Commit Graph

36 Commits

Author SHA1 Message Date
Robert Hensing
922556a3d3
Merge pull request #191355 from GenericNerdyUsername/fetchzip-include-hidden
fetchzip: don't error out if the directory inside the archive starts with a "."
2022-12-16 18:08:36 +01:00
GenericNerdyUsername
725f4bc5be
fetchzip: don't error out if the directory inside the archive starts with a "." 2022-12-16 16:26:03 +00:00
Hraban Luyat
24268b64b0 fetchzip: auto generate name from pname-version 2022-11-07 13:40:57 -05:00
Sandro Jäckel
01dfd3cb58
fetchzip: add pname+version support 2022-10-04 18:57:46 +02:00
Sergei Trofimovich
ffb456ae61 fetchzip: force UTF-8 compatibel locale to unpack non-ASCII symbols
musl and darwin support UTF-8 locales without any extras. As a result
unzip can unpack UTF-8 filenames there as is. But on glibc without
locale archive presence files get mangled as:

    deps/αβ -> deps/#U03b1#U03b2

This makes `fetchzip` fixed-output derivations unstable.

Tested this change to fail in `coq.src` which was generated in system
that mangles UTF-8 symbols:

    $ nix build -f. coq.src --rebuild -L
    source> trying https://github.com/coq/coq/archive/V8.15.2.zip
    source>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
    source>                                  Dload  Upload   Total   Spent    Left  Speed
    source>   0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
    source> 100 8945k  100 8945k    0     0  1513k      0  0:00:05  0:00:05 --:--:-- 1989k
    source> unpacking source archive /build/V8.15.2.zip
    error: hash mismatch in fixed-output derivation '/nix/store/hrnyykm7wgw8vxisgq7hc2bg5gr0y6s8-source.drv':
             specified: sha256-h81nFqkuvZkMR7YLHy7laTq5yOhjMW+w6rYzncxvyD4=
                got:    sha256-DTspmwyD3Evl1CUmvUy2MonbLGUezvsHN3prmP9eK2I=

Note: it means that some of existing caches for fixed output
derivations become incorrect. It should not break already cached
tarballs on cache.nixos.org thus the impact should not be widespread.
2022-06-11 11:25:58 +01:00
Artturin
b3caa2f1fe treewide: extraPostFetch -> postFetch 2022-05-23 17:18:50 +03:00
Artturin
5a6a31e54d fetchzip: extraPostFetch -> postFetch && tests 2022-05-22 21:18:52 +03:00
Jonathan Ringer
24cbda98f3 fetchzip: remove need for overrideAttrs 2022-02-26 08:58:08 -08:00
Artturin
5c37214abb fetchzip: recursively run chmod +w on the unpackDir
Fixes https://github.com/NixOS/nixpkgs/issues/136030
2021-09-08 03:58:18 +03:00
Alexey Shmalko
1893e2be4d
fetchzip: add extension parameter
fetchzip downloads the file from specified URL, renames it to basename
of that url, and then relies on unzip to do the unpacking.

The first consequence is that this requires URL to end with proper
extension—otherwise it will fail to unpack. This is not always the
case and input-fonts workarounds this by adding “&.zip” query
parameter (which is obviously a hack and is not guaranteed to work
with every URL).

The second consequence is that basename of the url must be a valid
filename. I’ve tried to build a custom configuration of input-fonts
and I get an error from mv that the filename is too long:

> trying https://input.djr.com/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip
>   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
>                                  Dload  Upload   Total   Spent    Left  Speed
> 100  406k  100  406k    0     0   230k      0  0:00:01  0:00:01 --:--:--  230k
> mv: failed to access '/build/?fontSelection=fourStyleFamily&regular=InputMonoNarrow-Regular&italic=InputMonoNarrow-Italic&bold=InputMonoNarrow-Bold&boldItalic=InputMonoNarrow-BoldItalic&a=0&g=0&i=topserif&l=serifs_round&zero=0&asterisk=height&braces=straight&preset=default&line-height=1.2&accept=I+do&email=&.zip': File name too long

We could use “name” parameter as the filename (that’s how it is used
in fetchurl). However, the previous attempt to do
so (fc01353703) was
reverted (24b5eb61eb) because of the
introduced regression—many fetchzip invocations use names without
extension (also the default name is just “source”).

This commit adds an optional “extension” parameter. If it is set,
fetchzip renames the downloaded file to “download.${extension}”
effectively solving both problems above without introducing a massive
regression.

This is a no-op for all existing packages.

Tested by updating my NixOS setup + the extra inputs-fonts
configuration mentioned above + tons of unstable emacs packages after
a nix-collect-garbage (3Gb downloaded) with this patch applied.
2021-08-10 19:57:08 +03:00
Sandro Jäckel
9f78dd4a1c fetchzip: accept urls
Required after 9bbfb26854
2021-04-07 02:41:25 +02:00
Jan Malakhovski
2b94e5d934 fetchzip: simplify postFetch as per suggestions
of @veprbl, @adisbladis, and @MetaDark
2021-02-05 13:08:57 +00:00
Jan Malakhovski
dcb6103bf2 fetchzip: fix extraPostFetch concatenation
4a5c49363a added some more commands after
`extraPostFetch` but concatenated them without a separating newline.

Which means, that since that commit

  fetchzip { ..., extraPostFetch = ''rm -f "$out"/some-file''; }

now actually runs the following shell command

  rm -f "$out"/some-file"chmod -R a-w "$out"

thus deleting "$out". Which is very unfortunate.

Especially since this actually happens on master for all `fetchFromBitbucket`
derivations. But since the results are fixed-output users bulding with hydra
cache enabled are not hitting this for not recently updated derivations yet.
2021-02-04 14:48:47 +00:00
Luke Granger-Brown
1f6d750097 fetchzip: get write permission on unpacked directory
This is a workaround for NixOS/nix#4295, which caused single-user Linux
Nix installations using sandboxed builds to start failing to build
fetchzip derivations after 4a5c49363a.

In short: removing write permissions for the entire directory is great,
except we then can't rename(2) it to the final Nix store path out of the
sandbox, because we don't have write permission on the directory and
thus cannot update the ".." directory entry.
2020-12-04 03:28:00 +00:00
MetaDark
4a5c49363a fetchzip: remove write permissions for unpacked files
Fixes https://github.com/NixOS/nixpkgs/issues/38649
2020-11-26 15:30:12 -05:00
Benjamin Hipple
259f3b3b68
Revert "fetchzip, fetchgit: cleanup handling of optional features and whitespace" 2020-03-26 00:35:40 -04:00
Benjamin Hipple
caff183402 fetchzip, fetchgit: cleanup handling of optional features and whitespace
No intended functional difference, just trying to polish to make this easier to
extend and more aligned with other nix conventions.
2020-02-08 21:51:46 -05:00
volth
f3282c8d1e treewide: remove unused variables (#63177)
* treewide: remove unused variables

* making ofborg happy
2019-06-16 19:59:05 +00:00
John Ericson
b97242238d fetchzip: Use unzip from buildPackages
Additionally, the manual path manipulation becomes no longer needed.
2018-09-21 12:55:32 -04:00
Eelco Dolstra
c3255fe8ec
fetchzip and friends: Set "name" to "source" by default
This makes them produce the same store paths as builtins.fetchgit,
builtins.fetchTarball etc. See
65b5f177b5.
2017-10-30 17:17:07 +01:00
luigy
eb6c45bcae Fix #27448 argument list too long in fetchzip for darwin.
Paired with @Ericson2314
2017-07-17 11:23:20 -04:00
Eelco Dolstra
97fc806a7b
nix-prefetch-zip: Remove
This script is not needed anymore since "nix-prefetch-url --unpack
<url>" and "nix-prefetch-url -A foo.src" (where "foo.src" is a
fetchzip / fetchFromGitHub call) work fine.
2017-02-16 11:28:23 +01:00
Domen Kožar
a01b6a0d07 fetchzip: improve error message 2016-05-17 17:32:53 +01:00
zimbatm
d2f3e250cf Merge pull request #8576 from obadz/nix-prefetch-zip
Add --ext option to nix-prefetch-zip
2016-02-26 00:57:30 +00:00
Vladimír Čunát
b91dcad4bc fetchFromBitBucket: auto-remove an impure file
Also fix the hash in goPackages.inflect, the only user of the fetcher ATM.
Closes #12002 (different `inflect` fix), fixes #12012.
Using fetchzip-derived functions is likely more efficient than fetchhg,
and it's lighter on dependencies (hash is the same as with fetchhg in this case).
2015-12-29 16:54:50 +01:00
Thomas Tuegel
55544c1e68 nix-prefetch-zip: keep downloaded file extension
Keep the correct file extension on the downloaded file so that
`unpackFile` will know how to unpack it correctly.
2015-12-19 09:32:22 -06:00
obadz
91a2973dc3 Add --ext option to nix-prefetch-zip 2015-07-15 16:15:41 +01:00
Charles Strahan
24b5eb61eb fetchzip: fix unpacking regression
This resolves a regression introduced in fc01353703, where providing a
name without a proper extension breaks existing uses of fetchzip (they
now fail to unpack). Of particular note, that commit broke all uses of
fetchFromGitHub because it uses a name like so: "${repo}-${rev}-src"

Fixes #5954
2015-01-25 20:16:21 -05:00
Charles Strahan
fc01353703 fetchzip: various fixes
This fixes:

 * Passing stripRoot.
 * Archives containing a single file.
 * Archives where the root folder has the same name as one of it's children.

Fixes #5851
2015-01-23 17:10:40 -05:00
Charles Strahan
4a3e689519 nix-prefetch-zip: support single files and recuring directories
See #5851
2015-01-23 17:10:40 -05:00
Charles Strahan
b6f9975b6a nix-prefetch-zip: unset $name
nix-shell leaks $name, breaking nix-prefetch-zip.
2015-01-22 21:18:16 -05:00
Michael Raskin
9ed097ab50 Fix fetchzip for actuall zip files 2014-12-12 15:16:57 +03:00
Eelco Dolstra
1a44dbbbb9 unpackFile: Always copy directories
If $src refers to a directory, then always copy it. Previously, we
checked the extension first, so if the directory had an extension like
.tar, unpackPhase would fail.
2014-08-09 12:44:49 +02:00
Charles Strahan
9bd8dffd77 require gnused for all prefetch scripts 2014-06-23 01:25:40 -04:00
Eelco Dolstra
ea36f3b868 fetchFromGitHub: Use .tar.gz instead of .zip
Also clean up the name attribute of fetchzip derivations a bit.
2014-05-09 15:53:44 +02:00
Eelco Dolstra
c8df888858 Add a function "fetchzip"
This function downloads and unpacks a file in one fixed-output
derivation. This is primarily useful for dynamically generated zip
files, such as GitHub's /archive URLs, where the unpacked content of
the zip file doesn't change, but the zip file itself may (e.g. due to
minor changes in the compression algorithm, or changes in timestamps).

Fetchzip is implemented by extending fetchurl with a "postFetch" hook
that is executed after the file has been downloaded. This hook can
thus perform arbitrary checks or transformations on the downloaded
file.
2014-05-08 15:30:17 +02:00