Merge pull request #136022 from hercules-ci/fetcher-testing

Fetcher testing

doc/builders/fetchers.chapter.md

@@ -1,8 +1,16 @@
 # Fetchers {#chap-pkgs-fetchers}
 
-When using Nix, you will frequently need to download source code and other files from the internet. Nixpkgs comes with a few helper functions that allow you to fetch fixed-output derivations in a structured way.
+When using Nix, you will frequently need to download source code and other files from the internet. For this purpose, Nix provides the [_fixed output derivation_](https://nixos.org/manual/nix/stable/#fixed-output-drvs) feature and Nixpkgs provides various functions that implement the actual fetching from various protocols and services.
 
-The two fetcher primitives are `fetchurl` and `fetchzip`. Both of these have two required arguments, a URL and a hash. The hash is typically `sha256`, although many more hash algorithms are supported. Nixpkgs contributors are currently recommended to use `sha256`. This hash will be used by Nix to identify your source. A typical usage of fetchurl is provided below.
+## Caveats
+
+Because fixed output derivations are _identified_ by their hash, a common mistake is to update a fetcher's URL or a version parameter, without updating the hash. **This will cause the old contents to be used.** So remember to always invalidate the hash argument.
+
+For those who develop and maintain fetcheres, a similar problem arises with changes to the implementation of a fetcher. These may cause a fixed output derivation to fail, but won't normally be caught by tests because the supposed output is already in the store or cache. For the purpose of testing, you can use a trick that is embodied by the [`invalidateFetcherByDrvHash`](#sec-pkgs-invalidateFetcherByDrvHash) function. It uses the derivation `name` to create a unique output path per fetcher implementation, defeating the caching precisely where it would be harmful.
+
+## `fetchurl` and `fetchzip` {#fetchurl}
+
+Two basic fetchers are `fetchurl` and `fetchzip`. Both of these have two required arguments, a URL and a hash. The hash is typically `sha256`, although many more hash algorithms are supported. Nixpkgs contributors are currently recommended to use `sha256`. This hash will be used by Nix to identify your source. A typical usage of fetchurl is provided below.
 
 ```nix
 { stdenv, fetchurl }:
@@ -20,7 +28,7 @@ The main difference between `fetchurl` and `fetchzip` is in how they store the c
 
 `fetchpatch` works very similarly to `fetchurl` with the same arguments expected. It expects patch files as a source and performs normalization on them before computing the checksum. For example it will remove comments or other unstable parts that are sometimes added by version control systems and can change over time.
 
-Other fetcher functions allow you to add source code directly from a VCS such as subversion or git. These are mostly straightforward nambes based on the name of the command used with the VCS system. Because they give you a working repository, they act most like `fetchzip`.
+Most other fetchers return a directory rather than a single file.
 
 ## `fetchsvn` {#fetchsvn}
 

doc/builders/special.xml

@@ -7,4 +7,5 @@
  </para>
  <xi:include href="special/fhs-environments.section.xml" />
  <xi:include href="special/mkshell.section.xml" />
+ <xi:include href="special/invalidateFetcherByDrvHash.section.xml" />
 </chapter>

doc/builders/special/invalidateFetcherByDrvHash.section.md

@@ -0,0 +1,31 @@
+
+## `invalidateFetcherByDrvHash` {#sec-pkgs-invalidateFetcherByDrvHash}
+
+Use the derivation hash to invalidate the output via name, for testing.
+
+Type: `(a@{ name, ... } -> Derivation) -> a -> Derivation`
+
+Normally, fixed output derivations can and should be cached by their output
+hash only, but for testing we want to re-fetch everytime the fetcher changes.
+
+Changes to the fetcher become apparent in the drvPath, which is a hash of
+how to fetch, rather than a fixed store path.
+By inserting this hash into the name, we can make sure to re-run the fetcher
+every time the fetcher changes.
+
+This relies on the assumption that Nix isn't clever enough to reuse its
+database of local store contents to optimize fetching.
+
+You might notice that the "salted" name derives from the normal invocation,
+not the final derivation. `invalidateFetcherByDrvHash` has to invoke the fetcher
+function twice: once to get a derivation hash, and again to produce the final
+fixed output derivation.
+
+Example:
+
+    tests.fetchgit = invalidateFetcherByDrvHash fetchgit {
+      name = "nix-source";
+      url = "https://github.com/NixOS/nix";
+      rev = "9d9dbe6ed05854e03811c361a3380e09183f4f4a";
+      sha256 = "sha256-7DszvbCNTjpzGRmpIVAWXk20P0/XTrWZ79KSOGLrUWY=";
+    };

pkgs/build-support/fetchgit/tests.nix

@@ -0,0 +1,10 @@
+{ invalidateFetcherByDrvHash, fetchgit, ... }:
+
+{
+  simple = invalidateFetcherByDrvHash fetchgit {
+    name = "nix-source";
+    url = "https://github.com/NixOS/nix";
+    rev = "9d9dbe6ed05854e03811c361a3380e09183f4f4a";
+    sha256 = "sha256-7DszvbCNTjpzGRmpIVAWXk20P0/XTrWZ79KSOGLrUWY=";
+  };
+}

pkgs/top-level/all-packages.nix

@@ -449,10 +449,12 @@ with pkgs;
 
   fetchfossil = callPackage ../build-support/fetchfossil { };
 
-  fetchgit = callPackage ../build-support/fetchgit {
+  fetchgit = (callPackage ../build-support/fetchgit {
     git = buildPackages.gitMinimal;
     cacert = buildPackages.cacert;
     git-lfs = buildPackages.git-lfs;
+  }) // { # fetchgit is a function, so we use // instead of passthru.
+    tests = callPackages ../build-support/fetchgit/tests.nix {};
   };
 
   fetchgitLocal = callPackage ../build-support/fetchgitlocal { };
@@ -602,6 +604,15 @@ with pkgs;
 
   installShellFiles = callPackage ../build-support/install-shell-files {};
 
+  # See doc/builders/special/invalidateFetcherByDrvHash.section.md or
+  # https://nixos.org/manual/nixpkgs/unstable/#sec-pkgs-invalidateFetcherByDrvHash
+  invalidateFetcherByDrvHash = f: args:
+    let
+      drvPath = (f args).drvPath;
+      # It's safe to discard the context, because we don't access the path.
+      salt = builtins.unsafeDiscardStringContext (lib.substring 0 12 (baseNameOf drvPath));
+    in f (args // { name = "${args.name or "source"}-salted-${salt}"; });
+
   lazydocker = callPackage ../tools/misc/lazydocker { };
 
   ld-is-cc-hook = makeSetupHook { name = "ld-is-cc-hook"; }