git.iko.soy/typeable/nixpkgs
on master help sign in
nixpkgs-docs: when to prefer passthru.tests over installCheckPhase

And mention you can have either lightweight 'package' or more heavyweight 'NixOS' (module) tests. This was suggested at https://github.com/ryantm/nixpkgs-update/issues/260#issuecomment-821287971 and discussed further at https://github.com/NixOS/nixpkgs/pull/119731

commitd09e0be1c4876fe99340b03e56ebb941f93842c2 authorArnout Engelen <arnout@bzzt.net> dateSat 17 Apr 2021, 19:02 (6 years ago) parent28135ad
3 files changed+46−2
doc/contributing/coding-conventions.chapter.md+20−1 doc/stdenv/meta.chapter.md+24−1 doc/stdenv/stdenv.chapter.md+2
doc/contributing/coding-conventions.chapter.md+20−1
@@ -545,7 +545,26 @@ The following types of tests exists:
545545
546546Here in the nixpkgs manual we describe mostly _package tests_; for _module tests_ head over to the corresponding [section in the NixOS manual](https://nixos.org/manual/nixos/stable/#sec-nixos-tests).
547547
548−### Writing package tests {#ssec-package-tests-writing}
548+### Writing inline package tests {#ssec-inline-package-tests-writing}
549+
550+For very simple tests, they can be written inline:
551+
552+```nix
553+{ …, yq-go }:
554+
555+buildGoModule rec {
556+ …
557+
558+ passthru.tests = {
559+ simple = runCommand "${pname}-test" {} ''
560+ echo "test: 1" | ${yq-go}/bin/yq eval -j > $out
561+ [ "$(cat $out | tr -d $'\n ')" = '{"test":1}' ]
562+ '';
563+ };
564+}
565+```
566+
567+### Writing larger package tests {#ssec-package-tests-writing}
549568
550569This is an example using the `phoronix-test-suite` package with the current best practices.
551570
doc/stdenv/meta.chapter.md+24−1
@@ -134,7 +134,28 @@ Attribute Set `lib.platforms` defines [various common lists](https://github.com/
134134This attribute is special in that it is not actually under the `meta` attribute set but rather under the `passthru` attribute set. This is due to how `meta` attributes work, and the fact that they are supposed to contain only metadata, not derivations.
135135:::
136136
137−An attribute set with as values tests. A test is a derivation, which builds successfully when the test passes, and fails to build otherwise. A derivation that is a test needs to have `meta.timeout` defined.
137+An attribute set with tests as values. A test is a derivation that builds when the test passes and fails to build otherwise.
138+
139+You can run these tests with:
140+
141+```ShellSession
142+$ cd path/to/nixpkgs
143+$ nix-build -A your-package.tests
144+```
145+
146+#### Package tests
147+
148+Tests that are part of the source package are often executed in the `installCheckPhase`.
149+
150+Prefer `passthru.tests` for tests that are introduced in nixpkgs because:
151+
152+* `passthru.tests` tests the 'real' package, independently from the environment in which it was built
153+* we can run `passthru.tests` independently
154+* `installCheckPhase` adds overhead to each build
155+
156+For more on how to write and run package tests, see <xref linkend="sec-package-tests"/>.
157+
158+#### NixOS tests
138159
139160The NixOS tests are available as `nixosTests` in parameters of derivations. For instance, the OpenSMTPD derivation includes lines similar to:
140161
@@ -148,6 +169,8 @@ The NixOS tests are available as `nixosTests` in parameters of derivations. For
148169}
149170```
150171
172+NixOS tests run in a VM, so they are slower than regular package tests. For more information see [NixOS module tests](https://nixos.org/manual/nixos/stable/#sec-nixos-tests).
173+
151174### `timeout` {#var-meta-timeout}
152175
153176A timeout (in seconds) for building the derivation. If the derivation takes longer than this time to build, it can fail due to breaking the timeout. However, all computers do not have the same computing power, hence some builders may decide to apply a multiplicative factor to this value. When filling this value in, try to keep it approximately consistent with other values already present in `nixpkgs`.
doc/stdenv/stdenv.chapter.md+2
@@ -714,6 +714,8 @@ to `~/.gdbinit`. GDB will then be able to find debug information installed via `
714714
715715The installCheck phase checks whether the package was installed correctly by running its test suite against the installed directories. The default `installCheck` calls `make installcheck`.
716716
717+It is often better to add tests that are not part of the source distribution to `passthru.tests` (see <xref linkend="var-meta-tests"/>). This avoids adding overhead to every build and enables us to run them independently.
718+
717719#### Variables controlling the installCheck phase {#variables-controlling-the-installcheck-phase}
718720
719721##### `doInstallCheck` {#var-stdenv-doInstallCheck}