git.iko.soy/typeable/nixpkgs
on master help sign in
Merge pull request #119731 from raboof/document-preferring-passthru-tests-over-installChecks

nixpkgs-docs: when to prefer passthru.tests over installCheckPhase

commitc580b69c108d61e6dd95d08f4d636458001cc9f4 authorArnout Engelen <arnout@bzzt.net> dateSat 14 Aug 2021, 10:04 (5 years ago) parent28135add09e0be
3 files changed+46−2
doc/contributing/coding-conventions.chapter.md+20−1 doc/stdenv/meta.chapter.md+24−1 doc/stdenv/stdenv.chapter.md+2
doc/contributing/coding-conventions.chapter.md+20−1
@@ -545,7 +545,26 @@ The following types of tests exists:
545545
546546Here in the nixpkgs manual we describe mostly _package tests_; for _module tests_ head over to the corresponding [section in the NixOS manual](https://nixos.org/manual/nixos/stable/#sec-nixos-tests).
547547
548−### Writing package tests {#ssec-package-tests-writing}
548+### Writing inline package tests {#ssec-inline-package-tests-writing}
549+
550+For very simple tests, they can be written inline:
551+
552+```nix
553+{ …, yq-go }:
554+
555+buildGoModule rec {
556+ …
557+
558+ passthru.tests = {
559+ simple = runCommand "${pname}-test" {} ''
560+ echo "test: 1" | ${yq-go}/bin/yq eval -j > $out
561+ [ "$(cat $out | tr -d $'\n ')" = '{"test":1}' ]
562+ '';
563+ };
564+}
565+```
566+
567+### Writing larger package tests {#ssec-package-tests-writing}
549568
550569This is an example using the `phoronix-test-suite` package with the current best practices.
551570
doc/stdenv/meta.chapter.md+24−1
@@ -134,7 +134,28 @@ Attribute Set `lib.platforms` defines [various common lists](https://github.com/
134134This attribute is special in that it is not actually under the `meta` attribute set but rather under the `passthru` attribute set. This is due to how `meta` attributes work, and the fact that they are supposed to contain only metadata, not derivations.
135135:::
136136
137−An attribute set with as values tests. A test is a derivation, which builds successfully when the test passes, and fails to build otherwise. A derivation that is a test needs to have `meta.timeout` defined.
137+An attribute set with tests as values. A test is a derivation that builds when the test passes and fails to build otherwise.
138+
139+You can run these tests with:
140+
141+```ShellSession
142+$ cd path/to/nixpkgs
143+$ nix-build -A your-package.tests
144+```
145+
146+#### Package tests
147+
148+Tests that are part of the source package are often executed in the `installCheckPhase`.
149+
150+Prefer `passthru.tests` for tests that are introduced in nixpkgs because:
151+
152+* `passthru.tests` tests the 'real' package, independently from the environment in which it was built
153+* we can run `passthru.tests` independently
154+* `installCheckPhase` adds overhead to each build
155+
156+For more on how to write and run package tests, see <xref linkend="sec-package-tests"/>.
157+
158+#### NixOS tests
138159
139160The NixOS tests are available as `nixosTests` in parameters of derivations. For instance, the OpenSMTPD derivation includes lines similar to:
140161
@@ -148,6 +169,8 @@ The NixOS tests are available as `nixosTests` in parameters of derivations. For
148169}
149170```
150171
172+NixOS tests run in a VM, so they are slower than regular package tests. For more information see [NixOS module tests](https://nixos.org/manual/nixos/stable/#sec-nixos-tests).
173+
151174### `timeout` {#var-meta-timeout}
152175
153176A timeout (in seconds) for building the derivation. If the derivation takes longer than this time to build, it can fail due to breaking the timeout. However, all computers do not have the same computing power, hence some builders may decide to apply a multiplicative factor to this value. When filling this value in, try to keep it approximately consistent with other values already present in `nixpkgs`.
doc/stdenv/stdenv.chapter.md+2
@@ -714,6 +714,8 @@ to `~/.gdbinit`. GDB will then be able to find debug information installed via `
714714
715715The installCheck phase checks whether the package was installed correctly by running its test suite against the installed directories. The default `installCheck` calls `make installcheck`.
716716
717+It is often better to add tests that are not part of the source distribution to `passthru.tests` (see <xref linkend="var-meta-tests"/>). This avoids adding overhead to every build and enables us to run them independently.
718+
717719#### Variables controlling the installCheck phase {#variables-controlling-the-installcheck-phase}
718720
719721##### `doInstallCheck` {#var-stdenv-doInstallCheck}