@@ -2,7 +2,7 @@
3## What is materialization?
3## What is materialization?
5−Capturing and storing the nix files for a project so that they do
5+Capturing and storing the Nix files for a project so that they do
6not need to be built (or checked). This allows us to cache the input
6not need to be built (or checked). This allows us to cache the input
7of an IFD (import from derivation).
7of an IFD (import from derivation).
@@ -16,143 +16,146 @@ dependencies of nix-tools for instance).
16 is not unusual for it to take 5 seconds per project).
16 is not unusual for it to take 5 seconds per project).
18* They can be slow to build (or download) on machines that do not
18* They can be slow to build (or download) on machines that do not
19− yet have them in the nix store.
19+ yet have them in the Nix store.
21−* Hydra does not show progress because it does not provide feedback
21+* Hydra does not show progress because it does not provide feedback until it
22− until it has a list of jobs and the list of jobs cannot depends
22+ has a list of jobs and the list of jobs cannot depend on the Nix expressions
23− on the nix being present (although this is often blamed on IFD
23+ being present (although this is often blamed on IFD it would be the same if
24− it would be the same if it wrote out JSON files and read them in)
24+ it wrote out JSON files and read them in)
26−## When is it ok to materialize?
26+## When is it OK to materialize?
28−* The nix is unlikely to change frequently (and when it does you
28+* The Nix expressions are unlikely to change frequently (and when it does you
29 are happy to manually update it).
29 are happy to manually update it).
31−* You are happy to script something to update the materialized
31+* You are happy to script something to update the materialized Nix files
32− nix files automatically.
32+ automatically.
33−* You are certain that the IFD you materialize is not `system`-dependent. If it was you'd
34− obtain different nix expressions depending on which `system` the IFD was evaluated.
36−## How can we materialize the nix files?
34+* You are certain that the IFD you materialize is not `system`-dependent. If it
35+ was you'd obtain different Nix expressions depending on which `system` the
38+## How can we materialize the Nix files?
38−Lets say we want to build `hlint`. We might start with an `hlint`
40+Lets say we want to build `hlint`. We might start with an `hlint.nix`
39file that looks like this:
41file that looks like this:
42let inherit (import ./. {}) sources nixpkgsArgs;
44let inherit (import ./. {}) sources nixpkgsArgs;
43 pkgs = import sources.nixpkgs nixpkgsArgs;
45 pkgs = import sources.nixpkgs nixpkgsArgs;
44 hlint = pkgs.haskell-nix.hackage-package {
46 hlint = pkgs.haskell-nix.hackage-package {
45− compiler-nix-name = "ghc884";
47+ compiler-nix-name = "ghc8102";
46 name = "hlint";
48 name = "hlint";
47 version = "2.2.11";
49 version = "2.2.11";
49−in hlint.components.exes.hlint
51+in hlint
52Building this may result in a lot of output, but if you build
54Building this may result in a lot of output, but if you build
53it again it should give just:
55it again it should give just:
56−$ nix-build hlint.nix
58+$ nix-build hlint.nix -A components.exes.hlint
57−trace: Using latest index state for hlint!
59+trace: No index state specified for hlint, using the latest index state that we know about (2021-01-04T00:00:00Z)!
58−trace: Using index-state: 2020-04-15T00:00:00Z for hlint
60+/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
59−/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
62−To materialize the nix files we need to take care to pin down the
63+To materialize the Nix files we need to take care to pin down the inputs. Stack
64+projects have their inputs pinned through specifying the snapshot. For cabal
63−inputs. For cabal projects this means we must specify the
65+projects this means we must specify the `index-state` of hackage we want to
64−`index-state` of hackage we want to use:
66+use:
67let inherit (import ./. {}) sources nixpkgsArgs;
69let inherit (import ./. {}) sources nixpkgsArgs;
68 pkgs = import sources.nixpkgs nixpkgsArgs;
70 pkgs = import sources.nixpkgs nixpkgsArgs;
69 hlint = pkgs.haskell-nix.hackage-package {
71 hlint = pkgs.haskell-nix.hackage-package {
70− compiler-nix-name = "ghc884";
72+ compiler-nix-name = "ghc8102";
71 name = "hlint";
73 name = "hlint";
72 version = "2.2.11";
74 version = "2.2.11";
73− index-state = "2020-04-15T00:00:00Z";
75+ index-state = "2021-01-04T00:00:00Z";
75−in hlint.components.exes.hlint
77+in hlint
78−Now if we build again we get a hint telling use how to
80+Now if we build again we get a hint telling use how to calculate a suitable
79−calculate a suitable sha256 hash to turn the derivation
80−containing the nix files into a fixed output derivation:
81+sha256 hash to turn the derivation containing the Nix files into a fixed-output
83−$ nix-build hlint.nix
85+$ nix-build hlint.nix -A components.exes.hlint
84−trace: Using index-state: 2020-04-15T00:00:00Z for hlint
86+trace: To make project.plan-nix for hlint a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
85−trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs/`
87+trace: To materialize project.plan-nix for hlint entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
86−/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
88+/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
88−$ nix-hash --base32 --type sha256 /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs/
90+$ nix-build hlint.nix -A project.plan-nix.passthru.calculateMaterializedSha | bash
89−02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4
91+trace: To make project.plan-nix for hlint a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
92+trace: To materialize project.plan-nix for hlint entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
93+04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7
92−We can add the hash as `plan-sha256` or (`stack-sha256` for
96+For a Stack project all occurences of `plan-nix` and `plan-sha256` are replaced
93−stack projects)
97+by `stack-nix` and `stack-sha256`, respectively. We can add the hash as
96let inherit (import ./. {}) sources nixpkgsArgs;
101let inherit (import ./. {}) sources nixpkgsArgs;
97 pkgs = import sources.nixpkgs nixpkgsArgs;
102 pkgs = import sources.nixpkgs nixpkgsArgs;
98 hlint = pkgs.haskell-nix.hackage-package {
103 hlint = pkgs.haskell-nix.hackage-package {
99− compiler-nix-name = "ghc884";
104+ compiler-nix-name = "ghc8102";
100 name = "hlint";
105 name = "hlint";
101 version = "2.2.11";
106 version = "2.2.11";
102− index-state = "2020-04-15T00:00:00Z";
107+ index-state = "2021-01-04T00:00:00Z";
103− plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
108+ plan-sha256 = "04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7";
105−in hlint.components.exes.hlint
110+in hlint
108−Just adding the hash might help reuse of the cached nix, but nix will
113+Just adding the hash might help reuse of the cached Nix expressions, but Nix
109−still calculate all the dependencies (which can add seconds to
114+will still calculate all the dependencies (which can add seconds to `nix-build`
110−`nix-build` and `nix-shell` commands when no other work is needed)
115+and `nix-shell` commands when no other work is needed) and users who do not yet
111−and users who do not yet have the dependencies in their store will have
116+have the dependencies in their store will have to wait while they are built or
112−to wait while they are built or downloaded.
117+downloaded.
114−Running nix build again gives us a hint on what we can do next:
119+Running `nix-build` again gives us a hint on what we can do next:
117−$ nix-build hlint.nix
122+$ nix-build hlint.nix -A components.exes.hlint
118−trace: Using index-state: 2020-04-15T00:00:00Z for hlint
123+trace: To materialize project.plan-nix for hlint entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
119−trace: To materialize, point `materialized` to a copy of /nix/store/kk047cqsjvbj4w8psv4l05abdcnyrqdc-hlint-plan-to-nix-pkgs
124+/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
120−/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
123−To capture the nix we can do something like:
127+To capture the Nix expressions we can do something like:
126let inherit (import ./. {}) sources nixpkgsArgs;
130let inherit (import ./. {}) sources nixpkgsArgs;
127 pkgs = import sources.nixpkgs nixpkgsArgs;
131 pkgs = import sources.nixpkgs nixpkgsArgs;
128 hlint = pkgs.haskell-nix.hackage-package {
132 hlint = pkgs.haskell-nix.hackage-package {
129− compiler-nix-name = "ghc884";
133+ compiler-nix-name = "ghc8102";
130 name = "hlint";
134 name = "hlint";
131 version = "2.2.11";
135 version = "2.2.11";
132− index-state = "2020-04-15T00:00:00Z";
136+ index-state = "2021-01-04T00:00:00Z";
133− plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
137+ plan-sha256 = "04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7";
134 materialized = ./hlint.materialized;
138 materialized = ./hlint.materialized;
136−in hlint.components.exes.hlint
140+in hlint
139−Now we can copy the nix files needed and build with:
143+Now we can copy the Nix files needed and build with:
142−$ cp -r /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs hlint.materialized
146+$ nix-build hlint.nix 2>&1 | grep -om1 '/nix/store/.*-updateMaterialized' | bash
143−$ nix-build hlint.nix
147+$ nix-build hlint.nix -A components.exes.hlint
144−/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
148+building '/nix/store/wpxsgzl1z4jnhfqzmzg3xxv3ljpmzr5h-hlint-plan-to-nix-pkgs.drv'...
149+/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
147−We may want to run `chmod -R +w hlint.materialized` as the files copied from the
148−store will be read only.
150## How can we check `sha256` and `materialized` are up to date?
152## How can we check `sha256` and `materialized` are up to date?
152Let's pretend we had to go back to `hlint` version `2.2.10`.
154Let's pretend we had to go back to `hlint` version `2.2.10`.
153We can tell haskell.nix to check the materialization either by:
155We can tell haskell.nix to check the materialization either by:
155* Removing the materialization files with `rm -rf hlint.materialized`
157* Removing the materialization files with `rm -rf hlint.materialized`
156* Temporarily adding `checkMaterialization = true;`
159* Temporarily adding `checkMaterialization = true;`
158If we choose to add the `checkMaterialization` flag you would have:
161If we choose to add the `checkMaterialization` flag you would have:
@@ -161,68 +164,50 @@ If we choose to add the `checkMaterialization` flag you would have:
161let inherit (import ./. {}) sources nixpkgsArgs;
164let inherit (import ./. {}) sources nixpkgsArgs;
162 pkgs = import sources.nixpkgs nixpkgsArgs;
165 pkgs = import sources.nixpkgs nixpkgsArgs;
163 hlint = pkgs.haskell-nix.hackage-package {
166 hlint = pkgs.haskell-nix.hackage-package {
164− compiler-nix-name = "ghc884";
167+ compiler-nix-name = "ghc8102";
165 name = "hlint";
168 name = "hlint";
166 version = "2.2.10";
169 version = "2.2.10";
167− index-state = "2020-04-15T00:00:00Z";
170+ index-state = "2021-01-04T00:00:00Z";
168− plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
171+ plan-sha256 = "04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7";
169 materialized = ./hlint.materialized;
172 materialized = ./hlint.materialized;
170 checkMaterialization = true;
173 checkMaterialization = true;
172−in hlint.components.exes.hlint
175+in hlint
175This will fail and report the details of what is wrong and how to fix it:
178This will fail and report the details of what is wrong and how to fix it:
178−$ nix-build hlint.nix
181+$ nix-build hlint.nix -A components.exes.hlint
182−Calculated hash for hlint-plan-to-nix-pkgs was not 02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4. New hash is :
185+Calculated hash for hlint-plan-to-nix-pkgs was not 04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7. New hash is :
183− plan-sha256 = "0zsi3wv92qax33ic4n5dfsqd1r9qam1k75za3c5jqgdxl3hy8vph";
186+ plan-sha256 = "0jsgdmii0a6b35sd42cpbc83s4sp4fbx8slphzvamq8n9x49i5b6";
184−Materialized nix used for hlint-plan-to-nix-pkgs incorrect. To fix run :
187+Materialized nix used for hlint-plan-to-nix-pkgs incorrect. To fix run: /nix/store/6wp0zzal40ls874f5ddpaac7qmii9y4z-updateMaterialized
185− rm -rf /Users/hamish/iohk/haskell.nix/hlint.materialized
186− cp -r /nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs /Users/hamish/iohk/haskell.nix/hlint.materialized
187− chmod -R +w /Users/hamish/iohk/haskell.nix/hlint.materialized
188−builder for '/nix/store/a5zmgfjfxahapw0q8hd2da5bg7knqvbx-hlint-plan-to-nix-pkgs.drv' failed with exit code 1
188+builder for '/nix/store/61a0vginv76w4p9ycyd628pjanav06pl-hlint-plan-to-nix-pkgs.drv' failed with exit code 1
189−error: build of '/nix/store/a5zmgfjfxahapw0q8hd2da5bg7knqvbx-hlint-plan-to-nix-pkgs.drv' failed
189+error: build of '/nix/store/61a0vginv76w4p9ycyd628pjanav06pl-hlint-plan-to-nix-pkgs.drv' failed
190(use '--show-trace' to show detailed location information)
190(use '--show-trace' to show detailed location information)
193−Checking the materialization requires nix to do all the work that materialization
193+Checking the materialization requires Nix to do all the work that
194−avoids. So while it might be tempting to leave `checkMaterialization = true` all
194+materialization avoids. So while it might be tempting to leave
195+`checkMaterialization = true` all the time, we would be better off just
195−the time, we would be better off just removing `materialized` and `plan-sha256`.
196+removing `materialized` and `plan-sha256`.
197−## How can we update the nix files with a script?
198+## How can we update the Nix files with a script?
199−There are versions of the functions (`project'`, `cabalProject'`,
200+We can simply put the commands we used earlier in a script:
200−`stackProject'` and `hackage-project`) that also return the nix as
201−`plan-nix` or `stack-nix`. By calling one of these functions without
202−the hash and materialized nix we can find out what nix files should be.
206−let inherit (import ./. {}) sources nixpkgsArgs;
203+#!/bin/sh
207− pkgs = import sources.nixpkgs nixpkgsArgs;
208− hlint = pkgs.haskell-nix.hackage-project {
209− compiler-nix-name = "ghc884";
212− index-state = "2020-04-15T00:00:00Z";
217−```
205+# Output new plan-sha256
218−$ nix-build hlint.nix -A plan-nix
206+nix-build hlint.nix -A project.plan-nix.passthru.calculateMaterializedSha | bash
219−trace: Using index-state: 2020-04-15T00:00:00Z for hlint
220−trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs/`
221−/nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs
224−We can have the script copy `$(nix-build hlint.nix -A plan-nix --no-out-link)`
208+# Update materialized Nix expressions
225−and use `nix-hash` to calculate the new value for `plan-sha256`.
209+nix-build hlint.nix 2>&1 | grep -om1 '/nix/store/.*-updateMaterialized' | bash
227## Can we skip making a copy and use `materialized = /nix/store/...`?
212## Can we skip making a copy and use `materialized = /nix/store/...`?
@@ -231,7 +216,7 @@ Yes and it gives us the same speed improvement, however:
231* It does not help at all in `restricted-eval` mode (Hydra).
216* It does not help at all in `restricted-eval` mode (Hydra).
233* Users will still wind up building or downloading the dependencies
218* Users will still wind up building or downloading the dependencies
234− needed to build the nix files (if they do not have them).
219+ needed to build the Nix files (if they do not have them).
236For those reasons it might be best to make a copy instead
221For those reasons it might be best to make a copy instead
237of using the `/nix/store/...` path directly.
222of using the `/nix/store/...` path directly.
@@ -243,30 +228,29 @@ a non-existing path is now an error:
244let inherit (import ./. {}) sources nixpkgsArgs;
229let inherit (import ./. {}) sources nixpkgsArgs;
245 pkgs = import sources.nixpkgs nixpkgsArgs;
230 pkgs = import sources.nixpkgs nixpkgsArgs;
246− hlintPlan = /nix/store/kk047cqsjvbj4w8psv4l05abdcnyrqdc-hlint-plan-to-nix-pkgs;
231+ hlintPlan = /nix/store/63k3f8bvsnag7v36vb3149208jyx61rk-hlint-plan-to-nix-pkgs;
247 hlint = pkgs.haskell-nix.hackage-package {
232 hlint = pkgs.haskell-nix.hackage-package {
248− compiler-nix-name = "ghc884";
233+ compiler-nix-name = "ghc8102";
249 name = "hlint";
234 name = "hlint";
250 version = "2.2.11";
235 version = "2.2.11";
251− index-state = "2020-04-15T00:00:00Z";
236+ index-state = "2021-01-04T00:00:00Z";
252− plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
237+ plan-sha256 = "04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7";
253 materialized = if __pathExists hlintPlan then hlintPlan else null;
238 materialized = if __pathExists hlintPlan then hlintPlan else null;
255−in hlint.components.exes.hlint
240+in hlint
258Running when no building is needed is still slow in restricted evaluation mode.
243Running when no building is needed is still slow in restricted evaluation mode.
261−$ time nix-build --option restrict-eval true -I . --option allowed-uris "https://github.com/NixOS https://github.com/input-output-hk" hlint.nix --show-trace
246+$ time nix-build --option restrict-eval true -I . --option allowed-uris "https://github.com/NixOS https://github.com/input-output-hk" hlint.nix -A components.exes.hlint --show-trace
262−trace: Using index-state: 2020-04-15T00:00:00Z for hlint
247+/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
263−/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
265real 0m4.463s
249real 0m4.463s
266user 0m4.440s
250user 0m4.440s
267sys 0m0.461s
251sys 0m0.461s
268−$ time nix-build hlint.nix
252+$ time nix-build hlint.nix -A components.exes.hlint
269−/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
253+/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
271real 0m2.206s
255real 0m2.206s
272user 0m1.665s
256user 0m1.665s