@@ -2,7 +2,7 @@
22
33## What is materialization?
44
5−Capturing and storing the nix files for a project so that they do
5+Capturing and storing the Nix files for a project so that they do
66not need to be built (or checked). This allows us to cache the input
77of an IFD (import from derivation).
88
@@ -16,143 +16,146 @@ dependencies of nix-tools for instance).
1616 is not unusual for it to take 5 seconds per project).
1717
1818* They can be slow to build (or download) on machines that do not
19− yet have them in the nix store.
19+ yet have them in the Nix store.
2020
21−* Hydra does not show progress because it does not provide feedback
22− until it has a list of jobs and the list of jobs cannot depends
23− on the nix being present (although this is often blamed on IFD
24− it would be the same if it wrote out JSON files and read them in)
21+* Hydra does not show progress because it does not provide feedback until it
22+ has a list of jobs and the list of jobs cannot depend on the Nix expressions
23+ being present (although this is often blamed on IFD it would be the same if
24+ it wrote out JSON files and read them in)
2525
26−## When is it ok to materialize?
26+## When is it OK to materialize?
2727
28−* The nix is unlikely to change frequently (and when it does you
28+* The Nix expressions are unlikely to change frequently (and when it does you
2929 are happy to manually update it).
3030
31−* You are happy to script something to update the materialized
32− nix files automatically.
33−* You are certain that the IFD you materialize is not `system`-dependent. If it was you'd
34− obtain different nix expressions depending on which `system` the IFD was evaluated.
31+* You are happy to script something to update the materialized Nix files
32+ automatically.
3533
36−## How can we materialize the nix files?
34+* You are certain that the IFD you materialize is not `system`-dependent. If it
35+ was you'd obtain different Nix expressions depending on which `system` the
36+ IFD was evaluated.
3737
38−Lets say we want to build `hlint`. We might start with an `hlint`
38+## How can we materialize the Nix files?
39+
40+Lets say we want to build `hlint`. We might start with an `hlint.nix`
3941file that looks like this:
4042
4143```nix
4244let inherit (import ./. {}) sources nixpkgsArgs;
4345 pkgs = import sources.nixpkgs nixpkgsArgs;
4446 hlint = pkgs.haskell-nix.hackage-package {
45− compiler-nix-name = "ghc884";
47+ compiler-nix-name = "ghc8102";
4648 name = "hlint";
4749 version = "2.2.11";
4850 };
49−in hlint.components.exes.hlint
51+in hlint
5052```
5153
5254Building this may result in a lot of output, but if you build
5355it again it should give just:
5456
5557```
56−$ nix-build hlint.nix
57−trace: Using latest index state for hlint!
58−trace: Using index-state: 2020-04-15T00:00:00Z for hlint
59−/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
58+$ nix-build hlint.nix -A components.exes.hlint
59+trace: No index state specified for hlint, using the latest index state that we know about (2021-01-04T00:00:00Z)!
60+/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
6061```
6162
62−To materialize the nix files we need to take care to pin down the
63−inputs. For cabal projects this means we must specify the
64−`index-state` of hackage we want to use:
63+To materialize the Nix files we need to take care to pin down the inputs. Stack
64+projects have their inputs pinned through specifying the snapshot. For cabal
65+projects this means we must specify the `index-state` of hackage we want to
66+use:
6567
6668```nix
6769let inherit (import ./. {}) sources nixpkgsArgs;
6870 pkgs = import sources.nixpkgs nixpkgsArgs;
6971 hlint = pkgs.haskell-nix.hackage-package {
70− compiler-nix-name = "ghc884";
72+ compiler-nix-name = "ghc8102";
7173 name = "hlint";
7274 version = "2.2.11";
73− index-state = "2020-04-15T00:00:00Z";
75+ index-state = "2021-01-04T00:00:00Z";
7476 };
75−in hlint.components.exes.hlint
77+in hlint
7678```
7779
78−Now if we build again we get a hint telling use how to
79−calculate a suitable sha256 hash to turn the derivation
80−containing the nix files into a fixed output derivation:
80+Now if we build again we get a hint telling use how to calculate a suitable
81+sha256 hash to turn the derivation containing the Nix files into a fixed-output
82+derivation:
8183
8284```
83−$ nix-build hlint.nix
84−trace: Using index-state: 2020-04-15T00:00:00Z for hlint
85−trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs/`
86−/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
87−
88−$ nix-hash --base32 --type sha256 /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs/
89−02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4
85+$ nix-build hlint.nix -A components.exes.hlint
86+trace: To make project.plan-nix for hlint a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
87+trace: To materialize project.plan-nix for hlint entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
88+/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
89+
90+$ nix-build hlint.nix -A project.plan-nix.passthru.calculateMaterializedSha | bash
91+trace: To make project.plan-nix for hlint a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
92+trace: To materialize project.plan-nix for hlint entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
93+04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7
9094```
9195
92−We can add the hash as `plan-sha256` or (`stack-sha256` for
93−stack projects)
96+For a Stack project all occurences of `plan-nix` and `plan-sha256` are replaced
97+by `stack-nix` and `stack-sha256`, respectively. We can add the hash as
98+`plan-sha256`:
9499
95100```nix
96101let inherit (import ./. {}) sources nixpkgsArgs;
97102 pkgs = import sources.nixpkgs nixpkgsArgs;
98103 hlint = pkgs.haskell-nix.hackage-package {
99− compiler-nix-name = "ghc884";
104+ compiler-nix-name = "ghc8102";
100105 name = "hlint";
101106 version = "2.2.11";
102− index-state = "2020-04-15T00:00:00Z";
103− plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
107+ index-state = "2021-01-04T00:00:00Z";
108+ plan-sha256 = "04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7";
104109 };
105−in hlint.components.exes.hlint
110+in hlint
106111```
107112
108−Just adding the hash might help reuse of the cached nix, but nix will
109−still calculate all the dependencies (which can add seconds to
110−`nix-build` and `nix-shell` commands when no other work is needed)
111−and users who do not yet have the dependencies in their store will have
112−to wait while they are built or downloaded.
113+Just adding the hash might help reuse of the cached Nix expressions, but Nix
114+will still calculate all the dependencies (which can add seconds to `nix-build`
115+and `nix-shell` commands when no other work is needed) and users who do not yet
116+have the dependencies in their store will have to wait while they are built or
117+downloaded.
113118
114−Running nix build again gives us a hint on what we can do next:
119+Running `nix-build` again gives us a hint on what we can do next:
115120
116121```
117−$ nix-build hlint.nix
118−trace: Using index-state: 2020-04-15T00:00:00Z for hlint
119−trace: To materialize, point `materialized` to a copy of /nix/store/kk047cqsjvbj4w8psv4l05abdcnyrqdc-hlint-plan-to-nix-pkgs
120−/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
122+$ nix-build hlint.nix -A components.exes.hlint
123+trace: To materialize project.plan-nix for hlint entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
124+/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
121125```
122126
123−To capture the nix we can do something like:
127+To capture the Nix expressions we can do something like:
124128
125129```nix
126130let inherit (import ./. {}) sources nixpkgsArgs;
127131 pkgs = import sources.nixpkgs nixpkgsArgs;
128132 hlint = pkgs.haskell-nix.hackage-package {
129− compiler-nix-name = "ghc884";
133+ compiler-nix-name = "ghc8102";
130134 name = "hlint";
131135 version = "2.2.11";
132− index-state = "2020-04-15T00:00:00Z";
133− plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
136+ index-state = "2021-01-04T00:00:00Z";
137+ plan-sha256 = "04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7";
134138 materialized = ./hlint.materialized;
135139 };
136−in hlint.components.exes.hlint
140+in hlint
137141```
138142
139−Now we can copy the nix files needed and build with:
143+Now we can copy the Nix files needed and build with:
140144
141145```
142−$ cp -r /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs hlint.materialized
143−$ nix-build hlint.nix
144−/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
146+$ nix-build hlint.nix 2>&1 | grep -om1 '/nix/store/.*-updateMaterialized' | bash
147+$ nix-build hlint.nix -A components.exes.hlint
148+building '/nix/store/wpxsgzl1z4jnhfqzmzg3xxv3ljpmzr5h-hlint-plan-to-nix-pkgs.drv'...
149+/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
145150```
146151
147−We may want to run `chmod -R +w hlint.materialized` as the files copied from the
148−store will be read only.
149−
150152## How can we check `sha256` and `materialized` are up to date?
151153
152154Let's pretend we had to go back to `hlint` version `2.2.10`.
153155We can tell haskell.nix to check the materialization either by:
154156
155157* Removing the materialization files with `rm -rf hlint.materialized`
158+
156159* Temporarily adding `checkMaterialization = true;`
157160
158161If we choose to add the `checkMaterialization` flag you would have:
@@ -161,68 +164,50 @@ If we choose to add the `checkMaterialization` flag you would have:
161164let inherit (import ./. {}) sources nixpkgsArgs;
162165 pkgs = import sources.nixpkgs nixpkgsArgs;
163166 hlint = pkgs.haskell-nix.hackage-package {
164− compiler-nix-name = "ghc884";
167+ compiler-nix-name = "ghc8102";
165168 name = "hlint";
166169 version = "2.2.10";
167− index-state = "2020-04-15T00:00:00Z";
168− plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
170+ index-state = "2021-01-04T00:00:00Z";
171+ plan-sha256 = "04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7";
169172 materialized = ./hlint.materialized;
170173 checkMaterialization = true;
171174 };
172−in hlint.components.exes.hlint
175+in hlint
173176```
174177
175178This will fail and report the details of what is wrong and how to fix it:
176179
177180```
178−$ nix-build hlint.nix
181+$ nix-build hlint.nix -A components.exes.hlint
179182
180183...
181184
182−Calculated hash for hlint-plan-to-nix-pkgs was not 02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4. New hash is :
183− plan-sha256 = "0zsi3wv92qax33ic4n5dfsqd1r9qam1k75za3c5jqgdxl3hy8vph";
184−Materialized nix used for hlint-plan-to-nix-pkgs incorrect. To fix run :
185− rm -rf /Users/hamish/iohk/haskell.nix/hlint.materialized
186− cp -r /nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs /Users/hamish/iohk/haskell.nix/hlint.materialized
187− chmod -R +w /Users/hamish/iohk/haskell.nix/hlint.materialized
188−builder for '/nix/store/a5zmgfjfxahapw0q8hd2da5bg7knqvbx-hlint-plan-to-nix-pkgs.drv' failed with exit code 1
189−error: build of '/nix/store/a5zmgfjfxahapw0q8hd2da5bg7knqvbx-hlint-plan-to-nix-pkgs.drv' failed
185+Calculated hash for hlint-plan-to-nix-pkgs was not 04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7. New hash is :
186+ plan-sha256 = "0jsgdmii0a6b35sd42cpbc83s4sp4fbx8slphzvamq8n9x49i5b6";
187+Materialized nix used for hlint-plan-to-nix-pkgs incorrect. To fix run: /nix/store/6wp0zzal40ls874f5ddpaac7qmii9y4z-updateMaterialized
188+builder for '/nix/store/61a0vginv76w4p9ycyd628pjanav06pl-hlint-plan-to-nix-pkgs.drv' failed with exit code 1
189+error: build of '/nix/store/61a0vginv76w4p9ycyd628pjanav06pl-hlint-plan-to-nix-pkgs.drv' failed
190190(use '--show-trace' to show detailed location information)
191191```
192192
193−Checking the materialization requires nix to do all the work that materialization
194−avoids. So while it might be tempting to leave `checkMaterialization = true` all
195−the time, we would be better off just removing `materialized` and `plan-sha256`.
193+Checking the materialization requires Nix to do all the work that
194+materialization avoids. So while it might be tempting to leave
195+`checkMaterialization = true` all the time, we would be better off just
196+removing `materialized` and `plan-sha256`.
196197
197−## How can we update the nix files with a script?
198+## How can we update the Nix files with a script?
198199
199−There are versions of the functions (`project'`, `cabalProject'`,
200−`stackProject'` and `hackage-project`) that also return the nix as
201−`plan-nix` or `stack-nix`. By calling one of these functions without
202−the hash and materialized nix we can find out what nix files should be.
203−For instance:
200+We can simply put the commands we used earlier in a script:
204201
205202```nix
206−let inherit (import ./. {}) sources nixpkgsArgs;
207− pkgs = import sources.nixpkgs nixpkgsArgs;
208− hlint = pkgs.haskell-nix.hackage-project {
209− compiler-nix-name = "ghc884";
210− name = "hlint";
211− version = "2.2.10";
212− index-state = "2020-04-15T00:00:00Z";
213− };
214−in hlint
215−```
203+#!/bin/sh
216204
217−```
218−$ nix-build hlint.nix -A plan-nix
219−trace: Using index-state: 2020-04-15T00:00:00Z for hlint
220−trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs/`
221−/nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs
222−```
205+# Output new plan-sha256
206+nix-build hlint.nix -A project.plan-nix.passthru.calculateMaterializedSha | bash
223207
224−We can have the script copy `$(nix-build hlint.nix -A plan-nix --no-out-link)`
225−and use `nix-hash` to calculate the new value for `plan-sha256`.
208+# Update materialized Nix expressions
209+nix-build hlint.nix 2>&1 | grep -om1 '/nix/store/.*-updateMaterialized' | bash
210+```
226211
227212## Can we skip making a copy and use `materialized = /nix/store/...`?
228213
@@ -231,7 +216,7 @@ Yes and it gives us the same speed improvement, however:
231216* It does not help at all in `restricted-eval` mode (Hydra).
232217
233218* Users will still wind up building or downloading the dependencies
234− needed to build the nix files (if they do not have them).
219+ needed to build the Nix files (if they do not have them).
235220
236221For those reasons it might be best to make a copy instead
237222of using the `/nix/store/...` path directly.
@@ -243,30 +228,29 @@ a non-existing path is now an error:
243228```nix
244229let inherit (import ./. {}) sources nixpkgsArgs;
245230 pkgs = import sources.nixpkgs nixpkgsArgs;
246− hlintPlan = /nix/store/kk047cqsjvbj4w8psv4l05abdcnyrqdc-hlint-plan-to-nix-pkgs;
231+ hlintPlan = /nix/store/63k3f8bvsnag7v36vb3149208jyx61rk-hlint-plan-to-nix-pkgs;
247232 hlint = pkgs.haskell-nix.hackage-package {
248− compiler-nix-name = "ghc884";
233+ compiler-nix-name = "ghc8102";
249234 name = "hlint";
250235 version = "2.2.11";
251− index-state = "2020-04-15T00:00:00Z";
252− plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
236+ index-state = "2021-01-04T00:00:00Z";
237+ plan-sha256 = "04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7";
253238 materialized = if __pathExists hlintPlan then hlintPlan else null;
254239 };
255−in hlint.components.exes.hlint
240+in hlint
256241```
257242
258243Running when no building is needed is still slow in restricted evaluation mode.
259244
260245```
261−$ time nix-build --option restrict-eval true -I . --option allowed-uris "https://github.com/NixOS https://github.com/input-output-hk" hlint.nix --show-trace
262−trace: Using index-state: 2020-04-15T00:00:00Z for hlint
263−/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
246+$ time nix-build --option restrict-eval true -I . --option allowed-uris "https://github.com/NixOS https://github.com/input-output-hk" hlint.nix -A components.exes.hlint --show-trace
247+/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
264248
265249real 0m4.463s
266250user 0m4.440s
267251sys 0m0.461s
268−$ time nix-build hlint.nix
269−/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
252+$ time nix-build hlint.nix -A components.exes.hlint
253+/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
270254
271255real 0m2.206s
272256user 0m1.665s