git.iko.soy/haskell.nix
on patch-1 help sign in
haskell.nix/docs/tutorials/materialization.md
258 lines 9.6 kB history raw
1# Materialization
3## What is materialization?
5Capturing and storing the Nix files for a project so that they do
6not need to be built (or checked). This allows us to cache the input
7of an IFD (import from derivation).
9## Why use materialization?
11Using functions like `project`, `cabalProject`, `stackProject`
12and `hackage-package` results in a lot of dependencies (all the
13dependencies of nix-tools for instance).
15* They can be slow to calculate (even if no work needs to be done it
16 is not unusual for it to take 5 seconds per project).
18* They can be slow to build (or download) on machines that do not
19 yet have them in the Nix store.
21* Hydra does not show progress because it does not provide feedback until it
22 has a list of jobs and the list of jobs cannot depend on the Nix expressions
23 being present (although this is often blamed on IFD it would be the same if
24 it wrote out JSON files and read them in)
26## When is it OK to materialize?
28* The Nix expressions are unlikely to change frequently (and when it does you
29 are happy to manually update it).
31* You are happy to script something to update the materialized Nix files
32 automatically.
34* You are certain that the IFD you materialize is not `system`-dependent. If it
35 was you'd obtain different Nix expressions depending on which `system` the
36 IFD was evaluated.
38## How can we materialize the Nix files?
40Lets say we want to build `hlint`. We might start with an `hlint.nix`
41file that looks like this:
43```nix
44let inherit (import ./. {}) sources nixpkgsArgs;
45 pkgs = import sources.nixpkgs nixpkgsArgs;
46 hlint = pkgs.haskell-nix.hackage-package {
47 compiler-nix-name = "ghc8102";
48 name = "hlint";
49 version = "2.2.11";
50 };
51in hlint
52```
54Building this may result in a lot of output, but if you build
55it again it should give just:
57```
58$ nix-build hlint.nix -A components.exes.hlint
59trace: No index state specified for hlint, using the latest index state that we know about (2021-01-04T00:00:00Z)!
60/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
61```
63To materialize the Nix files we need to take care to pin down the inputs. Stack
64projects have their inputs pinned through specifying the snapshot. For cabal
65projects this means we must specify the `index-state` of hackage we want to
66use:
68```nix
69let inherit (import ./. {}) sources nixpkgsArgs;
70 pkgs = import sources.nixpkgs nixpkgsArgs;
71 hlint = pkgs.haskell-nix.hackage-package {
72 compiler-nix-name = "ghc8102";
73 name = "hlint";
74 version = "2.2.11";
75 index-state = "2021-01-04T00:00:00Z";
76 };
77in hlint
78```
80Now if we build again we get a hint telling use how to calculate a suitable
81sha256 hash to turn the derivation containing the Nix files into a fixed-output
82derivation:
84```
85$ nix-build hlint.nix -A components.exes.hlint
86trace: To make project.plan-nix for hlint a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
87trace: To materialize project.plan-nix for hlint entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
88/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
90$ nix-build hlint.nix -A project.plan-nix.passthru.calculateMaterializedSha | bash
91trace: To make project.plan-nix for hlint a fixed-output derivation but not materialized, set `plan-sha256` to the output of the 'calculateMaterializedSha' script in 'passthru'.
92trace: To materialize project.plan-nix for hlint entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
9304hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7
94```
96For a Stack project all occurences of `plan-nix` and `plan-sha256` are replaced
97by `stack-nix` and `stack-sha256`, respectively. We can add the hash as
98`plan-sha256`:
100```nix
101let inherit (import ./. {}) sources nixpkgsArgs;
102 pkgs = import sources.nixpkgs nixpkgsArgs;
103 hlint = pkgs.haskell-nix.hackage-package {
104 compiler-nix-name = "ghc8102";
105 name = "hlint";
106 version = "2.2.11";
107 index-state = "2021-01-04T00:00:00Z";
108 plan-sha256 = "04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7";
109 };
110in hlint
111```
113Just adding the hash might help reuse of the cached Nix expressions, but Nix
114will still calculate all the dependencies (which can add seconds to `nix-build`
115and `nix-shell` commands when no other work is needed) and users who do not yet
116have the dependencies in their store will have to wait while they are built or
117downloaded.
119Running `nix-build` again gives us a hint on what we can do next:
121```
122$ nix-build hlint.nix -A components.exes.hlint
123trace: To materialize project.plan-nix for hlint entirely, pass a writable path as the `materialized` argument and run the 'updateMaterialized' script in 'passthru'.
124/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
125```
127To capture the Nix expressions we can do something like:
129```nix
130let inherit (import ./. {}) sources nixpkgsArgs;
131 pkgs = import sources.nixpkgs nixpkgsArgs;
132 hlint = pkgs.haskell-nix.hackage-package {
133 compiler-nix-name = "ghc8102";
134 name = "hlint";
135 version = "2.2.11";
136 index-state = "2021-01-04T00:00:00Z";
137 plan-sha256 = "04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7";
138 materialized = ./hlint.materialized;
139 };
140in hlint
141```
143Now we can copy the Nix files needed and build with:
145```
146$ nix-build hlint.nix 2>&1 | grep -om1 '/nix/store/.*-updateMaterialized' | bash
147$ nix-build hlint.nix -A components.exes.hlint
148building '/nix/store/wpxsgzl1z4jnhfqzmzg3xxv3ljpmzr5h-hlint-plan-to-nix-pkgs.drv'...
149/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
150```
152## How can we check `sha256` and `materialized` are up to date?
154Let's pretend we had to go back to `hlint` version `2.2.10`.
155We can tell haskell.nix to check the materialization either by:
157* Removing the materialization files with `rm -rf hlint.materialized`
159* Temporarily adding `checkMaterialization = true;`
161If we choose to add the `checkMaterialization` flag you would have:
163```nix
164let inherit (import ./. {}) sources nixpkgsArgs;
165 pkgs = import sources.nixpkgs nixpkgsArgs;
166 hlint = pkgs.haskell-nix.hackage-package {
167 compiler-nix-name = "ghc8102";
168 name = "hlint";
169 version = "2.2.10";
170 index-state = "2021-01-04T00:00:00Z";
171 plan-sha256 = "04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7";
172 materialized = ./hlint.materialized;
173 checkMaterialization = true;
174 };
175in hlint
176```
178This will fail and report the details of what is wrong and how to fix it:
180```
181$ nix-build hlint.nix -A components.exes.hlint
183...
185Calculated hash for hlint-plan-to-nix-pkgs was not 04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7. New hash is :
186 plan-sha256 = "0jsgdmii0a6b35sd42cpbc83s4sp4fbx8slphzvamq8n9x49i5b6";
187Materialized nix used for hlint-plan-to-nix-pkgs incorrect. To fix run: /nix/store/6wp0zzal40ls874f5ddpaac7qmii9y4z-updateMaterialized
188builder for '/nix/store/61a0vginv76w4p9ycyd628pjanav06pl-hlint-plan-to-nix-pkgs.drv' failed with exit code 1
189error: build of '/nix/store/61a0vginv76w4p9ycyd628pjanav06pl-hlint-plan-to-nix-pkgs.drv' failed
190(use '--show-trace' to show detailed location information)
191```
193Checking the materialization requires Nix to do all the work that
194materialization avoids. So while it might be tempting to leave
195`checkMaterialization = true` all the time, we would be better off just
196removing `materialized` and `plan-sha256`.
198## How can we update the Nix files with a script?
200We can simply put the commands we used earlier in a script:
202```nix
203#!/bin/sh
205# Output new plan-sha256
206nix-build hlint.nix -A project.plan-nix.passthru.calculateMaterializedSha | bash
208# Update materialized Nix expressions
209nix-build hlint.nix 2>&1 | grep -om1 '/nix/store/.*-updateMaterialized' | bash
210```
212## Can we skip making a copy and use `materialized = /nix/store/...`?
214Yes and it gives us the same speed improvement, however:
216* It does not help at all in `restricted-eval` mode (Hydra).
218* Users will still wind up building or downloading the dependencies
219 needed to build the Nix files (if they do not have them).
221For those reasons it might be best to make a copy instead
222of using the `/nix/store/...` path directly.
224If you really want to use the `/nix/store/...` path directly
225you should guard against the path not existing as passing in
226a non-existing path is now an error:
228```nix
229let inherit (import ./. {}) sources nixpkgsArgs;
230 pkgs = import sources.nixpkgs nixpkgsArgs;
231 hlintPlan = /nix/store/63k3f8bvsnag7v36vb3149208jyx61rk-hlint-plan-to-nix-pkgs;
232 hlint = pkgs.haskell-nix.hackage-package {
233 compiler-nix-name = "ghc8102";
234 name = "hlint";
235 version = "2.2.11";
236 index-state = "2021-01-04T00:00:00Z";
237 plan-sha256 = "04hdgqwpaswmyb0ili7fwi6czzihd6x0jlvivw52d1i7wv4gaqy7";
238 materialized = if __pathExists hlintPlan then hlintPlan else null;
239 };
240in hlint
241```
243Running when no building is needed is still slow in restricted evaluation mode.
245```
246$ time nix-build --option restrict-eval true -I . --option allowed-uris "https://github.com/NixOS https://github.com/input-output-hk" hlint.nix -A components.exes.hlint --show-trace
247/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
249real 0m4.463s
250user 0m4.440s
251sys 0m0.461s
252$ time nix-build hlint.nix -A components.exes.hlint
253/nix/store/2ybrfmcp79gg75ad4pr1cbxjak70yg8b-hlint-exe-hlint-2.2.11
255real 0m2.206s
256user 0m1.665s
257sys 0m0.332s
258```