git.iko.soy/haskell.nix
on bc4400d help sign in
haskell.nix/docs/tutorials/materialization.md
274 lines 9.9 kB history raw
1# Materialization
3## What is materialization?
5Capturing and storing the nix files for a project so that they do
6not need to be built (or checked). This allows us to cache the input
7of an IFD (import from derivation).
9## Why use materialization?
11Using functions like `project`, `cabalProject`, `stackProject`
12and `hackage-package` results in a lot of dependencies (all the
13dependencies of nix-tools for instance).
15* They can be slow to calculate (even if no work needs to be done it
16 is not unusual for it to take 5 seconds per project).
18* They can be slow to build (or download) on machines that do not
19 yet have them in the nix store.
21* Hydra does not show progress because it does not provide feedback
22 until it has a list of jobs and the list of jobs cannot depends
23 on the nix being present (although this is often blamed on IFD
24 it would be the same if it wrote out JSON files and read them in)
26## When is it ok to materialize?
28* The nix is unlikely to change frequently (and when it does you
29 are happy to manually update it).
31* You are happy to script something to update the materialized
32 nix files automatically.
33* You are certain that the IFD you materialize is not `system`-dependent. If it was you'd
34 obtain different nix expressions depending on which `system` the IFD was evaluated.
36## How can we materialize the nix files?
38Lets say we want to build `hlint`. We might start with an `hlint`
39file that looks like this:
41```nix
42let inherit (import ./. {}) sources nixpkgsArgs;
43 pkgs = import sources.nixpkgs nixpkgsArgs;
44 hlint = pkgs.haskell-nix.hackage-package {
45 compiler-nix-name = "ghc884";
46 name = "hlint";
47 version = "2.2.11";
48 };
49in hlint.components.exes.hlint
50```
52Building this may result in a lot of output, but if you build
53it again it should give just:
55```
56$ nix-build hlint.nix
57trace: Using latest index state for hlint!
58trace: Using index-state: 2020-04-15T00:00:00Z for hlint
59/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
60```
62To materialize the nix files we need to take care to pin down the
63inputs. For cabal projects this means we must specify the
64`index-state` of hackage we want to use:
66```nix
67let inherit (import ./. {}) sources nixpkgsArgs;
68 pkgs = import sources.nixpkgs nixpkgsArgs;
69 hlint = pkgs.haskell-nix.hackage-package {
70 compiler-nix-name = "ghc884";
71 name = "hlint";
72 version = "2.2.11";
73 index-state = "2020-04-15T00:00:00Z";
74 };
75in hlint.components.exes.hlint
76```
78Now if we build again we get a hint telling use how to
79calculate a suitable sha256 hash to turn the derivation
80containing the nix files into a fixed output derivation:
82```
83$ nix-build hlint.nix
84trace: Using index-state: 2020-04-15T00:00:00Z for hlint
85trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs/`
86/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
88$ nix-hash --base32 --type sha256 /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs/
8902hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4
90```
92We can add the hash as `plan-sha256` or (`stack-sha256` for
93stack projects)
95```nix
96let inherit (import ./. {}) sources nixpkgsArgs;
97 pkgs = import sources.nixpkgs nixpkgsArgs;
98 hlint = pkgs.haskell-nix.hackage-package {
99 compiler-nix-name = "ghc884";
100 name = "hlint";
101 version = "2.2.11";
102 index-state = "2020-04-15T00:00:00Z";
103 plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
104 };
105in hlint.components.exes.hlint
106```
108Just adding the hash might help reuse of the cached nix, but nix will
109still calculate all the dependencies (which can add seconds to
110`nix-build` and `nix-shell` commands when no other work is needed)
111and users who do not yet have the dependencies in their store will have
112to wait while they are built or downloaded.
114Running nix build again gives us a hint on what we can do next:
116```
117$ nix-build hlint.nix
118trace: Using index-state: 2020-04-15T00:00:00Z for hlint
119trace: To materialize, point `materialized` to a copy of /nix/store/kk047cqsjvbj4w8psv4l05abdcnyrqdc-hlint-plan-to-nix-pkgs
120/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
121```
123To capture the nix we can do something like:
125```nix
126let inherit (import ./. {}) sources nixpkgsArgs;
127 pkgs = import sources.nixpkgs nixpkgsArgs;
128 hlint = pkgs.haskell-nix.hackage-package {
129 compiler-nix-name = "ghc884";
130 name = "hlint";
131 version = "2.2.11";
132 index-state = "2020-04-15T00:00:00Z";
133 plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
134 materialized = ./hlint.materialized;
135 };
136in hlint.components.exes.hlint
137```
139Now we can copy the nix files needed and build with:
141```
142$ cp -r /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs hlint.materialized
143$ nix-build hlint.nix
144/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
145```
147We may want to run `chmod -R +w hlint.materialized` as the files copied from the
148store will be read only.
150## How can we check `sha256` and `materialized` are up to date?
152Let's pretend we had to go back to `hlint` version `2.2.10`.
153We can tell haskell.nix to check the materialization either by:
155* Removing the materialization files with `rm -rf hlint.materialized`
156* Temporarily adding `checkMaterialization = true;`
158If we choose to add the `checkMaterialization` flag you would have:
160```nix
161let inherit (import ./. {}) sources nixpkgsArgs;
162 pkgs = import sources.nixpkgs nixpkgsArgs;
163 hlint = pkgs.haskell-nix.hackage-package {
164 compiler-nix-name = "ghc884";
165 name = "hlint";
166 version = "2.2.10";
167 index-state = "2020-04-15T00:00:00Z";
168 plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
169 materialized = ./hlint.materialized;
170 checkMaterialization = true;
171 };
172in hlint.components.exes.hlint
173```
175This will fail and report the details of what is wrong and how to fix it:
177```
178$ nix-build hlint.nix
180...
182Calculated hash for hlint-plan-to-nix-pkgs was not 02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4. New hash is :
183 plan-sha256 = "0zsi3wv92qax33ic4n5dfsqd1r9qam1k75za3c5jqgdxl3hy8vph";
184Materialized nix used for hlint-plan-to-nix-pkgs incorrect. To fix run :
185 rm -rf /Users/hamish/iohk/haskell.nix/hlint.materialized
186 cp -r /nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs /Users/hamish/iohk/haskell.nix/hlint.materialized
187 chmod -R +w /Users/hamish/iohk/haskell.nix/hlint.materialized
188builder for '/nix/store/a5zmgfjfxahapw0q8hd2da5bg7knqvbx-hlint-plan-to-nix-pkgs.drv' failed with exit code 1
189error: build of '/nix/store/a5zmgfjfxahapw0q8hd2da5bg7knqvbx-hlint-plan-to-nix-pkgs.drv' failed
190(use '--show-trace' to show detailed location information)
191```
193Checking the materialization requires nix to do all the work that materialization
194avoids. So while it might be tempting to leave `checkMaterialization = true` all
195the time, we would be better off just removing `materialized` and `plan-sha256`.
197## How can we update the nix files with a script?
199There are versions of the functions (`project'`, `cabalProject'`,
200`stackProject'` and `hackage-project`) that also return the nix as
201`plan-nix` or `stack-nix`. By calling one of these functions without
202the hash and materialized nix we can find out what nix files should be.
203For instance:
205```nix
206let inherit (import ./. {}) sources nixpkgsArgs;
207 pkgs = import sources.nixpkgs nixpkgsArgs;
208 hlint = pkgs.haskell-nix.hackage-project {
209 compiler-nix-name = "ghc884";
210 name = "hlint";
211 version = "2.2.10";
212 index-state = "2020-04-15T00:00:00Z";
213 };
214in hlint
215```
217```
218$ nix-build hlint.nix -A plan-nix
219trace: Using index-state: 2020-04-15T00:00:00Z for hlint
220trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs/`
221/nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs
222```
224We can have the script copy `$(nix-build hlint.nix -A plan-nix --no-out-link)`
225and use `nix-hash` to calculate the new value for `plan-sha256`.
227## Can we skip making a copy and use `materialized = /nix/store/...`?
229Yes and it gives us the same speed improvement, however:
231* It does not help at all in `restricted-eval` mode (Hydra).
233* Users will still wind up building or downloading the dependencies
234 needed to build the nix files (if they do not have them).
236For those reasons it might be best to make a copy instead
237of using the `/nix/store/...` path directly.
239If you really want to use the `/nix/store/...` path directly
240you should guard against the path not existing as passing in
241a non-existing path is now an error:
243```nix
244let inherit (import ./. {}) sources nixpkgsArgs;
245 pkgs = import sources.nixpkgs nixpkgsArgs;
246 hlintPlan = /nix/store/kk047cqsjvbj4w8psv4l05abdcnyrqdc-hlint-plan-to-nix-pkgs;
247 hlint = pkgs.haskell-nix.hackage-package {
248 compiler-nix-name = "ghc884";
249 name = "hlint";
250 version = "2.2.11";
251 index-state = "2020-04-15T00:00:00Z";
252 plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
253 materialized = if __pathExists hlintPlan then hlintPlan else null;
254 };
255in hlint.components.exes.hlint
256```
258Running when no building is needed is still slow in restricted evaluation mode.
260```
261$ time nix-build --option restrict-eval true -I . --option allowed-uris "https://github.com/NixOS https://github.com/input-output-hk" hlint.nix --show-trace
262trace: Using index-state: 2020-04-15T00:00:00Z for hlint
263/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
265real 0m4.463s
266user 0m4.440s
267sys 0m0.461s
268$ time nix-build hlint.nix
269/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
271real 0m2.206s
272user 0m1.665s
273sys 0m0.332s
274```