git.iko.soy/haskell.nix
on c773606 help sign in
haskell.nix/docs/tutorials/materialization.md
267 lines 9.7 kB history raw
1# Materialization
3## What is materialization?
5Capturing and storing the nix files for a project so that they do
6not need to be built (or checked). This allows us to cache the input
7of an IFD (import from derviation).
9## Why use materialization?
11Using functions like `project`, `cabalProject`, `stackProject`
12and `hackage-package` results in a lot of dependencies (all the
13dependencies of nix-tools for instance).
15* They can be slow to calculate (even if no work needs to be done it
16 is not unusual for it to take 5 seconds per project).
18* They can be slow to build (or download) on machines that do not
19 yet have them in the nix store.
21* Hydra does not show progress because it does not provide feedback
22 until it has a list of jobs and the list of jobs cannot depends
23 on the nix being present (although this is often blamed on IFD
24 it would be the same if it wrote out JSON files and read them in)
26## When is it ok to materialize?
28* The nix is unlikely to change frequently (and when it does you
29 are happy to manually update it).
31* You are happy to script something to update the materialized
32 nix files automatically.
33* You are certain that the IFD you materialize is not `system`-dependent. If it was you'd
34 obtain different nix expressions depending on which `system` the IFD was evaluated.
36## How can we materialize the nix files?
38Lets say we want to build `hlint`. We might start with an `hlint`
39file that looks like this:
41```nix
42let inherit (import ./. {}) sources nixpkgsArgs;
43 pkgs = import sources.nixpkgs-default nixpkgsArgs;
44 hlint = pkgs.haskell-nix.hackage-package {
45 name = "hlint";
46 version = "2.2.11";
47 };
48in hlint.components.exes.hlint
49```
51Building this may result in a lot of output, but if you build
52it again it should give just:
54```
55$ nix-build hlint.nix
56trace: Using latest index state for hlint!
57trace: Using index-state: 2020-04-15T00:00:00Z for hlint
58/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
59```
61To materialize the nix files we need to take care to pin down the
62inputs. For cabal projects this means we must specify the
63`index-state` of hackage we want to use:
65```nix
66let inherit (import ./. {}) sources nixpkgsArgs;
67 pkgs = import sources.nixpkgs-default nixpkgsArgs;
68 hlint = pkgs.haskell-nix.hackage-package {
69 name = "hlint";
70 version = "2.2.11";
71 index-state = "2020-04-15T00:00:00Z";
72 };
73in hlint.components.exes.hlint
74```
76Now if we build again we get a hint telling use how to
77calculate a suitable sha256 hash to turn the derivation
78containing the nix files into a fixed output derivation:
80```
81$ nix-build hlint.nix
82trace: Using index-state: 2020-04-15T00:00:00Z for hlint
83trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs/`
84/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
86$ nix-hash --base32 --type sha256 /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs/
8702hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4
88```
90We can add the hash as `plan-sha256` or (`stack-sha256` for
91stack projects)
93```nix
94let inherit (import ./. {}) sources nixpkgsArgs;
95 pkgs = import sources.nixpkgs-default nixpkgsArgs;
96 hlint = pkgs.haskell-nix.hackage-package {
97 name = "hlint";
98 version = "2.2.11";
99 index-state = "2020-04-15T00:00:00Z";
100 plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
101 };
102in hlint.components.exes.hlint
103```
105Just adding the hash might help reuse of the cached nix, but nix will
106still calculate all the dependencies (which can add seconds to
107`nix-build` and `nix-shell` commands when no other work is needed)
108and users who do not yet have the dependencies in their store will have
109to wait while they are built or downloaded.
111Running nix build again gives us a hint on what we can do next:
113```
114$ nix-build hlint.nix
115trace: Using index-state: 2020-04-15T00:00:00Z for hlint
116trace: To materialize, point `materialized` to a copy of /nix/store/kk047cqsjvbj4w8psv4l05abdcnyrqdc-hlint-plan-to-nix-pkgs
117/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
118```
120To capture the nix we can do something like:
122```nix
123let inherit (import ./. {}) sources nixpkgsArgs;
124 pkgs = import sources.nixpkgs-default nixpkgsArgs;
125 hlint = pkgs.haskell-nix.hackage-package {
126 name = "hlint";
127 version = "2.2.11";
128 index-state = "2020-04-15T00:00:00Z";
129 plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
130 materialized = ./hlint.materialized;
131 };
132in hlint.components.exes.hlint
133```
135Now we can copy the nix files needed and build with:
137```
138$ cp -r /nix/store/8z6p4237rin3c6c1lmjwshmj8rdqrhw2-hlint-plan-to-nix-pkgs hlint.materialized
139$ nix-build hlint.nix
140/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
141```
143We may want to run `chmod -R +w hlint.materialized` as the files copied from the
144store will be read only.
146## How can we check `sha256` and `materialized` are up to date?
148Let's pretend we had to go back to `hlint` version `2.2.10`.
149We can tell haskell.nix to check the materialiazation either by:
151* Removing the materialization files with `rm -rf hlint.materialized`
152* Temporarily adding `checkMaterialization = true;`
154If we choose to add the `checkMaterialization` flag you would have:
156```nix
157let inherit (import ./. {}) sources nixpkgsArgs;
158 pkgs = import sources.nixpkgs-default nixpkgsArgs;
159 hlint = pkgs.haskell-nix.hackage-package {
160 name = "hlint";
161 version = "2.2.10";
162 index-state = "2020-04-15T00:00:00Z";
163 plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
164 materialized = ./hlint.materialized;
165 checkMaterialization = true;
166 };
167in hlint.components.exes.hlint
168```
170This will fail and report the details of what is wrong and how to fix it:
172```
173$ nix-build hlint.nix
175...
177Calculated hash for hlint-plan-to-nix-pkgs was not 02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4. New hash is :
178 plan-sha256 = "0zsi3wv92qax33ic4n5dfsqd1r9qam1k75za3c5jqgdxl3hy8vph";
179Materialized nix used for hlint-plan-to-nix-pkgs incorrect. To fix run :
180 rm -rf /Users/hamish/iohk/haskell.nix/hlint.materialized
181 cp -r /nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs /Users/hamish/iohk/haskell.nix/hlint.materialized
182 chmod -R +w /Users/hamish/iohk/haskell.nix/hlint.materialized
183builder for '/nix/store/a5zmgfjfxahapw0q8hd2da5bg7knqvbx-hlint-plan-to-nix-pkgs.drv' failed with exit code 1
184error: build of '/nix/store/a5zmgfjfxahapw0q8hd2da5bg7knqvbx-hlint-plan-to-nix-pkgs.drv' failed
185(use '--show-trace' to show detailed location information)
186```
188Checking the materialization requires nix to do all the work that materialization
189avoids. So while it might be tempting to leave `checkMaterialization = true` all
190the time, we would be better off just removing `materialized` and `plan-sha256`.
192## How can we update the nix files with a script?
194There are versions of the functions (`project'`, `cabalProject'`,
195`stackProject'` and `hackage-project`) that also return the nix as
196`plan-nix` or `stack-nix`. By calling one of these functions without
197the hash and materialized nix we can find out what nix files should be.
198For instance:
200```nix
201let inherit (import ./. {}) sources nixpkgsArgs;
202 pkgs = import sources.nixpkgs-default nixpkgsArgs;
203 hlint = pkgs.haskell-nix.hackage-project {
204 name = "hlint";
205 version = "2.2.10";
206 index-state = "2020-04-15T00:00:00Z";
207 };
208in hlint
209```
211```
212$ nix-build hlint.nix -A plan-nix
213trace: Using index-state: 2020-04-15T00:00:00Z for hlint
214trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs/`
215/nix/store/ywdhbx9rzzkfc60c5vzk7cins2hnvkgx-hlint-plan-to-nix-pkgs
216```
218We can have the script copy `$(nix-build hlint.nix -A plan-nix --no-out-link)`
219and use `nix-hash` to calculate the new value for `plan-sha256`.
221## Can we skip making a copy and use `materialized = /nix/store/...`?
223Yes and it gives us the same speed improvement, however:
225* It does not help at all in `restricted-eval` mode (Hydra).
227* Users will still wind up building or downloading the dependencies
228 needed to build the nix fileds (if they do not have them).
230For those reasons it might be best to make a copy instead
231of using the `/nix/store/...` path directly.
233If you really want to use the `/nix/store/...` path directly
234you should gaurd against the path not existing as passing in
235a non-existing path is now an error:
237```nix
238let inherit (import ./. {}) sources nixpkgsArgs;
239 pkgs = import sources.nixpkgs-default nixpkgsArgs;
240 hlintPlan = /nix/store/kk047cqsjvbj4w8psv4l05abdcnyrqdc-hlint-plan-to-nix-pkgs;
241 hlint = pkgs.haskell-nix.hackage-package {
242 name = "hlint";
243 version = "2.2.11";
244 index-state = "2020-04-15T00:00:00Z";
245 plan-sha256 = "02hasr27a994sml1fzf8swb716lm6lgixxr53y0gxkhw437xkck4";
246 materialized = if __pathExists hlintPlan then hlintPlan else null;
247 };
248in hlint.components.exes.hlint
249```
251Running when no building is needed is still slow in restricted evaluation mode.
253```
254$ time nix-build --option restrict-eval true -I . --option allowed-uris "https://github.com/NixOS https://github.com/input-output-hk" hlint.nix --show-trace
255trace: Using index-state: 2020-04-15T00:00:00Z for hlint
256/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
258real 0m4.463s
259user 0m4.440s
260sys 0m0.461s
261$ time nix-build hlint.nix
262/nix/store/rnfz66v7k8i38c8rsmchzsyqjrmrbdpk-hlint-2.2.11-exe-hlint
264real 0m2.206s
265user 0m1.665s
266sys 0m0.332s
267```