@@ -0,0 +1,270 @@
1+# Materialization
2+
3+## What is materialization?
4+
5+Capturing and storing the nix files for a project so that they do
6+not need to be built (or checked). This allows us to cache the input
7+of an IFD (import from derviation).
8+
9+## Why use materialization?
10+
11+Using functions like `cabalProject`, `stackProject` and `hackage-package`
12+results in a lot of dependencies (all the dependencies of nix-tools
13+for instance).
14+
15+* They can be slow to calculate (even if no work needs to be done it
16+ is not unusual for it to take 5 seconds per project).
17+
18+* They can be slow to build (or download) on machines that do not
19+ yet have them in the nix store.
20+
21+* Hydra does not show progress because it does not provide feedback
22+ until it has a list of jobs and the list of jobs cannot depends
23+ on the nix being present (although this is often blamed on IFD
24+ it would be the same if it wrote out JSON files and read them in)
25+
26+## When is it ok to materialize?
27+
28+* The nix is unlikely to change frequently (and when it does you
29+ are happy to manually update it).
30+
31+* You are happy to script something to update the materialized
32+ nix files automatically.
33+* You are certain that the IFD you materialize is not `system`-dependent. If it was you'd
34+ obtain different nix expressions depending on which `system` the IFD was evaluated.
35+
36+## How can we materialize the nix files?
37+
38+Lets say we want to build `hlint`. We might start with an `hlint`
39+file that looks like this:
40+
41+```nix
42+((import ./nixpkgs (import ./.)).haskell-nix.hackage-package {
43+ name = "hlint";
44+ version = "2.2.4";
45+}).components.exes.hlint
46+```
47+
48+Building this may result in a lot of output, but if youb build
49+it again it should give just:
50+
51+```
52+$ nix-build hlint.nix
53+trace: Using latest index state for hlint!
54+trace: Using index-state: 2019-12-09T00:00:00Z for hlint
55+/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint
56+```
57+
58+To materialize the nix files we need to take care to pin down the
59+inputs. For `cabalProject` and `hackage-package` this means
60+we must specify the `index-state` of hackage we want to use:
61+
62+```nix
63+((import ./nixpkgs (import ./.)).haskell-nix.hackage-package {
64+ name = "hlint";
65+ version = "2.2.4";
66+ index-state = "2019-12-03T00:00:00Z";
67+}).components.exes.hlint
68+```
69+
70+Now if we build again we get a hint telling use how to
71+calculate a suitable sha256 hash to turn the derivation
72+containing the nix files into a fixed output derivation:
73+
74+```
75+$ nix-build hlint.nix
76+trace: Using index-state: 2019-12-03T00:00:00Z for hlint
77+trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/qk1fvza1alkvs51vzmpjp2xsg8xklyxk-hlint-plan-to-nix-pkgs/`
78+/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint
79+
80+$ nix-hash --base32 --type sha256 /nix/store/qk1fvza1alkvs51vzmpjp2xsg8xklyxk-hlint-plan-to-nix-pkgs/
81+1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss
82+```
83+
84+We can add the hash as `plan-sha256` or (`stack-sha256` for
85+`stackProject`)
86+
87+```nix
88+((import ./nixpkgs (import ./.)).haskell-nix.hackage-package {
89+ name = "hlint";
90+ version = "2.2.4";
91+ index-state = "2019-12-03T00:00:00Z";
92+ plan-sha256 = "1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss";
93+}).components.exes.hlint
94+```
95+
96+Just adding the hash might help reuse of the cached nix, but nix will
97+still calculate all the dependencies (which can add seconds to
98+`nix-build` and `nix-shell` commands when no other work is needed)
99+and users who do not yet have the dependencies in their store will have
100+to wait while they are built or downloaded.
101+
102+Running nix build again gives us a hint on what we can do next:
103+
104+```
105+$ nix-build hlint.nix
106+trace: Using index-state: 2019-12-03T00:00:00Z for hlint
107+trace: To materialize, point `materialized` to a copy of /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs
108+/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint
109+```
110+
111+To capture the nix we can do something like:
112+
113+```nix
114+((import ./nixpkgs (import ./.)).haskell-nix.hackage-package {
115+ name = "hlint";
116+ version = "2.2.4";
117+ index-state = "2019-12-03T00:00:00Z";
118+ plan-sha256 = "1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss";
119+ materialized = ./hlint.materialized;
120+}).components.exes.hlint
121+```
122+
123+Now we can copy the nix files needed and build with:
124+
125+```
126+$ cp -r /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs hlint.materialized
127+$ nix-build hlint.nix
128+/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint
129+```
130+
131+We may want to run `cmod -R +w hlint.materialized` as the files copied from the
132+store will be read only.
133+
134+## How can we check `sha256` and `materialized` are up to date?
135+
136+Let's pretend we had to go back to `hlint` version `2.2.3`.
137+We can change `version` and temporarily add
138+`checkMaterialization = true;`:
139+
140+```nix
141+((import ./nixpkgs (import ./.)).haskell-nix.hackage-package {
142+ name = "hlint";
143+ version = "2.2.3";
144+ index-state = "2019-12-03T00:00:00Z";
145+ plan-sha256 = "1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss";
146+ materialized = ./hlint.materialized;
147+ checkMaterialization = true;
148+}).components.exes.hlint
149+```
150+
151+This will fail and report the details of what is wrong:
152+
153+```
154+$ nix-build hlint.nix
155+trace: Using index-state: 2019-12-03T00:00:00Z for hlint
156+building '/nix/store/zmif4gk52ynh57pf4dikzgsk30haqi2b-hlint-plan-to-nix-pkgs.drv'...
157+Changes to hlint-plan-to-nix-pkgs not reflected in plan-sha256
158+diff -ru /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs/.plan.nix/hlint.nix /nix/store/h5j8k3y5lansyfss25gd7knbninzr6z4-hlint-plan-to-nix-pkgs/.plan.nix/hlint.nix
159+--- /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs/.plan.nix/hlint.nix 1970-01-01 00:00:01.000000000 +0000
160++++ /nix/store/h5j8k3y5lansyfss25gd7knbninzr6z4-hlint-plan-to-nix-pkgs/.plan.nix/hlint.nix 1970-01-01 00:00:01.000000000 +0000
161+@@ -42,7 +42,7 @@
162+ flags = { threaded = true; gpl = true; ghc-lib = false; };
163+ package = {
164+ specVersion = "1.18";
165+- identifier = { name = "hlint"; version = "2.2.4"; };
166++ identifier = { name = "hlint"; version = "2.2.3"; };
167+ license = "BSD-3-Clause";
168+ copyright = "Neil Mitchell 2006-2019";
169+ maintainer = "Neil Mitchell <ndmitchell@gmail.com>";
170+@@ -95,7 +95,6 @@
171+ (hsPkgs."extra" or (buildDepError "extra"))
172+ (hsPkgs."refact" or (buildDepError "refact"))
173+ (hsPkgs."aeson" or (buildDepError "aeson"))
174+- (hsPkgs."filepattern" or (buildDepError "filepattern"))
175+ (hsPkgs."syb" or (buildDepError "syb"))
176+ (hsPkgs."mtl" or (buildDepError "mtl"))
177+ ] ++ (if !flags.ghc-lib && (compiler.isGhc && (compiler.version).ge "8.8.0") && (compiler.isGhc && (compiler.version).lt "8.9.0")
178+diff -ru /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs/default.nix /nix/store/h5j8k3y5lansyfss25gd7knbninzr6z4-hlint-plan-to-nix-pkgs/default.nix
179+--- /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs/default.nix 1970-01-01 00:00:01.000000000 +0000
180++++ /nix/store/h5j8k3y5lansyfss25gd7knbninzr6z4-hlint-plan-to-nix-pkgs/default.nix 1970-01-01 00:00:01.000000000 +0000
181+@@ -76,7 +76,7 @@
182+ "tagged".revision = (((hackage."tagged")."0.8.6").revisions).default;
183+ "tagged".flags.transformers = true;
184+ "tagged".flags.deepseq = true;
185+- "haskell-src-exts".revision = (((hackage."haskell-src-exts")."1.22.0").revisions).default;
186++ "haskell-src-exts".revision = (((hackage."haskell-src-exts")."1.21.1").revisions).default;
187+ "unliftio-core".revision = (((hackage."unliftio-core")."0.1.2.0").revisions).default;
188+ "ghc-lib-parser".revision = (((hackage."ghc-lib-parser")."8.8.1").revisions).default;
189+ "containers".revision = (((hackage."containers")."0.6.0.1").revisions).default;
190+@@ -116,7 +116,6 @@
191+ "hpc".revision = (((hackage."hpc")."0.6.0.3").revisions).default;
192+ "filepath".revision = (((hackage."filepath")."1.4.2.1").revisions).default;
193+ "process".revision = (((hackage."process")."1.6.5.0").revisions).default;
194+- "filepattern".revision = (((hackage."filepattern")."0.1.1").revisions).default;
195+ "libyaml".revision = (((hackage."libyaml")."0.1.1.1").revisions).default;
196+ "libyaml".flags.system-libyaml = false;
197+ "libyaml".flags.no-unicode = false;
198+Calculated hash is 1qjmhlb4rw6mggs7y57f6zr5zjmkhkx7sn9q8pb18308n5nxgxcs expected hash was 1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss for hlint-plan-to-nix-pkgs
199+builder for '/nix/store/zmif4gk52ynh57pf4dikzgsk30haqi2b-hlint-plan-to-nix-pkgs.drv' failed with exit code 1
200+error: build of '/nix/store/zmif4gk52ynh57pf4dikzgsk30haqi2b-hlint-plan-to-nix-pkgs.drv' failed
201+```
202+
203+Checking the materialization requires nix to do all the work that materialization
204+avoids. So while it might be tempting to leave `checkMaterialization = true` all
205+the time, we would be better off just removing `materialized` and `plan-sha256`.
206+
207+## How can we update the nix files with a script?
208+
209+There are versions of the functions (`cabalProject'`, `stackProject'`
210+and `hackage-project`) that also return the nix as `plan-nix` or
211+`stack-nix`. By calling one of these functions without the
212+hash and materialized nix we can find out what nix files should be.
213+For instance:
214+
215+```nix
216+(import ./nixpkgs (import ./.)).haskell-nix.hackage-project {
217+ name = "hlint";
218+ version = "2.2.4";
219+ index-state = "2019-12-03T00:00:00Z";
220+}
221+```
222+
223+```
224+$ nix-build hlint.nix -A plan-nix
225+trace: Using index-state: 2019-12-03T00:00:00Z for hlint
226+trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/qk1fvza1alkvs51vzmpjp2xsg8xklyxk-hlint-plan-to-nix-pkgs/`
227+/nix/store/qk1fvza1alkvs51vzmpjp2xsg8xklyxk-hlint-plan-to-nix-pkgs
228+```
229+
230+We can have the script copy `$(nix-build hlint.nix -A plan-nix --no-out-link)`
231+and use `nix-hash` to calculate the new value for `plan-sha256`.
232+
233+## Can we skip making a copy and use `materialized = /nix/store/...`?
234+
235+Yes and it gives us the same speed improvement, however:
236+
237+* It does not help at all in `restricted-eval` mode (Hydra).
238+
239+* Users will still wind up building or downloading the dependencies
240+ needed to build the nix fileds (if they do not have them).
241+
242+For instance:
243+
244+```nix
245+((import ./nixpkgs (import ./.)).haskell-nix.hackage-package {
246+ name = "hlint";
247+ version = "2.2.4";
248+ index-state = "2019-12-03T00:00:00Z";
249+ plan-sha256 = "1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss";
250+ materialized = /nix/store/qk1fvza1alkvs51vzmpjp2xsg8xklyxk-hlint-plan-to-nix-pkgs;
251+}).components.exes.hlint
252+```
253+
254+Running when no building is needed is still slow in restricted evaluation mode.
255+
256+```
257+$ time nix-build --option restrict-eval true -I . --option allowed-uris "https://github.com/NixOS https://github.com/input-output-hk" hlint.nix
258+trace: Using index-state: 2019-12-03T00:00:00Z for hlint
259+/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint
260+
261+real 0m10.066s
262+user 0m8.563s
263+sys 0m0.630s
264+$ time nix-build hlint.nix
265+/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint
266+
267+real 0m4.628s
268+user 0m3.889s
269+sys 0m0.389s
270+```