docs/user-guide/materialization.md · haskell.nix

git.iko.soy/haskell.nix

on 33dc7bc help sign in

haskell.nix/docs/user-guide/materialization.md

270 lines 11 kB history raw

1# Materialization

3## What is materialization?

5Capturing and storing the nix files for a project so that they do

6not need to be built (or checked). This allows us to cache the input

7of an IFD (import from derviation).

9## Why use materialization?

11Using functions like `cabalProject`, `stackProject` and `hackage-package`

12results in a lot of dependencies (all the dependencies of nix-tools

13for instance).

15* They can be slow to calculate (even if no work needs to be done it

16 is not unusual for it to take 5 seconds per project).

18* They can be slow to build (or download) on machines that do not

19 yet have them in the nix store.

21* Hydra does not show progress because it does not provide feedback

22 until it has a list of jobs and the list of jobs cannot depends

23 on the nix being present (although this is often blamed on IFD

24 it would be the same if it wrote out JSON files and read them in)

26## When is it ok to materialize?

28* The nix is unlikely to change frequently (and when it does you

29 are happy to manually update it).

31* You are happy to script something to update the materialized

32 nix files automatically.

33* You are certain that the IFD you materialize is not `system`-dependent. If it was you'd

34 obtain different nix expressions depending on which `system` the IFD was evaluated.

36## How can we materialize the nix files?

38Lets say we want to build `hlint`. We might start with an `hlint`

39file that looks like this:

41```nix

42((import ./nixpkgs (import ./.)).haskell-nix.hackage-package {

43 name = "hlint";

44 version = "2.2.4";

45}).components.exes.hlint

46```

48Building this may result in a lot of output, but if youb build

49it again it should give just:

51```

52$ nix-build hlint.nix

53trace: Using latest index state for hlint!

54trace: Using index-state: 2019-12-09T00:00:00Z for hlint

55/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint

56```

58To materialize the nix files we need to take care to pin down the

59inputs. For `cabalProject` and `hackage-package` this means

60we must specify the `index-state` of hackage we want to use:

62```nix

63((import ./nixpkgs (import ./.)).haskell-nix.hackage-package {

64 name = "hlint";

65 version = "2.2.4";

66 index-state = "2019-12-03T00:00:00Z";

67}).components.exes.hlint

68```

70Now if we build again we get a hint telling use how to

71calculate a suitable sha256 hash to turn the derivation

72containing the nix files into a fixed output derivation:

74```

75$ nix-build hlint.nix

76trace: Using index-state: 2019-12-03T00:00:00Z for hlint

77trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/qk1fvza1alkvs51vzmpjp2xsg8xklyxk-hlint-plan-to-nix-pkgs/`

78/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint

80$ nix-hash --base32 --type sha256 /nix/store/qk1fvza1alkvs51vzmpjp2xsg8xklyxk-hlint-plan-to-nix-pkgs/

811a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss

82```

84We can add the hash as `plan-sha256` or (`stack-sha256` for

85`stackProject`)

87```nix

88((import ./nixpkgs (import ./.)).haskell-nix.hackage-package {

89 name = "hlint";

90 version = "2.2.4";

91 index-state = "2019-12-03T00:00:00Z";

92 plan-sha256 = "1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss";

93}).components.exes.hlint

94```

96Just adding the hash might help reuse of the cached nix, but nix will

97still calculate all the dependencies (which can add seconds to

98`nix-build` and `nix-shell` commands when no other work is needed)

99and users who do not yet have the dependencies in their store will have

100to wait while they are built or downloaded.

101

102Running nix build again gives us a hint on what we can do next:

103

104```

105$ nix-build hlint.nix

106trace: Using index-state: 2019-12-03T00:00:00Z for hlint

107trace: To materialize, point `materialized` to a copy of /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs

108/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint

109```

110

111To capture the nix we can do something like:

112

113```nix

114((import ./nixpkgs (import ./.)).haskell-nix.hackage-package {

115 name = "hlint";

116 version = "2.2.4";

117 index-state = "2019-12-03T00:00:00Z";

118 plan-sha256 = "1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss";

119 materialized = ./hlint.materialized;

120}).components.exes.hlint

121```

122

123Now we can copy the nix files needed and build with:

124

125```

126$ cp -r /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs hlint.materialized

127$ nix-build hlint.nix

128/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint

129```

130

131We may want to run `cmod -R +w hlint.materialized` as the files copied from the

132store will be read only.

133

134## How can we check `sha256` and `materialized` are up to date?

135

136Let's pretend we had to go back to `hlint` version `2.2.3`.

137We can change `version` and temporarily add

138`checkMaterialization = true;`:

139

140```nix

141((import ./nixpkgs (import ./.)).haskell-nix.hackage-package {

142 name = "hlint";

143 version = "2.2.3";

144 index-state = "2019-12-03T00:00:00Z";

145 plan-sha256 = "1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss";

146 materialized = ./hlint.materialized;

147 checkMaterialization = true;

148}).components.exes.hlint

149```

150

151This will fail and report the details of what is wrong:

152

153```

154$ nix-build hlint.nix

155trace: Using index-state: 2019-12-03T00:00:00Z for hlint

156building '/nix/store/zmif4gk52ynh57pf4dikzgsk30haqi2b-hlint-plan-to-nix-pkgs.drv'...

157Changes to hlint-plan-to-nix-pkgs not reflected in plan-sha256

158diff -ru /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs/.plan.nix/hlint.nix /nix/store/h5j8k3y5lansyfss25gd7knbninzr6z4-hlint-plan-to-nix-pkgs/.plan.nix/hlint.nix

159--- /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs/.plan.nix/hlint.nix 1970-01-01 00:00:01.000000000 +0000

160+++ /nix/store/h5j8k3y5lansyfss25gd7knbninzr6z4-hlint-plan-to-nix-pkgs/.plan.nix/hlint.nix 1970-01-01 00:00:01.000000000 +0000

161@@ -42,7 +42,7 @@

162 flags = { threaded = true; gpl = true; ghc-lib = false; };

163 package = {

164 specVersion = "1.18";

165- identifier = { name = "hlint"; version = "2.2.4"; };

166+ identifier = { name = "hlint"; version = "2.2.3"; };

167 license = "BSD-3-Clause";

168 copyright = "Neil Mitchell 2006-2019";

169 maintainer = "Neil Mitchell <ndmitchell@gmail.com>";

170@@ -95,7 +95,6 @@

171 (hsPkgs."extra" or (buildDepError "extra"))

172 (hsPkgs."refact" or (buildDepError "refact"))

173 (hsPkgs."aeson" or (buildDepError "aeson"))

174- (hsPkgs."filepattern" or (buildDepError "filepattern"))

175 (hsPkgs."syb" or (buildDepError "syb"))

176 (hsPkgs."mtl" or (buildDepError "mtl"))

177 ] ++ (if !flags.ghc-lib && (compiler.isGhc && (compiler.version).ge "8.8.0") && (compiler.isGhc && (compiler.version).lt "8.9.0")

178diff -ru /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs/default.nix /nix/store/h5j8k3y5lansyfss25gd7knbninzr6z4-hlint-plan-to-nix-pkgs/default.nix

179--- /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs/default.nix 1970-01-01 00:00:01.000000000 +0000

180+++ /nix/store/h5j8k3y5lansyfss25gd7knbninzr6z4-hlint-plan-to-nix-pkgs/default.nix 1970-01-01 00:00:01.000000000 +0000

181@@ -76,7 +76,7 @@

182 "tagged".revision = (((hackage."tagged")."0.8.6").revisions).default;

183 "tagged".flags.transformers = true;

184 "tagged".flags.deepseq = true;

185- "haskell-src-exts".revision = (((hackage."haskell-src-exts")."1.22.0").revisions).default;

186+ "haskell-src-exts".revision = (((hackage."haskell-src-exts")."1.21.1").revisions).default;

187 "unliftio-core".revision = (((hackage."unliftio-core")."0.1.2.0").revisions).default;

188 "ghc-lib-parser".revision = (((hackage."ghc-lib-parser")."8.8.1").revisions).default;

189 "containers".revision = (((hackage."containers")."0.6.0.1").revisions).default;

190@@ -116,7 +116,6 @@

191 "hpc".revision = (((hackage."hpc")."0.6.0.3").revisions).default;

192 "filepath".revision = (((hackage."filepath")."1.4.2.1").revisions).default;

193 "process".revision = (((hackage."process")."1.6.5.0").revisions).default;

194- "filepattern".revision = (((hackage."filepattern")."0.1.1").revisions).default;

195 "libyaml".revision = (((hackage."libyaml")."0.1.1.1").revisions).default;

196 "libyaml".flags.system-libyaml = false;

197 "libyaml".flags.no-unicode = false;

198Calculated hash is 1qjmhlb4rw6mggs7y57f6zr5zjmkhkx7sn9q8pb18308n5nxgxcs expected hash was 1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss for hlint-plan-to-nix-pkgs

199builder for '/nix/store/zmif4gk52ynh57pf4dikzgsk30haqi2b-hlint-plan-to-nix-pkgs.drv' failed with exit code 1

200error: build of '/nix/store/zmif4gk52ynh57pf4dikzgsk30haqi2b-hlint-plan-to-nix-pkgs.drv' failed

201```

202

203Checking the materialization requires nix to do all the work that materialization

204avoids. So while it might be tempting to leave `checkMaterialization = true` all

205the time, we would be better off just removing `materialized` and `plan-sha256`.

206

207## How can we update the nix files with a script?

208

209There are versions of the functions (`cabalProject'`, `stackProject'`

210and `hackage-project`) that also return the nix as `plan-nix` or