docs/user-guide/materialization.md · haskell.nix

git.iko.soy/haskell.nix

on 930ec0d help sign in

haskell.nix/docs/user-guide/materialization.md

291 lines 12 kB history raw

1# Materialization

3## What is materialization?

5Capturing and storing the nix files for a project so that they do

6not need to be built (or checked). This allows us to cache the input

7of an IFD (import from derviation).

9## Why use materialization?

11Using functions like `cabalProject`, `stackProject` and `hackage-package`

12results in a lot of dependencies (all the dependencies of nix-tools

13for instance).

15* They can be slow to calculate (even if no work needs to be done it

16 is not unusual for it to take 5 seconds per project).

18* They can be slow to build (or download) on machines that do not

19 yet have them in the nix store.

21* Hydra does not show progress because it does not provide feedback

22 until it has a list of jobs and the list of jobs cannot depends

23 on the nix being present (although this is often blamed on IFD

24 it would be the same if it wrote out JSON files and read them in)

26## When is it ok to materialize?

28* The nix is unlikely to change frequently (and when it does you

29 are happy to manually update it).

31* You are happy to script something to update the materialized

32 nix files automatically.

33* You are certain that the IFD you materialize is not `system`-dependent. If it was you'd

34 obtain different nix expressions depending on which `system` the IFD was evaluated.

36## How can we materialize the nix files?

38Lets say we want to build `hlint`. We might start with an `hlint`

39file that looks like this:

41```nix

42let inherit (import ./. {}) sources nixpkgsArgs;

43 pkgs = import sources.nixpkgs-default nixpkgsArgs;

44 hlint = pkgs.haskell-nix.hackage-package {

45 name = "hlint";

46 version = "2.2.4";

47 };

48in hlint.components.exes.hlint

49```

51Building this may result in a lot of output, but if you build

52it again it should give just:

54```

55$ nix-build hlint.nix

56trace: Using latest index state for hlint!

57trace: Using index-state: 2019-12-09T00:00:00Z for hlint

58/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint

59```

61To materialize the nix files we need to take care to pin down the

62inputs. For `cabalProject` and `hackage-package` this means

63we must specify the `index-state` of hackage we want to use:

65```nix

66let inherit (import ./. {}) sources nixpkgsArgs;

67 pkgs = import sources.nixpkgs-default nixpkgsArgs;

68 hlint = pkgs.haskell-nix.hackage-package {

69 name = "hlint";

70 version = "2.2.4";

71 index-state = "2019-12-03T00:00:00Z";

72 };

73in hlint.components.exes.hlint

74```

76Now if we build again we get a hint telling use how to

77calculate a suitable sha256 hash to turn the derivation

78containing the nix files into a fixed output derivation:

80```

81$ nix-build hlint.nix

82trace: Using index-state: 2019-12-03T00:00:00Z for hlint

83trace: Get `plan-sha256` with `nix-hash --base32 --type sha256 /nix/store/qk1fvza1alkvs51vzmpjp2xsg8xklyxk-hlint-plan-to-nix-pkgs/`

84/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint

86$ nix-hash --base32 --type sha256 /nix/store/qk1fvza1alkvs51vzmpjp2xsg8xklyxk-hlint-plan-to-nix-pkgs/

871a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss

88```

90We can add the hash as `plan-sha256` or (`stack-sha256` for

91`stackProject`)

93```nix

94let inherit (import ./. {}) sources nixpkgsArgs;

95 pkgs = import sources.nixpkgs-default nixpkgsArgs;

96 hlint = pkgs.haskell-nix.hackage-package {

97 name = "hlint";

98 version = "2.2.4";

99 index-state = "2019-12-03T00:00:00Z";

100 plan-sha256 = "1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss";

101 };

102in hlint.components.exes.hlint

103```

104

105Just adding the hash might help reuse of the cached nix, but nix will

106still calculate all the dependencies (which can add seconds to

107`nix-build` and `nix-shell` commands when no other work is needed)

108and users who do not yet have the dependencies in their store will have

109to wait while they are built or downloaded.

110

111Running nix build again gives us a hint on what we can do next:

112

113```

114$ nix-build hlint.nix

115trace: Using index-state: 2019-12-03T00:00:00Z for hlint

116trace: To materialize, point `materialized` to a copy of /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs

117/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint

118```

119

120To capture the nix we can do something like:

121

122```nix

123let inherit (import ./. {}) sources nixpkgsArgs;

124 pkgs = import sources.nixpkgs-default nixpkgsArgs;

125 hlint = pkgs.haskell-nix.hackage-package {

126 name = "hlint";

127 version = "2.2.4";

128 index-state = "2019-12-03T00:00:00Z";

129 plan-sha256 = "1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss";

130 materialized = ./hlint.materialized;

131 };

132in hlint.components.exes.hlint

133```

134

135Now we can copy the nix files needed and build with:

136

137```

138$ cp -r /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs hlint.materialized

139$ nix-build hlint.nix

140/nix/store/7wwqq5v42gm6iiz2d3ngbnkyw7i4py11-hlint-2.2.4-exe-hlint

141```

142

143We may want to run `chmod -R +w hlint.materialized` as the files copied from the

144store will be read only.

145

146## How can we check `sha256` and `materialized` are up to date?

147

148Let's pretend we had to go back to `hlint` version `2.2.3`.

149We can change `version` and temporarily add

150`checkMaterialization = true;`:

151

152```nix

153let inherit (import ./. {}) sources nixpkgsArgs;

154 pkgs = import sources.nixpkgs-default nixpkgsArgs;

155 hlint = pkgs.haskell-nix.hackage-package {

156 name = "hlint";

157 version = "2.2.4";

158 index-state = "2019-12-03T00:00:00Z";

159 plan-sha256 = "1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss";

160 materialized = ./hlint.materialized;

161 checkMaterialization = true;

162 };

163in hlint.components.exes.hlint

164```

165

166This will fail and report the details of what is wrong:

167

168```

169$ nix-build hlint.nix

170trace: Using index-state: 2019-12-03T00:00:00Z for hlint

171building '/nix/store/zmif4gk52ynh57pf4dikzgsk30haqi2b-hlint-plan-to-nix-pkgs.drv'...

172Changes to hlint-plan-to-nix-pkgs not reflected in plan-sha256

173diff -ru /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs/.plan.nix/hlint.nix /nix/store/h5j8k3y5lansyfss25gd7knbninzr6z4-hlint-plan-to-nix-pkgs/.plan.nix/hlint.nix

174--- /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs/.plan.nix/hlint.nix 1970-01-01 00:00:01.000000000 +0000

175+++ /nix/store/h5j8k3y5lansyfss25gd7knbninzr6z4-hlint-plan-to-nix-pkgs/.plan.nix/hlint.nix 1970-01-01 00:00:01.000000000 +0000

176@@ -42,7 +42,7 @@

177 flags = { threaded = true; gpl = true; ghc-lib = false; };

178 package = {

179 specVersion = "1.18";

180- identifier = { name = "hlint"; version = "2.2.4"; };

181+ identifier = { name = "hlint"; version = "2.2.3"; };

182 license = "BSD-3-Clause";

183 copyright = "Neil Mitchell 2006-2019";

184 maintainer = "Neil Mitchell <ndmitchell@gmail.com>";

185@@ -95,7 +95,6 @@

186 (hsPkgs."extra" or (buildDepError "extra"))

187 (hsPkgs."refact" or (buildDepError "refact"))

188 (hsPkgs."aeson" or (buildDepError "aeson"))

189- (hsPkgs."filepattern" or (buildDepError "filepattern"))

190 (hsPkgs."syb" or (buildDepError "syb"))

191 (hsPkgs."mtl" or (buildDepError "mtl"))

192 ] ++ (if !flags.ghc-lib && (compiler.isGhc && (compiler.version).ge "8.8.0") && (compiler.isGhc && (compiler.version).lt "8.9.0")

193diff -ru /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs/default.nix /nix/store/h5j8k3y5lansyfss25gd7knbninzr6z4-hlint-plan-to-nix-pkgs/default.nix

194--- /nix/store/0xalcphb7ifvy5fc9dpwj40fij6nn5av-hlint-plan-to-nix-pkgs/default.nix 1970-01-01 00:00:01.000000000 +0000

195+++ /nix/store/h5j8k3y5lansyfss25gd7knbninzr6z4-hlint-plan-to-nix-pkgs/default.nix 1970-01-01 00:00:01.000000000 +0000

196@@ -76,7 +76,7 @@

197 "tagged".revision = (((hackage."tagged")."0.8.6").revisions).default;

198 "tagged".flags.transformers = true;

199 "tagged".flags.deepseq = true;

200- "haskell-src-exts".revision = (((hackage."haskell-src-exts")."1.22.0").revisions).default;

201+ "haskell-src-exts".revision = (((hackage."haskell-src-exts")."1.21.1").revisions).default;

202 "unliftio-core".revision = (((hackage."unliftio-core")."0.1.2.0").revisions).default;

203 "ghc-lib-parser".revision = (((hackage."ghc-lib-parser")."8.8.1").revisions).default;

204 "containers".revision = (((hackage."containers")."0.6.0.1").revisions).default;

205@@ -116,7 +116,6 @@

206 "hpc".revision = (((hackage."hpc")."0.6.0.3").revisions).default;

207 "filepath".revision = (((hackage."filepath")."1.4.2.1").revisions).default;

208 "process".revision = (((hackage."process")."1.6.5.0").revisions).default;

209- "filepattern".revision = (((hackage."filepattern")."0.1.1").revisions).default;

210 "libyaml".revision = (((hackage."libyaml")."0.1.1.1").revisions).default;

211 "libyaml".flags.system-libyaml = false;

212 "libyaml".flags.no-unicode = false;

213Calculated hash is 1qjmhlb4rw6mggs7y57f6zr5zjmkhkx7sn9q8pb18308n5nxgxcs expected hash was 1a4rhv3h2daz6dzwzfl3w7l1v556n7aqfiagw6m0rvqq230iabss for hlint-plan-to-nix-pkgs

214builder for '/nix/store/zmif4gk52ynh57pf4dikzgsk30haqi2b-hlint-plan-to-nix-pkgs.drv' failed with exit code 1

215error: build of '/nix/store/zmif4gk52ynh57pf4dikzgsk30haqi2b-hlint-plan-to-nix-pkgs.drv' failed

216```

217

218Checking the materialization requires nix to do all the work that materialization

219avoids. So while it might be tempting to leave `checkMaterialization = true` all

220the time, we would be better off just removing `materialized` and `plan-sha256`.

221

222## How can we update the nix files with a script?

223

224There are versions of the functions (`cabalProject'`, `stackProject'`

225and `hackage-project`) that also return the nix as `plan-nix` or