git.iko.soy/typeable/cabal2nix
on v20160308 help sign in
typeable/cabal2nix/doc/02-guarantee-consistent-builds-and-obsolete-overridescope.md
182 lines 8.1 kB history raw
1Guarantee Consistent Builds and Obsolete `overrideScope`
2========================================================
4The Problem
5-----------
7A lot of effort goes into curated package sets like Stackage, but even so we
8can compile only ~50% of the packages available from Hackage. It appears to be
9the nature of the game: when `lens` 2.x comes out with a fundamentally new API,
10then some packages will adopt the new version and others won't. A consistent
11package set must chose on which side of the fence it wants to live. Either
12those packages that depend on `lens` 1.x compile or those that depend on 2.x
13compile --- but not both.
15Now, Nix is not limited to one particular version of `lens` --- we can have
16both versions available at the same time. But it's difficult to take advantage
17of that feature, because once you start mixing `lens` 1.x and 2.x in the same
18package set, you risk inconsistent builds, i.e. builds where some part of the
19dependency tree refers to `lens` 1.x and another part refers to `lens` 2.x.
20It's a bad idea to try and link those two trees together into one executable;
21we are fortunate that `Cabal` detects this error during the configure phase and
22aborts the build!
24We need a mechanism that can mix multiple package versions within a package
25set, but that also guarantees consistency for every single build. We hoped
26`overrideScope` would be that mechanism, but somehow it hasn't quite lived up
27to the promise, mostly because it is hard to understand.
29The Situation Today
30-------------------
32Consider an executable package `foobar` that depends on the libraries `foo` and
33`bar`, each of which depends on `lens`. The corresponding definitions in
34`hackage-packages.nix` --- stripped down to the relevant bits --- look as
35follows:
37 "lens" = ... lens version 1.x ...;
38 "lens_2_0" = ... lens version 2.x ...;
40 "foo" = callPackage
41 ({ mkDerivation, lens }:
42 mkDerivation {
43 pname = "foo";
44 libraryHaskellDepends = [lens];
45 }) {};
47 "bar" = callPackage
48 ({ mkDerivation, lens }:
49 mkDerivation {
50 pname = "bar";
51 libraryHaskellDepends = [lens];
52 }) {};
54 "foobar" = callPackage
55 ({ mkDerivation, lens }:
56 mkDerivation {
57 pname = "foobar"; [...]
58 libraryHaskellDepends = [foo bar];
59 }) {};
61Let's assume that `foo` won't compile in that setup because it requires `lens`
62version 2.x. We can remedy that by adding an override to
63`configuration-common.nix` that says:
65 foo = super.foo.override { lens = self.lens_2_0; };
67That change fixes the build of `foo`, but `foobar` remains broken, because now
68it pulls in both `lens` 1.x and 2.x simultaneously through its dependencies. If
69`bar` works only with `lens` 1.x, then there is nothing we can do: the version
70constraints conflict and we cannot compile `foobar`. If `bar` *does* support
71`lens` 2.x, however, then we can just switch it to the newer version with:
73 bar = super.bar.override { lens = self.lens_2_0; };
75Now we can compile `foobar`! Unfortunately, that change may break other builds.
76There is a reason why `lens` 1.x is our default choice. If any other package
77depends on `bar` as well as `lens` 1.x (directly or indirectly), then it will
78no longer compile after that change.
80We can avoid that side-effect by localizing the override to `foobar`:
82 foobar = super.foobar.override {
83 bar = self.bar.override { lens = self.lens_2_0; };
84 };
86That approach allows us to compile `foobar`, while still leaving the default
87version of `bar` at `lens` 1.x, like most of our packages require. Overriding
88build inputs this way works fine, and we have used this technique for many
89years to fix builds that require non-default versions to compile. The downside
90of these nested overrides is that the tend to become freaky complicated if a
91package needs overriding that is sufficiently deep in the dependency tree. The
92GHC 7.8.4 package set, for example, needed many such overrides because its
93default version of `mtl` was stuck at version 2.1.x all the while large parts
94of Hackage had moved on to `mtl` 2.2.x. Since `mtl` is a rather fundamental
95package, we had nested overrides 3-4 levels deep that were highly repetitious,
96too. It was a mess.
98Haskell NG improved on that situation by adding `overrideScope`. That function
99changes the package set ("scope") in which Nix evaluates a build expression.
100The override
102 foobar = super.foobar.overrideScope (self: super: { lens = self.lens_2_0; });
104creates a new temporary package set, replaces `lens` with `lens_2_0` in it, and
105then evaluates `foobar`. The `callPackage` function picks up the re-written
106`lens` attribute, which means that there's no need to override that choice
107explicitly in all dependencies of `foobar`. One could say that `overrideScope`
108implements "deep overriding", i.e. it applies an override to the given
109derivation as well as all sub-derivations that it refers to.
111Unfortunately, we lack a proper understanding of how expensive that technique
112is memory and performance-wise. In the past, we've occasionally crashed Nix
113with this kind of stuff --- keep in mind that the interpreter creates a whole
114new package set for every build that uses this mechanism ---, but when used
115sparingly, `overrideScope` seems to work okay.
117In some cases, `overrideScope` won't work at all, i.e. when confronted with
118builds that have explicitly passed arguments. For example, let's say that
119`lens` 3.x comes. So we try to compile `foobar` like this:
121 foobar = super.foobar.overrideScope (self: super: { lens = self.lens_3_0; });
123That build will fail, because we added an explicit override for `foo` earlier
124that committed the build to `lens_2_0`, and `overrideScope` will not affect
125that choice since that build input is not picked up with `callPackage`. So
126`foobar` will pull in both `lens` 2.x and 3.x despite the use of
127`overrideScope`.
130Possible Improvements
131---------------------
133We generate Haskell build expressions automatically with `cabal2nix`, and that
134tool knows the complete dependency tree for every package. So it would be
135possible to generate builds that expect as function arguments not just their
136immediate dependencies but the transitive closure of all dependencies. Build
137expressions would then call their direct dependencies, passing in appropriate
138versions of their respective dependencies, etc. For example:
140 "foobar" = callPackage
141 ({ mkDerivation, foo, bar, many, other, inputs, of, lens }:
142 let lens' = lens.override { inherit many other inputs of lens; };
143 foo' = foo.override { lens = lens'; };
144 bar' = bar.override { lens = lens'; };
145 in
146 mkDerivation {
147 pname = "foobar"; [...]
148 libraryHaskellDepends = [foo' bar'];
149 }) {};
151Now, `foobar` expects every single package that occurs anywhere inside of its
152dependency tree as an argument, and it constructs the dependency tree using
153those arguments. So the build must be consistent. It's impossible for `foobar`
154to refer to two incompatible versions of `lens`, because its inputs always use
155the same version.
157Consequently,
159 foobar.override { mtl = self.mtl_2.4.0; }
161gives us is a version of `foobar` that has its entire dependency tree built
162with `mtl` 2.4.x. We could even get rid of the override altogether if we adopt
163the suggestions from "Use Function Application To Escape Override Hell" and
164remove `callPackage` from `hackage-packages.nix`. We'd define all builds as
165straight functions
167 "foobar" =
168 { mkDerivation, foo, bar, many, other, inputs, of, lens }:
169 let lens' = lens { inherit many other inputs of lens; };
170 foo' = foo { lens = lens'; };
171 bar' = bar { lens = lens'; };
172 in
173 mkDerivation {
174 pname = "foobar"; [...]
175 libraryHaskellDepends = [foo' bar'];
176 };
178and invoke them from inside of a package set with:
180 callPackage foobar { mtl = self.mtl_2.4.0; }
182This would give us a guarantee for consistent builds without any overrides.