@@ -0,0 +1,182 @@
1+Guarantee Consistent Builds and Obsolete `overrideScope`
2+========================================================
3+
4+The Problem
5+-----------
6+
7+A lot of effort goes into curated package sets like Stackage, but even so we
8+can compile only ~50% of the packages available from Hackage. It appears to be
9+the nature of the game: when `lens` 2.x comes out with a fundamentally new API,
10+then some packages will adopt the new version and others won't. A consistent
11+package set must chose on which side of the fence it wants to live. Either
12+those packages that depend on `lens` 1.x compile or those that depend on 2.x
13+compile --- but not both.
14+
15+Now, Nix is not limited to one particular version of `lens` --- we can have
16+both versions available at the same time. But it's difficult to take advantage
17+of that feature, because once you start mixing `lens` 1.x and 2.x in the same
18+package set, you risk inconsistent builds, i.e. builds where some part of the
19+dependency tree refers to `lens` 1.x and another part refers to `lens` 2.x.
20+It's a bad idea to try and link those two trees together into one executable;
21+we are fortunate that `Cabal` detects this error during the configure phase and
22+aborts the build!
23+
24+We need a mechanism that can mix multiple package versions within a package
25+set, but that also guarantees consistency for every single build. We hoped
26+`overrideScope` would be that mechanism, but somehow it hasn't quite lived up
27+to the promise, mostly because it is hard to understand.
28+
29+The Situation Today
30+-------------------
31+
32+Consider an executable package `foobar` that depends on the libraries `foo` and
33+`bar`, each of which depends on `lens`. The corresponding definitions in
34+`hackage-packages.nix` --- stripped down to the relevant bits --- look as
35+follows:
36+
37+ "lens" = ... lens version 1.x ...;
38+ "lens_2_0" = ... lens version 2.x ...;
39+
40+ "foo" = callPackage
41+ ({ mkDerivation, lens }:
42+ mkDerivation {
43+ pname = "foo";
44+ libraryHaskellDepends = [lens];
45+ }) {};
46+
47+ "bar" = callPackage
48+ ({ mkDerivation, lens }:
49+ mkDerivation {
50+ pname = "bar";
51+ libraryHaskellDepends = [lens];
52+ }) {};
53+
54+ "foobar" = callPackage
55+ ({ mkDerivation, lens }:
56+ mkDerivation {
57+ pname = "foobar"; [...]
58+ libraryHaskellDepends = [foo bar];
59+ }) {};
60+
61+Let's assume that `foo` won't compile in that setup because it requires `lens`
62+version 2.x. We can remedy that by adding an override to
63+`configuration-common.nix` that says:
64+
65+ foo = super.foo.override { lens = self.lens_2_0; };
66+
67+That change fixes the build of `foo`, but `foobar` remains broken, because now
68+it pulls in both `lens` 1.x and 2.x simultaneously through its dependencies. If
69+`bar` works only with `lens` 1.x, then there is nothing we can do: the version
70+constraints conflict and we cannot compile `foobar`. If `bar` *does* support
71+`lens` 2.x, however, then we can just switch it to the newer version with:
72+
73+ bar = super.bar.override { lens = self.lens_2_0; };
74+
75+Now we can compile `foobar`! Unfortunately, that change may break other builds.
76+There is a reason why `lens` 1.x is our default choice. If any other package
77+depends on `bar` as well as `lens` 1.x (directly or indirectly), then it will
78+no longer compile after that change.
79+
80+We can avoid that side-effect by localizing the override to `foobar`:
81+
82+ foobar = super.foobar.override {
83+ bar = self.bar.override { lens = self.lens_2_0; };
84+ };
85+
86+That approach allows us to compile `foobar`, while still leaving the default
87+version of `bar` at `lens` 1.x, like most of our packages require. Overriding
88+build inputs this way works fine, and we have used this technique for many
89+years to fix builds that require non-default versions to compile. The downside
90+of these nested overrides is that the tend to become freaky complicated if a
91+package needs overriding that is sufficiently deep in the dependency tree. The
92+GHC 7.8.4 package set, for example, needed many such overrides because its
93+default version of `mtl` was stuck at version 2.1.x all the while large parts
94+of Hackage had moved on to `mtl` 2.2.x. Since `mtl` is a rather fundamental
95+package, we had nested overrides 3-4 levels deep that were highly repetitious,
96+too. It was a mess.
97+
98+Haskell NG improved on that situation by adding `overrideScope`. That function
99+changes the package set ("scope") in which Nix evaluates a build expression.
100+The override
101+
102+ foobar = super.foobar.overrideScope (self: super: { lens = self.lens_2_0; });
103+
104+creates a new temporary package set, replaces `lens` with `lens_2_0` in it, and
105+then evaluates `foobar`. The `callPackage` function picks up the re-written
106+`lens` attribute, which means that there's no need to override that choice
107+explicitly in all dependencies of `foobar`. One could say that `overrideScope`
108+implements "deep overriding", i.e. it applies an override to the given
109+derivation as well as all sub-derivations that it refers to.
110+
111+Unfortunately, we lack a proper understanding of how expensive that technique
112+is memory and performance-wise. In the past, we've occasionally crashed Nix
113+with this kind of stuff --- keep in mind that the interpreter creates a whole
114+new package set for every build that uses this mechanism ---, but when used
115+sparingly, `overrideScope` seems to work okay.
116+
117+In some cases, `overrideScope` won't work at all, i.e. when confronted with
118+builds that have explicitly passed arguments. For example, let's say that
119+`lens` 3.x comes. So we try to compile `foobar` like this:
120+
121+ foobar = super.foobar.overrideScope (self: super: { lens = self.lens_3_0; });
122+
123+That build will fail, because we added an explicit override for `foo` earlier
124+that committed the build to `lens_2_0`, and `overrideScope` will not affect
125+that choice since that build input is not picked up with `callPackage`. So
126+`foobar` will pull in both `lens` 2.x and 3.x despite the use of
127+`overrideScope`.
128+
129+
130+Possible Improvements
131+---------------------
132+
133+We generate Haskell build expressions automatically with `cabal2nix`, and that
134+tool knows the complete dependency tree for every package. So it would be
135+possible to generate builds that expect as function arguments not just their
136+immediate dependencies but the transitive closure of all dependencies. Build
137+expressions would then call their direct dependencies, passing in appropriate
138+versions of their respective dependencies, etc. For example:
139+
140+ "foobar" = callPackage
141+ ({ mkDerivation, foo, bar, many, other, inputs, of, lens }:
142+ let lens' = lens.override { inherit many other inputs of lens; };
143+ foo' = foo.override { lens = lens'; };
144+ bar' = bar.override { lens = lens'; };
145+ in
146+ mkDerivation {
147+ pname = "foobar"; [...]
148+ libraryHaskellDepends = [foo' bar'];
149+ }) {};
150+
151+Now, `foobar` expects every single package that occurs anywhere inside of its
152+dependency tree as an argument, and it constructs the dependency tree using
153+those arguments. So the build must be consistent. It's impossible for `foobar`
154+to refer to two incompatible versions of `lens`, because its inputs always use
155+the same version.
156+
157+Consequently,
158+
159+ foobar.override { mtl = self.mtl_2.4.0; }
160+
161+gives us is a version of `foobar` that has its entire dependency tree built
162+with `mtl` 2.4.x. We could even get rid of the override altogether if we adopt
163+the suggestions from "Use Function Application To Escape Override Hell" and
164+remove `callPackage` from `hackage-packages.nix`. We'd define all builds as
165+straight functions
166+
167+ "foobar" =
168+ { mkDerivation, foo, bar, many, other, inputs, of, lens }:
169+ let lens' = lens { inherit many other inputs of lens; };
170+ foo' = foo { lens = lens'; };
171+ bar' = bar { lens = lens'; };
172+ in
173+ mkDerivation {
174+ pname = "foobar"; [...]
175+ libraryHaskellDepends = [foo' bar'];
176+ };
177+
178+and invoke them from inside of a package set with:
179+
180+ callPackage foobar { mtl = self.mtl_2.4.0; }
181+
182+This would give us a guarantee for consistent builds without any overrides.