1Use Function Application To Escape Override Hell
2================================================
7The architecture of the `haskell.packages.*` hierarchy worked fine when Nixpkgs
8shipped a handful of manually maintained package sets. The addition of LTS
9Haskell, however, increased the number of active package sets by a factor of
1010, and that number will grow further since there is a new LTS release coming
11out every week. Our preferred mechanism to cope with that diversity is the
12override: `haskellPackages` provides the foundation that other package sets
13override according to their needs. As of today, this approach requires
14approximately 25,000 overrides. Furthermore, we use 3 different types of
15overrides (`override`, `overrideCabal`, and `overrideScope`) that have subtly
16different purposes and capabilities. The complexity of this technique caused
17non-trivial issues in the past that manifested in [severe memory leaks][5],
18[insufficient flexibility][6], and [counter-intuitive behavior][7]. The current
19approach is not going to scale forever --- particularly in terms of
20maintainability. We need a better architecture to define Haskell package sets
21that supports variant builds in a fashion that's simpler and thus easier to
22understand and maintain.
27Nixpkgs distributes over 50 Haskell package sets that differ in which packages
28they contain, which versions of those packages they contain, and how those
29builds are configured. For example, package sets intended for use with GHC
30versions prior to 7.4.x neither build nor run regression test suites, because
31Cabal introduced that feature only in later versions of the compiler. Package
32sets compiling with GHC versions prior to 7.8.x don't support shared library
33builds, because that feature never worked reliably until more recent versions
34of the compiler. And builds running on Darwin don't generate the Haddock
35documentation when compiling with GHC versions 7.6.x or earlier, because the
36Haddock binary on that platform used to interact poorly with Darwin's `cpp` and
37thus failed until that bug was eventually fixed in GHC 7.8.1.
39This means that seemingly similar builds like `haskell.packages.ghc763.foo` and
40`haskell.packages.ghc784.foo` actually differ in multiple dimensions: those two
41`foo`s may be different versions of the same package; they are almost certainly
42compiled with different versions of their respective dependencies; and their
43configurations potentially differ, i.e both builds use different Cabal flags
44and have different features enabled. In fact, a build of `haskellPackages.foo`
45that's run on Linux/x86_64 most likely differs non-trivially from the exact
46same build run on Darwin/x86_64.
48Nix can handle this diversity, because it describes builds in a Turing-complete
49functional language. Builds that exist in multiple variants are functions that
50take the necessary dependencies plus a set of flags to enable/disable the
51features we intend to support, and then map that information to appropriate
52build instructions. Those functions live in one central place,
53[`hackage-packages.nix`][1], and all our crazy package sets simply call those
54functions with appropriate arguments to define exactly those build variants
57What sounds simple and obvious in theory, however, is a bit of a mess in
58practice. Consider the package `hslua` as an example. `hackage-packages.nix`
59defines builds for versions 0.3.13, 0.4.0, and 0.4.1. Those definitions look
60all quite similar, so we'll omit the third definition for brevity:
62 "hslua_0_3_13" = callPackage
63 ({ mkDerivation, base, lua5_1, mtl }:
67 sha256 = "e95774d21ac4378e799af4b8dce9068bfedd3109bf13b717d9ecb5817f86430a";
68 configureFlags = [ "-fsystem-lua" ];
69 libraryHaskellDepends = [ base mtl ];
70 libraryPkgconfigDepends = [ lua5_1 ];
71 testHaskellDepends = [ base ];
72 description = "A Lua language interpreter embedding in Haskell";
73 license = stdenv.lib.licenses.mit;
74 hydraPlatforms = stdenv.lib.platforms.none;
75 }) { inherit (pkgs) lua5_1; };
78 ({ mkDerivation, base, bytestring, hspec, hspec-contrib, HUnit
79 , lua5_1, QuickCheck, quickcheck-instances, text
84 sha256 = "2df2b4f0566ef2244506f9830e0207fce3bece7c331129f69f446c722136173f";
85 configureFlags = [ "-fsystem-lua" ];
86 libraryHaskellDepends = [ base bytestring ];
87 librarySystemDepends = [ lua5_1 ];
88 testHaskellDepends = [
89 base bytestring hspec hspec-contrib HUnit QuickCheck
90 quickcheck-instances text
92 description = "A Lua language interpreter embedding in Haskell";
93 license = stdenv.lib.licenses.mit;
94 }) { inherit (pkgs) lua5_1; };
96Let's go through that code step by step. The basic structure of those
99 name_version = callPackage f { ... system dependencies ... };
101where `f` is a function that given a set of build inputs produces a build. The
102code invokes that function right away, so apparently `name_version` defines a
103*build*, not a function that generates a build. The arguments passed to `f`
104contain only system-level dependencies like `lua5_1`. Haskell dependencies like
105`mtl` are not passed explicitly since the magic `callPackage` function can find
106them automatically. Haskell attributes have a `_version` suffix only if the
107attribute refers to an old version, like `hslua_0_3_13`. The latest version,
1080.4.1, is simply called `hslua`. Last but not least, every Haskell build
109expects a parameter `mkDerivation` --- our generic Haskell builder, which turns
110those attribute sets into build instructions that Nix can execute. This
111argument is also not passed explicitly but found by `callPackage`.
113The `callPackage` magic relies on two advanced techniques: fixed points and
114reflection. Nixpkgs defines the Haskell package set recursively as a function
115`self: { ... }` that takes as an argument the attribute set it's going to
116produce and then uses that information to the produce the attribute set it's
117already received as an argument. It's a mind-boggling way to define a recursive
118attribute set `rec { ... }` without actually using the `rec` keyword. Every
119computable function of this kind has a fixed point: a value `x` that satisfies
120`f x = x`. That `x` is the result we actually want --- it's an ordinary
121attribute set with all the `self`-references resolved and replaced by actual
122values. The details of fixed points aren't important, though. What's important
123is that our package set knows its final value already while we're still
124defining it! This allows for the existence of the `callPackage` function:
127 callPackage = stdenv.lib.callPackageWith self;
129 self // { ... Haskell package definitions here ... }
131The expression `callPackage f args` translates to `stdenv.lib.callPackageWith
132self f args`. [`callPackageWith`][2] then uses reflection to determine the
133names of all arguments expected by `f`. It finds those names that don't exist
134in `args` in `self` and adds them to `args` to complete the function call. In
135other words, parameters of `f` that are not specified in `args` are
136automatically filled in from `self` --- the Haskell package set. This is why
137system packages are passed explicitly (`self` doesn't contain them), but
138Haskell dependencies are not (`self` does contain them).
140This feature is nice in package sets human beings write manually, because it
141allows us to keep function calls short and concise. A computer program
142generates the file `hackage-packages.nix`, though, so arguably we could have it
143bind all Haskell names explicitly with `inherit (self) base mtl` just as well.
144But `callPackage` has another feature that's more important than implicit name
145resolution: it adds the method `override` to the generated derivation, which
146can "undo" the function application to change the value of arguments that were
147used originally. We can say
149 hslua.override { mtl = self.mtl_2_1_3_1; }
151to instantiate a variant of `hslua` that compiles with that particular version
152of `mtl` rather than the default one. The same way, we can change the behavior
153of `mkDerivation`! For example, the expression
156 mkDerivation = args: self.mkDerivation (args // { doCheck=false; });
159gives us a variant of `hslua` that has the test suite disabled. These kind of
160overrides are so useful that we've defined a whole [set of helper functions][3]
161that use this technique to customize the default definitions from
162`hackage-packages.nix` in any way we need.
164Now, Haskell packages sets are just collections of overrides that change the
165default choices made in `hackage-packages.nix` as required. For example, the
166package set for [LTS Haskell version 2.8][4] specifies:
168 "hslua" = doDistribute super."hslua_0_3_13";
170It renames `hslua_0_3_13` to `hslua` to ensure that all Haskell builds in that
171package set use the older version instead of the latest one. Furthermore, we
172would like Hydra builds of `haskell.packages.lts-2_8.hslua` enabled. The
173original definition from `hackage-packages.nix` disabled those builds via
175 hydraPlatforms = stdenv.lib.platforms.none
177which makes sense in `haskellPackages`, because it prefers the latest version
178of the package, but the older LTS set isn't happy with that choice, so it uses
179a `doDistribute` override to replace the original empty set of `hydraPlatforms`
180with something more appropriate.
182Generally speaking, all of our 50+ Haskell package sets are essentially just
183variants of `hackage-packages.nix` with certain choices overridden if
189It's unclear how to improve upon the current architecture.
191The override approach produces a weird structure: first, we define a build
192function for every Haskell package --- like one would expect in a purely
193functional language ---, but then that function isn't exposed. Instead, we call
194the function right away to create a derivation. However, we can't use an honest
195function call, because we really need a *function* later, so we're forced to
196use freaky lazy-evaluation features that allow us to undo the function call
197with `override` later to make the derivation look like the function again that
198it probably should have been in the first place.
200One possible approach is to take `callPackage` out of `hackage-packages.nix`.
201Instead of derivations, `hackage-packages.nix` could provide functions that
202package sets then call with appropriate parameters to define the actual build.
203This would require some 10,000 functions plus another ~7,000 function
204applications per package set. In the current architecture, some attributes from
205`hackage-packages.nix` are passed through unmodified, whereas the new approach
206requires each package set to define every build explicitly. It's unclear how
207that would affect the Nix interpreter in terms of memory requirements and
210Furthermore, just taking `callPackage` out of `hackage-packages.nix` won't
211remedy the need for overriding, because our definitions don't mention any of
212the build configuration flags, like `doCheck`. It's not possible to call the
216 { mkDerivation, base, bytestring, [... more inputs omitted ...] }:
221 [... more attributes omitted ...]
224with `hslua { doCheck = false; ... }` because `doCheck` is an argument of
225`mkDerivation`, which is hidden inside of that function. The established
226approach to solve that problem is to wrap `mkDerivation`, i.e.
229 mkDerivation = args: self.mkDerivation (args // { doCheck=false; });
232but this feels cumbersome compared to straight function application. We can
233lift all those `mkDerivation` arguments into the outer function:
236 { mkDerivation, base, bytestring, [... more inputs omitted ...]
237 , doCheck ? true, doHaddock ? true, enableSharedLibraries ? true
242 inherit doCheck doHaddock;
243 [... more attributes omitted ...]
246but this adds a dozen highly redundant arguments to every single definition!
247There might be clever ways around mentioning each of them explicitly, like:
250 { mkDerivation, base, bytestring, [... more inputs omitted ...]
253 stdenv.lib.callPackageWith args mkDerivation {
256 [... more attributes omitted ...]
259Yet that is exactly the kind of stuff we wanted to avoid in the first place. So
260maybe it would be best to batch all those variant options in an attribute set
264 { mkDerivation, base, bytestring, [... more inputs omitted ...]
270 [... more attributes omitted ...]
273This approach allows us to call `hslua { config.doCheck = false; }`, which
274seems nice enough. At the same time, it allows us to call
276 hslua { config.pname = "this-is-not-hslua"; }
278as well, which feels like a bug.
280An entirely different approach is to take advantage of lazy evaluation to
281define all possible build variants at once:
284 showFlag = str: b: (if b then "with" else "without") + "-" + str;
286 mkDerivation = { pname, version, buildDepends ? []
290 name = pname + "-" + version + "-" + showFlag "check" doCheck;
292 enableChecking = mkDerivation (args // { doCheck = true; });
293 disableChecking = mkDerivation (args // { doCheck = false; });
296 "hslua" = { base ? null }:
300 buildDepends = [ base ];
303 [ ((hslua { base = "base"; }).name)
304 ((hslua { base = "base"; }).disableChecking.name)
305 ((hslua { base = "base"; }).disableChecking.disableChecking.enableChecking.name)
308Each of those styles has its own advantages and disadvantages which aren't
309fully understood yet. Hopefully, this summary provides some food for thought
310and helps kick off a fruitful discussion.
314[1]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/hackage-packages.nix
315[2]: https://github.com/NixOS/nixpkgs/blob/master/lib/customisation.nix#L72-L97
316[3]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/lib.nix
317[4]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/configuration-lts-2.8.nix#L4171
318[5]: https://github.com/NixOS/nixpkgs/issues/6192
319[6]: https://github.com/NixOS/nixpkgs/issues/7953
320[7]: https://github.com/NixOS/nixpkgs/issues/9871
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list