@@ -0,0 +1,69 @@
1+Publish All of Hackage
2+======================
3+
4+The Problem
5+-----------
6+
7+Users of Nixpkgs want to build Haskell packages that have no recent release,
8+i.e. the software's authors have not updated their code to deal with modern
9+versions of the respective build dependencies. The required old versions,
10+however, may not be available in Nixpkgs, and then the build cannot succeed. At
11+that point, we recommend our users to
12+[add the missing package versions themselves][1] through the override mechanism
13+in `~/.nixpkgs/config.nix`. That approach works fine, but it's challenging for
14+new users and it's certainly inconvenient.
15+
16+The Situation Today
17+-------------------
18+
19+We publish 11,216 package versions out of 62,354 available ones, i.e. 18% of
20+Hackage. We select packages for distribution using the following criteria:
21+
22+1. the latest version of every package (9,058 builds),
23+
24+2. old versions required by LTS Haskell (2,137 builds), and
25+
26+3. old versions manually configured in the `extra-packages` field of
27+ [`configuration-hackage2nix.yaml`][2] (21 builds).
28+
29+The resulting [`hackage-packages.nix`][3] file is ~8.3 MByte large.
30+
31+Now, we could simply publish all of Hackage, i.e. all 62,354 available builds.
32+That would certainly make the lives of many Haskell hackers easier, but
33+unfortunately the resulting `hackage-packages.nix` file would require 52 MByte
34+of space. Furthermore, simple benchmarks suggest that `nix-instantiate` needs a
35+whopping 4 seconds to instantiate a `ghcWithPackages` environment from the new
36+database where instantiating the same environment used to take 0.45 seconds
37+before --- a performance drop by a factor of 8.
38+
39+Possible Improvements
40+---------------------
41+
42+We could continue to publish the "small" `hackage-packages.nix` database by
43+default, but generate an additional large database that we'd store in a
44+separate location. We would then add support for an option to
45+`~/.nixpkgs/config.nix` users could set to opt in to using the large database
46+if they want to. That way, Haskell users would have an option to get all of
47+Hackage without causing a performance drop for everyone else.
48+
49+One issue is that checking that ~50MB file into Nixpkgs might be a bad idea,
50+because it sets a dangerous precedent. Arguably, if we check all of Hackage
51+into Nixpkgs, then we cannot reasonably say *no* to someone who wants to check
52+all of CPAN into Nixpkgs too, and before we know it our Git repository triples
53+in size. So it might be wise to put the full Hackage variant of
54+`hackage-packages.nix` into a separate repository that we fetch via
55+`builtins.fetchurl` if the corresponding option is enabled.
56+
57+An altogether different approach is to improve our tools like `cabal2nix` to
58+make it convenient to add old versions from Hackage into a local Nixpkgs
59+instance, i.e. by extending the program to generate builds for all missing
60+versions to complete the transitive closure of the requested build so that
61+users don't have to run it a dozen times because multiple old versions are
62+missing. Conceivably, we could also provide tools that manage
63+`~/.nixpkgs/config.nix`, which might be a good idea independent of these
64+Haskell-specific issues.
65+
66+
67+[1]: http://nixos.org/nixpkgs/manual/#how-to-create-nix-builds-for-your-own-private-haskell-packages
68+[2]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/configuration-hackage2nix.yaml
69+[3]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/hackage-packages.nix