git.iko.soy/typeable/cabal2nix
on v20160613 help sign in
typeable/cabal2nix/doc/05-publish-all-of-hackage.md
69 lines 3.5 kB history raw
1Publish All of Hackage
2======================
4The Problem
5-----------
7Users of Nixpkgs want to build Haskell packages that have no recent release,
8i.e. the software's authors have not updated their code to deal with modern
9versions of the respective build dependencies. The required old versions,
10however, may not be available in Nixpkgs, and then the build cannot succeed. At
11that point, we recommend our users to
12[add the missing package versions themselves][1] through the override mechanism
13in `~/.nixpkgs/config.nix`. That approach works fine, but it's challenging for
14new users and it's certainly inconvenient.
16The Situation Today
17-------------------
19We publish 11,216 package versions out of 62,354 available ones, i.e. 18% of
20Hackage. We select packages for distribution using the following criteria:
221. the latest version of every package (9,058 builds),
242. old versions required by LTS Haskell (2,137 builds), and
263. old versions manually configured in the `extra-packages` field of
27 [`configuration-hackage2nix.yaml`][2] (21 builds).
29The resulting [`hackage-packages.nix`][3] file is ~8.3 MByte large.
31Now, we could simply publish all of Hackage, i.e. all 62,354 available builds.
32That would certainly make the lives of many Haskell hackers easier, but
33unfortunately the resulting `hackage-packages.nix` file would require 52 MByte
34of space. Furthermore, simple benchmarks suggest that `nix-instantiate` needs a
35whopping 4 seconds to instantiate a `ghcWithPackages` environment from the new
36database where instantiating the same environment used to take 0.45 seconds
37before --- a performance drop by a factor of 8.
39Possible Improvements
40---------------------
42We could continue to publish the "small" `hackage-packages.nix` database by
43default, but generate an additional large database that we'd store in a
44separate location. We would then add support for an option to
45`~/.nixpkgs/config.nix` users could set to opt in to using the large database
46if they want to. That way, Haskell users would have an option to get all of
47Hackage without causing a performance drop for everyone else.
49One issue is that checking that ~50MB file into Nixpkgs might be a bad idea,
50because it sets a dangerous precedent. Arguably, if we check all of Hackage
51into Nixpkgs, then we cannot reasonably say *no* to someone who wants to check
52all of CPAN into Nixpkgs too, and before we know it our Git repository triples
53in size. So it might be wise to put the full Hackage variant of
54`hackage-packages.nix` into a separate repository that we fetch via
55`builtins.fetchurl` if the corresponding option is enabled.
57An altogether different approach is to improve our tools like `cabal2nix` to
58make it convenient to add old versions from Hackage into a local Nixpkgs
59instance, i.e. by extending the program to generate builds for all missing
60versions to complete the transitive closure of the requested build so that
61users don't have to run it a dozen times because multiple old versions are
62missing. Conceivably, we could also provide tools that manage
63`~/.nixpkgs/config.nix`, which might be a good idea independent of these
64Haskell-specific issues.
67[1]: http://nixos.org/nixpkgs/manual/#how-to-create-nix-builds-for-your-own-private-haskell-packages
68[2]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/configuration-hackage2nix.yaml
69[3]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/hackage-packages.nix