@@ -0,0 +1,164 @@
1+Set up a Sufficiently Powerful Build Farm
2+=========================================
7+`hydra.nixos.org` does not compile any of our LTS Haskell package sets. This
8+means that users of `haskell.packages.lts-x_y` cannot get any pre-compiled
9+binaries. It also means that those builds aren't verified, i.e. we won't notice
10+when changes to Nixpkgs break builds in those package sets. Furthermore, we
11+have [no pre-compiled binaries with profiling support][1].
16+We have 66 active package sets that define the following number of active
88+That gives a total of 111,647 active builds. Many of those builds are
89+identical: all package sets combined define 77,445 distinct store paths, i.e.
90+some 34,202 builds can be shared across package sets.
92+Now, `hydra.nixos.org` compiles only `haskellPackages` at the moment. Out of a
93+total of [46,862 builds in trunk][2], 15,446 (33%) come from the Haskell
94+package set. If we'd enable every Haskell package set on Linux/i686,
95+Linux/x86_64, and Darwin/x86_64, then we'd have a total of 263,751 builds ---
96+5.6 times as much as before ---, and 88% of all builds would be related to
99+A complete set of builds of `haskellPackages` takes up approx. 80 GByte of disk
100+space per platform, i.e. 240 GByte for all our 3 active platforms. How would
101+that number develop if we'd enable everything? The store path sizes in MByte
102+are distributed as follows (based on 13,070 samples excluding `ghc`):
104+ Min. 1st Qu. Median Mean 3rd Qu. Max.
105+ 0.02 0.37 0.91 3.92 2.71 678.90
107+Multiplying the average size by the number of distinct store paths tells us
108+that storing *everything* requires approx. 300 GByte per platform. With 3
109+active platforms, we'd need about 1 TByte of disk space for one complete set of
112+Now, we might be able to reduce that number by disabling some particularly
113+large builds. It turns out that the size distribution is skewed to the left,
114+i.e. towards smaller builds. Approximately 81% of all store paths lie below the
115+average. Our top-20 biggest Haskell builds are:
123+ 6: amazonka-ec2 214.8
126+ 9: unicode-properties 175.0
127+ 10: scholdoc-texmath 165.0
131+ 14: unicode-names 118.4
133+ 16: java-character 112.1
136+ 19: open-symbology 107.6
137+ 20: turkish-deasciifier 104.0
139+If we'd make an effort to disable some of those expensive builds --- or maybe
140+reduce their output size ---, then that would make a noticeable dent into the
143+Anyway, it's clear that `hydra.nixos.org` cannot cope with that load. Curiously
144+enough, compiling the packages is the least of our problems; that's no big
145+deal. But evaluating 66 package sets with some 111,000 derivations in them
146+every 10 minutes might be unfeasible. Hydra has undergone some architectural
147+changes recently that might make a difference, but I don't have any reliable
148+data concerning the performance of the `hydra-evaluator` process, so I cannot
151+We know for sure that the currently available disk space would not suffice.
152+Disk space is notoriously low on `hydra.nixos.org` and storing another terabyte
153+Haskell data is certainly impossible at the moment.
155+Possible Improvements
156+---------------------
163+[1]: https://github.com/NixOS/nixpkgs/issues/10143
164+[2]: http://hydra.nixos.org/jobset/nixpkgs/trunk