@@ -4,11 +4,14 @@ Set up a Sufficiently Powerful Build Farm
44The Problem
55-----------
66
7−`hydra.nixos.org` does not compile any of our LTS Haskell package sets. This
8−means that users of `haskell.packages.lts-x_y` cannot get any pre-compiled
9−binaries. It also means that those builds aren't verified, i.e. we won't notice
10−when changes to Nixpkgs break builds in those package sets. Furthermore, we
11−have [no pre-compiled binaries with profiling support][1].
7+`hydra.nixos.org` compiles and provides binaries only for the `haskellPackages`
8+package set. The build farm compiles none of our LTS Haskell package sets,
9+which means that users of `haskell.packages.lts-x_y` cannot get any
10+pre-compiled binaries. It also means that those builds aren't verified, i.e. we
11+won't notice when changes to Nixpkgs break builds in those package sets.
12+
13+Furthermore, we have [no pre-compiled binaries with profiling support][1] for
14+any of our package sets.
1215
1316The Situation Today
1417-------------------
@@ -85,9 +88,9 @@ builds per platform:
8588 66: lts-3_9 1324
8689 pkgset builds
8790
88−That gives a total of 111,647 active builds. Many of those builds are
89−identical: all package sets combined define 77,445 distinct store paths, i.e.
90−some 34,202 builds can be shared across package sets.
91+That gives a total of 111,647 active builds, many of which are identical. All
92+package sets combined define 77,445 distinct store paths, i.e. some 34,202
93+builds are shared across package sets.
9194
9295Now, `hydra.nixos.org` compiles only `haskellPackages` at the moment. Out of a
9396total of [46,862 builds in trunk][2], 15,446 (33%) come from the Haskell
@@ -96,23 +99,24 @@ Linux/x86_64, and Darwin/x86_64, then we'd have a total of 263,751 builds ---
96995.6 times as much as before ---, and 88% of all builds would be related to
97100Haskell.
98101
99−A complete set of builds of `haskellPackages` takes up approx. 80 GByte of disk
100−space per platform, i.e. 240 GByte for all our 3 active platforms. How would
101−that number develop if we'd enable everything? The store path sizes in MByte
102−are distributed as follows (based on 13,070 samples excluding `ghc`):
102+A complete build of the active derivations in `haskellPackages` takes up
103+approx. 27 GByte of disk space per platform. That gives about 80 GByte for all
104+of our 3 active platforms. How would that number develop if we'd enable
105+everything? The store path sizes in MByte are distributed as follows (based on
106+7,620 samples excluding `ghc`):
103107
104− Min. 1st Qu. Median Mean 3rd Qu. Max.
105− 0.02 0.37 0.91 3.92 2.71 678.90
108+ Minimum 1st Quart. Median Mean 3rd Quart. Maximum
109+ 0.0169 0.3557 0.9497 4.6300 3.0640 678.9000
106110
107−Multiplying the average size by the number of distinct store paths tells us
108−that storing *everything* requires approx. 300 GByte per platform. With 3
109−active platforms, we'd need about 1 TByte of disk space for one complete set of
110−Haskell packages.
111+Multiplying the average store path size by the number of distinct store paths
112+tells us that storing *everything* requires approx. 360 GByte per platform.
113+With 3 active platforms, we'd need about 1 TByte of disk space for one complete
114+set of Haskell packages.
111115
112116Now, we might be able to reduce that number by disabling some particularly
113−large builds. It turns out that the size distribution is skewed to the left,
114−i.e. towards smaller builds. Approximately 81% of all store paths lie below the
115−average. Our top-20 biggest Haskell builds are:
117+large builds. The store path size distribution is skewed to the left, i.e.
118+towards smaller builds. Approximately 82% of all store paths are actually
119+smaller than the numerical average. Our top-20 biggest Haskell builds are:
116120
117121 pkg size
118122 1: ghc 895.5
@@ -137,27 +141,67 @@ average. Our top-20 biggest Haskell builds are:
137141 20: turkish-deasciifier 104.0
138142
139143If we'd make an effort to disable some of those expensive builds --- or maybe
140−reduce their output size ---, then that would make a noticeable dent into the
141−space requirements.
142−
143−Anyway, it's clear that `hydra.nixos.org` cannot cope with that load. Curiously
144−enough, compiling the packages is the least of our problems; that's no big
145−deal. But evaluating 66 package sets with some 111,000 derivations in them
146−every 10 minutes might be unfeasible. Hydra has undergone some architectural
147−changes recently that might make a difference, but I don't have any reliable
148−data concerning the performance of the `hydra-evaluator` process, so I cannot
149−say.
150−
151−We know for sure that the currently available disk space would not suffice.
152−Disk space is notoriously low on `hydra.nixos.org` and storing another terabyte
144+reduce their output size ---, then we'd make a noticeable dent into the space
145+requirements. Even so, it's clear that `hydra.nixos.org` cannot provide that
146+much disk space today.
147+
148+Curiously enough, the CPU power necessary to compile all those packages is the
149+least of our problems. Our build farm can easily re-compile everything from
150+scratch within 2-3 days, which is "good enough" for all practical purposes.
151+Also, changes to `stdenv` occur rarely (and we typically know about them in
152+advance). The normal update cycle triggers only a handful of builds -- maybe
153+20-300 per day --, because the versions fundamental Haskell packages are fixed
154+in the LTS package sets.
155+
156+It's unclear whether the Hydra software would cope with 66 package sets with
157+some 111,000 derivations in them that need to be evaluated, say, once an hour.
158+Hydra has undergone some architectural changes recently that might make such a
159+load possible --- i.e. `hydra-evaluator` is more efficient than it used to be
160+---, but I don't have any reliable data concerning the performance of the
161+process, so I cannot say what is possible and what is not.
162+
163+We know for sure that the currently available disk space doesn't suffice. Disk
164+space is notoriously low on `hydra.nixos.org`, and storing another terabyte
153165Haskell data is certainly impossible at the moment.
154166
155167Possible Improvements
156168---------------------
157169
158−
159−
160−
170+We have basically two alternatives:
171+
172+1. Throw hardware (money) at `hydra.nixos.org`.
173+2. Establish a separate build farm for Haskell packages.
174+
175+Either solution requires money, which we could probably raise through crowd
176+funding. At the moment, the NixOS Foundation collects donations for purposes of
177+NixOS in general, but it should be possible to start a funding campaign that
178+collects donations specifically for the purposes of establishing a Haskell
179+build farm so that people who care about that particular topic have an
180+incentive to participate.
181+
182+Now, if we'd go for approach (1), then we could use those funds to buy bigger
183+disks and more RAM for `hydra.nixos.org`, which would be beneficial for
184+everyone -- not just Haskell users. The downside is that `hydra.nixos.org` is a
185+bit of a black box. Only very few people have access to those machines, and
186+that situation is not going to change any time soon. Personally, I have no idea
187+whether adding RAM or disks to the cluster is feasible at all, and whether
188+those upgrades would enable the build farm to cope with the number of builds
189+that we're considering here.
190+
191+Solution (2) seems more manageable, because we could set up an environment from
192+scratch as we see fit. Experience from managing `hydra.cryp.to` suggests that
193+one powerful KVM-based virtual server can serve as the Hydra master. In
194+addition, we'd need 2-3 additional build slaves to compile packages. For
195+massive re-builds, we could spawn another 10-15 builds slaves in EC2 to reduce
196+the time it takes to re-build everything from scratch. Such a setup would
197+probably work well in practice, and it should be available at a yearly cost of
198+1,000 dollars or less.
199+
200+Anyhow, that's just a rough estimate. I don't know, really, what an ideal
201+hardware / service platform for running such a virtual service would be. It
202+would be great if a resident virtual server / NAS / system management guru
203+could chime in with suggestions; I'm sure the NixOS crowd has people who know
204+that kind of stuff and who can design the infrastructure for such a build farm.
161205
162206
163207[1]: https://github.com/NixOS/nixpkgs/issues/10143