1Set up a Sufficiently Powerful Build Farm
2=========================================
7`hydra.nixos.org` compiles and provides binaries only for the `haskellPackages`
8package set. The build farm compiles none of our LTS Haskell package sets,
9which means that users of `haskell.packages.lts-x_y` cannot get any
10pre-compiled binaries. It also means that those builds aren't verified, i.e. we
11won't notice when changes to Nixpkgs break builds in those package sets.
13Furthermore, we have [no pre-compiled binaries with profiling support][1] for
14any of our package sets.
19We have 66 active package sets that define the following number of active
91That gives a total of 111,647 active builds, many of which are identical. All
92package sets combined define 77,445 distinct store paths, i.e. some 34,202
93builds are shared across package sets.
95Now, `hydra.nixos.org` compiles only `haskellPackages` at the moment. Out of a
96total of [46,862 builds in trunk][2], 15,446 (33%) come from the Haskell
97package set. If we'd enable every Haskell package set on Linux/i686,
98Linux/x86_64, and Darwin/x86_64, then we'd have a total of 263,751 builds ---
995.6 times as much as before ---, and 88% of all builds would be related to
102A complete build of the active derivations in `haskellPackages` takes up
103approx. 27 GByte of disk space per platform. That gives about 80 GByte for all
104of our 3 active platforms. How would that number develop if we'd enable
105everything? The store path sizes in MByte are distributed as follows (based on
1067,620 samples excluding `ghc`):
108 Minimum 1st Quart. Median Mean 3rd Quart. Maximum
109 0.0169 0.3557 0.9497 4.6300 3.0640 678.9000
111Multiplying the average store path size by the number of distinct store paths
112tells us that storing *everything* requires approx. 360 GByte per platform.
113With 3 active platforms, we'd need about 1 TByte of disk space for one complete
114set of Haskell packages.
116Now, we might be able to reduce that number by disabling some particularly
117large builds. The store path size distribution is skewed to the left, i.e.
118towards smaller builds. Approximately 82% of all store paths are actually
119smaller than the numerical average. Our top-20 biggest Haskell builds are:
127 6: amazonka-ec2 214.8
130 9: unicode-properties 175.0
131 10: scholdoc-texmath 165.0
135 14: unicode-names 118.4
137 16: java-character 112.1
140 19: open-symbology 107.6
141 20: turkish-deasciifier 104.0
143If we'd make an effort to disable some of those expensive builds --- or maybe
144reduce their output size ---, then we'd make a noticeable dent into the space
145requirements. Even so, it's clear that `hydra.nixos.org` cannot provide that
146much disk space today.
148Curiously enough, the CPU power necessary to compile all those packages is the
149least of our problems. Our build farm can easily re-compile everything from
150scratch within 2-3 days, which is "good enough" for all practical purposes.
151Also, changes to `stdenv` occur rarely (and we typically know about them in
152advance). The normal update cycle triggers only a handful of builds -- maybe
15320-300 per day --, because the versions fundamental Haskell packages are fixed
154in the LTS package sets.
156It's unclear whether the Hydra software would cope with 66 package sets with
157some 111,000 derivations in them that need to be evaluated, say, once an hour.
158Hydra has undergone some architectural changes recently that might make such a
159load possible --- i.e. `hydra-evaluator` is more efficient than it used to be
160---, but I don't have any reliable data concerning the performance of the
161process, so I cannot say what is possible and what is not.
163We know for sure that the currently available disk space doesn't suffice. Disk
164space is notoriously low on `hydra.nixos.org`, and storing another terabyte
165Haskell data is certainly impossible at the moment.
170We have basically two alternatives:
1721. Throw hardware (money) at `hydra.nixos.org`.
1732. Establish a separate build farm for Haskell packages.
175Either solution requires money, which we could probably raise through crowd
176funding. At the moment, the NixOS Foundation collects donations for purposes of
177NixOS in general, but it should be possible to start a funding campaign that
178collects donations specifically for the purposes of establishing a Haskell
179build farm so that people who care about that particular topic have an
180incentive to participate.
182Now, if we'd go for approach (1), then we could use those funds to buy bigger
183disks and more RAM for `hydra.nixos.org`, which would be beneficial for
184everyone -- not just Haskell users. The downside is that `hydra.nixos.org` is a
185bit of a black box. Only very few people have access to those machines, and
186that situation is not going to change any time soon. Personally, I have no idea
187whether adding RAM or disks to the cluster is feasible at all, and whether
188those upgrades would enable the build farm to cope with the number of builds
189that we're considering here.
191Solution (2) seems more manageable, because we could set up an environment from
192scratch as we see fit. Experience from managing `hydra.cryp.to` suggests that
193one powerful KVM-based virtual server can serve as the Hydra master. In
194addition, we'd need 2-3 additional build slaves to compile packages. For
195massive re-builds, we could spawn another 10-15 builds slaves in EC2 to reduce
196the time it takes to re-build everything from scratch. Such a setup would
197probably work well in practice, and it should be available at a yearly cost of
1981,000 dollars or less.
200Anyhow, that's just a rough estimate. I don't know, really, what an ideal
201hardware / service platform for running such a virtual service would be. It
202would be great if a resident virtual server / NAS / system management guru
203could chime in with suggestions; I'm sure the NixOS crowd has people who know
204that kind of stuff and who can design the infrastructure for such a build farm.
207[1]: https://github.com/NixOS/nixpkgs/issues/10143
208[2]: http://hydra.nixos.org/jobset/nixpkgs/trunk
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list