git.iko.soy/typeable/cabal2nix
on master help sign in
doc: 4th installment of the future series
commite2ffbf646d98fb63140f043d7ca91eccceec7394 authorPeter Simons <simons@cryp.to> dateFri 16 Oct 2015, 14:02 (11 years ago) parentca76c91
8 files changed+262−4
compare with master
unifiedsplit
doc/.gitignore+3 doc/00-future-of-haskell-packaging-in-nixpkgs.md+1 doc/04-set-up-a-sufficiently-powerful-build-farm.md+164 doc/GNUmakefile+2 doc/create-workspace.r+50 doc/future.org+4−4 doc/list-haskell-store-paths.nix+27 doc/list-haskell-store-paths.sh+11
doc/.gitignore+3
@@ -5,3 +5,6 @@
55/*.log
66/*.out
77/*.pdf
8+/*.json
9+/.RData
10+/.Rhistory
doc/00-future-of-haskell-packaging-in-nixpkgs.md+1
@@ -34,6 +34,7 @@ The issues we've identified so far are:
3434- [Use Function Application To Escape Override Hell](01-use-function-application-to-escape-override-hell.md)
3535- [Guarantee Consistent Builds and Obsolete `overrideScope`](02-guarantee-consistent-builds-and-obsolete-overridescope.md)
3636- [Map Cabal Files To Nix Without Information Loss](03-map-cabal-files-to-nix-without-information-loss.md)
37+- [Set up a Sufficiently Powerful Build Farm](04-set-up-a-sufficiently-powerful-build-farm.md)
3738
3839[1]: https://github.com/NixOS/nixpkgs/commit/c0c82ea2ebbcf0632260a931cf832cac1c8a014e
3940[2]: https://github.com/NixOS/nixpkgs/blob/master/pkgs/top-level/haskell-packages.nix
doc/04-set-up-a-sufficiently-powerful-build-farm.mdnew file+164
@@ -0,0 +1,164 @@
1+Set up a Sufficiently Powerful Build Farm
2+=========================================
3+
4+The Problem
5+-----------
6+
7+`hydra.nixos.org` does not compile any of our LTS Haskell package sets. This
8+means that users of `haskell.packages.lts-x_y` cannot get any pre-compiled
9+binaries. It also means that those builds aren't verified, i.e. we won't notice
10+when changes to Nixpkgs break builds in those package sets. Furthermore, we
11+have [no pre-compiled binaries with profiling support][1].
12+
13+The Situation Today
14+-------------------
15+
16+We have 66 active package sets that define the following number of active
17+builds per platform:
18+
19+ pkgset builds
20+ 1: ghc6123 5173
21+ 2: ghc704 5182
22+ 3: ghc7102 5189
23+ 4: ghc722 5183
24+ 5: ghc742 5183
25+ 6: ghc763 5182
26+ 7: ghc783 5173
27+ 8: ghc784 5173
28+ 9: ghcHEAD 5188
29+ 10: ghcNokinds 5188
30+ 11: ghcjs 5172
31+ 12: lts-0_0 795
32+ 13: lts-0_1 795
33+ 14: lts-0_2 795
34+ 15: lts-0_3 795
35+ 16: lts-0_4 795
36+ 17: lts-0_5 795
37+ 18: lts-0_6 795
38+ 19: lts-0_7 795
39+ 20: lts-1_0 827
40+ 21: lts-1_1 827
41+ 22: lts-1_10 828
42+ 23: lts-1_11 829
43+ 24: lts-1_12 829
44+ 25: lts-1_13 829
45+ 26: lts-1_14 830
46+ 27: lts-1_15 831
47+ 28: lts-1_2 828
48+ 29: lts-1_4 828
49+ 30: lts-1_5 828
50+ 31: lts-1_7 828
51+ 32: lts-1_8 828
52+ 33: lts-1_9 827
53+ 34: lts-2_0 1019
54+ 35: lts-2_1 1019
55+ 36: lts-2_10 1023
56+ 37: lts-2_11 1023
57+ 38: lts-2_12 1023
58+ 39: lts-2_13 1022
59+ 40: lts-2_14 1022
60+ 41: lts-2_15 1022
61+ 42: lts-2_16 1022
62+ 43: lts-2_17 1023
63+ 44: lts-2_18 1022
64+ 45: lts-2_19 1022
65+ 46: lts-2_2 1018
66+ 47: lts-2_20 1024
67+ 48: lts-2_21 1023
68+ 49: lts-2_22 1023
69+ 50: lts-2_3 1018
70+ 51: lts-2_4 1018
71+ 52: lts-2_5 1018
72+ 53: lts-2_6 1017
73+ 54: lts-2_7 1017
74+ 55: lts-2_8 1023
75+ 56: lts-2_9 1023
76+ 57: lts-3_0 1322
77+ 58: lts-3_1 1322
78+ 59: lts-3_2 1321
79+ 60: lts-3_3 1321
80+ 61: lts-3_4 1321
81+ 62: lts-3_5 1322
82+ 63: lts-3_6 1321
83+ 64: lts-3_7 1323
84+ 65: lts-3_8 1323
85+ 66: lts-3_9 1324
86+ pkgset builds
87+
88+That gives a total of 111,647 active builds. Many of those builds are
89+identical: all package sets combined define 77,445 distinct store paths, i.e.
90+some 34,202 builds can be shared across package sets.
91+
92+Now, `hydra.nixos.org` compiles only `haskellPackages` at the moment. Out of a
93+total of [46,862 builds in trunk][2], 15,446 (33%) come from the Haskell
94+package set. If we'd enable every Haskell package set on Linux/i686,
95+Linux/x86_64, and Darwin/x86_64, then we'd have a total of 263,751 builds ---
96+5.6 times as much as before ---, and 88% of all builds would be related to
97+Haskell.
98+
99+A complete set of builds of `haskellPackages` takes up approx. 80 GByte of disk
100+space per platform, i.e. 240 GByte for all our 3 active platforms. How would
101+that number develop if we'd enable everything? The store path sizes in MByte
102+are distributed as follows (based on 13,070 samples excluding `ghc`):
103+
104+ Min. 1st Qu. Median Mean 3rd Qu. Max.
105+ 0.02 0.37 0.91 3.92 2.71 678.90
106+
107+Multiplying the average size by the number of distinct store paths tells us
108+that storing *everything* requires approx. 300 GByte per platform. With 3
109+active platforms, we'd need about 1 TByte of disk space for one complete set of
110+Haskell packages.
111+
112+Now, we might be able to reduce that number by disabling some particularly
113+large builds. It turns out that the size distribution is skewed to the left,
114+i.e. towards smaller builds. Approximately 81% of all store paths lie below the
115+average. Our top-20 biggest Haskell builds are:
116+
117+ pkg size
118+ 1: ghc 895.5
119+ 2: metadata 678.9
120+ 3: uhc-light 253.5
121+ 4: OpenGLRaw 249.3
122+ 5: FpMLv53 229.5
123+ 6: amazonka-ec2 214.8
124+ 7: Agda 212.6
125+ 8: xhb 189.0
126+ 9: unicode-properties 175.0
127+ 10: scholdoc-texmath 165.0
128+ 11: idris 137.7
129+ 12: gf 126.4
130+ 13: pandoc 124.8
131+ 14: unicode-names 118.4
132+ 15: wxcore 113.8
133+ 16: java-character 112.1
134+ 17: hat 111.1
135+ 18: texmath 109.6
136+ 19: open-symbology 107.6
137+ 20: turkish-deasciifier 104.0
138+
139+If we'd make an effort to disable some of those expensive builds --- or maybe
140+reduce their output size ---, then that would make a noticeable dent into the
141+space requirements.
142+
143+Anyway, it's clear that `hydra.nixos.org` cannot cope with that load. Curiously
144+enough, compiling the packages is the least of our problems; that's no big
145+deal. But evaluating 66 package sets with some 111,000 derivations in them
146+every 10 minutes might be unfeasible. Hydra has undergone some architectural
147+changes recently that might make a difference, but I don't have any reliable
148+data concerning the performance of the `hydra-evaluator` process, so I cannot
149+say.
150+
151+We know for sure that the currently available disk space would not suffice.
152+Disk space is notoriously low on `hydra.nixos.org` and storing another terabyte
153+Haskell data is certainly impossible at the moment.
154+
155+Possible Improvements
156+---------------------
157+
158+
159+
160+
161+
162+
163+[1]: https://github.com/NixOS/nixpkgs/issues/10143
164+[2]: http://hydra.nixos.org/jobset/nixpkgs/trunk
doc/GNUmakefile+2
@@ -9,6 +9,8 @@ all:: 00-future-of-haskell-packaging-in-nixpkgs.pdf
99all:: 01-use-function-application-to-escape-override-hell.pdf
1010all:: 02-guarantee-consistent-builds-and-obsolete-overridescope.pdf
1111all:: 03-map-cabal-files-to-nix-without-information-loss.md
12+all:: 04-set-up-a-sufficiently-powerful-build-farm.md
13+
1214
1315%.tex : %.md
1416 pandoc -s $(PANDOC_FLAGS) -f markdown -t latex $< -o $@
doc/create-workspace.rnew file+50
@@ -0,0 +1,50 @@
1+# cabal2nix/doc/create-workspace.r
2+
3+library(data.table)
4+library(rjson)
5+library(foreach)
6+library(doParallel)
7+
8+### Set up the workspace.
9+
10+cores <- detectCores()
11+options(cores=cores, mc.cores=cores)
12+cl <- makeForkCluster(cores)
13+registerDoParallel(cl)
14+setDefaultCluster(cl)
15+
16+builds <- foreach (i=list.files(".", "*.json"), .combine="rbind") %dopar% {
17+ print(i)
18+ l <- fromJSON(file=i)
19+ pkgset <- names(l)
20+ data.table(pkgset, pkg=names(l[[pkgset]]), out=unlist(l[[pkgset]]))
21+}
22+setkey(builds, pkgset, pkg)
23+
24+t <- foreach (out=unique(builds$out), .combine="rbind") %dopar% {
25+ size=system(paste("( du -sb", out, "| cut -f1 ) 2>/dev/null || true"), intern=TRUE)
26+ size <- if (identical(size, character(0))) "" else size
27+ data.table(out, size=as.integer(size))
28+}
29+
30+builds <- merge(builds, t, by="out")
31+
32+### Analyze the data.
33+
34+# Total number of attributes in all package sets.
35+builds_total <- nrow(builds)
36+
37+# Number of distict builds (output paths).
38+builds_unique <- length(unique(builds$out))
39+
40+# Average output path size.
41+avg_build_size <- mean(builds[pkg != "ghc", size], na.rm=T)
42+sd_build_size <- sd(builds[pkg != "ghc", size], na.rm=T)
43+summary(builds[pkg != "ghc", size/1e6])
44+
45+# Biggest builds.
46+biggest_builds <- head(builds[!is.na(size),list(size=mean(size)),by=pkg][order(size, decreasing=T)], n=20)
47+biggest_builds$size <- round(biggest_builds$size / 1e6, 1)
48+
49+# Number of builds, distinguished by package set.
50+builds_by_pkgset <- builds[,list(attributes=length(pkg)), by=pkgset][order(pkgset)]
doc/future.org+4−4
@@ -1,13 +1,13 @@
11#+TITLE: The Future of Haskell Packaging in Nix
22
3−* TODO Use Function Application To Escape Override Hell
4−* TODO Guarantee Consistent Builds and Obsolete =overrideScope=
5−* TODO Map Cabal Files To Nix Without Information Loss
3+* DONE Use Function Application To Escape Override Hell
4+* DONE Guarantee Consistent Builds and Obsolete =overrideScope=
5+* DONE Map Cabal Files To Nix Without Information Loss
66* TODO Implement Proper Dependency Resolution
77* TODO Develop Intelligent Policies for Post-processing and Patching
88* TODO Users Need a Configuration Mechanism That Is Both Simpler and More Powerful
99* TODO Deal With Infinite Recursions in Test Dependencies
10−* TODO Set up a Sufficiently Powerful Build Farm
10+* DONE Set up a Sufficiently Powerful Build Farm
1111* TODO Define and Publish Processes for Managing the Package Set(s)
1212* TODO Publish All of Hackage
1313
doc/list-haskell-store-paths.nixnew file+27
@@ -0,0 +1,27 @@
1+# list-haskell-store-paths.nix
2+
3+let
4+
5+ inherit (import <nixpkgs> { config.allowBroken = true; }) lib haskell;
6+
7+ listStorePaths = pkgset: lib.mapAttrs
8+ (attr: drv: builtins.toPath drv)
9+ (lib.filterAttrs isActiveBuild haskell.packages.${pkgset});
10+
11+ isActiveBuild = name: drv: let r = builtins.tryEval (hasNonEmptyPlatforms drv && isNotBroken drv);
12+ in if r.success then r.value else false;
13+
14+ hasNonEmptyPlatforms = drv: lib.isDerivation drv && getPlatforms drv != [];
15+
16+ isNotBroken = drv: !(lib.attrByPath ["meta" "broken"] false drv);
17+
18+ getPlatforms = drv: lib.attrByPath ["hydraPlatforms"]
19+ (lib.attrByPath ["Platforms"] [])
20+ (drv.meta or {});
21+
22+in
23+
24+# Using "lib.attrNames haskell.packages" instead of pkgset would be nice, but
25+# nix-instantiate doesn't cope.
26+
27+{ pkgset }: lib.genAttrs [pkgset] listStorePaths
doc/list-haskell-store-paths.shnew file+11
@@ -0,0 +1,11 @@
1+#! /usr/bin/env bash
2+
3+set -eu -o pipefail
4+
5+pkgsets=( $(nix-instantiate --eval -E 'with (import <nixpkgs> {}); builtins.attrNames haskell.packages') )
6+n=$(( ${#pkgsets[@]} - 2 ))
7+
8+for n in ${pkgsets[@]:1:$n}; do
9+ eval n=$n
10+ nix-instantiate --eval --strict --json list-haskell-store-paths.nix --argstr pkgset $n >$n.json
11+done