@@ -0,0 +1,191 @@
1+`stackage-curator` is the tool used by the Stackage Curator team to:
2+
3+* Calculate Stackage Nightly and LTS Haskell build plans
4+* Check that these plans build and pass tests
5+* Generate Haddock files
6+* Upload the plans and Haddocks to the appropriate places:
7+ * S3 for the docs
8+ * Github for the YAML files
9+
10+The current version of the tool was started a while ago by Michael
11+Snoyman, and has evolved significantly since then. It's currently
12+something of a mess, and is holding us back. This document is intended
13+to list out goals for a rewrite, and spell out the responsibilities
14+this tool needs to meet.
15+
16+Note that there was an earlier document beginning to lay out some of
17+this, but it was not as comprehensive:
18+https://github.com/fpco/stackage-curator/blob/use-stack/PLAN2.md.
19+
20+## Goals for rewrite
21+
22+* __Use Stack for building__. Currently, `stackage-curator` builds by
23+ calling out to `Setup.hs` files directly. If we move over to Stack,
24+ we'll automatically inherit a bunch of nice functionality
25+ (especially caching), and have less maintenance burden.
26+* __Share library code with Stack__. There's an
27+ [open issue for Stack](https://github.com/commercialhaskell/stack/issues/3620)
28+ to separate off some general-purpose code around the Hackage index
29+ and cabal file parsing. The primary codebase to share the code with
30+ is `stackage-curator`. Doing this would reduce the
31+ `stackage-curator` maintenance burden.
32+* __Reduce size of the YAML files__. The YAML files currently include
33+ a _lot_ of information that most people don't need, such as the list
34+ of modules per package and reverse dependencies. This information is
35+ needed for `stackage-server`, but `stackage-server` itself could
36+ derive that information by parsing cabal files instead. We should
37+ reduce the unneeded information in these files to save user
38+ bandwidth.
39+* __Hash based downloads__. It would be nice to implement the
40+ necessary changes to make
41+ [hash based downloads](https://www.fpcomplete.com/blog/2018/01/hash-based-package-downloads-part-2-of-2)
42+ a reality, further increasing security and decreasing bandwidth
43+ usage.
44+* __Allow modifications to packages__. Currently, all packages we
45+ include are identical to their Hackage variants. We theoretically
46+ want to start allowing modifications to packages, and maybe even
47+ packages that are not on Hackage. (The latter is significantly more
48+ contentious; see
49+ [my SLURP blog post](https://www.snoyman.com/blog/2018/01/slurp).)
50+ The hash-based downloads helps with this somewhat, but I haven't
51+ fully fleshed out the ideas in the document below.
52+
53+## Responsibilities for stackage-curator
54+
55+### Calculate the build constraints
56+
57+The `fpco/stackage` repo has a `build-constraints.yaml` file which
58+places a bunch of _constraints_ on the build. This includes which
59+packages are present, which upper bounds to respect, tests/benchmarks
60+to skip, etc. For a Stackage Nightly run, calculating these
61+constraints is simple: parse that YAML file.
62+
63+For an LTS major release (e.g., `lts-11.0`), we do exactly the same
64+thing: take a current `build-constraints.yaml` and calculate a set of
65+constraints.
66+
67+For an LTS minor release (e.g., `lts-11.5`), the process is
68+different. We take the previous LTS plan (e.g., `lts-11.4`) and
69+calculate the constraints from it. This means inheriting information
70+on the tests/benchmarks to skip, for example. It also means creating
71+some upper bounds. The rule here is:
72+
73+* If the build plan includes an explicit version bound, like `mtl <
74+ 2.2.3`, we respect that bound. As the curator team knows, we'll
75+ often add such explicit bounds due to some new breaking change.
76+* If no explicit version bound exists, then we follow PVP rules and
77+ keep the same major version. So, for instance, if we have
78+ `stm-2.3.4` in the previous LTS snapshot, and no explicit upper
79+ bounds, we'll generate a bound `stm < 2.4`.
80+
81+__NOTE__ This implies that the version bound information needs to be
82+in the LTS build plans, even though it is only needed for
83+`stackage-curator` itself. An altnerative would be to store a separate
84+file with additional metadata that only `stackage-curator` cares
85+about, including tests/benchmarks to skip. We do not do this today.
86+
87+Sometimes, when creating this constraints from a previous LTS, it
88+becomes necessary to add new, additional constraints. We do that today
89+via the `CONSTRAINTS` environment variable. I'm not convinced this is
90+the best way to handle things: it's tedious, and allows us no ability
91+to add comments as to why constraints were added. Here's a
92+theoretically better design, meant mostly as a strawman:
93+
94+* `stackage-curator` spits out a new `build-constraints-lts-11.5.yaml`
95+ file, with all of the constraints it calculated.
96+* The curator team can edit that file directly to modify constraints,
97+ and add any extra metadata (such as via comments) desired.
98+* This file gets uploaded together with the actual `lts-11.5.yaml`
99+ file.
100+
101+This step definitely deserves some discussion.
102+
103+### Calculate the build plan
104+
105+Now that we have all of the constraints, we can figure out an actual
106+plan. This will combine the constraints with the current state of the
107+Hackage package index to decide on specific package versions to build.
108+
109+__NOTE__ We currently hard-code in `build-constraints.yaml` which
110+version of the `Win32` package we have. It would be nice if we had
111+some kind of a lookup table that correlated GHC versions to `Win32`
112+package versions, so we never accidentally forget to update this
113+version when changing GHC versions.
114+
115+### Check the build plan
116+
117+Confirm that the version bounds in the plan are compatible with each
118+other, and generate the error messages we all know and love/hate
119+:). We could theoretically stop doing this, see
120+[ignore version bounds](https://www.stackage.org/blog/2018/01/ignore-version-bounds)
121+post. We could add explicit "ignore version bounds" metadata (I think
122+Gershom brought up this option).
123+
124+One idea for improvement: currently, `stackage-curator` only checks
125+plans for compatibility with Linux 64-bit. While we can't easily test
126+the builds themselves on Windows and OS X, we _can_ improve this step
127+to check that the plans work for Windows and OS X as well.
128+
129+### Perform the build
130+
131+Fairly straightforward to describe, harder to implement:
132+
133+* Build all packages
134+* Build all non-skipped tests and benchmarks
135+* Build all Haddocks
136+* Fail if a test suite, benchmark or Haddock fails, unless it was an expected failure
137+* Create the "bundle", a tar.xz file with all of the generated Haddocks and some other metadata
138+ * __NOTE__ This bundle isn't strictly necessary anymore, since
139+ stackage-server gets all of the contents from S3 directly. See
140+ discussion at https://github.com/fpco/stackage/issues/2788.
141+
142+### Upload Haddocks
143+
144+Upload all of the Haddock files to S3. This is relatively
145+straightforward, assuming we figure out a good way to build the
146+Haddocks (see Gotchas). There's a good argument for S3 being a
147+terrible mechanism for this, however, and instead using some kind of
148+hash-based download to save a lot of space and make the uploads
149+faster. That kind of change is probably out of scope for now. Any
150+changes made here would need to be mirrored in stackage-server.
151+
152+### Upload plan files
153+
154+The plan files need to be added to the relevant Github repos. We use
155+signed commits to prove that it was the stackage-curator tool that
156+created the commits.
157+
158+## Build scripts
159+
160+It's tempting to put all of the logic above into a single execution of
161+the `stackage-curator` tool. However, that's not how it's currently
162+implemented. Instead, each of the actions above (more or less) is its
163+own command. This allows a set of bash scripts to call the commands
164+individually. This is nice for two reasons:
165+
166+1. It lets us skip some parts of the run easily, such as the
167+ `NOPLAN=1` builds that do not recreate a build plan.
168+2. We're able to play with read/write permissions and access to secret
169+ files depending on the stage we're in. For example, we need a GPG
170+ key when uploading the plan files, but do _not_ want to have access
171+ to that key when performing the build, since a nefarious `Setup.hs`
172+ file or Template Haskell code could leak our secret information. We
173+ leverage Docker for this.
174+
175+## Gotchas
176+
177+* Stack doesn't know anything right now about:
178+ * expected test/benchmark failures
179+ * skipped tests/benchmarks
180+* Haddock generation from Stack probably generates links in a way that
181+ won't be amenable for the S3 upload. There's also likely an inherent
182+ mismatch with the way this will play with caching. I'm not sure what
183+ the right solution is for this. I've spent _many_ brain cycles
184+ thinking through this one, and it's prevented me from making
185+ progress. I recommend we start getting other bits of this working
186+ and then circle back to this problem.
187+* We need to think about backwards compatibility with older tools
188+ (especially older Stacks) pointing to fpco/lts-haskell and
189+ fpco/stackage-nightly. See the
190+ [PLAN2.md doc](https://github.com/fpco/stackage-curator/blob/use-stack/PLAN2.md)
191+ for some ideas around this.