git.iko.soy/stackage
on c46738e help sign in
stackage/CURATORS.md
154 lines 8.2 kB history raw
1This is a collection of instructions covering the processes that the Stackage curators - the
2guys who maintain the Stackage project itself - should be doing on a regular basis.
3Originally this was handled largely by Michael Snoyman,
4but now we are a team of 4 people handling requests weekly in rotation.
5Curation activities are mostly automated, and do not take up a significant amount of time.
7## Workflow
9This section sketches out at a high level how the entire Stackage build/curation
10process works:
12* [build-constraints.yaml](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) specifies packages to be included in Stackage
13* [stackage-curator](http://www.stackage.org/package/stackage-curator) combines build-constraints.yaml with the current state of Hackage to create a build plan for a Stackage Nightly
14* stackage-curator can check that build plan to ensure all version bounds are consistent
15 * The [Travis job](https://github.com/fpco/stackage/blob/master/.travis.yml) performs these two steps to provide immediate feedback on pull requests
16* Docker Hub [builds](https://github.com/fpco/stackage/blob/master/Dockerfile) a [Docker image](https://registry.hub.docker.com/u/snoyberg/stackage/) for running builds
17* The stackage-build server (described below) is able to run automated builds using the [build.sh script](https://github.com/fpco/stackage/blob/master/automated/build.sh)
18* When a new Nightly build is completed, it is uploaded to [the nightly repo](https://github.com/fpco/stackage-nightly)
19* Once a week, we run an LTS minor bump. Instead of using build-constraints.yaml, that job takes the previous LTS release, turns it into constraints, and then bumps the version numbers to the latest on Hackage, in accordance with the version bounds in the build plan. This plans are uploaded to [the LTS repo](https://github.com/fpco/lts-haskell)
20* Cutting a new LTS major release is essentially just a Stackage Nightly that gets rebuilt and uploaded as an LTS
22## Pull requests
24The typical story on pull requests is: If Travis accepts it and the
25author only added packages under his/her own name, merge it. If the
26build later fails (see below), then block the package until it's
27fixed.
29Optionally we can check if packdeps says the package is up to date.
31## Fixing bounds issues
33The most common activity you'll deal with in Stackage curation is a version
34bound issue, usually a restrictive upper bound. You fix this by opening an
35issue on the Stackage repo about the problem, and modifying the
36build-constraints.yaml file to work around it in one of the ways below. Be sure
37to refer to the issue for workarounds added to that file.
39* __Temporary upper bounds__ Most common technique, just prevent a new version of a library from being included immediately
40* __Skipping tests and benchmarks__ If the upper bound is only in a test suite or benchmark, you can add the relevant package to skipped-tests or skipped-benchmarks. For example, if conduit had an upper bound on criterion for a benchmark, you could added conduit as a skipped benchmark.
41* __Excluding packages__ In an extreme case of a non-responsive maintainer, you can remove the package entirely from Stackage. We try to avoid that whenever possible
43## Updating the content of the Docker image used for building
45### Adding Debian packages for required system tools or libraries
46Additional (non-Haskell) system libraries or tools should be added to `stackage/debian-bootstrap.sh`.
47Committing the changes to a branch should trigger a DockerHub. Normally only the nightly branch needs to be updated
48since new packages are not added to the current lts release.
50Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to xenial which is the version used to build Nightly).
52Note we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be add to expected-test-failures.
54### Upgrading GHC version
55The Dockerfile contains information on which GHC versions should be used. You
56can modify it and push it to Github to trigger a DockerHub build. The nightly
57branch is used for nightlies. For LTSes, we use the ltsX branch, where X is the
58major version number (e.g., lts3 for lts-3.\*).
60Note that when starting a new LTS major release, you'll need to modify Docker
61Hub to create a new Docker tag for the relevant branch name.
63### Getting the new image to the build server
64Once a new Docker image is available, you'll need to pull it onto the stackage-build server (see
65below). Instead of pulling an unbounded number of images, I typically just
66delete all of the old images and let the new ones get downloaded:
68```
69docker rm $(docker ps -a -q)
70docker rmi $(docker images -q)
71```
73but `docker pull snoyberg/stackage:nightly` can also be run instead just to update the nightly image say.
75For a new GHC version you should also delete the cache directories on the stackage-build server to
76force all packages to be rebuilt. See: [issue#746](https://github.com/fpco/stackage/issues/746). Eg:
77```
78rm -r nightly/work/builds/nightly/
79```
80This should also be done when moving the Nightly docker image to a new version of Ubuntu.
82## stackage-build server
84You'll need to get your SSH public key added to the machine. ~/.ssh/config info:
86```
87Host stackage-build
88 User curators
89 Hostname ec2-52-5-20-252.compute-1.amazonaws.com
90```
92### Running the build script
94We currently run the builds manually so make it easy to see when there are
95bounds issues that need to be corrected. Automated this would be even better,
96we're just not there yet.
98```
99# Run a nightly build
100/opt/stackage-build/stackage/automated/run-nightly.sh
102# Run an LTS minor bump
103/opt/stackage-build/stackage/automated/build.sh lts-2.17
105# Run an LTS major bump
106/opt/stackage-build/stackage/automated/build.sh lts-3.0
107```
109Recommended: run these from inside a `screen` session. If you get version bound
110problems on nightly or LTS major, you need to fix build-constraints.yaml (see
111info above). For an LTS minor bump, you'll typically want to use the
112`CONSTRAINTS` environment variable, e.g.:
114```
115CONSTRAINTS='--constraint "conduit < 1.4.5" --constraint "criterion < 1.2.3"' /opt/stackage-build/stackage/automated/build.sh lts-2.17
116```
118Valid arguments to include in this environment variable:
120* `--constraint` to modify an upper or lower bound
121* `--add-package` to add a brand new package
122* `--expect-test-failure` to expect tests to fail
123* `--expect-haddock-failure` to expect haddocks to fail
125If a build fails for bounds reasons, see all of the advice above. If the code
126itself doesn't build, or tests fail, open up an issue and then either put in a
127version bound to avoid that version or something else. It's difficult to give
128universal advice on how to solve things, since each situation is unique. Let's
129develop this advice over time. For now: if you're not sure, ask Michael for
130guidance.
132__`NOPLAN=1`__ If you wish to rerun a build without recalculating a
133build plan, you can set the environment variable `NOPLAN=1`. This is
134useful for such cases as an intermittent test failure, out of memory
135condition, or manually tweaking the plan file.
137### Timing
139A looping script on the build server keeps trying to build nightly
140with `sleep 30m` interleaved. It only publishes the nightly once per
141day. This way new package versions or build failures can be caught
142early and hopefully the nightlies will be timely.
144LTS minor bumps typically are run on Sundays.
146### Website sync debugging (and other out of disk space errors)
148* You can detect the problem by running `df`. If you see that `/` is out of space, we have a problem
149* There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
150* You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
152### Wiping the cache
154Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`. You can wipe the nightly cache and rebuild everything by doing `rm -rf /opt/stackage-build/stackage/automated/nightly`.