1This is a collection of instructions covering the processes that the Stackage curators - the
2guys who maintain the Stackage project itself - should be doing on a regular basis.
3Originally this was handled largely by Michael Snoyman,
4but now we are a team of 4 people handling requests weekly in rotation.
5Curation activities are mostly automated, and do not take up a significant amount of time.
9This section sketches out at a high level how the entire Stackage build/curation
12* [build-constraints.yaml](https://github.com/fpco/stackage/blob/master/build-constraints.yaml) specifies packages to be included in Stackage
13* [stackage-curator](http://www.stackage.org/package/stackage-curator) combines build-constraints.yaml with the current state of Hackage to create a build plan for a Stackage Nightly
14* stackage-curator can check that build plan to ensure all version bounds are consistent
15 * The [Travis job](https://github.com/fpco/stackage/blob/master/.travis.yml) performs these two steps to provide immediate feedback on pull requests
16* Docker Hub [builds](https://github.com/fpco/stackage/blob/master/Dockerfile) a [Docker image](https://registry.hub.docker.com/u/snoyberg/stackage/) for running builds
17* The stackage-build server (described below) is able to run automated builds using the [build.sh script](https://github.com/fpco/stackage/blob/master/automated/build.sh)
18* When a new Nightly build is completed, it is uploaded to [the nightly repo](https://github.com/fpco/stackage-nightly)
19* Once a week, we run an LTS minor bump. Instead of using build-constraints.yaml, that job takes the previous LTS release, turns it into constraints, and then bumps the version numbers to the latest on Hackage, in accordance with the version bounds in the build plan. This plans are uploaded to [the LTS repo](https://github.com/fpco/lts-haskell)
20* Cutting a new LTS major release is essentially just a Stackage Nightly that gets rebuilt and uploaded as an LTS
24The typical story on pull requests is: If Travis accepts it and the
25author only added packages under his/her own name, merge it. If the
26build later fails (see "Adding Debian packages for required system tools or libraries"),
27then block the package until it's fixed.
29If benchmarks, haddocks, or test suites fails at this point we
30typically also block the package until these issues are fixed. This in
31order to add packages with a clean slate.
33Optionally we can check if packdeps says the package is up to date.
34Visit http://packdeps.haskellers.com/feed?needle=<package-name>
36Builds may fail because of unrelated bounds changes. If this happens,
37first add any version bounds to get master into a passing state (see
38"Fixing bounds issues"), then re-run the travis build.
40A common issue is that authors submit newly uploaded packages, it can
41take up to an hour before this has synced across the stack
42infrastructure. You can usually compare the versions of the package in
43https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/
44to what's on hackage to see if this is the case. Wait an hour and
45re-run the pull request.
47Tests also commonly fail due to missing test files, and sometimes due
48to doctest limitations. You can point the maintainer to
49https://github.com/bergmark/blog/blob/master/2016/package-faq.md
51## Fixing bounds issues
53The most common activity you'll deal with in Stackage curation is a version
54bound issue, usually a restrictive upper bound. You fix this by opening an
55issue on the Stackage repo about the problem, and modifying the
56build-constraints.yaml file to work around it in one of the ways below. Be sure
57to refer to the issue for workarounds added to that file.
59### Temporary upper bounds
61Most common technique, just prevent a new version of a library from
62being included immediately. This also applies to when only benchmarks
63and tests are affected.
65* Copy the stackage-curator output and create a new issue, see e.g
66https://github.com/fpco/stackage/issues/2108
68* Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
71 "Stackage upper bounds":
72 # https://github.com/fpco/stackage/issues/2108
76* Commit (message e.g. "Upper bound for #2108")
77* Optionally: Verify with `stackage-curator check` locally
79* Verify that everything works on the build server (you can restart the build or wait for it to to run again)
81Sometimes releases for different packages are tightly coupled. Then it
82can make sense to combine them into one issue, as in
83https://github.com/fpco/stackage/issues/2143.
85If a dependency that is not explicitly in stackage is causing test or
86benchmark failures you can skip or expect them to fail (see "Skipping
87tests and benchmarks" and "Expecting test/benchmark/haddock
88failures"). Bonus points for reporting this upstream to that packages'
91### Lifting upper bounds
93You can try this when you notice that a package has been updated. You
94can also periodically try to lift bounds (I think it's good to do this
95at the start of your week /@bergmark)
97If stackage-curator is happy commit the change ("Remove upper bounds and close #X").
99### Amending upper bounds
101With the `pipes` example above there was later a new release of
102`pipes-safe` that required the **newer** version of `pipes`. You can
103add that package to the same upper bounds section,
104(e.g. https://github.com/fpco/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
105as we want to lift them simultaneously.
107### Skipping tests and benchmarks
109Sometimes tests and benchmark dependencies are forgotten or not cared
110for. To disable compilation for them add them to `skipped-tests` or
111`skipped-benchmarks`. If a package is added to these sections they
112won't be compiled, and their dependencies won't be taken into account.
114There are sub sections under these headers that is used to group types
115of failures together, and also to document what type of failures
118### Expecting test/benchmark/haddock failures
120The difference from the `skipped` sections is that items listed here
121are compiled and their dependencies are taken into account. These
122sections also have sub sections with groups and descriptions.
124One big category of test suites in this section are those requiring
125running services. We don't want to run those, but we do want to check
126dependencies and compile them.
128If there are no version bounds that would fix the issue or if you
129can't figure it out, file it
130(e.g. https://github.com/fpco/stackage/issues/2133) to ask the
133### Waiting for new releases
135Sometimes there is a failure reported on a (now possibly closed) issue
136on an external tracker. If an issue gets resolved but there is no
137hackage release yet we'd like to get notified when it's uploaded.
139Add the package with its current version to the
140`tell-me-when-its-released` section. This will cause the build to stop
141when the new version is out.
143### Excluding packages
145In an extreme case of a non-responsive maintainer, you can remove the
146package entirely from Stackage. We try to avoid that whenever
149This typically happens when we move to a new major GHC release or when
150there are only a few packages waiting for updates on an upper bounds
153Comment out the offending packages from the "packages" section and add
154a comment saying why it was disabled:
157 # - swagger # BLOCKED aeson 1.0
161## Updating the content of the Docker image used for building
163### Adding Debian packages for required system tools or libraries
164Additional (non-Haskell) system libraries or tools should be added to `stackage/debian-bootstrap.sh`.
165Committing the changes to a branch should trigger a DockerHub. Normally only the `nightly` branch needs to be updated
166since new packages are not added to the current lts release.
168Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to xenial which is the version used to build Nightly).
170Note that we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be added to `expected-test-failures`.
171It's good to inform the maintainer of any disabled tests (commenting in the PR is sufficient).
173If a new package fails to build because of missing system libraries we often ask the maintainer to help figure out what to install.
175### Upgrading GHC version
176The Dockerfile contains information on which GHC versions should be used. You
177can modify it and push it to Github to trigger a DockerHub build. The nightly
178branch is used for nightlies. For LTSes, we use the ltsX branch, where X is the
179major version number (e.g., lts3 for lts-3.\*).
181Note that when starting a new LTS major release, you'll need to modify Docker
182Hub to create a new Docker tag for the relevant branch name.
184### Getting the new image to the build server
185Once a new Docker image is available, you'll need to pull it onto the stackage-build server (see
186below). Instead of pulling an unbounded number of images, I typically just
187delete all of the old images and let the new ones get downloaded:
190docker rm $(docker ps -a -q)
191docker rmi $(docker images -q)
194but `docker pull snoyberg/stackage:nightly` can also be run instead just to update the nightly image say.
196For a new GHC version you should also delete the cache directories on the stackage-build server to
197force all packages to be rebuilt. See: [issue#746](https://github.com/fpco/stackage/issues/746). Eg:
199rm -r nightly/work/builds/nightly/
201This should also be done when moving the Nightly docker image to a new version of Ubuntu.
203## stackage-build server
205You'll need to get your SSH public key added to the machine. ~/.ssh/config info:
210 Hostname build.stackage.org
213### Running the build script
215We currently run the builds manually so make it easy to see when there are
216bounds issues that need to be corrected. Automated this would be even better,
217we're just not there yet.
221/opt/stackage-build/stackage/automated/run-nightly.sh
223# Run an LTS minor bump
224/opt/stackage-build/stackage/automated/build.sh lts-2.17
226# Run an LTS major bump
227/opt/stackage-build/stackage/automated/build.sh lts-3.0
230Recommended: run these from inside a `tmux` session. If you get version bound
231problems on nightly or LTS major, you need to fix build-constraints.yaml (see
232info above). For an LTS minor bump, you'll typically want to use the
233`CONSTRAINTS` environment variable, e.g.:
236CONSTRAINTS='--constraint "conduit < 1.4.5" --constraint "criterion < 1.2.3"' /opt/stackage-build/stackage/automated/build.sh lts-2.17
239Valid arguments to include in this environment variable:
241* `--constraint` to modify an upper or lower bound
242* `--add-package` to add a brand new package
243* `--expect-test-failure` to expect tests to fail
244* `--expect-haddock-failure` to expect haddocks to fail
246If a build fails for bounds reasons, see all of the advice above. If the code
247itself doesn't build, or tests fail, open up an issue and then either put in a
248version bound to avoid that version or something else. It's difficult to give
249universal advice on how to solve things, since each situation is unique. Let's
250develop this advice over time. For now: if you're not sure, ask for guidance.
252__`NOPLAN=1`__ If you wish to rerun a build without recalculating a
253build plan, you can set the environment variable `NOPLAN=1`. This is
254useful for such cases as an intermittent test failure, out of memory
255condition, or manually tweaking the plan file. This is the default for
260A looping script on the build server keeps trying to build nightly
261with `sleep 30m` interleaved. It only publishes the nightly once per
262day. This way new package versions or build failures can be caught
263early and hopefully the nightlies will be timely.
265LTS minor bumps typically are run on Sundays.
267### Website sync debugging (and other out of disk space errors)
269* You can detect the problem by running `df`. If you see that `/` is out of space, we have a problem
270* (outdated) There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
271* (outdated) You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
275Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`.
276You can wipe the nightly cache and rebuild everything by doing
277`rm -rf /var/stackage/stackage/automated/nightly`.
278Replace nightly with `lts7` to wipe the LTS 7 cache.
280## Local curator setup
282We do not run the full stackage build locally as that might take too
283much time. Some steps on the other hand are much faster to do
286It is useful to be able to modify constraints locally before pushing to
287the repository. To do this first install stackage-curator:
288`git clone git@github.com:fpco/stackage-curator.git && cd stackage-curator && stack install`
289or get the linux binary: https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
290(it is a good idea to upgrade stackage-curator at least at the start of your week as curator).
291Then clone the stackage repo `git clone git@github.com:fpco/stackage.git`.
292Inside it run `stack update && stackage-curator check` to get new packages and do dependency resolution.
294This can be used to make sure all version bounds are in place
295(including for test suites and benchmarks), to check whether bounds
296can be lifted, and to get `tell-me-when-its-released` notifications.
298Notably this does not build anything, so you wont see any compilation
299errors for builds/tests/benchmarks.
301## Adding new curators
3031. Add public ssh key to `~/.ssh/authorized_keys` on build server
3042. Add to fpco/stackage project.
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list