1This is a collection of instructions covering the processes that the Stackage curators - the
2team who maintain the Stackage project itself - should be doing on a regular basis.
3Curation activities are mostly automated, and do not take up a significant amount of time.
4The following is the current list of curators, in alphabetical order:
6* Adam Bergmark (@bergmark)
7* Alexey Zabelin (@alexeyzab)
8* Chris Dornan (@cdornan)
9* Dan Burton (@danburton)
10* Jens Petersen (@juhp)
11* Joe Kachmar (@jkachmar)
12* Michael Snoyman (@snoyberg)
13* Mihai Maruseac (@mihaimaruseac)
17This section sketches out at a high level how the entire Stackage build/curation
20* [build-constraints.yaml](https://github.com/commercialhaskell/stackage/blob/master/build-constraints.yaml) specifies packages to be included in Stackage
21* [curator](https://github.com/commercialhaskell/curator) combines build-constraints.yaml with the current state of Hackage to create a build plan for a Stackage Nightly
22* `curator` can check that build plan to ensure all version bounds are consistent
23 * The [Travis job](https://github.com/commercialhaskell/stackage/blob/master/.travis.yml) performs these two steps to provide immediate feedback on pull requests
24* Docker Hub [builds](https://github.com/commercialhaskell/stackage/blob/master/Dockerfile) a [Docker image](https://hub.docker.com/r/commercialhaskell/stackage/) for running builds
25* The stackage-build server (described below) is able to run automated builds using the [build.sh script](https://github.com/commercialhaskell/stackage/blob/master/automated/build.sh)
26* When a new Nightly build is completed, it is uploaded to [the nightly repo](https://github.com/commercialhaskell/stackage-nightly)
27* Once a week, we run an LTS minor bump. Instead of using build-constraints.yaml, that job takes the previous LTS release, turns it into constraints, and then bumps the version numbers to the latest on Hackage, in accordance with the version bounds in the build plan. This plans are uploaded to [the LTS repo](https://github.com/commercialhaskell/lts-haskell)
28* Cutting a new LTS major release is essentially just a Stackage Nightly that gets rebuilt and uploaded as an LTS
32The typical story on pull requests is: If Travis accepts it and the
33author only added packages under his/her own name, merge it. If the
34build later fails (see [Adding Debian packages]), then block the
35package until it's fixed.
37[Adding Debian packages]: https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md#adding-debian-packages-for-required-system-tools-or-libraries
39If benchmarks, haddocks, or test suites fails at this point we
40typically also block the package until these issues are fixed. This in
41order to add packages with a clean slate.
43Optionally we can check if packdeps says the package is up to date.
44Visit http://packdeps.haskellers.com/feed?needle=<package-name>
46Builds may fail because of unrelated bounds changes. If this happens,
47first add any version bounds to get master into a passing state (see
48"Fixing bounds issues"), then re-run the travis build.
50A common issue is that authors submit newly uploaded packages, it can
51take up to an hour before this has synced across the stack
52infrastructure. You can usually compare the versions of the package in
53https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/
54to what's on hackage to see if this is the case. Wait an hour and
55re-run the pull request.
57Tests also commonly fail due to missing test files, and sometimes due
58to doctest limitations. You can point the maintainer to
59https://github.com/bergmark/blog/blob/master/2016/package-faq.md
61## Fixing bounds issues
63The most common activity you'll deal with in Stackage curation is a version
64bound issue, usually a restrictive upper bound. You fix this by opening an
65issue on the Stackage repo about the problem, and modifying the
66build-constraints.yaml file to work around it in one of the ways below. Be sure
67to refer to the issue for workarounds added to that file.
69### Temporary upper bounds
71Most common technique, just prevent a new version of a library from
72being included immediately. This also applies to when only benchmarks
73and tests are affected.
75* Copy the `curator` output and create a new issue, see e.g
76https://github.com/commercialhaskell/stackage/issues/2108
78* Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
81 "Stackage upper bounds":
82 # https://github.com/commercialhaskell/stackage/issues/2108
86* Commit (message e.g. "Upper bound for #2108")
87* Optionally: Verify with `./check` locally
89* Verify that everything works on the build server (you can restart the build or wait for it to to run again)
91Sometimes releases for different packages are tightly coupled. Then it
92can make sense to combine them into one issue, as in
93https://github.com/commercialhaskell/stackage/issues/2143.
95If a dependency that is not explicitly in stackage is causing test or
96benchmark failures you can skip or expect them to fail (see "Skipping
97tests and benchmarks" and "Expecting test/benchmark/haddock
98failures"). Bonus points for reporting this upstream to that packages'
101### Lifting upper bounds
103You can try this when you notice that a package has been updated. You
104can also periodically try to lift bounds (I think it's good to do this
105at the start of your week /@bergmark)
107If not all packages have been updated check if any of them are missing
108from the original issue and if so add a new comment mentioning them. A
109new package may appear if its dependencies were part of this issue but
110have been updated since the last time we checked. We want to give
111these new packages ample time to be upgraded.
113If `curator` is happy commit the change ("Remove upper bounds
114and close #X"). After doing this the next nightly build may fail
115because some packages didn't have an upper bound in place, but
116compilation failed. In this case revert the previous commit so any
117disabled packages are enabled again, re-open the issue, and add a new
118comment with the failing packages. This is to give all maintainers
119enough time to upgrade for this case as well.
121### Amending upper bounds
123With the `pipes` example above there was later a new release of
124`pipes-safe` that required the **newer** version of `pipes`. You can
125add that package to the same upper bounds section,
126(e.g. https://github.com/commercialhaskell/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
127as we want to lift them simultaneously.
129### Skipping tests and benchmarks
131Sometimes tests and benchmark dependencies are forgotten or not cared
132for. To disable compilation for them add them to `skipped-tests` or
133`skipped-benchmarks`. If a package is added to these sections they
134won't be compiled, and their dependencies won't be taken into account.
136There are sub sections under these headers that is used to group types
137of failures together, and also to document what type of failures
140### Expecting test/benchmark/haddock failures
142The difference from the `skipped` sections is that items listed here
143are compiled and their dependencies are taken into account
144(but they are allowed to fail to build).
145These sections also have subsections with groups and descriptions.
147One big category of test suites in this section are those requiring
148running services. We don't want to run those, but we do want to check
149dependencies and compile them.
151If there are no version bounds that would fix the issue or if you
152can't figure it out, file it
153(e.g. https://github.com/commercialhaskell/stackage/issues/2133) to ask the
156### Waiting for new releases
158Sometimes there is a failure reported on a (now possibly closed) issue
159on an external tracker. If an issue gets resolved but there is no
160hackage release yet we'd like to get notified when it's uploaded.
162Add the package with its current version to the
163`tell-me-when-its-released` section. This will cause the build to stop
164when the new version is out.
166### Excluding packages
168In an extreme case of a non-responsive maintainer, you can remove the
169package entirely from Stackage. We try to avoid that whenever
172This typically happens when we move to a new major GHC release or when
173there are only a few packages waiting for updates on an upper bounds
176If a package needs to be disabled due to build failures: Add a `< 0`
177bound to the package to exclude it, and add a comment stating why it
178was disabled: `- swagger < 0 # compile failure againts aeson 1.0`
180If a package needs to be disabled due to bounds issues, see the "Large
181scale enabling/disabling of packages" section below.
183If this causes reverse dependencies to be disabled we should notify
184the maintainers of those packages.
187## Updating the content of the Docker image used for building
189### Adding Debian packages for required system tools or libraries
190Additional (non-Haskell) system libraries or tools should be added to `stackage/debian-bootstrap.sh`.
191After you've committed those changes, merging them into the `nightly` branch should
192trigger a DockerHub build. Simply run:
195 $ git checkout nightly
200Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to xenial which is the version used to build Nightly).
202Note that we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be added to `expected-test-failures`.
203It's good to inform the maintainer of any disabled tests (commenting in the PR is sufficient).
205If a new package fails to build because of missing system libraries we often ask the maintainer to help figure out what to install.
207### Upgrading GHC version
208The Dockerfile contains information on which GHC versions should be used. You
209can modify it and push it to Github to trigger a build. The nightly
210branch is used for nightlies. For LTSes, we use the ltsX branch, where X is the
211major version number (e.g., lts3 for lts-3.\*).
213Note that when starting a new LTS major release, you'll need to modify `.github/workflows/image.yml` to add a new lts branch.
215Update `GHCVER` in `Dockerfile`. (This env var automatically gets passed to `debian-bootstrap.sh`.)
217Ensure that the [global-hints.yaml
218file](https://github.com/fpco/stackage-content/blob/master/stack/global-hints.yaml)
219is updated with information on the latest GHC release by cloning that
220repo and running `./update-global-hints.hs ghc-X.Y.Z`.
222If enountering an error like the following, this means that the Stack metadata
223has not yet been updated, so wait some time until this happens:
226This probably means a GHC bindist has not yet been added for OS key 'linux64', 'linux64-ncurses6', 'linux64-tinfo6'.
227Supported versions: ...
228update-global-hints.hs: Received ExitFailure 1 when running
231Also required to build an LTS minor bump with a ghc version change: On the build server, modify `/var/stackage/stackage/automated/work/lts-$THIS_LTS_MAJOR_VER/constraints.yaml` and update the ghc-version. (You may need to update sibling files as well.) Then run `NOPLAN=1 /var/stackage/stackage/automated/build.sh lts-$THIS_LTS_MINOR_BUMP` to build the LTS.
233### Getting the new image to the build server
234Once a new Docker image is available, you'll need to pull it onto the stackage-build server (see
235below). Instead of pulling an unbounded number of images, I typically just
236delete all of the old images and let the new ones get downloaded:
239docker rm $(docker ps -a -q)
240docker rmi $(docker images -q)
243but `docker pull ghcr.io/commercialhaskell/stackage/build:nightly` can also be run instead just to update the nightly image say.
245For a new GHC version you should also delete the ~~cache~~ .stack-work snapshot install directories on the stackage-build server to
246~~force all packages to be rebuilt~~ clear up some space. See: [issue#746](https://github.com/commercialhaskell/stackage/issues/746). Eg:
250SNAP_SERIES=nightly # or lts16
252rm -r work/$SNAP_SERIES/unpack-dir/.stack-work/install/x86_64-linux/*/$OLD_GHCVER/
254This should also be done when moving the Nightly docker image to a new version of Ubuntu.
256If you're impatient and would like to build the Docker image on the
257build server instead of waiting for Docker Hub, you can run the
258following command (replacing `BRANCH=nightly` if the image for a different branch is desired):
264 && git clone https://github.com/commercialhaskell/stackage \
266 && git checkout $BRANCH \
267 && docker build --tag commercialhaskell/stackage:$BRANCH .)
271Note that we do a clean clone of the `stackage` repo instead of using
272the existing checkout because of how `docker build` works: it will
273send the entire local directory contents as context to the Docker
274daemon, which in the case of the build tree is a _lot_ of content. (We
275can discuss the wisdom—or lack thereof—of Docker's
278## stackage-build server
280You'll need to get your SSH public key added to the machine. ~/.ssh/config info:
285 Hostname build.stackage.org
288### Running the build script
290We currently run the builds manually so make it easy to see when there are
291bounds issues that need to be corrected. Automated this would be even better,
292we're just not there yet.
296/var/stackage/stackage/automated/run-nightly.sh
298# Run an LTS minor bump
299/var/stackage/stackage/automated/build.sh lts-15.1
301# Run an LTS major bump
302/var/stackage/stackage/automated/build.sh lts-16.0
305Recommended: run these from inside a `tmux` session. If you get version bound
306problems on nightly or LTS major, you need to fix build-constraints.yaml (see
309### Building LTS minor releases
310Before running the build, please make sure that the Dockerfile in `automated/dockerfiles/lts-X.Y` is up to date, where X is the major version that you're building and Y is the latest minor version of X for which a Dockerfile exists.
311 * If any changes need to be made, (eg, new GHC version), copy `automated/lts-X.Y/Dockerfile` to `automated/lts-X.Z/Dockerfile`, where Z is the minor version you're building, and include the new changes.
312 * If you are building the first release of a new LTS major version, create a new `lts-X.Z/Dockerfile` based on the previous LTS's, and adjust the variables at the top to match the requirements of the snapshot. Ensure that `STACK_VERSION` is the latest release of Stack, and `BOOTSTRAP_COMMIT` is the commit ID of this repo containing the version of the `bootstrap-commit.sh` used to build the snapshot. Also ensure the FROM image's Ubuntu version matches that used in the [root Dockerfile](Dockerfile) used to build this snapshot.
314First run `build.sh` to regenerate updated `ltsXX/work/constraints.yaml` and `ltsXX/work/snapshot-incomplete.yaml` files.
316For an LTS minor bump, you'll typically want to:
318* Add constraints to package `range:` fields _under_ the `source:` field in that `constraints.yaml`, and edit `snapshot-incomplete.yaml` to change the version used for that package, if necessary.
319* Add new packages to the `constraints.yaml` file
320* Test, benchmark, haddock failures can also be added to package fields in the `constraints.yaml` if necessary, though it should be avoided if possible for LTS.
322Then run `NOPLAN=1 build.sh` to build the generate an updated snapshot.
324If a build fails for bounds reasons, see all of the advice above. If the code
325itself doesn't build, or tests fail, open up an issue and then either put in a
326version bound to avoid that version or something else. It's difficult to give
327universal advice on how to solve things, since each situation is unique. Let's
328develop this advice over time. For now: if you're not sure, ask for guidance.
330__`NOPLAN=1`__ If you wish to rerun a build without recalculating a
331build plan, you can set the environment variable `NOPLAN=1`. This is
332useful for such cases as an intermittent test failure, out of memory
333condition, or manually tweaking the plan file. (When using `NOPLAN=1`,
334if one needs to revert one package, say due to a build or test regression,
335one can edit `current-plan.yaml` and updated the SHA256 hash of the .cabal file,
336to avoid having to rebuild everything again.)
338Note LTS builds without NOPLAN will use the latest Hackage data.
340If you need to make further modifications beyond what `constraints.yaml` allows, you can directly edit the `snapshot-incomplete.yaml` file. Then, instead of `NOPLAN=1 build.sh`, you need to use `NOPLAN=2 build.sh`. Note that from this point on, further changes to `constraints.yaml` will not impact the build plan.
344A looping script on the build server keeps trying to build nightly
345with `sleep 30m` interleaved. It only publishes the nightly once per
346day. This way new package versions or build failures can be caught
347early and hopefully the nightlies will be timely.
349LTS minor bumps are typically run on weekends. It can be a good idea
350to start the build on friday or saturday to have enough time to
351resolve any issues before the next curator shift the coming monday.
353### Diskspace errors (and website sync debugging)
355* You can detect the problem by running `df`. If you see that `/` is out of space, we have a problem.
356* If you see that `/var/stackage/` is out of space, you can:
357 * run `./etc/diskspace/clean-old-stack-libs.sh [nightly|lts-XX]`
358 * If that is insufficient then remove all the old builds under the previous ghc/Cabal version:
359 * `rm -r /var/stackage/stackage/automated/work/[nightly|lts-XX]/unpack-dir/unpacked/*/.stack-work/dist/x86_64-linux/Cabal-X.Y.0.0/`
361 optionally (not recommended?):
362 * `rm -r /var/stackage/stackage/automated/work/lts*/unpack-dir/unpacked/`
363 * `rm -r /var/stackage/stackage/automated/work/nightly/unpack-dir/unpacked/`
367Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`.
368You can wipe the nightly cache and rebuild everything by doing
369`rm -rf /var/stackage/stackage/automated/nightly`.
370Replace nightly with `lts7` to wipe the LTS 7 cache.
372### Force a single package rebuild
374You can force a single package to rebuild by deleting its "previous result"
378$ rm /var/stackage/stackage/automated/nightly/work/builds/nightly/prevres/Build/cryptohash-0.11.9
383If docker hangs with e.g. `docker: Error response from daemon:
384connection error: desc = "transport: dial unix
385/var/run/docker/containerd/docker-containerd .sock: connect:
386connection refused".` you can restart it with `sudo systemctl restart
389## Local curator setup
391We do not run the full stackage build locally as that might take too
392much time. However, some steps on the other hand are much faster to do
393yourself, e.g. verifying constraints without building anything.
395To get started, install `curator` via Git:
398$ git clone git@github.com:commercialhaskell/curator.git
399$ cd curator && stack install curator
402It is a good idea to upgrade `curator` at the start of your week.
403Then, clone the stackage repo, get the latest packages and run dependency
407$ git clone git@github.com:commercialhaskell/stackage.git
412This can be used to make sure all version bounds are in place, including for
413test suites and benchmarks, to check whether bounds can be lifted, and to get
414[tell-me-when-its-released] notifications.
416`curator` does not build anything, so you wont see any compilation
417errors for builds, tests and benchmarks.
419[tell-me-when-its-released]: https://github.com/commercialhaskell/stackage/blob/master/CURATORS.md#waiting-for-new-releases
421### Large scale enabling/disabling of packages
423`etc/commenter` is a binary that mostly automates the translation of
424`./check` errors into lines that go into `build-constraints.yaml`. It
425can only handle bounds issues, compilation issues still need to be
428It disables all offending packages/test suites/benchmarks, so it is
429only meant to be used when we close bounds issues and want to disable
430packages, and when upgrading GHC.
433This is currently a rust program, You can install the rust toolchain
434by using [rustup](https://rustup.rs/).
436Then `cargo install --locked --path etc/commenter`
440After disabling a few packages you get this curator output:
443ConfigFile (GHC 9 bounds issues, @maintainer) (not present) depended on by:
444- [ ] xdg-desktop-entry-0.1.1.1 (-any). @maintainer. Used by: library
447pipes-misc (GHC 9 bounds issues, @maintainer) (not present) depended on by:
448- [ ] pipes-fluid-0.6.0.1 (>=0.5). @handles. Used by: test-suite
451testing-feat (GHC 9 bounds issues, Grandfathered dependencies) (not present) depended on by:
452- [ ] dual-tree-0.2.3.0 (-any). Grandfathered dependencies. @handles. Used by: test-suite
457./check 2>&1 >/dev/null | commenter add
460You will get this output:
465 - xdg-desktop-entry < 0 # tried xdg-desktop-entry-0.1.1.1, but its *library* requires the disabled package: ConfigFile
469 - dual-tree # tried dual-tree-0.2.3.0, but its *test-suite* requires the disabled package: testing-feat
470 - pipes-fluid # tried pipes-fluid-0.6.0.1, but its *test-suite* requires the disabled package: pipes-misc
472Adding 1 libs, 2 tests, 0 benches to build-constraints.yaml
475These bounds are added to build-constraints.yaml automatically.
477Re-run this command until no more packages are disabled.
481We can periodically remove all packages under the bounds sections and then re-run the disabling flow above until we get a clean plan. This will automatically pick up packages that have been fixed.
485./check 2>&1 >/dev/null | commenter add
488Repeat the second command until no updates are made to build-constraints.yaml.
490#### Checking for new releases
492Run `stack update` before doing this.
494`commenter outdated` looks through all bounds issues and library
495compilation failures and compares the marked version with the latest hackage release. Example output is
498Fin mismatch, manual: 0.2.8.0, hackage: 0.2.9.0
499aeson mismatch, auto: 1.5.6.0, hackage: 2.0.2.0
502where "manual" means the bound was added manually by a curator,
503perhaps due to a compilation failure so we could try re-enabling the
504package. "auto" means it's part of the sections generated by
505`commenter`, to update that run the `Re-enabling` step as documented
508`outdated` only finds packages that are in the auto generated
509sections, or that are of the form `- package < 0 # $version`.
513* Please make sure to separate bounds issues from compilation failures/test run failures, as we cannot verify that a package builds or that tests pass without running the build!
516## Adding new curators
5181. Add public ssh key to `~/.ssh/authorized_keys` on build server
5192. Add to commercialhaskell/stackage project.
521## Dealing with a new GHC release
523As mentioned in the [GHC upgrade note], the major impact of a new GHC release
524is on the packages that are causing upper bounds to be put in place. In order
525to minimise out-of-date breakage and allow maintainers to have a solid chance
526of getting their packages into the newest LTS, we try to do the following:
528Make an early announcement (in the form of a blog post, typically) of the new
529GHC release on the nightly build and the planned deadline for the new LTS release.
530Make it clear, that in the time coming up to this, we hope package maintainers
531will upgrade their packages to allow for the new GHC release.
533We prefer to prune packages causing upper bounds constraints **after** the LTS
534release to allow the maximum amount of packages to get into the newest LTS.
536You will almost always need to update the Win32 package version listed in the
537build-constraints.yaml file.
539After the first LTS release, the package pruning process may begin in the
540nightly build in order to move forward with getting the latest versions of
541packages compatible with the new GHC release.
543[GHC upgrade note]: https://github.com/commercialhaskell/stackage/blob/master/MAINTAINERS.md#upgrading-to-a-new-ghc-version
547Every 3-6 months, we make a new major release of LTS. The procedure we follow for this is:
5491. Write a blog post on stackage.org announcing the intent to cut a major
550 release. Give an estimated date two weeks in the future from the publication
5522. Spread the blog post on social media and mailing lists as much as possible.
5533. Expect maintainers to send significant requests for added packages and
554 relaxed upper bounds. There will likely be some hard decisions to be made
555 regarding relaxing a bound versus keeping more packages. All of these changes
556 occur on master and affect nightly.
5574. Once the estimated date hits, push a new `ltsXX` and wait for the docker image build.
5585. Run the build procedure for the new LTS release.
5596. After the LTS build completes, more aggressively prune upper bounds from
560 `build-constraints.yaml`.
5617. Once both (5) and (6) are done, publish a new blog post on stackage.org
562 announcing the new LTS and Nightly, with links to the change pages on
563 stackage.org. Include a reminder that requests for packages to be added to LTS
564 may be made on commercialhaskell/lts-haskell.
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list