@@ -4,7 +4,7 @@ Originally this was handled largely by Michael Snoyman,
4but now we are a team of 4 people handling requests weekly in rotation.
4but now we are a team of 4 people handling requests weekly in rotation.
5Curation activities are mostly automated, and do not take up a significant amount of time.
5Curation activities are mostly automated, and do not take up a significant amount of time.
7−## Workflow
7+## Workflow overview
9This section sketches out at a high level how the entire Stackage build/curation
9This section sketches out at a high level how the entire Stackage build/curation
10process works:
10process works:
@@ -23,10 +23,30 @@ process works:
24The typical story on pull requests is: If Travis accepts it and the
24The typical story on pull requests is: If Travis accepts it and the
25author only added packages under his/her own name, merge it. If the
25author only added packages under his/her own name, merge it. If the
26+build later fails (see "Adding Debian packages for required system tools or libraries"),
26−build later fails (see below), then block the package until it's
27+then block the package until it's fixed.
29+If benchmarks, haddocks, or test suites fails at this point we
30+typically also block the package until these issues are fixed. This in
31+order to add packages with a clean slate.
29Optionally we can check if packdeps says the package is up to date.
33Optionally we can check if packdeps says the package is up to date.
34+Visit http://packdeps.haskellers.com/feed?needle=<package-name>
36+Builds may fail because of unrelated bounds changes. If this happens,
37+first add any version bounds to get master into a passing state (see
38+"Fixing bounds issues"), then re-run the travis build.
40+A common issue is that authors submit newly uploaded packages, it can
41+take up to an hour before this has synced across the stack
42+infrastructure. You can usually compare the versions of the package in
43+https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/
44+to what's on hackage to see if this is the case. Wait an hour and
45+re-run the pull request.
47+Tests also commonly fail due to missing test files, and sometimes due
48+to doctest limitations. You can point the maintainer to
49+https://github.com/bergmark/blog/blob/master/2016/package-faq.md
31## Fixing bounds issues
51## Fixing bounds issues
@@ -36,20 +56,121 @@ issue on the Stackage repo about the problem, and modifying the
36build-constraints.yaml file to work around it in one of the ways below. Be sure
56build-constraints.yaml file to work around it in one of the ways below. Be sure
37to refer to the issue for workarounds added to that file.
57to refer to the issue for workarounds added to that file.
59+### Temporary upper bounds
39−* __Temporary upper bounds__ Most common technique, just prevent a new version of a library from being included immediately
61+Most common technique, just prevent a new version of a library from
40−* __Skipping tests and benchmarks__ If the upper bound is only in a test suite or benchmark, you can add the relevant package to skipped-tests or skipped-benchmarks. For example, if conduit had an upper bound on criterion for a benchmark, you could added conduit as a skipped benchmark.
62+being included immediately. This also applies to when only benchmarks
63+and tests are affected.
65+* Copy the stackage-curator output and create a new issue, see e.g
66+https://github.com/fpco/stackage/issues/2108
68+* Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
71+ "Stackage upper bounds":
72+ # https://github.com/fpco/stackage/issues/2108
76+* Commit (message e.g. "Upper bound for #2108")
77+* Optionally: Verify with `stackage-curator check` locally
79+* Verify that everything works on the build server (you can restart the build or wait for it to to run again)
81+Sometimes releases for different packages are tightly coupled. Then it
82+can make sense to combine them into one issue, as in
83+https://github.com/fpco/stackage/issues/2143.
85+If a dependency that is not explicitly in stackage is causing test or
86+benchmark failures you can skip or expect them to fail (see "Skipping
87+tests and benchmarks" and "Expecting test/benchmark/haddock
88+failures"). Bonus points for reporting this upstream to that packages'
91+### Lifting upper bounds
93+You can try this when you notice that a package has been updated. You
94+can also periodically try to lift bounds (I think it's good to do this
95+at the start of your week /@bergmark)
97+If stackage-curator is happy commit the change ("Remove upper bounds and close #X").
99+### Amending upper bounds
101+With the `pipes` example above there was later a new release of
102+`pipes-safe` that required the **newer** version of `pipes`. You can
103+add that package to the same upper bounds section,
104+(e.g. https://github.com/fpco/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
105+as we want to lift them simultaneously.
107+### Skipping tests and benchmarks
109+Sometimes tests and benchmark dependencies are forgotten or not cared
110+for. To disable compilation for them add them to `skipped-tests` or
111+`skipped-benchmarks`. If a package is added to these sections they
112+won't be compiled, and their dependencies won't be taken into account.
114+There are sub sections under these headers that is used to group types
115+of failures together, and also to document what type of failures
118+### Expecting test/benchmark/haddock failures
120+The difference from the `skipped` sections is that items listed here
121+are compiled and their dependencies are taken into account. These
122+sections also have sub sections with groups and descriptions.
124+One big category of test suites in this section are those requiring
125+running services. We don't want to run those, but we do want to check
126+dependencies and compile them.
128+If there are no version bounds that would fix the issue or if you
129+can't figure it out, file it
130+(e.g. https://github.com/fpco/stackage/issues/2133) to ask the
133+### Waiting for new releases
135+Sometimes there is a failure reported on a (now possibly closed) issue
136+on an external tracker. If an issue gets resolved but there is no
137+hackage release yet we'd like to get notified when it's uploaded.
139+Add the package with its current version to the
140+`tell-me-when-its-released` section. This will cause the build to stop
141+when the new version is out.
143+### Excluding packages
41−* __Excluding packages__ In an extreme case of a non-responsive maintainer, you can remove the package entirely from Stackage. We try to avoid that whenever possible
145+In an extreme case of a non-responsive maintainer, you can remove the
146+package entirely from Stackage. We try to avoid that whenever
149+This typically happens when we move to a new major GHC release or when
150+there are only a few packages waiting for updates on an upper bounds
153+Comment out the offending packages from the "packages" section and add
154+a comment saying why it was disabled:
157+ # - swagger # BLOCKED aeson 1.0
43## Updating the content of the Docker image used for building
161## Updating the content of the Docker image used for building
45### Adding Debian packages for required system tools or libraries
163### Adding Debian packages for required system tools or libraries
46Additional (non-Haskell) system libraries or tools should be added to `stackage/debian-bootstrap.sh`.
164Additional (non-Haskell) system libraries or tools should be added to `stackage/debian-bootstrap.sh`.
47−Committing the changes to a branch should trigger a DockerHub. Normally only the nightly branch needs to be updated
165+Committing the changes to a branch should trigger a DockerHub. Normally only the `nightly` branch needs to be updated
48since new packages are not added to the current lts release.
166since new packages are not added to the current lts release.
50Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to xenial which is the version used to build Nightly).
168Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to xenial which is the version used to build Nightly).
52−Note we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be add to expected-test-failures.
170+Note that we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be added to `expected-test-failures`.
171+It's good to inform the maintainer of any disabled tests (commenting in the PR is sufficient).
173+If a new package fails to build because of missing system libraries we often ask the maintainer to help figure out what to install.
54### Upgrading GHC version
175### Upgrading GHC version
55The Dockerfile contains information on which GHC versions should be used. You
176The Dockerfile contains information on which GHC versions should be used. You
@@ -106,7 +227,7 @@ we're just not there yet.
106/opt/stackage-build/stackage/automated/build.sh lts-3.0
227/opt/stackage-build/stackage/automated/build.sh lts-3.0
109−Recommended: run these from inside a `screen` session. If you get version bound
230+Recommended: run these from inside a `tmux` session. If you get version bound
110problems on nightly or LTS major, you need to fix build-constraints.yaml (see
231problems on nightly or LTS major, you need to fix build-constraints.yaml (see
111info above). For an LTS minor bump, you'll typically want to use the
232info above). For an LTS minor bump, you'll typically want to use the
112`CONSTRAINTS` environment variable, e.g.:
233`CONSTRAINTS` environment variable, e.g.:
@@ -126,13 +247,13 @@ If a build fails for bounds reasons, see all of the advice above. If the code
126itself doesn't build, or tests fail, open up an issue and then either put in a
247itself doesn't build, or tests fail, open up an issue and then either put in a
127version bound to avoid that version or something else. It's difficult to give
248version bound to avoid that version or something else. It's difficult to give
128universal advice on how to solve things, since each situation is unique. Let's
249universal advice on how to solve things, since each situation is unique. Let's
129−develop this advice over time. For now: if you're not sure, ask Michael for
250+develop this advice over time. For now: if you're not sure, ask for guidance.
132__`NOPLAN=1`__ If you wish to rerun a build without recalculating a
252__`NOPLAN=1`__ If you wish to rerun a build without recalculating a
133build plan, you can set the environment variable `NOPLAN=1`. This is
253build plan, you can set the environment variable `NOPLAN=1`. This is
134useful for such cases as an intermittent test failure, out of memory
254useful for such cases as an intermittent test failure, out of memory
135−condition, or manually tweaking the plan file.
255+condition, or manually tweaking the plan file. This is the default for
137### Timing
258### Timing
@@ -146,9 +267,33 @@ LTS minor bumps typically are run on Sundays.
146### Website sync debugging (and other out of disk space errors)
267### Website sync debugging (and other out of disk space errors)
148* You can detect the problem by running `df`. If you see that `/` is out of space, we have a problem
269* You can detect the problem by running `df`. If you see that `/` is out of space, we have a problem
149−* There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
270+* (outdated) There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
150−* You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
271+* (outdated) You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
152### Wiping the cache
273### Wiping the cache
154−Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`. You can wipe the nightly cache and rebuild everything by doing `rm -rf /opt/stackage-build/stackage/automated/nightly`.
275+Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`.
276+You can wipe the nightly cache and rebuild everything by doing
277+`rm -rf /var/stackage/stackage/automated/nightly`.
278+Replace nightly with `lts7` to wipe the LTS 7 cache.
280+## Local curator setup
282+We don't run the full stackage build locally as that might take too
283+much time. Some steps on the other hand are much faster to do
286+It's useful to be able to modify constraints locally before pushing to
287+the repository. To do this first install stackage-curator:
288+`git clone git@github.com:fpco/stackage-curator.git && cd stackage-curator && stack install`
289+or get the linux binary: https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
290+(it's a good idea to upgrade stackage-curator at least at the start of your week as curator).
291+Then clone the stackage repo `git clone git@github.com:fpco/stackage.git`.
292+Inside it run `stack update && stackage-curator check` to get new packages and do dependency resolution.
294+This can be used to make sure all version bounds are in place
295+(including for test suites and benchmarks), to check whether bounds
296+can be lifted, and to get `tell-me-when-its-released` notifications.
298+Notably this doesn't build anything, so you won't see any compilation
299+errors for builds/tests/benchmarks.