@@ -4,7 +4,7 @@ Originally this was handled largely by Michael Snoyman,
44but now we are a team of 4 people handling requests weekly in rotation.
55Curation activities are mostly automated, and do not take up a significant amount of time.
66
7−## Workflow
7+## Workflow overview
88
99This section sketches out at a high level how the entire Stackage build/curation
1010process works:
@@ -23,10 +23,30 @@ process works:
2323
2424The typical story on pull requests is: If Travis accepts it and the
2525author only added packages under his/her own name, merge it. If the
26−build later fails (see below), then block the package until it's
27−fixed.
26+build later fails (see "Adding Debian packages for required system tools or libraries"),
27+then block the package until it's fixed.
28+
29+If benchmarks, haddocks, or test suites fails at this point we
30+typically also block the package until these issues are fixed. This in
31+order to add packages with a clean slate.
2832
2933Optionally we can check if packdeps says the package is up to date.
34+Visit http://packdeps.haskellers.com/feed?needle=<package-name>
35+
36+Builds may fail because of unrelated bounds changes. If this happens,
37+first add any version bounds to get master into a passing state (see
38+"Fixing bounds issues"), then re-run the travis build.
39+
40+A common issue is that authors submit newly uploaded packages, it can
41+take up to an hour before this has synced across the stack
42+infrastructure. You can usually compare the versions of the package in
43+https://github.com/commercialhaskell/all-cabal-metadata/tree/master/packages/
44+to what's on hackage to see if this is the case. Wait an hour and
45+re-run the pull request.
46+
47+Tests also commonly fail due to missing test files, and sometimes due
48+to doctest limitations. You can point the maintainer to
49+https://github.com/bergmark/blog/blob/master/2016/package-faq.md
3050
3151## Fixing bounds issues
3252
@@ -36,20 +56,121 @@ issue on the Stackage repo about the problem, and modifying the
3656build-constraints.yaml file to work around it in one of the ways below. Be sure
3757to refer to the issue for workarounds added to that file.
3858
39−* __Temporary upper bounds__ Most common technique, just prevent a new version of a library from being included immediately
40−* __Skipping tests and benchmarks__ If the upper bound is only in a test suite or benchmark, you can add the relevant package to skipped-tests or skipped-benchmarks. For example, if conduit had an upper bound on criterion for a benchmark, you could added conduit as a skipped benchmark.
41−* __Excluding packages__ In an extreme case of a non-responsive maintainer, you can remove the package entirely from Stackage. We try to avoid that whenever possible
59+### Temporary upper bounds
60+
61+Most common technique, just prevent a new version of a library from
62+being included immediately. This also applies to when only benchmarks
63+and tests are affected.
64+
65+* Copy the stackage-curator output and create a new issue, see e.g
66+https://github.com/fpco/stackage/issues/2108
67+
68+* Add a new entry under the "stackage upper bounds" section of `build-constraints.yaml`. For the above example it would be
69+
70+```yaml
71+ "Stackage upper bounds":
72+ # https://github.com/fpco/stackage/issues/2108
73+ - pipes < 4.3.0
74+```
75+
76+* Commit (message e.g. "Upper bound for #2108")
77+* Optionally: Verify with `stackage-curator check` locally
78+* Push
79+* Verify that everything works on the build server (you can restart the build or wait for it to to run again)
80+
81+Sometimes releases for different packages are tightly coupled. Then it
82+can make sense to combine them into one issue, as in
83+https://github.com/fpco/stackage/issues/2143.
84+
85+If a dependency that is not explicitly in stackage is causing test or
86+benchmark failures you can skip or expect them to fail (see "Skipping
87+tests and benchmarks" and "Expecting test/benchmark/haddock
88+failures"). Bonus points for reporting this upstream to that packages'
89+maintainer.
90+
91+### Lifting upper bounds
92+
93+You can try this when you notice that a package has been updated. You
94+can also periodically try to lift bounds (I think it's good to do this
95+at the start of your week /@bergmark)
96+
97+If stackage-curator is happy commit the change ("Remove upper bounds and close #X").
98+
99+### Amending upper bounds
100+
101+With the `pipes` example above there was later a new release of
102+`pipes-safe` that required the **newer** version of `pipes`. You can
103+add that package to the same upper bounds section,
104+(e.g. https://github.com/fpco/stackage/commit/6429b1eb14db3f2a0779813ef2927085fa4ad673)
105+as we want to lift them simultaneously.
106+
107+### Skipping tests and benchmarks
108+
109+Sometimes tests and benchmark dependencies are forgotten or not cared
110+for. To disable compilation for them add them to `skipped-tests` or
111+`skipped-benchmarks`. If a package is added to these sections they
112+won't be compiled, and their dependencies won't be taken into account.
113+
114+There are sub sections under these headers that is used to group types
115+of failures together, and also to document what type of failures
116+exist.
117+
118+### Expecting test/benchmark/haddock failures
119+
120+The difference from the `skipped` sections is that items listed here
121+are compiled and their dependencies are taken into account. These
122+sections also have sub sections with groups and descriptions.
123+
124+One big category of test suites in this section are those requiring
125+running services. We don't want to run those, but we do want to check
126+dependencies and compile them.
127+
128+If there are no version bounds that would fix the issue or if you
129+can't figure it out, file it
130+(e.g. https://github.com/fpco/stackage/issues/2133) to ask the
131+maintainer for help.
132+
133+### Waiting for new releases
134+
135+Sometimes there is a failure reported on a (now possibly closed) issue
136+on an external tracker. If an issue gets resolved but there is no
137+hackage release yet we'd like to get notified when it's uploaded.
138+
139+Add the package with its current version to the
140+`tell-me-when-its-released` section. This will cause the build to stop
141+when the new version is out.
142+
143+### Excluding packages
144+
145+In an extreme case of a non-responsive maintainer, you can remove the
146+package entirely from Stackage. We try to avoid that whenever
147+possible.
148+
149+This typically happens when we move to a new major GHC release or when
150+there are only a few packages waiting for updates on an upper bounds
151+issue.
152+
153+Comment out the offending packages from the "packages" section and add
154+a comment saying why it was disabled:
155+
156+```
157+ # - swagger # BLOCKED aeson 1.0
158+```
159+
42160
43161## Updating the content of the Docker image used for building
44162
45163### Adding Debian packages for required system tools or libraries
46164Additional (non-Haskell) system libraries or tools should be added to `stackage/debian-bootstrap.sh`.
47−Committing the changes to a branch should trigger a DockerHub. Normally only the nightly branch needs to be updated
165+Committing the changes to a branch should trigger a DockerHub. Normally only the `nightly` branch needs to be updated
48166since new packages are not added to the current lts release.
49167
50168Use [Ubuntu Package content search](http://packages.ubuntu.com/) to determine which package provides particular dev files (it defaults to xenial which is the version used to build Nightly).
51169
52−Note we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be add to expected-test-failures.
170+Note that we generally don't install/run services needed for testsuites in the docker images - packages with tests requiring some system service can be added to `expected-test-failures`.
171+It's good to inform the maintainer of any disabled tests (commenting in the PR is sufficient).
172+
173+If a new package fails to build because of missing system libraries we often ask the maintainer to help figure out what to install.
53174
54175### Upgrading GHC version
55176The Dockerfile contains information on which GHC versions should be used. You
@@ -106,7 +227,7 @@ we're just not there yet.
106227/opt/stackage-build/stackage/automated/build.sh lts-3.0
107228```
108229
109−Recommended: run these from inside a `screen` session. If you get version bound
230+Recommended: run these from inside a `tmux` session. If you get version bound
110231problems on nightly or LTS major, you need to fix build-constraints.yaml (see
111232info above). For an LTS minor bump, you'll typically want to use the
112233`CONSTRAINTS` environment variable, e.g.:
@@ -126,13 +247,13 @@ If a build fails for bounds reasons, see all of the advice above. If the code
126247itself doesn't build, or tests fail, open up an issue and then either put in a
127248version bound to avoid that version or something else. It's difficult to give
128249universal advice on how to solve things, since each situation is unique. Let's
129−develop this advice over time. For now: if you're not sure, ask Michael for
130−guidance.
250+develop this advice over time. For now: if you're not sure, ask for guidance.
131251
132252__`NOPLAN=1`__ If you wish to rerun a build without recalculating a
133253build plan, you can set the environment variable `NOPLAN=1`. This is
134254useful for such cases as an intermittent test failure, out of memory
135−condition, or manually tweaking the plan file.
255+condition, or manually tweaking the plan file. This is the default for
256+LTS builds.
136257
137258### Timing
138259
@@ -146,9 +267,33 @@ LTS minor bumps typically are run on Sundays.
146267### Website sync debugging (and other out of disk space errors)
147268
148269* You can detect the problem by running `df`. If you see that `/` is out of space, we have a problem
149−* There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
150−* You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
270+* (outdated) There are many temp files inside `/home/ubuntu/stackage-server-cron` that can be cleared out occasionally
271+* (outdated) You can then manually run `/home/ubuntu/stackage-server-cron.sh`, or wait for the cron job to do it
151272
152273### Wiping the cache
153274
154−Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`. You can wipe the nightly cache and rebuild everything by doing `rm -rf /opt/stackage-build/stackage/automated/nightly`.
275+Sometimes the cache can get corrupted which might manifest as `can't load .so/.DLL`.
276+You can wipe the nightly cache and rebuild everything by doing
277+`rm -rf /var/stackage/stackage/automated/nightly`.
278+Replace nightly with `lts7` to wipe the LTS 7 cache.
279+
280+## Local curator setup
281+
282+We don't run the full stackage build locally as that might take too
283+much time. Some steps on the other hand are much faster to do
284+yourself.
285+
286+It's useful to be able to modify constraints locally before pushing to
287+the repository. To do this first install stackage-curator:
288+`git clone git@github.com:fpco/stackage-curator.git && cd stackage-curator && stack install`
289+or get the linux binary: https://s3.amazonaws.com/stackage-travis/stackage-curator/stackage-curator.bz2
290+(it's a good idea to upgrade stackage-curator at least at the start of your week as curator).
291+Then clone the stackage repo `git clone git@github.com:fpco/stackage.git`.
292+Inside it run `stack update && stackage-curator check` to get new packages and do dependency resolution.
293+
294+This can be used to make sure all version bounds are in place
295+(including for test suites and benchmarks), to check whether bounds
296+can be lifted, and to get `tell-me-when-its-released` notifications.
297+
298+Notably this doesn't build anything, so you won't see any compilation
299+errors for builds/tests/benchmarks.