@@ -0,0 +1,129 @@
1+# The Stackage data flow
2+
3+The Stackage project is really built on top of a number of different
4+subcomponents. This page covers how they fit together. The Stackage data flow
5+diagram gives a good bird's-eye view:
6+
7+
8+
9+## Inputs
10+
11+There are three inputs into the data flow:
12+
13+* [Hackage](http://hackage.haskell.org/) is the upstream repository of all
14+ available open source Haskell packages that are part of our ecosystem.
15+ Hackage provides both cabal file metadata (via the 00-index.tar file) and
16+ tarballs of the individual packages.
17+
18+* [build-constraints.yaml](https://github.com/fpco/stackage/blob/master/build-constraints.yaml)
19+ is the primary Stackage input file. This is where package maintainers can add
20+ packages to the Stackage package set. This also defines upper bounds, skipped
21+ tests, and a few other pieces of metadata.
22+
23+* [stackage-content](https://github.com/fpco/stackage-content) is a Github
24+ repository containing static file content served from stackage.org
25+
26+## Travis
27+
28+For [various
29+reasons](https://www.fpcomplete.com/blog/2015/05/distributing-packages-without-sysadmin),
30+we leverage Travis CI for running some processes. In particular:
31+
32+* [all-cabal-files](https://github.com/commercialhaskell/all-cabal-files/blob/hackage/.travis.yml)
33+ clones all cabal files from Hackage's 00-index.tar file into a Git repository
34+ without any modification
35+
36+* [all-cabal-hashes](https://github.com/commercialhaskell/all-cabal-hashes/blob/hackage/.travis.yml)
37+ is mostly the same, but also includes cryptographic hashes of the package
38+ tarballs for more secure download (as leveraged by
39+ [Stack](http://haskellstack.com). It is powered by [all-cabal-hashes-tool](https://github.com/commercialhaskell/all-cabal-hashes-tool)
40+
41+* [all-cabal-packages](https://github.com/commercialhaskell/all-cabal-packages)
42+ uses [hackage-mirror](http://github.com/fpco/hackage-mirror) to populate the
43+ hackage.fpcomplete.com mirror of Hackage, which provides S3-backed high
44+ availability hosting of all package tarballs
45+
46+* [all-cabal-metadata](https://github.com/commercialhaskell/all-cabal-metadata)
47+ uses
48+ [all-cabal-metadata-tool](https://github.com/commercialhaskell/all-cabal-metadata-tool)
49+ to query extra metadata from Hackage about packages and put them into YAML
50+ files. As we'll see later, this avoids the need to make a lot of costly calls
51+ to Hackage APIs
52+
53+Travis does not currently provide a means of running jobs on a regular basis.
54+Therefore, we have a simple cron job on the Stackage build server that triggers
55+each of the above builds every 30 minutes.
56+
57+## stackage-curator
58+
59+The heart of running Stackage builds is the
60+[stackage-curator](https://github.com/fpco/stackage-curator) tool. We run this
61+on a daily basis on the Stackage build server for Stackage Nightly, and on a
62+weekly basis for LTS Haskell. The build process is [highly
63+automated](https://github.com/fpco/stackage/blob/master/automated/build.sh) and
64+leverages Docker quite a bit.
65+
66+stackage-curator needs to know about the most recent versions of all packages,
67+their tarball contents, and some metadata, all of which it gets from the
68+Travis-generated sources mentioned in the previous section. In addition, it
69+needs to know about build constraints, which can come from one of two places:
70+
71+* When doing an LTS Haskell minor version bump (e.g., building lts-5.13), it
72+ grabs the previous version (e.g., lts-5.12) and converts the previous package
73+ set into constraints. For example, if lts-5.12 contains the package foo-5.6.7,
74+ this will be converted into the constraint `foo >= 5.6.7 && < 5.7`.
75+* When doing a Stackage Nightly build or LTS Haskell major version bump (e.g.,
76+ building lts-6.0), it grabs the latest version of the build-constraints.yaml
77+ file.
78+
79+By combining these constraints with the current package data, stackage-curator
80+can generate a build plan and check it. (As an aside, this build plan
81+generation and checking also occurs every time you make a pull request to the
82+stackage repo.) If there are version bounds problems, one of the [Stackage
83+curators](https://github.com/fpco/stackage/blob/master/CURATORS.md) will open
84+up a Github issue and will add upper bounds, temporarily block a package, or
85+some other corrective action.
86+
87+Once a valid build plan is found, stackage-curator will build all packages,
88+build docs, and run test suites. Assuming that all succeeds, it generates some
89+artifacts:
90+
91+* Uploads the build plan as a YAML file to either
92+ [stackage-nightly](https://github.com/fpco/stackage-nightly) or
93+ [lts-haskell](https://github.com/fpco/lts-haskell)
94+* Uploads the generated Haddock docs and a package index (containing all used
95+ .cabal files) to haddock.stackage.org.
96+
97+## stackage-server-cron
98+
99+On the Stackage build server, we run the [stackage-server-cron
100+executable](https://github.com/fpco/stackage-server/blob/master/app/stackage-server-cron.hs)
101+regularly, which generates:
102+
103+* A [SQLite
104+ database](https://github.com/fpco/stackage-server/blob/master/Stackage/Database.hs)
105+ containing information on snapshots, the packages they contain, Hackage
106+ metadata about packages, and a bit more. This database is uploaded to S3.
107+* A Hoogle database for each snapshot, which is also uploaded to S3
108+
109+## stackage-server
110+
111+The [software running stackage.org](https://github.com/fpco/stackage-server) is
112+a relatively simple Yesod web application. It pulls data from the
113+stackage-content repo, the SQLite database, the Hoogle databases, and the build
114+plans for Stackage Nightly and LTS Haskell. It doesn't generate anything
115+important of its own except for a user interface.
116+
117+## Stack
118+
119+[Stack](http://haskellstack.com) takes advantage of many of the pieces listed above as well:
120+
121+* It by default uses the all-cabal-hashes repo for getting package metadata,
122+ and downloads package contents from the hackage.fpcomplete.com mirror (using
123+ the hashes in the repo for verification)
124+* There are some metadata files in stackage-content which contain information
125+ on, for example, where to download GHC tarballs from to make `stack setup`
126+ work
127+* Stack downloads the raw build plans for Stackage Nightly and LTS Haskell from
128+ the Github repo and uses them when deciding which packages to build for a
129+ given stack.yaml file