git.iko.soy/haskell.nix
on iko/ghcjs-patch-master help sign in
Adds documentation (#46)

* Adds documentation

commit79a704960e3ed2fb09e7453cb9f0a8c66518e681 authorMoritz Angermann <moritz.angermann@gmail.com> dateFri 22 Feb 2019, 10:54 (8 years ago) parent24b6d6a
8 files changed+536
compare with iko/ghcjs-patch-master
unifiedsplit
docs/architecture.md+64 docs/index.md+90 docs/iohk-nix.md+122 doc/removing-with-package-wrapper.md → docs/removing-with-package-wrapper.md docs/user-guide-cabal.md+65 docs/user-guide-stack.md+49 docs/user-guide.md+117 mkdocs.yml+29
docs/architecture.mdnew file+64
@@ -0,0 +1,64 @@
1+# Architecture
2+
3+There are multiple components that play a part in the haskell.nix
4+infrastructure. These are `nix-tools`, `haskell.nix`, `hackage.nix`,
5+and `stackage.nix`.
6+
7+```no-highlight
8+ .-------------. .-------------.
9+.- nix-tools ------. | haskell.nix | .- | hackage.nix |
10+| .--------------. | .----------------. '-------------' | '-------------'
11+| | stack-to-nix |---> | stack-pkgs.nix |-. | | |
12+| '--------------' | '----------------' | v | v
13+| .-------------. | .----------. '--> .----------. <-' .--------------.
14+| | plan-to-nix |----> | plan.nix |------.---> | pkgs.nix | <--- | stackage.nix |
15+| '-------------' | '----------' | '----------' '--------------'
16+| .--------------. | .--------------. | |
17+| | cabal-to-nix |---> | $package.nix |--' v
18+| '--------------' | '--------------' .-------------.
19+'------------------' | default.nix |
20+ '-------------'
21+ |
22+ v
23+ .-------------.
24+ | release.nix |
25+ '-------------'
26+```
27+haskell.nix diagram
28+
29+## [nix-tools](https://github.com/input-output-hk/nix-tools)
30+
31+nix-tools is a Haskell package that provides the following tools:
32+
33+- `cabal-to-nix`: a `.cabal` to `.nix` transformer that retains
34+ conditional expressions.
35+
36+- `stack-to-nix`: a `stack.yaml` to `.nix` transformer that will read
37+ in a `stack.yaml` expression an generate a `pkgs.nix` file suited for
38+ use with `haskell.nix`.
39+
40+- `plan-to-nix`: a `plan.json` to `.nix` transformer that will read in
41+ a `plan.json` file and generate a `pkgs.nix` file suited for use
42+ with `haskell.nix`.
43+
44+as well as a few other tools used to generate `hackage.nix` and `stackage.nix`.
45+
46+## [haskell.nix](https://github.com/input-output-hk/haskell.nix)
47+
48+haskell.nix is the runtime system for this Haskell infrastructure. It
49+contains the component builder, as well as the system package and
50+license mapping. Without haskell.nix the expressions generated by
51+either of the `nix-tools` tools make little sense on their own.
52+
53+## [hackage.nix](https://github.com/input-output-hk/hackage.nix)
54+
55+hackage.nix provides all cabal expressions from hackage as nix
56+expressions. It is periodically updated to keep in sync with the set
57+of packages available on hackage.
58+
59+## [stackage.nix](https://github.com/input-output-hk/stackage.nix)
60+
61+stackage.nix is similar to hackage.nix but provides all stackage
62+snapshots (lts, and nightly) as nix expressions. It naturally depends
63+on hackage.nix to resolve package names, versions and revisions to the
64+repsective packages from hackage.nix.
docs/index.mdnew file+90
@@ -0,0 +1,90 @@
1+# haskell.nix
2+
3+haskell.nix is an alternative Haskell infrastructure for nixpkgs. See
4+[Nixpkgs current Users' Guide to Haskell Infrastructure](https://nixos.org/nixpkgs/manual/#users-guide-to-the-haskell-infrastructure) for comparison.
5+
6+## Motivation
7+
8+Why do we need another Haskell infrastructure for nix? Doesn't nixpkgs
9+provide a sufficiently good Haskell infrastructure already? These are
10+good questions. And it boils down to the following reasons for us to
11+embark on a new infrastructure:
12+
13+- first class support for cross compilation
14+- first class support for package sets
15+- component level control when building packages
16+- reduction of `dontCheck` for cyclic dependencies
17+- reducing build times by building libraries and tests in parallel
18+- more logic encoded in nix expressions
19+- decoupling of Haskell and nixpkgs
20+
21+### cross compilation
22+
23+`nixpkgs` has quite good support for cross compilation, however the
24+Haskell infrastructure suffers from the fact that it heavily relies on
25+the `cabal2nix` tool. `cabal2nix` (as well as tools that depend on it
26+like `stack2nix`) flattens the `.cabal` file at conversion time to a
27+given os/arch/flags configuration. Thus to make cross compilation
28+work with `cabal2nix` you will have to generate a separate `nix`
29+expression for each configuration. This becomes a major maintenance
30+burden over time. Therefore the tooling that translates cabal files
31+into nix-expressions for use with Haskell.nix retains the full
32+conditional tree from the cabal file and exposes it to `nix`. In
33+addition it will also expose the `build-type` value, which allows us
34+to cache the `Setup.hs` for build-type simple and not have to rebuild
35+it every time.
36+
37+### package sets
38+
39+We often rely on either package sets as provided by stackage or
40+computed by cabal. `nixpkgs` provides it's own curated package set
41+which might or might not work for the projects we work on.
42+`stack2nix` tries to solve this issue, here we go one step further and
43+provide the infrastructure to allow any form of package set.
44+
45+### component level control
46+
47+The Haskell builder in `nixpkgs` provides control over executables and
48+libraries, to build a specific executable only however is rather
49+tricky to do. This also leads to the cyclic dependencies issue.
50+
51+### cyclic dependencies
52+
53+Because the Haskell builder in `nixpkgs` exposes packages at the
54+package level, if packages mutually depend on each other through tests
55+and libraries, lead to cyclic dependencies that nix can't resolve. By
56+exposing the components to nix as separate derivations this will only
57+occur if you have mutually dependent components.
58+
59+### build times
60+
61+The Haskell builder in nixpkgs build package sequentially, first the
62+library than the executables and finally the tests. It then executes
63+the tests before the package is considered done. The upshot of this
64+is that packages are only considered done if the test-suites
65+passed. The downside is that if you have to compile multiple packages
66+the likelihood of them failing is low, you have unnecessarily
67+serialized you build. In a more aggressive setting libraries could
68+start building as early as their dependent libraries are built. Of
69+course they will have to be invalidated later should the test-suites
70+of their dependencies fail, but this way we can make use of parallel
71+building. In an ideal scenario this will reduce build times close to
72+the optimum.
73+
74+### more logic in nix
75+
76+The `cabal2nix` tool has a resolver that resolved system dependencies
77+and licenses to values in `nixpkgs`. This logic end up being a simple
78+dictionary lookup and can be a simple nix expression. This also
79+offloads some of the work the cabal to nix translation tool needs to
80+do into nix, and as such if changes are necessary (or needed to be
81+performed ad hoc) there is no need to rebuild the conversion tool and
82+subsequently mark every derived expression as out of date.
83+
84+### decoupleing
85+
86+Finally by treating Haskell.nix and nixpkgs as separate entities we
87+can decouple the Haskell packages and infrastructure from the nixpkgs
88+package set, and rely on it to provide us with system packages while
89+staying up to date with Haskell packages from hackage while retaining
90+a stable (or known to be good) nixpkgs revision.
docs/iohk-nix.mdnew file+122
@@ -0,0 +1,122 @@
1+# IOHK's nix tooling
2+
3+## [iohk-nix](https://github.com/input-output-hk/iohk-nix)
4+
5+iohk-nix is IOHK's shared nix library. It provides some templates to
6+make working with haskell.nix trivial but is non-essential to use
7+haskell.nix infrastructure.
8+
9+lib.nix
10+```nix
11+let
12+ # iohk-nix can be overridden for debugging purposes by setting
13+ # NIX_PATH=iohk_nix=/path/to/iohk-nix
14+ iohkNix = import (
15+ let try = builtins.tryEval <iohk_nix>;
16+ in if try.success
17+ then builtins.trace "using host <iohk_nix>" try.value
18+ else
19+ let
20+ spec = builtins.fromJSON (builtins.readFile ./iohk-nix.json);
21+ in builtins.fetchTarball {
22+ url = "${spec.url}/archive/${spec.rev}.tar.gz";
23+ inherit (spec) sha256;
24+ }) {};
25+
26+ pkgs = iohkNix.pkgs;
27+ lib = pkgs.lib;
28+in lib // { inherit iohkNix pkgs; inherit (iohkNix) nix-tools; }
29+```
30+
31+iohk-nix.json
32+```json
33+{
34+ "url": "https://github.com/input-output-hk/iohk-nix",
35+ "rev": "c92f0119ef5814b0ed1f445c2fdcf8894e326294",
36+ "sha256": "05r90x6x3yp1nb66rkc4n0i8q15c634rrdsr2zvb118s3sdcmmrm",
37+ "fetchSubmodules": false
38+}
39+```
40+
41+nix/pkgs.nix
42+```
43+{ pkgs ? import <nixpkgs> {}
44+, iohk-overlay ? {}
45+, iohk-module ? {}
46+, haskell
47+, hackage
48+, stackage
49+, ...
50+}:
51+let
52+ # our packages
53+ stack-pkgs = import ./.stack-pkgs.nix;
54+
55+ # packages which will require TH and thus
56+ # will need -fexternal-interpreter treatment
57+ # when cross compiling.
58+ th-packages = [
59+ "hedgehog" "cardano-crypto-wrapper"
60+ "cardano-crypto-test" "cardano-chain"
61+ "small-steps" "cs-ledger" ];
62+
63+ # Build the packageset with module support.
64+ # We can essentially override anything in the modules
65+ # section.
66+ #
67+ # packages.cbors.patches = [ ./one.patch ];
68+ # packages.cbors.flags.optimize-gmp = false;
69+ #
70+ compiler = (stack-pkgs.overlay hackage).compiler.nix-name;
71+ pkgSet = haskell.mkNewPkgSet {
72+ inherit pkgs;
73+ pkg-def = stackage.${stack-pkgs.resolver};
74+ # The overlay allows extension or restriction of the set of
75+ # packages we are interested in. By using the stack-pkgs.overlay
76+ # we restrict our package set to the ones provided in stack.yaml.
77+ pkg-def-overlays = [
78+ stack-pkgs.overlay
79+ iohk-overlay.${compiler}
80+ ];
81+ # package customizations
82+ modules = [
83+ # This module will ensure that we get the necessary
84+ # patches ontop of GHC packages that for which the
85+ # ones that GHC ships are not identical to the ones
86+ # we find on hackage. These patches will make sure
87+ # they are identical by augmenting the packages on
88+ # hackage to match those that ship with ghc.
89+ haskell.ghcHackagePatches.${compiler}
90+
91+ # the iohk-module will supply us with the necessary
92+ # cross compilation plumbing to make Template Haskell
93+ # work when cross compiling. For now we need to
94+ # list the packages that require template haskell
95+ # explicity here.
96+ (iohk-module { nixpkgs = pkgs;
97+ inherit th-packages; })
98+ ];
99+ };
100+in
101+ pkgSet.config.hsPkgs // { _config = pkgSet.config; }
102+```
103+
104+
105+default.nix
106+```
107+let
108+ localLib = import ./lib.nix;
109+in
110+# This file needs to export a function that takes
111+# the arguments it is passed and forwards them to
112+# the default-nix template from iohk-nix. This is
113+# important so that the release.nix file can properly
114+# parameterize this file when targetting different
115+# hosts.
116+{ ... }@args:
117+# We will instantiate the defaul-nix template with the
118+# nix/pkgs.nix file...
119+localLib.nix-tools.default-nix ./nix/pkgs.nix args
120+# ... and add a few custom packages as well.
121+// { }
122+```
doc/removing-with-package-wrapper.md → docs/removing-with-package-wrapper.mdrenamed

The content is the same.

docs/user-guide-cabal.mdnew file+65
@@ -0,0 +1,65 @@
1+# User Guide (cabal project)
2+
3+Here we will look into how to generate the `pkgs.nix` file for a
4+`cabal.project` project. For the full integration please see the [User
5+Guide](/user-guide)
6+
7+## Using `plan-to-nix`
8+
9+*We currently don't have a `project-to-nix` tool yet, as such creating
10+ the relevant `pkgs.nix` file for a `cabal.project` is slightly more
11+ involved than for a corresponding stack project*.
12+
13+With [nix-tools](https://github.com/input-output-hk/nix-tools) in
14+`PATH`, we can simply run the following command on a stack project:
15+
16+```bash
17+# make sure the cabal project is configured (the plan.json file is generated)
18+cabal new-configure
19+# convert the plan.json file into a pkgs.nix file
20+plan-to-nix dist-newstyle/cache/plan.json > nix/plan.nix
21+```
22+
23+This will produce a `nix/plan.nix` file that looks like the following:
24+```nix
25+hackage:
26+ {
27+ packages = {
28+ "o-clock" = hackage.o-clock."0.1.1".revisions.default;
29+ ...
30+ };
31+ compiler = { ... };
32+ }
33+```
34+
35+it specifically does not include any of our local packages yet. We
36+will need to run
37+
38+```bash
39+cabal-to-nix $path > nix/$pkg.nix
40+```
41+or
42+```bash
43+cabal-to-nix $url $rev > nix/$pkg.nix
44+```
45+for each local (or source) package.
46+
47+With this in place we can then proceed to build the `nix/pkgs.nix`
48+file as follows:
49+
50+```nix
51+let plan = import ./plan.nix; in
52+{ ... }:
53+{ pkg-def = plan;
54+ overlay =
55+ { local-package-a = ./local-package-a.nix;
56+ local-package-b = ./local-package-b.nix;
57+ source-import-a = ./source-import-a.nix;
58+ source-import-b = ./source-import-b.nix;
59+ ...
60+ };
61+}
62+```
63+
64+*If you came here from the [User Guide](/user-guide), go back and
65+ complete the setup.*
docs/user-guide-stack.mdnew file+49
@@ -0,0 +1,49 @@
1+# User Guide (stack project)
2+
3+Here we will look into how to generate the `pkgs.nix` file for a
4+`stack.yaml` project. For the full integration please see the [User
5+Guide](/user-guide)
6+
7+## Using `stack-to-nix`
8+
9+With [nix-tools](https://github.com/input-output-hk/nix-tools) in
10+`PATH`, we can simply run the following command on a stack project:
11+
12+```bash
13+stack-to-nix -o nix stack.yaml > nix/.stack-pkgs.nix
14+```
15+
16+This will produce a `nix/.stack-pkgs.nix` file that looks like the following:
17+```nix
18+{
19+ resolver = "lts-12.17";
20+ overlay = hackage:
21+ {
22+ packages = {
23+ "o-clock" = hackage.o-clock."0.1.1".revisions.default;
24+ ...
25+ } // {
26+ my-package = ./.stack.nix/my-package.nix;
27+ ...
28+ };
29+ };
30+}
31+```
32+
33+This file contains the stackage resolver, as well as an overlay of
34+packages. The overlay specifies which `extra-deps` (here: clock-0.1.1)
35+we wanted to overlay over the stackage snapshot, and what local
36+packages we want (here: my-package).
37+
38+We will then create the following `nix/pkgs.nix` file:
39+
40+```nix
41+let stack-pkgs = import ./.stack-pkgs.nix; in
42+{ stackage, ... }:
43+{ pkg-def = stackage.${stack-pkgs.resolver};
44+ inherit (stack-pkgs) overlay;
45+}
46+```
47+
48+*If you came here from the [User Guide](/user-guide), go back and
49+ complete the setup.*
docs/user-guide.mdnew file+117
@@ -0,0 +1,117 @@
1+# User Guide
2+
3+So you want to use `haskell.nix` with your stack or cabal project. The
4+general approach will be to pick the right tool from `nix-tools` and
5+produce a `pkgs.nix` expressions. Getting a copy of the haskell.nix,
6+hackage.nix (and potentially stackage.nix) source will then equip us
7+to produce derivations that we can `nix build`.
8+
9+## Setup
10+
11+The general structure will be the same for haskell.nix, independent of
12+the use of stack or cabal. Let us assume for now that we have
13+generated a `pkgs.nix` expression in `nix`.
14+
15+- [Generating a `pkgs.nix` for a stack project](/user-guide-stack)
16+- [Generating a `pkgs.nix` for a cabal project](/user-guide-cabal)
17+
18+## default.nix
19+
20+We will start with defining a helper function in our `default.nix`
21+that will allow us to override the source imports with `-I
22+haskell=/path/to/haskell.nix` in case we need it.
23+
24+```nix
25+{ pkgs ? import <nixpkgs> {}
26+}:
27+let
28+ overrideWith = override: default:
29+ let
30+ try = builtins.tryEval (builtins.findFile builtins.nixPath override);
31+ in if try.success then
32+ builtins.trace "using search host <${override}>" try.value
33+ else
34+ default;
35+in
36+```
37+
38+Next we will use this to import `haskell.nix`, `hackage.nix` and
39+`stackage.nix` (if we use a stack project).
40+
41+**NOTE**: update the `rev` and `sha256` values to the recent ones as
42+ found on GitHub. Especially `hackage.nix` and `stackage.nix` will
43+ evolve with package release on hackage and stackage releases
44+ respectively.
45+
46+```nix
47+let
48+ # all packages from hackage as nix expressions
49+ hackage = import (overrideWith "hackage"
50+ (pkgs.fetchFromGitHub { owner = "input-output-hk";
51+ repo = "hackage.nix";
52+ rev = "3180384b563ec7c7b46bca86b3ace0f32d04cde8";
53+ sha256 = "19ndkn8pivli9plwq0wnx1cj126l89yk7jw9a0dj51ic3b2qhlb2";
54+ name = "hackage-exprs-source"; }))
55+ ;
56+ # a different haskell infrastructure
57+ haskell = import (overrideWith "haskell"
58+ (pkgs.fetchFromGitHub { owner = "input-output-hk";
59+ repo = "haskell.nix";
60+ rev = "73f733ba8bbd11443dda713d1a2d4b7c50a5d408";
61+ sha256 = "1p2srrxw2lac5krrg35waa251by98r2miwyg6zac9glpg2vmq3ip";
62+ name = "haskell-lib-source"; }))
63+ hackage;
64+
65+ # the set of all stackage snapshots
66+ stackage = import (overrideWith "stackage"
67+ (pkgs.fetchFromGitHub { owner = "input-output-hk";
68+ repo = "stackage.nix";
69+ rev = "2615a4e6b1651215ee400e62fcdcb195062a3d35";
70+ sha256 = "08c8lb8x047hndwm1cb2zxixnjmrswfp5y18xp1v79cjqlva0qj6";
71+ name = "stackage-snapshot-source"; }))
72+ ;
73+in
74+```
75+
76+Finally we string this together and produce a package set:
77+
78+```nix
79+let
80+ # Import the file you will create in the stack-to-nix or cabal-to-nix step.
81+ my-pkgs = import ./nix/pkgs.nix { inherit stackage; };
82+
83+ pkgSet = haskell.mkPkgSet {
84+ inherit pkgs;
85+ pkg-def = my-pkgs.pkg-def;
86+ pkg-def-overlays = [
87+ # this overlay will provide additional packages
88+ # ontop of the package set. E.g. extra-deps
89+ # for stack packages. or local packages for
90+ # cabal.projects
91+ my-pkgs.overlay
92+ ];
93+ modules = [
94+ # specific package overrides would go here
95+ # example:
96+ # packages.cbors.patches = [ ./one.patch ];
97+ # packages.cbors.flags.optimize-gmp = false;
98+ ];
99+ };
100+
101+in pkgSet.config.hsPkgs // { _config = pkgSet.config; }
102+```
103+
104+With this setup you can then start building the components of
105+interest:
106+
107+```bash
108+nix build -f default.nix $pkg.components.library
109+```
110+
111+to build the library for `$pkg` or
112+
113+```bash
114+nix build -f default.nix $pkg.components.exes.$exe
115+```
116+
117+to build a specific executable. The same holds for test suites and benchmarks.
mkdocs.ymlnew file+29
@@ -0,0 +1,29 @@
1+# You can use mkdocs from nix, by running the following:
2+#
3+# nix-shell -p mkdocs
4+#
5+# that should pop you into a shell, where you can then
6+# run
7+#
8+# mkdocs serve
9+#
10+# and point your browser to localhost:8000
11+#
12+site_name: Alternative Haskell Infrastructure for Nixpkgs
13+theme: readthedocs
14+site_url: null
15+repo_url: https://github.com/input-output-hk/haskell.nix
16+
17+# this is called `nav` in more recent mkdocs versions. Mine
18+# right now is 0.17.5
19+pages:
20+ - 'Introduction': index.md
21+ - 'Architecture': architecture.md
22+ - 'User Guide':
23+ - 'General': user-guide.md
24+ - 'Stack Projects': user-guide-stack.md
25+ - 'Cabal Proejcts': user-guide-cabal.md
26+ - 'Templates / Abstraction':
27+ - 'IOHKs nix library': iohk-nix.md
28+ - 'Dev Notes':
29+ - 'Removing withPackage wrapper': removing-with-package-wrapper.md