daml/ghc-lib/working-on-ghc-lib.md

# Working on `ghc-lib`

Copyright 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All Rights Reserved.
SPDX-License-Identifier: (Apache-2.0 OR BSD-3-Clause)

If you need to build, test, deploy or develop [`ghc-lib`](https://github.com/digital-asset/ghc-lib) as used by DAML and utilizing the Digital Asset [GHC fork](https://github.com/digital-asset/ghc) these notes are for you.

## Table of contents
* [Prerequisites](#prerequisites)
* [How to build `ghc` from the DA GHC fork](#how-to-build-ghc-lib-from-the-da-ghc-fork)
* [How to build `ghc-lib` from the DA GHC fork](#how-to-build-ghc-lib-from-the-da-ghc-fork)
* [How to test `ghc-lib`](#how-to-test-ghc-lib)
* [How to deploy `ghc-lib`](#how-to-deploy-ghc-lib)
* [How to rebase `ghc-lib` on upstream master](#how-to-rebase-ghc-lib-on-upstream-master)
* [How to develop `ghc-lib`](#how-to-develop-ghc-lib)

## Prerequisites

- Download `stack` and other tools:
```bash
cd ~
mkdir -p ~/.local/bin
cat << EOF >> ~/.bashrc
export PATH=~/.local/bin:$PATH
EOF

brew install autoconf automake python3 gmp
curl -sSL https://get.haskellstack.org/ > install.sh
chmod +x install.sh && ./install.sh -f -d ~/.local/bin
ln -s /usr/bin/make ~/.local/bin/make
source ~/.bashrc
```

- Optional: So that you can generate a SHA (because this function is missing on MacOS), add this to your `~/.bashrc`:
```bash
function sha256sum() { shasum -a 256 "$@" ; } && export -f sha256sum
```

## How to build `ghc` from the DA GHC fork
To build DA's fork of `ghc` (which incorporates our extensions and DAML syntax):
```
mkdir -p ~/tmp && cd ~/tmp
git clone https://gitlab.haskell.org/ghc/ghc.git
cd ghc
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream
git checkout `git merge-base upstream/da-master master`
git merge --no-edit upstream/da-master
git submodule update --init --recursive
stack build --stack-yaml=hadrian/stack.yaml --only-dependencies
hadrian/build.stack.sh --configure --flavour=quickest -j
```
Note that the `git checkout` step will put you in detached HEAD state - that's expected. The compiler is built to `_build/stage1/bin/ghc`.

The equivalent commands to build the `8.8.1` compatible branch are:
```
git clone https://gitlab.haskell.org/ghc/ghc.git
cd ghc
git fetch --tags
git checkout ghc-8.8.1-rc1
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream
git merge --no-edit upstream/da-master-8.8.1
git submodule update --init --recursive
stack build --stack-yaml=hadrian/stack.yaml --only-dependencies
hadrian/build.stack.sh --configure --flavour=quickest -j
```

## Iterating on Template Desugaring

Modifying GHC, building `ghc-lib` and then building `damlc` is quite time
intensive and makes mistakes very costly. Therefore it is usually preferable to
first take a look at the new output from template desugaring before building `ghc-lib`.
The fastest option for that is to build GHC with

```
./hadrian/build.sh -j --flavour=quickest --freeze1
```

You can then run GHC on a DAML file as follows

```
./_build/stage1/bin/ghc  ~/tmp/Test.hs -ddump-parsed
```

Note that the file should end with `.hs`, otherwise GHC will think that it is an additional input
for the linking phase and your DAML file should start with:

```
{-# LANGUAGE DamlSyntax #-}
```

You will get compile errors after the parse tree has been emitted since the standard library is missing
but if you just want to see the output from template desugaring, this is sufficient.

## How to build `ghc-lib` from the DA GHC fork
(You don't need to follow the previous step in order to do this.)

These instructions detail how to generate `ghc-lib-parser` and `ghc-lib` packages intended for use by `damlc`.

1. Generate `ghc-lib-parser.cabal` by running:
```bash
mkdir -p ~/tmp && cd ~/tmp
git clone git@github.com:digital-asset/ghc-lib.git
cd ghc-lib && git clone https://gitlab.haskell.org/ghc/ghc.git
cd ghc
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream
git checkout `git merge-base upstream/da-master master`
git merge --no-edit upstream/da-master upstream/da-unit-ids
git submodule update --init --recursive
cd ..
stack setup > /dev/null 2>&1
stack build --no-terminal --interleaved-output
stack exec -- ghc-lib-gen ghc --ghc-lib-parser
```
Note that the `git checkout` step will put you in detached HEAD state - that's expected.

The equivalent 8.8.1 commands are:
```
mkdir -p ~/tmp && cd ~/tmp
git clone git@github.com:digital-asset/ghc-lib.git
cd ghc-lib
git checkout ghc-8.8.1-rc1
git clone https://gitlab.haskell.org/ghc/ghc.git
cd ghc
git fetch --tags
git checkout ghc-8.8.1-rc1
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream
git merge --no-edit upstream/da-master-8.8.1 upstream/da-unit-ids-8.8.1
git submodule update --init --recursive
cd ..
stack setup > /dev/null 2>&1
stack build --no-terminal --interleaved-output
stack exec -- ghc-lib-gen ghc --ghc-lib-parser
```

2. Edit `~/tmp/ghc-lib/ghc/ghc-lib-parser.cabal` to (a) change the version number (we use a datestamp, e.g. `0.20190219`) and (b) add clause `extra-libraries:ffi` to the `library` stanza. Then run:
```bash
cat << EOF >> stack.yaml
- ghc
EOF
stack sdist ghc --tar-dir=.
```
This creates `~tmp/ghc-lib/ghc-lib-parser-xxx.tar.gz` where `xxx` is the version number.

3. Generate `ghc-lib.cabal` by running:
```bash
git checkout stack.yaml
(cd ghc && git clean -xf && git checkout .)
stack exec -- ghc-lib-gen ghc --ghc-lib
```

4. Edit `~/tmp/ghc-lib/ghc/ghc-lib.cabal` to (a) change the version number (we use a datestamp, e.g. `0.20190219`), (b) change the `ghc-lib-parser` version number in the `build-depends` stanza and (c) add clause `extra-libraries:ffi` to the `library` stanza. Then run:
```
stack sdist ghc --tar-dir=.
```
This creates `~tmp/ghc-lib/ghc-lib-xxx.tar.gz` where `xxx` is the version number.

5. You can (optionally) test that `ghc-lib-parser` and `ghc-lib` sdists build with these commands:
```bash
tar xvf ghc-lib-parser-xxx.tar.gz
tar xvf ghc-lib-xxx.tar.gz
mv ghc-lib-parser-xxx ghc-lib-parser
mv ghc-lib-xxx ghc-lib
sed '$d' stack.yaml > stack.yaml.tmp&&cp stack.yaml.tmp stack.yaml
cat << EOF >> stack.yaml
- ghc-lib-parser
- ghc-lib
EOF
stack build ghc-lib-parser --no-terminal --interleaved-output
stack build ghc-lib --no-terminal --interleaved-output
```
where, as in steps 3 and 4, `xxx` is the version number.

## How to test `ghc-lib`
Once you've [built `ghc-lib`](#how-to-build-ghc-lib-from-the-da-ghc-fork), you should test it locally:

1. Get the SHAs of the tar.gz files. If you followed the last step in the prerequsites, you can do this by running:
```
sha256sum ghc-lib-parser-xxx.tar.gz
sha256sum ghc-lib-xxx.tar.gz
```
where as before, `xxx` is the version number.

2. At the root of the `daml` repo, edit `WORKSPACE` (determines where Bazel gets `ghc-lib` from).
Update the lines for `ghc-lib-parser` and `ghc-lib` with the new `url`s, `stripPrefix`s and `sha256s`:
```bash
  ("ghc-lib-parser", {"url": "file:///path/to/the/ghc-lib-parser-xxx.tar.gz", "stripPrefix": "ghc-lib-parser-xxx", "sha256": "a422c86eaf6efe7cec8086b1b0f361355d4415825cf0513502755736a191ab44"})
, ("ghc-lib", {"url": "file:///path/to/the/ghc-lib-xxx.tar.gz", "stripPrefix": "ghc-lib-xxx", "sha256": "d422c86eaf6efe7cec8086b1b0f361355d4415825cf0513502755736a191ab66"})
```
3. Check that the DAML tests pass:
```bash
bazel run //compiler/damlc:daml-ghc-test -- --pattern=
```
If they pass, you can move on to [deploying](#how-to-deploy-ghc-lib).

## How to deploy `ghc-lib`
Now you've [built](#how-to-build-ghc-lib-from-the-da-ghc-fork) and [tested `ghc-lib`](#how-to-test-ghc-lib), you can deploy it:

1. Upload `ghc-lib-parser-xxx.tar.gz` and `ghc-lib-xxx.tar.gz` to [bintray](https://bintray.com/digitalassetsdk/ghc-lib) with commands like the following
```bash
API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx;export API_KEY

# Upload commands are formed as follows.
#   curl -T <FILE.EXT> -ushayne.fletcher@digitalassetsdk:<API_KEY> \
#   https://api.bintray.com/content/digitalassetsdk/ghc-lib/<YOUR_COOL_PACKAGE_NAME>/<VERSION_NAME>/<FILE_TARGET_PATH>
curl -T /path/to/ghc-lib-parser-xxx.tar.gz \
     -ushayne.fletcher@digitalassetsdk:$API_KEY \
     https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/ghc-lib-parser-xxx.tar.gz
curl -T /path/to/ghc-lib-xxx.tar.gz \
     -ushayne.fletcher@digitalassetsdk:$API_KEY \
     https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/ghc-lib-xxx.tar.gz
curl -X POST -ushayne.fletcher@digitalassetsdk:$API_KEY \
     https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/publish
```
(where `API_KEY` is replaced by your bintray `API_KEY` which you can retrieve by looking into your bintray profile).

2. In the `daml` repo, create a new feature branch:
```bash
cd daml
git checkout -b update-ghc-lib
```
3. Edit `WORKSPACE` _again_ with the `url`s pointing to bintray this time:
```bash
# Download commands are formed as follows.
#   curl -L "https://digitalassetsdk.bintray.com/ghc-lib/<FILE_PATH>" -o <FILE.EXT>
# Example download URL - https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-xxx.tar.gz.
          ("ghc-lib-parser", {"url": "https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-parser-0.20190401.1.tar.gz", "stripPrefix": "ghc-lib-parser-0.20190401.1", "sha256": "3036ed084ca57668faab25f8ae0420a992e21ad484c6f82acce73705dfed9e33"})
        , ("ghc-lib", {"url": "https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-0.20190401.1.tar.gz", "stripPrefix": "ghc-lib-0.20190401.1", "sha256": "82e94f26729c35fddc7a3d7d6b0c89f397109342b2c092c70173bb537af6f5c9"})
```
4. If you didn't do this before, make sure the DAML tests pass by running:
```bash
bazel run //compiler/damlc:daml-ghc-test -- --pattern=
```
5. When the tests pass, push your branch to origin and raise a PR.

## How to rebase `ghc-lib` on upstream master

To keep `ghc-lib` consistent with changes to upstream GHC source code, it is neccessary to rebase our branches on the upstream `master` from time to time. The procedure for doing this is as follows:
```bash
mkdir -p ~/tmp && cd ~/tmp
git clone git@github.com:digital-asset/ghc.git
cd ghc
git remote add upstream https://gitlab.haskell.org/ghc/ghc.git
git fetch upstream master
# These checkout commands take into account that `da-master` is the "default" branch.
git checkout -t origin/master && git merge upstream/master
git checkout da-master && git rebase master
git checkout -t origin/da-unit-ids && git rebase master
```
Obviously, you will need to deal with any rebase conflicts that come up (hopefully not often). You can test `ghc-lib` after rebasing by following the [build procedure](#how-to-build-ghc-lib-from-the-da-ghc-fork) replacing the line
```bash
git remote add upstream git@github.com:digital-asset/ghc.git
```
with
```bash
git remote add upstream $HOME/tmp/ghc
```
and then the [test procedure](#how-to-test-ghc-lib).

When you are satisfied that the tests pass, you can push the changes to origin with these commands:
```bash
cd ~/tmp/ghc
git push origin master:master
git push -f origin da-master:da-master
git push -f origin da-unit-ids:da-unit-ids
```
After this, release the updated `ghc-lib` following the usual [deployment procedure](#how-to-deploy-ghc-lib).

## How to develop `ghc-lib`

The following procedure sets up a new feature branch with starting point `da-master`.
```bash
mkdir ~/tmp && cd ~/tmp
git clone https://gitlab.haskell.org/ghc/ghc.git ghc.git
cd ghc.git
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream da-master
git checkout -t upstream/da-master
git checkout -b feature-xxx da-master
git push upstream feature-xxx:feature-xxx
```
where `feature-xxx` is replaced by the desired name of your feature branch.

To prepare to produce a `ghc` from your feature branch, remember to first initialize submodules and build hadrian's dependencies (hadrian itself will be built on the first ghc build invocation).
```
git submodule update --init --recursive
stack build --stack-yaml=hadrian/stack.yaml --only-dependencies
```
To build `ghc` invoke hadrian via `hadrian/build.stack.sh`.
```bash
hadrian/build.stack.sh --configure --flavour=quickest -j
```
As usual, the compiler is built to `_build/stage1/bin/ghc`.

When you are ready to publish your feature branch, push to `upstream` and raise your PR with base `da-master`.