daml/ghc-lib/working-on-ghc-lib.md

Working on ghc-lib

Copyright 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All Rights Reserved. SPDX-License-Identifier: (Apache-2.0 OR BSD-3-Clause)

If you need to build, test, deploy or develop ghc-lib as used by DAML and utilizing the Digital Asset GHC fork these notes are for you.

Table of contents

• Prerequisites
• How to build ghc from the DA GHC fork
• How to build ghc-lib from the DA GHC fork
• How to test ghc-lib
• How to deploy ghc-lib
• How to rebase ghc-lib on upstream master
• How to develop ghc-lib

Prerequisites

• Download stack and other tools:

cd ~
mkdir -p ~/.local/bin
cat << EOF >> ~/.bashrc
export PATH=~/.local/bin:$PATH
EOF

brew install autoconf automake python3 gmp
curl -sSL https://get.haskellstack.org/ > install.sh
chmod +x install.sh && ./install.sh -f -d ~/.local/bin
ln -s /usr/bin/make ~/.local/bin/make
source ~/.bashrc

• Optional: So that you can generate a SHA (because this function is missing on MacOS), add this to your ~/.bashrc:

function sha256sum() { shasum -a 256 "$@" ; } && export -f sha256sum

How to build ghc from the DA GHC fork

To build DA's fork of ghc (which incorporates our extensions and DAML syntax):

mkdir -p ~/tmp && cd ~/tmp
git clone https://gitlab.haskell.org/ghc/ghc.git
cd ghc
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream
git checkout `git merge-base upstream/da-master master`
git merge --no-edit upstream/da-master
git submodule update --init --recursive
stack build --stack-yaml=hadrian/stack.yaml --only-dependencies
hadrian/build.stack.sh --configure --flavour=quickest -j

Note that the git checkout step will put you in detached HEAD state - that's expected. The compiler is built to _build/stage1/bin/ghc.

The equivalent commands to build the 8.8.1 compatible branch are:

git clone https://gitlab.haskell.org/ghc/ghc.git
cd ghc
git fetch --tags
git checkout ghc-8.8.1-rc1
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream
git merge --no-edit upstream/da-master-8.8.1
git submodule update --init --recursive
stack build --stack-yaml=hadrian/stack.yaml --only-dependencies
hadrian/build.stack.sh --configure --flavour=quickest -j

Iterating on Template Desugaring

Modifying GHC, building ghc-lib and then building damlc is quite time intensive and makes mistakes very costly. Therefore it is usually preferable to first take a look at the new output from template desugaring before building ghc-lib. The fastest option for that is to build GHC with

./hadrian/build.sh -j --flavour=quickest --freeze1

You can then run GHC on a DAML file as follows

./_build/stage1/bin/ghc  ~/tmp/Test.hs -ddump-parsed

Note that the file should end with .hs, otherwise GHC will think that it is an additional input for the linking phase and your DAML file should start with:

{-# LANGUAGE DamlSyntax #-}

You will get compile errors after the parse tree has been emitted since the standard library is missing but if you just want to see the output from template desugaring, this is sufficient.

How to build ghc-lib from the DA GHC fork

(You don't need to follow the previous step in order to do this.)

These instructions detail how to generate ghc-lib-parser and ghc-lib packages intended for use by damlc.

1. Generate ghc-lib-parser.cabal by running:

mkdir -p ~/tmp && cd ~/tmp
git clone git@github.com:digital-asset/ghc-lib.git
cd ghc-lib && git clone https://gitlab.haskell.org/ghc/ghc.git
cd ghc
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream
git checkout `git merge-base upstream/da-master master`
git merge --no-edit upstream/da-master upstream/da-unit-ids
git submodule update --init --recursive
cd ..
stack setup > /dev/null 2>&1
stack build --no-terminal --interleaved-output
stack exec -- ghc-lib-gen ghc --ghc-lib-parser

Note that the git checkout step will put you in detached HEAD state - that's expected.

The equivalent 8.8.1 commands are:

mkdir -p ~/tmp && cd ~/tmp
git clone git@github.com:digital-asset/ghc-lib.git
cd ghc-lib
git checkout ghc-8.8.1-rc1
git clone https://gitlab.haskell.org/ghc/ghc.git
cd ghc
git fetch --tags
git checkout ghc-8.8.1-rc1
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream
git merge --no-edit upstream/da-master-8.8.1 upstream/da-unit-ids-8.8.1
git submodule update --init --recursive
cd ..
stack setup > /dev/null 2>&1
stack build --no-terminal --interleaved-output
stack exec -- ghc-lib-gen ghc --ghc-lib-parser

2. Edit ~/tmp/ghc-lib/ghc/ghc-lib-parser.cabal to (a) change the version number (we use a datestamp, e.g. 0.20190219) and (b) add clause extra-libraries:ffi to the library stanza. Then run:

cat << EOF >> stack.yaml
- ghc
EOF
stack sdist ghc --tar-dir=.

This creates ~tmp/ghc-lib/ghc-lib-parser-xxx.tar.gz where xxx is the version number.

3. Generate ghc-lib.cabal by running:

git checkout stack.yaml
(cd ghc && git clean -xf && git checkout .)
stack exec -- ghc-lib-gen ghc --ghc-lib

4. Edit ~/tmp/ghc-lib/ghc/ghc-lib.cabal to (a) change the version number (we use a datestamp, e.g. 0.20190219), (b) change the ghc-lib-parser version number in the build-depends stanza and (c) add clause extra-libraries:ffi to the library stanza. Then run:

stack sdist ghc --tar-dir=.

This creates ~tmp/ghc-lib/ghc-lib-xxx.tar.gz where xxx is the version number.

5. You can (optionally) test that ghc-lib-parser and ghc-lib sdists build with these commands:

tar xvf ghc-lib-parser-xxx.tar.gz
tar xvf ghc-lib-xxx.tar.gz
mv ghc-lib-parser-xxx ghc-lib-parser
mv ghc-lib-xxx ghc-lib
sed '$d' stack.yaml > stack.yaml.tmp&&cp stack.yaml.tmp stack.yaml
cat << EOF >> stack.yaml
- ghc-lib-parser
- ghc-lib
EOF
stack build ghc-lib-parser --no-terminal --interleaved-output
stack build ghc-lib --no-terminal --interleaved-output

where, as in steps 3 and 4, xxx is the version number.

How to test ghc-lib

Once you've built ghc-lib, you should test it locally:

1. Get the SHAs of the tar.gz files. If you followed the last step in the prerequsites, you can do this by running:

sha256sum ghc-lib-parser-xxx.tar.gz
sha256sum ghc-lib-xxx.tar.gz

where as before, xxx is the version number.

2. At the root of the daml repo, edit WORKSPACE (determines where Bazel gets ghc-lib from). Update the lines for ghc-lib-parser and ghc-lib with the new urls, stripPrefixs and sha256s:

  ("ghc-lib-parser", {"url": "file:///path/to/the/ghc-lib-parser-xxx.tar.gz", "stripPrefix": "ghc-lib-parser-xxx", "sha256": "a422c86eaf6efe7cec8086b1b0f361355d4415825cf0513502755736a191ab44"})
, ("ghc-lib", {"url": "file:///path/to/the/ghc-lib-xxx.tar.gz", "stripPrefix": "ghc-lib-xxx", "sha256": "d422c86eaf6efe7cec8086b1b0f361355d4415825cf0513502755736a191ab66"})

3. Check that the DAML tests pass:

bazel run //compiler/damlc:daml-ghc-test -- --pattern=

If they pass, you can move on to deploying.

How to deploy ghc-lib

Now you've built and tested ghc-lib, you can deploy it:

1. Upload ghc-lib-parser-xxx.tar.gz and ghc-lib-xxx.tar.gz to bintray with commands like the following

API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx;export API_KEY

# Upload commands are formed as follows.
#   curl -T <FILE.EXT> -ushayne.fletcher@digitalassetsdk:<API_KEY> \
#   https://api.bintray.com/content/digitalassetsdk/ghc-lib/<YOUR_COOL_PACKAGE_NAME>/<VERSION_NAME>/<FILE_TARGET_PATH>
curl -T /path/to/ghc-lib-parser-xxx.tar.gz \
     -ushayne.fletcher@digitalassetsdk:$API_KEY \
     https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/ghc-lib-parser-xxx.tar.gz
curl -T /path/to/ghc-lib-xxx.tar.gz \
     -ushayne.fletcher@digitalassetsdk:$API_KEY \
     https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/ghc-lib-xxx.tar.gz
curl -X POST -ushayne.fletcher@digitalassetsdk:$API_KEY \
     https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/publish

(where API_KEY is replaced by your bintray API_KEY which you can retrieve by looking into your bintray profile).

2. In the daml repo, create a new feature branch:

cd daml
git checkout -b update-ghc-lib

3. Edit WORKSPACE again with the urls pointing to bintray this time:

# Download commands are formed as follows.
#   curl -L "https://digitalassetsdk.bintray.com/ghc-lib/<FILE_PATH>" -o <FILE.EXT>
# Example download URL - https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-xxx.tar.gz.
          ("ghc-lib-parser", {"url": "https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-parser-0.20190401.1.tar.gz", "stripPrefix": "ghc-lib-parser-0.20190401.1", "sha256": "3036ed084ca57668faab25f8ae0420a992e21ad484c6f82acce73705dfed9e33"})
        , ("ghc-lib", {"url": "https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-0.20190401.1.tar.gz", "stripPrefix": "ghc-lib-0.20190401.1", "sha256": "82e94f26729c35fddc7a3d7d6b0c89f397109342b2c092c70173bb537af6f5c9"})

4. If you didn't do this before, make sure the DAML tests pass by running:

bazel run //compiler/damlc:daml-ghc-test -- --pattern=

5. When the tests pass, push your branch to origin and raise a PR.

How to rebase ghc-lib on upstream master

To keep ghc-lib consistent with changes to upstream GHC source code, it is neccessary to rebase our branches on the upstream master from time to time. The procedure for doing this is as follows:

mkdir -p ~/tmp && cd ~/tmp
git clone git@github.com:digital-asset/ghc.git
cd ghc
git remote add upstream https://gitlab.haskell.org/ghc/ghc.git
git fetch upstream master
# These checkout commands take into account that `da-master` is the "default" branch.
git checkout -t origin/master && git merge upstream/master
git checkout da-master && git rebase master
git checkout -t origin/da-unit-ids && git rebase master

Obviously, you will need to deal with any rebase conflicts that come up (hopefully not often). You can test ghc-lib after rebasing by following the build procedure replacing the line

git remote add upstream git@github.com:digital-asset/ghc.git

with

git remote add upstream $HOME/tmp/ghc

and then the test procedure.

When you are satisfied that the tests pass, you can push the changes to origin with these commands:

cd ~/tmp/ghc
git push origin master:master
git push -f origin da-master:da-master
git push -f origin da-unit-ids:da-unit-ids

After this, release the updated ghc-lib following the usual deployment procedure.

How to develop ghc-lib

The following procedure sets up a new feature branch with starting point da-master.

mkdir ~/tmp && cd ~/tmp
git clone https://gitlab.haskell.org/ghc/ghc.git ghc.git
cd ghc.git
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream da-master
git checkout -t upstream/da-master
git checkout -b feature-xxx da-master
git push upstream feature-xxx:feature-xxx

where feature-xxx is replaced by the desired name of your feature branch.

To prepare to produce a ghc from your feature branch, remember to first initialize submodules and build hadrian's dependencies (hadrian itself will be built on the first ghc build invocation).

git submodule update --init --recursive
stack build --stack-yaml=hadrian/stack.yaml --only-dependencies

To build ghc invoke hadrian via hadrian/build.stack.sh.

hadrian/build.stack.sh --configure --flavour=quickest -j

As usual, the compiler is built to _build/stage1/bin/ghc.

When you are ready to publish your feature branch, push to upstream and raise your PR with base da-master.