From 9bf4ef9ba6e6310aef9ceb22bd741bade4f367fb Mon Sep 17 00:00:00 2001 From: nickchapman-da <49153372+nickchapman-da@users.noreply.github.com> Date: Tue, 17 Nov 2020 10:02:19 +0000 Subject: [PATCH] update instructions for working with ghc-lib (#7958) changelog_begin changelog_end typos undo change to old instructions test & fix new instructions remove old instructions rename new instructions copyright at top remove update date from text tee when running example instruct to make initial build --- ghc-lib/new-working-on-ghc-lib.md | 67 +++++++ ghc-lib/working-on-ghc-lib.md | 298 ------------------------------ 2 files changed, 67 insertions(+), 298 deletions(-) create mode 100644 ghc-lib/new-working-on-ghc-lib.md delete mode 100644 ghc-lib/working-on-ghc-lib.md diff --git a/ghc-lib/new-working-on-ghc-lib.md b/ghc-lib/new-working-on-ghc-lib.md new file mode 100644 index 0000000000..3e46261bff --- /dev/null +++ b/ghc-lib/new-working-on-ghc-lib.md @@ -0,0 +1,67 @@ +Copyright 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All Rights Reserved. +SPDX-License-Identifier: (Apache-2.0 OR BSD-3-Clause) + +# Working on `ghc-lib` + +If you need to build, test, deploy or develop [`ghc-lib`](https://github.com/digital-asset/ghc-lib) as used by DAML and utilizing the Digital Asset [GHC fork](https://github.com/digital-asset/ghc) these notes are for you. + +Here are instructions for when working on DAML surface syntax, as implemented in the the digital-assert fork of `ghc`. (linked in via `ghc-lib`). + + +### Cloning the digital-assert fork of `ghc` + +1. Make initial clone from the main ghc gitlab repo: +``` +git clone --recurse-submodules https://gitlab.haskell.org/ghc/ghc.git +cd ghc +``` + +2. Add the DA fork as a remote +``` +git remote add da-fork git@github.com:digital-asset/ghc.git +git fetch da-fork +``` + +3. Checkout the version of interest and update the submodules: +``` +git checkout da-master-8.8.1 +git submodule update --init --recursive +``` + +4. Made initial build (takes about 15 mins) +``` +hadrian/build.stack.sh --configure --flavour=quickest -j +``` + + +### Iterating on parser/desugaring in `ghc` + +Working locally in a branch from `da-master-8.8.1`, there are two files which generally need changing to update syntax and desugaring: + +- [`compiler/parser/Parser.y`](https://github.com/digital-asset/ghc/blob/da-master-8.8.1/compiler/parser/Parser.y) + +- [`compiler/parser/RdrHsSyn.hs`](https://github.com/digital-asset/ghc/blob/da-master-8.8.1/compiler/parser/RdrHsSyn.hs) + + +The quickest way to build and test is: + +1. `hadrian/build.stack.sh --configure --flavour=quickest -j` + +2. `./_build/stage1/bin/ghc ./Example.hs -ddump-parsed | tee desugar.out` + +Step 1 gives immediate feedback on build failures, but takes about 2-3 minutes when successful. For Step 2 you need a DAML example file. The input file must end in `.hs` suffix. It must begin with the pragma: `{-# LANGUAGE DamlSyntax #-}` + + +### Building `daml` following a change to `ghc` + +Once you have the GHC patch you want to incorporate into the DAML repo, here's the steps you'll need to take: + +1. Open a PR in the daml repo with the commit hash for the GHC patch in `ci/da-ghc-lib/compile.yml`. See [here](https://github.com/digital-asset/daml/pull/7489/commits/fedc456260f598f9924ce62d9765c3c09b8ad861) + +2. Wait for CI to build `ghc-lib`/`ghc-lib-parser`, and get the new SHA from the end of the azure CI logs. The CI/azure log you are looking for is in the `Bash` subtab of the `da_ghc_lib` job. The lines of interest are at the very end of the log. See [here](https://dev.azure.com/digitalasset/adadc18a-d7df-446a-aacb-86042c1619c6/_apis/build/builds/60342/logs/52) + +3. Update `stack-snapshot.yaml` with the new SHAs. See [here](https://github.com/digital-asset/daml/pull/7489/commits/f0198dc694238437357706c81b0c3d1979483d7a) + +3. Run the pin command on linux or mac `bazel run @stackage-unpinned//:pin` and commit those changes as well + +4. Before merging the PR, the pin command will also have to be run on windows, and those changes committed as well. You will need access to a windows machine for that: `ad-hoc.sh windows create` diff --git a/ghc-lib/working-on-ghc-lib.md b/ghc-lib/working-on-ghc-lib.md deleted file mode 100644 index cec9604ab2..0000000000 --- a/ghc-lib/working-on-ghc-lib.md +++ /dev/null @@ -1,298 +0,0 @@ -# Working on `ghc-lib` - -Copyright 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All Rights Reserved. -SPDX-License-Identifier: (Apache-2.0 OR BSD-3-Clause) - -If you need to build, test, deploy or develop [`ghc-lib`](https://github.com/digital-asset/ghc-lib) as used by DAML and utilizing the Digital Asset [GHC fork](https://github.com/digital-asset/ghc) these notes are for you. - -## Table of contents -* [Prerequisites](#prerequisites) -* [How to build `ghc` from the DA GHC fork](#how-to-build-ghc-lib-from-the-da-ghc-fork) -* [How to build `ghc-lib` from the DA GHC fork](#how-to-build-ghc-lib-from-the-da-ghc-fork) -* [How to test `ghc-lib`](#how-to-test-ghc-lib) -* [How to deploy `ghc-lib`](#how-to-deploy-ghc-lib) -* [How to rebase `ghc-lib` on upstream master](#how-to-rebase-ghc-lib-on-upstream-master) -* [How to develop `ghc-lib`](#how-to-develop-ghc-lib) - -## Prerequisites - -- Download `stack` and other tools: -```bash -cd ~ -mkdir -p ~/.local/bin -cat << EOF >> ~/.bashrc -export PATH=~/.local/bin:$PATH -EOF - -brew install autoconf automake python3 gmp -curl -sSL https://get.haskellstack.org/ > install.sh -chmod +x install.sh && ./install.sh -f -d ~/.local/bin -ln -s /usr/bin/make ~/.local/bin/make -source ~/.bashrc -``` - -- Optional: So that you can generate a SHA (because this function is missing on MacOS), add this to your `~/.bashrc`: -```bash -function sha256sum() { shasum -a 256 "$@" ; } && export -f sha256sum -``` - -## How to build `ghc` from the DA GHC fork -To build DA's fork of `ghc` (which incorporates our extensions and DAML syntax): -``` -mkdir -p ~/tmp && cd ~/tmp -git clone https://gitlab.haskell.org/ghc/ghc.git -cd ghc -git remote add upstream git@github.com:digital-asset/ghc.git -git fetch upstream -git checkout `git merge-base upstream/da-master master` -git merge --no-edit upstream/da-master -git submodule update --init --recursive -stack build --stack-yaml=hadrian/stack.yaml --only-dependencies -hadrian/build.stack.sh --configure --flavour=quickest -j -``` -Note that the `git checkout` step will put you in detached HEAD state - that's expected. The compiler is built to `_build/stage1/bin/ghc`. - -The equivalent commands to build the `8.8.1` compatible branch are: -``` -git clone https://gitlab.haskell.org/ghc/ghc.git -cd ghc -git fetch --tags -git checkout ghc-8.8.1-rc1 -git remote add upstream git@github.com:digital-asset/ghc.git -git fetch upstream -git merge --no-edit upstream/da-master-8.8.1 -git submodule update --init --recursive -stack build --stack-yaml=hadrian/stack.yaml --only-dependencies -hadrian/build.stack.sh --configure --flavour=quickest -j -``` - -## Iterating on Template Desugaring - -Modifying GHC, building `ghc-lib` and then building `damlc` is quite time -intensive and makes mistakes very costly. Therefore it is usually preferable to -first take a look at the new output from template desugaring before building `ghc-lib`. -The fastest option for that is to build GHC with - -``` -./hadrian/build.sh -j --flavour=quickest --freeze1 -``` - -You can then run GHC on a DAML file as follows - -``` -./_build/stage1/bin/ghc ~/tmp/Test.hs -ddump-parsed -``` - -Note that the file should end with `.hs`, otherwise GHC will think that it is an additional input -for the linking phase and your DAML file should start with: - -``` -{-# LANGUAGE DamlSyntax #-} -``` - -You will get compile errors after the parse tree has been emitted since the standard library is missing -but if you just want to see the output from template desugaring, this is sufficient. - -## How to build `ghc-lib` from the DA GHC fork -(You don't need to follow the previous step in order to do this.) - -These instructions detail how to generate `ghc-lib-parser` and `ghc-lib` packages intended for use by `damlc`. - -1. Generate `ghc-lib-parser.cabal` by running: -```bash -mkdir -p ~/tmp && cd ~/tmp -git clone git@github.com:digital-asset/ghc-lib.git -cd ghc-lib && git clone https://gitlab.haskell.org/ghc/ghc.git -cd ghc -git remote add upstream git@github.com:digital-asset/ghc.git -git fetch upstream -git checkout `git merge-base upstream/da-master master` -git merge --no-edit upstream/da-master upstream/da-unit-ids -git submodule update --init --recursive -cd .. -stack setup > /dev/null 2>&1 -stack build --no-terminal --interleaved-output -stack exec -- ghc-lib-gen ghc --ghc-lib-parser -``` -Note that the `git checkout` step will put you in detached HEAD state - that's expected. - -The equivalent 8.8.1 commands are: -``` -mkdir -p ~/tmp && cd ~/tmp -git clone git@github.com:digital-asset/ghc-lib.git -cd ghc-lib -git checkout ghc-8.8.1-rc1 -git clone https://gitlab.haskell.org/ghc/ghc.git -cd ghc -git fetch --tags -git checkout ghc-8.8.1-rc1 -git remote add upstream git@github.com:digital-asset/ghc.git -git fetch upstream -git merge --no-edit upstream/da-master-8.8.1 upstream/da-unit-ids-8.8.1 -git submodule update --init --recursive -cd .. -stack setup > /dev/null 2>&1 -stack build --no-terminal --interleaved-output -stack exec -- ghc-lib-gen ghc --ghc-lib-parser -``` - -2. Edit `~/tmp/ghc-lib/ghc/ghc-lib-parser.cabal` to (a) change the version number (we use a datestamp, e.g. `0.20190219`) and (b) add clause `extra-libraries:ffi` to the `library` stanza. Then run: -```bash -cat << EOF >> stack.yaml -- ghc -EOF -stack sdist ghc --tar-dir=. -``` -This creates `~tmp/ghc-lib/ghc-lib-parser-xxx.tar.gz` where `xxx` is the version number. - -3. Generate `ghc-lib.cabal` by running: -```bash -git checkout stack.yaml -(cd ghc && git clean -xf && git checkout .) -stack exec -- ghc-lib-gen ghc --ghc-lib -``` - -4. Edit `~/tmp/ghc-lib/ghc/ghc-lib.cabal` to (a) change the version number (we use a datestamp, e.g. `0.20190219`), (b) change the `ghc-lib-parser` version number in the `build-depends` stanza and (c) add clause `extra-libraries:ffi` to the `library` stanza. Then run: -``` -stack sdist ghc --tar-dir=. -``` -This creates `~tmp/ghc-lib/ghc-lib-xxx.tar.gz` where `xxx` is the version number. - -5. You can (optionally) test that `ghc-lib-parser` and `ghc-lib` sdists build with these commands: -```bash -tar xvf ghc-lib-parser-xxx.tar.gz -tar xvf ghc-lib-xxx.tar.gz -mv ghc-lib-parser-xxx ghc-lib-parser -mv ghc-lib-xxx ghc-lib -sed '$d' stack.yaml > stack.yaml.tmp&&cp stack.yaml.tmp stack.yaml -cat << EOF >> stack.yaml -- ghc-lib-parser -- ghc-lib -EOF -stack build ghc-lib-parser --no-terminal --interleaved-output -stack build ghc-lib --no-terminal --interleaved-output -``` -where, as in steps 3 and 4, `xxx` is the version number. - -## How to test `ghc-lib` -Once you've [built `ghc-lib`](#how-to-build-ghc-lib-from-the-da-ghc-fork), you should test it locally: - -1. Get the SHAs of the tar.gz files. If you followed the last step in the prerequsites, you can do this by running: -``` -sha256sum ghc-lib-parser-xxx.tar.gz -sha256sum ghc-lib-xxx.tar.gz -``` -where as before, `xxx` is the version number. - -2. At the root of the `daml` repo, edit `WORKSPACE` (determines where Bazel gets `ghc-lib` from). -Update the lines for `ghc-lib-parser` and `ghc-lib` with the new `url`s, `stripPrefix`s and `sha256s`: -```bash - ("ghc-lib-parser", {"url": "file:///path/to/the/ghc-lib-parser-xxx.tar.gz", "stripPrefix": "ghc-lib-parser-xxx", "sha256": "a422c86eaf6efe7cec8086b1b0f361355d4415825cf0513502755736a191ab44"}) -, ("ghc-lib", {"url": "file:///path/to/the/ghc-lib-xxx.tar.gz", "stripPrefix": "ghc-lib-xxx", "sha256": "d422c86eaf6efe7cec8086b1b0f361355d4415825cf0513502755736a191ab66"}) -``` -3. Check that the DAML tests pass: -```bash -bazel run //compiler/damlc:daml-ghc-test -- --pattern= -``` -If they pass, you can move on to [deploying](#how-to-deploy-ghc-lib). - -## How to deploy `ghc-lib` -Now you've [built](#how-to-build-ghc-lib-from-the-da-ghc-fork) and [tested `ghc-lib`](#how-to-test-ghc-lib), you can deploy it: - -1. Upload `ghc-lib-parser-xxx.tar.gz` and `ghc-lib-xxx.tar.gz` to [bintray](https://bintray.com/digitalassetsdk/ghc-lib) with commands like the following -```bash -API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx;export API_KEY - -# Upload commands are formed as follows. -# curl -T -ushayne.fletcher@digitalassetsdk: \ -# https://api.bintray.com/content/digitalassetsdk/ghc-lib/// -curl -T /path/to/ghc-lib-parser-xxx.tar.gz \ - -ushayne.fletcher@digitalassetsdk:$API_KEY \ - https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/ghc-lib-parser-xxx.tar.gz -curl -T /path/to/ghc-lib-xxx.tar.gz \ - -ushayne.fletcher@digitalassetsdk:$API_KEY \ - https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/ghc-lib-xxx.tar.gz -curl -X POST -ushayne.fletcher@digitalassetsdk:$API_KEY \ - https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/publish -``` -(where `API_KEY` is replaced by your bintray `API_KEY` which you can retrieve by looking into your bintray profile). - -2. In the `daml` repo, create a new feature branch: -```bash -cd daml -git checkout -b update-ghc-lib -``` -3. Edit `WORKSPACE` _again_ with the `url`s pointing to bintray this time: -```bash -# Download commands are formed as follows. -# curl -L "https://digitalassetsdk.bintray.com/ghc-lib/" -o -# Example download URL - https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-xxx.tar.gz. - ("ghc-lib-parser", {"url": "https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-parser-0.20190401.1.tar.gz", "stripPrefix": "ghc-lib-parser-0.20190401.1", "sha256": "3036ed084ca57668faab25f8ae0420a992e21ad484c6f82acce73705dfed9e33"}) - , ("ghc-lib", {"url": "https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-0.20190401.1.tar.gz", "stripPrefix": "ghc-lib-0.20190401.1", "sha256": "82e94f26729c35fddc7a3d7d6b0c89f397109342b2c092c70173bb537af6f5c9"}) -``` -4. If you didn't do this before, make sure the DAML tests pass by running: -```bash -bazel run //compiler/damlc:daml-ghc-test -- --pattern= -``` -5. When the tests pass, push your branch to origin and raise a PR. - -## How to rebase `ghc-lib` on upstream master - -To keep `ghc-lib` consistent with changes to upstream GHC source code, it is neccessary to rebase our branches on the upstream `master` from time to time. The procedure for doing this is as follows: -```bash -mkdir -p ~/tmp && cd ~/tmp -git clone git@github.com:digital-asset/ghc.git -cd ghc -git remote add upstream https://gitlab.haskell.org/ghc/ghc.git -git fetch upstream master -# These checkout commands take into account that `da-master` is the "default" branch. -git checkout -t origin/master && git merge upstream/master -git checkout da-master && git rebase master -git checkout -t origin/da-unit-ids && git rebase master -``` -Obviously, you will need to deal with any rebase conflicts that come up (hopefully not often). You can test `ghc-lib` after rebasing by following the [build procedure](#how-to-build-ghc-lib-from-the-da-ghc-fork) replacing the line -```bash -git remote add upstream git@github.com:digital-asset/ghc.git -``` -with -```bash -git remote add upstream $HOME/tmp/ghc -``` -and then the [test procedure](#how-to-test-ghc-lib). - -When you are satisfied that the tests pass, you can push the changes to origin with these commands: -```bash -cd ~/tmp/ghc -git push origin master:master -git push -f origin da-master:da-master -git push -f origin da-unit-ids:da-unit-ids -``` -After this, release the updated `ghc-lib` following the usual [deployment procedure](#how-to-deploy-ghc-lib). - -## How to develop `ghc-lib` - -The following procedure sets up a new feature branch with starting point `da-master`. -```bash -mkdir ~/tmp && cd ~/tmp -git clone https://gitlab.haskell.org/ghc/ghc.git ghc.git -cd ghc.git -git remote add upstream git@github.com:digital-asset/ghc.git -git fetch upstream da-master -git checkout -t upstream/da-master -git checkout -b feature-xxx da-master -git push upstream feature-xxx:feature-xxx -``` -where `feature-xxx` is replaced by the desired name of your feature branch. - -To prepare to produce a `ghc` from your feature branch, remember to first initialize submodules and build hadrian's dependencies (hadrian itself will be built on the first ghc build invocation). -``` -git submodule update --init --recursive -stack build --stack-yaml=hadrian/stack.yaml --only-dependencies -``` -To build `ghc` invoke hadrian via `hadrian/build.stack.sh`. -```bash -hadrian/build.stack.sh --configure --flavour=quickest -j -``` -As usual, the compiler is built to `_build/stage1/bin/ghc`. - -When you are ready to publish your feature branch, push to `upstream` and raise your PR with base `da-master`.