digital-asset/daml
1
1
Fork 0
mirror of https://github.com/digital-asset/daml.git synced 2024-09-20 01:07:18 +03:00
Code Issues Projects Releases Wiki Activity

update instructions for working with ghc-lib (#7958)

Browse Source 
changelog_begin
changelog_end

typos

undo change to old instructions

test & fix new instructions

remove old instructions

rename new instructions

copyright at top

remove update date from text

tee when running example

instruct to make initial build
This commit is contained in:
nickchapman-da 2020-11-17 10:02:19 +00:00 committed by GitHub
parent 10792ed6fd
commit 9bf4ef9ba6
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 67 additions and 298 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

67
ghc-lib/new-working-on-ghc-lib.md Normal file
View File

@ -0,0 +1,67 @@
Copyright 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All Rights Reserved.
SPDX-License-Identifier: (Apache-2.0 OR BSD-3-Clause)
# Working on `ghc-lib`
If you need to build, test, deploy or develop [`ghc-lib`](https://github.com/digital-asset/ghc-lib) as used by DAML and utilizing the Digital Asset [GHC fork](https://github.com/digital-asset/ghc) these notes are for you.
Here are instructions for when working on DAML surface syntax, as implemented in the the digital-assert fork of `ghc`. (linked in via `ghc-lib`).
### Cloning the digital-assert fork of `ghc`
1. Make initial clone from the main ghc gitlab repo:
```
git clone --recurse-submodules https://gitlab.haskell.org/ghc/ghc.git
cd ghc
```
2. Add the DA fork as a remote
```
git remote add da-fork git@github.com:digital-asset/ghc.git
git fetch da-fork
```
3. Checkout the version of interest and update the submodules:
```
git checkout da-master-8.8.1
git submodule update --init --recursive
```
4. Made initial build (takes about 15 mins)
```
hadrian/build.stack.sh --configure --flavour=quickest -j
```
### Iterating on parser/desugaring in `ghc`
Working locally in a branch from `da-master-8.8.1`, there are two files which generally need changing to update syntax and desugaring:
- [`compiler/parser/Parser.y`](https://github.com/digital-asset/ghc/blob/da-master-8.8.1/compiler/parser/Parser.y)
- [`compiler/parser/RdrHsSyn.hs`](https://github.com/digital-asset/ghc/blob/da-master-8.8.1/compiler/parser/RdrHsSyn.hs)
The quickest way to build and test is:
1. `hadrian/build.stack.sh --configure --flavour=quickest -j`
2. `./_build/stage1/bin/ghc ./Example.hs -ddump-parsed | tee desugar.out`
Step 1 gives immediate feedback on build failures, but takes about 2-3 minutes when successful. For Step 2 you need a DAML example file. The input file must end in `.hs` suffix. It must begin with the pragma: `{-# LANGUAGE DamlSyntax #-}`
### Building `daml` following a change to `ghc`
Once you have the GHC patch you want to incorporate into the DAML repo, here's the steps you'll need to take:
1. Open a PR in the daml repo with the commit hash for the GHC patch in `ci/da-ghc-lib/compile.yml`. See [here](https://github.com/digital-asset/daml/pull/7489/commits/fedc456260f598f9924ce62d9765c3c09b8ad861)
2. Wait for CI to build `ghc-lib`/`ghc-lib-parser`, and get the new SHA from the end of the azure CI logs. The CI/azure log you are looking for is in the `Bash` subtab of the `da_ghc_lib` job. The lines of interest are at the very end of the log. See [here](https://dev.azure.com/digitalasset/adadc18a-d7df-446a-aacb-86042c1619c6/_apis/build/builds/60342/logs/52)
3. Update `stack-snapshot.yaml` with the new SHAs. See [here](https://github.com/digital-asset/daml/pull/7489/commits/f0198dc694238437357706c81b0c3d1979483d7a)
3. Run the pin command on linux or mac `bazel run @stackage-unpinned//:pin` and commit those changes as well
4. Before merging the PR, the pin command will also have to be run on windows, and those changes committed as well. You will need access to a windows machine for that: `ad-hoc.sh windows create`

298
ghc-lib/working-on-ghc-lib.md
View File

@ -1,298 +0,0 @@
# Working on `ghc-lib`
Copyright 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All Rights Reserved.
SPDX-License-Identifier: (Apache-2.0 OR BSD-3-Clause)
If you need to build, test, deploy or develop [`ghc-lib`](https://github.com/digital-asset/ghc-lib) as used by DAML and utilizing the Digital Asset [GHC fork](https://github.com/digital-asset/ghc) these notes are for you.
## Table of contents
* [Prerequisites](#prerequisites)
* [How to build `ghc` from the DA GHC fork](#how-to-build-ghc-lib-from-the-da-ghc-fork)
* [How to build `ghc-lib` from the DA GHC fork](#how-to-build-ghc-lib-from-the-da-ghc-fork)
* [How to test `ghc-lib`](#how-to-test-ghc-lib)
* [How to deploy `ghc-lib`](#how-to-deploy-ghc-lib)
* [How to rebase `ghc-lib` on upstream master](#how-to-rebase-ghc-lib-on-upstream-master)
* [How to develop `ghc-lib`](#how-to-develop-ghc-lib)
## Prerequisites
- Download `stack` and other tools:
```bash
cd ~
mkdir -p ~/.local/bin
cat << EOF >> ~/.bashrc
export PATH=~/.local/bin:$PATH
EOF
brew install autoconf automake python3 gmp
curl -sSL https://get.haskellstack.org/ > install.sh
chmod +x install.sh && ./install.sh -f -d ~/.local/bin
ln -s /usr/bin/make ~/.local/bin/make
source ~/.bashrc
```
- Optional: So that you can generate a SHA (because this function is missing on MacOS), add this to your `~/.bashrc`:
```bash
function sha256sum() { shasum -a 256 "$@" ; } && export -f sha256sum
```
## How to build `ghc` from the DA GHC fork
To build DA's fork of `ghc` (which incorporates our extensions and DAML syntax):
```
mkdir -p ~/tmp && cd ~/tmp
git clone https://gitlab.haskell.org/ghc/ghc.git
cd ghc
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream
git checkout `git merge-base upstream/da-master master`
git merge --no-edit upstream/da-master
git submodule update --init --recursive
stack build --stack-yaml=hadrian/stack.yaml --only-dependencies
hadrian/build.stack.sh --configure --flavour=quickest -j
```
Note that the `git checkout` step will put you in detached HEAD state - that's expected. The compiler is built to `_build/stage1/bin/ghc`.
The equivalent commands to build the `8.8.1` compatible branch are:
```
git clone https://gitlab.haskell.org/ghc/ghc.git
cd ghc
git fetch --tags
git checkout ghc-8.8.1-rc1
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream
git merge --no-edit upstream/da-master-8.8.1
git submodule update --init --recursive
stack build --stack-yaml=hadrian/stack.yaml --only-dependencies
hadrian/build.stack.sh --configure --flavour=quickest -j
```
## Iterating on Template Desugaring
Modifying GHC, building `ghc-lib` and then building `damlc` is quite time
intensive and makes mistakes very costly. Therefore it is usually preferable to
first take a look at the new output from template desugaring before building `ghc-lib`.
The fastest option for that is to build GHC with
```
./hadrian/build.sh -j --flavour=quickest --freeze1
```
You can then run GHC on a DAML file as follows
```
./_build/stage1/bin/ghc ~/tmp/Test.hs -ddump-parsed
```
Note that the file should end with `.hs`, otherwise GHC will think that it is an additional input
for the linking phase and your DAML file should start with:
```
{-# LANGUAGE DamlSyntax #-}
```
You will get compile errors after the parse tree has been emitted since the standard library is missing
but if you just want to see the output from template desugaring, this is sufficient.
## How to build `ghc-lib` from the DA GHC fork
(You don't need to follow the previous step in order to do this.)
These instructions detail how to generate `ghc-lib-parser` and `ghc-lib` packages intended for use by `damlc`.
1. Generate `ghc-lib-parser.cabal` by running:
```bash
mkdir -p ~/tmp && cd ~/tmp
git clone git@github.com:digital-asset/ghc-lib.git
cd ghc-lib && git clone https://gitlab.haskell.org/ghc/ghc.git
cd ghc
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream
git checkout `git merge-base upstream/da-master master`
git merge --no-edit upstream/da-master upstream/da-unit-ids
git submodule update --init --recursive
cd ..
stack setup > /dev/null 2>&1
stack build --no-terminal --interleaved-output
stack exec -- ghc-lib-gen ghc --ghc-lib-parser
```
Note that the `git checkout` step will put you in detached HEAD state - that's expected.
The equivalent 8.8.1 commands are:
```
mkdir -p ~/tmp && cd ~/tmp
git clone git@github.com:digital-asset/ghc-lib.git
cd ghc-lib
git checkout ghc-8.8.1-rc1
git clone https://gitlab.haskell.org/ghc/ghc.git
cd ghc
git fetch --tags
git checkout ghc-8.8.1-rc1
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream
git merge --no-edit upstream/da-master-8.8.1 upstream/da-unit-ids-8.8.1
git submodule update --init --recursive
cd ..
stack setup > /dev/null 2>&1
stack build --no-terminal --interleaved-output
stack exec -- ghc-lib-gen ghc --ghc-lib-parser
```
2. Edit `~/tmp/ghc-lib/ghc/ghc-lib-parser.cabal` to (a) change the version number (we use a datestamp, e.g. `0.20190219`) and (b) add clause `extra-libraries:ffi` to the `library` stanza. Then run:
```bash
cat << EOF >> stack.yaml
- ghc
EOF
stack sdist ghc --tar-dir=.
```
This creates `~tmp/ghc-lib/ghc-lib-parser-xxx.tar.gz` where `xxx` is the version number.
3. Generate `ghc-lib.cabal` by running:
```bash
git checkout stack.yaml
(cd ghc && git clean -xf && git checkout .)
stack exec -- ghc-lib-gen ghc --ghc-lib
```
4. Edit `~/tmp/ghc-lib/ghc/ghc-lib.cabal` to (a) change the version number (we use a datestamp, e.g. `0.20190219`), (b) change the `ghc-lib-parser` version number in the `build-depends` stanza and (c) add clause `extra-libraries:ffi` to the `library` stanza. Then run:
```
stack sdist ghc --tar-dir=.
```
This creates `~tmp/ghc-lib/ghc-lib-xxx.tar.gz` where `xxx` is the version number.
5. You can (optionally) test that `ghc-lib-parser` and `ghc-lib` sdists build with these commands:
```bash
tar xvf ghc-lib-parser-xxx.tar.gz
tar xvf ghc-lib-xxx.tar.gz
mv ghc-lib-parser-xxx ghc-lib-parser
mv ghc-lib-xxx ghc-lib
sed '$d' stack.yaml > stack.yaml.tmp&&cp stack.yaml.tmp stack.yaml
cat << EOF >> stack.yaml
- ghc-lib-parser
- ghc-lib
EOF
stack build ghc-lib-parser --no-terminal --interleaved-output
stack build ghc-lib --no-terminal --interleaved-output
```
where, as in steps 3 and 4, `xxx` is the version number.
## How to test `ghc-lib`
Once you've [built `ghc-lib`](#how-to-build-ghc-lib-from-the-da-ghc-fork), you should test it locally:
1. Get the SHAs of the tar.gz files. If you followed the last step in the prerequsites, you can do this by running:
```
sha256sum ghc-lib-parser-xxx.tar.gz
sha256sum ghc-lib-xxx.tar.gz
```
where as before, `xxx` is the version number.
2. At the root of the `daml` repo, edit `WORKSPACE` (determines where Bazel gets `ghc-lib` from).
Update the lines for `ghc-lib-parser` and `ghc-lib` with the new `url`s, `stripPrefix`s and `sha256s`:
```bash
("ghc-lib-parser", {"url": "file:///path/to/the/ghc-lib-parser-xxx.tar.gz", "stripPrefix": "ghc-lib-parser-xxx", "sha256": "a422c86eaf6efe7cec8086b1b0f361355d4415825cf0513502755736a191ab44"})
, ("ghc-lib", {"url": "file:///path/to/the/ghc-lib-xxx.tar.gz", "stripPrefix": "ghc-lib-xxx", "sha256": "d422c86eaf6efe7cec8086b1b0f361355d4415825cf0513502755736a191ab66"})
```
3. Check that the DAML tests pass:
```bash
bazel run //compiler/damlc:daml-ghc-test -- --pattern=
```
If they pass, you can move on to [deploying](#how-to-deploy-ghc-lib).
## How to deploy `ghc-lib`
Now you've [built](#how-to-build-ghc-lib-from-the-da-ghc-fork) and [tested `ghc-lib`](#how-to-test-ghc-lib), you can deploy it:
1. Upload `ghc-lib-parser-xxx.tar.gz` and `ghc-lib-xxx.tar.gz` to [bintray](https://bintray.com/digitalassetsdk/ghc-lib) with commands like the following
```bash
API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx;export API_KEY
# Upload commands are formed as follows.
# curl -T <FILE.EXT> -ushayne.fletcher@digitalassetsdk:<API_KEY> \
# https://api.bintray.com/content/digitalassetsdk/ghc-lib/<YOUR_COOL_PACKAGE_NAME>/<VERSION_NAME>/<FILE_TARGET_PATH>
curl -T /path/to/ghc-lib-parser-xxx.tar.gz \
-ushayne.fletcher@digitalassetsdk:$API_KEY \
https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/ghc-lib-parser-xxx.tar.gz
curl -T /path/to/ghc-lib-xxx.tar.gz \
-ushayne.fletcher@digitalassetsdk:$API_KEY \
https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/ghc-lib-xxx.tar.gz
curl -X POST -ushayne.fletcher@digitalassetsdk:$API_KEY \
https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/publish
```
(where `API_KEY` is replaced by your bintray `API_KEY` which you can retrieve by looking into your bintray profile).
2. In the `daml` repo, create a new feature branch:
```bash
cd daml
git checkout -b update-ghc-lib
```
3. Edit `WORKSPACE` _again_ with the `url`s pointing to bintray this time:
```bash
# Download commands are formed as follows.
# curl -L "https://digitalassetsdk.bintray.com/ghc-lib/<FILE_PATH>" -o <FILE.EXT>
# Example download URL - https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-xxx.tar.gz.
("ghc-lib-parser", {"url": "https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-parser-0.20190401.1.tar.gz", "stripPrefix": "ghc-lib-parser-0.20190401.1", "sha256": "3036ed084ca57668faab25f8ae0420a992e21ad484c6f82acce73705dfed9e33"})
, ("ghc-lib", {"url": "https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-0.20190401.1.tar.gz", "stripPrefix": "ghc-lib-0.20190401.1", "sha256": "82e94f26729c35fddc7a3d7d6b0c89f397109342b2c092c70173bb537af6f5c9"})
```
4. If you didn't do this before, make sure the DAML tests pass by running:
```bash
bazel run //compiler/damlc:daml-ghc-test -- --pattern=
```
5. When the tests pass, push your branch to origin and raise a PR.
## How to rebase `ghc-lib` on upstream master
To keep `ghc-lib` consistent with changes to upstream GHC source code, it is neccessary to rebase our branches on the upstream `master` from time to time. The procedure for doing this is as follows:
```bash
mkdir -p ~/tmp && cd ~/tmp
git clone git@github.com:digital-asset/ghc.git
cd ghc
git remote add upstream https://gitlab.haskell.org/ghc/ghc.git
git fetch upstream master
# These checkout commands take into account that `da-master` is the "default" branch.
git checkout -t origin/master && git merge upstream/master
git checkout da-master && git rebase master
git checkout -t origin/da-unit-ids && git rebase master
```
Obviously, you will need to deal with any rebase conflicts that come up (hopefully not often). You can test `ghc-lib` after rebasing by following the [build procedure](#how-to-build-ghc-lib-from-the-da-ghc-fork) replacing the line
```bash
git remote add upstream git@github.com:digital-asset/ghc.git
```
with
```bash
git remote add upstream $HOME/tmp/ghc
```
and then the [test procedure](#how-to-test-ghc-lib).
When you are satisfied that the tests pass, you can push the changes to origin with these commands:
```bash
cd ~/tmp/ghc
git push origin master:master
git push -f origin da-master:da-master
git push -f origin da-unit-ids:da-unit-ids
```
After this, release the updated `ghc-lib` following the usual [deployment procedure](#how-to-deploy-ghc-lib).
## How to develop `ghc-lib`
The following procedure sets up a new feature branch with starting point `da-master`.
```bash
mkdir ~/tmp && cd ~/tmp
git clone https://gitlab.haskell.org/ghc/ghc.git ghc.git
cd ghc.git
git remote add upstream git@github.com:digital-asset/ghc.git
git fetch upstream da-master
git checkout -t upstream/da-master
git checkout -b feature-xxx da-master
git push upstream feature-xxx:feature-xxx
```
where `feature-xxx` is replaced by the desired name of your feature branch.
To prepare to produce a `ghc` from your feature branch, remember to first initialize submodules and build hadrian's dependencies (hadrian itself will be built on the first ghc build invocation).
```
git submodule update --init --recursive
stack build --stack-yaml=hadrian/stack.yaml --only-dependencies
```
To build `ghc` invoke hadrian via `hadrian/build.stack.sh`.
```bash
hadrian/build.stack.sh --configure --flavour=quickest -j
```
As usual, the compiler is built to `_build/stage1/bin/ghc`.
When you are ready to publish your feature branch, push to `upstream` and raise your PR with base `da-master`.