mirror of
https://github.com/digital-asset/daml.git
synced 2024-09-20 01:07:18 +03:00
update instructions for working with ghc-lib (#7958)
changelog_begin changelog_end typos undo change to old instructions test & fix new instructions remove old instructions rename new instructions copyright at top remove update date from text tee when running example instruct to make initial build
This commit is contained in:
parent
10792ed6fd
commit
9bf4ef9ba6
67
ghc-lib/new-working-on-ghc-lib.md
Normal file
67
ghc-lib/new-working-on-ghc-lib.md
Normal file
@ -0,0 +1,67 @@
|
||||
Copyright 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All Rights Reserved.
|
||||
SPDX-License-Identifier: (Apache-2.0 OR BSD-3-Clause)
|
||||
|
||||
# Working on `ghc-lib`
|
||||
|
||||
If you need to build, test, deploy or develop [`ghc-lib`](https://github.com/digital-asset/ghc-lib) as used by DAML and utilizing the Digital Asset [GHC fork](https://github.com/digital-asset/ghc) these notes are for you.
|
||||
|
||||
Here are instructions for when working on DAML surface syntax, as implemented in the the digital-assert fork of `ghc`. (linked in via `ghc-lib`).
|
||||
|
||||
|
||||
### Cloning the digital-assert fork of `ghc`
|
||||
|
||||
1. Make initial clone from the main ghc gitlab repo:
|
||||
```
|
||||
git clone --recurse-submodules https://gitlab.haskell.org/ghc/ghc.git
|
||||
cd ghc
|
||||
```
|
||||
|
||||
2. Add the DA fork as a remote
|
||||
```
|
||||
git remote add da-fork git@github.com:digital-asset/ghc.git
|
||||
git fetch da-fork
|
||||
```
|
||||
|
||||
3. Checkout the version of interest and update the submodules:
|
||||
```
|
||||
git checkout da-master-8.8.1
|
||||
git submodule update --init --recursive
|
||||
```
|
||||
|
||||
4. Made initial build (takes about 15 mins)
|
||||
```
|
||||
hadrian/build.stack.sh --configure --flavour=quickest -j
|
||||
```
|
||||
|
||||
|
||||
### Iterating on parser/desugaring in `ghc`
|
||||
|
||||
Working locally in a branch from `da-master-8.8.1`, there are two files which generally need changing to update syntax and desugaring:
|
||||
|
||||
- [`compiler/parser/Parser.y`](https://github.com/digital-asset/ghc/blob/da-master-8.8.1/compiler/parser/Parser.y)
|
||||
|
||||
- [`compiler/parser/RdrHsSyn.hs`](https://github.com/digital-asset/ghc/blob/da-master-8.8.1/compiler/parser/RdrHsSyn.hs)
|
||||
|
||||
|
||||
The quickest way to build and test is:
|
||||
|
||||
1. `hadrian/build.stack.sh --configure --flavour=quickest -j`
|
||||
|
||||
2. `./_build/stage1/bin/ghc ./Example.hs -ddump-parsed | tee desugar.out`
|
||||
|
||||
Step 1 gives immediate feedback on build failures, but takes about 2-3 minutes when successful. For Step 2 you need a DAML example file. The input file must end in `.hs` suffix. It must begin with the pragma: `{-# LANGUAGE DamlSyntax #-}`
|
||||
|
||||
|
||||
### Building `daml` following a change to `ghc`
|
||||
|
||||
Once you have the GHC patch you want to incorporate into the DAML repo, here's the steps you'll need to take:
|
||||
|
||||
1. Open a PR in the daml repo with the commit hash for the GHC patch in `ci/da-ghc-lib/compile.yml`. See [here](https://github.com/digital-asset/daml/pull/7489/commits/fedc456260f598f9924ce62d9765c3c09b8ad861)
|
||||
|
||||
2. Wait for CI to build `ghc-lib`/`ghc-lib-parser`, and get the new SHA from the end of the azure CI logs. The CI/azure log you are looking for is in the `Bash` subtab of the `da_ghc_lib` job. The lines of interest are at the very end of the log. See [here](https://dev.azure.com/digitalasset/adadc18a-d7df-446a-aacb-86042c1619c6/_apis/build/builds/60342/logs/52)
|
||||
|
||||
3. Update `stack-snapshot.yaml` with the new SHAs. See [here](https://github.com/digital-asset/daml/pull/7489/commits/f0198dc694238437357706c81b0c3d1979483d7a)
|
||||
|
||||
3. Run the pin command on linux or mac `bazel run @stackage-unpinned//:pin` and commit those changes as well
|
||||
|
||||
4. Before merging the PR, the pin command will also have to be run on windows, and those changes committed as well. You will need access to a windows machine for that: `ad-hoc.sh windows create`
|
@ -1,298 +0,0 @@
|
||||
# Working on `ghc-lib`
|
||||
|
||||
Copyright 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All Rights Reserved.
|
||||
SPDX-License-Identifier: (Apache-2.0 OR BSD-3-Clause)
|
||||
|
||||
If you need to build, test, deploy or develop [`ghc-lib`](https://github.com/digital-asset/ghc-lib) as used by DAML and utilizing the Digital Asset [GHC fork](https://github.com/digital-asset/ghc) these notes are for you.
|
||||
|
||||
## Table of contents
|
||||
* [Prerequisites](#prerequisites)
|
||||
* [How to build `ghc` from the DA GHC fork](#how-to-build-ghc-lib-from-the-da-ghc-fork)
|
||||
* [How to build `ghc-lib` from the DA GHC fork](#how-to-build-ghc-lib-from-the-da-ghc-fork)
|
||||
* [How to test `ghc-lib`](#how-to-test-ghc-lib)
|
||||
* [How to deploy `ghc-lib`](#how-to-deploy-ghc-lib)
|
||||
* [How to rebase `ghc-lib` on upstream master](#how-to-rebase-ghc-lib-on-upstream-master)
|
||||
* [How to develop `ghc-lib`](#how-to-develop-ghc-lib)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Download `stack` and other tools:
|
||||
```bash
|
||||
cd ~
|
||||
mkdir -p ~/.local/bin
|
||||
cat << EOF >> ~/.bashrc
|
||||
export PATH=~/.local/bin:$PATH
|
||||
EOF
|
||||
|
||||
brew install autoconf automake python3 gmp
|
||||
curl -sSL https://get.haskellstack.org/ > install.sh
|
||||
chmod +x install.sh && ./install.sh -f -d ~/.local/bin
|
||||
ln -s /usr/bin/make ~/.local/bin/make
|
||||
source ~/.bashrc
|
||||
```
|
||||
|
||||
- Optional: So that you can generate a SHA (because this function is missing on MacOS), add this to your `~/.bashrc`:
|
||||
```bash
|
||||
function sha256sum() { shasum -a 256 "$@" ; } && export -f sha256sum
|
||||
```
|
||||
|
||||
## How to build `ghc` from the DA GHC fork
|
||||
To build DA's fork of `ghc` (which incorporates our extensions and DAML syntax):
|
||||
```
|
||||
mkdir -p ~/tmp && cd ~/tmp
|
||||
git clone https://gitlab.haskell.org/ghc/ghc.git
|
||||
cd ghc
|
||||
git remote add upstream git@github.com:digital-asset/ghc.git
|
||||
git fetch upstream
|
||||
git checkout `git merge-base upstream/da-master master`
|
||||
git merge --no-edit upstream/da-master
|
||||
git submodule update --init --recursive
|
||||
stack build --stack-yaml=hadrian/stack.yaml --only-dependencies
|
||||
hadrian/build.stack.sh --configure --flavour=quickest -j
|
||||
```
|
||||
Note that the `git checkout` step will put you in detached HEAD state - that's expected. The compiler is built to `_build/stage1/bin/ghc`.
|
||||
|
||||
The equivalent commands to build the `8.8.1` compatible branch are:
|
||||
```
|
||||
git clone https://gitlab.haskell.org/ghc/ghc.git
|
||||
cd ghc
|
||||
git fetch --tags
|
||||
git checkout ghc-8.8.1-rc1
|
||||
git remote add upstream git@github.com:digital-asset/ghc.git
|
||||
git fetch upstream
|
||||
git merge --no-edit upstream/da-master-8.8.1
|
||||
git submodule update --init --recursive
|
||||
stack build --stack-yaml=hadrian/stack.yaml --only-dependencies
|
||||
hadrian/build.stack.sh --configure --flavour=quickest -j
|
||||
```
|
||||
|
||||
## Iterating on Template Desugaring
|
||||
|
||||
Modifying GHC, building `ghc-lib` and then building `damlc` is quite time
|
||||
intensive and makes mistakes very costly. Therefore it is usually preferable to
|
||||
first take a look at the new output from template desugaring before building `ghc-lib`.
|
||||
The fastest option for that is to build GHC with
|
||||
|
||||
```
|
||||
./hadrian/build.sh -j --flavour=quickest --freeze1
|
||||
```
|
||||
|
||||
You can then run GHC on a DAML file as follows
|
||||
|
||||
```
|
||||
./_build/stage1/bin/ghc ~/tmp/Test.hs -ddump-parsed
|
||||
```
|
||||
|
||||
Note that the file should end with `.hs`, otherwise GHC will think that it is an additional input
|
||||
for the linking phase and your DAML file should start with:
|
||||
|
||||
```
|
||||
{-# LANGUAGE DamlSyntax #-}
|
||||
```
|
||||
|
||||
You will get compile errors after the parse tree has been emitted since the standard library is missing
|
||||
but if you just want to see the output from template desugaring, this is sufficient.
|
||||
|
||||
## How to build `ghc-lib` from the DA GHC fork
|
||||
(You don't need to follow the previous step in order to do this.)
|
||||
|
||||
These instructions detail how to generate `ghc-lib-parser` and `ghc-lib` packages intended for use by `damlc`.
|
||||
|
||||
1. Generate `ghc-lib-parser.cabal` by running:
|
||||
```bash
|
||||
mkdir -p ~/tmp && cd ~/tmp
|
||||
git clone git@github.com:digital-asset/ghc-lib.git
|
||||
cd ghc-lib && git clone https://gitlab.haskell.org/ghc/ghc.git
|
||||
cd ghc
|
||||
git remote add upstream git@github.com:digital-asset/ghc.git
|
||||
git fetch upstream
|
||||
git checkout `git merge-base upstream/da-master master`
|
||||
git merge --no-edit upstream/da-master upstream/da-unit-ids
|
||||
git submodule update --init --recursive
|
||||
cd ..
|
||||
stack setup > /dev/null 2>&1
|
||||
stack build --no-terminal --interleaved-output
|
||||
stack exec -- ghc-lib-gen ghc --ghc-lib-parser
|
||||
```
|
||||
Note that the `git checkout` step will put you in detached HEAD state - that's expected.
|
||||
|
||||
The equivalent 8.8.1 commands are:
|
||||
```
|
||||
mkdir -p ~/tmp && cd ~/tmp
|
||||
git clone git@github.com:digital-asset/ghc-lib.git
|
||||
cd ghc-lib
|
||||
git checkout ghc-8.8.1-rc1
|
||||
git clone https://gitlab.haskell.org/ghc/ghc.git
|
||||
cd ghc
|
||||
git fetch --tags
|
||||
git checkout ghc-8.8.1-rc1
|
||||
git remote add upstream git@github.com:digital-asset/ghc.git
|
||||
git fetch upstream
|
||||
git merge --no-edit upstream/da-master-8.8.1 upstream/da-unit-ids-8.8.1
|
||||
git submodule update --init --recursive
|
||||
cd ..
|
||||
stack setup > /dev/null 2>&1
|
||||
stack build --no-terminal --interleaved-output
|
||||
stack exec -- ghc-lib-gen ghc --ghc-lib-parser
|
||||
```
|
||||
|
||||
2. Edit `~/tmp/ghc-lib/ghc/ghc-lib-parser.cabal` to (a) change the version number (we use a datestamp, e.g. `0.20190219`) and (b) add clause `extra-libraries:ffi` to the `library` stanza. Then run:
|
||||
```bash
|
||||
cat << EOF >> stack.yaml
|
||||
- ghc
|
||||
EOF
|
||||
stack sdist ghc --tar-dir=.
|
||||
```
|
||||
This creates `~tmp/ghc-lib/ghc-lib-parser-xxx.tar.gz` where `xxx` is the version number.
|
||||
|
||||
3. Generate `ghc-lib.cabal` by running:
|
||||
```bash
|
||||
git checkout stack.yaml
|
||||
(cd ghc && git clean -xf && git checkout .)
|
||||
stack exec -- ghc-lib-gen ghc --ghc-lib
|
||||
```
|
||||
|
||||
4. Edit `~/tmp/ghc-lib/ghc/ghc-lib.cabal` to (a) change the version number (we use a datestamp, e.g. `0.20190219`), (b) change the `ghc-lib-parser` version number in the `build-depends` stanza and (c) add clause `extra-libraries:ffi` to the `library` stanza. Then run:
|
||||
```
|
||||
stack sdist ghc --tar-dir=.
|
||||
```
|
||||
This creates `~tmp/ghc-lib/ghc-lib-xxx.tar.gz` where `xxx` is the version number.
|
||||
|
||||
5. You can (optionally) test that `ghc-lib-parser` and `ghc-lib` sdists build with these commands:
|
||||
```bash
|
||||
tar xvf ghc-lib-parser-xxx.tar.gz
|
||||
tar xvf ghc-lib-xxx.tar.gz
|
||||
mv ghc-lib-parser-xxx ghc-lib-parser
|
||||
mv ghc-lib-xxx ghc-lib
|
||||
sed '$d' stack.yaml > stack.yaml.tmp&&cp stack.yaml.tmp stack.yaml
|
||||
cat << EOF >> stack.yaml
|
||||
- ghc-lib-parser
|
||||
- ghc-lib
|
||||
EOF
|
||||
stack build ghc-lib-parser --no-terminal --interleaved-output
|
||||
stack build ghc-lib --no-terminal --interleaved-output
|
||||
```
|
||||
where, as in steps 3 and 4, `xxx` is the version number.
|
||||
|
||||
## How to test `ghc-lib`
|
||||
Once you've [built `ghc-lib`](#how-to-build-ghc-lib-from-the-da-ghc-fork), you should test it locally:
|
||||
|
||||
1. Get the SHAs of the tar.gz files. If you followed the last step in the prerequsites, you can do this by running:
|
||||
```
|
||||
sha256sum ghc-lib-parser-xxx.tar.gz
|
||||
sha256sum ghc-lib-xxx.tar.gz
|
||||
```
|
||||
where as before, `xxx` is the version number.
|
||||
|
||||
2. At the root of the `daml` repo, edit `WORKSPACE` (determines where Bazel gets `ghc-lib` from).
|
||||
Update the lines for `ghc-lib-parser` and `ghc-lib` with the new `url`s, `stripPrefix`s and `sha256s`:
|
||||
```bash
|
||||
("ghc-lib-parser", {"url": "file:///path/to/the/ghc-lib-parser-xxx.tar.gz", "stripPrefix": "ghc-lib-parser-xxx", "sha256": "a422c86eaf6efe7cec8086b1b0f361355d4415825cf0513502755736a191ab44"})
|
||||
, ("ghc-lib", {"url": "file:///path/to/the/ghc-lib-xxx.tar.gz", "stripPrefix": "ghc-lib-xxx", "sha256": "d422c86eaf6efe7cec8086b1b0f361355d4415825cf0513502755736a191ab66"})
|
||||
```
|
||||
3. Check that the DAML tests pass:
|
||||
```bash
|
||||
bazel run //compiler/damlc:daml-ghc-test -- --pattern=
|
||||
```
|
||||
If they pass, you can move on to [deploying](#how-to-deploy-ghc-lib).
|
||||
|
||||
## How to deploy `ghc-lib`
|
||||
Now you've [built](#how-to-build-ghc-lib-from-the-da-ghc-fork) and [tested `ghc-lib`](#how-to-test-ghc-lib), you can deploy it:
|
||||
|
||||
1. Upload `ghc-lib-parser-xxx.tar.gz` and `ghc-lib-xxx.tar.gz` to [bintray](https://bintray.com/digitalassetsdk/ghc-lib) with commands like the following
|
||||
```bash
|
||||
API_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx;export API_KEY
|
||||
|
||||
# Upload commands are formed as follows.
|
||||
# curl -T <FILE.EXT> -ushayne.fletcher@digitalassetsdk:<API_KEY> \
|
||||
# https://api.bintray.com/content/digitalassetsdk/ghc-lib/<YOUR_COOL_PACKAGE_NAME>/<VERSION_NAME>/<FILE_TARGET_PATH>
|
||||
curl -T /path/to/ghc-lib-parser-xxx.tar.gz \
|
||||
-ushayne.fletcher@digitalassetsdk:$API_KEY \
|
||||
https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/ghc-lib-parser-xxx.tar.gz
|
||||
curl -T /path/to/ghc-lib-xxx.tar.gz \
|
||||
-ushayne.fletcher@digitalassetsdk:$API_KEY \
|
||||
https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/ghc-lib-xxx.tar.gz
|
||||
curl -X POST -ushayne.fletcher@digitalassetsdk:$API_KEY \
|
||||
https://api.bintray.com/content/digitalassetsdk/ghc-lib/da-ghc-lib/xxx/publish
|
||||
```
|
||||
(where `API_KEY` is replaced by your bintray `API_KEY` which you can retrieve by looking into your bintray profile).
|
||||
|
||||
2. In the `daml` repo, create a new feature branch:
|
||||
```bash
|
||||
cd daml
|
||||
git checkout -b update-ghc-lib
|
||||
```
|
||||
3. Edit `WORKSPACE` _again_ with the `url`s pointing to bintray this time:
|
||||
```bash
|
||||
# Download commands are formed as follows.
|
||||
# curl -L "https://digitalassetsdk.bintray.com/ghc-lib/<FILE_PATH>" -o <FILE.EXT>
|
||||
# Example download URL - https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-xxx.tar.gz.
|
||||
("ghc-lib-parser", {"url": "https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-parser-0.20190401.1.tar.gz", "stripPrefix": "ghc-lib-parser-0.20190401.1", "sha256": "3036ed084ca57668faab25f8ae0420a992e21ad484c6f82acce73705dfed9e33"})
|
||||
, ("ghc-lib", {"url": "https://digitalassetsdk.bintray.com/ghc-lib/ghc-lib-0.20190401.1.tar.gz", "stripPrefix": "ghc-lib-0.20190401.1", "sha256": "82e94f26729c35fddc7a3d7d6b0c89f397109342b2c092c70173bb537af6f5c9"})
|
||||
```
|
||||
4. If you didn't do this before, make sure the DAML tests pass by running:
|
||||
```bash
|
||||
bazel run //compiler/damlc:daml-ghc-test -- --pattern=
|
||||
```
|
||||
5. When the tests pass, push your branch to origin and raise a PR.
|
||||
|
||||
## How to rebase `ghc-lib` on upstream master
|
||||
|
||||
To keep `ghc-lib` consistent with changes to upstream GHC source code, it is neccessary to rebase our branches on the upstream `master` from time to time. The procedure for doing this is as follows:
|
||||
```bash
|
||||
mkdir -p ~/tmp && cd ~/tmp
|
||||
git clone git@github.com:digital-asset/ghc.git
|
||||
cd ghc
|
||||
git remote add upstream https://gitlab.haskell.org/ghc/ghc.git
|
||||
git fetch upstream master
|
||||
# These checkout commands take into account that `da-master` is the "default" branch.
|
||||
git checkout -t origin/master && git merge upstream/master
|
||||
git checkout da-master && git rebase master
|
||||
git checkout -t origin/da-unit-ids && git rebase master
|
||||
```
|
||||
Obviously, you will need to deal with any rebase conflicts that come up (hopefully not often). You can test `ghc-lib` after rebasing by following the [build procedure](#how-to-build-ghc-lib-from-the-da-ghc-fork) replacing the line
|
||||
```bash
|
||||
git remote add upstream git@github.com:digital-asset/ghc.git
|
||||
```
|
||||
with
|
||||
```bash
|
||||
git remote add upstream $HOME/tmp/ghc
|
||||
```
|
||||
and then the [test procedure](#how-to-test-ghc-lib).
|
||||
|
||||
When you are satisfied that the tests pass, you can push the changes to origin with these commands:
|
||||
```bash
|
||||
cd ~/tmp/ghc
|
||||
git push origin master:master
|
||||
git push -f origin da-master:da-master
|
||||
git push -f origin da-unit-ids:da-unit-ids
|
||||
```
|
||||
After this, release the updated `ghc-lib` following the usual [deployment procedure](#how-to-deploy-ghc-lib).
|
||||
|
||||
## How to develop `ghc-lib`
|
||||
|
||||
The following procedure sets up a new feature branch with starting point `da-master`.
|
||||
```bash
|
||||
mkdir ~/tmp && cd ~/tmp
|
||||
git clone https://gitlab.haskell.org/ghc/ghc.git ghc.git
|
||||
cd ghc.git
|
||||
git remote add upstream git@github.com:digital-asset/ghc.git
|
||||
git fetch upstream da-master
|
||||
git checkout -t upstream/da-master
|
||||
git checkout -b feature-xxx da-master
|
||||
git push upstream feature-xxx:feature-xxx
|
||||
```
|
||||
where `feature-xxx` is replaced by the desired name of your feature branch.
|
||||
|
||||
To prepare to produce a `ghc` from your feature branch, remember to first initialize submodules and build hadrian's dependencies (hadrian itself will be built on the first ghc build invocation).
|
||||
```
|
||||
git submodule update --init --recursive
|
||||
stack build --stack-yaml=hadrian/stack.yaml --only-dependencies
|
||||
```
|
||||
To build `ghc` invoke hadrian via `hadrian/build.stack.sh`.
|
||||
```bash
|
||||
hadrian/build.stack.sh --configure --flavour=quickest -j
|
||||
```
|
||||
As usual, the compiler is built to `_build/stage1/bin/ghc`.
|
||||
|
||||
When you are ready to publish your feature branch, push to `upstream` and raise your PR with base `da-master`.
|
Loading…
Reference in New Issue
Block a user