cryptol/CONTRIBUTING.md

# Cryptol development

This document explains our standards for developing Cryptol. Our goals
are to have a development process that:

- Consistently yields reliable software artifacts
- Quickly incorporates improvements and gets them into user hands
- Allows new contributors to have an immediate impact

It describes our methods and practices for:

- Testing and continuous integration
- Organizing, branching, and merging this repository
- Producing and publishing release artifacts
- **TODO**: documentation
- **TODO**: feature/release planning, ticket assignment, etc

This is a living document that is not (and possibly cannot be)
comprehensive. If something is missing or unclear, or if you have
suggestions for improving our processes, please file an issue or open
a pull request.

# Testing

Cryptol primarily uses golden testing on the Cryptol interpreter
executable. These tests provide the interpreter with input and then
check the output against an expected output file. We make at least one
test for each new issue, and keep the accumulated tests in our suite
as regression tests. The test suite itself is written using the
`test-framework` library, so it can readily output XML for consumption
by Jenkins and other CI systems.

## Stackage

On the Jenkins machines, we `cp cabal.GHCNNN.config cabal.config`
before building in order to build against a Stackage LTS snapshot
(updated periodically). This is to ensure compatibility with
downstream dependencies that rely on Stackage for their stability. We
do not have `cabal.config` in place by default, though, so developers
can use different versions of the compiler.

## Running tests

To run the test suite, run `make test` from the root of the
repository. By default, you'll get output on the console for each test
that fails, either with an explanation for why it failed, or a command
line you can paste in order to compare the test results against the
expected output.

The `make test` target invokes the `cryptol-test-runner` executable,
which is defined in the `/tests/` directory. It is invoked with the
location of the `cryptol` executable, an output directory, and
standard `test-framework` command line arguments. The `test` target in
the `Makefile` provides a template for how to invoke it if you need to
use advanced parameters.

## Running benchmarks

To run the benchmark suite, run `make bench` from the root of the
repository. By default, you'll get Criterion output on the console
from the benchmarking executable in `/bench`. The easiest way to pass
custom arguments to this executable is to run the suite subsequently
with `cabal bench --benchmark-options="..."`.

## Creaing a new test

A test consists at minimum of an `.icry` file, which is a batch-mode
file for the interpreter, and an `.icry.stdout` file, which contains
expected output (the "golden" file). As opposed to `.cry` Cryptol
source files, `.icry` files are run by the interpreter line-by-line as
if a user has typed each one in and pressed Enter.

Frequently, one creates an `.icry` file by interactively producing a
desired behavior in the interpreter, and then copying the relevant
lines of input into the file. Remember that, as with unit testing,
golden testing will only test the examples you give it, so make sure
your examples exercise many instances and corner cases of the bug or
feature.

## Expected test failures

We try to keep as few failing tests as possible in the `master`
branch. Usually tests for new features are merged into the `master`
branch in a working state. However if a new bug is reported, we often
write tests for it before it is fixed, particularly if it will take
some time to implement the fix.

To prevent confusion over which tests ought and ought not to be
failing, we add an `.icry.fails` file with an explanatory message
alongside the `.icry` script that defines the test. This will usually
reference an issue number, so that anyone running the test suite will
understand that the reason for the failure is not _their_ changes, but
rather a known issue that is already being handled.

### Example

Issue #6 was a feature request to add let-binding to the
interpreter. @dylanmc gave an example of the input he wanted to be
able to enter, so we created a file `/tests/issues/issue006.icry`
with the contents:

    :let timesTwo x = x * 2
    :let double x = x + x
    :prove \x = timesTwo x == double x

We might not yet know what the expected output should be, so we
created a dummy file `/tests/issues/issue006.icry.stdout`:

    TODO: once implemented, do something sensible here

Since this is not the output we got when running the `.icry` file,
this was now a failing test. To prevent confusion, we marked that it
was expected to fail by putting creating a
`/tests/issues/issue006.icry.fails` file with a reference to the
issue:

    In development, see issue #6

As the issue progressed and we refined the design, @acfoltzer
implemented the `let` feature and came up with some additional
examples that stretch the legs of the feature further, so we updated
our `.icry` file, this time loading a supplemental `.cry` file so we
could test behavior within a module context.

`issue006.cry`:

    f : [32] -> [32]
    f x = x + 2

    g : [32] -> [32]
    g x = f x + 1

`issue006.icry`:

    :l issue006.cry
    g 5
    let f x = 0
    g 5
    (f : [32] -> [32]) 5
    let f x = if (x : [32]) == 0 then 1 else x * (f (x - 1))
    f 5
    let h x = g x
    h 5

Since the feature was now implemented, we could also produce expected
output. The easiest way to do this is to interpret the `.icry` file
using the `-b` flag outside of the test runner, see if the results
look as expected, and then save those results as the new
`.icry.stdout`:

    # start with a fresh build
    % make
    ...
    # tests are run from within the directory of the .icry file
    % cd tests/issues
    % ../../.cabal-sandbox/bin/cryptol -b issue006.icry
    Loading module Cryptol
    Loading module Cryptol
    Loading module Main
    0x00000008
    0x00000008
    0x00000000
    0x00000078
    0x00000008

At this point, it's very important to compare the results you see
against the results you expect from the inputs in the `.icry`
script. Since the results looked correct, we piped the same command
into the matching `.icry.stdout` file and removed the `.icry.fails`
file:

    % ../../.cabal-sandbox/bin/cryptol -b issue006.icry.stdout
    % rm issue006.icry.fails

Now the test case `issue006` passes, and will (hopefully!) break if
the let-binding feature breaks.

# Repository organization and practices

The top-level repository directories are:

- `/bench`: Benchmarking executable and suite
- `/cryptol`: Haskell sources for the front-end `cryptol` executable
  and read-eval-print loop
- `/cryptol-server`: Experimental Cryptol JSON-over-ZeroMQ server,
  currently built to support
  [pycryptol](http://pycryptol.readthedocs.org/en/latest/)
- `/docs`: LaTeX and Markdown sources for the Cryptol documentation
- `/examples`: Cryptol sources implementing several interesting
  algorithms
- `/lib`: Cryptol standard library sources
- `/src`: Haskell sources for the `cryptol` library (the bulk of the
  implementation)
- `/tests`: Haskell sources for the Cryptol regression test suite, as
  well as the Cryptol sources and expected outputs that comprise that
  suite
- `/win32`: Support files for building the Windows installer

## Branching and merging

Within the `GaloisInc/cryptol` repository, we use the
[git-flow model](http://nvie.com/posts/a-successful-git-branching-model/)
for branches and merging. Our version has two notable differences:

1. Our `master` (rather than `develop`) branch serves as the cutting
   edge development branch, and our `release` (rather than `master`)
   branch is where only pristine, tagged releases are committed.

2. We use `wip` (work-in-progress) branches as a centralized storage
   place for (usually individual) work in progress. Whereas a
   `feature` branch is expected to be relatively stable, all bets are
   off with a `wip` branch. Typically `wip` branches are not actually
   merged directly into `master`, but instead are rebased into a new
   branch where the history is cleaned up before merging into
   `master`.

In short:

- Any substantial new features should be developed on a branch
  prefixed with `feature/`, and then merged into `master` when
  completed.
- When we reach a feature freeze for a release, we create a new branch
  prefixed with `release/`, for example `release/2.1.0`. When the
  release is made, we merge those changes back into `master` and make
  a snapshot commit on the `release` branch.
- If a critical bug emerges in already-released software, we create a
  branch off of the relevant `release` branch commit prefixed with
  `hotfix/2.1.1`. When the hotfix is complete, we merge those changes
  back into `master` and make a snapshot commit on the `release`
  branch.

# Releases

We take the stability and reliability of our releases very
seriously. To that end, our release process is based on principles of
_automation_, _reproducibility_, and _assurance_.

Automation is essential for reducing the possibility of human
error. The checklist for a successful release is fairly lengthy, and
most of the steps need not be done by hand. The real points of
judgment for an individual release are deciding _when_ the codebase is
ready to be released, not _how_ it is released.

Reproducibility is essential for fixing bugs both in hotfixes and
future mainline development. If we cannot reproduce the circumstances
of a release, we might not be able to reproduce bugs that are reported
by users of that release. Bugs are often very particular about the
environment where they will arise, so it is critical to make the
environment of a release consistent.

Assurance is crucial due to the nature of development done with
Cryptol. When people use Cryptol to develop the next generations of
trustworthy systems, we want them to be sure the software was built by
the Cryptol developers, and was not corrupted during download or
replaced by a malicious third party. To this end, we sign our releases
with a [GPG key](http://www.cryptol.net/files/Galois.asc).

## Cutting releases

**TODO**: make this relevant to folks outside Galois; right now the
 build farm exists within the Galois network only, and Galois also
 controls the release signing key.

The release process is:

1. Make sure the `release/n.n.n` branch is in a release/ready state,
   with successful build artifacts across all platforms on the
   relevant Jenkins job. **TODO**: get a Jenkins job running per
   release branch, rather than just `master`.
1. Merge the `release/n.n.n` branch into the pristine `release` branch
   and add a git tag.
1. Merge the `release/n.n.n` branch back into `master` for future
   development, and delete the `release/n.n.n` branch.
1. Run the `cryptol-release` Jenkins job to create a draft
   release. Specify the build number with the successful artifacts,
   the textual version tag (e.g., "2.1.0"), whether it's a prerelease
   (e.g., an alpha), and keep the `DRAFT` option checked.
1. On the Github page for the draft release and add a changelog
   (**TODO**: automate changelogs?).
1. (**TODO**: this part of the process needs to be better and
   automated) Download the successfully-built artifacts _from
   Jenkins_, and in the same directory run the script
   `/release-infrastructure/sign.sh` from the `cryptol-internal.git`
   repository. You must have the correct GPG key (D3103D7E) for this
   to work.
1. Upload the `.sig` files to the draft release on Github.
1. Publish the release and announce it (**TODO**: compile mailing lists)

    - <http://www.cryptol.net> (in the `cryptol2-web.git` repo)
    - <cryptol-team@lists.galois.com>
    - <cryptol-users@community.galois.com>
    - @galois on Twitter (for major releases)
    - TODO: more?