3## Running the test suite
5Do `make test_c` or `make test_js`. It will run the tests in parallel and through `cabal test`.
6You can also manually call `cabal test` or `stack test`.
10#### To the test program
12You can pass arguments to the test program. For example, to pass the `--help` argument:
16make TEST-ARGS="--help" test_c
18cabal test --test-options="--help"
20stack test --test-arguments="--help"
23Try it to learn more about the other arguments you can provide. Two are of particular interest: `--node` to test against the Node code generator and `--pattern <regex>` to only run the test that match the provided `<regex>`.
27You can pass arguments to idris in each of its invocation by the tests. There are two ways to this. You can either modify the `idrisFlags` term in `TestRun`, or set the `$IDRIS` environment variable accordingly
29### Specifying the number of parallel jobs
31With make, the test suite runs in parallel by default. You can specify the number of threads with `TEST-JOBS`. For stack and cabal, you need to explicitly enable parallelism as you would do with any other GHC-compiled executable. Examples:
34# Two test jobs via make
35make TEST-JOBS=2 test_c
36# Two test jobs via cabal
37cabal test --test-options="+RTS -N2 -RTS"
38# Two test jobs via stack
39stack test --test-arguments="+RTS -N2 -RTS"
42### Running only previously failed tests
44Because of the `--rerun-update` option, `make test_c` will create a `.tasty-rerun-log` file in the root directory of the project. Each time the test suite is run, the file will be written with the result of the tests. The next time you do `make test`, you can specify the `rerun-filter` argument to, for example, only run previously failed tests. Valid values are given in the `--help`.
46## Extending the test suite
48### Content of the directory
50To each test there shall be a folder that holds at least a bash script `run` and a golden file `expected`. When running the test suite, the standard output of the script is compared against the golden file. Any mismatch is reported as a failure.
52The name of a test folder is the identifier of the *test family* followed by a three digit number. A test family gathers related tests.
54Add to that four top level files:
56- The program `TestRun.hs` that runs the test suite
57- A Haskell module `TestData.hs` with the metadata for each test
58- The Perl script `mktest.pl` to ease the creation of a new test
63Tests are categorised with the following test families (and their identifiers):
65+ *basic*: Basic language features, some complete programs
66+ *bignum*: Bignums and GMP
67+ *bounded*: Bounds testing for bits
68+ *corecords*: Tests for corecord
69+ *delab*: De-elaboration tests
70+ *directives*: Tests for directives
71+ *disambig*: Disambiguaton tests
72+ *docs*: Testing documentation functionality
73+ *dsl*: Embedded DSLs and features to support DSL development
74+ *effects*: Effects package
75+ *error*: Error messages and error reflection
76+ *ffi*: FFI calls, including type providers
77+ *folding* Testing some folds
78+ *idrisdoc*: Documentation tool functionality
79+ *interactive*: Interactive editing, proof search
80+ *interfaces*: Testing interfaces.
82+ *literate*: `.lidr` files; literate programming
83+ *meta*: Meta-programming
85+ *primitives*: Primitive types
86+ *proof*: Theorem proving, tactics
87+ *proofsearch*: Proof search
88+ *pruviloj*: Elaborator reflection
89+ *quasiquote*: Quasiquotations
91+ *reg*: Regression tests, covering previous bug fixes
92+ *regression*: Regression tests, covering previous bug fixes
93+ *sourceLocation*: Interaction with files from Idris
94+ *sugar*: Syntactic sugar, syntax extensions
95+ *syntax*: Syntax extensions
96+ *tactics*: Testing for tactics
97+ *totality*: Totality checking
98+ *tutorial*: Examples from the tutorial
99+ *unique*: Uniqueness types
100+ *universes*: Universes
101+ *views*: Views and covering functions
103### Adding a test family
1051. Choose an available identifier for your test family. It should be short
106and somewhat self-explanatory.
1072. Add it in the `README.md` file, i.e. just above.
1083. Add it in `TestData.hs`, in the `testFamiliesData` list. Say the chosen identifier is `foo`. You should append to the list `("foo", "A proper name for foo", [ ])`. The empty list will be replaced with the metadata of each test in the family.
1131. Choose the family your test shall belong to and remember its identifier.
1142. Pick the next available integer in the test family. It will be the index.
1153. Say the family's identifier is `foo` and the index is `42`. You should call `./mktest.pl foo042` ; this will create the directory and a simple `run` script.
1164. Modify the `run` script to your liking. If you want to call the idris executable, write `${IDRIS:-idris} $@`.
1175. Add any file you may need in the directory. If they don't end in `.idr`, you should remember them for the next step.
1186. Add your test in `TestData.hs`. Each family has a list of `(Index, CompatCodegen)`. See the next section for the available values in `CompatCodegen`. In most cases, you should write `( 42, ANY)`.
1197. Generate the `expected` file by doing:
122 cabal test --test-options="--pattern foo042 --accept"
124 stack test --test-arguments="--pattern foo042 --accept"
1268. Check the content of `expected`. Maybe the test didn't do what you thought it would. Fix and go back to 7 until it's ok.
128### Specifying compatible backends
130Some tests only make sense with specific code generators. Others don't generate code. You need to supply this information in `testFamiliesData`. Available values
131for the compatible backends are:
133- `ANY`: choose this if your test will work with any code generator
134- `C_CG`: choose this to only test against the C code generator
135- `NODE_CG`: choose this to only test against the Node code generator
136- `NONE`: choose this if you don't want your test to be run with multiple code
137generators (mainly for tests that only perform type checking)
139Currently, `NONE` has the same effect as `ANY`, but this will change.
1431. Delete the corresponding directory
1442. Remove the corresponding line in the definition of `testFamiliesData` in `TestData.hs`
146### Updating golden files
148To update the `expected` file for every test, do one of the following:
154cabal test --test-options="--accept"
156stack test --test-arguments="--accept"
159"Accepted" tests are the ones that update the golden file. A test can still fail if the `run` script itself crashes.
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list