catala/tests
2024-06-27 14:00:11 +02:00
..
arithmetic Message formatting: some more space for results 2024-05-03 15:27:06 +02:00
array Types printer: use commas rather than * for tuples 2024-06-21 12:16:49 +02:00
backends Generate tests reports from 'clerk test' 2024-06-19 16:10:26 +02:00
bool More space between multiple errors, no numbers if just one 2024-06-20 11:15:53 +02:00
date Message formatting: some more space for results 2024-05-03 15:27:06 +02:00
dec/good Message formatting: some more space for results 2024-05-03 15:27:06 +02:00
default Refactor and improve suggestions' pretty-printing 2024-06-21 11:11:53 +02:00
enum Refactor and improve suggestions' pretty-printing 2024-06-21 11:11:53 +02:00
exception Message formatting: some more space for results 2024-05-03 15:27:06 +02:00
func Update test outputs 2024-06-21 12:23:01 +02:00
io Message formatting: some more space for results 2024-05-03 15:27:06 +02:00
literate/good Workaround pandoc version changing the result of some tests 2024-06-27 14:00:11 +02:00
metadata/good Message formatting: some more space for results 2024-05-03 15:27:06 +02:00
modules Replace module hashes for external modules 2024-05-28 11:43:50 +02:00
money Message formatting: some more space for results 2024-05-03 15:27:06 +02:00
name_resolution Fix error message typo 2024-06-05 15:19:12 +02:00
parsing/bad Refactor and improve suggestions' pretty-printing 2024-06-21 11:11:53 +02:00
proof Message formatting: some more space for results 2024-05-03 15:27:06 +02:00
scope Refactor and improve suggestions' pretty-printing 2024-06-21 11:11:53 +02:00
struct Message formatting: some more space for results 2024-05-03 15:27:06 +02:00
tuples/good Types printer: use commas rather than * for tuples 2024-06-21 12:16:49 +02:00
typing Add error messages numbering (#634) 2024-06-20 15:41:55 +02:00
variable_state Message formatting: some more space for results 2024-05-03 15:27:06 +02:00
.ocamlformat Forbid reformatting in tests/ 2024-05-30 17:09:13 +02:00
README.md Tests: add catala test-scope pseudo-command for inline tests 2024-02-26 14:56:43 +01:00

Catala test suite

This folder contains Catala source files designed to test the features of the language.

Tests are declared inside plain Catala files with the following format:

```catala-test-inline
$ catala ARGS
... output from the catala command ...
```

In addition to the normal Catala commands, you can write catala test-scope SCOPE. This will behave like catala Interpret -s SCOPE, but allows to further customise the flags applied to it (so that the same test can be used to check with optimisations enabled or disabled, with the lcalc interpreter, and so on)

Workflow for adding new tests

  1. Create a new test file in foo/{good,bad}/bar.catala_<language> (pick the right directory and an informative name for your test)
  2. Write your test, and pick a toplevel scope A to run.
  3. Add the following section to your file:
    ```catala-test-inline
    $ catala test-scope A
    ```
    
  4. Run clerk test tests/foo/{good,bad}/bar.catala_<language> --reset from the root of the Catala repository. This will update the test with the actual output of the catala command.
  5. Don't forget to git add the test file.

Workflow for fixing regressions

  1. Run clerk test from the root of the Catala repository, if a test fails you should see something like FAILED: foo/{good,bad}/bar.catala_<language> followed by the diff between the expected output and the current output of the command.
  2. Debug the compiler and the test and repeat. Run clerk test tests/foo/{good,bad}/bar.catala_<language> to check the test again.
  3. When you're finished debugging, if you are positive that a change in the test output is legitimate, record the new outputs with clerk test tests/foo/{good,bad}/bar.catala_<language> --reset.
  4. Re-run make tests from the root of the Catala repository to check that everything passes.
  5. Run git diff to double-check your changes to the test outputs are expected. If necessary, justify them in your commit message.

If a compiler change causes a lot of regressions (error message formatting changes for instance), you can mass-reset the expected outputs with clerk test --reset. git diff will then allow to check all the changes at once. Caution: It's your responsability to check all the changes before committing them.

Tips

  • Running a single test-file just to check changes when tweaking either the compiler or the test file itself, but without updating or diffing with the reference can be useful when debugging. The following command outputs the result to stdout and can be used from within text editors:

    clerk runtest test-file.catala_en
    # Or, to use the current build artefacts, wrap with `dune exec`:
    dune exec -- clerk runtest test-file.catala_en