CatalaLang/catala
1
1
Fork 0
mirror of https://github.com/CatalaLang/catala.git synced 2024-09-19 16:28:12 +03:00
Code Issues Projects Releases Wiki Activity

Update doc for handling tests

Browse Source
This commit is contained in:
Louis Gesbert 2022-07-11 16:03:47 +02:00
parent 1d268a940b
commit d2a182043b
2 changed files with 36 additions and 27 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
Makefile
View File

@ -292,6 +292,9 @@ bench_js: run_french_law_library_benchmark_js
#> bench_python : Run Python benchmarks for the Catala-generated code
bench_python: run_french_law_library_benchmark_python
tests/%: .FORCE
@$(MAKE) -C tests $*
##########################################
# Website assets
##########################################

60
tests/README.md
View File

@ -3,47 +3,53 @@
This folder contains Catala source files designed to test the features of the
language.
Expected outputs are stored using the convention
`<name_of_test>.catala_<language>.<name_of_scope>.Interpret` in the corresponding test output folder.
Tests are declared inside plain Catala files with the following format:
For both workflows: use `CATALA_OPTS="..." make ...` to pass in Catala compiler
options when debugging.
~~~markdown
```catala-test {id="TEST-IDENT"}
catala ARGS
```
~~~
The output of running catala with the given `ARGS` for a `foo.catala_en` file is
expected to be in an `output/foo.catala_en.TEST-IDENT` file (use
`CLERK_OPTS=--reset` to (re-)generate it).
You can use `CATALA_OPTS="..." make ...` to pass in Catala compiler options when
debugging.
## Workflow for adding new tests
1. Create a new test file in `foo/{good,bad}/bar.catala_<language>` (pick the right directory and
an informative name for your test)
2. Write your test, and pick a toplevel scope `A` to run.
3. From this directory, create `mkdir -p foo/{good,bad}/output` and the empty expected
output file `touch foo/{good,bad}/output/bar.catala_<language>.A.Interpret`.
4. From the root of the Catala repository, Run `make -C tests foo/{good,bad}/bar.catala_<language>` to get the output of
your test (compared with the empty file). The failing test will give you the command to use to reproduce
the fail from the `tests` directory.
5. When you're happy with the output, launch `CLERK_OPTS=--reset make -C tests foo/{good,bad}/bar.catala_<language>`
3. Add the following section to your file:
~~~markdown
```catala-test {id="A.Interpret"}
catala Interpret -s A
```
~~~
4. Run `CLERK_OPTS=--reset make tests/foo/{good,bad}/bar.catala_<language>`
from the root of the Catala repository. This
will record the content of the output of your test into a file.
6. Check that your test pass with `make -C tests foo/{good,bad}/bar.catala_<language>`
from the root of the Catala repository.
7. That's it, you've added a new test for the Catala language!
will record the content of the output of your test into the target file.
5. Check that the expected output is present in `tests/foo/{good,bad}/output/bar.catala_<language>.A.Interpret`
6. Don't forget to `git add` both the test file and the output file.
## Workflow for fixing regressions
1. Run `make test_suite` from the root of the Catala repository,
if a test fails you should see something like
`[ERROR] Test failed: foo/{good,bad}/output/bar.catala_<language>.A.Interpret]`.
2. Compare the computed and expected output with `make -C tests foo/{good,bad}/bar.catala_<language>`
from the root of the Catala repository.
3. Debug the compiler and/or the test, running `make -C tests foo/{good,bad}/bar.catala_<language>`
from the root of the Catala repository periodically to check the output of Catala
on the test case.
4. When you're finished debugging, record the new test output with
`CLERK_OPTS=--reset make foo/{good,bad}/bar.catala_<language>`.
5. Re-run `make test_suite` from the root of the Catala repository
to check that everything passes.
6. That's it, you've fixed the Catala test suite to adapt for changes in the
language.
2. Compare the computed and expected output with `make tests/foo/{good,bad}/bar.catala_<language>`
from the root of the Catala repository. Debug the compiler and the test and repeat.
3. When you're finished debugging, if you are positive that a change in the test
output is legitimate, record the new outputs with `CLERK_OPTS=--reset make
tests/foo/{good,bad}/bar.catala_<language>`.
4. Re-run `make test_suite` from the root of the Catala repository to check that
everything passes.
5. Run `git diff` to double-check your changes to the test outputs are expected.
If necessary, justify them in your commit message.
If a compiler change causes a lot of regressions (error message formatting changes
for instance), you can mass-reset the expected the outputs with `CLERK_OPTS=--reset make test_suite`.
**Caution**: use at your own risk, regressions should be fixed one by one in
general.
**Caution**: It's your responsability to check all the changes before committing them.