2023-07-18 17:00:53 +03:00
|
|
|
(* This file is part of the Catala build system, a specification language for
|
2023-08-25 15:12:02 +03:00
|
|
|
tax and social benefits computation rules. Copyright (C) 2022-2023 Inria,
|
2023-07-18 17:00:53 +03:00
|
|
|
contributors: Louis Gesbert <louis.gesbert@inria.fr>
|
|
|
|
|
|
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not
|
|
|
|
|
use this file except in compliance with the License. You may obtain a copy of
|
|
|
|
|
the License at
|
|
|
|
|
|
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
|
|
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
|
|
|
distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
|
|
|
|
|
WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
|
|
|
|
|
License for the specific language governing permissions and limitations under
|
|
|
|
|
the License. *)
|
|
|
|
|
|
|
|
|
|
(** This module contains specific commands used to detect and run inline tests
|
|
|
|
|
in Catala files. The functionality is built into the `clerk runtest`
|
|
|
|
|
subcommand, but is separate from the normal Clerk behaviour: Clerk drives
|
|
|
|
|
Ninja, which in turn might need to evaluate tests as part of some rules and
|
|
|
|
|
can run `clerk runtest` in a reentrant way. *)
|
|
|
|
|
|
2023-09-26 12:42:46 +03:00
|
|
|
open Catala_utils
|
2023-07-18 17:00:53 +03:00
|
|
|
|
Generate tests reports from 'clerk test'
This is a proper replacement for the previous shell-based placeholder hack.
Here is a summary:
- `clerk runtest` (normally run by ninja) is much extended:
* besides generating the test@out file, it checks individual tests for success
and can write a report file containing their status, and the positions for
their (expected/current) outputs (this uses `Marshal`)
* it now handles out-tests directly in addition to inline-tests, for which
it generates the separate output file ; they are included in the report
- ninja is now tasked with building all the test reports (which shouldn't fail);
for directories, individual reports are concatenated (as before).
Removing intermediate report rules, and out-test rules means that the ninja
file is much simplified.
- then, clerk takes back control, reads the final reports and formats them in a
user-friendly way. Printing the reports may imply running `diff` internally.
In particular, the commands to easily reproduce each test are provided.
Resetting the test results if required is also done directly by clerk, at this
stage.
A few switches are available to customise the output, but I am waiting for some
feedback before deciding what to make available from the CLI.
The `clerk report` command is available to manually explore test reports, but
normally the processing is done directly at the end of `clerk test` (i.e. ninja
will no longer call that command)
2024-06-14 22:05:19 +03:00
|
|
|
val run_tests :
|
|
|
|
|
catala_exe:string ->
|
|
|
|
|
catala_opts:string list ->
|
|
|
|
|
test_flags:string list ->
|
|
|
|
|
report:File.t option ->
|
|
|
|
|
out:File.t option ->
|
|
|
|
|
File.t ->
|
|
|
|
|
unit
|
|
|
|
|
(** [run_tests ~catala_exe ~catala_opts ~test_flags ~report ~out file] runs the
|
|
|
|
|
tests in Catala [file] using the given path to the Catala executable and the
|
|
|
|
|
provided options. Output is printed to [stdout] if [out] is [None]. *)
|