leo/tests/README.md

106 lines
3.3 KiB
Markdown
Raw Normal View History

2021-05-11 21:16:55 +03:00
# Leo Test Framework
This directory includes Leo code samples, which are parsed and used as tests by `test-framework`.
2021-05-11 21:16:55 +03:00
## Structure
Currently, the test framework covers only two areas: compiler and parser; tests for both destinations are placed in
matching folders. The third folder - expectations - contains results of test execution which are saved in git and then
2021-05-11 21:16:55 +03:00
compared to test output.
## Test Structure
Tests can be placed either in the `compiler/` or in the `parser/` directories. Each test is a Leo file with correct (or intentionally
incorrect) Leo code. What makes Leo file a test is a block comment at the top of the file:
2021-05-11 21:16:55 +03:00
```
/*
namespace: Parse
expectation: Pass
*/
2022-09-30 20:48:47 +03:00
struct X {
2021-05-11 21:16:55 +03:00
x: u32,
y: u32,
}
```
This comment contains YAML structure with set of mandatory and optional fields.
## Test Expectations
After an initial run of the tests, test expectations will be autogenerated and placed under the `expectations/` directory.
They will contain the results of execution in detail (for example, in the compiler tests, they include number of constraints and
2021-05-11 21:16:55 +03:00
output registers).
During subsequent test runs, the results of each test are compared to the stored expectations, and if the stored expectations (say, number
of constraints in Pedersen Hash example) don't match actual results, an error will be thrown and the test won't pass. Of course,
2021-05-11 21:16:55 +03:00
there are two possible scenarios:
1. If the test has failed because the logic was changed intentionally, then expectations need to be deleted. New ones will be
generated instead. A PR should contain changes to expectations as well as to tests or code.
2. If the test should pass, then expectations should not be changed or removed.
2021-05-11 21:16:55 +03:00
## Test Configuration
Here is the list of all possible configuration options for compiler and parser tests.
2022-05-23 18:22:11 +03:00
### namespace
2021-05-11 21:16:55 +03:00
2022-05-23 18:22:11 +03:00
```yaml
2021-05-11 21:16:55 +03:00
- Mandatory: yes
- Namespace: all
2022-05-23 18:22:11 +03:00
- Values: ...
2021-05-11 21:16:55 +03:00
```
2022-05-23 18:22:11 +03:00
Several values are supported, but they vary depending on the directory you are in.
Parser Directory namespaces:
2022-05-24 19:45:11 +03:00
- `Parse` - Test a file to check that it is a valid Leo program.
- `ParseExpression` - Test a file line by line to check that each line is a valid Leo expression.
- `ParseStatement` - Test a file consuming multiple lines till a blank line to check that it contains a valid Leo statement.
- `Serialize` - Test a file to check that it can be serialized to JSON.
- `Input` - Test an input file to check that it is a valid Leo input file.
- `Token` - Test a file line by line to check that it contains zero or more valid Leo parser tokens.
2021-05-11 21:16:55 +03:00
2022-05-23 18:22:11 +03:00
Compiler Directory namespaces:
2021-05-11 21:16:55 +03:00
2022-05-24 19:45:11 +03:00
- `Compiler` - Test a file to check that it is a valid Leo program, and it can be compiled without errors.
2021-05-11 21:16:55 +03:00
### expectation
2022-05-23 18:22:11 +03:00
```yaml
2021-05-11 21:16:55 +03:00
- Mandatory: yes
- Namespace: all
- Values: Pass / Fail
```
This setting indicates whether the tested code is supposed to succeed or to fail.
If the test was marked as `Pass` but it actually failed,
you'll know that something went wrong and the test or the compiler/parser needs fixing.
2021-05-11 21:16:55 +03:00
### input_file (Compile)
2022-05-23 18:22:11 +03:00
```yaml
2021-05-11 21:16:55 +03:00
- Mandatory: no
- Namespace: Compile
- Values: <input file path>, ...
```
This setting allows using one or more input files for the Leo program.
The program will be run with every provided input.
2021-05-11 21:16:55 +03:00
See this example:
2022-05-23 18:22:11 +03:00
```yaml
2021-05-11 21:16:55 +03:00
/*
namespace: Compile
expectation: Pass
input_file:
2021-05-11 21:16:55 +03:00
- inputs/a_0.in
- inputs/a_1.in
*/
function main(a: u32) {}
```