Merge pull request #1781 from AleoHQ/doc-test-update

[Testing] Update doc of test framework.
This commit is contained in:
Collin Chin 2022-05-04 13:02:22 -07:00 committed by GitHub
commit 4a1cd29b53
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -1,17 +1,17 @@
# Leo Test Framework
This directory includes Leo code samples, which are parsed and used as tests by test-framework.
This directory includes Leo code samples, which are parsed and used as tests by `test-framework`.
## Structure
Currently, test framework covers only two areas: compiler and parser, tests for both destinations are placed in
matching folders. Third folder - expectations - contains results of test execution which are saved in git and then
Currently, the test framework covers only two areas: compiler and parser; tests for both destinations are placed in
matching folders. The third folder - expectations - contains results of test execution which are saved in git and then
compared to test output.
## Test Structure
Tests can be placed either in `compiler/` or `parser/` directories. Each test is a Leo file with correct (or intentionally
incorrect) Leo code. What makes Leo file a test is block comment at the top of the file:
Tests can be placed either in the `compiler/` or in the `parser/` directories. Each test is a Leo file with correct (or intentionally
incorrect) Leo code. What makes Leo file a test is a block comment at the top of the file:
```
/*
@ -29,17 +29,17 @@ This comment contains YAML structure with set of mandatory and optional fields.
## Test Expectations
After first run of the tests, test expectations will be autogenerated and placed under `tests/expectations` directory.
They will contain results of exection in detail (for example, in Compiler tests they include number of constraints and
After an initial run of the tests, test expectations will be autogenerated and placed under the `expectations/` directory.
They will contain the results of execution in detail (for example, in the compiler tests, they include number of constraints and
output registers).
During next test runs, results of each test are compared to stored expectations, and if stored expectations (say, number
of constaints in Pedersen Hash example) don't match actual results, error will be through and test won't pass. Of course,
During subsequent test runs, the results of each test are compared to the stored expectations, and if the stored expectations (say, number
of constraints in Pedersen Hash example) don't match actual results, an error will be thrown and the test won't pass. Of course,
there are two possible scenarios:
1. If test has failed because logic was changed intentionally, then expectations need to be deleted. New ones will be
generated instead. And commit or PR should contain changes to expectations as well as to tests or code.
2. If test should pass, then expectations should not be changed or removed.
1. If the test has failed because the logic was changed intentionally, then expectations need to be deleted. New ones will be
generated instead. A PR should contain changes to expectations as well as to tests or code.
2. If the test should pass, then expectations should not be changed or removed.
## Test Configuration
@ -50,13 +50,15 @@ Here is the list of all possible configuration options for compiler and parser t
```
- Mandatory: yes
- Namespace: all
- Values: Compile / Parser
- Values: Compile / Parse
```
Only two values are supported: `Parser` and `Compile`, the former is meant to be a parser test, the latter
is, obviously, compiler test. Mind that it's `Compile` and NOT ~Compiler~.
Only two values are supported: `Parse` and `Compile`. The former is meant to be a parser test, the latter
is a full compiler test.
In Parser namespace there are additional possible values to this field: `ParseStatement`, `ParseExpression` and `Token`.
Besides the `Parse` value,
there are actually additional possible values for this field:
`ParseStatement`, `ParseExpression`, and `Token`.
Each one of them allows testing Leo parser on different levels - lexer tokens or just expressions/statements.
Compiler tests always include complete Leo programs.
@ -69,9 +71,9 @@ Compiler tests always include complete Leo programs.
- Values: Pass / Fail
```
This setting indicates whether given code is supposed to be run successfuly or we expect failure. Then, if test
was marked as `Pass` and it actually failed, you'll know that something went wrong and test or compiler/parser need
fixing.
This setting indicates whether the tested code is supposed to succeed or to fail.
If the test was marked as `Pass` but it actually failed,
you'll know that something went wrong and the test or the compiler/parser needs fixing.
### input_file (Compile)
@ -81,70 +83,18 @@ fixing.
- Values: <input file path>, ...
```
This setting allows using one or multiple input files for Leo program. Program will then be run with every provided input.
This setting allows using one or more input files for the Leo program.
The program will be run with every provided input.
See this example:
```
/*
namespace: Compile
expectation: Pass
input_file:
input_file:
- inputs/a_0.in
- inputs/a_1.in
*/
function main(a: u32) {}
```
### inputs (Compile)
```
- Mandatory: no
- Namespace: Compile
- Values: <input file contents>, ...
```
With this setting you can specify inputs right in the test description. It is useful for not creating too many files for
each single case.
```
/*
namespace: Compile
expectation: Pass
inputs:
- first.in: |
[main]
a: u32 = 100;
[registers]
r0: bool = false;
# - second.in: | ....
*/
function main(a: u32) -> bool {
return a == 100;
}
```
### state_file (Compile)
```
- Mandatory: no
- Namespace: Compile
- Values: <path to state file>
```
This setting allows you to specify state file, which can be accessed in Leo program:
```
/*
namespace: Compile
expectation: Pass
state_file: input/basic.state
input_file: input/basic.in
*/
function main(a: bool) -> bool {
return a == input.registers.b;
}
```