Tested on python 3.5 and 3.7
We make light use of pyenv to set an appropriate python version if
installed. We could easily install a correct version too if we wanted
but that seemed invasive.
The newer ruamel was an annoying upgrade but also offers some
improvements that exposed some test suite issues (fixed later).
We add a new pytest flag `--accept` that will automatically write back
yaml files with updated responses. This makes it much easier and less
error-prone to update test cases when we expect output to change, or
when authoring new tests.
Second we make sure to test that we actually preserve the order of the
selection set when returning results. This is a "SHOULD" part of the
spec but seems pretty important and something that users will rely on.
To support both of the above we use ruamel.yaml which preserves a
certain amount of formatting and comments (so that --accept can work in
a failry ergonomic way), as well as ordering (so that when we write yaml
the order of keys has meaning that's preserved during parsing).
Use ruamel.yaml everywhere for consistency (since both libraries have
different quirks).
Quirks of ruamel.yaml:
- trailing whitespace in multiline strings in yaml files isn't written
back out as we'd like: https://bitbucket.org/ruamel/yaml/issues/47/multiline-strings-being-changed-if-they
- formatting is only sort of preserved; ruamel e.g. normalizes
indentation. Normally the diff is pretty clean though, and you can
always just check in portions of your test file after --accept
fixup
Examples
1) `
pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv
`
2) `pytest --hge-urls "http://127.0.0.1:8080" "http://127.0.0.1:8081" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" "postgresql://admin@127.0.0.1:5432/hge_tests2" -vv
`
### Solution and Design
<!-- How is this issue solved/fixed? What is the design? -->
<!-- It's better if we elaborate -->
#### Reducing execution time of tests
- The Schema setup and teardown, which were earlier done per test method, usually takes around 1 sec.
- For mutations, the model has now been changed to only do schema setup and teardown once per test class.
- A data setup and teardown will be done once per test instead (usually takes ~10ms).
- For the test class to get this behaviour, one can can extend the class `DefaultTestMutations`.
- The function `dir()` should be define which returns the location of the configuration folder.
- Inside the configuration folder, there should be
- Files `<conf_dir>/schema_setup.yaml` and `<conf_dir>/schema_teardown.yaml`, which has the metadata query executed during schema setup and teardown respectively
- Files named `<conf_dir>/values_setup.yaml` and `<conf_dir>/values_teardown.yaml`. These files are executed to setup and remove data from the tables respectively.
#### Running Graphql queries on both http and websockets
- Each GraphQL query/mutation is run on the both HTTP and websocket protocols
- Pytests test parameterisation is used to achieve this
- The errors over websockets are slightly different from that on HTTP
- The code takes care of converting the errors in HTTP to errors in websockets
#### Parallel executation of tests.
- The plugin pytest-xdist helps in running tests on parallel workers.
- We are using this plugin to group tests by file and run on different workers.
- Parallel test worker processes operate on separate postgres databases(and separate graphql-engines connected to these databases). Thus tests on one worker will not affect the tests on the other worker.
- With two workers, this decreases execution times by half, as the tests on event triggers usually takes a long time, but does not consume much CPU.
* 1) Tests for creating permissions
2) Test for constraint_on with GraphQL insert on_conflict
* Run tests with access key and webhook
* Tests for GraphQL query with quoted columns
* Rewrite test-server.sh so that it can be run locally
* JWT based tests
* Tests with various postgres types
* For tests on select queries, run setup only once per class
* Tests for v1 count queries
* Skip teardown for tests that does not modify data
* Workaround for hpc 'parse error when reading .tix file'
* Move GeoJson tests to the new structure
* Basic tests for v1 queries
* Tests for column, table or operator not found error cases on GraphQL queries
* Skip test teardown for mutation tests which does not change database state, even when it returns 200.
