graphql-engine/server/tests-py/README.md

## Running tests

The easiest way to run the test suite is to do:

    $ scripts/dev.sh test

This should install python dependencies if required, and run in isolation.  The output format is described in the [pytest documentation](https://docs.pytest.org/en/latest/usage.html#detailed-summary-report).  Errors and failures are indicated by `F`s and `E`s.

## Tests Structure

- Tests are grouped as test classes in test modules (names starting with `test_`)

- The configuration files (if needed) for the tests in a class are usually kept in one folder.
  - The folder name is usually either the `dir` variable or the `dir()` function


-  Some tests (like in `test_graphql_queries.py`) requires a setup and teardown per class.
   - Here we are extending the `DefaultTestSelectQueries` class.
   - This class defines a fixture which will run the configurations in `setup.yaml` and `teardown.yaml` once per class
   - Extending test class should define a function name `dir()`, which returns the configuration folder


- For mutation tests (like in `test_graphql_mutations.py`)
   - We need a `schema_setup` and `schema_teardown` per class
   - And `values_setup` and `values_teardown` per test
   - Doing schema setup and teardown per test is expensive.
   - We are extending the `DefaultTestMutations` class for this.
   - This class defines a fixture which will run the configuration in `setup.yaml` and `teardown.yaml` once per class.
   - Another fixture defined in this class runs the configuration in `values_setup.yaml` and `values_teardown.yaml` once per class.
update multiple files (#943) 2018-11-08 09:32:57 +03:00			`## Running tests`

Add local development swiss army knife script `scripts/dev.sh` At the moment we can... ...run tests in isolation, generating coverage report: $ dev.sh test You can pass args to pytest as well. e.g. to run a specific test: $ dev.sh test -k "test_jsonb_has_all" Launch a postgres container with useful dev defaults, with PostGIS, cleaning up afterwards: $ dev.sh postgres Build and launch graphql-engine in dev mode, connecting with a `postgres` launched above $ dev.sh graphql-engine 2019-07-12 06:59:29 +03:00			`The easiest way to run the test suite is to do:`

			`$ scripts/dev.sh test`

Link to pytest docs in the server test suite READMEs (#4425) 2020-04-21 12:29:16 +03:00			This should install python dependencies if required, and run in isolation. The output format is described in the [pytest documentation](https://docs.pytest.org/en/latest/usage.html#detailed-summary-report). Errors and failures are indicated by `F`s and `E`s.
run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) Examples 1) ` pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv ` 2) `pytest --hge-urls "http://127.0.0.1:8080" "http://127.0.0.1:8081" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" "postgresql://admin@127.0.0.1:5432/hge_tests2" -vv ` ### Solution and Design <!-- How is this issue solved/fixed? What is the design? --> <!-- It's better if we elaborate --> #### Reducing execution time of tests - The Schema setup and teardown, which were earlier done per test method, usually takes around 1 sec. - For mutations, the model has now been changed to only do schema setup and teardown once per test class. - A data setup and teardown will be done once per test instead (usually takes ~10ms). - For the test class to get this behaviour, one can can extend the class `DefaultTestMutations`. - The function `dir()` should be define which returns the location of the configuration folder. - Inside the configuration folder, there should be - Files `<conf_dir>/schema_setup.yaml` and `<conf_dir>/schema_teardown.yaml`, which has the metadata query executed during schema setup and teardown respectively - Files named `<conf_dir>/values_setup.yaml` and `<conf_dir>/values_teardown.yaml`. These files are executed to setup and remove data from the tables respectively. #### Running Graphql queries on both http and websockets - Each GraphQL query/mutation is run on the both HTTP and websocket protocols - Pytests test parameterisation is used to achieve this - The errors over websockets are slightly different from that on HTTP - The code takes care of converting the errors in HTTP to errors in websockets #### Parallel executation of tests. - The plugin pytest-xdist helps in running tests on parallel workers. - We are using this plugin to group tests by file and run on different workers. - Parallel test worker processes operate on separate postgres databases(and separate graphql-engines connected to these databases). Thus tests on one worker will not affect the tests on the other worker. - With two workers, this decreases execution times by half, as the tests on event triggers usually takes a long time, but does not consume much CPU. 2019-04-08 10:22:38 +03:00
			`## Tests Structure`

			- Tests are grouped as test classes in test modules (names starting with `test_`)

			`- The configuration files (if needed) for the tests in a class are usually kept in one folder.`
			- The folder name is usually either the `dir` variable or the `dir()` function


			- Some tests (like in `test_graphql_queries.py`) requires a setup and teardown per class.
			- Here we are extending the `DefaultTestSelectQueries` class.
			- This class defines a fixture which will run the configurations in `setup.yaml` and `teardown.yaml` once per class
			- Extending test class should define a function name `dir()`, which returns the configuration folder


			- For mutation tests (like in `test_graphql_mutations.py`)
			- We need a `schema_setup` and `schema_teardown` per class
			- And `values_setup` and `values_teardown` per test
			`- Doing schema setup and teardown per test is expensive.`
			- We are extending the `DefaultTestMutations` class for this.
			- This class defines a fixture which will run the configuration in `setup.yaml` and `teardown.yaml` once per class.
			- Another fixture defined in this class runs the configuration in `values_setup.yaml` and `values_teardown.yaml` once per class.