graphql-engine/server/tests-py
nizar-m a40bf10b9f run graphql tests on both http and websocket; add parallelism (close #1868) (#1921)
Examples 
1)  `
pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv
`
2)  `pytest --hge-urls "http://127.0.0.1:8080"   "http://127.0.0.1:8081" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests"  "postgresql://admin@127.0.0.1:5432/hge_tests2" -vv
`
### Solution and Design
<!-- How is this issue solved/fixed? What is the design? -->
<!-- It's better if we elaborate -->
#### Reducing execution time of tests
- The Schema setup and teardown, which were earlier done per test method, usually takes around 1 sec. 
- For mutations, the model has now been changed to only do schema setup and teardown once per test class.
-  A data setup and teardown will be done once per test instead (usually takes ~10ms).
- For the test class to get this behaviour, one can can extend the class `DefaultTestMutations`. 
    - The function  `dir()` should be define which returns the location of the configuration folder.
    - Inside the configuration folder, there should be 
        - Files `<conf_dir>/schema_setup.yaml` and `<conf_dir>/schema_teardown.yaml`, which has the metadata query executed during schema setup and teardown respectively
        - Files named `<conf_dir>/values_setup.yaml` and `<conf_dir>/values_teardown.yaml`. These files are executed to setup and remove data from the tables respectively. 

#### Running Graphql queries on both http and websockets
- Each GraphQL query/mutation is run on the both HTTP and websocket protocols
- Pytests test parameterisation is used to achieve this
- The errors over websockets are slightly different from that on HTTP
   - The code takes care of converting the errors in HTTP to errors in websockets

#### Parallel executation of tests.
- The plugin pytest-xdist helps in running tests on parallel workers.
- We are using this plugin to group tests by file and run on different workers.
- Parallel test worker processes operate on separate postgres databases(and separate graphql-engines connected to these databases). Thus tests on one worker will not affect the tests on the other worker.
- With two workers, this decreases execution times by half, as the tests on event triggers usually takes a long time, but does not consume much CPU.
2019-04-08 12:52:38 +05:30
..
queries run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
webhook/insecure apply update permissions for upsert mutations (#628) 2018-12-15 21:40:29 +05:30
.gitignore run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
conftest.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
context.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
graphql_server.py revert "forward response headers from remote servers (#1664)" 2019-02-28 17:20:56 +05:30
pytest.ini Tests for server with access control, and some more tests (#710) 2018-10-28 23:57:49 +05:30
README.md run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
requirements.txt run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
super_classes.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
test_apis_disabled.py add option to disable metadata and graphql apis (close #1088) (#1650) 2019-02-28 19:23:03 +05:30
test_compat.py rename access-key to admin-secret (close #1347) (#1540) 2019-02-14 15:07:47 +05:30
test_cookie_webhook.py read cookie while initialising websocket connection (fix #1660) (#1668) 2019-03-04 13:16:53 +05:30
test_cors.py add support for multiple domains in cors config (close #1436) (#1536) 2019-02-14 11:28:38 +05:30
test_events.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
test_graphql_introspection.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
test_graphql_mutations.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
test_graphql_queries.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
test_horizontal_scale.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
test_jwt.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
test_schema_stitching.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
test_subscriptions.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
test_v1_queries.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
test_version.py pep8 fixes for python files (#875) 2018-10-30 14:51:58 +05:30
test_webhook_insecure.py Tests for server with access control, and some more tests (#710) 2018-10-28 23:57:49 +05:30
test_websocket_init_cookie.py read cookie while initialising websocket connection (fix #1660) (#1668) 2019-03-04 13:16:53 +05:30
validate.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
webhook.py allow authentication webhook with POST (close #1138) (#1147) 2018-12-03 16:49:08 +05:30
webserver.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30

Running tests

pip install -r requirements.txt
pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv

Tests Structure

  • Tests are grouped as test classes in test modules (names starting with test_)

  • The configuration files (if needed) for the tests in a class are usually kept in one folder.

    • The folder name is usually either the dir variable or the dir() function
  • Some tests (like in test_graphql_queries.py) requires a setup and teardown per class.

    • Here we are extending the DefaultTestSelectQueries class.
    • This class defines a fixture which will run the configurations in setup.yaml and teardown.yaml once per class
    • Extending test class should define a function name dir(), which returns the configuration folder
  • For mutation tests (like in test_graphql_mutations.py)

    • We need a schema_setup and schema_teardown per class
    • And values_setup and values_teardown per test
    • Doing schema setup and teardown per test is expensive.
    • We are extending the DefaultTestMutations class for this.
    • This class defines a fixture which will run the configuration in setup.yaml and teardown.yaml once per class.
    • Another fixture defined in this class runs the configuration in values_setup.yaml and values_teardown.yaml once per class.