graphql-engine/server/tests-py
Brandon Simmons 91aee7fdeb Test result ordering, add --accept test mode to automatically accept changed test cases
We add a new pytest flag `--accept` that will automatically write back
yaml files with updated responses. This makes it much easier and less
error-prone to update test cases when we expect output to change, or
when authoring new tests.

Second we make sure to test that we actually preserve the order of the
selection set when returning results. This is a "SHOULD" part of the
spec but seems pretty important and something that users will rely on.

To support both of the above we use ruamel.yaml which preserves a
certain amount of formatting and comments (so that --accept can work in
a failry ergonomic way), as well as ordering (so that when we write yaml
the order of keys has meaning that's preserved during parsing).

Use ruamel.yaml everywhere for consistency (since both libraries have
different quirks).

Quirks of ruamel.yaml:
- trailing whitespace in multiline strings in yaml files isn't written
  back out as we'd like: https://bitbucket.org/ruamel/yaml/issues/47/multiline-strings-being-changed-if-they
- formatting is only sort of preserved; ruamel e.g. normalizes
  indentation. Normally the diff is pretty clean though, and you can
  always just check in portions of your test file after --accept

fixup
2019-11-05 15:15:25 -06:00
..
pgdump add an api to dump postgres schema (close #1939) (#1967) 2019-04-30 14:04:08 +05:30
queries Test fixup: 'yes' being YAML synonym for 'true' is dubious 2019-11-05 15:15:25 -06:00
test_tests Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
webhook/insecure apply update permissions for upsert mutations (#628) 2018-12-15 21:40:29 +05:30
.gitignore run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
conftest.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
context.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
graphql_server.py allow configuring timeout for remote schema calls (close #2501) (#2753) 2019-08-23 14:27:19 +05:30
pytest.ini Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
README.md Add local development swiss army knife script scripts/dev.sh 2019-07-26 01:17:35 -04:00
requirements-top-level.txt Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
requirements.txt Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
super_classes.py fix validation of null values, closes #1981 (#2057) 2019-04-24 13:19:39 +05:30
test_allowlist_queries.py keep allowed queries in sync when collections are changed, fix #2221 (#2230) 2019-05-20 15:59:32 +05:30
test_apis_disabled.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_compat.py rename access-key to admin-secret (close #1347) (#1540) 2019-02-14 15:07:47 +05:30
test_compression.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_config_api.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_cookie_webhook.py read cookie while initialising websocket connection (fix #1660) (#1668) 2019-03-04 13:16:53 +05:30
test_cors.py add support for multiple domains in cors config (close #1436) (#1536) 2019-02-14 11:28:38 +05:30
test_events.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_graphql_introspection.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_graphql_mutations.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_graphql_queries.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_horizontal_scale.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_inconsistent_meta.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_jwt.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_logging.py server: Fix flaky logging test (#3081) 2019-10-10 08:40:19 -05:00
test_pg_dump.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_schema_stitching.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_subscriptions.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_tests.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_v1_queries.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_v1alpha1_endpoint.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_validation.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
test_version.py Add local development swiss army knife script scripts/dev.sh 2019-07-26 01:17:35 -04:00
test_webhook_insecure.py Tests for server with access control, and some more tests (#710) 2018-10-28 23:57:49 +05:30
test_websocket_init_cookie.py read cookie while initialising websocket connection (fix #1660) (#1668) 2019-03-04 13:16:53 +05:30
validate.py Test result ordering, add --accept test mode to automatically accept changed test cases 2019-11-05 15:15:25 -06:00
webhook.py allow authentication webhook with POST (close #1138) (#1147) 2018-12-03 16:49:08 +05:30
webserver.py run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30

Running tests

The easiest way to run the test suite is to do:

$ scripts/dev.sh test

This should install python dependencies if required, and run in isolation.

Tests Structure

  • Tests are grouped as test classes in test modules (names starting with test_)

  • The configuration files (if needed) for the tests in a class are usually kept in one folder.

    • The folder name is usually either the dir variable or the dir() function
  • Some tests (like in test_graphql_queries.py) requires a setup and teardown per class.

    • Here we are extending the DefaultTestSelectQueries class.
    • This class defines a fixture which will run the configurations in setup.yaml and teardown.yaml once per class
    • Extending test class should define a function name dir(), which returns the configuration folder
  • For mutation tests (like in test_graphql_mutations.py)

    • We need a schema_setup and schema_teardown per class
    • And values_setup and values_teardown per test
    • Doing schema setup and teardown per test is expensive.
    • We are extending the DefaultTestMutations class for this.
    • This class defines a fixture which will run the configuration in setup.yaml and teardown.yaml once per class.
    • Another fixture defined in this class runs the configuration in values_setup.yaml and values_teardown.yaml once per class.