graphql-engine/server/tests-py/test_graphql_introspection.py

136 lines
5.9 KiB
Python
Raw Normal View History

run default tests in test_server_upgrade (#3718) * run basic tests after upgrade * terminate before specifying file in pytest cmd * Move fixture definitions out of test classes Previously we had abstract classes with the fixtures defined in them. The test classes then inherits these super classes. This is creating inheritence problems, especially when you want to just inherit the tests in class, but not the fixtures. We have now moved all those fixture definitions outside of the class (in conftest.py). These fixtures are now used by the test classes when and where they are required. * Run pytests on server upgrade Server upgrade tests are run by 1) Run pytest with schema/metadata setup but do not do schema/metadata teardown 2) Upgrade the server 3) Run pytest using the above schema and teardown at the end of the tests 4) Cleanup hasura metadata and start again with next set of tests We have added options --skip-schema-setup and --skip-schema-teardown to help running server upgrade tests. While running the tests, we noticed that error codes and messages for some of the tests have changed. So we have added another option to pytest `--avoid-error-message-checks`. If this flag is set, and if comparing expected and response message fails, and if the expected response has an error message, Pytest will throw warnings instead of an error. * Use marks to specify server-upgrade tests Not all tests can be run as serve upgrade tests, particularly those which themselves change the schema. We introduce two pytest markers. Marker allow_server_upgrade_test will add the test into the list of server upgrade tests that can be run. skip_server_upgrade_test removes it from the list. With this we have added tests for queries, mutations, and selected event trigger and remote schema tests to the list of server upgrade tests. * Remove components not needed anymore * Install curl * Fix error in query validation * Fix error in test_v1_queries.py * install procps for server upgrade tests * Use postgres image which has postgis installed * set pager off with psql * quote the bash variable WORKTREE_DIR Co-authored-by: nizar-m <19857260+nizar-m@users.noreply.github.com> Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-02-13 12:14:02 +03:00
import pytest
import ruamel.yaml as yaml
from validate import check_query_f, check_query
run default tests in test_server_upgrade (#3718) * run basic tests after upgrade * terminate before specifying file in pytest cmd * Move fixture definitions out of test classes Previously we had abstract classes with the fixtures defined in them. The test classes then inherits these super classes. This is creating inheritence problems, especially when you want to just inherit the tests in class, but not the fixtures. We have now moved all those fixture definitions outside of the class (in conftest.py). These fixtures are now used by the test classes when and where they are required. * Run pytests on server upgrade Server upgrade tests are run by 1) Run pytest with schema/metadata setup but do not do schema/metadata teardown 2) Upgrade the server 3) Run pytest using the above schema and teardown at the end of the tests 4) Cleanup hasura metadata and start again with next set of tests We have added options --skip-schema-setup and --skip-schema-teardown to help running server upgrade tests. While running the tests, we noticed that error codes and messages for some of the tests have changed. So we have added another option to pytest `--avoid-error-message-checks`. If this flag is set, and if comparing expected and response message fails, and if the expected response has an error message, Pytest will throw warnings instead of an error. * Use marks to specify server-upgrade tests Not all tests can be run as serve upgrade tests, particularly those which themselves change the schema. We introduce two pytest markers. Marker allow_server_upgrade_test will add the test into the list of server upgrade tests that can be run. skip_server_upgrade_test removes it from the list. With this we have added tests for queries, mutations, and selected event trigger and remote schema tests to the list of server upgrade tests. * Remove components not needed anymore * Install curl * Fix error in query validation * Fix error in test_v1_queries.py * install procps for server upgrade tests * Use postgres image which has postgis installed * set pager off with psql * quote the bash variable WORKTREE_DIR Co-authored-by: nizar-m <19857260+nizar-m@users.noreply.github.com> Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-02-13 12:14:02 +03:00
@pytest.mark.usefixtures('per_class_tests_db_state')
class TestGraphqlIntrospection:
def test_introspection(self, hge_ctx):
with open(self.dir() + "/introspection.yaml") as c:
run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) Examples 1) ` pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv ` 2) `pytest --hge-urls "http://127.0.0.1:8080" "http://127.0.0.1:8081" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" "postgresql://admin@127.0.0.1:5432/hge_tests2" -vv ` ### Solution and Design <!-- How is this issue solved/fixed? What is the design? --> <!-- It's better if we elaborate --> #### Reducing execution time of tests - The Schema setup and teardown, which were earlier done per test method, usually takes around 1 sec. - For mutations, the model has now been changed to only do schema setup and teardown once per test class. - A data setup and teardown will be done once per test instead (usually takes ~10ms). - For the test class to get this behaviour, one can can extend the class `DefaultTestMutations`. - The function `dir()` should be define which returns the location of the configuration folder. - Inside the configuration folder, there should be - Files `<conf_dir>/schema_setup.yaml` and `<conf_dir>/schema_teardown.yaml`, which has the metadata query executed during schema setup and teardown respectively - Files named `<conf_dir>/values_setup.yaml` and `<conf_dir>/values_teardown.yaml`. These files are executed to setup and remove data from the tables respectively. #### Running Graphql queries on both http and websockets - Each GraphQL query/mutation is run on the both HTTP and websocket protocols - Pytests test parameterisation is used to achieve this - The errors over websockets are slightly different from that on HTTP - The code takes care of converting the errors in HTTP to errors in websockets #### Parallel executation of tests. - The plugin pytest-xdist helps in running tests on parallel workers. - We are using this plugin to group tests by file and run on different workers. - Parallel test worker processes operate on separate postgres databases(and separate graphql-engines connected to these databases). Thus tests on one worker will not affect the tests on the other worker. - With two workers, this decreases execution times by half, as the tests on event triggers usually takes a long time, but does not consume much CPU.
2019-04-08 10:22:38 +03:00
conf = yaml.safe_load(c)
resp, _ = check_query(hge_ctx, conf)
hasArticle = False
hasArticleAuthorFKRel = False
hasArticleAuthorManualRel = False
for t in resp['data']['__schema']['types']:
if t['name'] == 'article':
hasArticle = True
for fld in t['fields']:
if fld['name'] == 'author_obj_rel_manual':
hasArticleAuthorManualRel = True
assert fld['type']['kind'] == 'OBJECT'
elif fld['name'] == 'author_obj_rel_fk':
hasArticleAuthorFKRel = True
assert fld['type']['kind'] == 'NON_NULL'
assert hasArticle
assert hasArticleAuthorFKRel
assert hasArticleAuthorManualRel
def test_introspection_user(self, hge_ctx):
check_query_f(hge_ctx, self.dir() + "/introspection_user_role.yaml")
@classmethod
def dir(cls):
return "queries/graphql_introspection"
@pytest.mark.usefixtures('per_class_tests_db_state')
class TestNullableObjectRelationshipInSchema:
@classmethod
def dir(cls):
return "queries/graphql_introspection/nullable_object_relationship"
def test_introspection(self, hge_ctx):
with open(self.dir() + "/../introspection.yaml") as c:
conf = yaml.safe_load(c)
resp, _ = check_query(hge_ctx, conf)
for t in resp['data']['__schema']['types']:
if t['name'] == 'table2':
for fld in t['fields']:
if fld['name'] == 'via_table1':
# graphql schema introspection doesn't explictly mark
# the fields to be nullable (it does in the non-null
# case). So checking this should be sufficient to detect
# if its nullable. If this is not nullable, the
# top-level kind would be `NON_NULL` and its `ofType`
# would have the actual type
assert fld['type']['kind'] == 'OBJECT'
def getTypeNameFromType(typeObject):
if typeObject['name'] != None:
return typeObject['name']
elif isinstance(typeObject['ofType'],dict):
return getTypeNameFromType(typeObject['ofType'])
else:
raise Exception("typeObject doesn't have name and ofType is not an object")
@pytest.mark.usefixtures('per_class_tests_db_state')
class TestGraphqlIntrospectionWithCustomTableName:
# test to check some of the type names that are generated
# while tracking a table with a custom name
def test_introspection(self, hge_ctx):
with open(self.dir() + "/introspection.yaml") as c:
conf = yaml.safe_load(c)
resp, _ = check_query(hge_ctx, conf)
hasMultiSelect = False
hasAggregate = False
hasSelectByPk = False
hasQueryRoot = False
for t in resp['data']['__schema']['types']:
if t['name'] == 'query_root':
hasQueryRoot = True
for field in t['fields']:
if field['name'] == 'user_address':
hasMultiSelect = True
assert 'args' in field
for args in field['args']:
if args['name'] == 'distinct_on':
assert "user_address_select_column" == getTypeNameFromType(args['type'])
elif args['name'] == 'order_by':
assert "user_address_order_by" == getTypeNameFromType(args['type'])
elif args['name'] == 'where':
assert 'user_address_bool_exp' == getTypeNameFromType(args['type'])
elif field['name'] == 'user_address_aggregate':
hasAggregate = True
assert "user_address_aggregate" == getTypeNameFromType(field['type'])
elif field['name'] == 'user_address_by_pk':
assert "user_address" == getTypeNameFromType(field['type'])
hasSelectByPk = True
elif t['name'] == 'mutation_root':
for field in t['fields']:
if field['name'] == 'insert_user_address':
hasMultiInsert = True
assert "user_address_mutation_response" == getTypeNameFromType(field['type'])
for args in field['args']:
if args['name'] == 'object':
assert "user_address_insert_input" == getTypeNameFromType(args['type'])
elif field['name'] == 'update_user_address_by_pk':
hasUpdateByPk = True
assert "user_address" == getTypeNameFromType(field['type'])
for args in field['args']:
if args['name'] == 'object':
assert "user_address" == getTypeNameFromType(args['type'])
assert hasQueryRoot
assert hasMultiSelect
assert hasAggregate
assert hasSelectByPk
assert hasMultiInsert
assert hasUpdateByPk
@classmethod
def dir(cls):
return "queries/graphql_introspection/custom_table_name"
@pytest.mark.usefixtures('per_class_tests_db_state', 'pro_tests_fixtures')
class TestDisableGraphQLIntrospection:
@classmethod
def dir(cls):
return "queries/graphql_introspection/disable_introspection"
setup_metadata_api_version = "v2"
def test_disable_introspection(self, hge_ctx):
check_query_f(hge_ctx, self.dir() + "/disable_introspection.yaml")