hasura/graphql-engine
1
0
Fork 0
mirror of https://github.com/hasura/graphql-engine.git synced 2024-09-11 10:46:25 +03:00
Code Issues Projects Releases Wiki Activity

server/tests: Fix BigQuery test failure Job exceeded rate limits error in CI

Browse Source 
Fixes https://github.com/hasura/graphql-engine-mono/issues/3695.

Error: [BigQuery test failure Job exceeded rate limits](https://github.com/hasura/graphql-engine-mono/issues/3695)

Cause:
1. [this command](2325755954/.circleci/test-server.sh (L1263)) runs tests matching the `Bigquery or Common` string, for the `test-oss-server-bigquery` CI job.
2. in this case, the pytest filter matched on `TestGraphQLQueryBoolExpSearchCommon`. Although unrelated pytests are skipped, BQ setup and teardown runs uneccesarily for the [MSSQL and Postgres backends](e444cf1f5d/server/tests-py/test_graphql_queries.py (L868)).
4. the setup and teardown runs three times in quick succession, _for each of_ SQL Server, Postgres and BigQuery. Occasionally, this surpassed [BigQuery's maximum rate of 5 table update operations in 10 seconds](https://cloud.google.com/bigquery/quotas#load_job_per_table.long).

Fix: restrict setup/teardown to only the relevant backends...
- Hotfix (this PR): ...by renaming pytest classes and changing the pytest filters in `test-server`
	- ok, this is faintly horrifying and an inelegant convention change. On the bright side, it shaves a minute or so off our integration test suite run by skipping fewer tests. Anecdata for `test-oss-server-bigquery`
		- before: 87 passed, 299 skipped, 1 warning, 1 error in 192.99s
		- after: 87 passed, 20 skipped, 1 warning in 170.82s
	- [`Common` was a terrible name, anyway](https://github.com/hasura/graphql-engine-mono/issues/2079), for `AnyCombinationOfBackends`.
- Better fix: ...by refactoring the `conftest.py` helpers. I ran out of a timebox so will write up a separate issue. Given we're actively [porting pytests over to hspec](https://github.com/hasura/graphql-engine/issues/8432), I don't know how much it's worth investing time in a refactor.

To verify the fix: I ran a full CI build a few times [[1]](https://buildkite.com/hasura/graphql-engine-mono/builds/8069#078c781a-c8ef-44f2-a400-15f91fb88e42)[[2]](https://buildkite.com/hasura/graphql-engine-mono/builds/8072#f9e7f59d-264f-46a4-973d-21aa762cca35)[[3]](https://buildkite.com/hasura/graphql-engine-mono/builds/8075#bb104e80-ff76-408c-a46b-6f40e92e6317) whilst troubleshooting to convince myself this fixed the problem.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4362
GitOrigin-RevId: 4c3283f0654b70e9dcda642d9012f6376aa95290
This commit is contained in:
Abby Sassel 2022-04-27 19:39:40 +01:00 committed by hasura-bot
parent f684fecc6c
commit 25d77afaff
5 changed files with 19 additions and 28 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.circleci/test-server.sh
View File

@ -1223,7 +1223,7 @@ backend-mssql)
add_mssql_source 8080 "$HASURA_GRAPHQL_MSSQL_SOURCE_URL"
# See note [Specifying Pytests with -k flag]
pytest -n 1 --hge-urls "$HGE_URL" --pg-urls "$HASURA_GRAPHQL_DATABASE_URL" --backend mssql -k "MSSQL or Common"
pytest -n 1 --hge-urls "$HGE_URL" --pg-urls "$HASURA_GRAPHQL_DATABASE_URL" --backend mssql -k "MSSQL"
# start inherited roles test
echo -e "\n$(time_elapsed): <########## TEST INHERITED-ROLES WITH SQL SERVER BACKEND ###########################################>\n"
@ -1244,7 +1244,7 @@ backend-citus)
add_citus_source 8080 "$HASURA_GRAPHQL_CITUS_SOURCE_URL"
# See note [Specifying Pytests with -k flag]
pytest -n 1 --hge-urls "$HGE_URL" --pg-urls "$HASURA_GRAPHQL_DATABASE_URL" --backend citus -k "Citus or Common"
pytest -n 1 --hge-urls "$HGE_URL" --pg-urls "$HASURA_GRAPHQL_DATABASE_URL" --backend citus -k "Citus"
kill_hge_servers
;;
@ -1260,7 +1260,7 @@ backend-bigquery)
add_bigquery_source 8080
# See note [Specifying Pytests with -k flag]
pytest -n 1 --hge-urls "$HGE_URL" --pg-urls "$HASURA_GRAPHQL_DATABASE_URL" --backend bigquery -k "Bigquery or Common"
pytest -n 1 --hge-urls "$HGE_URL" --pg-urls "$HASURA_GRAPHQL_DATABASE_URL" --backend bigquery -k "Bigquery"
kill_hge_servers
;;

10
server/tests-py/test_events.py
View File

@ -175,7 +175,7 @@ class TestEventCreateAndDeleteMSSQL:
# HASURA_GRAPHQL_EVENTS_FETCH_BATCH_SIZE=100 (the default)
@pytest.mark.parametrize("backend", ['mssql','postgres'])
@usefixtures("per_method_tests_db_state")
class TestEventFloodCommon(object):
class TestEventFloodPostgresMSSQL(object):
@classmethod
def dir(cls):
@ -376,7 +376,7 @@ class TestEventDataFormatGeoJSONMSSQL(object):
@pytest.mark.parametrize("backend", ['mssql','postgres'])
@usefixtures("per_class_tests_db_state")
class TestCreateEventQueryCommon(object):
class TestCreateEventQueryPostgresMSSQL(object):
@classmethod
def dir(cls):
@ -458,7 +458,7 @@ class TestCreateEventQueryCommon(object):
@pytest.mark.parametrize("backend", ['mssql','postgres'])
@usefixtures('per_method_tests_db_state')
class TestEventRetryConfCommon(object):
class TestEventRetryConfPostgresMSSQL(object):
@classmethod
def dir(cls):
@ -541,7 +541,7 @@ class TestEventRetryConfCommon(object):
@pytest.mark.parametrize("backend", ['mssql','postgres'])
@usefixtures('per_method_tests_db_state')
class TestEventHeadersCommon(object):
class TestEventHeadersPostgresMSSQL(object):
@classmethod
def dir(cls):
@ -1663,7 +1663,7 @@ class TestManualEventsMSSQL(object):
@pytest.mark.parametrize("backend", ['mssql','postgres'])
@usefixtures('per_method_tests_db_state')
class TestEventsAsynchronousExecutionCommon(object):
class TestEventsAsynchronousExecutionPostgresMSSQL(object):
@classmethod
def dir(cls):

27
server/tests-py/test_graphql_queries.py
View File

@ -248,7 +248,7 @@ class TestGraphQLQueryBoolExpSearchBigquery:
@pytest.mark.parametrize("transport", ['http', 'websocket'])
@pytest.mark.parametrize("backend", ['citus', 'mssql', 'postgres'])
@usefixtures('per_class_tests_db_state')
class TestGraphQLQueryBasicCommon:
class TestGraphQLQueryBasicPostgresMSSQLCitus:
# This also exercises support for multiple operations in a document:
def test_select_query_author(self, hge_ctx, transport):
check_query_f(hge_ctx, self.dir() + '/select_query_author.yaml', transport)
@ -439,7 +439,7 @@ class TestGraphQLQueryBasicCitus:
@pytest.mark.parametrize("transport", ['http', 'websocket'])
@pytest.mark.parametrize("backend", ['citus', 'postgres'])
@usefixtures('per_class_tests_db_state')
class TestGraphQLQueryFragmentsCommon:
class TestGraphQLQueryFragmentsPostgresCitus:
def test_select_query_top_level_fragment(self, hge_ctx, transport):
check_query_f(hge_ctx, self.dir() + '/select_query_top_level_fragment.yaml', transport)
@ -484,7 +484,7 @@ class TestGraphQLQueryAgg:
@pytest.mark.parametrize("transport", ['http', 'websocket'])
@pytest.mark.parametrize("backend", ['mssql', 'postgres'])
@usefixtures('per_class_tests_db_state')
class TestGraphQLQueryAggPermCommon:
class TestGraphQLQueryAggPermPostgresMSSQL:
def test_author_agg_articles(self, hge_ctx, transport):
check_query_f(hge_ctx, self.dir() + '/author_agg_articles.yaml', transport)
@ -576,7 +576,7 @@ class TestGraphQLQueryOffsets:
@pytest.mark.parametrize("transport", ['http', 'websocket', 'subscription'])
@pytest.mark.parametrize("backend", ['mssql', 'postgres'])
@usefixtures('per_class_tests_db_state')
class TestGraphQLQueryBoolExpBasicCommon:
class TestGraphQLQueryBoolExpBasicPostgresMSSQL:
def test_order_delivered_at_is_null(self, hge_ctx, transport):
check_query_f(hge_ctx, self.dir() + '/select_query_order_delivered_at_is_null.yaml', transport)
@ -864,10 +864,10 @@ class TestGraphQLInheritedRolesMSSQL:
def test_inherited_role_when_some_roles_may_not_have_permission_configured(self, hge_ctx, transport):
check_query_f(hge_ctx, self.dir() + '/inherited_role_with_some_roles_having_no_permissions.yaml')
@pytest.mark.parametrize("transport", ['http', 'websocket', 'subscription'])
@pytest.mark.parametrize("backend", ['postgres', 'mssql'])
@pytest.mark.parametrize("transport", ['http', 'websocket'])
@pytest.mark.parametrize("backend", ['postgres'])
@usefixtures('per_class_tests_db_state')
class TestGraphQLQueryBoolExpSearchCommon:
class TestGraphQLQueryBoolExpSearchPostgres:
def test_city_where_like(self, hge_ctx, transport):
check_query_f(hge_ctx, self.dir() + '/select_city_where_like.yaml', transport)
@ -875,15 +875,6 @@ class TestGraphQLQueryBoolExpSearchCommon:
def test_city_where_not_like(self, hge_ctx, transport):
check_query_f(hge_ctx, self.dir() + '/select_city_where_nlike.yaml', transport)
@classmethod
def dir(cls):
return 'queries/graphql_query/boolexp/search'
@pytest.mark.parametrize("transport", ['http', 'websocket'])
@pytest.mark.parametrize("backend", ['postgres'])
@usefixtures('per_class_tests_db_state')
class TestGraphQLQueryBoolExpSearchPostgres:
def test_city_where_ilike(self, hge_ctx, transport):
check_query_f(hge_ctx, self.dir() + '/select_city_where_ilike_postgres.yaml', transport)
@ -915,7 +906,7 @@ class TestGraphQLQueryBoolExpSearchPostgres:
def dir(cls):
return 'queries/graphql_query/boolexp/search'
@pytest.mark.parametrize("transport", ['http', 'websocket'])
@pytest.mark.parametrize("transport", ['http', 'websocket', 'subscription'])
@pytest.mark.parametrize("backend", ['mssql'])
@usefixtures('per_class_tests_db_state')
class TestGraphQLQueryBoolExpSearchMSSQL:
@ -1276,7 +1267,7 @@ class TestMissingUnauthorizedRoleAndCookie:
check_query_f(hge_ctx, self.dir() + '/cookie_header_absent_unauth_role_not_set.yaml', transport, add_auth=False)
@usefixtures('per_class_tests_db_state')
class TestGraphQLExplainCommon:
class TestGraphQLExplainPostgresMSSQLMySQL:
@classmethod
def dir(cls):
return 'queries/explain'

2
server/tests-py/test_metadata.py
View File

@ -454,7 +454,7 @@ class TestMetadataOrder:
@pytest.mark.parametrize("backend", ['citus', 'mssql', 'postgres', 'bigquery'])
@usefixtures('per_class_tests_db_state')
class TestSetTableCustomizationCommon:
class TestSetTableCustomizationPostgresMSSQLCitusBigquery:
@classmethod
def dir(cls):

2
server/tests-py/test_subscriptions.py
View File

@ -682,7 +682,7 @@ class TestSubscriptionUDFWithSessionArg:
@pytest.mark.parametrize("backend", ['mssql', 'postgres'])
@usefixtures('per_class_tests_db_state', 'ws_conn_init')
class TestSubscriptionCustomizedSourceCommon:
class TestSubscriptionCustomizedSourceMSSQLPostgres:
@classmethod
def dir(cls):
return 'queries/subscriptions/customized_source'