hasura/graphql-engine
1
0
Fork 0
mirror of https://github.com/hasura/graphql-engine.git synced 2024-09-11 02:35:32 +03:00
Code Issues Projects Releases Wiki Activity
3d33017296
graphql-engine/.circleci
History
Abby Sassel 25d77afaff server/tests: Fix BigQuery test failure Job exceeded rate limits error in CI 
Fixes https://github.com/hasura/graphql-engine-mono/issues/3695.

Error: [BigQuery test failure Job exceeded rate limits](https://github.com/hasura/graphql-engine-mono/issues/3695)

Cause:
1. [this command](2325755954/.circleci/test-server.sh (L1263)) runs tests matching the `Bigquery or Common` string, for the `test-oss-server-bigquery` CI job.
2. in this case, the pytest filter matched on `TestGraphQLQueryBoolExpSearchCommon`. Although unrelated pytests are skipped, BQ setup and teardown runs uneccesarily for the [MSSQL and Postgres backends](e444cf1f5d/server/tests-py/test_graphql_queries.py (L868)).
4. the setup and teardown runs three times in quick succession, _for each of_ SQL Server, Postgres and BigQuery. Occasionally, this surpassed [BigQuery's maximum rate of 5 table update operations in 10 seconds](https://cloud.google.com/bigquery/quotas#load_job_per_table.long).

Fix: restrict setup/teardown to only the relevant backends...
- Hotfix (this PR): ...by renaming pytest classes and changing the pytest filters in `test-server`
	- ok, this is faintly horrifying and an inelegant convention change. On the bright side, it shaves a minute or so off our integration test suite run by skipping fewer tests. Anecdata for `test-oss-server-bigquery`
		- before: 87 passed, 299 skipped, 1 warning, 1 error in 192.99s
		- after: 87 passed, 20 skipped, 1 warning in 170.82s
	- [`Common` was a terrible name, anyway](https://github.com/hasura/graphql-engine-mono/issues/2079), for `AnyCombinationOfBackends`.
- Better fix: ...by refactoring the `conftest.py` helpers. I ran out of a timebox so will write up a separate issue. Given we're actively [porting pytests over to hspec](https://github.com/hasura/graphql-engine/issues/8432), I don't know how much it's worth investing time in a refactor.

To verify the fix: I ran a full CI build a few times [[1]](https://buildkite.com/hasura/graphql-engine-mono/builds/8069#078c781a-c8ef-44f2-a400-15f91fb88e42)[[2]](https://buildkite.com/hasura/graphql-engine-mono/builds/8072#f9e7f59d-264f-46a4-973d-21aa762cca35)[[3]](https://buildkite.com/hasura/graphql-engine-mono/builds/8075#bb104e80-ff76-408c-a46b-6f40e92e6317) whilst troubleshooting to convince myself this fixed the problem.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4362
GitOrigin-RevId: 4c3283f0654b70e9dcda642d9012f6376aa95290
2022-04-27 18:40:41 +00:00
..
pgbouncer run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
postgres-12.0-alpine-postgis3 build: rename file to adhere to windows rules (close #4002) (#4003) 2020-03-03 12:33:21 +05:30
postgres-13.0-alpine-postgis3 server: hasura on PG v13 (#125) 2020-12-01 12:22:42 +00:00
server-upgrade-downgrade server: Clean up Cabal project files 2022-04-07 07:16:12 +00:00
.gitignore run graphql tests on both http and websocket; add parallelism (close #1868) (#1921) 2019-04-08 12:52:38 +05:30
build-docker-image.sh circleci integration (#74) 2018-07-10 10:01:02 +00:00
ciignore.sh server: split server test suite to run in parallel on circleci (#309) 2021-01-25 08:24:13 +00:00
cli-migrations-builder.dockerfile ci: add tests for cli-migrations image (#4396) 2020-04-15 07:26:00 +05:30
config.yml ci,server: upgrades to GHC 8.10.7 and reworks server builder image 2021-11-15 13:22:39 +00:00
install-cli-builder-deps.sh cli, ci: enables docker based tests 2021-05-17 00:30:08 +00:00
README.md circleci: Use backport of Matt Pickering's memory reclamation RTS patch in CI 2021-03-12 02:09:21 +00:00
server-test-names.txt server: streaming subscriptions schema generation and tests (incremental PR - 3) 2022-04-22 19:54:11 +00:00
test-console.sh build: use the debian buster version in the docker image 2021-05-27 07:50:15 +00:00
test-deprecated-server-flags.sh tests: run server test shell files through shfmt for consistent indentation 2022-01-11 11:22:42 +00:00
test-server-flags.sh tests: run server test shell files through shfmt for consistent indentation 2022-01-11 11:22:42 +00:00
test-server.sh server/tests: Fix BigQuery test failure Job exceeded rate limits error in CI 2022-04-27 18:40:41 +00:00

README.md

TODO(swann): write this out better

Any tests in test-server.sh should also have their names added to server-test-names.txt, otherwise the CI pipelines will not run them.