Abby Sassel
|
fc0352a995
|
server/tests: fix BigQuery test failure
This PR addresses a couple of recent issues with BigQuery tests in CI, plus some refactoring.
TL;DR: most issues were fixed by being [explicit about failures](http://redsymbol.net/articles/unofficial-bash-strict-mode/) and variables in scope. Thanks to @qrilka for addressing a couple of these in your last PR already, sharing my notes about them here too for posterity as I found the troubleshooting exercise useful.
_commands listed in execution order_
`generate_test_project`
* issue: this rarely fails AFAICT, but a failure in any of the `gcloud` commands will result in unclear failures further along the call stack
* fix: fail faster with `set -e`
`create_bigquery_dataset`
* error: `BigQuery error in mk operation: Not found: Project`. examples [[1]](https://github.com/hasura/graphql-engine-mono/issues/3600)[[2]](https://buildkite.com/hasura/graphql-engine-mono/builds/4857#f70990eb-3721-45c6-b11a-af7e4ebd9fe3)
* issue: a failure on a missing project, which I could verify was created successfully
* cause: missing `HASURA_BIGQUERY_PROJECT_ID` , most likely caused by the above failure and dodgy subshell scoping
* fix: pass `project_id` as an argument. This might not have been necessary since `generate_test_project` fails faster, but I thought it was clearer to pass explicitly.
`ensure_bigquery_db`
* issue: this could fail, but return a 0 exit code. [example](https://buildkite.com/hasura/graphql-engine-mono/builds/4836#e3dfbc1e-4034-40bb-beb1-181fb5d9489f)
* fix: add a `--fail` flag to `curl` call. Kirill addressed in https://github.com/hasura/graphql-engine-mono/pull/3435/files#diff-0525000dbb36f436dcc17570541378de51471200d294b468c4b288e0292441b6R94
`delete_temp_bigquery_project`
* error: `(gcloud.projects.delete) value for field [projectId] in collection [cloudresourcemanager.projects] is required but was not provided`. [example](https://buildkite.com/hasura/graphql-engine-mono/builds/4836#e3dfbc1e-4034-40bb-beb1-181fb5d9489f)
* issue: attempting to delete a tmp file that didn't exist due to earlier failures
* fix: Kirill addressed in https://github.com/hasura/graphql-engine-mono/pull/3435/files#diff-0525000dbb36f436dcc17570541378de51471200d294b468c4b288e0292441b6R106
to verify: check [bigquery](https://buildkite.com/hasura/graphql-engine-mono/builds/5115#41d059eb-329c-46fb-a745-b2b97fffd328) and [hspec](https://buildkite.com/hasura/graphql-engine-mono/builds/5115#5623a53d-c18d-478e-bf44-446e1287453b) jobs in CI
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3616
Co-authored-by: Divi <32202683+imperfect-fourth@users.noreply.github.com>
GitOrigin-RevId: b85420ef57036b4dbb05202b73ee9e954113bf9d
|
2022-02-21 11:45:03 +00:00 |
|
Abby Sassel
|
a091110364
|
server/tests: ephemeral BigQuery projects for CI test jobs
_Problem_
We currently run teardown/`DELETE` statements on the same, shared `hasura_test` dataset. This is not ideal as parallel test runs can and do affect each other, resulting in nondeterministic CI failures.
Closes https://github.com/hasura/graphql-engine-mono/issues/2521
_Solution and design_
This PR introduces ephemeral, isolated projects for each test run _in CI only_. Projects are created within [the Google Cloud Platform `data-sources-test-bigquery` directory](https://console.cloud.google.com/iam-admin/settings?folder=704256416468&orgonly=true&supportedpurview=organizationId) on each test run, and destroyed afterwards.
I've only introduced this change in CI for the time being:
1. this isn't as much of an issue locally because we're less likely to run bigquery tests in parallel.
2. to more quickly unblock https://github.com/hasura/graphql-engine/issues/7929.
3. to limit the number of new projects created until we have a better idea of our usage vs GCP quota/limits.
Also updated the [internal wiki here](https://github.com/hasura/graphql-engine-mono/wiki/Testing-BigQuery) with this info.
_To verify_
- CI: [this job](https://buildkite.com/hasura/graphql-engine-mono/builds/3770#89e5bac6-16fe-447e-bcda-85cd47ea1b77) successfully runs all tests on a temporary project & dataset
- local: follow [these steps](https://github.com/hasura/graphql-engine-mono/wiki/Testing-BigQuery#ci--optional-dedicated-gcp-project-for-tests) to try the same setup locally
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3240
GitOrigin-RevId: d88d9cb7922266bfa962cfcb481e0272b8929a5d
|
2022-01-25 08:28:06 +00:00 |
|
