We seem to be getting flakes where we try and use the same port for two different servers. This is because in certain cases we cannot simply allocate the port dynamically, but have to decide it in advance, leading to a race condition.
We resolve this by keeping track of the ports we allocate when using this method, making sure we never allocate them twice. We also make sure we allocate from a different pool of ports to the usual dynamic port pool (typically above port 32768, and often above port 49152).
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8903
GitOrigin-RevId: 375a23867591a4566493dddbc550c58cf88ea392
I'm hoping this will reduce the flakiness of the tests, by ensuring the servers stop appropriately.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8880
GitOrigin-RevId: 6379b18cbe43527a1c2e73a14d93179b54bf1b75
These tests are intended to catch issues in upgrading HGE. However:
* the tests are very convoluted and hard to understand,
* we can only run a small subset of Python tests that don't mutate any data or metadata, and
* I have never seen them fail for a legitimate reason, but I've seen a lot of flakes.
While we do believe it's important to test that upgrades don't break the stored introspection, these tests don't seem to be doing that any more. I humbly request that we delete them now and either (a) figure out how to test this properly, or (b) just wait for v3, which does away with reintrospecting on server startup entirely.
[NDAT-259]: https://hasurahq.atlassian.net/browse/NDAT-259?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8844
GitOrigin-RevId: 528bc632fce377b7eff2026b832bd58586ac5a0b
This rewrites the JWT tests to generate and specify the secrets per test class, and to provide the server configuration to the HGE fixture.
It covers the tests in:
- *test_jwt.py*
- *test_jwt_claims_map.py*
- *test_config_api.py*
- *test_graphql_queries.py* (just a couple here)
This does reduce the number of code paths exercised with JWT, as we were previously running *all* tests with JWT tokens. However, this seems excessive; we don't need to tread every code path, just enough to ensure we handle the tokens appropriately. I believe that the test coverage in *test_jwt.py* does this well enough (though I'd prefer if we moved the coverage lower down in the stack as unit tests).
These tests were configured in multiple different ways by *test-server.sh*; this configuration is now moved to test subclasses within the various files. This results in a bit of duplication.
Unfortunately, the tests would ideally use parameterization rather than subclassing, but that doesn't work because of `hge_fixture_env`, which creates a "soft" dependency between the environment variables and `hge_server`. Parameterizing the former *should* force the latter to be recreated for each new set of environment variables, but `hge_server` isn't actually aware there's a dependency.
It currently looks like this adds lines of code; we'll more than make up for it when we delete the relevant lines from *test-server.sh*. I am not doing that here because I plan on deleting the whole file in a subsequent changeset.
[NDAT-538]: https://hasurahq.atlassian.net/browse/NDAT-538?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8803
GitOrigin-RevId: f7f2caa62de0b0a45e42964b69a8ae73d1575fe8
This adds the ability to capture logs to the HGE fixture, and uses this in test_logging.py to analyze the logs, instead of relying on a shell script redirecting the logs to a file.
We then inject the logs into the tests and parse the JSON. Because we're no longer reading a file, we need to do this in a separate thread, as we'll block on reading rather than the stream ending. (Once HGE stops, the stream will be closed.)
Some of the tests require a JWK server, so this has been extracted from test_jwk.py.
[NDAT-540]: https://hasurahq.atlassian.net/browse/NDAT-540?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8772
GitOrigin-RevId: 9413e714f1c42b8a0991d0d30c4358209fd30c0c
This requires rewriting the test class to split it into 3, each specifying the correct environment variables for HGE.
It would be lovely to use parameterization rather than subclassing, but that doesn't work because of `hge_fixture_env`, which creates a "soft" dependency between the environment variables and `hge_server`. Parameterizing the former *should* force the latter to be recreated for each new set of environment variables, but `hge_server` isn't actually aware there's a dependency. See `TestParameterizedFixtures` in test_tests.py for more information.
[NDAT-539]: https://hasurahq.atlassian.net/browse/NDAT-539?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8747
GitOrigin-RevId: 878b2fc20f39f962a67cd950046a99c283cfc6fc
We were previously using the Docker Compose file in the root directory
for manual testing _and_ the server API tests.
This splits them so we can e.g. add Yugabyte for easy manual testing.
In the future, this will also allow us to use ephemeral ports for API
test databases, while keeping the fixed ports for manual testing.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7524
GitOrigin-RevId: 7244e296b0ed0ace9782b6f44f321933a9d9a49d
This rewrites the last couple of Python tests that were failing when run with a separate HGE binary per test class. The changes are as follows:
1. The event triggers tests, naming conventions tests, and subscriptions tests all generate a new source DB per test, so can run in parallel.
2. The scheduled triggers tests use the correct URL for the trigger service when the port is generated randomly.
3. Whitespace and trailing commas are added to the scheduled triggers tests.
4. Support for SQL Server is added to _hge.py_ so the naming conventions test that runs on SQL Server passes. (The other SQL Server tests do not pass and we're not going to bother with them for now.)
5. Container names are fixed in _run.sh_.
6. _run.sh_ and _run-new.sh_ don't pull images explicitly as it's annoying when running tests a lot. If you want to pull the latest versions, just run `docker compose pull` from the _server/tests-py_ directory, or the root directory. (If you don't have the images at all, they'll still be pulled automatically.)
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7350
GitOrigin-RevId: db58f310f017b2a0884fcf61ccc56d15583f99bd
We no longer support this and therefore don't run tests against it.
This also refactors the code a little so it doesn't have to skip running a PostgreSQL-specific test against MS SQL Server.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7201
GitOrigin-RevId: 307c2ab0052162c012f7b1c55866b57f2fa6d9a6
This test did not work when splitting the metadata and source backends. Fixed mostly by running the relevant SQL using `source_backend.engine`, but I also took the time to clean it up a little, and broke up _test.yaml_ into 3 files.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6957
GitOrigin-RevId: bbca60a8906caba2d0cffd834b3b8595fca058fd
Rather than varying it, let's just use `postgis/postgis` everywhere.
This uses the latest version of PostGIS, in which some of the raster codes have changed. This seems benign (it's just one digit) in the hex stream. I can't find the relevant release notes though.
Also syncs _images.go_ and _databases.yaml_ so we use the same thing where possible.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6903
GitOrigin-RevId: bb5c56f2e7ff69e4c008f1d658850af08c96badc
When running using the "new" style (with a HGE binary, not a URL), a new PostgreSQL metadata and source database are created for each test. When we get this into CI, this should drastically reduce the flakiness.
I have also enabled parallelization by default when using `run-new.sh`. It's much faster.
I had to basically rewrite _server/tests-py/test_graphql_read_only_source.py_ so that it does two different things depending on how it's run. It's unfortunate, but it should eventually go away.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6879
GitOrigin-RevId: a121b9035f8da3e61a3e36d8b1fbc6ccae918fad
This enables sharing the Docker Compose-based database configuration across the Haskell-based API tests and the legacy Python integration tests.
Why? Because we depend on different database versions and I keep running out of disk space. I am far too lazy to buy another disk and set up my operating system _again_.
The files in question are:
- _docker-compose/databases.yaml_, which is the base specification for the databases
- _docker-compose.yml_, used by the API tests locally (and for other manual testing), which extends the above
- _.buildkite/docker-compose-files/test-oss-server-hspec.yml_, used by the API tests in CI, which extends _databases.yaml_
- _server/tests-py/docker-compose.yml_, used by the Python integration tests
The changes are summarized as follows:
1. The following snippets are moved from _docker-compose/databases.yaml_ to _docker-compose.yml_ and _.buildkite/docker-compose-files/test-oss-server-hspec.yml_, as they're not strictly necessary for other forms of testing:
- the fixed port mappings (in the range 65000–65010)
- the PostgreSQL initialization
- the SQL Server initialization
2. Environment variables are used a little more in health checks and initialization scripts, as usernames, passwords, etc. can be overridden.
3. The volumes in _docker-compose/databases.yaml_ are made anonymous (unnamed), and the names are only specified in _docker-compose.yml_. We don't need to do this elsewhere.
- For extra fun, I have removed all named volumes from the CI Docker Compose files, as they seem to be unnecessary.
4. _server/tests-py/docker-compose.yml_ now depends on _docker-compose/databases.yaml_.
- This was the point.
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6864
GitOrigin-RevId: f22f2839716f543ce8a62f890da244de7e23abaa
