<!-- The PR description should answer 2 important questions: -->
### What
We need to bump `tonic` because of `cargo audit`, and to do so we need
to bump OpenTelemetry, and to do that we need to finally bump `http`,
and to do that...
### How
Follow the breadcrumb trail, fix the errors.
Checked, trace propagation still works:
<img width="1781" alt="Screenshot 2024-10-02 at 15 56 28"
src="https://github.com/user-attachments/assets/4c3f8763-e646-445d-84de-f507da15ad2e">
V3_GIT_ORIGIN_REV_ID: aa219e91375aa310dc656183948edc33f77cdfdc
<!-- The PR description should answer 2 important questions: -->
### What
<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->
<!-- Consider: do we need to add a changelog entry? -->
<!-- Does this PR introduce new validation that might break old builds?
-->
<!-- Consider: do we need to put new checks behind a flag? -->
Write tests to confirm websocket connection behavior in conjunction with
[graphl-ws](https://github.com/enisdenjo/graphql-ws/blob/master/PROTOCOL.md)
subprotocol.
### How
<!-- How is it trying to accomplish it (what are the implementation
steps)? -->
Test the websocket by spinning up a server in an async tokio task. Use
tokio-tungstenite for websocket client.
V3_GIT_ORIGIN_REV_ID: 32c19298b6a5b23649b22d8d820ef8d47ef1d293
<!-- The PR description should answer 2 (maybe 3) important questions:
-->
### What
We have a bunch of local development infra for building the engine
inside a Docker container. This is helpful for Buildkite which doesn't
come with stuff like `cargo` preinstalled. We've not using Buildkite
anymore, let's remove it.
V3_GIT_ORIGIN_REV_ID: b4b7679aab5b14081288df25d139944f160a61fe
This PR updates as many tests as possible that use the custom connector
so that the tests run over two versions of the custom connector:
1. The custom connector in the repo, which currently speaks `ndc_models`
v0.2.x
2. The custom connector from the past (commit ), which is the last
version to speak `ndc_models` v0.1.x
This helps us test both the NDC v0.1.x and v0.2.x code paths. When the
postgres connector upgrades to v0.2.x, we can use the same approach as
in this PR to get the tests to run over multiple versions of the
postgres connector too, for much better coverage. This approach with the
custom connector will become less useful over time as the v0.1.x
connector is not updated and will diverge in data from the v0.2.x
connector. The postgres connector is likely to be longer-lasting, as it
is more stable.
The basic test used for `execute` integration tests is
`test_execution_expectation` (in `crates/engine/tests/common.rs`) and it
has been extended into a version called
`test_execution_expectation_for_multiple_ndc_versions` that takes
metadata on a per NDC version basis and then runs the test multiple
times, once for each NDC version. This allows one to swap out the
DataConnectorLink involved in the test to a different one that points at
either the v0.1.x or v0.2.x versions of the connector. The assertion is
that both connectors should produce the same results, even if they talk
a different version of the NDC protocol. As each version runs, we
`println!` the version so that if the test fails you can look in stdout
for the test and see which one was executing when it failed.
Tests that use the custom connector now use
`test_execution_expectation_for_multiple_ndc_versions` and run across
both connector versions. Some tests were unable to be used across both
version as the data between the two versions has changed. Some tests
were modified to avoid the changed data so as to support running across
both versions. Any tests that use `test_execution_expectation_legacy`
don't run across both versions because those tests aren't backed by the
same test implementation as
`test_execution_expectation_for_multiple_ndc_versions`.
Unfortunately the custom connector doesn't use the standard connector
SDK, so it doesn't support `HASURA_CONNECTOR_PORT`. This means that the
old connector is stuck on 8101. To work around this, I've moved the
current connector port to 8102 instead. Technically we might be able to
use docker to remap the ports, but then this binds us into always
running the connectors in docker in order to move their ports around, so
I avoided that approach.
Completes APIPG-703
V3_GIT_ORIGIN_REV_ID: fb0e410ddbee0ea699815388bc63584d6ff5dd70
<!-- The PR description should answer 2 (maybe 3) important questions:
-->
### What
We've had our CI mixed between Github and Buildkite for a while, it's
time to commit. First step is moving the "tests" step to Github Actions.
<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->
<!-- Consider: do we need to add a changelog entry? -->
### How
This PR:
- Moves the `test` step to Github Actions
- Creates a new `custom_connector.Dockerfile` which builds custom
connector only, more quickly.
- Changes the metadata tests to use `localhost` instead of their Docker
internal names (ie `custom_connector` or `postgres_connector`) - this is
because the tests are being run from outside Docker now
- Removes the `test` Buildkite step
It does not:
- Remove the code coverage or benchmarks steps from Buildkite
- Tidy up `justfile` or Dockerfiles
<!-- How is it trying to accomplish it (what are the implementation
steps)? -->
---------
Co-authored-by: Philip Lykke Carlsen <plcplc@gmail.com>
V3_GIT_ORIGIN_REV_ID: a67534ebc1634a24b48d2620c45003221852e199
Enable HTTP/2 support for Axum, so that we can make use of request
multiplexing from the engine in the future.
I also added an `env_logger` to the custom connector so we can see
what's going on there.
V3_GIT_ORIGIN_REV_ID: 3082b988ae3d149ee548f94638308876734b26df
This splits out a `debug.Dockerfile` which makes use of out-of-band
caching to speed up builds drastically, at the expense of
reproducibility.
It is used to run tests and auxiliary test services (i.e. the custom
connector).
The new `debug.Dockerfile` marks the Cargo dependency and build caches
as Docker caches, which means they are shared between builds. This is
probably fine for local work and testing. The `Dockerfile` continues to
not use a cache like this, to guarantee that it is not polluted by extra
information, at the expense of build speed.
In addition, we build a `nextest` archive ahead of time to avoid
building tests when attempting to run them.
On my machine, a re-run of `just test` now takes seconds.
I have also sped up the `postgres` container start time by creating a
database called "finished" last, and then waiting for that to show up.
V3_GIT_ORIGIN_REV_ID: 7ef0548361987175b68a0cad44c8f2295110a1fb
<!-- Thank you for submitting this PR! :) -->
## Description
Set our `ndc-postgres` connector in tests to use new mutations versions
so we can test Boolean Expressions. Also does some house-keeping, like
ensuring we pull the latest `ndc-postgres` in CI and exposing `8080`
from `ndc-postgres` to fix local dev flow.
V3_GIT_ORIGIN_REV_ID: 4c92670e9976a3f75ec31e1224079799380ef6e2
This seems appropriate now that we've stabilized the new configuration.
Of note are the configuration updates and the use of an environment
variable to specify the connection URI. This upgrade also fixes the
health checks.
Regenerating the configuration lost the table descriptions, which seems
to be because they were not present in the Chinook SQL. I have dragged
the Chinook SQL in from ndc-postgres and kept it separate from the
initialization of other tables.
The auto-generated configuration is slightly different from the
manually-created configuration in that the collection names are
singular, not plural. This means that I had to change a lot of test
metadata files too.
V3_GIT_ORIGIN_REV_ID: 2b66fd3049aaf4daeb386915ea3b64a209b1f393
- Use `rust-toolchain.toml` to specify tools.
- Drop the unused version field from the compose files.
- Use a modern Docker base for the dev-auth-webhook.
- Handle signals in the test servers so `docker stop` is quick.
- Upgrade Jaeger.
V3_GIT_ORIGIN_REV_ID: b5fee1d5dc953a1fdb0aabd75ba02df6846b7518
<!-- Thank you for submitting this PR! :) -->
## Description
Missed this in https://github.com/hasura/v3-engine/pull/355, apologies.
V3_GIT_ORIGIN_REV_ID: 360b4a3a218f6ba754814c5b3083a8151a7886fb
I got frustrated by the lack of useful output when services failed to
start in CI, so I thought I'd mess around.
I made some changes so that:
1. we wait for services with health checks to report as healthy,
2. we do not rebuild the engine or custom connector when they start,
3. the health checks work (apart from ndc-postgres, which will come
later), and
4. we use environment variables rather than command-line arguments where
possible.
I have also renamed the "agent" binary to "custom-connector", matching
its crate, because it was driving me a little crazy.
V3_GIT_ORIGIN_REV_ID: 8d672b0b25438b54d47368ce82cd236cfdd4e554
<!-- Thank you for submitting this PR! :) -->
## Description
This moves all the crates into a `/crates` folder. Everything appears to
just work, thanks Cargo!
V3_GIT_ORIGIN_REV_ID: 8e3ef287b1a46cabdb4d919a50e813ab2cddf8b1
