graphql-engine/v3
Daniel Harvey 774bf78786 Tidy up metadata resolve (#527)
<!-- Thank you for submitting this PR! :) -->

## Description

This stacks on top of https://github.com/hasura/v3-engine/pull/526 and
is the tidying up step following it.

There is a lot going on here, none of it particularly interesting.

- We remove the old `resolve_metadata` function, so all of "the action"
for resolving metadata happens inside `resolved::stages::mod`.
- We move the remaining modules into two folders, `helpers` and `types`.
This is a pretty rough cut, and there are definitely better homes for
the things here, I just wanted the separation of stages and "other bits"
to be clearer.
- We split the `types` helpers into `types` and `type_mappings`, as the
`type_mappings` stuff was clearly defined and didn't belong with the
grab-bag of functions that remain in `types`.
- We explicitly export everything used outside metadata resolve, and
stop other modules dipping into it's internal structure. This will make
future changes (particularly refactors that change the stages etc) a lot
easier as we don't need the rest of the app concerning itself with it.
- I _think_ we may now be able to move metadata resolve into a separate
crate. I have not tried yet, this PR is already silly enough.

When given a choice between "fixing" something, and doing the most
mechanical obvious change, I have opted for the latter. That is to
derisk this big change, and to ensure it doesn't linger, collecting
awful merge conflicts.

Functional no-op.

V3_GIT_ORIGIN_REV_ID: 2b52403cbfddb0427b8a3e61ad2edaef9d1b3b46
2024-04-30 10:56:28 +00:00
..
.cargo v3: open-source hasura v3 engine 2023-12-19 09:05:39 +00:00
crates Tidy up metadata resolve (#527) 2024-04-30 10:56:28 +00:00
docs docs: attempt to document roles / annotations (#358) 2024-04-10 08:58:13 +00:00
nix create Docker images with Nix (#507) 2024-04-26 08:44:04 +00:00
rfcs add boolean expression types to the metadata (#331) 2024-03-12 00:29:28 +00:00
.dockerignore Include the dev-auth-webhook crate in the workspace. (#500) 2024-04-24 08:12:37 +00:00
.envrc basic Nix flake for deps (#333) 2024-03-06 11:39:19 +00:00
.envrc.example basic Nix flake for deps (#333) 2024-03-06 11:39:19 +00:00
.gitignore Set the crate properties once, to improve the Nix build. (#334) 2024-03-06 17:15:13 +00:00
auth_config.json v3: open-source hasura v3 engine 2023-12-19 09:05:39 +00:00
benchmark.sh Fix the benchmarks. (#494) 2024-04-23 09:20:18 +00:00
Cargo.lock Bump strum_macros from 0.25.3 to 0.26.2 (#514) 2024-04-29 10:11:26 +00:00
Cargo.toml Do not include debug symbols in release builds. (#493) 2024-04-23 09:42:45 +00:00
ci.docker-compose.yaml Speed up debug builds in Docker. (#484) 2024-04-22 13:43:29 +00:00
CONTRIBUTING.md v3: open-source hasura v3 engine 2023-12-19 09:05:39 +00:00
coverage.sh v3: open-source hasura v3 engine 2023-12-19 09:05:39 +00:00
debug.Dockerfile Fix the benchmarks. (#494) 2024-04-23 09:20:18 +00:00
dev-auth-webhook.Dockerfile Include the dev-auth-webhook crate in the workspace. (#500) 2024-04-24 08:12:37 +00:00
docker-compose.yaml Speed up debug builds in Docker. (#484) 2024-04-22 13:43:29 +00:00
Dockerfile Fix the benchmarks. (#494) 2024-04-23 09:20:18 +00:00
drill.yml v3: open-source hasura v3 engine 2023-12-19 09:05:39 +00:00
flake.lock Nix: Switch to nixpkgs-unstable, rather than nixos-unstable. (#407) 2024-03-26 15:38:33 +00:00
flake.nix create Docker images with Nix (#507) 2024-04-26 08:44:04 +00:00
justfile create Docker images with Nix (#507) 2024-04-26 08:44:04 +00:00
README.md Include the dev-auth-webhook crate in the workspace. (#500) 2024-04-24 08:12:37 +00:00
rust-toolchain.toml Improvements to Docker configuration. (#414) 2024-03-28 08:52:55 +00:00

Hasura GraphQL Engine V3

Docs

Hasura V3 is the API execution engine, based over the Open Data Domain Specification (OpenDD spec) and Native Data Connector Specifications (NDC spec), which powers the Hasura Data Delivery Network (DDN). The engine expects to run against an OpenDDS metadata file and exposes a GraphQL endpoint according to the specified metadata. The engine needs a data connector to run alongside, for the execution of data source specific queries.

Data connectors

Hasura V3 engine does not execute queries directly - instead it sends IR (Abstracted, intermediate query) to NDC agents (aka data connectors). To run queries on a database, we'll need to run the data connector that supports the database.

Available Data connectors are listed at the Connector Hub

For local development, we use the reference agent implementation that is a part of the NDC spec.

To start the reference agent only, you can do:

docker compose up reference_agent

and point the host name reference_agent to localhost in your /etc/hosts file.

Run V3 engine (with reference agent)

Using cargo

Hasura V3 engine is written in rust, hence cargo is required to build and run V3 engine locally.

To start the v3 engine locally, we need a metadata.json file and an auth config file.

Following are steps to run V3 engine with a reference agent (read only, in memory, relational database with sample tables), and an sample metadata file, exposing a fixed GraphQL schema. This can be used to understand the build setup and the new V3 engine concepts.

RUST_LOG=DEBUG cargo run --release --bin engine -- \
  --metadata-path crates/open-dds/examples/reference.json \
 --authn-config-path auth_config.json

A dev webhook implementation is provided in dev-auth-webhook, that exposes the POST /validate-request which accepts converts the headers present in the incoming request to a object containing session variables, note that only headers that start with x-hasura- will be returned in the response.

The dev webhook can be run using the following command:

docker compose up auth_hook

and point the host name auth_hook to localhost in your /etc/hosts file.

Open http://localhost:3000 for GraphiQL.

Use --port option to start v3-engine on a different port.

RUST_LOG=DEBUG cargo run --release --bin engine -- \
     --port 8000 --metadata-path crates/open-dds/examples/reference.json

Now, open http://localhost:8000 for GraphiQL.

With docker

You can also start Hasura V3 engine, reference_agent, dev Authentication webhook and jaegar for tracing (accessible at localhost:4002) using docker (without the need of using cargo)

METADATA_PATH=crates/open-dds/examples/reference.json AUTHN_CONFIG_PATH=auth_config.json docker compose up

Run V3 engine (with Postgres)

NDC Postgres is the official connector by Hasura for Postgres Database. For running V3 engine for GraphQL API on Postgres, you need to run NDC Postgres Connector and have a metadata.json file that is authored specifically for your Postgres database and models (tables, views, functions).

The recommended way to author metadata.json for Postgres, is via Hasura DDN.

Follow the Hasura DDN Guide to create a Hasura DDN project, connect your cloud or local Postgres Database (Hasura DDN provides a secure tunnel mechanism to connect your local database easily), and model your GraphQL API. You can then download the authored metadata.json and use the following steps to run GraphQL API on your local Hasura V3 engine.

Steps to run metadata with V3 engine locally

  1. Download metadata from DDN project, using Hasura V3 CLI

    hasura3 build create --dry-run > ddn-metadata.json
    
  2. Following steps are to generate Postgres metadata object and run the Postgres Connector. These steps refer to the NDC Postgres repository:

    1. Start the Postgres connector in configuration mode (Config server). A config server provides additional endpoints for database instrospection and provide the schema of the database. Output of the config server will form the Postgres Metadata object.

    2. Run the following command in the ndc-postgres repository:

      just run-config
      
    3. Generate the postgres configuration using the new-configuration.sh script by running the following command (in another terminal) in the ndc-postgres repository:

      ./scripts/new-configuration.sh localhost:9100 '<postgres database url>' > pg-config.json
      
    4. Now shutdown the postgres config server and start the Postgres Connector using the pg-config.json generated in the above step, by running the following command:

      Please specify different PORT for different data connectors:

      PORT=8100 \
      RUST_LOG=INFO \
          cargo run --bin ndc-postgres --release -- serve --configuration pg-config.json > /tmp/ndc-postgres.log
      
    5. Fetch the schema for the data connector object by running the following command:

      curl -X GET http://localhost:8100/schema | jq . > pg-schema.json
      
    6. Finally, generate the DataConnector object:

      jq --null-input --arg name 'default' --arg port '8100' --slurpfile schema pg-schema.json '{"kind":"DataConnector","version":"v2","definition":{"name":"\($name)","url":{"singleUrl":{"value":"http://localhost:\($port)"}},"schema":$schema[0]}}' > pg-metadata.json
      
  3. Now you have the NDC Postgres connector running, and have obtained the Postgres metadata (pg-metadata.json) which is required for the V3 engine.

  4. In ddn-metadata.json (from step 1.), replace the HasuraHubDataConnector objects with DataConnector objects generated inside the pg-metadata.json file.

  5. Remove the object for kind: AuthConfig from ddn-metadata.json, move it to a separate file auth_config.json, and remove the kind field from it.

  6. Remove the object for kind: CompatibilityConfig from ddn-metadata.json. If desired, a flags field can be added to the OSS metadata to enable the flags corresponding to that compatibility date in the DDN metadata.

  7. Finally, start the v3-engine using the modified metadata using the following command (using the modified ddn-metadata.json and auth_config.json from Step 5):

    RUST_LOG=DEBUG cargo run --release --bin engine -- \
     --metadata-path ddn-metadata.json auth_config.json
    

    You should have the v3-engine up and running at http://localhost:3000

Note: We understand that these steps are not very straightforward, and we intend to continuously improve the developer experience of running OSS V3 Engine.

Running tests

To run the test suite, you need to docker login to ghcr.io first:

docker login -u <username> -p <token> ghcr.io

where username is your github username, and token is your github PAT. The PAT needs to have the read:packages scope and Hasura SSO configured. See this for more details.

Next run the postgres NDC locally using docker compose up postgres_connector and point the host name postgres_connector to localhost in your /etc/hosts file.

Next run the custom NDC locally using docker compose up custom_connector and point the host name custom_connector to localhost in your /etc/hosts file OR you can run cargo run --bin agent and then do cargo test.

Testing/Development with the chinook database

The crates/engine/tests/chinook contains static files required to run v3-engine run with the chinook database as a data connector.

To get this running, you can run the following command:

METADATA_PATH=crates/engine/tests/schema.json AUTHN_CONFIG_PATH=auth_config.json docker compose up postgres_connector engine

If you are running the v3-engine locally through cargo, then you'll need to update the value of the singleUrl present in crates/engine/tests/chinook/schema.json** from "http://postgres_connector:8080"to"http://localhost:8100"`.

Running tests with a single command

Alternatively, the tests can be run in the same Docker image as CI:

just test

Updating goldenfiles

There are some tests where we compare the output of the test against an expected golden file. If you make some changes which expectedly change the goldenfile, you can regenerate them like this:

Locally (with postgres_connector pointing to localhost)

  REGENERATE_GOLDENFILES=1 cargo test

Docker:

  just update-golden-files

Running coverage report

We can check for coverage of unit tests by running:

just coverage

You can also give a filter expression (which is passed to grep -E) to give coverage only for matched files:

just coverage "open-dds|engine"

Run benchmarks

The benchmarks operate against the reference agent using the same test cases as the test suite, and need a similar setup.

To run benchmarks for the lexer, parser and validation:

cargo bench -p lang-graphql "lexer"
cargo bench -p lang-graphql "parser"
cargo bench -p lang-graphql "validation/.*"

Alternatively, the benchmarks can be run in the same Docker image as CI:

just ci-bench