Commit Graph

42 Commits

Author SHA1 Message Date
Samir Talwar
38c0e1470b CI: Fix the Rust cache. (#942)
### What

Try not to pollute the CI cache with the wrong thing.

### How

1. Remove the package selector as we don't care about production builds
here any more.
2. Tell the test build to not save the cache so there is no write
contention.
3. Downgrade `mockito` to v1.4 reduce the size of the cache, because
`v1.5` depends on `http` v1.

V3_GIT_ORIGIN_REV_ID: 2109c8c7db5d80e3b2c29d2949423e8faebd10b2
2024-08-07 15:34:22 +00:00
Daniel Harvey
07f0a90332 Split out IR crate (#909)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

`execute` is now the biggest `crate` in engine and does a lot, let's
split it into it's constituent steps.

Functional no-op.

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

Split out `ir` crate from the `execute` crate. Replace export of entire
modules with that of specific types / functions. Therefore, consumers
outside the crate talk about `ir::CommandInfo` rather than
`ir::command::CommandInfo`. There is no need for other crates to know
about the internal structure of this crate.

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

V3_GIT_ORIGIN_REV_ID: 47553aec63e80af7f95e659a170a2685e9ac2ce3
2024-07-30 15:03:49 +00:00
Daniel Chambers
f84c2f3695 Validate that the capabilities version matches the DataConnectorLink schema version (#880)
This PR adds validation code to `metadata_resolve` that prevents someone
from putting a schema/capabilities from the wrong NDC version into the
DataConnectorLink while specifying a different schema version in the
DataConnectorLink. For example:

```
kind: DataConnectorLink
version: v1
definition:
  name: data_connector
  schema:
    version: v0.2
    schema: {}
    capabilities:
      version: 0.1.5 # Not allowed for version v0.2!
      capabilities: {}
```

This PR has two commits. One is a refactor where we rearrange the
DataConnectorError types so that the name of the data connector is
captured centrally in `NamedDataConnectorError`, so that it doesn't have
to be passed around and included in every error manually. The other is
the validation changes to `metadata_resolve`.

Completes APIPG-705

V3_GIT_ORIGIN_REV_ID: baed571f36f4cbed824ca546128f5df360d5b298
2024-07-25 14:31:28 +00:00
Daniel Chambers
01539cd7c1 Upgrade ndc_models to v0.2.0-rc0 (#835)
This PR adds true support for ndc_models v0.2.0 to v3-engine. Note that
v0.2.0 is not finalized yet, so we're pointing at v0.2.0-rc0. The
support still comes via the migration methodology, where v0.2.x ndc
models are downgraded to v0.1.x to support backwards compatibility. In
the future we want to remove this and have the engine generate the
different versioned ndc models separately instead of performing a
migration.

The ndc_models_v01 crate reference has been bumped to the official
v0.1.5 version, which brings the newtypes to the v0.1.x version. The
ndc_models crate reference is now on v0.2.0-rc0.

The custom connector has been updated to support ndc-spec v0.2.0. All
tests that talk to the custom connector have been updated with its
latest v0.2.0 schema/capabilities.

In `metadata_resolve` the v01->v02 schema/capabilities migration code
has been updated to handle the new v0.2.0 types. This includes inferring
v0.2.0 capabilities from what was possible in v0.1.x.

In `execution`, the migration code has been updated to deal with the new
v0.1.5 newtypes and v0.2.0 types. This means there are now cases where a
downgrade is impossible and produces an error (see `NdcDowngradeError`
in `execute::ndc::migration`). A bug has also been fixed where NDC
expressions in arguments were not being serialized to the correct NDC
version.

V3_GIT_ORIGIN_REV_ID: 5b4afcde64c307b2bd7c985c588d6c74d9623a0f
2024-07-16 01:53:42 +00:00
Daniel Chambers
862ad8e0dc Update ndc-models to latest version which includes new name newtypes (#791)
This PR updated ndc-models to the latest version on main. This version
is still a 0.1.x version, but it now includes all the [new
newtypes](https://github.com/hasura/ndc-spec/pull/156) that wrap
previously stringly-typed things. For example, `ArgumentName`,
`FieldName`, etc.

This pervades across the entire engine, but thankfully the changes are
mostly mechanical repetitive changes. Usually you will see conversions
from `String`-typed variables into the newtypes using this sort of form:
`FieldName::from(string.as_str())`, which is the most efficient way
copying the value (the str slice is copied). Or you will see usages of
the newtype as a raw string by `.as_str()`-ing it. Converting the
newtypes into a String can be done with `.into()` if owned, but if
referenced `.as_str().to_owned()` performs the clone and type
conversion.

Other changes:
* A few minor instances of `ok_or()` usages (or similar) have been
converted into lazy error construction variants (eg `ok_or_else()`)

V3_GIT_ORIGIN_REV_ID: 64a371ae6197ef3be98a6f7cdc4052d654a43da0
2024-07-04 08:58:12 +00:00
paritosh-08
08014cab88 pre-execution plugin middleware (#771)
The RFC:
https://docs.google.com/document/d/1NB9fA6J8_dKtWknJkfTiqN5qWhPVDfYVOKFK25gYE7Y/edit

The JIRA: https://hasurahq.atlassian.net/browse/V3ENGINE-234

<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

This PR adds a new middleware to the `/graphql` endpoint handler. This
new middleware will be used to handle the pre-execution plugins.

### How

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

We are doing something similar to
https://github.com/tokio-rs/axum/blob/axum-v0.6.20/examples/consume-body-in-extractor-or-middleware/src/main.rs

V3_GIT_ORIGIN_REV_ID: a7e0a7a252efd1f266b3e90df23a8307a9b35fc7
2024-07-03 12:58:26 +00:00
Daniel Chambers
a72d45d165 Add support for multiple versions of ndc-spec to Open DD and metadata-resolve (#781)
This PR introduces support for multiple versions of the ndc-spec by
adding a new `VersionedSchemaAndCapabilities` enum variant under the
`DataConnectorLink` in OpenDD. This allows the capture of both ndc
v0.1.* and v0.2.* schema and capabilities.

This is achieved by referencing the `ndc-models` crate twice, once for
`v0.1.4` and once for the first commit after `v0.1.4`. That commit was
chosen to avoid actual v0.2.0 breaking changes for now, while we lay in
this multiple version support plumbing. Future PRs will use a newer
commit and adopt the breaking changes where necessary. The
`VersionedSchemaAndCapabilities::V02` variant uses the the v0.2
reference of `ndc-models`.

Then, during metadata resolve, when we resolve the
`DataConnectorContext` from `DataConnectorLink`, we perform a migration
of v0.1 types to v0.2 types and store and use the v0.2 types during
metadata resolve. This migration is performed in the new module
`ndc_migration`. We also record the `NdcVersion` (either `V01` or `V02`)
in the `DataConnectorLink`. The `execute` crate will need to use this to
determine which version to send to the connector at runtime (to be
implemented in a future PR).

The new changes to OpenDD are hidden from the JSON Schema via a new
`UnstableFeatures` flag, and the use of the new variant is gated behind
it in metadata resolve, since we don't yet support it upstream in the
`execute` crate.

V3_GIT_ORIGIN_REV_ID: d6d8a768ea3537c0b5e620799e94d3dd1e529526
2024-06-28 11:57:41 +00:00
Daniel Harvey
64657639d4 Use Rust 1.79.0 (#751)
<!-- Thank you for submitting this PR! :) -->

## Description

Upgrade Rust, as a treat. Functional no-op.

---------

Co-authored-by: Samir Talwar <samir@functional.computer>
V3_GIT_ORIGIN_REV_ID: 1e0014049e89b8658326c8d8f652df800c415526
2024-06-27 22:22:01 +00:00
Philip Lykke Carlsen
0ea629ae59 Stdout otel exporter (#776)
### What

Output all traces to stdout.

---------

Co-authored-by: Daniel Harvey <danieljamesharvey@gmail.com>
V3_GIT_ORIGIN_REV_ID: 06330076ca305a331996530ddcd4d4c13d46bd95
2024-06-27 14:45:05 +00:00
Samir Talwar
5d619a540f Replace lazy_static with stdlib equivalents. (#758)
### What

The `lazy_static` macro is poorly maintained, fairly bloated, and has
been mostly superseded by
[`OnceLock`](https://doc.rust-lang.org/stable/std/sync/struct.OnceLock.html)
in the stdlib.

### How

1. I turned a couple of `static ref` values into `const`, sometimes by
creating `const fn` equivalents to other functions.
2. I inlined static behavior to construct a JSON pointer into some
tests, where we don't care too much about losing a few milliseconds.
3. For the rest, I replaced `lazy_static` with a `static OnceLock` and a
call to `OnceLock::get_or_init`.

V3_GIT_ORIGIN_REV_ID: 18e4150a5fb24fe71f6ed77fe6178b7942405aa3
2024-06-26 12:45:41 +00:00
Samir Talwar
63ac02bc07 Take snapshots of passing and failing metadata resolution. (#763)
### What

In order to more easily monitor and review changes to metadata
resolution, this introduces snapshot testing for both successful and
failing calls to `resolve`. I used [Insta](https://insta.rs/) for this.

### How

For tests of the failure case, we already had a text file with the
expected error, so I have turned those files into snapshot files. I
wrote a small script to move the files rather than deleting and
recreating them so I could guarantee that the contents have not changed.
(Unfortunately, Git's diff doesn't always recognise the move as a move
because Insta has added a header.)

For tests of the successful case, I added a line to snapshot the
metadata rather than discarding it.

I also rewrote the tests to use `insta::glob` so we could get rid of
`test_each`.

V3_GIT_ORIGIN_REV_ID: 41bef4cf77bddb8d20d7c101df52ae149e8b0476
2024-06-26 08:15:29 +00:00
Vamshi Surabhi
8af78227a0 Experimental SQL interface (#742)
Adds a very experimental SQL interface to v3-engine for GenAI use cases.

---------

Co-authored-by: Abhinav Gupta <127770473+abhinav-hasura@users.noreply.github.com>
Co-authored-by: Gil Mizrahi <gil@gilmi.net>
Co-authored-by: Anon Ray <ecthiender@users.noreply.github.com>
V3_GIT_ORIGIN_REV_ID: 077779ec4e7843abdffdac1ed6aa655210649b93
2024-06-25 18:46:39 +00:00
Samir Talwar
1378730d43 Remove redundant clones. (#752)
I noticed a few extra calls to `.clone()` while working on an unrelated
refactor. I want to remove them for brevity and simplicity; I don't
expect a performance improvement.

This turns on the Clippy warning `redundant_clone`, which detects
unnecessary calls to `.clone()` (and `.to_string()`).

It is an unstable warning and so might reports some false positives. If
we find any, we can suppress the warning there.

V3_GIT_ORIGIN_REV_ID: a713f29cf862d6f4cb40300105c6b9f96df00676
2024-06-24 14:09:02 +00:00
Daniel Harvey
eba79698d5 Ban println! in code with clippy (#711)
<!-- Thank you for submitting this PR! :) -->

## Description

A few debug lines slipped in recently, let's make `clippy` `warn` on
those, so they are kicked out by CI. Functional no-op.

V3_GIT_ORIGIN_REV_ID: 290f6de35f9315b68811eb5f15969fb0333e9d06
2024-06-24 11:46:53 +00:00
Samir Talwar
726d5d8bd3 Bump ndc-spec to v0.1.4. (#735)
V3_GIT_ORIGIN_REV_ID: 74e7ca646cab98569c455e497a9c211930900a77
2024-06-20 12:49:07 +00:00
Samir Talwar
783aec942d Move all dependency versions into the workspace file. (#734)
This keeps versions in one place so we can more easily ensure we upgrade
crates together.

V3_GIT_ORIGIN_REV_ID: 6a929bb6196c19a1f66a768585b669127035e9be
2024-06-20 12:25:55 +00:00
Samir Talwar
5ad33bd6a5 Avoid returning a Result when it is unnecessary. (#726)
Return a `T` instead of a `Result<T, E>` when we never return an error
(`E`) case.

I also enabled some more warnings. `unnecessary_box_returns` has been
suppressed where appropriate, and `unused_async` doesn't seem to be
violated anywhere any more.

I got rid of some calls to `.unwrap()` too.

V3_GIT_ORIGIN_REV_ID: 015ebd05978cf8c2d87474a90e0cd4333779a761
2024-06-18 12:03:04 +00:00
Samir Talwar
5df3e6fac4 Inline format arguments into the format string when possible. (#703)
V3_GIT_ORIGIN_REV_ID: 50d6a12eefbfcc6b217d226759856e957fac0f4b
2024-06-12 11:25:38 +00:00
Samir Talwar
0c6d6a67d2 Semicolons, everywhere! (#700)
If a function doesn't return a value, terminate with a semicolon.

I also moved `implicit_hasher` and `return_self_not_must_use` to the
"definitely keep disabling this" list, and installed
[Bacon](https://dystroy.org/bacon/) in the Nix shell to make it easier
to run Clippy.

V3_GIT_ORIGIN_REV_ID: ffb17b42d982518aec433a1676dba0a0dd0ad95d
2024-06-11 15:33:32 +00:00
Samir Talwar
ca830c05fb Be more explicit about clones, and remove a few. (#671)
Calling `.to_owned()` on a reference, `.to_vec()` on a vector reference,
etc. are just synonyms for `.clone()` which are less explicit about
cloning. Let's be explicit.

This also removes some unnecessary clones.

V3_GIT_ORIGIN_REV_ID: 1bc00c4106f0346303d73e4268c89030c0ce93fc
2024-06-05 13:26:33 +00:00
Samir Talwar
7281322f51 Enable a few lint warnings around conditions, and fix the highlighted areas. (#662)
1. Use `map_or(…, …)` instead of `.map(…).unwrap_or(…)`.
2. Use `.is_some_and(…)` instead of `.map(…).unwrap_or_default(…)`.
3. Nest `|` patterns where possible.
4. Be more specific about match patterns.

I found I could also simplify `typecheck_qualified_type_reference`
considerably.

V3_GIT_ORIGIN_REV_ID: 6a3b1a4c525c0187c2fdb6df0c979ca0b7b3016c
2024-06-05 08:42:03 +00:00
Samir Talwar
40e23db51e Apply a few lints to make it struct construction clearer. (#661)
1. Always name the struct when calling `default()`.
2. Sort construction according to the definitions.
3. Approve allowing `struct_field_names` because it doesn't seem to be
helpful.
4. Enable `manual_string_new`; nothing seems to be triggering it now.

V3_GIT_ORIGIN_REV_ID: 868742114b0bf27bc3ea03cdf1e63a0f710ebe33
2024-06-04 08:42:48 +00:00
Samir Talwar
d8277974a8 Move a couple of lint suppressions closer to the code. (#658)
Rather than allowing the `cast_precision_loss` and `inline_always`
warnings everywhere, we just suppress them in the few places they're
already used.

V3_GIT_ORIGIN_REV_ID: c28350a21efa029c8c6aae2a602eec2d75f42216
2024-06-03 12:16:14 +00:00
Samir Talwar
02ee05bed4 Merge match branches to reduce duplication. (#649)
V3_GIT_ORIGIN_REV_ID: 69b2e0a370c547aed9ed798fa54168bc67f98569
2024-05-31 14:44:11 +00:00
Samir Talwar
dfccac348e Use Clippy to help simplify conditions. (#648)
Just because it's fewer lines of code.

1. Invert `if`/`else` blocks with negative conditions.
2. Unwrap redundant `else` blocks.
3. Simplify a few branches to `let … else`.
4. Replace a `match` with `if let`.

V3_GIT_ORIGIN_REV_ID: 4f10730b688d21c1fc86a45ee5fb4adf008b3d94
2024-05-31 13:02:00 +00:00
Samir Talwar
79074bef84 Idiomatic iteration patterns. (#632)
Fix some warnings flagged by Clippy.

1. Elide `.into_iter()` where it's unnecessary.
2. Favor `&` over `.iter()`.
3. Use `.values()` on maps instead of discarding keys by destructuring.
4. Avoid `::from_iter(…)` in favor of `.collect()`.

I also replaced a call to `.cloned()` with `.copied()`.

V3_GIT_ORIGIN_REV_ID: 7d39665b0cd04f5bae9405c0ff5f044f57433f32
2024-05-30 06:22:45 +00:00
Anon Ray
543ffd4727 data shape for query usage analytics (#606)
## Description

The expectation is, that engine should emit usages of OpenDD objects
(i.e. models, commands, relationships, permissions, fields of types
etc.) from a GraphQL query.

This PR adds the types required to gather query usage analytics, under a new crate `query-usage-analytics`.

V3_GIT_ORIGIN_REV_ID: 49778c25a9019e0c8c9a2d13eaa8ba28638b8b55
2024-05-28 09:51:09 +00:00
Daniel Harvey
2bb8be9fa7 Split out execute crate (#588)
<!-- Thank you for submitting this PR! :) -->

## Description

Following `metadata-resolve` and `schema` crates, this splits out
`execute`, the largest folder in `engine`. Undoubtedly this could be
split further.

Functional no-op.

V3_GIT_ORIGIN_REV_ID: c272908153f78212d1f5dd58819707ac3cbcd439
2024-05-17 14:42:23 +00:00
Anon Ray
fb1fc29f34 fix cargo doc warnings (#585)
## Description

As advertised.

Now you can run the following to browse our internal code

```
cargo doc --no-deps --document-private-items --open
```

V3_GIT_ORIGIN_REV_ID: 5b7091a00ed6148b8a91168807b07aa6a925cac9
2024-05-16 09:04:54 +00:00
Daniel Harvey
de342fb2bf Split schema into own crate (#556)
<!-- Thank you for submitting this PR! :) -->

## Description

This PR splits the GraphQL schema generation into the `schema` crate.
Functional no-op.

V3_GIT_ORIGIN_REV_ID: 4f1a91387305d88e9b5fbe4bc8df0575292cf878
2024-05-10 12:28:22 +00:00
Daniel Harvey
11240e2bff Split metadata-resolve into own crate (#543)
<!-- Thank you for submitting this PR! :) -->

## Description

Now that metadata resolve has a clear interface with the rest of the
engine, let's take it out into it's own crate. This will make it easier
to maintain a strong boundary between things.

To simplify imports etc, removed nested layers of modules, so now we
import `use metadata_resolve::Qualified` instead of `use
crate::metadata::resolved::Qualified`.

The changes in `engine` crate are all just updating imports.

Functional no-op.

V3_GIT_ORIGIN_REV_ID: fb94304f7ed8883287c18bd6870045dfd69e3fe3
2024-05-02 12:29:22 +00:00
Gil Mizrahi
0323da8144 create auth and utils subdirectories in crates, and move architecture information to a separate doc. (#534)
## Description

1. I've moved the architecture information we had in `CONTRIBUTING.md`
to a separate document `docs/architecture.md` so we can evolve both
separately in the future.
2. I've introduced a couple of sub directories: `utils` and `auth`, for
supporting crates that are not the core functionality of the engine so
it is easier to find the most relevant crates.

New structure:

```
crates
├── auth
│   ├── dev-auth-webhook
│   ├── hasura-authn-core
│   ├── hasura-authn-jwt
│   └── hasura-authn-webhook
├── custom-connector
├── engine
├── lang-graphql
├── metadata-schema-generator
├── open-dds
└── utils
    ├── opendds-derive
    ├── recursion_limit_macro
    └── tracing-util
```

V3_GIT_ORIGIN_REV_ID: e0e9394da2fcd911f329c48107a76f8492fa304c
2024-05-01 09:05:15 +00:00
Samir Talwar
117c11d18f Do not include debug symbols in release builds. (#493)
This shrinks the Docker image size by half.

I have also normalized the two Dockerfiles so they share a cache for
longer.

V3_GIT_ORIGIN_REV_ID: f976725b09ad2c8022a912b15cdcde55ce5a9486
2024-04-23 09:42:45 +00:00
Samir Talwar
c7d9baaf66 Use references where possible instead of cloning. (#478)
When trying to reduce the number of dependencies we use in the engine, I
was blocked by a few `.clone()` calls that, on inspection, turned out to
be completely unnecessary.

I have replaced those with passing by reference, and then gone on a
pedant spree. I enabled the `needless_pass_by_value` Clippy warning and
fixed it everywhere that it highlighted. In most places, this meant
adding `&`, but I also marked some types as `Copy`, which makes
pass-by-value the right move.

In one place, I replaced calls to `async_map` with `if` and `else`, to
avoid constructing closures that capture across async boundaries. This
means I could just delete `async_map`.

V3_GIT_ORIGIN_REV_ID: 6ff71f0c553b707889d89552eff3e8c001e898cc
2024-04-18 17:35:48 +00:00
Daniel Harvey
90082f9fba Enable redundant_closure_for_method_calls clippy rule and fix (#468)
<!-- Thank you for submitting this PR! :) -->

## Description

In https://github.com/hasura/v3-engine/pull/441 we made all our skipped
Clippy rules explicit. This enables one (pretty arbitrarily) and fixes
what comes up.

V3_GIT_ORIGIN_REV_ID: 406692a2a134cb2a6cf5785acd0ac7c5b9f90c61
2024-04-17 14:58:18 +00:00
Philip Lykke Carlsen
8752081323 Add /metadata and /metadata-hash endpoints (#461)
## Description

This PR iterates on #459.

Rather than serving the engine metadata it serves an arbitrary file,
given by the command line argument `--introspection-metadata`.
Specifying this argument gives rise to endpoints `/metadata` and
`/metadata-hash`.

![image](https://github.com/hasura/v3-engine/assets/358550/63040f02-876a-4c29-8cf1-52a305ffff67)

Update: We only load the file in at engine startup and serve that
version. Changing the file on disk will not change what the engine
serves.

---------

Co-authored-by: Gil Mizrahi <gil@gilmi.net>
V3_GIT_ORIGIN_REV_ID: db88adb5c08c4489cc1abd5fb5236b8d5ba51b9a
2024-04-11 21:32:40 +00:00
Daniel Harvey
2a24b3060a use clippy settings in Cargo workspace (#441)
<!-- Thank you for submitting this PR! :) -->

## Description

Following the approach taken here:
https://github.com/hasura/ndc-postgres/pull/402

This moves the `clippy` settings into the Cargo workspace file instead
of passing them for each invocation.

We enable all pedantic settings, run `cargo clippy --fix` to auto fix a
few things, and then manually disable all other lints.

Plenty of them are worth enabling and fixing in future IMO.

---------

Co-authored-by: Samir Talwar <samir.talwar@hasura.io>
V3_GIT_ORIGIN_REV_ID: aa0e6ccb8d72a7393e14b5c58b82077a67d9cb15
2024-04-08 10:14:11 +00:00
Daniel Harvey
bdd5d06437 Move all crates into a folder (#355)
<!-- Thank you for submitting this PR! :) -->

## Description

This moves all the crates into a `/crates` folder. Everything appears to
just work, thanks Cargo!

V3_GIT_ORIGIN_REV_ID: 8e3ef287b1a46cabdb4d919a50e813ab2cddf8b1
2024-03-19 18:07:14 +00:00
Daniel Harvey
7e0f0bb1ab migrate build job to Github Actions (#352)
V3_GIT_ORIGIN_REV_ID: b26d1479508aeb9168459f0db82591f8995bb174
2024-03-15 17:24:02 +00:00
Rakesh Emmadi
f5c9a6f291 Custom deserializer for OpenDd objects (#313)
V3_GIT_ORIGIN_REV_ID: 4c7a9380e517ac8e7ac1bd44dbe879918bd0d9a2
2024-03-11 12:17:50 +00:00
Samir Talwar
dea588255b Set the crate properties once, to improve the Nix build. (#334)
V3_GIT_ORIGIN_REV_ID: ec5f3d968306a64ad5c63df324ca5c524838c598
2024-03-06 17:15:13 +00:00
Manas Agarwal
e27e5b7ffe v3: open-source hasura v3 engine
Hasura V3 Engine v3.alpha.12-19-2023
V3-GitOrigin-RevId: 6605575a52b347b5e9a14ecd1cc736f113c663b3

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/10567
Co-authored-by: Vishnu Bharathi <4211715+scriptnull@users.noreply.github.com>
GitOrigin-RevId: 38c98a4b1971efe3ac724c2371c43ceb7d31f140
2023-12-19 09:05:39 +00:00