Commit Graph

465 Commits

Author SHA1 Message Date
Daniel Harvey
788fd9e197 Newtypes around object collections (#807)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

We pass around a lot of `BTreeMap` and `IndexMap` types. This has two
problems:

a) everytime we change the items inside, we have to manually update lots
of call sites
b) we can't attach useful behaviour to it

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

This adds some wrappers around `object_types`, and adds a `.get()`
function with a useful default error. This error only contains the
missing `type_name`, so the idea is that it would be wrapped in another
error that would provide more context. The hope is that we can get away
from one giant error enum, and instead have a set of smaller error types
that live along each resolving stage.

In isolation this PR isn't very interesting, as it's a tiny drop in the
ocean, so really it's here to say, "is this a thing we vaguely like?"

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

V3_GIT_ORIGIN_REV_ID: 804a703d7155ce5b929937689d5649c6571357b0
2024-07-09 16:26:58 +00:00
Daniel Harvey
adc1ad5bf9 Break out DataConnectorError (#813)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

Put all the errors that happen in the `data_connectors` stage into a new
type `DataConnectorError`.

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

Moving enum members to the new type. Functional no-op.

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

V3_GIT_ORIGIN_REV_ID: 2e50eb561ce6b7c7fac4de4b31c9957f7ee4696c
2024-07-09 15:55:22 +00:00
Daniel Harvey
054600d4d3 Move errors to type_permissions stage (#809)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

Much like https://github.com/hasura/v3-engine/pull/808, move an error to
the place it is thrown.

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

Move files around. Functional no-op.

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

V3_GIT_ORIGIN_REV_ID: a61527b57662efada2c7d2049e12f103deec1e4e
2024-07-09 11:58:57 +00:00
Daniel Harvey
83df9e81bb Move GraphqlConfigError to graphql_config stage (#808)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

We have a giant error file in `metadata-resolve`, and it's really
unclear what can go wrong where. Let's improve it in some small way.

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

Move `GraphqlConfigError` to the `graphql_config` stage, and make that
stage only return that kind of error.

Functional no-op.

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

---------

Co-authored-by: Samir Talwar <samir.talwar@hasura.io>
V3_GIT_ORIGIN_REV_ID: 9aa58875a24d66e6945c9ead6434fedb124f1dd8
2024-07-09 10:02:40 +00:00
Daniel Chambers
1564fcce17 Remove usage of ndc_models Expression types from the IR (#810)
This PR removes the usage of `ndc_models` `Expression` types from the
IR. It then adds logic to map from the new IR `FilterExpression` types
to `ndc_models` types in the plan part of the code where IR ->
ndc_models mapping is performed. The new IR types and the change to the
IR creation logic is mostly in `crates/execute/src/ir/filter.rs`.

The new IR types have some optimisations applied to them. In particular,
some basic boolean expression simplification logic is applied when
`FilterExpression::mk_and` `mk_or` and `mk_not` are used. This strips
out redundant and/ors/nots resulting in simpler boolexps for connectors
to process. (This logic is similar to what GDC did in Hasura v2).

Unfortunately it turned out that arguments had JSON-serialized
`ndc_models` types embedded in them, in particular, where argument
presets had been used to set a BooleanExpression as an argument value.
The old code was simply creating an ndc_models::Expression and
serializing it directly to JSON. This has now been refactored to retain
the new IR `FilterExpression` until `plan` time, at which point it will
be mapped into the ndc type and serialized to JSON and embedded in the
argument value. The types change for this can be seen in
`crates/execute/src/ir/arguments.rs`, but the real logic is actually in
`crates/execute/src/ir/permissions.rs`, with the
`make_value_from_value_expression` and
`make_argument_from_value_expression` functions.

The mapping of expression and argument IR into `ndc_models` can be found
in `crates/execute/src/plan/common.rs`.

In `metadata_resolve`, some ndc_models types were removed and
non-ndc_model types were used instead. In particular
`ndc_models::ComparisonOperatorName` (swapped with
`DataConnectorOperatorName`) and `ndc_models::UnaryComparisonOperator`
(swapped with a new `UnaryComparisonOperator` enum type).

This PR is a functional no-op, other than the boolean expression
simplification.

V3_GIT_ORIGIN_REV_ID: 13805430fb2e2ca5653406a094d719138f0d5b51
2024-07-09 08:23:24 +00:00
dependabot[bot]
f8a2abc0df Bump serde from 1.0.203 to 1.0.204 (#801)
Bumps [serde](https://github.com/serde-rs/serde) from 1.0.203 to 1.0.204.

V3_GIT_ORIGIN_REV_ID: cbe13bf30a6aa7cc5480d174d7d435f18a5cc7f2
2024-07-09 06:54:43 +00:00
Daniel Harvey
8579570459 Add deprecation notice to old boolean expression types (#806)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

Since these code comments end up in the docs, let's make sure we point
out these old types are no longer in favour.

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

Updating doc comments.

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

V3_GIT_ORIGIN_REV_ID: a21190fbf2327e1ae265c76d1f467af063f1dd64
2024-07-08 14:19:28 +00:00
Daniel Harvey
29ccf96784 Skip capability check for local relationships (#805)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

If the target data connector for a relationship does not have the
`foreach` capability, we threw an error, however we should allow this.

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

Check data connector names for relationship target before throwing
capability error.

V3_GIT_ORIGIN_REV_ID: 157908f0077b7c3af67a77600e5f8c9fc65df7f2
2024-07-08 13:03:13 +00:00
dependabot[bot]
7a0eb6847a Bump async-trait from 0.1.80 to 0.1.81 (#804)
Bumps [async-trait](https://github.com/dtolnay/async-trait) from 0.1.80 to 0.1.81.

V3_GIT_ORIGIN_REV_ID: 1e9a6a6a40b6b0f732774df1e7a52b8a182f70c1
2024-07-08 08:32:04 +00:00
dependabot[bot]
0d9bb020f1 Bump serde_json from 1.0.119 to 1.0.120 (#803)
Bumps [serde_json](https://github.com/serde-rs/json) from 1.0.119 to 1.0.120.

V3_GIT_ORIGIN_REV_ID: 0c432282a8ae7daaca22a608881ed0f69da8dd4a
2024-07-08 08:02:06 +00:00
dependabot[bot]
5feabdfd08 Bump syn from 2.0.68 to 2.0.69 (#802)
Bumps [syn](https://github.com/dtolnay/syn) from 2.0.68 to 2.0.69.

V3_GIT_ORIGIN_REV_ID: 6e8a895e4b928418e87c820effcdfbbecdfbfba7
2024-07-08 07:32:24 +00:00
dependabot[bot]
0be3e9f6c0 Bump serde_with from 3.8.2 to 3.8.3 (#800)
Bumps [serde_with](https://github.com/jonasbb/serde_with) from 3.8.2 to 3.8.3

V3_GIT_ORIGIN_REV_ID: d2b0fbecea1997959f644b90c2a57075635a0293
2024-07-08 07:03:01 +00:00
Daniel Chambers
90dbbccce5 Remove ndc_models types from arguments and order by IR (#798)
Note: This PR is stacked on #797 and should be merged after it.

This PR removes the use of ndc_models types from the arguments and order
by IR. It then shifts the logic that maps the IR to the ndc_models into
the plan part of the code where IR -> ndc_model mapping is performed.

This will help isolate the IR from ndc_models and move us towards being
able to have multiple IR -> ndc_models mapping codes, one per supported
ndc version.

This is a functional no-op PR.

V3_GIT_ORIGIN_REV_ID: 66be868ff4c8185c6190537d570d88813cb7f410
2024-07-05 10:44:41 +00:00
Daniel Chambers
e44589931c Reimplement most OpenDD newtypes using SmolStr and a macro (#797)
This PR replaces most of the string newtypes in the open-dds crate with
new implementations based on `SmolStr`, using a macro adapted from the
macro used in ndc-models. `SmolStr` usage should result in less heap
allocations and the macro ensures all the newtypes get all the same
trait implementations (such as various `From` and `Borrow` instances)
and helper impl functions (such as `as_str`).

The new macro, `str_newtype`, creates a newtype wrapper around `SmolStr`
or another type (typically `Identifier`), and writes out all the various
trait implementations. It also takes a documentation string which
ensures that every newtype has a description in JSON Schema.

The changes in the downstream crates are just adjusting to use the new
newtypes.

Also:
* `QueryGraphqlConfig` used a lot of String typed-properties; these have
been changed for the appropriate `GraphQlTypeName` and
`GraphQlFieldName` types.
* metadata-resolve had a duplicate newtype called
`ConnectorArgumentName`, which duplicated open-dds's
`DataConnectorArgumentName`. This has been removed and replaced.
* A new newtype called `CollectionName` has been added in open-dds to
represent the name of the ndc collection that a model points to. This
was previously just a string.
V3_GIT_ORIGIN_REV_ID: d5a33756ef0e110b2255478358a38e42e6ff89a2
2024-07-05 10:16:33 +00:00
Philip Lykke Carlsen
9910df8154 Add ServerOptions to initial server startup logs (#794)
### What

Add ServerOptions to initial server startup logs.

![image](https://github.com/hasura/v3-engine/assets/358550/33e49344-a7d8-44be-8bce-b6c8b11a06ec)

### Update

Use json, as well as a span field rather than an event:

![image](https://github.com/hasura/v3-engine/assets/358550/724bcdf4-8125-4a5e-83aa-db7de1229e3a)

V3_GIT_ORIGIN_REV_ID: 28d0c8edd70357b3460a4fbea389f5fa801fd4bf
2024-07-05 08:10:17 +00:00
Daniel Chambers
62e76aac67 Sort open-dds JSON schema for better diffing (#796)
This PR just sorts the definitions in the Open DD JSON schema file, so
that when it changes, we can get better diffs because the types won't
move around inside the file.

V3_GIT_ORIGIN_REV_ID: 80221fae1b4d6e4b98b658f9ba2f19413abf7389
2024-07-05 06:45:45 +00:00
Tom Harding
fa5673cd1f Allow non-nullable variables to fulfil nullable values (#795)
### What

This PR fixes a bug with variable nullability coercion. Specifically,
providing a non-null variable for a nullable field should work, as all
non-nullable variables can be used as nullable variables via "coercion".

Related issues:
* https://hasurahq.atlassian.net/browse/V3ENGINE-243
* https://github.com/hasura/v3-e2e-testing/pull/224 (the test for this
change)

### How

I think someone did something clever with macros, so I tried to unpack
the macro to make sense of it, and in the process, spotted the bug: it's
not that both the location must be nullable AND the variable not, it's
that _if_ the variable is nullable, the location must not be nullable.

V3_GIT_ORIGIN_REV_ID: 4f4ba3fe6898220bbba9ae2867233cd762bef1cb
2024-07-04 16:07:35 +00:00
paritosh-08
4b62f7decf release v2024.07.04 (#792)
Update the changelog for new version

---------

Co-authored-by: Anon Ray <ecthiender@users.noreply.github.com>
V3_GIT_ORIGIN_REV_ID: b87f16ebaa1471f010ec461be097bcfd6648c99a
2024-07-04 10:25:48 +00:00
Daniel Chambers
862ad8e0dc Update ndc-models to latest version which includes new name newtypes (#791)
This PR updated ndc-models to the latest version on main. This version
is still a 0.1.x version, but it now includes all the [new
newtypes](https://github.com/hasura/ndc-spec/pull/156) that wrap
previously stringly-typed things. For example, `ArgumentName`,
`FieldName`, etc.

This pervades across the entire engine, but thankfully the changes are
mostly mechanical repetitive changes. Usually you will see conversions
from `String`-typed variables into the newtypes using this sort of form:
`FieldName::from(string.as_str())`, which is the most efficient way
copying the value (the str slice is copied). Or you will see usages of
the newtype as a raw string by `.as_str()`-ing it. Converting the
newtypes into a String can be done with `.into()` if owned, but if
referenced `.as_str().to_owned()` performs the clone and type
conversion.

Other changes:
* A few minor instances of `ok_or()` usages (or similar) have been
converted into lazy error construction variants (eg `ok_or_else()`)

V3_GIT_ORIGIN_REV_ID: 64a371ae6197ef3be98a6f7cdc4052d654a43da0
2024-07-04 08:58:12 +00:00
paritosh-08
08014cab88 pre-execution plugin middleware (#771)
The RFC:
https://docs.google.com/document/d/1NB9fA6J8_dKtWknJkfTiqN5qWhPVDfYVOKFK25gYE7Y/edit

The JIRA: https://hasurahq.atlassian.net/browse/V3ENGINE-234

<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

This PR adds a new middleware to the `/graphql` endpoint handler. This
new middleware will be used to handle the pre-execution plugins.

### How

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

We are doing something similar to
https://github.com/tokio-rs/axum/blob/axum-v0.6.20/examples/consume-body-in-extractor-or-middleware/src/main.rs

V3_GIT_ORIGIN_REV_ID: a7e0a7a252efd1f266b3e90df23a8307a9b35fc7
2024-07-03 12:58:26 +00:00
Daniel Chambers
e380876823 Handle multiple ndc-model versions during query planning and execution (#790)
The PR adds code that handles multiple versions of the ndc-models during
the execution pipeline. Depending on whether a connector supports `v01`
or `v02` models, a different set of types is used to send and receive
the http requests.

However, the engine internally still uses the latest (v02) models inside
its IR. Unfortunately, it was going to be quite traumatic to prevent the
engine from using ndc models inside the IR and during response
processing and remote joins. This means that the engine generates v02
requests, and these are downgraded into v01 requests. v01 responses are
upgraded to v02 responses before being processed by the engine.

The ndc client (`execute::ndc::client`) now only takes new wrapper enum
types (`execute::ndc::types`) that split between v01 or v02
requests/responses. Every place that carries an ndc request now carries
this type instead, which allows it to carry either a v01 or a v02
request.

When ndc requests are created during planning, all creation goes via the
new `execute::plan::ndc_request` module. This inspects the connector's
supported version, creates the necessary request, and if needed,
downgrades it to v01.

When ndc responses are read during planning or during remote joins, they
are upgraded to v02 via helper functions defined on the types in
`execute::ndc::types`.

The upgrade/downgrade code is located in `execute::ndc::migration`. Keep
in mind the "v02" types are currently the same as the "v01" types so the
migration code is not doing much. This will change as the v02 types are
modified.

However, this approach has its drawbacks. One is that it prevents
changes to the ndc types [like
this](https://github.com/hasura/ndc-spec/pull/158) without a fair bit of
pain (see
[comment](https://github.com/hasura/ndc-spec/pull/158#issuecomment-2202127094)).
Another is that the downgrade code can fail at runtime and it is not
immediately obvious to developers using new, but unused, v02 features
that their new feature would fail on v01, because that mapping to v01
has already been written. Another is that we're paying some (small,
probably?) performance cost by upgrading/downgrading types because we
need to rebuild data structures.

Also:
* `execute::ndc::response` has been merged into `execute::ndc::client`,
since it was inextricably linked.
V3_GIT_ORIGIN_REV_ID: f3f36736b52058323d789c378fed06af566f39a3
2024-07-03 08:32:37 +00:00
Anon Ray
c1b9592f6b respect relationship comparison capability when generating relationship fields in model filter (#789)
### What

When generating GraphQL schema for relationship fields in model filter,
engine ignores `relation_comparisons` capability of the data connector.
Engine would generate schema for data connectors which don't have this
capability. This PR fixes that.

### How

While generating fields for the filter input type, take the relationship
capabilities into account.

The `ObjectBooleanExpressionType` and `BooleanExpressionType` objects
are quite different, hence their schema generation part is also
different, and is split in two different functions
(`build_comparable_relationships_schema`, and
`build_new_comparable_relationships_schema`). Added checking of
relationship comparison capability in both the functions.

V3_GIT_ORIGIN_REV_ID: dce2b88f7792e01e5bb390ecdb580e223ec80f01
2024-07-02 07:49:10 +00:00
Daniel Harvey
0123c558f1 Enable BooleanExpressionTypes (#783)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

# BooleanExpressionType

A new metadata kind `BooleanExpressionType` can now be defined. These
can be used in place of `ObjectBooleanExpressionType` and
`DataConnectorScalarRepresentation`, and allow more granular control of
comparison operators and how they are used.

The old metadata types still work, but will eventually be deprecated.

```yaml
kind: BooleanExpressionType
version: v1
definition:
  name: album_bool_exp
  operand:
    object:
      type: Album
      comparableFields:
        - fieldName: AlbumId
          booleanExpressionType: pg_int_comparison_exp
        - fieldName: ArtistId
          booleanExpressionType: pg_int_comparison_exp_with_is_null
        - field: Address
          booleanExpressionType: address_bool_exp
      comparableRelationships:
        - relationshipName: Artist
          booleanExpressionType: artist_bool_exp
  logicalOperators:
    enable: true
  isNull:
    enable: true
  graphql:
    typeName: app_album_bool_exp
```

```yaml
kind: BooleanExpressionType
version: v1
definition:
  name: pg_int_comparison_exp
  operand:
    scalar:
      type: Int
      comparisonOperators:
        - name: equals
          argumentType: String!
        - name: _in
          argumentType: [String!]!
      dataConnectorOperatorMapping:
        - dataConnectorName: postgres_db
          dataConnectorScalarType: String
          operatorMapping:
            equals: _eq
  logicalOperators:
    enable: true
  isNull:
    enable: true
  graphql:
    typeName: app_postgres_int_bool_exp
```

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

Remove feature flag, unhide JsonSchema items, fix a few missing bits of
JsonSchema the tests didn't warn us about before.

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

V3_GIT_ORIGIN_REV_ID: dd3055d926fdeb7446cd57085679f2492a4358a1
2024-07-01 15:29:14 +00:00
dependabot[bot]
ec59b4e9d3 Bump serde_with from 3.8.1 to 3.8.2 (#785)
V3_GIT_ORIGIN_REV_ID: 4592f1a7fcde9326b7091351c916e710bd5602bc
2024-07-01 15:05:55 +00:00
dependabot[bot]
45f55cf90a Bump serde_json from 1.0.118 to 1.0.119 (#786)
V3_GIT_ORIGIN_REV_ID: ef0c6ab406e71c497e31d26a112083cc5108eeb9
2024-07-01 14:36:18 +00:00
dependabot[bot]
dbda5110f8 Bump clap from 4.5.7 to 4.5.8 (#784)
V3_GIT_ORIGIN_REV_ID: 38de46553688ac1eeae1b364dc9e99c1bcb13fde
2024-07-01 14:08:10 +00:00
Rakesh Emmadi
5efbb09500 Unify get_underlying_type_name and get_base_type functions (#788)
### What
This is a no-op refactor that involves unifying `get_base_type` and
`get_underlying_type_name` functions, whose motives are the same.

### How
Replace the name of erstwhile `get_base_type` fn with
`get_underlying_type_name`. Remove the latter. Update the rest of the
code to use the new function.

V3_GIT_ORIGIN_REV_ID: 0b0d999670641ba265fde153af9b43b4d865e215
2024-07-01 12:57:50 +00:00
Rakesh Emmadi
9e98ed2310 Implement query usage analytics (#713)
JIRA: https://hasurahq.atlassian.net/browse/V3ENGINE-176

This PR implements usage analytics
([RFC](https://docs.google.com/document/d/1k8_ZniiejxHhD0SphtEJ2mg6MoJON0dkk0q4b5WZWDE/edit))
for a GraphQL query when a request is made. The analytics data JSON is
attached to the span, whose details are provided below:
Span name - `execute`
Attribute - `internal.query_usage_analytics = <JSON-string>` OR
`internal.query_usage_analytics_error = <String>`

Refs:
- [Link](https://github.com/hasura/v3-engine/pull/606) to the PR, where
data types for usage analytics were added.
- [Link](https://github.com/hasura/v3-engine/pull/715) to the PR, where
some improvements are made to the analytics data types.

Note: Changelog entry is added.
V3_GIT_ORIGIN_REV_ID: f4089f3354b638533ff72fbec72951e883c31100
2024-07-01 12:10:12 +00:00
Gil Mizrahi
4dd20a7e5e update pg configuration for tests (#787)
### What

Update ndc-postgres configuration to v4, including the new
`mutationsVersion: "v4"`.

### How

- sed `s/experimental_/v2_/g`
- `ndc-postgres-cli upgrade`
- `ndc-postgres-cli update`

V3_GIT_ORIGIN_REV_ID: 67c608c2a3f1e232d5e0725fe8817672aa3dd627
2024-07-01 09:12:36 +00:00
Daniel Chambers
a72d45d165 Add support for multiple versions of ndc-spec to Open DD and metadata-resolve (#781)
This PR introduces support for multiple versions of the ndc-spec by
adding a new `VersionedSchemaAndCapabilities` enum variant under the
`DataConnectorLink` in OpenDD. This allows the capture of both ndc
v0.1.* and v0.2.* schema and capabilities.

This is achieved by referencing the `ndc-models` crate twice, once for
`v0.1.4` and once for the first commit after `v0.1.4`. That commit was
chosen to avoid actual v0.2.0 breaking changes for now, while we lay in
this multiple version support plumbing. Future PRs will use a newer
commit and adopt the breaking changes where necessary. The
`VersionedSchemaAndCapabilities::V02` variant uses the the v0.2
reference of `ndc-models`.

Then, during metadata resolve, when we resolve the
`DataConnectorContext` from `DataConnectorLink`, we perform a migration
of v0.1 types to v0.2 types and store and use the v0.2 types during
metadata resolve. This migration is performed in the new module
`ndc_migration`. We also record the `NdcVersion` (either `V01` or `V02`)
in the `DataConnectorLink`. The `execute` crate will need to use this to
determine which version to send to the connector at runtime (to be
implemented in a future PR).

The new changes to OpenDD are hidden from the JSON Schema via a new
`UnstableFeatures` flag, and the use of the new variant is gated behind
it in metadata resolve, since we don't yet support it upstream in the
`execute` crate.

V3_GIT_ORIGIN_REV_ID: d6d8a768ea3537c0b5e620799e94d3dd1e529526
2024-06-28 11:57:41 +00:00
David Overton
77ddaf360e Open DD changes for OrderByExpression (#780)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

Adds new Open DD type `OrderByExpression` and defines `ModelV2` type, as
described in
https://github.com/hasura/v3-engine/blob/main/rfcs/open-dd-expression-type-changes.md.

### How

- Added new types `OrderByExpression`, `ModelV2` and
`ModelGraphQlDefinitionV2` to the `open-dds` crate.
- Added new `UnstableFeatures` flag `enable_order_by_expressions` to
`metadata-resolve`. This is not yet used, as `metadata-resolve` does not
yet use the new Open DD types.

---------

Co-authored-by: Daniel Harvey <danieljamesharvey@gmail.com>
V3_GIT_ORIGIN_REV_ID: e861e88f293f1b380e6346de8490b96cec6f65bb
2024-06-28 11:26:30 +00:00
Daniel Harvey
bd3bc77808 Add scalar boolean expression types to data_connector_scalar_types (#778)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

Changed tests to use the new `BooleanExpressionType` until something
broke. Fortunately something broke - we need to add the scalar boolean
expression types to the data connector scalar types for various lookups.

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

Split scalar boolean expression type resolve into own step so it's
available earlier in the pipeline.

Loop through them and add them to the `data_connector_scalar_types`
outputs.

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

V3_GIT_ORIGIN_REV_ID: dbd8969c3e9e9d8db1d4a34e93aefc34bdf31421
2024-06-28 10:35:44 +00:00
Daniel Harvey
a47327a9bf Add cacert to docker image (#782)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

Allow engine to connect to NDCs via HTTPS.

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

Add `cacert` to Docker image using Nix.

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

V3_GIT_ORIGIN_REV_ID: 52458920236f3868cc8daf18e140f8536d9bc674
2024-06-28 10:06:18 +00:00
Daniel Chambers
8f5cfba867 Simplify DataConnectorContext type and remove duplicate scalar types (#779)
This PR refactors the `DataConnectorContext` type in
`crates/metadata-resolve/src/stages/data_connectors/types.rs` to remove
the extra copy of scalar types it had and simplifies and removes the
nesting in the types.

`DataConnectorContext` used to have its own copy of `scalars` which
contained `ScalarTypeInfo`s. However, it already contains a more
complete copy of scalar types inside `inner.schema.scalar_types`! Turns
out the `scalars` copy (and `ScalarTypeInfo`) is unnecessary. The only
value it added was having the computed `ComparisonOperators` struct,
which is simply copied from there to where it really lives on
`ScalarTypeWithRepresentationInfo` during the
`data_connector_scalar_types` stage.

So I've moved `ComparisonOperators`, and the code that creates it, to
`data_connector_scalar_types` and deleted `ScalarTypeInfo`. This meant
that `DataConnectorContext` only contained `DataConnectorCoreInfo`
(pointless!), so I inlined `DataConnectorCoreInfo` into
`DataConnectorContext` to simplify things. This removed `.inner` calls
all through the code.

These changes help my work with supporting multiple `ndc-models`
versions because it simplifies the number of places we store and deal
with scalar types.

This PR is a functional no-op.

V3_GIT_ORIGIN_REV_ID: 3beb0b07abc7ff5cfaa7e9b60eb46aec94d7ec1a
2024-06-28 07:58:43 +00:00
Daniel Harvey
64657639d4 Use Rust 1.79.0 (#751)
<!-- Thank you for submitting this PR! :) -->

## Description

Upgrade Rust, as a treat. Functional no-op.

---------

Co-authored-by: Samir Talwar <samir@functional.computer>
V3_GIT_ORIGIN_REV_ID: 1e0014049e89b8658326c8d8f652df800c415526
2024-06-27 22:22:01 +00:00
Daniel Harvey
f87561921c Validate boolean expression types when used as arguments (#777)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

In https://github.com/hasura/v3-engine/pull/750 and
https://github.com/hasura/v3-engine/pull/754 we added a number of checks
for boolean expression types that can be run once we know the data
source they will be used against.

Previously these checks were only used for model `where` clauses, this
pull request moves them into a shared folder and also checks them when
they are used as model or command arguments.

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

Add a new `arguments` resolve step. It doesn't fit inside the usual
`commands` or `models` step as we need the data sources for both
validated, plus access to all the resolved `relationships` outputs.

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

V3_GIT_ORIGIN_REV_ID: f713659962e3f20b2c85f287b6c362fb52ffa1ed
2024-06-27 18:44:40 +00:00
Daniel Harvey
b31d6099d7 Tidy model_permissions and command_permissions stages (#775)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

Much like https://github.com/hasura/v3-engine/pull/774, we split the big
files into separate modules. Functional no-op.

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

Copy pasta, `just fix-local`, mostly.

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

V3_GIT_ORIGIN_REV_ID: ba7e40057583f98df54573c72663a4a2d2c4a4ab
2024-06-27 16:30:43 +00:00
Daniel Harvey
9232af913a Split up command resolve stage (#774)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

Tiny no-op PR to split the `command` stage of `metadata-resolve` into
smaller modules the same way `models` and others work.

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

Copy and paste, run `just fix-local` to remove unrequired imports.

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

V3_GIT_ORIGIN_REV_ID: ab2846be83ba0e948ea222b42eb4fd7ffa5b3523
2024-06-27 15:48:43 +00:00
Philip Lykke Carlsen
0ea629ae59 Stdout otel exporter (#776)
### What

Output all traces to stdout.

---------

Co-authored-by: Daniel Harvey <danieljamesharvey@gmail.com>
V3_GIT_ORIGIN_REV_ID: 06330076ca305a331996530ddcd4d4c13d46bd95
2024-06-27 14:45:05 +00:00
Samir Talwar
291d564d82 Support for partial supergraphs in metadata resolution. (#772)
### What

This adds a flag, `--partial-supergraph`, which instructs the metadata
resolver to prune relationships to unknown subgraphs rather than failing
to resolve.

### How

The flag gets passed through as
`metadata_resolve::configuration::Configuration`, and known subgraphs
are now tracked in `MetadataAccessor`. If the flag is set and a
relationship target refers to an unknown subgraph, we return an empty
list of relationships instead of failing.

Some test infrastructure has been added to set configuration flags per
test.

V3_GIT_ORIGIN_REV_ID: 6f0de2442a3bfc7c7a4c48e3dc7296dc1538cd67
2024-06-27 14:17:31 +00:00
Philip Lykke Carlsen
25d4a3b5ea Add flag to toggle whether to expose internal errors (#759)
### What

This PR adds the ability to include internal errors in API responses via
a command line argument `--expose-internal-errors`.

The default behavior remains not to show the contents of internal error
messages.

V3_GIT_ORIGIN_REV_ID: 11c47286d3fbceeda71df3a224853633aeea8902
2024-06-27 12:43:23 +00:00
Daniel Harvey
599b4abdc5 Fix boolean expression type operator mapping (#767)
<!-- The PR description should answer 2 (maybe 3) important questions:
-->

### What

A scalar `BooleanExpressionType` lets us give operators fancy names.
This PR makes them actually work.

Functional no-op as this feature is behind a feature flag.

<!-- What is this PR trying to accomplish (and why, if it's not
obvious)? -->

<!-- Consider: do we need to add a changelog entry? -->

### How

<!-- How is it trying to accomplish it (what are the implementation
steps)? -->

V3_GIT_ORIGIN_REV_ID: b49ef95d3d6672a1e27371fa5f4df63acd0849fc
2024-06-27 11:23:58 +00:00
Anon Ray
832ceb1f97 tests: use a different argument name for header forwarding (#773)
### What

There was a report that using a different argument name other than
`headers` for `DataConnectorLink.argumentPresets` doesn't work. This PR
updates the test to use a different name, and this seems to work.

### How
Update the headers argument name in custom connector schema, and the
argument name in `DataConnectorLink.argumentPresets` metadata.json.

V3_GIT_ORIGIN_REV_ID: 91346c01573f666e5707c0f41f4635d689bf5b98
2024-06-27 09:53:57 +00:00
Samir Talwar
00db411c6e Remove ModelsGraphqlOutput::graphql_types, as it's not used. (#769)
### What

This removes an unused field. The remaining struct had just one field
and is private to the crate, so I converted it to a type alias.

### How

In order to make `rustc` highlight the issue, I reduced the number of
types we export as `pub`, changing some to `pub(crate)`.

Then I just deleted the field once the warning showed up.

V3_GIT_ORIGIN_REV_ID: 7a26e99f062ed0f2c7449e1f57bc76068f059afb
2024-06-27 09:13:21 +00:00
Daniel Chambers
5b2e36cd4f Fix NDC version used in JSON Schema references (#770)
This PR fixes the version of the NDC Spec that Open DD's JSON schema
references when it uses NDC's schema and capabilities types in the
`DataConnectorLink`. OpenDD references 0.1.4 of NDC, but uses 0.1.3 in
the JSON schema. This corrects that.

V3_GIT_ORIGIN_REV_ID: bdbb417b3227861dae7835f6d3bda0d1bf935ea7
2024-06-27 03:50:35 +00:00
Samir Talwar
eb1dbffbd1 Use SubgraphIdentifier in the metadata accessor, and SmolStr internally. (#768)
### What

Using `String` everywhere to represent subgraph identifiers is
definitely going to cause problems at some point. We can avoid this by
using `open_dds::identifier::SubgraphIdentifier` instead.

`Identifier` and `SubgraphIdentifier` were wrappers around `String`. I
have also changed them to wrap `SmolStr` instead, for better performance
as they're cloned a lot.

### How

First of all, I hid the innards of `Identifier` and
`SubgraphIdentifier`, instead exposing certain behaviors as methods.
Mostly, this means exposing `as_str()` and `to_string()`.

Then I replaced the internals with `SmolStr`. I had to change one place
where `RefCast` was used, but that was pretty much it.

Finally, I switched out `String` for `SubgraphIdentifier` in
`QualifiedObject`. This necessitated a couple of constants for the two
"magic" subgraphs, `__globals` and `__unknown_namespace`, but was
otherwise fine.

I didn't touch `Qualified` for now.

V3_GIT_ORIGIN_REV_ID: 28664609c3173b181c3789093cb9796896642eb7
2024-06-26 22:04:22 +00:00
Samir Talwar
5d619a540f Replace lazy_static with stdlib equivalents. (#758)
### What

The `lazy_static` macro is poorly maintained, fairly bloated, and has
been mostly superseded by
[`OnceLock`](https://doc.rust-lang.org/stable/std/sync/struct.OnceLock.html)
in the stdlib.

### How

1. I turned a couple of `static ref` values into `const`, sometimes by
creating `const fn` equivalents to other functions.
2. I inlined static behavior to construct a JSON pointer into some
tests, where we don't care too much about losing a few milliseconds.
3. For the rest, I replaced `lazy_static` with a `static OnceLock` and a
call to `OnceLock::get_or_init`.

V3_GIT_ORIGIN_REV_ID: 18e4150a5fb24fe71f6ed77fe6178b7942405aa3
2024-06-26 12:45:41 +00:00
Daniel Chambers
82c0c65bd0 Enable aggregate relationships by default (#765)
NOTE: This PR is stacked on #756 and should be shipped after that is
merged.

This PR enables the existing aggregate relationships work (see #725,
#731, #756) by default by removing the experimental flag it used to be
disabled behind.

The new OpenDD schema changes that were added are also unhidden so that
they are visible in the OpenDD JSON Schema.

V3_GIT_ORIGIN_REV_ID: cfd86d8a9ea61887ccf0f1a5d08bdcc3dda59cdc
2024-06-26 10:49:10 +00:00
Daniel Chambers
0624a7553d Aggregate relationships GraphQL schema and execution (#756)
## Description
This PR implements the GraphQL schema and execution for aggregate
relationships.

In the `schema` crate, the new `model_aggregate_relationship_field`
function handles generating schema for ModelAggregateTarget
relationships. It mostly delegates the meat of its implementation to
reused logic; some refactoring has occurred to make this possible.
This involved changes in `select_many`, `select_aggregate` and
`model_arguments`. The creation of the model arguments field argument
now exists in `model_arguments` and is reused by `select_many` and
`select_aggregate`. The creation of all aggregate field arguments is now
in `select_aggregate::generate_select_aggregate_arguments`, and is then
reused when generating the aggregate relationship field. That field is
annotated with the new `RelationshipToModelAggregate` annotation.

In the `execute` crate, the logic around generating an the aggregate
selection IR was moved from `select_aggregate` into `model_selection`.
This was so it can be reused by the logic in `relationship` that now
uses it to generate an aggregate selection when encountering an
`RelationshipToModelAggregate` field.
Inside `relationship` some rearranging was done so that
`build_local_model_relationship` and `build_remote_relationship` could
work with either a normal model selection IR or the new aggregate
selection IR. The necessitated moving the creation of that IR outside
those functions into the caller, so the different callers can create
different IR (normal vs aggregate IR). This also reduced code
duplication.

New tests have been added to `engine` that cover aggregate relationships
and also remote joined aggregate relationships.

This PR also corrects two bugs in metadata resolve revealed by new
testing:
* The filter input field name in `GraphqlConfig` must be specified if
using an aggregate relationship
* The filter input type name defined on a `Model` must be specified if
that model is the target of an aggregate relationship. Conversely, the
filter input type name can be specified if the `Model` itself doesn't
define an aggregate, but is still involved in a aggregate relationship
(this previously produced an error).

This PR completes the feature, but it is still hidden behind the
experimental flag. There will be a follow up PR to remove that and
expose the functionality by default.

JIRA: [V3ENGINE-160](https://hasurahq.atlassian.net/browse/V3ENGINE-160)

[V3ENGINE-160]:
https://hasurahq.atlassian.net/browse/V3ENGINE-160?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

V3_GIT_ORIGIN_REV_ID: d499371906f7af71a4017c7c3ae75b7693cd3fa7
2024-06-26 09:10:33 +00:00
Samir Talwar
63ac02bc07 Take snapshots of passing and failing metadata resolution. (#763)
### What

In order to more easily monitor and review changes to metadata
resolution, this introduces snapshot testing for both successful and
failing calls to `resolve`. I used [Insta](https://insta.rs/) for this.

### How

For tests of the failure case, we already had a text file with the
expected error, so I have turned those files into snapshot files. I
wrote a small script to move the files rather than deleting and
recreating them so I could guarantee that the contents have not changed.
(Unfortunately, Git's diff doesn't always recognise the move as a move
because Insta has added a header.)

For tests of the successful case, I added a line to snapshot the
metadata rather than discarding it.

I also rewrote the tests to use `insta::glob` so we could get rid of
`test_each`.

V3_GIT_ORIGIN_REV_ID: 41bef4cf77bddb8d20d7c101df52ae149e8b0476
2024-06-26 08:15:29 +00:00