NOTE: This PR is stacked on #756 and should be shipped after that is
merged.
This PR enables the existing aggregate relationships work (see #725,
#731, #756) by default by removing the experimental flag it used to be
disabled behind.
The new OpenDD schema changes that were added are also unhidden so that
they are visible in the OpenDD JSON Schema.
V3_GIT_ORIGIN_REV_ID: cfd86d8a9ea61887ccf0f1a5d08bdcc3dda59cdc
## Description
This PR implements the GraphQL schema and execution for aggregate
relationships.
In the `schema` crate, the new `model_aggregate_relationship_field`
function handles generating schema for ModelAggregateTarget
relationships. It mostly delegates the meat of its implementation to
reused logic; some refactoring has occurred to make this possible.
This involved changes in `select_many`, `select_aggregate` and
`model_arguments`. The creation of the model arguments field argument
now exists in `model_arguments` and is reused by `select_many` and
`select_aggregate`. The creation of all aggregate field arguments is now
in `select_aggregate::generate_select_aggregate_arguments`, and is then
reused when generating the aggregate relationship field. That field is
annotated with the new `RelationshipToModelAggregate` annotation.
In the `execute` crate, the logic around generating an the aggregate
selection IR was moved from `select_aggregate` into `model_selection`.
This was so it can be reused by the logic in `relationship` that now
uses it to generate an aggregate selection when encountering an
`RelationshipToModelAggregate` field.
Inside `relationship` some rearranging was done so that
`build_local_model_relationship` and `build_remote_relationship` could
work with either a normal model selection IR or the new aggregate
selection IR. The necessitated moving the creation of that IR outside
those functions into the caller, so the different callers can create
different IR (normal vs aggregate IR). This also reduced code
duplication.
New tests have been added to `engine` that cover aggregate relationships
and also remote joined aggregate relationships.
This PR also corrects two bugs in metadata resolve revealed by new
testing:
* The filter input field name in `GraphqlConfig` must be specified if
using an aggregate relationship
* The filter input type name defined on a `Model` must be specified if
that model is the target of an aggregate relationship. Conversely, the
filter input type name can be specified if the `Model` itself doesn't
define an aggregate, but is still involved in a aggregate relationship
(this previously produced an error).
This PR completes the feature, but it is still hidden behind the
experimental flag. There will be a follow up PR to remove that and
expose the functionality by default.
JIRA: [V3ENGINE-160](https://hasurahq.atlassian.net/browse/V3ENGINE-160)
[V3ENGINE-160]:
https://hasurahq.atlassian.net/browse/V3ENGINE-160?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
V3_GIT_ORIGIN_REV_ID: d499371906f7af71a4017c7c3ae75b7693cd3fa7
### What
In order to more easily monitor and review changes to metadata
resolution, this introduces snapshot testing for both successful and
failing calls to `resolve`. I used [Insta](https://insta.rs/) for this.
### How
For tests of the failure case, we already had a text file with the
expected error, so I have turned those files into snapshot files. I
wrote a small script to move the files rather than deleting and
recreating them so I could guarantee that the contents have not changed.
(Unfortunately, Git's diff doesn't always recognise the move as a move
because Insta has added a header.)
For tests of the successful case, I added a line to snapshot the
metadata rather than discarding it.
I also rewrote the tests to use `insta::glob` so we could get rid of
`test_each`.
V3_GIT_ORIGIN_REV_ID: 41bef4cf77bddb8d20d7c101df52ae149e8b0476
### What
This just means we can drop a bunch of references and pass by value.
### How
`#[derive(Clone, Copy)]` and fixing lints.
V3_GIT_ORIGIN_REV_ID: e15d323f8232755294d1f7a2c70ccf0de8a1632f
<!-- Thank you for submitting this PR! :) -->
## Description
Adding a test for this and fixing a few missing parts. Mostly threading
`boolean_expression_types` everywhere and adding them to our type
lookups. Behind a feature flag so this is a functional no-op.
V3_GIT_ORIGIN_REV_ID: 5fd6d5b9e06f0216e770b0715c59c0479881017f
<!-- Thank you for submitting this PR! :) -->
## Description
Previously we didn't check whether a boolean expression over strings was
used on a `String` field or a `User` field. Now we look up the types and
actually find out.
Functional no-op as behind a feature flag.
V3_GIT_ORIGIN_REV_ID: 8b7e94c4b873c49e206caa84e24c0d17c049c899
<!-- Thank you for submitting this PR! :) -->
## Description
When using boolean expression types on models, we have to check that any
relationships they define are local, as we currently do not support
remote predicates. This adds these checks in the `models_graphql` stage,
once we know about a) models and their sources b) boolean expressions c)
relationships.
Behind a feature flag, so strictly a no-op.
V3_GIT_ORIGIN_REV_ID: 70b2e4b316f5b8d57fa06d5492cccdddca0aaf1c
## Description
This PR continues on from #725 and adds the metadata resolve logic to
validate the usage of aggregates applied to relationships. The metadata
resolve logic is gated behind a new `enable_aggregate_relationships`
flag and disabled by default.
The new resolve logic lives in
`crates/metadata-resolve/src/stages/relationships/mod.rs`, however, most
of the actual logic that validates the usage of the aggregate expression
with the relationship has been reused from the model aggregate resolve
code. Subsequently, that code
(`crates/metadata-resolve/src/stages/models/aggregation.rs`) was
refactored to return a distinct error type
`ModelAggregateExpressionError` that can be composed with the
`RelationshipError` type, so the same errors can be returned with the
additional context of the relationship they were found in.
New metadata resolve error tests have been added in
`crates/metadata-resolve/tests/failing/aggregate_expression_in_relationship/*`.
Also, two new passing metadata resolve tests have been added to cover
the happy case of root field and relationship aggregate expressions:
`crates/metadata-resolve/tests/passing/aggregate_expressions/*`
JIRA: [V3ENGINE-160](https://hasurahq.atlassian.net/browse/V3ENGINE-160)
[V3ENGINE-160]:
https://hasurahq.atlassian.net/browse/V3ENGINE-160?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
V3_GIT_ORIGIN_REV_ID: 8cd7040cc9277940641d5ac239d7b34f8c1462c5
<!-- Thank you for submitting this PR! :) -->
## Description
We cannot validate `BooleanExpressionType`s much until they are used.
This adds checks for when they are used as model `where` clauses. We
ensure the data connector in question is allowed to use nested filtering
(if we try and do any), and ensure that every scalar field has mappings
for the appropriate data connector.
Hidden behind a feature flag, so technically a no-op.
V3_GIT_ORIGIN_REV_ID: ab54f913157954e8197798febafc70012bfad9ad
<!-- Thank you for submitting this PR! :) -->
## Description
Because we globbed for folders, we had to put a test in the root of each
folder which is messy. If we glob for files instead, like the `failing`
tests do, we can have a better time. Functional no-op.
V3_GIT_ORIGIN_REV_ID: 381f5a44d9ae6100198bd28a286493bc3a2ba98e
Return a `T` instead of a `Result<T, E>` when we never return an error
(`E`) case.
I also enabled some more warnings. `unnecessary_box_returns` has been
suppressed where appropriate, and `unused_async` doesn't seem to be
violated anywhere any more.
I got rid of some calls to `.unwrap()` too.
V3_GIT_ORIGIN_REV_ID: 015ebd05978cf8c2d87474a90e0cd4333779a761
## Description
This PR moves all the tests that were added to
`crates/engine/tests/validate_metadata_artifacts/aggregate_expressions`
to the metadata-resolve `metadata_golden_tests`.
All the `error.txt` files got renamed to `expected_error.txt` and the
`metadata is not consistent:` bit on the front of the error got trimmed
off.
I had to change the way that `test_failing_metadata` works so that
instead of looking for directories under `failing/` it looks for
`metadata.json` files under `failing/`. This is because not every
directory has a test in it, some directories are just used for grouping
(eg `failing/aggregate_expressions/`). This doesn't change what tests
get run. The only side effect is that every test name has `_metadata`
suffixed to it (the filename). 😢 `test-each` doesn't appear to offer a
way to disable this behaviour.
V3_GIT_ORIGIN_REV_ID: dece7cd3bb4623c51050d12471d2c0990fd69bfd
<!-- Thank you for submitting this PR! :) -->
## Description
We have a set of tests in `metadata-resolve` that uses globs to find
files in either `passing` or `failing` folders and check them. This
allows those files to live in nested folders for tidyness. Functional
no-op.
V3_GIT_ORIGIN_REV_ID: 22d42e986676a110f455c59c3df9e6946c5d996c
<!-- Thank you for submitting this PR! :) -->
## Description
This adds the ability to describe nested object boolean expressions,
which become `fieldPath` items in the generated
`ndc_models::ComparisonTarget::Column` items. This allows us to describe
filtering a `User` based on some element in their nested `address` field
(like `postcode`, for example).
Like the other `BooleanExpressionType` work, this remains behind a
feature flag so should make no user-facing changes.
It is also missing a whole heap of metadata resolve checks, going to
follow with these after doing the happy path to unblock other work.
V3_GIT_ORIGIN_REV_ID: c89e2942a651d349fca97706affcf40d91afeefb
## Description
There is a bug in NDC validation of command where it returns early
without validating. This is because we perform the NDC validation right
after resolving the command source, but haven't attached the "resolved
command source" to the command yet. As a result the command's source is
`None`, and the validation does an early return.
This PR fixes it by taking the command source directly. If there is no
source to resolve, we don't have to perform any NDC validation of the
command.
Note: this is not the case with models. Model's source resolving
attaches the "resolved source", then performs the validation.
## Changelog
- Add a changelog entry (in the "Changelog entry" section below) if the
changes
in this PR have any user-facing impact. See
[changelog
guide](https://github.com/hasura/graphql-engine-mono/wiki/Changelog-Guide).
- If no changelog is required ignore/remove this section and add a
`no-changelog-required` label to the PR.
### Product
_(Select all products this will be available in)_
- [x] community-edition
- [x] cloud
<!-- product : end : DO NOT REMOVE -->
### Type
<!-- See changelog structure:
https://github.com/hasura/graphql-engine-mono/wiki/Changelog-Guide#structure-of-our-changelog
-->
_(Select only one. In case of multiple, choose the most appropriate)_
- [ ] highlight
- [ ] enhancement
- [x] bugfix
- [ ] behaviour-change
- [ ] performance-enhancement
- [ ] security-fix
<!-- type : end : DO NOT REMOVE -->
### Changelog entry
<!--
- Add a user understandable changelog entry
- Include all details needed to understand the change. Try including
links to docs or issues if relevant
- For Highlights start with a H4 heading (#### <entry title>)
- Get the changelog entry reviewed by your team
-->
Fix skipping of NDC validation of commands due to wrong assumption.
<!-- changelog-entry : end : DO NOT REMOVE -->
<!-- changelog : end : DO NOT REMOVE -->
V3_GIT_ORIGIN_REV_ID: 5b9f4c922ee83394f1b6fe840cd934b6e138e2e9
<!-- Thank you for submitting this PR! :) -->
## Description
This somewhat resolves the `BooleanExpressionType` metadata type.
There is a lot that is missing, but everything is behind a feature so I
want to merge what is here before moving onto getting it working with
the later parts of the pipeline. We'll start with a lot of duplication
and then work to remove as much as we can.
Functional no-op for users, the feature flag is not exposed to the
actual binary.
How to review this PR:
- Check that any changes to existing code are cosmetic and naming-only
to ensure this is a no-op
- Look through the new resolve step and compare it to the
[RFC](https://github.com/hasura/v3-engine/blob/main/rfcs/open-dd-expression-type-changes.md)
and see if it is doing roughly what you might expect.
- Don't get too bogged down in missing things / style stuff, there are
definitely missing things and bad style.
V3_GIT_ORIGIN_REV_ID: 24e3b3e72e62d0094db3b461cb8c6d359982755d
`test_each::path` always gives us a `PathBuf` even if we don't want one,
so we need to suppress the associated warning.
V3_GIT_ORIGIN_REV_ID: 4f2bb29122df6979ad378227fb88e4632d3551f7
## Description
If the user specified their orderings in their GraphqlConfig in a
different order, they would be presented with an error.
```yaml
enumTypeNames:
- directions: # Reversed ordering of this list
- Desc
- Asc
typeName: OrderBy
```
> ERR Code=opendds-validation Message="invalid metadata: error building
schema: unable to build schema: metadata is not consistent: invalid
directions: Desc,Asc defined in orderByInput of GraphqlConfig ,
currenlty there is no support for partial directions. Please specify a
type which has both 'asc' and 'desc' directions"
7:56PM ERR Supergraph Build failed.
This has been fixed.
## Changelog
### Product
_(Select all products this will be available in)_
- [x] community-edition
- [x] cloud
<!-- product : end : DO NOT REMOVE -->
### Type
<!-- See changelog structure:
https://github.com/hasura/graphql-engine-mono/wiki/Changelog-Guide#structure-of-our-changelog
-->
_(Select only one. In case of multiple, choose the most appropriate)_
- [ ] highlight
- [ ] enhancement
- [x] bugfix
- [ ] behaviour-change
- [ ] performance-enhancement
- [ ] security-fix
<!-- type : end : DO NOT REMOVE -->
### Changelog entry
<!--
- Add a user understandable changelog entry
- Include all details needed to understand the change. Try including
links to docs or issues if relevant
- For Highlights start with a H4 heading (#### <entry title>)
- Get the changelog entry reviewed by your team
-->
Fixed arbitrary ordering of directions in GraphqlConfig causing invalid
error
<!-- changelog-entry : end : DO NOT REMOVE -->
<!-- changelog : end : DO NOT REMOVE -->
V3_GIT_ORIGIN_REV_ID: b27a0dd3b5c665f54ea58a40af8b2b1bfb0d2434
## Description
Parallel to #611, in service of removing supergraph config, we'd like to
map all current supergraph config into a generated subgraph. To ensure
that we don't end up with an unfortunate naming collision, I've decided
to refine the definition of a subgraph identifier to exclude any string
starting with `__internal`, which we can then have as our "internal
namespace". This change introduces the new type, and updates the
"unknown_namespace" subgraph to come under this umbrella.
This change should be a no-op to any user who hasn't used a subgraph
name prefixed with `__internal`.
<!--
Questions to consider answering:
1. What user-facing changes are being made?
2. What are issues related to this PR? (Consider adding `(close
#<issue-no>)` to the PR title)
3. What is the conceptual design behind this PR?
4. How can this PR be tested/verified?
5. Does the PR have limitations?
6. Does the PR introduce breaking changes?
-->
## Changelog
User-defined subgraphs may no longer be given names that start with
`__internal`, as this will be reserved for internally generated
subgraphs.
- Add a changelog entry (in the "Changelog entry" section below) if the
changes
in this PR have any user-facing impact. See
[changelog
guide](https://github.com/hasura/graphql-engine-mono/wiki/Changelog-Guide).
- If no changelog is required ignore/remove this section and add a
`no-changelog-required` label to the PR.
### Product
_(Select all products this will be available in)_
- [x] community-edition
- [x] cloud
<!-- product : end : DO NOT REMOVE -->
### Type
<!-- See changelog structure:
https://github.com/hasura/graphql-engine-mono/wiki/Changelog-Guide#structure-of-our-changelog
-->
_(Select only one. In case of multiple, choose the most appropriate)_
- [ ] highlight
- [ ] enhancement
- [ ] bugfix
- [x] behaviour-change
- [ ] performance-enhancement
- [ ] security-fix
<!-- type : end : DO NOT REMOVE -->
### Changelog entry
<!--
- Add a user understandable changelog entry
- Include all details needed to understand the change. Try including
links to docs or issues if relevant
- For Highlights start with a H4 heading (#### <entry title>)
- Get the changelog entry reviewed by your team
-->
User-defined subgraphs may no longer have names that start with
`__internal`.
<!-- changelog-entry : end : DO NOT REMOVE -->
<!-- changelog : end : DO NOT REMOVE -->
V3_GIT_ORIGIN_REV_ID: a6df1bba7b1f2cda4593654c15d168f25b7134b2
## Description
As per [APG-113](https://hasurahq.atlassian.net/browse/APG-113), we
would like to remove the separate concept of supergraphs. This is the
first step: we allow the supergraph config to appear in subgraphs. The
next step will be to remove the non-subgraph version of it.
This PR updates the subgraph enum type to allow for supergraph config
objects, but does nothing else. This means it should be a no-op for
current users, but will be forward-compatible with the eventual goal of
the epic. We also throw in a makeshift test framework that we can
probably replace with Insta at a later date.
<!--
Questions to consider answering:
1. What user-facing changes are being made?
2. What are issues related to this PR? (Consider adding `(close
#<issue-no>)` to the PR title)
3. What is the conceptual design behind this PR?
4. How can this PR be tested/verified?
5. Does the PR have limitations?
6. Does the PR introduce breaking changes?
-->
## Changelog
- Add a changelog entry (in the "Changelog entry" section below) if the
changes
in this PR have any user-facing impact. See
[changelog
guide](https://github.com/hasura/graphql-engine-mono/wiki/Changelog-Guide).
- If no changelog is required ignore/remove this section and add a
`no-changelog-required` label to the PR.
### Product
_(Select all products this will be available in)_
- [x] community-edition
- [x] cloud
<!-- product : end : DO NOT REMOVE -->
### Type
<!-- See changelog structure:
https://github.com/hasura/graphql-engine-mono/wiki/Changelog-Guide#structure-of-our-changelog
-->
_(Select only one. In case of multiple, choose the most appropriate)_
- [ ] highlight
- [ ] enhancement
- [ ] bugfix
- [x] behaviour-change
- [ ] performance-enhancement
- [ ] security-fix
<!-- type : end : DO NOT REMOVE -->
### Changelog entry
<!--
- Add a user understandable changelog entry
- Include all details needed to understand the change. Try including
links to docs or issues if relevant
- For Highlights start with a H4 heading (#### <entry title>)
- Get the changelog entry reviewed by your team
-->
Supergraph configuration (`CompatibilityConfig`, `AuthConfig`, and
`GraphqlConfig`) may now be defined in any subgraph. Note that
supergraph configuration may still only be defined once for any given
supergraph.
<!-- changelog-entry : end : DO NOT REMOVE -->
<!-- changelog : end : DO NOT REMOVE -->
[APG-113]:
https://hasurahq.atlassian.net/browse/APG-113?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ
V3_GIT_ORIGIN_REV_ID: a1b9911c4abb332fe37914efd1c3afbbd554a5d0
<!-- Thank you for submitting this PR! :) -->
## Description
We have a formatting check job but it was not a required check so it got
missed, this fixes everything and will make the check required in
Github. Functional no-op.
V3_GIT_ORIGIN_REV_ID: 514478f2a48482ca34a860fedcfe6185b29c1dc3
<!-- Thank you for submitting this PR! :) -->
## Description
We have a new `BooleanExpressionType` metadata kind. This adds it, tests
it can be parsed, but hides it from generated metadata and throws an
error if one is actually used in the engine.
V3_GIT_ORIGIN_REV_ID: 036b5fd9c32475d1c5a5e5e6321fb736fe6caefa
