graphql-engine/server/src-lib/Hasura/RQL/DDL
David Overton e5f88d8039 Nested array support for Data Connectors Backend and MongoDB
## Description

This change adds support for querying into nested arrays in Data Connector agents that support such a concept (currently MongoDB).

### DC API changes

- New API type `ColumnType` which allows representing the type of a "column" as either a scalar type, an object reference or an array of `ColumnType`s. This recursive definition allows arbitrary nesting of arrays of types.
- The `type` fields in the API types `ColumnInfo` and `ColumnInsertSchema` now take a `ColumnType` instead of a `ScalarType`.
- To ensure backwards compatibility, a `ColumnType` representing a scalar serialises and deserialises to the same representation as `ScalarType`.
- In queries, the `Field` type now has a new constructor `NestedArrayField`. This contains a nested `Field` along with optional `limit`, `offset`, `where` and `order_by` arguments. (These optional arguments are not yet used by either HGE or the MongoDB agent.)

### MongoDB Haskell agent changes

- The `/schema` endpoint will now recognise arrays within the JSON validation schema and generate corresponding arrays in the DC schema.
- The `/query` endpoint will now handle `NestedArrayField`s within queries (although it does not yet handle `limit`, `offset`, `where` and `order_by`).

### HGE server changes

- The `Backend` type class adds a new type family `XNestedArrays b` to enable nested arrays on a per-backend basis (currently enabled only for the `DataConnector` backend.
- Within `RawColumnInfo` the column type is now represented by a new type `RawColumnType b` which mirrors the shape of the DC API `ColumnType`, but uses `XNestedObjects b` and `XNestedArrays b` type families to allow turning nested object and array supports on or off for a particular backend. In the `DataConnector` backend `API.CustomType` is converted into `RawColumnInfo 'DataConnector` while building the schema.
- In the next stage of schema building, the `RawColumnInfo` is converted into a `StructuredColumnInfo` which allows us to represent the three different types of columns: scalar, object and array. TODO: the `StructuredColumnInfo` looks very similar to the Logical Model types. The main difference is that it uses the `XNestedObjects` and `XNestedArrays` type families. We should be able to combine these two representations.
- The `StructuredColumnInfo` is then placed into a `FIColumn` `FieldInfo`. This involved some refactoring of `FieldInfo` as I had previously split out `FINestedObject` into a separate constructor. However it works out better to represent all "column" fields (i.e. scalar, object and array) using `FIColumn` as this make it easier to implement permission checking correctly. This is the reason the `StructuredColumnInfo` was needed.
- Next, the `FieldInfo` are used to generate `FieldParser`s. We add a new constructor to `AnnFieldG` for `AFNestedArray`. An `AFNestedArray` field parser can contain either a simple array selection or an array aggregate. Simple array `FieldParsers` are currently limited to subfield selection. We will add support for limit, offset, where and order_by in a future PR. We also don't yet generate array aggregate `FieldParsers.
- The new `AFNestedArray` field is handled by the `QueryPlan` module in the `DataConnector` backend. There we generate an `API.NestedArrayField` from the AFNestedArray. We also handle nested arrays when reshaping the response from the DC agent.

## Limitations

- Support for limit, offset, filter (where) and order_by is not yet fully implemented, although it should not be hard to add this
- Support for aggregations on nested arrays is not yet fully implemented
- Permissions involving nested arrays (and objects) not yet implemented
- This should be integrated with Logical Model types, but that will happen in a separate PR

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/9149
GitOrigin-RevId: 0e7b71a994fc1d2ca1ef73bfe7b96e95b5328531
2023-05-24 08:02:43 +00:00
..
Action chore(server): remove TH aeson, separate lenses, in Hasura.RQL.DDL 2023-04-28 15:13:32 +00:00
Metadata chore(server): remove TH aeson, separate lenses, in Hasura.RQL.DDL 2023-04-28 15:13:32 +00:00
Permission Nested array support for Data Connectors Backend and MongoDB 2023-05-24 08:02:43 +00:00
Relationship chore(server): move table related things to Hasura.Table.* 2023-05-17 08:55:32 +00:00
Schema Nested array support for Data Connectors Backend and MongoDB 2023-05-24 08:02:43 +00:00
Webhook server: add HTTP response status in the Kriti context for Actions 2023-05-15 16:35:22 +00:00
Action.hs chore(server): remove TH aeson, separate lenses, in Hasura.RQL.DDL 2023-04-28 15:13:32 +00:00
ApiLimit.hs server: add the env var for toggling metric granularity 2023-05-11 05:21:16 +00:00
ComputedField.hs chore(server): move table related things to Hasura.Table.* 2023-05-17 08:55:32 +00:00
ConnectionTemplate.hs server: add connection_template as an argument in pg_test_connection_template API 2023-04-03 14:27:12 +00:00
CustomTypes.hs chore(server): move table related things to Hasura.Table.* 2023-05-17 08:55:32 +00:00
DataConnector.hs Data Connectors: Support using array literals in permission checks 2023-05-24 04:42:02 +00:00
Endpoint.hs Import InsOrdHashMap, not OMap, OM, Map, HM, ... 2023-04-27 07:43:22 +00:00
EventTrigger.hs chore(server): move table related things to Hasura.Table.* 2023-05-17 08:55:32 +00:00
FeatureFlag.hs Import J, not Aeson, A, JSON, Yaml... 2023-04-26 17:30:24 +00:00
GraphqlSchemaIntrospection.hs Remove HasServerConfigCtx from the schema cache build. 2023-03-27 17:44:27 +00:00
Headers.hs Split Hasura.RQL.DDL.Headers to put types in Hasura.RQL.Types.Headers 2023-04-24 16:45:55 +00:00
InheritedRoles.hs Import InsOrdHashMap, not OMap, OM, Map, HM, ... 2023-04-27 07:43:22 +00:00
Metadata.hs chore(server): move table related things to Hasura.Table.* 2023-05-17 08:55:32 +00:00
Network.hs Extract Network and Net into hasura-extras 2023-04-24 11:13:34 +00:00
OpenTelemetry.hs Split Hasura.RQL.DDL.Headers to put types in Hasura.RQL.Types.Headers 2023-04-24 16:45:55 +00:00
Permission.hs Nested array support for Data Connectors Backend and MongoDB 2023-05-24 08:02:43 +00:00
QueryCollection.hs Import InsOrdHashMap, not OMap, OM, Map, HM, ... 2023-04-27 07:43:22 +00:00
QueryTags.hs chore(server): remove TH aeson, separate lenses, in Hasura.RQL.DDL 2023-04-28 15:13:32 +00:00
Relationship.hs chore(server): move table related things to Hasura.Table.* 2023-05-17 08:55:32 +00:00
RemoteRelationship.hs Nested array support for Data Connectors Backend and MongoDB 2023-05-24 08:02:43 +00:00
ScheduledTrigger.hs Import InsOrdHashMap, not OMap, OM, Map, HM, ... 2023-04-27 07:43:22 +00:00
Schema.hs chore(server): move table related things to Hasura.Table.* 2023-05-17 08:55:32 +00:00
SourceKinds.hs Import J, not Aeson, A, JSON, Yaml... 2023-04-26 17:30:24 +00:00
Warnings.hs Add untrack_tables metadata API 2023-05-17 15:31:56 +00:00