graphql-engine/server/src-lib/Hasura
David Overton e5f88d8039 Nested array support for Data Connectors Backend and MongoDB
## Description

This change adds support for querying into nested arrays in Data Connector agents that support such a concept (currently MongoDB).

### DC API changes

- New API type `ColumnType` which allows representing the type of a "column" as either a scalar type, an object reference or an array of `ColumnType`s. This recursive definition allows arbitrary nesting of arrays of types.
- The `type` fields in the API types `ColumnInfo` and `ColumnInsertSchema` now take a `ColumnType` instead of a `ScalarType`.
- To ensure backwards compatibility, a `ColumnType` representing a scalar serialises and deserialises to the same representation as `ScalarType`.
- In queries, the `Field` type now has a new constructor `NestedArrayField`. This contains a nested `Field` along with optional `limit`, `offset`, `where` and `order_by` arguments. (These optional arguments are not yet used by either HGE or the MongoDB agent.)

### MongoDB Haskell agent changes

- The `/schema` endpoint will now recognise arrays within the JSON validation schema and generate corresponding arrays in the DC schema.
- The `/query` endpoint will now handle `NestedArrayField`s within queries (although it does not yet handle `limit`, `offset`, `where` and `order_by`).

### HGE server changes

- The `Backend` type class adds a new type family `XNestedArrays b` to enable nested arrays on a per-backend basis (currently enabled only for the `DataConnector` backend.
- Within `RawColumnInfo` the column type is now represented by a new type `RawColumnType b` which mirrors the shape of the DC API `ColumnType`, but uses `XNestedObjects b` and `XNestedArrays b` type families to allow turning nested object and array supports on or off for a particular backend. In the `DataConnector` backend `API.CustomType` is converted into `RawColumnInfo 'DataConnector` while building the schema.
- In the next stage of schema building, the `RawColumnInfo` is converted into a `StructuredColumnInfo` which allows us to represent the three different types of columns: scalar, object and array. TODO: the `StructuredColumnInfo` looks very similar to the Logical Model types. The main difference is that it uses the `XNestedObjects` and `XNestedArrays` type families. We should be able to combine these two representations.
- The `StructuredColumnInfo` is then placed into a `FIColumn` `FieldInfo`. This involved some refactoring of `FieldInfo` as I had previously split out `FINestedObject` into a separate constructor. However it works out better to represent all "column" fields (i.e. scalar, object and array) using `FIColumn` as this make it easier to implement permission checking correctly. This is the reason the `StructuredColumnInfo` was needed.
- Next, the `FieldInfo` are used to generate `FieldParser`s. We add a new constructor to `AnnFieldG` for `AFNestedArray`. An `AFNestedArray` field parser can contain either a simple array selection or an array aggregate. Simple array `FieldParsers` are currently limited to subfield selection. We will add support for limit, offset, where and order_by in a future PR. We also don't yet generate array aggregate `FieldParsers.
- The new `AFNestedArray` field is handled by the `QueryPlan` module in the `DataConnector` backend. There we generate an `API.NestedArrayField` from the AFNestedArray. We also handle nested arrays when reshaping the response from the DC agent.

## Limitations

- Support for limit, offset, filter (where) and order_by is not yet fully implemented, although it should not be hard to add this
- Support for aggregations on nested arrays is not yet fully implemented
- Permissions involving nested arrays (and objects) not yet implemented
- This should be integrated with Logical Model types, but that will happen in a separate PR

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/9149
GitOrigin-RevId: 0e7b71a994fc1d2ca1ef73bfe7b96e95b5328531
2023-05-24 08:02:43 +00:00
..
App chore(server): split NamingCase type from functions 2023-05-17 14:03:40 +00:00
Backends Nested array support for Data Connectors Backend and MongoDB 2023-05-24 08:02:43 +00:00
Cache server, pro: actually reformat the code-base using ormolu 2021-09-23 22:57:37 +00:00
Eventing chore(server): move table related things to Hasura.Table.* 2023-05-17 08:55:32 +00:00
Function Adding UDF (user-defined-functions) support to Data Connectors - GDC-820 2023-05-19 04:48:46 +00:00
GraphQL Nested array support for Data Connectors Backend and MongoDB 2023-05-24 08:02:43 +00:00
LogicalModel Nested array support for Data Connectors Backend and MongoDB 2023-05-24 08:02:43 +00:00
LogicalModelResolver chore(server): remove NQ -> table relationships 2023-05-16 15:09:18 +00:00
Metadata server: methods for loading and storing source introspection 2023-05-15 15:58:43 +00:00
NativeQuery chore(server): move table related things to Hasura.Table.* 2023-05-17 08:55:32 +00:00
QueryTags chore(server): move some query tags code to a sensible place 2023-03-30 21:19:38 +00:00
RemoteSchema server: use stored introspection only when upstream remote schema unreachable 2023-05-16 17:03:10 +00:00
RQL Nested array support for Data Connectors Backend and MongoDB 2023-05-24 08:02:43 +00:00
Server server: collect oss resource usage metrics 2023-05-24 02:44:12 +00:00
Services harmonize network manager handling 2023-02-22 15:55:54 +00:00
SQL chore(server): delete MySQL native backend 2023-05-05 10:32:56 +00:00
StoredProcedure chore(server): move table related things to Hasura.Table.* 2023-05-17 08:55:32 +00:00
Table Nested array support for Data Connectors Backend and MongoDB 2023-05-24 08:02:43 +00:00
Tracing server: add the env var for toggling metric granularity 2023-05-11 05:21:16 +00:00
App.hs server: collect oss resource usage metrics 2023-05-24 02:44:12 +00:00
ClientCredentials.hs RemoveHasura.RQL.Types dependency on Backends.Postgres.Execute.Types 2023-04-24 14:41:19 +00:00
CredentialCache.hs server: Make HLint pass, by hook or by crook. 2023-04-25 14:03:02 +00:00
GC.hs Upgrade Ormolu to v0.5. 2022-11-02 20:55:13 +00:00
HTTP.hs server: improve logging around HTTP Exception errors for Actions and Remote Schemas 2023-05-03 07:56:37 +00:00
Logging.hs Always flush the loggers without explicitly relying on onException 2023-04-05 11:07:14 +00:00
Name.hs chore(server): delete MySQL native backend 2023-05-05 10:32:56 +00:00
PingSources.hs Fix: Unable to use environment variable for CockroachDB connection string in Cloud 2023-02-14 12:15:58 +00:00
QueryTags.hs chore(server): move some query tags code to a sensible place 2023-03-30 21:19:38 +00:00
Services.hs harmonize network manager handling 2023-02-22 15:55:54 +00:00
Session.hs Import HashMap, not HM, Map, M... 2023-04-26 15:43:44 +00:00
ShutdownLatch.hs multitenant: make shutdown logic more abstract 2022-11-23 07:53:19 +00:00
Tracing.hs PLAT-438: improve traces from OpenTelemetry 2023-04-26 16:21:29 +00:00