Commit Graph

36 Commits

Author SHA1 Message Date
Lyndon Maydwell
e9dcbf6491 Adding custom scalar types to SQLite DC Agent - GDC-610
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6783
Co-authored-by: David Overton <7734777+dmoverton@users.noreply.github.com>
Co-authored-by: Daniel Chambers <1214352+daniel-chambers@users.noreply.github.com>
GitOrigin-RevId: ee53c20b0090b6b3c88792ddc98b8287872fe0f3
2022-12-08 06:50:08 +00:00
Daniel Chambers
c14fd3ba4c Add mutability properties to the Data Connector schema API [GDC-664]
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7190
GitOrigin-RevId: ce602b5e5cc5aee8716ff3f7a036b18b3bf47188
2022-12-08 02:07:01 +00:00
Lyndon Maydwell
3d5fb984b0 Fix metadata defaults bug - Defaults serialised into metadata table - GDC-647
## Description

There is a bug in the metadata defaults code, see [the original PR](https://github.com/hasura/graphql-engine-mono/pull/6286).

Steps to reproduce this issue:

* Start a new HGE project
* Start HGE with a defaults argument: `HASURA_GRAPHQL_LOG_LEVEL=debug cabal run exe:graphql-engine -- serve --enable-console --console-assets-dir=./console/static/dist --metadata-defaults='{"backend_configs": {"dataconnector": {"mongo": {"display_name": "BONGOBB", "uri": "http://localhost:8123"}}}}'`
* Add a source (doesn't need to be related to the defaults)
* Export metadata
* See that the defaults are present in the exported metadata

## Related Issues

* Github Issue: https://github.com/hasura/graphql-engine/issues/9237
* Jira: https://hasurahq.atlassian.net/browse/GDC-647
* Original PR: https://github.com/hasura/graphql-engine-mono/pull/6286

## Solution

* The test for if defaults should be included for metadata api operations has been extended to check for updates
* Metadata inconsistencies have been hidden for `/capabilities` calls on startup

## TODO

* [x] Fix bug
* [x] Write tests
* [x] OSS Metadata Migration to correct persisted data - `server/src-rsr/migrations/47_to_48.sql`
* [x] Cloud Metadata Migration - `pro/server/res/cloud/migrations/6_to_7.sql`
* [x] Bump Catalog Version - `server/src-rsr/catalog_version.txt`
* [x] Update Catalog Versions - `server/src-rsr/catalog_versions.txt` (This will be done by Infra when creating a release)
* [x] Log connection error as it occurs *(Already being logged. Requires `--enabled-log-types startup,webhook-log,websocket-log,http-log,data-connector-log`)
* [x] Don't mark metadata inconsistencies for this call.

## Questions

* [ ] Does the `pro/server/res/cloud/migrations/6_to_7.sql` cover the cloud scenarios?
* [ ] Should we have `SET search_path` in migrations?
* [x] What should be in `server/src-rsr/catalog_versions.txt`?

## Testing

To test the solution locally run:

> docker compose up -d

and

> cabal run  -- exe:api-tests --skip BigQuery --skip SQLServer --skip '/Test.API.Explain/Postgres/'

## Solution

In `runMetadataQuery` in `server/src-lib/Hasura/Server/API/Metadata.hs`:

```diff
-        if (exportsMetadata _rqlMetadata)
+        if (exportsMetadata _rqlMetadata || queryModifiesMetadata _rqlMetadata)
```

This ensures that defaults aren't present in operations that serialise metadata.

Note: You might think that `X_add_source` would need the defaults to be present to add a source that references the defaults, but since the resolution occurs in the schema-cache building phase, the defaults can be excluded for the metadata modifications required for `X_add_source`.

In addition to the code-change, a metadata migration has been introduced in order to clean up serialised defaults.

The following scenarios need to be considered for both OSS and Cloud:

* The user has not had defaults serialised
* The user has had the defaults serialised and no other backends configured
* The user has had the defaults serialised and has also configured other backends

We want to remove as much of the metadata as possible without any user-specified data and this should be reflected in migration `server/src-rsr/migrations/47_to_48.sql`.

## Server checklist

### Catalog upgrade

Does this PR change Hasura Catalog version?
-  Yes

### Metadata
Does this PR add a new Metadata feature?
-  No

### GraphQL
-  No new GraphQL schema is generated

### Breaking changes
-  No Breaking changes

## Changelog

__Component__ : server
__Type__: bugfix
__Product__: community-edition

### Short Changelog

Fixes a metadata defaults serialization bug and introduces a metadata migration to correct data that has been persisted due to the bug.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7034
GitOrigin-RevId: ad7d4f748397a1a607f2c0c886bf0fbbc3f873f2
2022-12-06 22:35:19 +00:00
David Overton
a18c6976f8 Map scalar types to GraphQL built-in types for parsing [GDC-587]
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6818
GitOrigin-RevId: 6d1887fb7865fe8ec24a73c77da291c7ecf8dfd2
2022-12-01 00:08:40 +00:00
Daniel Chambers
ed79049637 Mutation Data Connector API types [GDC-594]
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6893
GitOrigin-RevId: edf0f72027197ae61bfa8d27d53eabf6ca626112
2022-11-29 03:37:13 +00:00
Daniel Chambers
7e9f5bdd96 Fixed filtering not being applied across object relations in SQLite agent
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6937
GitOrigin-RevId: 655e9be98059a1e86566b975ae646b49d67fa06e
2022-11-18 08:15:50 +00:00
Lyndon Maydwell
7228d0327f Add display_name, release_name fields to MD Agent APIs - GDC-626
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6849
GitOrigin-RevId: 0ab90aaf281cc1c043f73fd6d63c4c18d58c7c92
2022-11-18 04:19:08 +00:00
Daniel Chambers
1a7d7fd0c0 Fix Data Connector TS type generation CI check and regenerate types
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6866
GitOrigin-RevId: 7ecfdaa4af9ca7dc7b349bd8981f6ef083f5a543
2022-11-15 07:51:51 +00:00
Daniel Chambers
431311e74e Add support for ordering by related tables and aggregates to the SQLite agent [GDC-313]
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6773
GitOrigin-RevId: 9cca50511aa9e8ef7c4e2a20f280be0fa51aab85
2022-11-09 06:06:58 +00:00
Daniel Harvey
48bb73ecb4 [server/tests] create new Postgres DB for each test
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6741
GitOrigin-RevId: 0fa10ff8e3ab0ca9c36b1146ebf70434836f2531
2022-11-08 11:44:13 +00:00
David Overton
87bcdb97c7 Use JSON instead of GraphQL for comparison operators
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6563
GitOrigin-RevId: 0819414df199292e0146ce3009295ce3e49f2439
2022-10-28 01:14:09 +00:00
David Overton
9921823915 GDC-189 custom aggregations
>

## Description
->

This PR allows DC agents to define custom aggregate functions for their scalar types.

### Related Issues
->

GDC-189

### Solution and Design
>

We added a new property `aggregate_functions` to the scalar types capabilities. This allows the agent author to specify a set of aggregate functions supported by each scalar type, along with the function's result type.

During GraphQL schema generation, the custom aggregate functions are available via a new method `getCustomAggregateOperators` on the `Backend` type class.
Custom functions are merged with the builtin aggregate functions when building GraphQL schemas for table aggregate fields and for `order_by` operators on array relations.

### Steps to test and verify
>

• Codec tests for aggregate function capabilities have been added to the unit tests.
• Some custom aggregate operators have been added to the reference agent and are used in a new test in `api-tests`.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6199
GitOrigin-RevId: e9c0d1617af93847c1493671fdbb794f573bde0c
2022-10-27 00:44:06 +00:00
Daniel Chambers
84b84a78e1 Add column scalar type information Data Connector query API request [GDC-493]
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6433
GitOrigin-RevId: c5f5e0c2e25c6820d9f73a1e90699cf18dc4cd47
2022-10-20 03:24:53 +00:00
Solomon
5e85b0c534 Mutable Sqlite GDC Fixture - GDC-455
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6351
GitOrigin-RevId: bdcb3cfa694dda6a0d1473331bc689aae418a5bb
2022-10-17 22:53:14 +00:00
Daniel Chambers
6d8231903c Reworked scalar type determination in SQLite agent [GDC-455]
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6335
GitOrigin-RevId: baf7619a0c705bddcbd75f8842bf8fcfeeaa05c9
2022-10-13 19:03:25 +00:00
Daniel Chambers
da8533526c Adds explicit main schema option to SQLite data connector agent [GDC-455]
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6307
Co-authored-by: Solomon <24038+solomon-b@users.noreply.github.com>
GitOrigin-RevId: 90433478867ffc21826f07df8587da86bd43dc88
2022-10-13 04:20:11 +00:00
Matthew Goodwin
ea6d9226d7 Sql lite testing revisions
As I have been onboarding, in order to get a working sqlite agent/db running locally, I got some helpful steps from @mattshardman which I adapted into this testing setup.

Hopefully this should ease the pain for the next person needing to do this!

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6261
GitOrigin-RevId: 03a3f04e38f118898bfd1d27de9305739dee5276
2022-10-12 20:28:24 +00:00
Lyndon Maydwell
d54bb30d3b Structured Error Protocol for Data Connectors Agents - GDW-137
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6061
Co-authored-by: Vishnu Bharathi <4211715+scriptnull@users.noreply.github.com>
GitOrigin-RevId: 855d96378030f4e01b0c74b00e20e592e51e7a49
2022-10-11 00:26:24 +00:00
Daniel Chambers
8369cac3bd Data Connector agent data schema capabilities [GDC-479]
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6268
GitOrigin-RevId: 4ec29566d3c2ab2144dad8055b4442a4027915ec
2022-10-10 06:59:30 +00:00
Daniel Chambers
c862c64b33 Break down Data Connector agent schema tests into finer grained, more accurate, tests [GDC-479]
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6225
GitOrigin-RevId: 8eaadcc14cf4bc90a22450e924daf8cc340ed2ea
2022-10-10 02:25:09 +00:00
Tom Harding
a1c5ac46f6 Extract dc-api and its tests from graphql-engine.cabal
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6000
Co-authored-by: Daniel Chambers <1214352+daniel-chambers@users.noreply.github.com>
GitOrigin-RevId: d8f3cd0194191ab4c51a5a7d350cb2c3a0a5a7f3
2022-09-27 07:21:54 +00:00
Lyndon Maydwell
4d2e37b3e6 Raw Query Support for Data Connectors - GDW-394
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5890
Co-authored-by: Vijay Prasanna <11921040+vijayprasanna13@users.noreply.github.com>
GitOrigin-RevId: f6bd2ed5fe170bcce262564cf4f45c95c9bdff94
2022-09-22 21:09:06 +00:00
Daniel Chambers
3365a18516 Rename DataConnector cross_table capability to subquery
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5980
GitOrigin-RevId: c289b2a7afb4d07214aac2b4d89c9b87a75e98aa
2022-09-21 10:48:05 +00:00
Daniel Chambers
dc9a86680c Gardening: Clean up Data Connector API types
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5977
GitOrigin-RevId: 1a6898d6416fff265a8add74d414c979f7fa3bc5
2022-09-21 05:13:03 +00:00
Daniel Chambers
04ae6abf78 Exists support in filter expressions for Data Connector queries [GDW-133]
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5904
GitOrigin-RevId: 6bad4c29a7d14d3881f9c57fe983d14cc41bdc4b
2022-09-20 04:01:33 +00:00
Lyndon Maydwell
ba0a685211 CI Support for Data Connector Agent Tests - GDW-336
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5808
GitOrigin-RevId: 817575185cdbd21985c647f22d5b643ce0852ef6
2022-09-15 09:47:24 +00:00
David Overton
f4419236ed Gdc capabilities scalar types - GDW-87
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5597
GitOrigin-RevId: 4f561bf476266955d7b1d3dbca4d406a97bf8b34
2022-09-06 04:26:03 +00:00
Daniel Chambers
97b0e4c591 Move Typescript types for Data Connector agent into their own package
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5596
GitOrigin-RevId: c5da90eb4e61a9d9a5ddc34f7bfbaa2d00c698b8
2022-09-05 06:09:23 +00:00
Daniel Chambers
9ef243f0b2 Fixed Constraint's foreign table not being typed with TableName in Data Connector API
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5687
GitOrigin-RevId: 5f616e65c493e640770d0b53f265bd2b62ad9369
2022-09-01 00:52:35 +00:00
Daniel Chambers
d5af9dfa05 Fixed SQLite agent health endpoint
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5688
GitOrigin-RevId: 17813d348cd37e379daaebf4854daf0ae2c51b1a
2022-08-31 05:11:50 +00:00
Lyndon Maydwell
88c5b14c3d Adding EXPLAIN endpoint capability and implementation for SQLite data connector agent GDW-181
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5603
GitOrigin-RevId: b9255d265ceb88d96643c95f409387fb93df6676
2022-08-29 03:40:16 +00:00
Lyndon Maydwell
8fdd0ac8f5 Adding FK support to the SQLite data connectors agent schema endpoint
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5598
GitOrigin-RevId: 3613c39958bd5dcb2aca497475f4bd69a3479e6f
2022-08-25 05:07:15 +00:00
Daniel Chambers
f6d15e4ef2 SQLite Data Connector Agent fixes
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5505
GitOrigin-RevId: 928d83b3913864258f5cd452f6e5d410a5778cbd
2022-08-22 05:27:47 +00:00
Lyndon Maydwell
5d751bf0ba Adding metrics capability to data-connector agents (with specs)
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5444
GitOrigin-RevId: 49242e83fc1efeb5665515b44368e930818baf27
2022-08-17 01:47:36 +00:00
Lyndon Maydwell
4b86dfa4ad Adding metrics endpoint to SQLite Data Connector Agent
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5418
GitOrigin-RevId: 28f3b3e14f610ca6b880711d03cf01486413d5ab
2022-08-10 06:38:07 +00:00
Lyndon Maydwell
fc907201a0 Sqlite reference agent
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5263
Co-authored-by: Daniel Chambers <1214352+daniel-chambers@users.noreply.github.com>
GitOrigin-RevId: fb5cf3cb80ab59cd5ee0d064e3f776c062856aa3
2022-08-05 07:12:51 +00:00