graphql-engine/server/src-rsr/migrations
Lyndon Maydwell 3d5fb984b0 Fix metadata defaults bug - Defaults serialised into metadata table - GDC-647
## Description

There is a bug in the metadata defaults code, see [the original PR](https://github.com/hasura/graphql-engine-mono/pull/6286).

Steps to reproduce this issue:

* Start a new HGE project
* Start HGE with a defaults argument: `HASURA_GRAPHQL_LOG_LEVEL=debug cabal run exe:graphql-engine -- serve --enable-console --console-assets-dir=./console/static/dist --metadata-defaults='{"backend_configs": {"dataconnector": {"mongo": {"display_name": "BONGOBB", "uri": "http://localhost:8123"}}}}'`
* Add a source (doesn't need to be related to the defaults)
* Export metadata
* See that the defaults are present in the exported metadata

## Related Issues

* Github Issue: https://github.com/hasura/graphql-engine/issues/9237
* Jira: https://hasurahq.atlassian.net/browse/GDC-647
* Original PR: https://github.com/hasura/graphql-engine-mono/pull/6286

## Solution

* The test for if defaults should be included for metadata api operations has been extended to check for updates
* Metadata inconsistencies have been hidden for `/capabilities` calls on startup

## TODO

* [x] Fix bug
* [x] Write tests
* [x] OSS Metadata Migration to correct persisted data - `server/src-rsr/migrations/47_to_48.sql`
* [x] Cloud Metadata Migration - `pro/server/res/cloud/migrations/6_to_7.sql`
* [x] Bump Catalog Version - `server/src-rsr/catalog_version.txt`
* [x] Update Catalog Versions - `server/src-rsr/catalog_versions.txt` (This will be done by Infra when creating a release)
* [x] Log connection error as it occurs *(Already being logged. Requires `--enabled-log-types startup,webhook-log,websocket-log,http-log,data-connector-log`)
* [x] Don't mark metadata inconsistencies for this call.

## Questions

* [ ] Does the `pro/server/res/cloud/migrations/6_to_7.sql` cover the cloud scenarios?
* [ ] Should we have `SET search_path` in migrations?
* [x] What should be in `server/src-rsr/catalog_versions.txt`?

## Testing

To test the solution locally run:

> docker compose up -d

and

> cabal run  -- exe:api-tests --skip BigQuery --skip SQLServer --skip '/Test.API.Explain/Postgres/'

## Solution

In `runMetadataQuery` in `server/src-lib/Hasura/Server/API/Metadata.hs`:

```diff
-        if (exportsMetadata _rqlMetadata)
+        if (exportsMetadata _rqlMetadata || queryModifiesMetadata _rqlMetadata)
```

This ensures that defaults aren't present in operations that serialise metadata.

Note: You might think that `X_add_source` would need the defaults to be present to add a source that references the defaults, but since the resolution occurs in the schema-cache building phase, the defaults can be excluded for the metadata modifications required for `X_add_source`.

In addition to the code-change, a metadata migration has been introduced in order to clean up serialised defaults.

The following scenarios need to be considered for both OSS and Cloud:

* The user has not had defaults serialised
* The user has had the defaults serialised and no other backends configured
* The user has had the defaults serialised and has also configured other backends

We want to remove as much of the metadata as possible without any user-specified data and this should be reflected in migration `server/src-rsr/migrations/47_to_48.sql`.

## Server checklist

### Catalog upgrade

Does this PR change Hasura Catalog version?
-  Yes

### Metadata
Does this PR add a new Metadata feature?
-  No

### GraphQL
-  No new GraphQL schema is generated

### Breaking changes
-  No Breaking changes

## Changelog

__Component__ : server
__Type__: bugfix
__Product__: community-edition

### Short Changelog

Fixes a metadata defaults serialization bug and introduces a metadata migration to correct data that has been persisted due to the bug.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7034
GitOrigin-RevId: ad7d4f748397a1a607f2c0c886bf0fbbc3f873f2
2022-12-06 22:35:19 +00:00
..
1_to_2.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
2_to_3.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
4_to_5.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
5_to_6.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
6_to_7.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
7_to_8.sql escape sql underscores in LIKE expressions (close #8435) 2022-06-08 18:54:19 +00:00
8_to_9.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
9_to_10.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
08_to_1.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
10_to_11.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
11_to_12.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
12_to_13.sql escape sql underscores in LIKE expressions (close #8435) 2022-06-08 18:54:19 +00:00
13_to_12.sql escape sql underscores in LIKE expressions (close #8435) 2022-06-08 18:54:19 +00:00
13_to_14.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
14_to_13.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
14_to_15.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
15_to_14.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
15_to_16.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
16_to_15.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
16_to_17.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
17_to_16.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
17_to_18.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
18_to_17.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
18_to_19.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
19_to_18.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
19_to_20.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
20_to_19.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
20_to_21.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
21_to_20.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
21_to_22.sql escape sql underscores in LIKE expressions (close #8435) 2022-06-08 18:54:19 +00:00
22_to_21.sql escape sql underscores in LIKE expressions (close #8435) 2022-06-08 18:54:19 +00:00
22_to_23.sql escape sql underscores in LIKE expressions (close #8435) 2022-06-08 18:54:19 +00:00
23_to_22.sql escape sql underscores in LIKE expressions (close #8435) 2022-06-08 18:54:19 +00:00
23_to_24.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
24_to_23.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
24_to_25.sql Make catalog metadata migrations work on all schema versions (fix #2826) (#2379) 2019-10-11 00:13:57 -05:00
25_to_24.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
25_to_26.sql escape sql underscores in LIKE expressions (close #8435) 2022-06-08 18:54:19 +00:00
26_to_25.sql escape sql underscores in LIKE expressions (close #8435) 2022-06-08 18:54:19 +00:00
26_to_27.sql improve event fetch query (#3236) 2019-11-13 12:59:19 +05:30
27_to_26.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
27_to_28.sql functions can access session info via input arg (close #2322) (#3143) 2019-11-20 12:17:06 +05:30
28_to_27.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
28_to_29.sql Update 28_to_29.sql 2021-08-09 16:36:44 +00:00
29_to_28.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
29_to_30.sql allow custom mutations through actions (#3042) 2020-02-13 23:08:23 +05:30
30_to_29.sql Add downgrade command (close #1156) (#3760) 2020-02-07 16:33:12 +05:30
30_to_31.sql Explicitly invalidate enum values when metadata is reloaded 2020-01-30 18:17:29 -06:00
31_to_30.sql Explicitly invalidate enum values when metadata is reloaded 2020-01-30 18:17:29 -06:00
31_to_32.sql allow custom mutations through actions (#3042) 2020-02-13 23:08:23 +05:30
32_to_31.sql Fix catalog version for v1.1.1 (close #4354) (#4355) 2020-04-09 19:57:59 -05:00
32_to_33.sql server(events): utilize proper backpressure scheme (close #3839) (#4013) 2020-03-11 11:57:31 +05:30
33_to_32.sql server(events): utilize proper backpressure scheme (close #3839) (#4013) 2020-03-11 11:57:31 +05:30
33_to_34.sql Remove any outstanding hdb_views (fixes #3940) (#3965) 2020-03-26 15:42:33 -05:00
34_to_33.sql Scheduled triggers (close #1914) (#3553) 2020-05-13 18:03:16 +05:30
34_to_35.sql Scheduled triggers (close #1914) (#3553) 2020-05-13 18:03:16 +05:30
35_to_34.sql Scheduled triggers (close #1914) (#3553) 2020-05-13 18:03:16 +05:30
35_to_36.sql Remote Joins: Create relationships across database and remote schemas (#2392) 2020-05-27 20:32:58 +05:30
36_to_35.sql Remote Joins: Create relationships across database and remote schemas (#2392) 2020-05-27 20:32:58 +05:30
36_to_37.sql Propagate the trace context to event triggers (#5409) 2020-07-23 13:39:26 -07:00
37_to_36.sql Propagate the trace context to event triggers (#5409) 2020-07-23 13:39:26 -07:00
37_to_38.sql Merge oss/master onto mono/main 2020-11-12 22:37:19 +05:30
38_to_37.sql Merge oss/master onto mono/main 2020-11-12 22:37:19 +05:30
38_to_39.sql Merge oss/master onto mono/main 2020-11-12 22:37:19 +05:30
39_to_38.sql Merge oss/master onto mono/main 2020-11-12 22:37:19 +05:30
39_to_40.sql server: move to storing metadata as a json blob (#115) 2020-12-08 14:23:28 +00:00
40_to_39.sql Merge oss/master onto mono/main 2020-11-12 22:37:19 +05:30
41_to_40.sql server: relocate some of the pre metadata separation source catalog migrations 2021-11-03 14:21:40 +00:00
41_to_42.sql server: relocate some of the pre metadata separation source catalog migrations 2021-11-03 14:21:40 +00:00
42_to_41.sql Merge oss/master onto mono/main 2020-11-12 22:37:19 +05:30
42_to_43.sql server: V2 maintenance mode fixes 2021-04-21 10:56:13 +00:00
43_to_42.sql escape sql underscores in LIKE expressions (close #8435) 2022-06-08 18:54:19 +00:00
43_to_44.sql Use exceptions to handle compatibility with pg 9.6 in 2.0 upgrade migration 2021-06-24 02:20:11 +00:00
44_to_43.sql Optimistic Metadata Locking 2021-02-19 02:40:23 +00:00
44_to_45.sql server/postgres: fix resetting the metadata catalog version to 43 while initializing postgres source with 1.0 catalog (#1645) 2021-06-23 18:01:09 +00:00
45_to_44.sql Schema-Sync Improvements for Cloud, Pro, and OSS - Version Checking for Metadata (#738) 2021-04-06 03:25:53 +00:00
45_to_46.sql server/postgres: fix resetting the metadata catalog version to 43 while initializing postgres source with 1.0 catalog (#1645) 2021-06-23 18:01:09 +00:00
46_to_45.sql server: add unique constraints to the cron events table (#1131) 2021-04-15 11:21:58 +00:00
46_to_47.sql server: add an index on event_id in hdb_cron_event_invocation_logs table 2021-08-17 05:45:11 +00:00
47_to_46.sql server: add an index on event_id in hdb_cron_event_invocation_logs table 2021-08-17 05:45:11 +00:00
47_to_48.sql Fix metadata defaults bug - Defaults serialised into metadata table - GDC-647 2022-12-06 22:35:19 +00:00
48_to_47.sql Fix metadata defaults bug - Defaults serialised into metadata table - GDC-647 2022-12-06 22:35:19 +00:00