3d5fb984b0
## Description There is a bug in the metadata defaults code, see [the original PR](https://github.com/hasura/graphql-engine-mono/pull/6286). Steps to reproduce this issue: * Start a new HGE project * Start HGE with a defaults argument: `HASURA_GRAPHQL_LOG_LEVEL=debug cabal run exe:graphql-engine -- serve --enable-console --console-assets-dir=./console/static/dist --metadata-defaults='{"backend_configs": {"dataconnector": {"mongo": {"display_name": "BONGOBB", "uri": "http://localhost:8123"}}}}'` * Add a source (doesn't need to be related to the defaults) * Export metadata * See that the defaults are present in the exported metadata ## Related Issues * Github Issue: https://github.com/hasura/graphql-engine/issues/9237 * Jira: https://hasurahq.atlassian.net/browse/GDC-647 * Original PR: https://github.com/hasura/graphql-engine-mono/pull/6286 ## Solution * The test for if defaults should be included for metadata api operations has been extended to check for updates * Metadata inconsistencies have been hidden for `/capabilities` calls on startup ## TODO * [x] Fix bug * [x] Write tests * [x] OSS Metadata Migration to correct persisted data - `server/src-rsr/migrations/47_to_48.sql` * [x] Cloud Metadata Migration - `pro/server/res/cloud/migrations/6_to_7.sql` * [x] Bump Catalog Version - `server/src-rsr/catalog_version.txt` * [x] Update Catalog Versions - `server/src-rsr/catalog_versions.txt` (This will be done by Infra when creating a release) * [x] Log connection error as it occurs *(Already being logged. Requires `--enabled-log-types startup,webhook-log,websocket-log,http-log,data-connector-log`) * [x] Don't mark metadata inconsistencies for this call. ## Questions * [ ] Does the `pro/server/res/cloud/migrations/6_to_7.sql` cover the cloud scenarios? * [ ] Should we have `SET search_path` in migrations? * [x] What should be in `server/src-rsr/catalog_versions.txt`? ## Testing To test the solution locally run: > docker compose up -d and > cabal run -- exe:api-tests --skip BigQuery --skip SQLServer --skip '/Test.API.Explain/Postgres/' ## Solution In `runMetadataQuery` in `server/src-lib/Hasura/Server/API/Metadata.hs`: ```diff - if (exportsMetadata _rqlMetadata) + if (exportsMetadata _rqlMetadata || queryModifiesMetadata _rqlMetadata) ``` This ensures that defaults aren't present in operations that serialise metadata. Note: You might think that `X_add_source` would need the defaults to be present to add a source that references the defaults, but since the resolution occurs in the schema-cache building phase, the defaults can be excluded for the metadata modifications required for `X_add_source`. In addition to the code-change, a metadata migration has been introduced in order to clean up serialised defaults. The following scenarios need to be considered for both OSS and Cloud: * The user has not had defaults serialised * The user has had the defaults serialised and no other backends configured * The user has had the defaults serialised and has also configured other backends We want to remove as much of the metadata as possible without any user-specified data and this should be reflected in migration `server/src-rsr/migrations/47_to_48.sql`. ## Server checklist ### Catalog upgrade Does this PR change Hasura Catalog version? - ✅ Yes ### Metadata Does this PR add a new Metadata feature? - ✅ No ### GraphQL - ✅ No new GraphQL schema is generated ### Breaking changes - ✅ No Breaking changes ## Changelog __Component__ : server __Type__: bugfix __Product__: community-edition ### Short Changelog Fixes a metadata defaults serialization bug and introduces a metadata migration to correct data that has been persisted due to the bug. PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7034 GitOrigin-RevId: ad7d4f748397a1a607f2c0c886bf0fbbc3f873f2 |
||
---|---|---|
.. | ||
src | ||
test | ||
.gitignore | ||
.nvmrc | ||
Dockerfile | ||
package-lock.json | ||
package.json | ||
README.md | ||
tsconfig.json |
Data Connector Agent for SQLite
This directory contains an SQLite implementation of a data connector agent. It can use local SQLite database files as referenced by the "db" config field.
Capabilities
The SQLite agent currently supports the following capabilities:
- GraphQL Schema
- GraphQL Queries
- Relationships
- Aggregations
- Prometheus Metrics
- Exposing Foreign-Key Information
- Mutations
- Subscriptions
- Streaming Subscriptions
Note: You are able to get detailed metadata about the agent's capabilities by
GET
ting the /capabilities
endpoint of the running agent.
Requirements
- NodeJS 16
- SQLite
>= 3.38.0
or compiled in JSON support- Required for the json_group_array() and json_group_object() aggregate SQL functions
- https://www.sqlite.org/json1.html#jgrouparray
- Note: NPM is used for the TS Types for the DC-API protocol
Build & Run
npm install
npm run build
npm run start
Or a simple dev-loop via entr
:
echo src/**/*.ts | xargs -n1 echo | DB_READONLY=y entr -r npm run start
Docker Build & Run
> docker build . -t dc-sqlite-agent:latest
> docker run -it --rm -p 8100:8100 dc-sqlite-agent:latest
You will want to mount a volume with your database(s) so that they can be referenced in configuration.
Options / Environment Variables
Note: Boolean flags {FLAG}
can be provided as 1
, true
, yes
, or omitted and default to false
.
ENV Variable Name | Format | Default | Info |
---|---|---|---|
PORT |
INT |
8100 |
Port for agent to listen on. |
PERMISSIVE_CORS |
{FLAG} |
false |
Allows all requests - Useful for testing with SwaggerUI. Turn off on production. |
DB_CREATE |
{FLAG} |
false |
Allows new databases to be created. |
DB_READONLY |
{FLAG} |
false |
Makes databases readonly. |
DB_ALLOW_LIST |
DB1[,DB2]* |
Any Allowed | Restrict what databases can be connected to. |
DB_PRIVATECACHE |
{FLAG} |
Shared | Keep caches between connections private. |
DEBUGGING_TAGS |
{FLAG} |
false |
Outputs xml style tags in query comments for deugging purposes. |
PRETTY_PRINT_LOGS |
{FLAG} |
false |
Uses pino-pretty to pretty print request logs |
LOG_LEVEL |
fatal | error | info | debug | trace | silent |
info |
The minimum log level to output |
METRICS |
{FLAG} |
false |
Enables a /metrics prometheus metrics endpoint. |
QUERY_LENGTH_LIMIT |
INT |
Infinity |
Puts a limit on the length of generated SQL before execution. |
Agent usage
The agent is configured as per the configuration schema. The valid configuration properties are:
Property | Type | Default |
---|---|---|
db |
string |
|
tables |
string[] |
null |
include_sqlite_meta_tables |
boolean |
false |
explicit_main_schema |
boolean |
false |
The only required property is db
which specifies a local sqlite database to use.
The schema is exposed via introspection, but you can limit which tables are referenced by
- Explicitly enumerating them via the
tables
property, or - Toggling the
include_sqlite_meta_tables
to include or exclude sqlite meta tables.
The explicit_main_schema
field can be set to opt into exposing tables by their fully qualified names (ie ["main", "MyTable"]
instead of just ["MyTable"]
).
Dataset
The dataset used for testing the reference agent is sourced from:
Testing Changes to the Agent
Run:
cabal run dc-api:test:tests-dc-api -- test --agent-base-url http://localhost:8100 --agent-config '{"db": "db.chinook2.sqlite"}'
From the HGE repo.
TODO
- Prometheus metrics hosted at
/metrics
- Pull reference types from a package rather than checked-in files
- Health Check
- DB Specific Health Checks
- Schema
- Capabilities
- Query
- Array Relationships
- Object Relationships
- Ensure everything is escaped correctly - https://sequelize.org/api/v6/class/src/sequelize.js~sequelize#instance-method-escape
- Or... Use parameterized queries if possible - https://sequelize.org/docs/v6/core-concepts/raw-queries/#bind-parameter
- Run test-suite from SDK
- Remove old queries module
- Relationships / Joins
- Rename
resultTT
and other badly named types in theschema.ts
module - Add ENV Variable for restriction on what databases can be used
- Update to the latest types
- Port back to hge codebase as an official reference agent
- Make escapeSQL global to the query module
- Make CORS permissions configurable
- Optional DB Allowlist
- Fix SDK Test suite to be more flexible about descriptions
- READONLY option
- CREATE option
- Don't create DB option
- Aggregate queries
- Verbosity settings
- Cache settings
- Missing WHERE clause from object relationships
- Reuse
find_table_relationship
in more scenarios - ORDER clause in aggregates breaks SQLite parser for some reason
- Check that looped exist check doesn't cause name conflicts
NOT EXISTS IS NULL
!=EXISTS IS NOT NULL
, Example: