## Description This change adds support for nested object fields in HGE IR and Schema Cache, the Data Connectors backend and API, and the MongoDB agent. ### Data Connector API changes - The `/schema` endpoint response now includes an optional set of GraphQL type definitions. Table column types can refer to these definitions by name. - Queries can now include a new field type `object` which contains a column name and a nested query. This allows querying into a nested object within a field. ### MongoDB agent changes - Add support for querying into nested documents using the new `object` field type. ### HGE changes - The `Backend` type class has a new type family `XNestedObjects b` which controls whether or not a backend supports querying into nested objects. This is currently enabled only for the `DataConnector` backend. - For backends that support nested objects, the `FieldInfo` type gets a new constructor `FINestedObject`, and the `AnnFieldG` type gets a new constructor `AFNestedObject`. - If the DC `/schema` endpoint returns any custom GraphQL type definitions they are stored in the `TableInfo` for each table in the source. - During schema cache building, the function `addNonColumnFields` will check whether any column types match custom GraphQL object types stored in the `TableInfo`. If so, they are converted into `FINestedObject` instead of `FIColumn` in the `FieldInfoMap`. - When building the `FieldParser`s from `FieldInfo` (function `fieldSelection`) any `FINestedObject` fields are converted into nested object parsers returning `AFNestedObject`. - The `DataConnector` query planner converts `AFNestedObject` fields into `object` field types in the query sent to the agent. ## Limitations ### HGE not yet implemented: - Support for nested arrays - Support for nested objects/arrays in mutations - Support for nested objects/arrays in order-by - Support for filters (`where`) in nested objects/arrays - Support for adding custom GraphQL types via track table metadata API - Support for interface and union types - Tests for nested objects ### Mongo agent not yet implemented: - Generate nested object types from validation schema - Support for aggregates - Support for order-by - Configure agent port - Build agent in CI - Agent tests for nested objects and MongoDB agent PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7844 GitOrigin-RevId: aec9ec1e4216293286a68f9b1af6f3f5317db423 |
||
---|---|---|
.. | ||
dataset_templates | ||
src | ||
test | ||
.gitignore | ||
.nvmrc | ||
Dockerfile | ||
package-lock.json | ||
package.json | ||
README.md | ||
tsconfig.json |
Data Connector Agent for SQLite
This directory contains an SQLite implementation of a data connector agent. It can use local SQLite database files as referenced by the "db" config field.
Capabilities
The SQLite agent currently supports the following capabilities:
- GraphQL Schema
- GraphQL Queries
- Relationships
- Aggregations
- Prometheus Metrics
- Exposing Foreign-Key Information
- Mutations
- Subscriptions
- Streaming Subscriptions
Note: You are able to get detailed metadata about the agent's capabilities by
GET
ting the /capabilities
endpoint of the running agent.
Requirements
- NodeJS 16
- SQLite
>= 3.38.0
or compiled in JSON support- Required for the json_group_array() and json_group_object() aggregate SQL functions
- https://www.sqlite.org/json1.html#jgrouparray
- Note: NPM is used for the TS Types for the DC-API protocol
Build & Run
npm install
npm run build
npm run start
Or a simple dev-loop via entr
:
echo src/**/*.ts | xargs -n1 echo | DB_READONLY=y entr -r npm run start
Docker Build & Run
> docker build . -t dc-sqlite-agent:latest
> docker run -it --rm -p 8100:8100 dc-sqlite-agent:latest
You will want to mount a volume with your database(s) so that they can be referenced in configuration.
Options / Environment Variables
Note: Boolean flags {FLAG}
can be provided as 1
, true
, t
, yes
, y
, or omitted and default to false
.
ENV Variable Name | Format | Default | Info |
---|---|---|---|
PORT |
INT |
8100 |
Port for agent to listen on. |
PERMISSIVE_CORS |
{FLAG} |
false |
Allows all requests - Useful for testing with SwaggerUI. Turn off on production. |
DB_CREATE |
{FLAG} |
false |
Allows new databases to be created. |
DB_READONLY |
{FLAG} |
false |
Makes databases readonly. |
DB_ALLOW_LIST |
DB1[,DB2]* |
Any Allowed | Restrict what databases can be connected to. |
DB_PRIVATECACHE |
{FLAG} |
Shared | Keep caches between connections private. |
DEBUGGING_TAGS |
{FLAG} |
false |
Outputs xml style tags in query comments for deugging purposes. |
PRETTY_PRINT_LOGS |
{FLAG} |
false |
Uses pino-pretty to pretty print request logs |
LOG_LEVEL |
fatal | error | info | debug | trace | silent |
info |
The minimum log level to output |
METRICS |
{FLAG} |
false |
Enables a /metrics prometheus metrics endpoint. |
QUERY_LENGTH_LIMIT |
INT |
Infinity |
Puts a limit on the length of generated SQL before execution. |
DATASETS |
{FLAG} |
false |
Enable dataset operations |
DATASET_DELETE |
{FLAG} |
false |
Enable DELETE /datasets/:name |
DATASET_TEMPLATES |
DIRECTORY |
./dataset_templates |
Directory to clone datasets from. |
DATASET_CLONES |
DIRECTORY |
./dataset_clones |
Directory to clone datasets to. |
MUTATIONS |
{FLAG} |
false |
Enable Mutation Support. |
Agent usage
The agent is configured as per the configuration schema. The valid configuration properties are:
Property | Type | Default |
---|---|---|
db |
string |
|
tables |
string[] |
null |
include_sqlite_meta_tables |
boolean |
false |
explicit_main_schema |
boolean |
false |
The only required property is db
which specifies a local sqlite database to use.
The schema is exposed via introspection, but you can limit which tables are referenced by
- Explicitly enumerating them via the
tables
property, or - Toggling the
include_sqlite_meta_tables
to include or exclude sqlite meta tables.
The explicit_main_schema
field can be set to opt into exposing tables by their fully qualified names (ie ["main", "MyTable"]
instead of just ["MyTable"]
).
Dataset
The dataset used for testing the reference agent is sourced from:
Datasets
Datasets support is enabled via the ENV variables:
DATASETS
DATASET_DELETE
DATASET_TEMPLATES
DATASET_CLONES
Templates will be looked up at ${DATASET_TEMPLATES}/${template_name}.sqlite
or ${DATASET_TEMPLATES}/${template_name}.sql
. The .sqlite
templates are just SQLite database files that will be copied as a clone. The .sql
templates are SQL script files that will be run against a blank SQLite database in order to create a clone.
Clones will be copied to ${DATASET_CLONES}/${clone_name}.sqlite
.
Testing Changes to the Agent
Ensure you run the agent with DATASETS=1 DATASET_DELETE=1 MUTATIONS=1
in order to enable testing of mutations.
Then run:
cabal run dc-api:test:tests-dc-api -- test --agent-base-url http://localhost:8100 sandwich --tui
From the HGE repo.
Known Issues
- Using "returning" in insert/update/delete mutations where you join across relationships that are affected by the insert/update/delete mutation itself may return inconsistent results. This is because of this issue with SQLite: https://sqlite.org/forum/forumpost/9470611066
TODO
- Prometheus metrics hosted at
/metrics
- Pull reference types from a package rather than checked-in files
- Health Check
- DB Specific Health Checks
- Schema
- Capabilities
- Query
- Array Relationships
- Object Relationships
- Ensure everything is escaped correctly - https://sequelize.org/api/v6/class/src/sequelize.js~sequelize#instance-method-escape
- Or... Use parameterized queries if possible - https://sequelize.org/docs/v6/core-concepts/raw-queries/#bind-parameter
- Run test-suite from SDK
- Remove old queries module
- Relationships / Joins
- Rename
resultTT
and other badly named types in theschema.ts
module - Add ENV Variable for restriction on what databases can be used
- Update to the latest types
- Port back to hge codebase as an official reference agent
- Make escapeSQL global to the query module
- Make CORS permissions configurable
- Optional DB Allowlist
- Fix SDK Test suite to be more flexible about descriptions
- READONLY option
- CREATE option
- Don't create DB option
- Aggregate queries
- Verbosity settings
- Cache settings
- Missing WHERE clause from object relationships
- Reuse
find_table_relationship
in more scenarios - ORDER clause in aggregates breaks SQLite parser for some reason
- Check that looped exist check doesn't cause name conflicts
NOT EXISTS IS NULL
!=EXISTS IS NOT NULL
- Mutation support