graphql-engine/dc-agents/sqlite
Lyndon Maydwell d54bb30d3b Structured Error Protocol for Data Connectors Agents - GDW-137
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6061
Co-authored-by: Vishnu Bharathi <4211715+scriptnull@users.noreply.github.com>
GitOrigin-RevId: 855d96378030f4e01b0c74b00e20e592e51e7a49
2022-10-11 00:26:24 +00:00
..
src Structured Error Protocol for Data Connectors Agents - GDW-137 2022-10-11 00:26:24 +00:00
test Sqlite reference agent 2022-08-05 07:12:51 +00:00
.gitignore Sqlite reference agent 2022-08-05 07:12:51 +00:00
.nvmrc Move Typescript types for Data Connector agent into their own package 2022-09-05 06:09:23 +00:00
Dockerfile Sqlite reference agent 2022-08-05 07:12:51 +00:00
package-lock.json Structured Error Protocol for Data Connectors Agents - GDW-137 2022-10-11 00:26:24 +00:00
package.json Structured Error Protocol for Data Connectors Agents - GDW-137 2022-10-11 00:26:24 +00:00
README.md Structured Error Protocol for Data Connectors Agents - GDW-137 2022-10-11 00:26:24 +00:00
tsconfig.json Sqlite reference agent 2022-08-05 07:12:51 +00:00

Data Connector Agent for SQLite

This directory contains an SQLite implementation of a data connector agent. It can use local SQLite database files as referenced by the "db" config field.

Capabilities

The SQLite agent currently supports the following capabilities:

  • GraphQL Schema
  • GraphQL Queries
  • Relationships
  • Aggregations
  • Prometheus Metrics
  • Exposing Foreign-Key Information
  • Mutations
  • Subscriptions
  • Streaming Subscriptions

Note: You are able to get detailed metadata about the agent's capabilities by GETting the /capabilities endpoint of the running agent.

Requirements

Build & Run

npm install
npm run build
npm run start

Or a simple dev-loop via entr:

echo src/**/*.ts | xargs -n1 echo | DB_READONLY=y entr -r npm run start

Docker Build & Run

> docker build . -t dc-sqlite-agent:latest
> docker run -it --rm -p 8100:8100 dc-sqlite-agent:latest

You will want to mount a volume with your database(s) so that they can be referenced in configuration.

Options / Environment Variables

Note: Boolean flags {FLAG} can be provided as 1, true, yes, or omitted and default to false.

ENV Variable Name Format Default Info
PORT INT 8100 Port for agent to listen on.
PERMISSIVE_CORS {FLAG} false Allows all requests - Useful for testing with SwaggerUI. Turn off on production.
DB_CREATE {FLAG} false Allows new databases to be created.
DB_READONLY {FLAG} false Makes databases readonly.
DB_ALLOW_LIST DB1[,DB2]* Any Allowed Restrict what databases can be connected to.
DB_PRIVATECACHE {FLAG} Shared Keep caches between connections private.
DEBUGGING_TAGS {FLAG} false Outputs xml style tags in query comments for deugging purposes.
PRETTY_PRINT_LOGS {FLAG} false Uses pino-pretty to pretty print request logs
LOG_LEVEL fatal | error | info | debug | trace | silent info The minimum log level to output
METRICS {FLAG} false Enables a /metrics prometheus metrics endpoint.
QUERY_LENGTH_LIMIT INT Infinity Puts a limit on the length of generated SQL before execution.

Agent usage

The agent is configured as per the configuration schema.

The only required field is db which specifies a local sqlite database to use.

The schema is exposed via introspection, but you can limit which tables are referenced by

  • Explicitly enumerating them via the tables field, or
  • Toggling the include_sqlite_meta_tables to include or exclude sqlite meta tables.

Dataset

The dataset used for testing the reference agent is sourced from:

Testing Changes to the Agent

Run:

cabal run dc-api:test:tests-dc-api -- test --agent-base-url http://localhost:8100 --agent-config '{"db": "db.chinook2.sqlite"}'

From the HGE repo.

TODO

  • Prometheus metrics hosted at /metrics
  • Pull reference types from a package rather than checked-in files
  • Health Check
  • DB Specific Health Checks
  • Schema
  • Capabilities
  • Query
  • Array Relationships
  • Object Relationships
  • Ensure everything is escaped correctly - https://sequelize.org/api/v6/class/src/sequelize.js~sequelize#instance-method-escape
  • Or... Use parameterized queries if possible - https://sequelize.org/docs/v6/core-concepts/raw-queries/#bind-parameter
  • Run test-suite from SDK
  • Remove old queries module
  • Relationships / Joins
  • Rename resultTT and other badly named types in the schema.ts module
  • Add ENV Variable for restriction on what databases can be used
  • Update to the latest types
  • Port back to hge codebase as an official reference agent
  • Make escapeSQL global to the query module
  • Make CORS permissions configurable
  • Optional DB Allowlist
  • Fix SDK Test suite to be more flexible about descriptions
  • READONLY option
  • CREATE option
  • Don't create DB option
  • Aggregate queries
  • Verbosity settings
  • Cache settings
  • Missing WHERE clause from object relationships
  • Reuse find_table_relationship in more scenarios
  • ORDER clause in aggregates breaks SQLite parser for some reason
  • Check that looped exist check doesn't cause name conflicts
  • NOT EXISTS IS NULL != EXISTS IS NOT NULL, Example:

Known Bugs

Tricky Aggregates may have logic bug

Replicate by running the agent test-suite.