graphql-engine/dc-agents/sqlite

Data Connector Agent for SQLite

This directory contains an SQLite implementation of a data connector agent. It can use local SQLite database files as referenced by the "db" config field.

Capabilities

The SQLite agent currently supports the following capabilities:

• GraphQL Schema
• GraphQL Queries
• Relationships
• Aggregations
• Prometheus Metrics
• Exposing Foreign-Key Information
• Mutations
• Subscriptions
• Streaming Subscriptions

Note: You are able to get detailed metadata about the agent's capabilities by GETting the /capabilities endpoint of the running agent.

Requirements

• NodeJS 16
• SQLite >= 3.38.0 or compiled in JSON support
  • Required for the json_group_array() and json_group_object() aggregate SQL functions
  • https://www.sqlite.org/json1.html#jgrouparray
• Note: NPM is used for the TS Types for the DC-API protocol

Build & Run

npm install
npm run build
npm run start

Or a simple dev-loop via entr:

echo src/**/*.ts | xargs -n1 echo | DB_READONLY=y entr -r npm run start

Docker Build & Run

> docker build . -t dc-sqlite-agent:latest
> docker run -it --rm -p 8100:8100 dc-sqlite-agent:latest

You will want to mount a volume with your database(s) so that they can be referenced in configuration.

Options / Environment Variables

Note: Boolean flags {FLAG} can be provided as 1, true, yes, or omitted and default to false.

ENV Variable Name	Format	Default	Info
PORT	INT	8100	Port for agent to listen on.
PERMISSIVE_CORS	{FLAG}	false	Allows all requests - Useful for testing with SwaggerUI. Turn off on production.
DB_CREATE	{FLAG}	false	Allows new databases to be created.
DB_READONLY	{FLAG}	false	Makes databases readonly.
DB_ALLOW_LIST	DB1[,DB2]*	Any Allowed	Restrict what databases can be connected to.
DB_PRIVATECACHE	{FLAG}	Shared	Keep caches between connections private.
DEBUGGING_TAGS	{FLAG}	false	Outputs xml style tags in query comments for deugging purposes.
PRETTY_PRINT_LOGS	{FLAG}	false	Uses pino-pretty to pretty print request logs
LOG_LEVEL	fatal | error | info | debug | trace | silent	info	The minimum log level to output
METRICS	{FLAG}	false	Enables a /metrics prometheus metrics endpoint.
QUERY_LENGTH_LIMIT	INT	Infinity	Puts a limit on the length of generated SQL before execution.
DATASETS	{FLAG}	false	Enable dataset operations
DATASET_DELETE	{FLAG}	false	Enable DELETE /datasets/:name
DATASET_TEMPLATES	DIRECTORY	./dataset_templates	Directory to clone datasets from.
DATASET_CLONES	DIRECTORY	./dataset_clones	Directory to clone datasets to.

Agent usage

The agent is configured as per the configuration schema. The valid configuration properties are:

Property	Type	Default
db	string
tables	string[]	null
include_sqlite_meta_tables	boolean	false
explicit_main_schema	boolean	false

The only required property is db which specifies a local sqlite database to use.

The schema is exposed via introspection, but you can limit which tables are referenced by

• Explicitly enumerating them via the tables property, or
• Toggling the include_sqlite_meta_tables to include or exclude sqlite meta tables.

The explicit_main_schema field can be set to opt into exposing tables by their fully qualified names (ie ["main", "MyTable"] instead of just ["MyTable"]).

Dataset

The dataset used for testing the reference agent is sourced from:

• https://raw.githubusercontent.com/lerocha/chinook-database/master/ChinookDatabase/DataSources/Chinook_Sqlite.sql

Datasets

Datasets support is enabled via the ENV variables:

• DATASETS
• DATASET_DELETE
• DATASET_TEMPLATES
• DATASET_CLONES

Templates will be looked up at ${DATASET_TEMPLATES}/${template_name}.db.

Clones will be copied to ${DATASET_CLONES}/${clone_name}.db.

Testing Changes to the Agent

Run:

cabal run dc-api:test:tests-dc-api -- test --agent-base-url http://localhost:8100 --agent-config '{"db": "db.chinook2.sqlite"}'

From the HGE repo.

TODO

• Prometheus metrics hosted at /metrics
• Pull reference types from a package rather than checked-in files
• Health Check
• DB Specific Health Checks
• Schema
• Capabilities
• Query
• Array Relationships
• Object Relationships
• Ensure everything is escaped correctly - https://sequelize.org/api/v6/class/src/sequelize.js~sequelize#instance-method-escape
• Or... Use parameterized queries if possible - https://sequelize.org/docs/v6/core-concepts/raw-queries/#bind-parameter
• Run test-suite from SDK
• Remove old queries module
• Relationships / Joins
• Rename resultTT and other badly named types in the schema.ts module
• Add ENV Variable for restriction on what databases can be used
• Update to the latest types
• Port back to hge codebase as an official reference agent
• Make escapeSQL global to the query module
• Make CORS permissions configurable
• Optional DB Allowlist
• Fix SDK Test suite to be more flexible about descriptions
• READONLY option
• CREATE option
• Don't create DB option
• Aggregate queries
• Verbosity settings
• Cache settings
• Missing WHERE clause from object relationships
• Reuse find_table_relationship in more scenarios
• ORDER clause in aggregates breaks SQLite parser for some reason
• Check that looped exist check doesn't cause name conflicts
• NOT EXISTS IS NULL != EXISTS IS NOT NULL, Example: