graphql-engine/dc-agents/sqlite
Antoine Leblanc 30e772d3fa add content-length header.
## Description

Adds a content-length response header to all endpoints. This PR tests this feature by checking the content-length of every request we send in the tests.

## Changelog ✍️

__Component__ : server

__Type__: enhancement

__Product__: community-edition

### Short Changelog

add a content-length response header to all endpoints

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7444
Co-authored-by: Manas Agarwal <5352361+manasag@users.noreply.github.com>
GitOrigin-RevId: a0a811852053c5dde4b11b71ba11a7d456c84d76
2023-02-01 21:32:50 +00:00
..
src Implemented datasets support in Data Connector agent test suite 2023-01-30 07:00:26 +00:00
test add content-length header. 2023-02-01 21:32:50 +00:00
.gitignore Datasets implementation for dataconnectors 2023-01-17 05:49:10 +00:00
.nvmrc Move Typescript types for Data Connector agent into their own package 2022-09-05 06:09:23 +00:00
Dockerfile Sqlite reference agent 2022-08-05 07:12:51 +00:00
package-lock.json Implemented datasets support in Data Connector agent test suite 2023-01-30 07:00:26 +00:00
package.json Implemented datasets support in Data Connector agent test suite 2023-01-30 07:00:26 +00:00
README.md Datasets implementation for dataconnectors 2023-01-17 05:49:10 +00:00
tsconfig.json Sqlite reference agent 2022-08-05 07:12:51 +00:00

Data Connector Agent for SQLite

This directory contains an SQLite implementation of a data connector agent. It can use local SQLite database files as referenced by the "db" config field.

Capabilities

The SQLite agent currently supports the following capabilities:

  • GraphQL Schema
  • GraphQL Queries
  • Relationships
  • Aggregations
  • Prometheus Metrics
  • Exposing Foreign-Key Information
  • Mutations
  • Subscriptions
  • Streaming Subscriptions

Note: You are able to get detailed metadata about the agent's capabilities by GETting the /capabilities endpoint of the running agent.

Requirements

Build & Run

npm install
npm run build
npm run start

Or a simple dev-loop via entr:

echo src/**/*.ts | xargs -n1 echo | DB_READONLY=y entr -r npm run start

Docker Build & Run

> docker build . -t dc-sqlite-agent:latest
> docker run -it --rm -p 8100:8100 dc-sqlite-agent:latest

You will want to mount a volume with your database(s) so that they can be referenced in configuration.

Options / Environment Variables

Note: Boolean flags {FLAG} can be provided as 1, true, yes, or omitted and default to false.

ENV Variable Name Format Default Info
PORT INT 8100 Port for agent to listen on.
PERMISSIVE_CORS {FLAG} false Allows all requests - Useful for testing with SwaggerUI. Turn off on production.
DB_CREATE {FLAG} false Allows new databases to be created.
DB_READONLY {FLAG} false Makes databases readonly.
DB_ALLOW_LIST DB1[,DB2]* Any Allowed Restrict what databases can be connected to.
DB_PRIVATECACHE {FLAG} Shared Keep caches between connections private.
DEBUGGING_TAGS {FLAG} false Outputs xml style tags in query comments for deugging purposes.
PRETTY_PRINT_LOGS {FLAG} false Uses pino-pretty to pretty print request logs
LOG_LEVEL fatal | error | info | debug | trace | silent info The minimum log level to output
METRICS {FLAG} false Enables a /metrics prometheus metrics endpoint.
QUERY_LENGTH_LIMIT INT Infinity Puts a limit on the length of generated SQL before execution.
DATASETS {FLAG} false Enable dataset operations
DATASET_DELETE {FLAG} false Enable DELETE /datasets/:name
DATASET_TEMPLATES DIRECTORY ./dataset_templates Directory to clone datasets from.
DATASET_CLONES DIRECTORY ./dataset_clones Directory to clone datasets to.

Agent usage

The agent is configured as per the configuration schema. The valid configuration properties are:

Property Type Default
db string
tables string[] null
include_sqlite_meta_tables boolean false
explicit_main_schema boolean false

The only required property is db which specifies a local sqlite database to use.

The schema is exposed via introspection, but you can limit which tables are referenced by

  • Explicitly enumerating them via the tables property, or
  • Toggling the include_sqlite_meta_tables to include or exclude sqlite meta tables.

The explicit_main_schema field can be set to opt into exposing tables by their fully qualified names (ie ["main", "MyTable"] instead of just ["MyTable"]).

Dataset

The dataset used for testing the reference agent is sourced from:

Datasets

Datasets support is enabled via the ENV variables:

  • DATASETS
  • DATASET_DELETE
  • DATASET_TEMPLATES
  • DATASET_CLONES

Templates will be looked up at ${DATASET_TEMPLATES}/${template_name}.db.

Clones will be copied to ${DATASET_CLONES}/${clone_name}.db.

Testing Changes to the Agent

Run:

cabal run dc-api:test:tests-dc-api -- test --agent-base-url http://localhost:8100 --agent-config '{"db": "db.chinook2.sqlite"}'

From the HGE repo.

TODO

  • Prometheus metrics hosted at /metrics
  • Pull reference types from a package rather than checked-in files
  • Health Check
  • DB Specific Health Checks
  • Schema
  • Capabilities
  • Query
  • Array Relationships
  • Object Relationships
  • Ensure everything is escaped correctly - https://sequelize.org/api/v6/class/src/sequelize.js~sequelize#instance-method-escape
  • Or... Use parameterized queries if possible - https://sequelize.org/docs/v6/core-concepts/raw-queries/#bind-parameter
  • Run test-suite from SDK
  • Remove old queries module
  • Relationships / Joins
  • Rename resultTT and other badly named types in the schema.ts module
  • Add ENV Variable for restriction on what databases can be used
  • Update to the latest types
  • Port back to hge codebase as an official reference agent
  • Make escapeSQL global to the query module
  • Make CORS permissions configurable
  • Optional DB Allowlist
  • Fix SDK Test suite to be more flexible about descriptions
  • READONLY option
  • CREATE option
  • Don't create DB option
  • Aggregate queries
  • Verbosity settings
  • Cache settings
  • Missing WHERE clause from object relationships
  • Reuse find_table_relationship in more scenarios
  • ORDER clause in aggregates breaks SQLite parser for some reason
  • Check that looped exist check doesn't cause name conflicts
  • NOT EXISTS IS NULL != EXISTS IS NOT NULL, Example: