Changes compared to `/v1alpha1/graphql`
* Changed all graphql responses in **/v1/graphql** endpoint to be 200. All graphql clients expect responses to be HTTP 200. Non-200 responses are considered transport layer errors.
* Errors in http and websocket layer are now consistent and have similar structure.
1. Reuses postgres connections during startup which reduces the overhead of opening and closing connections.
2. Faster schema cache building. This is done by fetching all the required data in a single sql statement.
* build schema cache function without db setup
The setup shouldn't happen for sync. The database is already setup by the instance which generated the event. This means that the sync is now faster.
* use SQL loop to drop hdb_views schema views and routines with ordering
This avoids deadlocks when schema is being changed concurrently
* schema sync now only processes the latest event
This becomes useful when a lot of schema change
events happen while we are still processing an
earlier event.
* split stm transactions when snapshotting to make it faster
* mx subs: push to both old and new sinks at the same time
* expose dev APIs through allowed APIs flag
* add types to represent unparsed http gql requests
This will help when we add caching of frequently used ASTs
* query plan caching
* move livequery to execute
* add multiplexed module
* session variable can be customised depending on the context
Previously the value was always "current_setting('hasura.user')"
* get rid of typemap requirement in reusable plan
* subscriptions are multiplexed when possible
* use lazytx for introspection to avoid acquiring a pg connection
* refactor to make execute a completely decoupled module
* don't issue a transaction for a query
* don't use current setting for explained sql
* move postgres related types to a different module
* validate variableValues on postgres before multiplexing subs
* don't user current_setting for queries over ws
* plan_cache is only visible when developer flag is enabled
* introduce 'batch size' when multiplexing subscriptions
* bump stackage to 13.16
* fix schema_stitching test case error code
* store hashes instead of actual responses for subscriptions
* internal api to dump subscriptions state
* remove PlanCache from SchemaCacheRef
* allow live query options to be configured on server startup
* capture metrics for multiplexed subscriptions
* more metrics captured for multiplexed subs
* switch to tvar based hashmap for faster snapshotting
* livequery modules do not expose internal details
* fix typo in live query env vars
* switch to hasura's pg-client-hs
* read version from env var at build time (close#1398)
* remove un-used imports, edit makefile
* edit makefile to add new targets and export variables
* only export VERSION in makefile
* read version by executing the script if env var is absent
Examples
1) `
pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv
`
2) `pytest --hge-urls "http://127.0.0.1:8080" "http://127.0.0.1:8081" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" "postgresql://admin@127.0.0.1:5432/hge_tests2" -vv
`
### Solution and Design
<!-- How is this issue solved/fixed? What is the design? -->
<!-- It's better if we elaborate -->
#### Reducing execution time of tests
- The Schema setup and teardown, which were earlier done per test method, usually takes around 1 sec.
- For mutations, the model has now been changed to only do schema setup and teardown once per test class.
- A data setup and teardown will be done once per test instead (usually takes ~10ms).
- For the test class to get this behaviour, one can can extend the class `DefaultTestMutations`.
- The function `dir()` should be define which returns the location of the configuration folder.
- Inside the configuration folder, there should be
- Files `<conf_dir>/schema_setup.yaml` and `<conf_dir>/schema_teardown.yaml`, which has the metadata query executed during schema setup and teardown respectively
- Files named `<conf_dir>/values_setup.yaml` and `<conf_dir>/values_teardown.yaml`. These files are executed to setup and remove data from the tables respectively.
#### Running Graphql queries on both http and websockets
- Each GraphQL query/mutation is run on the both HTTP and websocket protocols
- Pytests test parameterisation is used to achieve this
- The errors over websockets are slightly different from that on HTTP
- The code takes care of converting the errors in HTTP to errors in websockets
#### Parallel executation of tests.
- The plugin pytest-xdist helps in running tests on parallel workers.
- We are using this plugin to group tests by file and run on different workers.
- Parallel test worker processes operate on separate postgres databases(and separate graphql-engines connected to these databases). Thus tests on one worker will not affect the tests on the other worker.
- With two workers, this decreases execution times by half, as the tests on event triggers usually takes a long time, but does not consume much CPU.
From `alpha-40` we've been using a `WHERE` clause to fetch required rows and generate mutation response. This has a few limitations like the requirement of a primary key/unique constraint. This also returns inconsistent data on `delete` mutation as mentioned in #1794.
Now, we're using `VALUES (..)` (refer [here](https://www.postgresql.org/docs/current/sql-values.html)) expression to form virtual table rows in `SQL` to generate mutation response.
Internal changes:-
- Not to use primary key/unique constraint columns:-
- Revert back to `ConstraintName` from `TableConstraint` in `TableInfo` type
- Remove `tcCols` field in `TableConstraint` type
- Modify `table_info.sql` and `fetchTableMeta` function `SQL`
- A test case to perform `delete` mutation and returning relational objects.
1. Haskel library `pg-client-hs` has been updated to expose a function that helps listen to `postgres` notifications over a `channel` in this [PR](https://github.com/hasura/pg-client-hs/pull/5)
2. The server records an event in a table `hdb_catalog.hdb_cache_update_event` whenever any `/v1/query` (that changes metadata) is requested. A trigger notifies a `cache update` event via `hasura_cache_update` channel
3. The server runs two concurrent threads namely `listener` and `processor`. The `listener` thread listens to events on `hasura_cache_update` channel and pushed into a `Queue`. The `processor` thread fetches events from that `Queue` and processes it. Thus server rebuilds schema cache from database and updates.
If returning field contains nested selections then mutation is performed in two steps
1. Mutation is performed with returning columns of any primary key and unique constraints
2. returning fields are queried on rows returned by selecting from table by filtering with column values returned in Step 1.
Since mutation takes two courses based on selecting relations in returning field, it is hard to maintain sequence of prepared arguments (PrepArg) generated while resolving returning field. So, we're using txtConverter instead of prepare to resolve mutation fields.
* read cookie while initialising websocket connection (fix#1660)
* add tests for cookie on websocket init
* fix logic for tests
* enforce cors, and flag to force read cookie when cors disabled
- as browsers don't enforce SOP on websockets, we enforce CORS policy
on websocket handshake
- if CORS is disabled, by default cookie is not read (because XSS
risk!). Add special flag to force override this behaviour
* add log and forward origin header to webhook
- add log notice when cors is disabled, and cookie is not read on
websocket handshake
- forward origin header to webhook in POST mode. So that when CORS is
disabled, webhook can also enforce CORS independently.
* add docs, and forward all client headers to webhook
