Commit Graph

271 Commits

Author SHA1 Message Date
Rakesh Emmadi
6d92e4f9db save permissions, relationships and collections in catalog with 'is_system_defined' explicitly (#3165)
* save permissions, relationships and collections in catalog with 'is_system_defined'
* Use common stanzas in the .cabal file
* Refactor migration code into lib instead of exe
* Add new server test suite that exercises migrations
* Make graphql-engine clean succeed even if the schema does not exist
2019-10-21 11:01:05 -05:00
Alexis King
e01008413e
Track variable uses within query validation for caching (fix #3097) (#3135)
This fix is a little ugly, but it’s the only simple solution without a
significant refactoring that restructures the relationship between
GraphQL/Validate and GraphQL/Resolve. The ugliness should go away if we
implement something like #2801.
2019-10-16 09:33:34 -05:00
nizar-m
44da458c81 fix hpc combine error (close #2946) (#2947)
* Fix hpc combine error

* Do not perform ciignore

* xfail test jsonb_has_all

* Bring back ciignore

* Refer jsonb_has_all xfaul to the corresponding issue in graphql-engine-internal
2019-10-02 12:06:27 +05:30
Rakesh Emmadi
8a0615ff70 add gzip brotli compression to http responses (close #2674) (#2751) 2019-09-19 18:24:40 +05:30
Alexis King
ed26da59a6 Add support for GraphQL enum types via enum table references
These changes also add a new type, PGColumnType, between PGColInfo and
PGScalarType, and they process PGRawColumnType values into PGColumnType
values during schema cache generation.
2019-08-26 00:54:56 -05:00
José Lorenzo Rodríguez
c7a2320456 Implemented graceful shutdown for HTTP requests (close #2698) (#2717)
* Listens for SIGTERM as the termination signal
* Stops accepting new connections once the signal is received
* Waits for all connections to be drained, before shutting down
* Forcefully kills all pending connections after 30 seconds

Currently this does not send a close message to websocket clients, I'd
like to submit that change as a separate pull request, but at least this
solve my biggest concern which is not getting confirmation for mutations
while restarting the server.
2019-08-26 00:31:27 -05:00
Shahidh K Muhammed
3042fd175e
build latest.cli-migrations image on release (close #2591) (#2707) 2019-08-10 12:39:27 +00:00
Vamshi Surabhi
66a2f33fa9 use all available cpu cores by default (#2511)
Use all available cores by default. sets -N rts flag by default. Can be disabled by
setting the `GHCRTS` env variable to `-N1`.
2019-07-12 05:25:15 +00:00
Anon Ray
f2f14e727b Merge pull request from GHSA-2j98-fw5g-j43v
* fix bug in audience check while verifying JWT

  - previously the check was converting the audience type into a string
  and then comparing with the conf value. all audience types (as it is a
  string or URI) will convert to plain strings
  - use the Audience type from the jose library for comparing

* add docs for audience

* add issuer check as well

* docs minor syntax fix

* skip audience check if not given in conf

* minor docs update

* qualify import jose library
2019-07-11 09:58:39 +00:00
Anon Ray
8f1e7dbc8d breaking: server logging changes (close #507, close #2171) (#1835) 2019-07-11 05:37:06 +00:00
Shahidh K Muhammed
76ceb707f4
bundle console assets into server (close #516, close #521, close #2130) (#2192)
This PR builds console static assets into the server docker image at `/srv/console-assets`. When env var `HASURA_GRAPHQL_CONSOLE_ASSETS_DIR=/srv/console-assets` or flag `--console-assets-dir=/srv/console-assets` is set on the server, the files in this directory are served at `/console/assets/*`.

The console html template will have a variable called `cdnAssets: false` when this flag is set and it loads assets from server itself instead of CDN.

The assets are moved to a new bucket with a new naming scheme:

```
graphql-engine-cdn.hasura.io/console/assets/
   /common/{}
   /versioned/<version/{}
   /channel/<channel>/<version>/{}
```

Console served by CLI will still load assets from CDN - will fix that in the next release.
2019-05-16 13:15:29 +05:30
Rakesh Emmadi
0b210cc245 support allow-list for graphql queries (closes #989) (#2075) 2019-05-16 11:43:25 +05:30
Rakesh Emmadi
204cd3514b optimise server startup time (close #1430) (#2120)
1. Reuses postgres connections during startup which reduces the overhead of opening and closing connections. 
2. Faster schema cache building. This is done by fetching all the required data in a single sql statement.
2019-05-08 13:06:42 +05:30
Shahidh K Muhammed
71cf017197 add an api to dump postgres schema (close #1939) (#1967) 2019-04-30 14:04:08 +05:30
nizar-m
cadc41b09d Use UTF-8 encoding for stack hpc report generation (#1974) 2019-04-10 14:41:50 +05:30
nizar-m
a40bf10b9f run graphql tests on both http and websocket; add parallelism (close #1868) (#1921)
Examples 
1)  `
pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv
`
2)  `pytest --hge-urls "http://127.0.0.1:8080"   "http://127.0.0.1:8081" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests"  "postgresql://admin@127.0.0.1:5432/hge_tests2" -vv
`
### Solution and Design
<!-- How is this issue solved/fixed? What is the design? -->
<!-- It's better if we elaborate -->
#### Reducing execution time of tests
- The Schema setup and teardown, which were earlier done per test method, usually takes around 1 sec. 
- For mutations, the model has now been changed to only do schema setup and teardown once per test class.
-  A data setup and teardown will be done once per test instead (usually takes ~10ms).
- For the test class to get this behaviour, one can can extend the class `DefaultTestMutations`. 
    - The function  `dir()` should be define which returns the location of the configuration folder.
    - Inside the configuration folder, there should be 
        - Files `<conf_dir>/schema_setup.yaml` and `<conf_dir>/schema_teardown.yaml`, which has the metadata query executed during schema setup and teardown respectively
        - Files named `<conf_dir>/values_setup.yaml` and `<conf_dir>/values_teardown.yaml`. These files are executed to setup and remove data from the tables respectively. 

#### Running Graphql queries on both http and websockets
- Each GraphQL query/mutation is run on the both HTTP and websocket protocols
- Pytests test parameterisation is used to achieve this
- The errors over websockets are slightly different from that on HTTP
   - The code takes care of converting the errors in HTTP to errors in websockets

#### Parallel executation of tests.
- The plugin pytest-xdist helps in running tests on parallel workers.
- We are using this plugin to group tests by file and run on different workers.
- Parallel test worker processes operate on separate postgres databases(and separate graphql-engines connected to these databases). Thus tests on one worker will not affect the tests on the other worker.
- With two workers, this decreases execution times by half, as the tests on event triggers usually takes a long time, but does not consume much CPU.
2019-04-08 12:52:38 +05:30
Shahidh K Muhammed
2274e2748b Revert "reduce server build time on ci (#1836)"
This reverts commit cbd6bb1f15.
2019-03-25 09:49:23 +05:30
Anon Ray
cbd6bb1f15 reduce server build time on ci (#1836)
Due to a misconfiguration in the caching logic, a server was building dependencies on every build. This fixes it to reduce server build time.
2019-03-22 18:11:15 +05:30
Vamshi Surabhi
c7346fd55a bump stackage to lts 13 and refer to hasura's pg-client-hs (#1747) 2019-03-14 20:25:33 +05:30
Rakesh Emmadi
e32f5a1fb1 sync metadata cache across multiple instances connected to same db (closes #1182) (#1574)
1. Haskel library `pg-client-hs` has been updated to expose a function that helps listen to `postgres` notifications over a `channel` in this [PR](https://github.com/hasura/pg-client-hs/pull/5)
2. The server records an event in a table `hdb_catalog.hdb_cache_update_event` whenever any `/v1/query` (that changes metadata) is requested. A trigger notifies a `cache update` event via `hasura_cache_update` channel
3. The server runs two concurrent threads namely `listener` and `processor`. The `listener` thread listens to events on `hasura_cache_update` channel and pushed into a `Queue`. The `processor` thread fetches events from that `Queue` and processes it. Thus server rebuilds schema cache from database and updates.
2019-03-12 11:16:27 +05:30
Shahidh K Muhammed
a83adf6503 skip ciignore check for release branches 2019-03-07 21:47:27 +05:30
Shahidh K Muhammed
a1089c1ec5
support ciignore for master builds too (#1708) 2019-03-07 17:48:06 +05:30
Shahidh K Muhammed
17183caed4
adds a check to see if the jobs should be run or not (close #1161) (#1705)
CircleCI jobs are run for any PR that is submitted to the repo. This PR adds a check to decide whether the job should be run or not.

Figured out that CircleCI has a way to gracefully terminate a job:
```
circleci-agent step halt
```

A `.ciignore` file is ran against all the changes in the PR to decide whether the PR should be built or not. If the answer comes out as `no`, a file is written at `/buid/skip_job.txt`. This is done in the `check_build_worthiness` step.

All further jobs, in the beginning, looks for this file and gracefully terminates the job if this file is present. The directory is passed down to the jobs as the workspace.

```yaml
  skip_job_on_ciignore: &skip_job_on_ciignore
    run: |
      if [ -f /build/skip_job.txt ]; then
        echo "halting job due to /build/skip_job.txt"
        circleci-agent step halt
      fi
```

ref: https://support.circleci.com/hc/en-us/articles/360015562253-Conditionally-end-a-running-job-gracefully

There are some known issues on jobs that are run when PR is merged to master, need to address them after this PR is merged.
2019-03-07 11:28:03 +05:30
Anon Ray
02d80c9ac6 read cookie while initialising websocket connection (fix #1660) (#1668)
* read cookie while initialising websocket connection (fix #1660)

* add tests for cookie on websocket init

* fix logic for tests

* enforce cors, and flag to force read cookie when cors disabled

  - as browsers don't enforce SOP on websockets, we enforce CORS policy
  on websocket handshake
  - if CORS is disabled, by default cookie is not read (because XSS
  risk!). Add special flag to force override this behaviour

* add log and forward origin header to webhook

  - add log notice when cors is disabled, and cookie is not read on
  websocket handshake
  - forward origin header to webhook in POST mode. So that when CORS is
  disabled, webhook can also enforce CORS independently.

* add docs, and forward all client headers to webhook
2019-03-04 13:16:53 +05:30
Rakesh Emmadi
377290a058 breaking: correct (de)serialisation of postgres numeric types in json (fix #1523) (#1662) 2019-03-01 17:15:04 +05:30
nizar-m
1fa66dc622 add option to disable metadata and graphql apis (close #1088) (#1650) 2019-02-28 19:23:03 +05:30
Shahidh K Muhammed
fdd5784bf7
test server upgrade from last release (close #570) (#1667) 2019-02-27 20:42:47 +05:30
nizar-m
f83a8e591f rename access-key to admin-secret (close #1347) (#1540)
Rename the admin secret key header used to access GraphQL engine from X-Hasura-Access-Key to X-Hasura-Admin-Secret.

Server CLI and console all support the older flag but marks it as deprecated.
2019-02-14 15:07:47 +05:30
Anon Ray
199a24d050 add support for multiple domains in cors config (close #1436) (#1536)
Support for multiple domains (as CSV) in the `--cors-domain` flag and `HASURA_GRAPHQL_CORS_DOMAIN` env var.

Following are all valid configurations (must include scheme and optional port):
```shell
HASURA_GRAPHQL_CORS_DOMAIN="https://*.foo.bar.com:8080"
HASURA_GRAPHQL_CORS_DOMAIN="https://*.foo.bar.com, http://*.localhost, https://example.com"
HASURA_GRAPHQL_CORS_DOMAIN="*"
HASURA_GRAPHQL_CORS_DOMAIN="http://example.com, http://*.localhost, http://localhost:3000, https://*.foo.bar.com, https://foo.bar.com"
```

**Note**: top-level domains are not considered as part of wildcard domains. You have to add them separately. E.g - `https://*.foo.com` doesn't include `https://foo.com`.

The default (if the flag or env var is not specified) is `*`. Which means CORS headers are sent for all domains.
2019-02-14 11:28:38 +05:30
Anon Ray
4f6462e98f add config for stringified hasura claims in JWT (fix #1176) (#1538) 2019-02-05 17:34:16 +05:30
nizar-m
916caf1575 add flag to disable prepared statements (close #1392) (#1396) 2019-01-18 19:50:41 +05:30
Aravind Shankar
a8dd49aa4c remove sleep for the console tests to start (close #1377) (#1378) 2019-01-15 10:49:06 +05:30
Anon Ray
ee3e099eb5 add circleci tests for pg 11 (closes #810) (#1192) 2018-12-14 11:30:36 +05:30
Shahidh K Muhammed
07181123b4
disable checking build worthiness until a better solution is found (#1162) 2018-12-04 10:33:20 +05:30
Rakesh Emmadi
3ea20bc4d7 allow authentication webhook with POST (close #1138) (#1147) 2018-12-03 16:49:08 +05:30
Tirumarai Selvan
317efb81f1 event triggers: take webhook url from env (close #966) (#968) 2018-11-14 12:43:01 +05:30
Shahidh K Muhammed
fbcae53efa
build cli-migrations image on tagged deploys (close #534) (#953) 2018-11-02 19:38:25 +05:30
Shahidh K Muhammed
1470caa1df
update digitalocean one-click app manifests on deploy (close #878) (#937) 2018-11-02 15:51:44 +05:30
nizar-m
c80a37a647 make tests work without hpc (fix #903) (#904) 2018-10-29 13:13:47 +05:30
nizar-m
0ffb0478b9 Tests for server with access control, and some more tests (#710)
* 1) Tests for creating permissions
2) Test for constraint_on with GraphQL insert on_conflict

* Run tests with access key and webhook

* Tests for GraphQL query with quoted columns

* Rewrite test-server.sh so that it can be run locally

* JWT based tests

* Tests with various postgres types

* For tests on select queries, run setup only once per class

* Tests for v1 count queries

* Skip teardown for tests that does not modify data

* Workaround for hpc 'parse error when reading .tix file'

* Move GeoJson tests to the new structure

* Basic tests for v1 queries

* Tests for column, table or operator not found error cases on GraphQL queries

* Skip test teardown for mutation tests which does not change database state, even when it returns 200.
2018-10-28 23:57:49 +05:30
nizar-m
cd030068c2 GeoJSON: Fix MultiPolygon parse error (closes #840) 2018-10-24 13:51:37 +05:30
gy741
45c347552e avoid run apt-get upgrade and dist-upgrade (#796) 2018-10-20 09:02:57 +05:30
Anon Ray
7d9851b0a9 generate coverage report for server tests (close #464) (#512) 2018-10-05 18:07:19 +05:30
nizar-m
596bccde49 add python based tests, remove haskell tests
this does not generate coverage report yet
2018-10-04 18:14:15 +05:30
Aravind Shankar
44f64a1410 changelog script to consider only core components (fix #433) (#624) 2018-10-03 18:23:23 +05:30
Aravind Shankar
f847758edf changelog script to consider only core components (close #433) (#564) 2018-09-29 11:19:24 +05:30
Aravind Shankar
e82eb52305 changelog script to consider only core components (close #433) (#529) 2018-09-26 19:52:28 +05:30
Tirumarai Selvan
2cd2b23b2d add custom headers for webhooks, refactor retry logic (#419) 2018-09-24 17:20:11 +05:30
Vamshi Surabhi
534f23d1a6 python based tests (#387) 2018-09-18 11:51:57 +05:30
Shahidh K Muhammed
5929ec5be3
add installation manifests (#432) 2018-09-13 15:03:13 +05:30
Aravind Shankar
79a6849241 add parallel tests and update dependencies (#374) 2018-09-08 23:01:14 +05:30
Aravind Shankar
ea9b187e7c update tests to use access key (close #113) (#296)
Closes #113
2018-08-30 21:24:12 +05:30
Vamshi Surabhi
dcde969d66 ignore certain headers from the request when calling the webhook (close #260) (#261) 2018-08-06 19:36:48 +05:30
Shahidh K Muhammed
0ed3ff6418
remove manifests directory from init (close #254) (#258) 2018-08-06 17:03:17 +05:30
Anon Ray
c747971f2d server tests now run across supported postgres versions >= 9.5 (fix #154) (#199) 2018-07-26 11:17:21 +05:30
Aravind Shankar
aef56ba196 configure git while updating manifests repo (#190) 2018-07-24 09:35:32 +05:30
Shahidh
0b15987db9
ciignore: always pass checks for tags 2018-07-21 01:38:16 +05:30
Shahidh K Muhammed
8aa481c639
deploy script enhancements (close #112) (#155)
[skip ci]
2018-07-18 17:57:09 +05:30
Shahidh K Muhammed
7a42e7a7fa
ciignore: always build master branch (#162)
* always pass build worthiness check for master

[skip ci]
2018-07-18 17:13:37 +05:30
Shahidh K Muhammed
3cb62622b2
update ciignore to take circleci compare url (#158) 2018-07-18 11:16:52 +05:30
Shahidh K Muhammed
92935bed40
add ciignore (#153)
Builds with changes only to files mentioned in `.ciignore` will fail on circleci at the `check_build_worthiness` step.
2018-07-18 11:00:52 +05:30
Aravind Shankar
5985bf8e9e deploy script enhancements (#124) 2018-07-17 22:30:26 +05:30
Rishichandra Wawhal
3a085da736 testing console tests in the ci (#83)
* testing console tests in the ci

* console: making cypress wait for the server to start

* console: fixing failing tests

* console: update failing test

* console: cleaned up modify tests

* console: fixed a failing test for api-explorer
2018-07-11 17:32:29 +05:30
shahidhk
00e4da5389 update deploy workflow 2018-07-11 14:07:34 +05:30
Anon Ray
4270529c11 basic test suite (#78)
* server: basic test setup

* server: use the default transaction mode

* server: basic tests in yaml files

* server: restructure test setup and some more tests
2018-07-11 12:43:07 +05:30
Praveen Durairaj
6826cec1df console: add gzip compression to deployments (#94) 2018-07-11 12:19:03 +05:30
Shahidh K Muhammed
57f9bfc478
close #92 add waits for cli and server test (#93) 2018-07-11 12:17:51 +05:30
Shahidh K Muhammed
183cbb169b only deploy dev|release branches (#76)
* remove special characters from version string
2018-07-10 11:52:13 +00:00
Shahidh K Muhammed
50165fc1ed circleci integration (#74) 2018-07-10 10:01:02 +00:00
Shahidh K Muhammed
98c827449b
build: integrate circleci
integration with circleci for various workflows
2018-07-03 22:40:13 +05:30
Shahidh K Muhammed
6170d6b094 cli: add circleci config for cli (#26) 2018-06-29 13:09:21 +00:00