Commit Graph

209 Commits

Author SHA1 Message Date
Rakesh Emmadi
204cd3514b optimise server startup time (close #1430) (#2120)
1. Reuses postgres connections during startup which reduces the overhead of opening and closing connections. 
2. Faster schema cache building. This is done by fetching all the required data in a single sql statement.
2019-05-08 13:06:42 +05:30
Shahidh K Muhammed
71cf017197 add an api to dump postgres schema (close #1939) (#1967) 2019-04-30 14:04:08 +05:30
nizar-m
cadc41b09d Use UTF-8 encoding for stack hpc report generation (#1974) 2019-04-10 14:41:50 +05:30
nizar-m
a40bf10b9f run graphql tests on both http and websocket; add parallelism (close #1868) (#1921)
Examples 
1)  `
pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv
`
2)  `pytest --hge-urls "http://127.0.0.1:8080"   "http://127.0.0.1:8081" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests"  "postgresql://admin@127.0.0.1:5432/hge_tests2" -vv
`
### Solution and Design
<!-- How is this issue solved/fixed? What is the design? -->
<!-- It's better if we elaborate -->
#### Reducing execution time of tests
- The Schema setup and teardown, which were earlier done per test method, usually takes around 1 sec. 
- For mutations, the model has now been changed to only do schema setup and teardown once per test class.
-  A data setup and teardown will be done once per test instead (usually takes ~10ms).
- For the test class to get this behaviour, one can can extend the class `DefaultTestMutations`. 
    - The function  `dir()` should be define which returns the location of the configuration folder.
    - Inside the configuration folder, there should be 
        - Files `<conf_dir>/schema_setup.yaml` and `<conf_dir>/schema_teardown.yaml`, which has the metadata query executed during schema setup and teardown respectively
        - Files named `<conf_dir>/values_setup.yaml` and `<conf_dir>/values_teardown.yaml`. These files are executed to setup and remove data from the tables respectively. 

#### Running Graphql queries on both http and websockets
- Each GraphQL query/mutation is run on the both HTTP and websocket protocols
- Pytests test parameterisation is used to achieve this
- The errors over websockets are slightly different from that on HTTP
   - The code takes care of converting the errors in HTTP to errors in websockets

#### Parallel executation of tests.
- The plugin pytest-xdist helps in running tests on parallel workers.
- We are using this plugin to group tests by file and run on different workers.
- Parallel test worker processes operate on separate postgres databases(and separate graphql-engines connected to these databases). Thus tests on one worker will not affect the tests on the other worker.
- With two workers, this decreases execution times by half, as the tests on event triggers usually takes a long time, but does not consume much CPU.
2019-04-08 12:52:38 +05:30
Shahidh K Muhammed
2274e2748b Revert "reduce server build time on ci (#1836)"
This reverts commit cbd6bb1f15.
2019-03-25 09:49:23 +05:30
Anon Ray
cbd6bb1f15 reduce server build time on ci (#1836)
Due to a misconfiguration in the caching logic, a server was building dependencies on every build. This fixes it to reduce server build time.
2019-03-22 18:11:15 +05:30
Vamshi Surabhi
c7346fd55a bump stackage to lts 13 and refer to hasura's pg-client-hs (#1747) 2019-03-14 20:25:33 +05:30
Rakesh Emmadi
e32f5a1fb1 sync metadata cache across multiple instances connected to same db (closes #1182) (#1574)
1. Haskel library `pg-client-hs` has been updated to expose a function that helps listen to `postgres` notifications over a `channel` in this [PR](https://github.com/hasura/pg-client-hs/pull/5)
2. The server records an event in a table `hdb_catalog.hdb_cache_update_event` whenever any `/v1/query` (that changes metadata) is requested. A trigger notifies a `cache update` event via `hasura_cache_update` channel
3. The server runs two concurrent threads namely `listener` and `processor`. The `listener` thread listens to events on `hasura_cache_update` channel and pushed into a `Queue`. The `processor` thread fetches events from that `Queue` and processes it. Thus server rebuilds schema cache from database and updates.
2019-03-12 11:16:27 +05:30
Shahidh K Muhammed
a83adf6503 skip ciignore check for release branches 2019-03-07 21:47:27 +05:30
Shahidh K Muhammed
a1089c1ec5
support ciignore for master builds too (#1708) 2019-03-07 17:48:06 +05:30
Shahidh K Muhammed
17183caed4
adds a check to see if the jobs should be run or not (close #1161) (#1705)
CircleCI jobs are run for any PR that is submitted to the repo. This PR adds a check to decide whether the job should be run or not.

Figured out that CircleCI has a way to gracefully terminate a job:
```
circleci-agent step halt
```

A `.ciignore` file is ran against all the changes in the PR to decide whether the PR should be built or not. If the answer comes out as `no`, a file is written at `/buid/skip_job.txt`. This is done in the `check_build_worthiness` step.

All further jobs, in the beginning, looks for this file and gracefully terminates the job if this file is present. The directory is passed down to the jobs as the workspace.

```yaml
  skip_job_on_ciignore: &skip_job_on_ciignore
    run: |
      if [ -f /build/skip_job.txt ]; then
        echo "halting job due to /build/skip_job.txt"
        circleci-agent step halt
      fi
```

ref: https://support.circleci.com/hc/en-us/articles/360015562253-Conditionally-end-a-running-job-gracefully

There are some known issues on jobs that are run when PR is merged to master, need to address them after this PR is merged.
2019-03-07 11:28:03 +05:30
Anon Ray
02d80c9ac6 read cookie while initialising websocket connection (fix #1660) (#1668)
* read cookie while initialising websocket connection (fix #1660)

* add tests for cookie on websocket init

* fix logic for tests

* enforce cors, and flag to force read cookie when cors disabled

  - as browsers don't enforce SOP on websockets, we enforce CORS policy
  on websocket handshake
  - if CORS is disabled, by default cookie is not read (because XSS
  risk!). Add special flag to force override this behaviour

* add log and forward origin header to webhook

  - add log notice when cors is disabled, and cookie is not read on
  websocket handshake
  - forward origin header to webhook in POST mode. So that when CORS is
  disabled, webhook can also enforce CORS independently.

* add docs, and forward all client headers to webhook
2019-03-04 13:16:53 +05:30
Rakesh Emmadi
377290a058 breaking: correct (de)serialisation of postgres numeric types in json (fix #1523) (#1662) 2019-03-01 17:15:04 +05:30
nizar-m
1fa66dc622 add option to disable metadata and graphql apis (close #1088) (#1650) 2019-02-28 19:23:03 +05:30
Shahidh K Muhammed
fdd5784bf7
test server upgrade from last release (close #570) (#1667) 2019-02-27 20:42:47 +05:30
nizar-m
f83a8e591f rename access-key to admin-secret (close #1347) (#1540)
Rename the admin secret key header used to access GraphQL engine from X-Hasura-Access-Key to X-Hasura-Admin-Secret.

Server CLI and console all support the older flag but marks it as deprecated.
2019-02-14 15:07:47 +05:30
Anon Ray
199a24d050 add support for multiple domains in cors config (close #1436) (#1536)
Support for multiple domains (as CSV) in the `--cors-domain` flag and `HASURA_GRAPHQL_CORS_DOMAIN` env var.

Following are all valid configurations (must include scheme and optional port):
```shell
HASURA_GRAPHQL_CORS_DOMAIN="https://*.foo.bar.com:8080"
HASURA_GRAPHQL_CORS_DOMAIN="https://*.foo.bar.com, http://*.localhost, https://example.com"
HASURA_GRAPHQL_CORS_DOMAIN="*"
HASURA_GRAPHQL_CORS_DOMAIN="http://example.com, http://*.localhost, http://localhost:3000, https://*.foo.bar.com, https://foo.bar.com"
```

**Note**: top-level domains are not considered as part of wildcard domains. You have to add them separately. E.g - `https://*.foo.com` doesn't include `https://foo.com`.

The default (if the flag or env var is not specified) is `*`. Which means CORS headers are sent for all domains.
2019-02-14 11:28:38 +05:30
Anon Ray
4f6462e98f add config for stringified hasura claims in JWT (fix #1176) (#1538) 2019-02-05 17:34:16 +05:30
nizar-m
916caf1575 add flag to disable prepared statements (close #1392) (#1396) 2019-01-18 19:50:41 +05:30
Aravind Shankar
a8dd49aa4c remove sleep for the console tests to start (close #1377) (#1378) 2019-01-15 10:49:06 +05:30
Anon Ray
ee3e099eb5 add circleci tests for pg 11 (closes #810) (#1192) 2018-12-14 11:30:36 +05:30
Shahidh K Muhammed
07181123b4
disable checking build worthiness until a better solution is found (#1162) 2018-12-04 10:33:20 +05:30
Rakesh Emmadi
3ea20bc4d7 allow authentication webhook with POST (close #1138) (#1147) 2018-12-03 16:49:08 +05:30
Tirumarai Selvan
317efb81f1 event triggers: take webhook url from env (close #966) (#968) 2018-11-14 12:43:01 +05:30
Shahidh K Muhammed
fbcae53efa
build cli-migrations image on tagged deploys (close #534) (#953) 2018-11-02 19:38:25 +05:30
Shahidh K Muhammed
1470caa1df
update digitalocean one-click app manifests on deploy (close #878) (#937) 2018-11-02 15:51:44 +05:30
nizar-m
c80a37a647 make tests work without hpc (fix #903) (#904) 2018-10-29 13:13:47 +05:30
nizar-m
0ffb0478b9 Tests for server with access control, and some more tests (#710)
* 1) Tests for creating permissions
2) Test for constraint_on with GraphQL insert on_conflict

* Run tests with access key and webhook

* Tests for GraphQL query with quoted columns

* Rewrite test-server.sh so that it can be run locally

* JWT based tests

* Tests with various postgres types

* For tests on select queries, run setup only once per class

* Tests for v1 count queries

* Skip teardown for tests that does not modify data

* Workaround for hpc 'parse error when reading .tix file'

* Move GeoJson tests to the new structure

* Basic tests for v1 queries

* Tests for column, table or operator not found error cases on GraphQL queries

* Skip test teardown for mutation tests which does not change database state, even when it returns 200.
2018-10-28 23:57:49 +05:30
nizar-m
cd030068c2 GeoJSON: Fix MultiPolygon parse error (closes #840) 2018-10-24 13:51:37 +05:30
gy741
45c347552e avoid run apt-get upgrade and dist-upgrade (#796) 2018-10-20 09:02:57 +05:30
Anon Ray
7d9851b0a9 generate coverage report for server tests (close #464) (#512) 2018-10-05 18:07:19 +05:30
nizar-m
596bccde49 add python based tests, remove haskell tests
this does not generate coverage report yet
2018-10-04 18:14:15 +05:30
Aravind Shankar
44f64a1410 changelog script to consider only core components (fix #433) (#624) 2018-10-03 18:23:23 +05:30
Aravind Shankar
f847758edf changelog script to consider only core components (close #433) (#564) 2018-09-29 11:19:24 +05:30
Aravind Shankar
e82eb52305 changelog script to consider only core components (close #433) (#529) 2018-09-26 19:52:28 +05:30
Tirumarai Selvan
2cd2b23b2d add custom headers for webhooks, refactor retry logic (#419) 2018-09-24 17:20:11 +05:30
Vamshi Surabhi
534f23d1a6 python based tests (#387) 2018-09-18 11:51:57 +05:30
Shahidh K Muhammed
5929ec5be3
add installation manifests (#432) 2018-09-13 15:03:13 +05:30
Aravind Shankar
79a6849241 add parallel tests and update dependencies (#374) 2018-09-08 23:01:14 +05:30
Aravind Shankar
ea9b187e7c update tests to use access key (close #113) (#296)
Closes #113
2018-08-30 21:24:12 +05:30
Vamshi Surabhi
dcde969d66 ignore certain headers from the request when calling the webhook (close #260) (#261) 2018-08-06 19:36:48 +05:30
Shahidh K Muhammed
0ed3ff6418
remove manifests directory from init (close #254) (#258) 2018-08-06 17:03:17 +05:30
Anon Ray
c747971f2d server tests now run across supported postgres versions >= 9.5 (fix #154) (#199) 2018-07-26 11:17:21 +05:30
Aravind Shankar
aef56ba196 configure git while updating manifests repo (#190) 2018-07-24 09:35:32 +05:30
Shahidh
0b15987db9
ciignore: always pass checks for tags 2018-07-21 01:38:16 +05:30
Shahidh K Muhammed
8aa481c639
deploy script enhancements (close #112) (#155)
[skip ci]
2018-07-18 17:57:09 +05:30
Shahidh K Muhammed
7a42e7a7fa
ciignore: always build master branch (#162)
* always pass build worthiness check for master

[skip ci]
2018-07-18 17:13:37 +05:30
Shahidh K Muhammed
3cb62622b2
update ciignore to take circleci compare url (#158) 2018-07-18 11:16:52 +05:30
Shahidh K Muhammed
92935bed40
add ciignore (#153)
Builds with changes only to files mentioned in `.ciignore` will fail on circleci at the `check_build_worthiness` step.
2018-07-18 11:00:52 +05:30
Aravind Shankar
5985bf8e9e deploy script enhancements (#124) 2018-07-17 22:30:26 +05:30
Rishichandra Wawhal
3a085da736 testing console tests in the ci (#83)
* testing console tests in the ci

* console: making cypress wait for the server to start

* console: fixing failing tests

* console: update failing test

* console: cleaned up modify tests

* console: fixed a failing test for api-explorer
2018-07-11 17:32:29 +05:30
shahidhk
00e4da5389 update deploy workflow 2018-07-11 14:07:34 +05:30
Anon Ray
4270529c11 basic test suite (#78)
* server: basic test setup

* server: use the default transaction mode

* server: basic tests in yaml files

* server: restructure test setup and some more tests
2018-07-11 12:43:07 +05:30
Praveen Durairaj
6826cec1df console: add gzip compression to deployments (#94) 2018-07-11 12:19:03 +05:30
Shahidh K Muhammed
57f9bfc478
close #92 add waits for cli and server test (#93) 2018-07-11 12:17:51 +05:30
Shahidh K Muhammed
183cbb169b only deploy dev|release branches (#76)
* remove special characters from version string
2018-07-10 11:52:13 +00:00
Shahidh K Muhammed
50165fc1ed circleci integration (#74) 2018-07-10 10:01:02 +00:00
Shahidh K Muhammed
98c827449b
build: integrate circleci
integration with circleci for various workflows
2018-07-03 22:40:13 +05:30
Shahidh K Muhammed
6170d6b094 cli: add circleci config for cli (#26) 2018-06-29 13:09:21 +00:00