Commit Graph

32 Commits

Author SHA1 Message Date
Karthikeyan Chinnakonda
72c24eea69 server: fix cron trigger bug of new events not getting generated when cron trigger is imported via metadata
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
GitOrigin-RevId: 1a9076e039ba2c8da898683c64ef988fb8512eae
2021-05-26 16:20:19 +00:00
Karthikeyan Chinnakonda
ecb4b9d098 server: graceful shutdown for event and cron triggers and async action (II)
GitOrigin-RevId: 2090c3cc34f633c691b4e4ff9ae918a60c37ce26
2021-05-14 09:39:33 +00:00
Swann Moreau
04a87970f4 server: sanitise event trigger logs (fix #1233)
- sanitise the response field in event trigger and scheduled trigger logs, removing the body and the headers
- flatten the log structure to include the event id at `$.detail.event_id` instead of `$.detail.context.event_id`

the log format changes as follows:
```diff
diff --git a/log b/log
index 96127e0..1fb909a 100644
--- a/log
+++ b/log
@@ -1,48 +1,15 @@
 {
   "detail": {
-    "context": {
-      "event_id": "b9d4e627-6029-43f2-9d46-31c532b07070"
-    },
+    "event_id": "adcc8dcd-2f32-4554-bd55-90c787aee137",
     "request": {
       "size": 416
     },
     "response": {
-      "body": "{\n  \"args\": {}, \n  \"data\": \"{\\\"event\\\":{\\\"session_variables\\\":{\\\"x-hasura-role\\\":\\\"admin\\\"},\\\"op\\\":\\\"INSERT\\\",\\\"data\\\":{\\\"old\\\":null,\\\"new\\\":{\\\"name\\\":\\\"someotheranimal\\\",\\\"id\\\":3}},\\\"trace_context\\\":{\\\"trace_id\\\":\\\"e8237c03de151634\\\",\\\"span_id\\\":\\\"8c5f8952e9e06da8\\\"}},\\\"created_at\\\":\\\"2021-05-06T07:52:58.796611Z\\\",\\\"id>
-      "headers": [
-        {
-          "name": "Date",
-          "value": "Thu, 06 May 2021 07:53:00 GMT"
-        },
-        {
-          "name": "Content-Type",
-          "value": "application/json"
-        },
-        {
-          "name": "Content-Length",
-          "value": "1692"
-        },
-        {
-          "name": "Connection",
-          "value": "keep-alive"
-        },
-        {
-          "name": "Server",
-          "value": "gunicorn/19.9.0"
-        },
-        {
-          "name": "Access-Control-Allow-Origin",
-          "value": "*"
-        },
-        {
-          "name": "Access-Control-Allow-Credentials",
-          "value": "true"
-        }
-      ],
-      "size": 1692,
+      "size": 1719,
       "status": 200
     }
   },
   "level": "info",
-  "timestamp": "2021-05-06T13:23:00.376+0530",
+  "timestamp": "2021-05-06T13:25:14.481+0530",
   "type": "event-trigger"
 }
```

GitOrigin-RevId: d9622de366737da04dc6d9ff73238be16ec9305e
2021-05-12 12:10:53 +00:00
Antoine Leblanc
2152911e24 server: introduce Hasura.Base (take 2)
GitOrigin-RevId: 0dd10f1ccd338b1cf382ebff59b6ee7f209d39a1
2021-05-11 15:19:33 +00:00
kodiakhq[bot]
a935746e17 Integration test improvements (for speed/clarity), also increase polling interval for scheduled events
This claws back ~7min from integration tests (run serially, as with `dev.sh test --integration`
Further improvements would do well to focus on optimizing metadata operations, as `setup` dominates

GitOrigin-RevId: 76637d6fa953c2404627c4391447a05bf09355fa
2021-04-27 05:35:26 +00:00
Phil Freeman
3a3da164d7 [server] fix invocation log queries leaking data across tenants
GitOrigin-RevId: ceb568bc3fe05934db28fa7e3f897815bf7af414
2021-04-14 01:24:45 +00:00
Karthikeyan Chinnakonda
3e880be55f server: efficient polling for cron triggers and async actions
GitOrigin-RevId: 210df9e6e660b90fb74e930a8909129edba4e7a3
2021-03-30 12:58:40 +00:00
Anon Ray
cad013c039 multitenant: fix cloud schema files
GitOrigin-RevId: 99f04df3654ab83f64489960b5f7ecf5f86c9a99
2021-02-22 16:03:07 +00:00
Karthikeyan Chinnakonda
7be72d7bee server: support for maintenance mode in v1.4
Co-authored-by: Anon Ray <616387+ecthiender@users.noreply.github.com>
GitOrigin-RevId: 85f636b1bc8063689845da90e85cc480e87dca7e
2021-02-18 16:47:22 +00:00
Rakesh Emmadi
9ef603360c server: generalize schema cache building (#496)
Co-authored-by: Vamshi Surabhi <vamshi@hasura.io>
Co-authored-by: Vladimir Ciobanu <admin@cvlad.info>
Co-authored-by: Antoine Leblanc <antoine@hasura.io>
Co-authored-by: Stylish Haskell Bot <stylish-haskell@users.noreply.github.com>
GitOrigin-RevId: 9d631878037637f3ed2994b5d0525efd978f7b8f
2021-02-14 06:08:46 +00:00
Anon Ray
06b599b747 server: multitenant metadata storage
The metadata storage implementation for graphql-engine-multitenant.

- It uses a centralized PG database to store metadata of all tenants (instead of per tenant database)
- Similarly, it uses a single schema-sync listener thread per MT worker (instead of listener thread per tenant) (PS: although, the processor thread is spawned per tenant)
- 2 new flags are introduced - `--metadataDatabaseUrl` and (optional) `--metadataDatabaseRetries`

Internally, a "metadata mode" is introduced to indicate an external/managed store vs a store managed by each pro-server.

To run :
- obtain the schema file (located at `pro/server/res/cloud/metadata_db_schema.sql`)
- apply the schema on a PG database
- set the `--metadataDatabaseUrl` flag to point to the above database
- run the MT executable

The schema (and its migrations) for the metadata db is managed outside the MT worker.

### New metadata

The following is the new portion of `Metadata` added :

```yaml
version: 3
metrics_config:
  analyze_query_variables: true
  analyze_response_body: false
api_limits:
  disabled: false
  depth_limit:
    global: 5
    per_role:
      user: 7
      editor: 9
  rate_limit:
    per_role:
      user:
        unique_params:
        - x-hasura-user-id
        - x-hasura-team-id
        max_reqs_per_min: 20
    global:
      unique_params: IP
      max_reqs_per_min: 10
```

- In Pro, the code around fetching/updating/syncing pro-config is removed
- That also means, `hdb_pro_catalog` for keeping the config cache is not required. Hence the `hdb_pro_catalog` is also removed
- The required config comes from metadata / schema cache

### New Metadata APIs

- `set_api_limits`
- `remove_api_limits`
- `set_metrics_config`
- `remove_metrics_config`

#### `set_api_limits`

```yaml
type: set_api_limits
args:
  disabled: false
  depth_limit:
    global: 5
    per_role:
      user: 7
      editor: 9
  rate_limit:
    per_role:
      anonymous:
         max_reqs_per_min: 10
         unique_params: "ip"
      editor:
        max_reqs_per_min: 30
        unique_params:
        - x-hasura-user-id
      user:
        unique_params:
        - x-hasura-user-id
        - x-hasura-team-id
        max_reqs_per_min: 20
    global:
      unique_params: IP
      max_reqs_per_min: 10
```

#### `remove_api_limits`

```yaml
type: remove_api_limits
args: {}
```

#### `set_metrics_config`

```yaml
type: set_metrics_config
args:
  analyze_query_variables: true
  analyze_response_body: false
```

#### `remove_metrics_config`

```yaml
type: remove_metrics_config
args: {}
```

#### TODO
- [x] on-prem pro implementation for `MonadMetadataStorage`
- [x] move the project config from Lux to pro metadata (PR: #379)
- [ ] console changes for pro config/api limits, subscription workers (cc @soorajshankar @beerose)
- [x] address other minor TODOs
  - [x] TxIso for `MonadSourceResolver`
  - [x] enable EKG connection pool metrics
  - [x] add logging of connection info when sources are added?
  - [x] confirm if the `buildReason` for schema cache is correct
- [ ] testing
- [x] 1.3 -> 1.4 cloud migration script (#465; PR: #508)
  - [x] one-time migration of existing metadata from users' db to centralized PG
  - [x] one-time migration of pro project config + api limits + regression tests from metrics API  to metadata
- [ ] integrate with infra team (WIP - cc @hgiasac)
  - [x] benchmark with 1000+ tenants + each tenant making read/update metadata query every second (PR: https://github.com/hasura/graphql-engine-mono/pull/411)
  - [ ] benchmark with few tenants having large metadata (100+ tables etc.)
  - [ ] when user moves regions (https://github.com/hasura/lux/issues/1717)
    - [ ] metadata has to be migrated from one regional PG to another
    - [ ] migrate metrics data as well ?
      - [ ] operation logs
      - [ ] regression test runs

- [ ] find a way to share the schema files with the infra team

Co-authored-by: Naveen Naidu <30195193+Naveenaidu@users.noreply.github.com>
GitOrigin-RevId: 39e8361f2c0e96e0f9e8f8fb45e6cc14857f31f1
2021-02-11 17:55:21 +00:00
hasura-bot
2c56254e5a server: simplify JSON instances
GITHUB_PR_NUMBER: 6152
GITHUB_PR_URL: https://github.com/hasura/graphql-engine/pull/6152

Co-authored-by: Antoine Leblanc <1618949+nicuveo@users.noreply.github.com>
GitOrigin-RevId: 6c94aef8c57e852b3d41b8355c09e64fce756a7c
2021-01-19 19:15:42 +00:00
Rakesh Emmadi
be62641f68 server: multi source metadata APIs (#217)
Co-authored-by: Aleksandra Sikora <aleksandra@hasura.io>
Co-authored-by: Anon Ray <ecthiender@users.noreply.github.com>
Co-authored-by: Vishnu Bharathi <vishnubharathi04@gmail.com>
Co-authored-by: Aleksandra Sikora <aleksandra@hasura.io>
Co-authored-by: Sameer Kolhar <kolhar730@gmail.com>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
GitOrigin-RevId: 0dd1e4d58ab81f1b4ce24de2d3eab709c2755e6d
2021-01-07 09:05:19 +00:00
Rakesh Emmadi
a153e96309 server: expand metadata storage class with async actions and core metadata operations (#184)
An incremental PR towards https://github.com/hasura/graphql-engine/pull/5797
- Expands `MonadMetadataStorage` with operations related to async actions and setting/updating metadata

GitOrigin-RevId: 53386b7b2d007e162050b826d0708897f0b4c8f6
2020-12-14 04:31:20 +00:00
hasura-bot
29925eb08d server: metadata storage abstraction for scheduled triggers
An incremental PR towards https://github.com/hasura/graphql-engine/pull/5797

* metadata storage abstraction for scheduled triggers

Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Auke Booij <auke@hasura.io>
GITHUB_PR_NUMBER: 6131
GITHUB_PR_URL: https://github.com/hasura/graphql-engine/pull/6131

* update pro server code

Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Auke Booij <auke@hasura.io>
GitOrigin-RevId: 17244a47b3e8633acf2492e0b0734b72025f0a09
2020-11-25 10:57:38 +00:00
Auke Booij
0540b279db
server: make more use of hlint (#6059)
https://github.com/hasura/graphql-engine/pull/6059
2020-10-28 16:40:33 +00:00
Antoine Leblanc
a8ed6a82e2
server: move Hasura.SQL to Hasura.Backends.Postgres (#6053)
https://github.com/hasura/graphql-engine/pull/6053
2020-10-27 13:53:49 +00:00
Antoine Leblanc
f46f807942 clean SQL text functions 2020-10-22 14:07:47 +01:00
Sameer Kolhar
af32ccff36
server: limit length of event trigger names (close #5786) (#5826)
https://github.com/hasura/graphql-engine/pull/5826
2020-10-06 15:22:09 +00:00
Karthikeyan Chinnakonda
43a6290124
server: lower the isolation level to ReadCommited while inserting/updating invocation logs and events (#5824)
https://github.com/hasura/graphql-engine/pull/5824
2020-09-24 06:46:24 +00:00
Karthikeyan Chinnakonda
6eb7a7dd8a Merge branch 'master' into scheduled-triggers-created-at-bug-5272 2020-09-16 03:33:41 +05:30
Karthikeyan Chinnakonda
15c3bd9799 include created_at field in CronEventPartial 2020-09-09 12:17:34 +05:30
Karthikeyan Chinnakonda
3e10eb6694 don't send created_at in the payload in case of cron triggers 2020-09-08 12:18:00 +05:30
Karthikeyan Chinnakonda
ee6797a8d8 refactor StandAloneEvent to OneOffEvents 2020-09-07 11:45:15 +05:30
Karthikeyan Chinnakonda
772e991079 use the created_at time of the scheduled event
- in case of cron events, use the created_at of the cron trigger
2020-09-04 22:44:54 +05:30
Phil Freeman
9e792f2e61 server: Use event trigger name as tracing span name 2020-08-31 15:05:04 -07:00
Anon Ray
434c78267c
server: log request/response sizes for event triggers (#5463)
* server: log request/response sizes for event triggers

  event triggers (and scheduled triggers) now have request/response size
  in their logs.

* add changelog entry
2020-07-28 10:52:44 -07:00
Phil Freeman
0dddbe9e9d
Add MonadTrace and MonadExecuteQuery abstractions (#5383)
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-07-15 16:10:48 +05:30
Lyndon Maydwell
24592a516b
Pass environment variables around as a data structure, via @sordina (#5374)
* Pass environment variables around as a data structure, via @sordina

* Resolving build error

* Adding Environment passing note to changelog

* Removing references to ILTPollerLog as this seems to have been reintroduced from a bad merge

* removing commented-out imports

* Language pragmas already set by project

* Linking async thread

* Apply suggestions from code review

Use `runQueryTx` instead of `runLazyTx` for queries.

* remove the non-user facing entry in the changelog

Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-07-14 12:00:58 -07:00
Karthikeyan Chinnakonda
9ef6de5113
server: include additional fields in scheduled trigger webhook payload (#5262)
* include scheduled triggers metadata in the webhook body

Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
2020-07-03 06:25:07 +05:30
Karthikeyan Chinnakonda
97b1155bf8
server: unlock scheduled events on graceful shutdown (#4928) 2020-07-02 17:27:09 +05:30
Tirumarai Selvan
cc8e2ccc78
Scheduled triggers (close #1914) (#3553)
server: add scheduled triggers 

Co-authored-by: Alexis King <lexi.lambda@gmail.com>
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>
2020-05-13 18:03:16 +05:30