Commit Graph

84 Commits

Author SHA1 Message Date
paritosh-08
00005020a6 server: rename event trigger metric hasura_event_processing_time_seconds to hasura_event_webhook_processing_time_seconds
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6950
GitOrigin-RevId: a32ce3f167eb6bb5c48645a614288c043c252d26
2022-11-21 07:49:28 +00:00
paritosh-08
5be4d37240 server,pro: add metrics related to event trigger runtime
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6825
GitOrigin-RevId: 913e1bd21912db781d69f579aee943faf6ce3a11
2022-11-16 17:12:48 +00:00
Samir Talwar
342391f39d Upgrade Ormolu to v0.5.
This upgrades the version of Ormolu required by the HGE repository to v0.5.0.1, and reformats all code accordingly.

Ormolu v0.5 reformats code that uses infix operators. This is mostly useful, adding newlines and indentation to make it clear which operators are applied first, but in some cases, it's unpleasant. To make this easier on the eyes, I had to do the following:

* Add a few fixity declarations (search for `infix`)
* Add parentheses to make precedence clear, allowing Ormolu to keep everything on one line
* Rename `relevantEq` to `(==~)` in #6651 and set it to `infix 4`
* Add a few _.ormolu_ files (thanks to @hallettj for helping me get started), mostly for Autodocodec operators that don't have explicit fixity declarations

In general, I think these changes are quite reasonable. They mostly affect indentation.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6675
GitOrigin-RevId: cd47d87f1d089fb0bc9dcbbe7798dbceedcd7d83
2022-11-02 20:55:13 +00:00
Tom Harding
e71496efa5 Replace Hasura.RQL.Types.Numeric with refined
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5913
GitOrigin-RevId: 96e218229a08dfbc5a598d709be1ee2083d41ec6
2022-09-21 18:03:04 +00:00
paritosh-08
ab71adc3a0 server, pro: event trigger auto cleanup (increment 3)
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5823
Co-authored-by: pranshi06 <85474619+pranshi06@users.noreply.github.com>
Co-authored-by: Puru Gupta <32328846+purugupta99@users.noreply.github.com>
Co-authored-by: Karthikeyan Chinnakonda <15602904+codingkarthik@users.noreply.github.com>
GitOrigin-RevId: 3d4af905d5df4c05107ad46ef29f4c0a567ff754
2022-09-15 11:46:22 +00:00
Solomon
418d0e2cc2 Use NonNegative types for arg/env parsing
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5305
GitOrigin-RevId: 8274ca8c4217d15939fa2a78664a2be486426bf7
2022-08-17 01:09:19 +00:00
awjchen
1a4e3ae001 server: add event trigger metrics for Prometheus
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5346
GitOrigin-RevId: 0db925f85e06e99b636995887c0e19a49fe9be2d
2022-08-15 05:34:04 +00:00
Solomon
3855efa9a5 Move soEventsHttpPoolSize and soEventsFetchInterval defaulting into arg parser
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5284
GitOrigin-RevId: f2302b3219f3aa9ebde623b4efa7f3ee8a97c031
2022-08-10 19:18:11 +00:00
Solomon
07e4a49e5e Consolidate Arg/Env Parser Defaults and cleanup arg merging code
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5262
GitOrigin-RevId: 2501e44a8fe0a57b13141c307d2dd44b028052b6
2022-08-05 00:29:52 +00:00
Samir Talwar
40617719ef server: Remove the Show instance from QErr and anything that touches it.
We only use these `Show` instances in error messages (where we call
`show` explicitly anyway) and test cases (in which Hspec requires `Show
a` for any `a` in an assertion).

This removes the instance in favor of a custom `showQErr` function
(which serializes the error to JSON). It is then used in certain error
message production which previously called `show` on a `QErr`.

There are two places where we serialize a QErr and then construct a new
QErr from the resulting string. Instead, we modify the existing QErr to
add extra information.

An orphan `Show QErr` instance is retained for tests so that we can have
nice test failure messages.

This is preparation for future changes in which the error message within
`QErr` will not be exposed directly, and therefore will not have a
`Show` instance. That said, it feels like a sensible kind of cleanup
anyway.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4897
GitOrigin-RevId: 8f79f7a356f0aea571156f39aefac242bf751f3a
2022-07-01 11:48:26 +00:00
Daniel Harvey
03c93ad96f server: Refactor TByteString to SerializableBlob
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4736
GitOrigin-RevId: 446832b00c24205c2237666ac9d954277c5fd05d
2022-06-17 09:57:38 +00:00
Puru Gupta
ebc8cd5134 server: do not serialize env vars in logs or errors: PR III - Event Triggers
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4521
Co-authored-by: Naveen Naidu <30195193+Naveenaidu@users.noreply.github.com>
Co-authored-by: Brandon Simmons <210815+jberryman@users.noreply.github.com>
GitOrigin-RevId: c7702e8f3da466dc108936328de30ceb87df258a
2022-06-05 20:28:13 +00:00
Naveen Naidu
222419527f server: refactor MaintenanceMode datatype
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4298
GitOrigin-RevId: 0b88d040bd4b24c61c4ae91e18dc6edf67b3672e
2022-04-28 20:56:11 +00:00
Antoine Leblanc
3cbcbd9291 Remove RQL/Types.hs
## Description

This PR removes `RQL.Types`, which was now only re-exporting a bunch of unrelated modules.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4363
GitOrigin-RevId: 894f29a19bff70b3dad8abc5d9858434d5065417
2022-04-27 13:58:47 +00:00
jkachmar
647231b685 Yeet some default-extensions
Manually enables:
* EmptyCase
* ExistentialQuantification
* QuantifiedConstraints
* QuasiQuotes
* TemplateHaskell
* TypeFamilyDependencies

...in the following components:
* 'graphql-engine' library
* 'graphql-engine' 'src-test'
* 'graphql-engine' 'tests/integration'
* 'graphql-engine' tests-hspec'

Additionally, performs some light refactoring and documentation.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3991
GitOrigin-RevId: 514477d3466b01f60eca8935d0fef60dd0756838
2022-03-16 00:40:17 +00:00
Karthikeyan Chinnakonda
a2da867fc8 Fetch only events of the ETs which exist in the metadata
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3857
GitOrigin-RevId: 1ee501116103e2be49d4538c80f87c11c066af8e
2022-03-08 09:06:32 +00:00
Solomon
d67d4e2310 Webhook Transform Cleanup / Refactor
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3624
GitOrigin-RevId: 849e6dd70d6fe3d84056a485b20928ff813881d4
2022-03-08 00:43:08 +00:00
Solomon
4b792abdcc Feature/webhook response transforms
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3074
Co-authored-by: David Overton <7734777+dmoverton@users.noreply.github.com>
Co-authored-by: Lyndon Maydwell <92299+sordina@users.noreply.github.com>
Co-authored-by: Tirumarai Selvan <8663570+tirumaraiselvan@users.noreply.github.com>
GitOrigin-RevId: 006c5c9b71cdca1c2f47962230e6189e09557fab
2022-01-19 04:47:36 +00:00
Solomon
0faa92337d Use transformed request body in event invocation logs
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3057
GitOrigin-RevId: 7e001830f97c81289553e37c35098ccaeadbdf22
2021-12-06 22:40:31 +00:00
David Overton
aac64f2c81 Source typename customization (close graphql-engine#6974)
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/1616
GitOrigin-RevId: f7eefd2367929209aa77895ea585e96a99a78d47
2021-10-29 14:43:14 +00:00
Robert
71af68e9e5 server: drop HasVersion implicit parameter (closes #2236)
The only real use was for the dubious multitenant option
--consoleAssetsVersion, which actually overrode not just
the assets version. I.e., as far as I can tell, if you pass
--consoleAssetsVersion to multitenant, that version will
also make it into e.g. HTTP client user agent headers as
the proper graphql-engine version.

I'm dropping that option, since it seems unused in production
and I don't want to go to the effort of fixing it, but am happy
to look into that if folks feels strongly that it should be
kept.

(Reason for attacking this is that I was looking into http
client things around blacklisting, and the versioning thing
is a bit painful around http client headers.)

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/2458
GitOrigin-RevId: a02b05557124bdba9f65e96b3aa2746aeee03f4a
2021-10-13 16:39:58 +00:00
Solomon Bothwell
4e05bdcaec Feature/request transform string interpolation
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/2443
Co-authored-by: Tirumarai Selvan <8663570+tirumaraiselvan@users.noreply.github.com>
GitOrigin-RevId: d7d68984d0ae1403bb414572e9704c01ed27deab
2021-09-29 08:14:29 +00:00
Robert
11a454c2d6 server, pro: actually reformat the code-base using ormolu
This commit applies ormolu to the whole Haskell code base by running `make format`.

For in-flight branches, simply merging changes from `main` will result in merge conflicts.
To avoid this, update your branch using the following instructions. Replace `<format-commit>`
by the hash of *this* commit.

$ git checkout my-feature-branch
$ git merge <format-commit>^    # and resolve conflicts normally
$ make format
$ git commit -a -m "reformat with ormolu"
$ git merge -s ours post-ormolu

https://github.com/hasura/graphql-engine-mono/pull/2404

GitOrigin-RevId: 75049f5c12f430c615eafb4c6b8e83e371e01c8e
2021-09-23 22:57:37 +00:00
Naveen Naidu
3d95c67748 server/query-tags: implement new metadata specification and API
https://github.com/hasura/graphql-engine-mono/pull/2203

GitOrigin-RevId: 6169fcce0c6dccaec055bf8e723e9aa7c19a7f0c
2021-09-23 12:38:56 +00:00
Karthikeyan Chinnakonda
ccf97ab6b0 server: improve event trigger logging
https://github.com/hasura/graphql-engine-mono/pull/2286

GitOrigin-RevId: b3232a08dd7ec2aa0c9e7a2fada0e05e34a1897b
2021-09-20 14:32:31 +00:00
Karthikeyan Chinnakonda
5f79b5f102 server: generalize the event triggers codepath for all backends
https://github.com/hasura/graphql-engine-mono/pull/2189

Co-authored-by: hasura-bot <30118761+hasura-bot@users.noreply.github.com>
Co-authored-by: Martin Mark <74692114+martin-hasura@users.noreply.github.com>
Co-authored-by: Vishnu Bharathi <4211715+scriptnull@users.noreply.github.com>
Co-authored-by: Sameer Kolhar <6604943+kolharsam@users.noreply.github.com>
Co-authored-by: Antoine Leblanc <1618949+nicuveo@users.noreply.github.com>
Co-authored-by: Matt Hardman <28978422+mattshardman@users.noreply.github.com>
Co-authored-by: Vijay Prasanna <11921040+vijayprasanna13@users.noreply.github.com>
Co-authored-by: Divi <32202683+imperfect-fourth@users.noreply.github.com>
GitOrigin-RevId: 97c71571656c6e0c57d06f2d38193833180901c0
2021-09-20 07:35:49 +00:00
Antoine Leblanc
07fd7324c3 server: enable StrictData by default (fix hasura/graphql-engine#3941)
## Description

Almost all our data structures use strictness annotations, following [our styleguide's principle](https://github.com/hasura/graphql-engine/blob/master/server/STYLE.md#dealing-with-laziness) of "by default, use strict data types and lazy functions". The very few cases where we actually need laziness were already explicitly labelled as lazy with the `~` prefix operator.

This PR simply globally enables `StrictData`, allowing us to express records without `!()` on every field, but makes no attempt at cleaning existing code.

https://github.com/hasura/graphql-engine-mono/pull/1869

Co-authored-by: Philip Lykke Carlsen <358550+plcplc@users.noreply.github.com>
GitOrigin-RevId: e65c6e2f89413188da250122f64c2173615946ec
2021-09-16 12:23:15 +00:00
Solomon Bothwell
af5ff07614 Request Transformations
https://github.com/hasura/graphql-engine-mono/pull/1984

Co-authored-by: jkachmar <8461423+jkachmar@users.noreply.github.com>
GitOrigin-RevId: 1767d6bdde48c156fe171b5a9b7e44d7f2eb4869
2021-09-16 11:03:57 +00:00
Karthikeyan Chinnakonda
94f3ad041c server: generalize event triggers - incremental PR 1
https://github.com/hasura/graphql-engine-mono/pull/2269

GitOrigin-RevId: c4ea0cc41a1c66d418219cc1d41bf95656426733
2021-09-06 11:16:32 +00:00
awjchen
3aa0027c40 server: add support for tagged and dynamic metrics
https://github.com/hasura/graphql-engine-mono/pull/1680

GitOrigin-RevId: c92d4f977b5b7fe9a0b71d3841e960a95f722299
2021-08-05 21:08:17 +00:00
Robert
5cc91aebc3 server: Log action and event names (close #1603)
- add name fields to log output in several spots:
  - action logs get the action name in detail.action_name
  - events (triggered and scheduled) get the trigger name in detail.event_name
  - one-off scheduled events don't have a trigger name; instead, they get the
    comment if it exists
- remove unused event creation timestamp from ExtraLogContext

https://github.com/hasura/graphql-engine-mono/pull/1712

GitOrigin-RevId: 28907340d4e2d9adc0c48cc5d3010eef1fa902e1
2021-07-05 07:48:39 +00:00
Robert
a7a4791b96 server/eventing: clean-up around logging configuration
- Add export list to Hasura.Eventing.Common
- Group logging options in one type / argument
- Group request header processing code in one place
  This doesn't address the convoluted logic, but should make it a bit easier
  to figure out what's going on for the next person.

https://github.com/hasura/graphql-engine-mono/pull/1710

GitOrigin-RevId: 34b0abdd1b86b5836eb512484acb0db8c81f3014
2021-07-02 17:25:59 +00:00
Karthikeyan Chinnakonda
47de5e06cf server: fetch pending events across sources concurrently
https://github.com/hasura/graphql-engine-mono/pull/1562

GitOrigin-RevId: 763e60e8b175ba4b44aefbbf4ebe91455b8e3fff
2021-06-14 16:09:39 +00:00
kodiakhq[bot]
01d8a37706 server: fix asymptotics of event_log batch fetching
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
GitOrigin-RevId: 9b8afb502e94dd371a8596ccde97d023bd2344a9
2021-06-01 17:34:38 +00:00
Karthikeyan Chinnakonda
f1f56ccf75 server: source initialization fix
GitOrigin-RevId: fcb94ca743a99ee3ffe30d40717bb1f0a13cf751
2021-05-31 13:54:59 +00:00
Karthikeyan Chinnakonda
72c24eea69 server: fix cron trigger bug of new events not getting generated when cron trigger is imported via metadata
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
GitOrigin-RevId: 1a9076e039ba2c8da898683c64ef988fb8512eae
2021-05-26 16:20:19 +00:00
Swann Moreau
d3611af58d server: lazy event catalog initialisation
event catalog:
- `hdb_catalog` is no longer automatically created
- catalog is initialised when the first event trigger is created
- catalog initialisation is done during the schema cache build, using `ArrowCache` so it is only run in response to a change to the set of event triggers

event queue:
- `processEventQueue` thread is prevented from starting when `HASURA_GRAPHQL_EVENTS_FETCH_INTERVAL=0`
- `processEventQueue` thread only processes sources for which at least one event trigger exists in some table in the source

Co-authored-by: Anon Ray <616387+ecthiender@users.noreply.github.com>
GitOrigin-RevId: 73f256465d62490cd2b59dcd074718679993d4fe
2021-05-25 06:51:03 +00:00
Naveen Naidu
14f80c55ff EventTrigger: Fix num_event_fetched EKG metric
GitOrigin-RevId: 6ca047edb235f1b30986c08698410ce71f587849
2021-05-17 09:50:29 +00:00
Karthikeyan Chinnakonda
ecb4b9d098 server: graceful shutdown for event and cron triggers and async action (II)
GitOrigin-RevId: 2090c3cc34f633c691b4e4ff9ae918a60c37ce26
2021-05-14 09:39:33 +00:00
Swann Moreau
04a87970f4 server: sanitise event trigger logs (fix #1233)
- sanitise the response field in event trigger and scheduled trigger logs, removing the body and the headers
- flatten the log structure to include the event id at `$.detail.event_id` instead of `$.detail.context.event_id`

the log format changes as follows:
```diff
diff --git a/log b/log
index 96127e0..1fb909a 100644
--- a/log
+++ b/log
@@ -1,48 +1,15 @@
 {
   "detail": {
-    "context": {
-      "event_id": "b9d4e627-6029-43f2-9d46-31c532b07070"
-    },
+    "event_id": "adcc8dcd-2f32-4554-bd55-90c787aee137",
     "request": {
       "size": 416
     },
     "response": {
-      "body": "{\n  \"args\": {}, \n  \"data\": \"{\\\"event\\\":{\\\"session_variables\\\":{\\\"x-hasura-role\\\":\\\"admin\\\"},\\\"op\\\":\\\"INSERT\\\",\\\"data\\\":{\\\"old\\\":null,\\\"new\\\":{\\\"name\\\":\\\"someotheranimal\\\",\\\"id\\\":3}},\\\"trace_context\\\":{\\\"trace_id\\\":\\\"e8237c03de151634\\\",\\\"span_id\\\":\\\"8c5f8952e9e06da8\\\"}},\\\"created_at\\\":\\\"2021-05-06T07:52:58.796611Z\\\",\\\"id>
-      "headers": [
-        {
-          "name": "Date",
-          "value": "Thu, 06 May 2021 07:53:00 GMT"
-        },
-        {
-          "name": "Content-Type",
-          "value": "application/json"
-        },
-        {
-          "name": "Content-Length",
-          "value": "1692"
-        },
-        {
-          "name": "Connection",
-          "value": "keep-alive"
-        },
-        {
-          "name": "Server",
-          "value": "gunicorn/19.9.0"
-        },
-        {
-          "name": "Access-Control-Allow-Origin",
-          "value": "*"
-        },
-        {
-          "name": "Access-Control-Allow-Credentials",
-          "value": "true"
-        }
-      ],
-      "size": 1692,
+      "size": 1719,
       "status": 200
     }
   },
   "level": "info",
-  "timestamp": "2021-05-06T13:23:00.376+0530",
+  "timestamp": "2021-05-06T13:25:14.481+0530",
   "type": "event-trigger"
 }
```

GitOrigin-RevId: d9622de366737da04dc6d9ff73238be16ec9305e
2021-05-12 12:10:53 +00:00
Antoine Leblanc
2152911e24 server: introduce Hasura.Base (take 2)
GitOrigin-RevId: 0dd10f1ccd338b1cf382ebff59b6ee7f209d39a1
2021-05-11 15:19:33 +00:00
Brandon Simmons
9c9bb43a53 server: restore proper batching behavior in event trigger processing (#1237)
This essentially restores the original code from c425b554b8
(https://github.com/hasura/graphql-engine/pull/4013). Prior to this
commit we would slurp messages as fast as possible from the database
(one thing c425b55 fixed).

Another thing broken as a consequence of the same logic was the
removeEventFromLockedEvents logic which unlocks in-flight events
(breaking at-least-once delivery)

Some archeology, post-c425b55:

- cc8e2ccc erroneously attempted to refactor using `bracket`, resulting
  in the same slurp-all-events behavior (since we don't ever wait for
  processEvent to complete)
- at some point event processing within a batch is made serial, this
  reported as a bug. See: https://github.com/hasura/graphql-engine/issues/5189
- in 0ef52292b5 (which I approved...) an `async` is added, again
  causing the same issue...

GitOrigin-RevId: d8cbaab385267a4c3f1f173e268a385265980fb1
2021-04-29 04:02:05 +00:00
Karthikeyan Chinnakonda
bc1e131717 server: accept new env var HASURA_GRAPHQL_EVENTS_FETCH_BATCH_SIZE
GitOrigin-RevId: 62c463d3ee754ce9f05ba09afa8cd74ca807a96c
2021-04-27 17:23:46 +00:00
Antoine Leblanc
ba70ca427a server: switch to a sub-backend approach
GitOrigin-RevId: 660126d5f65620fb58a3ffcbed564e9e35f59938
2021-04-21 21:45:32 +00:00
Karthikeyan Chinnakonda
d09b00040f server: V2 maintenance mode fixes
GitOrigin-RevId: dcd69fb217a4e976523f1bc8e7b2693111091002
2021-04-21 10:56:13 +00:00
Antoine Leblanc
dd1192ca2c server: cleanup of language extensions [gardening]
GitOrigin-RevId: d862c724715cb8a4c2f37d2e0e525d12c46b18eb
2021-04-16 06:55:56 +00:00
Antoine Leblanc
9672cc9b32 sever: clean extension types
GitOrigin-RevId: 43f48f90dc96bec87b94a69ca1825511ec9b7507
2021-03-24 02:56:36 +00:00
Naveen Naidu
f55df228fe Server: Add Event Trigger Metrics
- [x] **Event Triggers Metrics**
  - [x] Distribution of size of event trigger fetches / Number of events fetched in the last `event trigger fetch`
  - [x] Event Triggers: Number of event trigger HTTP workers in process
  - [x]  Event Triggers: Avg event trigger lock time (if an event has been fetched but not processed because http worker is not free)

#### Sample response
The metrics can be viewed from the `/dev/ekg` endpoint

```json
{
"num_events_fetched":{
      "max":0,
      "mean":0,
      "count":1,
      "min":0,
      "variance":null,
      "type":"d",
      "sum":0
   },
 "num_event_trigger_http_workers":{
      "type":"g",
      "val":0
   },
"event_lock_time":{
      "max":0,
      "mean":0,
      "count":0,
      "min":0,
      "variance":0,
      "type":"d",
      "sum":0
   },
```

#### Todo
- [ ]  Group similar metrics together (Eg: Group all the metrics related to Event trigger, How do we do it??)

Closes: https://github.com/hasura/graphql-engine-mono/issues/202
GitOrigin-RevId: bada11d871272b04c8a09d006d9d037a8464a472
2021-03-07 05:25:24 +00:00
Vladimir Ciobanu
281cb771ff server: add MSSQL support
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Antoine Leblanc <1618949+nicuveo@users.noreply.github.com>
Co-authored-by: Vamshi Surabhi <6562944+0x777@users.noreply.github.com>
Co-authored-by: Aravind K P <8335904+scriptonist@users.noreply.github.com>
GitOrigin-RevId: 699c453b9692e1b822f393f23ff5e6db4e010d57
2021-02-23 17:38:36 +00:00
Rakesh Emmadi
9ef603360c server: generalize schema cache building (#496)
Co-authored-by: Vamshi Surabhi <vamshi@hasura.io>
Co-authored-by: Vladimir Ciobanu <admin@cvlad.info>
Co-authored-by: Antoine Leblanc <antoine@hasura.io>
Co-authored-by: Stylish Haskell Bot <stylish-haskell@users.noreply.github.com>
GitOrigin-RevId: 9d631878037637f3ed2994b5d0525efd978f7b8f
2021-02-14 06:08:46 +00:00