Commit Graph

93 Commits

Author SHA1 Message Date
Rakesh Emmadi
0c3244c2c0 multitenant: switch schema sync listener to polling mechanism
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8146
GitOrigin-RevId: 7fe82899ca8ef9eba526fe4b298b0ffbce32be82
2023-03-14 04:39:34 +00:00
Antoine Leblanc
cf531b05cb Rewrite Tracing to allow for only one TraceT in the entire stack.
This PR is on top of #7789.

### Description

This PR entirely rewrites the API of the Tracing library, to make `interpTraceT` a thing of the past. Before this change, we ran traces by sticking a `TraceT` on top of whatever we were doing. This had several major drawbacks:
- we were carrying a bunch of `TraceT` across the codebase, and the entire codebase had to know about it
- we needed to carry a second class constraint around (`HasReporterM`) to be able to run all of those traces
- we kept having to do stack rewriting with `interpTraceT`, which went from inconvenient to horrible
- we had to declare several behavioral instances on `TraceT m`

This PR rewrite all of `Tracing` using a more conventional model: there is ONE `TraceT` at the bottom of the stack, and there is an associated class constraint `MonadTrace`: any part of the code that happens to satisfy `MonadTrace` is able to create new traces. We NEVER have to do stack rewriting, `interpTraceT` is gone, and `TraceT` and `Reporter` become  implementation details that 99% of the code is blissfully unaware of: code that needs to do tracing only needs to declare that the monad in which it operates implements `MonadTrace`.

In doing so, this PR revealed **several bugs in the codebase**: places where we were expecting to trace something, but due to the default instance of `HasReporterM IO` we would actually not do anything. This PR also splits the code of `Tracing` in more byte-sized modules, with the goal of potentially moving to `server/lib` down the line.

### Remaining work

This PR is a draft; what's left to do is:
- [x] make Pro compile; i haven't updated `HasuraPro/Main` yet
- [x] document Tracing by writing a note that explains how to use the library, and the meaning of "reporter", "trace" and "span", as well as the pitfalls
- [x] discuss some of the trade-offs in the implementation, which is why i'm opening this PR already despite it not fully building yet
- [x] it depends on #7789 being merged first

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7791
GitOrigin-RevId: cadd32d039134c93ddbf364599a2f4dd988adea8
2023-03-13 17:38:39 +00:00
Philip Lykke Carlsen
e979a39f6d chore: Fix all outstanding hlint hints
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8042
GitOrigin-RevId: 87c718fa7b09f375ea0e7c2465788ac49f575290
2023-02-20 17:43:28 +00:00
Rakesh Emmadi
ab99215828 server: log when events of scheduled & event triggers are fetched from database for delivery
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7651
Co-authored-by: Sean Park-Ross <94021366+seanparkross@users.noreply.github.com>
GitOrigin-RevId: 3149dae41c6a8a16caa1d44fbc0606451d6d2cfa
2023-01-30 06:08:08 +00:00
awjchen
12fdac004f server: fix tracing bug where some errors prevent spans from being emitted
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7450
GitOrigin-RevId: 23f6c9cfea8e7ca64b39866d15d2e6187aaaa0d9
2023-01-25 03:38:21 +00:00
Daniel Harvey
06b284cf33 [server] metadata API for native access
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7476
Co-authored-by: Tom Harding <6302310+i-am-tom@users.noreply.github.com>
GitOrigin-RevId: 781c29666e92004dc82918c2292fdacc27fded4c
2023-01-16 17:21:22 +00:00
awjchen
1ec5efd5d3 server: replicate log-based data transfer metrics as prometheus metrics
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7354
GitOrigin-RevId: 3f49b8ebba515b42a9d7b22e83e6f39d9d6087c6
2022-12-28 03:49:33 +00:00
awjchen
ee78e32c6e server: implement trace sampling
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/7300
GitOrigin-RevId: d96d7fa5aaf0c1e71d1c4c0fa8f0162abce39e18
2022-12-22 19:48:51 +00:00
paritosh-08
3d4c66041f server: add event_processing_time metric
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6987
GitOrigin-RevId: 4f5948a225b18e2db59572482d2a608117fbb7d0
2022-12-06 15:11:04 +00:00
paritosh-08
00005020a6 server: rename event trigger metric hasura_event_processing_time_seconds to hasura_event_webhook_processing_time_seconds
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6950
GitOrigin-RevId: a32ce3f167eb6bb5c48645a614288c043c252d26
2022-11-21 07:49:28 +00:00
paritosh-08
5be4d37240 server,pro: add metrics related to event trigger runtime
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6825
GitOrigin-RevId: 913e1bd21912db781d69f579aee943faf6ce3a11
2022-11-16 17:12:48 +00:00
Samir Talwar
342391f39d Upgrade Ormolu to v0.5.
This upgrades the version of Ormolu required by the HGE repository to v0.5.0.1, and reformats all code accordingly.

Ormolu v0.5 reformats code that uses infix operators. This is mostly useful, adding newlines and indentation to make it clear which operators are applied first, but in some cases, it's unpleasant. To make this easier on the eyes, I had to do the following:

* Add a few fixity declarations (search for `infix`)
* Add parentheses to make precedence clear, allowing Ormolu to keep everything on one line
* Rename `relevantEq` to `(==~)` in #6651 and set it to `infix 4`
* Add a few _.ormolu_ files (thanks to @hallettj for helping me get started), mostly for Autodocodec operators that don't have explicit fixity declarations

In general, I think these changes are quite reasonable. They mostly affect indentation.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6675
GitOrigin-RevId: cd47d87f1d089fb0bc9dcbbe7798dbceedcd7d83
2022-11-02 20:55:13 +00:00
Tom Harding
e71496efa5 Replace Hasura.RQL.Types.Numeric with refined
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5913
GitOrigin-RevId: 96e218229a08dfbc5a598d709be1ee2083d41ec6
2022-09-21 18:03:04 +00:00
paritosh-08
ab71adc3a0 server, pro: event trigger auto cleanup (increment 3)
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5823
Co-authored-by: pranshi06 <85474619+pranshi06@users.noreply.github.com>
Co-authored-by: Puru Gupta <32328846+purugupta99@users.noreply.github.com>
Co-authored-by: Karthikeyan Chinnakonda <15602904+codingkarthik@users.noreply.github.com>
GitOrigin-RevId: 3d4af905d5df4c05107ad46ef29f4c0a567ff754
2022-09-15 11:46:22 +00:00
Solomon
418d0e2cc2 Use NonNegative types for arg/env parsing
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5305
GitOrigin-RevId: 8274ca8c4217d15939fa2a78664a2be486426bf7
2022-08-17 01:09:19 +00:00
awjchen
1a4e3ae001 server: add event trigger metrics for Prometheus
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5346
GitOrigin-RevId: 0db925f85e06e99b636995887c0e19a49fe9be2d
2022-08-15 05:34:04 +00:00
Solomon
3855efa9a5 Move soEventsHttpPoolSize and soEventsFetchInterval defaulting into arg parser
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5284
GitOrigin-RevId: f2302b3219f3aa9ebde623b4efa7f3ee8a97c031
2022-08-10 19:18:11 +00:00
Solomon
07e4a49e5e Consolidate Arg/Env Parser Defaults and cleanup arg merging code
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5262
GitOrigin-RevId: 2501e44a8fe0a57b13141c307d2dd44b028052b6
2022-08-05 00:29:52 +00:00
Samir Talwar
40617719ef server: Remove the Show instance from QErr and anything that touches it.
We only use these `Show` instances in error messages (where we call
`show` explicitly anyway) and test cases (in which Hspec requires `Show
a` for any `a` in an assertion).

This removes the instance in favor of a custom `showQErr` function
(which serializes the error to JSON). It is then used in certain error
message production which previously called `show` on a `QErr`.

There are two places where we serialize a QErr and then construct a new
QErr from the resulting string. Instead, we modify the existing QErr to
add extra information.

An orphan `Show QErr` instance is retained for tests so that we can have
nice test failure messages.

This is preparation for future changes in which the error message within
`QErr` will not be exposed directly, and therefore will not have a
`Show` instance. That said, it feels like a sensible kind of cleanup
anyway.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4897
GitOrigin-RevId: 8f79f7a356f0aea571156f39aefac242bf751f3a
2022-07-01 11:48:26 +00:00
Daniel Harvey
03c93ad96f server: Refactor TByteString to SerializableBlob
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4736
GitOrigin-RevId: 446832b00c24205c2237666ac9d954277c5fd05d
2022-06-17 09:57:38 +00:00
Puru Gupta
ebc8cd5134 server: do not serialize env vars in logs or errors: PR III - Event Triggers
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4521
Co-authored-by: Naveen Naidu <30195193+Naveenaidu@users.noreply.github.com>
Co-authored-by: Brandon Simmons <210815+jberryman@users.noreply.github.com>
GitOrigin-RevId: c7702e8f3da466dc108936328de30ceb87df258a
2022-06-05 20:28:13 +00:00
Naveen Naidu
222419527f server: refactor MaintenanceMode datatype
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4298
GitOrigin-RevId: 0b88d040bd4b24c61c4ae91e18dc6edf67b3672e
2022-04-28 20:56:11 +00:00
Antoine Leblanc
3cbcbd9291 Remove RQL/Types.hs
## Description

This PR removes `RQL.Types`, which was now only re-exporting a bunch of unrelated modules.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4363
GitOrigin-RevId: 894f29a19bff70b3dad8abc5d9858434d5065417
2022-04-27 13:58:47 +00:00
jkachmar
647231b685 Yeet some default-extensions
Manually enables:
* EmptyCase
* ExistentialQuantification
* QuantifiedConstraints
* QuasiQuotes
* TemplateHaskell
* TypeFamilyDependencies

...in the following components:
* 'graphql-engine' library
* 'graphql-engine' 'src-test'
* 'graphql-engine' 'tests/integration'
* 'graphql-engine' tests-hspec'

Additionally, performs some light refactoring and documentation.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3991
GitOrigin-RevId: 514477d3466b01f60eca8935d0fef60dd0756838
2022-03-16 00:40:17 +00:00
Karthikeyan Chinnakonda
a2da867fc8 Fetch only events of the ETs which exist in the metadata
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3857
GitOrigin-RevId: 1ee501116103e2be49d4538c80f87c11c066af8e
2022-03-08 09:06:32 +00:00
Solomon
d67d4e2310 Webhook Transform Cleanup / Refactor
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3624
GitOrigin-RevId: 849e6dd70d6fe3d84056a485b20928ff813881d4
2022-03-08 00:43:08 +00:00
Solomon
4b792abdcc Feature/webhook response transforms
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3074
Co-authored-by: David Overton <7734777+dmoverton@users.noreply.github.com>
Co-authored-by: Lyndon Maydwell <92299+sordina@users.noreply.github.com>
Co-authored-by: Tirumarai Selvan <8663570+tirumaraiselvan@users.noreply.github.com>
GitOrigin-RevId: 006c5c9b71cdca1c2f47962230e6189e09557fab
2022-01-19 04:47:36 +00:00
Solomon
0faa92337d Use transformed request body in event invocation logs
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3057
GitOrigin-RevId: 7e001830f97c81289553e37c35098ccaeadbdf22
2021-12-06 22:40:31 +00:00
David Overton
aac64f2c81 Source typename customization (close graphql-engine#6974)
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/1616
GitOrigin-RevId: f7eefd2367929209aa77895ea585e96a99a78d47
2021-10-29 14:43:14 +00:00
Robert
71af68e9e5 server: drop HasVersion implicit parameter (closes #2236)
The only real use was for the dubious multitenant option
--consoleAssetsVersion, which actually overrode not just
the assets version. I.e., as far as I can tell, if you pass
--consoleAssetsVersion to multitenant, that version will
also make it into e.g. HTTP client user agent headers as
the proper graphql-engine version.

I'm dropping that option, since it seems unused in production
and I don't want to go to the effort of fixing it, but am happy
to look into that if folks feels strongly that it should be
kept.

(Reason for attacking this is that I was looking into http
client things around blacklisting, and the versioning thing
is a bit painful around http client headers.)

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/2458
GitOrigin-RevId: a02b05557124bdba9f65e96b3aa2746aeee03f4a
2021-10-13 16:39:58 +00:00
Solomon Bothwell
4e05bdcaec Feature/request transform string interpolation
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/2443
Co-authored-by: Tirumarai Selvan <8663570+tirumaraiselvan@users.noreply.github.com>
GitOrigin-RevId: d7d68984d0ae1403bb414572e9704c01ed27deab
2021-09-29 08:14:29 +00:00
Robert
11a454c2d6 server, pro: actually reformat the code-base using ormolu
This commit applies ormolu to the whole Haskell code base by running `make format`.

For in-flight branches, simply merging changes from `main` will result in merge conflicts.
To avoid this, update your branch using the following instructions. Replace `<format-commit>`
by the hash of *this* commit.

$ git checkout my-feature-branch
$ git merge <format-commit>^    # and resolve conflicts normally
$ make format
$ git commit -a -m "reformat with ormolu"
$ git merge -s ours post-ormolu

https://github.com/hasura/graphql-engine-mono/pull/2404

GitOrigin-RevId: 75049f5c12f430c615eafb4c6b8e83e371e01c8e
2021-09-23 22:57:37 +00:00
Naveen Naidu
3d95c67748 server/query-tags: implement new metadata specification and API
https://github.com/hasura/graphql-engine-mono/pull/2203

GitOrigin-RevId: 6169fcce0c6dccaec055bf8e723e9aa7c19a7f0c
2021-09-23 12:38:56 +00:00
Karthikeyan Chinnakonda
ccf97ab6b0 server: improve event trigger logging
https://github.com/hasura/graphql-engine-mono/pull/2286

GitOrigin-RevId: b3232a08dd7ec2aa0c9e7a2fada0e05e34a1897b
2021-09-20 14:32:31 +00:00
Karthikeyan Chinnakonda
5f79b5f102 server: generalize the event triggers codepath for all backends
https://github.com/hasura/graphql-engine-mono/pull/2189

Co-authored-by: hasura-bot <30118761+hasura-bot@users.noreply.github.com>
Co-authored-by: Martin Mark <74692114+martin-hasura@users.noreply.github.com>
Co-authored-by: Vishnu Bharathi <4211715+scriptnull@users.noreply.github.com>
Co-authored-by: Sameer Kolhar <6604943+kolharsam@users.noreply.github.com>
Co-authored-by: Antoine Leblanc <1618949+nicuveo@users.noreply.github.com>
Co-authored-by: Matt Hardman <28978422+mattshardman@users.noreply.github.com>
Co-authored-by: Vijay Prasanna <11921040+vijayprasanna13@users.noreply.github.com>
Co-authored-by: Divi <32202683+imperfect-fourth@users.noreply.github.com>
GitOrigin-RevId: 97c71571656c6e0c57d06f2d38193833180901c0
2021-09-20 07:35:49 +00:00
Antoine Leblanc
07fd7324c3 server: enable StrictData by default (fix hasura/graphql-engine#3941)
## Description

Almost all our data structures use strictness annotations, following [our styleguide's principle](https://github.com/hasura/graphql-engine/blob/master/server/STYLE.md#dealing-with-laziness) of "by default, use strict data types and lazy functions". The very few cases where we actually need laziness were already explicitly labelled as lazy with the `~` prefix operator.

This PR simply globally enables `StrictData`, allowing us to express records without `!()` on every field, but makes no attempt at cleaning existing code.

https://github.com/hasura/graphql-engine-mono/pull/1869

Co-authored-by: Philip Lykke Carlsen <358550+plcplc@users.noreply.github.com>
GitOrigin-RevId: e65c6e2f89413188da250122f64c2173615946ec
2021-09-16 12:23:15 +00:00
Solomon Bothwell
af5ff07614 Request Transformations
https://github.com/hasura/graphql-engine-mono/pull/1984

Co-authored-by: jkachmar <8461423+jkachmar@users.noreply.github.com>
GitOrigin-RevId: 1767d6bdde48c156fe171b5a9b7e44d7f2eb4869
2021-09-16 11:03:57 +00:00
Karthikeyan Chinnakonda
94f3ad041c server: generalize event triggers - incremental PR 1
https://github.com/hasura/graphql-engine-mono/pull/2269

GitOrigin-RevId: c4ea0cc41a1c66d418219cc1d41bf95656426733
2021-09-06 11:16:32 +00:00
awjchen
3aa0027c40 server: add support for tagged and dynamic metrics
https://github.com/hasura/graphql-engine-mono/pull/1680

GitOrigin-RevId: c92d4f977b5b7fe9a0b71d3841e960a95f722299
2021-08-05 21:08:17 +00:00
Robert
5cc91aebc3 server: Log action and event names (close #1603)
- add name fields to log output in several spots:
  - action logs get the action name in detail.action_name
  - events (triggered and scheduled) get the trigger name in detail.event_name
  - one-off scheduled events don't have a trigger name; instead, they get the
    comment if it exists
- remove unused event creation timestamp from ExtraLogContext

https://github.com/hasura/graphql-engine-mono/pull/1712

GitOrigin-RevId: 28907340d4e2d9adc0c48cc5d3010eef1fa902e1
2021-07-05 07:48:39 +00:00
Robert
a7a4791b96 server/eventing: clean-up around logging configuration
- Add export list to Hasura.Eventing.Common
- Group logging options in one type / argument
- Group request header processing code in one place
  This doesn't address the convoluted logic, but should make it a bit easier
  to figure out what's going on for the next person.

https://github.com/hasura/graphql-engine-mono/pull/1710

GitOrigin-RevId: 34b0abdd1b86b5836eb512484acb0db8c81f3014
2021-07-02 17:25:59 +00:00
Karthikeyan Chinnakonda
47de5e06cf server: fetch pending events across sources concurrently
https://github.com/hasura/graphql-engine-mono/pull/1562

GitOrigin-RevId: 763e60e8b175ba4b44aefbbf4ebe91455b8e3fff
2021-06-14 16:09:39 +00:00
kodiakhq[bot]
01d8a37706 server: fix asymptotics of event_log batch fetching
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
GitOrigin-RevId: 9b8afb502e94dd371a8596ccde97d023bd2344a9
2021-06-01 17:34:38 +00:00
Karthikeyan Chinnakonda
f1f56ccf75 server: source initialization fix
GitOrigin-RevId: fcb94ca743a99ee3ffe30d40717bb1f0a13cf751
2021-05-31 13:54:59 +00:00
Karthikeyan Chinnakonda
72c24eea69 server: fix cron trigger bug of new events not getting generated when cron trigger is imported via metadata
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
GitOrigin-RevId: 1a9076e039ba2c8da898683c64ef988fb8512eae
2021-05-26 16:20:19 +00:00
Swann Moreau
d3611af58d server: lazy event catalog initialisation
event catalog:
- `hdb_catalog` is no longer automatically created
- catalog is initialised when the first event trigger is created
- catalog initialisation is done during the schema cache build, using `ArrowCache` so it is only run in response to a change to the set of event triggers

event queue:
- `processEventQueue` thread is prevented from starting when `HASURA_GRAPHQL_EVENTS_FETCH_INTERVAL=0`
- `processEventQueue` thread only processes sources for which at least one event trigger exists in some table in the source

Co-authored-by: Anon Ray <616387+ecthiender@users.noreply.github.com>
GitOrigin-RevId: 73f256465d62490cd2b59dcd074718679993d4fe
2021-05-25 06:51:03 +00:00
Naveen Naidu
14f80c55ff EventTrigger: Fix num_event_fetched EKG metric
GitOrigin-RevId: 6ca047edb235f1b30986c08698410ce71f587849
2021-05-17 09:50:29 +00:00
Karthikeyan Chinnakonda
ecb4b9d098 server: graceful shutdown for event and cron triggers and async action (II)
GitOrigin-RevId: 2090c3cc34f633c691b4e4ff9ae918a60c37ce26
2021-05-14 09:39:33 +00:00
Swann Moreau
04a87970f4 server: sanitise event trigger logs (fix #1233)
- sanitise the response field in event trigger and scheduled trigger logs, removing the body and the headers
- flatten the log structure to include the event id at `$.detail.event_id` instead of `$.detail.context.event_id`

the log format changes as follows:
```diff
diff --git a/log b/log
index 96127e0..1fb909a 100644
--- a/log
+++ b/log
@@ -1,48 +1,15 @@
 {
   "detail": {
-    "context": {
-      "event_id": "b9d4e627-6029-43f2-9d46-31c532b07070"
-    },
+    "event_id": "adcc8dcd-2f32-4554-bd55-90c787aee137",
     "request": {
       "size": 416
     },
     "response": {
-      "body": "{\n  \"args\": {}, \n  \"data\": \"{\\\"event\\\":{\\\"session_variables\\\":{\\\"x-hasura-role\\\":\\\"admin\\\"},\\\"op\\\":\\\"INSERT\\\",\\\"data\\\":{\\\"old\\\":null,\\\"new\\\":{\\\"name\\\":\\\"someotheranimal\\\",\\\"id\\\":3}},\\\"trace_context\\\":{\\\"trace_id\\\":\\\"e8237c03de151634\\\",\\\"span_id\\\":\\\"8c5f8952e9e06da8\\\"}},\\\"created_at\\\":\\\"2021-05-06T07:52:58.796611Z\\\",\\\"id>
-      "headers": [
-        {
-          "name": "Date",
-          "value": "Thu, 06 May 2021 07:53:00 GMT"
-        },
-        {
-          "name": "Content-Type",
-          "value": "application/json"
-        },
-        {
-          "name": "Content-Length",
-          "value": "1692"
-        },
-        {
-          "name": "Connection",
-          "value": "keep-alive"
-        },
-        {
-          "name": "Server",
-          "value": "gunicorn/19.9.0"
-        },
-        {
-          "name": "Access-Control-Allow-Origin",
-          "value": "*"
-        },
-        {
-          "name": "Access-Control-Allow-Credentials",
-          "value": "true"
-        }
-      ],
-      "size": 1692,
+      "size": 1719,
       "status": 200
     }
   },
   "level": "info",
-  "timestamp": "2021-05-06T13:23:00.376+0530",
+  "timestamp": "2021-05-06T13:25:14.481+0530",
   "type": "event-trigger"
 }
```

GitOrigin-RevId: d9622de366737da04dc6d9ff73238be16ec9305e
2021-05-12 12:10:53 +00:00
Antoine Leblanc
2152911e24 server: introduce Hasura.Base (take 2)
GitOrigin-RevId: 0dd10f1ccd338b1cf382ebff59b6ee7f209d39a1
2021-05-11 15:19:33 +00:00