Commit Graph

51 Commits

Author SHA1 Message Date
Brandon Simmons
440a8664de server: Upgrade the HGE server to GHC 9.4.5
Upgrade to GHC 9.4.5, and update any tests.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8954
Co-authored-by: Mohd Bilal <24944223+m-Bilal@users.noreply.github.com>
Co-authored-by: Samir Talwar <47582+SamirTalwar@users.noreply.github.com>
Co-authored-by: Philip Lykke Carlsen <358550+plcplc@users.noreply.github.com>
GitOrigin-RevId: 5261126777cb478567ea471c4bf5441bc345ea0d
2023-06-06 13:29:30 +00:00
Samir Talwar
6ed9f36125 Remove the server upgrade/downgrade tests.
These tests are intended to catch issues in upgrading HGE. However:

* the tests are very convoluted and hard to understand,
* we can only run a small subset of Python tests that don't mutate any data or metadata, and
* I have never seen them fail for a legitimate reason, but I've seen a lot of flakes.

While we do believe it's important to test that upgrades don't break the stored introspection, these tests don't seem to be doing that any more. I humbly request that we delete them now and either (a) figure out how to test this properly, or (b) just wait for v3, which does away with reintrospecting on server startup entirely.

[NDAT-259]: https://hasurahq.atlassian.net/browse/NDAT-259?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/8844
GitOrigin-RevId: 528bc632fce377b7eff2026b832bd58586ac5a0b
2023-04-24 10:04:15 +00:00
Samir Talwar
133a9b7452 server/tests-py: Generate random ports for helper services.
If the tests are run with specific ports assigned to specific services,
set through the environment variables, we continue to use those ports.
We just don't hard-code them now, we pick them up from the environment
variables.

However, if the environment variables are not set, we generate a random
port for each service. This allows us to run multiple tests in parallel
in the future, independently.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6218
GitOrigin-RevId: 3d2a1880bf67544c848951888ce7b4fa1ba379dc
2022-10-21 17:34:21 +00:00
Samir Talwar
0d4d7e6b1e server/tests-py: Start helper services in the test harness.
This makes it possible for the test harness to start the test JWK server and the test remote schema server.

In order to do this, we still generate the TLS certificates in the test script (because we need to install the generated CA certificate in the OS certificate store), and then pass the certificate and key paths into the test runner.

Because we are still using _test-server.sh_ for now, we don't use the JWK server fixture in that case, as HGE needs the JWK server to be up and running when it starts. Instead, we keep running it outside (for now).

This is also the case for the GraphQL server fixture when we are running the server upgrade/downgrade tests.

I have also refactored _graphql_server.py_ so there isn't a global `HGE_URLS` value, but instead the value is passed through.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/6303
GitOrigin-RevId: 06f05ff674372dc5d632e55d68e661f5c7a17c10
2022-10-13 15:45:24 +00:00
Samir Talwar
bff1309344 server/tests-py: Make HGECtxGQLServer a fixture.
Starting it and stopping it for the various tests that actually use it.

There are only a few.

This also removes some dead code and fixes warnings in _test_webhook_request_context.py_.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5846
GitOrigin-RevId: 7760467f9de7b1f9718e7482275c298eeaa3ad3a
2022-09-14 21:42:40 +00:00
paritosh-08
25a027ba79 server/console: use root_field_namespace as prefix for remote schema
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/5000
GitOrigin-RevId: 6427691e1609e939b4245d90cff20de72b18e99a
2022-07-13 05:30:04 +00:00
Samir Talwar
987b55f981 server/tests-py: Reduce the number of locations we check the status code.
We have a lot of `assert st_code == 200` scattered about. This is a
problem because (a) it makes the code harder to parse and (b) the error
message is lacking; I have seen a few flaky tests which were impossible
to diagnose because I didn't know what the response _should_ be.

This reduces the number of places in which we perform this assertion
(moving most of them to `HGECtx.execute_query`), so that we can have a
better chance of seeing a useful error message on test failure.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4957
GitOrigin-RevId: 3ff388bccf49f96569aa6b7db85266a0c5ee27ea
2022-07-05 18:01:07 +00:00
Puru Gupta
5501f686df server: do not serialize env vars in logs or errors: PR I - Remote Schemas
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4463
Co-authored-by: Naveen Naidu <30195193+Naveenaidu@users.noreply.github.com>
Co-authored-by: Brandon Simmons <210815+jberryman@users.noreply.github.com>
GitOrigin-RevId: 71c6824130d71312f5bd5ae94fc268c0544c6ca3
2022-05-21 09:05:59 +00:00
Auke Booij
dfd09159ec Fix JSON encoding issue (fix hasura/graphql-engine#7543 and hasura/graphql-engine#8200)
`encJFromAssocList` is broken since it does not appropriately escape the keys of the JSON dictionaries. Presumably it should.

This is a hotfix for a JSON encoding issue. Longer-term solutions are being discussed in hasura/graphql-engine-mono#3736.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/4139
GitOrigin-RevId: 27afff319d268a4ec017c8eafd694cc17c2fd7f0
2022-04-05 19:11:30 +00:00
Antoine Leblanc
ccea1da1d5 Enable and test remote relationships from remote schemas.
### Description

This is it! This PR enables the Metadata API for remote relationships from remote schemas, adds tests, ~~adds documentation~~, adds an entry to the Changelog. This is the release PR that enables the feature.

### Checklist
- [ ] Tests:
  - [x] RS-to-Postgres (high level)
  - [x] RS-to-RS (high level)
  - [x] From RS specifically (testing for edge cases)
  - [x] Metadata API tests
  - [ ] Unit testing the actual engine?
- [x] Changelog entry
- [ ] Documentation?

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3974
Co-authored-by: Vamshi Surabhi <6562944+0x777@users.noreply.github.com>
Co-authored-by: Vishnu Bharathi <4211715+scriptnull@users.noreply.github.com>
Co-authored-by: jkachmar <8461423+jkachmar@users.noreply.github.com>
GitOrigin-RevId: c9aebf12e6eebef8d264ea831a327b968d4be9d2
2022-03-17 20:54:57 +00:00
paritosh-08
a0c83b8e99 server: add inconsistency information in reload_metadata API call
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3860
GitOrigin-RevId: 02d1963946d69fdf6886aa61806eb71db60f8931
2022-03-03 13:34:44 +00:00
Naveen Naidu
2e823234f9 server/remote-schema: Include OperationName in the request sent to remote schemas
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3581
Co-authored-by: Puru Gupta <32328846+purugupta99@users.noreply.github.com>
GitOrigin-RevId: 2db21d0df158a5a3229b883ee426b2f38f864f80
2022-02-16 07:55:19 +00:00
David Overton
8eca322b94 Fix for customized remote schema input object types in query variables
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3417
GitOrigin-RevId: a4d3e59f5177f971a64a2b9e62fd1acaa8079a6b
2022-01-24 04:46:46 +00:00
Vamshi Surabhi
2e1be1d2c5 bump python test suite dependencies
This PR upgrades some of the pinned dependencies do not build with python 3.10 - cffi, ruamel, py. Further, it upgrades other packages where the effort is minimal.

For the reviewers: Please review it commit by commit.

PR-URL: https://github.com/hasura/graphql-engine-mono/pull/3367
GitOrigin-RevId: c5401fe289d3185a79c4d382297f86fbde139825
2022-01-17 07:40:56 +00:00
David Overton
5bfce057c6 Refactor remote schema customization
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/2771
GitOrigin-RevId: 0c90136f956df3f4552140e6ca3d2f4766f8b3f5
2021-11-30 00:38:27 +00:00
David Overton
ec60386f9c Allow "extensions" field in remote schema response
PR-URL: https://github.com/hasura/graphql-engine-mono/pull/2818
GitOrigin-RevId: 505e8bae6d3e11199c229bd2b86af09161eb8b66
2021-11-10 02:34:56 +00:00
David Overton
3e1e0773fe Test customization with update_remote_schema
https://github.com/hasura/graphql-engine-mono/pull/1971

GitOrigin-RevId: 617f15193e1149d08327d3a7eb0b21e11e280561
2021-08-02 05:50:49 +00:00
David Overton
1abb1dee69 Remote Schema Customization take 2 using parser tranformations
https://github.com/hasura/graphql-engine-mono/pull/1740

GitOrigin-RevId: e807952058243a97f67cd9969fa434933a08652f
2021-07-30 11:33:59 +00:00
Sameer Kolhar
608e4fbb20 server, console, docs: add update_remote_schema API
https://github.com/hasura/graphql-engine-mono/pull/1546

GitOrigin-RevId: 142b0d0e0ffc35b2679c91c411868c45a8b8e221
2021-06-21 17:42:40 +00:00
Vamshi Surabhi
96104ec1a8 Revert "remote schema typename customisation"
This reverts the remote schema type customisation and namespacing feature temporarily as we test for certain conditions.

GitOrigin-RevId: f8ee97233da4597f703970c3998664c03582d8e7
2021-06-10 09:57:16 +00:00
David Overton
4a69fdeb01 Dmoverton/5863 prefix namespacing
GitOrigin-RevId: 108e8b25e745cb4f74d143d316262049cef62b70
2021-06-09 22:42:05 +00:00
Abby Sassel
3ba5d1e540 server: include more detail in inconsistent metadata error messages (fix #6684)
GitOrigin-RevId: 1a0d8fa2b465320e9ab4ea43259d0e8d92364473
2021-05-19 05:02:53 +00:00
Rakesh Emmadi
29f2ddc289 server: support separate metadata database and server code setup for multi sources (#197)
This is an incremental PR towards https://github.com/hasura/graphql-engine/pull/5797

Co-authored-by: Anon Ray <ecthiender@users.noreply.github.com>
GitOrigin-RevId: a6cb8c239b2ff840a0095e78845f682af0e588a9
2020-12-28 12:56:55 +00:00
Rakesh Emmadi
a2cf9a53c2 server: move to storing metadata as a json blob (#115)
GitOrigin-RevId: 3d1a7618a4ec086c2d255549a6c15087201e9ab0
2020-12-08 14:23:28 +00:00
Karthikeyan Chinnakonda
3ea611f9fd
Server: Validate remote schema queries (#5938)
* [skip ci] use the args while making the fieldParser

* modify the execution part of the remote queries

* parse union queries deeply

* add test for remote schema field validation

* add tests for validating remote query arguments


Co-authored-by: Auke Booij <auke@hasura.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2020-10-13 14:03:11 +05:30
Alexis King
7e970177c1
Rewrite GraphQL schema generation and query parsing (close #2801) (#4111)
Aka “the PDV refactor.” History is preserved on the branch 2801-graphql-schema-parser-refactor.

* [skip ci] remove stale benchmark commit from commit_diff

* [skip ci] Check for root field name conflicts between remotes

* [skip ci] Additionally check for conflicts between remotes and DB

* [skip ci] Check for conflicts in schema when tracking a table

* [skip ci] Fix equality checking in GraphQL AST

* server: fix mishandling of GeoJSON inputs in subscriptions (fix #3239) (#4551)

* Add support for multiple top-level fields in a subscription to improve testability of subscriptions

* Add an internal flag to enable multiple subscriptions

* Add missing call to withConstructorFn in live queries (fix #3239)

Co-authored-by: Alexis King <lexi.lambda@gmail.com>

* Scheduled triggers (close #1914) (#3553)

server: add scheduled triggers

Co-authored-by: Alexis King <lexi.lambda@gmail.com>
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>

* dev.sh: bump version due to addition of croniter python dependency

* server: fix an introspection query caching issue (fix #4547) (#4661)

Introspection queries accept variables, but we need to make sure to
also touch the variables that we ignore, so that an introspection
query is marked not reusable if we are not able to build a correct
query plan for it.

A better solution here would be to deal with such unused variables
correctly, so that more introspection queries become reusable.

An even better solution would be to type-safely track *how* to reuse
which variables, rather than to split the reusage marking from the
planning.

Co-authored-by: Tirumarai Selvan <tiru@hasura.io>

* flush log buffer on exception in mkWaiApp ( fix #4772 ) (#4801)

* flush log buffer on exception in mkWaiApp

* add comment to explain the introduced change

* add changelog

* allow logging details of a live query polling thread (#4959)

* changes for poller-log

add various multiplexed query info in poller-log

* minor cleanup, also fixes a bug which will return duplicate data

* Live query poller stats can now be logged

This also removes in-memory stats that are collected about batched
query execution as the log lines when piped into an monitoring tool
will give us better insights.

* allow poller-log to be configurable

* log minimal information in the livequery-poller-log

Other information can be retrieved from /dev/subscriptions/extended

* fix few review comments

* avoid marshalling and unmarshalling from ByteString to EncJSON

* separate out SubscriberId and SubscriberMetadata

Co-authored-by: Anon Ray <rayanon004@gmail.com>

* Don't compile in developer APIs by default

* Tighten up handling of admin secret, more docs

Store the admin secret only as a hash to prevent leaking the secret
inadvertently, and to prevent timing attacks on the secret.

NOTE: best practice for stored user passwords is a function with a
tunable cost like bcrypt, but our threat model is quite different (even
if we thought we could reasonably protect the secret from an attacker
who could read arbitrary regions of memory), and bcrypt is far too slow
(by design) to perform on each request. We'd have to rely on our
(technically savvy) users to choose high entropy passwords in any case.

Referencing #4736

* server/docs: add instructions to fix loss of float precision in PostgreSQL <= 11 (#5187)

This adds a server flag, --pg-connection-options, that can be used to set a PostgreSQL connection parameter, extra_float_digits, that needs to be used to avoid loss of data on older versions of PostgreSQL, which have odd default behavior when returning float values. (fixes #5092)

* [skip ci] Add new commits from master to the commit diff

* [skip ci] serve default directives (skip & include) over introspection

* [skip ci] Update non-Haskell assets with the version on master

* server: refactor GQL execution check and config API (#5094)

Co-authored-by: Vamshi Surabhi <vamshi@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* [skip ci] fix js issues in tests by pinning dependencies version

* [skip ci] bump graphql version

* [skip ci] Add note about memory usage

* generalize query execution logic on Postgres (#5110)

* generalize PGExecCtx to support specialized functions for various operations

* fix tests compilation

* allow customising PGExecCtx when starting the web server

* server: changes catalog initialization and logging for pro customization (#5139)

* new typeclass to abstract the logic of QueryLog-ing

* abstract the logic of logging websocket-server logs

  introduce a MonadWSLog typeclass

* move catalog initialization to init step

  expose a helper function to migrate catalog
  create schema cache in initialiseCtx

* expose various modules and functions for pro

* [skip ci] cosmetic change

* [skip ci] fix test calling a mutation that does not exist

* [skip ci] minor text change

* [skip ci] refactored input values

* [skip ci] remove VString Origin

* server: fix updating of headers behaviour in the update cron trigger API and create future events immediately (#5151)

* server: fix bug to update headers in an existing cron trigger and create future events

Co-authored-by: Tirumarai Selvan <tiru@hasura.io>

* Lower stack chunk size in RTS to reduce thread STACK memory (closes #5190)

This reduces memory consumption for new idle subscriptions significantly
(see linked ticket).

The hypothesis is: we fork a lot of threads per websocket, and some of
these use slightly more than the initial 1K stack size, so the first
overflow balloons to 32K, when significantly less is required.

However: running with `+RTS -K1K -xc` did not seem to show evidence of
any overflows! So it's a mystery why this improves things.

GHC should probably also be doubling the stack buffer at each overflow
or doing something even smarter; the knobs we have aren't so helpful.

* [skip ci] fix todo and schema generation for aggregate fields

* 5087 libpq pool leak (#5089)

Shrink libpq buffers to 1MB before returning connection to pool. Closes #5087

See: https://github.com/hasura/pg-client-hs/pull/19

Also related: #3388 #4077

* bump pg-client-hs version (fixes a build issue on some environments) (#5267)

* do not use prepared statements for mutations

* server: unlock scheduled events on graceful shutdown (#4928)

* Fix buggy parsing of new --conn-lifetime flag in 2b0e3774

* [skip ci] remove cherry-picked commit from commit_diff.txt

* server: include additional fields in scheduled trigger webhook payload (#5262)

* include scheduled triggers metadata in the webhook body

Co-authored-by: Tirumarai Selvan <tiru@hasura.io>

* server: call the webhook asynchronously in event triggers (#5352)

* server: call the webhook asynchronosly in event triggers

* Expose all modules in Cabal file (#5371)

* [skip ci] update commit_diff.txt

* [skip ci] fix cast exp parser & few TODOs

* [skip ci] fix remote fields arguments

* [skip ci] fix few more TODO, no-op refactor, move resolve/action.hs to execute/action.hs

* Pass environment variables around as a data structure, via @sordina (#5374)

* Pass environment variables around as a data structure, via @sordina

* Resolving build error

* Adding Environment passing note to changelog

* Removing references to ILTPollerLog as this seems to have been reintroduced from a bad merge

* removing commented-out imports

* Language pragmas already set by project

* Linking async thread

* Apply suggestions from code review

Use `runQueryTx` instead of `runLazyTx` for queries.

* remove the non-user facing entry in the changelog

Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* [skip ci] fix: restrict remote relationship field generation for hasura queries

* [skip ci] no-op refactor; move insert execution code from schema parser module

* server: call the webhook asynchronously in event triggers (#5352)

* server: call the webhook asynchronosly in event triggers

* Expose all modules in Cabal file (#5371)

* [skip ci] update commit_diff.txt

* Pass environment variables around as a data structure, via @sordina (#5374)

* Pass environment variables around as a data structure, via @sordina

* Resolving build error

* Adding Environment passing note to changelog

* Removing references to ILTPollerLog as this seems to have been reintroduced from a bad merge

* removing commented-out imports

* Language pragmas already set by project

* Linking async thread

* Apply suggestions from code review

Use `runQueryTx` instead of `runLazyTx` for queries.

* remove the non-user facing entry in the changelog

Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* [skip ci] implement header checking

Probably closes #14 and #3659.

* server: refactor 'pollQuery' to have a hook to process 'PollDetails' (#5391)

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* update pg-client (#5421)

* [skip ci] update commit_diff

* Fix latency buckets for telemetry data

These must have gotten messed up during a refactor. As a consequence
almost all samples received so far fall into the single erroneous 0 to
1K seconds (originally supposed to be 1ms?) bucket.

I also re-thought what the numbers should be, but these are still
arbitrary and might want adjusting in the future.

* [skip ci] include the latest commit compared against master in commit_diff

* [skip ci] include new commits from master in commit_diff

* [skip ci] improve description generation

* [skip ci] sort all introspect arrays

* [skip ci] allow parsers to specify error codes

* [skip ci] fix integer and float parsing error code

* [skip ci] scalar from json errors are now parse errors

* [skip ci] fixed negative integer error message and code

* [skip ci] Re-fix nullability in relationships

* [skip ci] no-op refactor and removed couple of FIXMEs

* [skip ci] uncomment code in 'deleteMetadataObject'

* [skip ci] Fix re-fix of nullability for relationships

* [skip ci] fix default arguments error code

* [skip ci] updated test error message

!!! WARNING !!!
Since all fields accept `null`, they all are technically optional in
the new schema. Meaning there's no such thing as a missing mandatory
field anymore: a field that doesn't have a default value, and which
therefore isn't labelled as "optional" in the schema, will be assumed
to be null if it's missing, meaning it isn't possible anymore to have
an error for a missing mandatory field. The only possible error is now
when a optional positional argument is omitted but is not the last
positional argument.

* [skip ci] cleanup of int scalar parser

* [skip ci] retro-compatibility of offset as string

* [skip ci] Remove commit from commit_diff.txt

Although strictly speaking we don't know if this will work correctly in PDV
if we would implement query plan caching, the fact is that in the theoretical
case that we would have the same issue in PDV, it would probably apply not just
to introspection, and the fix would be written completely differently.  So this
old commit is of no value to us other than the heads-up "make sure query plan
caching works correctly even in the presence of unused variables", which is
already part of the test suite.

* Add MonadTrace and MonadExecuteQuery abstractions (#5383)

* [skip ci] Fix accumulation of input object types

Just like object types, interface types, and union types, we have to avoid
circularities when collecting input types from the GraphQL AST.

Additionally, this fixes equality checks for input object types (whose fields
are unordered, and hence should be compared as sets) and enum types (ditto).

* [skip ci] fix fragment error path

* [skip ci] fix node error code

* [skip ci] fix paths in insert queries

* [skip ci] fix path in objects

* [skip ci] manually alter node id path for consistency

* [skip ci] more node error fixups

* [skip ci] one last relay error message fix

* [skip ci] update commit_diff

* Propagate the trace context to event triggers (#5409)

* Propagate the trace context to event triggers

* Handle missing trace and span IDs

* Store trace context as one LOCAL

* Add migrations

* Documentation

* changelog

* Fix warnings

* Respond to code review suggestions

* Respond to code review

* Undo changelog

* Update CHANGELOG.md

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* server: log request/response sizes for event triggers (#5463)

* server: log request/response sizes for event triggers

  event triggers (and scheduled triggers) now have request/response size
  in their logs.

* add changelog entry

* Tracing: Simplify HTTP traced request (#5451)

Remove the Inversion of Control (SuspendRequest) and simplify
the tracing of HTTP Requests.

Co-authored-by: Phil Freeman <phil@hasura.io>

* Attach request ID as tracing metadata (#5456)

* Propagate the trace context to event triggers

* Handle missing trace and span IDs

* Store trace context as one LOCAL

* Add migrations

* Documentation

* Include the request ID as trace metadata

* changelog

* Fix warnings

* Respond to code review suggestions

* Respond to code review

* Undo changelog

* Update CHANGELOG.md

* Typo

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* server: add logging for action handlers (#5471)

* server: add logging for action handlers

* add changelog entry

* change action-handler log type from internal to non-internal

* fix action-handler-log name

* server: pass http and websocket request to logging context (#5470)

* pass request body to logging context in all cases

* add message size logging on the websocket API

  this is required by graphql-engine-pro/#416

* message size logging on websocket API

  As we need to log all messages recieved/sent by the websocket server,
  it makes sense to log them as part of the websocket server event logs.
  Previously message recieved were logged inside the onMessage handler,
  and messages sent were logged only for "data" messages (as a server event log)

* fix review comments

Co-authored-by: Phil Freeman <phil@hasura.io>

* server: stop eventing subsystem threads when shutting down (#5479)

* server: stop eventing subsystem threads when shutting down

* Apply suggestions from code review

Co-authored-by: Karthikeyan Chinnakonda <chkarthikeyan95@gmail.com>

Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Karthikeyan Chinnakonda <chkarthikeyan95@gmail.com>

* [skip ci] update commit_diff with new commits added in master

* Bugfix to support 0-size HASURA_GRAPHQL_QUERY_PLAN_CACHE_SIZE

Also some minor refactoring of bounded cache module:
 - the maxBound check in `trim` was confusing and unnecessary
 - consequently trim was unnecessary for lookupPure

Also add some basic tests

* Support only the bounded cache, with default HASURA_GRAPHQL_QUERY_PLAN_CACHE_SIZE of 4000. Closes #5363

* [skip ci] remove merge commit from commit_diff

* server: Fix compiler warning caused by GHC upgrade (#5489)

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* [skip ci] update all non server code from master

* [skip ci] aligned object field error message with master

* [skip ci] fix remaining undefined?

* [skip ci] remove unused import

* [skip ci] revert to previous error message, fix tests

* Move nullableType/nonNullableType to Schema.hs

These are functions on Types, not on Parsers.

* [skip ci] fix setup to fix backend only test

the order in which permission checks are performed on the branch is
slightly different than on master, resulting in a slightly different
error if there are no other mutations the user has access to. By
adding update permissions, we go back to the expected case.

* [skip ci] fix insert geojson tests to reflect new paths

* [skip ci] fix enum test for better error message

* [skip ci] fix header test for better error message

* [skip ci] fix fragment cycle test for better error message

* [skip ci] fix error message for type mismatch

* [skip ci] fix variable path in test

* [skip ci] adjust tests after bug fix

* [skip ci] more tests fixing

* Add hdb_catalog.current_setting abstraction for reading Hasura settings

As the comment in the function’s definition explains, this is needed to
work around an awkward Postgres behavior.

* [skip ci] Update CONTRIBUTING.md to mention Node setup for Python tests

* [skip ci] Add missing Python tests env var to CONTRIBUTING.md

* [skip ci] fix order of result when subscription is run with multiple nodes

* [skip ci] no-op refactor: fix a warning in Internal/Parser.hs

* [skip ci] throw error when a subscription contains remote joins

* [skip ci] Enable easier profiling by hiding AssertNF behind a flag

In order to compile a profiling build, run:

$ cabal new-build -f profiling --enable-profiling

* [skip ci] Fix two warnings

We used to lookup the objects that implement a given interface by filtering all
objects in the schema document.  However, one of the tests expects us to
generate a warning if the provided `implements` field of an introspection query
specifies an object not implementing some interface.  So we use that field
instead.

* [skip ci] Fix warnings by commenting out query plan caching

* [skip ci] improve masking/commenting query caching related code & few warning fixes

* [skip ci] Fixed compiler warnings in graphql-parser-hs

* Sync non-Haskell assets with master

* [skip ci] add a test inserting invalid GraphQL but valid JSON value in a jsonb column

* [skip ci] Avoid converting to/from Map

* [skip ci] Apply some hlint suggestions

* [skip ci] remove redundant constraints from buildLiveQueryPlan and explainGQLQuery

* [skip ci] add NOTEs about missing Tracing constraints in PDV from master

* Remove -fdefer-typed-holes, fix warnings

* Update cabal.project.freeze

* Limit GHC’s heap size to 8GB in CI to avoid the OOM killer

* Commit package-lock.json for Python tests’ remote schema server

* restrict env variables start with HASURA_GRAPHQL_ for headers configuration in actions, event triggers & remote schemas (#5519)

* restrict env variables start with HASURA_GRAPHQL_ for headers definition in actions & event triggers

* update CHANGELOG.md

* Apply suggestions from code review

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* add test for table_by_pk node when roles doesn't have permission to PK

* [skip ci] fix introspection query if any enum column present in primary key (fix #5200) (#5522)

* [skip ci] test case fix for a6450e126b

* [skip ci] add tests to agg queries when role doesn't have access to any cols

* fix backend test

* Simplify subscription execution

* [skip ci] add test to check if required headers are present while querying

* Suppose, table B is related to table A and to query B certain headers are
  necessary, then the test checks that we are throwing error when the header
  is not set when B is queried through A

* fix mutations not checking for view mutability

* [skip ci] add variable type checking and corresponding tests

* [skip ci] add test to check if update headers are present while doing an upsert

* [skip ci] add positive counterparts to some of the negative permission tests

* fix args missing their description in introspect

* [skip ci] Remove unused function; insert missing markNotReusable call

* [skip ci] Add a Note about InputValue

* [skip ci] Delete LegacySchema/ 🎉

* [skip ci] Delete GraphQL/{Resolve,Validate}/ 🎉

* [skip ci] Delete top-level Resolve/Validate modules; tidy .cabal file

* [skip ci] Delete LegacySchema top-level module

Somehow I missed this one.

* fix input value to json

* [skip ci] elaborate on JSON objects in GraphQL

* [skip ci] add missing file

* [skip ci] add a test with subscription containing remote joins

* add a test with remote joins in mutation output

* [skip ci] Add some comments to Schema/Mutation.hs

* [skip ci] Remove no longer needed code from RemoteServer.hs

* [skip ci] Use a helper function to generate conflict clause parsers

* [skip ci] fix type checker error in fields with default value

* capitalize the header keys in select_articles_without_required_headers

* Somehow, this was the reason the tests were failing. I have no idea, why!

* [skip ci] Add a long Note about optional fields and nullability

* Improve comments a bit; simplify Schema/Common.hs a bit

* [skip ci] full implementation of 5.8.5 type checking.

* [skip ci] fix validation test teardown

* [skip ci] fix schema stitching test

* fix remote schema ignoring enum nullability

* [skip ci] fix fieldOptional to not discard nullability

* revert nullability of use_spheroid

* fix comment

* add required remote fields with arguments for tests

* [skip ci] add missing docstrings

* [skip ci] fixed description of remote fields

* [skip ci] change docstring for consistency

* fix several schema inconsistencies

* revert behaviour change in function arguments parsing

* fix remaining nullability issues in new schema

* minor no-op refactor; use isListType from graphql-parser-hs

* use nullability of remote schema node, while creating a Remote reln

* fix 'ID' input coercing & action 'ID' type relationship mapping

* include ASTs in MonadExecuteQuery

* needed for PRO code-base

* Delete code for "interfaces implementing ifaces" (draft GraphQL spec)

Previously I started writing some code that adds support for a future GraphQL
feature where interfaces may themselves be sub-types of other interfaces.
However, this code was incomplete, and partially incorrect.  So this commit
deletes support for that entirely.

* Ignore a remote schema test during the upgrade/downgrade test

The PDV refactor does a better job at exposing a minimal set of types through
introspection.  In particular, not every type that is present in a remote schema
is re-exposed by Hasura.  The test
test_schema_stitching.py::TestRemoteSchemaBasic::test_introspection assumed that
all types were re-exposed, which is not required for GraphQL compatibility, in
order to test some aspect of our support for remote schemas.

So while this particular test has been updated on PDV, the PDV branch now does
not pass the old test, which we argue to be incorrect.  Hence this test is
disabled while we await a release, after which we can re-enable it.

This also re-enables a test that was previously disabled for similar, though
unrelated, reasons.

* add haddock documentation to the action's field parsers

* Deslecting some tests in server-upgrade

Some tests with current build are failing on server upgrade
which it should not. The response is more accurate than
what it was.

Also the upgrade tests were not throwing errors when the test is
expected to return an error, but succeeds. The test framework is
patched to catch this case.

* [skip ci] Add a long Note about interfaces and object types

* send the response headers back to client after running a query

* Deselect a few more tests during upgrade/downgrade test

* Update commit_diff.txt

* change log kind from db_migrate to catalog_migrate (#5531)

* Show method and complete URI in traced HTTP calls (#5525)

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* restrict env variables start with HASURA_GRAPHQL_ for headers configuration in actions, event triggers & remote schemas (#5519)

* restrict env variables start with HASURA_GRAPHQL_ for headers definition in actions & event triggers

* update CHANGELOG.md

* Apply suggestions from code review

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* fix introspection query if any enum column present in primary key (fix #5200) (#5522)

* Fix telemetry reporting of transport (websocket was reported as http)

* add log kinds in cli-migrations image (#5529)

* add log kinds in cli-migrations image

* give hint to resolve timeout error

* minor changes and CHANGELOG

* server: set hasura.tracecontext in RQL mutations [#5542] (#5555)

* server: set hasura.tracecontext in RQL mutations [#5542]

* Update test suite

Co-authored-by: Tirumarai Selvan <tiru@hasura.io>

* Add bulldozer auto-merge and -update configuration

We still need to add the github app (as of time of opening this PR)

Afterwards devs should be able to allow bulldozer to automatically
"update" the branch, merging in parent when it changes, as well as
automatically merge when all checks pass.

This is opt-in by adding the `auto-update-auto-merge` label to the PR.

* Remove 'bulldozer' config, try 'kodiak' for auto-merge

see: https://github.com/chdsbd/kodiak

The main issue that bit us was not being able to auto update forked
branches, also:
https://github.com/palantir/bulldozer/issues/66
https://github.com/palantir/bulldozer/issues/145

* Cherry-picked all commits

* [skip ci] Slightly improve formatting

* Revert "fix introspection query if any enum column present in primary key (fix #5200) (#5522)"

This reverts commit 0f9a5afa59.

This undoes a cherry-pick of 34288e1eb5 that was
already done previously in a6450e126b, and
subsequently fixed for PDV in 70e89dc250

* Do a small bit of tidying in Hasura.GraphQL.Parser.Collect

* Fix cherry-picking work

Some previous cherry-picks ended up modifying code that is commented out

* [skip ci] clarified comment regarding insert representation

* [skip ci] removed obsolete todos

* cosmetic change

* fix action error message

* [skip ci] remove obsolete comment

* [skip ci] synchronize stylish haskell extensions list

* use previously defined scalar names in parsers rather than ad-hoc literals

* Apply most syntax hlint hints.

* Clarify comment on update mutation.

* [skip ci] Clarify what fields should be specified for objects

* Update "_inc" description.

* Use record types rather than tuples fo IntrospectionResult and ParsedIntrospection

* Get rid of checkFieldNamesUnique (use Data.List.Extended.duplicates)

* Throw more errors when collecting query root names

* [skip ci] clean column parser comment

* Remove dead code inserted in ab65b39

* avoid converting to non-empty list where not needed

* add note and TODO about the disabled checks in PDV

* minor refactor in remoteField' function

* Unify two getObject methods

* Nitpicks in Remote.hs

* Update CHANGELOG.md

* Revert "Unify two getObject methods"

This reverts commit bd6bb40355.

We do need two different getObject functions as the corresponding error message is different

* Fix error message in Remote.hs

* Update CHANGELOG.md

Co-authored-by: Auke Booij <auke@tulcod.com>

* Apply suggested Changelog fix.

Co-authored-by: Auke Booij <auke@tulcod.com>

* Fix typo in Changelog.

* [skip ci] Update changelog.

* reuse type names to avoid duplication

* Fix Hashable instance for Definition

The presence of `Maybe Unique`, and an optional description, as part of
`Definition`s, means that `Definition`s that are considered `Eq`ual may get
different hashes.  This can happen, for instance, when one object is memoized
but another is not.

* [skip ci] Update commit_diff.txt

* Bump parser version.

* Bump freeze file after changes in parser.

* [skip ci] Incorporate commits from master

* Fix developer flag in server/cabal.project.freeze

Co-authored-by: Auke Booij <auke@tulcod.com>

* Deselect a changed ENUM test for upgrade/downgrade CI

* Deselect test here as well

* [skip ci] remove dead code

* Disable more tests for upgrade/downgrade

* Fix which test gets deselected

* Revert "Add hdb_catalog.current_setting abstraction for reading Hasura settings"

This reverts commit 66e85ab9fb.

* Remove circular reference in cabal.project.freeze

Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
Co-authored-by: Auke Booij <auke@hasura.io>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>
Co-authored-by: Brandon Simmons <brandon.m.simmons@gmail.com>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
Co-authored-by: Anon Ray <rayanon004@gmail.com>
Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Anon Ray <ecthiender@users.noreply.github.com>
Co-authored-by: Vamshi Surabhi <vamshi@hasura.io>
Co-authored-by: Antoine Leblanc <antoine@hasura.io>
Co-authored-by: Brandon Simmons <brandon@hasura.io>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Lyndon Maydwell <lyndon@sordina.net>
Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Naveen Naidu <naveennaidu479@gmail.com>
Co-authored-by: Karthikeyan Chinnakonda <chkarthikeyan95@gmail.com>
Co-authored-by: Nizar Malangadan <nizar-m@users.noreply.github.com>
Co-authored-by: Antoine Leblanc <crucuny@gmail.com>
Co-authored-by: Auke Booij <auke@tulcod.com>
2020-08-21 12:27:01 -05:00
Rakesh Emmadi
fd6535b861
option to reload remote schemas in 'reload metadata' (fix #3792, #4117) (#4141)
* option to reload remote schemas in 'reload_metadata' API, fix #3792, #4117

* add tests

* update changelog

* update docs api reference for 'reload_metadata'

* send reload_remote_schemas: true with the reload_metadata query

* add reload remote schemas checkbox; minor refactor

* Add a Note about cache invalidation and inconsistent metadata objects

* Small pluralization agreement tweak in docs

* Remove duplicated line in CHANGELOG

* no-op refactor

Suggested by Alexis @lexi-lambda

* Update server/src-lib/Hasura/RQL/DDL/RemoteSchema.hs

As suggested by @lexi-lambda

Co-Authored-By: Alexis King <lexi.lambda@gmail.com>

* fix tests

* requested changes

* comment 'replaceMetadataToOrdJson' unit tests

Co-authored-by: Rishichandra Wawhal <rishi@hasura.io>
Co-authored-by: Alexis King <lexi.lambda@gmail.com>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
2020-03-26 17:22:20 +05:30
Tirumarai Selvan
f8e133070b
run default tests in test_server_upgrade (#3718)
* run basic tests after upgrade

* terminate before specifying file in pytest cmd

* Move fixture definitions out of test classes

Previously we had abstract classes with the fixtures defined
in them. The test classes then inherits these super classes. This
is creating inheritence problems, especially when you want to just
inherit the tests in class, but not the fixtures. We have now moved
all those fixture definitions outside of the class (in conftest.py).
These fixtures are now used by the test classes when and where they
are required.

* Run pytests on server upgrade

Server upgrade tests are run by
  1) Run pytest with schema/metadata setup but do not do schema/metadata
teardown
  2) Upgrade the server
  3) Run pytest using the above schema and teardown at the end of the
tests
  4) Cleanup hasura metadata and start again with next set of tests

We have added options --skip-schema-setup and --skip-schema-teardown to
help running server upgrade tests.

While running the tests, we noticed that error codes and messages for
some of the tests have changed. So we have added another option to
pytest `--avoid-error-message-checks`. If this flag is set, and if
comparing expected and response message fails, and if the expected
response has an error message, Pytest will throw warnings instead of an
error.

* Use marks to specify server-upgrade tests

Not all tests can be run as serve upgrade tests, particularly those
which themselves change the schema. We introduce two pytest markers.
Marker allow_server_upgrade_test will add the test into the list of
server  upgrade  tests  that  can  be run. skip_server_upgrade_test
removes it from the list.

With this we have added tests for queries, mutations, and selected
event trigger and remote schema tests to the list of server upgrade
tests.

* Remove components not needed anymore

* Install curl

* Fix error in query validation

* Fix error in test_v1_queries.py

* install procps for server upgrade tests

* Use postgres image which has postgis installed

* set pager off with psql

* quote the bash variable WORKTREE_DIR

Co-authored-by: nizar-m <19857260+nizar-m@users.noreply.github.com>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-02-13 14:44:02 +05:30
Alexis King
3cdb3841e6 Build the GraphQL context after pruning dangling dependents
fixes #3791
2020-01-30 18:17:29 -06:00
Alexis King
1387722970 Refactor schema cache construction to avoid imperative updates
wip: fix error codes in remote schema tests
2020-01-08 16:43:06 -06:00
Brandon Simmons
91aee7fdeb Test result ordering, add --accept test mode to automatically accept changed test cases
We add a new pytest flag `--accept` that will automatically write back
yaml files with updated responses. This makes it much easier and less
error-prone to update test cases when we expect output to change, or
when authoring new tests.

Second we make sure to test that we actually preserve the order of the
selection set when returning results. This is a "SHOULD" part of the
spec but seems pretty important and something that users will rely on.

To support both of the above we use ruamel.yaml which preserves a
certain amount of formatting and comments (so that --accept can work in
a failry ergonomic way), as well as ordering (so that when we write yaml
the order of keys has meaning that's preserved during parsing).

Use ruamel.yaml everywhere for consistency (since both libraries have
different quirks).

Quirks of ruamel.yaml:
- trailing whitespace in multiline strings in yaml files isn't written
  back out as we'd like: https://bitbucket.org/ruamel/yaml/issues/47/multiline-strings-being-changed-if-they
- formatting is only sort of preserved; ruamel e.g. normalizes
  indentation. Normally the diff is pretty clean though, and you can
  always just check in portions of your test file after --accept

fixup
2019-11-05 15:15:25 -06:00
Tirumarai Selvan
98784212e2 allow configuring timeout for remote schema calls (close #2501) (#2753) 2019-08-23 14:27:19 +05:30
Anon Ray
8f1e7dbc8d breaking: server logging changes (close #507, close #2171) (#1835) 2019-07-11 05:37:06 +00:00
Rakesh Emmadi
9eb38e6c96 cache remote schema's introspection query response (fix #1679) (#2089) 2019-07-08 11:21:41 +05:30
Anon Ray
278f26b073 forward set-cookie headers from remote servers (fix #1654) (#2305) 2019-06-04 15:40:28 +05:30
Anon Ray
95a27fba89 fix response for remote schema queries over ws (fix #2246) (#2248) 2019-05-29 17:21:09 +05:30
Anon Ray
ee783e142e fix sending duplicate content-type header to remote schemas (fix #2159) (#2170) 2019-05-13 18:17:01 +05:30
Anon Ray
a21f6cd648 introduce v1/graphql (fix #1368) (#2064)
Changes compared to `/v1alpha1/graphql`

* Changed all graphql responses in **/v1/graphql** endpoint to be 200. All graphql clients expect responses to be HTTP 200. Non-200 responses are considered transport layer errors. 

* Errors in http and websocket layer are now consistent and have similar structure.
2019-05-10 11:35:10 +05:30
Anon Ray
dfaf7d3a1f fix duplicate headers being sent to remote schema (#2118) 2019-05-06 16:57:34 +05:30
Vamshi Surabhi
ce243f5899
multiplexed subscriptions (#1934)
* add types to represent unparsed http gql requests

This will help when we add caching of frequently used ASTs

* query plan caching

* move livequery to execute

* add multiplexed module

* session variable can be customised depending on the context

Previously the value was always "current_setting('hasura.user')"

* get rid of typemap requirement in reusable plan

* subscriptions are multiplexed when possible

* use lazytx for introspection to avoid acquiring a pg connection

* refactor to make execute a completely decoupled module

* don't issue a transaction for a query

* don't use current setting for explained sql

* move postgres related types to a different module

* validate variableValues on postgres before multiplexing subs

* don't user current_setting for queries over ws

* plan_cache is only visible when developer flag is enabled

* introduce 'batch size' when multiplexing subscriptions

* bump stackage to 13.16

* fix schema_stitching test case error code

* store hashes instead of actual responses for subscriptions

* internal api to dump subscriptions state

* remove PlanCache from SchemaCacheRef

* allow live query options to be configured on server startup

* capture metrics for multiplexed subscriptions

* more metrics captured for multiplexed subs

* switch to tvar based hashmap for faster snapshotting

* livequery modules do not expose internal details

* fix typo in live query env vars

* switch to hasura's pg-client-hs
2019-04-17 15:18:41 +05:30
nizar-m
a40bf10b9f run graphql tests on both http and websocket; add parallelism (close #1868) (#1921)
Examples 
1)  `
pytest --hge-urls "http://127.0.0.1:8080" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests" -vv
`
2)  `pytest --hge-urls "http://127.0.0.1:8080"   "http://127.0.0.1:8081" --pg-urls "postgresql://admin@127.0.0.1:5432/hge_tests"  "postgresql://admin@127.0.0.1:5432/hge_tests2" -vv
`
### Solution and Design
<!-- How is this issue solved/fixed? What is the design? -->
<!-- It's better if we elaborate -->
#### Reducing execution time of tests
- The Schema setup and teardown, which were earlier done per test method, usually takes around 1 sec. 
- For mutations, the model has now been changed to only do schema setup and teardown once per test class.
-  A data setup and teardown will be done once per test instead (usually takes ~10ms).
- For the test class to get this behaviour, one can can extend the class `DefaultTestMutations`. 
    - The function  `dir()` should be define which returns the location of the configuration folder.
    - Inside the configuration folder, there should be 
        - Files `<conf_dir>/schema_setup.yaml` and `<conf_dir>/schema_teardown.yaml`, which has the metadata query executed during schema setup and teardown respectively
        - Files named `<conf_dir>/values_setup.yaml` and `<conf_dir>/values_teardown.yaml`. These files are executed to setup and remove data from the tables respectively. 

#### Running Graphql queries on both http and websockets
- Each GraphQL query/mutation is run on the both HTTP and websocket protocols
- Pytests test parameterisation is used to achieve this
- The errors over websockets are slightly different from that on HTTP
   - The code takes care of converting the errors in HTTP to errors in websockets

#### Parallel executation of tests.
- The plugin pytest-xdist helps in running tests on parallel workers.
- We are using this plugin to group tests by file and run on different workers.
- Parallel test worker processes operate on separate postgres databases(and separate graphql-engines connected to these databases). Thus tests on one worker will not affect the tests on the other worker.
- With two workers, this decreases execution times by half, as the tests on event triggers usually takes a long time, but does not consume much CPU.
2019-04-08 12:52:38 +05:30
Anon Ray
d9882fcb03 fix remote queries/mutations to work over websocket (fix #1619) (#1621) 2019-03-05 16:39:02 +05:30
nizar-m
1fa66dc622 add option to disable metadata and graphql apis (close #1088) (#1650) 2019-02-28 19:23:03 +05:30
Shahidh K Muhammed
097bfb6bfa revert "forward response headers from remote servers (#1664)"
This reverts commit c19fe35f4e.
2019-02-28 17:20:56 +05:30
Anon Ray
c19fe35f4e forward response headers from remote servers (fix #1654) (#1664) 2019-02-28 17:15:07 +05:30
nizar-m
32387ba964 support union and interface types in remote schema (close #1276) (#1361) 2019-01-28 22:15:10 +05:30
Anon Ray
39bc3acffd fix conversion when merging remote schema scalars with hasura (fix #1244) (#1497) 2019-01-28 21:01:37 +05:30
nizar-m
8e3b8f51c9 Support default values (in inputvalue) from the remote schema (close #1491) (#1493) 2019-01-28 18:08:38 +05:30
Anon Ray
77cbf12bb7 merge types with same structure in remote schema (closes #1112, #1135) (#1145) 2018-12-12 17:31:18 +05:30
Anon Ray
a509a86eaa implement internal graphql server for remote schema tests (#1117)
* implement internal graphql server for remote schema tests

* add one more graphql endpoint and more tests

* add missing dependency
2018-11-26 18:38:16 +05:30