* Pass environment variables around as a data structure, via @sordina
* Resolving build error
* Adding Environment passing note to changelog
* Removing references to ILTPollerLog as this seems to have been reintroduced from a bad merge
* removing commented-out imports
* Language pragmas already set by project
* Linking async thread
* Apply suggestions from code review
Use `runQueryTx` instead of `runLazyTx` for queries.
* remove the non-user facing entry in the changelog
Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
This adds a server flag, --pg-connection-options, that can be used to set a PostgreSQL connection parameter, extra_float_digits, that needs to be used to avoid loss of data on older versions of PostgreSQL, which have odd default behavior when returning float values. (fixes#5092)
* new typeclass to abstract the logic of QueryLog-ing
* abstract the logic of logging websocket-server logs
introduce a MonadWSLog typeclass
* move catalog initialization to init step
expose a helper function to migrate catalog
create schema cache in initialiseCtx
* expose various modules and functions for pro
* generalize PGExecCtx to support specialized functions for various operations
* fix tests compilation
* allow customising PGExecCtx when starting the web server
The bulk of changes here is some shifting of code around and a little
parameterizing of functions for easier testing.
Also: comments, some renaming for clarity/less-chance-for-misue.
Store the admin secret only as a hash to prevent leaking the secret
inadvertently, and to prevent timing attacks on the secret.
NOTE: best practice for stored user passwords is a function with a
tunable cost like bcrypt, but our threat model is quite different (even
if we thought we could reasonably protect the secret from an attacker
who could read arbitrary regions of memory), and bcrypt is far too slow
(by design) to perform on each request. We'd have to rely on our
(technically savvy) users to choose high entropy passwords in any case.
Referencing #4736
* validation support for unions and interfaces
* refactor SQL generation logic for improved readability
* '/v1/relay' endpoint for relay schema
* implement 'Node' interface and top level 'node' field resolver
* add relay toggle on graphiql
* fix explain api response & index plan id with query type
* add hasura mutations to relay
* add relay pytests
* update CHANGELOG.md
Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Rishichandra Wawhal <rishi@hasura.io>
Co-authored-by: Rikin Kachhia <54616969+rikinsk@users.noreply.github.com>
When compiling the graphql-engine binary with `-O2`, ghc-8.10 seems to
be stuck at the module `Server.Init` while consuming `17G` of RAM (for 5
minutes at least before I forcefully terminated the compilation). With
this pragma, ghc-8.10 now takes under `12G` to compile graphql-engine
binary.
This also seems to squash a stubborn space leak we see with
subscriptions (linking to canonical #3388 for reference).
This may also fix some of the "Unexpected exception" websockets
exceptions we are now surfacing (see e.g. #4344)
Also: dev.sh: fix hpc reporting
Initial work on this done by Vamshi.
There are two implementations of a Cache, namely a bounded and an
unbounded variant. This can be elegantly captured in a type class.
In addition to reducing the amount of error-prone code in the
definition of the cache, this version reduces the amount of
error-prone code in usage sites of the cache, as it makes the cache
into an abstract object, so that a calling site cannot distinguish
between cache types. Any decision about what should be cached should
be made through the interface of a cache, rather than at the callsite,
and this is captured by this variant.
* move user info related code to Hasura.User module
* the RFC #4120 implementation; insert permissions with admin secret
* revert back to old RoleName based schema maps
An attempt made to avoid duplication of schema contexts in types
if any role doesn't possess any admin secret specific schema
* fix compile errors in haskell test
* keep 'user_vars' for session variables in http-logs
* no-op refacto
* tests for admin only inserts
* update docs for admin only inserts
* updated CHANGELOG.md
* default behaviour when admin secret is not set
* fix x-hasura-role to X-Hasura-Role in pytests
* introduce effective timeout in actions async tests
* update docs for admin-secret not configured case
* Update docs/graphql/manual/api-reference/schema-metadata-api/permission.rst
Co-Authored-By: Marion Schleifer <marion@hasura.io>
* Apply suggestions from code review
Co-Authored-By: Marion Schleifer <marion@hasura.io>
* a complete iteration
backend insert permissions accessable via 'x-hasura-backend-privilege'
session variable
* console changes for backend-only permissions
* provide tooltip id; update labels and tooltips;
* requested changes
* requested changes
- remove className from Toggle component
- use appropriate function name (capitalizeFirstChar -> capitalize)
* use toggle props from definitelyTyped
* fix accidental commit
* Revert "introduce effective timeout in actions async tests"
This reverts commit b7a59c19d6.
* generate complete schema for both 'default' and 'backend' sessions
* Apply suggestions from code review
Co-Authored-By: Marion Schleifer <marion@hasura.io>
* remove unnecessary import, export Toggle as is
* update session variable in tooltip
* 'x-hasura-use-backend-only-permissions' variable to switch
* update help texts
* update docs
* update docs
* update console help text
* regenerate package-lock
* serve no backend schema when backend_only: false and header set to true
- Few type name refactor as suggested by @0x777
* update CHANGELOG.md
* Update CHANGELOG.md
* Update CHANGELOG.md
* fix a merge bug where a certain entity didn't get removed
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Rishichandra Wawhal <rishi@hasura.io>
Co-authored-by: rikinsk <rikin.kachhia@gmail.com>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
* config options for internal errors for non-admin role, close#4031
More detailed action debug info is added in response 'internal' field
* add docs
* update CHANGELOG.md
* set admin graphql errors option in ci tests, minor changes to docs
* fix tests
Don't use any auth for sync actions error tests. The request body
changes based on auth type in session_variables (x-hasura-auth-mode)
* Apply suggestions from code review
Co-Authored-By: Marion Schleifer <marion@hasura.io>
* use a new sum type to represent the inclusion of internal errors
As suggested in review by @0x777
-> Move around few modules in to specific API folder
-> Saperate types from Init.hs
* fix tests
Don't use any auth for sync actions error tests. The request body
changes based on auth type in session_variables (x-hasura-auth-mode)
* move 'HttpResponse' to 'Hasura.HTTP' module
* update change log with breaking change warning
* Update CHANGELOG.md
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
The previous check was too conservative and acquired a lock on the
schema cache in situations where it was unnecessary. This change
exposes the logic run_sql uses to determine whether to use the
metadata check to make the check more precise.
* add support for action queries
* a new parameter `type` is added in the ArgumentDefinition, its value
can be either `query` or `mutation` and it defaults to the latter
* throw 400 when a query action is tried to explain
* update the actions docs to include query actions
* refactor the ToJSON and ToOrdJSON of ActionDefinition
Co-authored-by: Rishichandra Wawhal <rishi@hasura.io>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
* add new optional field `claims_namespace_path` in JWT config
* return value when empty array is found in executeJSONPath
* update the docs related to claims_namespace_path
* improve encodeJSONPath, add property tests for parseJSONPath
* throw error if both claims_namespace_path and claims_namespace are set
* refactor the Data.Parser.JsonPath to Data.Parser.JSONPathSpec
* update the JWT docs
Co-Authored-By: Marion Schleifer <marion@hasura.io>
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Tirumarai Selvan <tirumarai.selvan@gmail.com>
* type is not required for jwk_url
* remove type from JWTConfig
* Omit type field in JWTConfig serialization if jwk_url is provided
* remove type from jwk_url test suite
* add changelog
* fix docs with new format
Co-authored-by: Alexis King <lexi.lambda@gmail.com>
* add expiry time to webhook user info
This also adds an optional message to webhook errors: if we fail to
parse an expiry time, we will log a warning with the parse error.
* refactored Auth
This change had one main goal: put in common all expiry time
extraction code between the JWT and WebHook parts of the
code. Furthermore, this change also moves all WebHook specific code to
its own module, similarly to what is done for JWT.
* Remove dependency on string-conversions in favor of text-conversions
string-conversions silently uses UTF8 instead of being explicit about
it, and it uses lenientDecode when decoding ByteStrings when it’s
usually better to reject invalid UTF8 input outright. text-conversions
solves both those problems.
Co-authored-by: Alexis King <lexi.lambda@gmail.com>
* do not perform the metadata check in read-only mode
* improve the isAltrDropReplace regex
* quote the regex at compile-time to handle syntax errors statically
Co-authored-by: Alexis King <lexi.lambda@gmail.com>
* add new column "pg_version" while sending telemetry data
* make a new type for PGVersion and use serverVersion func
* define runTxIO action to run transaction(which exits on error)
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
- The metrics will include
- no of synchronous actions
- no of asynchronous actions
- no of type relationships with the output
- no of custom types defined
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* add 'ID' to default scalars for custom types, fix#4061
* preserve cookie headers from sync action webhook, close#4021
* validate action webhook response to conform to output type, fix#3977
* fix tests, don't run actions' tests on PG version < 10
* update CHANGELOG.md
* no-op refactor, use types from http-network more
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
Writing to a mutable var is a particularly potent source of leaks since
it mostly defeats GHC's analysis. Here we add assertions to all mutable
writes, and fix a couple spots where we wrote some thunks to a mutable
var (compiled with -O2).
Some of these thunks were probably benign, but others looked liked they
might be retaining big args. Didn't do much analysis, just fixed.
Actually pretty happy with how easy this was to use and as a diagnostic,
once I sorted out some issues. We should consider using it elsewhere,
and maybe extending so that we can use it with tests, enable when
`-fenable-assertsions` etc.
Relates #3388
Also simplified codepaths that use `AcceptWith`, which has unnecessary
`Maybe` fields.
* Test working through a backlog of change events
* Use a slightly more performant threaded http server in eventing pytests
This helped locally but not on CI it seems...
* Rework event processing for backpressure. Closes#3839
With loo low `HASURA_GRAPHQL_EVENTS_FETCH_INTERVAL` and/or slow webhooks
and/or too small `HASURA_GRAPHQL_EVENTS_HTTP_POOL_SIZE` we might
previously check out events from the DB faster than we can service them,
leading to space leaks, weirdness, etc.
Other changes:
- avoid fetch interval sleep latency when we previously did a non-empty
fetch
- prefetch event batch while http pool is working
- warn when it appears we can't keep up with events being generated
- make some effort to process events in creation order so we don't
starve older ones.
ALSO NOTE: HASURA_GRAPHQL_EVENTS_FETCH_INTERVAL changes semantics
slightly, since it only comes into play after an empty fetch. The old
semantics weren't documented in detail, so I think this is fine.
This is the result of a general audit of how we fork threads, with a
detour into how we're using mutable state especially in websocket
codepaths, making more robust to async exceptions and exceptions
resulting from bugs.
Some highlights:
- use a wrapper around 'immortal' so threads that die due to bugs are
restarted, and log the error
- use 'withAsync' some places
- use bracket a few places where we might break invariants
- log some codepaths that represent bugs
- export UnstructuredLog for ad hoc logging (the alternative is we
continue not logging useful stuff)
I had to timebox this. There are a few TODOs I didn't want to address.
And we'll wait until this is merged to attempt #3705 for
Control.Concurrent.Extended
* basic doc for actions
* custom_types, sync and async actions
* switch to graphql-parser-hs on github
* update docs
* metadata import/export
* webhook calls are now supported
* relationships in sync actions
* initialise.sql is now in sync with the migration file
* fix metadata tests
* allow specifying arguments of actions
* fix blacklist check on check_build_worthiness job
* track custom_types and actions related tables
* handlers are now triggered on async actions
* default to pgjson unless a field is involved in relationships, for generating definition list
* use 'true' for action filter for non admin role
* fix create_action_permission sql query
* drop permissions when dropping an action
* add a hdb_role view (and relationships) to fetch all roles in the system
* rename 'webhook' key in action definition to 'handler'
* allow templating actions wehook URLs with env vars
* add 'update_action' /v1/query type
* allow forwarding client headers by setting `forward_client_headers` in action definition
* add 'headers' configuration in action definition
* handle webhook error response based on status codes
* support array relationships for custom types
* implement single row mutation, see https://github.com/hasura/graphql-engine/issues/3731
* single row mutation: rename 'pk_columns' -> 'columns' and no-op refactor
* use top level primary key inputs for delete_by_pk & account select permissions for single row mutations
* use only REST semantics to resolve the webhook response
* use 'pk_columns' instead of 'columns' for update_by_pk input
* add python basic tests for single row mutations
* add action context (name) in webhook payload
* Async action response is accessible for non admin roles only if
the request session vars equals to action's
* clean nulls, empty arrays for actions, custom types in export metadata
* async action mutation returns only the UUID of the action
* unit tests for URL template parser
* Basic sync actions python tests
* fix output in async query & add async tests
* add admin secret header in async actions python test
* document async action architecture in Resolve/Action.hs file
* support actions returning array of objects
* tests for list type response actions
* update docs with actions and custom types metadata API reference
* update actions python tests as per #f8e1330
Co-authored-by: Tirumarai Selvan <tirumarai.selvan@gmail.com>
Co-authored-by: Aravind Shankar <face11301@gmail.com>
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
* Add downgrade command
* Add docs per @lexi-lambda's suggestions
* make tests pass
* Update hdb_version once, from Haskell
* more work based on feedback
* Improve the usage message
* Small docs changes
* Test downgrades exist for each tag
* Update downgrading.rst
* Use git-log to find tags which are ancestors of the current commit
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
We upload a set of accumulating timers and counters to track service
time for different types of operations, across several dimensions (e.g.
did we hit the plan cache, was a remote involved, etc.)
Also...
Standardize on DiffTime as a standard duration type, and try to use it
consistently.
See discussion here:
https://github.com/hasura/graphql-engine/pull/3584#pullrequestreview-340679369
It should be possible to overwrite that module so the new threadDelay
sticks per the pattern in #3705 blocked on #3558
Rename the Control.Concurrent.Extended.threadDelay to `sleep` since a
naive use with a literal argument would be very bad!
We catch a bug in 'computeTimeDiff'.
Add convenient 'Read' instances to the time unit utility types. Make
'Second' a newtype to support this.
This fixes#3759. Also, while we’re at it, also improve the way
invalidations are synced across instances so enums and remote schemas
are appropriately reloaded by the schema syncing process.
* WIP: Remove hdb_views for inserts
* Show failing row in check constraint error
* Revert "Show failing row in check constraint error"
This reverts commit dd2cac29d0.
* Use the better query plan
* Simplify things
* fix cli test
* Update downgrading.rst
* remove 1.1 asset for cli
- Move MonadBase/MonadBaseControl instances for TxE into pg-client-hs
- Set the -qn2 RTS option by default to limit the parallel GC to 2
threads
- Remove eventlog instrumentation
- Don’t rebuild the schema cache again after running a query that needs
it to be rebuilt, since we do that explicitly now.
- Remove some redundant checks, and relocate a couple others.
The connection handler in websocket transport was not using the
'UserAuthentication' interface to resolve user info. Fix resolving
user info in websocket transport to use the common
'UserAuthentication' interface
* save permissions, relationships and collections in catalog with 'is_system_defined'
* Use common stanzas in the .cabal file
* Refactor migration code into lib instead of exe
* Add new server test suite that exercises migrations
* Make graphql-engine clean succeed even if the schema does not exist
* Separate DB and metadata migrations
* Refactor Migrate.hs to generate list of migrations at compile-time
* Replace ginger with shakespeare to improve performance
* Improve migration log messages
Although brotli itself is MIT-licensed, the Haskell brotli library that provides bindings to it is GPL-licensed, so we cannot use it unless we get a response on haskell-hvr/brotli#1.
* allow customizing GraphQL root field names, close#981
* document v2 track_table API in reference
* support customising column field names in GraphQL schema
* [docs] add custom column fields doc in API reference
* add tests
* rename 'ColField' to 'ColumnField'
* embed column's graphql field in 'PGColumnInfo'
-> Value constructor of 'PGCol' is not exposed
-> Using 'parseJSON' to construct 'PGCol' in 'FromJSON' instances
* avoid using 'Maybe TableConfig'
* refactors & 'custom_column_fields' -> 'custom_column_names'
* cli-test: add configuration field in metadata export test
* update expected keys in `FromJSON` instance of `TableMeta`
* use `buildSchemaCacheFor` to update configuration in v2 track_table
* remove 'GraphQLName' type and use 'isValidName' exposed from parser lib
* point graphql-parser-hs library git repo to hasura
* support 'set_table_custom_fields' query API & added docs and tests
* use positional arguments in SQL functions
* only allow omitting set of last arguments in functions
* disallow omitting of a non default argument in functions
These changes also add a new type, PGColumnType, between PGColInfo and
PGScalarType, and they process PGRawColumnType values into PGColumnType
values during schema cache generation.
