>
### Description
>
### Changelog
- [x] `CHANGELOG.md` is updated with user-facing content relevant to this PR. If no changelog is required, then add the `no-changelog-required` label.
### Affected components
- [x] Server
### Related Issues
->
Fixes#1528
### Solution and Design
>
Only replace connection configuration instead of replacing entire metadata with empty one.
GitOrigin-RevId: f9a16dcc7b1219ec1af915bf083622fcb7dde69a
event catalog:
- `hdb_catalog` is no longer automatically created
- catalog is initialised when the first event trigger is created
- catalog initialisation is done during the schema cache build, using `ArrowCache` so it is only run in response to a change to the set of event triggers
event queue:
- `processEventQueue` thread is prevented from starting when `HASURA_GRAPHQL_EVENTS_FETCH_INTERVAL=0`
- `processEventQueue` thread only processes sources for which at least one event trigger exists in some table in the source
Co-authored-by: Anon Ray <616387+ecthiender@users.noreply.github.com>
GitOrigin-RevId: 73f256465d62490cd2b59dcd074718679993d4fe
This claws back ~7min from integration tests (run serially, as with `dev.sh test --integration`
Further improvements would do well to focus on optimizing metadata operations, as `setup` dominates
GitOrigin-RevId: 76637d6fa953c2404627c4391447a05bf09355fa
Modifying schema-sync implementation to use polling for OSS/Pro. Invalidations are now propagated via the `hdb_catalog.hdb_schema_notifications` table in OSS/Pro. Pattern followed is now a Listener/Processor split with Cloud listening for changes via a LISTEN/NOTIFY channel and OSS polling for resource version changes in the metadata table. See issue #460 for more details.
GitOrigin-RevId: 48434426df02e006f4ec328c0d5cd5b30183db25
Multi source support had limited the availability of async action queries in subscriptions. This PR
adds support for async action query subscriptions with new implementation. Also addresses https://github.com/hasura/graphql-engine/issues/6460.
GitOrigin-RevId: 5ddc321073d224f287dc4b86ce2239ff55190b36
* add `use_prepared_statements` option to add_pg_source API
* update CHANGELOG.md
* change default value for 'use_prepared_statements' to False
* update CHANGELOG.md
GitOrigin-RevId: 6f5b90991f4a8c03a51e5829d2650771bb0e29c1
While debugging issues with HLS, Reed Mullanix noticed that we don't use relative paths. This leads to problems when using HLS + Emacs due to a bug in `lsp-mode` which prevents it from finding the correct project root.
However, it is still a good practice to use relative paths in TH for other reasons, including being able to import these modules in GHCI.
This PR should make it so HLS-1.0 & emacs provide type inference, imports, etc., in all modules in our codebase.
GitOrigin-RevId: 5f53b9a7ccf46df1ea7be94ff0a5c6ec861f4ead
Fixes https://github.com/hasura/graphql-engine-mono/issues/712
Main point of interest: the `Hasura.SQL.Backend` module.
This PR creates an `Exists` type indexed by indexed type and packed constraint while hiding all of its complexity by not exporting the constructor.
Existential constructors/types which are no longer (directly) existential:
- [X] BackendSourceInfo :: BackendSourceInfo
- [x] BackendSourceMetadata :: BackendSourceMetadata
- [x] MOSourceObjId :: MetadatObjId
- [x] SOSourceObj :: SchemaObjId
- [x] RFDB :: RootField
- [x] LQP :: LiveQueryPlan
- [x] ExecutionStep :: ExecStepDB
This PR also removes ALL usages of `Typeable.cast` from our codebase. We still need to derive `Typeable` in a few places in order to be able to derive `Data` in one place. I have not dug deeper to see why this is needed.
GitOrigin-RevId: bb47e957192e4bb0af4c4116aee7bb92f7983445
Previously invalid REST endpoints would throw errors during schema cache build.
This PR changes the validation to instead add to the inconsistent metadata objects in order to allow use of `allow_inconsistent_metadata` with inconsistent REST endpoints.
All non-fatal endpoint definition errors are returned as inconsistent metadata warnings/errors depending on the use of `allow_inconsistent_metadata`. The endpoints with issues are then created and return informational runtime errors when they are called.
Console impact when creating endpoints is that error messages now refer to metadata inconsistencies rather than REST feature at the top level:
<img width="969" alt="image" src="https://user-images.githubusercontent.com/92299/110258597-8336fa00-7ff7-11eb-872c-bfca945aa0e8.png">
Note: Conflicting endpoints generate one error per conflicting set of endpoints due to the implementation of `groupInconsistentMetadataById` and `imObjectIds`. This is done to ensure that error messages are terse, but may pose errors if there are some assumptions made surrounding `imObjectIds`.
Related to https://github.com/hasura/graphql-engine-mono/pull/473 (Allow Inconsistent Metadata (v2) #473 (Merged))
---
### Kodiak commit message
Changes the validation to use inconsistent metadata objects for REST endpoint issues.
#### Commit title
Inconsistent metadata for REST endpoints
GitOrigin-RevId: b9de971208e9bb0a319c57df8dace44cb115ff66
fixes#3868
docker image - `hasura/graphql-engine:inherited-roles-preview-48b73a2de`
Note:
To be able to use the inherited roles feature, the graphql-engine should be started with the env variable `HASURA_GRAPHQL_EXPERIMENTAL_FEATURES` set to `inherited_roles`.
Introduction
------------
This PR implements the idea of multiple roles as presented in this [paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/FGALanguageICDE07.pdf). The multiple roles feature in this PR can be used via inherited roles. An inherited role is a role which can be created by combining multiple singular roles. For example, if there are two roles `author` and `editor` configured in the graphql-engine, then we can create a inherited role with the name of `combined_author_editor` role which will combine the select permissions of the `author` and `editor` roles and then make GraphQL queries using the `combined_author_editor`.
How are select permissions of different roles are combined?
------------------------------------------------------------
A select permission includes 5 things:
1. Columns accessible to the role
2. Row selection filter
3. Limit
4. Allow aggregation
5. Scalar computed fields accessible to the role
Suppose there are two roles, `role1` gives access to the `address` column with row filter `P1` and `role2` gives access to both the `address` and the `phone` column with row filter `P2` and we create a new role `combined_roles` which combines `role1` and `role2`.
Let's say the following GraphQL query is queried with the `combined_roles` role.
```graphql
query {
employees {
address
phone
}
}
```
This will translate to the following SQL query:
```sql
select
(case when (P1 or P2) then address else null end) as address,
(case when P2 then phone else null end) as phone
from employee
where (P1 or P2)
```
The other parameters of the select permission will be combined in the following manner:
1. Limit - Minimum of the limits will be the limit of the inherited role
2. Allow aggregations - If any of the role allows aggregation, then the inherited role will allow aggregation
3. Scalar computed fields - same as table column fields, as in the above example
APIs for inherited roles:
----------------------
1. `add_inherited_role`
`add_inherited_role` is the [metadata API](https://hasura.io/docs/1.0/graphql/core/api-reference/index.html#schema-metadata-api) to create a new inherited role. It accepts two arguments
`role_name`: the name of the inherited role to be added (String)
`role_set`: list of roles that need to be combined (Array of Strings)
Example:
```json
{
"type": "add_inherited_role",
"args": {
"role_name":"combined_user",
"role_set":[
"user",
"user1"
]
}
}
```
After adding the inherited role, the inherited role can be used like single roles like earlier
Note:
An inherited role can only be created with non-inherited/singular roles.
2. `drop_inherited_role`
The `drop_inherited_role` API accepts the name of the inherited role and drops it from the metadata. It accepts a single argument:
`role_name`: name of the inherited role to be dropped
Example:
```json
{
"type": "drop_inherited_role",
"args": {
"role_name":"combined_user"
}
}
```
Metadata
---------
The derived roles metadata will be included under the `experimental_features` key while exporting the metadata.
```json
{
"experimental_features": {
"derived_roles": [
{
"role_name": "manager_is_employee_too",
"role_set": [
"employee",
"manager"
]
}
]
}
}
```
Scope
------
Only postgres queries and subscriptions are supported in this PR.
Important points:
-----------------
1. All columns exposed to an inherited role will be marked as `nullable`, this is done so that cell value nullification can be done.
TODOs
-------
- [ ] Tests
- [ ] Test a GraphQL query running with a inherited role without enabling inherited roles in experimental features
- [] Tests for aggregate queries, limit, computed fields, functions, subscriptions (?)
- [ ] Introspection test with a inherited role (nullability changes in a inherited role)
- [ ] Docs
- [ ] Changelog
Co-authored-by: Vamshi Surabhi <6562944+0x777@users.noreply.github.com>
GitOrigin-RevId: 3b8ee1e11f5ceca80fe294f8c074d42fbccfec63
Added a note on existentials. I plan to create a subsequent PR with a note on how we use the singletons trick to recover the type inside an existential.
GitOrigin-RevId: 1f227d859dcc384b4ac7e103053f643f879827d1
When adding a rest endpoint that references a query with multiple definitions, throw an error.
Needed a second PR for this as Kodiak merged the prior PR before this commit was added.
---
### Kodiak commit message
Adding check for multiple query definitions for rest endpoints.
GitOrigin-RevId: 6e3977a210e29f143b61282fbb37c93eb5b9d73c
Resolves Issues:
* https://github.com/hasura/graphql-engine-mono/issues/658 - Invalid Slashes
* https://github.com/hasura/graphql-engine-mono/issues/628 - Subscriptions
Implementation:
* Moved some logic from Endpoint.hs to allow reuse of splitting url into PathSegments.
* Additional validation steps alongside checking for overlapping routes
* Logging potential misuse of GET for mutations
Future Work:
* [ ] GET is allowed for mutations (Ignore/Log warning for Now)
* [ ] Add to scInconsistentObjs rather than throwing error
* Add information to scInconsistentObjs instead of raising errors directly.
TODO:
* [x] Duplicate variable segments with the same name in the location should not be allowed
* [x] We should throw an error on trailing and leading slashes and URLs which contain empty segments
* [x] Endpoints can be created using subscriptions. But the error only shows at the time of the query
* [x] Tests
---
### Kodiak commit message
Prohibit Invalid slashes, duplicate variables, subscriptions for REST endpoints.
GitOrigin-RevId: 86c0d4af97984c8afd02699e6071e9c1658710b8
Provides queries for fetching and inserting metadata into that database that do not assume there is a `resource_version` column. This means that will work when migrating to/from older versions.
Co-authored-by: Lyndon Maydwell <92299+sordina@users.noreply.github.com>
GitOrigin-RevId: dac636d530524082c5a13ae0f016a2d4ced16f7f
Add optimistic concurrency control to the ‘replace_metadata’ call.
Prevents users from submitting out-of-date metadata to metadata-mutating APIs.
See https://github.com/hasura/graphql-engine-mono/issues/472 for details.
GitOrigin-RevId: 5f220f347a3eba288a9098b01e9913ffd7e38166
The metadata storage implementation for graphql-engine-multitenant.
- It uses a centralized PG database to store metadata of all tenants (instead of per tenant database)
- Similarly, it uses a single schema-sync listener thread per MT worker (instead of listener thread per tenant) (PS: although, the processor thread is spawned per tenant)
- 2 new flags are introduced - `--metadataDatabaseUrl` and (optional) `--metadataDatabaseRetries`
Internally, a "metadata mode" is introduced to indicate an external/managed store vs a store managed by each pro-server.
To run :
- obtain the schema file (located at `pro/server/res/cloud/metadata_db_schema.sql`)
- apply the schema on a PG database
- set the `--metadataDatabaseUrl` flag to point to the above database
- run the MT executable
The schema (and its migrations) for the metadata db is managed outside the MT worker.
### New metadata
The following is the new portion of `Metadata` added :
```yaml
version: 3
metrics_config:
analyze_query_variables: true
analyze_response_body: false
api_limits:
disabled: false
depth_limit:
global: 5
per_role:
user: 7
editor: 9
rate_limit:
per_role:
user:
unique_params:
- x-hasura-user-id
- x-hasura-team-id
max_reqs_per_min: 20
global:
unique_params: IP
max_reqs_per_min: 10
```
- In Pro, the code around fetching/updating/syncing pro-config is removed
- That also means, `hdb_pro_catalog` for keeping the config cache is not required. Hence the `hdb_pro_catalog` is also removed
- The required config comes from metadata / schema cache
### New Metadata APIs
- `set_api_limits`
- `remove_api_limits`
- `set_metrics_config`
- `remove_metrics_config`
#### `set_api_limits`
```yaml
type: set_api_limits
args:
disabled: false
depth_limit:
global: 5
per_role:
user: 7
editor: 9
rate_limit:
per_role:
anonymous:
max_reqs_per_min: 10
unique_params: "ip"
editor:
max_reqs_per_min: 30
unique_params:
- x-hasura-user-id
user:
unique_params:
- x-hasura-user-id
- x-hasura-team-id
max_reqs_per_min: 20
global:
unique_params: IP
max_reqs_per_min: 10
```
#### `remove_api_limits`
```yaml
type: remove_api_limits
args: {}
```
#### `set_metrics_config`
```yaml
type: set_metrics_config
args:
analyze_query_variables: true
analyze_response_body: false
```
#### `remove_metrics_config`
```yaml
type: remove_metrics_config
args: {}
```
#### TODO
- [x] on-prem pro implementation for `MonadMetadataStorage`
- [x] move the project config from Lux to pro metadata (PR: #379)
- [ ] console changes for pro config/api limits, subscription workers (cc @soorajshankar @beerose)
- [x] address other minor TODOs
- [x] TxIso for `MonadSourceResolver`
- [x] enable EKG connection pool metrics
- [x] add logging of connection info when sources are added?
- [x] confirm if the `buildReason` for schema cache is correct
- [ ] testing
- [x] 1.3 -> 1.4 cloud migration script (#465; PR: #508)
- [x] one-time migration of existing metadata from users' db to centralized PG
- [x] one-time migration of pro project config + api limits + regression tests from metrics API to metadata
- [ ] integrate with infra team (WIP - cc @hgiasac)
- [x] benchmark with 1000+ tenants + each tenant making read/update metadata query every second (PR: https://github.com/hasura/graphql-engine-mono/pull/411)
- [ ] benchmark with few tenants having large metadata (100+ tables etc.)
- [ ] when user moves regions (https://github.com/hasura/lux/issues/1717)
- [ ] metadata has to be migrated from one regional PG to another
- [ ] migrate metrics data as well ?
- [ ] operation logs
- [ ] regression test runs
- [ ] find a way to share the schema files with the infra team
Co-authored-by: Naveen Naidu <30195193+Naveenaidu@users.noreply.github.com>
GitOrigin-RevId: 39e8361f2c0e96e0f9e8f8fb45e6cc14857f31f1
This PR generalizes a bunch of metadata structures.
Most importantly, it changes `SourceCache` to hold existentially quantified values:
```
data BackendSourceInfo =
forall b. Backend b => BackendSourceInfo (SourceInfo b)
type SourceCache = HashMap SourceName BackendSourceInfo
```
This changes a *lot* of things throughout the code. For now, all code using the schema cache explicitly casts sources to Postgres, meaning that if any non-Postgres `SourceInfo` makes it to the cache, it'll be ignored.
That means that after this PR is submitted, we can split work between two different aspects:
- creating `SourceInfo` for other backends
- handling those other sources down the line
GitOrigin-RevId: fb9ea00f32e840fc33c5467896fb1dfa5283ab42
Fixes https://github.com/hasura/graphql-engine/issues/5704 by checking, for aggregate fields whether we are handling a numeric aggregation.
This PR also adds type information to `ColFld` such that we know the type of the field.
This is the second attempt. See #319 for a less invasive approach. @nicuveo suggested type information might be useful, and since it wasn't hard to add, I think this version is better as well.
GitOrigin-RevId: aa6a259fd5debe9466df6302839ddbbd0ea659b5
This PR is a combination of the following other PRs:
- #169: move HasHttpManager out of RQL.Types
- #170: move UserInfoM to Hasura.Session
- #179: delete dead code from RQL.Types
- #180: move event related code to EventTrigger
GitOrigin-RevId: d97608d7945f2c7a0a37e307369983653eb62eb1
Earlier, while creating the event trigger's internal postgres trigger, we used to get the name of the table from the `TG_TABLE_NAME` special trigger variable. Using this with normal tables works fine, but it breaks when the parent table is partitioned because we associate the ET configuration in the schema only with the original table (as it should be).
In this PR, we supply the table name and schema name through template variables instead of using `TG_TABLE_NAME` and `TG_TABLE_SCHEMA`, so that event triggers work with a partitioned table as well.
TODO:
- [x] Changelog
- [x] unit test (ET on partition table)
GitOrigin-RevId: 556376881a85525300dcf64da0611ee9ad387eb0
This is an incremental PR towards https://github.com/hasura/graphql-engine/pull/5797
Co-authored-by: Anon Ray <ecthiender@users.noreply.github.com>
GitOrigin-RevId: a6cb8c239b2ff840a0095e78845f682af0e588a9
* server: fix handling of empty array values in relationships of set_custom_types
* add a test for dropping the relationship
GitOrigin-RevId: abff138b0021647a1cb6c6c041c2ba53c1ff9b77
Generalize TableCoreInfoRM, TableCoreCacheRT, some table metadata data types, generalize fromPGCol to fromCol, generalize some schema cache functions, prepare some enum schema cache code for generalization
GitOrigin-RevId: a65112bc1688e00fd707d27af087cb2585961da2
An incremental PR towards https://github.com/hasura/graphql-engine/pull/5797
- Expands `MonadMetadataStorage` with operations related to async actions and setting/updating metadata
GitOrigin-RevId: 53386b7b2d007e162050b826d0708897f0b4c8f6
This PR makes a bunch of schema generation code in Hasura.GraphQL.Schema backend-agnostic, by moving the backend-specific parts into a new BackendSchema type class. This way, the schema generation code can be reused for other backends, simply by implementing new instances of the BackendSchema type class.
This work is now in a state where the schema generators are sufficiently generic to accept the implementation of a new backend. That means that we can start exposing MS SQL schema. Execution is not implemented yet, of course.
The branch currently does not support computed fields or Relay. This is, in a sense, intentional: computed field support is normally baked into the schema generation (through the fieldSelection schema generator), and so this branch shows a programming technique that allows us to expose certain GraphQL schema depending on backend support. We can write support for computed fields and Relay at a later stage.
Co-authored-by: Antoine Leblanc <antoine@hasura.io>
GitOrigin-RevId: df369fc3d189cbda1b931d31678e9450a6601314
* add support for joining to remote interface and union fields
* add test for making a remote relationship with an Union type
* add CHANGELOG
* remove the pretty-simple library
* remove unused imports from Remote.hs
* Apply suggestions from code review
Co-authored-by: Brandon Simmons <brandon.m.simmons@gmail.com>
* support remote relationships against enum fields as well
* update CHANGELOG
Co-authored-by: Brandon Simmons <brandon.m.simmons@gmail.com>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
Add a backend type extension parameter to some RQL types, following the ideas of the paper "Trees that grow" (Najd & Jones 2016)
Co-authored-by: Antoine Leblanc <antoine@hasura.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* [skip ci] use the args while making the fieldParser
* modify the execution part of the remote queries
* parse union queries deeply
* add test for remote schema field validation
* add tests for validating remote query arguments
Co-authored-by: Auke Booij <auke@hasura.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
* improve jsonpath parser to accept special characters and property tests for the same
* make the JWTClaimsMapValueG parametrizable
* add documentation in the JWT file
* modify processAuthZHeader
Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
Co-authored-by: Marion Schleifer <marion@hasura.io>
Aka “the PDV refactor.” History is preserved on the branch 2801-graphql-schema-parser-refactor.
* [skip ci] remove stale benchmark commit from commit_diff
* [skip ci] Check for root field name conflicts between remotes
* [skip ci] Additionally check for conflicts between remotes and DB
* [skip ci] Check for conflicts in schema when tracking a table
* [skip ci] Fix equality checking in GraphQL AST
* server: fix mishandling of GeoJSON inputs in subscriptions (fix#3239) (#4551)
* Add support for multiple top-level fields in a subscription to improve testability of subscriptions
* Add an internal flag to enable multiple subscriptions
* Add missing call to withConstructorFn in live queries (fix#3239)
Co-authored-by: Alexis King <lexi.lambda@gmail.com>
* Scheduled triggers (close#1914) (#3553)
server: add scheduled triggers
Co-authored-by: Alexis King <lexi.lambda@gmail.com>
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>
* dev.sh: bump version due to addition of croniter python dependency
* server: fix an introspection query caching issue (fix#4547) (#4661)
Introspection queries accept variables, but we need to make sure to
also touch the variables that we ignore, so that an introspection
query is marked not reusable if we are not able to build a correct
query plan for it.
A better solution here would be to deal with such unused variables
correctly, so that more introspection queries become reusable.
An even better solution would be to type-safely track *how* to reuse
which variables, rather than to split the reusage marking from the
planning.
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
* flush log buffer on exception in mkWaiApp ( fix#4772 ) (#4801)
* flush log buffer on exception in mkWaiApp
* add comment to explain the introduced change
* add changelog
* allow logging details of a live query polling thread (#4959)
* changes for poller-log
add various multiplexed query info in poller-log
* minor cleanup, also fixes a bug which will return duplicate data
* Live query poller stats can now be logged
This also removes in-memory stats that are collected about batched
query execution as the log lines when piped into an monitoring tool
will give us better insights.
* allow poller-log to be configurable
* log minimal information in the livequery-poller-log
Other information can be retrieved from /dev/subscriptions/extended
* fix few review comments
* avoid marshalling and unmarshalling from ByteString to EncJSON
* separate out SubscriberId and SubscriberMetadata
Co-authored-by: Anon Ray <rayanon004@gmail.com>
* Don't compile in developer APIs by default
* Tighten up handling of admin secret, more docs
Store the admin secret only as a hash to prevent leaking the secret
inadvertently, and to prevent timing attacks on the secret.
NOTE: best practice for stored user passwords is a function with a
tunable cost like bcrypt, but our threat model is quite different (even
if we thought we could reasonably protect the secret from an attacker
who could read arbitrary regions of memory), and bcrypt is far too slow
(by design) to perform on each request. We'd have to rely on our
(technically savvy) users to choose high entropy passwords in any case.
Referencing #4736
* server/docs: add instructions to fix loss of float precision in PostgreSQL <= 11 (#5187)
This adds a server flag, --pg-connection-options, that can be used to set a PostgreSQL connection parameter, extra_float_digits, that needs to be used to avoid loss of data on older versions of PostgreSQL, which have odd default behavior when returning float values. (fixes#5092)
* [skip ci] Add new commits from master to the commit diff
* [skip ci] serve default directives (skip & include) over introspection
* [skip ci] Update non-Haskell assets with the version on master
* server: refactor GQL execution check and config API (#5094)
Co-authored-by: Vamshi Surabhi <vamshi@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* [skip ci] fix js issues in tests by pinning dependencies version
* [skip ci] bump graphql version
* [skip ci] Add note about memory usage
* generalize query execution logic on Postgres (#5110)
* generalize PGExecCtx to support specialized functions for various operations
* fix tests compilation
* allow customising PGExecCtx when starting the web server
* server: changes catalog initialization and logging for pro customization (#5139)
* new typeclass to abstract the logic of QueryLog-ing
* abstract the logic of logging websocket-server logs
introduce a MonadWSLog typeclass
* move catalog initialization to init step
expose a helper function to migrate catalog
create schema cache in initialiseCtx
* expose various modules and functions for pro
* [skip ci] cosmetic change
* [skip ci] fix test calling a mutation that does not exist
* [skip ci] minor text change
* [skip ci] refactored input values
* [skip ci] remove VString Origin
* server: fix updating of headers behaviour in the update cron trigger API and create future events immediately (#5151)
* server: fix bug to update headers in an existing cron trigger and create future events
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
* Lower stack chunk size in RTS to reduce thread STACK memory (closes#5190)
This reduces memory consumption for new idle subscriptions significantly
(see linked ticket).
The hypothesis is: we fork a lot of threads per websocket, and some of
these use slightly more than the initial 1K stack size, so the first
overflow balloons to 32K, when significantly less is required.
However: running with `+RTS -K1K -xc` did not seem to show evidence of
any overflows! So it's a mystery why this improves things.
GHC should probably also be doubling the stack buffer at each overflow
or doing something even smarter; the knobs we have aren't so helpful.
* [skip ci] fix todo and schema generation for aggregate fields
* 5087 libpq pool leak (#5089)
Shrink libpq buffers to 1MB before returning connection to pool. Closes#5087
See: https://github.com/hasura/pg-client-hs/pull/19
Also related: #3388#4077
* bump pg-client-hs version (fixes a build issue on some environments) (#5267)
* do not use prepared statements for mutations
* server: unlock scheduled events on graceful shutdown (#4928)
* Fix buggy parsing of new --conn-lifetime flag in 2b0e3774
* [skip ci] remove cherry-picked commit from commit_diff.txt
* server: include additional fields in scheduled trigger webhook payload (#5262)
* include scheduled triggers metadata in the webhook body
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
* server: call the webhook asynchronously in event triggers (#5352)
* server: call the webhook asynchronosly in event triggers
* Expose all modules in Cabal file (#5371)
* [skip ci] update commit_diff.txt
* [skip ci] fix cast exp parser & few TODOs
* [skip ci] fix remote fields arguments
* [skip ci] fix few more TODO, no-op refactor, move resolve/action.hs to execute/action.hs
* Pass environment variables around as a data structure, via @sordina (#5374)
* Pass environment variables around as a data structure, via @sordina
* Resolving build error
* Adding Environment passing note to changelog
* Removing references to ILTPollerLog as this seems to have been reintroduced from a bad merge
* removing commented-out imports
* Language pragmas already set by project
* Linking async thread
* Apply suggestions from code review
Use `runQueryTx` instead of `runLazyTx` for queries.
* remove the non-user facing entry in the changelog
Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* [skip ci] fix: restrict remote relationship field generation for hasura queries
* [skip ci] no-op refactor; move insert execution code from schema parser module
* server: call the webhook asynchronously in event triggers (#5352)
* server: call the webhook asynchronosly in event triggers
* Expose all modules in Cabal file (#5371)
* [skip ci] update commit_diff.txt
* Pass environment variables around as a data structure, via @sordina (#5374)
* Pass environment variables around as a data structure, via @sordina
* Resolving build error
* Adding Environment passing note to changelog
* Removing references to ILTPollerLog as this seems to have been reintroduced from a bad merge
* removing commented-out imports
* Language pragmas already set by project
* Linking async thread
* Apply suggestions from code review
Use `runQueryTx` instead of `runLazyTx` for queries.
* remove the non-user facing entry in the changelog
Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* [skip ci] implement header checking
Probably closes#14 and #3659.
* server: refactor 'pollQuery' to have a hook to process 'PollDetails' (#5391)
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* update pg-client (#5421)
* [skip ci] update commit_diff
* Fix latency buckets for telemetry data
These must have gotten messed up during a refactor. As a consequence
almost all samples received so far fall into the single erroneous 0 to
1K seconds (originally supposed to be 1ms?) bucket.
I also re-thought what the numbers should be, but these are still
arbitrary and might want adjusting in the future.
* [skip ci] include the latest commit compared against master in commit_diff
* [skip ci] include new commits from master in commit_diff
* [skip ci] improve description generation
* [skip ci] sort all introspect arrays
* [skip ci] allow parsers to specify error codes
* [skip ci] fix integer and float parsing error code
* [skip ci] scalar from json errors are now parse errors
* [skip ci] fixed negative integer error message and code
* [skip ci] Re-fix nullability in relationships
* [skip ci] no-op refactor and removed couple of FIXMEs
* [skip ci] uncomment code in 'deleteMetadataObject'
* [skip ci] Fix re-fix of nullability for relationships
* [skip ci] fix default arguments error code
* [skip ci] updated test error message
!!! WARNING !!!
Since all fields accept `null`, they all are technically optional in
the new schema. Meaning there's no such thing as a missing mandatory
field anymore: a field that doesn't have a default value, and which
therefore isn't labelled as "optional" in the schema, will be assumed
to be null if it's missing, meaning it isn't possible anymore to have
an error for a missing mandatory field. The only possible error is now
when a optional positional argument is omitted but is not the last
positional argument.
* [skip ci] cleanup of int scalar parser
* [skip ci] retro-compatibility of offset as string
* [skip ci] Remove commit from commit_diff.txt
Although strictly speaking we don't know if this will work correctly in PDV
if we would implement query plan caching, the fact is that in the theoretical
case that we would have the same issue in PDV, it would probably apply not just
to introspection, and the fix would be written completely differently. So this
old commit is of no value to us other than the heads-up "make sure query plan
caching works correctly even in the presence of unused variables", which is
already part of the test suite.
* Add MonadTrace and MonadExecuteQuery abstractions (#5383)
* [skip ci] Fix accumulation of input object types
Just like object types, interface types, and union types, we have to avoid
circularities when collecting input types from the GraphQL AST.
Additionally, this fixes equality checks for input object types (whose fields
are unordered, and hence should be compared as sets) and enum types (ditto).
* [skip ci] fix fragment error path
* [skip ci] fix node error code
* [skip ci] fix paths in insert queries
* [skip ci] fix path in objects
* [skip ci] manually alter node id path for consistency
* [skip ci] more node error fixups
* [skip ci] one last relay error message fix
* [skip ci] update commit_diff
* Propagate the trace context to event triggers (#5409)
* Propagate the trace context to event triggers
* Handle missing trace and span IDs
* Store trace context as one LOCAL
* Add migrations
* Documentation
* changelog
* Fix warnings
* Respond to code review suggestions
* Respond to code review
* Undo changelog
* Update CHANGELOG.md
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* server: log request/response sizes for event triggers (#5463)
* server: log request/response sizes for event triggers
event triggers (and scheduled triggers) now have request/response size
in their logs.
* add changelog entry
* Tracing: Simplify HTTP traced request (#5451)
Remove the Inversion of Control (SuspendRequest) and simplify
the tracing of HTTP Requests.
Co-authored-by: Phil Freeman <phil@hasura.io>
* Attach request ID as tracing metadata (#5456)
* Propagate the trace context to event triggers
* Handle missing trace and span IDs
* Store trace context as one LOCAL
* Add migrations
* Documentation
* Include the request ID as trace metadata
* changelog
* Fix warnings
* Respond to code review suggestions
* Respond to code review
* Undo changelog
* Update CHANGELOG.md
* Typo
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* server: add logging for action handlers (#5471)
* server: add logging for action handlers
* add changelog entry
* change action-handler log type from internal to non-internal
* fix action-handler-log name
* server: pass http and websocket request to logging context (#5470)
* pass request body to logging context in all cases
* add message size logging on the websocket API
this is required by graphql-engine-pro/#416
* message size logging on websocket API
As we need to log all messages recieved/sent by the websocket server,
it makes sense to log them as part of the websocket server event logs.
Previously message recieved were logged inside the onMessage handler,
and messages sent were logged only for "data" messages (as a server event log)
* fix review comments
Co-authored-by: Phil Freeman <phil@hasura.io>
* server: stop eventing subsystem threads when shutting down (#5479)
* server: stop eventing subsystem threads when shutting down
* Apply suggestions from code review
Co-authored-by: Karthikeyan Chinnakonda <chkarthikeyan95@gmail.com>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Karthikeyan Chinnakonda <chkarthikeyan95@gmail.com>
* [skip ci] update commit_diff with new commits added in master
* Bugfix to support 0-size HASURA_GRAPHQL_QUERY_PLAN_CACHE_SIZE
Also some minor refactoring of bounded cache module:
- the maxBound check in `trim` was confusing and unnecessary
- consequently trim was unnecessary for lookupPure
Also add some basic tests
* Support only the bounded cache, with default HASURA_GRAPHQL_QUERY_PLAN_CACHE_SIZE of 4000. Closes#5363
* [skip ci] remove merge commit from commit_diff
* server: Fix compiler warning caused by GHC upgrade (#5489)
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* [skip ci] update all non server code from master
* [skip ci] aligned object field error message with master
* [skip ci] fix remaining undefined?
* [skip ci] remove unused import
* [skip ci] revert to previous error message, fix tests
* Move nullableType/nonNullableType to Schema.hs
These are functions on Types, not on Parsers.
* [skip ci] fix setup to fix backend only test
the order in which permission checks are performed on the branch is
slightly different than on master, resulting in a slightly different
error if there are no other mutations the user has access to. By
adding update permissions, we go back to the expected case.
* [skip ci] fix insert geojson tests to reflect new paths
* [skip ci] fix enum test for better error message
* [skip ci] fix header test for better error message
* [skip ci] fix fragment cycle test for better error message
* [skip ci] fix error message for type mismatch
* [skip ci] fix variable path in test
* [skip ci] adjust tests after bug fix
* [skip ci] more tests fixing
* Add hdb_catalog.current_setting abstraction for reading Hasura settings
As the comment in the function’s definition explains, this is needed to
work around an awkward Postgres behavior.
* [skip ci] Update CONTRIBUTING.md to mention Node setup for Python tests
* [skip ci] Add missing Python tests env var to CONTRIBUTING.md
* [skip ci] fix order of result when subscription is run with multiple nodes
* [skip ci] no-op refactor: fix a warning in Internal/Parser.hs
* [skip ci] throw error when a subscription contains remote joins
* [skip ci] Enable easier profiling by hiding AssertNF behind a flag
In order to compile a profiling build, run:
$ cabal new-build -f profiling --enable-profiling
* [skip ci] Fix two warnings
We used to lookup the objects that implement a given interface by filtering all
objects in the schema document. However, one of the tests expects us to
generate a warning if the provided `implements` field of an introspection query
specifies an object not implementing some interface. So we use that field
instead.
* [skip ci] Fix warnings by commenting out query plan caching
* [skip ci] improve masking/commenting query caching related code & few warning fixes
* [skip ci] Fixed compiler warnings in graphql-parser-hs
* Sync non-Haskell assets with master
* [skip ci] add a test inserting invalid GraphQL but valid JSON value in a jsonb column
* [skip ci] Avoid converting to/from Map
* [skip ci] Apply some hlint suggestions
* [skip ci] remove redundant constraints from buildLiveQueryPlan and explainGQLQuery
* [skip ci] add NOTEs about missing Tracing constraints in PDV from master
* Remove -fdefer-typed-holes, fix warnings
* Update cabal.project.freeze
* Limit GHC’s heap size to 8GB in CI to avoid the OOM killer
* Commit package-lock.json for Python tests’ remote schema server
* restrict env variables start with HASURA_GRAPHQL_ for headers configuration in actions, event triggers & remote schemas (#5519)
* restrict env variables start with HASURA_GRAPHQL_ for headers definition in actions & event triggers
* update CHANGELOG.md
* Apply suggestions from code review
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* add test for table_by_pk node when roles doesn't have permission to PK
* [skip ci] fix introspection query if any enum column present in primary key (fix#5200) (#5522)
* [skip ci] test case fix for a6450e126b
* [skip ci] add tests to agg queries when role doesn't have access to any cols
* fix backend test
* Simplify subscription execution
* [skip ci] add test to check if required headers are present while querying
* Suppose, table B is related to table A and to query B certain headers are
necessary, then the test checks that we are throwing error when the header
is not set when B is queried through A
* fix mutations not checking for view mutability
* [skip ci] add variable type checking and corresponding tests
* [skip ci] add test to check if update headers are present while doing an upsert
* [skip ci] add positive counterparts to some of the negative permission tests
* fix args missing their description in introspect
* [skip ci] Remove unused function; insert missing markNotReusable call
* [skip ci] Add a Note about InputValue
* [skip ci] Delete LegacySchema/ 🎉
* [skip ci] Delete GraphQL/{Resolve,Validate}/ 🎉
* [skip ci] Delete top-level Resolve/Validate modules; tidy .cabal file
* [skip ci] Delete LegacySchema top-level module
Somehow I missed this one.
* fix input value to json
* [skip ci] elaborate on JSON objects in GraphQL
* [skip ci] add missing file
* [skip ci] add a test with subscription containing remote joins
* add a test with remote joins in mutation output
* [skip ci] Add some comments to Schema/Mutation.hs
* [skip ci] Remove no longer needed code from RemoteServer.hs
* [skip ci] Use a helper function to generate conflict clause parsers
* [skip ci] fix type checker error in fields with default value
* capitalize the header keys in select_articles_without_required_headers
* Somehow, this was the reason the tests were failing. I have no idea, why!
* [skip ci] Add a long Note about optional fields and nullability
* Improve comments a bit; simplify Schema/Common.hs a bit
* [skip ci] full implementation of 5.8.5 type checking.
* [skip ci] fix validation test teardown
* [skip ci] fix schema stitching test
* fix remote schema ignoring enum nullability
* [skip ci] fix fieldOptional to not discard nullability
* revert nullability of use_spheroid
* fix comment
* add required remote fields with arguments for tests
* [skip ci] add missing docstrings
* [skip ci] fixed description of remote fields
* [skip ci] change docstring for consistency
* fix several schema inconsistencies
* revert behaviour change in function arguments parsing
* fix remaining nullability issues in new schema
* minor no-op refactor; use isListType from graphql-parser-hs
* use nullability of remote schema node, while creating a Remote reln
* fix 'ID' input coercing & action 'ID' type relationship mapping
* include ASTs in MonadExecuteQuery
* needed for PRO code-base
* Delete code for "interfaces implementing ifaces" (draft GraphQL spec)
Previously I started writing some code that adds support for a future GraphQL
feature where interfaces may themselves be sub-types of other interfaces.
However, this code was incomplete, and partially incorrect. So this commit
deletes support for that entirely.
* Ignore a remote schema test during the upgrade/downgrade test
The PDV refactor does a better job at exposing a minimal set of types through
introspection. In particular, not every type that is present in a remote schema
is re-exposed by Hasura. The test
test_schema_stitching.py::TestRemoteSchemaBasic::test_introspection assumed that
all types were re-exposed, which is not required for GraphQL compatibility, in
order to test some aspect of our support for remote schemas.
So while this particular test has been updated on PDV, the PDV branch now does
not pass the old test, which we argue to be incorrect. Hence this test is
disabled while we await a release, after which we can re-enable it.
This also re-enables a test that was previously disabled for similar, though
unrelated, reasons.
* add haddock documentation to the action's field parsers
* Deslecting some tests in server-upgrade
Some tests with current build are failing on server upgrade
which it should not. The response is more accurate than
what it was.
Also the upgrade tests were not throwing errors when the test is
expected to return an error, but succeeds. The test framework is
patched to catch this case.
* [skip ci] Add a long Note about interfaces and object types
* send the response headers back to client after running a query
* Deselect a few more tests during upgrade/downgrade test
* Update commit_diff.txt
* change log kind from db_migrate to catalog_migrate (#5531)
* Show method and complete URI in traced HTTP calls (#5525)
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* restrict env variables start with HASURA_GRAPHQL_ for headers configuration in actions, event triggers & remote schemas (#5519)
* restrict env variables start with HASURA_GRAPHQL_ for headers definition in actions & event triggers
* update CHANGELOG.md
* Apply suggestions from code review
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* fix introspection query if any enum column present in primary key (fix#5200) (#5522)
* Fix telemetry reporting of transport (websocket was reported as http)
* add log kinds in cli-migrations image (#5529)
* add log kinds in cli-migrations image
* give hint to resolve timeout error
* minor changes and CHANGELOG
* server: set hasura.tracecontext in RQL mutations [#5542] (#5555)
* server: set hasura.tracecontext in RQL mutations [#5542]
* Update test suite
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
* Add bulldozer auto-merge and -update configuration
We still need to add the github app (as of time of opening this PR)
Afterwards devs should be able to allow bulldozer to automatically
"update" the branch, merging in parent when it changes, as well as
automatically merge when all checks pass.
This is opt-in by adding the `auto-update-auto-merge` label to the PR.
* Remove 'bulldozer' config, try 'kodiak' for auto-merge
see: https://github.com/chdsbd/kodiak
The main issue that bit us was not being able to auto update forked
branches, also:
https://github.com/palantir/bulldozer/issues/66https://github.com/palantir/bulldozer/issues/145
* Cherry-picked all commits
* [skip ci] Slightly improve formatting
* Revert "fix introspection query if any enum column present in primary key (fix#5200) (#5522)"
This reverts commit 0f9a5afa59.
This undoes a cherry-pick of 34288e1eb5 that was
already done previously in a6450e126b, and
subsequently fixed for PDV in 70e89dc250
* Do a small bit of tidying in Hasura.GraphQL.Parser.Collect
* Fix cherry-picking work
Some previous cherry-picks ended up modifying code that is commented out
* [skip ci] clarified comment regarding insert representation
* [skip ci] removed obsolete todos
* cosmetic change
* fix action error message
* [skip ci] remove obsolete comment
* [skip ci] synchronize stylish haskell extensions list
* use previously defined scalar names in parsers rather than ad-hoc literals
* Apply most syntax hlint hints.
* Clarify comment on update mutation.
* [skip ci] Clarify what fields should be specified for objects
* Update "_inc" description.
* Use record types rather than tuples fo IntrospectionResult and ParsedIntrospection
* Get rid of checkFieldNamesUnique (use Data.List.Extended.duplicates)
* Throw more errors when collecting query root names
* [skip ci] clean column parser comment
* Remove dead code inserted in ab65b39
* avoid converting to non-empty list where not needed
* add note and TODO about the disabled checks in PDV
* minor refactor in remoteField' function
* Unify two getObject methods
* Nitpicks in Remote.hs
* Update CHANGELOG.md
* Revert "Unify two getObject methods"
This reverts commit bd6bb40355.
We do need two different getObject functions as the corresponding error message is different
* Fix error message in Remote.hs
* Update CHANGELOG.md
Co-authored-by: Auke Booij <auke@tulcod.com>
* Apply suggested Changelog fix.
Co-authored-by: Auke Booij <auke@tulcod.com>
* Fix typo in Changelog.
* [skip ci] Update changelog.
* reuse type names to avoid duplication
* Fix Hashable instance for Definition
The presence of `Maybe Unique`, and an optional description, as part of
`Definition`s, means that `Definition`s that are considered `Eq`ual may get
different hashes. This can happen, for instance, when one object is memoized
but another is not.
* [skip ci] Update commit_diff.txt
* Bump parser version.
* Bump freeze file after changes in parser.
* [skip ci] Incorporate commits from master
* Fix developer flag in server/cabal.project.freeze
Co-authored-by: Auke Booij <auke@tulcod.com>
* Deselect a changed ENUM test for upgrade/downgrade CI
* Deselect test here as well
* [skip ci] remove dead code
* Disable more tests for upgrade/downgrade
* Fix which test gets deselected
* Revert "Add hdb_catalog.current_setting abstraction for reading Hasura settings"
This reverts commit 66e85ab9fb.
* Remove circular reference in cabal.project.freeze
Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
Co-authored-by: Auke Booij <auke@hasura.io>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>
Co-authored-by: Brandon Simmons <brandon.m.simmons@gmail.com>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
Co-authored-by: Anon Ray <rayanon004@gmail.com>
Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Anon Ray <ecthiender@users.noreply.github.com>
Co-authored-by: Vamshi Surabhi <vamshi@hasura.io>
Co-authored-by: Antoine Leblanc <antoine@hasura.io>
Co-authored-by: Brandon Simmons <brandon@hasura.io>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Lyndon Maydwell <lyndon@sordina.net>
Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Naveen Naidu <naveennaidu479@gmail.com>
Co-authored-by: Karthikeyan Chinnakonda <chkarthikeyan95@gmail.com>
Co-authored-by: Nizar Malangadan <nizar-m@users.noreply.github.com>
Co-authored-by: Antoine Leblanc <crucuny@gmail.com>
Co-authored-by: Auke Booij <auke@tulcod.com>
* Pass environment variables around as a data structure, via @sordina
* Resolving build error
* Adding Environment passing note to changelog
* Removing references to ILTPollerLog as this seems to have been reintroduced from a bad merge
* removing commented-out imports
* Language pragmas already set by project
* Linking async thread
* Apply suggestions from code review
Use `runQueryTx` instead of `runLazyTx` for queries.
* remove the non-user facing entry in the changelog
Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* fix relay introspection failing if any views exist, fix#5020
* reduce base64 encoded node id length, close#5037
* make node field type non-nullable in an edge
* more relay tests with permissions & complete restructure of test yaml files
Co-authored-by: Aravind <aravindkp@outlook.in>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* validation support for unions and interfaces
* refactor SQL generation logic for improved readability
* '/v1/relay' endpoint for relay schema
* implement 'Node' interface and top level 'node' field resolver
* add relay toggle on graphiql
* fix explain api response & index plan id with query type
* add hasura mutations to relay
* add relay pytests
* update CHANGELOG.md
Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Rishichandra Wawhal <rishi@hasura.io>
Co-authored-by: Rikin Kachhia <54616969+rikinsk@users.noreply.github.com>
* changes for poller-log
add various multiplexed query info in poller-log
* minor cleanup, also fixes a bug which will return duplicate data
* Live query poller stats can now be logged
This also removes in-memory stats that are collected about batched
query execution as the log lines when piped into an monitoring tool
will give us better insights.
* allow poller-log to be configurable
* log minimal information in the livequery-poller-log
Other information can be retrieved from /dev/subscriptions/extended
* fix few review comments
* avoid marshalling and unmarshalling from ByteString to EncJSON
* separate out SubscriberId and SubscriberMetadata
Co-authored-by: Anon Ray <rayanon004@gmail.com>
This also seems to squash a stubborn space leak we see with
subscriptions (linking to canonical #3388 for reference).
This may also fix some of the "Unexpected exception" websockets
exceptions we are now surfacing (see e.g. #4344)
Also: dev.sh: fix hpc reporting
Initial work on this done by Vamshi.
* Add support for multiple top-level fields in a subscription to improve testability of subscriptions
* Add an internal flag to enable multiple subscriptions
* Add missing call to withConstructorFn in live queries (fix#3239)
Co-authored-by: Alexis King <lexi.lambda@gmail.com>
* Allow computed fields to have access to Hasura's session variables
* Inform about session args for computed fields in changelog and docs
* Add tests for session arguments for computed fields (and the respective errors)
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
* move user info related code to Hasura.User module
* the RFC #4120 implementation; insert permissions with admin secret
* revert back to old RoleName based schema maps
An attempt made to avoid duplication of schema contexts in types
if any role doesn't possess any admin secret specific schema
* fix compile errors in haskell test
* keep 'user_vars' for session variables in http-logs
* no-op refacto
* tests for admin only inserts
* update docs for admin only inserts
* updated CHANGELOG.md
* default behaviour when admin secret is not set
* fix x-hasura-role to X-Hasura-Role in pytests
* introduce effective timeout in actions async tests
* update docs for admin-secret not configured case
* Update docs/graphql/manual/api-reference/schema-metadata-api/permission.rst
Co-Authored-By: Marion Schleifer <marion@hasura.io>
* Apply suggestions from code review
Co-Authored-By: Marion Schleifer <marion@hasura.io>
* a complete iteration
backend insert permissions accessable via 'x-hasura-backend-privilege'
session variable
* console changes for backend-only permissions
* provide tooltip id; update labels and tooltips;
* requested changes
* requested changes
- remove className from Toggle component
- use appropriate function name (capitalizeFirstChar -> capitalize)
* use toggle props from definitelyTyped
* fix accidental commit
* Revert "introduce effective timeout in actions async tests"
This reverts commit b7a59c19d6.
* generate complete schema for both 'default' and 'backend' sessions
* Apply suggestions from code review
Co-Authored-By: Marion Schleifer <marion@hasura.io>
* remove unnecessary import, export Toggle as is
* update session variable in tooltip
* 'x-hasura-use-backend-only-permissions' variable to switch
* update help texts
* update docs
* update docs
* update console help text
* regenerate package-lock
* serve no backend schema when backend_only: false and header set to true
- Few type name refactor as suggested by @0x777
* update CHANGELOG.md
* Update CHANGELOG.md
* Update CHANGELOG.md
* fix a merge bug where a certain entity didn't get removed
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Rishichandra Wawhal <rishi@hasura.io>
Co-authored-by: rikinsk <rikin.kachhia@gmail.com>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
* config options for internal errors for non-admin role, close#4031
More detailed action debug info is added in response 'internal' field
* add docs
* update CHANGELOG.md
* set admin graphql errors option in ci tests, minor changes to docs
* fix tests
Don't use any auth for sync actions error tests. The request body
changes based on auth type in session_variables (x-hasura-auth-mode)
* Apply suggestions from code review
Co-Authored-By: Marion Schleifer <marion@hasura.io>
* use a new sum type to represent the inclusion of internal errors
As suggested in review by @0x777
-> Move around few modules in to specific API folder
-> Saperate types from Init.hs
* fix tests
Don't use any auth for sync actions error tests. The request body
changes based on auth type in session_variables (x-hasura-auth-mode)
* move 'HttpResponse' to 'Hasura.HTTP' module
* update change log with breaking change warning
* Update CHANGELOG.md
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
The previous check was too conservative and acquired a lock on the
schema cache in situations where it was unnecessary. This change
exposes the logic run_sql uses to determine whether to use the
metadata check to make the check more precise.
* allow underscore prefix and special characters in json path
* server: Rewrite/refactor JSONPath parser
The JSONPath parser is also rewritten, the previous implementation
was written in a very explicitly “recursive descent” style, but the whole
point of using attoparsec is to be able to backtrack! Taking advantage
of the combinators makes for a much simpler parser.
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
Co-authored-by: Alexis King <lexi.lambda@gmail.com>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>
Co-authored-by: Shahidh K Muhammed <shahidh@hasura.io>
* add support for action queries
* a new parameter `type` is added in the ArgumentDefinition, its value
can be either `query` or `mutation` and it defaults to the latter
* throw 400 when a query action is tried to explain
* update the actions docs to include query actions
* refactor the ToJSON and ToOrdJSON of ActionDefinition
Co-authored-by: Rishichandra Wawhal <rishi@hasura.io>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
* add new optional field `claims_namespace_path` in JWT config
* return value when empty array is found in executeJSONPath
* update the docs related to claims_namespace_path
* improve encodeJSONPath, add property tests for parseJSONPath
* throw error if both claims_namespace_path and claims_namespace are set
* refactor the Data.Parser.JsonPath to Data.Parser.JSONPathSpec
* update the JWT docs
Co-Authored-By: Marion Schleifer <marion@hasura.io>
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Tirumarai Selvan <tirumarai.selvan@gmail.com>
* allow re-using Postgres scalars in custom types, close#4125
* add pytest tests
* update CHANGELOG.md
* add a doc pointer for reusable postgres scalars
* document the code, improve the CHANGELOG entry
As suggested by @lexi-lambda
* a bit more source code documentation, use WriterT to collect reused scalars
* Apply suggestions from code review
Co-Authored-By: Marion Schleifer <marion@hasura.io>
* improve doc for Postgres scalars in custom graphql types
* Add some more references to Note; fix Haddock syntax
Also a few very minor tweaks:
* Use HashSet instead of [] more pervasively
* Export execWriterT from Hasura.Prelude
* Use pattern guards in multi-way if
* Tweak a few names/comments
* Pull buildActions out of buildAndCollectInfo, use buildInfoMap
* Tweak wording in documentation
* incorporate changes in console code
* account Postgres scalars for action input arguments
-> Avoid unnecessary 'throw500' in making action schema
* Review changes
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Alexis King <lexi.lambda@gmail.com>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>
* do not perform the metadata check in read-only mode
* improve the isAltrDropReplace regex
* quote the regex at compile-time to handle syntax errors statically
Co-authored-by: Alexis King <lexi.lambda@gmail.com>
* Improve performance of replace_metadata on tracking tables. Closes#3802
On the 1000 table track case this went from >20min to 8 seconds, the
bottleneck being all the former calls (for each table) to
buildSchemaCache.
* refactor replace metadata
Only save metadata in hdb_catalog schema and build schema cache (strictly)
* remove `withBuildSchemaCache` function
Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* add 'ID' to default scalars for custom types, fix#4061
* preserve cookie headers from sync action webhook, close#4021
* validate action webhook response to conform to output type, fix#3977
* fix tests, don't run actions' tests on PG version < 10
* update CHANGELOG.md
* no-op refactor, use types from http-network more
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* Test working through a backlog of change events
* Use a slightly more performant threaded http server in eventing pytests
This helped locally but not on CI it seems...
* Rework event processing for backpressure. Closes#3839
With loo low `HASURA_GRAPHQL_EVENTS_FETCH_INTERVAL` and/or slow webhooks
and/or too small `HASURA_GRAPHQL_EVENTS_HTTP_POOL_SIZE` we might
previously check out events from the DB faster than we can service them,
leading to space leaks, weirdness, etc.
Other changes:
- avoid fetch interval sleep latency when we previously did a non-empty
fetch
- prefetch event batch while http pool is working
- warn when it appears we can't keep up with events being generated
- make some effort to process events in creation order so we don't
starve older ones.
ALSO NOTE: HASURA_GRAPHQL_EVENTS_FETCH_INTERVAL changes semantics
slightly, since it only comes into play after an empty fetch. The old
semantics weren't documented in detail, so I think this is fine.
* basic doc for actions
* custom_types, sync and async actions
* switch to graphql-parser-hs on github
* update docs
* metadata import/export
* webhook calls are now supported
* relationships in sync actions
* initialise.sql is now in sync with the migration file
* fix metadata tests
* allow specifying arguments of actions
* fix blacklist check on check_build_worthiness job
* track custom_types and actions related tables
* handlers are now triggered on async actions
* default to pgjson unless a field is involved in relationships, for generating definition list
* use 'true' for action filter for non admin role
* fix create_action_permission sql query
* drop permissions when dropping an action
* add a hdb_role view (and relationships) to fetch all roles in the system
* rename 'webhook' key in action definition to 'handler'
* allow templating actions wehook URLs with env vars
* add 'update_action' /v1/query type
* allow forwarding client headers by setting `forward_client_headers` in action definition
* add 'headers' configuration in action definition
* handle webhook error response based on status codes
* support array relationships for custom types
* implement single row mutation, see https://github.com/hasura/graphql-engine/issues/3731
* single row mutation: rename 'pk_columns' -> 'columns' and no-op refactor
* use top level primary key inputs for delete_by_pk & account select permissions for single row mutations
* use only REST semantics to resolve the webhook response
* use 'pk_columns' instead of 'columns' for update_by_pk input
* add python basic tests for single row mutations
* add action context (name) in webhook payload
* Async action response is accessible for non admin roles only if
the request session vars equals to action's
* clean nulls, empty arrays for actions, custom types in export metadata
* async action mutation returns only the UUID of the action
* unit tests for URL template parser
* Basic sync actions python tests
* fix output in async query & add async tests
* add admin secret header in async actions python test
* document async action architecture in Resolve/Action.hs file
* support actions returning array of objects
* tests for list type response actions
* update docs with actions and custom types metadata API reference
* update actions python tests as per #f8e1330
Co-authored-by: Tirumarai Selvan <tirumarai.selvan@gmail.com>
Co-authored-by: Aravind Shankar <face11301@gmail.com>
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
* Add check expresion to update permissions (close#384)
* wip on conflict behavior
* Handle upserts for views properly
* Use insert check if there is no update check
* Fix the test
* Improve error message slightly
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
* fix nested insert with returning computed fields gives error, fix#3609
* revert using ordered hashmaps, sort columns based on ordinal postion
* fix 1. keys order 2. json/jsonb column value in nested insert returning
* add a note for sorted columns
* cast 'VALUES' expression as table row type
* use single CTE expression for generating returning for nested inserts
This fixes#3759. Also, while we’re at it, also improve the way
invalidations are synced across instances so enums and remote schemas
are appropriately reloaded by the schema syncing process.
* WIP: Remove hdb_views for inserts
* Show failing row in check constraint error
* Revert "Show failing row in check constraint error"
This reverts commit dd2cac29d0.
* Use the better query plan
* Simplify things
* fix cli test
* Update downgrading.rst
* remove 1.1 asset for cli
- Move MonadBase/MonadBaseControl instances for TxE into pg-client-hs
- Set the -qn2 RTS option by default to limit the parallel GC to 2
threads
- Remove eventlog instrumentation
- Don’t rebuild the schema cache again after running a query that needs
it to be rebuilt, since we do that explicitly now.
- Remove some redundant checks, and relocate a couple others.
This changes TableCoreCacheT to internally record dependencies at a
per-table level. In practice, this dramatically improves the performance
of building permissions: it makes it far, far less likely for
permissions to be needlessly rebuilt because some unrelated table
changed.
* export metadata without nulls, empty arrays
* property tests for 'ReplaceMetadata' using QuickCheck
-> Derive Arbitrary class for 'ReplaceMetadata' dependant types
* reduce property test cases number to 30
QuickCheck generates the `ReplaceMetadata` value really large
for higher number test cases. Encoded JSON for such values is large and
consumes more memory. Thus, CI is giving up while running property
tests.
* circle-ci: Add property tests as saperate job
* add no command mode to tests
* add yaml.v2 to go mod
* remove indirect comment for yaml.v2 dependency
Instead of
'WITH some_alias (SELECT * from some_func()) SELECT <rows> FROM some_alias'
for SQL function queries, Use
'SELECT <rows> FROM some_func() AS some_alias'
* save permissions, relationships and collections in catalog with 'is_system_defined'
* Use common stanzas in the .cabal file
* Refactor migration code into lib instead of exe
* Add new server test suite that exercises migrations
* Make graphql-engine clean succeed even if the schema does not exist
* Separate DB and metadata migrations
* Refactor Migrate.hs to generate list of migrations at compile-time
* Replace ginger with shakespeare to improve performance
* Improve migration log messages
* allow customizing GraphQL root field names, close#981
* document v2 track_table API in reference
* support customising column field names in GraphQL schema
* [docs] add custom column fields doc in API reference
* add tests
* rename 'ColField' to 'ColumnField'
* embed column's graphql field in 'PGColumnInfo'
-> Value constructor of 'PGCol' is not exposed
-> Using 'parseJSON' to construct 'PGCol' in 'FromJSON' instances
* avoid using 'Maybe TableConfig'
* refactors & 'custom_column_fields' -> 'custom_column_names'
* cli-test: add configuration field in metadata export test
* update expected keys in `FromJSON` instance of `TableMeta`
* use `buildSchemaCacheFor` to update configuration in v2 track_table
* remove 'GraphQLName' type and use 'isValidName' exposed from parser lib
* point graphql-parser-hs library git repo to hasura
* support 'set_table_custom_fields' query API & added docs and tests
This fixes an issue where queries could incorrectly be considered
reusable if a variable was used in two positions: one where it affected
SQL generation and one where it did not.
* initial raster support
* _st_intersects_geom -> _st_intersects_geom_nband
* add tests
* update docs
* improve docs
As requested by @marionschleifer
* new type for raster values
Suggested by @lexi-lambda
* replace `SEUnsafe "NULL"` with SENull
* use positional arguments in SQL functions
* only allow omitting set of last arguments in functions
* disallow omitting of a non default argument in functions
These changes also add a new type, PGColumnType, between PGColInfo and
PGScalarType, and they process PGRawColumnType values into PGColumnType
values during schema cache generation.
* allow altering type of a column iff session vars are defined in permissions
* use a sum type to define dependency reason
* set jwt expiry test's expiry time to 4 seconds
* derive Data instance for necessary types to simplify 'hasStaticExp'
query templates is a little known feature that lets you template rql
queries and serve them as rest apis. This is not relevant anymore
given the GraphQL interface and getting rid of it reduces the dev
time when adding features in few subsystems.
This feature has never been used outside hasura's internal projects or
documented or exposed through console and hence can safely be removed.
Currently, we allow tracking of a table with the same name as an already tracked function, and vice-versa. This causes an issue when querying from GraphQL since it will only query the table and not the function. I've made changes to disallow this by throwing an error.
Changes compared to `/v1alpha1/graphql`
* Changed all graphql responses in **/v1/graphql** endpoint to be 200. All graphql clients expect responses to be HTTP 200. Non-200 responses are considered transport layer errors.
* Errors in http and websocket layer are now consistent and have similar structure.
1. Reuses postgres connections during startup which reduces the overhead of opening and closing connections.
2. Faster schema cache building. This is done by fetching all the required data in a single sql statement.
* build schema cache function without db setup
The setup shouldn't happen for sync. The database is already setup by the instance which generated the event. This means that the sync is now faster.
* use SQL loop to drop hdb_views schema views and routines with ordering
This avoids deadlocks when schema is being changed concurrently
* schema sync now only processes the latest event
This becomes useful when a lot of schema change
events happen while we are still processing an
earlier event.
* add types to represent unparsed http gql requests
This will help when we add caching of frequently used ASTs
* query plan caching
* move livequery to execute
* add multiplexed module
* session variable can be customised depending on the context
Previously the value was always "current_setting('hasura.user')"
* get rid of typemap requirement in reusable plan
* subscriptions are multiplexed when possible
* use lazytx for introspection to avoid acquiring a pg connection
* refactor to make execute a completely decoupled module
* don't issue a transaction for a query
* don't use current setting for explained sql
* move postgres related types to a different module
* validate variableValues on postgres before multiplexing subs
* don't user current_setting for queries over ws
* plan_cache is only visible when developer flag is enabled
* introduce 'batch size' when multiplexing subscriptions
* bump stackage to 13.16
* fix schema_stitching test case error code
* store hashes instead of actual responses for subscriptions
* internal api to dump subscriptions state
* remove PlanCache from SchemaCacheRef
* allow live query options to be configured on server startup
* capture metrics for multiplexed subscriptions
* more metrics captured for multiplexed subs
* switch to tvar based hashmap for faster snapshotting
* livequery modules do not expose internal details
* fix typo in live query env vars
* switch to hasura's pg-client-hs
From `alpha-40` we've been using a `WHERE` clause to fetch required rows and generate mutation response. This has a few limitations like the requirement of a primary key/unique constraint. This also returns inconsistent data on `delete` mutation as mentioned in #1794.
Now, we're using `VALUES (..)` (refer [here](https://www.postgresql.org/docs/current/sql-values.html)) expression to form virtual table rows in `SQL` to generate mutation response.
Internal changes:-
- Not to use primary key/unique constraint columns:-
- Revert back to `ConstraintName` from `TableConstraint` in `TableInfo` type
- Remove `tcCols` field in `TableConstraint` type
- Modify `table_info.sql` and `fetchTableMeta` function `SQL`
- A test case to perform `delete` mutation and returning relational objects.
If returning field contains nested selections then mutation is performed in two steps
1. Mutation is performed with returning columns of any primary key and unique constraints
2. returning fields are queried on rows returned by selecting from table by filtering with column values returned in Step 1.
Since mutation takes two courses based on selecting relations in returning field, it is hard to maintain sequence of prepared arguments (PrepArg) generated while resolving returning field. So, we're using txtConverter instead of prepare to resolve mutation fields.
Rename the admin secret key header used to access GraphQL engine from X-Hasura-Access-Key to X-Hasura-Admin-Secret.
Server CLI and console all support the older flag but marks it as deprecated.
* support jsonb and geometry operators on RQL bool exps, close#1408
* add tests for jsonb operators in /v1/query
TODO:-
-> add tests for geometry (postgis) operators
* support parsing session variables for st_d_within and has_key ops
-> Add tests for boolExp operators and select permissions
* improve parsing $st_d_within op's json value logic
* remove phase one/two distinction and hdbquery typeclass
* move extensions to default-extensions
* switch to LazyTx which only acquires a connection if needed
* move defns from TH module into Ops module
* remove tojson orphan instance for http exception
* remove orphan instance for dmlp1
* getTopLevelNodes will not throw any exceptions
When using self referential relationships in boolean expressions, the exists clause incorrectly uses the table names to qualify columns which will be the same for parent table and the child table. This is now fixed by generating unique aliases as we traverse down the relationships.
* improved nested insert execution logic
* integrate error path, improve executing 'withExp' and improve tests
* add more readable types in '/Resolve/Insert.hs'
* set conflict context just before executing WITH expression
* allow ordering using columns from object relationships, close#463
* validate table fields in nested insert
* add tests
* add docs
* change 'table_order_by' type from enums to ordered map
* remove unwanted code from 'Schema.hs' file
* 'AnnGObject' is not list of field name and value tuple
* update docs for new order_by type
* use 'InsOrdHashMap' for 'AnnGObj'
* handle empty fields in order_by
* remove '_' prefixes for asc/desc
* fix the changed order_by syntax across the repo
* support for count and aggregations on columns, close#786
* support explain query for aggregations
* '<arr-rel>_agg' in '<table>' type, fix order by for aggregations
* add 'allow_aggregations' key in select permissions
* Add checkbox to toggle count and aggregations on columns on select permission
* align aggregation checkbox with columns div
* improve readability of the generated sql
* alias is needed at the top level aggregation
* throw internal errors for unexpected fields
* rename SelFld to more readable TableAggFld
* rename agg to aggregate
Better SQL generation for select queries (the query plans will be the same but much more readable). This closes some long standing issues (#6, #121, #278).
This is breaking change where bigint and bigserial Postgres types will be encoded as GraphQL String types, as opposed to Int as present in earlier releases.
Input types were already encoded as String.
This is achieved by selecting `bigint` and `bigserial` columns as `text`s in the SQL query: `select "big_id"::text ..` instead of `select "big_id" .. `.
Reason for that change is outlined in #633 where JavaScript cannot decode 64 bit Integers.
Insert trigger function: If query affects no rows then return `null`
Insert trigger function is modified to have
`IF r IS NULL THEN RETURN null; ELSE RETURN r; END IF;` in return statement.
Fix migration logic to accommodate for non superuser permissions. Closes#567
- [x] Server
By clearing the `hdb_views` schema of existing views and functions instead of dropping and creating it again.
- [x] Bug fix (non-breaking change which fixes an issue)
### Description
What component does this PR affect?
- [x] Server
### Related Issue
#494
### Solution and Design
Use `quote_ident()` SQL function over `constraint_name` in insert trigger function definition.
### Type
- [x] Bug fix (non-breaking change which fixes an issue)
* fix primary key changing on upsert, fix#342
* add 'update_columns' in 'on_conflict' object, consider 'allowUpsert'
* 'ConflictCtx' type should respect upsert cases
* validation for not null fields in an object
The API:
1. HGE has `--jwt-secret` flag or `HASURA_GRAPHQL_JWT_SECRET` env var. The value of which is a JSON.
2. The structure of this JSON is: `{"type": "<standard-JWT-algorithms>", "key": "<the-key>"}`
`type` : Standard JWT algos : `HS256`, `RS256`, `RS512` etc. (see jwt.io).
`key`:
i. Incase of symmetric key, the key as it is.
ii. Incase of asymmetric keys, only the public key, in a PEM encoded string or as a X509 certificate.
3. The claims in the JWT token must contain the following:
i. `x-hasura-default-role` field: default role of that user
ii. `x-hasura-allowed-roles` : A list of allowed roles for the user. The default role is overriden by `x-hasura-role` header.
4. The claims in the JWT token, can have other `x-hasura-*` fields where their values can only be strings.
5. The JWT tokens are sent as `Authorization: Bearer <token>` headers.
---
To test:
1. Generate a shared secret (for HMAC-SHA256) or RSA key pair.
2. Goto https://jwt.io/ , add the keys
3. Edit the claims to have `x-hasura-role` (mandatory) and other `x-hasura-*` fields. Add permissions related to the claims to test permissions.
4. Start HGE with `--jwt-secret` flag or `HASURA_GRAPHQL_JWT_SECRET` env var, which takes a JSON string: `{"type": "HS256", "key": "mylongsharedsecret"}` or `{"type":"RS256", "key": "<PEM-encoded-public-key>"}`
5. Copy the JWT token from jwt.io and use it in the `Authorization: Bearer <token>` header.
---
TODO: Support EC public keys. It is blocked on frasertweedale/hs-jose#61
* don't allow fkey based relations from/to a table that isn't tracked, fix#185
Check if remote table exist in metadata when creating foreign-key
based object relationship.
* add tests for adding object relation using fkey if remote table is untracked
* add 'on_conflict' condition to allow upsert mutation, closes#105
* check for empty unique or primary key constraints
* add 'on_conflict' condition test cases and introspection test case
* update 'conflict_action' enum values' description
