>
### Description
>
While adding [insert mutation schema parser for MSSQL backend](https://github.com/hasura/graphql-engine-mono/pull/2141) I also included [identity](https://en.wikipedia.org/wiki/Identity_column) notion to table columns across all backends. In MSSQL we cannot insert any value (even `DEFAULT` expression) into Identity columns. This behavior of identity columns is not same in Postgres as we can insert values. This PR drops the notion of identity in the column info. The context of identity columns for MSSQL is carried in `ExtraTableMetadata` type.
### Changelog
- [x] `CHANGELOG.md` is updated with user-facing content relevant to this PR. If no changelog is required, then add the `no-changelog-required` label.
### Affected components
- [x] Server
- [ ] Console
- [ ] CLI
- [ ] Docs
- [ ] Community Content
- [ ] Build System
- [x] Tests
- [ ] Other (list it)
### Related Issues
->
Fix https://github.com/hasura/graphql-engine/issues/7557https://github.com/hasura/graphql-engine-mono/pull/2378
GitOrigin-RevId: c18b5708e2e6107423a0a95a7fc2e9721e8a21a1
Some of our use of CPP causes trouble for ormolu, compare https://github.com/tweag/ormolu/issues/774.
Specifically, for understandable reasons, it can't deal well with `#ifdef` use that is not at the top-level.
This PR removes the problematic usage in ways that I hope are also a net non-loss regardless of helping
out ormolu (or other tooling).
- The default value for enabled APIs moves to the top level, next to the command line help, so
they'll stay in sync more easily.
- All the CPP around using `assertNFHere` is moved to one module.
https://github.com/hasura/graphql-engine-mono/pull/2361
GitOrigin-RevId: ed6e039e6d8960322fd8d1312df762ad197c29b1
## Description
Almost all our data structures use strictness annotations, following [our styleguide's principle](https://github.com/hasura/graphql-engine/blob/master/server/STYLE.md#dealing-with-laziness) of "by default, use strict data types and lazy functions". The very few cases where we actually need laziness were already explicitly labelled as lazy with the `~` prefix operator.
This PR simply globally enables `StrictData`, allowing us to express records without `!()` on every field, but makes no attempt at cleaning existing code.
https://github.com/hasura/graphql-engine-mono/pull/1869
Co-authored-by: Philip Lykke Carlsen <358550+plcplc@users.noreply.github.com>
GitOrigin-RevId: e65c6e2f89413188da250122f64c2173615946ec
### Description
We always build a subscription root, even when there was no possible fields. This breaks some third party clients, as the spec does not allow empty types in the schema. This PR fixes this by changing the `buildSubscriptionParser` helper to return a `Maybe` value, and harmonizes / cleans places where we build the subscription root.
https://github.com/hasura/graphql-engine-mono/pull/2357
GitOrigin-RevId: 1aeae25e321eee957e7645c436d17e69207309fd
This is a follow-up to #1959.
Today, I spent a while in review figuring out that a harmless PR change didn't do anything,
because it was moving from a `runLazy...` to something without the `Lazy`. So let's get
that source of confusion removed.
This should be a bit easier to review commit by commit, since some of the functions had
confusing names. (E.g. there was a misnamed `Migrate.Internal.runTx` before.)
The change should be a no-op.
https://github.com/hasura/graphql-engine-mono/pull/2335
GitOrigin-RevId: 0f284c4c0f814482d7827e7732a6d49e7735b302
### Description
During the PDV refactor that led to 2.0, we broke an undocumented and untested semantic of inserts: accepting _explicit_ null values in nested object inserts.
In short: in the schema, we often distinguish between _explicit_ null values `{id: 3, author: null}` and _implicit_ null values that correspond to the field being omitted `{id: 3}`. In this particular case, we forgot to accept explicit null values. Since the field is optional (meaning we accept implicit null values), it was nullable in the schema, like it was in pre-PDV times. But in practice we would reject explicit nulls.
This PR fixes this, and adds a test. Furthermore, it does a bit of a cleanup of the Mutation part of the schema, and more specifically of all insertion code.
https://github.com/hasura/graphql-engine-mono/pull/2341
GitOrigin-RevId: 895cfeecef7e8e49903a3fb37987707150446eb0
>
### Description
>
This PR is an incremental work towards [enabling insert mutations on MSSQL](https://github.com/hasura/graphql-engine-mono/pull/1974). In this PR, we generate insert mutation schema parser for MSSQL backend.
### Changelog
- [ ] `CHANGELOG.md` is updated with user-facing content relevant to this PR. If no changelog is required, then add the `no-changelog-required` label.
### Affected components
- [x] Server
https://github.com/hasura/graphql-engine-mono/pull/2141
GitOrigin-RevId: 8595008dece35f7fded9c52e134de8b97b64f53f
When adding object relationships, we set the nullability of the generated GraphQL field based on whether the database backend enforces that the referenced data always exists. For manual relationships (corresponding to `manual_configuration`), the database backend is unaware of any relationship between data, and hence such fields are always set to be nullable.
For relationships generated from foreign key constraints (corresponding to `foreign_key_constraint_on`), we distinguish between two cases:
1. The "forward" object relationship from a referencing table (i.e. which has the foreign key constraint) to a referenced table. This should be set to be non-nullable when all referencing columns are non-nullable. But in fact, it used to set it to be non-nullable if *any* referencing column is non-nullable, which is only correct in Postgres when `MATCH FULL` is set (a flag we don't consider). This fixes that by changing a boolean conjunction to a disjunction.
2. The "reverse" object relationship from a referenced table to a referencing table which has the foreign key constraint. This should always be set to be nullable. But in fact, it used to always be set to non-nullable, as was reported in hasura/graphql-engine#7201. This fixes that.
Moreover, we have moved the computation of the nullability from `Hasura.RQL.DDL.Relationship` to `Hasura.GraphQL.Schema.Select`: this nullability used to be passed through the `riIsNullable` field of `RelInfo`, but for array relationships this information is not actually used, and moreover the remaining fields of `RelInfo` are already enough to deduce the nullability.
This also adds regression tests for both (1) and (2) above.
https://github.com/hasura/graphql-engine-mono/pull/2159
GitOrigin-RevId: 617f12765614f49746d18d3368f41dfae2f3e6ca
This removes the module re-exports of [Data.Align](https://hackage.haskell.org/package/semialign-1.2/docs/Data-Align.html) and [Data.These](https://hackage.haskell.org/package/these-1.1.1.1/docs/Data-These.html) from `Hasura.Prelude`. The reasoning being that they're not used widely and reasonably obscure, and that being explicit about the imports makes for an easier to understand codebase.
(I spent longer than I'd have liked earlier today figuring out where `align` in multitenant came from.
The right one not showing up on the first hoogle page doesn't help. Yes, better tool use could have
avoided that, but still...)
Do feel free to shoot this down, I won't insist on the change.
https://github.com/hasura/graphql-engine-mono/pull/2194
GitOrigin-RevId: 10f887b74538b17623bee6d6451c5aba11573fbd
Replaces one instance of `mtl`-style effects with `transformers`-style, as this results in a measurable reduction in memory usage. The change is kept completely within one module.
https://github.com/hasura/graphql-engine-mono/pull/1944
GitOrigin-RevId: 587b8e61725bb4a505404bbe741185759b7bceeb
This is just a one-off fix, based on running ormolu across
the code base, which uses GHC's parser in haddock mode.
### Description
Fixes several instances of illegal haddock comments.
### Related Issues
#1679
### Steps to test and verify
Run ormolu over the codebase. Prior to this change, it complains that it
can't parse certain files due to malformed Haddock comments, after it
doesn't (there are still some other errors).
### Limitations, known bugs & workarounds
This doesn't ensure that we don't introduce similar issues in the future;
that'll be dealt with once we implement #1679.
#### Breaking changes
- [x] No Breaking changes, only touches code comments
https://github.com/hasura/graphql-engine-mono/pull/2010
GitOrigin-RevId: 7fbab0325ce13a16a04ff98d351f1af768e25d7c
>
### Description
>
From HGE version 2.0 onwards, all remote relationship fields are generated as plain types without non-nullable and lists. This PR fixes the same.
### Changelog
- [x] `CHANGELOG.md` is updated with user-facing content relevant to this PR. If no changelog is required, then add the `no-changelog-required` label.
### Affected components
- [x] Server
- [x] Tests
### Related Issues
->
fix https://github.com/hasura/graphql-engine/issues/7284
### Steps to test and verify
>
- Create a remote relationship to a field in remote schema with non-nullable or list type
- The HGE introspection should give the remote relationship field type correctly as like in the remote schema
https://github.com/hasura/graphql-engine-mono/pull/2071
GitOrigin-RevId: e113f5d17b62bfa0a25028c20260ae1782ae224b
Query plan caching was introduced by - I believe - hasura/graphql-engine#1934 in order to reduce the query response latency. During the development of PDV in hasura/graphql-engine#4111, it was found out that the new architecture (for which query plan caching wasn't implemented) performed comparably to the pre-PDV architecture with caching. Hence, it was decided to leave query plan caching until some day in the future when it was deemed necessary.
Well, we're in the future now, and there still isn't a convincing argument for query plan caching. So the time has come to remove some references to query plan caching from the codebase. For the most part, any code being removed would probably not be very well suited to the post-PDV architecture of query execution, so arguably not much is lost.
Apart from simplifying the code, this PR will contribute towards making the GraphQL schema generation more modular, testable, and easier to profile. I'd like to eventually work towards a situation in which it's easy to generate a GraphQL schema parser *in isolation*, without being connected to a database, and then parse a GraphQL query *in isolation*, without even listening any HTTP port. It is important that both of these operations can be examined in detail, and in isolation, since they are two major performance bottlenecks, as well as phases where many important upcoming features hook into.
Implementation
The following have been removed:
- The entirety of `server/src-lib/Hasura/GraphQL/Execute/Plan.hs`
- The core phases of query parsing and execution no longer have any references to query plan caching. Note that this is not to be confused with query *response* caching, which is not affected by this PR. This includes removal of the types:
- - `Opaque`, which is replaced by a tuple. Note that the old implementation was broken and did not adequately hide the constructors.
- - `QueryReusability` (and the `markNotReusable` method). Notably, the implementation of the `ParseT` monad now consists of two, rather than three, monad transformers.
- Cache-related tests (in `server/src-test/Hasura/CacheBoundedSpec.hs`) have been removed .
- References to query plan caching in the documentation.
- The `planCacheOptions` in the `TenantConfig` type class was removed. However, during parsing, unrecognized fields in the YAML config get ignored, so this does not cause a breaking change. (Confirmed manually, as well as in consultation with @sordina.)
- The metrics no longer send cache hit/miss messages.
There are a few places in which one can still find references to query plan caching:
- We still accept the `--query-plan-cache-size` command-line option for backwards compatibility. The `HASURA_QUERY_PLAN_CACHE_SIZE` environment variable is not read.
https://github.com/hasura/graphql-engine-mono/pull/1815
GitOrigin-RevId: 17d92b254ec093c62a7dfeec478658ede0813eb7
GJ IR changes cherry-picked from the original GJ branch. There is a separate (can be merged independently) PR for metadata changes (#1727) and there will be a different PR upcoming PR for execution changes.
https://github.com/hasura/graphql-engine-mono/pull/1810
Co-authored-by: Vamshi Surabhi <6562944+0x777@users.noreply.github.com>
GitOrigin-RevId: c31956af29dc9c9b75d002aba7d93c230697c5f4
### Description
This PR is the first of several PRs meant to introduce Generalized Joins. In this first PR, we add non-breaking changes to the Metadata types for DB-to-DB remote joins. Note that we are currently rejecting the new remote join format in order to keep folks from breaking their metadata (in case of a downgrade). These issues will be tackled (and JSON changes reverted) in subsequent PRs.
This PR also changes the way we construct the schema cache, and breaks the way we process sources in two steps: we first resolve each source and construct a cache of their tables' raw info, then in a second step we build the source output. This is so that we have access to the target source's tables when building db-to-db relationships.
### Notes
- this PR contains a few minor cleanups of the schema
- it also fixes a bug in how we do renames in remote schema relationships
- it introduces cross-source schema dependencies
https://github.com/hasura/graphql-engine-mono/pull/1727
Co-authored-by: Evie Ciobanu <1017953+eviefp@users.noreply.github.com>
GitOrigin-RevId: f625473077bc5fff5d941b70e9a116192bc1eb22
### Description
In our haste to generalize everything for MSSQL, we put every single "suspicious" type in Backend, including ones that weren't required. `Alias` is one of those: it's only used in a type alias, and is actually just an implementation detail of the translation layer. This PR removes it.
https://github.com/hasura/graphql-engine-mono/pull/1759
GitOrigin-RevId: fb348934ec65a51aae7f95d93c83c3bb704587b5
### Description
This PR removes all `fmapX` and `traverseX` functions from RQL.IR, favouring instead `Functor` and `Traversable` instances throughout the code. This was a relatively straightforward change, except for two small pain points: `AnnSelectG` and `AnnInsert`. Both were parametric over two types `a` and `v`, making it impossible to make them traversable functors... But it turns out that in every single use case, `a ~ f v`. By changing those types to take such an `f :: Type -> Type` as an argument instead of `a :: Type` makes it possible to make them functors.
The only small difference is for `AnnIns`, I had to introduce one `Identity` transformation for one of the `f` parameters. This is relatively straightforward.
### Notes
This PR fixes the most verbose BigQuery hint (`let` instead of `<- pure`).
https://github.com/hasura/graphql-engine-mono/pull/1668
GitOrigin-RevId: e632263a8c559aa04aeae10dcaec915b4a81ad1a
- add name fields to log output in several spots:
- action logs get the action name in detail.action_name
- events (triggered and scheduled) get the trigger name in detail.event_name
- one-off scheduled events don't have a trigger name; instead, they get the
comment if it exists
- remove unused event creation timestamp from ExtraLogContext
https://github.com/hasura/graphql-engine-mono/pull/1712
GitOrigin-RevId: 28907340d4e2d9adc0c48cc5d3010eef1fa902e1
### Description
This PR adds the required IR for DB to DB joins, based on @paf31 and @0x777 's `feature/db-to-db` branch.
To do so, it also refactors the IR to introduce a new type parameter, `r`, which is used to recursively constructs the `v` parameter of remote QueryDBs. When collecting remote joins, we replace `r` with `Const Void`, indicating at the type level that there cannot be any leftover remote join.
Furthermore, this PR refactors IR.Select for readability, moves some code from IR.Root to IR.Select to avoid having to deal with circular dependencies, and makes it compile by adding `error` in all new cases in the execution pipeline.
The diff doesn't make it clear, but most of Select.hs is actually unchanged. Declarations have just been reordered by topic, in the following order:
- type declarations
- instance declarations
- type aliases
- constructor functions
- traverse functions
https://github.com/hasura/graphql-engine-mono/pull/1580
Co-authored-by: Phil Freeman <630306+paf31@users.noreply.github.com>
GitOrigin-RevId: bbdcb4119cec8bb3fc32f1294f91b8dea0728721
Remote relationships are now supported on SQL Server and BigQuery. The major change though is the re-architecture of remote join execution logic. Prior to this PR, each backend is responsible for processing the remote relationships that are part of their AST.
This is not ideal as there is nothing specific about a remote join's execution that ties it to a backend. The only backend specific part is whether or not the specification of the remote relationship is valid (i.e, we'll need to validate whether the scalars are compatible).
The approach now changes to this:
1. Before delegating the AST to the backend, we traverse the AST, collect all the remote joins while modifying the AST to add necessary join fields where needed.
1. Once the remote joins are collected from the AST, the database call is made to fetch the response. The necessary data for the remote join(s) is collected from the database's response and one or more remote schema calls are constructed as necessary.
1. The remote schema calls are then executed and the data from the database and from the remote schemas is joined to produce the final response.
### Known issues
1. Ideally the traversal of the IR to collect remote joins should return an AST which does not include remote join fields. This operation can be type safe but isn't taken up as part of the PR.
1. There is a lot of code duplication between `Transport/HTTP.hs` and `Transport/Websocket.hs` which needs to be fixed ASAP. This too hasn't been taken up by this PR.
1. The type which represents the execution plan is only modified to handle our current remote joins and as such it will have to be changed to accommodate general remote joins.
1. Use of lenses would have reduced the boilerplate code to collect remote joins from the base AST.
1. The current remote join logic assumes that the join columns of a remote relationship appear with their names in the database response. This however is incorrect as they could be aliased. This can be taken up by anyone, I've left a comment in the code.
### Notes to the reviewers
I think it is best reviewed commit by commit.
1. The first one is very straight forward.
1. The second one refactors the remote join execution logic but other than moving things around, it doesn't change the user facing functionality. This moves Postgres specific parts to `Backends/Postgres` module from `Execute`. Some IR related code to `Hasura.RQL.IR` module. Simplifies various type class function signatures as a backend doesn't have to handle remote joins anymore
1. The third one fixes partial case matches that for some weird reason weren't shown as warnings before this refactor
1. The fourth one generalizes the validation logic of remote relationships and implements `scalarTypeGraphQLName` function on SQL Server and BigQuery which is used by the validation logic. This enables remote relationships on BigQuery and SQL Server.
https://github.com/hasura/graphql-engine-mono/pull/1497
GitOrigin-RevId: 77dd8eed326602b16e9a8496f52f46d22b795598
This reverts the remote schema type customisation and namespacing feature temporarily as we test for certain conditions.
GitOrigin-RevId: f8ee97233da4597f703970c3998664c03582d8e7
This is a minor refactor (part of `Internal/Parser.hs` is moved into `Internal/Input.hs`) to remove `Collect.hs-boot` and `Directives.hs-boot` files. Without these changes:
1. Most changes would trigger recompilation from the modules with hs-boot files.
1. haskell-language-server fails for some reason in the presence of hs-boot files.
GitOrigin-RevId: 77a2e443417b449c5d7d9d418fc75fcdf076a9ae
Multi source support had limited the availability of async action queries in subscriptions. This PR
adds support for async action query subscriptions with new implementation. Also addresses https://github.com/hasura/graphql-engine/issues/6460.
GitOrigin-RevId: 5ddc321073d224f287dc4b86ce2239ff55190b36
While debugging issues with HLS, Reed Mullanix noticed that we don't use relative paths. This leads to problems when using HLS + Emacs due to a bug in `lsp-mode` which prevents it from finding the correct project root.
However, it is still a good practice to use relative paths in TH for other reasons, including being able to import these modules in GHCI.
This PR should make it so HLS-1.0 & emacs provide type inference, imports, etc., in all modules in our codebase.
GitOrigin-RevId: 5f53b9a7ccf46df1ea7be94ff0a5c6ec861f4ead
Fixes https://github.com/hasura/graphql-engine-mono/issues/712
Main point of interest: the `Hasura.SQL.Backend` module.
This PR creates an `Exists` type indexed by indexed type and packed constraint while hiding all of its complexity by not exporting the constructor.
Existential constructors/types which are no longer (directly) existential:
- [X] BackendSourceInfo :: BackendSourceInfo
- [x] BackendSourceMetadata :: BackendSourceMetadata
- [x] MOSourceObjId :: MetadatObjId
- [x] SOSourceObj :: SchemaObjId
- [x] RFDB :: RootField
- [x] LQP :: LiveQueryPlan
- [x] ExecutionStep :: ExecStepDB
This PR also removes ALL usages of `Typeable.cast` from our codebase. We still need to derive `Typeable` in a few places in order to be able to derive `Data` in one place. I have not dug deeper to see why this is needed.
GitOrigin-RevId: bb47e957192e4bb0af4c4116aee7bb92f7983445
fixes#3868
docker image - `hasura/graphql-engine:inherited-roles-preview-48b73a2de`
Note:
To be able to use the inherited roles feature, the graphql-engine should be started with the env variable `HASURA_GRAPHQL_EXPERIMENTAL_FEATURES` set to `inherited_roles`.
Introduction
------------
This PR implements the idea of multiple roles as presented in this [paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/FGALanguageICDE07.pdf). The multiple roles feature in this PR can be used via inherited roles. An inherited role is a role which can be created by combining multiple singular roles. For example, if there are two roles `author` and `editor` configured in the graphql-engine, then we can create a inherited role with the name of `combined_author_editor` role which will combine the select permissions of the `author` and `editor` roles and then make GraphQL queries using the `combined_author_editor`.
How are select permissions of different roles are combined?
------------------------------------------------------------
A select permission includes 5 things:
1. Columns accessible to the role
2. Row selection filter
3. Limit
4. Allow aggregation
5. Scalar computed fields accessible to the role
Suppose there are two roles, `role1` gives access to the `address` column with row filter `P1` and `role2` gives access to both the `address` and the `phone` column with row filter `P2` and we create a new role `combined_roles` which combines `role1` and `role2`.
Let's say the following GraphQL query is queried with the `combined_roles` role.
```graphql
query {
employees {
address
phone
}
}
```
This will translate to the following SQL query:
```sql
select
(case when (P1 or P2) then address else null end) as address,
(case when P2 then phone else null end) as phone
from employee
where (P1 or P2)
```
The other parameters of the select permission will be combined in the following manner:
1. Limit - Minimum of the limits will be the limit of the inherited role
2. Allow aggregations - If any of the role allows aggregation, then the inherited role will allow aggregation
3. Scalar computed fields - same as table column fields, as in the above example
APIs for inherited roles:
----------------------
1. `add_inherited_role`
`add_inherited_role` is the [metadata API](https://hasura.io/docs/1.0/graphql/core/api-reference/index.html#schema-metadata-api) to create a new inherited role. It accepts two arguments
`role_name`: the name of the inherited role to be added (String)
`role_set`: list of roles that need to be combined (Array of Strings)
Example:
```json
{
"type": "add_inherited_role",
"args": {
"role_name":"combined_user",
"role_set":[
"user",
"user1"
]
}
}
```
After adding the inherited role, the inherited role can be used like single roles like earlier
Note:
An inherited role can only be created with non-inherited/singular roles.
2. `drop_inherited_role`
The `drop_inherited_role` API accepts the name of the inherited role and drops it from the metadata. It accepts a single argument:
`role_name`: name of the inherited role to be dropped
Example:
```json
{
"type": "drop_inherited_role",
"args": {
"role_name":"combined_user"
}
}
```
Metadata
---------
The derived roles metadata will be included under the `experimental_features` key while exporting the metadata.
```json
{
"experimental_features": {
"derived_roles": [
{
"role_name": "manager_is_employee_too",
"role_set": [
"employee",
"manager"
]
}
]
}
}
```
Scope
------
Only postgres queries and subscriptions are supported in this PR.
Important points:
-----------------
1. All columns exposed to an inherited role will be marked as `nullable`, this is done so that cell value nullification can be done.
TODOs
-------
- [ ] Tests
- [ ] Test a GraphQL query running with a inherited role without enabling inherited roles in experimental features
- [] Tests for aggregate queries, limit, computed fields, functions, subscriptions (?)
- [ ] Introspection test with a inherited role (nullability changes in a inherited role)
- [ ] Docs
- [ ] Changelog
Co-authored-by: Vamshi Surabhi <6562944+0x777@users.noreply.github.com>
GitOrigin-RevId: 3b8ee1e11f5ceca80fe294f8c074d42fbccfec63
Added a note on existentials. I plan to create a subsequent PR with a note on how we use the singletons trick to recover the type inside an existential.
GitOrigin-RevId: 1f227d859dcc384b4ac7e103053f643f879827d1
* server: use a leaky bucket algorithm for bytes-per-second cache rate limiting
* Use evalsha properly
* Adds redis cache limit parameters to PoliciesConfig
* Loads Leaky Bucket Script On Server Start
* Adds more redis logging and moves cache update into lua script
* reverts setex in lua and adds notes
* Refactors cacheStore and adds max TTL and cache size limits
* Filter session vars in cache key
* WIP
* parens
* cache-clear-hander POC implementation
* cache-clear-hander POC implementation
* Pro projectId used as cache key
* POC working!
* prefixing query-response keys in redis
* Add cacheClearer to RedisScripts
* Partial implementation of cacheClearer from scripts record
* updating tests
* [automated] stylish-haskell commit
* Adds query look with up with metrics script
* Adds missing module and lua script from last commit
* Changes redis script module structure to match cache clearing branch
* minor change to lua script
* cleaning up cache clearing
* generalising JsonLog
* [automated] stylish-haskell commit
* Draft Cache Metrics Endpoint
* Adds Cache Metrics Handler
* Adds hook handler module
* Missed HandlerHook module in last commit
* glob
* Fixes redis mget bug
* Removes cache totals and changes dashes to colons in metric cache keys
* Adds query param to clear clear endpoint for deleting specific keys
* Adds query param to clear clear endpoint for deleting specific keys
* Cache Metrics on query families rather then queries
* Replace Set with nub
* Base16 Redis Hashes
* Query Family Redis Keys With Roles
* response headers for cache keys
* fixing bug in family key by excluding operation name; using hash for response header instead of entire key
* Adds query family to redis cache keys and cache clear endpoint
* Fixes queryfamily hash bug
* Moves cache endpoints to /pro
* Moved cache clear to POST
* Refactors cache clear function
* Fixes query family format bug
* Adds query cache tests and optional --redis-url flag to python test suite
* Adds session variable cache test
* Update pro changelog
* adding documentation for additional caching features
* more docs
* clearing up units of leaky bucket params
* Adds comments to leaky bucket script
* removes old todo
* Fixes session variable filtering to work with new query rootfield
* more advanced defaulting behaviour for bucket rate and capacity.
* Updates Docs
* Moves Role into QueryFamily hash
* Use Aeson for Cache Clear endpoint response
* Moves trace to bracket the leaky bucket script
* Misc review tweaks
* Adds sum type for cache clear query params
* Hardcodes RegisReplyLog log level
* Update docs/graphql/cloud/response-caching.rst
Co-authored-by: Phil Freeman <phil@hasura.io>
* new prose for rate limiting docs
* [automated] stylish-haskell commit
* make rootToSessVarPreds total
* [automated] stylish-haskell commit
* Fixes out of scope error
* Renamed _acRedis to _acCacheStore
Co-authored-by: Solomon Bothwell <ssbothwell@gmail.com>
Co-authored-by: Lyndon Maydwell <lyndon@sordina.net>
Co-authored-by: David Overton <david@hasura.io>
Co-authored-by: Stylish Haskell Bot <stylish-haskell@users.noreply.github.com>
Co-authored-by: Lyndon Maydwell <lyndon@hasura.io>
GitOrigin-RevId: dda5c1a3f902967b3d78310f950541a55fabb1b0
* Stop shutdown handler retaining the whole serveCtx
This might look like quite a strange way to write the function but it's
the only way I could get GHC to not capture `serveCtx` in the shutdown
handler.
Fixes the metadata issue in #344
* Force argumentNames
The arguments list is often empty so we end up with a lot of duplicate
thunks if this value is not forced.
* Increase sharing in nullableType and nonNullableType
The previous definitions would lead to increased allocation as it would
destory any previously created sharing. The new definition only allocate
a fresh constructor if the value is changed.
* Add memoization for field parsers
It was observed in #344 that many parsers were not being memoised which
led to an increase in memory usage. This patch generalisation memoisation so
that it works for FieldParsers as well as normal Parsers.
There can still be substantial improvement made by also memoising
InputFieldParsers but that is left for future work.
Co-authored-by: Antoine Leblanc <antoine@hasura.io>
* [automated] stylish-haskell commit
* changelog
Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Antoine Leblanc <antoine@hasura.io>
Co-authored-by: Stylish Haskell Bot <stylish-haskell@users.noreply.github.com>
Co-authored-by: Phil Freeman <phil@hasura.io>
GitOrigin-RevId: 36255f77a47cf283ea61df9d6a4f9138d4e5834c
The metadata storage implementation for graphql-engine-multitenant.
- It uses a centralized PG database to store metadata of all tenants (instead of per tenant database)
- Similarly, it uses a single schema-sync listener thread per MT worker (instead of listener thread per tenant) (PS: although, the processor thread is spawned per tenant)
- 2 new flags are introduced - `--metadataDatabaseUrl` and (optional) `--metadataDatabaseRetries`
Internally, a "metadata mode" is introduced to indicate an external/managed store vs a store managed by each pro-server.
To run :
- obtain the schema file (located at `pro/server/res/cloud/metadata_db_schema.sql`)
- apply the schema on a PG database
- set the `--metadataDatabaseUrl` flag to point to the above database
- run the MT executable
The schema (and its migrations) for the metadata db is managed outside the MT worker.
### New metadata
The following is the new portion of `Metadata` added :
```yaml
version: 3
metrics_config:
analyze_query_variables: true
analyze_response_body: false
api_limits:
disabled: false
depth_limit:
global: 5
per_role:
user: 7
editor: 9
rate_limit:
per_role:
user:
unique_params:
- x-hasura-user-id
- x-hasura-team-id
max_reqs_per_min: 20
global:
unique_params: IP
max_reqs_per_min: 10
```
- In Pro, the code around fetching/updating/syncing pro-config is removed
- That also means, `hdb_pro_catalog` for keeping the config cache is not required. Hence the `hdb_pro_catalog` is also removed
- The required config comes from metadata / schema cache
### New Metadata APIs
- `set_api_limits`
- `remove_api_limits`
- `set_metrics_config`
- `remove_metrics_config`
#### `set_api_limits`
```yaml
type: set_api_limits
args:
disabled: false
depth_limit:
global: 5
per_role:
user: 7
editor: 9
rate_limit:
per_role:
anonymous:
max_reqs_per_min: 10
unique_params: "ip"
editor:
max_reqs_per_min: 30
unique_params:
- x-hasura-user-id
user:
unique_params:
- x-hasura-user-id
- x-hasura-team-id
max_reqs_per_min: 20
global:
unique_params: IP
max_reqs_per_min: 10
```
#### `remove_api_limits`
```yaml
type: remove_api_limits
args: {}
```
#### `set_metrics_config`
```yaml
type: set_metrics_config
args:
analyze_query_variables: true
analyze_response_body: false
```
#### `remove_metrics_config`
```yaml
type: remove_metrics_config
args: {}
```
#### TODO
- [x] on-prem pro implementation for `MonadMetadataStorage`
- [x] move the project config from Lux to pro metadata (PR: #379)
- [ ] console changes for pro config/api limits, subscription workers (cc @soorajshankar @beerose)
- [x] address other minor TODOs
- [x] TxIso for `MonadSourceResolver`
- [x] enable EKG connection pool metrics
- [x] add logging of connection info when sources are added?
- [x] confirm if the `buildReason` for schema cache is correct
- [ ] testing
- [x] 1.3 -> 1.4 cloud migration script (#465; PR: #508)
- [x] one-time migration of existing metadata from users' db to centralized PG
- [x] one-time migration of pro project config + api limits + regression tests from metrics API to metadata
- [ ] integrate with infra team (WIP - cc @hgiasac)
- [x] benchmark with 1000+ tenants + each tenant making read/update metadata query every second (PR: https://github.com/hasura/graphql-engine-mono/pull/411)
- [ ] benchmark with few tenants having large metadata (100+ tables etc.)
- [ ] when user moves regions (https://github.com/hasura/lux/issues/1717)
- [ ] metadata has to be migrated from one regional PG to another
- [ ] migrate metrics data as well ?
- [ ] operation logs
- [ ] regression test runs
- [ ] find a way to share the schema files with the infra team
Co-authored-by: Naveen Naidu <30195193+Naveenaidu@users.noreply.github.com>
GitOrigin-RevId: 39e8361f2c0e96e0f9e8f8fb45e6cc14857f31f1
This PR generalizes a bunch of metadata structures.
Most importantly, it changes `SourceCache` to hold existentially quantified values:
```
data BackendSourceInfo =
forall b. Backend b => BackendSourceInfo (SourceInfo b)
type SourceCache = HashMap SourceName BackendSourceInfo
```
This changes a *lot* of things throughout the code. For now, all code using the schema cache explicitly casts sources to Postgres, meaning that if any non-Postgres `SourceInfo` makes it to the cache, it'll be ignored.
That means that after this PR is submitted, we can split work between two different aspects:
- creating `SourceInfo` for other backends
- handling those other sources down the line
GitOrigin-RevId: fb9ea00f32e840fc33c5467896fb1dfa5283ab42
### Description
Our Prelude provides the very convenient `hoistMaybe :: Maybe b -> MaybeT m b`. This PR adds hlint rules to replace uses of `MaybeT $ pure $ x` with the cleaner `hoistMaybe x`, and rules to specifically replace `MaybeT $ pure Nothing` with `empty`.
GitOrigin-RevId: 7254f4954e34e4d7ca972dc7c12073d3ab8cb0b8
Fixes https://github.com/hasura/graphql-engine/issues/5704 by checking, for aggregate fields whether we are handling a numeric aggregation.
This PR also adds type information to `ColFld` such that we know the type of the field.
This is the second attempt. See #319 for a less invasive approach. @nicuveo suggested type information might be useful, and since it wasn't hard to add, I think this version is better as well.
GitOrigin-RevId: aa6a259fd5debe9466df6302839ddbbd0ea659b5
Fixes https://github.com/hasura/graphql-engine/issues/6385
In the v1.3.4-beta.2 version, the SQL generated for a query action containing relationship and configured with permissions parsed the `x-hasura-user-id` session variable value through the `hasura.user` postgres setting instead of passing the session variables as an JSON object to the query as it was in v1.3.3.
This PR fixes the SQL generation to that of v1.3.3
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
GitOrigin-RevId: 838ba812f89b51df7fcead81b9d3c2885dfa39b4
### Description
This PR updates the graphql schema to be backend-agnostic. To do so, it also moves the definition of operators to `BackendSchema`, to be specified differently per backend.
GitOrigin-RevId: 35b9d2d1bff93fb68b872d6ab0d3d12ec12c1b93
This is an incremental PR towards https://github.com/hasura/graphql-engine/pull/5797
Co-authored-by: Anon Ray <ecthiender@users.noreply.github.com>
GitOrigin-RevId: a6cb8c239b2ff840a0095e78845f682af0e588a9
