Commit Graph

677 Commits

Author SHA1 Message Date
Vladimir Ciobanu
d5ff1acf2d better handling for one-to-one relationships
Co-authored-by: Rikin Kachhia <54616969+rikinsk@users.noreply.github.com>
GitOrigin-RevId: 1bb5bc0c4ac8109ee1d20563d23cf98e0906a483
2021-03-03 13:02:59 +00:00
Karthikeyan Chinnakonda
15ed0cf536 server: document the remote joins architecture
GitOrigin-RevId: f142908fa3dd3b0cb8887521c639be4a017e606a
2021-03-03 11:47:13 +00:00
David Overton
e4c8d560c5 Fix/missing source error
GitOrigin-RevId: 074f4eb2d13a2c8ab002a355c6734eed257482d3
2021-03-02 04:27:29 +00:00
Vladimir Ciobanu
88ce387919 docs: add note about recovering existentially quantified type info
This should be merged after #686

GitOrigin-RevId: a372ca475a9c4a182b4819310634d4c3985225ab
2021-03-01 22:31:55 +00:00
Vladimir Ciobanu
ddbc497506 docs: add note on existentials and references to it
Added a note on existentials. I plan to create a subsequent PR with a note on how we use the singletons trick to recover the type inside an existential.

GitOrigin-RevId: 1f227d859dcc384b4ac7e103053f643f879827d1
2021-03-01 21:38:50 +00:00
Praveen Durairaju
0ff26100bf update docs link to avoid redirects
GitOrigin-RevId: 1f2a1d21bfb9b2908d56305fa2dfb69270deafdf
2021-03-01 18:51:18 +00:00
Vamshi Surabhi
e9f85ce6e6 mssql: connection pooling
GitOrigin-RevId: c1a6509f19a903724ce2b770ae23cbd925b537f8
2021-02-25 18:16:52 +00:00
Abby Sassel
fb1a0d286d server: support ltree operators (close #625)
GitOrigin-RevId: fb6a3eb8cbe4604789938bcbc78916fbcd1af515
2021-02-25 11:06:49 +00:00
Antoine Leblanc
f55bbe96ef server: fix MSSQL returning malformed JSON on empty select
GitOrigin-RevId: b0b9f68dbdb2b015e4c0640670a2e4cf978fe5fb
2021-02-25 09:58:24 +00:00
Karthikeyan Chinnakonda
4211d27272 server: support reading JWT from Cookie header
GitOrigin-RevId: 1de90242d3c000361f87256c2dddce1677863231
2021-02-25 09:03:46 +00:00
Vamshi Surabhi
74a58fb66c mssql: basic support for geography and geometry types
GitOrigin-RevId: 97bbf0e8d53b00c9f4365b145adb19bdcd769ceb
2021-02-24 12:53:15 +00:00
Lyndon Maydwell
b944d598e0 Adding check for multiple query definitions for rest endpoints
When adding a rest endpoint that references a query with multiple definitions, throw an error.

Needed a second PR for this as Kodiak merged the prior PR before this commit was added.

---

### Kodiak commit message

Adding check for multiple query definitions for rest endpoints.

GitOrigin-RevId: 6e3977a210e29f143b61282fbb37c93eb5b9d73c
2021-02-24 10:13:11 +00:00
Lyndon Maydwell
08da0c63b6 REST Endpoints - Prohibit Invalid slashes, duplicate variables, non-singular query definitions, subscriptions
Resolves Issues:

* https://github.com/hasura/graphql-engine-mono/issues/658 - Invalid Slashes
* https://github.com/hasura/graphql-engine-mono/issues/628 - Subscriptions

Implementation:

* Moved some logic from Endpoint.hs to allow reuse of splitting url into PathSegments.
* Additional validation steps alongside checking for overlapping routes
* Logging potential misuse of GET for mutations

Future Work:

* [ ] GET is allowed for mutations (Ignore/Log warning for Now)
* [ ] Add to scInconsistentObjs rather than throwing error
  * Add information to scInconsistentObjs instead of raising errors directly.

TODO:

* [x] Duplicate variable segments with the same name in the location should not be allowed
* [x] We should throw an error on trailing and leading slashes and URLs which contain empty segments
* [x] Endpoints can be created using subscriptions. But the error only shows at the time of the query
* [x] Tests

---

### Kodiak commit message

Prohibit Invalid slashes, duplicate variables, subscriptions for REST endpoints.

GitOrigin-RevId: 86c0d4af97984c8afd02699e6071e9c1658710b8
2021-02-24 04:31:05 +00:00
Antoine Leblanc
7527039f08 server: fix build
GitOrigin-RevId: 8ba687fce6892ac32182311cccc1d4d149049d62
2021-02-23 21:43:56 +00:00
Vladimir Ciobanu
281cb771ff server: add MSSQL support
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Antoine Leblanc <1618949+nicuveo@users.noreply.github.com>
Co-authored-by: Vamshi Surabhi <6562944+0x777@users.noreply.github.com>
Co-authored-by: Aravind K P <8335904+scriptonist@users.noreply.github.com>
GitOrigin-RevId: 699c453b9692e1b822f393f23ff5e6db4e010d57
2021-02-23 17:38:36 +00:00
Anon Ray
cad013c039 multitenant: fix cloud schema files
GitOrigin-RevId: 99f04df3654ab83f64489960b5f7ecf5f86c9a99
2021-02-22 16:03:07 +00:00
Naveen Naidu
4fa34e1d5e server: Apply migrations when catalog version < 43
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
GitOrigin-RevId: 4fa1a04f4924002faac1eca2e5092dfa8767fdb9
2021-02-22 15:00:24 +00:00
Abby Sassel
b00009f4bb server: fix issue with queries on character column types (close #6217)
GitOrigin-RevId: 81eaa0db63656b1d37b9b71736d4309af902cc47
2021-02-22 10:21:21 +00:00
Anon Ray
0dfe83f931 server: optimize resolving source
GitOrigin-RevId: e0513a714b5e3f6d730b0470e1836ed44badd3c0
2021-02-22 07:53:36 +00:00
Antoine Leblanc
377425ff2d server: generalize subscriptions
GitOrigin-RevId: 464e80abf151032dc50eaf6cf8dafc5e7cfa51cd
2021-02-20 13:46:43 +00:00
David Overton
7e79f8dc64 Fix migrations
Provides queries for fetching and inserting metadata into that database that do not assume there is a `resource_version` column. This means that will work when migrating to/from older versions.

Co-authored-by: Lyndon Maydwell <92299+sordina@users.noreply.github.com>
GitOrigin-RevId: dac636d530524082c5a13ae0f016a2d4ced16f7f
2021-02-19 08:47:10 +00:00
Lyndon Maydwell
59c01786fe Optimistic Metadata Locking
Add optimistic concurrency control to the ‘replace_metadata’ call.

Prevents users from submitting out-of-date metadata to metadata-mutating APIs.

See https://github.com/hasura/graphql-engine-mono/issues/472 for details.

GitOrigin-RevId: 5f220f347a3eba288a9098b01e9913ffd7e38166
2021-02-19 02:40:23 +00:00
Karthikeyan Chinnakonda
7be72d7bee server: support for maintenance mode in v1.4
Co-authored-by: Anon Ray <616387+ecthiender@users.noreply.github.com>
GitOrigin-RevId: 85f636b1bc8063689845da90e85cc480e87dca7e
2021-02-18 16:47:22 +00:00
Anon Ray
1091203ced multitenant: fix issues with cloud schema sync
GitOrigin-RevId: 7749fe71a1174ac8dbfb79734584c60d882f07b7
2021-02-18 15:12:26 +00:00
Lyndon Maydwell
0d67b5cb53 Allow Inconsistent Metadata (v2) (#473)
Allow Inconsistent Metadata

GitOrigin-RevId: 7b7e1fdccbeaa1b7dccf66ff22c4b40f8cf5bb27
2021-02-16 08:10:21 +00:00
Vladimir Ciobanu
7290a57308 rename 'Connection Timeout' to 'Pool Timeout'
GitOrigin-RevId: f2f03a368b372d83bd9c9e71e2069310ba7e6f09
2021-02-16 05:19:49 +00:00
Vladimir Ciobanu
83c74ac133 connection acquisition timeout for oss and pro
Co-authored-by: Vamshi Surabhi <6562944+0x777@users.noreply.github.com>
GitOrigin-RevId: 673452c1fed2494b437b5fdd1f195e48b10f7634
2021-02-15 13:33:21 +00:00
Rakesh Emmadi
9ef603360c server: generalize schema cache building (#496)
Co-authored-by: Vamshi Surabhi <vamshi@hasura.io>
Co-authored-by: Vladimir Ciobanu <admin@cvlad.info>
Co-authored-by: Antoine Leblanc <antoine@hasura.io>
Co-authored-by: Stylish Haskell Bot <stylish-haskell@users.noreply.github.com>
GitOrigin-RevId: 9d631878037637f3ed2994b5d0525efd978f7b8f
2021-02-14 06:08:46 +00:00
Phil Freeman
7fffc11077 Caching, Rate Limiting, Metrics & Session Variable Improvements (#376)
* server: use a leaky bucket algorithm for bytes-per-second cache rate limiting

* Use evalsha properly

* Adds redis cache limit parameters to PoliciesConfig

* Loads Leaky Bucket Script On Server Start

* Adds more redis logging and moves cache update into lua script

* reverts setex in lua and adds notes

* Refactors cacheStore and adds max TTL and cache size limits

* Filter session vars in cache key

* WIP

* parens

* cache-clear-hander POC implementation

* cache-clear-hander POC implementation

* Pro projectId used as cache key

* POC working!

* prefixing query-response keys in redis

* Add cacheClearer to RedisScripts

* Partial implementation of cacheClearer from scripts record

* updating tests

* [automated] stylish-haskell commit

* Adds query look with up with metrics script

* Adds missing module and lua script from last commit

* Changes redis script module structure to match cache clearing branch

* minor change to lua script

* cleaning up cache clearing

* generalising JsonLog

* [automated] stylish-haskell commit

* Draft Cache Metrics Endpoint

* Adds Cache Metrics Handler

* Adds hook handler module

* Missed HandlerHook module in last commit

* glob

* Fixes redis mget bug

* Removes cache totals and changes dashes to colons in metric cache keys

* Adds query param to clear clear endpoint for deleting specific keys

* Adds query param to clear clear endpoint for deleting specific keys

* Cache Metrics on query families rather then queries

* Replace Set with nub

* Base16 Redis Hashes

* Query Family Redis Keys With Roles

* response headers for cache keys

* fixing bug in family key by excluding operation name; using hash for response header instead of entire key

* Adds query family to redis cache keys and cache clear endpoint

* Fixes queryfamily hash bug

* Moves cache endpoints to /pro

* Moved cache clear to POST

* Refactors cache clear function

* Fixes query family format bug

* Adds query cache tests and optional --redis-url flag to python test suite

* Adds session variable cache test

* Update pro changelog

* adding documentation for additional caching features

* more docs

* clearing up units of leaky bucket params

* Adds comments to leaky bucket script

* removes old todo

* Fixes session variable filtering to work with new query rootfield

* more advanced defaulting behaviour for bucket rate and capacity.

* Updates Docs

* Moves Role into QueryFamily hash

* Use Aeson for Cache Clear endpoint response

* Moves trace to bracket the leaky bucket script

* Misc review tweaks

* Adds sum type for cache clear query params

* Hardcodes RegisReplyLog log level

* Update docs/graphql/cloud/response-caching.rst

Co-authored-by: Phil Freeman <phil@hasura.io>

* new prose for rate limiting docs

* [automated] stylish-haskell commit

* make rootToSessVarPreds total

* [automated] stylish-haskell commit

* Fixes out of scope error

* Renamed _acRedis to _acCacheStore

Co-authored-by: Solomon Bothwell <ssbothwell@gmail.com>
Co-authored-by: Lyndon Maydwell <lyndon@sordina.net>
Co-authored-by: David Overton <david@hasura.io>
Co-authored-by: Stylish Haskell Bot <stylish-haskell@users.noreply.github.com>
Co-authored-by: Lyndon Maydwell <lyndon@hasura.io>
GitOrigin-RevId: dda5c1a3f902967b3d78310f950541a55fabb1b0
2021-02-13 00:06:18 +00:00
Sameer Kolhar
8655e6fd3a console: function permissions UI (#413)
GitOrigin-RevId: ccdfab19751b0d238a4ebcec59ba73a798103ca9
2021-02-12 17:02:41 +00:00
Karthikeyan Chinnakonda
8a68cc6650 server: don't expose remote schema mutations in relay
GitOrigin-RevId: 21c126bfd46a22d3b0e64d57e92d78ea3c4df9ab
2021-02-12 13:29:19 +00:00
Antoine Leblanc
1ca4034697 server: generalize execution of queries and mutations
GitOrigin-RevId: aff477a0849d4667f2f6dc6804d6ca2982e53f95
2021-02-12 03:05:05 +00:00
Matthew Pickering
35b81f39e9 Memory performance improvements from Cherre (#518)
* Stop shutdown handler retaining the whole serveCtx

This might look like quite a strange way to write the function but it's
the only way I could get GHC to not capture `serveCtx` in the shutdown
handler.

Fixes the metadata issue in #344

* Force argumentNames

The arguments list is often empty so we end up with a lot of duplicate
thunks if this value is not forced.

* Increase sharing in nullableType and nonNullableType

The previous definitions would lead to increased allocation as it would
destory any previously created sharing. The new definition only allocate
a fresh constructor if the value is changed.

* Add memoization for field parsers

It was observed in #344 that many parsers were not being memoised which
led to an increase in memory usage. This patch generalisation memoisation so
that it works for FieldParsers as well as normal Parsers.

There can still be substantial improvement made by also memoising
InputFieldParsers but that is left for future work.

Co-authored-by: Antoine Leblanc <antoine@hasura.io>

* [automated] stylish-haskell commit

* changelog

Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Antoine Leblanc <antoine@hasura.io>
Co-authored-by: Stylish Haskell Bot <stylish-haskell@users.noreply.github.com>
Co-authored-by: Phil Freeman <phil@hasura.io>
GitOrigin-RevId: 36255f77a47cf283ea61df9d6a4f9138d4e5834c
2021-02-12 01:34:56 +00:00
Anon Ray
06b599b747 server: multitenant metadata storage
The metadata storage implementation for graphql-engine-multitenant.

- It uses a centralized PG database to store metadata of all tenants (instead of per tenant database)
- Similarly, it uses a single schema-sync listener thread per MT worker (instead of listener thread per tenant) (PS: although, the processor thread is spawned per tenant)
- 2 new flags are introduced - `--metadataDatabaseUrl` and (optional) `--metadataDatabaseRetries`

Internally, a "metadata mode" is introduced to indicate an external/managed store vs a store managed by each pro-server.

To run :
- obtain the schema file (located at `pro/server/res/cloud/metadata_db_schema.sql`)
- apply the schema on a PG database
- set the `--metadataDatabaseUrl` flag to point to the above database
- run the MT executable

The schema (and its migrations) for the metadata db is managed outside the MT worker.

### New metadata

The following is the new portion of `Metadata` added :

```yaml
version: 3
metrics_config:
  analyze_query_variables: true
  analyze_response_body: false
api_limits:
  disabled: false
  depth_limit:
    global: 5
    per_role:
      user: 7
      editor: 9
  rate_limit:
    per_role:
      user:
        unique_params:
        - x-hasura-user-id
        - x-hasura-team-id
        max_reqs_per_min: 20
    global:
      unique_params: IP
      max_reqs_per_min: 10
```

- In Pro, the code around fetching/updating/syncing pro-config is removed
- That also means, `hdb_pro_catalog` for keeping the config cache is not required. Hence the `hdb_pro_catalog` is also removed
- The required config comes from metadata / schema cache

### New Metadata APIs

- `set_api_limits`
- `remove_api_limits`
- `set_metrics_config`
- `remove_metrics_config`

#### `set_api_limits`

```yaml
type: set_api_limits
args:
  disabled: false
  depth_limit:
    global: 5
    per_role:
      user: 7
      editor: 9
  rate_limit:
    per_role:
      anonymous:
         max_reqs_per_min: 10
         unique_params: "ip"
      editor:
        max_reqs_per_min: 30
        unique_params:
        - x-hasura-user-id
      user:
        unique_params:
        - x-hasura-user-id
        - x-hasura-team-id
        max_reqs_per_min: 20
    global:
      unique_params: IP
      max_reqs_per_min: 10
```

#### `remove_api_limits`

```yaml
type: remove_api_limits
args: {}
```

#### `set_metrics_config`

```yaml
type: set_metrics_config
args:
  analyze_query_variables: true
  analyze_response_body: false
```

#### `remove_metrics_config`

```yaml
type: remove_metrics_config
args: {}
```

#### TODO
- [x] on-prem pro implementation for `MonadMetadataStorage`
- [x] move the project config from Lux to pro metadata (PR: #379)
- [ ] console changes for pro config/api limits, subscription workers (cc @soorajshankar @beerose)
- [x] address other minor TODOs
  - [x] TxIso for `MonadSourceResolver`
  - [x] enable EKG connection pool metrics
  - [x] add logging of connection info when sources are added?
  - [x] confirm if the `buildReason` for schema cache is correct
- [ ] testing
- [x] 1.3 -> 1.4 cloud migration script (#465; PR: #508)
  - [x] one-time migration of existing metadata from users' db to centralized PG
  - [x] one-time migration of pro project config + api limits + regression tests from metrics API  to metadata
- [ ] integrate with infra team (WIP - cc @hgiasac)
  - [x] benchmark with 1000+ tenants + each tenant making read/update metadata query every second (PR: https://github.com/hasura/graphql-engine-mono/pull/411)
  - [ ] benchmark with few tenants having large metadata (100+ tables etc.)
  - [ ] when user moves regions (https://github.com/hasura/lux/issues/1717)
    - [ ] metadata has to be migrated from one regional PG to another
    - [ ] migrate metrics data as well ?
      - [ ] operation logs
      - [ ] regression test runs

- [ ] find a way to share the schema files with the infra team

Co-authored-by: Naveen Naidu <30195193+Naveenaidu@users.noreply.github.com>
GitOrigin-RevId: 39e8361f2c0e96e0f9e8f8fb45e6cc14857f31f1
2021-02-11 17:55:21 +00:00
Karthikeyan Chinnakonda
0cfad08993 server: function permissions changes
GitOrigin-RevId: b0d784dd23aacd7a62cf471dd33eca869e25c13e
2021-02-11 09:44:38 +00:00
Antoine Leblanc
9b38863c73 server: change RootField so that only the DB component is existential
GitOrigin-RevId: ff620b775ea9b1c8433255705254ed9d58eba290
2021-02-09 19:30:08 +00:00
Antoine Leblanc
7760ed2d98 server: generalize top-level build functions
GitOrigin-RevId: 5ace65f702b411bf75c84e949742c7d657382fab
2021-02-09 12:48:22 +00:00
Vladimir Ciobanu
6b7b68504a server: improve the UX around SourceInfo
GitOrigin-RevId: 47f7e9b7daa3f89771dd3883e9f6050274827e27
2021-02-08 16:31:17 +00:00
Antoine Leblanc
e449cf9990 server: finish generalizing mutations
GitOrigin-RevId: 645eefadee35f1642eee805e20161417ab38b949
2021-02-08 10:40:51 +00:00
Antoine Leblanc
83701fb63e server: changes to support other backends
GitOrigin-RevId: ec0ad47957ab6f9a0855623fffedb23924e7c75d
2021-02-03 16:25:17 +00:00
Karthikeyan Chinnakonda
312e56dfa6 server: support tracking of custom functions which return single row
GitOrigin-RevId: fbf99816a70e666bb9b02fca92a0ee7d419cdff8
2021-02-03 14:07:30 +00:00
Swann Moreau
c14dcd5792 pass gql requests into auth webhook POST body (#149)
* fix arg order in UserAuthentication instance [force ci]

* change the constructor name to AHGraphQLRequest

Co-authored-by: Stylish Haskell Bot <stylish-haskell@users.noreply.github.com>
Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
GitOrigin-RevId: fb3258f4a84efc6c730b0c6222ebd8cea1b91081
2021-02-03 07:11:39 +00:00
Karthikeyan Chinnakonda
94a886ee94 server: fix mutation functions not being exposed to admin when function permissions are inferred
GitOrigin-RevId: 7d8031341351bdbf25d4244ce57d4f9396dde437
2021-02-01 12:58:31 +00:00
Karthikeyan Chinnakonda
05f21a2768 server: fix the FromJSON instance of FunctionPermissionsCtx
GitOrigin-RevId: 506d654ef9cad95ffdddbf832a2779fc34e6bd00
2021-02-01 06:04:28 +00:00
Karthikeyan Chinnakonda
10a3f9960d server: new function permissions layer
Co-authored-by: Rikin Kachhia <54616969+rikinsk@users.noreply.github.com>
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
GitOrigin-RevId: 35645121242294cb6bb500ea598e9a1f2ca67fa1
2021-01-29 05:49:09 +00:00
Lyndon Maydwell
0767333597 server: support restified versions of graphql queries (#303)
Restified GraphQL Endpoints feature.

GitOrigin-RevId: 3d6e589426ec21a60a915b47f579f0ac4934af45
2021-01-29 01:03:35 +00:00
Antoine Leblanc
353859db09 server: remove GraphQL.Utils
GitOrigin-RevId: 90639f9f3d263ccb0ce4e3b8b6e19ce784f4b25d
2021-01-26 13:14:35 +00:00
Antoine Leblanc
62a3bb0d9e Move text operator out of SQL.Types. (#406)
GitOrigin-RevId: 25adad545b302fd1dde25ea0365d492dfcdaa71d
2021-01-26 12:23:20 +00:00
Antoine Leblanc
6494229f54 server: generalize functions (#393)
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
GitOrigin-RevId: 5d2140152a2a18601c785ea80a7689cbe3bd277e
2021-01-25 10:13:54 +00:00
Karthikeyan Chinnakonda
3020150274 server: allow mapping session variables to standard JWT claims
fixes https://github.com/hasura/graphql-engine/issues/6449

A while back we added [support for customizing JWT claims](https://github.com/hasura/graphql-engine/pull/3575) and this enabled to map a session variable to any value within the unregistered claims, but as reported in #6449 , users aren't able to map the `x-hasura-user-id` session variable to the `sub` standard JWT claim.

This PR fixes the above issue by allowing mapping session variables to standard JWT claims as well.

GitOrigin-RevId: d3e63d7580adac55eb212e0a1ecf7c33f5b3ac4b
2021-01-21 16:50:46 +00:00