Commit Graph

496 Commits

Author SHA1 Message Date
Rakesh Emmadi
362230e8d0 server: async action query subscription
Multi source support had limited the availability of async action queries in subscriptions. This PR
adds support for async action query subscriptions with new implementation. Also addresses https://github.com/hasura/graphql-engine/issues/6460.

GitOrigin-RevId: 5ddc321073d224f287dc4b86ce2239ff55190b36
2021-03-31 10:40:15 +00:00
Rakesh Emmadi
97549ccc8e server: fix action output type generation (#850)
* fix action output type generation

* update CHANGELOG.md

GitOrigin-RevId: 558ff640f5fe5dd5ebf15a3c063f1b989a5fb28a
2021-03-22 10:57:46 +00:00
Vladimir Ciobanu
91710bba58 server: use relative paths in TH splices
While debugging issues with HLS, Reed Mullanix noticed that we don't use relative paths. This leads to problems when using HLS + Emacs due to a bug in `lsp-mode` which prevents it from finding the correct project root.

However, it is still a good practice to use relative paths in TH for other reasons, including being able to import these modules in GHCI.

This PR should make it so HLS-1.0 & emacs provide type inference, imports, etc., in all modules in our codebase.

GitOrigin-RevId: 5f53b9a7ccf46df1ea7be94ff0a5c6ec861f4ead
2021-03-16 17:36:39 +00:00
Karthikeyan Chinnakonda
d1a2fabb26 server: fix the FromJSON instance of ExperimentalFeature
GitOrigin-RevId: 4d3c296021bfc26295632fb4b331c1fe0c902734
2021-03-16 11:49:00 +00:00
Vladimir Ciobanu
da8f6981d4 server: reduce the number of backend dispatches
Fixes https://github.com/hasura/graphql-engine-mono/issues/712

Main point of interest: the `Hasura.SQL.Backend` module.

This PR creates an `Exists` type indexed by indexed type and packed constraint while hiding all of its complexity by not exporting the constructor.

Existential constructors/types which are no longer (directly) existential:
- [X] BackendSourceInfo :: BackendSourceInfo
- [x] BackendSourceMetadata :: BackendSourceMetadata
- [x] MOSourceObjId :: MetadatObjId
- [x] SOSourceObj :: SchemaObjId
- [x] RFDB :: RootField
- [x] LQP :: LiveQueryPlan
- [x] ExecutionStep :: ExecStepDB

This PR also removes ALL usages of `Typeable.cast` from our codebase. We still need to derive `Typeable` in a few places in order to be able to derive `Data` in one place. I have not dug deeper to see why this is needed.

GitOrigin-RevId: bb47e957192e4bb0af4c4116aee7bb92f7983445
2021-03-15 13:03:55 +00:00
Antoine Leblanc
a544fe9822 server: generalized LogQueryLog (fix #708)
GitOrigin-RevId: 0e18e0f5caaa6889804debf7111ef9aa99e080b5
2021-03-13 14:41:44 +00:00
Abby Sassel
64d52f5fa3 server: generalize integration tests
Co-authored-by: Antoine Leblanc <1618949+nicuveo@users.noreply.github.com>
GitOrigin-RevId: 647edb8d293bf783c504b7e15ce02c56858b3b72
2021-03-11 18:18:41 +00:00
Karthikeyan Chinnakonda
71beba0c54 server: add experimental features in the alpha config API
GitOrigin-RevId: b4841e3dbea8340f8fad638359f3b1cb28188db6
2021-03-09 18:46:42 +00:00
Karthikeyan Chinnakonda
92026b769f [Preview] Inherited roles for postgres read queries
fixes #3868

docker image - `hasura/graphql-engine:inherited-roles-preview-48b73a2de`

Note:

To be able to use the inherited roles feature, the graphql-engine should be started with the env variable `HASURA_GRAPHQL_EXPERIMENTAL_FEATURES` set to `inherited_roles`.

Introduction
------------

This PR implements the idea of multiple roles as presented in this [paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/FGALanguageICDE07.pdf). The multiple roles feature in this PR can be used via inherited roles. An inherited role is a role which can be created by combining multiple singular roles. For example, if there are two roles `author` and `editor` configured in the graphql-engine, then we can create a inherited role with the name of `combined_author_editor` role which will combine the select permissions of the `author` and `editor` roles and then make GraphQL queries using the `combined_author_editor`.

How are select permissions of different roles are combined?
------------------------------------------------------------

A select permission includes 5 things:

1. Columns accessible to the role
2. Row selection filter
3. Limit
4. Allow aggregation
5. Scalar computed fields accessible to the role

 Suppose there are two roles, `role1` gives access to the `address` column with row filter `P1` and `role2` gives access to both the `address` and the `phone` column with row filter `P2` and we create a new role `combined_roles` which combines `role1` and `role2`.

Let's say the following GraphQL query is queried with the `combined_roles` role.

```graphql
query {
   employees {
     address
     phone
   }
}
```

This will translate to the following SQL query:

```sql

 select
    (case when (P1 or P2) then address else null end) as address,
    (case when P2 then phone else null end) as phone
 from employee
 where (P1 or P2)
```

The other parameters of the select permission will be combined in the following manner:

1. Limit - Minimum of the limits will be the limit of the inherited role
2. Allow aggregations - If any of the role allows aggregation, then the inherited role will allow aggregation
3. Scalar computed fields - same as table column fields, as in the above example

APIs for inherited roles:
----------------------

1. `add_inherited_role`

`add_inherited_role` is the [metadata API](https://hasura.io/docs/1.0/graphql/core/api-reference/index.html#schema-metadata-api) to create a new inherited role. It accepts two arguments

`role_name`: the name of the inherited role to be added (String)
`role_set`: list of roles that need to be combined (Array of Strings)

Example:

```json
{
  "type": "add_inherited_role",
  "args": {
      "role_name":"combined_user",
      "role_set":[
          "user",
          "user1"
      ]
  }
}
```

After adding the inherited role, the inherited role can be used like single roles like earlier

Note:

An inherited role can only be created with non-inherited/singular roles.

2. `drop_inherited_role`

The `drop_inherited_role` API accepts the name of the inherited role and drops it from the metadata. It accepts a single argument:

`role_name`: name of the inherited role to be dropped

Example:

```json

{
  "type": "drop_inherited_role",
  "args": {
      "role_name":"combined_user"
  }
}
```

Metadata
---------

The derived roles metadata will be included under the `experimental_features` key while exporting the metadata.

```json
{
  "experimental_features": {
    "derived_roles": [
      {
        "role_name": "manager_is_employee_too",
        "role_set": [
          "employee",
          "manager"
        ]
      }
    ]
  }
}
```

Scope
------

Only postgres queries and subscriptions are supported in this PR.

Important points:
-----------------

1. All columns exposed to an inherited role will be marked as `nullable`, this is done so that cell value nullification can be done.

TODOs
-------

- [ ] Tests
   - [ ] Test a GraphQL query running with a inherited role without enabling inherited roles in experimental features
   - [] Tests for aggregate queries, limit, computed fields, functions, subscriptions (?)
   - [ ] Introspection test with a inherited role (nullability changes in a inherited role)
- [ ] Docs
- [ ] Changelog

Co-authored-by: Vamshi Surabhi <6562944+0x777@users.noreply.github.com>
GitOrigin-RevId: 3b8ee1e11f5ceca80fe294f8c074d42fbccfec63
2021-03-08 11:15:10 +00:00
Naveen Naidu
f55df228fe Server: Add Event Trigger Metrics
- [x] **Event Triggers Metrics**
  - [x] Distribution of size of event trigger fetches / Number of events fetched in the last `event trigger fetch`
  - [x] Event Triggers: Number of event trigger HTTP workers in process
  - [x]  Event Triggers: Avg event trigger lock time (if an event has been fetched but not processed because http worker is not free)

#### Sample response
The metrics can be viewed from the `/dev/ekg` endpoint

```json
{
"num_events_fetched":{
      "max":0,
      "mean":0,
      "count":1,
      "min":0,
      "variance":null,
      "type":"d",
      "sum":0
   },
 "num_event_trigger_http_workers":{
      "type":"g",
      "val":0
   },
"event_lock_time":{
      "max":0,
      "mean":0,
      "count":0,
      "min":0,
      "variance":0,
      "type":"d",
      "sum":0
   },
```

#### Todo
- [ ]  Group similar metrics together (Eg: Group all the metrics related to Event trigger, How do we do it??)

Closes: https://github.com/hasura/graphql-engine-mono/issues/202
GitOrigin-RevId: bada11d871272b04c8a09d006d9d037a8464a472
2021-03-07 05:25:24 +00:00
Sameer Kolhar
ee56b741d8 server: add is_remote_schema_permissions_enabled to config API
GitOrigin-RevId: 7129d0fba25edc2450c74140335df9398154ce52
2021-03-04 16:22:31 +00:00
Praveen Durairaju
0ff26100bf update docs link to avoid redirects
GitOrigin-RevId: 1f2a1d21bfb9b2908d56305fa2dfb69270deafdf
2021-03-01 18:51:18 +00:00
Vamshi Surabhi
e9f85ce6e6 mssql: connection pooling
GitOrigin-RevId: c1a6509f19a903724ce2b770ae23cbd925b537f8
2021-02-25 18:16:52 +00:00
Karthikeyan Chinnakonda
4211d27272 server: support reading JWT from Cookie header
GitOrigin-RevId: 1de90242d3c000361f87256c2dddce1677863231
2021-02-25 09:03:46 +00:00
Vladimir Ciobanu
281cb771ff server: add MSSQL support
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Antoine Leblanc <1618949+nicuveo@users.noreply.github.com>
Co-authored-by: Vamshi Surabhi <6562944+0x777@users.noreply.github.com>
Co-authored-by: Aravind K P <8335904+scriptonist@users.noreply.github.com>
GitOrigin-RevId: 699c453b9692e1b822f393f23ff5e6db4e010d57
2021-02-23 17:38:36 +00:00
Naveen Naidu
4fa34e1d5e server: Apply migrations when catalog version < 43
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
GitOrigin-RevId: 4fa1a04f4924002faac1eca2e5092dfa8767fdb9
2021-02-22 15:00:24 +00:00
Antoine Leblanc
377425ff2d server: generalize subscriptions
GitOrigin-RevId: 464e80abf151032dc50eaf6cf8dafc5e7cfa51cd
2021-02-20 13:46:43 +00:00
David Overton
7e79f8dc64 Fix migrations
Provides queries for fetching and inserting metadata into that database that do not assume there is a `resource_version` column. This means that will work when migrating to/from older versions.

Co-authored-by: Lyndon Maydwell <92299+sordina@users.noreply.github.com>
GitOrigin-RevId: dac636d530524082c5a13ae0f016a2d4ced16f7f
2021-02-19 08:47:10 +00:00
Lyndon Maydwell
59c01786fe Optimistic Metadata Locking
Add optimistic concurrency control to the ‘replace_metadata’ call.

Prevents users from submitting out-of-date metadata to metadata-mutating APIs.

See https://github.com/hasura/graphql-engine-mono/issues/472 for details.

GitOrigin-RevId: 5f220f347a3eba288a9098b01e9913ffd7e38166
2021-02-19 02:40:23 +00:00
Karthikeyan Chinnakonda
7be72d7bee server: support for maintenance mode in v1.4
Co-authored-by: Anon Ray <616387+ecthiender@users.noreply.github.com>
GitOrigin-RevId: 85f636b1bc8063689845da90e85cc480e87dca7e
2021-02-18 16:47:22 +00:00
Anon Ray
1091203ced multitenant: fix issues with cloud schema sync
GitOrigin-RevId: 7749fe71a1174ac8dbfb79734584c60d882f07b7
2021-02-18 15:12:26 +00:00
Lyndon Maydwell
0d67b5cb53 Allow Inconsistent Metadata (v2) (#473)
Allow Inconsistent Metadata

GitOrigin-RevId: 7b7e1fdccbeaa1b7dccf66ff22c4b40f8cf5bb27
2021-02-16 08:10:21 +00:00
Vladimir Ciobanu
7290a57308 rename 'Connection Timeout' to 'Pool Timeout'
GitOrigin-RevId: f2f03a368b372d83bd9c9e71e2069310ba7e6f09
2021-02-16 05:19:49 +00:00
Vladimir Ciobanu
83c74ac133 connection acquisition timeout for oss and pro
Co-authored-by: Vamshi Surabhi <6562944+0x777@users.noreply.github.com>
GitOrigin-RevId: 673452c1fed2494b437b5fdd1f195e48b10f7634
2021-02-15 13:33:21 +00:00
Rakesh Emmadi
9ef603360c server: generalize schema cache building (#496)
Co-authored-by: Vamshi Surabhi <vamshi@hasura.io>
Co-authored-by: Vladimir Ciobanu <admin@cvlad.info>
Co-authored-by: Antoine Leblanc <antoine@hasura.io>
Co-authored-by: Stylish Haskell Bot <stylish-haskell@users.noreply.github.com>
GitOrigin-RevId: 9d631878037637f3ed2994b5d0525efd978f7b8f
2021-02-14 06:08:46 +00:00
Phil Freeman
7fffc11077 Caching, Rate Limiting, Metrics & Session Variable Improvements (#376)
* server: use a leaky bucket algorithm for bytes-per-second cache rate limiting

* Use evalsha properly

* Adds redis cache limit parameters to PoliciesConfig

* Loads Leaky Bucket Script On Server Start

* Adds more redis logging and moves cache update into lua script

* reverts setex in lua and adds notes

* Refactors cacheStore and adds max TTL and cache size limits

* Filter session vars in cache key

* WIP

* parens

* cache-clear-hander POC implementation

* cache-clear-hander POC implementation

* Pro projectId used as cache key

* POC working!

* prefixing query-response keys in redis

* Add cacheClearer to RedisScripts

* Partial implementation of cacheClearer from scripts record

* updating tests

* [automated] stylish-haskell commit

* Adds query look with up with metrics script

* Adds missing module and lua script from last commit

* Changes redis script module structure to match cache clearing branch

* minor change to lua script

* cleaning up cache clearing

* generalising JsonLog

* [automated] stylish-haskell commit

* Draft Cache Metrics Endpoint

* Adds Cache Metrics Handler

* Adds hook handler module

* Missed HandlerHook module in last commit

* glob

* Fixes redis mget bug

* Removes cache totals and changes dashes to colons in metric cache keys

* Adds query param to clear clear endpoint for deleting specific keys

* Adds query param to clear clear endpoint for deleting specific keys

* Cache Metrics on query families rather then queries

* Replace Set with nub

* Base16 Redis Hashes

* Query Family Redis Keys With Roles

* response headers for cache keys

* fixing bug in family key by excluding operation name; using hash for response header instead of entire key

* Adds query family to redis cache keys and cache clear endpoint

* Fixes queryfamily hash bug

* Moves cache endpoints to /pro

* Moved cache clear to POST

* Refactors cache clear function

* Fixes query family format bug

* Adds query cache tests and optional --redis-url flag to python test suite

* Adds session variable cache test

* Update pro changelog

* adding documentation for additional caching features

* more docs

* clearing up units of leaky bucket params

* Adds comments to leaky bucket script

* removes old todo

* Fixes session variable filtering to work with new query rootfield

* more advanced defaulting behaviour for bucket rate and capacity.

* Updates Docs

* Moves Role into QueryFamily hash

* Use Aeson for Cache Clear endpoint response

* Moves trace to bracket the leaky bucket script

* Misc review tweaks

* Adds sum type for cache clear query params

* Hardcodes RegisReplyLog log level

* Update docs/graphql/cloud/response-caching.rst

Co-authored-by: Phil Freeman <phil@hasura.io>

* new prose for rate limiting docs

* [automated] stylish-haskell commit

* make rootToSessVarPreds total

* [automated] stylish-haskell commit

* Fixes out of scope error

* Renamed _acRedis to _acCacheStore

Co-authored-by: Solomon Bothwell <ssbothwell@gmail.com>
Co-authored-by: Lyndon Maydwell <lyndon@sordina.net>
Co-authored-by: David Overton <david@hasura.io>
Co-authored-by: Stylish Haskell Bot <stylish-haskell@users.noreply.github.com>
Co-authored-by: Lyndon Maydwell <lyndon@hasura.io>
GitOrigin-RevId: dda5c1a3f902967b3d78310f950541a55fabb1b0
2021-02-13 00:06:18 +00:00
Sameer Kolhar
8655e6fd3a console: function permissions UI (#413)
GitOrigin-RevId: ccdfab19751b0d238a4ebcec59ba73a798103ca9
2021-02-12 17:02:41 +00:00
Antoine Leblanc
1ca4034697 server: generalize execution of queries and mutations
GitOrigin-RevId: aff477a0849d4667f2f6dc6804d6ca2982e53f95
2021-02-12 03:05:05 +00:00
Anon Ray
06b599b747 server: multitenant metadata storage
The metadata storage implementation for graphql-engine-multitenant.

- It uses a centralized PG database to store metadata of all tenants (instead of per tenant database)
- Similarly, it uses a single schema-sync listener thread per MT worker (instead of listener thread per tenant) (PS: although, the processor thread is spawned per tenant)
- 2 new flags are introduced - `--metadataDatabaseUrl` and (optional) `--metadataDatabaseRetries`

Internally, a "metadata mode" is introduced to indicate an external/managed store vs a store managed by each pro-server.

To run :
- obtain the schema file (located at `pro/server/res/cloud/metadata_db_schema.sql`)
- apply the schema on a PG database
- set the `--metadataDatabaseUrl` flag to point to the above database
- run the MT executable

The schema (and its migrations) for the metadata db is managed outside the MT worker.

### New metadata

The following is the new portion of `Metadata` added :

```yaml
version: 3
metrics_config:
  analyze_query_variables: true
  analyze_response_body: false
api_limits:
  disabled: false
  depth_limit:
    global: 5
    per_role:
      user: 7
      editor: 9
  rate_limit:
    per_role:
      user:
        unique_params:
        - x-hasura-user-id
        - x-hasura-team-id
        max_reqs_per_min: 20
    global:
      unique_params: IP
      max_reqs_per_min: 10
```

- In Pro, the code around fetching/updating/syncing pro-config is removed
- That also means, `hdb_pro_catalog` for keeping the config cache is not required. Hence the `hdb_pro_catalog` is also removed
- The required config comes from metadata / schema cache

### New Metadata APIs

- `set_api_limits`
- `remove_api_limits`
- `set_metrics_config`
- `remove_metrics_config`

#### `set_api_limits`

```yaml
type: set_api_limits
args:
  disabled: false
  depth_limit:
    global: 5
    per_role:
      user: 7
      editor: 9
  rate_limit:
    per_role:
      anonymous:
         max_reqs_per_min: 10
         unique_params: "ip"
      editor:
        max_reqs_per_min: 30
        unique_params:
        - x-hasura-user-id
      user:
        unique_params:
        - x-hasura-user-id
        - x-hasura-team-id
        max_reqs_per_min: 20
    global:
      unique_params: IP
      max_reqs_per_min: 10
```

#### `remove_api_limits`

```yaml
type: remove_api_limits
args: {}
```

#### `set_metrics_config`

```yaml
type: set_metrics_config
args:
  analyze_query_variables: true
  analyze_response_body: false
```

#### `remove_metrics_config`

```yaml
type: remove_metrics_config
args: {}
```

#### TODO
- [x] on-prem pro implementation for `MonadMetadataStorage`
- [x] move the project config from Lux to pro metadata (PR: #379)
- [ ] console changes for pro config/api limits, subscription workers (cc @soorajshankar @beerose)
- [x] address other minor TODOs
  - [x] TxIso for `MonadSourceResolver`
  - [x] enable EKG connection pool metrics
  - [x] add logging of connection info when sources are added?
  - [x] confirm if the `buildReason` for schema cache is correct
- [ ] testing
- [x] 1.3 -> 1.4 cloud migration script (#465; PR: #508)
  - [x] one-time migration of existing metadata from users' db to centralized PG
  - [x] one-time migration of pro project config + api limits + regression tests from metrics API  to metadata
- [ ] integrate with infra team (WIP - cc @hgiasac)
  - [x] benchmark with 1000+ tenants + each tenant making read/update metadata query every second (PR: https://github.com/hasura/graphql-engine-mono/pull/411)
  - [ ] benchmark with few tenants having large metadata (100+ tables etc.)
  - [ ] when user moves regions (https://github.com/hasura/lux/issues/1717)
    - [ ] metadata has to be migrated from one regional PG to another
    - [ ] migrate metrics data as well ?
      - [ ] operation logs
      - [ ] regression test runs

- [ ] find a way to share the schema files with the infra team

Co-authored-by: Naveen Naidu <30195193+Naveenaidu@users.noreply.github.com>
GitOrigin-RevId: 39e8361f2c0e96e0f9e8f8fb45e6cc14857f31f1
2021-02-11 17:55:21 +00:00
Swann Moreau
c14dcd5792 pass gql requests into auth webhook POST body (#149)
* fix arg order in UserAuthentication instance [force ci]

* change the constructor name to AHGraphQLRequest

Co-authored-by: Stylish Haskell Bot <stylish-haskell@users.noreply.github.com>
Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
GitOrigin-RevId: fb3258f4a84efc6c730b0c6222ebd8cea1b91081
2021-02-03 07:11:39 +00:00
Karthikeyan Chinnakonda
10a3f9960d server: new function permissions layer
Co-authored-by: Rikin Kachhia <54616969+rikinsk@users.noreply.github.com>
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
GitOrigin-RevId: 35645121242294cb6bb500ea598e9a1f2ca67fa1
2021-01-29 05:49:09 +00:00
Lyndon Maydwell
0767333597 server: support restified versions of graphql queries (#303)
Restified GraphQL Endpoints feature.

GitOrigin-RevId: 3d6e589426ec21a60a915b47f579f0ac4934af45
2021-01-29 01:03:35 +00:00
Karthikeyan Chinnakonda
3020150274 server: allow mapping session variables to standard JWT claims
fixes https://github.com/hasura/graphql-engine/issues/6449

A while back we added [support for customizing JWT claims](https://github.com/hasura/graphql-engine/pull/3575) and this enabled to map a session variable to any value within the unregistered claims, but as reported in #6449 , users aren't able to map the `x-hasura-user-id` session variable to the `sub` standard JWT claim.

This PR fixes the above issue by allowing mapping session variables to standard JWT claims as well.

GitOrigin-RevId: d3e63d7580adac55eb212e0a1ecf7c33f5b3ac4b
2021-01-21 16:50:46 +00:00
Antoine Leblanc
4815fcd500 server: progress on generic metadata
This PR generalizes a bunch of metadata structures.

Most importantly, it changes `SourceCache` to hold existentially quantified values:
```
data BackendSourceInfo =
  forall b. Backend b => BackendSourceInfo (SourceInfo b)

type SourceCache = HashMap SourceName BackendSourceInfo
```

This changes a *lot* of things throughout the code. For now, all code using the schema cache explicitly casts sources to Postgres, meaning that if any non-Postgres `SourceInfo` makes it to the cache, it'll be ignored.

That means that after this PR is submitted, we can split work between two different aspects:
  - creating `SourceInfo` for other backends
  - handling those other sources down the line

GitOrigin-RevId: fb9ea00f32e840fc33c5467896fb1dfa5283ab42
2021-01-20 00:32:45 +00:00
hasura-bot
98ccd81704 Server: Remote relationships permissions
GITHUB_PR_NUMBER: 6125
GITHUB_PR_URL: https://github.com/hasura/graphql-engine/pull/6125

Co-authored-by: Karthikeyan Chinnakonda <15602904+codingkarthik@users.noreply.github.com>
GitOrigin-RevId: 53d0671e6335dad1af7cb00e3e05e7021a910673
2021-01-19 20:57:58 +00:00
hasura-bot
2c56254e5a server: simplify JSON instances
GITHUB_PR_NUMBER: 6152
GITHUB_PR_URL: https://github.com/hasura/graphql-engine/pull/6152

Co-authored-by: Antoine Leblanc <1618949+nicuveo@users.noreply.github.com>
GitOrigin-RevId: 6c94aef8c57e852b3d41b8355c09e64fce756a7c
2021-01-19 19:15:42 +00:00
hasura-bot
38fc4cb677 server: consistently log request_id at the same level
GITHUB_PR_NUMBER: 6244
GITHUB_PR_URL: https://github.com/hasura/graphql-engine/pull/6244

Co-authored-by: José Lorenzo Rodríguez <37621+lorenzo@users.noreply.github.com>
GitOrigin-RevId: fef22d98ac7be23ca21a63dc1c696fa7169253a4
2021-01-19 17:26:45 +00:00
Karthikeyan Chinnakonda
6c22132061 server: move remote schema permissions APIs to /v1/metadata
Earlier (pre catalog separation), the remote schema permissions were in `/v1/query`. This PR moves it to `/v1/metadata`.

GitOrigin-RevId: cb39d9df4cc2288f67231504e3a7909f2f8df4da
2021-01-18 10:39:29 +00:00
Karthikeyan Chinnakonda
c14bcb6967 server: accept new config allowed_skew in JWT config to provide leeway in JWT expiry
fixes https://github.com/hasura/graphql-engine/issues/2109

This PR accepts a new config `allowed_skew` in the JWT config to provide for some leeway while comparing the JWT expiry time.

GitOrigin-RevId: ef50cf77d8e2780478685096ed13794b5c4c9de4
2021-01-13 08:39:18 +00:00
Lyndon Maydwell
ece4fb4bce Merge pull request #342 from hasura/naveen/move-serverCtx-to-Config
server: Move 'serverMetrics' to Config.hs for better visibility
GitOrigin-RevId: 4f2b40ec81d4d4bb662cc16693b993fb3606b64a
2021-01-13 06:28:12 +00:00
Antoine Leblanc
3948ca84da server: RQL code health
This PR is a combination of the following other PRs:
- #169: move HasHttpManager out of RQL.Types
- #170: move UserInfoM to Hasura.Session
- #179: delete dead code from RQL.Types
- #180: move event related code to EventTrigger

GitOrigin-RevId: d97608d7945f2c7a0a37e307369983653eb62eb1
2021-01-08 23:10:36 +00:00
Rakesh Emmadi
84b5153dc1 replace 'pg_set_table_custom_fields' with 'pg_set_table_customization' in /v1/metadata API (#321)
GitOrigin-RevId: ac770273630321eec224c436c09bd61134bf28c3
2021-01-08 07:42:46 +00:00
Rakesh Emmadi
be62641f68 server: multi source metadata APIs (#217)
Co-authored-by: Aleksandra Sikora <aleksandra@hasura.io>
Co-authored-by: Anon Ray <ecthiender@users.noreply.github.com>
Co-authored-by: Vishnu Bharathi <vishnubharathi04@gmail.com>
Co-authored-by: Aleksandra Sikora <aleksandra@hasura.io>
Co-authored-by: Sameer Kolhar <kolhar730@gmail.com>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
GitOrigin-RevId: 0dd1e4d58ab81f1b4ce24de2d3eab709c2755e6d
2021-01-07 09:05:19 +00:00
Rakesh Emmadi
29f2ddc289 server: support separate metadata database and server code setup for multi sources (#197)
This is an incremental PR towards https://github.com/hasura/graphql-engine/pull/5797

Co-authored-by: Anon Ray <ecthiender@users.noreply.github.com>
GitOrigin-RevId: a6cb8c239b2ff840a0095e78845f682af0e588a9
2020-12-28 12:56:55 +00:00
Phil Freeman
2dfbf99b41 server: simplify shutdown logic, improve resource management (#218) (#195)
* Remove unused ExitCode constructors

* Simplify shutdown logic

* Update server/src-lib/Hasura/App.hs

Co-authored-by: Brandon Simmons <brandon@hasura.io>

* WIP: fix zombie thread issue

* Use forkCodensity for the schema sync thread

* Use forkCodensity for the oauthTokenUpdateWorker

* Use forkCodensity for the schema update processor thread

* Add deprecation notice

* Logger threads use Codensity

* Add the MonadFix instance for Codensity to get log-sender thread logs

* Move outIdleGC out to the top level, WIP

* Update forkImmortal fuction for more logging info

* add back the idle GC to Pro

* setupAuth

* use ImmortalThreadLog

* Fix tests

* Add another finally block

* loud warnings

* Change log level

* hlint

* Finalize the logger in the correct place

* Add ManagedT

* Update server/src-lib/Hasura/Server/Auth.hs

Co-authored-by: Brandon Simmons <brandon@hasura.io>

* Comments etc.

Co-authored-by: Brandon Simmons <brandon@hasura.io>
Co-authored-by: Naveen Naidu <naveennaidu479@gmail.com>
GitOrigin-RevId: 156065c5c3ace0e13d1997daef6921cc2e9f641c
2020-12-21 18:56:57 +00:00
Karthikeyan Chinnakonda
39a4352569 Merge pull request #113 from hasura/karthikeyan/remote-schema-permissions
server: remote schema permissions
GitOrigin-RevId: 63b9717e30351676c9474bdfddd3ad1ee1409eea
2020-12-21 09:12:35 +00:00
Tirumarai Selvan
b544b87b9b Merge pull request #223 from hasura/jberryman/5863-prep-refactoring
GitOrigin-RevId: 71b1453edf4b93ffc16a15ea3c6057bb865b6606
2020-12-20 06:53:38 +00:00
Rakesh Emmadi
a153e96309 server: expand metadata storage class with async actions and core metadata operations (#184)
An incremental PR towards https://github.com/hasura/graphql-engine/pull/5797
- Expands `MonadMetadataStorage` with operations related to async actions and setting/updating metadata

GitOrigin-RevId: 53386b7b2d007e162050b826d0708897f0b4c8f6
2020-12-14 04:31:20 +00:00
Rakesh Emmadi
a2cf9a53c2 server: move to storing metadata as a json blob (#115)
GitOrigin-RevId: 3d1a7618a4ec086c2d255549a6c15087201e9ab0
2020-12-08 14:23:28 +00:00
hasura-bot
3451413d1a server: do not block catalog migration on inconsistent metadata (#139)
Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
GITHUB_PR_NUMBER: 6286
GITHUB_PR_URL: https://github.com/hasura/graphql-engine/pull/6286

Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
GitOrigin-RevId: 27bbfc960da87e4ad54bad5943782f28aa113874
2020-12-04 08:39:05 +00:00
Phil Freeman
20913c254d server: allocation limits (#82)
* server: allocation limits

* Move most of the allocation limits work into Pro

Co-authored-by: Brandon Simmons <brandon@hasura.io>

* code review changes

* fix build

Co-authored-by: Brandon Simmons <brandon@hasura.io>
GitOrigin-RevId: 035f15def48f4a0376cb2701eff9982c43a78e2d
2020-12-03 04:07:22 +00:00
Auke Booij
86aef93d31 server: remove a few unused Lift instances (#114)
GitOrigin-RevId: b5679e9579d226f7209b73ac966d29391c308c06
2020-11-26 13:58:07 +00:00
hasura-bot
29925eb08d server: metadata storage abstraction for scheduled triggers
An incremental PR towards https://github.com/hasura/graphql-engine/pull/5797

* metadata storage abstraction for scheduled triggers

Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Auke Booij <auke@hasura.io>
GITHUB_PR_NUMBER: 6131
GITHUB_PR_URL: https://github.com/hasura/graphql-engine/pull/6131

* update pro server code

Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Auke Booij <auke@hasura.io>
GitOrigin-RevId: 17244a47b3e8633acf2492e0b0734b72025f0a09
2020-11-25 10:57:38 +00:00
hasura-bot
d955001046 server: schema cache syncing without relying on a Postgres table (#40)
An incremental PR towards # 5797 (OSS repo)

* schema cache syncing without relying on a Postgres table

Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
GITHUB_PR_NUMBER: 6173
GITHUB_PR_URL: https://github.com/hasura/graphql-engine/pull/6173

* fix pro compile issues

Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
GitOrigin-RevId: 055b06ae477af80892124ff55d005d04a8034c34
2020-11-24 06:11:07 +00:00
Vishnu Bharathi P
58c44f55dd Merge oss/master onto mono/main
GitOrigin-RevId: 1c8c4d60e033c8a0bc8b2beed24c5bceb7d4bcc8
2020-11-12 22:37:19 +05:30
Vishnu Bharathi P
666058ab7f oss: renames dot files and folders
GitOrigin-RevId: 540aeec3be091e1cfb7b05a988f50445534ed663
2020-11-12 22:37:19 +05:30
Antoine Leblanc
7960cd1c87
server: generalize table name IR (#6127)
https://github.com/hasura/graphql-engine/pull/6127
2020-11-10 07:50:26 +00:00
Antoine Leblanc
39843e48f2
server: split RQL.Types.DML (#6101)
https://github.com/hasura/graphql-engine/pull/6101
2020-11-09 10:03:38 +00:00
Rakesh Emmadi
8bf84dff17
Avoid hdb_views schema for storing event trigger procedures (#6135)
https://github.com/hasura/graphql-engine/pull/6135
2020-11-05 12:12:13 +00:00
Auke Booij
3bcde3d4b8
server: metadata separation: reorganize metadata types (#6103)
https://github.com/hasura/graphql-engine/pull/6103
2020-11-03 18:01:33 +00:00
Sasha Bogicevic
81e836a12c
server: configurable websocket keep alive interval (#6092)
Accept new server flag --websocket-keepalive to control
websockets keep-alive interval

Co-authored-by: Auke Booij <auke@hasura.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2020-11-03 18:04:48 +01:00
Antoine Leblanc
88bd2ed022
server: generalize BoolExp (#6082)
https://github.com/hasura/graphql-engine/pull/6082
2020-11-02 11:50:40 +00:00
Auke Booij
62da89eb9d
server: multiple sources: generalize LazyTx to LazyTxT (#6097)
https://github.com/hasura/graphql-engine/pull/6097
2020-10-30 11:00:39 +00:00
Karthikeyan Chinnakonda
2cb08a89cb
server: customize tracking tables with a custom name (#5719)
https://github.com/hasura/graphql-engine/pull/5719
2020-10-29 12:48:45 +00:00
Antoine Leblanc
8b577adfae
move NonEmptyText out of RQL.Types.Common (#6086) 2020-10-29 00:04:21 +00:00
Auke Booij
0540b279db
server: make more use of hlint (#6059)
https://github.com/hasura/graphql-engine/pull/6059
2020-10-28 16:40:33 +00:00
Antoine Leblanc
a8ed6a82e2
server: move Hasura.SQL to Hasura.Backends.Postgres (#6053)
https://github.com/hasura/graphql-engine/pull/6053
2020-10-27 13:53:49 +00:00
Auke Booij
8d0afb6f92
server: SQL that grows (#6003)
Add a backend type extension parameter to some RQL types, following the ideas of the paper "Trees that grow" (Najd & Jones 2016)

Co-authored-by: Antoine Leblanc <antoine@hasura.io>
Co-authored-by: kodiakhq[bot] <49736102+kodiakhq[bot]@users.noreply.github.com>
2020-10-22 22:42:27 +02:00
Antoine Leblanc
91d8a7ab61 remove SQL.Text, clean text functions 2020-10-22 14:07:48 +01:00
gahag
19b4f55ca1
server: implement websocket compression setting (fixes #3292) (#5928)
https://github.com/hasura/graphql-engine/pull/5928
2020-10-12 09:14:23 +00:00
Auke Booij
84a129c8e4
server: heterogeneous execution of GraphQL queries (#5869)
https://github.com/hasura/graphql-engine/pull/5869
2020-10-07 10:23:17 +00:00
Sameer Kolhar
af32ccff36
server: limit length of event trigger names (close #5786) (#5826)
https://github.com/hasura/graphql-engine/pull/5826
2020-10-06 15:22:09 +00:00
Sameer Kolhar
10f41e7559
server: accept only non-negative integers for batch size and refetch interval (close #5653) (#5759)
https://github.com/hasura/graphql-engine/pull/5759
2020-09-17 10:56:41 +00:00
Phil Freeman
7302abeed0
server: Add MonadQueryInstrumentation (#5644)
https://github.com/hasura/graphql-engine/pull/5644
2020-09-17 00:23:39 +00:00
Anon Ray
41c819881d
Merge branch 'master' into disable-opt-server-init 2020-09-10 16:31:59 +05:30
Phil Freeman
1b2cda3d30 Move server_timestamp_ms out into Main.hs 2020-09-08 11:13:35 -07:00
Phil Freeman
6b61aef2d8 server: move registerGcMetrics to runApp 2020-09-08 09:19:52 -07:00
Anon Ray
893a45a95a server: disable optimization for Hasura.Server.Init module
- builds on CircleCI seems to run out of memory. Hence disabling
  optimizations for this module.
2020-09-07 12:10:02 +05:30
Rakesh Emmadi
4ce6002af2
support customizing JWT claims (close #3485) (#3575)
* improve jsonpath parser to accept special characters and property tests for the same

* make the JWTClaimsMapValueG parametrizable

* add documentation in the JWT file

* modify processAuthZHeader

Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
Co-authored-by: Marion Schleifer <marion@hasura.io>
2020-08-31 22:10:01 +05:30
Alexis King
8c29f15b4d
server: Bump GHC version to 8.10.2 (#5659) 2020-08-26 23:16:23 -05:00
Alexis King
7e970177c1
Rewrite GraphQL schema generation and query parsing (close #2801) (#4111)
Aka “the PDV refactor.” History is preserved on the branch 2801-graphql-schema-parser-refactor.

* [skip ci] remove stale benchmark commit from commit_diff

* [skip ci] Check for root field name conflicts between remotes

* [skip ci] Additionally check for conflicts between remotes and DB

* [skip ci] Check for conflicts in schema when tracking a table

* [skip ci] Fix equality checking in GraphQL AST

* server: fix mishandling of GeoJSON inputs in subscriptions (fix #3239) (#4551)

* Add support for multiple top-level fields in a subscription to improve testability of subscriptions

* Add an internal flag to enable multiple subscriptions

* Add missing call to withConstructorFn in live queries (fix #3239)

Co-authored-by: Alexis King <lexi.lambda@gmail.com>

* Scheduled triggers (close #1914) (#3553)

server: add scheduled triggers

Co-authored-by: Alexis King <lexi.lambda@gmail.com>
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>

* dev.sh: bump version due to addition of croniter python dependency

* server: fix an introspection query caching issue (fix #4547) (#4661)

Introspection queries accept variables, but we need to make sure to
also touch the variables that we ignore, so that an introspection
query is marked not reusable if we are not able to build a correct
query plan for it.

A better solution here would be to deal with such unused variables
correctly, so that more introspection queries become reusable.

An even better solution would be to type-safely track *how* to reuse
which variables, rather than to split the reusage marking from the
planning.

Co-authored-by: Tirumarai Selvan <tiru@hasura.io>

* flush log buffer on exception in mkWaiApp ( fix #4772 ) (#4801)

* flush log buffer on exception in mkWaiApp

* add comment to explain the introduced change

* add changelog

* allow logging details of a live query polling thread (#4959)

* changes for poller-log

add various multiplexed query info in poller-log

* minor cleanup, also fixes a bug which will return duplicate data

* Live query poller stats can now be logged

This also removes in-memory stats that are collected about batched
query execution as the log lines when piped into an monitoring tool
will give us better insights.

* allow poller-log to be configurable

* log minimal information in the livequery-poller-log

Other information can be retrieved from /dev/subscriptions/extended

* fix few review comments

* avoid marshalling and unmarshalling from ByteString to EncJSON

* separate out SubscriberId and SubscriberMetadata

Co-authored-by: Anon Ray <rayanon004@gmail.com>

* Don't compile in developer APIs by default

* Tighten up handling of admin secret, more docs

Store the admin secret only as a hash to prevent leaking the secret
inadvertently, and to prevent timing attacks on the secret.

NOTE: best practice for stored user passwords is a function with a
tunable cost like bcrypt, but our threat model is quite different (even
if we thought we could reasonably protect the secret from an attacker
who could read arbitrary regions of memory), and bcrypt is far too slow
(by design) to perform on each request. We'd have to rely on our
(technically savvy) users to choose high entropy passwords in any case.

Referencing #4736

* server/docs: add instructions to fix loss of float precision in PostgreSQL <= 11 (#5187)

This adds a server flag, --pg-connection-options, that can be used to set a PostgreSQL connection parameter, extra_float_digits, that needs to be used to avoid loss of data on older versions of PostgreSQL, which have odd default behavior when returning float values. (fixes #5092)

* [skip ci] Add new commits from master to the commit diff

* [skip ci] serve default directives (skip & include) over introspection

* [skip ci] Update non-Haskell assets with the version on master

* server: refactor GQL execution check and config API (#5094)

Co-authored-by: Vamshi Surabhi <vamshi@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* [skip ci] fix js issues in tests by pinning dependencies version

* [skip ci] bump graphql version

* [skip ci] Add note about memory usage

* generalize query execution logic on Postgres (#5110)

* generalize PGExecCtx to support specialized functions for various operations

* fix tests compilation

* allow customising PGExecCtx when starting the web server

* server: changes catalog initialization and logging for pro customization (#5139)

* new typeclass to abstract the logic of QueryLog-ing

* abstract the logic of logging websocket-server logs

  introduce a MonadWSLog typeclass

* move catalog initialization to init step

  expose a helper function to migrate catalog
  create schema cache in initialiseCtx

* expose various modules and functions for pro

* [skip ci] cosmetic change

* [skip ci] fix test calling a mutation that does not exist

* [skip ci] minor text change

* [skip ci] refactored input values

* [skip ci] remove VString Origin

* server: fix updating of headers behaviour in the update cron trigger API and create future events immediately (#5151)

* server: fix bug to update headers in an existing cron trigger and create future events

Co-authored-by: Tirumarai Selvan <tiru@hasura.io>

* Lower stack chunk size in RTS to reduce thread STACK memory (closes #5190)

This reduces memory consumption for new idle subscriptions significantly
(see linked ticket).

The hypothesis is: we fork a lot of threads per websocket, and some of
these use slightly more than the initial 1K stack size, so the first
overflow balloons to 32K, when significantly less is required.

However: running with `+RTS -K1K -xc` did not seem to show evidence of
any overflows! So it's a mystery why this improves things.

GHC should probably also be doubling the stack buffer at each overflow
or doing something even smarter; the knobs we have aren't so helpful.

* [skip ci] fix todo and schema generation for aggregate fields

* 5087 libpq pool leak (#5089)

Shrink libpq buffers to 1MB before returning connection to pool. Closes #5087

See: https://github.com/hasura/pg-client-hs/pull/19

Also related: #3388 #4077

* bump pg-client-hs version (fixes a build issue on some environments) (#5267)

* do not use prepared statements for mutations

* server: unlock scheduled events on graceful shutdown (#4928)

* Fix buggy parsing of new --conn-lifetime flag in 2b0e3774

* [skip ci] remove cherry-picked commit from commit_diff.txt

* server: include additional fields in scheduled trigger webhook payload (#5262)

* include scheduled triggers metadata in the webhook body

Co-authored-by: Tirumarai Selvan <tiru@hasura.io>

* server: call the webhook asynchronously in event triggers (#5352)

* server: call the webhook asynchronosly in event triggers

* Expose all modules in Cabal file (#5371)

* [skip ci] update commit_diff.txt

* [skip ci] fix cast exp parser & few TODOs

* [skip ci] fix remote fields arguments

* [skip ci] fix few more TODO, no-op refactor, move resolve/action.hs to execute/action.hs

* Pass environment variables around as a data structure, via @sordina (#5374)

* Pass environment variables around as a data structure, via @sordina

* Resolving build error

* Adding Environment passing note to changelog

* Removing references to ILTPollerLog as this seems to have been reintroduced from a bad merge

* removing commented-out imports

* Language pragmas already set by project

* Linking async thread

* Apply suggestions from code review

Use `runQueryTx` instead of `runLazyTx` for queries.

* remove the non-user facing entry in the changelog

Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* [skip ci] fix: restrict remote relationship field generation for hasura queries

* [skip ci] no-op refactor; move insert execution code from schema parser module

* server: call the webhook asynchronously in event triggers (#5352)

* server: call the webhook asynchronosly in event triggers

* Expose all modules in Cabal file (#5371)

* [skip ci] update commit_diff.txt

* Pass environment variables around as a data structure, via @sordina (#5374)

* Pass environment variables around as a data structure, via @sordina

* Resolving build error

* Adding Environment passing note to changelog

* Removing references to ILTPollerLog as this seems to have been reintroduced from a bad merge

* removing commented-out imports

* Language pragmas already set by project

* Linking async thread

* Apply suggestions from code review

Use `runQueryTx` instead of `runLazyTx` for queries.

* remove the non-user facing entry in the changelog

Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* [skip ci] implement header checking

Probably closes #14 and #3659.

* server: refactor 'pollQuery' to have a hook to process 'PollDetails' (#5391)

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* update pg-client (#5421)

* [skip ci] update commit_diff

* Fix latency buckets for telemetry data

These must have gotten messed up during a refactor. As a consequence
almost all samples received so far fall into the single erroneous 0 to
1K seconds (originally supposed to be 1ms?) bucket.

I also re-thought what the numbers should be, but these are still
arbitrary and might want adjusting in the future.

* [skip ci] include the latest commit compared against master in commit_diff

* [skip ci] include new commits from master in commit_diff

* [skip ci] improve description generation

* [skip ci] sort all introspect arrays

* [skip ci] allow parsers to specify error codes

* [skip ci] fix integer and float parsing error code

* [skip ci] scalar from json errors are now parse errors

* [skip ci] fixed negative integer error message and code

* [skip ci] Re-fix nullability in relationships

* [skip ci] no-op refactor and removed couple of FIXMEs

* [skip ci] uncomment code in 'deleteMetadataObject'

* [skip ci] Fix re-fix of nullability for relationships

* [skip ci] fix default arguments error code

* [skip ci] updated test error message

!!! WARNING !!!
Since all fields accept `null`, they all are technically optional in
the new schema. Meaning there's no such thing as a missing mandatory
field anymore: a field that doesn't have a default value, and which
therefore isn't labelled as "optional" in the schema, will be assumed
to be null if it's missing, meaning it isn't possible anymore to have
an error for a missing mandatory field. The only possible error is now
when a optional positional argument is omitted but is not the last
positional argument.

* [skip ci] cleanup of int scalar parser

* [skip ci] retro-compatibility of offset as string

* [skip ci] Remove commit from commit_diff.txt

Although strictly speaking we don't know if this will work correctly in PDV
if we would implement query plan caching, the fact is that in the theoretical
case that we would have the same issue in PDV, it would probably apply not just
to introspection, and the fix would be written completely differently.  So this
old commit is of no value to us other than the heads-up "make sure query plan
caching works correctly even in the presence of unused variables", which is
already part of the test suite.

* Add MonadTrace and MonadExecuteQuery abstractions (#5383)

* [skip ci] Fix accumulation of input object types

Just like object types, interface types, and union types, we have to avoid
circularities when collecting input types from the GraphQL AST.

Additionally, this fixes equality checks for input object types (whose fields
are unordered, and hence should be compared as sets) and enum types (ditto).

* [skip ci] fix fragment error path

* [skip ci] fix node error code

* [skip ci] fix paths in insert queries

* [skip ci] fix path in objects

* [skip ci] manually alter node id path for consistency

* [skip ci] more node error fixups

* [skip ci] one last relay error message fix

* [skip ci] update commit_diff

* Propagate the trace context to event triggers (#5409)

* Propagate the trace context to event triggers

* Handle missing trace and span IDs

* Store trace context as one LOCAL

* Add migrations

* Documentation

* changelog

* Fix warnings

* Respond to code review suggestions

* Respond to code review

* Undo changelog

* Update CHANGELOG.md

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* server: log request/response sizes for event triggers (#5463)

* server: log request/response sizes for event triggers

  event triggers (and scheduled triggers) now have request/response size
  in their logs.

* add changelog entry

* Tracing: Simplify HTTP traced request (#5451)

Remove the Inversion of Control (SuspendRequest) and simplify
the tracing of HTTP Requests.

Co-authored-by: Phil Freeman <phil@hasura.io>

* Attach request ID as tracing metadata (#5456)

* Propagate the trace context to event triggers

* Handle missing trace and span IDs

* Store trace context as one LOCAL

* Add migrations

* Documentation

* Include the request ID as trace metadata

* changelog

* Fix warnings

* Respond to code review suggestions

* Respond to code review

* Undo changelog

* Update CHANGELOG.md

* Typo

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* server: add logging for action handlers (#5471)

* server: add logging for action handlers

* add changelog entry

* change action-handler log type from internal to non-internal

* fix action-handler-log name

* server: pass http and websocket request to logging context (#5470)

* pass request body to logging context in all cases

* add message size logging on the websocket API

  this is required by graphql-engine-pro/#416

* message size logging on websocket API

  As we need to log all messages recieved/sent by the websocket server,
  it makes sense to log them as part of the websocket server event logs.
  Previously message recieved were logged inside the onMessage handler,
  and messages sent were logged only for "data" messages (as a server event log)

* fix review comments

Co-authored-by: Phil Freeman <phil@hasura.io>

* server: stop eventing subsystem threads when shutting down (#5479)

* server: stop eventing subsystem threads when shutting down

* Apply suggestions from code review

Co-authored-by: Karthikeyan Chinnakonda <chkarthikeyan95@gmail.com>

Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Karthikeyan Chinnakonda <chkarthikeyan95@gmail.com>

* [skip ci] update commit_diff with new commits added in master

* Bugfix to support 0-size HASURA_GRAPHQL_QUERY_PLAN_CACHE_SIZE

Also some minor refactoring of bounded cache module:
 - the maxBound check in `trim` was confusing and unnecessary
 - consequently trim was unnecessary for lookupPure

Also add some basic tests

* Support only the bounded cache, with default HASURA_GRAPHQL_QUERY_PLAN_CACHE_SIZE of 4000. Closes #5363

* [skip ci] remove merge commit from commit_diff

* server: Fix compiler warning caused by GHC upgrade (#5489)

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* [skip ci] update all non server code from master

* [skip ci] aligned object field error message with master

* [skip ci] fix remaining undefined?

* [skip ci] remove unused import

* [skip ci] revert to previous error message, fix tests

* Move nullableType/nonNullableType to Schema.hs

These are functions on Types, not on Parsers.

* [skip ci] fix setup to fix backend only test

the order in which permission checks are performed on the branch is
slightly different than on master, resulting in a slightly different
error if there are no other mutations the user has access to. By
adding update permissions, we go back to the expected case.

* [skip ci] fix insert geojson tests to reflect new paths

* [skip ci] fix enum test for better error message

* [skip ci] fix header test for better error message

* [skip ci] fix fragment cycle test for better error message

* [skip ci] fix error message for type mismatch

* [skip ci] fix variable path in test

* [skip ci] adjust tests after bug fix

* [skip ci] more tests fixing

* Add hdb_catalog.current_setting abstraction for reading Hasura settings

As the comment in the function’s definition explains, this is needed to
work around an awkward Postgres behavior.

* [skip ci] Update CONTRIBUTING.md to mention Node setup for Python tests

* [skip ci] Add missing Python tests env var to CONTRIBUTING.md

* [skip ci] fix order of result when subscription is run with multiple nodes

* [skip ci] no-op refactor: fix a warning in Internal/Parser.hs

* [skip ci] throw error when a subscription contains remote joins

* [skip ci] Enable easier profiling by hiding AssertNF behind a flag

In order to compile a profiling build, run:

$ cabal new-build -f profiling --enable-profiling

* [skip ci] Fix two warnings

We used to lookup the objects that implement a given interface by filtering all
objects in the schema document.  However, one of the tests expects us to
generate a warning if the provided `implements` field of an introspection query
specifies an object not implementing some interface.  So we use that field
instead.

* [skip ci] Fix warnings by commenting out query plan caching

* [skip ci] improve masking/commenting query caching related code & few warning fixes

* [skip ci] Fixed compiler warnings in graphql-parser-hs

* Sync non-Haskell assets with master

* [skip ci] add a test inserting invalid GraphQL but valid JSON value in a jsonb column

* [skip ci] Avoid converting to/from Map

* [skip ci] Apply some hlint suggestions

* [skip ci] remove redundant constraints from buildLiveQueryPlan and explainGQLQuery

* [skip ci] add NOTEs about missing Tracing constraints in PDV from master

* Remove -fdefer-typed-holes, fix warnings

* Update cabal.project.freeze

* Limit GHC’s heap size to 8GB in CI to avoid the OOM killer

* Commit package-lock.json for Python tests’ remote schema server

* restrict env variables start with HASURA_GRAPHQL_ for headers configuration in actions, event triggers & remote schemas (#5519)

* restrict env variables start with HASURA_GRAPHQL_ for headers definition in actions & event triggers

* update CHANGELOG.md

* Apply suggestions from code review

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* add test for table_by_pk node when roles doesn't have permission to PK

* [skip ci] fix introspection query if any enum column present in primary key (fix #5200) (#5522)

* [skip ci] test case fix for a6450e126b

* [skip ci] add tests to agg queries when role doesn't have access to any cols

* fix backend test

* Simplify subscription execution

* [skip ci] add test to check if required headers are present while querying

* Suppose, table B is related to table A and to query B certain headers are
  necessary, then the test checks that we are throwing error when the header
  is not set when B is queried through A

* fix mutations not checking for view mutability

* [skip ci] add variable type checking and corresponding tests

* [skip ci] add test to check if update headers are present while doing an upsert

* [skip ci] add positive counterparts to some of the negative permission tests

* fix args missing their description in introspect

* [skip ci] Remove unused function; insert missing markNotReusable call

* [skip ci] Add a Note about InputValue

* [skip ci] Delete LegacySchema/ 🎉

* [skip ci] Delete GraphQL/{Resolve,Validate}/ 🎉

* [skip ci] Delete top-level Resolve/Validate modules; tidy .cabal file

* [skip ci] Delete LegacySchema top-level module

Somehow I missed this one.

* fix input value to json

* [skip ci] elaborate on JSON objects in GraphQL

* [skip ci] add missing file

* [skip ci] add a test with subscription containing remote joins

* add a test with remote joins in mutation output

* [skip ci] Add some comments to Schema/Mutation.hs

* [skip ci] Remove no longer needed code from RemoteServer.hs

* [skip ci] Use a helper function to generate conflict clause parsers

* [skip ci] fix type checker error in fields with default value

* capitalize the header keys in select_articles_without_required_headers

* Somehow, this was the reason the tests were failing. I have no idea, why!

* [skip ci] Add a long Note about optional fields and nullability

* Improve comments a bit; simplify Schema/Common.hs a bit

* [skip ci] full implementation of 5.8.5 type checking.

* [skip ci] fix validation test teardown

* [skip ci] fix schema stitching test

* fix remote schema ignoring enum nullability

* [skip ci] fix fieldOptional to not discard nullability

* revert nullability of use_spheroid

* fix comment

* add required remote fields with arguments for tests

* [skip ci] add missing docstrings

* [skip ci] fixed description of remote fields

* [skip ci] change docstring for consistency

* fix several schema inconsistencies

* revert behaviour change in function arguments parsing

* fix remaining nullability issues in new schema

* minor no-op refactor; use isListType from graphql-parser-hs

* use nullability of remote schema node, while creating a Remote reln

* fix 'ID' input coercing & action 'ID' type relationship mapping

* include ASTs in MonadExecuteQuery

* needed for PRO code-base

* Delete code for "interfaces implementing ifaces" (draft GraphQL spec)

Previously I started writing some code that adds support for a future GraphQL
feature where interfaces may themselves be sub-types of other interfaces.
However, this code was incomplete, and partially incorrect.  So this commit
deletes support for that entirely.

* Ignore a remote schema test during the upgrade/downgrade test

The PDV refactor does a better job at exposing a minimal set of types through
introspection.  In particular, not every type that is present in a remote schema
is re-exposed by Hasura.  The test
test_schema_stitching.py::TestRemoteSchemaBasic::test_introspection assumed that
all types were re-exposed, which is not required for GraphQL compatibility, in
order to test some aspect of our support for remote schemas.

So while this particular test has been updated on PDV, the PDV branch now does
not pass the old test, which we argue to be incorrect.  Hence this test is
disabled while we await a release, after which we can re-enable it.

This also re-enables a test that was previously disabled for similar, though
unrelated, reasons.

* add haddock documentation to the action's field parsers

* Deslecting some tests in server-upgrade

Some tests with current build are failing on server upgrade
which it should not. The response is more accurate than
what it was.

Also the upgrade tests were not throwing errors when the test is
expected to return an error, but succeeds. The test framework is
patched to catch this case.

* [skip ci] Add a long Note about interfaces and object types

* send the response headers back to client after running a query

* Deselect a few more tests during upgrade/downgrade test

* Update commit_diff.txt

* change log kind from db_migrate to catalog_migrate (#5531)

* Show method and complete URI in traced HTTP calls (#5525)

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* restrict env variables start with HASURA_GRAPHQL_ for headers configuration in actions, event triggers & remote schemas (#5519)

* restrict env variables start with HASURA_GRAPHQL_ for headers definition in actions & event triggers

* update CHANGELOG.md

* Apply suggestions from code review

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>

* fix introspection query if any enum column present in primary key (fix #5200) (#5522)

* Fix telemetry reporting of transport (websocket was reported as http)

* add log kinds in cli-migrations image (#5529)

* add log kinds in cli-migrations image

* give hint to resolve timeout error

* minor changes and CHANGELOG

* server: set hasura.tracecontext in RQL mutations [#5542] (#5555)

* server: set hasura.tracecontext in RQL mutations [#5542]

* Update test suite

Co-authored-by: Tirumarai Selvan <tiru@hasura.io>

* Add bulldozer auto-merge and -update configuration

We still need to add the github app (as of time of opening this PR)

Afterwards devs should be able to allow bulldozer to automatically
"update" the branch, merging in parent when it changes, as well as
automatically merge when all checks pass.

This is opt-in by adding the `auto-update-auto-merge` label to the PR.

* Remove 'bulldozer' config, try 'kodiak' for auto-merge

see: https://github.com/chdsbd/kodiak

The main issue that bit us was not being able to auto update forked
branches, also:
https://github.com/palantir/bulldozer/issues/66
https://github.com/palantir/bulldozer/issues/145

* Cherry-picked all commits

* [skip ci] Slightly improve formatting

* Revert "fix introspection query if any enum column present in primary key (fix #5200) (#5522)"

This reverts commit 0f9a5afa59.

This undoes a cherry-pick of 34288e1eb5 that was
already done previously in a6450e126b, and
subsequently fixed for PDV in 70e89dc250

* Do a small bit of tidying in Hasura.GraphQL.Parser.Collect

* Fix cherry-picking work

Some previous cherry-picks ended up modifying code that is commented out

* [skip ci] clarified comment regarding insert representation

* [skip ci] removed obsolete todos

* cosmetic change

* fix action error message

* [skip ci] remove obsolete comment

* [skip ci] synchronize stylish haskell extensions list

* use previously defined scalar names in parsers rather than ad-hoc literals

* Apply most syntax hlint hints.

* Clarify comment on update mutation.

* [skip ci] Clarify what fields should be specified for objects

* Update "_inc" description.

* Use record types rather than tuples fo IntrospectionResult and ParsedIntrospection

* Get rid of checkFieldNamesUnique (use Data.List.Extended.duplicates)

* Throw more errors when collecting query root names

* [skip ci] clean column parser comment

* Remove dead code inserted in ab65b39

* avoid converting to non-empty list where not needed

* add note and TODO about the disabled checks in PDV

* minor refactor in remoteField' function

* Unify two getObject methods

* Nitpicks in Remote.hs

* Update CHANGELOG.md

* Revert "Unify two getObject methods"

This reverts commit bd6bb40355.

We do need two different getObject functions as the corresponding error message is different

* Fix error message in Remote.hs

* Update CHANGELOG.md

Co-authored-by: Auke Booij <auke@tulcod.com>

* Apply suggested Changelog fix.

Co-authored-by: Auke Booij <auke@tulcod.com>

* Fix typo in Changelog.

* [skip ci] Update changelog.

* reuse type names to avoid duplication

* Fix Hashable instance for Definition

The presence of `Maybe Unique`, and an optional description, as part of
`Definition`s, means that `Definition`s that are considered `Eq`ual may get
different hashes.  This can happen, for instance, when one object is memoized
but another is not.

* [skip ci] Update commit_diff.txt

* Bump parser version.

* Bump freeze file after changes in parser.

* [skip ci] Incorporate commits from master

* Fix developer flag in server/cabal.project.freeze

Co-authored-by: Auke Booij <auke@tulcod.com>

* Deselect a changed ENUM test for upgrade/downgrade CI

* Deselect test here as well

* [skip ci] remove dead code

* Disable more tests for upgrade/downgrade

* Fix which test gets deselected

* Revert "Add hdb_catalog.current_setting abstraction for reading Hasura settings"

This reverts commit 66e85ab9fb.

* Remove circular reference in cabal.project.freeze

Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
Co-authored-by: Auke Booij <auke@hasura.io>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>
Co-authored-by: Brandon Simmons <brandon.m.simmons@gmail.com>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
Co-authored-by: Anon Ray <rayanon004@gmail.com>
Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Anon Ray <ecthiender@users.noreply.github.com>
Co-authored-by: Vamshi Surabhi <vamshi@hasura.io>
Co-authored-by: Antoine Leblanc <antoine@hasura.io>
Co-authored-by: Brandon Simmons <brandon@hasura.io>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Lyndon Maydwell <lyndon@sordina.net>
Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Naveen Naidu <naveennaidu479@gmail.com>
Co-authored-by: Karthikeyan Chinnakonda <chkarthikeyan95@gmail.com>
Co-authored-by: Nizar Malangadan <nizar-m@users.noreply.github.com>
Co-authored-by: Antoine Leblanc <crucuny@gmail.com>
Co-authored-by: Auke Booij <auke@tulcod.com>
2020-08-21 12:27:01 -05:00
Naveen Naidu
b14a4f255a
server: Pass EKG Metrics Store as argument to runHGEServer (#5560)
* Add ekg-core to build-executable .cabal

* Move creation of EKG Store to Main.hs

This helps to share metrics between pro and OSS and
helps surface the metrics from OSS in Datadog via
Pro.

Co-authored-by: Phil Freeman <phil@hasura.io>
2020-08-18 12:53:12 -07:00
Phil Freeman
b2561a719b
server: set hasura.tracecontext in RQL mutations [#5542] (#5555)
* server: set hasura.tracecontext in RQL mutations [#5542]

* Update test suite

Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
2020-08-10 12:34:24 -07:00
Tirumarai Selvan
eb80d97b0a
change log kind from db_migrate to catalog_migrate (#5531) 2020-08-05 15:53:14 +05:30
Brandon Simmons
b973479631
Merge branch 'master' into 5363-default-bounded-plan-cache 2020-07-29 12:09:21 -04:00
Anon Ray
046a783a14
server: pass http and websocket request to logging context (#5470)
* pass request body to logging context in all cases

* add message size logging on the websocket API

  this is required by graphql-engine-pro/#416

* message size logging on websocket API

  As we need to log all messages recieved/sent by the websocket server,
  it makes sense to log them as part of the websocket server event logs.
  Previously message recieved were logged inside the onMessage handler,
  and messages sent were logged only for "data" messages (as a server event log)

* fix review comments

Co-authored-by: Phil Freeman <phil@hasura.io>
2020-07-29 20:18:36 +05:30
Anon Ray
6c7e63791f
server: add logging for action handlers (#5471)
* server: add logging for action handlers

* add changelog entry

* change action-handler log type from internal to non-internal

* fix action-handler-log name
2020-07-29 19:00:29 +05:30
Brandon Simmons
2a0768d7ad
Merge branch 'master' into 5363-default-bounded-plan-cache 2020-07-28 20:23:26 -04:00
Brandon Simmons
1d4ec4eafb Support only the bounded cache, with default HASURA_GRAPHQL_QUERY_PLAN_CACHE_SIZE of 4000. Closes #5363 2020-07-28 19:02:44 -04:00
Phil Freeman
df51a8eb18
Attach request ID as tracing metadata (#5456)
* Propagate the trace context to event triggers

* Handle missing trace and span IDs

* Store trace context as one LOCAL

* Add migrations

* Documentation

* Include the request ID as trace metadata

* changelog

* Fix warnings

* Respond to code review suggestions

* Respond to code review

* Undo changelog

* Update CHANGELOG.md

* Typo

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-07-28 13:06:54 -07:00
Naveen Naidu
664e9df9c6
Tracing: Simplify HTTP traced request (#5451)
Remove the Inversion of Control (SuspendRequest) and simplify
the tracing of HTTP Requests.

Co-authored-by: Phil Freeman <phil@hasura.io>
2020-07-28 11:51:56 -07:00
Brandon Simmons
2eab6a89aa Fix latency buckets for telemetry data
These must have gotten messed up during a refactor. As a consequence
almost all samples received so far fall into the single erroneous 0 to
1K seconds (originally supposed to be 1ms?) bucket.

I also re-thought what the numbers should be, but these are still
arbitrary and might want adjusting in the future.
2020-07-22 12:29:38 -04:00
Anon Ray
1eb36bbbb3
server: refactor 'pollQuery' to have a hook to process 'PollDetails' (#5391)
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-07-16 18:49:42 +05:30
Phil Freeman
0dddbe9e9d
Add MonadTrace and MonadExecuteQuery abstractions (#5383)
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-07-15 16:10:48 +05:30
Lyndon Maydwell
24592a516b
Pass environment variables around as a data structure, via @sordina (#5374)
* Pass environment variables around as a data structure, via @sordina

* Resolving build error

* Adding Environment passing note to changelog

* Removing references to ILTPollerLog as this seems to have been reintroduced from a bad merge

* removing commented-out imports

* Language pragmas already set by project

* Linking async thread

* Apply suggestions from code review

Use `runQueryTx` instead of `runLazyTx` for queries.

* remove the non-user facing entry in the changelog

Co-authored-by: Phil Freeman <paf31@cantab.net>
Co-authored-by: Phil Freeman <phil@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-07-14 12:00:58 -07:00
Rakesh Emmadi
2fe353a294
allow array relation connection fields regardless of aggregation permission & change relay endpoint to '/v1beta1/relay' (fix #5218) (#5257)
* fix error when array relation connections are queried, fix #5218

* change relay endpoint to '/v1beta1/relay'

* Update CHANGELOG.md

Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
2020-07-03 12:00:35 +05:30
Brandon Simmons
9e423a3c55 Fix buggy parsing of new --conn-lifetime flag in 2b0e3774 2020-07-02 13:27:46 -04:00
Brandon Simmons
2b0e3774a3
5087 libpq pool leak (#5089)
Shrink libpq buffers to 1MB before returning connection to pool. Closes #5087

See: https://github.com/hasura/pg-client-hs/pull/19

Also related: #3388 #4077
2020-07-01 09:23:10 +05:30
Auke Booij
bc3d735bf3
server/docs: add instructions to fix loss of float precision in PostgreSQL <= 11 (#5187)
This adds a server flag, --pg-connection-options, that can be used to set a PostgreSQL connection parameter, extra_float_digits, that needs to be used to avoid loss of data on older versions of PostgreSQL, which have odd default behavior when returning float values. (fixes #5092)
2020-06-30 10:39:25 +02:00
Anon Ray
a7a60c2dfe
server: changes catalog initialization and logging for pro customization (#5139)
* new typeclass to abstract the logic of QueryLog-ing

* abstract the logic of logging websocket-server logs

  introduce a MonadWSLog typeclass

* move catalog initialization to init step

  expose a helper function to migrate catalog
  create schema cache in initialiseCtx

* expose various modules and functions for pro
2020-06-19 12:12:32 +05:30
Vamshi Surabhi
6fc404329a
generalize query execution logic on Postgres (#5110)
* generalize PGExecCtx to support specialized functions for various operations

* fix tests compilation

* allow customising PGExecCtx when starting the web server
2020-06-16 23:14:59 +05:30
Anon Ray
0cf4cbc5c6
server: refactor GQL execution check and config API (#5094)
Co-authored-by: Vamshi Surabhi <vamshi@hasura.io>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-06-16 20:53:06 +05:30
Brandon Simmons
5e37350561 Refactor and unit test authentication code paths (closes #4736)
The bulk of changes here is some shifting of code around and a little
parameterizing of functions for easier testing.

Also: comments, some renaming for clarity/less-chance-for-misue.
2020-06-08 13:10:58 -04:00
Brandon Simmons
d747bc1148 Tighten up handling of admin secret, more docs
Store the admin secret only as a hash to prevent leaking the secret
inadvertently, and to prevent timing attacks on the secret.

NOTE: best practice for stored user passwords is a function with a
tunable cost like bcrypt, but our threat model is quite different (even
if we thought we could reasonably protect the secret from an attacker
who could read arbitrary regions of memory), and bcrypt is far too slow
(by design) to perform on each request. We'd have to rely on our
(technically savvy) users to choose high entropy passwords in any case.

Referencing #4736
2020-06-08 13:09:25 -04:00
Vamshi Surabhi
2a9bc2354f
add relay modern support (#4458)
* validation support for unions and interfaces

* refactor SQL generation logic for improved readability

* '/v1/relay' endpoint for relay schema

* implement 'Node' interface and top level 'node' field resolver

* add relay toggle on graphiql

* fix explain api response & index plan id with query type

* add hasura mutations to relay

* add relay pytests

* update CHANGELOG.md

Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Rishichandra Wawhal <rishi@hasura.io>
Co-authored-by: Rikin Kachhia <54616969+rikinsk@users.noreply.github.com>
2020-06-08 17:43:01 +05:30
Aravind
edbd9c4f48
cli: load assets from server if cdn is disabled (close #3382) (#3851) 2020-06-03 09:36:23 +05:30
Phil Freeman
34a41e5632
Add some useful FromJSON instances (#4707)
These data types already had ToJSON instances, and it is useful to be able to parse the resulting JSON values back into Haskell values.
2020-06-03 05:01:50 +05:30
Vamshi Surabhi
2d2def8ecb Disables optimizations on Hasura.Server.Init module
When compiling the graphql-engine binary with `-O2`, ghc-8.10 seems to
be stuck at the module `Server.Init` while consuming `17G` of RAM (for 5
minutes at least before I forcefully terminated the compilation). With
this pragma, ghc-8.10 now takes under `12G` to compile graphql-engine
binary.
2020-06-02 10:55:36 +05:30
Tirumarai Selvan
c0d2bc6653
Remote Joins: Create relationships across database and remote schemas (#2392)
add remote joins: Create relationships across database and remote schemas (#2392)

Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>

Co-authored-by: Chris Done <chrisdone@gmail.com>
Co-authored-by: Chris Done <github@chrisdone.com>
Co-authored-by: wawhal <rishichandra.wawhal@gmail.com>
Co-authored-by: Aravind Shankar <aravind@hasura.io>
Co-authored-by: Brandon Simmons <brandon.m.simmons@gmail.com>
Co-authored-by: Rishichandra Wawhal <rishi@hasura.io>
Co-authored-by: Brandon Simmons <brandon@hasura.io>
Co-authored-by: nizar-m <19857260+nizar-m@users.noreply.github.com>
Co-authored-by: Praveen Durairaju <praveend.web@gmail.com>
Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Anon Ray <rayanon004@gmail.com>
Co-authored-by: Shahidh K Muhammed <shahidh@hasura.io>
Co-authored-by: soorajshankar <soorajshankar@users.noreply.github.com>
Co-authored-by: Sooraj Sanker <sooraj@Soorajs-MacBook-Pro.local>
Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>
2020-05-27 20:32:58 +05:30
Vamshi Surabhi
ff99b24031 fix a security issue with 1.2 2020-05-22 14:40:57 +05:30
Brandon Simmons
ff62d5e0bf Migrate to GHC 8.10, upgrade dependencies. Closes #4517
This also seems to squash a stubborn space leak we see with
subscriptions (linking to canonical #3388 for reference).

This may also fix some of the "Unexpected exception" websockets
exceptions we are now surfacing (see e.g. #4344)

Also: dev.sh: fix hpc reporting

Initial work on this done by Vamshi.
2020-05-13 19:13:02 -04:00
Tirumarai Selvan
cc8e2ccc78
Scheduled triggers (close #1914) (#3553)
server: add scheduled triggers 

Co-authored-by: Alexis King <lexi.lambda@gmail.com>
Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Karthikeyan Chinnakonda <karthikeyan@hasura.io>
Co-authored-by: Aleksandra Sikora <ola.zxcvbnm@gmail.com>
2020-05-13 18:03:16 +05:30
Auke Booij
e17e47ef8c
server: avoid code duplication using type classes (#4624)
There are two implementations of a Cache, namely a bounded and an
unbounded variant.  This can be elegantly captured in a type class.
In addition to reducing the amount of error-prone code in the
definition of the cache, this version reduces the amount of
error-prone code in usage sites of the cache, as it makes the cache
into an abstract object, so that a calling site cannot distinguish
between cache types.  Any decision about what should be cached should
be made through the interface of a cache, rather than at the callsite,
and this is captured by this variant.
2020-05-13 11:17:32 +02:00
Rakesh Emmadi
d52bfcda4e
backend only insert permissions (rfc #4120) (#4224)
* move user info related code to Hasura.User module

* the RFC #4120 implementation; insert permissions with admin secret

* revert back to old RoleName based schema maps

An attempt made to avoid duplication of schema contexts in types
if any role doesn't possess any admin secret specific schema

* fix compile errors in haskell test

* keep 'user_vars' for session variables in http-logs

* no-op refacto

* tests for admin only inserts

* update docs for admin only inserts

* updated CHANGELOG.md

* default behaviour when admin secret is not set

* fix x-hasura-role to X-Hasura-Role in pytests

* introduce effective timeout in actions async tests

* update docs for admin-secret not configured case

* Update docs/graphql/manual/api-reference/schema-metadata-api/permission.rst

Co-Authored-By: Marion Schleifer <marion@hasura.io>

* Apply suggestions from code review

Co-Authored-By: Marion Schleifer <marion@hasura.io>

* a complete iteration

backend insert permissions accessable via 'x-hasura-backend-privilege'
session variable

* console changes for backend-only permissions

* provide tooltip id; update labels and tooltips;

* requested changes

* requested changes

- remove className from Toggle component
- use appropriate function name (capitalizeFirstChar -> capitalize)

* use toggle props from definitelyTyped

* fix accidental commit

* Revert "introduce effective timeout in actions async tests"

This reverts commit b7a59c19d6.

* generate complete schema for both 'default' and 'backend' sessions

* Apply suggestions from code review

Co-Authored-By: Marion Schleifer <marion@hasura.io>

* remove unnecessary import, export Toggle as is

* update session variable in tooltip

* 'x-hasura-use-backend-only-permissions' variable to switch

* update help texts

* update docs

* update docs

* update console help text

* regenerate package-lock

* serve no backend schema when backend_only: false and header set to true

- Few type name refactor as suggested by @0x777

* update CHANGELOG.md

* Update CHANGELOG.md

* Update CHANGELOG.md

* fix a merge bug where a certain entity didn't get removed

Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Rishichandra Wawhal <rishi@hasura.io>
Co-authored-by: rikinsk <rikin.kachhia@gmail.com>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
2020-04-24 14:40:53 +05:30
Rakesh Emmadi
6f100e0009
improve debug information in actions errors response (close #4031) (#4432)
* config options for internal errors for non-admin role, close #4031

More detailed action debug info is added in response 'internal' field

* add docs

* update CHANGELOG.md

* set admin graphql errors option in ci tests, minor changes to docs

* fix tests

Don't use any auth for sync actions error tests. The request body
changes based on auth type in session_variables (x-hasura-auth-mode)

* Apply suggestions from code review

Co-Authored-By: Marion Schleifer <marion@hasura.io>

* use a new sum type to represent the inclusion of internal errors

As suggested in review by @0x777
-> Move around few modules in to specific API folder
-> Saperate types from Init.hs

* fix tests

Don't use any auth for sync actions error tests. The request body
changes based on auth type in session_variables (x-hasura-auth-mode)

* move 'HttpResponse' to 'Hasura.HTTP' module

* update change log with breaking change warning

* Update CHANGELOG.md

Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
2020-04-24 13:25:51 +05:30
Karthikeyan Chinnakonda
246a0b7ab8
server: Improve queryModifiesSchemaCache check for run_sql (#4283)
The previous check was too conservative and acquired a lock on the
schema cache in situations where it was unnecessary. This change
exposes the logic run_sql uses to determine whether to use the
metadata check to make the check more precise.
2020-04-22 16:03:23 -05:00
Karthikeyan Chinnakonda
5116e16e18
server(actions): add support for queries (close #4032) (#4309)
* add support for action queries

* a new parameter `type` is added in the ArgumentDefinition, its value
  can be either `query` or `mutation` and it defaults to the latter

* throw 400 when a query action is tried to explain

* update the actions docs to include query actions

* refactor the ToJSON and ToOrdJSON of ActionDefinition

Co-authored-by: Rishichandra Wawhal <rishi@hasura.io>
Co-authored-by: Tirumarai Selvan <tiru@hasura.io>
2020-04-16 12:55:19 +05:30
Karthikeyan Chinnakonda
a26bc80496
accept a new argument claims_namespace_path in JWT config (#4365)
* add new optional field `claims_namespace_path` in JWT config

* return value when empty array is found in executeJSONPath

* update the docs related to claims_namespace_path

* improve encodeJSONPath, add property tests for parseJSONPath

* throw error if both claims_namespace_path and claims_namespace are set

* refactor the Data.Parser.JsonPath to Data.Parser.JSONPathSpec

* update the JWT docs

Co-Authored-By: Marion Schleifer <marion@hasura.io>

Co-authored-by: Marion Schleifer <marion@hasura.io>
Co-authored-by: rakeshkky <12475069+rakeshkky@users.noreply.github.com>
Co-authored-by: Tirumarai Selvan <tirumarai.selvan@gmail.com>
2020-04-16 12:15:21 +05:30
Tirumarai Selvan
76fbe90b60
type is not required for jwk_url in JWT config (#4334)
* type is not required for jwk_url

* remove type from JWTConfig

* Omit type field in JWTConfig serialization if jwk_url is provided

* remove type from jwk_url test suite

* add changelog

* fix docs with new format

Co-authored-by: Alexis King <lexi.lambda@gmail.com>
2020-04-10 19:25:59 +05:30
Tirumarai Selvan
ca15ef8c96
Refactor initialisation and relax tx isolation levels where possible (#3901)
Co-authored-by: Alexis King <lexi.lambda@gmail.com>
2020-04-09 02:41:24 -05:00
Shahidh K Muhammed
b2461c5899
server: add more info about server to config api (#3412)
Co-authored-by: Toan Nguyen <hgiasac@gmail.com>
Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-04-06 10:23:58 +05:30
Antoine Leblanc
5b54f9d766
server: add support for webhook connection expiration (#4196)
* add expiry time to webhook user info

This also adds an optional message to webhook errors: if we fail to
parse an expiry time, we will log a warning with the parse error.

* refactored Auth

This change had one main goal: put in common all expiry time
extraction code between the JWT and WebHook parts of the
code. Furthermore, this change also moves all WebHook specific code to
its own module, similarly to what is done for JWT.

* Remove dependency on string-conversions in favor of text-conversions

string-conversions silently uses UTF8 instead of being explicit about
it, and it uses lenientDecode when decoding ByteStrings when it’s
usually better to reject invalid UTF8 input outright. text-conversions
solves both those problems.

Co-authored-by: Alexis King <lexi.lambda@gmail.com>
2020-04-02 19:00:13 -05:00
Antoine Leblanc
5b74b2e2df
server: prevent metadata checks in read-only mode (#4250)
* do not perform the metadata check in read-only mode
* improve the isAltrDropReplace regex
* quote the regex at compile-time to handle syntax errors statically

Co-authored-by: Alexis King <lexi.lambda@gmail.com>
2020-04-01 18:22:55 -05:00
Karthikeyan Chinnakonda
1264fad97e
fix unnecessary schema update event in run_sql API (#4266)
When consistent_metadata_check is set to false, then a new schema update
event should not be created.
2020-04-01 16:53:02 -05:00
Karthikeyan Chinnakonda
54dd3c94b9
include postgres version of the database while sending telemetry data (#4122)
* add new column "pg_version" while sending telemetry data

* make a new type for PGVersion and use serverVersion func

* define runTxIO action to run transaction(which exits on error)

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-04-01 20:44:26 +05:30
Karthikeyan Chinnakonda
482fa41e83
add metrics for actions (#4139)
- The metrics will include
  - no of synchronous actions
  - no of asynchronous actions
  - no of type relationships with the output
  - no of custom types defined

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-03-27 17:46:22 +05:30
Brandon Simmons
eafdda4f33
Add missing migration to initialise.sql for #3839 (#4212) 2020-03-26 23:50:45 -05:00
Phil Freeman
4ce4eb1d2f
Remove any outstanding hdb_views (fixes #3940) (#3965) 2020-03-26 15:42:33 -05:00
Rakesh Emmadi
f80b69e931
few actions' fixes and improvements (fix #3977, #4061 & close #4021) (#4109)
* add 'ID' to default scalars for custom types, fix #4061

* preserve cookie headers from sync action webhook, close #4021

* validate action webhook response to conform to output type, fix #3977

* fix tests, don't run actions' tests on PG version < 10

* update CHANGELOG.md

* no-op refactor, use types from http-network more

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-03-20 12:16:45 +05:30
Brandon Simmons
4d2dbe68fa
Use AssertNF for leak prophylaxis when doing mutable writes. Closes #3937 (#4097)
Writing to a mutable var is a particularly potent source of leaks since
it mostly defeats GHC's analysis. Here we add assertions to all mutable
writes, and fix a couple spots where we wrote some thunks to a mutable
var (compiled with -O2).

Some of these thunks were probably benign, but others looked liked they
might be retaining big args. Didn't do much analysis, just fixed.

Actually pretty happy with how easy this was to use and as a diagnostic,
once I sorted out some issues. We should consider using it elsewhere,
and maybe extending so that we can use it with tests, enable when
`-fenable-assertsions` etc.

Relates #3388

Also simplified codepaths that use `AcceptWith`, which has unnecessary
`Maybe` fields.
2020-03-17 20:31:22 -05:00
Brandon Simmons
c425b554b8
server(events): utilize proper backpressure scheme (close #3839) (#4013)
* Test working through a backlog of change events

* Use a slightly more performant threaded http server in eventing pytests

This helped locally but not on CI it seems...

* Rework event processing for backpressure. Closes #3839

With loo low `HASURA_GRAPHQL_EVENTS_FETCH_INTERVAL` and/or slow webhooks
and/or too small `HASURA_GRAPHQL_EVENTS_HTTP_POOL_SIZE` we might
previously check out events from the DB faster than we can service them,
leading to space leaks, weirdness, etc.

Other changes:
- avoid fetch interval sleep latency when we previously did a non-empty
  fetch
- prefetch event batch while http pool is working
- warn when it appears we can't keep up with events being generated
- make some effort to process events in creation order so we don't
  starve older ones.

ALSO NOTE: HASURA_GRAPHQL_EVENTS_FETCH_INTERVAL changes semantics
slightly, since it only comes into play after an empty fetch. The old
semantics weren't documented in detail, so I think this is fine.
2020-03-11 11:57:31 +05:30
Vamshi Surabhi
c52bfc540d
More robust forking, exception safety. Closes #3768 (#3860)
This is the result of a general audit of how we fork threads, with a
detour into how we're using mutable state especially in websocket
codepaths, making more robust to async exceptions and exceptions
resulting from bugs.

Some highlights:
- use a wrapper around 'immortal' so threads that die due to bugs are
  restarted, and log the error
- use 'withAsync' some places
- use bracket a few places where we might break invariants
- log some codepaths that represent bugs
- export UnstructuredLog for ad hoc logging (the alternative is we
  continue not logging useful stuff)

I had to timebox this. There are a few TODOs I didn't want to address.
And we'll wait until this is merged to attempt #3705 for
Control.Concurrent.Extended
2020-03-05 23:29:26 +05:30
Vamshi Surabhi
8a648e77f3
Don't update catalog version if using --dryRun (#3970) 2020-03-05 11:12:38 +05:30
Aravind Shankar
45bcb6b536
cli, server: use prerelease tag as channel for console assets cdn (#3975)
Co-authored-by: Shahidh K Muhammed <muhammedshahid.k@gmail.com>
2020-03-04 20:10:47 +05:30
Praveen Durairaju
865b150c82
noop: replace subdomain links with subpath (#3869)
Co-authored-by: Rikin Kachhia <54616969+rikinsk@users.noreply.github.com>
2020-02-27 15:43:07 +05:30
Phil Freeman
8df0151f99
Maintain downgrade commands in a text file, update when tagging (#3933) 2020-02-24 20:03:56 +05:30
Phil Freeman
e72bd922dc
Update Init.hs for newer tags (#3931) 2020-02-24 12:45:24 +05:30
Vamshi Surabhi
b84db36ebb
allow custom mutations through actions (#3042)
* basic doc for actions

* custom_types, sync and async actions

* switch to graphql-parser-hs on github

* update docs

* metadata import/export

* webhook calls are now supported

* relationships in sync actions

* initialise.sql is now in sync with the migration file

* fix metadata tests

* allow specifying arguments of actions

* fix blacklist check on check_build_worthiness job

* track custom_types and actions related tables

* handlers are now triggered on async actions

* default to pgjson unless a field is involved in relationships, for generating definition list

* use 'true' for action filter for non admin role

* fix create_action_permission sql query

* drop permissions when dropping an action

* add a hdb_role view (and relationships) to fetch all roles in the system

* rename 'webhook' key in action definition to 'handler'

* allow templating actions wehook URLs with env vars

* add 'update_action' /v1/query type

* allow forwarding client headers by setting `forward_client_headers` in action definition

* add 'headers' configuration in action definition

* handle webhook error response based on status codes

* support array relationships for custom types

* implement single row mutation, see https://github.com/hasura/graphql-engine/issues/3731

* single row mutation: rename 'pk_columns' -> 'columns' and no-op refactor

* use top level primary key inputs for delete_by_pk & account select permissions for single row mutations

* use only REST semantics to resolve the webhook response

* use 'pk_columns' instead of 'columns' for update_by_pk input

* add python basic tests for single row mutations

* add action context (name) in webhook payload

* Async action response is accessible for non admin roles only if
  the request session vars equals to action's

* clean nulls, empty arrays for actions, custom types in export metadata

* async action mutation returns only the UUID of the action

* unit tests for URL template parser

* Basic sync actions python tests

* fix output in async query & add async tests

* add admin secret header in async actions python test

* document async action architecture in Resolve/Action.hs file

* support actions returning array of objects

* tests for list type response actions

* update docs with actions and custom types metadata API reference

* update actions python tests as per #f8e1330

Co-authored-by: Tirumarai Selvan <tirumarai.selvan@gmail.com>
Co-authored-by: Aravind Shankar <face11301@gmail.com>
Co-authored-by: Rakesh Emmadi <12475069+rakeshkky@users.noreply.github.com>
2020-02-13 23:08:23 +05:30
Phil Freeman
94102c0460
Add downgrade command (close #1156) (#3760)
* Add downgrade command

* Add docs per @lexi-lambda's suggestions

* make tests pass

* Update hdb_version once, from Haskell

* more work based on feedback

* Improve the usage message

* Small docs changes

* Test downgrades exist for each tag

* Update downgrading.rst

* Use git-log to find tags which are ancestors of the current commit

Co-authored-by: Vamshi Surabhi <0x777@users.noreply.github.com>
2020-02-07 16:33:12 +05:30
Vamshi Surabhi
2de663d2a8
Merge branch 'master' into 3759-3791-minor-metadata-build-bugfixes 2020-02-05 18:25:46 +05:30
Anon Ray
c6e34baad8
fix parsing JWK expiry time from headers on startup (fix #3655) (#3779) 2020-02-05 12:37:30 +05:30
Brandon Simmons
f506cb397f Log time spent consuming request bytes from socket. Closes #3547
Add a new 'request_read_time' to the logging record.

We also use 'withElapsedTime' to get proper monotonic clock for
'query_execution_time' as well.
2020-02-03 18:50:10 -06:00
Brandon Simmons
58ef316118 Add request timings and count histograms to telemetry. Closes #3552
We upload a set of accumulating timers and counters to track service
time for different types of operations, across several dimensions (e.g.
did we hit the plan cache, was a remote involved, etc.)

Also...

Standardize on DiffTime as a standard duration type, and try to use it
consistently.

See discussion here:
https://github.com/hasura/graphql-engine/pull/3584#pullrequestreview-340679369

It should be possible to overwrite that module so the new threadDelay
sticks per the pattern in #3705 blocked on #3558

Rename the Control.Concurrent.Extended.threadDelay to `sleep` since a
naive use with a literal argument would be very bad!

We catch a bug in 'computeTimeDiff'.

Add convenient 'Read' instances to the time unit utility types. Make
'Second' a newtype to support this.
2020-02-03 18:50:10 -06:00
Lyndon Maydwell
be6f966ce0
New ENV variable support for disabling CORS headers (close #3782) (#3785) 2020-02-03 16:43:03 +05:30
Alexis King
8ef205fba5 Explicitly invalidate enum values when metadata is reloaded
This fixes #3759. Also, while we’re at it, also improve the way
invalidations are synced across instances so enums and remote schemas
are appropriately reloaded by the schema syncing process.
2020-01-30 18:17:29 -06:00
Alexis King
5bd5a548fa
server: Parameterize the graphql-engine library over the version (#3668) 2020-01-22 15:55:55 -06:00
Karthikeyan Chinnakonda
b344e7f39c disable explain end-points when metadata API is disabled, fix #3717 (#3751) 2020-01-22 16:20:49 +05:30
Phil Freeman
9ed8f717a7 remove hdb_views for inserts (#3598)
* WIP: Remove hdb_views for inserts

* Show failing row in check constraint error

* Revert "Show failing row in check constraint error"

This reverts commit dd2cac29d0.

* Use the better query plan

* Simplify things

* fix cli test

* Update downgrading.rst

* remove 1.1 asset for cli
2020-01-16 10:23:28 +05:30
Alexis King
feb4a05618
Add fast path for server internal metadata migrations (#3686) 2020-01-14 01:09:10 -06:00
Anon Ray
dca8559703 fix cache-control header parsing for JWK (fix #3655) (#3676)
write a proper parser according to the RFC
https://tools.ietf.org/html/rfc7234#section-5.2
2020-01-13 15:56:51 -06:00
Alexis King
e47a8c4b86 incremental metadata: Clean up a few lingering loose ends
- Move MonadBase/MonadBaseControl instances for TxE into pg-client-hs
- Set the -qn2 RTS option by default to limit the parallel GC to 2
  threads
- Remove eventlog instrumentation
- Don’t rebuild the schema cache again after running a query that needs
  it to be rebuilt, since we do that explicitly now.
- Remove some redundant checks, and relocate a couple others.
2020-01-08 17:19:03 -06:00
Alexis King
e2eabcd54e Properly check that custom field names do not conflict with other fields 2020-01-08 16:45:53 -06:00
Alexis King
27997107ab Add caching for recreating event trigger functions 2020-01-08 16:45:46 -06:00
Alexis King
780857fb19 Switch to a CPS implementation of Rule
This is significantly more performance, even without specialization,
which dramatically improves compile times.
2020-01-08 16:45:46 -06:00
Alexis King
c322e8a5d4 Use a significantly more efficient table_info_agg view
Also, use the view in Schema.Diff to share some more logic.
2020-01-08 16:45:46 -06:00
Alexis King
447bac74e7 Split up Hasura.RQL.DDL.Schema.Cache
This should hopefully improve compile times by avoiding the need to
specialize everything at once.
2020-01-08 16:45:46 -06:00
Alexis King
5b969208c6 Use arrows instead of monads to define the schema cache construction 2020-01-08 16:43:06 -06:00
Alexis King
1387722970 Refactor schema cache construction to avoid imperative updates
wip: fix error codes in remote schema tests
2020-01-08 16:43:06 -06:00
Toan Nguyen
2aba7edf81 add query execution time and response size to ws-server logs in websocket transport (#3584) 2020-01-07 14:25:32 -06:00
Phil Freeman
c766881125 Support batched queries (fix #1812) (#3490) 2019-12-20 10:04:02 -06:00
nizar-m
c0038a74a5 Allow request body to be logged even with successful queries (#3529) 2019-12-16 15:02:05 -06:00
Brandon Simmons
55bc9d57ef Fix documentation of HASURA_GRAPHQL_PG_CONNECTIONS (#3495) 2019-12-12 14:10:04 -06:00
Anon Ray
b81d22f583 fix resolving user info in websocket transport (#3509)
The connection handler in websocket transport was not using the
  'UserAuthentication' interface to resolve user info. Fix resolving
  user info in websocket transport to use the common
  'UserAuthentication' interface
2019-12-10 19:04:49 -06:00
Tirumarai Selvan
ff4b2bf8b0 bulk query should not care about access mode of select or count queries (#3467) 2019-12-04 13:16:37 -06:00
Anon Ray
afd6f30e72 read cache control header to refresh JWK (fix #3301) (#3446) 2019-12-03 14:56:59 -06:00
Tirumarai Selvan
3cd656f3c7 check db connection in healthz endpoint (close #2645) (#3440)
* check db connection in healthz endpoint
* use Spock.text to automatically set Content-Type
* update docs
2019-12-03 13:18:10 -06:00
Anon Ray
c9b9ed5055 fix various functions to not create their own logger (#3439) 2019-11-28 14:33:14 +05:30
Anon Ray
490b639981 refactor some internal components (#3414) 2019-11-26 17:44:21 +05:30
Vamshi Surabhi
6abe8d7927 allow specifying an upper limit on the query plan cache size (#3012) 2019-11-25 11:12:23 -06:00
Rakesh Emmadi
9b8e6b42d1 functions can access session info via input arg (close #2322) (#3143) 2019-11-20 12:17:06 +05:30
Tirumarai Selvan
d2b2a58c0e add read_only to run_sql metadata api (#3191) 2019-11-14 18:20:18 -06:00
Tirumarai Selvan
3cad1319c4 improve event fetch query (#3236) 2019-11-13 12:59:19 +05:30
Shahidh K Muhammed
31e0225230
add tests with postgres 12 (close #2749) (#3102) 2019-10-26 11:09:57 +05:30
Rakesh Emmadi
6d92e4f9db save permissions, relationships and collections in catalog with 'is_system_defined' explicitly (#3165)
* save permissions, relationships and collections in catalog with 'is_system_defined'
* Use common stanzas in the .cabal file
* Refactor migration code into lib instead of exe
* Add new server test suite that exercises migrations
* Make graphql-engine clean succeed even if the schema does not exist
2019-10-21 11:01:05 -05:00
Rakesh Emmadi
d8d21c1487 support computed fields returning scalars or set of tables (close #1387) (#2917) 2019-10-18 13:59:47 +05:30
Ajeet D'Souza
a66fb42ce2 Make catalog metadata migrations work on all schema versions (fix #2826) (#2379)
* Separate DB and metadata migrations
* Refactor Migrate.hs to generate list of migrations at compile-time
* Replace ginger with shakespeare to improve performance
* Improve migration log messages
2019-10-11 00:13:57 -05:00
Toan Nguyen
0c74839934 add raw query field for error http log (close #2963) (#3020) 2019-10-07 13:04:33 -05:00
Rakesh Emmadi
8972dfb018 remove brotli compression (#2967)
Although brotli itself is MIT-licensed, the Haskell brotli library that provides bindings to it is GPL-licensed, so we cannot use it unless we get a response on haskell-hvr/brotli#1.
2019-10-05 02:20:50 -05:00
Vamshi Surabhi
0f143f0ea8 fixes to the subscriptions improvements introduced with #2942 (#3005)
* fix incorrect references to result variables
* remove docs/code related to 'fallback' backend
2019-10-03 14:35:55 -05:00
Rakesh Emmadi
8a0615ff70 add gzip brotli compression to http responses (close #2674) (#2751) 2019-09-19 18:24:40 +05:30
Rakesh Emmadi
9bd5826020 allow customising graphql schema for a table (close #981) (#2509)
* allow customizing GraphQL root field names, close #981

* document v2 track_table API in reference

* support customising column field names in GraphQL schema

* [docs] add custom column fields doc in API reference

* add tests

* rename 'ColField' to 'ColumnField'

* embed column's graphql field in 'PGColumnInfo'

-> Value constructor of 'PGCol' is not exposed
-> Using 'parseJSON' to construct 'PGCol' in 'FromJSON' instances

* avoid using 'Maybe TableConfig'

* refactors & 'custom_column_fields' -> 'custom_column_names'

* cli-test: add configuration field in metadata export test

* update expected keys in `FromJSON` instance of `TableMeta`

* use `buildSchemaCacheFor` to update configuration in v2 track_table

* remove 'GraphQLName' type and use 'isValidName' exposed from parser lib

* point graphql-parser-hs library git repo to hasura

* support 'set_table_custom_fields' query API & added docs and tests
2019-09-19 10:17:36 +05:30
Alexis King
264d70644b Multiplex all subscriptions, grouping them by their resolved SQL query 2019-09-16 22:00:46 -05:00
José Lorenzo Rodríguez
5609fba393 Implemented graceful shutdown for websockets (#2827) 2019-09-09 15:26:04 -05:00
Rakesh Emmadi
c148e5753a support optional parameters in database url (close #1709) (#2344) 2019-09-05 15:59:26 -07:00
Rakesh Emmadi
d9fb0f8780 use named notation for function arguments if any argument is not specified (fix #2730) (#2777)
* use positional arguments in SQL functions
* only allow omitting set of last arguments in functions
* disallow omitting of a non default argument in functions
2019-08-28 14:27:15 -05:00
Alexis King
78ec906794 server: Include number of enum tables in metrics 2019-08-26 00:54:56 -05:00
Alexis King
c46ecc72dc Refactor non-table functions out of Hasura.RQL.DDL.Table 2019-08-26 00:54:56 -05:00
Alexis King
7fa1452ca0 server: Rename a whole bunch of things
* PGTypeSimple → PGTypeScalar
* PGScalarTyped → WithScalarType
* PGColValue → PGScalarValue
* PGColInfo → PGColumnInfo
* PGRawColInfo → PGRawColumnInfo
* mkScalarSessionVar → mkTypedSessionVar
2019-08-26 00:54:56 -05:00
Alexis King
ed26da59a6 Add support for GraphQL enum types via enum table references
These changes also add a new type, PGColumnType, between PGColInfo and
PGScalarType, and they process PGRawColumnType values into PGColumnType
values during schema cache generation.
2019-08-26 00:54:56 -05:00
Vamshi Surabhi
a5a07634a5
Merge branch 'master' into dev-scripts-and-doc-tweaks 2019-08-07 12:08:46 +05:30
Anon Ray
b4a0a03631 log when request body parsing fails (fix #2555) (#2556) 2019-08-01 16:21:59 +05:30
Rakesh Emmadi
db24d3248f schema sync process will not generate garbage data (close #2542) (#2585)
schema syncing logic now requires only a single row in `hdb_schema_update_event`.
2019-07-31 14:31:48 +05:30
Brandon Simmons
bc32130d09 Clarify HASURA_GRAPHQL_EVENTS_FETCH_INTERVAL docs 2019-07-26 01:17:56 -04:00
Vamshi Surabhi
4facb3c780 remove support for query templates (#2560)
query templates is a little known feature that lets you template rql
queries and serve them as rest apis. This is not relevant anymore
given the GraphQL interface and getting rid of it reduces the dev
time when adding features in few subsystems.

This feature has never been used outside hasura's internal projects or
documented or exposed through console and hence can safely be removed.
2019-07-23 19:11:34 +00:00
Anon Ray
f2f14e727b Merge pull request from GHSA-2j98-fw5g-j43v
* fix bug in audience check while verifying JWT

  - previously the check was converting the audience type into a string
  and then comparing with the conf value. all audience types (as it is a
  string or URI) will convert to plain strings
  - use the Audience type from the jose library for comparing

* add docs for audience

* add issuer check as well

* docs minor syntax fix

* skip audience check if not given in conf

* minor docs update

* qualify import jose library
2019-07-11 09:58:39 +00:00
Ajeet D'Souza
92c4cff79e check input for empty strings for metadata api (close #2302) (#2300) 2019-07-11 09:00:45 +00:00
Anon Ray
8f1e7dbc8d breaking: server logging changes (close #507, close #2171) (#1835) 2019-07-11 05:37:06 +00:00
Vamshi Surabhi
5aa4a46e72
allow fetching gc stats through dev api (#2489) 2019-07-10 17:31:52 +05:30
Rakesh Emmadi
9eb38e6c96 cache remote schema's introspection query response (fix #1679) (#2089) 2019-07-08 11:21:41 +05:30
Ajeet D'Souza
666fd1bbc7 improve code for release channel detection (#2456) 2019-07-04 09:47:17 +00:00
Alexis King
fb3794c31a Support Node v12.x and fix tests on macOS (#2455)
* Bump node-sass to version ^4.12.0

This avoids sass/node-sass#2632, which causes compilation failures on
Node v12.x.

* Do most of the work in /pg_dump in Haskell instead of shell

The shell version caused problems on non-Linux systems since it relied
on the non-POSIX -i option for sed, which works slightly differently on
BSD and macOS.
2019-07-03 14:07:13 +05:30
Ajeet D'Souza
b184ce0188 add server version to console html (close #2444) (#2452) 2019-07-02 12:44:29 +00:00
Ajeet D'Souza
08df38aa8c add name of ci environment to telemetry data (#2382) 2019-06-26 11:53:40 +05:30
Shahidh K Muhammed
a8dd3b3deb add api to get server config details (close #1831); add jwt-analyzer (close #1369) (#1925) 2019-06-11 18:59:03 +05:30
Anon Ray
278f26b073 forward set-cookie headers from remote servers (fix #1654) (#2305) 2019-06-04 15:40:28 +05:30
Anon Ray
95a27fba89 fix response for remote schema queries over ws (fix #2246) (#2248) 2019-05-29 17:21:09 +05:30
Rakesh Emmadi
4aabbd5a28 fix and improve help description for stripes and connections (#2278) 2019-05-27 13:46:53 +05:30
Josha Inglis
277be9074e fix transaction isolation settings on server (#2211) 2019-05-20 19:48:17 +05:30
Shahidh K Muhammed
7c89e951f2 serve console assets without any authorization checks (#2201) 2019-05-16 16:58:50 +05:30
Anon Ray
81bdfafd69 ignore content-type header in auth webhook (#2197) 2019-05-16 14:41:15 +05:30
Shahidh K Muhammed
76ceb707f4
bundle console assets into server (close #516, close #521, close #2130) (#2192)
This PR builds console static assets into the server docker image at `/srv/console-assets`. When env var `HASURA_GRAPHQL_CONSOLE_ASSETS_DIR=/srv/console-assets` or flag `--console-assets-dir=/srv/console-assets` is set on the server, the files in this directory are served at `/console/assets/*`.

The console html template will have a variable called `cdnAssets: false` when this flag is set and it loads assets from server itself instead of CDN.

The assets are moved to a new bucket with a new naming scheme:

```
graphql-engine-cdn.hasura.io/console/assets/
   /common/{}
   /versioned/<version/{}
   /channel/<channel>/<version>/{}
```

Console served by CLI will still load assets from CDN - will fix that in the next release.
2019-05-16 13:15:29 +05:30
Rakesh Emmadi
0b210cc245 support allow-list for graphql queries (closes #989) (#2075) 2019-05-16 11:43:25 +05:30
Anon Ray
935eaf2211 booleans from env are case insensitve (fix #2177) (#2182) 2019-05-14 16:52:05 +05:30
Anon Ray
7d03e7af2f fix non-200 response for authorization errors on /v1/graphql (#2173) 2019-05-14 16:50:55 +05:30
Rakesh Emmadi
c6f40df6d5 close websocket connection on JWT expiry (fix #578) (#2156) 2019-05-14 11:54:46 +05:30
Tirumarai Selvan
36e49ea75b manual event triggers (close #1687) (#2077) 2019-05-13 15:11:07 +05:30
Anon Ray
a21f6cd648 introduce v1/graphql (fix #1368) (#2064)
Changes compared to `/v1alpha1/graphql`

* Changed all graphql responses in **/v1/graphql** endpoint to be 200. All graphql clients expect responses to be HTTP 200. Non-200 responses are considered transport layer errors. 

* Errors in http and websocket layer are now consistent and have similar structure.
2019-05-10 11:35:10 +05:30
Rakesh Emmadi
c4c36e0ef4 schema cache sync improvements (#2098)
* build schema cache function without db setup

The setup shouldn't happen for sync. The database is already setup by the instance which generated the event. This means that the sync is now faster.

* use SQL loop to drop hdb_views schema views and routines with ordering

This avoids deadlocks when schema is being changed concurrently

* schema sync now only processes the latest event

This becomes useful when a lot of schema change
events happen while we are still processing an
earlier event.
2019-05-03 16:12:26 +05:30
Shahidh K Muhammed
71cf017197 add an api to dump postgres schema (close #1939) (#1967) 2019-04-30 14:04:08 +05:30
Vamshi Surabhi
8389a7e273
multiplexed subscription improvements (#2081)
* split stm transactions when snapshotting to make it faster

* mx subs: push to both old and new sinks at the same time

* expose dev APIs through allowed APIs flag
2019-04-30 10:45:23 +05:30
Rakesh Emmadi
39f43bdd59 log inconsistent metadata objects only when there are any (close #2053) (#2058) 2019-04-29 11:52:48 +05:30
Rakesh Emmadi
1d7cbc7f24 recover from inconsistent database state (closes #231) (#1863) 2019-04-17 21:59:39 +05:30
Vamshi Surabhi
ce243f5899
multiplexed subscriptions (#1934)
* add types to represent unparsed http gql requests

This will help when we add caching of frequently used ASTs

* query plan caching

* move livequery to execute

* add multiplexed module

* session variable can be customised depending on the context

Previously the value was always "current_setting('hasura.user')"

* get rid of typemap requirement in reusable plan

* subscriptions are multiplexed when possible

* use lazytx for introspection to avoid acquiring a pg connection

* refactor to make execute a completely decoupled module

* don't issue a transaction for a query

* don't use current setting for explained sql

* move postgres related types to a different module

* validate variableValues on postgres before multiplexing subs

* don't user current_setting for queries over ws

* plan_cache is only visible when developer flag is enabled

* introduce 'batch size' when multiplexing subscriptions

* bump stackage to 13.16

* fix schema_stitching test case error code

* store hashes instead of actual responses for subscriptions

* internal api to dump subscriptions state

* remove PlanCache from SchemaCacheRef

* allow live query options to be configured on server startup

* capture metrics for multiplexed subscriptions

* more metrics captured for multiplexed subs

* switch to tvar based hashmap for faster snapshotting

* livequery modules do not expose internal details

* fix typo in live query env vars

* switch to hasura's pg-client-hs
2019-04-17 15:18:41 +05:30
Shahidh K Muhammed
de24cfd43a read version from env var during server build time (close #1398) (#1897)
* read version from env var at build time (close #1398)

* remove un-used imports, edit makefile

* edit makefile to add new targets and export variables

* only export VERSION in makefile

* read version by executing the script if env var is absent
2019-04-11 09:41:48 +05:30
Tirumarai Selvan
36781199d0 breaking: drop id from event_triggers table (fix #1840) (#1857) 2019-03-25 22:40:52 +05:30
Vamshi Surabhi
d61c90f1b7
validate relationship definitions on startup (#1849) 2019-03-25 14:26:29 +05:30
Vamshi Surabhi
24dcefb142
use bytestring builder to represent encoded json (#1800) 2019-03-18 21:52:21 +05:30
Rakesh Emmadi
e32f5a1fb1 sync metadata cache across multiple instances connected to same db (closes #1182) (#1574)
1. Haskel library `pg-client-hs` has been updated to expose a function that helps listen to `postgres` notifications over a `channel` in this [PR](https://github.com/hasura/pg-client-hs/pull/5)
2. The server records an event in a table `hdb_catalog.hdb_cache_update_event` whenever any `/v1/query` (that changes metadata) is requested. A trigger notifies a `cache update` event via `hasura_cache_update` channel
3. The server runs two concurrent threads namely `listener` and `processor`. The `listener` thread listens to events on `hasura_cache_update` channel and pushed into a `Queue`. The `processor` thread fetches events from that `Queue` and processes it. Thus server rebuilds schema cache from database and updates.
2019-03-12 11:16:27 +05:30
Vamshi Surabhi
98405fdc0c
allow x-hasura- req headers for jwt unauth role, closes #1686 (#1689) 2019-03-05 17:54:47 +05:30
Anon Ray
02d80c9ac6 read cookie while initialising websocket connection (fix #1660) (#1668)
* read cookie while initialising websocket connection (fix #1660)

* add tests for cookie on websocket init

* fix logic for tests

* enforce cors, and flag to force read cookie when cors disabled

  - as browsers don't enforce SOP on websockets, we enforce CORS policy
  on websocket handshake
  - if CORS is disabled, by default cookie is not read (because XSS
  risk!). Add special flag to force override this behaviour

* add log and forward origin header to webhook

  - add log notice when cors is disabled, and cookie is not read on
  websocket handshake
  - forward origin header to webhook in POST mode. So that when CORS is
  disabled, webhook can also enforce CORS independently.

* add docs, and forward all client headers to webhook
2019-03-04 13:16:53 +05:30
Rakesh Emmadi
377290a058 breaking: correct (de)serialisation of postgres numeric types in json (fix #1523) (#1662) 2019-03-01 17:15:04 +05:30
Rakesh Emmadi
6c20ca8a55 allow renaming tables, columns and relationships (close #79) (#1542) 2019-03-01 14:47:22 +05:30
nizar-m
1fa66dc622 add option to disable metadata and graphql apis (close #1088) (#1650) 2019-02-28 19:23:03 +05:30
Shahidh K Muhammed
097bfb6bfa revert "forward response headers from remote servers (#1664)"
This reverts commit c19fe35f4e.
2019-02-28 17:20:56 +05:30
Anon Ray
c19fe35f4e forward response headers from remote servers (fix #1654) (#1664) 2019-02-28 17:15:07 +05:30
nizar-m
f83a8e591f rename access-key to admin-secret (close #1347) (#1540)
Rename the admin secret key header used to access GraphQL engine from X-Hasura-Access-Key to X-Hasura-Admin-Secret.

Server CLI and console all support the older flag but marks it as deprecated.
2019-02-14 15:07:47 +05:30
Anon Ray
199a24d050 add support for multiple domains in cors config (close #1436) (#1536)
Support for multiple domains (as CSV) in the `--cors-domain` flag and `HASURA_GRAPHQL_CORS_DOMAIN` env var.

Following are all valid configurations (must include scheme and optional port):
```shell
HASURA_GRAPHQL_CORS_DOMAIN="https://*.foo.bar.com:8080"
HASURA_GRAPHQL_CORS_DOMAIN="https://*.foo.bar.com, http://*.localhost, https://example.com"
HASURA_GRAPHQL_CORS_DOMAIN="*"
HASURA_GRAPHQL_CORS_DOMAIN="http://example.com, http://*.localhost, http://localhost:3000, https://*.foo.bar.com, https://foo.bar.com"
```

**Note**: top-level domains are not considered as part of wildcard domains. You have to add them separately. E.g - `https://*.foo.com` doesn't include `https://foo.com`.

The default (if the flag or env var is not specified) is `*`. Which means CORS headers are sent for all domains.
2019-02-14 11:28:38 +05:30
Anon Ray
4f6462e98f add config for stringified hasura claims in JWT (fix #1176) (#1538) 2019-02-05 17:34:16 +05:30
Rakesh Emmadi
4ef50d95c7 hdb_views should always be cleared before building schema cache (#1513) 2019-01-29 15:39:58 +05:30
Anon Ray
4ae44f7b5d add functions to server telemetry (#1500) 2019-01-28 22:16:44 +05:30
Shahidh K Muhammed
11e7c3f9d6 add anonymous telemetry (#1401) 2019-01-28 19:25:28 +05:30
Rakesh Emmadi
0bf2457e23 allow exposing postgres functions through GraphQL interface (close #333) (#1073) 2019-01-25 09:01:54 +05:30
nizar-m
916caf1575 add flag to disable prepared statements (close #1392) (#1396) 2019-01-18 19:50:41 +05:30
Rakesh Emmadi
d91d7e658a optimise 'run_sql' query, closes #1362 (#1406) 2019-01-18 16:15:59 +05:30
Nathan Stitt
1b9540f996 allow specifying network interface with "server-host" option (#1280)
* allow specifying network interface with "server-host" option

* store host value as a HostPreference

* document server-host options
2019-01-11 16:37:13 +05:30
Rakesh Emmadi
41e487d203 fix auth hook mode env var (fix #1270) (#1285)
Support HASURA_GRAPHQL_AUTH_HOOK_MODE env var for --auth-hook-mode flag.
Drop support for HASURA_GRAPHQL_AUTH_HOOK_TYPE env var in next major
update (beta/stable)
2019-01-04 12:12:36 +05:30
Rakesh Emmadi
4d9d1505dd improve startup logging, close #1236 (#1258) 2019-01-02 16:54:17 +05:30
Vamshi Surabhi
ea4d2644e8
local console during development (#1252)
* console now works on local builds of the server

1. local console assets can be served at /static/ by a build time flag
'local-console'. This can be set with stack as follows:
   `stack build --flag graphql-engine:local-console`
2. the --root-dir option is removed which was used as a temporary hack
for serving graphiql
3. remove server's graphiql source code
2018-12-21 13:21:02 +05:30
Rakesh Emmadi
d4e6ffcae8 revert back to older cli options parser type (#1231) 2018-12-19 17:08:33 +05:30
Rakesh Emmadi
b5bbb966f2 add consolePath in console.html template (#1222)
Server templates `consolePath` key in `window.__env` object in console html template.
If server is hit at `/console/table/author` then `window.__env` in served html looks like
```
{
   consoleMode: "server",
   urlPrefix: "/console",
   consolePath: "/console/table/author",
   isAccessKeySet: true
}
```
2018-12-18 15:09:01 +05:30