hasura/graphql-engine
1
0
Fork 0
mirror of https://github.com/hasura/graphql-engine.git synced 2024-12-14 17:02:49 +03:00
Code Issues Projects Releases Wiki Activity

server: multitenant metadata storage

Browse Source 
The metadata storage implementation for graphql-engine-multitenant.

- It uses a centralized PG database to store metadata of all tenants (instead of per tenant database)
- Similarly, it uses a single schema-sync listener thread per MT worker (instead of listener thread per tenant) (PS: although, the processor thread is spawned per tenant)
- 2 new flags are introduced - `--metadataDatabaseUrl` and (optional) `--metadataDatabaseRetries`

Internally, a "metadata mode" is introduced to indicate an external/managed store vs a store managed by each pro-server.

To run :
- obtain the schema file (located at `pro/server/res/cloud/metadata_db_schema.sql`)
- apply the schema on a PG database
- set the `--metadataDatabaseUrl` flag to point to the above database
- run the MT executable

The schema (and its migrations) for the metadata db is managed outside the MT worker.

### New metadata

The following is the new portion of `Metadata` added :

```yaml
version: 3
metrics_config:
  analyze_query_variables: true
  analyze_response_body: false
api_limits:
  disabled: false
  depth_limit:
    global: 5
    per_role:
      user: 7
      editor: 9
  rate_limit:
    per_role:
      user:
        unique_params:
        - x-hasura-user-id
        - x-hasura-team-id
        max_reqs_per_min: 20
    global:
      unique_params: IP
      max_reqs_per_min: 10
```

- In Pro, the code around fetching/updating/syncing pro-config is removed
- That also means, `hdb_pro_catalog` for keeping the config cache is not required. Hence the `hdb_pro_catalog` is also removed
- The required config comes from metadata / schema cache

### New Metadata APIs

- `set_api_limits`
- `remove_api_limits`
- `set_metrics_config`
- `remove_metrics_config`

#### `set_api_limits`

```yaml
type: set_api_limits
args:
  disabled: false
  depth_limit:
    global: 5
    per_role:
      user: 7
      editor: 9
  rate_limit:
    per_role:
      anonymous:
         max_reqs_per_min: 10
         unique_params: "ip"
      editor:
        max_reqs_per_min: 30
        unique_params:
        - x-hasura-user-id
      user:
        unique_params:
        - x-hasura-user-id
        - x-hasura-team-id
        max_reqs_per_min: 20
    global:
      unique_params: IP
      max_reqs_per_min: 10
```

#### `remove_api_limits`

```yaml
type: remove_api_limits
args: {}
```

#### `set_metrics_config`

```yaml
type: set_metrics_config
args:
  analyze_query_variables: true
  analyze_response_body: false
```

#### `remove_metrics_config`

```yaml
type: remove_metrics_config
args: {}
```

#### TODO
- [x] on-prem pro implementation for `MonadMetadataStorage`
- [x] move the project config from Lux to pro metadata (PR: #379)
- [ ] console changes for pro config/api limits, subscription workers (cc @soorajshankar @beerose)
- [x] address other minor TODOs
  - [x] TxIso for `MonadSourceResolver`
  - [x] enable EKG connection pool metrics
  - [x] add logging of connection info when sources are added?
  - [x] confirm if the `buildReason` for schema cache is correct
- [ ] testing
- [x] 1.3 -> 1.4 cloud migration script (#465; PR: #508)
  - [x] one-time migration of existing metadata from users' db to centralized PG
  - [x] one-time migration of pro project config + api limits + regression tests from metrics API  to metadata
- [ ] integrate with infra team (WIP - cc @hgiasac)
  - [x] benchmark with 1000+ tenants + each tenant making read/update metadata query every second (PR: https://github.com/hasura/graphql-engine-mono/pull/411)
  - [ ] benchmark with few tenants having large metadata (100+ tables etc.)
  - [ ] when user moves regions (https://github.com/hasura/lux/issues/1717)
    - [ ] metadata has to be migrated from one regional PG to another
    - [ ] migrate metrics data as well ?
      - [ ] operation logs
      - [ ] regression test runs

- [ ] find a way to share the schema files with the infra team

Co-authored-by: Naveen Naidu <30195193+Naveenaidu@users.noreply.github.com>
GitOrigin-RevId: 39e8361f2c0e96e0f9e8f8fb45e6cc14857f31f1
This commit is contained in:
Anon Ray 2021-02-11 23:24:25 +05:30 committed by hasura-bot
parent 5f6e59b4db
commit 06b599b747
19 changed files with 328 additions and 50 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
server/graphql-engine.cabal
View File

@ -373,6 +373,7 @@ library
, Hasura.RQL.Instances
, Hasura.RQL.Types
, Hasura.RQL.Types.Action
, Hasura.RQL.Types.ApiLimit
, Hasura.RQL.Types.Column
, Hasura.RQL.Types.Common
, Hasura.RQL.Types.ComputedField
@ -395,6 +396,7 @@ library
, Hasura.RQL.Types.Source
, Hasura.RQL.Types.Table
, Hasura.RQL.DDL.Action
, Hasura.RQL.DDL.ApiLimit
, Hasura.RQL.DDL.ComputedField
, Hasura.RQL.DDL.CustomTypes
, Hasura.RQL.DDL.Deps

56
server/src-lib/Hasura/App.hs
View File

@ -90,6 +90,7 @@ data ExitCode
| AuthConfigurationError
| EventSubSystemError
| DatabaseMigrationError
| SchemaCacheInitError -- ^ used by MT because it initialises the schema cache only
-- these are used in app/Main.hs:
| MetadataExportError
| MetadataCleanError
@ -166,36 +167,48 @@ data GlobalCtx
-- and optional retries
}
initGlobalCtx
:: (MonadIO m)
=> Env.Environment
-> Maybe String
-- ^ the metadata DB URL
-> PostgresConnInfo (Maybe UrlConf)
-- ^ the user's DB URL
-> m GlobalCtx
initGlobalCtx env metadataDbUrl defaultPgConnInfo = do
httpManager <- liftIO $ HTTP.newManager HTTP.tlsManagerSettings
let PostgresConnInfo dbUrlConf maybeRetries = defaultPgConnInfo
maybeMetadataDbConnInfo =
let retries = fromMaybe 1 $ _pciRetries defaultPgConnInfo
in (Q.ConnInfo retries . Q.CDDatabaseURI . txtToBs . T.pack)
<$> metadataDbUrl
mkConnInfoFromSource dbUrl = do
resolvePostgresConnInfo env dbUrl maybeRetries
maybeDbUrlAndConnInfo <- forM dbUrlConf $ \dbUrl -> do
connInfo <- resolvePostgresConnInfo env dbUrl maybeRetries
pure (dbUrl, connInfo)
mkConnInfoFromMDb mdbUrl =
let retries = fromMaybe 1 maybeRetries
in (Q.ConnInfo retries . Q.CDDatabaseURI . txtToBs . T.pack) mdbUrl
metadataDbConnInfo <-
case (maybeMetadataDbConnInfo, maybeDbUrlAndConnInfo) of
(Nothing, Nothing) ->
printErrExit InvalidDatabaseConnectionParamsError
"Fatal Error: Either of --metadata-database-url or --database-url option expected"
-- If no metadata storage specified consider use default database as
-- metadata storage
(Nothing, Just (_, dbConnInfo)) -> pure dbConnInfo
(Just mdConnInfo, _) -> pure mdConnInfo
mkGlobalCtx mdbConnInfo sourceConnInfo =
pure $ GlobalCtx httpManager mdbConnInfo (sourceConnInfo, maybeRetries)
pure $ GlobalCtx httpManager metadataDbConnInfo (maybeDbUrlAndConnInfo, maybeRetries)
case (metadataDbUrl, dbUrlConf) of
(Nothing, Nothing) ->
printErrExit InvalidDatabaseConnectionParamsError
"Fatal Error: Either of --metadata-database-url or --database-url option expected"
-- If no metadata storage specified consider use default database as
-- metadata storage
(Nothing, Just dbUrl) -> do
connInfo <- mkConnInfoFromSource dbUrl
mkGlobalCtx connInfo $ Just (dbUrl, connInfo)
(Just mdUrl, Nothing) -> do
let mdConnInfo = mkConnInfoFromMDb mdUrl
mkGlobalCtx mdConnInfo Nothing
(Just mdUrl, Just dbUrl) -> do
srcConnInfo <- mkConnInfoFromSource dbUrl
let mdConnInfo = mkConnInfoFromMDb mdUrl
mkGlobalCtx mdConnInfo (Just (dbUrl, srcConnInfo))
-- | Context required for the 'serve' CLI command.
@ -207,6 +220,7 @@ data ServeCtx
, _scMetadataDbPool :: !Q.PGPool
, _scShutdownLatch :: !ShutdownLatch
, _scSchemaCache :: !RebuildableSchemaCache
, _scSchemaCacheRef :: !SchemaCacheRef
, _scSchemaSyncCtx :: !SchemaSyncCtx
}
@ -278,8 +292,12 @@ initialiseServeCtx env GlobalCtx{..} so@ServeOptions{..} = do
sqlGenCtx soEnableRemoteSchemaPermissions soInferFunctionPermissions (mkPgSourceResolver pgLogger)
let schemaSyncCtx = SchemaSyncCtx schemaSyncListenerThread schemaSyncEventRef cacheInitStartTime
-- See Note [Temporarily disabling query plan caching]
-- (planCache, schemaCacheRef) <- initialiseCache
schemaCacheRef <- initialiseCache rebuildableSchemaCache
pure $ ServeCtx _gcHttpManager instanceId loggers metadataDbPool latch
rebuildableSchemaCache schemaSyncCtx
rebuildableSchemaCache schemaCacheRef schemaSyncCtx
mkLoggers
:: (MonadIO m, MonadBaseControl IO m)
@ -453,7 +471,7 @@ runHGEServer env ServeOptions{..} ServeCtx{..} initTime postPollHook serverMetri
soPlanCacheOptions
soResponseInternalErrorsConfig
postPollHook
_scSchemaCache
_scSchemaCacheRef
ekgStore
soEnableRemoteSchemaPermissions
soInferFunctionPermissions

7
server/src-lib/Hasura/Eventing/ScheduledTrigger.hs
View File

@ -87,6 +87,7 @@ module Hasura.Eventing.ScheduledTrigger
, getCronEventsTx
, deleteScheduledEventTx
, getInvocationsTx
, getInvocationsQuery
-- * Export utility functions which are useful to build
-- SQLs for fetching data from metadata storage
@ -811,8 +812,12 @@ getInvocationsTx
-> ScheduledEventPagination
-> Q.TxE QErr (WithTotalCount [ScheduledEventInvocation])
getInvocationsTx invocationsBy pagination = do
let sql = Q.fromBuilder $ toSQL $ mkPaginationSelectExp allRowsSelect pagination
let sql = Q.fromBuilder $ toSQL $ getInvocationsQuery invocationsBy pagination
(withCount . Q.getRow) <$> Q.withQE defaultTxErrorHandler sql () True
getInvocationsQuery :: GetInvocationsBy -> ScheduledEventPagination -> S.Select
getInvocationsQuery invocationsBy pagination =
mkPaginationSelectExp allRowsSelect pagination
where
createdAtOrderBy table =
let createdAtCol = S.SEQIdentifier $ S.mkQIdentifierTable table $ Identifier "created_at"

2
server/src-lib/Hasura/GraphQL/Logging.hs
View File

@ -43,9 +43,7 @@ instance L.ToEngineLog QueryLog L.Hasura where
class Monad m => MonadQueryLog m where
logQueryLog
:: L.Logger L.Hasura
-- ^ logger
-> GQLReqUnparsed
-- ^ GraphQL request
-> Maybe (G.Name, EQ.PreparedSql)
-- ^ Generated SQL if any
-> RequestId

2
server/src-lib/Hasura/GraphQL/Transport/WebSocket/Server.hs
View File

@ -1,6 +1,6 @@
{-# LANGUAGE CPP #-}
{-# LANGUAGE NondecreasingIndentation #-}
{-# LANGUAGE RankNTypes #-}
{-# LANGUAGE CPP #-}
module Hasura.GraphQL.Transport.WebSocket.Server
( WSId(..)

29
server/src-lib/Hasura/RQL/DDL/ApiLimit.hs Normal file
View File

@ -0,0 +1,29 @@
-- |
module Hasura.RQL.DDL.ApiLimit where
import Hasura.Prelude
import Control.Lens ((.~))
import Hasura.EncJSON
import Hasura.RQL.Types
runSetApiLimits
:: (MonadError QErr m, MetadataM m, CacheRWM m)
=> ApiLimit -> m EncJSON
runSetApiLimits al = do
withNewInconsistentObjsCheck
$ buildSchemaCache
$ MetadataModifier
$ metaApiLimits .~ al
return successMsg
runRemoveApiLimits
:: (MonadError QErr m, MetadataM m, CacheRWM m)
=> m EncJSON
runRemoveApiLimits = do
withNewInconsistentObjsCheck
$ buildSchemaCache
$ MetadataModifier
$ metaApiLimits .~ emptyApiLimit
return successMsg

26
server/src-lib/Hasura/RQL/DDL/Metadata.hs
View File

@ -9,6 +9,9 @@ module Hasura.RQL.DDL.Metadata
, runGetCatalogState
, runSetCatalogState
, runSetMetricsConfig
, runRemoveMetricsConfig
, module Hasura.RQL.DDL.Metadata.Types
) where
@ -20,7 +23,7 @@ import qualified Data.HashMap.Strict.InsOrd as OMap
import qualified Data.HashSet as HS
import qualified Data.List as L
import Control.Lens ((^?))
import Control.Lens ((.~), (^?))
import Data.Aeson
import Hasura.Metadata.Class
@ -93,6 +96,7 @@ runReplaceMetadata replaceMetadata = do
pure $ Metadata (OMap.singleton defaultSource newDefaultSourceMetadata)
_mnsRemoteSchemas _mnsQueryCollections _mnsAllowlist
_mnsCustomTypes _mnsActions _mnsCronTriggers (_metaRestEndpoints oldMetadata)
emptyApiLimit emptyMetricsConfig
putMetadata metadata
buildSchemaCacheStrict
-- See Note [Clear postgres schema for dropped triggers]
@ -199,3 +203,23 @@ runSetCatalogState
runSetCatalogState SetCatalogState{..} = do
updateCatalogState _scsType _scsState
pure successMsg
runSetMetricsConfig
:: (MonadIO m, CacheRWM m, MetadataM m, MonadError QErr m)
=> MetricsConfig -> m EncJSON
runSetMetricsConfig mc = do
withNewInconsistentObjsCheck
$ buildSchemaCache
$ MetadataModifier
$ metaMetricsConfig .~ mc
pure successMsg
runRemoveMetricsConfig
:: (MonadIO m, CacheRWM m, MetadataM m, MonadError QErr m)
=> m EncJSON
runRemoveMetricsConfig = do
withNewInconsistentObjsCheck
$ buildSchemaCache
$ MetadataModifier
$ metaMetricsConfig .~ emptyMetricsConfig
pure successMsg

28
server/src-lib/Hasura/RQL/DDL/Metadata/Generator.hs
View File

@ -48,6 +48,8 @@ genMetadata =
<*> arbitrary
<*> arbitrary
<*> arbitrary
<*> arbitrary
<*> arbitrary
instance (Arbitrary k, Eq k, Hashable k, Arbitrary v) => Arbitrary (InsOrdHashMap k v) where
arbitrary = OM.fromList <$> arbitrary
@ -454,3 +456,29 @@ sampleGraphQLValues = [ G.VInt 1
, G.VString "article"
, G.VBoolean True
]
instance Arbitrary MetricsConfig where
arbitrary = genericArbitrary
instance Arbitrary ApiLimit where
arbitrary = genericArbitrary
instance Arbitrary DepthLimit where
arbitrary = genericArbitrary
instance Arbitrary RateLimit where
arbitrary = genericArbitrary
instance Arbitrary RateLimitConfig where
arbitrary = genericArbitrary
instance Arbitrary UniqueParamConfig where
arbitrary = elements sampleUniqueParamConfigs
sampleUniqueParamConfigs :: [UniqueParamConfig]
sampleUniqueParamConfigs = [ UPCIpAddress
, UPCSessionVar ["x-hasura-user-id"]
, UPCSessionVar ["x-hasura-user-id", "x-hasura-team-id"]
, UPCSessionVar ["x-hasura-user-id", "x-hasura-team-id", "x-hasura-org-id"]
]

6
server/src-lib/Hasura/RQL/DDL/Schema/Cache.hs
View File

@ -195,6 +195,8 @@ buildSchemaCacheRule env = proc (metadata, invalidationKeys) -> do
<> dependencyInconsistentObjects
<> toList gqlSchemaInconsistentObjects
<> toList relaySchemaInconsistentObjects
, scApiLimits = _boApiLimits resolvedOutputs
, scMetricsConfig = _boMetricsConfig resolvedOutputs
}
where
resolveSourceIfNeeded
@ -286,7 +288,7 @@ buildSchemaCacheRule env = proc (metadata, invalidationKeys) -> do
=> (Metadata, Inc.Dependency InvalidationKeys) `arr` BuildOutputs 'Postgres
buildAndCollectInfo = proc (metadata, invalidationKeys) -> do
let Metadata sources remoteSchemas collections allowlists
customTypes actions cronTriggers endpoints = metadata
customTypes actions cronTriggers endpoints apiLimits metricsConfig = metadata
remoteSchemaPermissions =
let remoteSchemaPermsList = OMap.toList $ _rsmPermissions <$> remoteSchemas
in concat $ flip map remoteSchemaPermsList $
@ -371,6 +373,8 @@ buildSchemaCacheRule env = proc (metadata, invalidationKeys) -> do
, _boCustomTypes = annotatedCustomTypes
, _boCronTriggers = cronTriggersMap
, _boEndpoints = resolvedEndpoints
, _boApiLimits = apiLimits
, _boMetricsConfig = metricsConfig
}
mkEndpointMetadataObject (name, createEndpoint) =

2
server/src-lib/Hasura/RQL/DDL/Schema/Cache/Common.hs
View File

@ -111,6 +111,8 @@ data BuildOutputs (b :: BackendType)
, _boCustomTypes :: !(AnnotatedCustomTypes b)
, _boCronTriggers :: !(M.HashMap TriggerName CronTriggerInfo)
, _boEndpoints :: !(M.HashMap EndpointName (EndpointMetadata GQLQueryWithText))
, _boApiLimits :: !ApiLimit
, _boMetricsConfig :: !MetricsConfig
}
$(makeLenses ''BuildOutputs)

3
server/src-lib/Hasura/RQL/DDL/Schema/LegacyCatalog.hs
View File

@ -342,8 +342,7 @@ fetchMetadataFromHdbTables = liftTx do
actions <- oMapFromL _amName <$> fetchActions
MetadataNoSources fullTableMetaMap functions remoteSchemas collections
allowlist customTypes actions <$> fetchCronTriggers
allowlist customTypes actions <$> fetchCronTriggers
where
modMetaMap l f xs = do
st <- get

1
server/src-lib/Hasura/RQL/Types.hs
View File

@ -49,6 +49,7 @@ import qualified Hasura.Backends.Postgres.SQL.Types as PG
import Hasura.Backends.Postgres.Connection as R
import Hasura.RQL.IR.BoolExp as R
import Hasura.RQL.Types.Action as R
import Hasura.RQL.Types.ApiLimit as R
import Hasura.RQL.Types.Column as R
import Hasura.RQL.Types.Common as R
import Hasura.RQL.Types.ComputedField as R

114
server/src-lib/Hasura/RQL/Types/ApiLimit.hs Normal file
View File

@ -0,0 +1,114 @@
-- |
module Hasura.RQL.Types.ApiLimit where
import Control.Lens
import Hasura.Prelude
import qualified Data.Aeson.Casing as Casing
import qualified Data.Text as T
import Data.Aeson
import Hasura.Server.Utils (isSessionVariable)
import Hasura.Session (RoleName)
data ApiLimit
= ApiLimit
{ _alRateLimit :: !(Maybe RateLimit)
, _alDepthLimit :: !(Maybe DepthLimit)
, _alDisabled :: !Bool
} deriving (Show, Eq, Generic)
instance FromJSON ApiLimit where
parseJSON = withObject "ApiLimit" $ \o ->
ApiLimit
<$> o .:? "rate_limit"
<*> o .:? "depth_limit"
<*> o .:? "disabled" .!= False
instance ToJSON ApiLimit where
toJSON =
genericToJSON (Casing.aesonPrefix Casing.snakeCase) { omitNothingFields = True }
emptyApiLimit :: ApiLimit
emptyApiLimit = ApiLimit Nothing Nothing False
data RateLimit
= RateLimit
{ _rlGlobal :: !RateLimitConfig
, _rlPerRole :: !(InsOrdHashMap RoleName RateLimitConfig)
} deriving (Show, Eq, Generic)
instance FromJSON RateLimit where
parseJSON = withObject "RateLimit" $ \o ->
RateLimit <$> o .: "global" <*> o .:? "per_role" .!= mempty
instance ToJSON RateLimit where
toJSON =
genericToJSON (Casing.aesonPrefix Casing.snakeCase)
data RateLimitConfig
= RateLimitConfig
{ _rlcMaxReqsPerMin :: !Int
, _rlcUniqueParams :: !(Maybe UniqueParamConfig)
} deriving (Show, Eq, Generic)
instance FromJSON RateLimitConfig where
parseJSON =
genericParseJSON (Casing.aesonPrefix Casing.snakeCase)
instance ToJSON RateLimitConfig where
toJSON =
genericToJSON (Casing.aesonPrefix Casing.snakeCase)
-- | The unique key using which an authenticated client can be identified
data UniqueParamConfig
= UPCSessionVar ![Text]
-- ^ it can be a list of session variable (like session var in 'UserInfo')
| UPCIpAddress
-- ^ or it can be an IP address
deriving (Show, Eq, Generic)
instance ToJSON UniqueParamConfig where
toJSON = \case
UPCSessionVar xs -> toJSON xs
UPCIpAddress -> "IP"
instance FromJSON UniqueParamConfig where
parseJSON = \case
String v -> case T.toLower v of
"ip" -> pure UPCIpAddress
_ -> fail errMsg
Array xs -> traverse parseSessVar xs <&> UPCSessionVar . toList
_ -> fail errMsg
where
parseSessVar = \case
String s
| isSessionVariable s && s /= "x-hasura-role" -> pure s
| otherwise -> fail errMsg
_ -> fail errMsg
errMsg = "Not a valid value. Should be either: 'IP' or a list of Hasura session variables"
data DepthLimit
= DepthLimit
{ _dlGlobal :: !MaxDepth
, _dlPerRole :: !(InsOrdHashMap RoleName MaxDepth)
} deriving (Show, Eq, Generic)
instance FromJSON DepthLimit where
parseJSON = withObject "DepthLimit" $ \o ->
DepthLimit <$> o .: "global" <*> o .:? "per_role" .!= mempty
instance ToJSON DepthLimit where
toJSON =
genericToJSON (Casing.aesonPrefix Casing.snakeCase)
newtype MaxDepth
= MaxDepth { unMaxDepth :: Int }
deriving stock (Show, Eq, Ord, Generic)
deriving newtype (ToJSON, FromJSON, Arbitrary)
$(makeLenses ''ApiLimit)
$(makeLenses ''RateLimit)
$(makeLenses ''DepthLimit)

47
server/src-lib/Hasura/RQL/Types/Metadata.hs
View File

@ -3,6 +3,7 @@ module Hasura.RQL.Types.Metadata where
import Hasura.Prelude
import qualified Data.Aeson.Casing as Casing
import qualified Data.Aeson.Ordered as AO
import qualified Data.HashMap.Strict.Extended as M
import qualified Data.HashMap.Strict.InsOrd.Extended as OM
@ -21,6 +22,7 @@ import Data.Text.Extended
import Hasura.Backends.Postgres.SQL.Types
import Hasura.Incremental (Cacheable)
import Hasura.RQL.Types.Action
import Hasura.RQL.Types.ApiLimit
import Hasura.RQL.Types.Common
import Hasura.RQL.Types.ComputedField
import Hasura.RQL.Types.CustomTypes
@ -386,6 +388,20 @@ mkSourceMetadata name urlConf connSettings =
type Sources = InsOrdHashMap SourceName SourceMetadata
-- | Various user-controlled configuration for metrics used by Pro
data MetricsConfig
= MetricsConfig
{ _mcAnalyzeQueryVariables :: !Bool
-- ^ should the query-variables be logged and analyzed for metrics
, _mcAnalyzeResponseBody :: !Bool
-- ^ should the response-body be analyzed for empty and null responses
} deriving (Show, Eq, Generic)
$(deriveJSON (Casing.aesonPrefix Casing.snakeCase) ''MetricsConfig)
emptyMetricsConfig :: MetricsConfig
emptyMetricsConfig = MetricsConfig False False
parseNonSourcesMetadata
:: Object
-> Parser
@ -395,6 +411,8 @@ parseNonSourcesMetadata
, CustomTypes
, Actions
, CronTriggers
, ApiLimit
, MetricsConfig
)
parseNonSourcesMetadata o = do
remoteSchemas <- parseListAsMap "remote schemas" _rsmName $
@ -406,10 +424,15 @@ parseNonSourcesMetadata o = do
actions <- parseListAsMap "actions" _amName $ o .:? "actions" .!= []
cronTriggers <- parseListAsMap "cron triggers" ctName $
o .:? "cron_triggers" .!= []
apiLimits <- o .:? "api_limits" .!= emptyApiLimit
metricsConfig <- o .:? "metrics_config" .!= emptyMetricsConfig
pure ( remoteSchemas, queryCollections, allowlist, customTypes
, actions, cronTriggers
, actions, cronTriggers, apiLimits, metricsConfig
)
-- | A complete GraphQL Engine metadata representation to be stored,
-- exported/replaced via metadata queries.
data Metadata
@ -422,7 +445,10 @@ data Metadata
, _metaActions :: !Actions
, _metaCronTriggers :: !CronTriggers
, _metaRestEndpoints :: !Endpoints
, _metaApiLimits :: !ApiLimit
, _metaMetricsConfig :: !MetricsConfig
} deriving (Show, Eq)
$(makeLenses ''Metadata)
instance FromJSON Metadata where
@ -434,13 +460,14 @@ instance FromJSON Metadata where
endpoints <- oMapFromL _ceName <$> o .:? "rest_endpoints" .!= []
(remoteSchemas, queryCollections, allowlist, customTypes,
actions, cronTriggers) <- parseNonSourcesMetadata o
actions, cronTriggers, apiLimits, metricsConfig) <- parseNonSourcesMetadata o
pure $ Metadata sources remoteSchemas queryCollections allowlist
customTypes actions cronTriggers endpoints
customTypes actions cronTriggers endpoints apiLimits metricsConfig
emptyMetadata :: Metadata
emptyMetadata =
Metadata mempty mempty mempty mempty emptyCustomTypes mempty mempty mempty
emptyApiLimit emptyMetricsConfig
tableMetadataSetter
:: SourceName -> QualifiedTable -> ASetter' Metadata TableMetadata
@ -477,7 +504,7 @@ instance FromJSON MetadataNoSources where
pure (tables, functions)
MVVersion3 -> fail "unexpected version for metadata without sources: 3"
(remoteSchemas, queryCollections, allowlist, customTypes,
actions, cronTriggers) <- parseNonSourcesMetadata o
actions, cronTriggers, _, _) <- parseNonSourcesMetadata o
pure $ MetadataNoSources tables functions remoteSchemas queryCollections
allowlist customTypes actions cronTriggers
@ -516,7 +543,9 @@ metadataToOrdJSON ( Metadata
actions
cronTriggers
endpoints
) = AO.object $ [versionPair, sourcesPair] <>
apiLimits
metricsConfig
) = AO.object $ [ versionPair , sourcesPair] <>
catMaybes [ remoteSchemasPair
, queryCollectionsPair
, allowlistPair
@ -524,6 +553,8 @@ metadataToOrdJSON ( Metadata
, customTypesPair
, cronTriggersPair
, endpointsPair
, apiLimitsPair
, metricsConfigPair
]
where
versionPair = ("version", AO.toOrdered currentMetadataVersion)
@ -537,6 +568,12 @@ metadataToOrdJSON ( Metadata
cronTriggersPair = listToMaybeOrdPairSort "cron_triggers" crontriggerQToOrdJSON ctName cronTriggers
endpointsPair = listToMaybeOrdPairSort "rest_endpoints" AO.toOrdered _ceUrl endpoints
apiLimitsPair = if apiLimits == emptyApiLimit then Nothing
else Just ("api_limits", AO.toOrdered apiLimits)
metricsConfigPair = if metricsConfig == emptyMetricsConfig then Nothing
else Just ("metrics_config", AO.toOrdered metricsConfig)
sourceMetaToOrdJSON :: SourceMetadata -> AO.Value
sourceMetaToOrdJSON SourceMetadata{..} =
let sourceNamePair = ("name", AO.toOrdered _smName)

3
server/src-lib/Hasura/RQL/Types/SchemaCache.hs
View File

@ -147,6 +147,7 @@ import Hasura.Incremental (Cacheable, Dependency, Mon
selectKeyD)
import Hasura.RQL.IR.BoolExp
import Hasura.RQL.Types.Action
import Hasura.RQL.Types.ApiLimit
import Hasura.RQL.Types.Common
import Hasura.RQL.Types.ComputedField
import Hasura.RQL.Types.CustomTypes
@ -285,6 +286,8 @@ data SchemaCache
, scInconsistentObjs :: ![InconsistentMetadata]
, scCronTriggers :: !(M.HashMap TriggerName CronTriggerInfo)
, scEndpoints :: !(EndpointTrie GQLQueryWithText)
, scApiLimits :: !ApiLimit
, scMetricsConfig :: !MetricsConfig
}
$(deriveToJSON hasuraJSON ''SchemaCache)

15
server/src-lib/Hasura/Server/API/Metadata.hs
View File

@ -17,6 +17,7 @@ import qualified Hasura.Tracing as Tracing
import Hasura.EncJSON
import Hasura.Metadata.Class
import Hasura.RQL.DDL.Action
import Hasura.RQL.DDL.ApiLimit
import Hasura.RQL.DDL.ComputedField
import Hasura.RQL.DDL.CustomTypes
import Hasura.RQL.DDL.Endpoint
@ -142,6 +143,14 @@ data RQLMetadata
| RMGetCatalogState !GetCatalogState
| RMSetCatalogState !SetCatalogState
-- 'ApiLimit' related
| RMSetApiLimits !ApiLimit
| RMRemoveApiLimits
-- 'MetricsConfig' related
| RMSetMetricsConfig !MetricsConfig
| RMRemoveMetricsConfig
-- bulk metadata queries
| RMBulk [RQLMetadata]
deriving (Show, Eq)
@ -290,4 +299,10 @@ runMetadataQueryM env = withPathK "args" . \case
RMGetCatalogState q -> runGetCatalogState q
RMSetCatalogState q -> runSetCatalogState q
RMSetApiLimits q -> runSetApiLimits q
RMRemoveApiLimits -> runRemoveApiLimits
RMSetMetricsConfig q -> runSetMetricsConfig q
RMRemoveMetricsConfig -> runRemoveMetricsConfig
RMBulk q -> encJFromList <$> indexedMapM (runMetadataQueryM env) q

33
server/src-lib/Hasura/Server/App.hs
View File

@ -754,7 +754,7 @@ mkWaiApp
-> E.PlanCacheOptions
-> ResponseInternalErrorsConfig
-> Maybe EL.LiveQueryPostPollHook
-> RebuildableSchemaCache
-> SchemaCacheRef
-> EKG.Store
-> RemoteSchemaPermsCtx
-> FunctionPermissionsCtx
@ -764,11 +764,8 @@ mkWaiApp
-> m HasuraApp
mkWaiApp env logger sqlGenCtx enableAL httpManager mode corsCfg enableConsole consoleAssetsDir
enableTelemetry instanceId apis lqOpts _ {- planCacheOptions -} responseErrorsConfig
liveQueryHook schemaCache ekgStore enableRSPermsCtx functionPermsCtx connectionOptions keepAliveDelay = do
liveQueryHook schemaCacheRef ekgStore enableRSPermsCtx functionPermsCtx connectionOptions keepAliveDelay = do
-- See Note [Temporarily disabling query plan caching]
-- (planCache, schemaCacheRef) <- initialiseCache
schemaCacheRef <- initialiseCache
let getSchemaCache = first lastBuiltSchemaCache <$> readIORef (_scrCache schemaCacheRef)
let corsPolicy = mkDefaultCorsPolicy corsCfg
@ -807,17 +804,17 @@ mkWaiApp env logger sqlGenCtx enableAL httpManager mode corsCfg enableConsole co
pure $ WSC.websocketsOr connectionOptions (\ip conn -> lowerIO $ wsServerApp ip conn) spockApp
return $ HasuraApp waiApp schemaCacheRef stopWSServer
where
-- initialiseCache :: m (E.PlanCache, SchemaCacheRef)
initialiseCache :: m SchemaCacheRef
initialiseCache = do
cacheLock <- liftIO $ newMVar ()
cacheCell <- liftIO $ newIORef (schemaCache, initSchemaCacheVer)
-- planCache <- liftIO $ E.initPlanCache planCacheOptions
let cacheRef = SchemaCacheRef cacheLock cacheCell E.clearPlanCache
-- pure (planCache, cacheRef)
pure cacheRef
-- initialiseCache :: m (E.PlanCache, SchemaCacheRef)
initialiseCache :: MonadIO m => RebuildableSchemaCache -> m SchemaCacheRef
initialiseCache schemaCache = do
cacheLock <- liftIO $ newMVar ()
cacheCell <- liftIO $ newIORef (schemaCache, initSchemaCacheVer)
-- planCache <- liftIO $ E.initPlanCache planCacheOptions
let cacheRef = SchemaCacheRef cacheLock cacheCell E.clearPlanCache
-- pure (planCache, cacheRef)
pure cacheRef
httpApp
@ -979,7 +976,7 @@ httpApp corsCfg serverCtx enableConsole consoleAssetsDir enableTelemetry = do
forM_ [Spock.GET, Spock.POST] $ \m -> Spock.hookAny m $ \_ -> do
req <- Spock.request
let headers = Wai.requestHeaders req
let qErr = err404 NotFound "resource does not exist"
qErr = err404 NotFound "resource does not exist"
raiseGenericApiError logger headers qErr
where
@ -1005,14 +1002,14 @@ httpApp corsCfg serverCtx enableConsole consoleAssetsDir enableTelemetry = do
-- serve static files if consoleAssetsDir is set
onJust consoleAssetsDir $ \dir ->
Spock.get ("console/assets" <//> Spock.wildcard) $ \path ->
Spock.get ("console/assets" <//> Spock.wildcard) $ \path -> do
consoleAssetsHandler logger dir (T.unpack path)
-- serve console html
Spock.get ("console" <//> Spock.wildcard) $ \path -> do
req <- Spock.request
let headers = Wai.requestHeaders req
let authMode = scAuthMode serverCtx
authMode = scAuthMode serverCtx
consoleHtml <- lift $ renderConsole path authMode enableTelemetry consoleAssetsDir
either (raiseGenericApiError logger headers . err500 Unexpected . T.pack) Spock.html consoleHtml

1
server/src-lib/Hasura/Server/Migrate.hs
View File

@ -292,6 +292,7 @@ migrations maybeDefaultSourceConfig dryRun =
SourceMetadata defaultSource _mnsTables _mnsFunctions defaultSourceConfig
in Metadata (OMap.singleton defaultSource defaultSourceMetadata)
_mnsRemoteSchemas _mnsQueryCollections _mnsAllowlist _mnsCustomTypes _mnsActions _mnsCronTriggers mempty
emptyApiLimit emptyMetricsConfig
liftTx $ setMetadataInCatalog metadataV3
from43To42 = do

1
server/src-lib/Hasura/Server/SchemaUpdate.hs
View File

@ -4,6 +4,7 @@ module Hasura.Server.SchemaUpdate
, startSchemaSyncProcessorThread
, EventPayload(..)
, SchemaSyncCtx(..)
, SchemaSyncEventRef
)
where