digital-asset/daml
1
1
Fork 0
mirror of https://github.com/digital-asset/daml.git synced 2024-09-20 01:07:18 +03:00
Code Issues Projects Releases Wiki Activity

[JSON-API] HOCON config json api (#12236)

Browse Source 
* Change heartBeatPer to more intuitive naming of heartbeatPeriod

CHANGELOG_BEGIN
CHANGELOG_END

* Initial changes to add HOCON config for json_api

CHANGELOG_BEGIN
CHANGELOG_END

* avoid IllegalArgumentException noise

* use named arguments in big config conversion

* Changes include
 - tests for a full http-json-api config file
 - logging config and non-repudiation config is still specified via cli args.
 - config readers for MetricsReporter

* Add defaults to WebsocketConfig case class to allow partially specifying fields on typeconf file

* changes to the JwtVerifierBase config reader and equivalent test

* message already describes the value

* replace manual succeed/fails with scalatest combinators

* use qualified imports for WebsocketConfig defaults

* add back autodeleted empty lines

* collapse two lists of token verifiers into one

* add new line to config files

* rename dbStartupMode to startMode to keep consistent with cli option and for easy documentation

* Changes to daml docs to specify ways to run JSON-API by supplying a HOCON config file.

CHANGELOG_BEGIN
JSON-API can now be started supplying a HOCON application config file using the `--config` option.
All CLI flags except `logging` and `non-repudiation` one's are now deprecated and will be cleaned up in some future releases.
CHANGELOG_END

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
This commit is contained in:
akshayshirahatti-da 2022-01-10 23:07:07 +00:00 committed by GitHub
parent 9f5a2f9778
commit 50de6e3639
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
30 changed files with 853 additions and 116 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

149
docs/source/json-api/index.rst
View File

@ -63,12 +63,128 @@ The most basic way to start the JSON API is with the command:
.. code-block:: shell
daml json-api --ledger-host localhost --ledger-port 6865 --http-port 7575
daml json-api --config json-api-app.conf
where a corresponding minimal config file is
.. code-block:: none
{
server {
address = "localhost"
port = 7575
}
ledger-api {
address = "localhost"
port = 6865
}
}
This will start the JSON API on port 7575 and connect it to a ledger running on ``localhost:6865``.
.. note:: Your JSON API service should never be exposed to the internet. When running in production the JSON API should be behind a `reverse proxy, such as via NGINX <https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/>`_.
The full set of configurable options that can be specified via config file is listed below
.. code-block:: none
{
server {
//IP address that HTTP JSON API service listens on. Defaults to 127.0.0.1.
address = "127.0.0.1"
//HTTP JSON API service port number. A port number of 0 will let the system pick an ephemeral port.
port = 7575
}
ledger-api {
address = "127.0.0.1"
port = 6865
tls {
enabled = "true"
// the certificate to be used by the server
cert-chain-file = "cert-chain.crt"
// private key of the server
private-key-file = "pvt-key.pem"
// trust collection, which means that all client certificates will be verified using the trusted
// certificates in this store. if omitted, the JVM default trust store is used.
trust-collection-file = "root-ca.crt"
}
}
query-store {
base-config {
user = "postgres"
password = "password"
driver = "org.postgresql.Driver"
url = "jdbc:postgresql://localhost:5432/test?&ssl=true"
// prefix for table names to avoid collisions, empty by default
table-prefix = "foo"
// max pool size for the database connection pool
pool-size = 12
//specifies the min idle connections for database connection pool.
min-idle = 4
//specifies the idle timeout for the database connection pool.
idle-timeout = 12s
//specifies the connection timeout for database connection pool.
connection-timeout = 90s
}
// option setting how the schema should be handled.
// Valid options are start-only, create-only, create-if-needed-and-start and create-and-start
start-mode = "start-only"
}
// Optional interval to poll for package updates. Examples: 500ms, 5s, 10min, 1h, 1d. Defaults to 5 seconds
package-reload-interval = 5s
//Optional max inbound message size in bytes. Defaults to 4194304.
max-inbound-message-size = 4194304
//Optional max inbound message size in bytes used for uploading and downloading package updates. Defaults to the `max-inbound-message-size` setting.
package-max-inbound-message-size = 4194304
//Optional max cache size in entries for storing surrogate template id mappings. Defaults to None
max-template-id-cache-entries = 1000
//health check timeout in seconds
health-timeout-seconds = 5
//Optional websocket configuration parameters
websocket-config {
//Maximum websocket session duration
max-duration = 120m
//Server-side heartbeat interval duration
heartbeat-period = 5s
//akka stream throttle-mode one of either `shaping` or `enforcing`
mode = "shaping"
}
metrics {
//Start a metrics reporter. Must be one of "console", "csv:///PATH", "graphite://HOST[:PORT][/METRIC_PREFIX]", or "prometheus://HOST[:PORT]".
reporter = "console"
//Set metric reporting interval , examples : 1s, 30s, 1m, 1h
reporting-interval = 30s
}
// DEV MODE ONLY (not recommended for production)
// Allow connections without a reverse proxy providing HTTPS.
allow-insecure-tokens = false
// Optional static content configuration string. Contains comma-separated key-value pairs, where:
// prefix -- URL prefix,
// directory -- local directory that will be mapped to the URL prefix.
// Example: "prefix=static,directory=./static-content"
static-content {
prefix = "static"
directory = "static-content-dir"
}
}
.. note:: You can also start JSON API using CLI args (example below) however this is now deprecated
.. code-block:: shell
daml json-api --ledger-host localhost --ledger-port 6865 --http-port 7575
Standalone JAR
--------------
@ -84,7 +200,7 @@ start the standalone JAR, you can use the following command:
.. code-block:: shell
java -jar http-json-1.5.0.jar --ledger-host localhost --ledger-port 6865 --http-port 7575
java -jar http-json-1.5.0.jar --config json-api-app.conf
Replace the version number ``1.5.0`` by the version of the SDK you are
using.
@ -99,10 +215,37 @@ your query every time so it is generally not recommended to rely on
this in production. Note that the PostgreSQL backend acts purely as a
cache. It is safe to reinitialize the database at any time.
To enable the PostgreSQL backend you can use the ``--query-store-jdbc-config`` flag, an example of which is below.
To enable the PostgreSQL backend you can add the ``query-store`` config block in your application config file
.. code-block:: none
query-store {
base-config {
user = "postgres"
password = "password"
driver = "org.postgresql.Driver"
url = "jdbc:postgresql://localhost:5432/test?&ssl=true"
// prefix for table names to avoid collisions, empty by default
table-prefix = "foo"
// max pool size for the database connection pool
pool-size = 12
//specifies the min idle connections for database connection pool.
min-idle = 4
//specifies the idle timeout for the database connection pool.
idle-timeout = 12s
//specifies the connection timeout for database connection pool.
connection-timeout = 90s
}
// option setting how the schema should be handled.
// Valid options are start-only, create-only, create-if-needed-and-start and create-and-start
start-mode = "create-if-needed-and-start"
}
.. note:: When you use the Query Store you'll want to use ``start-mode=create-if-needed-and-start`` so that all the necessary tables are created if they don't exist.
you can also use the ``--query-store-jdbc-config`` CLI flag (deprecated), an example of which is below.
.. code-block:: shell

72
docs/source/json-api/production-setup.rst
View File

@ -30,7 +30,35 @@ The query store is built by saving the state of the ACS up to the current ledger
offset. This allows the *HTTP JSON API* to only request the delta on subsequent queries,
making it much faster than having to request the entire ACS every time.
For example to enable the PostgreSQL backend you can use the ``--query-store-jdbc-config`` flag, as shown below.
For example to enable the PostgreSQL backend you can add the ``query-store`` config block in your application config file
.. code-block:: none
query-store {
base-config {
user = "postgres"
password = "password"
driver = "org.postgresql.Driver"
url = "jdbc:postgresql://localhost:5432/test?&ssl=true"
// prefix for table names to avoid collisions, empty by default
table-prefix = "foo"
// max pool size for the database connection pool
pool-size = 12
//specifies the min idle connections for database connection pool.
min-idle = 4
//specifies the idle timeout for the database connection pool.
idle-timeout = 12s
//specifies the connection timeout for database connection pool.
connection-timeout = 90s
}
// option setting how the schema should be handled.
// Valid options are start-only, create-only, create-if-needed-and-start and create-and-start
start-mode = "start-only"
}
You can also use the ``--query-store-jdbc-config`` CLI flag (deprecated), as shown below.
.. code-block:: shell
@ -39,9 +67,8 @@ For example to enable the PostgreSQL backend you can use the ``--query-store-jdb
Consult your database vendor's JDBC driver documentation to learn how to specify a JDBC connection string that suits your needs.
Despite appearing in the JDBC connection string, the ``start-mode`` is a custom parameter defined by
the query store configuration itself which allows to deal with the initialization and usage of the
database which backs the query store.
The ``start-mode`` is a custom parameter defined by the query store configuration itself which allows to deal
with the initialization and usage of the database which backs the query store.
Depending on how you prefer to operate it, you can either choose to:
@ -92,8 +119,28 @@ rest and using a secure communication channel between the *HTTP JSON API* server
To protect data in transit and over untrusted networks, the *HTTP JSON API* server provides
TLS support, to enable TLS you need to specify the private key for your server and the
certificate chain via ``daml json-api --pem server.pem --crt server.crt``. You can also
set a custom root CA certificate used to validate client certificates via ``--cacrt ca.crt``
certificate chain via the below config block specifying the ``cert-chain-file``, ``private-key-file``, you can also set
a custom root CA certificate used to validate client certificates via ``trust-collection-file`` parameter.
.. code-block:: none
ledger-api {
address = "127.0.0.1"
port = 6400
tls {
enabled = "true"
// the certificate to be used by the server
cert-chain-file = "cert-chain.crt"
// private key of the server
private-key-file = "pvt-key.pem"
// trust collection, which means that all client certificates will be verified using the trusted
// certificates in this store. if omitted, the JVM default trust store is used.
trust-collection-file = "root-ca.crt"
}
}
Using the cli options (deprecated), you can specify tls options using``daml json-api --pem server.pem --crt server.crt``.
Custom root CA certificate can be set via ``--cacrt ca.crt``
For more details on secure DAML infrastructure setup please refer to this `reference implementation <https://github.com/digital-asset/ex-secure-daml-infra>`__
@ -193,7 +240,18 @@ Enable and configure reporting
------------------------------
To enable metrics and configure reporting, you can use the two following CLI options:
To enable metrics and configure reporting, you can use the below config block in application config
.. code-block:: none
metrics {
//Start a metrics reporter. Must be one of "console", "csv:///PATH", "graphite://HOST[:PORT][/METRIC_PREFIX]", or "prometheus://HOST[:PORT]".
reporter = "console"
//Set metric reporting interval , examples : 1s, 30s, 1m, 1h
reporting-interval = 30s
}
or the two following CLI options (deprecated):
- ``--metrics-reporter``: passing a legal value will enable reporting; the accepted values
are as follows:

5
ledger-service/http-json-cli/BUILD.bazel
View File

@ -12,6 +12,9 @@ da_scala_library(
srcs = glob(["src/main/scala/**/*.scala"]),
plugins = [silencer_plugin],
scala_deps = [
"@maven//:com_chuusai_shapeless",
"@maven//:com_github_pureconfig_pureconfig_core",
"@maven//:com_github_pureconfig_pureconfig_generic",
"@maven//:com_github_scopt_scopt",
"@maven//:org_scala_lang_modules_scala_collection_compat",
"@maven//:org_scalaz_scalaz_core",
@ -21,10 +24,12 @@ da_scala_library(
visibility = ["//ledger-service:__subpackages__"],
deps = [
"//ledger-service/cli-opts",
"//ledger-service/pureconfig-utils",
"//ledger/ledger-api-common",
"//ledger/metrics",
"//libs-scala/db-utils",
"@maven//:ch_qos_logback_logback_classic",
"@maven//:com_typesafe_config",
"@maven//:io_netty_netty_handler",
"@maven//:org_slf4j_slf4j_api",
],

2
ledger-service/http-json-cli/ee/src/main/scala/com/daml/http/NonRepudiationOptions.scala
View File

@ -5,7 +5,7 @@ package com.daml.http
import java.nio.file.Paths
trait NonRepudiationOptions { this: scopt.OptionParser[Config] =>
trait NonRepudiationOptions { this: scopt.OptionParser[JsonApiCli] =>
opt[String]("non-repudiation-certificate-path")
.action((path, config) =>

2
ledger-service/http-json-cli/src/main/scala/com/daml/http/CliBase.scala
View File

@ -14,7 +14,7 @@ trait CliBase {
): Option[Config] = {
implicit val jcd: DBConfig.JdbcConfigDefaults =
DBConfig.JdbcConfigDefaults(supportedJdbcDriverNames)
configParser(getEnvVar).parse(args, Config.Empty)
configParser(getEnvVar).parse(args, JsonApiCli.Default).flatMap(_.loadConfig)
}
protected[this] def configParser(getEnvVar: String => Option[String])(implicit

51
ledger-service/http-json-cli/src/main/scala/com/daml/http/Config.scala
View File

@ -18,6 +18,7 @@ import scala.concurrent.duration._
import ch.qos.logback.classic.{Level => LogLevel}
import com.daml.cliopts.Logging.LogEncoder
import com.daml.http.{WebsocketConfig => WSC}
import com.daml.metrics.MetricsReporter
import com.daml.http.dbbackend.JdbcConfig
@ -42,41 +43,38 @@ private[http] final case class Config(
logLevel: Option[LogLevel] = None, // the default is in logback.xml
logEncoder: LogEncoder = LogEncoder.Plain,
metricsReporter: Option[MetricsReporter] = None,
metricsReportingInterval: FiniteDuration = 10 seconds,
metricsReportingInterval: FiniteDuration = StartSettings.DefaultMetricsReportingInterval,
surrogateTpIdCacheMaxEntries: Option[Long] = None,
) extends StartSettings
private[http] object Config {
import scala.language.postfixOps
val Empty = Config(ledgerHost = "", ledgerPort = -1, httpPort = -1)
val DefaultWsConfig =
WebsocketConfig(
maxDuration = 120 minutes,
throttleElem = 20,
throttlePer = 1 second,
maxBurst = 20,
ThrottleMode.Shaping,
heartBeatPer = 5 second,
)
}
// It is public for Daml Hub
final case class WebsocketConfig(
maxDuration: FiniteDuration,
throttleElem: Int,
throttlePer: FiniteDuration,
maxBurst: Int,
mode: ThrottleMode,
heartBeatPer: FiniteDuration,
maxDuration: FiniteDuration = WSC.DefaultMaxDuration,
throttleElem: Int = WSC.DefaultThrottleElem,
throttlePer: FiniteDuration = WSC.DefaultThrottlePer,
maxBurst: Int = WSC.DefaultMaxBurst,
mode: ThrottleMode = WSC.DefaultThrottleMode,
heartbeatPeriod: FiniteDuration = WSC.DefaultHeartbeatPeriod,
)
private[http] object WebsocketConfig
extends ConfigCompanion[WebsocketConfig, DummyImplicit]("WebsocketConfig") {
implicit val showInstance: Show[WebsocketConfig] = Show.shows(c =>
s"WebsocketConfig(maxDuration=${c.maxDuration}, heartBeatPer=${c.heartBeatPer})"
s"WebsocketConfig(maxDuration=${c.maxDuration}, heartBeatPer=${c.heartbeatPeriod})"
)
val DefaultMaxDuration: FiniteDuration = 120.minutes
val DefaultThrottleElem: Int = 20
val DefaultThrottlePer: FiniteDuration = 1.second
val DefaultMaxBurst: Int = 20
val DefaultThrottleMode: ThrottleMode = ThrottleMode.Shaping
val DefaultHeartbeatPeriod: FiniteDuration = 5.second
lazy val help: String =
"Contains comma-separated key-value pairs. Where:\n" +
s"${indent}maxDuration -- Maximum websocket session duration in minutes\n" +
@ -92,15 +90,14 @@ private[http] object WebsocketConfig
for {
md <- optionalLongField("maxDuration")
hbp <- optionalLongField("heartBeatPer")
} yield Config.DefaultWsConfig
.copy(
maxDuration = md
.map(t => FiniteDuration(t, TimeUnit.MINUTES))
.getOrElse(Config.DefaultWsConfig.maxDuration),
heartBeatPer = hbp
.map(t => FiniteDuration(t, TimeUnit.SECONDS))
.getOrElse(Config.DefaultWsConfig.heartBeatPer),
)
} yield WebsocketConfig(
maxDuration = md
.map(t => FiniteDuration(t, TimeUnit.MINUTES))
.getOrElse(WebsocketConfig.DefaultMaxDuration),
heartbeatPeriod = hbp
.map(t => FiniteDuration(t, TimeUnit.SECONDS))
.getOrElse(WebsocketConfig.DefaultHeartbeatPeriod),
)
private def helpString(maxDuration: String, heartBeatPer: String): String =
s"""\"maxDuration=$maxDuration,heartBeatPer=$heartBeatPer\""""

94
ledger-service/http-json-cli/src/main/scala/com/daml/http/FileBasedConfig.scala Normal file
View File

@ -0,0 +1,94 @@
// Copyright (c) 2022 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
package com.daml.http
import akka.stream.ThrottleMode
import com.daml.cliopts
import com.daml.cliopts.Logging.LogEncoder
import com.daml.http.dbbackend.{DbStartupMode, JdbcConfig}
import com.daml.pureconfigutils.{HttpServerConfig, LedgerApiConfig, MetricsConfig}
import com.daml.pureconfigutils.SharedConfigReaders._
import pureconfig.ConfigReader
import pureconfig.generic.semiauto._
import ch.qos.logback.classic.{Level => LogLevel}
import scala.concurrent.duration._
private[http] object FileBasedConfig {
implicit val throttleModeCfgReader: ConfigReader[ThrottleMode] =
ConfigReader.fromString[ThrottleMode](catchConvertError { s =>
s.toLowerCase() match {
case "enforcing" => Right(ThrottleMode.Enforcing)
case "shaping" => Right(ThrottleMode.Shaping)
case _ => Left("not one of 'shaping' or 'enforcing'")
}
})
implicit val websocketCfgReader: ConfigReader[WebsocketConfig] =
deriveReader[WebsocketConfig]
implicit val staticContentCfgReader: ConfigReader[StaticContentConfig] =
deriveReader[StaticContentConfig]
implicit val dbStartupModeReader: ConfigReader[DbStartupMode] =
ConfigReader.fromString[DbStartupMode](catchConvertError { s =>
DbStartupMode.configValuesMap
.get(s.toLowerCase())
.toRight(
s"not one of ${DbStartupMode.allConfigValues.mkString(",")}"
)
})
implicit val queryStoreCfgReader: ConfigReader[JdbcConfig] = deriveReader[JdbcConfig]
implicit val httpJsonApiCfgReader: ConfigReader[FileBasedConfig] =
deriveReader[FileBasedConfig]
val Empty: FileBasedConfig = FileBasedConfig(
HttpServerConfig(cliopts.Http.defaultAddress, -1),
LedgerApiConfig("", -1),
)
}
private[http] final case class FileBasedConfig(
server: HttpServerConfig,
ledgerApi: LedgerApiConfig,
queryStore: Option[JdbcConfig] = None,
packageReloadInterval: FiniteDuration = StartSettings.DefaultPackageReloadInterval,
maxInboundMessageSize: Int = StartSettings.DefaultMaxInboundMessageSize,
healthTimeoutSeconds: Int = StartSettings.DefaultHealthTimeoutSeconds,
packageMaxInboundMessageSize: Option[Int] = None,
maxTemplateIdCacheEntries: Option[Long] = None,
websocketConfig: Option[WebsocketConfig] = None,
metrics: Option[MetricsConfig] = None,
allowInsecureTokens: Boolean = false,
staticContent: Option[StaticContentConfig] = None,
) {
def toConfig(
nonRepudiation: nonrepudiation.Configuration.Cli,
logLevel: Option[LogLevel], // the default is in logback.xml
logEncoder: LogEncoder,
): Config = {
Config(
ledgerHost = ledgerApi.address,
ledgerPort = ledgerApi.port,
address = server.address,
httpPort = server.port,
portFile = server.portFile,
packageReloadInterval = packageReloadInterval,
packageMaxInboundMessageSize = packageMaxInboundMessageSize,
maxInboundMessageSize = maxInboundMessageSize,
healthTimeoutSeconds = healthTimeoutSeconds,
tlsConfig = ledgerApi.tls.tlsConfiguration,
jdbcConfig = queryStore,
staticContentConfig = staticContent,
allowNonHttps = allowInsecureTokens,
wsConfig = websocketConfig,
nonRepudiation = nonRepudiation,
logLevel = logLevel,
logEncoder = logEncoder,
metricsReporter = metrics.map(_.reporter),
metricsReportingInterval =
metrics.map(_.reportingInterval).getOrElse(StartSettings.DefaultMetricsReportingInterval),
surrogateTpIdCacheMaxEntries = maxTemplateIdCacheEntries,
)
}
}

97
ledger-service/http-json-cli/src/main/scala/com/daml/http/JsonApiCli.scala Normal file
View File

@ -0,0 +1,97 @@
// Copyright (c) 2022 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
// SPDX-License-Identifier: Apache-2.0
package com.daml.http
import com.daml.cliopts.Logging.LogEncoder
import com.daml.http.dbbackend.JdbcConfig
import com.daml.ledger.api.tls.TlsConfiguration
import com.daml.metrics.MetricsReporter
import java.io.File
import java.nio.file.Path
import scala.concurrent.duration._
import ch.qos.logback.classic.{Level => LogLevel}
import com.typesafe.scalalogging.StrictLogging
import pureconfig.ConfigSource
import pureconfig.error.ConfigReaderFailures
import scalaz.syntax.std.option._
private[http] final case class JsonApiCli(
configFile: Option[File],
ledgerHost: String,
ledgerPort: Int,
address: String = com.daml.cliopts.Http.defaultAddress,
httpPort: Int,
portFile: Option[Path] = None,
packageReloadInterval: FiniteDuration = StartSettings.DefaultPackageReloadInterval,
packageMaxInboundMessageSize: Option[Int] = None,
maxInboundMessageSize: Int = StartSettings.DefaultMaxInboundMessageSize,
healthTimeoutSeconds: Int = StartSettings.DefaultHealthTimeoutSeconds,
tlsConfig: TlsConfiguration = TlsConfiguration(enabled = false, None, None, None),
jdbcConfig: Option[JdbcConfig] = None,
staticContentConfig: Option[StaticContentConfig] = None,
allowNonHttps: Boolean = false,
wsConfig: Option[WebsocketConfig] = None,
nonRepudiation: nonrepudiation.Configuration.Cli = nonrepudiation.Configuration.Cli.Empty,
logLevel: Option[LogLevel] = None, // the default is in logback.xml
logEncoder: LogEncoder = LogEncoder.Plain,
metricsReporter: Option[MetricsReporter] = None,
metricsReportingInterval: FiniteDuration = 10 seconds,
surrogateTpIdCacheMaxEntries: Option[Long] = None,
) extends StartSettings
with StrictLogging {
def loadFromConfigFile: Option[Either[ConfigReaderFailures, FileBasedConfig]] =
configFile.map(cf => ConfigSource.file(cf).load[FileBasedConfig])
def loadFromCliArgs: Config = {
Config(
address = address,
httpPort = httpPort,
portFile = portFile,
ledgerHost = ledgerHost,
ledgerPort = ledgerPort,
packageReloadInterval = packageReloadInterval,
packageMaxInboundMessageSize = packageMaxInboundMessageSize,
maxInboundMessageSize = maxInboundMessageSize,
healthTimeoutSeconds = healthTimeoutSeconds,
tlsConfig = tlsConfig,
jdbcConfig = jdbcConfig,
staticContentConfig = staticContentConfig,
allowNonHttps = allowNonHttps,
wsConfig = wsConfig,
nonRepudiation = nonRepudiation,
logLevel = logLevel,
logEncoder = logEncoder,
metricsReporter = metricsReporter,
metricsReportingInterval = metricsReportingInterval,
surrogateTpIdCacheMaxEntries = surrogateTpIdCacheMaxEntries,
)
}
def loadConfig: Option[Config] =
loadFromConfigFile.cata(
{
case Right(fileBasedConfig) =>
Some(
fileBasedConfig.toConfig(
nonRepudiation,
logLevel,
logEncoder,
)
)
case Left(ex) =>
logger.error(
s"Error loading json-api service config from file ${configFile}",
ex.prettyPrint(),
)
None
},
Some(loadFromCliArgs),
)
}
private[http] object JsonApiCli {
val Default = JsonApiCli(configFile = None, ledgerHost = "", ledgerPort = -1, httpPort = -1)
}

47
ledger-service/http-json-cli/src/main/scala/com/daml/http/OptionParser.scala
View File

@ -9,18 +9,19 @@ import com.daml.ledger.api.tls.TlsConfigurationCli
import com.typesafe.scalalogging.StrictLogging
import scopt.{Read, RenderingMode}
import java.io.File
import scala.concurrent.duration.{Duration, FiniteDuration}
import scala.util.Try
class OptionParser(getEnvVar: String => Option[String])(implicit
jdbcConfigDefaults: JdbcConfigDefaults
) extends scopt.OptionParser[Config]("http-json-binary")
) extends scopt.OptionParser[JsonApiCli]("http-json-binary")
with StrictLogging {
private def setJdbcConfig(
config: Config,
config: JsonApiCli,
jdbcConfig: JdbcConfig,
): Config = {
): JsonApiCli = {
if (config.jdbcConfig.exists(_ != jdbcConfig)) {
throw new IllegalStateException(
"--query-store-jdbc-config and --query-store-jdbc-config-env are mutually exclusive."
@ -47,14 +48,21 @@ class OptionParser(getEnvVar: String => Option[String])(implicit
help("help").text("Print this usage text")
opt[Option[File]]('c', "config")
.text(
"The application config file, this is the recommended way to run the service, cli-args are now deprecated"
)
.valueName("<file>")
.action((file, cli) => cli.copy(configFile = file))
opt[String]("ledger-host")
.action((x, c) => c.copy(ledgerHost = x))
.required()
.optional()
.text("Ledger host name or IP address")
opt[Int]("ledger-port")
.action((x, c) => c.copy(ledgerPort = x))
.required()
.optional()
.text("Ledger port number")
import com.daml.cliopts
@ -62,7 +70,7 @@ class OptionParser(getEnvVar: String => Option[String])(implicit
cliopts.Http.serverParse(this, serviceName = "HTTP JSON API")(
address = (f, c) => c copy (address = f(c.address)),
httpPort = (f, c) => c copy (httpPort = f(c.httpPort)),
defaultHttpPort = None,
defaultHttpPort = Some(-1),
portFile = Some((f, c) => c copy (portFile = f(c.portFile))),
)
@ -172,4 +180,31 @@ class OptionParser(getEnvVar: String => Option[String])(implicit
(f, c) => c.copy(metricsReportingInterval = f(c.metricsReportingInterval)),
)
checkConfig { cfg =>
if (cfg.configFile.isEmpty && (cfg.ledgerHost == null || cfg.ledgerPort == -1))
failure(
"Missing required values --ledger-host and/or --ledger-port values for cli args"
)
else
success
}
checkConfig { cfg =>
if (cfg.configFile.isEmpty && (cfg.httpPort == -1))
failure(
"Missing required value --http-port for HTTP-JSON-API"
)
else
success
}
//this check only checks for "required" fields to conclude that both config file and cli args were supplied
checkConfig { cfg =>
if (
cfg.configFile.isDefined && (cfg.ledgerHost != "" || cfg.ledgerPort != -1 || cfg.httpPort != -1)
)
Left("Found both config file and cli opts for the app, please provide only one of them")
else Right(())
}
}

5
ledger-service/http-json-cli/src/main/scala/com/daml/http/StartSettings.scala
View File

@ -7,7 +7,7 @@ import java.nio.file.Path
import com.daml.ledger.api.tls.TlsConfiguration
import scala.concurrent.duration.FiniteDuration
import scala.concurrent.duration._
import ch.qos.logback.classic.{Level => LogLevel}
import com.daml.cliopts.Logging.LogEncoder
@ -39,9 +39,10 @@ trait StartSettings {
object StartSettings {
val DefaultPackageReloadInterval: FiniteDuration = FiniteDuration(5, "s")
val DefaultPackageReloadInterval: FiniteDuration = 5.seconds
val DefaultMaxInboundMessageSize: Int = 4194304
val DefaultHealthTimeoutSeconds: Int = 5
val DefaultMetricsReportingInterval: FiniteDuration = 10.seconds
trait Default extends StartSettings {
override val staticContentConfig: Option[StaticContentConfig] = None

6
ledger-service/http-json-cli/src/main/scala/com/daml/http/dbbackend/JdbcConfig.scala
View File

@ -13,7 +13,7 @@ import com.daml.dbutils, dbutils.DBConfig
private[http] final case class JdbcConfig(
baseConfig: dbutils.JdbcConfig,
dbStartupMode: DbStartupMode = DbStartupMode.StartOnly,
startMode: DbStartupMode = DbStartupMode.StartOnly,
backendSpecificConf: Map[String, String] = Map.empty,
)
@ -23,7 +23,7 @@ private[http] object JdbcConfig
implicit val showInstance: Show[JdbcConfig] = Show.shows { a =>
import a._, baseConfig._
s"JdbcConfig(driver=$driver, url=$url, user=$user, start-mode=$dbStartupMode)"
s"JdbcConfig(driver=$driver, url=$url, user=$user, start-mode=$startMode)"
}
private[this] val DisableContractPayloadIndexing = "disableContractPayloadIndexing"
@ -68,7 +68,7 @@ private[http] object JdbcConfig
remainingConf <- StateT.get: Fields[Map[String, String]]
} yield JdbcConfig(
baseConfig = baseConfig,
dbStartupMode = createSchema orElse dbStartupMode getOrElse DbStartupMode.StartOnly,
startMode = createSchema orElse dbStartupMode getOrElse DbStartupMode.StartOnly,
backendSpecificConf = remainingConf,
)

2
ledger-service/http-json-oracle/src/itlib/scala/http/HttpServiceOracleInt.scala
View File

@ -44,7 +44,7 @@ object HttpServiceOracleInt {
tablePrefix = "some_nice_prefix_",
poolSize = ConnectionPool.PoolSize.Integration,
),
dbStartupMode = DbStartupMode.CreateOnly,
startMode = DbStartupMode.CreateOnly,
backendSpecificConf =
if (disableContractPayloadIndexing) Map(DisableContractPayloadIndexing -> "true")
else Map.empty,

4
ledger-service/http-json-perf/src/main/scala/com/daml/http/perf/Main.scala
View File

@ -228,7 +228,7 @@ object Main extends StrictLogging {
user.pwd,
ConnectionPool.PoolSize.Production,
),
dbStartupMode = startupMode,
startMode = startupMode,
)
}
@ -254,7 +254,7 @@ object Main extends StrictLogging {
password = "",
ConnectionPool.PoolSize.Production,
),
dbStartupMode = DbStartupMode.CreateOnly,
startMode = DbStartupMode.CreateOnly,
)
private def resolveSimulationClass(str: String): Throwable \/ Class[_ <: Simulation] = {

2
ledger-service/http-json-testing/src/main/scala/com/daml/http/HttpServiceTestFixture.scala
View File

@ -272,7 +272,7 @@ object HttpServiceTestFixture extends LazyLogging with Assertions with Inside {
for {
dao <- Future(ContractDao(c))
isSuccess <- DbStartupOps
.fromStartupMode(dao, c.dbStartupMode)
.fromStartupMode(dao, c.startMode)
.unsafeToFuture()
_ = if (!isSuccess) throw new Exception("Db startup failed")
} yield dao

3
ledger-service/http-json/BUILD.bazel
View File

@ -200,6 +200,8 @@ daml_compile(
size = "medium",
srcs = glob(["src/test/scala/**/*.scala"]),
data = [
":src/test/resources/http-json-api.conf",
":src/test/resources/http-json-api-minimal.conf",
"//ledger/test-common/test-certificates",
],
plugins = [
@ -229,6 +231,7 @@ daml_compile(
scalacopts = hj_scalacopts,
deps = [
":http-json-{}".format(edition),
"//bazel_tools/runfiles:scala_runfiles",
"//daml-lf/data",
"//daml-lf/interface",
"//daml-lf/transaction",

4
ledger-service/http-json/src/bench/scala/com/daml/http/dbbackend/ContractDaoBenchmark.scala
View File

@ -126,7 +126,7 @@ trait OracleBenchmarkDbConn extends BenchmarkDbConnection with OracleAround {
password = user.pwd,
poolSize = ConnectionPool.PoolSize.Integration,
),
dbStartupMode = DbStartupMode.CreateOnly,
startMode = DbStartupMode.CreateOnly,
backendSpecificConf =
if (disableContractPayloadIndexing) Map(DisableContractPayloadIndexing -> "true")
else Map.empty,
@ -158,7 +158,7 @@ trait PostgresBenchmarkDbConn extends BenchmarkDbConnection with PostgresAround
password = database.password,
poolSize = ConnectionPool.PoolSize.Integration,
),
dbStartupMode = DbStartupMode.CreateOnly,
startMode = DbStartupMode.CreateOnly,
)
}

4
ledger-service/http-json/src/failure/scala/http/HttpTestFixture.scala
View File

@ -44,7 +44,7 @@ trait HttpFailureTestFixture extends ToxicSandboxFixture with PostgresAroundAll
password = "",
poolSize = ConnectionPool.PoolSize.Integration,
),
dbStartupMode = DbStartupMode.CreateOnly,
startMode = DbStartupMode.CreateOnly,
)
override def packageFiles =
@ -67,7 +67,7 @@ trait HttpFailureTestFixture extends ToxicSandboxFixture with PostgresAroundAll
proxiedPort,
Some(jdbcConfig_),
None,
wsConfig = Some(Config.DefaultWsConfig),
wsConfig = Some(WebsocketConfig()),
ledgerIdOverwrite = Some(ledgerId(None)),
)
}

2
ledger-service/http-json/src/it/scala/http/WebsocketServiceOffsetTickIntTest.scala
View File

@ -31,7 +31,7 @@ class WebsocketServiceOffsetTickIntTest
// make sure websocket heartbeats non-stop, DO NOT CHANGE `0.second`
override def wsConfig: Option[WebsocketConfig] =
Some(Config.DefaultWsConfig.copy(heartBeatPer = 0.second))
Some(WebsocketConfig(heartbeatPeriod = 0.second))
import WebsocketTestFixture._

2
ledger-service/http-json/src/itlib/scala/http/AbstractWebsocketServiceIntegrationTest.scala
View File

@ -43,7 +43,7 @@ abstract class AbstractWebsocketServiceIntegrationTest
override def useTls = UseTls.NoTls
override def wsConfig: Option[WebsocketConfig] = Some(Config.DefaultWsConfig)
override def wsConfig: Option[WebsocketConfig] = Some(WebsocketConfig())
private val baseQueryInput: Source[Message, NotUsed] =
Source.single(TextMessage.Strict("""{"templateIds": ["Account:Account"]}"""))

2
ledger-service/http-json/src/itpostgres/scala/http/HttpServicePostgresInt.scala
View File

@ -41,6 +41,6 @@ object HttpServicePostgresInt {
tablePrefix = "some_nice_prefix_",
poolSize = ConnectionPool.PoolSize.Integration,
),
dbStartupMode = DbStartupMode.CreateOnly,
startMode = DbStartupMode.CreateOnly,
)
}

4
ledger-service/http-json/src/main/scala/com/digitalasset/http/Main.scala
View File

@ -130,10 +130,10 @@ object Main {
IO.pure(some(ErrorCodes.StartupError))
case Right(true) =>
DbStartupOps
.fromStartupMode(dao, c.dbStartupMode)
.fromStartupMode(dao, c.startMode)
.map(success =>
if (success)
if (DbStartupOps.shouldStart(c.dbStartupMode)) none
if (DbStartupOps.shouldStart(c.startMode)) none
else some(ErrorCodes.Ok)
else some(ErrorCodes.StartupError)
)

4
ledger-service/http-json/src/main/scala/com/digitalasset/http/WebSocketService.scala
View File

@ -604,7 +604,7 @@ class WebSocketService(
import util.ErrorOps._
import com.daml.http.json.JsonProtocol._
private val config = wsConfig.getOrElse(Config.DefaultWsConfig)
private val config = wsConfig.getOrElse(WebsocketConfig())
private val numConns = new java.util.concurrent.atomic.AtomicInteger(0)
@ -884,7 +884,7 @@ class WebSocketService(
Flow[StepAndErrors[Pos, JsValue]]
.map(a => Step(a))
.keepAlive(config.heartBeatPer, () => TickTrigger)
.keepAlive(config.heartbeatPeriod, () => TickTrigger)
.scan(zero) {
case ((None, _), TickTrigger) =>
// skip all ticks we don't have the offset yet

10
ledger-service/http-json/src/test/resources/http-json-api-minimal.conf Normal file
View File

@ -0,0 +1,10 @@
{
server {
address = "127.0.0.1"
port = 7500
}
ledger-api {
address = "127.0.0.1"
port = 6400
}
}

94
ledger-service/http-json/src/test/resources/http-json-api.conf Normal file
View File

@ -0,0 +1,94 @@
{
server {
//IP address that HTTP JSON API service listens on. Defaults to 127.0.0.1.
address = "127.0.0.1"
//HTTP JSON API service port number. A port number of 0 will let the system pick an ephemeral port.
port = 7500
//Optional unique file name where to write the allocated HTTP port number. If process terminates gracefully, this file will be deleted automatically. Used to inform clients in CI about which port HTTP JSON API listens on. Defaults to none, that is, no file gets created.
port-file = "port-file"
}
ledger-api {
address = "127.0.0.1"
port = 6400
tls {
enabled = "true"
// the certificate to be used by the server
cert-chain-file = "cert-chain.crt"
// private key of the server
private-key-file = "pvt-key.pem"
// trust collection, which means that all client certificates will be verified using the trusted
// certificates in this store. if omitted, the JVM default trust store is used.
trust-collection-file = "root-ca.crt"
}
}
query-store {
base-config {
user = "postgres"
password = "password"
driver = "org.postgresql.Driver"
url = "jdbc:postgresql://localhost:5432/test?&ssl=true"
// prefix for table names to avoid collisions, empty by default
table-prefix = "foo"
// max pool size for the database connection pool
pool-size = 12
//specifies the min idle connections for database connection pool.
min-idle = 4
//specifies the idle timeout for the database connection pool.
idle-timeout = 12s
//specifies the connection timeout for database connection pool.
connection-timeout = 90s
}
// option setting how the schema should be handled.
// Valid options are start-only, create-only, create-if-needed-and-start and create-and-start
start-mode = "start-only"
// any backend db specific values.
backend-specific-conf {
foo = "bar"
}
}
// Optional interval to poll for package updates. Examples: 500ms, 5s, 10min, 1h, 1d. Defaults to 5 seconds
package-reload-interval = 5s
//Optional max inbound message size in bytes. Defaults to 4194304.
max-inbound-message-size = 4194304
//Optional max inbound message size in bytes used for uploading and downloading package updates. Defaults to the `max-inbound-message-size` setting.
package-max-inbound-message-size = 4194304
//Optional max cache size in entries for storing surrogate template id mappings. Defaults to None
max-template-id-cache-entries = 2000
//health check timeout
health-timeout-seconds = 5
//Optional websocket configuration parameters
websocket-config {
//Maximum websocket session duration
max-duration = 180m
//Server-side heartbeat interval duration
heartbeat-period = 1s
//akka stream throttle-mode one of either `shaping` or `enforcing`
mode = "enforcing"
}
metrics {
//Start a metrics reporter. Must be one of "console", "csv:///PATH", "graphite://HOST[:PORT][/METRIC_PREFIX]", or "prometheus://HOST[:PORT]".
reporter = "console"
//Set metric reporting interval , examples : 1s, 30s, 1m, 1h
reporting-interval = 30s
}
// DEV MODE ONLY (not recommended for production)
// Allow connections without a reverse proxy providing HTTPS.
allow-insecure-tokens = false
// Optional static content configuration string. Contains comma-separated key-value pairs, where:
// prefix -- URL prefix,
// directory -- local directory that will be mapped to the URL prefix.
// Example: "prefix=static,directory=./static-content"
static-content {
prefix = "static"
directory = "static-content-dir"
}
}

114
ledger-service/http-json/src/test/scala/com/digitalasset/http/CliSpec.scala
View File

@ -3,11 +3,20 @@
package com.daml.http
import akka.stream.ThrottleMode
import com.daml.bazeltools.BazelRunfiles.requiredResource
import org.scalatest.freespec.AnyFreeSpec
import org.scalatest.matchers.should.Matchers
import com.daml.dbutils
import com.daml.dbutils.{JdbcConfig => DbUtilsJdbcConfig}
import ch.qos.logback.classic.{Level => LogLevel}
import com.daml.cliopts.Logging.LogEncoder
import com.daml.http.dbbackend.{DbStartupMode, JdbcConfig}
import com.daml.ledger.api.tls.TlsConfiguration
import com.daml.metrics.MetricsReporter
import java.io.File
import java.nio.file.Paths
import scala.concurrent.duration._
object CliSpec {
@ -38,7 +47,7 @@ final class CliSpec extends AnyFreeSpec with Matchers {
idleTimeout,
tablePrefix,
),
dbStartupMode = DbStartupMode.StartOnly,
startMode = DbStartupMode.StartOnly,
)
val jdbcConfigString =
"driver=org.postgresql.Driver,url=jdbc:postgresql://localhost:5432/test?&ssl=true,user=postgres,password=password," +
@ -132,7 +141,7 @@ final class CliSpec extends AnyFreeSpec with Matchers {
val config =
configParser(Seq("--query-store-jdbc-config", jdbcConfigString) ++ sharedOptions)
.getOrElse(fail())
config.jdbcConfig shouldBe Some(jdbcConfig.copy(dbStartupMode = DbStartupMode.CreateOnly))
config.jdbcConfig shouldBe Some(jdbcConfig.copy(startMode = DbStartupMode.CreateOnly))
}
"should get the StartOnly startup mode from the string" in {
@ -140,7 +149,7 @@ final class CliSpec extends AnyFreeSpec with Matchers {
val config =
configParser(Seq("--query-store-jdbc-config", jdbcConfigString) ++ sharedOptions)
.getOrElse(fail())
config.jdbcConfig shouldBe Some(jdbcConfig.copy(dbStartupMode = DbStartupMode.StartOnly))
config.jdbcConfig shouldBe Some(jdbcConfig.copy(startMode = DbStartupMode.StartOnly))
}
"should get the CreateIfNeededAndStart startup mode from the string" in {
@ -149,7 +158,7 @@ final class CliSpec extends AnyFreeSpec with Matchers {
configParser(Seq("--query-store-jdbc-config", jdbcConfigString) ++ sharedOptions)
.getOrElse(fail())
config.jdbcConfig shouldBe Some(
jdbcConfig.copy(dbStartupMode = DbStartupMode.CreateIfNeededAndStart)
jdbcConfig.copy(startMode = DbStartupMode.CreateIfNeededAndStart)
)
}
@ -159,7 +168,7 @@ final class CliSpec extends AnyFreeSpec with Matchers {
configParser(Seq("--query-store-jdbc-config", jdbcConfigString) ++ sharedOptions)
.getOrElse(fail())
config.jdbcConfig shouldBe Some(
jdbcConfig.copy(dbStartupMode = DbStartupMode.CreateAndStart)
jdbcConfig.copy(startMode = DbStartupMode.CreateAndStart)
)
}
@ -169,7 +178,7 @@ final class CliSpec extends AnyFreeSpec with Matchers {
configParser(Seq("--query-store-jdbc-config", jdbcConfigString) ++ sharedOptions)
.getOrElse(fail())
config.jdbcConfig shouldBe Some(
jdbcConfig.copy(dbStartupMode = DbStartupMode.StartOnly)
jdbcConfig.copy(startMode = DbStartupMode.StartOnly)
)
}
@ -179,7 +188,7 @@ final class CliSpec extends AnyFreeSpec with Matchers {
configParser(Seq("--query-store-jdbc-config", jdbcConfigString) ++ sharedOptions)
.getOrElse(fail())
config.jdbcConfig shouldBe Some(
jdbcConfig.copy(dbStartupMode = DbStartupMode.CreateOnly)
jdbcConfig.copy(startMode = DbStartupMode.CreateOnly)
)
}
}
@ -216,4 +225,95 @@ final class CliSpec extends AnyFreeSpec with Matchers {
}
}
"TypeConfig app Conf" - {
val confFile = "ledger-service/http-json/src/test/resources/http-json-api-minimal.conf"
"should fail on missing ledgerHost and ledgerPort if no config file supplied" in {
configParser(sharedOptions.drop(4)) should ===(None)
}
"should fail on missing httpPort and no config file is supplied" in {
configParser(sharedOptions.take(4)) should ===(None)
}
"should successfully load a minimal config file" in {
val cfg = configParser(Seq("--config", requiredResource(confFile).getAbsolutePath))
cfg shouldBe Some(
Config.Empty.copy(httpPort = 7500, ledgerHost = "127.0.0.1", ledgerPort = 6400)
)
}
"should load a minimal config file along with logging opts from cli" in {
val cfg = configParser(
Seq(
"--config",
requiredResource(confFile).getAbsolutePath,
"--log-level",
"DEBUG",
"--log-encoder",
"json",
)
)
cfg shouldBe Some(
Config(
httpPort = 7500,
ledgerHost = "127.0.0.1",
ledgerPort = 6400,
logLevel = Some(LogLevel.DEBUG),
logEncoder = LogEncoder.Json,
)
)
}
"should fail when config file and cli args both are supplied" in {
configParser(
Seq("--config", requiredResource(confFile).getAbsolutePath) ++ sharedOptions
) should ===(None)
}
"should successfully load a complete config file" in {
val baseConfig = DbUtilsJdbcConfig(
url = "jdbc:postgresql://localhost:5432/test?&ssl=true",
driver = "org.postgresql.Driver",
user = "postgres",
password = "password",
poolSize = 12,
idleTimeout = 12.seconds,
connectionTimeout = 90.seconds,
tablePrefix = "foo",
minIdle = 4,
)
val expectedWsConfig = WebsocketConfig(
maxDuration = 180.minutes,
throttleElem = 20,
throttlePer = 1.second,
maxBurst = 20,
mode = ThrottleMode.Enforcing,
heartbeatPeriod = 1.second,
)
val expectedConfig = Config(
ledgerHost = "127.0.0.1",
ledgerPort = 6400,
address = "127.0.0.1",
httpPort = 7500,
portFile = Some(Paths.get("port-file")),
tlsConfig = TlsConfiguration(
enabled = true,
Some(new File("cert-chain.crt")),
Some(new File("pvt-key.pem")),
Some(new File("root-ca.crt")),
),
jdbcConfig = Some(JdbcConfig(baseConfig, DbStartupMode.StartOnly, Map("foo" -> "bar"))),
staticContentConfig = Some(StaticContentConfig("static", new File("static-content-dir"))),
metricsReporter = Some(MetricsReporter.Console),
metricsReportingInterval = 30.seconds,
wsConfig = Some(expectedWsConfig),
surrogateTpIdCacheMaxEntries = Some(2000L),
packageMaxInboundMessageSize = Some(StartSettings.DefaultMaxInboundMessageSize),
)
val confFile = "ledger-service/http-json/src/test/resources/http-json-api.conf"
val cfg = configParser(Seq("--config", requiredResource(confFile).getAbsolutePath))
cfg shouldBe Some(expectedConfig)
}
}
}

4
ledger-service/pureconfig-utils/BUILD.bazel
View File

@ -36,9 +36,11 @@ da_scala_library(
deps = [
"//ledger-service/jwt",
"//ledger/ledger-api-common",
"//ledger/metrics",
"//libs-scala/db-utils",
"@maven//:com_auth0_java_jwt",
"@maven//:com_typesafe_config",
"@maven//:io_netty_netty_handler",
],
)
@ -58,6 +60,8 @@ da_scala_test(
scalacopts = lf_scalacopts,
deps = [
":pureconfig-utils",
"//ledger-service/jwt",
"//ledger/metrics",
"@maven//:org_scalatest_scalatest_compatible",
],
)

136
ledger-service/pureconfig-utils/src/main/scala/com/daml/pureconfigutils/SharedConfigReaders.scala
View File

@ -6,63 +6,129 @@ package com.daml.pureconfigutils
import akka.http.scaladsl.model.Uri
import com.auth0.jwt.algorithms.Algorithm
import com.daml.dbutils.JdbcConfig
import com.daml.jwt.{ECDSAVerifier, HMAC256Verifier, JwksVerifier, JwtVerifierBase, RSA256Verifier}
import com.daml.jwt.{
ECDSAVerifier,
HMAC256Verifier,
JwksVerifier,
JwtVerifier,
JwtVerifierBase,
RSA256Verifier,
}
import com.daml.ledger.api.tls.TlsConfiguration
import com.daml.metrics.MetricsReporter
import com.daml.platform.services.time.TimeProviderType
import pureconfig.{ConfigReader, ConvertHelpers}
import pureconfig.error.{CannotConvert, ConvertFailure, FailureReason}
import pureconfig.{ConfigObjectCursor, ConfigReader, ConvertHelpers}
import pureconfig.generic.semiauto.deriveReader
import scalaz.\/
import scalaz.syntax.std.option._
import java.nio.file.Path
import java.io.File
import scala.concurrent.duration.FiniteDuration
final case class HttpServerConfig(address: String, port: Int, portFile: Option[Path] = None)
final case class LedgerApiConfig(address: String, port: Int)
final case class MetricsConfig(reporter: String, reportingInterval: FiniteDuration)
final case class LedgerTlsConfig(
enabled: Boolean = false,
certChainFile: Option[File] = None,
privateKeyFile: Option[File] = None,
trustCollectionFile: Option[File] = None,
) {
def tlsConfiguration: TlsConfiguration =
TlsConfiguration(enabled, certChainFile, privateKeyFile, trustCollectionFile)
}
final case class LedgerApiConfig(
address: String,
port: Int,
tls: LedgerTlsConfig = LedgerTlsConfig(),
)
final case class MetricsConfig(reporter: MetricsReporter, reportingInterval: FiniteDuration)
object TokenVerifierConfig {
private val knownTokenVerifiers: Map[String, String => JwtVerifier.Error \/ JwtVerifierBase] =
Map(
"rs256-crt" -> RSA256Verifier.fromCrtFile,
"es256-crt" -> (ECDSAVerifier
.fromCrtFile(_, Algorithm.ECDSA256(_, null))),
"es512-crt" -> (ECDSAVerifier
.fromCrtFile(_, Algorithm.ECDSA512(_, null))),
"rs256-jwks" -> (valueStr =>
\/.attempt(JwksVerifier(valueStr))(e => JwtVerifier.Error(Symbol("RS256"), e.getMessage))
),
)
private val unsafeTokenVerifier: (String, String => JwtVerifier.Error \/ JwtVerifierBase) =
"hs256-unsafe" -> (HMAC256Verifier(_))
def extractByType(
typeStr: String,
valueStr: String,
objectCursor: ConfigObjectCursor,
): ConfigReader.Result[JwtVerifierBase] = {
def convertFailure(msg: String) = {
ConfigReader.Result.fail(
ConvertFailure(
CannotConvert(typeStr, "JwtVerifier", msg),
objectCursor,
)
)
}
(knownTokenVerifiers + unsafeTokenVerifier)
.get(typeStr)
.cata(
{ conv =>
conv(valueStr).fold(
err => convertFailure(s"Failed to create $typeStr verifier: $err"),
(Right(_)),
)
},
convertFailure(s"value not one of ${knownTokenVerifiers.keys.mkString(", ")}"),
)
}
}
object SharedConfigReaders {
implicit val tokenVerifierReader: ConfigReader[JwtVerifierBase] =
ConfigReader.forProduct2[JwtVerifierBase, String, String]("type", "uri") {
case (t: String, p: String) =>
// hs256-unsafe, rs256-crt, es256-crt, es512-crt, rs256-jwks
t match {
case "hs256-unsafe" =>
HMAC256Verifier(p)
.valueOr(err => sys.error(s"Failed to create HMAC256 verifier: $err"))
case "rs256-crt" =>
RSA256Verifier
.fromCrtFile(p)
.valueOr(err => sys.error(s"Failed to create RSA256 verifier: $err"))
case "es256-crt" =>
ECDSAVerifier
.fromCrtFile(p, Algorithm.ECDSA256(_, null))
.valueOr(err => sys.error(s"Failed to create ECDSA256 verifier: $err"))
case "es512-crt" =>
ECDSAVerifier
.fromCrtFile(p, Algorithm.ECDSA512(_, null))
.valueOr(err => sys.error(s"Failed to create ECDSA512 verifier: $err"))
case "rs256-jwks" =>
JwksVerifier(p)
}
def catchConvertError[A, B](f: String => Either[String, B])(implicit
B: reflect.ClassTag[B]
): String => Either[FailureReason, B] =
s => f(s).left.map(CannotConvert(s, B.toString, _))
implicit val tokenVerifierCfgRead: ConfigReader[JwtVerifierBase] =
ConfigReader.fromCursor { cur =>
for {
objCur <- cur.asObjectCursor
typeCur <- objCur.atKey("type")
typeStr <- typeCur.asString
valueCur <- objCur.atKey("uri")
valueStr <- valueCur.asString
ident <- TokenVerifierConfig.extractByType(typeStr, valueStr, objCur)
} yield ident
}
implicit val uriCfgReader: ConfigReader[Uri] =
ConfigReader.fromString[Uri](ConvertHelpers.catchReadError(s => Uri(s)))
implicit val timeProviderTypeCfgReader: ConfigReader[TimeProviderType] =
ConfigReader.fromString[TimeProviderType](ConvertHelpers.catchReadError { s =>
implicit val timeProviderTypeCfgReader: ConfigReader[TimeProviderType] = {
ConfigReader.fromString[TimeProviderType](catchConvertError { s =>
s.toLowerCase() match {
case "static" => TimeProviderType.Static
case "wall-clock" => TimeProviderType.WallClock
case s =>
throw new IllegalArgumentException(
s"Value '$s' for time-provider-type is not one of 'static' or 'wall-clock'"
)
case "static" => Right(TimeProviderType.Static)
case "wall-clock" => Right(TimeProviderType.WallClock)
case _ => Left("not one of 'static' or 'wall-clock'")
}
})
}
implicit val metricReporterReader: ConfigReader[MetricsReporter] = {
ConfigReader.fromString[MetricsReporter](ConvertHelpers.catchReadError { s =>
MetricsReporter.parseMetricsReporter(s.toLowerCase())
})
}
implicit val jdbcCfgReader: ConfigReader[JdbcConfig] = deriveReader[JdbcConfig]
implicit val httpServerCfgReader: ConfigReader[HttpServerConfig] =
deriveReader[HttpServerConfig]
implicit val ledgerTlsCfgReader: ConfigReader[LedgerTlsConfig] =
deriveReader[LedgerTlsConfig]
implicit val ledgerApiConfReader: ConfigReader[LedgerApiConfig] =
deriveReader[LedgerApiConfig]
implicit val metricsConfigReader: ConfigReader[MetricsConfig] = deriveReader[MetricsConfig]

35
ledger-service/pureconfig-utils/src/test/scala/com/daml/pureconfigutils/SharedConfigReadersTest.scala
View File

@ -3,11 +3,18 @@
package com.daml.pureconfigutils
import com.daml.jwt.JwtVerifierBase
import com.daml.metrics.MetricsReporter
import org.scalatest.Inside.inside
import org.scalatest.matchers.should.Matchers
import org.scalatest.wordspec.AsyncWordSpec
import pureconfig.error.{ConfigReaderFailures, ConvertFailure}
import pureconfig.generic.semiauto.deriveReader
import pureconfig.{ConfigReader, ConfigSource}
import java.nio.file.Paths
import scala.concurrent.duration._
class SharedConfigReadersTest extends AsyncWordSpec with Matchers {
import SharedConfigReaders._
@ -19,6 +26,9 @@ class SharedConfigReadersTest extends AsyncWordSpec with Matchers {
implicit val serviceConfigReader: ConfigReader[SampleServiceConfig] =
deriveReader[SampleServiceConfig]
case class DummyConfig(tokenVerifier: JwtVerifierBase)
implicit val dummyCfgReader: ConfigReader[DummyConfig] = deriveReader[DummyConfig]
"should be able to parse a sample config with shared config objects" in {
val conf = """
|{
@ -38,9 +48,28 @@ class SharedConfigReadersTest extends AsyncWordSpec with Matchers {
|}
|""".stripMargin
ConfigSource.string(conf).load[SampleServiceConfig] match {
case Right(_) => succeed
case Left(ex) => fail(s"Failed to successfully parse service conf: ${ex.head.description}")
val expectedConf = SampleServiceConfig(
HttpServerConfig("127.0.0.1", 8890, Some(Paths.get("port-file"))),
LedgerApiConfig("127.0.0.1", 8098),
MetricsConfig(MetricsReporter.Console, 10.seconds),
)
ConfigSource.string(conf).load[SampleServiceConfig] shouldBe Right(expectedConf)
}
"should fail on loading unknown tokenVerifiers" in {
val conf = """
|{
| token-verifier {
| type = "foo"
| uri = "bar"
| }
|}
|""".stripMargin
val cfg = ConfigSource.string(conf).load[DummyConfig]
inside(cfg) { case Left(ConfigReaderFailures(ex)) =>
ex shouldBe a[ConvertFailure]
}
}

8
ledger/metrics/src/main/scala/com/daml/metrics/MetricsReporter.scala
View File

@ -58,7 +58,7 @@ object MetricsReporter {
val defaultPort: Int = 55001
}
implicit val metricsReporterRead: Read[MetricsReporter] = {
def parseMetricsReporter(s: String): MetricsReporter = {
def getAddress(uri: URI, defaultPort: Int) = {
if (uri.getHost == null) {
throw invalidRead
@ -66,7 +66,7 @@ object MetricsReporter {
val port = if (uri.getPort > 0) uri.getPort else defaultPort
new InetSocketAddress(uri.getHost, port)
}
Read.reads {
s match {
case "console" =>
Console
case value if value.startsWith("csv://") =>
@ -89,6 +89,10 @@ object MetricsReporter {
}
}
implicit val metricsReporterRead: Read[MetricsReporter] = {
Read.reads(parseMetricsReporter)
}
val cliHint: String =
"""Must be one of "console", "csv:///PATH", "graphite://HOST[:PORT][/METRIC_PREFIX]", or "prometheus://HOST[:PORT]"."""

5
triggers/service/auth/src/test/scala/com/daml/auth/middleware/oauth2/CliSpec.scala
View File

@ -43,10 +43,7 @@ class CliSpec extends AsyncWordSpec with Matchers {
inside(cfg) { case Some(Right(c)) =>
c.copy(tokenVerifier = null) shouldBe minimalCfg
// token verifier needs to be set.
c.tokenVerifier match {
case _: JwksVerifier => succeed
case _ => fail("expected JwksVerifier based on supplied config")
}
c.tokenVerifier shouldBe a[JwksVerifier]
}
}