daml/ledger
Samir Talwar 5cea224a36
kvutils: Flush export files after each write. (#7527)
This will allow consumers to stream writes as they're written.

CHANGELOG_BEGIN
CHANGELOG_END
2020-09-30 10:48:13 +00:00
..
caching caching: Wait for the cache to evict some values in tests. (#6438) 2020-06-19 19:30:58 +02:00
daml-on-sql DAML for PostgreSQL: prevent dev version of DAML-LF (#7176) 2020-08-20 12:15:38 +02:00
ledger-api-akka Organize maven coordinates (#5272) 2020-04-01 11:41:18 +02:00
ledger-api-auth concurrent: Tag DirectExecutionContext. (#7517) 2020-09-29 17:23:57 +00:00
ledger-api-auth-client Set the Bearer prefix in bindings. (#5484) 2020-04-08 13:07:28 +02:00
ledger-api-client concurrent: Tag DirectExecutionContext. (#7517) 2020-09-29 17:23:57 +00:00
ledger-api-common concurrent: Tag DirectExecutionContext. (#7517) 2020-09-29 17:23:57 +00:00
ledger-api-domain participant-integration-api: Drop unused participant_id columns. (#6874) 2020-08-17 17:29:24 +02:00
ledger-api-health sandbox + kvutils: Add the "read" component back to the health checks. [KVL-475] [KVL-483] (#7486) 2020-09-25 11:06:46 +00:00
ledger-api-test-tool ledger-api-test-tool: Retry the participant connection for a minute. (#7475) 2020-09-24 08:29:29 +00:00
ledger-api-test-tool-on-canton Canton conformance test debug logging and reduced test parallelism (#7322) 2020-09-04 14:25:30 +02:00
ledger-on-memory concurrent: Tag DirectExecutionContext. (#7517) 2020-09-29 17:23:57 +00:00
ledger-on-sql kvutils: Sort the output state before writing. [KVL-558] (#7515) 2020-09-29 17:07:11 +00:00
metrics concurrent: Tag DirectExecutionContext. (#7517) 2020-09-29 17:23:57 +00:00
participant-integration-api more error logging for random transaction stream testing (#7482) 2020-09-30 06:18:16 +00:00
participant-state kvutils: Flush export files after each write. (#7527) 2020-09-30 10:48:13 +00:00
participant-state-index participant-integration-api: Factor out similar behavior in the admin service implementations. (#7171) 2020-08-19 10:24:48 +00:00
participant-state-metrics Make interpretation cost available for LedgerWriter/WriteService (#6515) 2020-06-30 16:45:12 +00:00
recovering-indexer-integration-tests Pre-execution support for DAML on memory KV (#6793) 2020-08-05 08:00:27 +00:00
sandbox concurrent: Tag DirectExecutionContext. (#7517) 2020-09-29 17:23:57 +00:00
sandbox-classic concurrent: Tag DirectExecutionContext. (#7517) 2020-09-29 17:23:57 +00:00
sandbox-common concurrent: Tag DirectExecutionContext. (#7517) 2020-09-29 17:23:57 +00:00
sandbox-perf concurrent: Tag DirectExecutionContext. (#7517) 2020-09-29 17:23:57 +00:00
scripts replace DAML Authors with DA in copyright headers (#5228) 2020-03-27 01:26:10 +01:00
test-common concurrent: Tag DirectExecutionContext. (#7517) 2020-09-29 17:23:57 +00:00
README.md Use com.daml as root package (#5343) 2020-04-05 19:49:57 +02:00

ledger

Home of our reference ledger implementation (Sandbox) and various ledger related libraries.

Logging

Logging Configuration

The Sandbox and Ledger API Server use Logback for logging configuration.

Log Files

The Sandbox logs at INFO level to standard out and to the file sandbox.log in the current working directory.

Log levels

As most Java libraries and frameworks, the Sandbox and Ledger API Server use INFO as the default logging level. This level is for minimal and important information (usually only startup and normal shutdown events). INFO level logging should not produce increasing volume of logging during normal operation.

WARN level should be used for transition between healthy/unhealthy state, or in other close to error scenarios.

DEBUG level should be turned on only when investigating issues in the system, and usually that means we want the trail loggers. Normal loggers at DEBUG level can be useful sometimes (e.g. DAML interpretation).

Metrics

Sandbox and Ledger API Server provide a couple of useful metrics:

Sandbox and Ledger API Server

The Ledger API Server exposes basic metrics for all gRPC services and some additional ones.

Metric NameDescription
LedgerApi.com.daml.ledger.api.v1.$SERVICE.$METHOD
A meter that tracks the number of calls to the respective service and method.
CommandSubmission.failedCommandInterpretations
A meter that tracks the failed command interpretations.
CommandSubmission.submittedTransactions
A timer that tracks the commands submitted to the backing ledger.

Indexer

Metric NameDescription
JdbcIndexer.processedStateUpdates
A timer that tracks duration of state update processing.
JdbcIndexer.lastReceivedRecordTime
A gauge that returns the last received record time in milliseconds since EPOCH.
JdbcIndexer.lastReceivedOffset
A gauge that returns that last received offset from the ledger.
JdbcIndexer.currentRecordTimeLag
A gauge that returns the difference between the Indexer's wallclock time and the last received record time in milliseconds.

Metrics Reporting

The Sandbox automatically makes all metrics available via JMX under the JMX domain com.daml.platform.sandbox.

When building an Indexer or Ledger API Server the implementer/ledger integrator is responsible to set up a MetricRegistry and a suitable metric reporting strategy that fits their needs.

Health Checks

Ledger API Server health checks

The Ledger API Server exposes health checks over the gRPC Health Checking Protocol. You can check the health of the overall server by making a gRPC request to grpc.health.v1.Health.Check.

You can also perform a streaming health check by making a request to grpc.health.v1.Health.Watch. The server will immediately send the current health of the Ledger API Server, and then send a new message whenever the health changes.

The ledger may optionally expose health checks for underlying services and connections; the names of the services are ledger-dependent. For example, the Sandbox exposes two service health checks:

  • the "index" service tests the health of the connection to the index database
  • the "write" service tests the health of the connection to the ledger database

To use these, make a request with the service field set to the name of the service. An unknown service name will result in a gRPC NOT_FOUND error.

Indexer health checks

The Indexer does not currently run a gRPC server, and so does not expose any health checks on its own.

In the situation where it is run in the same process as the Ledger API Server, the authors of the binary are encouraged to add specific health checks for the Indexer. This is the case in the Sandbox and Reference implementations.

Checking the server health in production

We encourage you to use the grpc-health-probe tool to periodically check the health of your Ledger API Server in production. On the command line, you can run it as follows (changing the address to match your ledger):

$ grpc-health-probe -addr=localhost:6865
status: SERVING

An example of how to naively configure Kubernetes to run the Sandbox, with accompanying health checks, can be found in sandbox/kubernetes.yaml.

More details can be found on the Kubernetes blog, in the post titled Health checking gRPC servers on Kubernetes.

gRPC and back-pressure

RPC

Standard RPC requests should return with RESOURCE_EXHAUSTED status code to signal back-pressure. Envoy can be configured to retry on these errors. We have to be careful not to have any persistent changes when returning with such an error as the same original request can be retried on another service instance.

Streaming

gRPC's streaming protocol has built-in flow-control, but it's not fully active by default. What it does it controls the flow between the TCP/HTTP layer and the library so it builds on top of TCP's own flow control. The inbound flow control is active by default, but the outbound does not signal back-pressure out of the box.

AutoInboundFlowControl: The default behaviour for handling incoming items in a stream is to automatically signal demand after every onNext call. This is the correct thing to do if the handler logic is CPU bound and does not depend on other reactive downstream services. By default it's active on all inbound streams. One can disable this and signal demand by manually calling request to follow demands of downstream services. Disabling this feature is possible by calling disableAutoInboundFlowControl on CallStreamObserver.

ServerCallStreamObserver: casting an outbound StreamObserver manually to ServerCallStreamObserver gives us access to isReady and onReadyHandler. With these methods we can check if there is available capacity in the channel i.e. we are safe to push into it. This can be used to signal demand to our upstream flow. Note that gRPC buffers 32Kb data per channel and isReady will return false only when this buffer gets full.