From deb8a8b740401c04345ce22ed8c8aff62087df70 Mon Sep 17 00:00:00 2001 From: tudor-da Date: Wed, 8 Sep 2021 09:58:55 +0200 Subject: [PATCH] [Divulgence pruning] Daml SDK docs update [DPP-536] (#10764) * [Divulgence pruning] Minor update to participant_pruning_service.proto divulgence docs CHANGELOG_BEGIN [Ledger API Specification] Participant pruning of all divulged contracts is fully implemented: Participant operators can choose to prune all immediately and retroactively divulged contracts, by setting the newly-added prune_all_divulged_contracts flag in the ParticipantPruningService/Prune request. CHANGELOG_END * Enrich `Daml Participant Pruning` in the `Operating Daml` docs section * Apply suggestions from code review Co-authored-by: mziolekda Co-authored-by: mziolekda --- docs/source/ops/index.rst | 15 ++++++++++++--- .../v1/admin/participant_pruning_service.proto | 7 ++++--- 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/docs/source/ops/index.rst b/docs/source/ops/index.rst index 5a80860ba1..17967c903a 100644 --- a/docs/source/ops/index.rst +++ b/docs/source/ops/index.rst @@ -29,11 +29,12 @@ Still, Daml applications may be affected in the following ways: - Pruning may degrade the behavior of or abort in-progress requests if the pruning offset is too recent. In particular, the system might misbehave if command completions are pruned before the command trackers are able to process the completions. - Command deduplication and command tracker retention should always configured in such a way, that the associated windows don't overlap with the pruning window, so that their operation is unaffected by pruning. - Pruning may affect the behavior of Ledger API calls that allow to read data from the ledger: see the next sub-section for more information about API impacts. +- Pruning of all divulged contracts (see :ref:`Prune Request `) does not preserve application visibility over contracts divulged up to the pruning offset, hence applications making use of pruned divulged contracts might start experiencing failed command submissions: see the section below for determining a suitable pruning offset. .. warning:: Participants may know of contracts for which they don't know the current activeness status. This happens through :ref:`divulgence ` where a party learns of the existence of a contract without being guaranteed to ever see its archival. Such contracts are pruned by the feature described on this page as not doing so could easily lead to an ever growing participant state. -During command submission, parties can currently resolve the contract ids of divulged contracts. This is mutually incompatible with the pruning behaviour described above. Daml code that may end up running on pruned participants should therefore never rely on that behaviour. +During command submission, parties can fetch divulged contracts. This is incompatible with the pruning behaviour described above which allows participant operators to reclaim storage space by pruning divulged contracts. Daml code running on pruned participants should therefore never rely on existence of divulged contracts prior to or at the pruning offset. Instead, such applications MUST ensure re-divulgence of the used contracts. How the Daml Ledger API is affected ----------------------------------- @@ -49,7 +50,6 @@ Please refer to the :doc:`protobuf documentation of the API ` and the :ref:`Active Contract Service ` provide offsets of the ledger end, as well as of transactions, and of Active Contracts snapshots respectively; such offsets can be passed unchanged to `prune` calls. +The :ref:`Transaction Service ` and the :ref:`Active Contract Service ` provide offsets of the ledger end of the Transactions, and of Active Contracts snapshots respectively. Such offsets can be passed unchanged to `prune` calls, as long as they are lexicographically lower than the current ledger end. + +When pruning all divulged contracts, the participant operator can choose the pruning offset as follows: + +- Just before the ledger end, if no application hosted on the participant makes use of divulgence OR + +- An offset old enough (e.g. older than an arbitrary multi-day grace period) that it ensures that pruning does not affect any recently-divulged contract needed by the applications hosted on the participant. Scheduled jobs, applications and/or operator tools can be built on top of the Daml Ledger API to implement pruning automatically, for example at regular intervals, or on-demand, for example according to a user-initiated process. For instance, pruning at regular intervals could be performed by a cron job that: 1. If a pruning interval has been saved to a well-known location: + a. Backs up the Daml Participant Index DB. + b. Performs pruning. + c. (If using PostgreSQL) Performs a `VACUUM FULL` command on the Daml Participant Index DB. 2. Queries the current ledger end and saves its offset. diff --git a/ledger-api/grpc-definitions/com/daml/ledger/api/v1/admin/participant_pruning_service.proto b/ledger-api/grpc-definitions/com/daml/ledger/api/v1/admin/participant_pruning_service.proto index 31fe27438f..dd68c183f8 100644 --- a/ledger-api/grpc-definitions/com/daml/ledger/api/v1/admin/participant_pruning_service.proto +++ b/ledger-api/grpc-definitions/com/daml/ledger/api/v1/admin/participant_pruning_service.proto @@ -58,9 +58,10 @@ message PruneRequest { // such that they do not rely on divulged contracts; i.e., no warnings from // using divulged contracts as inputs to transactions are emitted. // - // Participant node operators SHOULD set this flag to avoid leaking - // storage due to accumulating unarchived divulged contracts PROVIDED no - // application using this participant node relies on divulgence. + // Participant node operators SHOULD set the `prune_all_divulged_contracts` flag to avoid leaking + // storage due to accumulating unarchived divulged contracts PROVIDED that: + // 1. no application using this participant node relies on divulgence OR + // 2. divulged contracts on which applications rely have been re-divulged after the `prune_up_to` offset. bool prune_all_divulged_contracts = 3; }