Commit Graph

96 Commits

Author SHA1 Message Date
Samir Talwar
bd12170af2
participant-state: Remove the ParticipantId, PackageId, and Party aliases. [KVL-1002] (#10308)
* participant-state: Remove the `ParticipantId` alias.

This alias adds nothing. By using `Ref.ParticipantId` directly, many
packages can remove their dependency on the _participant-state_ package.

CHANGELOG_BEGIN
CHANGELOG_END

* participant-state: Remove the `PackageId` and `Party` aliases.

They don't add anything. Let's just use `Ref`.

* kvutils: Restore missing compat imports.
2021-07-19 12:31:25 +00:00
Samir Talwar
1ee53c0736
participant-state-index: Use Ref directly. (#10299)
This avoids the dependency on _participant-state_.

CHANGELOG_BEGIN
CHANGELOG_END
2021-07-15 17:21:03 +00:00
Samir Talwar
cb29f34d4b
ledger-offset: Move Offset to a new package. [KVL-1002] (#10296)
* ledger-offset: Move `Offset` to a new package.

CHANGELOG_BEGIN
- [Integration Kit] The ``Offset`` type has been moved to a new Maven
  package, ``ledger-offset``, from the ``participant-state`` package.
  The Java package has been renamed to ``com.daml.ledger.offset``. If
  you are using this type, you will need to update your dependencies and
  imports.
CHANGELOG_END

* Avoid rewrapping offsets for no reason.

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>

* participant-integration-api: Sort some imports.

* participant-integration-api: Fix dependencies for the Oracle tests.

I didn't add `ledger-offset`.

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
2021-07-15 17:53:03 +02:00
Samir Talwar
a9a0b70b85
ledger-configuration: Extract the configuration from participant-state. [KVL-1002] (#10279)
* ledger-configuration: Extract configuration from participant-state.

The configuration is often used without the state, and doesn't need to
be versioned in the same way.

CHANGELOG_BEGIN
- [Integration Kit] The ledger configuration classes, ``Configuration``,
  ``LedgerInitialConditions``, and ``TimeModel``, have been moved from
  *participant-state* to a separate package named
  *ledger-configuration*, in the Java package
  ``com.daml.ledger.configuration``. You will need to update your
  dependencies and imports.
CHANGELOG_END

* participant-state: Remove the `LedgerId` aliases.

* ledger-configuration: Rename `TimeModel` to `LedgerTimeModel`.

This avoids confusion with the protobuf-generated `TimeModel` classes.

CHANGELOG_BEGIN
- [Integration Kit] ``TimeModel`` has been renamed to
  ``LedgerTimeModel``. If you are using the ledger configuration classes
  directly, you may need to update your code.
CHANGELOG_END

* ledger-configuration: Remove colons in LedgerInitialConditions' docs.

* kvutils: Restore a missing compat import.

* participant-integration-api: Add ledger-configuration to Oracle tests.

* sandbox-common: Fix `--max-ledger-time-skew` docs.

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
2021-07-15 13:03:19 +00:00
Miklos
b1ca310866
ledger: Damlification of Scala files (#9667)
* Damlification of Scala files (primarily comments and strings).

* Corrected a typo.
CHANGELOG_BEGIN
CHANGELOG_END

* Fixed build.

* Fixed test case for acronyms.

* Reformatted.
2021-05-20 10:21:04 +00:00
Moritz Kiefer
22ce940954
Upgrade scalafmt and enable trailing commas (#8437)
This PR updates scalafmt and enables trailingCommas =
multiple. Unfortunately, scalafmt broke the version field which means
we cannot fully preserve the rest of the config. I’ve made some
attempts to stay reasonably close to the original config but couldn’t
find an exact equivalent in a lot of cases. I don’t feel strongly
about any of the settings so happy to change them to something else.

As announced, this will be merged on Saturday to avoid too many conflicts.

changelog_begin
changelog_end
2021-01-09 11:37:37 +01:00
Moritz Kiefer
424faa923a
Port damlc dependencies to Scala 2.13 (#8423)
* Port damlc dependencies to Scala 2.13

I got a bit fed up by the fact that going directory by directory
didn’t really work since there are two many interdependencies in
tests (e.g., client tests depend on sandbox, sandbox tests depend on
clients, engine tests depend on DARs which depend on damlc, …).

So before attempting to continue with the per-directory process, this
is a bruteforce approach to break a lot of those cycles by porting all
dependencies of damlc which includes client bindings (for DAML Script)
and Sandbox Classic (also for DAML Script).

If this is too annoying to review let me know and I’ll try to split it
up into a few chunks.

changelog_begin
changelog_end

* Update daml-lf/data/src/main/2.13/com/daml/lf/data/LawlessTraversals.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* fixup lawlesstraversal

changelog_begin
changelog_end

* less iterator more view

changelog_begin
changelog_end

* document safety of unsafeWrapArray

changelog_begin
changelog_end

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2021-01-08 07:22:38 +01:00
Gary Verhaegen
a925f0174c
update copyright notices for 2021 (#8257)
* update copyright notices for 2021

To be merged on 2021-01-01.

CHANGELOG_BEGIN
CHANGELOG_END

* patch-bazel-windows & da-ghc-lib
2021-01-01 19:49:51 +01:00
Kamil Bożek
5db4ca687b
Multi-party command submission [KVL-699] (#8152)
Changes:
- replaced submitter: Party with actAs: Set[Party] and readAs: Set[Party] in com.daml.ledger.api.domain.Commands
- adapted contract and key lookups for multi-party submissions

* Changed single Commands to accept multiple submitters

* Removed submitter field from Commands

* Replaced submitters with actAs and readAs in Commands

* Moved actAs and readAs to com.daml.ledger.api.Commands

* Contract lookup tests

* Key lookup tests

CHANGELOG_BEGIN
CHANGELOG_END

* Formatted code changes

* Remove unused method

* Formatting fixes

* Simplified logical condition for active contracts visibility

* Removed unused temporary method in Commands

* Unified contract key lookup query for different type of db types

* Simplified ContractReader class structure

* Improved dao unit tests

* Unit tests for lookups as a divulgee

* Reduced code duplication in unit tests

* Minor improvement

* Formatted code

* Fixed a ContractsReader's query

* Improved unit tests

* Explanation for engine's command authorizers and contract readers

* Minor fixes

* Adapted SpannedIndexService for the multi-party submission
2020-12-11 10:04:15 +01:00
Robert Autenrieth
35d52e956e
Prepare index database for multi-party submissions [KVL-700] (#8035)
* Support multiple submitters in LedgerEntry

CHANGELOG_BEGIN
CHANGELOG_END

* Adapt the database schema

* Fix party formatting

* Add some tests

* Change logic for when requesters see commands

* Disable test that can't work

* Handle empty sets in array intersection

* Don't query completions as an empty set

... this should never happen in practice

* Add test for filtering command IDs

* Enable multiple parties in command deduplication (#8092)

* Enable multiple parties in command deduplication

CHANGELOG_BEGIN
CHANGELOG_END

* Update ledger/participant-integration-api/src/main/scala/platform/store/dao/LedgerDao.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update ledger/participant-integration-api/src/main/scala/platform/store/dao/LedgerDao.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Apply review suggestions

* Remove useless function

* Fix EventsTablePostgresql

* Apply review comments

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-12-04 21:47:52 +01:00
Samir Talwar
052f69cde9
ledger-api: Use proto_jars, and publish Protobuf sources separately from the Scala classes. [KVL-714] (#8091)
* ledger-api: Use `proto_jars`.

CHANGELOG_BEGIN
- [Ledger API] The Scala JARs containing the gRPC definitions no longer
  contain the *.proto files used to generate the ScalaPB-based classes.
CHANGELOG_END

* Create a source JAR for *.proto files in `proto_jars`.

* ledger-api: Publish the protobuf sources as "ledger-api-proto".

CHANGELOG_BEGIN
- [Ledger API] The *.proto files containing the gRPC definitions are now
  provided by a new Maven Central artifact, with the group "com.daml"
  and the artifact name "ledger-api-proto".
CHANGELOG_END

* release: We don't need the "main-jar" option.

* Bazel: Proto JARs will always have a Maven artifact suffix.

* Bazel: Simplify Protobuf source file TAR and JAR targets.

* Bazel: Extract out Protobuf functions.
2020-11-27 17:14:48 +00:00
Samir Talwar
9976b4cd50
Bazel: Factor out logic around Protobuf JARs. [KVL-714] (#8084)
* kvutils: Use ScalaPB to generate a Scala JAR for daml_kvutils.proto.

* Bazel: Delete the unused `da_java_binary` rule, and inline `_wrap_rule`.

* Bazel: Factor out Java/Scala protobuf class generation into a helper.

CHANGELOG_BEGIN
CHANGELOG_END

* daml-lf/archive: Use `proto_jars`.

* Bazel: Remove the visibility modifier from `proto_jars`.

It's too confusing. Just make everything public.

* daml-lf/archive: Push protobuf source tarballs into `proto_jars`.

* Bazel: Add comments to the various parts of `proto_jars`.

* daml-assistant: Do unpleasant things with `location` in Bazel.
2020-11-27 08:34:53 +00:00
Oliver Seeliger
83f11aa784
Participant pruning ledger api server support ported from Canton (#7988)
* Participant pruning ledger api server support ported from Canton

CHANGELOG_BEGIN
- [Ledger API]: The preview of `ParticipantPruningService` enables ledger participants to prune the "front" of ledger state at the participant including the ledger api server index.
CHANGELOG_END

* Review feedback from Stefano

* Add pruning tests plus missed command completions change

* Review feedback from Robert

* Improved test readability by having populate helper return offsets

* Review feedback

* Ledger api changes to pruning api and disable canton pruning test

- Change return result to PruneResponse
- Change type of PruneRequest.prune_up_to to string

* Review feedback: Use ApiOffsetSConverter for logged offsets
2020-11-23 15:28:04 +01:00
Samir Talwar
d0ffc2c8c5
participant-integration-api: Factor out similar behavior in the admin service implementations. (#7171)
* participant-integration-api: Factor out `pollUntilPersisted`.

CHANGELOG_BEGIN
CHANGELOG_END

* participant-integration-api: Flatten `flatMap`s in admin services.

* participant-integration-api: Use the `FutureConverters` implicits.

* participant-integration-api: Inline `waitForEntry` again.

* participant-integration-api: Commas are great.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* participant-integration-api: Pass the EC to `pollUntilPersisted`.

* participant-integration-api: Extract out more into SynchronousResponse.

* participant-state-index: Stream entries from an optional offset.

Makes them consistent with configuration entries.

* participant-integration-api: Move the ledger end into the strategy.

* participant-integration-api: Use SynchronousResponse everywhere.

* participant-integration-api: Pass inputs to SynchronousResponse#submit.

* participant-integration-api: Add docs to SynchronousResponse.

* participant-integration-api: Add a changelog entry.

CHANGELOG_BEGIN
- [Ledger API] The ConfigurationManagementService will now use the same
  description as other services in case of certain errors. The error
  status codes have not changed.
CHANGELOG_END

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-08-19 10:24:48 +00:00
Stefano Baghino
e972872128
Properly use LoggingContext in Participant Server (#6924)
* Properly use LoggingContext in Participant Server

Fixes #6837

Spreads usage of LoggingContext and ContextualizedLogger throughout the participant server.

changelog_begin
[Sandbox/Integration Kit] We have enriched the contextual information
exposed by the Ledger API server. You should note richer logging information,
which can be read either via unstructured or structured logging frameworks.
A paragraph on how to configure structured logging has been added to the docs.
For more on the issue, see https://github.com/digital-asset/daml/issues/6837.
changelog_end

* Make everything compile

* Leave only actual commands on trace level on submission

* Shorter log messages on submission

* Don't add values down the call stack

* Document good practices regarding the logging context

* Add context to write service implementations

* Add logging context to tests

* Document logging for DAML on SQL

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463503013

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463508117

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463515665

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463513157

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463516359

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463516895

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463518813

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463520210

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463521501

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463521593

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525453

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525560

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525672

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525742

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463526837

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527054

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527523

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527814

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527958

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527900

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527997

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463528050

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463542877

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463543051

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463543614

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463543692
2020-07-31 12:40:13 +00:00
Stephen Compall
4355406259
add more scalac 2.12 warnings (#6798)
* set many extra scalac -Xlint options for all Scala projects

CHANGELOG_BEGIN
CHANGELOG_END

* move NoCopy to its own file

package.scala:18: warning: it is not recommended to define classes/objects inside of package objects.
If possible, define trait NoCopy in package data instead.
  trait NoCopy {
        ^

* move more traits, classes, and objects to proper packages

- note that `package` is itself a scoping construct, so if your reason
  is the apparent aesthetic of placing a bunch of things in one `package
  object`, that is easily remedied by deleting the `object` keyword

* fix some type-parameter-shadow warnings

- I'm generally in favor of sensible name-shadowing, following the
  "deliberately hide variables that should not be accessed here" school
  of thought.  But I think type name shadowing isn't quite as valuable
  and more likely to confuse than general variable shadowing, so have
  experimentally linted it out.

  Example warning:

EventsTableFlatEventsRangeQueries.scala:11: warning: type parameter
 Offset defined in trait EventsTableFlatEventsRangeQueries shadows class
 Offset defined in package v1. You may want to rename your type
 parameter, or possibly remove it.
private[events] sealed trait EventsTableFlatEventsRangeQueries[Offset] {
                                                               ^

* fix more package-object-classes warnings

* fix an inaccessible warning

ContractsService.scala:197: warning: method searchDb in class ContractsService references private class ContractsFetch.
Classes which cannot access ContractsFetch may be unable to override searchDb.
  def searchDb(dao: dbbackend.ContractDao, fetch: ContractsFetch)(
      ^

* enable -Xlint:infer-any

- continuing the saga of #6116, #6132

* enable -explaintypes for more detailed type errors

* missed header for NoCopy; probably should have left it in the package file

* misspelling in comment

* revert -Xlint:doc-detached

- there are a lot of these fixes, and they are noisy, so shifting to a
  separate PR
- thanks to @leo-da for pointing out
2020-07-21 08:18:01 -04:00
Remy
a629959c09
LF: Move GlobalKey and GlobalKeyWithMaintainers outside Node (#6795)
CHANGELOG_BEGIN
CHANGELOG_END
2020-07-21 09:08:36 +02:00
Remy
9e456a1016
DAML-LF: Kill RelativeContractId (#5991)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-25 22:30:45 +02:00
Robert Autenrieth
7e448d810c
Always use the latest ledger config (#5669)
CHANGELOG_BEGIN
- [Sandbox] The ledger API server will now always use the most recent ledger configuration.
  Until a ledger configuration is read from the ledger, command submissions will fail with the UNAVAILABLE error.
CHANGELOG_END

In kvutils, the first ledger configuration change needs
to have a generation one higher than the one returned
by getLedgerInitialConditions().

Remove initial config writing from sandbox as it's now written by the ledger API server
2020-05-06 12:12:23 +02:00
Stefano Baghino
f81efb3d9d
Add new participant schema contracts table (#5541)
* Add new participant schema contracts table

- full implementation (both read and write path)
- fixes #5514

changelog_begin
[Ledger API Server] Fixed a bug that prevented validating transactions
containing reference to contracts that did not originate on the local
participant. See `issue #5514 <https://github.com/digital-asset/daml/issues/5514>`_.
changelog_end

* Format fixes and addressing comments

- address https://github.com/digital-asset/daml/pull/5541#discussion_r408806775
- address https://github.com/digital-asset/daml/pull/5541#discussion_r408806964
- address https://github.com/digital-asset/daml/pull/5541#discussion_r408812745

* Address https://github.com/digital-asset/daml/pull/5541#discussion_r408815185

* Address https://github.com/digital-asset/daml/pull/5541#discussion_r408813067

* Fix mistake in comment

* Apply suggestions from code review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@daml.com>

* Fix compilation error

* Add data migration

Tested by https://github.com/stefanobaghino-da/test-5541-data-migration

* Address https://github.com/digital-asset/daml/pull/5541#discussion_r409386378

* Address https://github.com/digital-asset/daml/pull/5541#discussion_r409385174

* Address https://github.com/digital-asset/daml/pull/5541#discussion_r409367606

* Simplify migration

* Address review

- address https://github.com/digital-asset/daml/pull/5541#discussion_r410344423
- address https://github.com/digital-asset/daml/pull/5541#discussion_r410339914

* Update ledger/sandbox/src/main/resources/db/migration/postgres/V26_0__Contracts_new_schema.sql

Co-Authored-By: Gerolf Seitz <gerolf.seitz@daml.com>

* Fix migration for privacy-aware ledgers

* Remove leftover query

* Fix migration hashes

Co-authored-by: Gerolf Seitz <gerolf.seitz@daml.com>
2020-04-17 20:07:20 +00:00
Gerolf Seitz
a2d785e3ee
Use com.daml as root package (#5343)
Packages com.digitalasset.daml and com.daml have been unified under com.daml

Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.

CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
2020-04-05 19:49:57 +02:00
Stefano Baghino
e2cab1c162
Integrate new active contracts reader in Ledger API (#5397)
changelog_begin
changelog_end
2020-04-03 08:17:42 +00:00
Robert Autenrieth
3945708192
Don't deduplicate failed submissions (#5333)
* Pass explicit submitter and command Id

* Stop deduplicating commands on submission failure

CHANGELOG_BEGIN
CHANGELOG_END

* Add test
2020-04-01 23:38:23 +00:00
Gerolf Seitz
329320bad9
Organize maven coordinates (#5272)
* Use com.daml as groupId for all artifacts

CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END

* Add 2 additional maven related checks to the release binary

1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId

* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
2020-04-01 11:41:18 +02:00
Robert Autenrieth
f8cd628c2f
Remove old time model from ledger config (#5216)
* Remove old time model from ledger config

CHANGELOG_BEGIN
- [Ledger API] Fields related to the old ledger time
  model have been removed from the configuration
  management service and the ledger configuration service.
CHANGELOG_END

* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/tests/LedgerConfigurationService.scala

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2020-03-30 08:46:22 +02:00
Gary Verhaegen
1872c668a5
replace DAML Authors with DA in copyright headers (#5228)
Change requested by Manoj.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 01:26:10 +01:00
Robert Autenrieth
3f597aae16
New ledger time (#5100)
* Tighten result type

Command execution can't result in a sequencer error

* New helper method for extracting used contracts

* New error clause

* Add a DAO query for the maximum time of contracts

* Implement algorithm for finding ledger time

CHANGELOG_BEGIN
CHANGELOG_END

* fixup ledgerTimeHelper

* Use new ledger time algorithm

* Mark LET/MRT as deprecated

CHANGELOG_BEGIN
- [Ledger API] DAML ledgers have switched to a new ledger time model.
  The ledger_effective_time and maximum_record_time fields of command submission are deprecated,
  the ledger time of transactions is instead set automatically by the ledger API server.
  Ledger time is no longer strictly monotonically increasing, but only follows causal monotonicity:
  ledger time of transactions is greater than or equal to the ledger time of any used contract.
  See `#4345 <https://github.com/digital-asset/daml/issues/4345>`__.
CHANGELOG_END

* Add ledger time skew check

* Remove command updater

LET/MRT are now deprecated, this class is now useless

* Remove old time model validator

* Switch to new time model check: kvutils

* Switch to new time model check: in-memory ledger

* Switch to new time model check: SqlLedger

* Use initial ledger config

* Ignore user provided LET

* Use TimeProvider in submission services

* Use deduplication_time in daml-script runner

- Also remove unnecessary command completion output of CommandTracker.
- Remove usage of maximum record time in CommandTracker.

* Use arbitrary default value for deduplication time

* Use built-in Instant ordering

* Remove obsolete test

* Remove obsolete test: CommandStaticTimeIT

* Refactor test: TransactionMRTCompliance

* Disable test: CommandTrackerFlow timeout

* thread maxDeduplicationTime through to CommandTracker

* Improve test

* Refactor command client configuration

* Deduplication time should always use UTC

* Add missing method in TimedIndexService after rebase

* Put more details into the deduplication error response.

* Use system time for command dedup submittedAt.

* Use explicit UTC time source in command validator

* Revert CommandTracker[Flow] to previous completion-recovering-behavior

* Adapt scala client command config to new config params

Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2020-03-25 09:28:56 +01:00
Gerolf Seitz
cf032a02cf
Introduce Stable Offsets (#4953)
A "stable offset" in the context of the Participant Server is the offset
that was provided by the ledger backend (be it kvutils, corda, daml on sql).

The Participant Server does not keep a participant-local offset anymore.
In a single domain/kvutil setup, this makes offsets stable across participants,
since all participants will see the same offset for the same transaction.

The following changes were needed to achieve this:
- The participant server always uses the offset provided by the backend
  AS IS (no more +1 magic).
- Offsets provided to the Ledger API in requests must be treated as
  startExclusive and endInclusive (previously beginInclusive and
  endExclusive).

CHANGELOG_BEGIN
[Ledger API]: Offsets have been redefined. Instead of being represented
by a number or a structured string, an offset is now an opaque string
that can be compared lexicographically.
[DAML Integration Kit]: The bounds for ``Dispatcher`` are now
startExclusive and endInclusive.
CHANGELOG_END

---------
ledger api:
ledger_offset.proto
  Changed definition of offsets, since they can now be compared
  lexicographically.

---------
participant-state-api:
Offset:
  Changed from Array[Long] to ByteString. Ledgers need to make sure that the
  offsets produced are strictly monotonically increasing according to
  lexicographical order.

---------
akka-streams:
Dispatcher, DispatcherImpl, SubSource:
  Changed interval handling to exclusive/inclusive.

---------
ledger-on-memory:
InMemoryLedgerReaderWriter, InMemoryState:
  Changed interval handling to exclusive/inclusive.

---------
ledger-on-sql:
CommonQueries, SqlLedgerReaderWriter:
  Change interval in query and boundary handling.

---------
kvutils:
KeyValueParticipantStateReader, KVOffset:
  Convenience functions for kvutils to add or remove sub-indexes for
  offsets.
  KV ledger implementations can use KVOffset to construct a structured offset.

---------
Participant Server:
JdbcLedgerDao:
  Use Offset instead of Long.
  Fetch offsets directly as Offset from the database with proper anorm
  integration.
  Change interval handling to exclusive/inclusive.

CommandCompletionsReader, CommandCompletionsTable:
  Change interval handling to exclusive/inclusive.

BaseLedger:
  Use Offset instead of Long.
  Change interval handling to exclusive/inclusive.

Conversions:
  Anorm integration for using Offset in queries and result parsers.

JdbcIndexer:
  Remove references to "extenalLedgerEnd" and participant-local Long
  offset (headRef).

---------
sandbox:
In general:
  Use the Offset type everywhere instead of Long.

SQL migrations:
  Change all offset columns to bytea or BINARY.

LedgerBackedIndexService:
  Proper bounds checking has been pushed down to Dispatcher, which
  allowed simplifying the acceptedTransactions implementation.

InMemoryLedger, LedgerEntries:
  Change interval handling to exclusive/inclusive.
  Transaction lookup by ID is now O(n) because transaction IDs are not
  necessarily the same as the offset.

SqlLedger:
  Remove external offset references.
2020-03-18 08:43:41 +01:00
Samir Talwar
3e7d016875
sandbox: Normalize language around listing all known parties. (#4895)
This renames methods backing the `ListKnownParties` request to
from `parties`, `getParties` or `listParties` to `listKnownParties`.

CHANGELOG_BEGIN
- [Ledger API Server] Renamed two metrics:
  ``daml.index.parties`` was renamed to ``daml.index.list_known_parties``
  ``daml.index.db.get_parties`` was renamed to ``daml.index.db.list_known_parties``
CHANGELOG_END
2020-03-09 13:15:33 +00:00
Samir Talwar
0f4fb9a8f5
Ledger API Server: Add the GetParties endpoint. (#4888)
* sandbox: Add a database test for storing and retrieving parties.

* sandbox: Add database queries for selecting one or many parties.

* ledger-api-test-tool: Add a test for `ListKnownParties`.

* sandbox: Add an endpoint to retrieve a single party's details.

CHANGELOG_BEGIN
- [Ledger API] Added an endpoint to retrieve a single party's details at
  ``com.digitalasset.ledger.api.v1.admin.PartyManagementService.GetParty``.
  Please consult the ledger API reference documentation for more
  information.
CHANGELOG_END

* sandbox: Add an endpoint to retrieve a multiple parties' details.

CHANGELOG_BEGIN
- [Ledger API] Added an endpoint to retrieve multiple parties's details at
  ``com.digitalasset.ledger.api.v1.admin.PartyManagementService.GetParties``.
  Please consult the ledger API reference documentation for more
  information.
CHANGELOG_END

* sandbox: Getting a single party is a special case of multiple parties.

So let's use that code path and stop duplicating work.

* sandbox: Remove `GetParty`, as it's subsumed by `GetParties`.

"Subsumed" is a great word.
2020-03-09 12:09:56 +00:00
Robert Autenrieth
98095cf80d
Clean up command deduplication (#4801)
* Always return error on duplicate submissions

* Remove unnecessary submission information

Now that duplicate submissions always return an error,
we don't need to store the original submission result.

CHANGELOG_BEGIN
CHANGELOG_END

* Rename ttl to deduplicationTime/deduplicateUntil

* Store absolute deduplicateUntil in domain commands

* Fix my own initials

* Remove CommandDeduplicationEntry

Instead, use CommandDeduplicationResult everywhere,
removing the extra layer.
2020-03-04 14:08:08 +01:00
Stefano Baghino
00a3a1c2bb
Tighten the loop: backend services to return API responses (#4763)
* Tighten the loop: backend services to return API responses

CHANGELOG_BEGIN
CHANGELOG_END

* Use transaction filter directly

* Remove unnecessary transition through domain objects

* Ensure transient contract remover compares sets of witnesses

* Honor verbosity in request

* Address review https://github.com/digital-asset/daml/pull/4763#pullrequestreview-367012726

- using named parameters when creating the API objects
- renamed EventOps accessors to easily recognizable names
- dropped unnecessary usage of views
- honoring verbosity level in request in all places
- replaced usage of lenses with simple copying where it made sense
2020-03-02 15:06:38 +00:00
Robert Autenrieth
79e7ca0627
Implement new command deduplication (#4467)
* Add TTL field to protobuf

* Add command deduplication to index service

* Wire command deduplication to DAO

* Implement in-memory command deduplication

* Remove Deduplicator

* Implement JDBC command deduplication

* Add TTL field to domain commands

* Deduplicate commands in the submission service

CHANGELOG_BEGIN
- [Sandbox] Implement a new command submission deduplication mechanism
  based on a time-to-live (TTL) for commands.
  See https://github.com/digital-asset/daml/issues/4193
CHANGELOG_END

* Remove unused command service parameter

* fixup protobuf

* Add configuration for TTL

* Fix Haskell bindings

* Rename SQL table

* Add command deduplication test

* Redesign command deduplication queries

* Address review comment

* Address review comment

* Address review comments

* Make command deduplication test optional

* Disable more tests

* Address review comments

* Address review comments

* Refine test

* Address review comments

* scalafmt

* Truncate new table on reset

* Store original command result

* Rename table columns

... to be consistent with other upcoming tables

* Rename migrations to solve conflicts

Fixes #4193.
2020-02-26 12:00:02 +01:00
Stefano Baghino
0e02690382
Make the completion service return API response directly (#4675)
* Make the completion service return API response directly

Skip intermediate transformation to CompletionEvent and return the API
response directly. This approach will be followed by a new implementation
that directly stores to and retrieves from the index instead of using the
transaction stream and filtering it.

CHANGELOG_BEGIN
CHANGELOG_END

* Fix build errors

* Address https://github.com/digital-asset/daml/pull/4675#discussion_r383405441

* Address https://github.com/digital-asset/daml/pull/4675#discussion_r383406125

* Address https://github.com/digital-asset/daml/pull/4675#discussion_r383405965
2020-02-24 18:40:45 +00:00
Andreas Herrmann
f33e79c787
Remove unused dependencies to da_scala_library (#3938)
* Inline all scala_library dependencies

* Run //:buildifier-fix

* TMP scala_library_suite --> scala_library

* da_scala_library: Enable unused dependency checker

* scala_library: Enable unused dependency checker

* //daml-lf/data:data

* //daml-lf/engine:engine

* //ledger-api/rs-grpc-akka:rs-grpc-akka

* //ledger/participant-state:participant-state

* //ledger/ledger-api-client:ledger-api-client

* //scala-protoc-plugins/scala-logging:scala-logging-lib

* //ledger/ledger-api-scala-logging:ledger-api-scala-logging

* //ledger/ledger-api-common:ledger-api-common

* //ledger-service/utils:utils

* //ledger-service/jwt:jwt

* //ledger/ledger-api-auth:ledger-api-auth

* //extractor:extractor

* //daml-assistant/scala-daml-project-config:scala-daml-project-config

* //language-support/codegen-common:codegen-common

* //language-support/scala/codegen:codegen

* //language-support/codegen-main:codegen-main-lib

* //ledger-service/db-backend:db-backend

* //ledger-service/http-json:http-json

* //daml-lf/scenario-interpreter:scenario-interpreter

* //ledger/sandbox:sandbox

* //navigator/backend:navigator-library

* //daml-assistant/daml-sdk:sdk-lib

* //daml-lf/data-scalacheck:data-scalacheck

* //daml-script/test:test-lib

* //ledger/ledger-api-common:ledger-api-common-scala-tests-lib

* //ledger/test-common:test-common

* //ledger/sandbox:sandbox-scala-tests-lib

* //extractor:extractor-scala-tests-lib

* //language-support/java/bindings:bindings-java-tests-lib

* //language-support/java/bindings-rxjava:bindings-java-tests-lib

* //language-support/scala/bindings-akka-testing:bindings-akka-testing

* //language-support/scala/codegen-testing:codegen-testing

* //language-support/scala/codegen-sample-app:daml-lf-codegen-sample-app

* //language-support/scala/codegen-sample-app:daml-lf-codegen-sample-app-testing

* //language-support/scala/codegen-testing:codegen-testing-testing

* //ledger-api/sample-service:sample-service

* //ledger-api/rs-grpc-akka:rs-grpc-akka-tests-lib

* //ledger/ledger-api-test-tool:ledger-api-test-tool-lib

* //ledger/ledger-api-test-tool:ledger-api-test-tool-tests

* //ledger/participant-state/kvutils:kvutils

* //ledger/sandbox:ledger-api-server

* //ledger/sandbox-perf:sandbox-perf-lib

* //navigator/backend:navigator-tests-library

* UNDO scala_library_suite --> scala_library

This reverts commit ab3eb1ae23139e2ec431ab4551fbb0371e0354e1.

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-01-06 18:14:21 +01:00
Gary Verhaegen
878429e3bf
update copyright notices to 2020 (#3939)
copyright update 2020

* update template
* run script: `dade-copyright-headers update .`
* update script
* manual adjustments
* exclude frozen proto files from further header checks (by adding NO_AUTO_COPYRIGHT files)
2020-01-02 21:21:13 +01:00
Andreas Herrmann
9fbb787062 Remove unused dependencies to da_scala_test(_suite) (#3925)
* Remove unused scala.bzl imports

* override_targets org.scalatest.scalatest_2.12

Otherwise, rules_scala implicitly adds a different version to scala_test
than other packages transitively depending on scalatest. This causes
unused dependency checker to raise an error.

* Handle rules_scala scalatest in pom_file.bzl

* Inline all scala_test dependencies

So that `unused_dependency_checker = "error"` can be applied to them.

* Run //:buildifier-fix

* TMP scala_test_suite --> scala_test

* da_scala_test: Enable unused dependency checker

* //navigator/backend:navigator-scala-tests

* //ledger/sandbox:sandbox-scala-tests

* //ledger/participant-state/kvutils:kvutils-tests

* //ledger/participant-state:participant-state-tests

* //ledger/ledger-api-scala-logging:ledger-api-scala-logging-test

* //ledger/ledger-api-common:ledger-api-common-scala-tests

* //ledger/ledger-api-client:ledger-api-client-tests

* //ledger/ledger-api-auth:ledger-api-auth-scala-tests

* //ledger-service/lf-value-json:tests

* //ledger-service/jwt:tests

* //ledger-service/http-json:tests

* //ledger-api/rs-grpc-akka:rs-grpc-akka-tests

* //language-support/scala/codegen-sample-app:tests

* //language-support/scala/codegen-sample-app:ScalaCodeGenIT

* //language-support/scala/codegen:tests

* //language-support/scala/bindings-akka:tests

* //language-support/java/codegen:test

* //language-support/java/codegen:ledger-tests

* //language-support/java/bindings-rxjava:bindings-java-tests

* //language-support/codegen-common:test

* //extractor:extractor-scala-tests

* //daml-lf/scenario-interpreter:scenario-interpreter_tests

* //daml-lf/language:language-test

* //daml-lf/interface:tests

* //daml-lf/engine:tests

* //daml-lf/encoder:tests

* //daml-lf/archive:daml_lf_archive_reader_tests

* //daml-assistant/scala-daml-project-config:scala-daml-project-config-tests

* UNDO scala_test_suite --> scala_test

This reverts commit 13ed47ba725e944533ca1157a070cb8dc30569ac.

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2019-12-30 13:49:59 +00:00
Jussi Mäki
33c7a1aace Implementation for the configuration management service (#3756)
* Implementation for the configuration management service

- Add configuration generation to the response of SetTimeModel
- Implement the ConfigManagementService
- Implement integration test into test tool

This is still a draft as it has unsolved FIXMEs and
it conflicts with #3744 which should go first.

The main conflict is with changes to PartyAllocationResponse which cannot
no longer reference "domain.PartyDetail" as we would have a cyclic dependency
(participant-state contains Configuration which we point to from domain.ConfigurationEntry).

The still open issues are:
- Revisit PartyDetail
- Naming: LedgerConfigurationService and ConfigManagementService are not talking
  about the same configuration and it feels confusing.
- Remove duplication of ConfigurationEntry? Do we need both domain.ConfigurationEntry
  and ledger.store.ConfigurationEntry? Only difference is in the types of participantId
  and submissionId.

* Address review part 1

* Fix up tests after rebase and address PR review

* Post-merge fixes

* Add missing config MRT checks and fixes to tests

- Check config MRT in InMemoryLedger and SqlLedger
- Use proper source of time in ConfigManagement
- Separate out ConfigManagementServiceIT in sandbox conformance tsets

* Reformat
2019-12-13 17:58:44 +00:00
Jussi Mäki
ce70ad4a2f
Async package management (#3806)
* Add package_entries table

* Change PublicPackageUpload event to cover list of packages.

Add PublicPackageUploadRejected.

* Produce new package update events in KeyValueConsumption

* Update signature of uploadPackages

* Cleanup InMemoryKVParticipantState. Add submissionId to uploadPackages.

* Fix up InMemoryKVParticipantStateIT

* Initial ledger dao changes for package entries

Drop the participant_id as we never expect to see
entries of other participants. This should be done
for party_entries as well.

* Drop UploadPackagesResult

* Implement getPackageEntries and refactor callers

* Add maxRecordTime to uploadPackages

* First cut at updating ApiPackageManagementService

* Update tests, wire through the packageEntries

* Don't extend IndexPackagesService in InMemoryPackageStore

It does not implement the full interface and it isn't used
directly as one anyway.

* Drop maximum_record_time from package_management_service

Adding maximum record times touches the whole stack. Leaving
this change to another PR.

* Wire through the removal of maximum_record_time.

And remove dead code from InMemoryKVParticipantState

* Remove checking for duplicate package uploads

This aligns with the behaviour of WriteService.

* Reformat

* Fix PackageManagementService after adding of submission_id to the service
2019-12-13 09:56:47 +01:00
Gerolf Seitz
0c00b28c83
Party management (#3744)
* Implement Async party allocation

* Address Jussi's review.

- Remove comments about implicit party entries
- Remove PollingUtils#pollSingleUntilPersisted
- Remove ill-placed comment in SqlLedger

Review: https://github.com/digital-asset/daml/pull/3744#pullrequestreview-327453144

* Address Samir's review

- Slight improvement in InMemoryKVParticipantStateIT
- Fix typo in scaladoc
- Fix SqlLedgerSpec: remove unused param and rephrase clues

Review: https://github.com/digital-asset/daml/pull/3744#pullrequestreview-327461368

* Address Stefano's review

Review: https://github.com/digital-asset/daml/pull/3744#pullrequestreview-327455206

* Only poll from the ledger end just before submitting the party allocation request.
2019-12-05 17:50:21 +01:00
Samir Talwar
5dd38d54e8 sandbox: PostgreSQL health checks. (#3655)
* ledger-api-test-tool: Increase the duration when watching health.

This should hopefully stop CI from flaking out.

* reference-v2/sandbox: Avoid unnecessary companion object constructors.

I like indirection… when it does something.

* ledger: Propagate empty health checks throughout the services.

* reference: Remove duplication from the ReferenceServer object.

* ledger-api-common: Actually query a "reporter" in the health service.

* ledger-api-common: Report health per-component when required.

* ledger-api-health: Use a Map to represent components for health checks.

* sandbox: Fix warnings in SqlLedgerSpec.

* ledger-api-common: Throw GrpcHealthService errors inside the Future.

* ledger: Implement health checks against the PostgreSQL connection.

Without proper testing, because I am not great at this.

* sandbox: Remove duplication and fix warnings in PostgresAround.

* sandbox: Test the SQL Ledger's health reporting on failure.

* sandbox: Don't report as unhealthy until 3 connections fail.

* ledger-api-health: Remove unused parts of the API.

Bit of premature design there.

* sandbox: Rename the "ledger" health check to "write".

* participant-state: Add the ReportsHealth trait to ReadService.

* ledger-api-common: `Future.fromTry(Try(…))` -> `Future(…)`.

* ledger-api-common: Make it clearer that StubReporter closes over health.

* ledger-api-common: Explain the HealthService watch tests with comments.

* sandbox: Clean up SqlLedger a bit.

* sandbox: Don't try and stop PostgreSQL twice in PostgresAround.

* bazel_tools: Windows rlocation lookups need to be with forward slashes.

* release: Fix case of "true".

* ledger-api-common: Make `GrpcHealthService::matchResponse` return a Try.

* ledger-api-common: Make `GrpcHealthServiceSpec` async.

* sandbox: Make a couple of DB classes final.

* sandbox: Avoid importing `X._` in PostgresAround.

* sandbox: Add clues to the SqlLedgerSpec's multiple assertions.

* sandbox: If PostgreSQL doesn't come back up, keep retrying.

* sandbox: Remove duplication in SqlLedgerSpec.

* sandbox: In SqlLedgerSpec, actually wait for the health to change.

* sandbox: In PostgresAround, make stopping PostgreSQL idempotent.

* sandbox: Simplify the SqlLedgerSpec to make it work on CI.

It's worth a shot.

* ledger-api-common: Simplify the GrpcHealthServiceSpec a little.

And add a changelog.

CHANGELOG_BEGIN

- [Ledger API Server] Add a health check endpoint conforming to the
  `GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
- [Ledger API Server] Add health checks for index database connectivity.
- [Participant State API] Add a mandatory ``currentHealth()`` method to
  ``IndexService``, ``ReadService`` and ``WriteService``.

CHANGELOG_END

* sandbox: Improve the Javadoc layout for DbDispatcher.

* sandbox: Capitalize constants in SqlExecutor.

* ledger-api-health: Convert HealthStatus to an abstract class.
2019-11-29 15:07:43 +00:00
Andreas Herrmann
2bd1db490a
Replace bazel-deps by rules_jvm_external (#3253)
* Update bazel-common to fix javadoc issues

Specifically, to fix the following error

```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel:7:1: in javadoc_library rule //ledger-api/rs-grpc-bridge:rs-grpc-bridge_javadoc:
Traceback (most recent call last):
        File "/home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel", line 7
                javadoc_library(name = 'rs-grpc-bridge_javadoc')
        File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/com_github_google_bazel_common/tools/javadoc/javadoc.bzl", line 27, in _javadoc_library
                dep.java.transitive_deps
object of type 'JavaSkylarkApiProvider' has no field 'transitive_deps'
```

* Define Maven deps using rules_jvm_external

* Pin artifacts

* Remove bazel-deps generated targets

* Remove bazel-deps

* Switch to rules_jvm_external targets

* update bazel documentation

* pom_file: There are no more bazel-deps targets

* BAZEL-JVM.md `maven_install` typo
2019-10-28 13:53:14 +01:00
Remy
e45b585307 daml-lf: repackage daml-lf archive proto (daml_lf -> daml_lf_dev) (#3169) 2019-10-15 09:36:11 +00:00
Remy
d64df4b5f7 daml-lf: clean achive bazel build (#3175)
* clean the mess in daml-lf achive bazel build

* add release note

* Address Moritz's comments

* more release notes
2019-10-15 08:36:46 +00:00
Gerolf Seitz
5592092cc7
Remove DAML-on-X IndexServer and Reference Server (#3108)
It is not needed anymore and can therefore be deleted.
2019-10-08 11:47:21 +02:00
Robert Autenrieth
b97ef28de1
Rename BlindingInfo fields (#2883) 2019-09-18 00:28:53 +02:00
Robert Autenrieth
e288437dc0
Support divulged contracts in a distributed ledger (#2739)
Fixes #2488

Includes some refactoring of ActiveContracts (now ActiveLedgerState).
2019-09-11 14:21:41 +02:00
Gerolf Seitz
cd3a6f9ffe
Port integration test for ActiveContractsService (#2580) 2019-08-22 11:35:55 +02:00
Jussi Mäki
4ecf070860
kvutils: Authorization and config submissions (#2433)
* Initial version for configuration submissions and authorization

* Refactor config submission based on review. Add tests.

* Cleanup test output and comments

* kvutils test-suite

- Add unit tests for kvutils
- Thread the input state into Commit monad
- Fix a bug in handling rejected transactions (from previous refactoring)

* Address PR review

- Add submissionId to configuration changes.
- Revert the dedup of the ParticipantNotAuthorized message. This did not buy much
  and we need to duplicate it anyway when modularizing kvutils.
2019-08-21 18:24:16 +02:00
Stefano Baghino
eb965866cc
Simplify new test tool (#2573)
* Simplify new test tool

* Shorten exercise to leverage ContractId

* Remove dead code

* Address https://github.com/digital-asset/daml/pull/2573#discussion_r314702191

* Address https://github.com/digital-asset/daml/pull/2573#discussion_r314700739

* Simplify transaction requests
2019-08-16 16:24:00 +02:00