* Modularize JdbcLedgerDaoSpec
Adding more tests, but breaking it up a bit to make sense of it beforehand.
CHANGELOG_BEGIN
CHANGELOG_END
* Remove unnecessary suppressed warnings
* sandbox: Fail to start if a time mode is not explicitly specified.
CHANGELOG_BEGIN
- [Sandbox] Sandbox is switching from Static Time mode to Wall Clock
Time mode as the default. To ensure that our users know about this,
for one version, there will be no default time mode. Instead, users
will have to explicitly select their preferred time mode by means of
the `--static-time` or `--wall-clock-time` switches. In the next
release, Wall Clock Time will become the default, and users who are
happy with the defaults will no longer need to specify the time mode.
CHANGELOG_END
* daml-script|triggers: Specify time mode when testing against Sandbox.
* daml-assistant: Default the Sandbox to wall clock time.
CHANGELOG_BEGIN
- [DAML Assistant] Initializing a new DAML project adds a switch to
``daml.yaml`` to ensure Sandbox can continue to start with ``daml
start``::
sandbox-options:
- --wall-clock-time
CHANGELOG_END
* docs: Update the DAML Script and Triggers docs to use Wall Clock time.
It's now what Sandbox will use by default when using `daml init`.
* docs: Change the Quickstart to run Sandbox in wall clock time.
This explains why the contract IDs may vary.
It also updates the manual release testing script to match.
A "stable offset" in the context of the Participant Server is the offset
that was provided by the ledger backend (be it kvutils, corda, daml on sql).
The Participant Server does not keep a participant-local offset anymore.
In a single domain/kvutil setup, this makes offsets stable across participants,
since all participants will see the same offset for the same transaction.
The following changes were needed to achieve this:
- The participant server always uses the offset provided by the backend
AS IS (no more +1 magic).
- Offsets provided to the Ledger API in requests must be treated as
startExclusive and endInclusive (previously beginInclusive and
endExclusive).
CHANGELOG_BEGIN
[Ledger API]: Offsets have been redefined. Instead of being represented
by a number or a structured string, an offset is now an opaque string
that can be compared lexicographically.
[DAML Integration Kit]: The bounds for ``Dispatcher`` are now
startExclusive and endInclusive.
CHANGELOG_END
---------
ledger api:
ledger_offset.proto
Changed definition of offsets, since they can now be compared
lexicographically.
---------
participant-state-api:
Offset:
Changed from Array[Long] to ByteString. Ledgers need to make sure that the
offsets produced are strictly monotonically increasing according to
lexicographical order.
---------
akka-streams:
Dispatcher, DispatcherImpl, SubSource:
Changed interval handling to exclusive/inclusive.
---------
ledger-on-memory:
InMemoryLedgerReaderWriter, InMemoryState:
Changed interval handling to exclusive/inclusive.
---------
ledger-on-sql:
CommonQueries, SqlLedgerReaderWriter:
Change interval in query and boundary handling.
---------
kvutils:
KeyValueParticipantStateReader, KVOffset:
Convenience functions for kvutils to add or remove sub-indexes for
offsets.
KV ledger implementations can use KVOffset to construct a structured offset.
---------
Participant Server:
JdbcLedgerDao:
Use Offset instead of Long.
Fetch offsets directly as Offset from the database with proper anorm
integration.
Change interval handling to exclusive/inclusive.
CommandCompletionsReader, CommandCompletionsTable:
Change interval handling to exclusive/inclusive.
BaseLedger:
Use Offset instead of Long.
Change interval handling to exclusive/inclusive.
Conversions:
Anorm integration for using Offset in queries and result parsers.
JdbcIndexer:
Remove references to "extenalLedgerEnd" and participant-local Long
offset (headRef).
---------
sandbox:
In general:
Use the Offset type everywhere instead of Long.
SQL migrations:
Change all offset columns to bytea or BINARY.
LedgerBackedIndexService:
Proper bounds checking has been pushed down to Dispatcher, which
allowed simplifying the acceptedTransactions implementation.
InMemoryLedger, LedgerEntries:
Change interval handling to exclusive/inclusive.
Transaction lookup by ID is now O(n) because transaction IDs are not
necessarily the same as the offset.
SqlLedger:
Remove external offset references.
* Switched to ByteString from Array[Byte] on almost all simplified API interfaces.
* Sort output by keys.
* Added comment.
CHANGELOG_BEGIN
CHANGELOG_END
* Removed DamlLogEntryId from LedgerEntry.
* Return a SortedMap ordering output state by its keys' hash in order to have deterministic ordering.
* Code tidying.
* Added implicit conversion for anorm for ByteStrings to make SQL queries cleaner.
* Ooops, missed adding a header.
* Avoid copying bytes by anorm by using ByteString.newInput()
* Added some Scaladoc to simplified API interfaces.
* Added docs to LedgerStateAccess.
* Reverted some changes.
* Added some docs to ValidatingCommitter.
* Corrected some typos.
* Added package-level documentation to kvutils.api.
* Clarified convenience classes for LedgerStateOperations.
* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/participant/state/kvutils/Version.scala
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Minor rewording.
* Added missing header.
* Fixed problem with merge.
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* kvutils: Remove the unnecessary execution context from the test base.
* kvutils: Remove the unnecessary execution context from the writer.
* ledger-on-sql: Make a proper owner so it has a proper execution context.
This means the parallelization now needs to come from the test, so I've
augmented ParticipantStateIntegrationSpecBase to take a proper execution
context instead of the serial one that ScalaTest provides, with a
default of `ExecutionContext.global`.
* ledger-on-memory: Make a proper owner with a proper execution context.
* kvutils/app: Remove `executionContext` from LedgerFactory.
Shouldn't need it in `ResourceOwner`. I was bad.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-memory: Make ResourceOwners real classes.
* ledger-on-sql: Make the ResourceOwner a real class.
* ledger-on-sql: Cause side effects on resource acquisition.
Not on owner construction.
This would fail only on PostgreSQL because `IN ()` is invalid. H2 seems
to be fine with it.
CHANGELOG_BEGIN
- [Ledger API Server] Support a call to `GetParties` with an empty list
of parties.
CHANGELOG_END
* Periodically clear expired deduplication entries
CHANGELOG_BEGIN
CHANGELOG_END
Fixes#4959
* Increase cache maintenance frequency
The previous value was only good for testing purposes
* Actually remove deduplication entries
* Clear deduplication cache for IndexAndWriteService
* Share test certificates
This is primarily an attempt at making sure my contribution stats
remain negative but I think it’s a nice cleanup. The only difference
in the certs used by daml-helper which are now used everywhere is that
they use a different CN for the CA and the server. This is required to
make openssl happy (which is used by the daml-helper).
changelog_begin
changelog_end
* Fix script and trigger tests
This PR fixes the tls configuration to work if client auth is not
enabled and adds a `--tls` flag to extractor and navigator which
allows you to enable tls without overriding any certificates.
There is a test for extractor but none for navigator since there are
no tls tests at all afaict atm. I did however test it manually.
changelog_begin
- [Navigator] Navigator can now run a TLS enabled ledger without
client authentication. You can enable TLS without any special
certificates by passing ``--tls``.
- [Extractor] Extractor can now run a TLS enabled ledger without
client authentication. You can enable TLS without any special
certificates by passing ``--tls``.
changelog_end
Currently sandbox only supports TLS if you also enable client
authentication. There is no reason for why this has to be the case and
for things like DABL we want TLS without client authentication so it’s
useful to be able to test this in sandbox. This PR introduces a
`--client-auth` flag that allows you to configure the behavior. The
default is the current one of requiring client authentication.
This PR does not yet update Java clients, however, the Haskell client
supports this already and is used to test this functionality.
I’ve also added a section in the documentation on TLS (there were no
docs at all so far).
changelog_begin
- [DAML Sandbox] When Sandbox is run with TLS enabled, you can now
configure the requirement for client authentication via
``--client-auth``. See
https://docs.daml.com/tools/sandbox.html#running-with-tls for more information.
changelog_end
We exclude the tests that create lots of data.
CommandDeduplicationIT is disabled as kvutils does not yet
have time-based deduplication.
CHANGELOG_BEGIN
CHANGELOG_END
* Don't read exclusive end in completions query
CHANGELOG_BEGIN
CHANGELOG_END
* Store offsets directly and do +1 only on read side
* Fix existing completions
* Add test for the completion service
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Rename EC auth cmdline options in line with the standard and document them.
CHANGELOG_BEGIN
CHANGELOG_END
* 📝 Fix doc
* Auth docs: change `RSA DSA` -> `RSA Signature` (clashed with DSA algo)
As proposed by @SamirTalwar-DA
CHANGELOG_BEGIN
[Sandbox] Rename the `--auth-jwt-ec256-crt` command line option to `--auth-jwt-es256-crt` as well as `--auth-jwt-ec256-crt` to `--auth-jwt-es256-crt` and fix their docs
CHANGELOG_END
* kvutils: Avoid casting `ArgumentCaptor` and friends in tests.
Instead, use generics the way they're intended.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: In KeyValueParticipantStateWriterSpec, drop the Option.
After some investigation, canton does not currently expose a nice way
to tell ammonite where it should write its files or even better use
the in-memory mode. However, ammonite respects $HOME so we can just
set that to a temp directory which fixes the issue.
changelog_begin
changelog_end
* Include Bazel patch to mark tests as exclusive
This should hopefully avoid rerunning the conformance tests as often
as we do now. While this patch is not applied on Windows (since we get
it from nix), this is not really an issue since most of the exclusive
tests (in particular, all conformance tests) are disabled on Windows
anyway.
I’ve tested this locally accross a couple of runs and I get the
caching I want and looking at the code in the patch, the change looks
very reasonable. I somewhat wonder if it just broke internally at
google because they marked tests as exclusive that should have gotten
no-cache.
changelog_begin
changelog_end
* Disable caching for canton
* ledger-api-test-tool: Fix warnings flagged by IntelliJ IDEA.
* ledger-api-test-tool: Open-world mode.
In open-world mode, parties aren't allocated; their names are just
reserved for the test case, so that no other test will accidentally use
the same party name.
This is so we can test ledgers which dynamically allocate parties, such
as Sandbox.
* sandbox: Run conformance tests in "open-world" mode.
This means that the tests don't explicitly allocate parties (except for
a few), instead relying on Sandbox's implicit party allocation feature.
This is not enabled for Sandbox Next yet.
* sandbox-next: Implicit party allocation.
This is added to the command submission service.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-next: Don't implicitly allocate pre-existing parties.
* ledger-api-test-tool: Move pre-allocation into ParticipantTestContext.
* ledger-api-test-tool: We can reserve parties or wait for them. Not both.
Make illegal states unrepresentable as early as possible.
* sandbox: Name ApiSubmissionService's private methods a little better.
* sandbox: Move ApiSubmissionService's conditional logic into methods.
* sandbox: Document why we set `implicitPartyAllocation` to `false`.
* sandbox: Document why `implicitPartyAllocation` is dangerous.
* Rework ValueSerializer
- Handle errors directly in the convenience method (it's the only way in which it's used)
- Don't drop the root cause when a serialization error occurs
- Remove outdated comment
CHANGELOG_BEGIN
CHANGELOG_END
* Add alternative with error context for deserialize method
* Make error context evaluated lazily
* Rename inner helper to avoid name clash
This renames methods backing the `ListKnownParties` request to
from `parties`, `getParties` or `listParties` to `listKnownParties`.
CHANGELOG_BEGIN
- [Ledger API Server] Renamed two metrics:
``daml.index.parties`` was renamed to ``daml.index.list_known_parties``
``daml.index.db.get_parties`` was renamed to ``daml.index.db.list_known_parties``
CHANGELOG_END
* sandbox: Add a database test for storing and retrieving parties.
* sandbox: Add database queries for selecting one or many parties.
* ledger-api-test-tool: Add a test for `ListKnownParties`.
* sandbox: Add an endpoint to retrieve a single party's details.
CHANGELOG_BEGIN
- [Ledger API] Added an endpoint to retrieve a single party's details at
``com.digitalasset.ledger.api.v1.admin.PartyManagementService.GetParty``.
Please consult the ledger API reference documentation for more
information.
CHANGELOG_END
* sandbox: Add an endpoint to retrieve a multiple parties' details.
CHANGELOG_BEGIN
- [Ledger API] Added an endpoint to retrieve multiple parties's details at
``com.digitalasset.ledger.api.v1.admin.PartyManagementService.GetParties``.
Please consult the ledger API reference documentation for more
information.
CHANGELOG_END
* sandbox: Getting a single party is a special case of multiple parties.
So let's use that code path and stop duplicating work.
* sandbox: Remove `GetParty`, as it's subsumed by `GetParties`.
"Subsumed" is a great word.
Events in transaction trees should only reference other events
that:
1) are either create or exercise events
2) the requesting parties are a witness of
This applies to recalculated root nodes as well as
the child event ids referenced in exercise nodes.
CHANGELOG_BEGIN
[Sandbox]: fixed projection of transaction trees.
CHANGELOG_END
* Deprecate ledger initialization with scenarios
CHANGELOG_BEGIN
[Sandbox] Initializing the sandbox with scenarios is now deprecated in
favor of using DAML Script. The scenario parameter will be removed in
the near future. A warning is logged on startup.
The DAML SDK templates and quickstart guide are using DAML Script.
See the DAML Script migration guide for more information:
https://docs.daml.com/daml-script/index.html#using-daml-script-for-ledger-initialization
CHANGELOG_END
* sandbox-next: Pull runner configuration into the constructor.
No need to do it on `acquire()` if it's pure.
* sandbox-next: Error if a scenario is provided.
Sandbox-Next doesn't support scenarios, instead favoring DAML Script.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: On error opening an envelope, throw the correct message.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: On error when querying state, throw the correct error.
* kvutils|ledger-on-sql: Remove unnecessary curly braces around `throw`.
* Refactor extraction of events from transaction
Closes#1909
CHANGELOG_BEGIN
CHANGELOG_END
* Remove unnecessary filtering
* Fix disclosure rule for flat transaction
* Refactor and split collection and filtering
* Replace transaction filtration with blinding info
* Move transient contract remover in transaction conversion
* Remove dangling file
* Simplify transient contract filtering
* Further refinements
* Simplify transaction tree event extraction
* Move newRoots up the file for consistency and readability
* Remove collect from GenTransaction, replace with custom iterator
* Address https://github.com/digital-asset/daml/pull/4781#discussion_r388167562
* Switch to a strict collect method
* Replaced direct access to map with contains
* sandbox: Re-use the root actor system in the StandaloneIndexerServer.
* kvutils/app: Don't use the ActorSystem execution context randomly.
Instead, make `Runner` a proper ResourceOwner, with an `acquire` method.
* sandbox: Re-use the root actor system in the StandaloneApiServer.
CHANGELOG_BEGIN
CHANGELOG_END
* resources: Remove the now-unused `ResourceOwner.sequence` functions.
They weren't well-thought-out anyway; they acquire resources
sequentially, rather than in parallel.
* Allow `LedgerFactory` to provide a full-blown `ReadWriterService` rather than a `LedgerReaderWriter` (needed by at least vDAML)
CHANGELOG_BEGIN
CHANGELOG_END
* Address review points
* Address Samir's review point in `SqlLedgerFactory`
* Finish addressing Samir's review point in `SqlLedgerFactory`
* Split reader and writer owners in `LedgerFactory` as suggested by Gerolf
* Remove unneeded `val` from `KeyvalueParticipantState[Reader|Writer]` constructor params
* Remove unneeded type parameter from `app.Runner`
* Leave `LedgerFactory` a full ledger builder but split hierarchy upwards and clarify responsibilities
* Rename `SimpleLedgerFactory` to `KeyValueLedgerFactory`
* sandbox-next: Make the Runner a real ResourceOwner.
* sandbox: Don't construct the ResetService twice.
* sandbox: Inline and simplify methods in StandaloneApiServer.
* resources: Define a `ResettableResource`, which can be `reset()`.
`reset()` releases the resource, performs an optional reset operation,
and then re-acquires it, binding it to the same variable.
* resources: Pass the resource value into the reset operation.
* sandbox: Fix warnings in `TestCommands`.
* sandbox-next: Add the ResetService.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Make sure the SandboxResetService resets asynchronously.
It was being too clever and negating its own asynchronous behavior.
* sandbox-next: Forbid no seeding.
This double negative is really hard to phrase well.
* sandbox-next: Implement ResetService for a persistent ledger.
* sandbox: Delete the comment heading StandaloneIndexerServer.
It's no longer meaningful.
* sandbox-next: No need to wrap the SandboxResetService in an owner.
* sandbox-next: Bump the ResetService test timeouts.
It looks like it's definitely slower than on Sandbox Classic™. Gonna
look into this as part of future work.
* Revert to previous asynchronous reset behavior
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Always return error on duplicate submissions
* Remove unnecessary submission information
Now that duplicate submissions always return an error,
we don't need to store the original submission result.
CHANGELOG_BEGIN
CHANGELOG_END
* Rename ttl to deduplicationTime/deduplicateUntil
* Store absolute deduplicateUntil in domain commands
* Fix my own initials
* Remove CommandDeduplicationEntry
Instead, use CommandDeduplicationResult everywhere,
removing the extra layer.
It is basically impossible to not hit this all the time if you upload
more than one package so issuing a warning is a bit confusing.
changelog_begin
- [Sandbox] The warning about duplicate package uploads is no longer
emitted by default. You can enable them by passing
``--log-level=debug``.
changelog_end
* sandbox: Return `Future[Unit]` from migrations rather than awaiting.
I've removed the explicit error-handling, because this will be
propagated and handled at the top level.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Pass the JDBC URL into the JdbcIndexerFactory constructor.
* sandbox: Replace the JdbcIndexerFactory's `InitStatus` with two classes.
The `asInstanceOf` conversions put me off.
* sandbox: Stop passing around the ledger ID in JdbcIndexerFactory.
* sandbox: Remove the indexer `asyncTolerance`; it's no longer used.
The change to `EventFilter` and to the query in `JdbcLedgerDao` are
"duplicate work", but we need the change in EventFilter for the
InMemoryLedger, and the change in JdbcLedgerDao so that we avoid
fetching a contract that anyway would be discarded later.
CHANGELOG_BEGIN
[Sandbox]: Witnessed contracts for which a party is not a stakeholder
are no longer returned in the active contract stream.
CHANGELOG_END
Fixes#3254.
* libs-scala/ports: Wrap socket ports in a type, `Port`.
* sandbox: Use `Port` for the API server port, and propagate.
CHANGELOG_BEGIN
CHANGELOG_END
* extractor: Use `Port` for the server port.
* ports: Make Port a compile-time class only.
* ports: Allow port 0; it can be specified by a user.
* ports: Publish to Maven Central.
* kvutils: Make the `KeyValueParticipantStateReader` tests more rigorous.
If the `offset` is specified as `None`, expect it to be `None`, not
just anything.
* kvutils: Simplify `KeyValueParticipantStateReader#stateUpdates`.
Construct the Source with `Source.apply`, not `Source.fromIterator`.
* kvutils: Use multiple entry IDs in `KeyValueParticipantStateReaderSpec`.
* kvutils: Add basic tests to `KeyValueParticipantStateReaderSpec`.
* kvutils: Add heartbeats to `LedgerReader`'s `events` output.
Heartbeats are optional, to be delivered by the ledger if and when it
deems necessary.
* sandbox-next: An observing time service backend using Akka streams.
* sandbox-next: A regular heartbeat based on Akka Streams' `tick`.
* sandbox: Replace `TimeServiceBackend.withObserver` with `.observing`.
More code, but it's more decoupled, so can more easily be sent to the
underlying backend in Sandbox Next.
CHANGELOG_BEGIN
- [Sandbox] Fixed a bug in the command completions stream when running
Sandbox in static time. Previously, upon updating the time, the old
time was emitted on the completions stream. The new time is now
emitted.
CHANGELOG_END
* sandbox: TimeServiceBackend should only emit accepted changes.
* ledger-on-memory: Use `LedgerRecord` directly.
* ledger-on-memory: Stream heartbeats to the log.
* ledger-on-memory: Encapsulate mutations behind locks at all times.
* ledger-on-memory: Differentiate between reading and writing.
* ledger-on-memory: Factor out appending to the log.
* kvutils: Move the heartbeat test into the base from ledger-on-memory.
* kvutils: Log when the submission validation fails unexpectedly.
* ledger-on-sql: Add a script to hash all migrations.
* ledger-on-sql: Publish heartbeats to the log, and stream them out.
* ledger-on-sql: Log if publishing the heartbeat failed.
* ledger-on-sql: Wrap all queries in `Try`.
Just to make sure that we don't throw from a function that returns `Try`
or `Future`.
* ledger-on-sql: Allow `Long` values as the heartbeat timestamp.
`INTEGER` really does mean 32-bit, apparently.
* sandbox-next: Pipe heartbeats to the ledger.
* ledger-on-sql: Make sure we publish the correct head after a heartbeat.
Off-by-one errors are the best errors.
* ledger-on-(memory|sql): Just accept heartbeats, not their owner.
* sandbox: Update CIDs in tests to account for the extra heartbeat.
* ledger-on-memory: Fix a reference to variable in a comment.
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* ledger-on-sql: `flatMap` over `Try` rather than `Future` when possible.
* sandbox: Make sure the heartbeat queues are thread-safe.
* kvutils: Remove `LoggingContext` from the interfaces.
Keep it internally. This means we'll drop any context, but otherwise
things should work as expected.
* sandbox-next: Pull out the heartbeat interval into a constant.
* ledger-on-sql|sandbox: Clarify large levels of nesting.
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
Rejected submissions are a user-error and don't indicate a problem with the server not functioning properly. Such user
errors make it hard to spot "real" server-side warnings and errors.
i
Closes#4772
CHANGELOG_BEGIN
- The Ledger API Server now logs rejected submissions at a lower "INFO" level to remove a source of warnings/errors without relation to server health.
CHANGELOG_END
* Freeze DAML-LF 1.8
Two minor points that I did not mention in the previous PR:
We also include the renaming of structural records to `struct` and the
renaming of `Map` to `TextMap`.
There are some minor changes around the LF encoder tests which need to
be able to emit package metadata properly so I’ve added it to the
parser. Sorry for not splitting that out.
Following the process used for the DAML-LF 1.7 release, this does not
yet include the frozen proto file.
changelog_begin
- [DAML-LF] Release DAML-LF 1.8:
* Rename structural records to ``Struct``. Note that
structural records are not exposed in DAML.
* Rename ``Map`` to ``TextMap``.
* Add type synonyms. Note that type synonyms are not serializable.
* Add package metadata, i.e., package names and versions.
Note that the default output of ``damlc`` is stil DAML-LF 1.7. You
can produce DAML-LF 1.8 by passing ``--target=1.8``.
changelog_end
* Update encoder
* Update java codegen tests
* Update comment in scala codegen
* Handle TSynApp in interface reader
* Bump lf_stable_version to 1.7
* Fix kvutils tests
* Make kvutils work with the new contract id scheme
CHANGELOG_BEGIN
- [KVUtils] uses random contract id. Contract ids are made of 65 hexa decimal characters.
CHANGELOG_END
Co-authored-by: Jussi Mäki <jussi.maki@digitalasset.com>
* Tighten the loop: backend services to return API responses
CHANGELOG_BEGIN
CHANGELOG_END
* Use transaction filter directly
* Remove unnecessary transition through domain objects
* Ensure transient contract remover compares sets of witnesses
* Honor verbosity in request
* Address review https://github.com/digital-asset/daml/pull/4763#pullrequestreview-367012726
- using named parameters when creating the API objects
- renamed EventOps accessors to easily recognizable names
- dropped unnecessary usage of views
- honoring verbosity level in request in all places
- replaced usage of lenses with simple copying where it made sense
* sandbox: Name the arguments to `ApiServices.create` for clarity.
* sandbox: Clarify numbers and types in configuration classes.
* sandbox-next: Log the correct port on startup.
* sandbox-next: Connect up the command configuration.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-next: Wire up TLS configuration.
* sandbox-next: Wire up the maximum inbound message size.
* sandbox-next: Set the global log level if specified.
And if it's not specified, default to the level in logback.xml, INFO.
* sandbox-next: Connect up the submission configuration.
* sandbox-next: Log the correct ledger ID.
* sandbox-next: Use `TimeProvider.UTC`.
* Make completion service return checkpoints
The new table for #4681 and the query used to retrieve completions
currently does not return checkpoints. These do not have to match
the application_id and submitting_party query since those fields
are not populated.
CHANGELOG_BEGIN
CHANGELOG_END
* Address https://github.com/digital-asset/daml/pull/4735#discussion_r384713277
This removes the sample/reference implementation of kvutils
InMemoryKVParticipantState.
This used to be the only implementation of kvutils, but now with the
simplified kvutils api we have ledger-on-memory and ledger-on-sql.
InMemoryKVParticipantState was also used for the ledger dump utility,
which now uses ledger-on-memory.
* Runner now supports a multi participant configuration
This change removes the "extra participants" config and goes for consistent
participant setup with --participant.
* Run all conformance tests in the repository in verbose mode.
This means we'll print stack traces on error, which should make it
easier to figure out what's going on with flaky tests on CI.
This doesn't change the default for other users of the
ledger-api-test-tool; we just add the flag for:
- ledger-api-test-tool-on-canton
- ledger-on-memory
- ledger-on-sql
- sandbox
Fixes#4225.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-next: Get the authorization service from configuration.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-next: Add parameter names to Runner function calls.
It's getting very confusing what's what without them, and the mix of
with and without is more confusing.
* Add TTL field to protobuf
* Add command deduplication to index service
* Wire command deduplication to DAO
* Implement in-memory command deduplication
* Remove Deduplicator
* Implement JDBC command deduplication
* Add TTL field to domain commands
* Deduplicate commands in the submission service
CHANGELOG_BEGIN
- [Sandbox] Implement a new command submission deduplication mechanism
based on a time-to-live (TTL) for commands.
See https://github.com/digital-asset/daml/issues/4193
CHANGELOG_END
* Remove unused command service parameter
* fixup protobuf
* Add configuration for TTL
* Fix Haskell bindings
* Rename SQL table
* Add command deduplication test
* Redesign command deduplication queries
* Address review comment
* Address review comment
* Address review comments
* Make command deduplication test optional
* Disable more tests
* Address review comments
* Address review comments
* Refine test
* Address review comments
* scalafmt
* Truncate new table on reset
* Store original command result
* Rename table columns
... to be consistent with other upcoming tables
* Rename migrations to solve conflicts
Fixes#4193.
* Add overridable indexer, api and auth configuration to `LedgerFactory`
CHANGELOG_BEGIN
CHANGELOG_END
* Add overridable indexer and api metrics creation to `LedgerFactory`
CHANGELOG_BEGIN
CHANGELOG_END
* Add overridable api's `TimeServiceBackend` to `LedgerFactory`
* 🎨 Fix formatting
* Port SDK ledgers based on `Runner` (and the sandbox) to `TimeServiceBackend`
* Revert to `TimeProvider` for committer usage and to `None` default for API server.
Also removed now unused `TimeServiceProvider.wallClock()`.
* Move TimeServiceBackend back to the API server.
* 🎨 Remove unneeded argument passed for parameter w/default
* Restore sandbox ledger time support
* Simplify passing a `TimeProvider` to the sandbox ledger
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Context
=======
After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.
This PR is a proposal for a new release process designed under the
following constraints:
- Reuse as much as possible of the existing infrastructure, to minimize
effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
not meant for general public consumption, but can still be used by savvy
users without jumping through too many extra hoops (ideally just
swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
release status, with as few changes as possible in-between, so we can be
confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
releases, to avoid discovering non-transient problems while trying to
promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
keep the same approval process for SOC2 auditability.
The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:
```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```
This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.
Version numbers
===============
Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:
- Semantic versioning (which we chose as the version format for the SDK
version number) allows for "prerelease" version numbers as well as
"metadata"; an example of a complete version string would be
`1.2.3-nightly.201+server12.43`. The "main" part of the version string
always has to have 3 numbers separated by dots; the "prerelease"
(after the `-` but before the `+`) and the "metadata" (after the `+`)
parts are optional and, if present, must consist of one or more segments
separated by dots, where a segment can be either a number or an
alphanumeric string. In terms of ordering, metadata is irrelevant and
any version with a prerelease string is before the corresponding "main"
version string alone. Amongst prereleases, segments are compared in
order with purely numeric ones compared as numbers and mixed ones
compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
`-`, or a transition between a number and a letter. Version strings
are compared element-wise, with numeric segments being compared as
numbers. Alphabetic segments are treated specially if they happen to be
one of a handful of magic words (such as "alpha", "beta" or "snapshot"
for example) which count as "qualifiers"; a version string with a
qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
special ordering amongst qualifiers. Other alphabetic segments are
compared alphabetically and count as being "after" their prefix
(`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
separated by `.`, plus an optional (though deprecated) alphanumeric
"version tag" separated by a `-`. I could not find any official
documentation on ordering for the version tag; numeric segments are
compared as numbers.
- npm uses semantic versioning so that is covered already.
After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:
```
0.13.53-alpha.20200214.3025.ef5d32b7
```
where the components are, respectively:
- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
master.
- `3025`: the number of commits in master up to the release commit
(included). Because we have a linear, append-only master branch, this
uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
not strictly speaking necessary, but makes it a lot more convenient to
identify the commit.
The main downsides of this format are:
1. It is not a valid format for GHC packages. We do not publish GHC
packages from the SDK (so far we have instead opted to release our
Haskell code as separate packages entirely), so this should not be an
issue. However, our SDK version currently leaks to `ghc-pkg` as the
version string for the stdlib (and prim) packages. This PR addresses
that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
would see the above version number as `0.13.53.20200214.3025`, which
should be enough to uniquely identify it. Note that, as far as I could
find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
it makes it more scary. However, I have been told that this may be
long enough to cause issues on Windows by pushing us past the max path
size limitation of that "OS". I suggest we try it and see what
happens.
The upsides are:
- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
they have the same date, allowing us to release same-day patches if
needed. (Note: that is, commits that were made on the same day; the
release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
GHC).
Alternatives I considered:
- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
be accepted by all schemes, but will not sort as expected under semantic
versioning (though Maven will be fine). I have no idea how it will sort
under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
This is not valid semantic versioning and is therefore rejected by
npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
what is generally done in the Java world, but we then lose track of what
version is actually in use and I'm concerned about bug reports. This
would also not let us publish to the main Maven repo (at least not more
than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
version formats. However, it would not clearly indicate to humans that
it is not meant as a stable version, and would sort differently under
semantic versioning (which counts it as a prerelease, i.e. before
`0.13.53`) than under maven (which counts it as a patch, so after
`0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
number of prereleases in-between `0.13.52` and the next. This is
currently the fallback plan if Windows path length causes issues. It
would be less convenient to map releases to commits, but it could still
be done via querying the history of the `LATEST` file.
Release notes
=============
> Note: We have decided not to have release notes for snapshot releases.
Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.
But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.
The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.
> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.
New release process
===================
Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.
Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:
1. Before running the release step, CI will checkout the commit
specified in the LATEST file. This requires separating the release
step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
that gets the version to build from an environment variable, with a
default of `0.0.0` to remain consistent with the current `daml-head`
behaviour.
Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.
The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.
I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Make ReadOnlySqlLedger provide a ResourceOwner, not a Resource.
* sandbox: Fix two race conditions on shutdown in ReadOnlySqlLedger.
It appears that there are two race conditions regarding the ledger end
update mechanism.
1. The dispatcher can keep firing for a little while even after we shut
down the source, which can cause a spurious connection failure as it
makes a query on a closed database connection.
2. We don't wait for the sink to complete, which means, again, we could
shut down the connection before the last `lookupLedgerEnd` query is
issued.
This also makes sure we actually construct a new source if the updates
fail. Previously we were re-using the same source, which looked like a
crash-loop waiting to happen.
Tested by constructing `ReadOnlySqlLedger` and closing it in a loop, and
watching for errors.
CHANGELOG_BEGIN
- [Ledger API Server] Fix a race condition on shutdown in which polling
for the ledger end could continue even after the database connection
was closed.
CHANGELOG_END
* Split Ledger API Test Tool output
Makes failure pop up even without text coloring (e.g. on Azure Pipelines)
CHANGELOG_BEGIN
[DAML Ledger Integration Kit] Ledger API Test Tool now prints errors as a separate section
CHANGELOG_END
* Successes on the right, failures on the left :)
* Add missing newline
* sandbox: Fix a bug in the ResetServiceIT `timedReset` function.
It was computing the start and end times almost simultaneously.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Better error messages in ResetServiceIT if resets are slow.
Let the matchers do their magic.
This moves IndexerIT into its own package, and swaps the dependency from
reference-v2 to ledger-on-memory.
This test should ideally live in the sandbox code, but because it
depends on ledger-on-memory, it's easier to keep it separate.
Also rewrites a lot of the code because the API is different. The tests
should now be clearer too.
I've also marked the test as flaky, because, well, it is.
CHANGELOG_BEGIN
CHANGELOG_END
* Revert "sandbox: Log, explaining the removal of static time and scenarios. (#4582)"
This reverts commit b5cb341e8d.
CHANGELOG_BEGIN
- [Sandbox] Removed the warnings regarding static time and scenarios on
initialization. We will not deprecate these until we have a stable
path forward.
CHANGELOG_END
* Sandbox: Include the log level in output.
* ledger-on-sql: Provide queries in the transaction lambda.
* ledger-on-sql: Split read queries out from write queries.
Can't run write queries from a read transaction.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Pass the connection into the `Queries` constructors.
Way less typing this way round.
* sandbox: If the ledger ID isn't provided, use the one in the database.
Previously, we would fail if working against an existing ledger, and not
explicitly providing the ledger ID. This was the case even if the ledger
ID was randomly generated initially.
CHANGELOG_BEGIN
- [Sandbox] If no ledger ID is provided when running against an existing
ledger, use the existing ID. Previously, Sandbox would fail to start.
CHANGELOG_END
* sandbox: The ReadOnlySqlLedger should always receive a ledger ID.
It's read-only; it can't create one.
* sandbox: Stop using `equal` in SqlLedgerSpec.
* sandbox: Test that the ledger ID is as specified in SqlLedgerSpec.
* sandbox: Let the top-level runner handle a ledger ID mismatch.
And clean up the log text.
* sandbox: Initialize the ledger properly when the ID is dynamic.
* sandbox: Use `Vector`, not `List`, for SqlLedger initialization.
Append with Vector, good. List, bad.
* ledger-api-common: Make `LedgerApiMode.Dynamic` an object.
And add Java-style static factory methods.
* kvutils/app | ledger-on-{memory,sql}: Make `ledgerId` optional.
It should be generated or retrieved from the persistence layer by the
ledger itself.
* kvutils: Make the ledger ID optional in the tests.
* ledger-on-sql: Store the ledger ID, and reject conflicting IDs.
* ledger-on-sql: Make more things final.
* ledger-on-sql: Document the `ledger_meta.table_key` column better.
* sandbox: Don't hardcode the number of packages in the test DAR.
It changes.
* ledger-on-sql: Merge the `head` resource owner with the `dispatcher`.
* sandbox: Use backticks to simplify pattern match in ReadOnlySqlLedger.
* ledger-on-sql: Extract methods in `owner`.
* Push down completion requests to data access layer
This is largely a refactoring. The externally observable behavior is unchanged, but:
- a sub-dao is created for command completions (with the intent of breaking up the dao completely in future commits)
- the command completions dao can, in theory, directly fetch completions off the index
- in practice this is not implemented here to keep this PR as small as possible
Filtering ledger entries to get completions is moved to a function that is in turn used by:
- the ledger dao
- the in-memory sandbox
The plan for the former is to add a new table where completion-relevant data is stored so that it can be fetched quickly.
The plan for the latter is to get rid of it once DAML-on-SQL ships.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix off-by-one error in the in-memory sandbox
* Add type-level strings in DAML.
This PR adds a `PromotedText` stable package, with `PromotedText` type, which is used to encode type-level strings from DAML into DAML-LF. The reason for this is to preserve the `HasField` instance argument. This PR adds a test that `HasField` is succesfully reconstructed incontexts, during data-dependencies, which wasn't possible before.
changelog_begin
changelog_end
* adresss comments
* fix overly specific tests
* Use KeyHasher to serialize contract keys in kvutils
- Use Value instead of VersionedValue in GlobalKey as the versioning does not make sense here
and may be misleading as the a value with a different version but same meaning would still
be the same key.
- Relocate the KeyHasher to ledger-api-common so kvutils can use it (otherwise cyclic dependencies)
- Replace storing of the contract key as a VersionedValue with the hash produced by KeyHasher.
This is backwards incompatible. A compatible option would require us to query the key with both
the old way and the new way which is unattenable. We're making a calculated breaking change.
CHANGELOG_BEGIN
- [DAML Ledger Integration Kit] Serialize contract keys using a hash instead of the value in kvutils.
This is a backwards incompatible change to kvutils.
CHANGELOG_END
* Use proper hasher for contract keys and not KeyHasher
- Use Hash.scala, not KeyHasher.scala.
- Add hash to GlobalKey as we want the hash to be computed from the inside.
The use of KeyHasher will be later deprecated and replaced by this.
* Use "sealed abstract case class" trick instead of private ctor
and rebase fix
* Revert change to unsupported value version decode error
* Reformat code
* Add kvutils changelog entry and bump the version
CHANGELOG_BEGIN
- [Sandbox] Static time mode is being deprecated in the future. A warning has been added to notify users of this fact.
- [Sandbox] Scenarios are being deprecated in the future, in favor of `DAML Script <https://docs.daml.com/daml-script/>`_. A warning has been added to notify users of this fact.
CHANGELOG_END
* kvutils: Extract a committer from the uses of `SubmissionValidator`.
This makes the clock injectable too.
* kvutils: Provide logging contexts in the `Runner`.
* sandbox: Remove the `StaticAllowBackwards` time provider type.
It's not used anywhere.
* sandbox: Fix warnings in CliSpec.
* sandbox: Ensure that we cannot specify both static and wall-clock time.
* sandbox-next: Crash if wall clock time is not specified.
* sandbox-next: Document more known issues in the new Sandbox.
* sandbox: Add a Clock (and some tests) to TimeServiceBackend.
* sandbox-next: Support static time.
CHANGELOG_BEGIN
- [Sandbox Next] Re-establish static time mode.
CHANGELOG_END
* ledger-on-(memory|sql): Expect a `() => Instant`, not a `Clock`.
* Add more tests to default run of Ledger API Test Tool
Furthermore, drop TimeIT, which is a sandbox-only property (and tested already as part of its integration tests)
CHANGELOG_BEGIN
[DAML Ledger Integration Kit] Ledger API Test Tool default tests modified. Use --list for the updated list of default tests. Time service test dropped from the suite.
CHANGELOG_END
* Address https://github.com/digital-asset/daml/pull/4561#discussion_r380635979
* Address https://github.com/digital-asset/daml/pull/4561#discussion_r380636989
* Optimize imports
* Only run semantic tests on Canton
* kvutils/ledger-on-sql: Avoid a race condition in dispatching.
This changes the API of kvutils to allow for passing data out of the
transaction, which makes it much easier to ensure the new head makes it
to the dispatcher. Previously, we would use an `AtomicLong` to
communicate the data, but this was problematic because values could
arrive out in the wrong order. For example:
- log index 5 is committed
- `head` is updated to 6
- the dispatcher is signalled with a head of 6
- log index 6 is committed
- log index 7 is committed
- `head` is updated to 8
- `head` is updated to 7
- the dispatcher is signalled with a head of 7
In this scenario, we would have to wait until a new commit comes in
before the indexer finds out about log index 7.
* kvutils: Just return an `Either`from `SubmissionValidator`.
It was either that or introduce yet another type to split
`SubmissionValidated` into two.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Make ValidationFailed extend NoStackTrace.
* Simplify participant index access for contract lookups
CHANGELOG_BEGIN
[Sandbox] Participant index contract lookups simplified, should speed up command interpretation.
CHANGELOG_END
* Deserialize contracts off of the database connection thread pool
* Return proper code for invalid authentication
CHANGELOG_BEGIN
[Sandbox] If authentication is enabled, requests without a valid
authentication are going to be rejected with an ``UNAUTHENTICATED``
return code instead of ``PERMISSION_DENIED``.
CHANGELOG_END
* Reduce logging noise from java-rxbindings tests
* Fix rxjava bindings tests to match new behavior
* Fix extractor tests to match new behavior
* Address https://github.com/digital-asset/daml/pull/4485#discussion_r378507478
* kvutils: Make it easier to import the Key and Value types.
Moved them into a companion object.
* kvutils: Flatten the nested `for` comprehensions in SubmissionValidator.
* kvutils: Test that results succeed before querying state updates.
If they don't, we get idle timeouts and we don't see the error, which is
unhelpful.
* sandbox: Increase log levels for failed submissions.
* kvutils: Avoid converting DamlStateKey -> ByteString -> Array[Byte].
Just convert once.
* ledger-on-sql: Use the `SubmissionValidator` from kvutils.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Remove unnecessary interpolation in ApiSubmissionService.
* sandbox: Move more resource acquisition into the `owner`.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Reimplement SandboxClientResource as a resources.Resource.
* codegen: Use resources in TestUtil.
* sandbox: Manage PostgreSQL in tests with ResourceOwners.
* Translate unsupported kinds to a special Erased type
This should simplify `data-dependencies` and avoid issues like #4470
since we can match on the type instead of having to guess which types
can and which cannot be translated back to DAML.
changelog_begin
changelog_end
* sandbox: Don't hold on to old resources when resetting.
Now there's one hell of a memory leak.
CHANGELOG_BEGIN
- [Sandbox] Fixed a memory leak when using the ResetService; not
everything was cleaned up correctly.
CHANGELOG_END
* sandbox: Split out SandboxClientResource from SandboxServerResource.
Gonna replace SandboxServerResource with a ResourceOwner acquisition.
* sandbox: Don't capture the API server in the SandboxServer resource.
When we reset, this is stored forever, leading to a memory leak.
Tested by rewriting the SandboxServerResource to use
`SandboxServer.owner`.
* sandbox: Revert the test client resource to calling `shutdownNow()`.
* sandbox: Make sure the fixture is recreated properly on each test run.
* sandbox: Make `SandboxState` a non-case class.
The `toString()` was unnecessarily heavy.
* sandbox: Futures, futures everywhere.
Avoid a race condition where the server is stopped before it starts by
storing a `Future[SandboxState]` rather than the `SandboxState` itself.
This doesn't trigger the same memory leak as storing a
`Resource[SandboxState]` because we don't capture the object itself in
the `flatMap` in the same way with `Future`.
* sandbox: Remove an unused parameter left in for debugging.
* sandbox: Replace `@VisibleForTesting` with a comment.
* sandbox: Add more comments to the weird logic in SandboxServer.
* sandbox: Get rid of the `Port` type alias; it was confusing.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Shuffle participants used in tests
* Add no-wait-for-parties flag to the test tool
This flag is needed for privacy-preserving ledgers that do not expose
party allocations to non-submitting participants.
* Add shuffle-participants flag to test-tool
CHANGELOG_BEGIN
CHANGELOG_END
* Address code review. Make --no-wait-for-parties hidden.
In a committed transaction (i.e. a transaction with only absolute
contract ids), local divulgence is a non-concept.
Therefore local divulgences don't need to be treated and all code
related to it can be removed.
This also removes some duplication between InMemoryLedger and SqlLedger.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix indexing of transactions referencing divulged contracts
When a transaction referenced a divulged contract that had no contract
metadata (e.g. it was only known to the participant via divulgence),
the indexer failed with InactiveDependencyError.
This fixes the issue by modifying lookupContractLet query to return
LetUnknown if the contract is known, but has no metadata.
No changelog entry to add as this bug did not affect any users as this
surfaced with privacy extensions. Canton not affected as they always
include divulged contracts, even if they already have been divulged.
CHANGELOG_BEGIN
CHANGELOG_END
* Failing test-case for lookupContractLet
* Address PR review
* Second pass addressing review
* Prevent building of en/de-coders of Cid, Val, and Nid outside Value/Transaction Coder
* DAML-LF: remove constrains on Cid for LF encoder/decoder
* Address Samir coments
* resources: Allow for startup exceptions without no stack trace.
* sandbox: Remove dependency duplication in the BUILD file.
* ledger-on-(memory|sql): Extract `Main` into its own file tree.
This is so we can remove the dependency on kvutils/app from the library.
logback.xml goes with it.
* sandbox: Implement "Sandbox Next" on top of ledger-on-sql.
CHANGELOG_BEGIN
- [Sandbox] Preliminary work to rebuild Sandbox on top of the DAML
Ledger Integration Kit. Currently not exposed through the CLI.
CHANGELOG_END
* ledger-on-sql: Publish to Maven Central.
* sandbox: If an unknown database is passed to Sandbox Next, handle it.
* resources: Catch errors in owner construction or acquisition.
Sometimes we're not in a future yet.
* resources: Don't run ProgramResource error handling inside the EC.
Otherwise we end up in deadlock which is only broken by
`awaitTermination` timing out.
* sandbox: Fix RecoveringIndexer so it doesn't wait too long.
If the indexer restart delay is too short, RecoveringIndexer will still
wait a second. This is not an issue in production, because the delay is
typically 10 seconds, but in tests, where we use 100 milliseconds, it's
enough to make the test flaky.
We need to wait in one-second increments to allow for quick
cancellation; we don't want the user to have to wait 10 seconds if the
process is terminated while restarting.
The fix is to take the minimum of 1 second and the remaining restart
time.
This also adds STDOUT logging back into the tests; it was invaluable
when debugging the issue.
I am not including a changelog entry, because this only affects tests.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Add an assertion in IndexerIT to ensure it stops quickly.
We already test the logs, but it's probably wise to test that the
indexer doesn't wait 10 seconds.
* kvutils: Add idle timeouts to all participant state tests.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Add nesting to test names in the participant state tests.
And clean up the names.
* kvutils: Use durations in participant state tests.
Makes it way easier to read than `addMicros(…)` calls.
* daml-lf/data: Fix warnings in Time.scala.
* daml-lf/data: Add `Timestamp#add`.
* kvutils: An MRT of (start time + 1 second) is too short for CI.
Increasing the MRT to (now + 10 seconds).
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Add a prefix of "ledger_" to the Flyway migrations table.
And while we're at it, make this a constant in the code and use Flyway
placeholders to substitute it in.
We don't have any users right now, so changing the migrations should be
fine.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run the indexer on PostgreSQL when possible.
Specifically, when running the PostgreSQL conformance tests.
It will also work when running it as a binary, but you need to specify
both the `--server-jdbc-url` and `--jdbc-url` switches. This is to keep
kvutils/app simple.
* ledger-on-sql: Make `ExtraConfig` final.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* resources: Revert `ResourceOwner.failed` to `ResourceOwner[Nothing]`.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* kvutils|ledger: Don't use Scala's global execution context.
I have just discovered that Scala's global execution context uses daemon
threads, which means they don't stick around past the end of the `main`
function. This is bad when application startup fails and we end up with
no other threads; the application terminates before it has a chance to
print the error message, which is very unhelpful.
We resolve this by creating a cached thread pool in the
`ProgramResource` class, specifically used for acquisition and release
of resources.
To do this, we need to pass the execution context through to the
resource constructors, which makes usage a little more verbose, because
now we can't use lambda functions (which can't take implicit arguments
such as the `executionContext`).
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils-app|reference-v2: Sort `Runner` and `ReferenceServer` methods.
This has outlived its usefulness as an experiment in how to implement a
kvutils-based leger, and is now just a source of flaky test runs.
🔥
CHANGELOG_BEGIN
CHANGELOG_END
`App` is fairly problematic. It depends on `DelayedInit`, which is
deprecated because it has some surprising behavior.
Best to avoid it for anything more than a simple demo application.
Details: https://github.com/scala/bug/issues/4330
CHANGELOG_BEGIN
CHANGELOG_END
clean up String Identifier in daml-lf
- Separate LedgerString from ContractIdString
- Drop TransactionIdString from daml-lf
- Create a new ContractIdString (aka. ContractIdStringV1) for new contractId comparable with relative contractIds
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Use `ResourceOwner#use` to simplify tests.
* kvutils: Merge in `RestartSpec` from ledger-on-sql.
CHANGELOG_BEGIN
- [KVUtils] Added a test case to the participant state tests to ensure
your ledger state is resumable upon restart.
CHANGELOG_END
* ledger-on-posix-filesystem: Extract paths into their own class.
* ledger-on-posix-filesystem: Seed the dispatcher with the correct head.
* kvutils: In participant state tests, be clear about different strings.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* sandbox: Move common API and indexer deps into their own packages.
Primarily `com.digitalasset.platform.index.store`.
* sandbox: Move API server deps into the `apiserver` and common packages.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Rename `….platform.index.store` to `….platform.store`.
* sandbox: Split `JdbcIndex` & co. out from `apiserver` into `index`.
Moar packages! And this one is nice and clean; there's only a few
dependencies.
* sandbox: Move `InMemoryPackageStore` into its own `packages` package.
`packages` package. What a tongue twister.
* sandbox: Fix warnings in `TransactionConversion`.
* sandbox: Inline `SandboxContractStore` into `LedgerBackedIndexService`.
* sandbox: Fix warnings in `InMemoryLedger` and `SqlLedger`.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Previously we were only cleaning up and exiting if loading the DAML
packages failed. Now we also handle API server errors.
An example of where this might happen is if the ledger ID specified is
different to the existing ledger ID, causing an error.
One day we'll kill the ResetService and all of this will go away. One
day…
CHANGELOG_BEGIN
- [Sandbox] Fix an error that stops the server from exiting cleanly if
API server initialization fails.
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* kvutils: Use matchers in ParticipantStateIntegrationSpecBase.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: In participant state tests, always test record time.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-posix-filesystem: Run `scalafmt`.
* ledger-on-posix-filesystem: Just use file channel locks. Seriously.
* ledger-on-posix-filesystem: Rename `FileSystemLock#run` to `apply`.
* ledger-on-posix-filesystem: Conformance tests now run quickly.
So we can stop constraining the number of concurrent test runs.
* ledger-on-posix-filesystem: Lock the entire ledger with a file lock.
This ensures we can't have two separate ledger servers running against
the same directory.
This is mostly an experiment to better understand how to do this on an
SQL ledger for #4039.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-posix-filesystem: Run `scalafmt`.
* ledger-on-posix-filesystem: Use `Resource#use` in `FileSystemLock`.
* resources: Move `Resource#use` to `ResourceOwner#use`.
Makes more sense for the same object to govern both `acquire()` and
`release()`.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* resources: Make it easy to use a resource and then release it.
* ledger-on-sql: Move database transaction methods to `Database`.
* ledger-on-sql: Make sure the ledger picks up where it left off.
This means seeding the dispatcher with the correct head, rather than
starting at 1 every time.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Move `Index` and `StartIndex` to the package object.
They're used everywhere.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
The `sandbox-perf` build has been failing for a while with the following
errors:
```
INFO: From Generating benchmark code for //ledger/sandbox-perf:sandbox-perf_codegen:
JMH benchmark generation: JMH Benchmark generator failed
JMH benchmark generation: Benchmark classes should not be final.
[com.digitalasset.platform.sandbox.perf.LargeTransactionBench]
JMH benchmark generation: The instantiated @State class cannot be abstract.
[com.digitalasset.platform.sandbox.perf.PerfBenchState]
```
However, these errors are ignored; running the benchmarks runs the
`AcsBench` benchmark and ignores the fact `LargeTransactionBench` and
`SimpleBench` failed to compile.
I've fixed the errors by making sure that classes are not final, and
that `SimpleBench` uses a concrete state class.
This also introduces a cheap patch to the Scala JMH Bazel rules that
makes sure they fail if there are any errors.
I'm not really sure how to patch the Bazel rules properly, but someone
else might have an idea. :-)
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Unfortunately, downgrading `scalafmt` resulted in files that would be
reformatted to invalid code, because lines would be merged but trailing
commas would be left in.
I've manually run it everywhere and fixed these issues so people don't
have to fix them as they go.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-sql: Don't allow unnamed in-memory databases.
It seems that changing the connection pool size at runtime causes weird
issues, predominantly that sometimes all connections are dropped and
therefore the in-memory database is lost after migration. Obviously, it
then stops working.
This works around this problem by simply not allowing unnamed in-memory
databases, guaranteeing that we can share the database between the
various connection pools.
This will probably be necessary anyway when attempting to share the
database between the ledger and the index.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Make it clear we're generating a new JDBC URL in tests.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-sql: Pull out a superclass for the various integration tests.
* ledger-on-sql: Use Flyway to migrate tables in an idempotent manner.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Make it easy to run it with Bazel for experimentation.
* ledger-on-sql: Test that migrations will never change in the future.
* ledger-on-sql: Add a prefix of "ledger_" to the tables.
This is so we don't accidentally conflict with the index when the
schemas are shared.
I am letting myself modify the migrations because the existing
migrations haven't been merged into `master` yet.
* ledger-on-sql: Explain why we change the pool size after migration.
* ledger-api-test-tool: Tests now specify a timeout scale, not a timeout.
Makes it easier to change the default timeout in one place.
* ledger-api-test-tool: Increase timeouts on slow tests.
These tests produce a lot of volume and can make CI flaky.
* ledger-on-sql: Only tear down PostgreSQL in tests.
For other databases, we just create a new file for each test case.
* ledger-on-sql: Reduce the log output in tests.
* ledger-on-sql: Use a separate connection pool for Flyway when possible.
Apparently `setMaximumPoolSize` doesn't really have the desired effect
after the connection pool has already been used. The new test case will
be flaky if we process more than one commit in parallel.
For SQLite, it seems to be OK.
* ledger-on-sql: Use a separate connection pool for Flyway with SQLite.
Except in memory.
* ledger-on-sql: Use a separate PostgreSQL database for each test.
Because performance, innit. Don't have to tear them down.
* ledger-on-sql: Move the `Queries` values inside the `RDBMS` objects.
* ledger-on-sql: Go into even more detail about pool size hijinks.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
It affects basically every file, so let's do it all at once.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Three problems here:
1. On failure, the Sandbox didn't close down the metrics reporting
thread, meaning the application would hang.
2. If there's a failure:
a. we didn't print it, and
b. we didn't exit with the correct status code.
These are now handled.
CHANGELOG_BEGIN
- [Sandbox] On initialization error, report the error correctly and exit
with a status code of 1. Previously, the program would hang
indefinitely. (This regression was introduced in v0.13.41.)
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* sandbox: Ensure that the Flyway tests actually pick up the SQL files.
* sandbox: Upgrade Flyway from v5 to v6.
CHANGELOG_BEGIN
- [Sandbox] Upgrade the Flyway database migrations library from v5 to v6.
CHANGELOG_END
* Add logging context information to ledger services
Rolls back that AllowedType implicit, which made the logging context
less usable and didn't provide that much of an advantage. Replaced with
helpers where it makes sense that take care of turning the value safely
into a string.
Closes#3699
CHANGELOG_BEGIN
CHANGELOG_END
* Remove unused import
* Address https://github.com/digital-asset/daml/pull/4205#discussion_r370704306
* ledger-on-sql: Don't bother cleaning up after integration tests.
Turns out Bazel cleans up before each test run, so we only have one
set of outputs at a time. This is far more useful for debugging anyway.
* ledger-on-sql: Pull out a test-lib to simplify the conformance tests.
* ledger-on-sql: Turn `Database` from a sealed trait into a case class.
* ledger-on-sql: Support for PostgreSQL!
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run the conformance tests against PostgreSQL.
* ledger-on-sql: Run the LotsOfParties conformance tests.
* ledger-on-sql: Use PostgreSQL's `RETURNING` keyword to save a query.
* ledger-on-sql: Ensure the reader connection pool is read-only.
* ledger-on-sql: Avoid cloning ByteStrings as byte arrays where possible.
Instead, use `ByteString#newInput()`, which returns an `InputStream`.
This is supported by H2 and PostgreSQL, but unfortunately not SQLite.
* ledger-on-sql: Run integration tests in parallel.
* Update the PostgreSQL and SQLite JDBC drivers.
* postgresql-testing: Extract `PostgresAround` from the Sandbox tests.
CHANGELOG_BEGIN
CHANGELOG_END
* postgresql-testing: Split into several files.
* postgresql-testing: Pull out the host and database names into variables.
* postgresql-testing: Don't publish to Maven.
I have no idea why the build wasn't failing before.
* postgresql-testing: Assimilate the `@postgresql_dev_env` dependency.
* kvutils: Use resources in KeyValueParticipantState, not AutoCloseable.
CHANGELOG_BEGIN
- [kvutils] The simplified API now uses ``com.digitalasset.resources``
to manage acquiring and releasing resources instead of ``Closeable``.
CHANGELOG_END
* ledger-on-posix-filesystem: Remove dead code due to a refactoring fail.
* Support DAML-LF type synonyms in scala.
CHANGELOG_BEGIN
CHANGELOG_END
* dont create synonymns in GenerateSimpleDalf
* extend DAML-LF parser to support type synonyms
* test: expand type synonyms correctly
* ledger-api-test-tool-on-canton: Upgrade Canton to v0.7.0.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api-test-tool-on-canton: Use the health endpoint instead of `nc`.
Netty prints a warning when kicking it with `nc -z` from Nix's
netcat-gnu. Not other netcats though… don't know why.
* ledger-api-test-tool-on-canton: Let IntelliJ format the shell script.
* ledger-api-test-tool-on-canton: Stop Canton with SIGINT.
For some reason, Canton doesn't print out a load of errors if it's
stopped with SIGINT (Ctrl+C). SIGTERM, on the other hand…
* ledger-api-test-tool-on-canton: Write Canton logs to STDOUT.
Now the logs are no longer noisy, we can just pump them to STDOUT. This
should make debugging any transient issues on CI a lot easier.
* libs-scala: Make sure `RetryStrategy` works if `attempts.isEmpty`.
* ledger-on-posix-filesystem: Only retry on expected lock errors.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-posix-filesystem: Only create the lock attempt directory once.
Soooo much faster.
* ledger-on-posix-filesystem: Don't run the unit tests on Windows.
* kvutils: Make logback.xml a base file, rather than the only option.
* kvutils/app: Simplify logback.base.xml.
Mostly by getting rid of unnecessary appenders.
* ledger-on-sql: Add trace logging for all database work.
* Upgrade H2; there's a few useful bug fixes.
* kvutils/app: Let the user override the server JDBC URL.
* kvutils/app: Provide a way to specify the ledger ID.
* ledger-on-sql: If there are missing entries in the log, fail on read.
This can happen right now because we insert in parallel. Next step: stop
doing that.
* ledger-on-sql: Stop writing in parallel; it causes race conditions.
This unfortunately means we also stop _reading_ in parallel, which is
less fun.
* Revert "ledger-on-sql: Use a sequential log entry ID."
This reverts commit c58265bf43.
* ledger-on-sql: Create tables with columns that are not nullable.
* ledger-on-sql: The H2 conformance tests now work, just slowly.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run tests against H2 and SQLite on memory and disk.
* ledger-on-sql: Allow H2 to read from the log in parallel with writes.
* ledger-on-sql: Use UUIDs for entry IDs, not 8 random bytes.
* ledger-on-sql: Make sure to log the correlation ID.
* ledger-on-sql: Do less while holding a database connection.
* ledger-on-sql: Log the connection.
* ledger-on-sql: We don't need to tell H2 not to drop the connection.
The connection pool takes care of that.
* ledger-on-sql: Disable H2 conformance tests on CI; they're too slow.
* ledger-on-sql: Rename `loggingContext` to `logCtx`.
* ledger-on-sql: Don't abuse log contexts; put the data in the message.
* ledger-on-sql: Make the connection log line easier to read.
And pull out the logger, even if it's only used once.
* ledger-on-sql: Run the unit tests against (H2, SQLite) * (memory, file).
* engine: the natural order in Transaction is pre-order traversal
- Do not use anymore SortedMap for nodes in transaction
- Drop unused transaction traversal
CHANGELOG_BEGIN
CHANGELOG_END
* daml-lg: improve generator of transactions
* Introduce new JWT payload format
... the reader still supports old formats
CHANGELOG_BEGIN
[Sandbox] The sandbox uses a new payload format for authentication tokens (JWTs).
The old format is deprecated, but still works.
[JSON API] The HTTP JSON API now uses the same payload format for authentication tokens as the sandbox.
The old format is deprecated, but still works.
CHANGELOG_END
* Add helper function for getting token party
* Support sandbox tokens in JSON API
* Add warning for deprecated formats
* Update documentation
* Add explicit test for new format
* Update JSON API documentation
* Fix test
CHANGELOG_BEGIN
[Sandbox] Metrics are now namespaced by "daml" and their names have been
standardize to snake_case
CHANGELOG_END
Please note that HikariJDBC records its own metrics but this is outside
of our reach, so connection-pool-related metrics will keep being called
with camelCase names.
Prohibit contract IDs in contract keys and add key maintainers to exercises
CHANGELOG_BEGIN
- [DAML-LF] Prohibit contract IDs in contract keys completely. Previously, creating keys containing absolute (but not relative) contract IDs was allowed, but `lookupByKey` on such a key would crash.
CHANGELOG_END
Co-authored-by: Remy <remy.haemmerle@daml.com>
Co-authored-by: Stephen Compall <scompall@nocandysw.com>
This was accidentally set to 1.7 before for no reason. Note that this
simply meant that we did not use the stable module when compiling with
--target=1.6 since we filter out newer stable packages.
Nevertheless, I’ve also added a test that verifies that we do not
accidentally introduce dependencies on packages in newer LF versions.
changelog_begin
changelog_end
* sandbox: Tidy up the RecoveringIndexerSpec a little.
* sandbox: Move the JdbcLedgerDao owner constructor to its object.
* sandbox/damlonx: Make sure the RecoveringIndexer terminates.
Previously, if there was a failure and a termination at the same time,
we could recover after terminating, which means we'd end up in a weird
state. This makes sure we stop correctly.
Unfortunately, I couldn't find a good place to put these tests apart
from in reference-v2, as they need to depend on both _kvutils_ and
_sandbox_. Perhaps they belong in their own Bazel package.
This might address #3945; it's hard to say.
CHANGELOG_BEGIN
- [Indexer] Potentially fix a bug when recovering from failure.
CHANGELOG_END
* reference-v2: Address a race condition in the IndexerIT assertions.
* sandbox: Make sure the RecoveringIndexer can stop during a restart.
* reference-v2: Use Mockito to simulate errors in IndexerIT.
* sandbox: When the RecoveringIndexer is stopped, stop quickly.
* reference-v2: Make sure each test has its own metrics registry.
* Replace TestNamedLoggerFactory with in-memory appender
* sandbox: Test RecoveringIndexerSpec with a mock logger.
Using a real logger was causing massive stack traces in all other tests
that didn't have `RecoveringIndexerSpec$Appender` available, and adding
the appender programmatically didn't work for some reason.
The mock gives much nicer error messages on failure, too.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* libs-scala/resources: Move `Resource` and friends to their own package.
CHANGELOG_BEGIN
- [Ledger API Server] Publish the resource management code as a library
under ``com.digitalasset:resources``.
CHANGELOG_END
* libs-scala/resources: Add to the artifacts list.
* libs-scala/resources: Factor out the test helpers into test/lib.
* libs-scala/resources-akka: Make the Akka resources an optional dep.
* EC256 rather than EC512 algo
CHANGELOG_BEGIN
[Ledger API Authorization]Support EC256 algorithm for JWT rather than EC512
CHANGELOG_END
* support both 256 and 512 ECDSA algo
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* scalafmt
* correct comment, avoid unnecessary private def
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* correct BAZEL formatter
* Introduce ContextualizedLogger and LoggingContext
A ContextualizedLogger is a wrapper around an Slf4j logger. It uses
call-by-name parameters to only construct logging lines when necessary.
The underlying context can be used by invoking withoutContext.
More interestingly, every call to one of its logging methods requires
the presence of an implicit LoggingContext.
A LoggingContext is a way of definining a set of pairs that a
ContextualizedLogger can log without the need of it being specified for
every call.
A new context can be created with newLoggingContext and, given an
implicit LoggingContext, pairs can be added or overwritten using
withEnrichedLoggingContext.
Pairs in the context will be appended to each event logged by a
ContextualizedLogger in the following form:
some text logged explicitly (context: {key1=value1, key2=value2)
If the underlying Logger is configured to use the Logstash encoder the
keys and values in the LoggingContext will also be added to the logging
event encoded in JSON format.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestion in https://github.com/digital-asset/daml/pull/4046#pullrequestreview-343224692
Thanks to @SamirTalwar
Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>
* Address https://github.com/digital-asset/daml/pull/4046#discussion_r366874330
Co-authored-by: Samir Talwar <samir@noodlesandwich.com>
* ledger-on-sql: Implement a ledger using the simplified kvutils on JDBC.
This tests against H2; PostgreSQL coming soon.
No changelog because we're not shipping this yet.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/app: Extract ledger construction into a trait.
* ledger-on-sql: Run the conformance tests against H2.
* ledger-on-sql: Use H2-specific queries to perform actions atomically.
* ledger-on-sql: Extract queries to `Queries` and `H2Queries`.
* ledger-on-sql: SQLite can't have more than one connection at a time.
* ledger-on-sql: Run conformance tests against SQLite.
* ledger-on-sql: Fix error message if the JDBC URL is missing.
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* kvutils/app: Rename LedgerConstructor to LedgerFactory.
* ledger-on-sql: Don't release connections early.
* ledger-on-sql: Rename `withConnection` so it's more meaningful.
* ledger-on-sql: Use a sequential log entry ID.
* ledger-on-sql: Disable H2 conformance tests.
Until we can figure out why some fail.
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
Fixes#3984.
CHANGELOG_BEGIN
[Sandbox] Fix bug that can cause the transaction stream to not terminate.
See `issue #3984 <https://github.com/digital-asset/daml/issues/3984>`__.
CHANGELOG_END
The bug is that the ledger end gets updated without writing any ledger entries,
for example when uploading packages or in case of duplicate commands.
The termination of the stream (i.e. the comparison of offsets) only happened
on actual ledger entries. If there are none, then the stream just waits for the next
entry to be written.
This change moves the stream termination/completion to BEFORE potential entries
are loaded. The stream knows that after a certain offset has been reached, that it can
be terminated/completed.
* kvutils, etc: Remove duplicate copyright notices.
* ledger-on-posix-filesystem: Implement a simplified KV ledger on FS.
CHANGELOG_BEGIN
- A simple KVUtils-based ledger that runs against a POSIX filesystem.
Intended as a reference implementation for testing and documentation.
CHANGELOG_END
* ledger-on-posix-filesystem: Make it runnable.
* ledger-on-posix-filesystem: Chunk long state paths for macOS.
macOS only supports file names of up to 255 characters.
* ledger-on-posix-filesystem: Run conformance tests.
The lock is temporarily implemented as an in-memory mutex.
* ledger-on-posix-filesystem: Move the paths into the class.
* ledger-on-posix-filesystem: Lock using the file system.
We use the movement of a directory as an atomic operation.
* ledger-on-posix-filesystem: Use a sequential log entry ID.
* ledger-on-posix-filesystem: Check exception types using patterns.
* ledger-on-posix-filesystem: Test that state key collisions can't happen.
* ledger-on-posix-filesystem: Document the various directories.
* ledger-on-posix-filesystem: Use a *persistent* sequential log entry ID.
Because this is a persistent store. Duh.
* ledger-on-posix-filesystem: Don't worry about using Futures.
It's a global lock anyway.
* Factor out common values
* Validate LookupByKey nodes
Fixes#3543
CHANGELOG_BEGIN
- [Sandbox] Validate LookupByKey transaction nodes.
See `issue #3543 <https://github.com/digital-asset/daml/issues/3543>`_.
CHANGELOG_END
* Add failing tests
* LookupByKey validation should not respect privacy
* lookupByKey should use stakeholders, not divulgees
Fixes#2311
CHANGELOG_BEGIN
- [Ledger API] lookupByKey now requires the submitter to be a stakeholder on the referenced contract.
See `issue #2311 <https://github.com/digital-asset/daml/issues/2311>`_.
CHANGELOG_END
* Adapt tests to new behavior
Our users seem to stumble over this problem over and over again. Since there's
no technical reason to ban `submitMustFail` during sandbix initialization, we
can also allow for it.
CHANGELOG_BEGIN
- [Sandbox] Allow ``submitMustFail`` in scenarios used for sandbox initialization.
CHANGELOG_END
* Initial version of simplified transaction reader/writer API.
* Code tidying.
* Ran ./fmt.sh
* Added verification of pre-declared vs. actual state updates to example.
* kvutils: Run `scalafmt`.
* kvutils: Move ParticipantStateIntegrationSpecBase to src/test/lib.
* kvutils: Clean up warnings in ParticipantStateIntegrationSpecBase.
* ledger-on-memory: Move InMemoryLedgerReaderWriter to its own package.
* ledger-on-memory: Add a Main class with a simple configuration.
* reference-v2/ledger-on-memory: Wait for DARs to upload.
* ledger-on-memory: Run conformance tests.
* kvutils/app: Extract the in-memory ledger `Main` class into a Runner.
* reference-v2: Move InMemoryKVParticipantState out of kvutils.
It's not used elsewhere.
* kvutils/app: Fix some of the config help text.
* kvutils/app: Don't register the archive files twice.
CHANGELOG_BEGIN
- [kvutils/app] A simple way to run KVUtils-based participants.
- [ledger-in-memory] Split out the simple in-memory ledger, previously
part of kvutils, into its own JAR.
CHANGELOG_END
* Revert "kvutils/app: Don't register the archive files twice."
This reverts commit 2489ae6964.
Turns out we need this for preloading, which is a useful optimization.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
CHANGELOG_BEGIN
[Sandbox] Fix the Sandbox command line help text to reflect the recent
change that the default address is ``localhost`` (``127.0.0.1`` or
``::1``)
CHANGELOG_END
This is done to restrict the usage of NamedLoggerFactory and most usages of MetricsRegistry
to a single Bazel package, thus allowing a possible replacement to not have to become a
dependency of multiple packages.
CHANGELOG_BEGIN
CHANGELOG_END
* ECDA512 algorithm support
* ECDA512
* happy day test for ECDA512 algorithm
* failure test for wrong key for ECDA512 algorithm
* add ability to use EC cert file
* update docs
* scalafmt
* Correct documentation
CHANGELOG_BEGIN
[Ledger API Authorization] Support elliptic curve algorithm for JWT verification
CHANGELOG_END
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* correct docs warning
Currently there is a constraint on the postgres `ledger_entries` table
ensuring that submitter, application_id and command_id are set on
transactions being persisted. These fields are coming from the submitter
info that will be empty when the transaction accepted event is received
on a participant which is not hosting the submitter.
This change updates the check to ensure that all of these fields are
set OR none of them are.
The same change has been made to the H2 migrations with the addition of
correcting the `is not null` check (was previously using `!= null`).
For postgres, as the check is just loosening the existing constraint of all fields
requiring a value I think there is no risk that the new constraint will fail
to apply to existing data. For H2 as the original constraint was
incorrect there is a possibility that existing data may be incorrect -
however as the same component is responsible for writing to H2 and postgres
I will assume the data was correctly persisted regardless of the constraint.
Plus as H2 is typically only used for ephemeral tests, there is
likely no existing data to migrate.
CHANGELOG_BEGIN
- [Sandbox] Loosens database schema to allow persistence of transaction
ledger entries where no submitter info is present (typically when the
submitter is hosted by another participant node).
CHANGELOG_END
