* Move public code into daml-integration-api
CHANGELOG_BEGIN
[DAML Integration Kit]: Removed sandbox specific code from the API intended to be used by ledger integrations. Use the maven coordinates ``com.daml:participant-integration-api:VERSION`` instead of ``com.daml:ledger-api-server`` or ``com.daml:sandbox``.
CHANGELOG_END
Also fixes#5635, removing usage of `ReadService` from the `StandaloneApiServer`.
The configuration stream of the Ledger API LedgerConfigurationService is now properly backed by the configuration entries instead of just serving the initial configuration.
CHANGELOG_BEGIN
[DAML Integration Kit]: ``StandaloneApiServer`` can now be run in a read-only mode.
- The type of the constructor parameter ``writeService`` of ``StandaloneApiServer`` changed to ``Option[WriteService]``. Passing ``None`` will not start any of the admin services, the command service, and the command submission service.
- The constructor parameter ``readService`` of ``StandaloneApiServer`` has been removed.
- A new constructor parameter ``ledgerId`` has been added to ``StandaloneApiServer``. It is used to verify that that ``StandaloneApiServer`` is run against an index storage for the same ledgerId. Initialization is aborted if this is not the case.
[DAML Integration Kit]: The ``LedgerConfigurationService`` now properly streams configuration changes.
CHANGELOG_END
* fixing flaky test, WIP
* introducing `storeSync`
to guarantee that transactions get processed in the specified order
removing unnecessary assertion that hid the fact that the order of offsets
read from DB was invalid
* cleaning up
changelog_begin
changelog_end
* addressing code review comments, thanks @S11001001!
* suspend it
* cleaning up a bit more
SubmittedTransaction and CommittedTransaction are moved from com.daml.lf.transaction.Transaction to
com.daml.lf.transaction
This helps intelliJ type inference.
CHANGELOG_BEGIN
CHANGELOG_END
Try to match transactions from the ledger export to a different non original dar.
This is useful to benchmark a different compilation of the same original daml source.
CHANGELOG_BEGIN
CHANGELOG_END
* Added metrics for decoding and total time of pre-execution.
CHANGELOG_BEGIN
CHANGELOG_END
* Code tidying.
* Added timer for tracking time spent with generating write sets.
* Removed engine constrcutor parameter.
* Basic abstractions for pre-executing submission validator.
* Fix build.
* Shared byteToLogEntryId with BatchedSubmissionValidator.
* Added some more metrics.
* Removed involved participants from PreExecutionResult.
CHANGELOG_BEGIN
CHANGELOG_END
* Reformatted/added headers.
* Fix build.
* Code tidying.
* Moved PreExecutionOutput into a separate file.
* Moved type definition into companion object.
* Added some tests for PreExecutingSubmissionValidator.
* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/validator/preexecution/LedgerStateReaderWithFingerprints.scala
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
* Removed left-in code.
* Added some assertions for .zip operations.
* Test result for unexpected message.
* Test setting min/max record time.
* Reuse code for generating a key-value pair for log entries.
* Tests for CachingDamlLedgerStateReaderWithFingerprints.
* Store empty values as well with their fingerprints.
* Moved weight function into companion object.
* Test that the results come back in right order.
* Code tidying.
* Basic test case for LogAppenderPreExecutingCommitStrategy. Added missing headers.
* Code tidying.
* Fix build.
* Fix build.
* Revert "Store empty values as well with their fingerprints."
This reverts commit ca68ea967f.
# Conflicts:
# ledger/participant-state/kvutils/src/test/suite/scala/com/daml/ledger/validator/caching/CachingDamlLedgerStateReaderWithFingerprintsSpec.scala
* Added some docs.
* Added test case for ensuring that we don't cache None values.
* Fix build.
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
* adding a test case for the offset order returning expected transactions,
moving the exiting test case under flat transactions behavior group
* Fixing event_offset to event_sequential_id conversion
handling the conversion on an empty ledger,
adding a bit more logging,
skipping DB query on an empty range
CHANGELOG_BEGIN
CHANGELOG_END
* addressing code review comments
simplifying the query
returning SQL that forces the event_offset index usage
* addressing code review comments
simplifying the query
returning SQL that forces the event_offset index usage
* simplifying it a bit more
[Sandbox][Ledger Integration Kit]: Metrics can now be exported to graphite with a prefix using the format ``--metrics-reporter=graphite://host:port/prefix``.
CHANGELOG_BEGIN
CHANGELOG_END
* Removed unused code.
* Basic interface draft for producing pre-execution results.
* Share code for running all steps for a committer.
* Code tidying.
* Removed type parameter Submission from Committer.
* Simplify creation and using of committers.
* ExecuteSubmission => SubmissionExecutor
* Code tidying.
* Code tidying.
CHANGELOG_BEGIN
CHANGELOG_END
* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/participant/state/kvutils/committer/Committer.scala
Co-authored-by: Andreas Lochbihler <36634420+andreaslochbihler-da@users.noreply.github.com>
* Fixed build.
* Basic test cases for dryRun.
* Set to-be-notified participants to be everyone.
* Some tests for runSteps.
* Fixing merge.
* Organized imports.
* Added separate timer for pre-execution.
* Set min/max record time from context.
* preexecution => preExecution
* Code tidying.
* Code tidying.
* Set out-of-time-bounds log entry from context.
* Code tidying.
* Code tidying.
* Keep track of deduplication time window for transactions.
* Set deduplicateUntil in the generated log entry.
* Made min/max record time optional in pre-execution result.
Co-authored-by: Andreas Lochbihler <36634420+andreaslochbihler-da@users.noreply.github.com>
* Fix fine-grained exclusions in ledger-api-test-tool
31995ee000 accidentally changed
`allTestCaseNames` to be a list of test suite names instead of a list
of test case names. That breaks fine-grained exclusions that we use in
our compatibility tests. I’ve also added a very basic test that both
the coarse-grained and fine-grained syntax is supported.
changelog_begin
changelog_end
* Update ledger/ledger-api-test-tool/BUILD.bazel
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Ledger API Test Tool: deprecate unneeded tests and options
Furthermore, now all active tests are now run be default without needing
to specify --all-tests. The scheduler makes sure to run non-isolated
tests last (and sequentially).
Fixes#3747Fixes#6518
A bit of historical context: tests in the Ledger API Test Tool used to
be the sandbox integration tests. As part of the Ledger API Test Tool
project, those tests have been ported one-to-one, with little review of
those same tests.
Examining the tests that cause #3747, it became evident that those tests
were originally put in place to verify that an isolated sandbox spun up
for a single test could withstand a number of commands slightly lower
than what at the time was the hard-coded back-pressure threshold. Hence,
implementing back-pressure on the Ledger API Test Tool would have
basically negated the usefulness of those tests. Furthermore, those
tests can be easily passed by simply raising the back-pressure
threshold. As such, they do not convey any meaningful information for
the DAML ledger implementor. This means we are retiring those tests. The
tests will be nominally kept in for a deprecation period to not break
existing build scripts, but running them explicitly will simply show
them as skipped. The documentation has been updated with the information
necessary to users to deal with retired tests.
Since all tests are now run by default --all-tests is being deprecated.
--load-scale-factor is also deprecated as it was used only by
TransactionScaleIT.
changelog_begin
[Ledger API Test Tool] The LotsOfPartiesIT and TransactionScaleIT test
suite have been deemed not providing relevant signal to DAML ledger
implementers and have been retired. The tests will be nominally kept in
but will be skipped while they are in a deprecation period. You are
advised to remove explicit references to those tests before they are
fully removed.
[Ledger API Test Tool] All tests are now run by default. The --all-tests
option is now ineffective and deprecated. You are advised to remove its
usages from your build scripts. Non-isolated tests that could affect the
global state of the ledger and interfere with other tests are now
automatically scheduled by the test tool to run sequentially at the end
of the run.
[Ledger API Test Tool] The --load-scale-factor option is now unused and
deprecated. You are advised to remove its usages from your build
scripts.
changelog_end
* Add test for deprecated CLI options
* Address https://github.com/digital-asset/daml/pull/6651#discussion_r452100158
* Address https://github.com/digital-asset/daml/pull/6651#discussion_r452126405
* Address https://github.com/digital-asset/daml/pull/6651#discussion_r452143811
* Address test failures
* sandbox: Move helpers in EmptyLedgerIT to a companion object.
* sandbox: Accept a new time model when none is set.
CHANGELOG_BEGIN
- [Ledger API Server + (Sandbox / Ledger Integration Kit)]
Accept a new time model if none is set. Previously, it would
erroneously be rejected because the generation number submitted to
was incorrectly set to `2` rather than `1`.
This would not affect most users of Sandbox or other kvutils-based
ledgers, as if a configuration is set automatically on startup when
creating a new ledger. It only affects users who explicitly override
the initial ledger configuration submit delay to something longer than
a few milliseconds.
CHANGELOG_END
* sandbox: Ensure that completions can be streamed from the beginning.
* sandbox: In EmptyLedgerIT, use the submission service.
* sandbox: Split EmptyLedgerIT into two.
* sandbox: In CompletionServiceWithEmptyLedgerIT, assume CI will be slow.
In other words, don't create a completion stream and expect a submission
to show up in a reasonable amount of time.
* row_id changes
* fixing inserts
* replacing offset with row_id in the flat transaction stream queries
* fixing flat transaction query, updating H2 migration script
* fixing formatting
* ACS query pagination relies on row_id instead of ledger offset
* give a name to the index that we have to drop
* give a name to the index
* Fixing events range query it can return SQL nulls on empty DB.
* remove the debug println
* remove outdated comment
* removing unused orderByColumns constant
* getting rid of new `Source.flatMapConcat` calls that were added as part of this PR.
CHANGELOG_BEGIN
1. ACS, Flat Transaction and Transaction Tree stream pagination based on event_sequential_id instead of event_offset.
2. Events ordering based on the order of insertion: order by event_sequential_id instead of order by (event_offset, transaction_id, node_index).
CHANGELOG_END
* reverting changes to V13 H2 migration script,
figuring out the name of the index that has to be dropped
* Addressing code review comments:
- replacing scalaz Option.cata with stdlib Option.fold
- moving implicit val def into import
* Addressing code review comments:
- extracting re-usable stream query functions
* forcing postgres to use index when looking up lower and upper bound row ids
* fixing the query when it is run on an empty ledger
* resolving rebase conflicts
* Update ledger/sandbox/src/main/scala/com/digitalasset/platform/store/dao/events/EventsRange.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* fetching a single row, fetchSize should NOT matter
* Adding integration test to reproduce invalid order of archived, created events
The test fails, which is expected.
* Fixing the order of archived, created events triggered by exercise
* Addressing code review comments and cleaning up
* Renaming row_id to event_sequential_id
* Investigating flaky tests
* Fixing formatting
* Revert HOTFIX-flaky-client-server changes
`bazel test --runs_per_test=50 //ledger/participant-state/kvutils:reference-ledger-dump` passed on this branch.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Return the ledger beginning as the ledger end on an empty ledger
Previously, the services would return `Offset.beforeBegin` to the
clients, and the other services deem this offset to be invalid.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Import DurationInt
* Kill unused import
* Use ApiOffset for the external ledger begin
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* replace traverseU and sequenceU with traverse and sequence
- with -Ypartial-unification on, the extra Unapply typeclass lookup is
unnecessary
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* limit imports; we only need *> and void
Buildifier now comes with a handy attachment to catch single `\`
characters inside strings and replace them with `\\` if the escape
sequence is invalid. Skylark/Python will do this at runtime anyway; this
just makes it clearer what the actual behavior is.
I needed to change `\` characters at the end of lines to `\\` manually
in order to stop Buildifier from simply concatenating the lines
together. Everything else was automatic.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/integrity-check: Always reset the output color.
Contrary to popular belief, terminal colors are hard. Resetting to
"white" isn't good enough; you don't know which color the user's
terminal should be set to. We need to print `AnsiColor.RESET` to ensure
the terminal is properly reset.
This changes the way we print so we _always_ set the color and then
reset it in one operation, making it impossible to accidentally leave
the terminal using a specific color. It also fixes the above bug as a
side effect.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/integrity-check: Print the total outside `go`.
* kvutils/integrity-check: Push the error outwards.
I'd rather avoid calling `sys.exit` inside a loop if I can.
* kvutils/integrity-check: Implicits for colors.
Before:
```scala
Print.green("Hello!")
```
After:
```scala
println("Hello!".green)
```
* kvutils/integrity-check: When there's an exception, crash.
The integrity checker would previously hang around for a long time in
the event of an exception. This makes sure all threads are shut down
immediately.
* kvutils/integrity-check: Increase the timeout per submission.
Larger submissions time out on my machine.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/integrity-check: Rewrite without `Await`.
* kvutils/integrity-check: Close the file after running.
small clean up in the testing library TransactionBuilder, bsaically add a Submitted and a Committed version for empty transaction constant and build method.
plus a bit of cleaning in tests using TransactionBuilder
CHANGELOG_BEGIN
CHANGELOG_END
* Removed archived_by_entry from CommitContext and DamlContractState.
* Removed references to getEntryId from test code.
* Do not pass log entry ID to committers.
* Log entry ID is no longer required for computing submission outputs.
* Log entry ID is no longer required for computing submission outputs.
* Code tidying.
CHANGELOG_BEGIN
CHANGELOG_END
Since we have only absolute contract ids, there is no more distinction
between local and global disclosure/divulgence. Let's please remove the
`global` prefix since it causes confusion (at least for me it did).
CHANGELOG_BEGIN
CHANGELOG_END
* `CommitContext` records accessed input keys
CHANGELOG_BEGIN
CHANGELOG_END
* Simplify state and tests for `accessedInputKeys`.
* Add note about stable VS deterministic iteration order
* Add test about not recording input keys that are not accessed.
* Use regular a regular set for accessed keys as nor iteration stability nor determinism are needed
* Measure time of interpretation and store it in CommandExecutionResult.
* Added parameters for passing in interpretation time for WriteService and LedgerWriter.
* Code tidying.
* Added CommitMetadata parameter to LedgerWriter.
* Thread through interpretation time. Aggregate interpretation times for a batch.
* Take max of interpretation times.
* Calculate interpretation time in nanos.
* Moved CommitMetadata into separate file.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Delegate deprecated methods to method with new signature.
* Code tidying.
* Suppress deprecation warnings.
* Made interpretation cost optional in CommitMetadata.
* Test that we populate interpretation time.
* Code tidying.
* Apply suggestions from code review
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Code tidying.
* Some more tests. Do not return interpretation cost for a batch if it only contains non-transaction submissions.
* Reformatted.
* Reformatted.
* Named arbitrary interpretation cost.
* Reverted changes for BatchingLedgerWriter.
* Always drop commit metadata for batches and don't report interpretation cost.
* More specific expectations.
* Include authorization check and blinding in interpretation time.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Use Distroless for the Java Docker base image.
We switched away from Distroless because it was causing issues with
`docker pull` when you had Docker configured to use `gcloud` for
authentication, but weren't actually authenticated.
Adding `docker-credential-gcloud` to dev-env should hopefully fix this,
meaning we can switch back to a base image that is better-maintained.
CHANGELOG_BEGIN
CHANGELOG_END
* Bump rules_docker to v0.14.3.
This fixes an issue when running `bazel sync`:
```
ERROR: java.io.IOException: Error downloading [http://central.maven.org/maven2/javax/servlet/javax.servlet-api/3.0.1/javax.servlet-api-3.0.1.jar] to [...]/external/javax_servlet_api/javax.servlet-api-3.0.1.jar: Unknown host: central.maven.org
```
