The current implementation of generating stack traces ignores the
`cause` of any exception, which means the actual error tends to be
masked. This changes the logic to rely on `Throwable::printStackTrace`,
which correctly prints the chain of causes too.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-common: Add tests for parsing the metrics reporter CLI args.
* participant-integration-api: Remove unused MetricsReporter stuff.
* sandbox-common: More rigorous parsing for metrics reporters.
This avoids the issue where a previously-valid value such as
"graphite:server:1234" fails with a cryptic error message:
"hostname can't be null".
CHANGELOG_BEGIN
- [Sandbox] Improved the error message when providing an invalid metric
reporter as a command line argument. Now the error message always
shows the correct syntax.
CHANGELOG_END
* participant-integration-api: Add cause to metrics reporter parse error.
Right now the error is logged, but the program stalls, because the
exception is outside any `Future`, and so the cleanup never happens.
CHANGELOG_BEGIN
CHANGELOG_END
This PR simplifies the deprecated `getScenarioLedger` to take the
whole engine instead of `compiledPackages` (from the engine) and the
rest of the `outputTransactionVersions` (form the engine conffig).
This prepares for some changes for #5164, where more options for the
engine config will have to be used by the method. Also we will need to
be sure the `compiledPackages` following the option of the engine
config.
CHANGELOG_BEGIN
CHANGELOG_END
* Use case class for grouping components needed for replay for a given commit strategy.
CHANGELOG_BEGIN
CHANGELOG_END
* Update ledger/participant-state/kvutils/tools/src/main/scala/com/daml/ledger/participant/state/kvutils/tools/export/CommitStrategySupport.scala
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Reformatted.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-memory: Split the tests into a test suite.
The ledger-on-memory tests are sporadically failing on CI because they
hit the timeout of 60s, which is the default for a "small" test target.
This is not outside the realm of possibility, as tests can take a while
on CI when lots of other things are running.
To mitigate this, rather than increasing the timeout, I am splitting the
tests up into a suite, which means they'll run as 3 targets, not 1, each
with a 60s timeout. This means they're far more likely to complete in
the allocated time.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-memory: Don't implicitly widen numerics.
Not sure why this breaks on CI and not on my machine, but 🤷🏽♂️.
* ledger-on-sql: When failing to acquire a connection, specify where.
* kvutils: Pass the execution context through the ValidatingCommitter.
The SubmissionValidator shouldn't be getting it through the constructor,
but this will do for now.
* kvutils: Remove the executionContext from BatchingLedgerStateOperations.
* ledger-on-sql: Inject the committer.
* ledger-on-sql: Construct a single-threaded executor for the committer.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Trailing commas are the best kind of commas.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* add -Ywarn-unused to all scalac options
* remove some unused arguments
* remove some unused definitions
* remove some unused variable names
* suppress some unused variable names
* changeExtension doesn't use baseName
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* work around no plugins in scenario interpreter perf tests
* remove many more unused things
* remove more unused things, restore some used things
* remove more unused things, restore a couple signature mistakes
* missed import
* unused argument
* remove more unused loggingContexts
* some unused code in triggers
* some unused code in sandbox and kvutils
* some unused code in repl-service and daml-script
* some unused code in bindings-rxjava tests
* some unused code in triggers runner
* more comments on silent usages
- suggested by @cocreature; thanks
* fix missing reference in TestCommands
* more unused in triggers
* more unused in sandbox
* more unused in daml-script
* more unused in ledger-client tests
* more unused in triggers
* more unused in kvutils
* more unused in daml-script
* more unused in sandbox
* remove unused in ledger-api-test-tool
* suppress final special case for codegen unused warnings
.../com/daml/sample/mymain/ContractIdNT.scala:24: warning: parameter value ev 0 in method ContractIdNT Value is never used
implicit def `ContractIdNT Value`[a_a1dk](implicit `ev 0`: ` lfdomainapi`.Value[a_a1dk]): ` lfdomainapi`.Value[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
^
.../com/daml/sample/mymain/ContractIdNT.scala:41: warning: parameter value eva_a1dk in method ContractIdNT LfEncodable is never used
implicit def `ContractIdNT LfEncodable`[a_a1dk](implicit eva_a1dk: ` lfdomainapi`.encoding.LfEncodable[a_a1dk]): ` lfdomainapi`.encoding.LfEncodable[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
^
* one more unused in daml-script
* special scaladoc rules may need silencer, too
* unused in compatibility/sandbox-migration
* more commas, a different way to `find`
- suggested by @remyhaemmerle-da; thanks
* remove unused definitions, params, args from sandbox Scala code
CHANGELOG_BEGIN
CHANGELOG_END
* remove unused loggingContext from sandbox
* pass pageSize along in JdbcLedgerDaoTransactionsSpec
- seems to have been the intent of the parameter, and at the moment it
is semantically identical
* remove unused definitions, params, args from kvutils Scala code
CHANGELOG_BEGIN
CHANGELOG_END
* label desired default for enclose compression argument, should it come into use
- suggested by @fabiotudone-da; thanks
* type-alias a couple of ProcessSubmission's args to label what they are
- suggested by @fabiotudone-da; thanks
* reformat after fixing merge
* more unused in kvutils
* define enclose's "default" compression as a constant
- suggested by @miklos-da; thanks
https://github.com/digital-asset/daml/pull/6992#discussion_r466489923
* participant-integration-api: In `JdbcIndexer`, log with context.
We were not providing the correct `loggingContext` to
`JdbcIndexer#handleStateUpdate`. This means we were just dropping useful
information. This adds the implicit so that it uses the correct logging
context.
There's a bigger problem, in that there are multiple logging contexts in
scope, making this very error prone. We'll need to figure out a way to
avoid this as much as possible.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Purge unnecessary newlines in JdbcIndexer.
* Create scala library for integrity checking tools.
CHANGELOG_BEGIN
CHANGELOG_END
* Moved integrity checking drivers into separate package.
* First define the scala library then the rest.
* Added missing header.
* Moved all export related code to under package kvutils.tools.export.
* Added missing header.
* Make all binaries depend on the library and not need sources.
* Factor out tar/gzip reproducibility flags
* use mktgz in package-app
* Bazel managed tar/gzip
* Remove quiet = True
As stated in the comment this is no longer required with Bazel >= 3.0.
* Build package-app as a sh_binary
This way Bazel will manage the runtime dependencies tar, gzip, mktgz,
and patchelf.
package-app.sh changes directory so it needs to make sure that all paths
are absolute and that the runfiles tree/manifest location is forwarded
to programs called by package-app.sh.
* Avoid file path too long errors
* Fix readlink -f on MacOS
* Document abspath
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Move in-mem writer's `ledgerStateAccess.inTransaction` down to committer
* Move `BatchedSubmissionValidator` and spec into `batch` subpackage
* Add `StateAccessingValidatingCommitter` and inherit it in batching one
* Document `StateAccessingValidatingCommitter`
* Generalize the committer for `InMemoryLedgerReaderWriter`
* `envelope` -> `submissionEnvelope` in validating committers
* Add `PreExecutingValidatingCommitter` and sub-components
* Add retry in case of conflict in `PreExecutingValidatingCommitter`
CHANGELOG_BEGIN
CHANGELOG_END
* Fix compilation error
* Hook pre-execution in `daml-on-memory-kv`
* Add fake time updates provider
* Fix `BatchedValidatingCommitterSpec`
* Don't use batched writer with pre-execution
* Fix conflict detection
* Fix out-of-time-bounds detection
* Prefix/unprefix serialized log entry IDs in pre-execution write sets
* Fix: produce an out-of-bounds rejection log entry in transaction rejected cases too
* Fix `SubmissionResult` return in case of repeated pre-exec conflict
* Fidelity level 1: sequential pre-execution
* Documentation for pre-execution support in DAML-on-Memory KV
* Add ledger-on-memory conformance test with pre-execution enabled
* Revert "Fix: produce an out-of-bounds rejection log entry in transaction rejected cases too"
This reverts commit 4df7e26b
* Fix test
* Improve naming and documentation
* Address review comments
* Fix test
* Fix wrong implementation used for `ParticipantStateIntegrationSpecBase` tests
* Address review comments
* Address review comments
* Address minor review comments
* remove unused definitions, params, args from ledger API Scala code
CHANGELOG_BEGIN
- [Ledger API] withTimeProvider removed from CommandClient; this method
has done nothing since the new ledger time model was introduced in
1.0.0. See `issue #6985 <https://github.com/digital-asset/daml/pull/6985>`__.
CHANGELOG_END
* percolate withTimeProvider and label removal elsewhere
* daml-on-sql: Fix references to the JAR file in the README.
* daml-on-sql: Add README info about architecture, OS, and dependencies.
* daml-on-sql: Make README headings consistent in style.
* daml-on-sql: Explain the `--ledgerid` flag in the README.
* daml-on-sql: Everything is important.
* daml-on-sql: Document parameters used to tune performance.
CHANGELOG_BEGIN
- [DAML on SQL] Document the architecture and OS we test against, native
system dependencies on Linux, and parameters for performance tuning.
CHANGELOG_END
* daml-on-sql: Document architecture requirements.
* event_sequential_id arithmetic
* add minPageSize fetch size to fallbacks
* use row_id arithmetic for single-party and multi-party tree queries, wildcard templates
- as a fallback, try to grab 10 or (pageSize / 10) rows with LIMIT,
whichever is larger
* add sequences of arnorm queries with parsed results
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* remove an extra LIMIT
* reformat SQL statements
* add newlines to other tx tree queries
* group by does nothing with this order by setting
* reformat all queries in EventsTableFlatEventsRangeQueries
* group by doesn't matter when you have order by and no aggregate exprs
* factor out the "faster read, then safer read" pattern from tree events
* make flat events all return SqlSequence with embedded parser
- with a path for EventsRange.readPage
* make flat transactions singleWildcardParty use arithmetic and fallback
* rename Fast to ByArith, Slow to ByLimit, because speed may vary
* missed Fast/Slow
* replace the other flat transaction queries with limited versions
replace
(?s)^ SQL"""(.*?)(\$\{range.endInclusive\})(.*?) limit \$pageSize"""
with
FrqK.ByArith(\n fasterRead = guessedPageEnd => SQL"""\n $1\$guessedPageEnd$3""",\n saferRead = minPageSize => SQL"""\n $1$2$3 limit \$minPageSize"""\n )
which is obviously better than being able to factor common parts of SQL
queries, so naturally I agree with anorm lacking a doobie-like append.
* remove readUpperBound, stray merge conflict inclusion
- thanks @leo-da for pointing it out
* rename SqlSequence.Elt to SqlSequence.Element
- suggested by @stefanobaghino-da; thanks
* rename FrqK to QueryParts
- suggested by @stefanobaghino-da; thanks
* reformat flatMap chain
* don't rescan first page; eliminate duplicate SQL exprs
- overload 'range' to mean "first page" then "search space after first page"
- page sizes are always safe to interpolate directly into SQL, as ints
(?s)^ fasterRead = guessedPageEnd => (.*?)\$guessedPageEnd(.*?)""",\n saferRead = minPageSize => SQL""".*?"""
read = (range, limitExpr) => $1\${range.endInclusive}$2 #\$limitExpr"""
* FilterRelation is used in a private[dao] context
- you won't get a warning for this because aliases are expanded before
this is checked, so the method can still be called, you simply can't
use the same type name used in the written signature
* generated sequences of transactions with different matching frequencies
- different occurrences of the matched transactions cause different SQL
queries to be used, so we try to exercise all of them
* generalize storeSync's traverse to let multiple tests run in order
- thanks to @leo-da for the inspiration
* a way for singleCreates to be slightly different
* test that matched transaction count and the specific offsets match
* test more code paths for flat events
* Added ledger writer that chooses between instances based on estimate interpretation cost.
CHANGELOG_BEGIN
CHANGELOG_END
* Code tidying.
* Delegate to pre-executing writer in case thershold is set to 0.
* Added ability to change metrics.
* Added metrics.
* Code tidying.
* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/participant/state/kvutils/api/InterpretationCostBasedLedgerWriterChooser.scala
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* add -Xlint:doc-detached
- reverts 1feae964e3 from #6798
* attach several scaladocs where they'll actually be included
* no changelog
* attach several more scaladocs where they'll actually be included
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* Change error code for invalid offsets for transaction stream and completion stream requests
* Expanded application architecture docs on how to build application with ledger api failover capabilities.
Fixes#6842.
CHANGELOG_BEGIN
- [Ledger API] The error code for requesting a transaction stream
with an offset beyond the ledger end changed from INVALID_ARGUMENT
to OUT_OF_RANGE. This makes it easier to handle scenarios where
an application fails over to a backup participant which hasn't
caught up with the ledger yet.
- [Ledger API] The command completion service now validates the offset and
returns the OUT_OF_RANGE error if the request offset is beyond the ledger end.
- [Documentation] Added a section on how to write DAML applications
that can fail over between multiple eventually consistent Ledger API endpoints
where command deduplication works across these Ledger API endpoints, which
can be useful for addressing HA and/or DR scenarios.
CHANGELOG_END
* daml-on-sql + sandbox: Remove unused Bazel `load` declarations.
* daml-on-sql: Build a Docker image.
* daml-on-sql: Move the example Kubernetes file over from Sandbox.
* Add a changelog entry that was dropped.
CHANGELOG_BEGIN
* [DAML on SQL] DAML on SQL is now available with every release,
from the GitHub Releases page.
CHANGELOG_END
* Upgrade the Java Docker base image to the latest version.
* Log all authorization errors
CHANGELOG_BEGIN
- [Ledger API Server] The ledger API server now prints detailed log messages
whenever a request was rejected due to a failed
authorization.
CHANGELOG_END
* Fix time model error message
CHANGELOG_BEGIN
CHANGELOG_END
* Restore ledger time based error message
* Add test for the error message
* Fix the error message
now for real?
* daml-on-sql: Pull out a new `Main` object that wraps sandbox-classic.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-on-sql: Fail if a JDBC URL is not provided or not for PostgreSQL.
* sandbox-classic: Rename the conformance test H2 database.
* daml-on-sql + sandbox-classic: Report configuration errors cleanly.
This means letting `ProgramResource` catch the errors, log, and exit.
* daml-on-sql: Change the name logged on startup.
* daml-on-sql: Change the default participant ID.
* sandbox-common: Give the ledger name its own tagged string type.
* sandbox-classic: Generate random ledger IDs using the ledger name.
* daml-on-sql: Remove the banner, replacing it with a blank line.
* daml-on-sql: Enable strong seeding by default.
And weak seeding in the conformance tests.
* sandbox-classic: Move the ledger name to a separate parameter.
It's not really configurable.
* sandbox-classic: Move LedgerName from sandbox-common.
* daml-on-sql: Remove "-participant" from the participant ID.
* daml-on-sql: Use `Name` where possible.
* daml-on-sql: Make the ledger ID mandatory.
* Revert "sandbox-classic: Move LedgerName from sandbox-common."
This reverts commit 0dad1584a7.
* daml-on-sql: Print "DAML-on-SQL" in the CLI help, not "Sandbox".
* daml-on-sql + sandbox + sandbox-classic: Split out custom CLI parsing. (#6846)
* participant-state: Simplify naming the seeding modes.
* Populate out-of-time-bounds entry when we set min/max record time.
* Populate out-of-time-bounds entry when we set max record time for ConfigCommitter.
* Code tidying.
* Do not throw in case no min/max record time has been specified and there's no out-of-time-bounds log entry.
CHANGELOG_BEGIN
CHANGELOG_END
* Code tidying.
* Add ledger-on-memory conformance test with pre-execution enabled
* Revert "Add ledger-on-memory conformance test with pre-execution enabled"
This reverts commit d2c4364a
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
* Upgrade nixpkgs revision
* Remove unused minio
It used to be used as a gateway to push the Nix cache to GCS, but has
since been replaced by nix-store-gcs-proxy.
* Update Bazel on Windows
changelog_begin
changelog_end
* Fix hlint warnings
The nixpkgs update implied an hlint update which enabled new warnings.
* Fix "Error applying patch"
Since Bazel 2.2.0 the order of generating `WORKSPACE` and `BUILD` files
and applying patches has been reversed. The allows users to define
patches to these files that will not be immediately overwritten.
However, it also means that patches on another repository's original
`WORKSPACE` file will likely become invalid.
* a948eb7255
* https://github.com/bazelbuild/bazel/issues/10681
Hint: If you're generating a patch with `git` then you can use the
following command to exclude the `WORKSPACE` file.
```
git diff ':(exclude)WORKSPACE'
```
* Update rules_nixpkgs
* nixpkgs location expansion escaping
* Drop --noincompatible_windows_native_test_wrapper
* client_server_test using sh_inline_test
client_server_test used to produce an executable shell script in form of
a text file output. However, since the removal of
`--noincompatible_windows_native_test_wrapper` this no longer works on
Windows since `.sh` files are not directly executable on Windows.
This change fixes the issue by producing the script file in a dedicated
rule and then wrapping it in a `sh_test` rule which also works on
Windows.
* daml_test using sh_inline_test
* daml_doc_test using sh_inline_test
* _daml_validate_test using sh_inline_test
* damlc_compile_test using sh_inline_test
* client_server_test find .exe on Windows
* Bump Windows cache for Bazel update
Remove `clean --expunge` after merge.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Validate performance test keys
Fixes#6823
changelog_begin
[Integration Kit] Fixed a bug in the Ledger API test tool that caused
the full conformance test suite to be run when trying to run performance
tests but using a wrong name. See https://github.com/digital-asset/daml/issues/6823
changelog_end
* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/Cli.scala
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Address https://github.com/digital-asset/daml/pull/6826#discussion_r458731629
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* sandbox: Inline dependency declarations in the BUILD files.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Inline test dependency declarations in the BUILD files.
And delete unnecessary test dependencies from the ResetService tests.
* sandbox-common: Move the default Sandbox participant ID here.
* sandbox-common: Move the participant ID into the config.
* sandbox-common: Allow the user to specify the participant ID in the CLI.
CHANGELOG_BEGIN
- [Sandbox] Allow the user to specify the participant ID with the
``--participant-id`` switch.
CHANGELOG_END
* participant-integration-api: `GrpcServerOwner` -> `GrpcServer.Owner`.
Mostly so I can create a test class named `GrpcServerSpec`.
* ports: Move the free port search from postgresql-testing.
* participant-integration-api: Test the basics of GrpcServer.
This uses the HelloService to make sure the server behaves normally.
* ledger-api-client: Extract out channel configuration from LedgerClient.
So we can test it independently of the LedgerClient itself.
* ledger-api-client: Increase the default maximum inbound header size.
Increased from 8 KB to 1 MB.
* participant-integration-api: Reduce the maximum error message size.
Truncate GRPC error descriptions to 256 KB.
* participant-integration-api: Use `Port.Dynamic` instead of `FreePort`.
In tests.
* participant-integration-api: Explicit null checks when they're shorter.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* ledger-api-client: Reduce the max inbound message size back to 8 KB.
And reduce the maximum size of an error description pushed out by the
server accordingly.
CHANGELOG_BEGIN
- [Integration Kit] Truncate GPRC error messages at 4 KB. This ensures
that we won't trigger a protocol error when sending errors to the
client.
CHANGELOG_END
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
