We’ve seen the tests time out despite already having a 900s test
timeout so this PR speeds up the tests using the following approaches:
1. Drop redundant test. We don’t need a test to test that daml repl
starts without a ledger if other tests also use that. Admittedly that
could make debugging slightly worse but I’m happy to accept that here.
2. Stop using a Ledger where we don’t need one. The import tests are
about testing the client side not the server side.
3. Share ledgers where we can. The inbound messages size tests and the
static time tests can reuse the ledger.
4. Stop using packages where we don’t need them. This speeds up both
ledger startup as well as Daml Repl startup.
changelog_begin
changelog_end
* trying to reliably reproduce the template ID constraint error
* tentative fix for template ID constraint error
* sequential simulation
* successfully reproduce the error pre-4633c3137a
- Batch entry 0
INSERT INTO some_fancy_prefix_contract
VALUES ('foo', 1, 'null'::jsonb, NULL, '{}'::jsonb, ?, ?, '')
ON CONFLICT (contract_id) DO NOTHING
was aborted: ERROR: insert or update on table "some_fancy_prefix_contract" violates foreign key constraint "some_fancy_prefix_contract_tpid_fkey"
Detail: Key (tpid)=(1) is not present in table "some_fancy_prefix_template_id".
* also reproduced the error pre-4633c3137a on Oracle
- ORA-02291: integrity constraint (UNA3GOHUV7YMSKT0MQXJKLKD9HKKAZ.SYS_C007859)
violated - parent key not found
* add changelog
CHANGELOG_BEGIN
- [JSON API] Fixed a rare error that manifested as
‘violates foreign key constraint "contract_tpid_fkey"
Detail: Key (tpid)=(...) is not present in table’
when attempting to run queries and goes away on JSON API restart.
See `issue #11330 <https://github.com/digital-asset/daml/pull/11330>`__.
CHANGELOG_END
* clean up some now-unneeded printlns
* interface: adding interfaces to the TS codegen
We add TS interfaces corresponding to Daml interfaces to the generated
TS code.
CHANGELOG_BEGIN
CHANGELOG_END
* Update language-support/ts/codegen/README.md
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Update language-support/ts/codegen/src/TsCodeGenMain.hs
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* comment on pure interface methods
* added comment on substitutions
* extend example/tests to two implementations
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Remove the mutating schema
changelog_begin
- [Participant] All participants now use the new append-only schema. Existing databases will
automatically upgrade to the new schema the first time a participant/ledger is started.
changelog_end
* Fix post-commit validation test
* Remove append-only flags from CI
* Don't crash when using deprecated flag
* Increase timeout for reset service tests
* Fix typos in parameter names
* Restore removed test
* Restore removed CLI check
* Improve CLI parameter description
* interfaces: Add fixed choice name collision check
Add a check that a template cannot have two choices with the same name,
even taking into account all of its "inherited" interface fixed choices.
Part of #11137
changelog_begin
changelog_end
* "Me want" -> "We want"
* [DPP-417][DPP-613] Adopt self-service error codes ApiTransactionService
CHANGELOG_BEGIN
CHANGELOG_END
* TV review
* TV review
* Uses ErrorFactories
* Error loggers use enriched logging contexts
* Deleted ApiTransactionServiceSpec
* Add TODO wrt to CommandValidation.invalidArgument
* Adapt transactionNotFound error assertions in conformance tests
* ErrorFactories.invalidArgumentWasNotFound for handling invalid event ids
* Revert improvement to TransactionNotFound cause message since
Canton still outputs the old message and assertions fail.
Co-authored-by: Tudor Voicu <tudor.voicu@digitalasset.com>
* Suport multi-party readAs in triggers
fixes#7640
This does not yet include the trigger service. We’ll tackle that separately.
changelog_begin
- [Daml Triggers] Triggers now support readAs parties. They can be
specified via `--ledger-readas a,b,c`. As part of this change
``testRule`` gained an extra argument to specify the `readAs`
parties. If you previously used
```
testRule trigger party acsBuilder commandsInFlight s
```
you now need to use
```
testRule trigger party [] acsBuilder commandsInFlight s
```
changelog_end
* Update triggers/tests/src/test/scala/com/digitalasset/daml/lf/engine/trigger/test/AbstractFuncTests.scala
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* [Self-service error codes] Implement V2 in Authorizer
CHANGELOG_BEGIN
CHANGELOG_END
* Added unit test for authorize (non-streamed)
* Fix after rebase
* Do not expose the error codes switching mechanism to the Java bindings
* Adjust InternalAuthorizationError to be SystemInternalAssumptionViolated
* Parameter names in test
* Testing AuthorizationInterceptor with regard to returned error codes
* Do not use default error code version switchers at instance creation
* Addressed Pawel's review comments
* Using ErrorFactories for error dispatching
* Pass loggingContext to Authorizer where available
* Generic internal authorization error
* Factor out npm install step of create-daml-app tests
This PR speeds up the create-daml-app tests by 30s or so by factoring
out the install of the unchanged npm deps into a genrule which we then
feed in again. This is a bit ugly but given how frequently we run
those tests I do think it is worth the uglyness.
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.
@sofiafaro-da is in charge of this release.
Commit log:
```
8ff347d8ed Expand type synonyms lazily (#11282)
2fd200fff8 KV: support contextual logging in KeyValueConsumption [KVL-1143] (#11288)
f4df1cc4d0 converting server errors to proper client errors (#11184)
525e4ceb5e Add GrpcStatus.toProto(JavaStatus) -> ScalaStatus conversion [KVL-1143] (#11292)
9b00a1aab5 Rotate release rotation (#11291)
dd09c38f4a Upgrade rules-nodejs (#11290)
87f1418d91 [Short] Remove unnecessary traits from ApiSubmissionServiceSpec (#11254)
6b65a72577 Merge npm install steps in create-daml-app tests (#11287)
a9f6afbfde kvutils: Rename `VersionedOffset` to `KVOffset`. (#11286)
98cf8d86b3 KV: introduce v2 error codes behind a CLI switch [KVL-1140] (#11224)
46f6877ee5 Increase time model skew limits (#11273)
8f94cffdd1 kvutils: Use `VersionedOffsetBuilder` where possible, and introduce `VersionedOffsetMutator`. [KVL-1154] (#11277)
81fde97b87 Bazel: Call `_wrap_rule` directly when building the Scala REPL rule. (#11279)
edb2b04abf Document scaladoc usage in Bazel (#11278)
62558374a7 Data dependencies support for reexports (#11147)
2fc7490df2 [Self-service error codes] Adapt error factories [DPP-656] (#11270)
b1a6b11d47 ledger-api-test-tool: Add static time awareness [KVL-1156] (#11266)
243c120939 Add an LF typechecking benchmark (#11276)
2db8c138a8 Test authorization within Exercise choice. (#11246)
ec58ed68b3 Typecheck nested LF type applications more efficiently (#11253)
a940016f46 Add debugRaw corresponding to traceM in Haskell (#11259)
a9885797fe Fix Scala repl targets (#11272)
e4808f620e Extract UtilLF module from daml-lf-conversion to its own library (#11263)
38712c0179 sandbox-next - Disable participant command deduplication [KVL-1156] (#11264)
82fa229385 Add daml-util-ghc lib (#11260)
68a93cf70f kvutils - Use the ledger configured time for command dedup [KVL-1149] (#11239)
c8cede074e Bump startup time in create-daml-app tests (#11261)
4fac87f112 Bump the schema version for the JSON API (#11252)
55076700ae update NOTICES file (#11256)
0063b1037f Retry faster during startup (#11255)
04feb40a8f Improve reset service tests (#11240)
b3375fd2a1 update compat versions for 1.18.0-snapshot.20211013.8071.0.514e8b50 (#11237)
9ed1eb3ce3 Address security notice on `ansi-regex` (#11243)
387d0e8f6c Make compatibility migrations `eternal` (#11242)
be899b32a8 Extract rejection_reason.proto from daml_kvutils.proto [KVL-1090] (#11235)
a1d94e10ba LF: Create special case class for VersionedContractInstance (#11127)
b7389885e8 Release 1.18 snapshot (#11234)
3c26852226 LF: Use template Id in exercise node of fixed choice. (#11229)
139b6f3b9e CI: Set `PROJ_DIR` inside the bash lib, not outside. (#11236)
ce64cb2a13 interfaces: Do some TODOs (#11231)
7a88c7d97c trigger-service: dev-mode-unsafe flag (#11233)
909a1bf896 [DPP-417][DDP-612] Adapt ApiSubmissionService to support V2 error codes (#11052)
```
Changelog:
```
- [JSON API] Several kinds of gRPC server errors are now reported with
associated HTTP statuses; for example, a Daml-LF interpreter error now
returns a 400 instead of a 500, and an exercise on an archived contract
returns a 409 Conflict instead of a 500. Errors internal to JSON API
(e.g. internal assertion failures) are no longer detailed in the HTTP
response; their details are only logged.
See `issue #11184 <https://github.com/digital-asset/daml/pull/11184>`__.
- [Daml Stdlib] Add `debugRaw` as a convenience wrapper around
`traceRaw` when used inside a do-block. `debugRaw` compares to
`debug` like `traceRaw` compares to `trace` meaning it expects a
`Text` instead of calling `show` on an expression.
sandbox-next - Disable participant command deduplication
kvutils - Command deduplication uses the configured time model (static/wall) and not running always using wall-clock
[JSON API] Solving a bug that could cause the JSON API to return
correct result if a contract with the same key is observed twice
required a schema change. The JSON API data needs to be dropped
and the query store needs to reset. If you are migrating from a
previous version, either reset your database manually or start
the HTTP JSON API with one of the options that regenerate the
schema (`create-only`, `create-if-needed-and-start`, `create-and-start`).
```
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines Daml Build <support@digitalasset.com>
Adding `Daml Documentation >> Building applications >> "Self-Service Error Codes (Experimental)` section in the HTML documentation.
The section is populated automatically with error code information retrieved from the classpath.
The process of generating documentation for error codes looks like this:
- first we find error codes information from the classpath,
- then we save it to a json file,
- then the json file is made available to a custom Sphinx extension,
- then the custom Sphinx extension generates documentation wherever we put a new custom Sphinx directive.
Try it out with
`./docs/scripts/live-preview.sh`
or
`./docs/scripts/preview.sh`
CHANGELOG_BEGIN
CHANGELOG_END
* Expand type synonyms lazily
We really only need to expand type synonyms in two cases:
1. When pattern matching e.g. to check that a projection on a struct
really happens on a struct. In this case we can first check if we
have a struct in which case we don’t need to expand anything. If we
don’t, we can expand afterwards.
2. When checking two types for alphaequivalence. This PR goes for the
simplest solution: First check alphaequivalence without expanding
anything treating type synonyms nominally. Then if that fails
expand everything and check again. We could be more clever and
interleave expansion with checking for alphaequivalence but this
seems to give us most of the performance benefits while keeping the
logic very simple.
On my local laptop this speeds up typechecking of the full DAR from
380ms to 176ms so roughly a 2x improvement.
For comparison: The Daml-LF 1.7 DAR which still relies on nonminal
typechecking of typeclasses everywhere takes 110ms. So we are still
leaving something on the table here but we’re much closer.
I think this also gets us to the point where speedy compilation is
slower than typechecking but I need to verify that again.
changelog_begin
changelog_end
* Apply suggestions from code review
Co-authored-by: Remy <remy.haemmerle@daml.com>
* fix biuld
changelog_begin
changelog_end
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Support contextual logging in KeyValueConsumption
CHANGELOG_BEGIN
CHANGELOG_END
* Fix formatting and imports
* Propagate the contextual logging per-readUpdates call
* No need for at-construction logging context in LogAppendingReadServiceFactory
* No need for logging context in CommitStrategySupport.newReadServiceFactory
* Fix JdbcIndexerSpec
* a model for trapping client errors in Scala bindings shim and reporting correctly
* clean up some nesting with an alias
* filter out client-side command service errors
* fix flattening error propagation of CommandService errors in endpoints
* remove todo
* Daml evaluation triggers INVALID_ARGUMENT; handle this for creates/exercises
* clean up lookupResult
* remove stripLeft utility; it is unused
* proper error propagation for /parties endpoint
* map grpc status codes to HTTP error codes
* add a case to pass-through gRPC errors in Endpoints errors
* handle gRPC status in all explicit top-level catches
* pass through gRPC errors in CommandService as well
* treat a gRPC status anywhere in the causal chain as indicating participant-server error
* propagate ContractsService errors without assuming they will always be ServerErrors
* filter ServerErrors' contents when rendering errorful streams
* log errors from websocket output instead of rendering full messages
* hide message in ServerError case
* remove Aborted
* transfer with bad contract ID now returns 409
* mention new error codes
* add changelog
CHANGELOG_BEGIN
- [JSON API] Several kinds of gRPC server errors are now reported with
associated HTTP statuses; for example, a Daml-LF interpreter error now
returns a 400 instead of a 500, and an exercise on an archived contract
returns a 409 Conflict instead of a 500. Errors internal to JSON API
(e.g. internal assertion failures) are no longer detailed in the HTTP
response; their details are only logged.
See `issue #11184 <https://github.com/digital-asset/daml/pull/11184>`__.
CHANGELOG_END
* remove unused Show and liftErr utility
* adapt daml-script to new error codes
* adapt typescript tests to new error codes
* adapt json-api failure tests to new error codes
There is no longer a non-versioned offset, so this is kind of
meaningless.
I also deleted `VersionedOffsetMutator`, which should not have made it
in.
CHANGELOG_BEGIN
CHANGELOG_END
* Propagate error version switch to KeyValueConsumption
CHANGELOG_BEGIN
CHANGELOG_END
* Introduce v2 (self-service) KV error codes behind the CLI switch
* Test v2 codes and fix them
* Keep newline separating methods in KeyValueConsumption
* TransactionRejections: don't wrap updates in `Some`
* Factor errorVersionsTable
* Reorder imports
* Split "convert rejection to proto models and back to expected grpc code"
* Remove unneeded Option returned by decodeTransactionRejectionEntry
* Formatting fix
* Fix 7537e93d
* kvutils: Remove `VersionedOffsetBuilder.apply`.
Doesn't do anything.
* ledger-on-memory: Use `VersionedOffsetBuilder`.
* indexer-benchmark: Use `VersionedOffsetBuilder`.
* ledger-on-sql: Use `VersionedOffsetBuilder`.
* kvutils: Use `VersionedOffsetBuilder` in tests.
* kvutils: Create a case class for `VersionedOffsetBuilder#split`.
* kvutils: Delete unused methods from the offset builders.
* kvutils: Use `Bytes#startWith` to check the offset version.
* kvutils: Move offset splitting into `VersionedOffset`.
* kvutils: Extract out versioned offset generators.
* kvutils: Replace `OffsetBuilder` with `VersionedOffsetMutator`.
This takes care of the last usages of `OffsetBuilder`, which were to
modify the lowest component of the offset.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Randomize the offset version in testing.
To make sure we don't use a hard-coded offset version anywhere.
* kvutils: `Random.between` is not available in Scala 2.12.
* kvutils: Move offset mutation methods to `VersionedOffset`.
* kvutils: Move the versioned offset construction into `VersionedOffset`.
The `VersionedOffsetBuilder` is still useful as it remembers the
version, meaning we only need to specify it once.
If we build the dependency lists in the `_create_scala_repl` function,
we don't add the correct `scalac` options; specifically, we don't add
the Silencer option to ignore unused imports from
`scala.collection .compat`.
By passing directly to `_wrap_rule`, we compute the `scalac` options
correctly.
CHANGELOG_BEGIN
CHANGELOG_END
* Add 'fsFromText' helper function in UtilGHC
* Define exports encoding/decoding
* Test exports encoding/decoding
* Tweak convertQualified to take a function on Name
* Generate $exports value
This includes all the values exposed by the module.
* Add norm_exports jq filter for QUERY-LF tests
* Test $exports value generation
* DataDeps reconstructs exports from $exports value
* Test data-dependencies preserves cross-package exports
* Only emit $exports entries for values defined in other modules
* Only emit $exports entries for Daml-LF versions that support type synonyms
* Do not emit $exports entries for internal DAML modules
* Use one $export<i> value per exported entity
changelog_begin
changelog_end
* Moved ErrorCodesVersionSwitcher to //ledger/error
CHANGELOG_BEGIN
CHANGELOG_END
* Rename ErrorCodeLoggingContext to ContextualizedErrorLogger
* Refactored ErrorFactories
* All error factories use ContextualizedErrorLogger for being able to dispatch self-service error codes.
* The ContextualizedErrorLogger is passed down from the dispatching Ledger API services.
* ErrorFactoriesSpec asserts both legacy (V1) and self-service error codes (V2).
* Adapted ApiSubmissionService
* Addressed Marcin's review comments
* ledger-api-test-tool: Add command line argument --static-time which makes the test tool aware that the ledger is running in static time
For command deduplication conformance tests this allows us to forward the time so that the tests can run in static time mode as well
CHANGELOG_BEGIN
CHANGELOG_END
* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/Cli.scala
* Do not run conformance test that cannot run in static time
* Remove hidden from static time flag for ledger api test tool
* ledger-api-test-tool: Clarify parts of the deduplication tests.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
