Commit Graph

8166 Commits

Author SHA1 Message Date
pbatko-da
2267429ae1
[DPP-656] Assert on self-service error code details in ErrorFactoriesSpec (#11289)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-20 09:18:20 +02:00
Remy
c06faf2764
LF: remove imperative environment from Speedy compiler (#11285)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-20 09:03:25 +02:00
pbatko-da
d3dad75d00
[DPP-592] Generate docs for self-service error codes. (#11129)
Adding `Daml Documentation >> Building applications >> "Self-Service Error Codes (Experimental)` section in the HTML documentation. 

The section is populated automatically with error code information retrieved from the classpath.
The process of generating documentation for error codes looks like this:
- first we find error codes information from the classpath,
- then we save it to a json file,
- then the json file is made available to a custom Sphinx extension,
- then the custom Sphinx extension generates documentation wherever we put a new custom Sphinx directive.

Try it out with
`./docs/scripts/live-preview.sh`
or
`./docs/scripts/preview.sh`


CHANGELOG_BEGIN
CHANGELOG_END
2021-10-20 08:47:54 +02:00
Moritz Kiefer
8ff347d8ed
Expand type synonyms lazily (#11282)
* Expand type synonyms lazily

We really only need to expand type synonyms in two cases:
1. When pattern matching e.g. to check that a projection on a struct
   really happens on a struct. In this case we can first check if we
   have a struct in which case we don’t need to expand anything. If we
   don’t, we can expand afterwards.
2. When checking two types for alphaequivalence. This PR goes for the
   simplest solution: First check alphaequivalence without expanding
   anything treating type synonyms nominally. Then if that fails
   expand everything and check again. We could be more clever and
   interleave expansion with checking for alphaequivalence but this
   seems to give us most of the performance benefits while keeping the
   logic very simple.

On my local laptop this speeds up typechecking of the full DAR from
380ms to 176ms so roughly a 2x improvement.

For comparison: The Daml-LF 1.7 DAR which still relies on nonminal
typechecking of typeclasses everywhere takes 110ms. So we are still
leaving something on the table here but we’re much closer.

I think this also gets us to the point where speedy compilation is
slower than typechecking but I need to verify that again.

changelog_begin
changelog_end

* Apply suggestions from code review

Co-authored-by: Remy <remy.haemmerle@daml.com>

* fix biuld

changelog_begin
changelog_end

Co-authored-by: Remy <remy.haemmerle@daml.com>
2021-10-19 20:48:32 +00:00
fabiotudone-da
2fd200fff8
KV: support contextual logging in KeyValueConsumption [KVL-1143] (#11288)
* Support contextual logging in KeyValueConsumption

CHANGELOG_BEGIN
CHANGELOG_END

* Fix formatting and imports

* Propagate the contextual logging per-readUpdates call

* No need for at-construction logging context in LogAppendingReadServiceFactory

* No need for logging context in CommitStrategySupport.newReadServiceFactory

* Fix JdbcIndexerSpec
2021-10-19 20:16:37 +00:00
Stephen Compall
f4df1cc4d0
converting server errors to proper client errors (#11184)
* a model for trapping client errors in Scala bindings shim and reporting correctly

* clean up some nesting with an alias

* filter out client-side command service errors

* fix flattening error propagation of CommandService errors in endpoints

* remove todo

* Daml evaluation triggers INVALID_ARGUMENT; handle this for creates/exercises

* clean up lookupResult

* remove stripLeft utility; it is unused

* proper error propagation for /parties endpoint

* map grpc status codes to HTTP error codes

* add a case to pass-through gRPC errors in Endpoints errors

* handle gRPC status in all explicit top-level catches

* pass through gRPC errors in CommandService as well

* treat a gRPC status anywhere in the causal chain as indicating participant-server error

* propagate ContractsService errors without assuming they will always be ServerErrors

* filter ServerErrors' contents when rendering errorful streams

* log errors from websocket output instead of rendering full messages

* hide message in ServerError case

* remove Aborted

* transfer with bad contract ID now returns 409

* mention new error codes

* add changelog

CHANGELOG_BEGIN
- [JSON API] Several kinds of gRPC server errors are now reported with
  associated HTTP statuses; for example, a Daml-LF interpreter error now
  returns a 400 instead of a 500, and an exercise on an archived contract
  returns a 409 Conflict instead of a 500.  Errors internal to JSON API
  (e.g. internal assertion failures) are no longer detailed in the HTTP
  response; their details are only logged.
  See `issue #11184 <https://github.com/digital-asset/daml/pull/11184>`__.
CHANGELOG_END

* remove unused Show and liftErr utility

* adapt daml-script to new error codes

* adapt typescript tests to new error codes

* adapt json-api failure tests to new error codes
2021-10-19 13:54:29 -04:00
fabiotudone-da
525e4ceb5e
Add GrpcStatus.toProto(JavaStatus) -> ScalaStatus conversion [KVL-1143] (#11292)
* Add GrpcStatus.toProto(JavaStatus) -> ScalaStatus conversion

CHANGELOG_BEGIN
CHANGELOG_END

* Improve readability of internal `details` utility

* Fix formatting

* details -> extractDetails
2021-10-19 16:57:41 +00:00
Moritz Kiefer
9b00a1aab5
Rotate release rotation (#11291)
changelog_begin
changelog_end
2021-10-19 14:47:39 +00:00
Moritz Kiefer
dd09c38f4a
Upgrade rules-nodejs (#11290)
I got confused by an error and thought I might as well debug it on the
latest version instead of debugging it on 3.5.1.

changelog_begin
changelog_end
2021-10-19 14:13:55 +00:00
pbatko-da
87f1418d91
[Short] Remove unnecessary traits from ApiSubmissionServiceSpec (#11254)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-19 15:49:07 +02:00
Moritz Kiefer
6b65a72577
Merge npm install steps in create-daml-app tests (#11287)
This cuts about 20s of the test time locally with no meaningful loss
in test coverage (imho).

changelog_begin
changelog_end
2021-10-19 15:32:44 +02:00
Samir Talwar
a9f6afbfde
kvutils: Rename VersionedOffset to KVOffset. (#11286)
There is no longer a non-versioned offset, so this is kind of
meaningless.

I also deleted `VersionedOffsetMutator`, which should not have made it
in.

CHANGELOG_BEGIN
CHANGELOG_END
2021-10-19 09:34:19 +00:00
fabiotudone-da
98cf8d86b3
KV: introduce v2 error codes behind a CLI switch [KVL-1140] (#11224)
* Propagate error version switch to KeyValueConsumption

CHANGELOG_BEGIN
CHANGELOG_END

* Introduce v2 (self-service) KV error codes behind the CLI switch

* Test v2 codes and fix them

* Keep newline separating methods in KeyValueConsumption

* TransactionRejections: don't wrap updates in `Some`

* Factor errorVersionsTable

* Reorder imports

* Split "convert rejection to proto models and back to expected grpc code"

* Remove unneeded Option returned by decodeTransactionRejectionEntry

* Formatting fix

* Fix 7537e93d
2021-10-19 08:32:57 +00:00
Robert Autenrieth
46f6877ee5
Increase time model skew limits (#11273)
* Increase time model skew limits

changelog_begin
changelog_end

* Only override time model for sandbox
2021-10-18 21:27:53 +02:00
Samir Talwar
8f94cffdd1
kvutils: Use VersionedOffsetBuilder where possible, and introduce VersionedOffsetMutator. [KVL-1154] (#11277)
* kvutils: Remove `VersionedOffsetBuilder.apply`.

Doesn't do anything.

* ledger-on-memory: Use `VersionedOffsetBuilder`.

* indexer-benchmark: Use `VersionedOffsetBuilder`.

* ledger-on-sql: Use `VersionedOffsetBuilder`.

* kvutils: Use `VersionedOffsetBuilder` in tests.

* kvutils: Create a case class for `VersionedOffsetBuilder#split`.

* kvutils: Delete unused methods from the offset builders.

* kvutils: Use `Bytes#startWith` to check the offset version.

* kvutils: Move offset splitting into `VersionedOffset`.

* kvutils: Extract out versioned offset generators.

* kvutils: Replace `OffsetBuilder` with `VersionedOffsetMutator`.

This takes care of the last usages of `OffsetBuilder`, which were to
modify the lowest component of the offset.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Randomize the offset version in testing.

To make sure we don't use a hard-coded offset version anywhere.

* kvutils: `Random.between` is not available in Scala 2.12.

* kvutils: Move offset mutation methods to `VersionedOffset`.

* kvutils: Move the versioned offset construction into `VersionedOffset`.

The `VersionedOffsetBuilder` is still useful as it remembers the
version, meaning we only need to specify it once.
2021-10-18 16:50:52 +00:00
Samir Talwar
81fde97b87
Bazel: Call _wrap_rule directly when building the Scala REPL rule. (#11279)
If we build the dependency lists in the `_create_scala_repl` function,
we don't add the correct `scalac` options; specifically, we don't add
the Silencer option to ignore unused imports from
`scala.collection .compat`.

By passing directly to `_wrap_rule`, we compute the `scalac` options
correctly.

CHANGELOG_BEGIN
CHANGELOG_END
2021-10-18 14:57:06 +00:00
Moritz Kiefer
edb2b04abf
Document scaladoc usage in Bazel (#11278)
changelog_begin
changelog_end
2021-10-18 14:54:47 +00:00
Moisés Ackerman
62558374a7
Data dependencies support for reexports (#11147)
* Add 'fsFromText' helper function in UtilGHC

* Define exports encoding/decoding

* Test exports encoding/decoding

* Tweak convertQualified to take a function on Name

* Generate $exports value

This includes all the values exposed by the module.

* Add norm_exports jq filter for QUERY-LF tests

* Test $exports value generation

* DataDeps reconstructs exports from $exports value

* Test data-dependencies preserves cross-package exports

* Only emit $exports entries for values defined in other modules

* Only emit $exports entries for Daml-LF versions that support type synonyms

* Do not emit $exports entries for internal DAML modules

* Use one $export<i> value per exported entity

changelog_begin
changelog_end
2021-10-18 14:51:11 +00:00
tudor-da
2fc7490df2
[Self-service error codes] Adapt error factories [DPP-656] (#11270)
* Moved ErrorCodesVersionSwitcher to //ledger/error

CHANGELOG_BEGIN
CHANGELOG_END

* Rename ErrorCodeLoggingContext to ContextualizedErrorLogger

* Refactored ErrorFactories
* All error factories use ContextualizedErrorLogger for being able to dispatch self-service error codes.
* The ContextualizedErrorLogger is passed down from the dispatching Ledger API services.
* ErrorFactoriesSpec asserts both legacy (V1) and self-service error codes (V2).

* Adapted ApiSubmissionService

* Addressed Marcin's review comments
2021-10-18 16:46:20 +02:00
nicu-da
b1a6b11d47
ledger-api-test-tool: Add static time awareness [KVL-1156] (#11266)
* ledger-api-test-tool: Add command line argument --static-time which makes the test tool aware that the ledger is running in static time

For command deduplication conformance tests this allows us to forward the time so that the tests can run in static time mode as well

CHANGELOG_BEGIN
CHANGELOG_END

* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/Cli.scala

* Do not run conformance test that cannot run in static time

* Remove hidden from static time flag for ledger api test tool

* ledger-api-test-tool: Clarify parts of the deduplication tests.

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
2021-10-18 14:16:25 +00:00
Moritz Kiefer
243c120939
Add an LF typechecking benchmark (#11276)
changelog_begin
changelog_end
2021-10-18 12:53:17 +00:00
nickchapman-da
2db8c138a8
Test authorization within Exercise choice. (#11246)
* Test authorization within Exercise choice.

CHANGELOG_BEGIN
CHANGELOG_END

* fix build following rebase

* 4x shouldBe a Right

* test for exercise within exercise

* implicit Name.assertFromString

* def --> val. massive speedup!

* be more private

* implicit Identifier creation

* use implicit for Party and ChoiceName creation

* comment test purpose

* delete stray text; fix build
2021-10-18 12:33:23 +01:00
Moritz Kiefer
ec58ed68b3
Typecheck nested LF type applications more efficiently (#11253)
* Typecheck nested LF type applications more efficiently

This improves performance typechecking in the Scala LF typechecker by
matching on nested foralls & type applications which allows us to
check them with only a single substitution as opposed to the previous
state of doing one for each type application.

This is about a 25% improvement on typechecking performance of
GHC.Classes from 200ms to 150ms.

changelog_begin
changelog_end
2021-10-18 11:14:59 +00:00
Moritz Kiefer
a940016f46
Add debugRaw corresponding to traceM in Haskell (#11259)
changelog_begin

- [Daml Stdlib] Add `debugRaw` as a convenience wrapper around
  `traceRaw` when used inside a do-block. `debugRaw` compares to
  `debug` like `traceRaw` compares to `trace` meaning it expects a
  `Text` instead of calling `show` on an expression.

changelog_end
2021-10-18 10:55:46 +02:00
Moritz Kiefer
a9885797fe
Fix Scala repl targets (#11272)
We need _wrap_rule to make sure plugins like wartremover are also
correctly applied here.

We need jline because otherwise the repl starts but then fails because
jline isn’t in scope.

changelog_begin
changelog_end
2021-10-18 07:19:41 +00:00
Moisés Ackerman
e4808f620e
Extract UtilLF module from daml-lf-conversion to its own library (#11263)
changelog_begin
changelog_end
2021-10-15 17:52:09 +00:00
nicu-da
38712c0179
sandbox-next - Disable participant command deduplication [KVL-1156] (#11264)
CHANGELOG_BEGIN

sandbox-next - Disable participant command deduplication

CHANGELOG_END
2021-10-15 09:04:36 -07:00
Moisés Ackerman
82fa229385
Add daml-util-ghc lib (#11260)
* Extract UtilGHC module from daml-lf-conversion to its own library

* Drop daml-preprocessor dependency on daml-lf-conversion

changelog_begin
changelog_end
2021-10-15 15:52:29 +00:00
nicu-da
68a93cf70f
kvutils - Use the ledger configured time for command dedup [KVL-1149] (#11239)
CHANGELOG_BEGIN
kvutils - Command deduplication uses the configured time model (static/wall) and not running always using wall-clock
CHANGELOG_END
2021-10-15 05:56:45 -07:00
Moritz Kiefer
c8cede074e
Bump startup time in create-daml-app tests (#11261)
We’ve seen this timeout a few times on CI. The version in the compat
tests already has a 60s timeout and that seems to work better so
here’s to hoping this helps.

I’ve also added some logs to see where it stops if it times out
again (jest hides console.log by default which is why this uses
console.debug by default).

changelog_begin
changelog_end
2021-10-15 14:53:43 +02:00
Stefano Baghino
4fac87f112
Bump the schema version for the JSON API (#11252)
Closes #11251

Schema changed as part of https://github.com/digital-asset/daml/pull/11102

Also backported to 1.17.1 in https://github.com/digital-asset/daml/pull/11143

changelog_begin
[JSON API] Solving a bug that could cause the JSON API to return
correct result if a contract with the same key is observed twice
required a schema change. The JSON API data needs to be dropped
and the query store needs to reset. If you are migrating from a
previous version, either reset your database manually or start
the HTTP JSON API with one of the options that regenerate the
schema (`create-only`, `create-if-needed-and-start`, `create-and-start`).
changelog_end
2021-10-15 13:53:18 +02:00
azure-pipelines[bot]
55076700ae
update NOTICES file (#11256)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-10-15 10:10:35 +00:00
Robert Autenrieth
0063b1037f
Retry faster during startup (#11255)
changelog_begin
changelog_end
2021-10-15 08:27:48 +02:00
Robert Autenrieth
04feb40a8f
Improve reset service tests (#11240)
* Improve reset service tests

- Increase timeout
- Decrease load
- Improve logging

changelog_begin
changelog_end

* Apply review comments
2021-10-15 02:15:58 +02:00
azure-pipelines[bot]
b3375fd2a1
update compat versions for 1.18.0-snapshot.20211013.8071.0.514e8b50 (#11237)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-10-14 16:33:01 +02:00
Stefano Baghino
9ed1eb3ce3
Address security notice on ansi-regex (#11243)
* Address security notice on `ansi-regex`

Addressed this via resolution of `chalk` on a common version
which seems close enough to other old versions not to break
anything (tested manually) while removing the `ansi-regex`
transitive dependency that causes our daily builds to break.

Also removed `package-lock.json` since we are using `yarn`.

changelog_begin
changelog_end
2021-10-14 11:15:34 +00:00
Stefano Baghino
387d0e8f6c
Make compatibility migrations eternal (#11242)
Not sure whether this is a good choice but I could not
find anything between `long` and `eternal`.

changelog_begin
changelog_end
2021-10-14 10:12:22 +00:00
nicu-da
be899b32a8
Extract rejection_reason.proto from daml_kvutils.proto [KVL-1090] (#11235)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-14 00:22:45 -07:00
Remy
a1d94e10ba
LF: Create special case class for VersionedContractInstance (#11127)
To replace usage of ContractInstance[VersionedValue]

CHANGELOG_BEGIN
CHANGELOG_END
2021-10-14 08:53:26 +02:00
Moritz Kiefer
b7389885e8
Release 1.18 snapshot (#11234)
changelog_begin
changelog_end
2021-10-13 13:27:14 +00:00
Remy
3c26852226
LF: Use template Id in exercise node of fixed choice. (#11229)
* LF: Put template Id in exercise node of fixed choice.

instead of interface id and fix ValueEnricher.

CHANGELOG_BEGIN
CHANGELOG_END
2021-10-13 15:24:00 +02:00
Samir Talwar
139b6f3b9e
CI: Set PROJ_DIR inside the bash lib, not outside. (#11236)
* CI: Set `PROJ_DIR` inside the bash lib, not outside.

We can't reference variables set outside.

CHANGELOG_BEGIN
CHANGELOG_END

* CI: Remove an unnecessary `export`.
2021-10-13 14:55:20 +02:00
Sofia Faro
ce64cb2a13
interfaces: Do some TODOs (#11231)
* interfaces: Do some TODOs

- Add uniqueness check between fixed choices and virtual choices in
  haskell decoder.
- Encode interface methods and fixed choices in scala encoder.
- ExprIterable for interfaces.

changelog_begin
changelog_end

* scalafmt
2021-10-13 13:36:53 +01:00
Robin Krom
7a88c7d97c
trigger-service: dev-mode-unsafe flag (#11233)
* trigger-service: dev-mode-unsafe flag

This adds a `--dev-mode-unsafe` flag analogous to the trigger service
analogous to its counterpart in the trigger runner.

CHANGELOG_BEGIN
CHANGELOG_END

* fix tests
2021-10-13 12:22:10 +00:00
pbatko-da
909a1bf896
[DPP-417][DDP-612] Adapt ApiSubmissionService to support V2 error codes (#11052)
Adapt ApiSubmissionService to support self-service (v2) error codes

CHANGELOG_BEGIN
CHANGELOG_END
2021-10-13 14:19:10 +02:00
nicu-da
514e8b50a3
Extract configuration_rejection.proto from daml_kvutils.proto [KVL-1090] (#11215)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-13 10:34:45 +00:00
Remy
d1611e8bc1
LF: Drop templateID from SEScopeExercise and KCloseExercise. (#11228)
We do not need to keep track of it because we have it in the partial
transaction.

CHANGELOG_BEGIN
CHANGELOG_END
2021-10-13 11:40:30 +02:00
Moritz Kiefer
cb1fb6f7bf
Reuse sandbox & JSON API across daml ledger tests (#11214)
Speeds them up by more than 2x and reduces flakiness. I also added a higher
timeout to the allocate party call since that seems to still be
sometimes too slow if lots of things run in parallel.

changelog_begin
changelog_end
2021-10-13 11:19:44 +02:00
Robin Krom
f4ca876756
interfaces: test for triggers (#11221)
This adds a test checking whether we can exercise choices on interface
contract IDs in triggers. We can only test this for daml-lf 1.dev.

CHANGELOG_BEGIN
CHANGELOG_END
2021-10-13 10:17:17 +02:00
nicu-da
e3ae424682
Extract party_allocation.proto from daml_kvutils.proto [KVL-1090] (#11213)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-13 08:15:17 +00:00