* Preparation: pull up initialisation
* Preparation: introduce partyArrayContext for more flexibility
* Preparation: add append-only-from-scratch feature to FlywayMigrations
* Add squashed Flyway migration for append-only H2
* Add support for H2 StorageBackend
* Extend conformance test suites for H2
* Extend JDBCLedgerDao unit test suites for H2
* Extend sandbox-classic unit test suites for H2
* Fixing typo, some renames based on review
changelog_begin
changelog_end
@stefanobaghino-da is taking care of 1.14.0-snapshot.20210615.7169.0.adeba206 (#10018), so they get pushed back to the end of the line.
Please do not merge this before #10018.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
* Do not terminate transactions/transaction trees streams when no end-offset provided in the ledger-api-bench-tool
CHANGELOG_BEGIN
CHANGELOG_END
* Minor change
* Split scenario & ledger execution
This PR by no means aims to solve everything we can do here. It is
rather the minimal change that I could get to work that provides us
with one Speedy machine for scenario execution (which is still an
on-ledger machine to avoid having to change the callsites to much in
this PR) and one speedy machine per submission.
There is tons of cleanup we can do afterwards but this should
hopefully set the right foundations.
changelog_begin
changelog_end
* [JSON-API] Concurrent query etc. metrics
changelog_begin
- [JSON-API] The metrics which describe the amount of these concurrent events is now available:
- running http request
- command submissions
- package allocations
- party allocations
changelog_end
* Rename running metrics to throughput ones & add comments on the metrics
* Adjust names of other existing metrics too, to have for the json api a more consistent metrics naming
* truncate party name for 64 chars for multi-valued submitters to avoid excessively long metrics name
CHANGELOG_BEGIN
[metrics] Limit size of multi party metrics to ease consumption
CHANGELOG_END
* Use just first party for dynamic metrics when there are multiple parties involved
* Log context of all updates written to the database
CHANGELOG_BEGIN
For every update in the index db log the full context at the INFO level.
CHANGELOG_END
* one more missing log
Commit log:
```
adeba20659 DPP-403 log flyway progress (#9964)
401069ef00 prepare for upgrade to Scalaz 7.3.3 (#9997)
3ad9cd58d9 [JSON-API] Log templateId & choice name (if present) on command submissions in the json API (#10005)
af144e8592 LF: Structure package loading errors (#9980)
3583fd1885 [JSON-API] Log source and path of incoming http requests & the response with info level (#10007)
e49c7c8db7 [In-memory fan-out] BufferedTransactionsReader implementation [DPP-413] [DPP-414] (#9882)
acb77336c8 Release 1.14 RC 2 (#10006)
3d2d4b9696 Don't duplicate log failed futures in the http json api (commandservice specific ones) (#9990)
0c32cf2968 Clarify the semantics of DIV_INT64 in LF spec. (#10003)
b79e739196 Fix Type (#10004)
998f106ee6 Drop pkgIds argument from Engine.validatePackages (#9986)
ae5010f3ef [Mutable state cache] Start cache from latest ledger head (#9998)
dd42a5a641 update maven_install_2.12.json (#9999)
66d38cd24d Storage query abstraction first sketch (#9959)
c75013622c Just print out the operator name for the operation logging in CommandService.scala (#9989)
b6faace335 Changing oracle backend from using varray to json array (#9943)
603a4a4b42 Add date to log statements in the HTTP JSON service (#9988)
836a82a6b6 Add proper error handling for missing contract keys (#9972)
85667af7be Address npm vulnerabilities (#9978)
d7d6e37785 LF: Structure Engine Errors. (#9971)
0adbbdcd58 update Daml Hub auth in create-daml-app (#9973)
997a7d449c Use symlinks on Windows (#9969)
e4258e61b1 docs: tweak Exception intro (#9967)
b2c4ef813d rotate release duty after 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9933)
de7a462082 Log call to submitAndWaitRequest with the command id provided in the log ctx (#9940)
630d021331 Collect build-event-stream (#9831)
26e199a719 Rerelease 1.11.2 snapshot (#9965)
8399039a8a Remove broken dep from release job (#9963)
e1db529162 DPP-390 configurable event decoding parallelism (#9931)
cc7af93afc [docs] Remove link to the #daml-contributors (#9928)
f7ac3ba784 Release 1.11.2 snapshot attempt 2 (#9962)
4634147ed1 Adds Indexer state to GRPC health checks [DPP-434] (#9951)
28c567b53d Release SDK 1.11.2 (#9957)
5f0f5756bf Define IncompleteTransaction for use by scenario-service (#9952)
b7fd8338c4 Increase `eventually` timeout when waiting for queue to be killed (#9955)
ca0fb16718 Fix collect_build_data job (#9956)
42030b25ed Skip Scala_2_12 job on releases (#9954)
5e1ddfbd5e LF: Fix specification of ArithmeticError (#9949)
3093a98b87 Forward port additional contract keys validation tests (#9944)
e541558f18 update NOTICES file (#9946)
db792a9774 Proxy ledger API health endpoint in JSON API health endpoint (#9945)
13767b5c5d Load the correct logback file for the http json service respecting the deployment situation (#9938)
4e49cf6814 `ledger-api-bench-tool` - Active contracts stream, completions stream [DPP-398, DPP-399] (#9857)
fce8e54469 release 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9932)
ab55fb2546 Handle module-prefixes in Java codegen (#9929)
7375e28271 Force resolution for css what to 5.0.1 (#9925)
```
Changelog:
```
- [JSON API] The template id & choice name (if present) are now logged on command submissions in the Json API (at trace level)
- [JSON-API] The source and the path for incoming http requests are now logged
- [JSON-API] The http response for a request is now logged
- [JSON API] Errors which arise from the CommandService are not logged twice anymore (thus reducing noise)
- [JSON API] log statements now include the date next to the time
- [HTTP-JSON] Calls which trigger a submitAndWaitRequest are logged with the command id provided in the log ctx
[Integration Kit] The state of the participant indexer can now be checked via the GRPC health endpoint
- [JSON API] The healthcheck endpoint on the JSON API now proxies the
health check endpoint on the underlying ledger. Previously it only
queried for ledger end to determine ledger health.
[HTTP-JSON]
- fixed that json log output could not be enabled via cli options except via usage of env vars
- [Integration Kit] - ledger-api-bench-tool - reading active contract streams
- [Integration Kit] - ledger-api-bench-tool - Reading completions stream
- [Java Codegen] The Java codegen will now pick up the
`module-prefixes` field from `daml.yaml` which can be used to handle
module name collisions between different DALFs.
```
CHANGELOG_BEGIN
CHANGELOG_END
* Force css-what resolution in compiler/daml-extension & fully regenerate the lock file
changelog_begin
changelog_end
* Don't force resolution as it's unecessary
This is a preparation for H2 integration: reworking/opening up the ingestion DSL
* Pull more execution logic to PGTable, as preparing for batch insert execution
* Pull out Table from PGTable, add support for deletion (preparation for JDBC batch execution)
* Abstract over Schema (preparation for JDBC batch execution)
* Opening up Field, abstract over table creation (preparation for JDBC batch execution)
* Move code to common (preparation for JDBC batch execution)
* Add generic JDBC batch execution
* Add generic JDBC batch deletion and some more generic Field-s
* Pulling out PGSmallintOptional to support the postgres flavor
* Generalizing idempotentInsert in FieldStrategy
* Capture non empty data logic and comment in a util function
* Seal access to Schema internals
changelog_begin
changelog_end
* Add information from the jwtpayload to the logging context
changelog_begin
- [JSON API] For applicable requests actAs, readAs, applicationId & ledgerId are included in the log context
changelog_end
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Revert changes to make the function generic
* Create JwtPayloadG trait from which both payload variants inherit
* Reduce code duplication in Endpoints.scala
* Apply review suggestion
* Update test name to reflect field name changes
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.
@stefanobaghino-da is in charge of this release.
Commit log:
```
adeba20659 DPP-403 log flyway progress (#9964)
401069ef00 prepare for upgrade to Scalaz 7.3.3 (#9997)
3ad9cd58d9 [JSON-API] Log templateId & choice name (if present) on command submissions in the json API (#10005)
af144e8592 LF: Structure package loading errors (#9980)
3583fd1885 [JSON-API] Log source and path of incoming http requests & the response with info level (#10007)
e49c7c8db7 [In-memory fan-out] BufferedTransactionsReader implementation [DPP-413] [DPP-414] (#9882)
acb77336c8 Release 1.14 RC 2 (#10006)
3d2d4b9696 Don't duplicate log failed futures in the http json api (commandservice specific ones) (#9990)
0c32cf2968 Clarify the semantics of DIV_INT64 in LF spec. (#10003)
b79e739196 Fix Type (#10004)
998f106ee6 Drop pkgIds argument from Engine.validatePackages (#9986)
ae5010f3ef [Mutable state cache] Start cache from latest ledger head (#9998)
dd42a5a641 update maven_install_2.12.json (#9999)
66d38cd24d Storage query abstraction first sketch (#9959)
c75013622c Just print out the operator name for the operation logging in CommandService.scala (#9989)
b6faace335 Changing oracle backend from using varray to json array (#9943)
603a4a4b42 Add date to log statements in the HTTP JSON service (#9988)
836a82a6b6 Add proper error handling for missing contract keys (#9972)
85667af7be Address npm vulnerabilities (#9978)
d7d6e37785 LF: Structure Engine Errors. (#9971)
0adbbdcd58 update Daml Hub auth in create-daml-app (#9973)
997a7d449c Use symlinks on Windows (#9969)
e4258e61b1 docs: tweak Exception intro (#9967)
b2c4ef813d rotate release duty after 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9933)
de7a462082 Log call to submitAndWaitRequest with the command id provided in the log ctx (#9940)
630d021331 Collect build-event-stream (#9831)
26e199a719 Rerelease 1.11.2 snapshot (#9965)
8399039a8a Remove broken dep from release job (#9963)
e1db529162 DPP-390 configurable event decoding parallelism (#9931)
cc7af93afc [docs] Remove link to the #daml-contributors (#9928)
f7ac3ba784 Release 1.11.2 snapshot attempt 2 (#9962)
4634147ed1 Adds Indexer state to GRPC health checks [DPP-434] (#9951)
28c567b53d Release SDK 1.11.2 (#9957)
5f0f5756bf Define IncompleteTransaction for use by scenario-service (#9952)
b7fd8338c4 Increase `eventually` timeout when waiting for queue to be killed (#9955)
ca0fb16718 Fix collect_build_data job (#9956)
42030b25ed Skip Scala_2_12 job on releases (#9954)
5e1ddfbd5e LF: Fix specification of ArithmeticError (#9949)
3093a98b87 Forward port additional contract keys validation tests (#9944)
e541558f18 update NOTICES file (#9946)
db792a9774 Proxy ledger API health endpoint in JSON API health endpoint (#9945)
13767b5c5d Load the correct logback file for the http json service respecting the deployment situation (#9938)
4e49cf6814 `ledger-api-bench-tool` - Active contracts stream, completions stream [DPP-398, DPP-399] (#9857)
fce8e54469 release 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9932)
ab55fb2546 Handle module-prefixes in Java codegen (#9929)
7375e28271 Force resolution for css what to 5.0.1 (#9925)
```
Changelog:
```
- [JSON API] The template id & choice name (if present) are now logged on command submissions in the Json API (at trace level)
- [JSON-API] The source and the path for incoming http requests are now logged
- [JSON-API] The http response for a request is now logged
- [JSON API] Errors which arise from the CommandService are not logged twice anymore (thus reducing noise)
- [JSON API] log statements now include the date next to the time
- [HTTP-JSON] Calls which trigger a submitAndWaitRequest are logged with the command id provided in the log ctx
[Integration Kit] The state of the participant indexer can now be checked via the GRPC health endpoint
- [JSON API] The healthcheck endpoint on the JSON API now proxies the
health check endpoint on the underlying ledger. Previously it only
queried for ledger end to determine ledger health.
[HTTP-JSON]
- fixed that json log output could not be enabled via cli options except via usage of env vars
- [Integration Kit] - ledger-api-bench-tool - reading active contract streams
- [Integration Kit] - ledger-api-bench-tool - Reading completions stream
- [Java Codegen] The Java codegen will now pick up the
`module-prefixes` field from `daml.yaml` which can be used to handle
module name collisions between different DALFs.
```
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
* upgrade scalacheck to 1.14.3
* regenerate maven_install files
* some different names and implicits
* remove some fromTryCatchNonFatal
* more porting
* port fromTryCatchNonFatal to attempt
* factor the assertions in SignatureSpec to avoid \/
* deal with invariant \/
* make partial unification do what we want
* \/, parse*, and toNel
* many uses of the .right method
* a legitimate use of fromTryCatchThrowable
* rebuild maven pins
* further invariant \/
* OneAnd and Nel interface changes
* further Either games
* \/ and reformatting
* \/ in http-json
* \/ in http-json
* deprecations
* more invariance
* cleanup unused
* more invariance; http-json compiles
* final either follies
* small 2.12 extra incompatibility
* rebuild deps
* revisit a couple earlier fixes using nicer expressions I learned later
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* repin 2.12
* Log templateId & choice name (if present) on command submissions in the json API
changelog_begin
- [JSON API] The template id & choice name (if present) are now logged on command submissions in the Json API (at trace level)
changelog_end
* Move the template id & the choice into the logging context
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/CommandService.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/CommandService.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Fix compile error due to scala 2.12 collection differences
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
changelog_begin
- [JSON-API] The source and the path for incoming http requests are now logged
- [JSON-API] The http response for a request is now logged
changelog_end
* Don't duplicate log failed futures in the http json api (commandservice specific)
changelog_begin
- [JSON API] Errors which arise from the CommandService are not logged twice anymore (thus reducing noise)
changelog_end
* Fix duplicate logging of a failed future differently via adding another error case in CommandService.scala
* Restore old formatting
* Improve comment
* Remove the wrong changes I accidentally made, the functions are quite similar xD
* remove the additional type and just make the id optional & remove unecessary comment
* Clarify semantics of DIV_INT64 in LF spec.
Especially around negative operands. Adds a couple of unit tests to make sure the behavior stays this way, and that MOD_INT64 is consistent with DIV_INT64.
changelog_begin
changelog_end
* spelling
* scalafmt
* hardcode numbers in MOD_INT64 test
* Drop pkgIds argument from Engine.validatePackages
Doesn’t do anything useful, we just use it to validate that it matches
the keys of pkgs.
changelog_begin
changelog_end
* add test showing broken behavior
* Fix buggy tests
changelog_begin
changelog_end
Co-authored-by: Remy Haemmerle <Remy.Haemmerle@daml.com>
* oracle varray to json
CHANGELOG_BEGIN
CHANGELOG_END
* various cleanup, mostly removing printlns
* remove oracle connection from conversions and dependencies, cleaning up vestigial scala side varray code
* fix lazylist/stream issue by using java stream. remove braces
* Add proper error handling for missing contract keys
These are clearly not internal errors so we should not be calling
`crash` here. Canton in fact already started string matching on the
crash message which is definitely not what we want.
changelog_begin
changelog_end
* backwards compat
changelog_begin
changelog_end
* fix tests
changelog_begin
changelog_end
Nothing sensible we can do about ws than force a newer version. It’s
pulled in via transitive deps that don’t have newer versions yet.
For glob-parent we can do something more sensible by bumping grunt-browserify.
changelog_begin
changelog_end
This PR is part of the effort to improve engine Errors. Here we
classify all the errors the engine can report in 4 categories:
- Errors happening during package loading
- Error happening during command/transaction preprocessing
- Error happening during interpretation
- Error happening during transaction validation
part of #9974
CHANGELOG_BEGIN
CHANGELOG_END
* update Daml Hub auth in create-daml-app
Current documentation says the token is passed as a cookie. This is a
little bit more secure than passing it as a URL param, as it used to be.
Note that while it is no longer documented behaviour, the token is
actually still passed as a param (in addition to the cookie), so
existing users are not broken (yet). Still, better if the code we
generate for them matches our current docs.
CHANGELOG_BEGIN
CHANGELOG_END
* review comment
@garyverhaegen-da is taking care of 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9932), so they get pushed back to the end of the line.
Please do not merge this before #9932.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
* Log call to submitAndWaitRequest with the command id provided in the log ctx
changelog_begin
- [HTTP-JSON] Calls which trigger a submitAndWaitRequest are logged with the command id provided in the log ctx
changelog_end
* Require the request id to be also in the log ctx
* Log command kind in submitAndWaitRequest of CommandService.scala
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/CommandService.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Generate Bazel logs and upload to GCS
changelog_begin
changelog_end
* Move git_*_sha into variables template
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Now that we skip the scala_2_12 job this dep results in us never
running the release job which is clearly not intentional.
changelog_begin
changelog_end
