* Run scenarios in off-ledger machine
This PR builds on the previous PR that split scenario execution in two
different speedy machines and now actually makes the machine that runs
scenarios run in off-ledger mode just like we handle Daml Script.
This required a bunch of refactoring to make it nice so apologies for
the slightly large PR. Hopefully it’s still relatively easy to follow
and luckily it deletes more code than it adds.
changelog_begin
changelog_end
* Apply suggestions from code review
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* review comments
changelog_begin
changelog_end
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
This morning we started with very restricted CI pools (2/6 for Windows
and 7/20 for Linux), apparently because the region we run in (us-east1)
has three zones, two of them were unable to allocate new nodes, and the
default policy is to distribute nodes evenly between zones.
I've manually changed the distribution policy. Unfortunately this option
is not yet available in our version of the GCP Terraform plugin.
CHANGELOG_BEGIN
CHANGELOG_END
changelog_begin
- [JSON-API] The database operations (regardless of in-memory or postgres) contain now metrics for fetching contracts by id or key (seperate metrics foreach)
- [JSON-API] The time for a repsonse payload construction of a request is now also tracked in a metric
changelog_end
It makes no sense to pass NotVisible to Speedy especially since that
is not how visibility is handled for the Engine. Also it lets us
delete code and I like deleted code.
changelog_begin
changelog_end
* [JSON-API] Add more timing metrics
changelog_begin
- [JSON-API] Timing metrics are now available for party management, package management, command submission and query endpoints.
- [JSON-API] Also added a timing metric for parsing and decoding of incoming json payloads
changelog_end
* Add comments to new metrics
* Split metrics up more & remove obsolete metric
* Split up timers for query endpoints
* Creating OR predicate to reduce length of string literals
CHANGELOG_BEGIN
CHANGELOG_END
* removing unnecessary thrown exception
CHANGELOG_BEGIN
CHANGELOG_END
* switching to camel case for constants
CHANGELOG_BEGIN
CHANGELOG_END
* run format
CHANGELOG_BEGIN
CHANGELOG_END
* remove tolist conversion
CHANGELOG_BEGIN
CHANGELOG_END
* scalafmt
Co-authored-by: Brian Healey <brian.healey@digitalasset.com>
* Use ScenarioRunner.submit in Daml Script
changelog_begin
changelog_end
* privatize ledger variable
changelog_begin
changelog_end
* drop space
changelog_begin
changelog_end
* nvarchar2 keys are text-incompatible, but varchar2 keys are fine
* commit the ACS update before query
* add changelog
CHANGELOG_BEGIN
- [JSON API] The Oracle database schema has changed; if using
``--query-store-jdbc-config``, you must rebuild the database by adding
``,createSchema=true``. See #9895.
CHANGELOG_END
* expand the InitDdl set to include materialized views
* replace search index with a materialized view that expands the stakeholders
* allow materialized views to be created in Oracle testing
* join and query the contract_stakeholders table for party-set membership
- restoring a few elements removed by 3e6661128d (#9484)
This solves two warts in the code:
- the validate/createUnsafe double-parse, because scopt doesn't let you flatMap;
- the non-JdbcConfig sub-configs appeared to need to know the JDBC drivers when
they really don't, because of a quirk in the inherited implementation
That coherence of scopt.Read instances calls for all its dependencies to be
coherent leads us to treat supportedJdbcDriverNames as a nullary typeclass
instance. This is a nullary typeclass by the same justification as
SupportedJdbcDriver; see scaladoc there for more.
And we solve the latter problem by...adding a type parameter, how else.
CHANGELOG_BEGIN
CHANGELOG_END
* Preparation: pull up initialisation
* Preparation: introduce partyArrayContext for more flexibility
* Preparation: add append-only-from-scratch feature to FlywayMigrations
* Add squashed Flyway migration for append-only H2
* Add support for H2 StorageBackend
* Extend conformance test suites for H2
* Extend JDBCLedgerDao unit test suites for H2
* Extend sandbox-classic unit test suites for H2
* Fixing typo, some renames based on review
changelog_begin
changelog_end
@stefanobaghino-da is taking care of 1.14.0-snapshot.20210615.7169.0.adeba206 (#10018), so they get pushed back to the end of the line.
Please do not merge this before #10018.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
* Do not terminate transactions/transaction trees streams when no end-offset provided in the ledger-api-bench-tool
CHANGELOG_BEGIN
CHANGELOG_END
* Minor change
* Split scenario & ledger execution
This PR by no means aims to solve everything we can do here. It is
rather the minimal change that I could get to work that provides us
with one Speedy machine for scenario execution (which is still an
on-ledger machine to avoid having to change the callsites to much in
this PR) and one speedy machine per submission.
There is tons of cleanup we can do afterwards but this should
hopefully set the right foundations.
changelog_begin
changelog_end
* [JSON-API] Concurrent query etc. metrics
changelog_begin
- [JSON-API] The metrics which describe the amount of these concurrent events is now available:
- running http request
- command submissions
- package allocations
- party allocations
changelog_end
* Rename running metrics to throughput ones & add comments on the metrics
* Adjust names of other existing metrics too, to have for the json api a more consistent metrics naming
* truncate party name for 64 chars for multi-valued submitters to avoid excessively long metrics name
CHANGELOG_BEGIN
[metrics] Limit size of multi party metrics to ease consumption
CHANGELOG_END
* Use just first party for dynamic metrics when there are multiple parties involved
* Log context of all updates written to the database
CHANGELOG_BEGIN
For every update in the index db log the full context at the INFO level.
CHANGELOG_END
* one more missing log
Commit log:
```
adeba20659 DPP-403 log flyway progress (#9964)
401069ef00 prepare for upgrade to Scalaz 7.3.3 (#9997)
3ad9cd58d9 [JSON-API] Log templateId & choice name (if present) on command submissions in the json API (#10005)
af144e8592 LF: Structure package loading errors (#9980)
3583fd1885 [JSON-API] Log source and path of incoming http requests & the response with info level (#10007)
e49c7c8db7 [In-memory fan-out] BufferedTransactionsReader implementation [DPP-413] [DPP-414] (#9882)
acb77336c8 Release 1.14 RC 2 (#10006)
3d2d4b9696 Don't duplicate log failed futures in the http json api (commandservice specific ones) (#9990)
0c32cf2968 Clarify the semantics of DIV_INT64 in LF spec. (#10003)
b79e739196 Fix Type (#10004)
998f106ee6 Drop pkgIds argument from Engine.validatePackages (#9986)
ae5010f3ef [Mutable state cache] Start cache from latest ledger head (#9998)
dd42a5a641 update maven_install_2.12.json (#9999)
66d38cd24d Storage query abstraction first sketch (#9959)
c75013622c Just print out the operator name for the operation logging in CommandService.scala (#9989)
b6faace335 Changing oracle backend from using varray to json array (#9943)
603a4a4b42 Add date to log statements in the HTTP JSON service (#9988)
836a82a6b6 Add proper error handling for missing contract keys (#9972)
85667af7be Address npm vulnerabilities (#9978)
d7d6e37785 LF: Structure Engine Errors. (#9971)
0adbbdcd58 update Daml Hub auth in create-daml-app (#9973)
997a7d449c Use symlinks on Windows (#9969)
e4258e61b1 docs: tweak Exception intro (#9967)
b2c4ef813d rotate release duty after 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9933)
de7a462082 Log call to submitAndWaitRequest with the command id provided in the log ctx (#9940)
630d021331 Collect build-event-stream (#9831)
26e199a719 Rerelease 1.11.2 snapshot (#9965)
8399039a8a Remove broken dep from release job (#9963)
e1db529162 DPP-390 configurable event decoding parallelism (#9931)
cc7af93afc [docs] Remove link to the #daml-contributors (#9928)
f7ac3ba784 Release 1.11.2 snapshot attempt 2 (#9962)
4634147ed1 Adds Indexer state to GRPC health checks [DPP-434] (#9951)
28c567b53d Release SDK 1.11.2 (#9957)
5f0f5756bf Define IncompleteTransaction for use by scenario-service (#9952)
b7fd8338c4 Increase `eventually` timeout when waiting for queue to be killed (#9955)
ca0fb16718 Fix collect_build_data job (#9956)
42030b25ed Skip Scala_2_12 job on releases (#9954)
5e1ddfbd5e LF: Fix specification of ArithmeticError (#9949)
3093a98b87 Forward port additional contract keys validation tests (#9944)
e541558f18 update NOTICES file (#9946)
db792a9774 Proxy ledger API health endpoint in JSON API health endpoint (#9945)
13767b5c5d Load the correct logback file for the http json service respecting the deployment situation (#9938)
4e49cf6814 `ledger-api-bench-tool` - Active contracts stream, completions stream [DPP-398, DPP-399] (#9857)
fce8e54469 release 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9932)
ab55fb2546 Handle module-prefixes in Java codegen (#9929)
7375e28271 Force resolution for css what to 5.0.1 (#9925)
```
Changelog:
```
- [JSON API] The template id & choice name (if present) are now logged on command submissions in the Json API (at trace level)
- [JSON-API] The source and the path for incoming http requests are now logged
- [JSON-API] The http response for a request is now logged
- [JSON API] Errors which arise from the CommandService are not logged twice anymore (thus reducing noise)
- [JSON API] log statements now include the date next to the time
- [HTTP-JSON] Calls which trigger a submitAndWaitRequest are logged with the command id provided in the log ctx
[Integration Kit] The state of the participant indexer can now be checked via the GRPC health endpoint
- [JSON API] The healthcheck endpoint on the JSON API now proxies the
health check endpoint on the underlying ledger. Previously it only
queried for ledger end to determine ledger health.
[HTTP-JSON]
- fixed that json log output could not be enabled via cli options except via usage of env vars
- [Integration Kit] - ledger-api-bench-tool - reading active contract streams
- [Integration Kit] - ledger-api-bench-tool - Reading completions stream
- [Java Codegen] The Java codegen will now pick up the
`module-prefixes` field from `daml.yaml` which can be used to handle
module name collisions between different DALFs.
```
CHANGELOG_BEGIN
CHANGELOG_END
* Force css-what resolution in compiler/daml-extension & fully regenerate the lock file
changelog_begin
changelog_end
* Don't force resolution as it's unecessary
This is a preparation for H2 integration: reworking/opening up the ingestion DSL
* Pull more execution logic to PGTable, as preparing for batch insert execution
* Pull out Table from PGTable, add support for deletion (preparation for JDBC batch execution)
* Abstract over Schema (preparation for JDBC batch execution)
* Opening up Field, abstract over table creation (preparation for JDBC batch execution)
* Move code to common (preparation for JDBC batch execution)
* Add generic JDBC batch execution
* Add generic JDBC batch deletion and some more generic Field-s
* Pulling out PGSmallintOptional to support the postgres flavor
* Generalizing idempotentInsert in FieldStrategy
* Capture non empty data logic and comment in a util function
* Seal access to Schema internals
changelog_begin
changelog_end
* Add information from the jwtpayload to the logging context
changelog_begin
- [JSON API] For applicable requests actAs, readAs, applicationId & ledgerId are included in the log context
changelog_end
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Revert changes to make the function generic
* Create JwtPayloadG trait from which both payload variants inherit
* Reduce code duplication in Endpoints.scala
* Apply review suggestion
* Update test name to reflect field name changes
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.
@stefanobaghino-da is in charge of this release.
Commit log:
```
adeba20659 DPP-403 log flyway progress (#9964)
401069ef00 prepare for upgrade to Scalaz 7.3.3 (#9997)
3ad9cd58d9 [JSON-API] Log templateId & choice name (if present) on command submissions in the json API (#10005)
af144e8592 LF: Structure package loading errors (#9980)
3583fd1885 [JSON-API] Log source and path of incoming http requests & the response with info level (#10007)
e49c7c8db7 [In-memory fan-out] BufferedTransactionsReader implementation [DPP-413] [DPP-414] (#9882)
acb77336c8 Release 1.14 RC 2 (#10006)
3d2d4b9696 Don't duplicate log failed futures in the http json api (commandservice specific ones) (#9990)
0c32cf2968 Clarify the semantics of DIV_INT64 in LF spec. (#10003)
b79e739196 Fix Type (#10004)
998f106ee6 Drop pkgIds argument from Engine.validatePackages (#9986)
ae5010f3ef [Mutable state cache] Start cache from latest ledger head (#9998)
dd42a5a641 update maven_install_2.12.json (#9999)
66d38cd24d Storage query abstraction first sketch (#9959)
c75013622c Just print out the operator name for the operation logging in CommandService.scala (#9989)
b6faace335 Changing oracle backend from using varray to json array (#9943)
603a4a4b42 Add date to log statements in the HTTP JSON service (#9988)
836a82a6b6 Add proper error handling for missing contract keys (#9972)
85667af7be Address npm vulnerabilities (#9978)
d7d6e37785 LF: Structure Engine Errors. (#9971)
0adbbdcd58 update Daml Hub auth in create-daml-app (#9973)
997a7d449c Use symlinks on Windows (#9969)
e4258e61b1 docs: tweak Exception intro (#9967)
b2c4ef813d rotate release duty after 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9933)
de7a462082 Log call to submitAndWaitRequest with the command id provided in the log ctx (#9940)
630d021331 Collect build-event-stream (#9831)
26e199a719 Rerelease 1.11.2 snapshot (#9965)
8399039a8a Remove broken dep from release job (#9963)
e1db529162 DPP-390 configurable event decoding parallelism (#9931)
cc7af93afc [docs] Remove link to the #daml-contributors (#9928)
f7ac3ba784 Release 1.11.2 snapshot attempt 2 (#9962)
4634147ed1 Adds Indexer state to GRPC health checks [DPP-434] (#9951)
28c567b53d Release SDK 1.11.2 (#9957)
5f0f5756bf Define IncompleteTransaction for use by scenario-service (#9952)
b7fd8338c4 Increase `eventually` timeout when waiting for queue to be killed (#9955)
ca0fb16718 Fix collect_build_data job (#9956)
42030b25ed Skip Scala_2_12 job on releases (#9954)
5e1ddfbd5e LF: Fix specification of ArithmeticError (#9949)
3093a98b87 Forward port additional contract keys validation tests (#9944)
e541558f18 update NOTICES file (#9946)
db792a9774 Proxy ledger API health endpoint in JSON API health endpoint (#9945)
13767b5c5d Load the correct logback file for the http json service respecting the deployment situation (#9938)
4e49cf6814 `ledger-api-bench-tool` - Active contracts stream, completions stream [DPP-398, DPP-399] (#9857)
fce8e54469 release 1.14.0-snapshot.20210608.7123.0.3cb8d5c2 (#9932)
ab55fb2546 Handle module-prefixes in Java codegen (#9929)
7375e28271 Force resolution for css what to 5.0.1 (#9925)
```
Changelog:
```
- [JSON API] The template id & choice name (if present) are now logged on command submissions in the Json API (at trace level)
- [JSON-API] The source and the path for incoming http requests are now logged
- [JSON-API] The http response for a request is now logged
- [JSON API] Errors which arise from the CommandService are not logged twice anymore (thus reducing noise)
- [JSON API] log statements now include the date next to the time
- [HTTP-JSON] Calls which trigger a submitAndWaitRequest are logged with the command id provided in the log ctx
[Integration Kit] The state of the participant indexer can now be checked via the GRPC health endpoint
- [JSON API] The healthcheck endpoint on the JSON API now proxies the
health check endpoint on the underlying ledger. Previously it only
queried for ledger end to determine ledger health.
[HTTP-JSON]
- fixed that json log output could not be enabled via cli options except via usage of env vars
- [Integration Kit] - ledger-api-bench-tool - reading active contract streams
- [Integration Kit] - ledger-api-bench-tool - Reading completions stream
- [Java Codegen] The Java codegen will now pick up the
`module-prefixes` field from `daml.yaml` which can be used to handle
module name collisions between different DALFs.
```
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
* upgrade scalacheck to 1.14.3
* regenerate maven_install files
* some different names and implicits
* remove some fromTryCatchNonFatal
* more porting
* port fromTryCatchNonFatal to attempt
* factor the assertions in SignatureSpec to avoid \/
* deal with invariant \/
* make partial unification do what we want
* \/, parse*, and toNel
* many uses of the .right method
* a legitimate use of fromTryCatchThrowable
* rebuild maven pins
* further invariant \/
* OneAnd and Nel interface changes
* further Either games
* \/ and reformatting
* \/ in http-json
* \/ in http-json
* deprecations
* more invariance
* cleanup unused
* more invariance; http-json compiles
* final either follies
* small 2.12 extra incompatibility
* rebuild deps
* revisit a couple earlier fixes using nicer expressions I learned later
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* repin 2.12
* Log templateId & choice name (if present) on command submissions in the json API
changelog_begin
- [JSON API] The template id & choice name (if present) are now logged on command submissions in the Json API (at trace level)
changelog_end
* Move the template id & the choice into the logging context
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/CommandService.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/CommandService.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Fix compile error due to scala 2.12 collection differences
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
changelog_begin
- [JSON-API] The source and the path for incoming http requests are now logged
- [JSON-API] The http response for a request is now logged
changelog_end
* Don't duplicate log failed futures in the http json api (commandservice specific)
changelog_begin
- [JSON API] Errors which arise from the CommandService are not logged twice anymore (thus reducing noise)
changelog_end
* Fix duplicate logging of a failed future differently via adding another error case in CommandService.scala
* Restore old formatting
* Improve comment
* Remove the wrong changes I accidentally made, the functions are quite similar xD
* remove the additional type and just make the id optional & remove unecessary comment
* Clarify semantics of DIV_INT64 in LF spec.
Especially around negative operands. Adds a couple of unit tests to make sure the behavior stays this way, and that MOD_INT64 is consistent with DIV_INT64.
changelog_begin
changelog_end
* spelling
* scalafmt
* hardcode numbers in MOD_INT64 test
* Drop pkgIds argument from Engine.validatePackages
Doesn’t do anything useful, we just use it to validate that it matches
the keys of pkgs.
changelog_begin
changelog_end
* add test showing broken behavior
* Fix buggy tests
changelog_begin
changelog_end
Co-authored-by: Remy Haemmerle <Remy.Haemmerle@daml.com>
