I haven’t found any conclusive information as to why ON COMMIT doesn’t
work incrementally but
https://docs.oracle.com/en/database/oracle/oracle-database/19/adjsn/json-query-rewrite-use-materialized-view-json_table.html#GUID-8B0922ED-C0D1-45BD-9588-B7719BE4ECF0
recommends that for rewriting (which isn’t what we do here but both
involve a materialized view on json_table).
Benchmarks:
before:
InsertBenchmark.run 1000 1 1000 avgt 5 0.327 ± 0.040 s/op
InsertBenchmark.run 1000 3 1000 avgt 5 0.656 ± 0.043 s/op
InsertBenchmark.run 1000 5 1000 avgt 5 1.034 ± 0.051 s/op
InsertBenchmark.run 1000 7 1000 avgt 5 1.416 ± 0.106 s/op
InsertBenchmark.run 1000 9 1000 avgt 5 1.734 ± 0.143 s/op
QueryBenchmark.run 1000 10 N/A avgt 5 0.071 ± 0.016 s/op
After:
Benchmark (batchSize) (batches) (numContracts) Mode Cnt Score Error Units
InsertBenchmark.run 1000 1 1000 avgt 5 0.217 ± 0.034 s/op
InsertBenchmark.run 1000 3 1000 avgt 5 0.232 ± 0.027 s/op
InsertBenchmark.run 1000 5 1000 avgt 5 0.226 ± 0.051 s/op
InsertBenchmark.run 1000 7 1000 avgt 5 0.225 ± 0.048 s/op
InsertBenchmark.run 1000 9 1000 avgt 5 0.232 ± 0.021 s/op
QueryBenchmark.run 1000 10 N/A avgt 5 0.080 ± 0.014 s/op
The difference in query times is just noise and changes across runs.
So we get the expected behavior of inserts being independent of the
total ACS size now. We could still explore if we gain something by
avoiding the materialized view to reduce constant factors but that’s
much less of an issue.
fixes#10243
changelog_begin
changelog_end
@akshayshirahatti-da is taking care of 1.15.0-snapshot.20210713.7343.0.1f35db17 (#10267), so they get pushed back to the end of the line.
Please do not merge this before #10267.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
* [In-memory fan-out] Oracle support
CHANGELOG_BEGIN
CHANGELOG_END
* Correct so Oracle backend works with fan out enabled - add implicit conversion and correct view defaulting for BLOB fields
Add missing tests to oracle append only spec as it lagged behind postgres and h2
Regen sql sha for oracle
CHANGELOG_BEGIN
CHANGELOG_END
* avoid global import of implicit that is not needed
* reduce duplication on conformance test targets
* update sha for oracle append only flyway scripts
Co-authored-by: Brian Healey <brian.healey@digitalasset.com>
* LF: change type from Try to Either in archive module
This is the first part of restructuring errors in archive module.
This is part of #9974.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* remove type alias
* apply stephen suggestion
* fix after rebase
* fix test
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Fix status check in collect_build_data
follow up to #10270 which caused the linux & macos builds to go
through but then screwed us over in collect_build_data. I hate CI.
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* CHANGELOG_BEGIN
Fix crash in ConfigManagement- and PackageManagement- services on duplicate submissionsIds from different participants.
Extend ledger-api-test-tool to cover the duplicate submissionId cases.
CHANGELOG_END
* Addressing review comments and fixing failing tests
* format
* address more review comments
* adopt after rebase to main
* [JSON-API] Refactor Endpoints.scala to use path directives etc.
changelog_begin
changelog_end
* Don't warn that the ev param in toRoute is not used
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Remove weird stuff to have nice stuff with the toRoute function
* Rename the toRoute function & remove comments as things are now clarified
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* release 1.15.0-snapshot.20210713.7343.0.1f35db17
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.
@akshayshirahatti-da is in charge of this release.
Commit log:
```
1f35db17c7 ledger-on-sql: Use random log entry ID allocation so we don't depend on `SeedService`. [KVL-1002] (#10255)
b8e2198873 Separate traces from warnings in engine. (#10253)
0a7f2b1a1b [JSON-API] Small refactoring to start using akka http routes internally (#10252)
6e8ec1d618 LF: Drop old depreated code (#10251)
de7a08fa7b [In-memory fan-out] Handle `null` submitters in getTransactionLogUpdates (#10248)
584169a2cc Participant state v1-to-v2 adaptors [KVL-1002] (#10246)
e4d7cd764a LF: Deprecate com.daml.lf.data.TryOps.Bracket (#10249)
b59f36591d [JSON-API] Correctly extract the request source URL/IP (#10244)
edaf541063 Adapt `ledger-api-bench-tool` to work with LR [DPP-424] (#10171)
6fe6ae075d LF: Make DarReader ZipEntries immulatble (#10243)
05e5218d06 Consolidating EventStorageBackend (#10224)
3fd3abfb26 Instrument buffer after contract state events stream (#10209)
643a2de006 v2 participant state API draft [KVL-998] (#10210)
5abcd0482e Adds Unit tests for parallel-ingestion [DPP-455] (#10238)
3cdedcf885 kvutils: Extract validators from TransactionCommitter [KVL-1015] (#10235)
00d622f268 Make @UNTIL-LF exclusive, add @UNTIL-LF-FEATURE. (#10240)
2bcbd4e177 es: switch to persistent nodes (#10236)
7f4bc2a472 update compat versions for 1.14.2 (#10233)
4eba4b00e8 Ledger: decouple BD value serialization library from archive library. (#10218)
729afa8c7b Write a few lines on how to run tests against an Oracle database (#10230)
2b67ebb5d4 tf: refactor appr var (#10232)
202b7f7ae7 add akshay to appr team (#10229)
2c5410c229 Release SDK 1.14.2 (#10226)
89d87ad17c update compat versions for 1.14.2-snapshot.20210708.7135.0.aa297840 (#10223)
48393444df TODO Cleanup: QueryNonPruned [DPP-459] (#10225)
0bea5e3ef7 Allow retriable lookupMaximumLedgerTime on contention (#10211)
ca2a9e9bbb Release 1.14.2 snapshot (#10222)
999577a1a7 tweak ES cluster (#10219)
6680d368a1 Warn on DA.Exception import (#10201)
f19f5b0821 LF: Simplify DarReader (#10217)
8f1a8f2390 Release DAML SDK 1.14.1 (#10204)
ebb76dca4c LF: reorganize errors in com.daml.lf.archive (#10213)
69646451f2 Improvements for validation tests (#10214)
5f124c3b64 Avoid collision in execution log postfix (#10205)
9e27ae0d85 Reshuffle cocreature in daml-lf codeowners (#10215)
41b8448b17 LF: Simplify archive reader. (#10208)
38734f02d7 es-feed: ignore invalid files (#10207)
6586dde11e Move exercise context computation to the client side (#10199)
c92c67832f Mark Daml profiler as stable (#10203)
cb1f4ec773 ci/windows: disable spool (#10200)
2326d425bc Publish execution logs on unix platforms (#10194)
42693452ce Drop cleanup of node_modules (#10195)
c8faed8a17 Deduplicate Java codegen running instructions (#10185)
e4c8c390f2 update compat versions for 1.14.1-snapshot.20210706.7133.0.8ec16e94 (#10196)
0d881f5e2b Improvements to the documentation with regards to offsets (#10180)
1d5ba4fa42 feed elasticsearch cluster (#10193)
e2bdca6be9 Use PartialFunction for more concise code. (#10191)
fe98c48b65 release 1.14.1-snapshot.20210706.7133.0.8ec16e94 (#10186)
4fe626a055 Drop logging on CompletionSource (#10190)
582aa5f08c Fix a typo in an exceptions example. (#10188)
8750c0c47c reduce noise in Scala bindings (#10187)
98b5ffee01 Add divulgence warning and test in script service. (#10179)
8578e56aa7 Tests for transaction validation (#10167)
05a72a3a15 update compat versions for 1.15.0-snapshot.20210705.7286.0.62aabcc4 (#10184)
7b93923c16 [In-memory fan-out] Performance optimizations [DPP-470] (#10127)
6c49619565 Document how to run the Java codegen from the Daml assistant (#10181)
61aa774988 Release SDK 1.15 RC (#10182)
```
Changelog:
```
- [JSON-API] If the service is put behind a proxy filling either of these headers X-Forwarded-For & X-Real-Ip then these will now be respected for logging the request source ip/url
- [Daml Compiler] Importing DA.Exception on LF < 1.14 now produces a
warning that exceptions require Daml-LF >= 1.14.
- [Daml Profiler] The Daml profiler is now a stable feature.
[Docs] Improvements to the documentation with regards to offsets
[Docs] Document how to use the Java codegen using the Daml assistant
```
CHANGELOG_BEGIN
CHANGELOG_END
* 1.16 not 1.15
changelog_begin
changelog_end
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Add a benchmark for contract insertion in the JSON API
Unfortunately the results seem to match up with my initial benchmark
in #10234
Benchmark (batchSize) (batches) (numContracts) Mode Cnt Score Error Units
InsertBenchmark.run 1000 1 1000 avgt 5 336.674 ± 42.058 ms/op
InsertBenchmark.run 1000 3 1000 avgt 5 787.086 ± 223.018 ms/op
InsertBenchmark.run 1000 5 1000 avgt 5 1181.041 ± 317.017 ms/op
InsertBenchmark.run 1000 7 1000 avgt 5 1531.185 ± 341.060 ms/op
InsertBenchmark.run 1000 9 1000 avgt 5 1945.345 ± 436.352 ms/op
Score should ideally be more or less constant but it goes up very
significantly as the total ACS size changes
fixes#10245
changelog_begin
changelog_end
* throughput -> average time
changelog_begin
changelog_end
This has now screwed us over for two releases (1.14 and currently
blocking 1.15) because we didn’t backport the change. While we could
backport this, it is annoying and provides little to no benefit given
that a failure here is harmless so let’s just ignore failures here.
changelog_begin
changelog_end
* Add a ContractDao benchmark
This PR adds a simple benchmark that uses the ContractDao directly and
is therefore a bit more fine-grained and easier to analyze than the
gatling benchmarks. I expect we’ll want to extend this, this really
only tests queries on reasonably large size ACS filtered by party but
let’s start somewhere.
fixes#10247
changelog_begin
changelog_end
* Factorize
changelog_begin
changelog_end
I don't see a reason why it's part of the participant state API, and
it definitely doesn't need to change between v1 and v2.
CHANGELOG_BEGIN
- [Integration Kit] The class ``SeedService`` has been moved from the
*participant-state* Maven package to the *participant-integration-api*
Maven package, under the Java package name
``com.daml.platform.apiserver`` to reflect its usage by the API
server, not the participant state API. If you use this class directly,
you will need to change your imports.
CHANGELOG_END
* ledger-on-sql: Abstract out the log entry ID allocator.
This is so we don't need to depend on `SeedService`.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Remove `allocateRandomLogEntryId` from production code.
* kvutils: Remove `SubmissionValidator.create`, as it's only for tests.
And remove default parameters for
`SubmissionValidator.createForTimeMode`.
* ledger-on-sql: Use the random log entry ID allocator.
In Sandbox, we still use the seed service, but this is not necessary
for ledger-on-sql tests.
* kvutils: Make `LogEntryIdAllocator.random` an `object`.
* ledger-on-sql: Move `RandomLogEntryIdAllocator` here.
It's not used anywhere else.
* ledger-on-sql: Add some documentation about the application code.
Just to make it clear it's not used in production.
* Separate traces from warnings in engine.
I decided to separate the engine warnings from the tracelog after all,
because I think it will make testing and maintenance easier in the
long run.
Part of #9947, follow up from #10179
changelog_begin
changelog_end
* scalafmt
* Apply suggestions from code review
Co-authored-by: Remy <remy.haemmerle@daml.com>
* dont use case class for WarningLog
* revert TraceLog changes from last PR
* Scala 2.12 doesnt have ArrayBuffer.addOne :(
* remove isWarnEnabled check
Co-authored-by: Remy <remy.haemmerle@daml.com>
* [JSON-API] Correctly extract the request source URL/IP
changelog_begin
- [JSON-API] If the service is put behind a proxy filling either of these headers X-Forwarded-For & X-Real-Ip then these will now be respected for logging the request source ip/url
changelog_end
* Return to the simple http server start code
* Remove unused import
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Print ledger-api-bench-tool final report when a stream in ended by shutting down API server
CHANGELOG_BEGIN
CHANGELOG_END
* Log ledger-api-bench-tool failure with specific header to make it compatible with the lr-sidecar
* Print final report on graceful shutdown even if streams are not complete
* trigger build
* LF: Make DarReader ZipEntries immulatble
CHANGELOG_BEGIN
CHANGELOG_END
* optim
* Revert "optim"
This reverts commit 3b25a8c68b.
* a function should not close a stream it didn’t open
* typo
* we do not support dar without manifest !
A few small tweaks, but the most important change is giving up on
preemptible instances (for now at least), because I saw GCP kill 8 out
of the 10 nodes at exactly the same time and I can't really expect the
cluster to sruvive that.
CHANGELOG_BEGIN
CHANGELOG_END
Two changes at the Terraform level, both with no impact on the actual
GCP state:
- There is no reason to make this value a `variable`: variables in
Terraforma are meant to be supplied at the CLI. `local` is the right
abstraction here (i.e. set in the file directly).
- Using an unordered `for_each` set rather than a list so we don't have
positional identity, meaning when adding someone at the top we don't
need to destroy and recreate everyone else.
CHANGELOG_BEGIN
CHANGELOG_END
