* [JSON-API] Shutdown on startup if the db connection is invalid
changelog_begin
- [JSON-API] The json api now correctly shutdowns at startup if the provided db connection is invalid in case of `createSchema=false`
changelog_end
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Main.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Switch ContractDao to use a HikariCP connection pool
CHANGELOG_BEGIN
CHANGELOG_END
* missed conn pool changes for PostgresTest and ContractDaoBenchmark
* shutdown db access await threadpool and fix formatting
* custom pool sizes for Prod and Integration similar to DbTriggerDao
* cleanup contract dao connection pool
* simplify Dao shutdown
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* remove redundant config setting
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* fix code formatting issue, NonUnitStatments warning
* use doobie 0.9.0 Fragment-in-Fragment interpolation in json-api db-backend
Since tpolecat/doobie#1045 (and therefore 4ca02e0eb6) doobie has
supported interpolating fragments in fragments. We've used this feature
for several fragments written since #7618, but have left the ones
written before alone to use ++. Here we change that where it
meaningfully clarifies the SQL subexpression.
Note that this does not entail a Put or Write instance for Fragment.
You cannot abstract over Fragment and arbitrary interpolated data in
this way, because Fragments are not treated as positional parameters;
that would mean being able to put arbitrary SQL substrings in positional
parameters.
CHANGELOG_BEGIN
CHANGELOG_END
* scalafmt
* useless whitespace accidentally removed
* new projection for aggregated matched-queries
We can redo all the template-ID matches (and payload query matches, if
needed) in the SELECT projection clause to emit a list of matchedQueries
indices SQL-side.
CHANGELOG_BEGIN
CHANGELOG_END
* selectContractsMultiTemplate always returns one query
* factoring
* remove multiquery deduplication from ContractDao
* test simplest case of projectedIndex; remove uniqueSets tests
* remove uniqueSets
* add more test cases for the 3 main varieties of potential inputs
* remove uniqueSets tests that were commented for reference
* remove unneeded left-join
* scala 2.12 port
* port Map test order to 2.12
* use SortedMap so the Scala version tests are unified
- suggested by @cocreature; thanks
* Support deletion of a large number of contracts
fixes#10339
There are two orthogonal issues here:
1. scalaz’s toVector from the Foldable[Set] instance
stackoverflows. I’ve just avoided using that altogether.
2. Oracle doesn’t like more than 1k items in the IN clause. I chunked
the queries into chunks of size 1k to fix this.
changelog_begin
- [JSON API] Fix an error where transactions that delete a large
number of contracts resulted in stackoverflows with the PostgreSQL
backend and database errors with Oracle.
changelog_end
* fix benchmark
changelog_begin
changelog_end
* Update ledger-service/db-backend/src/main/scala/com/digitalasset/http/dbbackend/Queries.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Update ledger-service/db-backend/src/main/scala/com/digitalasset/http/dbbackend/Queries.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* that's not how you foldA
changelog_begin
changelog_end
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Printing stacktraces is consider an antipattern by some people and
gets flagged by VeraCode. While this shouldn’t actually be an issue
here, it is also not super useful so dropping it is easier than
arguing that this is a false positive.
changelog_begin
changelog_end
changelog_begin
- [JSON-API] Connection tries from the json api to the ledger now include the logging context, more specifically the instance_uuid is included in each logging statement.
changelog_end
Was curious if there were any relevant performance improvements in
newer versions. Looks like the answer is no but we might as well
upgrade anyway.
changelog_begin
changelog_end
* daml-lf/data: Move ID aliases to `Ref` from _ledger-api-common_.
This allows us to remove a lot of dependencies on _ledger-api-common_,
and use these aliases in other places where that module is not used.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Remove an unused import.
* http-json-oracle: Remove `ledger-api-common` as a dependency.
* bindings-rxjava: Remove a now-unused dependency.
* [DOCS] Add documentation for the JSON API metrics
changelog_begin
- [JSON-API] You can now find a section `Metrics` in the http-json api documentation explaining how to enable metrics and which are available
changelog_end
* Fix rst build warnings
* Update docs/source/json-api/metrics.rst
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Adapt metrics doc to state that it IS an exhaustive list and remove wrong copy pasta text & add info about prometheus
* Update the legal values for the metrics reporter cli option
* shorten the description, the change prior was unnecessary ._.
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* [JSON-API] Log json request & response bodies in debug
This also readds logging of incoming requests and the responses which are being send out.
changelog_begin
- [JSON-API] Logging of the request and response bodies are now available for appropriate requests if the chosen log level is equal or lower than DEBUG. These can then be found in the logging context of the request begin & end log messages (The field names in the ctx are "request_body" and "response_body").
changelog_end
* Move the http request throughput marking to the right place including the logging of the processing time
* Ensure that the processing time measuring is implemented consistent
* participant-state: Remove the `ParticipantId` alias.
This alias adds nothing. By using `Ref.ParticipantId` directly, many
packages can remove their dependency on the _participant-state_ package.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-state: Remove the `PackageId` and `Party` aliases.
They don't add anything. Let's just use `Ref`.
* kvutils: Restore missing compat imports.
This PR extends the test to test a full matrix of different party &
template id numbers. Summarizing the results, as expected we index by
party but not by template id:
Benchmark (batchSize) (extraParties) (extraTemplates) Mode Cnt Score Error Units
QueryBenchmark.run 10000 1 0 avgt 5 0.255 ± 0.064 s/op
QueryBenchmark.run 10000 10 0 avgt 5 0.304 ± 0.245 s/op
QueryBenchmark.run 10000 100 0 avgt 5 0.296 ± 0.064 s/op
Benchmark (batchSize) (extraParties) (extraTemplates) Mode Cnt Score Error Units
QueryBenchmark.run 10000 0 1 avgt 5 0.277 ± 0.037 s/op
QueryBenchmark.run 10000 0 10 avgt 5 0.479 ± 0.301 s/op
QueryBenchmark.run 10000 0 100 avgt 5 2.131 ± 0.497 s/op
We know how to fix that so I’ll get on that.
changelog_begin
changelog_end
CHANGELOG_BEGIN
* [Integration Kit] Removed trace_context field from Ledger API and its bindings as we now have trace context propagation support via gRPC metadata. If you are constructing or consuming Ledger API requests or responses directly, you may need to update your code.
CHANGELOG_END
I haven’t found any conclusive information as to why ON COMMIT doesn’t
work incrementally but
https://docs.oracle.com/en/database/oracle/oracle-database/19/adjsn/json-query-rewrite-use-materialized-view-json_table.html#GUID-8B0922ED-C0D1-45BD-9588-B7719BE4ECF0
recommends that for rewriting (which isn’t what we do here but both
involve a materialized view on json_table).
Benchmarks:
before:
InsertBenchmark.run 1000 1 1000 avgt 5 0.327 ± 0.040 s/op
InsertBenchmark.run 1000 3 1000 avgt 5 0.656 ± 0.043 s/op
InsertBenchmark.run 1000 5 1000 avgt 5 1.034 ± 0.051 s/op
InsertBenchmark.run 1000 7 1000 avgt 5 1.416 ± 0.106 s/op
InsertBenchmark.run 1000 9 1000 avgt 5 1.734 ± 0.143 s/op
QueryBenchmark.run 1000 10 N/A avgt 5 0.071 ± 0.016 s/op
After:
Benchmark (batchSize) (batches) (numContracts) Mode Cnt Score Error Units
InsertBenchmark.run 1000 1 1000 avgt 5 0.217 ± 0.034 s/op
InsertBenchmark.run 1000 3 1000 avgt 5 0.232 ± 0.027 s/op
InsertBenchmark.run 1000 5 1000 avgt 5 0.226 ± 0.051 s/op
InsertBenchmark.run 1000 7 1000 avgt 5 0.225 ± 0.048 s/op
InsertBenchmark.run 1000 9 1000 avgt 5 0.232 ± 0.021 s/op
QueryBenchmark.run 1000 10 N/A avgt 5 0.080 ± 0.014 s/op
The difference in query times is just noise and changes across runs.
So we get the expected behavior of inserts being independent of the
total ACS size now. We could still explore if we gain something by
avoiding the materialized view to reduce constant factors but that’s
much less of an issue.
fixes#10243
changelog_begin
changelog_end
* LF: change type from Try to Either in archive module
This is the first part of restructuring errors in archive module.
This is part of #9974.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* remove type alias
* apply stephen suggestion
* fix after rebase
* fix test
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* [JSON-API] Refactor Endpoints.scala to use path directives etc.
changelog_begin
changelog_end
* Don't warn that the ev param in toRoute is not used
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Remove weird stuff to have nice stuff with the toRoute function
* Rename the toRoute function & remove comments as things are now clarified
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Add a benchmark for contract insertion in the JSON API
Unfortunately the results seem to match up with my initial benchmark
in #10234
Benchmark (batchSize) (batches) (numContracts) Mode Cnt Score Error Units
InsertBenchmark.run 1000 1 1000 avgt 5 336.674 ± 42.058 ms/op
InsertBenchmark.run 1000 3 1000 avgt 5 787.086 ± 223.018 ms/op
InsertBenchmark.run 1000 5 1000 avgt 5 1181.041 ± 317.017 ms/op
InsertBenchmark.run 1000 7 1000 avgt 5 1531.185 ± 341.060 ms/op
InsertBenchmark.run 1000 9 1000 avgt 5 1945.345 ± 436.352 ms/op
Score should ideally be more or less constant but it goes up very
significantly as the total ACS size changes
fixes#10245
changelog_begin
changelog_end
* throughput -> average time
changelog_begin
changelog_end
* Add a ContractDao benchmark
This PR adds a simple benchmark that uses the ContractDao directly and
is therefore a bit more fine-grained and easier to analyze than the
gatling benchmarks. I expect we’ll want to extend this, this really
only tests queries on reasonably large size ACS filtered by party but
let’s start somewhere.
fixes#10247
changelog_begin
changelog_end
* Factorize
changelog_begin
changelog_end
I don't see a reason why it's part of the participant state API, and
it definitely doesn't need to change between v1 and v2.
CHANGELOG_BEGIN
- [Integration Kit] The class ``SeedService`` has been moved from the
*participant-state* Maven package to the *participant-integration-api*
Maven package, under the Java package name
``com.daml.platform.apiserver`` to reflect its usage by the API
server, not the participant state API. If you use this class directly,
you will need to change your imports.
CHANGELOG_END
* [JSON-API] Correctly extract the request source URL/IP
changelog_begin
- [JSON-API] If the service is put behind a proxy filling either of these headers X-Forwarded-For & X-Real-Ip then these will now be respected for logging the request source ip/url
changelog_end
* Return to the simple http server start code
* Remove unused import
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* LF: Simplify archive reader.
- decouple Reader and Decoder
- introduce case class to handle hash, proto payload, and version
CHANGELOG_BEGIN
CHANGELOG_END
* Address Moritz' review
* cosmetic
These were originally hidden in the first PR because the metrics were
very shaky. Now they are actually useful and mentioned in release
notes so hiding this option makes no sense.
changelog_begin
changelog_end
Fixes#10161
changelog_begin
[JSON API] Fixed a bug that could sporadically make the streaming query
endpoint crash. This bug only affected 1.15.x snapshot releases.
changelog_end
* logging-entries: Split from contextualized-logging.
This allows us to introduce it to Daml-LF without bringing in the
Logback, Logstash, and gRPC dependencies.
CHANGELOG_BEGIN
CHANGELOG_END
* logging-entries: Fix dependencies for 2.12.
* logging-entries: Missed one more Scala 2.12 dependency.
* release: Publish logging-entries.
In #10016, 1% template ID and 1% party-set membership meant _the same_ 1%,
meaning that an index of both couldn't possibly yield interesting results. This
changes how LargeAcs builds the large ACS so that it's "1% of 1%", as you'd
expect.
CHANGELOG_BEGIN
CHANGELOG_END
changelog_begin
- [JSON-API] Timing metrics which measure how long the processing of a command submission request takes on the ledger are now available
changelog_end
changelog_begin
- [JSON-API] The database operations (regardless of in-memory or postgres) contain now metrics for fetching contracts by id or key (seperate metrics foreach)
- [JSON-API] The time for a repsonse payload construction of a request is now also tracked in a metric
changelog_end
* [JSON-API] Add more timing metrics
changelog_begin
- [JSON-API] Timing metrics are now available for party management, package management, command submission and query endpoints.
- [JSON-API] Also added a timing metric for parsing and decoding of incoming json payloads
changelog_end
* Add comments to new metrics
* Split metrics up more & remove obsolete metric
* Split up timers for query endpoints
* nvarchar2 keys are text-incompatible, but varchar2 keys are fine
* commit the ACS update before query
* add changelog
CHANGELOG_BEGIN
- [JSON API] The Oracle database schema has changed; if using
``--query-store-jdbc-config``, you must rebuild the database by adding
``,createSchema=true``. See #9895.
CHANGELOG_END
* expand the InitDdl set to include materialized views
* replace search index with a materialized view that expands the stakeholders
* allow materialized views to be created in Oracle testing
* join and query the contract_stakeholders table for party-set membership
- restoring a few elements removed by 3e6661128d (#9484)
This solves two warts in the code:
- the validate/createUnsafe double-parse, because scopt doesn't let you flatMap;
- the non-JdbcConfig sub-configs appeared to need to know the JDBC drivers when
they really don't, because of a quirk in the inherited implementation
That coherence of scopt.Read instances calls for all its dependencies to be
coherent leads us to treat supportedJdbcDriverNames as a nullary typeclass
instance. This is a nullary typeclass by the same justification as
SupportedJdbcDriver; see scaladoc there for more.
And we solve the latter problem by...adding a type parameter, how else.
CHANGELOG_BEGIN
CHANGELOG_END
* [JSON-API] Concurrent query etc. metrics
changelog_begin
- [JSON-API] The metrics which describe the amount of these concurrent events is now available:
- running http request
- command submissions
- package allocations
- party allocations
changelog_end
* Rename running metrics to throughput ones & add comments on the metrics
* Adjust names of other existing metrics too, to have for the json api a more consistent metrics naming
* Add information from the jwtpayload to the logging context
changelog_begin
- [JSON API] For applicable requests actAs, readAs, applicationId & ledgerId are included in the log context
changelog_end
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Revert changes to make the function generic
* Create JwtPayloadG trait from which both payload variants inherit
* Reduce code duplication in Endpoints.scala
* Apply review suggestion
* Update test name to reflect field name changes
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
