The old queries for building the transactions streams based on querying event tables directly are replaced with a two step process where we first fetch event sequential ids that match the input constraints from multiple id filter tables, and then fetch events by those fetched ids.
This approach is analogous to we have been already streaming ACS.
changelog_begin
Ledger API Server: This change introduces new db metrics and retires some of the old ones. The exact breakdown should be provided in a follow-up PR.
changelog_end
* Drop gRPC server side custom codegen
[CHANGELOG_BEGIN]
[CHANGELOG_END]
* Do not generate maven jar for ledger-api-akka and more cleanup
* Extract by-name source evaluation outside synchronized
* Addressed Martino's review comment
Co-authored-by: Tudor Voicu <tudor.voicu@digitalasset.com>
Previous version of etq_array_diff function proved to be vastly inefficient on a synthetic data set
resulting in "SQL Error: ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT" error
after many minutes of processing. PGA_AGGREGATE_LIMIT was set to 2GB in that testing setup.
This new implementation of etq_array_diff completed the migration successfully in 03min 34s
which is comparable with the data migration for PostgreSQL. No evidence for excessive consumption
of memory was found ('select * from V$PGASTAT;').
changelog_begin:
Ledger API:
Previously non-error-code exceptions coming from streaming endpoints but thrown outside of Akka streams were not logged, and reported to clients with gRPC status UNKNOWN. Now they are logged on the server side and reported to clients as errors with LEDGER_API_INTERNAL_ERROR error code
changelog_end
Changes:
1. Swap the DB index order to facilitate filtration by application ids
2. Filter parties in JVM
3. Fetch results from the DB in a paged fashion
* Add new security categories
* Use test-security framework for OAuth2 middleware
* Regenerate `security-evidence.md`
* Explicitely exit the test-evidence generator
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Add //ledger-service/http-json:integration-tests-lib to evidence generator
* Skip maven artefacts on the classpath
The checker framework artefact (`checker-2.5.4.jar`) causes an `IllegalAccessError` when included in the runpath:
```
Exception in thread "main" java.lang.IllegalAccessError: class com.sun.tools.javac.code.Scope$ImportScope$ImportEntry cannot access its superclass com.sun.tools.javac.code.Scope$Entry (com.sun.tools.javac.code.Scope$ImportScope$ImportEntry is in unnamed module of loader java.net.URLClassLoader @31000e60; com.sun.tools.javac.code.Scope$Entry is in module jdk.compiler of loader app)
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at org.scalatest.tools.SuiteDiscoveryHelper$.isDiscoverableSuite(SuiteDiscoveryHelper.scala:204)
at org.scalatest.tools.SuiteDiscoveryHelper$.processClassName(SuiteDiscoveryHelper.scala:243)
at org.scalatest.tools.SuiteDiscoveryHelper$.$anonfun$processFileNames$1(SuiteDiscoveryHelper.scala:279)
at scala.collection.Iterator$$anon$9.next(Iterator.scala:577)
at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:474)
at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:576)
at scala.collection.immutable.List.prependedAll(List.scala:152)
at scala.collection.immutable.List$.from(List.scala:684)
at scala.collection.immutable.List$.from(List.scala:681)
at scala.collection.IterableFactory$Delegate.from(Factory.scala:288)
at scala.collection.immutable.Iterable$.from(Iterable.scala:35)
at scala.collection.immutable.Iterable$.from(Iterable.scala:32)
at scala.collection.IterableFactory$Delegate.from(Factory.scala:288)
at scala.collection.IterableOnceExtensionMethods$.toIterable$extension(IterableOnce.scala:178)
at org.scalatest.tools.SuiteDiscoveryHelper$.processFileNames(SuiteDiscoveryHelper.scala:285)
at org.scalatest.tools.SuiteDiscoveryHelper$.$anonfun$discoverSuiteNames$1(SuiteDiscoveryHelper.scala:132)
at scala.collection.immutable.List.map(List.scala:250)
```
* Regenerate `security-evidence.md`
* Convert remaining TEST_EVIDENCE stanzas of HTTP JSON
* Regenerate `security-evidence.md`
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Print warning when a test suite could not be loaded
* Fix typo
* Use test-security framework for HTTP JSON
* Read files in tests lazily
The test-evidence generator tool needs to instantiate scalatest test suites in order
to access the tagged tests and collect relavant test entries.
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Regenerate `security-evidence.md`
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Use test-security framework for trigger service
* Use structural type to call `in` and `ignore` for different classes
* Remove Authentication category from EvidenceSecurity tool
There are no TEST_EVIDENCE annotations anymore.
* Add required trigger-runner-lib
* Import `scala.language.reflectiveCalls` where it is needed
* Remove left-over comments
* Add `test_evidence_binary` scala binary rule
This rule is a customized `scala_binary` rule which also accepts a `tests` attr and
generates a runpath file which is later consumed by scalatest to detect the relevant
scalatest test suites.
* Process test suites, add transitive deps
* Support nested tests_suites in `test_evidence_binary`
* Remove debug print's
* Add missing dependencies to test-evidence:generator
* Abort if test suites cannot be loaded
* Cleanup
* Reinstate scalacopts in http-json
* Reword the test description to not drop information
* Fix typo
* Explicitly exit the JVM on exceptions
This is required since non-daemon threads also prevent JVM shutdown when an exception was thrown.
* Format test-evidence/BUILD.bazel
* Resolve file paths lazily
This avoids a `NullPointerException` on Windows where Runfiles.rlocation returns `null`.
* Document new Security properties
* Print target directory and file name
* Clarify test descriptions
* Replace duplicate Security properties
Co-authored-by: Stephen Compall <stephen.compall@daml.com>Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Read driver_metadata for transaction trees as well
changelog_begin
changelog_end
* Test contract metadata in StorageBackendTestsTransactionStreamsEvents
Before this change, we had two error categories that were meant for the same purpose: malicious or faulty and security alert. Now, they have been merged together. As malicious or faulty was misunderstood and sometimes used instead of "SystemInternalAssumptionViolation", we've dropped that term.
Related to canton 10981
