* Drop gRPC server side custom codegen
[CHANGELOG_BEGIN]
[CHANGELOG_END]
* Do not generate maven jar for ledger-api-akka and more cleanup
* Extract by-name source evaluation outside synchronized
* Addressed Martino's review comment
Co-authored-by: Tudor Voicu <tudor.voicu@digitalasset.com>
Previous version of etq_array_diff function proved to be vastly inefficient on a synthetic data set
resulting in "SQL Error: ORA-04036: PGA memory used by the instance exceeds PGA_AGGREGATE_LIMIT" error
after many minutes of processing. PGA_AGGREGATE_LIMIT was set to 2GB in that testing setup.
This new implementation of etq_array_diff completed the migration successfully in 03min 34s
which is comparable with the data migration for PostgreSQL. No evidence for excessive consumption
of memory was found ('select * from V$PGASTAT;').
changelog_begin:
Ledger API:
Previously non-error-code exceptions coming from streaming endpoints but thrown outside of Akka streams were not logged, and reported to clients with gRPC status UNKNOWN. Now they are logged on the server side and reported to clients as errors with LEDGER_API_INTERNAL_ERROR error code
changelog_end
Changes:
1. Swap the DB index order to facilitate filtration by application ids
2. Filter parties in JVM
3. Fetch results from the DB in a paged fashion
* Add new security categories
* Use test-security framework for OAuth2 middleware
* Regenerate `security-evidence.md`
* Explicitely exit the test-evidence generator
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Add //ledger-service/http-json:integration-tests-lib to evidence generator
* Skip maven artefacts on the classpath
The checker framework artefact (`checker-2.5.4.jar`) causes an `IllegalAccessError` when included in the runpath:
```
Exception in thread "main" java.lang.IllegalAccessError: class com.sun.tools.javac.code.Scope$ImportScope$ImportEntry cannot access its superclass com.sun.tools.javac.code.Scope$Entry (com.sun.tools.javac.code.Scope$ImportScope$ImportEntry is in unnamed module of loader java.net.URLClassLoader @31000e60; com.sun.tools.javac.code.Scope$Entry is in module jdk.compiler of loader app)
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at org.scalatest.tools.SuiteDiscoveryHelper$.isDiscoverableSuite(SuiteDiscoveryHelper.scala:204)
at org.scalatest.tools.SuiteDiscoveryHelper$.processClassName(SuiteDiscoveryHelper.scala:243)
at org.scalatest.tools.SuiteDiscoveryHelper$.$anonfun$processFileNames$1(SuiteDiscoveryHelper.scala:279)
at scala.collection.Iterator$$anon$9.next(Iterator.scala:577)
at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:474)
at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:576)
at scala.collection.immutable.List.prependedAll(List.scala:152)
at scala.collection.immutable.List$.from(List.scala:684)
at scala.collection.immutable.List$.from(List.scala:681)
at scala.collection.IterableFactory$Delegate.from(Factory.scala:288)
at scala.collection.immutable.Iterable$.from(Iterable.scala:35)
at scala.collection.immutable.Iterable$.from(Iterable.scala:32)
at scala.collection.IterableFactory$Delegate.from(Factory.scala:288)
at scala.collection.IterableOnceExtensionMethods$.toIterable$extension(IterableOnce.scala:178)
at org.scalatest.tools.SuiteDiscoveryHelper$.processFileNames(SuiteDiscoveryHelper.scala:285)
at org.scalatest.tools.SuiteDiscoveryHelper$.$anonfun$discoverSuiteNames$1(SuiteDiscoveryHelper.scala:132)
at scala.collection.immutable.List.map(List.scala:250)
```
* Regenerate `security-evidence.md`
* Convert remaining TEST_EVIDENCE stanzas of HTTP JSON
* Regenerate `security-evidence.md`
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Print warning when a test suite could not be loaded
* Fix typo
* Use test-security framework for HTTP JSON
* Read files in tests lazily
The test-evidence generator tool needs to instantiate scalatest test suites in order
to access the tagged tests and collect relavant test entries.
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Regenerate `security-evidence.md`
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Use test-security framework for trigger service
* Use structural type to call `in` and `ignore` for different classes
* Remove Authentication category from EvidenceSecurity tool
There are no TEST_EVIDENCE annotations anymore.
* Add required trigger-runner-lib
* Import `scala.language.reflectiveCalls` where it is needed
* Remove left-over comments
* Add `test_evidence_binary` scala binary rule
This rule is a customized `scala_binary` rule which also accepts a `tests` attr and
generates a runpath file which is later consumed by scalatest to detect the relevant
scalatest test suites.
* Process test suites, add transitive deps
* Support nested tests_suites in `test_evidence_binary`
* Remove debug print's
* Add missing dependencies to test-evidence:generator
* Abort if test suites cannot be loaded
* Cleanup
* Reinstate scalacopts in http-json
* Reword the test description to not drop information
* Fix typo
* Explicitly exit the JVM on exceptions
This is required since non-daemon threads also prevent JVM shutdown when an exception was thrown.
* Format test-evidence/BUILD.bazel
* Resolve file paths lazily
This avoids a `NullPointerException` on Windows where Runfiles.rlocation returns `null`.
* Document new Security properties
* Print target directory and file name
* Clarify test descriptions
* Replace duplicate Security properties
Co-authored-by: Stephen Compall <stephen.compall@daml.com>Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Read driver_metadata for transaction trees as well
changelog_begin
changelog_end
* Test contract metadata in StorageBackendTestsTransactionStreamsEvents
Before this change, we had two error categories that were meant for the same purpose: malicious or faulty and security alert. Now, they have been merged together. As malicious or faulty was misunderstood and sometimes used instead of "SystemInternalAssumptionViolation", we've dropped that term.
Related to canton 10981
In rare situations a canton participant returns a SUBMISSION_ALREADY_IN_FLIGHT
error instead of DUPLICATE_COMMAND due to a possible race condition between
publishing the TransactionAccepted read service update and updating internal
in-flight transactions. Particularly in Oracle runs this can result in a
subsequence "duplicate" command to still be considered in-flight.
Extended the tests to also accept the "in-flight" error code.
Changelog_begin
Changelog_end
updateTime can return an error and the code correctly tried to turn
that into invalidArgument and logged it. However, the Either returned
by that then got completely discarded and the overall request
succeeded which definitely seems like a bug.
I added a test and verified that it fails withotu this change.
The code seems fairly messy. There are a few reasons for that:
1. We use Future[Either[…]] and we treat failures in the Either
differently from future failures. I'm not quite convinced that
makes sense but for this PR I didn’t want to introduce unrelated
changed.
2. Future[Either[…]] without some form of EitherT is inherently messy
but afaik you don’t use cats here so I kept it like this.
