Using `java.net.http.WebSocket` seemed like the simplest interface to be able to verify the closed status from the outside of the system. We looked into akka's `WSProbe::expectCompletion` but wiring that up would have required building our own `WebSocketService` and all its dependencies.
* Pruning needs to be retried, with artificial activity added, until the safe-offset has advanced far enough for it to succeed.
* The "max deduplication duration" needs to be dropped, otherwise pruning cannot be done for at least the default of 168h.
* The "reconciliation interval" needs to be lowered. This is a dynamic config, so we set it via a bootstrap script. The change is not effected immediately, but asynchronously some time after startup. Lowering this enables the safe-offset to catch up faster.
* We need to ensure the relevant tests are only enabled when testing against an Enterprise edition of Canton.
Contributes to https://digitalasset.atlassian.net/browse/LT-17
Adds a `disclosedContracts` optional list field to the `meta` argument
for `create`, `exercise` and `create-and-exercise` endpoints.
The argument is ignored in all cases but `exercise` (#16611 builds on
this PR to add `create-and-exercise` support). A single disclosed
contract looks more or less like follows:
{
"contractId": "abcd",
"templateId": "Mod:Tmpl",
$argumentsJsonField,
"metadata": {
"createdAt": "2023-03-21T18:00:33.246813Z",
"contractKeyHash": "77656c6c2068656c6c6f",
"driverMetadata": "dGhlcmUgcmVhZGVy"
}
}
where `argumentsJsonField` may be either one of these, setting aside the
extra quotes added for these tests:
"payload": {"owner": "Alice"}
"payloadBlob": {
"typeUrl": "type.googleapis.com/com.daml.ledger.api.v1.Record",
"value": "Eg4KBW93bmVyEgVaA0JvYg=="
}
(Note that `typeUrl` is variable, not constant; use the actual blob's
`typeUrl` contents, **do not assume it is exactly the above example**.)
This PR uses base-64 for `payloadBlob.value` and
`metadata.driverMetadata`, and base-16 for `metadata.contractKeyHash`.
* confirm that monadifying the package fetch still suppresses the error
* thread ExecutionContext from request
- makes the error less likely
- but still fairly easy to repro with 3 tabs
* experiment with setting executor
* explain that the cache isn't a cache
* random order, maybe
- #3090 mentions keeping the order as a goal; I don't see why we should,
though
* random order with groups of 8
* embed the decoding
- this slows down the processing of a group, yielding somewhat less
granular contention
- and also makes hits cost much less, at the cost of making granular
contention more expensive
* reduce diff size before resolution
- this won't improve contention, but does nearly eliminate the cost of
resolution for already-resolved packages, making hits nearly free
(amortized)
* randomize groups instead
- while groups themselves can overlap with this arrangement, each
costing ParallelLoadFactor granular contention, on average it seems to
perform a little better due to groups never overlapping
* refactor StatusEnvelope to utils
* constant 250ms retry
* detect contention earlier and skip decode
* factor traverseFM
* add -Xlint options requiring no changes
* add -Xlint:recurse-with-default
- very minor code changes
* factor http-json hj_scalacopts duplication
* use lf_scalacopts_stricter in libs-scala where NonUnitStatements was
* use hj_scalacopts in api-type-signature
* add nonlocal-return and nullary-unit to hj_scalacopts
* commented-out excluded options
* add unit-special globally
* check implicit-recursion for clients code
* Add new security categories
* Use test-security framework for OAuth2 middleware
* Regenerate `security-evidence.md`
* Explicitely exit the test-evidence generator
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Add //ledger-service/http-json:integration-tests-lib to evidence generator
* Skip maven artefacts on the classpath
The checker framework artefact (`checker-2.5.4.jar`) causes an `IllegalAccessError` when included in the runpath:
```
Exception in thread "main" java.lang.IllegalAccessError: class com.sun.tools.javac.code.Scope$ImportScope$ImportEntry cannot access its superclass com.sun.tools.javac.code.Scope$Entry (com.sun.tools.javac.code.Scope$ImportScope$ImportEntry is in unnamed module of loader java.net.URLClassLoader @31000e60; com.sun.tools.javac.code.Scope$Entry is in module jdk.compiler of loader app)
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at org.scalatest.tools.SuiteDiscoveryHelper$.isDiscoverableSuite(SuiteDiscoveryHelper.scala:204)
at org.scalatest.tools.SuiteDiscoveryHelper$.processClassName(SuiteDiscoveryHelper.scala:243)
at org.scalatest.tools.SuiteDiscoveryHelper$.$anonfun$processFileNames$1(SuiteDiscoveryHelper.scala:279)
at scala.collection.Iterator$$anon$9.next(Iterator.scala:577)
at scala.collection.Iterator$$anon$6.hasNext(Iterator.scala:474)
at scala.collection.Iterator$$anon$9.hasNext(Iterator.scala:576)
at scala.collection.immutable.List.prependedAll(List.scala:152)
at scala.collection.immutable.List$.from(List.scala:684)
at scala.collection.immutable.List$.from(List.scala:681)
at scala.collection.IterableFactory$Delegate.from(Factory.scala:288)
at scala.collection.immutable.Iterable$.from(Iterable.scala:35)
at scala.collection.immutable.Iterable$.from(Iterable.scala:32)
at scala.collection.IterableFactory$Delegate.from(Factory.scala:288)
at scala.collection.IterableOnceExtensionMethods$.toIterable$extension(IterableOnce.scala:178)
at org.scalatest.tools.SuiteDiscoveryHelper$.processFileNames(SuiteDiscoveryHelper.scala:285)
at org.scalatest.tools.SuiteDiscoveryHelper$.$anonfun$discoverSuiteNames$1(SuiteDiscoveryHelper.scala:132)
at scala.collection.immutable.List.map(List.scala:250)
```
* Regenerate `security-evidence.md`
* Convert remaining TEST_EVIDENCE stanzas of HTTP JSON
* Regenerate `security-evidence.md`
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Print warning when a test suite could not be loaded
* Fix typo
* Use test-security framework for HTTP JSON
* Read files in tests lazily
The test-evidence generator tool needs to instantiate scalatest test suites in order
to access the tagged tests and collect relavant test entries.
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Regenerate `security-evidence.md`
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Use test-security framework for HTTP JSON
* Regenerate `security-evidence.md`
* Use test-security framework for trigger service
* Use structural type to call `in` and `ignore` for different classes
* Remove Authentication category from EvidenceSecurity tool
There are no TEST_EVIDENCE annotations anymore.
* Add required trigger-runner-lib
* Import `scala.language.reflectiveCalls` where it is needed
* Remove left-over comments
* Add `test_evidence_binary` scala binary rule
This rule is a customized `scala_binary` rule which also accepts a `tests` attr and
generates a runpath file which is later consumed by scalatest to detect the relevant
scalatest test suites.
* Process test suites, add transitive deps
* Support nested tests_suites in `test_evidence_binary`
* Remove debug print's
* Add missing dependencies to test-evidence:generator
* Abort if test suites cannot be loaded
* Cleanup
* Reinstate scalacopts in http-json
* Reword the test description to not drop information
* Fix typo
* Explicitly exit the JVM on exceptions
This is required since non-daemon threads also prevent JVM shutdown when an exception was thrown.
* Format test-evidence/BUILD.bazel
* Resolve file paths lazily
This avoids a `NullPointerException` on Windows where Runfiles.rlocation returns `null`.
* Document new Security properties
* Print target directory and file name
* Clarify test descriptions
* Replace duplicate Security properties
Co-authored-by: Stephen Compall <stephen.compall@daml.com>Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* use domain.Party instead of String in jwtForParties, headersWithPartyAuth
CHANGELOG_BEGIN
CHANGELOG_END
* use domain.Party instead of String in headersWithPartyAuth
* fix sharedAccountCreateCommand and some corners
* domain.ContractTypeId.Template
* ResolveKeyType to use .Template
* CreateAndExerciseCommand to use .Template
* keyTypeMap
* bitraverse
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* use Consume.syntax
* added exercise in interface "sub query endpoint" test case. checked template id of archive. it is asserting to be template id but not interface id which is wrong. I will change it back to interface id after all CI passed
* test that the archive events have the interface ID; add the template-ID-to-interface-ID mapping to archive events in websocket result stream
* Add change log
CHANGELOG_BEGIM
CHANGELOG_END
* Add change log
CHANGELOG_BEGIN
CHANGELOG_END
* use JSON converter instead of hand-writing JSON
* factor assertions for created records
* combine value assertions
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* replace several TemplateId with ContractTypeId
* replace several TemplateId with ContractTypeId.Template
* deprecate, even though this misses so many usages
* link to #15098 for followups
CHANGELOG_BEGIN
CHANGELOG_END
CHANGELOG_BEGIN
- [JSON API] For exercise endpoints, when multiple choices with the same
name are defined, the one directly defined on the ``templateId``,
which may be a template or interface ID, will be selected; if a
template ID is specified that only inherits the choices with the same
name, the new ``choiceInterfaceId`` option must be specified to select
the source of the choice.
CHANGELOG_END
* move ImplicitPreference to scala-utils
- test it
- use it instead of tags for json-api testing
- fix a somewhat nonsensical type therein
* several new tests for the NonEmpty Foldable instances
CHANGELOG_BEGIN
CHANGELOG_END
Refactors a bunch of AbstractHttpServiceIntegrationTestFuns and
elements of #12922's JWT function abstractions into a new
HttpFailureUserFixture, using that split to split the
FailureTests as well.
Fixes#13112.
* restore failure test porting from #12922
This reverts b4942defa4.
* move token-relevant utilities to http-json-testing lib
* replace custom actorsystem setup with AkkaBeforeAndAfter
- suggested by @ray-roestenburg-da; thanks
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Victor Peter Rouven Müller <mueller.vpr@gmail.com>
* Normalize token parsing & remove custom parsing code
changelog_begin
- [HTTP-JSON] custom claim tokens without ledger id are now correctly recognized as such and not as user tokens
changelog_end
* Add test to check that tokens without ledger id work correctly
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/endpoints/UserManagement.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/EndpointsCompanion.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Remove the overload of jwtForParties & change how the wihtoutNamespace JWT is constructed
* Remove unnecessary code line
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* WIP
* First version that compiles (with debug print statements)
* Revert changes on the failure tests
* Make all tests pass with some major cleanup of the new code
* Remove debug prints & outcommented code
* Make decoding / encoding test also compatible with user token tests
changelog_begin
changelog_end
* Minimize diff
* Fix oracle tests
* leave note for followup
* Update ledger-service/http-json/src/itlib/scala/http/AbstractHttpServiceIntegrationTest.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Update kvutils Config and SandboxConfig
* Remove `eager-package-loading` and `scenario` from SandboxConfig
* Add `profile-dir` and `stackTraces` to kvutils Config
* Configurable authService in kvutils
changelog_begin
changelog_end
* Addressed Samir's review comments
* Add back Maximum supported deduplication duration enforcement of 1 hour
* SandboxNextFixture replaced by Sandbox-on-X based SandboxFixture
changelog_begin
changelog_end
* Some fixed tests
* No direct dependencies on //ledger/sandbox:sandbox and //ledger/sandbox:sandbox-scala-tests-lib
* Fix after rebase
* Rename SandboxFixture and add a missing dep
* Generate valid party names if hint is empty
* Smaller maxInboundMessageSize
* Added test for empty display name
* SandboxServer is a ResourceOwner
* Uses execution context passed as an input for resource management
* Fixes flaky FlywayMigrations issue with null Thread.currentThread.currentClassLoader
* SandboxServer simplification returns Port instead of ApiServer
* Dedicated PMAllocateWithoutDisplayName for non-Canton ledgers
* Created since Canton does not return empty display names
Since Scala 2.13.2, Scala introduced built-in support to
manage warnings in a more granular fashion, thus making
the silencer plugin we are currently using no longer
strictly useful. Removing compiler plugins also removes
friction from migrating to Scala 3 in the future. As a
cherry on top, the built-in warning configuration also
allows to check whether a `@nowarn` actually does
anything, allowing us to proactively remove unused
warnings should the need arise.
[Here][1] is s a blog post by the Scala team about it.
Warnings have been either solved or preserved if useful,
trying to minimize the scope (keeping it at the single
expression scope if possible). In particular, all
remaining usages of the Scala Collection API compatibility
module have been removed.
Using the silencer plugin also apparently hid a few
remaining usages of compatibility libraries that were used
as part of the transition from Scala 2.12 to Scala 2.13
that are no longer needed. Removing those warnings
highlighted those.
changelog_begin
changelog_end
[1]: https://www.scala-lang.org/2021/01/12/configuring-and-suppressing-warnings.html
* Drop support for no seeding in sandbox-classic
Sandbox classic is going away in SDK 2.0 and so is support for v0
contract ids which is the only thing no seeding was used for.
This has been cleared by product.
I’ll drop v0 contract ids completely in #12464 as a follow-up. Just
wanted to factor this out for ease of review.
changelog_begin
changelog_end
* .
* .
* Split channel configuration from LedgerClientConfiguration
Fixes#12391
The channel configuration now has to be provided separately from the
configuration specific to the ledger client. In this way we avoid
situations where the builder is provided with some configuration
that gets overridden.
changelog_begin
[Scala bindings] The channel configuration has been split from the
LedgerClientConfiguration class. Provide the gRPC channel specific
configuration separately or use a builder. The channel configuration
no longer overrides the builder.
changelog_end
* Fix compilation issues in //ledger-service/...
* Change heartBeatPer to more intuitive naming of heartbeatPeriod
CHANGELOG_BEGIN
CHANGELOG_END
* Initial changes to add HOCON config for json_api
CHANGELOG_BEGIN
CHANGELOG_END
* avoid IllegalArgumentException noise
* use named arguments in big config conversion
* Changes include
- tests for a full http-json-api config file
- logging config and non-repudiation config is still specified via cli args.
- config readers for MetricsReporter
* Add defaults to WebsocketConfig case class to allow partially specifying fields on typeconf file
* changes to the JwtVerifierBase config reader and equivalent test
* message already describes the value
* replace manual succeed/fails with scalatest combinators
* use qualified imports for WebsocketConfig defaults
* add back autodeleted empty lines
* collapse two lists of token verifiers into one
* add new line to config files
* rename dbStartupMode to startMode to keep consistent with cli option and for easy documentation
* Changes to daml docs to specify ways to run JSON-API by supplying a HOCON config file.
CHANGELOG_BEGIN
JSON-API can now be started supplying a HOCON application config file using the `--config` option.
All CLI flags except `logging` and `non-repudiation` one's are now deprecated and will be cleaned up in some future releases.
CHANGELOG_END
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Changelog entry and commit msg differ here because the bug described in
the changelog was already fixed by adding the user management support
because it caused for the affected endpoints that it will be interpret as
user token while only fetching the ledger id (without actually checking
that it is a user token).
changelog_begin
- [HTTP-JSON] Fixed a bug that caused jwt's without the daml namespace to be rejected for some endpoints (https://github.com/digital-asset/daml/issues/12215)
changelog_end
New year, new copyright, new expected unknown issues with various files
that won't be covered by the script and/or will be but shouldn't change.
I'll do the details on Jan 1, but would appreciate this being
preapproved so I can actually get it merged by then.
CHANGELOG_BEGIN
CHANGELOG_END
* WIP
* Remove the dummy implementation and replace it with an actual working implementation
* Make it compile!
* Add working tests for the user management support in the json api
CHANGELOG_BEGIN
- [JSON-API] Added basic support for the new user management feature of the ledger such that user tokens are now accepted instead of the legacy tokens
CHANGELOG_END
* Simplify the create iou test case and adjust the test case name to be correct
* Add additional test that covers that the overwrite of actAs&readAs still works via the meta object
* Make it work with unauthenticated ledgers too
* Fix compile error & wrong behaviour & add test coverage for non auth ledgers
* Clean up the diff
* Address 66312e9940 (r770782884)
* Address 66312e9940 (r770750653)
* Addressing 66312e9940 (r770751958)
* Address 66312e9940 (r770736671)
* Address 66312e9940 (r770734395) and 66312e9940 (r770783237)
Co-authored-by: Stefano Baghino <stefano.baghino@digitalasset.com>
* Update to Java 11
changelog_begin
changelog_end
* Fix RoundingMode deprecation warnings
* Fix dep-ann warning
* Integer constructor
* JavaX annotation dependency
* javax.xml.bind was removed in Java 11
Using Guava as a replacement, since it is already a project dependency.
* JDK 11 no longer has a separate JRE tree
* Remove unused jdk_nix import
* remove now redundant jdk11_nix
* Java 8 --> 9 increased Instant.now() precision
See https://bugs.openjdk.java.net/browse/JDK-8068730
The precision of `Instant.now()` increased between Java 8 and Java 9.
On Linux and MacOS this doesn't seem to be a problem, as the precision
still seems to be at micro seconds. However, on Windows this now causes
errors of the following form:
```
java.lang.IllegalArgumentException: Conversion of Instant
2021-11-05T13:58:56.726875100Z to microsecond granularity would result
in loss of precision.
```
Suggesting that it now offers sub-microsecond precision.
`TimestampConversion.instantToMicros` had a check to fail if the
conversion lead to a loss of precision. In the specific failing test
case this is not a concern, so this adds a `roundInstantToMicros`
variant that avoids this kind of error.
* TMP round timestamps
* Revert "TMP round timestamps"
This reverts commit af8e261278.
* Skip versions before 1.6.0 in migration tests
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
