Some cached threadpools weren't given names, meaning at runtime there
are a bunch of pool-x-thread-y threads. This makes it hard to understand
which threads are being used for what.
The following pool names were introduced:
append-only indexer: input-mapping-pool, batching-pool
ProgramResource: program-resource-pool
kvutils PackageCommitter: package-preload-executor
CHANGELOG_BEGIN
CHANGELOG_END
* Switch from `@silent` to `@nowarn`.
This annotation is native to Scala 2.12.13+ and 2.13.2+. It replaces
most usages of `@silent`.
I had to get creative about a couple of use cases that didn't work.
Specifically:
1. Suppressing deprecation warnings works, but Scala 2.12 erroneously
complains that the `@nowarn` is unnecessary. I had to suppress
this warning too with `-Ywarn-unused:-nowarn`.
2. I can't seem to suppress the warning, "The outer reference in this
type test cannot be checked at run time." Instead, I have
refactored the code to remove the warning.
We still need to use the silencer plugin to suppress some warnings about
unused imports (because of compatibility between Scala 2.12 and 2.13),
but this means we no longer need the library, and therefore it is not a
transitive dependency that downstream consumers need to worry about.
CHANGELOG_BEGIN
CHANGELOG_END
* Add some comments around `@nowarn` support.
* language-support/scala: Fix a warning suppression.
* Revert to the default warnings.
Compatibility was complaining.
* resources: Wrapper constructors around the normal Resource types.
* resources: Factor out `ResourceFactories#apply`.
* resources: Remove the global factories.
They're no longer necessary, and could cause confusion. Users should use
their own factory object, not a generic one.
CHANGELOG_BEGIN
CHANGELOG_END
* resources: Extract out `Resource.nest` into its own class.
* resources: Avoid needing a context for "pure" resources.
CHANGELOG_BEGIN
CHANGELOG_END
* resources: Turns out the higher kinds import fails in Scala 2.13.
This PR updates scalafmt and enables trailingCommas =
multiple. Unfortunately, scalafmt broke the version field which means
we cannot fully preserve the rest of the config. I’ve made some
attempts to stay reasonably close to the original config but couldn’t
find an exact equivalent in a lot of cases. I don’t feel strongly
about any of the settings so happy to change them to something else.
As announced, this will be merged on Saturday to avoid too many conflicts.
changelog_begin
changelog_end
* Build //libs-scala/... on 2.13
One test is unfortunately disabled at the moment since I utterly
failed to figure out why I get a ClassNotFoundException on 2.13.
changelog_begin
changelog_end
* Copyright headers
changelog_begin
changelog_end
* I can’t bazel today
changelog_begin
changelog_end
* Apply suggestions from code review
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Update libs-scala/resources/src/main/2.13/com/daml/resources/UnitCanBuildFrom.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* No split on view
changelog_begin
changelog_end
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.
This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.
Apologies for the giant PR, I don’t see a way to keep it smaller.
changelog_begin
changelog_end
* resources: Release sequenced resources in parallel.
This isn't used much, but has been bothering me for a while. While we
acquire the resources in parallel, we used to release them sequentially.
This reimplements `sequence` so they're released all at once.
CHANGELOG_BEGIN
CHANGELOG_END
* resources: Drop an unnecessary `.map`.
* resources: Fix the Scaladoc for `sequence`.
* resources: Move builders into //ledger/ledger-resources.
Keep the actual constructors in a trait, but instantiate it when working
with ledger code.
This allows us to later introduce an extra "context" type parameter to
ResourceOwner.
* resources-akka: Move the builders in to //ledger/ledger-resources.
* resources: Introduce an abstract `Context` parameter for owners.
This replaces the concrete `ExecutionContext`. While it _can_ be an
execution context, it really doesn't matter as long as we can get at one
somehow.
This is being introduced so we can wrap the context in a container,
either for type tagging or to include extra information.
Because our current context _is_ `ExecutionContext`, and an implicit is
provided to extract it, we can end up with two ways to get the same
value. We use shadowing to prevent this. This problem should go away in
the near future when a new context type is added.
CHANGELOG_BEGIN
- [Integration Kit] The `ResourceOwner` type is now parameterized by a
`Context`, which is filled in by the corresponding `Context` class in
the _ledger-resources_ dependency. This allows us to pass extra
information through resource acquisition.
CHANGELOG_END
* ledger-resources: Move `ResourceOwner` here from `resources`.
* ledger-resources: Remove dependencies from outside //ledger.
* ledger-resource: Wrap the acquisition execution context in `Context`.
So we can add a logging context to it.
* resources: Pass the Context, not the ExecutionContext, to Resource.
* Avoid importing `HasExecutionContext`.
* ledger-resources: Publish to Maven Central.
* resources: Make the small changes suggested by @stefanobaghino-da.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* ledger-resources: Pull out a trait for test resource contexts.
Saves a few lines of code.
* Restore some imports that were accidentally wildcarded.
* resources: Replace an `implicit def` with a couple of imports.
* participant-integration-api: Simplify the JdbcLedgerDaoBackend tests.
Try and use the right execution context where possible.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* add silent_annotations option to da scala bazel functions
* use silent_annotations for several scala targets
* use silencer_plugin instead when the lib isn't used
* use silent_annotations for several more scala targets
* use silencer_lib for strange indirect requirement for running tests
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* silent_annotations support for scaladoc
* add -Ywarn-unused to all scalac options
* remove some unused arguments
* remove some unused definitions
* remove some unused variable names
* suppress some unused variable names
* changeExtension doesn't use baseName
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* work around no plugins in scenario interpreter perf tests
* remove many more unused things
* remove more unused things, restore some used things
* remove more unused things, restore a couple signature mistakes
* missed import
* unused argument
* remove more unused loggingContexts
* some unused code in triggers
* some unused code in sandbox and kvutils
* some unused code in repl-service and daml-script
* some unused code in bindings-rxjava tests
* some unused code in triggers runner
* more comments on silent usages
- suggested by @cocreature; thanks
* fix missing reference in TestCommands
* more unused in triggers
* more unused in sandbox
* more unused in daml-script
* more unused in ledger-client tests
* more unused in triggers
* more unused in kvutils
* more unused in daml-script
* more unused in sandbox
* remove unused in ledger-api-test-tool
* suppress final special case for codegen unused warnings
.../com/daml/sample/mymain/ContractIdNT.scala:24: warning: parameter value ev 0 in method ContractIdNT Value is never used
implicit def `ContractIdNT Value`[a_a1dk](implicit `ev 0`: ` lfdomainapi`.Value[a_a1dk]): ` lfdomainapi`.Value[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
^
.../com/daml/sample/mymain/ContractIdNT.scala:41: warning: parameter value eva_a1dk in method ContractIdNT LfEncodable is never used
implicit def `ContractIdNT LfEncodable`[a_a1dk](implicit eva_a1dk: ` lfdomainapi`.encoding.LfEncodable[a_a1dk]): ` lfdomainapi`.encoding.LfEncodable[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
^
* one more unused in daml-script
* special scaladoc rules may need silencer, too
* unused in compatibility/sandbox-migration
* more commas, a different way to `find`
- suggested by @remyhaemmerle-da; thanks
CHANGELOG_BEGIN
- [Sandbox] The ledger API server will now always use the most recent ledger configuration.
Until a ledger configuration is read from the ledger, command submissions will fail with the UNAVAILABLE error.
CHANGELOG_END
In kvutils, the first ledger configuration change needs
to have a generation one higher than the one returned
by getLedgerInitialConditions().
Remove initial config writing from sandbox as it's now written by the ledger API server
* resources: Rename `tearDownDuration` to `tearDownTimeout`.
* resources: Don't time out `ProgramResource` on startup.
Let it go for as long as it needs to.
Sometimes Sandbox takes a while to start, especially if you're
preloading large DARs. We should not try and predict how long it will
take.
CHANGELOG_BEGIN
CHANGELOG_END
Packages com.digitalasset.daml and com.daml have been unified under com.daml
Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.
CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
* Use com.daml as groupId for all artifacts
CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END
* Add 2 additional maven related checks to the release binary
1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId
* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
* sandbox: Re-use the root actor system in the StandaloneIndexerServer.
* kvutils/app: Don't use the ActorSystem execution context randomly.
Instead, make `Runner` a proper ResourceOwner, with an `acquire` method.
* sandbox: Re-use the root actor system in the StandaloneApiServer.
CHANGELOG_BEGIN
CHANGELOG_END
* resources: Remove the now-unused `ResourceOwner.sequence` functions.
They weren't well-thought-out anyway; they acquire resources
sequentially, rather than in parallel.
* sandbox-next: Make the Runner a real ResourceOwner.
* sandbox: Don't construct the ResetService twice.
* sandbox: Inline and simplify methods in StandaloneApiServer.
* resources: Define a `ResettableResource`, which can be `reset()`.
`reset()` releases the resource, performs an optional reset operation,
and then re-acquires it, binding it to the same variable.
* resources: Pass the resource value into the reset operation.
* sandbox: Fix warnings in `TestCommands`.
* sandbox-next: Add the ResetService.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Make sure the SandboxResetService resets asynchronously.
It was being too clever and negating its own asynchronous behavior.
* sandbox-next: Forbid no seeding.
This double negative is really hard to phrase well.
* sandbox-next: Implement ResetService for a persistent ledger.
* sandbox: Delete the comment heading StandaloneIndexerServer.
It's no longer meaningful.
* sandbox-next: No need to wrap the SandboxResetService in an owner.
* sandbox-next: Bump the ResetService test timeouts.
It looks like it's definitely slower than on Sandbox Classic™. Gonna
look into this as part of future work.
* Revert to previous asynchronous reset behavior
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* sandbox: If the ledger ID isn't provided, use the one in the database.
Previously, we would fail if working against an existing ledger, and not
explicitly providing the ledger ID. This was the case even if the ledger
ID was randomly generated initially.
CHANGELOG_BEGIN
- [Sandbox] If no ledger ID is provided when running against an existing
ledger, use the existing ID. Previously, Sandbox would fail to start.
CHANGELOG_END
* sandbox: The ReadOnlySqlLedger should always receive a ledger ID.
It's read-only; it can't create one.
* sandbox: Stop using `equal` in SqlLedgerSpec.
* sandbox: Test that the ledger ID is as specified in SqlLedgerSpec.
* sandbox: Let the top-level runner handle a ledger ID mismatch.
And clean up the log text.
* sandbox: Initialize the ledger properly when the ID is dynamic.
* sandbox: Use `Vector`, not `List`, for SqlLedger initialization.
Append with Vector, good. List, bad.
* ledger-api-common: Make `LedgerApiMode.Dynamic` an object.
And add Java-style static factory methods.
* kvutils/app | ledger-on-{memory,sql}: Make `ledgerId` optional.
It should be generated or retrieved from the persistence layer by the
ledger itself.
* kvutils: Make the ledger ID optional in the tests.
* ledger-on-sql: Store the ledger ID, and reject conflicting IDs.
* ledger-on-sql: Make more things final.
* ledger-on-sql: Document the `ledger_meta.table_key` column better.
* sandbox: Don't hardcode the number of packages in the test DAR.
It changes.
* ledger-on-sql: Merge the `head` resource owner with the `dispatcher`.
* sandbox: Use backticks to simplify pattern match in ReadOnlySqlLedger.
* ledger-on-sql: Extract methods in `owner`.
* resources: Allow for startup exceptions without no stack trace.
* sandbox: Remove dependency duplication in the BUILD file.
* ledger-on-(memory|sql): Extract `Main` into its own file tree.
This is so we can remove the dependency on kvutils/app from the library.
logback.xml goes with it.
* sandbox: Implement "Sandbox Next" on top of ledger-on-sql.
CHANGELOG_BEGIN
- [Sandbox] Preliminary work to rebuild Sandbox on top of the DAML
Ledger Integration Kit. Currently not exposed through the CLI.
CHANGELOG_END
* ledger-on-sql: Publish to Maven Central.
* sandbox: If an unknown database is passed to Sandbox Next, handle it.
* resources: Catch errors in owner construction or acquisition.
Sometimes we're not in a future yet.
* resources: Don't run ProgramResource error handling inside the EC.
Otherwise we end up in deadlock which is only broken by
`awaitTermination` timing out.
* ledger-on-sql: Add a prefix of "ledger_" to the Flyway migrations table.
And while we're at it, make this a constant in the code and use Flyway
placeholders to substitute it in.
We don't have any users right now, so changing the migrations should be
fine.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run the indexer on PostgreSQL when possible.
Specifically, when running the PostgreSQL conformance tests.
It will also work when running it as a binary, but you need to specify
both the `--server-jdbc-url` and `--jdbc-url` switches. This is to keep
kvutils/app simple.
* ledger-on-sql: Make `ExtraConfig` final.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* resources: Revert `ResourceOwner.failed` to `ResourceOwner[Nothing]`.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* kvutils|ledger: Don't use Scala's global execution context.
I have just discovered that Scala's global execution context uses daemon
threads, which means they don't stick around past the end of the `main`
function. This is bad when application startup fails and we end up with
no other threads; the application terminates before it has a chance to
print the error message, which is very unhelpful.
We resolve this by creating a cached thread pool in the
`ProgramResource` class, specifically used for acquisition and release
of resources.
To do this, we need to pass the execution context through to the
resource constructors, which makes usage a little more verbose, because
now we can't use lambda functions (which can't take implicit arguments
such as the `executionContext`).
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils-app|reference-v2: Sort `Runner` and `ReferenceServer` methods.
`App` is fairly problematic. It depends on `DelayedInit`, which is
deprecated because it has some surprising behavior.
Best to avoid it for anything more than a simple demo application.
Details: https://github.com/scala/bug/issues/4330
CHANGELOG_BEGIN
CHANGELOG_END
* resources: Throw upon acquiring the implicit ExecutionContext.
If we try and release an executor service which is itself being used to
power the releasing, we end up in a deadlock—the executor can't shut
down, and therefore `awaitTermination` never completes.
Fixing this is hard, but banning it is easy.
CHANGELOG_BEGIN
CHANGELOG_END
* resources: Explain a little more how the ExecutionContext check works.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-posix-filesystem: Run `scalafmt`.
* ledger-on-posix-filesystem: Just use file channel locks. Seriously.
* ledger-on-posix-filesystem: Rename `FileSystemLock#run` to `apply`.
* ledger-on-posix-filesystem: Conformance tests now run quickly.
So we can stop constraining the number of concurrent test runs.
* ledger-on-posix-filesystem: Lock the entire ledger with a file lock.
This ensures we can't have two separate ledger servers running against
the same directory.
This is mostly an experiment to better understand how to do this on an
SQL ledger for #4039.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-posix-filesystem: Run `scalafmt`.
* ledger-on-posix-filesystem: Use `Resource#use` in `FileSystemLock`.
* resources: Move `Resource#use` to `ResourceOwner#use`.
Makes more sense for the same object to govern both `acquire()` and
`release()`.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* resources: Make it easy to use a resource and then release it.
* ledger-on-sql: Move database transaction methods to `Database`.
* ledger-on-sql: Make sure the ledger picks up where it left off.
This means seeding the dispatcher with the correct head, rather than
starting at 1 every time.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Move `Index` and `StartIndex` to the package object.
They're used everywhere.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* kvutils: Use resources in KeyValueParticipantState, not AutoCloseable.
CHANGELOG_BEGIN
- [kvutils] The simplified API now uses ``com.digitalasset.resources``
to manage acquiring and releasing resources instead of ``Closeable``.
CHANGELOG_END
* ledger-on-posix-filesystem: Remove dead code due to a refactoring fail.
* libs-scala/resources: Move `Resource` and friends to their own package.
CHANGELOG_BEGIN
- [Ledger API Server] Publish the resource management code as a library
under ``com.digitalasset:resources``.
CHANGELOG_END
* libs-scala/resources: Add to the artifacts list.
* libs-scala/resources: Factor out the test helpers into test/lib.
* libs-scala/resources-akka: Make the Akka resources an optional dep.
