Prohibit contract IDs in contract keys and add key maintainers to exercises
CHANGELOG_BEGIN
- [DAML-LF] Prohibit contract IDs in contract keys completely. Previously, creating keys containing absolute (but not relative) contract IDs was allowed, but `lookupByKey` on such a key would crash.
CHANGELOG_END
Co-authored-by: Remy <remy.haemmerle@daml.com>
Co-authored-by: Stephen Compall <scompall@nocandysw.com>
This was accidentally set to 1.7 before for no reason. Note that this
simply meant that we did not use the stable module when compiling with
--target=1.6 since we filter out newer stable packages.
Nevertheless, I’ve also added a test that verifies that we do not
accidentally introduce dependencies on packages in newer LF versions.
changelog_begin
changelog_end
* sandbox: Tidy up the RecoveringIndexerSpec a little.
* sandbox: Move the JdbcLedgerDao owner constructor to its object.
* sandbox/damlonx: Make sure the RecoveringIndexer terminates.
Previously, if there was a failure and a termination at the same time,
we could recover after terminating, which means we'd end up in a weird
state. This makes sure we stop correctly.
Unfortunately, I couldn't find a good place to put these tests apart
from in reference-v2, as they need to depend on both _kvutils_ and
_sandbox_. Perhaps they belong in their own Bazel package.
This might address #3945; it's hard to say.
CHANGELOG_BEGIN
- [Indexer] Potentially fix a bug when recovering from failure.
CHANGELOG_END
* reference-v2: Address a race condition in the IndexerIT assertions.
* sandbox: Make sure the RecoveringIndexer can stop during a restart.
* reference-v2: Use Mockito to simulate errors in IndexerIT.
* sandbox: When the RecoveringIndexer is stopped, stop quickly.
* reference-v2: Make sure each test has its own metrics registry.
* Replace TestNamedLoggerFactory with in-memory appender
* sandbox: Test RecoveringIndexerSpec with a mock logger.
Using a real logger was causing massive stack traces in all other tests
that didn't have `RecoveringIndexerSpec$Appender` available, and adding
the appender programmatically didn't work for some reason.
The mock gives much nicer error messages on failure, too.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* libs-scala/resources: Move `Resource` and friends to their own package.
CHANGELOG_BEGIN
- [Ledger API Server] Publish the resource management code as a library
under ``com.digitalasset:resources``.
CHANGELOG_END
* libs-scala/resources: Add to the artifacts list.
* libs-scala/resources: Factor out the test helpers into test/lib.
* libs-scala/resources-akka: Make the Akka resources an optional dep.
* EC256 rather than EC512 algo
CHANGELOG_BEGIN
[Ledger API Authorization]Support EC256 algorithm for JWT rather than EC512
CHANGELOG_END
* support both 256 and 512 ECDSA algo
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* scalafmt
* correct comment, avoid unnecessary private def
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* correct BAZEL formatter
* Introduce ContextualizedLogger and LoggingContext
A ContextualizedLogger is a wrapper around an Slf4j logger. It uses
call-by-name parameters to only construct logging lines when necessary.
The underlying context can be used by invoking withoutContext.
More interestingly, every call to one of its logging methods requires
the presence of an implicit LoggingContext.
A LoggingContext is a way of definining a set of pairs that a
ContextualizedLogger can log without the need of it being specified for
every call.
A new context can be created with newLoggingContext and, given an
implicit LoggingContext, pairs can be added or overwritten using
withEnrichedLoggingContext.
Pairs in the context will be appended to each event logged by a
ContextualizedLogger in the following form:
some text logged explicitly (context: {key1=value1, key2=value2)
If the underlying Logger is configured to use the Logstash encoder the
keys and values in the LoggingContext will also be added to the logging
event encoded in JSON format.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestion in https://github.com/digital-asset/daml/pull/4046#pullrequestreview-343224692
Thanks to @SamirTalwar
Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>
* Address https://github.com/digital-asset/daml/pull/4046#discussion_r366874330
Co-authored-by: Samir Talwar <samir@noodlesandwich.com>
* ledger-on-sql: Implement a ledger using the simplified kvutils on JDBC.
This tests against H2; PostgreSQL coming soon.
No changelog because we're not shipping this yet.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/app: Extract ledger construction into a trait.
* ledger-on-sql: Run the conformance tests against H2.
* ledger-on-sql: Use H2-specific queries to perform actions atomically.
* ledger-on-sql: Extract queries to `Queries` and `H2Queries`.
* ledger-on-sql: SQLite can't have more than one connection at a time.
* ledger-on-sql: Run conformance tests against SQLite.
* ledger-on-sql: Fix error message if the JDBC URL is missing.
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* kvutils/app: Rename LedgerConstructor to LedgerFactory.
* ledger-on-sql: Don't release connections early.
* ledger-on-sql: Rename `withConnection` so it's more meaningful.
* ledger-on-sql: Use a sequential log entry ID.
* ledger-on-sql: Disable H2 conformance tests.
Until we can figure out why some fail.
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
Fixes#3984.
CHANGELOG_BEGIN
[Sandbox] Fix bug that can cause the transaction stream to not terminate.
See `issue #3984 <https://github.com/digital-asset/daml/issues/3984>`__.
CHANGELOG_END
The bug is that the ledger end gets updated without writing any ledger entries,
for example when uploading packages or in case of duplicate commands.
The termination of the stream (i.e. the comparison of offsets) only happened
on actual ledger entries. If there are none, then the stream just waits for the next
entry to be written.
This change moves the stream termination/completion to BEFORE potential entries
are loaded. The stream knows that after a certain offset has been reached, that it can
be terminated/completed.
* kvutils, etc: Remove duplicate copyright notices.
* ledger-on-posix-filesystem: Implement a simplified KV ledger on FS.
CHANGELOG_BEGIN
- A simple KVUtils-based ledger that runs against a POSIX filesystem.
Intended as a reference implementation for testing and documentation.
CHANGELOG_END
* ledger-on-posix-filesystem: Make it runnable.
* ledger-on-posix-filesystem: Chunk long state paths for macOS.
macOS only supports file names of up to 255 characters.
* ledger-on-posix-filesystem: Run conformance tests.
The lock is temporarily implemented as an in-memory mutex.
* ledger-on-posix-filesystem: Move the paths into the class.
* ledger-on-posix-filesystem: Lock using the file system.
We use the movement of a directory as an atomic operation.
* ledger-on-posix-filesystem: Use a sequential log entry ID.
* ledger-on-posix-filesystem: Check exception types using patterns.
* ledger-on-posix-filesystem: Test that state key collisions can't happen.
* ledger-on-posix-filesystem: Document the various directories.
* ledger-on-posix-filesystem: Use a *persistent* sequential log entry ID.
Because this is a persistent store. Duh.
* ledger-on-posix-filesystem: Don't worry about using Futures.
It's a global lock anyway.
* Factor out common values
* Validate LookupByKey nodes
Fixes#3543
CHANGELOG_BEGIN
- [Sandbox] Validate LookupByKey transaction nodes.
See `issue #3543 <https://github.com/digital-asset/daml/issues/3543>`_.
CHANGELOG_END
* Add failing tests
* LookupByKey validation should not respect privacy
* lookupByKey should use stakeholders, not divulgees
Fixes#2311
CHANGELOG_BEGIN
- [Ledger API] lookupByKey now requires the submitter to be a stakeholder on the referenced contract.
See `issue #2311 <https://github.com/digital-asset/daml/issues/2311>`_.
CHANGELOG_END
* Adapt tests to new behavior
Our users seem to stumble over this problem over and over again. Since there's
no technical reason to ban `submitMustFail` during sandbix initialization, we
can also allow for it.
CHANGELOG_BEGIN
- [Sandbox] Allow ``submitMustFail`` in scenarios used for sandbox initialization.
CHANGELOG_END
* Initial version of simplified transaction reader/writer API.
* Code tidying.
* Ran ./fmt.sh
* Added verification of pre-declared vs. actual state updates to example.
* kvutils: Run `scalafmt`.
* kvutils: Move ParticipantStateIntegrationSpecBase to src/test/lib.
* kvutils: Clean up warnings in ParticipantStateIntegrationSpecBase.
* ledger-on-memory: Move InMemoryLedgerReaderWriter to its own package.
* ledger-on-memory: Add a Main class with a simple configuration.
* reference-v2/ledger-on-memory: Wait for DARs to upload.
* ledger-on-memory: Run conformance tests.
* kvutils/app: Extract the in-memory ledger `Main` class into a Runner.
* reference-v2: Move InMemoryKVParticipantState out of kvutils.
It's not used elsewhere.
* kvutils/app: Fix some of the config help text.
* kvutils/app: Don't register the archive files twice.
CHANGELOG_BEGIN
- [kvutils/app] A simple way to run KVUtils-based participants.
- [ledger-in-memory] Split out the simple in-memory ledger, previously
part of kvutils, into its own JAR.
CHANGELOG_END
* Revert "kvutils/app: Don't register the archive files twice."
This reverts commit 2489ae6964.
Turns out we need this for preloading, which is a useful optimization.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
CHANGELOG_BEGIN
[Sandbox] Fix the Sandbox command line help text to reflect the recent
change that the default address is ``localhost`` (``127.0.0.1`` or
``::1``)
CHANGELOG_END
This is done to restrict the usage of NamedLoggerFactory and most usages of MetricsRegistry
to a single Bazel package, thus allowing a possible replacement to not have to become a
dependency of multiple packages.
CHANGELOG_BEGIN
CHANGELOG_END
* ECDA512 algorithm support
* ECDA512
* happy day test for ECDA512 algorithm
* failure test for wrong key for ECDA512 algorithm
* add ability to use EC cert file
* update docs
* scalafmt
* Correct documentation
CHANGELOG_BEGIN
[Ledger API Authorization] Support elliptic curve algorithm for JWT verification
CHANGELOG_END
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* correct docs warning
Currently there is a constraint on the postgres `ledger_entries` table
ensuring that submitter, application_id and command_id are set on
transactions being persisted. These fields are coming from the submitter
info that will be empty when the transaction accepted event is received
on a participant which is not hosting the submitter.
This change updates the check to ensure that all of these fields are
set OR none of them are.
The same change has been made to the H2 migrations with the addition of
correcting the `is not null` check (was previously using `!= null`).
For postgres, as the check is just loosening the existing constraint of all fields
requiring a value I think there is no risk that the new constraint will fail
to apply to existing data. For H2 as the original constraint was
incorrect there is a possibility that existing data may be incorrect -
however as the same component is responsible for writing to H2 and postgres
I will assume the data was correctly persisted regardless of the constraint.
Plus as H2 is typically only used for ephemeral tests, there is
likely no existing data to migrate.
CHANGELOG_BEGIN
- [Sandbox] Loosens database schema to allow persistence of transaction
ledger entries where no submitter info is present (typically when the
submitter is hosted by another participant node).
CHANGELOG_END
* Do not assume submitter info is present in the message
* Make kvutils conversion utilities public
Required for implementing extensions on top of kvutils.
* Make kvutils.Conversions private to com.daml.ledger.participant.state
* Add test case for allowing optional submitter info
* Add changelog markers
CHANGELOG_BEGIN
CHANGELOG_END
* Move most of the remaining serializable types to stable LF packages
The only serializable types left in DAML stdlib after this PR are the
following:
- DA.Upgrade:MetaEquiv
- DA.Random:Minstd
- DA.Next.Set:Set
- DA.Next.Map:Map
- DA.Generics:MetaSel0
- DA.Generics:MetaData0
- DA.Generics:DecidedStrictness
- DA.Generics:SourceStrictness
- DA.Generics:SourceUnpackedness
- DA.Generics:Associativity
- DA.Generics:Infix0
- DA.Generics:Fixity
- DA.Generics:K1
- DA.Generics:Par1
- DA.Generics:U1
- DA.Internal.Prelude:Optional
Ignoring the Generics stuff which isn’t very urgent imho and the
Upgrade stuff which is probably going to change significantly anyway,
this leaves us with the weird Random module, the wrappers around
TextMap which will go away anyway and DA.Internal.Prelude:Optional
which shouldn’t exist in the first place (I’ll address that in a
separate PR).
CHANGELOG_BEGIN
- [DAML Compiler] Move more types from daml-stdlib to standalone LF
packages. The module names for the types have also changed
slightly. This only matters over the Ledger API when you specify the
module name explicitly. In DAML you should continue to use the
existing module names.
- The types from ``DA.Semigroup` are now in a separate package under
``DA.Semigroup.Types``
- The types from ``DA.Monoid` are now in a separate package under
``DA.Monoid.Types``
- The types from ``DA.Time` are now in a separate package under
``DA.Time.Types``
- The types from ``DA.Validation` are now in a separate package
under ``DA.Validation.Types``
- The types from ``DA.Logic` are now in a separate package under
``DA.Logic.Types``
- The types from `DA.Date` are now in a separate package under
`DA.Date.Types`.
- The `Down` type from `DA.Internal.Prelude` is now in a separate
package under `DA.Internal.Down`.
CHANGELOG_end
* Fix serializability of RelTime
* fix daml-docs
* Fix tests
* Expose address config option in StandaloneApiServer
Before this PR, it was not possible to set the ledger-api-server
address and so it defaulted to localhost. Now, there is a configuration
variable address in the ApiServerConfig.
* Bind to the loopback address by default instead of the public interface.
CHANGELOG_BEGIN
- [daml-on-x-server] Made ledger api server to bind to localhost by
default instead to the public interface for security reasons.
CHANGELOG_END
* Start on daml test-scripts
* Run all `Script a` as test cases
* LedgerClient: Expose PackageManagementClient
To enable DAR uploads
* Upload the DAR to the ledger
* Start sandbox if no ledger specified
* Format daml test-script
* Fix deprecation warning on ActorMaterializer
* Add test-case //daml-script/tests:test_daml_script_test_runner
* Add daml test-script command
CHANGELOG_BEGIN
- [DAML Script - Experimental] Allow running DAML scripts as test-cases.
Executing ``daml test-script --dar mydar.dar`` will execute all
definitions matching the type ``Script a`` as test-cases.
See `#3687 <https://github.com/digital-asset/daml/issues/3687>`__.
CHANGELOG_END
* daml-test-script enable logging
* Remove outdated TODO comment
* daml script-test More elaborate test-caseo
Compare to expected output and add failing test-case
* daml test-script Don't abort on test-failure
Before the test runner would abort on the first failed test-case. This
occasionally introduce additional test-failures if the sandbox was
torn down half-way through execution.
* ./fmt.sh
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
* sandbox: Create a monadic `ResourceOwner` to manage resources.
* sandbox: Rewrite `ResourceOwner` to be async.
* sandbox: Make sure failed resources are closed immediately.
* sandbox: Better naming in `Open`.
* sandbox: Rename `Open` to `Resource`, and open/close to acquire/release.
* sandbox: Convert `() => AutoCloseable` into `ResourceOwner`.
* sandbox: Refactor the LedgerApiServer in terms of resources.
* sandbox: Explicitly convert `() => AutoCloseable` to `ResourceOwner`.
Explicit > Implicit, right?
* sandbox: Create helpers for converting things to ResourceOwners.
Because I tried to start using them and there was so much code being
written at once.
* sandbox: Simplify construction of JdbcLedgerDao.
* sandbox: Releasing resources should be idempotent.
In that we should only do it once.
* sandbox: Fix the ResetService by closing the API services _first_.
They need to be shut down before the gRPC server.
* sandbox: Don't try and shut down PostgreSQL twice in tests.
* sandbox: Actually run the assertions in ResourceOwnerSpec.
Facepalm.
* sandbox: Test `Resource.sequence` more rigorously.
* sandbox: Move the helpers around `Resource` into `Resource.apply`.
* sandbox: Convert LedgerApiServer resource owners to classes.
* sandbox: Make `ResourceOwner` a monad too, delegating to `Resource`.
* sandbox: Turn `LedgerApiServer` into a ResourceOwner.
* sandbox: Simplify the public signature of `Resource.apply`.
* sandbox: Use ResourceOwners to simplify DB resource management.
This is one hell of a change. Sorry.
* sandbox: Try not to nest `Await.result` calls.
Causes issues when running in a `DirectExecutionContext`.
* sandbox: Turn index subscriptions into resources.
* sandbox: Fix warnings in RecoveringIndexerSpec.
* sandbox: Always release before recovering the indexer.
* sandbox: Add `flatten` and `transformWith` to `Resource`.
* sandbox: If releasing twice in parallel, the second should wait.
* sandbox: If the indexer recovers, clean up the old subscription.
* sandbox: Convert StandaloneIndexerServer into a resource owner.
* sandbox: Convert StandaloneApiServer into a resource owner.
* reference-v2: Rewrite ReferenceServer in terms of resources.
CHANGELOG_BEGIN
- [Reference v2] On an exception, shut down everything and crash.
Previously, the server would stay in a half-running state.
CHANGELOG_END
* sandbox: Rewrite SandboxServer in terms of resources.
* sandbox: Write the port file in a Future.
* sandbox: JdbcIndexer no longer needs to manage the actorSystem.
* sandbox: Shut down the LedgerApiServer when closing the Sandbox.
* sandbox: Rename `Resource.pure` to `Resource.successful`.
* sandbox: Rename `Resource.sequence_` to `sequenceIgnoringValues`.
* sandbox: Delete `CloseableResource`.
It's only used in once place. Just inline it.
* sandbox: `LedgerDao` no longer needs to be closeable.
* sandbox: Delete implicit materializers where they're not used.
* http-json: Wait for the Sandbox to start in tests.
* sandbox: Convert `scheduleHeartbeats` into a ResourceOwner.
* reference-v2: Explain why we steal ownership of the actor system.
* sandbox: Document why we only release resources once.
* sandbox: Add clues to ResourceOwnerSpec.
* http-json: Fix HttpServiceTestFixture to pass auth service through.
* codegen-sample-app: In ScalaCodeGenIT, wait for the server to start.
* Moves CommandCompletionIT to Ledger API Test Tool
Closes#2841
Also refactors test observers into a single API and unifies the existing implementations of the DirectExecutionContext
* Allow explicit offsets to get checkpoints
* Raise checkpoint test timeout
* Consolidate checkpoint tests in a single test case
* Fix compilation issues
* Exclude command submission/completion tests for rev-v2
* Drop test not relevant for Ledger API implementations
* Fix compilation errors
Closes#2842Closes#2843
The separation between high-level and low-level is not particularly meaningful, as the low-level variant issues commands with a component used by the high-level variant.
The separation aims at performing integration tests at two different levels but this can be addressed by a single suite of integration tests and relying on unit testing for individual components.
The first two tests:
- 'having many transactions all of them has a unique event id' reads transactions from an empty ledger and doesn't really test anything, so this has been dropped because the uniqueness of event identifiers is already more than covered by many tests in the 'TransactionService' suite.
- 'handle bad Decimals correctly' ensures the ledger api rejects unrepresentable decimals and it have been re-implemented as 'Reject unrepresentable numeric values' in the 'CommandService' suite.
The other tests are about command-and-exercise:
- 'process valid commands with workflow ids successfully' -> 'Implement create-and-exercise correctly'
- 'fail for invalid create arguments' -> 'Fail create-and-exercise on bad create arguments'
- 'fail for invalid choice arguments' -> 'Fail create-and-exercise on bad choice arguments'
- 'fail for an invalid choice' -> 'Fail create-and-exercise on invalid choice'
* Upgrade to Akka 2.6.1, akka-http 10.1.11 and Scala 2.12.10
Akka 2.6.1 Upgrade Changes
- Materializer in place of ActorMaterializer
- Source.future instead of Source.fromFuture
- The Scheduler.schedule method has been deprecated in favor of selecting scheduleWithFixedDelay or scheduleAtFixedRate
- onDownstreamFinish(cause: Throwable)
- ActorAttributes.supervisionStrategy(...) in place of ActorMaterializerSettings.withSupervisionStrategy
See https://doc.akka.io/docs/akka/current/project/migration-guide-2.5.x-2.6.x.html
* Akka 2.6.1 Upgrade Changes
- onDownstreamFinish(cause: Throwable)
See https://doc.akka.io/docs/akka/current/project/migration-guide-2.5.x-2.6.x.html
* code review: remove unnecessary supervision strategy
* sandbox: Fix warnings in RecoveringIndexerSpec.
* sandbox: Test RecoveringIndexer logs, and stop dumping them to STDERR.
Printing logs during test execution is really annoying. This stubs the
logger factory with a TestNamedLoggerFactory which captures them in a
list instead, for testing purposes.
* sandbox: Remove unnecessary comments from {,Simple}NamedLoggerFactory.
* ledger-api-common: Remove unused parts of the NamedLoggerFactory.
Using the default configuration as a base for this test meant binding on 6865 instead of using dynamic port binding, which could cause flaky errors if some other test is occupying the port
* Move Any wrappers and Archive to stable packages
There are no actual API or functionality changes in this PR but the
logic for locating the stable packages has slightly changed since the
Any wrappers package only makes sense for LF 1.7. To address this, we
simply filter out stable packages for newer LF versions since it
doesn’t make sense to depend on those anyway.
CHANGELOG_BEGIN
- [DAML Compiler] Move ``Archive`` type to a separate DALF.
CHANGELOG_END
* More comments
* Fix java codegen tests
* fix more tests
* Force cache reset on Windows
* Revert "Force cache reset on Windows"
This reverts commit 9f2b7d70b2.
This test suite was introduced in #3870 and seems to be extremely
flaky both locally (I get about 16/20 failures) and on CI.
The error is
- should return completions of commands that are submitted after subscription *** FAILED *** (4 seconds, 945 milliseconds)
Set() did not contain all elements of List("5104", "5100", "5105", "5109", "5103", "5106", "5107", "5102", "5108", "5101") (CommandClientIT.scala:330)
After bumping the timeout, I don’t get any failures locally.
Happy to choose another timeout, I have no clue what a sensible value
is here.
* Port FailingCommandsIT to the Ledger API Test Tool
Closes#2850
The FailingCommandsIT suite contains tests for mismatching ledger
identifier (which is already abundantly covered by the Ledger API Test
Tool) and for bad labels in contract creation, both via submit and
submitAndWait.
Some of the tests previously part of the CommandService Ledger API Test
Tool suite have been moved to a new home in CommandSubmissionCompletion
to reflect the fact that those use the submission/completion workflow
instead of leveraging the submit-and-wait alternatives.
There are also suites that for some reason cover subsequent invocations
of the same services but I'm not sure of the meaning of those and I
would drop them, unless we have a good explanation for those.
* Optimize imports
* Move Scala Ledger API client integration tests
Closes#2845
The Scala Ledger API client tests belong to the
//ledger/ledger-api-client package.
* Reduce noise coming from tests
* sandbox: Move SandboxServer's helper classes to the companion object.
And make them `private`, `final`, and non-`case`.
* sandbox: Drop SandboxServer.apply and just call the constructor.
* sandbox: Move the SandboxServer#SandboxState class into the object.
Leave the `resetAndRestartServer` method behind, though.
