* Add failing test case for transient contract keys
* Add kvutils tests for transient contracts and keys
* Fix handling of transient contracts and keys in kvutils
This removes the need for clients to handle the special case where the
stream might be empty.
Now the clients can always assume that they receive at least one
response element in the stream.
* Fix contract key uniqueness check in kvutils
Archival of a contract with a key and recreation within the
same transaction is now allowed in kvutils.
* Add assertions to check that new contract has been created with same key
Since we still have issues with parallel writes when using H2,
we should properly only use a single DB connection and executor thread
for that.
Because we didn't do that before for Sandbox, tests like PackageManagement
appeared to be flaky due to racy inserts in H2.
* Preload packages to engine during upload
* Improve logging in KeyValueCommitting and add timing information
* Fix scenario service tests now that logging is done in interpreter
This is believed to prevent race conditions that yield the following error:
java.lang.RuntimeException: Failed to get driver instance for jdbcUrl=jdbc:postgresql://...
The Ledger Indexer can now be started in 3 modes:
- validate schema and start
- migrate schema and start
- migrate schema only and exit
Contributes to #2660.
* Rename referencedContracts to divulgedContracts
We do not want to provide all referenced contracts as that would
require the ReadService to be able to read all those contracts, or
we would need to bundle the referenced contracts with the transaction.
The new type should match what was discussed in #2488.
* Add changelog entry to participant.state.v1.Version
* Remove divulgedTo from DivulgedContract
We decided not to prematurely add this feature.
* WIP
* Ensure identifier uniqueness across endpoints
* Use Greek alphabet letters for node naming to better tell them apart
* Allow reference server to specify extra participants
* Restore all semantic tests
* Update changelog
* Have exponential backoff retries when creating the endpoint test context
* Randomize hint to ensure tests are rerunnable
* Include descriptive party name prefix for randomized party name cases
* resolve scalafmt woes
* try to resolve failing release azure step
This allows using the dispatcher in situations where the index is common,
but different data streams are materialized from it.
The practical reason for this is to allow using a single dispatcher to read
from both the ledger_entries and configuration_entries tables, which share
the same ledger offset construction.
The time model is being reworked and the current stack is not prepared
to handle this additional check well.
We will reimplement time model related checks once the redesign has been
completed.
* Use proper names for the DB connection pools.
* Add some trace logging for DB operations.
This does not capture timing of individual statements but
rather the time for "units of work".
For example "lookup contract" doesn't mean only loading
a single row from the contracts table, but also 2 additional
queries for looking up witnesses and divulgences.
This is not a problem, because this is trace level logging
that helps us debug problems and shouldn't be made sense of by
users at this stage.
* Make description a by-name parameter.
This avoids building up the string if trace logging
is not enabled (i.e. most of the time).
The kvutils envelope is meant to be used whenever a kvutils produced
data is stored and transmitted over the network. It adds both versioning
and compression to the original message.
* Make sure ActiveContractService sends out a consistent snapshot
instead of looking up the latest ledger end, we can only take the latest known ledgerEnd in the scope of SqlLedger.
If we don't do that, we can miss contracts from a partially inserted batch insert of ledger entries
scenario:
1. batch insert transactions A and B at offsets 5 and 6 respectively; A is a huge transaction, B is a small transaction
2. B is inserted earlier than A and the ledger_end column in the parameters table is updated
3. A GetActiveContractsRequest comes in and we look at the latest ledger_end offset in the database. We will see 6 (from transaction B).
4. If we finish streaming the active contracts up to offset 6 before transaction A is properly inserted into the DB, the client will not see the contracts from transaction A
The fix to that is to use the latest known head to the dispatcher, which is updated AFTER a batch has been inserted completely.
This PR also factors out the basic readonly functionality that is
duplicated between SqlLedger and ReadOnlySqlLedger into a class
BaseLedger.
* copyright header
* add Numeric.java
* ledger-api: rename `decimal` field to `numeric` in value protobuf
* Address Gerolf's comment
* ledger-api: add missing renammings
* ledger-api: relax syntax of numbers that can be sent as numerics
* extractor: fix
* leger-api: change format of number though ledger api
* daml-lf: fix numeric regexp
* ledger: fix tests
* language: compile everything in the source directory
This removes the need to specify a 'main'. Instead we 'source' in
daml.yaml should point to the source root directory.
* Initial version for configuration submissions and authorization
* Refactor config submission based on review. Add tests.
* Cleanup test output and comments
* kvutils test-suite
- Add unit tests for kvutils
- Thread the input state into Commit monad
- Fix a bug in handling rejected transactions (from previous refactoring)
* Address PR review
- Add submissionId to configuration changes.
- Revert the dedup of the ParticipantNotAuthorized message. This did not buy much
and we need to duplicate it anyway when modularizing kvutils.
When diagnosing error logging from akka when shutting down the ledger
API server, I wanted to give upgrading akka to the latest good release
between akka and akka-http. Incidentally, this actually removed the
errors and the shutdown should now be successful.
* The first draft of LedgerClient JWT support
not integrated with HTTP JSON API gateway yet.
* JWT pass-through for create and exercise commands
* JWT pass-through for contracts search and lookup
* Removing unused class
* Readme updates
This test represents the behavior of contract keys before the change of behavior in DAML LF 1.DEV.
The existing ContractKeys test is renamed to ContractKeysSubmitterIsMaintainer.
The `LedgerTestSuiteRunner` now uses a fixed number of
threads for the runner threadpool. This way we don't flood
the CPU with threads when using a CachedThreadPool.
Example:
ContractKeys
- Divulged contracts cannot be fetched or looked up by key ... Success
- Contract Keys should reject fetching an undisclosed contract ... Success
- Contract keys should be scoped by maintainer ... Success
Divulgence
- Divulged contracts should not be exposed by the transaction service ... Success
- Divulged contracts should not be exposed by the active contract service ... Success
Identity
- A ledger should return a non-empty string as its identity ... Success
SemanticTests
- Test:consistency_doublespend1 ... Success
- Test:contract_keys_test ... Success
- Test:consistency_doublespend2 ... Success
- Test:authorization_success1 ... Success
- Test:authorization_rule2_failure ... Success
- Test:authorization_rule1_failure ... Success
- Test:privacy_projections1 ... Success
- Test:consistency_doublespend3 ... Success
- Test:authorization_success2_delegation ... Success
Time
- Advancing time should return the new time ... Success<Paste>
* New acceptance test tool prototype
* Fix transaction filter helper
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r308081386
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r308082432
Moreover, creates specific helpers for test DAML code.
* Move helpers from session to context
Remove a circular dependency between session and context, and limits the
usage of implicits.
* Run tests with ledgers out of process
* Prepare for drop in
* Temporary timeout increase
* Move bulk of the logic to the test runner
* Have a map of tests for CLI integration
* Timeout per-test, runner uses context, renaming
* Allow usage of time service, integrate semantic tests
* Integrate CLI SSL configuration
* Integrate failure expectation CLI option
* Integrate with failure output verbosity CLI option
* Integrate with test listing and picking CLI options
* Integrate with timeout scaling CLI option
* Integrate with command TTL CLI option
* Review CODEOWNERS
* Drop previously unimplemented CLI parameters
* Expand test todo list to encompass latest additions
* Deduplicate some testing infrastructure, swap in new test tool
* Remove unused rule
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311045444
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311051974
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311035287
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311055290
* WIP
* Make gRPC thread factory spawn non-deamon threads
* Remove debug prints, complete divulgence test (missing assertion messages)
* Try to detect assertion failure line number
* Add Sandbox binary that starts an ephemeral postgres instance
This allows us to use the client_server_test bazel macro to run against
the sandbox backed by postgres.
Fixes#1543
* Add Sandbox binary that starts an ephemeral postgres instance
This allows us to use the client_server_test bazel macro to run against
the sandbox backed by postgres.
Fixes#1543
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311400695
* Remove TlsConfiguration duplicate, move original to commons
* Make assertion more readable
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Fix build errors introduced in previous commit
- exclude time service tests from reference-v2
- add copyright header to TlsConfiguration
* Complete divulgence test and address review comments
Restores comments from original divulgence tests and strives to be as
close to them as possible.
- Address https://github.com/digital-asset/daml/pull/2282#discussion_r311429169
- Address https://github.com/digital-asset/daml/pull/2282#discussion_r311431156
- Address https://github.com/digital-asset/daml/pull/2282#discussion_r311431913
* Delete old divulgence tests
* Restore methods to fetch contracts by template identifier
* Run conformance tests on in-memory sandbox
* Run conformance tests on the persistent sandbox
* Remove workflow identifier workaround
* New contract keys test (#2452)
* Set channel type for LedgerSession
* Add ContractKeysIT
* Run tests against reference-server v2
* Removed the old ContractKeysIT. However, ...
... I cannot remove ContractKeysChecks completely, because we have
another test path for DARs created prior to DAML-LF 1.6, which doesn't
have the submitter == maintainer restriction for contract keys.
* Use codegen instead of custom code
* Add missing copyright headers
* Drop sandbox conformance test suite (seems to not work on Windows CI)
* Make time simpler
* Test tool tests to run only semantic tests
* Make CI behave more consistently
* Add missing files to run tests on Postrgres
* Fix flaky divulgence test, introduce readable identifiers
* Increase sandbox conformance tests timeout
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r312363246
* Reintroduce test resource extraction
* Read semantic tests from the classpath rather than Bazel
* Use codegen types for ContractKeys integration test
allocateParty returns Primitive.Party and create and exercise also
accept Primitive.Party. This way we don't have to keep a String and
Primitive.Party representation around.
* Bump timeout for sandbox conformance tests
* Remove dependency on Bazel runfiles
It looks like that closing the LedgerApiServer before closing the
AkkaExecutionSequencerPool (AESP) leads to a non-deterministic bunch of dead
letter warnings when stopping sandbox.
Closing the AESP first avoids these warnings.
Fixes#1886
This happened because we translate empty workflow ids to None in the
API-to-domain translation layer. When we store the workflow id, we actually
convert the None case back to an empty string (and not a null value).
When loading the workflow id from the database, we parse the column as a regular
LedgerString, but unfortunately these cannot be empty. Therefore using empty workflow
ids on postgres has been broken since workflow id has changed from a tagged string
to LedgerString.
* kvutils refactoring
Move transaction submission processing to its own class and refactor to
stop early on errors.
Throw a known error rather than use sys.error.
* Refactor package and party allocation
* Introduce "Commit" computation abstraction
This also fixes an issue where the command dedup entry was not created
when a later validation failed and a rejection log entry was created.
* Drop input_log_entries. Add contract instance to DamlContractState.
* Add copyright headers
* Add KeyValueCommitting.submissionOutputs
This cleans up implementations that need to know the outputs of a submission
up-front.
* Avoid double logging for API Services
* Make WitnessesIT a word spec with a "title"
* The test reporter should also print the "should" keyword.
* Enable ALL.THE.TESTS. for reference-server v2
* Read stakeholders through the active contract service
Fixes#2070
Also standardizes the use of American English spelling for `serialize`
and `deserialize`. Standardize. Serialize. Americanize.
* Remove Haskell bindings test workaround
* Address https://github.com/digital-asset/daml/pull/2316#discussion_r308193661
* Cleanup
* WIP
* first integration test + fixture
* minor cleanup
* Implementing ContractService.lookup
* Reverting back to endpoints.all (all2 did not work)
* Cleanup
* replace ApiValue ADT with aliases to daml-lf/transaction Value ADT
* porting rest of navigator to LF Value ADT
* Command Service WIP
* CommandService WIP
* porting more of navigator to LF Value ADT
* last error, not first
* rename ApiValueImplicits file
* special conversion features for ImmArray and FrontStack
- just .to[ImmArray] or .to[FrontStack] any random collection
* finish porting most of navigator main code
* use numeric indices for record field name fallback when pretty-printing
* tuples are not serializable
* use numeric indices for label fallback in JSON verbose encoding
* make traverseEitherStrictly more likely to preserve the seq's class
* to shortcut for ImmArraySeq .to[ImmArraySeq]
* compiling, passing navigator backend tests
* test traverseEitherStrictly more, er, strictly
* pass scalacopts through to scaladoc
* deal with unused warning
* remove unneeded function
* simpler error reporting, more private functions in ApiCodecCompressed
* move slowApply to FrontStack, test it so it actually works
* remove unneeded toStrings; better error from impossible ValueTuple case
* scalafmt FrontStackSpec
* support alternative, label-free record JSON encoding
* Adding domain.CreateCommand + corresponding json formats and dummy json format for lav1.value.Record
* CommandService.create should be done... need to test it
* TODO added
* Cleanup
* move ApiCodecCompressed, ApiValueImplicits, and some aliases to new lf-value-json package
* Using tagged TemplateId type instead of Identifier + exercise command WIP
* adapt navigator to moved pieces
* start defining scalacheck extension to ApiCodecCompressedSpec
* CommandService.exercise + introducing CommandMeta
* Adding command endpoints, can't test them yet, need lf value json formats
* fuse some list operations
- suggested by @stefanobaghino-da; thanks
* blue error message
* Minor fixes after merging librify-navigator-json-compressed, #2136
* experiment with an inductive case in TypedValueGenerators
* finish a List case for TypedValueGenerators; it's revealing
* Introducing API value to LF value converter,
CommandsValidator takes IdentifierResolverLike instead of IdentifierResolver
* cleanup
* remove accidentally readded duplicate aliases
* start tying knots in TypedValueGenerators
* verbatim copy ApiCodecCompressedSpec to lf-value-json
* shift some tests from navigator to lf-value-json
* test Optional and Map for ApiCodecCompressed
* heavier random testing of ApiCodecCompressed
* remove unused dependencies from lf-value-json
* adding value json writer
* cleanup
* Revert "cleanup"
This reverts commit 2e4d153f
* fixing the build
* cleanup
* cleaning up imports
* JsValue to API value is done, needs a test
* cleanup
* use scalac -Ypartial-unification in http-json
* simplify some Traverse instances
* factor CreateCommand and ExerciseCommand traverse instances
* Command create integration test WIP
* Command create integration test WIP, got rid of the JsonReader and JsonWriter for the values, converting values explicitly
* Extracting DomainJsonDecoder and DomainJsonEncoder
* LfV refactoring
* Create command serialize/deserialize test works
* cleanup
* resolving conflicts
* More json encode/decode tests
* logging
* command/create passes integration test now
* Adding readme
* grammar
* TODO added
* GetActiveContractsResponse encoding
* ideintifier conversion renaming
* PackageService resolveTemplateId returns domain.TemplateId now
* Resolving LF Identifier instead of Template ID, this should also work for Exercise command decoding
* cleaning up a bit
* daml-lf: show type in TypedValueGenerators-driven errors
* exercise command json encoding/decoding works
* command/exercise IOU_Transfer integration test passes now
* avoid filter for Gens; makes many contract ID gens not fail
* test ApiCodecCompressed against 100 random types, 20 random values each
* Updating README instructions
* improving error handling, failed futures, get logged and reported to the user now as 500
* [ROUTING DSL] Removing routing DSL, it did not work
* getting rid of HttpEntity.Strict match + cleanup
* fixing the merge conflict
* updating README
* use Show.shows instead of new Show
* List(_) isn't checked, but Seq(_) is slightly safer
* improving test assertions
* Adding /contracts/lookup implementation
* http-json: use ImmArraySeq instead of List; use toRightDisjuction
* http-json: .toList.toSet is shorter than fold
* http-json: replace .leftMap.map with .bimap
* http-json: use subst instead of reimplementing JsonFormat
* http-json: remove unused ExceptionHandler
* http-json: safer == comparison
* Adding two test cases for expected errors
* Adding BazelRunfiles.rlocation magic that supposed to handle windows path for bazel dependencies
* http-json: import, not extend
* Support for contract keys to kvutils
* Fix handling of CreateEvents with contract keys in semantic tester
* Add ContractKeysIT to tests and fix issues related to fetchByKey
* Address code review
This is a breaking change on the ledger api. So far we could rely on the
assumption that contractId == eventId. This assumption doesn't hold
anymore in a daml-on-x setting, where the eventId is created by the
api server, but the absolute contractId is created by the ledger
implementation.
Instead of going through weird contortions to store the relevant data in
the existing database schema, it is more viable in the long term to
remove that field and instead provide facilities to lookup transactions
by contractId.
Contributes to #2068.
* ledger: refactor validateValue from CommandsValidator to new ValueValidator
* daml-lf: add a matchable class for leaf Values except ValueContractId
* ledger: use IdentifierResolverLike for CommandsValidator
* daml-lf: add Traverse[FrontStack]
* navigator: define readRecordArgument and readArgument in terms of ledger-api-common ValueValidator
* navigator: clean up fillInVariantTI
* navigator: replace ledger-api Value writer with LfEngineToApi from ledger-api-common
* navigator: use traverse
* navigator: remove dead sequenceMap
* navigator: adapt to new ValueValidator structure
* ledger: dealias validateValue call
* navigator: a good use case for traverseEitherStrictly
* Add CommandSubmissionTtlIT to Ledger API Test Tool
* Add ActiveContractsServiceIT to Ledger API Test Tool
* Add WitnessesIT to Ledger API Test Tool
* Delete useless test
* Move time manipulation in TransactionServiceHelpers
* Add CommandServiceIT to Ledger API Test Tool
* Add the missing summary case to the ToolReporter
* replace ApiValue ADT with aliases to daml-lf/transaction Value ADT
* porting rest of navigator to LF Value ADT
* porting more of navigator to LF Value ADT
* last error, not first
* rename ApiValueImplicits file
* special conversion features for ImmArray and FrontStack
- just .to[ImmArray] or .to[FrontStack] any random collection
* finish porting most of navigator main code
* use numeric indices for record field name fallback when pretty-printing
* tuples are not serializable
* use numeric indices for label fallback in JSON verbose encoding
* make traverseEitherStrictly more likely to preserve the seq's class
* to shortcut for ImmArraySeq .to[ImmArraySeq]
* compiling, passing navigator backend tests
* test traverseEitherStrictly more, er, strictly
* pass scalacopts through to scaladoc
* deal with unused warning
* remove unneeded function
* simpler error reporting, more private functions in ApiCodecCompressed
* move slowApply to FrontStack, test it so it actually works
* remove unneeded toStrings; better error from impossible ValueTuple case
* scalafmt FrontStackSpec
* support alternative, label-free record JSON encoding
* fuse some list operations
- suggested by @stefanobaghino-da; thanks
* blue error message
* keeping loaded packages after reset
removing unnecessary extra reference
changelog
making InMemoryPackageStore immutable
* using Instant.now when loading packages in IndexServer
* loading Dars from ZipInputStreams
formatting and cleanup
fixing DarReaderTest not compiling
fmt
using IOUtils to read the ZipInputStream
fixing CodeGen
fixing broken legacy DAR loading
fixed v2 ReferenceServer
cleanup
* fixing rebase conflicts
* introduced a fromFile method to avoid having to do boiler-plating where the input is a File
* addressing leaking resources
* cleanup
* inlining closures
* closing the darStream
* make every PublicPackageUploaded update contain exactly one package
* implement update subscriptions that start with offsets with the same major number
* address review comments
* Do not convert the transaction id to Long in LedgerBackedIndexService
The InMemoryLedger does this, because it can make certain assumptions
(like transactionId == offset), but the sql based ledger cannot do that,
because the transactionId may be produced by another backend and thus
have a different format.
* Move tests that use contract keys into their own integration test file
This allows us to run the other tests via the ledger api test tool with
the postgres indexer while InMemoryKVParticipantState doesn't support
contract keys yet.
* Remove a few occurrences of LedgerBegin from TransactionServiceIT.
* Rename CommandTransactionChecksNo1866IT to ContractKeysNo1866IT
* Enable TransactionServiceTests,DivulgenceIT,TransactionBackpressureIT for postgres index|er
* format all the things
* Address a few review comments.
See https://github.com/digital-asset/daml/pull/2032#pullrequestreview-258771927
* Extract reusable base class for ContractKeys tests
This is needed because Bazel's scala_test_suite isolates the tests from
another, and one test cannot extend another one.
* Classify the reference-v2 conformance tests with a long timeout
* DamlOnXSubmissionService shut down of package load executor service
* ReferenceIndexServer specifies viable PackageId
Using dash (-) instead of slash (/)
* connenct LedgerAPI party and package management with KVUtils
* formatting
* address review comments
* add participantId parameter to sandbox and indexer
* annotate TODO's with github issue numbers
* check that submitter is in maintainers when looking up keys
Fixes#1866. Note that this limitation applies both for `lookupByKey`
and `fetchByKey` -- anything involving retrieving a key is affected.
* add UNTIL-LF to run tests up to a certain version of DAML-LF
* name targets for DAML tests better
* add notes about DAML-LF changes
* commit Test.daml with DAML-LF 1.5 rather than compiling it on the fly
* add scenario tests for #1866
* add warnings about future key behavior in docs
* use flag rather than version when executing
1) Change to SuiteResourceManagementAroundAll, transforming the test to
not assume a new/empty ledger for each test case
2) Use distinct workflow ids so the individual test cases don't clash
with each other
3) Use a new UUID for each randomized command id, instead of using the
uuid used per test run.
Semantic tests will be skipped if the specified Ledger API endpoint does not
implement TimeService.
This teaches MultiLedgerFixture to deal with cancelled tests.
This improves Ledger API Test Tool reporter to deal with cancellations better.
* Add external offset column to parameters
We only ever need to use the external offset when restarting the api
server from a persistent data store to resume the ReadService state
update stream from the last processed offset.
In the PostgresIndexer we then need to store the external offset while
adding a new ledger entry.
* Allow disabling the jmx reporter in metricsmanager
* Implement PostgresIndexer component
* Introduce read-only Ledger and Dao
* SandboxEventIdFormatter treats transactionId as opaque string
* Introduce LedgerBackedIndexService
This can be reused both in the Sandbox and the ApiServer
* Implement PostgresIndex component
* Reference Service based on kvutils-v2 and postgres(index|indexer)
* Disable conformance test for ReferenceServer
Adds a PostgreSQL package store to the sandbox.
Previously, packages were only stored in memory and
specified as CLI arguments. So if a user uploaded a
package at run time, he needed to add the package to
the CLI when restarting the sandbox.
* adapting docs to match state as is regarding not supporting dalf files
* patching release notes instead
* validating for ZIP files in Cli parser
* guarding for parsing non zipped archives
* Update release-notes.rst
* Update unreleased.rst
Co-Authored-By: Beth Aitman <bethaitman@users.noreply.github.com>
* Update unreleased.rst
* Update release-notes.rst
* Update release-notes.rst
* provide asynchronous interfaces for party and package management on the participant-state
* provide submission id for correlating req with resp on Write- and ReadService
* add asynchronous party allocation in KVUtils
* start conversion of the package upload and party allocation to synchronous paradigm
* implement synchronous versions of party allocamtion and packacge update
* document recent changes and unify concepts between v1 and v2 versions of
the participant-state interface
* scala formatting
* address review comments
* Ledger API Test Tool: Add --stable-identifiers option.
* Ledger API Test Tool: split stable party and command identifiers options.
This allows to control individually which identifiers are stable and which are
randomized.
* Ledger API tests: introduce TestIdsGenerator.
It combines all places where per-run unification of identifiers was generated.
* Ledger API Tests: use unification methods instead of run*Suffix
* Review fixes
Moves party allocation to its own WriteService trait (similar to package upload).
Changes uploadDar to uploadPackages, so that the write service does not depend on the DAR file format.
Adds or improves many comments.
* Introduce contract key to exercise node
Track the contract key in NodeExercises. This is in
preparation for adding the key into exercise events, and
for support for contract keys in kvutils, where we need
to be able to compute the outputs of the submission, e.g.
the contract key that would be unset by a consuming exercise.
This introduces transaction version 8.
* Move release note on contract keys in exercises to unreleased.rst
* Address Stephen's review
- Drop changelog entry for this change as it is not user facing
- Add info on version 8 to transaction.rst. Fix typo in version 7.
- Reorder VersionTimeline as per review
- Only compare the NodeExercise 'key' if it is set in the original
transaction (in order to compare transactions in a backwards
compatible manner)
* Unbork unreleased.rst
* Run integration tests for sandbox through the IT test suite
We currently don't run sandbox on postgres via ledger-api-test-tool. We
need to first figure out how to best do that before we can switch
sandbox to the test tool again.
* CommandClient#submitSingleCommand also uses commandUpdater
* The test reporter for the test tool now handles ignored tests
* More optional tests for ledger-api-test-tool
- TransactionBackpressureIT
- DivulgenceIT
- CommandTransactionChecksHighLevelIT
- CommandTransactionChecksLowLevelIT
Contributes to #1374.
* Add missing domain.RejectionReason cases
* Use java.time.Instant instead of damlf Timestamp
* Make applicationId, submitterParty, commandId optional in LedgerEntry.Transaction
* store archive size in `DarReader`
* rename `SandboxTemplateStore` to `SandboxPackageStore`
* store package info in `SandboxPackageStore`
* introduce package upload / read to write / index services
not tested yet, just a ton of plumbing
* WIP test the package service
* Fix build errors after rebase
* Move packages service to v2
* Ledger API client uses ledger API types
* Fix ReflectionIT
* Correctly handle uploading invalid dar files
* Fix reading DAR entry file sizes
* Improve package management IT
* Improve handling of duplicate packages
* Fix language-support build
* Use unique party and command names
* Rename lfpackage to language
* Rename SandboxPackageStore to InMemoryPackageStore
* Remove getCurrentTime ledger method
* Improve package management IT
* Move InMemoryActiveContracts and InMemoryPackageStore
* Use case object for UploadDarResult.Ok
* Address review comments
* Update release notes
Fixes#1311
* Use BazelRunfiles in test
This fixes two issues with the sandbox database migration:
* A typo in the migration script
* Error deserializing transactions due to protobuf recursion limits
Fixes#1682
* Fix various issues with the reference daml-on-x implementation
- Add authorization check to submission to catch authorization errors early
- Fix filtering by templates in transaction service
- Loosen the transaction_id validation to accept transaction_ids as used by
the reference implementation
- Fix handling of ledger end
- Add caching to archive decoding
- Enable TransactionIT for the reference implementation
- Fix time service event dedup
* fix transaction validation which started reporting different error type
* fix transaction fetching tests on sandbox that assumed that transaction ids would be verified as numbers by the api validation layer
* return NOT_FOUND in all places where a non-existing transaction id or event id is used to fetch transactions
Scale factor for time-to-live of commands sent for ledger processing
(captured as Maximum Record Time in submitted transactions) for
"SemanticTests" suite. Useful to tune Maximum Record Time depending on
the environment and the Ledger implementation under test. Defaults to
1.0. Use numbers higher than 1.0 to make timeouts more lax, use
numbers lower than 1.0 to make timeouts more strict.
Fixes#1312
This PR adds support for party management in the sandbox:
Both the in-memory and the SQL backend track a list of known
parties, and implicitly add any party mentioned in a transaction.
New calls were added to the IndexService and the WriteService
for managing parties. These calls are wired to the above mentioned
persistence backends, and to a new API service.
A corresponding client interface was added to the scala API client.
An integration test was added for checking that a call to allocate a
party succeeds.
An integration test for the sandbox was added for checking that the
sandbox implicitly adds all parties mentioned in a transaction.
Extra command line arguments to the client_server_test target are passed
to the ledger-api-test-tool instead of the arguments defined in the
bazel target.
For example:
`bazel run //ledger/sandbox:conformance-test-static-time --list` will
pass the --list parameter to the ledger-api-test-tool (to print all
available tests), instead of actually running the tests in static-time mode.
The Ledger API Test Tool takes the flag --all-tests to run all defaul
and optional tests
The tool also allows multiple occurrences of the --include and --exclude
flags.
Also removes StandaloneSemanticTestRunner.
Fixes#1371
* Provide initial implementation of package management in the KVUtils
* rename updateArchives to updatePublicPackages and add them to the abstract interface
* remove participant-state:reference app from published artifacts
* adopt the changes to the recent version split
Add tTansactionServiceIT into Ledger API Test Tool as an optional test.
Available tests can be listed with the --list option.
Tests can be included with --include, and excluded with --exclude.
Fixes#1372, #1472
* added recovery logic when ledger entry persistence fails
* adding release notes
* logging only when the queues are closed due to an error
* using error instead of warn
* Drop support for DAML-LF 1.3 from damlc
This means `damlc` now only supports versions allowing arbitrary expressions
as contract keys. This is very helpful for the upcoming new template
de/re-sugaring underpinning generic templates.
* Unpin a few DAML-LF target versions in tests
* Fix golden tests
* Adapt changelog to rebase
* Rename DAML-LF scenario tests to 1.dev
* Update tests to --target 1.dev
* Fix release notes
These were mistakenly disabled along with the move to tests-as-a-library
construct. The 'scala_test_suite' macro does discovery only over sources, not
dependencies.
Because the record time was mapped to the ledger effective time field
when deserializing contracts from the SQL database, a subsequent
comparison between the LETs of two nodes in the same transaction caused
the transaction to be rejected.
Fixes#1435.
* SandboxLedgerBackend implements CompletionsService from IndexService
* things compile
* cleanup
* cleanup
* commandId is not optional anymore
* proper filtering of completions
* Add RemoteApiProxy fixture type.
This is in preparation for using Sandbox IT suite as part of the Ledger API Test
Tool.
* ledger-api-test-tool: Drop reset functionality.
This is no longer necessary for the tool and it does not scale with the types of
tests in the suite.
* integration-tests: Fail if the server under fixture is stuck.
This makes sure that a server getting stuck will get detected by a test, instead
of ignoring it and potentially allowing the server to linger.
* integration-test: Make semantic testing runs independent.
It manges parties and command identifier to include a unique (random) suffix in
all ledger-commited identifiers. This allows the test to run against a Ledger
API without reseting it.
* ledger-api-test-tool: Unify test code using scenario runner with IT suite.
This reuses the scenario runner test code from the IT suite, instead of
reimplementing it. This should be a no-op (except for tests reports formatting).
* Review fixes.
* Ledger API Test Tool: Provide logback config.
This quites Ledger API Test Tool output.
* Make sure akka threads are terminated at end of test runs.
This makrs Akka threads to be daemons, hence forcing them to be closed at the
end of Ledger Api Test Tool.
* Use Ledger API Test Tool in tests of reference server.
* Add Apache commons-lang3.
* Ledger API Test Tool: Implement custom test reporter.
This addresses two needs:
- avoid using buggy scalatest test reporter;
- pretty-prints test results prettier.
* dade-copyright-headers: return success on successful reformatting.
* Isolate daml-on-x prototype
* Implement TransactionService of index api.
Towards the first milestone of the daml-on-x integration steps
* fix compiler error after last-minute refactoring
* formatting
* remove unnecessary dependency from api-server-damlonx
* Fix some inconsistencies.
This prevents binaries from including random logback configuration via library
dependencies.
Relevant binaries were found with:
bazel query --order_output=no --universe_scope='//...:*' \
'kind(".*(test|binary).*", allrdeps(//ledger/ledger-api-integration-tests:ledger-api-integration-tests-lib))'
This is a preparation for making the actors in DAML-LF's exercise
instruction optional (see #1347).
This is technically a breaking change but since `exerciseExplicit` has never
been exposed via the documentation, we don't add a release note.
* Rename Value's ContractId to VContractId
* daml-lf: a bit more about PartyId
* daml-lf: Concatenable MatchingStringModule
* daml-lf make clear type used for Scenarios only
* daml-lf create ContractId, LedgerId, TransactionId
* sandbox-sql conversion util
* LedgerName -> LedgerString
* futher type cleanup in the sandbox
* daml-lf add test for LedgerString
* fixing tests
* a bit more safety in the DB
* Address Stephen's comments
* fix rebase
* More fixes for StringModule
* change length of LedgerString (256 -> 255)
* Active Contacts Service from IndexService is wired through
* almost all tests pass
* fixing the offset toString issue
* removing irrelevant .await in RemoteServerResource
The index service serves as a "bridge" between the data on the ledger
and the ledger api users. That is also the reason why the index service
API is defined in terms of the types in ledger-api-domain. It is the
responsibility of the index service implementations to return compatible
types and adhere to the expected semantics, so we can actually reuse
index service implementations in the ledger api servers.
The foldWithPathState is a fold over the transaction that allows
keeping extra state along the path of the traversal.
We add computePerPartyProjectionRoots in preparation for the
work on privacy-aware kvutils, where transaction nodes are stored
separately rather than in one blob. This function allows us to compute
the projected transaction roots for each party mentioned in the
transaction. We require this information to establish the order
of the root nodes when the size and form of the transaction is
blinded by assigning random node ids.
* using IdentityService
* package service wired through
* using ledger id from identityService
* extracting out ApiServer trait
* making ApiServer creation async
* using async tolerance when waiting on ApiServer creation
Fixes#1250
The previous behavior when receiving a transaction filter without any
party was to reply with an empty stream. Since, given the current
situation, no data could ever be served for such request, it represents
a better feedback for the user to reject such requests as carrying an
`INVALID_ARGUMENT`.
* extracting some sub-interfaces
* dump
* dump
* things compile
* extracting more concerns
* comment change
* making configuration service return a stream
* adding documentation around the new interfaces
* Added agreement_text field to the CreatedEvent in Ledger API.
* Changed java bindings + java codegen
* Changed utilities for scala codegen
* Made necessary changes in Sandbox to propagate the agreement text from ContractInst to the CreatedEvent
* Made changes to the navigator to show the agreement text in the contract details page when it is set and not empty
Fixes#1110
* Add tracking of disclosure to contract state
And clean up some of the error handling in processSubmission.
* Do not add, but rather union the divulged parties
* no var no problem
further refactor
introduced InfraState
a bit less vars
encapsulating closes
SandboxServer starts automatically
rebase fixup
collecting state into a single object
some cleanup
removing exposed materializer
LedgerBackend is closed in SandboxServer
changed ownership of Ledger
fixing perf tests
fixing some compile errors
formatting
removing unused method
fixing integration test to use correct dar file
fixing issue with PostgresFixture and SandboxResource
Fix integration tests on Windows
* fixing rebase artifacts
* Store divulgence info for contracts
* Store divulgence info for contracts
* some documenting refactors
* Use fail instead of sys.error in test
* Add release note
* Rename migration files
* Fix typo
* Fix updating divulgence info in scenarios
The ACS was comparing differently formatted contract IDs.
Fixes#1166
* ledger-api-intgration-tests: Cleanup ledger ID handling.
This changes nothing in the logic of tests. It is a preparation for moving of
most of the tests to use Dynamic ledger ID mode for tests, which is a
preparation for running these tests against remote Ledger API endpoints.
* sandbox: Rework LedgerIdMode and move it up into common.
* daml-lf: make Decimal type safe
* daml-lf: create Utf8String type
* daml-lf: cleanup in data package
* Address Stephen Comments
* daml-lf: remove UTF8String
* enable Return wartremover wart
* remove return keyword from various places in daml-lf
* remove return keyword from various places in ledger
* simpler ImmArray equals
* move traverseEitherStrictly to point of use
* remove return from ledger-api-server-example
* Remove ExercisedEvent in Event oneof.
The Event message is only used in the Transaction message. Flat
transactions do not contain exercised events, but only created and
archived events. Therefore we can remove the ExercisedEvent from the
Event oneof, without breaking transport compatibility.
HOWEVER: The Java Bindings used to use the data.Event class for both flat
transactions and transaction trees. To properly represent the actual
event types in the two transaction structures,
1) Event is now and interface and is only used in the Transaction class.
2) there is a new interface TreeEvent, which is used in the TransactionTree class.
* CreatedEvent implements Event and TreeEvent
* ExercisedEvent implements TreeEvent
* ArchivedEvent implements Event
Some "pathological" cases where an occurrence of an exercised event
would have resulted only in an exception, are now removed (see change in
LedgerApiV1.scala).
Fixes#960.
* adding necessary @volatiles
* services are factored out
* no need for ResetService to stop the services before
* removing not needed callback methods
* mostly renaming things
* some further cleanup
* avoiding linear searching for transactions from genesis
* validating transaction ids
* one more failing test nailed down
* fixing off by 1 error
* docs + release notes
* fixing EventIdFormatterSpec
* fixing broken validation test
Up to now, the engine blindly assumed that contract ids pointed to
contracts of the right type. However, this assumption is faulty:
contract ids coming from the Ledger API cannot be type checked
in command translation since we need access to the contract itself
to do so.
This caused some seriously surprising / broken behavior: one could
send an exercise command with the wrong template id and still go
through, or break internal invariants about the type of choices.
This commit fixes this by checking that the type of the contract
instances we fetch is correct at runtime.
cc @hurryabit @dajmaki @remyhaemmerle-da @S11001001 @meiersi-da
* Remove ledger-api-server-example
This code does not serve any purpose anymore as it has been replaced
by ledger/participant-state and friends.
* Remove ledger-api-server-example from Windows build
* Use Sandbox's ActiveContracts in reference index service
This properly handles contract key updates and adds the required
checks for contract activeness and visibility.
Could consider moving the ActiveContracts.scala into another library,
but since we're soon starting the work to bring sandbox onto the
daml-on-x stack we might postpone that.
* reformat
* Exclude logback.xml from sandbox the library
This allows users of sandbox as a library to specify their
own logging configuration.
* Re-order fields in records if all labels are provided
Fixes#988
* Add line to release notes
* Fix line on release notes
* Document change on the Ledger API definition
* Simplify pattern matching
* Improve and fix tests
Integration tests now look for the "Missing field label" rather then the
"Mismatching record label" when inspecting errors. Furthermore, a test
has been added that ensures that repetitions in labels still raise an
error.
* Add comment
Comment about relationship between fields length check and fields reordering based on labels
Co-Authored-By: stefanobaghino-da <43749967+stefanobaghino-da@users.noreply.github.com>
* Improve Ledger API docstring
Explicitly state the semantics of labeled fields (i.e. repeated keys are not accepted)
Co-Authored-By: stefanobaghino-da <43749967+stefanobaghino-da@users.noreply.github.com>
