* ledger-api-test-tool: Allocate participants and parties declaratively.
Part of a solution to #3380.
Rather than allocating participants and parties line by line, instruct
the test to construct a number up-front.
This will make it easier to serve multi-participant ledgers, as we can
make sure we wait for all parties to be available on all participants
before starting the test. Once we can do that, tests will hopefully be
much less flaky on ledgers other than the Sandbox.
* ledger-api-test-tool: Rename `provision` to `allocate` and document it.
* ledger-api-test-tool: Make Allocation functions private where possible.
* ledger-api-test-tool: Fix the comment for `LedgerTestContext::allocate`.
It got the signature wrong.
* ledger-api-test-tool: Just use `Duration::toString` for logging.
* ledger-api-test-tool: Fix reporter highlighting.
Previously, the closing parenthesis around the test duration was not
highlighted. Now neither are.
Closes#2844.
It turns out that this test doesn't really exacerbate any backpressure;
it just fires lots of commands. In the interest of not losing this, I've
ported it to a test in TransactionService.
Without the throttling, we _do_ hit resource exhaustion on Sandbox with
the PostgreSQL server backend. Capping the number of commands at 500,
rather than 1000, seems to solve this.
* Implement proper stream pagination
The previous pagination mechanism for streaming ledger entries was implemented
as a recursive method call with manually concatenating akka Sources.
However, this didn't work properly and resulted in all the subsources being
forced immediately, resulting in parallel requests for all pages
(0-100, 100-200, 200-300, ...) instead of the expected mechanism of loading
the first page and on upstream request (i.e. client requests for more data
over grpc) loading the next page of data.
The current mechanism uses Source.unfoldAsync which makes the paging
mechanism work as expected: sequential loading of pages on demand.
* Serialize and deserialize Transactions outside the SQL Executor
This frees up the SQL Executor threads sooner for other work
and the conversion work only happens on a by need basis when
the consumer requests more data.
* Divulged contract visibility in multi-participant environments #3351
* Ledger api server time service optional ability for testing #3225
* Allow ledger api server to share DAML-on-X DAML engine #2975
* Allow ledger api server participant ids with LedgerString chars #3327
* Ledger api server includes SQL description in errors #3324
* Refactor proposal for "KeyValueCommitting"
This only changes the package uploads as an example.
Embrace the imperative style:
* we carry a lot of state through the commit steps anyway and immutable maps are actually
dangerous: we may end up reading from the wrong one etc (has happened).
* performs much better with transactions that have thousands of outputs (the current monadic commit would
have quite a lot of overhead for setting all these).
Drop the "Option[DamlStateValue]". This was only for dedup value and the extra complexity for catching the case
where some inputs weren't fetched isn't worth the complexity (tests will catch this anyway).
Return an Iterable[(DamlStateKey, DamlStateValue)]. This way we're clear about the ordering of the outputs.
* Address review and simplify
* Port the change to use daemon thread for preload executor
So that JVM doesn't get stuck when exiting.
* Fix typo
* Add client_server_build and integrity_test rules
And use them to implement ledger dump of the reference
server and to check it.
* Only build and test ledger dump on Linux. Only run tests relevant to dump.
* Make client_server_build quiet in happy path
* Reformat
* Remove unnecessary runfiles for client_server_build
* ledger-api-test-tool: Use Scala's `Duration` for timeouts.
* ledger-api-test-tool: Remove the parentheses from a getter.
* ledger-api-test-tool: Run tests sequentially.
Using Akka Streams to manage the concurrency.
* ledger-api-test-tool: Parallelize across available processors.
* sandbox/reference-v2: The test timeout scale factor is no longer needed.
Now that we don't run all tests at once, the measurements are much more
accurate.
* sandbox/reference-v2: Run LotsOfPartiesIT 11 times instead of 16.
* ledger-api-test-tool: Highlight slow tests in yellow and red.
* ledger-api-test-tool: Fix some warnings in the Reporter.
* ledger-api-test-tool: Sort the tests afterwards for better concurrency.
Turns out that Akka Streams tries to maintain order at the expense of
running more things, which is not ideal. This is a hack to let us use
`.mapAsyncUnordered` and then re-sort into the original order at the
end.
Ideally `.mapAsync` would detect when subsequent futures have finished
and start running more, even if the first hasn't finished yet.
* ledger-api-test-tool: Inline `availableProcessors` into Config.
* new library ledger-service/db-backend
* borrow contracts table schema from extractor
* borrow contract insertion, removing some data to be unused
* match contract schema with insert function
* factor insertContract arguments
* offset table declarations
* CLI argument for query store
* surrogate template IDs
* compute surrogate template IDs on-the-fly
* database init action
* incoherent typeclasses, eh
* newtype SurrogateTpId
* offset fetch/update functions
* bad sql
* bulk insert contracts, function for selecting contracts
* expose contract column name for query's usage
* Initializing DB on startup if configured
* dropping existing tables as part of initialization
* fix some query syntax errors
* createSchema flag
* function for streaming transactions with jwt party selected
* formatting
* usage
* collect acs contracts and the ledger offset at the end
* lastOffset
* fixing merge conflicts, updating the way 3rd party deps are specified
* Moving ContractDao into http-json module
so it can take domain AST as an input
* cleanup
* injecting new dependencies
* split transaction batches into inserts and deletes
* generate sql for deleting contracts
* `fetch_sources = True` for java_deps
* make the delete-constructed fragment more efficient; handle empty list here
* pass logHandler for insertContracts
* ContractDao returns ConnectionIO, it's up to the caller to wrap query into a transaction
* fixing typo
* minor cleanup, moving fromLedgerApi factory function into corresponding companion objects
* don't need it any more
* GetActiveContractsResponse => domain.Contract factory
* make concatFragment private
* add partition graph; move other contract-fetching experiments to ContractsFetch
* experimenting with akka sources
* introducing domain.Offset to work around API's empty/null offset cases
* minor cleanup
* decompose fetchActiveContractsFromOffset
* missed via
* ACS splitting graph
* finish doc for ACS splitting graph
* remove unneeded stages
* WIP
* lazily read a stream of ConnectionIO into a single ConnectionIO
* cancel on IO error
* figuring out how to put all the pieces together
* graph WIP
* Removing workflowId from the JSON API
* simplify acsAndBoundary; describe other flow pieces
* WIP
* use Vector in InsertDeleteStep; add variant for ACS (no deletes)
* `org.wartremover.warts.NonUnitStatements` enforced in `http-json` module
* evaluate InsertDeleteStep to a ConnectionIO
* database variant of LfValueCodec, using numbers for numbers
* convert input to JSON, combine insert plans, connect rest of contractsToOffsetIo
* remove strict contractsToOffset sink
* moving dao methods into an object
* putting pieces together
* contractsFromOffset WIP
* should be it
* cleanup
* cleanup
* contractsIo that takes List[domain.TemplateId.RequiredPkg])
* contractsIo that takes List[domain.TemplateId.RequiredPkg])
* cleanup
* put all pieces together, testing
something does not work yet
* diff is not required to return anything
that is why Sink.lastOption that gives Option[domain.Offset]
* factor out tuple split
* use traverse syntax in contractsIo2
* factor explicit flow steps out of graph DSL; remove aggregate
* locally model the Absolute/Begin distinction for offset bookmarking in DB
* Adding test cases to run HTTP Service with Postgres backend
the same set of test cases, run with and without DB backend
* make better use of domain.Offset in OffsetBoundary
* monomorphize InsertDeleteStep#append
* Disabling a test that fails with DB backend
* add release note
* add release note about workflowId
* a test case that checks the number or stored contracts
* trying to figure out why Postgres test fails on Windows with NPE
* ledger-api-test-tool: Avoid having to build the vector of tests.
Instead, maintain a buffer that is added to when tests are defined.
* ledger-api-test-tool: Inline `LedgerTest.apply` into LedgerTestSuite.
* ledger-api-test-tool: Rename `LedgerTest` to `LedgerTestCase`.
* ledger-api-test-tool: Port LotsOfPartiesIT.
* ledger-api-test-tool: Run LotsOfPartiesIT with multiple participants.
Odds of me getting this wrong: 80%. Let's find out!
* ledger-api-test-tool: More underscores, methinks.
* ledger-api-test-tool: Improve the readability of the LotsOfParties test.
Instead of all the functional magic, just do things twice. It's even
shorter in some cases.
* ledger-api-test-tool: Not so many parties. :-(
* sandbox: Extract duplication between conformance test definitions.
* sandbox: Run the LotsOfPartiesIT conformance test separately.
* Revert "ledger-api-test-tool: Not so many parties. :-("
This reverts commit 8039a7cc9b.
* sandbox: Add the copyright header to sandbox_conformance_test.bzl.
* Run `./fmt.sh`.
* ledger-api-test-tool: Generalize `sandbox_conformance_test`.
I want to use it with the DAML-on-X reference v2.
* api-server-damlonx/reference-v2: Use `server_conformance_test`.
* api-server-damlonx/reference-v2: Run LotsOfPartiesIT separately.
* Pre-port cleanup of helpers
* Further clean-up
* Remove dead code
* More cleaning and flattening
* Port first tests from CommandClientIT
* More cleanup and more dead code removal
* Port exercise by key test
* Port observer disclosure tests
* Port tests, remove duplicates and dead code
* Fix compilation error and remove more dead code
* Remove unused import
* Spread helpers to single users, remove materializer from helpers
* More test porting and cleanup
* Remove more dead code
* Restore tests deleted by mistake
* Restore final qualifier removed by mistake
* Fix typo
* Optimize imports
* Move timeout from helper to tests
* Remove unused class
* Never gonna give you up
* Address https://github.com/digital-asset/daml/pull/3320#discussion_r342018581
* Address https://github.com/digital-asset/daml/pull/3320#discussion_r342025161
* Remove removed test :)
* ledger/ledger-api-test-tool: Implement diff correctly for sequences.
Previously, we were using `x.ai.diff.conversions._`, which converts them
into sets. This is obviously not helpful when comparing, for example,
`Seq(7, 8, 9) and `Seq(7, 7, 8, 9)`, as the assertion will pass.
* ledger/ledger-api-test-tool: When diffing sequences, stay lazy.
* Fix the import paths of {value,transaction,blindinginfo}.proto
This cleans the imports in daml_kvutils.proto:
-import "daml-lf/transaction/src/main/protobuf/transaction.proto";
+import "com/digitalasset/daml/lf/transaction.proto";
This shouldn't be a breaking change to the outside as the proto files
themselves are not published, only the java libraries generated from them.
* Update missed deps
* ledger-api-test-tool: Use java.time instead of doing the sums ourselves.
* ledger-api-test-tool: Use `asProtobuf` everywhere.
* ledger-api-test-tool: Avoid scala.concurrent.duration where possible.
* ledger-api-test-tool: Add the copyright header to ProtobufConverters.
* Make Err an abstract class so it is throwable when used from Java
* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/participant/state/kvutils/Err.scala
bring back sealed
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
The symptom for receiving messages larger than the configured maxInboundMessageSize is a
gRPC error like:
Oct 31, 2019 1:52:37 PM io.grpc.internal.AbstractClientStream$TransportState inboundDataReceived
INFO: Received data on closed stream
Fixes#3301.
kvutils now supports dumping the ledger to a file via environment
variable: "KVUTILS_LEDGER_DUMP=/tmp/ledger.dump".
The integrity checker tool allows re-processing of ledger dumps to
validate that all submissions, state and log entries can be parsed
and processed. This forms the basis of kvutils data continuity test
suite.
Currently the checker strictly enforces that re-processed
outputs match the inputs exactly, which for now guarantees forwards
and backwards compatibility. The checker will be adapted when
forwards-incompatible changes or new kvutils message versions arise.
A select collection of ledger dumps will be maintained in a separate
repository (to be decided) and these dumps are to be validated by
the DAML CI.
The global implicit disclosure map was missing the `#` prefix for contract ids, which caused subsequent contract id lookups to not find the right contract.
Instead of loading and deserializing the transaction, we store the
stakeholders in tables contract_signatories and contract_observers.
To avoid having to do an extra roundtrip for fetching the stakeholders,
we aggregate the parties at query time into a % separated string (% is
not a valid character for parties), which we then split again in the
application.
Fixes#2330.
* Update bazel-common to fix javadoc issues
Specifically, to fix the following error
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel:7:1: in javadoc_library rule //ledger-api/rs-grpc-bridge:rs-grpc-bridge_javadoc:
Traceback (most recent call last):
File "/home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel", line 7
javadoc_library(name = 'rs-grpc-bridge_javadoc')
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/com_github_google_bazel_common/tools/javadoc/javadoc.bzl", line 27, in _javadoc_library
dep.java.transitive_deps
object of type 'JavaSkylarkApiProvider' has no field 'transitive_deps'
```
* Define Maven deps using rules_jvm_external
* Pin artifacts
* Remove bazel-deps generated targets
* Remove bazel-deps
* Switch to rules_jvm_external targets
* update bazel documentation
* pom_file: There are no more bazel-deps targets
* BAZEL-JVM.md `maven_install` typo
* speedy: limit allocation of trivial SValue constants
* daml-lf: limit allocation of constants for Value, Exp, Type
* weaken numeric equality in ledger service test
* Properly fill eventId for active contracts
This gets rid of the last remaining bit that assumes
contractId==eventId.
Fixes#65.
Contributes to #2068.
* Do not conflate eventId and contractId in the daml-lf interpreter
* Do not treat contractId as eventId in Ledger.scala
* Remember the transaction that divulged a contract.
* In this scope we can treat divulged contracts the same as disclosed ones
* revert a few more syntactical changes to make the overall diff smaller
* retain the same behavior on the scenario service api
* fix unreleased after rebase
* Add failing test case for transient contract keys
* Add kvutils tests for transient contracts and keys
* Fix handling of transient contracts and keys in kvutils
This removes the need for clients to handle the special case where the
stream might be empty.
Now the clients can always assume that they receive at least one
response element in the stream.
* Fix contract key uniqueness check in kvutils
Archival of a contract with a key and recreation within the
same transaction is now allowed in kvutils.
* Add assertions to check that new contract has been created with same key
Since we still have issues with parallel writes when using H2,
we should properly only use a single DB connection and executor thread
for that.
Because we didn't do that before for Sandbox, tests like PackageManagement
appeared to be flaky due to racy inserts in H2.
* Preload packages to engine during upload
* Improve logging in KeyValueCommitting and add timing information
* Fix scenario service tests now that logging is done in interpreter
This is believed to prevent race conditions that yield the following error:
java.lang.RuntimeException: Failed to get driver instance for jdbcUrl=jdbc:postgresql://...
The Ledger Indexer can now be started in 3 modes:
- validate schema and start
- migrate schema and start
- migrate schema only and exit
Contributes to #2660.
* Rename referencedContracts to divulgedContracts
We do not want to provide all referenced contracts as that would
require the ReadService to be able to read all those contracts, or
we would need to bundle the referenced contracts with the transaction.
The new type should match what was discussed in #2488.
* Add changelog entry to participant.state.v1.Version
* Remove divulgedTo from DivulgedContract
We decided not to prematurely add this feature.
* WIP
* Ensure identifier uniqueness across endpoints
* Use Greek alphabet letters for node naming to better tell them apart
* Allow reference server to specify extra participants
* Restore all semantic tests
* Update changelog
* Have exponential backoff retries when creating the endpoint test context
* Randomize hint to ensure tests are rerunnable
* Include descriptive party name prefix for randomized party name cases
* resolve scalafmt woes
* try to resolve failing release azure step
This allows using the dispatcher in situations where the index is common,
but different data streams are materialized from it.
The practical reason for this is to allow using a single dispatcher to read
from both the ledger_entries and configuration_entries tables, which share
the same ledger offset construction.
The time model is being reworked and the current stack is not prepared
to handle this additional check well.
We will reimplement time model related checks once the redesign has been
completed.
* Use proper names for the DB connection pools.
* Add some trace logging for DB operations.
This does not capture timing of individual statements but
rather the time for "units of work".
For example "lookup contract" doesn't mean only loading
a single row from the contracts table, but also 2 additional
queries for looking up witnesses and divulgences.
This is not a problem, because this is trace level logging
that helps us debug problems and shouldn't be made sense of by
users at this stage.
* Make description a by-name parameter.
This avoids building up the string if trace logging
is not enabled (i.e. most of the time).
The kvutils envelope is meant to be used whenever a kvutils produced
data is stored and transmitted over the network. It adds both versioning
and compression to the original message.
* Make sure ActiveContractService sends out a consistent snapshot
instead of looking up the latest ledger end, we can only take the latest known ledgerEnd in the scope of SqlLedger.
If we don't do that, we can miss contracts from a partially inserted batch insert of ledger entries
scenario:
1. batch insert transactions A and B at offsets 5 and 6 respectively; A is a huge transaction, B is a small transaction
2. B is inserted earlier than A and the ledger_end column in the parameters table is updated
3. A GetActiveContractsRequest comes in and we look at the latest ledger_end offset in the database. We will see 6 (from transaction B).
4. If we finish streaming the active contracts up to offset 6 before transaction A is properly inserted into the DB, the client will not see the contracts from transaction A
The fix to that is to use the latest known head to the dispatcher, which is updated AFTER a batch has been inserted completely.
This PR also factors out the basic readonly functionality that is
duplicated between SqlLedger and ReadOnlySqlLedger into a class
BaseLedger.
* copyright header
* add Numeric.java
* ledger-api: rename `decimal` field to `numeric` in value protobuf
* Address Gerolf's comment
* ledger-api: add missing renammings
* ledger-api: relax syntax of numbers that can be sent as numerics
* extractor: fix
* leger-api: change format of number though ledger api
* daml-lf: fix numeric regexp
* ledger: fix tests
* language: compile everything in the source directory
This removes the need to specify a 'main'. Instead we 'source' in
daml.yaml should point to the source root directory.
* Initial version for configuration submissions and authorization
* Refactor config submission based on review. Add tests.
* Cleanup test output and comments
* kvutils test-suite
- Add unit tests for kvutils
- Thread the input state into Commit monad
- Fix a bug in handling rejected transactions (from previous refactoring)
* Address PR review
- Add submissionId to configuration changes.
- Revert the dedup of the ParticipantNotAuthorized message. This did not buy much
and we need to duplicate it anyway when modularizing kvutils.
When diagnosing error logging from akka when shutting down the ledger
API server, I wanted to give upgrading akka to the latest good release
between akka and akka-http. Incidentally, this actually removed the
errors and the shutdown should now be successful.
* The first draft of LedgerClient JWT support
not integrated with HTTP JSON API gateway yet.
* JWT pass-through for create and exercise commands
* JWT pass-through for contracts search and lookup
* Removing unused class
* Readme updates
This test represents the behavior of contract keys before the change of behavior in DAML LF 1.DEV.
The existing ContractKeys test is renamed to ContractKeysSubmitterIsMaintainer.
The `LedgerTestSuiteRunner` now uses a fixed number of
threads for the runner threadpool. This way we don't flood
the CPU with threads when using a CachedThreadPool.
Example:
ContractKeys
- Divulged contracts cannot be fetched or looked up by key ... Success
- Contract Keys should reject fetching an undisclosed contract ... Success
- Contract keys should be scoped by maintainer ... Success
Divulgence
- Divulged contracts should not be exposed by the transaction service ... Success
- Divulged contracts should not be exposed by the active contract service ... Success
Identity
- A ledger should return a non-empty string as its identity ... Success
SemanticTests
- Test:consistency_doublespend1 ... Success
- Test:contract_keys_test ... Success
- Test:consistency_doublespend2 ... Success
- Test:authorization_success1 ... Success
- Test:authorization_rule2_failure ... Success
- Test:authorization_rule1_failure ... Success
- Test:privacy_projections1 ... Success
- Test:consistency_doublespend3 ... Success
- Test:authorization_success2_delegation ... Success
Time
- Advancing time should return the new time ... Success<Paste>
* New acceptance test tool prototype
* Fix transaction filter helper
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r308081386
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r308082432
Moreover, creates specific helpers for test DAML code.
* Move helpers from session to context
Remove a circular dependency between session and context, and limits the
usage of implicits.
* Run tests with ledgers out of process
* Prepare for drop in
* Temporary timeout increase
* Move bulk of the logic to the test runner
* Have a map of tests for CLI integration
* Timeout per-test, runner uses context, renaming
* Allow usage of time service, integrate semantic tests
* Integrate CLI SSL configuration
* Integrate failure expectation CLI option
* Integrate with failure output verbosity CLI option
* Integrate with test listing and picking CLI options
* Integrate with timeout scaling CLI option
* Integrate with command TTL CLI option
* Review CODEOWNERS
* Drop previously unimplemented CLI parameters
* Expand test todo list to encompass latest additions
* Deduplicate some testing infrastructure, swap in new test tool
* Remove unused rule
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311045444
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311051974
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311035287
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311055290
* WIP
* Make gRPC thread factory spawn non-deamon threads
* Remove debug prints, complete divulgence test (missing assertion messages)
* Try to detect assertion failure line number
* Add Sandbox binary that starts an ephemeral postgres instance
This allows us to use the client_server_test bazel macro to run against
the sandbox backed by postgres.
Fixes#1543
* Add Sandbox binary that starts an ephemeral postgres instance
This allows us to use the client_server_test bazel macro to run against
the sandbox backed by postgres.
Fixes#1543
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311400695
* Remove TlsConfiguration duplicate, move original to commons
* Make assertion more readable
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Fix build errors introduced in previous commit
- exclude time service tests from reference-v2
- add copyright header to TlsConfiguration
* Complete divulgence test and address review comments
Restores comments from original divulgence tests and strives to be as
close to them as possible.
- Address https://github.com/digital-asset/daml/pull/2282#discussion_r311429169
- Address https://github.com/digital-asset/daml/pull/2282#discussion_r311431156
- Address https://github.com/digital-asset/daml/pull/2282#discussion_r311431913
* Delete old divulgence tests
* Restore methods to fetch contracts by template identifier
* Run conformance tests on in-memory sandbox
* Run conformance tests on the persistent sandbox
* Remove workflow identifier workaround
* New contract keys test (#2452)
* Set channel type for LedgerSession
* Add ContractKeysIT
* Run tests against reference-server v2
* Removed the old ContractKeysIT. However, ...
... I cannot remove ContractKeysChecks completely, because we have
another test path for DARs created prior to DAML-LF 1.6, which doesn't
have the submitter == maintainer restriction for contract keys.
* Use codegen instead of custom code
* Add missing copyright headers
* Drop sandbox conformance test suite (seems to not work on Windows CI)
* Make time simpler
* Test tool tests to run only semantic tests
* Make CI behave more consistently
* Add missing files to run tests on Postrgres
* Fix flaky divulgence test, introduce readable identifiers
* Increase sandbox conformance tests timeout
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r312363246
* Reintroduce test resource extraction
* Read semantic tests from the classpath rather than Bazel
* Use codegen types for ContractKeys integration test
allocateParty returns Primitive.Party and create and exercise also
accept Primitive.Party. This way we don't have to keep a String and
Primitive.Party representation around.
* Bump timeout for sandbox conformance tests
* Remove dependency on Bazel runfiles
It looks like that closing the LedgerApiServer before closing the
AkkaExecutionSequencerPool (AESP) leads to a non-deterministic bunch of dead
letter warnings when stopping sandbox.
Closing the AESP first avoids these warnings.
Fixes#1886
This happened because we translate empty workflow ids to None in the
API-to-domain translation layer. When we store the workflow id, we actually
convert the None case back to an empty string (and not a null value).
When loading the workflow id from the database, we parse the column as a regular
LedgerString, but unfortunately these cannot be empty. Therefore using empty workflow
ids on postgres has been broken since workflow id has changed from a tagged string
to LedgerString.
* kvutils refactoring
Move transaction submission processing to its own class and refactor to
stop early on errors.
Throw a known error rather than use sys.error.
* Refactor package and party allocation
* Introduce "Commit" computation abstraction
This also fixes an issue where the command dedup entry was not created
when a later validation failed and a rejection log entry was created.
* Drop input_log_entries. Add contract instance to DamlContractState.
* Add copyright headers
* Add KeyValueCommitting.submissionOutputs
This cleans up implementations that need to know the outputs of a submission
up-front.
* Avoid double logging for API Services
* Make WitnessesIT a word spec with a "title"
* The test reporter should also print the "should" keyword.
* Enable ALL.THE.TESTS. for reference-server v2
* Read stakeholders through the active contract service
Fixes#2070
Also standardizes the use of American English spelling for `serialize`
and `deserialize`. Standardize. Serialize. Americanize.
* Remove Haskell bindings test workaround
* Address https://github.com/digital-asset/daml/pull/2316#discussion_r308193661
* Cleanup
* WIP
* first integration test + fixture
* minor cleanup
* Implementing ContractService.lookup
* Reverting back to endpoints.all (all2 did not work)
* Cleanup
* replace ApiValue ADT with aliases to daml-lf/transaction Value ADT
* porting rest of navigator to LF Value ADT
* Command Service WIP
* CommandService WIP
* porting more of navigator to LF Value ADT
* last error, not first
* rename ApiValueImplicits file
* special conversion features for ImmArray and FrontStack
- just .to[ImmArray] or .to[FrontStack] any random collection
* finish porting most of navigator main code
* use numeric indices for record field name fallback when pretty-printing
* tuples are not serializable
* use numeric indices for label fallback in JSON verbose encoding
* make traverseEitherStrictly more likely to preserve the seq's class
* to shortcut for ImmArraySeq .to[ImmArraySeq]
* compiling, passing navigator backend tests
* test traverseEitherStrictly more, er, strictly
* pass scalacopts through to scaladoc
* deal with unused warning
* remove unneeded function
* simpler error reporting, more private functions in ApiCodecCompressed
* move slowApply to FrontStack, test it so it actually works
* remove unneeded toStrings; better error from impossible ValueTuple case
* scalafmt FrontStackSpec
* support alternative, label-free record JSON encoding
* Adding domain.CreateCommand + corresponding json formats and dummy json format for lav1.value.Record
* CommandService.create should be done... need to test it
* TODO added
* Cleanup
* move ApiCodecCompressed, ApiValueImplicits, and some aliases to new lf-value-json package
* Using tagged TemplateId type instead of Identifier + exercise command WIP
* adapt navigator to moved pieces
* start defining scalacheck extension to ApiCodecCompressedSpec
* CommandService.exercise + introducing CommandMeta
* Adding command endpoints, can't test them yet, need lf value json formats
* fuse some list operations
- suggested by @stefanobaghino-da; thanks
* blue error message
* Minor fixes after merging librify-navigator-json-compressed, #2136
* experiment with an inductive case in TypedValueGenerators
* finish a List case for TypedValueGenerators; it's revealing
* Introducing API value to LF value converter,
CommandsValidator takes IdentifierResolverLike instead of IdentifierResolver
* cleanup
* remove accidentally readded duplicate aliases
* start tying knots in TypedValueGenerators
* verbatim copy ApiCodecCompressedSpec to lf-value-json
* shift some tests from navigator to lf-value-json
* test Optional and Map for ApiCodecCompressed
* heavier random testing of ApiCodecCompressed
* remove unused dependencies from lf-value-json
* adding value json writer
* cleanup
* Revert "cleanup"
This reverts commit 2e4d153f
* fixing the build
* cleanup
* cleaning up imports
* JsValue to API value is done, needs a test
* cleanup
* use scalac -Ypartial-unification in http-json
* simplify some Traverse instances
* factor CreateCommand and ExerciseCommand traverse instances
* Command create integration test WIP
* Command create integration test WIP, got rid of the JsonReader and JsonWriter for the values, converting values explicitly
* Extracting DomainJsonDecoder and DomainJsonEncoder
* LfV refactoring
* Create command serialize/deserialize test works
* cleanup
* resolving conflicts
* More json encode/decode tests
* logging
* command/create passes integration test now
* Adding readme
* grammar
* TODO added
* GetActiveContractsResponse encoding
* ideintifier conversion renaming
* PackageService resolveTemplateId returns domain.TemplateId now
* Resolving LF Identifier instead of Template ID, this should also work for Exercise command decoding
* cleaning up a bit
* daml-lf: show type in TypedValueGenerators-driven errors
* exercise command json encoding/decoding works
* command/exercise IOU_Transfer integration test passes now
* avoid filter for Gens; makes many contract ID gens not fail
* test ApiCodecCompressed against 100 random types, 20 random values each
* Updating README instructions
* improving error handling, failed futures, get logged and reported to the user now as 500
* [ROUTING DSL] Removing routing DSL, it did not work
* getting rid of HttpEntity.Strict match + cleanup
* fixing the merge conflict
* updating README
* use Show.shows instead of new Show
* List(_) isn't checked, but Seq(_) is slightly safer
* improving test assertions
* Adding /contracts/lookup implementation
* http-json: use ImmArraySeq instead of List; use toRightDisjuction
* http-json: .toList.toSet is shorter than fold
* http-json: replace .leftMap.map with .bimap
* http-json: use subst instead of reimplementing JsonFormat
* http-json: remove unused ExceptionHandler
* http-json: safer == comparison
* Adding two test cases for expected errors
* Adding BazelRunfiles.rlocation magic that supposed to handle windows path for bazel dependencies
* http-json: import, not extend
* Support for contract keys to kvutils
* Fix handling of CreateEvents with contract keys in semantic tester
* Add ContractKeysIT to tests and fix issues related to fetchByKey
* Address code review
This is a breaking change on the ledger api. So far we could rely on the
assumption that contractId == eventId. This assumption doesn't hold
anymore in a daml-on-x setting, where the eventId is created by the
api server, but the absolute contractId is created by the ledger
implementation.
Instead of going through weird contortions to store the relevant data in
the existing database schema, it is more viable in the long term to
remove that field and instead provide facilities to lookup transactions
by contractId.
Contributes to #2068.
* ledger: refactor validateValue from CommandsValidator to new ValueValidator
* daml-lf: add a matchable class for leaf Values except ValueContractId
* ledger: use IdentifierResolverLike for CommandsValidator
* daml-lf: add Traverse[FrontStack]
* navigator: define readRecordArgument and readArgument in terms of ledger-api-common ValueValidator
* navigator: clean up fillInVariantTI
* navigator: replace ledger-api Value writer with LfEngineToApi from ledger-api-common
* navigator: use traverse
* navigator: remove dead sequenceMap
* navigator: adapt to new ValueValidator structure
* ledger: dealias validateValue call
* navigator: a good use case for traverseEitherStrictly
* Add CommandSubmissionTtlIT to Ledger API Test Tool
* Add ActiveContractsServiceIT to Ledger API Test Tool
* Add WitnessesIT to Ledger API Test Tool
* Delete useless test
* Move time manipulation in TransactionServiceHelpers
* Add CommandServiceIT to Ledger API Test Tool
* Add the missing summary case to the ToolReporter
* replace ApiValue ADT with aliases to daml-lf/transaction Value ADT
* porting rest of navigator to LF Value ADT
* porting more of navigator to LF Value ADT
* last error, not first
* rename ApiValueImplicits file
* special conversion features for ImmArray and FrontStack
- just .to[ImmArray] or .to[FrontStack] any random collection
* finish porting most of navigator main code
* use numeric indices for record field name fallback when pretty-printing
* tuples are not serializable
* use numeric indices for label fallback in JSON verbose encoding
* make traverseEitherStrictly more likely to preserve the seq's class
* to shortcut for ImmArraySeq .to[ImmArraySeq]
* compiling, passing navigator backend tests
* test traverseEitherStrictly more, er, strictly
* pass scalacopts through to scaladoc
* deal with unused warning
* remove unneeded function
* simpler error reporting, more private functions in ApiCodecCompressed
* move slowApply to FrontStack, test it so it actually works
* remove unneeded toStrings; better error from impossible ValueTuple case
* scalafmt FrontStackSpec
* support alternative, label-free record JSON encoding
* fuse some list operations
- suggested by @stefanobaghino-da; thanks
* blue error message
* keeping loaded packages after reset
removing unnecessary extra reference
changelog
making InMemoryPackageStore immutable
* using Instant.now when loading packages in IndexServer
* loading Dars from ZipInputStreams
formatting and cleanup
fixing DarReaderTest not compiling
fmt
using IOUtils to read the ZipInputStream
fixing CodeGen
fixing broken legacy DAR loading
fixed v2 ReferenceServer
cleanup
* fixing rebase conflicts
* introduced a fromFile method to avoid having to do boiler-plating where the input is a File
* addressing leaking resources
* cleanup
* inlining closures
* closing the darStream
* make every PublicPackageUploaded update contain exactly one package
* implement update subscriptions that start with offsets with the same major number
* address review comments
* Do not convert the transaction id to Long in LedgerBackedIndexService
The InMemoryLedger does this, because it can make certain assumptions
(like transactionId == offset), but the sql based ledger cannot do that,
because the transactionId may be produced by another backend and thus
have a different format.
* Move tests that use contract keys into their own integration test file
This allows us to run the other tests via the ledger api test tool with
the postgres indexer while InMemoryKVParticipantState doesn't support
contract keys yet.
* Remove a few occurrences of LedgerBegin from TransactionServiceIT.
* Rename CommandTransactionChecksNo1866IT to ContractKeysNo1866IT
* Enable TransactionServiceTests,DivulgenceIT,TransactionBackpressureIT for postgres index|er
* format all the things
* Address a few review comments.
See https://github.com/digital-asset/daml/pull/2032#pullrequestreview-258771927
* Extract reusable base class for ContractKeys tests
This is needed because Bazel's scala_test_suite isolates the tests from
another, and one test cannot extend another one.
* Classify the reference-v2 conformance tests with a long timeout
* DamlOnXSubmissionService shut down of package load executor service
* ReferenceIndexServer specifies viable PackageId
Using dash (-) instead of slash (/)
* connenct LedgerAPI party and package management with KVUtils
* formatting
* address review comments
* add participantId parameter to sandbox and indexer
* annotate TODO's with github issue numbers
* check that submitter is in maintainers when looking up keys
Fixes#1866. Note that this limitation applies both for `lookupByKey`
and `fetchByKey` -- anything involving retrieving a key is affected.
* add UNTIL-LF to run tests up to a certain version of DAML-LF
* name targets for DAML tests better
* add notes about DAML-LF changes
* commit Test.daml with DAML-LF 1.5 rather than compiling it on the fly
* add scenario tests for #1866
* add warnings about future key behavior in docs
* use flag rather than version when executing
1) Change to SuiteResourceManagementAroundAll, transforming the test to
not assume a new/empty ledger for each test case
2) Use distinct workflow ids so the individual test cases don't clash
with each other
3) Use a new UUID for each randomized command id, instead of using the
uuid used per test run.
Semantic tests will be skipped if the specified Ledger API endpoint does not
implement TimeService.
This teaches MultiLedgerFixture to deal with cancelled tests.
This improves Ledger API Test Tool reporter to deal with cancellations better.
* Add external offset column to parameters
We only ever need to use the external offset when restarting the api
server from a persistent data store to resume the ReadService state
update stream from the last processed offset.
In the PostgresIndexer we then need to store the external offset while
adding a new ledger entry.
* Allow disabling the jmx reporter in metricsmanager
* Implement PostgresIndexer component
* Introduce read-only Ledger and Dao
* SandboxEventIdFormatter treats transactionId as opaque string
* Introduce LedgerBackedIndexService
This can be reused both in the Sandbox and the ApiServer
* Implement PostgresIndex component
* Reference Service based on kvutils-v2 and postgres(index|indexer)
* Disable conformance test for ReferenceServer
Adds a PostgreSQL package store to the sandbox.
Previously, packages were only stored in memory and
specified as CLI arguments. So if a user uploaded a
package at run time, he needed to add the package to
the CLI when restarting the sandbox.
* adapting docs to match state as is regarding not supporting dalf files
* patching release notes instead
* validating for ZIP files in Cli parser
* guarding for parsing non zipped archives
* Update release-notes.rst
* Update unreleased.rst
Co-Authored-By: Beth Aitman <bethaitman@users.noreply.github.com>
* Update unreleased.rst
* Update release-notes.rst
* Update release-notes.rst
* provide asynchronous interfaces for party and package management on the participant-state
* provide submission id for correlating req with resp on Write- and ReadService
* add asynchronous party allocation in KVUtils
* start conversion of the package upload and party allocation to synchronous paradigm
* implement synchronous versions of party allocamtion and packacge update
* document recent changes and unify concepts between v1 and v2 versions of
the participant-state interface
* scala formatting
* address review comments
* Ledger API Test Tool: Add --stable-identifiers option.
* Ledger API Test Tool: split stable party and command identifiers options.
This allows to control individually which identifiers are stable and which are
randomized.
* Ledger API tests: introduce TestIdsGenerator.
It combines all places where per-run unification of identifiers was generated.
* Ledger API Tests: use unification methods instead of run*Suffix
* Review fixes
