* provide asynchronous interfaces for party and package management on the participant-state
* provide submission id for correlating req with resp on Write- and ReadService
* add asynchronous party allocation in KVUtils
* start conversion of the package upload and party allocation to synchronous paradigm
* implement synchronous versions of party allocamtion and packacge update
* document recent changes and unify concepts between v1 and v2 versions of
the participant-state interface
* scala formatting
* address review comments
* Ledger API Test Tool: Add --stable-identifiers option.
* Ledger API Test Tool: split stable party and command identifiers options.
This allows to control individually which identifiers are stable and which are
randomized.
* Ledger API tests: introduce TestIdsGenerator.
It combines all places where per-run unification of identifiers was generated.
* Ledger API Tests: use unification methods instead of run*Suffix
* Review fixes
Moves party allocation to its own WriteService trait (similar to package upload).
Changes uploadDar to uploadPackages, so that the write service does not depend on the DAR file format.
Adds or improves many comments.
* Introduce contract key to exercise node
Track the contract key in NodeExercises. This is in
preparation for adding the key into exercise events, and
for support for contract keys in kvutils, where we need
to be able to compute the outputs of the submission, e.g.
the contract key that would be unset by a consuming exercise.
This introduces transaction version 8.
* Move release note on contract keys in exercises to unreleased.rst
* Address Stephen's review
- Drop changelog entry for this change as it is not user facing
- Add info on version 8 to transaction.rst. Fix typo in version 7.
- Reorder VersionTimeline as per review
- Only compare the NodeExercise 'key' if it is set in the original
transaction (in order to compare transactions in a backwards
compatible manner)
* Unbork unreleased.rst
* Run integration tests for sandbox through the IT test suite
We currently don't run sandbox on postgres via ledger-api-test-tool. We
need to first figure out how to best do that before we can switch
sandbox to the test tool again.
* CommandClient#submitSingleCommand also uses commandUpdater
* The test reporter for the test tool now handles ignored tests
* More optional tests for ledger-api-test-tool
- TransactionBackpressureIT
- DivulgenceIT
- CommandTransactionChecksHighLevelIT
- CommandTransactionChecksLowLevelIT
Contributes to #1374.
* Add missing domain.RejectionReason cases
* Use java.time.Instant instead of damlf Timestamp
* Make applicationId, submitterParty, commandId optional in LedgerEntry.Transaction
* store archive size in `DarReader`
* rename `SandboxTemplateStore` to `SandboxPackageStore`
* store package info in `SandboxPackageStore`
* introduce package upload / read to write / index services
not tested yet, just a ton of plumbing
* WIP test the package service
* Fix build errors after rebase
* Move packages service to v2
* Ledger API client uses ledger API types
* Fix ReflectionIT
* Correctly handle uploading invalid dar files
* Fix reading DAR entry file sizes
* Improve package management IT
* Improve handling of duplicate packages
* Fix language-support build
* Use unique party and command names
* Rename lfpackage to language
* Rename SandboxPackageStore to InMemoryPackageStore
* Remove getCurrentTime ledger method
* Improve package management IT
* Move InMemoryActiveContracts and InMemoryPackageStore
* Use case object for UploadDarResult.Ok
* Address review comments
* Update release notes
Fixes#1311
* Use BazelRunfiles in test
This fixes two issues with the sandbox database migration:
* A typo in the migration script
* Error deserializing transactions due to protobuf recursion limits
Fixes#1682
* Fix various issues with the reference daml-on-x implementation
- Add authorization check to submission to catch authorization errors early
- Fix filtering by templates in transaction service
- Loosen the transaction_id validation to accept transaction_ids as used by
the reference implementation
- Fix handling of ledger end
- Add caching to archive decoding
- Enable TransactionIT for the reference implementation
- Fix time service event dedup
* fix transaction validation which started reporting different error type
* fix transaction fetching tests on sandbox that assumed that transaction ids would be verified as numbers by the api validation layer
* return NOT_FOUND in all places where a non-existing transaction id or event id is used to fetch transactions
Scale factor for time-to-live of commands sent for ledger processing
(captured as Maximum Record Time in submitted transactions) for
"SemanticTests" suite. Useful to tune Maximum Record Time depending on
the environment and the Ledger implementation under test. Defaults to
1.0. Use numbers higher than 1.0 to make timeouts more lax, use
numbers lower than 1.0 to make timeouts more strict.
Fixes#1312
This PR adds support for party management in the sandbox:
Both the in-memory and the SQL backend track a list of known
parties, and implicitly add any party mentioned in a transaction.
New calls were added to the IndexService and the WriteService
for managing parties. These calls are wired to the above mentioned
persistence backends, and to a new API service.
A corresponding client interface was added to the scala API client.
An integration test was added for checking that a call to allocate a
party succeeds.
An integration test for the sandbox was added for checking that the
sandbox implicitly adds all parties mentioned in a transaction.
Extra command line arguments to the client_server_test target are passed
to the ledger-api-test-tool instead of the arguments defined in the
bazel target.
For example:
`bazel run //ledger/sandbox:conformance-test-static-time --list` will
pass the --list parameter to the ledger-api-test-tool (to print all
available tests), instead of actually running the tests in static-time mode.
The Ledger API Test Tool takes the flag --all-tests to run all defaul
and optional tests
The tool also allows multiple occurrences of the --include and --exclude
flags.
Also removes StandaloneSemanticTestRunner.
Fixes#1371
* Provide initial implementation of package management in the KVUtils
* rename updateArchives to updatePublicPackages and add them to the abstract interface
* remove participant-state:reference app from published artifacts
* adopt the changes to the recent version split
Add tTansactionServiceIT into Ledger API Test Tool as an optional test.
Available tests can be listed with the --list option.
Tests can be included with --include, and excluded with --exclude.
Fixes#1372, #1472
* added recovery logic when ledger entry persistence fails
* adding release notes
* logging only when the queues are closed due to an error
* using error instead of warn
* Drop support for DAML-LF 1.3 from damlc
This means `damlc` now only supports versions allowing arbitrary expressions
as contract keys. This is very helpful for the upcoming new template
de/re-sugaring underpinning generic templates.
* Unpin a few DAML-LF target versions in tests
* Fix golden tests
* Adapt changelog to rebase
* Rename DAML-LF scenario tests to 1.dev
* Update tests to --target 1.dev
* Fix release notes
These were mistakenly disabled along with the move to tests-as-a-library
construct. The 'scala_test_suite' macro does discovery only over sources, not
dependencies.
Because the record time was mapped to the ledger effective time field
when deserializing contracts from the SQL database, a subsequent
comparison between the LETs of two nodes in the same transaction caused
the transaction to be rejected.
Fixes#1435.
* SandboxLedgerBackend implements CompletionsService from IndexService
* things compile
* cleanup
* cleanup
* commandId is not optional anymore
* proper filtering of completions
* Add RemoteApiProxy fixture type.
This is in preparation for using Sandbox IT suite as part of the Ledger API Test
Tool.
* ledger-api-test-tool: Drop reset functionality.
This is no longer necessary for the tool and it does not scale with the types of
tests in the suite.
* integration-tests: Fail if the server under fixture is stuck.
This makes sure that a server getting stuck will get detected by a test, instead
of ignoring it and potentially allowing the server to linger.
* integration-test: Make semantic testing runs independent.
It manges parties and command identifier to include a unique (random) suffix in
all ledger-commited identifiers. This allows the test to run against a Ledger
API without reseting it.
* ledger-api-test-tool: Unify test code using scenario runner with IT suite.
This reuses the scenario runner test code from the IT suite, instead of
reimplementing it. This should be a no-op (except for tests reports formatting).
* Review fixes.
* Ledger API Test Tool: Provide logback config.
This quites Ledger API Test Tool output.
* Make sure akka threads are terminated at end of test runs.
This makrs Akka threads to be daemons, hence forcing them to be closed at the
end of Ledger Api Test Tool.
* Use Ledger API Test Tool in tests of reference server.
* Add Apache commons-lang3.
* Ledger API Test Tool: Implement custom test reporter.
This addresses two needs:
- avoid using buggy scalatest test reporter;
- pretty-prints test results prettier.
* dade-copyright-headers: return success on successful reformatting.
* Isolate daml-on-x prototype
* Implement TransactionService of index api.
Towards the first milestone of the daml-on-x integration steps
* fix compiler error after last-minute refactoring
* formatting
* remove unnecessary dependency from api-server-damlonx
* Fix some inconsistencies.
This prevents binaries from including random logback configuration via library
dependencies.
Relevant binaries were found with:
bazel query --order_output=no --universe_scope='//...:*' \
'kind(".*(test|binary).*", allrdeps(//ledger/ledger-api-integration-tests:ledger-api-integration-tests-lib))'
This is a preparation for making the actors in DAML-LF's exercise
instruction optional (see #1347).
This is technically a breaking change but since `exerciseExplicit` has never
been exposed via the documentation, we don't add a release note.
* Rename Value's ContractId to VContractId
* daml-lf: a bit more about PartyId
* daml-lf: Concatenable MatchingStringModule
* daml-lf make clear type used for Scenarios only
* daml-lf create ContractId, LedgerId, TransactionId
* sandbox-sql conversion util
* LedgerName -> LedgerString
* futher type cleanup in the sandbox
* daml-lf add test for LedgerString
* fixing tests
* a bit more safety in the DB
* Address Stephen's comments
* fix rebase
* More fixes for StringModule
* change length of LedgerString (256 -> 255)
* Active Contacts Service from IndexService is wired through
* almost all tests pass
* fixing the offset toString issue
* removing irrelevant .await in RemoteServerResource
The index service serves as a "bridge" between the data on the ledger
and the ledger api users. That is also the reason why the index service
API is defined in terms of the types in ledger-api-domain. It is the
responsibility of the index service implementations to return compatible
types and adhere to the expected semantics, so we can actually reuse
index service implementations in the ledger api servers.
The foldWithPathState is a fold over the transaction that allows
keeping extra state along the path of the traversal.
We add computePerPartyProjectionRoots in preparation for the
work on privacy-aware kvutils, where transaction nodes are stored
separately rather than in one blob. This function allows us to compute
the projected transaction roots for each party mentioned in the
transaction. We require this information to establish the order
of the root nodes when the size and form of the transaction is
blinded by assigning random node ids.
* using IdentityService
* package service wired through
* using ledger id from identityService
* extracting out ApiServer trait
* making ApiServer creation async
* using async tolerance when waiting on ApiServer creation
Fixes#1250
The previous behavior when receiving a transaction filter without any
party was to reply with an empty stream. Since, given the current
situation, no data could ever be served for such request, it represents
a better feedback for the user to reject such requests as carrying an
`INVALID_ARGUMENT`.
* extracting some sub-interfaces
* dump
* dump
* things compile
* extracting more concerns
* comment change
* making configuration service return a stream
* adding documentation around the new interfaces
* Added agreement_text field to the CreatedEvent in Ledger API.
* Changed java bindings + java codegen
* Changed utilities for scala codegen
* Made necessary changes in Sandbox to propagate the agreement text from ContractInst to the CreatedEvent
* Made changes to the navigator to show the agreement text in the contract details page when it is set and not empty
Fixes#1110
* Add tracking of disclosure to contract state
And clean up some of the error handling in processSubmission.
* Do not add, but rather union the divulged parties
* no var no problem
further refactor
introduced InfraState
a bit less vars
encapsulating closes
SandboxServer starts automatically
rebase fixup
collecting state into a single object
some cleanup
removing exposed materializer
LedgerBackend is closed in SandboxServer
changed ownership of Ledger
fixing perf tests
fixing some compile errors
formatting
removing unused method
fixing integration test to use correct dar file
fixing issue with PostgresFixture and SandboxResource
Fix integration tests on Windows
* fixing rebase artifacts
* Store divulgence info for contracts
* Store divulgence info for contracts
* some documenting refactors
* Use fail instead of sys.error in test
* Add release note
* Rename migration files
* Fix typo
* Fix updating divulgence info in scenarios
The ACS was comparing differently formatted contract IDs.
Fixes#1166
* ledger-api-intgration-tests: Cleanup ledger ID handling.
This changes nothing in the logic of tests. It is a preparation for moving of
most of the tests to use Dynamic ledger ID mode for tests, which is a
preparation for running these tests against remote Ledger API endpoints.
* sandbox: Rework LedgerIdMode and move it up into common.
* daml-lf: make Decimal type safe
* daml-lf: create Utf8String type
* daml-lf: cleanup in data package
* Address Stephen Comments
* daml-lf: remove UTF8String
* enable Return wartremover wart
* remove return keyword from various places in daml-lf
* remove return keyword from various places in ledger
* simpler ImmArray equals
* move traverseEitherStrictly to point of use
* remove return from ledger-api-server-example
* Remove ExercisedEvent in Event oneof.
The Event message is only used in the Transaction message. Flat
transactions do not contain exercised events, but only created and
archived events. Therefore we can remove the ExercisedEvent from the
Event oneof, without breaking transport compatibility.
HOWEVER: The Java Bindings used to use the data.Event class for both flat
transactions and transaction trees. To properly represent the actual
event types in the two transaction structures,
1) Event is now and interface and is only used in the Transaction class.
2) there is a new interface TreeEvent, which is used in the TransactionTree class.
* CreatedEvent implements Event and TreeEvent
* ExercisedEvent implements TreeEvent
* ArchivedEvent implements Event
Some "pathological" cases where an occurrence of an exercised event
would have resulted only in an exception, are now removed (see change in
LedgerApiV1.scala).
Fixes#960.
* adding necessary @volatiles
* services are factored out
* no need for ResetService to stop the services before
* removing not needed callback methods
* mostly renaming things
* some further cleanup
* avoiding linear searching for transactions from genesis
* validating transaction ids
* one more failing test nailed down
* fixing off by 1 error
* docs + release notes
* fixing EventIdFormatterSpec
* fixing broken validation test
Up to now, the engine blindly assumed that contract ids pointed to
contracts of the right type. However, this assumption is faulty:
contract ids coming from the Ledger API cannot be type checked
in command translation since we need access to the contract itself
to do so.
This caused some seriously surprising / broken behavior: one could
send an exercise command with the wrong template id and still go
through, or break internal invariants about the type of choices.
This commit fixes this by checking that the type of the contract
instances we fetch is correct at runtime.
cc @hurryabit @dajmaki @remyhaemmerle-da @S11001001 @meiersi-da
* Remove ledger-api-server-example
This code does not serve any purpose anymore as it has been replaced
by ledger/participant-state and friends.
* Remove ledger-api-server-example from Windows build
* Use Sandbox's ActiveContracts in reference index service
This properly handles contract key updates and adds the required
checks for contract activeness and visibility.
Could consider moving the ActiveContracts.scala into another library,
but since we're soon starting the work to bring sandbox onto the
daml-on-x stack we might postpone that.
* reformat
* Exclude logback.xml from sandbox the library
This allows users of sandbox as a library to specify their
own logging configuration.
* Re-order fields in records if all labels are provided
Fixes#988
* Add line to release notes
* Fix line on release notes
* Document change on the Ledger API definition
* Simplify pattern matching
* Improve and fix tests
Integration tests now look for the "Missing field label" rather then the
"Mismatching record label" when inspecting errors. Furthermore, a test
has been added that ensures that repetitions in labels still raise an
error.
* Add comment
Comment about relationship between fields length check and fields reordering based on labels
Co-Authored-By: stefanobaghino-da <43749967+stefanobaghino-da@users.noreply.github.com>
* Improve Ledger API docstring
Explicitly state the semantics of labeled fields (i.e. repeated keys are not accepted)
Co-Authored-By: stefanobaghino-da <43749967+stefanobaghino-da@users.noreply.github.com>
* Export daml_kvutils_java_proto and add (un)packDamlLogEntryId
Export needed when using kvutils via bazel from another workspace.
* Use 'String' for participant state's LedgerId
Ref.PackageId is the wrong thing, and likely MatchingStringModule
isn't very usable from Java.
* Drop LedgerTestingHelpers.asyncFromTimeService as unused.
* ledger-api-test-tool: Add clues to asserts in the main code paths.
This ensures that users get more informative errors for the currently used
semantic tests.
* semantic tester: add operation description to an underlying ledger.
This allows to provide richer error messages to users of LedgerApiTestTool.
* Extend test durations on CI for Ledger API Test Tool driven test.
This introduces a command-line argument to scale timeouts used in the test.
* ledger-api-its: Make FutureTimeouts.timeout duration scaled inside.
Also include more information in the error message.
* simple test to hedge against existing sql migration files
* copyright
* adding better failure message
* adding intructions on generating the digest file
* Add configuration to LedgerInitialConditions
* Wait until index state is established in reference index service
This fixes a race condition where the daml-on-x server has been
brought up before the index service can serve it.
* Remove ledgerId checks from index service
Now that we have decided on non-changing ledger identifiers, remove
the ledgerId checks from index service and simplify the interface.
* Reformat and fix leftover ??? from DamlOnXTransactionService
This reverts commit 3d8acde916.
For some reason that commit seems to have resulted in a lot of
"unexpected end of file" errors during cache downloads. I do not know
what is going on here or how to fix it so let’s revert it for now.
Submitting a command via the CommandService now returns either the
transaction id (SubmitAndWaitForTransactionId), the flat transaction
(SubmitAndWaitForTransactionResponse), or the transaction tree
(SubmitAndWaitForTransactionTreeResponse).
This means that users don't have to wade through the transaction stream
to retrieve the resulting transaction. This is particularly useful in
combination with #479.
Fixes#406
* Allow to mangle names used in scenarios before their executions.
This allows to run a scenario against a long-running server repeatedly and avoid
clashes between runs, since each party is unique (up to the randomness used).
* ScenarioRunner: add test for partyNameMangler
This change is needed in preparation of #406, where we want to return a
transaction tree and flat transaction after a SubmitAndWaitForTransaction(Tree).
* remove outdated flags from comments
* update tests in preparation for visibility rules
* check visibility of contract keys, fixes#751
* add release notes
* include visibility check in scenario runner
* stephen's suggestions
* fix type error in api server example
* scalafmt
* bring TransactionFiltration.scala up to speed with disclosure
see
<faf1cde1a1/ledger-api/grpc-definitions/com/digitalasset/ledger/api/v1/event.proto>
for an in-depth description. fixes#794.
* add tests for contract key visibility
* move a lot of ledger testing utils out of command checks
* move a lot of ledger testing utils out of command checks
* remove useless checks regarding divulgence
* add tests regarding event witnesses
* update release notes
* re-introduce safety when disclosing
* hedging against possibly gaps in offsets
* doing the math right
* adding some notes
* extracting pipeline logic
Please enter the commit message for your changes. Lines starting
* extracting "run until" logic to be testable
* testing _run until_ logic
* adding release note
* copyright
* adding link to PR
* changing lookUpByTransactionId to use TransactionPipeline
* fixing issue identified by Extractor tests
* reference-server: Add invariant that observed offset is monotonic.
* reference-server: Fix offset handling for empty resulting transaction streams.
If the requested transaction stream (by range) returns empty, code - previously
- would reset 'currentOffset' to None, which was breaking invariants for the
next transaction stream request. Ensure that 'currentOffset' is maintained in
such scenario.
- Make `getLedgerInitialConditions` a `Source` to mirror `stateUpdates`.
This makes it easier to implement in Java.
- Expose transaction submission outputs with `KeyValueSubmission.transactionOutputs`
for ledger implementations that require knowledge of outputs up-front.
- Fix typo in unpackDamLLogEntry.
* release: make 'ci/release.sh' runnable for dry runs.
release-dry-run.sh is outdated and duplicates logic from ci/release.sh, so it
got deleted.
* ledger-api-test-tool: release the tool together with the SDK components.
* ledger-api-test-tool: update docs to reflect distribution mechanism.
* ledger-api-test-tool: further docs refinements.
* Add Ledger API Test Tool mention into release notes.
* Work in progress for the key-value participant-state utilities
* WIP on kvutils reference
* Try out "InMemoryKVParticipantState" in the reference server
* Work in progress on the rewrite based on code review
Main idea here is that there's functions to produce
submissions, and function to process a submission and
produce the log entries, and a function to convert
a log entry into 'Update'.
* WIP
* Add documented traits to clean up the key-value utility interfaces
* reorganize and address code review
* Add tests for in-memory kvutils example plus fix few missing bits
* Documentation, logging and heartbeats
* Move heartbeat source init out of actor preStart
* Add DamlOnXTimeService. Package inputs. Fix entryId references.
Semantic tests pass now.
* copyright headers & reformat
* Fix input computation, improve docs in daml_kvutils.proto
The DamlContractState input entries were missing for fetches
and non-consuming exercises.
* Disable SemanticTests.timeTravel
This test only works against Sandbox with a static time model,
not against arbitrary daml-on-x implementations. We can re-enable
it once a test service shim has been implemented.
* docs: add structure for Ledger API Test Tool
* docs: ledger-api-test-tool: move docs to DAML docs.
* docs: dlik: add copyrights
* ledger-api-test-tool: improve docs
* ledger-api-test-tool: further docs refinements.
* ledger-api-test-tool: expose stacktraces of SemanticTester to the user.
Previously timeout errors would produce no useful output. This gives some -
albeit low quality - output to help pinpoing the issues. This needs improvement
for better UX.
* docs: dlik: summarize how to use the tool against a DAML ledger.
Add field 'usedPackages' to Transaction to record the packages
that were used during interpretation to produce said transaction.
This serves as a hint to specify what packages are required to
validate the transaction. Note that this may not be sufficient
for future implementations that e.g. compile DAML-LF to JVM
bytecode and require packages containing only type definitions
as inputs.
This hint is mainly required for ledger implementations for which the
packages are tracked similarly to contract instances, and which
require that all inputs to their transaction must be declared
beforehand.
This work is required for the participant-state key-value utilities
described in issue #410 and implemented in PR #637.
* removing major LF dev version from Haskell proto codecs
* removing major LF dev version from scenario service client
* missed import
* remove Scala support for dev major version; remove --allow-dev option from sandbox cli
* Version.minorFromCliOption function
* don't build daml-stdlib artifacts for dev major
* remove damlc CLI --target dev
* release note about removed dev major LF version
* governance now discusses minor dev, no more major dev
* don't build from daml_lf_dev.proto anymore
* remove daml_lf_dev.proto
* raise deprecated release
* reserve 9999 in the ArchivePayload sum, as suggested by @bitonic
* use reserved proto keyword, as suggested by @bitonic
- `reserved` cannot occur within `oneof` block
* remove --allow-dev test
* dev removal release note followed the previous release; move it back to HEAD
* GenTransaction.WithTxValue alias
* undo parts of parent dealing explicitly with serialization
* use GenNode.WithTxValue more often
* use Node*.WithTxValue more often
* introduced a SteppingMode abstraction for Dispatcher
* changing InMemoryLedger to use range queries
* SQL Ledger uses range queries
* documenting + fixing off by one semantics of Dispatcher
* adapting DispatcherIT
* added some docs
* reverting off-by-one offset semantics required on the ledger-api
* formatting
* introducing a case class ParsedEntry for sql result parsing
* testing Dispatcher with slow range queries as well
* adding missing ORDER BY ASC
* removing commented out code
* daml-on-x: Drop StateInit and use AtomicReference
- Drop the StateInit update message and introduce "getLedgerInitialConditions".
- Switch to AtomicReference from SyncVar as this is non-blocking and hence safer to use from Akka.
* ledger/participant-state: Rename recordTimeEpoch to initialRecordTime
and use the time from timeProvider at the ledger object initialization
time as the initial record time.
* fmt
* Fix use of repo as external workspace. Fix compilation issue in IndexState.scala
* Refix the use of this repo as external bazel workspace
* Fix compilation after rebase
* refmt
* seal engine.Error
* don't reinvent flatMap in CommandExecutorImpl
* add DivulgenceVisibility sandbox test case
- suggested by @oggy-; thanks
* add global implicit disclosure step to sandbox ACS transaction acceptance
* reorder ActiveContractsManager and include an implicitlyDisclose action
* add submitter argument to SubmissionHandle#lookupActiveContract
- the receiver must filter its response based on witnesses thereof
* propagate implicit disclosure in more places
* it's a bunch of functions
* proper error from the no-divulgence test
* rename ActiveContracts and ActiveContractsSteps to ...InMemory and ActiveContracts
* we need another table
* rename ActiveContract.witnesses to disclosedTo
* don't exclude other disclosures in ActiveContractsManager
* add local implicit disclosure when adding transaction to sandbox acs
* extra argument in ledger-api-server-example
* too late to blind?
* adapt to 10fcab8#339
* scalafmt
* non-total map get
* disable postgres test in SandboxSemanticTestsLfRunner until implicit disclosure added
* un-flatten witnesses and divulgences from "disclosedTo"; fixes DivulgenceIT
* comment on filtering of lookupActiveContract
- suggested by @francesco-da; thanks
* semantic errors from parent
* scenario to test divulgence success
* submit line 6 as owner; disable divulgence test under SQL
* comment what else to reenable
* Add release notes entry
* Add CreateAndExercise command to Java Bindings data layer
* Add CreateAndExercise command to DAMLe
The CreateAndExerciseCommand allows users to create a contract and
exercise a choice on it within the same transaction. Users can use this
method to implement "callable update functions" by creating a template
that calls the update function in a choice body.
Fixes#382.
* Add CreateAndExercise command handling to the sandbox
* Add CreateAndExercise command to the Ledger API
* ledger-api-test-tool: create with self-contained tests
Based on code by Tim Williams <tim.williams@digitalasset.com>
* ledger-api-test-tool: added an extract operation for resources
* api-server-damlonx: make ReferenceServer command-line-configurable.
This includes:
- port
- list of dars to load
- enable bad read service
* ledger-api-test-tool: Add test against Reference server.
* ledger-api-test-tool: allow all semantic tests.
* ledger-api-test-tool: move into //ledger/ledger-api-test-tool.
* ledger-api-test-tool: Add README
* ledger-api-test-tool: improve README according to the code review.
* ledger-api-test-tool: language and structure improvements to readme.
Based on review by bethaitman@digitalasset.com.
* extracting large command test from TransactionServiceIT
* make `preprocessCommands` more performant (see comment)
* fix duplicated command generation in `preprocessCommands`
* enable in-memory test-fixture as well
* Add semantic test for the reference server
Currently the semantic test is failing. Likely due to the
location annotation changes.
* Do not compare location annotations in isReplayedBy
The location annotations may not, and do not need to, match due to the
fact that the reconstructed update expression may not exactly match
the original one, and since the interpreter currently picks the closest
location annotation we cannot guarantee that they exactly match.
* scalafmt.
* buildifier.
* client_server_test: Increase timeout to 60s
Spawning a JVM-based server can easily take a long time on a very
loaded system (e.g. when running `bazel test //ledger/...` with enough
parallelism), so better have a high default timeout.
* ledger/api-server-damlonx: Address code review
* Fix client_server_test runner compilation. Bump timeout.
* Mark the reference server semantic test exclusive
* fmt reference/BUILD.bazel
* Explain failure of `fetch` in StdLib docs
* Explain `fetch` failure in DAML Reference docs
* bump integration tests timeout further, still seems to be flaky
* introduced back-pressure semantics for command submission
* fixup
* reenabling CommandServiceBackPressureIT
* fixing / enhancing the command submission backpressure test
* removed the unneccessary Error from the ADT
* routing CheckPoints/Heartbeats on a different channel to avoid back-pressure
* fixup
* remove early returns
* make reference index service stop repeating last event indefinitely
* make all services follow more consistent design, no unnecessary inherits, no unnecessary methods
* scala format
* HOTFIX: increasing 30.sec timeout to 300.sec
this to fix SQL-backed ledger tests that currently keep timing out.
It is NOT really a fix, but a temp work-around. Tried to get rid of
this timeout, but it propogates all over the place. Need a separate
story to address this
* HOTFIX: adding a link to the issue
Compared to older versions of the Engine, we do A LOT more validation
now. So when starting with a fresh engine after each reset (via
the ResetService), we also repeat the validation of the loaded packages
again and again. This is VERY expensive, especially for large DAML
packages.
Luckily, the specification of the ResetService states the following:
// Resets the ledger state. Note that loaded DARs won't be removed --
// this only rolls back the ledger to genesis.
This means that we can re-use the same Engine object and benefit from
not having "re-compile" packages via
ConcurrentCompiledPackages#addPackage, more specifically we can omit
this line: 1f2246c822/daml-lf/engine/src/main/scala/com/digitalasset/daml/lf/engine/ConcurrentCompiledPackages.scala (L56)Fixes#178
* fix various conversion functions from string to Decimal
Fixes#399.
This fixes a critical bug -- since:
* The DAML-LF spec specifies that the scale of `Decimal` is 10 --
that is, there are at most 10 digits past the decimal point:
<79bbf5c794/daml-lf/spec/value.rst (field-decimal)>.
* However, the code converting from the string representation that
we get on the wire was _not_ performing this check. This is due
to two reasons:
- `Decimal.check` is a function that checks whether a given
`Decimal` is within the upper and lower bounds of what the
DAML-LF spec specifies, but crucially it does _not_ check that
the scale is not exceeded:
<79bbf5c794/daml-lf/data/src/main/scala/com/digitalasset/daml/lf/data/Decimal.scala (L31)>.
This behavior is correct in some cases (more on that later),
but not in others. Crucially, `Decimal.fromString`, which was
supposed to check if a decimal string literal is valid, used
this function, which means that it accepted string literals
containing numbers out of the required scale, rounding them to
make them fit within the scale. This function has been renamed
to `Decimal.checkWithinBoundsAndRound`, and a new function
`Decimal.checkWithinBoundsAndWithinScale` has been added, which
fails if the number provided has data not within the scale.
`Decimal.fromString` now uses
`Decimal.checkWithinBoundsAndWithinScale`.
- `ApiToLfEngine.parseDecimal` assumed that `Decimal.fromString` _did_
fail when passed numbers that were in any way invalid, and
moreover did _not_ use the result of `Decimal.fromString`, but rather
re-parsed the string into an unconstrained `BigDecimal`:
<79bbf5c794/ledger/ledger-api-common/src/main/scala/com/digitalasset/platform/participant/util/ApiToLfEngine.scala (L96)>.
The reason for the code to work this way is that in the past
it was responsible for converting decimal strings both for the
current engine but also for an older version of the engine which
handled decimals of a different type. Both issues have been fixed.
* Therefore, `Decimal`s with scale exceeding the specified scale
made it into the engine, and contracts could be created containing
this invalid value.
* Once on the ledger, these bad numbers can be used to produce extremely
surprising results, due to how `Decimal` operations are
implemented. Generally speaking, all operations on `Decimal`
first compute the result and then run the output through
`Decimal.checkWithinBoundsAndRound`. The reason for this behavior
is that we specify multiplication and division as rounding their
output. Consider the case where the bad value 0.00000000005 made
it to the engine, and is then added to 100. The full-precision
result will be 100.00000000005, which after rounding becomes 100.
Therefore, on a ledger where such invalid values exist, it is not
the case that `y > 0 ==> x + y != x`, and so on.
Thanks a bunch to @briandbecker for the excellent bug report.
* fix failing test using to much precision
* Add buildifier targets.
The tool allows to check and format BUILD files in the repo.
To check if files are well formatted, run:
bazel run //:buildifier
To fix badly-formatted files run:
bazel run //:buildifier-fix
* Cleanup dade-copyright-headers formatting.
* Fix dade-copyright-headers on files with just the copyright.
* Run buildifier automatically on CI via 'fmt.sh'.
* Reformat all BUILD files with buildifier.
Excludes autogenerated Bazel files.
* TransactionServiceIT passes
* fixup
* using record time taken from TimeProvider
* taming TransactionBackPressureIT to avoid overloading of submission service
* implement AutoClosable on all Ledger related components having stateful resources
* reenabling InMemory fixture
* disabling contract key integration test for SandboxSQL fixture
* removing TODO
* bumping up timeout on TransactionServiceIT due to slow Azure pipeline
* 1 minute timeout for SqlLedgerSpec
* making jdbcurl CLI argument hidden
* updating release notes
* Add location information to DAML-LF produced by damlc
This is required to get error locations in the scenario view. Rigth now,
the location information for `create`/`exercise` still points to the
template/choice. I'll fix that in a separate PR.
* Fix test expectations
* Fix more tests
* Add participant-state and participant-state-index APIs and reference implementations
This adds the (still WIP) interfaces and the in-memory reference implementations
of participant-state and participant-state-index.
See issue #137.
* Pass ledgerId through to getLedgerEnd method
This is needed in the daml-on-x implementation of the
transaction service.
* Add api-server-damlonx and the reference server
This is the initial version of the ledger-api server built on top of
the participant state APIs.
While functionally complete (modulo test services), it has only yet
been tested with the semantic tester. The server and the participant
state APIs are still under active development.
See issue #137.
* Apply scalafmt and copyright headers to new damlonx code
* ledger/damlonx: Fix build errors after merge from master
* ledger/participant-state: separate out the reference impl
* ledger/participant-state-index: separate out reference impl
* ledger/damlonx: Refactoring
- Refactor participant-state into multiple modules
- Introduce structured offset and update id types (vector of ints)
- Properly parse ledger feature flags
* ledger: Disable failing semantic test. Decrease grouping duration.
* scalafmt, fix after merge
* Add missing copyright headers
* ledger/damlonx: Add Simon's comments
* ledger/damlonx: Remove UpdateId, use Offset in both state and state-index.
* ledger/damlonx: scalafmt
* damlonxserver: review participant-state interface
Includes adding issue links for postponed fixes.
* daml-on-x-server: review reference implementation
Fixme's added to https://github.com/digital-asset/daml/pull/348 and
extra issues creaed in https://github.com/digital-asset/daml/milestone/4
* ledger/damlonx: Use SimpleString
* ledgre/damlonx: Backtrack on SimpleString change
See issue #398
* daml-on-x-server: drop unused 'index.impl.reference.package.scala'
* daml-on-x-server: add note on potential transient contracts bug
* ledger/participant-state*: post-merge fixes
* Do not divulge contracts to observers in nonconsuming exercises
Disables support for non-default ledger feature flags, as they
are meaningless since ledger server logic does not respect the flags.
Instead of large refactoring to add support for the old flag settings,
it is best to disallow the deprecated flags, and later on phase out the
flags completely.
Re-enables test_divulgence_of_token in sandbox semantic tests.
Fixes#157.
* purge LedgerFlags entirely...
...since we only support one version of them anyway, and clearing them
* updated release notes
* SandboxSQL fixture + fixing various issues
WIP - playing around with Resources to support Sandbox Fixture with Postgres
Sandbox with Postgres fixture works - LedgerIdentityServiceGivenIT PASSES
adding guards for empty collections in PostgresLedgerDao
CommandCompletionServiceITi passed for both fixtures
fixing EventID formatting problem
another formatting fix
changed wrong autoCommit behaviour
CommandStaticTimeIT passed on both ledgers
rolling back transaction on duplicates
dealing with duplicate commands in postgres
fixing mismapped record_time
adding extra logging for fatal errors
disabling sql support as it's not finished yet
cleanup
removed unnecessary test classes
cleanup
formatting
* ADT was missing Product with Serializable
* fixed the compilation error
* Fetch status.proto from remote, simplify JS gRPC codegen
Fetch the `status.proto` file (part of the standard gRPC distribution)
from a distribution channel. _Moreover_, use the recently introduced
`proto_gen` rule to simplify how the gRPC code for the Node.js bindings
are generated (and remove the need to have `google/rpc/status.proto`
locally in the repository.
* Add plugin_runfiles option to proto_gen
This allows use to add additional files to the bazel sandbox so that
plugins can refer to them. This will subsequently be used by the
protoc-gen-doc plugin.
Also, pass the plugin options via --name_opt parameter.
* Add missing status.proto dependency /language-support/java and /ledger
* Build proto docs using the proto_gen rule
To make this work, I had to turn on the bazel build flag
`--protocopt=--include_source_info` because we cannot turn enable this
flag only for specific build rules.
* Make /ledger-api/grpc-definitions:docs public again
* Revert to the old style of passing plugin arguments to --name_out=options:path
* Suppress output of unzipping
* Fix link for google.rpc.Status in proto-docs
* Change contract key maintainers to have template argument in scope
More precisely, change the contract key type class we desugar the surface
syntax to such that the maintainers function has the template argument
in scope instead of the key. The compiler rewrites all references to the
template argument into references to the key and fails whenever that is
not possible.
* Add TemplateKey class to desugaring module
* Make template key check slightly more lenient
* Fix TODO descriptions
* draft of active contracts streaming - still WIP
dumping todos for Robert
Sandbox: fix getActiveContractSnapshot
Sandbox: Add comment on thread usage
Sandbox: Add jdbcuser and ledgerid CLI arguments
Sandbox: enable SQL ledger
Sandbox: fix ledgerEnd
Sandbox: Add postgres snapshot test
Sandbox: Refactor SQL ACS
Update ACS in the same transaction as updating ledger_entries
wip CommandTransactionIT
small refactoring of Postgres Ledger creation
removing jdbc user from CLI args as it should rather be part of the jdbc url
using a thread factory for the connection accessor thread
removing timestamps from contracts table
using Akka test helpers
handling failures as rejections
removing unnecessary fields from ActiveContractsCheck
minor cleanup
small optimisation in storing Anorm queries as constants
making ContractDataParser thinner
formatting
cleanup
PostgresLedger does not need initial contracts anymore
add docs to getStreamingConnection
configured streaming connection pool to grow and shrink dynamically
crashing on set jdbcUrl CLI arg
rolling back in case of an exception
adding some docs
adding some comments
minor renaming
Add runSQL comment
* wip ActiveContracts
* fixup
* creating class ActiveContractsManager to be explicit on responsibility
* cleanup
- Remove obsolete code introduced in this PR
* scalafmt
* changed wrong autoCommit behaviour
* formatting
* amended comment on denormalisation
* disabling failing assert for now
changing ports to use 6865 everywhere
no need for default ports
addressing missed outdated ports
changed more ports from 7600 to 6865
dealt with more 8080s
