Commit Graph

91 Commits

Author SHA1 Message Date
Jussi Mäki
0dc49ea811
Fixes to kvutils when used as external workspace (#984)
* Export daml_kvutils_java_proto and add (un)packDamlLogEntryId

Export needed when using kvutils via bazel from another workspace.

* Use 'String' for participant state's LedgerId

Ref.PackageId is the wrong thing, and likely MatchingStringModule
isn't very usable from Java.
2019-05-07 17:55:11 +02:00
gleber
e419420205
Improve error messages of Ledger API Test Tool. (#949)
* Drop LedgerTestingHelpers.asyncFromTimeService as unused.

* ledger-api-test-tool: Add clues to asserts in the main code paths.

This ensures that users get more informative errors for the currently used
semantic tests.

* semantic tester: add operation description to an underlying ledger.

This allows to provide richer error messages to users of LedgerApiTestTool.
2019-05-07 16:01:26 +02:00
Gabor Aranyossy
1a4a571488
making batch queries nicer (#968) 2019-05-07 15:34:26 +02:00
gleber
1225b45ce5
Extend test durations on CI for Ledger API Test Tool driven test. (#944)
* Extend test durations on CI for Ledger API Test Tool driven test.

This introduces a command-line argument to scale timeouts used in the test.

* ledger-api-its: Make FutureTimeouts.timeout duration scaled inside.

Also include more information in the error message.
2019-05-07 14:52:49 +02:00
Moritz Kiefer
55d5fa5dea
Upgrade to nixpkgs 19.03 (again) (#962)
* Upgrade to nixpkgs 19.03 (#871)

* Fix nix-build restart

Without || true we will never get to the next line.
2019-05-07 13:55:30 +02:00
Remy
0489c6e0a5
Daml lf type safty (Party & PackageId) (#761)
* daml-lf: split SimpleString into Party and PackageId

* daml-lf remove parameter from DefinitionRef
2019-05-06 20:40:43 +02:00
Gabor Aranyossy
dd1ac5067b
adding a simple test to hedge against existing sql migration files (#943)
* simple test to hedge against existing sql migration files

* copyright

* adding better failure message

* adding intructions on generating the digest file
2019-05-06 17:34:56 +02:00
Nick Smith
d725d50be9 add result of exercise to transaction -- fixes #479 (#583)
Modify the DAML Engine, Ledger API and Sandbox to pass the result of the
exercise as a field of the transactions.
2019-05-06 14:33:44 +02:00
Jussi Mäki
32a168a999
Index service refactoring (#928)
* Add configuration to LedgerInitialConditions

* Wait until index state is established in reference index service

This fixes a race condition where the daml-on-x server has been
brought up before the index service can serve it.

* Remove ledgerId checks from index service

Now that we have decided on non-changing ledger identifiers, remove
the ledgerId checks from index service and simplify the interface.

* Reformat and fix leftover ??? from DamlOnXTransactionService
2019-05-06 13:59:06 +02:00
Gabor Aranyossy
990a8ea5db
Dispatcher cleanup (#855)
* dispatcher cleanup

* doc improvements

* extracting SubSource from Dispatcher

* moving tests
2019-05-06 11:43:51 +02:00
Moritz Kiefer
9deb4ae7b2 Revert "Upgrade to nixpkgs 19.03 (#871)" (#917)
This reverts commit 3d8acde916.

For some reason that commit seems to have resulted in a lot of
"unexpected end of file" errors during cache downloads. I do not know
what is going on here or how to fix it so let’s revert it for now.
2019-05-05 09:10:20 +00:00
Moritz Kiefer
3d8acde916 Upgrade to nixpkgs 19.03 (#871) 2019-05-04 20:01:48 +00:00
gleber
c85503159b ledger-api-test-tool: uniquify command ids. (#884)
Add 'commandIdMangler' lambda to SemanticTester which forms command identifiers,
which can be passed via a constructor.
2019-05-03 16:46:15 +00:00
Francesco Mazzoli
e8261f2fd9
clarify the role of party sets in various locations (#766)
* clarify role of party sets in various locations

* use 1.dev for tests...

...so that we're sure we're testing the latest value / tx versions.
2019-05-03 17:09:59 +02:00
gleber
e057265af4 LedgerApiTestTool: exit with success when doing '--extract' (#889) 2019-05-03 14:41:55 +00:00
Gerolf Seitz
ecf6ece8c4
CommandService returns useful data for successful submissions (#875)
Submitting a command via the CommandService now returns either the
transaction id (SubmitAndWaitForTransactionId), the flat transaction
(SubmitAndWaitForTransactionResponse), or the transaction tree
(SubmitAndWaitForTransactionTreeResponse).

This means that users don't have to wade through the transaction stream
to retrieve the resulting transaction. This is particularly useful in
combination with #479.

Fixes #406
2019-05-03 16:01:41 +02:00
gleber
b3e2e10897 scenario-tester: Allow to mangle names used in scenarios before their executions. (#795)
* Allow to mangle names used in scenarios before their executions.

This allows to run a scenario against a long-running server repeatedly and avoid
clashes between runs, since each party is unique (up to the randomness used).

* ScenarioRunner: add test for partyNameMangler
2019-05-03 13:04:23 +00:00
Robert Autenrieth
44e43aa620 Reject running sandbox with --scenario and -w (#840)
Scenarios are always run in static time mode.
This change prevents users from thinking the scenario
was run in wallclock mode.
2019-05-03 10:38:03 +02:00
Gerolf Seitz
f4d8e134e3
Add TransactionService methods for looking up flat transactions (#830)
This change is needed in preparation of #406, where we want to return a
transaction tree and flat transaction after a SubmitAndWaitForTransaction(Tree).
2019-05-03 09:03:12 +02:00
Jussi Mäki
746c69255e
Move out classes from participant-state package object (#849)
Having the classes specified in the package object makes the
unusable from Java.
2019-05-02 17:16:27 +02:00
Francesco Mazzoli
cf89366ac6 respect contract visibility when looking up contract keys (#753)
* remove outdated flags from comments

* update tests in preparation for visibility rules

* check visibility of contract keys, fixes #751

* add release notes

* include visibility check in scenario runner

* stephen's suggestions

* fix type error in api server example

* scalafmt

* bring TransactionFiltration.scala up to speed with disclosure

see
<faf1cde1a1/ledger-api/grpc-definitions/com/digitalasset/ledger/api/v1/event.proto>
for an in-depth description. fixes #794.

* add tests for contract key visibility

* move a lot of ledger testing utils out of command checks

* move a lot of ledger testing utils out of command checks

* remove useless checks regarding divulgence

* add tests regarding event witnesses

* update release notes

* re-introduce safety when disclosing
2019-05-02 14:02:30 +00:00
Gabor Aranyossy
c501d64264
fixing bug for mishandling potential offset gaps (#802)
* hedging against possibly gaps in offsets

* doing the math right

* adding some notes

* extracting pipeline logic
 Please enter the commit message for your changes. Lines starting

* extracting "run until" logic to be testable

* testing _run until_ logic

* adding release note

* copyright

* adding link to PR

* changing lookUpByTransactionId to use TransactionPipeline

* fixing issue identified by Extractor tests
2019-05-02 15:00:41 +02:00
gleber
77edbba8d3 reference-server: Fix offset handling for empty resulting transaction streams. (#793)
* reference-server: Add invariant that observed offset is monotonic.

* reference-server: Fix offset handling for empty resulting transaction streams.

If the requested transaction stream (by range) returns empty, code - previously
- would reset 'currentOffset' to None, which was breaking invariants for the
next transaction stream request. Ensure that 'currentOffset' is maintained in
such scenario.
2019-04-30 14:50:17 +00:00
gleber
f6a29f3f69 Drop unused LedgerTestingHelpers from //ledger/sandbox. (#760)
Looks like only LedgerTestingHelpers from //ledger/ledger-api-integration-tests/
is used.
2019-04-30 13:20:13 +00:00
Gabor Aranyossy
43ce45c84e
changing parameters table to have typed entries (#787)
* chaning parameters table to have typed entries

* finishing thought on parameters table
2019-04-30 14:16:15 +02:00
Jussi Mäki
2328914ba4 Update participant-state-kvutils based on feedback (#780)
- Make `getLedgerInitialConditions` a `Source` to mirror `stateUpdates`.
  This makes it easier to implement in Java.

- Expose transaction submission outputs with `KeyValueSubmission.transactionOutputs`
  for ledger implementations that require knowledge of outputs up-front.

- Fix typo in unpackDamLLogEntry.
2019-04-30 09:14:20 +00:00
Gabor Aranyossy
f8a5fdc493
SQL performance improvements (#752)
* document performance bottleneck source

* making transaction ingestion parallel

* dump

* reverting test parameters

* updating ledger end in a race-safe way

* ignoring ledger entry persistence result

* fixing faulty update logic

* fixing tests relying on invalid semantics

* Update ledger/sandbox/src/main/scala/com/digitalasset/platform/sandbox/stores/ledger/sql/dao/PostgresLedgerDao.scala

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>

* Update ledger/sandbox/src/main/scala/com/digitalasset/platform/sandbox/stores/ledger/sql/SqlLedger.scala

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>
2019-04-29 21:22:01 +02:00
mziolekda
dc17c30d76 implement TLS in reference daml-on-x server (#739)
* implement TLS in reference daml-on-x server

* add tls configuration to LedgerApiTestTool

* add ssl tests to the reference ledger server
2019-04-29 17:01:26 +00:00
Francesco Mazzoli
71f61b0691
sign off / document first version of SQL schema. (#759) 2019-04-29 18:23:02 +02:00
Jussi Mäki
a8be6442bf
DamlOnX: release kvutils (#763)
* Move package comment to correct place

* Update kvutils package documentation

* Release participant-state-kvutils
2019-04-29 16:05:22 +02:00
Gabor Aranyossy
8e54bf2e67
documenting persistence with Postgres in SDK docs (#722)
* documenting persistence with Postgres in SDK docs

* revealing jdbcurl flag as it's docuemnted now

* updating release notes

* Update docs/source/tools/sandbox.rst

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>

* Update docs/source/tools/sandbox.rst

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>

* Update docs/source/tools/sandbox.rst

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>

* Apply suggestions from code review

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>

* changed release note entry as suggested

* Apply suggestions from code review

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>
2019-04-29 15:22:05 +02:00
gleber
c095a2e20c Ledger api test tool release (#756)
* release: make 'ci/release.sh' runnable for dry runs.

release-dry-run.sh is outdated and duplicates logic from ci/release.sh, so it
got deleted.

* ledger-api-test-tool: release the tool together with the SDK components.

* ledger-api-test-tool: update docs to reflect distribution mechanism.

* ledger-api-test-tool: further docs refinements.

* Add Ledger API Test Tool mention into release notes.
2019-04-29 12:16:19 +00:00
Jussi Mäki
105ed7aba6
DamlOnX participant-state key-value utilities (#637)
* Work in progress for the key-value participant-state utilities

* WIP on kvutils reference

* Try out "InMemoryKVParticipantState" in the reference server

* Work in progress on the rewrite based on code review

Main idea here is that there's functions to produce
submissions, and function to process a submission and
produce the log entries, and a function to convert
a log entry into 'Update'.

* WIP

* Add documented traits to clean up the key-value utility interfaces

* reorganize and address code review

* Add tests for in-memory kvutils example plus fix few missing bits

* Documentation, logging and heartbeats

* Move heartbeat source init out of actor preStart

* Add DamlOnXTimeService. Package inputs. Fix entryId references.

Semantic tests pass now.

* copyright headers & reformat

* Fix input computation, improve docs in daml_kvutils.proto

The DamlContractState input entries were missing for fetches
and non-consuming exercises.

* Disable SemanticTests.timeTravel

This test only works against Sandbox with a static time model,
not against arbitrary daml-on-x implementations. We can re-enable
it once a test service shim has been implemented.
2019-04-29 13:37:16 +02:00
Gabor Aranyossy
2fc40c3416
introduced paging when streaming ledger entries (#754)
* introduced paging when streaming ledger entries

* adding requested comment

* Add comment regarding safety of chunking
2019-04-29 12:56:02 +02:00
gleber
0ef84c19b3 Add Ledger API Test Tool docs into DAML docs (#587)
* docs: add structure for Ledger API Test Tool

* docs: ledger-api-test-tool: move docs to DAML docs.

* docs: dlik: add copyrights

* ledger-api-test-tool: improve docs

* ledger-api-test-tool: further docs refinements.

* ledger-api-test-tool: expose stacktraces of SemanticTester to the user.

Previously timeout errors would produce no useful output. This gives some -
albeit low quality - output to help pinpoing the issues. This needs improvement
for better UX.

* docs: dlik: summarize how to use the tool against a DAML ledger.
2019-04-29 09:14:14 +00:00
Jussi Mäki
fce04b9e4a Track the packages used to produce a transaction (#735)
Add field 'usedPackages' to Transaction to record the packages
that were used during interpretation to produce said transaction.
This serves as a hint to specify what packages are required to
validate the transaction. Note that this may not be sufficient
for future implementations that e.g. compile DAML-LF to JVM
bytecode and require packages containing only type definitions
as inputs.

This hint is mainly required for ledger implementations for which the
packages are tracked similarly to contract instances, and which
require that all inputs to their transaction must be declared
beforehand.

This work is required for the participant-state key-value utilities
described in issue #410 and implemented in PR #637.
2019-04-27 01:12:04 +00:00
Stephen Compall
0ba1e5b386 increase ledger-api IT timeout; daml-lf, language-support/scala: remove long-deprecated symbols (#741)
* remove long-deprecated symbols from daml-lf and language-support scala libraries

* remove useless usage of 2-parameter TemplateId constructor

* grow ledger-api-integration-tests to account for usual 300s-ish runtime

* buildifier
2019-04-26 21:05:26 +00:00
Stephen Compall
a3e9aad147
remove major LF dev version (#681)
* removing major LF dev version from Haskell proto codecs

* removing major LF dev version from scenario service client

* missed import

* remove Scala support for dev major version; remove --allow-dev option from sandbox cli

* Version.minorFromCliOption function

* don't build daml-stdlib artifacts for dev major

* remove damlc CLI --target dev

* release note about removed dev major LF version

* governance now discusses minor dev, no more major dev

* don't build from daml_lf_dev.proto anymore

* remove daml_lf_dev.proto

* raise deprecated release

* reserve 9999 in the ArchivePayload sum, as suggested by @bitonic

* use reserved proto keyword, as suggested by @bitonic

- `reserved` cannot occur within `oneof` block

* remove --allow-dev test

* dev removal release note followed the previous release; move it back to HEAD
2019-04-26 13:10:09 -04:00
Robert Autenrieth
7060ebf912
Implement stable hashing of contract keys in SQL sandbox (#544) 2019-04-26 16:34:12 +02:00
Gabor Aranyossy
aca0506c7c
Making api integration tests run exclusively to avoid flakiness (#717)
* making api integration tests run exclusively to avoid flakiness

* removing comment

* fixed bazel formatting
2019-04-26 12:02:26 +02:00
Gabor Aranyossy
cc32ee5025
adding missing tests around Sandbox's CLI module (#668)
* introducing CliSpec

* updating sandbox sdk docs

* fixing the missing code block

* adding missing copyright headers

* making allow-dev hidden

* HardCoded -> Predefined

* updated changelog
2019-04-26 10:12:23 +02:00
Remy
433184626d Ledger Api: drop ledger api domain values in favor of LF-values (#649) 2019-04-25 23:15:12 +00:00
mziolekda
cb56af6f79
validate invariants on the transaction streams in indexer (#561)
* validate invariants on the transaction streams, terminate them on violation

* provide unit tests for offset validators
2019-04-25 14:58:37 +02:00
Remy
680239b5cf daml-lf fix and cleanup Decimal.fromString (#606) 2019-04-25 00:22:45 +00:00
Michał Majcherski
bd7b61b1a9 windows: fix concurrent sandbox tars building (#667)
* windows: fix concurrent sandbox tars building

* windows: add bazel shutdown before build
2019-04-24 16:50:50 +00:00
Gabor Aranyossy
40f196f7f6
adding Metrics around Ledger operations (#646)
* adding metrics around ledger components

* adding missing copyright headers

* adding requested comment

* fixing merge conflicts
2019-04-24 16:51:20 +02:00
Stephen Compall
fef8a956ee use WithTxValue in place of VersionedValue more broadly (#643)
* GenTransaction.WithTxValue alias

* undo parts of parent dealing explicitly with serialization

* use GenNode.WithTxValue more often

* use Node*.WithTxValue more often
2019-04-24 13:54:41 +00:00
Gabor Aranyossy
f1f494ad60 Introducing range queries in Dispatcher (#611)
* introduced a SteppingMode abstraction for Dispatcher

* changing InMemoryLedger to use range queries

* SQL Ledger uses range queries

* documenting + fixing off by one semantics of Dispatcher

* adapting DispatcherIT

* added some docs

* reverting off-by-one offset semantics required on the ledger-api

* formatting

* introducing a case class ParsedEntry for sql result parsing

* testing Dispatcher with slow range queries as well

* adding missing ORDER BY ASC

* removing commented out code
2019-04-24 11:31:17 +00:00
Gabor Aranyossy
6422e2cafd Update README.md (#640)
adding missing description in README.md
2019-04-23 17:47:54 +00:00
Jussi Mäki
7c275b17fb DamlOnX refactoring: Drop StateInit, Long Offset (#519)
* daml-on-x: Drop StateInit and use AtomicReference

- Drop the StateInit update message and introduce "getLedgerInitialConditions".
- Switch to AtomicReference from SyncVar as this is non-blocking and hence safer to use from Akka.

* ledger/participant-state: Rename recordTimeEpoch to initialRecordTime

and use the time from timeProvider at the ledger object initialization
time as the initial record time.

* fmt

* Fix use of repo as external workspace. Fix compilation issue in IndexState.scala

* Refix the use of this repo as external bazel workspace

* Fix compilation after rebase

* refmt
2019-04-23 12:56:40 +00:00