Commit Graph

1559 Commits

Author SHA1 Message Date
Gabor Aranyossy
496cf5069f
Re-revert) "Ledger API Server refactor" (#1109)
* Revert "Revert "Ledger API Server refactor (#1077)" (#1096)"

This reverts commit e2fa13e62b.

* fixing race conditions why resetting the ledger
2019-05-13 17:23:03 +02:00
Gerolf Seitz
8158269b7d
Remove ExercisedEvent in Event oneof. (#1067)
* Remove ExercisedEvent in Event oneof.

The Event message is only used in the Transaction message. Flat
transactions do not contain exercised events, but only created and
archived events. Therefore we can remove the ExercisedEvent from the
Event oneof, without breaking transport compatibility.

HOWEVER: The Java Bindings used to use the data.Event class for both flat
transactions and transaction trees. To properly represent the actual
event types in the two transaction structures,
1) Event is now and interface and is only used in the Transaction class.
2) there is a new interface TreeEvent, which is used in the TransactionTree class.

* CreatedEvent implements Event and TreeEvent
* ExercisedEvent implements TreeEvent
* ArchivedEvent implements Event

Some "pathological" cases where an occurrence of an exercised event
would have resulted only in an exception, are now removed (see change in
LedgerApiV1.scala).

Fixes #960.
2019-05-13 14:36:13 +02:00
Remy
2e3a87934b Daml lf type safty (ChoiceName, VarName, FieldName, ConstructorName) (#983)
* daml-lf: make DefinitionRef more typesafe

* daml-lf: Identifier -> DefinitionRef

* daml-lf: remove unsafe apply and copy methods from DottedName

* daml-lf: create identifier

* daml-lf: make ChoiceNames Identifiers

* daml-lf: cleanup TVar

* daml-lf: FieldNames & VariantConstructors -> Identifiers

* bazel fmt

* daml-lf: VarName -> Identifier

* daml-lf: drop return inside Ref.scala

* daml-lf Identifier -> Name

* daml-lf DefinitionRef -> Identifier

* daml-lf make iface more type safe
+ address Francesco's comments

* daml-lf: remove unsafe unapply from MatchingStringModule

* fix navigator

* Address Stephen's Comments
2019-05-13 11:17:12 +00:00
Gabor Aranyossy
e2fa13e62b Revert "Ledger API Server refactor (#1077)" (#1096)
This reverts commit 17060b6225.
2019-05-13 10:31:14 +00:00
Gabor Aranyossy
17060b6225
Ledger API Server refactor (#1077)
* adding necessary @volatiles

* services are factored out

* no need for ResetService to stop the services before

* removing not needed callback methods

* mostly renaming things

* some further cleanup
2019-05-13 10:51:36 +02:00
Gabor Aranyossy
4abc18d8a1
avoiding linear searching for transactions from genesis (#994)
* avoiding linear searching for transactions from genesis

* validating transaction ids

* one more failing test nailed down

* fixing off by 1 error

* docs + release notes

* fixing EventIdFormatterSpec

* fixing broken validation test
2019-05-10 11:28:14 +02:00
Francesco Mazzoli
de54e8f60f
dynamically check type of contract ids, fixes #1005 (#1037)
Up to now, the engine blindly assumed that contract ids pointed to
contracts of the right type. However, this assumption is faulty:
contract ids coming from the Ledger API cannot be type checked
in command translation since we need access to the contract itself
to do so.

This caused some seriously surprising / broken behavior: one could
send an exercise command with the wrong template id and still go
through, or break internal invariants about the type of choices.

This commit fixes this by checking that the type of the contract
instances we fetch is correct at runtime.

cc @hurryabit @dajmaki @remyhaemmerle-da @S11001001 @meiersi-da
2019-05-09 17:11:05 +02:00
Jussi Mäki
66934f8a1e
Fix log output of ledger-api-integration-tests (#1016)
The logback-test.xml file was not included as a resource.
2019-05-09 17:09:32 +02:00
Jussi Mäki
17438382b8 Remove ledger-api-server-example (#1017)
* Remove ledger-api-server-example

This code does not serve any purpose anymore as it has been replaced
by ledger/participant-state and friends.

* Remove ledger-api-server-example from Windows build
2019-05-09 17:04:38 +02:00
Gabor Aranyossy
b72377a950
cleaning up LedgerApiServer (#1036)
* cleanup/refactoring of LedgerApiServer

* fixed LedgerResource

* fixed ResetService to be great again
2019-05-09 15:21:03 +02:00
Gerolf Seitz
662a21f493
Fix LedgerResource after concurrent changes being merged to master (#1033) 2019-05-09 13:01:49 +02:00
Gabor Aranyossy
b6b9a08c82
increasing a perf test timeout due to flakiness (#1028) 2019-05-09 11:26:13 +02:00
Gerolf Seitz
c89f3bdebe
Sandbox: Respect MRT (#990)
Transactions with a record time > maximum record time are now rejected
with a timeout error instead of being committed to the ledger.
2019-05-09 11:10:54 +02:00
Gabor Aranyossy
82edab41f8
making sql queue depth configurable -> making back-pressure tests more reliable (#1027)
* making sql queue depth configurable -> making back-pressure tests more reliable

* fixing SqlLedgerSpec
2019-05-09 10:42:22 +02:00
Francesco Mazzoli
5b1e82b4ef run ledger-api-integration-tests exclusively (#1018)
might help with #1012
2019-05-08 17:43:01 +00:00
Remy
67772baf61 sandbox: correct KeyHasherSpec (#995) 2019-05-08 15:00:16 +00:00
Jussi Mäki
a23000446b
Check contract visibility in reference index service (#947)
* Use Sandbox's ActiveContracts in reference index service

This properly handles contract key updates and adds the required
checks for contract activeness and visibility.

Could consider moving the ActiveContracts.scala into another library,
but since we're soon starting the work to bring sandbox onto the
daml-on-x stack we might postpone that.

* reformat

* Exclude logback.xml from sandbox the library

This allows users of sandbox as a library to specify their
own logging configuration.
2019-05-08 14:36:28 +02:00
Stefano Baghino
f9839a7e4c
Re-order fields in records if all labels are provided (#989)
* Re-order fields in records if all labels are provided

Fixes #988

* Add line to release notes

* Fix line on release notes

* Document change on the Ledger API definition

* Simplify pattern matching

* Improve and fix tests

Integration tests now look for the "Missing field label" rather then the
"Mismatching record label" when inspecting errors. Furthermore, a test
has been added that ensures that repetitions in labels still raise an
error.

* Add comment

Comment about relationship between fields length check and fields reordering based on labels

Co-Authored-By: stefanobaghino-da <43749967+stefanobaghino-da@users.noreply.github.com>

* Improve Ledger API docstring

Explicitly state the semantics of labeled fields (i.e. repeated keys are not accepted)

Co-Authored-By: stefanobaghino-da <43749967+stefanobaghino-da@users.noreply.github.com>
2019-05-08 12:54:53 +02:00
Jussi Mäki
0dc49ea811
Fixes to kvutils when used as external workspace (#984)
* Export daml_kvutils_java_proto and add (un)packDamlLogEntryId

Export needed when using kvutils via bazel from another workspace.

* Use 'String' for participant state's LedgerId

Ref.PackageId is the wrong thing, and likely MatchingStringModule
isn't very usable from Java.
2019-05-07 17:55:11 +02:00
gleber
e419420205
Improve error messages of Ledger API Test Tool. (#949)
* Drop LedgerTestingHelpers.asyncFromTimeService as unused.

* ledger-api-test-tool: Add clues to asserts in the main code paths.

This ensures that users get more informative errors for the currently used
semantic tests.

* semantic tester: add operation description to an underlying ledger.

This allows to provide richer error messages to users of LedgerApiTestTool.
2019-05-07 16:01:26 +02:00
Gabor Aranyossy
1a4a571488
making batch queries nicer (#968) 2019-05-07 15:34:26 +02:00
gleber
1225b45ce5
Extend test durations on CI for Ledger API Test Tool driven test. (#944)
* Extend test durations on CI for Ledger API Test Tool driven test.

This introduces a command-line argument to scale timeouts used in the test.

* ledger-api-its: Make FutureTimeouts.timeout duration scaled inside.

Also include more information in the error message.
2019-05-07 14:52:49 +02:00
Moritz Kiefer
55d5fa5dea
Upgrade to nixpkgs 19.03 (again) (#962)
* Upgrade to nixpkgs 19.03 (#871)

* Fix nix-build restart

Without || true we will never get to the next line.
2019-05-07 13:55:30 +02:00
Remy
0489c6e0a5
Daml lf type safty (Party & PackageId) (#761)
* daml-lf: split SimpleString into Party and PackageId

* daml-lf remove parameter from DefinitionRef
2019-05-06 20:40:43 +02:00
Gabor Aranyossy
dd1ac5067b
adding a simple test to hedge against existing sql migration files (#943)
* simple test to hedge against existing sql migration files

* copyright

* adding better failure message

* adding intructions on generating the digest file
2019-05-06 17:34:56 +02:00
Nick Smith
d725d50be9 add result of exercise to transaction -- fixes #479 (#583)
Modify the DAML Engine, Ledger API and Sandbox to pass the result of the
exercise as a field of the transactions.
2019-05-06 14:33:44 +02:00
Jussi Mäki
32a168a999
Index service refactoring (#928)
* Add configuration to LedgerInitialConditions

* Wait until index state is established in reference index service

This fixes a race condition where the daml-on-x server has been
brought up before the index service can serve it.

* Remove ledgerId checks from index service

Now that we have decided on non-changing ledger identifiers, remove
the ledgerId checks from index service and simplify the interface.

* Reformat and fix leftover ??? from DamlOnXTransactionService
2019-05-06 13:59:06 +02:00
Gabor Aranyossy
990a8ea5db
Dispatcher cleanup (#855)
* dispatcher cleanup

* doc improvements

* extracting SubSource from Dispatcher

* moving tests
2019-05-06 11:43:51 +02:00
Moritz Kiefer
9deb4ae7b2 Revert "Upgrade to nixpkgs 19.03 (#871)" (#917)
This reverts commit 3d8acde916.

For some reason that commit seems to have resulted in a lot of
"unexpected end of file" errors during cache downloads. I do not know
what is going on here or how to fix it so let’s revert it for now.
2019-05-05 09:10:20 +00:00
Moritz Kiefer
3d8acde916 Upgrade to nixpkgs 19.03 (#871) 2019-05-04 20:01:48 +00:00
gleber
c85503159b ledger-api-test-tool: uniquify command ids. (#884)
Add 'commandIdMangler' lambda to SemanticTester which forms command identifiers,
which can be passed via a constructor.
2019-05-03 16:46:15 +00:00
Francesco Mazzoli
e8261f2fd9
clarify the role of party sets in various locations (#766)
* clarify role of party sets in various locations

* use 1.dev for tests...

...so that we're sure we're testing the latest value / tx versions.
2019-05-03 17:09:59 +02:00
gleber
e057265af4 LedgerApiTestTool: exit with success when doing '--extract' (#889) 2019-05-03 14:41:55 +00:00
Gerolf Seitz
ecf6ece8c4
CommandService returns useful data for successful submissions (#875)
Submitting a command via the CommandService now returns either the
transaction id (SubmitAndWaitForTransactionId), the flat transaction
(SubmitAndWaitForTransactionResponse), or the transaction tree
(SubmitAndWaitForTransactionTreeResponse).

This means that users don't have to wade through the transaction stream
to retrieve the resulting transaction. This is particularly useful in
combination with #479.

Fixes #406
2019-05-03 16:01:41 +02:00
gleber
b3e2e10897 scenario-tester: Allow to mangle names used in scenarios before their executions. (#795)
* Allow to mangle names used in scenarios before their executions.

This allows to run a scenario against a long-running server repeatedly and avoid
clashes between runs, since each party is unique (up to the randomness used).

* ScenarioRunner: add test for partyNameMangler
2019-05-03 13:04:23 +00:00
Robert Autenrieth
44e43aa620 Reject running sandbox with --scenario and -w (#840)
Scenarios are always run in static time mode.
This change prevents users from thinking the scenario
was run in wallclock mode.
2019-05-03 10:38:03 +02:00
Gerolf Seitz
f4d8e134e3
Add TransactionService methods for looking up flat transactions (#830)
This change is needed in preparation of #406, where we want to return a
transaction tree and flat transaction after a SubmitAndWaitForTransaction(Tree).
2019-05-03 09:03:12 +02:00
Jussi Mäki
746c69255e
Move out classes from participant-state package object (#849)
Having the classes specified in the package object makes the
unusable from Java.
2019-05-02 17:16:27 +02:00
Francesco Mazzoli
cf89366ac6 respect contract visibility when looking up contract keys (#753)
* remove outdated flags from comments

* update tests in preparation for visibility rules

* check visibility of contract keys, fixes #751

* add release notes

* include visibility check in scenario runner

* stephen's suggestions

* fix type error in api server example

* scalafmt

* bring TransactionFiltration.scala up to speed with disclosure

see
<faf1cde1a1/ledger-api/grpc-definitions/com/digitalasset/ledger/api/v1/event.proto>
for an in-depth description. fixes #794.

* add tests for contract key visibility

* move a lot of ledger testing utils out of command checks

* move a lot of ledger testing utils out of command checks

* remove useless checks regarding divulgence

* add tests regarding event witnesses

* update release notes

* re-introduce safety when disclosing
2019-05-02 14:02:30 +00:00
Gabor Aranyossy
c501d64264
fixing bug for mishandling potential offset gaps (#802)
* hedging against possibly gaps in offsets

* doing the math right

* adding some notes

* extracting pipeline logic
 Please enter the commit message for your changes. Lines starting

* extracting "run until" logic to be testable

* testing _run until_ logic

* adding release note

* copyright

* adding link to PR

* changing lookUpByTransactionId to use TransactionPipeline

* fixing issue identified by Extractor tests
2019-05-02 15:00:41 +02:00
gleber
77edbba8d3 reference-server: Fix offset handling for empty resulting transaction streams. (#793)
* reference-server: Add invariant that observed offset is monotonic.

* reference-server: Fix offset handling for empty resulting transaction streams.

If the requested transaction stream (by range) returns empty, code - previously
- would reset 'currentOffset' to None, which was breaking invariants for the
next transaction stream request. Ensure that 'currentOffset' is maintained in
such scenario.
2019-04-30 14:50:17 +00:00
gleber
f6a29f3f69 Drop unused LedgerTestingHelpers from //ledger/sandbox. (#760)
Looks like only LedgerTestingHelpers from //ledger/ledger-api-integration-tests/
is used.
2019-04-30 13:20:13 +00:00
Gabor Aranyossy
43ce45c84e
changing parameters table to have typed entries (#787)
* chaning parameters table to have typed entries

* finishing thought on parameters table
2019-04-30 14:16:15 +02:00
Jussi Mäki
2328914ba4 Update participant-state-kvutils based on feedback (#780)
- Make `getLedgerInitialConditions` a `Source` to mirror `stateUpdates`.
  This makes it easier to implement in Java.

- Expose transaction submission outputs with `KeyValueSubmission.transactionOutputs`
  for ledger implementations that require knowledge of outputs up-front.

- Fix typo in unpackDamLLogEntry.
2019-04-30 09:14:20 +00:00
Gabor Aranyossy
f8a5fdc493
SQL performance improvements (#752)
* document performance bottleneck source

* making transaction ingestion parallel

* dump

* reverting test parameters

* updating ledger end in a race-safe way

* ignoring ledger entry persistence result

* fixing faulty update logic

* fixing tests relying on invalid semantics

* Update ledger/sandbox/src/main/scala/com/digitalasset/platform/sandbox/stores/ledger/sql/dao/PostgresLedgerDao.scala

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>

* Update ledger/sandbox/src/main/scala/com/digitalasset/platform/sandbox/stores/ledger/sql/SqlLedger.scala

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>
2019-04-29 21:22:01 +02:00
mziolekda
dc17c30d76 implement TLS in reference daml-on-x server (#739)
* implement TLS in reference daml-on-x server

* add tls configuration to LedgerApiTestTool

* add ssl tests to the reference ledger server
2019-04-29 17:01:26 +00:00
Francesco Mazzoli
71f61b0691
sign off / document first version of SQL schema. (#759) 2019-04-29 18:23:02 +02:00
Jussi Mäki
a8be6442bf
DamlOnX: release kvutils (#763)
* Move package comment to correct place

* Update kvutils package documentation

* Release participant-state-kvutils
2019-04-29 16:05:22 +02:00
Gabor Aranyossy
8e54bf2e67
documenting persistence with Postgres in SDK docs (#722)
* documenting persistence with Postgres in SDK docs

* revealing jdbcurl flag as it's docuemnted now

* updating release notes

* Update docs/source/tools/sandbox.rst

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>

* Update docs/source/tools/sandbox.rst

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>

* Update docs/source/tools/sandbox.rst

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>

* Apply suggestions from code review

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>

* changed release note entry as suggested

* Apply suggestions from code review

Co-Authored-By: gaboraranyossy-da <gabor.aranyossy@digitalasset.com>
2019-04-29 15:22:05 +02:00
gleber
c095a2e20c Ledger api test tool release (#756)
* release: make 'ci/release.sh' runnable for dry runs.

release-dry-run.sh is outdated and duplicates logic from ci/release.sh, so it
got deleted.

* ledger-api-test-tool: release the tool together with the SDK components.

* ledger-api-test-tool: update docs to reflect distribution mechanism.

* ledger-api-test-tool: further docs refinements.

* Add Ledger API Test Tool mention into release notes.
2019-04-29 12:16:19 +00:00
Jussi Mäki
105ed7aba6
DamlOnX participant-state key-value utilities (#637)
* Work in progress for the key-value participant-state utilities

* WIP on kvutils reference

* Try out "InMemoryKVParticipantState" in the reference server

* Work in progress on the rewrite based on code review

Main idea here is that there's functions to produce
submissions, and function to process a submission and
produce the log entries, and a function to convert
a log entry into 'Update'.

* WIP

* Add documented traits to clean up the key-value utility interfaces

* reorganize and address code review

* Add tests for in-memory kvutils example plus fix few missing bits

* Documentation, logging and heartbeats

* Move heartbeat source init out of actor preStart

* Add DamlOnXTimeService. Package inputs. Fix entryId references.

Semantic tests pass now.

* copyright headers & reformat

* Fix input computation, improve docs in daml_kvutils.proto

The DamlContractState input entries were missing for fetches
and non-consuming exercises.

* Disable SemanticTests.timeTravel

This test only works against Sandbox with a static time model,
not against arbitrary daml-on-x implementations. We can re-enable
it once a test service shim has been implemented.
2019-04-29 13:37:16 +02:00
Gabor Aranyossy
2fc40c3416
introduced paging when streaming ledger entries (#754)
* introduced paging when streaming ledger entries

* adding requested comment

* Add comment regarding safety of chunking
2019-04-29 12:56:02 +02:00
gleber
0ef84c19b3 Add Ledger API Test Tool docs into DAML docs (#587)
* docs: add structure for Ledger API Test Tool

* docs: ledger-api-test-tool: move docs to DAML docs.

* docs: dlik: add copyrights

* ledger-api-test-tool: improve docs

* ledger-api-test-tool: further docs refinements.

* ledger-api-test-tool: expose stacktraces of SemanticTester to the user.

Previously timeout errors would produce no useful output. This gives some -
albeit low quality - output to help pinpoing the issues. This needs improvement
for better UX.

* docs: dlik: summarize how to use the tool against a DAML ledger.
2019-04-29 09:14:14 +00:00
Jussi Mäki
fce04b9e4a Track the packages used to produce a transaction (#735)
Add field 'usedPackages' to Transaction to record the packages
that were used during interpretation to produce said transaction.
This serves as a hint to specify what packages are required to
validate the transaction. Note that this may not be sufficient
for future implementations that e.g. compile DAML-LF to JVM
bytecode and require packages containing only type definitions
as inputs.

This hint is mainly required for ledger implementations for which the
packages are tracked similarly to contract instances, and which
require that all inputs to their transaction must be declared
beforehand.

This work is required for the participant-state key-value utilities
described in issue #410 and implemented in PR #637.
2019-04-27 01:12:04 +00:00
Stephen Compall
0ba1e5b386 increase ledger-api IT timeout; daml-lf, language-support/scala: remove long-deprecated symbols (#741)
* remove long-deprecated symbols from daml-lf and language-support scala libraries

* remove useless usage of 2-parameter TemplateId constructor

* grow ledger-api-integration-tests to account for usual 300s-ish runtime

* buildifier
2019-04-26 21:05:26 +00:00
Stephen Compall
a3e9aad147
remove major LF dev version (#681)
* removing major LF dev version from Haskell proto codecs

* removing major LF dev version from scenario service client

* missed import

* remove Scala support for dev major version; remove --allow-dev option from sandbox cli

* Version.minorFromCliOption function

* don't build daml-stdlib artifacts for dev major

* remove damlc CLI --target dev

* release note about removed dev major LF version

* governance now discusses minor dev, no more major dev

* don't build from daml_lf_dev.proto anymore

* remove daml_lf_dev.proto

* raise deprecated release

* reserve 9999 in the ArchivePayload sum, as suggested by @bitonic

* use reserved proto keyword, as suggested by @bitonic

- `reserved` cannot occur within `oneof` block

* remove --allow-dev test

* dev removal release note followed the previous release; move it back to HEAD
2019-04-26 13:10:09 -04:00
Robert Autenrieth
7060ebf912
Implement stable hashing of contract keys in SQL sandbox (#544) 2019-04-26 16:34:12 +02:00
Gabor Aranyossy
aca0506c7c
Making api integration tests run exclusively to avoid flakiness (#717)
* making api integration tests run exclusively to avoid flakiness

* removing comment

* fixed bazel formatting
2019-04-26 12:02:26 +02:00
Gabor Aranyossy
cc32ee5025
adding missing tests around Sandbox's CLI module (#668)
* introducing CliSpec

* updating sandbox sdk docs

* fixing the missing code block

* adding missing copyright headers

* making allow-dev hidden

* HardCoded -> Predefined

* updated changelog
2019-04-26 10:12:23 +02:00
Remy
433184626d Ledger Api: drop ledger api domain values in favor of LF-values (#649) 2019-04-25 23:15:12 +00:00
mziolekda
cb56af6f79
validate invariants on the transaction streams in indexer (#561)
* validate invariants on the transaction streams, terminate them on violation

* provide unit tests for offset validators
2019-04-25 14:58:37 +02:00
Remy
680239b5cf daml-lf fix and cleanup Decimal.fromString (#606) 2019-04-25 00:22:45 +00:00
Michał Majcherski
bd7b61b1a9 windows: fix concurrent sandbox tars building (#667)
* windows: fix concurrent sandbox tars building

* windows: add bazel shutdown before build
2019-04-24 16:50:50 +00:00
Gabor Aranyossy
40f196f7f6
adding Metrics around Ledger operations (#646)
* adding metrics around ledger components

* adding missing copyright headers

* adding requested comment

* fixing merge conflicts
2019-04-24 16:51:20 +02:00
Stephen Compall
fef8a956ee use WithTxValue in place of VersionedValue more broadly (#643)
* GenTransaction.WithTxValue alias

* undo parts of parent dealing explicitly with serialization

* use GenNode.WithTxValue more often

* use Node*.WithTxValue more often
2019-04-24 13:54:41 +00:00
Gabor Aranyossy
f1f494ad60 Introducing range queries in Dispatcher (#611)
* introduced a SteppingMode abstraction for Dispatcher

* changing InMemoryLedger to use range queries

* SQL Ledger uses range queries

* documenting + fixing off by one semantics of Dispatcher

* adapting DispatcherIT

* added some docs

* reverting off-by-one offset semantics required on the ledger-api

* formatting

* introducing a case class ParsedEntry for sql result parsing

* testing Dispatcher with slow range queries as well

* adding missing ORDER BY ASC

* removing commented out code
2019-04-24 11:31:17 +00:00
Gabor Aranyossy
6422e2cafd Update README.md (#640)
adding missing description in README.md
2019-04-23 17:47:54 +00:00
Jussi Mäki
7c275b17fb DamlOnX refactoring: Drop StateInit, Long Offset (#519)
* daml-on-x: Drop StateInit and use AtomicReference

- Drop the StateInit update message and introduce "getLedgerInitialConditions".
- Switch to AtomicReference from SyncVar as this is non-blocking and hence safer to use from Akka.

* ledger/participant-state: Rename recordTimeEpoch to initialRecordTime

and use the time from timeProvider at the ledger object initialization
time as the initial record time.

* fmt

* Fix use of repo as external workspace. Fix compilation issue in IndexState.scala

* Refix the use of this repo as external bazel workspace

* Fix compilation after rebase

* refmt
2019-04-23 12:56:40 +00:00
Jussi Mäki
91449122f7
Add damlonx artifacts to the release (#627) 2019-04-23 14:00:55 +02:00
Stephen Compall
eb4d418be2 Check contract visibility when looking up contracts in sandbox (#405)
* seal engine.Error

* don't reinvent flatMap in CommandExecutorImpl

* add DivulgenceVisibility sandbox test case

- suggested by @oggy-; thanks

* add global implicit disclosure step to sandbox ACS transaction acceptance

* reorder ActiveContractsManager and include an implicitlyDisclose action

* add submitter argument to SubmissionHandle#lookupActiveContract

- the receiver must filter its response based on witnesses thereof

* propagate implicit disclosure in more places

* it's a bunch of functions

* proper error from the no-divulgence test

* rename ActiveContracts and ActiveContractsSteps to ...InMemory and ActiveContracts

* we need another table

* rename ActiveContract.witnesses to disclosedTo

* don't exclude other disclosures in ActiveContractsManager

* add local implicit disclosure when adding transaction to sandbox acs

* extra argument in ledger-api-server-example

* too late to blind?

* adapt to 10fcab8 #339

* scalafmt

* non-total map get

* disable postgres test in SandboxSemanticTestsLfRunner until implicit disclosure added

* un-flatten witnesses and divulgences from "disclosedTo"; fixes DivulgenceIT

* comment on filtering of lookupActiveContract

- suggested by @francesco-da; thanks

* semantic errors from parent

* scenario to test divulgence success

* submit line 6 as owner; disable divulgence test under SQL

* comment what else to reenable
2019-04-19 17:00:09 +00:00
Gerolf Seitz
7f8dbfeca0 Add CreateAndExercise command throughout the stack (#563)
* Add release notes entry

* Add CreateAndExercise command to Java Bindings data layer

* Add CreateAndExercise command to DAMLe

The CreateAndExerciseCommand allows users to create a contract and
exercise a choice on it within the same transaction. Users can use this
method to implement "callable update functions" by creating a template
that calls the update function in a choice body.

Fixes #382.

* Add CreateAndExercise command handling to the sandbox

* Add CreateAndExercise command to the Ledger API
2019-04-18 16:05:16 +00:00
gleber
91843213bc Ledger API test tool (#496)
* ledger-api-test-tool: create with self-contained tests

Based on code by Tim Williams <tim.williams@digitalasset.com>

* ledger-api-test-tool: added an extract operation for resources

* api-server-damlonx: make ReferenceServer command-line-configurable.

This includes:
- port
- list of dars to load
- enable bad read service

* ledger-api-test-tool: Add test against Reference server.

* ledger-api-test-tool: allow all semantic tests.

* ledger-api-test-tool: move into //ledger/ledger-api-test-tool.

* ledger-api-test-tool: Add README

* ledger-api-test-tool: improve README according to the code review.

* ledger-api-test-tool: language and structure improvements to readme.

Based on review by bethaitman@digitalasset.com.
2019-04-18 13:06:02 +00:00
Gabor Aranyossy
879edf5911 running sandbox performance tests with SQL backend (#580)
* running sandbox performance tests with SQL backend

* adding docs
2019-04-18 07:45:42 +00:00
Gabor Aranyossy
1b56d4385e Extracting large command test from TransactionServiceIT (#566)
* extracting large command test from TransactionServiceIT

* make `preprocessCommands` more performant (see comment)

* fix duplicated command generation in `preprocessCommands`

* enable in-memory test-fixture as well
2019-04-17 16:20:56 +00:00
Remy
de69ed61db
Leger Api. Simplify Ledger Api domain (#577)
* drop domain.Identifier in favor of Ref.Identifier
* drop domain.Party in favor of Ref.Party
2019-04-17 17:03:23 +02:00
Jussi Mäki
5233574d38 Add semantic test for the reference server (#424)
* Add semantic test for the reference server

Currently the semantic test is failing. Likely due to the
location annotation changes.

* Do not compare location annotations in isReplayedBy

The location annotations may not, and do not need to, match due to the
fact that the reconstructed update expression may not exactly match
the original one, and since the interpreter currently picks the closest
location annotation we cannot guarantee that they exactly match.

* scalafmt.

* buildifier.

* client_server_test: Increase timeout to 60s

Spawning a JVM-based server can easily take a long time on a very
loaded system (e.g. when running `bazel test //ledger/...` with enough
parallelism), so better have a high default timeout.

* ledger/api-server-damlonx: Address code review

* Fix client_server_test runner compilation. Bump timeout.

* Mark the reference server semantic test exclusive

* fmt reference/BUILD.bazel
2019-04-17 10:35:47 +02:00
Gabor Aranyossy
2ffe3fe8b4 making SQL Ledger reset-able (#559)
* making SQL Ledger resetable

* fixing compile errors

* enabling the test for the SQL Ledger
2019-04-17 08:30:16 +00:00
Bernhard Elsner
b2062e6ee9
Explain failure of fetch in StdLib docs (#490)
* Explain failure of `fetch` in StdLib docs

* Explain `fetch` failure in DAML Reference docs

* bump integration tests timeout further, still seems to be flaky
2019-04-16 19:53:51 +02:00
Gabor Aranyossy
d7966eb440
porting ResetServiceIT to the common testing suite (#552) 2019-04-16 18:40:09 +02:00
Gabor Aranyossy
a790a0d60d
adding explicit back pressure to Command Submission (#483)
* introduced back-pressure semantics for command submission

* fixup

* reenabling CommandServiceBackPressureIT

* fixing / enhancing the command submission backpressure test

* removed the unneccessary Error from the ADT

* routing CheckPoints/Heartbeats on a different channel to avoid back-pressure

* fixup
2019-04-16 13:50:24 +02:00
mziolekda
02dd917699
make reference index service stop repeating last event indefinitely + refactoring (#482)
* remove early returns

* make reference index service stop repeating last event indefinitely

* make all services follow more consistent design, no unnecessary inherits, no unnecessary methods

* scala format
2019-04-16 11:42:36 +02:00
Robert Autenrieth
d6da1e435b Add support for contract keys in the sandbox SQL backend (#444)
* Add support for contract key to SQL backend

* Address review comments

* Address reviewer comments
2019-04-16 10:11:29 +02:00
Leonid Shlyapnikov
1dfbdb903e
HOTFIX: increasing 30.sec timeout to 300.sec (#462)
* HOTFIX: increasing 30.sec timeout to 300.sec

this to fix SQL-backed ledger tests that currently keep timing out.
It is NOT really a fix, but a temp work-around. Tried to get rid of
this timeout, but it propogates all over the place. Need a separate
story to address this

* HOTFIX: adding a link to the issue
2019-04-15 12:17:53 -04:00
Francesco Mazzoli
f50cbef297 use .dars consistently in tests (#484) 2019-04-15 16:45:08 +02:00
Gerolf Seitz
2f3189db9e
Reuse the same Engine object between ledger resets (#466)
Compared to older versions of the Engine, we do A LOT more validation
now. So when starting with a fresh engine after each reset (via
the ResetService), we also repeat the validation of the loaded packages
again and again. This is VERY expensive, especially for large DAML
packages.

Luckily, the specification of the ResetService states the following:
// Resets the ledger state. Note that loaded DARs won't be removed --
// this only rolls back the ledger to genesis.
This means that we can re-use the same Engine object and benefit from
not having "re-compile" packages via
ConcurrentCompiledPackages#addPackage, more specifically we can omit
this line: 1f2246c822/daml-lf/engine/src/main/scala/com/digitalasset/daml/lf/engine/ConcurrentCompiledPackages.scala (L56)

Fixes #178
2019-04-15 09:19:04 +02:00
Francesco Mazzoli
6af84051ee
fix various conversion functions from string to Decimal (#439)
* fix various conversion functions from string to Decimal

Fixes #399.

This fixes a critical bug -- since:

* The DAML-LF spec specifies that the scale of `Decimal` is 10 --
    that is, there are at most 10 digits past the decimal point:
    <79bbf5c794/daml-lf/spec/value.rst (field-decimal)>.
* However, the code converting from the string representation that
    we get on the wire was _not_ performing this check. This is due
    to two reasons:

    - `Decimal.check` is a function that checks whether a given
        `Decimal` is within the upper and lower bounds of what the
        DAML-LF spec specifies, but crucially it does _not_ check that
        the scale is not exceeded:
        <79bbf5c794/daml-lf/data/src/main/scala/com/digitalasset/daml/lf/data/Decimal.scala (L31)>.
        This behavior is correct in some cases (more on that later),
        but not in others. Crucially, `Decimal.fromString`, which was
        supposed to check if a decimal string literal is valid, used
        this function, which means that it accepted string literals
        containing numbers out of the required scale, rounding them to
        make them fit within the scale. This function has been renamed
        to `Decimal.checkWithinBoundsAndRound`, and a new function
        `Decimal.checkWithinBoundsAndWithinScale` has been added, which
        fails if the number provided has data not within the scale.
        `Decimal.fromString` now uses
        `Decimal.checkWithinBoundsAndWithinScale`.

    - `ApiToLfEngine.parseDecimal` assumed that `Decimal.fromString` _did_
        fail when passed numbers that were in any way invalid, and
        moreover did _not_ use the result of `Decimal.fromString`, but rather
        re-parsed the string into an unconstrained `BigDecimal`:
        <79bbf5c794/ledger/ledger-api-common/src/main/scala/com/digitalasset/platform/participant/util/ApiToLfEngine.scala (L96)>.
        The reason for the code to work this way is that in the past
        it was responsible for converting decimal strings both for the
        current engine but also for an older version of the engine which
        handled decimals of a different type. Both issues have been fixed.

* Therefore, `Decimal`s with scale exceeding the specified scale
    made it into the engine, and contracts could be created containing
    this invalid value.

* Once on the ledger, these bad numbers can be used to produce extremely
    surprising results, due to how `Decimal` operations are
    implemented. Generally speaking, all operations on `Decimal`
    first compute the result and then run the output through
    `Decimal.checkWithinBoundsAndRound`. The reason for this behavior
    is that we specify multiplication and division as rounding their
    output. Consider the case where the bad value 0.00000000005 made
    it to the engine, and is then added to 100. The full-precision
    result will be 100.00000000005, which after rounding becomes 100.
    Therefore, on a ledger where such invalid values exist, it is not
    the case that `y > 0 ==> x + y != x`, and so on.

Thanks a bunch to @briandbecker for the excellent bug report.

* fix failing test using to much precision
2019-04-14 13:49:28 +02:00
Brian Healey
b594fcffc1
correct jq in dev-env (#463)
* correct jq in dev-env

* Remove references to lsof and dead scripts that called lsof since it is no longer in dev-env
2019-04-12 16:44:15 -04:00
Simon Meier
d522effecb
daml-on-x-server: document all participant state interfaces (#432)
* daml-on-x-server: document all participant state interfaces
2019-04-12 17:43:45 +02:00
gleber
aa70c7f64e
Enforce consistent formatting of BUILD files. (#412)
* Add buildifier targets.

The tool allows to check and format BUILD files in the repo.

To check if files are well formatted, run:

    bazel run //:buildifier

To fix badly-formatted files run:

    bazel run //:buildifier-fix

* Cleanup dade-copyright-headers formatting.

* Fix dade-copyright-headers on files with just the copyright.

* Run buildifier automatically on CI via 'fmt.sh'.

* Reformat all BUILD files with buildifier.

Excludes autogenerated Bazel files.
2019-04-12 13:10:16 +02:00
Gabor Aranyossy
3e154fd682
Sandbox Postgres support with passing api integration tests (#392)
* TransactionServiceIT passes

* fixup

* using record time taken from TimeProvider

* taming TransactionBackPressureIT to avoid overloading of submission service

* implement AutoClosable on all Ledger related components having stateful resources

* reenabling InMemory fixture

* disabling contract key integration test for SandboxSQL fixture

* removing TODO

* bumping up timeout on TransactionServiceIT due to slow Azure pipeline

* 1 minute timeout for SqlLedgerSpec

* making jdbcurl CLI argument hidden

* updating release notes
2019-04-12 12:25:48 +02:00
Martin Huschenbett
667a63a9b1
Add location information to DAML-LF produced by damlc (#390)
* Add location information to DAML-LF produced by damlc

This is required to get error locations in the scenario view. Rigth now,
the location information for `create`/`exercise` still points to the
template/choice. I'll fix that in a separate PR.

* Fix test expectations

* Fix more tests
2019-04-11 21:44:48 +02:00
Jussi Mäki
b77cf2c56b
DamlOnX reference implementation (#348)
* Add participant-state and participant-state-index APIs and reference implementations

This adds the (still WIP) interfaces and the in-memory reference implementations
of participant-state and participant-state-index.

See issue #137.

* Pass ledgerId through to getLedgerEnd method

This is needed in the daml-on-x implementation of the
transaction service.

* Add api-server-damlonx and the reference server

This is the initial version of the ledger-api server built on top of
the participant state APIs.

While functionally complete (modulo test services), it has only yet
been tested with the semantic tester. The server and the participant
state APIs are still under active development.

See issue #137.

* Apply scalafmt and copyright headers to new damlonx code

* ledger/damlonx: Fix build errors after merge from master

* ledger/participant-state: separate out the reference impl

* ledger/participant-state-index: separate out reference impl

* ledger/damlonx: Refactoring

- Refactor participant-state into multiple modules
- Introduce structured offset and update id types (vector of ints)
- Properly parse ledger feature flags

* ledger: Disable failing semantic test. Decrease grouping duration.

* scalafmt, fix after merge

* Add missing copyright headers

* ledger/damlonx: Add Simon's comments

* ledger/damlonx: Remove UpdateId, use Offset in both state and state-index.

* ledger/damlonx: scalafmt

* damlonxserver: review participant-state interface

Includes adding issue links for postponed fixes.

* daml-on-x-server: review reference implementation

Fixme's added to https://github.com/digital-asset/daml/pull/348 and
extra issues creaed in https://github.com/digital-asset/daml/milestone/4

* ledger/damlonx: Use SimpleString

* ledgre/damlonx: Backtrack on SimpleString change

See issue #398

* daml-on-x-server: drop unused 'index.impl.reference.package.scala'

* daml-on-x-server: add note on potential transient contracts bug

* ledger/participant-state*: post-merge fixes
2019-04-11 18:05:08 +02:00
Jussi Mäki
1f2246c822 Do not divulge contracts to observers in nonconsuming exercises (#325)
* Do not divulge contracts to observers in nonconsuming exercises

Disables support for non-default ledger feature flags, as they
are meaningless since ledger server logic does not respect the flags.
Instead of large refactoring to add support for the old flag settings,
it is best to disallow the deprecated flags, and later on phase out the
flags completely.

Re-enables test_divulgence_of_token in sandbox semantic tests.

Fixes #157.

* purge LedgerFlags entirely...

...since we only support one version of them anyway, and clearing them

* updated release notes
2019-04-11 14:41:35 +02:00
moritzkiefer-da
fa4067ad1b
Move POM file generation to Bazel rules (#374)
* Move POM file generation to Bazel rules
2019-04-11 11:24:52 +02:00
Francesco Mazzoli
10fcab8040
move ContractTransactionChecks to ledger-api-integration-tests (#339) 2019-04-11 10:22:34 +02:00
Remy
07858455c1
Fix DAML runtime for the new DAML-LF type Map (#204)
* add a test for daml-lf/interface
* fix scala code gen
* fix extractor
* fix navigator backend/frontend
* key of Map are strings in proto/json
2019-04-10 21:30:33 +02:00
Gabor Aranyossy
094b4b6959
Sandbox Sql fixture (#247)
* SandboxSQL fixture + fixing various issues

WIP - playing around with Resources to support Sandbox Fixture with Postgres

Sandbox with Postgres fixture works - LedgerIdentityServiceGivenIT PASSES

adding guards for empty collections in PostgresLedgerDao

CommandCompletionServiceITi passed for both fixtures

fixing EventID formatting problem

another formatting fix

changed wrong autoCommit behaviour

CommandStaticTimeIT passed on both ledgers

rolling back transaction on duplicates

dealing with duplicate commands in postgres

fixing mismapped record_time

adding extra logging for fatal errors

disabling sql support as it's not finished yet

cleanup

removed unnecessary test classes

cleanup

formatting

* ADT was missing Product with Serializable

* fixed the compilation error
2019-04-10 17:04:47 +02:00
Stefano Baghino
7467925aa1
Fetch status.proto from remote, simplify JS gRPC codegen (#285)
* Fetch status.proto from remote, simplify JS gRPC codegen

Fetch the `status.proto` file (part of the standard gRPC distribution)
from a distribution channel. _Moreover_, use the recently introduced
`proto_gen` rule to simplify how the gRPC code for the Node.js bindings
are generated (and remove the need to have `google/rpc/status.proto`
locally in the repository.

* Add plugin_runfiles option to proto_gen

This allows use to add additional files to the bazel sandbox so that
plugins can refer to them. This will subsequently be used by the
protoc-gen-doc plugin.

Also, pass the plugin options via --name_opt parameter.

* Add missing status.proto dependency /language-support/java and /ledger

* Build proto docs using the proto_gen rule

To make this work, I had to turn on the bazel build flag
`--protocopt=--include_source_info` because we cannot turn enable this
flag only for specific build rules.

* Make /ledger-api/grpc-definitions:docs public again

* Revert to the old style of passing plugin arguments to --name_out=options:path

* Suppress output of unzipping

* Fix link for google.rpc.Status in proto-docs
2019-04-09 10:30:01 +02:00
Martin Huschenbett
2e08e2a9d7
Change contract key maintainers to have template argument in scope (#298)
* Change contract key maintainers to have template argument in scope

More precisely, change the contract key type class we desugar the surface
syntax to such that the maintainers function has the template argument
in scope instead of the key. The compiler rewrites all references to the
template argument into references to the key and fails whenever that is
not possible.

* Add TemplateKey class to desugaring module

* Make template key check slightly more lenient

* Fix TODO descriptions
2019-04-09 09:14:03 +02:00
Francesco Mazzoli
f786210169
do not wait for 2 seconds before shutting down the server (#292)
* do not wait for 2 seconds before shutting down the server

should fix #178, will validate with @jberthold-da

* refactor & comment
2019-04-08 19:36:19 +02:00
Gabor Aranyossy
c730a13237
fixing PostgresDaoSpec (#289) 2019-04-08 15:56:56 +02:00
Gabor Aranyossy
8b9f2e5860
Sandbox - active contracts streaming + transaction validation (#197)
* draft of active contracts streaming - still WIP

dumping todos for Robert

Sandbox: fix getActiveContractSnapshot

Sandbox: Add comment on thread usage

Sandbox: Add jdbcuser and ledgerid CLI arguments

Sandbox: enable SQL ledger

Sandbox: fix ledgerEnd

Sandbox: Add postgres snapshot test

Sandbox: Refactor SQL ACS

Update ACS in the same transaction as updating ledger_entries

wip CommandTransactionIT

small refactoring of Postgres Ledger creation

removing jdbc user from CLI args as it should rather be part of the jdbc url

using a thread factory for the connection accessor thread

removing timestamps from contracts table

using Akka test helpers

handling failures as rejections

removing unnecessary fields from ActiveContractsCheck

minor cleanup

small optimisation in storing Anorm queries as constants

making ContractDataParser thinner

formatting

cleanup

PostgresLedger does not need initial contracts anymore

add docs to getStreamingConnection

configured streaming connection pool to grow and shrink dynamically

crashing on set jdbcUrl CLI arg

rolling back in case of an exception

adding some docs

adding some comments

minor renaming

Add runSQL comment

* wip ActiveContracts

* fixup

* creating class ActiveContractsManager to be explicit on responsibility

* cleanup

- Remove obsolete code introduced in this PR

* scalafmt

* changed wrong autoCommit behaviour

* formatting

* amended comment on denormalisation

* disabling failing assert for now
2019-04-08 11:56:14 +02:00
Robin Krom
a8b2d30b3d
Stdlib package (#229)
language: ship daml-stdlib as a package
2019-04-05 18:11:13 +02:00
Brian Healey
0205a1a932 #233 remove changelogs 2019-04-04 16:08:52 -04:00
Leonid Shlyapnikov
e655aae9e3 add fat dar support to DarReader, #109 2019-04-04 12:29:28 -04:00
Francesco Mazzoli
9c7357c7de update all references of old repo to new repo 2019-04-04 16:20:07 +02:00
Francesco Mazzoli
4d11dc8928 test divulgence in sandbox, see #164 2019-04-04 14:25:57 +02:00
Gabor Aranyossy
ab72aab696 fixes inconsistent help about Sandbox port
changing ports to use 6865 everywhere

no need for default ports

addressing missed outdated ports

changed more ports from 7600 to 6865

dealt with more 8080s
2019-04-04 13:03:48 +02:00
Digital Asset GmbH
05e691f558 open-sourcing daml 2019-04-04 09:33:38 +01:00