fix JSON API multikey stream
In the current state, the JSON API only handles multiple keys _from
different templates_. This makes it work for multiple keys from the same
template too.
Extracted from #7066 with the following changes:
- Use of a mutable `HashSet` to test for keys, because perf.
- Addition of a test at the JSON API level.
CHANGELOG_BEGIN
- [JSON API] Fix a bug where streaming multiple keys could silently
ignore some of the given keys.
CHANGELOG_END
* apply @cocreature's patch
https://gist.github.com/cocreature/d35367153a7331dc15cca4e5ea9098f0
* fix fmt
* Only include stakeholder contracts in result of query
This fixes a bug in the script service. We need to filter out divulged
contracts since this should behave exactly like the ACS endpoint on
the ledger API.
changelog_begin
changelog_end
* Update daml-lf/interpreter/src/main/scala/com/digitalasset/daml/lf/scenario/ScenarioLedger.scala
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Update daml-script/runner/src/main/scala/com/digitalasset/daml/lf/engine/script/LedgerInteraction.scala
Co-authored-by: Remy <remy.haemmerle@daml.com>
* fmt
changelog_begin
changelog_end
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Add queryContractId to DAML Script
I’ve switched between a few different naming options and in the end
settled on this one:
- `lookupContract`, not too bad but misses the connection to `query`
- `queryCid`, we don’t abbreviate this in other places in our API so I
don’t think we should here.
- `queryContractId`, makes the connection clear and no
abbreviation. We could also add `queryContractKey` later
changelog_begin
- [DAML Script] Add a `queryContractId` function for querying for a
contract with the given identifier.
changelog_end
* Fix test
changelog_begin
changelog_end
The executor may no longer be available. Fortunately we can avoid using
resources entirely in this case, and just acquire the channel the
old-fashioned way.
CHANGELOG_BEGIN
- [Scala Bindings] Fixed a bug in the ledger client which meant that the
client was not closed properly on application shutdown. You may have
seen some ``RejectedExecutionException`` errors in your logs if this
has affected you. We encourage all users of the `LedgerClient` to call
the ``close`` method explicitly to ensure it is closed at an
appropriate time in your application lifecycle.
CHANGELOG_END
An extraction script that makes the LF reading libraries (DAML LF archive decoding to an AST) usable in other Haskell projects.
This script once existed but is not available any more, so I have resurrected the script, adapted to current usage, and upgraded to the newest stackage LTS that can support it. These libraries _could_ be versioned manually to released SDKs and published if desirable (without making any guarantees about the stability, therefore prefixing the version with `0.`.
CHANGELOG_BEGIN
Tool to extract Haskell libraries for DAML LF archives for use in other Haskell projects
CHANGELOG_END
* reintroducing the main
* Introducing `ledger-service/http-json-testing`
* cleaning up
* Starting sandbox and json-api from perf-test main
changelog_begin
changelog_end
It turns out we were too aggressive in resetting the state. We only
want to reset in `submit` on successful commit and in `submitMustFail`
on failed commits. I’ve also split the reset in two parts, one that is
done at the beginning every time and one at the end. I don’t think it
really makes a difference but it matches what the scenario runner does
which at least makes me stop wondering if the difference matters.
fixes#7275fixes#7276
changelog_begin
changelog_end
* ledger-on-(memory + sql): Include logback as a test dependency.
Otherwise logs don't show up when we get failures, which makes debugging
a lot harder.
* kvutils: On unit test timeout, show a useful error.
* kvutils: Increase the streaming updates timeout in integration tests.
This is timing out every now and again on CI, and I can reproduce it
easily by running the tests in parallel.
CHANGELOG_BEGIN
CHANGELOG_END
I’ve also removed the restriction to only support going forward since
scenarios don’t have that restriction and it seems potentially useful
for tests.
factored out from #7264
changelog_begin
- [DAML Script] Add `passTime` helper to advance the time by the
given interval.
- [DAML Script] In DAML Studio, you can now set the time to the
past. This is not supported when running against a ledger.
changelog_end
Factored out from #7264
We have enough cases of scenarios that don’t actually do ledger
operations and need some hint for type inference. While adding type
annotations works, this is a bit easier at least for the migration and
some people might prefer it.
No test, since this seems a bit excessive here :)
changelog_begin
- [DAML Script] Add a `script` function to aid type inference. This
is the equivalent of `scenario`.
changelog_end
Factored out from #7264 and added a test. Just makes the migration a
tiny bit easier.
changelog_begin
- [DAML Script] Add `archiveCmd cid` as a convenience wrapper around
`exerciseCmd cid Archive`.
changelog_end
The new behavior here was only added in SDK 1.4.0 so the tests
fail (as expected) for older SDK releases. We only test against the
latest stable release on each PR so we didn’t catch this directly on CI.
changelog_begin
changelog_end
* java-quickstart replace scenarios by DAML Script
This replaces the scenarios in the java quickstart project by DAML
script test cases.
The following changes were required
* Replace `getParty` by `allocateParty`.
* Replace `create` and `exercise` by `createCmd` and `exerciseCmd`.
* Replace the pattern of `submit ... do create ...; exercise ...`
by `submit ... do createAndExerciseCmd ... ...`.
* Replace instances of `submit ... do fetch ...` by `query ... ...`.
To that end I've added the following helper function.
```
queryIou : Party -> ContractId Iou -> Optional Iou
```
changelog_begin
changelog_end
* Replace queryIou by FetchIou choice
changelog_begin
changelog_end
* Add comment explaining fetch choice
Addressing review comment
https://github.com/digital-asset/daml/pull/7183#discussion_r478918758
* Support dependencies in daml_test rule
Cherry-picked from https://github.com/digital-asset/daml/pull/7264/files
Renamed `dependencies` as `deps` to stick to Bazel conventions.
* Fix quickstart `daml_test` target
* fmt
* Quote deps in daml.yaml
To avoid issues with `:` in Windows paths.
* //docs:quickstart-java - use temp dir
This target repeatedly failed to build on Windows with
```
ERROR: D:/a/1/s/docs/BUILD.bazel:538:8: declared output
'docs/quickstart-java.tar.gz' was not created by genrule. This is
probably because the genrule actually didn't create this output, or
because the output was a directory and the genrule was run remotely
(note that only the contents of declared file outputs are copied from
genrules run remotely)
```
The genrule was operating in the current working directory and builds on
Windows are not sandboxed. Conflicting files in the current working
directory could cause this to fail.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Add a dummy OAuth2 authorization server
This is intended for use in the trigger service integration tests not
as an artifact that we ship to users.
changelog_begin
changelog_end
* Address review feedback
changelog_begin
changelog_end
* Refactor EnrichedTransaction
This PR prepares for work to interleave _authorization_ with _execution_, which has been an open issue for a long time: #132.
- Split computation of `BlindingInfo` and `FailedAuthorizations` into separate modules.
- Fixup callers: `Blinding.scala` and `ScenarioLedger.scala`, to do:
```
val failedAuthorizations =
AuthorizingTransaction.checkAuthFailures(authorization, tx)
val blindingInfo =
BlindingTransaction.calculateBlindingInfo(tx)
```
- Simplify code to remove `DontAuthorize` and `Authorization` types, which were only needed to restrict the combined _enrichment_ code from doing the authorization check when `blind`ing only is required. Now `Blinding.blind` calls just `calculateBlindingInfo`.
* address comments from Stefano
* Track source locations on Daml.Script.submit
Adds a `HasCallStack` constraint on `HasSubmit.submit` and
`HasSubmit.submitMustfail`.
We need to move the definition of `HasCallStack` to `GHC.Stack.Types` to
avoid a cyclic dependency between `DA.Stack` and `DA.Internal.LF`.
Alternatively, we could inline `HasCallStack` in `DA.Internal.LF` but it
seems preferable to use the alias on the signature of
`submit(MustFail)`.
We need to adapt the type of the builtins for scenario `submit` and
`submitMustfail`.
changelog_begin
changelog_end
* Test-case for scenario/script source locations
* Test second commit with intermediate definition
Addressing review comment
https://github.com/digital-asset/daml/pull/7252/files#r478518834
* Store the full callstack on submit
For forwards compatibility.
Addressing review comment
https://github.com/digital-asset/daml/pull/7252/files#r478520268
* Document unused location definition
Addressing review commit
https://github.com/digital-asset/daml/pull/7252/files#r478522373
* Make user facing `SrcLoc` zero based
Following LSP precedent rather than GHC
Addressing review comment
https://github.com/digital-asset/daml/pull/7252/files#r478529138
* Add comment for backwards compatibility code
Addressing review comment
https://github.com/digital-asset/daml/pull/7252/files#r478529138
* fmt
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* kvutils: Factor out a base class for ledger exporter implementations.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Use piped streams in LedgerDataExporterSpecBase.
* kvutils: Rename "FileBased…" to "SerializationBasedLedgerDataExporter".
* kvutils: Wrap Deserialization in a class for polymorphism purposes.
* kvutils: Pull out submission info variables in the ledger export test.
* kvutils: Inline (De)Serialization into the exporter.
* kvutils: Replace usage of Deserialization with LedgerDataImporter.
* kvutils: Don't synchronize on making an InMemorySubmissionAggregator.
* kvutils: Pass a SubmissionInfo to LedgerDataExporter.
* kvutils: Reflow a long string to make it readable.
It turns out that if you give the CSV reporter a non-existent directory,
it crashes. I did not expect this.
This constructs the directory so you don't have to worry about that.
CHANGELOG_BEGIN
CHANGELOG_END
This is the same technique as `DerivativeGauge` from the metrics
library, but with less code, because Scala is prettier than Java.
CHANGELOG_BEGIN
CHANGELOG_END
to distributed ledgers with differing ledger ids. This
accommodates distributed vDaml setups with participants with
matching ledger ids.
changelog_begin
changelog_end
* Remove authentication service from trigger service
This rips out the integration with ref-ledger-authenticator and
together with that the weird http basic auth which doesn’t actually
work for ref-ledger-authenticator outside of its testing mode and also
includes some very sketchy stuff like storing user
passwords (encrypted but still).
While the exact details of the new approach towards auth are still not
completely clear, it is clear that it is going to be sufficiently
different from what we did before that starting from a clean slate is
easier.
changelog_begin
- [Trigger Service] Parties are now specified in request bodies as
opposed to via HTTP Basic auth. This is done in preparation for
running the trigger service against authenticated ledgers.
changelog_end
* Fix integration tests
changelog_begin
changelog_end
* Deprecate noop `--application-id`
changelog_begin
[JSON API]
Hiding and deprecating `--application-id` command-line option. JSON API never used it.
It is required to instantiate LedgerClientConfiguration and was not used for any command submission.
JSON API uses Application ID specified in the JWT. See #7162
changelog_end
* removing further usage of noop applicationId
* a bit of explanation what this is for
Apparently `[[]]` links don't work to external resources. I wanted to
turn the first referencee into a link too but that contains too many
weird characters.
CHANGELOG_BEGIN
CHANGELOG_END
@leo-da is taking care of 1.5.0-snapshot.20200825.5071.0.d33e130f (#7233), so they get pushed back to the end of the line.
Please do not merge this before #7233.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
* Clarify how to start a test ledger, and run json api with & w/o query storage
* Remove the json-api --help output and replace with a note.
* Update JSON API Authorization/Authentication sections
* Minor improvements to query section
* CHANGELOG_BEGIN
[Docs] Improve the JSON API documentation, clarify the common json-api options, and update the authorization/authentication section.
CHANGELOG_END
* Note why you would specify a Command ID in the JSON API
* Clarify how the Authorization token works
* Add warning to always run the JSON API behind a reverse proxy
* Apply suggestions from code review
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Change how we refer to the different access tokens used by the JSON API
* applicationId doesn't need to match --application-id
* remove reference to planned jwt library
* Update party allocation instructions/explanation
* Remove IAM references from Ledger API Auth
* Stop lying about secrets
* Auth shouldn't be party-specific
* Needed more stars
* Apply suggestions from code review
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Update Ledger API access token description
* Additional clarifications on the two types of tokens
* Note that each request to the JSON API can only come from a single party
* Apply suggestions from code review
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Clarify getting access tokens from your ledger operator
* Add link to JWT format in the sandbox
* Fix inconsistent title level
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging.
@leo-da is in charge of this release.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
A `Struct[+X]` is a list of pair `(FieldName, X)` ordered by first
component.
We use this data-structure to represent TStruct LF type.
It will be used in upcomming PRs to sorted fields in Struct values.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Factor out ledger dump timers into a class.
* kvutils: Use `Paths` in the integrity checkers.
Strings give me the heebie-jeebies.
CHANGELOG_BEGIN
CHANGELOG_END
The similar scenario tests are also disabled on Windows and the script
tests are flaky on Windows due to timeouts.
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Enable DAML Script service by default and document it
fixes#7193
changelog_begin
- [DAML Script] DAML Scripts are now run in DAML Studio just like
scenarios. The functionality for inspecting the results is
identical. If you encounter issues, please report them.
changelog_end
* fix double "is identical"
changelog_begin
changelog_end
* kvutils: Add `override` annotations for the exporters.
* kvutils: Test the ledger export (v2).
* kvutils: Mark integrity tests as small.
* kvutils: Add assertions to the FileBasedLedgerDataExporter.
Just confirming my understanding of how this works. Verified with the
Ledger API Test Tool and some proprietary code.
* kvutils: Move the definition of `CorrelationId` to the package file.
* kvutils: Pass a submission write set through the batch pipeline.
This allows for less mutability all over the place.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Move export finishing into its own type.
* kvutils: Move some nested types upwards.
* kvutils: Split Deserialization from Serialization.
* kvutils: Extract out common behavior in (De)Serialization.
* kvutils: Don't use a singleton for LedgerDataExporter.
Instead, construct it once.
* kvutils: Make sure we close the export file when we're done.
* kvutils: Simplify `Debug` in the same manner as `LedgerDataExport`.
* kvutils: Fix a test name.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* kvutils: Remove backticks around "export".
* kvutils: Move the test `BatchedSubmissionValidator#apply` into the test.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* [KVL-222] Add participant id to index metadata dump
changelog_begin
changelog_end
* Test SqlLedger participant id initialization
* Test JdbcIndexer participant id initialization
* Make RecoveringIndexerSpec final and remove unused trait
* Normalize struct field order in the LF spec.
This commit changes the LF spec to ensure that struct field order is ignored whenever reasonable.
* Rename "type synonym resolution" to "type normalization", and change the rule for structs to also normalize the order of fields.
* Make the struct type "well-formed" only if the struct fields are sorted by field name. (This matches the way "well-formed" is only used for normalized types. E.g. type synonyms are not considered "well-formed" by the spec. This nomenclature should possibly be fixed in a separate PR.)
* Change the struct con expression type rule accordingly.
* Change the definition of a struct value. A struct value is now a struct expression where all the field expressions are values, and all the field names are sorted.
* Change the evaluation rule for struct expressions to also sort the fields by field name in the process.
As far as I know, this spec matches the behavior of the LF engine. I also tried to make the treatment of struct field order in the types match the treatment of struct field order in expressions (i.e. normalized structs always have sorted fields, but non-normalized structs do not).
changelog_begin
changelog_end
* Apply suggestions from code review
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Update daml-lf/spec/daml-lf-1.rst
Co-authored-by: Remy <remy.haemmerle@daml.com>