Commit Graph

5076 Commits

Author SHA1 Message Date
Remy
ada8ba033b
LF: ad-hoc data struct to store LF Struct like data (#7220)
A `Struct[+X]` is a list of pair `(FieldName, X)` ordered by first
component.

We use this data-structure to represent TStruct LF type.

It will be used in upcomming PRs to sorted fields in Struct values.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-26 12:18:18 +02:00
Moritz Kiefer
48e05e25e2
Fix name of get_gh_auth_header (#7239)
See
b812f09da9/ci/bash-lib.yml (L12),
this obviously failed on CI.

changelog_begin
changelog_end
2020-08-26 12:16:47 +02:00
Samir Talwar
19a7eadc57
kvutils: Use classes in the integrity checker code. (#7238)
* kvutils: Factor out ledger dump timers into a class.

* kvutils: Use `Paths` in the integrity checkers.

Strings give me the heebie-jeebies.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-26 09:08:26 +00:00
Andreas Herrmann
b812f09da9
Disable lsp-tests script tests on Windows (#7235)
The similar scenario tests are also disabled on Windows and the script
tests are flaky on Windows due to timeouts.

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-08-26 07:56:12 +00:00
Samir Talwar
b707bc825d
kvutils: Simpler constructors for common caching classes. (#7231)
* kvutils: Use `CachingDamlLedgerStateReader.apply`.

* kvutils: Simplify the constructor of CachingCommitStrategy.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-26 07:34:01 +00:00
Moritz Kiefer
d33e130f47
Enable DAML Script service by default and document it (#7229)
* Enable DAML Script service by default and document it

fixes #7193

changelog_begin

- [DAML Script] DAML Scripts are now run in DAML Studio just like
  scenarios. The functionality for inspecting the results is
  identical. If you encounter issues, please report them.

changelog_end

* fix double "is identical"

changelog_begin
changelog_end
2020-08-25 19:53:59 +02:00
Samir Talwar
aafb4a27f9
kvutils: Simplify exporting by pushing the mutations outwards. (#7215)
* kvutils: Add `override` annotations for the exporters.

* kvutils: Test the ledger export (v2).

* kvutils: Mark integrity tests as small.

* kvutils: Add assertions to the FileBasedLedgerDataExporter.

Just confirming my understanding of how this works. Verified with the
Ledger API Test Tool and some proprietary code.

* kvutils: Move the definition of `CorrelationId` to the package file.

* kvutils: Pass a submission write set through the batch pipeline.

This allows for less mutability all over the place.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Move export finishing into its own type.

* kvutils: Move some nested types upwards.

* kvutils: Split Deserialization from Serialization.

* kvutils: Extract out common behavior in (De)Serialization.

* kvutils: Don't use a singleton for LedgerDataExporter.

Instead, construct it once.

* kvutils: Make sure we close the export file when we're done.

* kvutils: Simplify `Debug` in the same manner as `LedgerDataExport`.

* kvutils: Fix a test name.

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>

* kvutils: Remove backticks around "export".

* kvutils: Move the test `BatchedSubmissionValidator#apply` into the test.

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
2020-08-25 17:52:59 +00:00
Remy
4e265f0c1c
Speedy: Make the fields 'fields' from SStruct immutable. (#7227)
Change the type of SStruct.fields from Array to ImmArray.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-25 18:03:33 +02:00
Remy
89770cf1c1
Speedy: make the field names from SRecord immutable. (#7225)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-25 17:02:27 +02:00
Stefano Baghino
b33bfda7fa
[KVL-222] Add participant id to index metadata dump (#7224)
* [KVL-222] Add participant id to index metadata dump

changelog_begin
changelog_end

* Test SqlLedger participant id initialization

* Test JdbcIndexer participant id initialization

* Make RecoveringIndexerSpec final and remove unused trait
2020-08-25 17:02:02 +02:00
Sofia Faro
3429ec24cc
Normalize struct field order in the LF spec. (#7228)
* Normalize struct field order in the LF spec.

This commit changes the LF spec to ensure that struct field order is ignored whenever reasonable.

* Rename "type synonym resolution" to "type normalization", and change the rule for structs to also normalize the order of fields.

* Make the struct type "well-formed" only if the struct fields are sorted by field name. (This matches the way "well-formed" is only used for normalized types. E.g. type synonyms are not considered "well-formed" by the spec. This nomenclature should possibly be fixed in a separate PR.)

* Change the struct con expression type rule accordingly.

* Change the definition of a struct value. A struct value is now a struct expression where all the field expressions are values, and all the field names are sorted.

* Change the evaluation rule for struct expressions to also sort the fields by field name in the process.

As far as I know, this spec matches the behavior of the LF engine. I also tried to make the treatment of struct field order in the types match the treatment of struct field order in expressions (i.e. normalized structs always have sorted fields, but non-normalized structs do not).

changelog_begin
changelog_end

* Apply suggestions from code review

Co-authored-by: Remy <remy.haemmerle@daml.com>

* Update daml-lf/spec/daml-lf-1.rst

Co-authored-by: Remy <remy.haemmerle@daml.com>
2020-08-25 14:32:03 +00:00
Samir Talwar
c08de213f9
kvutils: Fix flaky runs of BatchingQueueSpec. (#7223)
* kvutils: Use `eventually` in BatchingQueueSpec to fix a flaky test.

We cannot assume that the queue will shut down faster than we can check
its state.

* kvutils: Increase timeouts in BatchingQueueSpec to fix flakiness.

50 milliseconds is pretty fast when tests are running in parallel.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-25 14:21:38 +00:00
Moritz Kiefer
3058af8286
Support HasCallStack constraints (#7217)
This is primarily intended to improve error reporting in DAML Script,
in particular, https://github.com/digital-asset/daml/issues/7192 which
is a blocker for replacing scenarios by DAML Script.

However, I think it is also a very reasonable feature in general.

changelog_begin
changelog_end
2020-08-25 08:54:21 +00:00
Moritz Kiefer
c3d758a2d3
Ignore failed ipcrm calls (#7218)
* Ignore failed ipcrm calls

changelog_begin
changelog_end

* Update ci/clear-shared-segments-macos.yml

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-08-25 08:39:11 +00:00
Stefano Baghino
0919221d04
Fix off-by-one error in logging (#7211)
Furthermore, does not mention PostgreSQL in the message, as
the HikariJdbcConnectionProvider is shared across all our
usages of RDBMSs (including H2 and Sqlite).

changelog_begin
changelog_end
2020-08-25 08:23:42 +00:00
Stefano Baghino
57a47347da
Add run mode to kvutils to dump index metadata and exit (#7213)
* Add run mode to kvutils to dump index metadata and exit

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/7213#discussion_r475843480

* Address https://github.com/digital-asset/daml/pull/7213#discussion_r475847418
2020-08-25 10:09:05 +02:00
Samir Talwar
6b2fd4bfaa
participant-state: Run tests in parallel. (#7214)
* kvutils: Run tests in parallel.

* participant-state: Run tests in parallel, and fix the paths.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-25 07:44:44 +00:00
Gary Verhaegen
5f897e699f
reenable Windows cache (#7216)
All machines have been reset following #7212.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-24 20:02:08 +00:00
Moritz Kiefer
5936644970
Bump windows cache (#7212)
changelog_begin
changelog_end
2020-08-24 18:19:28 +02:00
Leonid Shlyapnikov
3ecfcc918c
Adding package-max-inbound-message-size (#7147)
* Adding `package-max-inbound-message-size`

this is to allow separate configuration settings for command submission
and package management ledger clients

* Fixing formatting

* Updating docs

changelog_begin

[JSON API] Adding `--package-max-inbound-message-size` command line option.
Optional max inbound message size in bytes used for uploading and downloading package updates. Defaults to the `max-inbound-message-size` setting.

changelog_end

* Addressing code review comments
2020-08-24 14:55:55 +00:00
Gary Verhaegen
f54e193df6
add visibleByKey (#7146)
Adds a Boolean check for the existence of an active contract with a
given key (that is visible with the current set of authorizations), and
explain `lookupByKey` in terms of it to hopefully clarify the relationship
between `lookupByKey` and `fetchByKey`.

Fixes #7143; see that for more information.

CHANGELOG_BEGIN

- [DAML Standard Library] Added a new function `visibleByKey`
  which, given a contract key, returns True if the current context can
  see an active contract with that key.

CHANGELOG_END
2020-08-24 15:04:05 +02:00
Moritz Kiefer
c997f5d145
Fix handling of transient contracts in extractor (#7210)
* Fix handling of transient contracts in extractor

fixes #7201

As described in the issue, the problem was that we completely ignored
the order of events and always processed exercises before creates
which falls apart as soon as they are in the same transaction.

I still think that in the current state doing creates before exercises
would be fine but this seems like the kind of assumption that breaks
very quickly during refactoring (e.g. let’s say you start creating
foreign key references) so instead this PR fixes this properly by
iterating over the transaction tree in pre-order.

This turns out to be a bit more complex than I expected it to be since
we filter the tree on the client side and afterwards we have lost
enough information so that we cannot traverse the tree instead
anymore. Therefore, the “tree” type in extractor now contains a list
of events in pre-order and a comment that it’s not actually a valid
tree or subtransaction.

changelog_begin

- [Extractor] Fix a bug that resulted in transient contracts (created
  and archived within the same transaction) to not be displayed as
  archived. See #7201 for details.

changelog_end

* Update extractor/src/main/scala/com/digitalasset/extractor/ledger/types/TransactionTree.scala

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>

* Suffix with template name

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
2020-08-24 14:43:24 +02:00
Remy
0e9616f784
LF: move the type ordering form interpreter to language package (#7203)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-24 07:18:41 +00:00
Remy
7f60560593
LF: make QualifiedName and Identifier Ordered (#7199)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-21 17:39:51 +02:00
Andreas Herrmann
037cf11710
Add :json command to DAML REPL (#7202)
* REPL test `:json` command

changelog_begin
changelog_end

* Implement :json command in DAML REPL

changelog_begin
- [DAML REPL] You can now convert DAML expressions to JSON in the DAML
  REPL using the meta-command ``:json`` for example ``:json [1, 2, 3]``.
changelog_end

* Extend DAML REPL documentation

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-08-21 14:53:06 +00:00
Stefano Baghino
ac86a9c5a3
Refactor SQLLedger initialization routine (#7200)
* Refactor SQLLedger initialization routine

Small refactoring to make initialization a bit more readable. Performed
while moving forward with the addition of the participant identifier to
the parameters table (so a few minor details have leaked into this PR).

changelog_begin
changelog_end

* Fix compilation errors

* Address https://github.com/digital-asset/daml/pull/7200#discussion_r474630880

* Fix test, lower test logging noise
2020-08-21 15:29:49 +02:00
Remy
6371f51930
PartialTransaction: replace nested Eithers by a custom ADT (#7191)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-21 10:16:36 +02:00
Stephen Compall
2369a3bbae
use sandbox-style flyway migrations for trigger service (#7092)
* depend on flyway in trigger service; duplicate FlywayMigrations from sandbox

* bring in postgres/h2 detection, other data-source elements needed; use scalalogging

* use DataSource instead of DriverManager for DbTriggerDao, pass to flyway too

- initialization order isn't quite right yet

* hardcode postgres in triggers flyway path

* found an init order that doesn't fail in tests

* better pools for production, and don't forget to close them

* move most init SQL to first migration file

* make logHandler implicit again

- seems to have been an accidental implication of #6334

* move remaining create to V1__Init.sql

* use Prod poolSize

* prove that PostStop signals are never seen (in tests)

* move all the shutdown to Stop message

* remove dead code in DbFlywayMigrations

* always "initialize" database

CHANGELOG_BEGIN
- [Trigger Service] The database format has changed to allow migrations in
  future releases.  Databases are always initialized or migrated to the current
  version on start, so use of ``--init-db`` is no longer required.
  See `issue #7092 <https://github.com/digital-asset/daml/pull/7092>`__.
CHANGELOG_END

* it should always be obvious when there is no escape

* remove logs from experiments

* remove receiveSignal PostStop

* remove unused imports
2020-08-20 18:52:37 +00:00
Gary Verhaegen
6d1adee92f
push script-runner and trigger-runner to Artifactory (#7196)
I have created the corresponding user and repositories on Artifactory,
and tested the `curl` command manually. I'll add the corresponding
credentials to Azure once this is approved.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-20 19:11:27 +02:00
Andreas Herrmann
9f2367f6cc
IDE distinguish scripts and scenarios (#7197)
* IDE distinguish scripts and scenarios

The code lens above DAML script definitions now reads `Script results`
instead of `Scenario results`.

This also adds DAML Script test cases to the lsp-tests.

The functionality for testing scenario and script code-lenses in the
`Utils` module is identical. However, this adds dedicated definitions
for scripts to clearly distinguish the scenario and script test-cases.

changelog_begin
changelog_end

* Add sum type to distinguish Scenario and Script

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-08-20 16:36:23 +00:00
Gary Verhaegen
2871ab7a11
add Windows tips to release instructions (#7180)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-20 17:26:21 +02:00
Samir Talwar
30564a7c90
ledger-api-client: Add test cases for LedgerClient. (#7195)
* ledger-api-client: Add integration tests for the simple stuff.

* sandbox-common: Make `SandboxFixtureWithAuth` a mixin.

This makes it useful with `SandboxNextFixture` as well as
`SandboxFixture`.

Also, add types to non-private fields and methods, and make more fields
protected rather than public.

* ledger-api-client: Add tests to make sure the token is passed through.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox-common: Tokens are for auth, not auth.
2020-08-20 15:00:32 +00:00
Remy
ba74146c10
Speedy: shortcup failure in recursive data structure equality (#7083)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-20 16:15:11 +02:00
Stefano Baghino
62fcd16de2
Factor parameters table access out of JdbcLedgerDao (#7194)
* Factor parameters table access out of JdbcLedgerDao

changelog_begin
changelog_end

* Make ParametersTable package private
2020-08-20 15:46:51 +02:00
Remy
17abe2f4cd
DAML for PostgreSQL: prevent dev version of DAML-LF (#7176)
Add a hidden flag to enable dev mode for testing purpose only.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-20 12:15:38 +02:00
Samir Talwar
5a228354bc
sandbox: Use a different H2 database for every test. (#7184)
* sandbox: Use a different H2 database for every test.

Previously tests would share an index database, but not a ledger
database, if not explicitly overridden. This is _very bad_.

Fortunately all tests did specify a backend explicitly, but it shouldn't
be necessary.

I caught this while writing a new test using `SandboxNextFixture`.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox-common: Simplify SandboxBackend a little.
2020-08-20 10:03:29 +00:00
Andreas Herrmann
3e6ffcad4b
Refactor IdeClient.submit - avoid Future.fromTry (#7179)
The code was using `result: Try` combined with `Future.fromTry` to
signal success or failure. However, the code also used a `try { }
finally { }` block to ensure cleanup. Wrapping the whole block in a
`Future { }` and simply throwing exceptions instead of setting `result =
Failure( )` has the same effect, but avoids the nesting of `try { }` and
`Future.fromTry`.

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-08-20 09:01:12 +00:00
Moritz Kiefer
255832cefc
Fix failed contract lookups in scenario runner (#7190)
lookupContract catches the exception which results in us continuing
and eventually running into a null-pointer exception.

This is another reason why we should enable the NonUnitStatements wart
but I’ll leave that for a separate PR (last I tried the scenario
service resulted in a ton of false positives).

fixes #7185

changelog_begin
changelog_end
2020-08-20 09:49:03 +02:00
Remy
cc6e8fa901
Engine: submit fail nicely when transaction output cannot be serialized (#7189)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-20 09:20:50 +02:00
Gary Verhaegen
29dfd2d227
fix typo in date docstring (#7187)
As reported by Gyorgy Balazsi on [the forum].

[the forum]: https://discuss.daml.com/t/day-parameter-overflow-for-the-date-function-simple-leap-year-handling/1068/7?u=gary_verhaegen

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-20 09:18:18 +02:00
anthonylusardi-da
3d067b5fa9
Fix some screen widths not showing the content menu collapse/expand icon (#7138)
* Fix some screen widths/zoom levels not showing the content menu collapse/expand icon

* CHANGELOG_BEGIN
[Docs] Fix some screen widths not showing the content menu collapse/expand icon
CHANGELOG_END
2020-08-19 16:37:47 -04:00
azure-pipelines[bot]
6dd542c2be
update compat versions for 1.5.0-snapshot.20200818.5027.0.1b33d374 (#7177)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2020-08-19 21:42:17 +02:00
Remy
e54c182032
Engine: move check of valide packages inside MutableCompiledPackages (#7186)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-19 19:37:53 +02:00
Gary Verhaegen
94f2a10e70
fix auto release pr message (#7178)
Because the build for the automated release PR is triggered "manually"
by the Azure cron (through the GH API), as far as the build of that PR
is concerned, this is a random commit build, not a PR build, and thus it
doesn't have a PR number.

This works around that by getting the corresponding PR number from the
GH API.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-19 13:58:56 +02:00
Samir Talwar
c8b10091db
participant-integration-api: Never use a delay of zero. (#7175)
* participant-integration-api: Never use a delay of zero.

If `akka.pattern.after` is passed a delay of zero, it will execute the
body synchronously, potentially leading to a stack overflow error.

This error was observed in tests.

CHANGELOG_BEGIN
CHANGELOG_END

* timer-utils: Add tests for Delayed.Future.

* timer-utils: Add tests for RetryStrategy.

* timer-utils: Remove duplication in RetryStrategy tests.

* timer-utils: Allow for more wiggle room in the RetryStrategy tests.

* timer-utils: Fail after retrying the correct number of times.

* timer-utils: Ensure we don't overflow the stack in RetryStrategy.

* timer-utils: Reject a negative number of retry attempts.
2020-08-19 10:43:31 +00:00
azure-pipelines[bot]
f69006264e
rotate release duty after 1.5.0-snapshot.20200818.5027.0.1b33d374 (#7174)
@garyverhaegen-da is taking care of 1.5.0-snapshot.20200818.5027.0.1b33d374 (#7173), so they get pushed back to the end of the line.

Please do not merge this before #7173.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2020-08-19 12:36:45 +02:00
Samir Talwar
d0ffc2c8c5
participant-integration-api: Factor out similar behavior in the admin service implementations. (#7171)
* participant-integration-api: Factor out `pollUntilPersisted`.

CHANGELOG_BEGIN
CHANGELOG_END

* participant-integration-api: Flatten `flatMap`s in admin services.

* participant-integration-api: Use the `FutureConverters` implicits.

* participant-integration-api: Inline `waitForEntry` again.

* participant-integration-api: Commas are great.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* participant-integration-api: Pass the EC to `pollUntilPersisted`.

* participant-integration-api: Extract out more into SynchronousResponse.

* participant-state-index: Stream entries from an optional offset.

Makes them consistent with configuration entries.

* participant-integration-api: Move the ledger end into the strategy.

* participant-integration-api: Use SynchronousResponse everywhere.

* participant-integration-api: Pass inputs to SynchronousResponse#submit.

* participant-integration-api: Add docs to SynchronousResponse.

* participant-integration-api: Add a changelog entry.

CHANGELOG_BEGIN
- [Ledger API] The ConfigurationManagementService will now use the same
  description as other services in case of certain errors. The error
  status codes have not changed.
CHANGELOG_END

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-08-19 10:24:48 +00:00
Andreas Herrmann
a080b47e34
Reset the machine on every submit in IdeClient (#7172)
* Test-case for operation after submitMustFail

Currently this triggers
```
BErrorClient (ClientIOError (GRPCIOBadStatusCode StatusUnknown (StatusDetails {unStatusDetails = ""})))
```

changelog_begin
changelog_end

* Reset the machine after every submit in IdeClient

This ensures that the machine is not stuck in a brocken state after a
`submitMustFail` completed.

changelog_begin
changelog_end

* Machine cleanup in finally block

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-08-19 10:26:15 +02:00
azure-pipelines[bot]
ed73352843
release 1.5.0-snapshot.20200818.5027.0.1b33d374 (#7173)
This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging.

@garyverhaegen-da is in charge of this release.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2020-08-19 10:09:16 +02:00
Remy
1b33d3742a
Revert "Sandbox: add a development mode (#7127)" (#7168)
This reverts commit 87a0ee9383.

CHANGELOG_BEGIN

WARNING remove the changelog entry from 87a0ee93 and starting by :
 "[Sandbox] By default Sandbox rejects the development versions of ..."

CHANGELOG_END
2020-08-18 19:17:56 +02:00