Commit Graph

3346 Commits

Author SHA1 Message Date
Gary Verhaegen
80652bd51f
report-std-change: handle GitHub errors (#4814)
There have been a few GitHub glitches last week that resulted in a few
commits on master not being associated with a PR (though they really
were created from merging a PR, and the correct PR number is in their
title).

This makes the report script crash on not finding the PR, so this PR
fixes that. And a comment.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-05 10:06:58 +01:00
mergify[bot]
9bb80e552c
daml2ts: Keep workspace package.json intact (#4820)
Keep all fields in the workspace `package.json` given to `daml2ts`
instead of only `private` and `workspaces`.

This also allows for removing the `sed` hack in the `build-and-lint`
test and use the `resolution` field of the workspace `package.json`
to point to our local versions of `@daml/types` and `@daml/ledger`.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-05 00:45:11 +00:00
Shayne Fletcher
0a0786d596
Set test size to large (#4817)
changelog_begin
changelog_end
2020-03-04 15:59:23 -05:00
mergify[bot]
cb1395e923
Remove damlc migrate (#4816)
* Remove damlc migrate

``damlc migrate`` hasn’t worked for quite a while and we emitted a
warning for months so given that we don’t have plans to make it work
again in the near future, I think it does more harm than good to keep
it around.

changelog_begin

- [DAML Compiler] After being deprecated for a while the ``damlc
  migrate`` command has now been removed. See
  https://docs.daml.com/upgrade/ for up to date documentation
  on model upgrades.

changelog_end

fixes #3704 (by removing the tests 😇)

* yeah the windows cache is once again broken \o/

* Revert "yeah the windows cache is once again broken \o/"

This reverts commit 38d7877aa4.
2020-03-04 20:36:48 +00:00
Leonid Shlyapnikov
f9057bea19
handle dual LiveBegin case (#4815)
* make append of two liveness markers proper with respect to offsets

- never happens in a real stream, but satisfying some random tests
- test randomly harder

CHANGELOG_BEGIN
CHANGELOG_END

* move Monoid and Semigroup instances for Steps to main code

- suggested by @leo-da; thanks

* stray empty InsertDeleteStep
2020-03-04 14:59:26 -05:00
Shayne Fletcher
48595f774d
daml2ts : Generate a package of each DALF (#4799)
* daml2ts : generate one package per dalf
2020-03-04 14:51:40 -05:00
Moritz Kiefer
c866ac5132
Speed up unmangleIdentifier (#4810)
* Speed up unmangleIdentifier

On my (admittedly not super scientific) benchmark, this brings the
time used to convert from the low-level proto Haskell AST to the
high-level AST from 20s down to 16s on a certain DALF that we are all
too familiar with.

changelog_begin
changelog_end

* address review comments
2020-03-04 19:48:03 +01:00
mergify[bot]
a40190d695
minor cleanup, using Option.exists instead of cata. (#4812)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-04 18:10:04 +00:00
mergify[bot]
c001b4f22c
test that ContractStreamStep#append preserves data sensibly (#4809)
* test that ContractStreamStep#append preserves data sensibly

CHANGELOG_BEGIN
CHANGELOG_END

* (C) header

* test more general associativity
2020-03-04 17:59:02 +00:00
Moritz Kiefer
c6e3bf5399
Only unmangle once in decodeValueName (#4813)
The `decodeValueName` code is rather confusing. It calls things
unmangled that are mangled and the other way around.

Furthermore, it unmangles twice, once in `decodeNameString` and once
directly in `decodeValueName`. The code claims that this is a
compatiblity hack but unless someone can explain to me what exactly is
failing here or CI fails, I would prefer to just kill this.

changelog_begin
changelog_end
2020-03-04 17:29:55 +00:00
Gary Verhaegen
c2240df083
change standard change report time (#4804)
In the current setup, the report generation starts at 4. However, the
daily killing of all machines also starts at 4, giving the report little
chance of ever finishing.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-04 18:22:32 +01:00
Moritz Kiefer
a928402050
Make debug a bit more lazy (#4807)
* Make `debug` a bit more lazy

Previously `debug x >>= f` would print the debug statement when it got
evaluated rather than when the monadic execution got executed. This is
rather confusing. Now we explicitly make it lazy by wrapping the trace
statement in a lambda passed to >>=.

changelog_begin

- [DAML Standard Library] Fix a bug where ``debug`` printed the trace
  statement before the action got executed. Note that this means that
  ``debug`` now has a slightly more restrictive type. You can use
  ``trace`` directly if this causes problems.

changelog_end

* Update compiler/damlc/tests/src/DA/Test/ShakeIdeClient.hs

Co-Authored-By: Martin Huschenbett <martin.huschenbett@posteo.me>

Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
2020-03-04 17:12:24 +00:00
Moritz Kiefer
569a8a2192
Address upgrade doc review comments (#4811)
This PR addresses the comments from @hurryabit on #4750.

changelog_begin
changelog_end
2020-03-04 18:09:45 +01:00
nickchapman-da
a64610925c
Extend the documentation for DAML upgrade to shows project setup, build and deploy steps. (#4794)
changelog_begin
changelog_end
2020-03-04 15:23:14 +00:00
Samir Talwar
85998f8f06
Sandbox-Next: Add the ResetService. (#4802)
* sandbox-next: Make the Runner a real ResourceOwner.

* sandbox: Don't construct the ResetService twice.

* sandbox: Inline and simplify methods in StandaloneApiServer.

* resources: Define a `ResettableResource`, which can be `reset()`.

`reset()` releases the resource, performs an optional reset operation,
and then re-acquires it, binding it to the same variable.

* resources: Pass the resource value into the reset operation.

* sandbox: Fix warnings in `TestCommands`.

* sandbox-next: Add the ResetService.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Make sure the SandboxResetService resets asynchronously.

It was being too clever and negating its own asynchronous behavior.

* sandbox-next: Forbid no seeding.

This double negative is really hard to phrase well.

* sandbox-next: Implement ResetService for a persistent ledger.

* sandbox: Delete the comment heading StandaloneIndexerServer.

It's no longer meaningful.

* sandbox-next: No need to wrap the SandboxResetService in an owner.

* sandbox-next: Bump the ResetService test timeouts.

It looks like it's definitely slower than on Sandbox Classic™. Gonna
look into this as part of future work.

* Revert to previous asynchronous reset behavior

Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2020-03-04 14:40:35 +00:00
nickchapman-da
3239a810eb
Snapshot release (#4806)
changelog_begin
changelog_end
2020-03-04 14:11:13 +00:00
Robert Autenrieth
98095cf80d
Clean up command deduplication (#4801)
* Always return error on duplicate submissions

* Remove unnecessary submission information

Now that duplicate submissions always return an error,
we don't need to store the original submission result.

CHANGELOG_BEGIN
CHANGELOG_END

* Rename ttl to deduplicationTime/deduplicateUntil

* Store absolute deduplicateUntil in domain commands

* Fix my own initials

* Remove CommandDeduplicationEntry

Instead, use CommandDeduplicationResult everywhere,
removing the extra layer.
2020-03-04 14:08:08 +01:00
Gary Verhaegen
6a1c75cfd4
fix Windows signing (maybe) (#4753)
During the latest attempt at making a snapshot release (#4749),
everything seemingly went well except for the step of signing the
Windows installer.

Based on a very obscure erorr message (Access Denied) and a bit of
Google search, my current hypothesis is that the signing fails because
the artifact produced by Bazel is read-only. This was not an issue in
the previous setup because we were getting the installer from an Azure
internal download between different jobs. We are now getting the binary
directly from Bazel.

This also adds a `set -e` to the relevant Bash snippet, because ideally
they should always have one.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-04 13:18:49 +01:00
Moritz Kiefer
351091a76e
Only log message about duplicate package uploads in debug level (#4803)
It is basically impossible to not hit this all the time if you upload
more than one package so issuing a warning is a bit confusing.

changelog_begin

- [Sandbox] The warning about duplicate package uploads is no longer
  emitted by default. You can enable them by passing
  ``--log-level=debug``.

changelog_end
2020-03-04 10:36:20 +00:00
Remy
267671c340
[engine] track usage of getTime (#4790)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-04 10:17:55 +01:00
Leonid Shlyapnikov
e80cda53ab
Fix for unknown template ID received from gRPC streaming API (#4800)
* Filter out contracts received from gRPC stream, if template IDs is unknown (pending package reload).

* minor cleanup

CHANGELOG_BEGIN

[JSON API - Experimental] Fix ``java.util.NoSuchElementException: key not found: TemplateId(xxx,yyy,zzz)`` error in the log for query and fetch endpoints. See #4761

CHANGELOG_END
2020-03-04 08:21:06 +00:00
Samir Talwar
06b3c26bfe
Sandbox: Refactoring and cleanup around FlywayMigrations and JdbcIndexerFactory. (#4797)
* sandbox: Return `Future[Unit]` from migrations rather than awaiting.

I've removed the explicit error-handling, because this will be
propagated and handled at the top level.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Pass the JDBC URL into the JdbcIndexerFactory constructor.

* sandbox: Replace the JdbcIndexerFactory's `InitStatus` with two classes.

The `asInstanceOf` conversions put me off.

* sandbox: Stop passing around the ledger ID in JdbcIndexerFactory.

* sandbox: Remove the indexer `asyncTolerance`; it's no longer used.
2020-03-04 08:13:19 +00:00
fabiotudone-da
85a10e5113
Ensure NodeInfo derived properties informeesOfNode and requireAuthorizers always return (#4554)
Fail with a more explanatory message if `Fetch.actingParties` is `None`

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-04 08:18:05 +01:00
Leonid Shlyapnikov
a3228bcea8
A few more map query tests (#4798)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-03 18:02:58 +00:00
Samir Talwar
a73b7459a7
resources: Resource.unit, as an alias for Resource.successful(()). (#4786)
* resources: `Resource.unit`, as an alias for `Resource.successful(())`.

CHANGELOG_BEGIN
CHANGELOG_END

* resources: Add `ResourceOwner.unit`.
2020-03-03 16:47:54 +00:00
Gerolf Seitz
536e793a44
Don't return witnessed contracts in ActiveContractsService (#4791)
The change to `EventFilter` and to the query in `JdbcLedgerDao` are
"duplicate work", but we need the change in EventFilter for the
InMemoryLedger, and the change in JdbcLedgerDao so that we avoid
fetching a contract that anyway would be discarded later.

CHANGELOG_BEGIN
[Sandbox]: Witnessed contracts for which a party is not a stakeholder
are no longer returned in the active contract stream.
CHANGELOG_END

Fixes #3254.
2020-03-03 16:29:55 +01:00
Remy
bb0d3f24cb
[engine] refactor transaction's usedPackages (#4793)
* [engine] refactor transaction's usedPackages

* Address Gerolf's comment

* changelog

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-03 13:59:40 +00:00
Samir Talwar
a2fd466240
ledger-on-sql: The tests are not that fast. (#4787)
Especially when they're running against PostgreSQL. Just spinning that
up takes a while.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-03 13:51:17 +00:00
Moritz Kiefer
ad0336d3a5
Freeze DAML-LF 1.8 proto (#4788)
* Freeze DAML-LF 1.8 proto

This is a copy of the 1.dev proto with the following changes that
overall remove all references to 1.dev:

* Change the versions and the reference to the spec.
* Remove `dev` from version history at the top.
* Remove `GENMAP` and corresponding primitives.
* Remove general equality `EQUAL` and the comments that the
  type-specific equality primitives are only available in < 1.dev
* Remove experimental text primitives.

changelog_begin
changelog_end

* windows is very bad

* bump windows hashes
2020-03-03 14:21:00 +01:00
Moritz Kiefer
3c30073b31
Merge cross-LF tests (#4785)
Given that these two group of tests run for the same combination of LF
versions, I don’t see any reason why we should keep them
separate. Merging them, speeds up the packaging tests by > 50s on my
local machine.

changelog_begin
changelog_end
2020-03-03 09:53:06 +00:00
Moritz Kiefer
cc7f3e7ad0
Avoid decoding package twice in validate-dar (#4780)
Rather than decoding everything twice, we can only rewrite references
which is measurably faster (> 5s out of 60s on the packaging tests).

changelog_begin
changelog_end
2020-03-03 10:28:11 +01:00
Samir Talwar
6711b04a30
sandbox: Standardize logging when initializing the ledger. (#4782)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-03 09:04:32 +00:00
Samir Talwar
a99156252d
libs-scala/ports: Wrap socket ports in a type, Port. (#4784)
* libs-scala/ports: Wrap socket ports in a type, `Port`.

* sandbox: Use `Port` for the API server port, and propagate.

CHANGELOG_BEGIN
CHANGELOG_END

* extractor: Use `Port` for the server port.

* ports: Make Port a compile-time class only.

* ports: Allow port 0; it can be specified by a user.

* ports: Publish to Maven Central.
2020-03-03 08:59:15 +00:00
Moritz Kiefer
ebbe135583
Add singleton to DA.List (#4777)
changelog_begin

- [DAML Standard Library] Add ``singleton`` to ``DA.List``.

changelog_end
2020-03-02 20:49:21 +00:00
Leonid Shlyapnikov
93329ac81d
Add create-and-exercise JSON API endpoint (#4754)
* CreateAndExerciseCommand JSON decoding

* create-and-exercise endpoint WIP

* create-and-exercise endpoint + test, WIP

* create-and-exercise test

* Update docs

CHANGELOG_BEGIN

[JSON API - Experimental]
Add ``/v1/create-and-exercise`` JSON API endpoint. See #4507.

CHANGELOG_END

* Cleaning up domain JSON encoder/decoder, trying to get rid of HasTemplate

* Cleaning up domain JSON encoder/decoder, trying to get rid of HasTemplate

* Addressing code review comments
2020-03-02 20:43:58 +00:00
Samir Talwar
89c23ef85d
Heartbeats for Sandbox-Next, ledger-on-sql, and ledger-on-memory. (#4755)
* kvutils: Make the `KeyValueParticipantStateReader` tests more rigorous.

If the `offset` is specified as `None`, expect it to be `None`, not
just anything.

* kvutils: Simplify `KeyValueParticipantStateReader#stateUpdates`.

Construct the Source with `Source.apply`, not `Source.fromIterator`.

* kvutils: Use multiple entry IDs in `KeyValueParticipantStateReaderSpec`.

* kvutils: Add basic tests to `KeyValueParticipantStateReaderSpec`.

* kvutils: Add heartbeats to `LedgerReader`'s `events` output.

Heartbeats are optional, to be delivered by the ledger if and when it
deems necessary.

* sandbox-next: An observing time service backend using Akka streams.

* sandbox-next: A regular heartbeat based on Akka Streams' `tick`.

* sandbox: Replace `TimeServiceBackend.withObserver` with `.observing`.

More code, but it's more decoupled, so can more easily be sent to the
underlying backend in Sandbox Next.

CHANGELOG_BEGIN
- [Sandbox] Fixed a bug in the command completions stream when running
  Sandbox in static time. Previously, upon updating the time, the old
  time was emitted on the completions stream. The new time is now
  emitted.
CHANGELOG_END

* sandbox: TimeServiceBackend should only emit accepted changes.

* ledger-on-memory: Use `LedgerRecord` directly.

* ledger-on-memory: Stream heartbeats to the log.

* ledger-on-memory: Encapsulate mutations behind locks at all times.

* ledger-on-memory: Differentiate between reading and writing.

* ledger-on-memory: Factor out appending to the log.

* kvutils: Move the heartbeat test into the base from ledger-on-memory.

* kvutils: Log when the submission validation fails unexpectedly.

* ledger-on-sql: Add a script to hash all migrations.

* ledger-on-sql: Publish heartbeats to the log, and stream them out.

* ledger-on-sql: Log if publishing the heartbeat failed.

* ledger-on-sql: Wrap all queries in `Try`.

Just to make sure that we don't throw from a function that returns `Try`
or `Future`.

* ledger-on-sql: Allow `Long` values as the heartbeat timestamp.

`INTEGER` really does mean 32-bit, apparently.

* sandbox-next: Pipe heartbeats to the ledger.

* ledger-on-sql: Make sure we publish the correct head after a heartbeat.

Off-by-one errors are the best errors.

* ledger-on-(memory|sql): Just accept heartbeats, not their owner.

* sandbox: Update CIDs in tests to account for the extra heartbeat.

* ledger-on-memory: Fix a reference to variable in a comment.

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* ledger-on-sql: `flatMap` over `Try` rather than `Future` when possible.

* sandbox: Make sure the heartbeat queues are thread-safe.

* kvutils: Remove `LoggingContext` from the interfaces.

Keep it internally. This means we'll drop any context, but otherwise
things should work as expected.

* sandbox-next: Pull out the heartbeat interval into a constant.

* ledger-on-sql|sandbox: Clarify large levels of nesting.

Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2020-03-02 19:44:42 +00:00
Oliver Seeliger
95f596ec56
Lower Ledger api server rejected submissions to info (#4773)
Rejected submissions are a user-error and don't indicate a problem with the server not functioning properly. Such user
errors make it hard to spot "real" server-side warnings and errors.
i
Closes #4772

CHANGELOG_BEGIN
- The Ledger API Server now logs rejected submissions at a lower "INFO" level to remove a source of warnings/errors without relation to server health.
CHANGELOG_END
2020-03-02 19:01:47 +00:00
nickchapman-da
8ba9a2647d
Update Upgrade.daml (#4775)
changelog_begin
changelog_end
2020-03-02 18:26:36 +00:00
Moritz Kiefer
6ebf455f62
Remove -j 200 (#4771)
Originally, we introduced this since it made fully cached builds
slightly faster. However, that was a long time ago. Looking at the
numbers now, things actually seem to be significantly faster without
this (possibly due to changes in how Bazel handles this). In
particular on MacOS where I’ve seen a couple of builds with < 20
minutes on CI which I did not see in quite some time without this.

changelog_begin
changelog_end
2020-03-02 18:53:18 +01:00
Moritz Kiefer
d68d3eb74a
Freeze DAML-LF 1.8 (#4770)
* Freeze DAML-LF 1.8

Two minor points that I did not mention in the previous PR:

We also include the renaming of structural records to `struct` and the
renaming of `Map` to `TextMap`.

There are some minor changes around the LF encoder tests which need to
be able to emit package metadata properly so I’ve added it to the
parser. Sorry for not splitting that out.

Following the process used for the DAML-LF 1.7 release, this does not
yet include the frozen proto file.

changelog_begin

- [DAML-LF] Release DAML-LF 1.8:

  * Rename structural records to ``Struct``. Note that
    structural records are not exposed in DAML.
  * Rename ``Map`` to ``TextMap``.
  * Add type synonyms. Note that type synonyms are not serializable.
  * Add package metadata, i.e., package names and versions.

  Note that the default output of ``damlc`` is stil DAML-LF 1.7. You
  can produce DAML-LF 1.8 by passing ``--target=1.8``.

changelog_end

* Update encoder

* Update java codegen tests

* Update comment in scala codegen

* Handle TSynApp in interface reader

* Bump lf_stable_version to 1.7

* Fix kvutils tests
2020-03-02 18:29:26 +01:00
Stefano Baghino
83da2c643b
Clean up ledger-api-common (#4774)
Removes unused classes

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-02 18:16:45 +01:00
Remy
e7ee8143cb
[kvutils] use new contract id scheme (#4741)
* Make kvutils work with the new contract id scheme

CHANGELOG_BEGIN
- [KVUtils] uses random contract id. Contract ids are made of 65 hexa decimal characters.
CHANGELOG_END

Co-authored-by: Jussi Mäki <jussi.maki@digitalasset.com>
2020-03-02 17:50:38 +01:00
Stefano Baghino
00a3a1c2bb
Tighten the loop: backend services to return API responses (#4763)
* Tighten the loop: backend services to return API responses

CHANGELOG_BEGIN
CHANGELOG_END

* Use transaction filter directly

* Remove unnecessary transition through domain objects

* Ensure transient contract remover compares sets of witnesses

* Honor verbosity in request

* Address review https://github.com/digital-asset/daml/pull/4763#pullrequestreview-367012726

- using named parameters when creating the API objects
- renamed EventOps accessors to easily recognizable names
- dropped unnecessary usage of views
- honoring verbosity level in request in all places
- replaced usage of lenses with simple copying where it made sense
2020-03-02 15:06:38 +00:00
Moritz Kiefer
501aa9e89e
Prepare DAML-LF 1.8 release (#4769)
We will only include type synonyms and package metadata (which are
used for Cross-SDK model upgrades). Everything else stays in 1.dev for
now.

changelog_begin
changelog_end
2020-03-02 14:23:07 +00:00
nickchapman-da
47fa9eb0ca
Add validate-dar to the assistant whitelist. (#4766)
changelog_begin
changelog_end
2020-03-02 11:57:24 +00:00
Moritz Kiefer
040d343b8c
Document package metadata in LF spec (#4764)
* Document package metadata in LF spec

changelog_begin
changelog_end

* remove space
2020-03-02 11:32:36 +00:00
Moritz Kiefer
d36bb65df2
Bump zip library (#4768)
This should hopefully fix the failure to delete the temporary zip file
on Windows. See https://github.com/mrkkrp/zip/pull/61

changelog_begin
changelog_end
2020-03-02 12:26:28 +01:00
Moritz Kiefer
1b5140ed50
Add a first version of new upgrade docs (#4750)
This PR adds a first version of update documentation and removes the
existing docs which were talking about `damlc migrate`.
So far the docs focus on the high-level approach to upgrades taken in
DAML and give an example of how to structure upgrade contracts.

What is not covered so far (and I’d like to leave that for a separate
PR) is:

1. Technical details: How are things split up into packages, which
restrictions apply to data-dependencies, …
2. Deployment and Running the upgrade via triggers/daml script/…
3. Common patterns for handling this in UIs (e.g. “locking” old contracts)

changelog_begin
changelog_end
2020-03-02 09:54:39 +01:00
Moritz Kiefer
7a9b9a3ace
Fix data-dependencies for impredicative types (#4744)
I’m not really happy with this “fix” but after having spend way too
much time on this, this was the best I came up with. (The details are
in an inline comment). If anyone has better ideas, I’m all ears.

changelog_begin
changelog_end
2020-03-02 09:37:46 +01:00
nickchapman-da
8af3857c09
Use damlc validate-dar in packaging tests. (#4760)
Replace `daml-lf-repl validate` in packaging tests with `damlc validate-dar`.
Simplify test setup a little by passing tools (damlc,validate, etc) in a record.

changelog_begin
changelog_end
2020-02-28 16:28:04 +00:00