Commit Graph

3385 Commits

Author SHA1 Message Date
Moritz Kiefer
d381603475
Detect whether sandbox has started using --port-file (#4880)
On MacOS using Java 13 (at least that was the only combination where
I’ve managed to reproduce this). Sandbox spits out errors of the
following form in ``daml start``

```
java.net.SocketException: Connection reset
```

This appears to be caused by the TCP requests that we send to detect
whether sandbox has started. Switching to using the ``--port-file``
option makes this issue go away.

fixes #4670

changelog_begin

- [DAML Assistant] In certain configurations ``daml start`` produced
  a (harmless) ``Connection reset`` log message. This has now been
  fixed.

changelog_end
2020-03-06 16:18:43 +00:00
associahedron
02ae2a29d9
data-dependencies: Call buildHiddenRefMap once per package. (#4881)
* Call buildHiddenRefMap once per pkg.

Instead of calling it once per module.

changelog_begin
changelog_end

* Remove special cases
2020-03-06 16:17:57 +00:00
Samir Talwar
a0f250a727
sandbox: Avoid converting to byte arrays before deserializing. (#4865)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-06 15:40:26 +00:00
Samir Talwar
c6954c086b
sandbox-next: Error if a scenario is provided. (#4875)
* sandbox-next: Pull runner configuration into the constructor.

No need to do it on `acquire()` if it's pure.

* sandbox-next: Error if a scenario is provided.

Sandbox-Next doesn't support scenarios, instead favoring DAML Script.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-06 14:51:42 +00:00
Samir Talwar
b4a529eda9
sandbox: Format Cli.scala. (#4876)
Not sure why formatting was turned off at one point, but it seems to be
OK now.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-06 14:43:04 +00:00
Samir Talwar
82c40f2ecb
kvutils: Throw meaningful errors when dealing with a Left. (#4835)
* kvutils: On error opening an envelope, throw the correct message.

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-on-sql: On error when querying state, throw the correct error.

* kvutils|ledger-on-sql: Remove unnecessary curly braces around `throw`.
2020-03-06 13:53:34 +00:00
Moritz Kiefer
d8afe7a4cc
Support specifying sandbox/navigator/json-api/script options in daml.yaml (#4877)
changelog_begin

- [DAML Assistant] You can now specify options for
  Sandbox/Navigator/the HTTP JSON API in ``daml.yaml`` via
  ``sandbox-options``/``navigator-options``/``json-api-options``. These
  options will be picked up by running ``daml start``. This is
  particularly useful for specifying ``--wall-clock-time`` and for
  specifying a fixed ledger ID during development.

changelog_end

fixes #2993
2020-03-06 14:52:45 +01:00
Moritz Kiefer
c7eb3f6b6c
Wrap Script in StateT to make evaluation order a bit less important (#4869)
* Wrap Script in StateT to make evaluation order a bit less important

This PR wraps the Script newtype in `StateT` which means that
evaluation won’t do much so `debug` behaves a bit more sensibly and
you don’t end up evaluating a script that only consists of `pure` and
`>>=` if you do not execute it.

fixes #4821

changelog_begin

- [DAML Script] Fix an issue where ``debug`` messages were output
  before the script was executed.

changelog_end

* Inline StateT and improve error messages
2020-03-06 13:42:11 +01:00
Shayne Fletcher
60fa929336
Eliminate duplication of package name calculation (#4874)
changelog_begin
changelog_end
2020-03-06 07:24:50 -05:00
Moritz Kiefer
ad55159989
Address review comments on script migration docs (#4870)
changelog_begin
changelog_end
2020-03-06 12:39:10 +01:00
Gary Verhaegen
2eedd291ec
bind repl to localhost (#4863)
Currently the repl server is bound to 0.0.0.0, which is not great for
security and makes running the tests a bit disruptive on macOS.

This binds it to 127.0.0.1 instead.

CHANGELOG_BEGIN
- [DAML Repl - Experimental] The REPL server will now bind to 127.0.0.1
  instead of 0.0.0.0.
CHANGELOG_END
2020-03-06 11:40:55 +01:00
Moritz Kiefer
52bf9b2a5c
Fix git tag of releases (#4862)
Previously, we tagged the commit that made the release instead of the
commit we are building the release off.

changelog_begin
changelog_end
2020-03-06 10:54:07 +01:00
Shayne Fletcher
9858a38d6b
version -> sdkVersionOrErr (#4861)
changelog_begin
changelog_end
2020-03-06 10:33:52 +01:00
fabiotudone-da
9635e14d56
Restore LedgerFactory inheritors' ability to use Config for setups (#4860)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-06 08:25:27 +00:00
Shayne Fletcher
7fe94be065
Drop special casing 'daml-types' (#4859)
changelog_begin
changelog_end
2020-03-05 18:02:49 -05:00
Shayne Fletcher
5171ed5313
Shift comment and metadata write calls around (#4858)
* Shift comment and metadata write calls around

Also remove licence line from package.json files

changelog_begin
changelog_end

* Broken Windows cache. Unbreak it. NOT FOR MERGING TO MASTER.

* Restore Windows cache; safe now to merge.
2020-03-05 17:47:23 -05:00
Moritz Kiefer
fa968699c8
Unexperimentalize DAML script (#4856)
The only remaining thing that I wanted to get in before doing that was
overloading ``submit`` which has now happened. I’m sure we can come up
with tons of improvements but I don’t expect breaking changes to the
current state.

changelog_begin

- [DAML Script] DAML Script is no longer experimental.

changelog_end

fixes #4615
2020-03-05 20:40:15 +01:00
Shayne Fletcher
e1a164ef8b
Introduce a parameter pack for function 'daml2ts' (#4852)
changelog_begin
changelog_end
2020-03-05 13:54:39 -05:00
mergify[bot]
6b851229b8
Refactor extraction of events from transaction (#4781)
* Refactor extraction of events from transaction

Closes #1909

CHANGELOG_BEGIN
CHANGELOG_END

* Remove unnecessary filtering

* Fix disclosure rule for flat transaction

* Refactor and split collection and filtering

* Replace transaction filtration with blinding info

* Move transient contract remover in transaction conversion

* Remove dangling file

* Simplify transient contract filtering

* Further refinements

* Simplify transaction tree event extraction

* Move newRoots up the file for consistency and readability

* Remove collect from GenTransaction, replace with custom iterator

* Address https://github.com/digital-asset/daml/pull/4781#discussion_r388167562

* Switch to a strict collect method

* Replaced direct access to map with contains
2020-03-05 18:06:05 +00:00
mergify[bot]
1007fd58e6
Track erased types in data-dependencies. (#4833)
* Track erased types in data-dependencies.

This PR introduces a tracker for Erased types, DA.Generics, and any type, typeclass, or typeclass instance that dependens on them transitively. This is designed to detect and cut out any such definition that will cause problems.

Apologies for the large PR. Originally this was just a small extension to the tracking of old-style typeclasses, but each bugfix uncovered a new bug, and the only way to get it to completely work was to do it all at once. That's why the tracker is so extensive -- the only thing it doesn't track is regular functions, because they will never introduce a cyclic reference to an erased type that gets exposed in the type system (until we implement dependent DAML).

Right now the tracker runs once per data-dependency module, but in a future PR I will make it run once per data-dependency package to reduce the repetitive. This was all tested on a very large DAR, and it runs fine, but it should be much faster once it runs once per package.

changelog_begin
changelog_end

* lint

* Address comments

* Add a great test case
2020-03-05 17:54:55 +00:00
mergify[bot]
c1d601586a
ledger-on-sql: Ensure log entries are returned in the correct order. (#4854)
I have no idea how this worked previously. I think we got lucky.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-05 17:18:08 +00:00
mergify[bot]
c6d9000369
Added --damlc option to daml-sdk-head (#4853)
changelog_begin
changelog_end
2020-03-05 16:38:56 +00:00
mergify[bot]
60013a1535
Remove DA.Upgrade module (#4839)
Given that Generic instances are not supported cross-SDK this module
only causes confusion and I’d rather remove it.

changelog_begin
changelog_end
2020-03-05 16:30:54 +00:00
Gary Verhaegen
950d8c3501
remove perf tests from CI (#4851)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-05 17:29:28 +01:00
Gary Verhaegen
75c7d48d5b
bind localhost for json-api and scenarios (#4576)
Both were previously binding 0.0.0.0, which is inherently insecure. More
importantly to me, that meant running `bazel test //...` essentially
rendered my computer unusable for however long it took (which is
_long_), as it kept popping up focus-stealing dialogs about whether or
not I wanted to trust "java" to open an incoming network connection.

The ScenarioService does not seem to have an existing setup for CLI args
and my Scala-fu is not good enough to add one, so I just changed the
hard-coded path.

The JSON API already had an option, just with the wrong default. This is
technically a breaking change, but I'm hoping to pass it under the
"experimental" flag we still have on the JSON API.

CHANGELOG_BEGIN
- [JSON API - Experimental] As a security improvement, the JSON API
server will now bind on ``127.0.0.1`` by default. Previous behaviour was
to bind on ``0.0.0.0``; you can get that behaviour back by passing in
the (existing) flag ``--address 0.0.0.0``.

- [DAML SDK] The Scenario Service will now bind on ``127.0.0.1``. Previous
behaviour was to bind on ``0.0.0.0``.

CHANGELOG_END
2020-03-05 17:10:49 +01:00
mergify[bot]
6defabe39e
Document migration from scenarios to DAML script (#4834)
* Document migration from scenarios to DAML script

changelog_begin
changelog_end

* Add a more general section on ledger initialization

* fix tests
2020-03-05 16:09:21 +00:00
Shayne Fletcher
ed75b64fef
'writeRootPackageJson' -> 'setupWorkspace' (#4847)
Rename this function in empathy with
https://github.com/digital-asset/daml/pull/4829 and compute dependency
pairs not triples as per https://github.com/digital-asset/daml/pull/4799#discussion_r387883681

changelog_begin
changelog_end
2020-03-05 11:00:53 -05:00
Andreas Herrmann
7a2bf268fe
ld -s --> ld -unexported_symbols_list=* on macOS (#4838)
On MacOS the `-s` linker flag (`--strip-all`) is obsolete. Instead, one
should pass `-unexported_symbols_list=*` to strip all symbols.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-03-05 16:56:01 +01:00
mergify[bot]
86362ee0bf
fix typo in daml-script docs (#4840)
Redundant.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-05 15:12:53 +00:00
mergify[bot]
35ee0b3057
daml2ts: Always use SDK version for @daml/types (#4829)
Currently, the version of `@daml/types` used by the generated TS is
configurable. This is not very helpful since `daml2ts` and `@daml/types`
are developed in lock step and still moving quite fast.

This PR changes `daml2ts` to always use the SDK version for as the
version of `@daml/types`. This also requires us to fix the test
for DAVL which used a hard coded version of `@daml/types` so far.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-05 15:12:06 +00:00
mergify[bot]
e785bb7402
Sandbox: Re-use actor systems between the StandaloneApiServer and StandaloneIndexerServer. (#4827)
* sandbox: Re-use the root actor system in the StandaloneIndexerServer.

* kvutils/app: Don't use the ActorSystem execution context randomly.

Instead, make `Runner` a proper ResourceOwner, with an `acquire` method.

* sandbox: Re-use the root actor system in the StandaloneApiServer.

CHANGELOG_BEGIN
CHANGELOG_END

* resources: Remove the now-unused `ResourceOwner.sequence` functions.

They weren't well-thought-out anyway; they acquire resources
sequentially, rather than in parallel.
2020-03-05 14:50:18 +00:00
Moritz Kiefer
0a6be2b341
Overload submit so that it doesn’t collide with DAML script (#4831)
This introduces a `HasSubmit` typeclass (following the naming scheme
of `HasCreate`, …) and instances for `Scenario` and `Script`. This
avoids the need to hide `submit` in every single DAML script.

changelog_begin

- [DAML Standard Library] ``submit`` and ``submitMustFail`` are now
  overloaded so that they can be used in both scenarios and DAML script.

changelog_end
2020-03-05 15:43:35 +01:00
Shayne Fletcher
d33b507344
Replace over-enthusiastic uses of 'maybe' with case expressions (#4837)
changelog_begin
changelog_end
2020-03-05 09:43:13 -05:00
mergify[bot]
46a218a22f
sandbox-next: Disable ResetService tests. (#4836)
They're too flaky. We'll bring them back once we work out the
performance issues.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-05 14:06:02 +00:00
mergify[bot]
1842015a46
daml2ts: Factor out assertFileExists in tests (#4830)
* daml2ts: Factor out assertFileExists in tests

The pattern `assertBool "..." =<< doesFileExist ...` appears so many
times that it definitely deserves its own function.

CHANGELOG_BEGIN
CHANGELOG_END

* Remove assertions for existence of DAR files

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-05 13:55:30 +00:00
Moritz Kiefer
42dd6349e9
Cache unmangleIdentifier (#4822)
* Cache unmangleIdentifier

Previously we unmangled once per reference to an identifier rather
than doing the unmangling once per entry in the string interning
table. This PR fixes this which brings a pretty decent performance
improvement. On my testcase the time for converting for converting
from the low-level proto AST to the high-level Haskell AST goes down
from 13.5 to 7.5s on a certain DALF that we know very well. Max
residency also goes down from 2GB to 1.5GB (although that number is
somewhat unreliable ime) and allocations drop by 8%.

changelog_begin
changelog_end

* Move error message to unmangleIdentifier
2020-03-05 13:35:28 +01:00
Shayne Fletcher
3b6bd630cc
Rearrange to avoid 'stripPrefix'; cut down on 'T.pack'/'T.unpack' (#4818)
* Rearrange to avoid 'stripPrefix'; cut down on 'T.pack'/'T.unpack'

changelog_begin
changelog_end

* Remove ticks as suggested in https://github.com/digital-asset/daml/pull/4799#discussion_r387886292

* Fix indentation (as per https://github.com/digital-asset/daml/pull/4799#discussion_r387892680)
2020-03-05 07:28:47 -05:00
mergify[bot]
95c1e5edaa
Allow LedgerFactory to provide a full-blown ReadWriterService (#4792)
* Allow `LedgerFactory` to provide a full-blown `ReadWriterService` rather than a `LedgerReaderWriter` (needed by at least vDAML)

CHANGELOG_BEGIN
CHANGELOG_END

* Address review points

* Address Samir's review point in `SqlLedgerFactory`

* Finish addressing Samir's review point in `SqlLedgerFactory`

* Split reader and writer owners in `LedgerFactory` as suggested by Gerolf

* Remove unneeded `val` from `KeyvalueParticipantState[Reader|Writer]` constructor params

* Remove unneeded type parameter from `app.Runner`

* Leave `LedgerFactory` a full ledger builder but split hierarchy upwards and clarify responsibilities

* Rename `SimpleLedgerFactory` to `KeyValueLedgerFactory`
2020-03-05 12:02:23 +00:00
Gary Verhaegen
3410000fc7
change msys2 mirror (#4825)
The `repo.msys2.org` server is currently broken and has been for about 7
hours at least from what I can find out online. This PR changes to the
next mirror in the list that seems to work for me locally; list taken
from [the GitHub
repo](https://github.com/msys2/MSYS2-packages/blob/master/pacman-mirrors/mirrorlist.msys).

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-05 11:56:32 +01:00
Gary Verhaegen
80652bd51f
report-std-change: handle GitHub errors (#4814)
There have been a few GitHub glitches last week that resulted in a few
commits on master not being associated with a PR (though they really
were created from merging a PR, and the correct PR number is in their
title).

This makes the report script crash on not finding the PR, so this PR
fixes that. And a comment.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-05 10:06:58 +01:00
mergify[bot]
9bb80e552c
daml2ts: Keep workspace package.json intact (#4820)
Keep all fields in the workspace `package.json` given to `daml2ts`
instead of only `private` and `workspaces`.

This also allows for removing the `sed` hack in the `build-and-lint`
test and use the `resolution` field of the workspace `package.json`
to point to our local versions of `@daml/types` and `@daml/ledger`.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-05 00:45:11 +00:00
Shayne Fletcher
0a0786d596
Set test size to large (#4817)
changelog_begin
changelog_end
2020-03-04 15:59:23 -05:00
mergify[bot]
cb1395e923
Remove damlc migrate (#4816)
* Remove damlc migrate

``damlc migrate`` hasn’t worked for quite a while and we emitted a
warning for months so given that we don’t have plans to make it work
again in the near future, I think it does more harm than good to keep
it around.

changelog_begin

- [DAML Compiler] After being deprecated for a while the ``damlc
  migrate`` command has now been removed. See
  https://docs.daml.com/upgrade/ for up to date documentation
  on model upgrades.

changelog_end

fixes #3704 (by removing the tests 😇)

* yeah the windows cache is once again broken \o/

* Revert "yeah the windows cache is once again broken \o/"

This reverts commit 38d7877aa4.
2020-03-04 20:36:48 +00:00
Leonid Shlyapnikov
f9057bea19
handle dual LiveBegin case (#4815)
* make append of two liveness markers proper with respect to offsets

- never happens in a real stream, but satisfying some random tests
- test randomly harder

CHANGELOG_BEGIN
CHANGELOG_END

* move Monoid and Semigroup instances for Steps to main code

- suggested by @leo-da; thanks

* stray empty InsertDeleteStep
2020-03-04 14:59:26 -05:00
Shayne Fletcher
48595f774d
daml2ts : Generate a package of each DALF (#4799)
* daml2ts : generate one package per dalf
2020-03-04 14:51:40 -05:00
Moritz Kiefer
c866ac5132
Speed up unmangleIdentifier (#4810)
* Speed up unmangleIdentifier

On my (admittedly not super scientific) benchmark, this brings the
time used to convert from the low-level proto Haskell AST to the
high-level AST from 20s down to 16s on a certain DALF that we are all
too familiar with.

changelog_begin
changelog_end

* address review comments
2020-03-04 19:48:03 +01:00
mergify[bot]
a40190d695
minor cleanup, using Option.exists instead of cata. (#4812)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-04 18:10:04 +00:00
mergify[bot]
c001b4f22c
test that ContractStreamStep#append preserves data sensibly (#4809)
* test that ContractStreamStep#append preserves data sensibly

CHANGELOG_BEGIN
CHANGELOG_END

* (C) header

* test more general associativity
2020-03-04 17:59:02 +00:00
Moritz Kiefer
c6e3bf5399
Only unmangle once in decodeValueName (#4813)
The `decodeValueName` code is rather confusing. It calls things
unmangled that are mangled and the other way around.

Furthermore, it unmangles twice, once in `decodeNameString` and once
directly in `decodeValueName`. The code claims that this is a
compatiblity hack but unless someone can explain to me what exactly is
failing here or CI fails, I would prefer to just kill this.

changelog_begin
changelog_end
2020-03-04 17:29:55 +00:00
Gary Verhaegen
c2240df083
change standard change report time (#4804)
In the current setup, the report generation starts at 4. However, the
daily killing of all machines also starts at 4, giving the report little
chance of ever finishing.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-04 18:22:32 +01:00