Commit Graph

3600 Commits

Author SHA1 Message Date
Moritz Kiefer
fdeac8d69c
Make extractor and navigator work without client auth (#4966)
This PR fixes the tls configuration to work if client auth is not
enabled and adds a `--tls` flag to extractor and navigator which
allows you to enable tls without overriding any certificates.

There is a test for extractor but none for navigator since there are
no tls tests at all afaict atm. I did however test it manually.

changelog_begin

- [Navigator] Navigator can now run a TLS enabled ledger without
  client authentication. You can enable TLS without any special
  certificates by passing ``--tls``.

- [Extractor] Extractor can now run a TLS enabled ledger without
  client authentication. You can enable TLS without any special
  certificates by passing ``--tls``.

changelog_end
2020-03-12 17:33:30 +01:00
Moritz Kiefer
edd73384c4
Make client authentication in Sandbox configurable (#4965)
Currently sandbox only supports TLS if you also enable client
authentication. There is no reason for why this has to be the case and
for things like DABL we want TLS without client authentication so it’s
useful to be able to test this in sandbox. This PR introduces a
`--client-auth` flag that allows you to configure the behavior. The
default is the current one of requiring client authentication.

This PR does not yet update Java clients, however, the Haskell client
supports this already and is used to test this functionality.

I’ve also added a section in the documentation on TLS (there were no
docs at all so far).

changelog_begin

- [DAML Sandbox] When Sandbox is run with TLS enabled, you can now
  configure the requirement for client authentication via
  ``--client-auth``. See
  https://docs.daml.com/tools/sandbox.html#running-with-tls for more information.

changelog_end
2020-03-12 16:04:46 +01:00
Stefano Baghino
1bf1a8e8e9
Add diff function between Relations (#4962)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-12 14:41:22 +01:00
Jussi Mäki
583c0a7749
Run most tests for ledger-dump (#4910)
We exclude the tests that create lots of data.
CommandDeduplicationIT is disabled as kvutils does not yet
have time-based deduplication.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-12 14:10:22 +01:00
associahedron
5b1698e553
Update packaging docs. (#4936)
* Update packaging docs.

changelog_begin
changelog_end

* Fix the code blocks

* Fix another code block

* Move comment so ordered list doesnt break.

* fix code blocks again???

* Apply suggestions from code review

Co-Authored-By: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* Make name collision section

* Incorporate feedback

* Small edit

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-03-12 12:33:53 +00:00
Moritz Kiefer
e0d652df91
Generate ssl certs in a genrule instead of checking them in (#4950)
changelog_begin
changelog_end

For now this is only used for the daml-helper tests. I’ll shuffle
things around and use it for all tests in a separate PR.
2020-03-12 10:36:40 +01:00
simonmaxen-da
4c55a67e38
Allow time for all nodes to see contract before executing (#4909)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-12 08:57:02 +01:00
Shayne Fletcher
cdc31ebe81
rename package 'build-and-lint' to 'build-and-lint-test' (#4957)
changelog_begin
changelog_end
2020-03-11 17:38:56 -04:00
Gary Verhaegen
872a5fc0df
do not send release notes to hubspot (#4956)
@bame-da wants a more manual process where he can control exactly when
release notes are posted, possibly in advance.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-11 21:41:46 +01:00
Robin Krom
254ee50b7c
docs: better readme's for ts libraries (#4954)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-11 21:35:42 +01:00
Leonid Shlyapnikov
19f5b86c79
Websocket stream to emit last seen offset instead of heartbeat message (#4886)
* Capture lastSeenOffset in the @volatile var

CHANGELOG_BEGIN
[JSON API - Experimental] Websocket stream now emits last seen offset instead of the heartbeat message.
``{"heartbeat": "ping"}`` is replaced by ``{"events":[],"offset":"<last seen offset>"}``. See #4510.
CHANGELOG_END

* updating docs

* moving the last seen offset into the stream, WIP

* adding in-stream state

* minor docs

* cleanup the heartbeat logic

* minot cleanup

* Change live and heartbeat msg handling + some debug logging (to be removed)

* fixing ts tests, cleaning up

* Adding todo with the reference to the follow-up ticket

* Adding todo with the reference to the follow-up ticket
2020-03-11 19:53:05 +00:00
Shayne Fletcher
901b28777e
Reduce repetition by reuse of 'modPath' (#4951)
* Reduce repetition by reuse of 'modPath'

changelog_begin
changelog_end

* Use intercalate not joinPath; import </> from FilePath.Posix

These two changes should remove any ambiguity about whether paths are
treated any differently on Windows vs 'nix.
2020-03-11 15:21:01 -04:00
Robert Autenrieth
d333d9e882
Add command deduplication changelog (#4944)
* Add command deduplication changelog

Fixes #4623

CHANGELOG_BEGIN
- [Sandbox] The command client is no longer idempontent with respect to duplicate
  submissions. Duplicate submissions now instead return an ALREADY_EXISTS error,
  consistent with the new deduplication mechanism of the submission client.
  See also `issue #4623 <https://github.com/digital-asset/daml/issues/4623>`_.
CHANGELOG_END

* Add changelog entry for command deduplication

Fixes #4623

CHANGELOG_BEGIN
- [Sandbox] The command client is no longer idempotent with respect to duplicate
  submissions. Duplicate submissions now instead return an ALREADY_EXISTS error,
  consistent with the new deduplication mechanism of the submission client.
  See also `issue #4623 <https://github.com/digital-asset/daml/issues/4623>`_.
CHANGELOG_END
2020-03-11 18:01:50 +00:00
Shayne Fletcher
793609fd04
client module import ideal syntax (#4946)
changelog_begin
changelog_end
2020-03-11 13:33:16 -04:00
Moritz Kiefer
ed5886051d
Support TLS in daml-helper (#4943)
This PR adds TLS support to DAML helper both via client certs and
without (although the latter is not tested so far since atm this is
not supported by sandbox). The CLI options follow the scheme used by
navigator/extractor/… with the addition that you can just pass `--tls`
which will turn on TLS without custom root certs or client certs.

changelog_begin

- [DAML Assistant] You can now connect to ledger via TLS for ``daml
  deploy`` and ``daml ledger`` commands. See
  https://docs.daml.com/deploy/generic_ledger.html for more information.

changelog_end
2020-03-11 17:58:05 +01:00
Andreas Herrmann
f3f39a23a3
daml-assistant tests: fix flaky test on Windows (#4945)
* daml-assistant tests: fix flaky test on Windows

```
bazel test //daml-assistant:test --test_arg=--quickcheck-replay=425714
```
failed on Windows in the test-case:
```
tail . ascendents == ascendents . takeDirectory
```
In that case the two path components given two the test case shrink to
`p1 = "a"` and `p2 = "\\"`. Confusingly, on Windows
```
isRelative "\\" == True
```
while
```
"a" </> "\\" = "\\"
```
This is documented behaviour on Windows, see [1] and [2].

Using `p1 </> p2 /= p2` instead of `isRelative p2` works around this.

[1]: https://hackage.haskell.org/package/filepath-1.4.2.1/docs/System-FilePath-Posix.html#v:-60--47--62-
[2]: https://hackage.haskell.org/package/filepath-1.4.2.1/docs/System-FilePath-Posix.html#v:isRelative

* Fix ascendants test group name

`ascendants` is defined in `DA.Daml.Project.Util`, not
`DA.Daml.Assistant`.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-03-11 16:04:31 +00:00
Robert Autenrieth
3cc17c0253
Command client remove deduplication (#4783)
* Remove deduplication from command client

CHANGELOG_BEGIN
CHANGELOG_END

* Adapt tests for the new behavior
2020-03-11 16:27:29 +01:00
Shayne Fletcher
7634b7cfad
Do not require the user type '@' (#4939)
changelog_begin
changelog_end
2020-03-11 11:27:15 -04:00
Leonid Shlyapnikov
953d55f338
Replace AsyncWordSpec with WordSpec, thanks @rautenrieth-da, see #4938 (#4941)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-11 14:58:38 +00:00
Robert Autenrieth
42adfb1887
Fix tests not running with bazel test (#4938)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-11 15:15:02 +01:00
Robert Autenrieth
9216a68ad1
Don't read exclusive end in completions query (#4899)
* Don't read exclusive end in completions query

CHANGELOG_BEGIN
CHANGELOG_END

* Store offsets directly and do +1 only on read side

* Fix existing completions

* Add test for the completion service

Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2020-03-11 14:10:32 +00:00
fabiotudone-da
e4ce69ea58
Rename EC auth cmdline options in line with the standard and document them. (#4930)
* Rename EC auth cmdline options in line with the standard and document them.

CHANGELOG_BEGIN
CHANGELOG_END

* 📝 Fix doc

* Auth docs: change `RSA DSA` -> `RSA Signature` (clashed with DSA algo)

As proposed by @SamirTalwar-DA

CHANGELOG_BEGIN
[Sandbox] Rename the `--auth-jwt-ec256-crt` command line option to `--auth-jwt-es256-crt` as well as `--auth-jwt-ec256-crt` to `--auth-jwt-es256-crt` and fix their docs
CHANGELOG_END
2020-03-11 13:18:47 +00:00
Shayne Fletcher
2e8f8e5192
make generated packages easier to use (#4926)
changelog_begin
changelog_end
2020-03-11 09:03:13 -04:00
Moritz Kiefer
0ed3bbf2ce
Bump nix version (#4934)
Using the newest seems like a good idea and the previous one has
network errors

changelog_begin
changelog_end
2020-03-11 12:26:29 +00:00
Remy
bdb6179405
[DAML-LF] add submission time for contract ids seeding (#4776)
* [DAML-LF] add submission time for contract ids seeding

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-11 12:17:22 +01:00
Samir Talwar
3c6f8ed627
kvutils: Avoid casting ArgumentCaptor and friends in tests. (#4928)
* kvutils: Avoid casting `ArgumentCaptor` and friends in tests.

Instead, use generics the way they're intended.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: In KeyValueParticipantStateWriterSpec, drop the Option.
2020-03-11 09:38:43 +00:00
Robert Autenrieth
9fd484469a
Add maxDeduplicationTime to the participant state API (#4722)
* Add max deduplication time to ledger configuration

* Add TTL to participant state API

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-11 09:49:37 +01:00
Robin Krom
8fe497ea5d
lanugage:docs: include generated docs on docs.daml.com (#4924)
This includes the generated docs for the typescipt libraries daml-react,
daml-ledger and daml-types in the documentation presented on
docs.daml.com. Next step is to create better readmes in this libraries.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-10 18:46:54 +01:00
Stefano Baghino
6f59832e45
Add lazy flattening function to Relation (#4922)
* Add lazy flattening function to Relation

CHANGELOG_BEGIN
CHANGELOG_END

* Restrict tests to non-empty relations and address https://github.com/digital-asset/daml/pull/4922#pullrequestreview-372050511
2020-03-10 17:39:45 +00:00
Moritz Kiefer
50a0e2eda3
Enable caching and sandboxing for canton ledger API test tool (#4923)
After some investigation, canton does not currently expose a nice way
to tell ammonite where it should write its files or even better use
the in-memory mode. However, ammonite respects $HOME so we can just
set that to a temp directory which fixes the issue.

changelog_begin
changelog_end
2020-03-10 16:34:58 +00:00
Moritz Kiefer
4510bae9b6
Split upgrade models into a separate package (#4921)
* Split upgrade models into a separate package

This PR splits the upgrade example into 3 packages instead of 2 which
avoids a dependency from the model on the old model. This is explained
in the documentation.

changelog_begin
changelog_end

* Fix typo
2020-03-10 17:24:17 +01:00
Shayne Fletcher
9cdb01773b
Make scope an (optional) CLI arg (#4908)
changelog_begin
changelog_end
2020-03-10 11:48:37 -04:00
nickchapman-da
944a23e302
New bazel rule -- daml_build_test -- to build a DAML project from its daml.yaml config, and setup a bazel test rule to validate the resulting .dar using daml validate-dar. (#4891)
Dependencies on other DAML projects are declared with the `dar_dict` attribute of the build rule. This attribute also declares the names by which the `.dar` files are known in the client project, corresponding to the references in the `daml.yaml` config.

The new rule is used build & test the upgrade documentation example code.

changelog_begin
changelog_end
2020-03-10 14:23:23 +00:00
Robin Krom
7520e9f48e
docs: build rules for typescript docs (#4907)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-10 13:33:39 +01:00
Moritz Kiefer
ffc6f4e520
Include Bazel patch to mark tests as exclusive (#4918)
* Include Bazel patch to mark tests as exclusive

This should hopefully avoid rerunning the conformance tests as often
as we do now. While this patch is not applied on Windows (since we get
it from nix), this is not really an issue since most of the exclusive
tests (in particular, all conformance tests) are disabled on Windows
anyway.

I’ve tested this locally accross a couple of runs and I get the
caching I want and looking at the code in the patch, the change looks
very reasonable. I somewhat wonder if it just broke internally at
google because they marked tests as exclusive that should have gotten
no-cache.

changelog_begin
changelog_end

* Disable caching for canton
2020-03-10 13:13:58 +01:00
Moritz Kiefer
048b3fa679
Fix warnings on mismatch between record type and constructor name (#4919)
changelog_begin
changelog_end
2020-03-10 13:01:16 +01:00
Gary Verhaegen
caf07566d2
add commands to extract relevant changelogs (#4905)
In the current state of the release instructions, the person in charge
of the release has to figure out how to produce the changelog. This PR
adds more specific (and hopefully simpler) instructions for producing
relevant changelogs.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-10 12:45:31 +01:00
Gary Verhaegen
ef931e0b72
skip testing release script after making a release (#4911)
Currently, on Linux, after the normal build, we try running the release
script (in "dry run" mode). This is to check that the release script not
only compiles, but actually runs. To be honest I'm not entirely sure why
we do that as a separate step (i.e. why does `bazel test //...` not give
us confidence about this script?), but the point of this PR is that,
while there may be some benefit in running this script on normal PRs to
check that we have not broken the release step, there is absolutely no
point in running it _on a release build_, i.e. right after we've used
the same script in "real" ("wet run"? 🤔) mode.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-10 12:45:14 +01:00
Moritz Kiefer
c4fa115b67
Remove experimental note about DAML script from daml --help (#4917)
changelog_begin
changelog_end
2020-03-10 11:26:46 +00:00
Stefano Baghino
5298ac8690
Fix CreatedEvent Protobuf documentation (#4916)
CHANGELOG_BEGIN
[Documentation] Ledger API documentation clarifies how witness_parties
are determined depending on whether CreatedEvents are served as part of
the flat or full transaction stream.
CHANGELOG_END
2020-03-10 11:19:59 +00:00
Gary Verhaegen
dd1b44eccc
add grep to dev-env (#4903)
Because it's not in there and we're all surprised to learn that.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-10 10:34:01 +01:00
Gerolf Seitz
72cafaef70
Update release testing to include running daml script (#4915)
Also the line numbers are updated.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-10 08:49:47 +01:00
Gary Verhaegen
390bcf2ae3
retry release (#4914)
Retrying #4902 with #4912 and #4913.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-09 20:46:22 +01:00
Gary Verhaegen
8f6be82660
retry releases (#4913)
This PR changes the release version format for snapshot releases to
allow for an optional "build number", i.e. "how many times have we
screwed up this release before this attempt?".

Adding this to the version string should be fine because:

- It is a number, so GHC will be happy.
- It is dot-separated and (manually) incrementing, so all three formats
  will sort it correctly.
- The SemVer->GHC conversion only looks at the beginning and ending of
  the snapshot string, and this changes the middle.

This is necessary because sometimes we screw up releases (e.g. #4902),
but also because sometimes releases screw themselves up by somehow
corrupting the Windows cache, and so far changing the version number has
been the only way out of that. So far that has meant changing the target
commit, but that's a very poor reason to choose a target commit.

We should not have to include additional code in a release just because
our release process is flaky.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-09 19:43:29 +01:00
Gary Verhaegen
41643315ac
hopefully fix Azure pipelines github tag release (#4912)
Maybe. If I'm reading
[that](https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/github-release?view=azure-devops)
right.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-09 19:39:08 +01:00
Gerolf Seitz
72d5d6e02e
Set LATEST version for snapshot release. (#4902)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-09 17:16:52 +01:00
Moritz Kiefer
58c101c3ab
Mark lsp tests as flaky on Windows (#4906)
:sadpanda:

changelog_begin
changelog_end
2020-03-09 16:53:36 +01:00
Samir Talwar
51f596daa7
Sandbox Next: Implicit party allocation. (#4894)
* ledger-api-test-tool: Fix warnings flagged by IntelliJ IDEA.

* ledger-api-test-tool: Open-world mode.

In open-world mode, parties aren't allocated; their names are just
reserved for the test case, so that no other test will accidentally use
the same party name.

This is so we can test ledgers which dynamically allocate parties, such
as Sandbox.

* sandbox: Run conformance tests in "open-world" mode.

This means that the tests don't explicitly allocate parties (except for
a few), instead relying on Sandbox's implicit party allocation feature.

This is not enabled for Sandbox Next yet.

* sandbox-next: Implicit party allocation.

This is added to the command submission service.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox-next: Don't implicitly allocate pre-existing parties.

* ledger-api-test-tool: Move pre-allocation into ParticipantTestContext.

* ledger-api-test-tool: We can reserve parties or wait for them. Not both.

Make illegal states unrepresentable as early as possible.

* sandbox: Name ApiSubmissionService's private methods a little better.

* sandbox: Move ApiSubmissionService's conditional logic into methods.

* sandbox: Document why we set `implicitPartyAllocation` to `false`.

* sandbox: Document why `implicitPartyAllocation` is dangerous.
2020-03-09 15:49:11 +00:00
Moritz Kiefer
7b1f193012
Error out on dependencies on newer LF versions (#4901)
This doesn’t really make sense since the main point of targetting an older
LF version is because your server does not support the newer LF
version but including dependencies in newer LF versions makes that
completely useless. In the current state, this also produces a bunch
of errors that look very confusing and while we might be able to fix
them, I don’t think it’s worth doing.

changelog_begin
changelog_end

fixes #4596
2020-03-09 14:59:34 +00:00
Robert Autenrieth
6f8c3ad832
Add command deduplication docs (#4693)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-09 15:33:45 +01:00