Commit Graph

5872 Commits

Author SHA1 Message Date
Robin Krom
0d37052206
fix: Create missing directories for script output (#8196)
This fixes #8142.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-11 11:48:31 +01:00
Kamil Bożek
5db4ca687b
Multi-party command submission [KVL-699] (#8152)
Changes:
- replaced submitter: Party with actAs: Set[Party] and readAs: Set[Party] in com.daml.ledger.api.domain.Commands
- adapted contract and key lookups for multi-party submissions

* Changed single Commands to accept multiple submitters

* Removed submitter field from Commands

* Replaced submitters with actAs and readAs in Commands

* Moved actAs and readAs to com.daml.ledger.api.Commands

* Contract lookup tests

* Key lookup tests

CHANGELOG_BEGIN
CHANGELOG_END

* Formatted code changes

* Remove unused method

* Formatting fixes

* Simplified logical condition for active contracts visibility

* Removed unused temporary method in Commands

* Unified contract key lookup query for different type of db types

* Simplified ContractReader class structure

* Improved dao unit tests

* Unit tests for lookups as a divulgee

* Reduced code duplication in unit tests

* Minor improvement

* Formatted code

* Fixed a ContractsReader's query

* Improved unit tests

* Explanation for engine's command authorizers and contract readers

* Minor fixes

* Adapted SpannedIndexService for the multi-party submission
2020-12-11 10:04:15 +01:00
dependabot[bot]
8decc31813
Bump ini from 1.3.5 to 1.3.7 (#8253)
Bumps [ini](https://github.com/isaacs/ini) from 1.3.5 to 1.3.7.
- [Release notes](https://github.com/isaacs/ini/releases)
- [Commits](https://github.com/isaacs/ini/compare/v1.3.5...v1.3.7)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-12-11 10:02:07 +01:00
Andreas Herrmann
d3b4042ade
Factor out the middleware client from the trigger server (#8244)
* Factor out authorization middleware client

changelog_begin
changelog_end

* Factor out token refresh

* Factor out auth request

* Factor out auth middleware URIs

* factor out auth directive

* Factor out login directive

* Add login response type to api library

* Allow error handling in login callback

* Factor error handling out of authorize directive

* Move tagged token types into middleware api

* Factor out the auth middleware client

* Dedicated exception types in middleware client

* Handle auth middleware client exception

Restores the behavior of authorize before factoring out the client

* expose middleware URIs

* Use the middleware client in the middleware tests

* Use localhost/CALLBACK in testing

https://github.com/digital-asset/daml/pull/8244#discussion_r540328001

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-12-10 17:26:06 +00:00
Gary Verhaegen
77a9592936
remove navigator licenses (#8246)
This looks like completely dead code, and its purpose is arguably better
fulfilled by the NOTICES file now.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-10 17:31:20 +01:00
Remy
76c4380fef
LF: drop support from deprecated LF value versions. (#8083) (#8237)
* LF: drop support from deprecated LF value versions. (#8083)

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-10 17:14:36 +01:00
Gary Verhaegen
5c8ac44049
update macOS nodes README (#8243)
This is far from perfect but removes the blatantly wrong sections of the
README.

Note: as a README change, this is not really a standard change, but
because the README is under the infra folder, this PR does need the tag
to pass CI.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-10 16:48:12 +01:00
Gary Verhaegen
05b5473a76
replace daml-licenses with NOTICES (#8241)
At some point in time, `daml-licenses` was supposed to list the licenses
for all of our Haskell dependencies. It's not been updated in ages, and
we now have a reliable way to get at all our dependencies.

This is arguably a bit weird since the command is `daml damlc license`,
and it lists licenses for everything (`daml license` would make more
sense), but it seems a lot better to give a correst superset than an
incorrect list.

Perhaps we can move the command to `daml license` at some point, but I'd
see that as a separate step. (That would technically be a breaking
change, but I think this one might be OK.)

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-10 16:46:13 +01:00
Gary Verhaegen
2de96b79a6
cleanup oss-compliance & NOTICES.md (#8242)
Mostly outdated/redundant information across those; hopefully this is
now more accurate.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-10 16:45:56 +01:00
Moritz Kiefer
4478d63c68
Improve error messages on use of contract ids in contract keys (#8236)
* Improve error messages on use of contract ids in contract keys

We got some feedback that "Unexpected contract id" sounds like an
internal error whereas this is really a user error.

changelog_begin
changelog_end

* Include templateId in error message

changelog_begin
changelog_end
2020-12-10 16:26:55 +01:00
Gary Verhaegen
be2ad76897
fix link to AST (#8238)
CHANGELOG_BEGIN
CHANGELOG_END
2020-12-10 15:48:58 +01:00
curiousleo-da
f4fa88c977
Trace index DB -> Ledger API read path [LPF-223] (#8233)
CHANGELOG_BEGIN
- [Integration Kit] Emit OpenTelemetry events for transactions streamed
  to the client from the Ledger API
CHANGELOG_END
2020-12-10 15:06:17 +01:00
Gary Verhaegen
e7b3ac39b5
delete merge-prohibition (#8235)
Dead file, part of the defunct "red master" system.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-10 14:37:47 +01:00
Gary Verhaegen
a564a97444
pom template: https link (#8234)
More security is better.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-10 14:37:33 +01:00
Oliver Seeliger
e23f44f47d
Ledger api server index h2 avoid reliance on PUBLIC schema (#8223)
by replacing entire `participant_command_completions` table to
prevent assumption that tables reside in the `PUBLIC` schema
which they are not in canton. This is follow-up to #8035

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-10 10:11:47 +01:00
Gary Verhaegen
029c655adc
blackduck: open PR on NOTICES file change (#8215)
CHANGELOG_BEGIN
CHANGELOG_END
2020-12-10 10:08:28 +01:00
Remy
e2e0523b78
LF: fix versioning output in LF Repl. (#8219)
With this PR, the Repl now:
* outputs the transaction version (not sure why it was not
there before)
* outputs the node version, because each node is now versioned
independently (See #7788)
* drops the value version, because it is dictated by the node version
(See #7788)

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-09 17:40:12 +01:00
Gerolf Seitz
ea84b975dd
daml-sdk-head: don't install to path (#8220)
When running daml-sdk-head from circleci without set-path=no, the
installation fails because the assistant tries to get a Yes/No answer
from stdin (which is not available).

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-09 15:39:16 +00:00
Samir Talwar
79a91669b2
kvutils: Use mockito-scala's ArgumentMatchersSugar. (#8218)
CHANGELOG_BEGIN
CHANGELOG_END
2020-12-09 15:34:38 +00:00
Moritz Kiefer
5f1d5ad118
Fix prettyprinting in DAML REPL (#8214)
We used a weird mix of prettyprinting and string concatenation. This
breaks as soon as you have a line break somewhere because indentation
is messed up at that point. This PR fixes that by consistently (at
least more consistently than before) using the prettyprinting lib.

fixes #8213

changelog_begin

- [DAML REPL] Fix a bug where bindings with very long types sometimes
  resulted in parse errors on following lines. See #8213

changelog_end
2020-12-09 15:24:43 +01:00
Samir Talwar
4e2e39f871
kvutils: Add a test case for conflicts during fetching by key. [KVL-759] (#8204)
* kvutils: Verify the engine handles an out-of-date pre-execution.

* kvutils: Add a test case for conflicts during fetching by key.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: In KVTest, use `Reader` instead of `State` where possible.

* kvutils: Pull out helper methods for pre-execution tests.

* kvutils: Use `Inside` to simplify checks in KVUtilsTransactionSpec.
2020-12-09 14:01:07 +00:00
Ognjen Maric
b535c33e4d
Document the tx service behavior for filters and transient contracts (#8016)
Document the tx service behavior for filters and transient contracts

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-09 14:48:39 +01:00
Moritz Kiefer
aa3caec47f
Bump rules_nodejs (#8211)
* Bump rules_nodejs

rules_nodejs finally added support for module remapping via js_library
so we can kill our custom rule for that.

changelog_begin
changelog_end

* Fix windows

changelog_begin
changelog_end

* .

changelog_begin
changelog_end
2020-12-09 10:45:35 +00:00
azure-pipelines[bot]
584cdd49d9
rotate release duty after 1.8.0-snapshot.20201209.5848.0.e27026ee (#8210)
@S11001001 is taking care of 1.8.0-snapshot.20201209.5848.0.e27026ee (#8209), so they get pushed back to the end of the line.

Please do not merge this before #8209.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2020-12-09 08:01:15 +01:00
azure-pipelines[bot]
e27026ee5d
update compat versions for 1.8.0-snapshot.20201208.5840.0.38455e8c (#8206)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2020-12-09 06:41:45 +01:00
Moritz Kiefer
ec0fcb39f1
Fix docs cron again (#8208)
Unfortunately, I missed the fact that we had our own logic for
handling process failures which resulted in uncatchable
exceptions. I’ve changed one place to use the upstream handling and
the other to call `fail` which throws an IOException like I would have
expected.

changelog_begin
changelog_end
2020-12-08 22:06:22 +01:00
Moritz Kiefer
0c7791bc1c
Fix docs cron (#8207)
The change in #8191 to publish daml on sql docs failed because the
versions.json and snapshots.json files don’t exist initially. This PR
fixes that by catching the exception and treating it as an empty file.

changelog_begin
changelog_end
2020-12-08 21:32:33 +01:00
Andreas Herrmann
bd09e8265d
Require authorization on DAR upload endpoint (#8193)
* Test authentication on upload_dar endpoint

changelog_begin
changelog_end

* require authentication on upload_dar endpoint

* push Directive into auth

* Fully upload request before auth redirection

* Make HTTP entity upload parameters configurable

changelog_begin
changelog_end

* Shorten help message

https://github.com/digital-asset/daml/pull/8193#discussion_r538428368

* maxHttpEntityUploadSize as Long

https://github.com/digital-asset/daml/pull/8193#discussion_r538431773

* use DefaultMaxInboundMessageSize for DefaultMaxHttpEntityUploadSize

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-12-08 18:09:06 +01:00
Samir Talwar
c588b5cc34
kvutils: Extract a SimplePackage class from TestHelpers. [KVL-759] (#8203)
* kvutils: Simplify KVTest a little.

* kvutils: Split out a SimplePackage manager from TestHelpers.

In an effort to clean up the TestHelpers and associated calls, I have
extracted a class that takes the additional contract data type _once_.
It also provides a useful place to move associated methods and values.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-08 16:08:30 +00:00
Samir Talwar
f7465f0646
kvutils: Extract out a trait from StateUpdates. (#8201)
This allows us to mock the object without resorting to mocking final
classes, which is _hella_ flaky.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-08 15:33:35 +00:00
Moritz Kiefer
f66166ec95
Release 1.8 RC (#8200)
changelog_begin
changelog_end
2020-12-08 16:20:37 +01:00
Remy
8f3c6a4494
LF: Add version directly in GenNode (#8154)
CHANGELOG_BEGIN
CHANGELOG_END
2020-12-08 16:12:12 +01:00
Moritz Kiefer
38455e8ca9
Fix Scala version overwrite (#8197)
The old format changed and resulted in that silently being ignored.

changelog_begin
changelog_end
2020-12-08 13:52:41 +00:00
Samir Talwar
fdde69a599
kvutils: Throw if CommitContext#read encounters missing input state. [KVL-757] (#8194)
* kvutils: Throw if CommitContext#read encounters missing input state.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Revert CommitContext#get.

* ledger-on-memory: The pre-execution tests are no longer flaky.

* kvutils: Update the Scaladoc for CommitContext to mention exceptions.

* kvutils: Moar CommitContext testing.
2020-12-08 13:25:11 +00:00
Samir Talwar
73514c53ea
kvutils: Fix a flaky batching queue test. (#8199)
This changes the API so that we can wait for the queue to finish before
asserting that it is, indeed, finished.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-08 13:17:26 +00:00
tudor-da
26b1b3dd13
Disable asynchronous commits in JdbcIndexer (#8195)
CHANGELOG_BEGIN
[JdbcIndexer] Asynchornous commits are disabled.
CHANGELOG_END
2020-12-08 13:05:42 +00:00
Moritz Kiefer
9447fdadca
Clarify DAML Script docs (#8198)
"UI testing" has caused some confusion since DAML Script clearly
doesn’t replace something like puppeteer.

changelog_begin
changelog_end
2020-12-08 12:54:15 +00:00
Moritz Kiefer
880d290285
Publish DAML on SQL Docs (#8191)
changelog_begin
changelog_end
2020-12-08 13:49:37 +01:00
Samir Talwar
0ee2159218
KVUtils: If there is a missing input state during model conformance validation, report "Disputed". [KVL-757] (#8180)
* kvutils: If there is a missing key, print the key.

* kvutils: Log retries during pre-execution.

* kvutils: Handle `Err.MissingInputState` in model validation.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Fetch contracts specified by contract keys for pre-execution.

This ensures that we can generate an appropriate read set even if
there's contention.

* kvutils: Make TestHelper a little more consistent.

* kvutils: Test generating read sets for inconsistent contract keys.

I have verified that this fails on `master` and passes on this branch.

* kvutils: Pull out helpers from PreExecutingSubmissionValidatorSpec.
2020-12-08 11:14:28 +00:00
Bernhard Elsner
7f81db4fc4
Add build rule for DAML Driver for PostgreSQL docs (#8188)
* Add build rule for DAML Driver for PostgreSQL docs

CHANGELOG_BEGIN
CHANGELOG_END

* Reimplement some JS functions

* fmt

changelog_begin
changelog_end

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-12-08 10:19:48 +00:00
Andreas Herrmann
59dae3c97b
Handle admin claims in OAuth2 test server (#8185)
* Middleware test resources

- Make OAuthServer accessible
- Don't access suiteResource directly in the test-suite

* OAuth2 test server test resources

- Make Server accessible
- Don't access suiteResource directly in the test-suite

* Use a deny list in OAuth2 test server

It is simpler to configure a set of disallowed parties and extend it on
demand than switching between blanket approval and allow list modes.

changelog_begin
changelog_end

* Handle admin claims in test server

Requests with admin claims are granted by default and denied if admin
requests have been disallowed.

* reset admin state in test fixtures

* test admin tokens in OAuth2 test server

* test admin claims in OAuth2 middleware

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-12-08 10:16:29 +00:00
Andreas Herrmann
e1b1e4d646
Wait for trigger shutdown before restart (#8186)
changelog_begin
changelog_end

This is to avoid a race condition where the old trigger runner may not
yet have completed shutdown by the time that the token has been
refreshed and we attempt to start a new trigger runner.

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-12-08 09:43:51 +00:00
Moritz Kiefer
7e05dc7932
Upgrade rules-scala and scalatest (#8187)
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.

This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.

Apologies for the giant PR, I don’t see a way to keep it smaller.

changelog_begin
changelog_end
2020-12-08 06:59:23 +01:00
Brian Healey
ca294eb14d
add blackduck scan to run on master (#6130) (#8161)
* add blackduck scan to run on master (#6130)

* add blackduck scan

* disable go scanning
exclude entire language-support/ts directory for node scanning
break to multiple lines to make command line params easier to parse

* Increase timeout for blackduck binary scan

* update blackduck scan config

* remove some exclusions, force python3

* exclude GO until path to go executable can be resolved

* added readme explanation of why we want this file

* fail in case of policy violation

* ensure haskell bazel scan completes before running second round scan for bazel jvm and node and other langs

* trigger notices file gen to ensure BOM complete

* remove trailing end of lines

* run with latest detect version and unique code location name changes to wrapper script

* Add blackduck to daily compat job

* DO NOT MERGE: condition false to disable other jobs for testing

* remove parameters not available to cronjob

* Revert changes to regular CI pipeline

CHANGELOG_BEGIN
CHANGELOG_END

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>

* Do not get branch name from variable

* Upgrade com.fasterxml.jackson.core:jackson-databind to 2.12.0 to address security vulnerability

* Remove disabling of other jobs, set to branch to be used on prod runs

* Apply suggestions from code review

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Address code review comments

* Updated NOTICES file

* Run bazel build, update NOTICES file

* Correct dade-assist

* do not have perms to pipe to dev/null

* Add md file explaining how to update NOTICES file

* Add instructions for running blackduck locally

* Add a link to full security-blackduck readme

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
2020-12-07 19:59:39 +00:00
Andreas Herrmann
4ac7b3f22d
Separate auth API from implementation and testing code (#8183)
* Rename and separate auth targets and packages

Split out the API types `Request`/`Response` into their own Bazel
targets, so that the trigger service does not have to depend on the full
middleware and OAuth2 test server targets.

Also rename packages: Generic auth code goes into `com.daml.auth`,
OAuth2 specific code goes into `com.daml.auth.oauth2` or
`com.daml.auth.middleware.oauth2`.

changelog_begin
changelog_end

* Remove empty test-case

This is a left-over from #7226.

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-12-07 16:17:46 +01:00
Robert Autenrieth
56f5cf107b
Prepare the ledger API for multi-party submissions [KVL-710] (#8125)
* Add multi-party submissions to the ledger API

CHANGELOG_BEGIN
* [Ledger API] Command submission requests now contain new optional
  fields used for multi-party submissions. Such submissions currently
  return UNIMPLEMENTED errors, they will be enabled in the future.
CHANGELOG_END

* Adapt Haskell bindings

* Handle the new ledger API fields

* Fix SubmitAndWaitDummyCommand

* Remove unused methods

* Redesign multi-party auth tests

* Remove direct access of request party

* Apply review comments

* Improve protobuf comments

* Multi-party tracker map

* Fix validation logic

* Consistent metric naming
2020-12-07 16:17:05 +01:00
Sofia Faro
84e5881236
Add serializability check for exceptions. (#8151)
* Add serializability check for exceptions.

Part of #8020

changelog_begin
changelog_end

* typo
2020-12-07 14:44:24 +00:00
Remy
ab43da3f7a
Speedy: Add a built to optimize cons with 1 elems (#8177)
We have a builtin SBConsMany use to concatenate a list with a prefix
of variable size. Its appear that most of the concatenation happens
with only one element. For this reason we add a new builtin to handle
the most efficiently possible the case with 1 element.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-07 15:13:33 +01:00
Richard Kapolnai
158e866f24
Improve consistency of describing consuming and nonconsuming choices (#8179)
* slightly improve consistency of describing consuming and nonconsuming choices

* CHANGELOG_BEGIN
CHANGELOG_END
2020-12-07 14:33:56 +01:00
Samir Talwar
88cb4c0981
daml-lf: Include position information in the testing parsers. (#8182)
So it's easier to figure out where I've gone wrong.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-07 12:33:00 +00:00