Commit Graph

8535 Commits

Author SHA1 Message Date
Marton Nagy
9350632eae
Fix releasing of resources in case connection initialization failed (#11915)
Previously InitHookDataSourceProxy was not closing the connection if initialization failed,
which causing worker-locks to stuck on PostgreSQL side, preventing subsequent HA initialization.
This PR fixes by closing explicitly the connection, if initialization failed.

changelog_begin
changelog_end
2021-11-30 18:21:37 +01:00
Hubert Slojewski
e1559af313
Update ModelConformanceValidator comments and prevent them from getting outdated easily (#11924) 2021-11-30 16:54:00 +00:00
nickchapman-da
16a41f7da2
Avoid package validation in speedy compilation benchmark. (#11927)
changelog_begin
changelog_end
2021-11-30 16:50:42 +00:00
Moritz Kiefer
16135e6f7c
Limit supported input versions in damlc to >= LF 1.8 (#11905)
* Limit supported input versions in damlc to >= LF 1.8

1.8 was the version that introduced type synonyms, we really don’t
gain much by dropping more since data-dependencies mainly depends on
that. and this provides for a very natural upgrade path for users
where pretty much everyone should be able to upgrade directly to SDK
2.0 without having to go through intermediate versions.

changelog_begin

- [Daml Compiler] The supported input LF versions for
  data-dependencies are now limited to LF 1.8 and newer.

changelog_end

* fix some tests

changelog_begin
changelog_end

* Drop export 1.6 tests

changelog_begin
changelog_end

* Drop daml2js support for LF < 1.8

changelog_begin

- [Daml2js] DARs with LF version < 1.8 are no longer supported.

changelog_end

* .

changelog_begin
changelog_end

* bash is bad, stop using it

changelog_begin
changelog_end

* .

changelog_begin
changelog_end
2021-11-30 17:24:08 +01:00
nickchapman-da
0ee4154972
Use Absolute-indexes as keys for the Env-mapping during closure-conversion (#11912)
* Use Absolute-indexes as keys for the Env-mapping during closure-conversion.

Do runtime check to confirm behaviour matches the existing Relative-indexes.

changelog_begin
changelog_end

* remove quadratic shift!

remove (dev)pretty-print code
remove relative-index keys from Env-mapping
remove runtime *diff* check
increase depth for stack-safety tests

* improve/simplify indexing calculation for Env-keys
2021-11-30 16:07:17 +00:00
Sofia Faro
1d7bca801e
Add optional typerep argument in UExerciseInterface. (#11910)
* Add type rep argument for interface exercises.

(Still WIP.)

Part of #11703. Fixes the order in which errors are raised ("wrong type"
takes priority over "does not implement interface"). This PR also simplifies
ExerciseInterface by making the guard mandatory, otherwise there's too
many variations. We can revisit that later if we want.

changelog_begin
changelog_end

* haskell side

* dont throw exception in checkTemplateId

* scalafmt

* evidence security

* fix TypingSpec test
2021-11-30 15:10:33 +00:00
Hubert Slojewski
c2c22f8dea
kvutils: Protos no longer depend on the Daml-LF transaction proto [KVL-1166] (#11909)
CHANGELOG_BEGIN

- [Integration Kit] kvutils protos no longer depend on the Daml-LF transaction proto

CHANGELOG_END
2021-11-30 15:25:18 +01:00
pbatko-da
5641948410
[Docs] Add labels to error codes to support references to them (#11913)
CHANGELOG_BEGIN
CHANGELOG_END
2021-11-30 14:30:58 +01:00
Moritz Kiefer
0e776769ee
Update protobuf docs template to handle oneOf (#11887)
Builds upon #1180 to render the oneof fields a bit nicer.

Note that we also need to kill newlines in the descriptions. That’s
not an issue in the plugin but in how rst renders tables.

fixes #11761

changelog_begin
changelog_end
2021-11-30 13:22:57 +01:00
Andreas Lochbihler
5a9481f9d8
unify heading markup according to README.md (#11919)
CHANGELOG_BEGIN
CHANGELOG_END
2021-11-30 11:32:09 +01:00
nicu-da
61334cff77
kvutils - Add Writer which can handle deduplication periods as offsets [KVL-1172] (#11900)
changelog_begin
changelog_end
2021-11-29 10:28:07 -08:00
Sofia Faro
0b9d57b95e
Add ContractDoesntImplementInterface error. (#11884)
* Add a ContractDoesntImplementInterface error

Part of #10810 and #11703

changelog_begin
changelog_end

* use inside

* rename error
2021-11-29 15:54:31 +00:00
Andreas Lochbihler
49e5d415f6
align index.rst files for HTML and PDF (#11907)
CHANGELOG_BEGIN
CHANGELOG_END
2021-11-29 14:48:20 +01:00
Moritz Kiefer
dbbb05fe2e
Split daml-lf encode/decode Haskell libraries (#11906)
* Split daml-lf encode/decode Haskell libraries

I was working on limiting the supported input versions in damlc. This
requires the list of stable package ids so we can allow those to be
decoded even if they are older.

However that leads to a cyclical dependency:
To get the package ids of the stable packages, I need to encode
them. The version check is in the decoder. If those live in the same
package, I now end up with a cyclical dependency and
Bazel (understandably) gets very sad.

This PR splits decoding/encoding into two libraries with the
generation of stable packages only depending on encoding.

changelog_begin
changelog_end

* Fix dependants

changelog_begin
changelog_end
2021-11-29 12:49:02 +00:00
Robin Krom
e5d3902372
iface: support for fixed choices in TS codegen (#11630)
This adds support for fixed choices to the TS codegen. However,
currently this fails because the JSON API does not accept interface
template ID's when exercising a choice.

CHANGELOG_BEGIN
CHANGELOG_END
2021-11-29 08:07:36 +00:00
Moritz Kiefer
31cc540cf9
Turn package name & version warnings into an error (#11859)
* Turn package name & version warnings into an error

fixes #11321

changelog_begin
changelog_end

* .

changelog_begin
changelog_end
2021-11-26 18:46:33 +00:00
Gary Verhaegen
4e50060021
self-service compat: set branch name to not main (#11902)
The "tell slack about failures" step is conditional on the branch name
being main; for whatever reason if we don't supply a branch name Azure
sets it to `main`, regardless of the fact that the commit is not in that
branch at all.

CHANGELOG_BEGIN
CHANGELOG_END
2021-11-26 16:50:53 +00:00
nickchapman-da
2f4aa4797e
refactor to avoid impossible code path (#11901)
changelog_begin
changelog_end
2021-11-26 15:46:26 +00:00
Gary Verhaegen
a81995c82e
switch dev images to Temurin (#11895)
The AdoptOpenJDK project has been renamed to Adoptium so they renamed
the adoptopenjdk images to Temurin.

CHANGELOG_BEGIN
CHANGELOG_END
2021-11-26 14:45:35 +00:00
Moritz Kiefer
f3a0e2eba1
Set scalafmt dialect explicitly (#11898)
Scalafmt started issuing warnings because we don’t do this since our
latest nixpkgs upgrade which included a scalafmt upgrade.

changelog_begin
changelog_end
2021-11-26 13:57:57 +00:00
mziolekda
60e372dcd2
Don't run pruning tests on H2, they time-out (#11897)
CHANGELOG_BEGIN
CHANGELOG_END
2021-11-26 13:06:03 +00:00
Remy
58e69ade1a
LF: replace "dev" LF version by "1.dev" in bazel files (#11894)
the more consistent, as asked by Moritz in review of #11820

CHANGELOG_BEGIN
CHANGELOG_END
2021-11-26 12:23:17 +00:00
nickchapman-da
8ef348dab0
Use absolute stack locations in SExpr1 (#11877)
* SExpr1.SELocS - carry relBad/absGood - abs unused so far

* compute/pass SELocS.abs in ClosureConversion, and check in Anf that it matches the reconstructed value

CHANGELOG_BEGIN
CHANGELOG_END

* Remove relative stack locations. Rename as SELocAbsoluteS. Simplify Anf. Remove shiftLoc in ClosureConversion.
2021-11-26 12:10:30 +00:00
azure-pipelines[bot]
071bcf7639
update NOTICES file (#11892)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-11-26 11:12:28 +00:00
nicu-da
a1705d669a
participant-state - Add an implicit logging context to the write service [kvl-1072] (#11838)
changelog_begin
changelog_end
2021-11-26 11:12:17 +00:00
Moritz Kiefer
9ff64f7faa
Change daml script’s sleep to sleep for a minimum amount of time (#11886)
* Change daml script’s sleep to sleep for a minimum amount of time

We’ve seen a few flaky test failures where we slept for less than the
expected amount of time which isn’t what we want. We definitely cannot
guarantee an exact sleep time but at least a minimum.

changelog_begin
changelog_end

* Update daml-script/runner/src/main/scala/com/digitalasset/daml/lf/engine/script/ScriptF.scala

Co-authored-by: Remy <remy.haemmerle@daml.com>

Co-authored-by: Remy <remy.haemmerle@daml.com>
2021-11-26 11:12:08 +00:00
Moritz Kiefer
132c277bda
Add a Canton sandbox to the SDK (#11881)
This is only the first step so we have something to use in tests,
eventually `daml sandbox` should map to this Sandbox.

part of #11831

changelog_begin
changelog_end
2021-11-26 11:27:41 +01:00
Moritz Kiefer
68a23436e8
Only run self-service compat job on PRs (#11893)
This is currently failing for the notice file update PR. While we
could try to hack around the git commands to do something sensible, I
don’t really see the point. We really only need this for PRs.

changelog_begin
changelog_end
2021-11-26 10:26:21 +01:00
pbatko-da
c27406cdcf
[DPP-762][Self-service error codes] Automate generation of inventory of error categories. #11879
Other changes:
- Fix ErrorCodeDocumentationGeneratorSpec not being in any bazel target.
- (partially) Fix inconsistent usage of deprecated, Deprecation and DeprecatedDocs annotation.

CHANGELOG_BEGIN
CHANGELOG_END
2021-11-26 10:22:14 +01:00
nicu-da
13797229cf
Adapt the compatibility exclusions (#11872)
changelog_begin
changelog_end
2021-11-26 10:07:02 +01:00
Remy
d66ecc90c9
LF: Drop Archive Snapshot for LF < 1.14 (#11820)
CHANGELOG_BEGIN
CHANGELOG_END
2021-11-26 09:18:52 +01:00
tudor-da
abc141b57e
Increase pruning tests timeout (#11891)
CHANGELOG_BEGIN
CHANGELOG_END
2021-11-26 08:16:35 +00:00
Andreas Herrmann
66b407467f
Update protobuf docs plugin (#11880)
* Try to upgrade protobuf docs plugin

changelog_begin
changelog_end

* Fix extension number 65020 is already registered

Building `//ledger-api/grpc-definitions:ledger-api-docs` [failed
with](https://github.com/digital-asset/daml/issues/11761#issuecomment-978947565)
```
panic: proto: extension number 65020 is already registered on message google.protobuf.FieldOptions
```

Go dependencies are now pulled in via Gazelle. By default Gazelle will
generate new proto rules for any `.proto` files encountered in third
party Go dependencies. However, many of these already have pregenerated
`.pb.go` files generate with the appropriate configuration.

The problem can be avoided by configuring Gazelle to not generate new
proto rules, but instead use pre-existing `.pb.go` files.

For reference the field number is set in
[go-proto-validators](32a686adf8/validator.proto (L19))
which is an indirect dependency through protoc-gen-doc.

In this case we need to update protoc-gen-validate to v0.6.2 to include
4f41f10dde
which fixes unknown label errors.

* ./fmt

* Expose gRPC status.proto for Haskell bindings

* Update Gazelle to support embedsrcs on Windows

`protoc-gen-doc` relies on `go:embed` file embedding
2dde01902b/resources.go (L8).
Gazelle supports `embedsrcs`, however, it did not generate the attribute
correctly on Windows due to the different directory separator. This is
fixed in https://github.com/bazelbuild/bazel-gazelle/pull/1101.

* Add gazelle to compatibility workspace

It's loaded into `@daml`'s top-level `BUILD` file and ends up being a
dependency of the compatibility workspace as well.

* shift go_googleapis import

* Delete dead code

protobuf is imported transitively.

* Document how to add Go dependencies

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-11-25 20:51:24 +00:00
Remy
b0dda53868
LF check stable proto with buf and md5sum. (#11888)
The resulting tests should be faster and more reliable

fixes #11443

CHANGELOG_BEGIN
CHANGELOG_END
2021-11-25 20:23:46 +00:00
mziolekda
056fc52706
Log while processing base64 encoded server key [DPP-761] (#11835)
* Log while processing base64 encoded server key

CHANGELOG_BEGIN
CHANGELOG_END

* address review comments

* demote logs from error to debug

* format
2021-11-25 18:02:35 +00:00
Gary Verhaegen
dbda67ba81
bump JVM in Docker image (#11883)
Please don't depend on this, it breaks all the time.

CHANGELOG_BEGIN
CHANGELOG_END
2021-11-25 18:08:38 +01:00
Kamil Bozek
f69bd688cc
ledger-api-bench-tool: Fix flaky MetricsCollectorSpec (#11750)
* Renamed MetricReporter to MetricFormatter

* Make MetricsCollector communicate via messages only, not by logging

* Logging reports from MetricManger

* Cancel periodic report request when stream is done

* Removed debug sleep

* Fix error when defining reporting period less than one second

* Fixed flaky MetricsCollectorSpec test

CHANGELOG_BEGIN
CHANGELOG_END
2021-11-25 17:39:30 +01:00
Moritz Kiefer
cb758e8d91
Fix call to experimental interface signatory builtin (#11882)
Turns out if you mistype the name it doesn’t work, who would have
thought.

changelog_begin
changelog_end
2021-11-25 16:33:18 +00:00
Sofia Faro
024400b9e4
Error when fetching the wrong template id (via fetch by interface). (#11862)
* Prevent wrongly typed fetch by interface.

When doing a "fetch by interface" command with a known template id,
error out with a WronglyTypedContract if the fetched contract has
a different template id. This doesn't affect daml, only affects
replays, so it's rather minor. I also enabled the engine test that
caught this.

Part of #11703, follow up to #11836.

changelog_begin
changelog_end

* strengthen test output checks
2021-11-25 16:19:36 +00:00
Sofia Faro
0852c8f6fa
Make DA.List.Total return Optional instead (#11878)
* Make DA.List.Total return Optional instead.

Instead of using ActionFail or CanAbort, return an Optional. The library
user can then convert this into the appropriate error if they wish.

This PR also removes DA.Optional.Total (and the deprecated DA.Maybe.Total)
for having questionable utility.

changelog_begin

- [Daml Standard Library] DA.List.Total functions now return Optional
  instead of being polymorphic in the return type. DA.Optional.Total
  has been removed.

changelog_end

* update Examples.daml
2021-11-25 14:33:48 +00:00
Victor Peter Rouven Müller
df373466dc
[JSON-API] Add query store metrics (#11809)
* Add query store metrics

CHANGELOG_BEGIN

- [JSON-API] added metrics to separately track:
    - time taken to update query-store ACS (from ledger)
    - lookup times for the query store

CHANGELOG_END

* Apply review comment
2021-11-25 13:04:28 +00:00
Sofia Faro
2f8f69e7cb
Drop DA.Next.Set and DA.Next.Map (#11864)
* Drop DA.Next.Set and DA.Next.Map

Fixes #11527. Also removed a bunch of unnecessary CPP, though there's
still a lot more CPP to remove.

changelog_begin

- [Daml Standard Library] DA.Next.Map and DA.Next.Set have been removed
  after being deprecated since Daml-LF 1.11

changelog_end

* Drop a deriving MapKey

* update unstable-types test
2021-11-25 13:02:08 +00:00
pbatko-da
5f3a4d2067
[Self-service error codes] Fix section numbering in pdf for error codes section by moving it a level higher. (#11867)
Also make section header markers consistent docs' reamde.

CHANGELOG_BEGIN
CHANGELOG_END
2021-11-25 13:46:12 +01:00
tudor-da
cf3ac011ca
[Self-service error codes] Do not return error code id and definite_answer in metadata for security sensitive errors (#11828)
* [Self-service error codes] Do not return error code id in metadata

CHANGELOG_BEGIN
CHANGELOG_END

* Do not propagate definite_answer
* Additionally, remove redundant context info from error codes

* Fix tests

* Create IndexDbException as a specialization
* Used to globally define error codes returned by the persistence layer
that are logging
2021-11-25 12:42:29 +01:00
Simon Meier
026b92a8b1
Add gRPC definitions for participant user management service (#11818)
* Introduce definitions for experimental user management service.

Only gRPC definitions are added. No new service is exposed by the Ledger
API server.

CHANGELOG_BEGIN

- [Ledger API] Introduce gRPC definitions for experimental user
  managament service to manage users and their rights for interacting
  with the Ledger API served by a participant node.

CHANGELOG_END
2021-11-25 12:26:47 +01:00
Andreas Herrmann
2fde30d916
Disable writing volatile bits in Scala statsfile (#11875)
Make Scala targets deterministic by disabling writing milliseconds
timestamps into the statsfile.

Before (cache disabled)
```
$ n=1; bazel clean; bazel build //libs-scala/adjustable-clock --execution_log_json_file=execlog$n.json; rsync -aL bazel-bin output$n
$ n=2; bazel clean; bazel build //libs-scala/adjustable-clock --execution_log_json_file=execlog$n.json; rsync -aL bazel-bin output$n
$ diff -u execlog1.json execlog2.json
--- execlog1.json       2021-11-25 09:53:45.638891575 +0100
+++ execlog2.json       2021-11-25 10:01:52.831225558 +0100
@@ -311392,7 +311392,7 @@
   }, {
     "path": "bazel-out/k8-opt/bin/external/io_bazel_rules_scala/third_party/dependency_analyzer/src/main/scala_version.statsfile",
     "digest": {
-      "hash": "1c535baa94812c16d33b6b3dcf2dec8e12b77e3a28cd1280ca2c9678840c4de6",
+      "hash": "a7f6f1b168649c1c5346ff92c69f0095b68781690e4b3f33321d8f35af8b3d77",
       "sizeBytes": "16",
       "hashFunctionName": "SHA-256"
     }
@@ -311705,7 +311705,7 @@
   }, {
     "path": "bazel-out/k8-opt/bin/external/io_bazel_rules_scala/third_party/dependency_analyzer/src/main/dependency_analyzer.statsfile",
     "digest": {
-      "hash": "b3af662512a906ec56abfb222f43af189b88b00ae0449bd44c25b7e90689c4db",
+      "hash": "d6c6e2a36754efea4863593a838fbfb500a66e0150d3c3e2e2490ad65c22a2cd",
       "sizeBytes": "16",
       "hashFunctionName": "SHA-256"
     }
@@ -312025,7 +312025,7 @@
   }, {
     "path": "bazel-out/k8-opt/bin/libs-scala/adjustable-clock/adjustable-clock.statsfile",
     "digest": {
-      "hash": "4326088227e5ecf2c21e76ddb6d77c9692ef1689dd146f65ec1e75c235382251",
+      "hash": "8a6b23f3c64ce3ff171772808ca0064af52bc145a32d1b6398de124aeb813c97",
       "sizeBytes": "16",
       "hashFunctionName": "SHA-256"
     }
$ diff -r output1 output2
diff -ur output1/bazel-bin/external/io_bazel_rules_scala/third_party/dependency_analyzer/src/main/dependency_analyzer.statsfile output2/bazel-bin/external/io_bazel_rules_scala/third_party/dependency_analyzer/src/main/dependency_analyzer.statsfile
--- output1/bazel-bin/external/io_bazel_rules_scala/third_party/dependency_analyzer/src/main/dependency_analyzer.statsfile      2021-11-25 09:53:41.706824833 +0100
+++ output2/bazel-bin/external/io_bazel_rules_scala/third_party/dependency_analyzer/src/main/dependency_analyzer.statsfile      2021-11-25 10:01:49.047160467 +0100
@@ -1 +1 @@
-build_time=2507
+build_time=2522
diff -ur output1/bazel-bin/external/io_bazel_rules_scala/third_party/dependency_analyzer/src/main/scala_version.statsfile output2/bazel-bin/external/io_bazel_rules_scala/third_party/dependency_analyzer/src/main/scala_version.statsfile
--- output1/bazel-bin/external/io_bazel_rules_scala/third_party/dependency_analyzer/src/main/scala_version.statsfile    2021-11-25 09:53:39.110780774 +0100
+++ output2/bazel-bin/external/io_bazel_rules_scala/third_party/dependency_analyzer/src/main/scala_version.statsfile    2021-11-25 10:01:46.427115402 +0100
@@ -1 +1 @@
-build_time=4162
+build_time=3946
diff -ur output1/bazel-bin/libs-scala/adjustable-clock/adjustable-clock.statsfile output2/bazel-bin/libs-scala/adjustable-clock/adjustable-clock.statsfile
--- output1/bazel-bin/libs-scala/adjustable-clock/adjustable-clock.statsfile    2021-11-25 09:53:45.282885532 +0100
+++ output2/bazel-bin/libs-scala/adjustable-clock/adjustable-clock.statsfile    2021-11-25 10:01:52.511220053 +0100
@@ -1 +1 @@
-build_time=3158
+build_time=3100
```

After (cache disabled)
```
$ n=3; bazel clean; bazel build //libs-scala/adjustable-clock --execution_log_json_file=execlog$n.json; rsync -aL bazel-bin output$n
$ n=4; bazel clean; bazel build //libs-scala/adjustable-clock --execution_log_json_file=execlog$n.json; rsync -aL bazel-bin output$n
$ diff -u execlog3.json execlog4.json
$ diff -ur output3 output4
```

See https://github.com/bazelbuild/rules_scala/pull/1298

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-11-25 11:01:17 +00:00
Andreas Herrmann
4ed9deda39
Remove xxd from dev-env (#11876)
The corresponding Nix attribute is missing
```
$ xxd
[dev-env] Building tools.xxd...
error: attribute 'xxd' in selection path 'tools.xxd' not found
[dev-env] Build of tools.xxd has failed!
```

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-11-25 11:01:04 +00:00
Gary Verhaegen
eaded41aca
remove mergify (#11866)
It's been working pretty well for us, but our usage is now completely
covered by GitHub's auto-merge feature, which has the advantage of not
requiring us to duplicate our branch protection rules.

This will also reduce our security audit surface as that's one fewer
provider to scan (not that we've found anything wrong with them so far,
but still, less work in the future).

CHANGELOG_BEGIN
CHANGELOG_END
2021-11-25 11:36:24 +01:00
Andreas Lochbihler
3cd5028b61
fix a few more things in the daml-lf spec (#11851)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Remy Haemmerle <Remy.Haemmerle@daml.com>
2021-11-25 10:46:19 +01:00
Marton Nagy
beca0ee3e9
Refactor StandaloneApiServer factory (#11842)
To enable direct IndexServer building (and enable two step ApiServer build process)
This will be needed for KV WriteService proxy: a wrapper is being build to serve deduplication time conversion, which needs access to completion service.

changelog_begin
changelog_end
2021-11-25 10:12:23 +01:00