In many network setups, there will be a more direct route for the
trigger service to contact the auth middleware than going back through
the frontend's public IP address (and possibly thus also through
intermediaries like an nginx reverse proxy etc.). In _some_ network
setups, it may not even be possible for the trigger service to reach the
auth middleware through its externally-visible address.
This PR caters to these cases by allowing the trigger service to use two
separate addresses for the auth middleware, an internal one the trigger
service uses when it needs to talk to the auth middleware, and an
external one used in generating URLs for external clients.
This is backwards-compatible: if the old option is used, we simply use
the same value for both.
CHANGELOG_BEGIN
- The Trigger Service can now accept separate `--auth-internal` and
`--auth-external` CLI arguments, where `--auth-internal` is the
address used by the Trigger Service to reach the Auth Middleware
directly, and `--auth-external` is the address the Trigger Service uses
in generated URLs sent back to the client. The `--auth` option remains
and keeps working as before, setting both internal and external
addresses to the same given value.
CHANGELOG_END
As I was adding support to a sample app for the third time, I decided I
might as well clean it up a bit and add support for everyone.
This is completely optional, activated by (build-time) environment
variables, and should be trivial to remove for users who don't care
about it. In an ideal world we'd have options to `daml new` and this
would be one, but I think adding it out-of-the-box is a better
alternative than letting people figure this out on their own.
CHANGELOG_BEGIN
- The `create-daml-app` template now includes support for a third
authentication scheme (in addition to the existing "dev mode" and Daml
Hub support): Auth0.
CHANGELOG_END
* Support adding tests as an hidden option
* Simplify existing suites
CHANGELOG_BEGIN
CHANGELOG_END
* Remove stale conformance suites from build.yml
* `--add` -> ``--additional`
* Re-add `--all-tests` as deprecated CLI option to be tested
* Move sandbox-classic pruning test to wall clock again
* Run KVCommandDeduplicationIT for sandbox append-only
* Tidy-up
* Also add participant pruning test to ledger-on-memory/single-participant
* Remove KVCommandDeduplicationIT on ledger-on-memory/append-only
* Run the full suite plus pruning (rather than just pruning) for ledger-on-memory with multiple participants and append-only
* Add KVCommandDeduplicationIT to ledger-on-memory append-only
* Exclude ConfigManagementServiceIT from ledger-on-memory append-only multi-participant
* Tidy-up
* Use KVCommandDeduplicationIT for sandbox-on-x too
* Fix merge
Add max dedup duration arg to all the test suites that include command dedup tests
* Make `--include` and `--additional` mutually exclusive
* Uniform formatting of multi-line strings
* Move exclusions after additions as they are applied last
* Re-disable deduplication test on sandbox with static time
* Re-disable deduplication test on sandbox-on-x
* [Ledger API error codes] ErrorCode interfaces and generator
* Implementation of annotation processor
* Implementation of DocItem generator
* Unit testing of the generator and error code logging
CHANGELOG_BEGIN
CHANGELOG_END
* Addressed review comments
- Add support for specifying either 1.2 or 1.3 as minimum TLS versions for ledger api server.
- Log enabled protocols (~TLS versions) and cipher suites at server and client startup.
- Add integration tests against Sandbox-classic and Sandbox
CHANGELOG_BEGIN
Sandbox: Add CLI flag to select minimum enabled TLS version for ledger API server.
CHANGELOG_END
* Minor tweaks.
* Use deduplication period instead of deduplication time.
* Introduced change ID and consistent use of deduplication duration/period.
* Consistent use of deduplication duration/period.
* Added ALREADY_EXISTS to the described gRPC errors.
* Minor tweak.
CHANGELOG_BEGIN
CHANGELOG_END
* Emphasize that applications should not change the deduplication time upon a resubmission
* Fix definition of change ID and always refer to it
* Try to clarify "Application-specific IDs"
* Make command deduplication explanation unspecific w.r.t. direction
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
* Extract deduplication "features" into a configuration to be used around the tests.
Better naming for assertions that support sync and async deduplication
CHANGELOG_BEGIN
CHANGELOG_END
* Fix broken test and use consistency for tests
As explained in #10853, we recently lost our ES cluster. While I'm not
planning on trusting Google's "rolling restart" feature ever again, we
can't exclude the possibility of future similar outages (without a
significant investment in the cluster, which I don't think we want to
do).
Losing the cluster is not a huge issue as we can always reingest the
data. Worst case we lose visibility for a few days. At least, as far as
the bazel logs are concerned.
Losing the Kibana data is a lot more annoying, as that is not derived
data and thus cannot be reingested. This PR aims to add a backup
mechanism for our Kibana configuration.
CHANGELOG_BEGIN
CHANGELOG_END
* Update daml documentation to reflect the deprecation of the package/internal access token
CHANGELOG_BEGIN
CHANGELOG_END
* drop the whole internal access token section
Following #10763, we drop the ad-hoc builders for `FrontStack`.
* Building a `Fronstack` from individuals elements should be done with
standard scala buidler.
* Building a `Fronstack` from a `TraversableOne` should be done with
the scala 2.13 `.to(FrontStack)` methd
* Building a `Fronstack` from a `ImmArray` should be done with the
`toImmArray` method.
CHANGELOG_BEGIN
CHANGELOG_END
* interfaces: protobuf encoder haskell side
This is the implementation of the protobuf encoder on the haskell side.
CHANGELOG_BEGIN
CHANGELOG_END
* missing decoder cases for added interface data constructor
* Add conformance test for KV committer usage of max deduplication duration as deduplication duration
CHANGELOG_BEGIN
CHANGELOG_END
* Run AppendOnlyKVCommandDeduplicationIT for KV ledgers that use the append-only schema
This only handles the interface definition, not the implementation in
the template. There are also a few rough edges:
1. It maks all choices as consuming.
2. it ignores locations
But for a poc that doesn’t seem too bad.
The tests don’t do anything super useful since the typechecker falls
over but I checked tha tthe generated LF looks more or less reasonable.
changelog_begin
changelog_end
On Sept 8 our ES cluster became unresponsive. I tried connecting to the
machines.
One machine had an ES Docker container that claimed to have started 7
weeks ago and stopped 5 weeks ago, while the machine's own uptime was 5
weeks. I assume GCP had decided to restart it for some reason. The init
script had failed on missing a TTY, hence the addition of the
`DEBIAN_FRONTEND` env var.
Two machines had a Docker container that had stopped on that day, resp.
6h and 2h before I started investigating. It wasn't immediately clear
what had caused the containers to stop.
On all three of these machines, I was abble to manually restart the
containers and they were abble to reform a cluster, though the state of
the cluster was red (missing shards).
The last two machines simply did not respond to SSH connection attempts.
Assuming it might help, I decided to try to restart the machines. As GCP
does not allow restarting individual machines when they're part of a
managed instance roup, I tried clicking the "rolling restart" button
on the GCP console, which seemed like it would restart the machines. I
carefully selected "restart" (and not "replace"), started the process,
and watched GCP proceed to immediately replace all five machines, losing
all data in the process.
I then started a new cluster and used bigger (and more) machines to
reingest all of the data, and then fell back to the existing
configuration for the "steady" state. I'll try to keep a better eye on
the state of the cluster from now on. In particular, we should not have
a node down for 5 weeks without noticing.
I'll also try to find some time to look into backing up the Kibana
configuration, as that's the one thing we can't just reingest at the
moment.
CHANGELOG_BEGIN
CHANGELOG_END
* Always use max_deduplication_duration as deduplication period for committer side deduplication
Doing this we get the following guarantees:
- Deduplication period is constant for all the submissions therefore this guarantees consistency between "backward-looking" and "forward-looking" deduplication
CHANGELOG_BEGIN
kvutils - committer side deduplication always uses max_deduplication_duration + min_skew as a deduplication period for all the requests.
CHANGELOG_END
* Set max_deduplication_duration to 10 seconds for all the conformance tests
* Update KV command deduplication conformance tests to account for max deduplication duration used as deduplication period for all the requests
* Add max deduplication wait for multi-participant conformance test
* Overwrite the submitter info deduplication period in the transaction committer so that we always set the max deduplication duration
* Update ledger/participant-state/kvutils/src/test/suite/scala/com/daml/ledger/participant/state/kvutils/committer/transaction/TransactionCommitterSpec.scala
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Add separate step for overwriting the deduplication period
* Code review - reformatting and renames
* Add comment for setting the deduplication duration
* [docs] Replace AdoptOpenJDK suggestion by Adoptium
In the installation instructions, suggest Adoptium as JDK source
[AdoptOpenJDK has moved to the Eclipse foundation](
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/)
* Without changelog entry
CHANGELOG_BEGIN
CHANGELOG_END
Manual release process. @stefanobaghino-da is in charge of this release.
Commit log (since 1.17.0-snapshot.20210907.7759.0.35a853fd):
```
976ca400eb Force JSON API to refresh packages for GET requests to /v1/query (#10835)
e42cd3a1c6 LF: add interface to scala AST and decoder (#10830)
07b72d0f3e Update Ledger API's proto documentation [KVL-1094] (#10815)
648acf52f8 Fix typo in app-arch.rst (#10798)
dc9fbb5aa9 Stringify errors in create-daml-app before calling alert (#10831)
90b008ef0d Stop using `controller … can` syntax in ledger tests (#10814)
e490b69471 Clarifications of code ownership in /ledger and /ledger-api (#10742)
8cdeb7539a make auth middleware aware that actAs => readAs (#10825)
e263d43aca Ledger API test tool: test that record time is monotonically increasing [KVL-1053] (#10552)
e5793a317b Account for multi participant conformance tests when updating the config [KVL-1092] (#10788)
2e39d7b647 Deprecate `Commands.deduplication_time` field [KVL-1096] (#10802)
7824201b8d Get rid of extra new line and prune precision for all gatling values (#10820)
a03f52a15e Add ledger API tests about command deduplication information in completions [KVL-1057] (#10748)
61a07b1986 Add command line option max-deduplication-duration for sandbox and KV [KVL-1098] (#10816)
0800504955 Fix DuplicateSubmissionID test exclusions (#10819)
1ff34ffbe8 Remove workaround for akka supervisor issue (#10818)
6d347d4e88 Add daml interfaces to the Haskell LF AST (#10811)
cecb27fe16 Stop using controller can syntax in skeleton template (#10799)
8830eaa2ec Drop controller can syntax from compiler tests (#10807)
9038a8051b Update code documentation for `participant-state` API [KVL-1094] (#10812)
60ffb79fb1 Command deduplication - Participant side/committer side command deduplication conformance tests split [KVL-1093] (#10784)
1e6540f8f2 Stop using `controller _ can` syntax in daml-lf tests (#10805)
0db15e6759 Ledger API test tool: ensure that test identifiers are unique (#10804)
f1e378a60b rotate release duty after 1.17.0-snapshot.20210907.7759.0.35a853fd (#10809)
1464177bf8 release 1.17.0-snapshot.20210907.7759.0.35a853fd (#10808)
deb8a8b740 [Divulgence pruning] Daml SDK docs update [DPP-536] (#10764)
8405ad58fd Enhance InMemoryLedger to use the ValueEnricher (#10757)
```
Changelog (since 1.17.0-snapshot.20210907.7759.0.35a853fd):
```
- [JSON API] Fixed a bug that prevented the JSON API to be aware of
packages uploaded directly via the Ledger API.
- The auth middleware will now consider that actAs credentials are valid
for readAs requests.
- [Ledger API Specification] Introduced a new conformance test about recort time in completions being monotonically increasing
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
- [Ledger API Specification] `Commands.deduplication_time` field has been deprecated, please use `Commands.deduplication_duration` instead.
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
[Ledger API Specification] Participant pruning of all divulged contracts is fully implemented: Participant operators can choose to prune all immediately and retroactively divulged contracts, by setting the newly-added prune_all_divulged_contracts flag in the ParticipantPruningService/Prune request.
```
changelog_begin
changelog_end
* Initial changes to add a surrogate_template_id cache to reduce db queries
CHANGELOG_BEGIN
CHANGELOG_END
* refactoring and addition of tests
* Code review based changes to use Contextual Logger and json-api metrics instance
* make max cache entries/size configurable
* Rename cache max entries default variable
* Add failing test that covers the bug we found in #10823
* Fix /v1/query endpoint bug
changelog_begin
- [JSON API] Fixed a bug that prevented the JSON API to be aware of
packages uploaded directly via the Ledger API.
changelog_end
* Stop using `controller … can` syntax in ledger tests
We’re planning to deprecate that in SDK 2.0 to reduce the confusion
around the implicit observer behavior so in preparation for that, this
PR drops the syntax from all ledger tests.
changelog_begin
changelog_end
* shuffle let above usage
changelog_begin
changelog_end
It seems a bit silly that the auth middleware would refuse a token that
has an `actAs` when it needs a `readAs`.
Reference: https://docs.daml.com/app-dev/authorization.html#access-tokens-and-claims
CHANGELOG_BEGIN
- The auth middleware will now consider that actAs credentials are valid
for readAs requests.
CHANGELOG_END