This can cause an issue because we cannot register JMX metrics twice. We
end up with fun exceptions.
The error looks like this:
```
14:37:26.230 [sandbox-worker-grpc-event-loop-e20f0542-e2c0-452d-bf7c-0ba58cda6145-31-1] DEBUG com.codahale.metrics.jmx.JmxReporter - Unable to register timer
javax.management.InstanceAlreadyExistsException: com.daml.platform.sandbox.SandboxServer:name=daml.lapi.reset_service.reset,type=timers
```
Seen in a flaky test on CI.
CHANGELOG_BEGIN
CHANGELOG_END
* Configure a token verifier on the auth middleware
changelog_begin
changelog_end
* Verify the token in the auth middleware
changelog_begin
changelog_end
* Test token verification in auth middleware
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* language: `daml ledger` commands against JSON API
This is the first step in offering the JSON API to run `daml ledger`
commands. First, only the `list-parties` command is implemented. The
other `daml ledger` commands will follow in separate PR's.
CHANGELOG_BEGIN
[Assistant] The `daml ledger list-parties` command can now query the
ledger via the HTTP JSON API instead the gRPC API for known parties on
the ledger.
CHANGELOG_END
Also adds a small test.
changelog_begin
- [DAML Compiler] Similar to functional dependencies, the
compiler now adds overlapping instance pragmas into the
DALF/DAR files. This way, overlapping instances pragmas
will be preserved when importing typeclass instances
via data-dependencies.
changelog_end
Requires authorization for `readAs:<party>` claims for the list and
status endpoints of the trigger service. In case of list the party is
provided by the request entity. However, in case of status the party is
determined by querying for the running trigger instance in the same way
as for the stop endpoint.
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* release: Check that the commit is on a release branch on origin.
I don't want to have to check out the branch locally to release from it.
This also replaces the loop with `grep`.
CHANGELOG_BEGIN
CHANGELOG_END
* release: Use `grep -q` instead of `grep ... > /dev/null`.
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
We use `xargs` a lot but rely on the version on the user's system.
This adds it to dev-env so it's consistent everywhere.
If nothing else, this means we can safely use the `--no-run-if-empty`
flag, which is not available on BSD (and therefore macOS) `xargs`.
CHANGELOG_BEGIN
CHANGELOG_END
* fmt: Do not try and format missing Haskell files.
CHANGELOG_BEGIN
CHANGELOG_END
* fmt: Filter in files for `hlint` instead of filtering out.
* fmt: Run scalafmt against the merge base too.
* fmt: Log "hlint" before running it.
* Replace occurrances of DAML SDK in the docs with appropriate new terms
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestions from code review
Revert two renamings.
* Fix broken internal link
* Add overlap mode annotations during LF conversion.
This PR adds an overlap mode binding `$$om$fFoo...` during LF conversion
per typeclass instance `$fFoo...` that uses an overlap mode pragma. The
idea is that we can then pick these up during data-dependencies
(left to a separate PR).
This PR also adds a test to make sure we're capturing this information
correctly.
changelog_begin
changelog_end
* git shrapnel
* ledger-api-test-tool: Specify the ledger clock granularity prettily.
I find "10s" easier to read than "10000".
CHANGELOG_BEGIN
- [Integration Kit] The ``--ledger-clock-granularity`` option now takes
a time duration (e.g. "10s" or "5m"), rather than an integer number of
milliseconds.
CHANGELOG_END
* ledger-api-test-tool: Default to a ledger clock granularity of 1 second.
The previous default, 10 seconds, means that these tests take way longer
than necessary on fast ledgers (e.g. in-memory or SQL-based ledgers).
The property is still configurable for ledgers that require it.
CHANGELOG_BEGIN
- [Integration Kit] The ledger API test tool now defaults to a ledger
clock granularity of 1 second, not 10, in line with most ledger
implementations.
CHANGELOG_END
* ledger-api-test-tool: Fall back to parsing clock granularity as millis.
So we maintain backwards compatibility for this option.
* ledger-api-test-tool: Remove a debugging fail.
* ledger-api-test-tool: Remove an unused import.
I saw a failure in CI due to timer fluctuation.
> 94 milliseconds was not around 100 milliseconds [95, 1000)
This widens the lower bound from 5ms to 20ms to be safe.
CHANGELOG_BEGIN
CHANGELOG_END
* choice observers, WIP
changelog_begin
changelog_end
fix generator driven test
Node.isReplayedBy, consider observers
add observers to NodeExercise in transaction.proto
add observers to TemplateChoice in Ast.scala
add observers to LF .proto, and Haskell Ast for TemplateChoice
reinstate trailing // for better format
fix validate tests
fix haskell LF decoder when choice-observers field is missing in .proto
fix build
make choice-observers optional in scala AST
make choice-observers optional in Haskell Ast
address comments from Remy and Martin
more review comments
check TransactionVersions.minChoiceObservers in Transaction encode/decode
featureChoiceObservers, and check in haskell type-checker
improve speedy Compiler for empty choice-observers
extend scala LF decoder for optional choice observers
extend scala parser for choices to allow optional choice-observers clause, and test
rename new field in scala Ast -> "choiceObservers"
var rename
extend TypingSpec tests for choice-observers. also add missing negative test for controllers
switch from keyword "ob" to identifier "observers" in scala parser choice syntax
add TODO for featureChoiceObservers to be part of DAML 1.9 (issue 7139)
* replace "NICK" comment markers with "FIXME #7709" comment markers
This PR has been created by a script, which is not very smart and does not have all the context. Please do double-check that the version prefix is correct before merging.
@nickchapman-da is in charge of this release.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
I tried a compiler with these new warnings on a giant codebase and I
turned out they are _extremely_ noisy. There are basically two things we
want to warn about:
1. Usage of the promoted kinds and types coming from the `DA.Generics`
module.
We already have a warning that importing `DA.Generics` will break
data-dependencies. Adding a warning to every single use site of its
exports seems rather excessive and doesn't add any new information.
2. Usage of the `Symbol` kind and type-level strings.
The `Symbol` kind seems to work better with data-dependencies than I
expected although it is erased into kind `*` during conversion to
DAML-LF. This is for two reasons: (1) The only _legal_ way to
introduce a type variable of kind `Symbol` is by means of the
`HasField` typeclass (you can technically get your hands on the
`Symbol` kind directly, but you need work around not being able to
import the internal module `GHC.Types`). (2) In data-dependencies,
we reconstruct type variables of kind `*` not as kind `*` but rather
ask GHC to infer their kinds (via
[`UserTyVar`](https://hackage.haskell.org/package/ghc-8.10.1/docs/src/GHC.Hs.Types.html#UserTyVar)).
This means that every type variable that was originally of kind
`Symbol` will be reconstructed at that kind again since the
necessary `HasField` constraint will force GHC to infer it.
I'm totally aware that it looks a lot like this works by accident
but I'm not completely opposed to making it a feature after the
fact. Before we make any decisions in this regard though, we should
decide on what uses of kind `Symbol` we want to make compatible with
data-dependencies. For instance, functions of types like
overField: forall x r a. HasField x r a => (a -> a) -> r -> r
might be something we want to support. We also need to support
constraints of the form `HasField "abc" ...` with a concrete
type-level string everywhere. The current implementation would have
warned about them. Thus, I see two open questions around the
`Symbol` kind: (1) Do we want to allow `HasField x r a`, where `r`
is a type variable and _not_ a type-level string, to be a superclass
constraint? (2) Do we want to allow the same constraint in the
context of a typeclass instance?
One thing we should definitely not allow is using the `Symbol` kind
directly under its name `GHC.Types.Symbol`, which is always in scope
for our record preprocessor. I will add an error when users try to
access that kind in a separate PR.
CHANGELOG_BEGIN
[DAML Compiler] The warning about advanced types combined with type
classes in a way that is supported by data-dependencies has been removed
again.
CHANGELOG_END
The previous restriction was both too lax and too strict now that we
have multi pary queries:
1. It allowed a party in `readAs` for command submissions which just
fails on the ledger side. Changing this is technically breaking but
only if you used a token that would have been rejected as soon as you
enabled auth so that seems very resonable to break.
2. It didn’t allow extra parties in `readAs`.
This PR switches to requiring exactly one party in `actAs` while
supporting multiple parties in `readAs`.
changelog_begin
- [JSON API] JWTs on command submissions can now contain extra parties
in the `readAs` field. `actAs` is still limited to a single party.
changelog_end
* Add tests for connection failures in the JSON API
This PR adds some toxiproxy based tests to see how the JSON API reacts
if the connection to the ledger is killed. There are a bunch of
inconsistencies here in the tests some of which we might want to
address and the others we should at least document but I’ll leave that
for future PRs.
changelog_begin
changelog_end
* Import HttpServiceTestFixture instead of prefixing
changelog_begin
changelog_end
* Regularize scala mod-parser syntax for controllers in preparation for adding choice-observers.
From keyword "by" to identifier "controllers" + leading ","
changelog_begin
changelog_end
* fix another use of the old "by" syntax -> ", controllers"
* queryContractId and queryContractKey, trivially
* add changelog
CHANGELOG_BEGIN
- [Triggers] Two new functions are available for querying the ACS:
``queryContractId``, for looking up a contract by ID, and ``queryContractKey``
for looking one up by key.
See `issue #7726 <https://github.com/digital-asset/daml/pull/7726>`__.
CHANGELOG_END
* more efficient, direct queryContractId implementation
* flip getContractById's arguments, avoid 'flip'
* test queryContractKey
* test queryContractId
* dev-env: Add a symlink, `dev-env/jdk`, to the current JDK.
This makes configuring IntelliJ IDEA to use the correct JDK much easier.
CHANGELOG_BEGIN
CHANGELOG_END
* dev-env: Document setting up the JDK in IntelliJ IDEA.
GitHub requests that when we use the API without a token, which is what
we do here, we use a user-agent header that allows them to contact us in
case there is an issue they need to discuss. This PR updates that
address, and cleans-up a bit of duplication around it.
CHANGELOG_BEGIN
CHANGELOG_END
This is pretty much a verbatim copy of
https://github.com/digital-asset/daml/pull/7740, but for updates of
structural records instead of projections.
This has no immediate performance impacts since the DAML compiler does
_currently_ not produce any structural record updates. Having both,
projections and updates, have the same runtime characteristics seems to
be a desirable property nevertheless, simply to avoid nasty surprises
should we ever start to use these updates.
Since the compiler does not produce any structural record updates, I
have no way to benchmark this. However, since the code is changed here
in the same way it was changed in the PR mentioned above, I expect the
same saving, which were roughly 80 ns per operation, where the previous
cost of an operation was _at most_ 210 ns.
CHANGELOG_BEGIN
CHANGELOG_END
* resources: Move builders into //ledger/ledger-resources.
Keep the actual constructors in a trait, but instantiate it when working
with ledger code.
This allows us to later introduce an extra "context" type parameter to
ResourceOwner.
* resources-akka: Move the builders in to //ledger/ledger-resources.
* resources: Introduce an abstract `Context` parameter for owners.
This replaces the concrete `ExecutionContext`. While it _can_ be an
execution context, it really doesn't matter as long as we can get at one
somehow.
This is being introduced so we can wrap the context in a container,
either for type tagging or to include extra information.
Because our current context _is_ `ExecutionContext`, and an implicit is
provided to extract it, we can end up with two ways to get the same
value. We use shadowing to prevent this. This problem should go away in
the near future when a new context type is added.
CHANGELOG_BEGIN
- [Integration Kit] The `ResourceOwner` type is now parameterized by a
`Context`, which is filled in by the corresponding `Context` class in
the _ledger-resources_ dependency. This allows us to pass extra
information through resource acquisition.
CHANGELOG_END
* ledger-resources: Move `ResourceOwner` here from `resources`.
* ledger-resources: Remove dependencies from outside //ledger.
* ledger-resource: Wrap the acquisition execution context in `Context`.
So we can add a logging context to it.
* resources: Pass the Context, not the ExecutionContext, to Resource.
* Avoid importing `HasExecutionContext`.
* ledger-resources: Publish to Maven Central.
* resources: Make the small changes suggested by @stefanobaghino-da.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* ledger-resources: Pull out a trait for test resource contexts.
Saves a few lines of code.
* Restore some imports that were accidentally wildcarded.
* resources: Replace an `implicit def` with a couple of imports.
* participant-integration-api: Simplify the JdbcLedgerDaoBackend tests.
Try and use the right execution context where possible.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Speed up projections of structural records
Currently, for every projection of a structural record, we first do a
linear search through the field names to find the index of the field
value in the record value. Then we use this index to get the actual
value. However, since we started sorting the fields in alphabetic order
both in types and in values, this lookup will always yield the same
result for a fixed record type and a fixed field name, regardless of the
value of the record.
This PR changes the AST node for structural record projection to
contain the index of the field within the record as well. This
information is not contained in the DAML-LF archive itself and must
hence be filled in by the DAML-LF type checker.
The only context in which we use structural record prejections are
typeclass method invocations, which do a little bit more than just the
projection. Unfortunately, typeclass method invocations are the small
unit of work I can benchmark using the setup I have. In the end, they
are also the feature our users really care about.
My benchmarks show that the time a single typeclass method invocation
takes has dropped from ca. 210ns to ca. 130ns. That's a speedup of
ca. 1.6x.
CHANGELOG_BEGIN
[DAML Interpreter] Typeclass method invocation is now 1.6x faster.
CHANGELOG_END
* Apply Remy's suggestions
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Make it compile again
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Remy <remy.haemmerle@daml.com>
PR #7732 illustrated a bug in our current changelog check: it only worked
if the PR was up-to-date with `master` at the time of pushing. This was
not the intention. Fortunately, the fix is fairly easy: check out the PR
commit before running the check, rather than run it from the merge
commit (as Azure does by default).
CHANGELOG_BEGIN
CHANGELOG_END
* Functional dependency support in data-dependencies
(WIP)
* Add test.
changelog_begin
- [DAML Compiler] Functional dependencies are now saved
by the DAML compiler in the DALF file / DAR package. When
importing a file with the functional dependency data via
data-dependencies, these will be picked up automatically.
changelog_end
* Move CallStack, HasCallStack docs to DA.Stack
This PR only modifies the docs. In particular, it:
- moves the `CallStack` and `HasCallStack` docs to `DA.Stack`
- gives an example of adding the `HasCallStack` constraint
- hides the rest of `GHC.Stack.Types` from the docs
Rationale: `GHC.Stack.Types` has a lot of `TextLit` stuff that
we don't want people to use directly, and we want people to
import `DA.Stack` instead anyway.
changelog_begin
changelog_end
* commit unsaved changes, doh
* grammar
Updating the expected dignostics for `damlc`'s integration tests can
be quite tedious. This PR adds a `sed` script to simplify it a bit.
See the included readme for details.
CHANGELOG_BEGIN
CHANGELOG_END
Haskell allows for promoting types to the kind level with the
`DataKinds` extension. Since DAML-LF has no similar concept (and
frankly speaking should't have one), this does not work well with
data-dependencies. Unfortunately, we need to enable the `DataKinds`
extension by default in `damlc` since our record system and the
`Numeric` type depend on it. Thus, issuing a warning that the
`DataKinds` is not supported with data-dependencies is not an option
and we need more refined warnings.
Most uses of promoted kinds won't even compile to DAML-LF and there's
no need to warn about those since they cause hard failures. However,
we need some limited support for type-level strings and the `Symbol`
kind, for type-level numbers and the `Nat` kind, as well as for some
stuff from `DA.Generics`. Type-level numbers and the `Nat` kind work
reasonable well since we have them in DAML-LF too. The `Symbol` kind,
and the types and kinds from `DA.Generics` are "erased" during
conversion to DAML-LF by replacing them with either a special `Erased`
type or the star kind. Thus, these entities are not restored properly
with data-dependencies. In combination with type classes this leads
easily to instances that for different types in the original code but
are at conflict during interface reconstruction. Long story short,
erased types/kinds and type classes don't work with data-dependencies.
We have a hack to somehow preserve type-level string since we need that
for our record system and the `HasField` type class. I have no desire
to support this hack for arbitrary type classes coming from the user.
Thus, I consider type-level strings to be erased as well here.
This PR adds a warning whenever a user is attempting to define a type
class that involves a kind that will be erased or an instance that
involves a type that will be erased. The warning tells the user that
this does not work with data-dependencies and is hence not upgradable.
There is one caveat: the `HasField` instances produced by the
propressor fall in the category we should warn about as well. Since not
supporting them would make our record system unusable, we need to give
them special treatment in data-dependencies. Thus, warning about them
would be unjustified noise and we explicitly exclude them in the code
producing the warnings.
CHANGELOG_BEGIN
[DAML Compiler] Issue warning when advanced types are with type classes
in a way that is supported by data-dependencies.
CHANGELOG_END
Stop emitting warnings about un-upgradability when our users use the
`InstanceSigs` or `MultiWayIf` extensions. These are safe.
Also don't issue warnings for `Cpp`, which can only be used internally.
CHANGELOG_BEGIN
[DAML Compiler] Add `InstanceSigs` and `MultiWayIf` to the list of
language extensions that don't cause problems with data-dependencies.
CHANGELOG_END
* Authorize trigger service on middleware
changelog_begin
changelog_end
* Trigger service auth callback handler
* Forward token
* Do not pin the application ID in the access token
The trigger service will assign an individual application ID to each
trigger based on its UUID. Requiring tokens on the granularity of
application IDs would break the idea of storing the token in a cookie to
be able to use it across multiple requests.
changelog_begin
changelog_end
* todo persist trigger token
* Add a state parameter to middleware login
* add documentation comments
* typo
* fmt
* Align Party type between middleware and trigger service
The middleware was using `com.daml.lf.data.Ref.Party` while the trigger
service is using `com.daml.ledger.api.refinements.ApiTypes.Party` which
requires conversions. This aligns the types to avoid such conversions.
* optional application id in oauth2 test server
* align party types
* configure auth middleware in trigger service tests
* handle empty cookie header
* follow redirects in trigger service tests
* keep track of cookies
* keep track of cookies
* Replace any previous Cookie header
Otherwise on old daml-ledger-token cookie might persist and be preferred
over a newly added instance.
* DEBUG
* Configure test ledger client readAs claims
* fmt
* docstrings
* remove debug output
* Avoid endless redirect loops
When the replay still fails to authorize on the middleware then we do
not want to attempt another login flow.
* Store callback routes in authCallbacks
* fmt
* Push AuthTestConfig into test target
https://github.com/digital-asset/daml/pull/7654#discussion_r506510193
* Unbind oauth2 server after middleware
https://github.com/digital-asset/daml/pull/7654/files#r506513251
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
daml-sdk-head: add optional sha information
This PR add an option, `--sha`, to `daml-sdk-head` so that it produces a
more accurate version number including the current git sha.
The main consequence is that calling `daml-sdk-head --sha` take
significantly longer than calling `daml-sdk-head`, because it needs to
recompile everything that depends on the version number.
> ## Wait, but why??
I started this work in support of an internal project that needed to
test against unreleased, and possibly unmerged, daml versions. However,
after further discussion I believe there is a better option for their
use-case. I've decided to still open this PR because the work was done
and there is no downside to it. It may still be useful if one wanted to
be able to maintain more than one non-released local version of daml.
CHANGELOG_BEGIN
CHANGELOG_END
* set doobie version to 0.9.2 and rerun maven pin
* port extractor and some of JSON API
* repin maven
* use doobie's own builder compatibility where required
* use probably bad derivations to supply Blockers where transactEC was required
- The point of using Blocker instead of ExecutionContext seems to be to
especially emphasize to API users that it isn't appropriate to use an
ExecutionContext with ordinary behavior. That is what we have done, which
should probably change, but just compiling for now.
* fix fragment inspection test for internal restructuring
- This test depends on implementation details of Doobie, so naturally it must be
altered when that runs. Fortunately, it's been made easier by the changes
in this upgrade.
* allow 256 blockers for navigator transaction blocker, like the global EC
* allow as many blockers as the pool size for trigger service
- The transactor shouldn't share ExecutionContext for transactions with the
caller, so we set up a new one based on configured pool size.
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
Removed ad hoc usages of UUID in ledger-api-test-tool for id generation for commands and keys.
This change should facilitate debugging.
CHANGELOG_BEGIN
CHANGELOG_END
There are two reasons for this:
1. 2018 was a long time ago and there seems to be no particular reason
for sticking to that version.
2. We have seen a bunch of issues where we get 404s when fetching msys
packages. The newer version has a larger mirrorlist which might help
with this. We could in principle try to patch the mirrorlist for
the old version but given that upgrading seems like a good idea
anyway, this seems easier.
changelog_begin
changelog_end
* Warn when importing DA.Generics
The module `DA.Generics` is hidden from the docs of `daml-stdlib`
because we consider it experimental. On top of that, the comments at the
top of the module make it clear that this module does _not_ work across
different SDK versions. Nevertheless, let's make it crystal clear that
`DA.Generics` does not work with data-dependencies and will stop you
from upgrading your package.
I've decided against using the a `{-# WARNING ... #-}` annotation
on the `DA.Generics` module declaration since that issues a warning for
every single time an enitity from the module is used rather than once
when importing the module. To me, that's a bit over the top since it is
conceivable that people still want to use the module during some stages
of their development. Another thing we might need to consider soon is a
`-fno-data-dependencies-warnings` flag which suppresses warning related
to data-dependencies. This would be impossible with a
`{-# WARNING ... #-}` annotation.
CHANGELOG_BEGIN
[DAML Stdlib] Issue a warning when importing the `DA.Generics` module,
which does not work with data-dependencies.
CHANGELOG_END
* Fix typo
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
