* Pass `damlc build` logs through daml logger
This gives us some timings which can be useful in some
circumstances (I want timings around data-dependencies next).
I also changed the default log level to INFO. The `damlc build` output
doesn’t get any noisier from that.
Example output:
```
2021-11-01 15:25:17.16 [INFO] [build] []
Compiling foobar to a DAR.
2021-11-01 15:25:17.80 [INFO] [build] []
Created .daml/dist/foobar-0.0.1.dar
```
changelog_begin
changelog_end
* Adjust scenario service log level
changelog_begin
changelog_end
* Adjust tests
changelog_begin
changelog_end
* delete comment
changelog_begin
changelog_end
Let me provide some justification:
1. We used to have a bug around this in the assistant. This got fixed
in https://github.com/digital-asset/daml/pull/7142 together with unit
tests. https://github.com/digital-asset/daml/pull/7150 then added an
integration test. Finally
https://github.com/digital-asset/daml/pull/8032 as part of refactoring
also added a test for absolute paths.
2. As shown in https://github.com/digital-asset/daml/issues/11218
those tests together currently take more than 50s (locally without
load).
3. On the other hand, they don’t really seem to add any meaningful
coverage over the unit tests.
So overall, those tests don’t seem to test anything particularly
useful that isn’t tested elsewhere while they are very slow so
deleting them seems like the best strategy.
changelog_begin
changelog_end
* Factor out npm install step of create-daml-app tests
This PR speeds up the create-daml-app tests by 30s or so by factoring
out the install of the unchanged npm deps into a genrule which we then
feed in again. This is a bit ugly but given how frequently we run
those tests I do think it is worth the uglyness.
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
Speeds them up by more than 2x and reduces flakiness. I also added a higher
timeout to the allocate party call since that seems to still be
sometimes too slow if lots of things run in parallel.
changelog_begin
changelog_end
Add support for `--max-inbound-message-size` flag to the [Ledger Export](https://docs.daml.com/tools/export/index.html) tool.
CHANGELOG_BEGIN
- [Daml Assistant] Add support for `--max-inbound-message-size` flag to the Ledger Export tool.
CHANGELOG_END
This PR accepts the --tls flag but rejects the flags to set certs
because those don’t really make sense here (or are at least not easily
settable).
Unfortunately no great way to test this without a huge amount of test
infrastructure to setup a custom TLS reverse proxy for tests.
If someone has an idea to add a reasonable test, I’m all ears.
changelog_begin
- [Daml Assistant] The `daml ledger` commands now accepts `--tls` in
combination with `--json-api` to access a JSON API behind a TLS
reverse proxy.
changelog_end
CHANGELOG_BEGIN
ledger-api - Command deduplication period can now be specified by setting `deduplication_offset` instead of `deduplication_time` (only valid for v2 WriteService). This change is backwards compatible.
CHANGELOG_END
* Propagate the enriched deduplicationPeriod instead of deduplication duration
* Update the Haskell bindings for the new deduplication period
* Calculate the deduplicateUntil using the new deduplication period for backward compat
* Use consistent naming for deduplication_period
* Cleanup command timeout extraction from deduplication period
* Add the required deduplication_offset to deduplication instead of deduplication_start
* Update haskell bindings to support deduplication_offset
* Add support for deduplication_offset in the ledger-api
* Remove the timestamp-based deduplication from our models to simplify upgrade for users
* Add optional conformance test for offset based deduplication
* Remove buf rule for FIELD_SAME_ONEOF as our change is backwards compatible
* Disable FIELD_SAME_ONEOF buf check for commands file
* Apply suggestions from code review
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Update comment for deduplication period
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* rewrite trigger docs to follow gsg
Per #10419 point 4, I've rewritten the Triggers section to build upon
the Getting Started Guide instead of inventing its own example.
Compared to #10395, this has a lot more explanations as this page must
now serve the dual purpose of being a possible "next step" from the GSG
and being the main reference page for triggers. It's also lost the "next
steps" section, which I think is a bit of a shame, but it doesn't really
make sense here.
There's also no easy way for people not interested in the GSG to follow
along; should we expose the "completed GSG" as a tempate?
CHANGELOG_BEGIN
CHANGELOG_END
* keep copy-trigger as a template
* fix copy-trigger project name
* make up gsg-trigger template
* remove awkward sentence, fix existing typo
* update code to use when{,Some}
* add to
* swap emitCommands and getCommandsInFlight
* typo
* insist on state-correction perspective
* fix copy-trigger tests
* add back copy-trigger to whitelist
* add gsg-trigger to whitelist
* Upgrade Scalatest to v3.2.9.
Because of some coupling we also have to upgrade Scalaz to the latest
v7.2 point release, v7.2.33.
The Scalatest changes are quite involved because the JAR has been broken
up into several smaller JARs. Because Bazel expects us to specify all
dependencies and doesn't allow transitive dependencies to be used
directly, this means that we need to specify the explicit Scalatest
components that we use.
As you can imagine, this results in quite a big set of changes. They
are, however, constrained to dependency management; all the code remains
the same.
CHANGELOG_BEGIN
CHANGELOG_END
* http-json-oracle: Fix a Scalatest dependency.
* ledger-api-client: Fix a Scalatest dependency.
No reason to force the existence of a config file if you can start
without one.
fixes#10516
changelog_begin
- [Navigator] Navigator will now start with an empty config if no
config file exists and it is run outside of a project.
changelog_end
* Add optional submission id to commands.proto
This allows to propagate a submission id. If no id is submitted (the submission id is empty) then we generate a new submission id
CHANGELOG_BEGIN
Add optional submission_id to the commands.proto.
CHANGELOG_END
* Update haskell bindings to include the submission id
* Code review - rename submission id extractor
* Code review - update comment and remove braces from if block
* Fix braces
changelog_begin
changelog_end
- DAML -> daml
- Assistant -> assistant
The capitalization of the latter fixed one occurrence of a capitalized 'A'
which was inconsistent with the vast majority of uncapitalized 'a's.
* Parallelize daml-assistant integration tests.
This mainly involves avoiding changing the working directory or the environment while inside tests.
AFAIK the daml start tests can't be parallelized without starting up a sandbox instance each time, which seems bad. These tests could be separated from the rest...
The magic number "2" was picked via experimentation:
```
1 threads -> 306 s
2 threads -> 199 s
3 threads -> 198 s
4 threads -> 195 s
```
changelog_begin
changelog_end
* buildifier-fix
Each of these installations is about 1gb so it’s not that unlikely
that this plays a significant factor in our out of disk space errors.
Although I think we cleanup partially so I’m not sure how large the
part of the SDK installation is that gets leftover.
changelog_begin
changelog_end
I started looking into failures in the integration tests and this
seems by far the most prevalent one (maybe even the only one but that
needs more investigation). The issue here is that sometimes waiting
40s isn’t enough. On the other hand, sometimes we wait unnecessarily
long which is also bad.
The fix in this PR is to wait until we see the "reload complete"
message which should be exactly the right amount of time to wait.
changelog_begin
changelog_end
Currently we have the following dependency chain:
1. The compiler depends on the scenario service for daml test
2. The scenario service depends on Daml Script
3. Daml Script depends on the Sandbox code only for daml test-script
The last one can easily be split. The scenario service does not care
about this code.
This means that now if we change ledger code, at least not all
compiler tests are going to rerun.
Verified that this actually breaks the dependency fully via
```
bazel query 'somepath(//compiler/damlc/tests:packaging, //ledger/participant-integration-api/...)'
```
changelog_begin
changelog_end
* Block concurrent SDK installations.
Fixes#9612
changelog_begin
- [Daml Assistant] The assistant will now avoid
installing SDK versions concurrently, waiting
for the previous installation to finish before
starting the next installation (if still
necessary). This fixes a bug where the SDK
would become corrupted because two
installations were started concurrently.
changelog_end
* Remove filelock from stack-snapshot.yaml
* Update windows snapshot pin
* Add message if file lock is not immediately available.
* Set supported jdbc driver names at compile time
This is mainly to unblock the work on Oracle support in the Ledger API
but I think it’s a sensible thing in general. For the Ledger API,
moving the dependency to the top-level is apparently rather
tricky. Because the SDK bundles everything into a single megajar,
Sandbox depending on the oracle library does also result in the JSON
API and the trigger service will also have the oracle library in scope
and will support Oracle in CE which they should not.
This PR simply hardcodes the list of supported drivers to address
that. Not pretty but does the job.
changelog_begin
changelog_end
* format
* Address review comments
changelog_begin
changelog_end
* Don't duplicate export script option parsing
Forward all flags to the wrapped executable instead of handling them in
daml-helper with optparse-applicative.
changelog_begin
changelog_end
* Fix hlint warning
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* rename "dump" to "export"
* Add Daml export command to assistant
* Make daml export script a subcommand of daml export
* daml-assistant IT for daml export script
* Expose export as daml ledger command (hidden for now)
Add Haskell side parser for ledger export flags
changelog_begin
* [Daml export] New feature: Use ``daml ledger export script`` to
generate a Daml script that will reconstruct a given ledger state.
This is an early access feature.
changelog_end
* Update integration test
* Remove daml export - it's daml ledger export now
* integration-test set ledger port
The integration tests don't configure the port in the project config,
but on the command-line, and they don't use the default value.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* daml build: add a --access-token-file for remote dependencies
This adds a `ledger.access-token-file` field in the `daml.yaml` project
file and a `--access-token-file` flag to `daml build` to authorize
querying/fetching of remote dependencies.
CHANGELOG_BEGIN
[daml build] A new flag `--access-token-file` is added for the `daml
build` command. It allows the specify the path to an access token to
authenticate against the ledger API. This is needed if the project
depends on a remote Daml package hosted on such a ledger. Alternatively,
the path to the token can also be specified in the `daml.yaml` project
file under the `ledger.access-token-file` field.
CHANGELOG_END
* Expose tho Oracle support in the EE trigger service
This PR builds on the previous PR that did all the actual work on
Oracle support and exposes it in the enterprise edition. This PR only
releases the enterprise edition via the SDK tarball. I’ll add
artifactory publishing separately.
changelog_begin
changelog_end
* Update daml-assistant/daml-sdk/validate.sh
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Fixes#9196.
This adds a new command `daml package list` which fetches all
available packages from the ledger and prints package name/versions to
stdout.
CHANGELOG_BEGIN
[daml packages] A new `daml packages list` command has been added to
list packages deployed on a remote Daml ledger.
CHANGELOG_END
Fixes#9187.
Instead of `daml: Must be called from within a project` you will get now
`daml: This command needs to be either run from within a project or the argument --host needs to be specified.`
CHANGELOG_BEGIN
CHANGELOG_END
* remote pkgs: resolve package names in data dependencies via ledger
This implements part 3) of #8976.
This adds the ability to specify package names/versions in daml.yaml in
the data-dependencies stanza. They are being resolved via the project
ledger and a daml.lock lock file.
CHANGELOG_BEGIN
[remote dependencies] Package names and versions, as well as package
ID's are allowed in the `data-dependencies` list of `daml.yaml`. These
packages are fetched from the project ledger. The auto-generated
`daml.lock` file keeps track of the package name/version to package
ID's resolution and should be checked in to version control of the
project.
CHANGELOG_END
* added docs
* Update compiler/damlc/lib/DA/Cli/Damlc/DependencyDb.hs
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Update docs/source/daml/reference/packages.rst
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* suggestions
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
--sandbox-port has caused confusion several times and cannot be
specified in daml.yaml which is a bit annoying. There isn’t really any
good reason not to support --sandbox-option --port since we can pick
up the port via the port file either way.
This change is fully backwards compatible.
changelog_begin
- [Daml Assistant] The sandbox port can now also be configured via
`--sandbox-option=--port=12345` instead of `--sandbox-port`. Other
tools like Navigator, the JSON API and Daml Script will pick up the
modified port automatically.
changelog_end
* damlc pkg: use cache for already present packages
This only downloads packages from a remote ledger, if the package is not
already present in the dependency db as a normal dependency.
CHANGELOG_BEGIN
CHANGELOG_END
* damlc: Allow package IDs in data-dependencies.
This is the next step outlined in issue #8976. If package id's are
present in the `data-dependency` section of the daml.yaml file, we try
to fetch them (and their transitive dependencies) from the default
ledger of the project.
CHANGELOG_BEGIN
CHANGELOG_END
* Support fetching SDK EE tarball in the assistant
This PR adds an `artifactory-api-key` field to `daml-config.yaml`. If
this is set, we will download the SDK EE tarball if it exists, falling
back to GH releases on 404s (for old releases).
changelog_begin
- [Daml Assistant] The assistant now supports an `artifactory-api-key`
field in `daml-config.yaml`. If you have a license for Daml Connect
EE you can specify this and the assistant will automatically fetch
the EE edition which provides additional functionality.
changelog_end
* Hardcode first EE snapshot
changelog_begin
changelog_end
* Release EE SDK tarballs and installer
As before, no way of testing this. I’ll do a snapshot afterwards.
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* Rename EE artifacts
changelog_begin
changelog_end
* Move Daml Profiler to EE version of sandbox/sandbox-classic
This splits Sandbox targets into EE/CE targets and exposes the option
in the EE version. The option still exists in the CE option for now
until we have released EE artifacts to not break users that might know
about it without an alternative.
There is also a small test that makes sure that this actually works
since classpaths are dumb and it didn’t work at first.
changelog_begin
changelog_end
* Fix publish target
changelog_begin
changelog_end
* Publish transitive dep
changelog_begin
changelog_end
* I hate bash
changelog_begin
changelog_end
* Build SDK EE tarball
This sets up the infrastructure to build an SDK EE tarball and allows
for swapping out all files included in the tarball depending on the
edition. As an example, this includes the JSON API with (partial)
Oracle support in the EE tarball.
This PR does not yet address publishing this artifact to Artifactory.
I’ll tackle that in a separate PR.
changelog_begin
changelog_end
* Build in temp dir because Windows is stupid
changelog_begin
changelog_end
* directories are bad
changelog_begin
changelog_end
* Navigator resources are actually needed
changelog_begin
changelog_end
We change the color from red to yellow to inform about the possibility
to rebuild and also output the success message in green.
CHANGELOG_BEGIN
CHANGELOG_END
* Draw the rest of the Scala 2.13 owl
Not quite but pretty close and this switches us over from inclusions
to exclusions which makes it much easier to track.
Ledger API test tool should be fixed by #8821. Non-repudiation needs a
tiny bit of work since unwrapArray doesn’t work the same on 2.13 but
shouldn’t be hard to fix.
changelog_begin
changelog_end
* Fix ScriptService tests
Those tests were all dumb. They asserted on a fixed order while the
function to sort the things was broken so we ended up with the random
Map order which is unsurprisingly not the same.
This is easily fixed by fixing the sort function.
There is also a second issue with query not sorting.
changelog_begin
changelog_end
* Turns out if you fix one test the next one breaks
And clearly nobody ever tested this or give this a second thought.
changelog_begin
changelog_end
* Use cliopts.Http for OAuth 2.0 middleware
changelog_begin
- [OAuth 2.0 Middleware] You can now configure the address that the
middleware listens to using the ``--address`` flag.
The port that the middleware listens to is now configured using the
``--http-port`` flag, use 0 to dynamically choose a free port.
You can now configure a port file where the chosen port will be
written to using the ``--port-file`` flag.
changelog_end
* Add test-case for OAuth 2.0 middleware port file
* Don't forget to close source
* Fix integration test
* Update triggers/service/auth/src/main/scala/com/daml/auth/middleware/oauth2/Config.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* add livez endpoint to auth middleware
* Add OAuth 2.0 middleware to Daml SDK
* unhide trigger service auth flags
changelog_begin
- [Triggers] The trigger service now supports authorization through an
auth middleware. The feature is enabled using the `--auth` and
`--auth-callback` command-line flags. Please refer to the
Authorization chapter of the trigger service documentation for further
instructions.
- [OAuth 2.0 middleware] Daml Connect now includes an implementation of
the auth middleware API that supports OAuth 2.0 Authorization Code
Grant. Please refer to the Auth Middleware and OAuth 2.0 Auth
Middleware chapters of the documentation.
changelog_end
* drop early access flag on triggers
Daml triggers, the trigger service, and the auth middleware are no
longer marked as early access features.
changelog_begin
- [Triggers] Daml Triggers and the Trigger Service are no longer in
early access status.
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
To avoid prompting the user to set the PATH variable on CI, we check the
`CI` environment variable and do nothing if set.
CHANGELOG_BEGIN
CHANGELOG_END
* fix: project parties are not required
This fixes a bug in daml assistant. `daml deploy` would fail if there
were no project parties listed in the daml.yaml project config file. Now
project parties are not required anymore.
Fixes#8487.
CHANGELOG_BEGIN
CHANGELOG_END
* added an integration test
We use the new cache folder specified with `DAML_CACHE` defaulting to
`.cache/daml` as a new cache folder for the assistant to store available
versions etc.
This does not migrate an existing cache folder and just starts a new
one.
CHANGELOG_BEGIN
CHANGELOG_END
* telemetry: use an extra cache directory for telemetry
If the sdk's `.daml` directory is not writable, telemetry fails and
kills the IDE. We add a new assistant environment variable "DAML_CACHE".
If set, this directory is used for telemetry cache data.
Fixes#8396.
CHANGELOG_BEGIN
CHANGELOG_END
* try creating directory
* check for already existing machine id file
* set DAML_CACHE to read-only in assistant integration tests
This PR updates scalafmt and enables trailingCommas =
multiple. Unfortunately, scalafmt broke the version field which means
we cannot fully preserve the rest of the config. I’ve made some
attempts to stay reasonably close to the original config but couldn’t
find an exact equivalent in a lot of cases. I don’t feel strongly
about any of the settings so happy to change them to something else.
As announced, this will be merged on Saturday to avoid too many conflicts.
changelog_begin
changelog_end
The one thing that is still missing is making the generated Scala code
from the codegen compatible with Scala 2.13 so the examples are
excluded for now.
changelog_begin
changelog_end
