This is another refactoring PR for createProjectPackageDb. There are
no actual changes but the `DependencyInfo` now includes the list of
main unit ids. This is not used in this PR but the plan is to start
emitting a file here containing those and then we only have to parse
that file instead of reading all DARs again just to figure out the
--package flags. (The reason why it needs to be written to a file
instead of simply returned is that this needs to work even with
--init-package-db=no which is going to become more important for
incremental package db initialization).
changelog_begin
changelog_end
* Upgrade rules_nodejs to version 1.6.0
closes#5367
This includes the fixes for the issues in jest that we’ve been seeing.
changelog_begin
changelog_end
* Fix eslint rules
* A bit of progress
* Try to add LinkablePackageInfo (doesn’t seem to work yet)
* Add rootDirs
* revert da_ts_library
* da_ts_library: add LinkablePackageInfo info
* Remove react hook workaround
Since rules_nodejs 1.6.0 this fails with the following error:
```
● Test suite failed to run
Configuration error:
Could not locate module react mapped as:
/.../execroot/com_github_digital_asset_daml/bazel-out/k8-opt/bin/language-support/ts/daml-react/test.sh.runfiles/com_github_digital_asset_daml/node_modules/react/umd/react.development.js.
Please check your configuration for these entries:
{
"moduleNameMapper": {
"/^react$/": "/.../execroot/com_github_digital_asset_daml/bazel-out/k8-opt/bin/language-support/ts/daml-react/test.sh.runfiles/com_github_digital_asset_daml/node_modules/react/umd/react.development.js"
},
"resolver": null
}
49 | // like a promis without being one.
50 | /* eslint-disable @typescript-eslint/no-floating-promises */
> 51 | var react_1 = __importStar(require("react"));
| ^
52 | var react_hooks_1 = require("@testing-library/react-hooks");
53 | var index_1 = __importStar(require("./index"));
54 | var events_1 = require("events");
at createNoMappedModuleFoundError (../../../../../../../../../../../node_modules/jest-resolve/build/index.js:501:17)
at Object.<anonymous> (index.test.js:51:28)
Test Suites: 1 failed, 1 total
Tests: 0 total
Snapshots: 0 total
Time: 1.88s
Ran all test suites within paths "language-support/ts/daml-react/DamlLedger.d.ts", "language-support/ts/daml-react/DamlLedger.js", "language-support/ts/daml-react/context.d.ts", "language-support/ts/daml-react/context.js", "language-support/ts/daml-react/hooks.d.ts", "language-support/ts/daml-react/hooks.js", "language-support/ts/daml-react/index.d.ts", "language-support/ts/daml-react/index.js", "language-support/ts/daml-react/index.test.d.ts", "language-support/ts/daml-react/index.test.js".
=
```
* rootDirs is not needed for tsc
This is only required for ts_project
* Update yarn Bazel packages
* docs/theme add missing dependencies
* Remove unused attribute module_root
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Properly mangle/unmangle names in the scenario service
Previously, we just passed in whatever we had in DAML land which broke
the scenario service as soon as a name needed mangling. This PR fixes
mangling when passing identifiers to the scenario service and
unmangles them again when pretty printing.
fixes#5593
changelog_begin
- [DAML Studio] Fix a bug where scenarios with names that are mangled
in DAML-LF resulted in a crash in the scenario service.
changelog_end
* Mangle module names in context updates
This is a first step in probably a series of PRs to cleanup the
packaging logic.
There is no change in functionality (or at least none that is
intended). The changes boil down to two things:
1. Add a `DependencyInfo` type that stores the various maps and has
detailed comments what exactly is in each map and hopefully somewhat
descriptive field names.
2. Move a lot of the logic in `createProjectPackageDb` into various
helper functions.
There is definitely a lot more cleanup and refactoring that we can do
here but I’ll try to keep it in manageable chunks.
changelog_begin
changelog_end
This has come up today where someone didn’t realize that removing the
field is sufficient to fix the warning.
I’ve also taken this as an opportunity to move the code out to a
helper function.
changelog_begin
changelog_end
* Fix conversion from contract to node ids in the scenario service
Previously we only included the mapping from the ledger. However, this
does not include any contracts which might have been created in a
partial transaction that failed. This caused a
`NoSuchElementException` when converting the error which is obviously
not what we want.
This PR extends the mapping to include any contracts created in the
current partial transaction.
changelog_begin
- [DAML Studio] Fix a bug where a failed transaction that included
references to a transient contract returned a gRPC error instead of
the expected error message.
changelog_end
* Update compiler/scenario-service/server/src/main/scala/com/digitalasset/daml/lf/scenario/Conversions.scala
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Update compiler/scenario-service/server/src/main/scala/com/digitalasset/daml/lf/scenario/Conversions.scala
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Update compiler/scenario-service/server/src/main/scala/com/digitalasset/daml/lf/scenario/Conversions.scala
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Update compiler/scenario-service/server/src/main/scala/com/digitalasset/daml/lf/scenario/Conversions.scala
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Update compiler/scenario-service/server/src/main/scala/com/digitalasset/daml/lf/scenario/Conversions.scala
Co-Authored-By: Remy <remy.haemmerle@daml.com>
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Improve error messages in daml repl on calls to `error`
There were two issues with calls to `error`:
1. This one is harmless but somewhat annoying: When calling `error` we
run into the log statement in `stepToValue` which prints out the
error message in a fairly reasonable form (you can argue whether
Error: User abort: is a super useful prefix but that’s a relatively
minor issue). Afterwards we then call `println` on the failed
future. However, that will just print the type of the exception
which isn’t something we want to show to users. I’ve just disabled
the println statement if the exception is `SError`.
2. This one is a bigger issue: `throw x` is not the same as
`Future.failed(x)`. I only fully realized the difference fairly
recently. The former fails before it produces a future. So `(throw
x).onComplete(…)` will never execute the callback. The latter is
just a failed future. It is rather confusing to have a function
that returns a future but then throws an exception instead of a
future and it confuses the grpc library which prints out a horrible
exception. I’ve changed all calls to `throw` in `runWithClients` to
instead use `Future.failed` and `flatMap` (in the form of
for-comprehensions).
There are still a few calls in `run` left which I’ll leave for a
separate PR.
I think we need to factor out some helper functions here to make this
a bit more manageable (e.g. for the Converter.toFuture) stuff but I’ll
leave that for a separate PR. You probably want to view this with
whitespace diffs disabled.
changelog_begin
- [DAML Repl] DAML Repl now produces better error messages on calls to
`error` and `abort`.
changelog_end
* Switch stepToValue to return Either
mkConfFile was a bit of a mess before. Half of the arguments we passed
in via `PackageConfigFields` were unused as shown by the fact that we
set them to `error …` in various places where we did not have that
information. More importantly rather than passing in the unit ids
which need to end up in the GHC package config, we passed in file
names which required us to parse the DAR again and have a bunch of
hacks where file names with .dalf endings were not translated.
This PR changes this to only pass in the fields we need and pass in
dependencies directly as unit ids.
changelog_begin
changelog_end
* Fix unstableSplitOn
unstableSplitOn is supposed to behave like splitOn but fast™ but it
didn’t live up to that promise for a few reasons:
1. Java’s split method by default drops empty strings at the
end. Passing a second negative parameter avoids that.
2. Java’s split method does regex splitting so we need to quote the
string.
3. Java’s split method splits on an empty pattern whereas we want to
simply preserve the input.
I’ve added regression tests for this and for unstableDrop and a few
other functions while I was at it.
changelog_begin
changelog_end
* Update compiler/damlc/tests/daml-test-files/UnstableText.daml
Co-Authored-By: Martin Huschenbett <martin.huschenbett@posteo.me>
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
* Haskell: Add assertFileExists to DA.Test.Util
This PR adds a helper function `assertFileExists` that captures the
`doesFileExist ... >>= assertBool ...` pattern that is very common in
our Haskell test suites. It also adds the inverse
`assertFileDoesNotExist` function. Both functions are now used where
appropriate.
CHANGELOG_BEGIN
CHANGELOG_END
* Add directory dependency to test-util lib
CHANGELOG_BEGIN
CHANGELOG_END
This is a spin off from the work for making module prefixing
work. There I wanted to be able to use the `GeneratePackageMap` rule
to get access to the modules in an LF package which requires to be in
an Action rather than in IO.
This resulted in a much larger refactoring than I expected, so here is
a list of changes to ease review:
1. I’ve split off Development/IDE/Core/RuleTypes/Daml.hs into its own
Bazel targets to avoid cycles. This allows us to use a rule without
depending on the implementation of that rule.
2. Development/IDE/Core/IdeState/Daml.hs has moved into its own Bazel
target. There is a lot of stuff that needs `withDamlIdeState` now
since the `ghcide` `initialise` function does not add our Shake rule
for getting the GHC session and making all of that depend on
damlc:lib creates a mess.
3. The actual logic of the initialization has stayed the same but
moved into a rule in Development/IDE/Core/Rules/Daml.hs.
4. As mentioned above, a few things that could previously get away
with only the rules provided by ghcide now need our own rules so
they switched to `withDamlIdeState`. This mainly affects `damlc
docs`.
5. Making daml-docs work with the ofInterestRule is a bit tricky.I did
find the issue and included the fix but also disabled the
ofInterestRule since it really does not make sense to generate Core
in the background.
changelog_begin
changelog_end
* Set the `Bearer ` prefix in bindings.
* Make the `Bearer ` prefix in the authorization header mandatory.
* Bearer prefix can be removed from the token file.
CHANGELOG_BEGIN
[Extractor]: The ``Bearer `` prefix can be removed from the token file.
It is added automatically.
[Navigator]: The ``Bearer `` prefix can be removed from the token file.
It is added automatically.
[DAML Script] The ``Bearer `` prefix can be removed from the token file. It
is added automatically.
[DAML Repl] The ``Bearer `` prefix can be removed from the token file. It is
added automatically.
[Scala Bindings] The ``Bearer `` prefix can be removed from the token. It is
added automatically.
[Java Bindings] The ``Bearer `` prefix can be removed from the token. It is
added automatically.
[DAML Integration Kit] ``AuthService`` implementations MUST read the
``Authorization`` header and the value of the header MUST start with
``Bearer ``.
CHANGELOG_END
* Split up repl tests to make them faster
This PR splits up the tests into the tests for TLS and Auth and the
tests for the actual functionality.
The func tests use the repl as a library which allows them to be
significantly faster:
1. We only need to start the service process once.
2. We only need to initialize the package db once.
There is at least one other point that I did not address for now and
that is only loading the packages into the repl service once. While
loading them multiple times is a noop, it still has a performance
implication.
Sadly, this has turned out much more messy than I thought it would be
due to various issues with haskeline/repline/tasty/computers. The
details are in a comment in DA.Test.Repl.FuncTests.
changelog_begin
changelog_end
* Try to nuke cache on window selectively
* Enable cache again
Packages com.digitalasset.daml and com.daml have been unified under com.daml
Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.
CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
I've completely removed the possibility to call `daml codegen ts`. I'm
happy to add in back in a followup PR once I've seen that all our tests
pass without it existing.
CHANGELOG_BEGIN
CHANGELOG_END
CHANGELOG_BEGIN
- Move daml2ts, bindings-ts and JSON API out of experimental section in docs
- Rename Experimental to Early Access in docs and assistant
- Reorganise the docs a little bit to de-emphasise the Ledger API
CHANGELOG_END
* REPL parse import statements
* Factor parsing out of handleLine
* Pull binding and error handling into handleStmt
This is specific to the case of handling a statemt and does not overlap
with handling imports.
* Track module imports as ImportDecl
* Add new module imports
* Validate module imports
* Add REPL interaction test for import
* REPL document import declarations
CHANGELOG_BEGIN
- [DAML REPL - Experimental] You can now use import declarations at the
REPL prompt to bring additional modules into scope.
CHANGELOG_END
* ReplState strict fields
* Use semigroupoids Alt to simplify parseReplInput
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Regression test for silent REPL server errors
CHANGELOG_BEGIN
CHANGELOG_END
* Fix script error test case
We get an error print out by the REPL server as well
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Limit memory usage of scenario service in tests
Previously the scenario service used over 2GB of memory during
//compiler/damlc/tests:integration-dev for absolutely no reason.
This PR adds an option to pass JVM options when starting the scenario
service and sets them to 200MB for the Shake tests and the damlc
integration tests which seems to be more than sufficient.
These option can also be set in `daml.yaml`. I did not change the
defaults since this is shipped to users where I’d rather have high
memory usage than a crashing scenario service and on certain large
DAML codebases we might actually use a fair bit.
changelog_begin
changelog_end
* Update docs/source/tools/assistant.rst
Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Replace unstable-types tests by a Haskell test suite
The JQ based test suite is absurdly slow, it takes around 60s to run
on my machine. The Haskell test suite on the other hand takes less
than 2s on my machine so this is a speedup of over 30x and I have to
read less bash.
changelog_begin
changelog_end
* Address review comments
- Move deployment tests (deployTest, fetchTest) out of integration-tests.
- Use DA.Test.Sandbox where appropriate.
- Split out code for useful test patterns: i.e. calling commands quietly, getFreePort.
changelog_begin
changelog_end
The details are in a comment since I want to preserve them. Please
review carefully, I don’t trust my async exception foo.
This is only an issue in the integration tests I believe since we
change files of interest all the time. As long as you stay stable at
some point for some amount of time, we will GC properly either way.
In my tests, this reduces memory usage of the scenario service in the
integration tests from 3 to 2 GiB. There is sadly
anther leak somewhere (contexts are properly GCd now, I verified that
we never have more than 3 contexts in the server). I’ll do some more
investigation but memory grows linearly to the 2GiB while it should
stay pretty much constant.
changelog_begin
changelog_end
* Use repline for DAML REPL
CHANGELOG_BEGIN
CHANGELOG_END
* REPL server print error messages
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Improve handling of exposed-modules with data-dependencies
Previously, we tried to rename all modules of a dependency via
--package. This fails if some of those modules are not exported. This
was trivial to hit as a user since the ``daml-trigger`` library made
use of this.
This PR adds a few things to improve the situation:
1. We only rename modules that are exposed. This fixes the issue if
you don’t actually reference a non-exposed module from your
data-dependency.
2. I’ve removed the exposed-modules from daml-trigger. I don’t think
they are essential here given that the module name has `Internal`
in the name and it’s too easy to have something that actually
references the non-exposed module since the types are reexported.
3. I’ve added documentation that mentions this issue.
4. I’ve added a warning if your exposed-modules are excluding some
modules. Maybe worth turning this into an error in the future.
changelog_begin
changelog_end
* Update compiler/damlc/lib/DA/Cli/Damlc/Packaging.hs
Co-Authored-By: associahedron <231829+associahedron@users.noreply.github.com>
Co-authored-by: associahedron <231829+associahedron@users.noreply.github.com>
Until #5294, we had some test cases for `damlc` flying around there
were never run because they had an upper bound on an unsupported
DAML-LF version. This PR makes sure we'll clean up such test cases in
the future.
CHANGELOG_BEGIN
CHANGELOG_END
* Extract package config type to a separate package.
changelog_begin
changelog_end
* update copyright headers
* run buildifier-fix
* fix src_strip_prefix
* Heres the file I accidentally deleted
* buildifier-fix again
DAML-LF 1.6 is the baseline vesion we want to support with the next
SDK release.
CHANGELOG_BEGIN
- [DAML Compiler] Drop support for reading DARs using DAML-LF 1.5.
Please recompile your DAML to a newer DAML-LF version in case you
have such DARs.
CHANGELOG_END
* Turn warnings for module name/record name mismatches into errors
The module name warning existed for ages. We started warning about the
record name mismatch in 0.13.55 so I’d like to turn it into an error
before 1.0
changelog_begin
- [DAML Compiler] File names must now match up with module names. This
already produced a warning in previous releases.
- [DAML Compiler] It is now an error to define a record with a single
constructor where the constructor does not match the type
name. To fix the error you need to change ```data X = Y { … }``` to
```data X = X { … }```. This restriction only applies to
single-constructor records. Variants and enums are not affected.
This already produced a warning in 0.13.55.
changelog_end
* Fix integration tests
* Fix docs
* Fix lsp tests
* New comamnd: daml ledger fetch-dar
`daml ledger fetch-dar [PID] [PATH]`
Download a `Package` and it's dependencies from a ledger, given a root `packageId`, and re-construct a valid `.dar` file. Addresses issue #5037.
The original package names are not reconstructed.
CHANGELOG_BEGIN
CHANGELOG_END
* address some comments
* fix spello
* attempt: recoverPackageName
* recover main package name & version
* Try to fix integration tests on Windows
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Adding more syntax is just confusing for beginners so these are not
particularly helpful and I would argue that they aren’t that useful in
general even once you move past the beginner stage.
changelog_begin
changelog_end
* Make name collision check more strict
This PR extends the name collision check to catch collisions between
A:B (type B in module A) and module A.B.C. For now this is just a
warning and not an error. Once we turn it into an error, we also need
to add this to the Scala collision checker.
There is a fair bit of plumbing required to make warnings work but on
the plus side we get multiple errors at once now instead of erroring
out on the first one.
changelog_begin
- [DAML Compiler] The name collision check has been extended to also
count the case as a collision where you have a type B in module A and a module
A.B.C (but no module A.B). This is a warning in this SDK release but
will become an error in a future release. The typescript codegen is
not usable on packages that don’t uphold this restriction.
changelog_end
* Address review comments
* Remove unused da_doc_package
The only use-site was `//compiler/daml-licenses:daml-licenses`, which
itself was unused.
* Remove unused notices-gen
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Add keys with maintainers to Fetch nodes
The new field is populated by the interpreter whenever the fetched
contract has a key. Used for contract key reinterpretation in Canton.
CHANGELOG_BEGIN
- [DAML-LF] Add keys with maintainers to Fetch nodes in transactions.
CHANGELOG_END
* Depend on LF version specific daml-libs
* daml-script.dar build multiple LF versions
CHANGELOG_BEGIN
[DAML Script] The `daml-script` library is now available in multiple LF
versions, namely 1.7, 1.8, and 1.dev.
CHANGELOG_END
* daml-trigger.dar build multiple LF versions
[DAML Triggers] The `daml-trigger` library is now available in multiple
LF versions, namely 1.7, 1.8, and 1.dev.
* Keep daml-script.dar available for tests
* Keep daml-trigger.dar available for tests
* daml-libs LF versions integration test
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This PR adds as `ScriptLedgerClient` trait (happy to change the name
if anyone has a better proposal) that abstracts over the interaction
with the ledger. This will allow us to plug in a different
implementation for interacting with the JSON API so we can run DAML
scripts against DABL or other environments where gRPC is not a
workable option. Note that this PR does not yet add the implementation
for interacting with the JSON API. I’ll leave that for a separate PR.
changelog_begin
changelog_end
* Tighten result type
Command execution can't result in a sequencer error
* New helper method for extracting used contracts
* New error clause
* Add a DAO query for the maximum time of contracts
* Implement algorithm for finding ledger time
CHANGELOG_BEGIN
CHANGELOG_END
* fixup ledgerTimeHelper
* Use new ledger time algorithm
* Mark LET/MRT as deprecated
CHANGELOG_BEGIN
- [Ledger API] DAML ledgers have switched to a new ledger time model.
The ledger_effective_time and maximum_record_time fields of command submission are deprecated,
the ledger time of transactions is instead set automatically by the ledger API server.
Ledger time is no longer strictly monotonically increasing, but only follows causal monotonicity:
ledger time of transactions is greater than or equal to the ledger time of any used contract.
See `#4345 <https://github.com/digital-asset/daml/issues/4345>`__.
CHANGELOG_END
* Add ledger time skew check
* Remove command updater
LET/MRT are now deprecated, this class is now useless
* Remove old time model validator
* Switch to new time model check: kvutils
* Switch to new time model check: in-memory ledger
* Switch to new time model check: SqlLedger
* Use initial ledger config
* Ignore user provided LET
* Use TimeProvider in submission services
* Use deduplication_time in daml-script runner
- Also remove unnecessary command completion output of CommandTracker.
- Remove usage of maximum record time in CommandTracker.
* Use arbitrary default value for deduplication time
* Use built-in Instant ordering
* Remove obsolete test
* Remove obsolete test: CommandStaticTimeIT
* Refactor test: TransactionMRTCompliance
* Disable test: CommandTrackerFlow timeout
* thread maxDeduplicationTime through to CommandTracker
* Improve test
* Refactor command client configuration
* Deduplication time should always use UTC
* Add missing method in TimedIndexService after rebase
* Put more details into the deduplication error response.
* Use system time for command dedup submittedAt.
* Use explicit UTC time source in command validator
* Revert CommandTracker[Flow] to previous completion-recovering-behavior
* Adapt scala client command config to new config params
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Integrate create-daml-app into the assistant
fixes#4868
changelog_begin
- [DAML Assistant] Add a new ``daml create-daml-app`` command for creating a project based on
`create-daml-app <https://github.com/digital-asset/create-daml-app>`_.
changelog_end
* Try random things hoping to fix windows
* Try random things hoping to fix things on macos
* Use MutableCompiledPackages in REPL
This avoids unnecessary recompilation of loaded packages.
CHANGELOG_BEGIN
CHANGELOG_END
* Track compiled definitions explicitly
Compile packages on load directly.
Use `PureCompiledPackages` extended with package for current line in
REPL service `runScript`.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
changelog_begin
- [DAML Compiler] The default output DAML-LF target version is now
1.8. You can target 1.7 by specifying ``--target=1.7`` in the
``build-options`` field in your ``daml.yaml``.
changelog_end
Fix daml2ts tests
