* Implement a simple profiler for DAML scenarios
The profiler runs a single scenario and records timing information when
each function (and some other closures) are entered and left. The
resulting information can be visualized as a flamegraph using
[speedscope](https://www.speedscope.app/).
The profiler works by instrumenting the CEK machine at the heart of
DAML Engine. Unfortunetaly, this causes a very small overhead on
non-profiling runs too. However, in my benchmarks I could not measure
any significant impact on the overall runtime at all. More precisely,
the overhead is as follows:
Every closure now has an additional field called `label`. In
non-profiling runs this field is always set to `null`. This field needs
to be allocated, copied whenever we copy a closure and scanned during
garbage collection. Additionally, whenever we enter a closure, we check
this field and whenever it is _not_ `null`, i.e. never during
non-profiling runs, we record an "open event" and set up a hook for the
corresponding "close event". Thus, the additional cost during
non-profiling runs are a single pointer comparison and a jump beyond
the "then branch".
Since this is still very much in active development, there are no
documentation, other than an entry in a README, and no tests yet. They
will come before we promote this. However, the UX will look very
different then since we already have plans to significantly change it.
CHANGELOG_BEGIN
CHANGELOG_END
* Run scalafmt
* Make profiling argument to PureCompiledPackges optional
* Fix a bunch of tests
CHANGELOG_BEGIN
CHANGELOG_END
* scalafmt is so annoying
CHANGELOG_BEGIN
CHANGELOG_END
* Apply simple suggestions
CHANGELOG_BEGIN
CHANGELOG_END
fixes#6005
Take a look at #6005 for details. I verified manually that things
still work as expected. It does require bumping the vscode version to
1.38 which was released almost a year ago so hopefully that should be
safe.
changelog_begin
changelog_end
* update rules_nixpkgs
* Use hermetic nixpkgs cc toolchain
CHANGELOG_BEGIN
CHANGELOG_END
* Work around Bazel's cc toolchain autodetection
* Use --crosstool_top for hermetic cc toolchain
When using --incompatible_enable_cc_toolchain_resolution instead
cc actions still depend on
`external/local_config_cc/builtin_include_directory_paths`
as well as
`external/nixpkgs_cc_toolchain_config/builtin_include_directory_paths`.
* override local_config_cc
* remove unused attribute
* Fix posix toolchain on Windows
* nixpkgs cc toolchain not on Windows
* Fix nixpkgs cc toolchain on MacOS
* nixpkgs cc toolchain uses bin/cc
* Use darwin.binutils on MacOS
* Remove clang(++) and gcc (g++) symlinks
The toolchain only considers `bin/cc` and having the other symlinks
around could lead to confusion
* Use hermetic toolchain in compatibility workspace
* Avoid empty linker flags
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Include Bazel patch to cache exclusive tests on Windows
This includes the patch that we already use on Linux and MacOS to fix
caching of things marked exclusive. I’ve kept in the debugging output
that I added in the last patch for now. While our workaround seems to
be working, I’d like to wait a bit longer in case the issue reappears.
changelog_begin
changelog_end
* Actually bump manifest
changelog_begin
changelog_end
* Fix alunchd killing VMWare process at end of script execution
* Fix alunchd killing VMWare process at end of script execution
CHANGELOG_BEGIN
Fix issue with MacOS Catalina Launchd killing VMWare instance on rebuild (AbandonProcessGrop)
CHANGELOG_END
Bash is not easy to add to dev-env because dev-env depends on Bash. This
has not been an issue so far because Bash behaves in very sensible ways
overall and is mostly backwards compatible.
A recent change to `daml-sdk-head` is, however, using some feature of
Bash 4 that is not evailable in Bash 3. Bash 3 is ancient so in an ideal
world that would not be an issue, but macOS still ships with Bash 3 for
some obscure (licensing) reason.
It turns our that the line
```bash
arr=()
```
in Bash 4 sets the variable `arr` to an empty array, whereas it leaves it
unset in Bash 3. This means that the later use of `arr`, in the case
where no further element has been added to the array, will yield an
error in combination with the `set -u` option we are using.
This PR changes the usage pattern from
```bash
"${arr[@]}"
```
which fails on Bash 3 to
```bash
${arr[@]:-}
```
which works as expected. Note that the quotes have been removed: the
quotes in this case are not useful assuming that the flags themselves
are never multiword. Without the quotes, this evaluates to `""` under
Bash 3 (as the variable is not set), which confuses Bazel because now it
thinks it's asked to build the `""` target, which it has no rule for.
Ignoring the quoting issue, the actual fix is to include `:-` inside the
`{}`, which instructs Bash to replace the variable with what follows the
`-` in case the variable is not set (in this case, an empty string), and
therefore not crash on this specific variable not being set despite the
`set -u` option.
CHANGELOG_BEGIN
CHANGELOG_END
* Include the first feature in compat tests for create-daml-app
This adds the step from the create-daml-app tests that applies the
patch for the messaging feature and tests that things will build. This
is the last step before we can actually run the puppeteer tests which
will turn this into an actual compatibility test.
changelog_begin
changelog_end
* temporarily run all tests
changelog_begin
changelog_end
* Fix exclusions
changelog_begin
changelog_end
* Revert "temporarily run all tests"
This reverts commit 7425dd09cf48f2bfd6064b55d857c76d51afc821.
* Remove accidentally committed messaging.patch
changelog_begin
changelog_end
* Include sources directory in the Bazel cache key
This should hopefully fix the “undeclared inclusion” errors we have
been getting daily on CI
The details are in a comment but the short summary is that
the daily cron job is running in D:\a\1 whereas jobs on the same
machine afterwards run in D:\a\2. Because absolute paths leak in some
places, this fucks things up.
changelog_begin
changelog_end
* Add debugging output to the cache
Previously, we used the stack to recurse when filling in the command
results which obviously breaks once you have large multi-command
submissions.
You probably want to view the diff with whitespace disabled.
changelog_begin
- [DAML Script] Fix a bug where large multi-command transactions
produced a stack overflow.
changelog_end
by adding synchronize call between an alpha-participant-create and the
corresponding beta-participant-exercise.
changelog_begin
changelog_end
We occasionally run into this in Canton
Fixes#5592
The CLI syntax and the defaults follow the JSON API here.
changelog_begin
- [DAML Script] The maximum inbound message size can now be configured
using `--max-inbound-message-size``. This matches the flag in the JSON
API.
- [DAML REPL] The maximum inbound message size can now be configured
using `--max-inbound-message-size``. This matches the flag in the JSON API.
changelog_end
Furthermore, uses a work stealing thread pool for work JdbcLedgerDao has
to perform outside of the database thread pool. This solves some
flakiness observed in conformance tests against PostgreSQL-backed
ledgers.
changelog_begin
changelog_end
* daml-sdk-head: Quote all variables.
And use arrays where necessary.
* daml-sdk-head: Use Bash tests (`[[` and `]]`) rather than `test`.
Most importantly, they support the `&&` and `||` operators, because
they're a shell builtin, not a program.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-sdk-head: Refer to `${BASH_SOURCE[0]}` explicitly.
`BASH_SOURCE` is an array. Expanding an array gets the first element, but this
is unclear. We can clarify by explicitly expanding the first element.
* daml-sdk-head: Use `command -v` instead of `which`, as it's more standard.
* daml-sdk-head: Use semicolons judiciously.
In our first design draft, the maintainers of a contract key were
computed from the template payload. Now, they are computed from the
key itself. The rule `EvUpdCreateWithKeySucceed` still reflect the old
behavior but changing a single index from `t` to `v` fixes the issue.
CHANGELOG_BEGIN
CHANGELOG_END
Currently, a call to `create` for a template with a contract key
evaluates the key even before _executing_ the `create`, which is _way_
too early. See #5967 for examples what this implies.
This change moves the evaluation of the key into the execution of the
`create` and therein _after_ the evaluation of the `ensure` clauses,
which is where it belongs according to the DAML-LF specification.
Unfortunately, the specification does not say anything about the
relative evaluation order of the `agreement`, `signatory`, `observer`
and `key` clauses. Thus, I've deciced to evaluate the key after the
other three, which seems to be its natural place given its optionality.
This fixes#5967.
CHANGELOG_BEGIN
CHANGELOG_END
* Consume language for listing [re]actions in a foldAsync linearly
* eagerly consume as many Futures as possible
* restate one foldAsync test with Consume
* report value that failed to match
* add error locations and new `drain` primitive
* use `drain` to properly handle heartbeats at end of stream
* it doesn't make sense to delay the drain step
* rewrite other websocket test; document syntax
* get rid of StreamState, finally
* split Consume to a separate file
* plan split tests
* functions for consuming SplitSeq
* converting SplitSeq into stream actions
* exercise and check recursively
* add one 100-sized sample to the tests
* don't violate Iou's invariant
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* disable test pending tracking down duplicates
- as discussed with @leo-da
* Add debugging output to inclusion errors
This adds some more debugging output to inclusion errors in Bazel
which should hopefully help us track it down. These are the only call sites
in the Bazel source that can produce them. My suspicion is that it’s
coming from HeaderDiscovery but I’m not entirely sure what is off.
We’ll almost certainly have to add more output once we know which of
those 3 cases we hit but let’s do it step by step.
changelog_begin
changelog_end
* Bump url and hash
changelog_begin
changelog_end
changelog_begin
OVERRIDES CHANGELOG ENTRY FROM 2a8c93a614
[Ledger Integration Kit] Added new metrics for
``daml.index.db.*.translation`` to measure the time spent
translating to and from the serialized DAML-LF values when
fetched from the participant index.
changelog_end
At least for now we still somewhat pretend to have a linear version
history so we want to include this in future releases.
changelog_begin
changelog_end
* Port PerformanceEnvelope.TransactionSize perf test from Canton
to LedgerApiTestTool --perf-tests
changelog_begin
[Ledger Api Test Tool] Added the TransactionSize performance benchmark test.
changelog_end
* Review feedback from Samir
* Formatting
This is the first part of #5700
It adds tests that build create-daml-app using `daml build` and then
run the codegen and build the UI. Contrary to our main tests these
also run on Windows. This is actually reasonably simple by first
building the typescript libraries on Linux and then downloading them
on Windows.
There are two parts that are still missing from the tests in the main
workspace:
1. Building the extra feature. This should be fairly easy to add.
2. Running the pupeeter tests. At least MacOS and Linux should be
reasonably easy. I don’t know what horrors Windows will throw at
us. This step is what actually makes this a compatibility
test. Currently it doesn’t actually launch Sandbox and the JSON API.
Since this PR is already pretty large, I’d like to tackle those things
separately.
changelog_begin
changelog_end
patch Bazel on Windows (ci setup)
We have a weird, intermittent bug on Windows where Bazel gets into a
broken state. To investigate, we need to patch Bazel to add more debug
output than present in the official distribution. This PR adds the basic
infrastructure we need to download the Bazel source code, apply a patch,
compile it, and make that binary available to the rest of the build.
This is for Windows only as we already have the ability to do similar
things on Linux and macOS through Nix.
This PR does not contain any intresting patch to Bazel, just the minimum
that we can check we are actually using the patched version.
CHANGELOG_BEGIN
CHANGELOG_END
* Extract caching from participant-state as a library
This will be used to keep a cache of values to cut on LF translation cost when serving transactions.
changelog_begin
changelog_end
* Add dependency where missing
* Updates to support VMWare vairant of Hypervisor
* Update infra/macos/scripts/rebuild-crontask.sh
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Update infra/macos/scripts/run-agent.sh
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
