This bumps the timeout of the compat tests on PRs to 360 minutes
matching other jobs on a PR (we mainly hit this if ghc-lib is rebuilt)
and the timeout on the daily jobs to 720 minutes (we hit this if
_everything_ is rebuilt).
I am slightly worried about the timeout on the daily job. After having
taken a look at it, there are a few reasons how we ended up here:
1. We started including more tests, e.g., sandbox-classic. Not much we
can do here, those tests are useful.
2. We have a very large number of snapshots for 1.3.0. There are a few
reasons for this:
1. Timing: We branched off early for the 1.2.0 release so the first
snapshot for 1.3 was on June 3th. For 1.4 it looks like the first
snapshot will be on July 15th so that’s roughly 2 extra
snapshots just due to timing.
2. Additional snapshots: We had one broken snapshot due to a broken
VSCode extension that we didn’t delete (probably not worth doing
at this point). We also had to backport to an old snapshot which
resulted in another extra snapshot. We also had one extra
snapshot which was supposed to be the RC but wasn’t since the
ANF revert needed to go in.
The only thing that is clearly useless is the one broken snapshot
but that doesn’t change things that much. I see 2 orthogonal
options for improving this assuming we agree that the current
runtime is worryingly high.
1. Prune snapshots more aggressively, e.g., only include the last 3
snapshots. That’s a pretty arbitrary decision but it would
enforce a hard limit.
2. Reduce test combinations. E.g., only test snapshots vs stable
releases but not snapshots vs snapshots.
3. We end up forcing a full build quite frequently. Here are just 2
examples of how we’ve done that so far.
1. Upgrade rules_haskell. Basically all tests are run by a Haskell
binary so this forces a full rebuild.
2. Change runfiles of `daml`.
I don’t think there is much we can do about 1 or 3 which leaves us
with 2. One not entirely unreasonable option is to just do nothing. We
did have periods where things went pretty smoothly for the most part
and each month we reset to a much smaller number of releases (we also
have to start throwing out old stable releases at some
point). Otherwise reducing the number of test combinations seems the
most promising option to me.
changelog_begin
changelog_end
* Fix fine-grained exclusions in ledger-api-test-tool
31995ee000 accidentally changed
`allTestCaseNames` to be a list of test suite names instead of a list
of test case names. That breaks fine-grained exclusions that we use in
our compatibility tests. I’ve also added a very basic test that both
the coarse-grained and fine-grained syntax is supported.
changelog_begin
changelog_end
* Update ledger/ledger-api-test-tool/BUILD.bazel
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Generate JS + typings to speed up dam2js
This speeds up daml2js to the point where it’s basically instant on
the GSG (based on 3 runs before and after it’s somewhere between a
40-50x speedup).
Most of this is fairly straightforward. I looked at the generated code
from typescript and the docs for declaration files and adapted things.
There is one interesting point here:
We set the TypeScript configuration to commonjs + ES5. While ES6
modules work fine via `yarn start`, they unfortunately work less fine
in nodejs and in particular in jest. I spent some time trying to fix
it and in the end decided that this is not worth doing since all our
users have to do the same which sucks.
Therefore, the codegen also emits ES5 + commonjs so this is not a
breaking change.
The tests pass and I went through the GSG manually to check things
still work and also verified that types show up correctly in VSCode.
What I was sadly unable to do is to keep the ESLint test for the .d.ts
files. typescript-eslint really wants everything to belong to a
typescript project with a tsconfig.json which doesn’t make sense
here. I don’t think that’s a huge loss. We still have ESLint for the
JS files.
changelog_begin
- [daml2js] daml2js now generates JS + typescript declaration files
directly instead of generating TypeScript code and invoking `yarn` to
generate the JS + the declaration files. This does not require any
changes to your code but it brings a significant speedup and removes the
dependency on `yarn` for running `daml codegen js`.
changelog_end
* Update language-support/ts/codegen/src/TsCodeGenMain.hs
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
* Update language-support/ts/codegen/src/TsCodeGenMain.hs
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
* Add details about validity windows
* Reword description of auth service concepts
* Elaborate on auth flow dealing with token/credential expiry
CHANGELOG_BEGIN
CHANGELOG_END
* declare key type in Scala codegen output
* add `key` type and function to TemplateCompanion
* template key exercise* case
* test idiomatic ExerciseByKey creation
* add changelog
CHANGELOG_BEGIN
- [Scala Codegen] Support for creating exercise-by-key commands, using same
exercise functions as exist for contract-IDs and the ``.createAnd`` pattern.
See `issue #6466 <https://github.com/digital-asset/daml/pull/6466>`_.
CHANGELOG_END
* Ledger API Test Tool: deprecate unneeded tests and options
Furthermore, now all active tests are now run be default without needing
to specify --all-tests. The scheduler makes sure to run non-isolated
tests last (and sequentially).
Fixes#3747Fixes#6518
A bit of historical context: tests in the Ledger API Test Tool used to
be the sandbox integration tests. As part of the Ledger API Test Tool
project, those tests have been ported one-to-one, with little review of
those same tests.
Examining the tests that cause #3747, it became evident that those tests
were originally put in place to verify that an isolated sandbox spun up
for a single test could withstand a number of commands slightly lower
than what at the time was the hard-coded back-pressure threshold. Hence,
implementing back-pressure on the Ledger API Test Tool would have
basically negated the usefulness of those tests. Furthermore, those
tests can be easily passed by simply raising the back-pressure
threshold. As such, they do not convey any meaningful information for
the DAML ledger implementor. This means we are retiring those tests. The
tests will be nominally kept in for a deprecation period to not break
existing build scripts, but running them explicitly will simply show
them as skipped. The documentation has been updated with the information
necessary to users to deal with retired tests.
Since all tests are now run by default --all-tests is being deprecated.
--load-scale-factor is also deprecated as it was used only by
TransactionScaleIT.
changelog_begin
[Ledger API Test Tool] The LotsOfPartiesIT and TransactionScaleIT test
suite have been deemed not providing relevant signal to DAML ledger
implementers and have been retired. The tests will be nominally kept in
but will be skipped while they are in a deprecation period. You are
advised to remove explicit references to those tests before they are
fully removed.
[Ledger API Test Tool] All tests are now run by default. The --all-tests
option is now ineffective and deprecated. You are advised to remove its
usages from your build scripts. Non-isolated tests that could affect the
global state of the ledger and interfere with other tests are now
automatically scheduled by the test tool to run sequentially at the end
of the run.
[Ledger API Test Tool] The --load-scale-factor option is now unused and
deprecated. You are advised to remove its usages from your build
scripts.
changelog_end
* Add test for deprecated CLI options
* Address https://github.com/digital-asset/daml/pull/6651#discussion_r452100158
* Address https://github.com/digital-asset/daml/pull/6651#discussion_r452126405
* Address https://github.com/digital-asset/daml/pull/6651#discussion_r452143811
* Address test failures
The default cache directories in yarn don’t work in the Bazel
sandbox. This results in Yarn creating tons of temp directories for
each test that are never cleared up.
This PR changes this to create actual temp directories that are
properly cleaned up.
changelog_begin
changelog_end
* REPL: Factor out Dar loading
* REPL accept multiple DARs
* REPL CLI take multiple packages
CHANGELOG_BEGIN
- [DAML REPL] The REPL now supports loading multiple DARs specified at
the command line. The DARs are loaded as data-dependencies, and the
modules of each DAR's main DALF will be imported into the REPL session
at startup.
CHANGELOG_END
* REPL test loading multiple DARs
* REPL Load packages from current session
* REPL func-test test symbols from two DARs
* Update compiler/damlc/daml-compiler/src/DA/Daml/Compiler/Repl.hs
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Fix leakage of SDK installations
This fixes an issue where the integration tests leaked SDK
installations. Easily verified by `ls /tmp/extra-dir*` before and
after running the tests.
changelog_begin
changelog_end
* I hate windows so much
changelog_begin
changelog_end
* sandbox: Move helpers in EmptyLedgerIT to a companion object.
* sandbox: Accept a new time model when none is set.
CHANGELOG_BEGIN
- [Ledger API Server + (Sandbox / Ledger Integration Kit)]
Accept a new time model if none is set. Previously, it would
erroneously be rejected because the generation number submitted to
was incorrectly set to `2` rather than `1`.
This would not affect most users of Sandbox or other kvutils-based
ledgers, as if a configuration is set automatically on startup when
creating a new ledger. It only affects users who explicitly override
the initial ledger configuration submit delay to something longer than
a few milliseconds.
CHANGELOG_END
* sandbox: Ensure that completions can be streamed from the beginning.
* sandbox: In EmptyLedgerIT, use the submission service.
* sandbox: Split EmptyLedgerIT into two.
* sandbox: In CompletionServiceWithEmptyLedgerIT, assume CI will be slow.
In other words, don't create a completion stream and expect a submission
to show up in a reasonable amount of time.
This changed by the revert of the ANF changes which is harmless by the
same reasoning that made bumping it harmless when we introduced it.
changelog_begin
changelog_end
We saw a flake recently where PostgreSQL stopped accepting connections
during a CI run, leading the build to fail. This increases the number of
connections to 200 from the default of 100, hopefully mitigating issues
such as this one.
CHANGELOG_BEGIN
CHANGELOG_END
* Revert ANF changes and add a testcase for evaluation order
After careful consideration, we decided that the change in evaluation
order that was accidentally introduced by the ANF changes should be
considered a breaking change or arguably even a bug and should not
land in 1.3.0.
Therefore, this PR reverts the following commits:
1. 353d0da6f7
2. a45b51042f
3. 04c7b2af7f
4. a624dd7242
5. b3aab72cee
Other PRs mostly had trivial merge conflicts that I resolved. The two
most interesting ones here are probably
1. https://github.com/digital-asset/daml/pull/6576 which was easy to
resolve and the change to return SEValue instead of SExpr is still
nice and useful even if we do not need the guarantees.
2. it https://github.com/digital-asset/daml/pull/6542 which required
some changes since the constructors changed. If you want to review
those changes in detail (they are pretty straightforward so not too
important), it’s probably easiest to check out this PR and run
```
git diff 2cd2a8f2a8
daml-lf/interpreter/src/main/scala/com/digitalasset/daml/lf/speedy/Compiler.scala
```
to see the diff to the parent commit of the first commit that
introduced ANF.
changelog_begin
changelog_end
* row_id changes
* fixing inserts
* replacing offset with row_id in the flat transaction stream queries
* fixing flat transaction query, updating H2 migration script
* fixing formatting
* ACS query pagination relies on row_id instead of ledger offset
* give a name to the index that we have to drop
* give a name to the index
* Fixing events range query it can return SQL nulls on empty DB.
* remove the debug println
* remove outdated comment
* removing unused orderByColumns constant
* getting rid of new `Source.flatMapConcat` calls that were added as part of this PR.
CHANGELOG_BEGIN
1. ACS, Flat Transaction and Transaction Tree stream pagination based on event_sequential_id instead of event_offset.
2. Events ordering based on the order of insertion: order by event_sequential_id instead of order by (event_offset, transaction_id, node_index).
CHANGELOG_END
* reverting changes to V13 H2 migration script,
figuring out the name of the index that has to be dropped
* Addressing code review comments:
- replacing scalaz Option.cata with stdlib Option.fold
- moving implicit val def into import
* Addressing code review comments:
- extracting re-usable stream query functions
* forcing postgres to use index when looking up lower and upper bound row ids
* fixing the query when it is run on an empty ledger
* resolving rebase conflicts
* Update ledger/sandbox/src/main/scala/com/digitalasset/platform/store/dao/events/EventsRange.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* fetching a single row, fetchSize should NOT matter
* Adding integration test to reproduce invalid order of archived, created events
The test fails, which is expected.
* Fixing the order of archived, created events triggered by exercise
* Addressing code review comments and cleaning up
* Renaming row_id to event_sequential_id
* Investigating flaky tests
* Fixing formatting
* Revert HOTFIX-flaky-client-server changes
`bazel test --runs_per_test=50 //ledger/participant-state/kvutils:reference-ledger-dump` passed on this branch.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Kill stale sandbox processes on macos and linux
changelog_begin
changelog_end
* Update build.sh
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* exitcodes are hard
changelog_begin
changelog_end
* Also kill stale sandboxes in compat tests
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Return the ledger beginning as the ledger end on an empty ledger
Previously, the services would return `Offset.beforeBegin` to the
clients, and the other services deem this offset to be invalid.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Import DurationInt
* Kill unused import
* Use ApiOffset for the external ledger begin
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* replace traverseU and sequenceU with traverse and sequence
- with -Ypartial-unification on, the extra Unapply typeclass lookup is
unnecessary
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* limit imports; we only need *> and void
While we do have additional tests, they pass in a single-participant
setup even on older SDK so no new exclusions necessary.
Side note: The autoupdate PR for yesterday’s (second) snapshot failed
since Maven was too slow and the ledger-api-test-tool wasn’t there
yet. I would suggest that for now we wait and see if this happens
frequently (it definitely doesn’t happen always). If it does, we could
get the checksum from the current build and avoid the race.
changelog_begin
changelog_end
* Update rules_haskell
- Fixes the issue where a `sh_test` wrapping a `haskell_binary` couldn't
add runfiles as is the case with other Bazel rules.
CHANGELOG_BEGIN
CHANGELOG_END
* Save the runfiles environment at start-up
To work around the fact that `withProgName` overwrites `argv[0]`.
See https://gitlab.haskell.org/ghc/ghc/-/issues/18418.
* damlc_compile_test include damlc runfiles
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Previously, daml2js generated Text without any intermediate
representation. This both made the code pretty messy and it makes it
super hard to implement things like generating JS source + TS typings
instead of generating typescript.
This PR addresses this by first generating intermediate types which
are then rendered into Text in a separate step. This should hopefully
make it almost trivial to generate JS source + typings in the future
my implementing two rendering functions.
The types are somewhat adhoc atm. I expect that they will probably
change a bit anyway once we switch to JS + typings so I would like to
avoid bikeshedding the exact definitions too much.
I didn’t bother to preserve the pretty printing exactly as before. If
we really care about pretty output (why should we?), I would suggest
to switch to a proper pretty printing library (in a separate PR).
I apologize for the large PR. Not quite sure how to make it smaller.
changelog_begin
changelog_end
So that the resulting value is returned from the `execute` function, instead of being assigned to the `returnValue` machine state.
Rename the parent class from `SBuiltinMaybeHungry` to `SBuiltinEffect`, and have this be the parent of a few more subclasses which cannot meat the _pure_ interface.
Declare method `executeEffect` of the parent class `SBuiltinEffect` which has the same type as the original `execute`.
changelog_begin
changelog_end
* Release 1.3.0 snapshot
This includes the fix for the indexer, a fix for the stackoverflow and
unpublishes daml-on-sql from GH releases.
changelog_begin
changelog_end
* Bump commit
changelog_begin
changelog_end
The running trigger store is now written to synchronously by the server, so there's no need for retry logic.
Also assert equality of trigger ids rather than an arbitrary predicate, as it gives better error messages and we don't use other types of predicates for this function.
CHANGELOG_BEGIN
CHANGELOG_END
If you have something like `http://localhost:8080` the port is handled
correctly. However, if you have `http://localhost/abc:8080` the port
will silently be ignored (`http://localhost:8080/abc` would be
correct). That is clearly wrong so this PR fixes it.
changelog_begin
- [DAML Script] Fix an issue where the `port` was ignored for
non-empty paths in the url when running DAML Script over the JSON API.
changelog_end
Buildifier now comes with a handy attachment to catch single `\`
characters inside strings and replace them with `\\` if the escape
sequence is invalid. Skylark/Python will do this at runtime anyway; this
just makes it clearer what the actual behavior is.
I needed to change `\` characters at the end of lines to `\\` manually
in order to stop Buildifier from simply concatenating the lines
together. Everything else was automatic.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/integrity-check: Always reset the output color.
Contrary to popular belief, terminal colors are hard. Resetting to
"white" isn't good enough; you don't know which color the user's
terminal should be set to. We need to print `AnsiColor.RESET` to ensure
the terminal is properly reset.
This changes the way we print so we _always_ set the color and then
reset it in one operation, making it impossible to accidentally leave
the terminal using a specific color. It also fixes the above bug as a
side effect.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/integrity-check: Print the total outside `go`.
* kvutils/integrity-check: Push the error outwards.
I'd rather avoid calling `sys.exit` inside a loop if I can.
* kvutils/integrity-check: Implicits for colors.
Before:
```scala
Print.green("Hello!")
```
After:
```scala
println("Hello!".green)
```
GHC has weird restriction on version numbers which damlc inserits so
we need to use `ghc_version` instead of `sdk_version`. That only makes
a difference for snapshot versions where the `-snapshot.` part is
replaced by `.`.
changelog_begin
changelog_end
Ask @nickchapman-da for details on what this transform does and why it's
better. I have no clue.
What I do understand is that current master has a stack-consuming
behaviour that just so happens to not reach the 1mb default stack size
on the tests we run, and that this presumably better version of a let
transform had been left out from the original ANF PR because it was
pushing the stack consumption over that 1mb limit.
Besides reintroducing the "better" let version, this PR applies an
almost full CPS transform to the set of mutually recursive functions
that grow the stack. Combined with the trampoline, this should mean the
ANF transform itself is done in (almost) constant stack space.
The one exception to that is the `flattenAlts` function, which has been
left as a plain imperative loop. This does mean that we consume stack
linearly with the nesting level of alts, so conceivably this could still
blow the stack. There is no fundamental reason that makes this hard, it
just seemed unnecessary for the stack consumption and potentially
damaging to performance (changing a native Java loop on an array to a
recursive list consumption).
To verify the stack-consuming behaviour on master, one can apply this
patch:
```patch
diff --git a/ledger/sandbox/BUILD.bazel b/ledger/sandbox/BUILD.bazel
index a717e66fb..4c953b3ea 100644
--- a/ledger/sandbox/BUILD.bazel
+++ b/ledger/sandbox/BUILD.bazel
@@ -124,6 +124,7 @@ da_scala_library(
da_scala_binary(
name = "sandbox-binary",
+ jvm_flags = ["-Xss256k"],
main_class = "com.daml.platform.sandbox.SandboxMain",
resources = ["src/main/resources/logback.xml"],
visibility = ["//visibility:public"],
```
then run
```
bazel test -t- //ledger/sandbox:conformance-test-contract-id-seeding-memory
```
The same process can be repeated on this branch to verify that, at
least, this now runs under 256kb of stack for all our tests. I don't
know of a good way to get a stronger guarantee on stack allocations.
While my primary motivation here is correctness (and termination, I
guess), this does seem to yield a modest performance improvement. On a
test machine, I got the following results as confidence intervals:
* `master` (b4915a4bd7): 54.00, 54.56
* this branch: 52.54, 53.11
I've only ran the benchmark once on each and don't have enough
experience with it to know how reliable those numbers are. I'll try to
run some more.
CHANGELOG_BEGIN
CHANGELOG_END
Note for reviewers: This is a hotfix for a snapshot. I’m not going to
into why we need this here. As for the version number we decided that
the least confusing option here is the following:
Use the sha and number of commits as usual but use the date of the
original snapshot that we are patching here.
changelog_begin
changelog_end
* kvutils/integrity-check: When there's an exception, crash.
The integrity checker would previously hang around for a long time in
the event of an exception. This makes sure all threads are shut down
immediately.
* kvutils/integrity-check: Increase the timeout per submission.
Larger submissions time out on my machine.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/integrity-check: Rewrite without `Await`.
* kvutils/integrity-check: Close the file after running.
small clean up in the testing library TransactionBuilder, bsaically add a Submitted and a Committed version for empty transaction constant and build method.
plus a bit of cleaning in tests using TransactionBuilder
CHANGELOG_BEGIN
CHANGELOG_END
* Removed archived_by_entry from CommitContext and DamlContractState.
* Removed references to getEntryId from test code.
* Do not pass log entry ID to committers.
* Log entry ID is no longer required for computing submission outputs.
* Log entry ID is no longer required for computing submission outputs.
* Code tidying.
CHANGELOG_BEGIN
CHANGELOG_END
