With this change, Daml exerciseByKey use the LF primitive
ExerciseByKey instead of the combinason of FetchByKey + Exercise.
CHANGELOG_BEGIN
CHANGELOG_END
The mapping from Azure job to local folder is managed by a handful of
JSON files under `$WORKDIR/SourceRootMapping`. Based on the sketchy
[documentation] that screams "don't mess with these definitely internal,
subject-to-change files", it seeems to me that our best bet for pinning
Windows work folders is to mess with these files. Specifically, take a
"version" of these files we like from a live CI machine and add those
specific files to the init script of our Windows nodes.
[documentation]: https://github.com/microsoft/azure-pipelines-agent/blob/master/docs/jobdirectories.md
So the first step is to collect these files, which is what this PR aims
to accomplish.
CHANGELOG_BEGIN
CHANGELOG_END
I'm not quite sure what's going wrong here, but this PR:
- Fixes a small bug in the process for determininng that we're _not_ in
a split release, which worked because the boolean expression crashing
is the same as the boolean expression returning false. But I'd still
prefer if it returned false without crashing.
- Changes the condition for skipping the Windows installer so that in
case of ternary+ logic we default to the harmless behaviour (including
it).
- Makes a change that might have some impact maybe on the way Azure
expands templates which may or may not result in the installer being
excluded when it should be excluded. But this time if it doesn't work
worst case we get an extra unused binary that uses some disk space,
instead of crashing the release.
CHANGELOG_BEGIN
CHANGELOG_END
In the split release process, the executable installer has to be created
later in the pipeline, once we have the Canton parts available.
CHANGELOG_BEGIN
CHANGELOG_END
Most of this is just documentation. The actual idea here is fairly
simple:
- The assembly repo pushes to S3 on each release (not yet implemented).
- The cronjob skips releases that are already there.
- Updating the top-level just piggybacks on what we already have.
changelog_begin
changelog_end
This follows the approach we used for the BazelCache stuff and splits
the 3 different operations in 3 separate modules + one Github utility
module to make it a bit easier to follow.
This is pure reshuffling, no functional change.
changelog_begin
changelog_end
Apologies for doing two things in one PR but they seemed somewhat
entangled.
For artifactory we follow the approach we use for Maven artifacts and
publish there directly.
For the SDK EE tarball, we just throw it in the split-release
directory and then the assembly repo can pick it up.
changelog_begin
changelog_end
Somewhat error-prone, so please review carefully.
Reasons we need this:
- Some file types are not properly handled by the script.
- The only exclusion mechanism we currently have (`NO_AUTO_COPYRIGHT`)
is overly coarse.
CHANGELOG_BEGIN
CHANGELOG_END
New year, new copyright, new expected unknown issues with various files
that won't be covered by the script and/or will be but shouldn't change.
I'll do the details on Jan 1, but would appreciate this being
preapproved so I can actually get it merged by then.
CHANGELOG_BEGIN
CHANGELOG_END
* Split up docs postprocessing & release in split-release
Follow up to #12172
The buld in the separate assembly should now only have to run the
sphinx-build step and then merge it with the non sphinx sources and
run lualatex on the pdf sources with the pdf fonts which is hopefully
simple enough that we can easily replicate it.
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
They migrated our account so hopefully this should work without any
other changes and fix our publishing issues.
I’m keeping the long timeout for now since I don’t know what an
appropriate timeout is.
changelog_begin
changelog_end
* revert to use synopsys detect 7.1.0 run-full-compat: true
CHANGELOG_BEGIN
CHANGELOG_END
* run on branch run full compat
run-full-compat: true
* run on branch run full compat
run-full-compat: true
* revert changes to run branch
For now, I’ve emitted Windows since this is not an artifact we want to
publish externally and for Canton Linux & MacOS covers our usecases.
changelog_begin
changelog_end
This publishes damlc & the Daml libraries as standalone artifacts so
Canton can avoid downloading the whole SDK tarball (hopefully).
changelog_begin
changelog_end
* Move toInterfaceContractId and fromInterfaceContractId out of Implements class
* Split Implements class into single-method classes
* Define toInterface outside its class to swap type arguments
This allows users to call 'toInterface @Interface', since the type of the template can usually be inferred
* Move interface classes and functions to DA.Internal.Interface
changelog_begin
changelog_end
* Split release process (PoC)
This PR adds the necessary infrastructure for the new split release
process. Releases are still triggered via the LATEST file but you can
choose between the old and the new release process by adding
SPLIT-RELEASE at the end of the line.
As a first step this publishes all artifacts we currently publish to
Github releases to our GCP bucket.
changelog_begin
changelog_end
* drop report-start
changelog_begin
changelog_end
* fix gcp bucket
changelog_begin
changelog_end
* review feedback
changelog_begin
changelog_end
* Update azure-pipelines.yml
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* SPLIT-RELEASE ->SPLIT_RELEASE
changelog_begin
changelog_end
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Closes#11317
* Add warning for uses of 'controller ... can' syntax
* Remove uses of 'controller ... can' syntax
* Add test cases for -W{,no-}controller-can
* Update docs to reflect 'controller ... can' syntax deprecation
changelog_begin
* Deprecate 'controller ... can' syntax.
* It will be removed in a future version of Daml.
* Instead, use 'choice ... with ... controller' syntax. Note that this does not implictly add the controller as an observer, so it must be added explictly as one (or as a signatory).
changelog_end
The "tell slack about failures" step is conditional on the branch name
being main; for whatever reason if we don't supply a branch name Azure
sets it to `main`, regardless of the fact that the commit is not in that
branch at all.
CHANGELOG_BEGIN
CHANGELOG_END
This is currently failing for the notice file update PR. While we
could try to hack around the git commands to do something sensible, I
don’t really see the point. We really only need this for PRs.
changelog_begin
changelog_end
It should now be possible for anyone to trigger the daily compat build
by adding a `run-full-compat: true` trailer to their commit message.
CHANGELOG_BEGIN
CHANGELOG_END
run-full-compat: true
* Bump go rules
Was debugging something else and this seemed simple enough to factor
it out.
changelog_begin
changelog_end
* Bump Bazel on Windows
changelog_begin
changelog_end
* bump bazel manifest
changelog_begin
changelog_end
* Update ghc patch
* Replace Implements proxy datatype with a class
* Expose 'Implements' class in Prelude
* Add DA.Internal.Desugar.HasMethod class
This allows us to get the type of a method of an interface through the functional dependency
* Convert interfaces from new desugaring
* Update Interface daml-test-file to use Interface class functions
* Update InterfaceDesugared daml-test-file
* Replace remaining uses of Is<Interface> methods with Implements
* Document HasMethod, Method and mkMethod
* Ignore _method_ bindings in convertBind
* Ignore interface desugaring types/classes/instances/functions in LF conversion
* update snapshot after pin on windows
changelog_begin
changelog_end
* Drop Scala 2.12 support
This only includes the CI/build system infrastructure
changes. Dropping compatibility layers from our code for 2.12 can be
done separately.
This is fine even in the context of backport builds since we already
disable the Scala 2.12 job for those anyway.
fixes#11315
changelog_begin
changelog_end
* Update bazel-java-deps.bzl
Co-authored-by: pbatko-da <pawel.batko@digitalasset.com>
Co-authored-by: pbatko-da <pawel.batko@digitalasset.com>
* interfaces: lf conversion of preconditions
This adds the LF conversion of preconditions from the desugared GHC code
to Daml LF for preconditions. This depends on the corresponding patch of
the GHC fork.
CHANGELOG_BEGIN
CHANGELOG_END
* pin linux stackage snapshot
* added a test
CHANGELOG_BEGIN
CHANGELOG_END
* pinned linux stackage snapshot
CHANGELOG_BEGIN
CHANGELOG_END
* additional InterfacePrecondition test
this checks that that conjunction of all preconditions of intererfaces
and templates is checked.
CHANGELOG_BEGIN
CHANGELOG_END
* pin stackage snapshot
* pin windows stackage snapshot
* update compile.yml (again)
* pin linux stack snapshot
* fix InterfacePrecondition test
* pin windows stackage
* Remove virtual choices
* Remove choices without a body in 'interface' definition
* Remove choices in 'template ... implements' section
part of #11372
changelog_begin
changelog_end
* Remove virtual choices cont.
Switch uses of virtual choices to fixed choice with method implementation
* update snapshot after pin on windows
* Disable failing interface tests with TODO #10810
* Make Oracle tests exclusive
This attempts to solve connection timeouts we often see in tests.
changelog_begin
changelog_end
* Revert "Make Oracle tests exclusive"
This reverts commit 7804c433b3476542fc156cad9d4da4755740b123.
* Run all Oracle tests sequentially
* ifaces: update ghc-parser
This updates the ghc-parser, including the `ifaceTypeRep` method
generation for interfaces.
CHANGELOG_BEGIN
CHANGELOG_END
* pin stack snapshot on unix
* add tests
* update ghc-lib hash
* pin stackage dependencies on unix
* pinned stackage windows
* CI: Set `PROJ_DIR` inside the bash lib, not outside.
We can't reference variables set outside.
CHANGELOG_BEGIN
CHANGELOG_END
* CI: Remove an unnecessary `export`.
* Use append-only schema in tests
changelog_begin
changelog_end
* Fix Oracle build
* Update ledger/ledger-on-memory/BUILD.bazel
Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>
* Run more conformance tests on append-only
* Run pruning tests in split-participant mode
* Disable multi-participant test in split participant
* Add missing ContractIdIT test
Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>
* interfaces: introduce TemplateOrInterface class in stdlib
The template typeclass is to strong for many applications. The new
constraint `TemplateOrInterface` only contains the methods to convert
contract IDs and choices.
CHANGELOG_BEGIN
CHANGELOG_END
* ghc-lib update, interface script tests
* pinned stackage on unix
* added missing implementation in preprocessor
* added test
* fixing tests
* remove Iface type
* pinned stackage windows
* make sure createAndExercise is not called on interfaces
Inspired by seeing `curl` TLS errors in a different repository's CI.
I don't know if this will help, but it's worth a try. Note that the
domain name has changed.
CHANGELOG_BEGIN
CHANGELOG_END
* Add the ability to check protobuf compatibility with the branch head alongside the latest release tag.
CHANGELOG_BEGIN
CI - add check to PR for protobuf breaking changes against the target branch head
CHANGELOG_END
Reminds people adding new users to the rotation that they should also
be added to DACH-NY/daml-language-ad-hoc to have a box in the cloud
for Windows testing.
Adds the possibility of adding comments at the beginning of the rotation file.
changelog_begin
changelog_end
* interfaces: consuming/non-consuming iface choices
We add the consumption behaviour to the interface choice definition and
typecheck accordingly.
CHANGELOG_BEGIN
CHANGELOG_END
update to new ghc-lib, conversion implementation
* update ghc-lib
* pinning stackage on unix
* pin stackage on windows
We believe the Blakduck logic is currently faulty. We have had a
violation on an NPM dependency, and Blackduck keeps reporting it despite
our having removed the dependency.
We believe that what is happening is that, in the first step of
checking, we udpate the Haskell dependencies, _and then check the
validity of the whole project_, which includes the NPM deps. Because
that fails, we never get to the step where we actually update the NPM
deps, and Blackduck is stuck forever.
The solution is to not fail on violations for the Haskell update steps.
Haskell deps are still checked in the second step, because, again, it is
checking the whole project.
CHANGELOG_BEGIN
CHANGELOG_END
* Check protobuf compatibility of release commits w.r.t. previous stable release
CHANGELOG_BEGIN
CHANGELOG_END
* Remove blank line
* Don't persist credentials
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* check-protobuf-against-stable.sh: SRC_DIR -> PROJECT ROOT + simplify
* Don't set LATEST_STABLE global in a function
* Simplify by using only the main work tree
* Simplify further as the check will be only run from `main`
* Move the check to `ci/build.yml` so that it is also run on PRs
* Enter the development environment to use tools
* Make variables read-only
* Support release branches and PRs targeting them
* Fix and document the reference tag finding logic
* Fix SYSTEM_PULLREQUEST_TARGETBRANCH and print it
* Don't log the source branch
* Fix comment formatting
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Enable Slack integration
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Don't check if the branch is a release one
...as the check won't be run on release branches.
* Add compatibility_stable_protobuf to collect_build_data
* Do not activate dev-env globally but only in sub-shells
* Add an explanation about why the check is not run on release branch commits
* Simplify further by leveraging `buf`'s ability to compare against branches
* Use `buf`'s `tag` locator instead of `branch`
* Split buf checks by module and remove previous manual check
* Explain how to run locally
* Use more future-proof WIRE_JSON for participant-integration-api
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
* Use stricter FILE for the ledger gRPC API
* Propose an explanation for WIRE in kvutils
* Fix comment typo
* Re-introduce linting configuration for kvutils
* Simplify explanation for KVUtils' breaking check rule
* Remove extra (C) header from 3rd-party proto
* Don't touch the copyright of google/rpc/status.proto
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
This bumps dotnet to the version required by the latest azuresigntool,
and pins azuresigntool for the future.
As usual for live CI upgrades, this will be rolled out using the
blue/green approach. I'll keep each deployed commit in this PR.
For future reference, this is PR [#10979].
[#10979]: https://github.com/digital-asset/daml/pull/10979
CHANGELOG_BEGIN
CHANGELOG_END
* Support adding tests as an hidden option
* Simplify existing suites
CHANGELOG_BEGIN
CHANGELOG_END
* Remove stale conformance suites from build.yml
* `--add` -> ``--additional`
* Re-add `--all-tests` as deprecated CLI option to be tested
* Move sandbox-classic pruning test to wall clock again
* Run KVCommandDeduplicationIT for sandbox append-only
* Tidy-up
* Also add participant pruning test to ledger-on-memory/single-participant
* Remove KVCommandDeduplicationIT on ledger-on-memory/append-only
* Run the full suite plus pruning (rather than just pruning) for ledger-on-memory with multiple participants and append-only
* Add KVCommandDeduplicationIT to ledger-on-memory append-only
* Exclude ConfigManagementServiceIT from ledger-on-memory append-only multi-participant
* Tidy-up
* Use KVCommandDeduplicationIT for sandbox-on-x too
* Fix merge
Add max dedup duration arg to all the test suites that include command dedup tests
* Make `--include` and `--additional` mutually exclusive
* Uniform formatting of multi-line strings
* Move exclusions after additions as they are applied last
* Re-disable deduplication test on sandbox with static time
* Re-disable deduplication test on sandbox-on-x
I've witnessed a build ([link], though that will likely expire soon)
that failed with a "No space left on device" error after skipping the
cleanup step because the machine still had 68GB free.
[link]: https://dev.azure.com/digitalasset/daml/_build/results?buildId=87591&view=logs&j=870bb40c-6da0-5bff-67ed-547f10fa97f2&t=deecee86-545a-596e-8b0d-fb7d606fe9f2
With the machines only having 200GB disk size total, cleaning up at 80
is probably going to start hampering the overall efficiency of the
cache. It may be time to think about increasing the disk size itself (or
finding ways to reduce the size requirements of our builds). Important
note, though: we can't actually increase the macOS disk size very much.
The failure happened on the `compatibility_linux` job.
CHANGELOG_BEGIN
CHANGELOG_END
* Add avg, stddev, p90, p99, requests_per_second numbers to be reported on slack similar to speedy_perf
changelog_begin
changelog_end
* changes based on code review
* fix failing job due to breaking function export
* vanilla job test on main pipeline
changelog_begin
changelog_end
* move job to daily compat tests
* add timeout to dev-env and changes based on code review
* Bump ghc-lib to include dropped parsing code for generic templates
changelog_begin
changelog_end
* bump snapshot
changelog_begin
changelog_end
* drop old generics file
changelog_begin
changelog_end
* drop other broken file
changelog_begin
changelog_end
* Bump again
changelog_begin
changelog_end
* bump to merged commit
changelog_begin
changelog_end
* and bump snapshots
changelog_begin
changelog_end
* Generate short to long name mapping in aspect
Maps shortened test names in da_scala_test_suite on Windows to their
long name on Linux and MacOS.
Names are shortened on Windows to avoid exceeding MAX_PATH.
* Script to generate scala test name mapping
* Generate scala-test-suite-name-map.json on Windows
changelog_begin
changelog_end
* Generate UTF-8 with Unix line endings
Otherwise the file will be formatted using UTF-16 with CRLF line
endings, which confuses `jq` on Linux.
* Apply Scala test name remapping before ES upload
* Pipe bazel output into intermediate file
Bazel writes the output of --experimental_show_artifacts to stderr
instead of stdout. In Powershell this means that these outputs are not
plain strings, but instead error objects. Simply redirecting these to
stdout and piping them into further processing will lead to
indeterministically missing items or indeterministically introduced
additional newlines which may break paths.
To work around this we extract the error message from error objects,
introduce appropriate newlines, and write the output to a temporary file
before further processing.
This solution is taken and adapted from
https://stackoverflow.com/a/48671797/841562
* Add copyright header
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Reminder to put and empty line between subject and body
changelog_begin
changelog_end
* Update ci/check-changelog.sh
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Skip subject in changelog check
This matches what unreleased.sh does.
Ideally we’d probably share the code but this is bash and I do not
like bash so I cannot be bothered to do this right now.
changelog_begin
changelog_end
* better error message
changelog_begin
changelog_end
Dropping the strict source name patch has resulted in empty
directories which Bazel is happy about but then fail at
runtime. They’re not quite empty, the digest points to 0x0a 0x00
so we match on that.
changelog_begin
changelog_end
I'm still not sure how or why this happens, but if we can detect it
"early" to fail and try to debug, we can also just try to fix it 🤷
CHANGELOG_BEGIN
CHANGELOG_END
