In #14622, I inadvertently changed the artifact names from using `macos`
to using `macOS`, which may break any number of external references,
from Canton's dependency (which is how I found out) to the assembly
script and the `daml install` command. This reverts that.
CHANGELOG_BEGIN
CHANGELOG_END
This adds parameters to the "split release" job. At the moment, when
run, the job just builds the current commit with a snapshot version.
With this change, we'll be able to build a split release for any commit
with any version tag.
The reason for this change is we want to trigger this job from the
assembly repo, to reduce the number of manual interventions needed in
the release process.
CHANGELOG_BEGIN
CHANGELOG_END
Azure is telling us to stop using 2019, so I've updated the job to be
compatible with 2022. (And future Windows images, too.)
CHANGELOG_BEGIN
CHANGELOG_END
Since #12645, we added a new pipeline, so we need to add a corresponding
entry.
As for #12645, the content of the files and the directory structure is
taken directly from a live CI node, as printed by the (now-named)
`workdirs` step.
CHANGELOG_BEGIN
CHANGELOG_END
This reverts a small part of 9e1e42d27c
(#12901).
Microsoft seems to have given us our free parallel jobs back, so we can
run the patch_bazel job again.
Fixes#12900.
CHANGELOG_BEGIN
CHANGELOG_END
This should hopefully get CI working again.
There are two changes in here:
1. We can no longer change our patched Bazel. I didn’t switch away
from the current patched version for now (we upload it to gcp bucket
so it still works fine even if we cannot build it) but if we upgrade,
we need to go to an unpatched version for now.
2. We need to get `az` from dev-env. I tested the self service compat
job stuff and it works fine with this but there is a chance other
parts don’t.
changelog_begin
changelog_end
Because there's no reason not to. The only obstacle to automating the
normal release process is that we need an explicit manual validation
step for our audit log when creating a release, but split-releases are
created in the Assemblty repo, so we can have the audit log over there.
The diff is going to be very messy because there's a lot of stuff moving
around. The only thing that is not just moving a job to a separate file
is the `ci/daily-snapshot.yml` file, which is 100% new and is meant as
a new Azure Pipelines entrypoint (which I will create after this gets
approved). I have made a reasonable effort to create individual commits
that simplify reviewing, but I expect it's still going to be kind of a
mess. I'm open to opening separate PRs to ~bump my stats~ move one job
at a time if that makes reviewing (and testing) easier.
CHANGELOG_BEGIN
CHANGELOG_END
The mapping from Azure job to local folder is managed by a handful of
JSON files under `$WORKDIR/SourceRootMapping`. Based on the sketchy
[documentation] that screams "don't mess with these definitely internal,
subject-to-change files", it seeems to me that our best bet for pinning
Windows work folders is to mess with these files. Specifically, take a
"version" of these files we like from a live CI machine and add those
specific files to the init script of our Windows nodes.
[documentation]: https://github.com/microsoft/azure-pipelines-agent/blob/master/docs/jobdirectories.md
So the first step is to collect these files, which is what this PR aims
to accomplish.
CHANGELOG_BEGIN
CHANGELOG_END
I'm not quite sure what's going wrong here, but this PR:
- Fixes a small bug in the process for determininng that we're _not_ in
a split release, which worked because the boolean expression crashing
is the same as the boolean expression returning false. But I'd still
prefer if it returned false without crashing.
- Changes the condition for skipping the Windows installer so that in
case of ternary+ logic we default to the harmless behaviour (including
it).
- Makes a change that might have some impact maybe on the way Azure
expands templates which may or may not result in the installer being
excluded when it should be excluded. But this time if it doesn't work
worst case we get an extra unused binary that uses some disk space,
instead of crashing the release.
CHANGELOG_BEGIN
CHANGELOG_END
In the split release process, the executable installer has to be created
later in the pipeline, once we have the Canton parts available.
CHANGELOG_BEGIN
CHANGELOG_END
New year, new copyright, new expected unknown issues with various files
that won't be covered by the script and/or will be but shouldn't change.
I'll do the details on Jan 1, but would appreciate this being
preapproved so I can actually get it merged by then.
CHANGELOG_BEGIN
CHANGELOG_END
* Split release process (PoC)
This PR adds the necessary infrastructure for the new split release
process. Releases are still triggered via the LATEST file but you can
choose between the old and the new release process by adding
SPLIT-RELEASE at the end of the line.
As a first step this publishes all artifacts we currently publish to
Github releases to our GCP bucket.
changelog_begin
changelog_end
* drop report-start
changelog_begin
changelog_end
* fix gcp bucket
changelog_begin
changelog_end
* review feedback
changelog_begin
changelog_end
* Update azure-pipelines.yml
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* SPLIT-RELEASE ->SPLIT_RELEASE
changelog_begin
changelog_end
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Drop Scala 2.12 support
This only includes the CI/build system infrastructure
changes. Dropping compatibility layers from our code for 2.12 can be
done separately.
This is fine even in the context of backport builds since we already
disable the Scala 2.12 job for those anyway.
fixes#11315
changelog_begin
changelog_end
* Update bazel-java-deps.bzl
Co-authored-by: pbatko-da <pawel.batko@digitalasset.com>
Co-authored-by: pbatko-da <pawel.batko@digitalasset.com>
* Make Oracle tests exclusive
This attempts to solve connection timeouts we often see in tests.
changelog_begin
changelog_end
* Revert "Make Oracle tests exclusive"
This reverts commit 7804c433b3476542fc156cad9d4da4755740b123.
* Run all Oracle tests sequentially
* Use append-only schema in tests
changelog_begin
changelog_end
* Fix Oracle build
* Update ledger/ledger-on-memory/BUILD.bazel
Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>
* Run more conformance tests on append-only
* Run pruning tests in split-participant mode
* Disable multi-participant test in split participant
* Add missing ContractIdIT test
Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>
* Add the ability to check protobuf compatibility with the branch head alongside the latest release tag.
CHANGELOG_BEGIN
CI - add check to PR for protobuf breaking changes against the target branch head
CHANGELOG_END
* Check protobuf compatibility of release commits w.r.t. previous stable release
CHANGELOG_BEGIN
CHANGELOG_END
* Remove blank line
* Don't persist credentials
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* check-protobuf-against-stable.sh: SRC_DIR -> PROJECT ROOT + simplify
* Don't set LATEST_STABLE global in a function
* Simplify by using only the main work tree
* Simplify further as the check will be only run from `main`
* Move the check to `ci/build.yml` so that it is also run on PRs
* Enter the development environment to use tools
* Make variables read-only
* Support release branches and PRs targeting them
* Fix and document the reference tag finding logic
* Fix SYSTEM_PULLREQUEST_TARGETBRANCH and print it
* Don't log the source branch
* Fix comment formatting
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Enable Slack integration
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Don't check if the branch is a release one
...as the check won't be run on release branches.
* Add compatibility_stable_protobuf to collect_build_data
* Do not activate dev-env globally but only in sub-shells
* Add an explanation about why the check is not run on release branch commits
* Simplify further by leveraging `buf`'s ability to compare against branches
* Use `buf`'s `tag` locator instead of `branch`
* Split buf checks by module and remove previous manual check
* Explain how to run locally
* Use more future-proof WIRE_JSON for participant-integration-api
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
* Use stricter FILE for the ledger gRPC API
* Propose an explanation for WIRE in kvutils
* Fix comment typo
* Re-introduce linting configuration for kvutils
* Simplify explanation for KVUtils' breaking check rule
* Remove extra (C) header from 3rd-party proto
* Don't touch the copyright of google/rpc/status.proto
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
* Support adding tests as an hidden option
* Simplify existing suites
CHANGELOG_BEGIN
CHANGELOG_END
* Remove stale conformance suites from build.yml
* `--add` -> ``--additional`
* Re-add `--all-tests` as deprecated CLI option to be tested
* Move sandbox-classic pruning test to wall clock again
* Run KVCommandDeduplicationIT for sandbox append-only
* Tidy-up
* Also add participant pruning test to ledger-on-memory/single-participant
* Remove KVCommandDeduplicationIT on ledger-on-memory/append-only
* Run the full suite plus pruning (rather than just pruning) for ledger-on-memory with multiple participants and append-only
* Add KVCommandDeduplicationIT to ledger-on-memory append-only
* Exclude ConfigManagementServiceIT from ledger-on-memory append-only multi-participant
* Tidy-up
* Use KVCommandDeduplicationIT for sandbox-on-x too
* Fix merge
Add max dedup duration arg to all the test suites that include command dedup tests
* Make `--include` and `--additional` mutually exclusive
* Uniform formatting of multi-line strings
* Move exclusions after additions as they are applied last
* Re-disable deduplication test on sandbox with static time
* Re-disable deduplication test on sandbox-on-x
* Fix status check in collect_build_data
follow up to #10270 which caused the linux & macos builds to go
through but then screwed us over in collect_build_data. I hate CI.
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* Oracle compliant append only schema
CHANGELOG_BEGIN
CHANGELOG_END
WIP : oracle on new appendonly schema
* diff to postgres dump, create consolidated view
* diff to postgres dump to ensure all oracle setup is equiv
* recompute sha for changed oracle flyway scripts
* drop old tables to prevent clash on name of new participant_events view
* recompute sha for flyway script
* prelim oracle StorageBackend
* Adds support for special preparedStatement for oracle
* Add support and wires to setObject by default everywhere
* Add the full OracleField suite with TODOs for convenience
* Wires OracleField suite to OracleFieldStrategy
changelog_begin
changelog_end
* enable debug version of oracle driver
* conversion Instant -> Timestamp for oracle
* WIP: primitive println debugging
* Passing PackagesSpec with appendonlyschema on Oracle
Rename size column to siz to avoid reserved word clash, including migration script for postgres
* include sha for new postgres migration script
* add missing copyright header
* cleanup
* passing party spec for appendonly on oracle
* passing configuration spec for appendonly on oracle
* scalafmt
* bazel buildifier reformat
* use db generic FETCH NEXT n ROW ONLY rather than limit for cross db compat
* siz instead of size for packages table on all dbs and schema
* revert enabling oracle jdbc debug
* Support Array[String] -> String conversion (and vice versa) for JSON array
Remove as aliases for tables as this does not work with oracle
Extract submitters clause for all db types
Use append transaction injector for oracle append only spec
* scalafmt
* correct oracle failing active contract spec tests
* wire in JdbcLedgerDaoCompletionsSpec
* remove semi-colons for ending statements that are problematic for oracle driver
* all tests up to divulgence passing for append only on oracle
* all appendonly tests passing on oracle
* remove ignore on fall back to limit-based query with consistent results
* do not change name of size column in packages table for mutable schema for all DBs
* do not change name of size column in packages table for mutable schema for all DBs
* standalone oracle appendonly schema script
regen shas on flyway scripts
revert some cosmetic refactoring in CommonStorageBackend
* Fixes conversion to parties from Oracle-JSON at flatEventWitnessesColumn
* Switches from composit SQLs to single SQLs at prepared statements to accommodate Oracle limitation
* Fixes arrayIntersectionWhereClause by applying patch from mutable Oracle schema integration
* Fixes queries with empty startExclusive Offsets
* First draw adding Oracle conformance test suites to CI
* wire in the oracle conformance tests for CI
* Use cross-db fetch next $n rows only syntax instead of limit syntax that works only for postgres/h2
* rename siz to package_size
* recompute shas
* scalafmt and include sha check for oracle append only flyway script
* correct missing package_size rename
* remove some todos -- correct corrupted V1__Init.sql
* Update ledger/ledger-on-sql/src/test/lib/scala/com/daml/ledger/on/sql/MainWithEphemeralOracleUser.scala
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* correct version number for postgres rename column scripts
* remove unnecessary migration tables for oracle append only
* review feedback: rename createEventFilter as requested, remove todos
* review feedback: case consistency
* review feedback: update todos with issue markers
* review feedback: cleanup
* review feedback: OracleField and OracleSchema cleanup
* Fixing Table generators to use preparedData for convenience
* Placing TODOs for refactorings later
* Renames initial append-only oracle script, for convenience
* Falls back to original behavior as far prepared statements go at couple of queries
Co-authored-by: Marton Nagy <marton.nagy@digitalasset.com>
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* Generate Bazel logs and upload to GCS
changelog_begin
changelog_end
* Move git_*_sha into variables template
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This currently breaks older releases because they require a different
Scala 2.12 version. It also adds zero value for a release that
defaults to Scala 2.12 and it adds basically no value for a release
that defaults to Scala 2.13 (see comment for details).
changelog_begin
changelog_end
This took me embarassingly long to understand and debug (partially
because afaict Azure is broken):
The issue is that in the current state, parameters.is_release is not
expanded when setting the env var. That makes sense. The variable is
only set at runtime but the ${{}} template expressions are expanded
before that (it works below in the condition since that’s not in a
${{}} and is evaluated at runtime).
Now if we look at the other env var that does work (the release tag)
we can see something interesting. We set it to the macro
$(release_tag) in build.yml. However, that is not expanded since
template expansion happens way earlier. So the template parameter is
set to the literal string "$(release_tag)". We then splice that in via
template expansion ${{parameters.release_tag}} and then at runtime
azure will expand the macro.
Just changing is_release to a macro however would break the use in the
condition (I think you might be able to fix that if you put it in a
string but that just seems even more hacky).
So this PR instead defines a new variable skip_tests which we define
in the job and the splice it in via a macro.
Confusingly, `$[variables.is_release]` is not expanded in an env
definition. Afaict this is simply a bug. The only difference between
macros and runtime expressions according to the docs is that runtime
expressions need to replace the full RHs and that macros can only
reference a single variable. Wouldn’t help much here either anyway if
we want to stick to the parameter instead of referencing a variable
directly (which maybe we don’t, it doesn’t seem to help much but
that’s a separate question).
changelog_begin
changelog_end
* Disable per commit windows compat tests
Windows version of #9370 to further reduce queues until we either get
more nodes or find another solution. We still test everything in the
daily run.
changelog_begin
changelog_end
* remove success check for skipped windows compat
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
This should reduce the pressure on CI nodes a bit. Note that we're still
running the full compatibility matrix on macOS as part of the daily
build.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Build Oracle tests, but don't run them.
CHANGELOG_BEGIN
CHANGELOG_END
* triggers: Switch to an environment variable for enabling Oracle tests.
* http-json: Switch to an environment variable for enabling Oracle tests.
* Disable running Oracle tests by default, not building them.
* triggers/service: Remove unused test dependencies.
* WIP : first cut at changed schema files for oracle
Define Oracle as DbType and handle necessary case match switches for it
recomputed shas for oracle migration scripts
Oracle fixtures
get things compiling
Able to connect to Oracle
Working through getting schema definitions functional with Oracle
runnable schema definitions only for active tables on oracle
delete commented lines in schema scripts
use oracle enterprise
correct inadvertently changed postgres schemas
WIP - latest oracle-ificiation
passing upload packages spec
add additional test for package upload entry read
correct typo in oracle database spec name
use BLOB for parties ledger_offset
package_entries use hex version of offset for range queries
reformat and update shas for sql scripts
binary numeric implicit conversion for oracle
correct duplicate exception text for oracle
parties test passing on oracle
add additional column to hold hex offset for party_entries
party_entries working for all dbs
scalafmt
Configuration ledger_offset should be BLOB
update sha of oracle sql files
enable passing tests in order
remove misleading null comments
define additional custom VARRAY types
add participant-integration-api-oracle tests to linux-oracle job
Add TODO for places where we need to deal with separate implicit imports for Oracle vs Postgres/H2
oracle implicit conversions for custom arrays and other problematic types
Do not override default debug level for all tests in participant-integration-api
CHANGELOG_BEGIN
Ledger API and Indexer Oracle Support
CHANGELOG_END
passing TransactionWriterSpec
passing JdbcLedgerDaoCompletionsSpec JdbcLedgerDaoDivulgenceSpec
passing JdbcLedgerDaoContractsSpec
All Oracle tests passing apart from one post-commit validation test
* Remove JdbcLedgerDaoValidatedOracleSpec as this is only relevant for classic postgres-backed sandbox
* rebase to master -- offsets are now varchar2 rather than blob
* remove use of DBMS_LOB operations
* remove all greater than/less than variants for DBMS_LOB
* revert postgres files that need not be touched
* code review feedback : avoid code duplication
* avoid indirection in type names for oracle arrays
* code review: HexString implicit conversions are not needed
* code review: Oracle case is not yet implemented for appendonlydao
* code review: Oracle case is not yet implemented for appendonlydao (cleanup import)
* code review: revert files that should not be touched
* address code review feedback: db specific imports for command completion become part of queries
* code review: perform db-specific reserved word escape to avoid case match
* code review: remove all dbms_lob comparison operations
* use simpler insert into with ignore dupes hint for oracle
* code review: avoid db specific match case in events range, use db specific limitClause
* code review: restore group by on Binary and Array fields for H2 and Postgres, disable for Oracle
* code review: restore group by on Binary and Array fields for H2 and Postgres, disable for Oracle
* code review: restore group by on binary and array fields for non-oracle dbs, honour the calculation of limit size from QueryParty.ByArith
* code review: honour the calculation of limit size from QueryParty.ByArith
* code review: drop user after oracle test
* code review: remove drop user as it throws errors due to dangling sessions
* code review: revert incorrectly changed postgres schema files
* code review: clean up TODOs
* Remove // before hostname for consistency with other oracle connection strings
* code review: unambiguously scope table column referenced in select and where queries
* code review: correct duplicate table alias
