* Check protobuf compatibility of release commits w.r.t. previous stable release
CHANGELOG_BEGIN
CHANGELOG_END
* Remove blank line
* Don't persist credentials
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* check-protobuf-against-stable.sh: SRC_DIR -> PROJECT ROOT + simplify
* Don't set LATEST_STABLE global in a function
* Simplify by using only the main work tree
* Simplify further as the check will be only run from `main`
* Move the check to `ci/build.yml` so that it is also run on PRs
* Enter the development environment to use tools
* Make variables read-only
* Support release branches and PRs targeting them
* Fix and document the reference tag finding logic
* Fix SYSTEM_PULLREQUEST_TARGETBRANCH and print it
* Don't log the source branch
* Fix comment formatting
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Enable Slack integration
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Don't check if the branch is a release one
...as the check won't be run on release branches.
* Add compatibility_stable_protobuf to collect_build_data
* Do not activate dev-env globally but only in sub-shells
* Add an explanation about why the check is not run on release branch commits
* Simplify further by leveraging `buf`'s ability to compare against branches
* Use `buf`'s `tag` locator instead of `branch`
* Split buf checks by module and remove previous manual check
* Explain how to run locally
* Use more future-proof WIRE_JSON for participant-integration-api
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
* Use stricter FILE for the ledger gRPC API
* Propose an explanation for WIRE in kvutils
* Fix comment typo
* Re-introduce linting configuration for kvutils
* Simplify explanation for KVUtils' breaking check rule
* Remove extra (C) header from 3rd-party proto
* Don't touch the copyright of google/rpc/status.proto
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
* Support adding tests as an hidden option
* Simplify existing suites
CHANGELOG_BEGIN
CHANGELOG_END
* Remove stale conformance suites from build.yml
* `--add` -> ``--additional`
* Re-add `--all-tests` as deprecated CLI option to be tested
* Move sandbox-classic pruning test to wall clock again
* Run KVCommandDeduplicationIT for sandbox append-only
* Tidy-up
* Also add participant pruning test to ledger-on-memory/single-participant
* Remove KVCommandDeduplicationIT on ledger-on-memory/append-only
* Run the full suite plus pruning (rather than just pruning) for ledger-on-memory with multiple participants and append-only
* Add KVCommandDeduplicationIT to ledger-on-memory append-only
* Exclude ConfigManagementServiceIT from ledger-on-memory append-only multi-participant
* Tidy-up
* Use KVCommandDeduplicationIT for sandbox-on-x too
* Fix merge
Add max dedup duration arg to all the test suites that include command dedup tests
* Make `--include` and `--additional` mutually exclusive
* Uniform formatting of multi-line strings
* Move exclusions after additions as they are applied last
* Re-disable deduplication test on sandbox with static time
* Re-disable deduplication test on sandbox-on-x
* Fix status check in collect_build_data
follow up to #10270 which caused the linux & macos builds to go
through but then screwed us over in collect_build_data. I hate CI.
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* Oracle compliant append only schema
CHANGELOG_BEGIN
CHANGELOG_END
WIP : oracle on new appendonly schema
* diff to postgres dump, create consolidated view
* diff to postgres dump to ensure all oracle setup is equiv
* recompute sha for changed oracle flyway scripts
* drop old tables to prevent clash on name of new participant_events view
* recompute sha for flyway script
* prelim oracle StorageBackend
* Adds support for special preparedStatement for oracle
* Add support and wires to setObject by default everywhere
* Add the full OracleField suite with TODOs for convenience
* Wires OracleField suite to OracleFieldStrategy
changelog_begin
changelog_end
* enable debug version of oracle driver
* conversion Instant -> Timestamp for oracle
* WIP: primitive println debugging
* Passing PackagesSpec with appendonlyschema on Oracle
Rename size column to siz to avoid reserved word clash, including migration script for postgres
* include sha for new postgres migration script
* add missing copyright header
* cleanup
* passing party spec for appendonly on oracle
* passing configuration spec for appendonly on oracle
* scalafmt
* bazel buildifier reformat
* use db generic FETCH NEXT n ROW ONLY rather than limit for cross db compat
* siz instead of size for packages table on all dbs and schema
* revert enabling oracle jdbc debug
* Support Array[String] -> String conversion (and vice versa) for JSON array
Remove as aliases for tables as this does not work with oracle
Extract submitters clause for all db types
Use append transaction injector for oracle append only spec
* scalafmt
* correct oracle failing active contract spec tests
* wire in JdbcLedgerDaoCompletionsSpec
* remove semi-colons for ending statements that are problematic for oracle driver
* all tests up to divulgence passing for append only on oracle
* all appendonly tests passing on oracle
* remove ignore on fall back to limit-based query with consistent results
* do not change name of size column in packages table for mutable schema for all DBs
* do not change name of size column in packages table for mutable schema for all DBs
* standalone oracle appendonly schema script
regen shas on flyway scripts
revert some cosmetic refactoring in CommonStorageBackend
* Fixes conversion to parties from Oracle-JSON at flatEventWitnessesColumn
* Switches from composit SQLs to single SQLs at prepared statements to accommodate Oracle limitation
* Fixes arrayIntersectionWhereClause by applying patch from mutable Oracle schema integration
* Fixes queries with empty startExclusive Offsets
* First draw adding Oracle conformance test suites to CI
* wire in the oracle conformance tests for CI
* Use cross-db fetch next $n rows only syntax instead of limit syntax that works only for postgres/h2
* rename siz to package_size
* recompute shas
* scalafmt and include sha check for oracle append only flyway script
* correct missing package_size rename
* remove some todos -- correct corrupted V1__Init.sql
* Update ledger/ledger-on-sql/src/test/lib/scala/com/daml/ledger/on/sql/MainWithEphemeralOracleUser.scala
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* correct version number for postgres rename column scripts
* remove unnecessary migration tables for oracle append only
* review feedback: rename createEventFilter as requested, remove todos
* review feedback: case consistency
* review feedback: update todos with issue markers
* review feedback: cleanup
* review feedback: OracleField and OracleSchema cleanup
* Fixing Table generators to use preparedData for convenience
* Placing TODOs for refactorings later
* Renames initial append-only oracle script, for convenience
* Falls back to original behavior as far prepared statements go at couple of queries
Co-authored-by: Marton Nagy <marton.nagy@digitalasset.com>
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* Generate Bazel logs and upload to GCS
changelog_begin
changelog_end
* Move git_*_sha into variables template
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This currently breaks older releases because they require a different
Scala 2.12 version. It also adds zero value for a release that
defaults to Scala 2.12 and it adds basically no value for a release
that defaults to Scala 2.13 (see comment for details).
changelog_begin
changelog_end
This took me embarassingly long to understand and debug (partially
because afaict Azure is broken):
The issue is that in the current state, parameters.is_release is not
expanded when setting the env var. That makes sense. The variable is
only set at runtime but the ${{}} template expressions are expanded
before that (it works below in the condition since that’s not in a
${{}} and is evaluated at runtime).
Now if we look at the other env var that does work (the release tag)
we can see something interesting. We set it to the macro
$(release_tag) in build.yml. However, that is not expanded since
template expansion happens way earlier. So the template parameter is
set to the literal string "$(release_tag)". We then splice that in via
template expansion ${{parameters.release_tag}} and then at runtime
azure will expand the macro.
Just changing is_release to a macro however would break the use in the
condition (I think you might be able to fix that if you put it in a
string but that just seems even more hacky).
So this PR instead defines a new variable skip_tests which we define
in the job and the splice it in via a macro.
Confusingly, `$[variables.is_release]` is not expanded in an env
definition. Afaict this is simply a bug. The only difference between
macros and runtime expressions according to the docs is that runtime
expressions need to replace the full RHs and that macros can only
reference a single variable. Wouldn’t help much here either anyway if
we want to stick to the parameter instead of referencing a variable
directly (which maybe we don’t, it doesn’t seem to help much but
that’s a separate question).
changelog_begin
changelog_end
* Disable per commit windows compat tests
Windows version of #9370 to further reduce queues until we either get
more nodes or find another solution. We still test everything in the
daily run.
changelog_begin
changelog_end
* remove success check for skipped windows compat
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
This should reduce the pressure on CI nodes a bit. Note that we're still
running the full compatibility matrix on macOS as part of the daily
build.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Build Oracle tests, but don't run them.
CHANGELOG_BEGIN
CHANGELOG_END
* triggers: Switch to an environment variable for enabling Oracle tests.
* http-json: Switch to an environment variable for enabling Oracle tests.
* Disable running Oracle tests by default, not building them.
* triggers/service: Remove unused test dependencies.
* WIP : first cut at changed schema files for oracle
Define Oracle as DbType and handle necessary case match switches for it
recomputed shas for oracle migration scripts
Oracle fixtures
get things compiling
Able to connect to Oracle
Working through getting schema definitions functional with Oracle
runnable schema definitions only for active tables on oracle
delete commented lines in schema scripts
use oracle enterprise
correct inadvertently changed postgres schemas
WIP - latest oracle-ificiation
passing upload packages spec
add additional test for package upload entry read
correct typo in oracle database spec name
use BLOB for parties ledger_offset
package_entries use hex version of offset for range queries
reformat and update shas for sql scripts
binary numeric implicit conversion for oracle
correct duplicate exception text for oracle
parties test passing on oracle
add additional column to hold hex offset for party_entries
party_entries working for all dbs
scalafmt
Configuration ledger_offset should be BLOB
update sha of oracle sql files
enable passing tests in order
remove misleading null comments
define additional custom VARRAY types
add participant-integration-api-oracle tests to linux-oracle job
Add TODO for places where we need to deal with separate implicit imports for Oracle vs Postgres/H2
oracle implicit conversions for custom arrays and other problematic types
Do not override default debug level for all tests in participant-integration-api
CHANGELOG_BEGIN
Ledger API and Indexer Oracle Support
CHANGELOG_END
passing TransactionWriterSpec
passing JdbcLedgerDaoCompletionsSpec JdbcLedgerDaoDivulgenceSpec
passing JdbcLedgerDaoContractsSpec
All Oracle tests passing apart from one post-commit validation test
* Remove JdbcLedgerDaoValidatedOracleSpec as this is only relevant for classic postgres-backed sandbox
* rebase to master -- offsets are now varchar2 rather than blob
* remove use of DBMS_LOB operations
* remove all greater than/less than variants for DBMS_LOB
* revert postgres files that need not be touched
* code review feedback : avoid code duplication
* avoid indirection in type names for oracle arrays
* code review: HexString implicit conversions are not needed
* code review: Oracle case is not yet implemented for appendonlydao
* code review: Oracle case is not yet implemented for appendonlydao (cleanup import)
* code review: revert files that should not be touched
* address code review feedback: db specific imports for command completion become part of queries
* code review: perform db-specific reserved word escape to avoid case match
* code review: remove all dbms_lob comparison operations
* use simpler insert into with ignore dupes hint for oracle
* code review: avoid db specific match case in events range, use db specific limitClause
* code review: restore group by on Binary and Array fields for H2 and Postgres, disable for Oracle
* code review: restore group by on Binary and Array fields for H2 and Postgres, disable for Oracle
* code review: restore group by on binary and array fields for non-oracle dbs, honour the calculation of limit size from QueryParty.ByArith
* code review: honour the calculation of limit size from QueryParty.ByArith
* code review: drop user after oracle test
* code review: remove drop user as it throws errors due to dangling sessions
* code review: revert incorrectly changed postgres schema files
* code review: clean up TODOs
* Remove // before hostname for consistency with other oracle connection strings
* code review: unambiguously scope table column referenced in select and where queries
* code review: correct duplicate table alias
* Add Oracle support in the trigger service
This PR migrates the ddl & queries and adds tests for this. It does
not yet expose this to users. I’ll handle that in a separate PR.
changelog_begin
changelog_end
* use getOrElse
changelog_begin
changelog_end
In #9169, I changed the compat jobs to not run on releases.
Unfortunately I forgot to update the `collect_build_data` job to know
about that. Hopefully after this has been merged we'll be able to rerun
\#9221.
CHANGELOG_BEGIN
CHANGELOG_END
As requested by @coceature.
Note: skipping the ts_lib job is enough to skip all compat tests because
they all depend on it, and in the Azure model if one of your
dependencies was skipped you get skipped too.
CHANGELOG_BEGIN
CHANGELOG_END
In Azure Yaml, by defualt, a step runs only if the previous step was
successful. However, that default _disappears if the step has an
explicit condition_. I believe we have a number of conditional steps
that have been written without that intention, and this is thus
restoring what I believe to be the original intention, i.e. _adding_ an
additional condition rather than _replacing_ the default one.
CHANGELOG_BEGIN
CHANGELOG_END
* Move artifact publishing out of yaml files
The current publishing process pretty much hardcodes the set of
artifacts we publish in the yaml config. This is a problem because we
always release from `main` so the yaml files are always
identical. However, we will add new artifacts over time and this
starts falling apart. This PR changes this such that the process
described in the yaml files is very generic and just uploads and
downloads everything in a directory whereas the details are handled in
bash scripts that will come from the respective release branch and are
therefore version-dependent.
As usual for these type of changes, I don’t have a great way to test
this. I did do some due diligence to test that at least the artifacts
are published correctly and I can download them but I can’t test the
actual publishing.
changelog_begin
changelog_end
* Update ci/copy-unix-release-artifacts.sh
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Update ci/copy-windows-release-artifacts.sh
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Update ci/publish-artifactory.sh
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Merge Maven uploads for different Scala versions
It turns out Maven will abort an existing staging operation if you
create a new one. This means our jobs race against each other. We
could try to fix that by either sequencing the jobs in a clever
way (annoying and can break things like rerunning if only parts
failed), or by creating more profiles (unclear if you can even have
two profiles for the same group id, even if you do, it’s annoying to
merge).
So in this PR I (grudgingly) merged both uploads into the Haskell
script. This isn’t all bad:
1. It moves some logic from bash embedded in yaml string literals into
Haskell code.
2. It duplicates some versions but it removes duplication in other
places so overall not too much worse.
3. It does however, make things slower. We don’t run this stuff in
parallel. That said, the release step is relatively small (< 5min) and
it only runs on Linux.
We could add CLI arguments to make the Scala versions configurable for
local development. Given that this is blocking releases, I wanted to
get something in that works first and then see what we need in that regard.
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* Stop pretending strings are booleans
Sorry for all the mess here. I’m not capable of programming in yaml.
It turns out is_release is a string not a boolean and
```
and('false', eq('true', 'true'))
```
is true.
I hate everything about this.
changelog_begin
changelog_end
* Name bash step to reduce confusion
changelog_begin
changelog_end
* fix version in test
changelog_begin
changelog_end
* names can’t have spaces apparently
changelog_begin
changelog_end
* Fixup scala 2.13 check
Somehow I managed to misread the helpcheck and get confused by my
experiments and thought semver produces an exit code of 1,0,-1 but
actually it writes that to stdout.
changelog_begin
changelog_end
* Update ci/build.yml
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Update ci/build.yml
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Port Ledger API Test Tool to Scala 2.13
And with that we’re finally at //... building on Scala 2.13.
changelog_begin
changelog_end
* Fix build on 2.12
changelog_begin
changelog_end
* Fix kvutils export on 2.13
changelog_begin
changelog_end
* separate OracleQueries from PostgresQueries
- with some changes from 8161e63189 courtesy @cocreature
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* abstract BIGINT
* json, signatories, observers columns
* compatible lastOffset
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* oracle functions for select (single template ID), insert
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* add oracle branch to integration tests
* oracle CLI configuration for json-api
* run integration tests with ojdbc in classpath
* update maven_install for ojdbc
* drop table if exists for Oracle
* make create DDLs and drops more planned out; drop in reverse order for Oracle integrity
* repin maven
* port agreement_text
* port (by removal) array part of ledger offset update
* use CASE instead of JSON map lookup for multiparty offset update
* simplify self types
* fix contract archival
* repin
* remove selectContracts in favor of selectContractsMultiTemplate
* move Oracle test execution to separate build target
* move websocket test to itlib
* make a bad array instance for Oracle
* report actually-available JDBC drivers only
* configure Oracle test from CI
* attempt with platforms and constraints
* a mismash of bazel to get it to conditionally enable oracle testing
* fix dep resolution in Scala 2.13
* make the Oracle test a stub (inits and does empty DB query)
* remove commented unused deps
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* repin
* we never supply a value for the surrogate ID columns
- suggested by @cocreature; thanks
* add not null to json in DB-specific place
- suggested by @cocreature; thanks
* why DBContractKey
- suggested by @cocreature; thanks
* textType isn't finalized
- suggested by @cocreature; thanks
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
This fixes Scaladoc and our pom file generation.
It also clears up the confusing error around gatling and removes a
redundant dependency on sbt (no idea why we had that in the first
place) both of which resulted in Scala 2.12 dependencies in our 2.13
lockfile which is obviously bad.
With this, we should now be ready to publish Scala 2.13 artifacts once
the ledger API test tool PR lands.
changelog_begin
changelog_end
Tests are still missing and blocked on #8821.
The main change here is the switch from `ArraySeq[Byte]` to
`ArraySeq.ofByte`. `ArraySeq` allows for boxed and unboxed
representaitons. That means that `ArraySeq[Byte]unsafeArray` does not always return
an Array[Byte] (boxed version would be Array[AnyRef]).
Apparently collection-compat has taken the yolo approach and pretends
it can give you an Array[Byte] anyway 🤷 Scala 2.13 on the other
hand, does things properly in this regard which means the code relying
on `unsafeArray` fails to compile.
`ArraySeq.ofByte` is the specialized unboxed version where none of
this is an issue on both 2.13 and 2.12.
changelog_begin
changelog_end
* Draw the rest of the Scala 2.13 owl
Not quite but pretty close and this switches us over from inclusions
to exclusions which makes it much easier to track.
Ledger API test tool should be fixed by #8821. Non-repudiation needs a
tiny bit of work since unwrapArray doesn’t work the same on 2.13 but
shouldn’t be hard to fix.
changelog_begin
changelog_end
* Fix ScriptService tests
Those tests were all dumb. They asserted on a fixed order while the
function to sort the things was broken so we ended up with the random
Map order which is unsurprisingly not the same.
This is easily fixed by fixing the sort function.
There is also a second issue with query not sorting.
changelog_begin
changelog_end
* Turns out if you fix one test the next one breaks
And clearly nobody ever tested this or give this a second thought.
changelog_begin
changelog_end
fixes#8498
This fixes the error in 2.13 wtr to the location change of Predef. It
doesn’t yet address the warning wtr to the import of higherKinds. For
now, our build ignores that warning. Trying to figure out if we can
get away with a breaking change here or if we need to hide that change
behind a flag but either way, no need to block fixing the actual error
on that.
changelog_begin
changelog_end
* Disable MacOS CI jobs
5/6 macos nodes are down and we cannot fix it quickly, so to unblock
everyone, let’s disable those jobs for now.
I deliberately did not remove MacOS from releases. Those really should run on MacOS.
changelog_begin
changelog_end
* Undo unnecessary changes
changelog_begin
changelog_end
* Allow skipping macos jobs
changelog_begin
changelog_end
