Commit Graph

415 Commits

Author SHA1 Message Date
Moritz Kiefer
b9c36c9a5b
Disable per commit windows compat tests (#9755)
* Disable per commit windows compat tests

Windows version of #9370 to further reduce queues until we either get
more nodes or find another solution. We still test everything in the
daily run.

changelog_begin
changelog_end

* remove success check for skipped windows compat

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
2021-05-20 10:37:59 +00:00
Gary Verhaegen
83d60f0561
more frequent Windows cache cleanup (#9743)
We'd like this to run more than once an hour, and Azure doesn't support
that as a cron.

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-19 14:24:47 +02:00
Gary Verhaegen
f249eec24f
drop per-commit compatibility_macos (#9730)
This should reduce the pressure on CI nodes a bit. Note that we're still
running the full compatibility matrix on macOS as part of the daily
build.

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-18 17:03:24 +02:00
Moritz Kiefer
b1738c7202
Switch Scala default to 2.13 (#9699)
changelog_begin
changelog_end
2021-05-17 15:04:53 +00:00
Moritz Kiefer
fb0551bdfd
Fix blackduck flag names (#9695)
changelog_begin
changelog_end
2021-05-16 16:45:31 +02:00
Gary Verhaegen
b58d30b9e3
skip Windows tests on release (#9656)
Caching doesn't seem to work very well here. On a release, we build an
old commit, which has already been tested twice (once as a commit on
`main`, once as part of the release PR).

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-12 14:23:05 +02:00
Gary Verhaegen
40f69ee5af
fix release PR notifications (#9658)
In the automated process, the Azure build is triggered on the branch
directly, which will be named `auto-release-pr-$(date -I)`. But if a
manual change needs to be made, and people subsequently use the `/azp
run` feature of Azure, the build then runs for the PR, which means it
actually runs on the merge commit of the branch and `main`, not on the
branch itself. In that case, the branch that we run the build on is
called `merge` and is thus not starting with `auto-release-pr-`.

This change should get us the notification back on manual PR builds too.

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-12 14:22:38 +02:00
Andreas Herrmann
0acc4f115c
Patch old Bazel derivation (#9622)
The patch that it downloads via Nix was taken from a GH PR instead of a
commit such that the hash is not fully stable. This adds a patch to
download the relevant patch directly from a GitHub commit.

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-05-07 18:23:47 +02:00
Robin Krom
84359ffa73
nested record updates (#9505)
* record dot updates: update to new ghc-lib-parser

This updates the ghc-lib-parser library featuring record dot updates and
adds tests for the new feature.

CHANGELOG_BEGIN
CHANGELOG_END

* update snapshot after pin on windows

* added a test for error locations

* nested record puns test

* update ghc commit

* update of stack dependencies (linux)

* update stack snapshot(windows)
2021-05-03 16:27:01 +02:00
Moritz Kiefer
4aca199bbd
Upgrade to Scala 2.13.5 (#9528)
changelog_begin
changelog_end
2021-04-29 13:05:57 +00:00
Stefano Baghino
e1e878a597
Simplify opt-in/out of Oracle when building (#9515)
* Simplify opt-in/out of Oracle when building

- defines Oracle-related environment variables as part of dev-env so that it's the same across CI and dev
- adds a Bazel 'oracle' configuration so that --config=oracle pulls all required environment

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/9515#discussion_r621180994

* Also use `$ORACLE_USERNAME` instead of `system` in the `sqlplus` connection string

* Address https://github.com/digital-asset/daml/pull/9515#discussion_r621186459
2021-04-28 06:10:43 +02:00
Samir Talwar
8cd3658b14
Switch to an environment variable for enabling Oracle tests. (#9511)
* participant-integration-api: Build Oracle tests, but don't run them.

CHANGELOG_BEGIN
CHANGELOG_END

* triggers: Switch to an environment variable for enabling Oracle tests.

* http-json: Switch to an environment variable for enabling Oracle tests.

* Disable running Oracle tests by default, not building them.

* triggers/service: Remove unused test dependencies.
2021-04-27 11:50:39 +00:00
Gary Verhaegen
e39c20ecbc
update GPG public key (#9488)
The existing public key is set to expire in May, so we've changed it.

Note: this _should_ require no other change as the private key is
unchanged (i.e. the new public key can be used to verify old
signatures), but my understanding of GPG is somewhat limited so 🤷.

CHANGELOG_BEGIN
CHANGELOG_END
2021-04-27 13:40:02 +02:00
Brian Healey
ca66061b28
Ledger Api Oracle support (#9277)
* WIP : first cut at changed schema files for oracle
Define Oracle as DbType and handle necessary case match switches for it
recomputed shas for oracle migration scripts
Oracle fixtures
get things compiling
Able to connect to Oracle
Working through getting schema definitions functional with Oracle
runnable schema definitions only for active tables on oracle
delete commented lines in schema scripts
use oracle enterprise
correct inadvertently changed postgres schemas
WIP - latest oracle-ificiation
passing upload packages spec
add additional test for package upload entry read
correct typo in oracle database spec name
use BLOB for parties ledger_offset
package_entries use hex version of offset for range queries
reformat and update shas for sql scripts
binary numeric implicit conversion for oracle
correct duplicate exception text for oracle
parties test passing on oracle
add additional column to hold hex offset for party_entries
party_entries working for all dbs
scalafmt
Configuration ledger_offset should be BLOB
update sha of oracle sql files
enable passing tests in order
remove misleading null comments
define additional custom VARRAY types
add participant-integration-api-oracle tests to linux-oracle job
Add TODO for places where we need to deal with separate implicit imports for Oracle vs Postgres/H2
oracle implicit conversions for custom arrays and other problematic types

Do not override default debug level for all tests in participant-integration-api

CHANGELOG_BEGIN
Ledger API and Indexer Oracle Support
CHANGELOG_END

passing TransactionWriterSpec
passing JdbcLedgerDaoCompletionsSpec JdbcLedgerDaoDivulgenceSpec
passing JdbcLedgerDaoContractsSpec
All Oracle tests passing apart from one post-commit validation test

* Remove JdbcLedgerDaoValidatedOracleSpec as this is only relevant for classic postgres-backed sandbox

* rebase to master -- offsets are now varchar2 rather than blob

* remove use of DBMS_LOB operations

* remove all greater than/less than variants for DBMS_LOB

* revert postgres files that need not be touched

* code review feedback : avoid code duplication

* avoid indirection in type names for oracle arrays

* code review: HexString implicit conversions are not needed

* code review: Oracle case is not yet implemented for appendonlydao

* code review: Oracle case is not yet implemented for appendonlydao (cleanup import)

* code review: revert files that should not be touched

* address code review feedback: db specific imports for command completion become part of queries

* code review: perform db-specific reserved word escape to avoid case match

* code review: remove all dbms_lob comparison operations

* use simpler insert into with ignore dupes hint for oracle

* code review: avoid db specific match case in events range, use db specific limitClause

* code review: restore group by on Binary and Array fields for H2 and Postgres, disable for Oracle

* code review: restore group by on Binary and Array fields for H2 and Postgres, disable for Oracle

* code review: restore group by on binary and array fields for non-oracle dbs, honour the calculation of limit size from QueryParty.ByArith

* code review: honour the calculation of limit size from QueryParty.ByArith

* code review: drop user after oracle test

* code review: remove drop user as it throws errors due to dangling sessions

* code review: revert incorrectly changed postgres schema files

* code review: clean up TODOs

* Remove // before hostname for consistency with other oracle connection strings

* code review: unambiguously scope table column referenced in select and where queries

* code review: correct duplicate table alias
2021-04-26 18:34:17 -04:00
Gary Verhaegen
8a6cfacbff
more robust macOS cleanup (#9456)
We've recently seen a few cases where the macOS nodes ended up not
having the cache partition mounted. So far this has only happened on
semi-broken nodes (guest VM still up and running but host unable to
connect to it), so I haven't been able to actually poke at a broken
machine, but I believe this should allow a machine in such a state to
recover.

While we haven't observed a similar issue on Linux nodes (as far as I'm
aware), I have made similar changes there to keep both scripts in sync.

CHANGELOG_BEGIN
CHANGELOG_END
2021-04-21 12:10:47 +02:00
Stefano Baghino
a551b6f874
Don't repeat the Oracle password in ci/build.yml (#9419)
Got it from https://stackoverflow.com/a/13802438

Incidentally, to write this commit message containing a single quote, I had to escape it with this https://stackoverflow.com/a/1250279

I guess today is "bring-your-quotation-marks-to-work day".

changelog_begin
changelog_end
2021-04-15 12:49:47 +02:00
Gary Verhaegen
2662163a23
cleanup: kill more stuff (#9411)
In this case we had Java and Daml processes running.

May be worth moving this to the reset-caches scripts later on.

changelog_begin
changelog_end
2021-04-14 15:15:03 +02:00
Moritz Kiefer
7890381cac
Publish EE JSON API to artifactory (#9392)
changelog_begin
changelog_end
2021-04-13 16:12:18 +02:00
Moritz Kiefer
dc4b9e5968
Publish trigger service EE fat JARs (#9363)
* Publish trigger service EE fat JARs

changelog_begin
changelog_end

* Include ee in artifact name

changelog_begin
changelog_end
2021-04-09 13:24:18 +02:00
Gary Verhaegen
631db446f0
try to fix non-repudiation pushing (#9323)
I've seen reports of Artifactory returning 409 when it detects an
invalid POM file, which would map cleanly to our observed behaviour (as
other files do seem to upload fine). I'm not a POM expert so not
entirely sure how to check the actual files, but I do see one error in
the existing, commented code: the path is not a valid Maven repository
path. It should be `groupid/artifactid/version`, i.e. it is currently
missing the `artifactid` bit. So I'd like to try adding that.

I don't know how to test this without making a release, so my plan is to
make a release once this is merged. Open to suggestion on faster ways to
test this.

CHANGELOG_BEGIN
CHANGELOG_END
2021-04-06 14:32:28 +02:00
Moritz Kiefer
f7b2f14294
Add Oracle support in the trigger service (#9286)
* Add Oracle support in the trigger service

This PR migrates the ddl & queries and adds tests for this. It does
not yet expose this to users. I’ll handle that in a separate PR.

changelog_begin
changelog_end

* use getOrElse

changelog_begin
changelog_end
2021-03-31 18:39:08 +02:00
Stefano Baghino
4b807899ca
Do not publish non-repudiation client JARs to Artifactory (#9312)
So far, every attempt failed with a 409 Conflict.
We'll try again next week.

changelog_begin
changelog_end
2021-03-31 16:02:21 +00:00
Stefano Baghino
cbd76c3465
Publish non-repudiation-core to Artifactory as well (#9305)
changelog_begin
changelog_end
2021-03-31 11:23:24 +00:00
Moritz Kiefer
e3ef08cc10
Avoid base64 encoding of cache suffixes (#9303)
base64 includes / which is at the very least pretty confusing and also
broke our cache cleanup which assumes that the cache suffix takes up
one directory. Afaict, we are not length-restricted in gcp paths so we
can just use the hex digits we get from md5

changelog_begin
changelog_end
2021-03-31 11:30:46 +02:00
Moritz Kiefer
29a6813465
Fix location of non-repudiation artifact (#9294)
This should end up in artifactory not github.

changelog_begin
changelog_end
2021-03-31 07:45:33 +02:00
Moritz Kiefer
eef158d6e8
Fix build command for non-repudiation-client (#9290)
The first attempt failed because the src jar wasn’t there. Afaict, the
path is fine but we didn’t build it.

changelog_begin
changelog_end
2021-03-30 18:36:34 +00:00
Stefano Baghino
4f0c72b0d8
Publish non-repudiation client libraries to Artifactory (#9282)
* Publish non-repudiation client libraries to Artifactory

changelog_begin
changelog_end

Closes #9097

* Address https://github.com/digital-asset/daml/pull/9282#discussion_r603875118
2021-03-30 12:18:07 +02:00
Gary Verhaegen
45c4ba2230
macos cache cleaning (#9245)
This is adapting the same approach as #9137 to the macOS machines. The
setup is very similar, except macOS apparently doesn't require any kind
of `sudo` access in the process.

The main reason for the change here is that while `~/.bazel-cache` is
reasonably fast to clean, cleaning just that has finally caught up to us
with a recent cleanup step that proudly claimed:

```
before: 638Mi free
after: 1.2Gi free
```

So we do need to start cleaning the other one after all.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-30 02:46:05 +02:00
Gary Verhaegen
cbba75a2c0
oracle: switch to entreprise (#9254)
* oracle: switch to entreprise

CHANGELOG_BEGIN
CHANGELOG_END

* --network host

changelog_begin
changelog_end

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-03-29 16:56:30 +02:00
Gary Verhaegen
205763e11a
correct notifs for new linux jobs (#9255)
Without this we sometimes get success notifications on Slack for failed
jobs.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-26 15:26:24 +00:00
Gary Verhaegen
3a91cbccba
get oracle image from Docker Hub (#9252)
CHANGELOG_BEGIN
CHANGELOG_END
2021-03-26 14:43:04 +01:00
Gary Verhaegen
839c97cff9
fix dev-env in clean-up (#9244)
CHANGELOG_BEGIN
CHANGELOG_END
2021-03-25 15:48:58 +00:00
Sofia Faro
f670daec6f
Add try-catch syntax. (#9203)
* Add try-catch syntax.

changelog_begin
changelog_end

* ++

* +++

* ++[>++<-]>

* ++[>++<-]>+

* ++[>+++<-]>

* ++[>+++<-]>+

* ++[>++++<-]>

* Use try-catch in ExceptionSyntax test

* add a syntax test for try catch

* +++[>+++<-]>

* +++[>+++<-]>+

* enable test4

* typo

* add a test with multiple patterns and guards

* final patch sha

* update stack snapshot (unix)

* update stack snapshot (win)
2021-03-25 13:36:09 +00:00
Stefano Baghino
b6f7b78990
Publish non-repudiation app fat JARs to Artifactory (#9238)
* Publish non-repudiation app fat JARs to Artifactory

changelog_begin
changelog_end

Contributes to #9097

* Address https://github.com/digital-asset/daml/pull/9238#discussion_r601264444
2021-03-25 10:48:49 +00:00
Stefano Baghino
3fa05804bf
Make non-repudiation integration exclusive to enterprise edition (#9219)
* Make non-repudiation integration exclusive to enterprise edition

Closes #9182

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/9219#discussion_r599578614

* Address https://github.com/digital-asset/daml/pull/9219#pullrequestreview-618605511

* Add aliases for CE test targets, thanks to @S11001001 for the feedback

* Uh, turns out you need a main...

* Address https://github.com/digital-asset/daml/pull/9219#discussion_r599717526

* Add missing dependency for Oracle
2021-03-24 20:25:47 +01:00
Gary Verhaegen
8f1fb6d0a1
fix slack notifs (#9234)
Revert change from #9166.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-24 18:38:41 +00:00
Gary Verhaegen
7c4b32aee9
use coreutils date on macos (#9228)
macOS uses BSD date by default which has slightly different options.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-24 13:35:02 +01:00
Gary Verhaegen
b0c0306696
fix release: skipping compat should not fail (#9227)
In #9169, I changed the compat jobs to not run on releases.
Unfortunately I forgot to update the `collect_build_data` job to know
about that. Hopefully after this has been merged we'll be able to rerun
\#9221.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-24 11:18:58 +00:00
Gary Verhaegen
90aa84193f
tuesday notif: newlines (#9213)
CHANGELOG_BEGIN
CHANGELOG_END
2021-03-23 17:03:22 +01:00
Gary Verhaegen
ad94c47605
ci/docker: sync with hub (#9184)
It looks like the last update was not propagated.

CHANGELOG_BEGIN
ChANGELOG_END
2021-03-18 16:01:58 +00:00
Gary Verhaegen
1ea00976f0
skip compat tests on release (#9169)
As requested by @coceature.

Note: skipping the ts_lib job is enough to skip all compat tests because
they all depend on it, and in the Azure model if one of your
dependencies was skipped you get skipped too.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-17 13:33:20 +00:00
Sofia Faro
c26c349c8b
Generate exception instances from syntax. (#9140)
* Generate exception instances from syntax.

changelog_begin
changelog_end

* II

* III

* VII

* update ghc patch and add test

* VIII

* IX

* Remove DatatypeContexts

* X

* update stack snapshot

* don't need datatypecontexts warning anymore

* X-2

* XII

* XIII
2021-03-17 12:19:22 +00:00
Gary Verhaegen
17f8bcd37d
fix release job (#9166)
Three issues here:

1. The release job runs on an Azure-hosted agent, so it doesn't have the
   `reset_caches.sh` script (and doesn't need it).
2. The `bash-lib` step should not run if the current job has already
   failed.
3. The `skip-github` jobs should also not run if the job  has failed.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-17 09:46:02 +00:00
Gary Verhaegen
691edeacf2
ci: fix cache cleanup (#9137)
This is a continuation of #8595 and #8599. I somehow had missed that
`/etc/fstab` can be used to tell `mount` to let users mount some
filesystems with preset options.

This is using the full history of `mount` hardening so should be safe
enough. The option `user` in `/etc/fstab` automatically disables any kind
of `setuid` feature on the mounted filesystem, which is the main attack
vector I know of.

This works flawlessly on my local VM, so hopefully this time's the
charm. (It also happens to be my third PR specifically targeted on this
issue, so, who knows, it may even work.)

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-16 17:51:38 +01:00
Moritz Kiefer
a3d6c9faf8
Fix artifactory upload path (#9115)
changelog_begin
changelog_end
2021-03-12 11:39:50 +01:00
Gary Verhaegen
c556db48ed
ci/clean-up: remove poweroff (#9108)
It's not working and I can't make it work (see #9096), so I'd rather
just remove it.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-12 10:48:34 +01:00
Gary Verhaegen
764f9af655
ci/build: fix conditions (#9109)
In Azure Yaml, by defualt, a step runs only if the previous step was
successful. However, that default _disappears if the step has an
explicit condition_. I believe we have a number of conditional steps
that have been written without that intention, and this is thus
restoring what I believe to be the original intention, i.e. _adding_ an
additional condition rather than _replacing_ the default one.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-12 07:13:26 +01:00
Moritz Kiefer
9a7e246549
Fix escaping in publish-artifactory (#9107)
changelog_begin
changelog_end
2021-03-12 01:29:55 +01:00
Gary Verhaegen
e2730d322d
ci/windows: add missing mkdir (#9103)
CHANGELOG_BEGIN
CHANGELOG_END
2021-03-11 18:16:19 +00:00
Moritz Kiefer
5cb857d902
Release EE SDK tarballs and installer (#9086)
* Release EE SDK tarballs and installer

As before, no way of testing this. I’ll do a snapshot afterwards.

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* Rename EE artifacts

changelog_begin
changelog_end
2021-03-11 14:49:48 +01:00