@nickchapman-da is taking care of 1.13.0-snapshot.20210428.6782.0.e1e878a5 (#9518), so they get pushed back to the end of the line.
Please do not merge this before #9518.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
* Add unit tests for Telemetry
CHANGELOG_BEGIN
CHANGELOG_END
* Make Tracers injectable, improve the TelemetryContextSpec, add a metrics-test-lib package
* Support rollback nodes in KeyValueCommitting.submissionOutputs
As discussed on Slack overapproximating and thereby not treating nodes
under a rollback special seems like a good first step and matches what
we do for transient contracts.
changelog_begin
changelog_end
* Newlines
changelog_begin
changelog_end
* better variable names
changelog_begin
changelog_end
* Outline test function
changelog_begin
changelog_end
* Apply suggestions from code review
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/participant/state/kvutils/KeyValueCommitting.scala
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Factor out result computation
changelog_begin
changelog_end
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Missing input state is the fault of the participant, not the ledger.
Logging it as a warning, with a stack trace, makes it look like the
ledger itself failed, when actually the participant failed to provide
the correct keys (potentially due to data corruption, but it could also
be malicious).
This drops the stack trace to make the error less scary. We still log at
WARN level because this sort of thing shouldn't happen very often, and
at the very least, the participant operator should be informed, and the
participant should be refreshed with the correct data somehow.
CHANGELOG_BEGIN
CHANGELOG_END
* add oracle option to http-json-perf-binary-ee
* add oracle path to perf Main's JDBC bracket
* adapt to availableJdbcDriverNames; missing deps
* add changelog
CHANGELOG_BEGIN
- [JSON-API Perf] ``--query-store-index=postgres`` must be passed
to select PostgreSQL query store performance testing; ``true``
and ``yes`` are no longer supported.
See `issue #9492 <https://github.com/digital-asset/daml/pull/9492>`__.
CHANGELOG_END
* participant-integration-api: Build Oracle tests, but don't run them.
CHANGELOG_BEGIN
CHANGELOG_END
* triggers: Switch to an environment variable for enabling Oracle tests.
* http-json: Switch to an environment variable for enabling Oracle tests.
* Disable running Oracle tests by default, not building them.
* triggers/service: Remove unused test dependencies.
The existing public key is set to expire in May, so we've changed it.
Note: this _should_ require no other change as the private key is
unchanged (i.e. the new public key can be used to verify old
signatures), but my understanding of GPG is somewhat limited so 🤷.
CHANGELOG_BEGIN
CHANGELOG_END
* WIP : first cut at changed schema files for oracle
Define Oracle as DbType and handle necessary case match switches for it
recomputed shas for oracle migration scripts
Oracle fixtures
get things compiling
Able to connect to Oracle
Working through getting schema definitions functional with Oracle
runnable schema definitions only for active tables on oracle
delete commented lines in schema scripts
use oracle enterprise
correct inadvertently changed postgres schemas
WIP - latest oracle-ificiation
passing upload packages spec
add additional test for package upload entry read
correct typo in oracle database spec name
use BLOB for parties ledger_offset
package_entries use hex version of offset for range queries
reformat and update shas for sql scripts
binary numeric implicit conversion for oracle
correct duplicate exception text for oracle
parties test passing on oracle
add additional column to hold hex offset for party_entries
party_entries working for all dbs
scalafmt
Configuration ledger_offset should be BLOB
update sha of oracle sql files
enable passing tests in order
remove misleading null comments
define additional custom VARRAY types
add participant-integration-api-oracle tests to linux-oracle job
Add TODO for places where we need to deal with separate implicit imports for Oracle vs Postgres/H2
oracle implicit conversions for custom arrays and other problematic types
Do not override default debug level for all tests in participant-integration-api
CHANGELOG_BEGIN
Ledger API and Indexer Oracle Support
CHANGELOG_END
passing TransactionWriterSpec
passing JdbcLedgerDaoCompletionsSpec JdbcLedgerDaoDivulgenceSpec
passing JdbcLedgerDaoContractsSpec
All Oracle tests passing apart from one post-commit validation test
* Remove JdbcLedgerDaoValidatedOracleSpec as this is only relevant for classic postgres-backed sandbox
* rebase to master -- offsets are now varchar2 rather than blob
* remove use of DBMS_LOB operations
* remove all greater than/less than variants for DBMS_LOB
* revert postgres files that need not be touched
* code review feedback : avoid code duplication
* avoid indirection in type names for oracle arrays
* code review: HexString implicit conversions are not needed
* code review: Oracle case is not yet implemented for appendonlydao
* code review: Oracle case is not yet implemented for appendonlydao (cleanup import)
* code review: revert files that should not be touched
* address code review feedback: db specific imports for command completion become part of queries
* code review: perform db-specific reserved word escape to avoid case match
* code review: remove all dbms_lob comparison operations
* use simpler insert into with ignore dupes hint for oracle
* code review: avoid db specific match case in events range, use db specific limitClause
* code review: restore group by on Binary and Array fields for H2 and Postgres, disable for Oracle
* code review: restore group by on Binary and Array fields for H2 and Postgres, disable for Oracle
* code review: restore group by on binary and array fields for non-oracle dbs, honour the calculation of limit size from QueryParty.ByArith
* code review: honour the calculation of limit size from QueryParty.ByArith
* code review: drop user after oracle test
* code review: remove drop user as it throws errors due to dangling sessions
* code review: revert incorrectly changed postgres schema files
* code review: clean up TODOs
* Remove // before hostname for consistency with other oracle connection strings
* code review: unambiguously scope table column referenced in select and where queries
* code review: correct duplicate table alias
* Switch from `@silent` to `@nowarn`.
This annotation is native to Scala 2.12.13+ and 2.13.2+. It replaces
most usages of `@silent`.
I had to get creative about a couple of use cases that didn't work.
Specifically:
1. Suppressing deprecation warnings works, but Scala 2.12 erroneously
complains that the `@nowarn` is unnecessary. I had to suppress
this warning too with `-Ywarn-unused:-nowarn`.
2. I can't seem to suppress the warning, "The outer reference in this
type test cannot be checked at run time." Instead, I have
refactored the code to remove the warning.
We still need to use the silencer plugin to suppress some warnings about
unused imports (because of compatibility between Scala 2.12 and 2.13),
but this means we no longer need the library, and therefore it is not a
transitive dependency that downstream consumers need to worry about.
CHANGELOG_BEGIN
CHANGELOG_END
* Add some comments around `@nowarn` support.
* language-support/scala: Fix a warning suppression.
* Revert to the default warnings.
Compatibility was complaining.
* Add a helper for getting all inactive contracts to Transaction
This comes up in a few places and rather than inlining the
implementation everywhere, I’d rather rely on this.
changelog_begin
changelog_end
* 2.13
changelog_begin
changelog_end
* compatibility: Use the same Scala version as the root.
It's confusing to get out of sync. `compatibility` was using Scala
2.12.12 while the root was on 2.12.13.
This brings them in sync to 2.12.13.
CHANGELOG_BEGIN
CHANGELOG_END
* Factor out Scala artifacts into scala_version.bzl.
* Bazel: Explicitly specify artifact hashes for Scala 2.13.
Not strictly necessary, as we _happen_ to depend on the same point
release as the Bazel Scala plugin, but this is incidental and we don't
want to bank on it.
CHANGELOG_BEGIN
- [Scala bindings] ``InsertOrdMap`` is deprecated; it was never
needed for the release version of ``GenMap``. Use an ordinary
``Map``, or ``Primitive.GenMap`` for Scala codegen, instead.
CHANGELOG_END
* Change the TelemetryContext to use SpanAttributes instead of raw AttributeKeys
and remove an unused object along with its dependencies
CHANGELOG_BEGIN
CHANGELOG_END
* Change the Tracer name and the instrumentation name from participant to com.daml.telemetry
* Support rollback nodes in kvutils ContractKeysValidation
Unfortunately, I couldn’t find any tests to update for this. If
someone can point me to something, I’ll happily add some rollback
tests.
changelog_begin
changelog_end
* Fix comment formatting
changelog_begin
changelog_end
* Add some tests an fix a bug
changelog_begin
changelog_end
* Update ledger/participant-state/kvutils/src/test/suite/scala/com/daml/ledger/participant/state/kvutils/committer/transaction/TransactionCommitterSpec.scala
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Name test cases
changelog_begin
changelog_end
* s/mykey/conflictingKey/
changelog_begin
changelog_end
* Fix missing $
changelog_begin
changelog_end
* Add typedefs
changelog_begin
changelog_end
* Add a testcase for rolled back archive
changelog_begin
changelog_end
* Add a type synonym
changelog_begin
changelog_end
* Add test for rollback of global archive
changelog_begin
changelog_end
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Fix validation of rollback nodes
To my own embarassment there is no test in this PR. We don’t have any
direct tests for validation atm. We absolutely need some and I already
discussed with Remy that this is something we should focus on once
exceptions are done but for now I’ll stick to being sad.
changelog_begin
changelog_end
* Fix bug
changelog_begin
changelog_end
* Set supported jdbc driver names at compile time
This is mainly to unblock the work on Oracle support in the Ledger API
but I think it’s a sensible thing in general. For the Ledger API,
moving the dependency to the top-level is apparently rather
tricky. Because the SDK bundles everything into a single megajar,
Sandbox depending on the oracle library does also result in the JSON
API and the trigger service will also have the oracle library in scope
and will support Oracle in CE which they should not.
This PR simply hardcodes the list of supported drivers to address
that. Not pretty but does the job.
changelog_begin
changelog_end
* format
* Address review comments
changelog_begin
changelog_end
* Add telemetry classes from the oem integration kit and use it for command submissions
* Change submitTransaction to submitTransactionWithTelemetry and add a deprecation
* Fix tests
* Revert "Change submitTransaction to submitTransactionWithTelemetry and add a deprecation"
CHANGELOG_BEGIN
- [Integration Kit] TelemetryContext has been introduced to the WriteService.submitTransaction method to support distributed tracing
CHANGELOG_END
Not quite sure what is going wrong and if this is the issue but I’m
failing to reproduce it reliably to test this so bumping and seeing if
we still hit it on `main` seems like the easiest option.
changelog_begin
changelog_end
* Check visibility for by-key operation of local contracts
fixes#9454
I tried out two approaches for this:
1. The one here where we add a new callback. This has the advantage
that the engine remains oblivious to visibility checks. They are all
done outside and the engine doesn’t even know about the reading
parties.
2. Make the engine aware of the reading parties. A start of that is in
#9458.
Both work in principle but I ended up going for 1 in the end. Doing
half of the visibility checks outside the engine and half inside just
seems worse than the current state.
changelog_begin
- [Daml Engine] Fix a bug where it was possible to
fetch/lookup/exercise a local contract by key even if the reading parties
are not stakeholders. See #9454 for details.
changelog_end
* Disable new test on Canton
changelog_begin
changelog_end
* Exclude from compat tests
changelog_begin
changelog_end
* s/LocalLookup/LocalFetch/
changelog_begin
changelog_end
* Address review
changelog_begin
changelog_end
* Extend Transaction proto format for rollback nodes.
- proto format
- encode/decode
- testcase
- always `allowRollback` from scalagen testing; and so remove control flag
- two more `8020` TODOs, for other code which matches `node.getNodeTypeCase` and sees `ROLLBACK`
changelog_begin
changelog_end
* add comments about version of rollback nodes in transaction.proto
* check rollback node version during encode & encode
* factor out code to decode children for exercise & rollback node
* test decode check for unexpected rollback node in old version
* add the MapOps argument to Compat.MapLike
* use immutable.Map newtype for GenMap instead of InsertOrdMap
- bindings tests and sample-app tests compile under 2.13 and 2.12
* add changelog
CHANGELOG_BEGIN
- [Scala Codegen] ``GenMap`` no longer requires ``InsertOrdMap``;
ordinary Scala immutable ``Map``s may be effectively used
instead. We still encourage use of the ``Primitive.GenMap``
alias for proper typing, future-proofing, and deprecation
warnings.
See `issue #9449 <https://github.com/digital-asset/daml/pull/9449>`__.
CHANGELOG_END
* implicitly convert from Map; test several aspects of the Scala GenMap API
* test best-effort class preservation
* drop the CBF for 2.12, as with TextMap
* test conversion in a contract construction
* remove unused CanBuildFrom compats
* combine Primitive tests
* reenable 2.13-only tests
* remove InsertOrdMap references in Shrink/Show instances
* remove test that GenMap order is preserved by Scala
* Add setTime flag
changelog_begin
* [Daml export] Enable the ``--set-time`` flag on
``daml ledger export script`` to replicate transaction time stamps.
This only works on ledgers operating in static time mode.
changelog_end
* Dump.scala --> Export.scala
So that the file name and class name match.
This was forgotten in 5e652bb2f8.
* Implement optional setTime actions
* Define Action type encompassing SetTime and Submit
Factor out construction of submit actions interleaved with setTime
actions.
This makes it easier to test the corresponding functionality in
isolation.
* EncodeSetTimeSpec
* ActionsFromTreesSpec
* update docs on daml ledger export
* Enable --set-time in integration test
To ensure that the generated code actually compiles
* Fix date/time encoding
Requires additional imports for DA.Time and DA.Date.
Requires qualified references to DA.Time.time, DA.Date.date, and
DA.Date.MONTH.
* Fix Scala 2.13
* Move setTime parameter to Action.fromTrees
* Move export transaction time docs out of caveats
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This focuses on the semantics rather than the display in Daml Studio
which needs more work (and seems not all that important at this
stage).
This already uncovered a bug which also applies outside of scenarios:
The consumedBy field was not affected by rollbacks which breaks the
mustBeActive check in partial transactions. This PR fixes this by
caching on try and restoring on rollback.
changelog_begin
changelog_end
* [Participant] Store offsets as text instead of bytea
This increases compatibility with other databases, in particular Oracle.
CHANGELOG_BEGIN
CHANGELOG_END
* also change the create_key_hash from bytea to varchar/text
* separate alter column statements for h2
* string key hash for append only schema
* fix migration script copy-pasta
* update comment, recompute hash
* Fixed append-only schema migration
* Fixed migration checksums
Co-authored-by: Kamil Bozek <kamil.bozek@digitalasset.com>
We've recently seen a few cases where the macOS nodes ended up not
having the cache partition mounted. So far this has only happened on
semi-broken nodes (guest VM still up and running but host unable to
connect to it), so I haven't been able to actually poke at a broken
machine, but I believe this should allow a machine in such a state to
recover.
While we haven't observed a similar issue on Linux nodes (as far as I'm
aware), I have made similar changes there to keep both scripts in sync.
CHANGELOG_BEGIN
CHANGELOG_END
* KV: do not use "Effects" part of InputsAndEffects
The PR is based #9429.
changelog_begin
changelog_end
* Factor out helpers
changelog_begin
changelog_end
* Apply suggestions from code review
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* review comments
changelog_begin
changelog_end
* fmt
changelog_begin
changelog_end
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Time lag between contract state events and general dispatcher
* Emit current event sequential id from the indexer.
* Time update registering metric
CHANGELOG_BEGIN
CHANGELOG_END
* A ledger-api-test-tool test case for too large deduplication key failures
* Hashing submitters in the deduplication key
CHANGELOG_BEGIN
- [Integration Kit] - deduplication key will contain hashed submitters instead of concatenated submitters
CHANGELOG_END
* Separate ValueLimitsIT test suite for the ledger-api-test-tool
* Excluded ValueLimitsIT for sandbox and ledger-on-sql conformance tests
* Checking contract creation result in the test case
* Update index on participant_command_completions table
Do not include submitters in the index to avoid issues with index row size limit
* Added a delay for more consistent results
* Fixed submitting a command
* Simplified the test case
* Reverted changing indexes
* Dedicated object for making deduplication keys
* Ported hashing deduplication keys to the appendonlydao
* Removed ValueLimitIT suite.
The suite will be restored when database index fixes for large submitters sets are implemented
* Additional unit test for hashing deduplication keys
* Test for duplicate contracts when querying on behalf of multiple parties
Fixes#9388
changelog_begin
changelog_end
* Optimize imports
* Thanks to @S11001001 for answering the comment
* Re-structure the test following @S11001001's input in https://github.com/digital-asset/daml/pull/9443#discussion_r616083932 -- thanks
* Document daml ledger export script
changelog_begin
* [Daml export] Refer to the "Ledger Export" chapter under the "Early
Access Features" for a description of the new Daml ledger export
command. This is an early access feature.
changelog_end
* Fix reference formatting
* Update docs/source/tools/export.rst
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Mark as alpha
* Clarify issue around archived contracts
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
