In order not to make the appearance that current state of sandbox-on-x can support anything than clean initialization form scratch, and clean initialization from already ingested database.
changelog_begin
changelog_end
* Simple scaffolding for the ledger-api-bench-tool
* Added README
* Added resource management and the LedgerIdentityService
* Changed the default log level to DEBUG
* Added the TransactionService
* Added stream configuration options
* Options for ledger configuration
* Minor improvements
* Refactored packages
* Minor improvement
* CHANGELOG_BEGIN
- [Integration Kit] - Created the ledger-api-bench-tool prototype for benchmarking ledger transaction streaming capabilities
CHANGELOG_END
* Unified endpoint argument with the ledger-api-test-tool + other minor fixes
* Logger as an argument to LogOnlyObserver
* Make error throw a GeneralError.
As well as abort, fail, etc.
changelog_begin
changelog_end
* keep the error message when you have an unhandled error in scenario
* Disable crashing opsem tests for now.
* Update CommandServiceIT regex pattern.
* Put | in wrong place :-|
* forgot to escape "
* Illegal repetition!
This PR includes :
- Adding ApiCommand to distinguish between generic command (that are
accepted by the engine) and command that are accepted by the ledger
API.
- Reimplement Canton's reinterpret method using commands instead of
node.
CHANGELOG_BEGIN
CHANGELOG_END
This change adds support for append-only sandbox-classic as well.
Temporarily enabled tests ensure correctness for now
In parallel-ingestion stabilisation epic further unit tests willbe added (hence keeping the TODO for now)
changelog_begin
changelog_end
* Filter divulgence to an empty set of parties
As @nmarton-da noticed painfully, we currently include divulgence to
an empty set of parties. While this is arguably not wrong it is at
least confusing and useless. The whole point of divulgence is to track
visibility. Divulging to an empty set of parties does not affect
visibility so it is not meaningfully different from no
divulgence. Therefore this PR filters it out and adds a doc comment
that the list of divulgees is always non-empty.
changelog_begin
changelog_end
* Fix tests
changelog_begin
changelog_end
The behavior of LedgerConfigProvider is time-sensitive at startup. The
test was accidentally supplying the serial execution context, which
meant that the initialization process could be blocked by a lack of
available threads, causing the wrong result.
Switching to the materializer's execution context avoids this issue in
testing, and does not impact production logic.
In general, resource owners should not accept execution contexts
implicitly. Looks like I added this one, so everyone else is off the
hook. I will punish myself by ordering a pizza.
CHANGELOG_BEGIN
CHANGELOG_END
This change adds support of append-only schema in sandbox-classic and daml-on-sql ledgers: this is available with feature flag.
The support is PoC grade, it will be stabilized/productionized in the upcoming epic.
Currently enabled CI tests in the respective projects guarding this implementation.
* Introduce SequentialWriteDao for simplified indexing in sandbox-classic
* Use this in appendonly.JdbcLedgerDao to implement necessary methods
* Add support for ledgerEnd query to StorageBackend
* Fix JdbcLedgerDao creation (supporting append-only)
* Add feature flag and wiring for sandbox-classic
* Activate conformance tests with append-only on sandbox-classic
* Add support/ci coverage for daml-on-sql
changelog_begin
changelog_end
Some cached threadpools weren't given names, meaning at runtime there
are a bunch of pool-x-thread-y threads. This makes it hard to understand
which threads are being used for what.
The following pool names were introduced:
append-only indexer: input-mapping-pool, batching-pool
ProgramResource: program-resource-pool
kvutils PackageCommitter: package-preload-executor
CHANGELOG_BEGIN
CHANGELOG_END
Just crashing here would actually be fine since this migration will
never run on a transaction with a rollback node but it’s easy enough
to fix it properly and that raises fewer questions.
changelog_begin
changelog_end
* Address more exception todos
A bit of a kitchen sink PR to address a bunch of the trivial todos
that didn’t seem worth splitting out into separate PRs.
changelog_begin
changelog_end
* Revert changes to TransactionSpec
changelog_begin
changelog_end
As discussed, we don’t want to expose this via serializable values at
least for now (and it’s not exposed on the ledger API anyway) so this
PR drops the type.
changelog_begin
changelog_end
* log all external requests at an info level
CHANGELOG_BEGIN
CHANGELOG_END
* additional logs in the transaction service
CHANGELOG_BEGIN
Log ledger-api client read requests at the info level. Affects following services
- Active Contracts Service
- Command Completion Service
- Ledger Configuration Service
- Ledger Identity Service
- Package Service
- Time Service
- Transaction Service
CHANGELOG_END
* Added a new test suite for testing limit API values - ValueLimitsIT
* Change index on participant_command_completions table
This change fixes issues with commands with large submitters number on sandbox-classic
* Added key_hash column to the ledger *state table
The new column is now the primary key of the table. Its values are hashed values of the 'key' column which allows to mitigate limitations on the index row size
* Backfill key_hash for ledger_state table
* Dynamic state table prefix form the backfill migration
* Removed redundant comments
* Backfill migration for all the db types
* Added missing copyright comment
* Fixed migration order after a rebase
* Added missing checksums for sql migrations
* Temporarily removed copyrights from one of sql migrations
* Removed unnecessary NOT NULL constraint
* Removed submitters from the index participant_command_completion_offset_application_idx in the append-only schema
* Disabled the test for old platforms
* CHANGELOG_BEGIN
- [Integration Kit] - a new test suite ValueLimitsIT for testing edge case values
- [Integration Kit] - modified index on the participant_command_completions table to avoid issuess with a large number of submitters
- [Sandbox] - added the key_hash column to the *state table
CHANGELOG_END
* Disabled concurrent testing for the ValueLimitsIT:VLLargeSubmittersNumberCreateContract test case
* Recomputed migrations checksums for the participant-integration-api
* Increase the sandbox-on-x queue size to 500
The motivation is that running conformance tests failed with RESOURCE_EXHAUSTED error due to 200 limit on the queue.
* Minor improvement
* Inlined key hashing for migrations to avoid external dependencies
* Minor improvement
* Fixed migrations hash
* Add Ledger API test tool tests for exceptions
changelog_begin
changelog_end
* Update daml-lf/language/daml-lf.bzl
Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>
* Address review comments
changelog_begin
changelog_end
* Shuffle around test
changelog_begin
changelog_end
Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>
Motivation of this PR: with the help of this combined method, wiring of the
append-only schema ingestion becomes possible.
* Add proper implementation for dao
* Adapts sandbox-classic usage
changelog_begin
changelog_end
* Precreate Oracle indexer scala migration package
This avoids needing to special case FlywayMigrations to not include
scala migrations which will be more robust when adding the first oracle-based
scala migration.
changelog_begin
changelog_end
* Review feedback - remove FIXME
The intent of this change is to make the first step in this direction
in order to support dpp-336 work of sanbdox-classic integration by
injecting DAO functionality via this interface.
Level of quality is still pre-production, hence TODO comments.
Planned next step: move StorageBackend interface, and implementation
to platform/store, to it's final place.
* Introduce StorageBackend interface
* Decouple event-seq-id assignment logic from storage specific batching
* Pull out of batching step from input mapping, execution too
* Switch to stateless DAO functions
* Switch to DBDispatcher instead of custom JDBC Connection pool
* Introduce/adapt metrics
* Naturally extend configuration
* Move RunningBatch layer to ParallelIndexerFactory
* Remove dead code
changelog_begin
changelog_end
* Support rollback nodes in ActiveStateManager
If some one can point me to existing tests for ActiveStateManager,
I’ll happily extend them I failed to find any.
I did test it against #9400 and it fixes most tests and the failing
ones fail in other parts so at that level it seems to work as
expected.
changelog_begin
changelog_end
* Better comments
changelog_begin
changelog_end
* less Set()
changelog_begin
changelog_end
* switch pattern matching
changelog_begin
changelog_end
* Clarify comment
changelog_begin
changelog_end
* Simplify tracking of archivedIds
changelog_begin
changelog_end
* Document where ActiveLedgerStateManager is used
changelog_begin
changelog_end
* Blow up when using rollback nodes with a mutable ALS
changelog_begin
changelog_end
This means we don't have to provide context explicitly on each log line;
it'll get passed through.
We include extra logging context, extracted from the state at each step.
CHANGELOG_BEGIN
CHANGELOG_END
* Add unit tests for Telemetry
CHANGELOG_BEGIN
CHANGELOG_END
* Make Tracers injectable, improve the TelemetryContextSpec, add a metrics-test-lib package
* Support rollback nodes in KeyValueCommitting.submissionOutputs
As discussed on Slack overapproximating and thereby not treating nodes
under a rollback special seems like a good first step and matches what
we do for transient contracts.
changelog_begin
changelog_end
* Newlines
changelog_begin
changelog_end
* better variable names
changelog_begin
changelog_end
* Outline test function
changelog_begin
changelog_end
* Apply suggestions from code review
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/participant/state/kvutils/KeyValueCommitting.scala
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Factor out result computation
changelog_begin
changelog_end
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Missing input state is the fault of the participant, not the ledger.
Logging it as a warning, with a stack trace, makes it look like the
ledger itself failed, when actually the participant failed to provide
the correct keys (potentially due to data corruption, but it could also
be malicious).
This drops the stack trace to make the error less scary. We still log at
WARN level because this sort of thing shouldn't happen very often, and
at the very least, the participant operator should be informed, and the
participant should be refreshed with the correct data somehow.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Build Oracle tests, but don't run them.
CHANGELOG_BEGIN
CHANGELOG_END
* triggers: Switch to an environment variable for enabling Oracle tests.
* http-json: Switch to an environment variable for enabling Oracle tests.
* Disable running Oracle tests by default, not building them.
* triggers/service: Remove unused test dependencies.
* WIP : first cut at changed schema files for oracle
Define Oracle as DbType and handle necessary case match switches for it
recomputed shas for oracle migration scripts
Oracle fixtures
get things compiling
Able to connect to Oracle
Working through getting schema definitions functional with Oracle
runnable schema definitions only for active tables on oracle
delete commented lines in schema scripts
use oracle enterprise
correct inadvertently changed postgres schemas
WIP - latest oracle-ificiation
passing upload packages spec
add additional test for package upload entry read
correct typo in oracle database spec name
use BLOB for parties ledger_offset
package_entries use hex version of offset for range queries
reformat and update shas for sql scripts
binary numeric implicit conversion for oracle
correct duplicate exception text for oracle
parties test passing on oracle
add additional column to hold hex offset for party_entries
party_entries working for all dbs
scalafmt
Configuration ledger_offset should be BLOB
update sha of oracle sql files
enable passing tests in order
remove misleading null comments
define additional custom VARRAY types
add participant-integration-api-oracle tests to linux-oracle job
Add TODO for places where we need to deal with separate implicit imports for Oracle vs Postgres/H2
oracle implicit conversions for custom arrays and other problematic types
Do not override default debug level for all tests in participant-integration-api
CHANGELOG_BEGIN
Ledger API and Indexer Oracle Support
CHANGELOG_END
passing TransactionWriterSpec
passing JdbcLedgerDaoCompletionsSpec JdbcLedgerDaoDivulgenceSpec
passing JdbcLedgerDaoContractsSpec
All Oracle tests passing apart from one post-commit validation test
* Remove JdbcLedgerDaoValidatedOracleSpec as this is only relevant for classic postgres-backed sandbox
* rebase to master -- offsets are now varchar2 rather than blob
* remove use of DBMS_LOB operations
* remove all greater than/less than variants for DBMS_LOB
* revert postgres files that need not be touched
* code review feedback : avoid code duplication
* avoid indirection in type names for oracle arrays
* code review: HexString implicit conversions are not needed
* code review: Oracle case is not yet implemented for appendonlydao
* code review: Oracle case is not yet implemented for appendonlydao (cleanup import)
* code review: revert files that should not be touched
* address code review feedback: db specific imports for command completion become part of queries
* code review: perform db-specific reserved word escape to avoid case match
* code review: remove all dbms_lob comparison operations
* use simpler insert into with ignore dupes hint for oracle
* code review: avoid db specific match case in events range, use db specific limitClause
* code review: restore group by on Binary and Array fields for H2 and Postgres, disable for Oracle
* code review: restore group by on Binary and Array fields for H2 and Postgres, disable for Oracle
* code review: restore group by on binary and array fields for non-oracle dbs, honour the calculation of limit size from QueryParty.ByArith
* code review: honour the calculation of limit size from QueryParty.ByArith
* code review: drop user after oracle test
* code review: remove drop user as it throws errors due to dangling sessions
* code review: revert incorrectly changed postgres schema files
* code review: clean up TODOs
* Remove // before hostname for consistency with other oracle connection strings
* code review: unambiguously scope table column referenced in select and where queries
* code review: correct duplicate table alias
* Switch from `@silent` to `@nowarn`.
This annotation is native to Scala 2.12.13+ and 2.13.2+. It replaces
most usages of `@silent`.
I had to get creative about a couple of use cases that didn't work.
Specifically:
1. Suppressing deprecation warnings works, but Scala 2.12 erroneously
complains that the `@nowarn` is unnecessary. I had to suppress
this warning too with `-Ywarn-unused:-nowarn`.
2. I can't seem to suppress the warning, "The outer reference in this
type test cannot be checked at run time." Instead, I have
refactored the code to remove the warning.
We still need to use the silencer plugin to suppress some warnings about
unused imports (because of compatibility between Scala 2.12 and 2.13),
but this means we no longer need the library, and therefore it is not a
transitive dependency that downstream consumers need to worry about.
CHANGELOG_BEGIN
CHANGELOG_END
* Add some comments around `@nowarn` support.
* language-support/scala: Fix a warning suppression.
* Revert to the default warnings.
Compatibility was complaining.
* Change the TelemetryContext to use SpanAttributes instead of raw AttributeKeys
and remove an unused object along with its dependencies
CHANGELOG_BEGIN
CHANGELOG_END
* Change the Tracer name and the instrumentation name from participant to com.daml.telemetry
