* LF: Simplify archive reader.
- decouple Reader and Decoder
- introduce case class to handle hash, proto payload, and version
CHANGELOG_BEGIN
CHANGELOG_END
* Address Moritz' review
* cosmetic
The debug logs don’t seem all that useful and we don’t have them for
other sources, e.g., TransactionSource. They also come with error logs
for failed sources which again is inconsistent with other sources,
noisy and confusing.
changelog_begin
changelog_end
* [In-memory fan-out] Fix execution contexts in ReadOnlySqlLedgerWithMutableCache
* servicesExecutionContext is used in all non-resource components
* Remove widespread use of resourceContext
CHANGELOG_BEGIN
CHANGELOG_END
* [In-memory fan-out] TransactionLogUpdatesConversions optimization
* Lazily convert to transaction trees in TransactionLogUpdatesConversions.ToTransactionTree
* Addressed review comments
* daml-lf/data: Truncate party names in log output, on request.
The party name can grow quite long, so we offer ledger implementors the
opportunity to truncate it in structured log output.
Unfortunately, because we use Logback through the global
`LoggerFactory`, there is no place to inject logging configuration. This
means we also need to use global, mutable state to configure logging
output. I have added a `LoggingConfiguration` class+object in Daml-LF
Data, which may not be the best place, but I can't think of a better
one right now. I suggest we leave it there until it has reason to grow,
at which point we may want to move it.
CHANGELOG_BEGIN
CHANGELOG_END
* logging-entries: Make `ToLoggingValue` mixin-able.
* participant-integration-api: Truncate parties in filters when logging.
* participant-integration-api: Cast to `Party` for logging.
Invalid input should not break the request at this point. No assertions.
* daml-lf/data: Move `Party to LoggingValue` to a new package.
This avoids the transitive dependency issue most of the time.
* daml-lf-data: Move the `Identifier` logging to another package.
Again, reduces the need for transitively depending on _logging-entries_.
I don't find the output from Bazel very helpful most of the time. It
does, however, produce a lot of noise which I have to filter through
when something else goes wrong.
CHANGELOG_BEGIN
CHANGELOG_END
* logging-entries: Split from contextualized-logging.
This allows us to introduce it to Daml-LF without bringing in the
Logback, Logstash, and gRPC dependencies.
CHANGELOG_BEGIN
CHANGELOG_END
* logging-entries: Fix dependencies for 2.12.
* logging-entries: Missed one more Scala 2.12 dependency.
* release: Publish logging-entries.
* participant-integration-api: Remove the subscription ID.
Pretty sure it's not used except for a single log line, which makes it
useless as a correlation ID.
If we want to correlate logs, let's add a correlation ID.
* participant-integration-api: Move transaction requests to trace logging.
Most of the useful information is already in the logging context. We
don't need to log the data structure too.
CHANGELOG_BEGIN
- [Ledger API Server] The amount of data logged in the API transaction
service has been reduced at INFO level. Please enable TRACE logging to
log the request data structures.
CHANGELOG_END
* participant-integration-api: Reorder methods in ApiTransactionService.
* participant-integration-api: Add the word "request" to some log lines.
* participant-integration-api: Add a logging prefix for string offsets.
* ledger-api-domain: `immutable.Set` -> `Set`.
It's an alias.
* participant-integration-api: Log transaction filters on subscription.
* participant-integration-api: Log transaction filters.
Just the parties isn't enough information.
* participant-integration-api: Log the entire transaction request.
Structured, because otherwise it's hard to throw things away later.
* contextualized-logging: Avoid `View` because it's not in Scala 2.12.
* contextualized-logging: Add tests for booleans.
* contextualized-logging: Avoid methods that accept views.
Scala 2.12 really doesn't like me when I do that.
* participant-integration-api: One more try at building with Scala 2.12.
Currently it is impossible to test new features until they land in the
default LF version. This is clearly not great. This PR releases one
Ledger API test tool per LF version (with the LF version being in the
name) to make it easier for Canton and others to test new features.
Note that at least the way things are setup now we won’t go back in
time. We will only publish stable, preview & dev but not older
versions. If needed, we could expand that in the future.
changelog_begin
changelog_end
* A test case for crashing participant in a multi-node env with divulged contracts
* Fixed test naming
CHANGELOG_BEGIN
CHANGELOG_END
* Additional usability check for the disclosed contract
* A newline
* Verify that fetching an archived contract is not possible
* Second test case
* A `ledger-api-test-tool` test case for divulging a contract with key twice on multiple participants [DPP-433] (#9970)
* A test case for divulging a contract with a key twice on multiple participants
CHANGELOG_BEGIN
CHANGELOG_END
* Use autogenerated archive method
* Trigger build
* Increased timeout for TransactionServiceIT:TXInvisibleTransactionTreeByEventId
* contextualized-logging: Automatically convert logging values to strings.
For now, this has almost the same behavior, but it allows us to
customize the output in the future.
The main change is that the log format has gone from:
context: {a=b, x=1, foo=bar}
to:
context: {a: "b", x: "1", foo: "bar"}
* contextualized-logging: Move `writeTo` inside `LoggingValue`.
* contextualized-logging: Allow for more than just strings.
`null`, numbers, and sequences are now correctly logged.
The log format has gone from:
context: {a: "b", x: "1", foo: "bar", parties: "[alice, bob]"}
to:
context: {a: "b", x: 1, foo: "bar", parties: ["alice", "bob"]}
CHANGELOG_BEGIN
- The log output of Daml components has changed so that the structured
part is closer to JSON. This allows us to distinguish and parse
numbers and lists. If you are parsing this log output, you may need to
change your parser.
The log output has changed from:
.. code-block::
context: {a=b, x=1, foo=bar, parties=[alice, bob]}
to:
.. code-block::
context: {a: "b", x: 1, foo: "bar", parties: ["alice", "bob"]}
CHANGELOG_END
* contextualized-logging: Extract the string serializer.
* Use non-string logging where possible.
* contextualized-logging: Split logging values from serialization.
So that callers don't have to know about Jackson.
* contextualized-logging: `SeqView` is `Iterable`. Don't need both.
* contextualized-logging: Make `ToStringToLoggingValue` a `val`.
Co-Authored-By: Stephen Compall <stephen.compall@daml.com>
* contextualized-logging: Add a transient dependency for 2.12 only.
This required more infrastructure than I thought it would.
* kvutils: Make it explicit that we're logging the hashes of archives.
The implicit was found to be a little confusing.
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Oracle compliant append only schema
CHANGELOG_BEGIN
CHANGELOG_END
WIP : oracle on new appendonly schema
* diff to postgres dump, create consolidated view
* diff to postgres dump to ensure all oracle setup is equiv
* recompute sha for changed oracle flyway scripts
* drop old tables to prevent clash on name of new participant_events view
* recompute sha for flyway script
* prelim oracle StorageBackend
* Adds support for special preparedStatement for oracle
* Add support and wires to setObject by default everywhere
* Add the full OracleField suite with TODOs for convenience
* Wires OracleField suite to OracleFieldStrategy
changelog_begin
changelog_end
* enable debug version of oracle driver
* conversion Instant -> Timestamp for oracle
* WIP: primitive println debugging
* Passing PackagesSpec with appendonlyschema on Oracle
Rename size column to siz to avoid reserved word clash, including migration script for postgres
* include sha for new postgres migration script
* add missing copyright header
* cleanup
* passing party spec for appendonly on oracle
* passing configuration spec for appendonly on oracle
* scalafmt
* bazel buildifier reformat
* use db generic FETCH NEXT n ROW ONLY rather than limit for cross db compat
* siz instead of size for packages table on all dbs and schema
* revert enabling oracle jdbc debug
* Support Array[String] -> String conversion (and vice versa) for JSON array
Remove as aliases for tables as this does not work with oracle
Extract submitters clause for all db types
Use append transaction injector for oracle append only spec
* scalafmt
* correct oracle failing active contract spec tests
* wire in JdbcLedgerDaoCompletionsSpec
* remove semi-colons for ending statements that are problematic for oracle driver
* all tests up to divulgence passing for append only on oracle
* all appendonly tests passing on oracle
* remove ignore on fall back to limit-based query with consistent results
* do not change name of size column in packages table for mutable schema for all DBs
* do not change name of size column in packages table for mutable schema for all DBs
* standalone oracle appendonly schema script
regen shas on flyway scripts
revert some cosmetic refactoring in CommonStorageBackend
* Fixes conversion to parties from Oracle-JSON at flatEventWitnessesColumn
* Switches from composit SQLs to single SQLs at prepared statements to accommodate Oracle limitation
* Fixes arrayIntersectionWhereClause by applying patch from mutable Oracle schema integration
* Fixes queries with empty startExclusive Offsets
* First draw adding Oracle conformance test suites to CI
* wire in the oracle conformance tests for CI
* Use cross-db fetch next $n rows only syntax instead of limit syntax that works only for postgres/h2
* rename siz to package_size
* recompute shas
* scalafmt and include sha check for oracle append only flyway script
* correct missing package_size rename
* remove some todos -- correct corrupted V1__Init.sql
* Update ledger/ledger-on-sql/src/test/lib/scala/com/daml/ledger/on/sql/MainWithEphemeralOracleUser.scala
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* correct version number for postgres rename column scripts
* remove unnecessary migration tables for oracle append only
* review feedback: rename createEventFilter as requested, remove todos
* review feedback: case consistency
* review feedback: update todos with issue markers
* review feedback: cleanup
* review feedback: OracleField and OracleSchema cleanup
* Fixing Table generators to use preparedData for convenience
* Placing TODOs for refactorings later
* Renames initial append-only oracle script, for convenience
* Falls back to original behavior as far prepared statements go at couple of queries
Co-authored-by: Marton Nagy <marton.nagy@digitalasset.com>
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* Resolve contract keys conflicts in disclosed contracts (#9948)
* build queries for nullifications of past key
* format code
CHANGELOG_BEGIN
In case a contract key is already present in a past contract in the contract table nullify it. This gets rid of contract keys of previously disclosed contracts. We never discover that disclosed contracts get archived, so we get conflicts on such past keys
CHANGELOG_END
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* correct oracle implemantation (new)
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Add missing indices for index initialization
changelog_begin
changelog_end
* Do not create unnecessary indices
* Create indices in H2
* Fix Oracle migration hash
* Verify all migration hashes
* Add exception tests
changelog_begin
changelog_end
* Add another test
This one tests a transaction that can only be
produced by a privacy-aware ledger.
* Address review comments
* CHANGELOG_BEGIN
Log ledger api validation failures at info level
CHANGELOG_END
* log validation failures in Api*Service family of classes
* address review comments
changelog_begin
- [JSON-API] Timing metrics which measure how long the processing of a command submission request takes on the ledger are now available
changelog_end
changelog_begin
- [JSON-API] The database operations (regardless of in-memory or postgres) contain now metrics for fetching contracts by id or key (seperate metrics foreach)
- [JSON-API] The time for a repsonse payload construction of a request is now also tracked in a metric
changelog_end
* [JSON-API] Add more timing metrics
changelog_begin
- [JSON-API] Timing metrics are now available for party management, package management, command submission and query endpoints.
- [JSON-API] Also added a timing metric for parsing and decoding of incoming json payloads
changelog_end
* Add comments to new metrics
* Split metrics up more & remove obsolete metric
* Split up timers for query endpoints
* Creating OR predicate to reduce length of string literals
CHANGELOG_BEGIN
CHANGELOG_END
* removing unnecessary thrown exception
CHANGELOG_BEGIN
CHANGELOG_END
* switching to camel case for constants
CHANGELOG_BEGIN
CHANGELOG_END
* run format
CHANGELOG_BEGIN
CHANGELOG_END
* remove tolist conversion
CHANGELOG_BEGIN
CHANGELOG_END
* scalafmt
Co-authored-by: Brian Healey <brian.healey@digitalasset.com>
* Preparation: pull up initialisation
* Preparation: introduce partyArrayContext for more flexibility
* Preparation: add append-only-from-scratch feature to FlywayMigrations
* Add squashed Flyway migration for append-only H2
* Add support for H2 StorageBackend
* Extend conformance test suites for H2
* Extend JDBCLedgerDao unit test suites for H2
* Extend sandbox-classic unit test suites for H2
* Fixing typo, some renames based on review
changelog_begin
changelog_end
* Do not terminate transactions/transaction trees streams when no end-offset provided in the ledger-api-bench-tool
CHANGELOG_BEGIN
CHANGELOG_END
* Minor change
* [JSON-API] Concurrent query etc. metrics
changelog_begin
- [JSON-API] The metrics which describe the amount of these concurrent events is now available:
- running http request
- command submissions
- package allocations
- party allocations
changelog_end
* Rename running metrics to throughput ones & add comments on the metrics
* Adjust names of other existing metrics too, to have for the json api a more consistent metrics naming
* truncate party name for 64 chars for multi-valued submitters to avoid excessively long metrics name
CHANGELOG_BEGIN
[metrics] Limit size of multi party metrics to ease consumption
CHANGELOG_END
* Use just first party for dynamic metrics when there are multiple parties involved
* Log context of all updates written to the database
CHANGELOG_BEGIN
For every update in the index db log the full context at the INFO level.
CHANGELOG_END
* one more missing log
This is a preparation for H2 integration: reworking/opening up the ingestion DSL
* Pull more execution logic to PGTable, as preparing for batch insert execution
* Pull out Table from PGTable, add support for deletion (preparation for JDBC batch execution)
* Abstract over Schema (preparation for JDBC batch execution)
* Opening up Field, abstract over table creation (preparation for JDBC batch execution)
* Move code to common (preparation for JDBC batch execution)
* Add generic JDBC batch execution
* Add generic JDBC batch deletion and some more generic Field-s
* Pulling out PGSmallintOptional to support the postgres flavor
* Generalizing idempotentInsert in FieldStrategy
* Capture non empty data logic and comment in a util function
* Seal access to Schema internals
changelog_begin
changelog_end
