This change adds support for append-only sandbox-classic as well.
Temporarily enabled tests ensure correctness for now
In parallel-ingestion stabilisation epic further unit tests willbe added (hence keeping the TODO for now)
changelog_begin
changelog_end
This change adds support of append-only schema in sandbox-classic and daml-on-sql ledgers: this is available with feature flag.
The support is PoC grade, it will be stabilized/productionized in the upcoming epic.
Currently enabled CI tests in the respective projects guarding this implementation.
* Introduce SequentialWriteDao for simplified indexing in sandbox-classic
* Use this in appendonly.JdbcLedgerDao to implement necessary methods
* Add support for ledgerEnd query to StorageBackend
* Fix JdbcLedgerDao creation (supporting append-only)
* Add feature flag and wiring for sandbox-classic
* Activate conformance tests with append-only on sandbox-classic
* Add support/ci coverage for daml-on-sql
changelog_begin
changelog_end
* Add Ledger API test tool tests for exceptions
changelog_begin
changelog_end
* Update daml-lf/language/daml-lf.bzl
Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>
* Address review comments
changelog_begin
changelog_end
* Shuffle around test
changelog_begin
changelog_end
Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>
Motivation of this PR: with the help of this combined method, wiring of the
append-only schema ingestion becomes possible.
* Add proper implementation for dao
* Adapts sandbox-classic usage
changelog_begin
changelog_end
* Support rollback nodes in ActiveStateManager
If some one can point me to existing tests for ActiveStateManager,
I’ll happily extend them I failed to find any.
I did test it against #9400 and it fixes most tests and the failing
ones fail in other parts so at that level it seems to work as
expected.
changelog_begin
changelog_end
* Better comments
changelog_begin
changelog_end
* less Set()
changelog_begin
changelog_end
* switch pattern matching
changelog_begin
changelog_end
* Clarify comment
changelog_begin
changelog_end
* Simplify tracking of archivedIds
changelog_begin
changelog_end
* Document where ActiveLedgerStateManager is used
changelog_begin
changelog_end
* Blow up when using rollback nodes with a mutable ALS
changelog_begin
changelog_end
* WIP : first cut at changed schema files for oracle
Define Oracle as DbType and handle necessary case match switches for it
recomputed shas for oracle migration scripts
Oracle fixtures
get things compiling
Able to connect to Oracle
Working through getting schema definitions functional with Oracle
runnable schema definitions only for active tables on oracle
delete commented lines in schema scripts
use oracle enterprise
correct inadvertently changed postgres schemas
WIP - latest oracle-ificiation
passing upload packages spec
add additional test for package upload entry read
correct typo in oracle database spec name
use BLOB for parties ledger_offset
package_entries use hex version of offset for range queries
reformat and update shas for sql scripts
binary numeric implicit conversion for oracle
correct duplicate exception text for oracle
parties test passing on oracle
add additional column to hold hex offset for party_entries
party_entries working for all dbs
scalafmt
Configuration ledger_offset should be BLOB
update sha of oracle sql files
enable passing tests in order
remove misleading null comments
define additional custom VARRAY types
add participant-integration-api-oracle tests to linux-oracle job
Add TODO for places where we need to deal with separate implicit imports for Oracle vs Postgres/H2
oracle implicit conversions for custom arrays and other problematic types
Do not override default debug level for all tests in participant-integration-api
CHANGELOG_BEGIN
Ledger API and Indexer Oracle Support
CHANGELOG_END
passing TransactionWriterSpec
passing JdbcLedgerDaoCompletionsSpec JdbcLedgerDaoDivulgenceSpec
passing JdbcLedgerDaoContractsSpec
All Oracle tests passing apart from one post-commit validation test
* Remove JdbcLedgerDaoValidatedOracleSpec as this is only relevant for classic postgres-backed sandbox
* rebase to master -- offsets are now varchar2 rather than blob
* remove use of DBMS_LOB operations
* remove all greater than/less than variants for DBMS_LOB
* revert postgres files that need not be touched
* code review feedback : avoid code duplication
* avoid indirection in type names for oracle arrays
* code review: HexString implicit conversions are not needed
* code review: Oracle case is not yet implemented for appendonlydao
* code review: Oracle case is not yet implemented for appendonlydao (cleanup import)
* code review: revert files that should not be touched
* address code review feedback: db specific imports for command completion become part of queries
* code review: perform db-specific reserved word escape to avoid case match
* code review: remove all dbms_lob comparison operations
* use simpler insert into with ignore dupes hint for oracle
* code review: avoid db specific match case in events range, use db specific limitClause
* code review: restore group by on Binary and Array fields for H2 and Postgres, disable for Oracle
* code review: restore group by on Binary and Array fields for H2 and Postgres, disable for Oracle
* code review: restore group by on binary and array fields for non-oracle dbs, honour the calculation of limit size from QueryParty.ByArith
* code review: honour the calculation of limit size from QueryParty.ByArith
* code review: drop user after oracle test
* code review: remove drop user as it throws errors due to dangling sessions
* code review: revert incorrectly changed postgres schema files
* code review: clean up TODOs
* Remove // before hostname for consistency with other oracle connection strings
* code review: unambiguously scope table column referenced in select and where queries
* code review: correct duplicate table alias
* Switch from `@silent` to `@nowarn`.
This annotation is native to Scala 2.12.13+ and 2.13.2+. It replaces
most usages of `@silent`.
I had to get creative about a couple of use cases that didn't work.
Specifically:
1. Suppressing deprecation warnings works, but Scala 2.12 erroneously
complains that the `@nowarn` is unnecessary. I had to suppress
this warning too with `-Ywarn-unused:-nowarn`.
2. I can't seem to suppress the warning, "The outer reference in this
type test cannot be checked at run time." Instead, I have
refactored the code to remove the warning.
We still need to use the silencer plugin to suppress some warnings about
unused imports (because of compatibility between Scala 2.12 and 2.13),
but this means we no longer need the library, and therefore it is not a
transitive dependency that downstream consumers need to worry about.
CHANGELOG_BEGIN
CHANGELOG_END
* Add some comments around `@nowarn` support.
* language-support/scala: Fix a warning suppression.
* Revert to the default warnings.
Compatibility was complaining.
* Add telemetry classes from the oem integration kit and use it for command submissions
* Change submitTransaction to submitTransactionWithTelemetry and add a deprecation
* Fix tests
* Revert "Change submitTransaction to submitTransactionWithTelemetry and add a deprecation"
CHANGELOG_BEGIN
- [Integration Kit] TelemetryContext has been introduced to the WriteService.submitTransaction method to support distributed tracing
CHANGELOG_END
* Check visibility for by-key operation of local contracts
fixes#9454
I tried out two approaches for this:
1. The one here where we add a new callback. This has the advantage
that the engine remains oblivious to visibility checks. They are all
done outside and the engine doesn’t even know about the reading
parties.
2. Make the engine aware of the reading parties. A start of that is in
#9458.
Both work in principle but I ended up going for 1 in the end. Doing
half of the visibility checks outside the engine and half inside just
seems worse than the current state.
changelog_begin
- [Daml Engine] Fix a bug where it was possible to
fetch/lookup/exercise a local contract by key even if the reading parties
are not stakeholders. See #9454 for details.
changelog_end
* Disable new test on Canton
changelog_begin
changelog_end
* Exclude from compat tests
changelog_begin
changelog_end
* s/LocalLookup/LocalFetch/
changelog_begin
changelog_end
* Address review
changelog_begin
changelog_end
* KV: do not use "Effects" part of InputsAndEffects
The PR is based #9429.
changelog_begin
changelog_end
* Factor out helpers
changelog_begin
changelog_end
* Apply suggestions from code review
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* review comments
changelog_begin
changelog_end
* fmt
changelog_begin
changelog_end
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* LedgerDao and ContractsReader interface updates for mutable state cache implementation
* Factored the LfValueSerialization cache out of the ContractStore
* Implemented state lookup methods (at valid_at)
This PR does not contain tests for the new lookup methods.
The tests have been extracted in a separate branch and will be merged
when the DAO integration testing suite will be adapted
for the append-only schema.
CHANGELOG_BEGIN
CHANGELOG_END
* Addressed review comments
* Expose libraries for integration testing purposes
The motivation of these changes is to eliminate manual work and reduce duplication between the SDK and oem-integration-kit repos by reusing the same test fixture for integration testing participant state implementations. Also, the DARs required for running these tests won't need to be manually updated.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix a concurrency issue in integration tests
* Fix Bazel error
* Fix conflict resolution
* Move inline daml-lf to separate dar files
* Add a comment
* Add a missing artifact
* Extract method
* Remove maven tags
* Add a macro for Scala libraries with dar resources
* Improve the macro
* Add missing artifact
* Simplify the tests
* Format signature
* Fix the maven tag
* Add missing copyright headers
* Format bazel files
* Make //ledger/test-common lf version dependent (to avoid jar hell)
* Move da_scala_dar_resources_library to a separate bzl file
* Add missing artifacts
Co-authored-by: Hubert Slojewski <hubert.slojewski@tesco.com>
In replicated Postgres setups, running with SYNCHRONOUS_COMMIT=LOCAL is a
suitable option as it is a trade off to only locally commit synchronously
but not to wait for a network ack.
CHANGELOG_BEGIN
CHANGELOG_END
* Move Daml Profiler to EE version of sandbox/sandbox-classic
This splits Sandbox targets into EE/CE targets and exposes the option
in the EE version. The option still exists in the CE option for now
until we have released EE artifacts to not break users that might know
about it without an alternative.
There is also a small test that makes sure that this actually works
since classpaths are dumb and it didn’t work at first.
changelog_begin
changelog_end
* Fix publish target
changelog_begin
changelog_end
* Publish transitive dep
changelog_begin
changelog_end
* I hate bash
changelog_begin
changelog_end
* Do not require a JWT token for Health and Reflection services
CHANGELOG_BEGIN
- A JWT token is no longer required to call methods of Health and Reflection services
CHANGELOG_END
* Let service's authorizer decide about rejections
* Updated authorization test
* Added integration test for unsecured authorisation test for the Health service
* Added integration test for unsecured authorisation test for the Server Reflection service
* Updated Claims doc comments
* Minor change
* Reduced code duplication with SecuredServiceCallAuthTests and UnsecuredServiceCallAuthTests
* Added copyrights
* Move response status handling logic to Authorizer
CHANGELOG_BEGIN
- [LF] Release LF 1.12. This version reduce the size of transaction
- [Compiler]: Change the default LF output from 1.8 to 1.11.
CHANGELOG_END
* Ledger API test tool: test against legacy and preview version.
+ use dictionary instead of alias to map version keywords to LF version
CHANGELOG_BEGIN
CHANGELOG_END
improve previous generalization from #8695
- use lf version instead keyword (like 'stable', 'latest', 'dev') to
tag actual target. This will allow two keywords to map to the same
versions without doing twice the compilation/test work.
- use alias to map keywords tag target to versioned tag target.
- move package manage dar to test_commong.
CHANGELOG_BEGIN
CHANGELOG_END
* Enable migration only option for startup
CHANGELOG_BEGIN
[Daml Driver for PostgreSQL] Enable migrate only option at startup
CHANGELOG_END
* scalafmt
* Rationalize names of the sql start modes
* no op ledger
* scalafmt
* WIP: choose to run migrationOnly in Main
* revert changes to startup options for sandboxnext
* add sql-start-mode parameter to CLI, validate it
* scalafmt
* Update documentation
* delete some WIP experiments
* removed unused import
* Reorder migrate to properly initialize dao
* use lowercase delimitted options rather than initial caps for sql-start-mode options
* remove redundant parens
* exit after running migration scripts
* return future for flyway migration, only exit on complete, use cached threadpool execution context
* scalafmt
* correct options in docs to lowercase dash delimited
[ledger-api-server] Make database connection pool size configurable
CHANGELOG_BEGIN
[Daml Driver for PostgreSQL]: Added CLI option --database-connection-pool-size
to configure the size of the database connection pool
[Integration Kit]: Added the CLI options api-server-connection-pool-size and
indexer-server-connection-pool-size to configure the database connection pool size
for the Ledger API Server and the indexer respectively.
CHANGELOG_END
* Add CLI options to enable daml-lf dev version
* Compile test-common with multiple compiler versions
* Run the ledger API test tool with multiple compiler versions
changelog_begin
changelog_end
* Fix scaladoc issues
* Apply reviewer comments
regarding lf_preview_version
* participant-integration-api: Reuse the services EC for data munging.
Rather than spawning a separate thread pool for manipulating data before
and after index queries, we can just re-use the services EC, which is
already a work-stealing thread pool.
* kvutils: Share the services EC with the indexer.
CHANGELOG_BEGIN
CHANGELOG_END
* Pipelined transaction indexing
CHANGELOG_BEGIN
[Integration Kit] The participant indexer (for PostgreSQL)
can now execute DAML transaction insertions in three pipelined stages.
CHANGELOG_END
* Make participant-integration-api test suite `large` for BAZEL
* Fixed constant timeout for MacOS builds
* Moved ledger end guard to TransactionReader
* Removed TransactionServiceResponseValidator
* Removed MetadataUpdate intermediary level from Update
* Added back store_ledger_entry timer
* Updated comment for idempotent insertions.
* Add test to verify that TLSv1.2 and TLSv1.3 work
As raised here: https://discuss.daml.com/t/what-version-of-tls-is-supported-by-the-ledger-api/1982
This has been effectively fixed by upgrading Netty here: https://github.com/digital-asset/daml/pull/8558
Note that this is an integration test for sandbox-classic only, but the
infrastructure used is shared with other Ledger API server
implementations, so this should be enough.
changelog_begin
changelog_end
* Fix Scala 2.13 compatibility issue
* Fix warning by adding type annotation to public field
CHANGELOG_BEGIN
- [Daml-LF] Release LF 1.11
- add suport for Generic Map
- add support for choice observer
- add generic comparison
- add convert of contract ID to string (available only in off-ledger mode)
- reduce size of dar using type interning
CHANGELOG_END
The default timeout of 250ms turned out to be problematic for Canton.
Therefore, I'm moving the hard coded 250ms up such that we can override
it from within the Canton code. Other users shouldn't be affected by the
change. Therefore, this is just an internal code change.
Additionally, I've added two new performance targets for the performance
envelope test.
Related to https://github.com/DACH-NY/canton/issues/4996
CHANGELOG_BEGIN
- Made the database connection timeout configurable.
CHANGELOG_END
* Port the rest of //ledger/... to Scala 2.13
draw the rest of the fcking owl
Omitted for now are the ledger API test tool which has a dependency
only compatible with 2.12 and the generated code of the Scala
codegen (the codegen compiles and runs with 2.13, the generated code
does not).
changelog_begin
changelog_end
* Less symbols
changelog_begin
changelog_end
* Enable asynchronous commit per connection level
CHANGELOG_BEGIN
CHANGELOG_END
* Added HikariConnectionSpec test for asserting async commits in connection
The flag allows the sandboxes (both classic and next) to use preview
version of the upcoming Daml-LF version (currently LF 1.11).
CHANGELOG_BEGIN
* [Sandbox-classic] add a flag to allow early access to the next
Daml-LF to be released.
* [Sandbox] add a flag to allow early access to the next Daml-LF
to be released.
CHANGELOG_END
This PR updates scalafmt and enables trailingCommas =
multiple. Unfortunately, scalafmt broke the version field which means
we cannot fully preserve the rest of the config. I’ve made some
attempts to stay reasonably close to the original config but couldn’t
find an exact equivalent in a lot of cases. I don’t feel strongly
about any of the settings so happy to change them to something else.
As announced, this will be merged on Saturday to avoid too many conflicts.
changelog_begin
changelog_end
The one thing that is still missing is making the generated Scala code
from the codegen compatible with Scala 2.13 so the examples are
excluded for now.
changelog_begin
changelog_end
* Replace many occurrences of DAML with Daml
* Update docs logo
* A few more CLI occurrences
CHANGELOG_BEGIN
- Change DAML capitalization and docs logo
CHANGELOG_END
* Fix some over-eager replacements
* A few mor occurrences in md files
* Address comments in *.proto files
* Change case in comments and strings in .ts files
* Revert changes to frozen proto files
* Also revert LF 1.11
* Update get-daml.sh
* Update windows installer
* Include .py files
* Include comments in .daml files
* More instances in the assistant CLI
* some more help texts
* Port damlc dependencies to Scala 2.13
I got a bit fed up by the fact that going directory by directory
didn’t really work since there are two many interdependencies in
tests (e.g., client tests depend on sandbox, sandbox tests depend on
clients, engine tests depend on DARs which depend on damlc, …).
So before attempting to continue with the per-directory process, this
is a bruteforce approach to break a lot of those cycles by porting all
dependencies of damlc which includes client bindings (for DAML Script)
and Sandbox Classic (also for DAML Script).
If this is too annoying to review let me know and I’ll try to split it
up into a few chunks.
changelog_begin
changelog_end
* Update daml-lf/data/src/main/2.13/com/daml/lf/data/LawlessTraversals.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* fixup lawlesstraversal
changelog_begin
changelog_end
* less iterator more view
changelog_begin
changelog_end
* document safety of unsafeWrapArray
changelog_begin
changelog_end
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
This commit fixes a few copyright headers that have been missed in the
automatic update on Jan 1, as well as the generation code in the compat
workspace so it generates the right headers.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Dedicated execution context for requests.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Construct the services executor outside.
* participant-integration-api: Share the services EC with the GRPC stack.
* participant-integration-api: Use the new EC wherever possible.
And stop using DirectExecutionContext.
* sandbox-classic: Fix DevModeIT to use the right configuration.
I have no idea why this breaks now, but it was always technically
broken, so I'm fixing it.
* participant-integration-api: Push the services EC up one more level.
* Update newly added ApiParticipantPruning service accordingly
see https://github.com/digital-asset/daml/pull/7988#discussion_r525319097
* participant-integration-api: Ensure the LedgerConfigProvider is ready.
Somehow this slipped through.
* sandbox-classic: Fix `DefaultConfig` in tests.
* language-support/java: Disable seeding in tests again.
Co-authored-by: Oliver Seeliger <oliver.seeliger@digitalasset.com>
