New instances and collection methods, many of which included
as-is from @andreaslochbihler-da's experiments downstream.
* mk, a version of apply with special inference
* new names for the conversion functions suggested by
@andreaslochbihler-da
* groupBy1, zipWithIndex, reduceLeft, last1, distinct, toVector1
* a derived Semigroup instance
* a more explicit subtyping utility to specify implicit preference than
what we get with shapeless @@
* from method, alias for unapply
* cats functorial, foldable instances in the new nonempty-cats library
* new NonEmpty utilities
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Lochbihler <36634420+andreaslochbihler-da@users.noreply.github.com>
Since Scala 2.13.2, Scala introduced built-in support to
manage warnings in a more granular fashion, thus making
the silencer plugin we are currently using no longer
strictly useful. Removing compiler plugins also removes
friction from migrating to Scala 3 in the future. As a
cherry on top, the built-in warning configuration also
allows to check whether a `@nowarn` actually does
anything, allowing us to proactively remove unused
warnings should the need arise.
[Here][1] is s a blog post by the Scala team about it.
Warnings have been either solved or preserved if useful,
trying to minimize the scope (keeping it at the single
expression scope if possible). In particular, all
remaining usages of the Scala Collection API compatibility
module have been removed.
Using the silencer plugin also apparently hid a few
remaining usages of compatibility libraries that were used
as part of the transition from Scala 2.12 to Scala 2.13
that are no longer needed. Removing those warnings
highlighted those.
changelog_begin
changelog_end
[1]: https://www.scala-lang.org/2021/01/12/configuring-and-suppressing-warnings.html
New year, new copyright, new expected unknown issues with various files
that won't be covered by the script and/or will be but shouldn't change.
I'll do the details on Jan 1, but would appreciate this being
preapproved so I can actually get it merged by then.
CHANGELOG_BEGIN
CHANGELOG_END
* add PartySet alias for db-backend
* add PartySet alias for fetch-contracts
* add PartySet alias for http-json
* deprecate old apply
* quick builder for NonEmpty collections
* replace PartySet in db-backend
* replace PartySet in fetch-contracts
* lar.Party is also domain.Party
* add incl1 operator
* replace PartySet in http-json
* port tests
* into with Scala 2.12 needs collection-compat
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* simplify a couple functions that don't need so much data transformation now
* clean up some OneAnds and HKTs
* deal with Scala 2.12 without having warning suppression
* better, more obscure choice for Scala 2.12
* trying to reliably reproduce the template ID constraint error
* tentative fix for template ID constraint error
* sequential simulation
* successfully reproduce the error pre-4633c3137a
- Batch entry 0
INSERT INTO some_fancy_prefix_contract
VALUES ('foo', 1, 'null'::jsonb, NULL, '{}'::jsonb, ?, ?, '')
ON CONFLICT (contract_id) DO NOTHING
was aborted: ERROR: insert or update on table "some_fancy_prefix_contract" violates foreign key constraint "some_fancy_prefix_contract_tpid_fkey"
Detail: Key (tpid)=(1) is not present in table "some_fancy_prefix_template_id".
* also reproduced the error pre-4633c3137a on Oracle
- ORA-02291: integrity constraint (UNA3GOHUV7YMSKT0MQXJKLKD9HKKAZ.SYS_C007859)
violated - parent key not found
* add changelog
CHANGELOG_BEGIN
- [JSON API] Fixed a rare error that manifested as
‘violates foreign key constraint "contract_tpid_fkey"
Detail: Key (tpid)=(...) is not present in table’
when attempting to run queries and goes away on JSON API restart.
See `issue #11330 <https://github.com/digital-asset/daml/pull/11330>`__.
CHANGELOG_END
* clean up some now-unneeded printlns
Closes#11251
Schema changed as part of https://github.com/digital-asset/daml/pull/11102
Also backported to 1.17.1 in https://github.com/digital-asset/daml/pull/11143
changelog_begin
[JSON API] Solving a bug that could cause the JSON API to return
correct result if a contract with the same key is observed twice
required a schema change. The JSON API data needs to be dropped
and the query store needs to reset. If you are migrating from a
previous version, either reset your database manually or start
the HTTP JSON API with one of the options that regenerate the
schema (`create-only`, `create-if-needed-and-start`, `create-and-start`).
changelog_end
* Add failing test that covers the bug
* Fix on conflict error for inserts into the contracts table
changelog_begin
- [JSON-API] make key_hash indexes non-unique, this fixes a bug where a duplicate key conflict was raised on the query store when the same contract was being witnessed twice by two separate parties
changelog_end
* move test to parent so as to test oracle query store
* make key_hash indexes non-unique
* use recordFromFields
Co-authored-by: Akshay <akshay.shirahatti@digitalasset.com>
* enumerating out-of-sync offsets at the DB level
* cleanup in lastOffset
* write the latest-requested-or-read offset when catching up
- Writing only the latest-read, as before, would imply unsynced offsets
that are actually well-synced. This puts the DB in a more uniform
state, i.e. it should actually reflect the single value that the
fetchAndPersist loop tries to catch everything up to.
* detecting lagging offsets from the unsynced-offsets set
- Treating every unsynced offset as a lag would make us needlessly retry
perfectly synchronized query results.
* add Foldable1 derived from Foldable for NonEmpty
* nicer version of the unsynced function
* ConnectionIO scalaz monad
* rename Offset.ordering to `Offset ordering` so it can be imported verbatim
* finish aggregating in the lag-detector function, compiles
* port sjd
* XTag, a scalaz 7.3-derived tag to allow stacked tags
* make the complicated aggregation properly testable
* extra semantic corner cases I didn't think of
* tests for laggingOffsets
* a way to rerun queries if the laggingOffsets check reveals inconsistency
* if bookmark is ever different, we always have to rerun anyway
* boolean blindness
* incorporate laggingOffsets into fetchAndPersistBracket
* split fetchAndPersist from getTermination and clean up its arguments
* just compose functors
* add looping to fetchAndPersistBracket
* more mvo tests
* test unsyncedOffsets, too
* Lagginess collector
* supply more likely actual data with mvo tests; don't trust Java equals
* rework minimumViableOffsets to track sync states across template IDs
* extra note
* fix the tests to work against the stricter mvo
* move surrogatesToDomains call
* more tests for lagginess accumulator
* add changelog
CHANGELOG_BEGIN
- [JSON API] Under rare conditions, a multi-template query backed by database
could have an ACS portion that doesn't match its transaction stream, if
updated concurrently. This conditions is now checked and accounted for.
See `issue #10617 <https://github.com/digital-asset/daml/pull/10617>`__.
CHANGELOG_END
* port toSeq to Scala 2.12
* handle a corner case with offsets being too close to expected values
* didn't need XTag
* Initial changes to add a surrogate_template_id cache to reduce db queries
CHANGELOG_BEGIN
CHANGELOG_END
* refactoring and addition of tests
* Code review based changes to use Contextual Logger and json-api metrics instance
* make max cache entries/size configurable
* Rename cache max entries default variable
* unconditionally enable JSON search index on Oracle
In '1kb of data' and larger Oracle integration tests:
ORA-29902: error in executing ODCIIndexStart() routine
ORA-20000: Oracle Text error:
DRG-50943: query token too long on line 1 on column 3
From https://docs.oracle.com/en/database/oracle/oracle-database/19/errmg/DRG-10000.html#GUID-46BC3B3F-4DB7-4EB4-85DA-55E9461966CB
Cause: A query token is longer than 256 bytes
Action: Rewrite query
* add changelog
CHANGELOG_BEGIN
- [JSON API] The Oracle database schema has changed; if using
``--query-store-jdbc-config``, you must rebuild the database by adding
``,start-mode=create-only``. See #10539.
CHANGELOG_END
* test only 1kb
* extra flag in db config string
* let Queries backends configure themselves from maps
* new Queries constructor dataflow to better support config values
* remove fields as we go, isolating backend-specific from -agnostic conf
- we use StateT to avoid the problems that will definitely arise if we
don't DRY.
* fix up DbConfig including DbStartupMode
* start to uncouple json-api's config from db-utils
* two JdbcConfigs with different purposes/scopes
- also moves db-utils contents to com.daml.dbutils
* adapt trigger service to refactoring
* fix JdbcConfig leftovers
* adapt http-json-cli to new JdbcConfig
* remove extra ConfigCompanion
* explain more about the QueryBackend/Queries distinction
* split SupportedJdbcDriver into two phases with a tparam
* use SupportedJdbcDriver.TC instead of SupportedJdbcDriver as the nullary typeclass
* patch around all the moved objects with imports
* missed import from moving ConnectionPool to dbutils
* use new 2-phase SupportedJdbcDriver for ContractDao setup
* left off part of a comment
* more q.queries imports
* other imports from the dbutils move
* nested JdbcConfig
* configure the driver in each backend-specific test
* very confusing error, but make the imports nicer and it goes away
* nested JdbcConfig in perf
* missing newline
* port contractdao-bench
* test new option parsing all the way through QueryBackend
* disable search index for some tests, enable for others
* add changelog
CHANGELOG_BEGIN
- [Trigger Service] ``--help`` no longer advertises unsupported JDBC
options from JSON API.
- [JSON API] [EE only] By default, on Oracle, sets up a JSON search
index to speed up the queries endpoints. However, Oracle versions
prior to 19.12 have an unrecoverably buggy implementation of this
index; in addition, the current implementation fails on queries with
strings >256 bytes, with no way to disable the index for that query.
Pass the ``disableContractPayloadIndexing=true`` option as part of
``--query-store-jdbc-config`` to disable this index when creating the
schema.
See `issue #10539 <https://github.com/digital-asset/daml/pull/10539>`__.
CHANGELOG_END
* port failure tests
* init version table last, drop first
- suggested by @realvictorprm; thanks
* rename split DBConfig.scala
- suggested by @realvictorprm; thanks
* move imports to not be in alphabetical order
- suggested by @realvictorprm; thanks
* remove createSchema
- suggested by @realvictorprm; thanks
* Revert "test only 1kb"
This reverts commit 616e173e63.
* port to scala 2.12
- bug in unused imports
- old name `-` for `removed`
* Addition of a key_hash field to speed up fetchByKey queries
CHANGELOG_BEGIN
CHANGELOG_END
* changes to make key_hash and Optional field
CHANGELOG_BEGIN
- Update schema version for http-json-api query store with new key_hash field
- Improved performance for fetchByKey query which now uses key_hash field
CHANGELOG_END
* remove btree index for postgres and other changes based on code review comments
* Upgrade Scalatest to v3.2.9.
Because of some coupling we also have to upgrade Scalaz to the latest
v7.2 point release, v7.2.33.
The Scalatest changes are quite involved because the JAR has been broken
up into several smaller JARs. Because Bazel expects us to specify all
dependencies and doesn't allow transitive dependencies to be used
directly, this means that we need to specify the explicit Scalatest
components that we use.
As you can imagine, this results in quite a big set of changes. They
are, however, constrained to dependency management; all the code remains
the same.
CHANGELOG_BEGIN
CHANGELOG_END
* http-json-oracle: Fix a Scalatest dependency.
* ledger-api-client: Fix a Scalatest dependency.
17709b5ba3 (#10344) brought the two implementations of
`selectContractsMultiTemplate` close together enough that they can be
usefully factored. Here is that factoring.
Several of the arguments to `queryByCondition` take the form
(Read[T], T => Out), i.e. Coyoneda; we could invert the control by
returning a data structure with coyonedas, but instead here we use a
sort of continuation-passing style, so the coyonedas are embedded in the
arguments to `queryByCondition`.
CHANGELOG_BEGIN
CHANGELOG_END
* Move ExceptionOps from ledger-service/utils to //libs-scala/scala-utils
* extract connection and JdbcConfig from //ledger-service to independent db-utils module
Changelog_begin
Changelog_end
* update trigger service to use new libs-scala/db-utils
* missed changes for http-json-oracle
* minor cleanup based on comments
* fix breaking scala 2_12 build
* cleanup db-utils/BAZEL.md file
* [JSON-API] Move database independent tests into a seperate abstract test
The DatabaseStartupOps tests are now also tested against Oracle.
Besides, an additional test will now cover that table creation
doesn't run into name collisions for different table prefixes within
the same database.
changelog_begin
changelog_end
* Add missing copyright headers
* Adjusting the version query slightly to fix the oracle db integration tests
* Rewrite the version query of oracle to fix it (hopefully)
* Test the prefix collision the other way around
* Put the table prefix also infront of the ensure_json constraint in the oracle queries
* Convert the table name of the jsonApSchemaVersion table to uppercase so it can be found in the list of the created tables in Oracle.
* Fix scala 2.12 collection compatibility compiler error by using :+
* Update ledger-service/db-backend/src/main/scala/com/digitalasset/http/dbbackend/Queries.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Use flatTraverse instead of flatMap to fix the compile error in Queries.scala
* Process the startup mode also in the tests & error if it failed
* Add collections compat import to fix scala 2.12 build failure
* Be confused about the build error prior, revert the change
* Move dropAllTablesIfExist a bit down to have a better declaration order
* Extract the tables vector combined with the version table into a seperate val
* Remove debug in Queries.scala logging
* Make the initDatabaseDdlsAndVersionTable val lazy, so we don't get a nullpointer exception
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
#9895 reintroduced the row_number over partition to eliminate duplicates
when querying the stakeholder-joined Oracle contract table. However, as
#10123 establishes, these duplicates cannot happen if we are querying
for only one party.
Therefore, we special-case the single-party query case, for which we
skip the partition + outer-query duplicate elimination steps.
CHANGELOG_BEGIN
CHANGELOG_END
* [JSON-API] Add option for setting a table prefix
changelog_begin
- [JSON-API] A table prefix can now be specified in the jdbc config via `tablePrefix=<YourFancyTablePrefix>`. This was added to allow running multiple instances of the json api without having collisions (independent of the chosen database).
changelog_end
* Extend the correct test in the oracle tests and simplify config override
* Fix formatting
* Fix postgres tests
* Fix bug in oracle query
* Fix typo
* Update ledger-service/db-backend/src/main/scala/com/digitalasset/http/dbbackend/Queries.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* add the table prefix to named constraints too
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* [JSON-API] Validate schema version & add minimal options for schema creation
* Add tests
* [JSON-API] Rework prior work and introduce the object SchemaHandling
* Add license headers & revert formatting changes
* Fix oracle build & scala 2_12 build
* correctly fix 2.12 build
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/SchemaHandlingResult.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* [JSON-API] Change case names & add backwards compat (but deprecate createSchema=true)
changelog_begin
- [JSON-API] Schema versioning was introduced to the db schema. Because of this the field `createSchema` in the jdbcConfig was deprecated. Via the field `start-mode` you can specify:
1. `create-only`: This is equal to the behaviour of `createSchema=true` so the db schema is created and then the application terminates.
2. `start-only`: With this the schema version is checked and if no version or an version was found which is not equal to the internal schema version then the application terminates. This is also the replacement of `createsSchema=false`.
3. `create-if-needed-and-start`: With this the schema version is checked and if no version or an version was found which is not equal to the internal schema version then the schema will be created/updated and the application proceeds with the startup.
4. `create-and-start`: Similar to the first option but instead of terminating the application proceeds with the startup.
changelog_end
* Add info about deprecated createSchema field
* Fix build & improve logging
* Give suggestions on what option to take, to fix an outdated or missing schema
* Renaming of schemaHandling to DbStartupMode, added more tests & correct exit codes depending on how the db startup went
* Align name with sandbox
* Improve tests
* Only add new sql code which strictly uses the interpolation to align with other pr's & minimally adjust statements
* Minimize diff
* Add backwards compat test
* Fix scala 2.12 build & oracle integration tests build
* Update ledger-service/http-json-cli/src/main/scala/com/daml/http/Config.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Adjust code according to review request & tests & add a failure test
* If the call to initialize fails also log the error which was thrown
* Fix formatting
* Add missing collections compat import in integration tests
* Fix last build errors (scala 2.12) & use Either instead of Option for getDbVersionState
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Switch ContractDao to use a HikariCP connection pool
CHANGELOG_BEGIN
CHANGELOG_END
* missed conn pool changes for PostgresTest and ContractDaoBenchmark
* shutdown db access await threadpool and fix formatting
* custom pool sizes for Prod and Integration similar to DbTriggerDao
* cleanup contract dao connection pool
* simplify Dao shutdown
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* remove redundant config setting
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* fix code formatting issue, NonUnitStatments warning
* use doobie 0.9.0 Fragment-in-Fragment interpolation in json-api db-backend
Since tpolecat/doobie#1045 (and therefore 4ca02e0eb6) doobie has
supported interpolating fragments in fragments. We've used this feature
for several fragments written since #7618, but have left the ones
written before alone to use ++. Here we change that where it
meaningfully clarifies the SQL subexpression.
Note that this does not entail a Put or Write instance for Fragment.
You cannot abstract over Fragment and arbitrary interpolated data in
this way, because Fragments are not treated as positional parameters;
that would mean being able to put arbitrary SQL substrings in positional
parameters.
CHANGELOG_BEGIN
CHANGELOG_END
* scalafmt
* useless whitespace accidentally removed
* new projection for aggregated matched-queries
We can redo all the template-ID matches (and payload query matches, if
needed) in the SELECT projection clause to emit a list of matchedQueries
indices SQL-side.
CHANGELOG_BEGIN
CHANGELOG_END
* selectContractsMultiTemplate always returns one query
* factoring
* remove multiquery deduplication from ContractDao
* test simplest case of projectedIndex; remove uniqueSets tests
* remove uniqueSets
* add more test cases for the 3 main varieties of potential inputs
* remove uniqueSets tests that were commented for reference
* remove unneeded left-join
* scala 2.12 port
* port Map test order to 2.12
* use SortedMap so the Scala version tests are unified
- suggested by @cocreature; thanks
* Support deletion of a large number of contracts
fixes#10339
There are two orthogonal issues here:
1. scalaz’s toVector from the Foldable[Set] instance
stackoverflows. I’ve just avoided using that altogether.
2. Oracle doesn’t like more than 1k items in the IN clause. I chunked
the queries into chunks of size 1k to fix this.
changelog_begin
- [JSON API] Fix an error where transactions that delete a large
number of contracts resulted in stackoverflows with the PostgreSQL
backend and database errors with Oracle.
changelog_end
* fix benchmark
changelog_begin
changelog_end
* Update ledger-service/db-backend/src/main/scala/com/digitalasset/http/dbbackend/Queries.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Update ledger-service/db-backend/src/main/scala/com/digitalasset/http/dbbackend/Queries.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* that's not how you foldA
changelog_begin
changelog_end
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Was curious if there were any relevant performance improvements in
newer versions. Looks like the answer is no but we might as well
upgrade anyway.
changelog_begin
changelog_end
I haven’t found any conclusive information as to why ON COMMIT doesn’t
work incrementally but
https://docs.oracle.com/en/database/oracle/oracle-database/19/adjsn/json-query-rewrite-use-materialized-view-json_table.html#GUID-8B0922ED-C0D1-45BD-9588-B7719BE4ECF0
recommends that for rewriting (which isn’t what we do here but both
involve a materialized view on json_table).
Benchmarks:
before:
InsertBenchmark.run 1000 1 1000 avgt 5 0.327 ± 0.040 s/op
InsertBenchmark.run 1000 3 1000 avgt 5 0.656 ± 0.043 s/op
InsertBenchmark.run 1000 5 1000 avgt 5 1.034 ± 0.051 s/op
InsertBenchmark.run 1000 7 1000 avgt 5 1.416 ± 0.106 s/op
InsertBenchmark.run 1000 9 1000 avgt 5 1.734 ± 0.143 s/op
QueryBenchmark.run 1000 10 N/A avgt 5 0.071 ± 0.016 s/op
After:
Benchmark (batchSize) (batches) (numContracts) Mode Cnt Score Error Units
InsertBenchmark.run 1000 1 1000 avgt 5 0.217 ± 0.034 s/op
InsertBenchmark.run 1000 3 1000 avgt 5 0.232 ± 0.027 s/op
InsertBenchmark.run 1000 5 1000 avgt 5 0.226 ± 0.051 s/op
InsertBenchmark.run 1000 7 1000 avgt 5 0.225 ± 0.048 s/op
InsertBenchmark.run 1000 9 1000 avgt 5 0.232 ± 0.021 s/op
QueryBenchmark.run 1000 10 N/A avgt 5 0.080 ± 0.014 s/op
The difference in query times is just noise and changes across runs.
So we get the expected behavior of inserts being independent of the
total ACS size now. We could still explore if we gain something by
avoiding the materialized view to reduce constant factors but that’s
much less of an issue.
fixes#10243
changelog_begin
changelog_end
* nvarchar2 keys are text-incompatible, but varchar2 keys are fine
* commit the ACS update before query
* add changelog
CHANGELOG_BEGIN
- [JSON API] The Oracle database schema has changed; if using
``--query-store-jdbc-config``, you must rebuild the database by adding
``,createSchema=true``. See #9895.
CHANGELOG_END
* expand the InitDdl set to include materialized views
* replace search index with a materialized view that expands the stakeholders
* allow materialized views to be created in Oracle testing
* join and query the contract_stakeholders table for party-set membership
- restoring a few elements removed by 3e6661128d (#9484)
* upgrade scalacheck to 1.14.3
* regenerate maven_install files
* some different names and implicits
* remove some fromTryCatchNonFatal
* more porting
* port fromTryCatchNonFatal to attempt
* factor the assertions in SignatureSpec to avoid \/
* deal with invariant \/
* make partial unification do what we want
* \/, parse*, and toNel
* many uses of the .right method
* a legitimate use of fromTryCatchThrowable
* rebuild maven pins
* further invariant \/
* OneAnd and Nel interface changes
* further Either games
* \/ and reformatting
* \/ in http-json
* \/ in http-json
* deprecations
* more invariance
* cleanup unused
* more invariance; http-json compiles
* final either follies
* small 2.12 extra incompatibility
* rebuild deps
* revisit a couple earlier fixes using nicer expressions I learned later
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* repin 2.12
* remove Array[String] instances from Oracle json-api driver layer
CHANGELOG_BEGIN
CHANGELOG_END
* fix Postgres integration test to deal with removed implicit API
* comparison queries
* name the contract primary key constraint
* use ignore_row_on_dupkey_index instead of merge
- suggested by @cocreature in #9286 f7b2f14294fa33d6804251ce841529a1e2bd298d; thanks
* retrySqlStates for oracle
* enable all non-websocket tests
* name the template_id primary key constraint
* clean up concatFragment calls
* add Websocket tests for oracle
* move iouCreateCommand to be usable by oracle integration tests
* work around Scala 2.12 NPE in Source
* multiquery support for Oracle
* matchedQueries, therefore query-stream support for Oracle
* enable websocket tests
* test '& bar' and 5kb strings
- 5kb string fails on Oracle with
ORA-01704: string literal too long
* refine the long data cases; gets too long at 4000 bytes as expected
- however, the predicate fails for unknown reason before then; possibly a missed
escape character case
* handle long data with a fallback
- now the predicate fails in all cases instead of a SQL error, which is...better
* only interpolate true, false, null into JSON predicate conditions
- the problem was with JSON-formatted data; it must be SQL-typed instead
* adapt equal's large-data solution for comparison as well
- only works for numbers and strings, but that's all we need to compare
* move Implicits to Queries
* remove stray spaces in output
* test Oracle query expressions alongside Postgresql
* test that bools aren't compared like numbers and strings
* test @> conjunctions and special {}-query handling
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* note on PASSING ... AS X
- suggested by @cocreature; thanks
* remove printlns; these functions don't really need scaffolding anymore
- suggested by @stefanobaghino-da; thanks
* support scalaz.Foldable1 in Fragments.in
* incorporating signatories and observers in Oracle contract query
* join syntax; allowed aggregation
* aggregate the signatories and observers independently before join
- prior: ERROR at line 8 (the GROUP BY line):
ORA-00932: inconsistent datatypes: expected - got CLOB
* make toSqlWhereClause portable, mostly
* name the constraints for debugging
* import cleanup
* skip inserting contract on conflict (for read committed)
* support lookup by contract ID
* remove ::jsonb from fetch-by-key for Oracle
* proper key comparison and retrieval
* on conflict ignore for signatories and observers
* contract ID, party, offset, package ID column types
* template module and entity name types
- nvarchar2 for name type because
,template_module_name CLOB NOT NULL
,template_entity_name CLOB NOT NULL
,UNIQUE (package_id, template_module_name, template_entity_name)
)
, Error Msg = ORA-02329: column of datatype LOB cannot be unique or a primary key
* type-aware == and @> output for Oracle
* pick arbitrary maximum module/entity name size
Cause: Error : 1450, Position : 0, Sql =
CREATE TABLE
template_id
(tpid NUMBER(19,0) GENERATED ALWAYS AS IDENTITY NOT NULL PRIMARY KEY
,package_id NVARCHAR2(64) NOT NULL
,template_module_name NVARCHAR2(1594) NOT NULL
,template_entity_name NVARCHAR2(1594) NOT NULL
,UNIQUE (package_id, template_module_name, template_entity_name)
)
, Error Msg = ORA-01450: maximum key length (6398) exceeded
* happy path for query-less queries
* done todo
CHANGELOG_BEGIN
CHANGELOG_END
* handle 2.13 deprecation
* factor NVARCHAR2(255)s
- suggested by @cocreature; thanks
* deal with where only a signatory OR observer matches
- suggested by @cocreature; thanks
* non-empty newtypes
* an operation
* add some map/set operations and make everything compile on 2.12 and 2.13
* +-: and :-+, with compatibility layer; docs
* move to nonempty package; add aliases for cons/snoc; fix SeqOps aliases
* ensure 2.12 aliases are inferrable
* groupBy1 and toList, use to prove uniqueSets's invariants
* prove immutability first
* matching variance in aliases
* prove the return property of uniqueSets, and use the proof
* tests for NonEmpty API
* rename sci alias to imm
* move RefinedOps to more obvious location
* more docs
CHANGELOG_BEGIN
CHANGELOG_END
* remove unused imports
* illustrate the scala.collection.Seq problem
* ideas for extension
* tests for toF
* tests for +-:
* explain difference with OneAnd
* separate OracleQueries from PostgresQueries
- with some changes from 8161e63189 courtesy @cocreature
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* abstract BIGINT
* json, signatories, observers columns
* compatible lastOffset
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* oracle functions for select (single template ID), insert
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* add oracle branch to integration tests
* oracle CLI configuration for json-api
* run integration tests with ojdbc in classpath
* update maven_install for ojdbc
* drop table if exists for Oracle
* make create DDLs and drops more planned out; drop in reverse order for Oracle integrity
* repin maven
* port agreement_text
* port (by removal) array part of ledger offset update
* use CASE instead of JSON map lookup for multiparty offset update
* simplify self types
* fix contract archival
* repin
* remove selectContracts in favor of selectContractsMultiTemplate
* move Oracle test execution to separate build target
* move websocket test to itlib
* make a bad array instance for Oracle
* report actually-available JDBC drivers only
* configure Oracle test from CI
* attempt with platforms and constraints
* a mismash of bazel to get it to conditionally enable oracle testing
* fix dep resolution in Scala 2.13
* make the Oracle test a stub (inits and does empty DB query)
* remove commented unused deps
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* repin
* we never supply a value for the surrogate ID columns
- suggested by @cocreature; thanks
* add not null to json in DB-specific place
- suggested by @cocreature; thanks
* why DBContractKey
- suggested by @cocreature; thanks
* textType isn't finalized
- suggested by @cocreature; thanks
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Unfortunately missing the actual interesting part since porting
`partitionBimap` seems to be rather annoying but this at least gets us
started on the easy parts.
changelog_begin
changelog_end
* SupportedJdbcDriver box for the required DB-specific implicits and magic values
* replace postgres references with the SupportedJdbcDriver box
* explaining the typeclass
* labels for debugging
* allow external initialization of SupportedJdbcDriver, but not usage
* thread SupportedJdbcDriver everywhere, hang it off of ContractDao
* remove unused dep from integration tests
* split Queries into an agnostic part and a DB-specific part
* document withOptPrefix
* reformat
* SQL syntax more amenable to refactoring
* different sets of DDL for different backends
* make everything use queries passed around everywhere (usually via ContractDao)
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* group database queries while still providing the matchedQueries part of the event
* SQL query for multiquery websocket request all at once
* fetchAndPersist responds with a single bookmark
* unify connection imports
* convert DB results to domain, reassociate with proper query indices
* reassociate multiple matches with proper query indices
* make overlap more likely in testing overlap
* simpler matchedQueries merging for multi-query case
* integrate DB query with metadata-ful StreamQuery's
* expose daoAndFetch; better insertDeleteStepSource doc
* more efficient query path for contract key streams
* missed LogHandler
* persist resolved template IDs; glue the prefiltered set into the stream
* ticks and phantom removal need the state from the ACS
* compile SQL queries for query language predicates on WS
- wrong matchedQueries order
* harmonize order of ACS-vector and Positives
* misc compilation conversions
* WebsocketServicePostgresInt mixin
* update (C) date
* test websocket queries under postgres
* looking for new way to compile queries with proper matchedQueries offsets
* model that querying without matchedQueries requires only one SQL query
* SQL path for contract key streams
* nondeterminism
* fix 3 fetch tests with SQL syntax
* nondeterminism mk 2
* fix multi-party query tests by dealing with nondeterminism properly
* temp logs to track down the contract duplication
* match new scalafmt from #8437
* remove completed TODOs
* add changelog
CHANGELOG_BEGIN
- [JSON API] If the JDBC query store is enabled, it will be used to optimize
Websocket queries as well as the previously-supported synchronous queries.
See `issue #8226 <https://github.com/digital-asset/daml/pull/8226>`__.
CHANGELOG_END
* fix up matchedQueries indices
* complete the fast path for by-id queries
* remove AS c
- suggested by @cocreature; thanks
* remove temporary debugging logs
- suggested by @cocreature; thanks
This PR updates scalafmt and enables trailingCommas =
multiple. Unfortunately, scalafmt broke the version field which means
we cannot fully preserve the rest of the config. I’ve made some
attempts to stay reasonably close to the original config but couldn’t
find an exact equivalent in a lot of cases. I don’t feel strongly
about any of the settings so happy to change them to something else.
As announced, this will be merged on Saturday to avoid too many conflicts.
changelog_begin
changelog_end
* mark some of dbbackend private
* fetchBy* functions for Queries
* shift in-memory filtering into the transaction stream
- removes irrelevant contracts from memory as soon as possible for fetch by
contract ID and key in-memory
* push the three synchronous search varieties into new signatures
* replace the core findByContract* functions with DB-delegating versions
* remove the GADT equality and most of the explicit traversals
- compiles again, finally
* factoring
* ContractDao wrappers for fetchById and fetchByKey
* DB version of findByContractId
* DB version of findByContractKey
* Search is the split of ContractsService
* fix SQL for keys
* trade the typeclass for a sum type
- sealed instead of final because of the path dependency on ContractsService
instance
* number conversion is done already in ContractDao
* make fetch-by-key tests depend on proper number conversion for SQL
* add changelog
CHANGELOG_BEGIN
- [JSON API] ``/v1/fetch`` now uses the Postgres database, if configured, to
look up contracts by ID or key, except when querying a contract by ID without
its corresponding template ID. The fallback in-memory version of
``/v1/fetch`` is also now significantly more efficient for large datasets,
though still linear.
You may optionally re-create JSON API's database to take full advantage.
See `issue #7993 <https://github.com/digital-asset/daml/pull/7993>`__.
CHANGELOG_END
* use search.search for search
- suggested by @cocreature; thanks
* add an index for contract key lookups
- suggested by @cocreature; thanks
* set doobie version to 0.9.2 and rerun maven pin
* port extractor and some of JSON API
* repin maven
* use doobie's own builder compatibility where required
* use probably bad derivations to supply Blockers where transactEC was required
- The point of using Blocker instead of ExecutionContext seems to be to
especially emphasize to API users that it isn't appropriate to use an
ExecutionContext with ordinary behavior. That is what we have done, which
should probably change, but just compiling for now.
* fix fragment inspection test for internal restructuring
- This test depends on implementation details of Doobie, so naturally it must be
altered when that runs. Fortunately, it's been made easier by the changes
in this upgrade.
* allow 256 blockers for navigator transaction blocker, like the global EC
* allow as many blockers as the pool size for trigger service
- The transactor shouldn't share ExecutionContext for transactions with the
caller, so we set up a new one based on configured pool size.
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* add silent_annotations option to da scala bazel functions
* use silent_annotations for several scala targets
* use silencer_plugin instead when the lib isn't used
* use silent_annotations for several more scala targets
* use silencer_lib for strange indirect requirement for running tests
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* silent_annotations support for scaladoc
* Support multi-party reads on the JSON API
Given that those aren’t going away and we’re instead doubling down on
this and adding multi-party writes as well, the JSON API needs to
support this. This PR only implements the read side (since the ledgers
do not yet support the write side).
This does not deviate from the approach chosen by the JSON API to
infer the parties from the token, we just don’t error out anymore when
more than one party is passed.
changelog_begin
changelog_end
* Apply suggestions from code review
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Remove dependency on doobie_postgres from db-backend
changelog_begin
changelog_end
* Fix offset update
changelog_begin
changelog_end
* Use nonempty sets for parties
changelog_begin
changelog_end
* Fix updateOffset under concurrent transactions
changelog_begin
changelog_end
* Add tests for multi-party websocket queries and fetches
changelog_begin
changelog_end
* fmt
changelog_begin
changelog_end
* Fix perf tests
changelog_begin
changelog_end
* Cleanup
changelog_begin
changelog_end
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/dbbackend/ContractDao.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Move ParsePayload instances, thanks Stephen!
changelog_begin
changelog_end
* More unsubst
changelog_begin
changelog_end
* Fix off by 1 error
changelog_begin
changelog_end
* Remove redundant type annotation
changelog_begin
changelog_end
* fmt
changelog_begin
changelog_end
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* add lf_scalacopts to ledger-service/ projects
* handle unused warnings in http-json main
* handle unused warnings in lf-value-json
* handle unused warnings in http-json tests
* handle unused warnings in jwt
* JwtGenerator.generate doesn't use its argument; remove it
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* format bazel files
