Commit Graph

7349 Commits

Author SHA1 Message Date
Remy
caf85a2270
LF: rationalize archive Parser/Reader/Decoder (#10239)
CHANGELOG_BEGIN
CHANGELOG_END
2021-07-14 11:05:17 +02:00
Moritz Kiefer
0043b81f00
Add a ContractDao benchmark (#10259)
* Add a ContractDao benchmark

This PR adds a simple benchmark that uses the ContractDao directly and
is therefore a bit more fine-grained and easier to analyze than the
gatling benchmarks. I expect we’ll want to extend this, this really
only tests queries on reasonably large size ACS filtered by party but
let’s start somewhere.

fixes #10247

changelog_begin
changelog_end

* Factorize

changelog_begin
changelog_end
2021-07-14 11:02:32 +02:00
Miklos
21a9e62f27
Add disclaimer to participant state v2 API [KVL-998] (#10254)
* Added disclaimer to Read/WriteService.

* Removed NeedCompletionOffsetForSubmissionId rejection reason.

* CHANGELOG_BEGIN
CHANGELOG_END

* Added disclaimers to adaptors regarding partial implementation.
2021-07-14 08:42:49 +00:00
Samir Talwar
6c63f96ba9
participant-integration-api: Move SeedService here. (#10261)
I don't see a reason why it's part of the participant state API, and
it definitely doesn't need to change between v1 and v2.

CHANGELOG_BEGIN
- [Integration Kit] The class ``SeedService`` has been moved from the
  *participant-state* Maven package to the *participant-integration-api*
  Maven package, under the Java package name
  ``com.daml.platform.apiserver`` to reflect its usage by the API
  server, not the participant state API. If you use this class directly,
  you will need to change your imports.
CHANGELOG_END
2021-07-14 08:35:29 +00:00
Hubert Slojewski
58c0b4694a
kvutils: Simplify usage of Rejections [KVL-1015] (#10258)
CHANGELOG_BEGIN
CHANGELOG_END
2021-07-14 09:15:39 +02:00
Hubert Slojewski
41aba1ca39
kvutils: Move unrelated test cases to a proper test class [KVL-1015] (#10257)
CHANGELOG_BEGIN
CHANGELOG_END
2021-07-14 09:15:25 +02:00
Samir Talwar
1f35db17c7
ledger-on-sql: Use random log entry ID allocation so we don't depend on SeedService. [KVL-1002] (#10255)
* ledger-on-sql: Abstract out the log entry ID allocator.

This is so we don't need to depend on `SeedService`.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Remove `allocateRandomLogEntryId` from production code.

* kvutils: Remove `SubmissionValidator.create`, as it's only for tests.

And remove default parameters for
`SubmissionValidator.createForTimeMode`.

* ledger-on-sql: Use the random log entry ID allocator.

In Sandbox, we still use the seed service, but this is not necessary
for ledger-on-sql tests.

* kvutils: Make `LogEntryIdAllocator.random` an `object`.

* ledger-on-sql: Move `RandomLogEntryIdAllocator` here.

It's not used anywhere else.

* ledger-on-sql: Add some documentation about the application code.

Just to make it clear it's not used in production.
2021-07-13 17:10:30 +00:00
Sofia Faro
b8e2198873
Separate traces from warnings in engine. (#10253)
* Separate traces from warnings in engine.

I decided to separate the engine warnings from the tracelog after all,
because I think it will make testing and maintenance easier in the
long run.

Part of #9947, follow up from #10179

changelog_begin
changelog_end

* scalafmt

* Apply suggestions from code review

Co-authored-by: Remy <remy.haemmerle@daml.com>

* dont use case class for WarningLog

* revert TraceLog changes from last PR

* Scala 2.12 doesnt have ArrayBuffer.addOne :(

* remove isWarnEnabled check

Co-authored-by: Remy <remy.haemmerle@daml.com>
2021-07-13 14:24:27 +00:00
Victor Peter Rouven Müller
0a7f2b1a1b
[JSON-API] Small refactoring to start using akka http routes internally (#10252)
* [JSON-API] Small refactoring to start using akka http routes internally

changelog_begin
changelog_end

* Fix build error

* Explicitly use the immutable seq via additional import

* Fix tests

* Fix scala 2.12 build
2021-07-13 14:07:03 +00:00
Remy
6e8ec1d618
LF: Drop old depreated code (#10251)
CHANGELOG_BEGIN
CHANGELOG_END
2021-07-13 13:10:00 +00:00
tudor-da
de7a08fa7b
[In-memory fan-out] Handle null submitters in getTransactionLogUpdates (#10248)
* [In-memory fan-out] Handle `null` submitters in getTransactionLogUpdates

CHANGELOG_BEGIN
CHANGELOG_END

* Address review comments
2021-07-13 12:11:27 +00:00
Miklos
584169a2cc
Participant state v1-to-v2 adaptors [KVL-1002] (#10246)
CHANGELOG_BEGIN
CHANGELOG_END
2021-07-13 13:35:35 +02:00
Remy
e4d7cd764a
LF: Deprecate com.daml.lf.data.TryOps.Bracket (#10249)
* LF: Deprecate com.daml.lf.data.TryOps.Bracket

CHANGELOG_BEGIN
CHANGELOG_END

* address reviews

* fix
2021-07-13 10:16:53 +00:00
Victor Peter Rouven Müller
b59f36591d
[JSON-API] Correctly extract the request source URL/IP (#10244)
* [JSON-API] Correctly extract the request source URL/IP

changelog_begin
- [JSON-API] If the service is put behind a proxy filling either of these headers X-Forwarded-For & X-Real-Ip then these will now be respected for logging the request source ip/url
changelog_end

* Return to the simple http server start code

* Remove unused import

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2021-07-13 08:33:40 +00:00
Kamil Bożek
edaf541063
Adapt ledger-api-bench-tool to work with LR [DPP-424] (#10171)
* Print ledger-api-bench-tool final report when a stream in ended by shutting down API server
CHANGELOG_BEGIN
CHANGELOG_END

* Log ledger-api-bench-tool failure with specific header to make it compatible with the lr-sidecar

* Print final report on graceful shutdown even if streams are not complete

* trigger build
2021-07-13 09:58:51 +02:00
Remy
6fe6ae075d
LF: Make DarReader ZipEntries immulatble (#10243)
* LF: Make DarReader ZipEntries immulatble

CHANGELOG_BEGIN
CHANGELOG_END

* optim

* Revert "optim"

This reverts commit 3b25a8c68b.

* a function should not close a stream it didn’t open

* typo

* we do not support dar without manifest !
2021-07-12 21:13:25 +00:00
Marton Nagy
05e5218d06
Consolidating EventStorageBackend (#10224)
* Switching to composed anorm queries
* Switching to data-driven prepared statements where possible

changelog_begin
changelog_end
2021-07-12 19:55:52 +02:00
tudor-da
3fd3abfb26
Instrument buffer after contract state events stream (#10209)
CHANGELOG_BEGIN
CHANGELOG_END
2021-07-12 19:37:32 +02:00
Miklos
643a2de006
v2 participant state API draft [KVL-998] (#10210)
CHANGELOG_BEGIN
CHANGELOG_END
2021-07-12 18:11:52 +02:00
Marton Nagy
5abcd0482e
Adds Unit tests for parallel-ingestion [DPP-455] (#10238)
changelog_begin
changelog_end
2021-07-12 14:46:12 +00:00
Hubert Slojewski
3cdedcf885
kvutils: Extract validators from TransactionCommitter [KVL-1015] (#10235)
CHANGELOG_BEGIN
CHANGELOG_END
2021-07-12 15:04:18 +02:00
Sofia Faro
00d622f268
Make @UNTIL-LF exclusive, add @UNTIL-LF-FEATURE. (#10240)
In the damlc integration tests.

changelog_begin
changelog_end
2021-07-12 10:46:36 +00:00
Gary Verhaegen
2bcbd4e177
es: switch to persistent nodes (#10236)
A few small tweaks, but the most important change is giving up on
preemptible instances (for now at least), because I saw GCP kill 8 out
of the 10 nodes at exactly the same time and I can't really expect the
cluster to sruvive that.

CHANGELOG_BEGIN
CHANGELOG_END
2021-07-12 06:27:23 +00:00
azure-pipelines[bot]
7f4bc2a472
update compat versions for 1.14.2 (#10233)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-07-12 08:24:55 +02:00
Remy
4eba4b00e8
Ledger: decouple BD value serialization library from archive library. (#10218)
The DB serialization library should no use damlLfCodedInputStream from
(archive decoding) to decode values.

CHANGELOG_BEGIN
CHANGELOG_END
2021-07-09 15:42:13 +02:00
Stefano Baghino
729afa8c7b
Write a few lines on how to run tests against an Oracle database (#10230)
changelog_begin
changelog_end
2021-07-09 14:14:03 +02:00
Gary Verhaegen
2b67ebb5d4
tf: refactor appr var (#10232)
Two changes at the Terraform level, both with no impact on the actual
GCP state:

- There is no reason to make this value a `variable`: variables in
  Terraforma are meant to be supplied at the CLI. `local` is the right
  abstraction here (i.e. set in the file directly).
- Using an unordered `for_each` set rather than a list so we don't have
  positional identity, meaning when adding someone at the top we don't
  need to destroy and recreate everyone else.

CHANGELOG_BEGIN
CHANGELOG_END
2021-07-09 13:41:46 +02:00
Gary Verhaegen
202b7f7ae7
add akshay to appr team (#10229)
CHANGELOG_BEGIN
CHANGELOG_END
2021-07-09 10:55:16 +00:00
Moritz Kiefer
2c5410c229
Release SDK 1.14.2 (#10226)
changelog_begin
changelog_end
2021-07-09 12:09:31 +02:00
azure-pipelines[bot]
89d87ad17c
update compat versions for 1.14.2-snapshot.20210708.7135.0.aa297840 (#10223)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-07-09 11:21:49 +02:00
Marton Nagy
48393444df
TODO Cleanup: QueryNonPruned [DPP-459] (#10225)
changelog_begin
changelog_end
2021-07-09 09:20:56 +00:00
tudor-da
0bea5e3ef7
Allow retriable lookupMaximumLedgerTime on contention (#10211)
CHANGELOG_BEGIN
CHANGELOG_END
2021-07-09 09:30:05 +02:00
Moritz Kiefer
ca2a9e9bbb
Release 1.14.2 snapshot (#10222)
changelog_begin
changelog_end
2021-07-08 19:48:45 +00:00
Gary Verhaegen
999577a1a7
tweak ES cluster (#10219)
This PR contains many small changes:

- A small refactoring whereby the "es-init" machine is now
  (syntactically) integrated with the two instance groups, to cut down a
  bit on repetition.
- The feeder machine is now preemptible, because I've seen it recover
  enough times that I'm confident this will not cause any issue.
- Indices are now sharded.
- Return values from ES are filtered, cutting down a bit on network
  usage and memory requirements to produce the responses.
- Bulk uploads for a single job are now done in parallel. This results
  in about a 2x speedup for ingestion.
- crontab was changed to very minute instead of every 5 minutes.

CHANGELOG_BEGIN
CHANGELOG_END
2021-07-08 19:20:35 +02:00
Moritz Kiefer
6680d368a1
Warn on DA.Exception import (#10201)
* Warn on DA.Exception import

We had a few confused users that imported that module and got weird
errors. The warning should hopefully make things a bit clearer. We
could do this for other modules as well but I’ll leave that for
separate PRs.

changelog_begin

- [Daml Compiler] Importing DA.Exception on LF < 1.14 now produces a
warning that exceptions require Daml-LF >= 1.14.

changelog_end

* Add a test for exception imports

changelog_begin
changelog_end
2021-07-08 16:51:19 +00:00
Remy
f19f5b0821
LF: Simplify DarReader (#10217)
This PR simplifies a bit DarReader and UniversalDarReader

This is a follow up of #10208.

CHANGELOG_BEGIN
CHANGELOG_END
2021-07-08 17:03:45 +02:00
tudor-da
8f1a8f2390
Release DAML SDK 1.14.1 (#10204)
* Release DAML SDK 1.14.1

CHANGELOG_BEGIN
CHANGELOG_END

* Update SHA for 1.14.1 that contains a fix for artifacts publishing
* Included change https://github.com/digital-asset/daml/pull/10212
2021-07-08 15:08:43 +02:00
Remy
ebb76dca4c
LF: reorganize errors in com.daml.lf.archive (#10213)
* LF: reorganize errors in com.daml.lf.archive

part of #9974

CHANGELOG_BEGIN
CHANGELOG_END

* fix
2021-07-08 13:03:42 +00:00
nickchapman-da
69646451f2
Improvements for validation tests (#10214)
* prefer Seq to List when generating samples

CHANGELOG_BEGIN
CHANGELOG_END

* use TableDrivenPropertyChecks

* remove count/println
2021-07-08 13:48:07 +01:00
Moritz Kiefer
5f124c3b64
Avoid collision in execution log postfix (#10205)
uname is the name for Linux and Linux_scala_2_12 which causes builds
to override each other and it looks like that might even break in case
of concurrent uploads although that could also be general flakiness in Azure.

changelog_begin
changelog_end
2021-07-08 11:23:53 +00:00
Moritz Kiefer
9e27ae0d85
Reshuffle cocreature in daml-lf codeowners (#10215)
No idea why I’m a codeowner for everything individually.

changelog_begin
changelog_end
2021-07-08 11:06:20 +00:00
Remy
41b8448b17
LF: Simplify archive reader. (#10208)
* LF: Simplify archive reader.

- decouple Reader and Decoder
- introduce case class to handle hash, proto payload, and version

CHANGELOG_BEGIN
CHANGELOG_END

* Address Moritz' review

* cosmetic
2021-07-07 19:56:40 +00:00
Gary Verhaegen
38734f02d7
es-feed: ignore invalid files (#10207)
We currently have about 1% (28 out of 2756) of our build logs that have
invalid JSON files. They are all about a `-profile` file being
incomplete, and since those files represent a single JSON object we
can't do smarter things like filtering invalid individual lines.

I haven't looked deeply into _why_ we create invalid files, but this
should let our ingestion process make some progress in the meantime.

CHANGELOG_BEGIN
CHANGELOG_END
2021-07-07 15:38:14 +00:00
Moritz Kiefer
6586dde11e
Move exercise context computation to the client side (#10199)
We probably want to start displaying the full ptx on the client side
but for now this at least moves things out of speedy and into the
rendering layer where they belong.

It also fixes IncompleteTransaction to unwind properly so the roots
are really the roots.

changelog_begin
changelog_end
2021-07-07 15:44:47 +02:00
Moritz Kiefer
c92c67832f
Mark Daml profiler as stable (#10203)
Nobody reported any issues and we’re not planning breaking changes so
ship it!

changelog_begin

- [Daml Profiler] The Daml profiler is now a stable feature.

changelog_end
2021-07-07 12:54:18 +00:00
Gary Verhaegen
cb1f4ec773
ci/windows: disable spool (#10200)
* ci/windows: disable spool

We're not expecting to print anything, and @RPS5' security newsletter
says this is a vector of attack.

CHANGELOG_BEGIN
CHANGELOG_END

* increase no-spool to 6

* Windows name truncation causing collisions

* update main group

* remove temp group
2021-07-07 12:44:33 +00:00
Moritz Kiefer
2326d425bc
Publish execution logs on unix platforms (#10194)
Even with the cache retries something still doesn’t seem to be cached
quite like I expect. I can’t really debug this without exec logs so
this PR starts publishing those.

changelog_begin
changelog_end
2021-07-07 11:37:43 +02:00
Moritz Kiefer
42693452ce
Drop cleanup of node_modules (#10195)
This is months old by now and I think it might be non-deterministic
forcing unnecessary rebuilds.

changelog_begin
changelog_end
2021-07-07 10:15:44 +02:00
Stefano Baghino
c8faed8a17
Deduplicate Java codegen running instructions (#10185)
Fixes #10183.

changelog_begin
changelog_end
2021-07-07 08:37:42 +02:00
azure-pipelines[bot]
e4c8c390f2
update compat versions for 1.14.1-snapshot.20210706.7133.0.8ec16e94 (#10196)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-07-07 08:34:25 +02:00