Commit Graph

235 Commits

Author SHA1 Message Date
Rohan Jacob-Rao
bdaf9aa216
Trigger service: restart triggers in database on server startup (#6556)
* Add optional trigger id to startTrigger

CHANGELOG_BEGIN
CHANGELOG_END

* Read running trigger table in DbTriggerDao

And get rid of traverse type application!
(since we have partial unification on)

* Restart triggers in database

* Test for trigger restart on recovery

* Add order by clause and remove accidental overloading
2020-07-01 13:22:23 +00:00
Rohan Jacob-Rao
546c75b74f
Trigger service: Remove data in messages from TriggerRunnerImpl to Server (#6554)
* Remove data from message sent from TriggerRunnerImpl to Server

* Remove unused trigger name from config

changelog_begin
changelog_end
2020-06-30 21:02:10 -04:00
Rohan Jacob-Rao
ea16ff350d
Trigger service: Change to restart strategy with backoff (#6552)
* Add min/maxRestartInterval to configs and use in trigger restart strategy

* Adapt tests with triggers failing due to lack of network

changelog_begin
changelog_end

* Adapt tests for triggers with errors

* Remove comment about number of restarts

* Use a small initial restart interval for testing

* Remove old restart params

* Move maxInboundMessageSize to LedgerConfig

* Rename TriggerRunnerConfig to TriggerRestartConfig
2020-06-30 20:46:23 -04:00
Rohan Jacob-Rao
99a9662781
Trigger service: change convention for updating running trigger store (#6532)
* Don't modify running trigger store via the trigger runner actor

changelog_begin
changelog_end

* Remove runner actor from RunningTrigger type

And parent actor from TriggerRunnerImpl

* Add to running trigger store on start request

* Update tests for new trigger store convention

* Fix test with network outage
2020-06-30 12:00:45 -04:00
Rohan Jacob-Rao
9c4daad473
Trigger service nits (#6528)
- Move Stop Message back to original place with other Server state messages
- Other style nits

changelog_begin
changelog_end
2020-06-29 13:25:35 -04:00
Shayne Fletcher
44bb7f5233
Spawn a ref ledger auth instance from text fixture (#6502)
changelog_begin
changelog_end
2020-06-25 15:53:34 -04:00
Rohan Jacob-Rao
674fecf73f
Trigger service: split up ServiceMain into multiple files (#6498)
* Move top level case classes to package object

changelog_begin
changelog_end

* Move Server Message classes to separate file

* Server in Server.scala and ServiceMain in ServiceMain.scala

* Copyright headers
2020-06-25 16:01:35 +00:00
Rohan Jacob-Rao
eb861ffd4a
Trigger service: split up Server companion object (#6491)
The Server object and especially the apply method is extremely big and
hard to follow. There are a great many functions nested within the apply
method. This PR attempt to organize things, mostly by moving local
functions into methods of the Server class. I think this makes things
easier to follow, and I think it's more conventional.

Note that with this change I added some implicit parameters to the
Server constructor, which I think makes sense for these kinds of values
(contexts, sequencer factory, etc.) I also moved the Message trait to
the top level, but we will probably it its own file, following this PR.
There are a few other shallow tweaks that I made along the way.

Open to feedback on the high level structuring, as I'm not super
familiar with idiomatic design in Scala.

changelog_begin
changelog_end
2020-06-25 10:22:53 -04:00
Shayne Fletcher
2f539c477a
exe lookup in runfiles (#6489)
changelog_begin
changelog_end
2020-06-25 09:48:55 -04:00
Stephen Compall
a51d0db8ff
set scalac -Xsource:2.13 -Ypartial-unification globally (#6469)
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts

* add now-referenced scalaz-core where needed

* work around bad type signatures in scalatest Aggregating, Containing

* unused Any suppression

* work around bad partial-unification wrought by type alias

* remove unused Conversions import

- not required in 4f68cfc480 either, so unsure how it's survived this long

* work around Future.traverse; remove unused show import

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused bounds

* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed

* longer comment on what the options do

- suggested by @stefanobaghino-da; thanks

* forget Future.traverse, just use scalaz, it knows how to do this
2020-06-24 16:51:24 -04:00
Shayne Fletcher
afc573d422
step 0 integration of ref-ledger-authentication (#6484)
changelog_begin
changelog_end
2020-06-24 13:40:03 -04:00
Rohan Jacob-Rao
10208fd194
Trigger service: read packages from database on startup (#6470)
* Move database initialization to Server apply method
* Reorder dao/server creation
* Read packages from database on startup
* Test starting a trigger after a shutdown
changelog_begin
changelog_end
2020-06-24 11:18:16 -04:00
Rohan Jacob-Rao
b10bfa1ba7
Trigger service: Write packages to database if we have one (#6424)
This is needed to recover state after the service shuts down or crashes.
We add a method to the RunningTriggerDao to persistPackages. This only
does something in the case of a DbTriggerDao. In any case the Server
keeps a package map in memory as it's required to construct a trigger runner.
Uploads of existing packages is considered harmless.

changelog_begin
changelog_end
2020-06-19 13:49:49 -04:00
Rohan Jacob-Rao
78d598ecf3
Trigger service: Improve formation of SQL queries (#6422)
Avoid the use of Fragment.const which interprets raw strings as SQL
queries without any checks. Use the `sql` string interpolation which I
found out does the right thing with Strings and other simple types.

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-19 09:32:27 -04:00
Rohan Jacob-Rao
0ca1288d9d
Trigger service: Increase server binding timeout (#6421)
To avoid timeouts in CI. Also factor out the implicit val one level
so it is used by both tests and main method.

changelog_begin
changelog_end
2020-06-18 19:54:24 +00:00
Rohan Jacob-Rao
5934b22dda
Trigger service: Refactor Server.addDar to take encoded dar (#6395)
Trigger service: Refactor Server.addDar to take encoded dar

This is so we can write the encoded packages to the database if we have
one (without re-encoding them).

changelog_begin
changelog_end
2020-06-18 12:37:43 -04:00
Rohan Jacob-Rao
5ca65ad69a
Trigger service: Refactor testing with and without database (#6373)
* Trigger service: Refactor testing with and without database

Following the approach taken in the HTTP JSON API.
Define an AbstractTriggerServiceTest class with abstract jdbcConfig
member. This is instantiated with None and Some in the in-memory and
database test classes respectively. Tests for the individual modes can
be added in these concrete classes.

changelog_begin
changelog_end

* Move db init and destroy into beforeEach/afterEach to avoid concurrent tests clashing
2020-06-16 18:00:20 -04:00
Rohan Jacob-Rao
3cfed4d7a3
Trigger Service: Refactor running trigger access using DAO interface (#6334)
Trigger Service: Refactor running trigger access using DAO interface

This introduces a RunningTriggerDao trait and two classes that implement
it, one for in-memory and one for database storage. This moves a lot of
code out of the Server class and into the DAO classes. The class chosen
at runtime depends on whether a JDBC config has been provided.

The database DAO also has methods for initialization and clean up used by
the server main function and the test fixture.

This is a pure refactoring (modulo some minor error message changes) so
functionality should stay the same.

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-13 14:33:46 -04:00
Shayne Fletcher
23e7dc826e
Trigger service docs and SDK integration basics (#6329)
* Integrate trigger-service into the assistant

changelog_begin
changelog_end

* Assistant integration and bare bones docs

changelog_begin
changelog_end
2020-06-12 18:54:38 +00:00
Simon Maxen
5329a4f97f
Ledger id requirement (#6323)
* Add option based constructor for LedgerIdRequirement

changelog_begin
changelog_end

* Make option based consructor the default, deprecate old constructor

* Update with review comments
2020-06-12 15:00:32 +00:00
Shayne Fletcher
0f47b8d1bf
Simplify findCredentials logic (#6301)
changelog_begin
changelog_end
2020-06-11 23:04:14 +00:00
Shayne Fletcher
b8126ee75c
Add --no-secret-key (#6309)
changelog_begin
changelog_end
2020-06-11 18:41:26 -04:00
Shayne Fletcher
07f24c0d93
Encrypt credentials (#6296)
* Encrypt credentials

changelog_begin
changelog_end

* Move key aquisition and write a big fat warning

* Only write encrypted token to DB.
2020-06-10 17:14:29 -04:00
Shayne Fletcher
033c58bc60
Switch to basic auth in readiness for auth service integration (#6279)
* Switch to basic auth in readiness for auth service integration

changelog_begin
changelog_end

* Newtype for user credentials
2020-06-10 09:48:33 -04:00
Rohan Jacob-Rao
e7017e6943
Trigger service: delete from database in stop trigger (#6249)
Includes a fair amount of refactoring to smooth the logic with/without a database. All tests now run with and without a database.

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-08 16:19:35 -04:00
Shayne Fletcher
ecad4ff7c3
Fix test descriptions (#6269)
changelog_begin
changelog_end
2020-06-08 19:11:59 +00:00
Shayne Fletcher
0b7989a715
Adjust status code response for unauthorized (#6268)
changelog_begin
changelog_end
2020-06-08 18:56:43 +00:00
Shayne Fletcher
0625e68988
Disambiguate initialization halted vs. failed (#6248)
changelog_begin
changelog_end
2020-06-05 17:18:38 -04:00
Rohan Jacob-Rao
3159c617f6
Tweak error handling in listTriggers helper (#6247)
changelog_begin
changelog_end
2020-06-05 20:32:49 +00:00
Rohan Jacob-Rao
de2b5d195e
Trigger service style touch ups (#6246)
* Touch up logTriggerStatus

* Touch up removeRunningTrigger

* Touch up addRunningTrigger

* Remove IntelliJ (scalastyle I think) warnings about public members without type annotations

CHANGELOG_BEGIN
CHANGELOG_END

* Redundant brackets
2020-06-05 16:07:10 -04:00
Shayne Fletcher
71ff65d5be
Trigger stop on db write failure (#6233)
* Stop running trigger under DB write failure

changelog_begin
changelog_end

* Another fix relating to initialization retries

changelog_begin
changelog_end

* Handle the error case directly
2020-06-04 17:00:12 -04:00
Shayne Fletcher
e23a488596
Put initialization failures through retries too (#6230)
changelog_begin
changelog_end
2020-06-04 15:24:51 -04:00
Shayne Fletcher
322b93c7e7
Temporarily disable flaky test (#6228)
changelog_begin
changelog_end
2020-06-04 12:04:35 -04:00
Shayne Fletcher
2c3efc6c4f
Refactor startTrigger (#6219)
changelog_begin
changelog_end
2020-06-04 09:30:47 -04:00
Rohan Jacob-Rao
09254a0fa8
Trigger service: streamline running tests with and without the database (#6205)
This adds a function withTriggerServiceAndDb which runs a test twice, once with and once without a database, and succeeds if both succeed. This will be useful for reusing test logic with both backends and making sure behaviour is consistent. I have used this function where possible, but it won't work for everything until stop is implemented on the DB side.

At the moment this new function squashes two tests into one making it hard to tell whether it failed with or without the database. In a future PR I will investigate using an abstract class to run the tests separately (hopefully with altered descriptions).

This feature required a few changes in the process, mainly:
- Use PostgresAroundAll to connect/disconnect to the database before and after all tests run
- Add a destroy method to the TriggerDao to reset the database between tests
- Use the TriggerDao in the withTriggerService functions to initialize / clean up the database at the start / end of each test
- Sort trigger instances from list using Scala's sort, not relying on Postgres' ordering of UUIDs. This also means we need to use UUIDs for trigger instances in the tests and sort nonempty vectors in expected results.
2020-06-04 09:30:26 -04:00
Shayne Fletcher
c2961b1957
Make the stop endpoint more robust and test (#6217)
* Make the stop endpoint more robust and test

changelog_begin
changelog_end

* Stopping an unknown trigger gives 404
2020-06-03 18:56:51 +00:00
Shayne Fletcher
6be68dc059
Renfinements to error test DAMLs (#6212)
changelog_begin
changelog_end
2020-06-03 15:02:53 +00:00
Shayne Fletcher
5ff8ec69df
Trigger error cases (#6202)
changelog_begin
changelog_end
2020-06-02 19:29:43 -04:00
Rohan Jacob-Rao
0b5762e437
Trigger service: use DB for start and list commands (#6153)
* Insert running trigger to DB when using one

If the DB write fails, the server sends itself a
TriggerInitializationFailure message so that the corresponding trigger
runner is stopped and the table is in sync with the actors.
We still need to retry writes here.

Includes basic test that runs the server with a JDBC config set and adds
a trigger, expecting a new entry to be added to the DB. However does not
check the running trigger table which we can do once reads are
implemented.

changelog_begin
changelog_end

* Await on future in test

* Update to new assertTriggerIds

* Apply scalafmt suggestions

* Create index on party token

* Read db in list command

* Update comment in test script

* Remove outdated comment

* Fix strings in insert and select

* Clean up test

* Add a second trigger in the db test

* Fix comment in test script

* Comment db tables

* Order trigger instances in list command

* Comment about TriggerDao execution context
2020-06-02 16:57:35 +00:00
Shayne Fletcher
256d48a431
Switch to timestamps not strings (#6191)
changelog_begin
changelog_end
2020-06-02 15:24:57 +00:00
Shayne Fletcher
2de23d93c2
Trigger record status (#6180)
* Store trigger history

changelog_begin
changelog_end

* Harvest trigger histories

changelog_begin
changelog_end

* Switch to Vector over List (and other bits and bobs)

* Use a better verb for updating trigger status method

* Add a comment

* Fix mangled comments
2020-06-01 20:19:51 +00:00
Shayne Fletcher
41c967f1f1
Failure path testing (#6178)
changelog_begin
changelog_end
2020-06-01 12:50:51 -04:00
Shayne Fletcher
f619dea380
Test trigger won't start when ledger connection not viable (#6150)
changelog_begin
changelog_end
2020-05-28 19:05:14 -04:00
Rohan Jacob-Rao
cc5d5d3fd8
Trigger service: refactor in prep for DB writes (#6134)
* Pass JdbcConfig object to TriggerDao apply

* No need to return TriggerDao from init db

* Refactor introducing RunningTrigger type

* Rename triggerId -> triggerInstance and triggerOrigId -> triggerName
Note this also changes the start request parameter name to triggerName.
However I have not yet renamed triggerId in the response messages. We
should probably make it triggerInstance there too but in a later PR.

changelog_begin
changelog_end
2020-05-28 13:47:24 -04:00
Stephen Compall
15350a7bc2
disable warts.Any and remove most suppressions (#6132)
* disable Any wart

* first pass removal of Any suppressions for false positives

* second pass removal of Any suppressions for false positives

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* third pass removal of Any suppressions for false positives

* fourth pass removal of Any suppressions for false positives

* reformat newly single-suppressions into single lines

- suggested by @SamirTalwar-DA; thanks
2020-05-28 16:53:41 +00:00
Shayne Fletcher
4fb13734ad
Extend proxy into test functions and gentle refactor (#6147)
changelog_begin
changelog_end
2020-05-28 12:39:34 -04:00
Shayne Fletcher
576e47f587
Toxify Windows (#6122)
changelog_begin
changelog_end
2020-05-27 12:07:54 -04:00
Shayne Fletcher
79563ee74b
ToxiProxy config (#6114)
changelog_begin
changelog_end
2020-05-27 07:08:45 -04:00
Shayne Fletcher
b95dd39a6e
Don't update running triggers until we know the trigger is running (#6079)
* Don't update running triggers until we know the trigger is running

changelog_begin
changelog_end

* Don't update running triggers until we know the trigger is running
2020-05-26 16:15:29 -04:00
Stefano Baghino
9af85e56e9
Introduce DAML-LF value caching for transaction service (#6052)
* Introduce DAML-LF value caching for transaction service

Allows to keep the DAML-LF values in the most recently indexed events in memory,
so that they don't have to be deserialized on the client from their Protobuf encoding.

Closes #6044

CHANGELOG_BEGIN
[Sandbox] The --max-lf-value-translation-cache-entries option allows to set a
number of events for which DAML-LF values are cached. Could reduce latency in
serving transactions for consumers that are reasonably fast.
CHANGELOG_END

* Add missing dependency

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428076003

* Update ledger/sandbox/src/main/scala/com/digitalasset/platform/sandboxnext/Runner.scala

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428071324

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428076905

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428081294

* Fix fatal warnings

* //ledger/caching has to be used whenever sandbox is run

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-05-26 08:33:53 +00:00
Rohan Jacob-Rao
f845bca12f
Trigger service: initialize database command (#6062)
Minimal database initialization with schemas for running_triggers and dalfs tables. The user passes in the database URL, username and password in a config string argument (approach and code adapted from the JSON API).

In future the idea is to also create a "service" role with permissions to read and write to the new tables. Then the user can pass in the service role to connect to the database when running the service for real.
2020-05-22 10:54:08 -04:00
Shayne Fletcher
90c0194dd9
Change trigger maps to store tokens not parties (#6061)
changelog_begin
changelog_end
2020-05-20 16:54:13 -04:00
Shayne Fletcher
639bec5a83
New module TokenManagement.scala (#6060)
changelog_begin
changelog_end
2020-05-20 16:51:40 -04:00
Shayne Fletcher
03d8425e16
Rework start/stop & list to require bearer header with JWT token (#6055)
changelog_begin
changelog_end
2020-05-20 15:40:22 -04:00
Shayne Fletcher
0e1ac4670c
Add maxInboundMessageSize param to trigger service (#6000)
changelog_begin
- trigger runner supports the `max-inbound-message-size` option
changelog_end
2020-05-15 13:52:59 +00:00
Shayne Fletcher
2b33a4d273
Version the existing paths (#5970)
changelog_begin
changelog_end
2020-05-13 16:09:41 -04:00
Shayne Fletcher
007ebeb56a
Add http-port option (#5951)
changelog_begin
changelog_end
2020-05-13 09:30:03 -04:00
Shayne Fletcher
221c6db0ae
Rename targets (#5950)
changelog_begin
changelog_end
2020-05-12 16:00:15 -04:00
Shayne Fletcher
6cbe36bf4d
Shutdown protocol (#5936)
changelog_begin
changelog_end
2020-05-12 08:33:05 -04:00
Shayne Fletcher
1007df46d2
script to start ledger and trigger service (#5939)
changelog_begin
changelog_end
2020-05-12 08:32:34 -04:00
Shayne Fletcher
75ed957c63
Restart triggers that fail (#5916)
changelog_begin
changelog_end
2020-05-08 15:23:54 -04:00
Rohan Jacob-Rao
4797834e1d
Trigger service: use "triggerId" and "triggerIds" fields for response messages (#5854)
* Use triggerId field in trigger start response

* Use triggerId field for stop trigger result

* Fix indentation and make yields consistent

* Use pair constructor for JsObject instead of Map

* Use triggerIds field in list triggers response

changelog_begin
changelog_end
2020-05-06 11:44:07 -04:00
Rohan Jacob-Rao
ac5f462f24
Trigger service: Use JSON object in result of upload dar response (#5849)
changelog_begin
changelog_end
2020-05-05 16:33:45 +00:00
Rohan Jacob-Rao
30fab0cbb9
Trigger service: Use response format from JSON API (#5829)
* Adapt ResponseFormat from JSON API

* Add some type annotations

* Use response format with status and errors/result fields

* Update and refactor tests

changelog_begin
changelog_end
2020-05-05 09:02:57 -04:00
Rohan Jacob-Rao
5a09a75859
Trigger service: Use tagged Party type instead of String (#5816)
changelog_begin
changelog_end
2020-05-01 18:08:06 +00:00
Rohan Jacob-Rao
947e6e5ebc
trigger service: Move request params to separate file (#5808)
* Move request types to new file
* Rename TriggerParams to StartParams and a bit more formatting
changelog_begin
changelog_end
2020-05-01 13:00:54 -04:00
Shayne Fletcher
e7741dd850
handle stop message received before acs query complete (#5801)
changelog_begin
changelog_end
2020-05-01 11:24:58 -04:00
Shayne Fletcher
1510d8ccd9
Rewrite acsQuery as a for-comprehension (#5803)
changelog_begin
changelog_end
2020-04-30 13:08:47 -04:00
Rohan Jacob-Rao
af8895b487
Trigger service: endpoint to list running triggers for a party (#5770)
* Endpoint to list all triggers (not yet by party)

* Clean up test code a little

changelog_begin
changelog_end

* Test for listing running triggers

* Respond with JSON list instead of random text

* List triggers by party

Pass party name in request body.
Store another map of party name to set of trigger ids.
Also store party names in the values of the original trigger id map, so
we can update the party map when stopping a trigger.
2020-04-29 14:43:56 -04:00
Gerolf Seitz
a2d785e3ee
Use com.daml as root package (#5343)
Packages com.digitalasset.daml and com.daml have been unified under com.daml

Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.

CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
2020-04-05 19:49:57 +02:00
Gary Verhaegen
1872c668a5
replace DAML Authors with DA in copyright headers (#5228)
Change requested by Manoj.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 01:26:10 +01:00
Robert Autenrieth
a3ddde3a9d
Remove MRT and LET (#5172)
Contributes to #4194.
Closes #4231.
Closes #5022.
CHANGELOG_BEGIN
- [Ledger API] The protobuf fields ledger_effective_time and maximum_record_time have been removed from
  command submission. These fields were previously deprecated following the introduction
  of a new ledger time model. See issue `#4194 <https://github.com/digital-asset/daml/issues/4194>`__.
[Java Bindings] removed the usage of ledgerEffectiveTime and
maximumRecordTime, and instead added minLedgerTimeAbsolute and
minLedgerTimeRelative in CommandSubmissionClient and CommandClient
CHANGELOG_END
2020-03-25 16:50:27 +01:00
Robert Autenrieth
3f597aae16
New ledger time (#5100)
* Tighten result type

Command execution can't result in a sequencer error

* New helper method for extracting used contracts

* New error clause

* Add a DAO query for the maximum time of contracts

* Implement algorithm for finding ledger time

CHANGELOG_BEGIN
CHANGELOG_END

* fixup ledgerTimeHelper

* Use new ledger time algorithm

* Mark LET/MRT as deprecated

CHANGELOG_BEGIN
- [Ledger API] DAML ledgers have switched to a new ledger time model.
  The ledger_effective_time and maximum_record_time fields of command submission are deprecated,
  the ledger time of transactions is instead set automatically by the ledger API server.
  Ledger time is no longer strictly monotonically increasing, but only follows causal monotonicity:
  ledger time of transactions is greater than or equal to the ledger time of any used contract.
  See `#4345 <https://github.com/digital-asset/daml/issues/4345>`__.
CHANGELOG_END

* Add ledger time skew check

* Remove command updater

LET/MRT are now deprecated, this class is now useless

* Remove old time model validator

* Switch to new time model check: kvutils

* Switch to new time model check: in-memory ledger

* Switch to new time model check: SqlLedger

* Use initial ledger config

* Ignore user provided LET

* Use TimeProvider in submission services

* Use deduplication_time in daml-script runner

- Also remove unnecessary command completion output of CommandTracker.
- Remove usage of maximum record time in CommandTracker.

* Use arbitrary default value for deduplication time

* Use built-in Instant ordering

* Remove obsolete test

* Remove obsolete test: CommandStaticTimeIT

* Refactor test: TransactionMRTCompliance

* Disable test: CommandTrackerFlow timeout

* thread maxDeduplicationTime through to CommandTracker

* Improve test

* Refactor command client configuration

* Deduplication time should always use UTC

* Add missing method in TimedIndexService after rebase

* Put more details into the deduplication error response.

* Use system time for command dedup submittedAt.

* Use explicit UTC time source in command validator

* Revert CommandTracker[Flow] to previous completion-recovering-behavior

* Adapt scala client command config to new config params

Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2020-03-25 09:28:56 +01:00
Moritz Kiefer
13816c75b3
Support uploading DARs to the trigger service (#5084)
* Support uploading DARs to the trigger service

This PR adds a new `upload_dar` endpoint that accepts a DAR as a
multi-part form request and adds it to the list of compiled packages.

I’ve also made the DAR passed in on startup optional now given the new
endpoint.

There is no endpoint for deleting a DAR so far but there is none on
the ledger API either so I think this not particularly urgent.

changelog_begin
changelog_end

* Address review comments
2020-03-19 13:02:04 +01:00
Moritz Kiefer
bd6db5be68
Handle failures during trigger initialization synchronously (#5046)
Previously the http endpoint for starting a trigger would always
return immediately. Based on the recent refactorings, we now do the
non-IO trigger initialization synchronously and return a failed http
status code with an error message.

This also refactors the code to only have one (mutable) set of
compiled packages which is a prerequisite for dynamic package uploads.

changelog_begin
changelog_end
2020-03-18 10:23:41 +01:00
Moritz Kiefer
fb86152f8c
Move non-IO initialization out of trigger runner (#5042)
Previously parts of the initialization, in particular, the code for
finding the filter and the heartbeat were part of the Runner. This led
to an akward API and didn’t really make any sense.

Now all of this code is part of a pure `Trigger.fromIdentifier`
method and the runner only takes care of actually running the
ledger. This could also be useful for the trigger service where we
might want to synchronously call `getIdentifier` so users get some
indication if there request even points to a valid trigger
directly. However, this is not tackled by this PR.

changelog_begin
changelog_end
2020-03-17 15:26:58 +01:00
Moritz Kiefer
cc2ca20336
Make trigger runner specific to a given trigger (#5038)
Previously the runner class was in a weird state where it was specific
to a DAR but not to an individual trigger. This meant that you had to
pass around a fair bit of state which got a bit awkward. This PR
addresses this by making the trigger class specific to the trigger.

It also now accepts `CompiledPackages` instead of a DAR which should
make it easier in the trigger service to support dynamic package
uploads.

changelog_begin
changelog_end
2020-03-17 11:51:06 +01:00
Moritz Kiefer
9aa68cac4f
Fix detection of package ids for triggers (#5016)
Previously we assumed that the module name was globally unique in the
DAR which is definitely not guaranteed. Now we instead detect the
package id of the trigger library based on the type of the trigger we
are running which doesn’t fall apart if there are multiple versions of
the trigger library.

I’ve also removed the check for the package id of the trigger library
since I’d like the trigger runner to be backwarts compatible from now on (we
didn’t break that in a while).

This is slightly ugly since the Runner class is currently not specific
to a single trigger but only the individual methods are aware of the
specific trigger identifier. I’ll refactor this in a separate PR.

changelog_begin
changelog_end
2020-03-16 16:13:42 +01:00
Samir Talwar
a99156252d
libs-scala/ports: Wrap socket ports in a type, Port. (#4784)
* libs-scala/ports: Wrap socket ports in a type, `Port`.

* sandbox: Use `Port` for the API server port, and propagate.

CHANGELOG_BEGIN
CHANGELOG_END

* extractor: Use `Port` for the server port.

* ports: Make Port a compile-time class only.

* ports: Allow port 0; it can be specified by a user.

* ports: Publish to Maven Central.
2020-03-03 08:59:15 +00:00
Shayne Fletcher
66dd112960
Remove pragma 'daml 1.2' (#4702)
changelog_begin
- The pragma 'daml 1.2' is now optional.
changelog_end
2020-02-26 13:17:45 -05:00
Gary Verhaegen
5a117dc358
introduce new release process (#4513)
Context
=======

After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.

This PR is a proposal for a new release process designed under the
following constraints:

- Reuse as much as possible of the existing infrastructure, to minimize
  effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
  not meant for general public consumption, but can still be used by savvy
  users without jumping through too many extra hoops (ideally just
  swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
  release status, with as few changes as possible in-between, so we can be
  confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
  releases, to avoid discovering non-transient problems while trying to
  promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
  keep the same approval process for SOC2 auditability.

The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:

```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```

This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.

Version numbers
===============

Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:

- Semantic versioning (which we chose as the version format for the SDK
  version number) allows for "prerelease" version numbers as well as
  "metadata"; an example of a complete version string would be
  `1.2.3-nightly.201+server12.43`. The "main" part of the version string
  always has to have 3 numbers separated by dots; the "prerelease"
  (after the `-` but before the `+`) and the "metadata" (after the `+`)
  parts are optional and, if present, must consist of one or more segments
  separated by dots, where a segment can be either a number or an
  alphanumeric string. In terms of ordering, metadata is irrelevant and
  any version with a prerelease string is before the corresponding "main"
  version string alone. Amongst prereleases, segments are compared in
  order with purely numeric ones compared as numbers and mixed ones
  compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
  which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
  `-`, or a transition between a number and a letter. Version strings
  are compared element-wise, with numeric segments being compared as
  numbers. Alphabetic segments are treated specially if they happen to be
  one of a handful of magic words (such as "alpha", "beta" or "snapshot"
  for example) which count as "qualifiers"; a version string with a
  qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
  which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
  special ordering amongst qualifiers. Other alphabetic segments are
  compared alphabetically and count as being "after" their prefix
  (`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
  separated by `.`, plus an optional (though deprecated) alphanumeric
  "version tag" separated by a `-`. I could not find any official
  documentation on ordering for the version tag; numeric segments are
  compared as numbers.
- npm uses semantic versioning so that is covered already.

After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:

```
0.13.53-alpha.20200214.3025.ef5d32b7
```

where the components are, respectively:

- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
  master.
- `3025`: the number of commits in master up to the release commit
  (included). Because we have a linear, append-only master branch, this
  uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
  not strictly speaking necessary, but makes it a lot more convenient to
  identify the commit.

The main downsides of this format are:

1. It is not a valid format for GHC packages. We do not publish GHC
  packages from the SDK (so far we have instead opted to release our
  Haskell code as separate packages entirely), so this should not be an
  issue. However, our SDK version currently leaks to `ghc-pkg` as the
  version string for the stdlib (and prim) packages. This PR addresses
  that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
  would see the above version number as `0.13.53.20200214.3025`, which
  should be enough to uniquely identify it. Note that, as far as I could
  find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
  it makes it more scary. However, I have been told that this may be
  long enough to cause issues on Windows by pushing us past the max path
  size limitation of that "OS". I suggest we try it and see what
  happens.

The upsides are:

- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
  they have the same date, allowing us to release same-day patches if
  needed. (Note: that is, commits that were made on the same day; the
  release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
  immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
  GHC).

Alternatives I considered:

- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
  be accepted by all schemes, but will not sort as expected under semantic
  versioning (though Maven will be fine). I have no idea how it will sort
  under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
  This is not valid semantic versioning and is therefore rejected by
  npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
  what is generally done in the Java world, but we then lose track of what
  version is actually in use and I'm concerned about bug reports. This
  would also not let us publish to the main Maven repo (at least not more
  than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
  version formats. However, it would not clearly indicate to humans that
  it is not meant as a stable version, and would sort differently under
  semantic versioning (which counts it as a prerelease, i.e. before
  `0.13.53`) than under maven (which counts it as a patch, so after
  `0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
  number of prereleases in-between `0.13.52` and the next. This is
  currently the fallback plan if Windows path length causes issues. It
  would be less convenient to map releases to commits, but it could still
  be done via querying the history of the `LATEST` file.

Release notes
=============

> Note: We have decided not to have release notes for snapshot releases.

Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.

But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.

The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.

> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.

New release process
===================

Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.

Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:

1. Before running the release step, CI will checkout the commit
  specified in the LATEST file. This requires separating the release
  step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
  that gets the version to build from an environment variable, with a
  default of `0.0.0` to remain consistent with the current `daml-head`
  behaviour.

Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.

The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.

I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-25 17:01:23 +01:00
Samir Talwar
f231ebc1b3
Sandbox-Next: Re-establish static time support. (#4581)
* kvutils: Extract a committer from the uses of `SubmissionValidator`.

This makes the clock injectable too.

* kvutils: Provide logging contexts in the `Runner`.

* sandbox: Remove the `StaticAllowBackwards` time provider type.

It's not used anywhere.

* sandbox: Fix warnings in CliSpec.

* sandbox: Ensure that we cannot specify both static and wall-clock time.

* sandbox-next: Crash if wall clock time is not specified.

* sandbox-next: Document more known issues in the new Sandbox.

* sandbox: Add a Clock (and some tests) to TimeServiceBackend.

* sandbox-next: Support static time.

CHANGELOG_BEGIN
- [Sandbox Next] Re-establish static time mode.
CHANGELOG_END

* ledger-on-(memory|sql): Expect a `() => Instant`, not a `Clock`.
2020-02-18 17:03:15 +00:00
Samir Talwar
a589f4af0b
sandbox: Move more resource acquisition into the owner. (#4501)
* sandbox: Move more resource acquisition into the `owner`.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Reimplement SandboxClientResource as a resources.Resource.

* codegen: Use resources in TestUtil.

* sandbox: Manage PostgreSQL in tests with ResourceOwners.
2020-02-14 13:52:45 +00:00
Miklos
dc0bca5a2f
Fix missing interpolators (#4372)
CHANGELOG_BEGIN
CHANGELOG_END
2020-02-04 17:09:49 +01:00
Moritz Kiefer
6453a73968
Add first prototype of triggers as a service (TaaS) (#4287)
* Add first prototype of triggers as a service (TaaS)

This is an extremely basic version of the trigger as a service thingy.

Right now, it supports spawning triggers and stopping them but nothing
else.

There is a very simple test to check that it’s not completely broken.

changelog_begin
changelog_end

* Apply suggestions from code review

Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>

* remove debugging output

* remove leftover import

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
2020-01-31 12:53:45 +01:00