Commit Graph

293 Commits

Author SHA1 Message Date
Remy
a899e5e4e6
Speedy: Check language version as part of Speedy compilation. (#7440)
Additionnally
+ move allowedLanguageVersion inside compiler config
+ add missing catch error insde
  ConcurrentCompiledPackages#addPackageInternal
+ implement EngineConfig method to easily produce a Compiler.Config

CHANGELOG_BEGIN
CHANGELOG_END
2020-09-22 12:43:48 +02:00
Stephen Compall
a670e68c72
add 'alter' function to DA.Map (#7427)
* add 'alter' function to DA.Map

CHANGELOG_BEGIN
CHANGELOG_END

* swap arguments for alter

- suggested by @cocreature; thanks

* alter moved to stdlib, remove from trigger lib

* use markdown formatting for alter doc

- suggested by @sofiafaro-da; thanks

* fix code block formatting

- suggested by @sofiafaro-da; thanks
2020-09-17 16:46:37 +00:00
Stephen Compall
1926ddbe9d
avoid ACS linear contract ID lookups in Trigger library (#7425)
* add Ord (ContractId a) instance

* change inner list in activeContracts to map

* adapt Internal module to new map

* support rest of Trigger daml lib

* rename adjust to alter, its Data.Map name

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* distribute type arguments for ACS activeContracts

- suggested by @cocreature; thanks

* swap arguments for alter

- suggested by @cocreature; thanks

* reformat groupActiveContracts signature

- suggested by @cocreature; thanks
2020-09-17 15:31:03 +00:00
Moritz Kiefer
b6673b6b4d
Add design doc for authentication in the trigger service (#7394)
* Add design doc for authentication in the trigger service

This is a draft of how I currently imagine authentication in the
trigger service to work. Since the authentication middleware has to be
pluggable in the end anyway we need public documentation in the end
anyway and I find this much easier to manage than a google doc outside
of the repo which just never sees updates.

changelog_begin
changelog_end

* Update triggers/service/authentication.md

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-09-16 15:57:26 +02:00
Stephen Compall
09014dc637
use GenMaps for trigger ACS (#7365)
* use GenMaps for trigger ACS

* Next removed for 1.dev

* temp port trigger test code to 1.dev only

* run trigger tests on 1.dev only

* move pending back to TextMap

* include trigger service in the 1.dev test lineup

- it takes >2min, so shouldn't be permanent

* add Ord TypeRep and Ord TemplateTypeRep when possible

* swap names in Internal to reduce the diff

* try to enable cpp for triggers compilation

$ bazel build //triggers/daml:daml-trigger-1.dev
<snip>
File:     daml/Daml/Trigger/Internal.daml
Hidden:   no
Range:    103:-1-103:-1
Source:   CPP
Severity: DsError
Message:  22 in hpp-0.6.1:Hpp.CmdLine
File:     daml/Daml/Trigger.daml
Hidden:   no
Range:    103:-1-103:-1
Source:   CPP
Severity: DsError
Message:  22 in hpp-0.6.1:Hpp.CmdLine
ERROR: Creation of DAR file failed.
<snip>

* remove problematic options for invoking cpp

hpp: Couldn't open input file: -Werror
CallStack (from HasCallStack):
  error, called at src/Hpp/CmdLine.hs:103:22 in hpp-0.6.1:Hpp.CmdLine

* enough cpp so default and 1.dev triggers compile

* cpp needed for docs as well

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* return trigger service to testing against sdk default lf version

* run trigger integration test against sdk default and LF 1.dev

* return trigger scenario test to SDK default LF version

* avoid import warnings in trigger lib

* Windows manifests a missing file differently. Hilarious
2020-09-15 12:30:23 -04:00
Moritz Kiefer
c0b6851abe
Generate hoogle docs for daml script/triggers (#7371)
* Generate hoogle docs for daml script/triggers

This PR switches over the documentation generation for daml script and
daml triggers to the multi-page format we already use for the standard
library and extends it to also generate hoogle documentation.

All 3 hoogle files are combined in a single hoogle_db.tar.gz archive.

Since the location in the multi-page format is different, I’ve added
redirects.

I verified locally, that I can generate the hoogle database and that
the links point to the right places.

changelog_begin
changelog_end

* Fix baseurl for daml-stdlib

changelog_begin
changelog_end
2020-09-10 15:58:15 +02:00
Moritz Kiefer
3171bb588a
Migrate trigger assertion library to DAML Script (#7318)
This is clearly a breaking change but triggers are still alpha so we
can get away this (confirmed with Bernhard).

changelog_begin

- [DAML Trigger] Daml.Trigger.Assert now uses DAML Script instead of scenarios.

changelog_end
2020-09-04 13:48:50 +02:00
Stephen Compall
0b72b6f716
more trigger service logs; tagged LoggingContext (#7205)
* get a LoggingContext into the TriggerRunnerImpl

* make some implicits more implicitly scoped and explicitly ascribed

* make some private/final markings

* most of JsonFormat[Identifier] is in companion

* experimental LoggingContext with phantom type

* ActorContext#log isn't really doing that much

* more details of LoggingContextOf

* make LoggingContextOf compile

* add trigger message logging, yet without context

* fix parent compile errors

* use Config as the phantom for its own logging extensions

* LocalDateTimeFormat cleanup

* switch TriggerRunner to contextual logging

* add trigger definition ID to logs

* log trigger-submitted commands, fix trigger test compile

* log trigger stopping and DAR uploads

* add context to PostStop/PreRestart logs

* add changelog

CHANGELOG_BEGIN
- [Triggers] More detailed logging of trigger actions and trigger service actions.
  See `issue #7205 <https://github.com/digital-asset/daml/pull/7205>`_.
CHANGELOG_END

* missed copyright header

* switch to Unit, scala/bug#9240 fixed
2020-09-03 17:47:09 +00:00
Samir Talwar
2b3cf1b3c0
ledger-api-client: Rename maxInboundMessageSize to maxInboundMetadataSize. (#7290)
* ledger-api-client: `maxInboundMessageSize` -> `maxInboundMetadataSize`.

CHANGELOG_BEGIN
- [Scala Bindings] Rename a field in the ``LedgerClientConfiguration``
  to ``maxInboundMetadataSize``, to match the builder Netty channel
  builder. It was incorrectly named ``maxInboundMessageSize``, which is
  a different channel property that configures the maximum message size,
  not the header size.
CHANGELOG_END

* ledger-api-client: Introduce a `maxInboundMessageSize` config property.

We use this a lot; easier if it's in the configuration.

CHANGELOG_BEGIN
- [Scala Bindings] Replace the
  ``LedgerClientConfiguration.maxInboundMessageSize`` property with a
  new one that represents the maximum size of the response body.
CHANGELOG_END
2020-09-02 08:41:34 +00:00
Moritz Kiefer
1df6ff49e3
Add a dummy OAuth2 authorization server (#7250)
* Add a dummy OAuth2 authorization server

This is intended for use in the trigger service integration tests not
as an artifact that we ship to users.

changelog_begin
changelog_end

* Address review feedback

changelog_begin
changelog_end
2020-08-28 17:28:06 +02:00
Moritz Kiefer
6e4f101aad
Remove ref-ledger-authenticator from trigger service (#7226)
* Remove authentication service from trigger service

This rips out the integration with ref-ledger-authenticator and
together with that the weird http basic auth which doesn’t actually
work for ref-ledger-authenticator outside of its testing mode and also
includes some very sketchy stuff like storing user
passwords (encrypted but still).

While the exact details of the new approach towards auth are still not
completely clear, it is clear that it is going to be sufficiently
different from what we did before that starting from a clean slate is
easier.

changelog_begin

- [Trigger Service] Parties are now specified in request bodies as
  opposed to via HTTP Basic auth. This is done in preparation for
  running the trigger service against authenticated ledgers.

changelog_end

* Fix integration tests

changelog_begin
changelog_end
2020-08-27 15:36:59 +02:00
Remy
89770cf1c1
Speedy: make the field names from SRecord immutable. (#7225)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-25 17:02:27 +02:00
Stephen Compall
2369a3bbae
use sandbox-style flyway migrations for trigger service (#7092)
* depend on flyway in trigger service; duplicate FlywayMigrations from sandbox

* bring in postgres/h2 detection, other data-source elements needed; use scalalogging

* use DataSource instead of DriverManager for DbTriggerDao, pass to flyway too

- initialization order isn't quite right yet

* hardcode postgres in triggers flyway path

* found an init order that doesn't fail in tests

* better pools for production, and don't forget to close them

* move most init SQL to first migration file

* make logHandler implicit again

- seems to have been an accidental implication of #6334

* move remaining create to V1__Init.sql

* use Prod poolSize

* prove that PostStop signals are never seen (in tests)

* move all the shutdown to Stop message

* remove dead code in DbFlywayMigrations

* always "initialize" database

CHANGELOG_BEGIN
- [Trigger Service] The database format has changed to allow migrations in
  future releases.  Databases are always initialized or migrated to the current
  version on start, so use of ``--init-db`` is no longer required.
  See `issue #7092 <https://github.com/digital-asset/daml/pull/7092>`__.
CHANGELOG_END

* it should always be obvious when there is no escape

* remove logs from experiments

* remove receiveSignal PostStop

* remove unused imports
2020-08-20 18:52:37 +00:00
Samir Talwar
30564a7c90
ledger-api-client: Add test cases for LedgerClient. (#7195)
* ledger-api-client: Add integration tests for the simple stuff.

* sandbox-common: Make `SandboxFixtureWithAuth` a mixin.

This makes it useful with `SandboxNextFixture` as well as
`SandboxFixture`.

Also, add types to non-private fields and methods, and make more fields
protected rather than public.

* ledger-api-client: Add tests to make sure the token is passed through.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox-common: Tokens are for auth, not auth.
2020-08-20 15:00:32 +00:00
Remy
e54c182032
Engine: move check of valide packages inside MutableCompiledPackages (#7186)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-19 19:37:53 +02:00
Moritz Kiefer
d4d0419c64
Use TO_TEXT_CONTRACT_ID in Show instance of ContractId (#7153)
fixes #7114

This PR changes the Show instance of ContractId and flips the switch
on triggers and DAML Script to run in off-ledger mode.

It also adds a test that for DAML Script we actually get back the
correct contract id.

There is a bit of a design decision here in how we want to print
contract ids, so let me list the options I considered. $cid will stand
for the actual cid and all options are wrapped in markdown inline
code.

1. `"$cid"`. Indistinguishable from string. Suggests that there might
be an IsString instance for ContractId.
2. `<$cid>`. Matches the dummy `<contract-id>` but it’s not a dummy so
I don’t think matching that is benefitial.
3. `$cid`. Easy to spot (contract ids start with # and have no
spaces), clearly not a string but might look slightly weird.

changelog_begin

- [DAML Script/DAML Triggers] When using DAML-LF 1.dev, the `Show` instance of `ContractId` will now display the actual contract id instead of a dummy `<contract-id>` value. Note that this only applies to DAML Script and DAML Triggers not to ledger code.

changelog_end
2020-08-17 17:06:24 +02:00
Stephen Compall
1737907415
add --address option to trigger service (#7090)
* factor --address, --http-port, --port-file options from http-json to cli-opts

- enabling reuse in trigger service

* use cli-opts for address and http-port options in Trigger service

* mark ServiceConfig and some defaults private

* use --address option to set up server

* document Setter

* test --address option is parsed

* missing (c) headers

* add changelog

CHANGELOG_BEGIN
- [Trigger Service] Accepts a new ``--address`` option to listen for HTTP connections on
  interfaces other than localhost, such as ``0.0.0.0`` for all addresses.
  See `issue #7090 <https://github.com/digital-asset/daml/pull/7090>`__.
CHANGELOG_END
2020-08-12 13:50:00 -04:00
Stephen Compall
7d5d6e125f
ensure /v1/status returns all results for restarted triggers (#7052)
* test only 'restart triggers with update errors' in CI, 100 times

CHANGELOG_BEGIN
CHANGELOG_END

* build fewer things and try 1000 runs

* test restart after shutdown, 200 times

* distrust the log

* report which withTriggerService is being used to provide log dividers

* log when TriggerStarting, TriggerStarted received in 'starting' state

- In mainline, we only add logs when we add triggers in the 'running'
  stat in Server.apply.  Let's see if we aren't sending these messages
  to the `Server.start` actor when it is still in the 'starting' loop

* log same thing in starting and running states

* revert build experiment and extra logs

- keep the fixture location log, as it's a very useful divider when
  parsing test results
2020-08-11 14:55:44 +00:00
Stephen Compall
0857c28874
turn on NonUnitStatements in trigger runner, service main code (#7079)
* turn on NonUnitStatements in trigger runner, service main code

* add some friendly discards

* add a not-so-friendly discard

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* one more discard to clarify an Await.ready's type

* explain unbind issue
2020-08-11 14:30:11 +00:00
Moritz Kiefer
39085493c8
Make ApplicationId in DAML Triggers configurable (#7075)
fixes #7030

This deliberately ignores the trigger service. The main reason for
setting the ApplicationId at the moment is authentication and this is
still very WIP in the trigger service, so I don’t think it makes sense
to add this in some form to the API until that has settled.

changelog_begin

- [DAML Trigger] You can now configure the application id via
  `--application-id` in `daml trigger`. This is primarily useful if
  you run with authentication.

changelog_end
2020-08-10 18:15:16 +02:00
Stephen Compall
d2ee532949
remove more unused definitions, params, args from triggers Scala code (#7032)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-05 19:06:22 +00:00
Rohan Jacob-Rao
765cd181b5
Trigger service test: Increase timeout for trigger to start running after service shutdown (#6986)
changelog_begin
changelog_end
2020-08-04 18:11:06 +00:00
Stephen Compall
c3e79878ff
remove unused definitions, params, args from ledger API Scala code (#6985)
* remove unused definitions, params, args from ledger API Scala code

CHANGELOG_BEGIN
- [Ledger API] withTimeProvider removed from CommandClient; this method
  has done nothing since the new ledger time model was introduced in
  1.0.0.  See `issue #6985 <https://github.com/digital-asset/daml/pull/6985>`__.
CHANGELOG_END

* percolate withTimeProvider and label removal elsewhere
2020-08-04 18:02:19 +00:00
Stephen Compall
4ba9ae4e2b
remove unused definitions, params, args from triggers Scala code (#6983)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-04 16:50:32 +00:00
Stephen Compall
3196ef7575
fix flaky 'restart triggers after shutdown' test (#6926)
* reenable 'restart triggers after shutdown'

CHANGELOG_BEGIN
CHANGELOG_END

* wait for everything to shut down before completing a withTriggerService fixture

- similar to a change to HttpServiceFixture.withHttpService in #4593,
  but without the suppression of shutdown errors

* label the WithDb tests

* in CI, test only 'recover packages after shutdown', 50 times

* experiment: Process#destroy appears to be async

* is it in the in-between period?

* partial -> total

* replace some booleans with assertions for better error reporting

* make triggerLog concurrent

* close channel and file in other error cases for port locking

- suggested by @leo-da; thanks

* use port locking instead of port 0 for trigger service fixtures

* destroy one service at a time

* missed continuation in build script

* use assertion language for "restart triggers with update errors"

* Revert "is it in the in-between period?"

This reverts commit 211ebfe9d2.

* use better assertion language for "restart triggers with update errors"

* restore full CI build
2020-08-04 08:15:08 -04:00
Moritz Kiefer
5ceeb476c8
Remove duplicated tls config parsing from DAML Script/Triggers (#6947)
This was not only unnecessarily duplicated, it also had a bug where
`--crt` behaved like `--pem` instead of setting the cert chain.

I didn’t add new tests since it seems like the wrong place to test
config parsing of a library. We do have tests for TLS in general for
both DAML Script and DAML Triggers.

changelog_begin
changelog_end
2020-07-31 18:25:18 +02:00
Rohan Jacob-Rao
e6cf6d0d51
Trigger service tests: sequence toxiproxy setup before starting sandbox (#6939)
This should avoid potential contention for ports.

changelog_begin
changelog_end
2020-07-30 15:32:46 -04:00
Rohan Jacob-Rao
91b174da91
Trigger service tests: Remove auth service process in trigger service fixture (#6928)
This code has a few problems: the auth service test mode is not enabled, the
admin ledger should be separate from main ledger if auth is enabled, and the
retry strategy not executed.

The auth service is not used in these tests yet and we have an example
of how to do it correctly in the auth service fixture (and auth service
client tests). We can try to reuse that fixture instead or copy the
code over when necessary.

changelog_begin
changelog_end
2020-07-29 23:29:04 +00:00
Rohan Jacob-Rao
4c76cf98ee
Trigger service tests: Make sure toxiproxy server is running before connecting the client (#6927)
* Trigger service tests: Make sure toxiproxy server is running before connecting client

changelog_begin
changelog_end

* Undo infuriatingly wrong camel case
2020-07-29 23:00:43 +00:00
Samir Talwar
99463577af
triggers + ports: Lock free ports before binding. (#6911)
* triggers: Use `FreePort.find()`.

* ports: Move `LockedFreePort` from postgresql-testing for reuse.

* triggers: Use `LockedFreePort` to avoid race conditions.

* ports + triggers: Move common port testing into the ports library.

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-29 14:29:55 +00:00
Samir Talwar
98de16606d
daml-on-sql: An elegant Main class, for a more civilized age. (#6829)
* daml-on-sql: Pull out a new `Main` object that wraps sandbox-classic.

CHANGELOG_BEGIN
CHANGELOG_END

* daml-on-sql: Fail if a JDBC URL is not provided or not for PostgreSQL.

* sandbox-classic: Rename the conformance test H2 database.

* daml-on-sql + sandbox-classic: Report configuration errors cleanly.

This means letting `ProgramResource` catch the errors, log, and exit.

* daml-on-sql: Change the name logged on startup.

* daml-on-sql: Change the default participant ID.

* sandbox-common: Give the ledger name its own tagged string type.

* sandbox-classic: Generate random ledger IDs using the ledger name.

* daml-on-sql: Remove the banner, replacing it with a blank line.

* daml-on-sql: Enable strong seeding by default.

And weak seeding in the conformance tests.

* sandbox-classic: Move the ledger name to a separate parameter.

It's not really configurable.

* sandbox-classic: Move LedgerName from sandbox-common.

* daml-on-sql: Remove "-participant" from the participant ID.

* daml-on-sql: Use `Name` where possible.

* daml-on-sql: Make the ledger ID mandatory.

* Revert "sandbox-classic: Move LedgerName from sandbox-common."

This reverts commit 0dad1584a7.

* daml-on-sql: Print "DAML-on-SQL" in the CLI help, not "Sandbox".

* daml-on-sql + sandbox + sandbox-classic: Split out custom CLI parsing. (#6846)

* participant-state: Simplify naming the seeding modes.
2020-07-24 18:54:19 +00:00
Rohan Jacob-Rao
4973bd3573
Trigger service: rest of auth service client (#6804)
* Method for service account request

* Extend domain and factor out sa/secure

* Methods to list service accounts and poll for service account

changelog_begin
changelog_end

* Run auth client tests 20 times

* Revert "Run auth client tests 20 times"

This reverts commit 5264ce6f30.

* Assert creds list for new service account is empty

* Request credential method

* Method to get a new credential id

* Method for fetching credential

* Login method (for ledger access token)

* Make authorize style consistent with rest

* Remove redundant test (subset of second test)

* Run auth client tests 20 times

* Fail future for http error responses and refine retry strategy

* Scalafmt

* Revert "Run auth client tests 20 times"

This reverts commit 2568dc6b47.

* Test getting service account before request
2020-07-21 21:26:49 +00:00
Samir Talwar
adcbfb45a1
sandbox-classic: Move the configuration from sandbox-common. (#6810)
* sandbox-common: Move the ledgerIdMode up in the configuration.

* sandbox-classic: Move the configuration from sandbox-common.

CHANGELOG_BEGIN
CHANGELOG_END

* Fix miscellaneous warnings caught by IntelliJ IDEA.
2020-07-21 14:43:53 +00:00
Stephen Compall
4355406259
add more scalac 2.12 warnings (#6798)
* set many extra scalac -Xlint options for all Scala projects

CHANGELOG_BEGIN
CHANGELOG_END

* move NoCopy to its own file

package.scala:18: warning: it is not recommended to define classes/objects inside of package objects.
If possible, define trait NoCopy in package data instead.
  trait NoCopy {
        ^

* move more traits, classes, and objects to proper packages

- note that `package` is itself a scoping construct, so if your reason
  is the apparent aesthetic of placing a bunch of things in one `package
  object`, that is easily remedied by deleting the `object` keyword

* fix some type-parameter-shadow warnings

- I'm generally in favor of sensible name-shadowing, following the
  "deliberately hide variables that should not be accessed here" school
  of thought.  But I think type name shadowing isn't quite as valuable
  and more likely to confuse than general variable shadowing, so have
  experimentally linted it out.

  Example warning:

EventsTableFlatEventsRangeQueries.scala:11: warning: type parameter
 Offset defined in trait EventsTableFlatEventsRangeQueries shadows class
 Offset defined in package v1. You may want to rename your type
 parameter, or possibly remove it.
private[events] sealed trait EventsTableFlatEventsRangeQueries[Offset] {
                                                               ^

* fix more package-object-classes warnings

* fix an inaccessible warning

ContractsService.scala:197: warning: method searchDb in class ContractsService references private class ContractsFetch.
Classes which cannot access ContractsFetch may be unable to override searchDb.
  def searchDb(dao: dbbackend.ContractDao, fetch: ContractsFetch)(
      ^

* enable -Xlint:infer-any

- continuing the saga of #6116, #6132

* enable -explaintypes for more detailed type errors

* missed header for NoCopy; probably should have left it in the package file

* misspelling in comment

* revert -Xlint:doc-detached

- there are a lot of these fixes, and they are noisy, so shifting to a
  separate PR
- thanks to @leo-da for pointing out
2020-07-21 08:18:01 -04:00
Moritz Kiefer
c524aee995
Bump timeouts in trigger auth service tests (#6797)
tested via --runs_per_test=8 and seems to be sufficient now.

changelog_begin
changelog_end
2020-07-20 17:06:45 +00:00
Rohan Jacob-Rao
9e979ced9a
Trigger service: Introduce auth service client and test fixture (#6768)
* WIP Call authorize before start trigger

* Intro auth service client

* Auth service client test setup

CHANGELOG_BEGIN
CHANGELOG_END

* scalafmt

* Use Spray JSON format and slightly better uri handling

* Get auth client test to work and fiddle with uris

* Remove changes to Server

* Add package for URI

* Format

* Copyrights

* Update sandbox packages

* Use default wallclock time for auth service sandbox
2020-07-20 13:35:14 +00:00
Robert Autenrieth
7ce9748066
Split sandbox code into separate packages (#6695)
* Move public code into daml-integration-api

CHANGELOG_BEGIN
[DAML Integration Kit]: Removed sandbox specific code from the API intended to be used by ledger integrations. Use the maven coordinates ``com.daml:participant-integration-api:VERSION`` instead of ``com.daml:ledger-api-server`` or ``com.daml:sandbox``.
CHANGELOG_END
2020-07-17 17:06:06 +02:00
Rohan Jacob-Rao
619209d0b1
Trigger service: Document steps to replicate auth flow from the command line (#6725)
CHANGELOG_BEGIN
CHANGELOG_END
2020-07-14 15:05:41 -04:00
Gary Verhaegen
8176fb0c8d
fix the the typo (#6723)
```
s/the the /the /
```

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-14 16:55:33 +00:00
Rohan Jacob-Rao
98613d9671
Trigger service: doc explaining planned use of auth service (#6659)
* Add details about validity windows
* Reword description of auth service concepts
* Elaborate on auth flow dealing with token/credential expiry

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-09 14:27:50 -04:00
Rohan Jacob-Rao
30f1bf8d5a
Disable recovery test until I can fix the flakiness (#6654)
CHANGELOG_BEGIN
CHANGELOG_END
2020-07-08 11:17:12 -04:00
Moritz Kiefer
02c59d4f2a
Revert ANF changes and add a testcase for evaluation order (#6645)
* Revert ANF changes and add a testcase for evaluation order

After careful consideration, we decided that the change in evaluation
order that was accidentally introduced by the ANF changes should be
considered a breaking change or arguably even a bug and should not
land in 1.3.0.

Therefore, this PR reverts the following commits:

1. 353d0da6f7
2. a45b51042f
3. 04c7b2af7f
4. a624dd7242
5. b3aab72cee

Other PRs mostly had trivial merge conflicts that I resolved. The two
most interesting ones here are probably

1. https://github.com/digital-asset/daml/pull/6576 which was easy to
   resolve and the change to return SEValue instead of SExpr is still
   nice and useful even if we do not need the guarantees.
2. it https://github.com/digital-asset/daml/pull/6542 which required
   some changes since the constructors changed. If you want to review
   those changes in detail (they are pretty straightforward so not too
   important), it’s probably easiest to check out this PR and run
   ```
   git diff 2cd2a8f2a8
   daml-lf/interpreter/src/main/scala/com/digitalasset/daml/lf/speedy/Compiler.scala
   ```
   to see the diff to the parent commit of the first commit that
   introduced ANF.

changelog_begin
changelog_end
2020-07-08 12:40:12 +02:00
Stephen Compall
4cd419623a
replace traverseU and sequenceU with traverse and sequence (#6594)
* replace traverseU and sequenceU with traverse and sequence

- with -Ypartial-unification on, the extra Unapply typeclass lookup is
  unnecessary

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* limit imports; we only need *> and void
2020-07-07 15:01:58 +00:00
Rohan Jacob-Rao
8261509312
Trigger service: Remove retries when checking trigger ids (#6628)
The running trigger store is now written to synchronously by the server, so there's no need for retry logic.

Also assert equality of trigger ids rather than an arbitrary predicate, as it gives better error messages and we don't use other types of predicates for this function.

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-06 13:48:38 -04:00
Rohan Jacob-Rao
7501805eae
Trigger service: increase timeout for db restart test (#6624)
changelog_begin
changelog_end
2020-07-06 16:29:47 +00:00
Moritz Kiefer
907e4fe678
Fix project versions of DAML Script and DAML Triggers (#6622)
GHC has weird restriction on version numbers which damlc inserits so
we need to use `ghc_version` instead of `sdk_version`. That only makes
a difference for snapshot versions where the `-snapshot.` part is
replaced by `.`.

changelog_begin
changelog_end
2020-07-06 15:07:39 +00:00
nickchapman-da
b3b30ae71d
Speedy issue 6573 (#6576)
* address issue 6573

changelog_begin
changelog_end

* fmt

* push switch from SExpr to SValue to callers of the trigger service
2020-07-02 13:15:00 +01:00
Moritz Kiefer
cf4fcb560b
Fix the package version of daml-trigger and daml-script (#6566)
It makes no sense to keep this at 0.0.1.

changelog_begin

- [DAML Script] The DAML Script library now has the version of the
  corresponding SDK.

- [DAML Trigger] The DAML Trigger library now has the version of the
  corresponding SDK.

changelog_end
2020-07-01 19:06:00 +02:00
Rohan Jacob-Rao
bdaf9aa216
Trigger service: restart triggers in database on server startup (#6556)
* Add optional trigger id to startTrigger

CHANGELOG_BEGIN
CHANGELOG_END

* Read running trigger table in DbTriggerDao

And get rid of traverse type application!
(since we have partial unification on)

* Restart triggers in database

* Test for trigger restart on recovery

* Add order by clause and remove accidental overloading
2020-07-01 13:22:23 +00:00
Rohan Jacob-Rao
546c75b74f
Trigger service: Remove data in messages from TriggerRunnerImpl to Server (#6554)
* Remove data from message sent from TriggerRunnerImpl to Server

* Remove unused trigger name from config

changelog_begin
changelog_end
2020-06-30 21:02:10 -04:00
Rohan Jacob-Rao
ea16ff350d
Trigger service: Change to restart strategy with backoff (#6552)
* Add min/maxRestartInterval to configs and use in trigger restart strategy

* Adapt tests with triggers failing due to lack of network

changelog_begin
changelog_end

* Adapt tests for triggers with errors

* Remove comment about number of restarts

* Use a small initial restart interval for testing

* Remove old restart params

* Move maxInboundMessageSize to LedgerConfig

* Rename TriggerRunnerConfig to TriggerRestartConfig
2020-06-30 20:46:23 -04:00
nickchapman-da
353d0da6f7
Speedy ANF (#6440)
* ANF transformation in Speedy.

The idea behind this PR is to transform speedy expressions into a simpler form where all non-atomic sub-expressions are made explicit by the introduction of let-forms. In particular, for the function-application form. These simpler forms allow the execution engine to take advantage of the atomic assumption, and often removes many additional execution steps. In particular the pushing of continuations to allow execution to continue after a compound expression has been reduced to a value.

changelog_begin
changelog_end

* improve comment

* inline functions relocateA/L

* remove comment about scalafmt

* remove commented out alterative def for transformLet1

* improve code by adding incr methods to DepthA/E

* remove (n == 0) special case in trackBindings

* clarify comment further

* improve validate/go to not consume stack for deeply right-nested let-expressions

* address comments from Remy: be private; use final case case; etc

* rename to unsafeCompilationPipeline

* add back some trailing commas

* remove commented-out debug line

* improve comment

* remove dev/debug code in compilationPipeline

* remove commented out code in SEAppGeneral.execute

* undo unrelated code improvement in SValue.scala

* fix compile. object Anf cannot be private
2020-06-30 20:52:44 +00:00
Rohan Jacob-Rao
99a9662781
Trigger service: change convention for updating running trigger store (#6532)
* Don't modify running trigger store via the trigger runner actor

changelog_begin
changelog_end

* Remove runner actor from RunningTrigger type

And parent actor from TriggerRunnerImpl

* Add to running trigger store on start request

* Update tests for new trigger store convention

* Fix test with network outage
2020-06-30 12:00:45 -04:00
Rohan Jacob-Rao
9c4daad473
Trigger service nits (#6528)
- Move Stop Message back to original place with other Server state messages
- Other style nits

changelog_begin
changelog_end
2020-06-29 13:25:35 -04:00
Shayne Fletcher
44bb7f5233
Spawn a ref ledger auth instance from text fixture (#6502)
changelog_begin
changelog_end
2020-06-25 15:53:34 -04:00
Rohan Jacob-Rao
674fecf73f
Trigger service: split up ServiceMain into multiple files (#6498)
* Move top level case classes to package object

changelog_begin
changelog_end

* Move Server Message classes to separate file

* Server in Server.scala and ServiceMain in ServiceMain.scala

* Copyright headers
2020-06-25 16:01:35 +00:00
Rohan Jacob-Rao
eb861ffd4a
Trigger service: split up Server companion object (#6491)
The Server object and especially the apply method is extremely big and
hard to follow. There are a great many functions nested within the apply
method. This PR attempt to organize things, mostly by moving local
functions into methods of the Server class. I think this makes things
easier to follow, and I think it's more conventional.

Note that with this change I added some implicit parameters to the
Server constructor, which I think makes sense for these kinds of values
(contexts, sequencer factory, etc.) I also moved the Message trait to
the top level, but we will probably it its own file, following this PR.
There are a few other shallow tweaks that I made along the way.

Open to feedback on the high level structuring, as I'm not super
familiar with idiomatic design in Scala.

changelog_begin
changelog_end
2020-06-25 10:22:53 -04:00
Shayne Fletcher
2f539c477a
exe lookup in runfiles (#6489)
changelog_begin
changelog_end
2020-06-25 09:48:55 -04:00
Stephen Compall
a51d0db8ff
set scalac -Xsource:2.13 -Ypartial-unification globally (#6469)
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts

* add now-referenced scalaz-core where needed

* work around bad type signatures in scalatest Aggregating, Containing

* unused Any suppression

* work around bad partial-unification wrought by type alias

* remove unused Conversions import

- not required in 4f68cfc480 either, so unsure how it's survived this long

* work around Future.traverse; remove unused show import

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused bounds

* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed

* longer comment on what the options do

- suggested by @stefanobaghino-da; thanks

* forget Future.traverse, just use scalaz, it knows how to do this
2020-06-24 16:51:24 -04:00
Shayne Fletcher
afc573d422
step 0 integration of ref-ledger-authentication (#6484)
changelog_begin
changelog_end
2020-06-24 13:40:03 -04:00
Rohan Jacob-Rao
10208fd194
Trigger service: read packages from database on startup (#6470)
* Move database initialization to Server apply method
* Reorder dao/server creation
* Read packages from database on startup
* Test starting a trigger after a shutdown
changelog_begin
changelog_end
2020-06-24 11:18:16 -04:00
Rohan Jacob-Rao
b10bfa1ba7
Trigger service: Write packages to database if we have one (#6424)
This is needed to recover state after the service shuts down or crashes.
We add a method to the RunningTriggerDao to persistPackages. This only
does something in the case of a DbTriggerDao. In any case the Server
keeps a package map in memory as it's required to construct a trigger runner.
Uploads of existing packages is considered harmless.

changelog_begin
changelog_end
2020-06-19 13:49:49 -04:00
Rohan Jacob-Rao
78d598ecf3
Trigger service: Improve formation of SQL queries (#6422)
Avoid the use of Fragment.const which interprets raw strings as SQL
queries without any checks. Use the `sql` string interpolation which I
found out does the right thing with Strings and other simple types.

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-19 09:32:27 -04:00
Remy
1b1b4eab2c
Speedy: clean machine builder name (#6427)
* Address comment martin made in #6368

* changelog

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-19 14:58:13 +02:00
Rohan Jacob-Rao
0ca1288d9d
Trigger service: Increase server binding timeout (#6421)
To avoid timeouts in CI. Also factor out the implicit val one level
so it is used by both tests and main method.

changelog_begin
changelog_end
2020-06-18 19:54:24 +00:00
Rohan Jacob-Rao
5934b22dda
Trigger service: Refactor Server.addDar to take encoded dar (#6395)
Trigger service: Refactor Server.addDar to take encoded dar

This is so we can write the encoded packages to the database if we have
one (without re-encoding them).

changelog_begin
changelog_end
2020-06-18 12:37:43 -04:00
Remy
5a9e7ebc7c
Speedy: refactor Machine builders (#6368)
In this PR we cleanup the constructor for the speedy Machine.

* We remove the `case`  keyword since `Machine` is a stateful class,
* We replace the pre-existing builders with
  + one generic builder `Machine.apply`,
  + scenario specific builder,

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-18 15:39:55 +02:00
Rohan Jacob-Rao
5ca65ad69a
Trigger service: Refactor testing with and without database (#6373)
* Trigger service: Refactor testing with and without database

Following the approach taken in the HTTP JSON API.
Define an AbstractTriggerServiceTest class with abstract jdbcConfig
member. This is instantiated with None and Some in the in-memory and
database test classes respectively. Tests for the individual modes can
be added in these concrete classes.

changelog_begin
changelog_end

* Move db init and destroy into beforeEach/afterEach to avoid concurrent tests clashing
2020-06-16 18:00:20 -04:00
Rohan Jacob-Rao
3cfed4d7a3
Trigger Service: Refactor running trigger access using DAO interface (#6334)
Trigger Service: Refactor running trigger access using DAO interface

This introduces a RunningTriggerDao trait and two classes that implement
it, one for in-memory and one for database storage. This moves a lot of
code out of the Server class and into the DAO classes. The class chosen
at runtime depends on whether a JDBC config has been provided.

The database DAO also has methods for initialization and clean up used by
the server main function and the test fixture.

This is a pure refactoring (modulo some minor error message changes) so
functionality should stay the same.

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-13 14:33:46 -04:00
Shayne Fletcher
23e7dc826e
Trigger service docs and SDK integration basics (#6329)
* Integrate trigger-service into the assistant

changelog_begin
changelog_end

* Assistant integration and bare bones docs

changelog_begin
changelog_end
2020-06-12 18:54:38 +00:00
Simon Maxen
5329a4f97f
Ledger id requirement (#6323)
* Add option based constructor for LedgerIdRequirement

changelog_begin
changelog_end

* Make option based consructor the default, deprecate old constructor

* Update with review comments
2020-06-12 15:00:32 +00:00
Shayne Fletcher
0f47b8d1bf
Simplify findCredentials logic (#6301)
changelog_begin
changelog_end
2020-06-11 23:04:14 +00:00
Shayne Fletcher
b8126ee75c
Add --no-secret-key (#6309)
changelog_begin
changelog_end
2020-06-11 18:41:26 -04:00
Shayne Fletcher
07f24c0d93
Encrypt credentials (#6296)
* Encrypt credentials

changelog_begin
changelog_end

* Move key aquisition and write a big fat warning

* Only write encrypted token to DB.
2020-06-10 17:14:29 -04:00
Shayne Fletcher
033c58bc60
Switch to basic auth in readiness for auth service integration (#6279)
* Switch to basic auth in readiness for auth service integration

changelog_begin
changelog_end

* Newtype for user credentials
2020-06-10 09:48:33 -04:00
Rohan Jacob-Rao
e7017e6943
Trigger service: delete from database in stop trigger (#6249)
Includes a fair amount of refactoring to smooth the logic with/without a database. All tests now run with and without a database.

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-08 16:19:35 -04:00
Shayne Fletcher
ecad4ff7c3
Fix test descriptions (#6269)
changelog_begin
changelog_end
2020-06-08 19:11:59 +00:00
Shayne Fletcher
0b7989a715
Adjust status code response for unauthorized (#6268)
changelog_begin
changelog_end
2020-06-08 18:56:43 +00:00
Shayne Fletcher
0625e68988
Disambiguate initialization halted vs. failed (#6248)
changelog_begin
changelog_end
2020-06-05 17:18:38 -04:00
Rohan Jacob-Rao
3159c617f6
Tweak error handling in listTriggers helper (#6247)
changelog_begin
changelog_end
2020-06-05 20:32:49 +00:00
Rohan Jacob-Rao
de2b5d195e
Trigger service style touch ups (#6246)
* Touch up logTriggerStatus

* Touch up removeRunningTrigger

* Touch up addRunningTrigger

* Remove IntelliJ (scalastyle I think) warnings about public members without type annotations

CHANGELOG_BEGIN
CHANGELOG_END

* Redundant brackets
2020-06-05 16:07:10 -04:00
Shayne Fletcher
71ff65d5be
Trigger stop on db write failure (#6233)
* Stop running trigger under DB write failure

changelog_begin
changelog_end

* Another fix relating to initialization retries

changelog_begin
changelog_end

* Handle the error case directly
2020-06-04 17:00:12 -04:00
Shayne Fletcher
e23a488596
Put initialization failures through retries too (#6230)
changelog_begin
changelog_end
2020-06-04 15:24:51 -04:00
Shayne Fletcher
322b93c7e7
Temporarily disable flaky test (#6228)
changelog_begin
changelog_end
2020-06-04 12:04:35 -04:00
Shayne Fletcher
2c3efc6c4f
Refactor startTrigger (#6219)
changelog_begin
changelog_end
2020-06-04 09:30:47 -04:00
Rohan Jacob-Rao
09254a0fa8
Trigger service: streamline running tests with and without the database (#6205)
This adds a function withTriggerServiceAndDb which runs a test twice, once with and once without a database, and succeeds if both succeed. This will be useful for reusing test logic with both backends and making sure behaviour is consistent. I have used this function where possible, but it won't work for everything until stop is implemented on the DB side.

At the moment this new function squashes two tests into one making it hard to tell whether it failed with or without the database. In a future PR I will investigate using an abstract class to run the tests separately (hopefully with altered descriptions).

This feature required a few changes in the process, mainly:
- Use PostgresAroundAll to connect/disconnect to the database before and after all tests run
- Add a destroy method to the TriggerDao to reset the database between tests
- Use the TriggerDao in the withTriggerService functions to initialize / clean up the database at the start / end of each test
- Sort trigger instances from list using Scala's sort, not relying on Postgres' ordering of UUIDs. This also means we need to use UUIDs for trigger instances in the tests and sort nonempty vectors in expected results.
2020-06-04 09:30:26 -04:00
Shayne Fletcher
c2961b1957
Make the stop endpoint more robust and test (#6217)
* Make the stop endpoint more robust and test

changelog_begin
changelog_end

* Stopping an unknown trigger gives 404
2020-06-03 18:56:51 +00:00
Shayne Fletcher
6be68dc059
Renfinements to error test DAMLs (#6212)
changelog_begin
changelog_end
2020-06-03 15:02:53 +00:00
Shayne Fletcher
5ff8ec69df
Trigger error cases (#6202)
changelog_begin
changelog_end
2020-06-02 19:29:43 -04:00
Rohan Jacob-Rao
0b5762e437
Trigger service: use DB for start and list commands (#6153)
* Insert running trigger to DB when using one

If the DB write fails, the server sends itself a
TriggerInitializationFailure message so that the corresponding trigger
runner is stopped and the table is in sync with the actors.
We still need to retry writes here.

Includes basic test that runs the server with a JDBC config set and adds
a trigger, expecting a new entry to be added to the DB. However does not
check the running trigger table which we can do once reads are
implemented.

changelog_begin
changelog_end

* Await on future in test

* Update to new assertTriggerIds

* Apply scalafmt suggestions

* Create index on party token

* Read db in list command

* Update comment in test script

* Remove outdated comment

* Fix strings in insert and select

* Clean up test

* Add a second trigger in the db test

* Fix comment in test script

* Comment db tables

* Order trigger instances in list command

* Comment about TriggerDao execution context
2020-06-02 16:57:35 +00:00
Shayne Fletcher
256d48a431
Switch to timestamps not strings (#6191)
changelog_begin
changelog_end
2020-06-02 15:24:57 +00:00
Shayne Fletcher
2de23d93c2
Trigger record status (#6180)
* Store trigger history

changelog_begin
changelog_end

* Harvest trigger histories

changelog_begin
changelog_end

* Switch to Vector over List (and other bits and bobs)

* Use a better verb for updating trigger status method

* Add a comment

* Fix mangled comments
2020-06-01 20:19:51 +00:00
Shayne Fletcher
41c967f1f1
Failure path testing (#6178)
changelog_begin
changelog_end
2020-06-01 12:50:51 -04:00
Shayne Fletcher
f619dea380
Test trigger won't start when ledger connection not viable (#6150)
changelog_begin
changelog_end
2020-05-28 19:05:14 -04:00
Rohan Jacob-Rao
cc5d5d3fd8
Trigger service: refactor in prep for DB writes (#6134)
* Pass JdbcConfig object to TriggerDao apply

* No need to return TriggerDao from init db

* Refactor introducing RunningTrigger type

* Rename triggerId -> triggerInstance and triggerOrigId -> triggerName
Note this also changes the start request parameter name to triggerName.
However I have not yet renamed triggerId in the response messages. We
should probably make it triggerInstance there too but in a later PR.

changelog_begin
changelog_end
2020-05-28 13:47:24 -04:00
Stephen Compall
15350a7bc2
disable warts.Any and remove most suppressions (#6132)
* disable Any wart

* first pass removal of Any suppressions for false positives

* second pass removal of Any suppressions for false positives

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* third pass removal of Any suppressions for false positives

* fourth pass removal of Any suppressions for false positives

* reformat newly single-suppressions into single lines

- suggested by @SamirTalwar-DA; thanks
2020-05-28 16:53:41 +00:00
Shayne Fletcher
4fb13734ad
Extend proxy into test functions and gentle refactor (#6147)
changelog_begin
changelog_end
2020-05-28 12:39:34 -04:00
Shayne Fletcher
576e47f587
Toxify Windows (#6122)
changelog_begin
changelog_end
2020-05-27 12:07:54 -04:00
Shayne Fletcher
79563ee74b
ToxiProxy config (#6114)
changelog_begin
changelog_end
2020-05-27 07:08:45 -04:00
Shayne Fletcher
b95dd39a6e
Don't update running triggers until we know the trigger is running (#6079)
* Don't update running triggers until we know the trigger is running

changelog_begin
changelog_end

* Don't update running triggers until we know the trigger is running
2020-05-26 16:15:29 -04:00