Commit Graph

426 Commits

Author SHA1 Message Date
Stephen Compall
8309cfef1e
group JSON API tests by endpoint, other categories (#14522)
* group query tests
* group remaining abstract integration tests
* group user management tests
* group websocket tests

CHANGELOG_BEGIN
CHANGELOG_END
2022-07-26 11:50:04 -04:00
Stephen Compall
6e3069d220
add retroImplements to lf/iface DefInterface (#14192)
* add resolveRetroImplements to EnvironmentInterface
* test that resolveRetroImplements does nothing with no data
* resolve retroactive interfaces in Java codegen
* resolve retroactive interfaces in Scala codegen
* resolve retroactive interfaces in json-api

CHANGELOG_BEGIN
CHANGELOG_END
2022-06-29 13:13:02 -04:00
Victor Peter Rouven Müller
8d21d19133
Add support for cmd deduplication & submission id for cmd submissions (#13771)
* Add completion offset to cmd submission responses

changelog_begin
- [HTTP-JSON] there is now a new meta field called deduplicationPeriod for cmd submissions with which one can specify the deduplication period. Moreover there also is another new meta field submissionId for the cmd submissions.
- [HTTP-JSON] Calls to endpoints for command submission now include the completion offset
changelog_end

Co-authored-by: Raymond Roestenburg <98821776+ray-roestenburg-da@users.noreply.github.com>
2022-06-29 16:57:05 +00:00
Stephen Compall
90ca397bc5
replace remaining RNil usages with shapeless.record.Record (#14275)
* deprecate RNil

CHANGELOG_BEGIN
CHANGELOG_END
2022-06-28 19:16:25 +00:00
Sergey Kisel
6a80a1a82f
Removal of sandbox-classic module and binary (#14201)
* Removal of sandbox-classic module and binary

CHANGELOG_BEGIN
CHANGELOG_END
2022-06-21 11:09:51 +02:00
Stephen Compall
19ad1be8f2
use domain.Party instead of String in json-api tests (#14158)
* use domain.Party instead of String for iouCreate{,AndExerciseTransfer}Command

CHANGELOG_BEGIN
CHANGELOG_END
2022-06-15 08:59:23 -04:00
Sergey Kisel
1d8f195438
Sandbox Classic Removal in Tests (#13934)
CHANGELOG_BEGING
CHANGELOG_END
2022-06-07 19:54:52 +00:00
Stephen Compall
7c23420876
use more domain models for json-api response tests (#14101)
CHANGELOG_BEGIN
CHANGELOG_END
2022-06-07 08:57:17 -04:00
Stephen Compall
96a6fe62e4
choice name overloading in lf/interface (#13938)
* introduce TemplateChoices to mediate template choice resolution
* if any resolutions fail in EnvironmentInterface, discard the template's resolution
* if any resolutions fail in Interface, discard the remaining unresolved

CHANGELOG_BEGIN
CHANGELOG_END
2022-05-31 08:41:48 -04:00
Stephen Compall
a716428ebc
replace hand-disassembly of json in tests with domain model values (#13960)
* make postJsonRequestWithMinimumAuth parse, and make all its callers declarative
* make getRequestWithMinimumAuth parse, and make all its callers declarative
* note about further cleanup

CHANGELOG_BEGIN
CHANGELOG_END
2022-05-24 06:31:06 -04:00
Stephen Compall
04c37349fc
refactor contract/party endpoint functions to endpoints package (#13955)
* move fetch, retrieveAll, query to separate module
* move parties endpoints to separate module
* narrow some access as all endpoint functions have moved

CHANGELOG_BEGIN
CHANGELOG_END
2022-05-24 06:29:51 -04:00
Stephen Compall
b307cd8406
remove some contract-ID generality from TypedValueGenerators (#13957)
* move where ContractId arbitrary instance passes into ValueAddend
* adjust ValueAddend users to deal with moved Arbitrary instance

CHANGELOG_BEGIN
CHANGELOG_END
2022-05-24 06:28:34 -04:00
Stephen Compall
97f3f4f956
reduce fixture-passing boilerplate in json-api tests (#13829)
* move the fixture data to a single value, make utilities extend off it
* start unifying the with* functions

CHANGELOG_BEGIN
CHANGELOG_END
2022-05-23 16:16:35 +00:00
Remy
6f6a3052a1
Repurpose templateId in ExerciseCommand and add interfaceId in ExerciseEvent (#13660)
part of #13653

CHANGELOG_BEGIN
CHANGELOG_END
2022-05-16 15:30:07 +00:00
Victor Peter Rouven Müller
3decea2a95
Use deriveFormat for ErrorDetailsFormat instead of hand written JsonFormat (#13770)
* Use deriveFormat for ErrorDetailsFormat instead of hand written JsonFormat

changelog_begin

- [HTTP-JSON] The field "@type" was renamed to "type" for encoding the ErrorDetails case

changelog_end

* Use moar types! :)

* Use proper tags, not aliases
2022-05-03 13:28:15 +00:00
Victor Peter Rouven Müller
2e6dbefa8c
Add optional ledgerApiError details to responses (#13751)
* Add optional ledgerApiError details to responses

* Finally get some proper error details

* Cleaning up the code

* Less bad words in the code

* Better deserialization error msg

changelog_begin

- [JSON-API] The response of the json api now includes a field ledgerApiError that contains extended information in the case of ledger errors. See issue https://github.com/digital-asset/daml/issues/9834 for more details.

changelog_end

* Reformat!

* Remove debug logging statements

* also update security evidence file

* more cleanup

* Improve Json decoder & encoder

* Add test for json encoder&decoder

* Add test which ensures that the LedgerApiError field in the response is filled with valid information

* Fix failing failure tests
2022-04-29 21:44:37 +02:00
Nicu Reut
a792de39a5
Update scalafmt to 3.5.1. (#13584)
CHANGELOG_BEGIN
CHANGELOG_END
2022-04-28 09:57:04 +00:00
Stephen Compall
f4d603757e
split AbstractHttpServiceIntegrationTest into two files (#13685)
* move AbstractHttpServiceIntegrationTestFuns to separate file
* security evidence moved

CHANGELOG_BEGIN
CHANGELOG_END
2022-04-25 15:25:23 -04:00
Stephen Compall
7c33a02d44
test and document error when querying with a pruned offset argument (#13633)
* pruning test successful, shows the real error
* document current pruning failure

CHANGELOG_BEGIN
CHANGELOG_END
2022-04-20 10:57:33 -04:00
Stephen Compall
9cafd894c1
nicer party throwaway literals in tests (#13619)
* replace NoCid with Aux
* xmap on ValueAddend, a way to e.g. let users "pun" on scalars
* nicer party specifications
- suggested by @realvictorprm; thanks

CHANGELOG_BEGIN
CHANGELOG_END
2022-04-19 09:54:50 -04:00
Stephen Compall
4bf0de8901
better errors when querying by interface ID (#13406)
- double the resolveTemplateId interface and use the "contract type" for
  exercise-by-id only
- lookup for template ID only
- 400 + error message test
- move query test to also run on PG/oracle

CHANGELOG_BEGIN
CHANGELOG_END
2022-03-31 15:53:39 +00:00
Victor Peter Rouven Müller
cc06073333
Java bindings/interface support (#13366)
* WIP

* First working version of java codegen daml interface support

* Update language-support/java/codegen/BUILD.bazel

Co-authored-by: Remy <remy.haemmerle@daml.com>

* Fix compile errors

* Simplify code massivly, enjoy less duplication

changelog_begin

- The Java codegen now has basic support for daml interface definitions. Converting a contract id of a template implementing an interface to a contract id of the interface is possible and both executing interface choices is possible on the contract id of the interface and implementing template.

changelog_end

* Rename the test file to reflect it is a test file & enhance the inner test name & extend it further

* Fix test

* Refactor parts of TemplateClass.scala into multiple files

* Format that files!

* Minimize duplication further

* Remove unused comment

* Simplify code
Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Update language-support/java/codegen/src/main/scala/com/digitalasset/daml/lf/codegen/CodeGenRunner.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Further refactoring and renaming of the TemplateClassSpec to ContractIdClassBuilderSpec

* Fix formatting

* Add interface docs

* Remove unnecessary code generation of the Contract class for interface types

* Use less bool flags and more good function names :)

* Fix build

Co-authored-by: Remy <remy.haemmerle@daml.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2022-03-29 12:38:00 +02:00
Stephen Compall
467b8fbbe7
shorter Value expressions in json-api tests (#13358)
* template ID variables
* remove duplicate function

CHANGELOG_BEGIN
CHANGELOG_END
2022-03-24 15:51:55 +00:00
Stephen Compall
2dd248cd68
move nonempty code to new nonempty library (#13350)
- new nonempty artifact, separate from scala-utils
- com.daml.scalautil.nonempty is now com.daml.nonempty

CHANGELOG_BEGIN
CHANGELOG_END
2022-03-21 17:14:17 +00:00
Victor Peter Rouven Müller
8e6493b02a
Add security evidence about the failure tests from the json api (#13229)
changelog_begin
changelog_end
2022-03-21 15:58:59 +01:00
Victor Peter Rouven Müller
47ca2cb953
Add security evidence annotations to the integration tests of the json api (#13225)
changelog_begin
changelog_end
2022-03-21 15:58:23 +01:00
Victor Peter Rouven Müller
31366b939c
Add test evidence annotation to unit test of the json api (#13219)
changelog_begin
changelog_end
2022-03-21 15:57:59 +01:00
Victor Peter Rouven Müller
bfd69b0bee
Test that we return a 404 when requesting information about non-existing packages (#13346)
changelog_begin

- [HTTP-JSON] If information about a package is requested that doesn't exist on the ledger you will now get a 404 instead of a 500 as response.

changelog_end
2022-03-21 15:57:32 +01:00
Stephen Compall
d37a63f56e
interface choices in json-api (#13301)
This expands the domain of the exercise endpoint to accept interface IDs
instead of template IDs when exercising by contract ID. The argument is
still [compatibly] called templateId; interface IDs are accepted nowhere
else.

CHANGELOG_BEGIN
- [JSON API] Supports interface choices, including exercise by contract
  ID accompanied by an interface ID.
  See `issue #13301 <https://github.com/digital-asset/daml/pull/13301>`__.
CHANGELOG_END
2022-03-18 17:47:30 +00:00
Stephen Compall
0b5ad7a7bd
interface support in the interface library (#13154)
This adds the presence of interface names, as well as lists of choices
and their argument/return types for each interface.

It also changes the semantics of DefTemplate and EnvironmentInterface:
each may contain "unresolved choices", which are the choices inherited
from interfaces for each implementing template;
EnvironmentInterface#resolveChoices will copy the full type information
as a concrete choice for each implementing template, which should be
much simpler for codegen and json-api.

A type generator should take account of the astInterfaces as well as
typeDecls in EnvironmentInterface; interfaces aren't included in
typeDecls because where I is an interface only ContractId I is
serializable.

* Ast.DataInterface is empty, just use the interfaces map directly
* port json-api for interface API changes; does not solve #12689
* port Scala codegen for interface API changes; does not add support
* port Java codegen for interface API changes; does not solve #11350

CHANGELOG_BEGIN
CHANGELOG_END
2022-03-15 19:34:54 +00:00
Stefano Baghino
739173bf84
Remove references to missing appender (#13294)
changelog_begin
[Trigger Service] Spurious logging warnings will not be printed when running `daml trigger-service`
[Oauth2 Middleware] Spurious logging warnings will not be printed when running `daml oauth2-middleware`
changelog_end

Removed references to a missing `stderr-appender` in several configuration file.

This removes warnings when running `daml trigger-service` and `daml oauth2-middleware`.

This will also remove those same warnings from several tests (HTTP JSON API and Trigger Service).
2022-03-15 18:02:02 +01:00
Stephen Compall
153a5e6bcc
parallelize long-running custom/user token tests (#13290)
Saves about 90 seconds in test wallclock time. Other tests considered
either ran too short to bother splitting, or were Oracle and thus won't
affect the main run at all.

CHANGELOG_BEGIN
CHANGELOG_END
2022-03-15 10:59:36 -04:00
Sergey Kisel
a4f4300502
Naming and logging improvements [DPP-956] (#13270)
* Naming and logging improvements [DPP-956]

CHANGELOG_BEGIN
CHANGELOG_END
2022-03-15 09:40:39 +01:00
Stephen Compall
2056f0a5cf
nonempty-cats refactoring (#13282)
* move ImplicitPreference to scala-utils
  - test it
  - use it instead of tags for json-api testing
  - fix a somewhat nonsensical type therein
* several new tests for the NonEmpty Foldable instances

CHANGELOG_BEGIN
CHANGELOG_END
2022-03-14 20:48:41 -04:00
Stephen Compall
c9b5750968
NonEmpty utilities, new nonempty-cats library (#13115)
New instances and collection methods, many of which included
as-is from @andreaslochbihler-da's experiments downstream.

* mk, a version of apply with special inference
* new names for the conversion functions suggested by
  @andreaslochbihler-da
* groupBy1, zipWithIndex, reduceLeft, last1, distinct, toVector1
* a derived Semigroup instance
* a more explicit subtyping utility to specify implicit preference than
  what we get with shapeless @@
* from method, alias for unapply
* cats functorial, foldable instances in the new nonempty-cats library
* new NonEmpty utilities

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Andreas Lochbihler <36634420+andreaslochbihler-da@users.noreply.github.com>
2022-03-10 16:30:42 +00:00
Sergey Kisel
9a27edd656
Improve MessageDigest and Mac instance creation to solve lock contention [DPP-956] (#13221)
* Improve MessageDigest and Mac instance creation to solve lock contention problem [DPP-956]

changelog_begin
Scalability bottleneck in regard to hashing has been fixed in multiple places.
changelog_end
2022-03-09 22:15:32 +01:00
Stephen Compall
1e562757fe
json-api failure tests with user tokens (#13113)
Refactors a bunch of AbstractHttpServiceIntegrationTestFuns and
elements of #12922's JWT function abstractions into a new
HttpFailureUserFixture, using that split to split the
FailureTests as well.

Fixes #13112.

* restore failure test porting from #12922
This reverts b4942defa4.

* move token-relevant utilities to http-json-testing lib

* replace custom actorsystem setup with AkkaBeforeAndAfter
- suggested by @ray-roestenburg-da; thanks

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Victor Peter Rouven Müller <mueller.vpr@gmail.com>
2022-03-09 18:15:49 +00:00
Stefano Baghino
00b7ad3803
Fix red main (#13224)
changelog_begin
changelog_end

The clash is between:
- https://github.com/digital-asset/daml/pull/13147
- https://github.com/digital-asset/daml/pull/12906

The latter changed the type of `ServerError` to only get `Throwable`s, the former used `ServerError`'s to wrap a string.
2022-03-09 17:09:51 +00:00
Simon Maxen
1d40c55177
Add metering report http endpoint [DPP-855] (#13147)
* Added metering report http endpoint

changelog_begin
Added metering report http endpoint
changelog_end

* Update with review comments
2022-03-09 14:59:08 +00:00
Victor Peter Rouven Müller
6829881302
[JSON-API] Fix duplicate error logging & improve error logs messages (#12906)
* Fix duplicate error logging & improve error logs messages

changelog_begin

- [HTTP-JSON] Fix duplicate logging of errors & improved error messages slightly

changelog_end

* Simplify error handling code

* Use simple functions instead of overloading methods

* minimize diff
2022-03-09 12:43:26 +00:00
Stewart Stewart
0a52e56ca1
Make http.domain a package (Fixes #11151) (#13164)
Use a package instead of object for organization so the namespace
remains open. Note that in Scala 3, the val/type aliases in this diff
can become export statements.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Stewart Stewart <stewart.stewart@digitalasset.com>
2022-03-07 09:44:31 -05:00
Victor Peter Rouven Müller
fdb5cdae0a
[JSON-API] Normalize token parsing & remove custom parsing code (#13161)
* Normalize token parsing & remove custom parsing code

changelog_begin

- [HTTP-JSON] custom claim tokens without ledger id are now correctly recognized as such and not as user tokens

changelog_end

* Add test to check that tokens without ledger id work correctly

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/endpoints/UserManagement.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/EndpointsCompanion.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Remove the overload of jwtForParties & change how the wihtoutNamespace JWT is constructed

* Remove unnecessary code line

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2022-03-07 13:35:37 +01:00
Victor Peter Rouven Müller
0dc167fa48
[JSON-API] Remove dependency on user provided tokens for the perf runner & refactor main (#13114)
* Don't error when user tokens are provided for the perf runner

changelog_begin
changelog_end

* Better token parsing error handling & extract user id to allocate the user if so

* make the code prettier

* Fix & simplify token parsing

* fix formatting of bazel file

* Update ledger-service/http-json-perf/src/main/scala/com/daml/http/perf/Main.scala

Co-authored-by: Raymond Roestenburg <98821776+ray-roestenburg-da@users.noreply.github.com>

* correctly handle the response of createUser & don't throw if no LedgerId was found in the token

* Wrap exceptions in Futures

* Refactor Main.scala of the perf runner completely & remove dependency on user provided JWT's

* Minimize diff

* simplify code further

* Update ledger-service/http-json-perf/src/main/scala/com/daml/http/perf/Main.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update ledger-service/http-json-perf/src/main/scala/com/daml/http/perf/Main.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update ledger-service/http-json-perf/src/main/scala/com/daml/http/perf/Main.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Fix build

* Don't pass a jwt to the perf runner anymore everywhere it was used & fix ledger id to be right

* Minimize diff

Co-authored-by: Raymond Roestenburg <98821776+ray-roestenburg-da@users.noreply.github.com>
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2022-03-03 16:12:40 +00:00
Victor Peter Rouven Müller
e1ccc2f4f4
duplicate json-api tests for full-lifecycle user token testing (#12922)
* WIP

* First version that compiles (with debug print statements)

* Revert changes on the failure tests

* Make all tests pass with some major cleanup of the new code

* Remove debug prints & outcommented code

* Make decoding / encoding test also compatible with user token tests

changelog_begin
changelog_end

* Minimize diff

* Fix oracle tests

* leave note for followup

* Update ledger-service/http-json/src/itlib/scala/http/AbstractHttpServiceIntegrationTest.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2022-03-01 11:05:58 +00:00
Victor Peter Rouven Müller
300bbf86ae
[JSON-API] Fix duplicate calls to the command service while debug logging is enabled (#12699)
* Fix duplicate calls to the command service while debug logging is enabled

changelog_begin

- [HTTP-JSON] Fixed duplicate calls to the command service while debug logging is enabled.
- [HTTP-JSON] Fixed internal server errors which can occur while debug logging is enabled.

changelog_end

* Cleanup the code a bit, test still failing tho :(

* Finally fix the bug for real now

* Address review comments

* don't cast

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2022-02-11 12:53:31 +00:00
Victor Peter Rouven Müller
9e77b8dbf6
Fix logging of internal server errors for the json api (#12822)
changelog_begin

- [HTTP-JSON] Internal server errors are now properly logged again. See `issue #12887 <https://github.com/digital-asset/daml/issues/12887>`__.

changelog_end
2022-02-11 13:51:44 +01:00
Stephen Compall
dc77d35f97
move endpoint functions to separate package (#12805)
* move user management endpoint functions to separate module

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* move create and exercise endpoints to separate module

* fix copyright headers

* mark the helpers package-private

* more privacy modifiers

* import cleanup

* move Forwarded header test to unit tests

* move decodeAndParseUserIdFromToken to UserManagement

* move package endpoints to separate module
2022-02-09 11:32:31 +01:00
Stephen Compall
f63165aa76
chunked HTTP response for /v1/users endpoint (#12777)
Enriches #12766 with a chunked, lazy internal approach to
aggregating pages from the participant server, rather than
aggregating all users directly in memory before returning them.

CHANGELOG_BEGIN
- [JSON API] If ``/v1/users`` fails while retrieving users from the participant,
  it will be reported as a warning, with a partial set of users, rather than
  failing.
  See `issue #12777 <https://github.com/digital-asset/daml/pull/12777>`__.
CHANGELOG_END
2022-02-07 18:20:19 +00:00
tudor-da
14a0459f0a
SandboxNextFixture replaced by SOX-based SandboxFixture (#12654)
* SandboxNextFixture replaced by Sandbox-on-X based SandboxFixture

changelog_begin
changelog_end

* Some fixed tests

* No direct dependencies on //ledger/sandbox:sandbox and //ledger/sandbox:sandbox-scala-tests-lib

* Fix after rebase

* Rename SandboxFixture and add a missing dep

* Generate valid party names if hint is empty

* Smaller maxInboundMessageSize

* Added test for empty display name

* SandboxServer is a ResourceOwner
* Uses execution context passed as an input for resource management
* Fixes flaky FlywayMigrations issue with null Thread.currentThread.currentClassLoader
* SandboxServer simplification returns Port instead of ApiServer

* Dedicated PMAllocateWithoutDisplayName for non-Canton ledgers
* Created since Canton does not return empty display names
2022-02-07 09:25:22 +01:00
Stefano Baghino
9c35fa286e
Return all users from GET /v1/users/, paginating 1k at a time (#12766)
* Return all users from `GET /v1/users/`, paginating 1k at a time

The test runs overall for ~30 seconds, mostly due to the time required to
create 20k users.

changelog_begin
[HTTP-JSON API] Previously, a 10k limit was in place on the number of users returned by
`GET /v1/users/`. This limit has been removed and users are retrieved in chunks from the ledger
changelog_end

As of now, this builds the response in-memory, which is likely not what we want to
support long-term.

* Relax test condition due to interference with other test setups
2022-02-04 17:34:20 +00:00