Commit Graph

3857 Commits

Author SHA1 Message Date
Samir Talwar
499bb924cf
kvutils: Prefix metrics with "daml.". (#5289)
Just to get them consistent with the others.

CHANGELOG_BEGIN
- [Ledger Integration Kit] Prefixed all metrics with "daml." for
  consistency.
CHANGELOG_END
2020-03-31 08:18:29 +00:00
Andreas Herrmann
89a9f5c7d2
tarball reproducibility (#5258)
* integration-tests reproducibility

* package-app reproducibility

* Make remaining tar czf reproducible

* package-app

CHANGELOG_BEGIN
CHANGELOG_END

* Reproducibility of remaing tar invocation

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-03-31 10:09:52 +02:00
Leonid Shlyapnikov
703b588384
Error and disconnect when multiple requests coming over the same WebSocket connection (#5288)
* minor refactoring in preparation for the next test case

* allow only one input request + test, WIP

* allow only one input request + test, WIP

* Error when multiple requests coming over the same websocket connection

This never worked, we never returned anything when 2nd request received,
now we return an error and drop the connection.

changelog_begin
changelog_end

* update docs

* Fixing copyright header

* changing test description

* trying to clean wsMessageHandler up

* Reverting getTransactionSourceForParty changes
2020-03-30 18:08:54 -04:00
Samir Talwar
2e9d7adfa9
sandbox|kvutils: Use MetricRegistry.name to create metric names. (#5274)
* kvutils: Use `MetricRegistry.name` to create metric names.

String interpolation is 4TL.

* sandbox: Make the metric names of the services consistent with kvutils.

CHANGELOG_BEGIN
- [Sandbox] Move the service metrics from ``daml.sandbox.indexService``
  and ``daml.sandbox.writeService`` to ``daml.services.index`` and
  ``daml.services.write`` respectively. This brings Sandbox in line with
  the Ledger Integration Kit.
CHANGELOG_END

* sandbox: Use `MetricRegistry.name` to create metric names.

String interpolation is 4TL.
2020-03-30 20:14:06 +00:00
Shayne Fletcher
de03006470
Use sandbox '--port-file' (#5284)
* increase timeout to eternal

changelog_begin
changelog_end

* Switch to using --port-file for sandbox

changelog_begin
changelog_end
2020-03-30 15:24:42 -04:00
Shayne Fletcher
2722e7ce95
Move 'withEnv', call it from daml2ts tests (#5276)
* Move 'withEnv', call it from daml2ts tests

changelog_begin
changelog_end

* Fix withEnv call to ensure that TASTY_NUM_THREADs is set

withEnv replaces the whole environment so we need to set everything we
care about.

* withEnv replaces the whole environment so we need to set everything we
care about.

* Apparently applying the same fix has destabilized Windows

* Try even harder to get daml assistant tests passing on Windows again

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-03-30 17:20:47 +00:00
Martin Huschenbett
72e0cb6374
@daml/react: Remove unused dependencies (#5283)
The `package.json` lists some dependencies which used to be
dependencies in the past but aren't anymore.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-30 16:53:23 +00:00
Moritz Kiefer
a84311ac39
Change withEnv to not clear the whole env (#5281)
The way we use `withEnv` it is really intended to set a few
environment variables in an already existing environment instead of
clearing everything before. This PR changes it to do only that.

changelog_begin
changelog_end

closes #5280
2020-03-30 12:21:25 -04:00
Remy
3fc1175436
Engine: force recent version for serializing transactions and values. (#5232) 2020-03-30 16:29:18 +02:00
Ognjen Maric
5be5036f27
Engine: support reinterpreting LookupByKey nodes (#5271)
Fixes #5264

CHANGELOG_BEGIN

- [Engine] Support reinterpreting LookupByKey nodes

CHANGELOG_END
2020-03-30 15:13:43 +02:00
Samir Talwar
66a7fc6384
kvutils/app: Publish metrics someplace. (#5267)
* kvutils/app: Publish JVM metrics.

* kvutils/app: Sort the metrics prefixes.

* kvutils/app: Allow for specifying a metrics reporter as a CLI argument.

No changelog as this is hidden.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Remove an unnecessary string interpolation mark.

Co-Authored-By: Miklos <57664299+miklos-da@users.noreply.github.com>

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
2020-03-30 13:00:42 +00:00
Stefano Baghino
746b774e82
Add canonical string representation for identifiers (#5270)
* Add canonical string representation for identifiers

An identifier is represented as a string as a package identifier and a qualified name separated by a colon.

changelog_begin
changelog_end

* Simplify explicit type signature for HexString

Thanks @remyhaemmerle-da

Co-Authored-By: Remy <remy.haemmerle@daml.com>

* Fix hashing test expected output

* Fix key hasher test expected output

Co-authored-by: Remy <remy.haemmerle@daml.com>
2020-03-30 12:55:11 +00:00
Samir Talwar
28b21d07c7
ledger-on-sql: Squash the migrations. (#5265)
Before this reaches production-ready status, it's time to squash the
migrations. This will improve startup performance a little.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-30 12:51:33 +00:00
Moritz Kiefer
19496482f6
Replace damlpatterns tarballs by a proper template (#5273)
There is no reason why patterns should be a tarball and the current
tarballs are broken anyway. Why do we even have 2 …

changelog_begin
changelog_end
2020-03-30 12:35:22 +00:00
Gary Verhaegen
38a5fea7a0
tweak periodic-killer (#5268)
1. Google says the instance is currently overutilized and suggests
   g1-small as a more appropriate size.
2. It occurred to me that the reason no error was logged might be that
   we lose them, so explicitly redirecting stderr too.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-30 14:12:14 +02:00
Martin Huschenbett
e11fced8c9
daml2ts: Update the TypeScript version (#5269)
Update the TypeScript version we're using in `daml2ts` and its support
libraries. Also update the versions of ESLint and its TypeScript
plugin we're using for linting the generated code. (And remove more
occurrences of `HashMap Text Text` at the same time.)

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-30 11:27:40 +00:00
Martin Huschenbett
dc617345c9
JSON API: Fix docs on WebSocket setup (#5253)
Using `on` to add event listeners is a Node.js idiom. In the brosers,
it's `addEventListener`, which works in Node.js too.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-30 13:19:25 +02:00
Moritz Kiefer
c0e28d83ee
Turn warnings for module name/record name mismatches into errors (#5266)
* Turn warnings for module name/record name mismatches into errors

The module name warning existed for ages. We started warning about the
record name mismatch in 0.13.55 so I’d like to turn it into an error
before 1.0

changelog_begin

- [DAML Compiler] File names must now match up with module names. This
  already produced a warning in previous releases.

- [DAML Compiler] It is now an error to define a record with a single
  constructor where the constructor does not match the type
  name. To fix the error you need to change ```data X = Y { … }``` to
  ```data X = X { … }```.  This restriction only applies to
  single-constructor records. Variants and enums are not affected.
  This already produced a warning in 0.13.55.

changelog_end

* Fix integration tests

* Fix docs

* Fix lsp tests
2020-03-30 10:21:53 +00:00
Moritz Kiefer
d70fa63a21
Add a proper scalatest test suite for triggers (#5250)
This PR adds a new test suite for DAML triggers based on scala test
rather than the client_server_test macro + a custom main. This seems
much nicer than the client_server_test (we get a lot of useful stuff
from scalatest, e.g., useful output of assertion failures, things
don’t blow up after the first test failure, …).

This PR only ports over a small fraction of the tests to make review
easier. The plan is then to port over everything and kill off the
existing test stuff once everything is ported over.

changelog_begin
changelog_end
2020-03-30 11:42:09 +02:00
Moritz Kiefer
a37d898e77
Remove dependency and data-dependency arguments from daml-helper (#5262)
Those were originally used for `damlc migrate`. However, that command
no longer exists and the only call site of `runNew` sets those
arguments to an empty list.

changelog_begin
changelog_end
2020-03-30 10:40:34 +02:00
Moritz Kiefer
1e296aa91f
Whitelist fetch-dar command in assistant telemetry (#5261)
changelog_begin
changelog_end
2020-03-30 10:39:55 +02:00
Remy
4cb3ea5c7c
Ledger: clean up contract id seeding (#5260)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-30 10:13:42 +02:00
Robert Autenrieth
f8cd628c2f
Remove old time model from ledger config (#5216)
* Remove old time model from ledger config

CHANGELOG_BEGIN
- [Ledger API] Fields related to the old ledger time
  model have been removed from the configuration
  management service and the ledger configuration service.
CHANGELOG_END

* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/tests/LedgerConfigurationService.scala

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2020-03-30 08:46:22 +02:00
mziolekda
0f3c0d66d8
docker image for ledger-on-memory (#5256)
* docker image for ledger-on-memory

CHANGELOG_BEGIN
CHANGELOG_END

* Apply suggestions from code review

Applying Samir's suggestions to the README file

Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>

* Update ledger/ledger-on-memory/README.md

Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-03-29 10:54:12 +02:00
Robert Autenrieth
d57f78c9ef
Remove useless timeProvider in quickstart (#5247)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-28 13:25:29 +01:00
Robert Autenrieth
0a83309f1b
TTL cleanup (#5246)
* Remove documentation for a removed option

CHANGELOG_BEGIN
- [DAML Integration Kit] The CLI option command-submission-ttl-scale-factor was
  removed, as the LET/MRT/TTL fields have recently been removed
  from the command submission service.
CHANGELOG_END

* Minor renaming
2020-03-28 00:13:08 +01:00
Gary Verhaegen
7e960eb454
log periodic reboots (#5235)
It appears that most of our Windows machines have not been rebooted
since Tuesday 24. We detected this because one of them has run out of
disk space.

This is not good, but what's worse is I currently have no idea what
could be going wrong, and we are not logging anything at all in the
current setup, so even ssh'ing into the machine provides no insight.

This PR hopefully addresses that by:

1. Redirecting the outputs of the script to a file, and
2. `tail`iing that file from the startup script, so the logs will appear
   directly in the GCP web console. (This is what we currently do for
   the Azure agent logs on Linux.)

This PR also tells the script to not stop on the first failed machine
and keep trying.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 21:35:49 +01:00
Gary Verhaegen
525c4d605a
prepare playground for perf testing (#4520)
* prepare playground for perf testing

The goal of this commit is to provide me with a playground to test out
various GCP machine configurations and assess what impact they have on
build performance. I need separate pools for that as changing the
machine type requires a machine restart, which would be disruptive to
any running job if I did that in the existing pools.

The first commit in this PR does the following:

- Deactivate Bazel caching.
- Create two new machine groups (of size 1) for Linux and Windows pools.
- Disable all jobs that are not building the project on Linux or
  Windows.
- Change the Linux and Windows jobs to use the new pools.

The plan is for me to manually change the machine types in the new pool
and rerun the build repeatedly. Once I believe I have the information I
need, the expected end result for this PR is only a change to the
machine type (or no change at all if the current machine size is already
optimal). All other changes will be undone and the additional pools will
be destroyed.

Note: as of the opening of this PR, the two new pools have not been
created yet at the Azure level (this is a separate, manual step not
managed by Terraform) and the Terraform files have not been applied.

See corresponding PR (#4520) for full details. The gist of it is:
- We need at least 28GB of memory to build at all, 30 is better.
- Additional memory beyond 30GB does not seem to provide any benefit.
- CPU speed matters more than number of CPU cores, but both help.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 20:14:54 +01:00
Remy
1b37f6c482
DAML-LF: redesign absolute contract ids (#5207)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 19:07:42 +01:00
nickchapman-da
aa53c30de1
New command: daml ledger fetch-dar (#5225)
* New comamnd: daml ledger fetch-dar

`daml ledger fetch-dar [PID] [PATH]`

Download a `Package` and it's dependencies from a ledger, given a root `packageId`, and re-construct a valid `.dar` file. Addresses issue #5037.

The original package names are not reconstructed.

CHANGELOG_BEGIN
CHANGELOG_END

* address some comments

* fix spello

* attempt: recoverPackageName

* recover main package name & version

* Try to fix integration tests on Windows

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-03-27 18:04:14 +00:00
Moritz Kiefer
e42111b31f
Fix template name in multiparty agreement docs (#5255)
fixes #5239

changelog_begin
changelog_end
2020-03-27 16:34:57 +00:00
Samir Talwar
7ef0972856
sandbox: On reset, wait for the API server to start before replacing it. (#5251)
* sandbox: On reset, wait for the API server to start before replacing it.

Hopefully this addresses the flickering behavior we're seeing in CI.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Scale the retries in ResetServiceIT along with everything else.
2020-03-27 16:29:56 +00:00
Moritz Kiefer
53994c807a
Disable tuple-section and lambda-case hints (#5229)
Adding more syntax is just confusing for beginners so these are not
particularly helpful and I would argue that they aren’t that useful in
general even once you move past the beginner stage.

changelog_begin
changelog_end
2020-03-27 17:16:25 +01:00
Stefano Baghino
9522c6e4b2
Allow to have both verbose and succinct row parsers (#5248)
* Allow to have both verbose and succinct row parsers

This will be used when we need to stream transactions and not individual
events.

changelog_begin
changelog_end

* Succinct parser should have verbose = false
2020-03-27 17:00:36 +01:00
Martin Huschenbett
454bd127a0
daml2ts: Don't use HashMap Text Text anywhere (#5252)
As I said previously, all instances of the dreaded `HashMap Text Text`
can go away if write our code more direct. Here we go.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 15:51:42 +00:00
Martin Huschenbett
b1e6298ec6
JSON API: Slightly improve the docs for the streaming endpoints (#5234)
Semicolons at the end of every line seem to be good style. Logging to
the console does not help with the example. The `WebSocket` class is
built-in in the browser and you only need a package like `ws` on
Node.js.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 15:17:55 +00:00
Remy
20218e4175
DAML-LF: restrict length of parties (#5237)
CHANGELOG_BEGIN
* [DAML-LF]. *Breaking* Restrict length of parties
CHANGELOG_END
2020-03-27 14:52:43 +01:00
mziolekda
ce62bfb596
Support for large grpc messages in ledger-on-memory (#5230)
* Support for large grpc messages in ledger-on-memory

* adress review comments

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 14:00:24 +01:00
Stefano Baghino
4543917f8b
Refactor query preparation for flat events (#5241)
* Refactor query preparation for flat events

This will allow to share code with the stream endpoints.

changelog_begin
changelog_end

* Allow for multiple requesting parties

* Safer approach to build queries

Thanks @SamirTalwar-DA

Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>

* Fix compilation errors

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-03-27 12:41:08 +00:00
Moritz Kiefer
9a1ddc8fa4
Remove no assistant tests from integration tests (#5240)
There was a brief period of time where `daml build` did not work
outside of the assistant. When we fixed that we added it to the
integration tests since there was no other test suite that used damlc
as a binary (as opposed to using it as a library which runs through
different codepaths). However, in the meantime we have tons of tests
all over the place (e.g. the packaging tests) that call `damlc build`
outside of the assistant so these tests serve no purpose.
Also they are somewhat confusing since the point of the integration
tests is to test an installed SDK whereas these tests do not need an
installed SDK (that’s the whole point of those tests).

changelog_begin
changelog_end
2020-03-27 13:11:10 +01:00
Martin Huschenbett
b398272fee
daml2ts: Don't reinvent assertInfixOf in tests (#5245)
Instead use the version from `DA.Test.Util`.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 11:58:53 +00:00
Martin Huschenbett
98f37dbe48
daml2ts: Make tests more concise (#5243)
We do not need to have templates in the `.daml` files we generate
during these templates, any serializable type is good enough. Thus,
let's make the tests shorter by using a simple serializable type
instead of templates.

Also make the tests easier to debug when expected failures of `daml2ts`
do not happen in the anticipated way.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 11:06:41 +00:00
Moritz Kiefer
0c5166dd8c
Support running DAML script over the JSON API (#5211)
* Support running DAML script over the JSON API

This is still in a somewhat messy state and some things don’t
work (documented in a comment) so I deliberately didn’t add this to
the documentation. However, there are tests and the PR is already
pretty large so I’d like to move the rest to separate PRs to not turn
this into more of a review nightmare than it already is.

changelog_begin
changelog_end

* Address review comments
2020-03-27 11:04:40 +00:00
Moritz Kiefer
3a7da97825
Move SDK installation to a tasty resource (#5238)
This ensures that -l and -p work properly in the integration tests
since they no longer depend on the order.

There is lots of other crap to cleanup in those tests but I’m trying
to keep it to small steps.

changelog_begin
changelog_end
2020-03-27 11:59:41 +01:00
Martin Huschenbett
d48698e43b
daml2ts: Fix the A:X vs A.B bug (#5226)
* daml2ts: Fix the A:X vs A.B bug

Currently, the TypeScript generated by `daml2ts` does not compile when
there are a modules `A` and `A.B` which both contain serializable types.
(Note: there can't be a type called `B` in `A` due to the name collision
check.)

Since fixing this within the current layout using TypeScript
namespaces would be very challenging since we would need to inject
subnamespaces into existing namespaces. Thus, we switch the code
generation to use ES2015 modules. Since TypeScript namespaces are
considered (somewhat) deprecated, this is a good move anyway.

As a result of this change to ES2015 we need to write `index.ts` files
at each node of the module hierarchy. These `index.ts` files reexport
the module at the current node (if present) and all child modules.

At the same time we also remove the default export of the top-level
`index.ts` file. It is not clear if they are actually useful and to
add them to the new scheme requires a technique that feels like a hack.

To give an example, assume we have modules `A` and `A.B.C`. We now write
the following directory structure:
```
src/
+- index.ts             <- reexports A
+- A/
   +- index.ts          <- reexports module.ts and B
   +- module.ts
   +- B/
      +- index.ts       <- reexport C
      +- C/
         +- index.ts    <- reexports module.ts
         +- module.ts
```
If you import the package like
```typescript
import * as foo from '@daml.js/foo-1.0.0';
```
you can acces the `A` and `A.B.C` modules as `foo.A` and `foo.A.B.C`.

CHANGELOG_BEGIN
CHANGELOG_END

* Adapt build-and-lint-test

CHANGELOG_BEGIN
CHANGELOG_END

* Add new test

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 10:27:18 +01:00
Gerolf Seitz
050f049e16
Add docs for the ledger time model to the app arch guide. (#5118)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 09:46:28 +01:00
Rohan Jacob-Rao
9d73aec229
GSG: update code samples (#5224)
* Update code in before section

* Update code in 'after' section, i.e. for messaging

changelog_begin
changelog_end

* Typo with END taggit st
2020-03-27 09:16:15 +01:00
Gary Verhaegen
1872c668a5
replace DAML Authors with DA in copyright headers (#5228)
Change requested by Manoj.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 01:26:10 +01:00
Gary Verhaegen
9f3900d870
pin copyright headers (#5227)
When files should be immutable, they should also be protected from the
copyright headers script.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-26 23:30:23 +01:00
Stephen Compall
332d35a347
stop filtering "phantom archives" when it won't work; reenable it on request (#5003)
* ContractKeyStreamRequest with different types for whether offset-preceded or not

- should replace EnrichedContractKey as the WS StreamQuery type

* add the ContractKeyStreamRequest layer everywhere

* split StreamQuery parse from the other steps

* make StreamQuery type depend on the input JsValue

* only StreamQueryReader is a typeclass now

* scalafmt

* wrong type arg

* letting the request data and phantom archive removal choice flow

- from work with @leo-da

* finish threading request-derived phantom state to removal flow

- from work with @leo-da

* make it clear that hints are contract IDs in StreamQuery

- from work with @leo-da

* treat StreamQueryReader's type parameter fully as a phantom

- from work with @leo-da

* remove unused type alias

- from work with @leo-da

* test fetch resume, with and without various offsetHints

* document offsetHints

* missing ` in rst

- thanks @hurryabit

* rename offsetHint to contractIdAtOffset

CHANGELOG_BEGIN
- [JSON API - Experimental] New field ``contractIdAtOffset`` for fetch-by-key streams
  to restore proper archive filtering.
  See `issue #4511 <https://github.com/digital-asset/daml/issues/4511>`_.
CHANGELOG_END

* we never unify the two ContractKeyStreamRequest types

* doc update for contractIdAtOffset
2020-03-26 15:31:27 -04:00