* daml-on-sql: Fix references to the JAR file in the README.
* daml-on-sql: Add README info about architecture, OS, and dependencies.
* daml-on-sql: Make README headings consistent in style.
* daml-on-sql: Explain the `--ledgerid` flag in the README.
* daml-on-sql: Everything is important.
* daml-on-sql: Document parameters used to tune performance.
CHANGELOG_BEGIN
- [DAML on SQL] Document the architecture and OS we test against, native
system dependencies on Linux, and parameters for performance tuning.
CHANGELOG_END
* daml-on-sql: Document architecture requirements.
* FrontStack: Ensure that claimed invariant is maintained
The documentation of `FrontStack` claims that the head of the
`FQPrepend` is never empty but there's a way to violate this claimed
invariant.
This PR makes sure the invariant is maintained and then uses the
invariant to remove a redundant `else`-branch.
CHANGELOG_BEGIN
CHANGELOG_END
* Remove initials from NOTE
CHANGELOG_BEGIN
CHANGELOG_END
* Add evaluation order tests for foldl and foldr
I'd like to have a play with the implementation of `foldl` and `foldr`
in Speedy in order to improve their performance. Doing so without any
tests for the evaluation order of those builtins, is too scary for me.
Thus, let's kick off the effort by adding some tests.
Except for the usual case where we need to test that we're running over
the list in the right order, we also need to test for the case where
the step function only consumes one argument and the case where the
accumulator is a function. In the latter case we need to make sure we
don't evaluate the argument applied to the final accumulator too early.
We're also very short on the semantics of `foldl` and `foldr` in the
DAML-LF spec but I leave that for another PR.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix copy/paste error
Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>
* Minor fixes
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>
* DAML Engine: Speed up updates of multiple fields in a record
Currently, updating multiple fields of a single record is not very
efficient. If you update, say, `n` fields, as in
```haskell
r with f_1 = ...; ...; f_n = ...
```
`n-1` intermediate records are created and almost immediately
discarded. Each creation of an intermediate record involves shallowly
cloning an array of size `m`, where `m` is the number of fields of the
record type. This does not only cost computation time but also a lot of
avoidable memory allocations. Overall, the cost of such an update is
`O(m*n)` in terms of time and memory.
This PR changes the DAML-LF interpreter such that multiple updates of
the same record are batched together. This avoids the creating of any
intermediate records but only creates the final record resulting from
the updates. This reduces the time complexity to `O(m+n)` and the
memory complexity to `O(m)`.
The code path for updating a single record field remains unchanged
(for now) and hence won't suffer from the additional complexity that
is required for the batching described above.
I've benchmarked this change by creating a single record with `m`
fields and updating all `m` fields in one single update expression,
for `m = 10` and `m = 20`. In both cases, the benchmarks showed a
speedup of ca. 2.5x.
This is part of #5766.
CHANGELOG_BEGIN
- [DAML Engine] Record update expressions of the form
`R with f_1 = E_1; ...; f_n = E_n` are now run as batch updates
in order to improve runtime performance and avoid unneccesary
memory allocations.
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Remy <remy.haemmerle@daml.com>
Co-authored-by: Remy <remy.haemmerle@daml.com>
This was not only unnecessarily duplicated, it also had a bug where
`--crt` behaved like `--pem` instead of setting the cert chain.
I didn’t add new tests since it seems like the wrong place to test
config parsing of a library. We do have tests for TLS in general for
both DAML Script and DAML Triggers.
changelog_begin
changelog_end
java.lang.NullPointerException:
at com.daml.ports.PortLock$Locked.unlock(PortLock.scala:55)
at com.daml.ports.PortLock$.lock(PortLock.scala:41)
at com.daml.ports.LockedFreePort$.find(LockedFreePort.scala:15)
at com.daml.lf.engine.trigger.TriggerServiceFixture$.$anonfun$withTriggerService$1(TriggerServiceFixture.scala:65)
CHANGELOG_BEGIN
CHANGELOG_END
* Moving `Statements.discard` from //ledger-server/http-json into //libs-scala/scala-utils
changelog_begin
changelog_end
* Add new module to the published artifacts
* `com.daml.scalautil` instead of `com.daml.scala.util`
@S11001001: That's because if this is in classpath and you import com.daml._,
you have a different scala in scope than the one you expect.
This is a spinoff from IDE support for DAML Script #6929. There I need
cannot quite use `run` from ScenarioRunner since I need to handle
results differently but I don’t want to replicate all the visibility
and authorization logic from `lookupContract` and `lookupKey`.
changelog_begin
changelog_end
* Use Bazel builtin pkg_tar rule
* Use @rules_pkg//:pkg.bzl%pkg_tar
The pkg_tar rule builtin to Bazel has been deprecated.
See https://docs.bazel.build/versions/master/be/pkg.html
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* daml-lf/parser: Add the ability to parse location annotations
Currently, the DAML-LF parser we use for testing in Scala land does not
understand location information and has in particular no way to produce
`ELocation` AST nodes. Having these nodes will be important in a test
for a feature I'm currently working on in order to demonstrate that my
Speedy AST transformation can actually look through location
information properly.
This PR adds a rule to the parser to allow for parsing location
information and producing `ELocation` nodes.
CHANGELOG_BEGIN
CHANGELOG_END
* Address Nick's feedback
CHANGELOG_BEGIN
CHANGELOG_END
* Strip ELocation nodes in simplifier rules.
Uses stripLoc in the patterns used by the simplifier. This change
reduces the size of the generated code by 1.5% for a very large project,
meaning we had rules (like `let x = e in x`) that weren't trigerring
because of the `ELocation` nodes that weren't being properly ignored.
changelog_begin
changelog_end
* More stripLoc
This code has a few problems: the auth service test mode is not enabled, the
admin ledger should be separate from main ledger if auth is enabled, and the
retry strategy not executed.
The auth service is not used in these tests yet and we have an example
of how to do it correctly in the auth service fixture (and auth service
client tests). We can try to reuse that fixture instead or copy the
code over when necessary.
changelog_begin
changelog_end
* Trigger service tests: Make sure toxiproxy server is running before connecting client
changelog_begin
changelog_end
* Undo infuriatingly wrong camel case
* event_sequential_id arithmetic
* add minPageSize fetch size to fallbacks
* use row_id arithmetic for single-party and multi-party tree queries, wildcard templates
- as a fallback, try to grab 10 or (pageSize / 10) rows with LIMIT,
whichever is larger
* add sequences of arnorm queries with parsed results
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* remove an extra LIMIT
* reformat SQL statements
* add newlines to other tx tree queries
* group by does nothing with this order by setting
* reformat all queries in EventsTableFlatEventsRangeQueries
* group by doesn't matter when you have order by and no aggregate exprs
* factor out the "faster read, then safer read" pattern from tree events
* make flat events all return SqlSequence with embedded parser
- with a path for EventsRange.readPage
* make flat transactions singleWildcardParty use arithmetic and fallback
* rename Fast to ByArith, Slow to ByLimit, because speed may vary
* missed Fast/Slow
* replace the other flat transaction queries with limited versions
replace
(?s)^ SQL"""(.*?)(\$\{range.endInclusive\})(.*?) limit \$pageSize"""
with
FrqK.ByArith(\n fasterRead = guessedPageEnd => SQL"""\n $1\$guessedPageEnd$3""",\n saferRead = minPageSize => SQL"""\n $1$2$3 limit \$minPageSize"""\n )
which is obviously better than being able to factor common parts of SQL
queries, so naturally I agree with anorm lacking a doobie-like append.
* remove readUpperBound, stray merge conflict inclusion
- thanks @leo-da for pointing it out
* rename SqlSequence.Elt to SqlSequence.Element
- suggested by @stefanobaghino-da; thanks
* rename FrqK to QueryParts
- suggested by @stefanobaghino-da; thanks
* reformat flatMap chain
* don't rescan first page; eliminate duplicate SQL exprs
- overload 'range' to mean "first page" then "search space after first page"
- page sizes are always safe to interpolate directly into SQL, as ints
(?s)^ fasterRead = guessedPageEnd => (.*?)\$guessedPageEnd(.*?)""",\n saferRead = minPageSize => SQL""".*?"""
read = (range, limitExpr) => $1\${range.endInclusive}$2 #\$limitExpr"""
* FilterRelation is used in a private[dao] context
- you won't get a warning for this because aliases are expanded before
this is checked, so the method can still be called, you simply can't
use the same type name used in the written signature
* generated sequences of transactions with different matching frequencies
- different occurrences of the matched transactions cause different SQL
queries to be used, so we try to exercise all of them
* generalize storeSync's traverse to let multiple tests run in order
- thanks to @leo-da for the inspiration
* a way for singleCreates to be slightly different
* test that matched transaction count and the specific offsets match
* test more code paths for flat events
The docs talked about `emitCommands` while the code talked about
`dedupExercise`. I’m increasingly leaning towards recommending
`emitCommands` for everything so I’ve switched the code instead of the
docs.
changelog_begin
changelog_end
* triggers: Use `FreePort.find()`.
* ports: Move `LockedFreePort` from postgresql-testing for reuse.
* triggers: Use `LockedFreePort` to avoid race conditions.
* ports + triggers: Move common port testing into the ports library.
CHANGELOG_BEGIN
CHANGELOG_END
This is running on Azure’s agents and we just call curl so there is
really no need for dev-env (as evidenced by the fact that the message
got sent despite dev-env failing).
changelog_begin
changelog_end
* Add a test for the conversion of (nested) record updates to DAML-LF
The new tests check that we generate the DAML-LF we intend to generate,
namely code using `ERecUpd` AST nodes rather than calls to the
`setField` function.
CHANGELOG_BEGIN
CHANGELOG_END
* Use get_value_name instead of get_dotted_name
CHANGELOG_BEGIN
CHANGELOG_END
* Evaluation order for exercise & other updates.
This adds more DAML tests and error cases for, finishing off the update
interpretation for now. I believe this is roughly the limit of what we
can test directly in DAML, and the rest is up to DAML-LF testing.
For example, there's no way to test that the contract inactivity check
in EvUpdExercInactive happens before the evaluation of the controller
expression, since there is no way to generate an 'exercise' with actors
from DAML, as far as I could tell, and 'exercise_without_actor' always
evaluates the controller before passing it off to 'exercise'.
For the same reason, the EvUpdExercBadActor check seems impossible to
trigger from DAML itself. The closest is an authorization test that
happens during submit (which I added a test for, because it is kinda
relevant to evaluation order of scenarios, even though scenario
interpretation isn't really important to the spec).
So at the very least we should add LF tests for those two cases.
changelog_begin
changelog_end
* Apply Moritz\'s suggestions
* Add test with fetch of consumed contract
* Added ledger writer that chooses between instances based on estimate interpretation cost.
CHANGELOG_BEGIN
CHANGELOG_END
* Code tidying.
* Delegate to pre-executing writer in case thershold is set to 0.
* Added ability to change metrics.
* Added metrics.
* Code tidying.
* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/participant/state/kvutils/api/InterpretationCostBasedLedgerWriterChooser.scala
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
I like keeping track of what happens in the areas I’m responsible
for. I’ve also removed Shayne since it doesn’t make sense to keep him.
changelog_begin
changelog_end
* add -Xlint:doc-detached
- reverts 1feae964e3 from #6798
* attach several scaladocs where they'll actually be included
* no changelog
* attach several more scaladocs where they'll actually be included
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
This fixes 3 issues:
1. Switch the order in which we assign to `$proc` and
`waitOn`. Without this the cleanup will not do anthing if `beforeAll`
fails since the variable has not been assigned. This results in jest
hanging forever until we hit the Bazel timeout.
2. Increase the timeout on `beforeAll` since we occasionally hit this
on CI.
3. Allocate 2 CPUs to reflect the number of resources required by this test.
changelog_begin
changelog_end
* Change error code for invalid offsets for transaction stream and completion stream requests
* Expanded application architecture docs on how to build application with ledger api failover capabilities.
Fixes#6842.
CHANGELOG_BEGIN
- [Ledger API] The error code for requesting a transaction stream
with an offset beyond the ledger end changed from INVALID_ARGUMENT
to OUT_OF_RANGE. This makes it easier to handle scenarios where
an application fails over to a backup participant which hasn't
caught up with the ledger yet.
- [Ledger API] The command completion service now validates the offset and
returns the OUT_OF_RANGE error if the request offset is beyond the ledger end.
- [Documentation] Added a section on how to write DAML applications
that can fail over between multiple eventually consistent Ledger API endpoints
where command deduplication works across these Ledger API endpoints, which
can be useful for addressing HA and/or DR scenarios.
CHANGELOG_END
* Expose type synonyms in data-dependencies.
This PR converts (non-unit) type synonyms during LF conversion, and then
exposes them via data-dependencies. This is possible only when using an
LF version that supports type synonyms (LF version >= 1.7), the
type synonym isn't of unit type (because it clashes with empty
typeclasses, and the (fully applied) type synonym has kind *
(e.g. you can't define a * -> * synonym in LF).
This fixes issue #6306
changelog_begin
- [DAML Compiler] The DAML compiler will now compile type synonyms
(``type X = Y``) into the DAR, whenever possible, and will expose
these synonyms via data-dependencies.
changelog_end
* Add more tests
* Fix formatting
We currently use two different mechanisms for breaking loops in the
`Serializable` companion objects generated for each type:
1. Explicit thunks of the form `() => Decoder<A>`, e.g., in the
definition of the `Serializable` interface itself.
2. the `lazyMemo` combinator, e.g. in enum types, `Optional`s and lists.
This is inconsistent and leads to the introduction of more loop
breakers than actually needed.
This PR changes the situation to use the `lazyMemo` combinator as the
only loop breaker. To this end, we change the definition of the
`Serializable` interface and related similar interfaces to use `Decoder`
directly rather than the thunked up version. This is a *breaking change*
of the affected interfaces. However, the libraries `@daml/ledger` and
`@daml/react` are updated accordingly and hence nothing breaks when
using `daml2js` in combination with these libraries. Furthermore, the
exact definition of the decoder types is considered an implementation
detail since we don't want to tie ourselves long-term to the use of the
`json-type-validation` library, which is mostly unmaintained.
Using `lazy` as the only loop breaker effectively pushes all loop
breakers to the top-level and allows for removing those nested more
deeply in types.
This change should not negatively impact the performance of the
decoding process since it only replaces thunks by memoized thunks and
removes a few calls to `lazyMemo` completely. In fact, we should gain
performance since the decoders in the companion objects are now
memoized.
CHANGELOG_BEGIN
CHANGELOG_END
Currently, if you have a record type `T` with a field of type, say,
`Optional S` and you're decoding a list of type `[T]`, then the decoder
for `S` has to be reconstructed for each element of the list. This is
because the `lazy` combinator from the `json-type-validation` does not
memoize the decoder it receives as a thunk.
This PR adds a new combinator `lazyMemo` which behaves like `lazy` but
also memoizes the decoder on its first invocation. All use sites of the
old `lazy` combinator are then replaced with `lazyMemo`.
We could consider upstreaming `lazyMemo` but I'm not sure how much
effort this is given that `json-type-validation` seems to be in
maintenance mode rather than active development.
CHANGELOG_BEGIN
CHANGELOG_END
We’ve had a few confused users run into issues because of
this. `fsevents` (which is basically impossible to avoid as a
dependency) requires NodeJS 8.16 but for some reason Ubuntu 18.04
sticks to the unsupported 8.10.
changelog_begin
changelog_end
* DAML REPL :show imports
changelog_begin
- [DAML REPL] You can now list the currently imported modules using
the REPL command ``:show imports``.
changelog_end
* Accept \r\n on Windows.
* REPL functests use text mode file for stdin/out
So that `\r\n` will be mapped to `\n` automatically on Windows.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* daml-on-sql + sandbox: Remove unused Bazel `load` declarations.
* daml-on-sql: Build a Docker image.
* daml-on-sql: Move the example Kubernetes file over from Sandbox.
* Add a changelog entry that was dropped.
CHANGELOG_BEGIN
* [DAML on SQL] DAML on SQL is now available with every release,
from the GitHub Releases page.
CHANGELOG_END
* Upgrade the Java Docker base image to the latest version.
After the node resets this should hopefully not be necessary
anymore (we still had an issue this morning but I believe all nodes
that hit the issue also got the fix and if not, I’ll schedule a
targetted clean --expunge). I’ve also added node_modules to
.bazelignore to match the other node_modules directories.
changelog_begin
changelog_end
Not quite sure why this didn’t fail on the PR but now it complains
that the version of @bazel/typescript and rules_nodejs are
incompatible.
changelog_begin
changelog_end
* Log all authorization errors
CHANGELOG_BEGIN
- [Ledger API Server] The ledger API server now prints detailed log messages
whenever a request was rejected due to a failed
authorization.
CHANGELOG_END
