* HOTFIX: increasing 30.sec timeout to 300.sec
this to fix SQL-backed ledger tests that currently keep timing out.
It is NOT really a fix, but a temp work-around. Tried to get rid of
this timeout, but it propogates all over the place. Need a separate
story to address this
* HOTFIX: adding a link to the issue
Compared to older versions of the Engine, we do A LOT more validation
now. So when starting with a fresh engine after each reset (via
the ResetService), we also repeat the validation of the loaded packages
again and again. This is VERY expensive, especially for large DAML
packages.
Luckily, the specification of the ResetService states the following:
// Resets the ledger state. Note that loaded DARs won't be removed --
// this only rolls back the ledger to genesis.
This means that we can re-use the same Engine object and benefit from
not having "re-compile" packages via
ConcurrentCompiledPackages#addPackage, more specifically we can omit
this line: 1f2246c822/daml-lf/engine/src/main/scala/com/digitalasset/daml/lf/engine/ConcurrentCompiledPackages.scala (L56)Fixes#178
* fix various conversion functions from string to Decimal
Fixes#399.
This fixes a critical bug -- since:
* The DAML-LF spec specifies that the scale of `Decimal` is 10 --
that is, there are at most 10 digits past the decimal point:
<79bbf5c794/daml-lf/spec/value.rst (field-decimal)>.
* However, the code converting from the string representation that
we get on the wire was _not_ performing this check. This is due
to two reasons:
- `Decimal.check` is a function that checks whether a given
`Decimal` is within the upper and lower bounds of what the
DAML-LF spec specifies, but crucially it does _not_ check that
the scale is not exceeded:
<79bbf5c794/daml-lf/data/src/main/scala/com/digitalasset/daml/lf/data/Decimal.scala (L31)>.
This behavior is correct in some cases (more on that later),
but not in others. Crucially, `Decimal.fromString`, which was
supposed to check if a decimal string literal is valid, used
this function, which means that it accepted string literals
containing numbers out of the required scale, rounding them to
make them fit within the scale. This function has been renamed
to `Decimal.checkWithinBoundsAndRound`, and a new function
`Decimal.checkWithinBoundsAndWithinScale` has been added, which
fails if the number provided has data not within the scale.
`Decimal.fromString` now uses
`Decimal.checkWithinBoundsAndWithinScale`.
- `ApiToLfEngine.parseDecimal` assumed that `Decimal.fromString` _did_
fail when passed numbers that were in any way invalid, and
moreover did _not_ use the result of `Decimal.fromString`, but rather
re-parsed the string into an unconstrained `BigDecimal`:
<79bbf5c794/ledger/ledger-api-common/src/main/scala/com/digitalasset/platform/participant/util/ApiToLfEngine.scala (L96)>.
The reason for the code to work this way is that in the past
it was responsible for converting decimal strings both for the
current engine but also for an older version of the engine which
handled decimals of a different type. Both issues have been fixed.
* Therefore, `Decimal`s with scale exceeding the specified scale
made it into the engine, and contracts could be created containing
this invalid value.
* Once on the ledger, these bad numbers can be used to produce extremely
surprising results, due to how `Decimal` operations are
implemented. Generally speaking, all operations on `Decimal`
first compute the result and then run the output through
`Decimal.checkWithinBoundsAndRound`. The reason for this behavior
is that we specify multiplication and division as rounding their
output. Consider the case where the bad value 0.00000000005 made
it to the engine, and is then added to 100. The full-precision
result will be 100.00000000005, which after rounding becomes 100.
Therefore, on a ledger where such invalid values exist, it is not
the case that `y > 0 ==> x + y != x`, and so on.
Thanks a bunch to @briandbecker for the excellent bug report.
* fix failing test using to much precision
* Add buildifier targets.
The tool allows to check and format BUILD files in the repo.
To check if files are well formatted, run:
bazel run //:buildifier
To fix badly-formatted files run:
bazel run //:buildifier-fix
* Cleanup dade-copyright-headers formatting.
* Fix dade-copyright-headers on files with just the copyright.
* Run buildifier automatically on CI via 'fmt.sh'.
* Reformat all BUILD files with buildifier.
Excludes autogenerated Bazel files.
* TransactionServiceIT passes
* fixup
* using record time taken from TimeProvider
* taming TransactionBackPressureIT to avoid overloading of submission service
* implement AutoClosable on all Ledger related components having stateful resources
* reenabling InMemory fixture
* disabling contract key integration test for SandboxSQL fixture
* removing TODO
* bumping up timeout on TransactionServiceIT due to slow Azure pipeline
* 1 minute timeout for SqlLedgerSpec
* making jdbcurl CLI argument hidden
* updating release notes
* Add location information to DAML-LF produced by damlc
This is required to get error locations in the scenario view. Rigth now,
the location information for `create`/`exercise` still points to the
template/choice. I'll fix that in a separate PR.
* Fix test expectations
* Fix more tests
* Add participant-state and participant-state-index APIs and reference implementations
This adds the (still WIP) interfaces and the in-memory reference implementations
of participant-state and participant-state-index.
See issue #137.
* Pass ledgerId through to getLedgerEnd method
This is needed in the daml-on-x implementation of the
transaction service.
* Add api-server-damlonx and the reference server
This is the initial version of the ledger-api server built on top of
the participant state APIs.
While functionally complete (modulo test services), it has only yet
been tested with the semantic tester. The server and the participant
state APIs are still under active development.
See issue #137.
* Apply scalafmt and copyright headers to new damlonx code
* ledger/damlonx: Fix build errors after merge from master
* ledger/participant-state: separate out the reference impl
* ledger/participant-state-index: separate out reference impl
* ledger/damlonx: Refactoring
- Refactor participant-state into multiple modules
- Introduce structured offset and update id types (vector of ints)
- Properly parse ledger feature flags
* ledger: Disable failing semantic test. Decrease grouping duration.
* scalafmt, fix after merge
* Add missing copyright headers
* ledger/damlonx: Add Simon's comments
* ledger/damlonx: Remove UpdateId, use Offset in both state and state-index.
* ledger/damlonx: scalafmt
* damlonxserver: review participant-state interface
Includes adding issue links for postponed fixes.
* daml-on-x-server: review reference implementation
Fixme's added to https://github.com/digital-asset/daml/pull/348 and
extra issues creaed in https://github.com/digital-asset/daml/milestone/4
* ledger/damlonx: Use SimpleString
* ledgre/damlonx: Backtrack on SimpleString change
See issue #398
* daml-on-x-server: drop unused 'index.impl.reference.package.scala'
* daml-on-x-server: add note on potential transient contracts bug
* ledger/participant-state*: post-merge fixes
* Do not divulge contracts to observers in nonconsuming exercises
Disables support for non-default ledger feature flags, as they
are meaningless since ledger server logic does not respect the flags.
Instead of large refactoring to add support for the old flag settings,
it is best to disallow the deprecated flags, and later on phase out the
flags completely.
Re-enables test_divulgence_of_token in sandbox semantic tests.
Fixes#157.
* purge LedgerFlags entirely...
...since we only support one version of them anyway, and clearing them
* updated release notes
* SandboxSQL fixture + fixing various issues
WIP - playing around with Resources to support Sandbox Fixture with Postgres
Sandbox with Postgres fixture works - LedgerIdentityServiceGivenIT PASSES
adding guards for empty collections in PostgresLedgerDao
CommandCompletionServiceITi passed for both fixtures
fixing EventID formatting problem
another formatting fix
changed wrong autoCommit behaviour
CommandStaticTimeIT passed on both ledgers
rolling back transaction on duplicates
dealing with duplicate commands in postgres
fixing mismapped record_time
adding extra logging for fatal errors
disabling sql support as it's not finished yet
cleanup
removed unnecessary test classes
cleanup
formatting
* ADT was missing Product with Serializable
* fixed the compilation error
* Fetch status.proto from remote, simplify JS gRPC codegen
Fetch the `status.proto` file (part of the standard gRPC distribution)
from a distribution channel. _Moreover_, use the recently introduced
`proto_gen` rule to simplify how the gRPC code for the Node.js bindings
are generated (and remove the need to have `google/rpc/status.proto`
locally in the repository.
* Add plugin_runfiles option to proto_gen
This allows use to add additional files to the bazel sandbox so that
plugins can refer to them. This will subsequently be used by the
protoc-gen-doc plugin.
Also, pass the plugin options via --name_opt parameter.
* Add missing status.proto dependency /language-support/java and /ledger
* Build proto docs using the proto_gen rule
To make this work, I had to turn on the bazel build flag
`--protocopt=--include_source_info` because we cannot turn enable this
flag only for specific build rules.
* Make /ledger-api/grpc-definitions:docs public again
* Revert to the old style of passing plugin arguments to --name_out=options:path
* Suppress output of unzipping
* Fix link for google.rpc.Status in proto-docs
* Change contract key maintainers to have template argument in scope
More precisely, change the contract key type class we desugar the surface
syntax to such that the maintainers function has the template argument
in scope instead of the key. The compiler rewrites all references to the
template argument into references to the key and fails whenever that is
not possible.
* Add TemplateKey class to desugaring module
* Make template key check slightly more lenient
* Fix TODO descriptions
* draft of active contracts streaming - still WIP
dumping todos for Robert
Sandbox: fix getActiveContractSnapshot
Sandbox: Add comment on thread usage
Sandbox: Add jdbcuser and ledgerid CLI arguments
Sandbox: enable SQL ledger
Sandbox: fix ledgerEnd
Sandbox: Add postgres snapshot test
Sandbox: Refactor SQL ACS
Update ACS in the same transaction as updating ledger_entries
wip CommandTransactionIT
small refactoring of Postgres Ledger creation
removing jdbc user from CLI args as it should rather be part of the jdbc url
using a thread factory for the connection accessor thread
removing timestamps from contracts table
using Akka test helpers
handling failures as rejections
removing unnecessary fields from ActiveContractsCheck
minor cleanup
small optimisation in storing Anorm queries as constants
making ContractDataParser thinner
formatting
cleanup
PostgresLedger does not need initial contracts anymore
add docs to getStreamingConnection
configured streaming connection pool to grow and shrink dynamically
crashing on set jdbcUrl CLI arg
rolling back in case of an exception
adding some docs
adding some comments
minor renaming
Add runSQL comment
* wip ActiveContracts
* fixup
* creating class ActiveContractsManager to be explicit on responsibility
* cleanup
- Remove obsolete code introduced in this PR
* scalafmt
* changed wrong autoCommit behaviour
* formatting
* amended comment on denormalisation
* disabling failing assert for now
changing ports to use 6865 everywhere
no need for default ports
addressing missed outdated ports
changed more ports from 7600 to 6865
dealt with more 8080s
