* Refactor extraction of events from transaction
Closes#1909
CHANGELOG_BEGIN
CHANGELOG_END
* Remove unnecessary filtering
* Fix disclosure rule for flat transaction
* Refactor and split collection and filtering
* Replace transaction filtration with blinding info
* Move transient contract remover in transaction conversion
* Remove dangling file
* Simplify transient contract filtering
* Further refinements
* Simplify transaction tree event extraction
* Move newRoots up the file for consistency and readability
* Remove collect from GenTransaction, replace with custom iterator
* Address https://github.com/digital-asset/daml/pull/4781#discussion_r388167562
* Switch to a strict collect method
* Replaced direct access to map with contains
The change to `EventFilter` and to the query in `JdbcLedgerDao` are
"duplicate work", but we need the change in EventFilter for the
InMemoryLedger, and the change in JdbcLedgerDao so that we avoid
fetching a contract that anyway would be discarded later.
CHANGELOG_BEGIN
[Sandbox]: Witnessed contracts for which a party is not a stakeholder
are no longer returned in the active contract stream.
CHANGELOG_END
Fixes#3254.
* Freeze DAML-LF 1.8 proto
This is a copy of the 1.dev proto with the following changes that
overall remove all references to 1.dev:
* Change the versions and the reference to the spec.
* Remove `dev` from version history at the top.
* Remove `GENMAP` and corresponding primitives.
* Remove general equality `EQUAL` and the comments that the
type-specific equality primitives are only available in < 1.dev
* Remove experimental text primitives.
changelog_begin
changelog_end
* windows is very bad
* bump windows hashes
* Freeze DAML-LF 1.8
Two minor points that I did not mention in the previous PR:
We also include the renaming of structural records to `struct` and the
renaming of `Map` to `TextMap`.
There are some minor changes around the LF encoder tests which need to
be able to emit package metadata properly so I’ve added it to the
parser. Sorry for not splitting that out.
Following the process used for the DAML-LF 1.7 release, this does not
yet include the frozen proto file.
changelog_begin
- [DAML-LF] Release DAML-LF 1.8:
* Rename structural records to ``Struct``. Note that
structural records are not exposed in DAML.
* Rename ``Map`` to ``TextMap``.
* Add type synonyms. Note that type synonyms are not serializable.
* Add package metadata, i.e., package names and versions.
Note that the default output of ``damlc`` is stil DAML-LF 1.7. You
can produce DAML-LF 1.8 by passing ``--target=1.8``.
changelog_end
* Update encoder
* Update java codegen tests
* Update comment in scala codegen
* Handle TSynApp in interface reader
* Bump lf_stable_version to 1.7
* Fix kvutils tests
* Make kvutils work with the new contract id scheme
CHANGELOG_BEGIN
- [KVUtils] uses random contract id. Contract ids are made of 65 hexa decimal characters.
CHANGELOG_END
Co-authored-by: Jussi Mäki <jussi.maki@digitalasset.com>
We will only include type synonyms and package metadata (which are
used for Cross-SDK model upgrades). Everything else stays in 1.dev for
now.
changelog_begin
changelog_end
* Ensure DarReader prevents zip bombs
CHANGELOG_BEGIN
[DAML-LF] The DarReader has a 1GB hard cap on ZIP archive entry size to prevent zip bombs
CHANGELOG_END
* Properly test UniversalArchiveReader, make it prevent bombs, break away memory heavy tests
* Exclude the zip bomb detection test from running on Mac CI nodes
As mentioned in the title, this is still very experimental and needs
more work before we want to advertise it. However, the code is in a
somewhat reasonable shape, there are tests and I think even in the
current state it is already useful. Also this PR is already getting
very large so I don’t want to hold off much longer before merging this.
It is included in the SDK but hidden from `damlc --help` and `daml
--help` until the most pressing issues are addressed (primarily around
making sure that it doesn’t just shut down if you have a type error
and better error messages in general).
changelog_begin
changelog_end
* Add package metadata to DAML-LF proto and the Haskell AST
This adds package metadata (currently only the package name and
version) to DAML-LF and the corresponding Haskell ASTs. This is useful
for debugging and “codegens” (typescript, damlc dependencies, …)
This PR does not yet add it to the Scala side or change the compiler
to actually produce this metadata.
Part of #4412
changelog_begin
changelog_end
* Address review comments
* [engine] optimize conversion of hash to string
* Revert "Bump daml-lf engine test timeout (#4591)"
This reverts commit 9495d78240.
* use guava to convert bytes to string
* changelog
CHANGELOG_BEGIN
CHANGELOG_END
* Engine:
Switch BaseEncoding to lowercase
* formatting
* Use KeyHasher to serialize contract keys in kvutils
- Use Value instead of VersionedValue in GlobalKey as the versioning does not make sense here
and may be misleading as the a value with a different version but same meaning would still
be the same key.
- Relocate the KeyHasher to ledger-api-common so kvutils can use it (otherwise cyclic dependencies)
- Replace storing of the contract key as a VersionedValue with the hash produced by KeyHasher.
This is backwards incompatible. A compatible option would require us to query the key with both
the old way and the new way which is unattenable. We're making a calculated breaking change.
CHANGELOG_BEGIN
- [DAML Ledger Integration Kit] Serialize contract keys using a hash instead of the value in kvutils.
This is a backwards incompatible change to kvutils.
CHANGELOG_END
* Use proper hasher for contract keys and not KeyHasher
- Use Hash.scala, not KeyHasher.scala.
- Add hash to GlobalKey as we want the hash to be computed from the inside.
The use of KeyHasher will be later deprecated and replaced by this.
* Use "sealed abstract case class" trick instead of private ctor
and rebase fix
* Revert change to unsupported value version decode error
* Reformat code
* Add kvutils changelog entry and bump the version
Manually matching on every package isn’t actually testing for anything
meaningful since the number of packages here really isn’t something
that we should test since we change it quite regularly in the
compiler.
changelog_begin
changelog_end
* Fix handling of tx versions in protoNodeInfo
The method incorrectly enforced a lower limit for a supported
transaction version. If a transaction does not contain a fetch
node then a lower transaction version (than minFetchActors) can
be chosen and this method would incorrectly fail.
CHANGELOG_BEGIN
CHANGELOG_END
* Add test for checking informeesOfNode with generated txs
* Translate unsupported kinds to a special Erased type
This should simplify `data-dependencies` and avoid issues like #4470
since we can match on the type instead of having to guess which types
can and which cannot be translated back to DAML.
changelog_begin
changelog_end
* Prevent building of en/de-coders of Cid, Val, and Nid outside Value/Transaction Coder
* DAML-LF: remove constrains on Cid for LF encoder/decoder
* Address Samir coments
This adds the NodeInfo trait to compute the informees of a node and the
required authorizers. This has been separated into its own trait as this
computation is also required (externally) for serialized transaction nodes
in a context where the full deserialization of the node would be prohibitively
expensive.
While doing this change and cleaning up transaction authorization it was
discovered that top-level fetches (which can only be created from scenarios)
did not include the submitter as a witnessing party.
CHANGELOG_BEGIN
[DAML SDK] Fix computation of witnesses of top-level fetch nodes in scenario results ("known to").
CHANGELOG_END
* kvutils: Use durations in participant state tests.
Makes it way easier to read than `addMicros(…)` calls.
* daml-lf/data: Fix warnings in Time.scala.
* daml-lf/data: Add `Timestamp#add`.
* kvutils: An MRT of (start time + 1 second) is too short for CI.
Increasing the MRT to (now + 10 seconds).
CHANGELOG_BEGIN
CHANGELOG_END
clean up String Identifier in daml-lf
- Separate LedgerString from ContractIdString
- Drop TransactionIdString from daml-lf
- Create a new ContractIdString (aka. ContractIdStringV1) for new contractId comparable with relative contractIds
CHANGELOG_BEGIN
CHANGELOG_END
* Added a partition function to prelude.
* Fixed partition function documentation.
* CHANGELOG_BEGIN
- Added `partition` function to prelude.
CHANGELOG_END
* Fixed partition documentation.
Unfortunately, downgrading `scalafmt` resulted in files that would be
reformatted to invalid code, because lines would be merged but trailing
commas would be left in.
I've manually run it everywhere and fixed these issues so people don't
have to fix them as they go.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Generic order relation, first pass
* First pass at generic order specification
changelog_begin
changelog_end
* Order functions raise an error if the arguments are incomparable
* refer to internal representation for cid ordering
* Add gen lt rules for maps.
* Support DAML-LF type synonyms in scala.
CHANGELOG_BEGIN
CHANGELOG_END
* dont create synonymns in GenerateSimpleDalf
* extend DAML-LF parser to support type synonyms
* test: expand type synonyms correctly
* engine: the natural order in Transaction is pre-order traversal
- Do not use anymore SortedMap for nodes in transaction
- Drop unused transaction traversal
CHANGELOG_BEGIN
CHANGELOG_END
* daml-lg: improve generator of transactions
Prohibit contract IDs in contract keys and add key maintainers to exercises
CHANGELOG_BEGIN
- [DAML-LF] Prohibit contract IDs in contract keys completely. Previously, creating keys containing absolute (but not relative) contract IDs was allowed, but `lookupByKey` on such a key would crash.
CHANGELOG_END
Co-authored-by: Remy <remy.haemmerle@daml.com>
Co-authored-by: Stephen Compall <scompall@nocandysw.com>
Fixes#28.
CHANGELOG_BEGIN
[Sandbox] DAML trace logs (trace, traceRaw, traceId) are now logged via the regular logging system (slf4j+logback) at interpretation time via the logger ``daml.tracelog`` at DEBUG level.
CHANGELOG_END
* Move most of the remaining serializable types to stable LF packages
The only serializable types left in DAML stdlib after this PR are the
following:
- DA.Upgrade:MetaEquiv
- DA.Random:Minstd
- DA.Next.Set:Set
- DA.Next.Map:Map
- DA.Generics:MetaSel0
- DA.Generics:MetaData0
- DA.Generics:DecidedStrictness
- DA.Generics:SourceStrictness
- DA.Generics:SourceUnpackedness
- DA.Generics:Associativity
- DA.Generics:Infix0
- DA.Generics:Fixity
- DA.Generics:K1
- DA.Generics:Par1
- DA.Generics:U1
- DA.Internal.Prelude:Optional
Ignoring the Generics stuff which isn’t very urgent imho and the
Upgrade stuff which is probably going to change significantly anyway,
this leaves us with the weird Random module, the wrappers around
TextMap which will go away anyway and DA.Internal.Prelude:Optional
which shouldn’t exist in the first place (I’ll address that in a
separate PR).
CHANGELOG_BEGIN
- [DAML Compiler] Move more types from daml-stdlib to standalone LF
packages. The module names for the types have also changed
slightly. This only matters over the Ledger API when you specify the
module name explicitly. In DAML you should continue to use the
existing module names.
- The types from ``DA.Semigroup` are now in a separate package under
``DA.Semigroup.Types``
- The types from ``DA.Monoid` are now in a separate package under
``DA.Monoid.Types``
- The types from ``DA.Time` are now in a separate package under
``DA.Time.Types``
- The types from ``DA.Validation` are now in a separate package
under ``DA.Validation.Types``
- The types from ``DA.Logic` are now in a separate package under
``DA.Logic.Types``
- The types from `DA.Date` are now in a separate package under
`DA.Date.Types`.
- The `Down` type from `DA.Internal.Prelude` is now in a separate
package under `DA.Internal.Down`.
CHANGELOG_end
* Fix serializability of RelTime
* fix daml-docs
* Fix tests
* Change variant json encoding,
adding integration test
* Add DamlLfTypeLookup dependencies
* Add MetadataReader
* Add test WIP
* Add serialize test cases
* Add serialize test cases, WIP
* Test for variant encoding decoding
* Solving merge conflicts
* Updating roundtrip test
* Minor cleanup
* Addressing code review comments
Add JsonVariant custom matcher
* Update specification
* Update link
* Add test case, WIP
* Add proper template key resolution
* Got rid of choice record ID resolution, resolving choice type and key type
* Fixing logging
* Add Contract Key decoding tests
* cleanup
* cleanup
* Update JSON variant encoding tests
* Add more contract key JSON decoding tests
* Fix variant JSON encoding
* Change value predicate to support new variant encoding
* Change value predicate to support new variant encoding
* Add lookup by contract key test case
where contract key contains variant and record
Add `requiredResource` to bazel utils
CHANGELOG_BEGIN
- [JSON API - Experimental] Change variant JSON encoding. The new format is ``{ tag: data-constructor, value: argument }``.
For example, if we have: ``data Foo = Bar Int | Baz``, these are all valid JSON encodings for values of type Foo:
- ``{"tag": "Bar", "value": 42}``
- ``{"tag": "Baz", "value": {}}``
See #3622
- [JSON API - Experimental] Fix ``/contracts/lookup` find by contract key.
- [JSON API - Experimental] Fix ``/command/exercise`` to support any LF type as a choice argument.
See #3390
CHANGELOG_END
* minor cleanup
* Fix copy/paste
* Renaming
* Got rid of DAML LF identifier resolution
resolving DAML LF Type based on command type
* Address code review comments, thanks @S11001001
* Address code review comments, thanks @S11001001
Do not include any error handling here; this partial function should
only match the successful case, JsonVariant.
* Address code review comments, thanks @S11001001
comment
* Address code review comments, thanks @S11001001
using `JsonVariant` for variant encoding/decoding
* Address code review comments, thanks @S11001001
replace `find` and `map` chain with collectFirst
* Update docs/source/json-api/lf-value-specification.rst
Co-Authored-By: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Stephen Compall <scompall@nocandysw.com>
* runtime-filtering-only-style enum for ValueAddend
- the alternative is a coproduct of singleton types, which would be more
like the other ValueAddend combinators, but done differently here as
an experiment
* test an enum case for lf-value-json
* drop Nones from records when JSON-encoding
* elide some needless conversions in ApiCodecCompressed
* record with visible Inj
- seems to be running up against scala/scala#5075
* use new 3rdparty lib reference
* construct record types field-by-field
- works around scala/scala#5075
* switch to record style for records' Inj
- suggested by @leo-da
* include some samples to experiment with record
* record parsing
* RecordVa doesn't benefit from kind-projector
* scalafmt
* make RecordVa details private; only intended API is public
* Arbitrary and Shrink record support
* rename RecordVa#Inj to HRec, to accommodate variants in same cons structure
* use record testing in lf-value-json
* add complex record type
* add complex record value and test it
* more documentation for RecordVa
* Table-ify prior record tests
* start adding variant support
* scalacheck support for variants
* more scalacheck support for variants
* variant examples
* example of using align for records
* test variant roundtrip
* test variant in record
* add sample tests for the JSON format of records and variants
* rename record-specific functions in RecordVa
* rename RecordVa to RecVarSpec
* Move Any wrappers and Archive to stable packages
There are no actual API or functionality changes in this PR but the
logic for locating the stable packages has slightly changed since the
Any wrappers package only makes sense for LF 1.7. To address this, we
simply filter out stable packages for newer LF versions since it
doesn’t make sense to depend on those anyway.
CHANGELOG_BEGIN
- [DAML Compiler] Move ``Archive`` type to a separate DALF.
CHANGELOG_END
* More comments
* Fix java codegen tests
* fix more tests
* Force cache reset on Windows
* Revert "Force cache reset on Windows"
This reverts commit 9f2b7d70b2.
* daml-lf: second version of type synonym spec
* cosmetic changes
* more fixes
* make special application for type synonymes
* Update daml-lf/spec/daml-lf-1.rst
Co-Authored-By: associahedron <231829+associahedron@users.noreply.github.com>
* make type synonym different from type constructor
* typos
* scala codegen: add GenMap support
* scala-codegen: use InsertOrdMap as underlying of Generic Map binding
* Address Stephen's comments
* make TextMap a strict subtype of immutable.Map to avoid overlap with GenMap
Otherwise, attempted GenMap encodings like
`Value.encode(InsertOrdMap("foo" -> "bar"))` (and any contract
containing such a structure) are either ambiguous or, worse, yield the
wrong result.
* ensure better return types for InsertOrdMap and similar wrappers' operations
* if is an expression
* adapt various primitive encodings to newtype TextMap
* easy cleanups of the TextMapApi
* proper alias for deprecated Map
* update deprecation releases
* Revert "proper alias for deprecated Map"
This reverts commit e85aa85b960c4bf5c4f9624896183ec6e2182bba.
* remove useless invisible deprecated notice
* add generic map tests for scala codegen
* please restart CI
* Move all datatypes out of daml-prim
This moves the remaining two modules DA.Types and GHC.Tuple to
separate LF packages with stable identifiers.
The only data types remaining are the ones for typeclasses which will
disappear once we move this to type synonyms.
CHANGELOG_BEGIN
- [DAML Compiler] The modules DA.Types and GHC.Tuple from daml-prim
have been moved to separate packages.
CHANGELOG_END
* Fix codegen tests
* Fix DarReader test
* Fix kvutils tests
* Fix jdbcdao tests
* Fix hs ledger bindings tests
* Sandbox: Remove streaming connections
The separate database connection pool for streaming connections
was only used for the active contracts stream. However, a single
db connection was being occupied until the last active contract was
streamed over the Ledger API to the client. This effectively means
that only ever 2 concurrent active contract streams could exist.
No need to say that this is bad design.
The following changes happened:
- remove the db connection pool for streaming connections
- replace the streaming mechanism for active contracts with
the already existing pagination mechanism in JdbcLedgerDao
- change the pagination mechanism to actually use database level
limit and offset instead of doing the pagination "client side"
- configure the HikariDataSource with the metric registry
CHANGELOG_BEGIN
- [Sandbox] Improve loading of active contracts for the Sandbox SQL backend.
CHANGELOG_END
* Extract PaginatingAsyncStream from JdbcLedgerDao for testing
* Reset metrics registry before each test
This is a first step towards making sure that the package ids for
types defined in daml-prim and daml-stdlib don’t change. This PR
mostly adds all the necessary infrastructure for that and moves
GHC.Types and GHC.Prim to make sure it works.
Until data-dependencies are really solid and we have verified that we
no longer have performance issues with an increasing number of Haskell
packages, we still include the source files in daml-prim and then just
rewrite the references.
We will also need to add tests that these packages really have stable
ids but I’ll leave that for separate PRs since this doesn’t make that
much sense anyway until all of the types have moved to stable
packages.
CHANGELOG_BEGIN
- [DAML Compiler] The modules GHC.Prim and GHC.Types from daml-prim
have been moved to separate packages.
CHANGELOG_END
* Track used packages during whole of engine submit
- Introduce MutableCompiledPackages interface
- Add TrackingCompiledPackages that tracks fetched packages
- Make used packages in transaction optional to distinguish between
missing dependencies and empty set of dependencies.
* Reimplement package dependency tracking
- Compute direct dependencies of a package during decoding
- Compute transitive dependencies of a package when adding a package
to engine.
- Annotate the resulting transaction with package dependencies
in Engine.submit.
* Create Ast.Package with proper direct deps in scenario service
While we don't have use for direct dependencies of a package in
scenario service (only Engine.submit needs it), it's better to be
accurate.
This of course overapproximates the direct dependencies.
* Compile a each new package once in ConcurrentCompiledPackages
* Include type synonyms in LF abstract syntax
* Use T instead of S for synonyms
Following a suggestion from @remyhaemmerle-da, we avoid having to
adapt all rules to synonyms by simply reusing the definition of type
constructors.
* Start moving all the jars into a single jar.
* Fold navigator into daml-sdk jar
* include sandbox
* Remove unnecessary compileDeps.
CHANGELOG_BEGIN
- [DAML SDK] Reduced the size of the DAML SDK by about
60% uncompressed, 70% compressed, by deduplicating Scala
dependencies.
CHANGELOG_END
* update copyright header
* buildifier fix
CHANGELOG_BEGIN
[DAM-LF Interface Reader]: **Rename** ``PrimTypeMap`` to ``PrimTypeTextMap`` and ``PrimType.Map`` to ``PrimType.TextMap``
CHANGELOG_END
* Add specification for type synonyms in DAML-LF
* Fix identifiers for type synonyms
* Use S for type synonyms
* Address review comments
* Split up TyTypeSynonym and DefTypeSynonym
