When buidling simple project that has our favourite large project as a
dependency, this decreased
- total allocations from 63GB to 57GB
- run time from 34.0s to 31.5s
* Add support for on-disk incremental builds in damlc build
* Normalise file paths of internal modules because Windows
* stop stealing my $s hlint
* Apparently jars are also called exe
* Address review comments
* Bump to proper ghcide revision
* Generalize AnyTemplate type to Any in DAML-LF
See #3131 for the motivation for this. The tl;dr is that we need
something like AnyTemplate for choice types as well.
Since the protobuf was already more general in anticipation of such a
change, this change only changes the internal AST on the Haskell and
Scala side.
Since AnyTemplate change has never made it out of 1.dev, I updated the
changelog in the LF spec instead of adding a new entry.
* Update daml-lf/spec/daml-lf-1.rst
Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* windows debugging
* more windows debugging
* clean expunge
* don’t cat the config file
* remove comment on type equality
* windows …
* gnah
* foobar
* foobar
* does anything ever work?
* reenable caching
* Do not build daml-lf-ast separately
* Intern all strings and dotted names in the DAML-LF protobuf encoding
On one of our huge DAML code bases this has brought down the size of the
DALF by 60% and cut the memory usage during decoding by 50%.
The `@QUERY-LF` tests in the damlc integration tests check that the
interning actually works.
* Add scala decoder support for interned strings
* Memoize versionisOlderThan. Speeds up decoding by 6x.
* Properly fix the performance problem with version ordering
LanguageVersion.ordering was a `def` and not `val` causing us to
recreate the ordering tables every time.
* Clean up //daml-lf/archive:decode-test
* Use int32 for interning indices
* Adapt QUERY-LF tests
* Strip a few useless field_ prefixes
* Add test for uniqueness in interning tables
* Drop field_ prefix from jq queries
* Fix Scala encoder
* Fix bond-trading tests
* Rename decode-test to decode-tester
* Address @cocreature's comments
* daml-lf: proper versioning for reading internedStrings
* daml-lf repl: revert change
* Remove polymorphic literals from library
* Add BECastNumeric and BEShiftNumeric in DAML-LF AST.
* Update rest
* Copyright header
* Remove polymorphic literals from LF conversion.
* Update tests
* Add test for forbidden polymorphic literals
* Add newline in DA.Numeric
* use MulNumeric instead of CastNumeric in DA.Internal.Prelude
* Refactor DAML-LF encoder/decoder in prep for string interning
Refactor the encoder/decoder such that all the functions concerned with
interning live next to each other and code will break once the types of
the protobuf messages change.
This PR is a pure refactoring and does not change any functionality.
* Fix typos
Right now, some functions are polymorphic in the monad we use. This is
absolutely unnecessary.
This PR fixes one monad and uses it all over the place. This is a pure
refactoring which doesn't change any functionality.
* Merge package reference detection into package id encoding
Currently, the encoding of a DAML-LF package works in two phases:
1. We compute the set of all package ids refernced by the package to encode.
2. We use this set of package ids to form a table for interning that is then
consulted during the encoding of package ids.
This PR changes it such that the interning table is computed while the
package is being encoded. New package ids are added to the table whenever
they are encountered for the first time.
This is in preparation for the general inlining of all strings, which will
use the same approach to building the interning table.
This PR is a pure refactoring and does not change any functionality.
* Addess @cocreature's comments
* Add unordered-containers as a dependency
This is in preparation for interning all strings during encoding, which
will replace the current reader monad by a state monad.
This PR does not change or add any functionality. It is purely a
refactoring.
* Add Any type and to_any/from_any primitives to protobuf
Following a suggestion by Rémy, the protobuf representation is more
general and is associated with an arbitrary type instead of a
typecon. This allows us to easily extend this later to a full Any
type.
I’ve still called the type in the protobuf Any instead of Haskell’s
Dynamic since I find AnyTemplate more clear than DynamicTemplate and
having AnyTemplate and Dynamic seems confusing.
Right now, the decoder enforces that the type is a TypeCon.
* Fix some mistakes in the spec
* Update daml-lf/spec/daml-lf-1.rst
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Update daml-lf/spec/daml-lf-1.rst
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Update daml-lf/spec/daml-lf-1.rst
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Update daml-lf/spec/daml-lf-1.rst
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Add evaluation rule for to_any_template
* Update daml-lf/spec/daml-lf-1.rst
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* daml-lf: add CAST_NUMERIC and SHIFT_NUMERIC internally
* daml-lf: add CAST_NUMERIC and SHIFT_NUMERIC to archive proto
* daml-lf: update spec with CAST_NUMERIC and SHIFT_NUMERIC
* update release notes
* fix spec
* Address comments from Fran and Gerolf
* fix unrel
* Delete obsolete proto3-suite patch
My patch has been upstreamed so no need to keep it around in our repo.
* Upgrade proto3-wire
* Adapt to changes in proto3-suite
* Optimize mangleIdentifier
I am slightly embarassed by importing Data.Text.Internal and
Data.Text.Array. However, this does make a very measurable difference:
On my testcase it brought down the time encoding modules from 22s to
17s, so it’s a 5s improvement.
I did quickcheck this against the old implementation, so I’m confident
it behaves correctly. I also ran some criterion benchmarks to see the
difference and on its own it’s even more of a difference.
https://gist.github.com/cocreature/822114257227473ecff1638a88f07788
* GHC is stupid :(
* language: put hash in package directories
We put the package id of the main dalf of a package into the directory
names, where we store the files of that package in the package database.
This way we make sure that two equally named packages don't overwrite
their dependencies and files.
* Supporting producing sdist tarballs for the HS ledger bindings
The README.md has an explanation for how you can use this.
This should hopefully allow others to experiment with the bindings.
My benchmark is sadly extremely noisy which makes it hard to figure
out how much of an improvement this is but it does seem to at least be
a minor improvement.
* daml-lf: prepare archive proto for Numeric
* replace DECIMAL by NUMERIC in messages and fields
* add nat kind and nat type
* add builtins CAST_NUMERIC and SHIFT_NUMERIC
* daml-lf: remove new builtins from archive proto
* daml-lf: some more doc about numeric
* daml-lf: fix spec
* specify the new fields for interning package IDs
* percolating intern tables
* crawl packages for package IDs as a pre-phase: generic prep
* stub case for interned_id in Scala packageref reader
* HasPackageRefs instances for the rest of the ast
* make intern table and use when encoding PackageRefs in v1
* don't need where
* stub out decode for interned package IDs
* no benefit to using uint32 instead of uint64
* percolate in encode one step
* interned case for decoding PackageRefs
* naming details
* intern table decoder
* finish propagating the intern table in encoder
* encode the package ID table
* document the vital assumption of encodeInternedPackageIds
* propagate the intern table through the LF decoder
- done by stacking ReaderT on top of Decode internally,
as discussed with @hurryabit
* daml-lf-proto requires mtl
* stub out interned case in Scala LF decoder
* stub interface decoder function
* get the interned table to most places in InterfaceReader
* support for interned package IDs in Scala decoder
* use ImmArraySeq instead of Vector for Scala intern decode table
* adding that ghc extension didn't make sense
* implement interned ID decoding for InterfaceReader
* scenario service won't have interned package IDs
* test the interned ID resolution in Scala by examining the proto -> AST in detail
* proper precondition for the dev phase of interned IDs testing
* better error reporting for malformed DALFs in intern test
* just import Data.Int
- suggested by @neil-da; thanks
* pass around the lookup function instead of the vector in decoder
- suggested by @neil-da; thanks
* remove derivations for types deleted in e63b012d2d
* rename VersionAware to EncodeCtx
- suggested by @hurryabit; thanks
* rename MDecode to MonadDecode
- suggested by @hurryabit; thanks
* pass a function through the encoder instead of a set
- based on suggestions by @hurryabit and @neil-da; thanks
* daml-ghc test that interned IDs are generated
- suggested by @hurryabit; thanks
* adapt to 5b480c99ec#1844
My primary goal is to avoid confusion when starting to support DAML-LF's
enum types. I also think these types were never particularly useful,
although I introduced them myself...
I find the current names very confusing and unintuitive.
I'm aware that there are other primitives called `TO_TEXT_X`. But there the
`X` is meant to be a type index to `TO_TEXT`. This doesn't make much sense in
the context of `TO_TEXT_CODE_POINTS` though since there's no type named
`CodePoints`.
* Drop support for DAML-LF 1.3 from damlc
This means `damlc` now only supports versions allowing arbitrary expressions
as contract keys. This is very helpful for the upcoming new template
de/re-sugaring underpinning generic templates.
* Unpin a few DAML-LF target versions in tests
* Fix golden tests
* Adapt changelog to rebase
* Rename DAML-LF scenario tests to 1.dev
* Update tests to --target 1.dev
* Fix release notes
