fixes#3150
This PR introduces a patch to GHC to fix the performance of the
pattern match checker in the presence of multiple packages which
is currently significantly (orders of magnitude) slower than having
everything in a single package. I also added a test case that hits
this. Here’s what you need to hit this issue:
1. A typeclass with a functional dependency. `HasField` is the obvious
candidate for this.
2. A lot of instances of this typeclass in a separate package (this is
the only part where the separate package matters).
3. A reasonably large ADT with a bunch of strict fields.
4. A pattern match in the context of some constraints of the
typeclass. The constraints can be completely unused.
In that case, you will get a significant slowdown in the number of
instances, number of constructors and number of constraints (didn’t
verify if it’s linear but it is significant which is all that
matters).
Here’s why this happens:
1. The pattern match checker checks for strict fields if the type is
inhabited.
2. This calls `pmTopNormaliseType_maybe` to normalize a type (the details don’t
matter) which in turn calls into the typechecker. This function is
called very often (presumably linear in the number of constructors
but didn’t verify.)
3. The typechecker has some logic in `improveFromInstEnv` for
generating additional equations by unifying functional
dependencies `a -> b` with constraints in scope
and thereby deducing information about `b`.
4. In the pattern match checker the list of instances of the home
package is empty since the pattern match checker (apparently)
doesn’t actually care about those extra equations. However, the
list of instances in the EPS is not empty. This is the issue here:
By moving it to an external package we suddenly end up with
thousands of instances that we try to unify with the functional
dependencies every time we normalize which happens very often.
Proposed fix:
The solution is rather simple: Since the pattern match checker
apparently does not care about the instances of the home package, it
almost certainly doesn’t care about instances in general so we just
empty the instances of external packages explicitly.
Is the fix correct?
1. I verified that the GHC test suite passes with this patch which
gives me a reasonable level of confidence.
2. I verified that our own test suite passes.
3. The most dodgy part is actually emptying the instance since the
whole EPS stuff is a mutable mess. What could in theory happen is that
the PM ends up loading an interface file that mutates this
again. However, afaiu it is impossible for the PM to need an
interface that the typechecker didnt already need. I did do a bunch
of debugging and this is exactly what I observed in my experiments.
Alternative ideas and upstreaming:
The other option would be to not try and mess with the EPS but somehow
have a conditional flag somewhere in the typechecker env to disable
this logic in the pattern match checker. However, that sounds
significantly more complex so I don’t think it’s worth the effort.
GHC 8.10 has a new pattern match checker that has different
performance characteristics and seems to do much better here so there
is little reason to try and upstream this. I strongly want to avoid
upgrading DAML to 8.10 at this point (too much risk, let’s wait until
things calm down)
changelog_begin
- [DAML Compiler] Fix an issue where compilation slowed down
significantly when code was split up into several packages. See
https://github.com/digital-asset/daml/issues/3150
changelog_end
automated ghc-lib build
This PR aims at automating the build of ghc-lib. The current process
still has a few manual steps; it needs to be updated because Bintray is
going away, so this seemed like a good opportunity to fully automate it.
This works like the "patch bazel on Windows" jobs: the filename will
contain a hash of the `ci/da-ghc-lib` folder, and the job will run only
if the corresponding filename does not yet exist on the GCS bucket. PRs
aiming at changing the ghc-lib version will need to run twice: once to
create the artifacts, and once to change the `stack-snapshot.yaml` file
to match.
CHANGELOG_BEGIN
CHANGELOG_END
First version of static verification tool.
The current state of the tool:
- Reads DAR files.
- Partially evaluates the code.
- Generates constraints for the field and choice to be verified.
- Passes the constraints to an SMT solver.
- Some basic tests.
This should hopefully fix the issues we have been seeing on CI. While
I’m not super keen on including non-upstreamable patches it seems
better than having CI be flaky.
changelog_begin
changelog_end
This PR bumps ghcide, haskell-lsp and haskell-lsp-types. There aren’t
really any important changes in ghcide itself but the haskell-lsp
update includes my fix for crashing completions.
One change in ghcide itself is that NormalizedFilePath got moved to
haskell-lsp. ghcide needs special treatment for empty file paths so we
use `toNormalizedFilePath'` from ghcide instead of
`toNormalizedFilePath` from `haskell-lsp`. I’ve added an hlint rule to
enforce this.
changelog_begin
changelog_end
This should hopefully fix the failure to delete the temporary zip file
on Windows. See https://github.com/mrkkrp/zip/pull/61
changelog_begin
changelog_end
* Upgrade to haskell-lsp 0.20
This includes another ghcide bump but the only change there is the
haskell-lsp ugprade. haskell-lsp 0.20 includes some performance
improvements.
changelog_begin
changelog_end
* Bump parser-combinators
changelog_begin
- [DAML Compiler] Choices marked explicitly as `preconsuming` are now equivalent to a non-consuming choice that calles `archive self` at the beginning.
changelog_end
Co-authored-by: Shayne Fletcher <shayne.fletcher@digitalasset.com>
* Patterns and test for new desugaring
* working on new template conversion
* Update ghc-lib
changelog_begin
changelog_end
* shut up hlint
* Update desugar stdlib
* update test
* remove unuseful templatebinds
* Add implicit qualified GHC.Types import
* Add missing primitives
* Remove chaff
* update comments
* Remove patterns that dont seem useful anymore
* Capture key data in template binds
* Dont make TypeRep/ToAny/FromAny classes conditional
* Remove some unnecessary TODOs
* Generate the template definition
* Remove new template desugaring test
* Fix jq query
* Rename makeDesugarDFunProjection to useSingleMethodDict
* Let TTypeRep and TAny be TUnit in primitives.
* Fix damlc visual wrt the new desugaring
* Fix visualization tests in shake test suite
* Fix damldocs
* Drop envTemplateKeyData
* Use the new ghc-lib release
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Update hlint
* Redefine base package to make more sense for daml
Also, elminate lens and attoparsesec packages and groups.
* Don't change the test; ignore the new lints
* More conversion haskell rules to daml
* Introduce a simpler template desugaring without support for generic templates
This adapts the LF conversion to the new template desugaring
introduced in our GHC fork. The guiding principle is that we use the
typeclasses directly to avoid generating, typechecking and converting
redundant code caused by indirections. I updated the template
desugaring documentation so that is probably a good starting point for
reviewing this.
* Address review comments
* Fix daml doc tests
* Fix data dependency tests
* Switch to new ghc-lib release
