* Preventively switch to uninterruptible mask in withMVar'
withMVar' is used to update the shakeSession var and it's crucial that the
third argument is not interrupted.
'mask' can still be interrupted for I/O actions and, while we were careful to
ensure none was used, if it ever breaks it will lead to very hard to debug
problems.
* refactor: move to RuleTypes
* Add a TestRequest to wait for arbitrary ide actions
Closes#955
* expectCurrentDiagnostics
* Add a test suite for cancellation
* Introduce --test-no-kick to fix cancellation tests reliability
* delete unsafeClearDiagnostics (unused)
* GetModSummaryWithoutTimestamps - remove StringBuffer
Since the contents of the buffer are not tracked by the fingerprint.
* Fix diagnostics bug
Given a FOI F with non null typechecking diagnostics D, imagine the following scenario:
1. An edit notification for F is received, creating a new version
2. GetModTime is executed, producing 0 diagnostics.
2.1 updateFileDiagnostics is called
2.2 setStageDiagnostics is called
2.3 LSP.updateDiagnostics is called with a new version, resetting all the
diagnostics for F
2.4 newDiags=[] in updateFileDiagnostics, which is different from D (the last
published diagnostics), which enqueues a new publishDiagnostics [] in the
Debouncer
3. An edit notification for F is received before typechecking has a chance to
run which undoes the previous edit
4. The debouncer publishes the empty set of diagnostics after waiting 0.1s
5. GetFileContents runs and since the contents of the file haven't changed since
the last time it ran, early cutoff skips everything donwstream
Since TypeCheck is skipped, the empty set of diagnostics stays published until
another edit comes.
The goal of this change is to prevent setStageDiagnostics from losing
diagnostics from other stages. To achieve this, we recover the old diagnostics
for all stages and merge them with the new stage.
* Fix hlint
* Use Map.insert for clarity
* Fix redundant imports
* Fix "code actions after edit" experiment"
* Trace rule errors
* Disable check parents in command line script
* Fix expectDiagnostics []
* Add a test
* remove uses of stale info within rules
The use of stale information should be limited to the leaves of the processing
tree, otherwise it becomes impossible to reason about the semantics of diagnostics
* Use stale info in the NeedsCompilation rule
* Use stale data in GetDocMap
* Fix tests that relied on unsupported behaviour of expectDiagnostics
* Move tracing functions to own module
* Bump opentelemetry to 0.6.0
* Write Values map size to OpenTelemetry metric
* Trace all requests and notifications
Instead of doing it in `HoverDefinition`, do it in
with{Response,Notification,...}. These wrap all handlers, so this should
cover everything. It also means that the span covers the entire
processing time for the request, where before we missed the setup
happening in the with* functions.
* Add flag for OpenTelemetry profiling
Run GC regularly with --ot-profiling
* Add flag to enable OT profiling in benchmark
* Use heapsize instead of ghc-datasize
I renamed the fork to distringuish from the original.
It is still being pulled from git using stack. This will be addressed
once I can push the fork to hackage.
* Bump opentelemetry to 0.6.1 - fixes 8.6 build
* Use heapsize from hackage
* Address HLint messages
* Record size of each key independently
* Refactor `startTelemetry` function
* Remove delay between measuring memory loops
* Each key in values map gets own OT instrument
* Measure values map length more rarely
* Rename --ot-profiling to --ot-memory-profiling
* Add docs for how to use the opentelemetry output
* Add instructions to build release version of tracy
* Clarify dependencies in opentelemetry instructions
* Fix LSP traces
* otTraced: delete unused
* Extract types out of D.IDE.Core.Shake
to avoid circular module dependencies
* Extract startTelemetry out of D.IDE.Shake and upgrade to 0.2
No more segfaults
* [nix] install opentelemetry
* [nix] install tracy
* Fix merge wibble
* Measure recursive sizes with sharing
* Sort keys for cost attribution
* Remove debug traces
* Allocate less, group keys, clean up hlints
* Add -A4G to the flags used for --ot-memory-profiling
* Modularize D.IDE.Core.Tracing
I want to reuse this code more directly in the non lsp driver
* Direct driver: report closure sizes when --ot-memory-profiling
An eventlog memory analysis doesnt' seem so relevant since this mode is not
interactive, but we could easily produce both if wanted to
* Everything is reachable from GhcSessionIO, so compute it last
I suspect the ShakeExtras record is reachable from GhcSessionIO
* bound recursion and use logger
* hlint suggestions
* Fix 8.6 build
* Format imports
* Do the memory analysis with full sharing. GhcSessionIO last
* Fail fast in the memory analysis
* error handling
* runHeapsize now takes initSize as an input argument
* Trace Shake sessions
* Reduced frequency for sampling values length
* Drop the -fexternal-interpreter flag in the Windows stack build
* Produce more benchmark artifacts
* Fix stack descriptors to use heapsize-0.2 from Hackage
* Bump to heapsize-0.3.0
* Record completions snippets (#900)
* Add field for RecordSnippets to CachcedCompletion
* Initial version of local record snippets
* Supprt record snippet completion for non local declarations.
* Better integration of local completions with current implementation
* Clean up non-local completions.
* Remove commented code.
* Switch from String to Text
* Remove ununsed definition
* Treat only Records and leave other defintions as is.
* Differentiate Records from Data constructors for external declaration
* Update test to include snippet in local record completions expected list.
* Update completionTest to also compare insertText.
* Add test for record snippet completion for imported records.
* Hlint fixes
* Hlint fixes
* Hlint suggestions.
* Update type.
* Consolidate imports
* Unpack tuple with explicit names
* Idiomatic changes
* Remove unused variable
* Better variable name
* Hlint suggestions
* Handle exhaustive pattern warning
* Add _ to snippet field name suggestions
* Remove type information passed around but not used
* Update to list comprehension style
* Eliminate intermediate function
* HLint suggestions.
* Idiomatic list comprehension
Co-authored-by: Pepe Iborra <pepeiborra@me.com>
* [nix] use gitignore.nix (#920)
* Ignore import list while producing completions (#919)
* Drop any items in explicit import list
* Test if imports not included in explicit list show up in completions
* Update README.md (#924)
* Custom cradle loading (#928)
When using ghcide as a library, it may be desirable to host the hie.yaml file
in a location other than the project root, or even avoid the file system altogether
* Favor `lookupPathToId` over `pathToId` (#926)
* Favor `lookupPathToId` over `pathToId`
* Fix `typecheckParentsAction`
* Fix `needsCompilationRule`
* Return completion snippets only when client supports it (#929)
* Use the real client capabilities on completions
* Return completion snippets only when supported by the client
Restored from https://github.com/haskell/ghcide/pull/900
* Redundant import
* Fix stack windows build
Co-authored-by: Michalis Pardalos <m.pardalos@gmail.com>
Co-authored-by: Michalis Pardalos <mpardalos@gmail.com>
Co-authored-by: Guru Devanla <gdevanla@users.noreply.github.com>
Co-authored-by: Samuel Ainsworth <skainsworth@gmail.com>
When using ghcide as a library, it may be desirable to host the hie.yaml file
in a location other than the project root, or even avoid the file system altogether
* Avoid calling kick explicitly
Leverages that rules are rerun by shakeRunDatabase.
Allows users of ghcide as a library to use their own kick
* Tweak doc comment
* Write ifaces on save
* Move isFileOfInterestRule to FileStore.hs and use real mtime for saved files
* hlint
* Add test
* fix flaky tests
* Only check for hie file in >= 8.6
* Cancellation of user actions
* Dispatch event handlers asynchronously
* add tests for asynchronous features
This adds a new Test plugin for custom requests
and a new blocking Command
* hlint
* Link the Testing plugin only when --testing
* Fix expectNoMoreDiagnostics
Needs also https://github.com/bubba/lsp-test/pull/74
* Upgrade lsp-test to a version that understands CustomClientMethod
* Add new command to GetModuleGraph for a session and propate changes to
modules
Only propagate changes to parent modules when saving
Typecheck files when they are opened, don't TC FOI
Add known files rule
Don't save ifaces for files with defered errors
Co-authored-by: Zubin Duggal <zubin@cmi.ac.in>
* Add configuration for parent typechecking
* hlint ignore
* Use targets to filter located imports (#10)
* Use targets to filter located imports
* Remove import paths from the GHC session
Otherwise GHC will prioritize source files found in the import path
* Update session-loader/Development/IDE/Session.hs
Co-authored-by: Pepe Iborra <pepeiborra@me.com>
* Add session-loader to hie.yaml (#714)
* move known files rule to RuleTypes
* Disable checkParents on open and close document (#12)
* Really disable expensive checkParents
* Add an option to check parents on close
Co-authored-by: Matthew Pickering <matthewtpickering@gmail.com>
Co-authored-by: Pepe Iborra <pepeiborra@me.com>
Co-authored-by: Luke Lau <luke_lau@icloud.com>
* Split out the session loading logic into a sublibrary
This way haskell-language-server can also reuse this logic.
Note that this sublibrary is public so it requires cabal-version: 3.0
Part of the work towards #478
* Move Development.IDE.Session into ghcide itself
Sublibraries do not seem to play well. Hide this behind the ghc-lib flag
so that the Haskell specific hie-bios stuff can be disabled
Note that we need to put the template-haskell part of this module into a
separate module because of an access exception when compiling with
Stack, GHC 8.10.1 and Windows.
* Update to hie-bios 0.6.1
* Obtain the GHC libdir at runtime using hie-bios
This replaces hardcoding the GHC libdir path with ghc-paths and instead
gets it at runtime through the hie-bios cradle. This means that the
ghcide binary should be a bit more distributable now, since it won't
rely on paths baked at compile time that are local to the machine it was
compiled on. And we also no longer need the ghcLibCheck since we are
comparing the coompiled and runtime versions with the installationCheck
(ghcVersionChecker)
Co-authored-by: Fendor <power.walross@gmail.com>
* Use global NameCache for reading HIE files
Co-authored-by: Matthew Pickering <matthewtpickering@gmail.com>
* ignore hlint
* redundant imports
* Use hie files as source of truth for name source spans.
Since we started reusing `.hi` files, this exposes a bug where definitions
aren't available since a bad source span from the `.hi` file gets put into
the NameCache. We rectify by ensuring the span in the NameCache always matches
the one from the `.hie` file.
This has surfaced because an interaction between the commit which uses `.hi`
instead of retypechecking and the change to use the shared global NameCache
to read `.hie` files.
* Add test for missing definitions
Co-authored-by: Matthew Pickering <matthewtpickering@gmail.com>
* Use stale information for hover and completions
This introduces a new function `useWithStaleFast` which returns with
stale information WITHOUT checking freshness like `use` and
`useWithStale`.
Greatly improve debug logging
All actions triggered by shakeRun now also pass an identifier which
means that the debug logging shows which actions are starting/finishing
We also distinguish between internal and external events. By default
external events are ones triggered by runAction and the debug output
is displayed to the user in command line and --lsp mode.
In order to see internal logging statements, there is a new flag called
--verbose which also prints out internal events such as file
modification flushes.
Cleaner variant using runAfter
Step 1: Do not run actions with shakeRun
Queue implementation, living, breathing
Use a priority queue to schedule shake actions.
Most user actions are answered immediately with a cache but also
spawn a shake action to check the cached value we consulted was up to
date.
* Remove DelayedActionExtra
* hlint
* Fix progress
* Always block instead of fail on initial computation
* Can block for code lens
* Update docs
Co-authored-by: Zubin Duggal <zubin@cmi.ac.in>
* Cache the results of loadSession until the components change
* Track the cradle dependencies
* hlint
* Add cradle to watched files test
* Add comment on sessionVersion field
* Write a cabal.project file
As suggested in #617. Taken fron #624
* Write a cabal.project.local
Otherwise Cabal still errors out
* Override default hie dir
Otherwise .hi and .hie files end up in different locations, which causes the getDefinition experiment to fail the second time it's run.
This is because we assume in ghcide that .hi and .hie files have the same lifetimes, which is not true when the ..hie files are wiped but the .hi files aren't.
There are a couple of cases to handle as seen below.
Thanks @jneira for help discovering them all.
There used to be linking errors but I no longer see those after the multi-cradle patch
Non Nix
=========
The table below shows a couple of combinations of cradles and ghcide versions in a
non-Nix environment. All the version mismatches are now handled as follows:
- "Cannot satisfy package" - `-package-id` flags referencing
package versions not available (generally base)
- "bad interface" - tried to load an interface file created by a different version of ghc
cradle/ghcide | 8.6 | 8.8 | 8.10
--------------|-----|----|---
Cabal 8.6 | success | cannot satisfy package | cannot satisfy package
Cabal 8.8 | cannot satisfy package | success | cannot satisfy package
Cabal 8.10 | cannot satisfy package | cannot satisfy package | success
Stack 8.6 | success | bad-interface | bad-interfac-
Stack 8.8 | bad-interface | success | bad-interface
Stack 8.10 | bad-interface | bad-interface | success
Nix
=========
Because Nix redefines the libdir to point at the run-time ghc installation,
it's actually much easier to detect a version mismatch:
just compare the compile-time and run-time libdirs
* ShakeSession and shakeRunGently
Currently we start a new Shake session for every interaction with the Shake
database, including type checking, hovers, code actions, completions, etc.
Since only one Shake session can ever exist, we abort the active session if any
in order to execute the new command in a responsive manner.
This is suboptimal in many, many ways:
- A hover in module M aborts the typechecking of module M, only to start over!
- Read-only commands (hover, code action, completion) need to typecheck all the
modules! (or rather, ask Shake to check that the typechecks are current)
- There is no way to run non-interfering commands concurrently
This is an experiment inspired by the 'ShakeQueue' of @mpickering, and
the follow-up discussion in https://github.com/mpickering/ghcide/issues/7
We introduce the concept of the 'ShakeSession' as part of the IDE state.
The 'ShakeSession' is initialized by a call to 'shakeRun', and survives until
the next call to 'shakeRun'. It is important that the session is restarted as
soon as the filesystem changes, to ensure that the database is current.
The 'ShakeSession' enables a new command 'shakeRunGently', which appends work to
the existing 'ShakeSession'. This command can be called in parallel without any
restriction.
* Simplify by assuming there is always a ShakeSession
* Improved naming and docs
* Define runActionSync on top of shakeEnqueue
shakeRun is not correct as it never returns anymore
* Drive progress reporting from newSession
The previous approach reused the shakeProgress thread, which doesn't work anymore as ShakeSession keeps the ShakeDatabase open until the next edit
* Deterministic progress messages in tests
Dropping the 0.1s sleep to ensure that progress messages during tests are
deterministic
* Make kick explicit
This is required for progress reporting to work, see notes in shakeRun
As to whether this is the right thing to do:
1. Less magic, more explicit
2. There's only 2 places where kick is actually used
* apply Neil's feedback
* avoid a deadlock when the enqueued action throws
* Simplify runAction + comments
* use a Barrier for clarity
A Barrier is a smaller abstraction than an MVar, and the next version of the extra package will come with a suitably small implementation:
98c2a83585
* Log timings for code actions, hovers and completions
* Rename shakeRun to shakeRestart
The action returned by shakeRun now blocks until another call to shakeRun is made, which is a change in behaviour,. but all the current uses of shakeRun ignore this action.
Since the new behaviour is not useful, this change simplifies and updates the docs and name accordingly
* delete runActionSync as it's just runAction
* restart shake session on new component created
* requeue pending actions on session restart
* hlint
* Bumped the delay from 5 to 6
* Add a test for the non-lsp command line
* Update exe/Main.hs
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Multi component support
In this commit we add support for loading multiple components into one
ghcide session.
The current behaviour is that each component is loaded lazily into the
session. When a file from an unrecognised component is loaded, the
cradle is consulted again to get a new set of options for the new
component. This will cause all the currently loaded files to be
reloaded into a new HscEnv which is shared by all the currently known
components. The result of this is that functions such as go-to
definition work between components if they have been loaded into the
same session but you have to open at least one file from each component
before it will work.
Only minimal changes are needed to the internals to ghcide to make the
file searching logic look in include directories for all currently
loaded components. The main changes are in exe/Main.hs which has been
heavily rewritten to avoid shake indirections. A global map is created
which maps a filepath to the HscEnv which should be used to compile it.
When a new component is created this map is completely refreshed so each
path maps to a new
Which paths belong to a componenent is determined by the targets listed
by the cradle. Therefore it is important that each cradle also lists all
the targets for the cradle. There are some other choices here as well
which are less accurate such as mapping via include directories which
is the aproach that I implemented in haskell-ide-engine.
The commit has been tested so far with cabal and hadrian.
Also deleted the .ghci file which was causing errors during testing and
seemed broken anyway.
Co-authored-by: Alan Zimmerman <alan.zimm@gmail.com>
Co-authored-by: fendor <power.walross@gmail.com>
* Final tweaks?
* Fix 8.4 build
* Add multi-component test
* Fix hlint
* Add cabal to CI images
* Modify path
* Set PATH in the right place (hopefully)
* Always generate interface files and hie files
* Use correct DynFlags in mkImportDirs
You have to use the DynFlags for the file we are currently compiling to
get the right packages in the package db so that lookupPackage doesn't
always fail.
* Revert "Always generate interface files and hie files"
This reverts commit 820aa24189.
* remove traces
* Another test
* lint
* Unset env vars set my stack
* Fix extra-source-files
As usual, stack doesn’t understand Cabal properly and doesn’t seem to
like ** wildcards so I’ve enumerated it manually.
* Unset env locally
Co-authored-by: Alan Zimmerman <alan.zimm@gmail.com>
Co-authored-by: fendor <power.walross@gmail.com>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Update to hie-bios 0.5.0
* Fix test-cases due to changes in the direct cradle
* Update test/exe/Main.hs comment
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Add a custom prefix to command IDs
A client can run more than one instance of ghcide. Some clients have a global
command registry, and all commands must be unique in that registry.
So to make the command ids unique, prefix them with the ghcide server process
id, as is done in haskell-ide-engine.
* Use same command naming scheme as in haskell-language-server
To ease interoperability
* Use makeLspCommandId for prefixing commands
This puts all the prefixing logic in one place.
* Add hlint exception for CPP in Development.IDE.Compat
* Bring in Win32 dependency for windows build
* Rules for loading interface files
* Typechecking with interface files
* Add a note in the README about the optimal project setup
* Improve support for hs-boot files
The branch was failing to load GHC because the module graph was missing
edges between a .hs file and its .hs-boot file. This means the .hs-boot
file was getting added into the HPT after the .hs file which led to
confusing errors about variables being out of scope.
The fix is to maintain a map from hs-boot to hs files and then add an
edge for this case when calling `transitiveDependencies`.
Also tidy up some code in setupEnv which I assume was attempting to fix
this but in an incorrect manner.
Add the -boot suffix when looking for hi-boot files.
For some reason, the `hi` path is not set to the right thing for
`hs-boot` files. I don't know why not perhaps it is ok to use an
existing `.hi` file in place of an `hs-boot` file. More investigation
needed. My experience is that GHC is quite bad a recompilation avoidance
for hs-boot files anyway.
For example: https://gitlab.haskell.org/ghc/ghc/issues/17434
Add the -boot suffix when writing interface files
* Generate .hi and .hie files during type checking
* Refactor GetModIface to not retain TypeChecked module in memory
This improves memory performance on a cold cache.
* Trailing whitespace
* Turn debug log messages into diagnostics
* Implement "hie" files for ghc-8.6.5
This means that the .hi files patch can also be used with 8.6.5
* Add tests for hover/definition on imported symbols
* hlints
* Generate .hie files when missing
* Fix subtle bug in setDefaultHieDir
* Simplify optimal project setup in README
* Move interface loading diagnostics behind --test flag
Reusing the --test flag for this seems harmless, I cannot justify introducing a
new flag
* Avoid expensive interface file generation for files of interest
* avoid redundant arguments (thanks Moritz K)
* qualify a DAML only comment
* Skip module source when generating hie file
thanks Moritz Kiefer for noting that we don't care for the generated .hie files
to embed module sources
* runGhcEnv <-> evalGhcEnv
* Apply suggestions from code review
Thanks Moritz Kiefer
Co-Authored-By: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Add suggested Show instance
Co-Authored-By: Matthew Pickering <matthewtpickering@gmail.com>
* Use Control.Exception.Safe
This is to avoid accidentally capturing asynchronous exceptions
* Rename atomicFileUpdate
* Fix a flaky test
We have to be careful with module naming in tests to avoid interference of .hi
files across tests
* Undo formatting of D.IDE.GHC.Util
* follow changes in master
Co-authored-by: Matthew Pickering <matthewtpickering@gmail.com>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Use custom version of h-l-t
* Use normalized path functions from h-l-t
* Restore empty path corner case
* Create a wrapper over NFP to override IsString
* Use maybe instead fromMaybe
* Use patched version of lsp-types in all yaml files
* Remove unused import
* Rename specific NormalizeFilePath to NormalizeFilePath'
* Remove specific newtype and IsString instance
* Use released haskell-lsp-0.21
* Adapt to type changes of haskell-lsp-0.21
* Add tags field to CompletionItem
* Fix test case about empty file path
* Correct stack.yaml used in azure ci cache
* Build ghcide including tests in windows azure ci
* Qualify haskell-lsp modules to avoid name clashes
* Working on Plugin support for hls
Fix PluginCommand reply type for executeCommand needs
* Remove PluginCommand
It will move to haskell-language-server instead
* Make azure CI hlint happy
By removing explicit OverloadedStrings pragma, in favour of the one already
enabled in the cabal file.
* Remove unneeded 'do'
* Fix more nits from review
* Fix isWorkspaceFile for relative paths
This fixes a performance regression on GetFileExists
* Avoid interrupting hie-bios when it's doing its thing
I noticed that the GHC hie-bios direct cradle, which uses Hadrian, a Shake build
system, was failing to start due to the following problem:
1. ghcide starts evaluating the LoadCradle node
2. The evaluation gets cancelled
3. Immediately after, ghcide starts evaluating LoadCradle again
4. Hadrian fails, since there is still another Hadrian process alive taking its
Shake lock
* Improve watched files test suite
* Refactor: extract Rules to a separate module
* Add tests for cradle loading
* Fix default extensions in exe target
* Move cradle loaded messages behind a flag
* Use satisfy
Following a suggestion by Moritz Kiefer (@cocreature)
We were calling runCradle multiple times per cradle, concurrently. For Cabal
cradles this function runs Cabal, which is neither fast nor designed to be run
concurrently
* Local hidir and hiedir folders to avoid conflicts with Cabal
hi files created by ghcide would have different optimization settings,
triggering Cabal recompilation if they were stored in Cabal folders
hie files would end up in the src folder as -hiedir is not set by Cabal
Telling GHC to write interface/hie files is pointless since we do it ourselves,
and it also leads to the recompilation checker getting confused
Using hie-bios initSession does things like setting up the cache directory for
interface files and other things which ghcide would rather manage itself.
* linker options
Co-authored-by: Matthew Pickering <matthewtpickering@gmail.com>
* Automatically pick up new dependencies
hie-bios's componentDependencies returns the dependencies of a cradle
that might change the cradle. Add those deps to the shake graph so that
the GHC session is newly created whenever they change.
For that, add a new rule type, GetHscEnvEq, to cache GHC sessions with
the key of GHC options and dependencies. And delete the optGhcSession
field from IdeOptions.
This is for https://github.com/digital-asset/ghcide/issues/50.
hie-bios's componentDependencies can return files that don't exist yet:
https://github.com/mpickering/hie-bios/blob/master/src/HIE/Bios/Types.hs#L90-L93.
This PR handles changes in the existing dependency files, but doesn't
handle newly created dependency files.
* address comments
* revert hie.yaml
* address more comments
* add test
* make direct cradles work; and use direct cradle in test
So that haskell-language-server can use its own config
And separate it out from the IdeConfiguration which is separately set by the
InitializeRequest message.
We have been experiencing a few flaky tests in DAML caused by our CLI
compiler losing diagnostics. The reason for that is the debouncer
which meant that messages got delayed and not send before the process
exited.
This PR makes the debouncer abstract and adds a noopDebouncer which
doesn’t do any debouncing. This is also what we use in the terminal
ghcide test thingy.
