haskell/haskell-language-server
1
1
Fork 0
mirror of https://github.com/haskell/haskell-language-server.git synced 2024-07-14 10:40:35 +03:00
Code Issues Projects Releases Wiki Activity
master
haskell-language-server/ghcide-bench/README.md
Pepe Iborra d0e3e0fe3f
HLS benchmarks (#3117) 
* extract ghcide:experiments-types

* extract haskell-language-server:plugins and let go of examples

The main goal here is to move the Plugins module into an internal
library so that it can be reused from the benchmark suite.

In order to
make that easier, and since they hardly serve a purpose in a repository
with 25 plugins, I delete the Example and Example2 plugin descriptors
and their dependencies.

* HLS benchmark suite

Port the ghcide benchmark suite to HLS and benchmark plugin
"configurations" independently.

This includes the following changes to the ghcide benchmark suite and
HLS:
- Support for "configurations" which are defined as sets of plugin ids.
  The benchmark will be run with only these plugins enabled and all
  others disabled
- Support for configurable concurrency. This relies on RTS -ol and -po
  flags to place the RTS traces in the target location rather than in
  the cwd

This change requires two commits, the next one places
ghcide/bench/hist/Main.hs into its final location to help 'git'
recognize the change as a file move

* ghcide/bench/hist/Main.hs -> bench/Main.hs

* CI - fix artifact names for uniqueness

* disable shorten HLS step

* Do not store eventlogs to avoid out of disk space

* render durations up to milliseconds

* shorten titles

Goal is to display the formatted CSV (via column) one row per line

* exclude formatting plugin configurations

* Extract ghcide-bench to a standalone package

* ghcide-bench: fix stderr capturing

* Fix mem stats

We parse maxResidency and allocatedBytes from the RTS -S output, but runSessionWithHandles kills the server without waiting for it to exit and these stats don't get logged.

The solution is to use runSessionWithHandles', but unfortunately it is internal and not exposed. I have raised a PR to expose it and in the meantime we need a source repo package.

* feedbacks

* delete Example plugins
2022-08-25 15:08:57 +01:00

2.8 KiB
Raw Permalink Blame History

A benchmark suite for measuring various performance-related metrics on ghcide and HLS.

Usage

Run with cabal ghcide bench, point it to a haskell-language-server or ghcide binary, specify:

  • the experiment to run, from the ones defined in src/Experiments.hs,
  • the example codebase (either a local folder or a Hackage package),
  • one or more module paths to run the experiment on,
  • the number of samples,
  • any extra command line options to pass to the binary,
Usage: ghcide-bench [(-v|--verbose) | (-q|--quiet)] [--shake-profiling PATH]
                    [--ot-profiling DIR] [--csv PATH] [--stack]
                    [--ghcide-options ARG] [-s|--select ARG] [--samples NAT]
                    [--ghcide PATH] [--timeout ARG]
                    [[--example-package-name ARG]
                      [--example-package-version ARG]
                      [(--example-module PATH)] |
                      --example-path ARG (--example-module PATH)] [--lsp-config]
                    [--no-clean]

Available options:
  --ot-profiling DIR       Enable OpenTelemetry and write eventlog for each
                           benchmark in DIR
  --stack                  Use stack (by default cabal is used)
  --ghcide-options ARG     additional options for ghcide
  -s,--select ARG          select which benchmarks to run
  --samples NAT            override sampling count
  --ghcide PATH            path to ghcide
  --timeout ARG            timeout for waiting for a ghcide response
  --lsp-config             Read an LSP config payload from standard input
  -h,--help                Show this help text

Experiments

Experiments are LSP sessions defined using the lsp-test DSL that run on one or more modules.

Currently the following experiments are defined:

  • edit: makes an edit and waits for re-typechecking
  • hover: asks for hover on an identifier
  • getDefinition: asks for the definitions of an identifier
  • documentsymbols
  • completions: asks for completions on an identifier position
  • code actions: makes an edit that breaks typechecking and asks for code actions
  • hole fit suggestions: measures the performance of hole fits
  • X after edit: combines the edit and X experiments
  • X after cradle edit: combines the X experiments with an edit to the hie.yaml file

One can define additional experiments easily, for e.g. formatting, code lenses, renames, etc. Experiments are defined in the src/Experiments.hs module.

Positions

ghcide-bench will analyze the modules prior to running the experiments, and try to identify the following designated source locations in the module:

  • stringLiteralP: a location that can be mutated without generating a diagnostic,
  • identifierP: a location with an identifier that is not locally defined in the module.
  • docP: a location containing a comment