haskell-language-server/ghcide-bench

A benchmark suite for measuring various performance-related metrics on ghcide and HLS.

Usage

Run with cabal ghcide bench, point it to a haskell-language-server or ghcide binary, specify:

• the experiment to run, from the ones defined in src/Experiments.hs,
• the example codebase (either a local folder or a Hackage package),
• one or more module paths to run the experiment on,
• the number of samples,
• any extra command line options to pass to the binary,

Usage: ghcide-bench [(-v|--verbose) | (-q|--quiet)] [--shake-profiling PATH]
                    [--ot-profiling DIR] [--csv PATH] [--stack]
                    [--ghcide-options ARG] [-s|--select ARG] [--samples NAT]
                    [--ghcide PATH] [--timeout ARG]
                    [[--example-package-name ARG]
                      [--example-package-version ARG]
                      [(--example-module PATH)] |
                      --example-path ARG (--example-module PATH)] [--lsp-config]
                    [--no-clean]

Available options:
  --ot-profiling DIR       Enable OpenTelemetry and write eventlog for each
                           benchmark in DIR
  --stack                  Use stack (by default cabal is used)
  --ghcide-options ARG     additional options for ghcide
  -s,--select ARG          select which benchmarks to run
  --samples NAT            override sampling count
  --ghcide PATH            path to ghcide
  --timeout ARG            timeout for waiting for a ghcide response
  --lsp-config             Read an LSP config payload from standard input
  -h,--help                Show this help text

Experiments

Experiments are LSP sessions defined using the lsp-test DSL that run on one or more modules.

Currently the following experiments are defined:

• edit: makes an edit and waits for re-typechecking
• hover: asks for hover on an identifier
• getDefinition: asks for the definitions of an identifier
• documentsymbols
• completions: asks for completions on an identifier position
• code actions: makes an edit that breaks typechecking and asks for code actions
• hole fit suggestions: measures the performance of hole fits
• X after edit: combines the edit and X experiments
• X after cradle edit: combines the X experiments with an edit to the hie.yaml file

One can define additional experiments easily, for e.g. formatting, code lenses, renames, etc. Experiments are defined in the src/Experiments.hs module.

Positions

ghcide-bench will analyze the modules prior to running the experiments, and try to identify the following designated source locations in the module:

• stringLiteralP: a location that can be mutated without generating a diagnostic,
• identifierP: a location with an identifier that is not locally defined in the module.
• docP: a location containing a comment