haskell-language-server/ghcide-bench/README.md

A benchmark suite for measuring various performance-related metrics on ghcide and HLS.

## Usage

Run with `cabal ghcide bench`, point it to a `haskell-language-server` or `ghcide` binary, specify:
- the experiment to run, from the ones defined in `src/Experiments.hs`,
- the example codebase (either a local folder or a Hackage package),
- one or more module paths to run the experiment on,
- the number of samples,
- any extra command line options to pass to the binary,

```
Usage: ghcide-bench [(-v|--verbose) | (-q|--quiet)] [--shake-profiling PATH]
                    [--ot-profiling DIR] [--csv PATH] [--stack]
                    [--ghcide-options ARG] [-s|--select ARG] [--samples NAT]
                    [--ghcide PATH] [--timeout ARG]
                    [[--example-package-name ARG]
                      [--example-package-version ARG]
                      [(--example-module PATH)] |
                      --example-path ARG (--example-module PATH)] [--lsp-config]
                    [--no-clean]

Available options:
  --ot-profiling DIR       Enable OpenTelemetry and write eventlog for each
                           benchmark in DIR
  --stack                  Use stack (by default cabal is used)
  --ghcide-options ARG     additional options for ghcide
  -s,--select ARG          select which benchmarks to run
  --samples NAT            override sampling count
  --ghcide PATH            path to ghcide
  --timeout ARG            timeout for waiting for a ghcide response
  --lsp-config             Read an LSP config payload from standard input
  -h,--help                Show this help text
```

## Experiments

Experiments are LSP sessions defined using the `lsp-test` DSL that run on one or
more modules.

Currently the following experiments are defined:
- *edit*: makes an edit and waits for re-typechecking
- *hover*: asks for hover on an identifier
- *getDefinition*: asks for the definitions of an identifier
- *documentsymbols*
- *completions*: asks for completions on an identifier position
- *code actions*: makes an edit that breaks typechecking and asks for code actions
- *hole fit suggestions*: measures the performance of hole fits
- *X after edit*: combines the *edit* and X experiments
- *X after cradle edit*: combines the X experiments with an edit to the `hie.yaml` file

One can define additional experiments easily, for e.g. formatting, code lenses, renames, etc.
Experiments are defined in the `src/Experiments.hs` module.

### Positions
`ghcide-bench` will analyze the modules prior to running the experiments,
and try to identify the following designated source locations in the module:

- *stringLiteralP*: a location that can be mutated without generating a diagnostic,
- *identifierP*: a location with an identifier that is not locally defined in the module.
- *docP*: a location containing a comment
HLS benchmarks (#3117) * extract ghcide:experiments-types * extract haskell-language-server:plugins and let go of examples The main goal here is to move the Plugins module into an internal library so that it can be reused from the benchmark suite. In order to make that easier, and since they hardly serve a purpose in a repository with 25 plugins, I delete the Example and Example2 plugin descriptors and their dependencies. * HLS benchmark suite Port the ghcide benchmark suite to HLS and benchmark plugin "configurations" independently. This includes the following changes to the ghcide benchmark suite and HLS: - Support for "configurations" which are defined as sets of plugin ids. The benchmark will be run with only these plugins enabled and all others disabled - Support for configurable concurrency. This relies on RTS -ol and -po flags to place the RTS traces in the target location rather than in the cwd This change requires two commits, the next one places ghcide/bench/hist/Main.hs into its final location to help 'git' recognize the change as a file move * ghcide/bench/hist/Main.hs -> bench/Main.hs * CI - fix artifact names for uniqueness * disable shorten HLS step * Do not store eventlogs to avoid out of disk space * render durations up to milliseconds * shorten titles Goal is to display the formatted CSV (via column) one row per line * exclude formatting plugin configurations * Extract ghcide-bench to a standalone package * ghcide-bench: fix stderr capturing * Fix mem stats We parse maxResidency and allocatedBytes from the RTS -S output, but runSessionWithHandles kills the server without waiting for it to exit and these stats don't get logged. The solution is to use runSessionWithHandles', but unfortunately it is internal and not exposed. I have raised a PR to expose it and in the meantime we need a source repo package. * feedbacks * delete Example plugins 2022-08-25 17:08:57 +03:00			`A benchmark suite for measuring various performance-related metrics on ghcide and HLS.`

			`## Usage`

			Run with `cabal ghcide bench`, point it to a `haskell-language-server` or `ghcide` binary, specify:
			- the experiment to run, from the ones defined in `src/Experiments.hs`,
			`- the example codebase (either a local folder or a Hackage package),`
			`- one or more module paths to run the experiment on,`
			`- the number of samples,`
			`- any extra command line options to pass to the binary,`

			```
			`Usage: ghcide-bench [(-v\|--verbose) \| (-q\|--quiet)] [--shake-profiling PATH]`
			`[--ot-profiling DIR] [--csv PATH] [--stack]`
			`[--ghcide-options ARG] [-s\|--select ARG] [--samples NAT]`
			`[--ghcide PATH] [--timeout ARG]`
			`[[--example-package-name ARG]`
			`[--example-package-version ARG]`
			`[(--example-module PATH)] \|`
			`--example-path ARG (--example-module PATH)] [--lsp-config]`
			`[--no-clean]`

			`Available options:`
			`--ot-profiling DIR Enable OpenTelemetry and write eventlog for each`
			`benchmark in DIR`
			`--stack Use stack (by default cabal is used)`
			`--ghcide-options ARG additional options for ghcide`
			`-s,--select ARG select which benchmarks to run`
			`--samples NAT override sampling count`
			`--ghcide PATH path to ghcide`
			`--timeout ARG timeout for waiting for a ghcide response`
			`--lsp-config Read an LSP config payload from standard input`
			`-h,--help Show this help text`
			```

			`## Experiments`

			Experiments are LSP sessions defined using the `lsp-test` DSL that run on one or
			`more modules.`

			`Currently the following experiments are defined:`
			`- edit: makes an edit and waits for re-typechecking`
			`- hover: asks for hover on an identifier`
			`- getDefinition: asks for the definitions of an identifier`
			`- documentsymbols`
			`- completions: asks for completions on an identifier position`
			`- code actions: makes an edit that breaks typechecking and asks for code actions`
			`- hole fit suggestions: measures the performance of hole fits`
			`- X after edit: combines the edit and X experiments`
			- X after cradle edit: combines the X experiments with an edit to the `hie.yaml` file

			`One can define additional experiments easily, for e.g. formatting, code lenses, renames, etc.`
			Experiments are defined in the `src/Experiments.hs` module.

			`### Positions`
			`ghcide-bench` will analyze the modules prior to running the experiments,
			`and try to identify the following designated source locations in the module:`

			`- stringLiteralP: a location that can be mutated without generating a diagnostic,`
			`- identifierP: a location with an identifier that is not locally defined in the module.`
			`- docP: a location containing a comment`