git.iko.soy/streamly
on bf4e5d8 help sign in
streamly/benchmark/README.md
102 lines 2.8 kB history raw
1## Running Benchmarks
3`bench.sh` script at the root of the repo is the top level driver for running
4benchmarks. It runs the requested benchmarks and then creates a report from the
5results using the `bench-show` package. Try `bench.sh --help` for available
6options to run it.
8## Quick start
10Run these commands from the root of the repo.
12To run the default benchmarks:
14```
15$ ./bench.sh
16```
18To run all benchmarks:
20```
21$ ./bench.sh --benchmarks all
22```
24To run `linear` and `linear-async` benchmarks:
26```
27$ ./bench.sh --benchmarks "linear linear-async"
28```
30To run only the base benchmark and only the benchmarks prefixed with
31`StreamD` in that (anything after a `--` is passed to gauge):
33```
34$ ./bench.sh --benchmarks base -- StreamD
35```
37## Comparing benchmarks
39To compare two sets of results, first run the benchmarks at the baseline
40commit:
42```
43$ ./bench.sh
44```
46And then run with the `--append` option at the commit that you want to compare
47with the baseline. It will show the comparison with the baseline:
49```
50$ ./bench.sh --append
51```
53Append just adds the next set of results in the same results file. You can keep
54appending more results and all of them will be compared with the baseline.
56You can use `--compare` to compare the previous commit with the head commit:
58```
59$ ./bench.sh --compare
60```
62To compare the head commit with some other base commit:
64```
65$ ./bench.sh --compare --base d918833
66```
68To compare two arbitrary commits:
70```
71$ ./bench.sh --compare --base d918833 --candidate 38aa5f2
72```
74Note that the above may not always work because the script and the benchmarks
75themselves might have changed across the commits. The `--append` method is more
76reliable to compare.
78## Available Benchmarks
80The benchmark names that you can use when running `bench.sh`:
82* `base`: a benchmark that measures the raw operations of the basic streams
83 `StreamD` and `StreamK`.
85* `linear`: measures the non-monadic operations of serial streams
86* `linear-async`: measures the non-monadic operations of concurrent streams
87* `linear-rate`: measures the rate limiting operations
88* `nested`: measures the monadic operations of all streams
89* `all`: runs all of the above benchmarks
91## Reporting without measuring
93You can use the `--no-measure` option to report the already measured results in
94the benchmarks results file. A results file may collect an arbitrary number of
95results by running with `--append` multiple times. Each benchmark has its own
96results file, for example the `linear` benchmark has the results file at
97`charts/linear/results.csv`.
99You can also manually edit the file to remove a set of results if you like or
100to append results from previously saved results or from some other results
101file. After editing you can run `bench.sh` with the `--no-measure` option to
102see the reports corresponding to the results.