@@ -0,0 +1,102 @@
1+## Running Benchmarks
2+
3+`bench.sh` script at the root of the repo is the top level driver for running
4+benchmarks. It runs the requested benchmarks and then creates a report from the
5+results using the `bench-show` package. Try `bench.sh --help` for available
6+options to run it.
7+
8+## Quick start
9+
10+Run these commands from the root of the repo.
11+
12+To run the default benchmarks:
13+
14+```
15+$ ./bench.sh
16+```
17+
18+To run all benchmarks:
19+
20+```
21+$ ./bench.sh --benchmarks all
22+```
23+
24+To run `linear` and `linear-async` benchmarks:
25+
26+```
27+$ ./bench.sh --benchmarks "linear linear-async"
28+```
29+
30+To run only the base benchmark and only the benchmarks prefixed with
31+`StreamD` in that (anything after a `--` is passed to gauge):
32+
33+```
34+$ ./bench.sh --benchmarks base -- StreamD
35+```
36+
37+## Comparing benchmarks
38+
39+To compare two sets of results, first run the benchmarks at the baseline
40+commit:
41+
42+```
43+$ ./bench.sh
44+```
45+
46+And then run with the `--append` option at the commit that you want to compare
47+with the baseline. It will show the comparison with the baseline:
48+
49+```
50+$ ./bench.sh --append
51+```
52+
53+Append just adds the next set of results in the same results file. You can keep
54+appending more results and all of them will be compared with the baseline.
55+
56+You can use `--compare` to compare the previous commit with the head commit:
57+
58+```
59+$ ./bench.sh --compare
60+```
61+
62+To compare the head commit with some other base commit:
63+
64+```
65+$ ./bench.sh --compare --base d918833
66+```
67+
68+To compare two arbitrary commits:
69+
70+```
71+$ ./bench.sh --compare --base d918833 --candidate 38aa5f2
72+```
73+
74+Note that the above may not always work because the script and the benchmarks
75+themselves might have changed across the commits. The `--append` method is more
76+reliable to compare.
77+
78+## Available Benchmarks
79+
80+The benchmark names that you can use when running `bench.sh`:
81+
82+* `base`: a benchmark that measures the raw operations of the basic streams
83+ `StreamD` and `StreamK`.
84+
85+* `linear`: measures the non-monadic operations of serial streams
86+* `linear-async`: measures the non-monadic operations of concurrent streams
87+* `linear-rate`: measures the rate limiting operations
88+* `nested`: measures the monadic operations of all streams
89+* `all`: runs all of the above benchmarks
90+
91+## Reporting without measuring
92+
93+You can use the `--no-measure` option to report the already measured results in
94+the benchmarks results file. A results file may collect an arbitrary number of
95+results by running with `--append` multiple times. Each benchmark has its own
96+results file, for example the `linear` benchmark has the results file at
97+`charts/linear/results.csv`.
98+
99+You can also manually edit the file to remove a set of results if you like or
100+to append results from previously saved results or from some other results
101+file. After editing you can run `bench.sh` with the `--no-measure` option to
102+see the reports corresponding to the results.