git.iko.soy/streamly
on master help sign in
Add a README for benchmarks
06242afHarendra Kumar
CONTRIBUTING.md+3
compare with master
unifiedsplit
@@ -75,6 +75,9 @@ any of the existing performance benchmarks. If you introduced something new
75then you may want to add benchmarks to check if it performs as well as expected
75then you may want to add benchmarks to check if it performs as well as expected
76by the programmers to deem it usable.
76by the programmers to deem it usable.
77
77
78+See the README file in the `benchmark` directory for more details on how to run
79+the benchmarks.
80+
78### Changelog
81### Changelog
79
82
80Any new changes that affect the user of the library in some way must be
83Any new changes that affect the user of the library in some way must be
benchmark/README.mdnew file+102
@@ -0,0 +1,102 @@
1+## Running Benchmarks
2+
3+`bench.sh` script at the root of the repo is the top level driver for running
4+benchmarks. It runs the requested benchmarks and then creates a report from the
5+results using the `bench-show` package. Try `bench.sh --help` for available
6+options to run it.
7+
8+## Quick start
9+
10+Run these commands from the root of the repo.
11+
12+To run the default benchmarks:
13+
14+```
15+$ ./bench.sh
16+```
17+
18+To run all benchmarks:
19+
20+```
21+$ ./bench.sh --benchmarks all
22+```
23+
24+To run `linear` and `linear-async` benchmarks:
25+
26+```
27+$ ./bench.sh --benchmarks "linear linear-async"
28+```
29+
30+To run only the base benchmark and only the benchmarks prefixed with
31+`StreamD` in that (anything after a `--` is passed to gauge):
32+
33+```
34+$ ./bench.sh --benchmarks base -- StreamD
35+```
36+
37+## Comparing benchmarks
38+
39+To compare two sets of results, first run the benchmarks at the baseline
40+commit:
41+
42+```
43+$ ./bench.sh
44+```
45+
46+And then run with the `--append` option at the commit that you want to compare
47+with the baseline. It will show the comparison with the baseline:
48+
49+```
50+$ ./bench.sh --append
51+```
52+
53+Append just adds the next set of results in the same results file. You can keep
54+appending more results and all of them will be compared with the baseline.
55+
56+You can use `--compare` to compare the previous commit with the head commit:
57+
58+```
59+$ ./bench.sh --compare
60+```
61+
62+To compare the head commit with some other base commit:
63+
64+```
65+$ ./bench.sh --compare --base d918833
66+```
67+
68+To compare two arbitrary commits:
69+
70+```
71+$ ./bench.sh --compare --base d918833 --candidate 38aa5f2
72+```
73+
74+Note that the above may not always work because the script and the benchmarks
75+themselves might have changed across the commits. The `--append` method is more
76+reliable to compare.
77+
78+## Available Benchmarks
79+
80+The benchmark names that you can use when running `bench.sh`:
81+
82+* `base`: a benchmark that measures the raw operations of the basic streams
83+ `StreamD` and `StreamK`.
84+
85+* `linear`: measures the non-monadic operations of serial streams
86+* `linear-async`: measures the non-monadic operations of concurrent streams
87+* `linear-rate`: measures the rate limiting operations
88+* `nested`: measures the monadic operations of all streams
89+* `all`: runs all of the above benchmarks
90+
91+## Reporting without measuring
92+
93+You can use the `--no-measure` option to report the already measured results in
94+the benchmarks results file. A results file may collect an arbitrary number of
95+results by running with `--append` multiple times. Each benchmark has its own
96+results file, for example the `linear` benchmark has the results file at
97+`charts/linear/results.csv`.
98+
99+You can also manually edit the file to remove a set of results if you like or
100+to append results from previously saved results or from some other results
101+file. After editing you can run `bench.sh` with the `--no-measure` option to
102+see the reports corresponding to the results.