@@ -7,34 +7,52 @@ options to run it.
8## Quick start
8## Quick start
10−Run these commands from the root of the repo.
10+You must be in the repo root directory to run these commands.
12−To run the default benchmarks:
12+Run the default benchmark suites:
15−$ ./bench.sh
15+$ ./bench.sh --quick
18+You can remove the `--quick` option to run benchmark with lower speed but
19+better accuracy, or use `--slow` to even further lower the speed and increase
22+Show available benchmark suites:
25+$ ./bench.sh --benchmarks help
28+Run all benchmark suites in the `serial_grp` group:
31+$ ./bench.sh --benchmarks serial_grp
18−To run all benchmarks:
34+Run `Prelude.Serial` and `Data.Parser` benchmark suites:
21−$ ./bench.sh --benchmarks all
37+$ ./bench.sh --benchmarks "Prelude.Serial Data.Parser"
24−To run `linear` and `linear-async` benchmarks:
40+Run all O(1) space complexity benchmarks in `Prelude.Serial` suite:
27−$ ./bench.sh --benchmarks "linear linear-async"
43+$ ./bench.sh --benchmarks Prelude.Serial -- Prelude.Serial/o-1-space
30−To run only the base benchmark and only the benchmarks prefixed with
31−`StreamD` in that (anything after a `--` is passed to gauge):
46+Anything after a `--` is passed to gauge, it basically selects all benchmarks
47+starting with `Prelude.Serial/o-1-space` prefix.
49+Run a specific benchmark in `Prelude.Serial` suite:
34−$ ./bench.sh --benchmarks base -- StreamD
52+$ ./bench.sh --benchmarks Prelude.Serial -- Prelude.Serial/o-1-space/generation/unfoldr
37−## Comparing benchmarks
55+## Comparing results for regression
39To compare two sets of results, first run the benchmarks at the baseline
57To compare two sets of results, first run the benchmarks at the baseline
@@ -75,26 +93,30 @@ Note that the above may not always work because the script and the benchmarks
75themselves might have changed across the commits. The `--append` method is more
93themselves might have changed across the commits. The `--append` method is more
76reliable to compare.
94reliable to compare.
78−## Available Benchmarks
96+## Comparing benchmark suites
98+First see the available benchmark suites:
80−The benchmark names that you can use when running `bench.sh`:
100+```
101+$ ./bench.sh --benchmarks help
82−* `base`: a benchmark that measures the raw operations of the basic streams
104+You will see some benchmark suites end with `_cmp`, these are comparison
83− `StreamD` and `StreamK`.
105+groups. If you run a comparsion group benchmark, comparison of all the
106+benchmark suites in that group will be shown in the end. For example to compare
107+all array benchmark suites:
85−* `linear`: measures the non-monadic operations of serial streams
109+```
86−* `linear-async`: measures the non-monadic operations of concurrent streams
110+$ ./bench.sh --benchmarks array_cmp
87−* `linear-rate`: measures the rate limiting operations
111+```
88−* `nested`: measures the monadic operations of all streams
89−* `all`: runs all of the above benchmarks
91## Reporting without measuring
113## Reporting without measuring
93You can use the `--no-measure` option to report the already measured results in
115You can use the `--no-measure` option to report the already measured results in
94the benchmarks results file. A results file may collect an arbitrary number of
116the benchmarks results file. A results file may collect an arbitrary number of
95results by running with `--append` multiple times. Each benchmark has its own
117results by running with `--append` multiple times. Each benchmark has its own
96−results file, for example the `linear` benchmark has the results file at
118+results file, for example the `Prelude.Serial` benchmark has the results file at
97−`charts/linear/results.csv`.
119+`charts/Prelude.Serial/results.csv`.
99You can also manually edit the file to remove a set of results if you like or
121You can also manually edit the file to remove a set of results if you like or
100to append results from previously saved results or from some other results
122to append results from previously saved results or from some other results