afc02e6970
Streaming benchmarks take constant memory whereas the buffering ones take memory proportional to the stream size. We would like to test them separately. For streaming benchmarks we can use a very large stream size to make sure there is no space leak and we are running in constant memory for long time. |
||
|---|---|---|
| .. | ||
| Streamly/Benchmark | ||
| Adaptive.hs | ||
| Array.hs | ||
| ArrayOps.hs | ||
| BaseStreams.hs | ||
| Chart.hs | ||
| Common.hs | ||
| Concurrent.hs | ||
| FileIO.hs | ||
| Linear.hs | ||
| LinearAsync.hs | ||
| LinearRate.hs | ||
| NanoBenchmarks.hs | ||
| Nested.hs | ||
| NestedConcurrent.hs | ||
| NestedOps.hs | ||
| NestedUnfold.hs | ||
| NestedUnfoldOps.hs | ||
| README.md | ||
| StreamDKOps.hs | ||
| StreamDOps.hs | ||
| StreamKOps.hs | ||
Running Benchmarks
bench.sh script at the root of the repo is the top level driver for running
benchmarks. It runs the requested benchmarks and then creates a report from the
results using the bench-show package. Try bench.sh --help for available
options to run it.
Quick start
Run these commands from the root of the repo.
To run the default benchmarks:
$ ./bench.sh
To run all benchmarks:
$ ./bench.sh --benchmarks all
To run linear and linear-async benchmarks:
$ ./bench.sh --benchmarks "linear linear-async"
To run only the base benchmark and only the benchmarks prefixed with
StreamD in that (anything after a -- is passed to gauge):
$ ./bench.sh --benchmarks base -- StreamD
Comparing benchmarks
To compare two sets of results, first run the benchmarks at the baseline commit:
$ ./bench.sh
And then run with the --append option at the commit that you want to compare
with the baseline. It will show the comparison with the baseline:
$ ./bench.sh --append
Append just adds the next set of results in the same results file. You can keep appending more results and all of them will be compared with the baseline.
You can use --compare to compare the previous commit with the head commit:
$ ./bench.sh --compare
To compare the head commit with some other base commit:
$ ./bench.sh --compare --base d918833
To compare two arbitrary commits:
$ ./bench.sh --compare --base d918833 --candidate 38aa5f2
Note that the above may not always work because the script and the benchmarks
themselves might have changed across the commits. The --append method is more
reliable to compare.
Available Benchmarks
The benchmark names that you can use when running bench.sh:
-
base: a benchmark that measures the raw operations of the basic streamsStreamDandStreamK. -
linear: measures the non-monadic operations of serial streams -
linear-async: measures the non-monadic operations of concurrent streams -
linear-rate: measures the rate limiting operations -
nested: measures the monadic operations of all streams -
all: runs all of the above benchmarks
Reporting without measuring
You can use the --no-measure option to report the already measured results in
the benchmarks results file. A results file may collect an arbitrary number of
results by running with --append multiple times. Each benchmark has its own
results file, for example the linear benchmark has the results file at
charts/linear/results.csv.
You can also manually edit the file to remove a set of results if you like or
to append results from previously saved results or from some other results
file. After editing you can run bench.sh with the --no-measure option to
see the reports corresponding to the results.