git.iko.soy/streamly
on master help sign in
Increase time-limit from 0 to 0.01

In the "reverse'" benchmark the GC stats were showing as 0 because no GC occurred in 1 iteration.

commit6a33a88980cae0f8309e3890507f21399ab28c80 authorHarendra Kumar <harendra@composewell.com> dateTue 26 Jan 2021, 15:32 (6 years ago) parent1e099de
3 files changed+22−2
compare with master
unifiedsplit
benchmark/README.md+14 bin/bench-exec-one.sh+2−1 bin/bench.sh+6−1
benchmark/README.md+14
@@ -227,3 +227,17 @@ $ cabal run FileSystem.Handle -- --quick FileSystem.Handle/o-1-space/reduce/read
227227
228228The automatic tests do not test unicode input, this option is useful to specify
229229a unicode text file manually.
230+
231+## Benchmarking notes
232+
233+We use gauge instead of criterion for benchmarking. We have fixed
234+several issues in gauge inherited from criterion. We have added several
235+features as well e.g. rusage stats, running benchmarks in an isolated
236+process etc which are crucial to our benchmarking analysis process.
237+
238+### Gotchas
239+
240+Gauge forces a GC before and after the measurement. However, we have observed
241+that sometimes the GC stats may not be accurate when the number of iterations
242+in the measurement is small (e.g. 1 iteration). In such cases usually the
243+number of GCs and GC times would also be 0.
bin/bench-exec-one.sh+2−1
@@ -95,7 +95,8 @@ bench_rts_opts_specific () {
9595# Speed options
9696#------------------------------------------------------------------------------
9797
98−SUPER_QUICK_OPTIONS="--quick --min-duration 0 --time-limit 0 --include-first-iter"
98+# Do not keep time limit as 0 otherwise GC stats may remain 0 in some cases.
99+SUPER_QUICK_OPTIONS="--quick --min-duration 0 --time-limit 0.01 --include-first-iter"
99100QUICKER_OPTIONS="--min-samples 3 --time-limit 1"
100101
101102# For certain long benchmarks if the user has not requested super quick
bin/bench.sh+6−1
@@ -163,7 +163,12 @@ target_exe_extra_args () {
163163 SPEED_OPTIONS="$QUICK_OPTS --min-samples 10 --time-limit 1"
164164 else
165165 # super quick but less accurate
166− SPEED_OPTIONS="$QUICK_OPTS --time-limit 0 --include-first-iter"
166+ # When the time-limit is too low and the benchmark is tiny,
167+ # then if the number of iterations is very small the GC stats
168+ # may remain 0. So keep the time-limit at a minimum of 10 ms
169+ # to collect significant stats. The problem was observed in
170+ # the Prelude.Serial/reverse' benchmark.
171+ SPEED_OPTIONS="$QUICK_OPTS --time-limit 0.01 --include-first-iter"
167172 fi
168173 else
169174 # Slow but more accurate mode