6a33a88 · streamly · git.iko.soy

benchmark/README.md+14

compare with master

unifiedsplit

@@ -227,3 +227,17 @@ $ cabal run FileSystem.Handle -- --quick FileSystem.Handle/o-1-space/reduce/read

227

228The automatic tests do not test unicode input, this option is useful to specify

229a unicode text file manually.

230+

231+## Benchmarking notes

232+

233+We use gauge instead of criterion for benchmarking. We have fixed

234+several issues in gauge inherited from criterion. We have added several

235+features as well e.g. rusage stats, running benchmarks in an isolated

236+process etc which are crucial to our benchmarking analysis process.

237+

238+### Gotchas

239+

240+Gauge forces a GC before and after the measurement. However, we have observed

241+that sometimes the GC stats may not be accurate when the number of iterations

242+in the measurement is small (e.g. 1 iteration). In such cases usually the

243+number of GCs and GC times would also be 0.

bin/bench-exec-one.sh+2−1

@@ -95,7 +95,8 @@ bench_rts_opts_specific () {

95# Speed options

96#------------------------------------------------------------------------------

97

98+# Do not keep time limit as 0 otherwise GC stats may remain 0 in some cases.

98−SUPER_QUICK_OPTIONS="--quick --min-duration 0 --time-limit 0 --include-first-iter"

99+SUPER_QUICK_OPTIONS="--quick --min-duration 0 --time-limit 0.01 --include-first-iter"

99QUICKER_OPTIONS="--min-samples 3 --time-limit 1"

100QUICKER_OPTIONS="--min-samples 3 --time-limit 1"

100

101

101# For certain long benchmarks if the user has not requested super quick

102# For certain long benchmarks if the user has not requested super quick

bin/bench.sh+6−1

@@ -163,7 +163,12 @@ target_exe_extra_args () {

163 SPEED_OPTIONS="$QUICK_OPTS --min-samples 10 --time-limit 1"

164 else

165 # super quick but less accurate

166+ # When the time-limit is too low and the benchmark is tiny,

167+ # then if the number of iterations is very small the GC stats

168+ # may remain 0. So keep the time-limit at a minimum of 10 ms

169+ # to collect significant stats. The problem was observed in

170+ # the Prelude.Serial/reverse' benchmark.

166− SPEED_OPTIONS="$QUICK_OPTS --time-limit 0 --include-first-iter"

171+ SPEED_OPTIONS="$QUICK_OPTS --time-limit 0.01 --include-first-iter"

167 fi

172 fi

168 else

173 else

169 # Slow but more accurate mode

174 # Slow but more accurate mode