3773fca · streamly · git.iko.soy

git.iko.soy/streamly

on master help sign in

Update the benchmarks guide and fix the default.nix of bench-runner

3773fcaAdithya Kumar4 years ago

docs/Developer/Benchmarks.md+43−38

compare with master

unifiedsplit

@@ -1,18 +1,7 @@

1# Benchmarking

3−## Benchmark Drivers

4−

5−Two benchmark drivers are supported:

6−

7−* `tasty-bench` (default)

8−* `gauge` (enabled by `--use-gauge` build flag)

9−

10## Build and run benchmarks directly

3## Build and run benchmarks directly

12−The benchmark executables are `tasty-bench` executables unless you have

13−passed `--use-gauge` cabal flag when building in which case it is a

14−`gauge` executable.

15−

16```

5```

17$ cabal run bench:Prelude.Serial # run selected

6$ cabal run bench:Prelude.Serial # run selected

18$ cabal run bench:Prelude.Serial -- --help # help on arguments

7$ cabal run bench:Prelude.Serial -- --help # help on arguments

@@ -26,53 +15,68 @@ $ cabal build --enable-benchmarks all # build all, alternate method

26$ cabal build --flag "-opt" ... # disable optimization, faster build

15$ cabal build --flag "-opt" ... # disable optimization, faster build

27```

16```

29−## Building and Running Benchmarks with bench.sh

18+## Building and Running Benchmarks with bench-runner

31−`<streamly repo>/bin/bench.sh` script is the top level driver for

20+The executable `bench-runner` is the top level driver for

32running benchmarks. It runs the requested benchmarks and then creates a

21running benchmarks. It runs the requested benchmarks and then creates a

33report from the results using the `bench-show` package.

22report from the results using the `bench-show` package.

35−IMPORTANT NOTE: The first time you run this script it may take a long

24+IMPORTANT NOTE: The first time you run this executable it may take a long

36time because it has to build the `bench-report` executable which has a

25time because it has to build the `bench-report` executable which has a

37−lot of dependencies. If you are using nix then use `--use-nix` flag

26+lot of dependencies.

38−for the first time so that the `bench-report` executable is built using

27+

39−nix. That can save a lot of time compiling it. However, once it is built

28+If you are using nix then use `benchmark/bench-runner/default.nix`. Build it

40−it will be cached in the `bin` directory of the repo and used from

29+manually and install it in the root of the repository like so,

41−there every time. You can also build it manually from the cabal file in

30+```

42−`benchmark/bench-report` and install it in the `bin` directory.

31+$ cd benchmark/bench-runner

32+$ nix-shell --run 'cabal install bench-runner --installdir=../../ --overwrite-policy=always'

33+$ cd ../../

34+```

35+

36+You can also use `cabal.project.report` to directly build an run the `bench-runner` like so,

37+```

38+$ cabal run bench-runner --project-file=cabal.project.report -- <bench-runner-args>

39+```

44## bench.sh: Quick start

41## bench.sh: Quick start

43+Assuming `bench-runner` is the executable. You can replace `bench-runner` with

44+`cabal run bench-runner --project-file=cabal.project.report --`

45+

46Useful commands:

48```

49−$ bin/bench.sh --help

49+$ ./bench-runner --help

50−$ bin/bench.sh --quick # run all the benchmark suites

50+$ ./bench-runner --quick # run all the benchmark suites

51−$ bin/bench.sh --benchmarks help # Show available benchmark suites

51+$ ./bench-runner --targets help # Show available benchmark suites

52−$ bin/bench.sh --benchmarks serial_grp # Run all serial benchmark suites

52+$ ./bench-runner --targets serial_grp # Run all serial benchmark suites

53−$ bin/bench.sh --benchmarks "Prelude.Serial Data.Parser" # run selected suites

53+$ ./bench-runner --targets "Prelude.Serial Data.Parser" # run selected suites

54−$ bin/bench.sh --no-measure # don't run benchmarks just show previous results

54+$ ./bench-runner --no-measure # don't run benchmarks just show previous results

56# Run all O(1) space complexity benchmarks in `Prelude.Serial` suite

57−$ bin/bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space

57+$ ./bench-runner --targets Prelude.Serial --prefix Prelude.Serial/o-1-space

59# Run a specific benchmark in `Prelude.Serial` suite

60−$ bin/bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr

60+$ ./bench-runner --targets Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr

61```

63−Note: `bench.sh` enables fusion-plugin by default.

63+Note: `bench-runner` enables fusion-plugin by default.

65## Comparing results with baseline

67```

68# Checkout baseline commit

69−$ bin/bench.sh --quick

69+$ ./bench-runner --quick

71# Checkout commit with new changes

72−$ bin/bench.sh --quick --append

72+$ ./bench-runner --quick --append

74# To add another result to comparisons just repeat the above command on

75# desired commit

76+

77+# To display the current results without running the benchmarks.

78+# See "Reporting without measuring" for more info.

79+$ ./bench-runner --no-measure

76```

80```

78## Comparing benchmark suites

82## Comparing benchmark suites

@@ -80,7 +84,7 @@ $ bin/bench.sh --quick --append

80First see the available benchmark suites:

84First see the available benchmark suites:

82```

86```

83−$ bin/bench.sh --benchmarks help

87+$ ./bench-runner --targets help

84```

88```

86You will see some benchmark suites end with `_cmp`, these are comparison

90You will see some benchmark suites end with `_cmp`, these are comparison

@@ -89,7 +93,7 @@ benchmark suites in that group will be shown in the end. For example to compare

89all array benchmark suites:

93all array benchmark suites:

91```

95```

92−$ bin/bench.sh --benchmarks array_cmp

96+$ ./bench-runner --targets array_cmp

93```

97```

95## Reporting without measuring

99## Reporting without measuring

@@ -102,7 +106,7 @@ results file, for example the `Prelude.Serial` benchmark has the results file at

102

106

103You can also manually edit the file to remove a set of results if you like or

107You can also manually edit the file to remove a set of results if you like or

104to append results from previously saved results or from some other results

108to append results from previously saved results or from some other results

105−file. After editing you can run `bench.sh` with the `--no-measure` option to

109+file. After editing you can run `bench-runner` with the `--no-measure` option to

106see the reports corresponding to the results.

110see the reports corresponding to the results.

107

111

108## Additional benchmark configuration

112## Additional benchmark configuration

@@ -134,7 +138,7 @@ a unicode text file manually.

134To run the unicode benchmarks on valid utf8 input, you can do the following,

138To run the unicode benchmarks on valid utf8 input, you can do the following,

135

139

136```

140```

137−$ Benchmark_FileSystem_Handle_InputFile=<valid-unicode-filepath> bin/bench.sh --benchmarks Unicode.Stream --cabal-build-options "-f include-strict-utf8"

141+$ Benchmark_FileSystem_Handle_InputFile=<valid-unicode-filepath> ./bench-runner --benchmarks Unicode.Stream --cabal-build-options "-f include-strict-utf8"

138```

142```

139

143

140## Benchmarking notes

144## Benchmarking notes

@@ -145,7 +149,8 @@ benchmark.

145

149

146### Gotchas

150### Gotchas

147

151

148−Gauge forces a GC before and after the measurement. However, we have observed

152+The benchmark driver forces a GC before and after the measurement.

153+However, we have observed

149that sometimes the GC stats may not be accurate when the number of iterations

154that sometimes the GC stats may not be accurate when the number of iterations

150in the measurement is small (e.g. 1 iteration). In such cases usually the

155in the measurement is small (e.g. 1 iteration). In such cases usually the

151number of GCs and GC times would also be 0.

156number of GCs and GC times would also be 0.

@@ -168,10 +173,10 @@ freeze file so that the versions of all libraries are pinned.

168

173

169There are two ways to find problematic code:

174There are two ways to find problematic code:

170

175

171−1. Run performance benchmarks using `bench.sh`, select the benchmarks

176+1. Run performance benchmarks using `bench-runner`, select the benchmarks

172 that are taking more than expected time.

177 that are taking more than expected time.

1732. When making a new change, compare with the baseline and select benchmarks

1782. When making a new change, compare with the baseline and select benchmarks

174− with the most regression reported by `bench.sh`.

179+ with the most regression reported by `bench-runner`.

175

180

176Number of allocations are the most stable measure that do not vary from

181Number of allocations are the most stable measure that do not vary from

177run to run. `cpuTime` and `bytesCopied` may vary. When comparing two

182run to run. `cpuTime` and `bytesCopied` may vary. When comparing two