ilyakooo0/streamly
1
0
Fork 0
mirror of https://github.com/ilyakooo0/streamly.git synced 2024-11-12 22:25:41 +03:00
Code Issues Packages Projects Releases Wiki Activity
f25a1dd28e
streamly/bin/bench-exec-one.sh

176 lines
5.5 KiB
Bash
Raw Normal View History

Use a "bash" shell from PATH In case of nix environment we may be using bash from non-standard path.
2020-10-23 12:00:04 +03:00
#!/usr/bin/env bash
Refactor Prelude benchmarks * Now benchmark modules correspond to source modules. The Prelude module in source corresponds to several modules one for each stream type. * Benchmarks in the same order/groupings as they appear in source * All benchmarks now have division according to space complexity * Refactoring reduces a lot of code duplication especially the stream generation and elimination functions. * The RTS options are now completely set in the shell script to run the benchmarks. * RTS options can be set on a per benchmark basis. RTS options work correctly now. * The set of streaming/infinite stream benchmarks is now complete and we can run all such benchmarks coneveniently. * Benchmark "quick"/"speed" options can now be specified on a per benchmark basis. Longer benchmarks can have fewer iterations/quick run time. * Benchmarks are grouped in several groups which can be run on a per group basis. Comparison groups are also defined for convenient comparisons of different modules (e.g. arrays or streamD/K). * The benchmark namespaces are grouped in a consistent manner. Benchmark executables have a consistent naming based on module names.
2020-06-06 22:21:33 +03:00
# Environment passed:
# BENCH_EXEC_PATH: the benchmark executable
# RTS_OPTIONS: additional RTS options
# QUICK_MODE: whether we are in quick mode
Optimize Functor/Applicative/Monad etc for serial Some of the benchmarks were order of magnitude off due to missing INLINE for type class operations. Now, all of them are in reasonable limits. Benchmarks affected for serial streams: * Functor, Applicative, Monad, transformers We need to do a similar exercise for other types of streams and for folds/parsers as well.
2020-06-23 10:01:59 +03:00
# $1: message
die () {
>&2 echo -e "Error: $1"
exit 1
}
warn () {
>&2 echo -e "Warning: $1"
}
test -n "$BENCH_EXEC_PATH" || die "BENCH_EXEC_PATH env var must be set"
test -n "$QUICK_MODE" || warn "QUICK_MODE env var not set (to 0 or 1)"
Refactor Prelude benchmarks * Now benchmark modules correspond to source modules. The Prelude module in source corresponds to several modules one for each stream type. * Benchmarks in the same order/groupings as they appear in source * All benchmarks now have division according to space complexity * Refactoring reduces a lot of code duplication especially the stream generation and elimination functions. * The RTS options are now completely set in the shell script to run the benchmarks. * RTS options can be set on a per benchmark basis. RTS options work correctly now. * The set of streaming/infinite stream benchmarks is now complete and we can run all such benchmarks coneveniently. * Benchmark "quick"/"speed" options can now be specified on a per benchmark basis. Longer benchmarks can have fewer iterations/quick run time. * Benchmarks are grouped in several groups which can be run on a per group basis. Comparison groups are also defined for convenient comparisons of different modules (e.g. arrays or streamD/K). * The benchmark namespaces are grouped in a consistent manner. Benchmark executables have a consistent naming based on module names.
2020-06-06 22:21:33 +03:00
#------------------------------------------------------------------------------
# RTS Options
#------------------------------------------------------------------------------
# RTS options based on the benchmark executable
bench_exe_rts_opts () {
case "$1" in
Prelude.Concurrent*) echo -n "-K256K -M384M" ;;
*) echo -n "" ;;
esac
}
# General RTS options for different classes of benchmarks
bench_rts_opts_default () {
case "$1" in
*/o-1-sp*) echo -n "-K36K -M16M" ;;
*/o-n-h*) echo -n "-K36K -M32M" ;;
*/o-n-st*) echo -n "-K1M -M16M" ;;
*/o-n-sp*) echo -n "-K1M -M32M" ;;
*) echo -n "" ;;
esac
}
# Overrides for specific benchmarks
bench_rts_opts_specific () {
case "$1" in
Increase stack limit for StreamK/toList benchmark
2020-07-02 14:33:25 +03:00
Data.Stream.StreamK/o-n-space/elimination/toList) echo -n "-K2M" ;;
Refactor Prelude benchmarks * Now benchmark modules correspond to source modules. The Prelude module in source corresponds to several modules one for each stream type. * Benchmarks in the same order/groupings as they appear in source * All benchmarks now have division according to space complexity * Refactoring reduces a lot of code duplication especially the stream generation and elimination functions. * The RTS options are now completely set in the shell script to run the benchmarks. * RTS options can be set on a per benchmark basis. RTS options work correctly now. * The set of streaming/infinite stream benchmarks is now complete and we can run all such benchmarks coneveniently. * Benchmark "quick"/"speed" options can now be specified on a per benchmark basis. Longer benchmarks can have fewer iterations/quick run time. * Benchmarks are grouped in several groups which can be run on a per group basis. Comparison groups are also defined for convenient comparisons of different modules (e.g. arrays or streamD/K). * The benchmark namespaces are grouped in a consistent manner. Benchmark executables have a consistent naming based on module names.
2020-06-06 22:21:33 +03:00
Prelude.Parallel/o-n-heap/mapping/mapM) echo -n "-M256M" ;;
Prelude.Parallel/o-n-heap/monad-outer-product/*) echo -n "-M256M" ;;
Prelude.Parallel/o-n-space/monad-outer-product/*) echo -n "-K4M -M256M" ;;
Add more benchmarks for applicative/monad Separate the applicative and monad benchmark groups Complete benchmarks for all type class operations
2020-06-21 12:45:55 +03:00
Break Benchmark.Prelude.Serial into smaller modules This module took a lot of time and 4GB RAM to compile. Its now broken into smaller module that can be compiled in less than 512 MB RAM. The memory limits are set based on a 3-pass fusion-plugin.
2020-11-29 10:27:41 +03:00
# XXX These should be moved to o-n-space?
Prelude.Serial/o-n-heap/grouping/classifySessionsOf) echo -n "-K1M -M32M" ;;
Prelude.Serial/o-n-heap/Functor/*) echo -n "-K4M -M32M" ;;
Prelude.Serial/o-n-heap/transformer/*) echo -n "-K8M -M64M" ;;
Optimize Functor/Applicative/Monad etc for serial Some of the benchmarks were order of magnitude off due to missing INLINE for type class operations. Now, all of them are in reasonable limits. Benchmarks affected for serial streams: * Functor, Applicative, Monad, transformers We need to do a similar exercise for other types of streams and for folds/parsers as well.
2020-06-23 10:01:59 +03:00
Prelude.Serial/o-n-space/Functor/*) echo -n "-K4M -M64M" ;;
Prelude.Serial/o-n-space/Applicative/*) echo -n "-K8M -M128M" ;;
Prelude.Serial/o-n-space/Monad/*) echo -n "-K8M -M64M" ;;
Break Benchmark.Prelude.Serial into smaller modules This module took a lot of time and 4GB RAM to compile. Its now broken into smaller module that can be compiled in less than 512 MB RAM. The memory limits are set based on a 3-pass fusion-plugin.
2020-11-29 10:27:41 +03:00
# Use -K4M for o-n-space except for grouping
Refactor Prelude benchmarks * Now benchmark modules correspond to source modules. The Prelude module in source corresponds to several modules one for each stream type. * Benchmarks in the same order/groupings as they appear in source * All benchmarks now have division according to space complexity * Refactoring reduces a lot of code duplication especially the stream generation and elimination functions. * The RTS options are now completely set in the shell script to run the benchmarks. * RTS options can be set on a per benchmark basis. RTS options work correctly now. * The set of streaming/infinite stream benchmarks is now complete and we can run all such benchmarks coneveniently. * Benchmark "quick"/"speed" options can now be specified on a per benchmark basis. Longer benchmarks can have fewer iterations/quick run time. * Benchmarks are grouped in several groups which can be run on a per group basis. Comparison groups are also defined for convenient comparisons of different modules (e.g. arrays or streamD/K). * The benchmark namespaces are grouped in a consistent manner. Benchmark executables have a consistent naming based on module names.
2020-06-06 22:21:33 +03:00
Prelude.Serial/o-n-space/grouping/*) echo -n "" ;;
Prelude.Serial/o-n-space/*) echo -n "-K4M" ;;
Add more benchmarks for applicative/monad Separate the applicative and monad benchmark groups Complete benchmarks for all type class operations
2020-06-21 12:45:55 +03:00
Refactor Prelude benchmarks * Now benchmark modules correspond to source modules. The Prelude module in source corresponds to several modules one for each stream type. * Benchmarks in the same order/groupings as they appear in source * All benchmarks now have division according to space complexity * Refactoring reduces a lot of code duplication especially the stream generation and elimination functions. * The RTS options are now completely set in the shell script to run the benchmarks. * RTS options can be set on a per benchmark basis. RTS options work correctly now. * The set of streaming/infinite stream benchmarks is now complete and we can run all such benchmarks coneveniently. * Benchmark "quick"/"speed" options can now be specified on a per benchmark basis. Longer benchmarks can have fewer iterations/quick run time. * Benchmarks are grouped in several groups which can be run on a per group basis. Comparison groups are also defined for convenient comparisons of different modules (e.g. arrays or streamD/K). * The benchmark namespaces are grouped in a consistent manner. Benchmark executables have a consistent naming based on module names.
2020-06-06 22:21:33 +03:00
Prelude.WSerial/o-n-space/*) echo -n "-K4M" ;;
Prelude.Async/o-n-space/monad-outer-product/*) echo -n "-K4M" ;;
Prelude.Ahead/o-n-space/monad-outer-product/*) echo -n "-K4M" ;;
Prelude.WAsync/o-n-heap/monad-outer-product/toNull3) echo -n "-M64M" ;;
Prelude.WAsync/o-n-space/monad-outer-product/*) echo -n "-K4M" ;;
# XXX need to investigate these, taking too much stack
Combine Yield and YieldB constructors Yield 0: now means a partial result is available Yield n: means a partial result is available, and we need to backtrack by n elements.
2020-06-23 13:31:19 +03:00
Data.Parser.ParserD/o-1-space/some) echo -n "-K8M" ;;
Data.Parser/o-1-space/some) echo -n "-K8M" ;;
Refactor Prelude benchmarks * Now benchmark modules correspond to source modules. The Prelude module in source corresponds to several modules one for each stream type. * Benchmarks in the same order/groupings as they appear in source * All benchmarks now have division according to space complexity * Refactoring reduces a lot of code duplication especially the stream generation and elimination functions. * The RTS options are now completely set in the shell script to run the benchmarks. * RTS options can be set on a per benchmark basis. RTS options work correctly now. * The set of streaming/infinite stream benchmarks is now complete and we can run all such benchmarks coneveniently. * Benchmark "quick"/"speed" options can now be specified on a per benchmark basis. Longer benchmarks can have fewer iterations/quick run time. * Benchmarks are grouped in several groups which can be run on a per group basis. Comparison groups are also defined for convenient comparisons of different modules (e.g. arrays or streamD/K). * The benchmark namespaces are grouped in a consistent manner. Benchmark executables have a consistent naming based on module names.
2020-06-06 22:21:33 +03:00
Data.Parser.ParserD/o-1-space/manyTill) echo -n "-K4M" ;;
Data.Parser/o-1-space/manyTill) echo -n "-K4M" ;;
Fix the backtrack buffer management for fromParserK This commit results in worse performance because now we are double buffering once in ParserD and once in ParserK. This can potentially be fixed but would require bigger changes to unify the backtracking buffer management for ParserD and ParserK.
2020-07-22 18:16:49 +03:00
Data.Parser/o-n-heap/manyAlt) echo -n "-K4M -M128M" ;;
Data.Parser/o-n-heap/someAlt) echo -n "-K4M -M128M" ;;
Data.Parser.ParserK/o-n-heap/manyAlt) echo -n "-K4M -M128M" ;;
Data.Parser.ParserK/o-n-heap/someAlt) echo -n "-K4M -M128M" ;;
Data.Parser.ParserK/o-n-heap/sequence) echo -n "-M64M";;
Data.Parser.ParserK/o-n-heap/sequenceA) echo -n "-M64M";;
Refactor Prelude benchmarks * Now benchmark modules correspond to source modules. The Prelude module in source corresponds to several modules one for each stream type. * Benchmarks in the same order/groupings as they appear in source * All benchmarks now have division according to space complexity * Refactoring reduces a lot of code duplication especially the stream generation and elimination functions. * The RTS options are now completely set in the shell script to run the benchmarks. * RTS options can be set on a per benchmark basis. RTS options work correctly now. * The set of streaming/infinite stream benchmarks is now complete and we can run all such benchmarks coneveniently. * Benchmark "quick"/"speed" options can now be specified on a per benchmark basis. Longer benchmarks can have fewer iterations/quick run time. * Benchmarks are grouped in several groups which can be run on a per group basis. Comparison groups are also defined for convenient comparisons of different modules (e.g. arrays or streamD/K). * The benchmark namespaces are grouped in a consistent manner. Benchmark executables have a consistent naming based on module names.
2020-06-06 22:21:33 +03:00
Data.SmallArray/o-1-sp*) echo -n "-K128K" ;;
*) echo -n "" ;;
esac
}
#------------------------------------------------------------------------------
# Speed options
#------------------------------------------------------------------------------
Add FileSystem.Handle benchmark, use --time-limit 0 In the quick mode we anyway use --include-first-iter, we should not be running many iterations, one iteration is enough, anyway allocations do not change across iterations. We can use this for super quick comparison of benchmarks.
2020-07-09 13:29:17 +03:00
SUPER_QUICK_OPTIONS="--quick --min-duration 0 --time-limit 0 --include-first-iter"
QUICKER_OPTIONS="--min-samples 3 --time-limit 1"
# For certain long benchmarks if the user has not requested super quick
# mode we anyway use a slightly quicker mode.
use_quicker_mode () {
if test -n "$QUICK_MODE"
Optimize Functor/Applicative/Monad etc for serial Some of the benchmarks were order of magnitude off due to missing INLINE for type class operations. Now, all of them are in reasonable limits. Benchmarks affected for serial streams: * Functor, Applicative, Monad, transformers We need to do a similar exercise for other types of streams and for folds/parsers as well.
2020-06-23 10:01:59 +03:00
then
Add FileSystem.Handle benchmark, use --time-limit 0 In the quick mode we anyway use --include-first-iter, we should not be running many iterations, one iteration is enough, anyway allocations do not change across iterations. We can use this for super quick comparison of benchmarks.
2020-07-09 13:29:17 +03:00
if test "$QUICK_MODE" -eq 0
then
echo $QUICKER_OPTIONS
fi
Optimize Functor/Applicative/Monad etc for serial Some of the benchmarks were order of magnitude off due to missing INLINE for type class operations. Now, all of them are in reasonable limits. Benchmarks affected for serial streams: * Functor, Applicative, Monad, transformers We need to do a similar exercise for other types of streams and for folds/parsers as well.
2020-06-23 10:01:59 +03:00
fi
Add FileSystem.Handle benchmark, use --time-limit 0 In the quick mode we anyway use --include-first-iter, we should not be running many iterations, one iteration is enough, anyway allocations do not change across iterations. We can use this for super quick comparison of benchmarks.
2020-07-09 13:29:17 +03:00
}
Refactor Prelude benchmarks * Now benchmark modules correspond to source modules. The Prelude module in source corresponds to several modules one for each stream type. * Benchmarks in the same order/groupings as they appear in source * All benchmarks now have division according to space complexity * Refactoring reduces a lot of code duplication especially the stream generation and elimination functions. * The RTS options are now completely set in the shell script to run the benchmarks. * RTS options can be set on a per benchmark basis. RTS options work correctly now. * The set of streaming/infinite stream benchmarks is now complete and we can run all such benchmarks coneveniently. * Benchmark "quick"/"speed" options can now be specified on a per benchmark basis. Longer benchmarks can have fewer iterations/quick run time. * Benchmarks are grouped in several groups which can be run on a per group basis. Comparison groups are also defined for convenient comparisons of different modules (e.g. arrays or streamD/K). * The benchmark namespaces are grouped in a consistent manner. Benchmark executables have a consistent naming based on module names.
2020-06-06 22:21:33 +03:00
bench_exe_quick_opts () {
case "$1" in
Prelude.Concurrent) echo -n "$SUPER_QUICK_OPTIONS" ;;
Prelude.Rate) echo -n "$SUPER_QUICK_OPTIONS" ;;
Prelude.Adaptive) echo -n "$SUPER_QUICK_OPTIONS" ;;
*) echo -n "" ;;
esac
}
# Use quick options for benchmarks that take too long
bench_quick_opts () {
case "$1" in
Prelude.Parallel/o-n-heap/mapping/mapM)
echo -n "$SUPER_QUICK_OPTIONS" ;;
Prelude.Parallel/o-n-heap/monad-outer-product/*)
echo -n "$SUPER_QUICK_OPTIONS" ;;
Prelude.Parallel/o-n-space/monad-outer-product/*)
echo -n "$SUPER_QUICK_OPTIONS" ;;
Add FileSystem.Handle benchmark, use --time-limit 0 In the quick mode we anyway use --include-first-iter, we should not be running many iterations, one iteration is enough, anyway allocations do not change across iterations. We can use this for super quick comparison of benchmarks.
2020-07-09 13:29:17 +03:00
Prelude.Parallel/o-n-heap/generation/*) use_quicker_mode ;;
Prelude.Parallel/o-n-heap/mapping/*) use_quicker_mode ;;
Prelude.Parallel/o-n-heap/concat-foldable/*) use_quicker_mode ;;
Prelude.Async/o-1-space/monad-outer-product/*) use_quicker_mode ;;
Prelude.Async/o-n-space/monad-outer-product/*) use_quicker_mode ;;
Refactor Prelude benchmarks * Now benchmark modules correspond to source modules. The Prelude module in source corresponds to several modules one for each stream type. * Benchmarks in the same order/groupings as they appear in source * All benchmarks now have division according to space complexity * Refactoring reduces a lot of code duplication especially the stream generation and elimination functions. * The RTS options are now completely set in the shell script to run the benchmarks. * RTS options can be set on a per benchmark basis. RTS options work correctly now. * The set of streaming/infinite stream benchmarks is now complete and we can run all such benchmarks coneveniently. * Benchmark "quick"/"speed" options can now be specified on a per benchmark basis. Longer benchmarks can have fewer iterations/quick run time. * Benchmarks are grouped in several groups which can be run on a per group basis. Comparison groups are also defined for convenient comparisons of different modules (e.g. arrays or streamD/K). * The benchmark namespaces are grouped in a consistent manner. Benchmark executables have a consistent naming based on module names.
2020-06-06 22:21:33 +03:00
Add FileSystem.Handle benchmark, use --time-limit 0 In the quick mode we anyway use --include-first-iter, we should not be running many iterations, one iteration is enough, anyway allocations do not change across iterations. We can use this for super quick comparison of benchmarks.
2020-07-09 13:29:17 +03:00
Prelude.Ahead/o-1-space/monad-outer-product/*) use_quicker_mode ;;
Prelude.Ahead/o-n-space/monad-outer-product/*) use_quicker_mode ;;
Refactor Prelude benchmarks * Now benchmark modules correspond to source modules. The Prelude module in source corresponds to several modules one for each stream type. * Benchmarks in the same order/groupings as they appear in source * All benchmarks now have division according to space complexity * Refactoring reduces a lot of code duplication especially the stream generation and elimination functions. * The RTS options are now completely set in the shell script to run the benchmarks. * RTS options can be set on a per benchmark basis. RTS options work correctly now. * The set of streaming/infinite stream benchmarks is now complete and we can run all such benchmarks coneveniently. * Benchmark "quick"/"speed" options can now be specified on a per benchmark basis. Longer benchmarks can have fewer iterations/quick run time. * Benchmarks are grouped in several groups which can be run on a per group basis. Comparison groups are also defined for convenient comparisons of different modules (e.g. arrays or streamD/K). * The benchmark namespaces are grouped in a consistent manner. Benchmark executables have a consistent naming based on module names.
2020-06-06 22:21:33 +03:00
Add FileSystem.Handle benchmark, use --time-limit 0 In the quick mode we anyway use --include-first-iter, we should not be running many iterations, one iteration is enough, anyway allocations do not change across iterations. We can use this for super quick comparison of benchmarks.
2020-07-09 13:29:17 +03:00
Prelude.WAsync/o-n-heap/monad-outer-product/*) use_quicker_mode ;;
Prelude.WAsync/o-n-space/monad-outer-product/*) use_quicker_mode ;;
Refactor Prelude benchmarks * Now benchmark modules correspond to source modules. The Prelude module in source corresponds to several modules one for each stream type. * Benchmarks in the same order/groupings as they appear in source * All benchmarks now have division according to space complexity * Refactoring reduces a lot of code duplication especially the stream generation and elimination functions. * The RTS options are now completely set in the shell script to run the benchmarks. * RTS options can be set on a per benchmark basis. RTS options work correctly now. * The set of streaming/infinite stream benchmarks is now complete and we can run all such benchmarks coneveniently. * Benchmark "quick"/"speed" options can now be specified on a per benchmark basis. Longer benchmarks can have fewer iterations/quick run time. * Benchmarks are grouped in several groups which can be run on a per group basis. Comparison groups are also defined for convenient comparisons of different modules (e.g. arrays or streamD/K). * The benchmark namespaces are grouped in a consistent manner. Benchmark executables have a consistent naming based on module names.
2020-06-06 22:21:33 +03:00
Add FileSystem.Handle benchmark, use --time-limit 0 In the quick mode we anyway use --include-first-iter, we should not be running many iterations, one iteration is enough, anyway allocations do not change across iterations. We can use this for super quick comparison of benchmarks.
2020-07-09 13:29:17 +03:00
FileSystem.Handle/*) use_quicker_mode ;;
Refactor Prelude benchmarks * Now benchmark modules correspond to source modules. The Prelude module in source corresponds to several modules one for each stream type. * Benchmarks in the same order/groupings as they appear in source * All benchmarks now have division according to space complexity * Refactoring reduces a lot of code duplication especially the stream generation and elimination functions. * The RTS options are now completely set in the shell script to run the benchmarks. * RTS options can be set on a per benchmark basis. RTS options work correctly now. * The set of streaming/infinite stream benchmarks is now complete and we can run all such benchmarks coneveniently. * Benchmark "quick"/"speed" options can now be specified on a per benchmark basis. Longer benchmarks can have fewer iterations/quick run time. * Benchmarks are grouped in several groups which can be run on a per group basis. Comparison groups are also defined for convenient comparisons of different modules (e.g. arrays or streamD/K). * The benchmark namespaces are grouped in a consistent manner. Benchmark executables have a consistent naming based on module names.
2020-06-06 22:21:33 +03:00
*) echo -n "" ;;
esac
}
last=""
for i in "$@"
do
BENCH_NAME="$last"
last="$i"
done
RTS_OPTIONS=\
"+RTS -T \
$(bench_exe_rts_opts $(basename $BENCH_EXEC_PATH)) \
$(bench_rts_opts_default $BENCH_NAME) \
$(bench_rts_opts_specific $BENCH_NAME) \
$RTS_OPTIONS \
-RTS"
QUICK_BENCH_OPTIONS="\
$(bench_exe_quick_opts $(basename $BENCH_EXEC_PATH)) \
$(bench_quick_opts $BENCH_NAME)"
if test -n "$STREAM_SIZE"
then
STREAM_LEN=$(env LC_ALL=en_US.UTF-8 printf "\--stream-size %'.f\n" $STREAM_SIZE)
fi
echo "$BENCH_NAME: \
$STREAM_LEN \
$QUICK_BENCH_OPTIONS \
$RTS_OPTIONS"
$BENCH_EXEC_PATH $RTS_OPTIONS "$@" $QUICK_BENCH_OPTIONS
Reference in New Issue Copy Permalink