git.iko.soy/streamly
on master help sign in
swap
30c5a56343 ahead, 0 behind
277 files changed+18831−12031
since splitsnapshots
unifiedsplit
.circleci/config.yml+73−277 .github/workflows/haskell.yml+84−91 .github/workflows/regression-check.yml+2−2 .hlint.ignore+5 benchmark/NanoBenchmarks.hs+1−1 benchmark/Streamly/Benchmark/Data/Array/CommonImports.hs+1−1 benchmark/Streamly/Benchmark/Data/Array/Stream.hs+3−3 benchmark/Streamly/Benchmark/Data/Fold.hs+28−5 benchmark/Streamly/Benchmark/Data/Fold/Window.hs+40−40 benchmark/Streamly/Benchmark/Data/MutArray.hs+1−1 benchmark/Streamly/Benchmark/Data/Parser.hs+10−10 benchmark/Streamly/Benchmark/Data/ParserK.hs+90−37 benchmark/Streamly/Benchmark/Data/Ring/Unboxed.hs+3−4 benchmark/Streamly/Benchmark/Data/Serialize.hs+591 benchmark/Streamly/Benchmark/Data/Serialize/RecCompatible.hs+24 benchmark/Streamly/Benchmark/Data/Serialize/RecNonCompatible.hs+21 benchmark/Streamly/Benchmark/Data/Serialize/TH.hs+76 benchmark/Streamly/Benchmark/Data/Stream/Common.hs+3−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentCommon.hs+3−3 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentEager.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentInterleaved.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentOrdered.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Eliminate.hs+3−1 benchmark/Streamly/Benchmark/Data/Stream/Exceptions.hs+12−3 benchmark/Streamly/Benchmark/Data/Stream/Expand.hs+9−1 benchmark/Streamly/Benchmark/Data/Stream/Generate.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Lift.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Reduce.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/Split.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/StreamD.hs+4−2 benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs+4−3 benchmark/Streamly/Benchmark/Data/Stream/StreamKAlt.hs+3−3 benchmark/Streamly/Benchmark/Data/Stream/ToStreamK.hs+3−4 benchmark/Streamly/Benchmark/Data/Stream/Transform.hs+2−2 benchmark/Streamly/Benchmark/Data/Unbox.hs−102 benchmark/Streamly/Benchmark/Data/Unfold.hs+11−2 benchmark/Streamly/Benchmark/FileSystem/Handle.hs+1−1 benchmark/Streamly/Benchmark/FileSystem/Handle/Read.hs+8−9 benchmark/Streamly/Benchmark/FileSystem/Handle/ReadWrite.hs+9−10 benchmark/Streamly/Benchmark/Prelude/Adaptive.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Ahead.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Async.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Concurrent.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Merge.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Parallel.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Rate.hs+1−1 benchmark/Streamly/Benchmark/Prelude/WAsync.hs+1−1 benchmark/Streamly/Benchmark/Prelude/WSerial.hs+1−1 benchmark/Streamly/Benchmark/Prelude/ZipAsync.hs+1−1 benchmark/Streamly/Benchmark/Prelude/ZipSerial.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Char.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Parser.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Stream.hs+26−8 benchmark/Streamly/Benchmark/Unicode/Utf8.hs+1−1 benchmark/bench-runner/Main.hs+34−6 benchmark/lib/Streamly/Benchmark/Common.hs+1−1 benchmark/lib/Streamly/Benchmark/Common/Handle.hs+3−1 benchmark/lib/Streamly/Benchmark/Prelude.hs+1−1 benchmark/streamly-benchmarks.cabal+64−40 configure+9−9 configure.ac+1−1 core/configure+9−9 core/configure.ac+1−1 core/docs/ApiChangelogs/0.1.0-0.2.0.txt+1471 core/docs/ApiChangelogs/0.2.0-0.2.2.txt+306 core/docs/Changelog.md+60−12 core/src/DocTestDataArray.hs+8 core/src/DocTestDataFold.hs−1 core/src/DocTestDataParserK.hs+15 core/src/DocTestDataStream.hs+2−1 core/src/DocTestFileSystemHandle.hs+15 core/src/DocTestUnicodeParser.hs+9 core/src/DocTestUnicodeStream.hs+12 core/src/DocTestUnicodeString.hs+5 core/src/Streamly/Console/Stdio.hs+1−27 core/src/Streamly/Data/Array.hs+48−48 core/src/Streamly/Data/Array/Generic.hs+16−13 core/src/Streamly/Data/Fold.hs+150−119 core/src/Streamly/Data/MutArray.hs+18−12 core/src/Streamly/Data/MutArray/Generic.hs+8−3 core/src/Streamly/Data/MutByteArray.hs+106 core/src/Streamly/Data/Parser.hs+130−60 core/src/Streamly/Data/ParserK.hs+105−10 core/src/Streamly/Data/Stream.hs+199−107 core/src/Streamly/Data/StreamK.hs+37−19 core/src/Streamly/Data/Unbox.hs−16 core/src/Streamly/Data/Unfold.hs+3−1 core/src/Streamly/FileSystem/Handle.hs+15−29 core/src/Streamly/Internal/Data/Array.hs+610−163 core/src/Streamly/Internal/Data/Array/Generic.hs+49−11 core/src/Streamly/Internal/Data/Stream/Chunked.hs → core/src/Streamly/Internal/Data/Array/Stream.hs+74−587 core/src/Streamly/Internal/Data/Array/Type.hs+538−108 core/src/Streamly/Internal/Serialize/FromBytes.hs → core/src/Streamly/Internal/Data/Binary/Parser.hs+10−4 core/src/Streamly/Internal/Serialize/ToBytes.hs → core/src/Streamly/Internal/Data/Binary/Stream.hs+12−2 core/src/Streamly/Internal/Data/Builder.hs+11−9 core/src/Streamly/Internal/Data/Fold.hs+9−2574 core/src/Streamly/Internal/Data/Fold/Chunked.hs+20−18 core/src/Streamly/Internal/Data/Fold/Combinators.hs+2452 core/src/Streamly/Internal/Data/Fold/Container.hs+75−32 core/src/Streamly/Internal/Data/Fold/Step.hs+2−2 core/src/Streamly/Internal/Data/Fold/Tee.hs+2 core/src/Streamly/Internal/Data/Fold/Type.hs+344−120 core/src/Streamly/Internal/Data/Fold/Window.hs+59−53 core/src/Streamly/Internal/Data/IORef/Unboxed.hs+23−20 core/src/Streamly/Internal/Data/IsMap.hs+8 core/src/Streamly/Internal/Data/List.hs+5−5 core/src/Streamly/Internal/Data/MutArray.hs+154−33 core/src/Streamly/Internal/Data/MutArray/Generic.hs+172−49 core/src/Streamly/Internal/Data/MutArray/Stream.hs+145 core/src/Streamly/Internal/Data/MutArray/Type.hs+1308−441 core/src/Streamly/Internal/Data/MutByteArray.hs+251 core/src/Streamly/Internal/Data/MutByteArray/Type.hs+321 core/src/Streamly/Internal/Data/Parser.hs+135−143 core/src/Streamly/Internal/Data/Parser/ParserDK.hs−1544 core/src/Streamly/Internal/Data/Parser/ParserD/Tee.hs → core/src/Streamly/Internal/Data/Parser/Tee.hs+1−1 core/src/Streamly/Internal/Data/Parser/ParserD/Type.hs → core/src/Streamly/Internal/Data/Parser/Type.hs+94−71 core/src/Streamly/Internal/Data/ParserK.hs+16 core/src/Streamly/Internal/Data/Parser/ParserK/Type.hs → core/src/Streamly/Internal/Data/ParserK/Type.hs+291−84 core/src/Streamly/Internal/Data/Pipe.hs+5−8 core/src/Streamly/Internal/Data/Producer.hs+6−13 core/src/Streamly/Internal/Data/Producer/Source.hs+1−1 core/src/Streamly/Internal/Data/Producer/Type.hs+1−1 core/src/Streamly/Internal/Data/Refold/Type.hs+1−1 core/src/Streamly/Internal/Data/Ring.hs+589−134 core/src/Streamly/Internal/Data/Ring/Generic.hs+185 core/src/Streamly/Internal/Data/Ring/Unboxed.hs−616 core/src/Streamly/Internal/Data/Serialize.hs−157 core/src/Streamly/Internal/Data/Serialize/TH.hs+530 core/src/Streamly/Internal/Data/Serialize/TH/Bottom.hs+475 core/src/Streamly/Internal/Data/Serialize/TH/Common.hs+70 core/src/Streamly/Internal/Data/Serialize/TH/RecHeader.hs+411 core/src/Streamly/Internal/Data/Serialize/Type.hs+317 core/src/Streamly/Internal/Data/Stream.hs+20−20 core/src/Streamly/Internal/Data/Stream/StreamD/Container.hs → core/src/Streamly/Internal/Data/Stream/Container.hs+65−57 core/src/Streamly/Internal/Data/Stream/StreamD/Eliminate.hs → core/src/Streamly/Internal/Data/Stream/Eliminate.hs+39−33 core/src/Streamly/Internal/Data/Stream/StreamD/Exception.hs → core/src/Streamly/Internal/Data/Stream/Exception.hs+12−9 core/src/Streamly/Internal/Data/Stream/StreamD/Generate.hs → core/src/Streamly/Internal/Data/Stream/Generate.hs+27−26 core/src/Streamly/Internal/Data/Stream/StreamD/Lift.hs → core/src/Streamly/Internal/Data/Stream/Lift.hs+3−3 core/src/Streamly/Internal/Data/Stream/MutChunked.hs−323 core/src/Streamly/Internal/Data/Stream/StreamD/Nesting.hs → core/src/Streamly/Internal/Data/Stream/Nesting.hs+115−228 core/src/Streamly/Internal/Data/Stream/StreamD/Step.hs → core/src/Streamly/Internal/Data/Stream/Step.hs+2−2 core/src/Streamly/Internal/Data/Stream/StreamD/Top.hs−353 core/src/Streamly/Internal/Data/Stream/Top.hs+480 core/src/Streamly/Internal/Data/Stream/StreamD/Transform.hs → core/src/Streamly/Internal/Data/Stream/Transform.hs+66−40 core/src/Streamly/Internal/Data/Stream/StreamD/Transformer.hs → core/src/Streamly/Internal/Data/Stream/Transformer.hs+5−4 core/src/Streamly/Internal/Data/Stream/StreamD/Type.hs → core/src/Streamly/Internal/Data/Stream/Type.hs+169−34 core/src/Streamly/Internal/Data/StreamK.hs+350−134 core/src/Streamly/Internal/Data/Stream/StreamK/Alt.hs → core/src/Streamly/Internal/Data/StreamK/Alt.hs+1−1 core/src/Streamly/Internal/Data/Stream/StreamK/Transformer.hs → core/src/Streamly/Internal/Data/StreamK/Transformer.hs+3−3 core/src/Streamly/Internal/Data/Stream/StreamK/Type.hs → core/src/Streamly/Internal/Data/StreamK/Type.hs+95−41 core/src/Streamly/Internal/Data/Time/Clock.hs+3−3 core/src/Streamly/Internal/Data/Time/Units.hs−3 core/src/Streamly/Internal/Data/Unbox.hs+295−280 core/src/Streamly/Internal/Data/Unbox/TH.hs+85−82 core/src/Streamly/Internal/Data/Unfold.hs+20−58 core/src/Streamly/Internal/Data/Unfold/Enumeration.hs−1 core/src/Streamly/Internal/Data/Unfold/Type.hs+4−8 core/src/Streamly/Internal/FileSystem/File.hs+11−13 core/src/Streamly/Internal/FileSystem/Handle.hs+33−42 core/src/Streamly/Internal/FileSystem/Path.hs+515 core/src/Streamly/Internal/Unicode/Array.hs+6 core/src/Streamly/Internal/Unicode/Parser.hs+12−4 core/src/Streamly/Internal/Unicode/Stream.hs+53−25 core/src/Streamly/Internal/Unicode/String.hs+17−12 core/src/Streamly/Unicode/Parser.hs+11−2 core/src/Streamly/Unicode/Stream.hs+9 core/src/Streamly/Unicode/String.hs+11−1 core/streamly-core.cabal+146−78 docs/Developer/optimization-guidelines.md+12 docs/Developer/paths.rst+25−2 docs/User/HowTo/Compiling.md+5−3 docs/User/Project/ApiChangelogs/0.9.0-0.10.0.txt+424 docs/User/Project/CONTRIBUTORS.md+8 docs/User/Project/Changelog.md+25−10 docs/User/Project/README.md+9−2 docs/User/Project/Upgrading.md → docs/User/Project/Upgrading-0.8-to-0.9.md+87−9 docs/User/Tutorials/quick-overview.md+10−14 docs/streamly-docs.cabal+1−1 hie.yaml+2−2 src/Streamly.hs−624 src/Streamly/Data/Fold/Prelude.hs+32−14 src/Streamly/Data/Fold/Tee.hs+1−1 src/Streamly/Data/Stream/Concurrent.hs−194 src/Streamly/Data/Stream/Exception.hs−24 src/Streamly/Data/Stream/MkType.hs+41−10 src/Streamly/Data/Stream/Prelude.hs+217−15 src/Streamly/Data/Stream/Time.hs−37 src/Streamly/Data/Unicode/Stream.hs−99 src/Streamly/Internal/Data/Array/ArrayMacros.h−40 src/Streamly/Internal/Data/Channel.hs+19 src/Streamly/Internal/Data/Stream/Channel/Dispatcher.hs → src/Streamly/Internal/Data/Channel/Dispatcher.hs+3−3 src/Streamly/Internal/Data/Stream/Channel/Types.hs → src/Streamly/Internal/Data/Channel/Types.hs+12−37 src/Streamly/Internal/Data/Stream/Channel/Worker.hs → src/Streamly/Internal/Data/Channel/Worker.hs+3−3 src/Streamly/Internal/Data/Fold/Async.hs+6−3 src/Streamly/Internal/Data/Fold/Concurrent.hs+1−44 src/Streamly/Internal/Data/Fold/Concurrent/Channel.hs+29−10 src/Streamly/Internal/Data/Fold/Concurrent/Channel/Type.hs+3−3 src/Streamly/Internal/Data/Fold/Prelude.hs+19 src/Streamly/Internal/Data/Fold/SVar.hs+8−7 src/Streamly/Internal/Data/IsMap/HashMap.hs+6 src/Streamly/Internal/Data/SVar.hs+1 src/Streamly/Internal/Data/SmallArray.hs+3−3 src/Streamly/Internal/Data/Stream/Ahead.hs+2−2 src/Streamly/Internal/Data/Stream/Async.hs+2−2 core/src/Streamly/Internal/Data/Stream/Common.hs → src/Streamly/Internal/Data/Stream/Common.hs+3−3 src/Streamly/Internal/Data/Stream/Concurrent.hs+51−42 src/Streamly/Internal/Data/Stream/Concurrent/Channel.hs+17−14 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Append.hs+11−11 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Consumer.hs+2−2 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Dispatcher.hs+3−3 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Interleave.hs+9−9 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Operations.hs+2−2 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Type.hs+4−4 src/Streamly/Internal/Data/Stream/IsStream.hs+6 src/Streamly/Internal/Data/Stream/IsStream/Combinators.hs+1−1 src/Streamly/Internal/Data/Stream/IsStream/Common.hs+6−6 src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs+2−2 src/Streamly/Internal/Data/Stream/IsStream/Exception.hs+1−1 src/Streamly/Internal/Data/Stream/IsStream/Expand.hs+7−3 src/Streamly/Internal/Data/Stream/IsStream/Generate.hs+6−6 src/Streamly/Internal/Data/Stream/IsStream/Reduce.hs+7−7 src/Streamly/Internal/Data/Stream/IsStream/Top.hs+2−3 src/Streamly/Internal/Data/Stream/IsStream/Transform.hs+3−3 src/Streamly/Internal/Data/Stream/IsStream/Type.hs+2−2 src/Streamly/Internal/Data/Stream/Exception/Lifted.hs → src/Streamly/Internal/Data/Stream/Lifted.hs+2−2 src/Streamly/Internal/Data/Stream/MkType.hs+33−34 src/Streamly/Internal/Data/Stream/Parallel.hs+1−1 src/Streamly/Internal/Data/Stream/Prelude.hs+19 src/Streamly/Internal/Data/Stream/SVar.hs+18 src/Streamly/Internal/Data/Stream/SVar/Eliminate.hs+2−2 src/Streamly/Internal/Data/Stream/SVar/Generate.hs+3−1 src/Streamly/Internal/Data/Stream/Serial.hs+7−2 src/Streamly/Internal/Data/Stream/Time.hs+3−3 src/Streamly/Internal/Data/Stream/Zip.hs+41−2 src/Streamly/Internal/Data/Stream/Zip/Concurrent.hs−50 src/Streamly/Internal/Data/Stream/ZipAsync.hs+3−2 src/Streamly/Internal/Data/Unfold/Exception.hs+1−3 src/Streamly/Internal/Data/Unfold/Prelude.hs+17 src/Streamly/Internal/Data/Unfold/SVar.hs+2−1 src/Streamly/Internal/FileSystem/Event.hs+5−2 src/Streamly/Internal/FileSystem/Event/Darwin.hs+5−5 src/Streamly/Internal/FileSystem/Event/Linux.hs+15−9 src/Streamly/Internal/FileSystem/Event/Windows.hs+11−8 src/Streamly/Internal/FileSystem/FD.hs+11−10 src/Streamly/Internal/Network/Inet/TCP.hs+82−47 src/Streamly/Internal/Network/Socket.hs+33−30 src/Streamly/Internal/System/IOVec.hs+3−3 src/Streamly/Internal/Unicode/Char.hs+9−7 src/Streamly/Memory/Array.hs−74 src/Streamly/Network/Inet/TCP.hs+41−30 src/Streamly/Network/Socket.hs+16−30 streamly.cabal+86−64 targets/Targets.hs+9−3 test/Streamly/Test/Data/Array.hs+7−8 test/Streamly/Test/Data/Array/Stream.hs+1−1 test/Streamly/Test/Data/Fold.hs+7−7 test/Streamly/Test/Data/Fold/Window.hs+8−9 test/Streamly/Test/Data/MutArray.hs+1−1 test/Streamly/Test/Data/Parser.hs+18−13 test/Streamly/Test/Data/ParserK.hs+13−9 test/Streamly/Test/Data/Ring/Unboxed.hs+2−2 test/Streamly/Test/Data/Serialize.hs+251−13 test/Streamly/Test/Data/Serialize/CompatV0.hs+46 test/Streamly/Test/Data/Serialize/CompatV1.hs+48 test/Streamly/Test/Data/Serialize/TH.hs+82 test/Streamly/Test/Data/Stream/Concurrent.hs+1−1 test/Streamly/Test/Data/Unbox.hs+125−26 test/Streamly/Test/Data/Unbox/TH.hs+11−10 test/Streamly/Test/FileSystem/Event/Common.hs+1−1 test/Streamly/Test/FileSystem/Handle.hs+2−2 test/Streamly/Test/Prelude/Serial.hs+7−3 test/Streamly/Test/Serialize/Serializable.hs+2−2 test/Streamly/Test/Unicode/Char.hs+1−1 test/Streamly/Test/Unicode/Parser.hs+6−16 test/Streamly/Test/Unicode/Stream.hs+1−1 test/lib/Streamly/Test/Prelude/Common.hs+9−2 test/streamly-tests.cabal+38−11
.circleci/config.yml+73−277
@@ -1,7 +1,7 @@
11version: 2.1
22
33#-----------------------------------------------------------------------------
4−# packcheck-0.4.2
4+# packcheck-0.7.0
55# Packcheck global environment variables
66#-----------------------------------------------------------------------------
77
@@ -11,6 +11,8 @@ env: &env
1111 # Common options
1212 # ------------------------------------------------------------------------
1313 # GHC_OPTIONS: "-Werror"
14+ # For updating see: https://downloads.haskell.org/~ghcup/
15+ GHCUP_VERSION: 0.1.20.0
1416 CABAL_REINIT_CONFIG: "y"
1517 LC_ALL: "C.UTF-8"
1618
@@ -22,7 +24,6 @@ env: &env
2224 # DISABLE_DOCS: "y"
2325 # DISABLE_SDIST_BUILD: "y"
2426 # DISABLE_DIST_CHECKS: "y"
25− ENABLE_INSTALL: "y"
2627
2728 # ------------------------------------------------------------------------
2829 # stack options
@@ -38,15 +39,12 @@ env: &env
3839 # cabal options
3940 # ------------------------------------------------------------------------
4041 CABAL_CHECK_RELAX: "y"
41− CABAL_NO_SANDBOX: "y"
42− CABAL_HACKAGE_MIRROR: "hackage.haskell.org:http://hackage.fpcomplete.com"
4342 CABAL_BUILD_OPTIONS: "-j1 --flag limit-build-mem"
4443
4544 # ------------------------------------------------------------------------
4645 # Where to find the required tools
4746 # ------------------------------------------------------------------------
48− PATH: /opt/ghc/bin:/opt/ghcjs/bin:/sbin:/usr/sbin:/bin:/usr/bin
49− TOOLS_DIR: /opt
47+ PATH: /sbin:/usr/sbin:/bin:/usr/bin
5048
5149 # ------------------------------------------------------------------------
5250 # Location of packcheck.sh (the shell script invoked to perform CI tests ).
@@ -59,15 +57,15 @@ env: &env
5957 # If you have not committed packcheck.sh in your repo at PACKCHECK
6058 # then it is automatically pulled from this URL.
6159 PACKCHECK_GITHUB_URL: "https://raw.githubusercontent.com/composewell/packcheck"
62− PACKCHECK_GITHUB_COMMIT: "a302772c8f3791c477edd1764b82f8d1f5b22fdf"
60+ PACKCHECK_GITHUB_COMMIT: "f6cfd7b7a33b4bf6fd63cce6a3fb3d26f5031403"
6361
6462executors:
6563 amd64-executor:
6664 docker:
67− - image: ubuntu:xenial
65+ - image: ubuntu:latest
6866 x86-executor:
6967 docker:
70− - image: i386/ubuntu:eoan
68+ - image: i386/ubuntu:focal
7169
7270#-----------------------------------------------------------------------------
7371# Common utility stuff, not to be modified usually
@@ -79,38 +77,50 @@ preinstall: &preinstall
7977 # required for https/cache save and restore
8078 apt-get install -y ca-certificates
8179
82− # required to (re)generate the configure script
83− apt-get install -y autoconf
84−
85− # For ghc and cabal-install packages from hvr's ppa
80+ # Alternative way of installing ghc and cabal, directly from
81+ # haskell.org instead of using ghcup. NOTE: this is for Debian
82+ # only and is debian release specific.
8683 # gnupg is required for apt-key to work
87− apt-get install -y gnupg
88− apt-get install -y apt-transport-https
89− apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA3CBA3FFE22B574
90− apt-get install -y software-properties-common
91− add-apt-repository -y ppa:hvr/ghc
92− # echo "deb http://downloads.haskell.org/debian stretch main" >> /etc/apt/sources.list
93− echo "deb-src http://ppa.launchpad.net/hvr/ghc/ubuntu xenial main" >> /etc/apt/sources.list
94− apt-get update
84+ #apt-get install -y gnupg
85+ #apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA3CBA3FFE22B574
86+ #echo "deb http://downloads.haskell.org/debian buster main" >> /etc/apt/sources.list
87+ #apt-get update
9588
96− # required for packcheck
97− apt-get install -y curl
89+ # required for outbound https for stack and for stack setup
90+ apt-get install -y netbase xz-utils make
91+ apt-get install -y zlib1g-dev
9892
99− # required for circleci.
100− apt-get install -y libcurl4-gnutls-dev
101− apt-get install -y git
93+ # For ghcup to install ghc
94+ if test -n "$GHCUP_VERSION"
95+ then
96+ apt-get install -y gcc
97+ apt-get install -y g++
98+ fi
99+
100+ # libgmp required by ghc for linking
101+ apt-get install -y libgmp-dev
102102 apt-get install -y libtinfo-dev
103103
104− # required for outbound https for stack and for stack setup
105− apt-get install -y netbase xz-utils make
104+ # Required by cabal when git URL is specified in project file
105+ apt-get install -y git
106+
107+ # Required for and by packcheck
108+ apt-get install -y curl
106109
107−project-preinstall: &project-preinstall
108− run: |
109110 # Get packcheck if needed
110− CURL=$(which curl)
111− PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
112− if test ! -e "$PACKCHECK"; then $CURL -sL -o "$PACKCHECK" $PACKCHECK_URL; fi;
113− chmod +x $PACKCHECK
111+ if test ! -e "$PACKCHECK"
112+ then
113+ if test -z "$PACKCHECK_GITHUB_COMMIT"
114+ then
115+ die "PACKCHECK_GITHUB_COMMIT is not specified."
116+ fi
117+ PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
118+ curl --fail -sL -o "$PACKCHECK" $PACKCHECK_URL || exit 1
119+ chmod +x $PACKCHECK
120+ elif test ! -x "$PACKCHECK"
121+ then
122+ chmod +x $PACKCHECK
123+ fi
114124
115125restore: &restore
116126 # Needs to happen after installing ca-certificates
@@ -126,271 +136,71 @@ save: &save
126136 save_cache:
127137 key: v1-{{ .Environment.CIRCLE_JOB }}-{{ .Branch }}
128138 paths:
129− - ~/.cabal
130− - ~/.ghc
131139 - ~/.local
140+ - ~/.cabal
132141 - ~/.stack
142+ - ~/.ghcup
133143
134144#-----------------------------------------------------------------------------
135145# Build matrix
136146#-----------------------------------------------------------------------------
137147
138148jobs:
139− cabal-ghc-8_10_4:
149+ cabal-ghc-8_6_5:
140150 <<: *env
141151 executor: amd64-executor
142152 steps:
143− - *preinstall
144153 - checkout
145− - *project-preinstall
146154 - *restore
155+ - *preinstall
147156 - run:
148157 environment:
158+ GHCVER: "8.6.5"
159+ CABAL_VERSION: "3.6.2.0"
149160 CABAL_PROJECT: "cabal.project"
150− DISABLE_TEST: "yes"
151− DISABLE_BENCH: "yes"
152161 DISABLE_SDIST_BUILD: "yes"
162+ CABAL_BUILD_OPTIONS: "--flag debug --flag -opt"
153163 command: |
154− apt-get install -y ghc-8.10.4
155− apt-get install -y cabal-install-3.4
156164 bash -c "$PACKCHECK cabal"
157165 - *save
158− stack-ghc-8_8:
166+ cabal-ghc-9_8_1-docspec:
159167 <<: *env
160168 executor: amd64-executor
161169 steps:
162− - *preinstall
163170 - checkout
164− - *project-preinstall
165171 - *restore
166− - run:
167− name: GHC 8.8 + stack lts-16.12
168− environment:
169− BUILD: "stack"
170− RESOLVER: "lts-16.12"
171− SDIST_OPTIONS: "--ignore-check"
172− STACK_BUILD_OPTIONS: "--flag streamly-benchmarks:-opt"
173− command: |
174− apt-get build-dep -y ghc-8.8.4
175− apt-get install -y cabal-install-3.2
176− bash -c "$PACKCHECK $BUILD"
177− - *save
178− cabal-ghc-8_6_5:
179− <<: *env
180− executor: amd64-executor
181− steps:
182172 - *preinstall
183− - checkout
184− - *project-preinstall
185− - *restore
186− - run: |
187− apt-get install -y ghc-8.6.3
188− apt-get install -y cabal-install-2.4
189− bash -c "$PACKCHECK cabal-v2"
190− - *save
191− cabal-ghc-8_4_4:
192− <<: *env
193− executor: x86-executor
194− steps:
195− - *preinstall
196− - checkout
197− - *project-preinstall
198− - *restore
199− - run:
200− name: GHC 8.4.4 + x86 + debug
201− environment:
202− BUILD: "cabal-v2"
203− GHCVER: "8.4.4"
204− CABAL_BUILD_OPTIONS: "--flag debug --flag limit-build-mem"
205− command: |
206− apt-get install -y ghc-8.4.4
207− apt-get install -y cabal-install-3.2
208− bash -c "$PACKCHECK $BUILD"
209− - *save
210− cabal-ghc-8_2_2:
211− <<: *env
212− executor: amd64-executor
213− steps:
214− - *preinstall
215− - checkout
216− - *project-preinstall
217− - *restore
218173 - run:
219174 environment:
220− BUILD: "cabal-v2"
221− DISABLE_TEST: "yes"
222− DISABLE_BENCH: "yes"
223− DISABLE_DOCS: "yes"
224− DISABLE_SDIST_BUILD: "yes"
225− DISABLE_DIST_CHECKS: "yes"
175+ GHCVER: "9.8.1"
176+ CABAL_VERSION: "3.10.1.0"
177+ CABAL_PROJECT: "cabal.project.doctest"
178+ DISABLE_SDIST_BUILD: "y"
179+ DISABLE_TEST: "y"
180+ DISABLE_BENCH: "y"
181+ DISABLE_DOCS: "y"
182+ ENABLE_DOCSPEC: "y"
183+ DOCSPEC_URL: https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz
184+ DOCSPEC_OPTIONS: "--timeout 60 --check-properties --property-variables xs"
226185 command: |
227− apt-get install -y ghc-8.2.2
228− apt-get install -y cabal-install-3.2
229− bash -c "$PACKCHECK $BUILD"
230− - *save
231− cabal-ghc-8_0_2:
232− <<: *env
233− executor: amd64-executor
234− steps:
235− - *preinstall
236− - checkout
237− - *project-preinstall
238− - *restore
239− - run:
240− environment:
241− BUILD: "cabal-v2"
242− DISABLE_TEST: "yes"
243− DISABLE_BENCH: "yes"
244− DISABLE_DOCS: "yes"
245− DISABLE_SDIST_BUILD: "yes"
246− DISABLE_DIST_CHECKS: "yes"
247− command: |
248− apt-get install -y ghc-8.0.2
249− apt-get install -y cabal-install-3.2
250− bash -c "$PACKCHECK $BUILD"
251− - *save
252− cabal-ghc-7_10_3:
253− <<: *env
254− executor: amd64-executor
255− steps:
256− - *preinstall
257− - checkout
258− - *project-preinstall
259− - *restore
260− - run: |
261− apt-get install -y ghc-7.10.3
262− apt-get install -y cabal-install-2.4
263− bash -c "$PACKCHECK cabal-v2"
186+ bash -c "$PACKCHECK cabal"
264187 - *save
265− stack-ghc-8_4:
188+ hlint-trailing-spaces:
266189 <<: *env
267190 executor: amd64-executor
268191 steps:
269− - *preinstall
270192 - checkout
271− - *project-preinstall
272193 - *restore
273− - run: |
274− bash -c "$PACKCHECK stack RESOLVER=lts-12"
275− - *save
276− stack-ghc-8_2:
277− <<: *env
278− executor: amd64-executor
279− steps:
280194 - *preinstall
281− - checkout
282− - *project-preinstall
283− - *restore
284− - run: |
285− bash -c "$PACKCHECK stack RESOLVER=lts-11"
286− - *save
287− cabal-ghcjs-8_4:
288− <<: *env
289− executor: amd64-executor
290− steps:
291− - *preinstall
292− - checkout
293− - *project-preinstall
294− - *restore
295195 - run:
296− name: GHCJS 8.4+no-test+no-docs
196+ name: hlint and trailing spaces
297197 environment:
298− BUILD: cabal-v2
299− ENABLE_GHCJS: "yes"
300− DISABLE_TEST: "yes"
301− ENABLE_INSTALL: ""
302− DISABLE_DOCS: "yes"
303198 DISABLE_SDIST_BUILD: "yes"
304− command: |
305− add-apt-repository -y ppa:hvr/ghcjs
306− add-apt-repository -y 'deb https://deb.nodesource.com/node_11.x xenial main'
307− curl -sSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -
308− apt-get update
309− apt-get install -y cabal-install-3.2
310− apt-get install -y nodejs
311− apt-get install -y ghcjs-8.4
312− cabal user-config update -a "jobs: 1"
313− bash -c "$PACKCHECK $BUILD"
314− - *save
315− coveralls-ghc-8_2_2:
316− <<: *env
317− executor: amd64-executor
318− steps:
319− - *preinstall
320− - checkout
321− - *project-preinstall
322− - *restore
323− - run:
324− environment:
325− BUILD: cabal
326− COVERALLS_OPTIONS: "--repo-token=\"$REPO_TOKEN\" --coverage-mode=StrictlyFullLines --exclude-dir=test test"
327− GHC_OPTIONS: "-DCOVERAGE_BUILD"
328− command: |
329− apt-get install -y ghc-8.2.2
330− apt-get install -y cabal-install-2.4
331− bash -c "$PACKCHECK cabal-v1"
332− - *save
333− coveralls-ghc-8_8_3:
334− <<: *env
335− executor: amd64-executor
336− steps:
337− - *preinstall
338− - checkout
339− - *project-preinstall
340− - *restore
341− - run:
342− environment:
343− BUILD: cabal-v2
344− COVERALLS_OPTIONS: "--repo-token=\"$REPO_TOKEN\" --coverage-mode=StrictlyFullLines --exclude-dir=test"
345− CABAL_PROJECT: "cabal.project.coverage"
346− CABAL_BUILD_TARGETS: ""
347− command: |
348− apt-get install -y ghc-8.8.3
349− apt-get install -y cabal-install-3.0
350− bash -c "$PACKCHECK cabal-v2"
351− no_output_timeout: 25m
352− - *save
353− coveralls-ghc-8_10_2:
354− <<: *env
355− executor: amd64-executor
356− steps:
357− - *preinstall
358− - checkout
359− - *project-preinstall
360− - *restore
361− - run:
362− name: Build and test with inspection, coverage, Werror
363− command: |
364− apt-get install -y ghc-8.10.2
365− apt-get install -y cabal-install-3.2
366− # Run tests with coverage
367− cabal update
368− cabal user-config update -a "jobs: 1"
369− bin/test.sh --coverage --raw
370− # Upload results to coveralls.io
371− PATH=$HOME/.cabal/bin:$PATH
372− export PATH
373− which hpc-coveralls 2>/dev/null || cabal install --project-file cabal.project.hpc-coveralls hpc-coveralls
374− hpc-coveralls --repo-token="$REPO_TOKEN" --coverage-mode=StrictlyFullLines
375− no_output_timeout: 25m
376− - *save
377− hlint-src:
378− <<: *env
379− executor: amd64-executor
380− steps:
381− - *preinstall
382− - checkout
383− - *project-preinstall
384− - *restore
385− - run:
386− name: Hlint src
387− environment:
388− BUILD: hlint
199+ HLINT_VERSION: 3.6.1
389200 HLINT_OPTIONS: lint
390− DISABLE_SDIST_BUILD: "yes"
391201 HLINT_TARGETS: core/src src test benchmark
392202 command: |
393− bash -c "$PACKCHECK $BUILD" || exit 1
203+ bash -c "$PACKCHECK hlint" || exit 1
394204 echo "Checking trailing spaces..."
395205 count=$(find . -name "*.hs" -exec grep -H '\ $' {} \; | tee /dev/tty | wc -l)
396206 exit $count
@@ -400,23 +210,9 @@ workflows:
400210 version: 2
401211 build:
402212 jobs:
403− - cabal-ghc-8_10_4:
404− name: GHC 8.10.4 + sdist + no-test + no-bench
405− #- cabal-ghc-8.6.5
406− #- cabal-ghc-8_4_4:
407− # name: GHC 8.4.4 + x86 + debug
408− #- cabal-ghc-8_2_2:
409− # name: GHC 8.2.2 + no-test + no-bench + no-docs
410− #- cabal-ghc-8_0_2:
411− # name: GHC 8.0.2 + no-test + no-bench + no-docs
412− #- cabal-ghc-7.10.3
413− # - cabal-ghcjs-8_4:
414− # name: GHCJS 8.4 + no-test + no-docs
415− #- stack-ghc-8.4
416− #- stack-ghc-8.2
417− #- coveralls-ghc-8.2.2
418− #- coveralls-ghc-8.8.3
419− #- coveralls-ghc-8_10_2:
420− # name: GHC 8.10.2 + inspection + coverage + Werror
421− - hlint-src:
422− name: Hlint src + Trailing Spaces
213+ #- cabal-ghc-8_6_5:
214+ # name: 8.6.5-debug-unoptimized
215+ - cabal-ghc-9_8_1-docspec:
216+ name: ghc-9.8.1-docspec
217+ - hlint-trailing-spaces:
218+ name: hlint and trailing spaces
.github/workflows/haskell.yml+84−91
@@ -13,9 +13,15 @@ jobs:
1313 # packcheck environment variables
1414 LC_ALL: C.UTF-8
1515 BUILD: ${{ matrix.build }}
16+ # For updating see: https://downloads.haskell.org/~ghcup/
17+ GHCUP_VERSION: 0.1.20.0
18+ DOCSPEC_URL: https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz
19+ DOCSPEC_OPTIONS: "--timeout 60 --check-properties --property-variables xs"
20+ GHCUP_GHC_OPTIONS: ${{ matrix.ghcup_ghc_options }}
1621 GHCVER: ${{ matrix.ghc_version }}
1722 CABALVER: ${{ matrix.cabal_version }}
1823 DISABLE_DOCS: ${{ matrix.disable_docs }}
24+ ENABLE_DOCSPEC: ${{ matrix.enable_docspec }}
1925 DISABLE_TEST: ${{ matrix.disable_test }}
2026 DISABLE_BENCH: ${{ matrix.disable_bench }}
2127 DISABLE_DIST_CHECKS: ${{ matrix.disable_dist_checks }}
@@ -39,14 +45,18 @@ jobs:
3945 STACK_BUILD_OPTIONS: ${{ matrix.stack_build_options }}
4046
4147 # packcheck location and revision
42− PACKCHECK_LOCAL_PATH: "./packcheck.sh"
48+ PACKCHECK: "./packcheck.sh"
4349 PACKCHECK_GITHUB_URL: "https://raw.githubusercontent.com/composewell/packcheck"
44− PACKCHECK_GITHUB_COMMIT: "a302772c8f3791c477edd1764b82f8d1f5b22fdf"
50+ PACKCHECK_GITHUB_COMMIT: "v0.7.0"
4551
4652 # Pull token from "secrets" setting of the github repo
4753 COVERALLS_TOKEN: ${{ secrets.COVERALLS_TOKEN }}
4854 COVERAGE: ${{ matrix.coverage }}
49− DOCTEST: ${{ matrix.doctest }}
55+
56+ # hlint
57+ HLINT_VERSION: 3.6.1
58+ HLINT_OPTIONS: "lint"
59+ HLINT_TARGETS: "core/src src test benchmark"
5060
5161 # Subdir
5262 SUBDIR: ${{ matrix.subdir }}
@@ -58,8 +68,8 @@ jobs:
5868 matrix:
5969 # The order is important to optimize fail-fast.
6070 name:
61− - 9.4.4-Werror
62− - 9.2.7-docspec
71+ - 9.8.1-Werror
72+ # - 9.8.1-docspec
6373 # - 8.10.7-coverage
6474
6575 # Note: if cabal.project is not specified benchmarks and tests won't
@@ -68,51 +78,74 @@ jobs:
6878 include:
6979 - name: head
7080 ghc_version: head
81+ # The URL may change, to find a working URL go to https://gitlab.haskell.org/ghc/ghc/-/jobs/
82+ # Find a debian10/11/12 job, click on a passed/failed status, at the
83+ # end of the output you will find the tar.xz name, put that tar
84+ # name after "raw/", and put the job name after "job=".
85+ # Also see https://github.com/mpickering/ghc-artefact-nix/blob/master/gitlab-artifact.nix
86+ #
87+ # May also use ghcup for installing ghc head version, use the
88+ # version "LatestNightly", and the following config:
89+ # ghcup config add-release-channel https://ghc.gitlab.haskell.org/ghcup-metadata/ghcup-nightlies-0.0.7.yaml
90+ ghcup_ghc_options: "-u https://gitlab.haskell.org/ghc/ghc/-/jobs/artifacts/master/raw/ghc-x86_64-linux-deb10-int_native-validate.tar.xz?job=x86_64-linux-deb10-int_native-validate"
7191 runner: ubuntu-latest
7292 build: cabal
73− cabal_version: 3.8.1.0
93+ cabal_version: 3.10.2.0
7494 cabal_project: cabal.project.ghc-head
7595 disable_sdist_build: "y"
7696 ignore_error: true
77− - name: 9.6.1
78− ghc_version: 9.6.1
97+ # Note: use linux for warning build for convenient dev testing
98+ - name: 9.8.1-Werror
99+ ghc_version: 9.8.1
79100 runner: ubuntu-latest
80101 build: cabal
81102 cabal_version: 3.10.1.0
82103 disable_sdist_build: "y"
83− cabal_project: cabal.project
84− ignore_error: true
85− - name: 9.4.4-Werror
86− ghc_version: 9.4.4
87− runner: ubuntu-latest
88− build: cabal
89− cabal_version: 3.8.1.0
90− disable_sdist_build: "y"
91104 cabal_project: cabal.project.Werror
92105 ignore_error: false
93− - name: 9.4.4-fusion-inspection
94− ghc_version: 9.4.4
106+ #- name: 9.8.1-docspec
107+ # ghc_version: 9.8.1
108+ # runner: ubuntu-latest
109+ # build: cabal
110+ # cabal_version: 3.10.1.0
111+ # cabal_project: cabal.project.doctest
112+ # disable_test: "y"
113+ # disable_bench: "y"
114+ # disable_docs: "y"
115+ # enable_docspec: "y"
116+ # disable_sdist_build: "y"
117+ # ignore_error: false
118+ - name: 9.8.1-fusion-inspection
119+ ghc_version: 9.8.1
95120 runner: ubuntu-latest
96121 build: cabal
97− cabal_version: 3.8.1.0
122+ cabal_version: 3.10.1.0
98123 disable_sdist_build: "y"
99124 cabal_project: cabal.project
100125 cabal_build_options: "--flag fusion-plugin --flag inspection"
101126 ignore_error: false
102− - name: 9.2.7
103− ghc_version: 9.2.7
104− runner: ubuntu-latest
127+ - name: 9.6.3-macos
128+ ghc_version: 9.6.3
129+ runner: macos-latest
105130 build: cabal
131+ cabal_version: 3.10.1.0
132+ disable_sdist_build: "y"
106133 cabal_project: cabal.project
107− cabal_version: 3.6.2.0
134+ ignore_error: false
135+ - name: 9.4.7
136+ ghc_version: 9.4.7
137+ runner: ubuntu-latest
138+ build: cabal
139+ cabal_version: 3.8.1.0
108140 disable_sdist_build: "y"
141+ cabal_project: cabal.project
109142 ignore_error: false
110− - name: 9.2.7-docspec
111− ghc_version: 9.2.7
143+ - name: 9.2.8
144+ ghc_version: 9.2.8
112145 runner: ubuntu-latest
113− doctest: "y"
114− cabal_version: 3.6.2.0
146+ build: cabal
115147 cabal_project: cabal.project
148+ cabal_version: 3.6.2.0
116149 disable_sdist_build: "y"
117150 ignore_error: false
118151 - name: 9.0.2-streamly-sdist
@@ -138,14 +171,6 @@ jobs:
138171 stack_build_options: "--flag streamly-benchmarks:-opt"
139172 cabal_version: 3.6.2.0
140173 ignore_error: false
141− - name: 8.10.7-macos
142− ghc_version: 8.10.7
143− runner: macos-latest
144− build: cabal
145− cabal_version: 3.6.2.0
146− cabal_project: cabal.project
147− disable_sdist_build: "y"
148− ignore_error: false
149174 # - name: 8.10.7-coverage
150175 # ghc_version: 8.10.7
151176 # runner: ubuntu-latest
@@ -160,15 +185,6 @@ jobs:
160185 cabal_project: cabal.project
161186 disable_sdist_build: "y"
162187 ignore_error: false
163− - name: 8.6.5-no-fusion-unoptimized
164− ghc_version: 8.6.5
165− runner: ubuntu-latest
166− build: cabal
167− cabal_version: 3.6.2.0
168− cabal_project: cabal.project
169− cabal_build_options: "--flag no-fusion --flag -opt"
170− disable_sdist_build: "y"
171− ignore_error: false
172188 - name: 8.6.5-debug-unoptimized
173189 ghc_version: 8.6.5
174190 runner: ubuntu-latest
@@ -178,66 +194,44 @@ jobs:
178194 cabal_build_options: "--flag debug --flag -opt"
179195 disable_sdist_build: "y"
180196 ignore_error: false
197+ - name: hlint
198+ build: hlint
199+ runner: ubuntu-latest
200+ ignore_error: true
181201
182202 steps:
183203 - uses: actions/checkout@v2
184204
185205 - uses: actions/cache@v1
186− name: Cache ~/.cabal
206+ name: Cache common directories
187207 with:
188− path: ~/.cabal
208+ path: |
209+ ~/.local
210+ ~/.cabal
211+ ~/.stack
212+ ~/.ghcup
189213 # Bump the key version to clear the cache
190214 key: ${{ runner.os }}-${{ matrix.ghc_version }}-cabal-v2
191215
192216 - name: Download packcheck
193217 run: |
194− # Get packcheck if needed
195− CURL=$(which curl)
196− PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
197− if test ! -e "$PACKCHECK_LOCAL_PATH"; then $CURL -sL -o "$PACKCHECK_LOCAL_PATH" $PACKCHECK_URL; fi;
198− chmod +x $PACKCHECK_LOCAL_PATH
199−
200− os=$(uname -s -m)
201− case "$os" in
202− "Linux x86_64") GHCUP_ARCH="x86_64-linux" ;;
203− "Darwin x86_64") GHCUP_ARCH="x86_64-apple-darwin" ;;
204− *) echo "Unknown OS/Arch: $os"; exit 1;;
205− esac
206−
207− # Check available versions here: https://downloads.haskell.org/~ghcup/
208− GHCUP_VER=0.1.18.0
209− $CURL -sL -o ./ghcup https://downloads.haskell.org/~ghcup/$GHCUP_VER/${GHCUP_ARCH}-ghcup-$GHCUP_VER
210− chmod +x ./ghcup
211− if test "$GHCVER" = "head"
218+ if test ! -e "$PACKCHECK"
212219 then
213− # The URL may change, to find a working URL go to https://gitlab.haskell.org/ghc/ghc/-/jobs/
214− # Find a debian10 job, click on a passed/failed job, at the
215− # end of the output you will find the tar.xz name, put that after
216− # "raw/", and put the job name after "job=".
217− # Also see https://github.com/mpickering/ghc-artefact-nix/blob/master/gitlab-artifact.nix
218− ./ghcup install ghc -u https://gitlab.haskell.org/ghc/ghc/-/jobs/artifacts/master/raw/ghc-x86_64-linux-deb10-int_native-validate.tar.xz?job=x86_64-linux-deb10-int_native-validate head
219− else
220− ./ghcup install ghc $GHCVER
220+ if test -z "$PACKCHECK_GITHUB_COMMIT"
221+ then
222+ die "PACKCHECK_GITHUB_COMMIT is not specified."
223+ fi
224+ PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
225+ curl --fail -sL -o "$PACKCHECK" $PACKCHECK_URL || exit 1
226+ chmod +x $PACKCHECK
227+ elif test ! -x "$PACKCHECK"
228+ then
229+ chmod +x $PACKCHECK
221230 fi
222− ./ghcup install cabal $CABALVER
223− ./ghcup set ghc $GHCVER
224− ./ghcup set cabal $CABALVER
225231
226232 - name: Run tests
227233 run: |
228− if test -n "$DOCTEST"
229− then
230− mkdir -p $HOME/.cabal/bin
231− curl -sL https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz > cabal-docspec.xz
232− echo '0829bd034fba901cbcfe491d98ed8b28fd54f9cb5c91fa8e1ac62dc4413c9562 cabal-docspec.xz' | sha256sum -c -
233− xz -d < cabal-docspec.xz > $HOME/.cabal/bin/cabal-docspec
234− rm -f cabal-docspec.xz
235− chmod a+x $HOME/.cabal/bin/cabal-docspec
236− cabal update
237− cabal build --project-file cabal.project.doctest all
238− $HOME/.cabal/bin/cabal-docspec --version
239− $HOME/.cabal/bin/cabal-docspec --timeout 60 --check-properties --property-variables xs
240− elif test -n "$COVERAGE"
234+ if test -n "$COVERAGE"
241235 then
242236 # Run tests with coverage
243237 cabal update
@@ -250,16 +244,15 @@ jobs:
250244 # XXX Uncomment this and fix it properly later
251245 # hpc-coveralls --repo-token="$COVERALLS_TOKEN" --coverage-mode=StrictlyFullLines
252246 else
253− export TOOLS_DIR=/opt
254247 # /usr/local/opt/curl/bin for macOS
255248 export PATH=$HOME/.local/bin:$HOME/.ghcup/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/opt/curl/bin
256249 if test -n "$SUBDIR"
257250 then
258− mv "$PACKCHECK_LOCAL_PATH" "$SUBDIR/packcheck.sh"
251+ mv "$PACKCHECK" "$SUBDIR/packcheck.sh"
259252 # This is required as dist-newstyle will be created on the top level
260253 # directory as it is considered the working dir.
261254 rm cabal.project
262255 cd "$SUBDIR"
263256 fi
264− bash -c "$PACKCHECK_LOCAL_PATH $BUILD"
257+ bash -c "$PACKCHECK $BUILD"
265258 fi
.github/workflows/regression-check.yml+2−2
@@ -26,8 +26,8 @@ jobs:
2626 Data.Fold.Window
2727 Data.Parser
2828 Data.ParserK
29− Data.Stream.StreamD
30− Data.Stream.StreamK:6
29+ Data.StreamD
30+ Data.StreamK:6
3131 Data.Unfold
3232 Data.Stream
3333 Data.Stream.Concurrent
.hlint.ignore+5
@@ -2,6 +2,9 @@ core/src/Streamly/Internal/Data/Stream/StreamK/Type.hs
22core/src/Streamly/Internal/Data/Pipe/Type.hs
33core/src/Streamly/Internal/Unicode/Stream.hs
44src/Streamly/Internal/Data/SmallArray/Type.hs
5+src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs
6+src/Streamly/Internal/Data/Stream/Serial.hs
7+src/Streamly/Internal/Data/Stream/Zip.hs
58test/Streamly/Test/Data/Array.hs
69test/Streamly/Test/Data/Parser.hs
710test/Streamly/Test/Data/ParserK.hs
@@ -10,6 +13,7 @@ test/Streamly/Test/Data/Unfold.hs
1013test/Streamly/Test/FileSystem/Event.hs
1114test/Streamly/Test/Prelude/Concurrent.hs
1215test/Streamly/Test/Data/Stream/Concurrent.hs
16+test/lib/Streamly/Test/Prelude/Common.hs
1317test/Streamly/Test/Prelude/Fold.hs
1418test/Streamly/Test/Prelude/Rate.hs
1519test/Streamly/Test/Prelude/Serial.hs
@@ -27,6 +31,7 @@ benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs
2731benchmark/Streamly/Benchmark/Data/Unfold.hs
2832benchmark/Streamly/Benchmark/FileSystem/Handle.hs
2933benchmark/Streamly/Benchmark/Prelude/Async.hs
34+benchmark/Streamly/Benchmark/Prelude/Concurrent.hs
3035benchmark/Streamly/Benchmark/Prelude/Merge.hs
3136benchmark/Streamly/Benchmark/Prelude/Parallel.hs
3237benchmark/Streamly/Benchmark/Prelude/Rate.hs
benchmark/NanoBenchmarks.hs+1−1
@@ -18,7 +18,7 @@ import qualified Streamly.Data.Fold as FL
1818import qualified Streamly.Internal.Data.Stream as S
1919import qualified Streamly.Internal.Data.Stream.IsStream as Internal
2020
21−import Gauge
21+import Test.Tasty.Bench
2222import System.Random
2323
2424maxValue :: Int
benchmark/Streamly/Benchmark/Data/Array/CommonImports.hs+1−1
@@ -9,7 +9,7 @@ import qualified Streamly.Data.Fold as Fold
99import qualified Streamly.Data.Stream as S
1010import qualified Streamly.Internal.Data.Stream as Stream
1111
12−import Gauge
12+import Test.Tasty.Bench
1313import Streamly.Benchmark.Common hiding (benchPureSrc)
1414import qualified Stream.Common as P
1515
benchmark/Streamly/Benchmark/Data/Array/Stream.hs+3−3
@@ -39,7 +39,7 @@ import Prelude hiding ()
3939
4040import qualified Streamly.Data.Stream as Stream
4141import qualified Streamly.Internal.Data.Array as Array
42−import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
42+import qualified Streamly.Internal.Data.Array.Stream as ArrayStream
4343import qualified Streamly.Internal.Data.Fold as Fold
4444import qualified Streamly.Internal.Data.Parser as Parser
4545import qualified Streamly.Internal.Data.Stream as Stream
@@ -47,13 +47,13 @@ import qualified Streamly.Internal.Data.StreamK as StreamK
4747import qualified Streamly.Internal.FileSystem.Handle as Handle
4848import qualified Streamly.Internal.Unicode.Stream as Unicode
4949
50−import Gauge hiding (env)
50+import Test.Tasty.Bench hiding (env)
5151import Streamly.Benchmark.Common
5252import Streamly.Benchmark.Common.Handle
5353import Control.Monad.IO.Class (MonadIO)
5454
5555#ifdef INSPECTION
56−import Streamly.Internal.Data.Unbox (Unbox)
56+import Streamly.Internal.Data.MutByteArray (Unbox)
5757import Streamly.Internal.Data.Stream (Step(..))
5858import Test.Inspection
5959#endif
benchmark/Streamly/Benchmark/Data/Fold.hs+28−5
@@ -12,6 +12,11 @@
1212{-# OPTIONS_GHC -fno-warn-warnings-deprecations #-}
1313{-# OPTIONS_GHC -Wno-orphans #-}
1414
15+#undef FUSION_CHECK
16+#ifdef FUSION_CHECK
17+{-# OPTIONS_GHC -ddump-simpl -ddump-to-file -dsuppress-all #-}
18+#endif
19+
1520module Main (main) where
1621
1722import Control.DeepSeq (NFData(..))
@@ -32,12 +37,11 @@ import Streamly.Internal.Data.MutArray (MutArray)
3237
3338import qualified Streamly.Internal.Data.Fold as FL
3439import qualified Streamly.Data.Fold.Prelude as Fold
35−import qualified Streamly.Internal.Data.Fold.Container as FL
3640import qualified Streamly.Internal.Data.Unfold as Unfold
3741import qualified Streamly.Internal.Data.Pipe as Pipe
3842import qualified Streamly.Internal.Data.Stream as Stream
3943
40−import Gauge
44+import Test.Tasty.Bench
4145import Streamly.Benchmark.Common
4246import Prelude hiding (all, any, take, unzip, sequence_, filter)
4347
@@ -265,13 +269,13 @@ unzip = Stream.fold $ FL.lmap (\a -> (a, a)) (FL.unzip FL.sum FL.length)
265269{-# INLINE unzipWithFstM #-}
266270unzipWithFstM :: Monad m => Stream m Int -> m (Int, Int)
267271unzipWithFstM = do
268− let f = \a -> return (a + 1, a)
272+ let f a = return (a + 1, a)
269273 Stream.fold (FL.unzipWithFstM f FL.sum FL.length)
270274
271275{-# INLINE unzipWithMinM #-}
272276unzipWithMinM :: Monad m => Stream m Int -> m (Int, Int)
273277unzipWithMinM = do
274− let f = \a -> return (a + 1, a)
278+ let f a = return (a + 1, a)
275279 Stream.fold (FL.unzipWithMinM f FL.sum FL.length)
276280
277281-------------------------------------------------------------------------------
@@ -503,7 +507,9 @@ o_n_heap_serial value =
503507-------------------------------------------------------------------------------
504508
505509main :: IO ()
506−main = runWithCLIOpts defaultStreamSize allBenchmarks
510+main = do
511+#ifndef FUSION_CHECK
512+ runWithCLIOpts defaultStreamSize allBenchmarks
507513
508514 where
509515
@@ -516,3 +522,20 @@ main = runWithCLIOpts defaultStreamSize allBenchmarks
516522 , bgroup (o_n_space_prefix moduleName) (o_n_space_serial value)
517523 , bgroup (o_n_heap_prefix moduleName) (o_n_heap_serial value)
518524 ]
525+#else
526+ -- Enable FUSION_CHECK macro at the beginning of the file
527+ -- Enable one benchmark below, and run the benchmark
528+ -- Check the .dump-simpl output
529+ let value = 100000
530+ let input = source value 1
531+ let getKey buckets = (`mod` buckets)
532+ let getFold k =
533+ return $ case k of
534+ 0 -> FL.sum
535+ 1 -> FL.length
536+ _ -> FL.length
537+
538+ -- demuxToMap (getKey 64) (getFold . getKey 64) input
539+ toIntMapIO (getKey 64) input
540+ return ()
541+#endif
benchmark/Streamly/Benchmark/Data/Fold/Window.hs+40−40
@@ -8,12 +8,12 @@ import Streamly.Internal.Data.Stream (Stream)
88import System.Random (randomRIO)
99
1010import qualified Streamly.Data.Fold as Fold
11−import qualified Streamly.Internal.Data.Ring.Unboxed as Ring
12−import qualified Streamly.Internal.Data.Fold.Window as Window
11+import qualified Streamly.Internal.Data.Fold as Window
12+import qualified Streamly.Internal.Data.Ring as Ring
1313import qualified Streamly.Internal.Data.Stream as Stream
1414
1515import Streamly.Benchmark.Common
16−import Gauge
16+import Test.Tasty.Bench
1717
1818{-# INLINE source #-}
1919source :: (Monad m, Num a, Stream.Enumerable a) =>
@@ -70,55 +70,55 @@ o_1_space_folds :: Int -> [Benchmark]
7070o_1_space_folds numElements =
7171 [ bgroup "fold"
7272 [ benchWithFold numElements "minimum (window size 100)"
73− (Window.minimum 100)
73+ (Window.windowMinimum 100)
7474 , benchWithFold numElements "minimum (window size 1000)"
75− (Window.minimum 1000)
75+ (Window.windowMinimum 1000)
7676 , benchWith sourceDescendingInt numElements
7777 "minimum descending (window size 1000)"
78− (Window.minimum 1000)
78+ (Window.windowMinimum 1000)
7979
8080 , benchWithFold numElements "maximum (window size 100)"
81− (Window.maximum 100)
81+ (Window.windowMaximum 100)
8282 , benchWithFold numElements "maximum (window size 1000)"
83− (Window.maximum 1000)
83+ (Window.windowMaximum 1000)
8484 , benchWith sourceDescendingInt numElements
8585 "maximum descending (window size 1000)"
86− (Window.maximum 1000)
86+ (Window.windowMaximum 1000)
8787
8888 , benchWithFold numElements "range (window size 100)"
89− (Window.range 100)
89+ (Window.windowRange 100)
9090 , benchWithFold numElements "range (window size 1000)"
91− (Window.range 1000)
91+ (Window.windowRange 1000)
9292 , benchWith sourceDescendingInt numElements
9393 "range descending (window size 1000)"
94− (Window.range 1000)
94+ (Window.windowRange 1000)
9595
9696 , benchWithFoldInt numElements "sumInt (window size 100)"
97− (Ring.slidingWindow 100 Window.sumInt)
97+ (Ring.slidingWindow 100 Window.windowSumInt)
9898 , benchWithFoldInt numElements "sum for Int (window size 100)"
99− (Ring.slidingWindow 100 Window.sum)
99+ (Ring.slidingWindow 100 Window.windowSum)
100100 , benchWithFold numElements "sum (window size 100)"
101− (Ring.slidingWindow 100 Window.sum)
101+ (Ring.slidingWindow 100 Window.windowSum)
102102 , benchWithFold numElements "sum (window size 1000)"
103− (Ring.slidingWindow 1000 Window.sum)
103+ (Ring.slidingWindow 1000 Window.windowSum)
104104 , benchWithFold numElements "sum (entire stream)"
105− (Window.cumulative Window.sum)
105+ (Window.cumulative Window.windowSum)
106106 , benchWithFold numElements "sum (Data.Fold)"
107107 Fold.sum
108108
109109 , benchWithFold numElements "mean (window size 100)"
110− (Ring.slidingWindow 100 Window.mean)
110+ (Ring.slidingWindow 100 Window.windowMean)
111111 , benchWithFold numElements "mean (window size 1000)"
112− (Ring.slidingWindow 1000 Window.mean)
112+ (Ring.slidingWindow 1000 Window.windowMean)
113113 , benchWithFold numElements "mean (entire stream)"
114− (Window.cumulative Window.mean)
114+ (Window.cumulative Window.windowMean)
115115 , benchWithFold numElements "mean (Data.Fold)"
116116 Fold.mean
117117
118118 , benchWithFold numElements "powerSum 2 (window size 100)"
119− (Ring.slidingWindow 100 (Window.powerSum 2))
119+ (Ring.slidingWindow 100 (Window.windowPowerSum 2))
120120 , benchWithFold numElements "powerSum 2 (entire stream)"
121− (Window.cumulative (Window.powerSum 2))
121+ (Window.cumulative (Window.windowPowerSum 2))
122122
123123 ]
124124 ]
@@ -127,51 +127,51 @@ o_1_space_scans :: Int -> [Benchmark]
127127o_1_space_scans numElements =
128128 [ bgroup "scan"
129129 [ benchWithPostscan numElements "minimum (window size 10)"
130− (Window.minimum 10)
130+ (Window.windowMinimum 10)
131131 -- Below window size 30 the linear search based impl performs better
132132 -- than the dequeue based implementation.
133133 , benchWithPostscan numElements "minimum (window size 30)"
134− (Window.minimum 30)
134+ (Window.windowMinimum 30)
135135 , benchWithPostscan numElements "minimum (window size 1000)"
136− (Window.minimum 1000)
136+ (Window.windowMinimum 1000)
137137 , benchScanWith sourceDescendingInt numElements
138138 "minimum descending (window size 1000)"
139− (Window.minimum 1000)
139+ (Window.windowMinimum 1000)
140140
141141 , benchWithPostscan numElements "maximum (window size 10)"
142− (Window.maximum 10)
142+ (Window.windowMaximum 10)
143143 , benchWithPostscan numElements "maximum (window size 30)"
144− (Window.maximum 30)
144+ (Window.windowMaximum 30)
145145 , benchWithPostscan numElements "maximum (window size 1000)"
146− (Window.maximum 1000)
146+ (Window.windowMaximum 1000)
147147 , benchScanWith sourceDescendingInt numElements
148148 "maximum descending (window size 1000)"
149− (Window.maximum 1000)
149+ (Window.windowMaximum 1000)
150150
151151 , benchWithPostscan numElements "range (window size 10)"
152− (Window.range 10)
152+ (Window.windowRange 10)
153153 , benchWithPostscan numElements "range (window size 30)"
154− (Window.range 30)
154+ (Window.windowRange 30)
155155 , benchWithPostscan numElements "range (window size 1000)"
156− (Window.range 1000)
156+ (Window.windowRange 1000)
157157 , benchScanWith sourceDescendingInt numElements
158158 "range descending (window size 1000)"
159− (Window.range 1000)
159+ (Window.windowRange 1000)
160160
161161 , benchWithPostscan numElements "sum (window size 100)"
162− (Ring.slidingWindow 100 Window.sum)
162+ (Ring.slidingWindow 100 Window.windowSum)
163163 , benchWithPostscan numElements "sum (window size 1000)"
164− (Ring.slidingWindow 1000 Window.sum)
164+ (Ring.slidingWindow 1000 Window.windowSum)
165165
166166 , benchWithPostscan numElements "mean (window size 100)"
167− (Ring.slidingWindow 100 Window.mean)
167+ (Ring.slidingWindow 100 Window.windowMean)
168168 , benchWithPostscan numElements "mean (window size 1000)"
169− (Ring.slidingWindow 1000 Window.mean)
169+ (Ring.slidingWindow 1000 Window.windowMean)
170170
171171 , benchWithPostscan numElements "powerSum 2 (window size 100)"
172− (Ring.slidingWindow 100 (Window.powerSum 2))
172+ (Ring.slidingWindow 100 (Window.windowPowerSum 2))
173173 , benchWithPostscan numElements "powerSum 2 (window size 1000)"
174− (Ring.slidingWindow 1000 (Window.powerSum 2))
174+ (Ring.slidingWindow 1000 (Window.windowPowerSum 2))
175175 ]
176176 ]
177177
benchmark/Streamly/Benchmark/Data/MutArray.hs+1−1
@@ -49,7 +49,7 @@ import qualified Streamly.Internal.Data.MutArray as MArray
4949import qualified Streamly.Internal.Data.Fold as Fold
5050import qualified Streamly.Internal.Data.Stream as Stream
5151
52−import Gauge
52+import Test.Tasty.Bench
5353import Streamly.Benchmark.Common hiding (benchPureSrc)
5454
5555type Stream = MutArray
benchmark/Streamly/Benchmark/Data/Parser.hs+10−10
@@ -47,10 +47,9 @@ import qualified Streamly.Internal.Data.Fold as Fold
4747import qualified Streamly.Internal.Data.Parser as PR
4848import qualified Streamly.Data.Stream as Stream
4949import qualified Streamly.Internal.Data.Producer as Producer
50−import qualified Streamly.Internal.Data.Producer.Source as Source
5150import qualified Streamly.Internal.Data.Stream as Stream
5251
53−import Gauge hiding (env)
52+import Test.Tasty.Bench hiding (env)
5453import Streamly.Benchmark.Common
5554import Streamly.Benchmark.Common.Handle
5655
@@ -349,7 +348,7 @@ split_ value =
349348-- PR.dropWhile (<= (value * 1 `div` 4)) *> PR.die "alt"
350349{-# INLINE takeWhileFail #-}
351350takeWhileFail :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
352−takeWhileFail predicate (Fold fstep finitial fextract) =
351+takeWhileFail predicate (Fold fstep finitial _ ffinal) =
353352 Parser step initial extract
354353
355354 where
@@ -370,7 +369,7 @@ takeWhileFail predicate (Fold fstep finitial fextract) =
370369 Fold.Done b -> Done 0 b
371370 else return $ Error "fail"
372371
373− extract s = fmap (Done 0) (fextract s)
372+ extract s = fmap (Done 0) (ffinal s)
374373
375374{-# INLINE alt2 #-}
376375alt2 :: Monad m
@@ -635,13 +634,13 @@ choice value =
635634{-# INLINE parseManyUnfoldArrays #-}
636635parseManyUnfoldArrays :: Int -> [Array.Array Int] -> IO ()
637636parseManyUnfoldArrays count arrays = do
638− let src = Source.source (Just (Producer.OuterLoop arrays))
637+ let src = Producer.source (Just (Producer.OuterLoop arrays))
639638 let parser = PR.fromFold (Fold.take count Fold.drain)
640639 let readSrc =
641− Source.producer
640+ Producer.producer
642641 $ Producer.concat Producer.fromList Array.producer
643642 let streamParser =
644− Producer.simplify (Source.parseMany parser readSrc)
643+ Producer.simplify (Producer.parseMany parser readSrc)
645644 Stream.fold Fold.drain $ Stream.unfold streamParser src
646645
647646-------------------------------------------------------------------------------
@@ -715,8 +714,6 @@ instance NFData ParseError where
715714o_1_space_serial :: Int -> [Benchmark]
716715o_1_space_serial value =
717716 [ benchIOSink value "takeBetween" $ takeBetween value
718− , benchIOSink value "takeEQ" $ takeEQ value
719− , benchIOSink value "takeGE" $ takeGE value
720717 , benchIOSink value "takeWhile" $ takeWhile value
721718 , benchIOSink value "takeWhileP" $ takeWhileP value
722719 , benchIOSink value "takeP" $ takeP value
@@ -813,8 +810,11 @@ o_1_space_serial_unfold bound arrays =
813810o_n_heap_serial :: Int -> [Benchmark]
814811o_n_heap_serial value =
815812 [
813+ benchIOSink value "takeEQ" $ takeEQ value
814+ , benchIOSink value "takeGE" $ takeGE value
815+
816816 -- lookahead benchmark holds the entire input till end
817− benchIOSink value "lookAhead" $ lookAhead value
817+ , benchIOSink value "lookAhead" $ lookAhead value
818818
819819 -- o-n-heap because of backtracking
820820 , benchIOSrc sourceEscapedFrames value "takeFramedByEsc_"
benchmark/Streamly/Benchmark/Data/ParserK.hs+90−37
@@ -5,6 +5,10 @@
55-- License : BSD-3-Clause
66-- Maintainer : streamly@composewell.com
77
8+-- BENCH_CHUNKED -> adaptC
9+-- BENCH_CHUNKED_GENERIC -> adaptCG
10+-- BENCH_SINGULAR -> adapt
11+
812{-# LANGUAGE CPP #-}
913{-# LANGUAGE FlexibleContexts #-}
1014{-# OPTIONS_GHC -Wno-orphans #-}
@@ -18,8 +22,14 @@ import Control.Applicative ((<|>))
1822import Control.DeepSeq (NFData(..))
1923import Control.Monad.IO.Class (MonadIO)
2024import Data.Foldable (asum)
25+#ifdef BENCH_CHUNKED
2126import Streamly.Data.Array (Array, Unbox)
22−import Streamly.Internal.Data.Fold.Type (Fold(..))
27+#endif
28+#ifdef BENCH_CHUNKED_GENERIC
29+import Streamly.Data.Array.Generic (Array)
30+import qualified Streamly.Internal.Data.Array.Generic as GenArr (chunksOf)
31+#endif
32+import Streamly.Internal.Data.Fold (Fold(..))
2333import Streamly.Data.StreamK (StreamK)
2434import Streamly.Internal.Data.Parser
2535 (ParseError(..), Parser(..), Initial(..), Step(..))
@@ -35,12 +45,52 @@ import qualified Streamly.Data.Stream as Stream
3545import qualified Streamly.Data.Fold as FL
3646import qualified Streamly.Internal.Data.Fold as Fold
3747import qualified Streamly.Data.Parser as PRD
38−import qualified Streamly.Data.ParserK as PR
48+import qualified Streamly.Internal.Data.ParserK as PR
3949import qualified Streamly.Internal.Data.StreamK as StreamK
4050
41−import Gauge
51+import Test.Tasty.Bench
4252import Streamly.Benchmark.Common
4353
54+-------------------------------------------------------------------------------
55+-- CPP Helpers
56+-------------------------------------------------------------------------------
57+
58+#ifdef BENCH_CHUNKED
59+
60+#define PARSE_OP StreamK.parseChunks
61+#define FROM_PARSER adaptC
62+#define INPUT (Array a)
63+#define PARSE_ELEM (Array Int)
64+#define CONSTRAINT_IO (MonadIO m, Unbox a)
65+#define CONSTRAINT (Monad m, Unbox a)
66+#define MODULE_NAME "Data.ParserK.Chunked"
67+
68+#endif
69+
70+#ifdef BENCH_CHUNKED_GENERIC
71+
72+#define PARSE_OP StreamK.parseChunksGeneric
73+#define FROM_PARSER adaptCG
74+#define INPUT (Array a)
75+#define PARSE_ELEM (Array Int)
76+#define CONSTRAINT_IO (MonadIO m)
77+#define CONSTRAINT (Monad m)
78+#define MODULE_NAME "Data.ParserK.Chunked.Generic"
79+
80+#endif
81+
82+#ifdef BENCH_SINGULAR
83+
84+#define PARSE_OP StreamK.parse
85+#define FROM_PARSER adapt
86+#define INPUT a
87+#define PARSE_ELEM Int
88+#define CONSTRAINT_IO (MonadIO m)
89+#define CONSTRAINT (Monad m)
90+#define MODULE_NAME "Data.ParserK"
91+
92+#endif
93+
4494-------------------------------------------------------------------------------
4595-- Utilities
4696-------------------------------------------------------------------------------
@@ -60,49 +110,52 @@ sourceUnfoldrM value n = Stream.unfoldrM step n
60110{-# INLINE benchIOSink #-}
61111benchIOSink
62112 :: NFData b
63− => Int -> String -> (StreamK IO (Array Int) -> IO b) -> Benchmark
113+ => Int -> String -> (StreamK IO PARSE_ELEM -> IO b) -> Benchmark
64114benchIOSink value name f =
65115 bench name $ nfIO $ randomRIO (1,1)
66116 >>= f
67117 . StreamK.fromStream
118+#ifdef BENCH_CHUNKED
68119 . Stream.chunksOf 4000
120+#endif
121+#ifdef BENCH_CHUNKED_GENERIC
122+ . GenArr.chunksOf 4000
123+#endif
69124 . sourceUnfoldrM value
70125
71126-------------------------------------------------------------------------------
72127-- Parsers
73128-------------------------------------------------------------------------------
74129
75−#define PARSE_OP StreamK.parseChunks
76−
77130{-# INLINE one #-}
78131one :: MonadIO m =>
79− Int -> StreamK m (Array Int) -> m (Either ParseError (Maybe Int))
80−one value = StreamK.parseChunks p
132+ Int -> StreamK m PARSE_ELEM -> m (Either ParseError (Maybe Int))
133+one value = PARSE_OP p
81134
82135 where
83136
84137 p = do
85− m <- PR.fromFold FL.one
138+ m <- PR.FROM_PARSER (PRD.fromFold FL.one)
86139 case m of
87140 Just i -> if i >= value then pure m else p
88141 Nothing -> pure Nothing
89142
90143{-# INLINE satisfy #-}
91−satisfy :: (MonadIO m, Unbox a) => (a -> Bool) -> PR.ParserK a m a
92−satisfy = PR.fromParser . PRD.satisfy
144+satisfy :: CONSTRAINT_IO => (a -> Bool) -> PR.ParserK INPUT m a
145+satisfy = PR.FROM_PARSER . PRD.satisfy
93146
94147{-# INLINE takeWhile #-}
95−takeWhile :: (MonadIO m, Unbox a) => (a -> Bool) -> PR.ParserK a m ()
96−takeWhile p = PR.fromParser $ PRD.takeWhile p FL.drain
148+takeWhile :: CONSTRAINT_IO => (a -> Bool) -> PR.ParserK INPUT m ()
149+takeWhile p = PR.FROM_PARSER $ PRD.takeWhile p FL.drain
97150
98151{-# INLINE takeWhileK #-}
99152takeWhileK :: MonadIO m =>
100− Int -> StreamK m (Array Int) -> m (Either ParseError ())
153+ Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
101154takeWhileK value = PARSE_OP (takeWhile (<= value))
102155
103156{-# INLINE splitAp2 #-}
104157splitAp2 :: MonadIO m
105− => Int -> StreamK m (Array Int) -> m (Either ParseError ((), ()))
158+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ((), ()))
106159splitAp2 value =
107160 PARSE_OP
108161 ((,)
@@ -112,7 +165,7 @@ splitAp2 value =
112165
113166{-# INLINE splitAp8 #-}
114167splitAp8 :: MonadIO m
115− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
168+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
116169splitAp8 value =
117170 PARSE_OP
118171 ( (\() () () () () () () () -> ())
@@ -127,7 +180,7 @@ splitAp8 value =
127180 )
128181
129182{-# INLINE sequenceA #-}
130−sequenceA :: MonadIO m => Int -> StreamK m (Array Int) -> m Int
183+sequenceA :: MonadIO m => Int -> StreamK m PARSE_ELEM -> m Int
131184sequenceA value xs = do
132185 let parser = satisfy (> 0)
133186 list = Prelude.replicate value parser
@@ -136,14 +189,14 @@ sequenceA value xs = do
136189
137190{-# INLINE sequenceA_ #-}
138191sequenceA_ :: MonadIO m =>
139− Int -> StreamK m (Array Int) -> m (Either ParseError ())
192+ Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
140193sequenceA_ value xs = do
141194 let parser = satisfy (> 0)
142195 list = Prelude.replicate value parser
143196 PARSE_OP (F.sequenceA_ list) xs
144197
145198{-# INLINE sequence #-}
146−sequence :: MonadIO m => Int -> StreamK m (Array Int) -> m Int
199+sequence :: MonadIO m => Int -> StreamK m PARSE_ELEM -> m Int
147200sequence value xs = do
148201 let parser = satisfy (> 0)
149202 list = Prelude.replicate value parser
@@ -152,7 +205,7 @@ sequence value xs = do
152205
153206{-# INLINE sequence_ #-}
154207sequence_ :: MonadIO m =>
155− Int -> StreamK m (Array Int) -> m (Either ParseError ())
208+ Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
156209sequence_ value =
157210 let parser = satisfy (> 0)
158211 list = Prelude.replicate value parser
@@ -160,7 +213,7 @@ sequence_ value =
160213
161214{-# INLINE takeWhileFailD #-}
162215takeWhileFailD :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
163−takeWhileFailD predicate (Fold fstep finitial fextract) =
216+takeWhileFailD predicate (Fold fstep finitial _ ffinal) =
164217 Parser step initial extract
165218
166219 where
@@ -181,16 +234,16 @@ takeWhileFailD predicate (Fold fstep finitial fextract) =
181234 Fold.Done b -> Done 0 b
182235 else return $ Error "fail"
183236
184− extract s = fmap (Done 0) (fextract s)
237+ extract s = fmap (Done 0) (ffinal s)
185238
186239{-# INLINE takeWhileFail #-}
187−takeWhileFail :: (Monad m, Unbox a) =>
188− (a -> Bool) -> Fold m a b -> PR.ParserK a m b
189−takeWhileFail p f = PR.fromParser (takeWhileFailD p f)
240+takeWhileFail :: CONSTRAINT =>
241+ (a -> Bool) -> Fold m a b -> PR.ParserK INPUT m b
242+takeWhileFail p f = PR.FROM_PARSER (takeWhileFailD p f)
190243
191244{-# INLINE alt2 #-}
192245alt2 :: MonadIO m
193− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
246+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
194247alt2 value =
195248 PARSE_OP
196249 ( takeWhileFail (<= (value `div` 2)) Fold.drain
@@ -199,7 +252,7 @@ alt2 value =
199252
200253{-# INLINE alt8 #-}
201254alt8 :: MonadIO m
202− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
255+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
203256alt8 value =
204257 PARSE_OP
205258 ( takeWhileFail (<= ( value `div` 8)) Fold.drain
@@ -214,7 +267,7 @@ alt8 value =
214267
215268{-# INLINE alt16 #-}
216269alt16 :: MonadIO m
217− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
270+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
218271alt16 value =
219272 PARSE_OP
220273 ( takeWhileFail (<= ( value `div` 16)) Fold.drain
@@ -236,27 +289,27 @@ alt16 value =
236289 )
237290
238291{-# INLINE manyAlt #-}
239−manyAlt :: MonadIO m => StreamK m (Array Int) -> m Int
292+manyAlt :: MonadIO m => StreamK m PARSE_ELEM -> m Int
240293manyAlt xs = do
241294 x <- PARSE_OP (AP.many (satisfy (> 0))) xs
242295 return $ Prelude.length x
243296
244297{-# INLINE someAlt #-}
245−someAlt :: MonadIO m => StreamK m (Array Int) -> m Int
298+someAlt :: MonadIO m => StreamK m PARSE_ELEM -> m Int
246299someAlt xs = do
247300 x <- PARSE_OP (AP.some (satisfy (> 0))) xs
248301 return $ Prelude.length x
249302
250303{-# INLINE choice #-}
251304choice :: MonadIO m =>
252− Int -> StreamK m (Array Int) -> m (Either ParseError Int)
305+ Int -> StreamK m PARSE_ELEM -> m (Either ParseError Int)
253306choice value =
254307 PARSE_OP (asum (replicate value (satisfy (< 0)))
255308 AP.<|> satisfy (> 0))
256309
257310{-# INLINE monad2 #-}
258311monad2 :: MonadIO m
259− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
312+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
260313monad2 value =
261314 PARSE_OP $ do
262315 takeWhile (<= (value `div` 2))
@@ -264,7 +317,7 @@ monad2 value =
264317
265318{-# INLINE monad4 #-}
266319monad4 :: MonadIO m
267− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
320+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
268321monad4 value =
269322 PARSE_OP $ do
270323 takeWhile (<= ( value `div` 4))
@@ -274,7 +327,7 @@ monad4 value =
274327
275328{-# INLINE monad8 #-}
276329monad8 :: MonadIO m
277− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
330+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
278331monad8 value =
279332 PARSE_OP $ do
280333 takeWhile (<= ( value `div` 8))
@@ -288,7 +341,7 @@ monad8 value =
288341
289342{-# INLINE monad16 #-}
290343monad16 :: MonadIO m
291− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
344+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
292345monad16 value =
293346 PARSE_OP $ do
294347 takeWhile (<= ( value `div` 16))
@@ -313,7 +366,7 @@ monad16 value =
313366-------------------------------------------------------------------------------
314367
315368moduleName :: String
316−moduleName = "Data.ParserK"
369+moduleName = MODULE_NAME
317370
318371instance NFData ParseError where
319372 {-# INLINE rnf #-}
@@ -335,7 +388,7 @@ o_1_space_serial value =
335388 ]
336389
337390{-# INLINE sepBy1 #-}
338−sepBy1 :: MonadIO m => StreamK m (Array Int) -> m Int
391+sepBy1 :: MonadIO m => StreamK m PARSE_ELEM -> m Int
339392sepBy1 xs = do
340393 x <- PARSE_OP (parser (satisfy odd) (satisfy even)) xs
341394 return $ Prelude.length x
benchmark/Streamly/Benchmark/Data/Ring/Unboxed.hs+3−4
@@ -12,12 +12,11 @@ module Main (main) where
1212
1313import Control.Monad (void)
1414import GHC.Ptr (Ptr(..))
15−
16−import qualified Streamly.Internal.Data.Array.Type as Array
17−import qualified Streamly.Internal.Data.Ring.Unboxed as Ring
15+import qualified Streamly.Internal.Data.Array as Array
16+import qualified Streamly.Internal.Data.Ring as Ring
1817import qualified Data.Foldable as P
1918
20−import Gauge
19+import Test.Tasty.Bench
2120import Streamly.Benchmark.Common
2221
2322import Prelude as P
benchmark/Streamly/Benchmark/Data/Serialize.hsnew file+591
@@ -0,0 +1,591 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+{- HLINT ignore -}
3+
4+#undef FUSION_CHECK
5+#ifdef FUSION_CHECK
6+{-# OPTIONS_GHC -ddump-simpl -ddump-to-file -dsuppress-all #-}
7+#endif
8+
9+-- |
10+-- Module : Streamly.Benchmark.Data.Serialize
11+-- Copyright : (c) 2023 Composewell
12+-- License : BSD-3-Clause
13+-- Maintainer : streamly@composewell.com
14+
15+module Main (main) where
16+
17+-------------------------------------------------------------------------------
18+-- Imports
19+-------------------------------------------------------------------------------
20+
21+import Control.DeepSeq (NFData(..), deepseq)
22+import Control.Monad (when, void)
23+import GHC.Generics (Generic)
24+import System.Random (randomRIO)
25+#ifndef USE_UNBOX
26+import Test.QuickCheck (Arbitrary, arbitrary)
27+#endif
28+
29+import Streamly.Internal.Data.MutByteArray
30+import qualified Streamly.Internal.Data.MutByteArray as MBA
31+#ifdef USE_UNBOX
32+import Data.Proxy (Proxy(..))
33+#else
34+import Control.DeepSeq (force)
35+import Test.QuickCheck (oneof, generate)
36+#endif
37+
38+import Test.Tasty.Bench
39+import Streamly.Benchmark.Common
40+
41+#ifndef USE_UNBOX
42+import Streamly.Benchmark.Data.Serialize.RecCompatible
43+import Streamly.Benchmark.Data.Serialize.RecNonCompatible
44+#endif
45+
46+#ifdef USE_UNBOX
47+#define SERIALIZE_CLASS Unbox
48+#define DERIVE_CLASS(typ) $(deriveUnbox [d|instance Unbox typ|])
49+#define SERIALIZE_OP pokeAt
50+#define DESERIALIZE_OP peekAt
51+#else
52+#define SERIALIZE_CLASS Serialize
53+#define DERIVE_CLASS(typ) $(deriveSerialize [d|instance Serialize typ|])
54+#define SERIALIZE_OP serializeAt
55+#define DESERIALIZE_OP deserializeAt
56+#endif
57+
58+-------------------------------------------------------------------------------
59+-- Types
60+-------------------------------------------------------------------------------
61+
62+data Unit = Unit
63+ deriving (Generic, Show, Eq)
64+
65+#ifndef USE_TH
66+instance SERIALIZE_CLASS Unit
67+#else
68+DERIVE_CLASS(Unit)
69+#endif
70+
71+instance NFData Unit where
72+ {-# INLINE rnf #-}
73+ rnf _ = ()
74+
75+data Sum2
76+ = Sum21
77+ | Sum22
78+ deriving (Generic, Show, Eq)
79+
80+#ifndef USE_TH
81+instance SERIALIZE_CLASS Sum2
82+#else
83+DERIVE_CLASS(Sum2)
84+#endif
85+
86+instance NFData Sum2 where
87+ {-# INLINE rnf #-}
88+ rnf _ = ()
89+
90+data Sum25
91+ = Sum251
92+ | Sum252
93+ | Sum253
94+ | Sum254
95+ | Sum255
96+ | Sum256
97+ | Sum257
98+ | Sum258
99+ | Sum259
100+ | Sum2510
101+ | Sum2511
102+ | Sum2512
103+ | Sum2513
104+ | Sum2514
105+ | Sum2515
106+ | Sum2516
107+ | Sum2517
108+ | Sum2518
109+ | Sum2519
110+ | Sum2520
111+ | Sum2521
112+ | Sum2522
113+ | Sum2523
114+ | Sum2524
115+ | Sum2525
116+ deriving (Generic, Show, Eq)
117+
118+#ifndef USE_TH
119+instance SERIALIZE_CLASS Sum25
120+#else
121+DERIVE_CLASS(Sum25)
122+#endif
123+
124+instance NFData Sum25 where
125+ {-# INLINE rnf #-}
126+ rnf _ = ()
127+
128+data Product25
129+ = Product25
130+ Int
131+ Int
132+ Int
133+ Int
134+ Int
135+ Int
136+ Int
137+ Int
138+ Int
139+ Int
140+ Int
141+ Int
142+ Int
143+ Int
144+ Int
145+ Int
146+ Int
147+ Int
148+ Int
149+ Int
150+ Int
151+ Int
152+ Int
153+ Int
154+ Int
155+ deriving (Generic, Show)
156+
157+#ifndef USE_TH
158+instance SERIALIZE_CLASS Product25
159+#else
160+DERIVE_CLASS(Product25)
161+#endif
162+
163+instance NFData Product25 where
164+ {-# INLINE rnf #-}
165+ rnf
166+ (Product25 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 a13 a14 a15 a16 a17 a18 a19 a20 a21 a22 a23 a24 a25)
167+ = a1
168+ `seq` a2
169+ `seq` a3
170+ `seq` a4
171+ `seq` a5
172+ `seq` a6
173+ `seq` a7
174+ `seq` a8
175+ `seq` a9
176+ `seq` a10
177+ `seq` a11
178+ `seq` a12
179+ `seq` a13
180+ `seq` a14
181+ `seq` a15
182+ `seq` a16
183+ `seq` a17
184+ `seq` a18
185+ `seq` a19
186+ `seq` a20
187+ `seq` a21
188+ `seq` a22
189+ `seq` a23
190+ `seq` a24
191+ `seq` a25
192+ `seq` ()
193+
194+-- XXX derived Eq instance is not inlined
195+instance Eq Product25 where
196+ {-# INLINE (==) #-}
197+ (==)
198+ (Product25 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 a13 a14 a15 a16 a17 a18 a19 a20 a21 a22 a23 a24 a25)
199+ (Product25 b1 b2 b3 b4 b5 b6 b7 b8 b9 b10 b11 b12 b13 b14 b15 b16 b17 b18 b19 b20 b21 b22 b23 b24 b25)
200+ = a1 == b1
201+ && a2 == b2
202+ && a3 == b3
203+ && a4 == b4
204+ && a5 == b5
205+ && a6 == b6
206+ && a7 == b7
207+ && a8 == b8
208+ && a9 == b9
209+ && a10 == b10
210+ && a11 == b11
211+ && a12 == b12
212+ && a13 == b13
213+ && a14 == b14
214+ && a15 == b15
215+ && a16 == b16
216+ && a17 == b17
217+ && a18 == b18
218+ && a19 == b19
219+ && a20 == b20
220+ && a21 == b21
221+ && a22 == b22
222+ && a23 == b23
223+ && a24 == b24
224+ && a25 == b25
225+
226+-------------------------------------------------------------------------------
227+-- Simple non-recursive ADT
228+-------------------------------------------------------------------------------
229+
230+{-
231+data CustomDT1
232+ = CDT1C1
233+ | CDT1C2 Int
234+ | CDT1C3 Int Int
235+ deriving (Generic, Show, Eq)
236+
237+#ifndef USE_TH
238+instance SERIALIZE_CLASS CustomDT1
239+#else
240+DERIVE_CLASS(CustomDT1)
241+#endif
242+
243+instance NFData CustomDT1 where
244+ {-# INLINE rnf #-}
245+ rnf CDT1C1 = ()
246+ rnf (CDT1C2 i) = i `seq` ()
247+ rnf (CDT1C3 i1 i2) = i1 `seq` i2 `seq` ()
248+-}
249+
250+-------------------------------------------------------------------------------
251+-- Recursive ADT
252+-------------------------------------------------------------------------------
253+
254+#ifndef USE_UNBOX
255+data BinTree a
256+ = Tree (BinTree a) (BinTree a)
257+ | Leaf a
258+ deriving (Show, Read, Eq, Generic)
259+
260+#ifndef USE_TH
261+instance Serialize (BinTree a)
262+#else
263+$(deriveSerialize [d|instance Serialize a => Serialize (BinTree a)|])
264+#endif
265+
266+instance NFData a => NFData (BinTree a) where
267+ {-# INLINE rnf #-}
268+ rnf (Leaf a) = rnf a `seq` ()
269+ rnf (Tree l r) = rnf l `seq` rnf r
270+
271+-- XXX This may not terminate, or could become really large.
272+instance Arbitrary a => Arbitrary (BinTree a) where
273+ arbitrary = oneof [Leaf <$> arbitrary, Tree <$> arbitrary <*> arbitrary]
274+
275+mkBinTree :: (Arbitrary a) => Int -> IO (BinTree a)
276+mkBinTree = go (generate arbitrary)
277+
278+ where
279+
280+ go r 0 = Leaf <$> r
281+ go r n = Tree <$> go r (n - 1) <*> go r (n - 1)
282+
283+#endif
284+
285+-------------------------------------------------------------------------------
286+-- Size helpers
287+-------------------------------------------------------------------------------
288+
289+{-# INLINE getSize #-}
290+getSize :: forall a. SERIALIZE_CLASS a => a -> Int
291+#ifdef USE_UNBOX
292+getSize _ = sizeOf (Proxy :: Proxy a)
293+#else
294+getSize = addSizeTo 0
295+#endif
296+
297+-------------------------------------------------------------------------------
298+-- Common helpers
299+-------------------------------------------------------------------------------
300+
301+{- HLINT ignore "Eta reduce" -}
302+-- Parts of "f" that are dependent on val will not be optimized out.
303+{-# INLINE loop #-}
304+loop :: Int -> (a -> IO b) -> a -> IO ()
305+loop count f val = go count val
306+ where
307+
308+ go n x = when (n > 0) $ f x >> go (n-1) x
309+
310+-- The first arg of "f" is the environment which is not threaded around in the
311+-- loop.
312+{-# INLINE loopWith #-}
313+loopWith :: Int -> (e -> a -> IO b) -> e -> a -> IO ()
314+loopWith count f e val = go count val
315+ where
316+
317+ go n x = when (n > 0) $ f e x >> go (n-1) x
318+
319+benchSink :: NFData b => String -> Int -> (Int -> IO b) -> Benchmark
320+benchSink name times f = bench name (nfIO (randomRIO (times, times) >>= f))
321+
322+-------------------------------------------------------------------------------
323+-- Serialization Helpers
324+-------------------------------------------------------------------------------
325+
326+{-
327+{-# INLINE pokeWithSize #-}
328+pokeWithSize :: SERIALIZE_CLASS a => MutByteArray -> a -> IO ()
329+pokeWithSize arr val = do
330+ let n = getSize val
331+ n `seq` SERIALIZE_OP 0 arr val >> return ()
332+
333+{-# INLINE pokeTimesWithSize #-}
334+pokeTimesWithSize :: SERIALIZE_CLASS a => a -> Int -> IO ()
335+pokeTimesWithSize val times = do
336+ let n = getSize val
337+ arr <- MBA.new n
338+ loopWith times pokeWithSize arr val
339+-}
340+
341+{-# INLINE poke #-}
342+poke :: SERIALIZE_CLASS a => MutByteArray -> a -> IO ()
343+poke arr val = void (SERIALIZE_OP 0 arr val)
344+
345+{-# INLINE pokeTimes #-}
346+pokeTimes :: SERIALIZE_CLASS a => a -> Int -> IO ()
347+pokeTimes val times = do
348+ let n = getSize val
349+ arr <- MBA.new n
350+ loopWith times poke arr val
351+
352+{-# INLINE encode #-}
353+encode :: SERIALIZE_CLASS a => a -> IO ()
354+encode val = do
355+ let n = getSize val
356+ arr <- MBA.new n
357+ void (SERIALIZE_OP 0 arr val)
358+
359+{-# INLINE encodeTimes #-}
360+encodeTimes :: SERIALIZE_CLASS a => a -> Int -> IO ()
361+encodeTimes val times = loop times encode val
362+
363+{-# INLINE peek #-}
364+peek :: forall a. (NFData a, SERIALIZE_CLASS a) =>
365+ (a, Int) -> MutByteArray -> IO ()
366+#ifdef USE_UNBOX
367+peek (_val, _) arr = do
368+ (val1 :: a) <- DESERIALIZE_OP 0 arr
369+#else
370+peek (_val, n) arr = do
371+ (_, val1 :: a) <- DESERIALIZE_OP 0 arr n
372+#endif
373+ -- If the datatype is not deeply strict or deepseq is not used then use
374+ -- Equality.
375+ -- Ensure that we are actually constructing the type and using it. This
376+ -- is important, otherwise the structure is created and discarded, the
377+ -- cost of creation of the structure is not accounted. Otherwise we may
378+ -- just read the values and discard them. The comparison adds to the
379+ -- cost though. We could use deepseq but then we need to write
380+ -- instances of NFData and ensure that they are correct and perform
381+ -- well. Equality check also ensures correctness.
382+ {-
383+ if (val1 /= val)
384+ then error "peek: no match"
385+ else return ()
386+ -}
387+ val1 `deepseq` return ()
388+
389+{-# INLINE peekTimes #-}
390+peekTimes :: (NFData a, SERIALIZE_CLASS a) => Int -> a -> Int -> IO ()
391+peekTimes n val times = do
392+ arr <- MBA.new n
393+ _ <- SERIALIZE_OP 0 arr val
394+ loopWith times peek (val, n) arr
395+
396+{-# INLINE trip #-}
397+trip :: forall a. (NFData a, SERIALIZE_CLASS a) => a -> IO ()
398+trip val = do
399+ let n = getSize val
400+ arr <- MBA.new n
401+ _ <- SERIALIZE_OP 0 arr val
402+#ifdef USE_UNBOX
403+ (val1 :: a) <- DESERIALIZE_OP 0 arr
404+#else
405+ (_, val1 :: a) <- DESERIALIZE_OP 0 arr n
406+#endif
407+ -- Do not remove this or use deepseq, see the comments in peek.
408+ {-
409+ if (val1 /= val)
410+ then error "roundtrip: no match"
411+ else return ()
412+ -}
413+ -- Note: deepseq is not needed if the datatype is strict
414+ val1 `deepseq` return ()
415+
416+{-# INLINE roundtrip #-}
417+roundtrip :: (NFData a, SERIALIZE_CLASS a) => a -> Int -> IO ()
418+roundtrip val times = loop times trip val
419+
420+-------------------------------------------------------------------------------
421+-- Benchmarks
422+-------------------------------------------------------------------------------
423+
424+{-# INLINE benchConst #-}
425+benchConst ::
426+ String
427+ -> (forall a. (NFData a, SERIALIZE_CLASS a) =>
428+ Int -> a -> Int -> IO ())
429+ -> Int
430+ -> Benchmark
431+benchConst gname f times =
432+ bgroup gname
433+ [ let !n = getSize Unit
434+ in benchSink "Unit" times (f n Unit)
435+ {-
436+ , let !n = getSize CDT1C1
437+ in benchSink "C1" times (f n CDT1C1)
438+ , let val = CDT1C2 5
439+ !n = getSize val
440+ in benchSink "C2" (times `div` 2) (f n val)
441+ , let val = CDT1C3 5 2
442+ !n = getSize val
443+ in benchSink "C3" (times `div` 3) (f n val)
444+ -}
445+ , let !n = getSize Sum21
446+ in benchSink "Sum2" times (f n Sum21)
447+ , let !n = getSize Sum2525
448+ in benchSink "Sum25" times (f n Sum2525)
449+ , let val = Product25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
450+ !n = getSize val
451+ in benchSink "Product25" (times `div` 26) (f n val)
452+ ]
453+
454+#ifndef USE_UNBOX
455+{-# INLINE benchVar #-}
456+benchVar ::
457+ String
458+ -> (forall a. (NFData a, SERIALIZE_CLASS a) =>
459+ Int -> a -> Int -> IO ())
460+ -> BinTree Int
461+ -> [Int]
462+ -> Int
463+ -> Benchmark
464+benchVar gname f tInt lInt times =
465+ bgroup gname
466+ [ let !n = getSize tInt
467+ in benchSink "bintree-int" times (f n tInt)
468+ , let !n = getSize lInt
469+ in benchSink "list-int" times (f n lInt)
470+ ]
471+
472+{-
473+{-# INLINE benchGeneric #-}
474+benchGeneric
475+ :: (NFData b, Serialize b)
476+ => String
477+ -> String
478+ -> (forall a. (NFData a, Serialize a) =>
479+ Int -> a -> Int -> IO ())
480+ -> b
481+ -> Int
482+ -> Benchmark
483+benchGeneric gname valId f val times =
484+ bgroup gname
485+ [ let !n = getSize val
486+ in benchSink valId times (f n val)
487+ ]
488+-}
489+
490+{-# INLINE benchLargeRec #-}
491+benchLargeRec ::
492+ String
493+ -> (forall a. (NFData a, Serialize a) =>
494+ Int -> a -> Int -> IO ())
495+ -> RecNonCompatible
496+ -> RecCompatible
497+ -> Int
498+ -> Benchmark
499+benchLargeRec gname f recL recR times =
500+ bgroup gname
501+ [ let !n = getSize recL
502+ in benchSink "RecNonCompatible" times (f n recL)
503+ , let !n = getSize recR
504+ in benchSink "RecCompatible" times (f n recR)
505+ ]
506+#endif
507+
508+-- Times is scaled by the number of constructors to normalize
509+#ifdef USE_UNBOX
510+allBenchmarks :: Int -> [Benchmark]
511+allBenchmarks times =
512+#else
513+allBenchmarks ::
514+ BinTree Int -> [Int] -> RecNonCompatible -> RecCompatible -> Int -> [Benchmark]
515+allBenchmarks tInt lInt recL recR times =
516+#endif
517+ [ bgroup "sizeOf"
518+ [
519+#ifndef USE_UNBOX
520+ bench "bintree-int" $ nf getSize tInt
521+ , bench "list-int" $ nf getSize lInt
522+#endif
523+ ]
524+ , benchConst "poke" (const pokeTimes) times
525+ -- , benchConst "pokeWithSize" (const pokeTimesWithSize) times
526+ , benchConst "encode" (const encodeTimes) times
527+ , benchConst "peek" peekTimes times
528+ , benchConst "roundtrip" (const roundtrip) times
529+#ifndef USE_UNBOX
530+ , benchVar "poke" (const pokeTimes) tInt lInt 1
531+ -- , benchVar "pokeWithSize" (const pokeTimesWithSize) tInt lInt 1
532+ , benchVar "encode" (const encodeTimes) tInt lInt 1
533+ , benchVar "peek" peekTimes tInt lInt 1
534+ , benchVar "roundtrip" (const roundtrip) tInt lInt 1
535+ , benchLargeRec "poke" (const pokeTimes) recL recR times
536+ , benchLargeRec "encode" (const encodeTimes) recL recR times
537+ , benchLargeRec "peek" peekTimes recL recR times
538+ , benchLargeRec "roundtrip" (const roundtrip) recL recR times
539+#endif
540+ ]
541+
542+-------------------------------------------------------------------------------
543+-- Driver
544+-------------------------------------------------------------------------------
545+
546+main :: IO ()
547+main = do
548+#ifndef USE_UNBOX
549+ -- Approximately 100000 constructors
550+ -- Assuming Leaf nodes constitute two constructors (Leaf, Int) and internal
551+ -- nodes 1 level = log_2 (100001/3) + 1 = 16
552+ !(tInt :: BinTree Int) <- force <$> mkBinTree 16
553+
554+ let !recL = valRecNonCompatible
555+ let !recR = valRecCompatible
556+
557+ -- Approximately 100000 constructors, assuming two constructors (Cons, Int)
558+ -- per element.
559+ let lInt = [1..50000 :: Int]
560+ let !len = length lInt -- evaluate the list
561+#endif
562+#ifndef FUSION_CHECK
563+#ifdef USE_UNBOX
564+ runWithCLIOpts defaultStreamSize allBenchmarks
565+#else
566+ len
567+ `seq` runWithCLIOpts
568+ defaultStreamSize
569+ (allBenchmarks tInt lInt recL recR)
570+#endif
571+#else
572+ -- Enable FUSION_CHECK macro at the beginning of the file
573+ -- Enable one benchmark below, and run the benchmark
574+ -- Check the .dump-simpl output
575+ let value = 100000
576+ -- print $ getSize (CDT1C3 4 2)
577+ -- print $ getSize tInt
578+ -- print $ getSize lInt
579+
580+ -- pokeTimes tInt 1
581+
582+ -- peekTimes ((CDT1C2 (5 :: Int)) :: CustomDT1) value
583+ -- peekTimes (Sum2525) value
584+ -- peekTimes (Product25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25) value
585+ -- peekTimes tInt 1
586+ let !n = getSize lInt
587+ peekTimes n lInt 1
588+
589+ -- roundtrip ((CDT1C2 (5 :: Int)) :: CustomDT1) value
590+ return ()
591+#endif
benchmark/Streamly/Benchmark/Data/Serialize/RecCompatible.hsnew file+24
@@ -0,0 +1,24 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Streamly.Benchmark.Data.Serialize.RecCompatible
4+ ( RecCompatible(..)
5+ , valRecCompatible
6+ ) where
7+
8+--------------------------------------------------------------------------------
9+-- Imports
10+--------------------------------------------------------------------------------
11+
12+import Streamly.Internal.Data.MutByteArray (Serialize)
13+import Streamly.Benchmark.Data.Serialize.TH (genLargeRecord)
14+
15+import qualified Streamly.Internal.Data.MutByteArray as Serialize
16+
17+--------------------------------------------------------------------------------
18+-- Code
19+--------------------------------------------------------------------------------
20+
21+$(genLargeRecord "RecCompatible" 50)
22+$(Serialize.deriveSerializeWith
23+ (Serialize.encodeRecordFields True)
24+ [d|instance Serialize RecCompatible|])
benchmark/Streamly/Benchmark/Data/Serialize/RecNonCompatible.hsnew file+21
@@ -0,0 +1,21 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Streamly.Benchmark.Data.Serialize.RecNonCompatible
4+ ( RecNonCompatible(..)
5+ , valRecNonCompatible
6+ ) where
7+
8+--------------------------------------------------------------------------------
9+-- Imports
10+--------------------------------------------------------------------------------
11+
12+import Streamly.Benchmark.Data.Serialize.TH (genLargeRecord)
13+
14+import qualified Streamly.Internal.Data.MutByteArray as Serialize
15+
16+--------------------------------------------------------------------------------
17+-- Code
18+--------------------------------------------------------------------------------
19+
20+$(genLargeRecord "RecNonCompatible" 50)
21+$(Serialize.deriveSerialize [d|instance Serialize.Serialize RecNonCompatible|])
benchmark/Streamly/Benchmark/Data/Serialize/TH.hsnew file+76
@@ -0,0 +1,76 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Streamly.Benchmark.Data.Serialize.TH (genLargeRecord) where
4+
5+--------------------------------------------------------------------------------
6+-- Imports
7+--------------------------------------------------------------------------------
8+
9+import Language.Haskell.TH
10+
11+import Streamly.Internal.Data.MutByteArray (makeI)
12+import Control.DeepSeq (NFData(..))
13+
14+--------------------------------------------------------------------------------
15+-- Large Record
16+--------------------------------------------------------------------------------
17+
18+genLargeRecord :: String -> Int -> Q [Dec]
19+genLargeRecord tyName numFields =
20+ sequence
21+ [ dataD
22+ (pure [])
23+ (mkName tyName)
24+ []
25+ Nothing
26+ [mkCon tyName]
27+ [derivClause Nothing [conT ''Eq, conT ''Show]]
28+ , mkValueSigDec
29+ , mkValueDec
30+ , nfDataInstance tyName
31+ ]
32+
33+ where
34+
35+ fieldTypeChoices = [conT ''()]
36+ chooseCycle i xs = xs !! (i `mod` length xs)
37+ nfDataInstance nm =
38+ instanceD
39+ (pure [])
40+ (appT (conT ''NFData) (conT (mkName nm)))
41+ [ funD
42+ 'rnf
43+ [ clause
44+ [ conP
45+ (mkName nm)
46+ (varP . makeI <$> [0 .. (numFields - 1)])
47+ ]
48+ (normalB
49+ (foldl
50+ (\b a -> [|rnf $(b) `seq` rnf $(a)|])
51+ [|()|]
52+ (varE . makeI <$> [0 .. (numFields - 1)])))
53+ []
54+ ]
55+ ]
56+ valueName = mkName $ "val" ++ tyName
57+ mkValueSigDec = sigD valueName [t|$(conT (mkName tyName))|]
58+ mkValueDec =
59+ funD
60+ valueName
61+ [ clause
62+ []
63+ (normalB
64+ (foldl
65+ (\b a -> [|$(b) $(a)|])
66+ (conE (mkName tyName))
67+ (conE '() <$ [0 .. (numFields - 1)])))
68+ []
69+ ]
70+ mkCon nm = recC (mkName nm) (mkField <$> [0 .. (numFields - 1)])
71+ mkField i =
72+ varBangType
73+ (mkName ("field" ++ show i))
74+ (bangType
75+ (bang noSourceUnpackedness noSourceStrictness)
76+ (chooseCycle i fieldTypeChoices))
benchmark/Streamly/Benchmark/Data/Stream/Common.hs+3−1
@@ -71,7 +71,9 @@ module Stream.Common
7171 )
7272where
7373
74+#if !MIN_VERSION_base(4,18,0)
7475import Control.Applicative (liftA2)
76+#endif
7577import Control.DeepSeq (NFData)
7678import Control.Exception (try)
7779import GHC.Exception (ErrorCall)
@@ -102,7 +104,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
102104#endif
103105#endif
104106
105−import Gauge
107+import Test.Tasty.Bench
106108import Prelude hiding (Foldable(..), mapM, replicate)
107109
108110#ifdef USE_STREAMK
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentCommon.hs+3−3
@@ -14,14 +14,14 @@ where
1414import Stream.Common
1515 (composeN, benchIO, benchIOSink, benchIOSrc, sourceUnfoldrM)
1616import Streamly.Data.Stream (Stream)
17−import Streamly.Internal.Data.Stream.Concurrent (MonadAsync, Config)
17+import Streamly.Internal.Data.Stream.Prelude (MonadAsync, Config)
1818
1919import qualified Data.List as List
2020import qualified Streamly.Data.Fold as Fold
2121import qualified Streamly.Data.Stream as Stream
22−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
22+import qualified Streamly.Internal.Data.Stream.Prelude as Async
2323
24−import Gauge
24+import Test.Tasty.Bench
2525import Prelude hiding (mapM)
2626import Streamly.Benchmark.Common
2727
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentEager.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
12+import qualified Streamly.Internal.Data.Stream.Prelude as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentEager"
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentInterleaved.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
12+import qualified Streamly.Internal.Data.Stream.Prelude as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentInterleaved"
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentOrdered.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
12+import qualified Streamly.Internal.Data.Stream.Prelude as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentOrdered"
benchmark/Streamly/Benchmark/Data/Stream/Eliminate.hs+3−1
@@ -57,7 +57,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
5757#endif
5858#endif
5959
60−import Gauge
60+import Test.Tasty.Bench
6161import Streamly.Benchmark.Common
6262import Prelude hiding (length, sum, or, and, any, all, notElem, elem, (!!),
6363 lookup, repeat, minimum, maximum, product, last, mapM_, init)
@@ -132,12 +132,14 @@ foldableAny :: Int -> Int -> Bool
132132foldableAny value n =
133133 Prelude.any (> (value + 1)) (sourceUnfoldr value n :: Stream Identity Int)
134134
135+{- HLINT ignore "Use all"-}
135136{-# INLINE foldableAnd #-}
136137foldableAnd :: Int -> Int -> Bool
137138foldableAnd value n =
138139 Prelude.and $ fmap
139140 (<= (value + 1)) (sourceUnfoldr value n :: Stream Identity Int)
140141
142+{- HLINT ignore "Use any"-}
141143{-# INLINE foldableOr #-}
142144foldableOr :: Int -> Int -> Bool
143145foldableOr value n =
benchmark/Streamly/Benchmark/Data/Stream/Exceptions.hs+12−3
@@ -36,21 +36,22 @@ import System.IO (Handle, hClose, hPutChar)
3636import qualified Streamly.FileSystem.Handle as FH
3737import qualified Streamly.Internal.FileSystem.Handle as IFH
3838import qualified Streamly.Internal.Data.Unfold as IUF
39−import qualified Streamly.Internal.Data.Unfold.Exception as IUF
39+import qualified Streamly.Internal.Data.Unfold.Prelude as IUF
4040
4141#ifdef USE_PRELUDE
4242import qualified Streamly.Internal.Data.Stream.IsStream as Stream
4343#else
4444import qualified Streamly.Internal.Data.Stream as Stream
45−import qualified Streamly.Internal.Data.Stream.Exception.Lifted as Stream
45+import qualified Streamly.Internal.Data.Stream.Prelude as Stream
4646#endif
4747
48−import Gauge hiding (env)
48+import Test.Tasty.Bench hiding (env)
4949import Prelude hiding (last, length)
5050import Streamly.Benchmark.Common
5151import Streamly.Benchmark.Common.Handle
5252
5353#ifdef INSPECTION
54+import Control.Monad.Catch (MonadCatch)
5455import Test.Inspection
5556
5657import qualified Streamly.Internal.Data.Stream as D
@@ -286,8 +287,12 @@ readChunksOnException inh devNull =
286287 in IUF.fold (IFH.writeChunks devNull) readEx inh
287288
288289#ifdef INSPECTION
290+#if __GLASGOW_HASKELL__ >= 906
291+inspect $ hasNoTypeClassesExcept 'readChunksOnException [''MonadCatch]
292+#else
289293inspect $ hasNoTypeClasses 'readChunksOnException
290294#endif
295+#endif
291296
292297-- | Send the file contents to /dev/null with exception handling
293298readChunksBracket_ :: Handle -> Handle -> IO ()
@@ -296,8 +301,12 @@ readChunksBracket_ inh devNull =
296301 in IUF.fold (IFH.writeChunks devNull) readEx inh
297302
298303#ifdef INSPECTION
304+#if __GLASGOW_HASKELL__ >= 906
305+inspect $ hasNoTypeClassesExcept 'readChunksBracket_ [''MonadCatch]
306+#else
299307inspect $ hasNoTypeClasses 'readChunksBracket_
300308#endif
309+#endif
301310
302311readChunksBracket :: Handle -> Handle -> IO ()
303312readChunksBracket inh devNull =
benchmark/Streamly/Benchmark/Data/Stream/Expand.hs+9−1
@@ -54,7 +54,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
5454#endif
5555#endif
5656
57−import Gauge
57+import Test.Tasty.Bench
5858import Stream.Common
5959import Streamly.Benchmark.Common
6060import Prelude hiding (concatMap)
@@ -198,7 +198,11 @@ concatMapPure outer inner n =
198198 (fromStream $ sourceUnfoldr outer n)
199199
200200#ifdef INSPECTION
201+#if __GLASGOW_HASKELL__ >= 906
202+inspect $ hasNoTypeClassesExcept 'concatMapPure [''Applicative]
203+#else
201204inspect $ hasNoTypeClasses 'concatMapPure
205+#endif
202206inspect $ 'concatMapPure `hasNoType` ''SPEC
203207#endif
204208
@@ -211,7 +215,11 @@ concatMapRepl outer inner n =
211215 (fromStream . S.replicate inner) (fromStream $ sourceUnfoldrM outer n)
212216
213217#ifdef INSPECTION
218+#if __GLASGOW_HASKELL__ >= 906
219+inspect $ hasNoTypeClassesExcept 'concatMapRepl [''Applicative]
220+#else
214221inspect $ hasNoTypeClasses 'concatMapRepl
222+#endif
215223inspect $ 'concatMapRepl `hasNoType` ''SPEC
216224#endif
217225
benchmark/Streamly/Benchmark/Data/Stream/Generate.hs+1−1
@@ -40,7 +40,7 @@ import qualified Stream.Common as Common
4040#endif
4141#endif
4242
43−import Gauge
43+import Test.Tasty.Bench
4444import Streamly.Benchmark.Common
4545import qualified Prelude
4646
benchmark/Streamly/Benchmark/Data/Stream/Lift.hs+1−1
@@ -37,7 +37,7 @@ import qualified Control.Monad.State.Strict as State
3737#endif
3838#endif
3939
40−import Gauge
40+import Test.Tasty.Bench
4141import Streamly.Benchmark.Common
4242import Prelude hiding (reverse, tail)
4343
benchmark/Streamly/Benchmark/Data/Stream/Reduce.hs+2−2
@@ -48,7 +48,7 @@ import Streamly.Internal.Data.Stream (Stream)
4848import qualified Streamly.Internal.Data.Stream as S
4949#ifndef USE_STREAMLY_CORE
5050import qualified Streamly.Data.Stream.Prelude as S
51−import qualified Streamly.Internal.Data.Stream.Time as S
51+import qualified Streamly.Internal.Data.Stream.Prelude as S
5252#endif
5353
5454#ifdef USE_STREAMK
@@ -61,7 +61,7 @@ import qualified Streamly.Internal.Data.Stream as K
6161
6262#endif
6363
64−import Gauge
64+import Test.Tasty.Bench
6565import Streamly.Benchmark.Common
6666import Stream.Common
6767import Prelude hiding (reverse, tail)
benchmark/Streamly/Benchmark/Data/Stream/Split.hs+2−2
@@ -35,7 +35,7 @@ import qualified Streamly.Internal.FileSystem.Handle as IFH
3535import qualified Streamly.Internal.Unicode.Stream as IUS
3636import qualified Streamly.Prelude as S
3737
38−import Gauge hiding (env)
38+import Test.Tasty.Bench hiding (env)
3939import Prelude hiding (last, length)
4040import Streamly.Benchmark.Common
4141import Streamly.Benchmark.Common.Handle
@@ -43,7 +43,7 @@ import Streamly.Benchmark.Common.Handle
4343#ifdef INSPECTION
4444import Streamly.Internal.Data.Stream (Step(..))
4545
46−import qualified Streamly.Internal.Data.MutArray.Type as MA
46+import qualified Streamly.Internal.Data.MutArray as MA
4747import qualified Streamly.Internal.Data.Unfold as IUF
4848
4949import Test.Inspection
benchmark/Streamly/Benchmark/Data/Stream/StreamD.hs+4−2
@@ -22,10 +22,12 @@
2222
2323module Main (main) where
2424
25+#if !MIN_VERSION_base(4,18,0)
2526import Control.Applicative (liftA2)
27+#endif
2628import Control.Monad (when)
2729import Data.Maybe (isJust)
28−import Gauge (bench, nfIO, bgroup, Benchmark, defaultMain)
30+import Test.Tasty.Bench (bench, nfIO, bgroup, Benchmark, defaultMain)
2931import System.Random (randomRIO)
3032import Prelude hiding (tail, mapM_, foldl, last, map, mapM, concatMap, zip)
3133
@@ -461,7 +463,7 @@ inspect $ 'unfoldManyRepl `hasNoType` ''SPEC
461463-------------------------------------------------------------------------------
462464
463465moduleName :: String
464−moduleName = "Data.Stream.StreamD"
466+moduleName = "Data.StreamD"
465467
466468o_1_space :: [Benchmark]
467469o_1_space =
benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs+4−3
@@ -21,7 +21,9 @@
2121
2222module Main (main) where
2323
24+#if !MIN_VERSION_base(4,18,0)
2425import Control.Applicative (liftA2)
26+#endif
2527import Control.Monad (when)
2628import Data.Maybe (isJust)
2729import System.Random (randomRIO)
@@ -33,10 +35,9 @@ import Prelude hiding
3335import qualified Prelude as P
3436import qualified Data.List as List
3537
36−import qualified Streamly.Internal.Data.Stream.StreamK.Type as S
3738import qualified Streamly.Internal.Data.StreamK as S
3839
39−import Gauge (bench, nfIO, bgroup, Benchmark, defaultMain)
40+import Test.Tasty.Bench (bench, nfIO, bgroup, Benchmark, defaultMain)
4041
4142import Streamly.Benchmark.Common
4243
@@ -633,7 +634,7 @@ filterAllInNestedList str = do
633634-------------------------------------------------------------------------------
634635
635636moduleName :: String
636−moduleName = "Data.Stream.StreamK"
637+moduleName = "Data.StreamK"
637638
638639o_1_space_generation :: Int -> Benchmark
639640o_1_space_generation streamLen =
benchmark/Streamly/Benchmark/Data/Stream/StreamKAlt.hs+3−3
@@ -18,12 +18,12 @@ import Prelude hiding ()
1818-- import qualified Prelude as P
1919-- import qualified Data.List as List
2020
21−import qualified Streamly.Internal.Data.Stream.StreamK.Alt as S
21+import qualified Streamly.Internal.Data.StreamK.Alt as S
2222-- import qualified Streamly.Internal.Data.Stream.Common as SP
2323-- import qualified Streamly.Internal.Data.SVar as S
2424
2525import Streamly.Benchmark.Common
26−import Gauge (bgroup, Benchmark, defaultMain)
26+import Test.Tasty.Bench (bgroup, Benchmark, defaultMain)
2727
2828value :: Int
2929value = 100000
@@ -452,4 +452,4 @@ o_1_space =
452452 ]
453453
454454main :: IO ()
455−main = defaultMain $ concat [o_1_space]
455+main = defaultMain [o_1_space]
benchmark/Streamly/Benchmark/Data/Stream/ToStreamK.hs+3−4
@@ -1,5 +1,5 @@
11-- |
2−-- Module : Streamly.Benchmark.Data.Stream.StreamK
2+-- Module : Streamly.Benchmark.Data.StreamK
33-- Copyright : (c) 2018 Composewell Technologies
44--
55-- License : BSD3
@@ -33,12 +33,11 @@ import Prelude hiding
3333import qualified Prelude as P
3434-- import qualified Data.List as List
3535
36−import qualified Streamly.Internal.Data.Stream.StreamK.Type as S
3736import qualified Streamly.Internal.Data.StreamK as S
3837import qualified Streamly.Internal.Data.Stream as D
3938import qualified Streamly.Internal.Data.Fold as Fold
4039
41−import Gauge (bgroup, Benchmark, defaultMain)
40+import Test.Tasty.Bench (bgroup, Benchmark, defaultMain)
4241
4342import Streamly.Benchmark.Common
4443
@@ -121,7 +120,7 @@ fromFoldable streamLen n = S.fromFoldable [n..n+streamLen]
121120{-# INLINE fromFoldableM #-}
122121fromFoldableM :: Monad m => Int -> Int -> Stream m Int
123122fromFoldableM streamLen n =
124− Prelude.foldr S.consM S.nil (Prelude.fmap return [n..n+streamLen])
123+ Prelude.foldr (S.consM . return) S.nil [n .. n + streamLen]
125124
126125{-
127126{-# INLINABLE concatMapFoldableWith #-}
benchmark/Streamly/Benchmark/Data/Stream/Transform.hs+2−2
@@ -46,7 +46,7 @@ import Streamly.Internal.Data.Time.Units
4646import Streamly.Internal.Data.Stream (Stream)
4747import qualified Streamly.Internal.Data.Stream as Stream
4848#ifndef USE_STREAMLY_CORE
49−import qualified Streamly.Internal.Data.Stream.Time as Stream
49+import qualified Streamly.Internal.Data.Stream.Prelude as Stream
5050#endif
5151#ifdef USE_STREAMK
5252import Control.DeepSeq (NFData(..))
@@ -58,7 +58,7 @@ import qualified Streamly.Internal.Data.StreamK as StreamK
5858#endif
5959#endif
6060
61−import Gauge
61+import Test.Tasty.Bench
6262import Stream.Common hiding (scanl')
6363import Streamly.Benchmark.Common
6464import Prelude hiding (sequence, mapM)
benchmark/Streamly/Benchmark/Data/Unbox.hsdeleted−102
@@ -1,102 +0,0 @@
1−{-# LANGUAGE TemplateHaskell #-}
2−
3−-- |
4−-- Module : Streamly.Benchmark.Data.Unbox
5−-- Copyright : (c) 2023 Composewell
6−-- License : BSD-3-Clause
7−-- Maintainer : streamly@composewell.com
8−
9−module Main (main) where
10−
11−-------------------------------------------------------------------------------
12−-- Imports
13−-------------------------------------------------------------------------------
14−
15−import Control.DeepSeq (NFData(..))
16−import Control.Exception (assert)
17−import Control.Monad (replicateM_)
18−import Data.Proxy (Proxy(..))
19−import GHC.Generics (Generic)
20−import System.Random (randomRIO)
21−
22−import Streamly.Internal.Data.Unbox
23−import Streamly.Internal.Data.Unbox.TH
24−
25−import Gauge
26−import Streamly.Benchmark.Common
27−
28−-------------------------------------------------------------------------------
29−-- Types
30−-------------------------------------------------------------------------------
31−
32−data CustomDT1 a b
33− = CDT1C1
34− | CDT1C2 a
35− | CDT1C3 a b
36− deriving (Generic, Show, Eq)
37−type CustomDT1_ = CustomDT1 Int Bool
38−
39−instance (Unbox a, Unbox b) => Unbox (CustomDT1 a b)
40−
41−data CustomDT2 a b
42− = CDT2C1
43− | CDT2C2 a
44− | CDT2C3 a b
45− deriving (Show, Eq)
46−type CustomDT2_ = CustomDT2 Int Bool
47−
48−$(deriveUnbox ''CustomDT2)
49−
50−-------------------------------------------------------------------------------
51−-- Helpers
52−-------------------------------------------------------------------------------
53−
54−serializeDeserializeTimes :: forall a. (Eq a, Unbox a) => a -> Int -> IO ()
55−serializeDeserializeTimes val times = do
56− arr <- newBytes (sizeOf (Proxy :: Proxy a))
57− replicateM_ times $ do
58− pokeByteIndex 0 arr val
59− val1 <- peekByteIndex 0 arr
60− assert (val == val1) (pure ())
61−
62−benchSink :: NFData b => String -> Int -> (Int -> IO b) -> Benchmark
63−benchSink name times f = bench name (nfIO (randomRIO (times, times) >>= f))
64−
65−-------------------------------------------------------------------------------
66−-- Benchmarks
67−-------------------------------------------------------------------------------
68−
69−allBenchmarks :: Int -> [Benchmark]
70−allBenchmarks times =
71− [ benchSink
72− "serializeDeserializeTimes CDT1C1 (Generic)"
73− times
74− (serializeDeserializeTimes (CDT1C1 :: CustomDT1_))
75− , benchSink
76− "serializeDeserializeTimes CDT2C1 (TH)"
77− times
78− (serializeDeserializeTimes (CDT2C1 :: CustomDT2_))
79− , benchSink
80− "serializeDeserializeTimes CDT1C2 (Generic)"
81− times
82− (serializeDeserializeTimes ((CDT1C2 (5 :: Int)) :: CustomDT1_))
83− , benchSink
84− "serializeDeserializeTimes CDT2C2 (TH)"
85− times
86− (serializeDeserializeTimes ((CDT2C2 (5 :: Int)) :: CustomDT2_))
87− , benchSink
88− "serializeDeserializeTimes CDT1C3 (Generic)"
89− times
90− (serializeDeserializeTimes ((CDT1C3 (5 :: Int) True) :: CustomDT1_))
91− , benchSink
92− "serializeDeserializeTimes CDT2C3 (TH)"
93− times
94− (serializeDeserializeTimes ((CDT2C3 (5 :: Int) True) :: CustomDT2_))
95− ]
96−
97−-------------------------------------------------------------------------------
98−-- Driver
99−-------------------------------------------------------------------------------
100−
101−main :: IO ()
102−main = runWithCLIOpts defaultStreamSize allBenchmarks
benchmark/Streamly/Benchmark/Data/Unfold.hs+11−2
@@ -31,17 +31,18 @@ import qualified Prelude
3131import qualified Streamly.FileSystem.Handle as FH
3232import qualified Streamly.Internal.Data.Fold as FL
3333import qualified Streamly.Internal.Data.Unfold as UF
34−import qualified Streamly.Internal.Data.Unfold.Exception as UF
34+import qualified Streamly.Internal.Data.Unfold.Prelude as UF
3535import qualified Streamly.Internal.Data.Stream as S
3636import qualified Streamly.Internal.Data.Stream as D
3737import qualified Streamly.Internal.Data.StreamK as K
3838
39−import Gauge hiding (env)
39+import Test.Tasty.Bench hiding (env)
4040import Prelude hiding (take, filter, zipWith, map, mapM, takeWhile)
4141import Streamly.Benchmark.Common
4242import Streamly.Benchmark.Common.Handle
4343
4444#ifdef INSPECTION
45+import Control.Monad.Catch (MonadCatch)
4546import Test.Inspection
4647#endif
4748
@@ -776,7 +777,11 @@ readWriteFinally_Unfold inh devNull =
776777 in S.fold (FH.write devNull) $ S.unfold readEx inh
777778
778779#ifdef INSPECTION
780+#if __GLASGOW_HASKELL__ >= 906
781+inspect $ hasNoTypeClassesExcept 'readWriteFinally_Unfold [''MonadCatch]
782+#else
779783inspect $ hasNoTypeClasses 'readWriteFinally_Unfold
784+#endif
780785-- inspect $ 'readWriteFinallyUnfold `hasNoType` ''Step
781786#endif
782787
@@ -792,7 +797,11 @@ readWriteBracket_Unfold inh devNull =
792797 in S.fold (FH.write devNull) $ S.unfold readEx inh
793798
794799#ifdef INSPECTION
800+#if __GLASGOW_HASKELL__ >= 906
801+inspect $ hasNoTypeClassesExcept 'readWriteBracket_Unfold [''MonadCatch]
802+#else
795803inspect $ hasNoTypeClasses 'readWriteBracket_Unfold
804+#endif
796805-- inspect $ 'readWriteBracketUnfold `hasNoType` ''Step
797806#endif
798807
benchmark/Streamly/Benchmark/FileSystem/Handle.hs+1−1
@@ -22,7 +22,7 @@ import Streamly.Benchmark.Common.Handle (mkHandleBenchEnv)
2222import qualified Handle.ReadWrite as RW
2323import qualified Handle.Read as RO
2424
25−import Gauge hiding (env)
25+import Test.Tasty.Bench hiding (env)
2626import Prelude hiding (last, length)
2727import Streamly.Benchmark.Common
2828
benchmark/Streamly/Benchmark/FileSystem/Handle/Read.hs+8−9
@@ -33,7 +33,6 @@ import qualified Streamly.Data.Stream as Stream
3333import qualified Streamly.Data.Fold as Fold
3434import qualified Streamly.FileSystem.Handle as FH
3535import qualified Streamly.Internal.Data.Array as A
36−import qualified Streamly.Internal.Data.Array.Type as AT
3736import qualified Streamly.Internal.Data.Fold as FL
3837import qualified Streamly.Internal.Data.Stream as IP
3938import qualified Streamly.Internal.FileSystem.Handle as IFH
@@ -41,14 +40,14 @@ import qualified Streamly.Internal.Unicode.Stream as IUS
4140import qualified Streamly.Data.Stream.Prelude as S
4241import qualified Streamly.Unicode.Stream as SS
4342
44−import Gauge hiding (env)
43+import Test.Tasty.Bench hiding (env)
4544import Prelude hiding (last, length)
4645import Streamly.Benchmark.Common.Handle
4746
4847#ifdef INSPECTION
4948import Streamly.Internal.Data.Stream (Step(..), FoldMany)
5049
51−import qualified Streamly.Internal.Data.MutArray.Type as MA
50+import qualified Streamly.Internal.Data.MutArray as MutArray
5251import qualified Streamly.Internal.Data.Stream as D
5352import qualified Streamly.Internal.Data.Unfold as IUF
5453
@@ -69,7 +68,7 @@ readLast = S.fold Fold.last . S.unfold FH.reader
6968inspect $ hasNoTypeClasses 'readLast
7069inspect $ 'readLast `hasNoType` ''Step -- S.unfold
7170inspect $ 'readLast `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
72−inspect $ 'readLast `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
71+inspect $ 'readLast `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
7372#endif
7473
7574-- assert that flattenArrays constructors are not present
@@ -81,7 +80,7 @@ readCountBytes = S.fold Fold.length . S.unfold FH.reader
8180inspect $ hasNoTypeClasses 'readCountBytes
8281inspect $ 'readCountBytes `hasNoType` ''Step -- S.unfold
8382inspect $ 'readCountBytes `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
84−inspect $ 'readCountBytes `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
83+inspect $ 'readCountBytes `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
8584#endif
8685
8786-- | Count the number of lines in a file.
@@ -96,7 +95,7 @@ readCountLines =
9695inspect $ hasNoTypeClasses 'readCountLines
9796inspect $ 'readCountLines `hasNoType` ''Step
9897inspect $ 'readCountLines `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
99−inspect $ 'readCountLines `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
98+inspect $ 'readCountLines `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
10099#endif
101100
102101-- | Count the number of words in a file.
@@ -120,7 +119,7 @@ readSumBytes = S.fold Fold.sum . S.unfold FH.reader
120119inspect $ hasNoTypeClasses 'readSumBytes
121120inspect $ 'readSumBytes `hasNoType` ''Step
122121inspect $ 'readSumBytes `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
123−inspect $ 'readSumBytes `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
122+inspect $ 'readSumBytes `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
124123#endif
125124
126125-- XXX When we mark this with INLINE and we have two benchmarks using S.drain
@@ -236,13 +235,13 @@ groupsOf :: Int -> Handle -> IO Int
236235groupsOf n inh =
237236 -- writeNUnsafe gives 2.5x boost here over writeN.
238237 S.fold Fold.length
239− $ IP.groupsOf n (AT.writeNUnsafe n) (S.unfold FH.reader inh)
238+ $ IP.groupsOf n (A.writeNUnsafe n) (S.unfold FH.reader inh)
240239
241240#ifdef INSPECTION
242241inspect $ hasNoTypeClasses 'groupsOf
243242inspect $ 'groupsOf `hasNoType` ''Step
244243inspect $ 'groupsOf `hasNoType` ''FoldMany
245−inspect $ 'groupsOf `hasNoType` ''AT.ArrayUnsafe -- AT.writeNUnsafe
244+inspect $ 'groupsOf `hasNoType` ''MutArray.ArrayUnsafe -- AT.writeNUnsafe
246245 -- FH.read/A.read
247246inspect $ 'groupsOf `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
248247#endif
benchmark/Streamly/Benchmark/FileSystem/Handle/ReadWrite.hs+9−10
@@ -34,7 +34,7 @@ import qualified Streamly.Internal.FileSystem.Handle as IFH
3434import qualified Streamly.Data.Array as A
3535import qualified Streamly.Data.Stream.Prelude as S
3636
37−import Gauge hiding (env)
37+import Test.Tasty.Bench hiding (env)
3838import Streamly.Benchmark.Common.Handle
3939
4040#ifdef INSPECTION
@@ -42,8 +42,7 @@ import Streamly.Internal.Data.Stream (Step(..))
4242
4343import qualified Streamly.Internal.Data.Stream as D
4444import qualified Streamly.Internal.Data.Tuple.Strict as Strict
45−import qualified Streamly.Internal.Data.Stream.MutChunked as MAS
46−import qualified Streamly.Internal.Data.Array.Type as AT
45+import qualified Streamly.Internal.Data.MutArray as MutArray
4746
4847import Test.Inspection
4948#endif
@@ -83,7 +82,7 @@ copyStream inh outh = S.fold (FH.write outh) (S.unfold FH.reader inh)
8382inspect $ hasNoTypeClasses 'copyStream
8483inspect $ 'copyStream `hasNoType` ''Step -- S.unfold
8584inspect $ 'copyStream `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
86−inspect $ 'copyStream `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
85+inspect $ 'copyStream `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
8786 -- FH.read/A.read
8887inspect $ 'copyStream `hasNoType` ''Strict.Tuple3' -- FH.write/chunksOf
8988#endif
@@ -109,8 +108,8 @@ readFromBytesNull inh devNull = IFH.putBytes devNull $ S.unfold FH.reader inh
109108#ifdef INSPECTION
110109inspect $ hasNoTypeClasses 'readFromBytesNull
111110inspect $ 'readFromBytesNull `hasNoType` ''Step
112−inspect $ 'readFromBytesNull `hasNoType` ''MAS.SpliceState
113−inspect $ 'readFromBytesNull `hasNoType` ''AT.ArrayUnsafe -- FH.fromBytes/S.chunksOf
111+inspect $ 'readFromBytesNull `hasNoType` ''MutArray.SpliceState
112+inspect $ 'readFromBytesNull `hasNoType` ''MutArray.ArrayUnsafe -- FH.fromBytes/S.chunksOf
114113inspect $ 'readFromBytesNull `hasNoType` ''D.FoldMany
115114#endif
116115
@@ -123,8 +122,8 @@ readWithFromBytesNull inh devNull =
123122#ifdef INSPECTION
124123inspect $ hasNoTypeClasses 'readWithFromBytesNull
125124inspect $ 'readWithFromBytesNull `hasNoType` ''Step
126−inspect $ 'readWithFromBytesNull `hasNoType` ''MAS.SpliceState
127−inspect $ 'readWithFromBytesNull `hasNoType` ''AT.ArrayUnsafe -- FH.fromBytes/S.chunksOf
125+inspect $ 'readWithFromBytesNull `hasNoType` ''MutArray.SpliceState
126+inspect $ 'readWithFromBytesNull `hasNoType` ''MutArray.ArrayUnsafe -- FH.fromBytes/S.chunksOf
128127inspect $ 'readWithFromBytesNull `hasNoType` ''D.FoldMany
129128#endif
130129
@@ -171,7 +170,7 @@ writeReadWith inh devNull = IUF.fold fld unf (defaultChunkSize, inh)
171170inspect $ hasNoTypeClasses 'writeReadWith
172171inspect $ 'writeReadWith `hasNoType` ''Step
173172inspect $ 'writeReadWith `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
174−inspect $ 'writeReadWith `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
173+inspect $ 'writeReadWith `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
175174 -- FH.read/A.read
176175#endif
177176
@@ -188,7 +187,7 @@ writeRead inh devNull = IUF.fold fld unf inh
188187inspect $ hasNoTypeClasses 'writeRead
189188inspect $ 'writeRead `hasNoType` ''Step
190189inspect $ 'writeRead `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
191−inspect $ 'writeRead `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
190+inspect $ 'writeRead `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
192191 -- FH.read/A.read
193192#endif
194193
benchmark/Streamly/Benchmark/Prelude/Adaptive.hs+1−1
@@ -10,7 +10,7 @@
1010import Control.Concurrent (threadDelay)
1111import Control.Monad (when)
1212import Control.Monad.IO.Class (liftIO)
13−import Gauge
13+import Test.Tasty.Bench
1414import Streamly.Prelude as S
1515import System.Random (randomRIO)
1616
benchmark/Streamly/Benchmark/Prelude/Ahead.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Gauge
18+import Test.Tasty.Bench
1919
2020moduleName :: String
2121moduleName = "Prelude.Ahead"
benchmark/Streamly/Benchmark/Prelude/Async.hs+1−1
@@ -16,7 +16,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream.Transform as Transform
1616import Streamly.Benchmark.Common
1717import Streamly.Benchmark.Prelude
1818
19−import Gauge
19+import Test.Tasty.Bench
2020
2121moduleName :: String
2222moduleName = "Prelude.Async"
benchmark/Streamly/Benchmark/Prelude/Concurrent.hs+1−1
@@ -15,7 +15,7 @@ import Streamly.Prelude
1515 , fromWAsync, parallel, fromParallel
1616 )
1717
18−import Gauge
18+import Test.Tasty.Bench
1919import qualified Streamly.Prelude as S
2020
2121-------------------------------------------------------------------------------
benchmark/Streamly/Benchmark/Prelude/Merge.hs+1−1
@@ -30,7 +30,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream as Stream
3030import Streamly.Benchmark.Common
3131import Streamly.Benchmark.Prelude
3232
33−import Gauge
33+import Test.Tasty.Bench
3434
3535#ifdef INSPECTION
3636import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Prelude/Parallel.hs+1−1
@@ -22,7 +22,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream as Internal
2222import Streamly.Benchmark.Common
2323import Streamly.Benchmark.Prelude
2424
25−import Gauge
25+import Test.Tasty.Bench
2626
2727moduleName :: String
2828moduleName = "Prelude.Parallel"
benchmark/Streamly/Benchmark/Prelude/Rate.hs+1−1
@@ -14,7 +14,7 @@ import qualified Streamly.Prelude as S
1414import Streamly.Benchmark.Common
1515import Streamly.Benchmark.Prelude
1616
17−import Gauge
17+import Test.Tasty.Bench
1818
1919moduleName :: String
2020moduleName = "Prelude.Rate"
benchmark/Streamly/Benchmark/Prelude/WAsync.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Gauge
18+import Test.Tasty.Bench
1919
2020moduleName :: String
2121moduleName = "Prelude.WAsync"
benchmark/Streamly/Benchmark/Prelude/WSerial.hs+1−1
@@ -29,7 +29,7 @@ import qualified Streamly.Internal.Data.Unfold as UF
2929import Streamly.Benchmark.Common
3030import Streamly.Benchmark.Prelude
3131
32−import Gauge
32+import Test.Tasty.Bench
3333
3434#ifdef INSPECTION
3535import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Prelude/ZipAsync.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Gauge
18+import Test.Tasty.Bench
1919
2020moduleName :: String
2121moduleName = "Prelude.ZipAsync"
benchmark/Streamly/Benchmark/Prelude/ZipSerial.hs+1−1
@@ -29,7 +29,7 @@ import qualified Streamly.Prelude as S
2929import Streamly.Benchmark.Common
3030import Streamly.Benchmark.Prelude hiding (sourceUnfoldrM)
3131
32−import Gauge
32+import Test.Tasty.Bench
3333
3434#ifdef INSPECTION
3535import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Unicode/Char.hs+1−1
@@ -16,7 +16,7 @@ import Control.DeepSeq (NFData(..))
1616import Streamly.Internal.Data.Array (Array)
1717import System.FilePath (dropExtensions, takeFileName)
1818import System.FilePath.Posix ((</>))
19−import Gauge.Main (Benchmark, bench, bgroup, defaultMain, env, nfIO)
19+import Test.Tasty.Bench (Benchmark, bench, bgroup, defaultMain, env, nfIO)
2020import Streamly.Internal.Unicode.Char
2121 ( NormalizationMode(NFC, NFD, NFKC, NFKD)
2222 , normalize
benchmark/Streamly/Benchmark/Unicode/Parser.hs+1−1
@@ -31,7 +31,7 @@ import qualified Streamly.Data.Array as Array
3131import qualified Streamly.Data.Stream as Stream
3232import qualified Streamly.Internal.Unicode.Parser as PRU
3333
34−import Gauge hiding (env)
34+import Test.Tasty.Bench hiding (env)
3535import Streamly.Benchmark.Common
3636
3737runParser :: Int -> (Stream IO Char -> IO a) -> IO ()
benchmark/Streamly/Benchmark/Unicode/Stream.hs+26−8
@@ -11,6 +11,11 @@
1111{-# LANGUAGE CPP #-}
1212{-# LANGUAGE ScopedTypeVariables #-}
1313
14+#undef FUSION_CHECK
15+#ifdef FUSION_CHECK
16+{-# OPTIONS_GHC -ddump-simpl -ddump-to-file -dsuppress-all #-}
17+#endif
18+
1419#ifdef __HADDOCK_VERSION__
1520#undef INSPECTION
1621#endif
@@ -33,16 +38,17 @@ import qualified Streamly.Internal.FileSystem.Handle as Handle
3338import qualified Streamly.Internal.Unicode.Array as UnicodeArr
3439import qualified Streamly.Internal.Unicode.Stream as Unicode
3540
36−import Gauge hiding (env)
41+import Test.Tasty.Bench hiding (env)
3742import Streamly.Benchmark.Common
3843import Streamly.Benchmark.Common.Handle
3944
4045#ifdef INSPECTION
41−import Streamly.Internal.Data.Unbox (Unbox)
42−import Streamly.Internal.Data.Stream.StreamD (Step(..))
43−import qualified Streamly.Internal.Data.Fold.Type as Fold
46+import Streamly.Internal.Data.MutByteArray (Unbox)
47+import Streamly.Internal.Data.Stream (Step(..))
48+import qualified Streamly.Internal.Data.Array as Array
49+import qualified Streamly.Internal.Data.MutArray as MutArray
50+import qualified Streamly.Internal.Data.Fold as Fold
4451import qualified Streamly.Internal.Data.Tuple.Strict as Strict
45−import qualified Streamly.Internal.Data.Array.Type as Array
4652
4753import Test.Inspection
4854#endif
@@ -218,7 +224,7 @@ inspect $ 'copyStreamLatin1' `hasNoType` ''Step
218224inspect $ 'copyStreamLatin1' `hasNoType` ''Unfold.ConcatState -- Handle.read/UF.many
219225
220226inspect $ 'copyStreamLatin1' `hasNoType` ''Fold.Step
221−inspect $ 'copyStreamLatin1' `hasNoType` ''Array.ArrayUnsafe -- Handle.write/writeNUnsafe
227+inspect $ 'copyStreamLatin1' `hasNoType` ''MutArray.ArrayUnsafe -- Handle.write/writeNUnsafe
222228 -- Handle.read/Array.read
223229inspect $ 'copyStreamLatin1' `hasNoType` ''Strict.Tuple3' -- Handle.write/chunksOf
224230#endif
@@ -239,7 +245,7 @@ inspect $ 'copyStreamLatin1 `hasNoType` ''Unfold.ConcatState -- Handle.read/UF.m
239245
240246inspect $ 'copyStreamLatin1 `hasNoType` ''Fold.ManyState
241247inspect $ 'copyStreamLatin1 `hasNoType` ''Fold.Step
242−inspect $ 'copyStreamLatin1 `hasNoType` ''Array.ArrayUnsafe -- Handle.write/writeNUnsafe
248+inspect $ 'copyStreamLatin1 `hasNoType` ''MutArray.ArrayUnsafe -- Handle.write/writeNUnsafe
243249 -- Handle.read/Array.read
244250inspect $ 'copyStreamLatin1 `hasNoType` ''Strict.Tuple3' -- Handle.write/chunksOf
245251#endif
@@ -280,7 +286,8 @@ _copyStreamUtf8'Fold :: Handle -> Handle -> IO ()
280286_copyStreamUtf8'Fold inh outh =
281287 Stream.fold (Handle.write outh)
282288 $ Unicode.encodeUtf8
283− $ Stream.foldMany Unicode.writeCharUtf8'
289+ $ Stream.catRights
290+ $ Stream.parseMany Unicode.writeCharUtf8'
284291 $ Stream.unfold Handle.reader inh
285292
286293{-# NOINLINE _copyStreamUtf8Parser #-}
@@ -317,6 +324,7 @@ o_1_space_decode_encode_read env =
317324
318325main :: IO ()
319326main = do
327+#ifndef FUSION_CHECK
320328 env <- mkHandleBenchEnv
321329 defaultMain (allBenchmarks env)
322330
@@ -329,3 +337,13 @@ main = do
329337 , o_1_space_decode_encode_read env
330338 ]
331339 ]
340+#else
341+ -- Enable FUSION_CHECK macro at the beginning of the file
342+ -- Enable one benchmark below, and run the benchmark
343+ -- Check the .dump-simpl output
344+ env <- mkHandleBenchEnv
345+ let mkHandles (RefHandles {bigInH = inh, outputH = outh}) = Handles inh outh
346+ (Handles inh outh) <- getHandles env mkHandles
347+ copyStreamLatin1' inh outh
348+ return ()
349+#endif
benchmark/Streamly/Benchmark/Unicode/Utf8.hs+1−1
@@ -13,7 +13,7 @@
1313import Data.Char (chr)
1414import Streamly.Internal.Unicode.Utf8 (Utf8)
1515
16−import Gauge hiding (env)
16+import Test.Tasty.Bench hiding (env)
1717import Streamly.Benchmark.Common
1818
1919import qualified Streamly.Internal.Unicode.Utf8 as Utf8
benchmark/bench-runner/Main.hs+34−6
@@ -16,9 +16,11 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
1616 benchName = drop 4 benchName0
1717
1818 general
19− | "o-1-sp" `isInfixOf` benchName = "-K36K -M16M"
19+ -- GHC-9.6 requires 32M heap vs 16M for GHC-9.4
20+ | "o-1-sp" `isInfixOf` benchName = "-K36K -M32M"
2021 | "o-n-h" `isInfixOf` benchName = "-K36K -M32M"
21− | "o-n-st" `isInfixOf` benchName = "-K1M -M16M"
22+ -- GHC-9.6 requires 32M heap vs 16M for GHC-9.4
23+ | "o-n-st" `isInfixOf` benchName = "-K1M -M32M"
2224 | "o-n-sp" `isInfixOf` benchName = "-K1M -M32M"
2325 | otherwise = ""
2426
@@ -27,6 +29,17 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
2729 | otherwise = ""
2830
2931 benchSpecific
32+ -- GHC-9.6 requires 64M, earlier it was 16M
33+ | "Data.Fold/o-n-heap.key-value.toHashMapIO (max buckets) sum" == benchName =
34+ "-M64M"
35+
36+ -- This is required only for the --long case because we allocate all
37+ -- the arrays upfront. depends on the size of the stream.
38+ | "Data.Parser/o-1-space" `isPrefixOf` benchName =
39+ "-M128M"
40+
41+ -----------------------------------------------------------------------
42+
3043 | "Prelude.Parallel/o-n-heap.mapping.mapM" == benchName = "-M256M"
3144 | "Prelude.Parallel/o-n-heap.monad-outer-product."
3245 `isPrefixOf` benchName = "-M256M"
@@ -57,12 +70,18 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
5770 -- GHC 9.4.4 requires 4M
5871 | "Data.Stream.StreamDK/o-n-space.traversable."
5972 `isPrefixOf` benchName = "-K4M"
73+ -- GHC-9.6 requires 64M, earlier it was 32M
74+ | "Data.Stream.StreamDK/o-n-heap.buffered.readsPrec pure streams" == benchName =
75+ "-M64M"
76+ -- GHC-9.6 requires 64M, earlier it was 32M
77+ | "Data.Stream.StreamDK/o-n-heap.buffered.showPrec Haskell lists" == benchName =
78+ "-M64M"
6079
6180 -----------------------------------------------------------------------
6281
63− | "Data.Stream.StreamD/o-n-space.elimination.toList" == benchName =
82+ | "Data.StreamD/o-n-space.elimination.toList" == benchName =
6483 "-K2M"
65− | "Data.Stream.StreamK/o-n-space.elimination.toList" == benchName =
84+ | "Data.StreamK/o-n-space.elimination.toList" == benchName =
6685 "-K2M"
6786
6887 -----------------------------------------------------------------------
@@ -74,6 +93,13 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
7493 | "Data.Stream/o-n-space.iterated."
7594 `isPrefixOf` benchName = "-K4M"
7695
96+ -- GHC-9.6 requires 64M, earlier it was 32M
97+ | "Data.Stream/o-n-heap.buffered.showPrec Haskell lists" == benchName =
98+ "-M64M"
99+ -- GHC-9.6 requires 64M, earlier it was 32M
100+ | "Data.Stream/o-n-heap.buffered.readsPrec pure streams" == benchName =
101+ "-M64M"
102+
77103 | "Data.Stream.ConcurrentEager/o-n-heap.monad-outer-product.toNullAp"
78104 `isPrefixOf` benchName = "-M1024M"
79105 | "Data.Stream.ConcurrentEager/o-1-space."
@@ -86,10 +112,12 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
86112
87113 | "Data.Array" `isPrefixOf` benchName
88114 && "/o-1-space.generation.read" `isSuffixOf` benchName = "-M32M"
115+ -- XXX GHC 9.6 onwards needs 64M, earlier it was 32M
89116 | "Data.Array" `isPrefixOf` benchName
90− && "/o-1-space.generation.show" `isSuffixOf` benchName = "-M32M"
117+ && "/o-1-space.generation.show" `isSuffixOf` benchName = "-M64M"
118+ -- XXX GHC 9.6 onwards needs 64M, earlier it was 32M
91119 | "Data.Array.Generic/o-1-space.transformationX4.map"
92− `isPrefixOf` benchName = "-M32M"
120+ `isPrefixOf` benchName = "-M64M"
93121
94122 -- XXX For --long option, need to check why so much heap is required.
95123 -- Note, if we remove the chunked stream module we need to keep the
benchmark/lib/Streamly/Benchmark/Common.hs+1−1
@@ -56,7 +56,7 @@ import System.Random (randomRIO)
5656import qualified Streamly.Internal.Data.Fold as Fold
5757import qualified Streamly.Internal.Data.Stream as S
5858
59−import Gauge
59+import Test.Tasty.Bench
6060
6161-------------------------------------------------------------------------------
6262-- Benchmark Prefixes
benchmark/lib/Streamly/Benchmark/Common/Handle.hs+3−1
@@ -30,6 +30,8 @@ module Streamly.Benchmark.Common.Handle
3030 , isSpace
3131 , isSp
3232 , mkHandleBenchEnv
33+ , Handles(..)
34+ , getHandles
3335 )
3436where
3537
@@ -46,7 +48,7 @@ import System.Process (callCommand)
4648
4749import Data.IORef
4850import Prelude hiding (last, length)
49−import Gauge hiding (env)
51+import Test.Tasty.Bench hiding (env)
5052
5153
5254scratchDir :: String
benchmark/lib/Streamly/Benchmark/Prelude.hs+1−1
@@ -82,7 +82,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream.Type as IsStream
8282import qualified Streamly.Internal.Data.Pipe as Pipe
8383import qualified Streamly.Internal.Data.Stream.Serial as Serial
8484
85−import Gauge
85+import Test.Tasty.Bench
8686import Streamly.Internal.Data.Time.Units
8787
8888-- Common polymorphic stream APIs used across multiple stream modules
benchmark/streamly-benchmarks.cabal+64−40
@@ -54,11 +54,6 @@ flag opt
5454 manual: True
5555 default: True
5656
57−flag use-gauge
58− description: Use gauge instead of tasty-bench for benchmarking
59− manual: True
60− default: False
61−
6257flag use-streamly-core
6358 description: Benchmark only core package
6459 manual: True
@@ -145,6 +140,10 @@ common compile-options
145140 -Wno-missing-kind-signatures
146141 -Wno-operator-whitespace
147142
143+ if impl(ghc >= 9.8)
144+ ghc-options:
145+ -Wno-missing-role-annotations
146+
148147 if flag(has-llvm)
149148 ghc-options: -fllvm
150149
@@ -152,15 +151,17 @@ common compile-options
152151 ghc-options: -Wmissed-specialisations
153152 -Wall-missed-specialisations
154153
155− if flag(dev) || flag(debug)
156− ghc-options: -fno-ignore-asserts
157−
158154common optimization-options
159155 if flag(opt)
160156 ghc-options: -O2
161157 -fdicts-strict
162158 -fspec-constr-recursive=16
163159 -fmax-worker-args=16
160+
161+ -- For this to be effective it must come after the -O2 option
162+ if flag(dev) || flag(debug)
163+ ghc-options: -fno-ignore-asserts
164+
164165 if flag(fusion-plugin) && !impl(ghcjs) && !impl(ghc < 8.6)
165166 ghc-options: -fplugin Fusion.Plugin
166167 else
@@ -174,43 +175,34 @@ common bench-depends
174175 -- Core libraries shipped with ghc, the min and max
175176 -- constraints of these libraries should match with
176177 -- the GHC versions we support
177− base >= 4.9 && < 4.19
178− , deepseq >= 1.4.1 && < 1.5
178+ base >= 4.9 && < 4.20
179+ , deepseq >= 1.4.1 && < 1.6
179180 , mtl >= 2.2 && < 2.3.2
180181
181182 -- other libraries
183+ , exceptions >= 0.8 && < 0.11
182184 , random >= 1.0 && < 2.0
183185 , transformers >= 0.4 && < 0.7
184− , containers >= 0.5 && < 0.7
186+ , containers >= 0.5 && < 0.8
185187 , hashable >= 1.3 && < 1.5
186188 , unordered-containers >= 0.2 && < 0.3
187189 , process >= 1.4 && < 1.7
188190 , directory >= 1.2.2 && < 1.4
189− , filepath >= 1.4.1 && < 1.5
190− , ghc-prim >= 0.4 && < 0.11
191+ , filepath >= 1.4.1 && < 1.6
192+ , ghc-prim >= 0.4 && < 0.12
193+ , tasty-bench >= 0.3 && < 0.4
194+ , tasty >= 1.4.1 && < 1.6
195+ , streamly-core
191196
192− if flag(use-streamly-core)
193− build-depends: streamly-core
194− else
197+ if !flag(use-streamly-core)
195198 build-depends: streamly
196− , streamly-core
197−
198− if flag(use-gauge)
199− build-depends: gauge >= 0.2.4 && < 0.3
200− else
201− build-depends: tasty-bench >= 0.3 && < 0.4
202− , tasty >= 1.4.1 && < 1.5
203− mixins: tasty-bench
204− (Test.Tasty.Bench as Gauge
205− , Test.Tasty.Bench as Gauge.Main
206− )
207199
208200 if flag(fusion-plugin) && !impl(ghcjs) && !impl(ghc < 8.6)
209201 build-depends:
210202 fusion-plugin >= 0.2 && < 0.3
211203 if flag(inspection)
212− build-depends: template-haskell >= 2.14 && < 2.21
213− , inspection-testing >= 0.4 && < 0.5
204+ build-depends: template-haskell >= 2.14 && < 2.22
205+ , inspection-testing >= 0.4 && < 0.6
214206 -- Array uses a Storable constraint in dev build making several inspection
215207 -- tests fail
216208 if flag(dev) && flag(inspection)
@@ -360,7 +352,30 @@ benchmark Data.Unbox
360352 import: bench-options
361353 type: exitcode-stdio-1.0
362354 hs-source-dirs: .
363− main-is: Streamly/Benchmark/Data/Unbox.hs
355+ cpp-options: -DUSE_UNBOX
356+ main-is: Streamly/Benchmark/Data/Serialize.hs
357+
358+benchmark Data.Unbox.Derive.TH
359+ import: bench-options
360+ type: exitcode-stdio-1.0
361+ hs-source-dirs: .
362+ cpp-options: -DUSE_UNBOX
363+ cpp-options: -DUSE_TH
364+ main-is: Streamly/Benchmark/Data/Serialize.hs
365+
366+benchmark Data.Serialize
367+ import: bench-options
368+ type: exitcode-stdio-1.0
369+ hs-source-dirs: .
370+ cpp-options: -DUSE_TH
371+ main-is: Streamly/Benchmark/Data/Serialize.hs
372+ build-depends: QuickCheck, template-haskell
373+ other-modules:
374+ Streamly.Benchmark.Data.Serialize.TH
375+ Streamly.Benchmark.Data.Serialize.RecCompatible
376+ Streamly.Benchmark.Data.Serialize.RecNonCompatible
377+ if flag(limit-build-mem)
378+ ghc-options: +RTS -M1000M -RTS
364379
365380benchmark Data.Unfold
366381 import: bench-options
@@ -412,6 +427,7 @@ benchmark Data.ParserK
412427 import: bench-options
413428 type: exitcode-stdio-1.0
414429 hs-source-dirs: Streamly/Benchmark/Data
430+ cpp-options: -DBENCH_SINGULAR
415431 main-is: ParserK.hs
416432 if impl(ghcjs)
417433 buildable: False
@@ -419,17 +435,25 @@ benchmark Data.ParserK
419435 buildable: True
420436 build-depends: exceptions >= 0.8 && < 0.11
421437
422−-- Note: to use this we have to set DISABLE_FUSION in ParserD.hs so that
423−-- the rewrite rules are disabled. We can also use the "no-fusion" build
424−-- flag but we need to keep in mind that it disables fusion for streams
425−-- as well.
426−benchmark Data.FromParserK
438+benchmark Data.ParserK.Chunked
427439 import: bench-options
428440 type: exitcode-stdio-1.0
429441 hs-source-dirs: Streamly/Benchmark/Data
430− cpp-options: -DFROM_PARSERK
442+ cpp-options: -DBENCH_CHUNKED
431443 main-is: ParserK.hs
432− if flag(use-streamly-core) || impl(ghcjs)
444+ if impl(ghcjs)
445+ buildable: False
446+ else
447+ buildable: True
448+ build-depends: exceptions >= 0.8 && < 0.11
449+
450+benchmark Data.ParserK.Chunked.Generic
451+ import: bench-options
452+ type: exitcode-stdio-1.0
453+ hs-source-dirs: Streamly/Benchmark/Data
454+ cpp-options: -DBENCH_CHUNKED_GENERIC
455+ main-is: ParserK.hs
456+ if impl(ghcjs)
433457 buildable: False
434458 else
435459 buildable: True
@@ -439,7 +463,7 @@ benchmark Data.FromParserK
439463-- Raw Streams
440464-------------------------------------------------------------------------------
441465
442−benchmark Data.Stream.StreamD
466+benchmark Data.StreamD
443467 import: bench-options
444468 type: exitcode-stdio-1.0
445469 hs-source-dirs: Streamly/Benchmark/Data/Stream
@@ -447,7 +471,7 @@ benchmark Data.Stream.StreamD
447471 if impl(ghcjs)
448472 buildable: False
449473
450−benchmark Data.Stream.StreamK
474+benchmark Data.StreamK
451475 import: bench-options
452476 type: exitcode-stdio-1.0
453477 hs-source-dirs: Streamly/Benchmark/Data/Stream
@@ -463,7 +487,7 @@ benchmark Data.Stream.ToStreamK
463487 if !flag(dev) || impl(ghcjs)
464488 buildable: False
465489
466−benchmark Data.Stream.StreamK.Alt
490+benchmark Data.StreamK.Alt
467491 import: bench-options
468492 type: exitcode-stdio-1.0
469493 hs-source-dirs: Streamly/Benchmark/Data/Stream
configure+9−9
@@ -1,6 +1,6 @@
11#! /bin/sh
22# Guess values for system-dependent variables and create Makefiles.
3−# Generated by GNU Autoconf 2.71 for streamly 0.10.0.
3+# Generated by GNU Autoconf 2.71 for streamly 0.10.1.
44#
55# Report bugs to <streamly@composewell.com>.
66#
@@ -610,8 +610,8 @@ MAKEFLAGS=
610610# Identity of this package.
611611PACKAGE_NAME='streamly'
612612PACKAGE_TARNAME='streamly'
613−PACKAGE_VERSION='0.10.0'
614−PACKAGE_STRING='streamly 0.10.0'
613+PACKAGE_VERSION='0.10.1'
614+PACKAGE_STRING='streamly 0.10.1'
615615PACKAGE_BUGREPORT='streamly@composewell.com'
616616PACKAGE_URL='https://streamly.composewell.com'
617617
@@ -1224,7 +1224,7 @@ if test "$ac_init_help" = "long"; then
12241224 # Omit some internal or obsolete options to make the list less imposing.
12251225 # This message is too long to be a string in the A/UX 3.1 sh.
12261226 cat <<_ACEOF
1227−\`configure' configures streamly 0.10.0 to adapt to many kinds of systems.
1227+\`configure' configures streamly 0.10.1 to adapt to many kinds of systems.
12281228
12291229Usage: $0 [OPTION]... [VAR=VALUE]...
12301230
@@ -1286,7 +1286,7 @@ fi
12861286
12871287if test -n "$ac_init_help"; then
12881288 case $ac_init_help in
1289− short | recursive ) echo "Configuration of streamly 0.10.0:";;
1289+ short | recursive ) echo "Configuration of streamly 0.10.1:";;
12901290 esac
12911291 cat <<\_ACEOF
12921292
@@ -1372,7 +1372,7 @@ fi
13721372test -n "$ac_init_help" && exit $ac_status
13731373if $ac_init_version; then
13741374 cat <<\_ACEOF
1375−streamly configure 0.10.0
1375+streamly configure 0.10.1
13761376generated by GNU Autoconf 2.71
13771377
13781378Copyright (C) 2021 Free Software Foundation, Inc.
@@ -1500,7 +1500,7 @@ cat >config.log <<_ACEOF
15001500This file contains any messages produced by compilers while
15011501running configure, to aid debugging if configure makes a mistake.
15021502
1503−It was created by streamly $as_me 0.10.0, which was
1503+It was created by streamly $as_me 0.10.1, which was
15041504generated by GNU Autoconf 2.71. Invocation command line was
15051505
15061506 $ $0$ac_configure_args_raw
@@ -3800,7 +3800,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
38003800# report actual input values of CONFIG_FILES etc. instead of their
38013801# values after options handling.
38023802ac_log="
3803−This file was extended by streamly $as_me 0.10.0, which was
3803+This file was extended by streamly $as_me 0.10.1, which was
38043804generated by GNU Autoconf 2.71. Invocation command line was
38053805
38063806 CONFIG_FILES = $CONFIG_FILES
@@ -3856,7 +3856,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\
38563856cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
38573857ac_cs_config='$ac_cs_config_escaped'
38583858ac_cs_version="\\
3859−streamly config.status 0.10.0
3859+streamly config.status 0.10.1
38603860configured by $0, generated by GNU Autoconf 2.71,
38613861 with options \\"\$ac_cs_config\\"
38623862
configure.ac+1−1
@@ -3,7 +3,7 @@
33# See https://www.gnu.org/software/autoconf/manual/autoconf.html for help on
44# the macros used in this file.
55
6−AC_INIT([streamly], [0.10.0], [streamly@composewell.com], [streamly], [https://streamly.composewell.com])
6+AC_INIT([streamly], [0.10.1], [streamly@composewell.com], [streamly], [https://streamly.composewell.com])
77
88# To suppress "WARNING: unrecognized options: --with-compiler"
99AC_ARG_WITH([compiler], [GHC])
core/configure+9−9
@@ -1,6 +1,6 @@
11#! /bin/sh
22# Guess values for system-dependent variables and create Makefiles.
3−# Generated by GNU Autoconf 2.71 for streamly-core 0.2.0.
3+# Generated by GNU Autoconf 2.71 for streamly-core 0.2.2.
44#
55# Report bugs to <streamly@composewell.com>.
66#
@@ -610,8 +610,8 @@ MAKEFLAGS=
610610# Identity of this package.
611611PACKAGE_NAME='streamly-core'
612612PACKAGE_TARNAME='streamly-core'
613−PACKAGE_VERSION='0.2.0'
614−PACKAGE_STRING='streamly-core 0.2.0'
613+PACKAGE_VERSION='0.2.2'
614+PACKAGE_STRING='streamly-core 0.2.2'
615615PACKAGE_BUGREPORT='streamly@composewell.com'
616616PACKAGE_URL='https://streamly.composewell.com'
617617
@@ -1256,7 +1256,7 @@ if test "$ac_init_help" = "long"; then
12561256 # Omit some internal or obsolete options to make the list less imposing.
12571257 # This message is too long to be a string in the A/UX 3.1 sh.
12581258 cat <<_ACEOF
1259−\`configure' configures streamly-core 0.2.0 to adapt to many kinds of systems.
1259+\`configure' configures streamly-core 0.2.2 to adapt to many kinds of systems.
12601260
12611261Usage: $0 [OPTION]... [VAR=VALUE]...
12621262
@@ -1318,7 +1318,7 @@ fi
13181318
13191319if test -n "$ac_init_help"; then
13201320 case $ac_init_help in
1321− short | recursive ) echo "Configuration of streamly-core 0.2.0:";;
1321+ short | recursive ) echo "Configuration of streamly-core 0.2.2:";;
13221322 esac
13231323 cat <<\_ACEOF
13241324
@@ -1404,7 +1404,7 @@ fi
14041404test -n "$ac_init_help" && exit $ac_status
14051405if $ac_init_version; then
14061406 cat <<\_ACEOF
1407−streamly-core configure 0.2.0
1407+streamly-core configure 0.2.2
14081408generated by GNU Autoconf 2.71
14091409
14101410Copyright (C) 2021 Free Software Foundation, Inc.
@@ -1622,7 +1622,7 @@ cat >config.log <<_ACEOF
16221622This file contains any messages produced by compilers while
16231623running configure, to aid debugging if configure makes a mistake.
16241624
1625−It was created by streamly-core $as_me 0.2.0, which was
1625+It was created by streamly-core $as_me 0.2.2, which was
16261626generated by GNU Autoconf 2.71. Invocation command line was
16271627
16281628 $ $0$ac_configure_args_raw
@@ -3832,7 +3832,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
38323832# report actual input values of CONFIG_FILES etc. instead of their
38333833# values after options handling.
38343834ac_log="
3835−This file was extended by streamly-core $as_me 0.2.0, which was
3835+This file was extended by streamly-core $as_me 0.2.2, which was
38363836generated by GNU Autoconf 2.71. Invocation command line was
38373837
38383838 CONFIG_FILES = $CONFIG_FILES
@@ -3888,7 +3888,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\
38883888cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
38893889ac_cs_config='$ac_cs_config_escaped'
38903890ac_cs_version="\\
3891−streamly-core config.status 0.2.0
3891+streamly-core config.status 0.2.2
38923892configured by $0, generated by GNU Autoconf 2.71,
38933893 with options \\"\$ac_cs_config\\"
38943894
core/configure.ac+1−1
@@ -3,7 +3,7 @@
33# See https://www.gnu.org/software/autoconf/manual/autoconf.html for help on
44# the macros used in this file.
55
6−AC_INIT([streamly-core], [0.2.0], [streamly@composewell.com], [streamly-core], [https://streamly.composewell.com])
6+AC_INIT([streamly-core], [0.2.2], [streamly@composewell.com], [streamly-core], [https://streamly.composewell.com])
77
88# To suppress "WARNING: unrecognized options: --with-compiler"
99AC_ARG_WITH([compiler], [GHC])
core/docs/ApiChangelogs/0.1.0-0.2.0.txtnew file+1471
@@ -0,0 +1,1471 @@
1+---------------------------------
2+API Annotations
3+---------------------------------
4+
5+[A] : Added
6+[R] : Removed
7+[C] : Changed
8+[O] : Old definition
9+[N] : New definition
10+[D] : Deprecated
11+
12+---------------------------------
13+API diff
14+---------------------------------
15+
16+[C] Streamly.Unicode.Parser
17+ [A] double :: Monad m => Parser Char m Double
18+[C] Streamly.FileSystem.Handle
19+ [A] readWith :: MonadIO m => Int -> Handle -> Stream m Word8
20+ [A] readChunksWith :: MonadIO m => Int -> Handle -> Stream m (Array Word8)
21+[C] Streamly.FileSystem.Dir
22+ [C] readEither
23+ [O] readEither :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
24+ [N] readEither :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
25+ [C] read
26+ [O] read :: MonadIO m => FilePath -> Stream m FilePath
27+ [N] read :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
28+[C] Streamly.Data.Unfold
29+ [A] class Enum a => Enumerable a
30+ [A] second :: b -> Unfold m (a, b) c -> Unfold m a c
31+ [A] first :: a -> Unfold m (a, b) c -> Unfold m b c
32+ [A] enumerateFromTo :: (Enumerable a, Monad m) => Unfold m (a, a) a
33+ [A] enumerateFromThenTo :: (Enumerable a, Monad m) => Unfold m (a, a, a) a
34+ [A] enumerateFromThen :: (Enumerable a, Monad m) => Unfold m (a, a) a
35+ [A] enumerateFrom :: (Enumerable a, Monad m) => Unfold m a a
36+[C] Streamly.Data.StreamK
37+ [C] parseChunks
38+ [O] parseChunks :: (Monad m, Unbox a) => ParserK a m b -> StreamK m (Array a) -> m (Either ParseError b)
39+ [N] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
40+ [C] parseBreakChunks
41+ [O] parseBreakChunks :: (Monad m, Unbox a) => ParserK a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
42+ [N] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
43+ [A] parseBreak :: forall m a b. Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
44+ [A] parse :: Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b)
45+ [A] handle :: (MonadCatch m, Exception e) => (e -> m (StreamK m a)) -> StreamK m a -> StreamK m a
46+ [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> StreamK m a) -> StreamK m a
47+[C] Streamly.Data.Stream
48+ [A] wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
49+ [A] splitOn :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
50+ [C] handle
51+ [O] handle :: (MonadCatch m, Exception e) => (e -> Stream m a) -> Stream m a -> Stream m a
52+ [N] handle :: (MonadCatch m, Exception e) => (e -> m (Stream m a)) -> Stream m a -> Stream m a
53+ [A] groupsOf :: Monad m => Int -> Fold m a b -> Stream m a -> Stream m b
54+[C] Streamly.Data.ParserK
55+ [D] fromParser :: (MonadIO m, Unbox a) => Parser a m b -> ParserK (Array a) m b
56+ [D] fromFold :: (MonadIO m, Unbox a) => Fold m a b -> ParserK (Array a) m b
57+ [A] adaptCG :: Monad m => Parser a m b -> ParserK (Array a) m b
58+ [A] adaptC :: (Monad m, Unbox a) => Parser a m b -> ParserK (Array a) m b
59+ [A] adapt :: Monad m => Parser a m b -> ParserK a m b
60+[C] Streamly.Data.Parser
61+ [A] groupByRollingEither :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
62+ [A] groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
63+[A] Streamly.Data.MutByteArray
64+ [A] class Unbox a
65+ [A] class Serialize a
66+ [A] SerializeConfig
67+ [A] MutByteArray
68+ [A] unpin :: MutByteArray -> IO MutByteArray
69+ [A] sizeOf :: (Unbox a, SizeOfRep (Rep a)) => Proxy a -> Int
70+ [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
71+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
72+ [A] pinnedNew :: Int -> IO MutByteArray
73+ [A] pin :: MutByteArray -> IO MutByteArray
74+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
75+ [A] new :: Int -> IO MutByteArray
76+ [A] isPinned :: MutByteArray -> Bool
77+ [A] inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
78+ [A] inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
79+ [A] inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
80+ [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
81+ [A] deriveUnbox :: Q [Dec] -> Q [Dec]
82+ [A] deriveSerializeWith :: (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
83+ [A] deriveSerialize :: Q [Dec] -> Q [Dec]
84+ [A] addSizeTo :: Serialize a => Int -> a -> Int
85+[C] Streamly.Data.MutArray.Generic
86+ [A] write :: MonadIO m => Fold m a (MutArray a)
87+ [A] readRev :: MonadIO m => MutArray a -> Stream m a
88+ [A] read :: MonadIO m => MutArray a -> Stream m a
89+ [A] putIndexUnsafe :: forall m a. MonadIO m => Int -> MutArray a -> a -> m ()
90+ [A] modifyIndexUnsafe :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
91+ [A] length :: MutArray a -> Int
92+ [A] getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
93+ [C] getIndex
94+ [O] getIndex :: MonadIO m => Int -> MutArray a -> m a
95+ [N] getIndex :: MonadIO m => Int -> MutArray a -> m (Maybe a)
96+ [A] fromListN :: MonadIO m => Int -> [a] -> m (MutArray a)
97+ [A] fromList :: MonadIO m => [a] -> m (MutArray a)
98+[C] Streamly.Data.MutArray
99+ [A] unpin :: MutArray a -> IO (MutArray a)
100+ [A] readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
101+ [A] read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
102+ [A] putIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
103+ [D] pokeByteIndex :: Unbox a => Int -> MutByteArray -> a -> IO ()
104+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
105+ [A] pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
106+ [A] pin :: MutArray a -> IO (MutArray a)
107+ [D] peekByteIndex :: Unbox a => Int -> MutByteArray -> IO a
108+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
109+ [D] newPinned :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
110+ [A] modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
111+ [A] modifyIndex :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
112+ [A] modify :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (a -> a) -> m ()
113+ [A] isPinned :: MutArray a -> Bool
114+ [A] getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
115+ [C] getIndex
116+ [O] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
117+ [N] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
118+[C] Streamly.Data.Array.Generic
119+ [A] toList :: Array a -> [a]
120+ [A] getIndex :: Int -> Array a -> Maybe a
121+[C] Streamly.Data.Array
122+ [A] unpin :: Array a -> IO (Array a)
123+ [D] pokeByteIndex :: Unbox a => Int -> MutByteArray -> a -> IO ()
124+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
125+ [A] pin :: Array a -> IO (Array a)
126+ [D] peekByteIndex :: Unbox a => Int -> MutByteArray -> IO a
127+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
128+ [A] isPinned :: Array a -> Bool
129+[C] Streamly.Console.Stdio
130+
131+---------------------------------
132+Internal API diff
133+---------------------------------
134+
135+[C] Streamly.Internal.Unicode.Stream
136+ [C] writeCharUtf8'
137+ [O] writeCharUtf8' :: Monad m => Fold m Word8 Char
138+ [N] writeCharUtf8' :: Monad m => Parser Word8 m Char
139+[C] Streamly.Internal.Unicode.Parser
140+ [A] number :: Monad m => Parser Char m (Integer, Int)
141+ [A] mkDouble :: Integer -> Int -> Double
142+ [A] doubleParser :: Monad m => Parser Char m (Int, Int)
143+ [C] double
144+ [O] double :: Parser Char m Double
145+ [N] double :: Monad m => Parser Char m Double
146+[R] Streamly.Internal.Serialize.ToBytes
147+[R] Streamly.Internal.Serialize.FromBytes
148+[C] Streamly.Internal.FileSystem.File
149+ [A] writeAppendWith :: (MonadIO m, MonadCatch m) => Int -> FilePath -> Stream m Word8 -> m ()
150+ [A] writeAppendChunks :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Array a) -> m ()
151+ [A] writeAppendArray :: FilePath -> Array a -> IO ()
152+ [A] writeAppend :: (MonadIO m, MonadCatch m) => FilePath -> Stream m Word8 -> m ()
153+ [R] appendWith :: (MonadIO m, MonadCatch m) => Int -> FilePath -> Stream m Word8 -> m ()
154+ [R] appendChunks :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Array a) -> m ()
155+ [R] appendArray :: FilePath -> Array a -> IO ()
156+ [R] append :: (MonadIO m, MonadCatch m) => FilePath -> Stream m Word8 -> m ()
157+[C] Streamly.Internal.FileSystem.Dir
158+ [C] reader
159+ [O] reader :: MonadIO m => Unfold m FilePath FilePath
160+ [N] reader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
161+ [C] readFiles
162+ [O] readFiles :: MonadIO m => FilePath -> Stream m FilePath
163+ [N] readFiles :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
164+ [C] readEitherPaths
165+ [O] readEitherPaths :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
166+ [N] readEitherPaths :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
167+ [C] readEither
168+ [O] readEither :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
169+ [N] readEither :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
170+ [C] readDirs
171+ [O] readDirs :: MonadIO m => FilePath -> Stream m FilePath
172+ [N] readDirs :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
173+ [C] read
174+ [O] read :: MonadIO m => FilePath -> Stream m FilePath
175+ [N] read :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
176+ [C] fileReader
177+ [O] fileReader :: MonadIO m => Unfold m FilePath FilePath
178+ [N] fileReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
179+ [C] eitherReaderPaths
180+ [O] eitherReaderPaths :: MonadIO m => Unfold m FilePath (Either FilePath FilePath)
181+ [N] eitherReaderPaths :: (MonadIO m, MonadCatch m) => Unfold m FilePath (Either FilePath FilePath)
182+ [C] eitherReader
183+ [O] eitherReader :: MonadIO m => Unfold m FilePath (Either FilePath FilePath)
184+ [N] eitherReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath (Either FilePath FilePath)
185+ [C] dirReader
186+ [O] dirReader :: MonadIO m => Unfold m FilePath FilePath
187+ [N] dirReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
188+[R] Streamly.Internal.Data.Unfold.Type
189+[R] Streamly.Internal.Data.Unfold.Enumeration
190+[C] Streamly.Internal.Data.Unfold
191+ [C] Unfold
192+ [A] Unfold :: (s -> m (Step s b)) -> (a -> m s) -> Unfold m a b
193+ [A] takeWhileMWithInput :: Monad m => (a -> b -> m Bool) -> Unfold m a b -> Unfold m a b
194+ [A] manyInterleave :: Monad m => Unfold m a b -> Unfold m c a -> Unfold m c b
195+ [A] enumerateFromStepIntegral :: (Monad m, Integral a) => Unfold m (a, a) a
196+ [A] crossApplySnd :: Unfold m a b -> Unfold m a c -> Unfold m a c
197+ [A] crossApplyFst :: Unfold m a b -> Unfold m a c -> Unfold m a b
198+ [A] concatMap :: Monad m => (b -> Unfold m a c) -> Unfold m a b -> Unfold m a c
199+[R] Streamly.Internal.Data.Unboxed
200+[C] Streamly.Internal.Data.Time.Units
201+ [R] Streamly.Internal.Data.Unboxed.Unbox
202+ [A] Streamly.Internal.Data.Unbox.Unbox
203+ [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.NanoSecond64
204+ [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.MilliSecond64
205+ [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.MicroSecond64
206+[R] Streamly.Internal.Data.Time.Clock.Type
207+[A] Streamly.Internal.Data.StreamK
208+ [A] CrossStreamK
209+ [A] (FixityR,6)
210+ [A] (FixityR,6)
211+ [A] (FixityR,6)
212+ [A] (FixityR,5)
213+ [A] (FixityR,5)
214+ [A] (FixityR,6)
215+ [A] (FixityR,5)
216+ [A] StreamK
217+ [A] MkStream :: (forall r. State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
218+ [A] zipWithM :: Monad m => (a -> b -> m c) -> StreamK m a -> StreamK m b -> StreamK m c
219+ [A] zipWith :: Monad m => (a -> b -> c) -> StreamK m a -> StreamK m b -> StreamK m c
220+ [A] unfoldrMWith :: Monad m => (m a -> StreamK m a -> StreamK m a) -> (b -> m (Maybe (a, b))) -> b -> StreamK m a
221+ [A] unfoldrM :: Monad m => (b -> m (Maybe (a, b))) -> b -> StreamK m a
222+ [A] unfoldr :: (b -> Maybe (a, b)) -> b -> StreamK m a
223+ [A] uncons :: Applicative m => StreamK m a -> m (Maybe (a, StreamK m a))
224+ [A] unShare :: StreamK m a -> StreamK m a
225+ [A] unCross :: CrossStreamK m a -> StreamK m a
226+ [A] toStream :: Applicative m => StreamK m a -> Stream m a
227+ [A] toList :: Monad m => StreamK m a -> m [a]
228+ [A] the :: (Eq a, Monad m) => StreamK m a -> m (Maybe a)
229+ [A] takeWhile :: (a -> Bool) -> StreamK m a -> StreamK m a
230+ [A] take :: Int -> StreamK m a -> StreamK m a
231+ [A] tail :: Applicative m => StreamK m a -> m (Maybe (StreamK m a))
232+ [A] sortBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> StreamK m a
233+ [A] sequence :: Monad m => StreamK m (m a) -> StreamK m a
234+ [A] scanlx' :: (x -> a -> x) -> x -> (x -> b) -> StreamK m a -> StreamK m b
235+ [A] scanl' :: (b -> a -> b) -> b -> StreamK m a -> StreamK m b
236+ [A] reverse :: StreamK m a -> StreamK m a
237+ [A] replicateMWith :: (m a -> StreamK m a -> StreamK m a) -> Int -> m a -> StreamK m a
238+ [A] replicateM :: Monad m => Int -> m a -> StreamK m a
239+ [A] replicate :: Int -> a -> StreamK m a
240+ [A] repeatMWith :: (m a -> t m a -> t m a) -> m a -> t m a
241+ [A] repeatM :: Monad m => m a -> StreamK m a
242+ [A] repeat :: a -> StreamK m a
243+ [A] parseDBreak :: Monad m => Parser a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
244+ [A] parseD :: Monad m => Parser a m b -> StreamK m a -> m (Either ParseError b)
245+ [A] parseChunksGeneric :: Monad m => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
246+ [A] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
247+ [A] parseBreakChunksGeneric :: forall m a b. Monad m => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
248+ [A] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
249+ [A] parseBreak :: forall m a b. Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
250+ [A] parse :: Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b)
251+ [A] null :: Monad m => StreamK m a -> m Bool
252+ [A] notElem :: (Monad m, Eq a) => a -> StreamK m a -> m Bool
253+ [A] nilM :: Applicative m => m b -> StreamK m a
254+ [A] nil :: StreamK m a
255+ [A] mkStream :: (forall r. State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
256+ [A] mkCross :: StreamK m a -> CrossStreamK m a
257+ [A] minimumBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> m (Maybe a)
258+ [A] minimum :: (Monad m, Ord a) => StreamK m a -> m (Maybe a)
259+ [A] mfix :: Monad m => (m a -> StreamK m a) -> StreamK m a
260+ [A] mergeMapWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m a -> StreamK m b
261+ [A] mergeIterateWith :: (StreamK m a -> StreamK m a -> StreamK m a) -> (a -> StreamK m a) -> StreamK m a -> StreamK m a
262+ [A] mergeByM :: Monad m => (a -> a -> m Ordering) -> StreamK m a -> StreamK m a -> StreamK m a
263+ [A] mergeBy :: (a -> a -> Ordering) -> StreamK m a -> StreamK m a -> StreamK m a
264+ [A] maximumBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> m (Maybe a)
265+ [A] maximum :: (Monad m, Ord a) => StreamK m a -> m (Maybe a)
266+ [A] mapMaybe :: (a -> Maybe b) -> StreamK m a -> StreamK m b
267+ [A] mapM_ :: Monad m => (a -> m b) -> StreamK m a -> m ()
268+ [A] mapMWith :: (m b -> StreamK m b -> StreamK m b) -> (a -> m b) -> StreamK m a -> StreamK m b
269+ [A] mapMSerial :: Monad m => (a -> m b) -> StreamK m a -> StreamK m b
270+ [A] mapM :: Monad m => (a -> m b) -> StreamK m a -> StreamK m b
271+ [A] map :: (a -> b) -> StreamK m a -> StreamK m b
272+ [A] lookup :: (Monad m, Eq a) => a -> StreamK m (a, b) -> m (Maybe b)
273+ [A] liftInner :: (Monad m, MonadTrans t, Monad (t m)) => StreamK m a -> StreamK (t m) a
274+ [A] last :: Monad m => StreamK m a -> m (Maybe a)
275+ [A] iterateMWith :: Monad m => (m a -> StreamK m a -> StreamK m a) -> (a -> m a) -> m a -> StreamK m a
276+ [A] iterateM :: Monad m => (a -> m a) -> m a -> StreamK m a
277+ [A] iterate :: (a -> a) -> a -> StreamK m a
278+ [A] intersperseM :: Monad m => m a -> StreamK m a -> StreamK m a
279+ [A] intersperse :: Monad m => a -> StreamK m a -> StreamK m a
280+ [A] interleaveMin :: StreamK m a -> StreamK m a -> StreamK m a
281+ [A] interleaveFst :: StreamK m a -> StreamK m a -> StreamK m a
282+ [A] interleave :: StreamK m a -> StreamK m a -> StreamK m a
283+ [A] insertBy :: (a -> a -> Ordering) -> a -> StreamK m a -> StreamK m a
284+ [A] init :: Applicative m => StreamK m a -> m (Maybe (StreamK m a))
285+ [A] hoist :: (Monad m, Monad n) => (forall x. m x -> n x) -> StreamK m a -> StreamK n a
286+ [A] head :: Monad m => StreamK m a -> m (Maybe a)
287+ [A] handle :: (MonadCatch m, Exception e) => (e -> m (StreamK m a)) -> StreamK m a -> StreamK m a
288+ [A] fromYieldK :: YieldK m a -> StreamK m a
289+ [A] fromStream :: Monad m => Stream m a -> StreamK m a
290+ [A] fromStopK :: StopK m -> StreamK m a
291+ [A] fromPure :: a -> StreamK m a
292+ [A] fromList :: [a] -> StreamK m a
293+ [A] fromIndicesMWith :: (m a -> StreamK m a -> StreamK m a) -> (Int -> m a) -> StreamK m a
294+ [A] fromIndicesM :: Monad m => (Int -> m a) -> StreamK m a
295+ [A] fromIndices :: (Int -> a) -> StreamK m a
296+ [A] fromFoldableM :: (Foldable f, Monad m) => f (m a) -> StreamK m a
297+ [A] fromFoldable :: Foldable f => f a -> StreamK m a
298+ [A] fromEffect :: Monad m => m a -> StreamK m a
299+ [A] foldrT :: (Monad m, Monad (s m), MonadTrans s) => (a -> s m b -> s m b) -> s m b -> StreamK m a -> s m b
300+ [A] foldrSShared :: (a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
301+ [A] foldrSM :: Monad m => (m a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
302+ [A] foldrS :: (a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
303+ [A] foldrM :: (a -> m b -> m b) -> m b -> StreamK m a -> m b
304+ [A] foldr1 :: Monad m => (a -> a -> a) -> StreamK m a -> m (Maybe a)
305+ [A] foldr :: Monad m => (a -> b -> b) -> b -> StreamK m a -> m b
306+ [A] foldlx' :: forall m a b x. Monad m => (x -> a -> x) -> x -> (x -> b) -> StreamK m a -> m b
307+ [A] foldlT :: (Monad m, Monad (s m), MonadTrans s) => (s m b -> a -> s m b) -> s m b -> StreamK m a -> s m b
308+ [A] foldlS :: (StreamK m b -> a -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
309+ [A] foldlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> StreamK m a -> m b
310+ [A] foldlM' :: Monad m => (b -> a -> m b) -> m b -> StreamK m a -> m b
311+ [A] foldl' :: Monad m => (b -> a -> b) -> b -> StreamK m a -> m b
312+ [A] foldStreamShared :: State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> StreamK m a -> m r
313+ [A] foldStream :: State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> StreamK m a -> m r
314+ [A] foldEither :: Monad m => Fold m a b -> StreamK m a -> m (Either (Fold m a b) (b, StreamK m a))
315+ [A] foldConcat :: Monad m => Producer m a b -> Fold m b c -> StreamK m a -> m (c, StreamK m a)
316+ [A] foldBreak :: Monad m => Fold m a b -> StreamK m a -> m (b, StreamK m a)
317+ [A] fold :: Monad m => Fold m a b -> StreamK m a -> m b
318+ [A] findM :: Monad m => (a -> m Bool) -> StreamK m a -> m (Maybe a)
319+ [A] findIndices :: (a -> Bool) -> StreamK m a -> StreamK m Int
320+ [A] find :: Monad m => (a -> Bool) -> StreamK m a -> m (Maybe a)
321+ [A] filter :: (a -> Bool) -> StreamK m a -> StreamK m a
322+ [A] evalStateT :: Monad m => m s -> StreamK (StateT s m) a -> StreamK m a
323+ [A] elem :: (Monad m, Eq a) => a -> StreamK m a -> m Bool
324+ [A] dropWhile :: (a -> Bool) -> StreamK m a -> StreamK m a
325+ [A] drop :: Int -> StreamK m a -> StreamK m a
326+ [A] drain :: Monad m => StreamK m a -> m ()
327+ [A] deleteBy :: (a -> a -> Bool) -> a -> StreamK m a -> StreamK m a
328+ [A] crossWith :: Monad m => (a -> b -> c) -> StreamK m a -> StreamK m b -> StreamK m c
329+ [A] crossApplyWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> StreamK m (a -> b) -> StreamK m a -> StreamK m b
330+ [A] crossApplySnd :: StreamK m a -> StreamK m b -> StreamK m b
331+ [A] crossApplyFst :: StreamK m a -> StreamK m b -> StreamK m a
332+ [A] crossApply :: StreamK m (a -> b) -> StreamK m a -> StreamK m b
333+ [A] cross :: Monad m => StreamK m a -> StreamK m b -> StreamK m (a, b)
334+ [A] consMBy :: Monad m => (StreamK m a -> StreamK m a -> StreamK m a) -> m a -> StreamK m a -> StreamK m a
335+ [A] consM :: Monad m => m a -> StreamK m a -> StreamK m a
336+ [A] consK :: YieldK m a -> StreamK m a -> StreamK m a
337+ [A] cons :: a -> StreamK m a -> StreamK m a
338+ [A] conjoin :: Monad m => StreamK m a -> StreamK m a -> StreamK m a
339+ [A] concatMapWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m a -> StreamK m b
340+ [A] concatMapEffect :: Monad m => (b -> StreamK m a) -> m b -> StreamK m a
341+ [A] concatMap :: (a -> StreamK m b) -> StreamK m a -> StreamK m b
342+ [A] concatIterateWith :: (StreamK m a -> StreamK m a -> StreamK m a) -> (a -> StreamK m a) -> StreamK m a -> StreamK m a
343+ [A] concatIterateScanWith :: Monad m => (StreamK m a -> StreamK m a -> StreamK m a) -> (b -> a -> m (b, StreamK m a)) -> m b -> StreamK m a -> StreamK m a
344+ [A] concatIterateLeftsWith :: b ~ Either a c => (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m b -> StreamK m b
345+ [A] concatEffect :: Monad m => m (StreamK m a) -> StreamK m a
346+ [A] buildSM :: Monad m => ((m a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a
347+ [A] buildS :: ((a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a
348+ [A] buildM :: Monad m => (forall r. (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
349+ [A] build :: forall m a. (forall b. (a -> b -> b) -> b -> b) -> StreamK m a
350+ [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> StreamK m a) -> StreamK m a
351+ [A] bindWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> StreamK m a -> (a -> StreamK m b) -> StreamK m b
352+ [A] before :: Monad m => m b -> StreamK m a -> StreamK m a
353+ [A] augmentSM :: Monad m => ((m a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a
354+ [A] augmentS :: ((a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a
355+ [A] append :: StreamK m a -> StreamK m a -> StreamK m a
356+ [A] any :: Monad m => (a -> Bool) -> StreamK m a -> m Bool
357+ [A] all :: Monad m => (a -> Bool) -> StreamK m a -> m Bool
358+ [A] (.:) :: a -> StreamK m a -> StreamK m a
359+ [A] (!!) :: Monad m => StreamK m a -> Int -> m (Maybe a)
360+[R] Streamly.Internal.Data.Stream.StreamK.Type
361+[R] Streamly.Internal.Data.Stream.StreamK.Transformer
362+[R] Streamly.Internal.Data.Stream.StreamK
363+[R] Streamly.Internal.Data.Stream.StreamD.Type
364+[R] Streamly.Internal.Data.Stream.StreamD.Transformer
365+[R] Streamly.Internal.Data.Stream.StreamD.Transform
366+[R] Streamly.Internal.Data.Stream.StreamD.Top
367+[R] Streamly.Internal.Data.Stream.StreamD.Step
368+[R] Streamly.Internal.Data.Stream.StreamD.Nesting
369+[R] Streamly.Internal.Data.Stream.StreamD.Lift
370+[R] Streamly.Internal.Data.Stream.StreamD.Generate
371+[R] Streamly.Internal.Data.Stream.StreamD.Exception
372+[R] Streamly.Internal.Data.Stream.StreamD.Eliminate
373+[R] Streamly.Internal.Data.Stream.StreamD.Container
374+[D] Streamly.Internal.Data.Stream.StreamD
375+[R] Streamly.Internal.Data.Stream.Common
376+[R] Streamly.Internal.Data.Stream.Chunked
377+[C] Streamly.Internal.Data.Stream
378+ [A] class Enum a => Enumerable a
379+ [A] Stream
380+ [A] UnStream :: (State StreamK m a -> s -> m (Step s a)) -> s -> Stream m a
381+ [A] Step
382+ [A] Yield :: a -> s -> Step s a
383+ [A] Stop :: Step s a
384+ [A] Skip :: s -> Step s a
385+ [A] InterleaveState
386+ [A] InterleaveSecondOnly :: s2 -> InterleaveState s1 s2
387+ [A] InterleaveSecond :: s1 -> s2 -> InterleaveState s1 s2
388+ [A] InterleaveFirstOnly :: s1 -> InterleaveState s1 s2
389+ [A] InterleaveFirst :: s1 -> s2 -> InterleaveState s1 s2
390+ [A] FoldManyPost
391+ [A] FoldManyPostYield :: b -> FoldManyPost s fs b a -> FoldManyPost s fs b a
392+ [A] FoldManyPostStart :: s -> FoldManyPost s fs b a
393+ [A] FoldManyPostLoop :: s -> fs -> FoldManyPost s fs b a
394+ [A] FoldManyPostDone :: FoldManyPost s fs b a
395+ [A] FoldMany
396+ [A] FoldManyYield :: b -> FoldMany s fs b a -> FoldMany s fs b a
397+ [A] FoldManyStart :: s -> FoldMany s fs b a
398+ [A] FoldManyLoop :: s -> fs -> FoldMany s fs b a
399+ [A] FoldManyFirst :: fs -> s -> FoldMany s fs b a
400+ [A] FoldManyDone :: FoldMany s fs b a
401+ [A] CrossStream
402+ [A] ConcatUnfoldInterleaveState
403+ [A] ConcatUnfoldInterleaveOuter :: o -> [i] -> ConcatUnfoldInterleaveState o i
404+ [A] ConcatUnfoldInterleaveInnerR :: [i] -> [i] -> ConcatUnfoldInterleaveState o i
405+ [A] ConcatUnfoldInterleaveInnerL :: [i] -> [i] -> ConcatUnfoldInterleaveState o i
406+ [A] ConcatUnfoldInterleaveInner :: o -> [i] -> ConcatUnfoldInterleaveState o i
407+ [A] ConcatMapUState
408+ [A] ConcatMapUOuter :: o -> ConcatMapUState o i
409+ [A] ConcatMapUInner :: o -> i -> ConcatMapUState o i
410+ [A] AppendState
411+ [A] AppendSecond :: s2 -> AppendState s1 s2
412+ [A] AppendFirst :: s1 -> AppendState s1 s2
413+ [A] pattern Stream :: (State StreamK m a -> s -> m (Step s a)) -> s -> Stream m a
414+ [A] zipWithM :: Monad m => (a -> b -> m c) -> Stream m a -> Stream m b -> Stream m c
415+ [A] zipWith :: Monad m => (a -> b -> c) -> Stream m a -> Stream m b -> Stream m c
416+ [A] wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
417+ [A] with :: Monad m => (Stream m a -> Stream m (s, a)) -> (((s, a) -> b) -> Stream m (s, a) -> Stream m (s, a)) -> ((s, a) -> b) -> Stream m a -> Stream m a
418+ [A] usingStateT :: Monad m => m s -> (Stream (StateT s m) a -> Stream (StateT s m) a) -> Stream m a -> Stream m a
419+ [A] usingReaderT :: Monad m => m r -> (Stream (ReaderT r m) a -> Stream (ReaderT r m) a) -> Stream m a -> Stream m a
420+ [A] uniqBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a
421+ [A] uniq :: (Eq a, Monad m) => Stream m a -> Stream m a
422+ [A] unionWithStreamGenericBy :: MonadIO m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
423+ [A] unionWithStreamAscBy :: (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
424+ [A] unfoldrM :: Monad m => (s -> m (Maybe (a, s))) -> s -> Stream m a
425+ [A] unfoldr :: Monad m => (s -> Maybe (a, s)) -> s -> Stream m a
426+ [A] unfoldRoundRobin :: Monad m => Unfold m a b -> Stream m a -> Stream m b
427+ [A] unfoldMany :: Monad m => Unfold m a b -> Stream m a -> Stream m b
428+ [A] unfoldIterateDfs :: Monad m => Unfold m a a -> Stream m a -> Stream m a
429+ [A] unfoldIterateBfsRev :: Monad m => Unfold m a a -> Stream m a -> Stream m a
430+ [A] unfoldIterateBfs :: Monad m => Unfold m a a -> Stream m a -> Stream m a
431+ [A] unfoldInterleave :: Monad m => Unfold m a b -> Stream m a -> Stream m b
432+ [A] unfold :: Applicative m => Unfold m a b -> a -> Stream m b
433+ [A] uncons :: Monad m => Stream m a -> m (Maybe (a, Stream m a))
434+ [A] unCross :: CrossStream m a -> Stream m a
435+ [A] transform :: Monad m => Pipe m a b -> Stream m a -> Stream m b
436+ [A] trace_ :: Monad m => m b -> Stream m a -> Stream m a
437+ [A] trace :: Monad m => (a -> m b) -> Stream m a -> Stream m a
438+ [A] toStreamK :: Monad m => Stream m a -> StreamK m a
439+ [A] toListRev :: Monad m => Stream m a -> m [a]
440+ [A] toList :: Monad m => Stream m a -> m [a]
441+ [A] timestamped :: MonadIO m => Stream m a -> Stream m (AbsTime, a)
442+ [A] timestampWith :: MonadIO m => Double -> Stream m a -> Stream m (AbsTime, a)
443+ [A] timesWith :: MonadIO m => Double -> Stream m (AbsTime, RelTime64)
444+ [A] times :: MonadIO m => Stream m (AbsTime, RelTime64)
445+ [A] timeout :: AbsTime -> t m ()
446+ [A] timeIndexed :: MonadIO m => Stream m a -> Stream m (RelTime64, a)
447+ [A] timeIndexWith :: MonadIO m => Double -> Stream m a -> Stream m (RelTime64, a)
448+ [A] the :: (Eq a, Monad m) => Stream m a -> m (Maybe a)
449+ [A] tapOffsetEvery :: Monad m => Int -> Int -> Fold m a b -> Stream m a -> Stream m a
450+ [A] tap :: Monad m => Fold m a b -> Stream m a -> Stream m a
451+ [A] takeWhileM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
452+ [A] takeWhileLast :: (a -> Bool) -> Stream m a -> Stream m a
453+ [A] takeWhileAround :: (a -> Bool) -> Stream m a -> Stream m a
454+ [A] takeWhile :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
455+ [A] takeEndByM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
456+ [A] takeEndBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
457+ [A] take :: Applicative m => Int -> Stream m a -> Stream m a
458+ [A] tail :: Monad m => Stream m a -> m (Maybe (Stream m a))
459+ [A] stripSuffixUnbox :: (MonadIO m, Eq a, Unbox a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
460+ [A] stripSuffix :: (Monad m, Eq a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
461+ [A] stripPrefix :: (Monad m, Eq a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
462+ [A] strideFromThen :: Monad m => Int -> Int -> Stream m a -> Stream m a
463+ [A] splitOnSuffixSeqAny :: [Array a] -> Fold m a b -> Stream m a -> Stream m b
464+ [A] splitOnSuffixSeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) => Bool -> Array a -> Fold m a b -> Stream m a -> Stream m b
465+ [A] splitOnSeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) => Array a -> Fold m a b -> Stream m a -> Stream m b
466+ [A] splitOnPrefix :: (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
467+ [A] splitOnAny :: [Array a] -> Fold m a b -> Stream m a -> Stream m b
468+ [A] splitOn :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
469+ [A] splitInnerBySuffix :: (Monad m, Eq (f a), Monoid (f a)) => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
470+ [A] splitInnerBy :: Monad m => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
471+ [A] slicesBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
472+ [A] sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
473+ [A] sequence :: Monad m => Stream m (m a) -> Stream m a
474+ [A] scanlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> Stream m b
475+ [A] scanlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> Stream m b
476+ [A] scanlMAfter' :: Monad m => (b -> a -> m b) -> m b -> (b -> m b) -> Stream m a -> Stream m b
477+ [A] scanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
478+ [A] scanlM :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
479+ [A] scanl1M' :: Monad m => (a -> a -> m a) -> Stream m a -> Stream m a
480+ [A] scanl1M :: Monad m => (a -> a -> m a) -> Stream m a -> Stream m a
481+ [A] scanl1' :: Monad m => (a -> a -> a) -> Stream m a -> Stream m a
482+ [A] scanl1 :: Monad m => (a -> a -> a) -> Stream m a -> Stream m a
483+ [A] scanl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
484+ [A] scanl :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
485+ [A] scanMaybe :: Monad m => Fold m a (Maybe b) -> Stream m a -> Stream m b
486+ [A] scanMany :: Monad m => Fold m a b -> Stream m a -> Stream m b
487+ [A] scan :: Monad m => Fold m a b -> Stream m a -> Stream m b
488+ [A] runStateT :: Monad m => m s -> Stream (StateT s m) a -> Stream m (s, a)
489+ [A] runReaderT :: Monad m => m s -> Stream (ReaderT s m) a -> Stream m a
490+ [A] runInnerWithState :: Monad m => (forall b. s -> t m b -> m (b, s)) -> m s -> Stream (t m) a -> Stream m (s, a)
491+ [A] runInnerWith :: Monad m => (forall b. t m b -> m b) -> Stream (t m) a -> Stream m a
492+ [A] roundRobin :: Monad m => Stream m a -> Stream m a -> Stream m a
493+ [A] rollingMapM :: Monad m => (Maybe a -> a -> m b) -> Stream m a -> Stream m b
494+ [A] rollingMap2 :: Monad m => (a -> a -> b) -> Stream m a -> Stream m b
495+ [A] rollingMap :: Monad m => (Maybe a -> a -> b) -> Stream m a -> Stream m b
496+ [A] reverseUnbox :: (MonadIO m, Unbox a) => Stream m a -> Stream m a
497+ [A] reverse :: Monad m => Stream m a -> Stream m a
498+ [A] replicateM :: Monad m => Int -> m a -> Stream m a
499+ [A] replicate :: Monad m => Int -> a -> Stream m a
500+ [A] repeated :: Stream m a -> Stream m a
501+ [A] repeatM :: Monad m => m a -> Stream m a
502+ [A] repeat :: Monad m => a -> Stream m a
503+ [A] relTimesWith :: MonadIO m => Double -> Stream m RelTime64
504+ [A] relTimes :: MonadIO m => Stream m RelTime64
505+ [A] refoldMany :: Monad m => Refold m x a b -> m x -> Stream m a -> Stream m b
506+ [A] refoldIterateM :: Monad m => Refold m b a b -> m b -> Stream m a -> Stream m b
507+ [A] reduceIterateBfs :: Monad m => (a -> a -> m a) -> Stream m a -> m (Maybe a)
508+ [A] reassembleBy :: Fold m a b -> (a -> a -> Int) -> Stream m a -> Stream m b
509+ [A] prune :: (a -> Bool) -> Stream m a -> Stream m a
510+ [A] prescanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
511+ [A] prescanl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
512+ [A] postscanlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> Stream m b
513+ [A] postscanlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> Stream m b
514+ [A] postscanlMAfter' :: Monad m => (b -> a -> m b) -> m b -> (b -> m b) -> Stream m a -> Stream m b
515+ [A] postscanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
516+ [A] postscanlM :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
517+ [A] postscanl' :: Monad m => (a -> b -> a) -> a -> Stream m b -> Stream m a
518+ [A] postscanl :: Monad m => (a -> b -> a) -> a -> Stream m b -> Stream m a
519+ [A] postscan :: Monad m => Fold m a b -> Stream m a -> Stream m b
520+ [A] parseSequence :: Stream m (Parser a m b) -> Stream m a -> Stream m b
521+ [A] parseManyTill :: Parser a m b -> Parser a m x -> Stream m a -> Stream m b
522+ [A] parseManyD :: Monad m => Parser a m b -> Stream m a -> Stream m (Either ParseError b)
523+ [A] parseMany :: Monad m => Parser a m b -> Stream m a -> Stream m (Either ParseError b)
524+ [A] parseIterateD :: Monad m => (b -> Parser a m b) -> b -> Stream m a -> Stream m (Either ParseError b)
525+ [A] parseIterate :: Monad m => (b -> Parser a m b) -> b -> Stream m a -> Stream m (Either ParseError b)
526+ [A] parseD :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b)
527+ [A] parseBreakD :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b, Stream m a)
528+ [A] parseBreak :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b, Stream m a)
529+ [A] parse :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b)
530+ [A] onException :: MonadCatch m => m b -> Stream m a -> Stream m a
531+ [A] null :: Monad m => Stream m a -> m Bool
532+ [A] nub :: (Monad m, Ord a) => Stream m a -> Stream m a
533+ [A] notElem :: (Monad m, Eq a) => a -> Stream m a -> m Bool
534+ [A] nilM :: Applicative m => m b -> Stream m a
535+ [A] nil :: Applicative m => Stream m a
536+ [A] morphInner :: Monad n => (forall x. m x -> n x) -> Stream m a -> Stream n a
537+ [A] mkCross :: Stream m a -> CrossStream m a
538+ [A] minimumBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> m (Maybe a)
539+ [A] minimum :: (Monad m, Ord a) => Stream m a -> m (Maybe a)
540+ [A] mergeMinBy :: (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
541+ [A] mergeFstBy :: (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
542+ [A] mergeByM :: Monad m => (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
543+ [A] mergeBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
544+ [A] maximumBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> m (Maybe a)
545+ [A] maximum :: (Monad m, Ord a) => Stream m a -> m (Maybe a)
546+ [A] mapMaybeM :: Monad m => (a -> m (Maybe b)) -> Stream m a -> Stream m b
547+ [A] mapMaybe :: Monad m => (a -> Maybe b) -> Stream m a -> Stream m b
548+ [A] mapM_ :: Monad m => (a -> m b) -> Stream m a -> m ()
549+ [A] mapM :: Monad m => (a -> m b) -> Stream m a -> Stream m b
550+ [A] map :: Monad m => (a -> b) -> Stream m a -> Stream m b
551+ [A] lookup :: (Monad m, Eq a) => a -> Stream m (a, b) -> m (Maybe b)
552+ [A] liftInnerWith :: Monad (t m) => (forall b. m b -> t m b) -> Stream m a -> Stream (t m) a
553+ [A] liftInner :: (Monad m, MonadTrans t, Monad (t m)) => Stream m a -> Stream (t m) a
554+ [A] last :: Monad m => Stream m a -> m (Maybe a)
555+ [A] joinOuterGeneric :: MonadIO m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (Maybe a, Maybe b)
556+ [A] joinOuterAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (Maybe a, Maybe b)
557+ [A] joinOuter :: (Ord k, MonadIO m) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, Maybe a, Maybe b)
558+ [A] joinLeftGeneric :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
559+ [A] joinLeftAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
560+ [A] joinLeft :: (Ord k, Monad m) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, Maybe b)
561+ [A] joinInnerGeneric :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, b)
562+ [A] joinInnerAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, b)
563+ [A] joinInner :: (Monad m, Ord k) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, b)
564+ [A] iterateM :: Monad m => (a -> m a) -> m a -> Stream m a
565+ [A] iterate :: Monad m => (a -> a) -> a -> Stream m a
566+ [A] isSuffixOfUnbox :: (MonadIO m, Eq a, Unbox a) => Stream m a -> Stream m a -> m Bool
567+ [A] isSuffixOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
568+ [A] isSubsequenceOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
569+ [A] isPrefixOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
570+ [A] isInfixOf :: (MonadIO m, Eq a, Enum a, Storable a, Unbox a) => Stream m a -> Stream m a -> m Bool
571+ [A] intersperseM_ :: Monad m => m b -> Stream m a -> Stream m a
572+ [A] intersperseMWith :: Int -> m a -> Stream m a -> Stream m a
573+ [A] intersperseMSuffix_ :: Monad m => m b -> Stream m a -> Stream m a
574+ [A] intersperseMSuffixWith :: forall m a. Monad m => Int -> m a -> Stream m a -> Stream m a
575+ [A] intersperseMSuffix :: forall m a. Monad m => m a -> Stream m a -> Stream m a
576+ [A] intersperseMPrefix_ :: Monad m => m b -> Stream m a -> Stream m a
577+ [A] intersperseM :: Monad m => m a -> Stream m a -> Stream m a
578+ [A] intersperse :: Monad m => a -> Stream m a -> Stream m a
579+ [A] intersectBySorted :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
580+ [A] interposeSuffixM :: Monad m => m c -> Unfold m b c -> Stream m b -> Stream m c
581+ [A] interposeSuffix :: Monad m => c -> Unfold m b c -> Stream m b -> Stream m c
582+ [A] interposeM :: Monad m => m c -> Unfold m b c -> Stream m b -> Stream m c
583+ [A] interpose :: Monad m => c -> Unfold m b c -> Stream m b -> Stream m c
584+ [A] interleaveMin :: Monad m => Stream m a -> Stream m a -> Stream m a
585+ [A] interleaveFstSuffix :: Monad m => Stream m a -> Stream m a -> Stream m a
586+ [A] interleaveFst :: Monad m => Stream m a -> Stream m a -> Stream m a
587+ [A] interleave :: Monad m => Stream m a -> Stream m a -> Stream m a
588+ [A] intercalateSuffix :: Monad m => Unfold m b c -> b -> Stream m b -> Stream m c
589+ [A] intercalate :: Monad m => Unfold m b c -> b -> Stream m b -> Stream m c
590+ [A] insertBy :: Monad m => (a -> a -> Ordering) -> a -> Stream m a -> Stream m a
591+ [A] indexedR :: Monad m => Int -> Stream m a -> Stream m (Int, a)
592+ [A] indexed :: Monad m => Stream m a -> Stream m (Int, a)
593+ [A] headElse :: Monad m => a -> Stream m a -> m a
594+ [A] head :: Monad m => Stream m a -> m (Maybe a)
595+ [A] handle :: (MonadCatch m, Exception e) => (e -> m (Stream m a)) -> Stream m a -> Stream m a
596+ [A] groupsWhile :: Monad m => (a -> a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
597+ [A] groupsRollingBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
598+ [A] groupsOf :: Monad m => Int -> Fold m a b -> Stream m a -> Stream m b
599+ [A] gintercalateSuffix :: Monad m => Unfold m a c -> Stream m a -> Unfold m b c -> Stream m b -> Stream m c
600+ [A] gintercalate :: Monad m => Unfold m a c -> Stream m a -> Unfold m b c -> Stream m b -> Stream m c
601+ [A] ghandle :: (MonadCatch m, Exception e) => (e -> Stream m a -> m (Stream m a)) -> Stream m a -> Stream m a
602+ [A] generateM :: Monad m => Int -> (Int -> m a) -> Stream m a
603+ [A] generate :: Monad m => Int -> (Int -> a) -> Stream m a
604+ [A] generalizeInner :: Monad m => Stream Identity a -> Stream m a
605+ [A] gbracket_ :: Monad m => m c -> (c -> m d) -> (c -> e -> Stream m b -> m (Stream m b)) -> (forall s. m s -> m (Either e s)) -> (c -> Stream m b) -> Stream m b
606+ [A] gbracket :: MonadIO m => IO c -> (c -> IO d1) -> (c -> e -> Stream m b -> IO (Stream m b)) -> (c -> IO d2) -> (forall s. m s -> m (Either e s)) -> (c -> Stream m b) -> Stream m b
607+ [A] fromStreamK :: Applicative m => StreamK m a -> Stream m a
608+ [A] fromPure :: Applicative m => a -> Stream m a
609+ [A] fromPtrN :: (Monad m, Storable a) => Int -> Ptr a -> Stream m a
610+ [A] fromPtr :: forall m a. (Monad m, Storable a) => Ptr a -> Stream m a
611+ [A] fromListM :: Monad m => [m a] -> Stream m a
612+ [A] fromList :: Applicative m => [a] -> Stream m a
613+ [A] fromIndicesM :: Monad m => (Int -> m a) -> Stream m a
614+ [A] fromIndices :: Monad m => (Int -> a) -> Stream m a
615+ [A] fromFoldableM :: (Monad m, Foldable f) => f (m a) -> Stream m a
616+ [A] fromFoldable :: (Monad m, Foldable f) => f a -> Stream m a
617+ [A] fromEffect :: Applicative m => m a -> Stream m a
618+ [A] fromByteStr# :: Monad m => Addr# -> Stream m Word8
619+ [A] foldrT :: (Monad m, Monad (t m), MonadTrans t) => (a -> t m b -> t m b) -> t m b -> Stream m a -> t m b
620+ [A] foldrS :: Monad m => (a -> Stream m b -> Stream m b) -> Stream m b -> Stream m a -> Stream m b
621+ [A] foldrMx :: Monad m => (a -> m x -> m x) -> m x -> (m x -> m b) -> Stream m a -> m b
622+ [A] foldrM :: Monad m => (a -> m b -> m b) -> m b -> Stream m a -> m b
623+ [A] foldr1 :: Monad m => (a -> a -> a) -> Stream m a -> m (Maybe a)
624+ [A] foldr :: Monad m => (a -> b -> b) -> b -> Stream m a -> m b
625+ [A] foldlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> m b
626+ [A] foldlT :: (Monad m, Monad (s m), MonadTrans s) => (s m b -> a -> s m b) -> s m b -> Stream m a -> s m b
627+ [A] foldlS :: Monad m => (Stream m b -> a -> Stream m b) -> Stream m b -> Stream m a -> Stream m b
628+ [A] foldlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> m b
629+ [A] foldlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> m b
630+ [A] foldl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> m b
631+ [A] foldSequence :: Stream m (Fold m a b) -> Stream m a -> Stream m b
632+ [A] foldManyPost :: Monad m => Fold m a b -> Stream m a -> Stream m b
633+ [A] foldMany :: Monad m => Fold m a b -> Stream m a -> Stream m b
634+ [A] foldIterateM :: Monad m => (b -> m (Fold m a b)) -> m b -> Stream m a -> Stream m b
635+ [A] foldIterateBfs :: Fold m a (Either a a) -> Stream m a -> m (Maybe a)
636+ [A] foldEither :: Monad m => Fold m a b -> Stream m a -> m (Either (Fold m a b) (b, Stream m a))
637+ [A] foldBreak :: Monad m => Fold m a b -> Stream m a -> m (b, Stream m a)
638+ [A] foldAddLazy :: Monad m => Fold m a b -> Stream m a -> Fold m a b
639+ [A] foldAdd :: Monad m => Fold m a b -> Stream m a -> m (Fold m a b)
640+ [A] fold :: Monad m => Fold m a b -> Stream m a -> m b
641+ [A] findM :: Monad m => (a -> m Bool) -> Stream m a -> m (Maybe a)
642+ [A] findIndices :: Monad m => (a -> Bool) -> Stream m a -> Stream m Int
643+ [A] find :: Monad m => (a -> Bool) -> Stream m a -> m (Maybe a)
644+ [A] finallyUnsafe :: MonadCatch m => m b -> Stream m a -> Stream m a
645+ [A] finallyIO :: (MonadIO m, MonadCatch m) => IO b -> Stream m a -> Stream m a
646+ [A] filterM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
647+ [A] filterInStreamGenericBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
648+ [A] filterInStreamAscBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
649+ [A] filter :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
650+ [A] evalStateT :: Monad m => m s -> Stream (StateT s m) a -> Stream m a
651+ [A] eqBy :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> m Bool
652+ [A] enumerateTo :: (Monad m, Bounded a, Enumerable a) => a -> Stream m a
653+ [A] enumerateFromToSmall :: (Monad m, Enum a) => a -> a -> Stream m a
654+ [A] enumerateFromToIntegral :: (Monad m, Integral a) => a -> a -> Stream m a
655+ [A] enumerateFromToFractional :: (Monad m, Fractional a, Ord a) => a -> a -> Stream m a
656+ [A] enumerateFromTo :: (Enumerable a, Monad m) => a -> a -> Stream m a
657+ [A] enumerateFromThenToSmall :: (Monad m, Enum a) => a -> a -> a -> Stream m a
658+ [A] enumerateFromThenToIntegral :: (Monad m, Integral a) => a -> a -> a -> Stream m a
659+ [A] enumerateFromThenToFractional :: (Monad m, Fractional a, Ord a) => a -> a -> a -> Stream m a
660+ [A] enumerateFromThenTo :: (Enumerable a, Monad m) => a -> a -> a -> Stream m a
661+ [A] enumerateFromThenSmallBounded :: (Monad m, Enumerable a, Bounded a) => a -> a -> Stream m a
662+ [A] enumerateFromThenNum :: (Monad m, Num a) => a -> a -> Stream m a
663+ [A] enumerateFromThenIntegral :: (Monad m, Integral a, Bounded a) => a -> a -> Stream m a
664+ [A] enumerateFromThenFractional :: (Monad m, Fractional a) => a -> a -> Stream m a
665+ [A] enumerateFromThen :: (Enumerable a, Monad m) => a -> a -> Stream m a
666+ [A] enumerateFromStepNum :: (Monad m, Num a) => a -> a -> Stream m a
667+ [A] enumerateFromStepIntegral :: (Integral a, Monad m) => a -> a -> Stream m a
668+ [A] enumerateFromNum :: (Monad m, Num a) => a -> Stream m a
669+ [A] enumerateFromIntegral :: (Monad m, Integral a, Bounded a) => a -> Stream m a
670+ [A] enumerateFromFractional :: (Monad m, Fractional a) => a -> Stream m a
671+ [A] enumerateFromBounded :: (Monad m, Enumerable a, Bounded a) => a -> Stream m a
672+ [A] enumerateFrom :: (Enumerable a, Monad m) => a -> Stream m a
673+ [A] enumerate :: (Monad m, Bounded a, Enumerable a) => Stream m a
674+ [A] elemIndices :: (Monad m, Eq a) => a -> Stream m a -> Stream m Int
675+ [A] elem :: (Monad m, Eq a) => a -> Stream m a -> m Bool
676+ [A] durations :: Double -> t m RelTime64
677+ [A] dropWhileM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
678+ [A] dropWhileLast :: (a -> Bool) -> Stream m a -> Stream m a
679+ [A] dropWhileAround :: (a -> Bool) -> Stream m a -> Stream m a
680+ [A] dropWhile :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
681+ [A] dropSuffix :: Stream m a -> Stream m a -> Stream m a
682+ [A] dropPrefix :: Stream m a -> Stream m a -> Stream m a
683+ [A] dropLast :: Int -> Stream m a -> Stream m a
684+ [A] dropInfix :: Stream m a -> Stream m a -> Stream m a
685+ [A] drop :: Monad m => Int -> Stream m a -> Stream m a
686+ [A] drain :: Monad m => Stream m a -> m ()
687+ [A] deleteInStreamGenericBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
688+ [A] deleteInStreamAscBy :: (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
689+ [A] deleteBy :: Monad m => (a -> a -> Bool) -> a -> Stream m a -> Stream m a
690+ [A] delayPre :: MonadIO m => Double -> Stream m a -> Stream m a
691+ [A] delayPost :: MonadIO m => Double -> Stream m a -> Stream m a
692+ [A] delay :: MonadIO m => Double -> Stream m a -> Stream m a
693+ [A] crossWith :: Monad m => (a -> b -> c) -> Stream m a -> Stream m b -> Stream m c
694+ [A] crossApplySnd :: Functor f => Stream f a -> Stream f b -> Stream f b
695+ [A] crossApplyFst :: Functor f => Stream f a -> Stream f b -> Stream f a
696+ [A] crossApply :: Functor f => Stream f (a -> b) -> Stream f a -> Stream f b
697+ [A] cross :: Monad m => Stream m a -> Stream m b -> Stream m (a, b)
698+ [A] consM :: Applicative m => m a -> Stream m a -> Stream m a
699+ [A] cons :: Applicative m => a -> Stream m a -> Stream m a
700+ [A] concatMapM :: Monad m => (a -> m (Stream m b)) -> Stream m a -> Stream m b
701+ [A] concatMap :: Monad m => (a -> Stream m b) -> Stream m a -> Stream m b
702+ [A] concatIterateScan :: Monad m => (b -> a -> m b) -> (b -> m (Maybe (b, Stream m a))) -> b -> Stream m a
703+ [A] concatIterateDfs :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
704+ [A] concatIterateBfsRev :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
705+ [A] concatIterateBfs :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
706+ [A] concatEffect :: Monad m => m (Stream m a) -> Stream m a
707+ [A] concat :: Monad m => Stream m (Stream m a) -> Stream m a
708+ [A] cmpBy :: Monad m => (a -> b -> Ordering) -> Stream m a -> Stream m b -> m Ordering
709+ [A] catRights :: Monad m => Stream m (Either a b) -> Stream m b
710+ [A] catMaybes :: Monad m => Stream m (Maybe a) -> Stream m a
711+ [A] catLefts :: Monad m => Stream m (Either a b) -> Stream m a
712+ [A] catEithers :: Monad m => Stream m (Either a a) -> Stream m a
713+ [A] bracketUnsafe :: MonadCatch m => m b -> (b -> m c) -> (b -> Stream m a) -> Stream m a
714+ [A] bracketIO3 :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> IO d) -> (b -> IO e) -> (b -> Stream m a) -> Stream m a
715+ [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> Stream m a) -> Stream m a
716+ [A] before :: Monad m => m b -> Stream m a -> Stream m a
717+ [A] append :: Monad m => Stream m a -> Stream m a -> Stream m a
718+ [A] any :: Monad m => (a -> Bool) -> Stream m a -> m Bool
719+ [A] all :: Monad m => (a -> Bool) -> Stream m a -> m Bool
720+ [A] afterUnsafe :: Monad m => m b -> Stream m a -> Stream m a
721+ [A] afterIO :: MonadIO m => IO b -> Stream m a -> Stream m a
722+ [A] absTimesWith :: MonadIO m => Double -> Stream m AbsTime
723+ [A] absTimes :: MonadIO m => Stream m AbsTime
724+ [A] (!!) :: Monad m => Stream m a -> Int -> m (Maybe a)
725+[R] Streamly.Internal.Data.Ring.Unboxed
726+[A] Streamly.Internal.Data.Ring.Generic
727+ [A] Ring
728+ [A] [ringMax] :: Ring a -> !Int
729+ [A] [ringHead] :: Ring a -> !Int
730+ [A] [ringArr] :: Ring a -> MutArray a
731+ [A] Ring :: MutArray a -> !Int -> !Int -> Ring a
732+ [A] writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
733+ [A] unsafeInsertRingWith :: Ring a -> a -> IO Int
734+ [A] toStreamWith :: Int -> Ring a -> Stream m a
735+ [A] toMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
736+ [A] seek :: MonadIO m => Int -> Ring a -> m (Ring a)
737+ [A] createRing :: MonadIO m => Int -> m (Ring a)
738+ [A] copyToMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
739+[C] Streamly.Internal.Data.Ring
740+ [C] Ring
741+ [A] [ringStart] :: Ring a -> {-# UNPACK #-} !ForeignPtr a
742+ [R] [ringMax] :: Ring a -> !Int
743+ [R] [ringHead] :: Ring a -> !Int
744+ [A] [ringBound] :: Ring a -> {-# UNPACK #-} !Ptr a
745+ [R] [ringArr] :: Ring a -> MutArray a
746+ [C] Ring
747+ [O] Ring :: MutArray a -> !Int -> !Int -> Ring a
748+ [N] Ring :: {-# UNPACK #-} !ForeignPtr a -> {-# UNPACK #-} !Ptr a -> Ring a
749+ [A] GHC.Show.Show
750+ [A] instance (GHC.Show.Show a, GHC.Show.Show b, GHC.Show.Show c, GHC.Show.Show d) => GHC.Show.Show (Streamly.Internal.Data.Ring.Tuple4' a b c d)
751+ [A] writeN :: Int -> Fold m a (Ring a)
752+ [R] writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
753+ [R] unsafeInsertRingWith :: Ring a -> a -> IO Int
754+ [A] unsafeInsert :: Storable a => Ring a -> Ptr a -> a -> IO (Ptr a)
755+ [A] unsafeFoldRingNM :: forall m a b. (MonadIO m, Storable a) => Int -> Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
756+ [A] unsafeFoldRingM :: forall m a b. (MonadIO m, Storable a) => Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
757+ [A] unsafeFoldRingFullM :: forall m a b. (MonadIO m, Storable a) => Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
758+ [A] unsafeFoldRing :: forall a b. Storable a => Ptr a -> (b -> a -> b) -> b -> Ring a -> b
759+ [A] unsafeEqArrayN :: Ring a -> Ptr a -> Array a -> Int -> Bool
760+ [A] unsafeEqArray :: Ring a -> Ptr a -> Array a -> Bool
761+ [R] toStreamWith :: Int -> Ring a -> Stream m a
762+ [R] toMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
763+ [A] startOf :: Ring a -> Ptr a
764+ [A] slidingWindowWith :: forall m a b. (MonadIO m, Storable a, Unbox a) => Int -> Fold m ((a, Maybe a), m (MutArray a)) b -> Fold m a b
765+ [A] slidingWindow :: forall m a b. (MonadIO m, Storable a, Unbox a) => Int -> Fold m (a, Maybe a) b -> Fold m a b
766+ [A] slide :: Ring a -> a -> m (Ring a)
767+ [R] seek :: MonadIO m => Int -> Ring a -> m (Ring a)
768+ [A] ringsOf :: Int -> Stream m a -> Stream m (MutArray a)
769+ [A] readRev :: Unfold m (MutArray a) a
770+ [A] read :: forall m a. (MonadIO m, Storable a) => Unfold m (Ring a, Ptr a, Int) a
771+ [A] putIndex :: Ring a -> Int -> a -> m ()
772+ [A] newRing :: Int -> m (Ring a)
773+ [A] new :: forall a. Storable a => Int -> IO (Ring a, Ptr a)
774+ [A] moveBy :: forall a. Storable a => Int -> Ring a -> Ptr a -> Ptr a
775+ [A] modifyIndex :: Ring a -> Int -> (a -> (a, b)) -> m b
776+ [A] length :: Ring a -> Int
777+ [A] getIndexUnsafe :: Ring a -> Int -> m a
778+ [A] getIndexRev :: Ring a -> Int -> m a
779+ [A] getIndex :: Ring a -> Int -> m a
780+ [A] fromArray :: MutArray a -> Ring a
781+ [R] createRing :: MonadIO m => Int -> m (Ring a)
782+ [A] castUnsafe :: Ring a -> Ring b
783+ [A] cast :: forall a b. Storable b => Ring a -> Maybe (Ring b)
784+ [A] bytesFree :: Ring a -> Int
785+ [A] byteLength :: Ring a -> Int
786+ [A] byteCapacity :: Ring a -> Int
787+ [A] asBytes :: Ring a -> Ring Word8
788+ [A] advance :: forall a. Storable a => Ring a -> Ptr a -> Ptr a
789+[R] Streamly.Internal.Data.Producer.Type
790+[R] Streamly.Internal.Data.Producer.Source
791+[C] Streamly.Internal.Data.Producer
792+ [A] Source
793+ [A] unread :: [b] -> Source a b -> Source a b
794+ [A] translate :: Functor m => (a -> c) -> (c -> a) -> Producer m c b -> Producer m a b
795+ [A] source :: Maybe a -> Source a b
796+ [A] producer :: Monad m => Producer m a b -> Producer m (Source a b) b
797+ [A] parseManyD :: Monad m => Parser a m b -> Producer m (Source x a) a -> Producer m (Source x a) (Either ParseError b)
798+ [A] parseMany :: Monad m => Parser a m b -> Producer m (Source x a) a -> Producer m (Source x a) (Either ParseError b)
799+ [A] parse :: Monad m => Parser a m b -> Producer m (Source s a) a -> Source s a -> m (Either ParseError b, Source s a)
800+ [A] lmap :: (a -> a) -> Producer m a b -> Producer m a b
801+ [A] isEmpty :: Source a b -> Bool
802+[R] Streamly.Internal.Data.Pipe.Type
803+[C] Streamly.Internal.Data.Pipe
804+ [A] Step
805+ [A] Yield :: a -> s -> Step s a
806+ [A] Continue :: s -> Step s a
807+ [A] PipeState
808+ [A] Produce :: s2 -> PipeState s1 s2
809+ [A] Consume :: s1 -> PipeState s1 s2
810+ [C] Pipe
811+ [A] Pipe :: (s1 -> a -> m (Step (PipeState s1 s2) b)) -> (s2 -> m (Step (PipeState s1 s2) b)) -> s1 -> Pipe m a b
812+[A] Streamly.Internal.Data.ParserK
813+ [A] Step
814+ [A] Partial :: !Int -> (Input a -> m (Step a m r)) -> Step a m r
815+ [A] Error :: !Int -> String -> Step a m r
816+ [A] Done :: !Int -> r -> Step a m r
817+ [A] Continue :: !Int -> (Input a -> m (Step a m r)) -> Step a m r
818+ [A] ParseResult
819+ [A] Success :: !Int -> !b -> ParseResult b
820+ [A] Failure :: !Int -> !String -> ParseResult b
821+ [A] Input
822+ [A] None :: Input a
823+ [A] Chunk :: a -> Input a
824+ [A] ParserK
825+ [A] [runParser] :: ParserK a m b -> forall r. (ParseResult b -> Int -> Input a -> m (Step a m r)) -> Int -> Int -> Input a -> m (Step a m r)
826+ [A] MkParser :: (forall r. (ParseResult b -> Int -> Input a -> m (Step a m r)) -> Int -> Int -> Input a -> m (Step a m r)) -> ParserK a m b
827+ [A] fromPure :: b -> ParserK a m b
828+ [A] fromEffect :: Monad m => m b -> ParserK a m b
829+ [A] die :: String -> ParserK a m b
830+ [A] adaptCG :: Monad m => Parser a m b -> ParserK (Array a) m b
831+ [A] adaptC :: (Monad m, Unbox a) => Parser a m b -> ParserK (Array a) m b
832+ [A] adapt :: Monad m => Parser a m b -> ParserK a m b
833+[R] Streamly.Internal.Data.Parser.ParserK.Type
834+[R] Streamly.Internal.Data.Parser.ParserD.Type
835+[R] Streamly.Internal.Data.Parser.ParserD.Tee
836+[R] Streamly.Internal.Data.Parser.ParserD
837+[C] Streamly.Internal.Data.Parser
838+ [A] Step
839+ [A] Partial :: !Int -> !s -> Step s b
840+ [A] Error :: !String -> Step s b
841+ [A] Done :: !Int -> !b -> Step s b
842+ [A] Continue :: !Int -> !s -> Step s b
843+ [A] Parser
844+ [A] Parser :: (s -> a -> m (Step s b)) -> m (Initial s b) -> (s -> m (Step s b)) -> Parser a m b
845+ [A] Initial
846+ [A] IPartial :: !s -> Initial s b
847+ [A] IError :: !String -> Initial s b
848+ [A] IDone :: !b -> Initial s b
849+ [A] GHC.Show.Show
850+ [A] instance (GHC.Show.Show a, GHC.Show.Show b) => GHC.Show.Show (Streamly.Internal.Data.Parser.Tuple'Fused a b)
851+ [A] ParseError
852+ [A] ParseError :: String -> ParseError
853+ [A] zipWithM :: Monad m => (a -> b -> m c) -> Stream m a -> Fold m c x -> Parser b m x
854+ [A] zip :: Monad m => Stream m a -> Fold m (a, b) x -> Parser b m x
855+ [A] wordWithQuotes :: (Monad m, Eq a) => Bool -> (a -> a -> Maybe a) -> a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
856+ [A] wordProcessQuotes :: (Monad m, Eq a) => a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
857+ [A] wordKeepQuotes :: (Monad m, Eq a) => a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
858+ [A] wordFramedBy :: Monad m => (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
859+ [A] wordBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
860+ [A] toFold :: Monad m => Parser a m b -> Fold m a b
861+ [A] takeWhileP :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
862+ [A] takeWhile1 :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
863+ [A] takeWhile :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
864+ [A] takeStartBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
865+ [A] takeStartBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
866+ [A] takeP :: Monad m => Int -> Parser a m b -> Parser a m b
867+ [A] takeGE :: Monad m => Int -> Fold m a b -> Parser a m b
868+ [A] takeFramedBy_ :: Monad m => (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
869+ [A] takeFramedByGeneric :: Monad m => Maybe (a -> Bool) -> Maybe (a -> Bool) -> Maybe (a -> Bool) -> Fold m a b -> Parser a m b
870+ [A] takeFramedByEsc_ :: Monad m => (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
871+ [A] takeEndBy_ :: (a -> Bool) -> Parser a m b -> Parser a m b
872+ [A] takeEndByEsc :: Monad m => (a -> Bool) -> (a -> Bool) -> Parser a m b -> Parser a m b
873+ [A] takeEndBy :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
874+ [A] takeEitherSepBy :: (a -> Bool) -> Fold m (Either a b) c -> Parser a m c
875+ [A] takeEQ :: Monad m => Int -> Fold m a b -> Parser a m b
876+ [A] takeBetween :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
877+ [A] subsequenceBy :: (a -> a -> Bool) -> Stream m a -> Parser a m ()
878+ [A] streamEqBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Parser a m ()
879+ [A] split_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
880+ [A] splitWith :: Monad m => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
881+ [A] splitSome :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
882+ [A] splitManyPost :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
883+ [A] splitMany :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
884+ [A] spanByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
885+ [A] spanBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
886+ [A] span :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
887+ [A] some :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
888+ [A] sequence :: Monad m => Stream m (Parser a m b) -> Fold m b c -> Parser a m c
889+ [A] sepByAll :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
890+ [A] sepBy1 :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
891+ [A] sepBy :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
892+ [A] satisfy :: Monad m => (a -> Bool) -> Parser a m a
893+ [A] sampleFromthen :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
894+ [A] roundRobin :: t (Parser a m b) -> Fold m b c -> Parser a m c
895+ [A] rmapM :: Monad m => (b -> m c) -> Parser a m b -> Parser a m c
896+ [A] retryMaxTotal :: Int -> Parser a m b -> Fold m b c -> Parser a m c
897+ [A] retryMaxSuccessive :: Int -> Parser a m b -> Fold m b c -> Parser a m c
898+ [A] retry :: Parser a m b -> Parser a m b
899+ [A] postscan :: Fold m a b -> Parser b m c -> Parser a m c
900+ [A] peek :: Monad m => Parser a m a
901+ [A] oneOf :: (Monad m, Eq a, Foldable f) => f a -> Parser a m a
902+ [A] oneNotEq :: (Monad m, Eq a) => a -> Parser a m a
903+ [A] oneEq :: (Monad m, Eq a) => a -> Parser a m a
904+ [A] one :: Monad m => Parser a m a
905+ [A] noneOf :: (Monad m, Eq a, Foldable f) => f a -> Parser a m a
906+ [A] noErrorUnsafeSplit_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
907+ [A] noErrorUnsafeSplitWith :: Monad m => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
908+ [A] noErrorUnsafeConcatMap :: Monad m => (b -> Parser a m c) -> Parser a m b -> Parser a m c
909+ [A] maybe :: Monad m => (a -> Maybe b) -> Parser a m b
910+ [A] manyTillP :: Parser a m b -> Parser a m x -> Parser b m c -> Parser a m c
911+ [A] manyTill :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
912+ [A] manyThen :: Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
913+ [A] manyP :: Parser a m b -> Parser b m c -> Parser a m c
914+ [A] many :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
915+ [A] makeIndexFilter :: (Fold m (s, a) b -> Parser a m b) -> (((s, a) -> Bool) -> Fold m (s, a) b -> Fold m (s, a) b) -> ((s, a) -> Bool) -> Fold m a b -> Parser a m b
916+ [A] lookAhead :: Monad m => Parser a m b -> Parser a m b
917+ [A] lmapM :: Monad m => (a -> m b) -> Parser b m r -> Parser a m r
918+ [A] lmap :: (a -> b) -> Parser b m r -> Parser a m r
919+ [A] listEqBy :: Monad m => (a -> a -> Bool) -> [a] -> Parser a m [a]
920+ [A] listEq :: (Monad m, Eq a) => [a] -> Parser a m [a]
921+ [A] indexed :: forall m a b. Monad m => Fold m (Int, a) b -> Parser a m b
922+ [A] groupByRollingEither :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
923+ [A] groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
924+ [A] groupBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
925+ [A] fromPure :: Monad m => b -> Parser a m b
926+ [A] fromFoldMaybe :: Monad m => String -> Fold m a (Maybe b) -> Parser a m b
927+ [A] fromFold :: Monad m => Fold m a b -> Parser a m b
928+ [A] fromEffect :: Monad m => m b -> Parser a m b
929+ [A] filter :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
930+ [A] extractStep :: Monad m => (s -> m (Step s1 b)) -> Step s b -> m (Step s1 b)
931+ [A] eof :: Monad m => Parser a m ()
932+ [A] either :: Monad m => (a -> Either String b) -> Parser a m b
933+ [A] dropWhile :: Monad m => (a -> Bool) -> Parser a m ()
934+ [A] dieM :: Monad m => m String -> Parser a m b
935+ [A] die :: Monad m => String -> Parser a m b
936+ [A] deintercalateAll :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
937+ [A] deintercalate1 :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
938+ [A] deintercalate :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
939+ [A] countBetween :: Int -> Int -> Parser a m b -> Fold m b c -> Parser a m c
940+ [A] count :: Int -> Parser a m b -> Fold m b c -> Parser a m c
941+ [A] concatMap :: Monad m => (b -> Parser a m c) -> Parser a m b -> Parser a m c
942+ [A] blockWithQuotes :: (Monad m, Eq a) => (a -> Bool) -> (a -> Bool) -> a -> a -> Fold m a b -> Parser a m b
943+ [A] bimapOverrideCount :: Int -> (s -> s1) -> (b -> b1) -> Step s b -> Step s1 b1
944+ [A] alt :: Monad m => Parser x m a -> Parser x m a -> Parser x m a
945+[A] Streamly.Internal.Data.MutByteArray
946+ [A] class Unbox a
947+ [A] class SizeOfRep (f :: Type -> Type)
948+ [A] class Serialize a
949+ [A] TypeOfType
950+ [A] UnitType :: Name -> TypeOfType
951+ [A] TheType :: SimpleDataCon -> TypeOfType
952+ [A] MultiType :: [SimpleDataCon] -> TypeOfType
953+ [A] SimpleDataCon
954+ [A] SimpleDataCon :: Name -> [Field] -> SimpleDataCon
955+ [A] SerializeConfig
956+ [A] [cfgRecordSyntaxWithHeader] :: SerializeConfig -> Bool
957+ [A] [cfgInlineSize] :: SerializeConfig -> Maybe Inline
958+ [A] [cfgInlineSerialize] :: SerializeConfig -> Maybe Inline
959+ [A] [cfgInlineDeserialize] :: SerializeConfig -> Maybe Inline
960+ [A] [cfgConstructorTagAsString] :: SerializeConfig -> Bool
961+ [A] SerializeConfig :: Maybe Inline -> Maybe Inline -> Maybe Inline -> Bool -> Bool -> SerializeConfig
962+ [A] PinnedState
963+ [A] Unpinned :: PinnedState
964+ [A] Pinned :: PinnedState
965+ [A] MutByteArray
966+ [A] MutByteArray :: MutableByteArray# RealWorld -> MutByteArray
967+ [A] DataType
968+ [A] [dtTvs] :: DataType -> [Name]
969+ [A] [dtName] :: DataType -> Name
970+ [A] [dtCxt] :: DataType -> Cxt
971+ [A] [dtCons] :: DataType -> [DataCon]
972+ [A] DataType :: Name -> [Name] -> Cxt -> [DataCon] -> DataType
973+ [A] DataCon
974+ [A] [dcTvs] :: DataCon -> [Name]
975+ [A] [dcName] :: DataCon -> Name
976+ [A] [dcFields] :: DataCon -> [(Maybe Name, Type)]
977+ [A] [dcCxt] :: DataCon -> Cxt
978+ [A] DataCon :: Name -> [Name] -> Cxt -> [(Maybe Name, Type)] -> DataCon
979+ [A] BoundedPtr
980+ [A] BoundedPtr :: MutByteArray -> Int -> Int -> BoundedPtr
981+ [A] Streamly.Internal.Data.Serialize.Type.Serialize
982+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize Streamly.Internal.Data.MutByteArray.LiftedInteger
983+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize GHC.Num.Integer.Integer
984+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize a => Streamly.Internal.Data.Serialize.Type.Serialize (GHC.Maybe.Maybe a)
985+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize (Data.Proxy.Proxy a)
986+ [A] instance (Streamly.Internal.Data.Serialize.Type.Serialize a, Streamly.Internal.Data.Serialize.Type.Serialize b) => Streamly.Internal.Data.Serialize.Type.Serialize (Data.Either.Either a b)
987+ [A] Peeker
988+ [A] Peeker :: Builder BoundedPtr IO a -> Peeker a
989+ [A] type Field = (Maybe Name, Type)
990+ [A] xorCmp :: [Word8] -> Name -> Name -> Q Exp
991+ [A] wListToString :: [Word8] -> String
992+ [A] w8_int :: Word8 -> Int
993+ [A] w32_int :: Word32 -> Int
994+ [A] unpin :: MutByteArray -> IO MutByteArray
995+ [A] typeOfType :: Type -> [DataCon] -> TypeOfType
996+ [A] skipByte :: Peeker ()
997+ [A] sizeOfRep :: SizeOfRep f => f x -> Int
998+ [A] sizeOfMutableByteArray :: MutByteArray -> IO Int
999+ [A] sizeOf :: (Unbox a, SizeOfRep (Rep a)) => Proxy a -> Int
1000+ [A] simplifyDataCon :: DataCon -> SimpleDataCon
1001+ [A] serializeW8List :: Name -> Name -> [Word8] -> Q Exp
1002+ [A] serializeConfig :: SerializeConfig
1003+ [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
1004+ [A] runPeeker :: Peeker a -> BoundedPtr -> IO a
1005+ [A] reifyDataType :: Name -> Q DataType
1006+ [A] readUnsafe :: Unbox a => Peeker a
1007+ [A] read :: Unbox a => Peeker a
1008+ [A] putSliceUnsafe :: MonadIO m => MutByteArray -> Int -> MutByteArray -> Int -> Int -> m ()
1009+ [A] pokeRep :: PokeRep f => f a -> BoundedPtr -> IO BoundedPtr
1010+ [A] pokeBoundedPtrUnsafe :: forall a. Unbox a => a -> BoundedPtr -> IO BoundedPtr
1011+ [A] pokeBoundedPtr :: forall a. Unbox a => a -> BoundedPtr -> IO BoundedPtr
1012+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
1013+ [A] pinnedNewAlignedBytes :: Int -> Int -> IO MutByteArray
1014+ [A] pinnedNew :: Int -> IO MutByteArray
1015+ [A] pinnedCloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
1016+ [A] pin :: MutByteArray -> IO MutByteArray
1017+ [A] peekRep :: PeekRep f => Peeker (f x)
1018+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
1019+ [A] openConstructor :: Name -> Int -> Q Pat
1020+ [A] nil :: MutByteArray
1021+ [A] newBytesAs :: PinnedState -> Int -> IO MutByteArray
1022+ [A] new :: Int -> IO MutByteArray
1023+ [A] mkSerializeExprFields :: Name -> [Field] -> Q Exp
1024+ [A] mkRecSizeOfExpr :: SimpleDataCon -> Q Exp
1025+ [A] mkRecSerializeExpr :: Name -> SimpleDataCon -> Q Exp
1026+ [A] mkRecDeserializeExpr :: Name -> Name -> Name -> SimpleDataCon -> Q Exp
1027+ [A] mkFieldName :: Int -> Name
1028+ [A] mkDeserializeKeysDec :: Name -> Name -> SimpleDataCon -> Q [Dec]
1029+ [A] mkDeserializeExprOne :: Name -> SimpleDataCon -> Q Exp
1030+ [A] matchConstructor :: Name -> Int -> Q Exp -> Q Match
1031+ [A] makeN :: Int -> Name
1032+ [A] makeI :: Int -> Name
1033+ [A] makeA :: Int -> Name
1034+ [A] litProxy :: Unbox a => Proxy a -> Q Exp
1035+ [A] litIntegral :: Integral a => a -> Q Exp
1036+ [A] isUnitType :: [DataCon] -> Bool
1037+ [A] isRecordSyntax :: SimpleDataCon -> Bool
1038+ [A] isPinned :: MutByteArray -> Bool
1039+ [A] int_w8 :: Int -> Word8
1040+ [A] int_w32 :: Int -> Word32
1041+ [A] inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
1042+ [A] inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
1043+ [A] inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
1044+ [A] getMutableByteArray# :: MutByteArray -> MutableByteArray# RealWorld
1045+ [A] genericSizeOf :: forall a. SizeOfRep (Rep a) => Proxy a -> Int
1046+ [A] genericPokeByteIndex :: (Generic a, PokeRep (Rep a)) => MutByteArray -> Int -> a -> IO ()
1047+ [A] genericPeekByteIndex :: (Generic a, PeekRep (Rep a)) => MutByteArray -> Int -> IO a
1048+ [A] errorUnsupported :: String -> a
1049+ [A] errorUnimplemented :: a
1050+ [A] encodeRecordFields :: Bool -> SerializeConfig -> SerializeConfig
1051+ [A] encodeConstrNames :: Bool -> SerializeConfig -> SerializeConfig
1052+ [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
1053+ [A] deriveUnbox :: Q [Dec] -> Q [Dec]
1054+ [A] deriveSerializeWith :: (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
1055+ [A] deriveSerialize :: Q [Dec] -> Q [Dec]
1056+ [A] conUpdateFuncDec :: Name -> [Field] -> Q [Dec]
1057+ [A] cloneSliceUnsafeAs :: MonadIO m => PinnedState -> Int -> Int -> MutByteArray -> m MutByteArray
1058+ [A] cloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
1059+ [A] c2w :: Char -> Word8
1060+ [A] asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
1061+ [A] addSizeTo :: Serialize a => Int -> a -> Int
1062+ [A] _x :: Name
1063+ [A] _val :: Name
1064+ [A] _tag :: Name
1065+ [A] _initialOffset :: Name
1066+ [A] _endOffset :: Name
1067+ [A] _arr :: Name
1068+ [A] _acc :: Name
1069+[A] Streamly.Internal.Data.MutArray.Stream
1070+ [A] SpliceState
1071+ [A] SpliceYielding :: arr -> SpliceState s arr -> SpliceState s arr
1072+ [A] SpliceInitial :: s -> SpliceState s arr
1073+ [A] SpliceFinish :: SpliceState s arr
1074+ [A] SpliceBuffering :: s -> arr -> SpliceState s arr
1075+ [A] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
1076+ [A] splitOn :: (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> Stream m (MutArray a)
1077+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1078+ [A] packArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1079+ [A] lpackArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
1080+ [A] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
1081+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1082+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1083+ [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
1084+ [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1085+ [A] compactEQ :: Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1086+ [A] compact :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1087+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1088+[A] Streamly.Internal.Data.MutArray.Generic
1089+ [A] MutArray
1090+ [A] [arrTrueLen] :: MutArray a -> {-# UNPACK #-} !Int
1091+ [A] [arrStart] :: MutArray a -> {-# UNPACK #-} !Int
1092+ [A] [arrLen] :: MutArray a -> {-# UNPACK #-} !Int
1093+ [A] [arrContents#] :: MutArray a -> MutableArray# RealWorld a
1094+ [A] MutArray :: MutableArray# RealWorld a -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> MutArray a
1095+ [A] writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
1096+ [A] writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
1097+ [A] writeN :: MonadIO m => Int -> Fold m a (MutArray a)
1098+ [A] write :: MonadIO m => Fold m a (MutArray a)
1099+ [A] uninit :: MonadIO m => MutArray a -> Int -> m (MutArray a)
1100+ [A] toStreamK :: MonadIO m => MutArray a -> StreamK m a
1101+ [A] toList :: MonadIO m => MutArray a -> m [a]
1102+ [A] strip :: MonadIO m => (a -> Bool) -> MutArray a -> m (MutArray a)
1103+ [A] snocWith :: MonadIO m => (Int -> Int) -> MutArray a -> a -> m (MutArray a)
1104+ [A] snocUnsafe :: MonadIO m => MutArray a -> a -> m (MutArray a)
1105+ [A] snoc :: MonadIO m => MutArray a -> a -> m (MutArray a)
1106+ [A] realloc :: MonadIO m => Int -> MutArray a -> m (MutArray a)
1107+ [A] reader :: MonadIO m => Unfold m (MutArray a) a
1108+ [A] readRev :: MonadIO m => MutArray a -> Stream m a
1109+ [A] read :: MonadIO m => MutArray a -> Stream m a
1110+ [A] putSliceUnsafe :: MonadIO m => MutArray a -> Int -> MutArray a -> Int -> Int -> m ()
1111+ [A] putIndices :: MonadIO m => MutArray a -> Fold m (Int, a) ()
1112+ [A] putIndexUnsafe :: forall m a. MonadIO m => Int -> MutArray a -> a -> m ()
1113+ [A] putIndex :: MonadIO m => Int -> MutArray a -> a -> m ()
1114+ [A] producerWith :: Monad m => (forall b. IO b -> m b) -> Producer m (MutArray a) a
1115+ [A] producer :: MonadIO m => Producer m (MutArray a) a
1116+ [A] nil :: MonadIO m => m (MutArray a)
1117+ [A] new :: MonadIO m => Int -> m (MutArray a)
1118+ [A] modifyIndexUnsafe :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
1119+ [A] modifyIndex :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
1120+ [A] length :: MutArray a -> Int
1121+ [A] getSliceUnsafe :: Int -> Int -> MutArray a -> MutArray a
1122+ [A] getSlice :: Int -> Int -> MutArray a -> MutArray a
1123+ [A] getIndexUnsafeWith :: MonadIO m => MutableArray# RealWorld a -> Int -> m a
1124+ [A] getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
1125+ [A] getIndex :: MonadIO m => Int -> MutArray a -> m (Maybe a)
1126+ [A] fromStreamN :: MonadIO m => Int -> Stream m a -> m (MutArray a)
1127+ [A] fromStream :: MonadIO m => Stream m a -> m (MutArray a)
1128+ [A] fromPureStream :: MonadIO m => Stream Identity a -> m (MutArray a)
1129+ [A] fromListN :: MonadIO m => Int -> [a] -> m (MutArray a)
1130+ [A] fromList :: MonadIO m => [a] -> m (MutArray a)
1131+ [A] eq :: (MonadIO m, Eq a) => MutArray a -> MutArray a -> m Bool
1132+ [A] cmp :: (MonadIO m, Ord a) => MutArray a -> MutArray a -> m Ordering
1133+ [A] clone :: MonadIO m => MutArray a -> m (MutArray a)
1134+ [A] chunksOf :: forall m a. MonadIO m => Int -> Stream m a -> Stream m (MutArray a)
1135+[A] Streamly.Internal.Data.MutArray
1136+ [A] MutByteArray
1137+ [A] MutArray
1138+ [A] [arrStart] :: MutArray a -> {-# UNPACK #-} !Int
1139+ [A] [arrEnd] :: MutArray a -> {-# UNPACK #-} !Int
1140+ [A] [arrContents] :: MutArray a -> {-# UNPACK #-} !MutByteArray
1141+ [A] [arrBound] :: MutArray a -> {-# UNPACK #-} !Int
1142+ [A] MutArray :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> MutArray a
1143+ [A] IORef
1144+ [A] ArrayUnsafe
1145+ [A] ArrayUnsafe :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> ArrayUnsafe a
1146+ [A] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1147+ [A] writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1148+ [A] writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1149+ [A] writeNWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1150+ [A] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1151+ [A] writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1152+ [A] writeIORef :: Unbox a => IORef a -> a -> IO ()
1153+ [A] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
1154+ [A] writeAppendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
1155+ [A] writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
1156+ [A] writeAppendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
1157+ [A] writeAppend :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
1158+ [A] write :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1159+ [A] unsafeSwapIndices :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> MutArray a -> m ()
1160+ [A] unpin :: MutArray a -> IO (MutArray a)
1161+ [A] toStreamKWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> StreamK m a
1162+ [A] toStreamKRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> StreamK m a
1163+ [A] toStreamKRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
1164+ [A] toStreamK :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
1165+ [A] toStreamDWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
1166+ [A] toStreamDRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
1167+ [A] toList :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m [a]
1168+ [A] swapIndices :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> MutArray a -> m ()
1169+ [A] strip :: forall a m. (Unbox a, MonadIO m) => (a -> Bool) -> MutArray a -> m (MutArray a)
1170+ [A] splitOn :: (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> Stream m (MutArray a)
1171+ [A] splitAt :: forall a. Unbox a => Int -> MutArray a -> (MutArray a, MutArray a)
1172+ [A] spliceWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int -> Int) -> MutArray a -> MutArray a -> m (MutArray a)
1173+ [A] spliceUnsafe :: MonadIO m => MutArray a -> MutArray a -> m (MutArray a)
1174+ [A] spliceExp :: (MonadIO m, Unbox a) => MutArray a -> MutArray a -> m (MutArray a)
1175+ [A] spliceCopy :: forall m a. MonadIO m => MutArray a -> MutArray a -> m (MutArray a)
1176+ [A] splice :: (MonadIO m, Unbox a) => MutArray a -> MutArray a -> m (MutArray a)
1177+ [A] snocWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> MutArray a -> a -> m (MutArray a)
1178+ [A] snocUnsafe :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1179+ [A] snocMay :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (Maybe (MutArray a))
1180+ [A] snocLinear :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1181+ [A] snoc :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1182+ [A] shuffleBy :: (a -> a -> m Bool) -> MutArray a -> MutArray a -> m ()
1183+ [A] roundUpToPower2 :: Int -> Int
1184+ [A] rightSize :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m (MutArray a)
1185+ [A] reverse :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m ()
1186+ [A] resizeExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1187+ [A] resize :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1188+ [A] realloc :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1189+ [A] readerRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Unfold m (MutArray a) a
1190+ [A] readerRev :: forall m a. (MonadIO m, Unbox a) => Unfold m (MutArray a) a
1191+ [A] reader :: forall m a. (MonadIO m, Unbox a) => Unfold m (MutArray a) a
1192+ [A] readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
1193+ [A] readIORef :: Unbox a => IORef a -> IO a
1194+ [A] read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
1195+ [A] putIndices :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Fold m (Int, a) ()
1196+ [A] putIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
1197+ [A] putIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
1198+ [A] producerWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Producer m (MutArray a) a
1199+ [A] producer :: forall m a. (MonadIO m, Unbox a) => Producer m (MutArray a) a
1200+ [A] pollIntIORef :: (MonadIO m, Unbox a) => IORef a -> Stream m a
1201+ [A] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1202+ [A] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (MutArray a)
1203+ [A] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1204+ [A] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1205+ [A] pinnedNewBytes :: MonadIO m => Int -> m (MutArray a)
1206+ [A] pinnedNewAligned :: (MonadIO m, Unbox a) => Int -> Int -> m (MutArray a)
1207+ [A] pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
1208+ [A] pinnedFromListN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1209+ [A] pinnedFromList :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1210+ [A] pinnedClone :: MonadIO m => MutArray a -> m (MutArray a)
1211+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1212+ [A] pin :: MutArray a -> IO (MutArray a)
1213+ [A] permute :: MutArray a -> m Bool
1214+ [A] partitionBy :: forall m a. (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> m (MutArray a, MutArray a)
1215+ [A] nil :: MutArray a
1216+ [A] newIORef :: forall a. Unbox a => a -> IO (IORef a)
1217+ [A] newArrayWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int -> m MutByteArray) -> Int -> Int -> m (MutArray a)
1218+ [A] new :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
1219+ [A] modifyIndices :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (Int -> a -> a) -> Fold m Int ()
1220+ [A] modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
1221+ [A] modifyIndex :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
1222+ [A] modifyIORef' :: Unbox a => IORef a -> (a -> a) -> IO ()
1223+ [A] modify :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (a -> a) -> m ()
1224+ [A] mergeBy :: Int -> (MutArray a -> MutArray a -> m ()) -> MutArray a -> m ()
1225+ [A] memcpy :: Ptr Word8 -> Ptr Word8 -> Int -> IO ()
1226+ [A] memcmp :: Ptr Word8 -> Ptr Word8 -> Int -> IO Bool
1227+ [A] length :: forall a. Unbox a => MutArray a -> Int
1228+ [A] isPinned :: MutArray a -> Bool
1229+ [A] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
1230+ [A] getSliceUnsafe :: forall a. Unbox a => Int -> Int -> MutArray a -> MutArray a
1231+ [A] getSlice :: forall a. Unbox a => Int -> Int -> MutArray a -> MutArray a
1232+ [A] getIndicesD :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
1233+ [A] getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1234+ [A] getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
1235+ [A] getIndexRev :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
1236+ [A] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
1237+ [A] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
1238+ [A] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
1239+ [A] fromStreamD :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
1240+ [A] fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
1241+ [A] fromPureStream :: (MonadIO m, Unbox a) => Stream Identity a -> m (MutArray a)
1242+ [A] fromListRevN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1243+ [A] fromListRev :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1244+ [A] fromListN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1245+ [A] fromList :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1246+ [A] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
1247+ [A] foldr :: (MonadIO m, Unbox a) => (a -> b -> b) -> b -> MutArray a -> m b
1248+ [A] foldl' :: (MonadIO m, Unbox a) => (b -> a -> b) -> b -> MutArray a -> m b
1249+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1250+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1251+ [A] divideBy :: Int -> (MutArray a -> m (MutArray a, MutArray a)) -> MutArray a -> m ()
1252+ [A] cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
1253+ [A] clone :: MonadIO m => MutArray a -> m (MutArray a)
1254+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1255+ [A] castUnsafe :: MutArray a -> MutArray b
1256+ [A] cast :: forall a b. Unbox b => MutArray a -> Maybe (MutArray b)
1257+ [A] c_memchr :: Ptr Word8 -> Word8 -> CSize -> IO (Ptr Word8)
1258+ [A] bytesFree :: MutArray a -> Int
1259+ [A] byteLength :: MutArray a -> Int
1260+ [A] byteCapacity :: MutArray a -> Int
1261+ [A] bubble :: (MonadIO m, Unbox a) => (a -> a -> Ordering) -> MutArray a -> m ()
1262+ [A] breakOn :: MonadIO m => Word8 -> MutArray Word8 -> m (MutArray Word8, Maybe (MutArray Word8))
1263+ [A] blockSize :: Int
1264+ [A] asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
1265+ [A] asBytes :: MutArray a -> MutArray Word8
1266+ [A] arrayChunkBytes :: Int
1267+ [A] allocBytesToElemCount :: Unbox a => a -> Int -> Int
1268+[C] Streamly.Internal.Data.IsMap
1269+ [A] mapTraverseWithKey :: (IsMap f, Applicative t) => (Key f -> a -> t b) -> f a -> t (f b)
1270+[R] Streamly.Internal.Data.IORef.Unboxed
1271+[R] Streamly.Internal.Data.Fold.Window
1272+[R] Streamly.Internal.Data.Fold.Type
1273+[R] Streamly.Internal.Data.Fold.Tee
1274+[R] Streamly.Internal.Data.Fold.Step
1275+[R] Streamly.Internal.Data.Fold.Container
1276+[D] Streamly.Internal.Data.Fold.Chunked
1277+[C] Streamly.Internal.Data.Fold
1278+ [A] ManyState
1279+ [C] Fold
1280+ [C] Fold
1281+ [O] Fold :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> Fold m a b
1282+ [N] Fold :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> (s -> m b) -> Fold m a b
1283+ [A] windowSumInt :: forall m a. (Monad m, Integral a) => Fold m (a, Maybe a) a
1284+ [A] windowSum :: forall m a. (Monad m, Num a) => Fold m (a, Maybe a) a
1285+ [A] windowRollingMapM :: Monad m => (Maybe a -> a -> m (Maybe b)) -> Fold m (a, Maybe a) (Maybe b)
1286+ [A] windowRollingMap :: Monad m => (Maybe a -> a -> Maybe b) -> Fold m (a, Maybe a) (Maybe b)
1287+ [A] windowRange :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe (a, a))
1288+ [A] windowPowerSumFrac :: (Monad m, Floating a) => a -> Fold m (a, Maybe a) a
1289+ [A] windowPowerSum :: (Monad m, Num a) => Int -> Fold m (a, Maybe a) a
1290+ [A] windowMinimum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
1291+ [A] windowMean :: forall m a. (Monad m, Fractional a) => Fold m (a, Maybe a) a
1292+ [A] windowMaximum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
1293+ [A] windowLmap :: (c -> a) -> Fold m (a, Maybe a) b -> Fold m (c, Maybe c) b
1294+ [A] windowLength :: (Monad m, Num b) => Fold m (a, Maybe a) b
1295+ [A] toSet :: (Monad m, Ord a) => Fold m a (Set a)
1296+ [A] toMapIO :: (MonadIO m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (Map k b)
1297+ [A] toMap :: (Monad m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (Map k b)
1298+ [A] toIntSet :: Monad m => Fold m Int IntSet
1299+ [A] toContainerIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (f b)
1300+ [A] toContainer :: (Monad m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (f b)
1301+ [A] nubInt :: Monad m => Fold m Int (Maybe Int)
1302+ [A] nub :: (Monad m, Ord a) => Fold m a (Maybe a)
1303+ [A] mapMStep :: Applicative m => (a -> m b) -> Step s a -> m (Step s b)
1304+ [A] kvToMap :: (Monad m, Ord k) => Fold m a b -> Fold m (k, a) (Map k b)
1305+ [A] frequency :: (Monad m, Ord a) => Fold m a (Map a Int)
1306+ [A] demuxToMapIO :: (MonadIO m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (Map k b)
1307+ [A] demuxToMap :: (Monad m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (Map k b)
1308+ [A] demuxToContainerIO :: (MonadIO m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
1309+ [A] demuxToContainer :: (Monad m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
1310+ [A] demuxKvToMap :: (Monad m, Ord k) => (k -> m (Fold m a b)) -> Fold m (k, a) (Map k b)
1311+ [A] demuxKvToContainer :: (Monad m, IsMap f, Traversable f) => (Key f -> m (Fold m a b)) -> Fold m (Key f, a) (f b)
1312+ [A] demuxIO :: (MonadIO m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (m (Map k b), Maybe (k, b))
1313+ [A] demuxGenericIO :: (MonadIO m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (m (f b), Maybe (Key f, b))
1314+ [A] demuxGeneric :: (Monad m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (m (f b), Maybe (Key f, b))
1315+ [A] demux :: (Monad m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (m (Map k b), Maybe (k, b))
1316+ [A] cumulative :: Fold m (a, Maybe a) b -> Fold m a b
1317+ [A] countDistinctInt :: Monad m => Fold m Int Int
1318+ [A] countDistinct :: (Monad m, Ord a) => Fold m a Int
1319+ [A] classifyIO :: (MonadIO m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
1320+ [A] classifyGenericIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
1321+ [A] classifyGeneric :: (Monad m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
1322+ [A] classify :: (Monad m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
1323+ [A] chainStepM :: Applicative m => (s1 -> m s2) -> (a -> m (Step s2 b)) -> Step s1 a -> m (Step s2 b)
1324+[C] Streamly.Internal.Data.Builder
1325+ [C] Builder
1326+ [C] Builder
1327+ [O] Builder :: (s -> m (s, a)) -> Builder s m a
1328+ [N] Builder :: (s -> m (a, s)) -> Builder s m a
1329+[A] Streamly.Internal.Data.Binary.Stream
1330+ [A] class ToBytes a
1331+ [A] word8 :: Applicative m => Word8 -> Stream m Word8
1332+ [A] word64le :: Monad m => Word64 -> Stream m Word8
1333+ [A] word64host :: Monad m => Word64 -> Stream m Word8
1334+ [A] word64be :: Monad m => Word64 -> Stream m Word8
1335+ [A] word32le :: Monad m => Word32 -> Stream m Word8
1336+ [A] word32be :: Monad m => Word32 -> Stream m Word8
1337+ [A] word16le :: Monad m => Word16 -> Stream m Word8
1338+ [A] word16be :: Monad m => Word16 -> Stream m Word8
1339+ [A] unit :: Applicative m => Stream m Word8
1340+ [A] toBytes :: ToBytes a => a -> Stream m Word8
1341+ [A] ordering :: Applicative m => Ordering -> Stream m Word8
1342+ [A] int8 :: Applicative m => Int8 -> Stream m Word8
1343+ [A] int64le :: Monad m => Int64 -> Stream m Word8
1344+ [A] int64be :: Monad m => Int64 -> Stream m Word8
1345+ [A] int32le :: Monad m => Int32 -> Stream m Word8
1346+ [A] int32be :: Monad m => Int32 -> Stream m Word8
1347+ [A] int16le :: Monad m => Int16 -> Stream m Word8
1348+ [A] int16be :: Monad m => Int16 -> Stream m Word8
1349+ [A] float32le :: Monad m => Float -> Stream m Word8
1350+ [A] float32be :: Monad m => Float -> Stream m Word8
1351+ [A] double64le :: Monad m => Double -> Stream m Word8
1352+ [A] double64be :: Monad m => Double -> Stream m Word8
1353+ [A] charUtf8 :: Monad m => Char -> Stream m Word8
1354+ [A] charLatin1 :: Applicative m => Char -> Stream m Word8
1355+ [A] bool :: Applicative m => Bool -> Stream m Word8
1356+[A] Streamly.Internal.Data.Binary.Parser
1357+ [A] class FromBytes a
1358+ [A] word8 :: Monad m => Parser Word8 m Word8
1359+ [A] word64le :: Monad m => Parser Word8 m Word64
1360+ [A] word64host :: MonadIO m => Parser Word8 m Word64
1361+ [A] word64be :: Monad m => Parser Word8 m Word64
1362+ [A] word32le :: Monad m => Parser Word8 m Word32
1363+ [A] word32be :: Monad m => Parser Word8 m Word32
1364+ [A] word16le :: Monad m => Parser Word8 m Word16
1365+ [A] word16be :: Monad m => Parser Word8 m Word16
1366+ [A] unit :: Monad m => Parser Word8 m ()
1367+ [A] ordering :: Monad m => Parser Word8 m Ordering
1368+ [A] int8 :: Monad m => Parser Word8 m Int8
1369+ [A] int64le :: Monad m => Parser Word8 m Int64
1370+ [A] int64be :: Monad m => Parser Word8 m Int64
1371+ [A] int32le :: Monad m => Parser Word8 m Int32
1372+ [A] int32be :: Monad m => Parser Word8 m Int32
1373+ [A] int16le :: Monad m => Parser Word8 m Int16
1374+ [A] int16be :: Monad m => Parser Word8 m Int16
1375+ [A] fromBytes :: FromBytes a => Parser Word8 m a
1376+ [A] float32le :: MonadIO m => Parser Word8 m Float
1377+ [A] float32be :: MonadIO m => Parser Word8 m Float
1378+ [A] eqWord8 :: Monad m => Word8 -> Parser Word8 m Word8
1379+ [A] double64le :: MonadIO m => Parser Word8 m Double
1380+ [A] double64be :: MonadIO m => Parser Word8 m Double
1381+ [A] charLatin1 :: Monad m => Parser Word8 m Char
1382+ [A] bool :: Monad m => Parser Word8 m Bool
1383+[R] Streamly.Internal.Data.Array.Type
1384+[A] Streamly.Internal.Data.Array.Stream
1385+ [A] unlines :: forall m a. (MonadIO m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1386+ [A] toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
1387+ [A] splitOnSuffix :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
1388+ [A] splitOn :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
1389+ [A] runArrayParserDBreak :: forall m a b. (MonadIO m, Unbox a) => Parser (Array a) m b -> Stream m (Array a) -> m (Either ParseError b, Stream m (Array a))
1390+ [A] runArrayFoldMany :: (Monad m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> StreamK m (Either ParseError b)
1391+ [A] runArrayFoldBreak :: (MonadIO m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1392+ [A] runArrayFold :: (MonadIO m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> m (Either ParseError b)
1393+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1394+ [A] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
1395+ [A] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1396+ [A] parseBreak :: (MonadIO m, Unbox a) => Parser a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1397+ [A] lpackArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
1398+ [A] interposeSuffix :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1399+ [A] interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1400+ [A] intercalateSuffix :: (Monad m, Unbox a) => Array a -> Stream m (Array a) -> Stream m a
1401+ [A] foldBreakD :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
1402+ [A] foldBreak :: (MonadIO m, Unbox a) => Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
1403+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1404+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1405+ [A] concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
1406+ [A] concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
1407+ [A] compact :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
1408+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1409+ [A] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
1410+[R] Streamly.Internal.Data.Array.Mut.Type
1411+[R] Streamly.Internal.Data.Array.Mut.Stream
1412+[R] Streamly.Internal.Data.Array.Mut
1413+[R] Streamly.Internal.Data.Array.Generic.Mut.Type
1414+[C] Streamly.Internal.Data.Array.Generic
1415+ [A] getIndex :: Int -> Array a -> Maybe a
1416+ [A] fromPureStream :: Stream Identity a -> Array a
1417+ [A] fromByteStr# :: Addr# -> Array Word8
1418+ [A] chunksOf :: forall m a. MonadIO m => Int -> Stream m a -> Stream m (Array a)
1419+[C] Streamly.Internal.Data.Array
1420+ [A] ArrayUnsafe
1421+ [A] ArrayUnsafe :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> ArrayUnsafe a
1422+ [C] Array
1423+ [A] [arrStart] :: Array a -> {-# UNPACK #-} !Int
1424+ [A] [arrEnd] :: Array a -> {-# UNPACK #-} !Int
1425+ [A] [arrContents] :: Array a -> {-# UNPACK #-} !MutByteArray
1426+ [A] Array :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> Array a
1427+ [A] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1428+ [A] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1429+ [R] writeNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
1430+ [A] unsafeMakePure :: Monad m => Fold IO a b -> Fold m a b
1431+ [A] unsafeIndexIO :: forall a. Unbox a => Int -> Array a -> IO a
1432+ [D] unsafeIndex :: forall a. Unbox a => Int -> Array a -> a
1433+ [A] unsafeFreezeWithShrink :: Unbox a => MutArray a -> Array a
1434+ [A] unpin :: Array a -> IO (Array a)
1435+ [A] toStreamKRev :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a
1436+ [A] toStreamK :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a
1437+ [A] toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
1438+ [A] toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
1439+ [A] splitAt :: Unbox a => Int -> Array a -> (Array a, Array a)
1440+ [A] splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
1441+ [A] serialize :: Serialize a => a -> Array Word8
1442+ [A] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1443+ [A] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
1444+ [A] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1445+ [A] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
1446+ [A] pinnedSerialize :: Serialize a => a -> Array Word8
1447+ [A] pinnedFromListN :: Unbox a => Int -> [a] -> Array a
1448+ [A] pinnedFromList :: Unbox a => [a] -> Array a
1449+ [A] pinnedClone :: MonadIO m => Array a -> m (Array a)
1450+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1451+ [A] pin :: Array a -> IO (Array a)
1452+ [A] nil :: Array a
1453+ [A] isPinned :: Array a -> Bool
1454+ [A] getIndexUnsafe :: forall a. Unbox a => Int -> Array a -> a
1455+ [A] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
1456+ [A] fromStreamD :: forall m a. (MonadIO m, Unbox a) => Stream m a -> m (Array a)
1457+ [A] fromPureStream :: Unbox a => Stream Identity a -> Array a
1458+ [A] fromListRevN :: Unbox a => Int -> [a] -> Array a
1459+ [A] fromListRev :: Unbox a => [a] -> Array a
1460+ [A] fromByteStr# :: Addr# -> Array Word8
1461+ [A] foldr :: Unbox a => (a -> b -> b) -> b -> Array a -> b
1462+ [A] foldl' :: forall a b. Unbox a => (b -> a -> b) -> b -> Array a -> b
1463+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1464+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1465+ [A] encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
1466+ [A] deserialize :: Serialize a => Array Word8 -> a
1467+ [A] clone :: MonadIO m => Array a -> m (Array a)
1468+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1469+ [A] byteLength :: Array a -> Int
1470+ [A] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
1471+ [A] breakOn :: MonadIO m => Word8 -> Array Word8 -> m (Array Word8, Maybe (Array Word8))
core/docs/ApiChangelogs/0.2.0-0.2.2.txtnew file+306
@@ -0,0 +1,306 @@
1+---------------------------------
2+API Annotations
3+---------------------------------
4+
5+[A] : Added
6+[R] : Removed
7+[C] : Changed
8+[O] : Old definition
9+[N] : New definition
10+[D] : Deprecated
11+
12+---------------------------------
13+API diff
14+---------------------------------
15+
16+[C] Streamly.Data.Stream
17+ [A] (FixityR,5)
18+ [A] (FixityR,5)
19+[C] Streamly.Data.MutArray.Generic
20+ [A] emptyOf :: MonadIO m => Int -> m (MutArray a)
21+ [A] createOf :: MonadIO m => Int -> Fold m a (MutArray a)
22+ [A] create :: MonadIO m => Fold m a (MutArray a)
23+[C] Streamly.Data.MutArray
24+ [A] pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
25+ [A] emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
26+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
27+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
28+ [A] appendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
29+ [A] append :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
30+[C] Streamly.Data.Array.Generic
31+ [A] createOf :: MonadIO m => Int -> Fold m a (Array a)
32+ [A] create :: MonadIO m => Fold m a (Array a)
33+[C] Streamly.Data.Array
34+ [A] class Serialize a
35+ [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
36+ [A] pinnedSerialize :: Serialize a => a -> Array Word8
37+ [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
38+ [A] deserialize :: Serialize a => Array Word8 -> a
39+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
40+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
41+ [A] addSizeTo :: Serialize a => Int -> a -> Int
42+
43+---------------------------------
44+Internal API diff
45+---------------------------------
46+
47+[C] Streamly.Internal.Unicode.Stream
48+ [A] encodeUtf16le' :: Stream m Char -> Stream m Word16
49+ [A] decodeUtf16le' :: Stream m Word16 -> Stream m Char
50+[A] Streamly.Internal.FileSystem.Path
51+ [A] class IsPath a
52+ [A] Rel
53+ [A] File
54+ [A] Dir
55+ [A] Abs
56+ [A] Streamly.Internal.FileSystem.Path.IsPath
57+ [A] instance Streamly.Internal.FileSystem.Path.IsPath Streamly.Internal.FileSystem.Path.Path
58+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel Streamly.Internal.FileSystem.Path.Path)
59+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path))
60+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path))
61+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path)
62+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path)
63+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs Streamly.Internal.FileSystem.Path.Path)
64+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path))
65+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path))
66+ [A] GHC.Show.Show
67+ [A] instance GHC.Show.Show Streamly.Internal.FileSystem.Path.PathException
68+ [A] GHC.Exception.Type.Exception
69+ [A] instance GHC.Exception.Type.Exception Streamly.Internal.FileSystem.Path.PathException
70+ [A] GHC.Classes.Eq
71+ [A] instance GHC.Classes.Eq Streamly.Internal.FileSystem.Path.PathException
72+ [A] Path
73+ [A] Path :: Array Word8 -> Path
74+ [A] toString :: Path -> [Char]
75+ [A] toPath :: IsPath a => a -> Path
76+ [A] toChunk :: Path -> Array Word8
77+ [A] toChars :: Monad m => Path -> Stream m Char
78+ [A] relfile :: QuasiQuoter
79+ [A] reldir :: QuasiQuoter
80+ [A] rel :: QuasiQuoter
81+ [A] primarySeparator :: Char
82+ [A] path :: QuasiQuoter
83+ [A] mkRelFile :: String -> Q Exp
84+ [A] mkRelDir :: String -> Q Exp
85+ [A] mkRel :: String -> Q Exp
86+ [A] mkPath :: String -> Q Exp
87+ [A] mkFile :: String -> Q Exp
88+ [A] mkDir :: String -> Q Exp
89+ [A] mkAbsFile :: String -> Q Exp
90+ [A] mkAbsDir :: String -> Q Exp
91+ [A] mkAbs :: String -> Q Exp
92+ [A] isSeparator :: Char -> Bool
93+ [A] fromString :: MonadThrow m => [Char] -> m Path
94+ [A] fromPathUnsafe :: IsPath a => Path -> a
95+ [A] fromPath :: (IsPath a, MonadThrow m) => Path -> m a
96+ [A] fromChunkUnsafe :: Array Word8 -> Path
97+ [A] fromChunk :: MonadThrow m => Array Word8 -> m Path
98+ [A] fromChars :: MonadThrow m => Stream Identity Char -> m Path
99+ [A] file :: QuasiQuoter
100+ [A] extendPath :: Path -> Path -> Path
101+ [A] extendDir :: (IsPath (a (Dir Path)), IsPath b, IsPath (a b)) => a (Dir Path) -> Rel b -> a b
102+ [A] dir :: QuasiQuoter
103+ [A] adaptPath :: (MonadThrow m, IsPath a, IsPath b) => a -> m b
104+ [A] absfile :: QuasiQuoter
105+ [A] absdir :: QuasiQuoter
106+ [A] abs :: QuasiQuoter
107+[C] Streamly.Internal.Data.Stream
108+ [A] (FixityR,5)
109+ [A] (FixityR,5)
110+ [C] splitInnerBySuffix
111+ [O] splitInnerBySuffix :: (Monad m, Eq (f a), Monoid (f a)) => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
112+ [N] splitInnerBySuffix :: Monad m => (f a -> Bool) -> (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
113+ [D] sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
114+ [A] indexOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
115+[C] Streamly.Internal.Data.MutByteArray
116+ [A] unsafePinnedAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
117+ [A] unsafeAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
118+ [D] nil :: MutByteArray
119+ [A] empty :: MutByteArray
120+ [D] asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
121+[D] Streamly.Internal.Data.MutArray.Stream
122+ [D] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
123+ [D] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
124+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
125+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
126+[C] Streamly.Internal.Data.MutArray.Generic
127+ [D] writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
128+ [D] writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
129+ [A] unsafeCreateOf :: MonadIO m => Int -> Fold m a (MutArray a)
130+ [A] emptyOf :: MonadIO m => Int -> m (MutArray a)
131+ [A] createWith :: MonadIO m => Int -> Fold m a (MutArray a)
132+ [A] createOf :: MonadIO m => Int -> Fold m a (MutArray a)
133+ [A] create :: MonadIO m => Fold m a (MutArray a)
134+[C] Streamly.Internal.Data.MutArray
135+ [A] SpliceState
136+ [A] SpliceYielding :: arr -> SpliceState s arr -> SpliceState s arr
137+ [A] SpliceInitial :: s -> SpliceState s arr
138+ [A] SpliceFinish :: SpliceState s arr
139+ [A] SpliceBuffering :: s -> arr -> SpliceState s arr
140+ [R] MutByteArray
141+ [D] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
142+ [D] writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
143+ [D] writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
144+ [D] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
145+ [D] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
146+ [D] writeAppendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
147+ [D] writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
148+ [A] unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
149+ [A] unsafePinnedAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
150+ [A] unsafeCreateOfWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
151+ [A] unsafeCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
152+ [A] unsafeAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
153+ [A] unsafeAppendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
154+ [A] toStreamWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
155+ [A] toStreamRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
156+ [R] toStreamDWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
157+ [R] toStreamDRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
158+ [A] slicerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
159+ [A] sliceIndexerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
160+ [A] revCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
161+ [D] resizeExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
162+ [D] resize :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
163+ [A] pokeSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
164+ [A] pokeAppendMay :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> a -> m (Maybe (MutArray Word8))
165+ [A] pokeAppend :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> a -> m (MutArray Word8)
166+ [D] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
167+ [D] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
168+ [D] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
169+ [D] pinnedNewBytes :: MonadIO m => Int -> m (MutArray a)
170+ [A] pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
171+ [A] pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
172+ [A] pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
173+ [A] pinnedCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
174+ [A] peekUnconsUnsafe :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> m (a, MutArray Word8)
175+ [A] peekUncons :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> m (Maybe a, MutArray Word8)
176+ [A] peekSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
177+ [A] pPinnedCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Parser (MutArray a) m (MutArray a)
178+ [A] pCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Parser (MutArray a) m (MutArray a)
179+ [D] nil :: MutArray a
180+ [A] lPinnedCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
181+ [A] lCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
182+ [A] indexReaderWith :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
183+ [A] indexReader :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
184+ [A] growExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
185+ [A] grow :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
186+ [D] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
187+ [R] getIndicesD :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
188+ [D] getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
189+ [D] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
190+ [A] fromStreamN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
191+ [D] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
192+ [D] fromStreamD :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
193+ [A] fromPureStreamN :: (MonadIO m, Unbox a) => Int -> Stream Identity a -> m (MutArray a)
194+ [A] fromPtrN :: MonadIO m => Int -> Ptr Word8 -> m (MutArray Word8)
195+ [A] fromChunksRealloced :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> m (MutArray a)
196+ [A] fromChunksK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
197+ [A] fromByteStr# :: MonadIO m => Addr# -> m (MutArray Word8)
198+ [D] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
199+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
200+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
201+ [A] fPinnedCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) (MutArray a)
202+ [A] fCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) (MutArray a)
203+ [A] emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
204+ [A] empty :: MutArray a
205+ [A] createWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
206+ [A] createOfWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
207+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
208+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
209+ [A] concatWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m (MutArray a) -> Stream m a
210+ [A] concatRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m (MutArray a) -> Stream m a
211+ [A] concatRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
212+ [A] concat :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
213+ [A] compactOnByteSuffix :: MonadIO m => Word8 -> Stream m (MutArray Word8) -> Stream m (MutArray Word8)
214+ [A] compactOnByte :: MonadIO m => Word8 -> Stream m (MutArray Word8) -> Stream m (MutArray Word8)
215+ [A] compactLeAs :: forall m a. (MonadIO m, Unbox a) => PinnedState -> Int -> Stream m (MutArray a) -> Stream m (MutArray a)
216+ [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
217+ [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
218+ [A] compactEQ :: Int -> Stream m (MutArray a) -> Stream m (MutArray a)
219+ [D] cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
220+ [A] byteEq :: MonadIO m => MutArray a -> MutArray a -> m Bool
221+ [A] byteCmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
222+ [A] buildChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
223+ [D] asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
224+ [A] appendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
225+ [A] appendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
226+ [A] append :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
227+[D] Streamly.Internal.Data.Array.Stream
228+ [C] interposeSuffix
229+ [O] interposeSuffix :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
230+ [N] interposeSuffix :: forall m a. (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
231+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
232+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
233+ [C] concatRev
234+ [O] concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
235+ [N] concatRev :: forall m a. (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
236+ [D] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
237+[C] Streamly.Internal.Data.Array.Generic
238+ [A] createOf :: MonadIO m => Int -> Fold m a (Array a)
239+ [A] create :: MonadIO m => Fold m a (Array a)
240+[C] Streamly.Internal.Data.Array
241+ [R] ArrayUnsafe
242+ [D] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
243+ [D] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
244+ [A] unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
245+ [A] unsafePinnedAsPtr :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
246+ [A] unsafeCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
247+ [D] toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
248+ [D] toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
249+ [C] splice
250+ [O] splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
251+ [N] splice :: MonadIO m => Array a -> Array a -> m (Array a)
252+ [A] slicerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Array a)
253+ [A] sliceIndexerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Int, Int)
254+ [D] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
255+ [D] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
256+ [D] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
257+ [D] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
258+ [A] pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
259+ [A] pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
260+ [A] pinnedCompactLE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
261+ [A] parseBreakChunksK :: forall m a b. (MonadIO m, Unbox a) => Parser a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
262+ [D] nil :: Array a
263+ [A] lPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
264+ [A] lCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
265+ [A] interposeSuffix :: forall m a. (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
266+ [A] interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
267+ [A] intercalateSuffix :: (Monad m, Unbox a) => Array a -> Stream m (Array a) -> Stream m a
268+ [A] indexReaderFromThenTo :: Unfold m (Int, Int, Int, Array a) a
269+ [A] indexReader :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
270+ [A] indexFinder :: (a -> Bool) -> Unfold Identity (Array a) Int
271+ [D] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Array a)
272+ [R] getIndicesFromThenTo :: Unfold m (Int, Int, Int, Array a) a
273+ [D] getIndices :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
274+ [D] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Int, Int)
275+ [D] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
276+ [D] fromStreamD :: forall m a. (MonadIO m, Unbox a) => Stream m a -> m (Array a)
277+ [A] fromPureStreamN :: Unbox a => Int -> Stream Identity a -> Array a
278+ [A] fromPtrN :: Int -> Ptr Word8 -> Array Word8
279+ [A] fromChunksK :: (MonadIO m, Unbox a) => StreamK m (Array a) -> m (Array a)
280+ [A] fromChunks :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
281+ [A] fromByteStr :: Ptr Word8 -> Array Word8
282+ [A] foldChunks :: (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m b
283+ [A] foldBreakChunksK :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
284+ [A] foldBreakChunks :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
285+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
286+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
287+ [C] findIndicesOf
288+ [O] findIndicesOf :: (a -> Bool) -> Unfold Identity (Array a) Int
289+ [N] findIndicesOf :: (a -> Bool) -> Array a -> Stream Identity Int
290+ [A] fPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
291+ [A] fCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
292+ [A] empty :: Array a
293+ [A] createWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
294+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
295+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
296+ [A] concatRev :: forall m a. (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
297+ [A] concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
298+ [A] compactOnByteSuffix :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
299+ [A] compactOnByte :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
300+ [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
301+ [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
302+ [A] byteEq :: Array a -> Array a -> Bool
303+ [A] byteCmp :: Array a -> Array a -> Ordering
304+ [A] buildChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
305+ [D] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
306+ [D] asPtrUnsafe :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
core/docs/Changelog.md+60−12
@@ -1,25 +1,73 @@
11# Changelog
22
3−## Unreleased
3+## 0.2.2 (Jan 2024)
44
5−* Arrays are now created unpinned by default, they were created pinned
6− earlier. During IO operations unpinned arrays are automatically copied
7− to pinned memory. When arrays are directly passed to IO operations
8− programmers can choose to create them pinned to avoid a copy. To
9− create pinned arrays, use the internal APIs with the `pinned*` prefix.
5+* Add fixities `infixr 5` for `cons` and `consM` functions.
6+* Fix a bug in Array `Eq` instance when the type is a sum type with
7+ differently sized constructors.
8+* lpackArraysChunksOf, compact, writeChunksWith, putChunksWith now take the
9+ buffer size in number of array elements instead of bytes.
1010
11−* Changed the signature of 'Streamly.Data.Stream.handle' to make the exception
12− handler monadic.
13−* Rethrow the exception promptly in bracketIO.
11+## 0.2.1 (Dec 2023)
12+
13+* Make the serialization of the unit constructor deterministic.
14+* Expose `pinnedSerialize` & `deserialize` via `Streamly.Data.Array`.
15+
16+## 0.2.0 (Nov 2023)
17+
18+See [0.1.0-0.2.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0-0.2.0.txt)
19+for a full list of API changes in this release. Only a few significant
20+changes are mentioned here.
21+
22+### Breaking Changes
23+
24+* `ParserK` in `Streamly.Data.ParserK` is not implicitly specialized
25+ to arrays anymore. To adapt to the new code, change `ParserK a m
26+ b` to `ParserK (Array a) m b` where the `Array` type comes from
27+ `Streamly.Data.Array`. This change also affected the signatures of
28+ `parseChunks` and `parseBreakChunks`.
29+* Changed the signature of 'Streamly.Data.Stream.handle' to make the
30+ exception handler monadic.
31+* Behavior change: Exceptions are now rethrown promptly in `bracketIO`.
32+
33+### Enhancements
34+
35+* __Serialization__: Added a `Streamly.Data.MutByteArray` module with a
36+ `Serialize` type class for fast binary serialization. The Data.Array
37+ module supplies the `serialize` and `deserialize` operations for arrays.
38+* __Unpinned Arrays__: Unboxed arrays are now created unpinned by default,
39+ they were created pinned earlier. During IO operations, unpinned arrays
40+ are automatically copied to pinned memory. When arrays are directly
41+ passed to IO operations programmers can choose to create them pinned to
42+ avoid a copy. To create pinned arrays, use the internal APIs with the
43+ `pinned*` prefix.
44+* StreamK now supports native exception handling routines (handle, bracketIO).
45+ Earlier we had to convert it to the `Stream` type for exception handling.
1446
1547### Deprecations
1648
17−* `Streamly.Data.MutArray.newUnpinned` is renamed to
18− `Streamly.Data.MutArray.pinnedNew`
49+See [0.1.0-0.2.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0-0.2.0.txt)
50+for a full list of deprecations.
51+
52+### Internal API Changes
53+
54+* Fold constructor has changed, added a `final` field to support
55+ finalization and cleanup of a chain of folds. The `extract` field is
56+ now used only for mapping the fold internal state to fold result for
57+ scanning purposes. If your fold does not require cleanup you can just use
58+ your existing `extract` function as `final` as well to adapt to this change.
59+* Many low level internal modules have been removed, they are entirely
60+ exported from higher level internal modules. If you were importing any
61+ of the missing low level modules then import the higher level modules instead.
62+* Internal module changes:
63+ * Streamly.Internal.Serialize.FromBytes -> Streamly.Internal.Data.Binary.Parser
64+ * Streamly.Internal.Serialize.ToBytes -> Streamly.Internal.Data.Binary.Stream
65+ * Streamly.Internal.Data.Unbox is now exported via Streamly.Internal.Data.Serialize
66+ * Streamly.Internal.Data.IORef.Unboxed is now exported via Streamly.Internal.Data.Serialize
1967
2068## 0.1.0 (March 2023)
2169
22−Also see [streamly-core-0.1.0 API Changelog](/core/docs/ApiChangelogs/0.1.0.txt) or
70+Also see [streamly-core-0.1.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0.txt) or
2371https://hackage.haskell.org/package/streamly-core-0.1.0/docs/docs/ApiChangelogs/0.1.0.txt
2472
2573`streamly` package is split into two packages, (1) `streamly-core` that
core/src/DocTestDataArray.hs+8
@@ -1,6 +1,7 @@
11{- $setup
22>>> :m
33>>> :set -XFlexibleContexts
4+>>> :set -XMagicHash
45>>> import Data.Function ((&))
56>>> import Data.Functor.Identity (Identity(..))
67>>> import System.IO.Unsafe (unsafePerformIO)
@@ -10,5 +11,12 @@
1011
1112>>> import qualified Streamly.Data.Array as Array
1213>>> import qualified Streamly.Data.Fold as Fold
14+>>> import qualified Streamly.Data.ParserK as ParserK
1315>>> import qualified Streamly.Data.Stream as Stream
16+>>> import qualified Streamly.Data.StreamK as StreamK
17+
18+For APIs that have not been released yet.
19+
20+>>> import qualified Streamly.Internal.Data.Array as Array
21+>>> import qualified Streamly.Internal.Data.Stream as Stream
1422-}
core/src/DocTestDataFold.hs−1
@@ -24,5 +24,4 @@
2424For APIs that have not been released yet.
2525
2626>>> import qualified Streamly.Internal.Data.Fold as Fold
27−>>> import qualified Streamly.Internal.Data.Fold.Window as FoldW
2827-}
core/src/DocTestDataParserK.hsnew file+15
@@ -0,0 +1,15 @@
1+{- $setup
2+>>> :m
3+>>> import Control.Applicative ((<|>))
4+>>> import Data.Char (isDigit, isAlpha)
5+
6+>>> import Streamly.Data.Parser (Parser)
7+>>> import Streamly.Data.ParserK (ParserK)
8+
9+>>> import qualified Streamly.Data.Parser as Parser
10+>>> import qualified Streamly.Data.ParserK as ParserK
11+
12+For APIs that have not been released yet.
13+
14+>>> import qualified Streamly.Internal.Data.ParserK as ParserK
15+-}
core/src/DocTestDataStream.hs+2−1
@@ -6,6 +6,7 @@
66>>> import Control.Monad.IO.Class (MonadIO (liftIO))
77>>> import Control.Monad.Trans.Class (lift)
88>>> import Control.Monad.Trans.Identity (runIdentityT)
9+>>> import Data.Char (isSpace)
910>>> import Data.Either (fromLeft, fromRight, isLeft, isRight, either)
1011>>> import Data.Maybe (fromJust, isJust)
1112>>> import Data.Function (fix, (&))
@@ -30,9 +31,9 @@
3031For APIs that have not been released yet.
3132
3233>>> import qualified Streamly.Internal.Data.Fold as Fold
33−>>> import qualified Streamly.Internal.Data.Fold.Window as Window
3434>>> import qualified Streamly.Internal.Data.Parser as Parser
3535>>> import qualified Streamly.Internal.Data.Stream as Stream
36+>>> import qualified Streamly.Internal.Data.StreamK as StreamK
3637>>> import qualified Streamly.Internal.Data.Unfold as Unfold
3738>>> import qualified Streamly.Internal.FileSystem.Dir as Dir
3839-}
core/src/DocTestFileSystemHandle.hsnew file+15
@@ -0,0 +1,15 @@
1+{- $setup
2+>>> :m
3+>>> import qualified Streamly.Data.Array as Array
4+>>> import qualified Streamly.FileSystem.Handle as Handle hiding (readChunks)
5+>>> import qualified Streamly.Data.Fold as Fold
6+>>> import qualified Streamly.Data.Stream as Stream
7+>>> import qualified Streamly.Data.Unfold as Unfold
8+
9+For APIs that have not been released yet.
10+
11+>>> import qualified Streamly.Internal.Data.Array as Array (unsafeCreateOf)
12+>>> import qualified Streamly.Internal.Data.Unfold as Unfold (first)
13+>>> import qualified Streamly.Internal.FileSystem.Handle as Handle
14+>>> import qualified Streamly.Internal.System.IO as IO (defaultChunkSize)
15+-}
core/src/DocTestUnicodeParser.hsnew file+9
@@ -0,0 +1,9 @@
1+{- $setup
2+>>> :m
3+>>> import qualified Streamly.Data.Stream as Stream
4+>>> import qualified Streamly.Unicode.Parser as Unicode
5+
6+For APIs that have not been released yet.
7+
8+>>> import qualified Streamly.Internal.Unicode.Parser as Unicode (number, mkDouble)
9+-}
core/src/DocTestUnicodeStream.hsnew file+12
@@ -0,0 +1,12 @@
1+{- $setup
2+>>> :m
3+
4+>>> import qualified Streamly.Data.Fold as Fold
5+>>> import qualified Streamly.Data.Stream as Stream
6+>>> import qualified Streamly.Unicode.Stream as Unicode
7+
8+For APIs that have not been released yet.
9+
10+>>> :set -XMagicHash
11+>>> import qualified Streamly.Internal.Unicode.Stream as Unicode
12+-}
core/src/DocTestUnicodeString.hsnew file+5
@@ -0,0 +1,5 @@
1+{- $setup
2+>>> :m
3+>>> :set -XQuasiQuotes
4+>>> import Streamly.Internal.Unicode.String
5+-}
core/src/Streamly/Console/Stdio.hs+1−27
@@ -24,33 +24,7 @@ module Streamly.Console.Stdio
2424 -- * Write (stderr)
2525 , writeErr
2626 , writeErrChunks
27−
28− -- * Deprecated
29− , read
30− , readChunks
3127 )
3228where
3329
34−import Control.Monad.IO.Class (MonadIO(..))
35−import Data.Word (Word8)
36−import Streamly.Internal.Data.Array.Type (Array)
37−import Streamly.Internal.Data.Unfold (Unfold)
38−
39−import Streamly.Internal.Console.Stdio hiding (read, readChunks)
40−import Prelude hiding (read)
41−
42−-- Same as 'reader'
43−--
44−-- @since 0.8.0
45−{-# DEPRECATED read "Please use 'reader' instead" #-}
46−{-# INLINE read #-}
47−read :: MonadIO m => Unfold m () Word8
48−read = reader
49−
50−-- Same as 'chunkReader'
51−--
52−-- @since 0.8.0
53−{-# DEPRECATED readChunks "Please use 'chunkReader' instead" #-}
54−{-# INLINE readChunks #-}
55−readChunks :: MonadIO m => Unfold m () (Array Word8)
56−readChunks = chunkReader
30+import Streamly.Internal.Console.Stdio
core/src/Streamly/Data/Array.hs+48−48
@@ -29,65 +29,80 @@ module Streamly.Data.Array
2929 -- $overview
3030
3131 -- * The Array Type
32− A.Array
32+ Array
33+
34+ -- * Pinning & Unpinning
35+ -- | Arrays are created unpinned by default unless pinned versions of
36+ -- creation APIs are used. Look for APIs with @pinned@ prefix in
37+ -- "Streamly.Internal.Data.Array" for some unreleased pinned creation APIs.
38+ -- If an array is to be sent to the OS without any further modification
39+ -- then it should be created pinned in the first place instead of pinning
40+ -- it later. Pinning an unpinned array has a copy overhead. OS interfacing
41+ -- APIs create a pinned array directly or convert an unpinned array to
42+ -- pinned array before sending it to the OS.
43+ , pin
44+ , unpin
45+ , isPinned
3346
3447 -- * Construction
3548 -- | When performance matters, the fastest way to generate an array is
36− -- 'writeN'. 'IsList' and 'IsString' instances can be
49+ -- 'createOf'. 'IsList' and 'IsString' instances can be
3750 -- used to conveniently construct arrays from literal values.
3851 -- 'OverloadedLists' extension or 'fromList' can be used to construct an
3952 -- array from a list literal. Similarly, 'OverloadedStrings' extension or
4053 -- 'fromList' can be used to construct an array from a string literal.
4154
42− -- Pure List APIs
43− , A.fromListN
44− , A.fromList
45−
46− -- Monadic APIs
47− , A.writeN -- drop new
48− , A.write -- full buffer
55+ -- ** From Stream
56+ , createOf
57+ , create
4958 , writeLastN -- drop old (ring buffer)
5059
51− -- * Conversion
60+ -- ** From List
61+ , fromListN
62+ , fromList
63+
64+ -- * To List
5265 -- 'GHC.Exts.toList' from "GHC.Exts" can be used to convert an array to a
5366 -- list.
54− , A.toList
67+ , toList
68+
69+ -- * To Stream
70+ , read
71+ , readRev
5572
5673 -- * Unfolds
57− , A.reader
58− , A.readerRev
74+ , reader
75+ , readerRev
5976
6077 -- * Casting
6178 , cast
6279 , asBytes
6380
6481 -- * Random Access
65− , A.length
82+ , length
6683 -- , (!!)
67− , A.getIndex
84+ , getIndex
6885
69− -- * Unbox Type Class
70− , Unbox (..)
86+ -- * Serialization
87+ , pinnedSerialize
88+ , deserialize
7189
72− -- * Pinning & Unpinning
73− , pin
74− , unpin
75− , isPinned
90+ -- * Re-exports
91+ , Unbox (..)
92+ , Serialize(..)
7693
7794 -- * Deprecated
78− , read
79− , readRev
95+ , writeN -- drop new
96+ , write -- full buffer
8097 )
8198where
8299
83100#include "inline.hs"
84101
85−import Streamly.Internal.Data.Unfold (Unfold)
86−import Streamly.Internal.Data.Array as A hiding (read, readRev)
87−import Streamly.Internal.Data.Array.Type (pin, unpin, isPinned)
102+import Streamly.Internal.Data.Array
103+import Streamly.Internal.Data.MutByteArray (Unbox(..), Serialize(..))
88104
89−import Streamly.Internal.Data.Unbox (Unbox (..))
90−import Prelude hiding (read)
105+import Prelude hiding (read, length)
91106
92107#include "DocTestDataArray.hs"
93108
@@ -103,7 +118,7 @@ import Prelude hiding (read)
103118--
104119-- Convert array to stream, and fold the stream:
105120--
106−-- >>> fold f arr = Stream.unfold Array.reader arr & Stream.fold f
121+-- >>> fold f arr = Array.read arr & Stream.fold f
107122-- >>> fold Fold.sum (Array.fromList [1,2,3::Int])
108123-- 6
109124--
@@ -111,7 +126,7 @@ import Prelude hiding (read)
111126--
112127-- Convert array to stream, transform, and fold back to array:
113128--
114−-- >>> amap f arr = Stream.unfold Array.reader arr & fmap f & Stream.fold Array.write
129+-- >>> amap f arr = Array.read arr & fmap f & Stream.fold (Array.createOf (Array.length arr))
115130-- >>> amap (+1) (Array.fromList [1,2,3::Int])
116131-- fromList [2,3,4]
117132--
@@ -138,12 +153,11 @@ import Prelude hiding (read)
138153-- See the 'fromPureStream' unreleased API to generate an array from an
139154-- Identity stream safely without using MonadIO constraint.
140155--
141−--
142156-- Note that 'Identity' streams can be generalized to IO streams:
143157--
144−-- >>> pure = Stream.fromList [1,2,3] :: Stream Identity Int
145−-- >>> generally = Stream.morphInner (return . runIdentity)
146−-- >>> Stream.fold Array.write (generally pure :: Stream IO Int)
158+-- >>> stream = Stream.fromList [1,2,3] :: Stream Identity Int
159+-- >>> pure2IO s = Stream.morphInner (return . runIdentity) s :: Stream IO Int
160+-- >>> Stream.fold Array.create (pure2IO stream)
147161-- fromList [1,2,3]
148162--
149163-- == Programming Tips
@@ -151,17 +165,3 @@ import Prelude hiding (read)
151165-- This module is designed to be imported qualified:
152166--
153167-- >>> import qualified Streamly.Data.Array as Array
154−
155−-- | Same as 'reader'
156−--
157−{-# DEPRECATED read "Please use 'reader' instead" #-}
158−{-# INLINE_NORMAL read #-}
159−read :: (Monad m, Unbox a) => Unfold m (Array a) a
160−read = reader
161−
162−-- | Same as 'readerRev'
163−--
164−{-# DEPRECATED readRev "Please use 'readerRev' instead" #-}
165−{-# INLINE_NORMAL readRev #-}
166−readRev :: (Monad m, Unbox a) => Unfold m (Array a) a
167−readRev = readerRev
core/src/Streamly/Data/Array/Generic.hs+16−13
@@ -15,34 +15,37 @@ module Streamly.Data.Array.Generic
1515 ( Array
1616
1717 -- * Construction
18− , A.fromListN
19− , A.fromList
18+ , fromListN
19+ , fromList
2020
2121 -- MonadicAPIs
22− , A.writeN
23− , A.write
22+ , createOf
23+ , create
2424
2525 -- * Conversion
26− , A.toList
26+ , toList
2727
2828 -- * Streams
29− , A.read
30− , A.readRev
29+ , read
30+ , readRev
3131
3232 -- * Unfolds
33− , A.reader
33+ , reader
3434 -- , A.readerRev
3535
3636 -- * Random Access
37− , A.length
38− , A.getIndex
37+ , length
38+ , getIndex
3939
4040 -- -- * Folding Arrays
4141 -- , A.streamFold
4242 -- , A.fold
43+
44+ -- * Deprecated
45+ , writeN
46+ , write
4347 )
4448where
4549
46−import Streamly.Internal.Data.Array.Generic (Array)
47−
48−import qualified Streamly.Internal.Data.Array.Generic as A
50+import Streamly.Internal.Data.Array.Generic
51+import Prelude hiding (length, read)
core/src/Streamly/Data/Fold.hs+150−119
@@ -10,6 +10,156 @@
1010-- Fast, composable stream consumers with ability to terminate, supporting
1111-- stream fusion.
1212--
13+-- == Using Folds
14+--
15+-- This module provides elementary folds and fold combinators that can be used
16+-- to consume a stream of data and reduce it to a final value, or transform it
17+-- in a stateful manner using scans. A data stream can be reduced into a stream
18+-- of folded data elements by folding segments of the stream. Fold combinators
19+-- can be used to compose multiple folds in parallel or to create a pipeline of
20+-- folds such that the next fold consumes the result of the previous fold. To
21+-- run these folds on a stream see 'Streamly.Data.Stream.fold',
22+-- 'Streamly.Data.Stream.scan', 'Streamly.Data.Stream.postscan',
23+-- 'Streamly.Data.Stream.scanMaybe', 'Streamly.Data.Stream.foldMany' and other
24+-- operations accepting 'Fold' type as argument "Streamly.Data.Stream".
25+--
26+-- == Reducing a Stream
27+--
28+-- A 'Fold' is a consumer of a stream of values. A fold driver (such as
29+-- 'Streamly.Data.Stream.fold') initializes the fold @accumulator@, runs the
30+-- fold @step@ function in a loop, processing the input stream one element at a
31+-- time and accumulating the result. The loop continues until the fold
32+-- terminates, at which point the accumulated result is returned.
33+--
34+-- For example, a 'sum' Fold represents a stream consumer that adds the values
35+-- in the input stream:
36+--
37+-- >>> Stream.fold Fold.sum $ Stream.fromList [1..100]
38+-- 5050
39+--
40+-- Conceptually, a 'Fold' is a data type that mimics a strict left fold
41+-- ('Data.List.foldl'). The above example is similar to a left fold using
42+-- @(+)@ as the step and @0@ as the initial value of the accumulator:
43+--
44+-- >>> Data.List.foldl' (+) 0 [1..100]
45+-- 5050
46+--
47+-- 'Fold's have an early termination capability e.g. the 'one' fold terminates
48+-- after consuming one element:
49+--
50+-- >>> Stream.fold Fold.one $ Stream.fromList [1..]
51+-- Just 1
52+--
53+-- The above example is similar to the following right fold:
54+--
55+-- >>> Prelude.foldr (\x _ -> Just x) Nothing [1..]
56+-- Just 1
57+--
58+-- 'Fold's can be combined together using combinators. For example, to create a
59+-- fold that sums first two elements in a stream:
60+--
61+-- >>> sumTwo = Fold.take 2 Fold.sum
62+-- >>> Stream.fold sumTwo $ Stream.fromList [1..100]
63+-- 3
64+--
65+-- == Parallel Composition
66+--
67+-- Folds can be combined to run in parallel on the same input. For example, to
68+-- compute the average of numbers in a stream without going through the stream
69+-- twice:
70+--
71+-- >>> avg = Fold.teeWith (/) Fold.sum (fmap fromIntegral Fold.length)
72+-- >>> Stream.fold avg $ Stream.fromList [1.0..100.0]
73+-- 50.5
74+--
75+-- Folds can be combined so as to partition the input stream over multiple
76+-- folds. For example, to count even and odd numbers in a stream:
77+--
78+-- >>> split n = if even n then Left n else Right n
79+-- >>> stream = fmap split $ Stream.fromList [1..100]
80+-- >>> countEven = fmap (("Even " ++) . show) Fold.length
81+-- >>> countOdd = fmap (("Odd " ++) . show) Fold.length
82+-- >>> f = Fold.partition countEven countOdd
83+-- >>> Stream.fold f stream
84+-- ("Even 50","Odd 50")
85+--
86+-- == Sequential Composition
87+--
88+-- Terminating folds can be combined to parse the stream serially such that the
89+-- first fold consumes the input until it terminates and the second fold
90+-- consumes the rest of the input until it terminates:
91+--
92+-- >>> f = Fold.splitWith (,) (Fold.take 8 Fold.toList) (Fold.takeEndBy (== '\n') Fold.toList)
93+-- >>> Stream.fold f $ Stream.fromList "header: hello\n"
94+-- ("header: ","hello\n")
95+--
96+-- == Splitting a Stream
97+--
98+-- A 'Fold' can be applied repeatedly on a stream to transform it to a stream
99+-- of fold results. To split a stream on newlines:
100+--
101+-- >>> f = Fold.takeEndBy (== '\n') Fold.toList
102+-- >>> Stream.fold Fold.toList $ Stream.foldMany f $ Stream.fromList "Hello there!\nHow are you\n"
103+-- ["Hello there!\n","How are you\n"]
104+--
105+-- Similarly, we can split the input of a fold too:
106+--
107+-- >>> Stream.fold (Fold.many f Fold.toList) $ Stream.fromList "Hello there!\nHow are you\n"
108+-- ["Hello there!\n","How are you\n"]
109+--
110+-- == Folds vs. Streams
111+--
112+-- We can often use streams or folds to achieve the same goal. However, streams
113+-- are more efficient in composition of producers (e.g.
114+-- 'Data.Stream.append' or 'Data.Stream.mergeBy') whereas folds are
115+-- more efficient in composition of consumers (e.g. 'splitWith', 'partition'
116+-- or 'teeWith').
117+--
118+-- Streams are producers, transformations on streams happen on the output side:
119+--
120+-- >>> :{
121+-- f stream =
122+-- Stream.filter odd stream
123+-- & fmap (+1)
124+-- & Stream.fold Fold.sum
125+-- :}
126+--
127+-- >>> f $ Stream.fromList [1..100 :: Int]
128+-- 2550
129+--
130+-- Folds are stream consumers with an input stream and an output value, stream
131+-- transformations on folds happen on the input side:
132+--
133+-- >>> :{
134+-- f =
135+-- Fold.filter odd
136+-- $ Fold.lmap (+1)
137+-- $ Fold.sum
138+-- :}
139+--
140+-- >>> Stream.fold f $ Stream.fromList [1..100 :: Int]
141+-- 2550
142+--
143+-- Notice the similiarity in the definition of @f@ in both cases, the only
144+-- difference is the composition by @&@ vs @$@ and the use @lmap@ vs @map@, the
145+-- difference is due to output vs input side transformations.
146+--
147+-- == Fusion Limitations
148+--
149+-- Folds support stream fusion for generating loops comparable to the speed of
150+-- C. However, it has some limitations. For fusion to work, the folds must be
151+-- inlined, folds must be statically known and not generated dynamically, folds
152+-- should not be passed recursively.
153+--
154+-- Another limitation is due to the quadratic complexity causing slowdown when
155+-- too many nested compositions are used. Especially, the performance of the
156+-- Applicative instance and splitting operations (e.g. 'splitWith') degrades
157+-- quadratically (O(n^2)) when combined @n@ times, roughly 8 or less sequenced
158+-- operations are fine. For these cases folds can be converted to parsers and
159+-- then used as ParserK.
160+--
161+-- == Experimental APIs
162+--
13163-- Please refer to "Streamly.Internal.Data.Fold" for more functions that have
14164-- not yet been released.
15165
@@ -21,9 +171,6 @@ module Streamly.Data.Fold
21171 --
22172 -- $setup
23173
24− -- * Overview
25− -- $overview
26−
27174 -- * Running A Fold
28175 drive
29176 -- XXX Should we have a stream returning function in fold module?
@@ -255,125 +402,9 @@ import Prelude
255402 span, splitAt, break, mapM, maybe)
256403
257404import Streamly.Internal.Data.Fold
258−import Streamly.Internal.Data.Fold.Container
259405
260406#include "DocTestDataFold.hs"
261407
262−-- $overview
263−--
264−-- A 'Fold' is a consumer of a stream of values. A fold driver (such as
265−-- 'Streamly.Data.Stream.fold') initializes the fold @accumulator@, runs the
266−-- fold @step@ function in a loop, processing the input stream one element at a
267−-- time and accumulating the result. The loop continues until the fold
268−-- terminates, at which point the accumulated result is returned.
269−--
270−-- For example, a 'sum' Fold represents a stream consumer that adds the values
271−-- in the input stream:
272−--
273−-- >>> Stream.fold Fold.sum $ Stream.fromList [1..100]
274−-- 5050
275−--
276−-- Conceptually, a 'Fold' is a data type that mimics a strict left fold
277−-- ('Data.List.foldl'). The above example is similar to a left fold using
278−-- @(+)@ as the step and @0@ as the initial value of the accumulator:
279−--
280−-- >>> Data.List.foldl' (+) 0 [1..100]
281−-- 5050
282−--
283−-- 'Fold's have an early termination capability e.g. the 'one' fold terminates
284−-- after consuming one element:
285−--
286−-- >>> Stream.fold Fold.one $ Stream.fromList [1..]
287−-- Just 1
288−--
289−-- The above example is similar to the following right fold:
290−--
291−-- >>> Prelude.foldr (\x _ -> Just x) Nothing [1..]
292−-- Just 1
293−--
294−-- 'Fold's can be combined together using combinators. For example, to create a
295−-- fold that sums first two elements in a stream:
296−--
297−-- >>> sumTwo = Fold.take 2 Fold.sum
298−-- >>> Stream.fold sumTwo $ Stream.fromList [1..100]
299−-- 3
300−--
301−-- Folds can be combined to run in parallel on the same input. For example, to
302−-- compute the average of numbers in a stream without going through the stream
303−-- twice:
304−--
305−-- >>> avg = Fold.teeWith (/) Fold.sum (fmap fromIntegral Fold.length)
306−-- >>> Stream.fold avg $ Stream.fromList [1.0..100.0]
307−-- 50.5
308−--
309−-- Folds can be combined so as to partition the input stream over multiple
310−-- folds. For example, to count even and odd numbers in a stream:
311−--
312−-- >>> split n = if even n then Left n else Right n
313−-- >>> stream = fmap split $ Stream.fromList [1..100]
314−-- >>> countEven = fmap (("Even " ++) . show) Fold.length
315−-- >>> countOdd = fmap (("Odd " ++) . show) Fold.length
316−-- >>> f = Fold.partition countEven countOdd
317−-- >>> Stream.fold f stream
318−-- ("Even 50","Odd 50")
319−--
320−-- Terminating folds can be combined to parse the stream serially such that the
321−-- first fold consumes the input until it terminates and the second fold
322−-- consumes the rest of the input until it terminates:
323−--
324−-- >>> f = Fold.splitWith (,) (Fold.take 8 Fold.toList) (Fold.takeEndBy (== '\n') Fold.toList)
325−-- >>> Stream.fold f $ Stream.fromList "header: hello\n"
326−-- ("header: ","hello\n")
327−--
328−-- A 'Fold' can be applied repeatedly on a stream to transform it to a stream
329−-- of fold results. To split a stream on newlines:
330−--
331−-- >>> f = Fold.takeEndBy (== '\n') Fold.toList
332−-- >>> Stream.fold Fold.toList $ Stream.foldMany f $ Stream.fromList "Hello there!\nHow are you\n"
333−-- ["Hello there!\n","How are you\n"]
334−--
335−-- Similarly, we can split the input of a fold too:
336−--
337−-- >>> Stream.fold (Fold.many f Fold.toList) $ Stream.fromList "Hello there!\nHow are you\n"
338−-- ["Hello there!\n","How are you\n"]
339−--
340−-- = Folds vs. Streams
341−--
342−-- We can often use streams or folds to achieve the same goal. However, streams
343−-- are more efficient in composition of producers (e.g.
344−-- 'Data.Stream.append' or 'Data.Stream.mergeBy') whereas folds are
345−-- more efficient in composition of consumers (e.g. 'splitWith', 'partition'
346−-- or 'teeWith').
347−--
348−-- Streams are producers, transformations on streams happen on the output side:
349−--
350−-- >>> :{
351−-- f stream =
352−-- Stream.filter odd stream
353−-- & fmap (+1)
354−-- & Stream.fold Fold.sum
355−-- :}
356−--
357−-- >>> f $ Stream.fromList [1..100 :: Int]
358−-- 2550
359−--
360−-- Folds are stream consumers with an input stream and an output value, stream
361−-- transformations on folds happen on the input side:
362−--
363−-- >>> :{
364−-- f =
365−-- Fold.filter odd
366−-- $ Fold.lmap (+1)
367−-- $ Fold.sum
368−-- :}
369−--
370−-- >>> Stream.fold f $ Stream.fromList [1..100 :: Int]
371−-- 2550
372−--
373−-- Notice the similiarity in the definition of @f@ in both cases, the only
374−-- difference is the composition by @&@ vs @$@ and the use @lmap@ vs @map@, the
375−-- difference is due to output vs input side transformations.
376−
377408--------------------------------------------------------------------------------
378409-- Deprecated
379410--------------------------------------------------------------------------------
core/src/Streamly/Data/MutArray.hs+18−12
@@ -32,22 +32,27 @@ module Streamly.Data.MutArray
3232 -- * Construction
3333
3434 -- Uninitialized Arrays
35− , new
36− , pinnedNew
35+ , emptyOf
36+ , pinnedEmptyOf
3737
3838 -- From containers
3939 , fromListN
4040 , fromList
41− , writeN -- drop new
42− , write -- full buffer
41+ , createOf
42+ , create
4343 -- writeLastN
4444
45+ -- * Pinning & Unpinning
46+ , pin
47+ , unpin
48+ , isPinned
49+
4550 -- * Appending elements
4651 , snoc
4752
4853 -- * Appending streams
49− , writeAppendN -- XXX Rename to writeSnocN?
50− , writeAppend -- XXX Rename to writeSnoc?
54+ , appendN
55+ , append
5156
5257 -- * Inplace mutation
5358 , putIndex
@@ -78,16 +83,17 @@ module Streamly.Data.MutArray
7883 -- * Size
7984 , length
8085
81− -- * Unbox Type Class
86+ -- * Re-exports
8287 , Unbox (..)
8388
84− -- * Pinning & Unpinning
85− , pin
86− , unpin
87− , isPinned
88−
8989 -- * Deprecated
9090 , newPinned
91+ , new
92+ , pinnedNew
93+ , writeN
94+ , write
95+ , writeAppendN
96+ , writeAppend
9197 )
9298where
9399
core/src/Streamly/Data/MutArray/Generic.hs+8−3
@@ -23,11 +23,11 @@ module Streamly.Data.MutArray.Generic
2323 MutArray
2424
2525 -- * Construction
26− , new
26+ , emptyOf
2727 , fromListN
2828 , fromList
29− , writeN
30− , write
29+ , createOf
30+ , create
3131
3232 -- * Appending elements
3333 , snoc
@@ -56,6 +56,11 @@ module Streamly.Data.MutArray.Generic
5656
5757 -- * Size
5858 , length
59+
60+ -- * Deprecated
61+ , new
62+ , writeN
63+ , write
5964 )
6065where
6166
core/src/Streamly/Data/MutByteArray.hsnew file+106
@@ -0,0 +1,106 @@
1+-- |
2+-- Module : Streamly.Data.MutByteArray
3+-- Copyright : (c) 2023 Composewell Technologies
4+-- License : BSD3-3-Clause
5+-- Maintainer : streamly@composewell.com
6+-- Portability : GHC
7+--
8+-- A low level byte Array type 'MutByteArray', along with type classes 'Unbox'
9+-- and 'Serialize' for fast binary serialization and deserialization of Haskell
10+-- values. Serialization, deserialization performance is similar to, and in
11+-- some cases many times better than the store package. Conceptually, the
12+-- 'Serialize' type class works in the same way as store.
13+--
14+-- Serialize instances are configurable to use constructor names (see
15+-- 'Streamly.Internal.Data.MutByteArray.encodeConstrNames'), record field names (see
16+-- 'Streamly.Internal.Data.MutByteArray.encodeRecordFields') instead of binary
17+-- encoded values. This is an experimental feature which allows JSON like
18+-- properties with faster speed. For example, you can change the order of
19+-- constructors or record fields without affecting serialized value.
20+--
21+-- Higher level unboxed array modules "Streamly.Data.Array" and
22+-- "Streamly.Data.MutArray" are built on top of this module. Unboxed arrays are
23+-- essentially serialized Haskell values. Array modules provide higher level
24+-- serialization routines like 'Streamly.Internal.Data.Array.pinnedSerialize'
25+-- and 'Streamly.Internal.Data.Array.deserialize' from the
26+-- "Streamly.Internal.Data.Array" module.
27+--
28+-- == Mutable Byte Array
29+--
30+-- 'MutByteArray' is a primitive mutable array in the IO monad. 'Unbox' and
31+-- 'Serialize' type classes use this primitive array to serialize data to and
32+-- deserialize it from. This array is used to build higher level unboxed
33+-- array types 'Streamly.Data.MutArray.MutArray' and 'Streamly.Data.Array.Array'.
34+--
35+-- == Using Unbox
36+--
37+-- The 'Unbox' type class is simple and used to serialize non-recursive fixed
38+-- size data types. This type class is primarily used to implement unboxed
39+-- arrays. Unboxed arrays are just a sequence of serialized fixed length
40+-- Haskell data types. Instances of this type class can be derived using
41+-- 'Generic' or template haskell based deriving functions provided in this
42+-- module.
43+--
44+-- Writing a data type to an array using the array creation routines in
45+-- "Streamly.Data.Array" or "Streamly.Data.MutArray" (e.g. @writeN@ or
46+-- @fromListN@), serializes the type to the array. Similarly, reading the data
47+-- type from the array deserializes it. You can also serialize and deserialize
48+-- directly to and from a 'MutByteArray', using the type class methods.
49+--
50+-- == Using Serialize
51+--
52+-- The 'Serialize' type class is a superset of the 'Unbox' type class, it can
53+-- serialize variable length data types as well e.g. Haskell lists. Use
54+-- 'deriveSerialize' to derive the instances of the type class automatically
55+-- and then use the type class methods to serialize and deserialize to and from
56+-- a 'MutByteArray'.
57+--
58+-- See 'Streamly.Internal.Data.Array.pinnedSerialize' and
59+-- 'Streamly.Internal.Data.Array.deserialize' for 'Array' type based
60+-- serialization.
61+--
62+-- == Comparing serialized values
63+--
64+-- When using the `Unbox` type class the same value may result in differing
65+-- serialized bytes because of unused uninitialized data in case of sum types.
66+-- Therefore, byte comparison of serialized values is not reliable.
67+--
68+-- However, the 'Serialize' type class guarantees that the serialized values
69+-- are always exactly the same and byte comparison of serialized is reliable.
70+--
71+module Streamly.Data.MutByteArray
72+ (
73+
74+ -- * Mutable Byte Array
75+ -- | The standard way to read from or write to a 'MutByteArray' is by using
76+ -- the 'Unbox' or 'Serialize' type class methods.
77+ MutByteArray
78+ , isPinned
79+ , pin
80+ , unpin
81+ , new
82+ , pinnedNew
83+
84+ -- * Unbox
85+ , Unbox(..)
86+ , deriveUnbox
87+
88+ -- * Serialize
89+ , Serialize(..)
90+
91+ -- ** Instance Config
92+ , SerializeConfig
93+ , inlineAddSizeTo
94+ , inlineSerializeAt
95+ , inlineDeserializeAt
96+
97+ -- ** Instance Deriving
98+ , deriveSerialize
99+ , deriveSerializeWith
100+ ) where
101+
102+--------------------------------------------------------------------------------
103+-- Imports
104+--------------------------------------------------------------------------------
105+
106+import Streamly.Internal.Data.MutByteArray
core/src/Streamly/Data/Parser.hs+130−60
@@ -7,12 +7,137 @@
77-- Stability : pre-release
88-- Portability : GHC
99--
10−-- Fast, composable stream consumers with ability to terminate, backtrack and
11−-- fail, supporting stream fusion. Parsers are a natural extension of
12−-- "Streamly.Data.Fold". Parsers and folds can be interconverted.
10+-- Parsers are stream consumers like folds with the following differences:
1311--
14−-- Please refer to "Streamly.Internal.Data.Parser" for functions that have
15−-- not yet been released.
12+-- * folds cannot fail but parsers can fail and backtrack.
13+-- * folds can be composed as a Tee but parsers cannot.
14+-- * folds can be used for scanning but parsers cannot.
15+-- * folds can be converted to parsers.
16+--
17+-- This module implements parsers with stream fusion which compile to efficient
18+-- loops comparable to the speed of C.
19+--
20+-- == Using Parsers
21+--
22+-- This module provides elementary parsers and parser combinators that can be
23+-- used to parse a stream of data. Additionally, all the folds from the
24+-- "Streamly.Data.Fold" module can be converted to parsers using 'fromFold'.
25+-- All the parsing functionality provided by popular parsing libraries, and
26+-- more is available. Also see "Streamly.Unicode.Parser" module for Char stream
27+-- parsers.
28+--
29+-- A data stream can be transformed to a stream of parsed data elements. Parser
30+-- combinators can be used to create a pipeline of folds or parsers such that
31+-- the next fold or parser consumes the result of the previous parser. See
32+-- 'Streamly.Data.Stream.parse' and 'Streamly.Data.Stream.parseMany' to run
33+-- these parsers on a stream.
34+--
35+-- == Parser vs ParserK
36+--
37+-- There are two functionally equivalent parsing modules,
38+-- "Streamly.Data.Parser" (this module) and "Streamly.Data.ParserK". The latter
39+-- is a CPS based wrapper over the former, and can be used for parsing in
40+-- general. "Streamly.Data.Parser" enables stream fusion and should be
41+-- preferred over "Streamly.Data.ParserK" for high performance stream parsing
42+-- use cases. However, there are a few cases where this module is not
43+-- suitable and ParserK should be used instead.
44+--
45+-- For static fusion, parser combinators have to use strict pattern matching on
46+-- arguments of type Parser. This leads to infinte loop when a parser is
47+-- defined recursively, due to strict evaluation of the recursive call. For
48+-- example, the following implementation loops infinitely because of the
49+-- recursive use of parser @p@ in the @*>@ combinator:
50+--
51+-- >>> import Streamly.Data.Parser (Parser)
52+-- >>> import qualified Streamly.Data.Fold as Fold
53+-- >>> import qualified Streamly.Data.Parser as Parser
54+-- >>> import qualified Streamly.Data.Stream as Stream
55+-- >>> import Control.Applicative ((<|>))
56+--
57+-- >>> :{
58+-- >>> p :: Monad m => Parser Char m String
59+-- >>> p = Parser.satisfy (== '(') *> p <|> Parser.fromFold Fold.toList
60+-- >>> :}
61+--
62+-- Use ParserK when recursive use is required:
63+--
64+-- >>> import Streamly.Data.ParserK (ParserK)
65+-- >>> import qualified Streamly.Data.StreamK as StreamK
66+-- >>> import qualified Streamly.Internal.Data.StreamK as StreamK (parse)
67+-- >>> import qualified Streamly.Internal.Data.ParserK as ParserK (adapt)
68+--
69+-- >>> :{
70+-- >>> p :: Monad m => ParserK Char m String
71+-- >>> p = ParserK.adapt (Parser.satisfy (== '(')) *> p <|> ParserK.adapt (Parser.fromFold Fold.toList)
72+-- >>> :}
73+--
74+-- >>> StreamK.parse p $ StreamK.fromStream $ Stream.fromList "hello"
75+-- Right "hello"
76+--
77+-- For this reason Applicative, Alternative or Monad compositions with
78+-- recursion cannot be used with the 'Parser' type. Alternative type class based
79+-- operations like 'asum' and Alternative based generic parser combinators use
80+-- recursion. Similarly, Applicative type class based operations like
81+-- 'Prelude.sequence' use recursion. Custom implementations of many such
82+-- operations are provided in this module (e.g. 'some', 'many'), and those
83+-- should be used instead.
84+--
85+-- Another limitation of Parser type is due to the quadratic complexity causing
86+-- slowdown when too many nested compositions are used. Especially Applicative,
87+-- Monad, Alternative instances, and sequenced parsing operations (e.g. nested
88+-- 'one', and 'splitWith') degrade the performance quadratically (O(n^2)) when
89+-- combined @n@ times, roughly 8 or less sequenced parsers are fine. READ THE
90+-- DOCS OF APPLICATIVE, MONAD AND ALTERNATIVE INSTANCES.
91+--
92+-- == Streaming Parsers
93+--
94+-- With 'Streamly.Data.ParserK.ParserK' you can use the generic Alternative
95+-- type class based parsers from the
96+-- <https://hackage.haskell.org/package/parser-combinators parser-combinators>
97+-- library or similar. However, we recommend that you use the equivalent
98+-- functionality from this module for better performance and for streaming
99+-- behavior.
100+--
101+-- Firstly, the combinators in this module are faster due to stream fusion.
102+-- Secondly, these are streaming in nature as the results can be passed
103+-- directly to other stream consumers (folds or parsers). The Alternative type
104+-- class based parsers would end up buffering all the results in lists before
105+-- they can be consumed.
106+--
107+-- When recursion or heavy nesting is needed use ParserK.
108+--
109+-- == Error Reporting
110+--
111+-- These parsers do not report the error context (e.g. line number or column).
112+-- This may be supported in future.
113+--
114+-- == Monad Transformer Stack
115+--
116+-- 'MonadTrans' instance is not provided. If the 'Parser' type is the top most
117+-- layer (which should be the case almost always) you can just use 'fromEffect'
118+-- to execute the lower layer monad effects.
119+--
120+-- == Parser vs ParserK Implementation
121+--
122+-- The 'Parser' type represents a stream consumer by composing state as data
123+-- which enables stream fusion. Stream fusion generates a tight loop without
124+-- any constructor allocations between the stages, providing C like performance
125+-- for the loop. Stream fusion works when multiple functions are combined in a
126+-- pipeline statically. Therefore, the operations in this module must be
127+-- inlined and must not be used recursively to allow for stream fusion.
128+--
129+-- The 'ParserK' type represents a stream consumer by composing function calls,
130+-- therefore, a function call overhead is incurred at each composition. It is
131+-- quite fast in general but may be a few times slower than a fused parser.
132+-- However, it allows for scalable dynamic composition especially parsers can
133+-- be used in recursive calls. Using the 'ParserK' type operations like
134+-- 'splitWith' provide linear (O(n)) performance with respect to the number of
135+-- compositions.
136+--
137+-- == Experimental APIs
138+--
139+-- Please refer to "Streamly.Internal.Data.Parser" for functions that have not
140+-- yet been released.
16141--
17142module Streamly.Data.Parser
18143 (
@@ -22,9 +147,6 @@ module Streamly.Data.Parser
22147 --
23148 -- $setup
24149
25− -- * Overview
26− -- $overview
27−
28150 -- * Parser Type
29151 Parser
30152
@@ -123,55 +245,3 @@ import Streamly.Internal.Data.Parser
123245import Prelude hiding (dropWhile, takeWhile, filter)
124246
125247#include "DocTestDataParser.hs"
126−
127−-- $overview
128−--
129−-- Several combinators in this module can be many times faster than CPS based
130−-- parsers because of stream fusion. For example,
131−-- 'Streamly.Internal.Data.Parser.many' combinator in this module is much
132−-- faster than the 'Control.Applicative.many' combinator of
133−-- 'Control.Applicative.Alternative' type class used by CPS based parsers.
134−--
135−-- The use of 'Alternative' type class, in parsers has another drawback.
136−-- Alternative based parsers use plain Haskell lists to collect the results. In
137−-- a strict Monad like IO, the results are necessarily buffered before they can
138−-- be consumed. This may not perform optimally in streaming applications
139−-- processing large amounts of data. Equivalent combinators in this module can
140−-- consume the results of parsing using a 'Fold' or another parser, thus
141−-- providing a scalable and composable consumer.
142−--
143−-- Note that these parsers do not report the error context (e.g. line number or
144−-- column). This may be supported in future.
145−--
146−-- mtl instances are not provided. If the 'Parser' type is the top most layer
147−-- (which should be the case almost always) you can just use 'fromEffect' to
148−-- execute the lower layer monad effects.
149−--
150−-- == Performance Notes
151−--
152−-- The 'Parser' type represents a stream consumer by composing state as data
153−-- which enables stream fusion. Stream fusion generates a tight loop without
154−-- any constructor allocations between the stages, providing C like performance
155−-- for the loop. Stream fusion works when multiple functions are combined in a
156−-- pipeline statically. Therefore, the operations in this module must be
157−-- inlined and must not be used recursively to allow for stream fusion. Note
158−-- that operations like 'sequence', and 'asum' that compose pasrers using
159−-- recursion should be avoided with these parsers. You can use these with the
160−-- 'ParserK' module instead.
161−--
162−-- Using the 'Parser' type, parsing operations like 'one', 'splitWith' etc.
163−-- degrade quadratically (O(n^2)) when combined many times. If you need to
164−-- combine these operations, say more than 8 times in a single loop, then you
165−-- should consider using the continuation style parser type 'ParserK' instead.
166−-- Also, if you need to use these operations in a recursive loop you should use
167−-- 'ParserK' instead.
168−--
169−-- The 'ParserK' type represents a stream consumer by composing function calls,
170−-- therefore, a function call overhead is incurred at each composition. It is
171−-- quite fast in general but may be a few times slower than a fused parser.
172−-- However, it allows for scalable dynamic composition especially parsers can
173−-- be used in recursive calls. Using the 'ParserK' type operations like
174−-- 'splitWith' provide linear (O(n)) performance with respect to the number of
175−-- compositions..
176−--
177−-- 'Parser' and 'ParserK' types can be interconverted.
core/src/Streamly/Data/ParserK.hs+105−10
@@ -1,3 +1,4 @@
1+{-# LANGUAGE CPP #-}
12-- |
23-- Module : Streamly.Data.ParserK
34-- Copyright : (c) 2023 Composewell Technologies
@@ -6,26 +7,112 @@
67-- Stability : pre-release
78-- Portability : GHC
89--
9−-- Parsers using Continuation Passing Style (CPS). See notes in
10−-- "Streamly.Data.Parser" module to know when to use this module.
10+-- See the general notes about parsing in the "Streamly.Data.Parser" module.
11+-- This module implements a using Continuation Passing Style (CPS) wrapper over
12+-- the "Streamly.Data.Parser" module. It is as fast or faster than attoparsec.
1113--
12−-- To run a 'ParserK' use 'Streamly.Data.StreamK.parseChunks'.
14+-- == Parser vs ParserK
15+--
16+-- 'ParserK' is preferred over 'Streamly.Data.Parser.Parser' when extensive
17+-- applicative, alternative and monadic composition is required, or when
18+-- recursive or dynamic composition of parsers is required. The
19+-- 'Streamly.Data.Parser.Parser' type fuses statically and creates efficient
20+-- loops whereas 'ParserK' uses function call based composition and has
21+-- comparatively larger runtime overhead but it is better suited to the
22+-- specific use cases mentioned above. 'ParserK' also allows to efficient parse
23+-- a stream of arrays, it can also break the input stream into a parse result
24+-- and remaining stream so that the stream can be parsed independently in
25+-- segments.
26+--
27+-- == Using ParserK
28+--
29+-- All the parsers from the "Streamly.Data.Parser" module can be adapted to
30+-- ParserK using the 'Streamly.Data.ParserK.adaptC',
31+-- 'Streamly.Internal.Data.ParserK.adapt', and
32+-- 'Streamly.Internal.Data.ParserK.adaptCG' combinators.
33+--
34+-- 'Streamly.Data.StreamK.parseChunks' runs a parser on a stream of unboxed
35+-- arrays, this is the preferred and most efficient way to parse chunked input.
36+-- The more general 'Streamly.Data.StreamK.parseBreakChunks' function returns
37+-- the remaining stream as well along with the parse result. There are
38+-- 'Streamly.Internal.Data.StreamK.parseChunksGeneric',
39+-- 'Streamly.Internal.Data.StreamK.parseBreakChunksGeneric' as well to run
40+-- parsers on boxed arrays. 'Streamly.Internal.Data.StreamK.parse',
41+-- 'Streamly.Internal.Data.StreamK.parseBreak' run parsers on a stream of
42+-- individual elements instead of stream of arrays.
43+--
44+-- == Monadic Composition
45+--
46+-- Monad composition can be used for lookbehind parsers, we can dynamically
47+-- compose new parsers based on the results of the previously parsed values.
48+--
49+-- If we have to parse "a9" or "9a" but not "99" or "aa" we can use the
50+-- following non-monadic, backtracking parser:
51+--
52+-- >>> digits p1 p2 = ((:) <$> p1 <*> ((:) <$> p2 <*> pure []))
53+-- >>> :{
54+-- backtracking :: Monad m => ParserK Char m String
55+-- backtracking = ParserK.adapt $
56+-- digits (Parser.satisfy isDigit) (Parser.satisfy isAlpha)
57+-- <|>
58+-- digits (Parser.satisfy isAlpha) (Parser.satisfy isDigit)
59+-- :}
60+--
61+-- We know that if the first parse resulted in a digit at the first place then
62+-- the second parse is going to fail. However, we waste that information and
63+-- parse the first character again in the second parse only to know that it is
64+-- not an alphabetic char. By using lookbehind in a 'Monad' composition we can
65+-- avoid redundant work:
66+--
67+-- >>> data DigitOrAlpha = Digit Char | Alpha Char
68+--
69+-- >>> :{
70+-- lookbehind :: Monad m => ParserK Char m String
71+-- lookbehind = do
72+-- x1 <- ParserK.adapt $
73+-- Digit <$> Parser.satisfy isDigit
74+-- <|> Alpha <$> Parser.satisfy isAlpha
75+-- -- Note: the parse depends on what we parsed already
76+-- x2 <- ParserK.adapt $
77+-- case x1 of
78+-- Digit _ -> Parser.satisfy isAlpha
79+-- Alpha _ -> Parser.satisfy isDigit
80+-- return $ case x1 of
81+-- Digit x -> [x,x2]
82+-- Alpha x -> [x,x2]
83+-- :}
84+--
85+-- == Experimental APIs
86+--
87+-- Please refer to "Streamly.Internal.Data.ParserK" for functions that have
88+-- not yet been released.
1389--
1490module Streamly.Data.ParserK
1591 (
92+ -- * Setup
93+ -- | To execute the code examples provided in this module in ghci, please
94+ -- run the following commands first.
95+ --
96+ -- $setup
97+
1698 -- * Parser Type
1799 ParserK
18100
19101 -- * Parsers
20102 -- ** Conversions
21− , fromFold
22− , fromParser
103+ , adapt
104+ , adaptC
105+ , adaptCG
23106 -- , toParser
24107
25108 -- ** Without Input
26109 , fromPure
27110 , fromEffect
28111 , die
112+
113+ -- * Deprecated
114+ , fromFold
115+ , fromParser
29116 )
30117
31118where
@@ -33,12 +120,20 @@ where
33120import Control.Monad.IO.Class (MonadIO)
34121import Streamly.Internal.Data.Fold (Fold)
35122import Streamly.Internal.Data.Unbox (Unbox)
123+import Streamly.Internal.Data.Array (Array)
36124import qualified Streamly.Internal.Data.Parser as ParserD
37125
38−import Streamly.Internal.Data.Parser.ParserK.Type
126+import Streamly.Internal.Data.ParserK.Type
39127
40−-- | Convert a 'Fold' to a 'ParserK'.
41−--
128+#include "DocTestDataParserK.hs"
129+
130+{-# DEPRECATED fromFold "Please use \"ParserK.adaptC . Parser.fromFold\" instead." #-}
42131{-# INLINE fromFold #-}
43−fromFold :: (MonadIO m, Unbox a) => Fold m a b -> ParserK a m b
44−fromFold = fromParser . ParserD.fromFold
132+fromFold :: (MonadIO m, Unbox a) => Fold m a b -> ParserK (Array a) m b
133+fromFold = adaptC . ParserD.fromFold
134+
135+{-# DEPRECATED fromParser "Please use \"adaptC\" instead." #-}
136+{-# INLINE fromParser #-}
137+fromParser ::
138+ (MonadIO m, Unbox a) => ParserD.Parser a m b -> ParserK (Array a) m b
139+fromParser = adaptC
core/src/Streamly/Data/Stream.hs+199−107
@@ -8,15 +8,20 @@
88-- Stability : released
99-- Portability : GHC
1010--
11−-- Fast, composable stream producers with ability to terminate, supporting
12−-- stream fusion.
11+-- Streams represented as state machines, that fuse together when composed
12+-- statically, eliminating function calls or intermediate constructor
13+-- allocations - generating tight, efficient loops. Suitable for high
14+-- performance looping operations.
15+--
16+-- If you need to call these operations recursively in a loop (i.e. composed
17+-- dynamically) then it is recommended to use the continuation passing style
18+-- (CPS) stream operations from the "Streamly.Data.StreamK" module. 'Stream'
19+-- and 'StreamK' types are interconvertible. See more details in the
20+-- documentation below regarding 'Stream' vs 'StreamK'.
1321--
1422-- Please refer to "Streamly.Internal.Data.Stream" for more functions that have
1523-- not yet been released.
1624--
17−-- For continuation passing style (CPS) stream type, please refer to
18−-- the "Streamly.Data.StreamK" module.
19−--
2025-- Checkout the <https://github.com/composewell/streamly-examples>
2126-- repository for many more real world examples of stream programming.
2227
@@ -37,9 +42,6 @@ module Streamly.Data.Stream
3742 -- * Construction
3843 -- | Functions ending in the general shape @b -> Stream m a@.
3944 --
40− -- See also: "Streamly.Internal.Data.Stream.StreamD.Generate" for
41− -- @Pre-release@ functions.
42− --
4345 -- Useful Idioms:
4446 --
4547 -- >>> fromIndices f = fmap f $ Stream.enumerateFrom 0
@@ -49,11 +51,12 @@ module Streamly.Data.Stream
4951 -- >>> fromFoldableM = Stream.sequence . fromFoldable
5052
5153 -- ** Primitives
52− -- | A fused 'Stream' is never constructed using 'cons', we always convert
53− -- other type of containers like list into streams, or generate it using
54− -- custom functions provided in this module. The 'cons' primitive has a
55− -- rare use in fusing a small number of elements. On the other hand, it is
56− -- common to construct 'StreamK' stream using the 'StreamK.cons' primitive.
54+ -- | A fused 'Stream' is never constructed using these primitives, they are
55+ -- typically generated by converting containers like list into streams, or
56+ -- generated using custom functions provided in this module. The 'cons'
57+ -- primitive in this module has a rare use in fusing a small number of
58+ -- elements. On the other hand, it is common to construct 'StreamK' stream
59+ -- using the StreamK.'StreamK.cons' primitive.
5760 , nil
5861 , nilM
5962 , cons
@@ -67,6 +70,15 @@ module Streamly.Data.Stream
6770
6871 -- ** From Values
6972 -- | Generate a monadic stream from a seed value or values.
73+ --
74+ -- Non-empty pure streams can be cycled using repeat and unfold:
75+ --
76+ -- >>> cycle = Stream.unfoldMany Unfold.fromList . Stream.repeat
77+ --
78+ -- Non-empty StreamK can be cycled using append:
79+ --
80+ -- >>> cycle xs = let ys = xs `StreamK.append` ys in ys
81+ --
7082 , fromPure
7183 , fromEffect
7284 , repeat
@@ -111,8 +123,6 @@ module Streamly.Data.Stream
111123 -- | Functions ending in the general shape @Stream m a -> m b@ or @Stream m
112124 -- a -> m (b, Stream m a)@
113125 --
114− -- See also: "Streamly.Internal.Data.Stream.StreamD.Eliminate" for @Pre-release@
115− -- functions.
116126
117127-- EXPLANATION: In imperative terms a fold can be considered as a loop over the stream
118128-- that reduces the stream to a single value.
@@ -211,21 +221,57 @@ module Streamly.Data.Stream
211221 -- operations like mapping a function over the stream.
212222 , foldrM
213223 , foldr
224+ -- foldr1
214225
215226 -- ** Specific Folds
216− -- | Usually you can use the folds in "Streamly.Data.Fold". However, some
217− -- folds that may be commonly used or may have an edge in performance in
218− -- some cases are provided here.
219− --
220− -- Useful idioms:
227+ -- | Streams are folded using folds in "Streamly.Data.Fold". Here are some
228+ -- idioms and equivalents of Data.List APIs using folds:
221229 --
222230 -- >>> foldlM' f a = Stream.fold (Fold.foldlM' f a)
223− -- >>> foldl1 f = Stream.fold (Fold.foldl1' f)
231+ -- >>> foldl1' f = Stream.fold (Fold.foldl1' f)
224232 -- >>> foldl' f a = Stream.fold (Fold.foldl' f a)
225233 -- >>> drain = Stream.fold Fold.drain
226234 -- >>> mapM_ f = Stream.fold (Fold.drainMapM f)
227235 -- >>> length = Stream.fold Fold.length
236+ -- >>> genericLength = Stream.fold Fold.lengthGeneric
228237 -- >>> head = Stream.fold Fold.one
238+ -- >>> last = Stream.fold Fold.latest
239+ -- >>> null = Stream.fold Fold.null
240+ -- >>> and = Stream.fold Fold.and
241+ -- >>> or = Stream.fold Fold.or
242+ -- >>> any p = Stream.fold (Fold.any p)
243+ -- >>> all p = Stream.fold (Fold.all p)
244+ -- >>> sum = Stream.fold Fold.sum
245+ -- >>> product = Stream.fold Fold.product
246+ -- >>> maximum = Stream.fold Fold.maximum
247+ -- >>> maximumBy cmp = Stream.fold (Fold.maximumBy cmp)
248+ -- >>> minimum = Stream.fold Fold.minimum
249+ -- >>> minimumBy cmp = Stream.fold (Fold.minimumBy cmp)
250+ -- >>> elem x = Stream.fold (Fold.elem x)
251+ -- >>> notElem x = Stream.fold (Fold.notElem x)
252+ -- >>> lookup x = Stream.fold (Fold.lookup x)
253+ -- >>> find p = Stream.fold (Fold.find p)
254+ -- >>> (!?) i = Stream.fold (Fold.index i)
255+ -- >>> genericIndex i = Stream.fold (Fold.indexGeneric i)
256+ -- >>> elemIndex x = Stream.fold (Fold.elemIndex x)
257+ -- >>> findIndex p = Stream.fold (Fold.findIndex p)
258+ --
259+ -- Some equivalents of Data.List APIs from the Stream module:
260+ --
261+ -- >>> head = fmap (fmap fst) . Stream.uncons
262+ -- >>> tail = fmap (fmap snd) . Stream.uncons
263+ -- >>> tail = Stream.tail -- unreleased API
264+ -- >>> init = Stream.init -- unreleased API
265+ --
266+ -- A Stream based toList fold implementation is provided below because it
267+ -- has a better performance compared to the fold.
268+
269+ -- Functions in Data.List, missing here:
270+ -- unsnoc = Stream.parseBreak (Parser.init Fold.toList)
271+ -- genericTake
272+ -- genericDrop
273+ -- genericSplitAt
274+ -- genericReplicate
229275 , toList
230276
231277 -- * Mapping
@@ -253,8 +299,6 @@ module Streamly.Data.Stream
253299 -- * Scanning
254300 -- | Stateful one-to-one transformations.
255301 --
256− -- See also: "Streamly.Internal.Data.Stream.StreamD.Transform" for
257− -- @Pre-release@ functions.
258302
259303 {-
260304 -- ** Left scans
@@ -281,6 +325,15 @@ module Streamly.Data.Stream
281325 , scan
282326 , postscan
283327 -- XXX postscan1 can be implemented using Monoids or Refolds.
328+ -- The following scans from Data.List are not provided.
329+ -- XXX scanl
330+ -- XXX scanl1
331+ -- XXX scanr
332+ -- XXX scanr1
333+ -- XXX mapAccumL
334+ -- XXX mapAccumR
335+ -- XXX inits
336+ -- XXX tails
284337
285338 -- ** Specific scans
286339 -- Indexing can be considered as a special type of zipping where we zip a
@@ -289,6 +342,8 @@ module Streamly.Data.Stream
289342
290343 -- * Insertion
291344 -- | Add elements to the stream.
345+ --
346+ -- >>> insert = Stream.insertBy compare
292347
293348 -- Inserting elements is a special case of interleaving/merging streams.
294349 , insertBy
@@ -324,12 +379,15 @@ module Streamly.Data.Stream
324379 -- "Streamly.Internal.Data.Fold" can be used along with 'scanMaybe' to
325380 -- perform stateful filtering operations in general.
326381 --
327− -- Useful idioms:
382+ -- Idioms and equivalents of Data.List APIs:
328383 --
329384 -- >>> deleteBy cmp x = Stream.scanMaybe (Fold.deleteBy cmp x)
385+ -- >>> deleteBy = Stream.deleteBy -- unreleased API
386+ -- >>> delete = deleteBy (==)
330387 -- >>> findIndices p = Stream.scanMaybe (Fold.findIndices p)
331388 -- >>> elemIndices a = findIndices (== a)
332389 -- >>> uniq = Stream.scanMaybe (Fold.uniqBy (==))
390+ -- >>> partition p = Stream.fold (Fold.partition Fold.toList Fold.toList) . fmap (if p then Left else Right)
333391 , scanMaybe
334392 , take
335393 , takeWhile
@@ -337,6 +395,8 @@ module Streamly.Data.Stream
337395 , drop
338396 , dropWhile
339397 , dropWhileM
398+ -- XXX write to an array in reverse and then read in reverse
399+ -- > dropWhileEnd = reverse . dropWhile p . reverse
340400
341401 -- XXX These are available as scans in folds. We need to check the
342402 -- performance though. If these are common and we need convenient stream
@@ -358,34 +418,42 @@ module Streamly.Data.Stream
358418 -- , elemIndices
359419
360420 -- * Combining Two Streams
421+ -- | Note that these operations are suitable for statically fusing a few
422+ -- streams, they have a quadratic O(n^2) time complexity wrt to the number
423+ -- of streams. If you want to compose many streams dynamically using binary
424+ -- combining operations see the corresponding operations in
425+ -- "Streamly.Data.StreamK".
426+ --
427+ -- When fusing more than two streams it is more efficient if the binary
428+ -- operations are composed as a balanced tree rather than a right
429+ -- associative or left associative one e.g.:
430+ --
431+ -- >>> s1 = Stream.fromList [1,2] `Stream.append` Stream.fromList [3,4]
432+ -- >>> s2 = Stream.fromList [4,5] `Stream.append` Stream.fromList [6,7]
433+ -- >>> s = s1 `Stream.append` s2
434+
361435 -- ** Appending
436+ -- | Equivalent of Data.List append:
437+ --
438+ -- >>> (++) = Stream.append
362439 , append
363440
364441 -- ** Interleaving
365− -- | When interleaving more than two streams you may want to interleave
366− -- them pairwise creating a balanced binary merge tree.
367442 , interleave
368443
369444 -- ** Merging
370− -- | When merging more than two streams you may want to merging them
371− -- pairwise creating a balanced binary merge tree.
372− --
373− -- Merging of @n@ streams can be performed by combining the streams pair
374− -- wise using 'mergeMapWith' to give O(n * log n) time complexity. If used
375− -- with 'concatMapWith' it will have O(n^2) performance.
376−
377445 , mergeBy
378446 , mergeByM
379447
380448 -- ** Zipping
381− -- | When zipping more than two streams you may want to zip them
382− -- pairwise creating a balanced binary tree.
449+ -- | Idioms and equivalents of Data.List APIs:
383450 --
384− -- Zipping of @n@ streams can be performed by combining the streams pair
385− -- wise using 'mergeMapWith' with O(n * log n) time complexity. If used
386− -- with 'concatMapWith' it will have O(n^2) performance.
451+ -- >>> zip = Stream.zipWith (,)
452+ -- >>> unzip = Stream.fold (Fold.unzip Fold.toList Fold.toList)
387453 , zipWith
388454 , zipWithM
455+ -- XXX zipWith3,4,5,6,7
456+ -- XXX unzip3,4,5,6,7
389457 -- , ZipStream (..)
390458
391459 -- ** Cross Product
@@ -418,48 +486,89 @@ module Streamly.Data.Stream
418486 -- One dimension loops are just a special case of nested loops. For
419487 -- example, 'concatMap' can degenerate to a simple map operation:
420488 --
421− -- > map f m = S.concatMap (\x -> S.fromPure (f x)) m
489+ -- >>> map f m = Stream.concatMap (\x -> Stream.fromPure (f x)) m
422490 --
423491 -- Similarly, 'concatMap' can perform filtering by mapping an element to a
424492 -- 'nil' stream:
425493 --
426− -- > filter p m = S.concatMap (\x -> if p x then S.fromPure x else S.nil) m
494+ -- >>> filter p m = Stream.concatMap (\x -> if p x then Stream.fromPure x else Stream.nil) m
427495 --
496+ -- Idioms and equivalents of Data.List APIs:
497+ --
498+ -- >>> concat = Stream.concatMap id
499+ -- >>> unlines = Stream.interposeSuffix '\n'
500+ -- >>> unlines = Stream.intercalateSuffix Unfold.fromList "\n"
501+ -- >>> unwords = Stream.interpose ' '
502+ -- >>> unwords = Stream.intercalate Unfold.fromList " "
428503
429504 , concatEffect
430505 , concatMap
431506 , concatMapM
432507
433508 -- * Repeated Fold
434− -- | Useful idioms:
509+ -- | Idioms and equivalents of Data.List APIs:
435510 --
436− -- >>> splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
437− -- >>> splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
438− -- >>> groupsBy eq f = Stream.parseMany (Parser.groupBy eq f)
439− -- >>> groupsByRolling eq f = Stream.parseMany (Parser.groupByRolling eq f)
440− -- >>> groupsOf n f = Stream.foldMany (Fold.take n f)
511+ -- >>> groupsOf n = Stream.foldMany (Fold.take n Fold.toList)
512+ -- >>> groupBy eq = Stream.groupsWhile eq Fold.toList
513+ -- >>> groupBy eq = Stream.parseMany (Parser.groupBy eq Fold.toList)
514+ -- >>> groupsByRolling eq = Stream.parseMany (Parser.groupByRolling eq Fold.toList)
515+ -- >>> groups = groupBy (==)
441516 , foldMany -- XXX Rename to foldRepeat
442517 , groupsOf
443518 , parseMany
444519
445520 -- * Splitting
521+ -- | Idioms and equivalents of Data.List APIs:
522+ --
523+ -- >>> splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
524+ -- >>> splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
525+ -- >>> lines = splitOnSuffix (== '\n')
526+ -- >>> words = Stream.wordsBy isSpace
527+ -- >>> splitAt n = Stream.fold (Fold.splitAt n Fold.toList Fold.toList)
528+ -- >>> span p = Parser.splitWith (,) (Parser.takeWhile p Fold.toList) (Parser.fromFold Fold.toList)
529+ -- >>> break p = span (not . p)
446530 , splitOn
447531 , wordsBy
448532
449533 -- * Buffered Operations
450534 -- | Operations that require buffering of the stream.
451535 -- Reverse is essentially a left fold followed by an unfold.
536+ --
537+ -- Idioms and equivalents of Data.List APIs:
538+ --
539+ -- >>> nub = Stream.fold Fold.toList . Stream.scanMaybe Fold.nub
540+ -- >>> nub = Stream.ordNub -- unreleased API
541+ -- >>> sortBy = StreamK.sortBy
542+ -- >>> sortOn f = StreamK.sortOn -- unreleased API
543+ -- >>> deleteFirstsBy = Stream.deleteFirstsBy -- unreleased
544+ -- >>> (\\) = Stream.deleteFirstsBy (==) -- unreleased
545+ -- >>> intersectBy = Stream.intersectBy -- unreleased
546+ -- >>> intersect = Stream.intersectBy (==) -- unreleased
547+ -- >>> unionBy = Stream.unionBy -- unreleased
548+ -- >>> union = Stream.unionBy (==) -- unreleased
549+ --
452550 , reverse
551+ -- XXX transpose: write the streams to arrays and then stream transposed.
552+ -- XXX subsequences
553+ -- XXX permutations
554+ -- , nub
555+ -- , ordNub
556+ -- , nubBy
453557
454558 -- * Multi-Stream folds
455559 -- | Operations that consume multiple streams at the same time.
456560 , eqBy
457561 , cmpBy
458562 , isPrefixOf
563+ , isInfixOf
564+ -- , isSuffixOf
565+ -- , isSuffixOfUnbox
459566 , isSubsequenceOf
460567
461568 -- trimming sequences
462− , stripPrefix
569+ -- , stripPrefix
570+ -- , stripSuffix
571+ -- , stripSuffixUnbox
463572
464573 -- Exceptions and resource management depend on the "exceptions" package
465574 -- XXX We can have IO Stream operations not depending on "exceptions"
@@ -480,9 +589,6 @@ module Streamly.Data.Stream
480589 -- For example, instead of calling them on a stream of chars call them on a
481590 -- stream of arrays before flattening it to a stream of chars.
482591 --
483− -- See also: "Streamly.Internal.Data.Stream.StreamD.Exception" for
484− -- @Pre-release@ functions.
485−
486592 , onException
487593 , handle
488594
@@ -516,25 +622,20 @@ module Streamly.Data.Stream
516622 , runReaderT
517623 , runStateT
518624
625+ -- XXX Arrays could be different types, therefore, this should be in
626+ -- specific array module. Or maybe we should abstract over array types.
519627 -- * Stream of Arrays
520628 , Array.chunksOf
521− , chunksSplitOn
522629 )
523630where
524631
525−import Control.Monad.IO.Class (MonadIO)
526−import Data.Word (Word8)
527−import Streamly.Data.Array (Array)
528632import Streamly.Internal.Data.Stream
529633import Prelude
530634 hiding (filter, drop, dropWhile, take, takeWhile, zipWith, foldr,
531− foldl, map, mapM, mapM_, sequence, all, any, sum, product, elem,
532− notElem, maximum, minimum, head, last, tail, length, null,
533− reverse, iterate, init, and, or, lookup, foldr1, (!!),
534− scanl, scanl1, repeat, replicate, concatMap, span)
635+ mapM, sequence, reverse, iterate, foldr1, repeat, replicate,
636+ concatMap)
535637
536638import qualified Streamly.Internal.Data.Array.Type as Array
537−import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
538639
539640#include "DocTestDataStream.hs"
540641
@@ -573,11 +674,11 @@ import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
573674--
574675-- == Stream Fusion
575676--
576−-- The fused 'Stream' type employs stream fusion for C-like performance when
577−-- looping over data. It represents a stream source or transformation by
578−-- defining a state machine with explicit state, and a step function working on
579−-- the state. A typical stream operation consumes elements from the previous
580−-- state machine in the pipeline, transforms them and yields new values for the
677+-- The fused 'Stream' type in this module employs stream fusion for C-like
678+-- performance when looping over data. It represents the stream as a state
679+-- machine using an explicit state, and a step function working on the state. A
680+-- typical stream operation consumes elements from the previous state machine
681+-- in a stream pipeline, transforms the elements and yields new values for the
581682-- next stage to consume. The stream operations are modular and represent a
582683-- single task, they have no knowledge of previous or next operation on the
583684-- elements.
@@ -586,7 +687,7 @@ import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
586687-- transformation operations and a stream consumer. All these operations taken
587688-- together form a closed loop processing the stream elements. Elements are
588689-- transferred between stages using a boxed data constructor. However, all the
589−-- stages of the pipeline are fused together by GHC, eliminating the
690+-- stages of the pipeline are fused together by GHC, eliminating the boxing of
590691-- intermediate constructors, and thus forming a tight C like loop without any
591692-- boxed data being used in the loop.
592693--
@@ -601,54 +702,45 @@ import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
601702--
602703-- == Stream vs StreamK
603704--
604−-- The fused stream model avoids constructor allocations or function call
605−-- overheads. However, the stream is represented as a state machine and to
606−-- generate elements it has to navigate the decision tree of the state machine.
607−-- Moreover, the state machine is cranked for each element in the stream. This
608−-- performs extremely well when the number of states are limited. The state
609−-- machine starts getting expensive as the number of states increase. For
610−-- example, generating a million element stream from a list requires a single
611−-- state and is very efficient. However, using fused 'cons' to generate a
612−-- million element stream would be a disaster.
705+-- The fused stream model avoids constructor allocations and function call
706+-- overheads. However, the stream is represented as a state machine, and to
707+-- generate stream elements it has to navigate the decision tree of the state
708+-- machine. Moreover, the state machine is cranked for each element in the
709+-- stream. This performs extremely well when the number of states are limited.
710+-- The state machine starts getting expensive as the number of states increase.
711+-- For example, generating a stream from a list requires a single state and is
712+-- very efficient, even if it has millions of elements. However, using 'cons'
713+-- to construct a million element stream would be a disaster.
613714--
614715-- A typical worst case scenario for fused stream model is a large number of
615716-- `cons` or `append` operations. A few static `cons` or `append` operations
616−-- are very fast and much faster than a CPS style stream. However, if we
617−-- construct a large stream using `cons` it introduces as many states in the
618−-- state machine as the number of elements. If we compose the `cons` as a
619−-- binary tree it will take @n * log n@ time to navigate the tree, and @n * n@
620−-- if it is a right associative composition.
621−--
622−-- For quadratic cases of fused stream, after a certain threshold the CPS
623−-- stream would perform much better and exhibit linear performance behavior.
717+-- are very fast and much faster than a CPS style stream because CPS involves a
718+-- function call for each element whereas fused stream involves a few
719+-- conditional branches in the state machine. However, constructing a large
720+-- stream using `cons` introduces as many states in the state machine as the
721+-- number of elements. If we compose `cons` as a balanced binary tree it will
722+-- take @n * log n@ time to navigate the tree, and @n * n@ if it is a right
723+-- associative composition.
724+--
624725-- Operations like 'cons' or 'append'; are typically recursively called to
625726-- construct a lazy infinite stream. For such use cases the CPS style 'StreamK'
626−-- type is provided. CPS streams do not have a state machine that needs to be
727+-- should be used. CPS streams do not have a state machine that needs to be
627728-- cranked for each element, past state has no effect on the future element
628−-- processing. However, it incurs a function call overhead for each operation
629−-- for each element, which could be very large overhead compared to fused state
630−-- machines even if it has many states and cranks it for each element. But in
631−-- some cases scales tip in favor of the CPS stream. In those cases even though
632−-- CPS has a large constant overhead, it has a linear performance rather than
633−-- quadratic.
634−--
635−-- As a general guideline, if you have to use 'cons' or 'append' or operations
636−-- of similar nature, at a large scale, then 'StreamK' should be used. When you
637−-- need to compose the stream dynamically or recursively, then 'StreamK' should
638−-- be used. Typically you would use a dynamically generated 'StreamK' with
639−-- chunks of data which can then be processed by statically fused stream
640−-- pipeline operations.
729+-- processing. However, CPS incurs a function call overhead for each element
730+-- processed, the overhead could be large compared to a fused state machine
731+-- even if it has many states. However, because of its linear performance
732+-- characterstics, after a certain threshold of stream compositions the CPS
733+-- stream would perform much better than the quadratic fused stream operations.
734+--
735+-- As a general guideline, you need to use 'StreamK' when you have to use
736+-- 'cons', 'append' or other operations having quadratic complexity at a large
737+-- scale. Typically, in such cases you need to compose the stream recursively,
738+-- by calling an operation in a loop. The decision to compose the stream is
739+-- taken at run time rather than statically at compile time.
740+--
741+-- Typically you would compose a 'StreamK' of chunks of data so that the
742+-- StreamK overhead is not high, and then process the chunks using 'Stream' by
743+-- using statically fused stream pipeline operations on the chunks.
641744--
642745-- 'Stream' and 'StreamK' types can be interconverted. See
643746-- "Streamly.Data.StreamK" module for conversion operations.
644−
645−-- | Split a stream of arrays on a given separator byte, dropping the separator
646−-- and coalescing all the arrays between two separators into a single array.
647−--
648−{-# INLINE chunksSplitOn #-}
649−chunksSplitOn
650− :: (MonadIO m)
651− => Word8
652− -> Stream m (Array Word8)
653− -> Stream m (Array Word8)
654−chunksSplitOn = ArrayStream.splitOn
core/src/Streamly/Data/StreamK.hs+37−19
@@ -8,18 +8,30 @@
88-- Stability : released
99-- Portability : GHC
1010--
11−-- Streams using Continuation Passing Style (CPS). See the @Stream vs StreamK@
12−-- section in the "Streamly.Data.Stream" module to know when to use this
13−-- module.
14−--
15−-- Please refer to "Streamly.Internal.Data.StreamK" for more functions
16−-- that have not yet been released.
17−--
18−-- Some useful idioms:
19−--
20−-- >>> concatFoldableWith f = Prelude.foldr f StreamK.nil
21−-- >>> concatMapFoldableWith f g = Prelude.foldr (f . g) StreamK.nil
22−-- >>> concatForFoldableWith f xs g = Prelude.foldr (f . g) StreamK.nil xs
11+-- Streams represented as chains of functions calls using Continuation Passing
12+-- Style (CPS), suitable for dynamically composing potentially large number of
13+-- streams.
14+--
15+-- Unlike the statically fused operations in "Streamly.Data.Stream", StreamK
16+-- operations are less efficient, involving a function call overhead for each
17+-- element, but they exhibit linear O(n) time complexity wrt to the number of
18+-- stream compositions. Therefore, they are suitable for dynamically composing
19+-- streams e.g. appending potentially infinite streams in recursive loops.
20+-- While fused streams can be used to efficiently process elements as small as
21+-- a single byte, CPS streams are typically used on bigger chunks of data to
22+-- avoid the larger overhead per element. For more details See the @Stream vs
23+-- StreamK@ section in the "Streamly.Data.Stream" module.
24+--
25+-- In addition to the combinators in this module, you can use operations from
26+-- "Streamly.Data.Stream" for StreamK as well by converting StreamK to Stream
27+-- ('toStream'), and vice-versa ('fromStream'). Please refer to
28+-- "Streamly.Internal.Data.StreamK" for more functions that have not yet been
29+-- released.
30+--
31+-- For documentation see the corresponding combinators in
32+-- "Streamly.Data.Stream". Documentation has been omitted in this module unless
33+-- there is a difference worth mentioning or if the combinator does not exist
34+-- in "Streamly.Data.Stream".
2335
2436-- Notes:
2537--
@@ -83,7 +95,8 @@ module Streamly.Data.StreamK
8395 -- , foldBreak
8496
8597 -- ** Parsing
86− -- , parseBreak
98+ , parse
99+ , parseBreak
87100 , parseBreakChunks
88101 , parseChunks
89102
@@ -93,6 +106,11 @@ module Streamly.Data.StreamK
93106 , take
94107
95108 -- * Combining Two Streams
109+ -- | Unlike the operations in "Streamly.Data.Stream", these operations can
110+ -- be used to dynamically compose large number of streams e.g. using the
111+ -- 'concatMapWith' and 'mergeMapWith' operations. They have a linear O(n)
112+ -- time complexity wrt to the number of streams being composed.
113+
96114 -- ** Appending
97115 , append
98116
@@ -116,6 +134,12 @@ module Streamly.Data.StreamK
116134 -- , CrossStreamK (..)
117135
118136 -- * Stream of streams
137+ -- | Some useful idioms:
138+ --
139+ -- >>> concatFoldableWith f = Prelude.foldr f StreamK.nil
140+ -- >>> concatMapFoldableWith f g = Prelude.foldr (f . g) StreamK.nil
141+ -- >>> concatForFoldableWith f xs g = Prelude.foldr (f . g) StreamK.nil xs
142+ --
119143 , concatEffect
120144 -- , concatMap
121145 , concatMapWith
@@ -168,9 +192,3 @@ import Prelude hiding (reverse, zipWith, mapM, dropWhile, take)
168192-- 1
169193-- 2
170194-- [1,2]
171−--
172−-- == Exception Handling
173−--
174−-- There are no native exception handling operations in the StreamK module,
175−-- please convert to 'Stream' type and use exception handling operations from
176−-- "Streamly.Data.Stream".
core/src/Streamly/Data/Unbox.hsdeleted−16
@@ -1,16 +0,0 @@
1−-- |
2−-- Module : Streamly.Data.Unbox
3−-- Copyright : (c) 2023 Composewell Technologies
4−-- License : BSD3-3-Clause
5−-- Maintainer : streamly@composewell.com
6−-- Portability : GHC
7−--
8−module Streamly.Data.Unbox
9− ( Unbox(..)
10− , MutableByteArray
11− , deriveUnbox
12− , deriveUnboxWith
13− ) where
14−
15−import Streamly.Internal.Data.Unbox
16−import Streamly.Internal.Data.Unbox.TH
core/src/Streamly/Data/Unfold.hs+3−1
@@ -39,7 +39,7 @@ module Streamly.Data.Unfold
3939
4040 -- * Unfolds
4141 -- One to one correspondence with
42− -- "Streamly.Internal.Data.Stream.StreamD.Generate"
42+ -- "Streamly.Internal.Data.Stream.Generate"
4343
4444 -- ** Basic Constructors
4545 , unfoldrM
@@ -65,6 +65,8 @@ module Streamly.Data.Unfold
6565 -- ** Mapping on Input
6666 , lmap
6767 , lmapM
68+ , first
69+ , second
6870
6971 -- ** Mapping on Output
7072 , mapM
core/src/Streamly/FileSystem/Handle.hs+15−29
@@ -1,4 +1,4 @@
1−#include "inline.hs"
1+{-# LANGUAGE CPP #-}
22
33-- |
44-- Module : Streamly.FileSystem.Handle
@@ -40,6 +40,12 @@
4040--
4141module Streamly.FileSystem.Handle
4242 (
43+ -- * Setup
44+ -- | To execute the code examples provided in this module in ghci, please
45+ -- run the following commands first.
46+ --
47+ -- $setup
48+
4349 -- * Singleton IO
4450 -- | Read or write a single buffer.
4551 getChunk
@@ -75,13 +81,13 @@ module Streamly.FileSystem.Handle
7581 -- position of the file handle. The stream ends as soon as EOF is
7682 -- encountered.
7783
78− -- -- *** Streams
79− -- , read
80− -- , readWith
81− -- , readChunks
82− -- , readChunksWith
84+ -- *** Streams
85+ , read
86+ , readWith
87+ , readChunks
88+ , readChunksWith
8389
84− -- -- *** Unfolds
90+ -- *** Unfolds
8591 , reader
8692 , readerWith
8793 , chunkReader
@@ -98,34 +104,14 @@ module Streamly.FileSystem.Handle
98104 , writeChunks
99105
100106 -- * Deprecated
101− , read
102107 , readWithBufferOf
103− , readChunks
104108 , readChunksWithBufferOf
105109 , writeChunksWithBufferOf
106110 , writeWithBufferOf
107111 )
108112where
109113
110−import Control.Monad.IO.Class (MonadIO(..))
111−import Data.Word (Word8)
112−import Streamly.Internal.Data.Array.Type (Array)
113−import Streamly.Internal.Data.Unfold.Type (Unfold)
114−import System.IO (Handle)
115−
116−import Streamly.Internal.FileSystem.Handle hiding (read, readChunks)
114+import Streamly.Internal.FileSystem.Handle
117115import Prelude hiding (read)
118116
119−-- | Same as 'reader'
120−--
121−{-# DEPRECATED read "Please use 'reader' instead" #-}
122−{-# INLINE read #-}
123−read :: MonadIO m => Unfold m Handle Word8
124−read = reader
125−
126−-- | Same as 'chunkReader'
127−--
128−{-# DEPRECATED readChunks "Please use 'chunkReader' instead" #-}
129−{-# INLINE readChunks #-}
130−readChunks :: MonadIO m => Unfold m Handle (Array Word8)
131−readChunks = chunkReader
117+#include "DocTestFileSystemHandle.hs"
core/src/Streamly/Internal/Data/Array.hs+610−163
@@ -2,7 +2,6 @@
22-- |
33-- Module : Streamly.Internal.Data.Array
44-- Copyright : (c) 2019 Composewell Technologies
5−--
65-- License : BSD3
76-- Maintainer : streamly@composewell.com
87-- Stability : experimental
@@ -17,59 +16,33 @@ module Streamly.Internal.Data.Array
1716 -- $design
1817
1918 -- * The Array Type
20− Array
19+ module Streamly.Internal.Data.Array.Type
2120
2221 -- * Construction
23−
24− -- Pure List APIs
25− , A.fromListN
26− , A.fromList
27−
28− -- Stream Folds
29− , fromStreamN
30− , fromStream
31−
3222 -- Monadic Folds
33− , A.writeN -- drop new
34− , A.pinnedWriteNAligned
35− , A.write -- full buffer
3623 , writeLastN
3724
38− -- * Elimination
39− -- ** Conversion
40− , A.toList
41−
42− -- ** Streams
43− , A.read
44− , A.readRev
45−
46− -- ** Unfolds
47− , reader
48− , readerUnsafe
49− , A.readerRev
50− , producer -- experimental
51−
5225 -- * Random Access
5326 -- , (!!)
5427 , getIndex
55− , A.unsafeIndex -- XXX Rename to getIndexUnsafe??
5628 , getIndexRev
57− , last -- XXX getIndexLast?
58− , getIndices
59− , getIndicesFromThenTo
29+ , last -- XXX getLastIndex?
6030 -- , getIndicesFrom -- read from a given position to the end of file
6131 -- , getIndicesUpto -- read from beginning up to the given position
6232 -- , getIndicesFromTo
6333 -- , getIndicesFromRev -- read from a given position to the beginning of file
6434 -- , getIndicesUptoRev -- read from end to the given position in file
35+ , indexReader
36+ , indexReaderFromThenTo
6537
6638 -- * Size
67− , length
6839 , null
6940
7041 -- * Search
7142 , binarySearch
7243 , findIndicesOf
44+ -- getIndicesOf
45+ , indexFinder -- see splitOn
7346 -- , findIndexOf
7447 -- , find
7548
@@ -77,66 +50,96 @@ module Streamly.Internal.Data.Array
7750 , cast
7851 , asBytes
7952 , castUnsafe
80− , asPtrUnsafe
8153 , asCStringUnsafe
82− , A.unsafeFreeze -- asImmutableUnsafe?
83− , A.unsafeThaw -- asMutableUnsafe?
8454
8555 -- * Subarrays
8656 , getSliceUnsafe
8757 -- , getSlice
88− , genSlicesFromLen
89− , getSlicesFromLen
58+ , sliceIndexerFromLen
59+ , slicerFromLen
9060 , splitOn
9161
9262 -- * Streaming Operations
9363 , streamTransform
9464
95− -- ** Folding
65+ -- * Folding
9666 , streamFold
9767 , fold
9868
69+ -- * Stream of Arrays
70+
71+ -- XXX these are probably not very useful to have in this module as we can
72+ -- express these idiomatically using streams.
73+ , interpose
74+ , interposeSuffix
75+ , intercalateSuffix
76+
77+ , compactLE
78+ , pinnedCompactLE
79+ , compactOnByte
80+ , compactOnByteSuffix
81+
82+ , foldBreakChunks
83+ , foldChunks
84+ , foldBreakChunksK
85+ , parseBreakChunksK
86+
87+ -- * Serialization
88+ , encodeAs
89+ , serialize
90+ , pinnedSerialize
91+ , deserialize
92+
9993 -- * Deprecated
100− , A.toStream
101− , A.toStreamRev
94+ , genSlicesFromLen
95+ , getSlicesFromLen
96+ , getIndices
10297 )
10398where
10499
100+#include "assert.hs"
105101#include "inline.hs"
106102#include "ArrayMacros.h"
107103
108−import Control.Exception (assert)
109−import Control.Monad (when)
110104import Control.Monad.IO.Class (MonadIO(..))
105+-- import Data.Bifunctor (first)
106+-- import Data.Either (fromRight)
111107import Data.Functor.Identity (Identity)
112108import Data.Proxy (Proxy(..))
113109import Data.Word (Word8)
114110import Foreign.C.String (CString)
115111import Foreign.Ptr (castPtr)
116112import Foreign.Storable (Storable)
113+import GHC.Types (SPEC(..))
117114import Streamly.Internal.Data.Unbox (Unbox(..))
118115import Prelude hiding (length, null, last, map, (!!), read, concat)
119116
120−import Streamly.Internal.Data.MutArray.Type (ArrayUnsafe(..))
121−import Streamly.Internal.Data.Array.Type
122− (Array(..), length, asPtrUnsafe)
117+import Streamly.Internal.Data.MutByteArray.Type (PinnedState(..), MutByteArray)
118+import Streamly.Internal.Data.Serialize.Type (Serialize)
123119import Streamly.Internal.Data.Fold.Type (Fold(..))
124−import Streamly.Internal.Data.Producer.Type (Producer(..))
125−import Streamly.Internal.Data.Stream (Stream)
126−import Streamly.Internal.Data.Tuple.Strict (Tuple3Fused'(..))
120+import Streamly.Internal.Data.Parser (Parser(..), Initial(..), ParseError(..))
121+import Streamly.Internal.Data.Stream (Stream(..))
122+import Streamly.Internal.Data.StreamK (StreamK)
123+import Streamly.Internal.Data.SVar.Type (adaptState, defState)
124+import Streamly.Internal.Data.Tuple.Strict (Tuple'(..), Tuple3Fused'(..))
127125import Streamly.Internal.Data.Unfold.Type (Unfold(..))
128126import Streamly.Internal.System.IO (unsafeInlineIO)
129127
130−import qualified Streamly.Internal.Data.MutArray.Type as MA
128+import qualified Streamly.Internal.Data.Fold.Type as Fold
129+import qualified Streamly.Internal.Data.Serialize.Type as Serialize
130+import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
131131import qualified Streamly.Internal.Data.MutArray as MA
132−import qualified Streamly.Internal.Data.Array.Type as A
133132import qualified Streamly.Internal.Data.Fold as FL
134−import qualified Streamly.Internal.Data.Producer.Type as Producer
135−import qualified Streamly.Internal.Data.Producer as Producer
136−import qualified Streamly.Internal.Data.Ring.Unboxed as RB
133+import qualified Streamly.Internal.Data.Ring as RB
134+import qualified Streamly.Internal.Data.Parser as Parser
135+-- import qualified Streamly.Internal.Data.ParserK as ParserK
137136import qualified Streamly.Internal.Data.Stream as D
138137import qualified Streamly.Internal.Data.Stream as Stream
138+import qualified Streamly.Internal.Data.StreamK as StreamK
139139import qualified Streamly.Internal.Data.Unfold as Unfold
140+import qualified Prelude
141+
142+import Streamly.Internal.Data.Array.Type
140143
141144#include "DocTestDataArray.hs"
142145
@@ -163,91 +166,18 @@ import qualified Streamly.Internal.Data.Unfold as Unfold
163166-- if we use one array for each chunk. This is still significant to add
164167-- pressure to GC.
165168
166−-------------------------------------------------------------------------------
167−-- Construction
168−-------------------------------------------------------------------------------
169−
170−-- | Create an 'Array' from the first N elements of a stream. The array is
171−-- allocated to size N, if the stream terminates before N elements then the
172−-- array may hold less than N elements.
173−--
174−-- /Pre-release/
175−{-# INLINE fromStreamN #-}
176−fromStreamN :: (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
177−fromStreamN n m = do
178− when (n < 0) $ error "writeN: negative write count specified"
179− A.fromStreamDN n m
180−
181−-- | Create an 'Array' from a stream. This is useful when we want to create a
182−-- single array from a stream of unknown size. 'writeN' is at least twice
183−-- as efficient when the size is already known.
184−--
185−-- Note that if the input stream is too large memory allocation for the array
186−-- may fail. When the stream size is not known, `chunksOf` followed by
187−-- processing of indvidual arrays in the resulting stream should be preferred.
188−--
189−-- /Pre-release/
190−{-# INLINE fromStream #-}
191−fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (Array a)
192−fromStream = Stream.fold A.write
193−-- write m = A.fromStreamD $ D.fromStreamK m
194−
195169-------------------------------------------------------------------------------
196170-- Elimination
197171-------------------------------------------------------------------------------
198172
199−{-# INLINE_NORMAL producer #-}
200−producer :: forall m a. (Monad m, Unbox a) => Producer m (Array a) a
201−producer =
202− Producer.translate A.unsafeThaw A.unsafeFreeze
203− $ MA.producerWith (return . unsafeInlineIO)
204−
205−-- | Unfold an array into a stream.
206−--
207−{-# INLINE_NORMAL reader #-}
208−reader :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
209−reader = Producer.simplify producer
210−
211−-- | Unfold an array into a stream, does not check the end of the array, the
212−-- user is responsible for terminating the stream within the array bounds. For
213−-- high performance application where the end condition can be determined by
214−-- a terminating fold.
215−--
216−-- Written in the hope that it may be faster than "read", however, in the case
217−-- for which this was written, "read" proves to be faster even though the core
218−-- generated with unsafeRead looks simpler.
219−--
220−-- /Pre-release/
221−--
222−{-# INLINE_NORMAL readerUnsafe #-}
223−readerUnsafe :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
224−readerUnsafe = Unfold step inject
225− where
226−
227− inject (Array contents start end) =
228− return (ArrayUnsafe contents end start)
229−
230− {-# INLINE_LATE step #-}
231− step (ArrayUnsafe contents end p) = do
232− -- unsafeInlineIO allows us to run this in Identity monad for pure
233− -- toList/foldr case which makes them much faster due to not
234− -- accumulating the list and fusing better with the pure consumers.
235− --
236− -- This should be safe as the array contents are guaranteed to be
237− -- evaluated/written to before we peek at them.
238− let !x = unsafeInlineIO $ peekByteIndex p contents
239− let !p1 = INDEX_NEXT(p,a)
240− return $ D.Yield x (ArrayUnsafe contents end p1)
241−
242173-- |
243174--
244−-- >>> import qualified Streamly.Internal.Data.Array.Type as Array
245175-- >>> null arr = Array.byteLength arr == 0
246176--
247177-- /Pre-release/
248178{-# INLINE null #-}
249179null :: Array a -> Bool
250−null arr = A.byteLength arr == 0
180+null arr = byteLength arr == 0
251181
252182-- | Like 'getIndex' but indexes the array in reverse from the end.
253183--
@@ -259,12 +189,11 @@ getIndexRev i arr =
259189 $ do
260190 let elemPtr = RINDEX_OF(arrEnd arr, i, a)
261191 if i >= 0 && elemPtr >= arrStart arr
262− then Just <$> peekByteIndex elemPtr (arrContents arr)
192+ then Just <$> peekAt elemPtr (arrContents arr)
263193 else return Nothing
264194
265195-- |
266196--
267−-- >>> import qualified Streamly.Internal.Data.Array as Array
268197-- >>> last arr = Array.getIndexRev arr 0
269198--
270199-- /Pre-release/
@@ -276,6 +205,8 @@ last = getIndexRev 0
276205-- Folds with Array as the container
277206-------------------------------------------------------------------------------
278207
208+-- XXX We should generate this from Ring.
209+
279210-- | @writeLastN n@ folds a maximum of @n@ elements from the end of the input
280211-- stream to an 'Array'.
281212--
@@ -284,7 +215,7 @@ writeLastN ::
284215 (Storable a, Unbox a, MonadIO m) => Int -> Fold m a (Array a)
285216writeLastN n
286217 | n <= 0 = fmap (const mempty) FL.drain
287− | otherwise = A.unsafeFreeze <$> Fold step initial done
218+ | otherwise = unsafeFreeze <$> Fold step initial done done
288219
289220 where
290221
@@ -297,11 +228,9 @@ writeLastN n
297228 in fmap f $ liftIO $ RB.new n
298229
299230 done (Tuple3Fused' rb rh i) = do
300− arr <- liftIO $ MA.new n
301− foldFunc i rh snoc' arr rb
302−
303− -- XXX We should write a read unfold for ring.
304− snoc' b a = liftIO $ MA.snocUnsafe b a
231+ arr <- MA.new n
232+ -- XXX We should write a read unfold for ring.
233+ foldFunc i rh MA.snocUnsafe arr rb
305234
306235 foldFunc i
307236 | i < n = RB.unsafeFoldRingM
@@ -327,19 +256,26 @@ binarySearch = undefined
327256-- compared to streams by using SIMD instructions.
328257-- We can also return a bit array instead.
329258
259+-- Can use SIMD.
260+
330261-- | Perform a linear search to find all the indices where a given element is
331262-- present in an array.
332263--
333264-- /Unimplemented/
334−findIndicesOf :: (a -> Bool) -> Unfold Identity (Array a) Int
335−findIndicesOf = undefined
265+indexFinder :: (a -> Bool) -> Unfold Identity (Array a) Int
266+indexFinder = undefined
267+
268+-- |
269+-- /Unimplemented/
270+findIndicesOf :: (a -> Bool) -> Array a -> Stream Identity Int
271+findIndicesOf p = Stream.unfold (indexFinder p)
336272
337273{-
338274findIndexOf :: (a -> Bool) -> Array a -> Maybe Int
339−findIndexOf p = Unfold.fold Fold.one . Stream.unfold (findIndicesOf p)
275+findIndexOf p = Unfold.fold Fold.one . Stream.unfold (indexFinder p)
340276
341277find :: (a -> Bool) -> Array a -> Bool
342−find = Unfold.fold Fold.null . Stream.unfold (findIndicesOf p)
278+find = Unfold.fold Fold.null . Stream.unfold (indexFinder p)
343279-}
344280
345281-------------------------------------------------------------------------------
@@ -381,29 +317,45 @@ splitOn :: (Monad m, Unbox a) =>
381317 (a -> Bool) -> Array a -> Stream m (Array a)
382318splitOn predicate arr =
383319 fmap (\(i, len) -> getSliceUnsafe i len arr)
384− $ D.sliceOnSuffix predicate (A.toStreamD arr)
320+ $ D.indexOnSuffix predicate (read arr)
321+
322+{-# INLINE sliceIndexerFromLen #-}
323+sliceIndexerFromLen :: forall m a. (Monad m, Unbox a)
324+ => Int -- ^ from index
325+ -> Int -- ^ length of the slice
326+ -> Unfold m (Array a) (Int, Int)
327+sliceIndexerFromLen from len =
328+ Unfold.lmap unsafeThaw (MA.sliceIndexerFromLen from len)
385329
330+{-# DEPRECATED genSlicesFromLen "Please use sliceIndexerFromLen instead." #-}
386331{-# INLINE genSlicesFromLen #-}
387332genSlicesFromLen :: forall m a. (Monad m, Unbox a)
388333 => Int -- ^ from index
389334 -> Int -- ^ length of the slice
390335 -> Unfold m (Array a) (Int, Int)
391−genSlicesFromLen from len =
392− Unfold.lmap A.unsafeThaw (MA.genSlicesFromLen from len)
336+genSlicesFromLen = sliceIndexerFromLen
393337
394338-- | Generate a stream of slices of specified length from an array, starting
395339-- from the supplied array index. The last slice may be shorter than the
396340-- requested length.
397341--
398342-- /Pre-release//
343+{-# INLINE slicerFromLen #-}
344+slicerFromLen :: forall m a. (Monad m, Unbox a)
345+ => Int -- ^ from index
346+ -> Int -- ^ length of the slice
347+ -> Unfold m (Array a) (Array a)
348+slicerFromLen from len =
349+ fmap unsafeFreeze
350+ $ Unfold.lmap unsafeThaw (MA.slicerFromLen from len)
351+
352+{-# DEPRECATED getSlicesFromLen "Please use slicerFromLen instead." #-}
399353{-# INLINE getSlicesFromLen #-}
400354getSlicesFromLen :: forall m a. (Monad m, Unbox a)
401355 => Int -- ^ from index
402356 -> Int -- ^ length of the slice
403357 -> Unfold m (Array a) (Array a)
404−getSlicesFromLen from len =
405− fmap A.unsafeFreeze
406− $ Unfold.lmap A.unsafeThaw (MA.getSlicesFromLen from len)
358+getSlicesFromLen = slicerFromLen
407359
408360-------------------------------------------------------------------------------
409361-- Random reads and writes
@@ -421,7 +373,7 @@ getIndex i arr =
421373 $ do
422374 let elemPtr = INDEX_OF(arrStart arr, i, a)
423375 if i >= 0 && INDEX_VALID(elemPtr, arrEnd arr, a)
424− then Just <$> peekByteIndex elemPtr (arrContents arr)
376+ then Just <$> peekAt elemPtr (arrContents arr)
425377 else return Nothing
426378
427379-- | Given a stream of array indices, read the elements on those indices from
@@ -433,19 +385,25 @@ getIndex i arr =
433385-- @
434386-- read =
435387-- let u = lmap (\arr -> (0, length arr - 1)) Unfold.enumerateFromTo
436−-- in Unfold.lmap f (getIndices arr)
388+-- in Unfold.lmap f (indexReader arr)
437389--
438390-- readRev =
439391-- let i = length arr - 1
440−-- in Unfold.lmap f (getIndicesFromThenTo i (i - 1) 0)
392+-- in Unfold.lmap f (indexReaderFromThenTo i (i - 1) 0)
441393-- @
442394--
443395-- /Pre-release/
396+{-# INLINE indexReader #-}
397+indexReader :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
398+indexReader m =
399+ let unf = MA.indexReaderWith (return . unsafeInlineIO) m
400+ in Unfold.lmap unsafeThaw unf
401+
402+-- XXX DO NOT REMOVE, change the signature to use Stream instead of unfold
403+{-# DEPRECATED getIndices "Please use getIndices instead." #-}
444404{-# INLINE getIndices #-}
445405getIndices :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
446−getIndices m =
447− let unf = MA.getIndicesD (return . unsafeInlineIO) m
448− in Unfold.lmap A.unsafeThaw unf
406+getIndices = indexReader
449407
450408-- | Unfolds @(from, then, to, array)@ generating a finite stream whose first
451409-- element is the array value from the index @from@ and the successive elements
@@ -461,9 +419,9 @@ getIndices m =
461419-- @
462420--
463421-- /Unimplemented/
464−{-# INLINE getIndicesFromThenTo #-}
465−getIndicesFromThenTo :: Unfold m (Int, Int, Int, Array a) a
466−getIndicesFromThenTo = undefined
422+{-# INLINE indexReaderFromThenTo #-}
423+indexReaderFromThenTo :: Unfold m (Int, Int, Int, Array a) a
424+indexReaderFromThenTo = undefined
467425
468426-------------------------------------------------------------------------------
469427-- Transform via stream operations
@@ -480,7 +438,7 @@ getIndicesFromThenTo = undefined
480438{-# INLINE runPipe #-}
481439runPipe :: (MonadIO m, Unbox a, Unbox b)
482440 => Pipe m a b -> Array a -> m (Array b)
483−runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (A.read arr)
441+runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (read arr)
484442-}
485443
486444-- XXX For transformations that cannot change the number of elements e.g. "map"
@@ -494,7 +452,7 @@ runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (A.read arr)
494452streamTransform :: forall m a b. (MonadIO m, Unbox a, Unbox b)
495453 => (Stream m a -> Stream m b) -> Array a -> m (Array b)
496454streamTransform f arr =
497− Stream.fold (A.writeWith (length arr)) $ f (A.read arr)
455+ Stream.fold (createWith (length arr)) $ f (read arr)
498456
499457-------------------------------------------------------------------------------
500458-- Casts
@@ -528,7 +486,7 @@ asBytes = castUnsafe
528486--
529487cast :: forall a b. (Unbox b) => Array a -> Maybe (Array b)
530488cast arr =
531− let len = A.byteLength arr
489+ let len = byteLength arr
532490 r = len `mod` SIZE_OF(b)
533491 in if r /= 0
534492 then Nothing
@@ -546,9 +504,9 @@ cast arr =
546504--
547505asCStringUnsafe :: Array a -> (CString -> IO b) -> IO b
548506asCStringUnsafe arr act = do
549− let arr1 = asBytes arr <> A.fromList [0]
550− -- asPtrUnsafe makes sure the array is pinned
551− asPtrUnsafe arr1 $ \ptr -> act (castPtr ptr)
507+ let arr1 = asBytes arr <> fromList [0]
508+ -- unsafePinnedAsPtr makes sure the array is pinned
509+ unsafePinnedAsPtr arr1 $ \ptr -> act (castPtr ptr)
552510
553511-------------------------------------------------------------------------------
554512-- Folds
@@ -561,11 +519,500 @@ asCStringUnsafe arr act = do
561519-- /Pre-release/
562520{-# INLINE fold #-}
563521fold :: forall m a b. (Monad m, Unbox a) => Fold m a b -> Array a -> m b
564−fold f arr = Stream.fold f (A.read arr)
522+fold f arr = Stream.fold f (read arr)
565523
566524-- | Fold an array using a stream fold operation.
567525--
568526-- /Pre-release/
569527{-# INLINE streamFold #-}
570528streamFold :: (Monad m, Unbox a) => (Stream m a -> m b) -> Array a -> m b
571−streamFold f arr = f (A.read arr)
529+streamFold f arr = f (read arr)
530+
531+--------------------------------------------------------------------------------
532+-- Serialization
533+--------------------------------------------------------------------------------
534+
535+{-# INLINE encodeAs #-}
536+encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
537+encodeAs ps a =
538+ unsafeInlineIO $ do
539+ let len = Serialize.addSizeTo 0 a
540+ mbarr <- MBA.newBytesAs ps len
541+ off <- Serialize.serializeAt 0 mbarr a
542+ assertM(len == off)
543+ pure $ Array mbarr 0 off
544+
545+-- |
546+-- Properties:
547+-- 1. Identity: @deserialize . serialize == id@
548+-- 2. Encoded equivalence: @serialize a == serialize a@
549+{-# INLINE serialize #-}
550+serialize :: Serialize a => a -> Array Word8
551+serialize = encodeAs Unpinned
552+
553+-- | Serialize a Haskell type to a pinned byte array. The array is allocated
554+-- using pinned memory so that it can be used directly in OS APIs for writing
555+-- to file or sending over the network.
556+--
557+-- Properties:
558+-- 1. Identity: @deserialize . pinnedSerialize == id@
559+-- 2. Encoded equivalence: @pinnedSerialize a == pinnedSerialize a@
560+{-# INLINE pinnedSerialize #-}
561+pinnedSerialize :: Serialize a => a -> Array Word8
562+pinnedSerialize = encodeAs Pinned
563+
564+-- | Decode a Haskell type from a byte array containing its serialized
565+-- representation.
566+{-# INLINE deserialize #-}
567+deserialize :: Serialize a => Array Word8 -> a
568+deserialize arr@(Array {..}) = unsafeInlineIO $ do
569+ let lenArr = length arr
570+ (off, val) <-
571+ Serialize.deserializeAt arrStart arrContents (arrStart + lenArr)
572+ assertM(off == arrStart + lenArr)
573+ pure val
574+
575+-------------------------------------------------------------------------------
576+-- Streams of Arrays
577+-------------------------------------------------------------------------------
578+
579+-- TODO: efficiently compare two streams of arrays. Two streams can have chunks
580+-- of different sizes, we can handle that in the stream comparison abstraction.
581+-- This could be useful e.g. to fast compare whether two files differ.
582+
583+-- | Insert the given element between arrays and flatten.
584+--
585+-- >>> interpose x = Stream.interpose x Array.reader
586+--
587+{-# INLINE interpose #-}
588+interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
589+interpose x = D.interpose x reader
590+
591+data FlattenState s =
592+ OuterLoop s
593+ | InnerLoop s !MutByteArray !Int !Int
594+
595+-- | Insert the given element after each array and flatten. This is similar to
596+-- unlines.
597+--
598+-- >>> interposeSuffix x = Stream.interposeSuffix x Array.reader
599+--
600+{-# INLINE_NORMAL interposeSuffix #-}
601+interposeSuffix :: forall m a. (Monad m, Unbox a)
602+ => a -> Stream m (Array a) -> Stream m a
603+-- This does not require MonadIO constraint.
604+-- interposeSuffix x = D.interposeSuffix x reader
605+interposeSuffix sep (D.Stream step state) = D.Stream step' (OuterLoop state)
606+
607+ where
608+
609+ {-# INLINE_LATE step' #-}
610+ step' gst (OuterLoop st) = do
611+ r <- step (adaptState gst) st
612+ return $ case r of
613+ D.Yield Array{..} s ->
614+ D.Skip (InnerLoop s arrContents arrStart arrEnd)
615+ D.Skip s -> D.Skip (OuterLoop s)
616+ D.Stop -> D.Stop
617+
618+ step' _ (InnerLoop st _ p end) | p == end =
619+ return $ D.Yield sep $ OuterLoop st
620+
621+ step' _ (InnerLoop st contents p end) = do
622+ x <- pure $ unsafeInlineIO $ peekAt p contents
623+ return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
624+
625+-- | Insert the given array after each array and flatten.
626+--
627+-- >>> intercalateSuffix = Stream.intercalateSuffix Array.reader
628+--
629+{-# INLINE intercalateSuffix #-}
630+intercalateSuffix :: (Monad m, Unbox a)
631+ => Array a -> Stream m (Array a) -> Stream m a
632+intercalateSuffix = D.intercalateSuffix reader
633+
634+-- | @compactLE n@ coalesces adjacent arrays in the input stream
635+-- only if the combined size would be less than or equal to n.
636+--
637+-- Generates unpinned arrays irrespective of the pinning status of input
638+-- arrays.
639+{-# INLINE_NORMAL compactLE #-}
640+compactLE :: (MonadIO m, Unbox a)
641+ => Int -> Stream m (Array a) -> Stream m (Array a)
642+compactLE n stream =
643+ D.map unsafeFreeze $ MA.compactLE n $ D.map unsafeThaw stream
644+
645+-- | Pinned version of 'compactLE'.
646+{-# INLINE_NORMAL pinnedCompactLE #-}
647+pinnedCompactLE :: (MonadIO m, Unbox a)
648+ => Int -> Stream m (Array a) -> Stream m (Array a)
649+pinnedCompactLE n stream =
650+ D.map unsafeFreeze $ MA.pinnedCompactLE n $ D.map unsafeThaw stream
651+
652+-- | Split a stream of arrays on a given separator byte, dropping the separator
653+-- and coalescing all the arrays between two separators into a single array.
654+--
655+{-# INLINE compactOnByte #-}
656+compactOnByte
657+ :: (MonadIO m)
658+ => Word8
659+ -> Stream m (Array Word8)
660+ -> Stream m (Array Word8)
661+compactOnByte byte =
662+ fmap unsafeFreeze . MA.compactOnByte byte . fmap unsafeThaw
663+
664+-- | Like 'compactOnByte' considers the separator in suffix position instead of
665+-- infix position.
666+{-# INLINE compactOnByteSuffix #-}
667+compactOnByteSuffix
668+ :: (MonadIO m)
669+ => Word8
670+ -> Stream m (Array Word8)
671+ -> Stream m (Array Word8)
672+compactOnByteSuffix byte =
673+ fmap unsafeFreeze . MA.compactOnByteSuffix byte . fmap unsafeThaw
674+
675+-------------------------------------------------------------------------------
676+-- Folding Streams of Arrays
677+-------------------------------------------------------------------------------
678+
679+-- XXX This should not be used for breaking a stream as the D.cons used in
680+-- reconstructing the stream could be very bad for performance. This can only
681+-- be useful in folding without breaking.
682+{-# INLINE_NORMAL foldBreakChunks #-}
683+foldBreakChunks :: forall m a b. (MonadIO m, Unbox a) =>
684+ Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
685+foldBreakChunks (Fold fstep initial _ final) stream@(Stream step state) = do
686+ res <- initial
687+ case res of
688+ Fold.Partial fs -> go SPEC state fs
689+ Fold.Done fb -> return $! (fb, stream)
690+
691+ where
692+
693+ {-# INLINE go #-}
694+ go !_ st !fs = do
695+ r <- step defState st
696+ case r of
697+ Stream.Yield (Array contents start end) s ->
698+ let fp = Tuple' end contents
699+ in goArray SPEC s fp start fs
700+ Stream.Skip s -> go SPEC s fs
701+ Stream.Stop -> do
702+ b <- final fs
703+ return (b, D.nil)
704+
705+ goArray !_ s (Tuple' end _) !cur !fs
706+ | cur == end = do
707+ go SPEC s fs
708+ goArray !_ st fp@(Tuple' end contents) !cur !fs = do
709+ x <- liftIO $ peekAt cur contents
710+ res <- fstep fs x
711+ let next = INDEX_NEXT(cur,a)
712+ case res of
713+ Fold.Done b -> do
714+ let arr = Array contents next end
715+ return $! (b, D.cons arr (D.Stream step st))
716+ Fold.Partial fs1 -> goArray SPEC st fp next fs1
717+
718+-- This may be more robust wrt fusion compared to unfoldMany?
719+
720+-- | Fold a stream of arrays using a 'Fold'. This is equivalent to the
721+-- following:
722+--
723+-- >>> foldChunks f = Stream.fold f . Stream.unfoldMany Array.reader
724+--
725+foldChunks :: (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m b
726+foldChunks f s = fmap fst (foldBreakChunks f s)
727+-- foldStream f = Stream.fold f . Stream.unfoldMany reader
728+
729+-- | Fold a stream of arrays using a 'Fold' and return the remaining stream.
730+--
731+-- The following alternative to this function allows composing the fold using
732+-- the parser Monad:
733+--
734+-- @
735+-- foldBreakStreamK f s =
736+-- fmap (first (fromRight undefined))
737+-- $ StreamK.parseBreakChunks (ParserK.adaptC (Parser.fromFold f)) s
738+-- @
739+--
740+-- We can compare perf and remove this one or define it in terms of that.
741+--
742+foldBreakChunksK :: forall m a b. (MonadIO m, Unbox a) =>
743+ Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
744+{-
745+foldBreakChunksK f s =
746+ fmap (first (fromRight undefined))
747+ $ StreamK.parseBreakChunks (ParserK.adaptC (Parser.fromFold f)) s
748+-}
749+foldBreakChunksK (Fold fstep initial _ final) stream = do
750+ res <- initial
751+ case res of
752+ Fold.Partial fs -> go fs stream
753+ Fold.Done fb -> return (fb, stream)
754+
755+ where
756+
757+ {-# INLINE go #-}
758+ go !fs st = do
759+ let stop = (, StreamK.nil) <$> final fs
760+ single a = yieldk a StreamK.nil
761+ yieldk (Array contents start end) r =
762+ let fp = Tuple' end contents
763+ in goArray fs r fp start
764+ in StreamK.foldStream defState yieldk single stop st
765+
766+ goArray !fs st (Tuple' end _) !cur
767+ | cur == end = do
768+ go fs st
769+ goArray !fs st fp@(Tuple' end contents) !cur = do
770+ x <- liftIO $ peekAt cur contents
771+ res <- fstep fs x
772+ let next = INDEX_NEXT(cur,a)
773+ case res of
774+ Fold.Done b -> do
775+ let arr = Array contents next end
776+ return $! (b, StreamK.cons arr st)
777+ Fold.Partial fs1 -> goArray fs1 st fp next
778+
779+{-
780+-- This can be generalized to any type provided it can be unfolded to a stream
781+-- and it can be combined using a semigroup operation.
782+--
783+{-# INLINE_NORMAL parseBreakD #-}
784+parseBreakD ::
785+ forall m a b. (MonadIO m, MonadThrow m, Unbox a)
786+ => PRD.Parser a m b
787+ -> D.Stream m (Array.Array a)
788+ -> m (b, D.Stream m (Array.Array a))
789+parseBreakD
790+ (PRD.Parser pstep initial extract) stream@(D.Stream step state) = do
791+
792+ res <- initial
793+ case res of
794+ PRD.IPartial s -> go SPEC state (List []) s
795+ PRD.IDone b -> return (b, stream)
796+ PRD.IError err -> throwM $ ParseError err
797+
798+ where
799+
800+ -- "backBuf" contains last few items in the stream that we may have to
801+ -- backtrack to.
802+ --
803+ -- XXX currently we are using a dumb list based approach for backtracking
804+ -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
805+ -- That will allow us more efficient random back and forth movement.
806+ go !_ st backBuf !pst = do
807+ r <- step defState st
808+ case r of
809+ D.Yield (Array contents start end) s ->
810+ gobuf SPEC s backBuf
811+ (Tuple' end contents) start pst
812+ D.Skip s -> go SPEC s backBuf pst
813+ D.Stop -> do
814+ b <- extract pst
815+ return (b, D.nil)
816+
817+ -- Use strictness on "cur" to keep it unboxed
818+ gobuf !_ s backBuf (Tuple' end _) !cur !pst
819+ | cur == end = do
820+ go SPEC s backBuf pst
821+ gobuf !_ s backBuf fp@(Tuple' end contents) !cur !pst = do
822+ x <- liftIO $ peekByteIndex contents cur
823+ pRes <- pstep pst x
824+ let next = INDEX_NEXT(cur,a)
825+ case pRes of
826+ PR.Partial 0 pst1 ->
827+ gobuf SPEC s (List []) fp next pst1
828+ PR.Partial n pst1 -> do
829+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
830+ let src0 = Prelude.take n (x:getList backBuf)
831+ arr0 = A.fromListN n (Prelude.reverse src0)
832+ arr1 = Array contents next end
833+ src = arr0 <> arr1
834+ let !(Array cont1 start end1) = src
835+ fp1 = Tuple' end1 cont1
836+ gobuf SPEC s (List []) fp1 start pst1
837+ PR.Continue 0 pst1 ->
838+ gobuf SPEC s (List (x:getList backBuf)) fp next pst1
839+ PR.Continue n pst1 -> do
840+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
841+ let (src0, buf1) = splitAt n (x:getList backBuf)
842+ arr0 = A.fromListN n (Prelude.reverse src0)
843+ arr1 = Array contents next end
844+ src = arr0 <> arr1
845+ let !(Array cont1 start end1) = src
846+ fp1 = Tuple' end1 cont1
847+ gobuf SPEC s (List buf1) fp1 start pst1
848+ PR.Done 0 b -> do
849+ let arr = Array contents next end
850+ return (b, D.cons arr (D.Stream step s))
851+ PR.Done n b -> do
852+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
853+ let src0 = Prelude.take n (x:getList backBuf)
854+ -- XXX create the array in reverse instead
855+ arr0 = A.fromListN n (Prelude.reverse src0)
856+ arr1 = Array contents next end
857+ -- XXX Use StreamK to avoid adding arbitrary layers of
858+ -- constructors every time.
859+ str = D.cons arr0 (D.cons arr1 (D.Stream step s))
860+ return (b, str)
861+ PR.Error err -> throwM $ ParseError err
862+-}
863+
864+-- | Parse an array stream using the supplied 'Parser'. Returns the parse
865+-- result and the unconsumed stream. Throws 'ParseError' if the parse fails.
866+--
867+-- The following alternative to this function allows composing the parser using
868+-- the parser Monad:
869+--
870+-- >>> parseBreakStreamK p = StreamK.parseBreakChunks (ParserK.adaptC p)
871+--
872+-- We can compare perf and remove this one or define it in terms of that.
873+--
874+-- /Internal/
875+--
876+{-# INLINE_NORMAL parseBreakChunksK #-}
877+parseBreakChunksK ::
878+ forall m a b. (MonadIO m, Unbox a)
879+ => Parser a m b
880+ -> StreamK m (Array a)
881+ -> m (Either ParseError b, StreamK m (Array a))
882+-- parseBreakStreamK p = StreamK.parseBreakChunks (ParserK.adaptC p)
883+parseBreakChunksK (Parser pstep initial extract) stream = do
884+ res <- initial
885+ case res of
886+ IPartial s -> go s stream []
887+ IDone b -> return (Right b, stream)
888+ IError err -> return (Left (ParseError err), stream)
889+
890+ where
891+
892+ -- "backBuf" contains last few items in the stream that we may have to
893+ -- backtrack to.
894+ --
895+ -- XXX currently we are using a dumb list based approach for backtracking
896+ -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
897+ -- That will allow us more efficient random back and forth movement.
898+ go !pst st backBuf = do
899+ let stop = goStop pst backBuf -- (, K.nil) <$> extract pst
900+ single a = yieldk a StreamK.nil
901+ yieldk arr r = goArray pst backBuf r arr
902+ in StreamK.foldStream defState yieldk single stop st
903+
904+ -- Use strictness on "cur" to keep it unboxed
905+ goArray !pst backBuf st (Array _ cur end) | cur == end = go pst st backBuf
906+ goArray !pst backBuf st (Array contents cur end) = do
907+ x <- liftIO $ peekAt cur contents
908+ pRes <- pstep pst x
909+ let next = INDEX_NEXT(cur,a)
910+ case pRes of
911+ Parser.Partial 0 s ->
912+ goArray s [] st (Array contents next end)
913+ Parser.Partial n s -> do
914+ assert (n <= Prelude.length (x:backBuf)) (return ())
915+ let src0 = Prelude.take n (x:backBuf)
916+ arr0 = fromListN n (Prelude.reverse src0)
917+ arr1 = Array contents next end
918+ src = arr0 <> arr1
919+ goArray s [] st src
920+ Parser.Continue 0 s ->
921+ goArray s (x:backBuf) st (Array contents next end)
922+ Parser.Continue n s -> do
923+ assert (n <= Prelude.length (x:backBuf)) (return ())
924+ let (src0, buf1) = Prelude.splitAt n (x:backBuf)
925+ arr0 = fromListN n (Prelude.reverse src0)
926+ arr1 = Array contents next end
927+ src = arr0 <> arr1
928+ goArray s buf1 st src
929+ Parser.Done 0 b -> do
930+ let arr = Array contents next end
931+ return (Right b, StreamK.cons arr st)
932+ Parser.Done n b -> do
933+ assert (n <= Prelude.length (x:backBuf)) (return ())
934+ let src0 = Prelude.take n (x:backBuf)
935+ -- XXX Use fromListRevN once implemented
936+ -- arr0 = A.fromListRevN n src0
937+ arr0 = fromListN n (Prelude.reverse src0)
938+ arr1 = Array contents next end
939+ str = StreamK.cons arr0 (StreamK.cons arr1 st)
940+ return (Right b, str)
941+ Parser.Error err -> do
942+ let n = Prelude.length backBuf
943+ arr0 = fromListN n (Prelude.reverse backBuf)
944+ arr1 = Array contents cur end
945+ str = StreamK.cons arr0 (StreamK.cons arr1 stream)
946+ return (Left (ParseError err), str)
947+
948+ -- This is a simplified goArray
949+ goExtract !pst backBuf (Array _ cur end)
950+ | cur == end = goStop pst backBuf
951+ goExtract !pst backBuf (Array contents cur end) = do
952+ x <- liftIO $ peekAt cur contents
953+ pRes <- pstep pst x
954+ let next = INDEX_NEXT(cur,a)
955+ case pRes of
956+ Parser.Partial 0 s ->
957+ goExtract s [] (Array contents next end)
958+ Parser.Partial n s -> do
959+ assert (n <= Prelude.length (x:backBuf)) (return ())
960+ let src0 = Prelude.take n (x:backBuf)
961+ arr0 = fromListN n (Prelude.reverse src0)
962+ arr1 = Array contents next end
963+ src = arr0 <> arr1
964+ goExtract s [] src
965+ Parser.Continue 0 s ->
966+ goExtract s backBuf (Array contents next end)
967+ Parser.Continue n s -> do
968+ assert (n <= Prelude.length (x:backBuf)) (return ())
969+ let (src0, buf1) = Prelude.splitAt n (x:backBuf)
970+ arr0 = fromListN n (Prelude.reverse src0)
971+ arr1 = Array contents next end
972+ src = arr0 <> arr1
973+ goExtract s buf1 src
974+ Parser.Done 0 b -> do
975+ let arr = Array contents next end
976+ return (Right b, StreamK.fromPure arr)
977+ Parser.Done n b -> do
978+ assert (n <= Prelude.length backBuf) (return ())
979+ let src0 = Prelude.take n backBuf
980+ -- XXX Use fromListRevN once implemented
981+ -- arr0 = A.fromListRevN n src0
982+ arr0 = fromListN n (Prelude.reverse src0)
983+ arr1 = Array contents next end
984+ str = StreamK.cons arr0 (StreamK.fromPure arr1)
985+ return (Right b, str)
986+ Parser.Error err -> do
987+ let n = Prelude.length backBuf
988+ arr0 = fromListN n (Prelude.reverse backBuf)
989+ arr1 = Array contents cur end
990+ str = StreamK.cons arr0 (StreamK.cons arr1 stream)
991+ return (Left (ParseError err), str)
992+
993+ -- This is a simplified goExtract
994+ {-# INLINE goStop #-}
995+ goStop !pst backBuf = do
996+ pRes <- extract pst
997+ case pRes of
998+ Parser.Partial _ _ -> error "Bug: parseBreak: Partial in extract"
999+ Parser.Continue 0 s ->
1000+ goStop s backBuf
1001+ Parser.Continue n s -> do
1002+ assert (n <= Prelude.length backBuf) (return ())
1003+ let (src0, buf1) = Prelude.splitAt n backBuf
1004+ arr = fromListN n (Prelude.reverse src0)
1005+ goExtract s buf1 arr
1006+ Parser.Done 0 b ->
1007+ return (Right b, StreamK.nil)
1008+ Parser.Done n b -> do
1009+ assert (n <= Prelude.length backBuf) (return ())
1010+ let src0 = Prelude.take n backBuf
1011+ -- XXX Use fromListRevN once implemented
1012+ -- arr0 = A.fromListRevN n src0
1013+ arr0 = fromListN n (Prelude.reverse src0)
1014+ return (Right b, StreamK.fromPure arr0)
1015+ Parser.Error err -> do
1016+ let n = Prelude.length backBuf
1017+ arr0 = fromListN n (Prelude.reverse backBuf)
1018+ return (Left (ParseError err), StreamK.fromPure arr0)
core/src/Streamly/Internal/Data/Array/Generic.hs+49−11
@@ -11,8 +11,8 @@ module Streamly.Internal.Data.Array.Generic
1111
1212 -- * Construction
1313 , nil
14− , writeN
15− , write
14+ , createOf
15+ , create
1616 , writeWith
1717 , writeLastN
1818
@@ -24,6 +24,8 @@ module Streamly.Internal.Data.Array.Generic
2424 , fromListN
2525 , fromList
2626
27+ , chunksOf
28+
2729 -- * Elimination
2830 , length
2931 , reader
@@ -42,6 +44,10 @@ module Streamly.Internal.Data.Array.Generic
4244 , getIndex
4345 , getSliceUnsafe
4446 , strip
47+
48+ -- * Deprecated
49+ , writeN
50+ , write
4551 )
4652where
4753
@@ -57,7 +63,7 @@ import GHC.IO (unsafePerformIO)
5763import Text.Read (readPrec)
5864
5965import Streamly.Internal.Data.Fold.Type (Fold(..))
60−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
66+import Streamly.Internal.Data.Stream.Type (Stream)
6167import Streamly.Internal.Data.Unfold.Type (Unfold(..))
6268import Streamly.Internal.System.IO (unsafeInlineIO)
6369
@@ -65,9 +71,9 @@ import qualified Streamly.Internal.Data.MutArray.Generic as MArray
6571import qualified Streamly.Internal.Data.Fold.Type as FL
6672import qualified Streamly.Internal.Data.Producer.Type as Producer
6773import qualified Streamly.Internal.Data.Producer as Producer
68−import qualified Streamly.Internal.Data.Ring as RB
69−import qualified Streamly.Internal.Data.Stream.StreamD.Type as D
70−import qualified Streamly.Internal.Data.Stream.StreamD.Generate as D
74+import qualified Streamly.Internal.Data.Ring.Generic as RB
75+import qualified Streamly.Internal.Data.Stream.Type as D
76+import qualified Streamly.Internal.Data.Stream.Generate as D
7177import qualified Text.ParserCombinators.ReadPrec as ReadPrec
7278
7379import Prelude hiding (Foldable(..), read)
@@ -102,21 +108,33 @@ nil = unsafePerformIO $ unsafeFreeze <$> MArray.nil
102108-- Construction - Folds
103109-------------------------------------------------------------------------------
104110
105−{-# INLINE_NORMAL writeN #-}
111+{-# INLINE_NORMAL createOf #-}
112+createOf :: MonadIO m => Int -> Fold m a (Array a)
113+createOf = fmap unsafeFreeze <$> MArray.createOf
114+
115+-- XXX Deprecate in major
116+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
117+{-# INLINE writeN #-}
106118writeN :: MonadIO m => Int -> Fold m a (Array a)
107−writeN = fmap unsafeFreeze <$> MArray.writeN
119+writeN = createOf
108120
109121{-# INLINE_NORMAL writeWith #-}
110122writeWith :: MonadIO m => Int -> Fold m a (Array a)
111−writeWith elemCount = unsafeFreeze <$> MArray.writeWith elemCount
123+writeWith elemCount = unsafeFreeze <$> MArray.createWith elemCount
112124
113125-- | Fold the whole input to a single array.
114126--
115127-- /Caution! Do not use this on infinite streams./
116128--
129+{-# INLINE create #-}
130+create :: MonadIO m => Fold m a (Array a)
131+create = fmap unsafeFreeze MArray.create
132+
133+-- XXX Deprecate in major
134+-- {-# DEPRECATED write "Please use create instead." #-}
117135{-# INLINE write #-}
118136write :: MonadIO m => Fold m a (Array a)
119−write = fmap unsafeFreeze MArray.write
137+write = create
120138
121139fromPureStream :: Stream Identity a -> Array a
122140fromPureStream x =
@@ -127,6 +145,15 @@ fromPureStream x =
127145fromByteStr# :: Addr# -> Array Word8
128146fromByteStr# addr = fromPureStream (D.fromByteStr# addr)
129147
148+-------------------------------------------------------------------------------
149+-- Stream Ops
150+-------------------------------------------------------------------------------
151+
152+{-# INLINE_NORMAL chunksOf #-}
153+chunksOf :: forall m a. MonadIO m
154+ => Int -> Stream m a -> Stream m (Array a)
155+chunksOf n strm = fmap unsafeFreeze $ MArray.chunksOf n strm
156+
130157-------------------------------------------------------------------------------
131158-- Construction - from streams
132159-------------------------------------------------------------------------------
@@ -231,6 +258,17 @@ getIndex i arr@Array {..} =
231258 then Just $ getIndexUnsafe i arr
232259 else Nothing
233260
261+-- >>> import qualified Streamly.Data.Stream as Stream
262+-- >>> import qualified Streamly.Data.Fold as Fold
263+-- >>> import qualified Streamly.Internal.Data.Array.Generic as Array
264+-- >>> import Data.Function ((&))
265+-- >>> :{
266+-- Stream.fromList [1,2,3,4,5::Int]
267+-- & Stream.scan (Array.writeLastN 2)
268+-- & Stream.fold Fold.toList
269+-- :}
270+-- [fromList [],fromList [1],fromList [1,2],fromList [2,3],fromList [3,4],fromList [4,5]]
271+--
234272{-# INLINE writeLastN #-}
235273writeLastN :: MonadIO m => Int -> Fold m a (Array a)
236274writeLastN n = FL.rmapM f (RB.writeLastN n)
@@ -238,7 +276,7 @@ writeLastN n = FL.rmapM f (RB.writeLastN n)
238276 where
239277
240278 f rb = do
241− arr <- RB.toMutArray 0 n rb
279+ arr <- RB.copyToMutArray 0 n rb
242280 return $ unsafeFreeze arr
243281
244282{-# INLINE getSliceUnsafe #-}
core/src/Streamly/Internal/Data/Stream/Chunked.hs → core/src/Streamly/Internal/Data/Array/Stream.hsrenamed+74−587
@@ -1,5 +1,6 @@
1+{-# OPTIONS_GHC -Wno-deprecations #-}
12-- |
2−-- Module : Streamly.Internal.Data.Stream.Chunked
3+-- Module : Streamly.Internal.Data.Array.Stream
34-- Copyright : (c) 2019 Composewell Technologies
45-- License : BSD3-3-Clause
56-- Maintainer : streamly@composewell.com
@@ -7,17 +8,25 @@
78--
89-- Combinators to efficiently manipulate streams of immutable arrays.
910--
10−module Streamly.Internal.Data.Stream.Chunked
11+-- We can either push these in the MutArray module with a "chunks" prefix or
12+-- keep this as a separate module and release it.
13+--
14+module Streamly.Internal.Data.Array.Stream
15+{-# DEPRECATED "Please use \"Streamly.Internal.Data.Array\" instead." #-}
1116 (
1217 -- * Creation
13− chunksOf
18+ Array.chunksOf
19+ , Array.pinnedChunksOf
20+ , Array.bufferChunks
1421
1522 -- * Flattening to elements
16− , concat
17− , concatRev
18− , interpose
19− , interposeSuffix
20− , intercalateSuffix
23+ , Array.concat
24+ , Array.flattenArrays
25+ , Array.concatRev
26+ , Array.flattenArraysRev
27+ , Array.interpose
28+ , Array.interposeSuffix
29+ , Array.intercalateSuffix
2130 , unlines
2231
2332 -- * Elimination
@@ -26,24 +35,27 @@ module Streamly.Internal.Data.Stream.Chunked
2635 -- can flatten the stream to byte stream and use that. But if we want the
2736 -- remaining stream to be a chunk stream then this could be handy. But it
2837 -- could also be implemented using parseBreak.
29− , foldBreak -- StreamK.foldBreakChunks
38+ , foldBreak
3039 , foldBreakD
31− -- The byte level parseBreak cannot work efficiently. Because the stream
32− -- will have to be a StreamK for backtracking, StreamK at byte level would
33− -- not be efficient.
34− , parseBreak -- StreamK.parseBreakChunks
40+ -- This is chunked parseBreak. A byte level parseBreak cannot work
41+ -- efficiently. Because the stream will have to be a StreamK for
42+ -- backtracking, StreamK at byte level would not be efficient.
43+ -- parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
44+ , parseBreak
3545 -- , parseBreakD
3646 -- , foldManyChunks
3747 -- , parseManyChunks
48+ , K.parseBreakChunks
49+ , K.parseChunks
3850
3951 -- ** Array Folds
40− -- XXX Use Parser.Chunked instead, need only chunkedParseBreak,
52+ -- XXX Use parseBreakChunks/parseChunks instead
4153 -- foldBreak can be implemented using parseBreak. Use StreamK.
4254 , runArrayFold
4355 , runArrayFoldBreak
4456 -- , parseArr
45− , runArrayParserDBreak -- StreamK.chunkedParseBreak
46− , runArrayFoldMany -- StreamK.chunkedParseMany
57+ , runArrayParserDBreak -- StreamK.parseBreakChunks
58+ , runArrayFoldMany
4759
4860 , toArray
4961
@@ -67,6 +79,8 @@ where
6779import Data.Bifunctor (second)
6880import Control.Exception (assert)
6981import Control.Monad.IO.Class (MonadIO(..))
82+-- import Data.Bifunctor (first)
83+-- import Data.Either (fromRight)
7084import Data.Proxy (Proxy(..))
7185import Data.Word (Word8)
7286import Streamly.Internal.Data.Unbox (Unbox(..))
@@ -82,143 +96,32 @@ import Streamly.Internal.Data.Parser (ParseError(..))
8296import Streamly.Internal.Data.Stream (Stream)
8397import Streamly.Internal.Data.StreamK (StreamK, fromStream, toStream)
8498import Streamly.Internal.Data.SVar.Type (adaptState, defState)
85−import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
8699
87−import qualified Streamly.Data.Fold as FL
88100import qualified Streamly.Internal.Data.Array as A
89101import qualified Streamly.Internal.Data.Array as Array
90−import qualified Streamly.Internal.Data.Array.Type as A
91−import qualified Streamly.Internal.Data.MutArray.Type as MA
92−import qualified Streamly.Internal.Data.Stream.MutChunked as AS
93−import qualified Streamly.Internal.Data.Fold.Type as FL (Fold(..), Step(..))
94102import qualified Streamly.Internal.Data.Parser as PR
95103import qualified Streamly.Internal.Data.Parser as PRD
96104 (Parser(..), Initial(..))
105+-- import qualified Streamly.Internal.Data.ParserK as ParserK
97106import qualified Streamly.Internal.Data.Stream as D
98107import qualified Streamly.Internal.Data.StreamK as K
99108
100109-- XXX Since these are immutable arrays MonadIO constraint can be removed from
101110-- most places.
102111
103−-------------------------------------------------------------------------------
104−-- Generation
105−-------------------------------------------------------------------------------
106−
107−-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
108−-- @n@ elements each.
109−--
110−-- > chunksOf n = Stream.groupsOf n (Array.writeN n)
111−--
112−-- /Pre-release/
113−{-# INLINE chunksOf #-}
114−chunksOf :: (MonadIO m, Unbox a)
115− => Int -> Stream m a -> Stream m (Array a)
116−chunksOf = A.chunksOf
117−
118−-------------------------------------------------------------------------------
119−-- Append
120−-------------------------------------------------------------------------------
121−
122−-- XXX efficiently compare two streams of arrays. Two streams can have chunks
123−-- of different sizes, we can handle that in the stream comparison abstraction.
124−-- This could be useful e.g. to fast compare whether two files differ.
125−
126−-- | Convert a stream of arrays into a stream of their elements.
127−--
128−-- Same as the following:
129−--
130−-- > concat = Stream.unfoldMany Array.read
131−--
132−-- @since 0.7.0
133−{-# INLINE concat #-}
134−concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
135−-- concat m = fromStreamD $ A.flattenArrays (toStreamD m)
136−-- concat m = fromStreamD $ D.concatMap A.toStreamD (toStreamD m)
137−concat = D.unfoldMany A.reader
138−
139−-- | Convert a stream of arrays into a stream of their elements reversing the
140−-- contents of each array before flattening.
141−--
142−-- > concatRev = Stream.unfoldMany Array.readerRev
143−--
144−-- @since 0.7.0
145−{-# INLINE concatRev #-}
146−concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
147−-- concatRev m = fromStreamD $ A.flattenArraysRev (toStreamD m)
148−concatRev = D.unfoldMany A.readerRev
149−
150112-------------------------------------------------------------------------------
151113-- Intersperse and append
152114-------------------------------------------------------------------------------
153115
154−-- | Flatten a stream of arrays after inserting the given element between
155−-- arrays.
156−--
157−-- /Pre-release/
158−{-# INLINE interpose #-}
159−interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
160−interpose x = D.interpose x A.reader
161−
162−{-# INLINE intercalateSuffix #-}
163−intercalateSuffix :: (Monad m, Unbox a)
164− => Array a -> Stream m (Array a) -> Stream m a
165−intercalateSuffix = D.intercalateSuffix A.reader
166−
167−-- | Flatten a stream of arrays appending the given element after each
168−-- array.
169−--
170−-- @since 0.7.0
171−{-# INLINE interposeSuffix #-}
172−interposeSuffix :: (Monad m, Unbox a)
173− => a -> Stream m (Array a) -> Stream m a
174−-- interposeSuffix x = fromStreamD . A.unlines x . toStreamD
175−interposeSuffix x = D.interposeSuffix x A.reader
176−
177−data FlattenState s =
178− OuterLoop s
179− | InnerLoop s !MA.MutableByteArray !Int !Int
180−
181−-- XXX This is a special case of interposeSuffix, can be removed.
182−-- XXX Remove monadIO constraint
183116{-# INLINE_NORMAL unlines #-}
184117unlines :: forall m a. (MonadIO m, Unbox a)
185118 => a -> D.Stream m (Array a) -> D.Stream m a
186−unlines sep (D.Stream step state) = D.Stream step' (OuterLoop state)
187− where
188− {-# INLINE_LATE step' #-}
189− step' gst (OuterLoop st) = do
190− r <- step (adaptState gst) st
191− return $ case r of
192− D.Yield Array{..} s ->
193− D.Skip (InnerLoop s arrContents arrStart arrEnd)
194− D.Skip s -> D.Skip (OuterLoop s)
195− D.Stop -> D.Stop
196−
197− step' _ (InnerLoop st _ p end) | p == end =
198− return $ D.Yield sep $ OuterLoop st
199−
200− step' _ (InnerLoop st contents p end) = do
201− x <- liftIO $ peekByteIndex p contents
202− return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
119+unlines = Array.interposeSuffix
203120
204121-------------------------------------------------------------------------------
205122-- Compact
206123-------------------------------------------------------------------------------
207124
208−-- XXX These would not be needed once we implement compactLEFold, see
209−-- module Streamly.Internal.Data.Stream.MutChunked
210−--
211−-- XXX Note that this thaws immutable arrays for appending, that may be
212−-- problematic if multiple users do the same thing, however, immutable arrays
213−-- would usually have no capacity to append, therefore, a copy will be forced
214−-- anyway. Confirm this. We can forcefully trim the array capacity before thaw
215−-- to ensure this.
216−{-# INLINE_NORMAL packArraysChunksOf #-}
217−packArraysChunksOf :: (MonadIO m, Unbox a)
218− => Int -> D.Stream m (Array a) -> D.Stream m (Array a)
219−packArraysChunksOf n str =
220− D.map A.unsafeFreeze $ AS.packArraysChunksOf n $ D.map A.unsafeThaw str
221−
222125-- XXX instead of writing two different versions of this operation, we should
223126-- write it as a pipe.
224127--
@@ -226,8 +129,7 @@ packArraysChunksOf n str =
226129{-# INLINE_NORMAL lpackArraysChunksOf #-}
227130lpackArraysChunksOf :: (MonadIO m, Unbox a)
228131 => Int -> Fold m (Array a) () -> Fold m (Array a) ()
229−lpackArraysChunksOf n fld =
230− FL.lmap A.unsafeThaw $ AS.lpackArraysChunksOf n (FL.lmap A.unsafeFreeze fld)
132+lpackArraysChunksOf = Array.lCompactGE
231133
232134-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
233135-- maximum specified size in bytes.
@@ -236,68 +138,19 @@ lpackArraysChunksOf n fld =
236138{-# INLINE compact #-}
237139compact :: (MonadIO m, Unbox a)
238140 => Int -> Stream m (Array a) -> Stream m (Array a)
239−compact = packArraysChunksOf
141+compact = Array.compactLE
240142
241−-------------------------------------------------------------------------------
242−-- Split
243−-------------------------------------------------------------------------------
244−
245−data SplitState s arr
246− = Initial s
247− | Buffering s arr
248− | Splitting s arr
249− | Yielding arr (SplitState s arr)
250− | Finishing
251−
252−-- | Split a stream of arrays on a given separator byte, dropping the separator
253−-- and coalescing all the arrays between two separators into a single array.
143+-- | Given a stream of arrays, splice them all together to generate a single
144+-- array. The stream must be /finite/.
254145--
255146-- @since 0.7.0
256−{-# INLINE_NORMAL _splitOn #-}
257−_splitOn
258− :: MonadIO m
259− => Word8
260− -> D.Stream m (Array Word8)
261− -> D.Stream m (Array Word8)
262−_splitOn byte (D.Stream step state) = D.Stream step' (Initial state)
263−
264− where
265−
266− {-# INLINE_LATE step' #-}
267− step' gst (Initial st) = do
268− r <- step gst st
269− case r of
270− D.Yield arr s -> do
271− (arr1, marr2) <- A.breakOn byte arr
272− return $ case marr2 of
273− Nothing -> D.Skip (Buffering s arr1)
274− Just arr2 -> D.Skip (Yielding arr1 (Splitting s arr2))
275− D.Skip s -> return $ D.Skip (Initial s)
276− D.Stop -> return D.Stop
277−
278− step' gst (Buffering st buf) = do
279− r <- step gst st
280− case r of
281− D.Yield arr s -> do
282− (arr1, marr2) <- A.breakOn byte arr
283− buf' <- A.splice buf arr1
284− return $ case marr2 of
285− Nothing -> D.Skip (Buffering s buf')
286− Just x -> D.Skip (Yielding buf' (Splitting s x))
287− D.Skip s -> return $ D.Skip (Buffering s buf)
288− D.Stop -> return $
289− if A.byteLength buf == 0
290− then D.Stop
291− else D.Skip (Yielding buf Finishing)
292−
293− step' _ (Splitting st buf) = do
294− (arr1, marr2) <- A.breakOn byte buf
295− return $ case marr2 of
296− Nothing -> D.Skip $ Buffering st arr1
297− Just arr2 -> D.Skip $ Yielding arr1 (Splitting st arr2)
147+{-# INLINE toArray #-}
148+toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
149+toArray = Array.fromChunks
298150
299− step' _ (Yielding arr next) = return $ D.Yield arr next
300− step' _ Finishing = return D.Stop
151+-------------------------------------------------------------------------------
152+-- Split
153+-------------------------------------------------------------------------------
301154
302155-- XXX Remove MonadIO constraint.
303156-- | Split a stream of arrays on a given separator byte, dropping the separator
@@ -310,7 +163,7 @@ splitOn
310163 => Word8
311164 -> Stream m (Array Word8)
312165 -> Stream m (Array Word8)
313−splitOn byte = D.splitInnerBy (A.breakOn byte) A.splice
166+splitOn = Array.compactOnByte
314167
315168{-# INLINE splitOnSuffix #-}
316169splitOnSuffix
@@ -318,8 +171,7 @@ splitOnSuffix
318171 => Word8
319172 -> Stream m (Array Word8)
320173 -> Stream m (Array Word8)
321−-- splitOn byte s = fromStreamD $ A.splitOn byte $ toStreamD s
322−splitOnSuffix byte = D.splitInnerBySuffix (A.breakOn byte) A.splice
174+splitOnSuffix = Array.compactOnByteSuffix
323175
324176-------------------------------------------------------------------------------
325177-- Elimination - Running folds
@@ -328,77 +180,24 @@ splitOnSuffix byte = D.splitInnerBySuffix (A.breakOn byte) A.splice
328180{-# INLINE_NORMAL foldBreakD #-}
329181foldBreakD :: forall m a b. (MonadIO m, Unbox a) =>
330182 Fold m a b -> D.Stream m (Array a) -> m (b, D.Stream m (Array a))
331−foldBreakD (FL.Fold fstep initial extract) stream@(D.Stream step state) = do
332− res <- initial
333− case res of
334− FL.Partial fs -> go SPEC state fs
335− FL.Done fb -> return $! (fb, stream)
336−
337− where
338−
339− {-# INLINE go #-}
340− go !_ st !fs = do
341− r <- step defState st
342− case r of
343− D.Yield (Array contents start end) s ->
344− let fp = Tuple' end contents
345− in goArray SPEC s fp start fs
346− D.Skip s -> go SPEC s fs
347− D.Stop -> do
348− b <- extract fs
349− return (b, D.nil)
350−
351− goArray !_ s (Tuple' end _) !cur !fs
352− | cur == end = do
353− go SPEC s fs
354− goArray !_ st fp@(Tuple' end contents) !cur !fs = do
355− x <- liftIO $ peekByteIndex cur contents
356− res <- fstep fs x
357− let next = INDEX_NEXT(cur,a)
358− case res of
359− FL.Done b -> do
360− let arr = Array contents next end
361− return $! (b, D.cons arr (D.Stream step st))
362− FL.Partial fs1 -> goArray SPEC st fp next fs1
363−
364−{-# INLINE_NORMAL foldBreakK #-}
365−foldBreakK :: forall m a b. (MonadIO m, Unbox a) =>
366− Fold m a b -> K.StreamK m (Array a) -> m (b, K.StreamK m (Array a))
367−foldBreakK (FL.Fold fstep initial extract) stream = do
368− res <- initial
369− case res of
370− FL.Partial fs -> go fs stream
371− FL.Done fb -> return (fb, stream)
372−
373− where
374−
375− {-# INLINE go #-}
376− go !fs st = do
377− let stop = (, K.nil) <$> extract fs
378− single a = yieldk a K.nil
379− yieldk (Array contents start end) r =
380− let fp = Tuple' end contents
381− in goArray fs r fp start
382− in K.foldStream defState yieldk single stop st
383−
384− goArray !fs st (Tuple' end _) !cur
385− | cur == end = do
386− go fs st
387− goArray !fs st fp@(Tuple' end contents) !cur = do
388− x <- liftIO $ peekByteIndex cur contents
389− res <- fstep fs x
390− let next = INDEX_NEXT(cur,a)
391− case res of
392− FL.Done b -> do
393− let arr = Array contents next end
394− return $! (b, K.cons arr st)
395− FL.Partial fs1 -> goArray fs1 st fp next
183+foldBreakD = Array.foldBreakChunks
396184
397185-- | Fold an array stream using the supplied 'Fold'. Returns the fold result
398186-- and the unconsumed stream.
399187--
400188-- > foldBreak f = runArrayFoldBreak (ChunkFold.fromFold f)
401189--
190+-- Instead of using this we can adapt the fold to ParserK and use
191+-- parseBreakChunks instead. ParserK allows composing using Monad as well.
192+--
193+-- @
194+-- foldBreak f s =
195+-- fmap (first (fromRight undefined))
196+-- $ K.parseBreakChunks (ParserK.adaptC (PR.fromFold f)) s
197+-- @
198+--
199+-- We can compare perf and remove this one or define it in terms of that.
200+--
402201-- /Internal/
403202--
404203{-# INLINE_NORMAL foldBreak #-}
@@ -407,14 +206,20 @@ foldBreak ::
407206 => Fold m a b
408207 -> StreamK m (A.Array a)
409208 -> m (b, StreamK m (A.Array a))
209+foldBreak = Array.foldBreakChunksK
210+--
410211-- foldBreak f s = fmap fromStreamD <$> foldBreakD f (toStreamD s)
411−foldBreak = foldBreakK
212+--
213+-- foldBreak f s =
214+-- fmap (first (fromRight undefined))
215+-- $ K.parseBreakChunks (ParserK.adaptC (PR.fromFold f)) s
216+--
412217-- If foldBreak performs better than runArrayFoldBreak we can use a rewrite
413218-- rule to rewrite runArrayFoldBreak to fold.
414219-- foldBreak f = runArrayFoldBreak (ChunkFold.fromFold f)
415220
416221-------------------------------------------------------------------------------
417−-- Fold to a single Array
222+-- Elimination - running element parsers
418223-------------------------------------------------------------------------------
419224
420225-- When we have to take an array partially, take the last part of the array.
@@ -460,341 +265,22 @@ splitAtArrayListRev n ls
460265 arr1 = Array contents end1 end
461266 in ([arr1], arr2:xs)
462267
463−-------------------------------------------------------------------------------
464−-- Fold to a single Array
465−-------------------------------------------------------------------------------
466−
467−{-
468−
469−-- XXX Both of these implementations of splicing seem to perform equally well.
470−-- We need to perform benchmarks over a range of sizes though.
471−
472−-- CAUTION! length must more than equal to lengths of all the arrays in the
473−-- stream.
474−{-# INLINE spliceArraysLenUnsafe #-}
475−spliceArraysLenUnsafe :: (MonadIO m, Unbox a)
476− => Int -> Stream m (MutArray a) -> m (MutArray a)
477−spliceArraysLenUnsafe len buffered = do
478− -- XXX The new array's pinned state should depend on the first element
479− -- of the stream. We should uncons the stream and do the required.
480− arr <- undefined
481− D.foldlM' MA.spliceUnsafe (return arr) buffered
482−
483−{-# INLINE _spliceArrays #-}
484−_spliceArrays :: (MonadIO m, Unbox a)
485− => Stream m (Array a) -> m (Array a)
486−_spliceArrays s = do
487− buffered <- D.foldr K.cons K.nil s
488− len <- K.fold FL.sum (fmap Array.length buffered)
489− -- XXX The new array's pinned state should depend on the first element
490− -- of the stream. We should uncons the stream and do the required.
491− arr <- undefined
492− final <- D.foldlM' writeArr (return arr) (toStream buffered)
493− return $ A.unsafeFreeze final
494−
495− where
496−
497− writeArr dst arr = MA.spliceUnsafe dst (A.unsafeThaw arr)
498−
499−{-# INLINE _spliceArraysBuffered #-}
500−_spliceArraysBuffered :: (MonadIO m, Unbox a)
501− => Stream m (Array a) -> m (Array a)
502−_spliceArraysBuffered s = do
503− buffered <- D.foldr K.cons K.nil s
504− len <- K.fold FL.sum (fmap Array.length buffered)
505− A.unsafeFreeze <$>
506− spliceArraysLenUnsafe len (fmap A.unsafeThaw (toStream buffered))
507−
508−-}
509−
510−{-# INLINE spliceArraysRealloced #-}
511−spliceArraysRealloced :: forall m a. (MonadIO m, Unbox a)
512− => Stream m (Array a) -> m (Array a)
513−spliceArraysRealloced s = do
514− res <- D.uncons s
515− case res of
516− Just (a, strm) -> do
517− arr <-
518− D.foldlM'
519− MA.spliceExp
520− (pure (A.unsafeThaw a))
521− (fmap A.unsafeThaw strm)
522− liftIO $ A.unsafeFreeze <$> MA.rightSize arr
523− Nothing -> pure A.nil
524−
525−-- XXX This should just be "fold A.write"
526−--
527−-- | Given a stream of arrays, splice them all together to generate a single
528−-- array. The stream must be /finite/.
529−--
530−-- @since 0.7.0
531−{-# INLINE toArray #-}
532−toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
533−toArray = spliceArraysRealloced
534−-- spliceArrays = _spliceArraysBuffered
535−
536−-- exponentially increasing sizes of the chunks upto the max limit.
537−-- XXX this will be easier to implement with parsers/terminating folds
538−-- With this we should be able to reduce the number of chunks/allocations.
539−-- The reallocation/copy based toArray can also be implemented using this.
540−--
541−{-
542−{-# INLINE toArraysInRange #-}
543−toArraysInRange :: (MonadIO m, Unbox a)
544− => Int -> Int -> Fold m (Array a) b -> Fold m a b
545−toArraysInRange low high (Fold step initial extract) =
546−-}
547−
548−{-
549−-- | Fold the input to a pure buffered stream (List) of arrays.
550−{-# INLINE _toArraysOf #-}
551−_toArraysOf :: (MonadIO m, Unbox a)
552− => Int -> Fold m a (Stream Identity (Array a))
553−_toArraysOf n = FL.groupsOf n (A.writeNF n) FL.toStream
554−-}
555−
556−-------------------------------------------------------------------------------
557−-- Elimination - running element parsers
558−-------------------------------------------------------------------------------
559−
560268-- GHC parser does not accept {-# ANN type [] NoSpecConstr #-}, so we need
561269-- to make a newtype.
562270{-# ANN type List NoSpecConstr #-}
563271newtype List a = List {getList :: [a]}
564272
565−{-
566−-- This can be generalized to any type provided it can be unfolded to a stream
567−-- and it can be combined using a semigroup operation.
568−--
569−-- XXX This should be written using CPS (as parseK) if we want it to scale wrt
570−-- to the number of times it can be called on the same stream.
571−{-# INLINE_NORMAL parseBreakD #-}
572−parseBreakD ::
573− forall m a b. (MonadIO m, MonadThrow m, Unbox a)
574− => PRD.Parser a m b
575− -> D.Stream m (Array.Array a)
576− -> m (b, D.Stream m (Array.Array a))
577−parseBreakD
578− (PRD.Parser pstep initial extract) stream@(D.Stream step state) = do
579−
580− res <- initial
581− case res of
582− PRD.IPartial s -> go SPEC state (List []) s
583− PRD.IDone b -> return (b, stream)
584− PRD.IError err -> throwM $ ParseError err
585−
586− where
587−
588− -- "backBuf" contains last few items in the stream that we may have to
589− -- backtrack to.
590− --
591− -- XXX currently we are using a dumb list based approach for backtracking
592− -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
593− -- That will allow us more efficient random back and forth movement.
594− go !_ st backBuf !pst = do
595− r <- step defState st
596− case r of
597− D.Yield (Array contents start end) s ->
598− gobuf SPEC s backBuf
599− (Tuple' end contents) start pst
600− D.Skip s -> go SPEC s backBuf pst
601− D.Stop -> do
602− b <- extract pst
603− return (b, D.nil)
604−
605− -- Use strictness on "cur" to keep it unboxed
606− gobuf !_ s backBuf (Tuple' end _) !cur !pst
607− | cur == end = do
608− go SPEC s backBuf pst
609− gobuf !_ s backBuf fp@(Tuple' end contents) !cur !pst = do
610− x <- liftIO $ peekByteIndex contents cur
611− pRes <- pstep pst x
612− let next = INDEX_NEXT(cur,a)
613− case pRes of
614− PR.Partial 0 pst1 ->
615− gobuf SPEC s (List []) fp next pst1
616− PR.Partial n pst1 -> do
617− assert (n <= Prelude.length (x:getList backBuf)) (return ())
618− let src0 = Prelude.take n (x:getList backBuf)
619− arr0 = A.fromListN n (Prelude.reverse src0)
620− arr1 = Array contents next end
621− src = arr0 <> arr1
622− let !(Array cont1 start end1) = src
623− fp1 = Tuple' end1 cont1
624− gobuf SPEC s (List []) fp1 start pst1
625− PR.Continue 0 pst1 ->
626− gobuf SPEC s (List (x:getList backBuf)) fp next pst1
627− PR.Continue n pst1 -> do
628− assert (n <= Prelude.length (x:getList backBuf)) (return ())
629− let (src0, buf1) = splitAt n (x:getList backBuf)
630− arr0 = A.fromListN n (Prelude.reverse src0)
631− arr1 = Array contents next end
632− src = arr0 <> arr1
633− let !(Array cont1 start end1) = src
634− fp1 = Tuple' end1 cont1
635− gobuf SPEC s (List buf1) fp1 start pst1
636− PR.Done 0 b -> do
637− let arr = Array contents next end
638− return (b, D.cons arr (D.Stream step s))
639− PR.Done n b -> do
640− assert (n <= Prelude.length (x:getList backBuf)) (return ())
641− let src0 = Prelude.take n (x:getList backBuf)
642− -- XXX create the array in reverse instead
643− arr0 = A.fromListN n (Prelude.reverse src0)
644− arr1 = Array contents next end
645− -- XXX Use StreamK to avoid adding arbitrary layers of
646− -- constructors every time.
647− str = D.cons arr0 (D.cons arr1 (D.Stream step s))
648− return (b, str)
649− PR.Error err -> throwM $ ParseError err
650−-}
651−
652−{-# INLINE_NORMAL parseBreakK #-}
653−parseBreakK ::
654− forall m a b. (MonadIO m, Unbox a)
655− => PRD.Parser a m b
656− -> K.StreamK m (Array.Array a)
657− -> m (Either ParseError b, K.StreamK m (Array.Array a))
658−parseBreakK (PRD.Parser pstep initial extract) stream = do
659− res <- initial
660− case res of
661− PRD.IPartial s -> go s stream []
662− PRD.IDone b -> return (Right b, stream)
663− PRD.IError err -> return (Left (ParseError err), stream)
664−
665− where
666−
667− -- "backBuf" contains last few items in the stream that we may have to
668− -- backtrack to.
669− --
670− -- XXX currently we are using a dumb list based approach for backtracking
671− -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
672− -- That will allow us more efficient random back and forth movement.
673− go !pst st backBuf = do
674− let stop = goStop pst backBuf -- (, K.nil) <$> extract pst
675− single a = yieldk a K.nil
676− yieldk arr r = goArray pst backBuf r arr
677− in K.foldStream defState yieldk single stop st
678−
679− -- Use strictness on "cur" to keep it unboxed
680− goArray !pst backBuf st (Array _ cur end) | cur == end = go pst st backBuf
681− goArray !pst backBuf st (Array contents cur end) = do
682− x <- liftIO $ peekByteIndex cur contents
683− pRes <- pstep pst x
684− let next = INDEX_NEXT(cur,a)
685− case pRes of
686− PR.Partial 0 s ->
687− goArray s [] st (Array contents next end)
688− PR.Partial n s -> do
689− assert (n <= Prelude.length (x:backBuf)) (return ())
690− let src0 = Prelude.take n (x:backBuf)
691− arr0 = A.fromListN n (Prelude.reverse src0)
692− arr1 = Array contents next end
693− src = arr0 <> arr1
694− goArray s [] st src
695− PR.Continue 0 s ->
696− goArray s (x:backBuf) st (Array contents next end)
697− PR.Continue n s -> do
698− assert (n <= Prelude.length (x:backBuf)) (return ())
699− let (src0, buf1) = splitAt n (x:backBuf)
700− arr0 = A.fromListN n (Prelude.reverse src0)
701− arr1 = Array contents next end
702− src = arr0 <> arr1
703− goArray s buf1 st src
704− PR.Done 0 b -> do
705− let arr = Array contents next end
706− return (Right b, K.cons arr st)
707− PR.Done n b -> do
708− assert (n <= Prelude.length (x:backBuf)) (return ())
709− let src0 = Prelude.take n (x:backBuf)
710− -- XXX Use fromListRevN once implemented
711− -- arr0 = A.fromListRevN n src0
712− arr0 = A.fromListN n (Prelude.reverse src0)
713− arr1 = Array contents next end
714− str = K.cons arr0 (K.cons arr1 st)
715− return (Right b, str)
716− PR.Error err -> do
717− let n = Prelude.length backBuf
718− arr0 = A.fromListN n (Prelude.reverse backBuf)
719− arr1 = Array contents cur end
720− str = K.cons arr0 (K.cons arr1 stream)
721− return (Left (ParseError err), str)
722−
723− -- This is a simplified goArray
724− goExtract !pst backBuf (Array _ cur end)
725− | cur == end = goStop pst backBuf
726− goExtract !pst backBuf (Array contents cur end) = do
727− x <- liftIO $ peekByteIndex cur contents
728− pRes <- pstep pst x
729− let next = INDEX_NEXT(cur,a)
730− case pRes of
731− PR.Partial 0 s ->
732− goExtract s [] (Array contents next end)
733− PR.Partial n s -> do
734− assert (n <= Prelude.length (x:backBuf)) (return ())
735− let src0 = Prelude.take n (x:backBuf)
736− arr0 = A.fromListN n (Prelude.reverse src0)
737− arr1 = Array contents next end
738− src = arr0 <> arr1
739− goExtract s [] src
740− PR.Continue 0 s ->
741− goExtract s backBuf (Array contents next end)
742− PR.Continue n s -> do
743− assert (n <= Prelude.length (x:backBuf)) (return ())
744− let (src0, buf1) = splitAt n (x:backBuf)
745− arr0 = A.fromListN n (Prelude.reverse src0)
746− arr1 = Array contents next end
747− src = arr0 <> arr1
748− goExtract s buf1 src
749− PR.Done 0 b -> do
750− let arr = Array contents next end
751− return (Right b, K.fromPure arr)
752− PR.Done n b -> do
753− assert (n <= Prelude.length backBuf) (return ())
754− let src0 = Prelude.take n backBuf
755− -- XXX Use fromListRevN once implemented
756− -- arr0 = A.fromListRevN n src0
757− arr0 = A.fromListN n (Prelude.reverse src0)
758− arr1 = Array contents next end
759− str = K.cons arr0 (K.fromPure arr1)
760− return (Right b, str)
761− PR.Error err -> do
762− let n = Prelude.length backBuf
763− arr0 = A.fromListN n (Prelude.reverse backBuf)
764− arr1 = Array contents cur end
765− str = K.cons arr0 (K.cons arr1 stream)
766− return (Left (ParseError err), str)
767−
768− -- This is a simplified goExtract
769− {-# INLINE goStop #-}
770− goStop !pst backBuf = do
771− pRes <- extract pst
772− case pRes of
773− PR.Partial _ _ -> error "Bug: parseBreak: Partial in extract"
774− PR.Continue 0 s ->
775− goStop s backBuf
776− PR.Continue n s -> do
777− assert (n <= Prelude.length backBuf) (return ())
778− let (src0, buf1) = splitAt n backBuf
779− arr = A.fromListN n (Prelude.reverse src0)
780− goExtract s buf1 arr
781− PR.Done 0 b ->
782− return (Right b, K.nil)
783− PR.Done n b -> do
784− assert (n <= Prelude.length backBuf) (return ())
785− let src0 = Prelude.take n backBuf
786− -- XXX Use fromListRevN once implemented
787− -- arr0 = A.fromListRevN n src0
788− arr0 = A.fromListN n (Prelude.reverse src0)
789− return (Right b, K.fromPure arr0)
790− PR.Error err -> do
791− let n = Prelude.length backBuf
792− arr0 = A.fromListN n (Prelude.reverse backBuf)
793− return (Left (ParseError err), K.fromPure arr0)
794−
795273-- | Parse an array stream using the supplied 'Parser'. Returns the parse
796274-- result and the unconsumed stream. Throws 'ParseError' if the parse fails.
797275--
276+-- >> parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
277+--
278+-- This is redundant and we can just use parseBreakChunks, as ParserK can be
279+-- composed using Monad. The only advantage of this is that we do not need to
280+-- adapt.
281+--
282+-- We can compare perf and remove this one or define it in terms of that.
283+--
798284-- /Internal/
799285--
800286{-# INLINE_NORMAL parseBreak #-}
@@ -807,7 +293,8 @@ parseBreak ::
807293parseBreak p s =
808294 fmap fromStreamD <$> parseBreakD (PRD.fromParserK p) (toStreamD s)
809295-}
810−parseBreak = parseBreakK
296+parseBreak = Array.parseBreakChunksK
297+-- parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
811298
812299-------------------------------------------------------------------------------
813300-- Elimination - Running Array Folds and parsers
core/src/Streamly/Internal/Data/Array/Type.hs+538−108
@@ -1,8 +1,11 @@
11{-# LANGUAGE CPP #-}
2+{-# LANGUAGE TypeFamilies #-}
3+-- Must come after TypeFamilies, otherwise it is re-enabled.
4+-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
5+{-# LANGUAGE NoMonoLocalBinds #-}
26-- |
37-- Module : Streamly.Internal.Data.Array.Type
48-- Copyright : (c) 2020 Composewell Technologies
5−--
69-- License : BSD3-3-Clause
710-- Maintainer : streamly@composewell.com
811-- Stability : experimental
@@ -10,79 +13,146 @@
1013--
1114-- See notes in "Streamly.Internal.Data.MutArray.Type"
1215--
16+
1317module Streamly.Internal.Data.Array.Type
1418 (
19+ -- ** Type
1520 -- $arrayNotes
1621 Array (..)
17− , asPtrUnsafe
18− , nil
1922
20− -- * Freezing and Thawing
23+ -- ** Conversion
24+ -- *** Mutable and Immutable
2125 , unsafeFreeze
2226 , unsafeFreezeWithShrink
2327 , unsafeThaw
2428
25− -- * Pinning and Unpinning
29+ -- *** Pinned and Unpinned
2630 , pin
2731 , unpin
2832 , isPinned
2933
30− -- * Construction
31− , splice
34+ -- *** Casting
35+ , unsafePinnedAsPtr
3236
33− , fromList
34− , pinnedFromList
37+ -- ** Construction
38+ , empty
39+
40+ -- *** Cloning
41+ -- XXX Why would we clone an immutable array?
42+ , clone
43+ , pinnedClone
44+
45+ -- *** Slicing
46+ -- | Get a subarray without copying
47+ , splitAt
48+ , breakOn -- XXX requires MonadIO
49+
50+ -- *** Stream Folds
51+ , unsafeMakePure
52+ , createOf
53+ , pinnedCreateOf
54+ , unsafeCreateOf
55+ , unsafePinnedCreateOf
56+ , create
57+ , pinnedCreate
58+ , createWith
59+
60+ -- *** From containers
3561 , fromListN
3662 , pinnedFromListN
37− , fromListRev
63+ , fromList
64+ , pinnedFromList
3865 , fromListRevN
39− , fromStreamDN
40− , fromStreamD
66+ , fromListRev
67+ , fromStreamN
68+ , fromStream
69+ , fromPureStreamN
4170 , fromPureStream
4271 , fromByteStr#
72+ , fromByteStr
73+ , fromPtrN
74+ , fromChunks
75+ , fromChunksK
4376
44− -- * Split
45− , breakOn
77+ -- ** Reading
4678
47− -- * Elimination
48− , unsafeIndexIO
49− , unsafeIndex -- getIndexUnsafe
50− , byteLength
79+ -- *** Indexing
80+ , unsafeIndexIO -- XXX unsafeGetIndexIO
81+ , getIndexUnsafe -- XXX unsafeGetIndex
82+
83+ -- *** To Streams
84+ , read
85+ , readRev
86+ , toStreamK
87+ , toStreamKRev
88+
89+ -- *** To Containers
90+ , toList
91+
92+ -- *** Unfolds
93+ , producer -- experimental
94+ , readerUnsafe
95+ , reader
96+ , readerRev
97+
98+ -- *** Size
5199 , length
100+ , byteLength
52101
102+ -- ** Folding
53103 , foldl'
54104 , foldr
55− , splitAt
105+ , byteCmp
106+ , byteEq
107+
108+ -- ** Appending
109+ , splice -- XXX requires MonadIO
110+
111+ -- ** Streams of arrays
112+ -- *** Chunk
113+ -- | Group a stream into arrays.
114+ , chunksOf
115+ , pinnedChunksOf
116+ , buildChunks
117+
118+ -- *** Split
119+ -- | Split an array into slices.
120+
121+ -- *** Concat
122+ -- | Append the arrays in a stream to form a stream of elements.
123+ , concat
124+ , concatRev
125+
126+ -- *** Compact
127+ -- | Append the arrays in a stream to form a stream of larger arrays.
128+ , fCompactGE
129+ , fPinnedCompactGE
130+ , lCompactGE
131+ , lPinnedCompactGE
132+ , compactGE
56133
134+ -- ** Deprecated
135+ , asPtrUnsafe
136+ , unsafeIndex
137+ , bufferChunks
138+ , flattenArrays
139+ , flattenArraysRev
140+ , fromArrayStreamK
141+ , fromStreamDN
142+ , fromStreamD
57143 , toStreamD
58144 , toStreamDRev
59− , toStreamK
60− , toStreamKRev
61145 , toStream
62146 , toStreamRev
63− , read
64− , readRev
65− , readerRev
66− , toList
67−
68− -- * Folds
147+ , nil
69148 , writeWith
70149 , writeN
71150 , pinnedWriteN
72151 , writeNUnsafe
73152 , pinnedWriteNUnsafe
74− , MA.ArrayUnsafe (..)
75153 , pinnedWriteNAligned
76154 , write
77155 , pinnedWrite
78− , unsafeMakePure
79−
80− -- * Streams of arrays
81− , chunksOf
82− , pinnedChunksOf
83− , bufferChunks
84− , flattenArrays
85− , flattenArraysRev
86156 )
87157where
88158
@@ -90,31 +160,36 @@ where
90160#include "inline.hs"
91161
92162import Control.Exception (assert)
93−import Control.Monad (replicateM)
163+import Control.Monad (replicateM, when)
94164import Control.Monad.IO.Class (MonadIO(..))
95165import Data.Functor.Identity (Identity(..))
166+import Data.Int (Int8, Int16, Int32, Int64)
96167import Data.Proxy (Proxy(..))
97−import Data.Word (Word8)
168+import Data.Word (Word8, Word16, Word32, Word64)
98169import GHC.Base (build)
99170import GHC.Exts (IsList, IsString(..), Addr#)
100171
101172import GHC.IO (unsafePerformIO)
102173import GHC.Ptr (Ptr(..))
103−import Streamly.Internal.Data.MutArray.Type (MutArray(..), MutableByteArray)
174+import Streamly.Internal.Data.Producer.Type (Producer(..))
175+import Streamly.Internal.Data.MutArray.Type (MutArray(..))
176+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
104177import Streamly.Internal.Data.Fold.Type (Fold(..))
105−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
178+import Streamly.Internal.Data.Stream.Type (Stream)
179+import Streamly.Internal.Data.StreamK.Type (StreamK)
106180import Streamly.Internal.Data.Unbox (Unbox(..))
107181import Streamly.Internal.Data.Unfold.Type (Unfold(..))
108182import Text.Read (readPrec)
109183
110−import Prelude hiding (Foldable(..), read, unlines, splitAt)
184+import Prelude hiding (Foldable(..), concat, read, unlines, splitAt)
111185
112186import qualified GHC.Exts as Exts
187+import qualified Streamly.Internal.Data.Fold.Type as Fold
113188import qualified Streamly.Internal.Data.MutArray.Type as MA
114−import qualified Streamly.Internal.Data.Stream.StreamD.Type as D
115−import qualified Streamly.Internal.Data.Stream.StreamD.Generate as D
116−import qualified Streamly.Internal.Data.Stream.StreamK.Type as K
117−import qualified Streamly.Internal.Data.Unbox as Unboxed
189+import qualified Streamly.Internal.Data.Stream.Type as D
190+import qualified Streamly.Internal.Data.StreamK.Type as K
191+import qualified Streamly.Internal.Data.MutByteArray.Type as Unboxed
192+import qualified Streamly.Internal.Data.Producer as Producer
118193import qualified Streamly.Internal.Data.Unfold.Type as Unfold
119194import qualified Text.ParserCombinators.ReadPrec as ReadPrec
120195
@@ -157,7 +232,7 @@ data Array a =
157232#endif
158233 -- All offsets are in terms of bytes from the start of arraycontents
159234 Array
160− { arrContents :: {-# UNPACK #-} !MutableByteArray
235+ { arrContents :: {-# UNPACK #-} !MutByteArray
161236 , arrStart :: {-# UNPACK #-} !Int -- offset
162237 , arrEnd :: {-# UNPACK #-} !Int -- offset + len
163238 }
@@ -168,14 +243,20 @@ data Array a =
168243
169244-- | Use an @Array a@ as @Ptr a@.
170245--
171−-- See 'MA.asPtrUnsafe' in the Mutable array module for more details.
246+-- See 'MA.unsafePinnedAsPtr' in the Mutable array module for more details.
172247--
173248-- /Unsafe/
174249--
175250-- /Pre-release/
176251--
252+{-# INLINE unsafePinnedAsPtr #-}
253+unsafePinnedAsPtr :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
254+unsafePinnedAsPtr arr = MA.unsafePinnedAsPtr (unsafeThaw arr)
255+
256+{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
257+{-# INLINE asPtrUnsafe #-}
177258asPtrUnsafe :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
178−asPtrUnsafe arr = MA.asPtrUnsafe (unsafeThaw arr)
259+asPtrUnsafe = unsafePinnedAsPtr
179260
180261-------------------------------------------------------------------------------
181262-- Freezing and Thawing
@@ -222,18 +303,19 @@ unsafeThaw (Array ac as ae) = MutArray ac as ae ae
222303-- Pinning & Unpinning
223304-------------------------------------------------------------------------------
224305
225−-- | Copy the 'Array' to pinned memory if unpinned, else do nothing. The
226−-- overhead is a copy if the input array is unpinned.
306+-- | Return a copy of the 'Array' in pinned memory if unpinned, else return the
307+-- original array.
227308{-# INLINE pin #-}
228309pin :: Array a -> IO (Array a)
229310pin = fmap unsafeFreeze . MA.pin . unsafeThaw
230311
231−-- | Copy the 'Array' to unpinned memory if pinned, else do nothing. The
232−-- overhead is a copy if the input array is pinned.
312+-- | Return a copy of the 'Array' in unpinned memory if pinned, else return the
313+-- original array.
233314{-# INLINE unpin #-}
234315unpin :: Array a -> IO (Array a)
235316unpin = fmap unsafeFreeze . MA.unpin . unsafeThaw
236317
318+-- | Return 'True' if the array is allocated in pinned memory.
237319{-# INLINE isPinned #-}
238320isPinned :: Array a -> Bool
239321isPinned = MA.isPinned . unsafeThaw
@@ -242,11 +324,14 @@ isPinned = MA.isPinned . unsafeThaw
242324-- Construction
243325-------------------------------------------------------------------------------
244326
245−-- Splice two immutable arrays creating a new array.
327+-- | Copy two immutable arrays into a new array. If you want to splice more
328+-- than two arrays then this operation would be highly inefficient because it
329+-- would make a copy on every splice operation, instead use the
330+-- 'fromChunksK' operation to combine n immutable arrays.
246331{-# INLINE splice #-}
247−splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
332+splice :: MonadIO m => Array a -> Array a -> m (Array a)
248333splice arr1 arr2 =
249− unsafeFreeze <$> MA.splice (unsafeThaw arr1) (unsafeThaw arr2)
334+ unsafeFreeze <$> MA.spliceCopy (unsafeThaw arr1) (unsafeThaw arr2)
250335
251336-- | Create an 'Array' from the first N elements of a list. The array is
252337-- allocated to size N, if the list terminates before N elements then the
@@ -290,24 +375,83 @@ pinnedFromList xs = unsafePerformIO $ unsafeFreeze <$> MA.pinnedFromList xs
290375fromListRev :: Unbox a => [a] -> Array a
291376fromListRev xs = unsafePerformIO $ unsafeFreeze <$> MA.fromListRev xs
292377
293−{-# INLINE_NORMAL fromStreamDN #-}
378+-- | Create an 'Array' from the first N elements of a stream. The array is
379+-- allocated to size N, if the stream terminates before N elements then the
380+-- array may hold less than N elements.
381+--
382+-- >>> fromStreamN n = Stream.fold (Array.writeN n)
383+--
384+-- /Pre-release/
385+{-# INLINE_NORMAL fromStreamN #-}
386+fromStreamN :: (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
387+fromStreamN n m = do
388+ when (n < 0) $ error "writeN: negative write count specified"
389+ unsafeFreeze <$> MA.fromStreamN n m
390+-- fromStreamN n = D.fold (writeN n)
391+
392+{-# DEPRECATED fromStreamDN "Please use fromStreamN instead." #-}
294393fromStreamDN :: forall m a. (MonadIO m, Unbox a)
295394 => Int -> D.Stream m a -> m (Array a)
296−fromStreamDN limit str = unsafeFreeze <$> MA.fromStreamDN limit str
395+fromStreamDN = fromStreamN
297396
397+-- | Create an 'Array' from a stream. This is useful when we want to create a
398+-- single array from a stream of unknown size. 'writeN' is at least twice
399+-- as efficient when the size is already known.
400+--
401+-- >>> fromStream = Stream.fold Array.write
402+--
403+-- Note that if the input stream is too large memory allocation for the array
404+-- may fail. When the stream size is not known, `chunksOf` followed by
405+-- processing of indvidual arrays in the resulting stream should be preferred.
406+--
407+-- /Pre-release/
298408{-# INLINE_NORMAL fromStreamD #-}
409+fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (Array a)
410+fromStream = D.fold write
411+-- fromStreamD str = unsafeFreeze <$> MA.fromStream str
412+
413+{-# DEPRECATED fromStreamD "Please use fromStream instead." #-}
299414fromStreamD :: forall m a. (MonadIO m, Unbox a)
300415 => D.Stream m a -> m (Array a)
301−fromStreamD str = unsafeFreeze <$> MA.fromStreamD str
416+fromStreamD = fromStream
417+
418+-------------------------------------------------------------------------------
419+-- Cloning
420+-------------------------------------------------------------------------------
421+
422+{-# INLINE clone #-}
423+clone ::
424+ ( MonadIO m
425+#ifdef DEVBUILD
426+ , Unbox a
427+#endif
428+ )
429+ => Array a -> m (Array a)
430+clone = fmap unsafeFreeze . MA.clone . unsafeThaw
431+
432+{-# INLINE pinnedClone #-}
433+pinnedClone ::
434+ ( MonadIO m
435+#ifdef DEVBUILD
436+ , Unbox a
437+#endif
438+ )
439+ => Array a -> m (Array a)
440+pinnedClone = fmap unsafeFreeze . MA.pinnedClone . unsafeThaw
302441
303442-------------------------------------------------------------------------------
304443-- Streams of arrays
305444-------------------------------------------------------------------------------
306445
307−{-# INLINE bufferChunks #-}
446+{-# INLINE buildChunks #-}
447+buildChunks :: (MonadIO m, Unbox a) =>
448+ D.Stream m a -> m (K.StreamK m (Array a))
449+buildChunks m = D.foldr K.cons K.nil $ chunksOf defaultChunkSize m
450+
451+{-# DEPRECATED bufferChunks "Please use buildChunks instead." #-}
308452bufferChunks :: (MonadIO m, Unbox a) =>
309453 D.Stream m a -> m (K.StreamK m (Array a))
310−bufferChunks m = D.foldr K.cons K.nil $ chunksOf defaultChunkSize m
454+bufferChunks = buildChunks
311455
312456-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
313457-- @n@ elements each.
@@ -328,27 +472,100 @@ pinnedChunksOf :: forall m a. (MonadIO m, Unbox a)
328472 => Int -> D.Stream m a -> D.Stream m (Array a)
329473pinnedChunksOf n str = D.map unsafeFreeze $ MA.pinnedChunksOf n str
330474
331−-- | Use the "read" unfold instead.
332−--
333−-- @flattenArrays = unfoldMany read@
475+-- | Convert a stream of arrays into a stream of their elements.
334476--
335−-- We can try this if there are any fusion issues in the unfold.
477+-- >>> concat = Stream.unfoldMany Array.reader
336478--
337−{-# INLINE_NORMAL flattenArrays #-}
479+{-# INLINE_NORMAL concat #-}
480+concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
481+-- XXX this requires MonadIO whereas the unfoldMany version does not
482+concat = MA.concatWith (pure . unsafeInlineIO) . D.map unsafeThaw
483+-- concat = D.unfoldMany reader
484+
485+{-# DEPRECATED flattenArrays "Please use \"unfoldMany reader\" instead." #-}
486+{-# INLINE flattenArrays #-}
338487flattenArrays :: forall m a. (MonadIO m, Unbox a)
339488 => D.Stream m (Array a) -> D.Stream m a
340−flattenArrays = MA.flattenArrays . D.map unsafeThaw
489+flattenArrays = concat
341490
342−-- | Use the "readRev" unfold instead.
491+-- | Convert a stream of arrays into a stream of their elements reversing the
492+-- contents of each array before flattening.
343493--
344−-- @flattenArrays = unfoldMany readRev@
494+-- >>> concatRev = Stream.unfoldMany Array.readerRev
345495--
346−-- We can try this if there are any fusion issues in the unfold.
347−--
348−{-# INLINE_NORMAL flattenArraysRev #-}
496+{-# INLINE_NORMAL concatRev #-}
497+concatRev :: forall m a. (Monad m, Unbox a)
498+ => D.Stream m (Array a) -> D.Stream m a
499+-- XXX this requires MonadIO whereas the unfoldMany version does not
500+concatRev = MA.concatRevWith (pure . unsafeInlineIO) . D.map unsafeThaw
501+-- concatRev = D.unfoldMany readerRev
502+
503+{-# DEPRECATED flattenArraysRev "Please use \"unfoldMany readerRev\" instead." #-}
504+{-# INLINE flattenArraysRev #-}
349505flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
350506 => D.Stream m (Array a) -> D.Stream m a
351−flattenArraysRev = MA.flattenArraysRev . D.map unsafeThaw
507+flattenArraysRev = concatRev
508+
509+-------------------------------------------------------------------------------
510+-- Compact
511+-------------------------------------------------------------------------------
512+
513+-- XXX Note that this thaws immutable arrays for appending, that may be
514+-- problematic if multiple users do the same thing, however, thawed immutable
515+-- arrays would have no capacity to append, therefore, a copy will be forced
516+-- anyway.
517+
518+-- | Fold @fCompactGE n@ coalesces adjacent arrays in the input stream
519+-- until the size becomes greater than or equal to n.
520+--
521+-- Generates unpinned arrays irrespective of the pinning status of input
522+-- arrays.
523+{-# INLINE_NORMAL fCompactGE #-}
524+fCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
525+fCompactGE n = fmap unsafeFreeze $ Fold.lmap unsafeThaw $ MA.fCompactGE n
526+
527+-- | PInned version of 'fCompactGE'.
528+{-# INLINE_NORMAL fPinnedCompactGE #-}
529+fPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
530+fPinnedCompactGE n =
531+ fmap unsafeFreeze $ Fold.lmap unsafeThaw $ MA.fPinnedCompactGE n
532+
533+-- | @compactGE n stream@ coalesces adjacent arrays in the @stream@ until
534+-- the size becomes greater than or equal to @n@.
535+--
536+-- >>> compactGE n = Stream.foldMany (Array.fCompactGE n)
537+--
538+-- Generates unpinned arrays irrespective of the pinning status of input
539+-- arrays.
540+{-# INLINE compactGE #-}
541+compactGE ::
542+ (MonadIO m, Unbox a)
543+ => Int -> Stream m (Array a) -> Stream m (Array a)
544+compactGE n stream =
545+ D.map unsafeFreeze $ MA.compactGE n $ D.map unsafeThaw stream
546+
547+-- | Like 'compactGE' but for transforming folds instead of stream.
548+--
549+-- >>> lCompactGE n = Fold.many (Array.fCompactGE n)
550+--
551+-- Generates unpinned arrays irrespective of the pinning status of input
552+-- arrays.
553+{-# INLINE_NORMAL lCompactGE #-}
554+lCompactGE :: (MonadIO m, Unbox a)
555+ => Int -> Fold m (Array a) () -> Fold m (Array a) ()
556+lCompactGE n fld =
557+ Fold.lmap unsafeThaw $ MA.lCompactGE n (Fold.lmap unsafeFreeze fld)
558+
559+-- | Pinned version of 'lCompactGE'.
560+{-# INLINE_NORMAL lPinnedCompactGE #-}
561+lPinnedCompactGE :: (MonadIO m, Unbox a)
562+ => Int -> Fold m (Array a) () -> Fold m (Array a) ()
563+lPinnedCompactGE n fld =
564+ Fold.lmap unsafeThaw $ MA.lPinnedCompactGE n (Fold.lmap unsafeFreeze fld)
565+
566+-------------------------------------------------------------------------------
567+-- Splitting
568+-------------------------------------------------------------------------------
352569
353570-- Drops the separator byte
354571{-# INLINE breakOn #-}
@@ -370,9 +587,14 @@ unsafeIndexIO :: forall a. Unbox a => Int -> Array a -> IO a
370587unsafeIndexIO i arr = MA.getIndexUnsafe i (unsafeThaw arr)
371588
372589-- | Return element at the specified index without checking the bounds.
590+{-# INLINE_NORMAL getIndexUnsafe #-}
591+getIndexUnsafe :: forall a. Unbox a => Int -> Array a -> a
592+getIndexUnsafe i arr = let !r = unsafeInlineIO $ unsafeIndexIO i arr in r
593+
594+{-# DEPRECATED unsafeIndex "Please use 'getIndexUnsafe' instead" #-}
373595{-# INLINE_NORMAL unsafeIndex #-}
374596unsafeIndex :: forall a. Unbox a => Int -> Array a -> a
375−unsafeIndex i arr = let !r = unsafeInlineIO $ unsafeIndexIO i arr in r
597+unsafeIndex = getIndexUnsafe
376598
377599-- | /O(1)/ Get the byte length of the array.
378600--
@@ -387,24 +609,68 @@ byteLength = MA.byteLength . unsafeThaw
387609length :: Unbox a => Array a -> Int
388610length arr = MA.length (unsafeThaw arr)
389611
612+{-# INLINE_NORMAL producer #-}
613+producer :: forall m a. (Monad m, Unbox a) => Producer m (Array a) a
614+producer =
615+ Producer.translate unsafeThaw unsafeFreeze
616+ $ MA.producerWith (return . unsafeInlineIO)
617+
618+-- | Unfold an array into a stream.
619+--
620+{-# INLINE_NORMAL reader #-}
621+reader :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
622+reader = Producer.simplify producer
623+
624+-- | Unfold an array into a stream, does not check the end of the array, the
625+-- user is responsible for terminating the stream within the array bounds. For
626+-- high performance application where the end condition can be determined by
627+-- a terminating fold.
628+--
629+-- Written in the hope that it may be faster than "read", however, in the case
630+-- for which this was written, "read" proves to be faster even though the core
631+-- generated with unsafeRead looks simpler.
632+--
633+-- /Pre-release/
634+--
635+{-# INLINE_NORMAL readerUnsafe #-}
636+readerUnsafe :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
637+readerUnsafe = Unfold step inject
638+ where
639+
640+ inject (Array contents start end) =
641+ return (MA.ArrayUnsafe contents end start)
642+
643+ {-# INLINE_LATE step #-}
644+ step (MA.ArrayUnsafe contents end p) = do
645+ -- unsafeInlineIO allows us to run this in Identity monad for pure
646+ -- toList/foldr case which makes them much faster due to not
647+ -- accumulating the list and fusing better with the pure consumers.
648+ --
649+ -- This should be safe as the array contents are guaranteed to be
650+ -- evaluated/written to before we peek at them.
651+ let !x = unsafeInlineIO $ peekAt p contents
652+ let !p1 = INDEX_NEXT(p,a)
653+ return $ D.Yield x (MA.ArrayUnsafe contents end p1)
654+
390655-- | Unfold an array into a stream in reverse order.
391656--
392657{-# INLINE_NORMAL readerRev #-}
393658readerRev :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
394659readerRev = Unfold.lmap unsafeThaw $ MA.readerRevWith (return . unsafeInlineIO)
395660
661+{-# DEPRECATED toStreamD "Please use 'read' instead." #-}
396662{-# INLINE_NORMAL toStreamD #-}
397663toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> D.Stream m a
398−toStreamD arr = MA.toStreamDWith (return . unsafeInlineIO) (unsafeThaw arr)
664+toStreamD = read
399665
400666{-# INLINE toStreamK #-}
401667toStreamK :: forall m a. (Monad m, Unbox a) => Array a -> K.StreamK m a
402668toStreamK arr = MA.toStreamKWith (return . unsafeInlineIO) (unsafeThaw arr)
403669
670+{-# DEPRECATED toStreamDRev "Please use 'readRev' instead." #-}
404671{-# INLINE_NORMAL toStreamDRev #-}
405672toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> D.Stream m a
406−toStreamDRev arr =
407− MA.toStreamDRevWith (return . unsafeInlineIO) (unsafeThaw arr)
673+toStreamDRev = readRev
408674
409675{-# INLINE toStreamKRev #-}
410676toStreamKRev :: forall m a. (Monad m, Unbox a) => Array a -> K.StreamK m a
@@ -416,7 +682,7 @@ toStreamKRev arr =
416682-- /Pre-release/
417683{-# INLINE_EARLY read #-}
418684read :: (Monad m, Unbox a) => Array a -> Stream m a
419−read = toStreamD
685+read arr = MA.toStreamWith (return . unsafeInlineIO) (unsafeThaw arr)
420686
421687-- | Same as 'read'
422688--
@@ -433,7 +699,7 @@ toStream = read
433699-- /Pre-release/
434700{-# INLINE_EARLY readRev #-}
435701readRev :: (Monad m, Unbox a) => Array a -> Stream m a
436−readRev = toStreamDRev
702+readRev arr = MA.toStreamRevWith (return . unsafeInlineIO) (unsafeThaw arr)
437703
438704-- | Same as 'readRev'
439705--
@@ -457,6 +723,7 @@ foldr f z arr = runIdentity $ D.foldr f z $ toStreamD arr
457723-- | Create two slices of an array without copying the original array. The
458724-- specified index @i@ is the first index of the second slice.
459725--
726+{-# INLINE splitAt #-}
460727splitAt :: Unbox a => Int -> Array a -> (Array a, Array a)
461728splitAt i arr = (unsafeFreeze a, unsafeFreeze b)
462729 where
@@ -476,8 +743,8 @@ toListFB c n Array{..} = go arrStart
476743 -- accumulating the list and fusing better with the pure consumers.
477744 --
478745 -- This should be safe as the array contents are guaranteed to be
479− -- evaluated/written to before we peekByteIndex at them.
480− let !x = unsafeInlineIO $ peekByteIndex p arrContents
746+ -- evaluated/written to before we peekAt at them.
747+ let !x = unsafeInlineIO $ peekAt p arrContents
481748 in c x (go (INDEX_NEXT(p,a)))
482749
483750-- | Convert an 'Array' into a list.
@@ -490,17 +757,28 @@ toList s = build (\c n -> toListFB c n s)
490757-- Folds
491758-------------------------------------------------------------------------------
492759
493−-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
760+-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
494761-- 'Array'.
495762--
496−{-# INLINE_NORMAL writeN #-}
763+{-# INLINE_NORMAL createOf #-}
764+createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
765+createOf = fmap unsafeFreeze . MA.createOf
766+
767+-- XXX Deprecate in major
768+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
769+{-# INLINE writeN #-}
497770writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
498−writeN = fmap unsafeFreeze . MA.writeN
771+writeN = createOf
499772
500−-- | Like 'fromListN' but creates a pinned array.
501−{-# INLINE_NORMAL pinnedWriteN #-}
773+-- | Like 'createOf' but creates a pinned array.
774+{-# INLINE_NORMAL pinnedCreateOf #-}
775+pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
776+pinnedCreateOf = fmap unsafeFreeze . MA.pinnedCreateOf
777+
778+{-# DEPRECATED pinnedWriteN "Please use pinnedCreateOf instead." #-}
779+{-# INLINE pinnedWriteN #-}
502780pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
503−pinnedWriteN = fmap unsafeFreeze . MA.pinnedWriteN
781+pinnedWriteN = pinnedCreateOf
504782
505783-- | @pinnedWriteNAligned alignment n@ folds a maximum of @n@ elements from the input
506784-- stream to an 'Array' aligned to the given size.
@@ -508,44 +786,74 @@ pinnedWriteN = fmap unsafeFreeze . MA.pinnedWriteN
508786-- /Pre-release/
509787--
510788{-# INLINE_NORMAL pinnedWriteNAligned #-}
789+{-# DEPRECATED pinnedWriteNAligned "To be removed." #-}
511790pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a)
512791 => Int -> Int -> Fold m a (Array a)
513792pinnedWriteNAligned alignSize = fmap unsafeFreeze . MA.pinnedWriteNAligned alignSize
514793
515−-- | Like 'writeN' but does not check the array bounds when writing. The fold
794+-- | Like 'createOf' but does not check the array bounds when writing. The fold
516795-- driver must not call the step function more than 'n' times otherwise it will
517796-- corrupt the memory and crash. This function exists mainly because any
518797-- conditional in the step function blocks fusion causing 10x performance
519798-- slowdown.
520799--
521−{-# INLINE_NORMAL writeNUnsafe #-}
800+{-# INLINE_NORMAL unsafeCreateOf #-}
801+unsafeCreateOf :: forall m a. (MonadIO m, Unbox a)
802+ => Int -> Fold m a (Array a)
803+unsafeCreateOf n = unsafeFreeze <$> MA.unsafeCreateOf n
804+
805+{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
806+{-# INLINE writeNUnsafe #-}
522807writeNUnsafe :: forall m a. (MonadIO m, Unbox a)
523808 => Int -> Fold m a (Array a)
524−writeNUnsafe n = unsafeFreeze <$> MA.writeNUnsafe n
809+writeNUnsafe = unsafeCreateOf
525810
526−{-# INLINE_NORMAL pinnedWriteNUnsafe #-}
811+{-# INLINE_NORMAL unsafePinnedCreateOf #-}
812+unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a)
813+ => Int -> Fold m a (Array a)
814+unsafePinnedCreateOf n = unsafeFreeze <$> MA.unsafePinnedCreateOf n
815+
816+{-# DEPRECATED pinnedWriteNUnsafe "Please use unsafePinnedCreateOf instead." #-}
817+{-# INLINE pinnedWriteNUnsafe #-}
527818pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a)
528819 => Int -> Fold m a (Array a)
529−pinnedWriteNUnsafe n = unsafeFreeze <$> MA.pinnedWriteNUnsafe n
820+pinnedWriteNUnsafe = unsafePinnedCreateOf
821+
822+{-# INLINE_NORMAL createWith #-}
823+createWith :: forall m a. (MonadIO m, Unbox a)
824+ => Int -> Fold m a (Array a)
825+-- createWith n = FL.rmapM spliceArrays $ toArraysOf n
826+createWith elemCount = unsafeFreeze <$> MA.createWith elemCount
530827
531−{-# INLINE_NORMAL writeWith #-}
828+{-# DEPRECATED writeWith "Please use createWith instead." #-}
829+{-# INLINE writeWith #-}
532830writeWith :: forall m a. (MonadIO m, Unbox a)
533831 => Int -> Fold m a (Array a)
534−-- writeWith n = FL.rmapM spliceArrays $ toArraysOf n
535−writeWith elemCount = unsafeFreeze <$> MA.writeWith elemCount
832+writeWith = createWith
536833
537834-- | Fold the whole input to a single array.
538835--
539836-- /Caution! Do not use this on infinite streams./
540837--
838+{-# INLINE create #-}
839+create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
840+create = fmap unsafeFreeze MA.create
841+
842+-- XXX Deprecate in major
843+-- {-# DEPRECATED write "Please use create instead." #-}
541844{-# INLINE write #-}
542845write :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
543−write = fmap unsafeFreeze MA.write
846+write = create
544847
545−-- | Like 'write' but creates a pinned array.
848+-- | Like 'create' but creates a pinned array.
849+{-# INLINE pinnedCreate #-}
850+pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
851+pinnedCreate = fmap unsafeFreeze MA.pinnedCreate
852+
853+{-# DEPRECATED pinnedWrite "Please use pinnedCreate instead." #-}
546854{-# INLINE pinnedWrite #-}
547855pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
548−pinnedWrite = fmap unsafeFreeze MA.pinnedWrite
856+pinnedWrite = pinnedCreate
549857
550858-- | Fold "step" has a dependency on "initial", and each step is dependent on
551859-- the previous invocation of step due to state passing, finally extract
@@ -567,15 +875,20 @@ pinnedWrite = fmap unsafeFreeze MA.pinnedWrite
567875-- could be unsafe and dangerous. This is dangerous especially when used with
568876-- foldMany like operations.
569877--
570−-- >>> import qualified Streamly.Internal.Data.Array.Type as Array
571878-- >>> unsafePureWrite = Array.unsafeMakePure Array.write
572879--
573880{-# INLINE unsafeMakePure #-}
574881unsafeMakePure :: Monad m => Fold IO a b -> Fold m a b
575−unsafeMakePure (Fold step initial extract) =
882+unsafeMakePure (Fold step initial extract final) =
576883 Fold (\x a -> return $! unsafeInlineIO (step x a))
577884 (return $! unsafePerformIO initial)
578885 (\s -> return $! unsafeInlineIO $ extract s)
886+ (\s -> return $! unsafeInlineIO $ final s)
887+
888+{-# INLINE fromPureStreamN #-}
889+fromPureStreamN :: Unbox a => Int -> Stream Identity a -> Array a
890+fromPureStreamN n x =
891+ unsafePerformIO $ fmap unsafeFreeze (MA.fromPureStreamN n x)
579892
580893-- | Convert a pure stream in Identity monad to an immutable array.
581894--
@@ -588,6 +901,19 @@ fromPureStream x = unsafePerformIO $ fmap (unsafeFreeze) (MA.fromPureStream x)
588901-- fromPureStream = runIdentity . D.fold (unsafeMakePure write)
589902-- fromPureStream = fromList . runIdentity . D.toList
590903
904+-- XXX This should be monadic.
905+
906+-- | Copy an immutable 'Ptr Word8' sequence into an array.
907+--
908+-- /Unsafe:/ The caller is responsible for safe addressing.
909+--
910+-- Note that this should be evaluated strictly to ensure that we do not hold
911+-- the reference to the pointer in a lazy thunk.
912+fromPtrN :: Int -> Ptr Word8 -> Array Word8
913+fromPtrN n addr = unsafePerformIO $ fmap unsafeFreeze (MA.fromPtrN n addr)
914+
915+-- XXX This should be monadic.
916+
591917-- | Copy a null terminated immutable 'Addr#' Word8 sequence into an array.
592918--
593919-- /Unsafe:/ The caller is responsible for safe addressing.
@@ -595,13 +921,49 @@ fromPureStream x = unsafePerformIO $ fmap (unsafeFreeze) (MA.fromPureStream x)
595921-- Note that this is completely safe when reading from Haskell string
596922-- literals because they are guaranteed to be NULL terminated:
597923--
598−-- >>> :set -XMagicHash
599−-- >>> import qualified Streamly.Internal.Data.Array.Type as Array
600924-- >>> Array.toList $ Array.fromByteStr# "\1\2\3\0"#
601925-- [1,2,3]
602926--
927+-- Note that this should be evaluated strictly to ensure that we do not hold
928+-- the reference to the pointer in a lazy thunk.
603929fromByteStr# :: Addr# -> Array Word8
604−fromByteStr# addr = fromPureStream (D.fromByteStr# addr)
930+fromByteStr# addr = unsafePerformIO $ fmap unsafeFreeze (MA.fromByteStr# addr)
931+
932+-- XXX This should be monadic.
933+
934+-- | Note that this should be evaluated strictly to ensure that we do not hold
935+-- the reference to the pointer in a lazy thunk.
936+fromByteStr :: Ptr Word8 -> Array Word8
937+fromByteStr (Ptr addr#) = fromByteStr# addr#
938+
939+-- XXX implement fromChunks/fromChunkList instead?
940+
941+-- | Convert an array stream to an array. Note that this requires peak memory
942+-- that is double the size of the array stream.
943+--
944+{-# INLINE fromChunksK #-}
945+fromChunksK :: (MonadIO m, Unbox a) => StreamK m (Array a) -> m (Array a)
946+fromChunksK stream =
947+ -- We buffer the entire stream and then allocate the target array of the
948+ -- same size, thus requiring double the memory.
949+ fmap unsafeFreeze $ MA.fromChunksK $ fmap unsafeThaw stream
950+
951+{-# DEPRECATED fromArrayStreamK "Please use fromChunksK instead." #-}
952+fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (Array a) -> m (Array a)
953+fromArrayStreamK = fromChunksK
954+
955+-- | Given a stream of arrays, splice them all together to generate a single
956+-- array. The stream must be /finite/.
957+--
958+{-# INLINE fromChunks #-}
959+fromChunks :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
960+fromChunks s =
961+ -- XXX Check which implementation is better
962+ -- This may also require double the memory as we double the space every
963+ -- time, when copying the last array we may have reallocated almost double
964+ -- the space required before we right size it.
965+ fmap unsafeFreeze $ MA.fromChunksRealloced (fmap unsafeThaw s)
966+ -- fromChunkStreamK $ D.toStreamK s
605967
606968-------------------------------------------------------------------------------
607969-- Instances
@@ -633,12 +995,66 @@ instance Unbox a => IsList (Array a) where
633995 {-# INLINE toList #-}
634996 toList = toList
635997
636−-- XXX we are assuming that Unboxed equality means element equality. This may
637−-- or may not be correct? arrcmp is 40% faster compared to stream equality.
638−instance (Unbox a, Eq a) => Eq (Array a) where
998+-- | Byte compare two arrays. Compare the length of the arrays. If the length
999+-- is equal, compare the lexicographical ordering of two underlying byte arrays
1000+-- otherwise return the result of length comparison.
1001+--
1002+-- /Unsafe/: Note that the 'Unbox' instance of sum types with constructors of
1003+-- different sizes may leave some memory uninitialized which can make byte
1004+-- comparison unreliable.
1005+--
1006+-- /Pre-release/
1007+{-# INLINE byteCmp #-}
1008+byteCmp :: Array a -> Array a -> Ordering
1009+byteCmp arr1 arr2 =
1010+ -- unsafePerformIO?
1011+ unsafeInlineIO $! unsafeThaw arr1 `MA.byteCmp` unsafeThaw arr2
1012+
1013+-- | Byte equality of two arrays.
1014+--
1015+-- >>> byteEq arr1 arr2 = (==) EQ $ Array.byteCmp arr1 arr2
1016+--
1017+-- /Unsafe/: See 'byteCmp'.
1018+{-# INLINE byteEq #-}
1019+byteEq :: Array a -> Array a -> Bool
1020+byteEq arr1 arr2 = (==) EQ $ byteCmp arr1 arr2
1021+
1022+#define MK_EQ_INSTANCE(typ) \
1023+instance {-# OVERLAPPING #-} Eq (Array typ) where { \
1024+; {-# INLINE (==) #-} \
1025+; (==) = byteEq \
1026+}
1027+
1028+MK_EQ_INSTANCE(Char)
1029+MK_EQ_INSTANCE(Word8)
1030+MK_EQ_INSTANCE(Word16)
1031+MK_EQ_INSTANCE(Word32)
1032+
1033+-- XXX The Word64 default instance should be as fast because we are comparing
1034+-- 64-bit at a time.
1035+MK_EQ_INSTANCE(Word64)
1036+MK_EQ_INSTANCE(Int)
1037+MK_EQ_INSTANCE(Int8)
1038+MK_EQ_INSTANCE(Int16)
1039+MK_EQ_INSTANCE(Int32)
1040+
1041+-- XXX The Int64 default instance should be as fast.
1042+MK_EQ_INSTANCE(Int64)
1043+
1044+-- | If the type allows a byte-by-byte comparison this instance can be
1045+-- overlapped by a more specific instance that uses 'byteCmp'. Byte comparison
1046+-- can be significantly faster.
1047+--
1048+instance {-# OVERLAPPABLE #-} (Unbox a, Eq a) => Eq (Array a) where
6391049 {-# INLINE (==) #-}
6401050 arr1 == arr2 =
641− (==) EQ $ unsafeInlineIO $! unsafeThaw arr1 `MA.cmp` unsafeThaw arr2
1051+ -- Does unboxed byte equality mean element equality?
1052+ -- XXX This is incorrect for sum types, as we may have some
1053+ -- uninitialized memory in that case. If we always initialize the
1054+ -- unused memory to zero we can use this.
1055+ -- Byte comparison is 40% faster compared to stream equality.
1056+ -- (==) EQ $ unsafeInlineIO $! unsafeThaw arr1 `MA.cmp` unsafeThaw arr2
1057+ (toStreamD arr1 :: Stream Identity a) == toStreamD arr2
6421058
6431059instance (Unbox a, Ord a) => Ord (Array a) where
6441060 {-# INLINE compare #-}
@@ -680,7 +1096,7 @@ _toStreamD_ size Array{..} = D.Stream step arrStart
6801096 {-# INLINE_LATE step #-}
6811097 step _ p | p == arrEnd = return D.Stop
6821098 step _ p = liftIO $ do
683− x <- peekByteIndex p arrContents
1099+ x <- peekAt p arrContents
6841100 return $ D.Yield x (p + size)
6851101
6861102{-
@@ -703,15 +1119,29 @@ instance Foldable Array where
7031119-- Semigroup and Monoid
7041120-------------------------------------------------------------------------------
7051121
1122+-- XXX Deprecate and remove the Semigroup and Monoid instances because of
1123+-- potential misuse chances.
1124+
1125+-- | This should not be used for combining many or N arrays as it would copy
1126+-- the two arrays everytime to a new array. For coalescing multiple arrays use
1127+-- 'fromChunksK' instead.
7061128instance Unbox a => Semigroup (Array a) where
7071129 arr1 <> arr2 = unsafePerformIO $ splice arr1 arr2
7081130
1131+empty ::
1132+#ifdef DEVBUILD
1133+ Unbox a =>
1134+#endif
1135+ Array a
1136+empty = Array Unboxed.empty 0 0
1137+
1138+{-# DEPRECATED nil "Please use empty instead." #-}
7091139nil ::
7101140#ifdef DEVBUILD
7111141 Unbox a =>
7121142#endif
7131143 Array a
714−nil = Array Unboxed.nil 0 0
1144+nil = empty
7151145
7161146instance Unbox a => Monoid (Array a) where
7171147 mempty = nil
core/src/Streamly/Internal/Serialize/FromBytes.hs → core/src/Streamly/Internal/Data/Binary/Parser.hsrenamed+10−4
@@ -1,13 +1,19 @@
11-- |
2−-- Module : Streamly.Internal.Serialize.FromBytes
2+-- Module : Streamly.Internal.Data.Binary.Parser
33-- Copyright : (c) 2020 Composewell Technologies
44-- License : BSD-3-Clause
55-- Maintainer : streamly@composewell.com
66-- Portability : GHC
77--
88-- Decode Haskell data types from byte streams.
9+--
10+-- It would be inefficient to use this to compose parsers for general algebraic
11+-- data types. For general deserialization of ADTs please use the Serialize
12+-- type class instances. The fastest way to deserialize byte streams
13+-- representing Haskell data types is to write them to arrays and deserialize
14+-- the array using the Serialize type class.
915
10−module Streamly.Internal.Serialize.FromBytes
16+module Streamly.Internal.Data.Binary.Parser
1117 (
1218 -- * Type class
1319 FromBytes (..)
@@ -51,7 +57,7 @@ import Streamly.Internal.Data.Maybe.Strict (Maybe'(..))
5157import Streamly.Internal.Data.Tuple.Strict (Tuple' (..))
5258import qualified Streamly.Data.Array as A
5359import qualified Streamly.Internal.Data.Array as A
54− (unsafeIndex, castUnsafe)
60+ (getIndexUnsafe, castUnsafe)
5561import qualified Streamly.Internal.Data.Parser as PR
5662 (fromPure, either, satisfy, takeEQ)
5763import qualified Streamly.Internal.Data.Parser as PRD
@@ -382,7 +388,7 @@ charLatin1 = fmap (chr . fromIntegral) word8
382388{-# INLINE word64host #-}
383389word64host :: MonadIO m => Parser Word8 m Word64
384390word64host =
385− fmap (A.unsafeIndex 0 . A.castUnsafe) $ PR.takeEQ 8 (A.writeN 8)
391+ fmap (A.getIndexUnsafe 0 . A.castUnsafe) $ PR.takeEQ 8 (A.writeN 8)
386392
387393-------------------------------------------------------------------------------
388394-- Type class
core/src/Streamly/Internal/Serialize/ToBytes.hs → core/src/Streamly/Internal/Data/Binary/Stream.hsrenamed+12−2
@@ -1,13 +1,23 @@
11-- |
2−-- Module : Streamly.Internal.Serialize.ToBytes
2+-- Module : Streamly.Internal.Data.Binary.Stream
33-- Copyright : (c) 2022 Composewell Technologies
44-- License : BSD-3-Clause
55-- Maintainer : streamly@composewell.com
66-- Portability : GHC
77--
88-- Encode Haskell data types to byte streams.
9+--
10+-- The primary purpose of this module is to serialize primitive Haskell types
11+-- to streams for convenient byte by byte processing when such a need arises.
12+--
13+-- It would be inefficient to use this to build byte streams from algebraic
14+-- data types. For general serialization of ADTs please use the Serialize type
15+-- class instances. The fastest way to convert general Haskell types to byte
16+-- streams is to serialize them to an array and then stream the array.
17+
18+-- XXX remove unit, bool, ordering, and the type class as well
919
10−module Streamly.Internal.Serialize.ToBytes
20+module Streamly.Internal.Data.Binary.Stream
1121 (
1222 -- * Type class
1323 ToBytes (..)
core/src/Streamly/Internal/Data/Builder.hs+11−9
@@ -16,7 +16,9 @@ module Streamly.Internal.Data.Builder
1616 )
1717where
1818
19+#if !MIN_VERSION_base(4,18,0)
1920import Control.Applicative (liftA2)
21+#endif
2022
2123------------------------------------------------------------------------------
2224-- The Builder type
@@ -27,16 +29,16 @@ import Control.Applicative (liftA2)
2729-- or even a Fold. Unlike fold the step function is one-shot and not called in
2830-- a loop.
2931newtype Builder s m a =
30− Builder (s -> m (s, a))
32+ Builder (s -> m (a, s))
3133
3234-- | Maps a function on the output of the fold (the type @b@).
3335instance Functor m => Functor (Builder s m) where
3436 {-# INLINE fmap #-}
35− fmap f (Builder step1) = Builder (fmap (fmap f) . step1)
37+ fmap f (Builder step1) = Builder (fmap (\ (a, s) -> (f a, s)) . step1)
3638
3739{-# INLINE fromPure #-}
3840fromPure :: Applicative m => b -> Builder s m b
39−fromPure b = Builder (\s -> pure (s, b))
41+fromPure b = Builder (\s -> pure (b, s))
4042
4143-- | Chain the actions and zip the outputs.
4244{-# INLINE sequenceWith #-}
@@ -47,9 +49,9 @@ sequenceWith func (Builder stepL) (Builder stepR) = Builder step
4749 where
4850
4951 step s = do
50− (s1, x) <- stepL s
51− (s2, y) <- stepR s1
52− pure (s2, func x y)
52+ (x, s1) <- stepL s
53+ (y, s2) <- stepR s1
54+ pure (func x y, s2)
5355
5456instance Monad m => Applicative (Builder a m) where
5557 {-# INLINE pure #-}
@@ -74,7 +76,7 @@ instance Monad m => Monad (Builder a m) where
7476 where
7577
7678 step s = do
77− (s1, x) <- stepL s
79+ (x, s1) <- stepL s
7880 let Builder stepR = f x
79− (s2, y) <- stepR s1
80− pure (s2, y)
81+ (y, s2) <- stepR s1
82+ pure (y, s2)
core/src/Streamly/Internal/Data/Fold.hs+9−2574
@@ -16,2583 +16,18 @@ module Streamly.Internal.Data.Fold
1616 -- * Imports
1717 -- $setup
1818
19− -- * Fold Type
20− Step (..)
21− , Fold (..)
22− , Tee (..)
23−
24− -- * Constructors
25− -- | Which constructor to use?
26− --
27− -- * @foldl*@: If the fold never terminates i.e. does not use the 'Done'
28− -- constructor otherwise use the @foldt*@ variants.
29− -- * @*M@: Use the @M@ suffix variants if any of the step, initial, or
30− -- extract function is monadic, otherwise use the pure variants.
31− --
32− , foldl'
33− , foldlM'
34− , foldl1'
35− , foldlM1'
36− , foldt'
37− , foldtM'
38− , foldr'
39− , foldrM'
40−
41− -- * Mappers
42− -- | Monadic functions useful with mapM/lmapM on folds or streams.
43− , tracing
44− , trace
45−
46− -- * Folds
47−
48− -- ** Accumulators
49− -- *** Semigroups and Monoids
50− , sconcat
51− , mconcat
52− , foldMap
53− , foldMapM
54−
55− -- *** Reducers
56− , drain
57− , drainMapM
58− , the
59− , length
60− , lengthGeneric
61− , mean
62− , rollingHash
63− , defaultSalt
64− , rollingHashWithSalt
65− , rollingHashFirstN
66− -- , rollingHashLastN
67−
68− -- *** Saturating Reducers
69− -- | 'product' terminates if it becomes 0. Other folds can theoretically
70− -- saturate on bounded types, and therefore terminate, however, they will
71− -- run forever on unbounded types like Integer/Double.
72− , sum
73− , product
74− , maximumBy
75− , maximum
76− , minimumBy
77− , minimum
78−
79− -- *** Collectors
80− -- | Avoid using these folds in scalable or performance critical
81− -- applications, they buffer all the input in GC memory which can be
82− -- detrimental to performance if the input is large.
83− , toList
84− , toListRev
85− -- $toListRev
86− , toStream
87− , toStreamRev
88− , toStreamK
89− , toStreamKRev
90− , topBy
91− , top
92− , bottomBy
93− , bottom
94−
95− -- *** Scanners
96− -- | Stateful transformation of the elements. Useful in combination with
97− -- the 'scanMaybe' combinator. For scanners the result of the fold is
98− -- usually a transformation of the current element rather than an
99− -- aggregation of all elements till now.
100− , latest
101− -- , nthLast -- using Ring array
102− , indexingWith
103− , indexing
104− , indexingRev
105− , rollingMapM
106−
107− -- *** Filters
108− -- | Useful in combination with the 'scanMaybe' combinator.
109− , filtering
110− , deleteBy
111− , uniqBy
112− , uniq
113− , repeated
114− , findIndices
115− , elemIndices
116−
117− -- ** Terminating Folds
118− -- *** Empty folds
119− -- | Folds that return a result without consuming any input.
120− , fromPure
121− , fromEffect
122− , fromRefold
123−
124− -- *** Singleton folds
125− -- | Folds that terminate after consuming exactly one input element. All
126− -- these can be implemented in terms of the 'maybe' fold.
127− , one
128− , null -- XXX not very useful and could be problematic, remove it?
129− , satisfy
130− , maybe
131−
132− -- *** Multi folds
133− -- | Terminate after consuming one or more elements.
134− , drainN
135− -- , lastN
136− -- , (!!)
137− , indexGeneric
138− , index
139− , findM
140− , find
141− , lookup
142− , findIndex
143− , elemIndex
144− , elem
145− , notElem
146− , all
147− , any
148− , and
149− , or
150−
151− -- ** Trimmers
152− -- | Useful in combination with the 'scanMaybe' combinator.
153− , taking
154− , dropping
155− , takingEndByM
156− , takingEndBy
157− , takingEndByM_
158− , takingEndBy_
159− , droppingWhileM
160− , droppingWhile
161− , prune
162−
163− -- * Running A Fold
164− , drive
165− -- , breakStream
166−
167− -- * Building Incrementally
168− , extractM
169− , reduce
170− , close
171− , isClosed
172− , snoc
173− , snocl
174− , snocM
175− , snoclM
176−
177− , addOne
178− , addStream
179−
180− -- * Combinators
181− -- ** Utilities
182− , with
183−
184− -- ** Transforming the Monad
185− , morphInner
186− , generalizeInner
187−
188− -- ** Mapping on output
189− , rmapM
190−
191− -- ** Mapping on Input
192− , transform
193− , lmap
194− --, lsequence
195− , lmapM
196−
197− -- ** Sliding Window
198− , slide2
199−
200− -- ** Scanning Input
201− , scan
202− , scanMany
203− , postscan
204− , indexed
205−
206− -- ** Zipping Input
207− , zipStreamWithM
208− , zipStream
209−
210− -- ** Filtering Input
211− , catMaybes
212− , mapMaybeM
213− , mapMaybe
214− , scanMaybe
215− , filter
216− , filterM
217− , sampleFromthen
218−
219− -- Either streams
220− , catLefts
221− , catRights
222− , catEithers
223−
224− {-
225− -- ** Insertion
226− -- | Insertion adds more elements to the stream.
227−
228− , insertBy
229− , intersperseM
230−
231− -- ** Reordering
232− , reverse
233− -}
234−
235− -- ** Trimming
236− , take
237−
238− -- By elements
239− , takeEndBy
240− , takeEndBy_
241− , takeEndBySeq
242− , takeEndBySeq_
243− {-
244− , drop
245− , dropWhile
246− , dropWhileM
247− -}
248−
249− -- ** Serial Append
250− , splitWith
251− , split_
252− -- , tail
253− -- , init
254− , splitAt -- spanN
255− -- , splitIn -- sessionN
256−
257− -- ** Parallel Distribution
258− , teeWith
259− , tee
260− , teeWithFst
261− , teeWithMin
262− , distribute
263− -- , distributeFst
264− -- , distributeMin
265−
266− -- ** Unzipping
267− , unzip
268− -- These two can be expressed using lmap/lmapM and unzip
269− , unzipWith
270− , unzipWithM
271− , unzipWithFstM
272− , unzipWithMinM
273−
274− -- ** Parallel Alternative
275− , shortest
276− , longest
277−
278− -- ** Partitioning
279− , partitionByM
280− , partitionByFstM
281− , partitionByMinM
282− , partitionBy
283− , partition
284−
285− -- ** Splitting
286− , many
287− , manyPost
288− , groupsOf
289− , chunksBetween
290− , refoldMany
291− , refoldMany1
292− , intersperseWithQuotes
293−
294− -- ** Nesting
295− , unfoldMany
296− , concatSequence
297− , concatMap
298− , duplicate
299− , refold
300−
301− -- * Deprecated
302− , foldr
303− , drainBy
304− , last
305− , head
306− , sequence
307− , mapM
308− , variance
309− , stdDev
310− , serialWith
19+ module Streamly.Internal.Data.Fold.Type
20+ , module Streamly.Internal.Data.Fold.Tee
21+ , module Streamly.Internal.Data.Fold.Combinators
22+ , module Streamly.Internal.Data.Fold.Container
23+ , module Streamly.Internal.Data.Fold.Window
31124 )
31225where
31326
314−#include "inline.hs"
315−#include "ArrayMacros.h"
316−
317−import Control.Monad (void)
318−import Control.Monad.IO.Class (MonadIO(..))
319−import Data.Bifunctor (first)
320−import Data.Bits (shiftL, shiftR, (.|.), (.&.))
321−import Data.Either (isLeft, isRight, fromLeft, fromRight)
322−import Data.Int (Int64)
323−import Data.Proxy (Proxy(..))
324−import Data.Word (Word32)
325−import Foreign.Storable (Storable, peek)
326−import Streamly.Internal.Data.MutArray.Type (MutArray(..))
327−import Streamly.Internal.Data.Maybe.Strict (Maybe'(..), toMaybe)
328−import Streamly.Internal.Data.Pipe.Type (Pipe (..), PipeState(..))
329−import Streamly.Internal.Data.Unbox (Unbox, sizeOf)
330−import Streamly.Internal.Data.Unfold.Type (Unfold(..))
331−import Streamly.Internal.Data.Tuple.Strict (Tuple'(..), Tuple3'(..))
332−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
333−
334−import qualified Prelude
335−import qualified Streamly.Internal.Data.MutArray.Type as MA
336−import qualified Streamly.Internal.Data.Array.Type as Array
337−import qualified Streamly.Internal.Data.Fold.Window as FoldW
338−import qualified Streamly.Internal.Data.Pipe.Type as Pipe
339−import qualified Streamly.Internal.Data.Ring.Unboxed as Ring
340−import qualified Streamly.Internal.Data.Stream.StreamD.Type as StreamD
341−
342−import Prelude hiding
343− ( Foldable(..), filter, drop, dropWhile, take, takeWhile, zipWith
344− , map, mapM_, sequence, all, any
345− , notElem, head, last, tail
346− , reverse, iterate, init, and, or, lookup, (!!)
347− , scanl, scanl1, replicate, concatMap, mconcat, unzip
348− , span, splitAt, break, mapM, zip, maybe)
349−import Streamly.Internal.Data.Fold.Type
27+import Streamly.Internal.Data.Fold.Combinators
28+import Streamly.Internal.Data.Fold.Container
35029import Streamly.Internal.Data.Fold.Tee
30+import Streamly.Internal.Data.Fold.Type
31+import Streamly.Internal.Data.Fold.Window
35132
35233#include "DocTestDataFold.hs"
353−
354−------------------------------------------------------------------------------
355−-- Running
356−------------------------------------------------------------------------------
357−
358−-- | Drive a fold using the supplied 'Stream', reducing the resulting
359−-- expression strictly at each step.
360−--
361−-- Definition:
362−--
363−-- >>> drive = flip Stream.fold
364−--
365−-- Example:
366−--
367−-- >>> Fold.drive (Stream.enumerateFromTo 1 100) Fold.sum
368−-- 5050
369−--
370−{-# INLINE drive #-}
371−drive :: Monad m => Stream m a -> Fold m a b -> m b
372−drive = flip StreamD.fold
373−
374−{-
375−-- | Like 'drive' but also returns the remaining stream. The resulting stream
376−-- would be 'Stream.nil' if the stream finished before the fold.
377−--
378−-- Definition:
379−--
380−-- >>> breakStream = flip Stream.foldBreak
381−--
382−-- /CPS/
383−--
384−{-# INLINE breakStreamK #-}
385−breakStreamK :: Monad m => StreamK m a -> Fold m a b -> m (b, StreamK m a)
386−breakStreamK strm fl = fmap f $ K.foldBreak fl (Stream.toStreamK strm)
387−
388− where
389−
390− f (b, str) = (b, Stream.fromStreamK str)
391−-}
392−
393−-- | Append a stream to a fold to build the fold accumulator incrementally. We
394−-- can repeatedly call 'addStream' on the same fold to continue building the
395−-- fold and finally use 'drive' to finish the fold and extract the result. Also
396−-- see the 'Streamly.Data.Fold.addOne' operation which is a singleton version
397−-- of 'addStream'.
398−--
399−-- Definitions:
400−--
401−-- >>> addStream stream = Fold.drive stream . Fold.duplicate
402−--
403−-- Example, build a list incrementally:
404−--
405−-- >>> :{
406−-- pure (Fold.toList :: Fold IO Int [Int])
407−-- >>= Fold.addOne 1
408−-- >>= Fold.addStream (Stream.enumerateFromTo 2 4)
409−-- >>= Fold.drive Stream.nil
410−-- >>= print
411−-- :}
412−-- [1,2,3,4]
413−--
414−-- This can be used as an O(n) list append compared to the O(n^2) @++@ when
415−-- used for incrementally building a list.
416−--
417−-- Example, build a stream incrementally:
418−--
419−-- >>> :{
420−-- pure (Fold.toStream :: Fold IO Int (Stream Identity Int))
421−-- >>= Fold.addOne 1
422−-- >>= Fold.addStream (Stream.enumerateFromTo 2 4)
423−-- >>= Fold.drive Stream.nil
424−-- >>= print
425−-- :}
426−-- fromList [1,2,3,4]
427−--
428−-- This can be used as an O(n) stream append compared to the O(n^2) @<>@ when
429−-- used for incrementally building a stream.
430−--
431−-- Example, build an array incrementally:
432−--
433−-- >>> :{
434−-- pure (Array.write :: Fold IO Int (Array Int))
435−-- >>= Fold.addOne 1
436−-- >>= Fold.addStream (Stream.enumerateFromTo 2 4)
437−-- >>= Fold.drive Stream.nil
438−-- >>= print
439−-- :}
440−-- fromList [1,2,3,4]
441−--
442−-- Example, build an array stream incrementally:
443−--
444−-- >>> :{
445−-- let f :: Fold IO Int (Stream Identity (Array Int))
446−-- f = Fold.groupsOf 2 (Array.writeN 3) Fold.toStream
447−-- in pure f
448−-- >>= Fold.addOne 1
449−-- >>= Fold.addStream (Stream.enumerateFromTo 2 4)
450−-- >>= Fold.drive Stream.nil
451−-- >>= print
452−-- :}
453−-- fromList [fromList [1,2],fromList [3,4]]
454−--
455−addStream :: Monad m => Stream m a -> Fold m a b -> m (Fold m a b)
456−addStream stream = drive stream . duplicate
457−
458−------------------------------------------------------------------------------
459−-- Transformations on fold inputs
460−------------------------------------------------------------------------------
461−
462−-- | Flatten the monadic output of a fold to pure output.
463−--
464−{-# DEPRECATED sequence "Use \"rmapM id\" instead" #-}
465−{-# INLINE sequence #-}
466−sequence :: Monad m => Fold m a (m b) -> Fold m a b
467−sequence = rmapM id
468−
469−-- | Map a monadic function on the output of a fold.
470−--
471−{-# DEPRECATED mapM "Use rmapM instead" #-}
472−{-# INLINE mapM #-}
473−mapM :: Monad m => (b -> m c) -> Fold m a b -> Fold m a c
474−mapM = rmapM
475−
476−-- |
477−-- >>> mapMaybeM f = Fold.lmapM f . Fold.catMaybes
478−--
479−{-# INLINE mapMaybeM #-}
480−mapMaybeM :: Monad m => (a -> m (Maybe b)) -> Fold m b r -> Fold m a r
481−mapMaybeM f = lmapM f . catMaybes
482−
483−-- | @mapMaybe f fold@ maps a 'Maybe' returning function @f@ on the input of
484−-- the fold, filters out 'Nothing' elements, and return the values extracted
485−-- from 'Just'.
486−--
487−-- >>> mapMaybe f = Fold.lmap f . Fold.catMaybes
488−-- >>> mapMaybe f = Fold.mapMaybeM (return . f)
489−--
490−-- >>> f x = if even x then Just x else Nothing
491−-- >>> fld = Fold.mapMaybe f Fold.toList
492−-- >>> Stream.fold fld (Stream.enumerateFromTo 1 10)
493−-- [2,4,6,8,10]
494−--
495−{-# INLINE mapMaybe #-}
496−mapMaybe :: Monad m => (a -> Maybe b) -> Fold m b r -> Fold m a r
497−mapMaybe f = lmap f . catMaybes
498−
499−------------------------------------------------------------------------------
500−-- Transformations on fold inputs
501−------------------------------------------------------------------------------
502−
503−-- | Apply a monadic function on the input and return the input.
504−--
505−-- >>> Stream.fold (Fold.lmapM (Fold.tracing print) Fold.drain) $ (Stream.enumerateFromTo (1 :: Int) 2)
506−-- 1
507−-- 2
508−--
509−-- /Pre-release/
510−--
511−{-# INLINE tracing #-}
512−tracing :: Monad m => (a -> m b) -> (a -> m a)
513−tracing f x = void (f x) >> return x
514−
515−-- | Apply a monadic function to each element flowing through and discard the
516−-- results.
517−--
518−-- >>> Stream.fold (Fold.trace print Fold.drain) $ (Stream.enumerateFromTo (1 :: Int) 2)
519−-- 1
520−-- 2
521−--
522−-- >>> trace f = Fold.lmapM (Fold.tracing f)
523−--
524−-- /Pre-release/
525−{-# INLINE trace #-}
526−trace :: Monad m => (a -> m b) -> Fold m a r -> Fold m a r
527−trace f = lmapM (tracing f)
528−
529−-- rename to lpipe?
530−--
531−-- | Apply a transformation on a 'Fold' using a 'Pipe'.
532−--
533−-- /Pre-release/
534−{-# INLINE transform #-}
535−transform :: Monad m => Pipe m a b -> Fold m b c -> Fold m a c
536−transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract) =
537− Fold step initial extract
538−
539− where
540−
541− initial = first (Tuple' pinitial) <$> finitial
542−
543− step (Tuple' ps fs) x = do
544− r <- pstep1 ps x
545− go fs r
546−
547− where
548−
549− -- XXX use SPEC?
550− go acc (Pipe.Yield b (Consume ps')) = do
551− acc' <- fstep acc b
552− return
553− $ case acc' of
554− Partial s -> Partial $ Tuple' ps' s
555− Done b2 -> Done b2
556− go acc (Pipe.Yield b (Produce ps')) = do
557− acc' <- fstep acc b
558− r <- pstep2 ps'
559− case acc' of
560− Partial s -> go s r
561− Done b2 -> return $ Done b2
562− go acc (Pipe.Continue (Consume ps')) =
563− return $ Partial $ Tuple' ps' acc
564− go acc (Pipe.Continue (Produce ps')) = do
565− r <- pstep2 ps'
566− go acc r
567−
568− extract (Tuple' _ fs) = fextract fs
569−
570−{-# INLINE scanWith #-}
571−scanWith :: Monad m => Bool -> Fold m a b -> Fold m b c -> Fold m a c
572−scanWith isMany (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
573− Fold step initial extract
574−
575− where
576−
577− {-# INLINE runStep #-}
578− runStep actionL sR = do
579− rL <- actionL
580− case rL of
581− Done bL -> do
582− rR <- stepR sR bL
583− case rR of
584− Partial sR1 ->
585− if isMany
586− then runStep initialL sR1
587− else Done <$> extractR sR1
588− Done bR -> return $ Done bR
589− Partial sL -> do
590− !b <- extractL sL
591− rR <- stepR sR b
592− return
593− $ case rR of
594− Partial sR1 -> Partial (sL, sR1)
595− Done bR -> Done bR
596−
597− initial = do
598− r <- initialR
599− case r of
600− Partial sR -> runStep initialL sR
601− Done b -> return $ Done b
602−
603− step (sL, sR) x = runStep (stepL sL x) sR
604−
605− extract = extractR . snd
606−
607−-- | Scan the input of a 'Fold' to change it in a stateful manner using another
608−-- 'Fold'. The scan stops as soon as the fold terminates.
609−--
610−-- /Pre-release/
611−{-# INLINE scan #-}
612−scan :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
613−scan = scanWith False
614−
615−-- XXX This does not fuse beacuse of the recursive step. Need to investigate.
616−--
617−-- | Scan the input of a 'Fold' to change it in a stateful manner using another
618−-- 'Fold'. The scan restarts with a fresh state if the fold terminates.
619−--
620−-- /Pre-release/
621−{-# INLINE scanMany #-}
622−scanMany :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
623−scanMany = scanWith True
624−
625−------------------------------------------------------------------------------
626−-- Filters
627−------------------------------------------------------------------------------
628−
629−-- | Returns the latest element omitting the first occurrence that satisfies
630−-- the given equality predicate.
631−--
632−-- Example:
633−--
634−-- >>> input = Stream.fromList [1,3,3,5]
635−-- >>> Stream.fold Fold.toList $ Stream.scanMaybe (Fold.deleteBy (==) 3) input
636−-- [1,3,5]
637−--
638−{-# INLINE_NORMAL deleteBy #-}
639−deleteBy :: Monad m => (a -> a -> Bool) -> a -> Fold m a (Maybe a)
640−deleteBy eq x0 = fmap extract $ foldl' step (Tuple' False Nothing)
641−
642− where
643−
644− step (Tuple' False _) x =
645− if eq x x0
646− then Tuple' True Nothing
647− else Tuple' False (Just x)
648− step (Tuple' True _) x = Tuple' True (Just x)
649−
650− extract (Tuple' _ x) = x
651−
652−-- | Provide a sliding window of length 2 elements.
653−--
654−-- See "Streamly.Internal.Data.Fold.Window".
655−--
656−{-# INLINE slide2 #-}
657−slide2 :: Monad m => Fold m (a, Maybe a) b -> Fold m a b
658−slide2 (Fold step1 initial1 extract1) = Fold step initial extract
659−
660− where
661−
662− initial =
663− first (Tuple' Nothing) <$> initial1
664−
665− step (Tuple' prev s) cur =
666− first (Tuple' (Just cur)) <$> step1 s (cur, prev)
667−
668− extract (Tuple' _ s) = extract1 s
669−
670−-- | Return the latest unique element using the supplied comparison function.
671−-- Returns 'Nothing' if the current element is same as the last element
672−-- otherwise returns 'Just'.
673−--
674−-- Example, strip duplicate path separators:
675−--
676−-- >>> input = Stream.fromList "//a//b"
677−-- >>> f x y = x == '/' && y == '/'
678−-- >>> Stream.fold Fold.toList $ Stream.scanMaybe (Fold.uniqBy f) input
679−-- "/a/b"
680−--
681−-- Space: @O(1)@
682−--
683−-- /Pre-release/
684−--
685−{-# INLINE uniqBy #-}
686−uniqBy :: Monad m => (a -> a -> Bool) -> Fold m a (Maybe a)
687−uniqBy eq = rollingMap f
688−
689− where
690−
691− f pre curr =
692− case pre of
693− Nothing -> Just curr
694− Just x -> if x `eq` curr then Nothing else Just curr
695−
696−-- | See 'uniqBy'.
697−--
698−-- Definition:
699−--
700−-- >>> uniq = Fold.uniqBy (==)
701−--
702−{-# INLINE uniq #-}
703−uniq :: (Monad m, Eq a) => Fold m a (Maybe a)
704−uniq = uniqBy (==)
705−
706−-- | Strip all leading and trailing occurrences of an element passing a
707−-- predicate and make all other consecutive occurrences uniq.
708−--
709−-- >> prune p = Stream.dropWhileAround p $ Stream.uniqBy (x y -> p x && p y)
710−--
711−-- @
712−-- > Stream.prune isSpace (Stream.fromList " hello world! ")
713−-- "hello world!"
714−--
715−-- @
716−--
717−-- Space: @O(1)@
718−--
719−-- /Unimplemented/
720−{-# INLINE prune #-}
721−prune ::
722− -- (Monad m, Eq a) =>
723− (a -> Bool) -> Fold m a (Maybe a)
724−prune = error "Not implemented yet!"
725−
726−-- | Emit only repeated elements, once.
727−--
728−-- /Unimplemented/
729−repeated :: -- (Monad m, Eq a) =>
730− Fold m a (Maybe a)
731−repeated = error "Not implemented yet!"
732−
733−------------------------------------------------------------------------------
734−-- Left folds
735−------------------------------------------------------------------------------
736−
737−------------------------------------------------------------------------------
738−-- Run Effects
739−------------------------------------------------------------------------------
740−
741−-- |
742−-- Definitions:
743−--
744−-- >>> drainMapM f = Fold.lmapM f Fold.drain
745−-- >>> drainMapM f = Fold.foldMapM (void . f)
746−--
747−-- Drain all input after passing it through a monadic function. This is the
748−-- dual of mapM_ on stream producers.
749−--
750−{-# INLINE drainMapM #-}
751−drainMapM :: Monad m => (a -> m b) -> Fold m a ()
752−drainMapM f = lmapM f drain
753−
754−{-# DEPRECATED drainBy "Please use 'drainMapM' instead." #-}
755−{-# INLINE drainBy #-}
756−drainBy :: Monad m => (a -> m b) -> Fold m a ()
757−drainBy = drainMapM
758−
759−-- | Returns the latest element of the input stream, if any.
760−--
761−-- >>> latest = Fold.foldl1' (\_ x -> x)
762−-- >>> latest = fmap getLast $ Fold.foldMap (Last . Just)
763−--
764−{-# INLINE latest #-}
765−latest :: Monad m => Fold m a (Maybe a)
766−latest = foldl1' (\_ x -> x)
767−
768−{-# DEPRECATED last "Please use 'latest' instead." #-}
769−{-# INLINE last #-}
770−last :: Monad m => Fold m a (Maybe a)
771−last = latest
772−
773−-- | Terminates with 'Nothing' as soon as it finds an element different than
774−-- the previous one, returns 'the' element if the entire input consists of the
775−-- same element.
776−--
777−{-# INLINE the #-}
778−the :: (Monad m, Eq a) => Fold m a (Maybe a)
779−the = foldt' step initial id
780−
781− where
782−
783− initial = Partial Nothing
784−
785− step Nothing x = Partial (Just x)
786− step old@(Just x0) x =
787− if x0 == x
788− then Partial old
789− else Done Nothing
790−
791−------------------------------------------------------------------------------
792−-- To Summary
793−------------------------------------------------------------------------------
794−
795−-- | Like 'length', except with a more general 'Num' return value
796−--
797−-- Definition:
798−--
799−-- >>> lengthGeneric = fmap getSum $ Fold.foldMap (Sum . const 1)
800−-- >>> lengthGeneric = Fold.foldl' (\n _ -> n + 1) 0
801−--
802−-- /Pre-release/
803−{-# INLINE lengthGeneric #-}
804−lengthGeneric :: (Monad m, Num b) => Fold m a b
805−lengthGeneric = foldl' (\n _ -> n + 1) 0
806−
807−-- | Determine the length of the input stream.
808−--
809−-- Definition:
810−--
811−-- >>> length = Fold.lengthGeneric
812−-- >>> length = fmap getSum $ Fold.foldMap (Sum . const 1)
813−--
814−{-# INLINE length #-}
815−length :: Monad m => Fold m a Int
816−length = lengthGeneric
817−
818−
819−-- | Determine the sum of all elements of a stream of numbers. Returns additive
820−-- identity (@0@) when the stream is empty. Note that this is not numerically
821−-- stable for floating point numbers.
822−--
823−-- >>> sum = FoldW.cumulative FoldW.sum
824−--
825−-- Same as following but numerically stable:
826−--
827−-- >>> sum = Fold.foldl' (+) 0
828−-- >>> sum = fmap Data.Monoid.getSum $ Fold.foldMap Data.Monoid.Sum
829−--
830−{-# INLINE sum #-}
831−sum :: (Monad m, Num a) => Fold m a a
832−sum = FoldW.cumulative FoldW.sum
833−
834−-- | Determine the product of all elements of a stream of numbers. Returns
835−-- multiplicative identity (@1@) when the stream is empty. The fold terminates
836−-- when it encounters (@0@) in its input.
837−--
838−-- Same as the following but terminates on multiplication by @0@:
839−--
840−-- >>> product = fmap Data.Monoid.getProduct $ Fold.foldMap Data.Monoid.Product
841−--
842−{-# INLINE product #-}
843−product :: (Monad m, Num a, Eq a) => Fold m a a
844−product = foldt' step (Partial 1) id
845−
846− where
847−
848− step x a =
849− if a == 0
850− then Done 0
851− else Partial $ x * a
852−
853−------------------------------------------------------------------------------
854−-- To Summary (Maybe)
855−------------------------------------------------------------------------------
856−
857−-- | Determine the maximum element in a stream using the supplied comparison
858−-- function.
859−--
860−{-# INLINE maximumBy #-}
861−maximumBy :: Monad m => (a -> a -> Ordering) -> Fold m a (Maybe a)
862−maximumBy cmp = foldl1' max'
863−
864− where
865−
866− max' x y =
867− case cmp x y of
868− GT -> x
869− _ -> y
870−
871−-- | Determine the maximum element in a stream.
872−--
873−-- Definitions:
874−--
875−-- >>> maximum = Fold.maximumBy compare
876−-- >>> maximum = Fold.foldl1' max
877−--
878−-- Same as the following but without a default maximum. The 'Max' Monoid uses
879−-- the 'minBound' as the default maximum:
880−--
881−-- >>> maximum = fmap Data.Semigroup.getMax $ Fold.foldMap Data.Semigroup.Max
882−--
883−{-# INLINE maximum #-}
884−maximum :: (Monad m, Ord a) => Fold m a (Maybe a)
885−maximum = foldl1' max
886−
887−-- | Computes the minimum element with respect to the given comparison function
888−--
889−{-# INLINE minimumBy #-}
890−minimumBy :: Monad m => (a -> a -> Ordering) -> Fold m a (Maybe a)
891−minimumBy cmp = foldl1' min'
892−
893− where
894−
895− min' x y =
896− case cmp x y of
897− GT -> y
898− _ -> x
899−
900−-- | Determine the minimum element in a stream using the supplied comparison
901−-- function.
902−--
903−-- Definitions:
904−--
905−-- >>> minimum = Fold.minimumBy compare
906−-- >>> minimum = Fold.foldl1' min
907−--
908−-- Same as the following but without a default minimum. The 'Min' Monoid uses the
909−-- 'maxBound' as the default maximum:
910−--
911−-- >>> maximum = fmap Data.Semigroup.getMin $ Fold.foldMap Data.Semigroup.Min
912−--
913−{-# INLINE minimum #-}
914−minimum :: (Monad m, Ord a) => Fold m a (Maybe a)
915−minimum = foldl1' min
916−
917−------------------------------------------------------------------------------
918−-- To Summary (Statistical)
919−------------------------------------------------------------------------------
920−
921−-- | Compute a numerically stable arithmetic mean of all elements in the input
922−-- stream.
923−--
924−{-# INLINE mean #-}
925−mean :: (Monad m, Fractional a) => Fold m a a
926−mean = fmap done $ foldl' step begin
927−
928− where
929−
930− begin = Tuple' 0 0
931−
932− step (Tuple' x n) y =
933− let n1 = n + 1
934− in Tuple' (x + (y - x) / n1) n1
935−
936− done (Tuple' x _) = x
937−
938−-- | Compute a numerically stable (population) variance over all elements in
939−-- the input stream.
940−--
941−{-# DEPRECATED variance "Use the streamly-statistics package instead" #-}
942−{-# INLINE variance #-}
943−variance :: (Monad m, Fractional a) => Fold m a a
944−variance = fmap done $ foldl' step begin
945−
946− where
947−
948− begin = Tuple3' 0 0 0
949−
950− step (Tuple3' n mean_ m2) x = Tuple3' n' mean' m2'
951−
952− where
953−
954− n' = n + 1
955− mean' = (n * mean_ + x) / (n + 1)
956− delta = x - mean_
957− m2' = m2 + delta * delta * n / (n + 1)
958−
959− done (Tuple3' n _ m2) = m2 / n
960−
961−-- | Compute a numerically stable (population) standard deviation over all
962−-- elements in the input stream.
963−--
964−{-# DEPRECATED stdDev "Use the streamly-statistics package instead" #-}
965−{-# INLINE stdDev #-}
966−stdDev :: (Monad m, Floating a) => Fold m a a
967−stdDev = sqrt <$> variance
968−
969−-- | Compute an 'Int' sized polynomial rolling hash
970−--
971−-- > H = salt * k ^ n + c1 * k ^ (n - 1) + c2 * k ^ (n - 2) + ... + cn * k ^ 0
972−--
973−-- Where @c1@, @c2@, @cn@ are the elements in the input stream and @k@ is a
974−-- constant.
975−--
976−-- This hash is often used in Rabin-Karp string search algorithm.
977−--
978−-- See https://en.wikipedia.org/wiki/Rolling_hash
979−--
980−{-# INLINE rollingHashWithSalt #-}
981−rollingHashWithSalt :: (Monad m, Enum a) => Int64 -> Fold m a Int64
982−rollingHashWithSalt = foldl' step
983−
984− where
985−
986− k = 2891336453 :: Int64
987−
988− step cksum a = cksum * k + fromIntegral (fromEnum a)
989−
990−-- | A default salt used in the implementation of 'rollingHash'.
991−{-# INLINE defaultSalt #-}
992−defaultSalt :: Int64
993−defaultSalt = -2578643520546668380
994−
995−-- | Compute an 'Int' sized polynomial rolling hash of a stream.
996−--
997−-- >>> rollingHash = Fold.rollingHashWithSalt Fold.defaultSalt
998−--
999−{-# INLINE rollingHash #-}
1000−rollingHash :: (Monad m, Enum a) => Fold m a Int64
1001−rollingHash = rollingHashWithSalt defaultSalt
1002−
1003−-- | Compute an 'Int' sized polynomial rolling hash of the first n elements of
1004−-- a stream.
1005−--
1006−-- >>> rollingHashFirstN n = Fold.take n Fold.rollingHash
1007−--
1008−-- /Pre-release/
1009−{-# INLINE rollingHashFirstN #-}
1010−rollingHashFirstN :: (Monad m, Enum a) => Int -> Fold m a Int64
1011−rollingHashFirstN n = take n rollingHash
1012−
1013−-- XXX Compare this with the implementation in Fold.Window, preferrably use the
1014−-- latter if performance is good.
1015−
1016−-- | Apply a function on every two successive elements of a stream. The first
1017−-- argument of the map function is the previous element and the second argument
1018−-- is the current element. When processing the very first element in the
1019−-- stream, the previous element is 'Nothing'.
1020−--
1021−-- /Pre-release/
1022−--
1023−{-# INLINE rollingMapM #-}
1024−rollingMapM :: Monad m => (Maybe a -> a -> m b) -> Fold m a b
1025−rollingMapM f = Fold step initial extract
1026−
1027− where
1028−
1029− -- XXX We need just a postscan. We do not need an initial result here.
1030− -- Or we can supply a default initial result as an argument to rollingMapM.
1031− initial = return $ Partial (Nothing, error "Empty stream")
1032−
1033− step (prev, _) cur = do
1034− x <- f prev cur
1035− return $ Partial (Just cur, x)
1036−
1037− extract = return . snd
1038−
1039−-- |
1040−-- >>> rollingMap f = Fold.rollingMapM (\x y -> return $ f x y)
1041−--
1042−{-# INLINE rollingMap #-}
1043−rollingMap :: Monad m => (Maybe a -> a -> b) -> Fold m a b
1044−rollingMap f = rollingMapM (\x y -> return $ f x y)
1045−
1046−------------------------------------------------------------------------------
1047−-- Monoidal left folds
1048−------------------------------------------------------------------------------
1049−
1050−-- | Semigroup concat. Append the elements of an input stream to a provided
1051−-- starting value.
1052−--
1053−-- Definition:
1054−--
1055−-- >>> sconcat = Fold.foldl' (<>)
1056−--
1057−-- >>> semigroups = fmap Data.Monoid.Sum $ Stream.enumerateFromTo 1 10
1058−-- >>> Stream.fold (Fold.sconcat 10) semigroups
1059−-- Sum {getSum = 65}
1060−--
1061−{-# INLINE sconcat #-}
1062−sconcat :: (Monad m, Semigroup a) => a -> Fold m a a
1063−sconcat = foldl' (<>)
1064−
1065−-- | Monoid concat. Fold an input stream consisting of monoidal elements using
1066−-- 'mappend' and 'mempty'.
1067−--
1068−-- Definition:
1069−--
1070−-- >>> mconcat = Fold.sconcat mempty
1071−--
1072−-- >>> monoids = fmap Data.Monoid.Sum $ Stream.enumerateFromTo 1 10
1073−-- >>> Stream.fold Fold.mconcat monoids
1074−-- Sum {getSum = 55}
1075−--
1076−{-# INLINE mconcat #-}
1077−mconcat ::
1078− ( Monad m
1079− , Monoid a) => Fold m a a
1080−mconcat = sconcat mempty
1081−
1082−-- |
1083−-- Definition:
1084−--
1085−-- >>> foldMap f = Fold.lmap f Fold.mconcat
1086−--
1087−-- Make a fold from a pure function that folds the output of the function
1088−-- using 'mappend' and 'mempty'.
1089−--
1090−-- >>> sum = Fold.foldMap Data.Monoid.Sum
1091−-- >>> Stream.fold sum $ Stream.enumerateFromTo 1 10
1092−-- Sum {getSum = 55}
1093−--
1094−{-# INLINE foldMap #-}
1095−foldMap :: (Monad m, Monoid b) => (a -> b) -> Fold m a b
1096−foldMap f = lmap f mconcat
1097−
1098−-- |
1099−-- Definition:
1100−--
1101−-- >>> foldMapM f = Fold.lmapM f Fold.mconcat
1102−--
1103−-- Make a fold from a monadic function that folds the output of the function
1104−-- using 'mappend' and 'mempty'.
1105−--
1106−-- >>> sum = Fold.foldMapM (return . Data.Monoid.Sum)
1107−-- >>> Stream.fold sum $ Stream.enumerateFromTo 1 10
1108−-- Sum {getSum = 55}
1109−--
1110−{-# INLINE foldMapM #-}
1111−foldMapM :: (Monad m, Monoid b) => (a -> m b) -> Fold m a b
1112−foldMapM act = foldlM' step (pure mempty)
1113−
1114− where
1115−
1116− step m a = do
1117− m' <- act a
1118− return $! mappend m m'
1119−
1120−------------------------------------------------------------------------------
1121−-- To Containers
1122−------------------------------------------------------------------------------
1123−
1124−-- $toListRev
1125−-- This is more efficient than 'Streamly.Internal.Data.Fold.toList'. toList is
1126−-- exactly the same as reversing the list after 'toListRev'.
1127−
1128−-- | Buffers the input stream to a list in the reverse order of the input.
1129−--
1130−-- Definition:
1131−--
1132−-- >>> toListRev = Fold.foldl' (flip (:)) []
1133−--
1134−-- /Warning!/ working on large lists accumulated as buffers in memory could be
1135−-- very inefficient, consider using "Streamly.Array" instead.
1136−--
1137−
1138−-- xn : ... : x2 : x1 : []
1139−{-# INLINE toListRev #-}
1140−toListRev :: Monad m => Fold m a [a]
1141−toListRev = foldl' (flip (:)) []
1142−
1143−------------------------------------------------------------------------------
1144−-- Partial Folds
1145−------------------------------------------------------------------------------
1146−
1147−-- | A fold that drains the first n elements of its input, running the effects
1148−-- and discarding the results.
1149−--
1150−-- Definition:
1151−--
1152−-- >>> drainN n = Fold.take n Fold.drain
1153−--
1154−-- /Pre-release/
1155−{-# INLINE drainN #-}
1156−drainN :: Monad m => Int -> Fold m a ()
1157−drainN n = take n drain
1158−
1159−------------------------------------------------------------------------------
1160−-- To Elements
1161−------------------------------------------------------------------------------
1162−
1163−-- | Like 'index', except with a more general 'Integral' argument
1164−--
1165−-- /Pre-release/
1166−{-# INLINE indexGeneric #-}
1167−indexGeneric :: (Integral i, Monad m) => i -> Fold m a (Maybe a)
1168−indexGeneric i = foldt' step (Partial 0) (const Nothing)
1169−
1170− where
1171−
1172− step j a =
1173− if i == j
1174− then Done $ Just a
1175− else Partial (j + 1)
1176−
1177−-- | Return the element at the given index.
1178−--
1179−-- Definition:
1180−--
1181−-- >>> index = Fold.indexGeneric
1182−--
1183−{-# INLINE index #-}
1184−index :: Monad m => Int -> Fold m a (Maybe a)
1185−index = indexGeneric
1186−
1187−-- | Consume a single input and transform it using the supplied 'Maybe'
1188−-- returning function.
1189−--
1190−-- /Pre-release/
1191−--
1192−{-# INLINE maybe #-}
1193−maybe :: Monad m => (a -> Maybe b) -> Fold m a (Maybe b)
1194−maybe f = foldt' (const (Done . f)) (Partial Nothing) id
1195−
1196−-- | Consume a single element and return it if it passes the predicate else
1197−-- return 'Nothing'.
1198−--
1199−-- Definition:
1200−--
1201−-- >>> satisfy f = Fold.maybe (\a -> if f a then Just a else Nothing)
1202−--
1203−-- /Pre-release/
1204−{-# INLINE satisfy #-}
1205−satisfy :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
1206−satisfy f = maybe (\a -> if f a then Just a else Nothing)
1207−{-
1208−satisfy f = Fold step (return $ Partial ()) (const (return Nothing))
1209−
1210− where
1211−
1212− step () a = return $ Done $ if f a then Just a else Nothing
1213−-}
1214−
1215−-- Naming notes:
1216−--
1217−-- "head" and "next" are two alternative names for the same API. head sounds
1218−-- apt in the context of lists but next sounds more apt in the context of
1219−-- streams where we think in terms of generating and consuming the next element
1220−-- rather than taking the head of some static/persistent structure.
1221−--
1222−-- We also want to keep the nomenclature consistent across folds and parsers,
1223−-- "head" becomes even more unintuitive for parsers because there are two
1224−-- possible variants viz. peek and next.
1225−--
1226−-- Also, the "head" fold creates confusion in situations like
1227−-- https://github.com/composewell/streamly/issues/1404 where intuitive
1228−-- expectation from head is to consume the entire stream and just give us the
1229−-- head. There we want to convey the notion that we consume one element from
1230−-- the stream and stop. The name "one" already being used in parsers for this
1231−-- purpose sounds more apt from this perspective.
1232−--
1233−-- The source of confusion is perhaps due to the fact that some folds consume
1234−-- the entire stream and others terminate early. It may have been clearer if we
1235−-- had separate abstractions for the two use cases.
1236−
1237−-- XXX We can possibly use "head" for the purposes of reducing the entire
1238−-- stream to the head element i.e. take the head and drain the rest.
1239−
1240−-- | Take one element from the stream and stop.
1241−--
1242−-- Definition:
1243−--
1244−-- >>> one = Fold.maybe Just
1245−--
1246−-- This is similar to the stream 'Stream.uncons' operation.
1247−--
1248−{-# INLINE one #-}
1249−one :: Monad m => Fold m a (Maybe a)
1250−one = maybe Just
1251−
1252−-- | Extract the first element of the stream, if any.
1253−--
1254−-- >>> head = Fold.one
1255−--
1256−{-# DEPRECATED head "Please use \"one\" instead" #-}
1257−{-# INLINE head #-}
1258−head :: Monad m => Fold m a (Maybe a)
1259−head = one
1260−
1261−-- | Returns the first element that satisfies the given predicate.
1262−--
1263−-- /Pre-release/
1264−{-# INLINE findM #-}
1265−findM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1266−findM predicate = Fold step (return $ Partial ()) (const $ return Nothing)
1267−
1268− where
1269−
1270− step () a =
1271− let f r =
1272− if r
1273− then Done (Just a)
1274− else Partial ()
1275− in f <$> predicate a
1276−
1277−-- | Returns the first element that satisfies the given predicate.
1278−--
1279−{-# INLINE find #-}
1280−find :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
1281−find p = findM (return . p)
1282−
1283−-- | In a stream of (key-value) pairs @(a, b)@, return the value @b@ of the
1284−-- first pair where the key equals the given value @a@.
1285−--
1286−-- Definition:
1287−--
1288−-- >>> lookup x = fmap snd <$> Fold.find ((== x) . fst)
1289−--
1290−{-# INLINE lookup #-}
1291−lookup :: (Eq a, Monad m) => a -> Fold m (a,b) (Maybe b)
1292−lookup a0 = foldt' step (Partial ()) (const Nothing)
1293−
1294− where
1295−
1296− step () (a, b) =
1297− if a == a0
1298− then Done $ Just b
1299− else Partial ()
1300−
1301−-- | Returns the first index that satisfies the given predicate.
1302−--
1303−{-# INLINE findIndex #-}
1304−findIndex :: Monad m => (a -> Bool) -> Fold m a (Maybe Int)
1305−findIndex predicate = foldt' step (Partial 0) (const Nothing)
1306−
1307− where
1308−
1309− step i a =
1310− if predicate a
1311− then Done $ Just i
1312− else Partial (i + 1)
1313−
1314−-- | Returns the index of the latest element if the element satisfies the given
1315−-- predicate.
1316−--
1317−{-# INLINE findIndices #-}
1318−findIndices :: Monad m => (a -> Bool) -> Fold m a (Maybe Int)
1319−findIndices predicate =
1320− -- XXX implement by combining indexing and filtering scans
1321− fmap (either (const Nothing) Just) $ foldl' step (Left (-1))
1322−
1323− where
1324−
1325− step i a =
1326− if predicate a
1327− then Right (either id id i + 1)
1328− else Left (either id id i + 1)
1329−
1330−-- | Returns the index of the latest element if the element matches the given
1331−-- value.
1332−--
1333−-- Definition:
1334−--
1335−-- >>> elemIndices a = Fold.findIndices (== a)
1336−--
1337−{-# INLINE elemIndices #-}
1338−elemIndices :: (Monad m, Eq a) => a -> Fold m a (Maybe Int)
1339−elemIndices a = findIndices (== a)
1340−
1341−-- | Returns the first index where a given value is found in the stream.
1342−--
1343−-- Definition:
1344−--
1345−-- >>> elemIndex a = Fold.findIndex (== a)
1346−--
1347−{-# INLINE elemIndex #-}
1348−elemIndex :: (Eq a, Monad m) => a -> Fold m a (Maybe Int)
1349−elemIndex a = findIndex (== a)
1350−
1351−------------------------------------------------------------------------------
1352−-- To Boolean
1353−------------------------------------------------------------------------------
1354−
1355−-- Similar to 'eof' parser, but the fold consumes and discards an input element
1356−-- when not at eof. XXX Remove or Rename to "eof"?
1357−
1358−-- | Consume one element, return 'True' if successful else return 'False'. In
1359−-- other words, test if the input is empty or not.
1360−--
1361−-- WARNING! It consumes one element if the stream is not empty. If that is not
1362−-- what you want please use the eof parser instead.
1363−--
1364−-- Definition:
1365−--
1366−-- >>> null = fmap isJust Fold.one
1367−--
1368−{-# INLINE null #-}
1369−null :: Monad m => Fold m a Bool
1370−null = foldt' (\() _ -> Done False) (Partial ()) (const True)
1371−
1372−-- | Returns 'True' if any element of the input satisfies the predicate.
1373−--
1374−-- Definition:
1375−--
1376−-- >>> any p = Fold.lmap p Fold.or
1377−--
1378−-- Example:
1379−--
1380−-- >>> Stream.fold (Fold.any (== 0)) $ Stream.fromList [1,0,1]
1381−-- True
1382−--
1383−{-# INLINE any #-}
1384−any :: Monad m => (a -> Bool) -> Fold m a Bool
1385−any predicate = foldt' step initial id
1386−
1387− where
1388−
1389− initial = Partial False
1390−
1391− step _ a =
1392− if predicate a
1393− then Done True
1394− else Partial False
1395−
1396−-- | Return 'True' if the given element is present in the stream.
1397−--
1398−-- Definition:
1399−--
1400−-- >>> elem a = Fold.any (== a)
1401−--
1402−{-# INLINE elem #-}
1403−elem :: (Eq a, Monad m) => a -> Fold m a Bool
1404−elem a = any (== a)
1405−
1406−-- | Returns 'True' if all elements of the input satisfy the predicate.
1407−--
1408−-- Definition:
1409−--
1410−-- >>> all p = Fold.lmap p Fold.and
1411−--
1412−-- Example:
1413−--
1414−-- >>> Stream.fold (Fold.all (== 0)) $ Stream.fromList [1,0,1]
1415−-- False
1416−--
1417−{-# INLINE all #-}
1418−all :: Monad m => (a -> Bool) -> Fold m a Bool
1419−all predicate = foldt' step initial id
1420−
1421− where
1422−
1423− initial = Partial True
1424−
1425− step _ a =
1426− if predicate a
1427− then Partial True
1428− else Done False
1429−
1430−-- | Returns 'True' if the given element is not present in the stream.
1431−--
1432−-- Definition:
1433−--
1434−-- >>> notElem a = Fold.all (/= a)
1435−--
1436−{-# INLINE notElem #-}
1437−notElem :: (Eq a, Monad m) => a -> Fold m a Bool
1438−notElem a = all (/= a)
1439−
1440−-- | Returns 'True' if all elements are 'True', 'False' otherwise
1441−--
1442−-- Definition:
1443−--
1444−-- >>> and = Fold.all (== True)
1445−--
1446−{-# INLINE and #-}
1447−and :: Monad m => Fold m Bool Bool
1448−and = all (== True)
1449−
1450−-- | Returns 'True' if any element is 'True', 'False' otherwise
1451−--
1452−-- Definition:
1453−--
1454−-- >>> or = Fold.any (== True)
1455−--
1456−{-# INLINE or #-}
1457−or :: Monad m => Fold m Bool Bool
1458−or = any (== True)
1459−
1460−------------------------------------------------------------------------------
1461−-- Grouping/Splitting
1462−------------------------------------------------------------------------------
1463−
1464−------------------------------------------------------------------------------
1465−-- Grouping without looking at elements
1466−------------------------------------------------------------------------------
1467−
1468−------------------------------------------------------------------------------
1469−-- Binary APIs
1470−------------------------------------------------------------------------------
1471−
1472−-- | @splitAt n f1 f2@ composes folds @f1@ and @f2@ such that first @n@
1473−-- elements of its input are consumed by fold @f1@ and the rest of the stream
1474−-- is consumed by fold @f2@.
1475−--
1476−-- >>> let splitAt_ n xs = Stream.fold (Fold.splitAt n Fold.toList Fold.toList) $ Stream.fromList xs
1477−--
1478−-- >>> splitAt_ 6 "Hello World!"
1479−-- ("Hello ","World!")
1480−--
1481−-- >>> splitAt_ (-1) [1,2,3]
1482−-- ([],[1,2,3])
1483−--
1484−-- >>> splitAt_ 0 [1,2,3]
1485−-- ([],[1,2,3])
1486−--
1487−-- >>> splitAt_ 1 [1,2,3]
1488−-- ([1],[2,3])
1489−--
1490−-- >>> splitAt_ 3 [1,2,3]
1491−-- ([1,2,3],[])
1492−--
1493−-- >>> splitAt_ 4 [1,2,3]
1494−-- ([1,2,3],[])
1495−--
1496−-- > splitAt n f1 f2 = Fold.splitWith (,) (Fold.take n f1) f2
1497−--
1498−-- /Internal/
1499−
1500−{-# INLINE splitAt #-}
1501−splitAt
1502− :: Monad m
1503− => Int
1504− -> Fold m a b
1505− -> Fold m a c
1506− -> Fold m a (b, c)
1507−splitAt n fld = splitWith (,) (take n fld)
1508−
1509−------------------------------------------------------------------------------
1510−-- Element Aware APIs
1511−------------------------------------------------------------------------------
1512−--
1513−------------------------------------------------------------------------------
1514−-- Binary APIs
1515−------------------------------------------------------------------------------
1516−
1517−{-# INLINE takingEndByM #-}
1518−takingEndByM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1519−takingEndByM p = Fold step initial (return . toMaybe)
1520−
1521− where
1522−
1523− initial = return $ Partial Nothing'
1524−
1525− step _ a = do
1526− r <- p a
1527− return
1528− $ if r
1529− then Done $ Just a
1530− else Partial $ Just' a
1531−
1532−-- |
1533−--
1534−-- >>> takingEndBy p = Fold.takingEndByM (return . p)
1535−--
1536−{-# INLINE takingEndBy #-}
1537−takingEndBy :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
1538−takingEndBy p = takingEndByM (return . p)
1539−
1540−{-# INLINE takingEndByM_ #-}
1541−takingEndByM_ :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1542−takingEndByM_ p = Fold step initial (return . toMaybe)
1543−
1544− where
1545−
1546− initial = return $ Partial Nothing'
1547−
1548− step _ a = do
1549− r <- p a
1550− return
1551− $ if r
1552− then Done Nothing
1553− else Partial $ Just' a
1554−
1555−-- |
1556−--
1557−-- >>> takingEndBy_ p = Fold.takingEndByM_ (return . p)
1558−--
1559−{-# INLINE takingEndBy_ #-}
1560−takingEndBy_ :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
1561−takingEndBy_ p = takingEndByM_ (return . p)
1562−
1563−{-# INLINE droppingWhileM #-}
1564−droppingWhileM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1565−droppingWhileM p = Fold step initial (return . toMaybe)
1566−
1567− where
1568−
1569− initial = return $ Partial Nothing'
1570−
1571− step Nothing' a = do
1572− r <- p a
1573− return
1574− $ Partial
1575− $ if r
1576− then Nothing'
1577− else Just' a
1578− step _ a = return $ Partial $ Just' a
1579−
1580−-- |
1581−-- >>> droppingWhile p = Fold.droppingWhileM (return . p)
1582−--
1583−{-# INLINE droppingWhile #-}
1584−droppingWhile :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
1585−droppingWhile p = droppingWhileM (return . p)
1586−
1587−-- Note: Keep this consistent with S.splitOn. In fact we should eliminate
1588−-- S.splitOn in favor of the fold.
1589−--
1590−-- XXX Use Fold.many instead once it is fixed.
1591−-- > Stream.splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
1592−
1593−-- | Like 'takeEndBy' but drops the element on which the predicate succeeds.
1594−--
1595−-- Example:
1596−--
1597−-- >>> input = Stream.fromList "hello\nthere\n"
1598−-- >>> line = Fold.takeEndBy_ (== '\n') Fold.toList
1599−-- >>> Stream.fold line input
1600−-- "hello"
1601−--
1602−-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1603−-- ["hello","there"]
1604−--
1605−{-# INLINE takeEndBy_ #-}
1606−takeEndBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1607−-- takeEndBy_ predicate = scanMaybe (takingEndBy_ predicate)
1608−takeEndBy_ predicate (Fold fstep finitial fextract) =
1609− Fold step finitial fextract
1610−
1611− where
1612−
1613− step s a =
1614− if not (predicate a)
1615− then fstep s a
1616− else Done <$> fextract s
1617−
1618−-- Note:
1619−-- > Stream.splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
1620−
1621−-- | Take the input, stop when the predicate succeeds taking the succeeding
1622−-- element as well.
1623−--
1624−-- Example:
1625−--
1626−-- >>> input = Stream.fromList "hello\nthere\n"
1627−-- >>> line = Fold.takeEndBy (== '\n') Fold.toList
1628−-- >>> Stream.fold line input
1629−-- "hello\n"
1630−--
1631−-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1632−-- ["hello\n","there\n"]
1633−--
1634−{-# INLINE takeEndBy #-}
1635−takeEndBy :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1636−-- takeEndBy predicate = scanMaybe (takingEndBy predicate)
1637−takeEndBy predicate (Fold fstep finitial fextract) =
1638− Fold step finitial fextract
1639−
1640− where
1641−
1642− step s a = do
1643− res <- fstep s a
1644− if not (predicate a)
1645− then return res
1646− else do
1647− case res of
1648− Partial s1 -> Done <$> fextract s1
1649− Done b -> return $ Done b
1650−
1651−------------------------------------------------------------------------------
1652−-- Binary splitting on a separator
1653−------------------------------------------------------------------------------
1654−
1655−data SplitOnSeqState acc a rb rh w ck =
1656− SplitOnSeqEmpty !acc
1657− | SplitOnSeqSingle !acc !a
1658− | SplitOnSeqWord !acc !Int !w
1659− | SplitOnSeqWordLoop !acc !w
1660− | SplitOnSeqKR !acc !Int !rb !rh
1661− | SplitOnSeqKRLoop !acc !ck !rb !rh
1662−
1663−-- XXX Need to add tests for takeEndBySeq, we have tests for takeEndBySeq_ .
1664−
1665−-- | Continue taking the input until the input sequence matches the supplied
1666−-- sequence, taking the supplied sequence as well. If the pattern is empty this
1667−-- acts as an identity fold.
1668−--
1669−-- >>> s = Stream.fromList "hello there. How are you?"
1670−-- >>> f = Fold.takeEndBySeq (Array.fromList "re") Fold.toList
1671−-- >>> Stream.fold f s
1672−-- "hello there"
1673−--
1674−-- >>> Stream.fold Fold.toList $ Stream.foldMany f s
1675−-- ["hello there",". How are"," you?"]
1676−--
1677−-- /Pre-release/
1678−{-# INLINE takeEndBySeq #-}
1679−takeEndBySeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) =>
1680− Array.Array a
1681− -> Fold m a b
1682− -> Fold m a b
1683−takeEndBySeq patArr (Fold fstep finitial fextract) =
1684− Fold step initial extract
1685−
1686− where
1687−
1688− patLen = Array.length patArr
1689−
1690− initial = do
1691− res <- finitial
1692− case res of
1693− Partial acc
1694− | patLen == 0 ->
1695− -- XXX Should we match nothing or everything on empty
1696− -- pattern?
1697− -- Done <$> fextract acc
1698− return $ Partial $ SplitOnSeqEmpty acc
1699− | patLen == 1 -> do
1700− pat <- liftIO $ Array.unsafeIndexIO 0 patArr
1701− return $ Partial $ SplitOnSeqSingle acc pat
1702− | SIZE_OF(a) * patLen <= sizeOf (Proxy :: Proxy Word) ->
1703− return $ Partial $ SplitOnSeqWord acc 0 0
1704− | otherwise -> do
1705− (rb, rhead) <- liftIO $ Ring.new patLen
1706− return $ Partial $ SplitOnSeqKR acc 0 rb rhead
1707− Done b -> return $ Done b
1708−
1709− -- Word pattern related
1710− maxIndex = patLen - 1
1711−
1712− elemBits = SIZE_OF(a) * 8
1713−
1714− wordMask :: Word
1715− wordMask = (1 `shiftL` (elemBits * patLen)) - 1
1716−
1717− wordPat :: Word
1718− wordPat = wordMask .&. Array.foldl' addToWord 0 patArr
1719−
1720− addToWord wd a = (wd `shiftL` elemBits) .|. fromIntegral (fromEnum a)
1721−
1722− -- For Rabin-Karp search
1723− k = 2891336453 :: Word32
1724− coeff = k ^ patLen
1725−
1726− addCksum cksum a = cksum * k + fromIntegral (fromEnum a)
1727−
1728− deltaCksum cksum old new =
1729− addCksum cksum new - coeff * fromIntegral (fromEnum old)
1730−
1731− -- XXX shall we use a random starting hash or 1 instead of 0?
1732− -- XXX Need to keep this cached across fold calls in foldmany
1733− -- XXX We may need refold to inject the cached state instead of
1734− -- initializing the state every time.
1735− -- XXX Allocation of ring buffer should also be done once
1736− patHash = Array.foldl' addCksum 0 patArr
1737−
1738− step (SplitOnSeqEmpty s) x = do
1739− res <- fstep s x
1740− case res of
1741− Partial s1 -> return $ Partial $ SplitOnSeqEmpty s1
1742− Done b -> return $ Done b
1743− step (SplitOnSeqSingle s pat) x = do
1744− res <- fstep s x
1745− case res of
1746− Partial s1
1747− | pat /= x -> return $ Partial $ SplitOnSeqSingle s1 pat
1748− | otherwise -> Done <$> fextract s1
1749− Done b -> return $ Done b
1750− step (SplitOnSeqWord s idx wrd) x = do
1751− res <- fstep s x
1752− let wrd1 = addToWord wrd x
1753− case res of
1754− Partial s1
1755− | idx == maxIndex -> do
1756− if wrd1 .&. wordMask == wordPat
1757− then Done <$> fextract s1
1758− else return $ Partial $ SplitOnSeqWordLoop s1 wrd1
1759− | otherwise ->
1760− return $ Partial $ SplitOnSeqWord s1 (idx + 1) wrd1
1761− Done b -> return $ Done b
1762− step (SplitOnSeqWordLoop s wrd) x = do
1763− res <- fstep s x
1764− let wrd1 = addToWord wrd x
1765− case res of
1766− Partial s1
1767− | wrd1 .&. wordMask == wordPat ->
1768− Done <$> fextract s1
1769− | otherwise ->
1770− return $ Partial $ SplitOnSeqWordLoop s1 wrd1
1771− Done b -> return $ Done b
1772− step (SplitOnSeqKR s idx rb rh) x = do
1773− res <- fstep s x
1774− case res of
1775− Partial s1 -> do
1776− rh1 <- liftIO $ Ring.unsafeInsert rb rh x
1777− if idx == maxIndex
1778− then do
1779− let fld = Ring.unsafeFoldRing (Ring.ringBound rb)
1780− let !ringHash = fld addCksum 0 rb
1781− if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1782− then Done <$> fextract s1
1783− else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
1784− else
1785− return $ Partial $ SplitOnSeqKR s1 (idx + 1) rb rh1
1786− Done b -> return $ Done b
1787− step (SplitOnSeqKRLoop s cksum rb rh) x = do
1788− res <- fstep s x
1789− case res of
1790− Partial s1 -> do
1791− old <- liftIO $ peek rh
1792− rh1 <- liftIO $ Ring.unsafeInsert rb rh x
1793− let ringHash = deltaCksum cksum old x
1794− if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1795− then Done <$> fextract s1
1796− else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
1797− Done b -> return $ Done b
1798−
1799− extract state =
1800− let st =
1801− case state of
1802− SplitOnSeqEmpty s -> s
1803− SplitOnSeqSingle s _ -> s
1804− SplitOnSeqWord s _ _ -> s
1805− SplitOnSeqWordLoop s _ -> s
1806− SplitOnSeqKR s _ _ _ -> s
1807− SplitOnSeqKRLoop s _ _ _ -> s
1808− in fextract st
1809−
1810−-- | Like 'takeEndBySeq' but discards the matched sequence.
1811−--
1812−-- /Pre-release/
1813−--
1814−{-# INLINE takeEndBySeq_ #-}
1815−takeEndBySeq_ :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) =>
1816− Array.Array a
1817− -> Fold m a b
1818− -> Fold m a b
1819−takeEndBySeq_ patArr (Fold fstep finitial fextract) =
1820− Fold step initial extract
1821−
1822− where
1823−
1824− patLen = Array.length patArr
1825−
1826− initial = do
1827− res <- finitial
1828− case res of
1829− Partial acc
1830− | patLen == 0 ->
1831− -- XXX Should we match nothing or everything on empty
1832− -- pattern?
1833− -- Done <$> fextract acc
1834− return $ Partial $ SplitOnSeqEmpty acc
1835− | patLen == 1 -> do
1836− pat <- liftIO $ Array.unsafeIndexIO 0 patArr
1837− return $ Partial $ SplitOnSeqSingle acc pat
1838− -- XXX Need to add tests for this case
1839− | SIZE_OF(a) * patLen <= sizeOf (Proxy :: Proxy Word) ->
1840− return $ Partial $ SplitOnSeqWord acc 0 0
1841− | otherwise -> do
1842− (rb, rhead) <- liftIO $ Ring.new patLen
1843− return $ Partial $ SplitOnSeqKR acc 0 rb rhead
1844− Done b -> return $ Done b
1845−
1846− -- Word pattern related
1847− maxIndex = patLen - 1
1848−
1849− elemBits = SIZE_OF(a) * 8
1850−
1851− wordMask :: Word
1852− wordMask = (1 `shiftL` (elemBits * patLen)) - 1
1853−
1854− elemMask :: Word
1855− elemMask = (1 `shiftL` elemBits) - 1
1856−
1857− wordPat :: Word
1858− wordPat = wordMask .&. Array.foldl' addToWord 0 patArr
1859−
1860− addToWord wd a = (wd `shiftL` elemBits) .|. fromIntegral (fromEnum a)
1861−
1862− -- For Rabin-Karp search
1863− k = 2891336453 :: Word32
1864− coeff = k ^ patLen
1865−
1866− addCksum cksum a = cksum * k + fromIntegral (fromEnum a)
1867−
1868− deltaCksum cksum old new =
1869− addCksum cksum new - coeff * fromIntegral (fromEnum old)
1870−
1871− -- XXX shall we use a random starting hash or 1 instead of 0?
1872− -- XXX Need to keep this cached across fold calls in foldMany
1873− -- XXX We may need refold to inject the cached state instead of
1874− -- initializing the state every time.
1875− -- XXX Allocation of ring buffer should also be done once
1876− patHash = Array.foldl' addCksum 0 patArr
1877−
1878− step (SplitOnSeqEmpty s) x = do
1879− res <- fstep s x
1880− case res of
1881− Partial s1 -> return $ Partial $ SplitOnSeqEmpty s1
1882− Done b -> return $ Done b
1883− step (SplitOnSeqSingle s pat) x = do
1884− if pat /= x
1885− then do
1886− res <- fstep s x
1887− case res of
1888− Partial s1 -> return $ Partial $ SplitOnSeqSingle s1 pat
1889− Done b -> return $ Done b
1890− else Done <$> fextract s
1891− step (SplitOnSeqWord s idx wrd) x = do
1892− let wrd1 = addToWord wrd x
1893− if idx == maxIndex
1894− then do
1895− if wrd1 .&. wordMask == wordPat
1896− then Done <$> fextract s
1897− else return $ Partial $ SplitOnSeqWordLoop s wrd1
1898− else return $ Partial $ SplitOnSeqWord s (idx + 1) wrd1
1899− step (SplitOnSeqWordLoop s wrd) x = do
1900− let wrd1 = addToWord wrd x
1901− old = (wordMask .&. wrd)
1902− `shiftR` (elemBits * (patLen - 1))
1903− res <- fstep s (toEnum $ fromIntegral old)
1904− case res of
1905− Partial s1
1906− | wrd1 .&. wordMask == wordPat ->
1907− Done <$> fextract s1
1908− | otherwise ->
1909− return $ Partial $ SplitOnSeqWordLoop s1 wrd1
1910− Done b -> return $ Done b
1911− step (SplitOnSeqKR s idx rb rh) x = do
1912− rh1 <- liftIO $ Ring.unsafeInsert rb rh x
1913− if idx == maxIndex
1914− then do
1915− let fld = Ring.unsafeFoldRing (Ring.ringBound rb)
1916− let !ringHash = fld addCksum 0 rb
1917− if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1918− then Done <$> fextract s
1919− else return $ Partial $ SplitOnSeqKRLoop s ringHash rb rh1
1920− else return $ Partial $ SplitOnSeqKR s (idx + 1) rb rh1
1921− step (SplitOnSeqKRLoop s cksum rb rh) x = do
1922− old <- liftIO $ peek rh
1923− res <- fstep s old
1924− case res of
1925− Partial s1 -> do
1926− rh1 <- liftIO $ Ring.unsafeInsert rb rh x
1927− let ringHash = deltaCksum cksum old x
1928− if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1929− then Done <$> fextract s1
1930− else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
1931− Done b -> return $ Done b
1932−
1933− -- XXX extract should return backtrack count as well. If the fold
1934− -- terminates early inside extract, we may still have buffered data
1935− -- remaining which will be lost if we do not communicate that to the
1936− -- driver.
1937− extract state = do
1938− let consumeWord s n wrd = do
1939− if n == 0
1940− then fextract s
1941− else do
1942− let old = elemMask .&. (wrd `shiftR` (elemBits * (n - 1)))
1943− r <- fstep s (toEnum $ fromIntegral old)
1944− case r of
1945− Partial s1 -> consumeWord s1 (n - 1) wrd
1946− Done b -> return b
1947−
1948− let consumeRing s n rb rh =
1949− if n == 0
1950− then fextract s
1951− else do
1952− old <- liftIO $ peek rh
1953− let rh1 = Ring.advance rb rh
1954− r <- fstep s old
1955− case r of
1956− Partial s1 -> consumeRing s1 (n - 1) rb rh1
1957− Done b -> return b
1958−
1959− case state of
1960− SplitOnSeqEmpty s -> fextract s
1961− SplitOnSeqSingle s _ -> fextract s
1962− SplitOnSeqWord s idx wrd -> consumeWord s idx wrd
1963− SplitOnSeqWordLoop s wrd -> consumeWord s patLen wrd
1964− SplitOnSeqKR s idx rb _ -> consumeRing s idx rb (Ring.startOf rb)
1965− SplitOnSeqKRLoop s _ rb rh -> consumeRing s patLen rb rh
1966−
1967−------------------------------------------------------------------------------
1968−-- Distributing
1969−------------------------------------------------------------------------------
1970−--
1971−-- | Distribute one copy of the stream to each fold and zip the results.
1972−--
1973−-- @
1974−-- |-------Fold m a b--------|
1975−-- ---stream m a---| |---m (b,c)
1976−-- |-------Fold m a c--------|
1977−-- @
1978−--
1979−-- Definition:
1980−--
1981−-- >>> tee = Fold.teeWith (,)
1982−--
1983−-- Example:
1984−--
1985−-- >>> t = Fold.tee Fold.sum Fold.length
1986−-- >>> Stream.fold t (Stream.enumerateFromTo 1.0 100.0)
1987−-- (5050.0,100)
1988−--
1989−{-# INLINE tee #-}
1990−tee :: Monad m => Fold m a b -> Fold m a c -> Fold m a (b,c)
1991−tee = teeWith (,)
1992−
1993−-- XXX use "List" instead of "[]"?, use Array for output to scale it to a large
1994−-- number of consumers? For polymorphic case a vector could be helpful. For
1995−-- Storables we can use arrays. Will need separate APIs for those.
1996−--
1997−-- | Distribute one copy of the stream to each fold and collect the results in
1998−-- a container.
1999−--
2000−-- @
2001−--
2002−-- |-------Fold m a b--------|
2003−-- ---stream m a---| |---m [b]
2004−-- |-------Fold m a b--------|
2005−-- | |
2006−-- ...
2007−-- @
2008−--
2009−-- >>> Stream.fold (Fold.distribute [Fold.sum, Fold.length]) (Stream.enumerateFromTo 1 5)
2010−-- [15,5]
2011−--
2012−-- >>> distribute = Prelude.foldr (Fold.teeWith (:)) (Fold.fromPure [])
2013−--
2014−-- This is the consumer side dual of the producer side 'sequence' operation.
2015−--
2016−-- Stops when all the folds stop.
2017−--
2018−{-# INLINE distribute #-}
2019−distribute :: Monad m => [Fold m a b] -> Fold m a [b]
2020−distribute = Prelude.foldr (teeWith (:)) (fromPure [])
2021−
2022−------------------------------------------------------------------------------
2023−-- Partitioning
2024−------------------------------------------------------------------------------
2025−
2026−{-# INLINE partitionByMUsing #-}
2027−partitionByMUsing :: Monad m =>
2028− ( (x -> y -> (x, y))
2029− -> Fold m (Either b c) x
2030− -> Fold m (Either b c) y
2031− -> Fold m (Either b c) (x, y)
2032− )
2033− -> (a -> m (Either b c))
2034− -> Fold m b x
2035− -> Fold m c y
2036− -> Fold m a (x, y)
2037−partitionByMUsing t f fld1 fld2 =
2038− let l = lmap (fromLeft undefined) fld1 -- :: Fold m (Either b c) x
2039− r = lmap (fromRight undefined) fld2 -- :: Fold m (Either b c) y
2040− in lmapM f (t (,) (filter isLeft l) (filter isRight r))
2041−
2042−-- | Partition the input over two folds using an 'Either' partitioning
2043−-- predicate.
2044−--
2045−-- @
2046−--
2047−-- |-------Fold b x--------|
2048−-- -----stream m a --> (Either b c)----| |----(x,y)
2049−-- |-------Fold c y--------|
2050−-- @
2051−--
2052−-- Example, send input to either fold randomly:
2053−--
2054−-- >>> :set -package random
2055−-- >>> import System.Random (randomIO)
2056−-- >>> randomly a = randomIO >>= \x -> return $ if x then Left a else Right a
2057−-- >>> f = Fold.partitionByM randomly Fold.length Fold.length
2058−-- >>> Stream.fold f (Stream.enumerateFromTo 1 100)
2059−-- ...
2060−--
2061−-- Example, send input to the two folds in a proportion of 2:1:
2062−--
2063−-- >>> :{
2064−-- proportionately m n = do
2065−-- ref <- newIORef $ cycle $ concat [replicate m Left, replicate n Right]
2066−-- return $ \a -> do
2067−-- r <- readIORef ref
2068−-- writeIORef ref $ tail r
2069−-- return $ Prelude.head r a
2070−-- :}
2071−--
2072−-- >>> :{
2073−-- main = do
2074−-- g <- proportionately 2 1
2075−-- let f = Fold.partitionByM g Fold.length Fold.length
2076−-- r <- Stream.fold f (Stream.enumerateFromTo (1 :: Int) 100)
2077−-- print r
2078−-- :}
2079−--
2080−-- >>> main
2081−-- (67,33)
2082−--
2083−--
2084−-- This is the consumer side dual of the producer side 'mergeBy' operation.
2085−--
2086−-- When one fold is done, any input meant for it is ignored until the other
2087−-- fold is also done.
2088−--
2089−-- Stops when both the folds stop.
2090−--
2091−-- /See also: 'partitionByFstM' and 'partitionByMinM'./
2092−--
2093−-- /Pre-release/
2094−{-# INLINE partitionByM #-}
2095−partitionByM :: Monad m
2096− => (a -> m (Either b c)) -> Fold m b x -> Fold m c y -> Fold m a (x, y)
2097−partitionByM = partitionByMUsing teeWith
2098−
2099−-- | Similar to 'partitionByM' but terminates when the first fold terminates.
2100−--
2101−{-# INLINE partitionByFstM #-}
2102−partitionByFstM :: Monad m
2103− => (a -> m (Either b c)) -> Fold m b x -> Fold m c y -> Fold m a (x, y)
2104−partitionByFstM = partitionByMUsing teeWithFst
2105−
2106−-- | Similar to 'partitionByM' but terminates when any fold terminates.
2107−--
2108−{-# INLINE partitionByMinM #-}
2109−partitionByMinM :: Monad m =>
2110− (a -> m (Either b c)) -> Fold m b x -> Fold m c y -> Fold m a (x, y)
2111−partitionByMinM = partitionByMUsing teeWithMin
2112−
2113−-- Note: we could use (a -> Bool) instead of (a -> Either b c), but the latter
2114−-- makes the signature clearer as to which case belongs to which fold.
2115−-- XXX need to check the performance in both cases.
2116−
2117−-- | Same as 'partitionByM' but with a pure partition function.
2118−--
2119−-- Example, count even and odd numbers in a stream:
2120−--
2121−-- >>> :{
2122−-- let f = Fold.partitionBy (\n -> if even n then Left n else Right n)
2123−-- (fmap (("Even " ++) . show) Fold.length)
2124−-- (fmap (("Odd " ++) . show) Fold.length)
2125−-- in Stream.fold f (Stream.enumerateFromTo 1 100)
2126−-- :}
2127−-- ("Even 50","Odd 50")
2128−--
2129−-- /Pre-release/
2130−{-# INLINE partitionBy #-}
2131−partitionBy :: Monad m
2132− => (a -> Either b c) -> Fold m b x -> Fold m c y -> Fold m a (x, y)
2133−partitionBy f = partitionByM (return . f)
2134−
2135−-- | Compose two folds such that the combined fold accepts a stream of 'Either'
2136−-- and routes the 'Left' values to the first fold and 'Right' values to the
2137−-- second fold.
2138−--
2139−-- Definition:
2140−--
2141−-- >>> partition = Fold.partitionBy id
2142−--
2143−{-# INLINE partition #-}
2144−partition :: Monad m
2145− => Fold m b x -> Fold m c y -> Fold m (Either b c) (x, y)
2146−partition = partitionBy id
2147−
2148−{-
2149−-- | Send one item to each fold in a round-robin fashion. This is the consumer
2150−-- side dual of producer side 'mergeN' operation.
2151−--
2152−-- partitionN :: Monad m => [Fold m a b] -> Fold m a [b]
2153−-- partitionN fs = Fold step begin done
2154−-}
2155−
2156−------------------------------------------------------------------------------
2157−-- Unzipping
2158−------------------------------------------------------------------------------
2159−
2160−{-# INLINE unzipWithMUsing #-}
2161−unzipWithMUsing :: Monad m =>
2162− ( (x -> y -> (x, y))
2163− -> Fold m (b, c) x
2164− -> Fold m (b, c) y
2165− -> Fold m (b, c) (x, y)
2166− )
2167− -> (a -> m (b, c))
2168− -> Fold m b x
2169− -> Fold m c y
2170− -> Fold m a (x, y)
2171−unzipWithMUsing t f fld1 fld2 =
2172− let f1 = lmap fst fld1 -- :: Fold m (b, c) b
2173− f2 = lmap snd fld2 -- :: Fold m (b, c) c
2174− in lmapM f (t (,) f1 f2)
2175−
2176−-- | Like 'unzipWith' but with a monadic splitter function.
2177−--
2178−-- Definition:
2179−--
2180−-- >>> unzipWithM k f1 f2 = Fold.lmapM k (Fold.unzip f1 f2)
2181−--
2182−-- /Pre-release/
2183−{-# INLINE unzipWithM #-}
2184−unzipWithM :: Monad m
2185− => (a -> m (b,c)) -> Fold m b x -> Fold m c y -> Fold m a (x,y)
2186−unzipWithM = unzipWithMUsing teeWith
2187−
2188−-- | Similar to 'unzipWithM' but terminates when the first fold terminates.
2189−--
2190−{-# INLINE unzipWithFstM #-}
2191−unzipWithFstM :: Monad m =>
2192− (a -> m (b, c)) -> Fold m b x -> Fold m c y -> Fold m a (x, y)
2193−unzipWithFstM = unzipWithMUsing teeWithFst
2194−
2195−-- | Similar to 'unzipWithM' but terminates when any fold terminates.
2196−--
2197−{-# INLINE unzipWithMinM #-}
2198−unzipWithMinM :: Monad m =>
2199− (a -> m (b,c)) -> Fold m b x -> Fold m c y -> Fold m a (x,y)
2200−unzipWithMinM = unzipWithMUsing teeWithMin
2201−
2202−-- | Split elements in the input stream into two parts using a pure splitter
2203−-- function, direct each part to a different fold and zip the results.
2204−--
2205−-- Definitions:
2206−--
2207−-- >>> unzipWith f = Fold.unzipWithM (return . f)
2208−-- >>> unzipWith f fld1 fld2 = Fold.lmap f (Fold.unzip fld1 fld2)
2209−--
2210−-- This fold terminates when both the input folds terminate.
2211−--
2212−-- /Pre-release/
2213−{-# INLINE unzipWith #-}
2214−unzipWith :: Monad m
2215− => (a -> (b,c)) -> Fold m b x -> Fold m c y -> Fold m a (x,y)
2216−unzipWith f = unzipWithM (return . f)
2217−
2218−-- | Send the elements of tuples in a stream of tuples through two different
2219−-- folds.
2220−--
2221−-- @
2222−--
2223−-- |-------Fold m a x--------|
2224−-- ---------stream of (a,b)--| |----m (x,y)
2225−-- |-------Fold m b y--------|
2226−--
2227−-- @
2228−--
2229−-- Definition:
2230−--
2231−-- >>> unzip = Fold.unzipWith id
2232−--
2233−-- This is the consumer side dual of the producer side 'zip' operation.
2234−--
2235−{-# INLINE unzip #-}
2236−unzip :: Monad m => Fold m a x -> Fold m b y -> Fold m (a,b) (x,y)
2237−unzip = unzipWith id
2238−
2239−------------------------------------------------------------------------------
2240−-- Combining streams and folds - Zipping
2241−------------------------------------------------------------------------------
2242−
2243−-- XXX These can be implemented using the fold scan, using the stream as a
2244−-- state.
2245−-- XXX Stream Skip state cannot be efficiently handled in folds but can be
2246−-- handled in parsers using the Continue facility. See zipWithM in the Parser
2247−-- module.
2248−--
2249−-- cmpBy, eqBy, isPrefixOf, isSubsequenceOf etc can be implemented using
2250−-- zipStream.
2251−
2252−-- | Zip a stream with the input of a fold using the supplied function.
2253−--
2254−-- /Unimplemented/
2255−--
2256−{-# INLINE zipStreamWithM #-}
2257−zipStreamWithM :: -- Monad m =>
2258− (a -> b -> m c) -> Stream m a -> Fold m c x -> Fold m b x
2259−zipStreamWithM = undefined
2260−
2261−-- | Zip a stream with the input of a fold.
2262−--
2263−-- >>> zip = Fold.zipStreamWithM (curry return)
2264−--
2265−-- /Unimplemented/
2266−--
2267−{-# INLINE zipStream #-}
2268−zipStream :: Monad m => Stream m a -> Fold m (a, b) x -> Fold m b x
2269−zipStream = zipStreamWithM (curry return)
2270−
2271−-- | Pair each element of a fold input with its index, starting from index 0.
2272−--
2273−{-# INLINE indexingWith #-}
2274−indexingWith :: Monad m => Int -> (Int -> Int) -> Fold m a (Maybe (Int, a))
2275−indexingWith i f = fmap toMaybe $ foldl' step initial
2276−
2277− where
2278−
2279− initial = Nothing'
2280−
2281− step Nothing' a = Just' (i, a)
2282− step (Just' (n, _)) a = Just' (f n, a)
2283−
2284−-- |
2285−-- >>> indexing = Fold.indexingWith 0 (+ 1)
2286−--
2287−{-# INLINE indexing #-}
2288−indexing :: Monad m => Fold m a (Maybe (Int, a))
2289−indexing = indexingWith 0 (+ 1)
2290−
2291−-- |
2292−-- >>> indexingRev n = Fold.indexingWith n (subtract 1)
2293−--
2294−{-# INLINE indexingRev #-}
2295−indexingRev :: Monad m => Int -> Fold m a (Maybe (Int, a))
2296−indexingRev n = indexingWith n (subtract 1)
2297−
2298−-- | Pair each element of a fold input with its index, starting from index 0.
2299−--
2300−-- >>> indexed = Fold.scanMaybe Fold.indexing
2301−--
2302−{-# INLINE indexed #-}
2303−indexed :: Monad m => Fold m (Int, a) b -> Fold m a b
2304−indexed = scanMaybe indexing
2305−
2306−-- | Change the predicate function of a Fold from @a -> b@ to accept an
2307−-- additional state input @(s, a) -> b@. Convenient to filter with an
2308−-- addiitonal index or time input.
2309−--
2310−-- >>> filterWithIndex = Fold.with Fold.indexed Fold.filter
2311−--
2312−-- @
2313−-- filterWithAbsTime = with timestamped filter
2314−-- filterWithRelTime = with timeIndexed filter
2315−-- @
2316−--
2317−-- /Pre-release/
2318−{-# INLINE with #-}
2319−with ::
2320− (Fold m (s, a) b -> Fold m a b)
2321− -> (((s, a) -> c) -> Fold m (s, a) b -> Fold m (s, a) b)
2322− -> (((s, a) -> c) -> Fold m a b -> Fold m a b)
2323−with f comb g = f . comb g . lmap snd
2324−
2325−-- XXX Implement as a filter
2326−-- sampleFromthen :: Monad m => Int -> Int -> Fold m a (Maybe a)
2327−
2328−-- | @sampleFromthen offset stride@ samples the element at @offset@ index and
2329−-- then every element at strides of @stride@.
2330−--
2331−{-# INLINE sampleFromthen #-}
2332−sampleFromthen :: Monad m => Int -> Int -> Fold m a b -> Fold m a b
2333−sampleFromthen offset size =
2334− with indexed filter (\(i, _) -> (i + offset) `mod` size == 0)
2335−
2336−------------------------------------------------------------------------------
2337−-- Nesting
2338−------------------------------------------------------------------------------
2339−
2340−-- | @concatSequence f t@ applies folds from stream @t@ sequentially and
2341−-- collects the results using the fold @f@.
2342−--
2343−-- /Unimplemented/
2344−--
2345−{-# INLINE concatSequence #-}
2346−concatSequence ::
2347− -- IsStream t =>
2348− Fold m b c -> t (Fold m a b) -> Fold m a c
2349−concatSequence _f _p = undefined
2350−
2351−-- | Group the input stream into groups of elements between @low@ and @high@.
2352−-- Collection starts in chunks of @low@ and then keeps doubling until we reach
2353−-- @high@. Each chunk is folded using the provided fold function.
2354−--
2355−-- This could be useful, for example, when we are folding a stream of unknown
2356−-- size to a stream of arrays and we want to minimize the number of
2357−-- allocations.
2358−--
2359−-- NOTE: this would be an application of "many" using a terminating fold.
2360−--
2361−-- /Unimplemented/
2362−--
2363−{-# INLINE chunksBetween #-}
2364−chunksBetween :: -- Monad m =>
2365− Int -> Int -> Fold m a b -> Fold m b c -> Fold m a c
2366−chunksBetween _low _high _f1 _f2 = undefined
2367−
2368−-- | A fold that buffers its input to a pure stream.
2369−--
2370−-- /Warning!/ working on large streams accumulated as buffers in memory could
2371−-- be very inefficient, consider using "Streamly.Data.Array" instead.
2372−--
2373−-- >>> toStream = fmap Stream.fromList Fold.toList
2374−--
2375−-- /Pre-release/
2376−{-# INLINE toStream #-}
2377−toStream :: (Monad m, Monad n) => Fold m a (Stream n a)
2378−toStream = fmap StreamD.fromList toList
2379−
2380−-- This is more efficient than 'toStream'. toStream is exactly the same as
2381−-- reversing the stream after toStreamRev.
2382−--
2383−-- | Buffers the input stream to a pure stream in the reverse order of the
2384−-- input.
2385−--
2386−-- >>> toStreamRev = fmap Stream.fromList Fold.toListRev
2387−--
2388−-- /Warning!/ working on large streams accumulated as buffers in memory could
2389−-- be very inefficient, consider using "Streamly.Data.Array" instead.
2390−--
2391−-- /Pre-release/
2392−
2393−-- xn : ... : x2 : x1 : []
2394−{-# INLINE toStreamRev #-}
2395−toStreamRev :: (Monad m, Monad n) => Fold m a (Stream n a)
2396−toStreamRev = fmap StreamD.fromList toListRev
2397−
2398−-- XXX This does not fuse. It contains a recursive step function. We will need
2399−-- a Skip input constructor in the fold type to make it fuse.
2400−--
2401−-- | Unfold and flatten the input stream of a fold.
2402−--
2403−-- @
2404−-- Stream.fold (unfoldMany u f) = Stream.fold f . Stream.unfoldMany u
2405−-- @
2406−--
2407−-- /Pre-release/
2408−{-# INLINE unfoldMany #-}
2409−unfoldMany :: Monad m => Unfold m a b -> Fold m b c -> Fold m a c
2410−unfoldMany (Unfold ustep inject) (Fold fstep initial extract) =
2411− Fold consume initial extract
2412−
2413− where
2414−
2415− {-# INLINE produce #-}
2416− produce fs us = do
2417− ures <- ustep us
2418− case ures of
2419− StreamD.Yield b us1 -> do
2420− fres <- fstep fs b
2421− case fres of
2422− Partial fs1 -> produce fs1 us1
2423− -- XXX What to do with the remaining stream?
2424− Done c -> return $ Done c
2425− StreamD.Skip us1 -> produce fs us1
2426− StreamD.Stop -> return $ Partial fs
2427−
2428− {-# INLINE_LATE consume #-}
2429− consume s a = inject a >>= produce s
2430−
2431−-- | Get the bottom most @n@ elements using the supplied comparison function.
2432−--
2433−{-# INLINE bottomBy #-}
2434−bottomBy :: (MonadIO m, Unbox a) =>
2435− (a -> a -> Ordering)
2436− -> Int
2437− -> Fold m a (MutArray a)
2438−bottomBy cmp n = Fold step initial extract
2439−
2440− where
2441−
2442− initial = do
2443− arr <- MA.pinnedNew n
2444− if n <= 0
2445− then return $ Done arr
2446− else return $ Partial (arr, 0)
2447−
2448− step (arr, i) x =
2449− if i < n
2450− then do
2451− arr' <- MA.snoc arr x
2452− MA.bubble cmp arr'
2453− return $ Partial (arr', i + 1)
2454− else do
2455− x1 <- MA.getIndexUnsafe (i - 1) arr
2456− case x `cmp` x1 of
2457− LT -> do
2458− MA.putIndexUnsafe (i - 1) arr x
2459− MA.bubble cmp arr
2460− return $ Partial (arr, i)
2461− _ -> return $ Partial (arr, i)
2462−
2463− extract = return . fst
2464−
2465−-- | Get the top @n@ elements using the supplied comparison function.
2466−--
2467−-- To get bottom n elements instead:
2468−--
2469−-- >>> bottomBy cmp = Fold.topBy (flip cmp)
2470−--
2471−-- Example:
2472−--
2473−-- >>> stream = Stream.fromList [2::Int,7,9,3,1,5,6,11,17]
2474−-- >>> Stream.fold (Fold.topBy compare 3) stream >>= MutArray.toList
2475−-- [17,11,9]
2476−--
2477−-- /Pre-release/
2478−--
2479−{-# INLINE topBy #-}
2480−topBy :: (MonadIO m, Unbox a) =>
2481− (a -> a -> Ordering)
2482− -> Int
2483− -> Fold m a (MutArray a)
2484−topBy cmp = bottomBy (flip cmp)
2485−
2486−-- | Fold the input stream to top n elements.
2487−--
2488−-- Definition:
2489−--
2490−-- >>> top = Fold.topBy compare
2491−--
2492−-- >>> stream = Stream.fromList [2::Int,7,9,3,1,5,6,11,17]
2493−-- >>> Stream.fold (Fold.top 3) stream >>= MutArray.toList
2494−-- [17,11,9]
2495−--
2496−-- /Pre-release/
2497−{-# INLINE top #-}
2498−top :: (MonadIO m, Unbox a, Ord a) => Int -> Fold m a (MutArray a)
2499−top = bottomBy $ flip compare
2500−
2501−-- | Fold the input stream to bottom n elements.
2502−--
2503−-- Definition:
2504−--
2505−-- >>> bottom = Fold.bottomBy compare
2506−--
2507−-- >>> stream = Stream.fromList [2::Int,7,9,3,1,5,6,11,17]
2508−-- >>> Stream.fold (Fold.bottom 3) stream >>= MutArray.toList
2509−-- [1,2,3]
2510−--
2511−-- /Pre-release/
2512−{-# INLINE bottom #-}
2513−bottom :: (MonadIO m, Unbox a, Ord a) => Int -> Fold m a (MutArray a)
2514−bottom = bottomBy compare
2515−
2516−------------------------------------------------------------------------------
2517−-- Interspersed parsing
2518−------------------------------------------------------------------------------
2519−
2520−data IntersperseQState fs ps =
2521− IntersperseQUnquoted !fs !ps
2522− | IntersperseQQuoted !fs !ps
2523− | IntersperseQQuotedEsc !fs !ps
2524−
2525−-- Useful for parsing CSV with quoting and escaping
2526−{-# INLINE intersperseWithQuotes #-}
2527−intersperseWithQuotes :: (Monad m, Eq a) =>
2528− a -> a -> a -> Fold m a b -> Fold m b c -> Fold m a c
2529−intersperseWithQuotes
2530− quote
2531− esc
2532− separator
2533− (Fold stepL initialL extractL)
2534− (Fold stepR initialR extractR) = Fold step initial extract
2535−
2536− where
2537−
2538− errMsg p status =
2539− error $ "intersperseWithQuotes: " ++ p ++ " parsing fold cannot "
2540− ++ status ++ " without input"
2541−
2542− {-# INLINE initL #-}
2543− initL mkState = do
2544− resL <- initialL
2545− case resL of
2546− Partial sL ->
2547− return $ Partial $ mkState sL
2548− Done _ ->
2549− errMsg "content" "succeed"
2550−
2551− initial = do
2552− res <- initialR
2553− case res of
2554− Partial sR -> initL (IntersperseQUnquoted sR)
2555− Done b -> return $ Done b
2556−
2557− {-# INLINE collect #-}
2558− collect nextS sR b = do
2559− res <- stepR sR b
2560− case res of
2561− Partial s ->
2562− initL (nextS s)
2563− Done c -> return (Done c)
2564−
2565− {-# INLINE process #-}
2566− process a sL sR nextState = do
2567− r <- stepL sL a
2568− case r of
2569− Partial s -> return $ Partial (nextState sR s)
2570− Done b -> collect nextState sR b
2571−
2572− {-# INLINE processQuoted #-}
2573− processQuoted a sL sR nextState = do
2574− r <- stepL sL a
2575− case r of
2576− Partial s -> return $ Partial (nextState sR s)
2577− Done _ -> error "Collecting fold finished inside quote"
2578−
2579− step (IntersperseQUnquoted sR sL) a
2580− | a == separator = do
2581− b <- extractL sL
2582− collect IntersperseQUnquoted sR b
2583− | a == quote = processQuoted a sL sR IntersperseQQuoted
2584− | otherwise = process a sL sR IntersperseQUnquoted
2585−
2586− step (IntersperseQQuoted sR sL) a
2587− | a == esc = processQuoted a sL sR IntersperseQQuotedEsc
2588− | a == quote = process a sL sR IntersperseQUnquoted
2589− | otherwise = processQuoted a sL sR IntersperseQQuoted
2590−
2591− step (IntersperseQQuotedEsc sR sL) a =
2592− processQuoted a sL sR IntersperseQQuoted
2593−
2594− extract (IntersperseQUnquoted sR _) = extractR sR
2595− extract (IntersperseQQuoted _ _) =
2596− error "intersperseWithQuotes: finished inside quote"
2597− extract (IntersperseQQuotedEsc _ _) =
2598− error "intersperseWithQuotes: finished inside quote, at escape char"
core/src/Streamly/Internal/Data/Fold/Chunked.hs+20−18
@@ -6,7 +6,7 @@
66-- Stability : experimental
77-- Portability : GHC
88--
9−-- Use "Streamly.Data.Parser.Chunked" instead.
9+-- Use "Streamly.Data.Parser" instead.
1010--
1111-- Fold a stream of foreign arrays. @Fold m a b@ in this module works
1212-- on a stream of "Array a" and produces an output of type @b@.
@@ -19,18 +19,19 @@
1919-- folds in Data.Fold to correctly work on an array stream as if it is an
2020-- element stream. For example:
2121--
22−-- >>> import qualified Streamly.Data.Fold as Fold
23−-- >>> import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
24−-- >>> import qualified Streamly.Internal.Data.Fold.Chunked as ChunkFold
25−-- >>> import qualified Streamly.Data.Stream as Stream
26−-- >>> import qualified Streamly.Data.StreamK as StreamK
22+-- >> import qualified Streamly.Data.Fold as Fold
23+-- >> import qualified Streamly.Internal.Data.Array.Stream as ArrayStream
24+-- >> import qualified Streamly.Internal.Data.Fold.Chunked as ChunkFold
25+-- >> import qualified Streamly.Data.Stream as Stream
26+-- >> import qualified Streamly.Data.StreamK as StreamK
2727--
28−-- >>> f = ChunkFold.fromFold (Fold.take 7 Fold.toList)
29−-- >>> s = Stream.chunksOf 5 $ Stream.fromList "hello world"
30−-- >>> ArrayStream.runArrayFold f (StreamK.fromStream s)
28+-- >> f = ChunkFold.fromFold (Fold.take 7 Fold.toList)
29+-- >> s = Stream.chunksOf 5 $ Stream.fromList "hello world"
30+-- >> ArrayStream.runArrayFold f (StreamK.fromStream s)
3131-- Right "hello w"
3232--
3333module Streamly.Internal.Data.Fold.Chunked
34+ {-# DEPRECATED "Please use Streamly.Data.Parser instead." #-}
3435 (
3536 ChunkFold (..)
3637
@@ -58,22 +59,22 @@ where
5859
5960#include "ArrayMacros.h"
6061
62+#if !MIN_VERSION_base(4,18,0)
6163import Control.Applicative (liftA2)
64+#endif
6265import Control.Exception (assert)
6366import Control.Monad.IO.Class (MonadIO(..))
6467import Data.Bifunctor (first)
6568import Data.Proxy (Proxy(..))
6669import Streamly.Internal.Data.Unbox (Unbox(..))
6770import GHC.Types (SPEC(..))
68−import Streamly.Internal.Data.MutArray.Type (touch)
69−import Streamly.Internal.Data.Array.Type (Array(..))
71+import Streamly.Internal.Data.Array (Array(..))
7072import Streamly.Internal.Data.Parser (Initial(..), Step(..))
7173import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
7274
73−import qualified Streamly.Internal.Data.Array as Array
75+import qualified Streamly.Internal.Data.Array.Type as Array
7476import qualified Streamly.Internal.Data.Fold as Fold
7577import qualified Streamly.Internal.Data.Parser as ParserD
76−import qualified Streamly.Internal.Data.Parser.ParserD.Type as ParserD
7778import qualified Streamly.Internal.Data.Parser as Parser
7879
7980import Prelude hiding (concatMap, take)
@@ -102,8 +103,8 @@ newtype ChunkFold m a b = ChunkFold (ParserD.Parser (Array a) m b)
102103{-# INLINE fromFold #-}
103104fromFold :: forall m a b. (MonadIO m, Unbox a) =>
104105 Fold.Fold m a b -> ChunkFold m a b
105−fromFold (Fold.Fold fstep finitial fextract) =
106− ChunkFold (ParserD.Parser step initial (fmap (Done 0) . fextract))
106+fromFold (Fold.Fold fstep finitial _ ffinal) =
107+ ChunkFold (ParserD.Parser step initial extract)
107108
108109 where
109110
@@ -123,7 +124,7 @@ fromFold (Fold.Fold fstep finitial fextract) =
123124 assert (cur == end) (return ())
124125 return $ Partial 0 fs
125126 goArray !_ !cur !fs = do
126− x <- liftIO $ peekByteIndex cur contents
127+ x <- liftIO $ peekAt cur contents
127128 res <- fstep fs x
128129 let elemSize = SIZE_OF(a)
129130 next = INDEX_NEXT(cur,a)
@@ -133,6 +134,8 @@ fromFold (Fold.Fold fstep finitial fextract) =
133134 Fold.Partial fs1 ->
134135 goArray SPEC next fs1
135136
137+ extract = fmap (Done 0) . ffinal
138+
136139-- | Convert an element 'ParserD.Parser' into an array stream fold. If the
137140-- parser fails the fold would throw an exception.
138141--
@@ -160,8 +163,7 @@ fromParserD (ParserD.Parser step1 initial1 extract1) =
160163 else return $ st (arrRem + n) fs1
161164
162165 goArray !_ !cur !fs = do
163− x <- liftIO $ peekByteIndex cur contents
164− liftIO $ touch contents
166+ x <- liftIO $ peekAt cur contents
165167 res <- step1 fs x
166168 let elemSize = SIZE_OF(a)
167169 next = INDEX_NEXT(cur,a)
core/src/Streamly/Internal/Data/Fold/Combinators.hsnew file+2452
@@ -0,0 +1,2452 @@
1+{-# LANGUAGE CPP #-}
2+-- |
3+-- Module : Streamly.Internal.Data.Fold.Combinators
4+-- Copyright : (c) 2019 Composewell Technologies
5+-- (c) 2013 Gabriel Gonzalez
6+-- License : BSD3
7+-- Maintainer : streamly@composewell.com
8+-- Stability : experimental
9+-- Portability : GHC
10+--
11+-- See "Streamly.Data.Fold" for an overview and
12+-- "Streamly.Internal.Data.Fold.Type" for design notes.
13+
14+module Streamly.Internal.Data.Fold.Combinators
15+ (
16+ -- * Mappers
17+ -- | Monadic functions useful with mapM/lmapM on folds or streams.
18+ tracing
19+ , trace
20+
21+ -- * Folds
22+
23+ -- ** Accumulators
24+ -- *** Semigroups and Monoids
25+ , sconcat
26+ , mconcat
27+ , foldMap
28+ , foldMapM
29+
30+ -- *** Reducers
31+ , drainMapM
32+ , the
33+ , mean
34+ , rollingHash
35+ , defaultSalt
36+ , rollingHashWithSalt
37+ , rollingHashFirstN
38+ -- , rollingHashLastN
39+
40+ -- *** Saturating Reducers
41+ -- | 'product' terminates if it becomes 0. Other folds can theoretically
42+ -- saturate on bounded types, and therefore terminate, however, they will
43+ -- run forever on unbounded types like Integer/Double.
44+ , sum
45+ , product
46+ , maximumBy
47+ , maximum
48+ , minimumBy
49+ , minimum
50+
51+ -- *** Collectors
52+ -- | Avoid using these folds in scalable or performance critical
53+ -- applications, they buffer all the input in GC memory which can be
54+ -- detrimental to performance if the input is large.
55+ , toListRev
56+ -- $toListRev
57+ , toStream
58+ , toStreamRev
59+ , topBy
60+ , top
61+ , bottomBy
62+ , bottom
63+
64+ -- *** Scanners
65+ -- | Stateful transformation of the elements. Useful in combination with
66+ -- the 'scanMaybe' combinator. For scanners the result of the fold is
67+ -- usually a transformation of the current element rather than an
68+ -- aggregation of all elements till now.
69+ , latest
70+ -- , nthLast -- using Ring array
71+ , indexingWith
72+ , indexing
73+ , indexingRev
74+ , rollingMapM
75+
76+ -- *** Filters
77+ -- | Useful in combination with the 'scanMaybe' combinator.
78+ , deleteBy
79+ , uniqBy
80+ , uniq
81+ , repeated
82+ , findIndices
83+ , elemIndices
84+
85+ -- *** Singleton folds
86+ -- | Folds that terminate after consuming exactly one input element. All
87+ -- these can be implemented in terms of the 'maybe' fold.
88+ , one
89+ , null -- XXX not very useful and could be problematic, remove it?
90+ , satisfy
91+ , maybe
92+
93+ -- *** Multi folds
94+ -- | Terminate after consuming one or more elements.
95+ , drainN
96+ -- , lastN
97+ -- , (!!)
98+ , indexGeneric
99+ , index
100+ , findM
101+ , find
102+ , lookup
103+ , findIndex
104+ , elemIndex
105+ , elem
106+ , notElem
107+ , all
108+ , any
109+ , and
110+ , or
111+
112+ -- ** Trimmers
113+ -- | Useful in combination with the 'scanMaybe' combinator.
114+ , takingEndByM
115+ , takingEndBy
116+ , takingEndByM_
117+ , takingEndBy_
118+ , droppingWhileM
119+ , droppingWhile
120+ , prune
121+
122+ -- * Running A Fold
123+ , drive
124+ -- , breakStream
125+
126+ -- * Building Incrementally
127+ , addStream
128+
129+ -- * Combinators
130+ -- ** Utilities
131+ , with
132+
133+ -- ** Mapping on Input
134+ , transform
135+
136+ -- ** Sliding Window
137+ , slide2
138+
139+ -- ** Scanning Input
140+ , scan
141+ , scanMany
142+ , indexed
143+
144+ -- ** Zipping Input
145+ , zipStreamWithM
146+ , zipStream
147+
148+ -- ** Filtering Input
149+ , mapMaybeM
150+ , mapMaybe
151+ , sampleFromthen
152+
153+ {-
154+ -- ** Insertion
155+ -- | Insertion adds more elements to the stream.
156+
157+ , insertBy
158+ , intersperseM
159+
160+ -- ** Reordering
161+ , reverse
162+ -}
163+
164+ -- ** Trimming
165+
166+ -- By elements
167+ , takeEndBySeq
168+ , takeEndBySeq_
169+ {-
170+ , drop
171+ , dropWhile
172+ , dropWhileM
173+ -}
174+
175+ -- ** Serial Append
176+ -- , tail
177+ -- , init
178+ , splitAt -- spanN
179+ -- , splitIn -- sessionN
180+
181+ -- ** Parallel Distribution
182+ , tee
183+ , distribute
184+ -- , distributeFst
185+ -- , distributeMin
186+
187+ -- ** Unzipping
188+ , unzip
189+ -- These two can be expressed using lmap/lmapM and unzip
190+ , unzipWith
191+ , unzipWithM
192+ , unzipWithFstM
193+ , unzipWithMinM
194+
195+ -- ** Partitioning
196+ , partitionByM
197+ , partitionByFstM
198+ , partitionByMinM
199+ , partitionBy
200+ , partition
201+
202+ -- ** Splitting
203+ , chunksBetween
204+ , intersperseWithQuotes
205+
206+ -- ** Nesting
207+ , unfoldMany
208+ , concatSequence
209+
210+ -- * Deprecated
211+ , drainBy
212+ , last
213+ , head
214+ , sequence
215+ , mapM
216+ , variance
217+ , stdDev
218+ )
219+where
220+
221+#include "inline.hs"
222+#include "ArrayMacros.h"
223+
224+import Control.Monad (void)
225+import Control.Monad.IO.Class (MonadIO(..))
226+import Data.Bifunctor (first)
227+import Data.Bits (shiftL, shiftR, (.|.), (.&.))
228+import Data.Either (isLeft, isRight, fromLeft, fromRight)
229+import Data.Int (Int64)
230+import Data.Proxy (Proxy(..))
231+import Data.Word (Word32)
232+import Foreign.Storable (Storable, peek)
233+import Streamly.Internal.Data.MutArray.Type (MutArray(..))
234+import Streamly.Internal.Data.Maybe.Strict (Maybe'(..), toMaybe)
235+import Streamly.Internal.Data.Pipe.Type (Pipe (..), PipeState(..))
236+import Streamly.Internal.Data.Unbox (Unbox, sizeOf)
237+import Streamly.Internal.Data.Unfold.Type (Unfold(..))
238+import Streamly.Internal.Data.Tuple.Strict (Tuple'(..), Tuple3'(..))
239+import Streamly.Internal.Data.Stream.Type (Stream)
240+
241+import qualified Prelude
242+import qualified Streamly.Internal.Data.MutArray.Type as MA
243+import qualified Streamly.Internal.Data.Array.Type as Array
244+import qualified Streamly.Internal.Data.Fold.Window as Fold
245+import qualified Streamly.Internal.Data.Pipe.Type as Pipe
246+import qualified Streamly.Internal.Data.Ring as Ring
247+import qualified Streamly.Internal.Data.Stream.Type as StreamD
248+
249+import Prelude hiding
250+ ( Foldable(..), filter, drop, dropWhile, take, takeWhile, zipWith
251+ , map, mapM_, sequence, all, any
252+ , notElem, head, last, tail
253+ , reverse, iterate, init, and, or, lookup, (!!)
254+ , scanl, scanl1, replicate, concatMap, mconcat, unzip
255+ , span, splitAt, break, mapM, zip, maybe)
256+import Streamly.Internal.Data.Fold.Type
257+
258+#include "DocTestDataFold.hs"
259+
260+------------------------------------------------------------------------------
261+-- Running
262+------------------------------------------------------------------------------
263+
264+-- | Drive a fold using the supplied 'Stream', reducing the resulting
265+-- expression strictly at each step.
266+--
267+-- Definition:
268+--
269+-- >>> drive = flip Stream.fold
270+--
271+-- Example:
272+--
273+-- >>> Fold.drive (Stream.enumerateFromTo 1 100) Fold.sum
274+-- 5050
275+--
276+{-# INLINE drive #-}
277+drive :: Monad m => Stream m a -> Fold m a b -> m b
278+drive = flip StreamD.fold
279+
280+{-
281+-- | Like 'drive' but also returns the remaining stream. The resulting stream
282+-- would be 'Stream.nil' if the stream finished before the fold.
283+--
284+-- Definition:
285+--
286+-- >>> breakStream = flip Stream.foldBreak
287+--
288+-- /CPS/
289+--
290+{-# INLINE breakStreamK #-}
291+breakStreamK :: Monad m => StreamK m a -> Fold m a b -> m (b, StreamK m a)
292+breakStreamK strm fl = fmap f $ K.foldBreak fl (Stream.toStreamK strm)
293+
294+ where
295+
296+ f (b, str) = (b, Stream.fromStreamK str)
297+-}
298+
299+-- | Append a stream to a fold to build the fold accumulator incrementally. We
300+-- can repeatedly call 'addStream' on the same fold to continue building the
301+-- fold and finally use 'drive' to finish the fold and extract the result. Also
302+-- see the 'Streamly.Data.Fold.addOne' operation which is a singleton version
303+-- of 'addStream'.
304+--
305+-- Definitions:
306+--
307+-- >>> addStream stream = Fold.drive stream . Fold.duplicate
308+--
309+-- Example, build a list incrementally:
310+--
311+-- >>> :{
312+-- pure (Fold.toList :: Fold IO Int [Int])
313+-- >>= Fold.addOne 1
314+-- >>= Fold.addStream (Stream.enumerateFromTo 2 4)
315+-- >>= Fold.drive Stream.nil
316+-- >>= print
317+-- :}
318+-- [1,2,3,4]
319+--
320+-- This can be used as an O(n) list append compared to the O(n^2) @++@ when
321+-- used for incrementally building a list.
322+--
323+-- Example, build a stream incrementally:
324+--
325+-- >>> :{
326+-- pure (Fold.toStream :: Fold IO Int (Stream Identity Int))
327+-- >>= Fold.addOne 1
328+-- >>= Fold.addStream (Stream.enumerateFromTo 2 4)
329+-- >>= Fold.drive Stream.nil
330+-- >>= print
331+-- :}
332+-- fromList [1,2,3,4]
333+--
334+-- This can be used as an O(n) stream append compared to the O(n^2) @<>@ when
335+-- used for incrementally building a stream.
336+--
337+-- Example, build an array incrementally:
338+--
339+-- >>> :{
340+-- pure (Array.write :: Fold IO Int (Array Int))
341+-- >>= Fold.addOne 1
342+-- >>= Fold.addStream (Stream.enumerateFromTo 2 4)
343+-- >>= Fold.drive Stream.nil
344+-- >>= print
345+-- :}
346+-- fromList [1,2,3,4]
347+--
348+-- Example, build an array stream incrementally:
349+--
350+-- >>> :{
351+-- let f :: Fold IO Int (Stream Identity (Array Int))
352+-- f = Fold.groupsOf 2 (Array.writeN 3) Fold.toStream
353+-- in pure f
354+-- >>= Fold.addOne 1
355+-- >>= Fold.addStream (Stream.enumerateFromTo 2 4)
356+-- >>= Fold.drive Stream.nil
357+-- >>= print
358+-- :}
359+-- fromList [fromList [1,2],fromList [3,4]]
360+--
361+addStream :: Monad m => Stream m a -> Fold m a b -> m (Fold m a b)
362+addStream stream = drive stream . duplicate
363+
364+------------------------------------------------------------------------------
365+-- Transformations on fold inputs
366+------------------------------------------------------------------------------
367+
368+-- | Flatten the monadic output of a fold to pure output.
369+--
370+{-# DEPRECATED sequence "Use \"rmapM id\" instead" #-}
371+{-# INLINE sequence #-}
372+sequence :: Monad m => Fold m a (m b) -> Fold m a b
373+sequence = rmapM id
374+
375+-- | Map a monadic function on the output of a fold.
376+--
377+{-# DEPRECATED mapM "Use rmapM instead" #-}
378+{-# INLINE mapM #-}
379+mapM :: Monad m => (b -> m c) -> Fold m a b -> Fold m a c
380+mapM = rmapM
381+
382+-- |
383+-- >>> mapMaybeM f = Fold.lmapM f . Fold.catMaybes
384+--
385+{-# INLINE mapMaybeM #-}
386+mapMaybeM :: Monad m => (a -> m (Maybe b)) -> Fold m b r -> Fold m a r
387+mapMaybeM f = lmapM f . catMaybes
388+
389+-- | @mapMaybe f fold@ maps a 'Maybe' returning function @f@ on the input of
390+-- the fold, filters out 'Nothing' elements, and return the values extracted
391+-- from 'Just'.
392+--
393+-- >>> mapMaybe f = Fold.lmap f . Fold.catMaybes
394+-- >>> mapMaybe f = Fold.mapMaybeM (return . f)
395+--
396+-- >>> f x = if even x then Just x else Nothing
397+-- >>> fld = Fold.mapMaybe f Fold.toList
398+-- >>> Stream.fold fld (Stream.enumerateFromTo 1 10)
399+-- [2,4,6,8,10]
400+--
401+{-# INLINE mapMaybe #-}
402+mapMaybe :: Monad m => (a -> Maybe b) -> Fold m b r -> Fold m a r
403+mapMaybe f = lmap f . catMaybes
404+
405+------------------------------------------------------------------------------
406+-- Transformations on fold inputs
407+------------------------------------------------------------------------------
408+
409+-- | Apply a monadic function on the input and return the input.
410+--
411+-- >>> Stream.fold (Fold.lmapM (Fold.tracing print) Fold.drain) $ (Stream.enumerateFromTo (1 :: Int) 2)
412+-- 1
413+-- 2
414+--
415+-- /Pre-release/
416+--
417+{-# INLINE tracing #-}
418+tracing :: Monad m => (a -> m b) -> (a -> m a)
419+tracing f x = void (f x) >> return x
420+
421+-- | Apply a monadic function to each element flowing through and discard the
422+-- results.
423+--
424+-- >>> Stream.fold (Fold.trace print Fold.drain) $ (Stream.enumerateFromTo (1 :: Int) 2)
425+-- 1
426+-- 2
427+--
428+-- >>> trace f = Fold.lmapM (Fold.tracing f)
429+--
430+-- /Pre-release/
431+{-# INLINE trace #-}
432+trace :: Monad m => (a -> m b) -> Fold m a r -> Fold m a r
433+trace f = lmapM (tracing f)
434+
435+-- rename to lpipe?
436+--
437+-- | Apply a transformation on a 'Fold' using a 'Pipe'.
438+--
439+-- /Pre-release/
440+{-# INLINE transform #-}
441+transform :: Monad m => Pipe m a b -> Fold m b c -> Fold m a c
442+transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract ffinal) =
443+ Fold step initial extract final
444+
445+ where
446+
447+ initial = first (Tuple' pinitial) <$> finitial
448+
449+ step (Tuple' ps fs) x = do
450+ r <- pstep1 ps x
451+ go fs r
452+
453+ where
454+
455+ -- XXX use SPEC?
456+ go acc (Pipe.Yield b (Consume ps')) = do
457+ acc' <- fstep acc b
458+ return
459+ $ case acc' of
460+ Partial s -> Partial $ Tuple' ps' s
461+ Done b2 -> Done b2
462+ go acc (Pipe.Yield b (Produce ps')) = do
463+ acc' <- fstep acc b
464+ r <- pstep2 ps'
465+ case acc' of
466+ Partial s -> go s r
467+ Done b2 -> return $ Done b2
468+ go acc (Pipe.Continue (Consume ps')) =
469+ return $ Partial $ Tuple' ps' acc
470+ go acc (Pipe.Continue (Produce ps')) = do
471+ r <- pstep2 ps'
472+ go acc r
473+
474+ extract (Tuple' _ fs) = fextract fs
475+
476+ final (Tuple' _ fs) = ffinal fs
477+
478+{-# INLINE scanWith #-}
479+scanWith :: Monad m => Bool -> Fold m a b -> Fold m b c -> Fold m a c
480+scanWith isMany
481+ (Fold stepL initialL extractL finalL)
482+ (Fold stepR initialR extractR finalR) =
483+ Fold step initial extract final
484+
485+ where
486+
487+ {-# INLINE runStep #-}
488+ runStep actionL sR = do
489+ rL <- actionL
490+ case rL of
491+ Done bL -> do
492+ rR <- stepR sR bL
493+ case rR of
494+ Partial sR1 ->
495+ if isMany
496+ then runStep initialL sR1
497+ else Done <$> finalR sR1
498+ Done bR -> return $ Done bR
499+ Partial sL -> do
500+ !b <- extractL sL
501+ rR <- stepR sR b
502+ case rR of
503+ Partial sR1 -> return $ Partial (sL, sR1)
504+ Done bR -> finalL sL >> return (Done bR)
505+
506+ initial = do
507+ r <- initialR
508+ case r of
509+ Partial sR -> runStep initialL sR
510+ Done b -> return $ Done b
511+
512+ step (sL, sR) x = runStep (stepL sL x) sR
513+
514+ extract = extractR . snd
515+
516+ final (sL, sR) = finalL sL *> finalR sR
517+
518+-- | Scan the input of a 'Fold' to change it in a stateful manner using another
519+-- 'Fold'. The scan stops as soon as the fold terminates.
520+--
521+-- /Pre-release/
522+{-# INLINE scan #-}
523+scan :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
524+scan = scanWith False
525+
526+-- XXX This does not fuse beacuse of the recursive step. Need to investigate.
527+--
528+-- | Scan the input of a 'Fold' to change it in a stateful manner using another
529+-- 'Fold'. The scan restarts with a fresh state if the fold terminates.
530+--
531+-- /Pre-release/
532+{-# INLINE scanMany #-}
533+scanMany :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
534+scanMany = scanWith True
535+
536+------------------------------------------------------------------------------
537+-- Filters
538+------------------------------------------------------------------------------
539+
540+-- | Returns the latest element omitting the first occurrence that satisfies
541+-- the given equality predicate.
542+--
543+-- Example:
544+--
545+-- >>> input = Stream.fromList [1,3,3,5]
546+-- >>> Stream.fold Fold.toList $ Stream.scanMaybe (Fold.deleteBy (==) 3) input
547+-- [1,3,5]
548+--
549+{-# INLINE_NORMAL deleteBy #-}
550+deleteBy :: Monad m => (a -> a -> Bool) -> a -> Fold m a (Maybe a)
551+deleteBy eq x0 = fmap extract $ foldl' step (Tuple' False Nothing)
552+
553+ where
554+
555+ step (Tuple' False _) x =
556+ if eq x x0
557+ then Tuple' True Nothing
558+ else Tuple' False (Just x)
559+ step (Tuple' True _) x = Tuple' True (Just x)
560+
561+ extract (Tuple' _ x) = x
562+
563+-- | Provide a sliding window of length 2 elements.
564+--
565+-- See "Streamly.Internal.Data.Fold.Window".
566+--
567+{-# INLINE slide2 #-}
568+slide2 :: Monad m => Fold m (a, Maybe a) b -> Fold m a b
569+slide2 (Fold step1 initial1 extract1 final1) = Fold step initial extract final
570+
571+ where
572+
573+ initial =
574+ first (Tuple' Nothing) <$> initial1
575+
576+ step (Tuple' prev s) cur =
577+ first (Tuple' (Just cur)) <$> step1 s (cur, prev)
578+
579+ extract (Tuple' _ s) = extract1 s
580+
581+ final (Tuple' _ s) = final1 s
582+
583+-- | Return the latest unique element using the supplied comparison function.
584+-- Returns 'Nothing' if the current element is same as the last element
585+-- otherwise returns 'Just'.
586+--
587+-- Example, strip duplicate path separators:
588+--
589+-- >>> input = Stream.fromList "//a//b"
590+-- >>> f x y = x == '/' && y == '/'
591+-- >>> Stream.fold Fold.toList $ Stream.scanMaybe (Fold.uniqBy f) input
592+-- "/a/b"
593+--
594+-- Space: @O(1)@
595+--
596+-- /Pre-release/
597+--
598+{-# INLINE uniqBy #-}
599+uniqBy :: Monad m => (a -> a -> Bool) -> Fold m a (Maybe a)
600+uniqBy eq = rollingMap f
601+
602+ where
603+
604+ f pre curr =
605+ case pre of
606+ Nothing -> Just curr
607+ Just x -> if x `eq` curr then Nothing else Just curr
608+
609+-- | See 'uniqBy'.
610+--
611+-- Definition:
612+--
613+-- >>> uniq = Fold.uniqBy (==)
614+--
615+{-# INLINE uniq #-}
616+uniq :: (Monad m, Eq a) => Fold m a (Maybe a)
617+uniq = uniqBy (==)
618+
619+-- | Strip all leading and trailing occurrences of an element passing a
620+-- predicate and make all other consecutive occurrences uniq.
621+--
622+-- >> prune p = Stream.dropWhileAround p $ Stream.uniqBy (x y -> p x && p y)
623+--
624+-- @
625+-- > Stream.prune isSpace (Stream.fromList " hello world! ")
626+-- "hello world!"
627+--
628+-- @
629+--
630+-- Space: @O(1)@
631+--
632+-- /Unimplemented/
633+{-# INLINE prune #-}
634+prune ::
635+ -- (Monad m, Eq a) =>
636+ (a -> Bool) -> Fold m a (Maybe a)
637+prune = error "Not implemented yet!"
638+
639+-- | Emit only repeated elements, once.
640+--
641+-- /Unimplemented/
642+repeated :: -- (Monad m, Eq a) =>
643+ Fold m a (Maybe a)
644+repeated = error "Not implemented yet!"
645+
646+------------------------------------------------------------------------------
647+-- Left folds
648+------------------------------------------------------------------------------
649+
650+------------------------------------------------------------------------------
651+-- Run Effects
652+------------------------------------------------------------------------------
653+
654+-- |
655+-- Definitions:
656+--
657+-- >>> drainMapM f = Fold.lmapM f Fold.drain
658+-- >>> drainMapM f = Fold.foldMapM (void . f)
659+--
660+-- Drain all input after passing it through a monadic function. This is the
661+-- dual of mapM_ on stream producers.
662+--
663+{-# INLINE drainMapM #-}
664+drainMapM :: Monad m => (a -> m b) -> Fold m a ()
665+drainMapM f = lmapM f drain
666+
667+{-# DEPRECATED drainBy "Please use 'drainMapM' instead." #-}
668+{-# INLINE drainBy #-}
669+drainBy :: Monad m => (a -> m b) -> Fold m a ()
670+drainBy = drainMapM
671+
672+-- | Returns the latest element of the input stream, if any.
673+--
674+-- >>> latest = Fold.foldl1' (\_ x -> x)
675+-- >>> latest = fmap getLast $ Fold.foldMap (Last . Just)
676+--
677+{-# INLINE latest #-}
678+latest :: Monad m => Fold m a (Maybe a)
679+latest = foldl1' (\_ x -> x)
680+
681+{-# DEPRECATED last "Please use 'latest' instead." #-}
682+{-# INLINE last #-}
683+last :: Monad m => Fold m a (Maybe a)
684+last = latest
685+
686+-- | Terminates with 'Nothing' as soon as it finds an element different than
687+-- the previous one, returns 'the' element if the entire input consists of the
688+-- same element.
689+--
690+{-# INLINE the #-}
691+the :: (Monad m, Eq a) => Fold m a (Maybe a)
692+the = foldt' step initial id
693+
694+ where
695+
696+ initial = Partial Nothing
697+
698+ step Nothing x = Partial (Just x)
699+ step old@(Just x0) x =
700+ if x0 == x
701+ then Partial old
702+ else Done Nothing
703+
704+------------------------------------------------------------------------------
705+-- To Summary
706+------------------------------------------------------------------------------
707+
708+-- | Determine the sum of all elements of a stream of numbers. Returns additive
709+-- identity (@0@) when the stream is empty. Note that this is not numerically
710+-- stable for floating point numbers.
711+--
712+-- >>> sum = Fold.cumulative Fold.windowSum
713+--
714+-- Same as following but numerically stable:
715+--
716+-- >>> sum = Fold.foldl' (+) 0
717+-- >>> sum = fmap Data.Monoid.getSum $ Fold.foldMap Data.Monoid.Sum
718+--
719+{-# INLINE sum #-}
720+sum :: (Monad m, Num a) => Fold m a a
721+sum = Fold.cumulative Fold.windowSum
722+
723+-- | Determine the product of all elements of a stream of numbers. Returns
724+-- multiplicative identity (@1@) when the stream is empty. The fold terminates
725+-- when it encounters (@0@) in its input.
726+--
727+-- Same as the following but terminates on multiplication by @0@:
728+--
729+-- >>> product = fmap Data.Monoid.getProduct $ Fold.foldMap Data.Monoid.Product
730+--
731+{-# INLINE product #-}
732+product :: (Monad m, Num a, Eq a) => Fold m a a
733+product = foldt' step (Partial 1) id
734+
735+ where
736+
737+ step x a =
738+ if a == 0
739+ then Done 0
740+ else Partial $ x * a
741+
742+------------------------------------------------------------------------------
743+-- To Summary (Maybe)
744+------------------------------------------------------------------------------
745+
746+-- | Determine the maximum element in a stream using the supplied comparison
747+-- function.
748+--
749+{-# INLINE maximumBy #-}
750+maximumBy :: Monad m => (a -> a -> Ordering) -> Fold m a (Maybe a)
751+maximumBy cmp = foldl1' max'
752+
753+ where
754+
755+ max' x y =
756+ case cmp x y of
757+ GT -> x
758+ _ -> y
759+
760+-- | Determine the maximum element in a stream.
761+--
762+-- Definitions:
763+--
764+-- >>> maximum = Fold.maximumBy compare
765+-- >>> maximum = Fold.foldl1' max
766+--
767+-- Same as the following but without a default maximum. The 'Max' Monoid uses
768+-- the 'minBound' as the default maximum:
769+--
770+-- >>> maximum = fmap Data.Semigroup.getMax $ Fold.foldMap Data.Semigroup.Max
771+--
772+{-# INLINE maximum #-}
773+maximum :: (Monad m, Ord a) => Fold m a (Maybe a)
774+maximum = foldl1' max
775+
776+-- | Computes the minimum element with respect to the given comparison function
777+--
778+{-# INLINE minimumBy #-}
779+minimumBy :: Monad m => (a -> a -> Ordering) -> Fold m a (Maybe a)
780+minimumBy cmp = foldl1' min'
781+
782+ where
783+
784+ min' x y =
785+ case cmp x y of
786+ GT -> y
787+ _ -> x
788+
789+-- | Determine the minimum element in a stream using the supplied comparison
790+-- function.
791+--
792+-- Definitions:
793+--
794+-- >>> minimum = Fold.minimumBy compare
795+-- >>> minimum = Fold.foldl1' min
796+--
797+-- Same as the following but without a default minimum. The 'Min' Monoid uses the
798+-- 'maxBound' as the default maximum:
799+--
800+-- >>> maximum = fmap Data.Semigroup.getMin $ Fold.foldMap Data.Semigroup.Min
801+--
802+{-# INLINE minimum #-}
803+minimum :: (Monad m, Ord a) => Fold m a (Maybe a)
804+minimum = foldl1' min
805+
806+------------------------------------------------------------------------------
807+-- To Summary (Statistical)
808+------------------------------------------------------------------------------
809+
810+-- | Compute a numerically stable arithmetic mean of all elements in the input
811+-- stream.
812+--
813+{-# INLINE mean #-}
814+mean :: (Monad m, Fractional a) => Fold m a a
815+mean = fmap done $ foldl' step begin
816+
817+ where
818+
819+ begin = Tuple' 0 0
820+
821+ step (Tuple' x n) y =
822+ let n1 = n + 1
823+ in Tuple' (x + (y - x) / n1) n1
824+
825+ done (Tuple' x _) = x
826+
827+-- | Compute a numerically stable (population) variance over all elements in
828+-- the input stream.
829+--
830+{-# DEPRECATED variance "Use the streamly-statistics package instead" #-}
831+{-# INLINE variance #-}
832+variance :: (Monad m, Fractional a) => Fold m a a
833+variance = fmap done $ foldl' step begin
834+
835+ where
836+
837+ begin = Tuple3' 0 0 0
838+
839+ step (Tuple3' n mean_ m2) x = Tuple3' n' mean' m2'
840+
841+ where
842+
843+ n' = n + 1
844+ mean' = (n * mean_ + x) / (n + 1)
845+ delta = x - mean_
846+ m2' = m2 + delta * delta * n / (n + 1)
847+
848+ done (Tuple3' n _ m2) = m2 / n
849+
850+-- | Compute a numerically stable (population) standard deviation over all
851+-- elements in the input stream.
852+--
853+{-# DEPRECATED stdDev "Use the streamly-statistics package instead" #-}
854+{-# INLINE stdDev #-}
855+stdDev :: (Monad m, Floating a) => Fold m a a
856+stdDev = sqrt <$> variance
857+
858+-- | Compute an 'Int' sized polynomial rolling hash
859+--
860+-- > H = salt * k ^ n + c1 * k ^ (n - 1) + c2 * k ^ (n - 2) + ... + cn * k ^ 0
861+--
862+-- Where @c1@, @c2@, @cn@ are the elements in the input stream and @k@ is a
863+-- constant.
864+--
865+-- This hash is often used in Rabin-Karp string search algorithm.
866+--
867+-- See https://en.wikipedia.org/wiki/Rolling_hash
868+--
869+{-# INLINE rollingHashWithSalt #-}
870+rollingHashWithSalt :: (Monad m, Enum a) => Int64 -> Fold m a Int64
871+rollingHashWithSalt = foldl' step
872+
873+ where
874+
875+ k = 2891336453 :: Int64
876+
877+ step cksum a = cksum * k + fromIntegral (fromEnum a)
878+
879+-- | A default salt used in the implementation of 'rollingHash'.
880+{-# INLINE defaultSalt #-}
881+defaultSalt :: Int64
882+defaultSalt = -2578643520546668380
883+
884+-- | Compute an 'Int' sized polynomial rolling hash of a stream.
885+--
886+-- >>> rollingHash = Fold.rollingHashWithSalt Fold.defaultSalt
887+--
888+{-# INLINE rollingHash #-}
889+rollingHash :: (Monad m, Enum a) => Fold m a Int64
890+rollingHash = rollingHashWithSalt defaultSalt
891+
892+-- | Compute an 'Int' sized polynomial rolling hash of the first n elements of
893+-- a stream.
894+--
895+-- >>> rollingHashFirstN n = Fold.take n Fold.rollingHash
896+--
897+-- /Pre-release/
898+{-# INLINE rollingHashFirstN #-}
899+rollingHashFirstN :: (Monad m, Enum a) => Int -> Fold m a Int64
900+rollingHashFirstN n = take n rollingHash
901+
902+-- XXX Compare this with the implementation in Fold.Window, preferrably use the
903+-- latter if performance is good.
904+
905+-- | Apply a function on every two successive elements of a stream. The first
906+-- argument of the map function is the previous element and the second argument
907+-- is the current element. When processing the very first element in the
908+-- stream, the previous element is 'Nothing'.
909+--
910+-- /Pre-release/
911+--
912+{-# INLINE rollingMapM #-}
913+rollingMapM :: Monad m => (Maybe a -> a -> m b) -> Fold m a b
914+rollingMapM f = Fold step initial extract extract
915+
916+ where
917+
918+ -- XXX We need just a postscan. We do not need an initial result here.
919+ -- Or we can supply a default initial result as an argument to rollingMapM.
920+ initial = return $ Partial (Nothing, error "Empty stream")
921+
922+ step (prev, _) cur = do
923+ x <- f prev cur
924+ return $ Partial (Just cur, x)
925+
926+ extract = return . snd
927+
928+-- |
929+-- >>> rollingMap f = Fold.rollingMapM (\x y -> return $ f x y)
930+--
931+{-# INLINE rollingMap #-}
932+rollingMap :: Monad m => (Maybe a -> a -> b) -> Fold m a b
933+rollingMap f = rollingMapM (\x y -> return $ f x y)
934+
935+------------------------------------------------------------------------------
936+-- Monoidal left folds
937+------------------------------------------------------------------------------
938+
939+-- | Semigroup concat. Append the elements of an input stream to a provided
940+-- starting value.
941+--
942+-- Definition:
943+--
944+-- >>> sconcat = Fold.foldl' (<>)
945+--
946+-- >>> semigroups = fmap Data.Monoid.Sum $ Stream.enumerateFromTo 1 10
947+-- >>> Stream.fold (Fold.sconcat 10) semigroups
948+-- Sum {getSum = 65}
949+--
950+{-# INLINE sconcat #-}
951+sconcat :: (Monad m, Semigroup a) => a -> Fold m a a
952+sconcat = foldl' (<>)
953+
954+-- | Monoid concat. Fold an input stream consisting of monoidal elements using
955+-- 'mappend' and 'mempty'.
956+--
957+-- Definition:
958+--
959+-- >>> mconcat = Fold.sconcat mempty
960+--
961+-- >>> monoids = fmap Data.Monoid.Sum $ Stream.enumerateFromTo 1 10
962+-- >>> Stream.fold Fold.mconcat monoids
963+-- Sum {getSum = 55}
964+--
965+{-# INLINE mconcat #-}
966+mconcat ::
967+ ( Monad m
968+ , Monoid a) => Fold m a a
969+mconcat = sconcat mempty
970+
971+-- |
972+-- Definition:
973+--
974+-- >>> foldMap f = Fold.lmap f Fold.mconcat
975+--
976+-- Make a fold from a pure function that folds the output of the function
977+-- using 'mappend' and 'mempty'.
978+--
979+-- >>> sum = Fold.foldMap Data.Monoid.Sum
980+-- >>> Stream.fold sum $ Stream.enumerateFromTo 1 10
981+-- Sum {getSum = 55}
982+--
983+{-# INLINE foldMap #-}
984+foldMap :: (Monad m, Monoid b) => (a -> b) -> Fold m a b
985+foldMap f = lmap f mconcat
986+
987+-- |
988+-- Definition:
989+--
990+-- >>> foldMapM f = Fold.lmapM f Fold.mconcat
991+--
992+-- Make a fold from a monadic function that folds the output of the function
993+-- using 'mappend' and 'mempty'.
994+--
995+-- >>> sum = Fold.foldMapM (return . Data.Monoid.Sum)
996+-- >>> Stream.fold sum $ Stream.enumerateFromTo 1 10
997+-- Sum {getSum = 55}
998+--
999+{-# INLINE foldMapM #-}
1000+foldMapM :: (Monad m, Monoid b) => (a -> m b) -> Fold m a b
1001+foldMapM act = foldlM' step (pure mempty)
1002+
1003+ where
1004+
1005+ step m a = do
1006+ m' <- act a
1007+ return $! mappend m m'
1008+
1009+------------------------------------------------------------------------------
1010+-- To Containers
1011+------------------------------------------------------------------------------
1012+
1013+-- $toListRev
1014+-- This is more efficient than 'Streamly.Internal.Data.Fold.toList'. toList is
1015+-- exactly the same as reversing the list after 'toListRev'.
1016+
1017+-- | Buffers the input stream to a list in the reverse order of the input.
1018+--
1019+-- Definition:
1020+--
1021+-- >>> toListRev = Fold.foldl' (flip (:)) []
1022+--
1023+-- /Warning!/ working on large lists accumulated as buffers in memory could be
1024+-- very inefficient, consider using "Streamly.Array" instead.
1025+--
1026+
1027+-- xn : ... : x2 : x1 : []
1028+{-# INLINE toListRev #-}
1029+toListRev :: Monad m => Fold m a [a]
1030+toListRev = foldl' (flip (:)) []
1031+
1032+------------------------------------------------------------------------------
1033+-- Partial Folds
1034+------------------------------------------------------------------------------
1035+
1036+-- | A fold that drains the first n elements of its input, running the effects
1037+-- and discarding the results.
1038+--
1039+-- Definition:
1040+--
1041+-- >>> drainN n = Fold.take n Fold.drain
1042+--
1043+-- /Pre-release/
1044+{-# INLINE drainN #-}
1045+drainN :: Monad m => Int -> Fold m a ()
1046+drainN n = take n drain
1047+
1048+------------------------------------------------------------------------------
1049+-- To Elements
1050+------------------------------------------------------------------------------
1051+
1052+-- | Like 'index', except with a more general 'Integral' argument
1053+--
1054+-- /Pre-release/
1055+{-# INLINE indexGeneric #-}
1056+indexGeneric :: (Integral i, Monad m) => i -> Fold m a (Maybe a)
1057+indexGeneric i = foldt' step (Partial 0) (const Nothing)
1058+
1059+ where
1060+
1061+ step j a =
1062+ if i == j
1063+ then Done $ Just a
1064+ else Partial (j + 1)
1065+
1066+-- | Return the element at the given index.
1067+--
1068+-- Definition:
1069+--
1070+-- >>> index = Fold.indexGeneric
1071+--
1072+{-# INLINE index #-}
1073+index :: Monad m => Int -> Fold m a (Maybe a)
1074+index = indexGeneric
1075+
1076+-- | Consume a single input and transform it using the supplied 'Maybe'
1077+-- returning function.
1078+--
1079+-- /Pre-release/
1080+--
1081+{-# INLINE maybe #-}
1082+maybe :: Monad m => (a -> Maybe b) -> Fold m a (Maybe b)
1083+maybe f = foldt' (const (Done . f)) (Partial Nothing) id
1084+
1085+-- | Consume a single element and return it if it passes the predicate else
1086+-- return 'Nothing'.
1087+--
1088+-- Definition:
1089+--
1090+-- >>> satisfy f = Fold.maybe (\a -> if f a then Just a else Nothing)
1091+--
1092+-- /Pre-release/
1093+{-# INLINE satisfy #-}
1094+satisfy :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
1095+satisfy f = maybe (\a -> if f a then Just a else Nothing)
1096+{-
1097+satisfy f = Fold step (return $ Partial ()) (const (return Nothing))
1098+
1099+ where
1100+
1101+ step () a = return $ Done $ if f a then Just a else Nothing
1102+-}
1103+
1104+-- Naming notes:
1105+--
1106+-- "head" and "next" are two alternative names for the same API. head sounds
1107+-- apt in the context of lists but next sounds more apt in the context of
1108+-- streams where we think in terms of generating and consuming the next element
1109+-- rather than taking the head of some static/persistent structure.
1110+--
1111+-- We also want to keep the nomenclature consistent across folds and parsers,
1112+-- "head" becomes even more unintuitive for parsers because there are two
1113+-- possible variants viz. peek and next.
1114+--
1115+-- Also, the "head" fold creates confusion in situations like
1116+-- https://github.com/composewell/streamly/issues/1404 where intuitive
1117+-- expectation from head is to consume the entire stream and just give us the
1118+-- head. There we want to convey the notion that we consume one element from
1119+-- the stream and stop. The name "one" already being used in parsers for this
1120+-- purpose sounds more apt from this perspective.
1121+--
1122+-- The source of confusion is perhaps due to the fact that some folds consume
1123+-- the entire stream and others terminate early. It may have been clearer if we
1124+-- had separate abstractions for the two use cases.
1125+
1126+-- XXX We can possibly use "head" for the purposes of reducing the entire
1127+-- stream to the head element i.e. take the head and drain the rest.
1128+
1129+-- | Take one element from the stream and stop.
1130+--
1131+-- Definition:
1132+--
1133+-- >>> one = Fold.maybe Just
1134+--
1135+-- This is similar to the stream 'Stream.uncons' operation.
1136+--
1137+{-# INLINE one #-}
1138+one :: Monad m => Fold m a (Maybe a)
1139+one = maybe Just
1140+
1141+-- | Extract the first element of the stream, if any.
1142+--
1143+-- >>> head = Fold.one
1144+--
1145+{-# DEPRECATED head "Please use \"one\" instead" #-}
1146+{-# INLINE head #-}
1147+head :: Monad m => Fold m a (Maybe a)
1148+head = one
1149+
1150+-- | Returns the first element that satisfies the given predicate.
1151+--
1152+-- /Pre-release/
1153+{-# INLINE findM #-}
1154+findM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1155+findM predicate =
1156+ Fold step (return $ Partial ()) extract extract
1157+
1158+ where
1159+
1160+ step () a =
1161+ let f r =
1162+ if r
1163+ then Done (Just a)
1164+ else Partial ()
1165+ in f <$> predicate a
1166+
1167+ extract = const $ return Nothing
1168+
1169+-- | Returns the first element that satisfies the given predicate.
1170+--
1171+{-# INLINE find #-}
1172+find :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
1173+find p = findM (return . p)
1174+
1175+-- | In a stream of (key-value) pairs @(a, b)@, return the value @b@ of the
1176+-- first pair where the key equals the given value @a@.
1177+--
1178+-- Definition:
1179+--
1180+-- >>> lookup x = fmap snd <$> Fold.find ((== x) . fst)
1181+--
1182+{-# INLINE lookup #-}
1183+lookup :: (Eq a, Monad m) => a -> Fold m (a,b) (Maybe b)
1184+lookup a0 = foldt' step (Partial ()) (const Nothing)
1185+
1186+ where
1187+
1188+ step () (a, b) =
1189+ if a == a0
1190+ then Done $ Just b
1191+ else Partial ()
1192+
1193+-- | Returns the first index that satisfies the given predicate.
1194+--
1195+{-# INLINE findIndex #-}
1196+findIndex :: Monad m => (a -> Bool) -> Fold m a (Maybe Int)
1197+findIndex predicate = foldt' step (Partial 0) (const Nothing)
1198+
1199+ where
1200+
1201+ step i a =
1202+ if predicate a
1203+ then Done $ Just i
1204+ else Partial (i + 1)
1205+
1206+-- | Returns the index of the latest element if the element satisfies the given
1207+-- predicate.
1208+--
1209+{-# INLINE findIndices #-}
1210+findIndices :: Monad m => (a -> Bool) -> Fold m a (Maybe Int)
1211+findIndices predicate =
1212+ -- XXX implement by combining indexing and filtering scans
1213+ fmap (either (const Nothing) Just) $ foldl' step (Left (-1))
1214+
1215+ where
1216+
1217+ step i a =
1218+ if predicate a
1219+ then Right (either id id i + 1)
1220+ else Left (either id id i + 1)
1221+
1222+-- | Returns the index of the latest element if the element matches the given
1223+-- value.
1224+--
1225+-- Definition:
1226+--
1227+-- >>> elemIndices a = Fold.findIndices (== a)
1228+--
1229+{-# INLINE elemIndices #-}
1230+elemIndices :: (Monad m, Eq a) => a -> Fold m a (Maybe Int)
1231+elemIndices a = findIndices (== a)
1232+
1233+-- | Returns the first index where a given value is found in the stream.
1234+--
1235+-- Definition:
1236+--
1237+-- >>> elemIndex a = Fold.findIndex (== a)
1238+--
1239+{-# INLINE elemIndex #-}
1240+elemIndex :: (Eq a, Monad m) => a -> Fold m a (Maybe Int)
1241+elemIndex a = findIndex (== a)
1242+
1243+------------------------------------------------------------------------------
1244+-- To Boolean
1245+------------------------------------------------------------------------------
1246+
1247+-- Similar to 'eof' parser, but the fold consumes and discards an input element
1248+-- when not at eof. XXX Remove or Rename to "eof"?
1249+
1250+-- | Consume one element, return 'True' if successful else return 'False'. In
1251+-- other words, test if the input is empty or not.
1252+--
1253+-- WARNING! It consumes one element if the stream is not empty. If that is not
1254+-- what you want please use the eof parser instead.
1255+--
1256+-- Definition:
1257+--
1258+-- >>> null = fmap isJust Fold.one
1259+--
1260+{-# INLINE null #-}
1261+null :: Monad m => Fold m a Bool
1262+null = foldt' (\() _ -> Done False) (Partial ()) (const True)
1263+
1264+-- | Returns 'True' if any element of the input satisfies the predicate.
1265+--
1266+-- Definition:
1267+--
1268+-- >>> any p = Fold.lmap p Fold.or
1269+--
1270+-- Example:
1271+--
1272+-- >>> Stream.fold (Fold.any (== 0)) $ Stream.fromList [1,0,1]
1273+-- True
1274+--
1275+{-# INLINE any #-}
1276+any :: Monad m => (a -> Bool) -> Fold m a Bool
1277+any predicate = foldt' step initial id
1278+
1279+ where
1280+
1281+ initial = Partial False
1282+
1283+ step _ a =
1284+ if predicate a
1285+ then Done True
1286+ else Partial False
1287+
1288+-- | Return 'True' if the given element is present in the stream.
1289+--
1290+-- Definition:
1291+--
1292+-- >>> elem a = Fold.any (== a)
1293+--
1294+{-# INLINE elem #-}
1295+elem :: (Eq a, Monad m) => a -> Fold m a Bool
1296+elem a = any (== a)
1297+
1298+-- | Returns 'True' if all elements of the input satisfy the predicate.
1299+--
1300+-- Definition:
1301+--
1302+-- >>> all p = Fold.lmap p Fold.and
1303+--
1304+-- Example:
1305+--
1306+-- >>> Stream.fold (Fold.all (== 0)) $ Stream.fromList [1,0,1]
1307+-- False
1308+--
1309+{-# INLINE all #-}
1310+all :: Monad m => (a -> Bool) -> Fold m a Bool
1311+all predicate = foldt' step initial id
1312+
1313+ where
1314+
1315+ initial = Partial True
1316+
1317+ step _ a =
1318+ if predicate a
1319+ then Partial True
1320+ else Done False
1321+
1322+-- | Returns 'True' if the given element is not present in the stream.
1323+--
1324+-- Definition:
1325+--
1326+-- >>> notElem a = Fold.all (/= a)
1327+--
1328+{-# INLINE notElem #-}
1329+notElem :: (Eq a, Monad m) => a -> Fold m a Bool
1330+notElem a = all (/= a)
1331+
1332+-- | Returns 'True' if all elements are 'True', 'False' otherwise
1333+--
1334+-- Definition:
1335+--
1336+-- >>> and = Fold.all (== True)
1337+--
1338+{-# INLINE and #-}
1339+and :: Monad m => Fold m Bool Bool
1340+and = all (== True)
1341+
1342+-- | Returns 'True' if any element is 'True', 'False' otherwise
1343+--
1344+-- Definition:
1345+--
1346+-- >>> or = Fold.any (== True)
1347+--
1348+{-# INLINE or #-}
1349+or :: Monad m => Fold m Bool Bool
1350+or = any (== True)
1351+
1352+------------------------------------------------------------------------------
1353+-- Grouping/Splitting
1354+------------------------------------------------------------------------------
1355+
1356+------------------------------------------------------------------------------
1357+-- Grouping without looking at elements
1358+------------------------------------------------------------------------------
1359+
1360+------------------------------------------------------------------------------
1361+-- Binary APIs
1362+------------------------------------------------------------------------------
1363+
1364+-- | @splitAt n f1 f2@ composes folds @f1@ and @f2@ such that first @n@
1365+-- elements of its input are consumed by fold @f1@ and the rest of the stream
1366+-- is consumed by fold @f2@.
1367+--
1368+-- >>> let splitAt_ n xs = Stream.fold (Fold.splitAt n Fold.toList Fold.toList) $ Stream.fromList xs
1369+--
1370+-- >>> splitAt_ 6 "Hello World!"
1371+-- ("Hello ","World!")
1372+--
1373+-- >>> splitAt_ (-1) [1,2,3]
1374+-- ([],[1,2,3])
1375+--
1376+-- >>> splitAt_ 0 [1,2,3]
1377+-- ([],[1,2,3])
1378+--
1379+-- >>> splitAt_ 1 [1,2,3]
1380+-- ([1],[2,3])
1381+--
1382+-- >>> splitAt_ 3 [1,2,3]
1383+-- ([1,2,3],[])
1384+--
1385+-- >>> splitAt_ 4 [1,2,3]
1386+-- ([1,2,3],[])
1387+--
1388+-- > splitAt n f1 f2 = Fold.splitWith (,) (Fold.take n f1) f2
1389+--
1390+-- /Internal/
1391+
1392+{-# INLINE splitAt #-}
1393+splitAt
1394+ :: Monad m
1395+ => Int
1396+ -> Fold m a b
1397+ -> Fold m a c
1398+ -> Fold m a (b, c)
1399+splitAt n fld = splitWith (,) (take n fld)
1400+
1401+------------------------------------------------------------------------------
1402+-- Element Aware APIs
1403+------------------------------------------------------------------------------
1404+--
1405+------------------------------------------------------------------------------
1406+-- Binary APIs
1407+------------------------------------------------------------------------------
1408+
1409+{-# INLINE takingEndByM #-}
1410+takingEndByM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1411+takingEndByM p = Fold step initial extract extract
1412+
1413+ where
1414+
1415+ initial = return $ Partial Nothing'
1416+
1417+ step _ a = do
1418+ r <- p a
1419+ return
1420+ $ if r
1421+ then Done $ Just a
1422+ else Partial $ Just' a
1423+
1424+ extract = return . toMaybe
1425+
1426+-- |
1427+--
1428+-- >>> takingEndBy p = Fold.takingEndByM (return . p)
1429+--
1430+{-# INLINE takingEndBy #-}
1431+takingEndBy :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
1432+takingEndBy p = takingEndByM (return . p)
1433+
1434+{-# INLINE takingEndByM_ #-}
1435+takingEndByM_ :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1436+takingEndByM_ p = Fold step initial extract extract
1437+
1438+ where
1439+
1440+ initial = return $ Partial Nothing'
1441+
1442+ step _ a = do
1443+ r <- p a
1444+ return
1445+ $ if r
1446+ then Done Nothing
1447+ else Partial $ Just' a
1448+
1449+ extract = return . toMaybe
1450+
1451+-- |
1452+--
1453+-- >>> takingEndBy_ p = Fold.takingEndByM_ (return . p)
1454+--
1455+{-# INLINE takingEndBy_ #-}
1456+takingEndBy_ :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
1457+takingEndBy_ p = takingEndByM_ (return . p)
1458+
1459+{-# INLINE droppingWhileM #-}
1460+droppingWhileM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1461+droppingWhileM p = Fold step initial extract extract
1462+
1463+ where
1464+
1465+ initial = return $ Partial Nothing'
1466+
1467+ step Nothing' a = do
1468+ r <- p a
1469+ return
1470+ $ Partial
1471+ $ if r
1472+ then Nothing'
1473+ else Just' a
1474+ step _ a = return $ Partial $ Just' a
1475+
1476+ extract = return . toMaybe
1477+
1478+-- |
1479+-- >>> droppingWhile p = Fold.droppingWhileM (return . p)
1480+--
1481+{-# INLINE droppingWhile #-}
1482+droppingWhile :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
1483+droppingWhile p = droppingWhileM (return . p)
1484+
1485+------------------------------------------------------------------------------
1486+-- Binary splitting on a separator
1487+------------------------------------------------------------------------------
1488+
1489+data SplitOnSeqState acc a rb rh w ck =
1490+ SplitOnSeqEmpty !acc
1491+ | SplitOnSeqSingle !acc !a
1492+ | SplitOnSeqWord !acc !Int !w
1493+ | SplitOnSeqWordLoop !acc !w
1494+ | SplitOnSeqKR !acc !Int !rb !rh
1495+ | SplitOnSeqKRLoop !acc !ck !rb !rh
1496+
1497+-- XXX Need to add tests for takeEndBySeq, we have tests for takeEndBySeq_ .
1498+
1499+-- | Continue taking the input until the input sequence matches the supplied
1500+-- sequence, taking the supplied sequence as well. If the pattern is empty this
1501+-- acts as an identity fold.
1502+--
1503+-- >>> s = Stream.fromList "hello there. How are you?"
1504+-- >>> f = Fold.takeEndBySeq (Array.fromList "re") Fold.toList
1505+-- >>> Stream.fold f s
1506+-- "hello there"
1507+--
1508+-- >>> Stream.fold Fold.toList $ Stream.foldMany f s
1509+-- ["hello there",". How are"," you?"]
1510+--
1511+-- /Pre-release/
1512+{-# INLINE takeEndBySeq #-}
1513+takeEndBySeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) =>
1514+ Array.Array a
1515+ -> Fold m a b
1516+ -> Fold m a b
1517+takeEndBySeq patArr (Fold fstep finitial fextract ffinal) =
1518+ Fold step initial extract final
1519+
1520+ where
1521+
1522+ patLen = Array.length patArr
1523+
1524+ initial = do
1525+ res <- finitial
1526+ case res of
1527+ Partial acc
1528+ | patLen == 0 ->
1529+ -- XXX Should we match nothing or everything on empty
1530+ -- pattern?
1531+ -- Done <$> ffinal acc
1532+ return $ Partial $ SplitOnSeqEmpty acc
1533+ | patLen == 1 -> do
1534+ pat <- liftIO $ Array.unsafeIndexIO 0 patArr
1535+ return $ Partial $ SplitOnSeqSingle acc pat
1536+ | SIZE_OF(a) * patLen <= sizeOf (Proxy :: Proxy Word) ->
1537+ return $ Partial $ SplitOnSeqWord acc 0 0
1538+ | otherwise -> do
1539+ (rb, rhead) <- liftIO $ Ring.new patLen
1540+ return $ Partial $ SplitOnSeqKR acc 0 rb rhead
1541+ Done b -> return $ Done b
1542+
1543+ -- Word pattern related
1544+ maxIndex = patLen - 1
1545+
1546+ elemBits = SIZE_OF(a) * 8
1547+
1548+ wordMask :: Word
1549+ wordMask = (1 `shiftL` (elemBits * patLen)) - 1
1550+
1551+ wordPat :: Word
1552+ wordPat = wordMask .&. Array.foldl' addToWord 0 patArr
1553+
1554+ addToWord wd a = (wd `shiftL` elemBits) .|. fromIntegral (fromEnum a)
1555+
1556+ -- For Rabin-Karp search
1557+ k = 2891336453 :: Word32
1558+ coeff = k ^ patLen
1559+
1560+ addCksum cksum a = cksum * k + fromIntegral (fromEnum a)
1561+
1562+ deltaCksum cksum old new =
1563+ addCksum cksum new - coeff * fromIntegral (fromEnum old)
1564+
1565+ -- XXX shall we use a random starting hash or 1 instead of 0?
1566+ -- XXX Need to keep this cached across fold calls in foldmany
1567+ -- XXX We may need refold to inject the cached state instead of
1568+ -- initializing the state every time.
1569+ -- XXX Allocation of ring buffer should also be done once
1570+ patHash = Array.foldl' addCksum 0 patArr
1571+
1572+ step (SplitOnSeqEmpty s) x = do
1573+ res <- fstep s x
1574+ case res of
1575+ Partial s1 -> return $ Partial $ SplitOnSeqEmpty s1
1576+ Done b -> return $ Done b
1577+ step (SplitOnSeqSingle s pat) x = do
1578+ res <- fstep s x
1579+ case res of
1580+ Partial s1
1581+ | pat /= x -> return $ Partial $ SplitOnSeqSingle s1 pat
1582+ | otherwise -> Done <$> ffinal s1
1583+ Done b -> return $ Done b
1584+ step (SplitOnSeqWord s idx wrd) x = do
1585+ res <- fstep s x
1586+ let wrd1 = addToWord wrd x
1587+ case res of
1588+ Partial s1
1589+ | idx == maxIndex -> do
1590+ if wrd1 .&. wordMask == wordPat
1591+ then Done <$> ffinal s1
1592+ else return $ Partial $ SplitOnSeqWordLoop s1 wrd1
1593+ | otherwise ->
1594+ return $ Partial $ SplitOnSeqWord s1 (idx + 1) wrd1
1595+ Done b -> return $ Done b
1596+ step (SplitOnSeqWordLoop s wrd) x = do
1597+ res <- fstep s x
1598+ let wrd1 = addToWord wrd x
1599+ case res of
1600+ Partial s1
1601+ | wrd1 .&. wordMask == wordPat ->
1602+ Done <$> ffinal s1
1603+ | otherwise ->
1604+ return $ Partial $ SplitOnSeqWordLoop s1 wrd1
1605+ Done b -> return $ Done b
1606+ step (SplitOnSeqKR s idx rb rh) x = do
1607+ res <- fstep s x
1608+ case res of
1609+ Partial s1 -> do
1610+ rh1 <- liftIO $ Ring.unsafeInsert rb rh x
1611+ if idx == maxIndex
1612+ then do
1613+ let fld = Ring.unsafeFoldRing (Ring.ringBound rb)
1614+ let !ringHash = fld addCksum 0 rb
1615+ if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1616+ then Done <$> ffinal s1
1617+ else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
1618+ else
1619+ return $ Partial $ SplitOnSeqKR s1 (idx + 1) rb rh1
1620+ Done b -> return $ Done b
1621+ step (SplitOnSeqKRLoop s cksum rb rh) x = do
1622+ res <- fstep s x
1623+ case res of
1624+ Partial s1 -> do
1625+ old <- liftIO $ peek rh
1626+ rh1 <- liftIO $ Ring.unsafeInsert rb rh x
1627+ let ringHash = deltaCksum cksum old x
1628+ if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1629+ then Done <$> ffinal s1
1630+ else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
1631+ Done b -> return $ Done b
1632+
1633+ extractFunc fex state =
1634+ let st =
1635+ case state of
1636+ SplitOnSeqEmpty s -> s
1637+ SplitOnSeqSingle s _ -> s
1638+ SplitOnSeqWord s _ _ -> s
1639+ SplitOnSeqWordLoop s _ -> s
1640+ SplitOnSeqKR s _ _ _ -> s
1641+ SplitOnSeqKRLoop s _ _ _ -> s
1642+ in fex st
1643+
1644+ extract state = extractFunc fextract state
1645+
1646+ final state = extractFunc ffinal state
1647+
1648+-- | Like 'takeEndBySeq' but discards the matched sequence.
1649+--
1650+-- /Pre-release/
1651+--
1652+{-# INLINE takeEndBySeq_ #-}
1653+takeEndBySeq_ :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) =>
1654+ Array.Array a
1655+ -> Fold m a b
1656+ -> Fold m a b
1657+takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
1658+ Fold step initial extract final
1659+
1660+ where
1661+
1662+ patLen = Array.length patArr
1663+
1664+ initial = do
1665+ res <- finitial
1666+ case res of
1667+ Partial acc
1668+ | patLen == 0 ->
1669+ -- XXX Should we match nothing or everything on empty
1670+ -- pattern?
1671+ -- Done <$> ffinal acc
1672+ return $ Partial $ SplitOnSeqEmpty acc
1673+ | patLen == 1 -> do
1674+ pat <- liftIO $ Array.unsafeIndexIO 0 patArr
1675+ return $ Partial $ SplitOnSeqSingle acc pat
1676+ -- XXX Need to add tests for this case
1677+ | SIZE_OF(a) * patLen <= sizeOf (Proxy :: Proxy Word) ->
1678+ return $ Partial $ SplitOnSeqWord acc 0 0
1679+ | otherwise -> do
1680+ (rb, rhead) <- liftIO $ Ring.new patLen
1681+ return $ Partial $ SplitOnSeqKR acc 0 rb rhead
1682+ Done b -> return $ Done b
1683+
1684+ -- Word pattern related
1685+ maxIndex = patLen - 1
1686+
1687+ elemBits = SIZE_OF(a) * 8
1688+
1689+ wordMask :: Word
1690+ wordMask = (1 `shiftL` (elemBits * patLen)) - 1
1691+
1692+ elemMask :: Word
1693+ elemMask = (1 `shiftL` elemBits) - 1
1694+
1695+ wordPat :: Word
1696+ wordPat = wordMask .&. Array.foldl' addToWord 0 patArr
1697+
1698+ addToWord wd a = (wd `shiftL` elemBits) .|. fromIntegral (fromEnum a)
1699+
1700+ -- For Rabin-Karp search
1701+ k = 2891336453 :: Word32
1702+ coeff = k ^ patLen
1703+
1704+ addCksum cksum a = cksum * k + fromIntegral (fromEnum a)
1705+
1706+ deltaCksum cksum old new =
1707+ addCksum cksum new - coeff * fromIntegral (fromEnum old)
1708+
1709+ -- XXX shall we use a random starting hash or 1 instead of 0?
1710+ -- XXX Need to keep this cached across fold calls in foldMany
1711+ -- XXX We may need refold to inject the cached state instead of
1712+ -- initializing the state every time.
1713+ -- XXX Allocation of ring buffer should also be done once
1714+ patHash = Array.foldl' addCksum 0 patArr
1715+
1716+ step (SplitOnSeqEmpty s) x = do
1717+ res <- fstep s x
1718+ case res of
1719+ Partial s1 -> return $ Partial $ SplitOnSeqEmpty s1
1720+ Done b -> return $ Done b
1721+ step (SplitOnSeqSingle s pat) x = do
1722+ if pat /= x
1723+ then do
1724+ res <- fstep s x
1725+ case res of
1726+ Partial s1 -> return $ Partial $ SplitOnSeqSingle s1 pat
1727+ Done b -> return $ Done b
1728+ else Done <$> ffinal s
1729+ step (SplitOnSeqWord s idx wrd) x = do
1730+ let wrd1 = addToWord wrd x
1731+ if idx == maxIndex
1732+ then do
1733+ if wrd1 .&. wordMask == wordPat
1734+ then Done <$> ffinal s
1735+ else return $ Partial $ SplitOnSeqWordLoop s wrd1
1736+ else return $ Partial $ SplitOnSeqWord s (idx + 1) wrd1
1737+ step (SplitOnSeqWordLoop s wrd) x = do
1738+ let wrd1 = addToWord wrd x
1739+ old = (wordMask .&. wrd)
1740+ `shiftR` (elemBits * (patLen - 1))
1741+ res <- fstep s (toEnum $ fromIntegral old)
1742+ case res of
1743+ Partial s1
1744+ | wrd1 .&. wordMask == wordPat ->
1745+ Done <$> ffinal s1
1746+ | otherwise ->
1747+ return $ Partial $ SplitOnSeqWordLoop s1 wrd1
1748+ Done b -> return $ Done b
1749+ step (SplitOnSeqKR s idx rb rh) x = do
1750+ rh1 <- liftIO $ Ring.unsafeInsert rb rh x
1751+ if idx == maxIndex
1752+ then do
1753+ let fld = Ring.unsafeFoldRing (Ring.ringBound rb)
1754+ let !ringHash = fld addCksum 0 rb
1755+ if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1756+ then Done <$> ffinal s
1757+ else return $ Partial $ SplitOnSeqKRLoop s ringHash rb rh1
1758+ else return $ Partial $ SplitOnSeqKR s (idx + 1) rb rh1
1759+ step (SplitOnSeqKRLoop s cksum rb rh) x = do
1760+ old <- liftIO $ peek rh
1761+ res <- fstep s old
1762+ case res of
1763+ Partial s1 -> do
1764+ rh1 <- liftIO $ Ring.unsafeInsert rb rh x
1765+ let ringHash = deltaCksum cksum old x
1766+ if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1767+ then Done <$> ffinal s1
1768+ else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
1769+ Done b -> return $ Done b
1770+
1771+ -- XXX extract should return backtrack count as well. If the fold
1772+ -- terminates early inside extract, we may still have buffered data
1773+ -- remaining which will be lost if we do not communicate that to the
1774+ -- driver.
1775+ extractFunc fex state = do
1776+ let consumeWord s n wrd = do
1777+ if n == 0
1778+ then fex s
1779+ else do
1780+ let old = elemMask .&. (wrd `shiftR` (elemBits * (n - 1)))
1781+ r <- fstep s (toEnum $ fromIntegral old)
1782+ case r of
1783+ Partial s1 -> consumeWord s1 (n - 1) wrd
1784+ Done b -> return b
1785+
1786+ let consumeRing s n rb rh =
1787+ if n == 0
1788+ then fex s
1789+ else do
1790+ old <- liftIO $ peek rh
1791+ let rh1 = Ring.advance rb rh
1792+ r <- fstep s old
1793+ case r of
1794+ Partial s1 -> consumeRing s1 (n - 1) rb rh1
1795+ Done b -> return b
1796+
1797+ case state of
1798+ SplitOnSeqEmpty s -> fex s
1799+ SplitOnSeqSingle s _ -> fex s
1800+ SplitOnSeqWord s idx wrd -> consumeWord s idx wrd
1801+ SplitOnSeqWordLoop s wrd -> consumeWord s patLen wrd
1802+ SplitOnSeqKR s idx rb _ -> consumeRing s idx rb (Ring.startOf rb)
1803+ SplitOnSeqKRLoop s _ rb rh -> consumeRing s patLen rb rh
1804+
1805+ extract state = extractFunc fextract state
1806+
1807+ final state = extractFunc ffinal state
1808+
1809+------------------------------------------------------------------------------
1810+-- Distributing
1811+------------------------------------------------------------------------------
1812+--
1813+-- | Distribute one copy of the stream to each fold and zip the results.
1814+--
1815+-- @
1816+-- |-------Fold m a b--------|
1817+-- ---stream m a---| |---m (b,c)
1818+-- |-------Fold m a c--------|
1819+-- @
1820+--
1821+-- Definition:
1822+--
1823+-- >>> tee = Fold.teeWith (,)
1824+--
1825+-- Example:
1826+--
1827+-- >>> t = Fold.tee Fold.sum Fold.length
1828+-- >>> Stream.fold t (Stream.enumerateFromTo 1.0 100.0)
1829+-- (5050.0,100)
1830+--
1831+{-# INLINE tee #-}
1832+tee :: Monad m => Fold m a b -> Fold m a c -> Fold m a (b,c)
1833+tee = teeWith (,)
1834+
1835+-- XXX use "List" instead of "[]"?, use Array for output to scale it to a large
1836+-- number of consumers? For polymorphic case a vector could be helpful. For
1837+-- Storables we can use arrays. Will need separate APIs for those.
1838+--
1839+-- | Distribute one copy of the stream to each fold and collect the results in
1840+-- a container.
1841+--
1842+-- @
1843+--
1844+-- |-------Fold m a b--------|
1845+-- ---stream m a---| |---m [b]
1846+-- |-------Fold m a b--------|
1847+-- | |
1848+-- ...
1849+-- @
1850+--
1851+-- >>> Stream.fold (Fold.distribute [Fold.sum, Fold.length]) (Stream.enumerateFromTo 1 5)
1852+-- [15,5]
1853+--
1854+-- >>> distribute = Prelude.foldr (Fold.teeWith (:)) (Fold.fromPure [])
1855+--
1856+-- This is the consumer side dual of the producer side 'sequence' operation.
1857+--
1858+-- Stops when all the folds stop.
1859+--
1860+{-# INLINE distribute #-}
1861+distribute :: Monad m => [Fold m a b] -> Fold m a [b]
1862+distribute = Prelude.foldr (teeWith (:)) (fromPure [])
1863+
1864+------------------------------------------------------------------------------
1865+-- Partitioning
1866+------------------------------------------------------------------------------
1867+
1868+{-# INLINE partitionByMUsing #-}
1869+partitionByMUsing :: Monad m =>
1870+ ( (x -> y -> (x, y))
1871+ -> Fold m (Either b c) x
1872+ -> Fold m (Either b c) y
1873+ -> Fold m (Either b c) (x, y)
1874+ )
1875+ -> (a -> m (Either b c))
1876+ -> Fold m b x
1877+ -> Fold m c y
1878+ -> Fold m a (x, y)
1879+partitionByMUsing t f fld1 fld2 =
1880+ let l = lmap (fromLeft undefined) fld1 -- :: Fold m (Either b c) x
1881+ r = lmap (fromRight undefined) fld2 -- :: Fold m (Either b c) y
1882+ in lmapM f (t (,) (filter isLeft l) (filter isRight r))
1883+
1884+-- | Partition the input over two folds using an 'Either' partitioning
1885+-- predicate.
1886+--
1887+-- @
1888+--
1889+-- |-------Fold b x--------|
1890+-- -----stream m a --> (Either b c)----| |----(x,y)
1891+-- |-------Fold c y--------|
1892+-- @
1893+--
1894+-- Example, send input to either fold randomly:
1895+--
1896+-- >>> :set -package random
1897+-- >>> import System.Random (randomIO)
1898+-- >>> randomly a = randomIO >>= \x -> return $ if x then Left a else Right a
1899+-- >>> f = Fold.partitionByM randomly Fold.length Fold.length
1900+-- >>> Stream.fold f (Stream.enumerateFromTo 1 100)
1901+-- ...
1902+--
1903+-- Example, send input to the two folds in a proportion of 2:1:
1904+--
1905+-- >>> :{
1906+-- proportionately m n = do
1907+-- ref <- newIORef $ cycle $ concat [replicate m Left, replicate n Right]
1908+-- return $ \a -> do
1909+-- r <- readIORef ref
1910+-- writeIORef ref $ tail r
1911+-- return $ Prelude.head r a
1912+-- :}
1913+--
1914+-- >>> :{
1915+-- main = do
1916+-- g <- proportionately 2 1
1917+-- let f = Fold.partitionByM g Fold.length Fold.length
1918+-- r <- Stream.fold f (Stream.enumerateFromTo (1 :: Int) 100)
1919+-- print r
1920+-- :}
1921+--
1922+-- >>> main
1923+-- (67,33)
1924+--
1925+--
1926+-- This is the consumer side dual of the producer side 'mergeBy' operation.
1927+--
1928+-- When one fold is done, any input meant for it is ignored until the other
1929+-- fold is also done.
1930+--
1931+-- Stops when both the folds stop.
1932+--
1933+-- /See also: 'partitionByFstM' and 'partitionByMinM'./
1934+--
1935+-- /Pre-release/
1936+{-# INLINE partitionByM #-}
1937+partitionByM :: Monad m
1938+ => (a -> m (Either b c)) -> Fold m b x -> Fold m c y -> Fold m a (x, y)
1939+partitionByM = partitionByMUsing teeWith
1940+
1941+-- | Similar to 'partitionByM' but terminates when the first fold terminates.
1942+--
1943+{-# INLINE partitionByFstM #-}
1944+partitionByFstM :: Monad m
1945+ => (a -> m (Either b c)) -> Fold m b x -> Fold m c y -> Fold m a (x, y)
1946+partitionByFstM = partitionByMUsing teeWithFst
1947+
1948+-- | Similar to 'partitionByM' but terminates when any fold terminates.
1949+--
1950+{-# INLINE partitionByMinM #-}
1951+partitionByMinM :: Monad m =>
1952+ (a -> m (Either b c)) -> Fold m b x -> Fold m c y -> Fold m a (x, y)
1953+partitionByMinM = partitionByMUsing teeWithMin
1954+
1955+-- Note: we could use (a -> Bool) instead of (a -> Either b c), but the latter
1956+-- makes the signature clearer as to which case belongs to which fold.
1957+-- XXX need to check the performance in both cases.
1958+
1959+-- | Same as 'partitionByM' but with a pure partition function.
1960+--
1961+-- Example, count even and odd numbers in a stream:
1962+--
1963+-- >>> :{
1964+-- let f = Fold.partitionBy (\n -> if even n then Left n else Right n)
1965+-- (fmap (("Even " ++) . show) Fold.length)
1966+-- (fmap (("Odd " ++) . show) Fold.length)
1967+-- in Stream.fold f (Stream.enumerateFromTo 1 100)
1968+-- :}
1969+-- ("Even 50","Odd 50")
1970+--
1971+-- /Pre-release/
1972+{-# INLINE partitionBy #-}
1973+partitionBy :: Monad m
1974+ => (a -> Either b c) -> Fold m b x -> Fold m c y -> Fold m a (x, y)
1975+partitionBy f = partitionByM (return . f)
1976+
1977+-- | Compose two folds such that the combined fold accepts a stream of 'Either'
1978+-- and routes the 'Left' values to the first fold and 'Right' values to the
1979+-- second fold.
1980+--
1981+-- Definition:
1982+--
1983+-- >>> partition = Fold.partitionBy id
1984+--
1985+{-# INLINE partition #-}
1986+partition :: Monad m
1987+ => Fold m b x -> Fold m c y -> Fold m (Either b c) (x, y)
1988+partition = partitionBy id
1989+
1990+{-
1991+-- | Send one item to each fold in a round-robin fashion. This is the consumer
1992+-- side dual of producer side 'mergeN' operation.
1993+--
1994+-- partitionN :: Monad m => [Fold m a b] -> Fold m a [b]
1995+-- partitionN fs = Fold step begin done
1996+-}
1997+
1998+------------------------------------------------------------------------------
1999+-- Unzipping
2000+------------------------------------------------------------------------------
2001+
2002+{-# INLINE unzipWithMUsing #-}
2003+unzipWithMUsing :: Monad m =>
2004+ ( (x -> y -> (x, y))
2005+ -> Fold m (b, c) x
2006+ -> Fold m (b, c) y
2007+ -> Fold m (b, c) (x, y)
2008+ )
2009+ -> (a -> m (b, c))
2010+ -> Fold m b x
2011+ -> Fold m c y
2012+ -> Fold m a (x, y)
2013+unzipWithMUsing t f fld1 fld2 =
2014+ let f1 = lmap fst fld1 -- :: Fold m (b, c) b
2015+ f2 = lmap snd fld2 -- :: Fold m (b, c) c
2016+ in lmapM f (t (,) f1 f2)
2017+
2018+-- | Like 'unzipWith' but with a monadic splitter function.
2019+--
2020+-- Definition:
2021+--
2022+-- >>> unzipWithM k f1 f2 = Fold.lmapM k (Fold.unzip f1 f2)
2023+--
2024+-- /Pre-release/
2025+{-# INLINE unzipWithM #-}
2026+unzipWithM :: Monad m
2027+ => (a -> m (b,c)) -> Fold m b x -> Fold m c y -> Fold m a (x,y)
2028+unzipWithM = unzipWithMUsing teeWith
2029+
2030+-- | Similar to 'unzipWithM' but terminates when the first fold terminates.
2031+--
2032+{-# INLINE unzipWithFstM #-}
2033+unzipWithFstM :: Monad m =>
2034+ (a -> m (b, c)) -> Fold m b x -> Fold m c y -> Fold m a (x, y)
2035+unzipWithFstM = unzipWithMUsing teeWithFst
2036+
2037+-- | Similar to 'unzipWithM' but terminates when any fold terminates.
2038+--
2039+{-# INLINE unzipWithMinM #-}
2040+unzipWithMinM :: Monad m =>
2041+ (a -> m (b,c)) -> Fold m b x -> Fold m c y -> Fold m a (x,y)
2042+unzipWithMinM = unzipWithMUsing teeWithMin
2043+
2044+-- | Split elements in the input stream into two parts using a pure splitter
2045+-- function, direct each part to a different fold and zip the results.
2046+--
2047+-- Definitions:
2048+--
2049+-- >>> unzipWith f = Fold.unzipWithM (return . f)
2050+-- >>> unzipWith f fld1 fld2 = Fold.lmap f (Fold.unzip fld1 fld2)
2051+--
2052+-- This fold terminates when both the input folds terminate.
2053+--
2054+-- /Pre-release/
2055+{-# INLINE unzipWith #-}
2056+unzipWith :: Monad m
2057+ => (a -> (b,c)) -> Fold m b x -> Fold m c y -> Fold m a (x,y)
2058+unzipWith f = unzipWithM (return . f)
2059+
2060+-- | Send the elements of tuples in a stream of tuples through two different
2061+-- folds.
2062+--
2063+-- @
2064+--
2065+-- |-------Fold m a x--------|
2066+-- ---------stream of (a,b)--| |----m (x,y)
2067+-- |-------Fold m b y--------|
2068+--
2069+-- @
2070+--
2071+-- Definition:
2072+--
2073+-- >>> unzip = Fold.unzipWith id
2074+--
2075+-- This is the consumer side dual of the producer side 'zip' operation.
2076+--
2077+{-# INLINE unzip #-}
2078+unzip :: Monad m => Fold m a x -> Fold m b y -> Fold m (a,b) (x,y)
2079+unzip = unzipWith id
2080+
2081+------------------------------------------------------------------------------
2082+-- Combining streams and folds - Zipping
2083+------------------------------------------------------------------------------
2084+
2085+-- XXX These can be implemented using the fold scan, using the stream as a
2086+-- state.
2087+-- XXX Stream Skip state cannot be efficiently handled in folds but can be
2088+-- handled in parsers using the Continue facility. See zipWithM in the Parser
2089+-- module.
2090+--
2091+-- cmpBy, eqBy, isPrefixOf, isSubsequenceOf etc can be implemented using
2092+-- zipStream.
2093+
2094+-- | Zip a stream with the input of a fold using the supplied function.
2095+--
2096+-- /Unimplemented/
2097+--
2098+{-# INLINE zipStreamWithM #-}
2099+zipStreamWithM :: -- Monad m =>
2100+ (a -> b -> m c) -> Stream m a -> Fold m c x -> Fold m b x
2101+zipStreamWithM = undefined
2102+
2103+-- | Zip a stream with the input of a fold.
2104+--
2105+-- >>> zip = Fold.zipStreamWithM (curry return)
2106+--
2107+-- /Unimplemented/
2108+--
2109+{-# INLINE zipStream #-}
2110+zipStream :: Monad m => Stream m a -> Fold m (a, b) x -> Fold m b x
2111+zipStream = zipStreamWithM (curry return)
2112+
2113+-- | Pair each element of a fold input with its index, starting from index 0.
2114+--
2115+{-# INLINE indexingWith #-}
2116+indexingWith :: Monad m => Int -> (Int -> Int) -> Fold m a (Maybe (Int, a))
2117+indexingWith i f = fmap toMaybe $ foldl' step initial
2118+
2119+ where
2120+
2121+ initial = Nothing'
2122+
2123+ step Nothing' a = Just' (i, a)
2124+ step (Just' (n, _)) a = Just' (f n, a)
2125+
2126+-- |
2127+-- >>> indexing = Fold.indexingWith 0 (+ 1)
2128+--
2129+{-# INLINE indexing #-}
2130+indexing :: Monad m => Fold m a (Maybe (Int, a))
2131+indexing = indexingWith 0 (+ 1)
2132+
2133+-- |
2134+-- >>> indexingRev n = Fold.indexingWith n (subtract 1)
2135+--
2136+{-# INLINE indexingRev #-}
2137+indexingRev :: Monad m => Int -> Fold m a (Maybe (Int, a))
2138+indexingRev n = indexingWith n (subtract 1)
2139+
2140+-- | Pair each element of a fold input with its index, starting from index 0.
2141+--
2142+-- >>> indexed = Fold.scanMaybe Fold.indexing
2143+--
2144+{-# INLINE indexed #-}
2145+indexed :: Monad m => Fold m (Int, a) b -> Fold m a b
2146+indexed = scanMaybe indexing
2147+
2148+-- | Change the predicate function of a Fold from @a -> b@ to accept an
2149+-- additional state input @(s, a) -> b@. Convenient to filter with an
2150+-- addiitonal index or time input.
2151+--
2152+-- >>> filterWithIndex = Fold.with Fold.indexed Fold.filter
2153+--
2154+-- @
2155+-- filterWithAbsTime = with timestamped filter
2156+-- filterWithRelTime = with timeIndexed filter
2157+-- @
2158+--
2159+-- /Pre-release/
2160+{-# INLINE with #-}
2161+with ::
2162+ (Fold m (s, a) b -> Fold m a b)
2163+ -> (((s, a) -> c) -> Fold m (s, a) b -> Fold m (s, a) b)
2164+ -> (((s, a) -> c) -> Fold m a b -> Fold m a b)
2165+with f comb g = f . comb g . lmap snd
2166+
2167+-- XXX Implement as a filter
2168+-- sampleFromthen :: Monad m => Int -> Int -> Fold m a (Maybe a)
2169+
2170+-- | @sampleFromthen offset stride@ samples the element at @offset@ index and
2171+-- then every element at strides of @stride@.
2172+--
2173+{-# INLINE sampleFromthen #-}
2174+sampleFromthen :: Monad m => Int -> Int -> Fold m a b -> Fold m a b
2175+sampleFromthen offset size =
2176+ with indexed filter (\(i, _) -> (i + offset) `mod` size == 0)
2177+
2178+------------------------------------------------------------------------------
2179+-- Nesting
2180+------------------------------------------------------------------------------
2181+
2182+-- | @concatSequence f t@ applies folds from stream @t@ sequentially and
2183+-- collects the results using the fold @f@.
2184+--
2185+-- /Unimplemented/
2186+--
2187+{-# INLINE concatSequence #-}
2188+concatSequence ::
2189+ -- IsStream t =>
2190+ Fold m b c -> t (Fold m a b) -> Fold m a c
2191+concatSequence _f _p = undefined
2192+
2193+-- | Group the input stream into groups of elements between @low@ and @high@.
2194+-- Collection starts in chunks of @low@ and then keeps doubling until we reach
2195+-- @high@. Each chunk is folded using the provided fold function.
2196+--
2197+-- This could be useful, for example, when we are folding a stream of unknown
2198+-- size to a stream of arrays and we want to minimize the number of
2199+-- allocations.
2200+--
2201+-- NOTE: this would be an application of "many" using a terminating fold.
2202+--
2203+-- /Unimplemented/
2204+--
2205+{-# INLINE chunksBetween #-}
2206+chunksBetween :: -- Monad m =>
2207+ Int -> Int -> Fold m a b -> Fold m b c -> Fold m a c
2208+chunksBetween _low _high _f1 _f2 = undefined
2209+
2210+-- | A fold that buffers its input to a pure stream.
2211+--
2212+-- /Warning!/ working on large streams accumulated as buffers in memory could
2213+-- be very inefficient, consider using "Streamly.Data.Array" instead.
2214+--
2215+-- >>> toStream = fmap Stream.fromList Fold.toList
2216+--
2217+-- /Pre-release/
2218+{-# INLINE toStream #-}
2219+toStream :: (Monad m, Monad n) => Fold m a (Stream n a)
2220+toStream = fmap StreamD.fromList toList
2221+
2222+-- This is more efficient than 'toStream'. toStream is exactly the same as
2223+-- reversing the stream after toStreamRev.
2224+--
2225+-- | Buffers the input stream to a pure stream in the reverse order of the
2226+-- input.
2227+--
2228+-- >>> toStreamRev = fmap Stream.fromList Fold.toListRev
2229+--
2230+-- /Warning!/ working on large streams accumulated as buffers in memory could
2231+-- be very inefficient, consider using "Streamly.Data.Array" instead.
2232+--
2233+-- /Pre-release/
2234+
2235+-- xn : ... : x2 : x1 : []
2236+{-# INLINE toStreamRev #-}
2237+toStreamRev :: (Monad m, Monad n) => Fold m a (Stream n a)
2238+toStreamRev = fmap StreamD.fromList toListRev
2239+
2240+-- XXX This does not fuse. It contains a recursive step function. We will need
2241+-- a Skip input constructor in the fold type to make it fuse.
2242+--
2243+-- | Unfold and flatten the input stream of a fold.
2244+--
2245+-- @
2246+-- Stream.fold (unfoldMany u f) = Stream.fold f . Stream.unfoldMany u
2247+-- @
2248+--
2249+-- /Pre-release/
2250+{-# INLINE unfoldMany #-}
2251+unfoldMany :: Monad m => Unfold m a b -> Fold m b c -> Fold m a c
2252+unfoldMany (Unfold ustep inject) (Fold fstep initial extract final) =
2253+ Fold consume initial extract final
2254+
2255+ where
2256+
2257+ {-# INLINE produce #-}
2258+ produce fs us = do
2259+ ures <- ustep us
2260+ case ures of
2261+ StreamD.Yield b us1 -> do
2262+ fres <- fstep fs b
2263+ case fres of
2264+ Partial fs1 -> produce fs1 us1
2265+ -- XXX What to do with the remaining stream?
2266+ Done c -> return $ Done c
2267+ StreamD.Skip us1 -> produce fs us1
2268+ StreamD.Stop -> return $ Partial fs
2269+
2270+ {-# INLINE_LATE consume #-}
2271+ consume s a = inject a >>= produce s
2272+
2273+-- | Get the bottom most @n@ elements using the supplied comparison function.
2274+--
2275+{-# INLINE bottomBy #-}
2276+bottomBy :: (MonadIO m, Unbox a) =>
2277+ (a -> a -> Ordering)
2278+ -> Int
2279+ -> Fold m a (MutArray a)
2280+bottomBy cmp n = Fold step initial extract extract
2281+
2282+ where
2283+
2284+ initial = do
2285+ arr <- MA.pinnedNew n
2286+ if n <= 0
2287+ then return $ Done arr
2288+ else return $ Partial (arr, 0)
2289+
2290+ step (arr, i) x =
2291+ if i < n
2292+ then do
2293+ arr' <- MA.snoc arr x
2294+ MA.bubble cmp arr'
2295+ return $ Partial (arr', i + 1)
2296+ else do
2297+ x1 <- MA.getIndexUnsafe (i - 1) arr
2298+ case x `cmp` x1 of
2299+ LT -> do
2300+ MA.putIndexUnsafe (i - 1) arr x
2301+ MA.bubble cmp arr
2302+ return $ Partial (arr, i)
2303+ _ -> return $ Partial (arr, i)
2304+
2305+ extract = return . fst
2306+
2307+-- | Get the top @n@ elements using the supplied comparison function.
2308+--
2309+-- To get bottom n elements instead:
2310+--
2311+-- >>> bottomBy cmp = Fold.topBy (flip cmp)
2312+--
2313+-- Example:
2314+--
2315+-- >>> stream = Stream.fromList [2::Int,7,9,3,1,5,6,11,17]
2316+-- >>> Stream.fold (Fold.topBy compare 3) stream >>= MutArray.toList
2317+-- [17,11,9]
2318+--
2319+-- /Pre-release/
2320+--
2321+{-# INLINE topBy #-}
2322+topBy :: (MonadIO m, Unbox a) =>
2323+ (a -> a -> Ordering)
2324+ -> Int
2325+ -> Fold m a (MutArray a)
2326+topBy cmp = bottomBy (flip cmp)
2327+
2328+-- | Fold the input stream to top n elements.
2329+--
2330+-- Definition:
2331+--
2332+-- >>> top = Fold.topBy compare
2333+--
2334+-- >>> stream = Stream.fromList [2::Int,7,9,3,1,5,6,11,17]
2335+-- >>> Stream.fold (Fold.top 3) stream >>= MutArray.toList
2336+-- [17,11,9]
2337+--
2338+-- /Pre-release/
2339+{-# INLINE top #-}
2340+top :: (MonadIO m, Unbox a, Ord a) => Int -> Fold m a (MutArray a)
2341+top = bottomBy $ flip compare
2342+
2343+-- | Fold the input stream to bottom n elements.
2344+--
2345+-- Definition:
2346+--
2347+-- >>> bottom = Fold.bottomBy compare
2348+--
2349+-- >>> stream = Stream.fromList [2::Int,7,9,3,1,5,6,11,17]
2350+-- >>> Stream.fold (Fold.bottom 3) stream >>= MutArray.toList
2351+-- [1,2,3]
2352+--
2353+-- /Pre-release/
2354+{-# INLINE bottom #-}
2355+bottom :: (MonadIO m, Unbox a, Ord a) => Int -> Fold m a (MutArray a)
2356+bottom = bottomBy compare
2357+
2358+------------------------------------------------------------------------------
2359+-- Interspersed parsing
2360+------------------------------------------------------------------------------
2361+
2362+data IntersperseQState fs ps =
2363+ IntersperseQUnquoted !fs !ps
2364+ | IntersperseQQuoted !fs !ps
2365+ | IntersperseQQuotedEsc !fs !ps
2366+
2367+-- Useful for parsing CSV with quoting and escaping
2368+{-# INLINE intersperseWithQuotes #-}
2369+intersperseWithQuotes :: (Monad m, Eq a) =>
2370+ a -> a -> a -> Fold m a b -> Fold m b c -> Fold m a c
2371+intersperseWithQuotes
2372+ quote
2373+ esc
2374+ separator
2375+ (Fold stepL initialL _ finalL)
2376+ (Fold stepR initialR extractR finalR) = Fold step initial extract final
2377+
2378+ where
2379+
2380+ errMsg p status =
2381+ error $ "intersperseWithQuotes: " ++ p ++ " parsing fold cannot "
2382+ ++ status ++ " without input"
2383+
2384+ {-# INLINE initL #-}
2385+ initL mkState = do
2386+ resL <- initialL
2387+ case resL of
2388+ Partial sL ->
2389+ return $ Partial $ mkState sL
2390+ Done _ ->
2391+ errMsg "content" "succeed"
2392+
2393+ initial = do
2394+ res <- initialR
2395+ case res of
2396+ Partial sR -> initL (IntersperseQUnquoted sR)
2397+ Done b -> return $ Done b
2398+
2399+ {-# INLINE collect #-}
2400+ collect nextS sR b = do
2401+ res <- stepR sR b
2402+ case res of
2403+ Partial s ->
2404+ initL (nextS s)
2405+ Done c -> return (Done c)
2406+
2407+ {-# INLINE process #-}
2408+ process a sL sR nextState = do
2409+ r <- stepL sL a
2410+ case r of
2411+ Partial s -> return $ Partial (nextState sR s)
2412+ Done b -> collect nextState sR b
2413+
2414+ {-# INLINE processQuoted #-}
2415+ processQuoted a sL sR nextState = do
2416+ r <- stepL sL a
2417+ case r of
2418+ Partial s -> return $ Partial (nextState sR s)
2419+ Done _ -> do
2420+ _ <- finalR sR
2421+ error "Collecting fold finished inside quote"
2422+
2423+ step (IntersperseQUnquoted sR sL) a
2424+ | a == separator = do
2425+ b <- finalL sL
2426+ collect IntersperseQUnquoted sR b
2427+ | a == quote = processQuoted a sL sR IntersperseQQuoted
2428+ | otherwise = process a sL sR IntersperseQUnquoted
2429+
2430+ step (IntersperseQQuoted sR sL) a
2431+ | a == esc = processQuoted a sL sR IntersperseQQuotedEsc
2432+ | a == quote = process a sL sR IntersperseQUnquoted
2433+ | otherwise = processQuoted a sL sR IntersperseQQuoted
2434+
2435+ step (IntersperseQQuotedEsc sR sL) a =
2436+ processQuoted a sL sR IntersperseQQuoted
2437+
2438+ extract (IntersperseQUnquoted sR _) = extractR sR
2439+ extract (IntersperseQQuoted _ _) =
2440+ error "intersperseWithQuotes: finished inside quote"
2441+ extract (IntersperseQQuotedEsc _ _) =
2442+ error "intersperseWithQuotes: finished inside quote, at escape char"
2443+
2444+ final (IntersperseQUnquoted sR sL) = finalL sL *> finalR sR
2445+ final (IntersperseQQuoted sR sL) = do
2446+ _ <- finalR sR
2447+ _ <- finalL sL
2448+ error "intersperseWithQuotes: finished inside quote"
2449+ final (IntersperseQQuotedEsc sR sL) = do
2450+ _ <- finalR sR
2451+ _ <- finalL sL
2452+ error "intersperseWithQuotes: finished inside quote, at escape char"
core/src/Streamly/Internal/Data/Fold/Container.hs+75−32
@@ -9,9 +9,6 @@
99
1010module Streamly.Internal.Data.Fold.Container
1111 (
12− -- * Imports
13− -- $setup
14−
1512 -- * Set operations
1613 toSet
1714 , toIntSet
@@ -103,7 +100,8 @@ import qualified Data.Set as Set
103100import qualified Streamly.Internal.Data.IsMap as IsMap
104101
105102import Prelude hiding (Foldable(..))
106−import Streamly.Internal.Data.Fold
103+import Streamly.Internal.Data.Fold.Type
104+import Streamly.Internal.Data.Fold.Combinators
107105
108106-- $setup
109107-- >>> :m
@@ -272,14 +270,15 @@ demuxGeneric :: (Monad m, IsMap f, Traversable f) =>
272270 (a -> Key f)
273271 -> (a -> m (Fold m a b))
274272 -> Fold m a (m (f b), Maybe (Key f, b))
275−demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
273+demuxGeneric getKey getFold =
274+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
276275
277276 where
278277
279278 initial = return $ Tuple' IsMap.mapEmpty Nothing
280279
281280 {-# INLINE runFold #-}
282− runFold kv (Fold step1 initial1 extract1) (k, a) = do
281+ runFold kv (Fold step1 initial1 extract1 final1) (k, a) = do
283282 res <- initial1
284283 case res of
285284 Partial s -> do
@@ -287,7 +286,7 @@ demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
287286 return
288287 $ case res1 of
289288 Partial _ ->
290− let fld = Fold step1 (return res1) extract1
289+ let fld = Fold step1 (return res1) extract1 final1
291290 in Tuple' (IsMap.mapInsert k fld kv) Nothing
292291 Done b -> Tuple' (IsMap.mapDelete k kv) (Just (k, b))
293292 Done b -> return $ Tuple' kv (Just (k, b))
@@ -300,15 +299,25 @@ demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
300299 runFold kv fld (k, a)
301300 Just f -> runFold kv f (k, a)
302301
303− extract (Tuple' kv x) = (Prelude.mapM f kv, x)
302+ extract (Tuple' kv x) = return (Prelude.mapM f kv, x)
304303
305304 where
306305
307− f (Fold _ i e) = do
306+ f (Fold _ i e _) = do
308307 r <- i
309308 case r of
310309 Partial s -> e s
311− Done b -> return b
310+ _ -> error "demuxGeneric: unreachable code"
311+
312+ final (Tuple' kv x) = return (Prelude.mapM f kv, x)
313+
314+ where
315+
316+ f (Fold _ i _ fin) = do
317+ r <- i
318+ case r of
319+ Partial s -> fin s
320+ _ -> error "demuxGeneric: unreachable code"
312321
313322-- | @demux getKey getFold@: In a key value stream, fold values corresponding
314323-- to each key using a key specific fold. @getFold@ is invoked to generate a
@@ -350,35 +359,36 @@ demuxGenericIO :: (MonadIO m, IsMap f, Traversable f) =>
350359 (a -> Key f)
351360 -> (a -> m (Fold m a b))
352361 -> Fold m a (m (f b), Maybe (Key f, b))
353−demuxGenericIO getKey getFold = fmap extract $ foldlM' step initial
362+demuxGenericIO getKey getFold =
363+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
354364
355365 where
356366
357367 initial = return $ Tuple' IsMap.mapEmpty Nothing
358368
359369 {-# INLINE initFold #-}
360− initFold kv (Fold step1 initial1 extract1) (k, a) = do
370+ initFold kv (Fold step1 initial1 extract1 final1) (k, a) = do
361371 res <- initial1
362372 case res of
363373 Partial s -> do
364374 res1 <- step1 s a
365375 case res1 of
366376 Partial _ -> do
367− let fld = Fold step1 (return res1) extract1
377+ let fld = Fold step1 (return res1) extract1 final1
368378 ref <- liftIO $ newIORef fld
369379 return $ Tuple' (IsMap.mapInsert k ref kv) Nothing
370380 Done b -> return $ Tuple' kv (Just (k, b))
371381 Done b -> return $ Tuple' kv (Just (k, b))
372382
373383 {-# INLINE runFold #-}
374− runFold kv ref (Fold step1 initial1 extract1) (k, a) = do
384+ runFold kv ref (Fold step1 initial1 extract1 final1) (k, a) = do
375385 res <- initial1
376386 case res of
377387 Partial s -> do
378388 res1 <- step1 s a
379389 case res1 of
380390 Partial _ -> do
381− let fld = Fold step1 (return res1) extract1
391+ let fld = Fold step1 (return res1) extract1 final1
382392 liftIO $ writeIORef ref fld
383393 return $ Tuple' kv Nothing
384394 Done b ->
@@ -396,16 +406,27 @@ demuxGenericIO getKey getFold = fmap extract $ foldlM' step initial
396406 f <- liftIO $ readIORef ref
397407 runFold kv ref f (k, a)
398408
399− extract (Tuple' kv x) = (Prelude.mapM f kv, x)
409+ extract (Tuple' kv x) = return (Prelude.mapM f kv, x)
400410
401411 where
402412
403413 f ref = do
404− (Fold _ i e) <- liftIO $ readIORef ref
414+ Fold _ i e _ <- liftIO $ readIORef ref
405415 r <- i
406416 case r of
407417 Partial s -> e s
408− Done b -> return b
418+ _ -> error "demuxGenericIO: unreachable code"
419+
420+ final (Tuple' kv x) = return (Prelude.mapM f kv, x)
421+
422+ where
423+
424+ f ref = do
425+ Fold _ i _ fin <- liftIO $ readIORef ref
426+ r <- i
427+ case r of
428+ Partial s -> fin s
429+ _ -> error "demuxGenericIO: unreachable code"
409430
410431-- | This is specialized version of 'demux' that uses mutable IO cells as
411432-- fold accumulators for better performance.
@@ -420,6 +441,13 @@ demuxIO :: (MonadIO m, Ord k) =>
420441 -> Fold m a (m (Map k b), Maybe (k, b))
421442demuxIO = demuxGenericIO
422443
444+-- | Fold a key value stream to a key-value Map. If the same key appears
445+-- multiple times, only the last value is retained.
446+{-# INLINE kvToMapOverwriteGeneric #-}
447+kvToMapOverwriteGeneric :: (Monad m, IsMap f) => Fold m (Key f, a) (f a)
448+kvToMapOverwriteGeneric =
449+ foldl' (\kv (k, v) -> IsMap.mapInsert k v kv) IsMap.mapEmpty
450+
423451{-# INLINE demuxToContainer #-}
424452demuxToContainer :: (Monad m, IsMap f, Traversable f) =>
425453 (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
@@ -507,8 +535,8 @@ classifyGeneric :: (Monad m, IsMap f, Traversable f, Ord (Key f)) =>
507535 -- for that use case. We return an action because we want it to be lazy so
508536 -- that the downstream consumers can choose to process or discard it.
509537 (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
510−classifyGeneric f (Fold step1 initial1 extract1) =
511− fmap extract $ foldlM' step initial
538+classifyGeneric f (Fold step1 initial1 extract1 final1) =
539+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
512540
513541 where
514542
@@ -545,7 +573,16 @@ classifyGeneric f (Fold step1 initial1 extract1) =
545573 let kv1 = IsMap.mapDelete k kv
546574 in Tuple3' kv1 (Set.insert k set) (Just (k, b))
547575
548− extract (Tuple3' kv _ x) = (Prelude.mapM extract1 kv, x)
576+ extract (Tuple3' kv _ x) = return (Prelude.mapM extract1 kv, x)
577+
578+ final (Tuple3' kv set x) = return (IsMap.mapTraverseWithKey f1 kv, x)
579+
580+ where
581+
582+ f1 k s = do
583+ if Set.member k set
584+ then extract1 s
585+ else final1 s
549586
550587-- | Folds the values for each key using the supplied fold. When scanning, as
551588-- soon as the fold is complete, its result is available in the second
@@ -570,8 +607,8 @@ classify = classifyGeneric
570607{-# INLINE classifyGenericIO #-}
571608classifyGenericIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) =>
572609 (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
573−classifyGenericIO f (Fold step1 initial1 extract1) =
574− fmap extract $ foldlM' step initial
610+classifyGenericIO f (Fold step1 initial1 extract1 final1) =
611+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
575612
576613 where
577614
@@ -610,8 +647,21 @@ classifyGenericIO f (Fold step1 initial1 extract1) =
610647 in return
611648 $ Tuple3' kv1 (Set.insert k set) (Just (k, b))
612649
613− extract (Tuple3' kv _ x) =
614− (Prelude.mapM (\ref -> liftIO (readIORef ref) >>= extract1) kv, x)
650+ extract (Tuple3' kv _ x) = return (Prelude.mapM g kv, x)
651+
652+ where
653+
654+ g ref = liftIO (readIORef ref) >>= extract1
655+
656+ final (Tuple3' kv set x) = return (IsMap.mapTraverseWithKey g kv, x)
657+
658+ where
659+
660+ g k ref = do
661+ s <- liftIO $ readIORef ref
662+ if Set.member k set
663+ then extract1 s
664+ else final1 s
615665
616666-- | Same as classify except that it uses mutable IORef cells in the
617667-- Map providing better performance. Be aware that if this is used as a scan,
@@ -626,13 +676,6 @@ classifyIO :: (MonadIO m, Ord k) =>
626676 (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
627677classifyIO = classifyGenericIO
628678
629−-- | Fold a key value stream to a key-value Map. If the same key appears
630−-- multiple times, only the last value is retained.
631−{-# INLINE kvToMapOverwriteGeneric #-}
632−kvToMapOverwriteGeneric :: (Monad m, IsMap f) => Fold m (Key f, a) (f a)
633−kvToMapOverwriteGeneric =
634− foldl' (\kv (k, v) -> IsMap.mapInsert k v kv) IsMap.mapEmpty
635−
636679{-# INLINE toContainer #-}
637680toContainer :: (Monad m, IsMap f, Traversable f, Ord (Key f)) =>
638681 (a -> Key f) -> Fold m a b -> Fold m a (f b)
core/src/Streamly/Internal/Data/Fold/Step.hs+2−2
@@ -8,7 +8,7 @@
88--
99module Streamly.Internal.Data.Fold.Step
1010 (
11− -- * Types
11+ -- * Step Type
1212 Step (..)
1313
1414 , mapMStep
@@ -40,7 +40,7 @@ data Step s b
4040 = Partial !s
4141 | Done !b
4242
43−-- | 'first' maps over 'Partial' and 'second' maps over 'Done'.
43+-- | 'first' maps over the fold state and 'second' maps over the fold result.
4444--
4545instance Bifunctor Step where
4646 {-# INLINE bimap #-}
core/src/Streamly/Internal/Data/Fold/Tee.hs+2
@@ -16,7 +16,9 @@ module Streamly.Internal.Data.Fold.Tee
1616 )
1717where
1818
19+#if !MIN_VERSION_base(4,18,0)
1920import Control.Applicative (liftA2)
21+#endif
2022import Streamly.Internal.Data.Fold.Type (Fold)
2123
2224import qualified Streamly.Internal.Data.Fold.Type as Fold
core/src/Streamly/Internal/Data/Fold/Type.hs+344−120
@@ -344,11 +344,9 @@
344344--
345345module Streamly.Internal.Data.Fold.Type
346346 (
347− -- * Imports
348− -- $setup
347+ module Streamly.Internal.Data.Fold.Step
349348
350− -- * Types
351− Step (..)
349+ -- * Fold Type
352350 , Fold (..)
353351
354352 -- * Constructors
@@ -369,6 +367,8 @@ module Streamly.Internal.Data.Fold.Type
369367 , toList
370368 , toStreamK
371369 , toStreamKRev
370+ , lengthGeneric
371+ , length
372372
373373 -- * Combinators
374374
@@ -393,6 +393,8 @@ module Streamly.Internal.Data.Fold.Type
393393 -- ** Trimming
394394 , take
395395 , taking
396+ , takeEndBy_
397+ , takeEndBy
396398 , dropping
397399
398400 -- ** Sequential application
@@ -444,21 +446,25 @@ where
444446
445447#include "inline.hs"
446448
449+#if !MIN_VERSION_base(4,18,0)
447450import Control.Applicative (liftA2)
448−import Control.Monad ((>=>))
451+#endif
452+import Control.Monad ((>=>), void)
449453import Data.Bifunctor (Bifunctor(..))
450454import Data.Either (fromLeft, fromRight, isLeft, isRight)
451455import Data.Functor.Identity (Identity(..))
452456import Fusion.Plugin.Types (Fuse(..))
453−import Streamly.Internal.Data.Fold.Step (Step(..), mapMStep, chainStepM)
454457import Streamly.Internal.Data.Maybe.Strict (Maybe'(..), toMaybe)
455458import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
456459import Streamly.Internal.Data.Refold.Type (Refold(..))
457460
458−import qualified Streamly.Internal.Data.Stream.StreamK.Type as K
461+import qualified Streamly.Internal.Data.StreamK.Type as K
459462
460463import Prelude hiding (Foldable(..), concatMap, filter, map, take)
461464
465+-- Entire module is exported, do not import selectively
466+import Streamly.Internal.Data.Fold.Step
467+
462468#include "DocTestDataFold.hs"
463469
464470------------------------------------------------------------------------------
@@ -469,23 +475,64 @@ import Prelude hiding (Foldable(..), concatMap, filter, map, take)
469475-- The type @b@ is the accumulator of the writer. That's the reason the
470476-- default folds in various modules are called "write".
471477
472−-- | The type @Fold m a b@ having constructor @Fold step initial extract@
473−-- represents a fold over an input stream of values of type @a@ to a final
474−-- value of type @b@ in 'Monad' @m@.
475−--
476−-- The fold uses an intermediate state @s@ as accumulator, the type @s@ is
477−-- internal to the specific fold definition. The initial value of the fold
478−-- state @s@ is returned by @initial@. The @step@ function consumes an input
479−-- and either returns the final result @b@ if the fold is done or the next
480−-- intermediate state (see 'Step'). At any point the fold driver can extract
481−-- the result from the intermediate state using the @extract@ function.
478+-- An alternative to using an "extract" function is to use "Partial s b" style
479+-- partial value so that we always emit the output value and there is no need
480+-- to extract. Then extract can be used for cleanup purposes. But in this case
481+-- in some cases we may need a "Continue" constructor where an output value is
482+-- not available, this was implicit earlier. Also, "b" should be lazy here so
483+-- that we do not always compute it even if we do not need it.
484+--
485+-- Partial s b --> extract :: s -> b
486+-- Continue --> extract :: s -> Maybe b
487+--
488+-- But keeping 'b' lazy does not let the fold optimize well. It leads to
489+-- significant regressions in the key-value folds.
490+--
491+-- The "final" function complicates combinators that take other folds as
492+-- argument because we need to call their finalizers at right places. An
493+-- alternative to reduce this complexity where it is not required is to use a
494+-- separate type for bracketed folds but then we need to manage the complexity
495+-- of two different fold types.
496+
497+-- The "final" function could be (s -> m (Step s b)), like in parsers so that
498+-- it can be called in a loop to drain the fold.
499+
500+-- | The type @Fold m a b@ represents a consumer of an input stream of values
501+-- of type @a@ and returning a final value of type @b@ in 'Monad' @m@. The
502+-- constructor of a fold is @Fold step initial extract final@.
503+--
504+-- The fold uses an internal state of type @s@. The initial value of the state
505+-- @s@ is created by @initial@. This function is called once and only once
506+-- before the fold starts consuming input. Any resource allocation can be done
507+-- in this function.
508+--
509+-- The @step@ function is called on each input, it consumes an input and
510+-- returns the next intermediate state (see 'Step') or the final result @b@ if
511+-- the fold terminates.
512+--
513+-- If the fold is used as a scan, the @extract@ function is used by the scan
514+-- driver to map the current state @s@ of the fold to the fold result. Thus
515+-- @extract@ can be called multiple times. In some folds, where scanning does
516+-- not make sense, this function is left unimplemented; such folds cannot be
517+-- used as scans.
518+--
519+-- Before a fold terminates, @final@ is called once and only once (unless the
520+-- fold terminated in @initial@ itself). Any resources allocated by @initial@
521+-- can be released in @final@. In folds that do not require any cleanup
522+-- @extract@ and @final@ are typically the same.
523+--
524+-- When implementing fold combinators, care should be taken to cleanup any
525+-- state of the argument folds held by the fold by calling the respective
526+-- @final@ at all exit points of the fold. Also, @final@ should not be called
527+-- more than once. Note that if a fold terminates by 'Done' constructor, there
528+-- is no state to cleanup.
482529--
483530-- NOTE: The constructor is not yet released, smart constructors are provided
484531-- to create folds.
485532--
486533data Fold m a b =
487− -- | @Fold @ @ step @ @ initial @ @ extract@
488− forall s. Fold (s -> a -> m (Step s b)) (m (Step s b)) (s -> m b)
534+ -- | @Fold@ @step@ @initial@ @extract@ @final@
535+ forall s. Fold (s -> a -> m (Step s b)) (m (Step s b)) (s -> m b) (s -> m b)
489536
490537------------------------------------------------------------------------------
491538-- Mapping on the output
@@ -495,7 +542,8 @@ data Fold m a b =
495542--
496543{-# INLINE rmapM #-}
497544rmapM :: Monad m => (b -> m c) -> Fold m a b -> Fold m a c
498−rmapM f (Fold step initial extract) = Fold step1 initial1 (extract >=> f)
545+rmapM f (Fold step initial extract final) =
546+ Fold step1 initial1 (extract >=> f) (final >=> f)
499547
500548 where
501549
@@ -526,6 +574,7 @@ foldl' step initial =
526574 (\s a -> return $ Partial $ step s a)
527575 (return (Partial initial))
528576 return
577+ return
529578
530579-- | Make a fold from a left fold style monadic step function and initial value
531580-- of the accumulator.
@@ -540,7 +589,7 @@ foldl' step initial =
540589{-# INLINE foldlM' #-}
541590foldlM' :: Monad m => (b -> a -> m b) -> m b -> Fold m a b
542591foldlM' step initial =
543− Fold (\s a -> Partial <$> step s a) (Partial <$> initial) return
592+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) return return
544593
545594-- | Make a strict left fold, for non-empty streams, using first element as the
546595-- starting value. Returns Nothing if the stream is empty.
@@ -638,7 +687,11 @@ foldrM' g z =
638687{-# INLINE foldt' #-}
639688foldt' :: Monad m => (s -> a -> Step s b) -> Step s b -> (s -> b) -> Fold m a b
640689foldt' step initial extract =
641− Fold (\s a -> return $ step s a) (return initial) (return . extract)
690+ Fold
691+ (\s a -> return $ step s a)
692+ (return initial)
693+ (return . extract)
694+ (return . extract)
642695
643696-- | Make a terminating fold with an effectful step function and initial state,
644697-- and a state extraction function.
@@ -651,7 +704,7 @@ foldt' step initial extract =
651704--
652705{-# INLINE foldtM' #-}
653706foldtM' :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> Fold m a b
654−foldtM' = Fold
707+foldtM' step initial extract = Fold step initial extract extract
655708
656709------------------------------------------------------------------------------
657710-- Refold
@@ -665,7 +718,7 @@ foldtM' = Fold
665718-- /Internal/
666719fromRefold :: Refold m c a b -> c -> Fold m a b
667720fromRefold (Refold step inject extract) c =
668− Fold step (inject c) extract
721+ Fold step (inject c) extract extract
669722
670723------------------------------------------------------------------------------
671724-- Basic Folds
@@ -718,6 +771,29 @@ toStreamKRev = foldl' (flip K.cons) K.nil
718771toStreamK :: Monad m => Fold m a (K.StreamK n a)
719772toStreamK = foldr K.cons K.nil
720773
774+-- | Like 'length', except with a more general 'Num' return value
775+--
776+-- Definition:
777+--
778+-- >>> lengthGeneric = fmap getSum $ Fold.foldMap (Sum . const 1)
779+-- >>> lengthGeneric = Fold.foldl' (\n _ -> n + 1) 0
780+--
781+-- /Pre-release/
782+{-# INLINE lengthGeneric #-}
783+lengthGeneric :: (Monad m, Num b) => Fold m a b
784+lengthGeneric = foldl' (\n _ -> n + 1) 0
785+
786+-- | Determine the length of the input stream.
787+--
788+-- Definition:
789+--
790+-- >>> length = Fold.lengthGeneric
791+-- >>> length = fmap getSum $ Fold.foldMap (Sum . const 1)
792+--
793+{-# INLINE length #-}
794+length :: Monad m => Fold m a Int
795+length = lengthGeneric
796+
721797------------------------------------------------------------------------------
722798-- Instances
723799------------------------------------------------------------------------------
@@ -725,7 +801,8 @@ toStreamK = foldr K.cons K.nil
725801-- | Maps a function on the output of the fold (the type @b@).
726802instance Functor m => Functor (Fold m a) where
727803 {-# INLINE fmap #-}
728− fmap f (Fold step1 initial1 extract) = Fold step initial (fmap2 f extract)
804+ fmap f (Fold step1 initial1 extract final) =
805+ Fold step initial (fmap2 f extract) (fmap2 f final)
729806
730807 where
731808
@@ -760,7 +837,7 @@ instance Functor m => Functor (Fold m a) where
760837--
761838{-# INLINE fromPure #-}
762839fromPure :: Applicative m => b -> Fold m a b
763−fromPure b = Fold undefined (pure $ Done b) pure
840+fromPure b = Fold undefined (pure $ Done b) pure pure
764841
765842-- | Make a fold that yields the result of the supplied effectful action
766843-- without consuming any further input.
@@ -769,7 +846,7 @@ fromPure b = Fold undefined (pure $ Done b) pure
769846--
770847{-# INLINE fromEffect #-}
771848fromEffect :: Applicative m => m b -> Fold m a b
772−fromEffect b = Fold undefined (Done <$> b) pure
849+fromEffect b = Fold undefined (Done <$> b) pure pure
773850
774851{-# ANN type SeqFoldState Fuse #-}
775852data SeqFoldState sl f sr = SeqFoldL !sl | SeqFoldR !f !sr
@@ -794,16 +871,20 @@ data SeqFoldState sl f sr = SeqFoldL !sl | SeqFoldR !f !sr
794871-- complexity, because each composition adds a new branch that each subsequent
795872-- fold's input element has to traverse, therefore, it cannot scale to a large
796873-- number of compositions. After around 100 compositions the performance starts
797−-- dipping rapidly compared to a CPS style implementation. When you need
798−-- scaling use parser monad instead.
874+-- dipping rapidly compared to a CPS style implementation.
875+--
876+-- For larger number of compositions you can convert the fold to a parser and
877+-- use ParserK.
799878--
800879-- /Time: O(n^2) where n is the number of compositions./
801880--
802881{-# INLINE splitWith #-}
803882splitWith :: Monad m =>
804883 (a -> b -> c) -> Fold m x a -> Fold m x b -> Fold m x c
805−splitWith func (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
806− Fold step initial extract
884+splitWith func
885+ (Fold stepL initialL _ finalL)
886+ (Fold stepR initialR _ finalR) =
887+ Fold step initial extract final
807888
808889 where
809890
@@ -820,13 +901,18 @@ splitWith func (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
820901 step (SeqFoldL st) a = runL (stepL st a)
821902 step (SeqFoldR f st) a = runR (stepR st a) f
822903
823− extract (SeqFoldR f sR) = fmap f (extractR sR)
824− extract (SeqFoldL sL) = do
825− rL <- extractL sL
904+ -- XXX splitWith should not be used for scanning
905+ -- It would rarely make sense and resource tracking and cleanup would be
906+ -- expensive. especially when multiple splitWith are chained.
907+ extract _ = error "splitWith: cannot be used for scanning"
908+
909+ final (SeqFoldR f sR) = fmap f (finalR sR)
910+ final (SeqFoldL sL) = do
911+ rL <- finalL sL
826912 res <- initialR
827913 fmap (func rL)
828914 $ case res of
829− Partial sR -> extractR sR
915+ Partial sR -> finalR sR
830916 Done rR -> return rR
831917
832918{-# DEPRECATED serialWith "Please use \"splitWith\" instead" #-}
@@ -846,8 +932,8 @@ data SeqFoldState_ sl sr = SeqFoldL_ !sl | SeqFoldR_ !sr
846932--
847933{-# INLINE split_ #-}
848934split_ :: Monad m => Fold m x a -> Fold m x b -> Fold m x b
849−split_ (Fold stepL initialL _) (Fold stepR initialR extractR) =
850− Fold step initial extract
935+split_ (Fold stepL initialL _ finalL) (Fold stepR initialR _ finalR) =
936+ Fold step initial extract final
851937
852938 where
853939
@@ -870,11 +956,16 @@ split_ (Fold stepL initialL _) (Fold stepR initialR extractR) =
870956 resR <- stepR st a
871957 return $ first SeqFoldR_ resR
872958
873− extract (SeqFoldR_ sR) = extractR sR
874− extract (SeqFoldL_ _) = do
959+ -- XXX split_ should not be used for scanning
960+ -- See splitWith for more details.
961+ extract _ = error "split_: cannot be used for scanning"
962+
963+ final (SeqFoldR_ sR) = finalR sR
964+ final (SeqFoldL_ sL) = do
965+ _ <- finalL sL
875966 res <- initialR
876967 case res of
877− Partial sR -> extractR sR
968+ Partial sR -> finalR sR
878969 Done rR -> return rR
879970
880971-- | 'Applicative' form of 'splitWith'. Split the input serially over two
@@ -922,8 +1013,10 @@ data TeeState sL sR bL bR
9221013--
9231014{-# INLINE teeWith #-}
9241015teeWith :: Monad m => (a -> b -> c) -> Fold m x a -> Fold m x b -> Fold m x c
925−teeWith f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
926− Fold step initial extract
1016+teeWith f
1017+ (Fold stepL initialL extractL finalL)
1018+ (Fold stepR initialR extractR finalR) =
1019+ Fold step initial extract final
9271020
9281021 where
9291022
@@ -950,6 +1043,10 @@ teeWith f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
9501043 extract (TeeLeft bR sL) = (`f` bR) <$> extractL sL
9511044 extract (TeeRight bL sR) = f bL <$> extractR sR
9521045
1046+ final (TeeBoth sL sR) = f <$> finalL sL <*> finalR sR
1047+ final (TeeLeft bR sL) = (`f` bR) <$> finalL sL
1048+ final (TeeRight bL sR) = f bL <$> finalR sR
1049+
9531050{-# ANN type TeeFstState Fuse #-}
9541051data TeeFstState sL sR b
9551052 = TeeFstBoth !sL !sR
@@ -962,8 +1059,10 @@ data TeeFstState sL sR b
9621059{-# INLINE teeWithFst #-}
9631060teeWithFst :: Monad m =>
9641061 (b -> c -> d) -> Fold m a b -> Fold m a c -> Fold m a d
965−teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
966− Fold step initial extract
1062+teeWithFst f
1063+ (Fold stepL initialL extractL finalL)
1064+ (Fold stepR initialR extractR finalR) =
1065+ Fold step initial extract final
9671066
9681067 where
9691068
@@ -982,7 +1081,7 @@ teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
9821081 Done bl -> do
9831082 Done . f bl <$>
9841083 case resR of
985− Partial sr -> extractR sr
1084+ Partial sr -> finalR sr
9861085 Done br -> return br
9871086
9881087 initial = runBoth initialL initialR
@@ -993,6 +1092,9 @@ teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
9931092 extract (TeeFstBoth sL sR) = f <$> extractL sL <*> extractR sR
9941093 extract (TeeFstLeft bR sL) = (`f` bR) <$> extractL sL
9951094
1095+ final (TeeFstBoth sL sR) = f <$> finalL sL <*> finalR sR
1096+ final (TeeFstLeft bR sL) = (`f` bR) <$> finalL sL
1097+
9961098-- | Like 'teeWith' but terminates as soon as any one of the two folds
9971099-- terminates.
9981100--
@@ -1001,8 +1103,10 @@ teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10011103{-# INLINE teeWithMin #-}
10021104teeWithMin :: Monad m =>
10031105 (b -> c -> d) -> Fold m a b -> Fold m a c -> Fold m a d
1004−teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1005− Fold step initial extract
1106+teeWithMin f
1107+ (Fold stepL initialL extractL finalL)
1108+ (Fold stepR initialR extractR finalR) =
1109+ Fold step initial extract final
10061110
10071111 where
10081112
@@ -1014,12 +1118,12 @@ teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10141118 Partial sl -> do
10151119 case resR of
10161120 Partial sr -> return $ Partial $ Tuple' sl sr
1017− Done br -> Done . (`f` br) <$> extractL sl
1121+ Done br -> Done . (`f` br) <$> finalL sl
10181122
10191123 Done bl -> do
10201124 Done . f bl <$>
10211125 case resR of
1022− Partial sr -> extractR sr
1126+ Partial sr -> finalR sr
10231127 Done br -> return br
10241128
10251129 initial = runBoth initialL initialR
@@ -1028,6 +1132,8 @@ teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10281132
10291133 extract (Tuple' sL sR) = f <$> extractL sL <*> extractR sR
10301134
1135+ final (Tuple' sL sR) = f <$> finalL sL <*> finalR sR
1136+
10311137-- | Shortest alternative. Apply both folds in parallel but choose the result
10321138-- from the one which consumed least input i.e. take the shortest succeeding
10331139-- fold.
@@ -1039,8 +1145,8 @@ teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10391145--
10401146{-# INLINE shortest #-}
10411147shortest :: Monad m => Fold m x a -> Fold m x b -> Fold m x (Either a b)
1042−shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
1043− Fold step initial extract
1148+shortest (Fold stepL initialL extractL finalL) (Fold stepR initialR _ finalR) =
1149+ Fold step initial extract final
10441150
10451151 where
10461152
@@ -1048,10 +1154,16 @@ shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
10481154 runBoth actionL actionR = do
10491155 resL <- actionL
10501156 resR <- actionR
1051− return $
1052− case resL of
1053− Partial sL -> bimap (Tuple' sL) Right resR
1054− Done bL -> Done $ Left bL
1157+ case resL of
1158+ Partial sL ->
1159+ case resR of
1160+ Partial sR -> return $ Partial $ Tuple' sL sR
1161+ Done bR -> finalL sL >> return (Done (Right bR))
1162+ Done bL -> do
1163+ case resR of
1164+ Partial sR -> void (finalR sR)
1165+ Done _ -> return ()
1166+ return (Done (Left bL))
10551167
10561168 initial = runBoth initialL initialR
10571169
@@ -1059,6 +1171,8 @@ shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
10591171
10601172 extract (Tuple' sL _) = Left <$> extractL sL
10611173
1174+ final (Tuple' sL sR) = Left <$> finalL sL <* finalR sR
1175+
10621176{-# ANN type LongestState Fuse #-}
10631177data LongestState sL sR
10641178 = LongestBoth !sL !sR
@@ -1076,8 +1190,10 @@ data LongestState sL sR
10761190--
10771191{-# INLINE longest #-}
10781192longest :: Monad m => Fold m x a -> Fold m x b -> Fold m x (Either a b)
1079−longest (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1080− Fold step initial extract
1193+longest
1194+ (Fold stepL initialL _ finalL)
1195+ (Fold stepR initialR _ finalR) =
1196+ Fold step initial extract final
10811197
10821198 where
10831199
@@ -1100,14 +1216,18 @@ longest (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
11001216 step (LongestLeft sL) a = bimap LongestLeft Left <$> stepL sL a
11011217 step (LongestRight sR) a = bimap LongestRight Right <$> stepR sR a
11021218
1103− left sL = Left <$> extractL sL
1104− extract (LongestLeft sL) = left sL
1105− extract (LongestRight sR) = Right <$> extractR sR
1106− extract (LongestBoth sL _) = left sL
1219+ -- XXX Scan with this may not make sense as we cannot determine the longest
1220+ -- until one of them have exhausted.
1221+ extract _ = error $ "longest: scan is not allowed as longest cannot be "
1222+ ++ "determined until one fold has exhausted."
1223+
1224+ final (LongestLeft sL) = Left <$> finalL sL
1225+ final (LongestRight sR) = Right <$> finalR sR
1226+ final (LongestBoth sL sR) = Left <$> finalL sL <* finalR sR
11071227
1108−data ConcatMapState m sa a c
1109− = B !sa
1110− | forall s. C (s -> a -> m (Step s c)) !s (s -> m c)
1228+data ConcatMapState m sa a b c
1229+ = B !sa (sa -> m b)
1230+ | forall s. C (s -> a -> m (Step s c)) !s (s -> m c) (s -> m c)
11111231
11121232-- | Map a 'Fold' returning function on the result of a 'Fold' and run the
11131233-- returned fold. This operation can be used to express data dependencies
@@ -1130,43 +1250,47 @@ data ConcatMapState m sa a c
11301250--
11311251{-# INLINE concatMap #-}
11321252concatMap :: Monad m => (b -> Fold m a c) -> Fold m a b -> Fold m a c
1133−concatMap f (Fold stepa initiala extracta) = Fold stepc initialc extractc
1253+concatMap f (Fold stepa initiala _ finala) =
1254+ Fold stepc initialc extractc finalc
11341255 where
11351256 initialc = do
11361257 r <- initiala
11371258 case r of
1138− Partial s -> return $ Partial (B s)
1259+ Partial s -> return $ Partial (B s finala)
11391260 Done b -> initInnerFold (f b)
11401261
1141− stepc (B s) a = do
1262+ stepc (B s fin) a = do
11421263 r <- stepa s a
11431264 case r of
1144− Partial s1 -> return $ Partial (B s1)
1265+ Partial s1 -> return $ Partial (B s1 fin)
11451266 Done b -> initInnerFold (f b)
11461267
1147− stepc (C stepInner s extractInner) a = do
1268+ stepc (C stepInner s extractInner fin) a = do
11481269 r <- stepInner s a
11491270 return $ case r of
1150− Partial sc -> Partial (C stepInner sc extractInner)
1271+ Partial sc -> Partial (C stepInner sc extractInner fin)
11511272 Done c -> Done c
11521273
1153− extractc (B s) = do
1154− r <- extracta s
1155− initExtract (f r)
1156− extractc (C _ sInner extractInner) = extractInner sInner
1274+ -- XXX Cannot use for scanning
1275+ extractc _ = error "concatMap: cannot be used for scanning"
11571276
1158− initInnerFold (Fold step i e) = do
1277+ initInnerFold (Fold step i e fin) = do
11591278 r <- i
11601279 return $ case r of
1161− Partial s -> Partial (C step s e)
1280+ Partial s -> Partial (C step s e fin)
11621281 Done c -> Done c
11631282
1164− initExtract (Fold _ i e) = do
1283+ initFinalize (Fold _ i _ fin) = do
11651284 r <- i
11661285 case r of
1167− Partial s -> e s
1286+ Partial s -> fin s
11681287 Done c -> return c
11691288
1289+ finalc (B s fin) = do
1290+ r <- fin s
1291+ initFinalize (f r)
1292+ finalc (C _ sInner _ fin) = fin sInner
1293+
11701294------------------------------------------------------------------------------
11711295-- Mapping on input
11721296------------------------------------------------------------------------------
@@ -1185,7 +1309,7 @@ concatMap f (Fold stepa initiala extracta) = Fold stepc initialc extractc
11851309--
11861310{-# INLINE lmap #-}
11871311lmap :: (a -> b) -> Fold m b r -> Fold m a r
1188−lmap f (Fold step begin done) = Fold step' begin done
1312+lmap f (Fold step begin done final) = Fold step' begin done final
11891313 where
11901314 step' x a = step x (f a)
11911315
@@ -1193,7 +1317,7 @@ lmap f (Fold step begin done) = Fold step' begin done
11931317--
11941318{-# INLINE lmapM #-}
11951319lmapM :: Monad m => (a -> m b) -> Fold m b r -> Fold m a r
1196−lmapM f (Fold step begin done) = Fold step' begin done
1320+lmapM f (Fold step begin done final) = Fold step' begin done final
11971321 where
11981322 step' x a = f a >>= step x
11991323
@@ -1205,8 +1329,10 @@ lmapM f (Fold step begin done) = Fold step' begin done
12051329-- /Pre-release/
12061330{-# INLINE postscan #-}
12071331postscan :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1208−postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1209− Fold step initial extract
1332+postscan
1333+ (Fold stepL initialL extractL finalL)
1334+ (Fold stepR initialR extractR finalR) =
1335+ Fold step initial extract final
12101336
12111337 where
12121338
@@ -1217,30 +1343,32 @@ postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
12171343 Done bL -> do
12181344 rR <- stepR sR bL
12191345 case rR of
1220− Partial sR1 -> Done <$> extractR sR1
1346+ Partial sR1 -> Done <$> finalR sR1
12211347 Done bR -> return $ Done bR
12221348 Partial sL -> do
12231349 !b <- extractL sL
12241350 rR <- stepR sR b
1225− return
1226− $ case rR of
1227− Partial sR1 -> Partial (sL, sR1)
1228− Done bR -> Done bR
1351+ case rR of
1352+ Partial sR1 -> return $ Partial (sL, sR1)
1353+ Done bR -> finalL sL >> return (Done bR)
12291354
12301355 initial = do
1231− r <- initialR
1232− rL <- initialL
1233− case r of
1234− Partial sR ->
1356+ rR <- initialR
1357+ case rR of
1358+ Partial sR -> do
1359+ rL <- initialL
12351360 case rL of
1236− Done _ -> Done <$> extractR sR
1361+ Done _ -> Done <$> finalR sR
12371362 Partial sL -> return $ Partial (sL, sR)
12381363 Done b -> return $ Done b
12391364
1365+ -- XXX should use Tuple'
12401366 step (sL, sR) x = runStep (stepL sL x) sR
12411367
12421368 extract = extractR . snd
12431369
1370+ final (sL, sR) = finalL sL *> finalR sR
1371+
12441372------------------------------------------------------------------------------
12451373-- Filtering
12461374------------------------------------------------------------------------------
@@ -1253,7 +1381,7 @@ postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
12531381--
12541382{-# INLINE_NORMAL catMaybes #-}
12551383catMaybes :: Monad m => Fold m a b -> Fold m (Maybe a) b
1256−catMaybes (Fold step initial extract) = Fold step1 initial extract
1384+catMaybes (Fold step initial extract final) = Fold step1 initial extract final
12571385
12581386 where
12591387
@@ -1293,7 +1421,7 @@ filtering f = foldl' step Nothing
12931421{-# INLINE filter #-}
12941422filter :: Monad m => (a -> Bool) -> Fold m a r -> Fold m a r
12951423-- filter p = scanMaybe (filtering p)
1296−filter f (Fold step begin done) = Fold step' begin done
1424+filter f (Fold step begin extract final) = Fold step' begin extract final
12971425 where
12981426 step' x a = if f a then step x a else return $ Partial x
12991427
@@ -1304,7 +1432,7 @@ filter f (Fold step begin done) = Fold step' begin done
13041432--
13051433{-# INLINE filterM #-}
13061434filterM :: Monad m => (a -> m Bool) -> Fold m a r -> Fold m a r
1307−filterM f (Fold step begin done) = Fold step' begin done
1435+filterM f (Fold step begin extract final) = Fold step' begin extract final
13081436 where
13091437 step' x a = do
13101438 use <- f a
@@ -1393,7 +1521,7 @@ dropping n = foldt' step initial extract
13931521{-# INLINE take #-}
13941522take :: Monad m => Int -> Fold m a b -> Fold m a b
13951523-- take n = scanMaybe (taking n)
1396−take n (Fold fstep finitial fextract) = Fold step initial extract
1524+take n (Fold fstep finitial fextract ffinal) = Fold step initial extract final
13971525
13981526 where
13991527
@@ -1405,7 +1533,7 @@ take n (Fold fstep finitial fextract) = Fold step initial extract
14051533 s1 = Tuple'Fused i1 s
14061534 if i1 < n
14071535 then return $ Partial s1
1408− else Done <$> fextract s
1536+ else Done <$> ffinal s
14091537 Done b -> return $ Done b
14101538
14111539 initial = finitial >>= next (-1)
@@ -1414,6 +1542,72 @@ take n (Fold fstep finitial fextract) = Fold step initial extract
14141542
14151543 extract (Tuple'Fused _ r) = fextract r
14161544
1545+ final (Tuple'Fused _ r) = ffinal r
1546+
1547+-- Note: Keep this consistent with S.splitOn. In fact we should eliminate
1548+-- S.splitOn in favor of the fold.
1549+--
1550+-- XXX Use Fold.many instead once it is fixed.
1551+-- > Stream.splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
1552+
1553+-- | Like 'takeEndBy' but drops the element on which the predicate succeeds.
1554+--
1555+-- Example:
1556+--
1557+-- >>> input = Stream.fromList "hello\nthere\n"
1558+-- >>> line = Fold.takeEndBy_ (== '\n') Fold.toList
1559+-- >>> Stream.fold line input
1560+-- "hello"
1561+--
1562+-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1563+-- ["hello","there"]
1564+--
1565+{-# INLINE takeEndBy_ #-}
1566+takeEndBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1567+-- takeEndBy_ predicate = scanMaybe (takingEndBy_ predicate)
1568+takeEndBy_ predicate (Fold fstep finitial fextract ffinal) =
1569+ Fold step finitial fextract ffinal
1570+
1571+ where
1572+
1573+ step s a =
1574+ if not (predicate a)
1575+ then fstep s a
1576+ else Done <$> ffinal s
1577+
1578+-- Note:
1579+-- > Stream.splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
1580+
1581+-- | Take the input, stop when the predicate succeeds taking the succeeding
1582+-- element as well.
1583+--
1584+-- Example:
1585+--
1586+-- >>> input = Stream.fromList "hello\nthere\n"
1587+-- >>> line = Fold.takeEndBy (== '\n') Fold.toList
1588+-- >>> Stream.fold line input
1589+-- "hello\n"
1590+--
1591+-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1592+-- ["hello\n","there\n"]
1593+--
1594+{-# INLINE takeEndBy #-}
1595+takeEndBy :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1596+-- takeEndBy predicate = scanMaybe (takingEndBy predicate)
1597+takeEndBy predicate (Fold fstep finitial fextract ffinal) =
1598+ Fold step finitial fextract ffinal
1599+
1600+ where
1601+
1602+ step s a = do
1603+ res <- fstep s a
1604+ if not (predicate a)
1605+ then return res
1606+ else do
1607+ case res of
1608+ Partial s1 -> Done <$> ffinal s1
1609+ Done b -> return $ Done b
1610+
14171611------------------------------------------------------------------------------
14181612-- Nesting
14191613------------------------------------------------------------------------------
@@ -1431,8 +1625,8 @@ take n (Fold fstep finitial fextract) = Fold step initial extract
14311625-- /Pre-release/
14321626{-# INLINE duplicate #-}
14331627duplicate :: Monad m => Fold m a b -> Fold m a (Fold m a b)
1434−duplicate (Fold step1 initial1 extract1) =
1435− Fold step initial (\s -> pure $ Fold step1 (pure $ Partial s) extract1)
1628+duplicate (Fold step1 initial1 extract1 final1) =
1629+ Fold step initial extract final
14361630
14371631 where
14381632
@@ -1440,6 +1634,11 @@ duplicate (Fold step1 initial1 extract1) =
14401634
14411635 step s a = second fromPure <$> step1 s a
14421636
1637+ -- Scanning may be problematic due to multiple finalizations.
1638+ extract = error "duplicate: scanning may be problematic"
1639+
1640+ final s = pure $ Fold step1 (pure $ Partial s) extract1 final1
1641+
14431642-- If there were a finalize/flushing action in the stream type that would be
14441643-- equivalent to running initialize in Fold. But we do not have a flushing
14451644-- action in streams.
@@ -1451,9 +1650,9 @@ duplicate (Fold step1 initial1 extract1) =
14511650-- /Pre-release/
14521651{-# INLINE reduce #-}
14531652reduce :: Monad m => Fold m a b -> m (Fold m a b)
1454−reduce (Fold step initial extract) = do
1653+reduce (Fold step initial extract final) = do
14551654 i <- initial
1456− return $ Fold step (return i) extract
1655+ return $ Fold step (return i) extract final
14571656
14581657-- This is the dual of Stream @cons@.
14591658
@@ -1463,7 +1662,8 @@ reduce (Fold step initial extract) = do
14631662-- /Pre-release/
14641663{-# INLINE snoclM #-}
14651664snoclM :: Monad m => Fold m a b -> m a -> Fold m a b
1466−snoclM (Fold fstep finitial fextract) action = Fold fstep initial fextract
1665+snoclM (Fold fstep finitial fextract ffinal) action =
1666+ Fold fstep initial fextract ffinal
14671667
14681668 where
14691669
@@ -1490,7 +1690,8 @@ snoclM (Fold fstep finitial fextract) action = Fold fstep initial fextract
14901690{-# INLINE snocl #-}
14911691snocl :: Monad m => Fold m a b -> a -> Fold m a b
14921692-- snocl f = snoclM f . return
1493−snocl (Fold fstep finitial fextract) a = Fold fstep initial fextract
1693+snocl (Fold fstep finitial fextract ffinal) a =
1694+ Fold fstep initial fextract ffinal
14941695
14951696 where
14961697
@@ -1510,12 +1711,12 @@ snocl (Fold fstep finitial fextract) a = Fold fstep initial fextract
15101711-- /Pre-release/
15111712{-# INLINE snocM #-}
15121713snocM :: Monad m => Fold m a b -> m a -> m (Fold m a b)
1513−snocM (Fold step initial extract) action = do
1714+snocM (Fold step initial extract final) action = do
15141715 res <- initial
15151716 r <- case res of
15161717 Partial fs -> action >>= step fs
15171718 Done _ -> return res
1518− return $ Fold step (return r) extract
1719+ return $ Fold step (return r) extract final
15191720
15201721-- Definitions:
15211722--
@@ -1534,12 +1735,12 @@ snocM (Fold step initial extract) action = do
15341735-- /Pre-release/
15351736{-# INLINE snoc #-}
15361737snoc :: Monad m => Fold m a b -> a -> m (Fold m a b)
1537−snoc (Fold step initial extract) a = do
1738+snoc (Fold step initial extract final) a = do
15381739 res <- initial
15391740 r <- case res of
15401741 Partial fs -> step fs a
15411742 Done _ -> return res
1542− return $ Fold step (return r) extract
1743+ return $ Fold step (return r) extract final
15431744
15441745-- | Append a singleton value to the fold.
15451746--
@@ -1567,7 +1768,7 @@ addOne = flip snoc
15671768-- /Pre-release/
15681769{-# INLINE extractM #-}
15691770extractM :: Monad m => Fold m a b -> m b
1570−extractM (Fold _ initial extract) = do
1771+extractM (Fold _ initial extract _) = do
15711772 res <- initial
15721773 case res of
15731774 Partial fs -> extract fs
@@ -1576,14 +1777,15 @@ extractM (Fold _ initial extract) = do
15761777-- | Close a fold so that it does not accept any more input.
15771778{-# INLINE close #-}
15781779close :: Monad m => Fold m a b -> Fold m a b
1579−close (Fold _ initial1 extract1) = Fold undefined initial undefined
1780+close (Fold _ initial1 _ final1) =
1781+ Fold undefined initial undefined undefined
15801782
15811783 where
15821784
15831785 initial = do
15841786 res <- initial1
15851787 case res of
1586− Partial s -> Done <$> extract1 s
1788+ Partial s -> Done <$> final1 s
15871789 Done b -> return $ Done b
15881790
15891791-- Corresponds to the null check for streams.
@@ -1593,7 +1795,7 @@ close (Fold _ initial1 extract1) = Fold undefined initial undefined
15931795-- /Pre-release/
15941796{-# INLINE isClosed #-}
15951797isClosed :: Monad m => Fold m a b -> m Bool
1596−isClosed (Fold _ initial _) = do
1798+isClosed (Fold _ initial _ _) = do
15971799 res <- initial
15981800 return $ case res of
15991801 Partial _ -> False
@@ -1627,8 +1829,10 @@ data ManyState s1 s2
16271829--
16281830{-# INLINE many #-}
16291831many :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1630−many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
1631− Fold step initial extract
1832+many
1833+ (Fold sstep sinitial sextract sfinal)
1834+ (Fold cstep cinitial cextract cfinal) =
1835+ Fold step initial extract final
16321836
16331837 where
16341838
@@ -1677,6 +1881,13 @@ many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
16771881 Partial s -> cextract s
16781882 Done b -> return b
16791883
1884+ final (ManyFirst ss cs) = sfinal ss *> cfinal cs
1885+ final (ManyLoop ss cs) = do
1886+ cres <- sfinal ss >>= cstep cs
1887+ case cres of
1888+ Partial s -> cfinal s
1889+ Done b -> return b
1890+
16801891-- | Like many, but the "first" fold emits an output at the end even if no
16811892-- input is received.
16821893--
@@ -1686,8 +1897,10 @@ many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
16861897--
16871898{-# INLINE manyPost #-}
16881899manyPost :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1689−manyPost (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
1690− Fold step initial extract
1900+manyPost
1901+ (Fold sstep sinitial sextract sfinal)
1902+ (Fold cstep cinitial cextract cfinal) =
1903+ Fold step initial extract final
16911904
16921905 where
16931906
@@ -1723,6 +1936,12 @@ manyPost (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
17231936 Partial s -> cextract s
17241937 Done b -> return b
17251938
1939+ final (Tuple' ss cs) = do
1940+ cres <- sfinal ss >>= cstep cs
1941+ case cres of
1942+ Partial s -> cfinal s
1943+ Done b -> return b
1944+
17261945-- | @groupsOf n split collect@ repeatedly applies the @split@ fold to chunks
17271946-- of @n@ items in the input stream and supplies the result to the @collect@
17281947-- fold.
@@ -1751,7 +1970,10 @@ groupsOf n split = many (take n split)
17511970--
17521971{-# INLINE refoldMany #-}
17531972refoldMany :: Monad m => Fold m a b -> Refold m x b c -> Refold m x a c
1754−refoldMany (Fold sstep sinitial sextract) (Refold cstep cinject cextract) =
1973+refoldMany
1974+ (Fold sstep sinitial sextract _sfinal)
1975+ -- XXX We will need a "final" in refold as well
1976+ (Refold cstep cinject cextract) =
17551977 Refold step inject extract
17561978
17571979 where
@@ -1802,7 +2024,9 @@ data ConsumeManyState x cs ss = ConsumeMany x cs (Either ss ss)
18022024-- /Internal/
18032025{-# INLINE refoldMany1 #-}
18042026refoldMany1 :: Monad m => Refold m x a b -> Fold m b c -> Refold m x a c
1805−refoldMany1 (Refold sstep sinject sextract) (Fold cstep cinitial cextract) =
2027+refoldMany1
2028+ (Refold sstep sinject sextract)
2029+ (Fold cstep cinitial cextract _cfinal) =
18062030 Refold step inject extract
18072031
18082032 where
@@ -1853,7 +2077,7 @@ refoldMany1 (Refold sstep sinject sextract) (Fold cstep cinitial cextract) =
18532077{-# INLINE refold #-}
18542078refold :: Monad m => Refold m b a c -> Fold m a b -> Fold m a c
18552079refold (Refold step inject extract) f =
1856− Fold step (extractM f >>= inject) extract
2080+ Fold step (extractM f >>= inject) extract extract
18572081
18582082------------------------------------------------------------------------------
18592083-- morphInner
@@ -1863,8 +2087,8 @@ refold (Refold step inject extract) f =
18632087--
18642088-- /Pre-release/
18652089morphInner :: (forall x. m x -> n x) -> Fold m a b -> Fold n a b
1866−morphInner f (Fold step initial extract) =
1867− Fold (\x a -> f $ step x a) (f initial) (f . extract)
2090+morphInner f (Fold step initial extract final) =
2091+ Fold (\x a -> f $ step x a) (f initial) (f . extract) (f . final)
18682092
18692093-- | Adapt a pure fold to any monad.
18702094--
core/src/Streamly/Internal/Data/Fold/Window.hs+59−53
@@ -36,24 +36,24 @@ module Streamly.Internal.Data.Fold.Window
3636 -- window folds by keeping the second element of the input tuple as
3737 -- @Nothing@.
3838 --
39− lmap
39+ windowLmap
4040 , cumulative
4141
42− , rollingMap
43− , rollingMapM
42+ , windowRollingMap
43+ , windowRollingMapM
4444
4545 -- ** Sums
46− , length
47− , sum
48− , sumInt
49− , powerSum
50− , powerSumFrac
46+ , windowLength
47+ , windowSum
48+ , windowSumInt
49+ , windowPowerSum
50+ , windowPowerSumFrac
5151
5252 -- ** Location
53− , minimum
54− , maximum
55− , range
56− , mean
53+ , windowMinimum
54+ , windowMaximum
55+ , windowRange
56+ , windowMean
5757 )
5858where
5959
@@ -65,16 +65,18 @@ import Streamly.Internal.Data.Fold.Type (Fold(..), Step(..))
6565import Streamly.Internal.Data.Tuple.Strict
6666 (Tuple'(..), Tuple3Fused' (Tuple3Fused'))
6767
68+import Foreign.ForeignPtr.Unsafe (unsafeForeignPtrToPtr)
69+
6870import qualified Streamly.Internal.Data.Fold.Type as Fold
69−import qualified Streamly.Internal.Data.Ring.Unboxed as Ring
71+import qualified Streamly.Internal.Data.Ring as Ring
7072
7173import Prelude hiding (length, sum, minimum, maximum)
7274
7375-- $setup
7476-- >>> import Data.Bifunctor(bimap)
7577-- >>> import qualified Streamly.Data.Fold as Fold
76−-- >>> import qualified Streamly.Internal.Data.Fold.Window as FoldW
77−-- >>> import qualified Streamly.Internal.Data.Ring.Unboxed as Ring
78+-- >>> import qualified Streamly.Internal.Data.Fold.Window as Fold
79+-- >>> import qualified Streamly.Internal.Data.Ring as Ring
7880-- >>> import qualified Streamly.Data.Stream as Stream
7981-- >>> import Prelude hiding (length, sum, minimum, maximum)
8082
@@ -87,9 +89,9 @@ import Prelude hiding (length, sum, minimum, maximum)
8789--
8890-- >>> lmap f = Fold.lmap (bimap f (f <$>))
8991--
90−{-# INLINE lmap #-}
91−lmap :: (c -> a) -> Fold m (a, Maybe a) b -> Fold m (c, Maybe c) b
92−lmap f = Fold.lmap (bimap f (f <$>))
92+{-# INLINE windowLmap #-}
93+windowLmap :: (c -> a) -> Fold m (a, Maybe a) b -> Fold m (c, Maybe c) b
94+windowLmap f = Fold.lmap (bimap f (f <$>))
9395
9496-- | Convert an incremental fold to a cumulative fold using the entire input
9597-- stream as a single window.
@@ -105,10 +107,10 @@ cumulative = Fold.lmap (, Nothing)
105107
106108-- | Apply an effectful function on the latest and the oldest element of the
107109-- window.
108−{-# INLINE rollingMapM #-}
109−rollingMapM :: Monad m =>
110+{-# INLINE windowRollingMapM #-}
111+windowRollingMapM :: Monad m =>
110112 (Maybe a -> a -> m (Maybe b)) -> Fold m (a, Maybe a) (Maybe b)
111−rollingMapM f = Fold.foldlM' f1 initial
113+windowRollingMapM f = Fold.foldlM' f1 initial
112114
113115 where
114116
@@ -118,12 +120,12 @@ rollingMapM f = Fold.foldlM' f1 initial
118120
119121-- | Apply a pure function on the latest and the oldest element of the window.
120122--
121−-- >>> rollingMap f = FoldW.rollingMapM (\x y -> return $ f x y)
123+-- >>> windowRollingMap f = Fold.windowRollingMapM (\x y -> return $ f x y)
122124--
123−{-# INLINE rollingMap #-}
124−rollingMap :: Monad m =>
125+{-# INLINE windowRollingMap #-}
126+windowRollingMap :: Monad m =>
125127 (Maybe a -> a -> Maybe b) -> Fold m (a, Maybe a) (Maybe b)
126−rollingMap f = Fold.foldl' f1 initial
128+windowRollingMap f = Fold.foldl' f1 initial
127129
128130 where
129131
@@ -146,9 +148,9 @@ rollingMap f = Fold.foldl' f1 initial
146148--
147149-- /Internal/
148150--
149−{-# INLINE sumInt #-}
150−sumInt :: forall m a. (Monad m, Integral a) => Fold m (a, Maybe a) a
151−sumInt = Fold step initial extract
151+{-# INLINE windowSumInt #-}
152+windowSumInt :: forall m a. (Monad m, Integral a) => Fold m (a, Maybe a) a
153+windowSumInt = Fold step initial extract extract
152154
153155 where
154156
@@ -180,9 +182,9 @@ sumInt = Fold step initial extract
180182--
181183-- /Time/: \(\mathcal{O}(n)\)
182184--
183−{-# INLINE sum #-}
184−sum :: forall m a. (Monad m, Num a) => Fold m (a, Maybe a) a
185−sum = Fold step initial extract
185+{-# INLINE windowSum #-}
186+windowSum :: forall m a. (Monad m, Num a) => Fold m (a, Maybe a) a
187+windowSum = Fold step initial extract extract
186188
187189 where
188190
@@ -219,9 +221,9 @@ sum = Fold step initial extract
219221--
220222-- >>> length = powerSum 0
221223--
222−{-# INLINE length #-}
223−length :: (Monad m, Num b) => Fold m (a, Maybe a) b
224−length = Fold.foldl' step 0
224+{-# INLINE windowLength #-}
225+windowLength :: (Monad m, Num b) => Fold m (a, Maybe a) b
226+windowLength = Fold.foldl' step 0
225227
226228 where
227229
@@ -237,18 +239,18 @@ length = Fold.foldl' step 0
237239-- /Space/: \(\mathcal{O}(1)\)
238240--
239241-- /Time/: \(\mathcal{O}(n)\)
240−{-# INLINE powerSum #-}
241−powerSum :: (Monad m, Num a) => Int -> Fold m (a, Maybe a) a
242−powerSum k = lmap (^ k) sum
242+{-# INLINE windowPowerSum #-}
243+windowPowerSum :: (Monad m, Num a) => Int -> Fold m (a, Maybe a) a
244+windowPowerSum k = windowLmap (^ k) windowSum
243245
244246-- | Like 'powerSum' but powers can be negative or fractional. This is slower
245247-- than 'powerSum' for positive intergal powers.
246248--
247249-- >>> powerSumFrac p = lmap (** p) sum
248250--
249−{-# INLINE powerSumFrac #-}
250−powerSumFrac :: (Monad m, Floating a) => a -> Fold m (a, Maybe a) a
251−powerSumFrac p = lmap (** p) sum
251+{-# INLINE windowPowerSumFrac #-}
252+windowPowerSumFrac :: (Monad m, Floating a) => a -> Fold m (a, Maybe a) a
253+windowPowerSumFrac p = windowLmap (** p) windowSum
252254
253255-------------------------------------------------------------------------------
254256-- Location
@@ -265,9 +267,9 @@ powerSumFrac p = lmap (** p) sum
265267--
266268-- /Time/: \(\mathcal{O}(n*w)\) where \(w\) is the window size.
267269--
268−{-# INLINE range #-}
269−range :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe (a, a))
270−range n = Fold step initial extract
270+{-# INLINE windowRange #-}
271+windowRange :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe (a, a))
272+windowRange n = Fold step initial extract extract
271273
272274 where
273275
@@ -300,7 +302,11 @@ range n = Fold step initial extract
300302 if i == 0
301303 then return Nothing
302304 else do
303− x <- liftIO $ peek rh
305+ -- Here we use "ringStart" over "ringHead" as "ringHead" will be
306+ -- uninitialized if the ring is not full.
307+ -- Using "unsafeForeignPtrToPtr" here is safe as we touch the ring
308+ -- again in "foldFunc".
309+ x <- liftIO $ peek (unsafeForeignPtrToPtr (Ring.ringStart rb))
304310 let accum (mn, mx) a = return (min mn a, max mx a)
305311 fmap Just $ foldFunc i rh accum (x, x) rb
306312
@@ -316,9 +322,9 @@ range n = Fold step initial extract
316322--
317323-- /Time/: \(\mathcal{O}(n*w)\) where \(w\) is the window size.
318324--
319−{-# INLINE minimum #-}
320−minimum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
321−minimum n = fmap (fmap fst) $ range n
325+{-# INLINE windowMinimum #-}
326+windowMinimum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
327+windowMinimum n = fmap (fmap fst) $ windowRange n
322328
323329-- | The maximum element in a rolling window.
324330--
@@ -329,9 +335,9 @@ minimum n = fmap (fmap fst) $ range n
329335--
330336-- /Time/: \(\mathcal{O}(n*w)\) where \(w\) is the window size.
331337--
332−{-# INLINE maximum #-}
333−maximum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
334−maximum n = fmap (fmap snd) $ range n
338+{-# INLINE windowMaximum #-}
339+windowMaximum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
340+windowMaximum n = fmap (fmap snd) $ windowRange n
335341
336342-- | Arithmetic mean of elements in a sliding window:
337343--
@@ -346,6 +352,6 @@ maximum n = fmap (fmap snd) $ range n
346352-- /Space/: \(\mathcal{O}(1)\)
347353--
348354-- /Time/: \(\mathcal{O}(n)\)
349−{-# INLINE mean #-}
350−mean :: forall m a. (Monad m, Fractional a) => Fold m (a, Maybe a) a
351−mean = Fold.teeWith (/) sum length
355+{-# INLINE windowMean #-}
356+windowMean :: forall m a. (Monad m, Fractional a) => Fold m (a, Maybe a) a
357+windowMean = Fold.teeWith (/) windowSum windowLength
core/src/Streamly/Internal/Data/IORef/Unboxed.hs+23−20
@@ -23,34 +23,31 @@ module Streamly.Internal.Data.IORef.Unboxed
2323 (
2424 IORef
2525
26− -- * Construction
26+ -- Construction
2727 , newIORef
2828
29− -- * Write
29+ -- Write
3030 , writeIORef
3131 , modifyIORef'
3232
33− -- * Read
33+ -- Read
3434 , readIORef
35− , toStreamD
35+ , pollIntIORef
3636 )
3737where
3838
3939#include "inline.hs"
4040
41−import Data.Proxy (Proxy(..))
4241import Control.Monad.IO.Class (MonadIO(..))
43−import Streamly.Internal.Data.Unbox
44− ( MutableByteArray(..)
45− , Unbox(..)
46− , newBytes
47− , sizeOf
48− )
42+import Data.Proxy (Proxy(..))
43+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
44+import Streamly.Internal.Data.Unbox (Unbox(..), sizeOf)
4945
50−import qualified Streamly.Internal.Data.Stream.StreamD.Type as D
46+import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
47+import qualified Streamly.Internal.Data.Stream.Type as D
5148
5249-- | An 'IORef' holds a single 'Unbox'-able value.
53−newtype IORef a = IORef MutableByteArray
50+newtype IORef a = IORef MutByteArray
5451
5552-- | Create a new 'IORef'.
5653--
@@ -58,8 +55,8 @@ newtype IORef a = IORef MutableByteArray
5855{-# INLINE newIORef #-}
5956newIORef :: forall a. Unbox a => a -> IO (IORef a)
6057newIORef x = do
61− var <- newBytes (sizeOf (Proxy :: Proxy a))
62− pokeByteIndex 0 var x
58+ var <- MBA.new (sizeOf (Proxy :: Proxy a))
59+ pokeAt 0 var x
6360 return $ IORef var
6461
6562-- | Write a value to an 'IORef'.
@@ -67,14 +64,14 @@ newIORef x = do
6764-- /Pre-release/
6865{-# INLINE writeIORef #-}
6966writeIORef :: Unbox a => IORef a -> a -> IO ()
70−writeIORef (IORef var) = pokeByteIndex 0 var
67+writeIORef (IORef var) = pokeAt 0 var
7168
7269-- | Read a value from an 'IORef'.
7370--
7471-- /Pre-release/
7572{-# INLINE readIORef #-}
7673readIORef :: Unbox a => IORef a -> IO a
77−readIORef (IORef var) = peekByteIndex 0 var
74+readIORef (IORef var) = peekAt 0 var
7875
7976-- | Modify the value of an 'IORef' using a function with strict application.
8077--
@@ -87,10 +84,16 @@ modifyIORef' var g = do
8784
8885-- | Generate a stream by continuously reading the IORef.
8986--
87+-- This operation reads the IORef without any synchronization. It can be
88+-- assumed to be atomic because the IORef (MutableByteArray) is always aligned
89+-- to Int boundaries, we are assuming that compiler uses single instructions to
90+-- access the memory. It may read stale values though until caches are
91+-- synchronised in a multiprocessor architecture.
92+--
9093-- /Pre-release/
91−{-# INLINE_NORMAL toStreamD #-}
92−toStreamD :: (MonadIO m, Unbox a) => IORef a -> D.Stream m a
93−toStreamD var = D.Stream step ()
94+{-# INLINE_NORMAL pollIntIORef #-}
95+pollIntIORef :: (MonadIO m, Unbox a) => IORef a -> D.Stream m a
96+pollIntIORef var = D.Stream step ()
9497
9598 where
9699
core/src/Streamly/Internal/Data/IsMap.hs+8
@@ -1,3 +1,7 @@
1+{-# LANGUAGE TypeFamilies #-}
2+-- Must come after TypeFamilies, otherwise it is re-enabled.
3+-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
4+{-# LANGUAGE NoMonoLocalBinds #-}
15-- |
26-- Module : Streamly.Internal.Data.IsMap
37-- Copyright : (c) 2022 Composewell Technologies
@@ -28,6 +32,8 @@ class IsMap f where
2832 mapDelete :: Key f -> f a -> f a
2933 mapUnion :: f a -> f a -> f a
3034 mapNull :: f a -> Bool
35+ mapTraverseWithKey ::
36+ Applicative t => (Key f -> a -> t b) -> f a -> t (f b)
3137
3238instance Ord k => IsMap (Map k) where
3339 type Key (Map k) = k
@@ -39,6 +45,7 @@ instance Ord k => IsMap (Map k) where
3945 mapDelete = Map.delete
4046 mapUnion = Map.union
4147 mapNull = Map.null
48+ mapTraverseWithKey = Map.traverseWithKey
4249
4350instance IsMap IntMap.IntMap where
4451 type Key IntMap.IntMap = Int
@@ -50,3 +57,4 @@ instance IsMap IntMap.IntMap where
5057 mapDelete = IntMap.delete
5158 mapUnion = IntMap.union
5259 mapNull = IntMap.null
60+ mapTraverseWithKey = IntMap.traverseWithKey
core/src/Streamly/Internal/Data/List.hs+5−5
@@ -66,19 +66,19 @@ where
6666import Control.Arrow (second)
6767import Data.Functor.Identity (Identity, runIdentity)
6868import GHC.Exts (IsList(..), IsString(..))
69−import Streamly.Internal.Data.Stream.StreamD.Type
69+import Streamly.Internal.Data.Stream.Type
7070 ( CrossStream
7171 , mkCross
7272 , unCross
7373 )
74−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
74+import Streamly.Internal.Data.Stream.Type (Stream)
7575import Text.Read
7676 ( Lexeme(Ident), lexP, parens, prec, readPrec, readListPrec
7777 , readListPrecDefault)
7878
79−import qualified Streamly.Internal.Data.Stream.StreamK.Type as K
80−import qualified Streamly.Internal.Data.Stream.StreamD.Type as Stream
81−import qualified Streamly.Internal.Data.Stream.StreamD.Generate as Stream
79+import qualified Streamly.Internal.Data.StreamK.Type as K
80+import qualified Streamly.Internal.Data.Stream.Type as Stream
81+import qualified Streamly.Internal.Data.Stream.Generate as Stream
8282
8383-- XXX Rename to PureStream.
8484
core/src/Streamly/Internal/Data/MutArray.hs+154−33
@@ -8,79 +8,200 @@
88--
99module Streamly.Internal.Data.MutArray
1010 (
11+ -- * MutArray.Type module
1112 module Streamly.Internal.Data.MutArray.Type
12− , splitOn
13+ -- * MutArray module
14+ , sliceIndexerFromLen
15+ , slicerFromLen
16+ , compactLE
17+ , pinnedCompactLE
18+ , compactOnByte
19+ , compactOnByteSuffix
20+ -- * Unboxed IORef
21+ , module Streamly.Internal.Data.IORef.Unboxed
22+
23+ -- * Deprecated
1324 , genSlicesFromLen
1425 , getSlicesFromLen
15− , fromStream
1626 )
1727where
1828
1929#include "inline.hs"
2030
2131import Control.Monad.IO.Class (MonadIO(..))
32+import Data.Word (Word8)
33+import Streamly.Internal.Data.MutByteArray.Type (PinnedState(..))
34+import Streamly.Internal.Data.Stream.Type (Stream)
2235import Streamly.Internal.Data.Unbox (Unbox)
23−import Streamly.Internal.Data.Stream (Stream)
2436import Streamly.Internal.Data.Unfold.Type (Unfold(..))
2537
26−import qualified Streamly.Internal.Data.Stream as D
38+import qualified Streamly.Internal.Data.Stream.Type as Stream
39+import qualified Streamly.Internal.Data.Stream.Nesting as Stream
40+-- import qualified Streamly.Internal.Data.Stream.Transform as Stream
2741import qualified Streamly.Internal.Data.Unfold as Unfold
2842
2943import Prelude hiding (foldr, length, read, splitAt)
3044import Streamly.Internal.Data.MutArray.Type
31−
32−-- | Split the array into a stream of slices using a predicate. The element
33−-- matching the predicate is dropped.
34−--
35−-- /Pre-release/
36−{-# INLINE splitOn #-}
37−splitOn :: (MonadIO m, Unbox a) =>
38− (a -> Bool) -> MutArray a -> Stream m (MutArray a)
39−splitOn predicate arr =
40− fmap (\(i, len) -> getSliceUnsafe i len arr)
41− $ D.sliceOnSuffix predicate (read arr)
45+import Streamly.Internal.Data.IORef.Unboxed
4246
4347-- | Generate a stream of array slice descriptors ((index, len)) of specified
4448-- length from an array, starting from the supplied array index. The last slice
4549-- may be shorter than the requested length depending on the array length.
4650--
4751-- /Pre-release/
48−{-# INLINE genSlicesFromLen #-}
49−genSlicesFromLen :: forall m a. (Monad m, Unbox a)
52+{-# INLINE sliceIndexerFromLen #-}
53+sliceIndexerFromLen :: forall m a. (Monad m, Unbox a)
5054 => Int -- ^ from index
5155 -> Int -- ^ length of the slice
5256 -> Unfold m (MutArray a) (Int, Int)
53−genSlicesFromLen from len =
57+sliceIndexerFromLen from len =
5458 let fromThenTo n = (from, from + len, n - 1)
5559 mkSlice n i = return (i, min len (n - i))
5660 in Unfold.lmap length
5761 $ Unfold.mapM2 mkSlice
5862 $ Unfold.lmap fromThenTo Unfold.enumerateFromThenTo
5963
64+{-# DEPRECATED genSlicesFromLen "Please use sliceIndexerFromLen instead." #-}
65+genSlicesFromLen :: forall m a. (Monad m, Unbox a)
66+ => Int -- ^ from index
67+ -> Int -- ^ length of the slice
68+ -> Unfold m (MutArray a) (Int, Int)
69+genSlicesFromLen = sliceIndexerFromLen
70+
6071-- | Generate a stream of slices of specified length from an array, starting
6172-- from the supplied array index. The last slice may be shorter than the
6273-- requested length depending on the array length.
6374--
6475-- /Pre-release/
65−{-# INLINE getSlicesFromLen #-}
66−getSlicesFromLen :: forall m a. (Monad m, Unbox a)
76+{-# INLINE slicerFromLen #-}
77+slicerFromLen :: forall m a. (Monad m, Unbox a)
6778 => Int -- ^ from index
6879 -> Int -- ^ length of the slice
6980 -> Unfold m (MutArray a) (MutArray a)
70−getSlicesFromLen from len =
81+slicerFromLen from len =
7182 let mkSlice arr (i, n) = return $ getSliceUnsafe i n arr
72− in Unfold.mapM2 mkSlice (genSlicesFromLen from len)
83+ in Unfold.mapM2 mkSlice (sliceIndexerFromLen from len)
84+
85+{-# DEPRECATED getSlicesFromLen "Please use slicerFromLen instead." #-}
86+getSlicesFromLen :: forall m a. (Monad m, Unbox a)
87+ => Int -- ^ from index
88+ -> Int -- ^ length of the slice
89+ -> Unfold m (MutArray a) (MutArray a)
90+getSlicesFromLen = slicerFromLen
7391
74−-- | Create an 'Array' from a stream. This is useful when we want to create a
75−-- single array from a stream of unknown size. 'writeN' is at least twice
76−-- as efficient when the size is already known.
92+-------------------------------------------------------------------------------
93+-- Compacting Streams of Arrays
94+-------------------------------------------------------------------------------
95+
96+-- | @compactLE maxElems@ coalesces adjacent arrays in the input stream
97+-- only if the combined size would be less than or equal to @maxElems@
98+-- elements. Note that it won't split an array if the original array is already
99+-- larger than maxElems.
77100--
78−-- Note that if the input stream is too large memory allocation for the array
79−-- may fail. When the stream size is not known, `chunksOf` followed by
80−-- processing of indvidual arrays in the resulting stream should be preferred.
101+-- @maxElems@ must be greater than 0.
81102--
82−-- /Pre-release/
83−{-# INLINE fromStream #-}
84−fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
85−fromStream = fromStreamD
86−-- fromStream (Stream m) = P.fold write m
103+-- Generates unpinned arrays irrespective of the pinning status of input
104+-- arrays.
105+{-# INLINE compactLE #-}
106+compactLE :: (MonadIO m, Unbox a) =>
107+ Int -> Stream m (MutArray a) -> Stream m (MutArray a)
108+-- XXX compactLE can be moved to MutArray/Type if we are not using the parser
109+-- to implement it.
110+compactLE = compactLeAs Unpinned
111+-- The parser version turns out to be a little bit slower.
112+-- compactLE n = Stream.catRights . Stream.parseManyD (pCompactLE n)
113+
114+-- | Pinned version of 'compactLE'.
115+{-# INLINE pinnedCompactLE #-}
116+pinnedCompactLE :: forall m a. (MonadIO m, Unbox a)
117+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
118+pinnedCompactLE = compactLeAs Pinned
119+-- pinnedCompactLE n = Stream.catRights . Stream.parseManyD (pPinnedCompactLE n)
120+
121+data SplitState s arr
122+ = Initial s
123+ | Buffering s arr
124+ | Splitting s arr
125+ | Yielding arr (SplitState s arr)
126+ | Finishing
127+
128+-- | Split a stream of arrays on a given separator byte, dropping the separator
129+-- and coalescing all the arrays between two separators into a single array.
130+--
131+{-# INLINE_NORMAL _compactOnByteCustom #-}
132+_compactOnByteCustom
133+ :: MonadIO m
134+ => Word8
135+ -> Stream m (MutArray Word8)
136+ -> Stream m (MutArray Word8)
137+_compactOnByteCustom byte (Stream.Stream step state) =
138+ Stream.Stream step' (Initial state)
139+
140+ where
141+
142+ {-# INLINE_LATE step' #-}
143+ step' gst (Initial st) = do
144+ r <- step gst st
145+ case r of
146+ Stream.Yield arr s -> do
147+ (arr1, marr2) <- breakOn byte arr
148+ return $ case marr2 of
149+ Nothing -> Stream.Skip (Buffering s arr1)
150+ Just arr2 -> Stream.Skip (Yielding arr1 (Splitting s arr2))
151+ Stream.Skip s -> return $ Stream.Skip (Initial s)
152+ Stream.Stop -> return Stream.Stop
153+
154+ step' gst (Buffering st buf) = do
155+ r <- step gst st
156+ case r of
157+ Stream.Yield arr s -> do
158+ (arr1, marr2) <- breakOn byte arr
159+ -- XXX Use spliceExp instead and then rightSize?
160+ buf1 <- splice buf arr1
161+ return $ case marr2 of
162+ Nothing -> Stream.Skip (Buffering s buf1)
163+ Just x -> Stream.Skip (Yielding buf1 (Splitting s x))
164+ Stream.Skip s -> return $ Stream.Skip (Buffering s buf)
165+ Stream.Stop -> return $
166+ if byteLength buf == 0
167+ then Stream.Stop
168+ else Stream.Skip (Yielding buf Finishing)
169+
170+ step' _ (Splitting st buf) = do
171+ (arr1, marr2) <- breakOn byte buf
172+ return $ case marr2 of
173+ Nothing -> Stream.Skip $ Buffering st arr1
174+ Just arr2 -> Stream.Skip $ Yielding arr1 (Splitting st arr2)
175+
176+ step' _ (Yielding arr next) = return $ Stream.Yield arr next
177+ step' _ Finishing = return Stream.Stop
178+
179+-- XXX implement predicate based version of this
180+-- XXX Naming of predicate based vs custom version
181+
182+-- | Split a stream of arrays on a given separator byte, dropping the separator
183+-- and coalescing all the arrays between two separators into a single array.
184+--
185+{-# INLINE compactOnByte #-}
186+compactOnByte
187+ :: (MonadIO m)
188+ => Word8
189+ -> Stream m (MutArray Word8)
190+ -> Stream m (MutArray Word8)
191+-- XXX compare perf of custom vs idiomatic version
192+-- compactOnByte = _compactOnByteCustom
193+-- XXX use spliceExp and rightSize?
194+compactOnByte byte = Stream.splitInnerBy (breakOn byte) splice
195+
196+-- | Like 'compactOnByte' considers the separator in suffix position instead of
197+-- infix position.
198+{-# INLINE compactOnByteSuffix #-}
199+compactOnByteSuffix
200+ :: (MonadIO m)
201+ => Word8
202+ -> Stream m (MutArray Word8)
203+ -> Stream m (MutArray Word8)
204+compactOnByteSuffix byte =
205+ -- XXX use spliceExp and rightSize?
206+ Stream.splitInnerBySuffix
207+ (\arr -> byteLength arr == 0) (breakOn byte) splice
core/src/Streamly/Internal/Data/MutArray/Generic.hs+172−49
@@ -19,14 +19,14 @@ module Streamly.Internal.Data.MutArray.Generic
1919 , nil
2020
2121 -- *** Uninitialized Arrays
22− , new
22+ , emptyOf
2323 -- , newArrayWith
2424
2525 -- *** From streams
26− , writeNUnsafe
27− , writeN
28− , writeWith
29− , write
26+ , unsafeCreateOf
27+ , createOf
28+ , createWith -- createOfMin/createMin/createGE?
29+ , create
3030 , fromStreamN
3131 , fromStream
3232 , fromPureStream
@@ -101,6 +101,7 @@ module Streamly.Internal.Data.MutArray.Generic
101101 -- ** Random reads
102102 , getIndex
103103 , getIndexUnsafe
104+ , getIndexUnsafeWith
104105 -- , getIndices
105106 -- , getFromThenTo
106107 -- , getIndexRev
@@ -132,7 +133,7 @@ module Streamly.Internal.Data.MutArray.Generic
132133 -- multidimensional array representations.
133134
134135 -- ** Construct from streams
135− -- , chunksOf
136+ , chunksOf
136137 -- , arrayStreamKFromStreamD
137138 -- , writeChunks
138139
@@ -159,6 +160,13 @@ module Streamly.Internal.Data.MutArray.Generic
159160 -- , appendSliceFrom
160161
161162 , clone
163+
164+ -- * Deprecated
165+ , new
166+ , writeNUnsafe
167+ , writeN
168+ , writeWith
169+ , write
162170 )
163171where
164172
@@ -181,14 +189,15 @@ import GHC.Int (Int(..))
181189import Streamly.Internal.Data.Fold.Type (Fold(..))
182190import Streamly.Internal.Data.Producer.Type (Producer (..))
183191import Streamly.Internal.Data.Unfold.Type (Unfold(..))
184−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
192+import Streamly.Internal.Data.Stream.Type (Stream)
193+import Streamly.Internal.Data.SVar.Type (adaptState)
185194
186195import qualified Streamly.Internal.Data.Fold.Type as FL
187196import qualified Streamly.Internal.Data.Producer as Producer
188−import qualified Streamly.Internal.Data.Stream.StreamD.Type as D
189−import qualified Streamly.Internal.Data.Stream.StreamD.Generate as D
190−import qualified Streamly.Internal.Data.Stream.StreamD.Lift as D
191−import qualified Streamly.Internal.Data.Stream.StreamK.Type as K
197+import qualified Streamly.Internal.Data.Stream.Type as D
198+import qualified Streamly.Internal.Data.Stream.Generate as D
199+import qualified Streamly.Internal.Data.Stream.Lift as D
200+import qualified Streamly.Internal.Data.StreamK.Type as K
192201
193202import Prelude hiding (read, length)
194203
@@ -234,13 +243,13 @@ bottomElement =
234243-- XXX Would be nice if GHC can provide something like newUninitializedArray# so
235244-- that we do not have to write undefined or error in the whole array.
236245
237−-- | @new count@ allocates a zero length array that can be extended to hold
246+-- | @emptyOf count@ allocates a zero length array that can be extended to hold
238247-- up to 'count' items without reallocating.
239248--
240249-- /Pre-release/
241−{-# INLINE new #-}
242−new :: MonadIO m => Int -> m (MutArray a)
243−new n@(I# n#) =
250+{-# INLINE emptyOf #-}
251+emptyOf :: MonadIO m => Int -> m (MutArray a)
252+emptyOf n@(I# n#) =
244253 liftIO
245254 $ IO
246255 $ \s# ->
@@ -249,6 +258,12 @@ new n@(I# n#) =
249258 let ma = MutArray arr# 0 0 n
250259 in (# s1#, ma #)
251260
261+-- XXX Deprecate in major
262+-- {-# DEPRECATED new "Please use emptyOf instead." #-}
263+{-# INLINE new #-}
264+new :: MonadIO m => Int -> m (MutArray a)
265+new = emptyOf
266+
252267-- XXX This could be pure?
253268
254269-- |
@@ -263,6 +278,21 @@ nil = new 0
263278-- Random writes
264279-------------------------------------------------------------------------------
265280
281+-- | Write the given element to the given index of the 'MutableArray#'. Does not
282+-- check if the index is out of bounds of the array.
283+--
284+-- /Pre-release/
285+{-# INLINE putIndexUnsafeWith #-}
286+putIndexUnsafeWith :: MonadIO m => Int -> MutableArray# RealWorld a -> a -> m ()
287+putIndexUnsafeWith n _arrContents# x =
288+ liftIO
289+ $ IO
290+ $ \s# ->
291+ case n of
292+ I# n# ->
293+ let s1# = writeArray# _arrContents# n# x s#
294+ in (# s1#, () #)
295+
266296-- | Write the given element to the given index of the array. Does not check if
267297-- the index is out of bounds of the array.
268298--
@@ -271,13 +301,7 @@ nil = new 0
271301putIndexUnsafe :: forall m a. MonadIO m => Int -> MutArray a -> a -> m ()
272302putIndexUnsafe i MutArray {..} x =
273303 assert (i >= 0 && i < arrLen)
274− (liftIO
275− $ IO
276− $ \s# ->
277− case i + arrStart of
278− I# n# ->
279− let s1# = writeArray# arrContents# n# x s#
280− in (# s1#, () #))
304+ putIndexUnsafeWith (i + arrStart) arrContents# x
281305
282306invalidIndex :: String -> Int -> a
283307invalidIndex label i =
@@ -307,7 +331,7 @@ putIndices arr = FL.foldlM' step (return ())
307331
308332 where
309333
310− step () (i, x) = liftIO (putIndex i arr x)
334+ step () (i, x) = putIndex i arr x
311335
312336-- | Modify a given index of an array using a modifier function without checking
313337-- the bounds.
@@ -456,17 +480,25 @@ uninit arr@MutArray{..} len =
456480-- Random reads
457481-------------------------------------------------------------------------------
458482
459−-- | Return the element at the specified index without checking the bounds.
483+-- | Return the element at the specified index without checking the bounds from
484+-- a @MutableArray# RealWorld@.
460485--
461486-- Unsafe because it does not check the bounds of the array.
462−{-# INLINE_NORMAL getIndexUnsafe #-}
463−getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
464−getIndexUnsafe n MutArray {..} =
487+{-# INLINE getIndexUnsafeWith #-}
488+getIndexUnsafeWith :: MonadIO m => MutableArray# RealWorld a -> Int -> m a
489+getIndexUnsafeWith _arrContents# n =
465490 liftIO
466491 $ IO
467492 $ \s# ->
468− let !(I# i#) = arrStart + n
469− in readArray# arrContents# i# s#
493+ let !(I# i#) = n
494+ in readArray# _arrContents# i# s#
495+
496+-- | Return the element at the specified index without checking the bounds.
497+--
498+-- Unsafe because it does not check the bounds of the array.
499+{-# INLINE_NORMAL getIndexUnsafe #-}
500+getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
501+getIndexUnsafe n MutArray {..} = getIndexUnsafeWith arrContents# (n + arrStart)
470502
471503-- | /O(1)/ Lookup the element at the given index. Index starts from 0.
472504--
@@ -570,16 +602,16 @@ readRev arr@MutArray{..} =
570602arrayChunkSize :: Int
571603arrayChunkSize = 1024
572604
573−-- | Like 'writeN' but does not check the array bounds when writing. The fold
605+-- | Like 'createOf' but does not check the array bounds when writing. The fold
574606-- driver must not call the step function more than 'n' times otherwise it will
575607-- corrupt the memory and crash. This function exists mainly because any
576608-- conditional in the step function blocks fusion causing 10x performance
577609-- slowdown.
578610--
579611-- /Pre-release/
580−{-# INLINE_NORMAL writeNUnsafe #-}
581−writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
582−writeNUnsafe n = Fold step initial return
612+{-# INLINE_NORMAL unsafeCreateOf #-}
613+unsafeCreateOf :: MonadIO m => Int -> Fold m a (MutArray a)
614+unsafeCreateOf n = Fold step initial return return
583615
584616 where
585617
@@ -587,59 +619,81 @@ writeNUnsafe n = Fold step initial return
587619
588620 step arr x = FL.Partial <$> snocUnsafe arr x
589621
590−-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
622+{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
623+{-# INLINE writeNUnsafe #-}
624+writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
625+writeNUnsafe = unsafeCreateOf
626+
627+-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
591628-- 'Array'.
592629--
593−-- >>> writeN n = Fold.take n (MutArray.writeNUnsafe n)
630+-- >>> createOf n = Fold.take n (MutArray.unsafeCreateOf n)
594631--
595632-- /Pre-release/
596−{-# INLINE_NORMAL writeN #-}
633+{-# INLINE_NORMAL createOf #-}
634+createOf :: MonadIO m => Int -> Fold m a (MutArray a)
635+createOf n = FL.take n $ unsafeCreateOf n
636+
637+-- XXX Deprecate in major
638+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
639+{-# INLINE writeN #-}
597640writeN :: MonadIO m => Int -> Fold m a (MutArray a)
598−writeN n = FL.take n $ writeNUnsafe n
641+writeN = createOf
599642
600643-- >>> f n = MutArray.writeAppendWith (* 2) (MutArray.pinnedNew n)
601644-- >>> writeWith n = Fold.rmapM MutArray.rightSize (f n)
602645-- >>> writeWith n = Fold.rmapM MutArray.fromArrayStreamK (MutArray.writeChunks n)
603646
604−-- | @writeWith minCount@ folds the whole input to a single array. The array
647+-- | @createWith minCount@ folds the whole input to a single array. The array
605648-- starts at a size big enough to hold minCount elements, the size is doubled
606649-- every time the array needs to be grown.
607650--
608651-- /Caution! Do not use this on infinite streams./
609652--
610653-- /Pre-release/
611−{-# INLINE_NORMAL writeWith #-}
612−writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
654+{-# INLINE_NORMAL createWith #-}
655+createWith :: MonadIO m => Int -> Fold m a (MutArray a)
613656-- writeWith n = FL.rmapM rightSize $ writeAppendWith (* 2) (pinnedNew n)
614−writeWith elemCount = FL.rmapM extract $ FL.foldlM' step initial
657+createWith elemCount = FL.rmapM extract $ FL.foldlM' step initial
615658
616659 where
617660
618661 initial = do
619− when (elemCount < 0) $ error "writeWith: elemCount is negative"
620− liftIO $ new elemCount
662+ when (elemCount < 0) $ error "createWith: elemCount is negative"
663+ new elemCount
621664
622665 step arr@(MutArray _ start end bound) x
623666 | end == bound = do
624667 let oldSize = end - start
625668 newSize = max (oldSize * 2) 1
626− arr1 <- liftIO $ realloc newSize arr
669+ arr1 <- realloc newSize arr
627670 snocUnsafe arr1 x
628671 step arr x = snocUnsafe arr x
629672
630− -- extract = liftIO . rightSize
673+ -- extract = rightSize
631674 extract = return
632675
676+{-# DEPRECATED writeWith "Please use createWith instead." #-}
677+{-# INLINE writeWith #-}
678+writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
679+writeWith = createWith
680+
633681-- | Fold the whole input to a single array.
634682--
635−-- Same as 'writeWith' using an initial array size of 'arrayChunkSize' bytes
683+-- Same as 'createWith' using an initial array size of 'arrayChunkSize' bytes
636684-- rounded up to the element size.
637685--
638686-- /Caution! Do not use this on infinite streams./
639687--
688+{-# INLINE create #-}
689+create :: MonadIO m => Fold m a (MutArray a)
690+create = writeWith arrayChunkSize
691+
692+-- XXX Deprecate in major
693+-- {-# DEPRECATED write "Please use create instead." #-}
640694{-# INLINE write #-}
641695write :: MonadIO m => Fold m a (MutArray a)
642−write = writeWith arrayChunkSize
696+write = create
643697
644698-- | Create a 'MutArray' from the first @n@ elements of a stream. The
645699-- array is allocated to size @n@, if the stream terminates before @n@
@@ -664,7 +718,76 @@ fromList xs = fromStream $ D.fromList xs
664718{-# INLINABLE fromPureStream #-}
665719fromPureStream :: MonadIO m => Stream Identity a -> m (MutArray a)
666720fromPureStream xs =
667− liftIO $ D.fold write $ D.morphInner (return . runIdentity) xs
721+ D.fold write $ D.morphInner (return . runIdentity) xs
722+
723+-------------------------------------------------------------------------------
724+-- Chunking
725+-------------------------------------------------------------------------------
726+
727+data GroupState s a start end bound
728+ = GroupStart s
729+ | GroupBuffer s (MutableArray# RealWorld a) start end bound
730+ | GroupYield
731+ (MutableArray# RealWorld a)
732+ start
733+ end
734+ bound
735+ (GroupState s a start end bound)
736+ | GroupFinish
737+
738+-- | @chunksOf n stream@ groups the input stream into a stream of
739+-- arrays of size n.
740+--
741+-- @chunksOf n = foldMany (MutArray.writeN n)@
742+--
743+-- /Pre-release/
744+{-# INLINE_NORMAL chunksOf #-}
745+chunksOf :: forall m a. MonadIO m
746+ => Int -> D.Stream m a -> D.Stream m (MutArray a)
747+-- XXX the idiomatic implementation leads to large regression in the D.reverse'
748+-- benchmark. It seems it has difficulty producing optimized code when
749+-- converting to StreamK. Investigate GHC optimizations.
750+-- chunksOf n = D.foldMany (writeN n)
751+chunksOf n (D.Stream step state) =
752+ D.Stream step' (GroupStart state)
753+
754+ where
755+
756+ -- start is always 0
757+ -- end and len are always equal
758+
759+ {-# INLINE_LATE step' #-}
760+ step' _ (GroupStart st) = do
761+ when (n <= 0) $
762+ -- XXX we can pass the module string from the higher level API
763+ error $ "Streamly.Internal.Data.Array.Generic.Mut.Type.chunksOf: "
764+ ++ "the size of arrays [" ++ show n
765+ ++ "] must be a natural number"
766+ (MutArray contents start end bound :: MutArray a) <- new n
767+ return $ D.Skip (GroupBuffer st contents start end bound)
768+
769+ step' gst (GroupBuffer st contents start end bound) = do
770+ r <- step (adaptState gst) st
771+ case r of
772+ D.Yield x s -> do
773+ putIndexUnsafeWith end contents x
774+ let end1 = end + 1
775+ return $
776+ if end1 >= bound
777+ then D.Skip
778+ (GroupYield
779+ contents start end1 bound (GroupStart s))
780+ else D.Skip (GroupBuffer s contents start end1 bound)
781+ D.Skip s ->
782+ return $ D.Skip (GroupBuffer s contents start end bound)
783+ D.Stop ->
784+ return
785+ $ D.Skip (GroupYield contents start end bound GroupFinish)
786+
787+ step' _ (GroupYield contents start end bound next) =
788+ return $ D.Yield (MutArray contents start end bound) next
789+
790+ step' _ GroupFinish = return D.Stop
668791
669792-------------------------------------------------------------------------------
670793-- Unfolds
@@ -727,7 +850,7 @@ putSliceUnsafe src srcStart dst dstStart len = liftIO $ do
727850
728851{-# INLINE clone #-}
729852clone :: MonadIO m => MutArray a -> m (MutArray a)
730−clone src = liftIO $ do
853+clone src = do
731854 let len = arrLen src
732855 dst <- new len
733856 putSliceUnsafe src 0 dst 0 len
core/src/Streamly/Internal/Data/MutArray/Stream.hsnew file+145
@@ -0,0 +1,145 @@
1+-- |
2+-- Module : Streamly.Internal.Data.MutArray.Stream
3+-- Copyright : (c) 2019 Composewell Technologies
4+-- License : BSD3-3-Clause
5+-- Maintainer : streamly@composewell.com
6+-- Stability : experimental
7+-- Portability : GHC
8+--
9+-- Combinators to efficiently manipulate streams of mutable arrays.
10+--
11+-- We can either push these in the MutArray module with a "chunks" prefix or
12+-- keep this as a separate module and release it.
13+--
14+module Streamly.Internal.Data.MutArray.Stream
15+{-# DEPRECATED "Please use \"Streamly.Internal.Data.MutArray\" instead." #-}
16+ (
17+ -- * Generation
18+ MArray.chunksOf
19+ , MArray.pinnedChunksOf
20+ , MArray.writeChunks -- chunksWrite?
21+ , MArray.splitOn -- chunksSplitOn
22+
23+ -- * Compaction
24+ , packArraysChunksOf
25+ , MArray.SpliceState (..)
26+ , lpackArraysChunksOf
27+ , compact -- chunksCompact
28+ , compactLE
29+ , compactEQ
30+ , compactGE
31+
32+ -- * Elimination
33+ , MArray.flattenArrays -- chunksConcat
34+ , MArray.flattenArraysRev -- chunksConcatRev
35+ , MArray.fromArrayStreamK -- chunksCoalesce
36+ )
37+where
38+
39+import Control.Monad.IO.Class (MonadIO(..))
40+import Streamly.Internal.Data.Unbox (Unbox)
41+import Streamly.Internal.Data.MutArray.Type (MutArray(..))
42+import Streamly.Internal.Data.Fold.Type (Fold(..))
43+import Streamly.Internal.Data.Parser (ParseError)
44+import Streamly.Internal.Data.Stream.Type (Stream)
45+
46+import qualified Streamly.Internal.Data.MutArray as MArray
47+import qualified Streamly.Internal.Data.Fold.Type as FL
48+import qualified Streamly.Internal.Data.Parser as ParserD
49+import qualified Streamly.Internal.Data.Stream as D
50+
51+-------------------------------------------------------------------------------
52+-- Compact
53+-------------------------------------------------------------------------------
54+
55+-- XXX This can be removed once compactLEFold/compactLE are implemented.
56+--
57+-- | This mutates the first array (if it has space) to append values from the
58+-- second one. This would work for immutable arrays as well because an
59+-- immutable array never has space so a new array is allocated instead of
60+-- mutating it.
61+--
62+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
63+-- maximum specified size. Note that if a single array is bigger than the
64+-- specified size we do not split it to fit. When we coalesce multiple arrays
65+-- if the size would exceed the specified size we do not coalesce therefore the
66+-- actual array size may be less than the specified chunk size.
67+--
68+-- @since 0.7.0
69+{-# INLINE packArraysChunksOf #-}
70+packArraysChunksOf :: (MonadIO m, Unbox a)
71+ => Int -> D.Stream m (MutArray a) -> D.Stream m (MutArray a)
72+packArraysChunksOf = MArray.compactLE
73+
74+-- XXX Remove this once compactLEFold is implemented
75+-- lpackArraysChunksOf = Fold.many compactLEFold
76+--
77+{-# INLINE lpackArraysChunksOf #-}
78+lpackArraysChunksOf :: (MonadIO m, Unbox a)
79+ => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
80+lpackArraysChunksOf = MArray.lCompactGE
81+
82+-- XXX Same as compactLE, to be removed once that is implemented.
83+--
84+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
85+-- maximum specified size in bytes.
86+--
87+-- /Internal/
88+{-# INLINE compact #-}
89+compact :: (MonadIO m, Unbox a)
90+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
91+compact = packArraysChunksOf
92+
93+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
94+-- maximum specified size. Note that if a single array is bigger than the
95+-- specified size we do not split it to fit. When we coalesce multiple arrays
96+-- if the size would exceed the specified size we do not coalesce therefore the
97+-- actual array size may be less than the specified chunk size.
98+--
99+-- /Internal/
100+{-# INLINE compactLEParserD #-}
101+compactLEParserD ::
102+ forall m a. (MonadIO m, Unbox a)
103+ => Int -> ParserD.Parser (MutArray a) m (MutArray a)
104+compactLEParserD = MArray.pCompactLE
105+
106+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
107+-- minimum specified size. Note that if all the arrays in the stream together
108+-- are smaller than the specified size the resulting array will be smaller than
109+-- the specified size. When we coalesce multiple arrays if the size would exceed
110+-- the specified size we stop coalescing further.
111+--
112+-- /Internal/
113+{-# INLINE compactGEFold #-}
114+compactGEFold ::
115+ forall m a. (MonadIO m, Unbox a)
116+ => Int -> FL.Fold m (MutArray a) (MutArray a)
117+compactGEFold = MArray.fCompactGE
118+
119+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
120+-- maximum specified size in bytes.
121+--
122+-- /Internal/
123+compactLE :: (MonadIO m, Unbox a) =>
124+ Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
125+compactLE n = D.parseManyD (compactLEParserD n)
126+
127+-- | Like 'compactLE' but generates arrays of exactly equal to the size
128+-- specified except for the last array in the stream which could be shorter.
129+--
130+-- /Unimplemented/
131+{-# INLINE compactEQ #-}
132+compactEQ :: -- (MonadIO m, Unbox a) =>
133+ Int -> Stream m (MutArray a) -> Stream m (MutArray a)
134+compactEQ _n _xs = undefined
135+ -- IsStream.fromStreamD $ D.foldMany (compactEQFold n) (IsStream.toStreamD xs)
136+
137+-- | Like 'compactLE' but generates arrays of size greater than or equal to the
138+-- specified except for the last array in the stream which could be shorter.
139+--
140+-- /Internal/
141+{-# INLINE compactGE #-}
142+compactGE ::
143+ (MonadIO m, Unbox a)
144+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
145+compactGE n = D.foldMany (compactGEFold n)
core/src/Streamly/Internal/Data/MutArray/Type.hs+1308−441

This diff is too big to show here.

core/src/Streamly/Internal/Data/MutByteArray.hsnew file+251

This diff is too big to show here.

core/src/Streamly/Internal/Data/MutByteArray/Type.hsnew file+321

This diff is too big to show here.

core/src/Streamly/Internal/Data/Parser.hs+135−143

This diff is too big to show here.

core/src/Streamly/Internal/Data/Parser/ParserDK.hsdeleted−1544

This diff is too big to show here.

core/src/Streamly/Internal/Data/Parser/ParserD/Tee.hs → core/src/Streamly/Internal/Data/Parser/Tee.hsrenamed+1−1

This diff is too big to show here.

core/src/Streamly/Internal/Data/Parser/ParserD/Type.hs → core/src/Streamly/Internal/Data/Parser/Type.hsrenamed+94−71

This diff is too big to show here.

core/src/Streamly/Internal/Data/ParserK.hsnew file+16

This diff is too big to show here.

core/src/Streamly/Internal/Data/Parser/ParserK/Type.hs → core/src/Streamly/Internal/Data/ParserK/Type.hsrenamed+291−84

This diff is too big to show here.

core/src/Streamly/Internal/Data/Pipe.hs+5−8

This diff is too big to show here.

core/src/Streamly/Internal/Data/Producer.hs+6−13

This diff is too big to show here.

core/src/Streamly/Internal/Data/Producer/Source.hs+1−1

This diff is too big to show here.

core/src/Streamly/Internal/Data/Producer/Type.hs+1−1

This diff is too big to show here.

core/src/Streamly/Internal/Data/Refold/Type.hs+1−1

This diff is too big to show here.

core/src/Streamly/Internal/Data/Ring.hs+589−134

This diff is too big to show here.

core/src/Streamly/Internal/Data/Ring/Generic.hsnew file+185

This diff is too big to show here.

core/src/Streamly/Internal/Data/Ring/Unboxed.hsdeleted−616

This diff is too big to show here.

core/src/Streamly/Internal/Data/Serialize.hsdeleted−157

This diff is too big to show here.

core/src/Streamly/Internal/Data/Serialize/TH.hsnew file+530

This diff is too big to show here.

core/src/Streamly/Internal/Data/Serialize/TH/Bottom.hsnew file+475

This diff is too big to show here.

core/src/Streamly/Internal/Data/Serialize/TH/Common.hsnew file+70

This diff is too big to show here.

core/src/Streamly/Internal/Data/Serialize/TH/RecHeader.hsnew file+411

This diff is too big to show here.

core/src/Streamly/Internal/Data/Serialize/Type.hsnew file+317

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream.hs+20−20

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Container.hs → core/src/Streamly/Internal/Data/Stream/Container.hsrenamed+65−57

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Eliminate.hs → core/src/Streamly/Internal/Data/Stream/Eliminate.hsrenamed+39−33

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Exception.hs → core/src/Streamly/Internal/Data/Stream/Exception.hsrenamed+12−9

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Generate.hs → core/src/Streamly/Internal/Data/Stream/Generate.hsrenamed+27−26

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Lift.hs → core/src/Streamly/Internal/Data/Stream/Lift.hsrenamed+3−3

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/MutChunked.hsdeleted−323

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Nesting.hs → core/src/Streamly/Internal/Data/Stream/Nesting.hsrenamed+115−228

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Step.hs → core/src/Streamly/Internal/Data/Stream/Step.hsrenamed+2−2

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Top.hsdeleted−353

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/Top.hsnew file+480

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Transform.hs → core/src/Streamly/Internal/Data/Stream/Transform.hsrenamed+66−40

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Transformer.hs → core/src/Streamly/Internal/Data/Stream/Transformer.hsrenamed+5−4

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Type.hs → core/src/Streamly/Internal/Data/Stream/Type.hsrenamed+169−34

This diff is too big to show here.

core/src/Streamly/Internal/Data/StreamK.hs+350−134

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamK/Alt.hs → core/src/Streamly/Internal/Data/StreamK/Alt.hsrenamed+1−1

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamK/Transformer.hs → core/src/Streamly/Internal/Data/StreamK/Transformer.hsrenamed+3−3

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamK/Type.hs → core/src/Streamly/Internal/Data/StreamK/Type.hsrenamed+95−41

This diff is too big to show here.

core/src/Streamly/Internal/Data/Time/Clock.hs+3−3

This diff is too big to show here.

core/src/Streamly/Internal/Data/Time/Units.hs−3

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unbox.hs+295−280

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unbox/TH.hs+85−82

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unfold.hs+20−58

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unfold/Enumeration.hs−1

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unfold/Type.hs+4−8

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/File.hs+11−13

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/Handle.hs+33−42

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/Path.hsnew file+515

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/Array.hs+6

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/Parser.hs+12−4

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/Stream.hs+53−25

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/String.hs+17−12

This diff is too big to show here.

core/src/Streamly/Unicode/Parser.hs+11−2

This diff is too big to show here.

core/src/Streamly/Unicode/Stream.hs+9

This diff is too big to show here.

core/src/Streamly/Unicode/String.hs+11−1

This diff is too big to show here.

core/streamly-core.cabal+146−78

This diff is too big to show here.

docs/Developer/optimization-guidelines.md+12

This diff is too big to show here.

docs/Developer/paths.rst+25−2

This diff is too big to show here.

docs/User/HowTo/Compiling.md+5−3

This diff is too big to show here.

docs/User/Project/ApiChangelogs/0.9.0-0.10.0.txtnew file+424

This diff is too big to show here.

docs/User/Project/CONTRIBUTORS.md+8

This diff is too big to show here.

docs/User/Project/Changelog.md+25−10

This diff is too big to show here.

docs/User/Project/README.md+9−2

This diff is too big to show here.

docs/User/Project/Upgrading.md → docs/User/Project/Upgrading-0.8-to-0.9.mdrenamed+87−9

This diff is too big to show here.

docs/User/Tutorials/quick-overview.md+10−14

This diff is too big to show here.

docs/streamly-docs.cabal+1−1

This diff is too big to show here.

hie.yaml+2−2

This diff is too big to show here.

src/Streamly.hsdeleted−624

This diff is too big to show here.

src/Streamly/Data/Fold/Prelude.hs+32−14

This diff is too big to show here.

src/Streamly/Data/Fold/Tee.hs+1−1

This diff is too big to show here.

src/Streamly/Data/Stream/Concurrent.hsdeleted−194

This diff is too big to show here.

src/Streamly/Data/Stream/Exception.hsdeleted−24

This diff is too big to show here.

src/Streamly/Data/Stream/MkType.hs+41−10

This diff is too big to show here.

src/Streamly/Data/Stream/Prelude.hs+217−15

This diff is too big to show here.

src/Streamly/Data/Stream/Time.hsdeleted−37

This diff is too big to show here.

src/Streamly/Data/Unicode/Stream.hsdeleted−99

This diff is too big to show here.

src/Streamly/Internal/Data/Array/ArrayMacros.hdeleted−40

This diff is too big to show here.

src/Streamly/Internal/Data/Channel.hsnew file+19

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Channel/Dispatcher.hs → src/Streamly/Internal/Data/Channel/Dispatcher.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Channel/Types.hs → src/Streamly/Internal/Data/Channel/Types.hsrenamed+12−37

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Channel/Worker.hs → src/Streamly/Internal/Data/Channel/Worker.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Async.hs+6−3

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent.hs+1−44

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent/Channel.hs+29−10

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent/Channel/Type.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Prelude.hsnew file+19

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/SVar.hs+8−7

This diff is too big to show here.

src/Streamly/Internal/Data/IsMap/HashMap.hs+6

This diff is too big to show here.

src/Streamly/Internal/Data/SVar.hs+1

This diff is too big to show here.

src/Streamly/Internal/Data/SmallArray.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Ahead.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Async.hs+2−2

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/Common.hs → src/Streamly/Internal/Data/Stream/Common.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent.hs+51−42

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel.hs+17−14

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Append.hs+11−11

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Consumer.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Dispatcher.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Interleave.hs+9−9

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Operations.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Type.hs+4−4

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream.hs+6

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Combinators.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Common.hs+6−6

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Exception.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Expand.hs+7−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Generate.hs+6−6

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Reduce.hs+7−7

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Top.hs+2−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Transform.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Type.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Exception/Lifted.hs → src/Streamly/Internal/Data/Stream/Lifted.hsrenamed+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/MkType.hs+33−34

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Parallel.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Prelude.hsnew file+19

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/SVar.hsnew file+18

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/SVar/Eliminate.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/SVar/Generate.hs+3−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Serial.hs+7−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Time.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Zip.hs+41−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Zip/Concurrent.hsdeleted−50

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/ZipAsync.hs+3−2

This diff is too big to show here.

src/Streamly/Internal/Data/Unfold/Exception.hs+1−3

This diff is too big to show here.

src/Streamly/Internal/Data/Unfold/Prelude.hsnew file+17

This diff is too big to show here.

src/Streamly/Internal/Data/Unfold/SVar.hs+2−1

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event.hs+5−2

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Darwin.hs+5−5

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Linux.hs+15−9

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Windows.hs+11−8

This diff is too big to show here.

src/Streamly/Internal/FileSystem/FD.hs+11−10

This diff is too big to show here.

src/Streamly/Internal/Network/Inet/TCP.hs+82−47

This diff is too big to show here.

src/Streamly/Internal/Network/Socket.hs+33−30

This diff is too big to show here.

src/Streamly/Internal/System/IOVec.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Unicode/Char.hs+9−7

This diff is too big to show here.

src/Streamly/Memory/Array.hsdeleted−74

This diff is too big to show here.

src/Streamly/Network/Inet/TCP.hs+41−30

This diff is too big to show here.

src/Streamly/Network/Socket.hs+16−30

This diff is too big to show here.

streamly.cabal+86−64

This diff is too big to show here.

targets/Targets.hs+9−3

This diff is too big to show here.

test/Streamly/Test/Data/Array.hs+7−8

This diff is too big to show here.

test/Streamly/Test/Data/Array/Stream.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Fold.hs+7−7

This diff is too big to show here.

test/Streamly/Test/Data/Fold/Window.hs+8−9

This diff is too big to show here.

test/Streamly/Test/Data/MutArray.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Parser.hs+18−13

This diff is too big to show here.

test/Streamly/Test/Data/ParserK.hs+13−9

This diff is too big to show here.

test/Streamly/Test/Data/Ring/Unboxed.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Data/Serialize.hs+251−13

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/CompatV0.hsnew file+46

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/CompatV1.hsnew file+48

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/TH.hsnew file+82

This diff is too big to show here.

test/Streamly/Test/Data/Stream/Concurrent.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Unbox.hs+125−26

This diff is too big to show here.

test/Streamly/Test/Data/Unbox/TH.hs+11−10

This diff is too big to show here.

test/Streamly/Test/FileSystem/Event/Common.hs+1−1

This diff is too big to show here.

test/Streamly/Test/FileSystem/Handle.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Prelude/Serial.hs+7−3

This diff is too big to show here.

test/Streamly/Test/Serialize/Serializable.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Unicode/Char.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Unicode/Parser.hs+6−16

This diff is too big to show here.

test/Streamly/Test/Unicode/Stream.hs+1−1

This diff is too big to show here.

test/lib/Streamly/Test/Prelude/Common.hs+9−2

This diff is too big to show here.

test/streamly-tests.cabal+38−11

This diff is too big to show here.