git.iko.soy/streamly
on master help sign in
swap
30c5a56331 ahead, 0 behind
266 files changed+15266−8615
since splitsnapshots
unifiedsplit
.circleci/config.yml+73−277 .github/workflows/haskell.yml+84−91 .github/workflows/regression-check.yml+2−2 .hlint.ignore+5 benchmark/NanoBenchmarks.hs+1−1 benchmark/Streamly/Benchmark/Data/Array/CommonImports.hs+1−1 benchmark/Streamly/Benchmark/Data/Array/Stream.hs+3−3 benchmark/Streamly/Benchmark/Data/Fold.hs+28−4 benchmark/Streamly/Benchmark/Data/Fold/Window.hs+39−39 benchmark/Streamly/Benchmark/Data/MutArray.hs+1−1 benchmark/Streamly/Benchmark/Data/Parser.hs+10−10 benchmark/Streamly/Benchmark/Data/ParserK.hs+89−36 benchmark/Streamly/Benchmark/Data/Ring/Unboxed.hs+2−3 benchmark/Streamly/Benchmark/Data/Serialize.hs+591 benchmark/Streamly/Benchmark/Data/Serialize/RecCompatible.hs+24 benchmark/Streamly/Benchmark/Data/Serialize/RecNonCompatible.hs+21 benchmark/Streamly/Benchmark/Data/Serialize/TH.hs+76 benchmark/Streamly/Benchmark/Data/Stream/Common.hs+3−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentCommon.hs+3−3 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentEager.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentInterleaved.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentOrdered.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Eliminate.hs+3−1 benchmark/Streamly/Benchmark/Data/Stream/Exceptions.hs+12−3 benchmark/Streamly/Benchmark/Data/Stream/Expand.hs+9−1 benchmark/Streamly/Benchmark/Data/Stream/Generate.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Lift.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Reduce.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/Split.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/StreamD.hs+4−2 benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs+4−3 benchmark/Streamly/Benchmark/Data/Stream/StreamKAlt.hs+3−3 benchmark/Streamly/Benchmark/Data/Stream/ToStreamK.hs+3−4 benchmark/Streamly/Benchmark/Data/Stream/Transform.hs+2−2 benchmark/Streamly/Benchmark/Data/Unbox.hs−132 benchmark/Streamly/Benchmark/Data/Unfold.hs+11−2 benchmark/Streamly/Benchmark/FileSystem/Handle.hs+1−1 benchmark/Streamly/Benchmark/FileSystem/Handle/Read.hs+8−9 benchmark/Streamly/Benchmark/FileSystem/Handle/ReadWrite.hs+9−10 benchmark/Streamly/Benchmark/Prelude/Adaptive.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Ahead.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Async.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Concurrent.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Merge.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Parallel.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Rate.hs+1−1 benchmark/Streamly/Benchmark/Prelude/WAsync.hs+1−1 benchmark/Streamly/Benchmark/Prelude/WSerial.hs+1−1 benchmark/Streamly/Benchmark/Prelude/ZipAsync.hs+1−1 benchmark/Streamly/Benchmark/Prelude/ZipSerial.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Char.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Parser.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Stream.hs+25−7 benchmark/Streamly/Benchmark/Unicode/Utf8.hs+1−1 benchmark/bench-runner/Main.hs+34−6 benchmark/lib/Streamly/Benchmark/Common.hs+1−1 benchmark/lib/Streamly/Benchmark/Common/Handle.hs+3−1 benchmark/lib/Streamly/Benchmark/Prelude.hs+1−1 benchmark/streamly-benchmarks.cabal+58−41 configure+9−9 configure.ac+1−1 core/configure+9−9 core/configure.ac+1−1 core/docs/ApiChangelogs/0.1.0-0.2.0.txt+1471 core/docs/ApiChangelogs/0.2.0-0.2.2.txt+306 core/docs/Changelog.md+60−12 core/src/DocTestDataArray.hs+8 core/src/DocTestDataFold.hs−1 core/src/DocTestDataParserK.hs+15 core/src/DocTestDataStream.hs+2−1 core/src/DocTestFileSystemHandle.hs+1−1 core/src/Streamly/Console/Stdio.hs+1−27 core/src/Streamly/Data/Array.hs+48−48 core/src/Streamly/Data/Array/Generic.hs+16−13 core/src/Streamly/Data/Fold.hs+150−118 core/src/Streamly/Data/MutArray.hs+18−12 core/src/Streamly/Data/MutArray/Generic.hs+8−3 core/src/Streamly/Data/MutByteArray.hs+106 core/src/Streamly/Data/Parser.hs+130−60 core/src/Streamly/Data/ParserK.hs+105−10 core/src/Streamly/Data/Stream.hs+131−46 core/src/Streamly/Data/StreamK.hs+2−1 core/src/Streamly/Data/Unbox.hs−16 core/src/Streamly/Data/Unfold.hs+3−1 core/src/Streamly/FileSystem/Handle.hs+7−30 core/src/Streamly/Internal/Data/Array.hs+609−162 core/src/Streamly/Internal/Data/Array/Generic.hs+48−10 core/src/Streamly/Internal/Data/Stream/Chunked.hs → core/src/Streamly/Internal/Data/Array/Stream.hs+74−587 core/src/Streamly/Internal/Data/Array/Type.hs+538−108 core/src/Streamly/Internal/Serialize/FromBytes.hs → core/src/Streamly/Internal/Data/Binary/Parser.hs+10−4 core/src/Streamly/Internal/Serialize/ToBytes.hs → core/src/Streamly/Internal/Data/Binary/Stream.hs+12−2 core/src/Streamly/Internal/Data/Builder.hs+11−9 core/src/Streamly/Internal/Data/Fold.hs+2−2 core/src/Streamly/Internal/Data/Fold/Chunked.hs+20−18 core/src/Streamly/Internal/Data/Fold/Combinators.hs+86−142 core/src/Streamly/Internal/Data/Fold/Container.hs+73−28 core/src/Streamly/Internal/Data/Fold/Step.hs+1−1 core/src/Streamly/Internal/Data/Fold/Tee.hs+2 core/src/Streamly/Internal/Data/Fold/Type.hs+339−115 core/src/Streamly/Internal/Data/Fold/Window.hs+57−51 core/src/Streamly/Internal/Data/IORef/Unboxed.hs+23−20 core/src/Streamly/Internal/Data/IsMap.hs+8 core/src/Streamly/Internal/Data/List.hs+5−5 core/src/Streamly/Internal/Data/MutArray.hs+154−33 core/src/Streamly/Internal/Data/MutArray/Generic.hs+172−49 core/src/Streamly/Internal/Data/MutArray/Stream.hs+145 core/src/Streamly/Internal/Data/MutArray/Type.hs+1308−441 core/src/Streamly/Internal/Data/MutByteArray.hs+251 core/src/Streamly/Internal/Data/MutByteArray/Type.hs+321 core/src/Streamly/Internal/Data/Parser.hs+135−143 core/src/Streamly/Internal/Data/Parser/ParserDK.hs−1544 core/src/Streamly/Internal/Data/Parser/ParserD/Tee.hs → core/src/Streamly/Internal/Data/Parser/Tee.hs+1−1 core/src/Streamly/Internal/Data/Parser/ParserD/Type.hs → core/src/Streamly/Internal/Data/Parser/Type.hs+94−71 core/src/Streamly/Internal/Data/ParserK.hs+16 core/src/Streamly/Internal/Data/Parser/ParserK/Type.hs → core/src/Streamly/Internal/Data/ParserK/Type.hs+288−83 core/src/Streamly/Internal/Data/Pipe.hs+4−7 core/src/Streamly/Internal/Data/Producer.hs+6−13 core/src/Streamly/Internal/Data/Producer/Source.hs+1−1 core/src/Streamly/Internal/Data/Producer/Type.hs+1−1 core/src/Streamly/Internal/Data/Ring.hs+9−6 core/src/Streamly/Internal/Data/Ring/Generic.hs+49−28 core/src/Streamly/Internal/Data/Serialize.hs−157 core/src/Streamly/Internal/Data/Serialize/TH.hs+530 core/src/Streamly/Internal/Data/Serialize/TH/Bottom.hs+475 core/src/Streamly/Internal/Data/Serialize/TH/Common.hs+70 core/src/Streamly/Internal/Data/Serialize/TH/RecHeader.hs+411 core/src/Streamly/Internal/Data/Serialize/Type.hs+317 core/src/Streamly/Internal/Data/Stream.hs+20−20 core/src/Streamly/Internal/Data/Stream/StreamD/Container.hs → core/src/Streamly/Internal/Data/Stream/Container.hs+65−57 core/src/Streamly/Internal/Data/Stream/StreamD/Eliminate.hs → core/src/Streamly/Internal/Data/Stream/Eliminate.hs+39−33 core/src/Streamly/Internal/Data/Stream/StreamD/Exception.hs → core/src/Streamly/Internal/Data/Stream/Exception.hs+12−9 core/src/Streamly/Internal/Data/Stream/StreamD/Generate.hs → core/src/Streamly/Internal/Data/Stream/Generate.hs+5−15 core/src/Streamly/Internal/Data/Stream/StreamD/Lift.hs → core/src/Streamly/Internal/Data/Stream/Lift.hs+3−3 core/src/Streamly/Internal/Data/Stream/MutChunked.hs−323 core/src/Streamly/Internal/Data/Stream/StreamD/Nesting.hs → core/src/Streamly/Internal/Data/Stream/Nesting.hs+90−206 core/src/Streamly/Internal/Data/Stream/StreamD/Step.hs → core/src/Streamly/Internal/Data/Stream/Step.hs+2−2 core/src/Streamly/Internal/Data/Stream/StreamD/Top.hs−353 core/src/Streamly/Internal/Data/Stream/Top.hs+480 core/src/Streamly/Internal/Data/Stream/StreamD/Transform.hs → core/src/Streamly/Internal/Data/Stream/Transform.hs+66−40 core/src/Streamly/Internal/Data/Stream/StreamD/Transformer.hs → core/src/Streamly/Internal/Data/Stream/Transformer.hs+5−4 core/src/Streamly/Internal/Data/Stream/StreamD/Type.hs → core/src/Streamly/Internal/Data/Stream/Type.hs+164−33 core/src/Streamly/Internal/Data/StreamK.hs+325−124 core/src/Streamly/Internal/Data/Stream/StreamK/Alt.hs → core/src/Streamly/Internal/Data/StreamK/Alt.hs+1−1 core/src/Streamly/Internal/Data/Stream/StreamK/Transformer.hs → core/src/Streamly/Internal/Data/StreamK/Transformer.hs+3−3 core/src/Streamly/Internal/Data/Stream/StreamK/Type.hs → core/src/Streamly/Internal/Data/StreamK/Type.hs+88−17 core/src/Streamly/Internal/Data/Time/Clock.hs+3−3 core/src/Streamly/Internal/Data/Time/Units.hs−3 core/src/Streamly/Internal/Data/Unbox.hs+195−260 core/src/Streamly/Internal/Data/Unbox/TH.hs+85−82 core/src/Streamly/Internal/Data/Unfold.hs+19−17 core/src/Streamly/Internal/Data/Unfold/Type.hs+1−1 core/src/Streamly/Internal/FileSystem/File.hs+11−13 core/src/Streamly/Internal/FileSystem/Handle.hs+23−30 core/src/Streamly/Internal/FileSystem/Path.hs+515 core/src/Streamly/Internal/Unicode/Parser.hs+3 core/src/Streamly/Internal/Unicode/Stream.hs+43−15 core/src/Streamly/Internal/Unicode/String.hs+7−7 core/streamly-core.cabal+135−73 docs/Developer/optimization-guidelines.md+12 docs/Developer/paths.rst+25−2 docs/User/HowTo/Compiling.md+5−3 docs/User/Project/ApiChangelogs/0.9.0-0.10.0.txt+424 docs/User/Project/CONTRIBUTORS.md+8 docs/User/Project/Changelog.md+25−10 docs/User/Project/README.md+9−2 docs/User/Project/Upgrading.md → docs/User/Project/Upgrading-0.8-to-0.9.md+87−9 docs/User/Tutorials/quick-overview.md+10−14 docs/streamly-docs.cabal+1−1 hie.yaml+2−2 src/Streamly.hs−625 src/Streamly/Data/Fold/Prelude.hs+31−13 src/Streamly/Data/Fold/Tee.hs+1−1 src/Streamly/Data/Stream/Concurrent.hs−194 src/Streamly/Data/Stream/Exception.hs−24 src/Streamly/Data/Stream/MkType.hs+41−10 src/Streamly/Data/Stream/Prelude.hs+217−15 src/Streamly/Data/Stream/Time.hs−37 src/Streamly/Data/Unicode/Stream.hs−99 src/Streamly/Internal/Data/Array/ArrayMacros.h−40 src/Streamly/Internal/Data/Channel.hs+19 src/Streamly/Internal/Data/Stream/Channel/Dispatcher.hs → src/Streamly/Internal/Data/Channel/Dispatcher.hs+3−3 src/Streamly/Internal/Data/Stream/Channel/Types.hs → src/Streamly/Internal/Data/Channel/Types.hs+12−37 src/Streamly/Internal/Data/Stream/Channel/Worker.hs → src/Streamly/Internal/Data/Channel/Worker.hs+3−3 src/Streamly/Internal/Data/Fold/Async.hs+5−2 src/Streamly/Internal/Data/Fold/Concurrent.hs+1−44 src/Streamly/Internal/Data/Fold/Concurrent/Channel.hs+29−10 src/Streamly/Internal/Data/Fold/Concurrent/Channel/Type.hs+3−3 src/Streamly/Internal/Data/Fold/Prelude.hs+19 src/Streamly/Internal/Data/Fold/SVar.hs+6−5 src/Streamly/Internal/Data/IsMap/HashMap.hs+6 src/Streamly/Internal/Data/SVar.hs+1 src/Streamly/Internal/Data/SmallArray.hs+1−1 src/Streamly/Internal/Data/Stream/Ahead.hs+2−2 src/Streamly/Internal/Data/Stream/Async.hs+2−2 core/src/Streamly/Internal/Data/Stream/Common.hs → src/Streamly/Internal/Data/Stream/Common.hs+3−3 src/Streamly/Internal/Data/Stream/Concurrent.hs+51−42 src/Streamly/Internal/Data/Stream/Concurrent/Channel.hs+17−14 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Append.hs+11−11 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Consumer.hs+2−2 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Dispatcher.hs+3−3 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Interleave.hs+9−9 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Operations.hs+2−2 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Type.hs+4−4 src/Streamly/Internal/Data/Stream/IsStream.hs+6 src/Streamly/Internal/Data/Stream/IsStream/Combinators.hs+1−1 src/Streamly/Internal/Data/Stream/IsStream/Common.hs+5−5 src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs+2−2 src/Streamly/Internal/Data/Stream/IsStream/Exception.hs+1−1 src/Streamly/Internal/Data/Stream/IsStream/Expand.hs+7−3 src/Streamly/Internal/Data/Stream/IsStream/Generate.hs+5−5 src/Streamly/Internal/Data/Stream/IsStream/Reduce.hs+6−6 src/Streamly/Internal/Data/Stream/IsStream/Top.hs+1−1 src/Streamly/Internal/Data/Stream/IsStream/Transform.hs+2−2 src/Streamly/Internal/Data/Stream/IsStream/Type.hs+1−1 src/Streamly/Internal/Data/Stream/Exception/Lifted.hs → src/Streamly/Internal/Data/Stream/Lifted.hs+2−2 src/Streamly/Internal/Data/Stream/MkType.hs+33−34 src/Streamly/Internal/Data/Stream/Parallel.hs+1−1 src/Streamly/Internal/Data/Stream/Prelude.hs+19 src/Streamly/Internal/Data/Stream/SVar.hs+18 src/Streamly/Internal/Data/Stream/SVar/Eliminate.hs+1−1 src/Streamly/Internal/Data/Stream/SVar/Generate.hs+3−1 src/Streamly/Internal/Data/Stream/Serial.hs+7−2 src/Streamly/Internal/Data/Stream/Time.hs+2−2 src/Streamly/Internal/Data/Stream/Zip.hs+41−2 src/Streamly/Internal/Data/Stream/Zip/Concurrent.hs−50 src/Streamly/Internal/Data/Stream/ZipAsync.hs+3−2 src/Streamly/Internal/Data/Unfold/Prelude.hs+17 src/Streamly/Internal/Data/Unfold/SVar.hs+1 src/Streamly/Internal/FileSystem/Event.hs+5−2 src/Streamly/Internal/FileSystem/Event/Darwin.hs+5−5 src/Streamly/Internal/FileSystem/Event/Linux.hs+15−9 src/Streamly/Internal/FileSystem/Event/Windows.hs+11−8 src/Streamly/Internal/FileSystem/FD.hs+11−10 src/Streamly/Internal/Network/Inet/TCP.hs+81−46 src/Streamly/Internal/Network/Socket.hs+32−29 src/Streamly/Internal/System/IOVec.hs+3−3 src/Streamly/Internal/Unicode/Char.hs+9−7 src/Streamly/Memory/Array.hs−74 src/Streamly/Network/Inet/TCP.hs+41−30 src/Streamly/Network/Socket.hs+16−30 streamly.cabal+84−62 targets/Targets.hs+8−3 test/Streamly/Test/Data/Array.hs+7−8 test/Streamly/Test/Data/Array/Stream.hs+1−1 test/Streamly/Test/Data/Fold.hs+7−6 test/Streamly/Test/Data/Fold/Window.hs+7−8 test/Streamly/Test/Data/MutArray.hs+1−1 test/Streamly/Test/Data/Parser.hs+18−13 test/Streamly/Test/Data/ParserK.hs+13−9 test/Streamly/Test/Data/Ring/Unboxed.hs+1−1 test/Streamly/Test/Data/Serialize.hs+251−13 test/Streamly/Test/Data/Serialize/CompatV0.hs+46 test/Streamly/Test/Data/Serialize/CompatV1.hs+48 test/Streamly/Test/Data/Serialize/TH.hs+82 test/Streamly/Test/Data/Stream/Concurrent.hs+1−1 test/Streamly/Test/Data/Unbox.hs+125−26 test/Streamly/Test/Data/Unbox/TH.hs+11−10 test/Streamly/Test/FileSystem/Event/Common.hs+1−1 test/Streamly/Test/FileSystem/Handle.hs+2−2 test/Streamly/Test/Prelude/Serial.hs+7−3 test/Streamly/Test/Serialize/Serializable.hs+2−2 test/Streamly/Test/Unicode/Char.hs+1−1 test/Streamly/Test/Unicode/Parser.hs+6−16 test/Streamly/Test/Unicode/Stream.hs+1−1 test/lib/Streamly/Test/Prelude/Common.hs+9−2 test/streamly-tests.cabal+38−11
.circleci/config.yml+73−277
@@ -1,7 +1,7 @@
11version: 2.1
22
33#-----------------------------------------------------------------------------
4−# packcheck-0.4.2
4+# packcheck-0.7.0
55# Packcheck global environment variables
66#-----------------------------------------------------------------------------
77
@@ -11,6 +11,8 @@ env: &env
1111 # Common options
1212 # ------------------------------------------------------------------------
1313 # GHC_OPTIONS: "-Werror"
14+ # For updating see: https://downloads.haskell.org/~ghcup/
15+ GHCUP_VERSION: 0.1.20.0
1416 CABAL_REINIT_CONFIG: "y"
1517 LC_ALL: "C.UTF-8"
1618
@@ -22,7 +24,6 @@ env: &env
2224 # DISABLE_DOCS: "y"
2325 # DISABLE_SDIST_BUILD: "y"
2426 # DISABLE_DIST_CHECKS: "y"
25− ENABLE_INSTALL: "y"
2627
2728 # ------------------------------------------------------------------------
2829 # stack options
@@ -38,15 +39,12 @@ env: &env
3839 # cabal options
3940 # ------------------------------------------------------------------------
4041 CABAL_CHECK_RELAX: "y"
41− CABAL_NO_SANDBOX: "y"
42− CABAL_HACKAGE_MIRROR: "hackage.haskell.org:http://hackage.fpcomplete.com"
4342 CABAL_BUILD_OPTIONS: "-j1 --flag limit-build-mem"
4443
4544 # ------------------------------------------------------------------------
4645 # Where to find the required tools
4746 # ------------------------------------------------------------------------
48− PATH: /opt/ghc/bin:/opt/ghcjs/bin:/sbin:/usr/sbin:/bin:/usr/bin
49− TOOLS_DIR: /opt
47+ PATH: /sbin:/usr/sbin:/bin:/usr/bin
5048
5149 # ------------------------------------------------------------------------
5250 # Location of packcheck.sh (the shell script invoked to perform CI tests ).
@@ -59,15 +57,15 @@ env: &env
5957 # If you have not committed packcheck.sh in your repo at PACKCHECK
6058 # then it is automatically pulled from this URL.
6159 PACKCHECK_GITHUB_URL: "https://raw.githubusercontent.com/composewell/packcheck"
62− PACKCHECK_GITHUB_COMMIT: "a302772c8f3791c477edd1764b82f8d1f5b22fdf"
60+ PACKCHECK_GITHUB_COMMIT: "f6cfd7b7a33b4bf6fd63cce6a3fb3d26f5031403"
6361
6462executors:
6563 amd64-executor:
6664 docker:
67− - image: ubuntu:xenial
65+ - image: ubuntu:latest
6866 x86-executor:
6967 docker:
70− - image: i386/ubuntu:eoan
68+ - image: i386/ubuntu:focal
7169
7270#-----------------------------------------------------------------------------
7371# Common utility stuff, not to be modified usually
@@ -79,38 +77,50 @@ preinstall: &preinstall
7977 # required for https/cache save and restore
8078 apt-get install -y ca-certificates
8179
82− # required to (re)generate the configure script
83− apt-get install -y autoconf
84−
85− # For ghc and cabal-install packages from hvr's ppa
80+ # Alternative way of installing ghc and cabal, directly from
81+ # haskell.org instead of using ghcup. NOTE: this is for Debian
82+ # only and is debian release specific.
8683 # gnupg is required for apt-key to work
87− apt-get install -y gnupg
88− apt-get install -y apt-transport-https
89− apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA3CBA3FFE22B574
90− apt-get install -y software-properties-common
91− add-apt-repository -y ppa:hvr/ghc
92− # echo "deb http://downloads.haskell.org/debian stretch main" >> /etc/apt/sources.list
93− echo "deb-src http://ppa.launchpad.net/hvr/ghc/ubuntu xenial main" >> /etc/apt/sources.list
94− apt-get update
84+ #apt-get install -y gnupg
85+ #apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA3CBA3FFE22B574
86+ #echo "deb http://downloads.haskell.org/debian buster main" >> /etc/apt/sources.list
87+ #apt-get update
9588
96− # required for packcheck
97− apt-get install -y curl
89+ # required for outbound https for stack and for stack setup
90+ apt-get install -y netbase xz-utils make
91+ apt-get install -y zlib1g-dev
9892
99− # required for circleci.
100− apt-get install -y libcurl4-gnutls-dev
101− apt-get install -y git
93+ # For ghcup to install ghc
94+ if test -n "$GHCUP_VERSION"
95+ then
96+ apt-get install -y gcc
97+ apt-get install -y g++
98+ fi
99+
100+ # libgmp required by ghc for linking
101+ apt-get install -y libgmp-dev
102102 apt-get install -y libtinfo-dev
103103
104− # required for outbound https for stack and for stack setup
105− apt-get install -y netbase xz-utils make
104+ # Required by cabal when git URL is specified in project file
105+ apt-get install -y git
106+
107+ # Required for and by packcheck
108+ apt-get install -y curl
106109
107−project-preinstall: &project-preinstall
108− run: |
109110 # Get packcheck if needed
110− CURL=$(which curl)
111− PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
112− if test ! -e "$PACKCHECK"; then $CURL -sL -o "$PACKCHECK" $PACKCHECK_URL; fi;
113− chmod +x $PACKCHECK
111+ if test ! -e "$PACKCHECK"
112+ then
113+ if test -z "$PACKCHECK_GITHUB_COMMIT"
114+ then
115+ die "PACKCHECK_GITHUB_COMMIT is not specified."
116+ fi
117+ PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
118+ curl --fail -sL -o "$PACKCHECK" $PACKCHECK_URL || exit 1
119+ chmod +x $PACKCHECK
120+ elif test ! -x "$PACKCHECK"
121+ then
122+ chmod +x $PACKCHECK
123+ fi
114124
115125restore: &restore
116126 # Needs to happen after installing ca-certificates
@@ -126,271 +136,71 @@ save: &save
126136 save_cache:
127137 key: v1-{{ .Environment.CIRCLE_JOB }}-{{ .Branch }}
128138 paths:
129− - ~/.cabal
130− - ~/.ghc
131139 - ~/.local
140+ - ~/.cabal
132141 - ~/.stack
142+ - ~/.ghcup
133143
134144#-----------------------------------------------------------------------------
135145# Build matrix
136146#-----------------------------------------------------------------------------
137147
138148jobs:
139− cabal-ghc-8_10_4:
149+ cabal-ghc-8_6_5:
140150 <<: *env
141151 executor: amd64-executor
142152 steps:
143− - *preinstall
144153 - checkout
145− - *project-preinstall
146154 - *restore
155+ - *preinstall
147156 - run:
148157 environment:
158+ GHCVER: "8.6.5"
159+ CABAL_VERSION: "3.6.2.0"
149160 CABAL_PROJECT: "cabal.project"
150− DISABLE_TEST: "yes"
151− DISABLE_BENCH: "yes"
152161 DISABLE_SDIST_BUILD: "yes"
162+ CABAL_BUILD_OPTIONS: "--flag debug --flag -opt"
153163 command: |
154− apt-get install -y ghc-8.10.4
155− apt-get install -y cabal-install-3.4
156164 bash -c "$PACKCHECK cabal"
157165 - *save
158− stack-ghc-8_8:
166+ cabal-ghc-9_8_1-docspec:
159167 <<: *env
160168 executor: amd64-executor
161169 steps:
162− - *preinstall
163170 - checkout
164− - *project-preinstall
165171 - *restore
166− - run:
167− name: GHC 8.8 + stack lts-16.12
168− environment:
169− BUILD: "stack"
170− RESOLVER: "lts-16.12"
171− SDIST_OPTIONS: "--ignore-check"
172− STACK_BUILD_OPTIONS: "--flag streamly-benchmarks:-opt"
173− command: |
174− apt-get build-dep -y ghc-8.8.4
175− apt-get install -y cabal-install-3.2
176− bash -c "$PACKCHECK $BUILD"
177− - *save
178− cabal-ghc-8_6_5:
179− <<: *env
180− executor: amd64-executor
181− steps:
182172 - *preinstall
183− - checkout
184− - *project-preinstall
185− - *restore
186− - run: |
187− apt-get install -y ghc-8.6.3
188− apt-get install -y cabal-install-2.4
189− bash -c "$PACKCHECK cabal-v2"
190− - *save
191− cabal-ghc-8_4_4:
192− <<: *env
193− executor: x86-executor
194− steps:
195− - *preinstall
196− - checkout
197− - *project-preinstall
198− - *restore
199− - run:
200− name: GHC 8.4.4 + x86 + debug
201− environment:
202− BUILD: "cabal-v2"
203− GHCVER: "8.4.4"
204− CABAL_BUILD_OPTIONS: "--flag debug --flag limit-build-mem"
205− command: |
206− apt-get install -y ghc-8.4.4
207− apt-get install -y cabal-install-3.2
208− bash -c "$PACKCHECK $BUILD"
209− - *save
210− cabal-ghc-8_2_2:
211− <<: *env
212− executor: amd64-executor
213− steps:
214− - *preinstall
215− - checkout
216− - *project-preinstall
217− - *restore
218173 - run:
219174 environment:
220− BUILD: "cabal-v2"
221− DISABLE_TEST: "yes"
222− DISABLE_BENCH: "yes"
223− DISABLE_DOCS: "yes"
224− DISABLE_SDIST_BUILD: "yes"
225− DISABLE_DIST_CHECKS: "yes"
175+ GHCVER: "9.8.1"
176+ CABAL_VERSION: "3.10.1.0"
177+ CABAL_PROJECT: "cabal.project.doctest"
178+ DISABLE_SDIST_BUILD: "y"
179+ DISABLE_TEST: "y"
180+ DISABLE_BENCH: "y"
181+ DISABLE_DOCS: "y"
182+ ENABLE_DOCSPEC: "y"
183+ DOCSPEC_URL: https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz
184+ DOCSPEC_OPTIONS: "--timeout 60 --check-properties --property-variables xs"
226185 command: |
227− apt-get install -y ghc-8.2.2
228− apt-get install -y cabal-install-3.2
229− bash -c "$PACKCHECK $BUILD"
230− - *save
231− cabal-ghc-8_0_2:
232− <<: *env
233− executor: amd64-executor
234− steps:
235− - *preinstall
236− - checkout
237− - *project-preinstall
238− - *restore
239− - run:
240− environment:
241− BUILD: "cabal-v2"
242− DISABLE_TEST: "yes"
243− DISABLE_BENCH: "yes"
244− DISABLE_DOCS: "yes"
245− DISABLE_SDIST_BUILD: "yes"
246− DISABLE_DIST_CHECKS: "yes"
247− command: |
248− apt-get install -y ghc-8.0.2
249− apt-get install -y cabal-install-3.2
250− bash -c "$PACKCHECK $BUILD"
251− - *save
252− cabal-ghc-7_10_3:
253− <<: *env
254− executor: amd64-executor
255− steps:
256− - *preinstall
257− - checkout
258− - *project-preinstall
259− - *restore
260− - run: |
261− apt-get install -y ghc-7.10.3
262− apt-get install -y cabal-install-2.4
263− bash -c "$PACKCHECK cabal-v2"
186+ bash -c "$PACKCHECK cabal"
264187 - *save
265− stack-ghc-8_4:
188+ hlint-trailing-spaces:
266189 <<: *env
267190 executor: amd64-executor
268191 steps:
269− - *preinstall
270192 - checkout
271− - *project-preinstall
272193 - *restore
273− - run: |
274− bash -c "$PACKCHECK stack RESOLVER=lts-12"
275− - *save
276− stack-ghc-8_2:
277− <<: *env
278− executor: amd64-executor
279− steps:
280194 - *preinstall
281− - checkout
282− - *project-preinstall
283− - *restore
284− - run: |
285− bash -c "$PACKCHECK stack RESOLVER=lts-11"
286− - *save
287− cabal-ghcjs-8_4:
288− <<: *env
289− executor: amd64-executor
290− steps:
291− - *preinstall
292− - checkout
293− - *project-preinstall
294− - *restore
295195 - run:
296− name: GHCJS 8.4+no-test+no-docs
196+ name: hlint and trailing spaces
297197 environment:
298− BUILD: cabal-v2
299− ENABLE_GHCJS: "yes"
300− DISABLE_TEST: "yes"
301− ENABLE_INSTALL: ""
302− DISABLE_DOCS: "yes"
303198 DISABLE_SDIST_BUILD: "yes"
304− command: |
305− add-apt-repository -y ppa:hvr/ghcjs
306− add-apt-repository -y 'deb https://deb.nodesource.com/node_11.x xenial main'
307− curl -sSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -
308− apt-get update
309− apt-get install -y cabal-install-3.2
310− apt-get install -y nodejs
311− apt-get install -y ghcjs-8.4
312− cabal user-config update -a "jobs: 1"
313− bash -c "$PACKCHECK $BUILD"
314− - *save
315− coveralls-ghc-8_2_2:
316− <<: *env
317− executor: amd64-executor
318− steps:
319− - *preinstall
320− - checkout
321− - *project-preinstall
322− - *restore
323− - run:
324− environment:
325− BUILD: cabal
326− COVERALLS_OPTIONS: "--repo-token=\"$REPO_TOKEN\" --coverage-mode=StrictlyFullLines --exclude-dir=test test"
327− GHC_OPTIONS: "-DCOVERAGE_BUILD"
328− command: |
329− apt-get install -y ghc-8.2.2
330− apt-get install -y cabal-install-2.4
331− bash -c "$PACKCHECK cabal-v1"
332− - *save
333− coveralls-ghc-8_8_3:
334− <<: *env
335− executor: amd64-executor
336− steps:
337− - *preinstall
338− - checkout
339− - *project-preinstall
340− - *restore
341− - run:
342− environment:
343− BUILD: cabal-v2
344− COVERALLS_OPTIONS: "--repo-token=\"$REPO_TOKEN\" --coverage-mode=StrictlyFullLines --exclude-dir=test"
345− CABAL_PROJECT: "cabal.project.coverage"
346− CABAL_BUILD_TARGETS: ""
347− command: |
348− apt-get install -y ghc-8.8.3
349− apt-get install -y cabal-install-3.0
350− bash -c "$PACKCHECK cabal-v2"
351− no_output_timeout: 25m
352− - *save
353− coveralls-ghc-8_10_2:
354− <<: *env
355− executor: amd64-executor
356− steps:
357− - *preinstall
358− - checkout
359− - *project-preinstall
360− - *restore
361− - run:
362− name: Build and test with inspection, coverage, Werror
363− command: |
364− apt-get install -y ghc-8.10.2
365− apt-get install -y cabal-install-3.2
366− # Run tests with coverage
367− cabal update
368− cabal user-config update -a "jobs: 1"
369− bin/test.sh --coverage --raw
370− # Upload results to coveralls.io
371− PATH=$HOME/.cabal/bin:$PATH
372− export PATH
373− which hpc-coveralls 2>/dev/null || cabal install --project-file cabal.project.hpc-coveralls hpc-coveralls
374− hpc-coveralls --repo-token="$REPO_TOKEN" --coverage-mode=StrictlyFullLines
375− no_output_timeout: 25m
376− - *save
377− hlint-src:
378− <<: *env
379− executor: amd64-executor
380− steps:
381− - *preinstall
382− - checkout
383− - *project-preinstall
384− - *restore
385− - run:
386− name: Hlint src
387− environment:
388− BUILD: hlint
199+ HLINT_VERSION: 3.6.1
389200 HLINT_OPTIONS: lint
390− DISABLE_SDIST_BUILD: "yes"
391201 HLINT_TARGETS: core/src src test benchmark
392202 command: |
393− bash -c "$PACKCHECK $BUILD" || exit 1
203+ bash -c "$PACKCHECK hlint" || exit 1
394204 echo "Checking trailing spaces..."
395205 count=$(find . -name "*.hs" -exec grep -H '\ $' {} \; | tee /dev/tty | wc -l)
396206 exit $count
@@ -400,23 +210,9 @@ workflows:
400210 version: 2
401211 build:
402212 jobs:
403− - cabal-ghc-8_10_4:
404− name: GHC 8.10.4 + sdist + no-test + no-bench
405− #- cabal-ghc-8.6.5
406− #- cabal-ghc-8_4_4:
407− # name: GHC 8.4.4 + x86 + debug
408− #- cabal-ghc-8_2_2:
409− # name: GHC 8.2.2 + no-test + no-bench + no-docs
410− #- cabal-ghc-8_0_2:
411− # name: GHC 8.0.2 + no-test + no-bench + no-docs
412− #- cabal-ghc-7.10.3
413− # - cabal-ghcjs-8_4:
414− # name: GHCJS 8.4 + no-test + no-docs
415− #- stack-ghc-8.4
416− #- stack-ghc-8.2
417− #- coveralls-ghc-8.2.2
418− #- coveralls-ghc-8.8.3
419− #- coveralls-ghc-8_10_2:
420− # name: GHC 8.10.2 + inspection + coverage + Werror
421− - hlint-src:
422− name: Hlint src + Trailing Spaces
213+ #- cabal-ghc-8_6_5:
214+ # name: 8.6.5-debug-unoptimized
215+ - cabal-ghc-9_8_1-docspec:
216+ name: ghc-9.8.1-docspec
217+ - hlint-trailing-spaces:
218+ name: hlint and trailing spaces
.github/workflows/haskell.yml+84−91
@@ -13,9 +13,15 @@ jobs:
1313 # packcheck environment variables
1414 LC_ALL: C.UTF-8
1515 BUILD: ${{ matrix.build }}
16+ # For updating see: https://downloads.haskell.org/~ghcup/
17+ GHCUP_VERSION: 0.1.20.0
18+ DOCSPEC_URL: https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz
19+ DOCSPEC_OPTIONS: "--timeout 60 --check-properties --property-variables xs"
20+ GHCUP_GHC_OPTIONS: ${{ matrix.ghcup_ghc_options }}
1621 GHCVER: ${{ matrix.ghc_version }}
1722 CABALVER: ${{ matrix.cabal_version }}
1823 DISABLE_DOCS: ${{ matrix.disable_docs }}
24+ ENABLE_DOCSPEC: ${{ matrix.enable_docspec }}
1925 DISABLE_TEST: ${{ matrix.disable_test }}
2026 DISABLE_BENCH: ${{ matrix.disable_bench }}
2127 DISABLE_DIST_CHECKS: ${{ matrix.disable_dist_checks }}
@@ -39,14 +45,18 @@ jobs:
3945 STACK_BUILD_OPTIONS: ${{ matrix.stack_build_options }}
4046
4147 # packcheck location and revision
42− PACKCHECK_LOCAL_PATH: "./packcheck.sh"
48+ PACKCHECK: "./packcheck.sh"
4349 PACKCHECK_GITHUB_URL: "https://raw.githubusercontent.com/composewell/packcheck"
44− PACKCHECK_GITHUB_COMMIT: "a302772c8f3791c477edd1764b82f8d1f5b22fdf"
50+ PACKCHECK_GITHUB_COMMIT: "v0.7.0"
4551
4652 # Pull token from "secrets" setting of the github repo
4753 COVERALLS_TOKEN: ${{ secrets.COVERALLS_TOKEN }}
4854 COVERAGE: ${{ matrix.coverage }}
49− DOCTEST: ${{ matrix.doctest }}
55+
56+ # hlint
57+ HLINT_VERSION: 3.6.1
58+ HLINT_OPTIONS: "lint"
59+ HLINT_TARGETS: "core/src src test benchmark"
5060
5161 # Subdir
5262 SUBDIR: ${{ matrix.subdir }}
@@ -58,8 +68,8 @@ jobs:
5868 matrix:
5969 # The order is important to optimize fail-fast.
6070 name:
61− - 9.4.4-Werror
62− - 9.2.7-docspec
71+ - 9.8.1-Werror
72+ # - 9.8.1-docspec
6373 # - 8.10.7-coverage
6474
6575 # Note: if cabal.project is not specified benchmarks and tests won't
@@ -68,51 +78,74 @@ jobs:
6878 include:
6979 - name: head
7080 ghc_version: head
81+ # The URL may change, to find a working URL go to https://gitlab.haskell.org/ghc/ghc/-/jobs/
82+ # Find a debian10/11/12 job, click on a passed/failed status, at the
83+ # end of the output you will find the tar.xz name, put that tar
84+ # name after "raw/", and put the job name after "job=".
85+ # Also see https://github.com/mpickering/ghc-artefact-nix/blob/master/gitlab-artifact.nix
86+ #
87+ # May also use ghcup for installing ghc head version, use the
88+ # version "LatestNightly", and the following config:
89+ # ghcup config add-release-channel https://ghc.gitlab.haskell.org/ghcup-metadata/ghcup-nightlies-0.0.7.yaml
90+ ghcup_ghc_options: "-u https://gitlab.haskell.org/ghc/ghc/-/jobs/artifacts/master/raw/ghc-x86_64-linux-deb10-int_native-validate.tar.xz?job=x86_64-linux-deb10-int_native-validate"
7191 runner: ubuntu-latest
7292 build: cabal
73− cabal_version: 3.8.1.0
93+ cabal_version: 3.10.2.0
7494 cabal_project: cabal.project.ghc-head
7595 disable_sdist_build: "y"
7696 ignore_error: true
77− - name: 9.6.1
78− ghc_version: 9.6.1
97+ # Note: use linux for warning build for convenient dev testing
98+ - name: 9.8.1-Werror
99+ ghc_version: 9.8.1
79100 runner: ubuntu-latest
80101 build: cabal
81102 cabal_version: 3.10.1.0
82103 disable_sdist_build: "y"
83− cabal_project: cabal.project
84− ignore_error: true
85− - name: 9.4.4-Werror
86− ghc_version: 9.4.4
87− runner: ubuntu-latest
88− build: cabal
89− cabal_version: 3.8.1.0
90− disable_sdist_build: "y"
91104 cabal_project: cabal.project.Werror
92105 ignore_error: false
93− - name: 9.4.4-fusion-inspection
94− ghc_version: 9.4.4
106+ #- name: 9.8.1-docspec
107+ # ghc_version: 9.8.1
108+ # runner: ubuntu-latest
109+ # build: cabal
110+ # cabal_version: 3.10.1.0
111+ # cabal_project: cabal.project.doctest
112+ # disable_test: "y"
113+ # disable_bench: "y"
114+ # disable_docs: "y"
115+ # enable_docspec: "y"
116+ # disable_sdist_build: "y"
117+ # ignore_error: false
118+ - name: 9.8.1-fusion-inspection
119+ ghc_version: 9.8.1
95120 runner: ubuntu-latest
96121 build: cabal
97− cabal_version: 3.8.1.0
122+ cabal_version: 3.10.1.0
98123 disable_sdist_build: "y"
99124 cabal_project: cabal.project
100125 cabal_build_options: "--flag fusion-plugin --flag inspection"
101126 ignore_error: false
102− - name: 9.2.7
103− ghc_version: 9.2.7
104− runner: ubuntu-latest
127+ - name: 9.6.3-macos
128+ ghc_version: 9.6.3
129+ runner: macos-latest
105130 build: cabal
131+ cabal_version: 3.10.1.0
132+ disable_sdist_build: "y"
106133 cabal_project: cabal.project
107− cabal_version: 3.6.2.0
134+ ignore_error: false
135+ - name: 9.4.7
136+ ghc_version: 9.4.7
137+ runner: ubuntu-latest
138+ build: cabal
139+ cabal_version: 3.8.1.0
108140 disable_sdist_build: "y"
141+ cabal_project: cabal.project
109142 ignore_error: false
110− - name: 9.2.7-docspec
111− ghc_version: 9.2.7
143+ - name: 9.2.8
144+ ghc_version: 9.2.8
112145 runner: ubuntu-latest
113− doctest: "y"
114− cabal_version: 3.6.2.0
146+ build: cabal
115147 cabal_project: cabal.project
148+ cabal_version: 3.6.2.0
116149 disable_sdist_build: "y"
117150 ignore_error: false
118151 - name: 9.0.2-streamly-sdist
@@ -138,14 +171,6 @@ jobs:
138171 stack_build_options: "--flag streamly-benchmarks:-opt"
139172 cabal_version: 3.6.2.0
140173 ignore_error: false
141− - name: 8.10.7-macos
142− ghc_version: 8.10.7
143− runner: macos-latest
144− build: cabal
145− cabal_version: 3.6.2.0
146− cabal_project: cabal.project
147− disable_sdist_build: "y"
148− ignore_error: false
149174 # - name: 8.10.7-coverage
150175 # ghc_version: 8.10.7
151176 # runner: ubuntu-latest
@@ -160,15 +185,6 @@ jobs:
160185 cabal_project: cabal.project
161186 disable_sdist_build: "y"
162187 ignore_error: false
163− - name: 8.6.5-no-fusion-unoptimized
164− ghc_version: 8.6.5
165− runner: ubuntu-latest
166− build: cabal
167− cabal_version: 3.6.2.0
168− cabal_project: cabal.project
169− cabal_build_options: "--flag no-fusion --flag -opt"
170− disable_sdist_build: "y"
171− ignore_error: false
172188 - name: 8.6.5-debug-unoptimized
173189 ghc_version: 8.6.5
174190 runner: ubuntu-latest
@@ -178,66 +194,44 @@ jobs:
178194 cabal_build_options: "--flag debug --flag -opt"
179195 disable_sdist_build: "y"
180196 ignore_error: false
197+ - name: hlint
198+ build: hlint
199+ runner: ubuntu-latest
200+ ignore_error: true
181201
182202 steps:
183203 - uses: actions/checkout@v2
184204
185205 - uses: actions/cache@v1
186− name: Cache ~/.cabal
206+ name: Cache common directories
187207 with:
188− path: ~/.cabal
208+ path: |
209+ ~/.local
210+ ~/.cabal
211+ ~/.stack
212+ ~/.ghcup
189213 # Bump the key version to clear the cache
190214 key: ${{ runner.os }}-${{ matrix.ghc_version }}-cabal-v2
191215
192216 - name: Download packcheck
193217 run: |
194− # Get packcheck if needed
195− CURL=$(which curl)
196− PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
197− if test ! -e "$PACKCHECK_LOCAL_PATH"; then $CURL -sL -o "$PACKCHECK_LOCAL_PATH" $PACKCHECK_URL; fi;
198− chmod +x $PACKCHECK_LOCAL_PATH
199−
200− os=$(uname -s -m)
201− case "$os" in
202− "Linux x86_64") GHCUP_ARCH="x86_64-linux" ;;
203− "Darwin x86_64") GHCUP_ARCH="x86_64-apple-darwin" ;;
204− *) echo "Unknown OS/Arch: $os"; exit 1;;
205− esac
206−
207− # Check available versions here: https://downloads.haskell.org/~ghcup/
208− GHCUP_VER=0.1.18.0
209− $CURL -sL -o ./ghcup https://downloads.haskell.org/~ghcup/$GHCUP_VER/${GHCUP_ARCH}-ghcup-$GHCUP_VER
210− chmod +x ./ghcup
211− if test "$GHCVER" = "head"
218+ if test ! -e "$PACKCHECK"
212219 then
213− # The URL may change, to find a working URL go to https://gitlab.haskell.org/ghc/ghc/-/jobs/
214− # Find a debian10 job, click on a passed/failed job, at the
215− # end of the output you will find the tar.xz name, put that after
216− # "raw/", and put the job name after "job=".
217− # Also see https://github.com/mpickering/ghc-artefact-nix/blob/master/gitlab-artifact.nix
218− ./ghcup install ghc -u https://gitlab.haskell.org/ghc/ghc/-/jobs/artifacts/master/raw/ghc-x86_64-linux-deb10-int_native-validate.tar.xz?job=x86_64-linux-deb10-int_native-validate head
219− else
220− ./ghcup install ghc $GHCVER
220+ if test -z "$PACKCHECK_GITHUB_COMMIT"
221+ then
222+ die "PACKCHECK_GITHUB_COMMIT is not specified."
223+ fi
224+ PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
225+ curl --fail -sL -o "$PACKCHECK" $PACKCHECK_URL || exit 1
226+ chmod +x $PACKCHECK
227+ elif test ! -x "$PACKCHECK"
228+ then
229+ chmod +x $PACKCHECK
221230 fi
222− ./ghcup install cabal $CABALVER
223− ./ghcup set ghc $GHCVER
224− ./ghcup set cabal $CABALVER
225231
226232 - name: Run tests
227233 run: |
228− if test -n "$DOCTEST"
229− then
230− mkdir -p $HOME/.cabal/bin
231− curl -sL https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz > cabal-docspec.xz
232− echo '0829bd034fba901cbcfe491d98ed8b28fd54f9cb5c91fa8e1ac62dc4413c9562 cabal-docspec.xz' | sha256sum -c -
233− xz -d < cabal-docspec.xz > $HOME/.cabal/bin/cabal-docspec
234− rm -f cabal-docspec.xz
235− chmod a+x $HOME/.cabal/bin/cabal-docspec
236− cabal update
237− cabal build --project-file cabal.project.doctest all
238− $HOME/.cabal/bin/cabal-docspec --version
239− $HOME/.cabal/bin/cabal-docspec --timeout 60 --check-properties --property-variables xs
240− elif test -n "$COVERAGE"
234+ if test -n "$COVERAGE"
241235 then
242236 # Run tests with coverage
243237 cabal update
@@ -250,16 +244,15 @@ jobs:
250244 # XXX Uncomment this and fix it properly later
251245 # hpc-coveralls --repo-token="$COVERALLS_TOKEN" --coverage-mode=StrictlyFullLines
252246 else
253− export TOOLS_DIR=/opt
254247 # /usr/local/opt/curl/bin for macOS
255248 export PATH=$HOME/.local/bin:$HOME/.ghcup/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/opt/curl/bin
256249 if test -n "$SUBDIR"
257250 then
258− mv "$PACKCHECK_LOCAL_PATH" "$SUBDIR/packcheck.sh"
251+ mv "$PACKCHECK" "$SUBDIR/packcheck.sh"
259252 # This is required as dist-newstyle will be created on the top level
260253 # directory as it is considered the working dir.
261254 rm cabal.project
262255 cd "$SUBDIR"
263256 fi
264− bash -c "$PACKCHECK_LOCAL_PATH $BUILD"
257+ bash -c "$PACKCHECK $BUILD"
265258 fi
.github/workflows/regression-check.yml+2−2
@@ -26,8 +26,8 @@ jobs:
2626 Data.Fold.Window
2727 Data.Parser
2828 Data.ParserK
29− Data.Stream.StreamD
30− Data.Stream.StreamK:6
29+ Data.StreamD
30+ Data.StreamK:6
3131 Data.Unfold
3232 Data.Stream
3333 Data.Stream.Concurrent
.hlint.ignore+5
@@ -2,6 +2,9 @@ core/src/Streamly/Internal/Data/Stream/StreamK/Type.hs
22core/src/Streamly/Internal/Data/Pipe/Type.hs
33core/src/Streamly/Internal/Unicode/Stream.hs
44src/Streamly/Internal/Data/SmallArray/Type.hs
5+src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs
6+src/Streamly/Internal/Data/Stream/Serial.hs
7+src/Streamly/Internal/Data/Stream/Zip.hs
58test/Streamly/Test/Data/Array.hs
69test/Streamly/Test/Data/Parser.hs
710test/Streamly/Test/Data/ParserK.hs
@@ -10,6 +13,7 @@ test/Streamly/Test/Data/Unfold.hs
1013test/Streamly/Test/FileSystem/Event.hs
1114test/Streamly/Test/Prelude/Concurrent.hs
1215test/Streamly/Test/Data/Stream/Concurrent.hs
16+test/lib/Streamly/Test/Prelude/Common.hs
1317test/Streamly/Test/Prelude/Fold.hs
1418test/Streamly/Test/Prelude/Rate.hs
1519test/Streamly/Test/Prelude/Serial.hs
@@ -27,6 +31,7 @@ benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs
2731benchmark/Streamly/Benchmark/Data/Unfold.hs
2832benchmark/Streamly/Benchmark/FileSystem/Handle.hs
2933benchmark/Streamly/Benchmark/Prelude/Async.hs
34+benchmark/Streamly/Benchmark/Prelude/Concurrent.hs
3035benchmark/Streamly/Benchmark/Prelude/Merge.hs
3136benchmark/Streamly/Benchmark/Prelude/Parallel.hs
3237benchmark/Streamly/Benchmark/Prelude/Rate.hs
benchmark/NanoBenchmarks.hs+1−1
@@ -18,7 +18,7 @@ import qualified Streamly.Data.Fold as FL
1818import qualified Streamly.Internal.Data.Stream as S
1919import qualified Streamly.Internal.Data.Stream.IsStream as Internal
2020
21−import Gauge
21+import Test.Tasty.Bench
2222import System.Random
2323
2424maxValue :: Int
benchmark/Streamly/Benchmark/Data/Array/CommonImports.hs+1−1
@@ -9,7 +9,7 @@ import qualified Streamly.Data.Fold as Fold
99import qualified Streamly.Data.Stream as S
1010import qualified Streamly.Internal.Data.Stream as Stream
1111
12−import Gauge
12+import Test.Tasty.Bench
1313import Streamly.Benchmark.Common hiding (benchPureSrc)
1414import qualified Stream.Common as P
1515
benchmark/Streamly/Benchmark/Data/Array/Stream.hs+3−3
@@ -39,7 +39,7 @@ import Prelude hiding ()
3939
4040import qualified Streamly.Data.Stream as Stream
4141import qualified Streamly.Internal.Data.Array as Array
42−import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
42+import qualified Streamly.Internal.Data.Array.Stream as ArrayStream
4343import qualified Streamly.Internal.Data.Fold as Fold
4444import qualified Streamly.Internal.Data.Parser as Parser
4545import qualified Streamly.Internal.Data.Stream as Stream
@@ -47,13 +47,13 @@ import qualified Streamly.Internal.Data.StreamK as StreamK
4747import qualified Streamly.Internal.FileSystem.Handle as Handle
4848import qualified Streamly.Internal.Unicode.Stream as Unicode
4949
50−import Gauge hiding (env)
50+import Test.Tasty.Bench hiding (env)
5151import Streamly.Benchmark.Common
5252import Streamly.Benchmark.Common.Handle
5353import Control.Monad.IO.Class (MonadIO)
5454
5555#ifdef INSPECTION
56−import Streamly.Internal.Data.Unbox (Unbox)
56+import Streamly.Internal.Data.MutByteArray (Unbox)
5757import Streamly.Internal.Data.Stream (Step(..))
5858import Test.Inspection
5959#endif
benchmark/Streamly/Benchmark/Data/Fold.hs+28−4
@@ -12,6 +12,11 @@
1212{-# OPTIONS_GHC -fno-warn-warnings-deprecations #-}
1313{-# OPTIONS_GHC -Wno-orphans #-}
1414
15+#undef FUSION_CHECK
16+#ifdef FUSION_CHECK
17+{-# OPTIONS_GHC -ddump-simpl -ddump-to-file -dsuppress-all #-}
18+#endif
19+
1520module Main (main) where
1621
1722import Control.DeepSeq (NFData(..))
@@ -36,7 +41,7 @@ import qualified Streamly.Internal.Data.Unfold as Unfold
3641import qualified Streamly.Internal.Data.Pipe as Pipe
3742import qualified Streamly.Internal.Data.Stream as Stream
3843
39−import Gauge
44+import Test.Tasty.Bench
4045import Streamly.Benchmark.Common
4146import Prelude hiding (all, any, take, unzip, sequence_, filter)
4247
@@ -264,13 +269,13 @@ unzip = Stream.fold $ FL.lmap (\a -> (a, a)) (FL.unzip FL.sum FL.length)
264269{-# INLINE unzipWithFstM #-}
265270unzipWithFstM :: Monad m => Stream m Int -> m (Int, Int)
266271unzipWithFstM = do
267− let f = \a -> return (a + 1, a)
272+ let f a = return (a + 1, a)
268273 Stream.fold (FL.unzipWithFstM f FL.sum FL.length)
269274
270275{-# INLINE unzipWithMinM #-}
271276unzipWithMinM :: Monad m => Stream m Int -> m (Int, Int)
272277unzipWithMinM = do
273− let f = \a -> return (a + 1, a)
278+ let f a = return (a + 1, a)
274279 Stream.fold (FL.unzipWithMinM f FL.sum FL.length)
275280
276281-------------------------------------------------------------------------------
@@ -502,7 +507,9 @@ o_n_heap_serial value =
502507-------------------------------------------------------------------------------
503508
504509main :: IO ()
505−main = runWithCLIOpts defaultStreamSize allBenchmarks
510+main = do
511+#ifndef FUSION_CHECK
512+ runWithCLIOpts defaultStreamSize allBenchmarks
506513
507514 where
508515
@@ -515,3 +522,20 @@ main = runWithCLIOpts defaultStreamSize allBenchmarks
515522 , bgroup (o_n_space_prefix moduleName) (o_n_space_serial value)
516523 , bgroup (o_n_heap_prefix moduleName) (o_n_heap_serial value)
517524 ]
525+#else
526+ -- Enable FUSION_CHECK macro at the beginning of the file
527+ -- Enable one benchmark below, and run the benchmark
528+ -- Check the .dump-simpl output
529+ let value = 100000
530+ let input = source value 1
531+ let getKey buckets = (`mod` buckets)
532+ let getFold k =
533+ return $ case k of
534+ 0 -> FL.sum
535+ 1 -> FL.length
536+ _ -> FL.length
537+
538+ -- demuxToMap (getKey 64) (getFold . getKey 64) input
539+ toIntMapIO (getKey 64) input
540+ return ()
541+#endif
benchmark/Streamly/Benchmark/Data/Fold/Window.hs+39−39
@@ -8,12 +8,12 @@ import Streamly.Internal.Data.Stream (Stream)
88import System.Random (randomRIO)
99
1010import qualified Streamly.Data.Fold as Fold
11+import qualified Streamly.Internal.Data.Fold as Window
1112import qualified Streamly.Internal.Data.Ring as Ring
12−import qualified Streamly.Internal.Data.Fold.Window as Window
1313import qualified Streamly.Internal.Data.Stream as Stream
1414
1515import Streamly.Benchmark.Common
16−import Gauge
16+import Test.Tasty.Bench
1717
1818{-# INLINE source #-}
1919source :: (Monad m, Num a, Stream.Enumerable a) =>
@@ -70,55 +70,55 @@ o_1_space_folds :: Int -> [Benchmark]
7070o_1_space_folds numElements =
7171 [ bgroup "fold"
7272 [ benchWithFold numElements "minimum (window size 100)"
73− (Window.minimum 100)
73+ (Window.windowMinimum 100)
7474 , benchWithFold numElements "minimum (window size 1000)"
75− (Window.minimum 1000)
75+ (Window.windowMinimum 1000)
7676 , benchWith sourceDescendingInt numElements
7777 "minimum descending (window size 1000)"
78− (Window.minimum 1000)
78+ (Window.windowMinimum 1000)
7979
8080 , benchWithFold numElements "maximum (window size 100)"
81− (Window.maximum 100)
81+ (Window.windowMaximum 100)
8282 , benchWithFold numElements "maximum (window size 1000)"
83− (Window.maximum 1000)
83+ (Window.windowMaximum 1000)
8484 , benchWith sourceDescendingInt numElements
8585 "maximum descending (window size 1000)"
86− (Window.maximum 1000)
86+ (Window.windowMaximum 1000)
8787
8888 , benchWithFold numElements "range (window size 100)"
89− (Window.range 100)
89+ (Window.windowRange 100)
9090 , benchWithFold numElements "range (window size 1000)"
91− (Window.range 1000)
91+ (Window.windowRange 1000)
9292 , benchWith sourceDescendingInt numElements
9393 "range descending (window size 1000)"
94− (Window.range 1000)
94+ (Window.windowRange 1000)
9595
9696 , benchWithFoldInt numElements "sumInt (window size 100)"
97− (Ring.slidingWindow 100 Window.sumInt)
97+ (Ring.slidingWindow 100 Window.windowSumInt)
9898 , benchWithFoldInt numElements "sum for Int (window size 100)"
99− (Ring.slidingWindow 100 Window.sum)
99+ (Ring.slidingWindow 100 Window.windowSum)
100100 , benchWithFold numElements "sum (window size 100)"
101− (Ring.slidingWindow 100 Window.sum)
101+ (Ring.slidingWindow 100 Window.windowSum)
102102 , benchWithFold numElements "sum (window size 1000)"
103− (Ring.slidingWindow 1000 Window.sum)
103+ (Ring.slidingWindow 1000 Window.windowSum)
104104 , benchWithFold numElements "sum (entire stream)"
105− (Window.cumulative Window.sum)
105+ (Window.cumulative Window.windowSum)
106106 , benchWithFold numElements "sum (Data.Fold)"
107107 Fold.sum
108108
109109 , benchWithFold numElements "mean (window size 100)"
110− (Ring.slidingWindow 100 Window.mean)
110+ (Ring.slidingWindow 100 Window.windowMean)
111111 , benchWithFold numElements "mean (window size 1000)"
112− (Ring.slidingWindow 1000 Window.mean)
112+ (Ring.slidingWindow 1000 Window.windowMean)
113113 , benchWithFold numElements "mean (entire stream)"
114− (Window.cumulative Window.mean)
114+ (Window.cumulative Window.windowMean)
115115 , benchWithFold numElements "mean (Data.Fold)"
116116 Fold.mean
117117
118118 , benchWithFold numElements "powerSum 2 (window size 100)"
119− (Ring.slidingWindow 100 (Window.powerSum 2))
119+ (Ring.slidingWindow 100 (Window.windowPowerSum 2))
120120 , benchWithFold numElements "powerSum 2 (entire stream)"
121− (Window.cumulative (Window.powerSum 2))
121+ (Window.cumulative (Window.windowPowerSum 2))
122122
123123 ]
124124 ]
@@ -127,51 +127,51 @@ o_1_space_scans :: Int -> [Benchmark]
127127o_1_space_scans numElements =
128128 [ bgroup "scan"
129129 [ benchWithPostscan numElements "minimum (window size 10)"
130− (Window.minimum 10)
130+ (Window.windowMinimum 10)
131131 -- Below window size 30 the linear search based impl performs better
132132 -- than the dequeue based implementation.
133133 , benchWithPostscan numElements "minimum (window size 30)"
134− (Window.minimum 30)
134+ (Window.windowMinimum 30)
135135 , benchWithPostscan numElements "minimum (window size 1000)"
136− (Window.minimum 1000)
136+ (Window.windowMinimum 1000)
137137 , benchScanWith sourceDescendingInt numElements
138138 "minimum descending (window size 1000)"
139− (Window.minimum 1000)
139+ (Window.windowMinimum 1000)
140140
141141 , benchWithPostscan numElements "maximum (window size 10)"
142− (Window.maximum 10)
142+ (Window.windowMaximum 10)
143143 , benchWithPostscan numElements "maximum (window size 30)"
144− (Window.maximum 30)
144+ (Window.windowMaximum 30)
145145 , benchWithPostscan numElements "maximum (window size 1000)"
146− (Window.maximum 1000)
146+ (Window.windowMaximum 1000)
147147 , benchScanWith sourceDescendingInt numElements
148148 "maximum descending (window size 1000)"
149− (Window.maximum 1000)
149+ (Window.windowMaximum 1000)
150150
151151 , benchWithPostscan numElements "range (window size 10)"
152− (Window.range 10)
152+ (Window.windowRange 10)
153153 , benchWithPostscan numElements "range (window size 30)"
154− (Window.range 30)
154+ (Window.windowRange 30)
155155 , benchWithPostscan numElements "range (window size 1000)"
156− (Window.range 1000)
156+ (Window.windowRange 1000)
157157 , benchScanWith sourceDescendingInt numElements
158158 "range descending (window size 1000)"
159− (Window.range 1000)
159+ (Window.windowRange 1000)
160160
161161 , benchWithPostscan numElements "sum (window size 100)"
162− (Ring.slidingWindow 100 Window.sum)
162+ (Ring.slidingWindow 100 Window.windowSum)
163163 , benchWithPostscan numElements "sum (window size 1000)"
164− (Ring.slidingWindow 1000 Window.sum)
164+ (Ring.slidingWindow 1000 Window.windowSum)
165165
166166 , benchWithPostscan numElements "mean (window size 100)"
167− (Ring.slidingWindow 100 Window.mean)
167+ (Ring.slidingWindow 100 Window.windowMean)
168168 , benchWithPostscan numElements "mean (window size 1000)"
169− (Ring.slidingWindow 1000 Window.mean)
169+ (Ring.slidingWindow 1000 Window.windowMean)
170170
171171 , benchWithPostscan numElements "powerSum 2 (window size 100)"
172− (Ring.slidingWindow 100 (Window.powerSum 2))
172+ (Ring.slidingWindow 100 (Window.windowPowerSum 2))
173173 , benchWithPostscan numElements "powerSum 2 (window size 1000)"
174− (Ring.slidingWindow 1000 (Window.powerSum 2))
174+ (Ring.slidingWindow 1000 (Window.windowPowerSum 2))
175175 ]
176176 ]
177177
benchmark/Streamly/Benchmark/Data/MutArray.hs+1−1
@@ -49,7 +49,7 @@ import qualified Streamly.Internal.Data.MutArray as MArray
4949import qualified Streamly.Internal.Data.Fold as Fold
5050import qualified Streamly.Internal.Data.Stream as Stream
5151
52−import Gauge
52+import Test.Tasty.Bench
5353import Streamly.Benchmark.Common hiding (benchPureSrc)
5454
5555type Stream = MutArray
benchmark/Streamly/Benchmark/Data/Parser.hs+10−10
@@ -47,10 +47,9 @@ import qualified Streamly.Internal.Data.Fold as Fold
4747import qualified Streamly.Internal.Data.Parser as PR
4848import qualified Streamly.Data.Stream as Stream
4949import qualified Streamly.Internal.Data.Producer as Producer
50−import qualified Streamly.Internal.Data.Producer.Source as Source
5150import qualified Streamly.Internal.Data.Stream as Stream
5251
53−import Gauge hiding (env)
52+import Test.Tasty.Bench hiding (env)
5453import Streamly.Benchmark.Common
5554import Streamly.Benchmark.Common.Handle
5655
@@ -349,7 +348,7 @@ split_ value =
349348-- PR.dropWhile (<= (value * 1 `div` 4)) *> PR.die "alt"
350349{-# INLINE takeWhileFail #-}
351350takeWhileFail :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
352−takeWhileFail predicate (Fold fstep finitial fextract) =
351+takeWhileFail predicate (Fold fstep finitial _ ffinal) =
353352 Parser step initial extract
354353
355354 where
@@ -370,7 +369,7 @@ takeWhileFail predicate (Fold fstep finitial fextract) =
370369 Fold.Done b -> Done 0 b
371370 else return $ Error "fail"
372371
373− extract s = fmap (Done 0) (fextract s)
372+ extract s = fmap (Done 0) (ffinal s)
374373
375374{-# INLINE alt2 #-}
376375alt2 :: Monad m
@@ -635,13 +634,13 @@ choice value =
635634{-# INLINE parseManyUnfoldArrays #-}
636635parseManyUnfoldArrays :: Int -> [Array.Array Int] -> IO ()
637636parseManyUnfoldArrays count arrays = do
638− let src = Source.source (Just (Producer.OuterLoop arrays))
637+ let src = Producer.source (Just (Producer.OuterLoop arrays))
639638 let parser = PR.fromFold (Fold.take count Fold.drain)
640639 let readSrc =
641− Source.producer
640+ Producer.producer
642641 $ Producer.concat Producer.fromList Array.producer
643642 let streamParser =
644− Producer.simplify (Source.parseMany parser readSrc)
643+ Producer.simplify (Producer.parseMany parser readSrc)
645644 Stream.fold Fold.drain $ Stream.unfold streamParser src
646645
647646-------------------------------------------------------------------------------
@@ -715,8 +714,6 @@ instance NFData ParseError where
715714o_1_space_serial :: Int -> [Benchmark]
716715o_1_space_serial value =
717716 [ benchIOSink value "takeBetween" $ takeBetween value
718− , benchIOSink value "takeEQ" $ takeEQ value
719− , benchIOSink value "takeGE" $ takeGE value
720717 , benchIOSink value "takeWhile" $ takeWhile value
721718 , benchIOSink value "takeWhileP" $ takeWhileP value
722719 , benchIOSink value "takeP" $ takeP value
@@ -813,8 +810,11 @@ o_1_space_serial_unfold bound arrays =
813810o_n_heap_serial :: Int -> [Benchmark]
814811o_n_heap_serial value =
815812 [
813+ benchIOSink value "takeEQ" $ takeEQ value
814+ , benchIOSink value "takeGE" $ takeGE value
815+
816816 -- lookahead benchmark holds the entire input till end
817− benchIOSink value "lookAhead" $ lookAhead value
817+ , benchIOSink value "lookAhead" $ lookAhead value
818818
819819 -- o-n-heap because of backtracking
820820 , benchIOSrc sourceEscapedFrames value "takeFramedByEsc_"
benchmark/Streamly/Benchmark/Data/ParserK.hs+89−36
@@ -5,6 +5,10 @@
55-- License : BSD-3-Clause
66-- Maintainer : streamly@composewell.com
77
8+-- BENCH_CHUNKED -> adaptC
9+-- BENCH_CHUNKED_GENERIC -> adaptCG
10+-- BENCH_SINGULAR -> adapt
11+
812{-# LANGUAGE CPP #-}
913{-# LANGUAGE FlexibleContexts #-}
1014{-# OPTIONS_GHC -Wno-orphans #-}
@@ -18,7 +22,13 @@ import Control.Applicative ((<|>))
1822import Control.DeepSeq (NFData(..))
1923import Control.Monad.IO.Class (MonadIO)
2024import Data.Foldable (asum)
25+#ifdef BENCH_CHUNKED
2126import Streamly.Data.Array (Array, Unbox)
27+#endif
28+#ifdef BENCH_CHUNKED_GENERIC
29+import Streamly.Data.Array.Generic (Array)
30+import qualified Streamly.Internal.Data.Array.Generic as GenArr (chunksOf)
31+#endif
2232import Streamly.Internal.Data.Fold (Fold(..))
2333import Streamly.Data.StreamK (StreamK)
2434import Streamly.Internal.Data.Parser
@@ -35,12 +45,52 @@ import qualified Streamly.Data.Stream as Stream
3545import qualified Streamly.Data.Fold as FL
3646import qualified Streamly.Internal.Data.Fold as Fold
3747import qualified Streamly.Data.Parser as PRD
38−import qualified Streamly.Data.ParserK as PR
48+import qualified Streamly.Internal.Data.ParserK as PR
3949import qualified Streamly.Internal.Data.StreamK as StreamK
4050
41−import Gauge
51+import Test.Tasty.Bench
4252import Streamly.Benchmark.Common
4353
54+-------------------------------------------------------------------------------
55+-- CPP Helpers
56+-------------------------------------------------------------------------------
57+
58+#ifdef BENCH_CHUNKED
59+
60+#define PARSE_OP StreamK.parseChunks
61+#define FROM_PARSER adaptC
62+#define INPUT (Array a)
63+#define PARSE_ELEM (Array Int)
64+#define CONSTRAINT_IO (MonadIO m, Unbox a)
65+#define CONSTRAINT (Monad m, Unbox a)
66+#define MODULE_NAME "Data.ParserK.Chunked"
67+
68+#endif
69+
70+#ifdef BENCH_CHUNKED_GENERIC
71+
72+#define PARSE_OP StreamK.parseChunksGeneric
73+#define FROM_PARSER adaptCG
74+#define INPUT (Array a)
75+#define PARSE_ELEM (Array Int)
76+#define CONSTRAINT_IO (MonadIO m)
77+#define CONSTRAINT (Monad m)
78+#define MODULE_NAME "Data.ParserK.Chunked.Generic"
79+
80+#endif
81+
82+#ifdef BENCH_SINGULAR
83+
84+#define PARSE_OP StreamK.parse
85+#define FROM_PARSER adapt
86+#define INPUT a
87+#define PARSE_ELEM Int
88+#define CONSTRAINT_IO (MonadIO m)
89+#define CONSTRAINT (Monad m)
90+#define MODULE_NAME "Data.ParserK"
91+
92+#endif
93+
4494-------------------------------------------------------------------------------
4595-- Utilities
4696-------------------------------------------------------------------------------
@@ -60,49 +110,52 @@ sourceUnfoldrM value n = Stream.unfoldrM step n
60110{-# INLINE benchIOSink #-}
61111benchIOSink
62112 :: NFData b
63− => Int -> String -> (StreamK IO (Array Int) -> IO b) -> Benchmark
113+ => Int -> String -> (StreamK IO PARSE_ELEM -> IO b) -> Benchmark
64114benchIOSink value name f =
65115 bench name $ nfIO $ randomRIO (1,1)
66116 >>= f
67117 . StreamK.fromStream
118+#ifdef BENCH_CHUNKED
68119 . Stream.chunksOf 4000
120+#endif
121+#ifdef BENCH_CHUNKED_GENERIC
122+ . GenArr.chunksOf 4000
123+#endif
69124 . sourceUnfoldrM value
70125
71126-------------------------------------------------------------------------------
72127-- Parsers
73128-------------------------------------------------------------------------------
74129
75−#define PARSE_OP StreamK.parseChunks
76−
77130{-# INLINE one #-}
78131one :: MonadIO m =>
79− Int -> StreamK m (Array Int) -> m (Either ParseError (Maybe Int))
80−one value = StreamK.parseChunks p
132+ Int -> StreamK m PARSE_ELEM -> m (Either ParseError (Maybe Int))
133+one value = PARSE_OP p
81134
82135 where
83136
84137 p = do
85− m <- PR.fromFold FL.one
138+ m <- PR.FROM_PARSER (PRD.fromFold FL.one)
86139 case m of
87140 Just i -> if i >= value then pure m else p
88141 Nothing -> pure Nothing
89142
90143{-# INLINE satisfy #-}
91−satisfy :: (MonadIO m, Unbox a) => (a -> Bool) -> PR.ParserK a m a
92−satisfy = PR.fromParser . PRD.satisfy
144+satisfy :: CONSTRAINT_IO => (a -> Bool) -> PR.ParserK INPUT m a
145+satisfy = PR.FROM_PARSER . PRD.satisfy
93146
94147{-# INLINE takeWhile #-}
95−takeWhile :: (MonadIO m, Unbox a) => (a -> Bool) -> PR.ParserK a m ()
96−takeWhile p = PR.fromParser $ PRD.takeWhile p FL.drain
148+takeWhile :: CONSTRAINT_IO => (a -> Bool) -> PR.ParserK INPUT m ()
149+takeWhile p = PR.FROM_PARSER $ PRD.takeWhile p FL.drain
97150
98151{-# INLINE takeWhileK #-}
99152takeWhileK :: MonadIO m =>
100− Int -> StreamK m (Array Int) -> m (Either ParseError ())
153+ Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
101154takeWhileK value = PARSE_OP (takeWhile (<= value))
102155
103156{-# INLINE splitAp2 #-}
104157splitAp2 :: MonadIO m
105− => Int -> StreamK m (Array Int) -> m (Either ParseError ((), ()))
158+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ((), ()))
106159splitAp2 value =
107160 PARSE_OP
108161 ((,)
@@ -112,7 +165,7 @@ splitAp2 value =
112165
113166{-# INLINE splitAp8 #-}
114167splitAp8 :: MonadIO m
115− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
168+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
116169splitAp8 value =
117170 PARSE_OP
118171 ( (\() () () () () () () () -> ())
@@ -127,7 +180,7 @@ splitAp8 value =
127180 )
128181
129182{-# INLINE sequenceA #-}
130−sequenceA :: MonadIO m => Int -> StreamK m (Array Int) -> m Int
183+sequenceA :: MonadIO m => Int -> StreamK m PARSE_ELEM -> m Int
131184sequenceA value xs = do
132185 let parser = satisfy (> 0)
133186 list = Prelude.replicate value parser
@@ -136,14 +189,14 @@ sequenceA value xs = do
136189
137190{-# INLINE sequenceA_ #-}
138191sequenceA_ :: MonadIO m =>
139− Int -> StreamK m (Array Int) -> m (Either ParseError ())
192+ Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
140193sequenceA_ value xs = do
141194 let parser = satisfy (> 0)
142195 list = Prelude.replicate value parser
143196 PARSE_OP (F.sequenceA_ list) xs
144197
145198{-# INLINE sequence #-}
146−sequence :: MonadIO m => Int -> StreamK m (Array Int) -> m Int
199+sequence :: MonadIO m => Int -> StreamK m PARSE_ELEM -> m Int
147200sequence value xs = do
148201 let parser = satisfy (> 0)
149202 list = Prelude.replicate value parser
@@ -152,7 +205,7 @@ sequence value xs = do
152205
153206{-# INLINE sequence_ #-}
154207sequence_ :: MonadIO m =>
155− Int -> StreamK m (Array Int) -> m (Either ParseError ())
208+ Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
156209sequence_ value =
157210 let parser = satisfy (> 0)
158211 list = Prelude.replicate value parser
@@ -160,7 +213,7 @@ sequence_ value =
160213
161214{-# INLINE takeWhileFailD #-}
162215takeWhileFailD :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
163−takeWhileFailD predicate (Fold fstep finitial fextract) =
216+takeWhileFailD predicate (Fold fstep finitial _ ffinal) =
164217 Parser step initial extract
165218
166219 where
@@ -181,16 +234,16 @@ takeWhileFailD predicate (Fold fstep finitial fextract) =
181234 Fold.Done b -> Done 0 b
182235 else return $ Error "fail"
183236
184− extract s = fmap (Done 0) (fextract s)
237+ extract s = fmap (Done 0) (ffinal s)
185238
186239{-# INLINE takeWhileFail #-}
187−takeWhileFail :: (Monad m, Unbox a) =>
188− (a -> Bool) -> Fold m a b -> PR.ParserK a m b
189−takeWhileFail p f = PR.fromParser (takeWhileFailD p f)
240+takeWhileFail :: CONSTRAINT =>
241+ (a -> Bool) -> Fold m a b -> PR.ParserK INPUT m b
242+takeWhileFail p f = PR.FROM_PARSER (takeWhileFailD p f)
190243
191244{-# INLINE alt2 #-}
192245alt2 :: MonadIO m
193− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
246+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
194247alt2 value =
195248 PARSE_OP
196249 ( takeWhileFail (<= (value `div` 2)) Fold.drain
@@ -199,7 +252,7 @@ alt2 value =
199252
200253{-# INLINE alt8 #-}
201254alt8 :: MonadIO m
202− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
255+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
203256alt8 value =
204257 PARSE_OP
205258 ( takeWhileFail (<= ( value `div` 8)) Fold.drain
@@ -214,7 +267,7 @@ alt8 value =
214267
215268{-# INLINE alt16 #-}
216269alt16 :: MonadIO m
217− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
270+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
218271alt16 value =
219272 PARSE_OP
220273 ( takeWhileFail (<= ( value `div` 16)) Fold.drain
@@ -236,27 +289,27 @@ alt16 value =
236289 )
237290
238291{-# INLINE manyAlt #-}
239−manyAlt :: MonadIO m => StreamK m (Array Int) -> m Int
292+manyAlt :: MonadIO m => StreamK m PARSE_ELEM -> m Int
240293manyAlt xs = do
241294 x <- PARSE_OP (AP.many (satisfy (> 0))) xs
242295 return $ Prelude.length x
243296
244297{-# INLINE someAlt #-}
245−someAlt :: MonadIO m => StreamK m (Array Int) -> m Int
298+someAlt :: MonadIO m => StreamK m PARSE_ELEM -> m Int
246299someAlt xs = do
247300 x <- PARSE_OP (AP.some (satisfy (> 0))) xs
248301 return $ Prelude.length x
249302
250303{-# INLINE choice #-}
251304choice :: MonadIO m =>
252− Int -> StreamK m (Array Int) -> m (Either ParseError Int)
305+ Int -> StreamK m PARSE_ELEM -> m (Either ParseError Int)
253306choice value =
254307 PARSE_OP (asum (replicate value (satisfy (< 0)))
255308 AP.<|> satisfy (> 0))
256309
257310{-# INLINE monad2 #-}
258311monad2 :: MonadIO m
259− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
312+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
260313monad2 value =
261314 PARSE_OP $ do
262315 takeWhile (<= (value `div` 2))
@@ -264,7 +317,7 @@ monad2 value =
264317
265318{-# INLINE monad4 #-}
266319monad4 :: MonadIO m
267− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
320+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
268321monad4 value =
269322 PARSE_OP $ do
270323 takeWhile (<= ( value `div` 4))
@@ -274,7 +327,7 @@ monad4 value =
274327
275328{-# INLINE monad8 #-}
276329monad8 :: MonadIO m
277− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
330+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
278331monad8 value =
279332 PARSE_OP $ do
280333 takeWhile (<= ( value `div` 8))
@@ -288,7 +341,7 @@ monad8 value =
288341
289342{-# INLINE monad16 #-}
290343monad16 :: MonadIO m
291− => Int -> StreamK m (Array Int) -> m (Either ParseError ())
344+ => Int -> StreamK m PARSE_ELEM -> m (Either ParseError ())
292345monad16 value =
293346 PARSE_OP $ do
294347 takeWhile (<= ( value `div` 16))
@@ -313,7 +366,7 @@ monad16 value =
313366-------------------------------------------------------------------------------
314367
315368moduleName :: String
316−moduleName = "Data.ParserK"
369+moduleName = MODULE_NAME
317370
318371instance NFData ParseError where
319372 {-# INLINE rnf #-}
@@ -335,7 +388,7 @@ o_1_space_serial value =
335388 ]
336389
337390{-# INLINE sepBy1 #-}
338−sepBy1 :: MonadIO m => StreamK m (Array Int) -> m Int
391+sepBy1 :: MonadIO m => StreamK m PARSE_ELEM -> m Int
339392sepBy1 xs = do
340393 x <- PARSE_OP (parser (satisfy odd) (satisfy even)) xs
341394 return $ Prelude.length x
benchmark/Streamly/Benchmark/Data/Ring/Unboxed.hs+2−3
@@ -12,12 +12,11 @@ module Main (main) where
1212
1313import Control.Monad (void)
1414import GHC.Ptr (Ptr(..))
15−
16−import qualified Streamly.Internal.Data.Array.Type as Array
15+import qualified Streamly.Internal.Data.Array as Array
1716import qualified Streamly.Internal.Data.Ring as Ring
1817import qualified Data.Foldable as P
1918
20−import Gauge
19+import Test.Tasty.Bench
2120import Streamly.Benchmark.Common
2221
2322import Prelude as P
benchmark/Streamly/Benchmark/Data/Serialize.hsnew file+591
@@ -0,0 +1,591 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+{- HLINT ignore -}
3+
4+#undef FUSION_CHECK
5+#ifdef FUSION_CHECK
6+{-# OPTIONS_GHC -ddump-simpl -ddump-to-file -dsuppress-all #-}
7+#endif
8+
9+-- |
10+-- Module : Streamly.Benchmark.Data.Serialize
11+-- Copyright : (c) 2023 Composewell
12+-- License : BSD-3-Clause
13+-- Maintainer : streamly@composewell.com
14+
15+module Main (main) where
16+
17+-------------------------------------------------------------------------------
18+-- Imports
19+-------------------------------------------------------------------------------
20+
21+import Control.DeepSeq (NFData(..), deepseq)
22+import Control.Monad (when, void)
23+import GHC.Generics (Generic)
24+import System.Random (randomRIO)
25+#ifndef USE_UNBOX
26+import Test.QuickCheck (Arbitrary, arbitrary)
27+#endif
28+
29+import Streamly.Internal.Data.MutByteArray
30+import qualified Streamly.Internal.Data.MutByteArray as MBA
31+#ifdef USE_UNBOX
32+import Data.Proxy (Proxy(..))
33+#else
34+import Control.DeepSeq (force)
35+import Test.QuickCheck (oneof, generate)
36+#endif
37+
38+import Test.Tasty.Bench
39+import Streamly.Benchmark.Common
40+
41+#ifndef USE_UNBOX
42+import Streamly.Benchmark.Data.Serialize.RecCompatible
43+import Streamly.Benchmark.Data.Serialize.RecNonCompatible
44+#endif
45+
46+#ifdef USE_UNBOX
47+#define SERIALIZE_CLASS Unbox
48+#define DERIVE_CLASS(typ) $(deriveUnbox [d|instance Unbox typ|])
49+#define SERIALIZE_OP pokeAt
50+#define DESERIALIZE_OP peekAt
51+#else
52+#define SERIALIZE_CLASS Serialize
53+#define DERIVE_CLASS(typ) $(deriveSerialize [d|instance Serialize typ|])
54+#define SERIALIZE_OP serializeAt
55+#define DESERIALIZE_OP deserializeAt
56+#endif
57+
58+-------------------------------------------------------------------------------
59+-- Types
60+-------------------------------------------------------------------------------
61+
62+data Unit = Unit
63+ deriving (Generic, Show, Eq)
64+
65+#ifndef USE_TH
66+instance SERIALIZE_CLASS Unit
67+#else
68+DERIVE_CLASS(Unit)
69+#endif
70+
71+instance NFData Unit where
72+ {-# INLINE rnf #-}
73+ rnf _ = ()
74+
75+data Sum2
76+ = Sum21
77+ | Sum22
78+ deriving (Generic, Show, Eq)
79+
80+#ifndef USE_TH
81+instance SERIALIZE_CLASS Sum2
82+#else
83+DERIVE_CLASS(Sum2)
84+#endif
85+
86+instance NFData Sum2 where
87+ {-# INLINE rnf #-}
88+ rnf _ = ()
89+
90+data Sum25
91+ = Sum251
92+ | Sum252
93+ | Sum253
94+ | Sum254
95+ | Sum255
96+ | Sum256
97+ | Sum257
98+ | Sum258
99+ | Sum259
100+ | Sum2510
101+ | Sum2511
102+ | Sum2512
103+ | Sum2513
104+ | Sum2514
105+ | Sum2515
106+ | Sum2516
107+ | Sum2517
108+ | Sum2518
109+ | Sum2519
110+ | Sum2520
111+ | Sum2521
112+ | Sum2522
113+ | Sum2523
114+ | Sum2524
115+ | Sum2525
116+ deriving (Generic, Show, Eq)
117+
118+#ifndef USE_TH
119+instance SERIALIZE_CLASS Sum25
120+#else
121+DERIVE_CLASS(Sum25)
122+#endif
123+
124+instance NFData Sum25 where
125+ {-# INLINE rnf #-}
126+ rnf _ = ()
127+
128+data Product25
129+ = Product25
130+ Int
131+ Int
132+ Int
133+ Int
134+ Int
135+ Int
136+ Int
137+ Int
138+ Int
139+ Int
140+ Int
141+ Int
142+ Int
143+ Int
144+ Int
145+ Int
146+ Int
147+ Int
148+ Int
149+ Int
150+ Int
151+ Int
152+ Int
153+ Int
154+ Int
155+ deriving (Generic, Show)
156+
157+#ifndef USE_TH
158+instance SERIALIZE_CLASS Product25
159+#else
160+DERIVE_CLASS(Product25)
161+#endif
162+
163+instance NFData Product25 where
164+ {-# INLINE rnf #-}
165+ rnf
166+ (Product25 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 a13 a14 a15 a16 a17 a18 a19 a20 a21 a22 a23 a24 a25)
167+ = a1
168+ `seq` a2
169+ `seq` a3
170+ `seq` a4
171+ `seq` a5
172+ `seq` a6
173+ `seq` a7
174+ `seq` a8
175+ `seq` a9
176+ `seq` a10
177+ `seq` a11
178+ `seq` a12
179+ `seq` a13
180+ `seq` a14
181+ `seq` a15
182+ `seq` a16
183+ `seq` a17
184+ `seq` a18
185+ `seq` a19
186+ `seq` a20
187+ `seq` a21
188+ `seq` a22
189+ `seq` a23
190+ `seq` a24
191+ `seq` a25
192+ `seq` ()
193+
194+-- XXX derived Eq instance is not inlined
195+instance Eq Product25 where
196+ {-# INLINE (==) #-}
197+ (==)
198+ (Product25 a1 a2 a3 a4 a5 a6 a7 a8 a9 a10 a11 a12 a13 a14 a15 a16 a17 a18 a19 a20 a21 a22 a23 a24 a25)
199+ (Product25 b1 b2 b3 b4 b5 b6 b7 b8 b9 b10 b11 b12 b13 b14 b15 b16 b17 b18 b19 b20 b21 b22 b23 b24 b25)
200+ = a1 == b1
201+ && a2 == b2
202+ && a3 == b3
203+ && a4 == b4
204+ && a5 == b5
205+ && a6 == b6
206+ && a7 == b7
207+ && a8 == b8
208+ && a9 == b9
209+ && a10 == b10
210+ && a11 == b11
211+ && a12 == b12
212+ && a13 == b13
213+ && a14 == b14
214+ && a15 == b15
215+ && a16 == b16
216+ && a17 == b17
217+ && a18 == b18
218+ && a19 == b19
219+ && a20 == b20
220+ && a21 == b21
221+ && a22 == b22
222+ && a23 == b23
223+ && a24 == b24
224+ && a25 == b25
225+
226+-------------------------------------------------------------------------------
227+-- Simple non-recursive ADT
228+-------------------------------------------------------------------------------
229+
230+{-
231+data CustomDT1
232+ = CDT1C1
233+ | CDT1C2 Int
234+ | CDT1C3 Int Int
235+ deriving (Generic, Show, Eq)
236+
237+#ifndef USE_TH
238+instance SERIALIZE_CLASS CustomDT1
239+#else
240+DERIVE_CLASS(CustomDT1)
241+#endif
242+
243+instance NFData CustomDT1 where
244+ {-# INLINE rnf #-}
245+ rnf CDT1C1 = ()
246+ rnf (CDT1C2 i) = i `seq` ()
247+ rnf (CDT1C3 i1 i2) = i1 `seq` i2 `seq` ()
248+-}
249+
250+-------------------------------------------------------------------------------
251+-- Recursive ADT
252+-------------------------------------------------------------------------------
253+
254+#ifndef USE_UNBOX
255+data BinTree a
256+ = Tree (BinTree a) (BinTree a)
257+ | Leaf a
258+ deriving (Show, Read, Eq, Generic)
259+
260+#ifndef USE_TH
261+instance Serialize (BinTree a)
262+#else
263+$(deriveSerialize [d|instance Serialize a => Serialize (BinTree a)|])
264+#endif
265+
266+instance NFData a => NFData (BinTree a) where
267+ {-# INLINE rnf #-}
268+ rnf (Leaf a) = rnf a `seq` ()
269+ rnf (Tree l r) = rnf l `seq` rnf r
270+
271+-- XXX This may not terminate, or could become really large.
272+instance Arbitrary a => Arbitrary (BinTree a) where
273+ arbitrary = oneof [Leaf <$> arbitrary, Tree <$> arbitrary <*> arbitrary]
274+
275+mkBinTree :: (Arbitrary a) => Int -> IO (BinTree a)
276+mkBinTree = go (generate arbitrary)
277+
278+ where
279+
280+ go r 0 = Leaf <$> r
281+ go r n = Tree <$> go r (n - 1) <*> go r (n - 1)
282+
283+#endif
284+
285+-------------------------------------------------------------------------------
286+-- Size helpers
287+-------------------------------------------------------------------------------
288+
289+{-# INLINE getSize #-}
290+getSize :: forall a. SERIALIZE_CLASS a => a -> Int
291+#ifdef USE_UNBOX
292+getSize _ = sizeOf (Proxy :: Proxy a)
293+#else
294+getSize = addSizeTo 0
295+#endif
296+
297+-------------------------------------------------------------------------------
298+-- Common helpers
299+-------------------------------------------------------------------------------
300+
301+{- HLINT ignore "Eta reduce" -}
302+-- Parts of "f" that are dependent on val will not be optimized out.
303+{-# INLINE loop #-}
304+loop :: Int -> (a -> IO b) -> a -> IO ()
305+loop count f val = go count val
306+ where
307+
308+ go n x = when (n > 0) $ f x >> go (n-1) x
309+
310+-- The first arg of "f" is the environment which is not threaded around in the
311+-- loop.
312+{-# INLINE loopWith #-}
313+loopWith :: Int -> (e -> a -> IO b) -> e -> a -> IO ()
314+loopWith count f e val = go count val
315+ where
316+
317+ go n x = when (n > 0) $ f e x >> go (n-1) x
318+
319+benchSink :: NFData b => String -> Int -> (Int -> IO b) -> Benchmark
320+benchSink name times f = bench name (nfIO (randomRIO (times, times) >>= f))
321+
322+-------------------------------------------------------------------------------
323+-- Serialization Helpers
324+-------------------------------------------------------------------------------
325+
326+{-
327+{-# INLINE pokeWithSize #-}
328+pokeWithSize :: SERIALIZE_CLASS a => MutByteArray -> a -> IO ()
329+pokeWithSize arr val = do
330+ let n = getSize val
331+ n `seq` SERIALIZE_OP 0 arr val >> return ()
332+
333+{-# INLINE pokeTimesWithSize #-}
334+pokeTimesWithSize :: SERIALIZE_CLASS a => a -> Int -> IO ()
335+pokeTimesWithSize val times = do
336+ let n = getSize val
337+ arr <- MBA.new n
338+ loopWith times pokeWithSize arr val
339+-}
340+
341+{-# INLINE poke #-}
342+poke :: SERIALIZE_CLASS a => MutByteArray -> a -> IO ()
343+poke arr val = void (SERIALIZE_OP 0 arr val)
344+
345+{-# INLINE pokeTimes #-}
346+pokeTimes :: SERIALIZE_CLASS a => a -> Int -> IO ()
347+pokeTimes val times = do
348+ let n = getSize val
349+ arr <- MBA.new n
350+ loopWith times poke arr val
351+
352+{-# INLINE encode #-}
353+encode :: SERIALIZE_CLASS a => a -> IO ()
354+encode val = do
355+ let n = getSize val
356+ arr <- MBA.new n
357+ void (SERIALIZE_OP 0 arr val)
358+
359+{-# INLINE encodeTimes #-}
360+encodeTimes :: SERIALIZE_CLASS a => a -> Int -> IO ()
361+encodeTimes val times = loop times encode val
362+
363+{-# INLINE peek #-}
364+peek :: forall a. (NFData a, SERIALIZE_CLASS a) =>
365+ (a, Int) -> MutByteArray -> IO ()
366+#ifdef USE_UNBOX
367+peek (_val, _) arr = do
368+ (val1 :: a) <- DESERIALIZE_OP 0 arr
369+#else
370+peek (_val, n) arr = do
371+ (_, val1 :: a) <- DESERIALIZE_OP 0 arr n
372+#endif
373+ -- If the datatype is not deeply strict or deepseq is not used then use
374+ -- Equality.
375+ -- Ensure that we are actually constructing the type and using it. This
376+ -- is important, otherwise the structure is created and discarded, the
377+ -- cost of creation of the structure is not accounted. Otherwise we may
378+ -- just read the values and discard them. The comparison adds to the
379+ -- cost though. We could use deepseq but then we need to write
380+ -- instances of NFData and ensure that they are correct and perform
381+ -- well. Equality check also ensures correctness.
382+ {-
383+ if (val1 /= val)
384+ then error "peek: no match"
385+ else return ()
386+ -}
387+ val1 `deepseq` return ()
388+
389+{-# INLINE peekTimes #-}
390+peekTimes :: (NFData a, SERIALIZE_CLASS a) => Int -> a -> Int -> IO ()
391+peekTimes n val times = do
392+ arr <- MBA.new n
393+ _ <- SERIALIZE_OP 0 arr val
394+ loopWith times peek (val, n) arr
395+
396+{-# INLINE trip #-}
397+trip :: forall a. (NFData a, SERIALIZE_CLASS a) => a -> IO ()
398+trip val = do
399+ let n = getSize val
400+ arr <- MBA.new n
401+ _ <- SERIALIZE_OP 0 arr val
402+#ifdef USE_UNBOX
403+ (val1 :: a) <- DESERIALIZE_OP 0 arr
404+#else
405+ (_, val1 :: a) <- DESERIALIZE_OP 0 arr n
406+#endif
407+ -- Do not remove this or use deepseq, see the comments in peek.
408+ {-
409+ if (val1 /= val)
410+ then error "roundtrip: no match"
411+ else return ()
412+ -}
413+ -- Note: deepseq is not needed if the datatype is strict
414+ val1 `deepseq` return ()
415+
416+{-# INLINE roundtrip #-}
417+roundtrip :: (NFData a, SERIALIZE_CLASS a) => a -> Int -> IO ()
418+roundtrip val times = loop times trip val
419+
420+-------------------------------------------------------------------------------
421+-- Benchmarks
422+-------------------------------------------------------------------------------
423+
424+{-# INLINE benchConst #-}
425+benchConst ::
426+ String
427+ -> (forall a. (NFData a, SERIALIZE_CLASS a) =>
428+ Int -> a -> Int -> IO ())
429+ -> Int
430+ -> Benchmark
431+benchConst gname f times =
432+ bgroup gname
433+ [ let !n = getSize Unit
434+ in benchSink "Unit" times (f n Unit)
435+ {-
436+ , let !n = getSize CDT1C1
437+ in benchSink "C1" times (f n CDT1C1)
438+ , let val = CDT1C2 5
439+ !n = getSize val
440+ in benchSink "C2" (times `div` 2) (f n val)
441+ , let val = CDT1C3 5 2
442+ !n = getSize val
443+ in benchSink "C3" (times `div` 3) (f n val)
444+ -}
445+ , let !n = getSize Sum21
446+ in benchSink "Sum2" times (f n Sum21)
447+ , let !n = getSize Sum2525
448+ in benchSink "Sum25" times (f n Sum2525)
449+ , let val = Product25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
450+ !n = getSize val
451+ in benchSink "Product25" (times `div` 26) (f n val)
452+ ]
453+
454+#ifndef USE_UNBOX
455+{-# INLINE benchVar #-}
456+benchVar ::
457+ String
458+ -> (forall a. (NFData a, SERIALIZE_CLASS a) =>
459+ Int -> a -> Int -> IO ())
460+ -> BinTree Int
461+ -> [Int]
462+ -> Int
463+ -> Benchmark
464+benchVar gname f tInt lInt times =
465+ bgroup gname
466+ [ let !n = getSize tInt
467+ in benchSink "bintree-int" times (f n tInt)
468+ , let !n = getSize lInt
469+ in benchSink "list-int" times (f n lInt)
470+ ]
471+
472+{-
473+{-# INLINE benchGeneric #-}
474+benchGeneric
475+ :: (NFData b, Serialize b)
476+ => String
477+ -> String
478+ -> (forall a. (NFData a, Serialize a) =>
479+ Int -> a -> Int -> IO ())
480+ -> b
481+ -> Int
482+ -> Benchmark
483+benchGeneric gname valId f val times =
484+ bgroup gname
485+ [ let !n = getSize val
486+ in benchSink valId times (f n val)
487+ ]
488+-}
489+
490+{-# INLINE benchLargeRec #-}
491+benchLargeRec ::
492+ String
493+ -> (forall a. (NFData a, Serialize a) =>
494+ Int -> a -> Int -> IO ())
495+ -> RecNonCompatible
496+ -> RecCompatible
497+ -> Int
498+ -> Benchmark
499+benchLargeRec gname f recL recR times =
500+ bgroup gname
501+ [ let !n = getSize recL
502+ in benchSink "RecNonCompatible" times (f n recL)
503+ , let !n = getSize recR
504+ in benchSink "RecCompatible" times (f n recR)
505+ ]
506+#endif
507+
508+-- Times is scaled by the number of constructors to normalize
509+#ifdef USE_UNBOX
510+allBenchmarks :: Int -> [Benchmark]
511+allBenchmarks times =
512+#else
513+allBenchmarks ::
514+ BinTree Int -> [Int] -> RecNonCompatible -> RecCompatible -> Int -> [Benchmark]
515+allBenchmarks tInt lInt recL recR times =
516+#endif
517+ [ bgroup "sizeOf"
518+ [
519+#ifndef USE_UNBOX
520+ bench "bintree-int" $ nf getSize tInt
521+ , bench "list-int" $ nf getSize lInt
522+#endif
523+ ]
524+ , benchConst "poke" (const pokeTimes) times
525+ -- , benchConst "pokeWithSize" (const pokeTimesWithSize) times
526+ , benchConst "encode" (const encodeTimes) times
527+ , benchConst "peek" peekTimes times
528+ , benchConst "roundtrip" (const roundtrip) times
529+#ifndef USE_UNBOX
530+ , benchVar "poke" (const pokeTimes) tInt lInt 1
531+ -- , benchVar "pokeWithSize" (const pokeTimesWithSize) tInt lInt 1
532+ , benchVar "encode" (const encodeTimes) tInt lInt 1
533+ , benchVar "peek" peekTimes tInt lInt 1
534+ , benchVar "roundtrip" (const roundtrip) tInt lInt 1
535+ , benchLargeRec "poke" (const pokeTimes) recL recR times
536+ , benchLargeRec "encode" (const encodeTimes) recL recR times
537+ , benchLargeRec "peek" peekTimes recL recR times
538+ , benchLargeRec "roundtrip" (const roundtrip) recL recR times
539+#endif
540+ ]
541+
542+-------------------------------------------------------------------------------
543+-- Driver
544+-------------------------------------------------------------------------------
545+
546+main :: IO ()
547+main = do
548+#ifndef USE_UNBOX
549+ -- Approximately 100000 constructors
550+ -- Assuming Leaf nodes constitute two constructors (Leaf, Int) and internal
551+ -- nodes 1 level = log_2 (100001/3) + 1 = 16
552+ !(tInt :: BinTree Int) <- force <$> mkBinTree 16
553+
554+ let !recL = valRecNonCompatible
555+ let !recR = valRecCompatible
556+
557+ -- Approximately 100000 constructors, assuming two constructors (Cons, Int)
558+ -- per element.
559+ let lInt = [1..50000 :: Int]
560+ let !len = length lInt -- evaluate the list
561+#endif
562+#ifndef FUSION_CHECK
563+#ifdef USE_UNBOX
564+ runWithCLIOpts defaultStreamSize allBenchmarks
565+#else
566+ len
567+ `seq` runWithCLIOpts
568+ defaultStreamSize
569+ (allBenchmarks tInt lInt recL recR)
570+#endif
571+#else
572+ -- Enable FUSION_CHECK macro at the beginning of the file
573+ -- Enable one benchmark below, and run the benchmark
574+ -- Check the .dump-simpl output
575+ let value = 100000
576+ -- print $ getSize (CDT1C3 4 2)
577+ -- print $ getSize tInt
578+ -- print $ getSize lInt
579+
580+ -- pokeTimes tInt 1
581+
582+ -- peekTimes ((CDT1C2 (5 :: Int)) :: CustomDT1) value
583+ -- peekTimes (Sum2525) value
584+ -- peekTimes (Product25 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25) value
585+ -- peekTimes tInt 1
586+ let !n = getSize lInt
587+ peekTimes n lInt 1
588+
589+ -- roundtrip ((CDT1C2 (5 :: Int)) :: CustomDT1) value
590+ return ()
591+#endif
benchmark/Streamly/Benchmark/Data/Serialize/RecCompatible.hsnew file+24
@@ -0,0 +1,24 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Streamly.Benchmark.Data.Serialize.RecCompatible
4+ ( RecCompatible(..)
5+ , valRecCompatible
6+ ) where
7+
8+--------------------------------------------------------------------------------
9+-- Imports
10+--------------------------------------------------------------------------------
11+
12+import Streamly.Internal.Data.MutByteArray (Serialize)
13+import Streamly.Benchmark.Data.Serialize.TH (genLargeRecord)
14+
15+import qualified Streamly.Internal.Data.MutByteArray as Serialize
16+
17+--------------------------------------------------------------------------------
18+-- Code
19+--------------------------------------------------------------------------------
20+
21+$(genLargeRecord "RecCompatible" 50)
22+$(Serialize.deriveSerializeWith
23+ (Serialize.encodeRecordFields True)
24+ [d|instance Serialize RecCompatible|])
benchmark/Streamly/Benchmark/Data/Serialize/RecNonCompatible.hsnew file+21
@@ -0,0 +1,21 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Streamly.Benchmark.Data.Serialize.RecNonCompatible
4+ ( RecNonCompatible(..)
5+ , valRecNonCompatible
6+ ) where
7+
8+--------------------------------------------------------------------------------
9+-- Imports
10+--------------------------------------------------------------------------------
11+
12+import Streamly.Benchmark.Data.Serialize.TH (genLargeRecord)
13+
14+import qualified Streamly.Internal.Data.MutByteArray as Serialize
15+
16+--------------------------------------------------------------------------------
17+-- Code
18+--------------------------------------------------------------------------------
19+
20+$(genLargeRecord "RecNonCompatible" 50)
21+$(Serialize.deriveSerialize [d|instance Serialize.Serialize RecNonCompatible|])
benchmark/Streamly/Benchmark/Data/Serialize/TH.hsnew file+76
@@ -0,0 +1,76 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Streamly.Benchmark.Data.Serialize.TH (genLargeRecord) where
4+
5+--------------------------------------------------------------------------------
6+-- Imports
7+--------------------------------------------------------------------------------
8+
9+import Language.Haskell.TH
10+
11+import Streamly.Internal.Data.MutByteArray (makeI)
12+import Control.DeepSeq (NFData(..))
13+
14+--------------------------------------------------------------------------------
15+-- Large Record
16+--------------------------------------------------------------------------------
17+
18+genLargeRecord :: String -> Int -> Q [Dec]
19+genLargeRecord tyName numFields =
20+ sequence
21+ [ dataD
22+ (pure [])
23+ (mkName tyName)
24+ []
25+ Nothing
26+ [mkCon tyName]
27+ [derivClause Nothing [conT ''Eq, conT ''Show]]
28+ , mkValueSigDec
29+ , mkValueDec
30+ , nfDataInstance tyName
31+ ]
32+
33+ where
34+
35+ fieldTypeChoices = [conT ''()]
36+ chooseCycle i xs = xs !! (i `mod` length xs)
37+ nfDataInstance nm =
38+ instanceD
39+ (pure [])
40+ (appT (conT ''NFData) (conT (mkName nm)))
41+ [ funD
42+ 'rnf
43+ [ clause
44+ [ conP
45+ (mkName nm)
46+ (varP . makeI <$> [0 .. (numFields - 1)])
47+ ]
48+ (normalB
49+ (foldl
50+ (\b a -> [|rnf $(b) `seq` rnf $(a)|])
51+ [|()|]
52+ (varE . makeI <$> [0 .. (numFields - 1)])))
53+ []
54+ ]
55+ ]
56+ valueName = mkName $ "val" ++ tyName
57+ mkValueSigDec = sigD valueName [t|$(conT (mkName tyName))|]
58+ mkValueDec =
59+ funD
60+ valueName
61+ [ clause
62+ []
63+ (normalB
64+ (foldl
65+ (\b a -> [|$(b) $(a)|])
66+ (conE (mkName tyName))
67+ (conE '() <$ [0 .. (numFields - 1)])))
68+ []
69+ ]
70+ mkCon nm = recC (mkName nm) (mkField <$> [0 .. (numFields - 1)])
71+ mkField i =
72+ varBangType
73+ (mkName ("field" ++ show i))
74+ (bangType
75+ (bang noSourceUnpackedness noSourceStrictness)
76+ (chooseCycle i fieldTypeChoices))
benchmark/Streamly/Benchmark/Data/Stream/Common.hs+3−1
@@ -71,7 +71,9 @@ module Stream.Common
7171 )
7272where
7373
74+#if !MIN_VERSION_base(4,18,0)
7475import Control.Applicative (liftA2)
76+#endif
7577import Control.DeepSeq (NFData)
7678import Control.Exception (try)
7779import GHC.Exception (ErrorCall)
@@ -102,7 +104,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
102104#endif
103105#endif
104106
105−import Gauge
107+import Test.Tasty.Bench
106108import Prelude hiding (Foldable(..), mapM, replicate)
107109
108110#ifdef USE_STREAMK
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentCommon.hs+3−3
@@ -14,14 +14,14 @@ where
1414import Stream.Common
1515 (composeN, benchIO, benchIOSink, benchIOSrc, sourceUnfoldrM)
1616import Streamly.Data.Stream (Stream)
17−import Streamly.Internal.Data.Stream.Concurrent (MonadAsync, Config)
17+import Streamly.Internal.Data.Stream.Prelude (MonadAsync, Config)
1818
1919import qualified Data.List as List
2020import qualified Streamly.Data.Fold as Fold
2121import qualified Streamly.Data.Stream as Stream
22−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
22+import qualified Streamly.Internal.Data.Stream.Prelude as Async
2323
24−import Gauge
24+import Test.Tasty.Bench
2525import Prelude hiding (mapM)
2626import Streamly.Benchmark.Common
2727
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentEager.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
12+import qualified Streamly.Internal.Data.Stream.Prelude as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentEager"
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentInterleaved.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
12+import qualified Streamly.Internal.Data.Stream.Prelude as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentInterleaved"
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentOrdered.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
12+import qualified Streamly.Internal.Data.Stream.Prelude as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentOrdered"
benchmark/Streamly/Benchmark/Data/Stream/Eliminate.hs+3−1
@@ -57,7 +57,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
5757#endif
5858#endif
5959
60−import Gauge
60+import Test.Tasty.Bench
6161import Streamly.Benchmark.Common
6262import Prelude hiding (length, sum, or, and, any, all, notElem, elem, (!!),
6363 lookup, repeat, minimum, maximum, product, last, mapM_, init)
@@ -132,12 +132,14 @@ foldableAny :: Int -> Int -> Bool
132132foldableAny value n =
133133 Prelude.any (> (value + 1)) (sourceUnfoldr value n :: Stream Identity Int)
134134
135+{- HLINT ignore "Use all"-}
135136{-# INLINE foldableAnd #-}
136137foldableAnd :: Int -> Int -> Bool
137138foldableAnd value n =
138139 Prelude.and $ fmap
139140 (<= (value + 1)) (sourceUnfoldr value n :: Stream Identity Int)
140141
142+{- HLINT ignore "Use any"-}
141143{-# INLINE foldableOr #-}
142144foldableOr :: Int -> Int -> Bool
143145foldableOr value n =
benchmark/Streamly/Benchmark/Data/Stream/Exceptions.hs+12−3
@@ -36,21 +36,22 @@ import System.IO (Handle, hClose, hPutChar)
3636import qualified Streamly.FileSystem.Handle as FH
3737import qualified Streamly.Internal.FileSystem.Handle as IFH
3838import qualified Streamly.Internal.Data.Unfold as IUF
39−import qualified Streamly.Internal.Data.Unfold.Exception as IUF
39+import qualified Streamly.Internal.Data.Unfold.Prelude as IUF
4040
4141#ifdef USE_PRELUDE
4242import qualified Streamly.Internal.Data.Stream.IsStream as Stream
4343#else
4444import qualified Streamly.Internal.Data.Stream as Stream
45−import qualified Streamly.Internal.Data.Stream.Exception.Lifted as Stream
45+import qualified Streamly.Internal.Data.Stream.Prelude as Stream
4646#endif
4747
48−import Gauge hiding (env)
48+import Test.Tasty.Bench hiding (env)
4949import Prelude hiding (last, length)
5050import Streamly.Benchmark.Common
5151import Streamly.Benchmark.Common.Handle
5252
5353#ifdef INSPECTION
54+import Control.Monad.Catch (MonadCatch)
5455import Test.Inspection
5556
5657import qualified Streamly.Internal.Data.Stream as D
@@ -286,8 +287,12 @@ readChunksOnException inh devNull =
286287 in IUF.fold (IFH.writeChunks devNull) readEx inh
287288
288289#ifdef INSPECTION
290+#if __GLASGOW_HASKELL__ >= 906
291+inspect $ hasNoTypeClassesExcept 'readChunksOnException [''MonadCatch]
292+#else
289293inspect $ hasNoTypeClasses 'readChunksOnException
290294#endif
295+#endif
291296
292297-- | Send the file contents to /dev/null with exception handling
293298readChunksBracket_ :: Handle -> Handle -> IO ()
@@ -296,8 +301,12 @@ readChunksBracket_ inh devNull =
296301 in IUF.fold (IFH.writeChunks devNull) readEx inh
297302
298303#ifdef INSPECTION
304+#if __GLASGOW_HASKELL__ >= 906
305+inspect $ hasNoTypeClassesExcept 'readChunksBracket_ [''MonadCatch]
306+#else
299307inspect $ hasNoTypeClasses 'readChunksBracket_
300308#endif
309+#endif
301310
302311readChunksBracket :: Handle -> Handle -> IO ()
303312readChunksBracket inh devNull =
benchmark/Streamly/Benchmark/Data/Stream/Expand.hs+9−1
@@ -54,7 +54,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
5454#endif
5555#endif
5656
57−import Gauge
57+import Test.Tasty.Bench
5858import Stream.Common
5959import Streamly.Benchmark.Common
6060import Prelude hiding (concatMap)
@@ -198,7 +198,11 @@ concatMapPure outer inner n =
198198 (fromStream $ sourceUnfoldr outer n)
199199
200200#ifdef INSPECTION
201+#if __GLASGOW_HASKELL__ >= 906
202+inspect $ hasNoTypeClassesExcept 'concatMapPure [''Applicative]
203+#else
201204inspect $ hasNoTypeClasses 'concatMapPure
205+#endif
202206inspect $ 'concatMapPure `hasNoType` ''SPEC
203207#endif
204208
@@ -211,7 +215,11 @@ concatMapRepl outer inner n =
211215 (fromStream . S.replicate inner) (fromStream $ sourceUnfoldrM outer n)
212216
213217#ifdef INSPECTION
218+#if __GLASGOW_HASKELL__ >= 906
219+inspect $ hasNoTypeClassesExcept 'concatMapRepl [''Applicative]
220+#else
214221inspect $ hasNoTypeClasses 'concatMapRepl
222+#endif
215223inspect $ 'concatMapRepl `hasNoType` ''SPEC
216224#endif
217225
benchmark/Streamly/Benchmark/Data/Stream/Generate.hs+1−1
@@ -40,7 +40,7 @@ import qualified Stream.Common as Common
4040#endif
4141#endif
4242
43−import Gauge
43+import Test.Tasty.Bench
4444import Streamly.Benchmark.Common
4545import qualified Prelude
4646
benchmark/Streamly/Benchmark/Data/Stream/Lift.hs+1−1
@@ -37,7 +37,7 @@ import qualified Control.Monad.State.Strict as State
3737#endif
3838#endif
3939
40−import Gauge
40+import Test.Tasty.Bench
4141import Streamly.Benchmark.Common
4242import Prelude hiding (reverse, tail)
4343
benchmark/Streamly/Benchmark/Data/Stream/Reduce.hs+2−2
@@ -48,7 +48,7 @@ import Streamly.Internal.Data.Stream (Stream)
4848import qualified Streamly.Internal.Data.Stream as S
4949#ifndef USE_STREAMLY_CORE
5050import qualified Streamly.Data.Stream.Prelude as S
51−import qualified Streamly.Internal.Data.Stream.Time as S
51+import qualified Streamly.Internal.Data.Stream.Prelude as S
5252#endif
5353
5454#ifdef USE_STREAMK
@@ -61,7 +61,7 @@ import qualified Streamly.Internal.Data.Stream as K
6161
6262#endif
6363
64−import Gauge
64+import Test.Tasty.Bench
6565import Streamly.Benchmark.Common
6666import Stream.Common
6767import Prelude hiding (reverse, tail)
benchmark/Streamly/Benchmark/Data/Stream/Split.hs+2−2
@@ -35,7 +35,7 @@ import qualified Streamly.Internal.FileSystem.Handle as IFH
3535import qualified Streamly.Internal.Unicode.Stream as IUS
3636import qualified Streamly.Prelude as S
3737
38−import Gauge hiding (env)
38+import Test.Tasty.Bench hiding (env)
3939import Prelude hiding (last, length)
4040import Streamly.Benchmark.Common
4141import Streamly.Benchmark.Common.Handle
@@ -43,7 +43,7 @@ import Streamly.Benchmark.Common.Handle
4343#ifdef INSPECTION
4444import Streamly.Internal.Data.Stream (Step(..))
4545
46−import qualified Streamly.Internal.Data.MutArray.Type as MA
46+import qualified Streamly.Internal.Data.MutArray as MA
4747import qualified Streamly.Internal.Data.Unfold as IUF
4848
4949import Test.Inspection
benchmark/Streamly/Benchmark/Data/Stream/StreamD.hs+4−2
@@ -22,10 +22,12 @@
2222
2323module Main (main) where
2424
25+#if !MIN_VERSION_base(4,18,0)
2526import Control.Applicative (liftA2)
27+#endif
2628import Control.Monad (when)
2729import Data.Maybe (isJust)
28−import Gauge (bench, nfIO, bgroup, Benchmark, defaultMain)
30+import Test.Tasty.Bench (bench, nfIO, bgroup, Benchmark, defaultMain)
2931import System.Random (randomRIO)
3032import Prelude hiding (tail, mapM_, foldl, last, map, mapM, concatMap, zip)
3133
@@ -461,7 +463,7 @@ inspect $ 'unfoldManyRepl `hasNoType` ''SPEC
461463-------------------------------------------------------------------------------
462464
463465moduleName :: String
464−moduleName = "Data.Stream.StreamD"
466+moduleName = "Data.StreamD"
465467
466468o_1_space :: [Benchmark]
467469o_1_space =
benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs+4−3
@@ -21,7 +21,9 @@
2121
2222module Main (main) where
2323
24+#if !MIN_VERSION_base(4,18,0)
2425import Control.Applicative (liftA2)
26+#endif
2527import Control.Monad (when)
2628import Data.Maybe (isJust)
2729import System.Random (randomRIO)
@@ -33,10 +35,9 @@ import Prelude hiding
3335import qualified Prelude as P
3436import qualified Data.List as List
3537
36−import qualified Streamly.Internal.Data.Stream.StreamK.Type as S
3738import qualified Streamly.Internal.Data.StreamK as S
3839
39−import Gauge (bench, nfIO, bgroup, Benchmark, defaultMain)
40+import Test.Tasty.Bench (bench, nfIO, bgroup, Benchmark, defaultMain)
4041
4142import Streamly.Benchmark.Common
4243
@@ -633,7 +634,7 @@ filterAllInNestedList str = do
633634-------------------------------------------------------------------------------
634635
635636moduleName :: String
636−moduleName = "Data.Stream.StreamK"
637+moduleName = "Data.StreamK"
637638
638639o_1_space_generation :: Int -> Benchmark
639640o_1_space_generation streamLen =
benchmark/Streamly/Benchmark/Data/Stream/StreamKAlt.hs+3−3
@@ -18,12 +18,12 @@ import Prelude hiding ()
1818-- import qualified Prelude as P
1919-- import qualified Data.List as List
2020
21−import qualified Streamly.Internal.Data.Stream.StreamK.Alt as S
21+import qualified Streamly.Internal.Data.StreamK.Alt as S
2222-- import qualified Streamly.Internal.Data.Stream.Common as SP
2323-- import qualified Streamly.Internal.Data.SVar as S
2424
2525import Streamly.Benchmark.Common
26−import Gauge (bgroup, Benchmark, defaultMain)
26+import Test.Tasty.Bench (bgroup, Benchmark, defaultMain)
2727
2828value :: Int
2929value = 100000
@@ -452,4 +452,4 @@ o_1_space =
452452 ]
453453
454454main :: IO ()
455−main = defaultMain $ concat [o_1_space]
455+main = defaultMain [o_1_space]
benchmark/Streamly/Benchmark/Data/Stream/ToStreamK.hs+3−4
@@ -1,5 +1,5 @@
11-- |
2−-- Module : Streamly.Benchmark.Data.Stream.StreamK
2+-- Module : Streamly.Benchmark.Data.StreamK
33-- Copyright : (c) 2018 Composewell Technologies
44--
55-- License : BSD3
@@ -33,12 +33,11 @@ import Prelude hiding
3333import qualified Prelude as P
3434-- import qualified Data.List as List
3535
36−import qualified Streamly.Internal.Data.Stream.StreamK.Type as S
3736import qualified Streamly.Internal.Data.StreamK as S
3837import qualified Streamly.Internal.Data.Stream as D
3938import qualified Streamly.Internal.Data.Fold as Fold
4039
41−import Gauge (bgroup, Benchmark, defaultMain)
40+import Test.Tasty.Bench (bgroup, Benchmark, defaultMain)
4241
4342import Streamly.Benchmark.Common
4443
@@ -121,7 +120,7 @@ fromFoldable streamLen n = S.fromFoldable [n..n+streamLen]
121120{-# INLINE fromFoldableM #-}
122121fromFoldableM :: Monad m => Int -> Int -> Stream m Int
123122fromFoldableM streamLen n =
124− Prelude.foldr S.consM S.nil (Prelude.fmap return [n..n+streamLen])
123+ Prelude.foldr (S.consM . return) S.nil [n .. n + streamLen]
125124
126125{-
127126{-# INLINABLE concatMapFoldableWith #-}
benchmark/Streamly/Benchmark/Data/Stream/Transform.hs+2−2
@@ -46,7 +46,7 @@ import Streamly.Internal.Data.Time.Units
4646import Streamly.Internal.Data.Stream (Stream)
4747import qualified Streamly.Internal.Data.Stream as Stream
4848#ifndef USE_STREAMLY_CORE
49−import qualified Streamly.Internal.Data.Stream.Time as Stream
49+import qualified Streamly.Internal.Data.Stream.Prelude as Stream
5050#endif
5151#ifdef USE_STREAMK
5252import Control.DeepSeq (NFData(..))
@@ -58,7 +58,7 @@ import qualified Streamly.Internal.Data.StreamK as StreamK
5858#endif
5959#endif
6060
61−import Gauge
61+import Test.Tasty.Bench
6262import Stream.Common hiding (scanl')
6363import Streamly.Benchmark.Common
6464import Prelude hiding (sequence, mapM)
benchmark/Streamly/Benchmark/Data/Unbox.hsdeleted−132
@@ -1,132 +0,0 @@
1−{-# LANGUAGE TemplateHaskell #-}
2−
3−#undef FUSION_CHECK
4−#ifdef FUSION_CHECK
5−{-# OPTIONS_GHC -ddump-simpl -ddump-to-file -dsuppress-all #-}
6−#endif
7−
8−
9−-- |
10−-- Module : Streamly.Benchmark.Data.Unbox
11−-- Copyright : (c) 2023 Composewell
12−-- License : BSD-3-Clause
13−-- Maintainer : streamly@composewell.com
14−
15−module Main (main) where
16−
17−-------------------------------------------------------------------------------
18−-- Imports
19−-------------------------------------------------------------------------------
20−
21−import Control.DeepSeq (NFData(..))
22−import Control.Exception (assert)
23−import Control.Monad (replicateM_)
24−import Data.Proxy (Proxy(..))
25−import GHC.Generics (Generic)
26−import System.Random (randomRIO)
27−
28−import Streamly.Internal.Data.Unbox
29−#ifdef USE_TH
30−import Streamly.Internal.Data.Unbox.TH
31−#endif
32−
33−import Gauge
34−import Streamly.Benchmark.Common
35−
36−-------------------------------------------------------------------------------
37−-- Types
38−-------------------------------------------------------------------------------
39−
40−data CustomDT1
41− = CDT1C1
42− | CDT1C2 Int
43− | CDT1C3 Int Bool
44− deriving (Generic, Show, Eq)
45−
46−#ifndef USE_TH
47−instance Unbox CustomDT1
48−#else
49−$(deriveUnbox ''CustomDT1)
50−#endif
51−
52−-------------------------------------------------------------------------------
53−-- Helpers
54−-------------------------------------------------------------------------------
55−
56−{-# INLINE pokeTimes #-}
57−pokeTimes :: forall a. Unbox a => a -> Int -> IO ()
58−pokeTimes val times = do
59− arr <- newBytes (sizeOf (Proxy :: Proxy a))
60− replicateM_ times $ do
61− pokeByteIndex 0 arr val
62−
63−{-# INLINE peekTimes #-}
64−peekTimes :: forall a. Unbox a => a -> Int -> IO ()
65−peekTimes val times = do
66− arr <- newBytes (sizeOf (Proxy :: Proxy a))
67− pokeByteIndex 0 arr val
68− replicateM_ times $ do
69− (_ :: a) <- peekByteIndex 0 arr
70− return ()
71−
72−{-# INLINE roundtrip #-}
73−roundtrip :: forall a. (Eq a, Unbox a) => a -> Int -> IO ()
74−roundtrip val times = do
75− arr <- newBytes (sizeOf (Proxy :: Proxy a))
76− replicateM_ times $ do
77− pokeByteIndex 0 arr val
78− val1 <- peekByteIndex 0 arr
79− assert (val == val1) (pure ())
80−
81−benchSink :: NFData b => String -> Int -> (Int -> IO b) -> Benchmark
82−benchSink name times f = bench name (nfIO (randomRIO (times, times) >>= f))
83−
84−-------------------------------------------------------------------------------
85−-- Benchmarks
86−-------------------------------------------------------------------------------
87−
88−allBenchmarks :: Int -> [Benchmark]
89−allBenchmarks times =
90− [ bgroup "poke"
91− [ benchSink "C1" times
92− (pokeTimes (CDT1C1 :: CustomDT1))
93− , benchSink "C2" times
94− (pokeTimes ((CDT1C2 (5 :: Int)) :: CustomDT1))
95− , benchSink "C3" times
96− (pokeTimes ((CDT1C3 (5 :: Int) True) :: CustomDT1))
97− ]
98− , bgroup "peek"
99− [ benchSink "C1" times
100− (peekTimes (CDT1C1 :: CustomDT1))
101− , benchSink "C2" times
102− (peekTimes ((CDT1C2 (5 :: Int)) :: CustomDT1))
103− , benchSink "C3" times
104− (peekTimes ((CDT1C3 (5 :: Int) True) :: CustomDT1))
105− ]
106− , bgroup "roundtrip"
107− [ benchSink "C1" times
108− (roundtrip (CDT1C1 :: CustomDT1))
109− , benchSink "C2" times
110− (roundtrip ((CDT1C2 (5 :: Int)) :: CustomDT1))
111− , benchSink "C3" times
112− (roundtrip ((CDT1C3 (5 :: Int) True) :: CustomDT1))
113− ]
114− ]
115−
116−-------------------------------------------------------------------------------
117−-- Driver
118−-------------------------------------------------------------------------------
119−
120−main :: IO ()
121−main = do
122−#ifndef FUSION_CHECK
123− runWithCLIOpts defaultStreamSize allBenchmarks
124−#else
125− -- Enable FUSION_CHECK macro at the beginning of the file
126− -- Enable one benchmark below, and run the benchmark
127− -- Check the .dump-simpl output
128− let value = 100000
129− -- peekTimes ((CDT1C2 (5 :: Int)) :: CustomDT1) value
130− roundtrip ((CDT1C2 (5 :: Int)) :: CustomDT1) value
131− return ()
132−#endif
benchmark/Streamly/Benchmark/Data/Unfold.hs+11−2
@@ -31,17 +31,18 @@ import qualified Prelude
3131import qualified Streamly.FileSystem.Handle as FH
3232import qualified Streamly.Internal.Data.Fold as FL
3333import qualified Streamly.Internal.Data.Unfold as UF
34−import qualified Streamly.Internal.Data.Unfold.Exception as UF
34+import qualified Streamly.Internal.Data.Unfold.Prelude as UF
3535import qualified Streamly.Internal.Data.Stream as S
3636import qualified Streamly.Internal.Data.Stream as D
3737import qualified Streamly.Internal.Data.StreamK as K
3838
39−import Gauge hiding (env)
39+import Test.Tasty.Bench hiding (env)
4040import Prelude hiding (take, filter, zipWith, map, mapM, takeWhile)
4141import Streamly.Benchmark.Common
4242import Streamly.Benchmark.Common.Handle
4343
4444#ifdef INSPECTION
45+import Control.Monad.Catch (MonadCatch)
4546import Test.Inspection
4647#endif
4748
@@ -776,7 +777,11 @@ readWriteFinally_Unfold inh devNull =
776777 in S.fold (FH.write devNull) $ S.unfold readEx inh
777778
778779#ifdef INSPECTION
780+#if __GLASGOW_HASKELL__ >= 906
781+inspect $ hasNoTypeClassesExcept 'readWriteFinally_Unfold [''MonadCatch]
782+#else
779783inspect $ hasNoTypeClasses 'readWriteFinally_Unfold
784+#endif
780785-- inspect $ 'readWriteFinallyUnfold `hasNoType` ''Step
781786#endif
782787
@@ -792,7 +797,11 @@ readWriteBracket_Unfold inh devNull =
792797 in S.fold (FH.write devNull) $ S.unfold readEx inh
793798
794799#ifdef INSPECTION
800+#if __GLASGOW_HASKELL__ >= 906
801+inspect $ hasNoTypeClassesExcept 'readWriteBracket_Unfold [''MonadCatch]
802+#else
795803inspect $ hasNoTypeClasses 'readWriteBracket_Unfold
804+#endif
796805-- inspect $ 'readWriteBracketUnfold `hasNoType` ''Step
797806#endif
798807
benchmark/Streamly/Benchmark/FileSystem/Handle.hs+1−1
@@ -22,7 +22,7 @@ import Streamly.Benchmark.Common.Handle (mkHandleBenchEnv)
2222import qualified Handle.ReadWrite as RW
2323import qualified Handle.Read as RO
2424
25−import Gauge hiding (env)
25+import Test.Tasty.Bench hiding (env)
2626import Prelude hiding (last, length)
2727import Streamly.Benchmark.Common
2828
benchmark/Streamly/Benchmark/FileSystem/Handle/Read.hs+8−9
@@ -33,7 +33,6 @@ import qualified Streamly.Data.Stream as Stream
3333import qualified Streamly.Data.Fold as Fold
3434import qualified Streamly.FileSystem.Handle as FH
3535import qualified Streamly.Internal.Data.Array as A
36−import qualified Streamly.Internal.Data.Array.Type as AT
3736import qualified Streamly.Internal.Data.Fold as FL
3837import qualified Streamly.Internal.Data.Stream as IP
3938import qualified Streamly.Internal.FileSystem.Handle as IFH
@@ -41,14 +40,14 @@ import qualified Streamly.Internal.Unicode.Stream as IUS
4140import qualified Streamly.Data.Stream.Prelude as S
4241import qualified Streamly.Unicode.Stream as SS
4342
44−import Gauge hiding (env)
43+import Test.Tasty.Bench hiding (env)
4544import Prelude hiding (last, length)
4645import Streamly.Benchmark.Common.Handle
4746
4847#ifdef INSPECTION
4948import Streamly.Internal.Data.Stream (Step(..), FoldMany)
5049
51−import qualified Streamly.Internal.Data.MutArray.Type as MA
50+import qualified Streamly.Internal.Data.MutArray as MutArray
5251import qualified Streamly.Internal.Data.Stream as D
5352import qualified Streamly.Internal.Data.Unfold as IUF
5453
@@ -69,7 +68,7 @@ readLast = S.fold Fold.last . S.unfold FH.reader
6968inspect $ hasNoTypeClasses 'readLast
7069inspect $ 'readLast `hasNoType` ''Step -- S.unfold
7170inspect $ 'readLast `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
72−inspect $ 'readLast `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
71+inspect $ 'readLast `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
7372#endif
7473
7574-- assert that flattenArrays constructors are not present
@@ -81,7 +80,7 @@ readCountBytes = S.fold Fold.length . S.unfold FH.reader
8180inspect $ hasNoTypeClasses 'readCountBytes
8281inspect $ 'readCountBytes `hasNoType` ''Step -- S.unfold
8382inspect $ 'readCountBytes `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
84−inspect $ 'readCountBytes `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
83+inspect $ 'readCountBytes `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
8584#endif
8685
8786-- | Count the number of lines in a file.
@@ -96,7 +95,7 @@ readCountLines =
9695inspect $ hasNoTypeClasses 'readCountLines
9796inspect $ 'readCountLines `hasNoType` ''Step
9897inspect $ 'readCountLines `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
99−inspect $ 'readCountLines `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
98+inspect $ 'readCountLines `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
10099#endif
101100
102101-- | Count the number of words in a file.
@@ -120,7 +119,7 @@ readSumBytes = S.fold Fold.sum . S.unfold FH.reader
120119inspect $ hasNoTypeClasses 'readSumBytes
121120inspect $ 'readSumBytes `hasNoType` ''Step
122121inspect $ 'readSumBytes `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
123−inspect $ 'readSumBytes `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
122+inspect $ 'readSumBytes `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
124123#endif
125124
126125-- XXX When we mark this with INLINE and we have two benchmarks using S.drain
@@ -236,13 +235,13 @@ groupsOf :: Int -> Handle -> IO Int
236235groupsOf n inh =
237236 -- writeNUnsafe gives 2.5x boost here over writeN.
238237 S.fold Fold.length
239− $ IP.groupsOf n (AT.writeNUnsafe n) (S.unfold FH.reader inh)
238+ $ IP.groupsOf n (A.writeNUnsafe n) (S.unfold FH.reader inh)
240239
241240#ifdef INSPECTION
242241inspect $ hasNoTypeClasses 'groupsOf
243242inspect $ 'groupsOf `hasNoType` ''Step
244243inspect $ 'groupsOf `hasNoType` ''FoldMany
245−inspect $ 'groupsOf `hasNoType` ''AT.ArrayUnsafe -- AT.writeNUnsafe
244+inspect $ 'groupsOf `hasNoType` ''MutArray.ArrayUnsafe -- AT.writeNUnsafe
246245 -- FH.read/A.read
247246inspect $ 'groupsOf `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
248247#endif
benchmark/Streamly/Benchmark/FileSystem/Handle/ReadWrite.hs+9−10
@@ -34,7 +34,7 @@ import qualified Streamly.Internal.FileSystem.Handle as IFH
3434import qualified Streamly.Data.Array as A
3535import qualified Streamly.Data.Stream.Prelude as S
3636
37−import Gauge hiding (env)
37+import Test.Tasty.Bench hiding (env)
3838import Streamly.Benchmark.Common.Handle
3939
4040#ifdef INSPECTION
@@ -42,8 +42,7 @@ import Streamly.Internal.Data.Stream (Step(..))
4242
4343import qualified Streamly.Internal.Data.Stream as D
4444import qualified Streamly.Internal.Data.Tuple.Strict as Strict
45−import qualified Streamly.Internal.Data.Stream.MutChunked as MAS
46−import qualified Streamly.Internal.Data.Array.Type as AT
45+import qualified Streamly.Internal.Data.MutArray as MutArray
4746
4847import Test.Inspection
4948#endif
@@ -83,7 +82,7 @@ copyStream inh outh = S.fold (FH.write outh) (S.unfold FH.reader inh)
8382inspect $ hasNoTypeClasses 'copyStream
8483inspect $ 'copyStream `hasNoType` ''Step -- S.unfold
8584inspect $ 'copyStream `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
86−inspect $ 'copyStream `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
85+inspect $ 'copyStream `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
8786 -- FH.read/A.read
8887inspect $ 'copyStream `hasNoType` ''Strict.Tuple3' -- FH.write/chunksOf
8988#endif
@@ -109,8 +108,8 @@ readFromBytesNull inh devNull = IFH.putBytes devNull $ S.unfold FH.reader inh
109108#ifdef INSPECTION
110109inspect $ hasNoTypeClasses 'readFromBytesNull
111110inspect $ 'readFromBytesNull `hasNoType` ''Step
112−inspect $ 'readFromBytesNull `hasNoType` ''MAS.SpliceState
113−inspect $ 'readFromBytesNull `hasNoType` ''AT.ArrayUnsafe -- FH.fromBytes/S.chunksOf
111+inspect $ 'readFromBytesNull `hasNoType` ''MutArray.SpliceState
112+inspect $ 'readFromBytesNull `hasNoType` ''MutArray.ArrayUnsafe -- FH.fromBytes/S.chunksOf
114113inspect $ 'readFromBytesNull `hasNoType` ''D.FoldMany
115114#endif
116115
@@ -123,8 +122,8 @@ readWithFromBytesNull inh devNull =
123122#ifdef INSPECTION
124123inspect $ hasNoTypeClasses 'readWithFromBytesNull
125124inspect $ 'readWithFromBytesNull `hasNoType` ''Step
126−inspect $ 'readWithFromBytesNull `hasNoType` ''MAS.SpliceState
127−inspect $ 'readWithFromBytesNull `hasNoType` ''AT.ArrayUnsafe -- FH.fromBytes/S.chunksOf
125+inspect $ 'readWithFromBytesNull `hasNoType` ''MutArray.SpliceState
126+inspect $ 'readWithFromBytesNull `hasNoType` ''MutArray.ArrayUnsafe -- FH.fromBytes/S.chunksOf
128127inspect $ 'readWithFromBytesNull `hasNoType` ''D.FoldMany
129128#endif
130129
@@ -171,7 +170,7 @@ writeReadWith inh devNull = IUF.fold fld unf (defaultChunkSize, inh)
171170inspect $ hasNoTypeClasses 'writeReadWith
172171inspect $ 'writeReadWith `hasNoType` ''Step
173172inspect $ 'writeReadWith `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
174−inspect $ 'writeReadWith `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
173+inspect $ 'writeReadWith `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
175174 -- FH.read/A.read
176175#endif
177176
@@ -188,7 +187,7 @@ writeRead inh devNull = IUF.fold fld unf inh
188187inspect $ hasNoTypeClasses 'writeRead
189188inspect $ 'writeRead `hasNoType` ''Step
190189inspect $ 'writeRead `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
191−inspect $ 'writeRead `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
190+inspect $ 'writeRead `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
192191 -- FH.read/A.read
193192#endif
194193
benchmark/Streamly/Benchmark/Prelude/Adaptive.hs+1−1
@@ -10,7 +10,7 @@
1010import Control.Concurrent (threadDelay)
1111import Control.Monad (when)
1212import Control.Monad.IO.Class (liftIO)
13−import Gauge
13+import Test.Tasty.Bench
1414import Streamly.Prelude as S
1515import System.Random (randomRIO)
1616
benchmark/Streamly/Benchmark/Prelude/Ahead.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Gauge
18+import Test.Tasty.Bench
1919
2020moduleName :: String
2121moduleName = "Prelude.Ahead"
benchmark/Streamly/Benchmark/Prelude/Async.hs+1−1
@@ -16,7 +16,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream.Transform as Transform
1616import Streamly.Benchmark.Common
1717import Streamly.Benchmark.Prelude
1818
19−import Gauge
19+import Test.Tasty.Bench
2020
2121moduleName :: String
2222moduleName = "Prelude.Async"
benchmark/Streamly/Benchmark/Prelude/Concurrent.hs+1−1
@@ -15,7 +15,7 @@ import Streamly.Prelude
1515 , fromWAsync, parallel, fromParallel
1616 )
1717
18−import Gauge
18+import Test.Tasty.Bench
1919import qualified Streamly.Prelude as S
2020
2121-------------------------------------------------------------------------------
benchmark/Streamly/Benchmark/Prelude/Merge.hs+1−1
@@ -30,7 +30,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream as Stream
3030import Streamly.Benchmark.Common
3131import Streamly.Benchmark.Prelude
3232
33−import Gauge
33+import Test.Tasty.Bench
3434
3535#ifdef INSPECTION
3636import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Prelude/Parallel.hs+1−1
@@ -22,7 +22,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream as Internal
2222import Streamly.Benchmark.Common
2323import Streamly.Benchmark.Prelude
2424
25−import Gauge
25+import Test.Tasty.Bench
2626
2727moduleName :: String
2828moduleName = "Prelude.Parallel"
benchmark/Streamly/Benchmark/Prelude/Rate.hs+1−1
@@ -14,7 +14,7 @@ import qualified Streamly.Prelude as S
1414import Streamly.Benchmark.Common
1515import Streamly.Benchmark.Prelude
1616
17−import Gauge
17+import Test.Tasty.Bench
1818
1919moduleName :: String
2020moduleName = "Prelude.Rate"
benchmark/Streamly/Benchmark/Prelude/WAsync.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Gauge
18+import Test.Tasty.Bench
1919
2020moduleName :: String
2121moduleName = "Prelude.WAsync"
benchmark/Streamly/Benchmark/Prelude/WSerial.hs+1−1
@@ -29,7 +29,7 @@ import qualified Streamly.Internal.Data.Unfold as UF
2929import Streamly.Benchmark.Common
3030import Streamly.Benchmark.Prelude
3131
32−import Gauge
32+import Test.Tasty.Bench
3333
3434#ifdef INSPECTION
3535import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Prelude/ZipAsync.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Gauge
18+import Test.Tasty.Bench
1919
2020moduleName :: String
2121moduleName = "Prelude.ZipAsync"
benchmark/Streamly/Benchmark/Prelude/ZipSerial.hs+1−1
@@ -29,7 +29,7 @@ import qualified Streamly.Prelude as S
2929import Streamly.Benchmark.Common
3030import Streamly.Benchmark.Prelude hiding (sourceUnfoldrM)
3131
32−import Gauge
32+import Test.Tasty.Bench
3333
3434#ifdef INSPECTION
3535import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Unicode/Char.hs+1−1
@@ -16,7 +16,7 @@ import Control.DeepSeq (NFData(..))
1616import Streamly.Internal.Data.Array (Array)
1717import System.FilePath (dropExtensions, takeFileName)
1818import System.FilePath.Posix ((</>))
19−import Gauge.Main (Benchmark, bench, bgroup, defaultMain, env, nfIO)
19+import Test.Tasty.Bench (Benchmark, bench, bgroup, defaultMain, env, nfIO)
2020import Streamly.Internal.Unicode.Char
2121 ( NormalizationMode(NFC, NFD, NFKC, NFKD)
2222 , normalize
benchmark/Streamly/Benchmark/Unicode/Parser.hs+1−1
@@ -31,7 +31,7 @@ import qualified Streamly.Data.Array as Array
3131import qualified Streamly.Data.Stream as Stream
3232import qualified Streamly.Internal.Unicode.Parser as PRU
3333
34−import Gauge hiding (env)
34+import Test.Tasty.Bench hiding (env)
3535import Streamly.Benchmark.Common
3636
3737runParser :: Int -> (Stream IO Char -> IO a) -> IO ()
benchmark/Streamly/Benchmark/Unicode/Stream.hs+25−7
@@ -11,6 +11,11 @@
1111{-# LANGUAGE CPP #-}
1212{-# LANGUAGE ScopedTypeVariables #-}
1313
14+#undef FUSION_CHECK
15+#ifdef FUSION_CHECK
16+{-# OPTIONS_GHC -ddump-simpl -ddump-to-file -dsuppress-all #-}
17+#endif
18+
1419#ifdef __HADDOCK_VERSION__
1520#undef INSPECTION
1621#endif
@@ -33,16 +38,17 @@ import qualified Streamly.Internal.FileSystem.Handle as Handle
3338import qualified Streamly.Internal.Unicode.Array as UnicodeArr
3439import qualified Streamly.Internal.Unicode.Stream as Unicode
3540
36−import Gauge hiding (env)
41+import Test.Tasty.Bench hiding (env)
3742import Streamly.Benchmark.Common
3843import Streamly.Benchmark.Common.Handle
3944
4045#ifdef INSPECTION
41−import Streamly.Internal.Data.Unbox (Unbox)
42−import Streamly.Internal.Data.Stream.StreamD (Step(..))
46+import Streamly.Internal.Data.MutByteArray (Unbox)
47+import Streamly.Internal.Data.Stream (Step(..))
48+import qualified Streamly.Internal.Data.Array as Array
49+import qualified Streamly.Internal.Data.MutArray as MutArray
4350import qualified Streamly.Internal.Data.Fold as Fold
4451import qualified Streamly.Internal.Data.Tuple.Strict as Strict
45−import qualified Streamly.Internal.Data.Array.Type as Array
4652
4753import Test.Inspection
4854#endif
@@ -218,7 +224,7 @@ inspect $ 'copyStreamLatin1' `hasNoType` ''Step
218224inspect $ 'copyStreamLatin1' `hasNoType` ''Unfold.ConcatState -- Handle.read/UF.many
219225
220226inspect $ 'copyStreamLatin1' `hasNoType` ''Fold.Step
221−inspect $ 'copyStreamLatin1' `hasNoType` ''Array.ArrayUnsafe -- Handle.write/writeNUnsafe
227+inspect $ 'copyStreamLatin1' `hasNoType` ''MutArray.ArrayUnsafe -- Handle.write/writeNUnsafe
222228 -- Handle.read/Array.read
223229inspect $ 'copyStreamLatin1' `hasNoType` ''Strict.Tuple3' -- Handle.write/chunksOf
224230#endif
@@ -239,7 +245,7 @@ inspect $ 'copyStreamLatin1 `hasNoType` ''Unfold.ConcatState -- Handle.read/UF.m
239245
240246inspect $ 'copyStreamLatin1 `hasNoType` ''Fold.ManyState
241247inspect $ 'copyStreamLatin1 `hasNoType` ''Fold.Step
242−inspect $ 'copyStreamLatin1 `hasNoType` ''Array.ArrayUnsafe -- Handle.write/writeNUnsafe
248+inspect $ 'copyStreamLatin1 `hasNoType` ''MutArray.ArrayUnsafe -- Handle.write/writeNUnsafe
243249 -- Handle.read/Array.read
244250inspect $ 'copyStreamLatin1 `hasNoType` ''Strict.Tuple3' -- Handle.write/chunksOf
245251#endif
@@ -280,7 +286,8 @@ _copyStreamUtf8'Fold :: Handle -> Handle -> IO ()
280286_copyStreamUtf8'Fold inh outh =
281287 Stream.fold (Handle.write outh)
282288 $ Unicode.encodeUtf8
283− $ Stream.foldMany Unicode.writeCharUtf8'
289+ $ Stream.catRights
290+ $ Stream.parseMany Unicode.writeCharUtf8'
284291 $ Stream.unfold Handle.reader inh
285292
286293{-# NOINLINE _copyStreamUtf8Parser #-}
@@ -317,6 +324,7 @@ o_1_space_decode_encode_read env =
317324
318325main :: IO ()
319326main = do
327+#ifndef FUSION_CHECK
320328 env <- mkHandleBenchEnv
321329 defaultMain (allBenchmarks env)
322330
@@ -329,3 +337,13 @@ main = do
329337 , o_1_space_decode_encode_read env
330338 ]
331339 ]
340+#else
341+ -- Enable FUSION_CHECK macro at the beginning of the file
342+ -- Enable one benchmark below, and run the benchmark
343+ -- Check the .dump-simpl output
344+ env <- mkHandleBenchEnv
345+ let mkHandles (RefHandles {bigInH = inh, outputH = outh}) = Handles inh outh
346+ (Handles inh outh) <- getHandles env mkHandles
347+ copyStreamLatin1' inh outh
348+ return ()
349+#endif
benchmark/Streamly/Benchmark/Unicode/Utf8.hs+1−1
@@ -13,7 +13,7 @@
1313import Data.Char (chr)
1414import Streamly.Internal.Unicode.Utf8 (Utf8)
1515
16−import Gauge hiding (env)
16+import Test.Tasty.Bench hiding (env)
1717import Streamly.Benchmark.Common
1818
1919import qualified Streamly.Internal.Unicode.Utf8 as Utf8
benchmark/bench-runner/Main.hs+34−6
@@ -16,9 +16,11 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
1616 benchName = drop 4 benchName0
1717
1818 general
19− | "o-1-sp" `isInfixOf` benchName = "-K36K -M16M"
19+ -- GHC-9.6 requires 32M heap vs 16M for GHC-9.4
20+ | "o-1-sp" `isInfixOf` benchName = "-K36K -M32M"
2021 | "o-n-h" `isInfixOf` benchName = "-K36K -M32M"
21− | "o-n-st" `isInfixOf` benchName = "-K1M -M16M"
22+ -- GHC-9.6 requires 32M heap vs 16M for GHC-9.4
23+ | "o-n-st" `isInfixOf` benchName = "-K1M -M32M"
2224 | "o-n-sp" `isInfixOf` benchName = "-K1M -M32M"
2325 | otherwise = ""
2426
@@ -27,6 +29,17 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
2729 | otherwise = ""
2830
2931 benchSpecific
32+ -- GHC-9.6 requires 64M, earlier it was 16M
33+ | "Data.Fold/o-n-heap.key-value.toHashMapIO (max buckets) sum" == benchName =
34+ "-M64M"
35+
36+ -- This is required only for the --long case because we allocate all
37+ -- the arrays upfront. depends on the size of the stream.
38+ | "Data.Parser/o-1-space" `isPrefixOf` benchName =
39+ "-M128M"
40+
41+ -----------------------------------------------------------------------
42+
3043 | "Prelude.Parallel/o-n-heap.mapping.mapM" == benchName = "-M256M"
3144 | "Prelude.Parallel/o-n-heap.monad-outer-product."
3245 `isPrefixOf` benchName = "-M256M"
@@ -57,12 +70,18 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
5770 -- GHC 9.4.4 requires 4M
5871 | "Data.Stream.StreamDK/o-n-space.traversable."
5972 `isPrefixOf` benchName = "-K4M"
73+ -- GHC-9.6 requires 64M, earlier it was 32M
74+ | "Data.Stream.StreamDK/o-n-heap.buffered.readsPrec pure streams" == benchName =
75+ "-M64M"
76+ -- GHC-9.6 requires 64M, earlier it was 32M
77+ | "Data.Stream.StreamDK/o-n-heap.buffered.showPrec Haskell lists" == benchName =
78+ "-M64M"
6079
6180 -----------------------------------------------------------------------
6281
63− | "Data.Stream.StreamD/o-n-space.elimination.toList" == benchName =
82+ | "Data.StreamD/o-n-space.elimination.toList" == benchName =
6483 "-K2M"
65− | "Data.Stream.StreamK/o-n-space.elimination.toList" == benchName =
84+ | "Data.StreamK/o-n-space.elimination.toList" == benchName =
6685 "-K2M"
6786
6887 -----------------------------------------------------------------------
@@ -74,6 +93,13 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
7493 | "Data.Stream/o-n-space.iterated."
7594 `isPrefixOf` benchName = "-K4M"
7695
96+ -- GHC-9.6 requires 64M, earlier it was 32M
97+ | "Data.Stream/o-n-heap.buffered.showPrec Haskell lists" == benchName =
98+ "-M64M"
99+ -- GHC-9.6 requires 64M, earlier it was 32M
100+ | "Data.Stream/o-n-heap.buffered.readsPrec pure streams" == benchName =
101+ "-M64M"
102+
77103 | "Data.Stream.ConcurrentEager/o-n-heap.monad-outer-product.toNullAp"
78104 `isPrefixOf` benchName = "-M1024M"
79105 | "Data.Stream.ConcurrentEager/o-1-space."
@@ -86,10 +112,12 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
86112
87113 | "Data.Array" `isPrefixOf` benchName
88114 && "/o-1-space.generation.read" `isSuffixOf` benchName = "-M32M"
115+ -- XXX GHC 9.6 onwards needs 64M, earlier it was 32M
89116 | "Data.Array" `isPrefixOf` benchName
90− && "/o-1-space.generation.show" `isSuffixOf` benchName = "-M32M"
117+ && "/o-1-space.generation.show" `isSuffixOf` benchName = "-M64M"
118+ -- XXX GHC 9.6 onwards needs 64M, earlier it was 32M
91119 | "Data.Array.Generic/o-1-space.transformationX4.map"
92− `isPrefixOf` benchName = "-M32M"
120+ `isPrefixOf` benchName = "-M64M"
93121
94122 -- XXX For --long option, need to check why so much heap is required.
95123 -- Note, if we remove the chunked stream module we need to keep the
benchmark/lib/Streamly/Benchmark/Common.hs+1−1
@@ -56,7 +56,7 @@ import System.Random (randomRIO)
5656import qualified Streamly.Internal.Data.Fold as Fold
5757import qualified Streamly.Internal.Data.Stream as S
5858
59−import Gauge
59+import Test.Tasty.Bench
6060
6161-------------------------------------------------------------------------------
6262-- Benchmark Prefixes
benchmark/lib/Streamly/Benchmark/Common/Handle.hs+3−1
@@ -30,6 +30,8 @@ module Streamly.Benchmark.Common.Handle
3030 , isSpace
3131 , isSp
3232 , mkHandleBenchEnv
33+ , Handles(..)
34+ , getHandles
3335 )
3436where
3537
@@ -46,7 +48,7 @@ import System.Process (callCommand)
4648
4749import Data.IORef
4850import Prelude hiding (last, length)
49−import Gauge hiding (env)
51+import Test.Tasty.Bench hiding (env)
5052
5153
5254scratchDir :: String
benchmark/lib/Streamly/Benchmark/Prelude.hs+1−1
@@ -82,7 +82,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream.Type as IsStream
8282import qualified Streamly.Internal.Data.Pipe as Pipe
8383import qualified Streamly.Internal.Data.Stream.Serial as Serial
8484
85−import Gauge
85+import Test.Tasty.Bench
8686import Streamly.Internal.Data.Time.Units
8787
8888-- Common polymorphic stream APIs used across multiple stream modules
benchmark/streamly-benchmarks.cabal+58−41
@@ -54,11 +54,6 @@ flag opt
5454 manual: True
5555 default: True
5656
57−flag use-gauge
58− description: Use gauge instead of tasty-bench for benchmarking
59− manual: True
60− default: False
61−
6257flag use-streamly-core
6358 description: Benchmark only core package
6459 manual: True
@@ -145,6 +140,10 @@ common compile-options
145140 -Wno-missing-kind-signatures
146141 -Wno-operator-whitespace
147142
143+ if impl(ghc >= 9.8)
144+ ghc-options:
145+ -Wno-missing-role-annotations
146+
148147 if flag(has-llvm)
149148 ghc-options: -fllvm
150149
@@ -152,15 +151,17 @@ common compile-options
152151 ghc-options: -Wmissed-specialisations
153152 -Wall-missed-specialisations
154153
155− if flag(dev) || flag(debug)
156− ghc-options: -fno-ignore-asserts
157−
158154common optimization-options
159155 if flag(opt)
160156 ghc-options: -O2
161157 -fdicts-strict
162158 -fspec-constr-recursive=16
163159 -fmax-worker-args=16
160+
161+ -- For this to be effective it must come after the -O2 option
162+ if flag(dev) || flag(debug)
163+ ghc-options: -fno-ignore-asserts
164+
164165 if flag(fusion-plugin) && !impl(ghcjs) && !impl(ghc < 8.6)
165166 ghc-options: -fplugin Fusion.Plugin
166167 else
@@ -174,43 +175,34 @@ common bench-depends
174175 -- Core libraries shipped with ghc, the min and max
175176 -- constraints of these libraries should match with
176177 -- the GHC versions we support
177− base >= 4.9 && < 4.19
178− , deepseq >= 1.4.1 && < 1.5
178+ base >= 4.9 && < 4.20
179+ , deepseq >= 1.4.1 && < 1.6
179180 , mtl >= 2.2 && < 2.3.2
180181
181182 -- other libraries
183+ , exceptions >= 0.8 && < 0.11
182184 , random >= 1.0 && < 2.0
183185 , transformers >= 0.4 && < 0.7
184− , containers >= 0.5 && < 0.7
186+ , containers >= 0.5 && < 0.8
185187 , hashable >= 1.3 && < 1.5
186188 , unordered-containers >= 0.2 && < 0.3
187189 , process >= 1.4 && < 1.7
188190 , directory >= 1.2.2 && < 1.4
189− , filepath >= 1.4.1 && < 1.5
190− , ghc-prim >= 0.4 && < 0.11
191+ , filepath >= 1.4.1 && < 1.6
192+ , ghc-prim >= 0.4 && < 0.12
193+ , tasty-bench >= 0.3 && < 0.4
194+ , tasty >= 1.4.1 && < 1.6
195+ , streamly-core
191196
192− if flag(use-streamly-core)
193− build-depends: streamly-core
194− else
197+ if !flag(use-streamly-core)
195198 build-depends: streamly
196− , streamly-core
197−
198− if flag(use-gauge)
199− build-depends: gauge >= 0.2.4 && < 0.3
200− else
201− build-depends: tasty-bench >= 0.3 && < 0.4
202− , tasty >= 1.4.1 && < 1.5
203− mixins: tasty-bench
204− (Test.Tasty.Bench as Gauge
205− , Test.Tasty.Bench as Gauge.Main
206− )
207199
208200 if flag(fusion-plugin) && !impl(ghcjs) && !impl(ghc < 8.6)
209201 build-depends:
210202 fusion-plugin >= 0.2 && < 0.3
211203 if flag(inspection)
212− build-depends: template-haskell >= 2.14 && < 2.21
213− , inspection-testing >= 0.4 && < 0.5
204+ build-depends: template-haskell >= 2.14 && < 2.22
205+ , inspection-testing >= 0.4 && < 0.6
214206 -- Array uses a Storable constraint in dev build making several inspection
215207 -- tests fail
216208 if flag(dev) && flag(inspection)
@@ -360,14 +352,30 @@ benchmark Data.Unbox
360352 import: bench-options
361353 type: exitcode-stdio-1.0
362354 hs-source-dirs: .
363− main-is: Streamly/Benchmark/Data/Unbox.hs
355+ cpp-options: -DUSE_UNBOX
356+ main-is: Streamly/Benchmark/Data/Serialize.hs
364357
365358benchmark Data.Unbox.Derive.TH
366359 import: bench-options
367360 type: exitcode-stdio-1.0
368361 hs-source-dirs: .
362+ cpp-options: -DUSE_UNBOX
369363 cpp-options: -DUSE_TH
370− main-is: Streamly/Benchmark/Data/Unbox.hs
364+ main-is: Streamly/Benchmark/Data/Serialize.hs
365+
366+benchmark Data.Serialize
367+ import: bench-options
368+ type: exitcode-stdio-1.0
369+ hs-source-dirs: .
370+ cpp-options: -DUSE_TH
371+ main-is: Streamly/Benchmark/Data/Serialize.hs
372+ build-depends: QuickCheck, template-haskell
373+ other-modules:
374+ Streamly.Benchmark.Data.Serialize.TH
375+ Streamly.Benchmark.Data.Serialize.RecCompatible
376+ Streamly.Benchmark.Data.Serialize.RecNonCompatible
377+ if flag(limit-build-mem)
378+ ghc-options: +RTS -M1000M -RTS
371379
372380benchmark Data.Unfold
373381 import: bench-options
@@ -419,6 +427,7 @@ benchmark Data.ParserK
419427 import: bench-options
420428 type: exitcode-stdio-1.0
421429 hs-source-dirs: Streamly/Benchmark/Data
430+ cpp-options: -DBENCH_SINGULAR
422431 main-is: ParserK.hs
423432 if impl(ghcjs)
424433 buildable: False
@@ -426,17 +435,25 @@ benchmark Data.ParserK
426435 buildable: True
427436 build-depends: exceptions >= 0.8 && < 0.11
428437
429−-- Note: to use this we have to set DISABLE_FUSION in ParserD.hs so that
430−-- the rewrite rules are disabled. We can also use the "no-fusion" build
431−-- flag but we need to keep in mind that it disables fusion for streams
432−-- as well.
433−benchmark Data.FromParserK
438+benchmark Data.ParserK.Chunked
434439 import: bench-options
435440 type: exitcode-stdio-1.0
436441 hs-source-dirs: Streamly/Benchmark/Data
437− cpp-options: -DFROM_PARSERK
442+ cpp-options: -DBENCH_CHUNKED
438443 main-is: ParserK.hs
439− if flag(use-streamly-core) || impl(ghcjs)
444+ if impl(ghcjs)
445+ buildable: False
446+ else
447+ buildable: True
448+ build-depends: exceptions >= 0.8 && < 0.11
449+
450+benchmark Data.ParserK.Chunked.Generic
451+ import: bench-options
452+ type: exitcode-stdio-1.0
453+ hs-source-dirs: Streamly/Benchmark/Data
454+ cpp-options: -DBENCH_CHUNKED_GENERIC
455+ main-is: ParserK.hs
456+ if impl(ghcjs)
440457 buildable: False
441458 else
442459 buildable: True
@@ -446,7 +463,7 @@ benchmark Data.FromParserK
446463-- Raw Streams
447464-------------------------------------------------------------------------------
448465
449−benchmark Data.Stream.StreamD
466+benchmark Data.StreamD
450467 import: bench-options
451468 type: exitcode-stdio-1.0
452469 hs-source-dirs: Streamly/Benchmark/Data/Stream
@@ -454,7 +471,7 @@ benchmark Data.Stream.StreamD
454471 if impl(ghcjs)
455472 buildable: False
456473
457−benchmark Data.Stream.StreamK
474+benchmark Data.StreamK
458475 import: bench-options
459476 type: exitcode-stdio-1.0
460477 hs-source-dirs: Streamly/Benchmark/Data/Stream
@@ -470,7 +487,7 @@ benchmark Data.Stream.ToStreamK
470487 if !flag(dev) || impl(ghcjs)
471488 buildable: False
472489
473−benchmark Data.Stream.StreamK.Alt
490+benchmark Data.StreamK.Alt
474491 import: bench-options
475492 type: exitcode-stdio-1.0
476493 hs-source-dirs: Streamly/Benchmark/Data/Stream
configure+9−9
@@ -1,6 +1,6 @@
11#! /bin/sh
22# Guess values for system-dependent variables and create Makefiles.
3−# Generated by GNU Autoconf 2.71 for streamly 0.10.0.
3+# Generated by GNU Autoconf 2.71 for streamly 0.10.1.
44#
55# Report bugs to <streamly@composewell.com>.
66#
@@ -610,8 +610,8 @@ MAKEFLAGS=
610610# Identity of this package.
611611PACKAGE_NAME='streamly'
612612PACKAGE_TARNAME='streamly'
613−PACKAGE_VERSION='0.10.0'
614−PACKAGE_STRING='streamly 0.10.0'
613+PACKAGE_VERSION='0.10.1'
614+PACKAGE_STRING='streamly 0.10.1'
615615PACKAGE_BUGREPORT='streamly@composewell.com'
616616PACKAGE_URL='https://streamly.composewell.com'
617617
@@ -1224,7 +1224,7 @@ if test "$ac_init_help" = "long"; then
12241224 # Omit some internal or obsolete options to make the list less imposing.
12251225 # This message is too long to be a string in the A/UX 3.1 sh.
12261226 cat <<_ACEOF
1227−\`configure' configures streamly 0.10.0 to adapt to many kinds of systems.
1227+\`configure' configures streamly 0.10.1 to adapt to many kinds of systems.
12281228
12291229Usage: $0 [OPTION]... [VAR=VALUE]...
12301230
@@ -1286,7 +1286,7 @@ fi
12861286
12871287if test -n "$ac_init_help"; then
12881288 case $ac_init_help in
1289− short | recursive ) echo "Configuration of streamly 0.10.0:";;
1289+ short | recursive ) echo "Configuration of streamly 0.10.1:";;
12901290 esac
12911291 cat <<\_ACEOF
12921292
@@ -1372,7 +1372,7 @@ fi
13721372test -n "$ac_init_help" && exit $ac_status
13731373if $ac_init_version; then
13741374 cat <<\_ACEOF
1375−streamly configure 0.10.0
1375+streamly configure 0.10.1
13761376generated by GNU Autoconf 2.71
13771377
13781378Copyright (C) 2021 Free Software Foundation, Inc.
@@ -1500,7 +1500,7 @@ cat >config.log <<_ACEOF
15001500This file contains any messages produced by compilers while
15011501running configure, to aid debugging if configure makes a mistake.
15021502
1503−It was created by streamly $as_me 0.10.0, which was
1503+It was created by streamly $as_me 0.10.1, which was
15041504generated by GNU Autoconf 2.71. Invocation command line was
15051505
15061506 $ $0$ac_configure_args_raw
@@ -3800,7 +3800,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
38003800# report actual input values of CONFIG_FILES etc. instead of their
38013801# values after options handling.
38023802ac_log="
3803−This file was extended by streamly $as_me 0.10.0, which was
3803+This file was extended by streamly $as_me 0.10.1, which was
38043804generated by GNU Autoconf 2.71. Invocation command line was
38053805
38063806 CONFIG_FILES = $CONFIG_FILES
@@ -3856,7 +3856,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\
38563856cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
38573857ac_cs_config='$ac_cs_config_escaped'
38583858ac_cs_version="\\
3859−streamly config.status 0.10.0
3859+streamly config.status 0.10.1
38603860configured by $0, generated by GNU Autoconf 2.71,
38613861 with options \\"\$ac_cs_config\\"
38623862
configure.ac+1−1
@@ -3,7 +3,7 @@
33# See https://www.gnu.org/software/autoconf/manual/autoconf.html for help on
44# the macros used in this file.
55
6−AC_INIT([streamly], [0.10.0], [streamly@composewell.com], [streamly], [https://streamly.composewell.com])
6+AC_INIT([streamly], [0.10.1], [streamly@composewell.com], [streamly], [https://streamly.composewell.com])
77
88# To suppress "WARNING: unrecognized options: --with-compiler"
99AC_ARG_WITH([compiler], [GHC])
core/configure+9−9
@@ -1,6 +1,6 @@
11#! /bin/sh
22# Guess values for system-dependent variables and create Makefiles.
3−# Generated by GNU Autoconf 2.71 for streamly-core 0.2.0.
3+# Generated by GNU Autoconf 2.71 for streamly-core 0.2.2.
44#
55# Report bugs to <streamly@composewell.com>.
66#
@@ -610,8 +610,8 @@ MAKEFLAGS=
610610# Identity of this package.
611611PACKAGE_NAME='streamly-core'
612612PACKAGE_TARNAME='streamly-core'
613−PACKAGE_VERSION='0.2.0'
614−PACKAGE_STRING='streamly-core 0.2.0'
613+PACKAGE_VERSION='0.2.2'
614+PACKAGE_STRING='streamly-core 0.2.2'
615615PACKAGE_BUGREPORT='streamly@composewell.com'
616616PACKAGE_URL='https://streamly.composewell.com'
617617
@@ -1256,7 +1256,7 @@ if test "$ac_init_help" = "long"; then
12561256 # Omit some internal or obsolete options to make the list less imposing.
12571257 # This message is too long to be a string in the A/UX 3.1 sh.
12581258 cat <<_ACEOF
1259−\`configure' configures streamly-core 0.2.0 to adapt to many kinds of systems.
1259+\`configure' configures streamly-core 0.2.2 to adapt to many kinds of systems.
12601260
12611261Usage: $0 [OPTION]... [VAR=VALUE]...
12621262
@@ -1318,7 +1318,7 @@ fi
13181318
13191319if test -n "$ac_init_help"; then
13201320 case $ac_init_help in
1321− short | recursive ) echo "Configuration of streamly-core 0.2.0:";;
1321+ short | recursive ) echo "Configuration of streamly-core 0.2.2:";;
13221322 esac
13231323 cat <<\_ACEOF
13241324
@@ -1404,7 +1404,7 @@ fi
14041404test -n "$ac_init_help" && exit $ac_status
14051405if $ac_init_version; then
14061406 cat <<\_ACEOF
1407−streamly-core configure 0.2.0
1407+streamly-core configure 0.2.2
14081408generated by GNU Autoconf 2.71
14091409
14101410Copyright (C) 2021 Free Software Foundation, Inc.
@@ -1622,7 +1622,7 @@ cat >config.log <<_ACEOF
16221622This file contains any messages produced by compilers while
16231623running configure, to aid debugging if configure makes a mistake.
16241624
1625−It was created by streamly-core $as_me 0.2.0, which was
1625+It was created by streamly-core $as_me 0.2.2, which was
16261626generated by GNU Autoconf 2.71. Invocation command line was
16271627
16281628 $ $0$ac_configure_args_raw
@@ -3832,7 +3832,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
38323832# report actual input values of CONFIG_FILES etc. instead of their
38333833# values after options handling.
38343834ac_log="
3835−This file was extended by streamly-core $as_me 0.2.0, which was
3835+This file was extended by streamly-core $as_me 0.2.2, which was
38363836generated by GNU Autoconf 2.71. Invocation command line was
38373837
38383838 CONFIG_FILES = $CONFIG_FILES
@@ -3888,7 +3888,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\
38883888cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
38893889ac_cs_config='$ac_cs_config_escaped'
38903890ac_cs_version="\\
3891−streamly-core config.status 0.2.0
3891+streamly-core config.status 0.2.2
38923892configured by $0, generated by GNU Autoconf 2.71,
38933893 with options \\"\$ac_cs_config\\"
38943894
core/configure.ac+1−1
@@ -3,7 +3,7 @@
33# See https://www.gnu.org/software/autoconf/manual/autoconf.html for help on
44# the macros used in this file.
55
6−AC_INIT([streamly-core], [0.2.0], [streamly@composewell.com], [streamly-core], [https://streamly.composewell.com])
6+AC_INIT([streamly-core], [0.2.2], [streamly@composewell.com], [streamly-core], [https://streamly.composewell.com])
77
88# To suppress "WARNING: unrecognized options: --with-compiler"
99AC_ARG_WITH([compiler], [GHC])
core/docs/ApiChangelogs/0.1.0-0.2.0.txtnew file+1471
@@ -0,0 +1,1471 @@
1+---------------------------------
2+API Annotations
3+---------------------------------
4+
5+[A] : Added
6+[R] : Removed
7+[C] : Changed
8+[O] : Old definition
9+[N] : New definition
10+[D] : Deprecated
11+
12+---------------------------------
13+API diff
14+---------------------------------
15+
16+[C] Streamly.Unicode.Parser
17+ [A] double :: Monad m => Parser Char m Double
18+[C] Streamly.FileSystem.Handle
19+ [A] readWith :: MonadIO m => Int -> Handle -> Stream m Word8
20+ [A] readChunksWith :: MonadIO m => Int -> Handle -> Stream m (Array Word8)
21+[C] Streamly.FileSystem.Dir
22+ [C] readEither
23+ [O] readEither :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
24+ [N] readEither :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
25+ [C] read
26+ [O] read :: MonadIO m => FilePath -> Stream m FilePath
27+ [N] read :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
28+[C] Streamly.Data.Unfold
29+ [A] class Enum a => Enumerable a
30+ [A] second :: b -> Unfold m (a, b) c -> Unfold m a c
31+ [A] first :: a -> Unfold m (a, b) c -> Unfold m b c
32+ [A] enumerateFromTo :: (Enumerable a, Monad m) => Unfold m (a, a) a
33+ [A] enumerateFromThenTo :: (Enumerable a, Monad m) => Unfold m (a, a, a) a
34+ [A] enumerateFromThen :: (Enumerable a, Monad m) => Unfold m (a, a) a
35+ [A] enumerateFrom :: (Enumerable a, Monad m) => Unfold m a a
36+[C] Streamly.Data.StreamK
37+ [C] parseChunks
38+ [O] parseChunks :: (Monad m, Unbox a) => ParserK a m b -> StreamK m (Array a) -> m (Either ParseError b)
39+ [N] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
40+ [C] parseBreakChunks
41+ [O] parseBreakChunks :: (Monad m, Unbox a) => ParserK a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
42+ [N] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
43+ [A] parseBreak :: forall m a b. Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
44+ [A] parse :: Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b)
45+ [A] handle :: (MonadCatch m, Exception e) => (e -> m (StreamK m a)) -> StreamK m a -> StreamK m a
46+ [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> StreamK m a) -> StreamK m a
47+[C] Streamly.Data.Stream
48+ [A] wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
49+ [A] splitOn :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
50+ [C] handle
51+ [O] handle :: (MonadCatch m, Exception e) => (e -> Stream m a) -> Stream m a -> Stream m a
52+ [N] handle :: (MonadCatch m, Exception e) => (e -> m (Stream m a)) -> Stream m a -> Stream m a
53+ [A] groupsOf :: Monad m => Int -> Fold m a b -> Stream m a -> Stream m b
54+[C] Streamly.Data.ParserK
55+ [D] fromParser :: (MonadIO m, Unbox a) => Parser a m b -> ParserK (Array a) m b
56+ [D] fromFold :: (MonadIO m, Unbox a) => Fold m a b -> ParserK (Array a) m b
57+ [A] adaptCG :: Monad m => Parser a m b -> ParserK (Array a) m b
58+ [A] adaptC :: (Monad m, Unbox a) => Parser a m b -> ParserK (Array a) m b
59+ [A] adapt :: Monad m => Parser a m b -> ParserK a m b
60+[C] Streamly.Data.Parser
61+ [A] groupByRollingEither :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
62+ [A] groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
63+[A] Streamly.Data.MutByteArray
64+ [A] class Unbox a
65+ [A] class Serialize a
66+ [A] SerializeConfig
67+ [A] MutByteArray
68+ [A] unpin :: MutByteArray -> IO MutByteArray
69+ [A] sizeOf :: (Unbox a, SizeOfRep (Rep a)) => Proxy a -> Int
70+ [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
71+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
72+ [A] pinnedNew :: Int -> IO MutByteArray
73+ [A] pin :: MutByteArray -> IO MutByteArray
74+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
75+ [A] new :: Int -> IO MutByteArray
76+ [A] isPinned :: MutByteArray -> Bool
77+ [A] inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
78+ [A] inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
79+ [A] inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
80+ [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
81+ [A] deriveUnbox :: Q [Dec] -> Q [Dec]
82+ [A] deriveSerializeWith :: (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
83+ [A] deriveSerialize :: Q [Dec] -> Q [Dec]
84+ [A] addSizeTo :: Serialize a => Int -> a -> Int
85+[C] Streamly.Data.MutArray.Generic
86+ [A] write :: MonadIO m => Fold m a (MutArray a)
87+ [A] readRev :: MonadIO m => MutArray a -> Stream m a
88+ [A] read :: MonadIO m => MutArray a -> Stream m a
89+ [A] putIndexUnsafe :: forall m a. MonadIO m => Int -> MutArray a -> a -> m ()
90+ [A] modifyIndexUnsafe :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
91+ [A] length :: MutArray a -> Int
92+ [A] getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
93+ [C] getIndex
94+ [O] getIndex :: MonadIO m => Int -> MutArray a -> m a
95+ [N] getIndex :: MonadIO m => Int -> MutArray a -> m (Maybe a)
96+ [A] fromListN :: MonadIO m => Int -> [a] -> m (MutArray a)
97+ [A] fromList :: MonadIO m => [a] -> m (MutArray a)
98+[C] Streamly.Data.MutArray
99+ [A] unpin :: MutArray a -> IO (MutArray a)
100+ [A] readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
101+ [A] read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
102+ [A] putIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
103+ [D] pokeByteIndex :: Unbox a => Int -> MutByteArray -> a -> IO ()
104+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
105+ [A] pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
106+ [A] pin :: MutArray a -> IO (MutArray a)
107+ [D] peekByteIndex :: Unbox a => Int -> MutByteArray -> IO a
108+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
109+ [D] newPinned :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
110+ [A] modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
111+ [A] modifyIndex :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
112+ [A] modify :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (a -> a) -> m ()
113+ [A] isPinned :: MutArray a -> Bool
114+ [A] getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
115+ [C] getIndex
116+ [O] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
117+ [N] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
118+[C] Streamly.Data.Array.Generic
119+ [A] toList :: Array a -> [a]
120+ [A] getIndex :: Int -> Array a -> Maybe a
121+[C] Streamly.Data.Array
122+ [A] unpin :: Array a -> IO (Array a)
123+ [D] pokeByteIndex :: Unbox a => Int -> MutByteArray -> a -> IO ()
124+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
125+ [A] pin :: Array a -> IO (Array a)
126+ [D] peekByteIndex :: Unbox a => Int -> MutByteArray -> IO a
127+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
128+ [A] isPinned :: Array a -> Bool
129+[C] Streamly.Console.Stdio
130+
131+---------------------------------
132+Internal API diff
133+---------------------------------
134+
135+[C] Streamly.Internal.Unicode.Stream
136+ [C] writeCharUtf8'
137+ [O] writeCharUtf8' :: Monad m => Fold m Word8 Char
138+ [N] writeCharUtf8' :: Monad m => Parser Word8 m Char
139+[C] Streamly.Internal.Unicode.Parser
140+ [A] number :: Monad m => Parser Char m (Integer, Int)
141+ [A] mkDouble :: Integer -> Int -> Double
142+ [A] doubleParser :: Monad m => Parser Char m (Int, Int)
143+ [C] double
144+ [O] double :: Parser Char m Double
145+ [N] double :: Monad m => Parser Char m Double
146+[R] Streamly.Internal.Serialize.ToBytes
147+[R] Streamly.Internal.Serialize.FromBytes
148+[C] Streamly.Internal.FileSystem.File
149+ [A] writeAppendWith :: (MonadIO m, MonadCatch m) => Int -> FilePath -> Stream m Word8 -> m ()
150+ [A] writeAppendChunks :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Array a) -> m ()
151+ [A] writeAppendArray :: FilePath -> Array a -> IO ()
152+ [A] writeAppend :: (MonadIO m, MonadCatch m) => FilePath -> Stream m Word8 -> m ()
153+ [R] appendWith :: (MonadIO m, MonadCatch m) => Int -> FilePath -> Stream m Word8 -> m ()
154+ [R] appendChunks :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Array a) -> m ()
155+ [R] appendArray :: FilePath -> Array a -> IO ()
156+ [R] append :: (MonadIO m, MonadCatch m) => FilePath -> Stream m Word8 -> m ()
157+[C] Streamly.Internal.FileSystem.Dir
158+ [C] reader
159+ [O] reader :: MonadIO m => Unfold m FilePath FilePath
160+ [N] reader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
161+ [C] readFiles
162+ [O] readFiles :: MonadIO m => FilePath -> Stream m FilePath
163+ [N] readFiles :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
164+ [C] readEitherPaths
165+ [O] readEitherPaths :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
166+ [N] readEitherPaths :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
167+ [C] readEither
168+ [O] readEither :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
169+ [N] readEither :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
170+ [C] readDirs
171+ [O] readDirs :: MonadIO m => FilePath -> Stream m FilePath
172+ [N] readDirs :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
173+ [C] read
174+ [O] read :: MonadIO m => FilePath -> Stream m FilePath
175+ [N] read :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
176+ [C] fileReader
177+ [O] fileReader :: MonadIO m => Unfold m FilePath FilePath
178+ [N] fileReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
179+ [C] eitherReaderPaths
180+ [O] eitherReaderPaths :: MonadIO m => Unfold m FilePath (Either FilePath FilePath)
181+ [N] eitherReaderPaths :: (MonadIO m, MonadCatch m) => Unfold m FilePath (Either FilePath FilePath)
182+ [C] eitherReader
183+ [O] eitherReader :: MonadIO m => Unfold m FilePath (Either FilePath FilePath)
184+ [N] eitherReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath (Either FilePath FilePath)
185+ [C] dirReader
186+ [O] dirReader :: MonadIO m => Unfold m FilePath FilePath
187+ [N] dirReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
188+[R] Streamly.Internal.Data.Unfold.Type
189+[R] Streamly.Internal.Data.Unfold.Enumeration
190+[C] Streamly.Internal.Data.Unfold
191+ [C] Unfold
192+ [A] Unfold :: (s -> m (Step s b)) -> (a -> m s) -> Unfold m a b
193+ [A] takeWhileMWithInput :: Monad m => (a -> b -> m Bool) -> Unfold m a b -> Unfold m a b
194+ [A] manyInterleave :: Monad m => Unfold m a b -> Unfold m c a -> Unfold m c b
195+ [A] enumerateFromStepIntegral :: (Monad m, Integral a) => Unfold m (a, a) a
196+ [A] crossApplySnd :: Unfold m a b -> Unfold m a c -> Unfold m a c
197+ [A] crossApplyFst :: Unfold m a b -> Unfold m a c -> Unfold m a b
198+ [A] concatMap :: Monad m => (b -> Unfold m a c) -> Unfold m a b -> Unfold m a c
199+[R] Streamly.Internal.Data.Unboxed
200+[C] Streamly.Internal.Data.Time.Units
201+ [R] Streamly.Internal.Data.Unboxed.Unbox
202+ [A] Streamly.Internal.Data.Unbox.Unbox
203+ [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.NanoSecond64
204+ [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.MilliSecond64
205+ [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.MicroSecond64
206+[R] Streamly.Internal.Data.Time.Clock.Type
207+[A] Streamly.Internal.Data.StreamK
208+ [A] CrossStreamK
209+ [A] (FixityR,6)
210+ [A] (FixityR,6)
211+ [A] (FixityR,6)
212+ [A] (FixityR,5)
213+ [A] (FixityR,5)
214+ [A] (FixityR,6)
215+ [A] (FixityR,5)
216+ [A] StreamK
217+ [A] MkStream :: (forall r. State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
218+ [A] zipWithM :: Monad m => (a -> b -> m c) -> StreamK m a -> StreamK m b -> StreamK m c
219+ [A] zipWith :: Monad m => (a -> b -> c) -> StreamK m a -> StreamK m b -> StreamK m c
220+ [A] unfoldrMWith :: Monad m => (m a -> StreamK m a -> StreamK m a) -> (b -> m (Maybe (a, b))) -> b -> StreamK m a
221+ [A] unfoldrM :: Monad m => (b -> m (Maybe (a, b))) -> b -> StreamK m a
222+ [A] unfoldr :: (b -> Maybe (a, b)) -> b -> StreamK m a
223+ [A] uncons :: Applicative m => StreamK m a -> m (Maybe (a, StreamK m a))
224+ [A] unShare :: StreamK m a -> StreamK m a
225+ [A] unCross :: CrossStreamK m a -> StreamK m a
226+ [A] toStream :: Applicative m => StreamK m a -> Stream m a
227+ [A] toList :: Monad m => StreamK m a -> m [a]
228+ [A] the :: (Eq a, Monad m) => StreamK m a -> m (Maybe a)
229+ [A] takeWhile :: (a -> Bool) -> StreamK m a -> StreamK m a
230+ [A] take :: Int -> StreamK m a -> StreamK m a
231+ [A] tail :: Applicative m => StreamK m a -> m (Maybe (StreamK m a))
232+ [A] sortBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> StreamK m a
233+ [A] sequence :: Monad m => StreamK m (m a) -> StreamK m a
234+ [A] scanlx' :: (x -> a -> x) -> x -> (x -> b) -> StreamK m a -> StreamK m b
235+ [A] scanl' :: (b -> a -> b) -> b -> StreamK m a -> StreamK m b
236+ [A] reverse :: StreamK m a -> StreamK m a
237+ [A] replicateMWith :: (m a -> StreamK m a -> StreamK m a) -> Int -> m a -> StreamK m a
238+ [A] replicateM :: Monad m => Int -> m a -> StreamK m a
239+ [A] replicate :: Int -> a -> StreamK m a
240+ [A] repeatMWith :: (m a -> t m a -> t m a) -> m a -> t m a
241+ [A] repeatM :: Monad m => m a -> StreamK m a
242+ [A] repeat :: a -> StreamK m a
243+ [A] parseDBreak :: Monad m => Parser a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
244+ [A] parseD :: Monad m => Parser a m b -> StreamK m a -> m (Either ParseError b)
245+ [A] parseChunksGeneric :: Monad m => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
246+ [A] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
247+ [A] parseBreakChunksGeneric :: forall m a b. Monad m => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
248+ [A] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
249+ [A] parseBreak :: forall m a b. Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
250+ [A] parse :: Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b)
251+ [A] null :: Monad m => StreamK m a -> m Bool
252+ [A] notElem :: (Monad m, Eq a) => a -> StreamK m a -> m Bool
253+ [A] nilM :: Applicative m => m b -> StreamK m a
254+ [A] nil :: StreamK m a
255+ [A] mkStream :: (forall r. State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
256+ [A] mkCross :: StreamK m a -> CrossStreamK m a
257+ [A] minimumBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> m (Maybe a)
258+ [A] minimum :: (Monad m, Ord a) => StreamK m a -> m (Maybe a)
259+ [A] mfix :: Monad m => (m a -> StreamK m a) -> StreamK m a
260+ [A] mergeMapWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m a -> StreamK m b
261+ [A] mergeIterateWith :: (StreamK m a -> StreamK m a -> StreamK m a) -> (a -> StreamK m a) -> StreamK m a -> StreamK m a
262+ [A] mergeByM :: Monad m => (a -> a -> m Ordering) -> StreamK m a -> StreamK m a -> StreamK m a
263+ [A] mergeBy :: (a -> a -> Ordering) -> StreamK m a -> StreamK m a -> StreamK m a
264+ [A] maximumBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> m (Maybe a)
265+ [A] maximum :: (Monad m, Ord a) => StreamK m a -> m (Maybe a)
266+ [A] mapMaybe :: (a -> Maybe b) -> StreamK m a -> StreamK m b
267+ [A] mapM_ :: Monad m => (a -> m b) -> StreamK m a -> m ()
268+ [A] mapMWith :: (m b -> StreamK m b -> StreamK m b) -> (a -> m b) -> StreamK m a -> StreamK m b
269+ [A] mapMSerial :: Monad m => (a -> m b) -> StreamK m a -> StreamK m b
270+ [A] mapM :: Monad m => (a -> m b) -> StreamK m a -> StreamK m b
271+ [A] map :: (a -> b) -> StreamK m a -> StreamK m b
272+ [A] lookup :: (Monad m, Eq a) => a -> StreamK m (a, b) -> m (Maybe b)
273+ [A] liftInner :: (Monad m, MonadTrans t, Monad (t m)) => StreamK m a -> StreamK (t m) a
274+ [A] last :: Monad m => StreamK m a -> m (Maybe a)
275+ [A] iterateMWith :: Monad m => (m a -> StreamK m a -> StreamK m a) -> (a -> m a) -> m a -> StreamK m a
276+ [A] iterateM :: Monad m => (a -> m a) -> m a -> StreamK m a
277+ [A] iterate :: (a -> a) -> a -> StreamK m a
278+ [A] intersperseM :: Monad m => m a -> StreamK m a -> StreamK m a
279+ [A] intersperse :: Monad m => a -> StreamK m a -> StreamK m a
280+ [A] interleaveMin :: StreamK m a -> StreamK m a -> StreamK m a
281+ [A] interleaveFst :: StreamK m a -> StreamK m a -> StreamK m a
282+ [A] interleave :: StreamK m a -> StreamK m a -> StreamK m a
283+ [A] insertBy :: (a -> a -> Ordering) -> a -> StreamK m a -> StreamK m a
284+ [A] init :: Applicative m => StreamK m a -> m (Maybe (StreamK m a))
285+ [A] hoist :: (Monad m, Monad n) => (forall x. m x -> n x) -> StreamK m a -> StreamK n a
286+ [A] head :: Monad m => StreamK m a -> m (Maybe a)
287+ [A] handle :: (MonadCatch m, Exception e) => (e -> m (StreamK m a)) -> StreamK m a -> StreamK m a
288+ [A] fromYieldK :: YieldK m a -> StreamK m a
289+ [A] fromStream :: Monad m => Stream m a -> StreamK m a
290+ [A] fromStopK :: StopK m -> StreamK m a
291+ [A] fromPure :: a -> StreamK m a
292+ [A] fromList :: [a] -> StreamK m a
293+ [A] fromIndicesMWith :: (m a -> StreamK m a -> StreamK m a) -> (Int -> m a) -> StreamK m a
294+ [A] fromIndicesM :: Monad m => (Int -> m a) -> StreamK m a
295+ [A] fromIndices :: (Int -> a) -> StreamK m a
296+ [A] fromFoldableM :: (Foldable f, Monad m) => f (m a) -> StreamK m a
297+ [A] fromFoldable :: Foldable f => f a -> StreamK m a
298+ [A] fromEffect :: Monad m => m a -> StreamK m a
299+ [A] foldrT :: (Monad m, Monad (s m), MonadTrans s) => (a -> s m b -> s m b) -> s m b -> StreamK m a -> s m b
300+ [A] foldrSShared :: (a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
301+ [A] foldrSM :: Monad m => (m a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
302+ [A] foldrS :: (a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
303+ [A] foldrM :: (a -> m b -> m b) -> m b -> StreamK m a -> m b
304+ [A] foldr1 :: Monad m => (a -> a -> a) -> StreamK m a -> m (Maybe a)
305+ [A] foldr :: Monad m => (a -> b -> b) -> b -> StreamK m a -> m b
306+ [A] foldlx' :: forall m a b x. Monad m => (x -> a -> x) -> x -> (x -> b) -> StreamK m a -> m b
307+ [A] foldlT :: (Monad m, Monad (s m), MonadTrans s) => (s m b -> a -> s m b) -> s m b -> StreamK m a -> s m b
308+ [A] foldlS :: (StreamK m b -> a -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
309+ [A] foldlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> StreamK m a -> m b
310+ [A] foldlM' :: Monad m => (b -> a -> m b) -> m b -> StreamK m a -> m b
311+ [A] foldl' :: Monad m => (b -> a -> b) -> b -> StreamK m a -> m b
312+ [A] foldStreamShared :: State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> StreamK m a -> m r
313+ [A] foldStream :: State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> StreamK m a -> m r
314+ [A] foldEither :: Monad m => Fold m a b -> StreamK m a -> m (Either (Fold m a b) (b, StreamK m a))
315+ [A] foldConcat :: Monad m => Producer m a b -> Fold m b c -> StreamK m a -> m (c, StreamK m a)
316+ [A] foldBreak :: Monad m => Fold m a b -> StreamK m a -> m (b, StreamK m a)
317+ [A] fold :: Monad m => Fold m a b -> StreamK m a -> m b
318+ [A] findM :: Monad m => (a -> m Bool) -> StreamK m a -> m (Maybe a)
319+ [A] findIndices :: (a -> Bool) -> StreamK m a -> StreamK m Int
320+ [A] find :: Monad m => (a -> Bool) -> StreamK m a -> m (Maybe a)
321+ [A] filter :: (a -> Bool) -> StreamK m a -> StreamK m a
322+ [A] evalStateT :: Monad m => m s -> StreamK (StateT s m) a -> StreamK m a
323+ [A] elem :: (Monad m, Eq a) => a -> StreamK m a -> m Bool
324+ [A] dropWhile :: (a -> Bool) -> StreamK m a -> StreamK m a
325+ [A] drop :: Int -> StreamK m a -> StreamK m a
326+ [A] drain :: Monad m => StreamK m a -> m ()
327+ [A] deleteBy :: (a -> a -> Bool) -> a -> StreamK m a -> StreamK m a
328+ [A] crossWith :: Monad m => (a -> b -> c) -> StreamK m a -> StreamK m b -> StreamK m c
329+ [A] crossApplyWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> StreamK m (a -> b) -> StreamK m a -> StreamK m b
330+ [A] crossApplySnd :: StreamK m a -> StreamK m b -> StreamK m b
331+ [A] crossApplyFst :: StreamK m a -> StreamK m b -> StreamK m a
332+ [A] crossApply :: StreamK m (a -> b) -> StreamK m a -> StreamK m b
333+ [A] cross :: Monad m => StreamK m a -> StreamK m b -> StreamK m (a, b)
334+ [A] consMBy :: Monad m => (StreamK m a -> StreamK m a -> StreamK m a) -> m a -> StreamK m a -> StreamK m a
335+ [A] consM :: Monad m => m a -> StreamK m a -> StreamK m a
336+ [A] consK :: YieldK m a -> StreamK m a -> StreamK m a
337+ [A] cons :: a -> StreamK m a -> StreamK m a
338+ [A] conjoin :: Monad m => StreamK m a -> StreamK m a -> StreamK m a
339+ [A] concatMapWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m a -> StreamK m b
340+ [A] concatMapEffect :: Monad m => (b -> StreamK m a) -> m b -> StreamK m a
341+ [A] concatMap :: (a -> StreamK m b) -> StreamK m a -> StreamK m b
342+ [A] concatIterateWith :: (StreamK m a -> StreamK m a -> StreamK m a) -> (a -> StreamK m a) -> StreamK m a -> StreamK m a
343+ [A] concatIterateScanWith :: Monad m => (StreamK m a -> StreamK m a -> StreamK m a) -> (b -> a -> m (b, StreamK m a)) -> m b -> StreamK m a -> StreamK m a
344+ [A] concatIterateLeftsWith :: b ~ Either a c => (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m b -> StreamK m b
345+ [A] concatEffect :: Monad m => m (StreamK m a) -> StreamK m a
346+ [A] buildSM :: Monad m => ((m a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a
347+ [A] buildS :: ((a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a
348+ [A] buildM :: Monad m => (forall r. (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
349+ [A] build :: forall m a. (forall b. (a -> b -> b) -> b -> b) -> StreamK m a
350+ [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> StreamK m a) -> StreamK m a
351+ [A] bindWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> StreamK m a -> (a -> StreamK m b) -> StreamK m b
352+ [A] before :: Monad m => m b -> StreamK m a -> StreamK m a
353+ [A] augmentSM :: Monad m => ((m a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a
354+ [A] augmentS :: ((a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a
355+ [A] append :: StreamK m a -> StreamK m a -> StreamK m a
356+ [A] any :: Monad m => (a -> Bool) -> StreamK m a -> m Bool
357+ [A] all :: Monad m => (a -> Bool) -> StreamK m a -> m Bool
358+ [A] (.:) :: a -> StreamK m a -> StreamK m a
359+ [A] (!!) :: Monad m => StreamK m a -> Int -> m (Maybe a)
360+[R] Streamly.Internal.Data.Stream.StreamK.Type
361+[R] Streamly.Internal.Data.Stream.StreamK.Transformer
362+[R] Streamly.Internal.Data.Stream.StreamK
363+[R] Streamly.Internal.Data.Stream.StreamD.Type
364+[R] Streamly.Internal.Data.Stream.StreamD.Transformer
365+[R] Streamly.Internal.Data.Stream.StreamD.Transform
366+[R] Streamly.Internal.Data.Stream.StreamD.Top
367+[R] Streamly.Internal.Data.Stream.StreamD.Step
368+[R] Streamly.Internal.Data.Stream.StreamD.Nesting
369+[R] Streamly.Internal.Data.Stream.StreamD.Lift
370+[R] Streamly.Internal.Data.Stream.StreamD.Generate
371+[R] Streamly.Internal.Data.Stream.StreamD.Exception
372+[R] Streamly.Internal.Data.Stream.StreamD.Eliminate
373+[R] Streamly.Internal.Data.Stream.StreamD.Container
374+[D] Streamly.Internal.Data.Stream.StreamD
375+[R] Streamly.Internal.Data.Stream.Common
376+[R] Streamly.Internal.Data.Stream.Chunked
377+[C] Streamly.Internal.Data.Stream
378+ [A] class Enum a => Enumerable a
379+ [A] Stream
380+ [A] UnStream :: (State StreamK m a -> s -> m (Step s a)) -> s -> Stream m a
381+ [A] Step
382+ [A] Yield :: a -> s -> Step s a
383+ [A] Stop :: Step s a
384+ [A] Skip :: s -> Step s a
385+ [A] InterleaveState
386+ [A] InterleaveSecondOnly :: s2 -> InterleaveState s1 s2
387+ [A] InterleaveSecond :: s1 -> s2 -> InterleaveState s1 s2
388+ [A] InterleaveFirstOnly :: s1 -> InterleaveState s1 s2
389+ [A] InterleaveFirst :: s1 -> s2 -> InterleaveState s1 s2
390+ [A] FoldManyPost
391+ [A] FoldManyPostYield :: b -> FoldManyPost s fs b a -> FoldManyPost s fs b a
392+ [A] FoldManyPostStart :: s -> FoldManyPost s fs b a
393+ [A] FoldManyPostLoop :: s -> fs -> FoldManyPost s fs b a
394+ [A] FoldManyPostDone :: FoldManyPost s fs b a
395+ [A] FoldMany
396+ [A] FoldManyYield :: b -> FoldMany s fs b a -> FoldMany s fs b a
397+ [A] FoldManyStart :: s -> FoldMany s fs b a
398+ [A] FoldManyLoop :: s -> fs -> FoldMany s fs b a
399+ [A] FoldManyFirst :: fs -> s -> FoldMany s fs b a
400+ [A] FoldManyDone :: FoldMany s fs b a
401+ [A] CrossStream
402+ [A] ConcatUnfoldInterleaveState
403+ [A] ConcatUnfoldInterleaveOuter :: o -> [i] -> ConcatUnfoldInterleaveState o i
404+ [A] ConcatUnfoldInterleaveInnerR :: [i] -> [i] -> ConcatUnfoldInterleaveState o i
405+ [A] ConcatUnfoldInterleaveInnerL :: [i] -> [i] -> ConcatUnfoldInterleaveState o i
406+ [A] ConcatUnfoldInterleaveInner :: o -> [i] -> ConcatUnfoldInterleaveState o i
407+ [A] ConcatMapUState
408+ [A] ConcatMapUOuter :: o -> ConcatMapUState o i
409+ [A] ConcatMapUInner :: o -> i -> ConcatMapUState o i
410+ [A] AppendState
411+ [A] AppendSecond :: s2 -> AppendState s1 s2
412+ [A] AppendFirst :: s1 -> AppendState s1 s2
413+ [A] pattern Stream :: (State StreamK m a -> s -> m (Step s a)) -> s -> Stream m a
414+ [A] zipWithM :: Monad m => (a -> b -> m c) -> Stream m a -> Stream m b -> Stream m c
415+ [A] zipWith :: Monad m => (a -> b -> c) -> Stream m a -> Stream m b -> Stream m c
416+ [A] wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
417+ [A] with :: Monad m => (Stream m a -> Stream m (s, a)) -> (((s, a) -> b) -> Stream m (s, a) -> Stream m (s, a)) -> ((s, a) -> b) -> Stream m a -> Stream m a
418+ [A] usingStateT :: Monad m => m s -> (Stream (StateT s m) a -> Stream (StateT s m) a) -> Stream m a -> Stream m a
419+ [A] usingReaderT :: Monad m => m r -> (Stream (ReaderT r m) a -> Stream (ReaderT r m) a) -> Stream m a -> Stream m a
420+ [A] uniqBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a
421+ [A] uniq :: (Eq a, Monad m) => Stream m a -> Stream m a
422+ [A] unionWithStreamGenericBy :: MonadIO m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
423+ [A] unionWithStreamAscBy :: (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
424+ [A] unfoldrM :: Monad m => (s -> m (Maybe (a, s))) -> s -> Stream m a
425+ [A] unfoldr :: Monad m => (s -> Maybe (a, s)) -> s -> Stream m a
426+ [A] unfoldRoundRobin :: Monad m => Unfold m a b -> Stream m a -> Stream m b
427+ [A] unfoldMany :: Monad m => Unfold m a b -> Stream m a -> Stream m b
428+ [A] unfoldIterateDfs :: Monad m => Unfold m a a -> Stream m a -> Stream m a
429+ [A] unfoldIterateBfsRev :: Monad m => Unfold m a a -> Stream m a -> Stream m a
430+ [A] unfoldIterateBfs :: Monad m => Unfold m a a -> Stream m a -> Stream m a
431+ [A] unfoldInterleave :: Monad m => Unfold m a b -> Stream m a -> Stream m b
432+ [A] unfold :: Applicative m => Unfold m a b -> a -> Stream m b
433+ [A] uncons :: Monad m => Stream m a -> m (Maybe (a, Stream m a))
434+ [A] unCross :: CrossStream m a -> Stream m a
435+ [A] transform :: Monad m => Pipe m a b -> Stream m a -> Stream m b
436+ [A] trace_ :: Monad m => m b -> Stream m a -> Stream m a
437+ [A] trace :: Monad m => (a -> m b) -> Stream m a -> Stream m a
438+ [A] toStreamK :: Monad m => Stream m a -> StreamK m a
439+ [A] toListRev :: Monad m => Stream m a -> m [a]
440+ [A] toList :: Monad m => Stream m a -> m [a]
441+ [A] timestamped :: MonadIO m => Stream m a -> Stream m (AbsTime, a)
442+ [A] timestampWith :: MonadIO m => Double -> Stream m a -> Stream m (AbsTime, a)
443+ [A] timesWith :: MonadIO m => Double -> Stream m (AbsTime, RelTime64)
444+ [A] times :: MonadIO m => Stream m (AbsTime, RelTime64)
445+ [A] timeout :: AbsTime -> t m ()
446+ [A] timeIndexed :: MonadIO m => Stream m a -> Stream m (RelTime64, a)
447+ [A] timeIndexWith :: MonadIO m => Double -> Stream m a -> Stream m (RelTime64, a)
448+ [A] the :: (Eq a, Monad m) => Stream m a -> m (Maybe a)
449+ [A] tapOffsetEvery :: Monad m => Int -> Int -> Fold m a b -> Stream m a -> Stream m a
450+ [A] tap :: Monad m => Fold m a b -> Stream m a -> Stream m a
451+ [A] takeWhileM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
452+ [A] takeWhileLast :: (a -> Bool) -> Stream m a -> Stream m a
453+ [A] takeWhileAround :: (a -> Bool) -> Stream m a -> Stream m a
454+ [A] takeWhile :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
455+ [A] takeEndByM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
456+ [A] takeEndBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
457+ [A] take :: Applicative m => Int -> Stream m a -> Stream m a
458+ [A] tail :: Monad m => Stream m a -> m (Maybe (Stream m a))
459+ [A] stripSuffixUnbox :: (MonadIO m, Eq a, Unbox a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
460+ [A] stripSuffix :: (Monad m, Eq a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
461+ [A] stripPrefix :: (Monad m, Eq a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
462+ [A] strideFromThen :: Monad m => Int -> Int -> Stream m a -> Stream m a
463+ [A] splitOnSuffixSeqAny :: [Array a] -> Fold m a b -> Stream m a -> Stream m b
464+ [A] splitOnSuffixSeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) => Bool -> Array a -> Fold m a b -> Stream m a -> Stream m b
465+ [A] splitOnSeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) => Array a -> Fold m a b -> Stream m a -> Stream m b
466+ [A] splitOnPrefix :: (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
467+ [A] splitOnAny :: [Array a] -> Fold m a b -> Stream m a -> Stream m b
468+ [A] splitOn :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
469+ [A] splitInnerBySuffix :: (Monad m, Eq (f a), Monoid (f a)) => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
470+ [A] splitInnerBy :: Monad m => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
471+ [A] slicesBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
472+ [A] sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
473+ [A] sequence :: Monad m => Stream m (m a) -> Stream m a
474+ [A] scanlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> Stream m b
475+ [A] scanlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> Stream m b
476+ [A] scanlMAfter' :: Monad m => (b -> a -> m b) -> m b -> (b -> m b) -> Stream m a -> Stream m b
477+ [A] scanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
478+ [A] scanlM :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
479+ [A] scanl1M' :: Monad m => (a -> a -> m a) -> Stream m a -> Stream m a
480+ [A] scanl1M :: Monad m => (a -> a -> m a) -> Stream m a -> Stream m a
481+ [A] scanl1' :: Monad m => (a -> a -> a) -> Stream m a -> Stream m a
482+ [A] scanl1 :: Monad m => (a -> a -> a) -> Stream m a -> Stream m a
483+ [A] scanl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
484+ [A] scanl :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
485+ [A] scanMaybe :: Monad m => Fold m a (Maybe b) -> Stream m a -> Stream m b
486+ [A] scanMany :: Monad m => Fold m a b -> Stream m a -> Stream m b
487+ [A] scan :: Monad m => Fold m a b -> Stream m a -> Stream m b
488+ [A] runStateT :: Monad m => m s -> Stream (StateT s m) a -> Stream m (s, a)
489+ [A] runReaderT :: Monad m => m s -> Stream (ReaderT s m) a -> Stream m a
490+ [A] runInnerWithState :: Monad m => (forall b. s -> t m b -> m (b, s)) -> m s -> Stream (t m) a -> Stream m (s, a)
491+ [A] runInnerWith :: Monad m => (forall b. t m b -> m b) -> Stream (t m) a -> Stream m a
492+ [A] roundRobin :: Monad m => Stream m a -> Stream m a -> Stream m a
493+ [A] rollingMapM :: Monad m => (Maybe a -> a -> m b) -> Stream m a -> Stream m b
494+ [A] rollingMap2 :: Monad m => (a -> a -> b) -> Stream m a -> Stream m b
495+ [A] rollingMap :: Monad m => (Maybe a -> a -> b) -> Stream m a -> Stream m b
496+ [A] reverseUnbox :: (MonadIO m, Unbox a) => Stream m a -> Stream m a
497+ [A] reverse :: Monad m => Stream m a -> Stream m a
498+ [A] replicateM :: Monad m => Int -> m a -> Stream m a
499+ [A] replicate :: Monad m => Int -> a -> Stream m a
500+ [A] repeated :: Stream m a -> Stream m a
501+ [A] repeatM :: Monad m => m a -> Stream m a
502+ [A] repeat :: Monad m => a -> Stream m a
503+ [A] relTimesWith :: MonadIO m => Double -> Stream m RelTime64
504+ [A] relTimes :: MonadIO m => Stream m RelTime64
505+ [A] refoldMany :: Monad m => Refold m x a b -> m x -> Stream m a -> Stream m b
506+ [A] refoldIterateM :: Monad m => Refold m b a b -> m b -> Stream m a -> Stream m b
507+ [A] reduceIterateBfs :: Monad m => (a -> a -> m a) -> Stream m a -> m (Maybe a)
508+ [A] reassembleBy :: Fold m a b -> (a -> a -> Int) -> Stream m a -> Stream m b
509+ [A] prune :: (a -> Bool) -> Stream m a -> Stream m a
510+ [A] prescanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
511+ [A] prescanl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
512+ [A] postscanlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> Stream m b
513+ [A] postscanlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> Stream m b
514+ [A] postscanlMAfter' :: Monad m => (b -> a -> m b) -> m b -> (b -> m b) -> Stream m a -> Stream m b
515+ [A] postscanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
516+ [A] postscanlM :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
517+ [A] postscanl' :: Monad m => (a -> b -> a) -> a -> Stream m b -> Stream m a
518+ [A] postscanl :: Monad m => (a -> b -> a) -> a -> Stream m b -> Stream m a
519+ [A] postscan :: Monad m => Fold m a b -> Stream m a -> Stream m b
520+ [A] parseSequence :: Stream m (Parser a m b) -> Stream m a -> Stream m b
521+ [A] parseManyTill :: Parser a m b -> Parser a m x -> Stream m a -> Stream m b
522+ [A] parseManyD :: Monad m => Parser a m b -> Stream m a -> Stream m (Either ParseError b)
523+ [A] parseMany :: Monad m => Parser a m b -> Stream m a -> Stream m (Either ParseError b)
524+ [A] parseIterateD :: Monad m => (b -> Parser a m b) -> b -> Stream m a -> Stream m (Either ParseError b)
525+ [A] parseIterate :: Monad m => (b -> Parser a m b) -> b -> Stream m a -> Stream m (Either ParseError b)
526+ [A] parseD :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b)
527+ [A] parseBreakD :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b, Stream m a)
528+ [A] parseBreak :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b, Stream m a)
529+ [A] parse :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b)
530+ [A] onException :: MonadCatch m => m b -> Stream m a -> Stream m a
531+ [A] null :: Monad m => Stream m a -> m Bool
532+ [A] nub :: (Monad m, Ord a) => Stream m a -> Stream m a
533+ [A] notElem :: (Monad m, Eq a) => a -> Stream m a -> m Bool
534+ [A] nilM :: Applicative m => m b -> Stream m a
535+ [A] nil :: Applicative m => Stream m a
536+ [A] morphInner :: Monad n => (forall x. m x -> n x) -> Stream m a -> Stream n a
537+ [A] mkCross :: Stream m a -> CrossStream m a
538+ [A] minimumBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> m (Maybe a)
539+ [A] minimum :: (Monad m, Ord a) => Stream m a -> m (Maybe a)
540+ [A] mergeMinBy :: (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
541+ [A] mergeFstBy :: (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
542+ [A] mergeByM :: Monad m => (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
543+ [A] mergeBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
544+ [A] maximumBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> m (Maybe a)
545+ [A] maximum :: (Monad m, Ord a) => Stream m a -> m (Maybe a)
546+ [A] mapMaybeM :: Monad m => (a -> m (Maybe b)) -> Stream m a -> Stream m b
547+ [A] mapMaybe :: Monad m => (a -> Maybe b) -> Stream m a -> Stream m b
548+ [A] mapM_ :: Monad m => (a -> m b) -> Stream m a -> m ()
549+ [A] mapM :: Monad m => (a -> m b) -> Stream m a -> Stream m b
550+ [A] map :: Monad m => (a -> b) -> Stream m a -> Stream m b
551+ [A] lookup :: (Monad m, Eq a) => a -> Stream m (a, b) -> m (Maybe b)
552+ [A] liftInnerWith :: Monad (t m) => (forall b. m b -> t m b) -> Stream m a -> Stream (t m) a
553+ [A] liftInner :: (Monad m, MonadTrans t, Monad (t m)) => Stream m a -> Stream (t m) a
554+ [A] last :: Monad m => Stream m a -> m (Maybe a)
555+ [A] joinOuterGeneric :: MonadIO m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (Maybe a, Maybe b)
556+ [A] joinOuterAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (Maybe a, Maybe b)
557+ [A] joinOuter :: (Ord k, MonadIO m) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, Maybe a, Maybe b)
558+ [A] joinLeftGeneric :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
559+ [A] joinLeftAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
560+ [A] joinLeft :: (Ord k, Monad m) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, Maybe b)
561+ [A] joinInnerGeneric :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, b)
562+ [A] joinInnerAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, b)
563+ [A] joinInner :: (Monad m, Ord k) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, b)
564+ [A] iterateM :: Monad m => (a -> m a) -> m a -> Stream m a
565+ [A] iterate :: Monad m => (a -> a) -> a -> Stream m a
566+ [A] isSuffixOfUnbox :: (MonadIO m, Eq a, Unbox a) => Stream m a -> Stream m a -> m Bool
567+ [A] isSuffixOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
568+ [A] isSubsequenceOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
569+ [A] isPrefixOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
570+ [A] isInfixOf :: (MonadIO m, Eq a, Enum a, Storable a, Unbox a) => Stream m a -> Stream m a -> m Bool
571+ [A] intersperseM_ :: Monad m => m b -> Stream m a -> Stream m a
572+ [A] intersperseMWith :: Int -> m a -> Stream m a -> Stream m a
573+ [A] intersperseMSuffix_ :: Monad m => m b -> Stream m a -> Stream m a
574+ [A] intersperseMSuffixWith :: forall m a. Monad m => Int -> m a -> Stream m a -> Stream m a
575+ [A] intersperseMSuffix :: forall m a. Monad m => m a -> Stream m a -> Stream m a
576+ [A] intersperseMPrefix_ :: Monad m => m b -> Stream m a -> Stream m a
577+ [A] intersperseM :: Monad m => m a -> Stream m a -> Stream m a
578+ [A] intersperse :: Monad m => a -> Stream m a -> Stream m a
579+ [A] intersectBySorted :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
580+ [A] interposeSuffixM :: Monad m => m c -> Unfold m b c -> Stream m b -> Stream m c
581+ [A] interposeSuffix :: Monad m => c -> Unfold m b c -> Stream m b -> Stream m c
582+ [A] interposeM :: Monad m => m c -> Unfold m b c -> Stream m b -> Stream m c
583+ [A] interpose :: Monad m => c -> Unfold m b c -> Stream m b -> Stream m c
584+ [A] interleaveMin :: Monad m => Stream m a -> Stream m a -> Stream m a
585+ [A] interleaveFstSuffix :: Monad m => Stream m a -> Stream m a -> Stream m a
586+ [A] interleaveFst :: Monad m => Stream m a -> Stream m a -> Stream m a
587+ [A] interleave :: Monad m => Stream m a -> Stream m a -> Stream m a
588+ [A] intercalateSuffix :: Monad m => Unfold m b c -> b -> Stream m b -> Stream m c
589+ [A] intercalate :: Monad m => Unfold m b c -> b -> Stream m b -> Stream m c
590+ [A] insertBy :: Monad m => (a -> a -> Ordering) -> a -> Stream m a -> Stream m a
591+ [A] indexedR :: Monad m => Int -> Stream m a -> Stream m (Int, a)
592+ [A] indexed :: Monad m => Stream m a -> Stream m (Int, a)
593+ [A] headElse :: Monad m => a -> Stream m a -> m a
594+ [A] head :: Monad m => Stream m a -> m (Maybe a)
595+ [A] handle :: (MonadCatch m, Exception e) => (e -> m (Stream m a)) -> Stream m a -> Stream m a
596+ [A] groupsWhile :: Monad m => (a -> a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
597+ [A] groupsRollingBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
598+ [A] groupsOf :: Monad m => Int -> Fold m a b -> Stream m a -> Stream m b
599+ [A] gintercalateSuffix :: Monad m => Unfold m a c -> Stream m a -> Unfold m b c -> Stream m b -> Stream m c
600+ [A] gintercalate :: Monad m => Unfold m a c -> Stream m a -> Unfold m b c -> Stream m b -> Stream m c
601+ [A] ghandle :: (MonadCatch m, Exception e) => (e -> Stream m a -> m (Stream m a)) -> Stream m a -> Stream m a
602+ [A] generateM :: Monad m => Int -> (Int -> m a) -> Stream m a
603+ [A] generate :: Monad m => Int -> (Int -> a) -> Stream m a
604+ [A] generalizeInner :: Monad m => Stream Identity a -> Stream m a
605+ [A] gbracket_ :: Monad m => m c -> (c -> m d) -> (c -> e -> Stream m b -> m (Stream m b)) -> (forall s. m s -> m (Either e s)) -> (c -> Stream m b) -> Stream m b
606+ [A] gbracket :: MonadIO m => IO c -> (c -> IO d1) -> (c -> e -> Stream m b -> IO (Stream m b)) -> (c -> IO d2) -> (forall s. m s -> m (Either e s)) -> (c -> Stream m b) -> Stream m b
607+ [A] fromStreamK :: Applicative m => StreamK m a -> Stream m a
608+ [A] fromPure :: Applicative m => a -> Stream m a
609+ [A] fromPtrN :: (Monad m, Storable a) => Int -> Ptr a -> Stream m a
610+ [A] fromPtr :: forall m a. (Monad m, Storable a) => Ptr a -> Stream m a
611+ [A] fromListM :: Monad m => [m a] -> Stream m a
612+ [A] fromList :: Applicative m => [a] -> Stream m a
613+ [A] fromIndicesM :: Monad m => (Int -> m a) -> Stream m a
614+ [A] fromIndices :: Monad m => (Int -> a) -> Stream m a
615+ [A] fromFoldableM :: (Monad m, Foldable f) => f (m a) -> Stream m a
616+ [A] fromFoldable :: (Monad m, Foldable f) => f a -> Stream m a
617+ [A] fromEffect :: Applicative m => m a -> Stream m a
618+ [A] fromByteStr# :: Monad m => Addr# -> Stream m Word8
619+ [A] foldrT :: (Monad m, Monad (t m), MonadTrans t) => (a -> t m b -> t m b) -> t m b -> Stream m a -> t m b
620+ [A] foldrS :: Monad m => (a -> Stream m b -> Stream m b) -> Stream m b -> Stream m a -> Stream m b
621+ [A] foldrMx :: Monad m => (a -> m x -> m x) -> m x -> (m x -> m b) -> Stream m a -> m b
622+ [A] foldrM :: Monad m => (a -> m b -> m b) -> m b -> Stream m a -> m b
623+ [A] foldr1 :: Monad m => (a -> a -> a) -> Stream m a -> m (Maybe a)
624+ [A] foldr :: Monad m => (a -> b -> b) -> b -> Stream m a -> m b
625+ [A] foldlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> m b
626+ [A] foldlT :: (Monad m, Monad (s m), MonadTrans s) => (s m b -> a -> s m b) -> s m b -> Stream m a -> s m b
627+ [A] foldlS :: Monad m => (Stream m b -> a -> Stream m b) -> Stream m b -> Stream m a -> Stream m b
628+ [A] foldlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> m b
629+ [A] foldlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> m b
630+ [A] foldl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> m b
631+ [A] foldSequence :: Stream m (Fold m a b) -> Stream m a -> Stream m b
632+ [A] foldManyPost :: Monad m => Fold m a b -> Stream m a -> Stream m b
633+ [A] foldMany :: Monad m => Fold m a b -> Stream m a -> Stream m b
634+ [A] foldIterateM :: Monad m => (b -> m (Fold m a b)) -> m b -> Stream m a -> Stream m b
635+ [A] foldIterateBfs :: Fold m a (Either a a) -> Stream m a -> m (Maybe a)
636+ [A] foldEither :: Monad m => Fold m a b -> Stream m a -> m (Either (Fold m a b) (b, Stream m a))
637+ [A] foldBreak :: Monad m => Fold m a b -> Stream m a -> m (b, Stream m a)
638+ [A] foldAddLazy :: Monad m => Fold m a b -> Stream m a -> Fold m a b
639+ [A] foldAdd :: Monad m => Fold m a b -> Stream m a -> m (Fold m a b)
640+ [A] fold :: Monad m => Fold m a b -> Stream m a -> m b
641+ [A] findM :: Monad m => (a -> m Bool) -> Stream m a -> m (Maybe a)
642+ [A] findIndices :: Monad m => (a -> Bool) -> Stream m a -> Stream m Int
643+ [A] find :: Monad m => (a -> Bool) -> Stream m a -> m (Maybe a)
644+ [A] finallyUnsafe :: MonadCatch m => m b -> Stream m a -> Stream m a
645+ [A] finallyIO :: (MonadIO m, MonadCatch m) => IO b -> Stream m a -> Stream m a
646+ [A] filterM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
647+ [A] filterInStreamGenericBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
648+ [A] filterInStreamAscBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
649+ [A] filter :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
650+ [A] evalStateT :: Monad m => m s -> Stream (StateT s m) a -> Stream m a
651+ [A] eqBy :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> m Bool
652+ [A] enumerateTo :: (Monad m, Bounded a, Enumerable a) => a -> Stream m a
653+ [A] enumerateFromToSmall :: (Monad m, Enum a) => a -> a -> Stream m a
654+ [A] enumerateFromToIntegral :: (Monad m, Integral a) => a -> a -> Stream m a
655+ [A] enumerateFromToFractional :: (Monad m, Fractional a, Ord a) => a -> a -> Stream m a
656+ [A] enumerateFromTo :: (Enumerable a, Monad m) => a -> a -> Stream m a
657+ [A] enumerateFromThenToSmall :: (Monad m, Enum a) => a -> a -> a -> Stream m a
658+ [A] enumerateFromThenToIntegral :: (Monad m, Integral a) => a -> a -> a -> Stream m a
659+ [A] enumerateFromThenToFractional :: (Monad m, Fractional a, Ord a) => a -> a -> a -> Stream m a
660+ [A] enumerateFromThenTo :: (Enumerable a, Monad m) => a -> a -> a -> Stream m a
661+ [A] enumerateFromThenSmallBounded :: (Monad m, Enumerable a, Bounded a) => a -> a -> Stream m a
662+ [A] enumerateFromThenNum :: (Monad m, Num a) => a -> a -> Stream m a
663+ [A] enumerateFromThenIntegral :: (Monad m, Integral a, Bounded a) => a -> a -> Stream m a
664+ [A] enumerateFromThenFractional :: (Monad m, Fractional a) => a -> a -> Stream m a
665+ [A] enumerateFromThen :: (Enumerable a, Monad m) => a -> a -> Stream m a
666+ [A] enumerateFromStepNum :: (Monad m, Num a) => a -> a -> Stream m a
667+ [A] enumerateFromStepIntegral :: (Integral a, Monad m) => a -> a -> Stream m a
668+ [A] enumerateFromNum :: (Monad m, Num a) => a -> Stream m a
669+ [A] enumerateFromIntegral :: (Monad m, Integral a, Bounded a) => a -> Stream m a
670+ [A] enumerateFromFractional :: (Monad m, Fractional a) => a -> Stream m a
671+ [A] enumerateFromBounded :: (Monad m, Enumerable a, Bounded a) => a -> Stream m a
672+ [A] enumerateFrom :: (Enumerable a, Monad m) => a -> Stream m a
673+ [A] enumerate :: (Monad m, Bounded a, Enumerable a) => Stream m a
674+ [A] elemIndices :: (Monad m, Eq a) => a -> Stream m a -> Stream m Int
675+ [A] elem :: (Monad m, Eq a) => a -> Stream m a -> m Bool
676+ [A] durations :: Double -> t m RelTime64
677+ [A] dropWhileM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
678+ [A] dropWhileLast :: (a -> Bool) -> Stream m a -> Stream m a
679+ [A] dropWhileAround :: (a -> Bool) -> Stream m a -> Stream m a
680+ [A] dropWhile :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
681+ [A] dropSuffix :: Stream m a -> Stream m a -> Stream m a
682+ [A] dropPrefix :: Stream m a -> Stream m a -> Stream m a
683+ [A] dropLast :: Int -> Stream m a -> Stream m a
684+ [A] dropInfix :: Stream m a -> Stream m a -> Stream m a
685+ [A] drop :: Monad m => Int -> Stream m a -> Stream m a
686+ [A] drain :: Monad m => Stream m a -> m ()
687+ [A] deleteInStreamGenericBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
688+ [A] deleteInStreamAscBy :: (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
689+ [A] deleteBy :: Monad m => (a -> a -> Bool) -> a -> Stream m a -> Stream m a
690+ [A] delayPre :: MonadIO m => Double -> Stream m a -> Stream m a
691+ [A] delayPost :: MonadIO m => Double -> Stream m a -> Stream m a
692+ [A] delay :: MonadIO m => Double -> Stream m a -> Stream m a
693+ [A] crossWith :: Monad m => (a -> b -> c) -> Stream m a -> Stream m b -> Stream m c
694+ [A] crossApplySnd :: Functor f => Stream f a -> Stream f b -> Stream f b
695+ [A] crossApplyFst :: Functor f => Stream f a -> Stream f b -> Stream f a
696+ [A] crossApply :: Functor f => Stream f (a -> b) -> Stream f a -> Stream f b
697+ [A] cross :: Monad m => Stream m a -> Stream m b -> Stream m (a, b)
698+ [A] consM :: Applicative m => m a -> Stream m a -> Stream m a
699+ [A] cons :: Applicative m => a -> Stream m a -> Stream m a
700+ [A] concatMapM :: Monad m => (a -> m (Stream m b)) -> Stream m a -> Stream m b
701+ [A] concatMap :: Monad m => (a -> Stream m b) -> Stream m a -> Stream m b
702+ [A] concatIterateScan :: Monad m => (b -> a -> m b) -> (b -> m (Maybe (b, Stream m a))) -> b -> Stream m a
703+ [A] concatIterateDfs :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
704+ [A] concatIterateBfsRev :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
705+ [A] concatIterateBfs :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
706+ [A] concatEffect :: Monad m => m (Stream m a) -> Stream m a
707+ [A] concat :: Monad m => Stream m (Stream m a) -> Stream m a
708+ [A] cmpBy :: Monad m => (a -> b -> Ordering) -> Stream m a -> Stream m b -> m Ordering
709+ [A] catRights :: Monad m => Stream m (Either a b) -> Stream m b
710+ [A] catMaybes :: Monad m => Stream m (Maybe a) -> Stream m a
711+ [A] catLefts :: Monad m => Stream m (Either a b) -> Stream m a
712+ [A] catEithers :: Monad m => Stream m (Either a a) -> Stream m a
713+ [A] bracketUnsafe :: MonadCatch m => m b -> (b -> m c) -> (b -> Stream m a) -> Stream m a
714+ [A] bracketIO3 :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> IO d) -> (b -> IO e) -> (b -> Stream m a) -> Stream m a
715+ [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> Stream m a) -> Stream m a
716+ [A] before :: Monad m => m b -> Stream m a -> Stream m a
717+ [A] append :: Monad m => Stream m a -> Stream m a -> Stream m a
718+ [A] any :: Monad m => (a -> Bool) -> Stream m a -> m Bool
719+ [A] all :: Monad m => (a -> Bool) -> Stream m a -> m Bool
720+ [A] afterUnsafe :: Monad m => m b -> Stream m a -> Stream m a
721+ [A] afterIO :: MonadIO m => IO b -> Stream m a -> Stream m a
722+ [A] absTimesWith :: MonadIO m => Double -> Stream m AbsTime
723+ [A] absTimes :: MonadIO m => Stream m AbsTime
724+ [A] (!!) :: Monad m => Stream m a -> Int -> m (Maybe a)
725+[R] Streamly.Internal.Data.Ring.Unboxed
726+[A] Streamly.Internal.Data.Ring.Generic
727+ [A] Ring
728+ [A] [ringMax] :: Ring a -> !Int
729+ [A] [ringHead] :: Ring a -> !Int
730+ [A] [ringArr] :: Ring a -> MutArray a
731+ [A] Ring :: MutArray a -> !Int -> !Int -> Ring a
732+ [A] writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
733+ [A] unsafeInsertRingWith :: Ring a -> a -> IO Int
734+ [A] toStreamWith :: Int -> Ring a -> Stream m a
735+ [A] toMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
736+ [A] seek :: MonadIO m => Int -> Ring a -> m (Ring a)
737+ [A] createRing :: MonadIO m => Int -> m (Ring a)
738+ [A] copyToMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
739+[C] Streamly.Internal.Data.Ring
740+ [C] Ring
741+ [A] [ringStart] :: Ring a -> {-# UNPACK #-} !ForeignPtr a
742+ [R] [ringMax] :: Ring a -> !Int
743+ [R] [ringHead] :: Ring a -> !Int
744+ [A] [ringBound] :: Ring a -> {-# UNPACK #-} !Ptr a
745+ [R] [ringArr] :: Ring a -> MutArray a
746+ [C] Ring
747+ [O] Ring :: MutArray a -> !Int -> !Int -> Ring a
748+ [N] Ring :: {-# UNPACK #-} !ForeignPtr a -> {-# UNPACK #-} !Ptr a -> Ring a
749+ [A] GHC.Show.Show
750+ [A] instance (GHC.Show.Show a, GHC.Show.Show b, GHC.Show.Show c, GHC.Show.Show d) => GHC.Show.Show (Streamly.Internal.Data.Ring.Tuple4' a b c d)
751+ [A] writeN :: Int -> Fold m a (Ring a)
752+ [R] writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
753+ [R] unsafeInsertRingWith :: Ring a -> a -> IO Int
754+ [A] unsafeInsert :: Storable a => Ring a -> Ptr a -> a -> IO (Ptr a)
755+ [A] unsafeFoldRingNM :: forall m a b. (MonadIO m, Storable a) => Int -> Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
756+ [A] unsafeFoldRingM :: forall m a b. (MonadIO m, Storable a) => Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
757+ [A] unsafeFoldRingFullM :: forall m a b. (MonadIO m, Storable a) => Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
758+ [A] unsafeFoldRing :: forall a b. Storable a => Ptr a -> (b -> a -> b) -> b -> Ring a -> b
759+ [A] unsafeEqArrayN :: Ring a -> Ptr a -> Array a -> Int -> Bool
760+ [A] unsafeEqArray :: Ring a -> Ptr a -> Array a -> Bool
761+ [R] toStreamWith :: Int -> Ring a -> Stream m a
762+ [R] toMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
763+ [A] startOf :: Ring a -> Ptr a
764+ [A] slidingWindowWith :: forall m a b. (MonadIO m, Storable a, Unbox a) => Int -> Fold m ((a, Maybe a), m (MutArray a)) b -> Fold m a b
765+ [A] slidingWindow :: forall m a b. (MonadIO m, Storable a, Unbox a) => Int -> Fold m (a, Maybe a) b -> Fold m a b
766+ [A] slide :: Ring a -> a -> m (Ring a)
767+ [R] seek :: MonadIO m => Int -> Ring a -> m (Ring a)
768+ [A] ringsOf :: Int -> Stream m a -> Stream m (MutArray a)
769+ [A] readRev :: Unfold m (MutArray a) a
770+ [A] read :: forall m a. (MonadIO m, Storable a) => Unfold m (Ring a, Ptr a, Int) a
771+ [A] putIndex :: Ring a -> Int -> a -> m ()
772+ [A] newRing :: Int -> m (Ring a)
773+ [A] new :: forall a. Storable a => Int -> IO (Ring a, Ptr a)
774+ [A] moveBy :: forall a. Storable a => Int -> Ring a -> Ptr a -> Ptr a
775+ [A] modifyIndex :: Ring a -> Int -> (a -> (a, b)) -> m b
776+ [A] length :: Ring a -> Int
777+ [A] getIndexUnsafe :: Ring a -> Int -> m a
778+ [A] getIndexRev :: Ring a -> Int -> m a
779+ [A] getIndex :: Ring a -> Int -> m a
780+ [A] fromArray :: MutArray a -> Ring a
781+ [R] createRing :: MonadIO m => Int -> m (Ring a)
782+ [A] castUnsafe :: Ring a -> Ring b
783+ [A] cast :: forall a b. Storable b => Ring a -> Maybe (Ring b)
784+ [A] bytesFree :: Ring a -> Int
785+ [A] byteLength :: Ring a -> Int
786+ [A] byteCapacity :: Ring a -> Int
787+ [A] asBytes :: Ring a -> Ring Word8
788+ [A] advance :: forall a. Storable a => Ring a -> Ptr a -> Ptr a
789+[R] Streamly.Internal.Data.Producer.Type
790+[R] Streamly.Internal.Data.Producer.Source
791+[C] Streamly.Internal.Data.Producer
792+ [A] Source
793+ [A] unread :: [b] -> Source a b -> Source a b
794+ [A] translate :: Functor m => (a -> c) -> (c -> a) -> Producer m c b -> Producer m a b
795+ [A] source :: Maybe a -> Source a b
796+ [A] producer :: Monad m => Producer m a b -> Producer m (Source a b) b
797+ [A] parseManyD :: Monad m => Parser a m b -> Producer m (Source x a) a -> Producer m (Source x a) (Either ParseError b)
798+ [A] parseMany :: Monad m => Parser a m b -> Producer m (Source x a) a -> Producer m (Source x a) (Either ParseError b)
799+ [A] parse :: Monad m => Parser a m b -> Producer m (Source s a) a -> Source s a -> m (Either ParseError b, Source s a)
800+ [A] lmap :: (a -> a) -> Producer m a b -> Producer m a b
801+ [A] isEmpty :: Source a b -> Bool
802+[R] Streamly.Internal.Data.Pipe.Type
803+[C] Streamly.Internal.Data.Pipe
804+ [A] Step
805+ [A] Yield :: a -> s -> Step s a
806+ [A] Continue :: s -> Step s a
807+ [A] PipeState
808+ [A] Produce :: s2 -> PipeState s1 s2
809+ [A] Consume :: s1 -> PipeState s1 s2
810+ [C] Pipe
811+ [A] Pipe :: (s1 -> a -> m (Step (PipeState s1 s2) b)) -> (s2 -> m (Step (PipeState s1 s2) b)) -> s1 -> Pipe m a b
812+[A] Streamly.Internal.Data.ParserK
813+ [A] Step
814+ [A] Partial :: !Int -> (Input a -> m (Step a m r)) -> Step a m r
815+ [A] Error :: !Int -> String -> Step a m r
816+ [A] Done :: !Int -> r -> Step a m r
817+ [A] Continue :: !Int -> (Input a -> m (Step a m r)) -> Step a m r
818+ [A] ParseResult
819+ [A] Success :: !Int -> !b -> ParseResult b
820+ [A] Failure :: !Int -> !String -> ParseResult b
821+ [A] Input
822+ [A] None :: Input a
823+ [A] Chunk :: a -> Input a
824+ [A] ParserK
825+ [A] [runParser] :: ParserK a m b -> forall r. (ParseResult b -> Int -> Input a -> m (Step a m r)) -> Int -> Int -> Input a -> m (Step a m r)
826+ [A] MkParser :: (forall r. (ParseResult b -> Int -> Input a -> m (Step a m r)) -> Int -> Int -> Input a -> m (Step a m r)) -> ParserK a m b
827+ [A] fromPure :: b -> ParserK a m b
828+ [A] fromEffect :: Monad m => m b -> ParserK a m b
829+ [A] die :: String -> ParserK a m b
830+ [A] adaptCG :: Monad m => Parser a m b -> ParserK (Array a) m b
831+ [A] adaptC :: (Monad m, Unbox a) => Parser a m b -> ParserK (Array a) m b
832+ [A] adapt :: Monad m => Parser a m b -> ParserK a m b
833+[R] Streamly.Internal.Data.Parser.ParserK.Type
834+[R] Streamly.Internal.Data.Parser.ParserD.Type
835+[R] Streamly.Internal.Data.Parser.ParserD.Tee
836+[R] Streamly.Internal.Data.Parser.ParserD
837+[C] Streamly.Internal.Data.Parser
838+ [A] Step
839+ [A] Partial :: !Int -> !s -> Step s b
840+ [A] Error :: !String -> Step s b
841+ [A] Done :: !Int -> !b -> Step s b
842+ [A] Continue :: !Int -> !s -> Step s b
843+ [A] Parser
844+ [A] Parser :: (s -> a -> m (Step s b)) -> m (Initial s b) -> (s -> m (Step s b)) -> Parser a m b
845+ [A] Initial
846+ [A] IPartial :: !s -> Initial s b
847+ [A] IError :: !String -> Initial s b
848+ [A] IDone :: !b -> Initial s b
849+ [A] GHC.Show.Show
850+ [A] instance (GHC.Show.Show a, GHC.Show.Show b) => GHC.Show.Show (Streamly.Internal.Data.Parser.Tuple'Fused a b)
851+ [A] ParseError
852+ [A] ParseError :: String -> ParseError
853+ [A] zipWithM :: Monad m => (a -> b -> m c) -> Stream m a -> Fold m c x -> Parser b m x
854+ [A] zip :: Monad m => Stream m a -> Fold m (a, b) x -> Parser b m x
855+ [A] wordWithQuotes :: (Monad m, Eq a) => Bool -> (a -> a -> Maybe a) -> a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
856+ [A] wordProcessQuotes :: (Monad m, Eq a) => a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
857+ [A] wordKeepQuotes :: (Monad m, Eq a) => a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
858+ [A] wordFramedBy :: Monad m => (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
859+ [A] wordBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
860+ [A] toFold :: Monad m => Parser a m b -> Fold m a b
861+ [A] takeWhileP :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
862+ [A] takeWhile1 :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
863+ [A] takeWhile :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
864+ [A] takeStartBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
865+ [A] takeStartBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
866+ [A] takeP :: Monad m => Int -> Parser a m b -> Parser a m b
867+ [A] takeGE :: Monad m => Int -> Fold m a b -> Parser a m b
868+ [A] takeFramedBy_ :: Monad m => (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
869+ [A] takeFramedByGeneric :: Monad m => Maybe (a -> Bool) -> Maybe (a -> Bool) -> Maybe (a -> Bool) -> Fold m a b -> Parser a m b
870+ [A] takeFramedByEsc_ :: Monad m => (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
871+ [A] takeEndBy_ :: (a -> Bool) -> Parser a m b -> Parser a m b
872+ [A] takeEndByEsc :: Monad m => (a -> Bool) -> (a -> Bool) -> Parser a m b -> Parser a m b
873+ [A] takeEndBy :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
874+ [A] takeEitherSepBy :: (a -> Bool) -> Fold m (Either a b) c -> Parser a m c
875+ [A] takeEQ :: Monad m => Int -> Fold m a b -> Parser a m b
876+ [A] takeBetween :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
877+ [A] subsequenceBy :: (a -> a -> Bool) -> Stream m a -> Parser a m ()
878+ [A] streamEqBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Parser a m ()
879+ [A] split_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
880+ [A] splitWith :: Monad m => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
881+ [A] splitSome :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
882+ [A] splitManyPost :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
883+ [A] splitMany :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
884+ [A] spanByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
885+ [A] spanBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
886+ [A] span :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
887+ [A] some :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
888+ [A] sequence :: Monad m => Stream m (Parser a m b) -> Fold m b c -> Parser a m c
889+ [A] sepByAll :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
890+ [A] sepBy1 :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
891+ [A] sepBy :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
892+ [A] satisfy :: Monad m => (a -> Bool) -> Parser a m a
893+ [A] sampleFromthen :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
894+ [A] roundRobin :: t (Parser a m b) -> Fold m b c -> Parser a m c
895+ [A] rmapM :: Monad m => (b -> m c) -> Parser a m b -> Parser a m c
896+ [A] retryMaxTotal :: Int -> Parser a m b -> Fold m b c -> Parser a m c
897+ [A] retryMaxSuccessive :: Int -> Parser a m b -> Fold m b c -> Parser a m c
898+ [A] retry :: Parser a m b -> Parser a m b
899+ [A] postscan :: Fold m a b -> Parser b m c -> Parser a m c
900+ [A] peek :: Monad m => Parser a m a
901+ [A] oneOf :: (Monad m, Eq a, Foldable f) => f a -> Parser a m a
902+ [A] oneNotEq :: (Monad m, Eq a) => a -> Parser a m a
903+ [A] oneEq :: (Monad m, Eq a) => a -> Parser a m a
904+ [A] one :: Monad m => Parser a m a
905+ [A] noneOf :: (Monad m, Eq a, Foldable f) => f a -> Parser a m a
906+ [A] noErrorUnsafeSplit_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
907+ [A] noErrorUnsafeSplitWith :: Monad m => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
908+ [A] noErrorUnsafeConcatMap :: Monad m => (b -> Parser a m c) -> Parser a m b -> Parser a m c
909+ [A] maybe :: Monad m => (a -> Maybe b) -> Parser a m b
910+ [A] manyTillP :: Parser a m b -> Parser a m x -> Parser b m c -> Parser a m c
911+ [A] manyTill :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
912+ [A] manyThen :: Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
913+ [A] manyP :: Parser a m b -> Parser b m c -> Parser a m c
914+ [A] many :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
915+ [A] makeIndexFilter :: (Fold m (s, a) b -> Parser a m b) -> (((s, a) -> Bool) -> Fold m (s, a) b -> Fold m (s, a) b) -> ((s, a) -> Bool) -> Fold m a b -> Parser a m b
916+ [A] lookAhead :: Monad m => Parser a m b -> Parser a m b
917+ [A] lmapM :: Monad m => (a -> m b) -> Parser b m r -> Parser a m r
918+ [A] lmap :: (a -> b) -> Parser b m r -> Parser a m r
919+ [A] listEqBy :: Monad m => (a -> a -> Bool) -> [a] -> Parser a m [a]
920+ [A] listEq :: (Monad m, Eq a) => [a] -> Parser a m [a]
921+ [A] indexed :: forall m a b. Monad m => Fold m (Int, a) b -> Parser a m b
922+ [A] groupByRollingEither :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
923+ [A] groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
924+ [A] groupBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
925+ [A] fromPure :: Monad m => b -> Parser a m b
926+ [A] fromFoldMaybe :: Monad m => String -> Fold m a (Maybe b) -> Parser a m b
927+ [A] fromFold :: Monad m => Fold m a b -> Parser a m b
928+ [A] fromEffect :: Monad m => m b -> Parser a m b
929+ [A] filter :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
930+ [A] extractStep :: Monad m => (s -> m (Step s1 b)) -> Step s b -> m (Step s1 b)
931+ [A] eof :: Monad m => Parser a m ()
932+ [A] either :: Monad m => (a -> Either String b) -> Parser a m b
933+ [A] dropWhile :: Monad m => (a -> Bool) -> Parser a m ()
934+ [A] dieM :: Monad m => m String -> Parser a m b
935+ [A] die :: Monad m => String -> Parser a m b
936+ [A] deintercalateAll :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
937+ [A] deintercalate1 :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
938+ [A] deintercalate :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
939+ [A] countBetween :: Int -> Int -> Parser a m b -> Fold m b c -> Parser a m c
940+ [A] count :: Int -> Parser a m b -> Fold m b c -> Parser a m c
941+ [A] concatMap :: Monad m => (b -> Parser a m c) -> Parser a m b -> Parser a m c
942+ [A] blockWithQuotes :: (Monad m, Eq a) => (a -> Bool) -> (a -> Bool) -> a -> a -> Fold m a b -> Parser a m b
943+ [A] bimapOverrideCount :: Int -> (s -> s1) -> (b -> b1) -> Step s b -> Step s1 b1
944+ [A] alt :: Monad m => Parser x m a -> Parser x m a -> Parser x m a
945+[A] Streamly.Internal.Data.MutByteArray
946+ [A] class Unbox a
947+ [A] class SizeOfRep (f :: Type -> Type)
948+ [A] class Serialize a
949+ [A] TypeOfType
950+ [A] UnitType :: Name -> TypeOfType
951+ [A] TheType :: SimpleDataCon -> TypeOfType
952+ [A] MultiType :: [SimpleDataCon] -> TypeOfType
953+ [A] SimpleDataCon
954+ [A] SimpleDataCon :: Name -> [Field] -> SimpleDataCon
955+ [A] SerializeConfig
956+ [A] [cfgRecordSyntaxWithHeader] :: SerializeConfig -> Bool
957+ [A] [cfgInlineSize] :: SerializeConfig -> Maybe Inline
958+ [A] [cfgInlineSerialize] :: SerializeConfig -> Maybe Inline
959+ [A] [cfgInlineDeserialize] :: SerializeConfig -> Maybe Inline
960+ [A] [cfgConstructorTagAsString] :: SerializeConfig -> Bool
961+ [A] SerializeConfig :: Maybe Inline -> Maybe Inline -> Maybe Inline -> Bool -> Bool -> SerializeConfig
962+ [A] PinnedState
963+ [A] Unpinned :: PinnedState
964+ [A] Pinned :: PinnedState
965+ [A] MutByteArray
966+ [A] MutByteArray :: MutableByteArray# RealWorld -> MutByteArray
967+ [A] DataType
968+ [A] [dtTvs] :: DataType -> [Name]
969+ [A] [dtName] :: DataType -> Name
970+ [A] [dtCxt] :: DataType -> Cxt
971+ [A] [dtCons] :: DataType -> [DataCon]
972+ [A] DataType :: Name -> [Name] -> Cxt -> [DataCon] -> DataType
973+ [A] DataCon
974+ [A] [dcTvs] :: DataCon -> [Name]
975+ [A] [dcName] :: DataCon -> Name
976+ [A] [dcFields] :: DataCon -> [(Maybe Name, Type)]
977+ [A] [dcCxt] :: DataCon -> Cxt
978+ [A] DataCon :: Name -> [Name] -> Cxt -> [(Maybe Name, Type)] -> DataCon
979+ [A] BoundedPtr
980+ [A] BoundedPtr :: MutByteArray -> Int -> Int -> BoundedPtr
981+ [A] Streamly.Internal.Data.Serialize.Type.Serialize
982+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize Streamly.Internal.Data.MutByteArray.LiftedInteger
983+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize GHC.Num.Integer.Integer
984+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize a => Streamly.Internal.Data.Serialize.Type.Serialize (GHC.Maybe.Maybe a)
985+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize (Data.Proxy.Proxy a)
986+ [A] instance (Streamly.Internal.Data.Serialize.Type.Serialize a, Streamly.Internal.Data.Serialize.Type.Serialize b) => Streamly.Internal.Data.Serialize.Type.Serialize (Data.Either.Either a b)
987+ [A] Peeker
988+ [A] Peeker :: Builder BoundedPtr IO a -> Peeker a
989+ [A] type Field = (Maybe Name, Type)
990+ [A] xorCmp :: [Word8] -> Name -> Name -> Q Exp
991+ [A] wListToString :: [Word8] -> String
992+ [A] w8_int :: Word8 -> Int
993+ [A] w32_int :: Word32 -> Int
994+ [A] unpin :: MutByteArray -> IO MutByteArray
995+ [A] typeOfType :: Type -> [DataCon] -> TypeOfType
996+ [A] skipByte :: Peeker ()
997+ [A] sizeOfRep :: SizeOfRep f => f x -> Int
998+ [A] sizeOfMutableByteArray :: MutByteArray -> IO Int
999+ [A] sizeOf :: (Unbox a, SizeOfRep (Rep a)) => Proxy a -> Int
1000+ [A] simplifyDataCon :: DataCon -> SimpleDataCon
1001+ [A] serializeW8List :: Name -> Name -> [Word8] -> Q Exp
1002+ [A] serializeConfig :: SerializeConfig
1003+ [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
1004+ [A] runPeeker :: Peeker a -> BoundedPtr -> IO a
1005+ [A] reifyDataType :: Name -> Q DataType
1006+ [A] readUnsafe :: Unbox a => Peeker a
1007+ [A] read :: Unbox a => Peeker a
1008+ [A] putSliceUnsafe :: MonadIO m => MutByteArray -> Int -> MutByteArray -> Int -> Int -> m ()
1009+ [A] pokeRep :: PokeRep f => f a -> BoundedPtr -> IO BoundedPtr
1010+ [A] pokeBoundedPtrUnsafe :: forall a. Unbox a => a -> BoundedPtr -> IO BoundedPtr
1011+ [A] pokeBoundedPtr :: forall a. Unbox a => a -> BoundedPtr -> IO BoundedPtr
1012+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
1013+ [A] pinnedNewAlignedBytes :: Int -> Int -> IO MutByteArray
1014+ [A] pinnedNew :: Int -> IO MutByteArray
1015+ [A] pinnedCloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
1016+ [A] pin :: MutByteArray -> IO MutByteArray
1017+ [A] peekRep :: PeekRep f => Peeker (f x)
1018+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
1019+ [A] openConstructor :: Name -> Int -> Q Pat
1020+ [A] nil :: MutByteArray
1021+ [A] newBytesAs :: PinnedState -> Int -> IO MutByteArray
1022+ [A] new :: Int -> IO MutByteArray
1023+ [A] mkSerializeExprFields :: Name -> [Field] -> Q Exp
1024+ [A] mkRecSizeOfExpr :: SimpleDataCon -> Q Exp
1025+ [A] mkRecSerializeExpr :: Name -> SimpleDataCon -> Q Exp
1026+ [A] mkRecDeserializeExpr :: Name -> Name -> Name -> SimpleDataCon -> Q Exp
1027+ [A] mkFieldName :: Int -> Name
1028+ [A] mkDeserializeKeysDec :: Name -> Name -> SimpleDataCon -> Q [Dec]
1029+ [A] mkDeserializeExprOne :: Name -> SimpleDataCon -> Q Exp
1030+ [A] matchConstructor :: Name -> Int -> Q Exp -> Q Match
1031+ [A] makeN :: Int -> Name
1032+ [A] makeI :: Int -> Name
1033+ [A] makeA :: Int -> Name
1034+ [A] litProxy :: Unbox a => Proxy a -> Q Exp
1035+ [A] litIntegral :: Integral a => a -> Q Exp
1036+ [A] isUnitType :: [DataCon] -> Bool
1037+ [A] isRecordSyntax :: SimpleDataCon -> Bool
1038+ [A] isPinned :: MutByteArray -> Bool
1039+ [A] int_w8 :: Int -> Word8
1040+ [A] int_w32 :: Int -> Word32
1041+ [A] inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
1042+ [A] inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
1043+ [A] inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
1044+ [A] getMutableByteArray# :: MutByteArray -> MutableByteArray# RealWorld
1045+ [A] genericSizeOf :: forall a. SizeOfRep (Rep a) => Proxy a -> Int
1046+ [A] genericPokeByteIndex :: (Generic a, PokeRep (Rep a)) => MutByteArray -> Int -> a -> IO ()
1047+ [A] genericPeekByteIndex :: (Generic a, PeekRep (Rep a)) => MutByteArray -> Int -> IO a
1048+ [A] errorUnsupported :: String -> a
1049+ [A] errorUnimplemented :: a
1050+ [A] encodeRecordFields :: Bool -> SerializeConfig -> SerializeConfig
1051+ [A] encodeConstrNames :: Bool -> SerializeConfig -> SerializeConfig
1052+ [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
1053+ [A] deriveUnbox :: Q [Dec] -> Q [Dec]
1054+ [A] deriveSerializeWith :: (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
1055+ [A] deriveSerialize :: Q [Dec] -> Q [Dec]
1056+ [A] conUpdateFuncDec :: Name -> [Field] -> Q [Dec]
1057+ [A] cloneSliceUnsafeAs :: MonadIO m => PinnedState -> Int -> Int -> MutByteArray -> m MutByteArray
1058+ [A] cloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
1059+ [A] c2w :: Char -> Word8
1060+ [A] asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
1061+ [A] addSizeTo :: Serialize a => Int -> a -> Int
1062+ [A] _x :: Name
1063+ [A] _val :: Name
1064+ [A] _tag :: Name
1065+ [A] _initialOffset :: Name
1066+ [A] _endOffset :: Name
1067+ [A] _arr :: Name
1068+ [A] _acc :: Name
1069+[A] Streamly.Internal.Data.MutArray.Stream
1070+ [A] SpliceState
1071+ [A] SpliceYielding :: arr -> SpliceState s arr -> SpliceState s arr
1072+ [A] SpliceInitial :: s -> SpliceState s arr
1073+ [A] SpliceFinish :: SpliceState s arr
1074+ [A] SpliceBuffering :: s -> arr -> SpliceState s arr
1075+ [A] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
1076+ [A] splitOn :: (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> Stream m (MutArray a)
1077+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1078+ [A] packArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1079+ [A] lpackArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
1080+ [A] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
1081+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1082+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1083+ [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
1084+ [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1085+ [A] compactEQ :: Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1086+ [A] compact :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1087+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1088+[A] Streamly.Internal.Data.MutArray.Generic
1089+ [A] MutArray
1090+ [A] [arrTrueLen] :: MutArray a -> {-# UNPACK #-} !Int
1091+ [A] [arrStart] :: MutArray a -> {-# UNPACK #-} !Int
1092+ [A] [arrLen] :: MutArray a -> {-# UNPACK #-} !Int
1093+ [A] [arrContents#] :: MutArray a -> MutableArray# RealWorld a
1094+ [A] MutArray :: MutableArray# RealWorld a -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> MutArray a
1095+ [A] writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
1096+ [A] writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
1097+ [A] writeN :: MonadIO m => Int -> Fold m a (MutArray a)
1098+ [A] write :: MonadIO m => Fold m a (MutArray a)
1099+ [A] uninit :: MonadIO m => MutArray a -> Int -> m (MutArray a)
1100+ [A] toStreamK :: MonadIO m => MutArray a -> StreamK m a
1101+ [A] toList :: MonadIO m => MutArray a -> m [a]
1102+ [A] strip :: MonadIO m => (a -> Bool) -> MutArray a -> m (MutArray a)
1103+ [A] snocWith :: MonadIO m => (Int -> Int) -> MutArray a -> a -> m (MutArray a)
1104+ [A] snocUnsafe :: MonadIO m => MutArray a -> a -> m (MutArray a)
1105+ [A] snoc :: MonadIO m => MutArray a -> a -> m (MutArray a)
1106+ [A] realloc :: MonadIO m => Int -> MutArray a -> m (MutArray a)
1107+ [A] reader :: MonadIO m => Unfold m (MutArray a) a
1108+ [A] readRev :: MonadIO m => MutArray a -> Stream m a
1109+ [A] read :: MonadIO m => MutArray a -> Stream m a
1110+ [A] putSliceUnsafe :: MonadIO m => MutArray a -> Int -> MutArray a -> Int -> Int -> m ()
1111+ [A] putIndices :: MonadIO m => MutArray a -> Fold m (Int, a) ()
1112+ [A] putIndexUnsafe :: forall m a. MonadIO m => Int -> MutArray a -> a -> m ()
1113+ [A] putIndex :: MonadIO m => Int -> MutArray a -> a -> m ()
1114+ [A] producerWith :: Monad m => (forall b. IO b -> m b) -> Producer m (MutArray a) a
1115+ [A] producer :: MonadIO m => Producer m (MutArray a) a
1116+ [A] nil :: MonadIO m => m (MutArray a)
1117+ [A] new :: MonadIO m => Int -> m (MutArray a)
1118+ [A] modifyIndexUnsafe :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
1119+ [A] modifyIndex :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
1120+ [A] length :: MutArray a -> Int
1121+ [A] getSliceUnsafe :: Int -> Int -> MutArray a -> MutArray a
1122+ [A] getSlice :: Int -> Int -> MutArray a -> MutArray a
1123+ [A] getIndexUnsafeWith :: MonadIO m => MutableArray# RealWorld a -> Int -> m a
1124+ [A] getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
1125+ [A] getIndex :: MonadIO m => Int -> MutArray a -> m (Maybe a)
1126+ [A] fromStreamN :: MonadIO m => Int -> Stream m a -> m (MutArray a)
1127+ [A] fromStream :: MonadIO m => Stream m a -> m (MutArray a)
1128+ [A] fromPureStream :: MonadIO m => Stream Identity a -> m (MutArray a)
1129+ [A] fromListN :: MonadIO m => Int -> [a] -> m (MutArray a)
1130+ [A] fromList :: MonadIO m => [a] -> m (MutArray a)
1131+ [A] eq :: (MonadIO m, Eq a) => MutArray a -> MutArray a -> m Bool
1132+ [A] cmp :: (MonadIO m, Ord a) => MutArray a -> MutArray a -> m Ordering
1133+ [A] clone :: MonadIO m => MutArray a -> m (MutArray a)
1134+ [A] chunksOf :: forall m a. MonadIO m => Int -> Stream m a -> Stream m (MutArray a)
1135+[A] Streamly.Internal.Data.MutArray
1136+ [A] MutByteArray
1137+ [A] MutArray
1138+ [A] [arrStart] :: MutArray a -> {-# UNPACK #-} !Int
1139+ [A] [arrEnd] :: MutArray a -> {-# UNPACK #-} !Int
1140+ [A] [arrContents] :: MutArray a -> {-# UNPACK #-} !MutByteArray
1141+ [A] [arrBound] :: MutArray a -> {-# UNPACK #-} !Int
1142+ [A] MutArray :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> MutArray a
1143+ [A] IORef
1144+ [A] ArrayUnsafe
1145+ [A] ArrayUnsafe :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> ArrayUnsafe a
1146+ [A] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1147+ [A] writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1148+ [A] writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1149+ [A] writeNWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1150+ [A] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1151+ [A] writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1152+ [A] writeIORef :: Unbox a => IORef a -> a -> IO ()
1153+ [A] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
1154+ [A] writeAppendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
1155+ [A] writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
1156+ [A] writeAppendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
1157+ [A] writeAppend :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
1158+ [A] write :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1159+ [A] unsafeSwapIndices :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> MutArray a -> m ()
1160+ [A] unpin :: MutArray a -> IO (MutArray a)
1161+ [A] toStreamKWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> StreamK m a
1162+ [A] toStreamKRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> StreamK m a
1163+ [A] toStreamKRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
1164+ [A] toStreamK :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
1165+ [A] toStreamDWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
1166+ [A] toStreamDRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
1167+ [A] toList :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m [a]
1168+ [A] swapIndices :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> MutArray a -> m ()
1169+ [A] strip :: forall a m. (Unbox a, MonadIO m) => (a -> Bool) -> MutArray a -> m (MutArray a)
1170+ [A] splitOn :: (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> Stream m (MutArray a)
1171+ [A] splitAt :: forall a. Unbox a => Int -> MutArray a -> (MutArray a, MutArray a)
1172+ [A] spliceWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int -> Int) -> MutArray a -> MutArray a -> m (MutArray a)
1173+ [A] spliceUnsafe :: MonadIO m => MutArray a -> MutArray a -> m (MutArray a)
1174+ [A] spliceExp :: (MonadIO m, Unbox a) => MutArray a -> MutArray a -> m (MutArray a)
1175+ [A] spliceCopy :: forall m a. MonadIO m => MutArray a -> MutArray a -> m (MutArray a)
1176+ [A] splice :: (MonadIO m, Unbox a) => MutArray a -> MutArray a -> m (MutArray a)
1177+ [A] snocWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> MutArray a -> a -> m (MutArray a)
1178+ [A] snocUnsafe :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1179+ [A] snocMay :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (Maybe (MutArray a))
1180+ [A] snocLinear :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1181+ [A] snoc :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1182+ [A] shuffleBy :: (a -> a -> m Bool) -> MutArray a -> MutArray a -> m ()
1183+ [A] roundUpToPower2 :: Int -> Int
1184+ [A] rightSize :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m (MutArray a)
1185+ [A] reverse :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m ()
1186+ [A] resizeExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1187+ [A] resize :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1188+ [A] realloc :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1189+ [A] readerRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Unfold m (MutArray a) a
1190+ [A] readerRev :: forall m a. (MonadIO m, Unbox a) => Unfold m (MutArray a) a
1191+ [A] reader :: forall m a. (MonadIO m, Unbox a) => Unfold m (MutArray a) a
1192+ [A] readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
1193+ [A] readIORef :: Unbox a => IORef a -> IO a
1194+ [A] read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
1195+ [A] putIndices :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Fold m (Int, a) ()
1196+ [A] putIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
1197+ [A] putIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
1198+ [A] producerWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Producer m (MutArray a) a
1199+ [A] producer :: forall m a. (MonadIO m, Unbox a) => Producer m (MutArray a) a
1200+ [A] pollIntIORef :: (MonadIO m, Unbox a) => IORef a -> Stream m a
1201+ [A] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1202+ [A] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (MutArray a)
1203+ [A] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1204+ [A] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1205+ [A] pinnedNewBytes :: MonadIO m => Int -> m (MutArray a)
1206+ [A] pinnedNewAligned :: (MonadIO m, Unbox a) => Int -> Int -> m (MutArray a)
1207+ [A] pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
1208+ [A] pinnedFromListN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1209+ [A] pinnedFromList :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1210+ [A] pinnedClone :: MonadIO m => MutArray a -> m (MutArray a)
1211+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1212+ [A] pin :: MutArray a -> IO (MutArray a)
1213+ [A] permute :: MutArray a -> m Bool
1214+ [A] partitionBy :: forall m a. (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> m (MutArray a, MutArray a)
1215+ [A] nil :: MutArray a
1216+ [A] newIORef :: forall a. Unbox a => a -> IO (IORef a)
1217+ [A] newArrayWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int -> m MutByteArray) -> Int -> Int -> m (MutArray a)
1218+ [A] new :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
1219+ [A] modifyIndices :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (Int -> a -> a) -> Fold m Int ()
1220+ [A] modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
1221+ [A] modifyIndex :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
1222+ [A] modifyIORef' :: Unbox a => IORef a -> (a -> a) -> IO ()
1223+ [A] modify :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (a -> a) -> m ()
1224+ [A] mergeBy :: Int -> (MutArray a -> MutArray a -> m ()) -> MutArray a -> m ()
1225+ [A] memcpy :: Ptr Word8 -> Ptr Word8 -> Int -> IO ()
1226+ [A] memcmp :: Ptr Word8 -> Ptr Word8 -> Int -> IO Bool
1227+ [A] length :: forall a. Unbox a => MutArray a -> Int
1228+ [A] isPinned :: MutArray a -> Bool
1229+ [A] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
1230+ [A] getSliceUnsafe :: forall a. Unbox a => Int -> Int -> MutArray a -> MutArray a
1231+ [A] getSlice :: forall a. Unbox a => Int -> Int -> MutArray a -> MutArray a
1232+ [A] getIndicesD :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
1233+ [A] getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1234+ [A] getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
1235+ [A] getIndexRev :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
1236+ [A] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
1237+ [A] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
1238+ [A] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
1239+ [A] fromStreamD :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
1240+ [A] fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
1241+ [A] fromPureStream :: (MonadIO m, Unbox a) => Stream Identity a -> m (MutArray a)
1242+ [A] fromListRevN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1243+ [A] fromListRev :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1244+ [A] fromListN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1245+ [A] fromList :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1246+ [A] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
1247+ [A] foldr :: (MonadIO m, Unbox a) => (a -> b -> b) -> b -> MutArray a -> m b
1248+ [A] foldl' :: (MonadIO m, Unbox a) => (b -> a -> b) -> b -> MutArray a -> m b
1249+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1250+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1251+ [A] divideBy :: Int -> (MutArray a -> m (MutArray a, MutArray a)) -> MutArray a -> m ()
1252+ [A] cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
1253+ [A] clone :: MonadIO m => MutArray a -> m (MutArray a)
1254+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1255+ [A] castUnsafe :: MutArray a -> MutArray b
1256+ [A] cast :: forall a b. Unbox b => MutArray a -> Maybe (MutArray b)
1257+ [A] c_memchr :: Ptr Word8 -> Word8 -> CSize -> IO (Ptr Word8)
1258+ [A] bytesFree :: MutArray a -> Int
1259+ [A] byteLength :: MutArray a -> Int
1260+ [A] byteCapacity :: MutArray a -> Int
1261+ [A] bubble :: (MonadIO m, Unbox a) => (a -> a -> Ordering) -> MutArray a -> m ()
1262+ [A] breakOn :: MonadIO m => Word8 -> MutArray Word8 -> m (MutArray Word8, Maybe (MutArray Word8))
1263+ [A] blockSize :: Int
1264+ [A] asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
1265+ [A] asBytes :: MutArray a -> MutArray Word8
1266+ [A] arrayChunkBytes :: Int
1267+ [A] allocBytesToElemCount :: Unbox a => a -> Int -> Int
1268+[C] Streamly.Internal.Data.IsMap
1269+ [A] mapTraverseWithKey :: (IsMap f, Applicative t) => (Key f -> a -> t b) -> f a -> t (f b)
1270+[R] Streamly.Internal.Data.IORef.Unboxed
1271+[R] Streamly.Internal.Data.Fold.Window
1272+[R] Streamly.Internal.Data.Fold.Type
1273+[R] Streamly.Internal.Data.Fold.Tee
1274+[R] Streamly.Internal.Data.Fold.Step
1275+[R] Streamly.Internal.Data.Fold.Container
1276+[D] Streamly.Internal.Data.Fold.Chunked
1277+[C] Streamly.Internal.Data.Fold
1278+ [A] ManyState
1279+ [C] Fold
1280+ [C] Fold
1281+ [O] Fold :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> Fold m a b
1282+ [N] Fold :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> (s -> m b) -> Fold m a b
1283+ [A] windowSumInt :: forall m a. (Monad m, Integral a) => Fold m (a, Maybe a) a
1284+ [A] windowSum :: forall m a. (Monad m, Num a) => Fold m (a, Maybe a) a
1285+ [A] windowRollingMapM :: Monad m => (Maybe a -> a -> m (Maybe b)) -> Fold m (a, Maybe a) (Maybe b)
1286+ [A] windowRollingMap :: Monad m => (Maybe a -> a -> Maybe b) -> Fold m (a, Maybe a) (Maybe b)
1287+ [A] windowRange :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe (a, a))
1288+ [A] windowPowerSumFrac :: (Monad m, Floating a) => a -> Fold m (a, Maybe a) a
1289+ [A] windowPowerSum :: (Monad m, Num a) => Int -> Fold m (a, Maybe a) a
1290+ [A] windowMinimum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
1291+ [A] windowMean :: forall m a. (Monad m, Fractional a) => Fold m (a, Maybe a) a
1292+ [A] windowMaximum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
1293+ [A] windowLmap :: (c -> a) -> Fold m (a, Maybe a) b -> Fold m (c, Maybe c) b
1294+ [A] windowLength :: (Monad m, Num b) => Fold m (a, Maybe a) b
1295+ [A] toSet :: (Monad m, Ord a) => Fold m a (Set a)
1296+ [A] toMapIO :: (MonadIO m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (Map k b)
1297+ [A] toMap :: (Monad m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (Map k b)
1298+ [A] toIntSet :: Monad m => Fold m Int IntSet
1299+ [A] toContainerIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (f b)
1300+ [A] toContainer :: (Monad m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (f b)
1301+ [A] nubInt :: Monad m => Fold m Int (Maybe Int)
1302+ [A] nub :: (Monad m, Ord a) => Fold m a (Maybe a)
1303+ [A] mapMStep :: Applicative m => (a -> m b) -> Step s a -> m (Step s b)
1304+ [A] kvToMap :: (Monad m, Ord k) => Fold m a b -> Fold m (k, a) (Map k b)
1305+ [A] frequency :: (Monad m, Ord a) => Fold m a (Map a Int)
1306+ [A] demuxToMapIO :: (MonadIO m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (Map k b)
1307+ [A] demuxToMap :: (Monad m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (Map k b)
1308+ [A] demuxToContainerIO :: (MonadIO m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
1309+ [A] demuxToContainer :: (Monad m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
1310+ [A] demuxKvToMap :: (Monad m, Ord k) => (k -> m (Fold m a b)) -> Fold m (k, a) (Map k b)
1311+ [A] demuxKvToContainer :: (Monad m, IsMap f, Traversable f) => (Key f -> m (Fold m a b)) -> Fold m (Key f, a) (f b)
1312+ [A] demuxIO :: (MonadIO m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (m (Map k b), Maybe (k, b))
1313+ [A] demuxGenericIO :: (MonadIO m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (m (f b), Maybe (Key f, b))
1314+ [A] demuxGeneric :: (Monad m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (m (f b), Maybe (Key f, b))
1315+ [A] demux :: (Monad m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (m (Map k b), Maybe (k, b))
1316+ [A] cumulative :: Fold m (a, Maybe a) b -> Fold m a b
1317+ [A] countDistinctInt :: Monad m => Fold m Int Int
1318+ [A] countDistinct :: (Monad m, Ord a) => Fold m a Int
1319+ [A] classifyIO :: (MonadIO m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
1320+ [A] classifyGenericIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
1321+ [A] classifyGeneric :: (Monad m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
1322+ [A] classify :: (Monad m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
1323+ [A] chainStepM :: Applicative m => (s1 -> m s2) -> (a -> m (Step s2 b)) -> Step s1 a -> m (Step s2 b)
1324+[C] Streamly.Internal.Data.Builder
1325+ [C] Builder
1326+ [C] Builder
1327+ [O] Builder :: (s -> m (s, a)) -> Builder s m a
1328+ [N] Builder :: (s -> m (a, s)) -> Builder s m a
1329+[A] Streamly.Internal.Data.Binary.Stream
1330+ [A] class ToBytes a
1331+ [A] word8 :: Applicative m => Word8 -> Stream m Word8
1332+ [A] word64le :: Monad m => Word64 -> Stream m Word8
1333+ [A] word64host :: Monad m => Word64 -> Stream m Word8
1334+ [A] word64be :: Monad m => Word64 -> Stream m Word8
1335+ [A] word32le :: Monad m => Word32 -> Stream m Word8
1336+ [A] word32be :: Monad m => Word32 -> Stream m Word8
1337+ [A] word16le :: Monad m => Word16 -> Stream m Word8
1338+ [A] word16be :: Monad m => Word16 -> Stream m Word8
1339+ [A] unit :: Applicative m => Stream m Word8
1340+ [A] toBytes :: ToBytes a => a -> Stream m Word8
1341+ [A] ordering :: Applicative m => Ordering -> Stream m Word8
1342+ [A] int8 :: Applicative m => Int8 -> Stream m Word8
1343+ [A] int64le :: Monad m => Int64 -> Stream m Word8
1344+ [A] int64be :: Monad m => Int64 -> Stream m Word8
1345+ [A] int32le :: Monad m => Int32 -> Stream m Word8
1346+ [A] int32be :: Monad m => Int32 -> Stream m Word8
1347+ [A] int16le :: Monad m => Int16 -> Stream m Word8
1348+ [A] int16be :: Monad m => Int16 -> Stream m Word8
1349+ [A] float32le :: Monad m => Float -> Stream m Word8
1350+ [A] float32be :: Monad m => Float -> Stream m Word8
1351+ [A] double64le :: Monad m => Double -> Stream m Word8
1352+ [A] double64be :: Monad m => Double -> Stream m Word8
1353+ [A] charUtf8 :: Monad m => Char -> Stream m Word8
1354+ [A] charLatin1 :: Applicative m => Char -> Stream m Word8
1355+ [A] bool :: Applicative m => Bool -> Stream m Word8
1356+[A] Streamly.Internal.Data.Binary.Parser
1357+ [A] class FromBytes a
1358+ [A] word8 :: Monad m => Parser Word8 m Word8
1359+ [A] word64le :: Monad m => Parser Word8 m Word64
1360+ [A] word64host :: MonadIO m => Parser Word8 m Word64
1361+ [A] word64be :: Monad m => Parser Word8 m Word64
1362+ [A] word32le :: Monad m => Parser Word8 m Word32
1363+ [A] word32be :: Monad m => Parser Word8 m Word32
1364+ [A] word16le :: Monad m => Parser Word8 m Word16
1365+ [A] word16be :: Monad m => Parser Word8 m Word16
1366+ [A] unit :: Monad m => Parser Word8 m ()
1367+ [A] ordering :: Monad m => Parser Word8 m Ordering
1368+ [A] int8 :: Monad m => Parser Word8 m Int8
1369+ [A] int64le :: Monad m => Parser Word8 m Int64
1370+ [A] int64be :: Monad m => Parser Word8 m Int64
1371+ [A] int32le :: Monad m => Parser Word8 m Int32
1372+ [A] int32be :: Monad m => Parser Word8 m Int32
1373+ [A] int16le :: Monad m => Parser Word8 m Int16
1374+ [A] int16be :: Monad m => Parser Word8 m Int16
1375+ [A] fromBytes :: FromBytes a => Parser Word8 m a
1376+ [A] float32le :: MonadIO m => Parser Word8 m Float
1377+ [A] float32be :: MonadIO m => Parser Word8 m Float
1378+ [A] eqWord8 :: Monad m => Word8 -> Parser Word8 m Word8
1379+ [A] double64le :: MonadIO m => Parser Word8 m Double
1380+ [A] double64be :: MonadIO m => Parser Word8 m Double
1381+ [A] charLatin1 :: Monad m => Parser Word8 m Char
1382+ [A] bool :: Monad m => Parser Word8 m Bool
1383+[R] Streamly.Internal.Data.Array.Type
1384+[A] Streamly.Internal.Data.Array.Stream
1385+ [A] unlines :: forall m a. (MonadIO m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1386+ [A] toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
1387+ [A] splitOnSuffix :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
1388+ [A] splitOn :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
1389+ [A] runArrayParserDBreak :: forall m a b. (MonadIO m, Unbox a) => Parser (Array a) m b -> Stream m (Array a) -> m (Either ParseError b, Stream m (Array a))
1390+ [A] runArrayFoldMany :: (Monad m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> StreamK m (Either ParseError b)
1391+ [A] runArrayFoldBreak :: (MonadIO m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1392+ [A] runArrayFold :: (MonadIO m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> m (Either ParseError b)
1393+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1394+ [A] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
1395+ [A] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1396+ [A] parseBreak :: (MonadIO m, Unbox a) => Parser a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1397+ [A] lpackArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
1398+ [A] interposeSuffix :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1399+ [A] interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1400+ [A] intercalateSuffix :: (Monad m, Unbox a) => Array a -> Stream m (Array a) -> Stream m a
1401+ [A] foldBreakD :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
1402+ [A] foldBreak :: (MonadIO m, Unbox a) => Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
1403+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1404+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1405+ [A] concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
1406+ [A] concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
1407+ [A] compact :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
1408+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1409+ [A] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
1410+[R] Streamly.Internal.Data.Array.Mut.Type
1411+[R] Streamly.Internal.Data.Array.Mut.Stream
1412+[R] Streamly.Internal.Data.Array.Mut
1413+[R] Streamly.Internal.Data.Array.Generic.Mut.Type
1414+[C] Streamly.Internal.Data.Array.Generic
1415+ [A] getIndex :: Int -> Array a -> Maybe a
1416+ [A] fromPureStream :: Stream Identity a -> Array a
1417+ [A] fromByteStr# :: Addr# -> Array Word8
1418+ [A] chunksOf :: forall m a. MonadIO m => Int -> Stream m a -> Stream m (Array a)
1419+[C] Streamly.Internal.Data.Array
1420+ [A] ArrayUnsafe
1421+ [A] ArrayUnsafe :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> ArrayUnsafe a
1422+ [C] Array
1423+ [A] [arrStart] :: Array a -> {-# UNPACK #-} !Int
1424+ [A] [arrEnd] :: Array a -> {-# UNPACK #-} !Int
1425+ [A] [arrContents] :: Array a -> {-# UNPACK #-} !MutByteArray
1426+ [A] Array :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> Array a
1427+ [A] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1428+ [A] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1429+ [R] writeNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
1430+ [A] unsafeMakePure :: Monad m => Fold IO a b -> Fold m a b
1431+ [A] unsafeIndexIO :: forall a. Unbox a => Int -> Array a -> IO a
1432+ [D] unsafeIndex :: forall a. Unbox a => Int -> Array a -> a
1433+ [A] unsafeFreezeWithShrink :: Unbox a => MutArray a -> Array a
1434+ [A] unpin :: Array a -> IO (Array a)
1435+ [A] toStreamKRev :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a
1436+ [A] toStreamK :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a
1437+ [A] toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
1438+ [A] toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
1439+ [A] splitAt :: Unbox a => Int -> Array a -> (Array a, Array a)
1440+ [A] splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
1441+ [A] serialize :: Serialize a => a -> Array Word8
1442+ [A] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1443+ [A] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
1444+ [A] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1445+ [A] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
1446+ [A] pinnedSerialize :: Serialize a => a -> Array Word8
1447+ [A] pinnedFromListN :: Unbox a => Int -> [a] -> Array a
1448+ [A] pinnedFromList :: Unbox a => [a] -> Array a
1449+ [A] pinnedClone :: MonadIO m => Array a -> m (Array a)
1450+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1451+ [A] pin :: Array a -> IO (Array a)
1452+ [A] nil :: Array a
1453+ [A] isPinned :: Array a -> Bool
1454+ [A] getIndexUnsafe :: forall a. Unbox a => Int -> Array a -> a
1455+ [A] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
1456+ [A] fromStreamD :: forall m a. (MonadIO m, Unbox a) => Stream m a -> m (Array a)
1457+ [A] fromPureStream :: Unbox a => Stream Identity a -> Array a
1458+ [A] fromListRevN :: Unbox a => Int -> [a] -> Array a
1459+ [A] fromListRev :: Unbox a => [a] -> Array a
1460+ [A] fromByteStr# :: Addr# -> Array Word8
1461+ [A] foldr :: Unbox a => (a -> b -> b) -> b -> Array a -> b
1462+ [A] foldl' :: forall a b. Unbox a => (b -> a -> b) -> b -> Array a -> b
1463+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1464+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1465+ [A] encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
1466+ [A] deserialize :: Serialize a => Array Word8 -> a
1467+ [A] clone :: MonadIO m => Array a -> m (Array a)
1468+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1469+ [A] byteLength :: Array a -> Int
1470+ [A] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
1471+ [A] breakOn :: MonadIO m => Word8 -> Array Word8 -> m (Array Word8, Maybe (Array Word8))
core/docs/ApiChangelogs/0.2.0-0.2.2.txtnew file+306
@@ -0,0 +1,306 @@
1+---------------------------------
2+API Annotations
3+---------------------------------
4+
5+[A] : Added
6+[R] : Removed
7+[C] : Changed
8+[O] : Old definition
9+[N] : New definition
10+[D] : Deprecated
11+
12+---------------------------------
13+API diff
14+---------------------------------
15+
16+[C] Streamly.Data.Stream
17+ [A] (FixityR,5)
18+ [A] (FixityR,5)
19+[C] Streamly.Data.MutArray.Generic
20+ [A] emptyOf :: MonadIO m => Int -> m (MutArray a)
21+ [A] createOf :: MonadIO m => Int -> Fold m a (MutArray a)
22+ [A] create :: MonadIO m => Fold m a (MutArray a)
23+[C] Streamly.Data.MutArray
24+ [A] pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
25+ [A] emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
26+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
27+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
28+ [A] appendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
29+ [A] append :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
30+[C] Streamly.Data.Array.Generic
31+ [A] createOf :: MonadIO m => Int -> Fold m a (Array a)
32+ [A] create :: MonadIO m => Fold m a (Array a)
33+[C] Streamly.Data.Array
34+ [A] class Serialize a
35+ [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
36+ [A] pinnedSerialize :: Serialize a => a -> Array Word8
37+ [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
38+ [A] deserialize :: Serialize a => Array Word8 -> a
39+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
40+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
41+ [A] addSizeTo :: Serialize a => Int -> a -> Int
42+
43+---------------------------------
44+Internal API diff
45+---------------------------------
46+
47+[C] Streamly.Internal.Unicode.Stream
48+ [A] encodeUtf16le' :: Stream m Char -> Stream m Word16
49+ [A] decodeUtf16le' :: Stream m Word16 -> Stream m Char
50+[A] Streamly.Internal.FileSystem.Path
51+ [A] class IsPath a
52+ [A] Rel
53+ [A] File
54+ [A] Dir
55+ [A] Abs
56+ [A] Streamly.Internal.FileSystem.Path.IsPath
57+ [A] instance Streamly.Internal.FileSystem.Path.IsPath Streamly.Internal.FileSystem.Path.Path
58+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel Streamly.Internal.FileSystem.Path.Path)
59+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path))
60+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path))
61+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path)
62+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path)
63+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs Streamly.Internal.FileSystem.Path.Path)
64+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path))
65+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path))
66+ [A] GHC.Show.Show
67+ [A] instance GHC.Show.Show Streamly.Internal.FileSystem.Path.PathException
68+ [A] GHC.Exception.Type.Exception
69+ [A] instance GHC.Exception.Type.Exception Streamly.Internal.FileSystem.Path.PathException
70+ [A] GHC.Classes.Eq
71+ [A] instance GHC.Classes.Eq Streamly.Internal.FileSystem.Path.PathException
72+ [A] Path
73+ [A] Path :: Array Word8 -> Path
74+ [A] toString :: Path -> [Char]
75+ [A] toPath :: IsPath a => a -> Path
76+ [A] toChunk :: Path -> Array Word8
77+ [A] toChars :: Monad m => Path -> Stream m Char
78+ [A] relfile :: QuasiQuoter
79+ [A] reldir :: QuasiQuoter
80+ [A] rel :: QuasiQuoter
81+ [A] primarySeparator :: Char
82+ [A] path :: QuasiQuoter
83+ [A] mkRelFile :: String -> Q Exp
84+ [A] mkRelDir :: String -> Q Exp
85+ [A] mkRel :: String -> Q Exp
86+ [A] mkPath :: String -> Q Exp
87+ [A] mkFile :: String -> Q Exp
88+ [A] mkDir :: String -> Q Exp
89+ [A] mkAbsFile :: String -> Q Exp
90+ [A] mkAbsDir :: String -> Q Exp
91+ [A] mkAbs :: String -> Q Exp
92+ [A] isSeparator :: Char -> Bool
93+ [A] fromString :: MonadThrow m => [Char] -> m Path
94+ [A] fromPathUnsafe :: IsPath a => Path -> a
95+ [A] fromPath :: (IsPath a, MonadThrow m) => Path -> m a
96+ [A] fromChunkUnsafe :: Array Word8 -> Path
97+ [A] fromChunk :: MonadThrow m => Array Word8 -> m Path
98+ [A] fromChars :: MonadThrow m => Stream Identity Char -> m Path
99+ [A] file :: QuasiQuoter
100+ [A] extendPath :: Path -> Path -> Path
101+ [A] extendDir :: (IsPath (a (Dir Path)), IsPath b, IsPath (a b)) => a (Dir Path) -> Rel b -> a b
102+ [A] dir :: QuasiQuoter
103+ [A] adaptPath :: (MonadThrow m, IsPath a, IsPath b) => a -> m b
104+ [A] absfile :: QuasiQuoter
105+ [A] absdir :: QuasiQuoter
106+ [A] abs :: QuasiQuoter
107+[C] Streamly.Internal.Data.Stream
108+ [A] (FixityR,5)
109+ [A] (FixityR,5)
110+ [C] splitInnerBySuffix
111+ [O] splitInnerBySuffix :: (Monad m, Eq (f a), Monoid (f a)) => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
112+ [N] splitInnerBySuffix :: Monad m => (f a -> Bool) -> (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
113+ [D] sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
114+ [A] indexOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
115+[C] Streamly.Internal.Data.MutByteArray
116+ [A] unsafePinnedAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
117+ [A] unsafeAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
118+ [D] nil :: MutByteArray
119+ [A] empty :: MutByteArray
120+ [D] asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
121+[D] Streamly.Internal.Data.MutArray.Stream
122+ [D] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
123+ [D] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
124+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
125+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
126+[C] Streamly.Internal.Data.MutArray.Generic
127+ [D] writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
128+ [D] writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
129+ [A] unsafeCreateOf :: MonadIO m => Int -> Fold m a (MutArray a)
130+ [A] emptyOf :: MonadIO m => Int -> m (MutArray a)
131+ [A] createWith :: MonadIO m => Int -> Fold m a (MutArray a)
132+ [A] createOf :: MonadIO m => Int -> Fold m a (MutArray a)
133+ [A] create :: MonadIO m => Fold m a (MutArray a)
134+[C] Streamly.Internal.Data.MutArray
135+ [A] SpliceState
136+ [A] SpliceYielding :: arr -> SpliceState s arr -> SpliceState s arr
137+ [A] SpliceInitial :: s -> SpliceState s arr
138+ [A] SpliceFinish :: SpliceState s arr
139+ [A] SpliceBuffering :: s -> arr -> SpliceState s arr
140+ [R] MutByteArray
141+ [D] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
142+ [D] writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
143+ [D] writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
144+ [D] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
145+ [D] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
146+ [D] writeAppendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
147+ [D] writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
148+ [A] unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
149+ [A] unsafePinnedAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
150+ [A] unsafeCreateOfWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
151+ [A] unsafeCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
152+ [A] unsafeAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
153+ [A] unsafeAppendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
154+ [A] toStreamWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
155+ [A] toStreamRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
156+ [R] toStreamDWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
157+ [R] toStreamDRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
158+ [A] slicerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
159+ [A] sliceIndexerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
160+ [A] revCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
161+ [D] resizeExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
162+ [D] resize :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
163+ [A] pokeSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
164+ [A] pokeAppendMay :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> a -> m (Maybe (MutArray Word8))
165+ [A] pokeAppend :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> a -> m (MutArray Word8)
166+ [D] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
167+ [D] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
168+ [D] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
169+ [D] pinnedNewBytes :: MonadIO m => Int -> m (MutArray a)
170+ [A] pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
171+ [A] pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
172+ [A] pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
173+ [A] pinnedCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
174+ [A] peekUnconsUnsafe :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> m (a, MutArray Word8)
175+ [A] peekUncons :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> m (Maybe a, MutArray Word8)
176+ [A] peekSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
177+ [A] pPinnedCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Parser (MutArray a) m (MutArray a)
178+ [A] pCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Parser (MutArray a) m (MutArray a)
179+ [D] nil :: MutArray a
180+ [A] lPinnedCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
181+ [A] lCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
182+ [A] indexReaderWith :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
183+ [A] indexReader :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
184+ [A] growExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
185+ [A] grow :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
186+ [D] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
187+ [R] getIndicesD :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
188+ [D] getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
189+ [D] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
190+ [A] fromStreamN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
191+ [D] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
192+ [D] fromStreamD :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
193+ [A] fromPureStreamN :: (MonadIO m, Unbox a) => Int -> Stream Identity a -> m (MutArray a)
194+ [A] fromPtrN :: MonadIO m => Int -> Ptr Word8 -> m (MutArray Word8)
195+ [A] fromChunksRealloced :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> m (MutArray a)
196+ [A] fromChunksK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
197+ [A] fromByteStr# :: MonadIO m => Addr# -> m (MutArray Word8)
198+ [D] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
199+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
200+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
201+ [A] fPinnedCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) (MutArray a)
202+ [A] fCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) (MutArray a)
203+ [A] emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
204+ [A] empty :: MutArray a
205+ [A] createWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
206+ [A] createOfWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
207+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
208+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
209+ [A] concatWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m (MutArray a) -> Stream m a
210+ [A] concatRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m (MutArray a) -> Stream m a
211+ [A] concatRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
212+ [A] concat :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
213+ [A] compactOnByteSuffix :: MonadIO m => Word8 -> Stream m (MutArray Word8) -> Stream m (MutArray Word8)
214+ [A] compactOnByte :: MonadIO m => Word8 -> Stream m (MutArray Word8) -> Stream m (MutArray Word8)
215+ [A] compactLeAs :: forall m a. (MonadIO m, Unbox a) => PinnedState -> Int -> Stream m (MutArray a) -> Stream m (MutArray a)
216+ [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
217+ [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
218+ [A] compactEQ :: Int -> Stream m (MutArray a) -> Stream m (MutArray a)
219+ [D] cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
220+ [A] byteEq :: MonadIO m => MutArray a -> MutArray a -> m Bool
221+ [A] byteCmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
222+ [A] buildChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
223+ [D] asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
224+ [A] appendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
225+ [A] appendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
226+ [A] append :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
227+[D] Streamly.Internal.Data.Array.Stream
228+ [C] interposeSuffix
229+ [O] interposeSuffix :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
230+ [N] interposeSuffix :: forall m a. (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
231+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
232+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
233+ [C] concatRev
234+ [O] concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
235+ [N] concatRev :: forall m a. (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
236+ [D] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
237+[C] Streamly.Internal.Data.Array.Generic
238+ [A] createOf :: MonadIO m => Int -> Fold m a (Array a)
239+ [A] create :: MonadIO m => Fold m a (Array a)
240+[C] Streamly.Internal.Data.Array
241+ [R] ArrayUnsafe
242+ [D] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
243+ [D] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
244+ [A] unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
245+ [A] unsafePinnedAsPtr :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
246+ [A] unsafeCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
247+ [D] toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
248+ [D] toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
249+ [C] splice
250+ [O] splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
251+ [N] splice :: MonadIO m => Array a -> Array a -> m (Array a)
252+ [A] slicerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Array a)
253+ [A] sliceIndexerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Int, Int)
254+ [D] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
255+ [D] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
256+ [D] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
257+ [D] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
258+ [A] pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
259+ [A] pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
260+ [A] pinnedCompactLE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
261+ [A] parseBreakChunksK :: forall m a b. (MonadIO m, Unbox a) => Parser a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
262+ [D] nil :: Array a
263+ [A] lPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
264+ [A] lCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
265+ [A] interposeSuffix :: forall m a. (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
266+ [A] interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
267+ [A] intercalateSuffix :: (Monad m, Unbox a) => Array a -> Stream m (Array a) -> Stream m a
268+ [A] indexReaderFromThenTo :: Unfold m (Int, Int, Int, Array a) a
269+ [A] indexReader :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
270+ [A] indexFinder :: (a -> Bool) -> Unfold Identity (Array a) Int
271+ [D] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Array a)
272+ [R] getIndicesFromThenTo :: Unfold m (Int, Int, Int, Array a) a
273+ [D] getIndices :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
274+ [D] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Int, Int)
275+ [D] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
276+ [D] fromStreamD :: forall m a. (MonadIO m, Unbox a) => Stream m a -> m (Array a)
277+ [A] fromPureStreamN :: Unbox a => Int -> Stream Identity a -> Array a
278+ [A] fromPtrN :: Int -> Ptr Word8 -> Array Word8
279+ [A] fromChunksK :: (MonadIO m, Unbox a) => StreamK m (Array a) -> m (Array a)
280+ [A] fromChunks :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
281+ [A] fromByteStr :: Ptr Word8 -> Array Word8
282+ [A] foldChunks :: (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m b
283+ [A] foldBreakChunksK :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
284+ [A] foldBreakChunks :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
285+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
286+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
287+ [C] findIndicesOf
288+ [O] findIndicesOf :: (a -> Bool) -> Unfold Identity (Array a) Int
289+ [N] findIndicesOf :: (a -> Bool) -> Array a -> Stream Identity Int
290+ [A] fPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
291+ [A] fCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
292+ [A] empty :: Array a
293+ [A] createWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
294+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
295+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
296+ [A] concatRev :: forall m a. (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
297+ [A] concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
298+ [A] compactOnByteSuffix :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
299+ [A] compactOnByte :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
300+ [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
301+ [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
302+ [A] byteEq :: Array a -> Array a -> Bool
303+ [A] byteCmp :: Array a -> Array a -> Ordering
304+ [A] buildChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
305+ [D] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
306+ [D] asPtrUnsafe :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
core/docs/Changelog.md+60−12
@@ -1,25 +1,73 @@
11# Changelog
22
3−## Unreleased
3+## 0.2.2 (Jan 2024)
44
5−* Arrays are now created unpinned by default, they were created pinned
6− earlier. During IO operations unpinned arrays are automatically copied
7− to pinned memory. When arrays are directly passed to IO operations
8− programmers can choose to create them pinned to avoid a copy. To
9− create pinned arrays, use the internal APIs with the `pinned*` prefix.
5+* Add fixities `infixr 5` for `cons` and `consM` functions.
6+* Fix a bug in Array `Eq` instance when the type is a sum type with
7+ differently sized constructors.
8+* lpackArraysChunksOf, compact, writeChunksWith, putChunksWith now take the
9+ buffer size in number of array elements instead of bytes.
1010
11−* Changed the signature of 'Streamly.Data.Stream.handle' to make the exception
12− handler monadic.
13−* Rethrow the exception promptly in bracketIO.
11+## 0.2.1 (Dec 2023)
12+
13+* Make the serialization of the unit constructor deterministic.
14+* Expose `pinnedSerialize` & `deserialize` via `Streamly.Data.Array`.
15+
16+## 0.2.0 (Nov 2023)
17+
18+See [0.1.0-0.2.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0-0.2.0.txt)
19+for a full list of API changes in this release. Only a few significant
20+changes are mentioned here.
21+
22+### Breaking Changes
23+
24+* `ParserK` in `Streamly.Data.ParserK` is not implicitly specialized
25+ to arrays anymore. To adapt to the new code, change `ParserK a m
26+ b` to `ParserK (Array a) m b` where the `Array` type comes from
27+ `Streamly.Data.Array`. This change also affected the signatures of
28+ `parseChunks` and `parseBreakChunks`.
29+* Changed the signature of 'Streamly.Data.Stream.handle' to make the
30+ exception handler monadic.
31+* Behavior change: Exceptions are now rethrown promptly in `bracketIO`.
32+
33+### Enhancements
34+
35+* __Serialization__: Added a `Streamly.Data.MutByteArray` module with a
36+ `Serialize` type class for fast binary serialization. The Data.Array
37+ module supplies the `serialize` and `deserialize` operations for arrays.
38+* __Unpinned Arrays__: Unboxed arrays are now created unpinned by default,
39+ they were created pinned earlier. During IO operations, unpinned arrays
40+ are automatically copied to pinned memory. When arrays are directly
41+ passed to IO operations programmers can choose to create them pinned to
42+ avoid a copy. To create pinned arrays, use the internal APIs with the
43+ `pinned*` prefix.
44+* StreamK now supports native exception handling routines (handle, bracketIO).
45+ Earlier we had to convert it to the `Stream` type for exception handling.
1446
1547### Deprecations
1648
17−* `Streamly.Data.MutArray.newUnpinned` is renamed to
18− `Streamly.Data.MutArray.pinnedNew`
49+See [0.1.0-0.2.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0-0.2.0.txt)
50+for a full list of deprecations.
51+
52+### Internal API Changes
53+
54+* Fold constructor has changed, added a `final` field to support
55+ finalization and cleanup of a chain of folds. The `extract` field is
56+ now used only for mapping the fold internal state to fold result for
57+ scanning purposes. If your fold does not require cleanup you can just use
58+ your existing `extract` function as `final` as well to adapt to this change.
59+* Many low level internal modules have been removed, they are entirely
60+ exported from higher level internal modules. If you were importing any
61+ of the missing low level modules then import the higher level modules instead.
62+* Internal module changes:
63+ * Streamly.Internal.Serialize.FromBytes -> Streamly.Internal.Data.Binary.Parser
64+ * Streamly.Internal.Serialize.ToBytes -> Streamly.Internal.Data.Binary.Stream
65+ * Streamly.Internal.Data.Unbox is now exported via Streamly.Internal.Data.Serialize
66+ * Streamly.Internal.Data.IORef.Unboxed is now exported via Streamly.Internal.Data.Serialize
1967
2068## 0.1.0 (March 2023)
2169
22−Also see [streamly-core-0.1.0 API Changelog](/core/docs/ApiChangelogs/0.1.0.txt) or
70+Also see [streamly-core-0.1.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0.txt) or
2371https://hackage.haskell.org/package/streamly-core-0.1.0/docs/docs/ApiChangelogs/0.1.0.txt
2472
2573`streamly` package is split into two packages, (1) `streamly-core` that
core/src/DocTestDataArray.hs+8
@@ -1,6 +1,7 @@
11{- $setup
22>>> :m
33>>> :set -XFlexibleContexts
4+>>> :set -XMagicHash
45>>> import Data.Function ((&))
56>>> import Data.Functor.Identity (Identity(..))
67>>> import System.IO.Unsafe (unsafePerformIO)
@@ -10,5 +11,12 @@
1011
1112>>> import qualified Streamly.Data.Array as Array
1213>>> import qualified Streamly.Data.Fold as Fold
14+>>> import qualified Streamly.Data.ParserK as ParserK
1315>>> import qualified Streamly.Data.Stream as Stream
16+>>> import qualified Streamly.Data.StreamK as StreamK
17+
18+For APIs that have not been released yet.
19+
20+>>> import qualified Streamly.Internal.Data.Array as Array
21+>>> import qualified Streamly.Internal.Data.Stream as Stream
1422-}
core/src/DocTestDataFold.hs−1
@@ -24,5 +24,4 @@
2424For APIs that have not been released yet.
2525
2626>>> import qualified Streamly.Internal.Data.Fold as Fold
27−>>> import qualified Streamly.Internal.Data.Fold.Window as FoldW
2827-}
core/src/DocTestDataParserK.hsnew file+15
@@ -0,0 +1,15 @@
1+{- $setup
2+>>> :m
3+>>> import Control.Applicative ((<|>))
4+>>> import Data.Char (isDigit, isAlpha)
5+
6+>>> import Streamly.Data.Parser (Parser)
7+>>> import Streamly.Data.ParserK (ParserK)
8+
9+>>> import qualified Streamly.Data.Parser as Parser
10+>>> import qualified Streamly.Data.ParserK as ParserK
11+
12+For APIs that have not been released yet.
13+
14+>>> import qualified Streamly.Internal.Data.ParserK as ParserK
15+-}
core/src/DocTestDataStream.hs+2−1
@@ -6,6 +6,7 @@
66>>> import Control.Monad.IO.Class (MonadIO (liftIO))
77>>> import Control.Monad.Trans.Class (lift)
88>>> import Control.Monad.Trans.Identity (runIdentityT)
9+>>> import Data.Char (isSpace)
910>>> import Data.Either (fromLeft, fromRight, isLeft, isRight, either)
1011>>> import Data.Maybe (fromJust, isJust)
1112>>> import Data.Function (fix, (&))
@@ -30,9 +31,9 @@
3031For APIs that have not been released yet.
3132
3233>>> import qualified Streamly.Internal.Data.Fold as Fold
33−>>> import qualified Streamly.Internal.Data.Fold.Window as Window
3434>>> import qualified Streamly.Internal.Data.Parser as Parser
3535>>> import qualified Streamly.Internal.Data.Stream as Stream
36+>>> import qualified Streamly.Internal.Data.StreamK as StreamK
3637>>> import qualified Streamly.Internal.Data.Unfold as Unfold
3738>>> import qualified Streamly.Internal.FileSystem.Dir as Dir
3839-}
core/src/DocTestFileSystemHandle.hs+1−1
@@ -8,7 +8,7 @@
88
99For APIs that have not been released yet.
1010
11−>>> import qualified Streamly.Internal.Data.Array.Type as Array (writeNUnsafe)
11+>>> import qualified Streamly.Internal.Data.Array as Array (unsafeCreateOf)
1212>>> import qualified Streamly.Internal.Data.Unfold as Unfold (first)
1313>>> import qualified Streamly.Internal.FileSystem.Handle as Handle
1414>>> import qualified Streamly.Internal.System.IO as IO (defaultChunkSize)
core/src/Streamly/Console/Stdio.hs+1−27
@@ -24,33 +24,7 @@ module Streamly.Console.Stdio
2424 -- * Write (stderr)
2525 , writeErr
2626 , writeErrChunks
27−
28− -- * Deprecated
29− , read
30− , readChunks
3127 )
3228where
3329
34−import Control.Monad.IO.Class (MonadIO(..))
35−import Data.Word (Word8)
36−import Streamly.Internal.Data.Array.Type (Array)
37−import Streamly.Internal.Data.Unfold (Unfold)
38−
39−import Streamly.Internal.Console.Stdio hiding (read, readChunks)
40−import Prelude hiding (read)
41−
42−-- Same as 'reader'
43−--
44−-- @since 0.8.0
45−{-# DEPRECATED read "Please use 'reader' instead" #-}
46−{-# INLINE read #-}
47−read :: MonadIO m => Unfold m () Word8
48−read = reader
49−
50−-- Same as 'chunkReader'
51−--
52−-- @since 0.8.0
53−{-# DEPRECATED readChunks "Please use 'chunkReader' instead" #-}
54−{-# INLINE readChunks #-}
55−readChunks :: MonadIO m => Unfold m () (Array Word8)
56−readChunks = chunkReader
30+import Streamly.Internal.Console.Stdio
core/src/Streamly/Data/Array.hs+48−48
@@ -29,65 +29,80 @@ module Streamly.Data.Array
2929 -- $overview
3030
3131 -- * The Array Type
32− A.Array
32+ Array
33+
34+ -- * Pinning & Unpinning
35+ -- | Arrays are created unpinned by default unless pinned versions of
36+ -- creation APIs are used. Look for APIs with @pinned@ prefix in
37+ -- "Streamly.Internal.Data.Array" for some unreleased pinned creation APIs.
38+ -- If an array is to be sent to the OS without any further modification
39+ -- then it should be created pinned in the first place instead of pinning
40+ -- it later. Pinning an unpinned array has a copy overhead. OS interfacing
41+ -- APIs create a pinned array directly or convert an unpinned array to
42+ -- pinned array before sending it to the OS.
43+ , pin
44+ , unpin
45+ , isPinned
3346
3447 -- * Construction
3548 -- | When performance matters, the fastest way to generate an array is
36− -- 'writeN'. 'IsList' and 'IsString' instances can be
49+ -- 'createOf'. 'IsList' and 'IsString' instances can be
3750 -- used to conveniently construct arrays from literal values.
3851 -- 'OverloadedLists' extension or 'fromList' can be used to construct an
3952 -- array from a list literal. Similarly, 'OverloadedStrings' extension or
4053 -- 'fromList' can be used to construct an array from a string literal.
4154
42− -- Pure List APIs
43− , A.fromListN
44− , A.fromList
45−
46− -- Monadic APIs
47− , A.writeN -- drop new
48− , A.write -- full buffer
55+ -- ** From Stream
56+ , createOf
57+ , create
4958 , writeLastN -- drop old (ring buffer)
5059
51− -- * Conversion
60+ -- ** From List
61+ , fromListN
62+ , fromList
63+
64+ -- * To List
5265 -- 'GHC.Exts.toList' from "GHC.Exts" can be used to convert an array to a
5366 -- list.
54− , A.toList
67+ , toList
68+
69+ -- * To Stream
70+ , read
71+ , readRev
5572
5673 -- * Unfolds
57− , A.reader
58− , A.readerRev
74+ , reader
75+ , readerRev
5976
6077 -- * Casting
6178 , cast
6279 , asBytes
6380
6481 -- * Random Access
65− , A.length
82+ , length
6683 -- , (!!)
67− , A.getIndex
84+ , getIndex
6885
69− -- * Unbox Type Class
70− , Unbox (..)
86+ -- * Serialization
87+ , pinnedSerialize
88+ , deserialize
7189
72− -- * Pinning & Unpinning
73− , pin
74− , unpin
75− , isPinned
90+ -- * Re-exports
91+ , Unbox (..)
92+ , Serialize(..)
7693
7794 -- * Deprecated
78− , read
79− , readRev
95+ , writeN -- drop new
96+ , write -- full buffer
8097 )
8198where
8299
83100#include "inline.hs"
84101
85−import Streamly.Internal.Data.Unfold (Unfold)
86−import Streamly.Internal.Data.Array as A hiding (read, readRev)
87−import Streamly.Internal.Data.Array.Type (pin, unpin, isPinned)
102+import Streamly.Internal.Data.Array
103+import Streamly.Internal.Data.MutByteArray (Unbox(..), Serialize(..))
88104
89−import Streamly.Internal.Data.Unbox (Unbox (..))
90−import Prelude hiding (read)
105+import Prelude hiding (read, length)
91106
92107#include "DocTestDataArray.hs"
93108
@@ -103,7 +118,7 @@ import Prelude hiding (read)
103118--
104119-- Convert array to stream, and fold the stream:
105120--
106−-- >>> fold f arr = Stream.unfold Array.reader arr & Stream.fold f
121+-- >>> fold f arr = Array.read arr & Stream.fold f
107122-- >>> fold Fold.sum (Array.fromList [1,2,3::Int])
108123-- 6
109124--
@@ -111,7 +126,7 @@ import Prelude hiding (read)
111126--
112127-- Convert array to stream, transform, and fold back to array:
113128--
114−-- >>> amap f arr = Stream.unfold Array.reader arr & fmap f & Stream.fold Array.write
129+-- >>> amap f arr = Array.read arr & fmap f & Stream.fold (Array.createOf (Array.length arr))
115130-- >>> amap (+1) (Array.fromList [1,2,3::Int])
116131-- fromList [2,3,4]
117132--
@@ -138,12 +153,11 @@ import Prelude hiding (read)
138153-- See the 'fromPureStream' unreleased API to generate an array from an
139154-- Identity stream safely without using MonadIO constraint.
140155--
141−--
142156-- Note that 'Identity' streams can be generalized to IO streams:
143157--
144−-- >>> pure = Stream.fromList [1,2,3] :: Stream Identity Int
145−-- >>> generally = Stream.morphInner (return . runIdentity)
146−-- >>> Stream.fold Array.write (generally pure :: Stream IO Int)
158+-- >>> stream = Stream.fromList [1,2,3] :: Stream Identity Int
159+-- >>> pure2IO s = Stream.morphInner (return . runIdentity) s :: Stream IO Int
160+-- >>> Stream.fold Array.create (pure2IO stream)
147161-- fromList [1,2,3]
148162--
149163-- == Programming Tips
@@ -151,17 +165,3 @@ import Prelude hiding (read)
151165-- This module is designed to be imported qualified:
152166--
153167-- >>> import qualified Streamly.Data.Array as Array
154−
155−-- | Same as 'reader'
156−--
157−{-# DEPRECATED read "Please use 'reader' instead" #-}
158−{-# INLINE_NORMAL read #-}
159−read :: (Monad m, Unbox a) => Unfold m (Array a) a
160−read = reader
161−
162−-- | Same as 'readerRev'
163−--
164−{-# DEPRECATED readRev "Please use 'readerRev' instead" #-}
165−{-# INLINE_NORMAL readRev #-}
166−readRev :: (Monad m, Unbox a) => Unfold m (Array a) a
167−readRev = readerRev
core/src/Streamly/Data/Array/Generic.hs+16−13
@@ -15,34 +15,37 @@ module Streamly.Data.Array.Generic
1515 ( Array
1616
1717 -- * Construction
18− , A.fromListN
19− , A.fromList
18+ , fromListN
19+ , fromList
2020
2121 -- MonadicAPIs
22− , A.writeN
23− , A.write
22+ , createOf
23+ , create
2424
2525 -- * Conversion
26− , A.toList
26+ , toList
2727
2828 -- * Streams
29− , A.read
30− , A.readRev
29+ , read
30+ , readRev
3131
3232 -- * Unfolds
33− , A.reader
33+ , reader
3434 -- , A.readerRev
3535
3636 -- * Random Access
37− , A.length
38− , A.getIndex
37+ , length
38+ , getIndex
3939
4040 -- -- * Folding Arrays
4141 -- , A.streamFold
4242 -- , A.fold
43+
44+ -- * Deprecated
45+ , writeN
46+ , write
4347 )
4448where
4549
46−import Streamly.Internal.Data.Array.Generic (Array)
47−
48−import qualified Streamly.Internal.Data.Array.Generic as A
50+import Streamly.Internal.Data.Array.Generic
51+import Prelude hiding (length, read)
core/src/Streamly/Data/Fold.hs+150−118
@@ -10,6 +10,156 @@
1010-- Fast, composable stream consumers with ability to terminate, supporting
1111-- stream fusion.
1212--
13+-- == Using Folds
14+--
15+-- This module provides elementary folds and fold combinators that can be used
16+-- to consume a stream of data and reduce it to a final value, or transform it
17+-- in a stateful manner using scans. A data stream can be reduced into a stream
18+-- of folded data elements by folding segments of the stream. Fold combinators
19+-- can be used to compose multiple folds in parallel or to create a pipeline of
20+-- folds such that the next fold consumes the result of the previous fold. To
21+-- run these folds on a stream see 'Streamly.Data.Stream.fold',
22+-- 'Streamly.Data.Stream.scan', 'Streamly.Data.Stream.postscan',
23+-- 'Streamly.Data.Stream.scanMaybe', 'Streamly.Data.Stream.foldMany' and other
24+-- operations accepting 'Fold' type as argument "Streamly.Data.Stream".
25+--
26+-- == Reducing a Stream
27+--
28+-- A 'Fold' is a consumer of a stream of values. A fold driver (such as
29+-- 'Streamly.Data.Stream.fold') initializes the fold @accumulator@, runs the
30+-- fold @step@ function in a loop, processing the input stream one element at a
31+-- time and accumulating the result. The loop continues until the fold
32+-- terminates, at which point the accumulated result is returned.
33+--
34+-- For example, a 'sum' Fold represents a stream consumer that adds the values
35+-- in the input stream:
36+--
37+-- >>> Stream.fold Fold.sum $ Stream.fromList [1..100]
38+-- 5050
39+--
40+-- Conceptually, a 'Fold' is a data type that mimics a strict left fold
41+-- ('Data.List.foldl'). The above example is similar to a left fold using
42+-- @(+)@ as the step and @0@ as the initial value of the accumulator:
43+--
44+-- >>> Data.List.foldl' (+) 0 [1..100]
45+-- 5050
46+--
47+-- 'Fold's have an early termination capability e.g. the 'one' fold terminates
48+-- after consuming one element:
49+--
50+-- >>> Stream.fold Fold.one $ Stream.fromList [1..]
51+-- Just 1
52+--
53+-- The above example is similar to the following right fold:
54+--
55+-- >>> Prelude.foldr (\x _ -> Just x) Nothing [1..]
56+-- Just 1
57+--
58+-- 'Fold's can be combined together using combinators. For example, to create a
59+-- fold that sums first two elements in a stream:
60+--
61+-- >>> sumTwo = Fold.take 2 Fold.sum
62+-- >>> Stream.fold sumTwo $ Stream.fromList [1..100]
63+-- 3
64+--
65+-- == Parallel Composition
66+--
67+-- Folds can be combined to run in parallel on the same input. For example, to
68+-- compute the average of numbers in a stream without going through the stream
69+-- twice:
70+--
71+-- >>> avg = Fold.teeWith (/) Fold.sum (fmap fromIntegral Fold.length)
72+-- >>> Stream.fold avg $ Stream.fromList [1.0..100.0]
73+-- 50.5
74+--
75+-- Folds can be combined so as to partition the input stream over multiple
76+-- folds. For example, to count even and odd numbers in a stream:
77+--
78+-- >>> split n = if even n then Left n else Right n
79+-- >>> stream = fmap split $ Stream.fromList [1..100]
80+-- >>> countEven = fmap (("Even " ++) . show) Fold.length
81+-- >>> countOdd = fmap (("Odd " ++) . show) Fold.length
82+-- >>> f = Fold.partition countEven countOdd
83+-- >>> Stream.fold f stream
84+-- ("Even 50","Odd 50")
85+--
86+-- == Sequential Composition
87+--
88+-- Terminating folds can be combined to parse the stream serially such that the
89+-- first fold consumes the input until it terminates and the second fold
90+-- consumes the rest of the input until it terminates:
91+--
92+-- >>> f = Fold.splitWith (,) (Fold.take 8 Fold.toList) (Fold.takeEndBy (== '\n') Fold.toList)
93+-- >>> Stream.fold f $ Stream.fromList "header: hello\n"
94+-- ("header: ","hello\n")
95+--
96+-- == Splitting a Stream
97+--
98+-- A 'Fold' can be applied repeatedly on a stream to transform it to a stream
99+-- of fold results. To split a stream on newlines:
100+--
101+-- >>> f = Fold.takeEndBy (== '\n') Fold.toList
102+-- >>> Stream.fold Fold.toList $ Stream.foldMany f $ Stream.fromList "Hello there!\nHow are you\n"
103+-- ["Hello there!\n","How are you\n"]
104+--
105+-- Similarly, we can split the input of a fold too:
106+--
107+-- >>> Stream.fold (Fold.many f Fold.toList) $ Stream.fromList "Hello there!\nHow are you\n"
108+-- ["Hello there!\n","How are you\n"]
109+--
110+-- == Folds vs. Streams
111+--
112+-- We can often use streams or folds to achieve the same goal. However, streams
113+-- are more efficient in composition of producers (e.g.
114+-- 'Data.Stream.append' or 'Data.Stream.mergeBy') whereas folds are
115+-- more efficient in composition of consumers (e.g. 'splitWith', 'partition'
116+-- or 'teeWith').
117+--
118+-- Streams are producers, transformations on streams happen on the output side:
119+--
120+-- >>> :{
121+-- f stream =
122+-- Stream.filter odd stream
123+-- & fmap (+1)
124+-- & Stream.fold Fold.sum
125+-- :}
126+--
127+-- >>> f $ Stream.fromList [1..100 :: Int]
128+-- 2550
129+--
130+-- Folds are stream consumers with an input stream and an output value, stream
131+-- transformations on folds happen on the input side:
132+--
133+-- >>> :{
134+-- f =
135+-- Fold.filter odd
136+-- $ Fold.lmap (+1)
137+-- $ Fold.sum
138+-- :}
139+--
140+-- >>> Stream.fold f $ Stream.fromList [1..100 :: Int]
141+-- 2550
142+--
143+-- Notice the similiarity in the definition of @f@ in both cases, the only
144+-- difference is the composition by @&@ vs @$@ and the use @lmap@ vs @map@, the
145+-- difference is due to output vs input side transformations.
146+--
147+-- == Fusion Limitations
148+--
149+-- Folds support stream fusion for generating loops comparable to the speed of
150+-- C. However, it has some limitations. For fusion to work, the folds must be
151+-- inlined, folds must be statically known and not generated dynamically, folds
152+-- should not be passed recursively.
153+--
154+-- Another limitation is due to the quadratic complexity causing slowdown when
155+-- too many nested compositions are used. Especially, the performance of the
156+-- Applicative instance and splitting operations (e.g. 'splitWith') degrades
157+-- quadratically (O(n^2)) when combined @n@ times, roughly 8 or less sequenced
158+-- operations are fine. For these cases folds can be converted to parsers and
159+-- then used as ParserK.
160+--
161+-- == Experimental APIs
162+--
13163-- Please refer to "Streamly.Internal.Data.Fold" for more functions that have
14164-- not yet been released.
15165
@@ -21,9 +171,6 @@ module Streamly.Data.Fold
21171 --
22172 -- $setup
23173
24− -- * Overview
25− -- $overview
26−
27174 -- * Running A Fold
28175 drive
29176 -- XXX Should we have a stream returning function in fold module?
@@ -258,121 +405,6 @@ import Streamly.Internal.Data.Fold
258405
259406#include "DocTestDataFold.hs"
260407
261−-- $overview
262−--
263−-- A 'Fold' is a consumer of a stream of values. A fold driver (such as
264−-- 'Streamly.Data.Stream.fold') initializes the fold @accumulator@, runs the
265−-- fold @step@ function in a loop, processing the input stream one element at a
266−-- time and accumulating the result. The loop continues until the fold
267−-- terminates, at which point the accumulated result is returned.
268−--
269−-- For example, a 'sum' Fold represents a stream consumer that adds the values
270−-- in the input stream:
271−--
272−-- >>> Stream.fold Fold.sum $ Stream.fromList [1..100]
273−-- 5050
274−--
275−-- Conceptually, a 'Fold' is a data type that mimics a strict left fold
276−-- ('Data.List.foldl'). The above example is similar to a left fold using
277−-- @(+)@ as the step and @0@ as the initial value of the accumulator:
278−--
279−-- >>> Data.List.foldl' (+) 0 [1..100]
280−-- 5050
281−--
282−-- 'Fold's have an early termination capability e.g. the 'one' fold terminates
283−-- after consuming one element:
284−--
285−-- >>> Stream.fold Fold.one $ Stream.fromList [1..]
286−-- Just 1
287−--
288−-- The above example is similar to the following right fold:
289−--
290−-- >>> Prelude.foldr (\x _ -> Just x) Nothing [1..]
291−-- Just 1
292−--
293−-- 'Fold's can be combined together using combinators. For example, to create a
294−-- fold that sums first two elements in a stream:
295−--
296−-- >>> sumTwo = Fold.take 2 Fold.sum
297−-- >>> Stream.fold sumTwo $ Stream.fromList [1..100]
298−-- 3
299−--
300−-- Folds can be combined to run in parallel on the same input. For example, to
301−-- compute the average of numbers in a stream without going through the stream
302−-- twice:
303−--
304−-- >>> avg = Fold.teeWith (/) Fold.sum (fmap fromIntegral Fold.length)
305−-- >>> Stream.fold avg $ Stream.fromList [1.0..100.0]
306−-- 50.5
307−--
308−-- Folds can be combined so as to partition the input stream over multiple
309−-- folds. For example, to count even and odd numbers in a stream:
310−--
311−-- >>> split n = if even n then Left n else Right n
312−-- >>> stream = fmap split $ Stream.fromList [1..100]
313−-- >>> countEven = fmap (("Even " ++) . show) Fold.length
314−-- >>> countOdd = fmap (("Odd " ++) . show) Fold.length
315−-- >>> f = Fold.partition countEven countOdd
316−-- >>> Stream.fold f stream
317−-- ("Even 50","Odd 50")
318−--
319−-- Terminating folds can be combined to parse the stream serially such that the
320−-- first fold consumes the input until it terminates and the second fold
321−-- consumes the rest of the input until it terminates:
322−--
323−-- >>> f = Fold.splitWith (,) (Fold.take 8 Fold.toList) (Fold.takeEndBy (== '\n') Fold.toList)
324−-- >>> Stream.fold f $ Stream.fromList "header: hello\n"
325−-- ("header: ","hello\n")
326−--
327−-- A 'Fold' can be applied repeatedly on a stream to transform it to a stream
328−-- of fold results. To split a stream on newlines:
329−--
330−-- >>> f = Fold.takeEndBy (== '\n') Fold.toList
331−-- >>> Stream.fold Fold.toList $ Stream.foldMany f $ Stream.fromList "Hello there!\nHow are you\n"
332−-- ["Hello there!\n","How are you\n"]
333−--
334−-- Similarly, we can split the input of a fold too:
335−--
336−-- >>> Stream.fold (Fold.many f Fold.toList) $ Stream.fromList "Hello there!\nHow are you\n"
337−-- ["Hello there!\n","How are you\n"]
338−--
339−-- = Folds vs. Streams
340−--
341−-- We can often use streams or folds to achieve the same goal. However, streams
342−-- are more efficient in composition of producers (e.g.
343−-- 'Data.Stream.append' or 'Data.Stream.mergeBy') whereas folds are
344−-- more efficient in composition of consumers (e.g. 'splitWith', 'partition'
345−-- or 'teeWith').
346−--
347−-- Streams are producers, transformations on streams happen on the output side:
348−--
349−-- >>> :{
350−-- f stream =
351−-- Stream.filter odd stream
352−-- & fmap (+1)
353−-- & Stream.fold Fold.sum
354−-- :}
355−--
356−-- >>> f $ Stream.fromList [1..100 :: Int]
357−-- 2550
358−--
359−-- Folds are stream consumers with an input stream and an output value, stream
360−-- transformations on folds happen on the input side:
361−--
362−-- >>> :{
363−-- f =
364−-- Fold.filter odd
365−-- $ Fold.lmap (+1)
366−-- $ Fold.sum
367−-- :}
368−--
369−-- >>> Stream.fold f $ Stream.fromList [1..100 :: Int]
370−-- 2550
371−--
372−-- Notice the similiarity in the definition of @f@ in both cases, the only
373−-- difference is the composition by @&@ vs @$@ and the use @lmap@ vs @map@, the
374−-- difference is due to output vs input side transformations.
375−
376408--------------------------------------------------------------------------------
377409-- Deprecated
378410--------------------------------------------------------------------------------
core/src/Streamly/Data/MutArray.hs+18−12
@@ -32,22 +32,27 @@ module Streamly.Data.MutArray
3232 -- * Construction
3333
3434 -- Uninitialized Arrays
35− , new
36− , pinnedNew
35+ , emptyOf
36+ , pinnedEmptyOf
3737
3838 -- From containers
3939 , fromListN
4040 , fromList
41− , writeN -- drop new
42− , write -- full buffer
41+ , createOf
42+ , create
4343 -- writeLastN
4444
45+ -- * Pinning & Unpinning
46+ , pin
47+ , unpin
48+ , isPinned
49+
4550 -- * Appending elements
4651 , snoc
4752
4853 -- * Appending streams
49− , writeAppendN -- XXX Rename to writeSnocN?
50− , writeAppend -- XXX Rename to writeSnoc?
54+ , appendN
55+ , append
5156
5257 -- * Inplace mutation
5358 , putIndex
@@ -78,16 +83,17 @@ module Streamly.Data.MutArray
7883 -- * Size
7984 , length
8085
81− -- * Unbox Type Class
86+ -- * Re-exports
8287 , Unbox (..)
8388
84− -- * Pinning & Unpinning
85− , pin
86− , unpin
87− , isPinned
88−
8989 -- * Deprecated
9090 , newPinned
91+ , new
92+ , pinnedNew
93+ , writeN
94+ , write
95+ , writeAppendN
96+ , writeAppend
9197 )
9298where
9399
core/src/Streamly/Data/MutArray/Generic.hs+8−3
@@ -23,11 +23,11 @@ module Streamly.Data.MutArray.Generic
2323 MutArray
2424
2525 -- * Construction
26− , new
26+ , emptyOf
2727 , fromListN
2828 , fromList
29− , writeN
30− , write
29+ , createOf
30+ , create
3131
3232 -- * Appending elements
3333 , snoc
@@ -56,6 +56,11 @@ module Streamly.Data.MutArray.Generic
5656
5757 -- * Size
5858 , length
59+
60+ -- * Deprecated
61+ , new
62+ , writeN
63+ , write
5964 )
6065where
6166
core/src/Streamly/Data/MutByteArray.hsnew file+106
@@ -0,0 +1,106 @@
1+-- |
2+-- Module : Streamly.Data.MutByteArray
3+-- Copyright : (c) 2023 Composewell Technologies
4+-- License : BSD3-3-Clause
5+-- Maintainer : streamly@composewell.com
6+-- Portability : GHC
7+--
8+-- A low level byte Array type 'MutByteArray', along with type classes 'Unbox'
9+-- and 'Serialize' for fast binary serialization and deserialization of Haskell
10+-- values. Serialization, deserialization performance is similar to, and in
11+-- some cases many times better than the store package. Conceptually, the
12+-- 'Serialize' type class works in the same way as store.
13+--
14+-- Serialize instances are configurable to use constructor names (see
15+-- 'Streamly.Internal.Data.MutByteArray.encodeConstrNames'), record field names (see
16+-- 'Streamly.Internal.Data.MutByteArray.encodeRecordFields') instead of binary
17+-- encoded values. This is an experimental feature which allows JSON like
18+-- properties with faster speed. For example, you can change the order of
19+-- constructors or record fields without affecting serialized value.
20+--
21+-- Higher level unboxed array modules "Streamly.Data.Array" and
22+-- "Streamly.Data.MutArray" are built on top of this module. Unboxed arrays are
23+-- essentially serialized Haskell values. Array modules provide higher level
24+-- serialization routines like 'Streamly.Internal.Data.Array.pinnedSerialize'
25+-- and 'Streamly.Internal.Data.Array.deserialize' from the
26+-- "Streamly.Internal.Data.Array" module.
27+--
28+-- == Mutable Byte Array
29+--
30+-- 'MutByteArray' is a primitive mutable array in the IO monad. 'Unbox' and
31+-- 'Serialize' type classes use this primitive array to serialize data to and
32+-- deserialize it from. This array is used to build higher level unboxed
33+-- array types 'Streamly.Data.MutArray.MutArray' and 'Streamly.Data.Array.Array'.
34+--
35+-- == Using Unbox
36+--
37+-- The 'Unbox' type class is simple and used to serialize non-recursive fixed
38+-- size data types. This type class is primarily used to implement unboxed
39+-- arrays. Unboxed arrays are just a sequence of serialized fixed length
40+-- Haskell data types. Instances of this type class can be derived using
41+-- 'Generic' or template haskell based deriving functions provided in this
42+-- module.
43+--
44+-- Writing a data type to an array using the array creation routines in
45+-- "Streamly.Data.Array" or "Streamly.Data.MutArray" (e.g. @writeN@ or
46+-- @fromListN@), serializes the type to the array. Similarly, reading the data
47+-- type from the array deserializes it. You can also serialize and deserialize
48+-- directly to and from a 'MutByteArray', using the type class methods.
49+--
50+-- == Using Serialize
51+--
52+-- The 'Serialize' type class is a superset of the 'Unbox' type class, it can
53+-- serialize variable length data types as well e.g. Haskell lists. Use
54+-- 'deriveSerialize' to derive the instances of the type class automatically
55+-- and then use the type class methods to serialize and deserialize to and from
56+-- a 'MutByteArray'.
57+--
58+-- See 'Streamly.Internal.Data.Array.pinnedSerialize' and
59+-- 'Streamly.Internal.Data.Array.deserialize' for 'Array' type based
60+-- serialization.
61+--
62+-- == Comparing serialized values
63+--
64+-- When using the `Unbox` type class the same value may result in differing
65+-- serialized bytes because of unused uninitialized data in case of sum types.
66+-- Therefore, byte comparison of serialized values is not reliable.
67+--
68+-- However, the 'Serialize' type class guarantees that the serialized values
69+-- are always exactly the same and byte comparison of serialized is reliable.
70+--
71+module Streamly.Data.MutByteArray
72+ (
73+
74+ -- * Mutable Byte Array
75+ -- | The standard way to read from or write to a 'MutByteArray' is by using
76+ -- the 'Unbox' or 'Serialize' type class methods.
77+ MutByteArray
78+ , isPinned
79+ , pin
80+ , unpin
81+ , new
82+ , pinnedNew
83+
84+ -- * Unbox
85+ , Unbox(..)
86+ , deriveUnbox
87+
88+ -- * Serialize
89+ , Serialize(..)
90+
91+ -- ** Instance Config
92+ , SerializeConfig
93+ , inlineAddSizeTo
94+ , inlineSerializeAt
95+ , inlineDeserializeAt
96+
97+ -- ** Instance Deriving
98+ , deriveSerialize
99+ , deriveSerializeWith
100+ ) where
101+
102+--------------------------------------------------------------------------------
103+-- Imports
104+--------------------------------------------------------------------------------
105+
106+import Streamly.Internal.Data.MutByteArray
core/src/Streamly/Data/Parser.hs+130−60
@@ -7,12 +7,137 @@
77-- Stability : pre-release
88-- Portability : GHC
99--
10−-- Fast, composable stream consumers with ability to terminate, backtrack and
11−-- fail, supporting stream fusion. Parsers are a natural extension of
12−-- "Streamly.Data.Fold". Parsers and folds can be interconverted.
10+-- Parsers are stream consumers like folds with the following differences:
1311--
14−-- Please refer to "Streamly.Internal.Data.Parser" for functions that have
15−-- not yet been released.
12+-- * folds cannot fail but parsers can fail and backtrack.
13+-- * folds can be composed as a Tee but parsers cannot.
14+-- * folds can be used for scanning but parsers cannot.
15+-- * folds can be converted to parsers.
16+--
17+-- This module implements parsers with stream fusion which compile to efficient
18+-- loops comparable to the speed of C.
19+--
20+-- == Using Parsers
21+--
22+-- This module provides elementary parsers and parser combinators that can be
23+-- used to parse a stream of data. Additionally, all the folds from the
24+-- "Streamly.Data.Fold" module can be converted to parsers using 'fromFold'.
25+-- All the parsing functionality provided by popular parsing libraries, and
26+-- more is available. Also see "Streamly.Unicode.Parser" module for Char stream
27+-- parsers.
28+--
29+-- A data stream can be transformed to a stream of parsed data elements. Parser
30+-- combinators can be used to create a pipeline of folds or parsers such that
31+-- the next fold or parser consumes the result of the previous parser. See
32+-- 'Streamly.Data.Stream.parse' and 'Streamly.Data.Stream.parseMany' to run
33+-- these parsers on a stream.
34+--
35+-- == Parser vs ParserK
36+--
37+-- There are two functionally equivalent parsing modules,
38+-- "Streamly.Data.Parser" (this module) and "Streamly.Data.ParserK". The latter
39+-- is a CPS based wrapper over the former, and can be used for parsing in
40+-- general. "Streamly.Data.Parser" enables stream fusion and should be
41+-- preferred over "Streamly.Data.ParserK" for high performance stream parsing
42+-- use cases. However, there are a few cases where this module is not
43+-- suitable and ParserK should be used instead.
44+--
45+-- For static fusion, parser combinators have to use strict pattern matching on
46+-- arguments of type Parser. This leads to infinte loop when a parser is
47+-- defined recursively, due to strict evaluation of the recursive call. For
48+-- example, the following implementation loops infinitely because of the
49+-- recursive use of parser @p@ in the @*>@ combinator:
50+--
51+-- >>> import Streamly.Data.Parser (Parser)
52+-- >>> import qualified Streamly.Data.Fold as Fold
53+-- >>> import qualified Streamly.Data.Parser as Parser
54+-- >>> import qualified Streamly.Data.Stream as Stream
55+-- >>> import Control.Applicative ((<|>))
56+--
57+-- >>> :{
58+-- >>> p :: Monad m => Parser Char m String
59+-- >>> p = Parser.satisfy (== '(') *> p <|> Parser.fromFold Fold.toList
60+-- >>> :}
61+--
62+-- Use ParserK when recursive use is required:
63+--
64+-- >>> import Streamly.Data.ParserK (ParserK)
65+-- >>> import qualified Streamly.Data.StreamK as StreamK
66+-- >>> import qualified Streamly.Internal.Data.StreamK as StreamK (parse)
67+-- >>> import qualified Streamly.Internal.Data.ParserK as ParserK (adapt)
68+--
69+-- >>> :{
70+-- >>> p :: Monad m => ParserK Char m String
71+-- >>> p = ParserK.adapt (Parser.satisfy (== '(')) *> p <|> ParserK.adapt (Parser.fromFold Fold.toList)
72+-- >>> :}
73+--
74+-- >>> StreamK.parse p $ StreamK.fromStream $ Stream.fromList "hello"
75+-- Right "hello"
76+--
77+-- For this reason Applicative, Alternative or Monad compositions with
78+-- recursion cannot be used with the 'Parser' type. Alternative type class based
79+-- operations like 'asum' and Alternative based generic parser combinators use
80+-- recursion. Similarly, Applicative type class based operations like
81+-- 'Prelude.sequence' use recursion. Custom implementations of many such
82+-- operations are provided in this module (e.g. 'some', 'many'), and those
83+-- should be used instead.
84+--
85+-- Another limitation of Parser type is due to the quadratic complexity causing
86+-- slowdown when too many nested compositions are used. Especially Applicative,
87+-- Monad, Alternative instances, and sequenced parsing operations (e.g. nested
88+-- 'one', and 'splitWith') degrade the performance quadratically (O(n^2)) when
89+-- combined @n@ times, roughly 8 or less sequenced parsers are fine. READ THE
90+-- DOCS OF APPLICATIVE, MONAD AND ALTERNATIVE INSTANCES.
91+--
92+-- == Streaming Parsers
93+--
94+-- With 'Streamly.Data.ParserK.ParserK' you can use the generic Alternative
95+-- type class based parsers from the
96+-- <https://hackage.haskell.org/package/parser-combinators parser-combinators>
97+-- library or similar. However, we recommend that you use the equivalent
98+-- functionality from this module for better performance and for streaming
99+-- behavior.
100+--
101+-- Firstly, the combinators in this module are faster due to stream fusion.
102+-- Secondly, these are streaming in nature as the results can be passed
103+-- directly to other stream consumers (folds or parsers). The Alternative type
104+-- class based parsers would end up buffering all the results in lists before
105+-- they can be consumed.
106+--
107+-- When recursion or heavy nesting is needed use ParserK.
108+--
109+-- == Error Reporting
110+--
111+-- These parsers do not report the error context (e.g. line number or column).
112+-- This may be supported in future.
113+--
114+-- == Monad Transformer Stack
115+--
116+-- 'MonadTrans' instance is not provided. If the 'Parser' type is the top most
117+-- layer (which should be the case almost always) you can just use 'fromEffect'
118+-- to execute the lower layer monad effects.
119+--
120+-- == Parser vs ParserK Implementation
121+--
122+-- The 'Parser' type represents a stream consumer by composing state as data
123+-- which enables stream fusion. Stream fusion generates a tight loop without
124+-- any constructor allocations between the stages, providing C like performance
125+-- for the loop. Stream fusion works when multiple functions are combined in a
126+-- pipeline statically. Therefore, the operations in this module must be
127+-- inlined and must not be used recursively to allow for stream fusion.
128+--
129+-- The 'ParserK' type represents a stream consumer by composing function calls,
130+-- therefore, a function call overhead is incurred at each composition. It is
131+-- quite fast in general but may be a few times slower than a fused parser.
132+-- However, it allows for scalable dynamic composition especially parsers can
133+-- be used in recursive calls. Using the 'ParserK' type operations like
134+-- 'splitWith' provide linear (O(n)) performance with respect to the number of
135+-- compositions.
136+--
137+-- == Experimental APIs
138+--
139+-- Please refer to "Streamly.Internal.Data.Parser" for functions that have not
140+-- yet been released.
16141--
17142module Streamly.Data.Parser
18143 (
@@ -22,9 +147,6 @@ module Streamly.Data.Parser
22147 --
23148 -- $setup
24149
25− -- * Overview
26− -- $overview
27−
28150 -- * Parser Type
29151 Parser
30152
@@ -123,55 +245,3 @@ import Streamly.Internal.Data.Parser
123245import Prelude hiding (dropWhile, takeWhile, filter)
124246
125247#include "DocTestDataParser.hs"
126−
127−-- $overview
128−--
129−-- Several combinators in this module can be many times faster than CPS based
130−-- parsers because of stream fusion. For example,
131−-- 'Streamly.Internal.Data.Parser.many' combinator in this module is much
132−-- faster than the 'Control.Applicative.many' combinator of
133−-- 'Control.Applicative.Alternative' type class used by CPS based parsers.
134−--
135−-- The use of 'Alternative' type class, in parsers has another drawback.
136−-- Alternative based parsers use plain Haskell lists to collect the results. In
137−-- a strict Monad like IO, the results are necessarily buffered before they can
138−-- be consumed. This may not perform optimally in streaming applications
139−-- processing large amounts of data. Equivalent combinators in this module can
140−-- consume the results of parsing using a 'Fold' or another parser, thus
141−-- providing a scalable and composable consumer.
142−--
143−-- Note that these parsers do not report the error context (e.g. line number or
144−-- column). This may be supported in future.
145−--
146−-- mtl instances are not provided. If the 'Parser' type is the top most layer
147−-- (which should be the case almost always) you can just use 'fromEffect' to
148−-- execute the lower layer monad effects.
149−--
150−-- == Performance Notes
151−--
152−-- The 'Parser' type represents a stream consumer by composing state as data
153−-- which enables stream fusion. Stream fusion generates a tight loop without
154−-- any constructor allocations between the stages, providing C like performance
155−-- for the loop. Stream fusion works when multiple functions are combined in a
156−-- pipeline statically. Therefore, the operations in this module must be
157−-- inlined and must not be used recursively to allow for stream fusion. Note
158−-- that operations like 'sequence', and 'asum' that compose pasrers using
159−-- recursion should be avoided with these parsers. You can use these with the
160−-- 'ParserK' module instead.
161−--
162−-- Using the 'Parser' type, parsing operations like 'one', 'splitWith' etc.
163−-- degrade quadratically (O(n^2)) when combined many times. If you need to
164−-- combine these operations, say more than 8 times in a single loop, then you
165−-- should consider using the continuation style parser type 'ParserK' instead.
166−-- Also, if you need to use these operations in a recursive loop you should use
167−-- 'ParserK' instead.
168−--
169−-- The 'ParserK' type represents a stream consumer by composing function calls,
170−-- therefore, a function call overhead is incurred at each composition. It is
171−-- quite fast in general but may be a few times slower than a fused parser.
172−-- However, it allows for scalable dynamic composition especially parsers can
173−-- be used in recursive calls. Using the 'ParserK' type operations like
174−-- 'splitWith' provide linear (O(n)) performance with respect to the number of
175−-- compositions..
176−--
177−-- 'Parser' and 'ParserK' types can be interconverted.
core/src/Streamly/Data/ParserK.hs+105−10
@@ -1,3 +1,4 @@
1+{-# LANGUAGE CPP #-}
12-- |
23-- Module : Streamly.Data.ParserK
34-- Copyright : (c) 2023 Composewell Technologies
@@ -6,26 +7,112 @@
67-- Stability : pre-release
78-- Portability : GHC
89--
9−-- Parsers using Continuation Passing Style (CPS). See notes in
10−-- "Streamly.Data.Parser" module to know when to use this module.
10+-- See the general notes about parsing in the "Streamly.Data.Parser" module.
11+-- This module implements a using Continuation Passing Style (CPS) wrapper over
12+-- the "Streamly.Data.Parser" module. It is as fast or faster than attoparsec.
1113--
12−-- To run a 'ParserK' use 'Streamly.Data.StreamK.parseChunks'.
14+-- == Parser vs ParserK
15+--
16+-- 'ParserK' is preferred over 'Streamly.Data.Parser.Parser' when extensive
17+-- applicative, alternative and monadic composition is required, or when
18+-- recursive or dynamic composition of parsers is required. The
19+-- 'Streamly.Data.Parser.Parser' type fuses statically and creates efficient
20+-- loops whereas 'ParserK' uses function call based composition and has
21+-- comparatively larger runtime overhead but it is better suited to the
22+-- specific use cases mentioned above. 'ParserK' also allows to efficient parse
23+-- a stream of arrays, it can also break the input stream into a parse result
24+-- and remaining stream so that the stream can be parsed independently in
25+-- segments.
26+--
27+-- == Using ParserK
28+--
29+-- All the parsers from the "Streamly.Data.Parser" module can be adapted to
30+-- ParserK using the 'Streamly.Data.ParserK.adaptC',
31+-- 'Streamly.Internal.Data.ParserK.adapt', and
32+-- 'Streamly.Internal.Data.ParserK.adaptCG' combinators.
33+--
34+-- 'Streamly.Data.StreamK.parseChunks' runs a parser on a stream of unboxed
35+-- arrays, this is the preferred and most efficient way to parse chunked input.
36+-- The more general 'Streamly.Data.StreamK.parseBreakChunks' function returns
37+-- the remaining stream as well along with the parse result. There are
38+-- 'Streamly.Internal.Data.StreamK.parseChunksGeneric',
39+-- 'Streamly.Internal.Data.StreamK.parseBreakChunksGeneric' as well to run
40+-- parsers on boxed arrays. 'Streamly.Internal.Data.StreamK.parse',
41+-- 'Streamly.Internal.Data.StreamK.parseBreak' run parsers on a stream of
42+-- individual elements instead of stream of arrays.
43+--
44+-- == Monadic Composition
45+--
46+-- Monad composition can be used for lookbehind parsers, we can dynamically
47+-- compose new parsers based on the results of the previously parsed values.
48+--
49+-- If we have to parse "a9" or "9a" but not "99" or "aa" we can use the
50+-- following non-monadic, backtracking parser:
51+--
52+-- >>> digits p1 p2 = ((:) <$> p1 <*> ((:) <$> p2 <*> pure []))
53+-- >>> :{
54+-- backtracking :: Monad m => ParserK Char m String
55+-- backtracking = ParserK.adapt $
56+-- digits (Parser.satisfy isDigit) (Parser.satisfy isAlpha)
57+-- <|>
58+-- digits (Parser.satisfy isAlpha) (Parser.satisfy isDigit)
59+-- :}
60+--
61+-- We know that if the first parse resulted in a digit at the first place then
62+-- the second parse is going to fail. However, we waste that information and
63+-- parse the first character again in the second parse only to know that it is
64+-- not an alphabetic char. By using lookbehind in a 'Monad' composition we can
65+-- avoid redundant work:
66+--
67+-- >>> data DigitOrAlpha = Digit Char | Alpha Char
68+--
69+-- >>> :{
70+-- lookbehind :: Monad m => ParserK Char m String
71+-- lookbehind = do
72+-- x1 <- ParserK.adapt $
73+-- Digit <$> Parser.satisfy isDigit
74+-- <|> Alpha <$> Parser.satisfy isAlpha
75+-- -- Note: the parse depends on what we parsed already
76+-- x2 <- ParserK.adapt $
77+-- case x1 of
78+-- Digit _ -> Parser.satisfy isAlpha
79+-- Alpha _ -> Parser.satisfy isDigit
80+-- return $ case x1 of
81+-- Digit x -> [x,x2]
82+-- Alpha x -> [x,x2]
83+-- :}
84+--
85+-- == Experimental APIs
86+--
87+-- Please refer to "Streamly.Internal.Data.ParserK" for functions that have
88+-- not yet been released.
1389--
1490module Streamly.Data.ParserK
1591 (
92+ -- * Setup
93+ -- | To execute the code examples provided in this module in ghci, please
94+ -- run the following commands first.
95+ --
96+ -- $setup
97+
1698 -- * Parser Type
1799 ParserK
18100
19101 -- * Parsers
20102 -- ** Conversions
21− , fromFold
22− , fromParser
103+ , adapt
104+ , adaptC
105+ , adaptCG
23106 -- , toParser
24107
25108 -- ** Without Input
26109 , fromPure
27110 , fromEffect
28111 , die
112+
113+ -- * Deprecated
114+ , fromFold
115+ , fromParser
29116 )
30117
31118where
@@ -33,12 +120,20 @@ where
33120import Control.Monad.IO.Class (MonadIO)
34121import Streamly.Internal.Data.Fold (Fold)
35122import Streamly.Internal.Data.Unbox (Unbox)
123+import Streamly.Internal.Data.Array (Array)
36124import qualified Streamly.Internal.Data.Parser as ParserD
37125
38−import Streamly.Internal.Data.Parser.ParserK.Type
126+import Streamly.Internal.Data.ParserK.Type
39127
40−-- | Convert a 'Fold' to a 'ParserK'.
41−--
128+#include "DocTestDataParserK.hs"
129+
130+{-# DEPRECATED fromFold "Please use \"ParserK.adaptC . Parser.fromFold\" instead." #-}
42131{-# INLINE fromFold #-}
43−fromFold :: (MonadIO m, Unbox a) => Fold m a b -> ParserK a m b
44−fromFold = fromParser . ParserD.fromFold
132+fromFold :: (MonadIO m, Unbox a) => Fold m a b -> ParserK (Array a) m b
133+fromFold = adaptC . ParserD.fromFold
134+
135+{-# DEPRECATED fromParser "Please use \"adaptC\" instead." #-}
136+{-# INLINE fromParser #-}
137+fromParser ::
138+ (MonadIO m, Unbox a) => ParserD.Parser a m b -> ParserK (Array a) m b
139+fromParser = adaptC
core/src/Streamly/Data/Stream.hs+131−46
@@ -42,9 +42,6 @@ module Streamly.Data.Stream
4242 -- * Construction
4343 -- | Functions ending in the general shape @b -> Stream m a@.
4444 --
45− -- See also: "Streamly.Internal.Data.Stream.StreamD.Generate" for
46− -- @Pre-release@ functions.
47− --
4845 -- Useful Idioms:
4946 --
5047 -- >>> fromIndices f = fmap f $ Stream.enumerateFrom 0
@@ -73,6 +70,15 @@ module Streamly.Data.Stream
7370
7471 -- ** From Values
7572 -- | Generate a monadic stream from a seed value or values.
73+ --
74+ -- Non-empty pure streams can be cycled using repeat and unfold:
75+ --
76+ -- >>> cycle = Stream.unfoldMany Unfold.fromList . Stream.repeat
77+ --
78+ -- Non-empty StreamK can be cycled using append:
79+ --
80+ -- >>> cycle xs = let ys = xs `StreamK.append` ys in ys
81+ --
7682 , fromPure
7783 , fromEffect
7884 , repeat
@@ -117,8 +123,6 @@ module Streamly.Data.Stream
117123 -- | Functions ending in the general shape @Stream m a -> m b@ or @Stream m
118124 -- a -> m (b, Stream m a)@
119125 --
120− -- See also: "Streamly.Internal.Data.Stream.StreamD.Eliminate" for @Pre-release@
121− -- functions.
122126
123127-- EXPLANATION: In imperative terms a fold can be considered as a loop over the stream
124128-- that reduces the stream to a single value.
@@ -217,21 +221,57 @@ module Streamly.Data.Stream
217221 -- operations like mapping a function over the stream.
218222 , foldrM
219223 , foldr
224+ -- foldr1
220225
221226 -- ** Specific Folds
222− -- | Usually you can use the folds in "Streamly.Data.Fold". However, some
223− -- folds that may be commonly used or may have an edge in performance in
224− -- some cases are provided here.
225− --
226− -- Useful idioms:
227+ -- | Streams are folded using folds in "Streamly.Data.Fold". Here are some
228+ -- idioms and equivalents of Data.List APIs using folds:
227229 --
228230 -- >>> foldlM' f a = Stream.fold (Fold.foldlM' f a)
229− -- >>> foldl1 f = Stream.fold (Fold.foldl1' f)
231+ -- >>> foldl1' f = Stream.fold (Fold.foldl1' f)
230232 -- >>> foldl' f a = Stream.fold (Fold.foldl' f a)
231233 -- >>> drain = Stream.fold Fold.drain
232234 -- >>> mapM_ f = Stream.fold (Fold.drainMapM f)
233235 -- >>> length = Stream.fold Fold.length
236+ -- >>> genericLength = Stream.fold Fold.lengthGeneric
234237 -- >>> head = Stream.fold Fold.one
238+ -- >>> last = Stream.fold Fold.latest
239+ -- >>> null = Stream.fold Fold.null
240+ -- >>> and = Stream.fold Fold.and
241+ -- >>> or = Stream.fold Fold.or
242+ -- >>> any p = Stream.fold (Fold.any p)
243+ -- >>> all p = Stream.fold (Fold.all p)
244+ -- >>> sum = Stream.fold Fold.sum
245+ -- >>> product = Stream.fold Fold.product
246+ -- >>> maximum = Stream.fold Fold.maximum
247+ -- >>> maximumBy cmp = Stream.fold (Fold.maximumBy cmp)
248+ -- >>> minimum = Stream.fold Fold.minimum
249+ -- >>> minimumBy cmp = Stream.fold (Fold.minimumBy cmp)
250+ -- >>> elem x = Stream.fold (Fold.elem x)
251+ -- >>> notElem x = Stream.fold (Fold.notElem x)
252+ -- >>> lookup x = Stream.fold (Fold.lookup x)
253+ -- >>> find p = Stream.fold (Fold.find p)
254+ -- >>> (!?) i = Stream.fold (Fold.index i)
255+ -- >>> genericIndex i = Stream.fold (Fold.indexGeneric i)
256+ -- >>> elemIndex x = Stream.fold (Fold.elemIndex x)
257+ -- >>> findIndex p = Stream.fold (Fold.findIndex p)
258+ --
259+ -- Some equivalents of Data.List APIs from the Stream module:
260+ --
261+ -- >>> head = fmap (fmap fst) . Stream.uncons
262+ -- >>> tail = fmap (fmap snd) . Stream.uncons
263+ -- >>> tail = Stream.tail -- unreleased API
264+ -- >>> init = Stream.init -- unreleased API
265+ --
266+ -- A Stream based toList fold implementation is provided below because it
267+ -- has a better performance compared to the fold.
268+
269+ -- Functions in Data.List, missing here:
270+ -- unsnoc = Stream.parseBreak (Parser.init Fold.toList)
271+ -- genericTake
272+ -- genericDrop
273+ -- genericSplitAt
274+ -- genericReplicate
235275 , toList
236276
237277 -- * Mapping
@@ -259,8 +299,6 @@ module Streamly.Data.Stream
259299 -- * Scanning
260300 -- | Stateful one-to-one transformations.
261301 --
262− -- See also: "Streamly.Internal.Data.Stream.StreamD.Transform" for
263− -- @Pre-release@ functions.
264302
265303 {-
266304 -- ** Left scans
@@ -287,6 +325,15 @@ module Streamly.Data.Stream
287325 , scan
288326 , postscan
289327 -- XXX postscan1 can be implemented using Monoids or Refolds.
328+ -- The following scans from Data.List are not provided.
329+ -- XXX scanl
330+ -- XXX scanl1
331+ -- XXX scanr
332+ -- XXX scanr1
333+ -- XXX mapAccumL
334+ -- XXX mapAccumR
335+ -- XXX inits
336+ -- XXX tails
290337
291338 -- ** Specific scans
292339 -- Indexing can be considered as a special type of zipping where we zip a
@@ -295,6 +342,8 @@ module Streamly.Data.Stream
295342
296343 -- * Insertion
297344 -- | Add elements to the stream.
345+ --
346+ -- >>> insert = Stream.insertBy compare
298347
299348 -- Inserting elements is a special case of interleaving/merging streams.
300349 , insertBy
@@ -330,12 +379,15 @@ module Streamly.Data.Stream
330379 -- "Streamly.Internal.Data.Fold" can be used along with 'scanMaybe' to
331380 -- perform stateful filtering operations in general.
332381 --
333− -- Useful idioms:
382+ -- Idioms and equivalents of Data.List APIs:
334383 --
335384 -- >>> deleteBy cmp x = Stream.scanMaybe (Fold.deleteBy cmp x)
385+ -- >>> deleteBy = Stream.deleteBy -- unreleased API
386+ -- >>> delete = deleteBy (==)
336387 -- >>> findIndices p = Stream.scanMaybe (Fold.findIndices p)
337388 -- >>> elemIndices a = findIndices (== a)
338389 -- >>> uniq = Stream.scanMaybe (Fold.uniqBy (==))
390+ -- >>> partition p = Stream.fold (Fold.partition Fold.toList Fold.toList) . fmap (if p then Left else Right)
339391 , scanMaybe
340392 , take
341393 , takeWhile
@@ -343,6 +395,8 @@ module Streamly.Data.Stream
343395 , drop
344396 , dropWhile
345397 , dropWhileM
398+ -- XXX write to an array in reverse and then read in reverse
399+ -- > dropWhileEnd = reverse . dropWhile p . reverse
346400
347401 -- XXX These are available as scans in folds. We need to check the
348402 -- performance though. If these are common and we need convenient stream
@@ -379,6 +433,9 @@ module Streamly.Data.Stream
379433 -- >>> s = s1 `Stream.append` s2
380434
381435 -- ** Appending
436+ -- | Equivalent of Data.List append:
437+ --
438+ -- >>> (++) = Stream.append
382439 , append
383440
384441 -- ** Interleaving
@@ -389,8 +446,14 @@ module Streamly.Data.Stream
389446 , mergeByM
390447
391448 -- ** Zipping
449+ -- | Idioms and equivalents of Data.List APIs:
450+ --
451+ -- >>> zip = Stream.zipWith (,)
452+ -- >>> unzip = Stream.fold (Fold.unzip Fold.toList Fold.toList)
392453 , zipWith
393454 , zipWithM
455+ -- XXX zipWith3,4,5,6,7
456+ -- XXX unzip3,4,5,6,7
394457 -- , ZipStream (..)
395458
396459 -- ** Cross Product
@@ -423,48 +486,89 @@ module Streamly.Data.Stream
423486 -- One dimension loops are just a special case of nested loops. For
424487 -- example, 'concatMap' can degenerate to a simple map operation:
425488 --
426− -- > map f m = S.concatMap (\x -> S.fromPure (f x)) m
489+ -- >>> map f m = Stream.concatMap (\x -> Stream.fromPure (f x)) m
427490 --
428491 -- Similarly, 'concatMap' can perform filtering by mapping an element to a
429492 -- 'nil' stream:
430493 --
431− -- > filter p m = S.concatMap (\x -> if p x then S.fromPure x else S.nil) m
494+ -- >>> filter p m = Stream.concatMap (\x -> if p x then Stream.fromPure x else Stream.nil) m
432495 --
496+ -- Idioms and equivalents of Data.List APIs:
497+ --
498+ -- >>> concat = Stream.concatMap id
499+ -- >>> unlines = Stream.interposeSuffix '\n'
500+ -- >>> unlines = Stream.intercalateSuffix Unfold.fromList "\n"
501+ -- >>> unwords = Stream.interpose ' '
502+ -- >>> unwords = Stream.intercalate Unfold.fromList " "
433503
434504 , concatEffect
435505 , concatMap
436506 , concatMapM
437507
438508 -- * Repeated Fold
439− -- | Useful idioms:
509+ -- | Idioms and equivalents of Data.List APIs:
440510 --
441− -- >>> splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
442− -- >>> splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
443− -- >>> groupsBy eq f = Stream.parseMany (Parser.groupBy eq f)
444− -- >>> groupsByRolling eq f = Stream.parseMany (Parser.groupByRolling eq f)
445− -- >>> groupsOf n f = Stream.foldMany (Fold.take n f)
511+ -- >>> groupsOf n = Stream.foldMany (Fold.take n Fold.toList)
512+ -- >>> groupBy eq = Stream.groupsWhile eq Fold.toList
513+ -- >>> groupBy eq = Stream.parseMany (Parser.groupBy eq Fold.toList)
514+ -- >>> groupsByRolling eq = Stream.parseMany (Parser.groupByRolling eq Fold.toList)
515+ -- >>> groups = groupBy (==)
446516 , foldMany -- XXX Rename to foldRepeat
447517 , groupsOf
448518 , parseMany
449519
450520 -- * Splitting
521+ -- | Idioms and equivalents of Data.List APIs:
522+ --
523+ -- >>> splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
524+ -- >>> splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
525+ -- >>> lines = splitOnSuffix (== '\n')
526+ -- >>> words = Stream.wordsBy isSpace
527+ -- >>> splitAt n = Stream.fold (Fold.splitAt n Fold.toList Fold.toList)
528+ -- >>> span p = Parser.splitWith (,) (Parser.takeWhile p Fold.toList) (Parser.fromFold Fold.toList)
529+ -- >>> break p = span (not . p)
451530 , splitOn
452531 , wordsBy
453532
454533 -- * Buffered Operations
455534 -- | Operations that require buffering of the stream.
456535 -- Reverse is essentially a left fold followed by an unfold.
536+ --
537+ -- Idioms and equivalents of Data.List APIs:
538+ --
539+ -- >>> nub = Stream.fold Fold.toList . Stream.scanMaybe Fold.nub
540+ -- >>> nub = Stream.ordNub -- unreleased API
541+ -- >>> sortBy = StreamK.sortBy
542+ -- >>> sortOn f = StreamK.sortOn -- unreleased API
543+ -- >>> deleteFirstsBy = Stream.deleteFirstsBy -- unreleased
544+ -- >>> (\\) = Stream.deleteFirstsBy (==) -- unreleased
545+ -- >>> intersectBy = Stream.intersectBy -- unreleased
546+ -- >>> intersect = Stream.intersectBy (==) -- unreleased
547+ -- >>> unionBy = Stream.unionBy -- unreleased
548+ -- >>> union = Stream.unionBy (==) -- unreleased
549+ --
457550 , reverse
551+ -- XXX transpose: write the streams to arrays and then stream transposed.
552+ -- XXX subsequences
553+ -- XXX permutations
554+ -- , nub
555+ -- , ordNub
556+ -- , nubBy
458557
459558 -- * Multi-Stream folds
460559 -- | Operations that consume multiple streams at the same time.
461560 , eqBy
462561 , cmpBy
463562 , isPrefixOf
563+ , isInfixOf
564+ -- , isSuffixOf
565+ -- , isSuffixOfUnbox
464566 , isSubsequenceOf
465567
466568 -- trimming sequences
467− , stripPrefix
569+ -- , stripPrefix
570+ -- , stripSuffix
571+ -- , stripSuffixUnbox
468572
469573 -- Exceptions and resource management depend on the "exceptions" package
470574 -- XXX We can have IO Stream operations not depending on "exceptions"
@@ -485,9 +589,6 @@ module Streamly.Data.Stream
485589 -- For example, instead of calling them on a stream of chars call them on a
486590 -- stream of arrays before flattening it to a stream of chars.
487591 --
488− -- See also: "Streamly.Internal.Data.Stream.StreamD.Exception" for
489− -- @Pre-release@ functions.
490−
491592 , onException
492593 , handle
493594
@@ -521,25 +622,20 @@ module Streamly.Data.Stream
521622 , runReaderT
522623 , runStateT
523624
625+ -- XXX Arrays could be different types, therefore, this should be in
626+ -- specific array module. Or maybe we should abstract over array types.
524627 -- * Stream of Arrays
525628 , Array.chunksOf
526− , chunksSplitOn
527629 )
528630where
529631
530−import Control.Monad.IO.Class (MonadIO)
531−import Data.Word (Word8)
532−import Streamly.Data.Array (Array)
533632import Streamly.Internal.Data.Stream
534633import Prelude
535634 hiding (filter, drop, dropWhile, take, takeWhile, zipWith, foldr,
536− foldl, map, mapM, mapM_, sequence, all, any, sum, product, elem,
537− notElem, maximum, minimum, head, last, tail, length, null,
538− reverse, iterate, init, and, or, lookup, foldr1, (!!),
539− scanl, scanl1, repeat, replicate, concatMap, span)
635+ mapM, sequence, reverse, iterate, foldr1, repeat, replicate,
636+ concatMap)
540637
541638import qualified Streamly.Internal.Data.Array.Type as Array
542−import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
543639
544640#include "DocTestDataStream.hs"
545641
@@ -648,14 +744,3 @@ import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
648744--
649745-- 'Stream' and 'StreamK' types can be interconverted. See
650746-- "Streamly.Data.StreamK" module for conversion operations.
651−
652−-- | Split a stream of arrays on a given separator byte, dropping the separator
653−-- and coalescing all the arrays between two separators into a single array.
654−--
655−{-# INLINE chunksSplitOn #-}
656−chunksSplitOn
657− :: (MonadIO m)
658− => Word8
659− -> Stream m (Array Word8)
660− -> Stream m (Array Word8)
661−chunksSplitOn = ArrayStream.splitOn
core/src/Streamly/Data/StreamK.hs+2−1
@@ -95,7 +95,8 @@ module Streamly.Data.StreamK
9595 -- , foldBreak
9696
9797 -- ** Parsing
98− -- , parseBreak
98+ , parse
99+ , parseBreak
99100 , parseBreakChunks
100101 , parseChunks
101102
core/src/Streamly/Data/Unbox.hsdeleted−16
@@ -1,16 +0,0 @@
1−-- |
2−-- Module : Streamly.Data.Unbox
3−-- Copyright : (c) 2023 Composewell Technologies
4−-- License : BSD3-3-Clause
5−-- Maintainer : streamly@composewell.com
6−-- Portability : GHC
7−--
8−module Streamly.Data.Unbox
9− ( Unbox(..)
10− , MutableByteArray
11− , deriveUnbox
12− , deriveUnboxWith
13− ) where
14−
15−import Streamly.Internal.Data.Unbox
16−import Streamly.Internal.Data.Unbox.TH
core/src/Streamly/Data/Unfold.hs+3−1
@@ -39,7 +39,7 @@ module Streamly.Data.Unfold
3939
4040 -- * Unfolds
4141 -- One to one correspondence with
42− -- "Streamly.Internal.Data.Stream.StreamD.Generate"
42+ -- "Streamly.Internal.Data.Stream.Generate"
4343
4444 -- ** Basic Constructors
4545 , unfoldrM
@@ -65,6 +65,8 @@ module Streamly.Data.Unfold
6565 -- ** Mapping on Input
6666 , lmap
6767 , lmapM
68+ , first
69+ , second
6870
6971 -- ** Mapping on Output
7072 , mapM
core/src/Streamly/FileSystem/Handle.hs+7−30
@@ -1,5 +1,4 @@
11{-# LANGUAGE CPP #-}
2−#include "inline.hs"
32
43-- |
54-- Module : Streamly.FileSystem.Handle
@@ -82,13 +81,13 @@ module Streamly.FileSystem.Handle
8281 -- position of the file handle. The stream ends as soon as EOF is
8382 -- encountered.
8483
85− -- -- *** Streams
86− -- , read
87− -- , readWith
88− -- , readChunks
89− -- , readChunksWith
84+ -- *** Streams
85+ , read
86+ , readWith
87+ , readChunks
88+ , readChunksWith
9089
91− -- -- *** Unfolds
90+ -- *** Unfolds
9291 , reader
9392 , readerWith
9493 , chunkReader
@@ -105,36 +104,14 @@ module Streamly.FileSystem.Handle
105104 , writeChunks
106105
107106 -- * Deprecated
108− , read
109107 , readWithBufferOf
110− , readChunks
111108 , readChunksWithBufferOf
112109 , writeChunksWithBufferOf
113110 , writeWithBufferOf
114111 )
115112where
116113
117−import Control.Monad.IO.Class (MonadIO(..))
118−import Data.Word (Word8)
119−import Streamly.Internal.Data.Array.Type (Array)
120−import Streamly.Internal.Data.Unfold.Type (Unfold)
121−import System.IO (Handle)
122−
123−import Streamly.Internal.FileSystem.Handle hiding (read, readChunks)
114+import Streamly.Internal.FileSystem.Handle
124115import Prelude hiding (read)
125116
126117#include "DocTestFileSystemHandle.hs"
127−
128−-- | Same as 'reader'
129−--
130−{-# DEPRECATED read "Please use 'reader' instead" #-}
131−{-# INLINE read #-}
132−read :: MonadIO m => Unfold m Handle Word8
133−read = reader
134−
135−-- | Same as 'chunkReader'
136−--
137−{-# DEPRECATED readChunks "Please use 'chunkReader' instead" #-}
138−{-# INLINE readChunks #-}
139−readChunks :: MonadIO m => Unfold m Handle (Array Word8)
140−readChunks = chunkReader
core/src/Streamly/Internal/Data/Array.hs+609−162
@@ -2,7 +2,6 @@
22-- |
33-- Module : Streamly.Internal.Data.Array
44-- Copyright : (c) 2019 Composewell Technologies
5−--
65-- License : BSD3
76-- Maintainer : streamly@composewell.com
87-- Stability : experimental
@@ -17,59 +16,33 @@ module Streamly.Internal.Data.Array
1716 -- $design
1817
1918 -- * The Array Type
20− Array
19+ module Streamly.Internal.Data.Array.Type
2120
2221 -- * Construction
23−
24− -- Pure List APIs
25− , A.fromListN
26− , A.fromList
27−
28− -- Stream Folds
29− , fromStreamN
30− , fromStream
31−
3222 -- Monadic Folds
33− , A.writeN -- drop new
34− , A.pinnedWriteNAligned
35− , A.write -- full buffer
3623 , writeLastN
3724
38− -- * Elimination
39− -- ** Conversion
40− , A.toList
41−
42− -- ** Streams
43− , A.read
44− , A.readRev
45−
46− -- ** Unfolds
47− , reader
48− , readerUnsafe
49− , A.readerRev
50− , producer -- experimental
51−
5225 -- * Random Access
5326 -- , (!!)
5427 , getIndex
55− , A.unsafeIndex -- XXX Rename to getIndexUnsafe??
5628 , getIndexRev
57− , last -- XXX getIndexLast?
58− , getIndices
59− , getIndicesFromThenTo
29+ , last -- XXX getLastIndex?
6030 -- , getIndicesFrom -- read from a given position to the end of file
6131 -- , getIndicesUpto -- read from beginning up to the given position
6232 -- , getIndicesFromTo
6333 -- , getIndicesFromRev -- read from a given position to the beginning of file
6434 -- , getIndicesUptoRev -- read from end to the given position in file
35+ , indexReader
36+ , indexReaderFromThenTo
6537
6638 -- * Size
67− , length
6839 , null
6940
7041 -- * Search
7142 , binarySearch
7243 , findIndicesOf
44+ -- getIndicesOf
45+ , indexFinder -- see splitOn
7346 -- , findIndexOf
7447 -- , find
7548
@@ -77,66 +50,96 @@ module Streamly.Internal.Data.Array
7750 , cast
7851 , asBytes
7952 , castUnsafe
80− , asPtrUnsafe
8153 , asCStringUnsafe
82− , A.unsafeFreeze -- asImmutableUnsafe?
83− , A.unsafeThaw -- asMutableUnsafe?
8454
8555 -- * Subarrays
8656 , getSliceUnsafe
8757 -- , getSlice
88− , genSlicesFromLen
89− , getSlicesFromLen
58+ , sliceIndexerFromLen
59+ , slicerFromLen
9060 , splitOn
9161
9262 -- * Streaming Operations
9363 , streamTransform
9464
95− -- ** Folding
65+ -- * Folding
9666 , streamFold
9767 , fold
9868
69+ -- * Stream of Arrays
70+
71+ -- XXX these are probably not very useful to have in this module as we can
72+ -- express these idiomatically using streams.
73+ , interpose
74+ , interposeSuffix
75+ , intercalateSuffix
76+
77+ , compactLE
78+ , pinnedCompactLE
79+ , compactOnByte
80+ , compactOnByteSuffix
81+
82+ , foldBreakChunks
83+ , foldChunks
84+ , foldBreakChunksK
85+ , parseBreakChunksK
86+
87+ -- * Serialization
88+ , encodeAs
89+ , serialize
90+ , pinnedSerialize
91+ , deserialize
92+
9993 -- * Deprecated
100− , A.toStream
101− , A.toStreamRev
94+ , genSlicesFromLen
95+ , getSlicesFromLen
96+ , getIndices
10297 )
10398where
10499
100+#include "assert.hs"
105101#include "inline.hs"
106102#include "ArrayMacros.h"
107103
108−import Control.Exception (assert)
109−import Control.Monad (when)
110104import Control.Monad.IO.Class (MonadIO(..))
105+-- import Data.Bifunctor (first)
106+-- import Data.Either (fromRight)
111107import Data.Functor.Identity (Identity)
112108import Data.Proxy (Proxy(..))
113109import Data.Word (Word8)
114110import Foreign.C.String (CString)
115111import Foreign.Ptr (castPtr)
116112import Foreign.Storable (Storable)
113+import GHC.Types (SPEC(..))
117114import Streamly.Internal.Data.Unbox (Unbox(..))
118115import Prelude hiding (length, null, last, map, (!!), read, concat)
119116
120−import Streamly.Internal.Data.MutArray.Type (ArrayUnsafe(..))
121−import Streamly.Internal.Data.Array.Type
122− (Array(..), length, asPtrUnsafe)
117+import Streamly.Internal.Data.MutByteArray.Type (PinnedState(..), MutByteArray)
118+import Streamly.Internal.Data.Serialize.Type (Serialize)
123119import Streamly.Internal.Data.Fold.Type (Fold(..))
124−import Streamly.Internal.Data.Producer.Type (Producer(..))
125−import Streamly.Internal.Data.Stream (Stream)
126−import Streamly.Internal.Data.Tuple.Strict (Tuple3Fused'(..))
120+import Streamly.Internal.Data.Parser (Parser(..), Initial(..), ParseError(..))
121+import Streamly.Internal.Data.Stream (Stream(..))
122+import Streamly.Internal.Data.StreamK (StreamK)
123+import Streamly.Internal.Data.SVar.Type (adaptState, defState)
124+import Streamly.Internal.Data.Tuple.Strict (Tuple'(..), Tuple3Fused'(..))
127125import Streamly.Internal.Data.Unfold.Type (Unfold(..))
128126import Streamly.Internal.System.IO (unsafeInlineIO)
129127
130−import qualified Streamly.Internal.Data.MutArray.Type as MA
128+import qualified Streamly.Internal.Data.Fold.Type as Fold
129+import qualified Streamly.Internal.Data.Serialize.Type as Serialize
130+import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
131131import qualified Streamly.Internal.Data.MutArray as MA
132−import qualified Streamly.Internal.Data.Array.Type as A
133132import qualified Streamly.Internal.Data.Fold as FL
134−import qualified Streamly.Internal.Data.Producer.Type as Producer
135−import qualified Streamly.Internal.Data.Producer as Producer
136133import qualified Streamly.Internal.Data.Ring as RB
134+import qualified Streamly.Internal.Data.Parser as Parser
135+-- import qualified Streamly.Internal.Data.ParserK as ParserK
137136import qualified Streamly.Internal.Data.Stream as D
138137import qualified Streamly.Internal.Data.Stream as Stream
138+import qualified Streamly.Internal.Data.StreamK as StreamK
139139import qualified Streamly.Internal.Data.Unfold as Unfold
140+import qualified Prelude
141+
142+import Streamly.Internal.Data.Array.Type
140143
141144#include "DocTestDataArray.hs"
142145
@@ -163,91 +166,18 @@ import qualified Streamly.Internal.Data.Unfold as Unfold
163166-- if we use one array for each chunk. This is still significant to add
164167-- pressure to GC.
165168
166−-------------------------------------------------------------------------------
167−-- Construction
168−-------------------------------------------------------------------------------
169−
170−-- | Create an 'Array' from the first N elements of a stream. The array is
171−-- allocated to size N, if the stream terminates before N elements then the
172−-- array may hold less than N elements.
173−--
174−-- /Pre-release/
175−{-# INLINE fromStreamN #-}
176−fromStreamN :: (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
177−fromStreamN n m = do
178− when (n < 0) $ error "writeN: negative write count specified"
179− A.fromStreamDN n m
180−
181−-- | Create an 'Array' from a stream. This is useful when we want to create a
182−-- single array from a stream of unknown size. 'writeN' is at least twice
183−-- as efficient when the size is already known.
184−--
185−-- Note that if the input stream is too large memory allocation for the array
186−-- may fail. When the stream size is not known, `chunksOf` followed by
187−-- processing of indvidual arrays in the resulting stream should be preferred.
188−--
189−-- /Pre-release/
190−{-# INLINE fromStream #-}
191−fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (Array a)
192−fromStream = Stream.fold A.write
193−-- write m = A.fromStreamD $ D.fromStreamK m
194−
195169-------------------------------------------------------------------------------
196170-- Elimination
197171-------------------------------------------------------------------------------
198172
199−{-# INLINE_NORMAL producer #-}
200−producer :: forall m a. (Monad m, Unbox a) => Producer m (Array a) a
201−producer =
202− Producer.translate A.unsafeThaw A.unsafeFreeze
203− $ MA.producerWith (return . unsafeInlineIO)
204−
205−-- | Unfold an array into a stream.
206−--
207−{-# INLINE_NORMAL reader #-}
208−reader :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
209−reader = Producer.simplify producer
210−
211−-- | Unfold an array into a stream, does not check the end of the array, the
212−-- user is responsible for terminating the stream within the array bounds. For
213−-- high performance application where the end condition can be determined by
214−-- a terminating fold.
215−--
216−-- Written in the hope that it may be faster than "read", however, in the case
217−-- for which this was written, "read" proves to be faster even though the core
218−-- generated with unsafeRead looks simpler.
219−--
220−-- /Pre-release/
221−--
222−{-# INLINE_NORMAL readerUnsafe #-}
223−readerUnsafe :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
224−readerUnsafe = Unfold step inject
225− where
226−
227− inject (Array contents start end) =
228− return (ArrayUnsafe contents end start)
229−
230− {-# INLINE_LATE step #-}
231− step (ArrayUnsafe contents end p) = do
232− -- unsafeInlineIO allows us to run this in Identity monad for pure
233− -- toList/foldr case which makes them much faster due to not
234− -- accumulating the list and fusing better with the pure consumers.
235− --
236− -- This should be safe as the array contents are guaranteed to be
237− -- evaluated/written to before we peek at them.
238− let !x = unsafeInlineIO $ peekByteIndex p contents
239− let !p1 = INDEX_NEXT(p,a)
240− return $ D.Yield x (ArrayUnsafe contents end p1)
241−
242173-- |
243174--
244−-- >>> import qualified Streamly.Internal.Data.Array.Type as Array
245175-- >>> null arr = Array.byteLength arr == 0
246176--
247177-- /Pre-release/
248178{-# INLINE null #-}
249179null :: Array a -> Bool
250−null arr = A.byteLength arr == 0
180+null arr = byteLength arr == 0
251181
252182-- | Like 'getIndex' but indexes the array in reverse from the end.
253183--
@@ -259,12 +189,11 @@ getIndexRev i arr =
259189 $ do
260190 let elemPtr = RINDEX_OF(arrEnd arr, i, a)
261191 if i >= 0 && elemPtr >= arrStart arr
262− then Just <$> peekByteIndex elemPtr (arrContents arr)
192+ then Just <$> peekAt elemPtr (arrContents arr)
263193 else return Nothing
264194
265195-- |
266196--
267−-- >>> import qualified Streamly.Internal.Data.Array as Array
268197-- >>> last arr = Array.getIndexRev arr 0
269198--
270199-- /Pre-release/
@@ -276,6 +205,8 @@ last = getIndexRev 0
276205-- Folds with Array as the container
277206-------------------------------------------------------------------------------
278207
208+-- XXX We should generate this from Ring.
209+
279210-- | @writeLastN n@ folds a maximum of @n@ elements from the end of the input
280211-- stream to an 'Array'.
281212--
@@ -284,7 +215,7 @@ writeLastN ::
284215 (Storable a, Unbox a, MonadIO m) => Int -> Fold m a (Array a)
285216writeLastN n
286217 | n <= 0 = fmap (const mempty) FL.drain
287− | otherwise = A.unsafeFreeze <$> Fold step initial done
218+ | otherwise = unsafeFreeze <$> Fold step initial done done
288219
289220 where
290221
@@ -297,11 +228,9 @@ writeLastN n
297228 in fmap f $ liftIO $ RB.new n
298229
299230 done (Tuple3Fused' rb rh i) = do
300− arr <- liftIO $ MA.new n
301− foldFunc i rh snoc' arr rb
302−
303− -- XXX We should write a read unfold for ring.
304− snoc' b a = liftIO $ MA.snocUnsafe b a
231+ arr <- MA.new n
232+ -- XXX We should write a read unfold for ring.
233+ foldFunc i rh MA.snocUnsafe arr rb
305234
306235 foldFunc i
307236 | i < n = RB.unsafeFoldRingM
@@ -327,19 +256,26 @@ binarySearch = undefined
327256-- compared to streams by using SIMD instructions.
328257-- We can also return a bit array instead.
329258
259+-- Can use SIMD.
260+
330261-- | Perform a linear search to find all the indices where a given element is
331262-- present in an array.
332263--
333264-- /Unimplemented/
334−findIndicesOf :: (a -> Bool) -> Unfold Identity (Array a) Int
335−findIndicesOf = undefined
265+indexFinder :: (a -> Bool) -> Unfold Identity (Array a) Int
266+indexFinder = undefined
267+
268+-- |
269+-- /Unimplemented/
270+findIndicesOf :: (a -> Bool) -> Array a -> Stream Identity Int
271+findIndicesOf p = Stream.unfold (indexFinder p)
336272
337273{-
338274findIndexOf :: (a -> Bool) -> Array a -> Maybe Int
339−findIndexOf p = Unfold.fold Fold.one . Stream.unfold (findIndicesOf p)
275+findIndexOf p = Unfold.fold Fold.one . Stream.unfold (indexFinder p)
340276
341277find :: (a -> Bool) -> Array a -> Bool
342−find = Unfold.fold Fold.null . Stream.unfold (findIndicesOf p)
278+find = Unfold.fold Fold.null . Stream.unfold (indexFinder p)
343279-}
344280
345281-------------------------------------------------------------------------------
@@ -381,29 +317,45 @@ splitOn :: (Monad m, Unbox a) =>
381317 (a -> Bool) -> Array a -> Stream m (Array a)
382318splitOn predicate arr =
383319 fmap (\(i, len) -> getSliceUnsafe i len arr)
384− $ D.sliceOnSuffix predicate (A.toStreamD arr)
320+ $ D.indexOnSuffix predicate (read arr)
321+
322+{-# INLINE sliceIndexerFromLen #-}
323+sliceIndexerFromLen :: forall m a. (Monad m, Unbox a)
324+ => Int -- ^ from index
325+ -> Int -- ^ length of the slice
326+ -> Unfold m (Array a) (Int, Int)
327+sliceIndexerFromLen from len =
328+ Unfold.lmap unsafeThaw (MA.sliceIndexerFromLen from len)
385329
330+{-# DEPRECATED genSlicesFromLen "Please use sliceIndexerFromLen instead." #-}
386331{-# INLINE genSlicesFromLen #-}
387332genSlicesFromLen :: forall m a. (Monad m, Unbox a)
388333 => Int -- ^ from index
389334 -> Int -- ^ length of the slice
390335 -> Unfold m (Array a) (Int, Int)
391−genSlicesFromLen from len =
392− Unfold.lmap A.unsafeThaw (MA.genSlicesFromLen from len)
336+genSlicesFromLen = sliceIndexerFromLen
393337
394338-- | Generate a stream of slices of specified length from an array, starting
395339-- from the supplied array index. The last slice may be shorter than the
396340-- requested length.
397341--
398342-- /Pre-release//
343+{-# INLINE slicerFromLen #-}
344+slicerFromLen :: forall m a. (Monad m, Unbox a)
345+ => Int -- ^ from index
346+ -> Int -- ^ length of the slice
347+ -> Unfold m (Array a) (Array a)
348+slicerFromLen from len =
349+ fmap unsafeFreeze
350+ $ Unfold.lmap unsafeThaw (MA.slicerFromLen from len)
351+
352+{-# DEPRECATED getSlicesFromLen "Please use slicerFromLen instead." #-}
399353{-# INLINE getSlicesFromLen #-}
400354getSlicesFromLen :: forall m a. (Monad m, Unbox a)
401355 => Int -- ^ from index
402356 -> Int -- ^ length of the slice
403357 -> Unfold m (Array a) (Array a)
404−getSlicesFromLen from len =
405− fmap A.unsafeFreeze
406− $ Unfold.lmap A.unsafeThaw (MA.getSlicesFromLen from len)
358+getSlicesFromLen = slicerFromLen
407359
408360-------------------------------------------------------------------------------
409361-- Random reads and writes
@@ -421,7 +373,7 @@ getIndex i arr =
421373 $ do
422374 let elemPtr = INDEX_OF(arrStart arr, i, a)
423375 if i >= 0 && INDEX_VALID(elemPtr, arrEnd arr, a)
424− then Just <$> peekByteIndex elemPtr (arrContents arr)
376+ then Just <$> peekAt elemPtr (arrContents arr)
425377 else return Nothing
426378
427379-- | Given a stream of array indices, read the elements on those indices from
@@ -433,19 +385,25 @@ getIndex i arr =
433385-- @
434386-- read =
435387-- let u = lmap (\arr -> (0, length arr - 1)) Unfold.enumerateFromTo
436−-- in Unfold.lmap f (getIndices arr)
388+-- in Unfold.lmap f (indexReader arr)
437389--
438390-- readRev =
439391-- let i = length arr - 1
440−-- in Unfold.lmap f (getIndicesFromThenTo i (i - 1) 0)
392+-- in Unfold.lmap f (indexReaderFromThenTo i (i - 1) 0)
441393-- @
442394--
443395-- /Pre-release/
396+{-# INLINE indexReader #-}
397+indexReader :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
398+indexReader m =
399+ let unf = MA.indexReaderWith (return . unsafeInlineIO) m
400+ in Unfold.lmap unsafeThaw unf
401+
402+-- XXX DO NOT REMOVE, change the signature to use Stream instead of unfold
403+{-# DEPRECATED getIndices "Please use getIndices instead." #-}
444404{-# INLINE getIndices #-}
445405getIndices :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
446−getIndices m =
447− let unf = MA.getIndicesD (return . unsafeInlineIO) m
448− in Unfold.lmap A.unsafeThaw unf
406+getIndices = indexReader
449407
450408-- | Unfolds @(from, then, to, array)@ generating a finite stream whose first
451409-- element is the array value from the index @from@ and the successive elements
@@ -461,9 +419,9 @@ getIndices m =
461419-- @
462420--
463421-- /Unimplemented/
464−{-# INLINE getIndicesFromThenTo #-}
465−getIndicesFromThenTo :: Unfold m (Int, Int, Int, Array a) a
466−getIndicesFromThenTo = undefined
422+{-# INLINE indexReaderFromThenTo #-}
423+indexReaderFromThenTo :: Unfold m (Int, Int, Int, Array a) a
424+indexReaderFromThenTo = undefined
467425
468426-------------------------------------------------------------------------------
469427-- Transform via stream operations
@@ -480,7 +438,7 @@ getIndicesFromThenTo = undefined
480438{-# INLINE runPipe #-}
481439runPipe :: (MonadIO m, Unbox a, Unbox b)
482440 => Pipe m a b -> Array a -> m (Array b)
483−runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (A.read arr)
441+runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (read arr)
484442-}
485443
486444-- XXX For transformations that cannot change the number of elements e.g. "map"
@@ -494,7 +452,7 @@ runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (A.read arr)
494452streamTransform :: forall m a b. (MonadIO m, Unbox a, Unbox b)
495453 => (Stream m a -> Stream m b) -> Array a -> m (Array b)
496454streamTransform f arr =
497− Stream.fold (A.writeWith (length arr)) $ f (A.read arr)
455+ Stream.fold (createWith (length arr)) $ f (read arr)
498456
499457-------------------------------------------------------------------------------
500458-- Casts
@@ -528,7 +486,7 @@ asBytes = castUnsafe
528486--
529487cast :: forall a b. (Unbox b) => Array a -> Maybe (Array b)
530488cast arr =
531− let len = A.byteLength arr
489+ let len = byteLength arr
532490 r = len `mod` SIZE_OF(b)
533491 in if r /= 0
534492 then Nothing
@@ -546,9 +504,9 @@ cast arr =
546504--
547505asCStringUnsafe :: Array a -> (CString -> IO b) -> IO b
548506asCStringUnsafe arr act = do
549− let arr1 = asBytes arr <> A.fromList [0]
550− -- asPtrUnsafe makes sure the array is pinned
551− asPtrUnsafe arr1 $ \ptr -> act (castPtr ptr)
507+ let arr1 = asBytes arr <> fromList [0]
508+ -- unsafePinnedAsPtr makes sure the array is pinned
509+ unsafePinnedAsPtr arr1 $ \ptr -> act (castPtr ptr)
552510
553511-------------------------------------------------------------------------------
554512-- Folds
@@ -561,11 +519,500 @@ asCStringUnsafe arr act = do
561519-- /Pre-release/
562520{-# INLINE fold #-}
563521fold :: forall m a b. (Monad m, Unbox a) => Fold m a b -> Array a -> m b
564−fold f arr = Stream.fold f (A.read arr)
522+fold f arr = Stream.fold f (read arr)
565523
566524-- | Fold an array using a stream fold operation.
567525--
568526-- /Pre-release/
569527{-# INLINE streamFold #-}
570528streamFold :: (Monad m, Unbox a) => (Stream m a -> m b) -> Array a -> m b
571−streamFold f arr = f (A.read arr)
529+streamFold f arr = f (read arr)
530+
531+--------------------------------------------------------------------------------
532+-- Serialization
533+--------------------------------------------------------------------------------
534+
535+{-# INLINE encodeAs #-}
536+encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
537+encodeAs ps a =
538+ unsafeInlineIO $ do
539+ let len = Serialize.addSizeTo 0 a
540+ mbarr <- MBA.newBytesAs ps len
541+ off <- Serialize.serializeAt 0 mbarr a
542+ assertM(len == off)
543+ pure $ Array mbarr 0 off
544+
545+-- |
546+-- Properties:
547+-- 1. Identity: @deserialize . serialize == id@
548+-- 2. Encoded equivalence: @serialize a == serialize a@
549+{-# INLINE serialize #-}
550+serialize :: Serialize a => a -> Array Word8
551+serialize = encodeAs Unpinned
552+
553+-- | Serialize a Haskell type to a pinned byte array. The array is allocated
554+-- using pinned memory so that it can be used directly in OS APIs for writing
555+-- to file or sending over the network.
556+--
557+-- Properties:
558+-- 1. Identity: @deserialize . pinnedSerialize == id@
559+-- 2. Encoded equivalence: @pinnedSerialize a == pinnedSerialize a@
560+{-# INLINE pinnedSerialize #-}
561+pinnedSerialize :: Serialize a => a -> Array Word8
562+pinnedSerialize = encodeAs Pinned
563+
564+-- | Decode a Haskell type from a byte array containing its serialized
565+-- representation.
566+{-# INLINE deserialize #-}
567+deserialize :: Serialize a => Array Word8 -> a
568+deserialize arr@(Array {..}) = unsafeInlineIO $ do
569+ let lenArr = length arr
570+ (off, val) <-
571+ Serialize.deserializeAt arrStart arrContents (arrStart + lenArr)
572+ assertM(off == arrStart + lenArr)
573+ pure val
574+
575+-------------------------------------------------------------------------------
576+-- Streams of Arrays
577+-------------------------------------------------------------------------------
578+
579+-- TODO: efficiently compare two streams of arrays. Two streams can have chunks
580+-- of different sizes, we can handle that in the stream comparison abstraction.
581+-- This could be useful e.g. to fast compare whether two files differ.
582+
583+-- | Insert the given element between arrays and flatten.
584+--
585+-- >>> interpose x = Stream.interpose x Array.reader
586+--
587+{-# INLINE interpose #-}
588+interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
589+interpose x = D.interpose x reader
590+
591+data FlattenState s =
592+ OuterLoop s
593+ | InnerLoop s !MutByteArray !Int !Int
594+
595+-- | Insert the given element after each array and flatten. This is similar to
596+-- unlines.
597+--
598+-- >>> interposeSuffix x = Stream.interposeSuffix x Array.reader
599+--
600+{-# INLINE_NORMAL interposeSuffix #-}
601+interposeSuffix :: forall m a. (Monad m, Unbox a)
602+ => a -> Stream m (Array a) -> Stream m a
603+-- This does not require MonadIO constraint.
604+-- interposeSuffix x = D.interposeSuffix x reader
605+interposeSuffix sep (D.Stream step state) = D.Stream step' (OuterLoop state)
606+
607+ where
608+
609+ {-# INLINE_LATE step' #-}
610+ step' gst (OuterLoop st) = do
611+ r <- step (adaptState gst) st
612+ return $ case r of
613+ D.Yield Array{..} s ->
614+ D.Skip (InnerLoop s arrContents arrStart arrEnd)
615+ D.Skip s -> D.Skip (OuterLoop s)
616+ D.Stop -> D.Stop
617+
618+ step' _ (InnerLoop st _ p end) | p == end =
619+ return $ D.Yield sep $ OuterLoop st
620+
621+ step' _ (InnerLoop st contents p end) = do
622+ x <- pure $ unsafeInlineIO $ peekAt p contents
623+ return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
624+
625+-- | Insert the given array after each array and flatten.
626+--
627+-- >>> intercalateSuffix = Stream.intercalateSuffix Array.reader
628+--
629+{-# INLINE intercalateSuffix #-}
630+intercalateSuffix :: (Monad m, Unbox a)
631+ => Array a -> Stream m (Array a) -> Stream m a
632+intercalateSuffix = D.intercalateSuffix reader
633+
634+-- | @compactLE n@ coalesces adjacent arrays in the input stream
635+-- only if the combined size would be less than or equal to n.
636+--
637+-- Generates unpinned arrays irrespective of the pinning status of input
638+-- arrays.
639+{-# INLINE_NORMAL compactLE #-}
640+compactLE :: (MonadIO m, Unbox a)
641+ => Int -> Stream m (Array a) -> Stream m (Array a)
642+compactLE n stream =
643+ D.map unsafeFreeze $ MA.compactLE n $ D.map unsafeThaw stream
644+
645+-- | Pinned version of 'compactLE'.
646+{-# INLINE_NORMAL pinnedCompactLE #-}
647+pinnedCompactLE :: (MonadIO m, Unbox a)
648+ => Int -> Stream m (Array a) -> Stream m (Array a)
649+pinnedCompactLE n stream =
650+ D.map unsafeFreeze $ MA.pinnedCompactLE n $ D.map unsafeThaw stream
651+
652+-- | Split a stream of arrays on a given separator byte, dropping the separator
653+-- and coalescing all the arrays between two separators into a single array.
654+--
655+{-# INLINE compactOnByte #-}
656+compactOnByte
657+ :: (MonadIO m)
658+ => Word8
659+ -> Stream m (Array Word8)
660+ -> Stream m (Array Word8)
661+compactOnByte byte =
662+ fmap unsafeFreeze . MA.compactOnByte byte . fmap unsafeThaw
663+
664+-- | Like 'compactOnByte' considers the separator in suffix position instead of
665+-- infix position.
666+{-# INLINE compactOnByteSuffix #-}
667+compactOnByteSuffix
668+ :: (MonadIO m)
669+ => Word8
670+ -> Stream m (Array Word8)
671+ -> Stream m (Array Word8)
672+compactOnByteSuffix byte =
673+ fmap unsafeFreeze . MA.compactOnByteSuffix byte . fmap unsafeThaw
674+
675+-------------------------------------------------------------------------------
676+-- Folding Streams of Arrays
677+-------------------------------------------------------------------------------
678+
679+-- XXX This should not be used for breaking a stream as the D.cons used in
680+-- reconstructing the stream could be very bad for performance. This can only
681+-- be useful in folding without breaking.
682+{-# INLINE_NORMAL foldBreakChunks #-}
683+foldBreakChunks :: forall m a b. (MonadIO m, Unbox a) =>
684+ Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
685+foldBreakChunks (Fold fstep initial _ final) stream@(Stream step state) = do
686+ res <- initial
687+ case res of
688+ Fold.Partial fs -> go SPEC state fs
689+ Fold.Done fb -> return $! (fb, stream)
690+
691+ where
692+
693+ {-# INLINE go #-}
694+ go !_ st !fs = do
695+ r <- step defState st
696+ case r of
697+ Stream.Yield (Array contents start end) s ->
698+ let fp = Tuple' end contents
699+ in goArray SPEC s fp start fs
700+ Stream.Skip s -> go SPEC s fs
701+ Stream.Stop -> do
702+ b <- final fs
703+ return (b, D.nil)
704+
705+ goArray !_ s (Tuple' end _) !cur !fs
706+ | cur == end = do
707+ go SPEC s fs
708+ goArray !_ st fp@(Tuple' end contents) !cur !fs = do
709+ x <- liftIO $ peekAt cur contents
710+ res <- fstep fs x
711+ let next = INDEX_NEXT(cur,a)
712+ case res of
713+ Fold.Done b -> do
714+ let arr = Array contents next end
715+ return $! (b, D.cons arr (D.Stream step st))
716+ Fold.Partial fs1 -> goArray SPEC st fp next fs1
717+
718+-- This may be more robust wrt fusion compared to unfoldMany?
719+
720+-- | Fold a stream of arrays using a 'Fold'. This is equivalent to the
721+-- following:
722+--
723+-- >>> foldChunks f = Stream.fold f . Stream.unfoldMany Array.reader
724+--
725+foldChunks :: (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m b
726+foldChunks f s = fmap fst (foldBreakChunks f s)
727+-- foldStream f = Stream.fold f . Stream.unfoldMany reader
728+
729+-- | Fold a stream of arrays using a 'Fold' and return the remaining stream.
730+--
731+-- The following alternative to this function allows composing the fold using
732+-- the parser Monad:
733+--
734+-- @
735+-- foldBreakStreamK f s =
736+-- fmap (first (fromRight undefined))
737+-- $ StreamK.parseBreakChunks (ParserK.adaptC (Parser.fromFold f)) s
738+-- @
739+--
740+-- We can compare perf and remove this one or define it in terms of that.
741+--
742+foldBreakChunksK :: forall m a b. (MonadIO m, Unbox a) =>
743+ Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
744+{-
745+foldBreakChunksK f s =
746+ fmap (first (fromRight undefined))
747+ $ StreamK.parseBreakChunks (ParserK.adaptC (Parser.fromFold f)) s
748+-}
749+foldBreakChunksK (Fold fstep initial _ final) stream = do
750+ res <- initial
751+ case res of
752+ Fold.Partial fs -> go fs stream
753+ Fold.Done fb -> return (fb, stream)
754+
755+ where
756+
757+ {-# INLINE go #-}
758+ go !fs st = do
759+ let stop = (, StreamK.nil) <$> final fs
760+ single a = yieldk a StreamK.nil
761+ yieldk (Array contents start end) r =
762+ let fp = Tuple' end contents
763+ in goArray fs r fp start
764+ in StreamK.foldStream defState yieldk single stop st
765+
766+ goArray !fs st (Tuple' end _) !cur
767+ | cur == end = do
768+ go fs st
769+ goArray !fs st fp@(Tuple' end contents) !cur = do
770+ x <- liftIO $ peekAt cur contents
771+ res <- fstep fs x
772+ let next = INDEX_NEXT(cur,a)
773+ case res of
774+ Fold.Done b -> do
775+ let arr = Array contents next end
776+ return $! (b, StreamK.cons arr st)
777+ Fold.Partial fs1 -> goArray fs1 st fp next
778+
779+{-
780+-- This can be generalized to any type provided it can be unfolded to a stream
781+-- and it can be combined using a semigroup operation.
782+--
783+{-# INLINE_NORMAL parseBreakD #-}
784+parseBreakD ::
785+ forall m a b. (MonadIO m, MonadThrow m, Unbox a)
786+ => PRD.Parser a m b
787+ -> D.Stream m (Array.Array a)
788+ -> m (b, D.Stream m (Array.Array a))
789+parseBreakD
790+ (PRD.Parser pstep initial extract) stream@(D.Stream step state) = do
791+
792+ res <- initial
793+ case res of
794+ PRD.IPartial s -> go SPEC state (List []) s
795+ PRD.IDone b -> return (b, stream)
796+ PRD.IError err -> throwM $ ParseError err
797+
798+ where
799+
800+ -- "backBuf" contains last few items in the stream that we may have to
801+ -- backtrack to.
802+ --
803+ -- XXX currently we are using a dumb list based approach for backtracking
804+ -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
805+ -- That will allow us more efficient random back and forth movement.
806+ go !_ st backBuf !pst = do
807+ r <- step defState st
808+ case r of
809+ D.Yield (Array contents start end) s ->
810+ gobuf SPEC s backBuf
811+ (Tuple' end contents) start pst
812+ D.Skip s -> go SPEC s backBuf pst
813+ D.Stop -> do
814+ b <- extract pst
815+ return (b, D.nil)
816+
817+ -- Use strictness on "cur" to keep it unboxed
818+ gobuf !_ s backBuf (Tuple' end _) !cur !pst
819+ | cur == end = do
820+ go SPEC s backBuf pst
821+ gobuf !_ s backBuf fp@(Tuple' end contents) !cur !pst = do
822+ x <- liftIO $ peekByteIndex contents cur
823+ pRes <- pstep pst x
824+ let next = INDEX_NEXT(cur,a)
825+ case pRes of
826+ PR.Partial 0 pst1 ->
827+ gobuf SPEC s (List []) fp next pst1
828+ PR.Partial n pst1 -> do
829+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
830+ let src0 = Prelude.take n (x:getList backBuf)
831+ arr0 = A.fromListN n (Prelude.reverse src0)
832+ arr1 = Array contents next end
833+ src = arr0 <> arr1
834+ let !(Array cont1 start end1) = src
835+ fp1 = Tuple' end1 cont1
836+ gobuf SPEC s (List []) fp1 start pst1
837+ PR.Continue 0 pst1 ->
838+ gobuf SPEC s (List (x:getList backBuf)) fp next pst1
839+ PR.Continue n pst1 -> do
840+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
841+ let (src0, buf1) = splitAt n (x:getList backBuf)
842+ arr0 = A.fromListN n (Prelude.reverse src0)
843+ arr1 = Array contents next end
844+ src = arr0 <> arr1
845+ let !(Array cont1 start end1) = src
846+ fp1 = Tuple' end1 cont1
847+ gobuf SPEC s (List buf1) fp1 start pst1
848+ PR.Done 0 b -> do
849+ let arr = Array contents next end
850+ return (b, D.cons arr (D.Stream step s))
851+ PR.Done n b -> do
852+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
853+ let src0 = Prelude.take n (x:getList backBuf)
854+ -- XXX create the array in reverse instead
855+ arr0 = A.fromListN n (Prelude.reverse src0)
856+ arr1 = Array contents next end
857+ -- XXX Use StreamK to avoid adding arbitrary layers of
858+ -- constructors every time.
859+ str = D.cons arr0 (D.cons arr1 (D.Stream step s))
860+ return (b, str)
861+ PR.Error err -> throwM $ ParseError err
862+-}
863+
864+-- | Parse an array stream using the supplied 'Parser'. Returns the parse
865+-- result and the unconsumed stream. Throws 'ParseError' if the parse fails.
866+--
867+-- The following alternative to this function allows composing the parser using
868+-- the parser Monad:
869+--
870+-- >>> parseBreakStreamK p = StreamK.parseBreakChunks (ParserK.adaptC p)
871+--
872+-- We can compare perf and remove this one or define it in terms of that.
873+--
874+-- /Internal/
875+--
876+{-# INLINE_NORMAL parseBreakChunksK #-}
877+parseBreakChunksK ::
878+ forall m a b. (MonadIO m, Unbox a)
879+ => Parser a m b
880+ -> StreamK m (Array a)
881+ -> m (Either ParseError b, StreamK m (Array a))
882+-- parseBreakStreamK p = StreamK.parseBreakChunks (ParserK.adaptC p)
883+parseBreakChunksK (Parser pstep initial extract) stream = do
884+ res <- initial
885+ case res of
886+ IPartial s -> go s stream []
887+ IDone b -> return (Right b, stream)
888+ IError err -> return (Left (ParseError err), stream)
889+
890+ where
891+
892+ -- "backBuf" contains last few items in the stream that we may have to
893+ -- backtrack to.
894+ --
895+ -- XXX currently we are using a dumb list based approach for backtracking
896+ -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
897+ -- That will allow us more efficient random back and forth movement.
898+ go !pst st backBuf = do
899+ let stop = goStop pst backBuf -- (, K.nil) <$> extract pst
900+ single a = yieldk a StreamK.nil
901+ yieldk arr r = goArray pst backBuf r arr
902+ in StreamK.foldStream defState yieldk single stop st
903+
904+ -- Use strictness on "cur" to keep it unboxed
905+ goArray !pst backBuf st (Array _ cur end) | cur == end = go pst st backBuf
906+ goArray !pst backBuf st (Array contents cur end) = do
907+ x <- liftIO $ peekAt cur contents
908+ pRes <- pstep pst x
909+ let next = INDEX_NEXT(cur,a)
910+ case pRes of
911+ Parser.Partial 0 s ->
912+ goArray s [] st (Array contents next end)
913+ Parser.Partial n s -> do
914+ assert (n <= Prelude.length (x:backBuf)) (return ())
915+ let src0 = Prelude.take n (x:backBuf)
916+ arr0 = fromListN n (Prelude.reverse src0)
917+ arr1 = Array contents next end
918+ src = arr0 <> arr1
919+ goArray s [] st src
920+ Parser.Continue 0 s ->
921+ goArray s (x:backBuf) st (Array contents next end)
922+ Parser.Continue n s -> do
923+ assert (n <= Prelude.length (x:backBuf)) (return ())
924+ let (src0, buf1) = Prelude.splitAt n (x:backBuf)
925+ arr0 = fromListN n (Prelude.reverse src0)
926+ arr1 = Array contents next end
927+ src = arr0 <> arr1
928+ goArray s buf1 st src
929+ Parser.Done 0 b -> do
930+ let arr = Array contents next end
931+ return (Right b, StreamK.cons arr st)
932+ Parser.Done n b -> do
933+ assert (n <= Prelude.length (x:backBuf)) (return ())
934+ let src0 = Prelude.take n (x:backBuf)
935+ -- XXX Use fromListRevN once implemented
936+ -- arr0 = A.fromListRevN n src0
937+ arr0 = fromListN n (Prelude.reverse src0)
938+ arr1 = Array contents next end
939+ str = StreamK.cons arr0 (StreamK.cons arr1 st)
940+ return (Right b, str)
941+ Parser.Error err -> do
942+ let n = Prelude.length backBuf
943+ arr0 = fromListN n (Prelude.reverse backBuf)
944+ arr1 = Array contents cur end
945+ str = StreamK.cons arr0 (StreamK.cons arr1 stream)
946+ return (Left (ParseError err), str)
947+
948+ -- This is a simplified goArray
949+ goExtract !pst backBuf (Array _ cur end)
950+ | cur == end = goStop pst backBuf
951+ goExtract !pst backBuf (Array contents cur end) = do
952+ x <- liftIO $ peekAt cur contents
953+ pRes <- pstep pst x
954+ let next = INDEX_NEXT(cur,a)
955+ case pRes of
956+ Parser.Partial 0 s ->
957+ goExtract s [] (Array contents next end)
958+ Parser.Partial n s -> do
959+ assert (n <= Prelude.length (x:backBuf)) (return ())
960+ let src0 = Prelude.take n (x:backBuf)
961+ arr0 = fromListN n (Prelude.reverse src0)
962+ arr1 = Array contents next end
963+ src = arr0 <> arr1
964+ goExtract s [] src
965+ Parser.Continue 0 s ->
966+ goExtract s backBuf (Array contents next end)
967+ Parser.Continue n s -> do
968+ assert (n <= Prelude.length (x:backBuf)) (return ())
969+ let (src0, buf1) = Prelude.splitAt n (x:backBuf)
970+ arr0 = fromListN n (Prelude.reverse src0)
971+ arr1 = Array contents next end
972+ src = arr0 <> arr1
973+ goExtract s buf1 src
974+ Parser.Done 0 b -> do
975+ let arr = Array contents next end
976+ return (Right b, StreamK.fromPure arr)
977+ Parser.Done n b -> do
978+ assert (n <= Prelude.length backBuf) (return ())
979+ let src0 = Prelude.take n backBuf
980+ -- XXX Use fromListRevN once implemented
981+ -- arr0 = A.fromListRevN n src0
982+ arr0 = fromListN n (Prelude.reverse src0)
983+ arr1 = Array contents next end
984+ str = StreamK.cons arr0 (StreamK.fromPure arr1)
985+ return (Right b, str)
986+ Parser.Error err -> do
987+ let n = Prelude.length backBuf
988+ arr0 = fromListN n (Prelude.reverse backBuf)
989+ arr1 = Array contents cur end
990+ str = StreamK.cons arr0 (StreamK.cons arr1 stream)
991+ return (Left (ParseError err), str)
992+
993+ -- This is a simplified goExtract
994+ {-# INLINE goStop #-}
995+ goStop !pst backBuf = do
996+ pRes <- extract pst
997+ case pRes of
998+ Parser.Partial _ _ -> error "Bug: parseBreak: Partial in extract"
999+ Parser.Continue 0 s ->
1000+ goStop s backBuf
1001+ Parser.Continue n s -> do
1002+ assert (n <= Prelude.length backBuf) (return ())
1003+ let (src0, buf1) = Prelude.splitAt n backBuf
1004+ arr = fromListN n (Prelude.reverse src0)
1005+ goExtract s buf1 arr
1006+ Parser.Done 0 b ->
1007+ return (Right b, StreamK.nil)
1008+ Parser.Done n b -> do
1009+ assert (n <= Prelude.length backBuf) (return ())
1010+ let src0 = Prelude.take n backBuf
1011+ -- XXX Use fromListRevN once implemented
1012+ -- arr0 = A.fromListRevN n src0
1013+ arr0 = fromListN n (Prelude.reverse src0)
1014+ return (Right b, StreamK.fromPure arr0)
1015+ Parser.Error err -> do
1016+ let n = Prelude.length backBuf
1017+ arr0 = fromListN n (Prelude.reverse backBuf)
1018+ return (Left (ParseError err), StreamK.fromPure arr0)
core/src/Streamly/Internal/Data/Array/Generic.hs+48−10
@@ -11,8 +11,8 @@ module Streamly.Internal.Data.Array.Generic
1111
1212 -- * Construction
1313 , nil
14− , writeN
15− , write
14+ , createOf
15+ , create
1616 , writeWith
1717 , writeLastN
1818
@@ -24,6 +24,8 @@ module Streamly.Internal.Data.Array.Generic
2424 , fromListN
2525 , fromList
2626
27+ , chunksOf
28+
2729 -- * Elimination
2830 , length
2931 , reader
@@ -42,6 +44,10 @@ module Streamly.Internal.Data.Array.Generic
4244 , getIndex
4345 , getSliceUnsafe
4446 , strip
47+
48+ -- * Deprecated
49+ , writeN
50+ , write
4551 )
4652where
4753
@@ -57,7 +63,7 @@ import GHC.IO (unsafePerformIO)
5763import Text.Read (readPrec)
5864
5965import Streamly.Internal.Data.Fold.Type (Fold(..))
60−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
66+import Streamly.Internal.Data.Stream.Type (Stream)
6167import Streamly.Internal.Data.Unfold.Type (Unfold(..))
6268import Streamly.Internal.System.IO (unsafeInlineIO)
6369
@@ -66,8 +72,8 @@ import qualified Streamly.Internal.Data.Fold.Type as FL
6672import qualified Streamly.Internal.Data.Producer.Type as Producer
6773import qualified Streamly.Internal.Data.Producer as Producer
6874import qualified Streamly.Internal.Data.Ring.Generic as RB
69−import qualified Streamly.Internal.Data.Stream.StreamD.Type as D
70−import qualified Streamly.Internal.Data.Stream.StreamD.Generate as D
75+import qualified Streamly.Internal.Data.Stream.Type as D
76+import qualified Streamly.Internal.Data.Stream.Generate as D
7177import qualified Text.ParserCombinators.ReadPrec as ReadPrec
7278
7379import Prelude hiding (Foldable(..), read)
@@ -102,21 +108,33 @@ nil = unsafePerformIO $ unsafeFreeze <$> MArray.nil
102108-- Construction - Folds
103109-------------------------------------------------------------------------------
104110
105−{-# INLINE_NORMAL writeN #-}
111+{-# INLINE_NORMAL createOf #-}
112+createOf :: MonadIO m => Int -> Fold m a (Array a)
113+createOf = fmap unsafeFreeze <$> MArray.createOf
114+
115+-- XXX Deprecate in major
116+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
117+{-# INLINE writeN #-}
106118writeN :: MonadIO m => Int -> Fold m a (Array a)
107−writeN = fmap unsafeFreeze <$> MArray.writeN
119+writeN = createOf
108120
109121{-# INLINE_NORMAL writeWith #-}
110122writeWith :: MonadIO m => Int -> Fold m a (Array a)
111−writeWith elemCount = unsafeFreeze <$> MArray.writeWith elemCount
123+writeWith elemCount = unsafeFreeze <$> MArray.createWith elemCount
112124
113125-- | Fold the whole input to a single array.
114126--
115127-- /Caution! Do not use this on infinite streams./
116128--
129+{-# INLINE create #-}
130+create :: MonadIO m => Fold m a (Array a)
131+create = fmap unsafeFreeze MArray.create
132+
133+-- XXX Deprecate in major
134+-- {-# DEPRECATED write "Please use create instead." #-}
117135{-# INLINE write #-}
118136write :: MonadIO m => Fold m a (Array a)
119−write = fmap unsafeFreeze MArray.write
137+write = create
120138
121139fromPureStream :: Stream Identity a -> Array a
122140fromPureStream x =
@@ -127,6 +145,15 @@ fromPureStream x =
127145fromByteStr# :: Addr# -> Array Word8
128146fromByteStr# addr = fromPureStream (D.fromByteStr# addr)
129147
148+-------------------------------------------------------------------------------
149+-- Stream Ops
150+-------------------------------------------------------------------------------
151+
152+{-# INLINE_NORMAL chunksOf #-}
153+chunksOf :: forall m a. MonadIO m
154+ => Int -> Stream m a -> Stream m (Array a)
155+chunksOf n strm = fmap unsafeFreeze $ MArray.chunksOf n strm
156+
130157-------------------------------------------------------------------------------
131158-- Construction - from streams
132159-------------------------------------------------------------------------------
@@ -231,6 +258,17 @@ getIndex i arr@Array {..} =
231258 then Just $ getIndexUnsafe i arr
232259 else Nothing
233260
261+-- >>> import qualified Streamly.Data.Stream as Stream
262+-- >>> import qualified Streamly.Data.Fold as Fold
263+-- >>> import qualified Streamly.Internal.Data.Array.Generic as Array
264+-- >>> import Data.Function ((&))
265+-- >>> :{
266+-- Stream.fromList [1,2,3,4,5::Int]
267+-- & Stream.scan (Array.writeLastN 2)
268+-- & Stream.fold Fold.toList
269+-- :}
270+-- [fromList [],fromList [1],fromList [1,2],fromList [2,3],fromList [3,4],fromList [4,5]]
271+--
234272{-# INLINE writeLastN #-}
235273writeLastN :: MonadIO m => Int -> Fold m a (Array a)
236274writeLastN n = FL.rmapM f (RB.writeLastN n)
@@ -238,7 +276,7 @@ writeLastN n = FL.rmapM f (RB.writeLastN n)
238276 where
239277
240278 f rb = do
241− arr <- RB.toMutArray 0 n rb
279+ arr <- RB.copyToMutArray 0 n rb
242280 return $ unsafeFreeze arr
243281
244282{-# INLINE getSliceUnsafe #-}
core/src/Streamly/Internal/Data/Stream/Chunked.hs → core/src/Streamly/Internal/Data/Array/Stream.hsrenamed+74−587
@@ -1,5 +1,6 @@
1+{-# OPTIONS_GHC -Wno-deprecations #-}
12-- |
2−-- Module : Streamly.Internal.Data.Stream.Chunked
3+-- Module : Streamly.Internal.Data.Array.Stream
34-- Copyright : (c) 2019 Composewell Technologies
45-- License : BSD3-3-Clause
56-- Maintainer : streamly@composewell.com
@@ -7,17 +8,25 @@
78--
89-- Combinators to efficiently manipulate streams of immutable arrays.
910--
10−module Streamly.Internal.Data.Stream.Chunked
11+-- We can either push these in the MutArray module with a "chunks" prefix or
12+-- keep this as a separate module and release it.
13+--
14+module Streamly.Internal.Data.Array.Stream
15+{-# DEPRECATED "Please use \"Streamly.Internal.Data.Array\" instead." #-}
1116 (
1217 -- * Creation
13− chunksOf
18+ Array.chunksOf
19+ , Array.pinnedChunksOf
20+ , Array.bufferChunks
1421
1522 -- * Flattening to elements
16− , concat
17− , concatRev
18− , interpose
19− , interposeSuffix
20− , intercalateSuffix
23+ , Array.concat
24+ , Array.flattenArrays
25+ , Array.concatRev
26+ , Array.flattenArraysRev
27+ , Array.interpose
28+ , Array.interposeSuffix
29+ , Array.intercalateSuffix
2130 , unlines
2231
2332 -- * Elimination
@@ -26,24 +35,27 @@ module Streamly.Internal.Data.Stream.Chunked
2635 -- can flatten the stream to byte stream and use that. But if we want the
2736 -- remaining stream to be a chunk stream then this could be handy. But it
2837 -- could also be implemented using parseBreak.
29− , foldBreak -- StreamK.foldBreakChunks
38+ , foldBreak
3039 , foldBreakD
31− -- The byte level parseBreak cannot work efficiently. Because the stream
32− -- will have to be a StreamK for backtracking, StreamK at byte level would
33− -- not be efficient.
34− , parseBreak -- StreamK.parseBreakChunks
40+ -- This is chunked parseBreak. A byte level parseBreak cannot work
41+ -- efficiently. Because the stream will have to be a StreamK for
42+ -- backtracking, StreamK at byte level would not be efficient.
43+ -- parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
44+ , parseBreak
3545 -- , parseBreakD
3646 -- , foldManyChunks
3747 -- , parseManyChunks
48+ , K.parseBreakChunks
49+ , K.parseChunks
3850
3951 -- ** Array Folds
40− -- XXX Use Parser.Chunked instead, need only chunkedParseBreak,
52+ -- XXX Use parseBreakChunks/parseChunks instead
4153 -- foldBreak can be implemented using parseBreak. Use StreamK.
4254 , runArrayFold
4355 , runArrayFoldBreak
4456 -- , parseArr
45− , runArrayParserDBreak -- StreamK.chunkedParseBreak
46− , runArrayFoldMany -- StreamK.chunkedParseMany
57+ , runArrayParserDBreak -- StreamK.parseBreakChunks
58+ , runArrayFoldMany
4759
4860 , toArray
4961
@@ -67,6 +79,8 @@ where
6779import Data.Bifunctor (second)
6880import Control.Exception (assert)
6981import Control.Monad.IO.Class (MonadIO(..))
82+-- import Data.Bifunctor (first)
83+-- import Data.Either (fromRight)
7084import Data.Proxy (Proxy(..))
7185import Data.Word (Word8)
7286import Streamly.Internal.Data.Unbox (Unbox(..))
@@ -82,143 +96,32 @@ import Streamly.Internal.Data.Parser (ParseError(..))
8296import Streamly.Internal.Data.Stream (Stream)
8397import Streamly.Internal.Data.StreamK (StreamK, fromStream, toStream)
8498import Streamly.Internal.Data.SVar.Type (adaptState, defState)
85−import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
8699
87−import qualified Streamly.Data.Fold as FL
88100import qualified Streamly.Internal.Data.Array as A
89101import qualified Streamly.Internal.Data.Array as Array
90−import qualified Streamly.Internal.Data.Array.Type as A
91−import qualified Streamly.Internal.Data.MutArray.Type as MA
92−import qualified Streamly.Internal.Data.Stream.MutChunked as AS
93−import qualified Streamly.Internal.Data.Fold.Type as FL (Fold(..), Step(..))
94102import qualified Streamly.Internal.Data.Parser as PR
95103import qualified Streamly.Internal.Data.Parser as PRD
96104 (Parser(..), Initial(..))
105+-- import qualified Streamly.Internal.Data.ParserK as ParserK
97106import qualified Streamly.Internal.Data.Stream as D
98107import qualified Streamly.Internal.Data.StreamK as K
99108
100109-- XXX Since these are immutable arrays MonadIO constraint can be removed from
101110-- most places.
102111
103−-------------------------------------------------------------------------------
104−-- Generation
105−-------------------------------------------------------------------------------
106−
107−-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
108−-- @n@ elements each.
109−--
110−-- > chunksOf n = Stream.groupsOf n (Array.writeN n)
111−--
112−-- /Pre-release/
113−{-# INLINE chunksOf #-}
114−chunksOf :: (MonadIO m, Unbox a)
115− => Int -> Stream m a -> Stream m (Array a)
116−chunksOf = A.chunksOf
117−
118−-------------------------------------------------------------------------------
119−-- Append
120−-------------------------------------------------------------------------------
121−
122−-- XXX efficiently compare two streams of arrays. Two streams can have chunks
123−-- of different sizes, we can handle that in the stream comparison abstraction.
124−-- This could be useful e.g. to fast compare whether two files differ.
125−
126−-- | Convert a stream of arrays into a stream of their elements.
127−--
128−-- Same as the following:
129−--
130−-- > concat = Stream.unfoldMany Array.read
131−--
132−-- @since 0.7.0
133−{-# INLINE concat #-}
134−concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
135−-- concat m = fromStreamD $ A.flattenArrays (toStreamD m)
136−-- concat m = fromStreamD $ D.concatMap A.toStreamD (toStreamD m)
137−concat = D.unfoldMany A.reader
138−
139−-- | Convert a stream of arrays into a stream of their elements reversing the
140−-- contents of each array before flattening.
141−--
142−-- > concatRev = Stream.unfoldMany Array.readerRev
143−--
144−-- @since 0.7.0
145−{-# INLINE concatRev #-}
146−concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
147−-- concatRev m = fromStreamD $ A.flattenArraysRev (toStreamD m)
148−concatRev = D.unfoldMany A.readerRev
149−
150112-------------------------------------------------------------------------------
151113-- Intersperse and append
152114-------------------------------------------------------------------------------
153115
154−-- | Flatten a stream of arrays after inserting the given element between
155−-- arrays.
156−--
157−-- /Pre-release/
158−{-# INLINE interpose #-}
159−interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
160−interpose x = D.interpose x A.reader
161−
162−{-# INLINE intercalateSuffix #-}
163−intercalateSuffix :: (Monad m, Unbox a)
164− => Array a -> Stream m (Array a) -> Stream m a
165−intercalateSuffix = D.intercalateSuffix A.reader
166−
167−-- | Flatten a stream of arrays appending the given element after each
168−-- array.
169−--
170−-- @since 0.7.0
171−{-# INLINE interposeSuffix #-}
172−interposeSuffix :: (Monad m, Unbox a)
173− => a -> Stream m (Array a) -> Stream m a
174−-- interposeSuffix x = fromStreamD . A.unlines x . toStreamD
175−interposeSuffix x = D.interposeSuffix x A.reader
176−
177−data FlattenState s =
178− OuterLoop s
179− | InnerLoop s !MA.MutableByteArray !Int !Int
180−
181−-- XXX This is a special case of interposeSuffix, can be removed.
182−-- XXX Remove monadIO constraint
183116{-# INLINE_NORMAL unlines #-}
184117unlines :: forall m a. (MonadIO m, Unbox a)
185118 => a -> D.Stream m (Array a) -> D.Stream m a
186−unlines sep (D.Stream step state) = D.Stream step' (OuterLoop state)
187− where
188− {-# INLINE_LATE step' #-}
189− step' gst (OuterLoop st) = do
190− r <- step (adaptState gst) st
191− return $ case r of
192− D.Yield Array{..} s ->
193− D.Skip (InnerLoop s arrContents arrStart arrEnd)
194− D.Skip s -> D.Skip (OuterLoop s)
195− D.Stop -> D.Stop
196−
197− step' _ (InnerLoop st _ p end) | p == end =
198− return $ D.Yield sep $ OuterLoop st
199−
200− step' _ (InnerLoop st contents p end) = do
201− x <- liftIO $ peekByteIndex p contents
202− return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
119+unlines = Array.interposeSuffix
203120
204121-------------------------------------------------------------------------------
205122-- Compact
206123-------------------------------------------------------------------------------
207124
208−-- XXX These would not be needed once we implement compactLEFold, see
209−-- module Streamly.Internal.Data.Stream.MutChunked
210−--
211−-- XXX Note that this thaws immutable arrays for appending, that may be
212−-- problematic if multiple users do the same thing, however, immutable arrays
213−-- would usually have no capacity to append, therefore, a copy will be forced
214−-- anyway. Confirm this. We can forcefully trim the array capacity before thaw
215−-- to ensure this.
216−{-# INLINE_NORMAL packArraysChunksOf #-}
217−packArraysChunksOf :: (MonadIO m, Unbox a)
218− => Int -> D.Stream m (Array a) -> D.Stream m (Array a)
219−packArraysChunksOf n str =
220− D.map A.unsafeFreeze $ AS.packArraysChunksOf n $ D.map A.unsafeThaw str
221−
222125-- XXX instead of writing two different versions of this operation, we should
223126-- write it as a pipe.
224127--
@@ -226,8 +129,7 @@ packArraysChunksOf n str =
226129{-# INLINE_NORMAL lpackArraysChunksOf #-}
227130lpackArraysChunksOf :: (MonadIO m, Unbox a)
228131 => Int -> Fold m (Array a) () -> Fold m (Array a) ()
229−lpackArraysChunksOf n fld =
230− FL.lmap A.unsafeThaw $ AS.lpackArraysChunksOf n (FL.lmap A.unsafeFreeze fld)
132+lpackArraysChunksOf = Array.lCompactGE
231133
232134-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
233135-- maximum specified size in bytes.
@@ -236,68 +138,19 @@ lpackArraysChunksOf n fld =
236138{-# INLINE compact #-}
237139compact :: (MonadIO m, Unbox a)
238140 => Int -> Stream m (Array a) -> Stream m (Array a)
239−compact = packArraysChunksOf
141+compact = Array.compactLE
240142
241−-------------------------------------------------------------------------------
242−-- Split
243−-------------------------------------------------------------------------------
244−
245−data SplitState s arr
246− = Initial s
247− | Buffering s arr
248− | Splitting s arr
249− | Yielding arr (SplitState s arr)
250− | Finishing
251−
252−-- | Split a stream of arrays on a given separator byte, dropping the separator
253−-- and coalescing all the arrays between two separators into a single array.
143+-- | Given a stream of arrays, splice them all together to generate a single
144+-- array. The stream must be /finite/.
254145--
255146-- @since 0.7.0
256−{-# INLINE_NORMAL _splitOn #-}
257−_splitOn
258− :: MonadIO m
259− => Word8
260− -> D.Stream m (Array Word8)
261− -> D.Stream m (Array Word8)
262−_splitOn byte (D.Stream step state) = D.Stream step' (Initial state)
263−
264− where
265−
266− {-# INLINE_LATE step' #-}
267− step' gst (Initial st) = do
268− r <- step gst st
269− case r of
270− D.Yield arr s -> do
271− (arr1, marr2) <- A.breakOn byte arr
272− return $ case marr2 of
273− Nothing -> D.Skip (Buffering s arr1)
274− Just arr2 -> D.Skip (Yielding arr1 (Splitting s arr2))
275− D.Skip s -> return $ D.Skip (Initial s)
276− D.Stop -> return D.Stop
277−
278− step' gst (Buffering st buf) = do
279− r <- step gst st
280− case r of
281− D.Yield arr s -> do
282− (arr1, marr2) <- A.breakOn byte arr
283− buf' <- A.splice buf arr1
284− return $ case marr2 of
285− Nothing -> D.Skip (Buffering s buf')
286− Just x -> D.Skip (Yielding buf' (Splitting s x))
287− D.Skip s -> return $ D.Skip (Buffering s buf)
288− D.Stop -> return $
289− if A.byteLength buf == 0
290− then D.Stop
291− else D.Skip (Yielding buf Finishing)
292−
293− step' _ (Splitting st buf) = do
294− (arr1, marr2) <- A.breakOn byte buf
295− return $ case marr2 of
296− Nothing -> D.Skip $ Buffering st arr1
297− Just arr2 -> D.Skip $ Yielding arr1 (Splitting st arr2)
147+{-# INLINE toArray #-}
148+toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
149+toArray = Array.fromChunks
298150
299− step' _ (Yielding arr next) = return $ D.Yield arr next
300− step' _ Finishing = return D.Stop
151+-------------------------------------------------------------------------------
152+-- Split
153+-------------------------------------------------------------------------------
301154
302155-- XXX Remove MonadIO constraint.
303156-- | Split a stream of arrays on a given separator byte, dropping the separator
@@ -310,7 +163,7 @@ splitOn
310163 => Word8
311164 -> Stream m (Array Word8)
312165 -> Stream m (Array Word8)
313−splitOn byte = D.splitInnerBy (A.breakOn byte) A.splice
166+splitOn = Array.compactOnByte
314167
315168{-# INLINE splitOnSuffix #-}
316169splitOnSuffix
@@ -318,8 +171,7 @@ splitOnSuffix
318171 => Word8
319172 -> Stream m (Array Word8)
320173 -> Stream m (Array Word8)
321−-- splitOn byte s = fromStreamD $ A.splitOn byte $ toStreamD s
322−splitOnSuffix byte = D.splitInnerBySuffix (A.breakOn byte) A.splice
174+splitOnSuffix = Array.compactOnByteSuffix
323175
324176-------------------------------------------------------------------------------
325177-- Elimination - Running folds
@@ -328,77 +180,24 @@ splitOnSuffix byte = D.splitInnerBySuffix (A.breakOn byte) A.splice
328180{-# INLINE_NORMAL foldBreakD #-}
329181foldBreakD :: forall m a b. (MonadIO m, Unbox a) =>
330182 Fold m a b -> D.Stream m (Array a) -> m (b, D.Stream m (Array a))
331−foldBreakD (FL.Fold fstep initial extract) stream@(D.Stream step state) = do
332− res <- initial
333− case res of
334− FL.Partial fs -> go SPEC state fs
335− FL.Done fb -> return $! (fb, stream)
336−
337− where
338−
339− {-# INLINE go #-}
340− go !_ st !fs = do
341− r <- step defState st
342− case r of
343− D.Yield (Array contents start end) s ->
344− let fp = Tuple' end contents
345− in goArray SPEC s fp start fs
346− D.Skip s -> go SPEC s fs
347− D.Stop -> do
348− b <- extract fs
349− return (b, D.nil)
350−
351− goArray !_ s (Tuple' end _) !cur !fs
352− | cur == end = do
353− go SPEC s fs
354− goArray !_ st fp@(Tuple' end contents) !cur !fs = do
355− x <- liftIO $ peekByteIndex cur contents
356− res <- fstep fs x
357− let next = INDEX_NEXT(cur,a)
358− case res of
359− FL.Done b -> do
360− let arr = Array contents next end
361− return $! (b, D.cons arr (D.Stream step st))
362− FL.Partial fs1 -> goArray SPEC st fp next fs1
363−
364−{-# INLINE_NORMAL foldBreakK #-}
365−foldBreakK :: forall m a b. (MonadIO m, Unbox a) =>
366− Fold m a b -> K.StreamK m (Array a) -> m (b, K.StreamK m (Array a))
367−foldBreakK (FL.Fold fstep initial extract) stream = do
368− res <- initial
369− case res of
370− FL.Partial fs -> go fs stream
371− FL.Done fb -> return (fb, stream)
372−
373− where
374−
375− {-# INLINE go #-}
376− go !fs st = do
377− let stop = (, K.nil) <$> extract fs
378− single a = yieldk a K.nil
379− yieldk (Array contents start end) r =
380− let fp = Tuple' end contents
381− in goArray fs r fp start
382− in K.foldStream defState yieldk single stop st
383−
384− goArray !fs st (Tuple' end _) !cur
385− | cur == end = do
386− go fs st
387− goArray !fs st fp@(Tuple' end contents) !cur = do
388− x <- liftIO $ peekByteIndex cur contents
389− res <- fstep fs x
390− let next = INDEX_NEXT(cur,a)
391− case res of
392− FL.Done b -> do
393− let arr = Array contents next end
394− return $! (b, K.cons arr st)
395− FL.Partial fs1 -> goArray fs1 st fp next
183+foldBreakD = Array.foldBreakChunks
396184
397185-- | Fold an array stream using the supplied 'Fold'. Returns the fold result
398186-- and the unconsumed stream.
399187--
400188-- > foldBreak f = runArrayFoldBreak (ChunkFold.fromFold f)
401189--
190+-- Instead of using this we can adapt the fold to ParserK and use
191+-- parseBreakChunks instead. ParserK allows composing using Monad as well.
192+--
193+-- @
194+-- foldBreak f s =
195+-- fmap (first (fromRight undefined))
196+-- $ K.parseBreakChunks (ParserK.adaptC (PR.fromFold f)) s
197+-- @
198+--
199+-- We can compare perf and remove this one or define it in terms of that.
200+--
402201-- /Internal/
403202--
404203{-# INLINE_NORMAL foldBreak #-}
@@ -407,14 +206,20 @@ foldBreak ::
407206 => Fold m a b
408207 -> StreamK m (A.Array a)
409208 -> m (b, StreamK m (A.Array a))
209+foldBreak = Array.foldBreakChunksK
210+--
410211-- foldBreak f s = fmap fromStreamD <$> foldBreakD f (toStreamD s)
411−foldBreak = foldBreakK
212+--
213+-- foldBreak f s =
214+-- fmap (first (fromRight undefined))
215+-- $ K.parseBreakChunks (ParserK.adaptC (PR.fromFold f)) s
216+--
412217-- If foldBreak performs better than runArrayFoldBreak we can use a rewrite
413218-- rule to rewrite runArrayFoldBreak to fold.
414219-- foldBreak f = runArrayFoldBreak (ChunkFold.fromFold f)
415220
416221-------------------------------------------------------------------------------
417−-- Fold to a single Array
222+-- Elimination - running element parsers
418223-------------------------------------------------------------------------------
419224
420225-- When we have to take an array partially, take the last part of the array.
@@ -460,341 +265,22 @@ splitAtArrayListRev n ls
460265 arr1 = Array contents end1 end
461266 in ([arr1], arr2:xs)
462267
463−-------------------------------------------------------------------------------
464−-- Fold to a single Array
465−-------------------------------------------------------------------------------
466−
467−{-
468−
469−-- XXX Both of these implementations of splicing seem to perform equally well.
470−-- We need to perform benchmarks over a range of sizes though.
471−
472−-- CAUTION! length must more than equal to lengths of all the arrays in the
473−-- stream.
474−{-# INLINE spliceArraysLenUnsafe #-}
475−spliceArraysLenUnsafe :: (MonadIO m, Unbox a)
476− => Int -> Stream m (MutArray a) -> m (MutArray a)
477−spliceArraysLenUnsafe len buffered = do
478− -- XXX The new array's pinned state should depend on the first element
479− -- of the stream. We should uncons the stream and do the required.
480− arr <- undefined
481− D.foldlM' MA.spliceUnsafe (return arr) buffered
482−
483−{-# INLINE _spliceArrays #-}
484−_spliceArrays :: (MonadIO m, Unbox a)
485− => Stream m (Array a) -> m (Array a)
486−_spliceArrays s = do
487− buffered <- D.foldr K.cons K.nil s
488− len <- K.fold FL.sum (fmap Array.length buffered)
489− -- XXX The new array's pinned state should depend on the first element
490− -- of the stream. We should uncons the stream and do the required.
491− arr <- undefined
492− final <- D.foldlM' writeArr (return arr) (toStream buffered)
493− return $ A.unsafeFreeze final
494−
495− where
496−
497− writeArr dst arr = MA.spliceUnsafe dst (A.unsafeThaw arr)
498−
499−{-# INLINE _spliceArraysBuffered #-}
500−_spliceArraysBuffered :: (MonadIO m, Unbox a)
501− => Stream m (Array a) -> m (Array a)
502−_spliceArraysBuffered s = do
503− buffered <- D.foldr K.cons K.nil s
504− len <- K.fold FL.sum (fmap Array.length buffered)
505− A.unsafeFreeze <$>
506− spliceArraysLenUnsafe len (fmap A.unsafeThaw (toStream buffered))
507−
508−-}
509−
510−{-# INLINE spliceArraysRealloced #-}
511−spliceArraysRealloced :: forall m a. (MonadIO m, Unbox a)
512− => Stream m (Array a) -> m (Array a)
513−spliceArraysRealloced s = do
514− res <- D.uncons s
515− case res of
516− Just (a, strm) -> do
517− arr <-
518− D.foldlM'
519− MA.spliceExp
520− (pure (A.unsafeThaw a))
521− (fmap A.unsafeThaw strm)
522− liftIO $ A.unsafeFreeze <$> MA.rightSize arr
523− Nothing -> pure A.nil
524−
525−-- XXX This should just be "fold A.write"
526−--
527−-- | Given a stream of arrays, splice them all together to generate a single
528−-- array. The stream must be /finite/.
529−--
530−-- @since 0.7.0
531−{-# INLINE toArray #-}
532−toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
533−toArray = spliceArraysRealloced
534−-- spliceArrays = _spliceArraysBuffered
535−
536−-- exponentially increasing sizes of the chunks upto the max limit.
537−-- XXX this will be easier to implement with parsers/terminating folds
538−-- With this we should be able to reduce the number of chunks/allocations.
539−-- The reallocation/copy based toArray can also be implemented using this.
540−--
541−{-
542−{-# INLINE toArraysInRange #-}
543−toArraysInRange :: (MonadIO m, Unbox a)
544− => Int -> Int -> Fold m (Array a) b -> Fold m a b
545−toArraysInRange low high (Fold step initial extract) =
546−-}
547−
548−{-
549−-- | Fold the input to a pure buffered stream (List) of arrays.
550−{-# INLINE _toArraysOf #-}
551−_toArraysOf :: (MonadIO m, Unbox a)
552− => Int -> Fold m a (Stream Identity (Array a))
553−_toArraysOf n = FL.groupsOf n (A.writeNF n) FL.toStream
554−-}
555−
556−-------------------------------------------------------------------------------
557−-- Elimination - running element parsers
558−-------------------------------------------------------------------------------
559−
560268-- GHC parser does not accept {-# ANN type [] NoSpecConstr #-}, so we need
561269-- to make a newtype.
562270{-# ANN type List NoSpecConstr #-}
563271newtype List a = List {getList :: [a]}
564272
565−{-
566−-- This can be generalized to any type provided it can be unfolded to a stream
567−-- and it can be combined using a semigroup operation.
568−--
569−-- XXX This should be written using CPS (as parseK) if we want it to scale wrt
570−-- to the number of times it can be called on the same stream.
571−{-# INLINE_NORMAL parseBreakD #-}
572−parseBreakD ::
573− forall m a b. (MonadIO m, MonadThrow m, Unbox a)
574− => PRD.Parser a m b
575− -> D.Stream m (Array.Array a)
576− -> m (b, D.Stream m (Array.Array a))
577−parseBreakD
578− (PRD.Parser pstep initial extract) stream@(D.Stream step state) = do
579−
580− res <- initial
581− case res of
582− PRD.IPartial s -> go SPEC state (List []) s
583− PRD.IDone b -> return (b, stream)
584− PRD.IError err -> throwM $ ParseError err
585−
586− where
587−
588− -- "backBuf" contains last few items in the stream that we may have to
589− -- backtrack to.
590− --
591− -- XXX currently we are using a dumb list based approach for backtracking
592− -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
593− -- That will allow us more efficient random back and forth movement.
594− go !_ st backBuf !pst = do
595− r <- step defState st
596− case r of
597− D.Yield (Array contents start end) s ->
598− gobuf SPEC s backBuf
599− (Tuple' end contents) start pst
600− D.Skip s -> go SPEC s backBuf pst
601− D.Stop -> do
602− b <- extract pst
603− return (b, D.nil)
604−
605− -- Use strictness on "cur" to keep it unboxed
606− gobuf !_ s backBuf (Tuple' end _) !cur !pst
607− | cur == end = do
608− go SPEC s backBuf pst
609− gobuf !_ s backBuf fp@(Tuple' end contents) !cur !pst = do
610− x <- liftIO $ peekByteIndex contents cur
611− pRes <- pstep pst x
612− let next = INDEX_NEXT(cur,a)
613− case pRes of
614− PR.Partial 0 pst1 ->
615− gobuf SPEC s (List []) fp next pst1
616− PR.Partial n pst1 -> do
617− assert (n <= Prelude.length (x:getList backBuf)) (return ())
618− let src0 = Prelude.take n (x:getList backBuf)
619− arr0 = A.fromListN n (Prelude.reverse src0)
620− arr1 = Array contents next end
621− src = arr0 <> arr1
622− let !(Array cont1 start end1) = src
623− fp1 = Tuple' end1 cont1
624− gobuf SPEC s (List []) fp1 start pst1
625− PR.Continue 0 pst1 ->
626− gobuf SPEC s (List (x:getList backBuf)) fp next pst1
627− PR.Continue n pst1 -> do
628− assert (n <= Prelude.length (x:getList backBuf)) (return ())
629− let (src0, buf1) = splitAt n (x:getList backBuf)
630− arr0 = A.fromListN n (Prelude.reverse src0)
631− arr1 = Array contents next end
632− src = arr0 <> arr1
633− let !(Array cont1 start end1) = src
634− fp1 = Tuple' end1 cont1
635− gobuf SPEC s (List buf1) fp1 start pst1
636− PR.Done 0 b -> do
637− let arr = Array contents next end
638− return (b, D.cons arr (D.Stream step s))
639− PR.Done n b -> do
640− assert (n <= Prelude.length (x:getList backBuf)) (return ())
641− let src0 = Prelude.take n (x:getList backBuf)
642− -- XXX create the array in reverse instead
643− arr0 = A.fromListN n (Prelude.reverse src0)
644− arr1 = Array contents next end
645− -- XXX Use StreamK to avoid adding arbitrary layers of
646− -- constructors every time.
647− str = D.cons arr0 (D.cons arr1 (D.Stream step s))
648− return (b, str)
649− PR.Error err -> throwM $ ParseError err
650−-}
651−
652−{-# INLINE_NORMAL parseBreakK #-}
653−parseBreakK ::
654− forall m a b. (MonadIO m, Unbox a)
655− => PRD.Parser a m b
656− -> K.StreamK m (Array.Array a)
657− -> m (Either ParseError b, K.StreamK m (Array.Array a))
658−parseBreakK (PRD.Parser pstep initial extract) stream = do
659− res <- initial
660− case res of
661− PRD.IPartial s -> go s stream []
662− PRD.IDone b -> return (Right b, stream)
663− PRD.IError err -> return (Left (ParseError err), stream)
664−
665− where
666−
667− -- "backBuf" contains last few items in the stream that we may have to
668− -- backtrack to.
669− --
670− -- XXX currently we are using a dumb list based approach for backtracking
671− -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
672− -- That will allow us more efficient random back and forth movement.
673− go !pst st backBuf = do
674− let stop = goStop pst backBuf -- (, K.nil) <$> extract pst
675− single a = yieldk a K.nil
676− yieldk arr r = goArray pst backBuf r arr
677− in K.foldStream defState yieldk single stop st
678−
679− -- Use strictness on "cur" to keep it unboxed
680− goArray !pst backBuf st (Array _ cur end) | cur == end = go pst st backBuf
681− goArray !pst backBuf st (Array contents cur end) = do
682− x <- liftIO $ peekByteIndex cur contents
683− pRes <- pstep pst x
684− let next = INDEX_NEXT(cur,a)
685− case pRes of
686− PR.Partial 0 s ->
687− goArray s [] st (Array contents next end)
688− PR.Partial n s -> do
689− assert (n <= Prelude.length (x:backBuf)) (return ())
690− let src0 = Prelude.take n (x:backBuf)
691− arr0 = A.fromListN n (Prelude.reverse src0)
692− arr1 = Array contents next end
693− src = arr0 <> arr1
694− goArray s [] st src
695− PR.Continue 0 s ->
696− goArray s (x:backBuf) st (Array contents next end)
697− PR.Continue n s -> do
698− assert (n <= Prelude.length (x:backBuf)) (return ())
699− let (src0, buf1) = splitAt n (x:backBuf)
700− arr0 = A.fromListN n (Prelude.reverse src0)
701− arr1 = Array contents next end
702− src = arr0 <> arr1
703− goArray s buf1 st src
704− PR.Done 0 b -> do
705− let arr = Array contents next end
706− return (Right b, K.cons arr st)
707− PR.Done n b -> do
708− assert (n <= Prelude.length (x:backBuf)) (return ())
709− let src0 = Prelude.take n (x:backBuf)
710− -- XXX Use fromListRevN once implemented
711− -- arr0 = A.fromListRevN n src0
712− arr0 = A.fromListN n (Prelude.reverse src0)
713− arr1 = Array contents next end
714− str = K.cons arr0 (K.cons arr1 st)
715− return (Right b, str)
716− PR.Error err -> do
717− let n = Prelude.length backBuf
718− arr0 = A.fromListN n (Prelude.reverse backBuf)
719− arr1 = Array contents cur end
720− str = K.cons arr0 (K.cons arr1 stream)
721− return (Left (ParseError err), str)
722−
723− -- This is a simplified goArray
724− goExtract !pst backBuf (Array _ cur end)
725− | cur == end = goStop pst backBuf
726− goExtract !pst backBuf (Array contents cur end) = do
727− x <- liftIO $ peekByteIndex cur contents
728− pRes <- pstep pst x
729− let next = INDEX_NEXT(cur,a)
730− case pRes of
731− PR.Partial 0 s ->
732− goExtract s [] (Array contents next end)
733− PR.Partial n s -> do
734− assert (n <= Prelude.length (x:backBuf)) (return ())
735− let src0 = Prelude.take n (x:backBuf)
736− arr0 = A.fromListN n (Prelude.reverse src0)
737− arr1 = Array contents next end
738− src = arr0 <> arr1
739− goExtract s [] src
740− PR.Continue 0 s ->
741− goExtract s backBuf (Array contents next end)
742− PR.Continue n s -> do
743− assert (n <= Prelude.length (x:backBuf)) (return ())
744− let (src0, buf1) = splitAt n (x:backBuf)
745− arr0 = A.fromListN n (Prelude.reverse src0)
746− arr1 = Array contents next end
747− src = arr0 <> arr1
748− goExtract s buf1 src
749− PR.Done 0 b -> do
750− let arr = Array contents next end
751− return (Right b, K.fromPure arr)
752− PR.Done n b -> do
753− assert (n <= Prelude.length backBuf) (return ())
754− let src0 = Prelude.take n backBuf
755− -- XXX Use fromListRevN once implemented
756− -- arr0 = A.fromListRevN n src0
757− arr0 = A.fromListN n (Prelude.reverse src0)
758− arr1 = Array contents next end
759− str = K.cons arr0 (K.fromPure arr1)
760− return (Right b, str)
761− PR.Error err -> do
762− let n = Prelude.length backBuf
763− arr0 = A.fromListN n (Prelude.reverse backBuf)
764− arr1 = Array contents cur end
765− str = K.cons arr0 (K.cons arr1 stream)
766− return (Left (ParseError err), str)
767−
768− -- This is a simplified goExtract
769− {-# INLINE goStop #-}
770− goStop !pst backBuf = do
771− pRes <- extract pst
772− case pRes of
773− PR.Partial _ _ -> error "Bug: parseBreak: Partial in extract"
774− PR.Continue 0 s ->
775− goStop s backBuf
776− PR.Continue n s -> do
777− assert (n <= Prelude.length backBuf) (return ())
778− let (src0, buf1) = splitAt n backBuf
779− arr = A.fromListN n (Prelude.reverse src0)
780− goExtract s buf1 arr
781− PR.Done 0 b ->
782− return (Right b, K.nil)
783− PR.Done n b -> do
784− assert (n <= Prelude.length backBuf) (return ())
785− let src0 = Prelude.take n backBuf
786− -- XXX Use fromListRevN once implemented
787− -- arr0 = A.fromListRevN n src0
788− arr0 = A.fromListN n (Prelude.reverse src0)
789− return (Right b, K.fromPure arr0)
790− PR.Error err -> do
791− let n = Prelude.length backBuf
792− arr0 = A.fromListN n (Prelude.reverse backBuf)
793− return (Left (ParseError err), K.fromPure arr0)
794−
795273-- | Parse an array stream using the supplied 'Parser'. Returns the parse
796274-- result and the unconsumed stream. Throws 'ParseError' if the parse fails.
797275--
276+-- >> parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
277+--
278+-- This is redundant and we can just use parseBreakChunks, as ParserK can be
279+-- composed using Monad. The only advantage of this is that we do not need to
280+-- adapt.
281+--
282+-- We can compare perf and remove this one or define it in terms of that.
283+--
798284-- /Internal/
799285--
800286{-# INLINE_NORMAL parseBreak #-}
@@ -807,7 +293,8 @@ parseBreak ::
807293parseBreak p s =
808294 fmap fromStreamD <$> parseBreakD (PRD.fromParserK p) (toStreamD s)
809295-}
810−parseBreak = parseBreakK
296+parseBreak = Array.parseBreakChunksK
297+-- parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
811298
812299-------------------------------------------------------------------------------
813300-- Elimination - Running Array Folds and parsers
core/src/Streamly/Internal/Data/Array/Type.hs+538−108
@@ -1,8 +1,11 @@
11{-# LANGUAGE CPP #-}
2+{-# LANGUAGE TypeFamilies #-}
3+-- Must come after TypeFamilies, otherwise it is re-enabled.
4+-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
5+{-# LANGUAGE NoMonoLocalBinds #-}
26-- |
37-- Module : Streamly.Internal.Data.Array.Type
48-- Copyright : (c) 2020 Composewell Technologies
5−--
69-- License : BSD3-3-Clause
710-- Maintainer : streamly@composewell.com
811-- Stability : experimental
@@ -10,79 +13,146 @@
1013--
1114-- See notes in "Streamly.Internal.Data.MutArray.Type"
1215--
16+
1317module Streamly.Internal.Data.Array.Type
1418 (
19+ -- ** Type
1520 -- $arrayNotes
1621 Array (..)
17− , asPtrUnsafe
18− , nil
1922
20− -- * Freezing and Thawing
23+ -- ** Conversion
24+ -- *** Mutable and Immutable
2125 , unsafeFreeze
2226 , unsafeFreezeWithShrink
2327 , unsafeThaw
2428
25− -- * Pinning and Unpinning
29+ -- *** Pinned and Unpinned
2630 , pin
2731 , unpin
2832 , isPinned
2933
30− -- * Construction
31− , splice
34+ -- *** Casting
35+ , unsafePinnedAsPtr
3236
33− , fromList
34− , pinnedFromList
37+ -- ** Construction
38+ , empty
39+
40+ -- *** Cloning
41+ -- XXX Why would we clone an immutable array?
42+ , clone
43+ , pinnedClone
44+
45+ -- *** Slicing
46+ -- | Get a subarray without copying
47+ , splitAt
48+ , breakOn -- XXX requires MonadIO
49+
50+ -- *** Stream Folds
51+ , unsafeMakePure
52+ , createOf
53+ , pinnedCreateOf
54+ , unsafeCreateOf
55+ , unsafePinnedCreateOf
56+ , create
57+ , pinnedCreate
58+ , createWith
59+
60+ -- *** From containers
3561 , fromListN
3662 , pinnedFromListN
37− , fromListRev
63+ , fromList
64+ , pinnedFromList
3865 , fromListRevN
39− , fromStreamDN
40− , fromStreamD
66+ , fromListRev
67+ , fromStreamN
68+ , fromStream
69+ , fromPureStreamN
4170 , fromPureStream
4271 , fromByteStr#
72+ , fromByteStr
73+ , fromPtrN
74+ , fromChunks
75+ , fromChunksK
4376
44− -- * Split
45− , breakOn
77+ -- ** Reading
4678
47− -- * Elimination
48− , unsafeIndexIO
49− , unsafeIndex -- getIndexUnsafe
50− , byteLength
79+ -- *** Indexing
80+ , unsafeIndexIO -- XXX unsafeGetIndexIO
81+ , getIndexUnsafe -- XXX unsafeGetIndex
82+
83+ -- *** To Streams
84+ , read
85+ , readRev
86+ , toStreamK
87+ , toStreamKRev
88+
89+ -- *** To Containers
90+ , toList
91+
92+ -- *** Unfolds
93+ , producer -- experimental
94+ , readerUnsafe
95+ , reader
96+ , readerRev
97+
98+ -- *** Size
5199 , length
100+ , byteLength
52101
102+ -- ** Folding
53103 , foldl'
54104 , foldr
55− , splitAt
105+ , byteCmp
106+ , byteEq
107+
108+ -- ** Appending
109+ , splice -- XXX requires MonadIO
110+
111+ -- ** Streams of arrays
112+ -- *** Chunk
113+ -- | Group a stream into arrays.
114+ , chunksOf
115+ , pinnedChunksOf
116+ , buildChunks
117+
118+ -- *** Split
119+ -- | Split an array into slices.
120+
121+ -- *** Concat
122+ -- | Append the arrays in a stream to form a stream of elements.
123+ , concat
124+ , concatRev
125+
126+ -- *** Compact
127+ -- | Append the arrays in a stream to form a stream of larger arrays.
128+ , fCompactGE
129+ , fPinnedCompactGE
130+ , lCompactGE
131+ , lPinnedCompactGE
132+ , compactGE
56133
134+ -- ** Deprecated
135+ , asPtrUnsafe
136+ , unsafeIndex
137+ , bufferChunks
138+ , flattenArrays
139+ , flattenArraysRev
140+ , fromArrayStreamK
141+ , fromStreamDN
142+ , fromStreamD
57143 , toStreamD
58144 , toStreamDRev
59− , toStreamK
60− , toStreamKRev
61145 , toStream
62146 , toStreamRev
63− , read
64− , readRev
65− , readerRev
66− , toList
67−
68− -- * Folds
147+ , nil
69148 , writeWith
70149 , writeN
71150 , pinnedWriteN
72151 , writeNUnsafe
73152 , pinnedWriteNUnsafe
74− , MA.ArrayUnsafe (..)
75153 , pinnedWriteNAligned
76154 , write
77155 , pinnedWrite
78− , unsafeMakePure
79−
80− -- * Streams of arrays
81− , chunksOf
82− , pinnedChunksOf
83− , bufferChunks
84− , flattenArrays
85− , flattenArraysRev
86156 )
87157where
88158
@@ -90,31 +160,36 @@ where
90160#include "inline.hs"
91161
92162import Control.Exception (assert)
93−import Control.Monad (replicateM)
163+import Control.Monad (replicateM, when)
94164import Control.Monad.IO.Class (MonadIO(..))
95165import Data.Functor.Identity (Identity(..))
166+import Data.Int (Int8, Int16, Int32, Int64)
96167import Data.Proxy (Proxy(..))
97−import Data.Word (Word8)
168+import Data.Word (Word8, Word16, Word32, Word64)
98169import GHC.Base (build)
99170import GHC.Exts (IsList, IsString(..), Addr#)
100171
101172import GHC.IO (unsafePerformIO)
102173import GHC.Ptr (Ptr(..))
103−import Streamly.Internal.Data.MutArray.Type (MutArray(..), MutableByteArray)
174+import Streamly.Internal.Data.Producer.Type (Producer(..))
175+import Streamly.Internal.Data.MutArray.Type (MutArray(..))
176+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
104177import Streamly.Internal.Data.Fold.Type (Fold(..))
105−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
178+import Streamly.Internal.Data.Stream.Type (Stream)
179+import Streamly.Internal.Data.StreamK.Type (StreamK)
106180import Streamly.Internal.Data.Unbox (Unbox(..))
107181import Streamly.Internal.Data.Unfold.Type (Unfold(..))
108182import Text.Read (readPrec)
109183
110−import Prelude hiding (Foldable(..), read, unlines, splitAt)
184+import Prelude hiding (Foldable(..), concat, read, unlines, splitAt)
111185
112186import qualified GHC.Exts as Exts
187+import qualified Streamly.Internal.Data.Fold.Type as Fold
113188import qualified Streamly.Internal.Data.MutArray.Type as MA
114−import qualified Streamly.Internal.Data.Stream.StreamD.Type as D
115−import qualified Streamly.Internal.Data.Stream.StreamD.Generate as D
116−import qualified Streamly.Internal.Data.Stream.StreamK.Type as K
117−import qualified Streamly.Internal.Data.Unbox as Unboxed
189+import qualified Streamly.Internal.Data.Stream.Type as D
190+import qualified Streamly.Internal.Data.StreamK.Type as K
191+import qualified Streamly.Internal.Data.MutByteArray.Type as Unboxed
192+import qualified Streamly.Internal.Data.Producer as Producer
118193import qualified Streamly.Internal.Data.Unfold.Type as Unfold
119194import qualified Text.ParserCombinators.ReadPrec as ReadPrec
120195
@@ -157,7 +232,7 @@ data Array a =
157232#endif
158233 -- All offsets are in terms of bytes from the start of arraycontents
159234 Array
160− { arrContents :: {-# UNPACK #-} !MutableByteArray
235+ { arrContents :: {-# UNPACK #-} !MutByteArray
161236 , arrStart :: {-# UNPACK #-} !Int -- offset
162237 , arrEnd :: {-# UNPACK #-} !Int -- offset + len
163238 }
@@ -168,14 +243,20 @@ data Array a =
168243
169244-- | Use an @Array a@ as @Ptr a@.
170245--
171−-- See 'MA.asPtrUnsafe' in the Mutable array module for more details.
246+-- See 'MA.unsafePinnedAsPtr' in the Mutable array module for more details.
172247--
173248-- /Unsafe/
174249--
175250-- /Pre-release/
176251--
252+{-# INLINE unsafePinnedAsPtr #-}
253+unsafePinnedAsPtr :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
254+unsafePinnedAsPtr arr = MA.unsafePinnedAsPtr (unsafeThaw arr)
255+
256+{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
257+{-# INLINE asPtrUnsafe #-}
177258asPtrUnsafe :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
178−asPtrUnsafe arr = MA.asPtrUnsafe (unsafeThaw arr)
259+asPtrUnsafe = unsafePinnedAsPtr
179260
180261-------------------------------------------------------------------------------
181262-- Freezing and Thawing
@@ -222,18 +303,19 @@ unsafeThaw (Array ac as ae) = MutArray ac as ae ae
222303-- Pinning & Unpinning
223304-------------------------------------------------------------------------------
224305
225−-- | Copy the 'Array' to pinned memory if unpinned, else do nothing. The
226−-- overhead is a copy if the input array is unpinned.
306+-- | Return a copy of the 'Array' in pinned memory if unpinned, else return the
307+-- original array.
227308{-# INLINE pin #-}
228309pin :: Array a -> IO (Array a)
229310pin = fmap unsafeFreeze . MA.pin . unsafeThaw
230311
231−-- | Copy the 'Array' to unpinned memory if pinned, else do nothing. The
232−-- overhead is a copy if the input array is pinned.
312+-- | Return a copy of the 'Array' in unpinned memory if pinned, else return the
313+-- original array.
233314{-# INLINE unpin #-}
234315unpin :: Array a -> IO (Array a)
235316unpin = fmap unsafeFreeze . MA.unpin . unsafeThaw
236317
318+-- | Return 'True' if the array is allocated in pinned memory.
237319{-# INLINE isPinned #-}
238320isPinned :: Array a -> Bool
239321isPinned = MA.isPinned . unsafeThaw
@@ -242,11 +324,14 @@ isPinned = MA.isPinned . unsafeThaw
242324-- Construction
243325-------------------------------------------------------------------------------
244326
245−-- Splice two immutable arrays creating a new array.
327+-- | Copy two immutable arrays into a new array. If you want to splice more
328+-- than two arrays then this operation would be highly inefficient because it
329+-- would make a copy on every splice operation, instead use the
330+-- 'fromChunksK' operation to combine n immutable arrays.
246331{-# INLINE splice #-}
247−splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
332+splice :: MonadIO m => Array a -> Array a -> m (Array a)
248333splice arr1 arr2 =
249− unsafeFreeze <$> MA.splice (unsafeThaw arr1) (unsafeThaw arr2)
334+ unsafeFreeze <$> MA.spliceCopy (unsafeThaw arr1) (unsafeThaw arr2)
250335
251336-- | Create an 'Array' from the first N elements of a list. The array is
252337-- allocated to size N, if the list terminates before N elements then the
@@ -290,24 +375,83 @@ pinnedFromList xs = unsafePerformIO $ unsafeFreeze <$> MA.pinnedFromList xs
290375fromListRev :: Unbox a => [a] -> Array a
291376fromListRev xs = unsafePerformIO $ unsafeFreeze <$> MA.fromListRev xs
292377
293−{-# INLINE_NORMAL fromStreamDN #-}
378+-- | Create an 'Array' from the first N elements of a stream. The array is
379+-- allocated to size N, if the stream terminates before N elements then the
380+-- array may hold less than N elements.
381+--
382+-- >>> fromStreamN n = Stream.fold (Array.writeN n)
383+--
384+-- /Pre-release/
385+{-# INLINE_NORMAL fromStreamN #-}
386+fromStreamN :: (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
387+fromStreamN n m = do
388+ when (n < 0) $ error "writeN: negative write count specified"
389+ unsafeFreeze <$> MA.fromStreamN n m
390+-- fromStreamN n = D.fold (writeN n)
391+
392+{-# DEPRECATED fromStreamDN "Please use fromStreamN instead." #-}
294393fromStreamDN :: forall m a. (MonadIO m, Unbox a)
295394 => Int -> D.Stream m a -> m (Array a)
296−fromStreamDN limit str = unsafeFreeze <$> MA.fromStreamDN limit str
395+fromStreamDN = fromStreamN
297396
397+-- | Create an 'Array' from a stream. This is useful when we want to create a
398+-- single array from a stream of unknown size. 'writeN' is at least twice
399+-- as efficient when the size is already known.
400+--
401+-- >>> fromStream = Stream.fold Array.write
402+--
403+-- Note that if the input stream is too large memory allocation for the array
404+-- may fail. When the stream size is not known, `chunksOf` followed by
405+-- processing of indvidual arrays in the resulting stream should be preferred.
406+--
407+-- /Pre-release/
298408{-# INLINE_NORMAL fromStreamD #-}
409+fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (Array a)
410+fromStream = D.fold write
411+-- fromStreamD str = unsafeFreeze <$> MA.fromStream str
412+
413+{-# DEPRECATED fromStreamD "Please use fromStream instead." #-}
299414fromStreamD :: forall m a. (MonadIO m, Unbox a)
300415 => D.Stream m a -> m (Array a)
301−fromStreamD str = unsafeFreeze <$> MA.fromStreamD str
416+fromStreamD = fromStream
417+
418+-------------------------------------------------------------------------------
419+-- Cloning
420+-------------------------------------------------------------------------------
421+
422+{-# INLINE clone #-}
423+clone ::
424+ ( MonadIO m
425+#ifdef DEVBUILD
426+ , Unbox a
427+#endif
428+ )
429+ => Array a -> m (Array a)
430+clone = fmap unsafeFreeze . MA.clone . unsafeThaw
431+
432+{-# INLINE pinnedClone #-}
433+pinnedClone ::
434+ ( MonadIO m
435+#ifdef DEVBUILD
436+ , Unbox a
437+#endif
438+ )
439+ => Array a -> m (Array a)
440+pinnedClone = fmap unsafeFreeze . MA.pinnedClone . unsafeThaw
302441
303442-------------------------------------------------------------------------------
304443-- Streams of arrays
305444-------------------------------------------------------------------------------
306445
307−{-# INLINE bufferChunks #-}
446+{-# INLINE buildChunks #-}
447+buildChunks :: (MonadIO m, Unbox a) =>
448+ D.Stream m a -> m (K.StreamK m (Array a))
449+buildChunks m = D.foldr K.cons K.nil $ chunksOf defaultChunkSize m
450+
451+{-# DEPRECATED bufferChunks "Please use buildChunks instead." #-}
308452bufferChunks :: (MonadIO m, Unbox a) =>
309453 D.Stream m a -> m (K.StreamK m (Array a))
310−bufferChunks m = D.foldr K.cons K.nil $ chunksOf defaultChunkSize m
454+bufferChunks = buildChunks
311455
312456-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
313457-- @n@ elements each.
@@ -328,27 +472,100 @@ pinnedChunksOf :: forall m a. (MonadIO m, Unbox a)
328472 => Int -> D.Stream m a -> D.Stream m (Array a)
329473pinnedChunksOf n str = D.map unsafeFreeze $ MA.pinnedChunksOf n str
330474
331−-- | Use the "read" unfold instead.
332−--
333−-- @flattenArrays = unfoldMany read@
475+-- | Convert a stream of arrays into a stream of their elements.
334476--
335−-- We can try this if there are any fusion issues in the unfold.
477+-- >>> concat = Stream.unfoldMany Array.reader
336478--
337−{-# INLINE_NORMAL flattenArrays #-}
479+{-# INLINE_NORMAL concat #-}
480+concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
481+-- XXX this requires MonadIO whereas the unfoldMany version does not
482+concat = MA.concatWith (pure . unsafeInlineIO) . D.map unsafeThaw
483+-- concat = D.unfoldMany reader
484+
485+{-# DEPRECATED flattenArrays "Please use \"unfoldMany reader\" instead." #-}
486+{-# INLINE flattenArrays #-}
338487flattenArrays :: forall m a. (MonadIO m, Unbox a)
339488 => D.Stream m (Array a) -> D.Stream m a
340−flattenArrays = MA.flattenArrays . D.map unsafeThaw
489+flattenArrays = concat
341490
342−-- | Use the "readRev" unfold instead.
491+-- | Convert a stream of arrays into a stream of their elements reversing the
492+-- contents of each array before flattening.
343493--
344−-- @flattenArrays = unfoldMany readRev@
494+-- >>> concatRev = Stream.unfoldMany Array.readerRev
345495--
346−-- We can try this if there are any fusion issues in the unfold.
347−--
348−{-# INLINE_NORMAL flattenArraysRev #-}
496+{-# INLINE_NORMAL concatRev #-}
497+concatRev :: forall m a. (Monad m, Unbox a)
498+ => D.Stream m (Array a) -> D.Stream m a
499+-- XXX this requires MonadIO whereas the unfoldMany version does not
500+concatRev = MA.concatRevWith (pure . unsafeInlineIO) . D.map unsafeThaw
501+-- concatRev = D.unfoldMany readerRev
502+
503+{-# DEPRECATED flattenArraysRev "Please use \"unfoldMany readerRev\" instead." #-}
504+{-# INLINE flattenArraysRev #-}
349505flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
350506 => D.Stream m (Array a) -> D.Stream m a
351−flattenArraysRev = MA.flattenArraysRev . D.map unsafeThaw
507+flattenArraysRev = concatRev
508+
509+-------------------------------------------------------------------------------
510+-- Compact
511+-------------------------------------------------------------------------------
512+
513+-- XXX Note that this thaws immutable arrays for appending, that may be
514+-- problematic if multiple users do the same thing, however, thawed immutable
515+-- arrays would have no capacity to append, therefore, a copy will be forced
516+-- anyway.
517+
518+-- | Fold @fCompactGE n@ coalesces adjacent arrays in the input stream
519+-- until the size becomes greater than or equal to n.
520+--
521+-- Generates unpinned arrays irrespective of the pinning status of input
522+-- arrays.
523+{-# INLINE_NORMAL fCompactGE #-}
524+fCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
525+fCompactGE n = fmap unsafeFreeze $ Fold.lmap unsafeThaw $ MA.fCompactGE n
526+
527+-- | PInned version of 'fCompactGE'.
528+{-# INLINE_NORMAL fPinnedCompactGE #-}
529+fPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
530+fPinnedCompactGE n =
531+ fmap unsafeFreeze $ Fold.lmap unsafeThaw $ MA.fPinnedCompactGE n
532+
533+-- | @compactGE n stream@ coalesces adjacent arrays in the @stream@ until
534+-- the size becomes greater than or equal to @n@.
535+--
536+-- >>> compactGE n = Stream.foldMany (Array.fCompactGE n)
537+--
538+-- Generates unpinned arrays irrespective of the pinning status of input
539+-- arrays.
540+{-# INLINE compactGE #-}
541+compactGE ::
542+ (MonadIO m, Unbox a)
543+ => Int -> Stream m (Array a) -> Stream m (Array a)
544+compactGE n stream =
545+ D.map unsafeFreeze $ MA.compactGE n $ D.map unsafeThaw stream
546+
547+-- | Like 'compactGE' but for transforming folds instead of stream.
548+--
549+-- >>> lCompactGE n = Fold.many (Array.fCompactGE n)
550+--
551+-- Generates unpinned arrays irrespective of the pinning status of input
552+-- arrays.
553+{-# INLINE_NORMAL lCompactGE #-}
554+lCompactGE :: (MonadIO m, Unbox a)
555+ => Int -> Fold m (Array a) () -> Fold m (Array a) ()
556+lCompactGE n fld =
557+ Fold.lmap unsafeThaw $ MA.lCompactGE n (Fold.lmap unsafeFreeze fld)
558+
559+-- | Pinned version of 'lCompactGE'.
560+{-# INLINE_NORMAL lPinnedCompactGE #-}
561+lPinnedCompactGE :: (MonadIO m, Unbox a)
562+ => Int -> Fold m (Array a) () -> Fold m (Array a) ()
563+lPinnedCompactGE n fld =
564+ Fold.lmap unsafeThaw $ MA.lPinnedCompactGE n (Fold.lmap unsafeFreeze fld)
565+
566+-------------------------------------------------------------------------------
567+-- Splitting
568+-------------------------------------------------------------------------------
352569
353570-- Drops the separator byte
354571{-# INLINE breakOn #-}
@@ -370,9 +587,14 @@ unsafeIndexIO :: forall a. Unbox a => Int -> Array a -> IO a
370587unsafeIndexIO i arr = MA.getIndexUnsafe i (unsafeThaw arr)
371588
372589-- | Return element at the specified index without checking the bounds.
590+{-# INLINE_NORMAL getIndexUnsafe #-}
591+getIndexUnsafe :: forall a. Unbox a => Int -> Array a -> a
592+getIndexUnsafe i arr = let !r = unsafeInlineIO $ unsafeIndexIO i arr in r
593+
594+{-# DEPRECATED unsafeIndex "Please use 'getIndexUnsafe' instead" #-}
373595{-# INLINE_NORMAL unsafeIndex #-}
374596unsafeIndex :: forall a. Unbox a => Int -> Array a -> a
375−unsafeIndex i arr = let !r = unsafeInlineIO $ unsafeIndexIO i arr in r
597+unsafeIndex = getIndexUnsafe
376598
377599-- | /O(1)/ Get the byte length of the array.
378600--
@@ -387,24 +609,68 @@ byteLength = MA.byteLength . unsafeThaw
387609length :: Unbox a => Array a -> Int
388610length arr = MA.length (unsafeThaw arr)
389611
612+{-# INLINE_NORMAL producer #-}
613+producer :: forall m a. (Monad m, Unbox a) => Producer m (Array a) a
614+producer =
615+ Producer.translate unsafeThaw unsafeFreeze
616+ $ MA.producerWith (return . unsafeInlineIO)
617+
618+-- | Unfold an array into a stream.
619+--
620+{-# INLINE_NORMAL reader #-}
621+reader :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
622+reader = Producer.simplify producer
623+
624+-- | Unfold an array into a stream, does not check the end of the array, the
625+-- user is responsible for terminating the stream within the array bounds. For
626+-- high performance application where the end condition can be determined by
627+-- a terminating fold.
628+--
629+-- Written in the hope that it may be faster than "read", however, in the case
630+-- for which this was written, "read" proves to be faster even though the core
631+-- generated with unsafeRead looks simpler.
632+--
633+-- /Pre-release/
634+--
635+{-# INLINE_NORMAL readerUnsafe #-}
636+readerUnsafe :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
637+readerUnsafe = Unfold step inject
638+ where
639+
640+ inject (Array contents start end) =
641+ return (MA.ArrayUnsafe contents end start)
642+
643+ {-# INLINE_LATE step #-}
644+ step (MA.ArrayUnsafe contents end p) = do
645+ -- unsafeInlineIO allows us to run this in Identity monad for pure
646+ -- toList/foldr case which makes them much faster due to not
647+ -- accumulating the list and fusing better with the pure consumers.
648+ --
649+ -- This should be safe as the array contents are guaranteed to be
650+ -- evaluated/written to before we peek at them.
651+ let !x = unsafeInlineIO $ peekAt p contents
652+ let !p1 = INDEX_NEXT(p,a)
653+ return $ D.Yield x (MA.ArrayUnsafe contents end p1)
654+
390655-- | Unfold an array into a stream in reverse order.
391656--
392657{-# INLINE_NORMAL readerRev #-}
393658readerRev :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
394659readerRev = Unfold.lmap unsafeThaw $ MA.readerRevWith (return . unsafeInlineIO)
395660
661+{-# DEPRECATED toStreamD "Please use 'read' instead." #-}
396662{-# INLINE_NORMAL toStreamD #-}
397663toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> D.Stream m a
398−toStreamD arr = MA.toStreamDWith (return . unsafeInlineIO) (unsafeThaw arr)
664+toStreamD = read
399665
400666{-# INLINE toStreamK #-}
401667toStreamK :: forall m a. (Monad m, Unbox a) => Array a -> K.StreamK m a
402668toStreamK arr = MA.toStreamKWith (return . unsafeInlineIO) (unsafeThaw arr)
403669
670+{-# DEPRECATED toStreamDRev "Please use 'readRev' instead." #-}
404671{-# INLINE_NORMAL toStreamDRev #-}
405672toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> D.Stream m a
406−toStreamDRev arr =
407− MA.toStreamDRevWith (return . unsafeInlineIO) (unsafeThaw arr)
673+toStreamDRev = readRev
408674
409675{-# INLINE toStreamKRev #-}
410676toStreamKRev :: forall m a. (Monad m, Unbox a) => Array a -> K.StreamK m a
@@ -416,7 +682,7 @@ toStreamKRev arr =
416682-- /Pre-release/
417683{-# INLINE_EARLY read #-}
418684read :: (Monad m, Unbox a) => Array a -> Stream m a
419−read = toStreamD
685+read arr = MA.toStreamWith (return . unsafeInlineIO) (unsafeThaw arr)
420686
421687-- | Same as 'read'
422688--
@@ -433,7 +699,7 @@ toStream = read
433699-- /Pre-release/
434700{-# INLINE_EARLY readRev #-}
435701readRev :: (Monad m, Unbox a) => Array a -> Stream m a
436−readRev = toStreamDRev
702+readRev arr = MA.toStreamRevWith (return . unsafeInlineIO) (unsafeThaw arr)
437703
438704-- | Same as 'readRev'
439705--
@@ -457,6 +723,7 @@ foldr f z arr = runIdentity $ D.foldr f z $ toStreamD arr
457723-- | Create two slices of an array without copying the original array. The
458724-- specified index @i@ is the first index of the second slice.
459725--
726+{-# INLINE splitAt #-}
460727splitAt :: Unbox a => Int -> Array a -> (Array a, Array a)
461728splitAt i arr = (unsafeFreeze a, unsafeFreeze b)
462729 where
@@ -476,8 +743,8 @@ toListFB c n Array{..} = go arrStart
476743 -- accumulating the list and fusing better with the pure consumers.
477744 --
478745 -- This should be safe as the array contents are guaranteed to be
479− -- evaluated/written to before we peekByteIndex at them.
480− let !x = unsafeInlineIO $ peekByteIndex p arrContents
746+ -- evaluated/written to before we peekAt at them.
747+ let !x = unsafeInlineIO $ peekAt p arrContents
481748 in c x (go (INDEX_NEXT(p,a)))
482749
483750-- | Convert an 'Array' into a list.
@@ -490,17 +757,28 @@ toList s = build (\c n -> toListFB c n s)
490757-- Folds
491758-------------------------------------------------------------------------------
492759
493−-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
760+-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
494761-- 'Array'.
495762--
496−{-# INLINE_NORMAL writeN #-}
763+{-# INLINE_NORMAL createOf #-}
764+createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
765+createOf = fmap unsafeFreeze . MA.createOf
766+
767+-- XXX Deprecate in major
768+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
769+{-# INLINE writeN #-}
497770writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
498−writeN = fmap unsafeFreeze . MA.writeN
771+writeN = createOf
499772
500−-- | Like 'fromListN' but creates a pinned array.
501−{-# INLINE_NORMAL pinnedWriteN #-}
773+-- | Like 'createOf' but creates a pinned array.
774+{-# INLINE_NORMAL pinnedCreateOf #-}
775+pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
776+pinnedCreateOf = fmap unsafeFreeze . MA.pinnedCreateOf
777+
778+{-# DEPRECATED pinnedWriteN "Please use pinnedCreateOf instead." #-}
779+{-# INLINE pinnedWriteN #-}
502780pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
503−pinnedWriteN = fmap unsafeFreeze . MA.pinnedWriteN
781+pinnedWriteN = pinnedCreateOf
504782
505783-- | @pinnedWriteNAligned alignment n@ folds a maximum of @n@ elements from the input
506784-- stream to an 'Array' aligned to the given size.
@@ -508,44 +786,74 @@ pinnedWriteN = fmap unsafeFreeze . MA.pinnedWriteN
508786-- /Pre-release/
509787--
510788{-# INLINE_NORMAL pinnedWriteNAligned #-}
789+{-# DEPRECATED pinnedWriteNAligned "To be removed." #-}
511790pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a)
512791 => Int -> Int -> Fold m a (Array a)
513792pinnedWriteNAligned alignSize = fmap unsafeFreeze . MA.pinnedWriteNAligned alignSize
514793
515−-- | Like 'writeN' but does not check the array bounds when writing. The fold
794+-- | Like 'createOf' but does not check the array bounds when writing. The fold
516795-- driver must not call the step function more than 'n' times otherwise it will
517796-- corrupt the memory and crash. This function exists mainly because any
518797-- conditional in the step function blocks fusion causing 10x performance
519798-- slowdown.
520799--
521−{-# INLINE_NORMAL writeNUnsafe #-}
800+{-# INLINE_NORMAL unsafeCreateOf #-}
801+unsafeCreateOf :: forall m a. (MonadIO m, Unbox a)
802+ => Int -> Fold m a (Array a)
803+unsafeCreateOf n = unsafeFreeze <$> MA.unsafeCreateOf n
804+
805+{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
806+{-# INLINE writeNUnsafe #-}
522807writeNUnsafe :: forall m a. (MonadIO m, Unbox a)
523808 => Int -> Fold m a (Array a)
524−writeNUnsafe n = unsafeFreeze <$> MA.writeNUnsafe n
809+writeNUnsafe = unsafeCreateOf
525810
526−{-# INLINE_NORMAL pinnedWriteNUnsafe #-}
811+{-# INLINE_NORMAL unsafePinnedCreateOf #-}
812+unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a)
813+ => Int -> Fold m a (Array a)
814+unsafePinnedCreateOf n = unsafeFreeze <$> MA.unsafePinnedCreateOf n
815+
816+{-# DEPRECATED pinnedWriteNUnsafe "Please use unsafePinnedCreateOf instead." #-}
817+{-# INLINE pinnedWriteNUnsafe #-}
527818pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a)
528819 => Int -> Fold m a (Array a)
529−pinnedWriteNUnsafe n = unsafeFreeze <$> MA.pinnedWriteNUnsafe n
820+pinnedWriteNUnsafe = unsafePinnedCreateOf
821+
822+{-# INLINE_NORMAL createWith #-}
823+createWith :: forall m a. (MonadIO m, Unbox a)
824+ => Int -> Fold m a (Array a)
825+-- createWith n = FL.rmapM spliceArrays $ toArraysOf n
826+createWith elemCount = unsafeFreeze <$> MA.createWith elemCount
530827
531−{-# INLINE_NORMAL writeWith #-}
828+{-# DEPRECATED writeWith "Please use createWith instead." #-}
829+{-# INLINE writeWith #-}
532830writeWith :: forall m a. (MonadIO m, Unbox a)
533831 => Int -> Fold m a (Array a)
534−-- writeWith n = FL.rmapM spliceArrays $ toArraysOf n
535−writeWith elemCount = unsafeFreeze <$> MA.writeWith elemCount
832+writeWith = createWith
536833
537834-- | Fold the whole input to a single array.
538835--
539836-- /Caution! Do not use this on infinite streams./
540837--
838+{-# INLINE create #-}
839+create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
840+create = fmap unsafeFreeze MA.create
841+
842+-- XXX Deprecate in major
843+-- {-# DEPRECATED write "Please use create instead." #-}
541844{-# INLINE write #-}
542845write :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
543−write = fmap unsafeFreeze MA.write
846+write = create
544847
545−-- | Like 'write' but creates a pinned array.
848+-- | Like 'create' but creates a pinned array.
849+{-# INLINE pinnedCreate #-}
850+pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
851+pinnedCreate = fmap unsafeFreeze MA.pinnedCreate
852+
853+{-# DEPRECATED pinnedWrite "Please use pinnedCreate instead." #-}
546854{-# INLINE pinnedWrite #-}
547855pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
548−pinnedWrite = fmap unsafeFreeze MA.pinnedWrite
856+pinnedWrite = pinnedCreate
549857
550858-- | Fold "step" has a dependency on "initial", and each step is dependent on
551859-- the previous invocation of step due to state passing, finally extract
@@ -567,15 +875,20 @@ pinnedWrite = fmap unsafeFreeze MA.pinnedWrite
567875-- could be unsafe and dangerous. This is dangerous especially when used with
568876-- foldMany like operations.
569877--
570−-- >>> import qualified Streamly.Internal.Data.Array.Type as Array
571878-- >>> unsafePureWrite = Array.unsafeMakePure Array.write
572879--
573880{-# INLINE unsafeMakePure #-}
574881unsafeMakePure :: Monad m => Fold IO a b -> Fold m a b
575−unsafeMakePure (Fold step initial extract) =
882+unsafeMakePure (Fold step initial extract final) =
576883 Fold (\x a -> return $! unsafeInlineIO (step x a))
577884 (return $! unsafePerformIO initial)
578885 (\s -> return $! unsafeInlineIO $ extract s)
886+ (\s -> return $! unsafeInlineIO $ final s)
887+
888+{-# INLINE fromPureStreamN #-}
889+fromPureStreamN :: Unbox a => Int -> Stream Identity a -> Array a
890+fromPureStreamN n x =
891+ unsafePerformIO $ fmap unsafeFreeze (MA.fromPureStreamN n x)
579892
580893-- | Convert a pure stream in Identity monad to an immutable array.
581894--
@@ -588,6 +901,19 @@ fromPureStream x = unsafePerformIO $ fmap (unsafeFreeze) (MA.fromPureStream x)
588901-- fromPureStream = runIdentity . D.fold (unsafeMakePure write)
589902-- fromPureStream = fromList . runIdentity . D.toList
590903
904+-- XXX This should be monadic.
905+
906+-- | Copy an immutable 'Ptr Word8' sequence into an array.
907+--
908+-- /Unsafe:/ The caller is responsible for safe addressing.
909+--
910+-- Note that this should be evaluated strictly to ensure that we do not hold
911+-- the reference to the pointer in a lazy thunk.
912+fromPtrN :: Int -> Ptr Word8 -> Array Word8
913+fromPtrN n addr = unsafePerformIO $ fmap unsafeFreeze (MA.fromPtrN n addr)
914+
915+-- XXX This should be monadic.
916+
591917-- | Copy a null terminated immutable 'Addr#' Word8 sequence into an array.
592918--
593919-- /Unsafe:/ The caller is responsible for safe addressing.
@@ -595,13 +921,49 @@ fromPureStream x = unsafePerformIO $ fmap (unsafeFreeze) (MA.fromPureStream x)
595921-- Note that this is completely safe when reading from Haskell string
596922-- literals because they are guaranteed to be NULL terminated:
597923--
598−-- >>> :set -XMagicHash
599−-- >>> import qualified Streamly.Internal.Data.Array.Type as Array
600924-- >>> Array.toList $ Array.fromByteStr# "\1\2\3\0"#
601925-- [1,2,3]
602926--
927+-- Note that this should be evaluated strictly to ensure that we do not hold
928+-- the reference to the pointer in a lazy thunk.
603929fromByteStr# :: Addr# -> Array Word8
604−fromByteStr# addr = fromPureStream (D.fromByteStr# addr)
930+fromByteStr# addr = unsafePerformIO $ fmap unsafeFreeze (MA.fromByteStr# addr)
931+
932+-- XXX This should be monadic.
933+
934+-- | Note that this should be evaluated strictly to ensure that we do not hold
935+-- the reference to the pointer in a lazy thunk.
936+fromByteStr :: Ptr Word8 -> Array Word8
937+fromByteStr (Ptr addr#) = fromByteStr# addr#
938+
939+-- XXX implement fromChunks/fromChunkList instead?
940+
941+-- | Convert an array stream to an array. Note that this requires peak memory
942+-- that is double the size of the array stream.
943+--
944+{-# INLINE fromChunksK #-}
945+fromChunksK :: (MonadIO m, Unbox a) => StreamK m (Array a) -> m (Array a)
946+fromChunksK stream =
947+ -- We buffer the entire stream and then allocate the target array of the
948+ -- same size, thus requiring double the memory.
949+ fmap unsafeFreeze $ MA.fromChunksK $ fmap unsafeThaw stream
950+
951+{-# DEPRECATED fromArrayStreamK "Please use fromChunksK instead." #-}
952+fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (Array a) -> m (Array a)
953+fromArrayStreamK = fromChunksK
954+
955+-- | Given a stream of arrays, splice them all together to generate a single
956+-- array. The stream must be /finite/.
957+--
958+{-# INLINE fromChunks #-}
959+fromChunks :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
960+fromChunks s =
961+ -- XXX Check which implementation is better
962+ -- This may also require double the memory as we double the space every
963+ -- time, when copying the last array we may have reallocated almost double
964+ -- the space required before we right size it.
965+ fmap unsafeFreeze $ MA.fromChunksRealloced (fmap unsafeThaw s)
966+ -- fromChunkStreamK $ D.toStreamK s
605967
606968-------------------------------------------------------------------------------
607969-- Instances
@@ -633,12 +995,66 @@ instance Unbox a => IsList (Array a) where
633995 {-# INLINE toList #-}
634996 toList = toList
635997
636−-- XXX we are assuming that Unboxed equality means element equality. This may
637−-- or may not be correct? arrcmp is 40% faster compared to stream equality.
638−instance (Unbox a, Eq a) => Eq (Array a) where
998+-- | Byte compare two arrays. Compare the length of the arrays. If the length
999+-- is equal, compare the lexicographical ordering of two underlying byte arrays
1000+-- otherwise return the result of length comparison.
1001+--
1002+-- /Unsafe/: Note that the 'Unbox' instance of sum types with constructors of
1003+-- different sizes may leave some memory uninitialized which can make byte
1004+-- comparison unreliable.
1005+--
1006+-- /Pre-release/
1007+{-# INLINE byteCmp #-}
1008+byteCmp :: Array a -> Array a -> Ordering
1009+byteCmp arr1 arr2 =
1010+ -- unsafePerformIO?
1011+ unsafeInlineIO $! unsafeThaw arr1 `MA.byteCmp` unsafeThaw arr2
1012+
1013+-- | Byte equality of two arrays.
1014+--
1015+-- >>> byteEq arr1 arr2 = (==) EQ $ Array.byteCmp arr1 arr2
1016+--
1017+-- /Unsafe/: See 'byteCmp'.
1018+{-# INLINE byteEq #-}
1019+byteEq :: Array a -> Array a -> Bool
1020+byteEq arr1 arr2 = (==) EQ $ byteCmp arr1 arr2
1021+
1022+#define MK_EQ_INSTANCE(typ) \
1023+instance {-# OVERLAPPING #-} Eq (Array typ) where { \
1024+; {-# INLINE (==) #-} \
1025+; (==) = byteEq \
1026+}
1027+
1028+MK_EQ_INSTANCE(Char)
1029+MK_EQ_INSTANCE(Word8)
1030+MK_EQ_INSTANCE(Word16)
1031+MK_EQ_INSTANCE(Word32)
1032+
1033+-- XXX The Word64 default instance should be as fast because we are comparing
1034+-- 64-bit at a time.
1035+MK_EQ_INSTANCE(Word64)
1036+MK_EQ_INSTANCE(Int)
1037+MK_EQ_INSTANCE(Int8)
1038+MK_EQ_INSTANCE(Int16)
1039+MK_EQ_INSTANCE(Int32)
1040+
1041+-- XXX The Int64 default instance should be as fast.
1042+MK_EQ_INSTANCE(Int64)
1043+
1044+-- | If the type allows a byte-by-byte comparison this instance can be
1045+-- overlapped by a more specific instance that uses 'byteCmp'. Byte comparison
1046+-- can be significantly faster.
1047+--
1048+instance {-# OVERLAPPABLE #-} (Unbox a, Eq a) => Eq (Array a) where
6391049 {-# INLINE (==) #-}
6401050 arr1 == arr2 =
641− (==) EQ $ unsafeInlineIO $! unsafeThaw arr1 `MA.cmp` unsafeThaw arr2
1051+ -- Does unboxed byte equality mean element equality?
1052+ -- XXX This is incorrect for sum types, as we may have some
1053+ -- uninitialized memory in that case. If we always initialize the
1054+ -- unused memory to zero we can use this.
1055+ -- Byte comparison is 40% faster compared to stream equality.
1056+ -- (==) EQ $ unsafeInlineIO $! unsafeThaw arr1 `MA.cmp` unsafeThaw arr2
1057+ (toStreamD arr1 :: Stream Identity a) == toStreamD arr2
6421058
6431059instance (Unbox a, Ord a) => Ord (Array a) where
6441060 {-# INLINE compare #-}
@@ -680,7 +1096,7 @@ _toStreamD_ size Array{..} = D.Stream step arrStart
6801096 {-# INLINE_LATE step #-}
6811097 step _ p | p == arrEnd = return D.Stop
6821098 step _ p = liftIO $ do
683− x <- peekByteIndex p arrContents
1099+ x <- peekAt p arrContents
6841100 return $ D.Yield x (p + size)
6851101
6861102{-
@@ -703,15 +1119,29 @@ instance Foldable Array where
7031119-- Semigroup and Monoid
7041120-------------------------------------------------------------------------------
7051121
1122+-- XXX Deprecate and remove the Semigroup and Monoid instances because of
1123+-- potential misuse chances.
1124+
1125+-- | This should not be used for combining many or N arrays as it would copy
1126+-- the two arrays everytime to a new array. For coalescing multiple arrays use
1127+-- 'fromChunksK' instead.
7061128instance Unbox a => Semigroup (Array a) where
7071129 arr1 <> arr2 = unsafePerformIO $ splice arr1 arr2
7081130
1131+empty ::
1132+#ifdef DEVBUILD
1133+ Unbox a =>
1134+#endif
1135+ Array a
1136+empty = Array Unboxed.empty 0 0
1137+
1138+{-# DEPRECATED nil "Please use empty instead." #-}
7091139nil ::
7101140#ifdef DEVBUILD
7111141 Unbox a =>
7121142#endif
7131143 Array a
714−nil = Array Unboxed.nil 0 0
1144+nil = empty
7151145
7161146instance Unbox a => Monoid (Array a) where
7171147 mempty = nil
core/src/Streamly/Internal/Serialize/FromBytes.hs → core/src/Streamly/Internal/Data/Binary/Parser.hsrenamed+10−4
@@ -1,13 +1,19 @@
11-- |
2−-- Module : Streamly.Internal.Serialize.FromBytes
2+-- Module : Streamly.Internal.Data.Binary.Parser
33-- Copyright : (c) 2020 Composewell Technologies
44-- License : BSD-3-Clause
55-- Maintainer : streamly@composewell.com
66-- Portability : GHC
77--
88-- Decode Haskell data types from byte streams.
9+--
10+-- It would be inefficient to use this to compose parsers for general algebraic
11+-- data types. For general deserialization of ADTs please use the Serialize
12+-- type class instances. The fastest way to deserialize byte streams
13+-- representing Haskell data types is to write them to arrays and deserialize
14+-- the array using the Serialize type class.
915
10−module Streamly.Internal.Serialize.FromBytes
16+module Streamly.Internal.Data.Binary.Parser
1117 (
1218 -- * Type class
1319 FromBytes (..)
@@ -51,7 +57,7 @@ import Streamly.Internal.Data.Maybe.Strict (Maybe'(..))
5157import Streamly.Internal.Data.Tuple.Strict (Tuple' (..))
5258import qualified Streamly.Data.Array as A
5359import qualified Streamly.Internal.Data.Array as A
54− (unsafeIndex, castUnsafe)
60+ (getIndexUnsafe, castUnsafe)
5561import qualified Streamly.Internal.Data.Parser as PR
5662 (fromPure, either, satisfy, takeEQ)
5763import qualified Streamly.Internal.Data.Parser as PRD
@@ -382,7 +388,7 @@ charLatin1 = fmap (chr . fromIntegral) word8
382388{-# INLINE word64host #-}
383389word64host :: MonadIO m => Parser Word8 m Word64
384390word64host =
385− fmap (A.unsafeIndex 0 . A.castUnsafe) $ PR.takeEQ 8 (A.writeN 8)
391+ fmap (A.getIndexUnsafe 0 . A.castUnsafe) $ PR.takeEQ 8 (A.writeN 8)
386392
387393-------------------------------------------------------------------------------
388394-- Type class
core/src/Streamly/Internal/Serialize/ToBytes.hs → core/src/Streamly/Internal/Data/Binary/Stream.hsrenamed+12−2
@@ -1,13 +1,23 @@
11-- |
2−-- Module : Streamly.Internal.Serialize.ToBytes
2+-- Module : Streamly.Internal.Data.Binary.Stream
33-- Copyright : (c) 2022 Composewell Technologies
44-- License : BSD-3-Clause
55-- Maintainer : streamly@composewell.com
66-- Portability : GHC
77--
88-- Encode Haskell data types to byte streams.
9+--
10+-- The primary purpose of this module is to serialize primitive Haskell types
11+-- to streams for convenient byte by byte processing when such a need arises.
12+--
13+-- It would be inefficient to use this to build byte streams from algebraic
14+-- data types. For general serialization of ADTs please use the Serialize type
15+-- class instances. The fastest way to convert general Haskell types to byte
16+-- streams is to serialize them to an array and then stream the array.
17+
18+-- XXX remove unit, bool, ordering, and the type class as well
919
10−module Streamly.Internal.Serialize.ToBytes
20+module Streamly.Internal.Data.Binary.Stream
1121 (
1222 -- * Type class
1323 ToBytes (..)
core/src/Streamly/Internal/Data/Builder.hs+11−9
@@ -16,7 +16,9 @@ module Streamly.Internal.Data.Builder
1616 )
1717where
1818
19+#if !MIN_VERSION_base(4,18,0)
1920import Control.Applicative (liftA2)
21+#endif
2022
2123------------------------------------------------------------------------------
2224-- The Builder type
@@ -27,16 +29,16 @@ import Control.Applicative (liftA2)
2729-- or even a Fold. Unlike fold the step function is one-shot and not called in
2830-- a loop.
2931newtype Builder s m a =
30− Builder (s -> m (s, a))
32+ Builder (s -> m (a, s))
3133
3234-- | Maps a function on the output of the fold (the type @b@).
3335instance Functor m => Functor (Builder s m) where
3436 {-# INLINE fmap #-}
35− fmap f (Builder step1) = Builder (fmap (fmap f) . step1)
37+ fmap f (Builder step1) = Builder (fmap (\ (a, s) -> (f a, s)) . step1)
3638
3739{-# INLINE fromPure #-}
3840fromPure :: Applicative m => b -> Builder s m b
39−fromPure b = Builder (\s -> pure (s, b))
41+fromPure b = Builder (\s -> pure (b, s))
4042
4143-- | Chain the actions and zip the outputs.
4244{-# INLINE sequenceWith #-}
@@ -47,9 +49,9 @@ sequenceWith func (Builder stepL) (Builder stepR) = Builder step
4749 where
4850
4951 step s = do
50− (s1, x) <- stepL s
51− (s2, y) <- stepR s1
52− pure (s2, func x y)
52+ (x, s1) <- stepL s
53+ (y, s2) <- stepR s1
54+ pure (func x y, s2)
5355
5456instance Monad m => Applicative (Builder a m) where
5557 {-# INLINE pure #-}
@@ -74,7 +76,7 @@ instance Monad m => Monad (Builder a m) where
7476 where
7577
7678 step s = do
77− (s1, x) <- stepL s
79+ (x, s1) <- stepL s
7880 let Builder stepR = f x
79− (s2, y) <- stepR s1
80− pure (s2, y)
81+ (y, s2) <- stepR s1
82+ pure (y, s2)
core/src/Streamly/Internal/Data/Fold.hs+2−2
@@ -20,7 +20,7 @@ module Streamly.Internal.Data.Fold
2020 , module Streamly.Internal.Data.Fold.Tee
2121 , module Streamly.Internal.Data.Fold.Combinators
2222 , module Streamly.Internal.Data.Fold.Container
23− -- , module Streamly.Internal.Data.Fold.Window
23+ , module Streamly.Internal.Data.Fold.Window
2424 )
2525where
2626
@@ -28,6 +28,6 @@ import Streamly.Internal.Data.Fold.Combinators
2828import Streamly.Internal.Data.Fold.Container
2929import Streamly.Internal.Data.Fold.Tee
3030import Streamly.Internal.Data.Fold.Type
31−-- import Streamly.Internal.Data.Fold.Window
31+import Streamly.Internal.Data.Fold.Window
3232
3333#include "DocTestDataFold.hs"
core/src/Streamly/Internal/Data/Fold/Chunked.hs+20−18
@@ -6,7 +6,7 @@
66-- Stability : experimental
77-- Portability : GHC
88--
9−-- Use "Streamly.Data.Parser.Chunked" instead.
9+-- Use "Streamly.Data.Parser" instead.
1010--
1111-- Fold a stream of foreign arrays. @Fold m a b@ in this module works
1212-- on a stream of "Array a" and produces an output of type @b@.
@@ -19,18 +19,19 @@
1919-- folds in Data.Fold to correctly work on an array stream as if it is an
2020-- element stream. For example:
2121--
22−-- >>> import qualified Streamly.Data.Fold as Fold
23−-- >>> import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
24−-- >>> import qualified Streamly.Internal.Data.Fold.Chunked as ChunkFold
25−-- >>> import qualified Streamly.Data.Stream as Stream
26−-- >>> import qualified Streamly.Data.StreamK as StreamK
22+-- >> import qualified Streamly.Data.Fold as Fold
23+-- >> import qualified Streamly.Internal.Data.Array.Stream as ArrayStream
24+-- >> import qualified Streamly.Internal.Data.Fold.Chunked as ChunkFold
25+-- >> import qualified Streamly.Data.Stream as Stream
26+-- >> import qualified Streamly.Data.StreamK as StreamK
2727--
28−-- >>> f = ChunkFold.fromFold (Fold.take 7 Fold.toList)
29−-- >>> s = Stream.chunksOf 5 $ Stream.fromList "hello world"
30−-- >>> ArrayStream.runArrayFold f (StreamK.fromStream s)
28+-- >> f = ChunkFold.fromFold (Fold.take 7 Fold.toList)
29+-- >> s = Stream.chunksOf 5 $ Stream.fromList "hello world"
30+-- >> ArrayStream.runArrayFold f (StreamK.fromStream s)
3131-- Right "hello w"
3232--
3333module Streamly.Internal.Data.Fold.Chunked
34+ {-# DEPRECATED "Please use Streamly.Data.Parser instead." #-}
3435 (
3536 ChunkFold (..)
3637
@@ -58,22 +59,22 @@ where
5859
5960#include "ArrayMacros.h"
6061
62+#if !MIN_VERSION_base(4,18,0)
6163import Control.Applicative (liftA2)
64+#endif
6265import Control.Exception (assert)
6366import Control.Monad.IO.Class (MonadIO(..))
6467import Data.Bifunctor (first)
6568import Data.Proxy (Proxy(..))
6669import Streamly.Internal.Data.Unbox (Unbox(..))
6770import GHC.Types (SPEC(..))
68−import Streamly.Internal.Data.MutArray.Type (touch)
69−import Streamly.Internal.Data.Array.Type (Array(..))
71+import Streamly.Internal.Data.Array (Array(..))
7072import Streamly.Internal.Data.Parser (Initial(..), Step(..))
7173import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
7274
73−import qualified Streamly.Internal.Data.Array as Array
75+import qualified Streamly.Internal.Data.Array.Type as Array
7476import qualified Streamly.Internal.Data.Fold as Fold
7577import qualified Streamly.Internal.Data.Parser as ParserD
76−import qualified Streamly.Internal.Data.Parser.ParserD.Type as ParserD
7778import qualified Streamly.Internal.Data.Parser as Parser
7879
7980import Prelude hiding (concatMap, take)
@@ -102,8 +103,8 @@ newtype ChunkFold m a b = ChunkFold (ParserD.Parser (Array a) m b)
102103{-# INLINE fromFold #-}
103104fromFold :: forall m a b. (MonadIO m, Unbox a) =>
104105 Fold.Fold m a b -> ChunkFold m a b
105−fromFold (Fold.Fold fstep finitial fextract) =
106− ChunkFold (ParserD.Parser step initial (fmap (Done 0) . fextract))
106+fromFold (Fold.Fold fstep finitial _ ffinal) =
107+ ChunkFold (ParserD.Parser step initial extract)
107108
108109 where
109110
@@ -123,7 +124,7 @@ fromFold (Fold.Fold fstep finitial fextract) =
123124 assert (cur == end) (return ())
124125 return $ Partial 0 fs
125126 goArray !_ !cur !fs = do
126− x <- liftIO $ peekByteIndex cur contents
127+ x <- liftIO $ peekAt cur contents
127128 res <- fstep fs x
128129 let elemSize = SIZE_OF(a)
129130 next = INDEX_NEXT(cur,a)
@@ -133,6 +134,8 @@ fromFold (Fold.Fold fstep finitial fextract) =
133134 Fold.Partial fs1 ->
134135 goArray SPEC next fs1
135136
137+ extract = fmap (Done 0) . ffinal
138+
136139-- | Convert an element 'ParserD.Parser' into an array stream fold. If the
137140-- parser fails the fold would throw an exception.
138141--
@@ -160,8 +163,7 @@ fromParserD (ParserD.Parser step1 initial1 extract1) =
160163 else return $ st (arrRem + n) fs1
161164
162165 goArray !_ !cur !fs = do
163− x <- liftIO $ peekByteIndex cur contents
164− liftIO $ touch contents
166+ x <- liftIO $ peekAt cur contents
165167 res <- step1 fs x
166168 let elemSize = SIZE_OF(a)
167169 next = INDEX_NEXT(cur,a)
core/src/Streamly/Internal/Data/Fold/Combinators.hs+86−142
@@ -30,8 +30,6 @@ module Streamly.Internal.Data.Fold.Combinators
3030 -- *** Reducers
3131 , drainMapM
3232 , the
33− , length
34− , lengthGeneric
3533 , mean
3634 , rollingHash
3735 , defaultSalt
@@ -166,8 +164,6 @@ module Streamly.Internal.Data.Fold.Combinators
166164 -- ** Trimming
167165
168166 -- By elements
169− , takeEndBy
170− , takeEndBy_
171167 , takeEndBySeq
172168 , takeEndBySeq_
173169 {-
@@ -240,15 +236,15 @@ import Streamly.Internal.Data.Pipe.Type (Pipe (..), PipeState(..))
240236import Streamly.Internal.Data.Unbox (Unbox, sizeOf)
241237import Streamly.Internal.Data.Unfold.Type (Unfold(..))
242238import Streamly.Internal.Data.Tuple.Strict (Tuple'(..), Tuple3'(..))
243−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
239+import Streamly.Internal.Data.Stream.Type (Stream)
244240
245241import qualified Prelude
246242import qualified Streamly.Internal.Data.MutArray.Type as MA
247243import qualified Streamly.Internal.Data.Array.Type as Array
248−import qualified Streamly.Internal.Data.Fold.Window as FoldW
244+import qualified Streamly.Internal.Data.Fold.Window as Fold
249245import qualified Streamly.Internal.Data.Pipe.Type as Pipe
250246import qualified Streamly.Internal.Data.Ring as Ring
251−import qualified Streamly.Internal.Data.Stream.StreamD.Type as StreamD
247+import qualified Streamly.Internal.Data.Stream.Type as StreamD
252248
253249import Prelude hiding
254250 ( Foldable(..), filter, drop, dropWhile, take, takeWhile, zipWith
@@ -443,8 +439,8 @@ trace f = lmapM (tracing f)
443439-- /Pre-release/
444440{-# INLINE transform #-}
445441transform :: Monad m => Pipe m a b -> Fold m b c -> Fold m a c
446−transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract) =
447− Fold step initial extract
442+transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract ffinal) =
443+ Fold step initial extract final
448444
449445 where
450446
@@ -477,10 +473,14 @@ transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract) =
477473
478474 extract (Tuple' _ fs) = fextract fs
479475
476+ final (Tuple' _ fs) = ffinal fs
477+
480478{-# INLINE scanWith #-}
481479scanWith :: Monad m => Bool -> Fold m a b -> Fold m b c -> Fold m a c
482−scanWith isMany (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
483− Fold step initial extract
480+scanWith isMany
481+ (Fold stepL initialL extractL finalL)
482+ (Fold stepR initialR extractR finalR) =
483+ Fold step initial extract final
484484
485485 where
486486
@@ -494,15 +494,14 @@ scanWith isMany (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
494494 Partial sR1 ->
495495 if isMany
496496 then runStep initialL sR1
497− else Done <$> extractR sR1
497+ else Done <$> finalR sR1
498498 Done bR -> return $ Done bR
499499 Partial sL -> do
500500 !b <- extractL sL
501501 rR <- stepR sR b
502− return
503− $ case rR of
504− Partial sR1 -> Partial (sL, sR1)
505− Done bR -> Done bR
502+ case rR of
503+ Partial sR1 -> return $ Partial (sL, sR1)
504+ Done bR -> finalL sL >> return (Done bR)
506505
507506 initial = do
508507 r <- initialR
@@ -514,6 +513,8 @@ scanWith isMany (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
514513
515514 extract = extractR . snd
516515
516+ final (sL, sR) = finalL sL *> finalR sR
517+
517518-- | Scan the input of a 'Fold' to change it in a stateful manner using another
518519-- 'Fold'. The scan stops as soon as the fold terminates.
519520--
@@ -565,7 +566,7 @@ deleteBy eq x0 = fmap extract $ foldl' step (Tuple' False Nothing)
565566--
566567{-# INLINE slide2 #-}
567568slide2 :: Monad m => Fold m (a, Maybe a) b -> Fold m a b
568−slide2 (Fold step1 initial1 extract1) = Fold step initial extract
569+slide2 (Fold step1 initial1 extract1 final1) = Fold step initial extract final
569570
570571 where
571572
@@ -577,6 +578,8 @@ slide2 (Fold step1 initial1 extract1) = Fold step initial extract
577578
578579 extract (Tuple' _ s) = extract1 s
579580
581+ final (Tuple' _ s) = final1 s
582+
580583-- | Return the latest unique element using the supplied comparison function.
581584-- Returns 'Nothing' if the current element is same as the last element
582585-- otherwise returns 'Just'.
@@ -702,35 +705,11 @@ the = foldt' step initial id
702705-- To Summary
703706------------------------------------------------------------------------------
704707
705−-- | Like 'length', except with a more general 'Num' return value
706−--
707−-- Definition:
708−--
709−-- >>> lengthGeneric = fmap getSum $ Fold.foldMap (Sum . const 1)
710−-- >>> lengthGeneric = Fold.foldl' (\n _ -> n + 1) 0
711−--
712−-- /Pre-release/
713−{-# INLINE lengthGeneric #-}
714−lengthGeneric :: (Monad m, Num b) => Fold m a b
715−lengthGeneric = foldl' (\n _ -> n + 1) 0
716−
717−-- | Determine the length of the input stream.
718−--
719−-- Definition:
720−--
721−-- >>> length = Fold.lengthGeneric
722−-- >>> length = fmap getSum $ Fold.foldMap (Sum . const 1)
723−--
724−{-# INLINE length #-}
725−length :: Monad m => Fold m a Int
726−length = lengthGeneric
727−
728−
729708-- | Determine the sum of all elements of a stream of numbers. Returns additive
730709-- identity (@0@) when the stream is empty. Note that this is not numerically
731710-- stable for floating point numbers.
732711--
733−-- >>> sum = FoldW.cumulative FoldW.sum
712+-- >>> sum = Fold.cumulative Fold.windowSum
734713--
735714-- Same as following but numerically stable:
736715--
@@ -739,7 +718,7 @@ length = lengthGeneric
739718--
740719{-# INLINE sum #-}
741720sum :: (Monad m, Num a) => Fold m a a
742−sum = FoldW.cumulative FoldW.sum
721+sum = Fold.cumulative Fold.windowSum
743722
744723-- | Determine the product of all elements of a stream of numbers. Returns
745724-- multiplicative identity (@1@) when the stream is empty. The fold terminates
@@ -932,7 +911,7 @@ rollingHashFirstN n = take n rollingHash
932911--
933912{-# INLINE rollingMapM #-}
934913rollingMapM :: Monad m => (Maybe a -> a -> m b) -> Fold m a b
935−rollingMapM f = Fold step initial extract
914+rollingMapM f = Fold step initial extract extract
936915
937916 where
938917
@@ -1173,7 +1152,8 @@ head = one
11731152-- /Pre-release/
11741153{-# INLINE findM #-}
11751154findM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1176−findM predicate = Fold step (return $ Partial ()) (const $ return Nothing)
1155+findM predicate =
1156+ Fold step (return $ Partial ()) extract extract
11771157
11781158 where
11791159
@@ -1184,6 +1164,8 @@ findM predicate = Fold step (return $ Partial ()) (const $ return Nothing)
11841164 else Partial ()
11851165 in f <$> predicate a
11861166
1167+ extract = const $ return Nothing
1168+
11871169-- | Returns the first element that satisfies the given predicate.
11881170--
11891171{-# INLINE find #-}
@@ -1426,7 +1408,7 @@ splitAt n fld = splitWith (,) (take n fld)
14261408
14271409{-# INLINE takingEndByM #-}
14281410takingEndByM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1429−takingEndByM p = Fold step initial (return . toMaybe)
1411+takingEndByM p = Fold step initial extract extract
14301412
14311413 where
14321414
@@ -1439,6 +1421,8 @@ takingEndByM p = Fold step initial (return . toMaybe)
14391421 then Done $ Just a
14401422 else Partial $ Just' a
14411423
1424+ extract = return . toMaybe
1425+
14421426-- |
14431427--
14441428-- >>> takingEndBy p = Fold.takingEndByM (return . p)
@@ -1449,7 +1433,7 @@ takingEndBy p = takingEndByM (return . p)
14491433
14501434{-# INLINE takingEndByM_ #-}
14511435takingEndByM_ :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1452−takingEndByM_ p = Fold step initial (return . toMaybe)
1436+takingEndByM_ p = Fold step initial extract extract
14531437
14541438 where
14551439
@@ -1462,6 +1446,8 @@ takingEndByM_ p = Fold step initial (return . toMaybe)
14621446 then Done Nothing
14631447 else Partial $ Just' a
14641448
1449+ extract = return . toMaybe
1450+
14651451-- |
14661452--
14671453-- >>> takingEndBy_ p = Fold.takingEndByM_ (return . p)
@@ -1472,7 +1458,7 @@ takingEndBy_ p = takingEndByM_ (return . p)
14721458
14731459{-# INLINE droppingWhileM #-}
14741460droppingWhileM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1475−droppingWhileM p = Fold step initial (return . toMaybe)
1461+droppingWhileM p = Fold step initial extract extract
14761462
14771463 where
14781464
@@ -1487,6 +1473,8 @@ droppingWhileM p = Fold step initial (return . toMaybe)
14871473 else Just' a
14881474 step _ a = return $ Partial $ Just' a
14891475
1476+ extract = return . toMaybe
1477+
14901478-- |
14911479-- >>> droppingWhile p = Fold.droppingWhileM (return . p)
14921480--
@@ -1494,70 +1482,6 @@ droppingWhileM p = Fold step initial (return . toMaybe)
14941482droppingWhile :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
14951483droppingWhile p = droppingWhileM (return . p)
14961484
1497−-- Note: Keep this consistent with S.splitOn. In fact we should eliminate
1498−-- S.splitOn in favor of the fold.
1499−--
1500−-- XXX Use Fold.many instead once it is fixed.
1501−-- > Stream.splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
1502−
1503−-- | Like 'takeEndBy' but drops the element on which the predicate succeeds.
1504−--
1505−-- Example:
1506−--
1507−-- >>> input = Stream.fromList "hello\nthere\n"
1508−-- >>> line = Fold.takeEndBy_ (== '\n') Fold.toList
1509−-- >>> Stream.fold line input
1510−-- "hello"
1511−--
1512−-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1513−-- ["hello","there"]
1514−--
1515−{-# INLINE takeEndBy_ #-}
1516−takeEndBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1517−-- takeEndBy_ predicate = scanMaybe (takingEndBy_ predicate)
1518−takeEndBy_ predicate (Fold fstep finitial fextract) =
1519− Fold step finitial fextract
1520−
1521− where
1522−
1523− step s a =
1524− if not (predicate a)
1525− then fstep s a
1526− else Done <$> fextract s
1527−
1528−-- Note:
1529−-- > Stream.splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
1530−
1531−-- | Take the input, stop when the predicate succeeds taking the succeeding
1532−-- element as well.
1533−--
1534−-- Example:
1535−--
1536−-- >>> input = Stream.fromList "hello\nthere\n"
1537−-- >>> line = Fold.takeEndBy (== '\n') Fold.toList
1538−-- >>> Stream.fold line input
1539−-- "hello\n"
1540−--
1541−-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1542−-- ["hello\n","there\n"]
1543−--
1544−{-# INLINE takeEndBy #-}
1545−takeEndBy :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1546−-- takeEndBy predicate = scanMaybe (takingEndBy predicate)
1547−takeEndBy predicate (Fold fstep finitial fextract) =
1548− Fold step finitial fextract
1549−
1550− where
1551−
1552− step s a = do
1553− res <- fstep s a
1554− if not (predicate a)
1555− then return res
1556− else do
1557− case res of
1558− Partial s1 -> Done <$> fextract s1
1559− Done b -> return $ Done b
1560−
15611485------------------------------------------------------------------------------
15621486-- Binary splitting on a separator
15631487------------------------------------------------------------------------------
@@ -1590,8 +1514,8 @@ takeEndBySeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) =>
15901514 Array.Array a
15911515 -> Fold m a b
15921516 -> Fold m a b
1593−takeEndBySeq patArr (Fold fstep finitial fextract) =
1594− Fold step initial extract
1517+takeEndBySeq patArr (Fold fstep finitial fextract ffinal) =
1518+ Fold step initial extract final
15951519
15961520 where
15971521
@@ -1604,7 +1528,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16041528 | patLen == 0 ->
16051529 -- XXX Should we match nothing or everything on empty
16061530 -- pattern?
1607− -- Done <$> fextract acc
1531+ -- Done <$> ffinal acc
16081532 return $ Partial $ SplitOnSeqEmpty acc
16091533 | patLen == 1 -> do
16101534 pat <- liftIO $ Array.unsafeIndexIO 0 patArr
@@ -1655,7 +1579,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16551579 case res of
16561580 Partial s1
16571581 | pat /= x -> return $ Partial $ SplitOnSeqSingle s1 pat
1658− | otherwise -> Done <$> fextract s1
1582+ | otherwise -> Done <$> ffinal s1
16591583 Done b -> return $ Done b
16601584 step (SplitOnSeqWord s idx wrd) x = do
16611585 res <- fstep s x
@@ -1664,7 +1588,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16641588 Partial s1
16651589 | idx == maxIndex -> do
16661590 if wrd1 .&. wordMask == wordPat
1667− then Done <$> fextract s1
1591+ then Done <$> ffinal s1
16681592 else return $ Partial $ SplitOnSeqWordLoop s1 wrd1
16691593 | otherwise ->
16701594 return $ Partial $ SplitOnSeqWord s1 (idx + 1) wrd1
@@ -1675,7 +1599,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16751599 case res of
16761600 Partial s1
16771601 | wrd1 .&. wordMask == wordPat ->
1678− Done <$> fextract s1
1602+ Done <$> ffinal s1
16791603 | otherwise ->
16801604 return $ Partial $ SplitOnSeqWordLoop s1 wrd1
16811605 Done b -> return $ Done b
@@ -1689,7 +1613,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16891613 let fld = Ring.unsafeFoldRing (Ring.ringBound rb)
16901614 let !ringHash = fld addCksum 0 rb
16911615 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1692− then Done <$> fextract s1
1616+ then Done <$> ffinal s1
16931617 else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
16941618 else
16951619 return $ Partial $ SplitOnSeqKR s1 (idx + 1) rb rh1
@@ -1702,11 +1626,11 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
17021626 rh1 <- liftIO $ Ring.unsafeInsert rb rh x
17031627 let ringHash = deltaCksum cksum old x
17041628 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1705− then Done <$> fextract s1
1629+ then Done <$> ffinal s1
17061630 else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
17071631 Done b -> return $ Done b
17081632
1709− extract state =
1633+ extractFunc fex state =
17101634 let st =
17111635 case state of
17121636 SplitOnSeqEmpty s -> s
@@ -1715,7 +1639,11 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
17151639 SplitOnSeqWordLoop s _ -> s
17161640 SplitOnSeqKR s _ _ _ -> s
17171641 SplitOnSeqKRLoop s _ _ _ -> s
1718− in fextract st
1642+ in fex st
1643+
1644+ extract state = extractFunc fextract state
1645+
1646+ final state = extractFunc ffinal state
17191647
17201648-- | Like 'takeEndBySeq' but discards the matched sequence.
17211649--
@@ -1726,8 +1654,8 @@ takeEndBySeq_ :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) =>
17261654 Array.Array a
17271655 -> Fold m a b
17281656 -> Fold m a b
1729−takeEndBySeq_ patArr (Fold fstep finitial fextract) =
1730− Fold step initial extract
1657+takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
1658+ Fold step initial extract final
17311659
17321660 where
17331661
@@ -1740,7 +1668,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
17401668 | patLen == 0 ->
17411669 -- XXX Should we match nothing or everything on empty
17421670 -- pattern?
1743− -- Done <$> fextract acc
1671+ -- Done <$> ffinal acc
17441672 return $ Partial $ SplitOnSeqEmpty acc
17451673 | patLen == 1 -> do
17461674 pat <- liftIO $ Array.unsafeIndexIO 0 patArr
@@ -1797,13 +1725,13 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
17971725 case res of
17981726 Partial s1 -> return $ Partial $ SplitOnSeqSingle s1 pat
17991727 Done b -> return $ Done b
1800− else Done <$> fextract s
1728+ else Done <$> ffinal s
18011729 step (SplitOnSeqWord s idx wrd) x = do
18021730 let wrd1 = addToWord wrd x
18031731 if idx == maxIndex
18041732 then do
18051733 if wrd1 .&. wordMask == wordPat
1806− then Done <$> fextract s
1734+ then Done <$> ffinal s
18071735 else return $ Partial $ SplitOnSeqWordLoop s wrd1
18081736 else return $ Partial $ SplitOnSeqWord s (idx + 1) wrd1
18091737 step (SplitOnSeqWordLoop s wrd) x = do
@@ -1814,7 +1742,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18141742 case res of
18151743 Partial s1
18161744 | wrd1 .&. wordMask == wordPat ->
1817− Done <$> fextract s1
1745+ Done <$> ffinal s1
18181746 | otherwise ->
18191747 return $ Partial $ SplitOnSeqWordLoop s1 wrd1
18201748 Done b -> return $ Done b
@@ -1825,7 +1753,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18251753 let fld = Ring.unsafeFoldRing (Ring.ringBound rb)
18261754 let !ringHash = fld addCksum 0 rb
18271755 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1828− then Done <$> fextract s
1756+ then Done <$> ffinal s
18291757 else return $ Partial $ SplitOnSeqKRLoop s ringHash rb rh1
18301758 else return $ Partial $ SplitOnSeqKR s (idx + 1) rb rh1
18311759 step (SplitOnSeqKRLoop s cksum rb rh) x = do
@@ -1836,7 +1764,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18361764 rh1 <- liftIO $ Ring.unsafeInsert rb rh x
18371765 let ringHash = deltaCksum cksum old x
18381766 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1839− then Done <$> fextract s1
1767+ then Done <$> ffinal s1
18401768 else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
18411769 Done b -> return $ Done b
18421770
@@ -1844,10 +1772,10 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18441772 -- terminates early inside extract, we may still have buffered data
18451773 -- remaining which will be lost if we do not communicate that to the
18461774 -- driver.
1847− extract state = do
1775+ extractFunc fex state = do
18481776 let consumeWord s n wrd = do
18491777 if n == 0
1850− then fextract s
1778+ then fex s
18511779 else do
18521780 let old = elemMask .&. (wrd `shiftR` (elemBits * (n - 1)))
18531781 r <- fstep s (toEnum $ fromIntegral old)
@@ -1857,7 +1785,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18571785
18581786 let consumeRing s n rb rh =
18591787 if n == 0
1860− then fextract s
1788+ then fex s
18611789 else do
18621790 old <- liftIO $ peek rh
18631791 let rh1 = Ring.advance rb rh
@@ -1867,13 +1795,17 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18671795 Done b -> return b
18681796
18691797 case state of
1870− SplitOnSeqEmpty s -> fextract s
1871− SplitOnSeqSingle s _ -> fextract s
1798+ SplitOnSeqEmpty s -> fex s
1799+ SplitOnSeqSingle s _ -> fex s
18721800 SplitOnSeqWord s idx wrd -> consumeWord s idx wrd
18731801 SplitOnSeqWordLoop s wrd -> consumeWord s patLen wrd
18741802 SplitOnSeqKR s idx rb _ -> consumeRing s idx rb (Ring.startOf rb)
18751803 SplitOnSeqKRLoop s _ rb rh -> consumeRing s patLen rb rh
18761804
1805+ extract state = extractFunc fextract state
1806+
1807+ final state = extractFunc ffinal state
1808+
18771809------------------------------------------------------------------------------
18781810-- Distributing
18791811------------------------------------------------------------------------------
@@ -2317,8 +2249,8 @@ toStreamRev = fmap StreamD.fromList toListRev
23172249-- /Pre-release/
23182250{-# INLINE unfoldMany #-}
23192251unfoldMany :: Monad m => Unfold m a b -> Fold m b c -> Fold m a c
2320−unfoldMany (Unfold ustep inject) (Fold fstep initial extract) =
2321− Fold consume initial extract
2252+unfoldMany (Unfold ustep inject) (Fold fstep initial extract final) =
2253+ Fold consume initial extract final
23222254
23232255 where
23242256
@@ -2345,7 +2277,7 @@ bottomBy :: (MonadIO m, Unbox a) =>
23452277 (a -> a -> Ordering)
23462278 -> Int
23472279 -> Fold m a (MutArray a)
2348−bottomBy cmp n = Fold step initial extract
2280+bottomBy cmp n = Fold step initial extract extract
23492281
23502282 where
23512283
@@ -2440,8 +2372,8 @@ intersperseWithQuotes
24402372 quote
24412373 esc
24422374 separator
2443− (Fold stepL initialL extractL)
2444− (Fold stepR initialR extractR) = Fold step initial extract
2375+ (Fold stepL initialL _ finalL)
2376+ (Fold stepR initialR extractR finalR) = Fold step initial extract final
24452377
24462378 where
24472379
@@ -2484,11 +2416,13 @@ intersperseWithQuotes
24842416 r <- stepL sL a
24852417 case r of
24862418 Partial s -> return $ Partial (nextState sR s)
2487− Done _ -> error "Collecting fold finished inside quote"
2419+ Done _ -> do
2420+ _ <- finalR sR
2421+ error "Collecting fold finished inside quote"
24882422
24892423 step (IntersperseQUnquoted sR sL) a
24902424 | a == separator = do
2491− b <- extractL sL
2425+ b <- finalL sL
24922426 collect IntersperseQUnquoted sR b
24932427 | a == quote = processQuoted a sL sR IntersperseQQuoted
24942428 | otherwise = process a sL sR IntersperseQUnquoted
@@ -2506,3 +2440,13 @@ intersperseWithQuotes
25062440 error "intersperseWithQuotes: finished inside quote"
25072441 extract (IntersperseQQuotedEsc _ _) =
25082442 error "intersperseWithQuotes: finished inside quote, at escape char"
2443+
2444+ final (IntersperseQUnquoted sR sL) = finalL sL *> finalR sR
2445+ final (IntersperseQQuoted sR sL) = do
2446+ _ <- finalR sR
2447+ _ <- finalL sL
2448+ error "intersperseWithQuotes: finished inside quote"
2449+ final (IntersperseQQuotedEsc sR sL) = do
2450+ _ <- finalR sR
2451+ _ <- finalL sL
2452+ error "intersperseWithQuotes: finished inside quote, at escape char"
core/src/Streamly/Internal/Data/Fold/Container.hs+73−28
@@ -270,14 +270,15 @@ demuxGeneric :: (Monad m, IsMap f, Traversable f) =>
270270 (a -> Key f)
271271 -> (a -> m (Fold m a b))
272272 -> Fold m a (m (f b), Maybe (Key f, b))
273−demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
273+demuxGeneric getKey getFold =
274+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
274275
275276 where
276277
277278 initial = return $ Tuple' IsMap.mapEmpty Nothing
278279
279280 {-# INLINE runFold #-}
280− runFold kv (Fold step1 initial1 extract1) (k, a) = do
281+ runFold kv (Fold step1 initial1 extract1 final1) (k, a) = do
281282 res <- initial1
282283 case res of
283284 Partial s -> do
@@ -285,7 +286,7 @@ demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
285286 return
286287 $ case res1 of
287288 Partial _ ->
288− let fld = Fold step1 (return res1) extract1
289+ let fld = Fold step1 (return res1) extract1 final1
289290 in Tuple' (IsMap.mapInsert k fld kv) Nothing
290291 Done b -> Tuple' (IsMap.mapDelete k kv) (Just (k, b))
291292 Done b -> return $ Tuple' kv (Just (k, b))
@@ -298,15 +299,25 @@ demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
298299 runFold kv fld (k, a)
299300 Just f -> runFold kv f (k, a)
300301
301− extract (Tuple' kv x) = (Prelude.mapM f kv, x)
302+ extract (Tuple' kv x) = return (Prelude.mapM f kv, x)
302303
303304 where
304305
305− f (Fold _ i e) = do
306+ f (Fold _ i e _) = do
306307 r <- i
307308 case r of
308309 Partial s -> e s
309− Done b -> return b
310+ _ -> error "demuxGeneric: unreachable code"
311+
312+ final (Tuple' kv x) = return (Prelude.mapM f kv, x)
313+
314+ where
315+
316+ f (Fold _ i _ fin) = do
317+ r <- i
318+ case r of
319+ Partial s -> fin s
320+ _ -> error "demuxGeneric: unreachable code"
310321
311322-- | @demux getKey getFold@: In a key value stream, fold values corresponding
312323-- to each key using a key specific fold. @getFold@ is invoked to generate a
@@ -348,35 +359,36 @@ demuxGenericIO :: (MonadIO m, IsMap f, Traversable f) =>
348359 (a -> Key f)
349360 -> (a -> m (Fold m a b))
350361 -> Fold m a (m (f b), Maybe (Key f, b))
351−demuxGenericIO getKey getFold = fmap extract $ foldlM' step initial
362+demuxGenericIO getKey getFold =
363+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
352364
353365 where
354366
355367 initial = return $ Tuple' IsMap.mapEmpty Nothing
356368
357369 {-# INLINE initFold #-}
358− initFold kv (Fold step1 initial1 extract1) (k, a) = do
370+ initFold kv (Fold step1 initial1 extract1 final1) (k, a) = do
359371 res <- initial1
360372 case res of
361373 Partial s -> do
362374 res1 <- step1 s a
363375 case res1 of
364376 Partial _ -> do
365− let fld = Fold step1 (return res1) extract1
377+ let fld = Fold step1 (return res1) extract1 final1
366378 ref <- liftIO $ newIORef fld
367379 return $ Tuple' (IsMap.mapInsert k ref kv) Nothing
368380 Done b -> return $ Tuple' kv (Just (k, b))
369381 Done b -> return $ Tuple' kv (Just (k, b))
370382
371383 {-# INLINE runFold #-}
372− runFold kv ref (Fold step1 initial1 extract1) (k, a) = do
384+ runFold kv ref (Fold step1 initial1 extract1 final1) (k, a) = do
373385 res <- initial1
374386 case res of
375387 Partial s -> do
376388 res1 <- step1 s a
377389 case res1 of
378390 Partial _ -> do
379− let fld = Fold step1 (return res1) extract1
391+ let fld = Fold step1 (return res1) extract1 final1
380392 liftIO $ writeIORef ref fld
381393 return $ Tuple' kv Nothing
382394 Done b ->
@@ -394,16 +406,27 @@ demuxGenericIO getKey getFold = fmap extract $ foldlM' step initial
394406 f <- liftIO $ readIORef ref
395407 runFold kv ref f (k, a)
396408
397− extract (Tuple' kv x) = (Prelude.mapM f kv, x)
409+ extract (Tuple' kv x) = return (Prelude.mapM f kv, x)
398410
399411 where
400412
401413 f ref = do
402− (Fold _ i e) <- liftIO $ readIORef ref
414+ Fold _ i e _ <- liftIO $ readIORef ref
403415 r <- i
404416 case r of
405417 Partial s -> e s
406− Done b -> return b
418+ _ -> error "demuxGenericIO: unreachable code"
419+
420+ final (Tuple' kv x) = return (Prelude.mapM f kv, x)
421+
422+ where
423+
424+ f ref = do
425+ Fold _ i _ fin <- liftIO $ readIORef ref
426+ r <- i
427+ case r of
428+ Partial s -> fin s
429+ _ -> error "demuxGenericIO: unreachable code"
407430
408431-- | This is specialized version of 'demux' that uses mutable IO cells as
409432-- fold accumulators for better performance.
@@ -418,6 +441,13 @@ demuxIO :: (MonadIO m, Ord k) =>
418441 -> Fold m a (m (Map k b), Maybe (k, b))
419442demuxIO = demuxGenericIO
420443
444+-- | Fold a key value stream to a key-value Map. If the same key appears
445+-- multiple times, only the last value is retained.
446+{-# INLINE kvToMapOverwriteGeneric #-}
447+kvToMapOverwriteGeneric :: (Monad m, IsMap f) => Fold m (Key f, a) (f a)
448+kvToMapOverwriteGeneric =
449+ foldl' (\kv (k, v) -> IsMap.mapInsert k v kv) IsMap.mapEmpty
450+
421451{-# INLINE demuxToContainer #-}
422452demuxToContainer :: (Monad m, IsMap f, Traversable f) =>
423453 (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
@@ -505,8 +535,8 @@ classifyGeneric :: (Monad m, IsMap f, Traversable f, Ord (Key f)) =>
505535 -- for that use case. We return an action because we want it to be lazy so
506536 -- that the downstream consumers can choose to process or discard it.
507537 (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
508−classifyGeneric f (Fold step1 initial1 extract1) =
509− fmap extract $ foldlM' step initial
538+classifyGeneric f (Fold step1 initial1 extract1 final1) =
539+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
510540
511541 where
512542
@@ -543,7 +573,16 @@ classifyGeneric f (Fold step1 initial1 extract1) =
543573 let kv1 = IsMap.mapDelete k kv
544574 in Tuple3' kv1 (Set.insert k set) (Just (k, b))
545575
546− extract (Tuple3' kv _ x) = (Prelude.mapM extract1 kv, x)
576+ extract (Tuple3' kv _ x) = return (Prelude.mapM extract1 kv, x)
577+
578+ final (Tuple3' kv set x) = return (IsMap.mapTraverseWithKey f1 kv, x)
579+
580+ where
581+
582+ f1 k s = do
583+ if Set.member k set
584+ then extract1 s
585+ else final1 s
547586
548587-- | Folds the values for each key using the supplied fold. When scanning, as
549588-- soon as the fold is complete, its result is available in the second
@@ -568,8 +607,8 @@ classify = classifyGeneric
568607{-# INLINE classifyGenericIO #-}
569608classifyGenericIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) =>
570609 (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
571−classifyGenericIO f (Fold step1 initial1 extract1) =
572− fmap extract $ foldlM' step initial
610+classifyGenericIO f (Fold step1 initial1 extract1 final1) =
611+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
573612
574613 where
575614
@@ -608,8 +647,21 @@ classifyGenericIO f (Fold step1 initial1 extract1) =
608647 in return
609648 $ Tuple3' kv1 (Set.insert k set) (Just (k, b))
610649
611− extract (Tuple3' kv _ x) =
612− (Prelude.mapM (\ref -> liftIO (readIORef ref) >>= extract1) kv, x)
650+ extract (Tuple3' kv _ x) = return (Prelude.mapM g kv, x)
651+
652+ where
653+
654+ g ref = liftIO (readIORef ref) >>= extract1
655+
656+ final (Tuple3' kv set x) = return (IsMap.mapTraverseWithKey g kv, x)
657+
658+ where
659+
660+ g k ref = do
661+ s <- liftIO $ readIORef ref
662+ if Set.member k set
663+ then extract1 s
664+ else final1 s
613665
614666-- | Same as classify except that it uses mutable IORef cells in the
615667-- Map providing better performance. Be aware that if this is used as a scan,
@@ -624,13 +676,6 @@ classifyIO :: (MonadIO m, Ord k) =>
624676 (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
625677classifyIO = classifyGenericIO
626678
627−-- | Fold a key value stream to a key-value Map. If the same key appears
628−-- multiple times, only the last value is retained.
629−{-# INLINE kvToMapOverwriteGeneric #-}
630−kvToMapOverwriteGeneric :: (Monad m, IsMap f) => Fold m (Key f, a) (f a)
631−kvToMapOverwriteGeneric =
632− foldl' (\kv (k, v) -> IsMap.mapInsert k v kv) IsMap.mapEmpty
633−
634679{-# INLINE toContainer #-}
635680toContainer :: (Monad m, IsMap f, Traversable f, Ord (Key f)) =>
636681 (a -> Key f) -> Fold m a b -> Fold m a (f b)
core/src/Streamly/Internal/Data/Fold/Step.hs+1−1
@@ -40,7 +40,7 @@ data Step s b
4040 = Partial !s
4141 | Done !b
4242
43−-- | 'first' maps over 'Partial' and 'second' maps over 'Done'.
43+-- | 'first' maps over the fold state and 'second' maps over the fold result.
4444--
4545instance Bifunctor Step where
4646 {-# INLINE bimap #-}
core/src/Streamly/Internal/Data/Fold/Tee.hs+2
@@ -16,7 +16,9 @@ module Streamly.Internal.Data.Fold.Tee
1616 )
1717where
1818
19+#if !MIN_VERSION_base(4,18,0)
1920import Control.Applicative (liftA2)
21+#endif
2022import Streamly.Internal.Data.Fold.Type (Fold)
2123
2224import qualified Streamly.Internal.Data.Fold.Type as Fold
core/src/Streamly/Internal/Data/Fold/Type.hs+339−115
@@ -367,6 +367,8 @@ module Streamly.Internal.Data.Fold.Type
367367 , toList
368368 , toStreamK
369369 , toStreamKRev
370+ , lengthGeneric
371+ , length
370372
371373 -- * Combinators
372374
@@ -391,6 +393,8 @@ module Streamly.Internal.Data.Fold.Type
391393 -- ** Trimming
392394 , take
393395 , taking
396+ , takeEndBy_
397+ , takeEndBy
394398 , dropping
395399
396400 -- ** Sequential application
@@ -442,8 +446,10 @@ where
442446
443447#include "inline.hs"
444448
449+#if !MIN_VERSION_base(4,18,0)
445450import Control.Applicative (liftA2)
446−import Control.Monad ((>=>))
451+#endif
452+import Control.Monad ((>=>), void)
447453import Data.Bifunctor (Bifunctor(..))
448454import Data.Either (fromLeft, fromRight, isLeft, isRight)
449455import Data.Functor.Identity (Identity(..))
@@ -452,7 +458,7 @@ import Streamly.Internal.Data.Maybe.Strict (Maybe'(..), toMaybe)
452458import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
453459import Streamly.Internal.Data.Refold.Type (Refold(..))
454460
455−import qualified Streamly.Internal.Data.Stream.StreamK.Type as K
461+import qualified Streamly.Internal.Data.StreamK.Type as K
456462
457463import Prelude hiding (Foldable(..), concatMap, filter, map, take)
458464
@@ -469,23 +475,64 @@ import Streamly.Internal.Data.Fold.Step
469475-- The type @b@ is the accumulator of the writer. That's the reason the
470476-- default folds in various modules are called "write".
471477
472−-- | The type @Fold m a b@ having constructor @Fold step initial extract@
473−-- represents a fold over an input stream of values of type @a@ to a final
474−-- value of type @b@ in 'Monad' @m@.
475−--
476−-- The fold uses an intermediate state @s@ as accumulator, the type @s@ is
477−-- internal to the specific fold definition. The initial value of the fold
478−-- state @s@ is returned by @initial@. The @step@ function consumes an input
479−-- and either returns the final result @b@ if the fold is done or the next
480−-- intermediate state (see 'Step'). At any point the fold driver can extract
481−-- the result from the intermediate state using the @extract@ function.
478+-- An alternative to using an "extract" function is to use "Partial s b" style
479+-- partial value so that we always emit the output value and there is no need
480+-- to extract. Then extract can be used for cleanup purposes. But in this case
481+-- in some cases we may need a "Continue" constructor where an output value is
482+-- not available, this was implicit earlier. Also, "b" should be lazy here so
483+-- that we do not always compute it even if we do not need it.
484+--
485+-- Partial s b --> extract :: s -> b
486+-- Continue --> extract :: s -> Maybe b
487+--
488+-- But keeping 'b' lazy does not let the fold optimize well. It leads to
489+-- significant regressions in the key-value folds.
490+--
491+-- The "final" function complicates combinators that take other folds as
492+-- argument because we need to call their finalizers at right places. An
493+-- alternative to reduce this complexity where it is not required is to use a
494+-- separate type for bracketed folds but then we need to manage the complexity
495+-- of two different fold types.
496+
497+-- The "final" function could be (s -> m (Step s b)), like in parsers so that
498+-- it can be called in a loop to drain the fold.
499+
500+-- | The type @Fold m a b@ represents a consumer of an input stream of values
501+-- of type @a@ and returning a final value of type @b@ in 'Monad' @m@. The
502+-- constructor of a fold is @Fold step initial extract final@.
503+--
504+-- The fold uses an internal state of type @s@. The initial value of the state
505+-- @s@ is created by @initial@. This function is called once and only once
506+-- before the fold starts consuming input. Any resource allocation can be done
507+-- in this function.
508+--
509+-- The @step@ function is called on each input, it consumes an input and
510+-- returns the next intermediate state (see 'Step') or the final result @b@ if
511+-- the fold terminates.
512+--
513+-- If the fold is used as a scan, the @extract@ function is used by the scan
514+-- driver to map the current state @s@ of the fold to the fold result. Thus
515+-- @extract@ can be called multiple times. In some folds, where scanning does
516+-- not make sense, this function is left unimplemented; such folds cannot be
517+-- used as scans.
518+--
519+-- Before a fold terminates, @final@ is called once and only once (unless the
520+-- fold terminated in @initial@ itself). Any resources allocated by @initial@
521+-- can be released in @final@. In folds that do not require any cleanup
522+-- @extract@ and @final@ are typically the same.
523+--
524+-- When implementing fold combinators, care should be taken to cleanup any
525+-- state of the argument folds held by the fold by calling the respective
526+-- @final@ at all exit points of the fold. Also, @final@ should not be called
527+-- more than once. Note that if a fold terminates by 'Done' constructor, there
528+-- is no state to cleanup.
482529--
483530-- NOTE: The constructor is not yet released, smart constructors are provided
484531-- to create folds.
485532--
486533data Fold m a b =
487− -- | @Fold @ @ step @ @ initial @ @ extract@
488− forall s. Fold (s -> a -> m (Step s b)) (m (Step s b)) (s -> m b)
534+ -- | @Fold@ @step@ @initial@ @extract@ @final@
535+ forall s. Fold (s -> a -> m (Step s b)) (m (Step s b)) (s -> m b) (s -> m b)
489536
490537------------------------------------------------------------------------------
491538-- Mapping on the output
@@ -495,7 +542,8 @@ data Fold m a b =
495542--
496543{-# INLINE rmapM #-}
497544rmapM :: Monad m => (b -> m c) -> Fold m a b -> Fold m a c
498−rmapM f (Fold step initial extract) = Fold step1 initial1 (extract >=> f)
545+rmapM f (Fold step initial extract final) =
546+ Fold step1 initial1 (extract >=> f) (final >=> f)
499547
500548 where
501549
@@ -526,6 +574,7 @@ foldl' step initial =
526574 (\s a -> return $ Partial $ step s a)
527575 (return (Partial initial))
528576 return
577+ return
529578
530579-- | Make a fold from a left fold style monadic step function and initial value
531580-- of the accumulator.
@@ -540,7 +589,7 @@ foldl' step initial =
540589{-# INLINE foldlM' #-}
541590foldlM' :: Monad m => (b -> a -> m b) -> m b -> Fold m a b
542591foldlM' step initial =
543− Fold (\s a -> Partial <$> step s a) (Partial <$> initial) return
592+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) return return
544593
545594-- | Make a strict left fold, for non-empty streams, using first element as the
546595-- starting value. Returns Nothing if the stream is empty.
@@ -638,7 +687,11 @@ foldrM' g z =
638687{-# INLINE foldt' #-}
639688foldt' :: Monad m => (s -> a -> Step s b) -> Step s b -> (s -> b) -> Fold m a b
640689foldt' step initial extract =
641− Fold (\s a -> return $ step s a) (return initial) (return . extract)
690+ Fold
691+ (\s a -> return $ step s a)
692+ (return initial)
693+ (return . extract)
694+ (return . extract)
642695
643696-- | Make a terminating fold with an effectful step function and initial state,
644697-- and a state extraction function.
@@ -651,7 +704,7 @@ foldt' step initial extract =
651704--
652705{-# INLINE foldtM' #-}
653706foldtM' :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> Fold m a b
654−foldtM' = Fold
707+foldtM' step initial extract = Fold step initial extract extract
655708
656709------------------------------------------------------------------------------
657710-- Refold
@@ -665,7 +718,7 @@ foldtM' = Fold
665718-- /Internal/
666719fromRefold :: Refold m c a b -> c -> Fold m a b
667720fromRefold (Refold step inject extract) c =
668− Fold step (inject c) extract
721+ Fold step (inject c) extract extract
669722
670723------------------------------------------------------------------------------
671724-- Basic Folds
@@ -718,6 +771,29 @@ toStreamKRev = foldl' (flip K.cons) K.nil
718771toStreamK :: Monad m => Fold m a (K.StreamK n a)
719772toStreamK = foldr K.cons K.nil
720773
774+-- | Like 'length', except with a more general 'Num' return value
775+--
776+-- Definition:
777+--
778+-- >>> lengthGeneric = fmap getSum $ Fold.foldMap (Sum . const 1)
779+-- >>> lengthGeneric = Fold.foldl' (\n _ -> n + 1) 0
780+--
781+-- /Pre-release/
782+{-# INLINE lengthGeneric #-}
783+lengthGeneric :: (Monad m, Num b) => Fold m a b
784+lengthGeneric = foldl' (\n _ -> n + 1) 0
785+
786+-- | Determine the length of the input stream.
787+--
788+-- Definition:
789+--
790+-- >>> length = Fold.lengthGeneric
791+-- >>> length = fmap getSum $ Fold.foldMap (Sum . const 1)
792+--
793+{-# INLINE length #-}
794+length :: Monad m => Fold m a Int
795+length = lengthGeneric
796+
721797------------------------------------------------------------------------------
722798-- Instances
723799------------------------------------------------------------------------------
@@ -725,7 +801,8 @@ toStreamK = foldr K.cons K.nil
725801-- | Maps a function on the output of the fold (the type @b@).
726802instance Functor m => Functor (Fold m a) where
727803 {-# INLINE fmap #-}
728− fmap f (Fold step1 initial1 extract) = Fold step initial (fmap2 f extract)
804+ fmap f (Fold step1 initial1 extract final) =
805+ Fold step initial (fmap2 f extract) (fmap2 f final)
729806
730807 where
731808
@@ -760,7 +837,7 @@ instance Functor m => Functor (Fold m a) where
760837--
761838{-# INLINE fromPure #-}
762839fromPure :: Applicative m => b -> Fold m a b
763−fromPure b = Fold undefined (pure $ Done b) pure
840+fromPure b = Fold undefined (pure $ Done b) pure pure
764841
765842-- | Make a fold that yields the result of the supplied effectful action
766843-- without consuming any further input.
@@ -769,7 +846,7 @@ fromPure b = Fold undefined (pure $ Done b) pure
769846--
770847{-# INLINE fromEffect #-}
771848fromEffect :: Applicative m => m b -> Fold m a b
772−fromEffect b = Fold undefined (Done <$> b) pure
849+fromEffect b = Fold undefined (Done <$> b) pure pure
773850
774851{-# ANN type SeqFoldState Fuse #-}
775852data SeqFoldState sl f sr = SeqFoldL !sl | SeqFoldR !f !sr
@@ -794,16 +871,20 @@ data SeqFoldState sl f sr = SeqFoldL !sl | SeqFoldR !f !sr
794871-- complexity, because each composition adds a new branch that each subsequent
795872-- fold's input element has to traverse, therefore, it cannot scale to a large
796873-- number of compositions. After around 100 compositions the performance starts
797−-- dipping rapidly compared to a CPS style implementation. When you need
798−-- scaling use parser monad instead.
874+-- dipping rapidly compared to a CPS style implementation.
875+--
876+-- For larger number of compositions you can convert the fold to a parser and
877+-- use ParserK.
799878--
800879-- /Time: O(n^2) where n is the number of compositions./
801880--
802881{-# INLINE splitWith #-}
803882splitWith :: Monad m =>
804883 (a -> b -> c) -> Fold m x a -> Fold m x b -> Fold m x c
805−splitWith func (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
806− Fold step initial extract
884+splitWith func
885+ (Fold stepL initialL _ finalL)
886+ (Fold stepR initialR _ finalR) =
887+ Fold step initial extract final
807888
808889 where
809890
@@ -820,13 +901,18 @@ splitWith func (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
820901 step (SeqFoldL st) a = runL (stepL st a)
821902 step (SeqFoldR f st) a = runR (stepR st a) f
822903
823− extract (SeqFoldR f sR) = fmap f (extractR sR)
824− extract (SeqFoldL sL) = do
825− rL <- extractL sL
904+ -- XXX splitWith should not be used for scanning
905+ -- It would rarely make sense and resource tracking and cleanup would be
906+ -- expensive. especially when multiple splitWith are chained.
907+ extract _ = error "splitWith: cannot be used for scanning"
908+
909+ final (SeqFoldR f sR) = fmap f (finalR sR)
910+ final (SeqFoldL sL) = do
911+ rL <- finalL sL
826912 res <- initialR
827913 fmap (func rL)
828914 $ case res of
829− Partial sR -> extractR sR
915+ Partial sR -> finalR sR
830916 Done rR -> return rR
831917
832918{-# DEPRECATED serialWith "Please use \"splitWith\" instead" #-}
@@ -846,8 +932,8 @@ data SeqFoldState_ sl sr = SeqFoldL_ !sl | SeqFoldR_ !sr
846932--
847933{-# INLINE split_ #-}
848934split_ :: Monad m => Fold m x a -> Fold m x b -> Fold m x b
849−split_ (Fold stepL initialL _) (Fold stepR initialR extractR) =
850− Fold step initial extract
935+split_ (Fold stepL initialL _ finalL) (Fold stepR initialR _ finalR) =
936+ Fold step initial extract final
851937
852938 where
853939
@@ -870,11 +956,16 @@ split_ (Fold stepL initialL _) (Fold stepR initialR extractR) =
870956 resR <- stepR st a
871957 return $ first SeqFoldR_ resR
872958
873− extract (SeqFoldR_ sR) = extractR sR
874− extract (SeqFoldL_ _) = do
959+ -- XXX split_ should not be used for scanning
960+ -- See splitWith for more details.
961+ extract _ = error "split_: cannot be used for scanning"
962+
963+ final (SeqFoldR_ sR) = finalR sR
964+ final (SeqFoldL_ sL) = do
965+ _ <- finalL sL
875966 res <- initialR
876967 case res of
877− Partial sR -> extractR sR
968+ Partial sR -> finalR sR
878969 Done rR -> return rR
879970
880971-- | 'Applicative' form of 'splitWith'. Split the input serially over two
@@ -922,8 +1013,10 @@ data TeeState sL sR bL bR
9221013--
9231014{-# INLINE teeWith #-}
9241015teeWith :: Monad m => (a -> b -> c) -> Fold m x a -> Fold m x b -> Fold m x c
925−teeWith f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
926− Fold step initial extract
1016+teeWith f
1017+ (Fold stepL initialL extractL finalL)
1018+ (Fold stepR initialR extractR finalR) =
1019+ Fold step initial extract final
9271020
9281021 where
9291022
@@ -950,6 +1043,10 @@ teeWith f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
9501043 extract (TeeLeft bR sL) = (`f` bR) <$> extractL sL
9511044 extract (TeeRight bL sR) = f bL <$> extractR sR
9521045
1046+ final (TeeBoth sL sR) = f <$> finalL sL <*> finalR sR
1047+ final (TeeLeft bR sL) = (`f` bR) <$> finalL sL
1048+ final (TeeRight bL sR) = f bL <$> finalR sR
1049+
9531050{-# ANN type TeeFstState Fuse #-}
9541051data TeeFstState sL sR b
9551052 = TeeFstBoth !sL !sR
@@ -962,8 +1059,10 @@ data TeeFstState sL sR b
9621059{-# INLINE teeWithFst #-}
9631060teeWithFst :: Monad m =>
9641061 (b -> c -> d) -> Fold m a b -> Fold m a c -> Fold m a d
965−teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
966− Fold step initial extract
1062+teeWithFst f
1063+ (Fold stepL initialL extractL finalL)
1064+ (Fold stepR initialR extractR finalR) =
1065+ Fold step initial extract final
9671066
9681067 where
9691068
@@ -982,7 +1081,7 @@ teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
9821081 Done bl -> do
9831082 Done . f bl <$>
9841083 case resR of
985− Partial sr -> extractR sr
1084+ Partial sr -> finalR sr
9861085 Done br -> return br
9871086
9881087 initial = runBoth initialL initialR
@@ -993,6 +1092,9 @@ teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
9931092 extract (TeeFstBoth sL sR) = f <$> extractL sL <*> extractR sR
9941093 extract (TeeFstLeft bR sL) = (`f` bR) <$> extractL sL
9951094
1095+ final (TeeFstBoth sL sR) = f <$> finalL sL <*> finalR sR
1096+ final (TeeFstLeft bR sL) = (`f` bR) <$> finalL sL
1097+
9961098-- | Like 'teeWith' but terminates as soon as any one of the two folds
9971099-- terminates.
9981100--
@@ -1001,8 +1103,10 @@ teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10011103{-# INLINE teeWithMin #-}
10021104teeWithMin :: Monad m =>
10031105 (b -> c -> d) -> Fold m a b -> Fold m a c -> Fold m a d
1004−teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1005− Fold step initial extract
1106+teeWithMin f
1107+ (Fold stepL initialL extractL finalL)
1108+ (Fold stepR initialR extractR finalR) =
1109+ Fold step initial extract final
10061110
10071111 where
10081112
@@ -1014,12 +1118,12 @@ teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10141118 Partial sl -> do
10151119 case resR of
10161120 Partial sr -> return $ Partial $ Tuple' sl sr
1017− Done br -> Done . (`f` br) <$> extractL sl
1121+ Done br -> Done . (`f` br) <$> finalL sl
10181122
10191123 Done bl -> do
10201124 Done . f bl <$>
10211125 case resR of
1022− Partial sr -> extractR sr
1126+ Partial sr -> finalR sr
10231127 Done br -> return br
10241128
10251129 initial = runBoth initialL initialR
@@ -1028,6 +1132,8 @@ teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10281132
10291133 extract (Tuple' sL sR) = f <$> extractL sL <*> extractR sR
10301134
1135+ final (Tuple' sL sR) = f <$> finalL sL <*> finalR sR
1136+
10311137-- | Shortest alternative. Apply both folds in parallel but choose the result
10321138-- from the one which consumed least input i.e. take the shortest succeeding
10331139-- fold.
@@ -1039,8 +1145,8 @@ teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10391145--
10401146{-# INLINE shortest #-}
10411147shortest :: Monad m => Fold m x a -> Fold m x b -> Fold m x (Either a b)
1042−shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
1043− Fold step initial extract
1148+shortest (Fold stepL initialL extractL finalL) (Fold stepR initialR _ finalR) =
1149+ Fold step initial extract final
10441150
10451151 where
10461152
@@ -1048,10 +1154,16 @@ shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
10481154 runBoth actionL actionR = do
10491155 resL <- actionL
10501156 resR <- actionR
1051− return $
1052− case resL of
1053− Partial sL -> bimap (Tuple' sL) Right resR
1054− Done bL -> Done $ Left bL
1157+ case resL of
1158+ Partial sL ->
1159+ case resR of
1160+ Partial sR -> return $ Partial $ Tuple' sL sR
1161+ Done bR -> finalL sL >> return (Done (Right bR))
1162+ Done bL -> do
1163+ case resR of
1164+ Partial sR -> void (finalR sR)
1165+ Done _ -> return ()
1166+ return (Done (Left bL))
10551167
10561168 initial = runBoth initialL initialR
10571169
@@ -1059,6 +1171,8 @@ shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
10591171
10601172 extract (Tuple' sL _) = Left <$> extractL sL
10611173
1174+ final (Tuple' sL sR) = Left <$> finalL sL <* finalR sR
1175+
10621176{-# ANN type LongestState Fuse #-}
10631177data LongestState sL sR
10641178 = LongestBoth !sL !sR
@@ -1076,8 +1190,10 @@ data LongestState sL sR
10761190--
10771191{-# INLINE longest #-}
10781192longest :: Monad m => Fold m x a -> Fold m x b -> Fold m x (Either a b)
1079−longest (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1080− Fold step initial extract
1193+longest
1194+ (Fold stepL initialL _ finalL)
1195+ (Fold stepR initialR _ finalR) =
1196+ Fold step initial extract final
10811197
10821198 where
10831199
@@ -1100,14 +1216,18 @@ longest (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
11001216 step (LongestLeft sL) a = bimap LongestLeft Left <$> stepL sL a
11011217 step (LongestRight sR) a = bimap LongestRight Right <$> stepR sR a
11021218
1103− left sL = Left <$> extractL sL
1104− extract (LongestLeft sL) = left sL
1105− extract (LongestRight sR) = Right <$> extractR sR
1106− extract (LongestBoth sL _) = left sL
1219+ -- XXX Scan with this may not make sense as we cannot determine the longest
1220+ -- until one of them have exhausted.
1221+ extract _ = error $ "longest: scan is not allowed as longest cannot be "
1222+ ++ "determined until one fold has exhausted."
1223+
1224+ final (LongestLeft sL) = Left <$> finalL sL
1225+ final (LongestRight sR) = Right <$> finalR sR
1226+ final (LongestBoth sL sR) = Left <$> finalL sL <* finalR sR
11071227
1108−data ConcatMapState m sa a c
1109− = B !sa
1110− | forall s. C (s -> a -> m (Step s c)) !s (s -> m c)
1228+data ConcatMapState m sa a b c
1229+ = B !sa (sa -> m b)
1230+ | forall s. C (s -> a -> m (Step s c)) !s (s -> m c) (s -> m c)
11111231
11121232-- | Map a 'Fold' returning function on the result of a 'Fold' and run the
11131233-- returned fold. This operation can be used to express data dependencies
@@ -1130,43 +1250,47 @@ data ConcatMapState m sa a c
11301250--
11311251{-# INLINE concatMap #-}
11321252concatMap :: Monad m => (b -> Fold m a c) -> Fold m a b -> Fold m a c
1133−concatMap f (Fold stepa initiala extracta) = Fold stepc initialc extractc
1253+concatMap f (Fold stepa initiala _ finala) =
1254+ Fold stepc initialc extractc finalc
11341255 where
11351256 initialc = do
11361257 r <- initiala
11371258 case r of
1138− Partial s -> return $ Partial (B s)
1259+ Partial s -> return $ Partial (B s finala)
11391260 Done b -> initInnerFold (f b)
11401261
1141− stepc (B s) a = do
1262+ stepc (B s fin) a = do
11421263 r <- stepa s a
11431264 case r of
1144− Partial s1 -> return $ Partial (B s1)
1265+ Partial s1 -> return $ Partial (B s1 fin)
11451266 Done b -> initInnerFold (f b)
11461267
1147− stepc (C stepInner s extractInner) a = do
1268+ stepc (C stepInner s extractInner fin) a = do
11481269 r <- stepInner s a
11491270 return $ case r of
1150− Partial sc -> Partial (C stepInner sc extractInner)
1271+ Partial sc -> Partial (C stepInner sc extractInner fin)
11511272 Done c -> Done c
11521273
1153− extractc (B s) = do
1154− r <- extracta s
1155− initExtract (f r)
1156− extractc (C _ sInner extractInner) = extractInner sInner
1274+ -- XXX Cannot use for scanning
1275+ extractc _ = error "concatMap: cannot be used for scanning"
11571276
1158− initInnerFold (Fold step i e) = do
1277+ initInnerFold (Fold step i e fin) = do
11591278 r <- i
11601279 return $ case r of
1161− Partial s -> Partial (C step s e)
1280+ Partial s -> Partial (C step s e fin)
11621281 Done c -> Done c
11631282
1164− initExtract (Fold _ i e) = do
1283+ initFinalize (Fold _ i _ fin) = do
11651284 r <- i
11661285 case r of
1167− Partial s -> e s
1286+ Partial s -> fin s
11681287 Done c -> return c
11691288
1289+ finalc (B s fin) = do
1290+ r <- fin s
1291+ initFinalize (f r)
1292+ finalc (C _ sInner _ fin) = fin sInner
1293+
11701294------------------------------------------------------------------------------
11711295-- Mapping on input
11721296------------------------------------------------------------------------------
@@ -1185,7 +1309,7 @@ concatMap f (Fold stepa initiala extracta) = Fold stepc initialc extractc
11851309--
11861310{-# INLINE lmap #-}
11871311lmap :: (a -> b) -> Fold m b r -> Fold m a r
1188−lmap f (Fold step begin done) = Fold step' begin done
1312+lmap f (Fold step begin done final) = Fold step' begin done final
11891313 where
11901314 step' x a = step x (f a)
11911315
@@ -1193,7 +1317,7 @@ lmap f (Fold step begin done) = Fold step' begin done
11931317--
11941318{-# INLINE lmapM #-}
11951319lmapM :: Monad m => (a -> m b) -> Fold m b r -> Fold m a r
1196−lmapM f (Fold step begin done) = Fold step' begin done
1320+lmapM f (Fold step begin done final) = Fold step' begin done final
11971321 where
11981322 step' x a = f a >>= step x
11991323
@@ -1205,8 +1329,10 @@ lmapM f (Fold step begin done) = Fold step' begin done
12051329-- /Pre-release/
12061330{-# INLINE postscan #-}
12071331postscan :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1208−postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1209− Fold step initial extract
1332+postscan
1333+ (Fold stepL initialL extractL finalL)
1334+ (Fold stepR initialR extractR finalR) =
1335+ Fold step initial extract final
12101336
12111337 where
12121338
@@ -1217,30 +1343,32 @@ postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
12171343 Done bL -> do
12181344 rR <- stepR sR bL
12191345 case rR of
1220− Partial sR1 -> Done <$> extractR sR1
1346+ Partial sR1 -> Done <$> finalR sR1
12211347 Done bR -> return $ Done bR
12221348 Partial sL -> do
12231349 !b <- extractL sL
12241350 rR <- stepR sR b
1225− return
1226− $ case rR of
1227− Partial sR1 -> Partial (sL, sR1)
1228− Done bR -> Done bR
1351+ case rR of
1352+ Partial sR1 -> return $ Partial (sL, sR1)
1353+ Done bR -> finalL sL >> return (Done bR)
12291354
12301355 initial = do
1231− r <- initialR
1232− rL <- initialL
1233− case r of
1234− Partial sR ->
1356+ rR <- initialR
1357+ case rR of
1358+ Partial sR -> do
1359+ rL <- initialL
12351360 case rL of
1236− Done _ -> Done <$> extractR sR
1361+ Done _ -> Done <$> finalR sR
12371362 Partial sL -> return $ Partial (sL, sR)
12381363 Done b -> return $ Done b
12391364
1365+ -- XXX should use Tuple'
12401366 step (sL, sR) x = runStep (stepL sL x) sR
12411367
12421368 extract = extractR . snd
12431369
1370+ final (sL, sR) = finalL sL *> finalR sR
1371+
12441372------------------------------------------------------------------------------
12451373-- Filtering
12461374------------------------------------------------------------------------------
@@ -1253,7 +1381,7 @@ postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
12531381--
12541382{-# INLINE_NORMAL catMaybes #-}
12551383catMaybes :: Monad m => Fold m a b -> Fold m (Maybe a) b
1256−catMaybes (Fold step initial extract) = Fold step1 initial extract
1384+catMaybes (Fold step initial extract final) = Fold step1 initial extract final
12571385
12581386 where
12591387
@@ -1293,7 +1421,7 @@ filtering f = foldl' step Nothing
12931421{-# INLINE filter #-}
12941422filter :: Monad m => (a -> Bool) -> Fold m a r -> Fold m a r
12951423-- filter p = scanMaybe (filtering p)
1296−filter f (Fold step begin done) = Fold step' begin done
1424+filter f (Fold step begin extract final) = Fold step' begin extract final
12971425 where
12981426 step' x a = if f a then step x a else return $ Partial x
12991427
@@ -1304,7 +1432,7 @@ filter f (Fold step begin done) = Fold step' begin done
13041432--
13051433{-# INLINE filterM #-}
13061434filterM :: Monad m => (a -> m Bool) -> Fold m a r -> Fold m a r
1307−filterM f (Fold step begin done) = Fold step' begin done
1435+filterM f (Fold step begin extract final) = Fold step' begin extract final
13081436 where
13091437 step' x a = do
13101438 use <- f a
@@ -1393,7 +1521,7 @@ dropping n = foldt' step initial extract
13931521{-# INLINE take #-}
13941522take :: Monad m => Int -> Fold m a b -> Fold m a b
13951523-- take n = scanMaybe (taking n)
1396−take n (Fold fstep finitial fextract) = Fold step initial extract
1524+take n (Fold fstep finitial fextract ffinal) = Fold step initial extract final
13971525
13981526 where
13991527
@@ -1405,7 +1533,7 @@ take n (Fold fstep finitial fextract) = Fold step initial extract
14051533 s1 = Tuple'Fused i1 s
14061534 if i1 < n
14071535 then return $ Partial s1
1408− else Done <$> fextract s
1536+ else Done <$> ffinal s
14091537 Done b -> return $ Done b
14101538
14111539 initial = finitial >>= next (-1)
@@ -1414,6 +1542,72 @@ take n (Fold fstep finitial fextract) = Fold step initial extract
14141542
14151543 extract (Tuple'Fused _ r) = fextract r
14161544
1545+ final (Tuple'Fused _ r) = ffinal r
1546+
1547+-- Note: Keep this consistent with S.splitOn. In fact we should eliminate
1548+-- S.splitOn in favor of the fold.
1549+--
1550+-- XXX Use Fold.many instead once it is fixed.
1551+-- > Stream.splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
1552+
1553+-- | Like 'takeEndBy' but drops the element on which the predicate succeeds.
1554+--
1555+-- Example:
1556+--
1557+-- >>> input = Stream.fromList "hello\nthere\n"
1558+-- >>> line = Fold.takeEndBy_ (== '\n') Fold.toList
1559+-- >>> Stream.fold line input
1560+-- "hello"
1561+--
1562+-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1563+-- ["hello","there"]
1564+--
1565+{-# INLINE takeEndBy_ #-}
1566+takeEndBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1567+-- takeEndBy_ predicate = scanMaybe (takingEndBy_ predicate)
1568+takeEndBy_ predicate (Fold fstep finitial fextract ffinal) =
1569+ Fold step finitial fextract ffinal
1570+
1571+ where
1572+
1573+ step s a =
1574+ if not (predicate a)
1575+ then fstep s a
1576+ else Done <$> ffinal s
1577+
1578+-- Note:
1579+-- > Stream.splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
1580+
1581+-- | Take the input, stop when the predicate succeeds taking the succeeding
1582+-- element as well.
1583+--
1584+-- Example:
1585+--
1586+-- >>> input = Stream.fromList "hello\nthere\n"
1587+-- >>> line = Fold.takeEndBy (== '\n') Fold.toList
1588+-- >>> Stream.fold line input
1589+-- "hello\n"
1590+--
1591+-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1592+-- ["hello\n","there\n"]
1593+--
1594+{-# INLINE takeEndBy #-}
1595+takeEndBy :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1596+-- takeEndBy predicate = scanMaybe (takingEndBy predicate)
1597+takeEndBy predicate (Fold fstep finitial fextract ffinal) =
1598+ Fold step finitial fextract ffinal
1599+
1600+ where
1601+
1602+ step s a = do
1603+ res <- fstep s a
1604+ if not (predicate a)
1605+ then return res
1606+ else do
1607+ case res of
1608+ Partial s1 -> Done <$> ffinal s1
1609+ Done b -> return $ Done b
1610+
14171611------------------------------------------------------------------------------
14181612-- Nesting
14191613------------------------------------------------------------------------------
@@ -1431,8 +1625,8 @@ take n (Fold fstep finitial fextract) = Fold step initial extract
14311625-- /Pre-release/
14321626{-# INLINE duplicate #-}
14331627duplicate :: Monad m => Fold m a b -> Fold m a (Fold m a b)
1434−duplicate (Fold step1 initial1 extract1) =
1435− Fold step initial (\s -> pure $ Fold step1 (pure $ Partial s) extract1)
1628+duplicate (Fold step1 initial1 extract1 final1) =
1629+ Fold step initial extract final
14361630
14371631 where
14381632
@@ -1440,6 +1634,11 @@ duplicate (Fold step1 initial1 extract1) =
14401634
14411635 step s a = second fromPure <$> step1 s a
14421636
1637+ -- Scanning may be problematic due to multiple finalizations.
1638+ extract = error "duplicate: scanning may be problematic"
1639+
1640+ final s = pure $ Fold step1 (pure $ Partial s) extract1 final1
1641+
14431642-- If there were a finalize/flushing action in the stream type that would be
14441643-- equivalent to running initialize in Fold. But we do not have a flushing
14451644-- action in streams.
@@ -1451,9 +1650,9 @@ duplicate (Fold step1 initial1 extract1) =
14511650-- /Pre-release/
14521651{-# INLINE reduce #-}
14531652reduce :: Monad m => Fold m a b -> m (Fold m a b)
1454−reduce (Fold step initial extract) = do
1653+reduce (Fold step initial extract final) = do
14551654 i <- initial
1456− return $ Fold step (return i) extract
1655+ return $ Fold step (return i) extract final
14571656
14581657-- This is the dual of Stream @cons@.
14591658
@@ -1463,7 +1662,8 @@ reduce (Fold step initial extract) = do
14631662-- /Pre-release/
14641663{-# INLINE snoclM #-}
14651664snoclM :: Monad m => Fold m a b -> m a -> Fold m a b
1466−snoclM (Fold fstep finitial fextract) action = Fold fstep initial fextract
1665+snoclM (Fold fstep finitial fextract ffinal) action =
1666+ Fold fstep initial fextract ffinal
14671667
14681668 where
14691669
@@ -1490,7 +1690,8 @@ snoclM (Fold fstep finitial fextract) action = Fold fstep initial fextract
14901690{-# INLINE snocl #-}
14911691snocl :: Monad m => Fold m a b -> a -> Fold m a b
14921692-- snocl f = snoclM f . return
1493−snocl (Fold fstep finitial fextract) a = Fold fstep initial fextract
1693+snocl (Fold fstep finitial fextract ffinal) a =
1694+ Fold fstep initial fextract ffinal
14941695
14951696 where
14961697
@@ -1510,12 +1711,12 @@ snocl (Fold fstep finitial fextract) a = Fold fstep initial fextract
15101711-- /Pre-release/
15111712{-# INLINE snocM #-}
15121713snocM :: Monad m => Fold m a b -> m a -> m (Fold m a b)
1513−snocM (Fold step initial extract) action = do
1714+snocM (Fold step initial extract final) action = do
15141715 res <- initial
15151716 r <- case res of
15161717 Partial fs -> action >>= step fs
15171718 Done _ -> return res
1518− return $ Fold step (return r) extract
1719+ return $ Fold step (return r) extract final
15191720
15201721-- Definitions:
15211722--
@@ -1534,12 +1735,12 @@ snocM (Fold step initial extract) action = do
15341735-- /Pre-release/
15351736{-# INLINE snoc #-}
15361737snoc :: Monad m => Fold m a b -> a -> m (Fold m a b)
1537−snoc (Fold step initial extract) a = do
1738+snoc (Fold step initial extract final) a = do
15381739 res <- initial
15391740 r <- case res of
15401741 Partial fs -> step fs a
15411742 Done _ -> return res
1542− return $ Fold step (return r) extract
1743+ return $ Fold step (return r) extract final
15431744
15441745-- | Append a singleton value to the fold.
15451746--
@@ -1567,7 +1768,7 @@ addOne = flip snoc
15671768-- /Pre-release/
15681769{-# INLINE extractM #-}
15691770extractM :: Monad m => Fold m a b -> m b
1570−extractM (Fold _ initial extract) = do
1771+extractM (Fold _ initial extract _) = do
15711772 res <- initial
15721773 case res of
15731774 Partial fs -> extract fs
@@ -1576,14 +1777,15 @@ extractM (Fold _ initial extract) = do
15761777-- | Close a fold so that it does not accept any more input.
15771778{-# INLINE close #-}
15781779close :: Monad m => Fold m a b -> Fold m a b
1579−close (Fold _ initial1 extract1) = Fold undefined initial undefined
1780+close (Fold _ initial1 _ final1) =
1781+ Fold undefined initial undefined undefined
15801782
15811783 where
15821784
15831785 initial = do
15841786 res <- initial1
15851787 case res of
1586− Partial s -> Done <$> extract1 s
1788+ Partial s -> Done <$> final1 s
15871789 Done b -> return $ Done b
15881790
15891791-- Corresponds to the null check for streams.
@@ -1593,7 +1795,7 @@ close (Fold _ initial1 extract1) = Fold undefined initial undefined
15931795-- /Pre-release/
15941796{-# INLINE isClosed #-}
15951797isClosed :: Monad m => Fold m a b -> m Bool
1596−isClosed (Fold _ initial _) = do
1798+isClosed (Fold _ initial _ _) = do
15971799 res <- initial
15981800 return $ case res of
15991801 Partial _ -> False
@@ -1627,8 +1829,10 @@ data ManyState s1 s2
16271829--
16281830{-# INLINE many #-}
16291831many :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1630−many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
1631− Fold step initial extract
1832+many
1833+ (Fold sstep sinitial sextract sfinal)
1834+ (Fold cstep cinitial cextract cfinal) =
1835+ Fold step initial extract final
16321836
16331837 where
16341838
@@ -1677,6 +1881,13 @@ many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
16771881 Partial s -> cextract s
16781882 Done b -> return b
16791883
1884+ final (ManyFirst ss cs) = sfinal ss *> cfinal cs
1885+ final (ManyLoop ss cs) = do
1886+ cres <- sfinal ss >>= cstep cs
1887+ case cres of
1888+ Partial s -> cfinal s
1889+ Done b -> return b
1890+
16801891-- | Like many, but the "first" fold emits an output at the end even if no
16811892-- input is received.
16821893--
@@ -1686,8 +1897,10 @@ many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
16861897--
16871898{-# INLINE manyPost #-}
16881899manyPost :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1689−manyPost (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
1690− Fold step initial extract
1900+manyPost
1901+ (Fold sstep sinitial sextract sfinal)
1902+ (Fold cstep cinitial cextract cfinal) =
1903+ Fold step initial extract final
16911904
16921905 where
16931906
@@ -1723,6 +1936,12 @@ manyPost (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
17231936 Partial s -> cextract s
17241937 Done b -> return b
17251938
1939+ final (Tuple' ss cs) = do
1940+ cres <- sfinal ss >>= cstep cs
1941+ case cres of
1942+ Partial s -> cfinal s
1943+ Done b -> return b
1944+
17261945-- | @groupsOf n split collect@ repeatedly applies the @split@ fold to chunks
17271946-- of @n@ items in the input stream and supplies the result to the @collect@
17281947-- fold.
@@ -1751,7 +1970,10 @@ groupsOf n split = many (take n split)
17511970--
17521971{-# INLINE refoldMany #-}
17531972refoldMany :: Monad m => Fold m a b -> Refold m x b c -> Refold m x a c
1754−refoldMany (Fold sstep sinitial sextract) (Refold cstep cinject cextract) =
1973+refoldMany
1974+ (Fold sstep sinitial sextract _sfinal)
1975+ -- XXX We will need a "final" in refold as well
1976+ (Refold cstep cinject cextract) =
17551977 Refold step inject extract
17561978
17571979 where
@@ -1802,7 +2024,9 @@ data ConsumeManyState x cs ss = ConsumeMany x cs (Either ss ss)
18022024-- /Internal/
18032025{-# INLINE refoldMany1 #-}
18042026refoldMany1 :: Monad m => Refold m x a b -> Fold m b c -> Refold m x a c
1805−refoldMany1 (Refold sstep sinject sextract) (Fold cstep cinitial cextract) =
2027+refoldMany1
2028+ (Refold sstep sinject sextract)
2029+ (Fold cstep cinitial cextract _cfinal) =
18062030 Refold step inject extract
18072031
18082032 where
@@ -1853,7 +2077,7 @@ refoldMany1 (Refold sstep sinject sextract) (Fold cstep cinitial cextract) =
18532077{-# INLINE refold #-}
18542078refold :: Monad m => Refold m b a c -> Fold m a b -> Fold m a c
18552079refold (Refold step inject extract) f =
1856− Fold step (extractM f >>= inject) extract
2080+ Fold step (extractM f >>= inject) extract extract
18572081
18582082------------------------------------------------------------------------------
18592083-- morphInner
@@ -1863,8 +2087,8 @@ refold (Refold step inject extract) f =
18632087--
18642088-- /Pre-release/
18652089morphInner :: (forall x. m x -> n x) -> Fold m a b -> Fold n a b
1866−morphInner f (Fold step initial extract) =
1867− Fold (\x a -> f $ step x a) (f initial) (f . extract)
2090+morphInner f (Fold step initial extract final) =
2091+ Fold (\x a -> f $ step x a) (f initial) (f . extract) (f . final)
18682092
18692093-- | Adapt a pure fold to any monad.
18702094--
core/src/Streamly/Internal/Data/Fold/Window.hs+57−51
@@ -36,24 +36,24 @@ module Streamly.Internal.Data.Fold.Window
3636 -- window folds by keeping the second element of the input tuple as
3737 -- @Nothing@.
3838 --
39− lmap
39+ windowLmap
4040 , cumulative
4141
42− , rollingMap
43− , rollingMapM
42+ , windowRollingMap
43+ , windowRollingMapM
4444
4545 -- ** Sums
46− , length
47− , sum
48− , sumInt
49− , powerSum
50− , powerSumFrac
46+ , windowLength
47+ , windowSum
48+ , windowSumInt
49+ , windowPowerSum
50+ , windowPowerSumFrac
5151
5252 -- ** Location
53− , minimum
54− , maximum
55− , range
56− , mean
53+ , windowMinimum
54+ , windowMaximum
55+ , windowRange
56+ , windowMean
5757 )
5858where
5959
@@ -65,6 +65,8 @@ import Streamly.Internal.Data.Fold.Type (Fold(..), Step(..))
6565import Streamly.Internal.Data.Tuple.Strict
6666 (Tuple'(..), Tuple3Fused' (Tuple3Fused'))
6767
68+import Foreign.ForeignPtr.Unsafe (unsafeForeignPtrToPtr)
69+
6870import qualified Streamly.Internal.Data.Fold.Type as Fold
6971import qualified Streamly.Internal.Data.Ring as Ring
7072
@@ -73,7 +75,7 @@ import Prelude hiding (length, sum, minimum, maximum)
7375-- $setup
7476-- >>> import Data.Bifunctor(bimap)
7577-- >>> import qualified Streamly.Data.Fold as Fold
76−-- >>> import qualified Streamly.Internal.Data.Fold.Window as FoldW
78+-- >>> import qualified Streamly.Internal.Data.Fold.Window as Fold
7779-- >>> import qualified Streamly.Internal.Data.Ring as Ring
7880-- >>> import qualified Streamly.Data.Stream as Stream
7981-- >>> import Prelude hiding (length, sum, minimum, maximum)
@@ -87,9 +89,9 @@ import Prelude hiding (length, sum, minimum, maximum)
8789--
8890-- >>> lmap f = Fold.lmap (bimap f (f <$>))
8991--
90−{-# INLINE lmap #-}
91−lmap :: (c -> a) -> Fold m (a, Maybe a) b -> Fold m (c, Maybe c) b
92−lmap f = Fold.lmap (bimap f (f <$>))
92+{-# INLINE windowLmap #-}
93+windowLmap :: (c -> a) -> Fold m (a, Maybe a) b -> Fold m (c, Maybe c) b
94+windowLmap f = Fold.lmap (bimap f (f <$>))
9395
9496-- | Convert an incremental fold to a cumulative fold using the entire input
9597-- stream as a single window.
@@ -105,10 +107,10 @@ cumulative = Fold.lmap (, Nothing)
105107
106108-- | Apply an effectful function on the latest and the oldest element of the
107109-- window.
108−{-# INLINE rollingMapM #-}
109−rollingMapM :: Monad m =>
110+{-# INLINE windowRollingMapM #-}
111+windowRollingMapM :: Monad m =>
110112 (Maybe a -> a -> m (Maybe b)) -> Fold m (a, Maybe a) (Maybe b)
111−rollingMapM f = Fold.foldlM' f1 initial
113+windowRollingMapM f = Fold.foldlM' f1 initial
112114
113115 where
114116
@@ -118,12 +120,12 @@ rollingMapM f = Fold.foldlM' f1 initial
118120
119121-- | Apply a pure function on the latest and the oldest element of the window.
120122--
121−-- >>> rollingMap f = FoldW.rollingMapM (\x y -> return $ f x y)
123+-- >>> windowRollingMap f = Fold.windowRollingMapM (\x y -> return $ f x y)
122124--
123−{-# INLINE rollingMap #-}
124−rollingMap :: Monad m =>
125+{-# INLINE windowRollingMap #-}
126+windowRollingMap :: Monad m =>
125127 (Maybe a -> a -> Maybe b) -> Fold m (a, Maybe a) (Maybe b)
126−rollingMap f = Fold.foldl' f1 initial
128+windowRollingMap f = Fold.foldl' f1 initial
127129
128130 where
129131
@@ -146,9 +148,9 @@ rollingMap f = Fold.foldl' f1 initial
146148--
147149-- /Internal/
148150--
149−{-# INLINE sumInt #-}
150−sumInt :: forall m a. (Monad m, Integral a) => Fold m (a, Maybe a) a
151−sumInt = Fold step initial extract
151+{-# INLINE windowSumInt #-}
152+windowSumInt :: forall m a. (Monad m, Integral a) => Fold m (a, Maybe a) a
153+windowSumInt = Fold step initial extract extract
152154
153155 where
154156
@@ -180,9 +182,9 @@ sumInt = Fold step initial extract
180182--
181183-- /Time/: \(\mathcal{O}(n)\)
182184--
183−{-# INLINE sum #-}
184−sum :: forall m a. (Monad m, Num a) => Fold m (a, Maybe a) a
185−sum = Fold step initial extract
185+{-# INLINE windowSum #-}
186+windowSum :: forall m a. (Monad m, Num a) => Fold m (a, Maybe a) a
187+windowSum = Fold step initial extract extract
186188
187189 where
188190
@@ -219,9 +221,9 @@ sum = Fold step initial extract
219221--
220222-- >>> length = powerSum 0
221223--
222−{-# INLINE length #-}
223−length :: (Monad m, Num b) => Fold m (a, Maybe a) b
224−length = Fold.foldl' step 0
224+{-# INLINE windowLength #-}
225+windowLength :: (Monad m, Num b) => Fold m (a, Maybe a) b
226+windowLength = Fold.foldl' step 0
225227
226228 where
227229
@@ -237,18 +239,18 @@ length = Fold.foldl' step 0
237239-- /Space/: \(\mathcal{O}(1)\)
238240--
239241-- /Time/: \(\mathcal{O}(n)\)
240−{-# INLINE powerSum #-}
241−powerSum :: (Monad m, Num a) => Int -> Fold m (a, Maybe a) a
242−powerSum k = lmap (^ k) sum
242+{-# INLINE windowPowerSum #-}
243+windowPowerSum :: (Monad m, Num a) => Int -> Fold m (a, Maybe a) a
244+windowPowerSum k = windowLmap (^ k) windowSum
243245
244246-- | Like 'powerSum' but powers can be negative or fractional. This is slower
245247-- than 'powerSum' for positive intergal powers.
246248--
247249-- >>> powerSumFrac p = lmap (** p) sum
248250--
249−{-# INLINE powerSumFrac #-}
250−powerSumFrac :: (Monad m, Floating a) => a -> Fold m (a, Maybe a) a
251−powerSumFrac p = lmap (** p) sum
251+{-# INLINE windowPowerSumFrac #-}
252+windowPowerSumFrac :: (Monad m, Floating a) => a -> Fold m (a, Maybe a) a
253+windowPowerSumFrac p = windowLmap (** p) windowSum
252254
253255-------------------------------------------------------------------------------
254256-- Location
@@ -265,9 +267,9 @@ powerSumFrac p = lmap (** p) sum
265267--
266268-- /Time/: \(\mathcal{O}(n*w)\) where \(w\) is the window size.
267269--
268−{-# INLINE range #-}
269−range :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe (a, a))
270−range n = Fold step initial extract
270+{-# INLINE windowRange #-}
271+windowRange :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe (a, a))
272+windowRange n = Fold step initial extract extract
271273
272274 where
273275
@@ -300,7 +302,11 @@ range n = Fold step initial extract
300302 if i == 0
301303 then return Nothing
302304 else do
303− x <- liftIO $ peek rh
305+ -- Here we use "ringStart" over "ringHead" as "ringHead" will be
306+ -- uninitialized if the ring is not full.
307+ -- Using "unsafeForeignPtrToPtr" here is safe as we touch the ring
308+ -- again in "foldFunc".
309+ x <- liftIO $ peek (unsafeForeignPtrToPtr (Ring.ringStart rb))
304310 let accum (mn, mx) a = return (min mn a, max mx a)
305311 fmap Just $ foldFunc i rh accum (x, x) rb
306312
@@ -316,9 +322,9 @@ range n = Fold step initial extract
316322--
317323-- /Time/: \(\mathcal{O}(n*w)\) where \(w\) is the window size.
318324--
319−{-# INLINE minimum #-}
320−minimum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
321−minimum n = fmap (fmap fst) $ range n
325+{-# INLINE windowMinimum #-}
326+windowMinimum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
327+windowMinimum n = fmap (fmap fst) $ windowRange n
322328
323329-- | The maximum element in a rolling window.
324330--
@@ -329,9 +335,9 @@ minimum n = fmap (fmap fst) $ range n
329335--
330336-- /Time/: \(\mathcal{O}(n*w)\) where \(w\) is the window size.
331337--
332−{-# INLINE maximum #-}
333−maximum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
334−maximum n = fmap (fmap snd) $ range n
338+{-# INLINE windowMaximum #-}
339+windowMaximum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
340+windowMaximum n = fmap (fmap snd) $ windowRange n
335341
336342-- | Arithmetic mean of elements in a sliding window:
337343--
@@ -346,6 +352,6 @@ maximum n = fmap (fmap snd) $ range n
346352-- /Space/: \(\mathcal{O}(1)\)
347353--
348354-- /Time/: \(\mathcal{O}(n)\)
349−{-# INLINE mean #-}
350−mean :: forall m a. (Monad m, Fractional a) => Fold m (a, Maybe a) a
351−mean = Fold.teeWith (/) sum length
355+{-# INLINE windowMean #-}
356+windowMean :: forall m a. (Monad m, Fractional a) => Fold m (a, Maybe a) a
357+windowMean = Fold.teeWith (/) windowSum windowLength
core/src/Streamly/Internal/Data/IORef/Unboxed.hs+23−20
@@ -23,34 +23,31 @@ module Streamly.Internal.Data.IORef.Unboxed
2323 (
2424 IORef
2525
26− -- * Construction
26+ -- Construction
2727 , newIORef
2828
29− -- * Write
29+ -- Write
3030 , writeIORef
3131 , modifyIORef'
3232
33− -- * Read
33+ -- Read
3434 , readIORef
35− , toStreamD
35+ , pollIntIORef
3636 )
3737where
3838
3939#include "inline.hs"
4040
41−import Data.Proxy (Proxy(..))
4241import Control.Monad.IO.Class (MonadIO(..))
43−import Streamly.Internal.Data.Unbox
44− ( MutableByteArray(..)
45− , Unbox(..)
46− , newBytes
47− , sizeOf
48− )
42+import Data.Proxy (Proxy(..))
43+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
44+import Streamly.Internal.Data.Unbox (Unbox(..), sizeOf)
4945
50−import qualified Streamly.Internal.Data.Stream.StreamD.Type as D
46+import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
47+import qualified Streamly.Internal.Data.Stream.Type as D
5148
5249-- | An 'IORef' holds a single 'Unbox'-able value.
53−newtype IORef a = IORef MutableByteArray
50+newtype IORef a = IORef MutByteArray
5451
5552-- | Create a new 'IORef'.
5653--
@@ -58,8 +55,8 @@ newtype IORef a = IORef MutableByteArray
5855{-# INLINE newIORef #-}
5956newIORef :: forall a. Unbox a => a -> IO (IORef a)
6057newIORef x = do
61− var <- newBytes (sizeOf (Proxy :: Proxy a))
62− pokeByteIndex 0 var x
58+ var <- MBA.new (sizeOf (Proxy :: Proxy a))
59+ pokeAt 0 var x
6360 return $ IORef var
6461
6562-- | Write a value to an 'IORef'.
@@ -67,14 +64,14 @@ newIORef x = do
6764-- /Pre-release/
6865{-# INLINE writeIORef #-}
6966writeIORef :: Unbox a => IORef a -> a -> IO ()
70−writeIORef (IORef var) = pokeByteIndex 0 var
67+writeIORef (IORef var) = pokeAt 0 var
7168
7269-- | Read a value from an 'IORef'.
7370--
7471-- /Pre-release/
7572{-# INLINE readIORef #-}
7673readIORef :: Unbox a => IORef a -> IO a
77−readIORef (IORef var) = peekByteIndex 0 var
74+readIORef (IORef var) = peekAt 0 var
7875
7976-- | Modify the value of an 'IORef' using a function with strict application.
8077--
@@ -87,10 +84,16 @@ modifyIORef' var g = do
8784
8885-- | Generate a stream by continuously reading the IORef.
8986--
87+-- This operation reads the IORef without any synchronization. It can be
88+-- assumed to be atomic because the IORef (MutableByteArray) is always aligned
89+-- to Int boundaries, we are assuming that compiler uses single instructions to
90+-- access the memory. It may read stale values though until caches are
91+-- synchronised in a multiprocessor architecture.
92+--
9093-- /Pre-release/
91−{-# INLINE_NORMAL toStreamD #-}
92−toStreamD :: (MonadIO m, Unbox a) => IORef a -> D.Stream m a
93−toStreamD var = D.Stream step ()
94+{-# INLINE_NORMAL pollIntIORef #-}
95+pollIntIORef :: (MonadIO m, Unbox a) => IORef a -> D.Stream m a
96+pollIntIORef var = D.Stream step ()
9497
9598 where
9699
core/src/Streamly/Internal/Data/IsMap.hs+8
@@ -1,3 +1,7 @@
1+{-# LANGUAGE TypeFamilies #-}
2+-- Must come after TypeFamilies, otherwise it is re-enabled.
3+-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
4+{-# LANGUAGE NoMonoLocalBinds #-}
15-- |
26-- Module : Streamly.Internal.Data.IsMap
37-- Copyright : (c) 2022 Composewell Technologies
@@ -28,6 +32,8 @@ class IsMap f where
2832 mapDelete :: Key f -> f a -> f a
2933 mapUnion :: f a -> f a -> f a
3034 mapNull :: f a -> Bool
35+ mapTraverseWithKey ::
36+ Applicative t => (Key f -> a -> t b) -> f a -> t (f b)
3137
3238instance Ord k => IsMap (Map k) where
3339 type Key (Map k) = k
@@ -39,6 +45,7 @@ instance Ord k => IsMap (Map k) where
3945 mapDelete = Map.delete
4046 mapUnion = Map.union
4147 mapNull = Map.null
48+ mapTraverseWithKey = Map.traverseWithKey
4249
4350instance IsMap IntMap.IntMap where
4451 type Key IntMap.IntMap = Int
@@ -50,3 +57,4 @@ instance IsMap IntMap.IntMap where
5057 mapDelete = IntMap.delete
5158 mapUnion = IntMap.union
5259 mapNull = IntMap.null
60+ mapTraverseWithKey = IntMap.traverseWithKey
core/src/Streamly/Internal/Data/List.hs+5−5
@@ -66,19 +66,19 @@ where
6666import Control.Arrow (second)
6767import Data.Functor.Identity (Identity, runIdentity)
6868import GHC.Exts (IsList(..), IsString(..))
69−import Streamly.Internal.Data.Stream.StreamD.Type
69+import Streamly.Internal.Data.Stream.Type
7070 ( CrossStream
7171 , mkCross
7272 , unCross
7373 )
74−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
74+import Streamly.Internal.Data.Stream.Type (Stream)
7575import Text.Read
7676 ( Lexeme(Ident), lexP, parens, prec, readPrec, readListPrec
7777 , readListPrecDefault)
7878
79−import qualified Streamly.Internal.Data.Stream.StreamK.Type as K
80−import qualified Streamly.Internal.Data.Stream.StreamD.Type as Stream
81−import qualified Streamly.Internal.Data.Stream.StreamD.Generate as Stream
79+import qualified Streamly.Internal.Data.StreamK.Type as K
80+import qualified Streamly.Internal.Data.Stream.Type as Stream
81+import qualified Streamly.Internal.Data.Stream.Generate as Stream
8282
8383-- XXX Rename to PureStream.
8484
core/src/Streamly/Internal/Data/MutArray.hs+154−33
@@ -8,79 +8,200 @@
88--
99module Streamly.Internal.Data.MutArray
1010 (
11+ -- * MutArray.Type module
1112 module Streamly.Internal.Data.MutArray.Type
12− , splitOn
13+ -- * MutArray module
14+ , sliceIndexerFromLen
15+ , slicerFromLen
16+ , compactLE
17+ , pinnedCompactLE
18+ , compactOnByte
19+ , compactOnByteSuffix
20+ -- * Unboxed IORef
21+ , module Streamly.Internal.Data.IORef.Unboxed
22+
23+ -- * Deprecated
1324 , genSlicesFromLen
1425 , getSlicesFromLen
15− , fromStream
1626 )
1727where
1828
1929#include "inline.hs"
2030
2131import Control.Monad.IO.Class (MonadIO(..))
32+import Data.Word (Word8)
33+import Streamly.Internal.Data.MutByteArray.Type (PinnedState(..))
34+import Streamly.Internal.Data.Stream.Type (Stream)
2235import Streamly.Internal.Data.Unbox (Unbox)
23−import Streamly.Internal.Data.Stream (Stream)
2436import Streamly.Internal.Data.Unfold.Type (Unfold(..))
2537
26−import qualified Streamly.Internal.Data.Stream as D
38+import qualified Streamly.Internal.Data.Stream.Type as Stream
39+import qualified Streamly.Internal.Data.Stream.Nesting as Stream
40+-- import qualified Streamly.Internal.Data.Stream.Transform as Stream
2741import qualified Streamly.Internal.Data.Unfold as Unfold
2842
2943import Prelude hiding (foldr, length, read, splitAt)
3044import Streamly.Internal.Data.MutArray.Type
31−
32−-- | Split the array into a stream of slices using a predicate. The element
33−-- matching the predicate is dropped.
34−--
35−-- /Pre-release/
36−{-# INLINE splitOn #-}
37−splitOn :: (MonadIO m, Unbox a) =>
38− (a -> Bool) -> MutArray a -> Stream m (MutArray a)
39−splitOn predicate arr =
40− fmap (\(i, len) -> getSliceUnsafe i len arr)
41− $ D.sliceOnSuffix predicate (read arr)
45+import Streamly.Internal.Data.IORef.Unboxed
4246
4347-- | Generate a stream of array slice descriptors ((index, len)) of specified
4448-- length from an array, starting from the supplied array index. The last slice
4549-- may be shorter than the requested length depending on the array length.
4650--
4751-- /Pre-release/
48−{-# INLINE genSlicesFromLen #-}
49−genSlicesFromLen :: forall m a. (Monad m, Unbox a)
52+{-# INLINE sliceIndexerFromLen #-}
53+sliceIndexerFromLen :: forall m a. (Monad m, Unbox a)
5054 => Int -- ^ from index
5155 -> Int -- ^ length of the slice
5256 -> Unfold m (MutArray a) (Int, Int)
53−genSlicesFromLen from len =
57+sliceIndexerFromLen from len =
5458 let fromThenTo n = (from, from + len, n - 1)
5559 mkSlice n i = return (i, min len (n - i))
5660 in Unfold.lmap length
5761 $ Unfold.mapM2 mkSlice
5862 $ Unfold.lmap fromThenTo Unfold.enumerateFromThenTo
5963
64+{-# DEPRECATED genSlicesFromLen "Please use sliceIndexerFromLen instead." #-}
65+genSlicesFromLen :: forall m a. (Monad m, Unbox a)
66+ => Int -- ^ from index
67+ -> Int -- ^ length of the slice
68+ -> Unfold m (MutArray a) (Int, Int)
69+genSlicesFromLen = sliceIndexerFromLen
70+
6071-- | Generate a stream of slices of specified length from an array, starting
6172-- from the supplied array index. The last slice may be shorter than the
6273-- requested length depending on the array length.
6374--
6475-- /Pre-release/
65−{-# INLINE getSlicesFromLen #-}
66−getSlicesFromLen :: forall m a. (Monad m, Unbox a)
76+{-# INLINE slicerFromLen #-}
77+slicerFromLen :: forall m a. (Monad m, Unbox a)
6778 => Int -- ^ from index
6879 -> Int -- ^ length of the slice
6980 -> Unfold m (MutArray a) (MutArray a)
70−getSlicesFromLen from len =
81+slicerFromLen from len =
7182 let mkSlice arr (i, n) = return $ getSliceUnsafe i n arr
72− in Unfold.mapM2 mkSlice (genSlicesFromLen from len)
83+ in Unfold.mapM2 mkSlice (sliceIndexerFromLen from len)
84+
85+{-# DEPRECATED getSlicesFromLen "Please use slicerFromLen instead." #-}
86+getSlicesFromLen :: forall m a. (Monad m, Unbox a)
87+ => Int -- ^ from index
88+ -> Int -- ^ length of the slice
89+ -> Unfold m (MutArray a) (MutArray a)
90+getSlicesFromLen = slicerFromLen
7391
74−-- | Create an 'Array' from a stream. This is useful when we want to create a
75−-- single array from a stream of unknown size. 'writeN' is at least twice
76−-- as efficient when the size is already known.
92+-------------------------------------------------------------------------------
93+-- Compacting Streams of Arrays
94+-------------------------------------------------------------------------------
95+
96+-- | @compactLE maxElems@ coalesces adjacent arrays in the input stream
97+-- only if the combined size would be less than or equal to @maxElems@
98+-- elements. Note that it won't split an array if the original array is already
99+-- larger than maxElems.
77100--
78−-- Note that if the input stream is too large memory allocation for the array
79−-- may fail. When the stream size is not known, `chunksOf` followed by
80−-- processing of indvidual arrays in the resulting stream should be preferred.
101+-- @maxElems@ must be greater than 0.
81102--
82−-- /Pre-release/
83−{-# INLINE fromStream #-}
84−fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
85−fromStream = fromStreamD
86−-- fromStream (Stream m) = P.fold write m
103+-- Generates unpinned arrays irrespective of the pinning status of input
104+-- arrays.
105+{-# INLINE compactLE #-}
106+compactLE :: (MonadIO m, Unbox a) =>
107+ Int -> Stream m (MutArray a) -> Stream m (MutArray a)
108+-- XXX compactLE can be moved to MutArray/Type if we are not using the parser
109+-- to implement it.
110+compactLE = compactLeAs Unpinned
111+-- The parser version turns out to be a little bit slower.
112+-- compactLE n = Stream.catRights . Stream.parseManyD (pCompactLE n)
113+
114+-- | Pinned version of 'compactLE'.
115+{-# INLINE pinnedCompactLE #-}
116+pinnedCompactLE :: forall m a. (MonadIO m, Unbox a)
117+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
118+pinnedCompactLE = compactLeAs Pinned
119+-- pinnedCompactLE n = Stream.catRights . Stream.parseManyD (pPinnedCompactLE n)
120+
121+data SplitState s arr
122+ = Initial s
123+ | Buffering s arr
124+ | Splitting s arr
125+ | Yielding arr (SplitState s arr)
126+ | Finishing
127+
128+-- | Split a stream of arrays on a given separator byte, dropping the separator
129+-- and coalescing all the arrays between two separators into a single array.
130+--
131+{-# INLINE_NORMAL _compactOnByteCustom #-}
132+_compactOnByteCustom
133+ :: MonadIO m
134+ => Word8
135+ -> Stream m (MutArray Word8)
136+ -> Stream m (MutArray Word8)
137+_compactOnByteCustom byte (Stream.Stream step state) =
138+ Stream.Stream step' (Initial state)
139+
140+ where
141+
142+ {-# INLINE_LATE step' #-}
143+ step' gst (Initial st) = do
144+ r <- step gst st
145+ case r of
146+ Stream.Yield arr s -> do
147+ (arr1, marr2) <- breakOn byte arr
148+ return $ case marr2 of
149+ Nothing -> Stream.Skip (Buffering s arr1)
150+ Just arr2 -> Stream.Skip (Yielding arr1 (Splitting s arr2))
151+ Stream.Skip s -> return $ Stream.Skip (Initial s)
152+ Stream.Stop -> return Stream.Stop
153+
154+ step' gst (Buffering st buf) = do
155+ r <- step gst st
156+ case r of
157+ Stream.Yield arr s -> do
158+ (arr1, marr2) <- breakOn byte arr
159+ -- XXX Use spliceExp instead and then rightSize?
160+ buf1 <- splice buf arr1
161+ return $ case marr2 of
162+ Nothing -> Stream.Skip (Buffering s buf1)
163+ Just x -> Stream.Skip (Yielding buf1 (Splitting s x))
164+ Stream.Skip s -> return $ Stream.Skip (Buffering s buf)
165+ Stream.Stop -> return $
166+ if byteLength buf == 0
167+ then Stream.Stop
168+ else Stream.Skip (Yielding buf Finishing)
169+
170+ step' _ (Splitting st buf) = do
171+ (arr1, marr2) <- breakOn byte buf
172+ return $ case marr2 of
173+ Nothing -> Stream.Skip $ Buffering st arr1
174+ Just arr2 -> Stream.Skip $ Yielding arr1 (Splitting st arr2)
175+
176+ step' _ (Yielding arr next) = return $ Stream.Yield arr next
177+ step' _ Finishing = return Stream.Stop
178+
179+-- XXX implement predicate based version of this
180+-- XXX Naming of predicate based vs custom version
181+
182+-- | Split a stream of arrays on a given separator byte, dropping the separator
183+-- and coalescing all the arrays between two separators into a single array.
184+--
185+{-# INLINE compactOnByte #-}
186+compactOnByte
187+ :: (MonadIO m)
188+ => Word8
189+ -> Stream m (MutArray Word8)
190+ -> Stream m (MutArray Word8)
191+-- XXX compare perf of custom vs idiomatic version
192+-- compactOnByte = _compactOnByteCustom
193+-- XXX use spliceExp and rightSize?
194+compactOnByte byte = Stream.splitInnerBy (breakOn byte) splice
195+
196+-- | Like 'compactOnByte' considers the separator in suffix position instead of
197+-- infix position.
198+{-# INLINE compactOnByteSuffix #-}
199+compactOnByteSuffix
200+ :: (MonadIO m)
201+ => Word8
202+ -> Stream m (MutArray Word8)
203+ -> Stream m (MutArray Word8)
204+compactOnByteSuffix byte =
205+ -- XXX use spliceExp and rightSize?
206+ Stream.splitInnerBySuffix
207+ (\arr -> byteLength arr == 0) (breakOn byte) splice
core/src/Streamly/Internal/Data/MutArray/Generic.hs+172−49
@@ -19,14 +19,14 @@ module Streamly.Internal.Data.MutArray.Generic
1919 , nil
2020
2121 -- *** Uninitialized Arrays
22− , new
22+ , emptyOf
2323 -- , newArrayWith
2424
2525 -- *** From streams
26− , writeNUnsafe
27− , writeN
28− , writeWith
29− , write
26+ , unsafeCreateOf
27+ , createOf
28+ , createWith -- createOfMin/createMin/createGE?
29+ , create
3030 , fromStreamN
3131 , fromStream
3232 , fromPureStream
@@ -101,6 +101,7 @@ module Streamly.Internal.Data.MutArray.Generic
101101 -- ** Random reads
102102 , getIndex
103103 , getIndexUnsafe
104+ , getIndexUnsafeWith
104105 -- , getIndices
105106 -- , getFromThenTo
106107 -- , getIndexRev
@@ -132,7 +133,7 @@ module Streamly.Internal.Data.MutArray.Generic
132133 -- multidimensional array representations.
133134
134135 -- ** Construct from streams
135− -- , chunksOf
136+ , chunksOf
136137 -- , arrayStreamKFromStreamD
137138 -- , writeChunks
138139
@@ -159,6 +160,13 @@ module Streamly.Internal.Data.MutArray.Generic
159160 -- , appendSliceFrom
160161
161162 , clone
163+
164+ -- * Deprecated
165+ , new
166+ , writeNUnsafe
167+ , writeN
168+ , writeWith
169+ , write
162170 )
163171where
164172
@@ -181,14 +189,15 @@ import GHC.Int (Int(..))
181189import Streamly.Internal.Data.Fold.Type (Fold(..))
182190import Streamly.Internal.Data.Producer.Type (Producer (..))
183191import Streamly.Internal.Data.Unfold.Type (Unfold(..))
184−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
192+import Streamly.Internal.Data.Stream.Type (Stream)
193+import Streamly.Internal.Data.SVar.Type (adaptState)
185194
186195import qualified Streamly.Internal.Data.Fold.Type as FL
187196import qualified Streamly.Internal.Data.Producer as Producer
188−import qualified Streamly.Internal.Data.Stream.StreamD.Type as D
189−import qualified Streamly.Internal.Data.Stream.StreamD.Generate as D
190−import qualified Streamly.Internal.Data.Stream.StreamD.Lift as D
191−import qualified Streamly.Internal.Data.Stream.StreamK.Type as K
197+import qualified Streamly.Internal.Data.Stream.Type as D
198+import qualified Streamly.Internal.Data.Stream.Generate as D
199+import qualified Streamly.Internal.Data.Stream.Lift as D
200+import qualified Streamly.Internal.Data.StreamK.Type as K
192201
193202import Prelude hiding (read, length)
194203
@@ -234,13 +243,13 @@ bottomElement =
234243-- XXX Would be nice if GHC can provide something like newUninitializedArray# so
235244-- that we do not have to write undefined or error in the whole array.
236245
237−-- | @new count@ allocates a zero length array that can be extended to hold
246+-- | @emptyOf count@ allocates a zero length array that can be extended to hold
238247-- up to 'count' items without reallocating.
239248--
240249-- /Pre-release/
241−{-# INLINE new #-}
242−new :: MonadIO m => Int -> m (MutArray a)
243−new n@(I# n#) =
250+{-# INLINE emptyOf #-}
251+emptyOf :: MonadIO m => Int -> m (MutArray a)
252+emptyOf n@(I# n#) =
244253 liftIO
245254 $ IO
246255 $ \s# ->
@@ -249,6 +258,12 @@ new n@(I# n#) =
249258 let ma = MutArray arr# 0 0 n
250259 in (# s1#, ma #)
251260
261+-- XXX Deprecate in major
262+-- {-# DEPRECATED new "Please use emptyOf instead." #-}
263+{-# INLINE new #-}
264+new :: MonadIO m => Int -> m (MutArray a)
265+new = emptyOf
266+
252267-- XXX This could be pure?
253268
254269-- |
@@ -263,6 +278,21 @@ nil = new 0
263278-- Random writes
264279-------------------------------------------------------------------------------
265280
281+-- | Write the given element to the given index of the 'MutableArray#'. Does not
282+-- check if the index is out of bounds of the array.
283+--
284+-- /Pre-release/
285+{-# INLINE putIndexUnsafeWith #-}
286+putIndexUnsafeWith :: MonadIO m => Int -> MutableArray# RealWorld a -> a -> m ()
287+putIndexUnsafeWith n _arrContents# x =
288+ liftIO
289+ $ IO
290+ $ \s# ->
291+ case n of
292+ I# n# ->
293+ let s1# = writeArray# _arrContents# n# x s#
294+ in (# s1#, () #)
295+
266296-- | Write the given element to the given index of the array. Does not check if
267297-- the index is out of bounds of the array.
268298--
@@ -271,13 +301,7 @@ nil = new 0
271301putIndexUnsafe :: forall m a. MonadIO m => Int -> MutArray a -> a -> m ()
272302putIndexUnsafe i MutArray {..} x =
273303 assert (i >= 0 && i < arrLen)
274− (liftIO
275− $ IO
276− $ \s# ->
277− case i + arrStart of
278− I# n# ->
279− let s1# = writeArray# arrContents# n# x s#
280− in (# s1#, () #))
304+ putIndexUnsafeWith (i + arrStart) arrContents# x
281305
282306invalidIndex :: String -> Int -> a
283307invalidIndex label i =
@@ -307,7 +331,7 @@ putIndices arr = FL.foldlM' step (return ())
307331
308332 where
309333
310− step () (i, x) = liftIO (putIndex i arr x)
334+ step () (i, x) = putIndex i arr x
311335
312336-- | Modify a given index of an array using a modifier function without checking
313337-- the bounds.
@@ -456,17 +480,25 @@ uninit arr@MutArray{..} len =
456480-- Random reads
457481-------------------------------------------------------------------------------
458482
459−-- | Return the element at the specified index without checking the bounds.
483+-- | Return the element at the specified index without checking the bounds from
484+-- a @MutableArray# RealWorld@.
460485--
461486-- Unsafe because it does not check the bounds of the array.
462−{-# INLINE_NORMAL getIndexUnsafe #-}
463−getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
464−getIndexUnsafe n MutArray {..} =
487+{-# INLINE getIndexUnsafeWith #-}
488+getIndexUnsafeWith :: MonadIO m => MutableArray# RealWorld a -> Int -> m a
489+getIndexUnsafeWith _arrContents# n =
465490 liftIO
466491 $ IO
467492 $ \s# ->
468− let !(I# i#) = arrStart + n
469− in readArray# arrContents# i# s#
493+ let !(I# i#) = n
494+ in readArray# _arrContents# i# s#
495+
496+-- | Return the element at the specified index without checking the bounds.
497+--
498+-- Unsafe because it does not check the bounds of the array.
499+{-# INLINE_NORMAL getIndexUnsafe #-}
500+getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
501+getIndexUnsafe n MutArray {..} = getIndexUnsafeWith arrContents# (n + arrStart)
470502
471503-- | /O(1)/ Lookup the element at the given index. Index starts from 0.
472504--
@@ -570,16 +602,16 @@ readRev arr@MutArray{..} =
570602arrayChunkSize :: Int
571603arrayChunkSize = 1024
572604
573−-- | Like 'writeN' but does not check the array bounds when writing. The fold
605+-- | Like 'createOf' but does not check the array bounds when writing. The fold
574606-- driver must not call the step function more than 'n' times otherwise it will
575607-- corrupt the memory and crash. This function exists mainly because any
576608-- conditional in the step function blocks fusion causing 10x performance
577609-- slowdown.
578610--
579611-- /Pre-release/
580−{-# INLINE_NORMAL writeNUnsafe #-}
581−writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
582−writeNUnsafe n = Fold step initial return
612+{-# INLINE_NORMAL unsafeCreateOf #-}
613+unsafeCreateOf :: MonadIO m => Int -> Fold m a (MutArray a)
614+unsafeCreateOf n = Fold step initial return return
583615
584616 where
585617
@@ -587,59 +619,81 @@ writeNUnsafe n = Fold step initial return
587619
588620 step arr x = FL.Partial <$> snocUnsafe arr x
589621
590−-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
622+{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
623+{-# INLINE writeNUnsafe #-}
624+writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
625+writeNUnsafe = unsafeCreateOf
626+
627+-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
591628-- 'Array'.
592629--
593−-- >>> writeN n = Fold.take n (MutArray.writeNUnsafe n)
630+-- >>> createOf n = Fold.take n (MutArray.unsafeCreateOf n)
594631--
595632-- /Pre-release/
596−{-# INLINE_NORMAL writeN #-}
633+{-# INLINE_NORMAL createOf #-}
634+createOf :: MonadIO m => Int -> Fold m a (MutArray a)
635+createOf n = FL.take n $ unsafeCreateOf n
636+
637+-- XXX Deprecate in major
638+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
639+{-# INLINE writeN #-}
597640writeN :: MonadIO m => Int -> Fold m a (MutArray a)
598−writeN n = FL.take n $ writeNUnsafe n
641+writeN = createOf
599642
600643-- >>> f n = MutArray.writeAppendWith (* 2) (MutArray.pinnedNew n)
601644-- >>> writeWith n = Fold.rmapM MutArray.rightSize (f n)
602645-- >>> writeWith n = Fold.rmapM MutArray.fromArrayStreamK (MutArray.writeChunks n)
603646
604−-- | @writeWith minCount@ folds the whole input to a single array. The array
647+-- | @createWith minCount@ folds the whole input to a single array. The array
605648-- starts at a size big enough to hold minCount elements, the size is doubled
606649-- every time the array needs to be grown.
607650--
608651-- /Caution! Do not use this on infinite streams./
609652--
610653-- /Pre-release/
611−{-# INLINE_NORMAL writeWith #-}
612−writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
654+{-# INLINE_NORMAL createWith #-}
655+createWith :: MonadIO m => Int -> Fold m a (MutArray a)
613656-- writeWith n = FL.rmapM rightSize $ writeAppendWith (* 2) (pinnedNew n)
614−writeWith elemCount = FL.rmapM extract $ FL.foldlM' step initial
657+createWith elemCount = FL.rmapM extract $ FL.foldlM' step initial
615658
616659 where
617660
618661 initial = do
619− when (elemCount < 0) $ error "writeWith: elemCount is negative"
620− liftIO $ new elemCount
662+ when (elemCount < 0) $ error "createWith: elemCount is negative"
663+ new elemCount
621664
622665 step arr@(MutArray _ start end bound) x
623666 | end == bound = do
624667 let oldSize = end - start
625668 newSize = max (oldSize * 2) 1
626− arr1 <- liftIO $ realloc newSize arr
669+ arr1 <- realloc newSize arr
627670 snocUnsafe arr1 x
628671 step arr x = snocUnsafe arr x
629672
630− -- extract = liftIO . rightSize
673+ -- extract = rightSize
631674 extract = return
632675
676+{-# DEPRECATED writeWith "Please use createWith instead." #-}
677+{-# INLINE writeWith #-}
678+writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
679+writeWith = createWith
680+
633681-- | Fold the whole input to a single array.
634682--
635−-- Same as 'writeWith' using an initial array size of 'arrayChunkSize' bytes
683+-- Same as 'createWith' using an initial array size of 'arrayChunkSize' bytes
636684-- rounded up to the element size.
637685--
638686-- /Caution! Do not use this on infinite streams./
639687--
688+{-# INLINE create #-}
689+create :: MonadIO m => Fold m a (MutArray a)
690+create = writeWith arrayChunkSize
691+
692+-- XXX Deprecate in major
693+-- {-# DEPRECATED write "Please use create instead." #-}
640694{-# INLINE write #-}
641695write :: MonadIO m => Fold m a (MutArray a)
642−write = writeWith arrayChunkSize
696+write = create
643697
644698-- | Create a 'MutArray' from the first @n@ elements of a stream. The
645699-- array is allocated to size @n@, if the stream terminates before @n@
@@ -664,7 +718,76 @@ fromList xs = fromStream $ D.fromList xs
664718{-# INLINABLE fromPureStream #-}
665719fromPureStream :: MonadIO m => Stream Identity a -> m (MutArray a)
666720fromPureStream xs =
667− liftIO $ D.fold write $ D.morphInner (return . runIdentity) xs
721+ D.fold write $ D.morphInner (return . runIdentity) xs
722+
723+-------------------------------------------------------------------------------
724+-- Chunking
725+-------------------------------------------------------------------------------
726+
727+data GroupState s a start end bound
728+ = GroupStart s
729+ | GroupBuffer s (MutableArray# RealWorld a) start end bound
730+ | GroupYield
731+ (MutableArray# RealWorld a)
732+ start
733+ end
734+ bound
735+ (GroupState s a start end bound)
736+ | GroupFinish
737+
738+-- | @chunksOf n stream@ groups the input stream into a stream of
739+-- arrays of size n.
740+--
741+-- @chunksOf n = foldMany (MutArray.writeN n)@
742+--
743+-- /Pre-release/
744+{-# INLINE_NORMAL chunksOf #-}
745+chunksOf :: forall m a. MonadIO m
746+ => Int -> D.Stream m a -> D.Stream m (MutArray a)
747+-- XXX the idiomatic implementation leads to large regression in the D.reverse'
748+-- benchmark. It seems it has difficulty producing optimized code when
749+-- converting to StreamK. Investigate GHC optimizations.
750+-- chunksOf n = D.foldMany (writeN n)
751+chunksOf n (D.Stream step state) =
752+ D.Stream step' (GroupStart state)
753+
754+ where
755+
756+ -- start is always 0
757+ -- end and len are always equal
758+
759+ {-# INLINE_LATE step' #-}
760+ step' _ (GroupStart st) = do
761+ when (n <= 0) $
762+ -- XXX we can pass the module string from the higher level API
763+ error $ "Streamly.Internal.Data.Array.Generic.Mut.Type.chunksOf: "
764+ ++ "the size of arrays [" ++ show n
765+ ++ "] must be a natural number"
766+ (MutArray contents start end bound :: MutArray a) <- new n
767+ return $ D.Skip (GroupBuffer st contents start end bound)
768+
769+ step' gst (GroupBuffer st contents start end bound) = do
770+ r <- step (adaptState gst) st
771+ case r of
772+ D.Yield x s -> do
773+ putIndexUnsafeWith end contents x
774+ let end1 = end + 1
775+ return $
776+ if end1 >= bound
777+ then D.Skip
778+ (GroupYield
779+ contents start end1 bound (GroupStart s))
780+ else D.Skip (GroupBuffer s contents start end1 bound)
781+ D.Skip s ->
782+ return $ D.Skip (GroupBuffer s contents start end bound)
783+ D.Stop ->
784+ return
785+ $ D.Skip (GroupYield contents start end bound GroupFinish)
786+
787+ step' _ (GroupYield contents start end bound next) =
788+ return $ D.Yield (MutArray contents start end bound) next
789+
790+ step' _ GroupFinish = return D.Stop
668791
669792-------------------------------------------------------------------------------
670793-- Unfolds
@@ -727,7 +850,7 @@ putSliceUnsafe src srcStart dst dstStart len = liftIO $ do
727850
728851{-# INLINE clone #-}
729852clone :: MonadIO m => MutArray a -> m (MutArray a)
730−clone src = liftIO $ do
853+clone src = do
731854 let len = arrLen src
732855 dst <- new len
733856 putSliceUnsafe src 0 dst 0 len
core/src/Streamly/Internal/Data/MutArray/Stream.hsnew file+145
@@ -0,0 +1,145 @@
1+-- |
2+-- Module : Streamly.Internal.Data.MutArray.Stream
3+-- Copyright : (c) 2019 Composewell Technologies
4+-- License : BSD3-3-Clause
5+-- Maintainer : streamly@composewell.com
6+-- Stability : experimental
7+-- Portability : GHC
8+--
9+-- Combinators to efficiently manipulate streams of mutable arrays.
10+--
11+-- We can either push these in the MutArray module with a "chunks" prefix or
12+-- keep this as a separate module and release it.
13+--
14+module Streamly.Internal.Data.MutArray.Stream
15+{-# DEPRECATED "Please use \"Streamly.Internal.Data.MutArray\" instead." #-}
16+ (
17+ -- * Generation
18+ MArray.chunksOf
19+ , MArray.pinnedChunksOf
20+ , MArray.writeChunks -- chunksWrite?
21+ , MArray.splitOn -- chunksSplitOn
22+
23+ -- * Compaction
24+ , packArraysChunksOf
25+ , MArray.SpliceState (..)
26+ , lpackArraysChunksOf
27+ , compact -- chunksCompact
28+ , compactLE
29+ , compactEQ
30+ , compactGE
31+
32+ -- * Elimination
33+ , MArray.flattenArrays -- chunksConcat
34+ , MArray.flattenArraysRev -- chunksConcatRev
35+ , MArray.fromArrayStreamK -- chunksCoalesce
36+ )
37+where
38+
39+import Control.Monad.IO.Class (MonadIO(..))
40+import Streamly.Internal.Data.Unbox (Unbox)
41+import Streamly.Internal.Data.MutArray.Type (MutArray(..))
42+import Streamly.Internal.Data.Fold.Type (Fold(..))
43+import Streamly.Internal.Data.Parser (ParseError)
44+import Streamly.Internal.Data.Stream.Type (Stream)
45+
46+import qualified Streamly.Internal.Data.MutArray as MArray
47+import qualified Streamly.Internal.Data.Fold.Type as FL
48+import qualified Streamly.Internal.Data.Parser as ParserD
49+import qualified Streamly.Internal.Data.Stream as D
50+
51+-------------------------------------------------------------------------------
52+-- Compact
53+-------------------------------------------------------------------------------
54+
55+-- XXX This can be removed once compactLEFold/compactLE are implemented.
56+--
57+-- | This mutates the first array (if it has space) to append values from the
58+-- second one. This would work for immutable arrays as well because an
59+-- immutable array never has space so a new array is allocated instead of
60+-- mutating it.
61+--
62+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
63+-- maximum specified size. Note that if a single array is bigger than the
64+-- specified size we do not split it to fit. When we coalesce multiple arrays
65+-- if the size would exceed the specified size we do not coalesce therefore the
66+-- actual array size may be less than the specified chunk size.
67+--
68+-- @since 0.7.0
69+{-# INLINE packArraysChunksOf #-}
70+packArraysChunksOf :: (MonadIO m, Unbox a)
71+ => Int -> D.Stream m (MutArray a) -> D.Stream m (MutArray a)
72+packArraysChunksOf = MArray.compactLE
73+
74+-- XXX Remove this once compactLEFold is implemented
75+-- lpackArraysChunksOf = Fold.many compactLEFold
76+--
77+{-# INLINE lpackArraysChunksOf #-}
78+lpackArraysChunksOf :: (MonadIO m, Unbox a)
79+ => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
80+lpackArraysChunksOf = MArray.lCompactGE
81+
82+-- XXX Same as compactLE, to be removed once that is implemented.
83+--
84+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
85+-- maximum specified size in bytes.
86+--
87+-- /Internal/
88+{-# INLINE compact #-}
89+compact :: (MonadIO m, Unbox a)
90+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
91+compact = packArraysChunksOf
92+
93+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
94+-- maximum specified size. Note that if a single array is bigger than the
95+-- specified size we do not split it to fit. When we coalesce multiple arrays
96+-- if the size would exceed the specified size we do not coalesce therefore the
97+-- actual array size may be less than the specified chunk size.
98+--
99+-- /Internal/
100+{-# INLINE compactLEParserD #-}
101+compactLEParserD ::
102+ forall m a. (MonadIO m, Unbox a)
103+ => Int -> ParserD.Parser (MutArray a) m (MutArray a)
104+compactLEParserD = MArray.pCompactLE
105+
106+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
107+-- minimum specified size. Note that if all the arrays in the stream together
108+-- are smaller than the specified size the resulting array will be smaller than
109+-- the specified size. When we coalesce multiple arrays if the size would exceed
110+-- the specified size we stop coalescing further.
111+--
112+-- /Internal/
113+{-# INLINE compactGEFold #-}
114+compactGEFold ::
115+ forall m a. (MonadIO m, Unbox a)
116+ => Int -> FL.Fold m (MutArray a) (MutArray a)
117+compactGEFold = MArray.fCompactGE
118+
119+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
120+-- maximum specified size in bytes.
121+--
122+-- /Internal/
123+compactLE :: (MonadIO m, Unbox a) =>
124+ Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
125+compactLE n = D.parseManyD (compactLEParserD n)
126+
127+-- | Like 'compactLE' but generates arrays of exactly equal to the size
128+-- specified except for the last array in the stream which could be shorter.
129+--
130+-- /Unimplemented/
131+{-# INLINE compactEQ #-}
132+compactEQ :: -- (MonadIO m, Unbox a) =>
133+ Int -> Stream m (MutArray a) -> Stream m (MutArray a)
134+compactEQ _n _xs = undefined
135+ -- IsStream.fromStreamD $ D.foldMany (compactEQFold n) (IsStream.toStreamD xs)
136+
137+-- | Like 'compactLE' but generates arrays of size greater than or equal to the
138+-- specified except for the last array in the stream which could be shorter.
139+--
140+-- /Internal/
141+{-# INLINE compactGE #-}
142+compactGE ::
143+ (MonadIO m, Unbox a)
144+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
145+compactGE n = D.foldMany (compactGEFold n)
core/src/Streamly/Internal/Data/MutArray/Type.hs+1308−441
@@ -28,57 +28,80 @@
2828
2929module Streamly.Internal.Data.MutArray.Type
3030 (
31− -- * Type
31+ -- ** Type
3232 -- $arrayNotes
3333 MutArray (..)
34− , MutableByteArray
35− , touch
34+
35+ -- ** Conversion
36+ -- *** Pinned and Unpinned
3637 , pin
3738 , unpin
3839 , isPinned
3940
40− -- * Constructing and Writing
41− -- ** Construction
42− , nil
41+ -- ** Casting
42+ , cast
43+ , castUnsafe -- XXX unsafeCast
44+ , asBytes
45+ , unsafePinnedAsPtr
46+ , unsafeAsPtr
4347
44− -- *** Uninitialized Arrays
45− , pinnedNew
46− , pinnedNewBytes
47− , pinnedNewAligned
48− , new
49− , newArrayWith
48+ -- ** Construction
49+ , empty
50+
51+ -- *** New
52+ -- | New arrays are always empty arrays with some reserve capacity to
53+ -- extend the length without reallocating.
54+ , emptyOf
55+ , newArrayWith -- emptyAlignedWith
56+ , pinnedEmptyOf
57+ , pinnedNewAligned -- XXX not required
58+ -- , new -- uninitialized array of specified length
59+
60+ -- *** Cloning
61+ , clone
62+ , pinnedClone
63+
64+ -- *** Slicing
65+ -- | Get a subarray without copying
66+ , getSliceUnsafe -- XXX unsafeGetSlice
67+ , getSlice
68+ , splitAt -- XXX should be able to express using getSlice
69+ , breakOn
5070
51− -- *** From streams
71+ -- *** Stream Folds
5272 , ArrayUnsafe (..)
53− , writeNWithUnsafe
54− , writeNWith
55− , writeNUnsafe
56− , pinnedWriteNUnsafe
57− , writeN
58− , pinnedWriteN
59− , pinnedWriteNAligned
60−
61− , writeWith
62− , write
63− , pinnedWrite
64−
65− , writeRevN
66− -- , writeRev
67−
68− -- ** From containers
73+ , unsafeCreateOfWith
74+ , unsafeCreateOf
75+ , unsafePinnedCreateOf
76+ , pinnedCreateOf
77+ , createOfWith
78+ , createOf
79+ , revCreateOf
80+
81+ , pinnedCreate
82+ , createWith
83+ , create
84+ -- , revCreate
85+
86+ -- *** From containers
6987 , fromListN
7088 , pinnedFromListN
7189 , fromList
7290 , pinnedFromList
7391 , fromListRevN
7492 , fromListRev
75− , fromStreamDN
76− , fromStreamD
93+ , fromStreamN
94+ , fromStream
95+ , fromPureStreamN
7796 , fromPureStream
97+ , fromByteStr#
98+ , fromPtrN
99+ , fromChunksK
100+ , fromChunksRealloced -- fromSmallChunks
78101
79− -- * Random writes
102+ -- ** Random writes
80103 , putIndex
81− , putIndexUnsafe
104+ , putIndexUnsafe -- XXX unsafePutIndex
82105 , putIndices
83106 -- , putFromThenTo
84107 -- , putFrom -- start writing at the given position
@@ -86,78 +109,71 @@ module Streamly.Internal.Data.MutArray.Type
86109 -- , putFromTo
87110 -- , putFromRev
88111 -- , putUptoRev
89− , modifyIndexUnsafe
112+ , modifyIndexUnsafe -- XXX unsafeModifyIndex
90113 , modifyIndex
91114 , modifyIndices
92115 , modify
93116 , swapIndices
94117 , unsafeSwapIndices
95118
96− -- * Growing and Shrinking
97− -- Arrays grow only at the end, though it is possible to grow on both sides
98− -- and therefore have a cons as well as snoc. But that will require two
99− -- bounds in the array representation.
119+ -- ** Reading
100120
101− -- ** Appending elements
102− , snocWith
103− , snoc
104− , snocLinear
105− , snocMay
106− , snocUnsafe
107−
108− -- ** Appending streams
109− , writeAppendNUnsafe
110− , writeAppendN
111− , writeAppendWith
112− , writeAppend
113−
114− -- * Eliminating and Reading
115−
116− -- ** To streams
117− , reader
118− , readerRevWith
119− , readerRev
121+ -- *** Indexing
122+ , getIndex
123+ , getIndexUnsafe -- XXX unsafeGetIndex
124+ -- , getFromThenTo
125+ , getIndexRev -- getRevIndex?
126+ , indexReader
127+ , indexReaderWith
120128
121− -- ** To containers
122− , toStreamDWith
123− , toStreamDRevWith
124− , toStreamKWith
125− , toStreamKRevWith
129+ -- *** To Streams
126130 , read
127131 , readRev
132+ , toStreamWith
133+ , toStreamRevWith
128134 , toStreamK
135+ , toStreamKWith
129136 , toStreamKRev
137+ , toStreamKRevWith
138+
139+ -- *** To Containers
130140 , toList
131141
142+ -- *** Unfolds
132143 -- experimental
133144 , producerWith
134145 , producer
135146
136− -- ** Random reads
137− , getIndex
138− , getIndexUnsafe
139− , getIndices
140− , getIndicesD
141− -- , getFromThenTo
142− , getIndexRev
143−
144− -- * Memory Management
145− , blockSize
146− , arrayChunkBytes
147− , allocBytesToElemCount
148− , realloc
149− , resize
150− , resizeExp
151− , rightSize
147+ , reader
148+ , readerRevWith
149+ , readerRev
152150
153− -- * Size
151+ -- ** Size and Capacity
152+ -- *** Size
154153 , length
155154 , byteLength
155+
156+ -- *** Capacity
156157 -- , capacity
157158 , byteCapacity
158159 , bytesFree
159160
160− -- * In-place Mutation Algorithms
161+ -- *** Capacity Management
162+ , blockSize
163+ , arrayChunkBytes
164+ , allocBytesToElemCount
165+ , realloc -- this is "resize"
166+ , grow
167+ , growExp
168+ , rightSize
169+
170+ -- ** Folding
171+ , foldl'
172+ , foldr
173+ , byteCmp
174+ , byteEq
175+
176+ -- ** In-place Mutation Algorithms
161177 , strip
162178 , reverse
163179 , permute
@@ -167,59 +183,121 @@ module Streamly.Internal.Data.MutArray.Type
167183 , mergeBy
168184 , bubble
169185
170− -- * Casting
171− , cast
172− , castUnsafe
173− , asBytes
174− , asPtrUnsafe
175−
176− -- * Folding
177− , foldl'
178− , foldr
179− , cmp
180−
181− -- * Arrays of arrays
182− -- We can add dimensionality parameter to the array type to get
183− -- multidimensional arrays. Multidimensional arrays would just be a
184− -- convenience wrapper on top of single dimensional arrays.
185−
186− -- | Operations dealing with multiple arrays, streams of arrays or
187− -- multidimensional array representations.
186+ -- ** Growing and Shrinking
187+ -- | Arrays grow only at the end, though it is possible to grow on both
188+ -- sides and therefore have a cons as well as snoc. But that will require
189+ -- both lower and upper bound in the array representation.
188190
189− -- ** Construct from streams
190− , chunksOf
191− , pinnedChunksOf
192− , writeChunks
193−
194− -- ** Eliminate to streams
195− , flattenArrays
196− , flattenArraysRev
197− , fromArrayStreamK
191+ -- *** Appending elements
192+ , snocWith
193+ , snoc
194+ , snocLinear
195+ , snocMay
196+ , snocUnsafe -- XXX unsafeSnoc
198197
199− -- ** Construct from arrays
200− -- get chunks without copying
201− , getSliceUnsafe
202− , getSlice
203− -- , getSlicesFromLenN
204− , splitAt -- XXX should be able to express using getSlice
205− , breakOn
198+ -- *** Appending streams
199+ , unsafeAppendN
200+ , appendN
201+ , appendWith
202+ , append
206203
207− -- ** Appending arrays
204+ -- *** Appending arrays
208205 , spliceCopy
209206 , spliceWith
210207 , splice
211208 , spliceExp
212− , spliceUnsafe
213− , putSliceUnsafe
209+ , spliceUnsafe -- XXX unsafeSplice
214210 -- , putSlice
215211 -- , appendSlice
216212 -- , appendSliceFrom
217213
218− -- * Utilities
214+ -- ** Serialization using Unbox
215+ , pokeAppend
216+ , pokeAppendMay
217+ , pokeSkipUnsafe -- unsafePokeSkip
218+
219+ -- ** Deserialization using Unbox
220+ , peekUncons
221+ , peekUnconsUnsafe -- unsafePeekUncons
222+ , peekSkipUnsafe -- unsafePeekSkip
223+
224+ -- Arrays of arrays
225+ -- We can add dimensionality parameter to the array type to get
226+ -- multidimensional arrays. Multidimensional arrays would just be a
227+ -- convenience wrapper on top of single dimensional arrays.
228+
229+ -- ** Streams of Arrays
230+ -- *** Chunk
231+ -- | Group a stream into arrays.
232+ , chunksOf
233+ , pinnedChunksOf
234+ , buildChunks
235+
236+ -- *** Split
237+ -- | Split an array into slices.
238+
239+ -- , getSlicesFromLenN
240+ , splitOn
241+ -- , slicesOf
242+
243+ -- *** Concat
244+ -- | Append the arrays in a stream to form a stream of elements.
245+ , concatWith
246+ , concatRevWith
247+ , concat
248+ , concatRev
249+
250+ -- *** Compact
251+ -- | Append the arrays in a stream to form a stream of larger arrays.
252+ , SpliceState (..)
253+ , pCompactLE
254+ , pPinnedCompactLE
255+ , compactLeAs
256+ , fCompactGE
257+ , fPinnedCompactGE
258+ , lCompactGE
259+ , lPinnedCompactGE
260+ , compactGE
261+ , compactEQ
262+
263+ -- ** Utilities
219264 , roundUpToPower2
220265 , memcpy
221266 , memcmp
222267 , c_memchr
268+
269+ -- * Deprecated
270+ , asPtrUnsafe
271+ , writeChunks
272+ , flattenArrays
273+ , flattenArraysRev
274+ , fromArrayStreamK
275+ , fromStreamDN
276+ , fromStreamD
277+ , cmp
278+ , getIndices
279+ , getIndicesWith
280+ , resize
281+ , resizeExp
282+ , nil
283+ , new
284+ , pinnedNew
285+ , pinnedNewBytes
286+ , writeAppendNUnsafe
287+ , writeAppendN
288+ , writeAppendWith
289+ , writeAppend
290+ , writeNWithUnsafe
291+ , writeNWith
292+ , writeNUnsafe
293+ , pinnedWriteNUnsafe
294+ , writeN
295+ , pinnedWriteN
296+ , pinnedWriteNAligned -- XXX not required
297+ , writeWith
298+ , write
299+ , pinnedWrite
300+ , writeRevN
223301 )
224302where
225303
@@ -230,48 +308,52 @@ where
230308
231309import Control.Monad (when, void)
232310import Control.Monad.IO.Class (MonadIO(..))
311+import Data.Bifunctor (first)
233312import Data.Bits (shiftR, (.|.), (.&.))
234313import Data.Functor.Identity (Identity(..))
235314import Data.Proxy (Proxy(..))
236315import Data.Word (Word8)
237316import Foreign.C.Types (CSize(..), CInt(..))
238317import Foreign.Ptr (plusPtr, minusPtr, nullPtr)
239−import Streamly.Internal.Data.Unbox
240− ( MutableByteArray(..)
241− , Unbox(..)
318+import Streamly.Internal.Data.MutByteArray.Type
319+ ( MutByteArray(..)
242320 , PinnedState(..)
243321 , getMutableByteArray#
244− , touch
322+ , putSliceUnsafe
245323 )
324+import Streamly.Internal.Data.Unbox (Unbox(..))
246325import GHC.Base
247326 ( IO(..)
248327 , Int(..)
249− , byteArrayContents#
250328 , compareByteArrays#
251329 , copyMutableByteArray#
252330 )
253331import GHC.Base (noinline)
254−import GHC.Exts (unsafeCoerce#)
332+import GHC.Exts (unsafeCoerce#, Addr#)
255333import GHC.Ptr (Ptr(..))
256334
257335import Streamly.Internal.Data.Fold.Type (Fold(..))
258336import Streamly.Internal.Data.Producer.Type (Producer (..))
259−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
260−import Streamly.Internal.Data.Stream.StreamK.Type (StreamK)
337+import Streamly.Internal.Data.Stream.Type (Stream)
338+import Streamly.Internal.Data.Parser.Type (Parser (..))
339+import Streamly.Internal.Data.StreamK.Type (StreamK)
261340import Streamly.Internal.Data.SVar.Type (adaptState, defState)
341+import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
262342import Streamly.Internal.Data.Unfold.Type (Unfold(..))
263343import Streamly.Internal.System.IO (arrayPayloadSize, defaultChunkSize)
264344
265345import qualified Streamly.Internal.Data.Fold.Type as FL
346+import qualified Streamly.Internal.Data.MutByteArray.Type as Unboxed
347+import qualified Streamly.Internal.Data.Parser.Type as Parser
348+-- import qualified Streamly.Internal.Data.Fold.Type as Fold
266349import qualified Streamly.Internal.Data.Producer as Producer
267−import qualified Streamly.Internal.Data.Stream.StreamD.Type as D
268−import qualified Streamly.Internal.Data.Stream.StreamD.Lift as D
269−import qualified Streamly.Internal.Data.Stream.StreamK.Type as K
270−import qualified Streamly.Internal.Data.Unbox as Unboxed
350+import qualified Streamly.Internal.Data.Stream.Type as D
351+import qualified Streamly.Internal.Data.Stream.Lift as D
352+import qualified Streamly.Internal.Data.StreamK.Type as K
271353import qualified Prelude
272354
273355import Prelude hiding
274− (Foldable(..), read, unlines, splitAt, reverse, truncate)
356+ (Foldable(..), concat, read, unlines, splitAt, reverse, truncate)
275357
276358#include "DocTestDataMutArray.hs"
277359
@@ -279,6 +361,7 @@ import Prelude hiding
279361-- Foreign helpers
280362-------------------------------------------------------------------------------
281363
364+-- NOTE: Have to be "ccall unsafe" so that we can pass unpinned memory to these
282365foreign import ccall unsafe "string.h memcpy" c_memcpy
283366 :: Ptr Word8 -> Ptr Word8 -> CSize -> IO (Ptr Word8)
284367
@@ -288,6 +371,9 @@ foreign import ccall unsafe "string.h memchr" c_memchr
288371foreign import ccall unsafe "string.h memcmp" c_memcmp
289372 :: Ptr Word8 -> Ptr Word8 -> CSize -> IO CInt
290373
374+foreign import ccall unsafe "string.h strlen" c_strlen
375+ :: Ptr Word8 -> IO CSize
376+
291377-- | Given an 'Unboxed' type (unused first arg) and a number of bytes, return
292378-- how many elements of that type will completely fit in those bytes.
293379--
@@ -342,7 +428,7 @@ data MutArray a =
342428 -- The array is a range into arrContents. arrContents may be a superset of
343429 -- the slice represented by the array. All offsets are in bytes.
344430 MutArray
345− { arrContents :: {-# UNPACK #-} !MutableByteArray
431+ { arrContents :: {-# UNPACK #-} !MutByteArray
346432 , arrStart :: {-# UNPACK #-} !Int -- ^ index into arrContents
347433 , arrEnd :: {-# UNPACK #-} !Int -- ^ index into arrContents
348434 -- Represents the first invalid index of
@@ -354,22 +440,25 @@ data MutArray a =
354440-- Pinning & Unpinning
355441-------------------------------------------------------------------------------
356442
357−-- | Copy the 'MutArray' to pinned memory if unpinned, else do nothing. The
358−-- overhead is a copy if the input array is unpinned.
443+-- | Return a copy of the array in pinned memory if unpinned, else return the
444+-- original array.
359445{-# INLINE pin #-}
360446pin :: MutArray a -> IO (MutArray a)
361−pin arr@MutArray{..} = do
362− contents <- Unboxed.pin arrContents
363− return $ arr {arrContents = contents}
447+pin arr@MutArray{..} =
448+ if Unboxed.isPinned arrContents
449+ then pure arr
450+ else pinnedClone arr
364451
365−-- | Copy the 'MutArray' to unpinned memory if pinned, else do nothing. The
366−-- overhead is a copy if the input array is pinned.
452+-- | Return a copy of the array in unpinned memory if pinned, else return the
453+-- original array.
367454{-# INLINE unpin #-}
368455unpin :: MutArray a -> IO (MutArray a)
369−unpin arr@MutArray{..} = do
370− contents <- Unboxed.unpin arrContents
371− return $ arr {arrContents = contents}
456+unpin arr@MutArray{..} =
457+ if Unboxed.isPinned arrContents
458+ then clone arr
459+ else pure arr
372460
461+-- | Return 'True' if the array is allocated in pinned memory.
373462{-# INLINE isPinned #-}
374463isPinned :: MutArray a -> Bool
375464isPinned MutArray{..} = Unboxed.isPinned arrContents
@@ -390,6 +479,8 @@ isPinned MutArray{..} = Unboxed.isPinned arrContents
390479-- we need more than that. See stg_pinnedNewAlignedByteArrayzh and
391480-- allocatePinned in GHC source.
392481
482+-- XXX Rename to emptyAlignedWith, alignSize should be first arg.
483+
393484-- | @newArrayWith allocator alignment count@ allocates a new array of zero
394485-- length and with a capacity to hold @count@ elements, using @allocator
395486-- size alignment@ as the memory allocator function.
@@ -402,7 +493,7 @@ isPinned MutArray{..} = Unboxed.isPinned arrContents
402493-- /Pre-release/
403494{-# INLINE newArrayWith #-}
404495newArrayWith :: forall m a. (MonadIO m, Unbox a)
405− => (Int -> Int -> m MutableByteArray) -> Int -> Int -> m (MutArray a)
496+ => (Int -> Int -> m MutByteArray) -> Int -> Int -> m (MutArray a)
406497newArrayWith alloc alignSize count = do
407498 let size = max (count * SIZE_OF(a)) 0
408499 contents <- alloc size alignSize
@@ -413,12 +504,25 @@ newArrayWith alloc alignSize count = do
413504 , arrBound = size
414505 }
415506
507+-- For arrays "nil" sounds a bit odd. empty is better. The only problem with
508+-- empty is that it is also used by the Alternative type class. But assuming we
509+-- will mostly import the Array module qualified this should be fine.
510+
511+-- | Create an empty array.
512+empty ::
513+#ifdef DEVBUILD
514+ Unbox a =>
515+#endif
516+ MutArray a
517+empty = MutArray Unboxed.empty 0 0 0
518+
519+{-# DEPRECATED nil "Please use empty instead." #-}
416520nil ::
417521#ifdef DEVBUILD
418522 Unbox a =>
419523#endif
420524 MutArray a
421−nil = MutArray Unboxed.nil 0 0 0
525+nil = empty
422526
423527{-# INLINE newBytesAs #-}
424528newBytesAs :: MonadIO m =>
@@ -435,12 +539,13 @@ newBytesAs ps bytes = do
435539 , arrBound = bytes
436540 }
437541
438−-- | Allocates a pinned empty array that can hold 'count' items. The memory of
439−-- the array is uninitialized and the allocation is aligned as per the
440−-- 'Unboxed' instance of the type.
542+-- | Allocates a pinned empty array that with a reserved capacity of bytes.
543+-- The memory of the array is uninitialized and the allocation is aligned as
544+-- per the 'Unboxed' instance of the type.
441545--
442546-- /Pre-release/
443547{-# INLINE pinnedNewBytes #-}
548+{-# DEPRECATED pinnedNewBytes "Please use pinnedEmptyOf with appropriate calculation" #-}
444549pinnedNewBytes :: MonadIO m =>
445550#ifdef DEVBUILD
446551 Unbox a =>
@@ -465,21 +570,31 @@ newAs ps =
465570 (error "new: alignment is not used in unpinned arrays.")
466571
467572-- XXX can unaligned allocation be more efficient when alignment is not needed?
468−--
469−-- | Allocates an empty pinned array that can hold 'count' items. The memory of
470−-- the array is uninitialized and the allocation is aligned as per the 'Unboxed'
471−-- instance of the type.
472−--
573+
574+-- | Allocates a pinned array of zero length but growable to the specified
575+-- capacity without reallocation.
576+{-# INLINE pinnedEmptyOf #-}
577+pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
578+pinnedEmptyOf = newAs Pinned
579+
580+-- XXX Deprecate in major
581+-- {-# DEPRECATED pinnedNew "Please use pinnedEmptyOf instead." #-}
473582{-# INLINE pinnedNew #-}
474583pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
475−pinnedNew = newAs Pinned
584+pinnedNew = pinnedEmptyOf
476585
477−-- | Allocates an empty unpinned array that can hold 'count' items. The memory
478−-- of the array is uninitialized.
586+-- | Allocates an unpinned array of zero length but growable to the specified
587+-- capacity without reallocation.
479588--
589+{-# INLINE emptyOf #-}
590+emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
591+emptyOf = newAs Unpinned
592+
593+-- XXX Deprecate in major
594+-- {-# DEPRECATED new "Please use emptyOf instead." #-}
480595{-# INLINE new #-}
481596new :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
482−new = newAs Unpinned
597+new = emptyOf
483598
484599-------------------------------------------------------------------------------
485600-- Random writes
@@ -495,7 +610,7 @@ putIndexUnsafe :: forall m a. (MonadIO m, Unbox a)
495610putIndexUnsafe i MutArray{..} x = do
496611 let index = INDEX_OF(arrStart, i, a)
497612 assert (i >= 0 && INDEX_VALID(index, arrEnd, a)) (return ())
498− liftIO $ pokeByteIndex index arrContents x
613+ liftIO $ pokeAt index arrContents x
499614
500615invalidIndex :: String -> Int -> a
501616invalidIndex label i =
@@ -513,7 +628,7 @@ putIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
513628putIndex i MutArray{..} x = do
514629 let index = INDEX_OF(arrStart,i,a)
515630 if i >= 0 && INDEX_VALID(index,arrEnd,a)
516− then liftIO $ pokeByteIndex index arrContents x
631+ then liftIO $ pokeAt index arrContents x
517632 else invalidIndex "putIndex" i
518633
519634-- | Write an input stream of (index, value) pairs to an array. Throws an
@@ -527,7 +642,7 @@ putIndices arr = FL.foldlM' step (return ())
527642
528643 where
529644
530− step () (i, x) = liftIO (putIndex i arr x)
645+ step () (i, x) = putIndex i arr x
531646
532647-- | Modify a given index of an array using a modifier function.
533648--
@@ -539,9 +654,9 @@ modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) =>
539654modifyIndexUnsafe i MutArray{..} f = liftIO $ do
540655 let index = INDEX_OF(arrStart,i,a)
541656 assert (i >= 0 && INDEX_NEXT(index,a) <= arrEnd) (return ())
542− r <- peekByteIndex index arrContents
657+ r <- peekAt index arrContents
543658 let (x, res) = f r
544− pokeByteIndex index arrContents x
659+ pokeAt index arrContents x
545660 return res
546661
547662-- | Modify a given index of an array using a modifier function.
@@ -553,9 +668,9 @@ modifyIndex i MutArray{..} f = do
553668 let index = INDEX_OF(arrStart,i,a)
554669 if i >= 0 && INDEX_VALID(index,arrEnd,a)
555670 then liftIO $ do
556− r <- peekByteIndex index arrContents
671+ r <- peekAt index arrContents
557672 let (x, res) = f r
558− pokeByteIndex index arrContents x
673+ pokeAt index arrContents x
559674 return res
560675 else invalidIndex "modifyIndex" i
561676
@@ -589,8 +704,8 @@ modify MutArray{..} f = liftIO $
589704
590705 go i =
591706 when (INDEX_VALID(i,arrEnd,a)) $ do
592− r <- peekByteIndex i arrContents
593− pokeByteIndex i arrContents (f r)
707+ r <- peekAt i arrContents
708+ pokeAt i arrContents (f r)
594709 go (INDEX_NEXT(i,a))
595710
596711-- XXX We could specify the number of bytes to swap instead of Proxy. Need
@@ -599,15 +714,15 @@ modify MutArray{..} f = liftIO $
599714swapArrayByteIndices ::
600715 forall a. Unbox a
601716 => Proxy a
602− -> MutableByteArray
717+ -> MutByteArray
603718 -> Int
604719 -> Int
605720 -> IO ()
606721swapArrayByteIndices _ arrContents i1 i2 = do
607− r1 <- peekByteIndex i1 arrContents
608− r2 <- peekByteIndex i2 arrContents
609− pokeByteIndex i1 arrContents (r2 :: a)
610− pokeByteIndex i2 arrContents (r1 :: a)
722+ r1 <- peekAt i1 arrContents
723+ r2 <- peekAt i2 arrContents
724+ pokeAt i1 arrContents (r2 :: a)
725+ pokeAt i2 arrContents (r1 :: a)
611726
612727-- | Swap the elements at two indices without validating the indices.
613728--
@@ -723,26 +838,23 @@ arrayChunkBytes = 1024
723838roundDownTo :: Int -> Int -> Int
724839roundDownTo elemSize size = size - (size `mod` elemSize)
725840
726−-- XXX See if resizing can be implemented by reading the old array as a stream
727−-- and then using writeN to the new array.
728−--
729841-- NOTE: we are passing elemSize explicitly to avoid an Unboxed constraint.
730842-- Since this is not inlined Unboxed consrraint leads to dictionary passing
731843-- which complicates some inspection tests.
732844--
733−{-# NOINLINE reallocExplicit #-}
734−reallocExplicit :: Int -> Int -> MutArray a -> IO (MutArray a)
735−reallocExplicit elemSize newCapacityInBytes MutArray{..} = do
845+{-# NOINLINE reallocExplicitAs #-}
846+reallocExplicitAs :: PinnedState -> Int -> Int -> MutArray a -> IO (MutArray a)
847+reallocExplicitAs ps elemSize newCapacityInBytes MutArray{..} = do
736848 assertM(arrEnd <= arrBound)
737849
738850 -- Allocate new array
739851 let newCapMaxInBytes = roundUpLargeArray newCapacityInBytes
740852 contents <-
741− if Unboxed.isPinned arrContents
742− then Unboxed.pinnedNewBytes newCapMaxInBytes
743− else Unboxed.newBytes newCapMaxInBytes
744− let !(MutableByteArray mbarrFrom#) = arrContents
745− !(MutableByteArray mbarrTo#) = contents
853+ if ps == Pinned
854+ then Unboxed.pinnedNew newCapMaxInBytes
855+ else Unboxed.new newCapMaxInBytes
856+ let !(MutByteArray mbarrFrom#) = arrContents
857+ !(MutByteArray mbarrTo#) = contents
746858
747859 -- Copy old data
748860 let oldStart = arrStart
@@ -763,6 +875,10 @@ reallocExplicit elemSize newCapacityInBytes MutArray{..} = do
763875 , arrBound = newCapInBytes
764876 }
765877
878+-- XXX Should these be called reallocBytes etc?
879+-- XXX We may also need reallocAs to allocate as pinned/unpinned explicitly. In
880+-- fact clone/pinnedClone can be implemented using reallocAs.
881+
766882-- | @realloc newCapacity array@ reallocates the array to the specified
767883-- capacity in bytes.
768884--
@@ -774,7 +890,12 @@ reallocExplicit elemSize newCapacityInBytes MutArray{..} = do
774890-- If the original array is pinned, the newly allocated array is also pinned.
775891{-# INLINABLE realloc #-}
776892realloc :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
777−realloc bytes arr = liftIO $ reallocExplicit (SIZE_OF(a)) bytes arr
893+realloc bytes arr =
894+ let ps =
895+ if isPinned arr
896+ then Pinned
897+ else Unpinned
898+ in liftIO $ reallocExplicitAs ps (SIZE_OF(a)) bytes arr
778899
779900-- | @reallocWith label capSizer minIncrBytes array@. The label is used
780901-- in error messages and the capSizer is used to determine the capacity of the
@@ -797,14 +918,14 @@ reallocWith label capSizer minIncrBytes arr = do
797918 where
798919
799920 badSize newSize =
800− concat
921+ Prelude.concat
801922 [ label
802923 , ": new array size (in bytes) is less than required size "
803924 , show newSize
804925 , ". Please check the sizing function passed."
805926 ]
806927
807−-- | @resize newCapacity array@ changes the total capacity of the array so that
928+-- | @grow newCapacity array@ changes the total capacity of the array so that
808929-- it is enough to hold the specified number of elements. Nothing is done if
809930-- the specified capacity is less than the length of the array.
810931--
@@ -812,34 +933,46 @@ reallocWith label capSizer minIncrBytes arr = do
812933-- the block size (4K).
813934--
814935-- /Pre-release/
815−{-# INLINE resize #-}
816−resize :: forall m a. (MonadIO m, Unbox a) =>
936+{-# INLINE grow #-}
937+grow :: forall m a. (MonadIO m, Unbox a) =>
817938 Int -> MutArray a -> m (MutArray a)
818−resize nElems arr@MutArray{..} = do
939+grow nElems arr@MutArray{..} = do
819940 let req = SIZE_OF(a) * nElems
820− len = arrEnd - arrStart
821− if req < len
941+ cap = arrBound - arrStart
942+ if req < cap
822943 then return arr
823944 else realloc req arr
824945
825−-- | Like 'resize' but if the byte capacity is more than 'largeObjectThreshold'
826−-- then it is rounded up to the closest power of 2.
946+{-# DEPRECATED resize "Please use grow instead." #-}
947+{-# INLINE resize #-}
948+resize :: forall m a. (MonadIO m, Unbox a) =>
949+ Int -> MutArray a -> m (MutArray a)
950+resize = grow
951+
952+-- | Like 'grow' but if the requested byte capacity is more than
953+-- 'largeObjectThreshold' then it is rounded up to the closest power of 2.
827954--
828955-- /Pre-release/
829−{-# INLINE resizeExp #-}
830−resizeExp :: forall m a. (MonadIO m, Unbox a) =>
956+{-# INLINE growExp #-}
957+growExp :: forall m a. (MonadIO m, Unbox a) =>
831958 Int -> MutArray a -> m (MutArray a)
832−resizeExp nElems arr@MutArray{..} = do
959+growExp nElems arr@MutArray{..} = do
833960 let req = roundUpLargeArray (SIZE_OF(a) * nElems)
834961 req1 =
835962 if req > largeObjectThreshold
836963 then roundUpToPower2 req
837964 else req
838− len = arrEnd - arrStart
839− if req1 < len
965+ cap = arrBound - arrStart
966+ if req1 < cap
840967 then return arr
841968 else realloc req1 arr
842969
970+{-# DEPRECATED resizeExp "Please use growExp instead." #-}
971+{-# INLINE resizeExp #-}
972+resizeExp :: forall m a. (MonadIO m, Unbox a) =>
973+ Int -> MutArray a -> m (MutArray a)
974+resizeExp = growExp
975+
843976-- | Resize the allocated memory to drop any reserved free space at the end of
844977-- the array and reallocate it to reduce wastage.
845978--
@@ -884,7 +1017,7 @@ rightSize arr@MutArray{..} = do
8841017snocNewEnd :: (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m (MutArray a)
8851018snocNewEnd newEnd arr@MutArray{..} x = liftIO $ do
8861019 assert (newEnd <= arrBound) (return ())
887− pokeByteIndex arrEnd arrContents x
1020+ pokeAt arrEnd arrContents x
8881021 return $ arr {arrEnd = newEnd}
8891022
8901023-- | Really really unsafe, appends the element into the first array, may
@@ -904,7 +1037,7 @@ snocUnsafe arr@MutArray{..} = snocNewEnd (INDEX_NEXT(arrEnd,a)) arr
9041037{-# INLINE snocMay #-}
9051038snocMay :: forall m a. (MonadIO m, Unbox a) =>
9061039 MutArray a -> a -> m (Maybe (MutArray a))
907−snocMay arr@MutArray{..} x = liftIO $ do
1040+snocMay arr@MutArray{..} x = do
9081041 let newEnd = INDEX_NEXT(arrEnd,a)
9091042 if newEnd <= arrBound
9101043 then Just <$> snocNewEnd newEnd arr x
@@ -918,7 +1051,7 @@ snocWithRealloc :: forall m a. (MonadIO m, Unbox a) =>
9181051 -> a
9191052 -> m (MutArray a)
9201053snocWithRealloc sizer arr x = do
921− arr1 <- liftIO $ reallocWith "snocWith" sizer (SIZE_OF(a)) arr
1054+ arr1 <- reallocWith "snocWith" sizer (SIZE_OF(a)) arr
9221055 snocUnsafe arr1 x
9231056
9241057-- | @snocWith sizer arr elem@ mutates @arr@ to append @elem@. The length of
@@ -940,7 +1073,7 @@ snocWith :: forall m a. (MonadIO m, Unbox a) =>
9401073 -> MutArray a
9411074 -> a
9421075 -> m (MutArray a)
943−snocWith allocSize arr x = liftIO $ do
1076+snocWith allocSize arr x = do
9441077 let newEnd = INDEX_NEXT(arrEnd arr,a)
9451078 if newEnd <= arrBound arr
9461079 then snocNewEnd newEnd arr x
@@ -984,6 +1117,126 @@ snoc = snocWith f
9841117 then oldSize * 2
9851118 else roundUpToPower2 oldSize * 2
9861119
1120+-------------------------------------------------------------------------------
1121+-- Serialization/Deserialization
1122+-------------------------------------------------------------------------------
1123+
1124+{-# INLINE pokeNewEnd #-}
1125+pokeNewEnd :: (MonadIO m, Unbox a) =>
1126+ Int -> MutArray Word8 -> a -> m (MutArray Word8)
1127+pokeNewEnd newEnd arr@MutArray{..} x = liftIO $ do
1128+ assert (newEnd <= arrBound) (return ())
1129+ liftIO $ pokeAt arrEnd arrContents x
1130+ return $ arr {arrEnd = newEnd}
1131+
1132+-- | Really really unsafe, unboxes a Haskell type and appends the resulting
1133+-- bytes to the byte array, may cause silent data corruption or if you are
1134+-- lucky a segfault if the array does not have enough space to append the
1135+-- element.
1136+--
1137+-- /Internal/
1138+{-# INLINE pokeAppendUnsafe #-}
1139+pokeAppendUnsafe :: forall m a. (MonadIO m, Unbox a) =>
1140+ MutArray Word8 -> a -> m (MutArray Word8)
1141+pokeAppendUnsafe arr@MutArray{..} = pokeNewEnd (arrEnd + SIZE_OF(a)) arr
1142+
1143+-- | Skip the specified number of bytes in the array. The data in the skipped
1144+-- region remains uninitialzed.
1145+{-# INLINE pokeSkipUnsafe #-}
1146+pokeSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
1147+pokeSkipUnsafe n arr@MutArray{..} = do
1148+ let newEnd = arrEnd + n
1149+ in assert (newEnd <= arrBound) (arr {arrEnd = newEnd})
1150+
1151+-- | Like 'pokeAppend' but does not grow the array when pre-allocated array
1152+-- capacity becomes full.
1153+--
1154+-- /Internal/
1155+{-# INLINE pokeAppendMay #-}
1156+pokeAppendMay :: forall m a. (MonadIO m, Unbox a) =>
1157+ MutArray Word8 -> a -> m (Maybe (MutArray Word8))
1158+pokeAppendMay arr@MutArray{..} x = liftIO $ do
1159+ let newEnd = arrEnd + SIZE_OF(a)
1160+ if newEnd <= arrBound
1161+ then Just <$> pokeNewEnd newEnd arr x
1162+ else return Nothing
1163+
1164+{-# NOINLINE pokeWithRealloc #-}
1165+pokeWithRealloc :: forall m a. (MonadIO m, Unbox a) =>
1166+ (Int -> Int)
1167+ -> MutArray Word8
1168+ -> a
1169+ -> m (MutArray Word8)
1170+pokeWithRealloc sizer arr x = do
1171+ arr1 <- liftIO $ reallocWith "pokeWithRealloc" sizer (SIZE_OF(a)) arr
1172+ pokeAppendUnsafe arr1 x
1173+
1174+{-# INLINE pokeAppendWith #-}
1175+pokeAppendWith :: forall m a. (MonadIO m, Unbox a) =>
1176+ (Int -> Int)
1177+ -> MutArray Word8
1178+ -> a
1179+ -> m (MutArray Word8)
1180+pokeAppendWith allocSize arr x = liftIO $ do
1181+ let newEnd = arrEnd arr + SIZE_OF(a)
1182+ if newEnd <= arrBound arr
1183+ then pokeNewEnd newEnd arr x
1184+ else pokeWithRealloc allocSize arr x
1185+
1186+-- | Unbox a Haskell type and append the resulting bytes to a mutable byte
1187+-- array. The array is grown exponentially when more space is needed.
1188+--
1189+-- Definition:
1190+--
1191+-- >>> pokeAppend arr x = MutArray.castUnsafe <$> MutArray.snoc (MutArray.castUnsafe arr) x
1192+--
1193+{-# INLINE pokeAppend #-}
1194+pokeAppend :: forall m a. (MonadIO m, Unbox a) =>
1195+ MutArray Word8 -> a -> m (MutArray Word8)
1196+pokeAppend = pokeAppendWith f
1197+
1198+ where
1199+
1200+ f oldSize =
1201+ if isPower2 oldSize
1202+ then oldSize * 2
1203+ else roundUpToPower2 oldSize * 2
1204+
1205+-- | Really really unsafe, create a Haskell value from an unboxed byte array,
1206+-- does not check if the array is big enough, may return garbage or if you are
1207+-- lucky may cause a segfault.
1208+--
1209+-- /Internal/
1210+{-# INLINE peekUnconsUnsafe #-}
1211+peekUnconsUnsafe :: forall m a. (MonadIO m, Unbox a) =>
1212+ MutArray Word8 -> m (a, MutArray Word8)
1213+peekUnconsUnsafe MutArray{..} = do
1214+ let start1 = arrStart + SIZE_OF(a)
1215+ assert (start1 <= arrEnd) (return ())
1216+ liftIO $ do
1217+ r <- peekAt arrStart arrContents
1218+ return (r, MutArray arrContents start1 arrEnd arrBound)
1219+
1220+-- | Discard the specified number of bytes in the array.
1221+{-# INLINE peekSkipUnsafe #-}
1222+peekSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
1223+peekSkipUnsafe n MutArray{..} =
1224+ let start1 = arrStart + n
1225+ in assert (start1 <= arrEnd) (MutArray arrContents start1 arrEnd arrBound)
1226+
1227+-- | Create a Haskell value from its unboxed representation from the head of a
1228+-- byte array, return the value and the remaining array.
1229+{-# INLINE peekUncons #-}
1230+peekUncons :: forall m a. (MonadIO m, Unbox a) =>
1231+ MutArray Word8 -> m (Maybe a, MutArray Word8)
1232+peekUncons arr@MutArray{..} = do
1233+ let start1 = arrStart + SIZE_OF(a)
1234+ if start1 > arrEnd
1235+ then return (Nothing, arr)
1236+ else liftIO $ do
1237+ r <- peekAt arrStart arrContents
1238+ return (Just r, MutArray arrContents start1 arrEnd arrBound)
1239+
9871240-------------------------------------------------------------------------------
9881241-- Random reads
9891242-------------------------------------------------------------------------------
@@ -998,7 +1251,7 @@ getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
9981251getIndexUnsafe i MutArray{..} = do
9991252 let index = INDEX_OF(arrStart,i,a)
10001253 assert (i >= 0 && INDEX_VALID(index,arrEnd,a)) (return ())
1001− liftIO $ peekByteIndex index arrContents
1254+ liftIO $ peekAt index arrContents
10021255
10031256-- | /O(1)/ Lookup the element at the given index. Index starts from 0.
10041257--
@@ -1007,7 +1260,7 @@ getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
10071260getIndex i MutArray{..} = do
10081261 let index = INDEX_OF(arrStart,i,a)
10091262 if i >= 0 && INDEX_VALID(index,arrEnd,a)
1010− then liftIO $ Just <$> peekByteIndex index arrContents
1263+ then liftIO $ Just <$> peekAt index arrContents
10111264 else return Nothing
10121265
10131266-- | /O(1)/ Lookup the element at the given index from the end of the array.
@@ -1020,21 +1273,16 @@ getIndexRev :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
10201273getIndexRev i MutArray{..} = do
10211274 let index = RINDEX_OF(arrEnd,i,a)
10221275 if i >= 0 && index >= arrStart
1023− then liftIO $ peekByteIndex index arrContents
1276+ then liftIO $ peekAt index arrContents
10241277 else invalidIndex "getIndexRev" i
10251278
10261279data GetIndicesState contents start end st =
10271280 GetIndicesState contents start end st
10281281
1029−-- | Given an unfold that generates array indices, read the elements on those
1030−-- indices from the supplied MutArray. An error is thrown if an index is out of
1031−-- bounds.
1032−--
1033−-- /Pre-release/
1034−{-# INLINE getIndicesD #-}
1035−getIndicesD :: (Monad m, Unbox a) =>
1282+{-# INLINE indexReaderWith #-}
1283+indexReaderWith :: (Monad m, Unbox a) =>
10361284 (forall b. IO b -> m b) -> D.Stream m Int -> Unfold m (MutArray a) a
1037−getIndicesD liftio (D.Stream stepi sti) = Unfold step inject
1285+indexReaderWith liftio (D.Stream stepi sti) = Unfold step inject
10381286
10391287 where
10401288
@@ -1053,9 +1301,26 @@ getIndicesD liftio (D.Stream stepi sti) = Unfold step inject
10531301 D.Skip s -> return $ D.Skip (GetIndicesState contents start end s)
10541302 D.Stop -> return D.Stop
10551303
1304+{-# DEPRECATED getIndicesWith "Please use indexReaderWith instead." #-}
1305+{-# INLINE getIndicesWith #-}
1306+getIndicesWith :: (Monad m, Unbox a) =>
1307+ (forall b. IO b -> m b) -> D.Stream m Int -> Unfold m (MutArray a) a
1308+getIndicesWith = indexReaderWith
1309+
1310+-- | Given an unfold that generates array indices, read the elements on those
1311+-- indices from the supplied MutArray. An error is thrown if an index is out of
1312+-- bounds.
1313+--
1314+-- /Pre-release/
1315+{-# INLINE indexReader #-}
1316+indexReader :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1317+indexReader = indexReaderWith liftIO
1318+
1319+-- XXX DO NOT REMOVE, change the signature to use Stream instead of unfold
1320+{-# DEPRECATED getIndices "Please use indexReader instead." #-}
10561321{-# INLINE getIndices #-}
10571322getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1058−getIndices = getIndicesD liftIO
1323+getIndices = indexReader
10591324
10601325-------------------------------------------------------------------------------
10611326-- Subarrays
@@ -1160,7 +1425,7 @@ partitionBy f arr@MutArray{..} = liftIO $ do
11601425
11611426 -- Invariant low < high on entry, and on return as well
11621427 moveHigh low high = do
1163− h <- peekByteIndex high arrContents
1428+ h <- peekAt high arrContents
11641429 if f h
11651430 then
11661431 -- Correctly classified, continue the loop
@@ -1180,7 +1445,7 @@ partitionBy f arr@MutArray{..} = liftIO $ do
11801445 -- low <= high
11811446 -- Both low and high are valid locations within the array
11821447 go low high = do
1183− l <- peekByteIndex low arrContents
1448+ l <- peekAt low arrContents
11841449 if f l
11851450 then
11861451 -- low is wrongly classified
@@ -1191,8 +1456,8 @@ partitionBy f arr@MutArray{..} = liftIO $ do
11911456 case r of
11921457 Nothing -> return low
11931458 Just (high1, h) -> do -- low < high1
1194− pokeByteIndex low arrContents h
1195− pokeByteIndex high1 arrContents l
1459+ pokeAt low arrContents h
1460+ pokeAt high1 arrContents l
11961461 let low1 = INDEX_NEXT(low,a)
11971462 high2 = INDEX_PREV(high1,a)
11981463 if low1 <= high2
@@ -1326,7 +1591,7 @@ chunksOfAs ps n (D.Stream step state) =
13261591 r <- step (adaptState gst) st
13271592 case r of
13281593 D.Yield x s -> do
1329− liftIO $ pokeByteIndex end contents x
1594+ liftIO $ pokeAt end contents x
13301595 let end1 = INDEX_NEXT(end,a)
13311596 return $
13321597 if end1 >= bound
@@ -1345,10 +1610,12 @@ chunksOfAs ps n (D.Stream step state) =
13451610
13461611 step' _ GroupFinish = return D.Stop
13471612
1348−-- | @chunksOf n stream@ groups the input stream into a stream of
1349−-- arrays of size n.
1613+-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
1614+-- @n@ elements each.
13501615--
1351−-- @chunksOf n = StreamD.foldMany (MutArray.writeN n)@
1616+-- Same as the following but may be more efficient:
1617+--
1618+-- >>> chunksOf n = Stream.foldMany (MutArray.createOf n)
13521619--
13531620-- /Pre-release/
13541621{-# INLINE_NORMAL chunksOf #-}
@@ -1357,17 +1624,29 @@ chunksOf :: forall m a. (MonadIO m, Unbox a)
13571624-- XXX the idiomatic implementation leads to large regression in the D.reverse'
13581625-- benchmark. It seems it has difficulty producing optimized code when
13591626-- converting to StreamK. Investigate GHC optimizations.
1360−-- chunksOf n = D.foldMany (writeN n)
1627+-- chunksOf n = D.foldMany (createOf n)
13611628chunksOf = chunksOfAs Unpinned
13621629
13631630-- | Like 'chunksOf' but creates pinned arrays.
13641631{-# INLINE_NORMAL pinnedChunksOf #-}
13651632pinnedChunksOf :: forall m a. (MonadIO m, Unbox a)
13661633 => Int -> D.Stream m a -> D.Stream m (MutArray a)
1367−-- pinnedChunksOf n = D.foldMany (pinnedWriteN n)
1634+-- pinnedChunksOf n = D.foldMany (pinnedCreateOf n)
13681635pinnedChunksOf = chunksOfAs Pinned
13691636
1370−-- XXX This should take a PinnedState
1637+-- | When we are buffering a stream of unknown size into an array we do not
1638+-- know how much space to pre-allocate. So we start with the min size and emit
1639+-- the array then keep on doubling the size every time. Thus we do not need to
1640+-- guess the optimum chunk size.
1641+--
1642+-- We can incorporate this in chunksOfAs if the additional size parameter does
1643+-- not impact perf.
1644+--
1645+{-# INLINE _chunksOfRange #-}
1646+_chunksOfRange :: -- (MonadIO m, Unbox a) =>
1647+ PinnedState -> Int -> Int -> D.Stream m a -> D.Stream m (MutArray a)
1648+_chunksOfRange _ps _low _hi = undefined
1649+
13711650-- XXX buffer to a list instead?
13721651-- | Buffer the stream into arrays in memory.
13731652{-# INLINE arrayStreamKFromStreamDAs #-}
@@ -1385,16 +1664,10 @@ data FlattenState s contents a =
13851664 OuterLoop s
13861665 | InnerLoop s contents !Int !Int
13871666
1388−-- | Use the "reader" unfold instead.
1389−--
1390−-- @flattenArrays = unfoldMany reader@
1391−--
1392−-- We can try this if there are any fusion issues in the unfold.
1393−--
1394−{-# INLINE_NORMAL flattenArrays #-}
1395−flattenArrays :: forall m a. (MonadIO m, Unbox a)
1396− => D.Stream m (MutArray a) -> D.Stream m a
1397−flattenArrays (D.Stream step state) = D.Stream step' (OuterLoop state)
1667+{-# INLINE_NORMAL concatWith #-}
1668+concatWith :: forall m a. (Monad m, Unbox a)
1669+ => (forall b. IO b -> m b) -> D.Stream m (MutArray a) -> D.Stream m a
1670+concatWith liftio (D.Stream step state) = D.Stream step' (OuterLoop state)
13981671
13991672 where
14001673
@@ -1411,19 +1684,30 @@ flattenArrays (D.Stream step state) = D.Stream step' (OuterLoop state)
14111684 return $ D.Skip $ OuterLoop st
14121685
14131686 step' _ (InnerLoop st contents p end) = do
1414− x <- liftIO $ peekByteIndex p contents
1687+ x <- liftio $ peekAt p contents
14151688 return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
14161689
1417−-- | Use the "readerRev" unfold instead.
1690+-- | Use the "reader" unfold instead.
14181691--
1419−-- @flattenArrays = unfoldMany readerRev@
1692+-- @concat = unfoldMany reader@
14201693--
14211694-- We can try this if there are any fusion issues in the unfold.
14221695--
1423−{-# INLINE_NORMAL flattenArraysRev #-}
1424−flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
1696+{-# INLINE_NORMAL concat #-}
1697+concat :: forall m a. (MonadIO m, Unbox a)
1698+ => D.Stream m (MutArray a) -> D.Stream m a
1699+concat = concatWith liftIO
1700+
1701+{-# DEPRECATED flattenArrays "Please use \"unfoldMany reader\" instead." #-}
1702+{-# INLINE flattenArrays #-}
1703+flattenArrays :: forall m a. (MonadIO m, Unbox a)
14251704 => D.Stream m (MutArray a) -> D.Stream m a
1426−flattenArraysRev (D.Stream step state) = D.Stream step' (OuterLoop state)
1705+flattenArrays = concat
1706+
1707+{-# INLINE_NORMAL concatRevWith #-}
1708+concatRevWith :: forall m a. (Monad m, Unbox a)
1709+ => (forall b. IO b -> m b) -> D.Stream m (MutArray a) -> D.Stream m a
1710+concatRevWith liftio (D.Stream step state) = D.Stream step' (OuterLoop state)
14271711
14281712 where
14291713
@@ -1441,16 +1725,33 @@ flattenArraysRev (D.Stream step state) = D.Stream step' (OuterLoop state)
14411725 return $ D.Skip $ OuterLoop st
14421726
14431727 step' _ (InnerLoop st contents p start) = do
1444− x <- liftIO $ peekByteIndex p contents
1728+ x <- liftio $ peekAt p contents
14451729 let cur = INDEX_PREV(p,a)
14461730 return $ D.Yield x (InnerLoop st contents cur start)
14471731
1732+-- | Use the "readerRev" unfold instead.
1733+--
1734+-- @concat = unfoldMany readerRev@
1735+--
1736+-- We can try this if there are any fusion issues in the unfold.
1737+--
1738+{-# INLINE_NORMAL concatRev #-}
1739+concatRev :: forall m a. (MonadIO m, Unbox a)
1740+ => D.Stream m (MutArray a) -> D.Stream m a
1741+concatRev = concatRevWith liftIO
1742+
1743+{-# DEPRECATED flattenArraysRev "Please use \"unfoldMany readerRev\" instead." #-}
1744+{-# INLINE flattenArraysRev #-}
1745+flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
1746+ => D.Stream m (MutArray a) -> D.Stream m a
1747+flattenArraysRev = concatRev
1748+
14481749-------------------------------------------------------------------------------
14491750-- Unfolds
14501751-------------------------------------------------------------------------------
14511752
14521753data ArrayUnsafe a = ArrayUnsafe
1453− {-# UNPACK #-} !MutableByteArray -- contents
1754+ {-# UNPACK #-} !MutByteArray -- contents
14541755 {-# UNPACK #-} !Int -- index 1
14551756 {-# UNPACK #-} !Int -- index 2
14561757
@@ -1480,7 +1781,7 @@ producerWith liftio = Producer step (return . toArrayUnsafe) extract
14801781 -- few actions for correctness and execution order sanity. We want
14811782 -- the peek to occur right here and not lazily at some later point
14821783 -- because we want the peek to be ordered with respect to the touch.
1483− !x <- liftio $ peekByteIndex cur contents
1784+ !x <- liftio $ peekAt cur contents
14841785 return $ D.Yield x (ArrayUnsafe contents (INDEX_NEXT(cur,a)) end)
14851786
14861787 extract = return . fromArrayUnsafe
@@ -1511,7 +1812,7 @@ readerRevWith liftio = Unfold step inject
15111812 {-# INLINE_LATE step #-}
15121813 step (ArrayUnsafe _ start p) | p < start = return D.Stop
15131814 step (ArrayUnsafe contents start p) = do
1514− !x <- liftio $ peekByteIndex p contents
1815+ !x <- liftio $ peekAt p contents
15151816 return $ D.Yield x (ArrayUnsafe contents start (INDEX_PREV(p,a)))
15161817
15171818-- | Unfold an array into a stream in reverse order.
@@ -1542,7 +1843,7 @@ toListFB c n MutArray{..} = go arrStart
15421843 -- evaluated/written to before we peek at them.
15431844 -- XXX
15441845 let !x = unsafeInlineIO $ do
1545− r <- peekByteIndex arrContents p
1846+ r <- peekAt arrContents p
15461847 return r
15471848 in c x (go (PTR_NEXT(p,a)))
15481849-}
@@ -1559,21 +1860,21 @@ toList MutArray{..} = liftIO $ go arrStart
15591860
15601861 go p | assert (p <= arrEnd) (p == arrEnd) = return []
15611862 go p = do
1562− x <- peekByteIndex p arrContents
1863+ x <- peekAt p arrContents
15631864 (:) x <$> go (INDEX_NEXT(p,a))
15641865
1565−{-# INLINE_NORMAL toStreamDWith #-}
1566−toStreamDWith ::
1866+{-# INLINE_NORMAL toStreamWith #-}
1867+toStreamWith ::
15671868 forall m a. (Monad m, Unbox a)
15681869 => (forall b. IO b -> m b) -> MutArray a -> D.Stream m a
1569−toStreamDWith liftio MutArray{..} = D.Stream step arrStart
1870+toStreamWith liftio MutArray{..} = D.Stream step arrStart
15701871
15711872 where
15721873
15731874 {-# INLINE_LATE step #-}
15741875 step _ p | assert (p <= arrEnd) (p == arrEnd) = return D.Stop
15751876 step _ p = liftio $ do
1576− r <- peekByteIndex p arrContents
1877+ r <- peekAt p arrContents
15771878 return $ D.Yield r (INDEX_NEXT(p,a))
15781879
15791880-- | Convert a 'MutArray' into a stream.
@@ -1582,7 +1883,7 @@ toStreamDWith liftio MutArray{..} = D.Stream step arrStart
15821883--
15831884{-# INLINE_NORMAL read #-}
15841885read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> D.Stream m a
1585−read = toStreamDWith liftIO
1886+read = toStreamWith liftIO
15861887
15871888{-# INLINE toStreamKWith #-}
15881889toStreamKWith ::
@@ -1594,18 +1895,18 @@ toStreamKWith liftio MutArray{..} = go arrStart
15941895
15951896 go p | assert (p <= arrEnd) (p == arrEnd) = K.nil
15961897 | otherwise =
1597− let elemM = peekByteIndex p arrContents
1898+ let elemM = peekAt p arrContents
15981899 in liftio elemM `K.consM` go (INDEX_NEXT(p,a))
15991900
16001901{-# INLINE toStreamK #-}
16011902toStreamK :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
16021903toStreamK = toStreamKWith liftIO
16031904
1604−{-# INLINE_NORMAL toStreamDRevWith #-}
1605−toStreamDRevWith ::
1905+{-# INLINE_NORMAL toStreamRevWith #-}
1906+toStreamRevWith ::
16061907 forall m a. (Monad m, Unbox a)
16071908 => (forall b. IO b -> m b) -> MutArray a -> D.Stream m a
1608−toStreamDRevWith liftio MutArray{..} =
1909+toStreamRevWith liftio MutArray{..} =
16091910 let p = INDEX_PREV(arrEnd,a)
16101911 in D.Stream step p
16111912
@@ -1614,7 +1915,7 @@ toStreamDRevWith liftio MutArray{..} =
16141915 {-# INLINE_LATE step #-}
16151916 step _ p | p < arrStart = return D.Stop
16161917 step _ p = liftio $ do
1617− r <- peekByteIndex p arrContents
1918+ r <- peekAt p arrContents
16181919 return $ D.Yield r (INDEX_PREV(p,a))
16191920
16201921-- | Convert a 'MutArray' into a stream in reverse order.
@@ -1623,7 +1924,7 @@ toStreamDRevWith liftio MutArray{..} =
16231924--
16241925{-# INLINE_NORMAL readRev #-}
16251926readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> D.Stream m a
1626−readRev = toStreamDRevWith liftIO
1927+readRev = toStreamRevWith liftIO
16271928
16281929{-# INLINE toStreamKRevWith #-}
16291930toStreamKRevWith ::
@@ -1637,7 +1938,7 @@ toStreamKRevWith liftio MutArray {..} =
16371938
16381939 go p | p < arrStart = K.nil
16391940 | otherwise =
1640− let elemM = peekByteIndex p arrContents
1941+ let elemM = peekAt p arrContents
16411942 in liftio elemM `K.consM` go (INDEX_PREV(p,a))
16421943
16431944{-# INLINE toStreamKRev #-}
@@ -1672,7 +1973,7 @@ foldr f z arr = D.foldr f z $ read arr
16721973
16731974-- XXX Keep the bound intact to not lose any free space? Perf impact?
16741975
1675−-- | @writeAppendNUnsafe n arr@ appends up to @n@ input items to the supplied
1976+-- | @unsafeAppendN n arr@ appends up to @n@ input items to the supplied
16761977-- array.
16771978--
16781979-- Unsafe: Do not drive the fold beyond @n@ elements, it will lead to memory
@@ -1681,13 +1982,12 @@ foldr f z arr = D.foldr f z $ read arr
16811982-- Any free space left in the array after appending @n@ elements is lost.
16821983--
16831984-- /Internal/
1684−{-# INLINE_NORMAL writeAppendNUnsafe #-}
1685−writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) =>
1985+{-# INLINE_NORMAL unsafeAppendN #-}
1986+unsafeAppendN :: forall m a. (MonadIO m, Unbox a) =>
16861987 Int
16871988 -> m (MutArray a)
16881989 -> Fold m a (MutArray a)
1689−writeAppendNUnsafe n action =
1690− fmap fromArrayUnsafe $ FL.foldlM' step initial
1990+unsafeAppendN n action = fmap fromArrayUnsafe $ FL.foldlM' step initial
16911991
16921992 where
16931993
@@ -1700,38 +2000,59 @@ writeAppendNUnsafe n action =
17002000 -- otherwise we lose that space.
17012001 arr1 <-
17022002 if free < needed
1703− then noinline reallocWith "writeAppendNUnsafeWith" (+ needed) needed arr
2003+ then noinline reallocWith "unsafeAppendN" (+ needed) needed arr
17042004 else return arr
17052005 return $ toArrayUnsafe arr1
17062006
17072007 step (ArrayUnsafe contents start end) x = do
1708− liftIO $ pokeByteIndex end contents x
2008+ liftIO $ pokeAt end contents x
17092009 return $ ArrayUnsafe contents start (INDEX_NEXT(end,a))
17102010
2011+{-# DEPRECATED writeAppendNUnsafe "Please use unsafeAppendN instead." #-}
2012+{-# INLINE writeAppendNUnsafe #-}
2013+writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) =>
2014+ Int
2015+ -> m (MutArray a)
2016+ -> Fold m a (MutArray a)
2017+writeAppendNUnsafe = unsafeAppendN
2018+
17112019-- | Append @n@ elements to an existing array. Any free space left in the array
17122020-- after appending @n@ elements is lost.
17132021--
1714−-- >>> writeAppendN n initial = Fold.take n (MutArray.writeAppendNUnsafe n initial)
2022+-- >>> appendN n initial = Fold.take n (MutArray.unsafeAppendN n initial)
17152023--
1716−{-# INLINE_NORMAL writeAppendN #-}
2024+{-# INLINE_NORMAL appendN #-}
2025+appendN :: forall m a. (MonadIO m, Unbox a) =>
2026+ Int -> m (MutArray a) -> Fold m a (MutArray a)
2027+appendN n initial = FL.take n (unsafeAppendN n initial)
2028+
2029+-- XXX Deprecate in major
2030+-- {-# DEPRECATED writeAppendN "Please use appendN instead." #-}
2031+{-# INLINE writeAppendN #-}
17172032writeAppendN :: forall m a. (MonadIO m, Unbox a) =>
17182033 Int -> m (MutArray a) -> Fold m a (MutArray a)
1719−writeAppendN n initial = FL.take n (writeAppendNUnsafe n initial)
2034+writeAppendN = appendN
17202035
1721−-- | @writeAppendWith realloc action@ mutates the array generated by @action@ to
2036+-- | @appendWith realloc action@ mutates the array generated by @action@ to
17222037-- append the input stream. If there is no reserved space available in the
17232038-- array it is reallocated to a size in bytes determined by @realloc oldSize@,
17242039-- where @oldSize@ is the current size of the array in bytes.
17252040--
17262041-- Note that the returned array may be a mutated version of original array.
17272042--
1728−-- >>> writeAppendWith sizer = Fold.foldlM' (MutArray.snocWith sizer)
2043+-- >>> appendWith sizer = Fold.foldlM' (MutArray.snocWith sizer)
17292044--
17302045-- /Pre-release/
2046+{-# INLINE appendWith #-}
2047+appendWith :: forall m a. (MonadIO m, Unbox a) =>
2048+ (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
2049+appendWith sizer = FL.foldlM' (snocWith sizer)
2050+
2051+{-# DEPRECATED writeAppendWith "Please use appendWith instead." #-}
17312052{-# INLINE writeAppendWith #-}
17322053writeAppendWith :: forall m a. (MonadIO m, Unbox a) =>
17332054 (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
1734−writeAppendWith sizer = FL.foldlM' (snocWith sizer)
2055+writeAppendWith = appendWith
17352056
17362057-- | @append action@ mutates the array generated by @action@ to append the
17372058-- input stream. If there is no reserved space available in the array it is
@@ -1739,70 +2060,101 @@ writeAppendWith sizer = FL.foldlM' (snocWith sizer)
17392060--
17402061-- Note that the returned array may be a mutated version of original array.
17412062--
1742−-- >>> writeAppend = MutArray.writeAppendWith (* 2)
2063+-- >>> append = MutArray.appendWith (* 2)
17432064--
2065+{-# INLINE append #-}
2066+append :: forall m a. (MonadIO m, Unbox a) =>
2067+ m (MutArray a) -> Fold m a (MutArray a)
2068+append = appendWith (* 2)
2069+
2070+-- XXX Deprecate in major
2071+-- {-# DEPRECATED writeAppend "Please use append instead." #-}
17442072{-# INLINE writeAppend #-}
17452073writeAppend :: forall m a. (MonadIO m, Unbox a) =>
17462074 m (MutArray a) -> Fold m a (MutArray a)
1747−writeAppend = writeAppendWith (* 2)
2075+writeAppend = append
17482076
17492077-- XXX We can carry bound as well in the state to make sure we do not lose the
17502078-- remaining capacity. Need to check perf impact.
17512079--
1752−-- | Like 'writeNUnsafe' but takes a new array allocator @alloc size@ function
1753−-- as argument.
2080+-- | Like 'unsafeCreateOf' but takes a new array allocator @alloc size@
2081+-- function as argument.
17542082--
1755−-- >>> writeNWithUnsafe alloc n = MutArray.writeAppendNUnsafe (alloc n) n
2083+-- >>> unsafeCreateOfWith alloc n = MutArray.unsafeAppendN (alloc n) n
17562084--
17572085-- /Pre-release/
1758−{-# INLINE_NORMAL writeNWithUnsafe #-}
1759−writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a)
2086+{-# INLINE_NORMAL unsafeCreateOfWith #-}
2087+unsafeCreateOfWith :: forall m a. (MonadIO m, Unbox a)
17602088 => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1761−writeNWithUnsafe alloc n = fromArrayUnsafe <$> FL.foldlM' step initial
2089+unsafeCreateOfWith alloc n = fromArrayUnsafe <$> FL.foldlM' step initial
17622090
17632091 where
17642092
17652093 initial = toArrayUnsafe <$> alloc (max n 0)
17662094
17672095 step (ArrayUnsafe contents start end) x = do
1768− liftIO $ pokeByteIndex end contents x
2096+ liftIO $ pokeAt end contents x
17692097 return
17702098 $ ArrayUnsafe contents start (INDEX_NEXT(end,a))
17712099
2100+{-# DEPRECATED writeNWithUnsafe "Please use unsafeCreateOfWith instead." #-}
2101+{-# INLINE writeNWithUnsafe #-}
2102+writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a)
2103+ => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
2104+writeNWithUnsafe = unsafeCreateOfWith
2105+
17722106{-# INLINE_NORMAL writeNUnsafeAs #-}
17732107writeNUnsafeAs :: forall m a. (MonadIO m, Unbox a)
17742108 => PinnedState -> Int -> Fold m a (MutArray a)
1775−writeNUnsafeAs ps = writeNWithUnsafe (newAs ps)
2109+writeNUnsafeAs ps = unsafeCreateOfWith (newAs ps)
17762110
1777−-- | Like 'writeN' but does not check the array bounds when writing. The fold
2111+-- | Like 'createOf' but does not check the array bounds when writing. The fold
17782112-- driver must not call the step function more than 'n' times otherwise it will
17792113-- corrupt the memory and crash. This function exists mainly because any
17802114-- conditional in the step function blocks fusion causing 10x performance
17812115-- slowdown.
17822116--
1783−-- >>> writeNUnsafe = MutArray.writeNWithUnsafe MutArray.new
2117+-- >>> unsafeCreateOf = MutArray.unsafeCreateOfWith MutArray.emptyOf
17842118--
1785−{-# INLINE_NORMAL writeNUnsafe #-}
2119+{-# INLINE_NORMAL unsafeCreateOf #-}
2120+unsafeCreateOf :: forall m a. (MonadIO m, Unbox a)
2121+ => Int -> Fold m a (MutArray a)
2122+unsafeCreateOf = writeNUnsafeAs Unpinned
2123+
2124+{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
2125+{-# INLINE writeNUnsafe #-}
17862126writeNUnsafe :: forall m a. (MonadIO m, Unbox a)
17872127 => Int -> Fold m a (MutArray a)
1788−writeNUnsafe = writeNUnsafeAs Unpinned
2128+writeNUnsafe = unsafeCreateOf
2129+
2130+-- | Like 'unsafeCreateOf' but creates a pinned array.
2131+{-# INLINE_NORMAL unsafePinnedCreateOf #-}
2132+unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a)
2133+ => Int -> Fold m a (MutArray a)
2134+unsafePinnedCreateOf = writeNUnsafeAs Pinned
17892135
1790−-- | Like 'writeNUnsafe' but creates a pinned array.
1791−{-# INLINE_NORMAL pinnedWriteNUnsafe #-}
2136+{-# DEPRECATED pinnedWriteNUnsafe "Please use unsafePinnedCreateOf instead." #-}
2137+{-# INLINE pinnedWriteNUnsafe #-}
17922138pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a)
17932139 => Int -> Fold m a (MutArray a)
1794−pinnedWriteNUnsafe = writeNUnsafeAs Pinned
2140+pinnedWriteNUnsafe = unsafePinnedCreateOf
17952141
1796−-- | @writeNWith alloc n@ folds a maximum of @n@ elements into an array
2142+-- | @createOfWith alloc n@ folds a maximum of @n@ elements into an array
17972143-- allocated using the @alloc@ function.
17982144--
1799−-- >>> writeNWith alloc n = Fold.take n (MutArray.writeNWithUnsafe alloc n)
1800−-- >>> writeNWith alloc n = MutArray.writeAppendN (alloc n) n
2145+-- >>> createOfWith alloc n = Fold.take n (MutArray.unsafeCreateOfWith alloc n)
2146+-- >>> createOfWith alloc n = MutArray.appendN (alloc n) n
18012147--
1802−{-# INLINE_NORMAL writeNWith #-}
2148+{-# INLINE_NORMAL createOfWith #-}
2149+createOfWith :: forall m a. (MonadIO m, Unbox a)
2150+ => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
2151+createOfWith alloc n = FL.take n (unsafeCreateOfWith alloc n)
2152+
2153+-- {-# DEPRECATED writeNWith "Please use createOfWith instead." #-}
2154+{-# INLINE writeNWith #-}
18032155writeNWith :: forall m a. (MonadIO m, Unbox a)
18042156 => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1805−writeNWith alloc n = FL.take n (writeNWithUnsafe alloc n)
2157+writeNWith = createOfWith
18062158
18072159{-# INLINE_NORMAL writeNAs #-}
18082160writeNAs ::
@@ -1810,28 +2162,42 @@ writeNAs ::
18102162 => PinnedState
18112163 -> Int
18122164 -> Fold m a (MutArray a)
1813−writeNAs ps = writeNWith (newAs ps)
2165+writeNAs ps = createOfWith (newAs ps)
18142166
1815−-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
2167+-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
18162168-- 'MutArray'.
18172169--
1818−-- >>> writeN = MutArray.writeNWith MutArray.new
1819−-- >>> writeN n = Fold.take n (MutArray.writeNUnsafe n)
1820−-- >>> writeN n = MutArray.writeAppendN n (MutArray.new n)
2170+-- >>> createOf = MutArray.createOfWith MutArray.new
2171+-- >>> createOf n = Fold.take n (MutArray.unsafeCreateOf n)
2172+-- >>> createOf n = MutArray.appendN n (MutArray.emptyOf n)
18212173--
1822−{-# INLINE_NORMAL writeN #-}
2174+{-# INLINE_NORMAL createOf #-}
2175+createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
2176+createOf = writeNAs Unpinned
2177+
2178+-- XXX Deprecate in major
2179+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
2180+{-# INLINE writeN #-}
18232181writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1824−writeN = writeNAs Unpinned
2182+writeN = createOf
18252183
1826−-- | Like 'writeN' but creates a pinned array.
1827−{-# INLINE_NORMAL pinnedWriteN #-}
2184+-- | Like 'createOf' but creates a pinned array.
2185+{-# INLINE_NORMAL pinnedCreateOf #-}
2186+pinnedCreateOf ::
2187+ forall m a. (MonadIO m, Unbox a)
2188+ => Int
2189+ -> Fold m a (MutArray a)
2190+pinnedCreateOf = writeNAs Pinned
2191+
2192+{-# DEPRECATED pinnedWriteN "Please use pinnedCreateOf instead." #-}
2193+{-# INLINE pinnedWriteN #-}
18282194pinnedWriteN ::
18292195 forall m a. (MonadIO m, Unbox a)
18302196 => Int
18312197 -> Fold m a (MutArray a)
1832−pinnedWriteN = writeNAs Pinned
2198+pinnedWriteN = pinnedCreateOf
18332199
1834−-- | Like writeNWithUnsafe but writes the array in reverse order.
2200+-- | Like unsafeCreateOfWith but writes the array in reverse order.
18352201--
18362202-- /Internal/
18372203{-# INLINE_NORMAL writeRevNWithUnsafe #-}
@@ -1848,11 +2214,11 @@ writeRevNWithUnsafe alloc n = fromArrayUnsafe <$> FL.foldlM' step initial
18482214
18492215 step (ArrayUnsafe contents start end) x = do
18502216 let ptr = INDEX_PREV(start,a)
1851− liftIO $ pokeByteIndex ptr contents x
2217+ liftIO $ pokeAt ptr contents x
18522218 return
18532219 $ ArrayUnsafe contents ptr end
18542220
1855−-- | Like writeNWith but writes the array in reverse order.
2221+-- | Like createOfWith but writes the array in reverse order.
18562222--
18572223-- /Internal/
18582224{-# INLINE_NORMAL writeRevNWith #-}
@@ -1860,31 +2226,36 @@ writeRevNWith :: forall m a. (MonadIO m, Unbox a)
18602226 => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
18612227writeRevNWith alloc n = FL.take n (writeRevNWithUnsafe alloc n)
18622228
1863−-- | Like writeN but writes the array in reverse order.
2229+-- | Like 'createOf' but writes the array in reverse order.
18642230--
18652231-- /Pre-release/
1866−{-# INLINE_NORMAL writeRevN #-}
2232+{-# INLINE_NORMAL revCreateOf #-}
2233+revCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
2234+revCreateOf = writeRevNWith new
2235+
2236+{-# DEPRECATED writeRevN "Please use revCreateOf instead." #-}
2237+{-# INLINE writeRevN #-}
18672238writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1868−writeRevN = writeRevNWith new
2239+writeRevN = revCreateOf
18692240
18702241-- | @pinnedWriteNAligned align n@ folds a maximum of @n@ elements from the
18712242-- input stream to a 'MutArray' aligned to the given size.
18722243--
1873−-- >>> pinnedWriteNAligned align = MutArray.writeNWith (MutArray.pinnedNewAligned align)
1874−-- >>> pinnedWriteNAligned align n = MutArray.writeAppendN n (MutArray.pinnedNewAligned align n)
2244+-- >>> pinnedWriteNAligned align = MutArray.createOfWith (MutArray.pinnedNewAligned align)
2245+-- >>> pinnedWriteNAligned align n = MutArray.appendN n (MutArray.pinnedNewAligned align n)
18752246--
18762247-- /Pre-release/
18772248--
18782249{-# INLINE_NORMAL pinnedWriteNAligned #-}
18792250pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a)
18802251 => Int -> Int -> Fold m a (MutArray a)
1881−pinnedWriteNAligned align = writeNWith (pinnedNewAligned align)
2252+pinnedWriteNAligned align = createOfWith (pinnedNewAligned align)
18822253
18832254-- XXX Buffer to a list instead?
1884−--
2255+
18852256-- | Buffer a stream into a stream of arrays.
18862257--
1887−-- >>> writeChunks n = Fold.many (MutArray.writeN n) Fold.toStreamK
2258+-- >>> buildChunks n = Fold.many (MutArray.createOf n) Fold.toStreamK
18882259--
18892260-- Breaking an array into an array stream can be useful to consume a large
18902261-- array sequentially such that memory of the array is released incrementatlly.
@@ -1893,37 +2264,43 @@ pinnedWriteNAligned align = writeNWith (pinnedNewAligned align)
18932264--
18942265-- /Unimplemented/
18952266--
1896−{-# INLINE_NORMAL writeChunks #-}
2267+{-# INLINE_NORMAL buildChunks #-}
2268+buildChunks :: (MonadIO m, Unbox a) =>
2269+ Int -> Fold m a (StreamK n (MutArray a))
2270+buildChunks n = FL.many (createOf n) FL.toStreamK
2271+
2272+{-# DEPRECATED writeChunks "Please use buildChunks instead." #-}
2273+{-# INLINE writeChunks #-}
18972274writeChunks :: (MonadIO m, Unbox a) =>
18982275 Int -> Fold m a (StreamK n (MutArray a))
1899−writeChunks n = FL.many (writeN n) FL.toStreamK
2276+writeChunks = buildChunks
19002277
19012278{-# INLINE_NORMAL writeWithAs #-}
19022279writeWithAs :: forall m a. (MonadIO m, Unbox a)
19032280 => PinnedState -> Int -> Fold m a (MutArray a)
1904−-- writeWithAs ps n = FL.rmapM rightSize $ writeAppendWith (* 2) (newAs ps n)
2281+-- writeWithAs ps n = FL.rmapM rightSize $ appendWith (* 2) (newAs ps n)
19052282writeWithAs ps elemCount =
19062283 FL.rmapM extract $ FL.foldlM' step initial
19072284
19082285 where
19092286
19102287 initial = do
1911− when (elemCount < 0) $ error "writeWith: elemCount is negative"
1912− liftIO $ newAs ps elemCount
2288+ when (elemCount < 0) $ error "createWith: elemCount is negative"
2289+ newAs ps elemCount
19132290
19142291 step arr@(MutArray _ start end bound) x
19152292 | INDEX_NEXT(end,a) > bound = do
19162293 let oldSize = end - start
19172294 newSize = max (oldSize * 2) 1
1918− arr1 <- liftIO $ reallocExplicit (SIZE_OF(a)) newSize arr
2295+ arr1 <- liftIO $ reallocExplicitAs ps (SIZE_OF(a)) newSize arr
19192296 snocUnsafe arr1 x
19202297 step arr x = snocUnsafe arr x
19212298
19222299 extract = liftIO . rightSize
19232300
1924−-- XXX Compare writeWith with fromStreamD which uses an array of streams
1925−-- implementation. We can write this using writeChunks above if that is faster.
1926−-- If writeWith is faster then we should use that to implement
2301+-- XXX Compare createWith with fromStreamD which uses an array of streams
2302+-- implementation. We can write this using buildChunks above if that is faster.
2303+-- If createWith is faster then we should use that to implement
19272304-- fromStreamD.
19282305--
19292306-- XXX The realloc based implementation needs to make one extra copy if we use
@@ -1938,39 +2315,56 @@ writeWithAs ps elemCount =
19382315-- XXX check if GHC's memory allocator is efficient enough. We can try the C
19392316-- malloc to compare against.
19402317
1941−-- | @writeWith minCount@ folds the whole input to a single array. The array
2318+-- | @createWith minCount@ folds the whole input to a single array. The array
19422319-- starts at a size big enough to hold minCount elements, the size is doubled
19432320-- every time the array needs to be grown.
19442321--
19452322-- /Caution! Do not use this on infinite streams./
19462323--
1947−-- >>> f n = MutArray.writeAppendWith (* 2) (MutArray.new n)
1948−-- >>> writeWith n = Fold.rmapM MutArray.rightSize (f n)
1949−-- >>> writeWith n = Fold.rmapM MutArray.fromArrayStreamK (MutArray.writeChunks n)
2324+-- >>> f n = MutArray.appendWith (* 2) (MutArray.emptyOf n)
2325+-- >>> createWith n = Fold.rmapM MutArray.rightSize (f n)
2326+-- >>> createWith n = Fold.rmapM MutArray.fromChunksK (MutArray.buildChunks n)
19502327--
19512328-- /Pre-release/
1952−{-# INLINE_NORMAL writeWith #-}
2329+{-# INLINE_NORMAL createWith #-}
2330+createWith :: forall m a. (MonadIO m, Unbox a)
2331+ => Int -> Fold m a (MutArray a)
2332+-- createWith n = FL.rmapM rightSize $ appendWith (* 2) (emptyOf n)
2333+createWith = writeWithAs Unpinned
2334+
2335+{-# DEPRECATED writeWith "Please use createWith instead." #-}
2336+{-# INLINE writeWith #-}
19532337writeWith :: forall m a. (MonadIO m, Unbox a)
19542338 => Int -> Fold m a (MutArray a)
1955−-- writeWith n = FL.rmapM rightSize $ writeAppendWith (* 2) (new n)
1956−writeWith = writeWithAs Unpinned
2339+writeWith = createWith
19572340
19582341-- | Fold the whole input to a single array.
19592342--
1960−-- Same as 'writeWith' using an initial array size of 'arrayChunkBytes' bytes
2343+-- Same as 'createWith' using an initial array size of 'arrayChunkBytes' bytes
19612344-- rounded up to the element size.
19622345--
19632346-- /Caution! Do not use this on infinite streams./
19642347--
2348+{-# INLINE create #-}
2349+create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
2350+create = createWith (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
2351+
2352+-- XXX Deprecate in major
2353+-- {-# DEPRECATED write "Please use create instead." #-}
19652354{-# INLINE write #-}
19662355write :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1967−write = writeWith (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
2356+write = create
19682357
1969−-- | Like 'write' but creates a pinned array.
2358+-- | Like 'create' but creates a pinned array.
2359+{-# INLINE pinnedCreate #-}
2360+pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
2361+pinnedCreate =
2362+ writeWithAs Pinned (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
2363+
2364+{-# DEPRECATED pinnedWrite "Please use pinnedCreate instead." #-}
19702365{-# INLINE pinnedWrite #-}
19712366pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1972−pinnedWrite =
1973− writeWithAs Pinned (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
2367+pinnedWrite = pinnedCreate
19742368
19752369-------------------------------------------------------------------------------
19762370-- construct from streams, known size
@@ -1980,25 +2374,31 @@ pinnedWrite =
19802374fromStreamDNAs :: forall m a. (MonadIO m, Unbox a)
19812375 => PinnedState -> Int -> D.Stream m a -> m (MutArray a)
19822376fromStreamDNAs ps limit str = do
1983− (arr :: MutArray a) <- liftIO $ newAs ps limit
2377+ (arr :: MutArray a) <- newAs ps limit
19842378 end <- D.foldlM' (fwrite (arrContents arr)) (return $ arrEnd arr) $ D.take limit str
19852379 return $ arr {arrEnd = end}
19862380
19872381 where
19882382
19892383 fwrite arrContents ptr x = do
1990− liftIO $ pokeByteIndex ptr arrContents x
2384+ liftIO $ pokeAt ptr arrContents x
19912385 return $ INDEX_NEXT(ptr,a)
19922386
1993−-- | Use the 'writeN' fold instead.
2387+-- | Use the 'createOf' fold instead.
19942388--
1995−-- >>> fromStreamDN n = Stream.fold (MutArray.writeN n)
2389+-- >>> fromStreamN n = Stream.fold (MutArray.createOf n)
19962390--
1997−{-# INLINE_NORMAL fromStreamDN #-}
2391+{-# INLINE_NORMAL fromStreamN #-}
2392+fromStreamN :: forall m a. (MonadIO m, Unbox a)
2393+ => Int -> D.Stream m a -> m (MutArray a)
2394+-- fromStreamDN n = D.fold (createOf n)
2395+fromStreamN = fromStreamDNAs Unpinned
2396+
2397+{-# DEPRECATED fromStreamDN "Please use fromStreamN instead." #-}
2398+{-# INLINE fromStreamDN #-}
19982399fromStreamDN :: forall m a. (MonadIO m, Unbox a)
19992400 => Int -> D.Stream m a -> m (MutArray a)
2000−-- fromStreamDN n = D.fold (writeN n)
2001−fromStreamDN = fromStreamDNAs Unpinned
2401+fromStreamDN = fromStreamN
20022402
20032403-- | Create a 'MutArray' from the first N elements of a list. The array is
20042404-- allocated to size N, if the list terminates before N elements then the
@@ -2018,16 +2418,70 @@ pinnedFromListN n xs = fromStreamDNAs Pinned n $ D.fromList xs
20182418-- /Pre-release/
20192419{-# INLINE fromListRevN #-}
20202420fromListRevN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
2021−fromListRevN n xs = D.fold (writeRevN n) $ D.fromList xs
2421+fromListRevN n xs = D.fold (revCreateOf n) $ D.fromList xs
2422+
2423+-- | Convert a pure stream in Identity monad to a mutable array.
2424+{-# INLINABLE fromPureStreamN #-}
2425+fromPureStreamN :: (MonadIO m, Unbox a) =>
2426+ Int -> Stream Identity a -> m (MutArray a)
2427+fromPureStreamN n xs =
2428+ D.fold (createOf n) $ D.morphInner (return . runIdentity) xs
20222429
20232430-- | Convert a pure stream in Identity monad to a mutable array.
20242431{-# INLINABLE fromPureStream #-}
20252432fromPureStream :: (MonadIO m, Unbox a) => Stream Identity a -> m (MutArray a)
20262433fromPureStream xs =
2027− liftIO $ D.fold write $ D.morphInner (return . runIdentity) xs
2434+ D.fold create $ D.morphInner (return . runIdentity) xs
2435+
2436+{-# INLINABLE fromPtrN #-}
2437+fromPtrN :: MonadIO m => Int -> Ptr Word8 -> m (MutArray Word8)
2438+fromPtrN len addr = do
2439+ -- memcpy is better than stream copy when the size is known.
2440+ -- XXX We can implement a stream copy in a similar way by streaming Word64
2441+ -- first and then remaining Word8.
2442+ arr <- new len
2443+ _ <- unsafeAsPtr arr
2444+ (\ptr -> liftIO $ c_memcpy ptr addr (fromIntegral len))
2445+ return (arr {arrEnd = len})
2446+
2447+{-# INLINABLE fromByteStr# #-}
2448+fromByteStr# :: MonadIO m => Addr# -> m (MutArray Word8)
2449+fromByteStr# addr = do
2450+ -- It is better to count the size first and allocate exact space.
2451+ -- Also, memcpy is better than stream copy when the size is known.
2452+ -- C strlen compares 4 bytes at a time, so is better than the stream
2453+ -- version. https://github.com/bminor/glibc/blob/master/string/strlen.c
2454+ -- XXX We can possibly use a stream of Word64 to do the same.
2455+ -- fromByteStr# addr = fromPureStream (D.fromByteStr# addr)
2456+ len <- liftIO $ c_strlen (Ptr addr)
2457+ let lenInt = fromIntegral len
2458+ arr <- new lenInt
2459+ _ <- unsafeAsPtr arr (\ptr -> liftIO $ c_memcpy ptr (Ptr addr) len)
2460+ return (arr {arrEnd = lenInt})
20282461
20292462-------------------------------------------------------------------------------
2030−-- convert stream to a single array
2463+-- convert a stream of arrays to a single array by reallocating and copying
2464+-------------------------------------------------------------------------------
2465+
2466+-- XXX Both of these implementations of splicing seem to perform equally well.
2467+-- We need to perform benchmarks over a range of sizes though.
2468+
2469+-- | Also see 'fromChunksK'.
2470+{-# INLINE fromChunksRealloced #-}
2471+fromChunksRealloced :: forall m a. (MonadIO m, Unbox a)
2472+ => Stream m (MutArray a) -> m (MutArray a)
2473+fromChunksRealloced s = do
2474+ res <- D.uncons s
2475+ case res of
2476+ Just (a, strm) -> do
2477+ arr <- D.foldlM' spliceExp (pure a) strm
2478+ -- Reallocation is exponential so there may be 50% empty space in
2479+ -- worst case. One more reallocation to reclaim the space.
2480+ rightSize arr
2481+ Nothing -> pure nil
2482+
2483+-------------------------------------------------------------------------------
2484+-- convert a stream of arrays to a single array by buffering arrays first
20312485-------------------------------------------------------------------------------
20322486
20332487{-# INLINE arrayStreamKLength #-}
@@ -2037,34 +2491,72 @@ arrayStreamKLength as = K.foldl' (+) 0 (K.map length as)
20372491-- | Convert an array stream to an array. Note that this requires peak memory
20382492-- that is double the size of the array stream.
20392493--
2494+{-# INLINE fromChunkskAs #-}
2495+fromChunkskAs :: (Unbox a, MonadIO m) =>
2496+ PinnedState -> StreamK m (MutArray a) -> m (MutArray a)
2497+fromChunkskAs ps as = do
2498+ len <- arrayStreamKLength as
2499+ arr <- newAs ps len
2500+ -- XXX is StreamK fold faster or StreamD fold?
2501+ K.foldlM' spliceUnsafe (pure arr) as
2502+ -- fromStreamDN len $ D.unfoldMany reader $ D.fromStreamK as
2503+
2504+-- XXX Need to compare this with fromChunks and fromChunkList and keep the
2505+-- fastest or simplest one if all are equally fast.
2506+
2507+-- | Convert an array stream to an array. Note that this requires peak memory
2508+-- that is double the size of the array stream.
2509+--
2510+-- Also see 'fromChunksRealloced'.
2511+--
2512+{-# INLINE fromChunksK #-}
2513+fromChunksK :: (Unbox a, MonadIO m) =>
2514+ StreamK m (MutArray a) -> m (MutArray a)
2515+fromChunksK = fromChunkskAs Unpinned
2516+
2517+{-# DEPRECATED fromArrayStreamK "Please use fromChunksK instead." #-}
20402518{-# INLINE fromArrayStreamK #-}
20412519fromArrayStreamK :: (Unbox a, MonadIO m) =>
20422520 StreamK m (MutArray a) -> m (MutArray a)
2043−fromArrayStreamK as = do
2044− len <- arrayStreamKLength as
2045− fromStreamDN len $ D.unfoldMany reader $ D.fromStreamK as
2521+fromArrayStreamK = fromChunksK
20462522
20472523{-# INLINE fromStreamDAs #-}
20482524fromStreamDAs ::
20492525 (MonadIO m, Unbox a) => PinnedState -> D.Stream m a -> m (MutArray a)
2050−fromStreamDAs ps m = arrayStreamKFromStreamDAs ps m >>= fromArrayStreamK
2526+fromStreamDAs ps m =
2527+ arrayStreamKFromStreamDAs Unpinned m >>= fromChunkskAs ps
2528+
2529+-- | Create an 'Array' from a stream. This is useful when we want to create a
2530+-- single array from a stream of unknown size. 'createOf' is at least twice
2531+-- as efficient when the size is already known.
2532+--
2533+-- Note that if the input stream is too large memory allocation for the array
2534+-- may fail. When the stream size is not known, `chunksOf` followed by
2535+-- processing of indvidual arrays in the resulting stream should be preferred.
2536+--
2537+-- /Pre-release/
2538+{-# INLINE fromStream #-}
2539+fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
2540+fromStream = fromStreamDAs Unpinned
20512541
2542+-- fromStream (Stream m) = P.fold create m
20522543-- CAUTION: a very large number (millions) of arrays can degrade performance
20532544-- due to GC overhead because we need to buffer the arrays before we flatten
20542545-- all the arrays.
20552546--
2056−-- XXX Compare if this is faster or "fold write".
2547+-- XXX Compare if this is faster or "fold create".
20572548--
20582549-- | We could take the approach of doubling the memory allocation on each
20592550-- overflow. This would result in more or less the same amount of copying as in
20602551-- the chunking approach. However, if we have to shrink in the end then it may
20612552-- result in an extra copy of the entire data.
20622553--
2063−-- >>> fromStreamD = StreamD.fold MutArray.write
2554+-- >>> fromStreamD = StreamD.fold MutArray.create
20642555--
20652556{-# INLINE fromStreamD #-}
2557+{-# DEPRECATED fromStreamD "Please use fromStream instead." #-}
20662558fromStreamD :: (MonadIO m, Unbox a) => D.Stream m a -> m (MutArray a)
2067−fromStreamD = fromStreamDAs Unpinned
2559+fromStreamD = fromStream
20682560
20692561-- | Create a 'MutArray' from a list. The list must be of finite size.
20702562--
@@ -2086,24 +2578,52 @@ fromListRev :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
20862578fromListRev xs = fromListRevN (Prelude.length xs) xs
20872579
20882580-------------------------------------------------------------------------------
2089−-- Combining
2581+-- Cloning
20902582-------------------------------------------------------------------------------
20912583
2092−-- | Put a sub range of a source array into a subrange of a destination array.
2093−-- This is not safe as it does not check the bounds.
2094−{-# INLINE putSliceUnsafe #-}
2095−putSliceUnsafe :: MonadIO m => MutArray a -> Int -> MutArray a -> Int -> Int -> m ()
2096−putSliceUnsafe src srcStartBytes dst dstStartBytes lenBytes = liftIO $ do
2097− assertM(lenBytes <= arrBound dst - dstStartBytes)
2098− assertM(lenBytes <= arrEnd src - srcStartBytes)
2099− let !(I# srcStartBytes#) = srcStartBytes
2100− !(I# dstStartBytes#) = dstStartBytes
2101− !(I# lenBytes#) = lenBytes
2102− let arrS# = getMutableByteArray# (arrContents src)
2103− arrD# = getMutableByteArray# (arrContents dst)
2104− IO $ \s# -> (# copyMutableByteArray#
2105− arrS# srcStartBytes# arrD# dstStartBytes# lenBytes# s#
2106− , () #)
2584+-- Arrays are aligned on 64-bit boundaries. The fastest way to copy an array is
2585+-- to unsafeCast it to Word64, read it, write it to Word64 array and unsafeCast
2586+-- it again. We can use SIMD read/write as well.
2587+
2588+{-# INLINE cloneAs #-}
2589+cloneAs ::
2590+ ( MonadIO m
2591+#ifdef DEVBUILD
2592+ , Unbox a
2593+#endif
2594+ )
2595+ => PinnedState -> MutArray a -> m (MutArray a)
2596+cloneAs ps src =
2597+ do
2598+ let startSrc = arrStart src
2599+ srcLen = arrEnd src - startSrc
2600+ newArrContents <-
2601+ Unboxed.cloneSliceUnsafeAs ps startSrc srcLen (arrContents src)
2602+ return $ MutArray newArrContents 0 srcLen srcLen
2603+
2604+{-# INLINE clone #-}
2605+clone ::
2606+ ( MonadIO m
2607+#ifdef DEVBUILD
2608+ , Unbox a
2609+#endif
2610+ )
2611+ => MutArray a -> m (MutArray a)
2612+clone = cloneAs Unpinned
2613+
2614+{-# INLINE pinnedClone #-}
2615+pinnedClone ::
2616+ ( MonadIO m
2617+#ifdef DEVBUILD
2618+ , Unbox a
2619+#endif
2620+ )
2621+ => MutArray a -> m (MutArray a)
2622+pinnedClone = cloneAs Pinned
2623+
2624+-------------------------------------------------------------------------------
2625+-- Combining
2626+-------------------------------------------------------------------------------
21072627
21082628-- | Copy two arrays into a newly allocated array. If the first array is pinned
21092629-- the spliced array is also pinned.
@@ -2113,21 +2633,19 @@ spliceCopy :: forall m a. MonadIO m =>
21132633 Unbox a =>
21142634#endif
21152635 MutArray a -> MutArray a -> m (MutArray a)
2116−spliceCopy arr1 arr2 = liftIO $ do
2636+spliceCopy arr1 arr2 = do
21172637 let start1 = arrStart arr1
21182638 start2 = arrStart arr2
21192639 len1 = arrEnd arr1 - start1
21202640 len2 = arrEnd arr2 - start2
2121− let newLen = len1 + len2
2641+ let len = len1 + len2
21222642 newArrContents <-
21232643 if Unboxed.isPinned (arrContents arr1)
2124− then Unboxed.pinnedNewBytes newLen
2125− else Unboxed.newBytes newLen
2126− let len = len1 + len2
2127− newArr = MutArray newArrContents 0 len len
2128− putSliceUnsafe arr1 start1 newArr 0 len1
2129− putSliceUnsafe arr2 start2 newArr len1 len2
2130− return newArr
2644+ then liftIO $ Unboxed.pinnedNew len
2645+ else liftIO $ Unboxed.new len
2646+ putSliceUnsafe (arrContents arr1) start1 newArrContents 0 len1
2647+ putSliceUnsafe (arrContents arr2) start2 newArrContents len1 len2
2648+ return $ MutArray newArrContents 0 len len
21312649
21322650-- | Really really unsafe, appends the second array into the first array. If
21332651-- the first array does not have enough space it may cause silent data
@@ -2136,12 +2654,13 @@ spliceCopy arr1 arr2 = liftIO $ do
21362654spliceUnsafe :: MonadIO m =>
21372655 MutArray a -> MutArray a -> m (MutArray a)
21382656spliceUnsafe dst src =
2139− liftIO $ do
2657+ do
21402658 let startSrc = arrStart src
21412659 srcLen = arrEnd src - startSrc
21422660 endDst = arrEnd dst
21432661 assertM(endDst + srcLen <= arrBound dst)
2144− putSliceUnsafe src startSrc dst endDst srcLen
2662+ putSliceUnsafe
2663+ (arrContents src) startSrc (arrContents dst) endDst srcLen
21452664 return $ dst {arrEnd = endDst + srcLen}
21462665
21472666-- | @spliceWith sizer dst src@ mutates @dst@ to append @src@. If there is no
@@ -2157,7 +2676,7 @@ spliceWith :: forall m a. (MonadIO m, Unbox a) =>
21572676 (Int -> Int -> Int) -> MutArray a -> MutArray a -> m (MutArray a)
21582677spliceWith sizer dst@(MutArray _ start end bound) src = do
21592678{-
2160− let f = writeAppendWith (`sizer` byteLength src) (return dst)
2679+ let f = appendWith (`sizer` byteLength src) (return dst)
21612680 in D.fold f (toStreamD src)
21622681-}
21632682 assert (end <= bound) (return ())
@@ -2173,7 +2692,7 @@ spliceWith sizer dst@(MutArray _ start end bound) src = do
21732692 $ "splice: newSize is less than the total size "
21742693 ++ "of arrays being appended. Please check the "
21752694 ++ "sizer function passed."
2176− liftIO $ realloc newSizeInBytes dst
2695+ realloc newSizeInBytes dst
21772696 else return dst
21782697 spliceUnsafe dst1 src
21792698
@@ -2210,14 +2729,24 @@ spliceExp = spliceWith (\l1 l2 -> max (l1 * 2) (l1 + l2))
22102729-- Splitting
22112730-------------------------------------------------------------------------------
22122731
2732+-- | Generate a stream of array slices using a predicate. The array element
2733+-- matching the predicate is dropped.
2734+--
2735+-- /Pre-release/
2736+{-# INLINE splitOn #-}
2737+splitOn :: (MonadIO m, Unbox a) =>
2738+ (a -> Bool) -> MutArray a -> Stream m (MutArray a)
2739+splitOn predicate arr =
2740+ fmap (\(i, len) -> getSliceUnsafe i len arr)
2741+ $ D.indexOnSuffix predicate (read arr)
2742+
22132743-- | Drops the separator byte
22142744{-# INLINE breakOn #-}
22152745breakOn :: MonadIO m
22162746 => Word8 -> MutArray Word8 -> m (MutArray Word8, Maybe (MutArray Word8))
2217−breakOn sep arr@MutArray{..} = asPtrUnsafe arr $ \p -> liftIO $ do
2218− -- XXX Instead of using asPtrUnsafe (pinning memory) we can pass unlifted
2219− -- Addr# to memchr and it should be safe (from ghc 8.4).
2747+breakOn sep arr@MutArray{..} = unsafeAsPtr arr $ \p -> liftIO $ do
22202748 -- XXX We do not need memchr here, we can use a Haskell equivalent.
2749+ -- Need efficient stream based primitives that work on Word64.
22212750 loc <- c_memchr p sep (fromIntegral $ byteLength arr)
22222751 let sepIndex = loc `minusPtr` p
22232752 return $
@@ -2238,32 +2767,41 @@ breakOn sep arr@MutArray{..} = asPtrUnsafe arr $ \p -> liftIO $ do
22382767 }
22392768 )
22402769
2770+-- | Like 'splitAt' but does not check whether the index is valid.
2771+--
2772+{-# INLINE unsafeSplitAt #-}
2773+unsafeSplitAt :: forall a. Unbox a =>
2774+ Int -> MutArray a -> (MutArray a, MutArray a)
2775+unsafeSplitAt i MutArray{..} =
2776+ let off = i * SIZE_OF(a)
2777+ p = arrStart + off
2778+ in ( MutArray
2779+ { arrContents = arrContents
2780+ , arrStart = arrStart
2781+ , arrEnd = p
2782+ , arrBound = p
2783+ }
2784+ , MutArray
2785+ { arrContents = arrContents
2786+ , arrStart = p
2787+ , arrEnd = arrEnd
2788+ , arrBound = arrBound
2789+ }
2790+ )
2791+
22412792-- | Create two slices of an array without copying the original array. The
22422793-- specified index @i@ is the first index of the second slice.
22432794--
2795+{-# INLINE splitAt #-}
22442796splitAt :: forall a. Unbox a => Int -> MutArray a -> (MutArray a, MutArray a)
2245−splitAt i arr@MutArray{..} =
2797+splitAt i arr =
22462798 let maxIndex = length arr - 1
22472799 in if i < 0
22482800 then error "sliceAt: negative array index"
22492801 else if i > maxIndex
22502802 then error $ "sliceAt: specified array index " ++ show i
22512803 ++ " is beyond the maximum index " ++ show maxIndex
2252− else let off = i * SIZE_OF(a)
2253− p = arrStart + off
2254− in ( MutArray
2255− { arrContents = arrContents
2256− , arrStart = arrStart
2257− , arrEnd = p
2258− , arrBound = p
2259− }
2260− , MutArray
2261− { arrContents = arrContents
2262− , arrStart = p
2263− , arrEnd = arrEnd
2264− , arrBound = arrBound
2265− }
2266− )
2804+ else unsafeSplitAt i arr
22672805
22682806-------------------------------------------------------------------------------
22692807-- Casting
@@ -2313,13 +2851,16 @@ cast arr =
23132851-- guarantees that unsafe calls will be performed in the calling thread. Making
23142852-- it safe to pass heap-allocated objects to unsafe functions.
23152853
2316−-- Unsafe because of direct pointer operations. The user must ensure that they
2317−-- are writing within the legal bounds of the array. Should we just name it
2318−-- asPtr, the unsafety is implicit for any pointer operations. And we are safe
2319−-- from Haskell perspective because we will be pinning the memory.
2854+-- XXX Should we just name it asPtr, the unsafety is implicit for any pointer
2855+-- operations. And we are safe from Haskell perspective because we will be
2856+-- pinning the memory.
2857+--
2858+-- XXX we cannot pass the length of the ptr here as in some cases it may not be
2859+-- available e.g. a null terminated C string. However, we can create another
2860+-- flavor of the API e.g. asPtrN.
23202861
2321−-- | Use an @MutArray a@ as @Ptr a@. This is useful when we want to pass an array
2322−-- as a pointer to some operating system call or to a "safe" FFI call.
2862+-- | Use a @MutArray a@ as @Ptr a@. This is useful when we want to pass an
2863+-- array as a pointer to some operating system call or to a "safe" FFI call.
23232864--
23242865-- If the array is not pinned it is copied to pinned memory before passing it
23252866-- to the monadic action.
@@ -2329,56 +2870,382 @@ cast arr =
23292870-- opportunistically before this operation occurs, to avoid the cost of a copy
23302871-- if possible.
23312872--
2332−-- /Unsafe/
2873+-- /Unsafe/ because of direct pointer operations. The user must ensure that
2874+-- they are writing within the legal bounds of the array.
23332875--
23342876-- /Pre-release/
23352877--
2878+{-# INLINE unsafePinnedAsPtr #-}
2879+unsafePinnedAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
2880+unsafePinnedAsPtr arr f =
2881+ Unboxed.unsafePinnedAsPtr
2882+ (arrContents arr) (\ptr -> f (ptr `plusPtr` arrStart arr))
2883+
2884+{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
2885+{-# INLINE asPtrUnsafe #-}
23362886asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
2337−asPtrUnsafe arr f = do
2338− contents <- liftIO $ Unboxed.pin $ arrContents arr
2339− let !ptr = Ptr (byteArrayContents#
2340− (unsafeCoerce# (getMutableByteArray# contents)))
2887+asPtrUnsafe = unsafePinnedAsPtr
23412888
2342− r <- f (ptr `plusPtr` arrStart arr)
2343− liftIO $ touch contents
2344− return r
2889+{-# INLINE unsafeAsPtr #-}
2890+unsafeAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
2891+unsafeAsPtr arr f =
2892+ Unboxed.unsafeAsPtr
2893+ (arrContents arr) (\ptr -> f (ptr `plusPtr` arrStart arr))
23452894
23462895-------------------------------------------------------------------------------
23472896-- Equality
23482897-------------------------------------------------------------------------------
23492898
2350−-- | Compare the length of the arrays. If the length is equal, compare the
2351−-- lexicographical ordering of two underlying byte arrays otherwise return the
2352−-- result of length comparison.
2899+-- | Byte compare two arrays. Compare the length of the arrays. If the length
2900+-- is equal, compare the lexicographical ordering of two underlying byte arrays
2901+-- otherwise return the result of length comparison.
2902+--
2903+-- /Unsafe/: Note that the 'Unbox' instance of sum types with constructors of
2904+-- different sizes may leave some memory uninitialized which can make byte
2905+-- comparison unreliable.
23532906--
23542907-- /Pre-release/
2908+{-# INLINE byteCmp #-}
2909+byteCmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
2910+byteCmp arr1 arr2 = do
2911+ let marr1 = getMutableByteArray# (arrContents arr1)
2912+ marr2 = getMutableByteArray# (arrContents arr2)
2913+ !(I# st1#) = arrStart arr1
2914+ !(I# st2#) = arrStart arr2
2915+ !(I# len#) = byteLength arr1
2916+ case compare (byteLength arr1) (byteLength arr2) of
2917+ EQ -> do
2918+ r <- liftIO $ IO $ \s# ->
2919+ let res =
2920+ I#
2921+ (compareByteArrays#
2922+ (unsafeCoerce# marr1)
2923+ st1#
2924+ (unsafeCoerce# marr2)
2925+ st2#
2926+ len#)
2927+ in (# s#, res #)
2928+ return $ compare r 0
2929+ x -> return x
2930+
23552931{-# INLINE cmp #-}
2932+{-# DEPRECATED cmp "Please use byteCmp instead." #-}
23562933cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
2357−cmp arr1 arr2 =
2358− liftIO
2359− $ do
2360− let marr1 = getMutableByteArray# (arrContents arr1)
2361− marr2 = getMutableByteArray# (arrContents arr2)
2362− !(I# st1#) = arrStart arr1
2363− !(I# st2#) = arrStart arr2
2364− !(I# len#) = byteLength arr1
2365− case compare (byteLength arr1) (byteLength arr2) of
2366− EQ -> do
2367− r <- liftIO $ IO $ \s# ->
2368− let res =
2369− I#
2370− (compareByteArrays#
2371− (unsafeCoerce# marr1)
2372− st1#
2373− (unsafeCoerce# marr2)
2374− st2#
2375− len#)
2376− in (# s#, res #)
2377− return $ compare r 0
2378− x -> return x
2934+cmp = byteCmp
2935+
2936+-- | Byte equality of two arrays.
2937+--
2938+-- >>> byteEq arr1 arr2 = (==) EQ $ MArray.byteCmp arr1 arr2
2939+--
2940+-- /Unsafe/: See 'byteCmp'.
2941+{-# INLINE byteEq #-}
2942+byteEq :: MonadIO m => MutArray a -> MutArray a -> m Bool
2943+byteEq arr1 arr2 = fmap (EQ ==) $ byteCmp arr1 arr2
2944+
2945+-------------------------------------------------------------------------------
2946+-- Compact
2947+-------------------------------------------------------------------------------
2948+
2949+-- Note: LE versions avoid an extra copy compared to GE. LE parser trades
2950+-- backtracking one array in lieu of avoiding a copy. However, LE and GE both
2951+-- can leave some memory unused. They can split the last array to fit it
2952+-- exactly in the space.
2953+
2954+{-# INLINE_NORMAL pCompactLeAs #-}
2955+pCompactLeAs ::
2956+ forall m a. (MonadIO m, Unbox a)
2957+ => PinnedState -> Int -> Parser (MutArray a) m (MutArray a)
2958+pCompactLeAs ps maxElems = Parser step initial extract
2959+
2960+ where
2961+
2962+ maxBytes = maxElems * SIZE_OF(a)
2963+
2964+ functionName = "Streamly.Internal.Data.MutArray.pCompactLE"
2965+
2966+ initial =
2967+ return
2968+ $ if maxElems <= 0
2969+ then error
2970+ $ functionName
2971+ ++ ": the size of arrays ["
2972+ ++ show maxElems ++ "] must be a natural number"
2973+ else Parser.IPartial Nothing
2974+
2975+ step Nothing arr =
2976+ return
2977+ $ let len = byteLength arr
2978+ in if len >= maxBytes
2979+ then Parser.Done 0 arr
2980+ else Parser.Partial 0 (Just arr)
2981+ -- XXX Split the last array to use the space more compactly.
2982+ step (Just buf) arr =
2983+ let len = byteLength buf + byteLength arr
2984+ in if len > maxBytes
2985+ then return $ Parser.Done 1 buf
2986+ else do
2987+ buf1 <-
2988+ if byteCapacity buf < maxBytes
2989+ then liftIO $ reallocExplicitAs
2990+ ps (SIZE_OF(a)) maxBytes buf
2991+ else return buf
2992+ buf2 <- spliceUnsafe buf1 arr
2993+ return $ Parser.Partial 0 (Just buf2)
2994+
2995+ extract Nothing = return $ Parser.Done 0 nil
2996+ extract (Just buf) = return $ Parser.Done 0 buf
2997+
2998+-- | Parser @pCompactLE maxElems@ coalesces adjacent arrays in the input stream
2999+-- only if the combined size would be less than or equal to @maxElems@
3000+-- elements. Note that it won't split an array if the original array is already
3001+-- larger than maxElems.
3002+--
3003+-- @maxElems@ must be greater than 0.
3004+--
3005+-- Generates unpinned arrays irrespective of the pinning status of input
3006+-- arrays.
3007+--
3008+-- /Internal/
3009+{-# INLINE pCompactLE #-}
3010+pCompactLE ::
3011+ forall m a. (MonadIO m, Unbox a)
3012+ => Int -> Parser (MutArray a) m (MutArray a)
3013+pCompactLE = pCompactLeAs Unpinned
3014+
3015+-- | Pinned version of 'pCompactLE'.
3016+{-# INLINE pPinnedCompactLE #-}
3017+pPinnedCompactLE ::
3018+ forall m a. (MonadIO m, Unbox a)
3019+ => Int -> Parser (MutArray a) m (MutArray a)
3020+pPinnedCompactLE = pCompactLeAs Pinned
3021+
3022+data SpliceState s arr
3023+ = SpliceInitial s
3024+ | SpliceBuffering s arr
3025+ | SpliceYielding arr (SpliceState s arr)
3026+ | SpliceFinish
3027+
3028+-- This mutates the first array (if it has space) to append values from the
3029+-- second one. This would work for immutable arrays as well because an
3030+-- immutable array never has additional space so a new array is allocated
3031+-- instead of mutating it.
3032+
3033+{-# INLINE_NORMAL compactLeAs #-}
3034+compactLeAs :: forall m a. (MonadIO m, Unbox a)
3035+ => PinnedState -> Int -> D.Stream m (MutArray a) -> D.Stream m (MutArray a)
3036+compactLeAs ps maxElems (D.Stream step state) =
3037+ D.Stream step' (SpliceInitial state)
3038+
3039+ where
3040+
3041+ maxBytes = maxElems * SIZE_OF(a)
3042+
3043+ functionName = "Streamly.Internal.Data.MutArray.rCompactLE"
3044+
3045+ {-# INLINE_LATE step' #-}
3046+ step' gst (SpliceInitial st) = do
3047+ when (maxElems <= 0) $
3048+ -- XXX we can pass the module string from the higher level API
3049+ error $ functionName ++ ": the size of arrays [" ++ show maxElems
3050+ ++ "] must be a natural number"
3051+ r <- step gst st
3052+ case r of
3053+ D.Yield arr s -> return $
3054+ let len = byteLength arr
3055+ in if len >= maxBytes
3056+ then D.Skip (SpliceYielding arr (SpliceInitial s))
3057+ else D.Skip (SpliceBuffering s arr)
3058+ D.Skip s -> return $ D.Skip (SpliceInitial s)
3059+ D.Stop -> return D.Stop
3060+
3061+ -- XXX Split the last array to use the space more compactly.
3062+ step' gst (SpliceBuffering st buf) = do
3063+ r <- step gst st
3064+ case r of
3065+ D.Yield arr s -> do
3066+ let len = byteLength buf + byteLength arr
3067+ if len > maxBytes
3068+ then return $
3069+ D.Skip (SpliceYielding buf (SpliceBuffering s arr))
3070+ else do
3071+ buf1 <- if byteCapacity buf < maxBytes
3072+ then liftIO $ reallocExplicitAs
3073+ ps (SIZE_OF(a)) maxBytes buf
3074+ else return buf
3075+ buf2 <- spliceUnsafe buf1 arr
3076+ return $ D.Skip (SpliceBuffering s buf2)
3077+ D.Skip s -> return $ D.Skip (SpliceBuffering s buf)
3078+ D.Stop -> return $ D.Skip (SpliceYielding buf SpliceFinish)
3079+
3080+ step' _ SpliceFinish = return D.Stop
3081+
3082+ step' _ (SpliceYielding arr next) = return $ D.Yield arr next
3083+
3084+
3085+{-# INLINE_NORMAL fCompactGeAs #-}
3086+fCompactGeAs ::
3087+ forall m a. (MonadIO m, Unbox a)
3088+ => PinnedState -> Int -> FL.Fold m (MutArray a) (MutArray a)
3089+fCompactGeAs ps minElems = Fold step initial extract extract
3090+
3091+ where
3092+
3093+ minBytes = minElems * SIZE_OF(a)
3094+
3095+ functionName = "Streamly.Internal.Data.MutArray.fCompactGE"
3096+
3097+ initial =
3098+ return
3099+ $ if minElems < 0
3100+ then error
3101+ $ functionName
3102+ ++ ": the size of arrays ["
3103+ ++ show minElems ++ "] must be a natural number"
3104+ else FL.Partial Nothing
3105+
3106+ step Nothing arr =
3107+ return
3108+ $ let len = byteLength arr
3109+ in if len >= minBytes
3110+ then FL.Done arr
3111+ else FL.Partial (Just arr)
3112+ -- XXX Buffer arrays as a list to avoid copy and reallocations
3113+ step (Just buf) arr = do
3114+ let len = byteLength buf + byteLength arr
3115+ buf1 <-
3116+ if byteCapacity buf < len
3117+ then liftIO $ reallocExplicitAs
3118+ ps (SIZE_OF(a)) (max minBytes len) buf
3119+ else return buf
3120+ buf2 <- spliceUnsafe buf1 arr
3121+ if len >= minBytes
3122+ then return $ FL.Done buf2
3123+ else return $ FL.Partial (Just buf2)
3124+
3125+ extract Nothing = return nil
3126+ extract (Just buf) = return buf
3127+
3128+-- | Fold @fCompactGE minElems@ coalesces adjacent arrays in the input stream
3129+-- until the size becomes greater than or equal to @minElems@.
3130+--
3131+-- Generates unpinned arrays irrespective of the pinning status of input
3132+-- arrays.
3133+{-# INLINE fCompactGE #-}
3134+fCompactGE ::
3135+ forall m a. (MonadIO m, Unbox a)
3136+ => Int -> FL.Fold m (MutArray a) (MutArray a)
3137+fCompactGE = fCompactGeAs Unpinned
3138+
3139+-- | Pinned version of 'fCompactGE'.
3140+{-# INLINE fPinnedCompactGE #-}
3141+fPinnedCompactGE ::
3142+ forall m a. (MonadIO m, Unbox a)
3143+ => Int -> FL.Fold m (MutArray a) (MutArray a)
3144+fPinnedCompactGE = fCompactGeAs Pinned
3145+
3146+{-# INLINE_NORMAL lCompactGeAs #-}
3147+lCompactGeAs :: forall m a. (MonadIO m, Unbox a)
3148+ => PinnedState -> Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
3149+-- The fold version turns out to be a little bit slower.
3150+-- lCompactGeAs ps n = FL.many (fCompactGeAs ps n)
3151+lCompactGeAs ps minElems (Fold step1 initial1 _ final1) =
3152+ Fold step initial extract final
3153+
3154+ where
3155+
3156+ minBytes = minElems * SIZE_OF(a)
3157+
3158+ functionName = "Streamly.Internal.Data.MutArray.lCompactGE"
3159+
3160+ initial = do
3161+ when (minElems <= 0) $
3162+ -- XXX we can pass the module string from the higher level API
3163+ error $ functionName ++ ": the size of arrays ["
3164+ ++ show minElems ++ "] must be a natural number"
3165+
3166+ r <- initial1
3167+ return $ first (Tuple' Nothing) r
3168+
3169+ {-# INLINE runInner #-}
3170+ runInner len acc buf =
3171+ if len >= minBytes
3172+ then do
3173+ r <- step1 acc buf
3174+ case r of
3175+ FL.Done _ -> return $ FL.Done ()
3176+ FL.Partial s -> do
3177+ _ <- final1 s
3178+ res <- initial1
3179+ return $ first (Tuple' Nothing) res
3180+ else return $ FL.Partial $ Tuple' (Just buf) acc
3181+
3182+ step (Tuple' Nothing r1) arr =
3183+ runInner (byteLength arr) r1 arr
3184+
3185+ -- XXX Buffer arrays as a list to avoid copy and reallocations
3186+ step (Tuple' (Just buf) r1) arr = do
3187+ let len = byteLength buf + byteLength arr
3188+ buf1 <- if byteCapacity buf < len
3189+ then liftIO $ reallocExplicitAs
3190+ ps (SIZE_OF(a)) (max minBytes len) buf
3191+ else return buf
3192+ buf2 <- spliceUnsafe buf1 arr
3193+ runInner len r1 buf2
3194+
3195+ -- XXX Several folds do extract >=> final, therefore, we need to make final
3196+ -- return "m b" rather than using extract post it if we want extract to be
3197+ -- partial.
3198+ --
3199+ -- extract forces the pending buffer to be sent to the fold which is not
3200+ -- what we want.
3201+ extract _ = error "lCompactGE: not designed for scanning"
3202+
3203+ final (Tuple' Nothing r1) = final1 r1
3204+ final (Tuple' (Just buf) r1) = do
3205+ r <- step1 r1 buf
3206+ case r of
3207+ FL.Partial rr -> final1 rr
3208+ FL.Done _ -> return ()
3209+
3210+-- | Like 'compactGE' but for transforming folds instead of stream.
3211+--
3212+-- >>> lCompactGE n = Fold.many (MutArray.fCompactGE n)
3213+--
3214+-- Generates unpinned arrays irrespective of the pinning status of input
3215+-- arrays.
3216+{-# INLINE lCompactGE #-}
3217+lCompactGE :: forall m a. (MonadIO m, Unbox a)
3218+ => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
3219+lCompactGE = lCompactGeAs Unpinned
3220+
3221+-- | Pinned version of 'lCompactGE'.
3222+{-# INLINE lPinnedCompactGE #-}
3223+lPinnedCompactGE :: forall m a. (MonadIO m, Unbox a)
3224+ => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
3225+lPinnedCompactGE = lCompactGeAs Pinned
3226+
3227+-- | @compactGE n stream@ coalesces adjacent arrays in the @stream@ until
3228+-- the size becomes greater than or equal to @n@.
3229+--
3230+-- >>> compactGE n = Stream.foldMany (MutArray.fCompactGE n)
3231+--
3232+{-# INLINE compactGE #-}
3233+compactGE ::
3234+ (MonadIO m, Unbox a)
3235+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
3236+compactGE n = D.foldMany (fCompactGE n)
3237+
3238+-- | 'compactEQ n' coalesces adajacent arrays in the input stream to
3239+-- arrays of exact size @n@.
3240+--
3241+-- /Unimplemented/
3242+{-# INLINE compactEQ #-}
3243+compactEQ :: -- (MonadIO m, Unbox a) =>
3244+ Int -> Stream m (MutArray a) -> Stream m (MutArray a)
3245+compactEQ _n = undefined -- D.parseManyD (pCompactEQ n)
23793246
23803247-------------------------------------------------------------------------------
2381−-- NFData
3248+-- In-place mutation algorithms
23823249-------------------------------------------------------------------------------
23833250
23843251-- | Strip elements which match with predicate from both ends.
@@ -2407,7 +3274,7 @@ strip eq arr@MutArray{..} = liftIO $ do
24073274 getStart cur = do
24083275 if cur < arrEnd
24093276 then do
2410− r <- peekByteIndex cur arrContents
3277+ r <- peekAt cur arrContents
24113278 if eq r
24123279 then getStart (INDEX_NEXT(cur,a))
24133280 else return cur
@@ -2417,7 +3284,7 @@ strip eq arr@MutArray{..} = liftIO $ do
24173284 if cur > low
24183285 then do
24193286 let prev = INDEX_PREV(cur,a)
2420− r <- peekByteIndex prev arrContents
3287+ r <- peekAt prev arrContents
24213288 if eq r
24223289 then getLast prev low
24233290 else return cur
core/src/Streamly/Internal/Data/MutByteArray.hsnew file+251
@@ -0,0 +1,251 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+-- This is required as all the instances in this module are orphan instances.
4+{-# OPTIONS_GHC -fno-warn-orphans #-}
5+
6+-- |
7+-- Module : Streamly.Internal.Data.MutByteArray
8+-- Copyright : (c) 2023 Composewell Technologies
9+-- License : BSD3-3-Clause
10+-- Maintainer : streamly@composewell.com
11+-- Portability : GHC
12+--
13+
14+module Streamly.Internal.Data.MutByteArray
15+ (
16+ -- * MutByteArray
17+ module Streamly.Internal.Data.MutByteArray.Type
18+ -- * Unbox
19+ , module Streamly.Internal.Data.Unbox
20+ , module Streamly.Internal.Data.Unbox.TH
21+ -- * Serialize
22+ , module Streamly.Internal.Data.Serialize.Type
23+ -- * Serialize TH
24+ , module Streamly.Internal.Data.Serialize.TH
25+ ) where
26+
27+--------------------------------------------------------------------------------
28+-- Imports
29+--------------------------------------------------------------------------------
30+
31+import Data.Proxy (Proxy(..))
32+import Streamly.Internal.Data.Array (Array(..))
33+import GHC.Exts (Int(..), sizeofByteArray#, unsafeCoerce#)
34+import GHC.Word (Word8)
35+
36+#if __GLASGOW_HASKELL__ >= 900
37+import GHC.Num.Integer (Integer(..))
38+#else
39+import GHC.Integer.GMP.Internals (Integer(..), BigNat(..))
40+#endif
41+
42+import Streamly.Internal.Data.MutByteArray.Type
43+import Streamly.Internal.Data.Serialize.TH
44+import Streamly.Internal.Data.Serialize.Type
45+import Streamly.Internal.Data.Unbox
46+import Streamly.Internal.Data.Unbox.TH
47+
48+--------------------------------------------------------------------------------
49+-- Common instances
50+--------------------------------------------------------------------------------
51+
52+-- Note
53+-- ====
54+--
55+-- Even a non-functional change such as changing the order of constructors will
56+-- change the instance derivation.
57+--
58+-- This will not pose a problem if both, encode, and decode are done by the same
59+-- version of the application. There *might* be a problem if version that
60+-- encodes differs from the version that decodes.
61+--
62+-- We need to add some compatibility tests using different versions of
63+-- dependencies.
64+--
65+-- Although such chages for the most basic types won't happen we need to detect
66+-- if it ever happens.
67+--
68+-- Should we worry about these kind of changes and this kind of compatibility?
69+-- This is a problem for all types of derivations that depend on the order of
70+-- constructors, for example, Enum.
71+
72+-- Note on Windows build
73+-- =====================
74+--
75+-- On Windows, having template haskell splices here fail the build with the
76+-- following error:
77+--
78+-- @
79+-- addLibrarySearchPath: C:\... (Win32 error 3): The system cannot find the path specified.
80+-- @
81+--
82+-- The error might be irrelavant but having these splices triggers it. We should
83+-- either fix the problem or avoid the use to template haskell splices in this
84+-- file.
85+--
86+-- Similar issue: https://github.com/haskell/cabal/issues/4741
87+
88+-- $(Serialize.deriveSerialize ''Maybe)
89+instance Serialize a => Serialize (Maybe a) where
90+
91+ {-# INLINE addSizeTo #-}
92+ addSizeTo acc x =
93+ case x of
94+ Nothing -> (acc + 1)
95+ Just field0 -> (addSizeTo (acc + 1)) field0
96+
97+ {-# INLINE deserializeAt #-}
98+ deserializeAt initialOffset arr endOffset = do
99+ (i0, tag) <- ((deserializeAt initialOffset) arr) endOffset
100+ case tag :: Word8 of
101+ 0 -> pure (i0, Nothing)
102+ 1 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
103+ pure (i1, Just a0)
104+ _ -> error "Found invalid tag while peeking (Maybe a)"
105+
106+ {-# INLINE serializeAt #-}
107+ serializeAt initialOffset arr val =
108+ case val of
109+ Nothing -> do
110+ i0 <- ((serializeAt initialOffset) arr) (0 :: Word8)
111+ pure i0
112+ Just field0 -> do
113+ i0 <- ((serializeAt initialOffset) arr) (1 :: Word8)
114+ i1 <- ((serializeAt i0) arr) field0
115+ pure i1
116+
117+-- $(Serialize.deriveSerialize ''Either)
118+instance (Serialize a, Serialize b) => Serialize (Either a b) where
119+
120+ {-# INLINE addSizeTo #-}
121+ addSizeTo acc x =
122+ case x of
123+ Left field0 -> (addSizeTo (acc + 1)) field0
124+ Right field0 -> (addSizeTo (acc + 1)) field0
125+
126+ {-# INLINE deserializeAt #-}
127+ deserializeAt initialOffset arr endOffset = do
128+ (i0, tag) <- ((deserializeAt initialOffset) arr) endOffset
129+ case tag :: Word8 of
130+ 0 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
131+ pure (i1, Left a0)
132+ 1 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
133+ pure (i1, Right a0)
134+ _ -> error "Found invalid tag while peeking (Either a b)"
135+
136+ {-# INLINE serializeAt #-}
137+ serializeAt initialOffset arr val =
138+ case val of
139+ Left field0 -> do
140+ i0 <- ((serializeAt initialOffset) arr) (0 :: Word8)
141+ i1 <- ((serializeAt i0) arr) field0
142+ pure i1
143+ Right field0 -> do
144+ i0 <- ((serializeAt initialOffset) arr) (1 :: Word8)
145+ i1 <- ((serializeAt i0) arr) field0
146+ pure i1
147+
148+instance Serialize (Proxy a) where
149+
150+ {-# INLINE addSizeTo #-}
151+ addSizeTo acc _ = (acc + 1)
152+
153+ {-# INLINE deserializeAt #-}
154+ deserializeAt initialOffset _ _ = pure ((initialOffset + 1), Proxy)
155+
156+ {-# INLINE serializeAt #-}
157+ serializeAt initialOffset _ _ = pure (initialOffset + 1)
158+
159+--------------------------------------------------------------------------------
160+-- Integer
161+--------------------------------------------------------------------------------
162+
163+data LiftedInteger
164+ = LIS Int
165+ | LIP (Array Word)
166+ | LIN (Array Word)
167+
168+-- $(Serialize.deriveSerialize ''LiftedInteger)
169+instance Serialize LiftedInteger where
170+
171+ {-# INLINE addSizeTo #-}
172+ addSizeTo acc x =
173+ case x of
174+ LIS field0 -> (addSizeTo (acc + 1)) field0
175+ LIP field0 -> (addSizeTo (acc + 1)) field0
176+ LIN field0 -> (addSizeTo (acc + 1)) field0
177+
178+ {-# INLINE deserializeAt #-}
179+ deserializeAt initialOffset arr endOffset = do
180+ (i0, tag) <- ((deserializeAt initialOffset) arr) endOffset
181+ case tag :: Word8 of
182+ 0 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
183+ pure (i1, LIS a0)
184+ 1 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
185+ pure (i1, LIP a0)
186+ 2 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
187+ pure (i1, LIN a0)
188+ _ -> error "Found invalid tag while peeking (LiftedInteger)"
189+
190+ {-# INLINE serializeAt #-}
191+ serializeAt initialOffset arr val =
192+ case val of
193+ LIS field0 -> do
194+ i0 <- ((serializeAt initialOffset) arr) (0 :: Word8)
195+ i1 <- ((serializeAt i0) arr) field0
196+ pure i1
197+ LIP field0 -> do
198+ i0 <- ((serializeAt initialOffset) arr) (1 :: Word8)
199+ i1 <- ((serializeAt i0) arr) field0
200+ pure i1
201+ LIN field0 -> do
202+ i0 <- ((serializeAt initialOffset) arr) (2 :: Word8)
203+ i1 <- ((serializeAt i0) arr) field0
204+ pure i1
205+
206+#if __GLASGOW_HASKELL__ >= 900
207+
208+{-# INLINE liftInteger #-}
209+liftInteger :: Integer -> LiftedInteger
210+liftInteger (IS x) = LIS (I# x)
211+liftInteger (IP x) =
212+ LIP (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
213+liftInteger (IN x) =
214+ LIN (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
215+
216+{-# INLINE unliftInteger #-}
217+unliftInteger :: LiftedInteger -> Integer
218+unliftInteger (LIS (I# x)) = IS x
219+unliftInteger (LIP (Array (MutByteArray x) _ _)) = IP (unsafeCoerce# x)
220+unliftInteger (LIN (Array (MutByteArray x) _ _)) = IN (unsafeCoerce# x)
221+
222+#else
223+
224+{-# INLINE liftInteger #-}
225+liftInteger :: Integer -> LiftedInteger
226+liftInteger (S# x) = LIS (I# x)
227+liftInteger (Jp# (BN# x)) =
228+ LIP (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
229+liftInteger (Jn# (BN# x)) =
230+ LIN (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
231+
232+{-# INLINE unliftInteger #-}
233+unliftInteger :: LiftedInteger -> Integer
234+unliftInteger (LIS (I# x)) = S# x
235+unliftInteger (LIP (Array (MutByteArray x) _ _)) =
236+ Jp# (BN# (unsafeCoerce# x))
237+unliftInteger (LIN (Array (MutByteArray x) _ _)) =
238+ Jn# (BN# (unsafeCoerce# x))
239+
240+#endif
241+
242+instance Serialize Integer where
243+ {-# INLINE addSizeTo #-}
244+ addSizeTo i a = addSizeTo i (liftInteger a)
245+
246+ {-# INLINE deserializeAt #-}
247+ deserializeAt off arr end =
248+ fmap unliftInteger <$> deserializeAt off arr end
249+
250+ {-# INLINE serializeAt #-}
251+ serializeAt off arr val = serializeAt off arr (liftInteger val)
core/src/Streamly/Internal/Data/MutByteArray/Type.hsnew file+321
@@ -0,0 +1,321 @@
1+{-# LANGUAGE UnboxedTuples #-}
2+
3+-- |
4+-- Module : Streamly.Internal.Data.MutByteArray.Type
5+-- Copyright : (c) 2023 Composewell Technologies
6+-- License : BSD3-3-Clause
7+-- Maintainer : streamly@composewell.com
8+-- Stability : experimental
9+-- Portability : GHC
10+--
11+module Streamly.Internal.Data.MutByteArray.Type
12+ (
13+ -- ** MutByteArray
14+ MutByteArray(..)
15+ , MutableByteArray
16+ , getMutableByteArray# -- XXX getMutByteArray#
17+
18+ -- ** Pinning
19+ , PinnedState(..)
20+ , isPinned
21+ , pin
22+ , unpin
23+
24+ -- ** Allocation
25+ , empty
26+ , newBytesAs -- XXX should be removed
27+ , new
28+ , pinnedNew
29+ , pinnedNewAlignedBytes -- XXX should be removed
30+
31+ -- ** Access
32+ , sizeOfMutableByteArray -- XXX length
33+ , putSliceUnsafe
34+ , cloneSliceUnsafeAs
35+ , cloneSliceUnsafe
36+ , pinnedCloneSliceUnsafe
37+ , unsafePinnedAsPtr
38+ , unsafeAsPtr
39+
40+ -- ** Deprecated
41+ , asPtrUnsafe
42+ , nil
43+ ) where
44+
45+import Control.Monad.IO.Class (MonadIO(..))
46+#ifdef DEBUG
47+import Control.Monad (when)
48+import Debug.Trace (trace)
49+#endif
50+import GHC.Base (IO(..))
51+import System.IO.Unsafe (unsafePerformIO)
52+
53+import GHC.Exts
54+
55+--------------------------------------------------------------------------------
56+-- The ArrayContents type
57+--------------------------------------------------------------------------------
58+
59+data PinnedState
60+ = Pinned
61+ | Unpinned deriving (Show, Eq)
62+
63+-- XXX can use UnliftedNewtypes
64+
65+-- | A lifted mutable byte array type wrapping @MutableByteArray# RealWorld@.
66+-- This is a low level array used to back high level unboxed arrays and
67+-- serialized data.
68+data MutByteArray = MutByteArray (MutableByteArray# RealWorld)
69+
70+{-# DEPRECATED MutableByteArray "Please use MutByteArray instead" #-}
71+type MutableByteArray = MutByteArray
72+
73+{-# INLINE getMutableByteArray# #-}
74+getMutableByteArray# :: MutByteArray -> MutableByteArray# RealWorld
75+getMutableByteArray# (MutByteArray mbarr) = mbarr
76+
77+-- | Return the size of the array in bytes.
78+{-# INLINE sizeOfMutableByteArray #-}
79+sizeOfMutableByteArray :: MutByteArray -> IO Int
80+sizeOfMutableByteArray (MutByteArray arr) =
81+ IO $ \s ->
82+ case getSizeofMutableByteArray# arr s of
83+ (# s1, i #) -> (# s1, I# i #)
84+
85+{-# INLINE touch #-}
86+touch :: MutByteArray -> IO ()
87+touch (MutByteArray contents) =
88+ IO $ \s -> case touch# contents s of s' -> (# s', () #)
89+
90+-- XXX Some functions in this module are "IO" and others are "m", we need to
91+-- make it consistent.
92+
93+-- XXX We can provide another API for "unsafe" FFI calls passing an unlifted
94+-- pointer to the FFI call. For unsafe calls we do not need to pin the array.
95+-- We can pass an unlifted pointer to the FFI routine to avoid GC kicking in
96+-- before the pointer is wrapped.
97+--
98+-- From the GHC manual:
99+--
100+-- GHC, since version 8.4, guarantees that garbage collection will never occur
101+-- during an unsafe call, even in the bytecode interpreter, and further
102+-- guarantees that unsafe calls will be performed in the calling thread. Making
103+-- it safe to pass heap-allocated objects to unsafe functions.
104+
105+-- | Use a @MutByteArray@ as @Ptr a@. This is useful when we want to pass
106+-- an array as a pointer to some operating system call or to a "safe" FFI call.
107+--
108+-- If the array is not pinned it is copied to pinned memory before passing it
109+-- to the monadic action.
110+--
111+-- /Performance Notes:/ Forces a copy if the array is not pinned. It is advised
112+-- that the programmer keeps this in mind and creates a pinned array
113+-- opportunistically before this operation occurs, to avoid the cost of a copy
114+-- if possible.
115+--
116+-- /Unsafe/ because of direct pointer operations. The user must ensure that
117+-- they are writing within the legal bounds of the array.
118+--
119+-- /Pre-release/
120+--
121+{-# INLINE unsafePinnedAsPtr #-}
122+unsafePinnedAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
123+unsafePinnedAsPtr arr f = do
124+ contents <- liftIO $ pin arr
125+ let !ptr = Ptr (byteArrayContents#
126+ (unsafeCoerce# (getMutableByteArray# contents)))
127+ r <- f ptr
128+ liftIO $ touch contents
129+ return r
130+
131+{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
132+{-# INLINE asPtrUnsafe #-}
133+asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
134+asPtrUnsafe = unsafePinnedAsPtr
135+
136+-- | For use with unsafe FFI functions. Does not force pin the array memory.
137+{-# INLINE unsafeAsPtr #-}
138+unsafeAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
139+unsafeAsPtr arr f = do
140+ let !ptr = Ptr (byteArrayContents#
141+ (unsafeCoerce# (getMutableByteArray# arr)))
142+ r <- f ptr
143+ liftIO $ touch arr
144+ return r
145+
146+--------------------------------------------------------------------------------
147+-- Creation
148+--------------------------------------------------------------------------------
149+
150+{-# NOINLINE empty #-}
151+empty :: MutByteArray
152+empty = unsafePerformIO $ new 0
153+
154+{-# DEPRECATED nil "Please use empty instead" #-}
155+nil :: MutByteArray
156+nil = empty
157+
158+-- XXX add "newRounded" to round up the large size to the next page boundary
159+-- and return the allocated size.
160+{-# INLINE new #-}
161+new :: Int -> IO MutByteArray
162+new nbytes | nbytes < 0 =
163+ errorWithoutStackTrace "newByteArray: size must be >= 0"
164+new (I# nbytes) = IO $ \s ->
165+ case newByteArray# nbytes s of
166+ (# s', mbarr# #) ->
167+ let c = MutByteArray mbarr#
168+ in (# s', c #)
169+
170+{-# INLINE pinnedNew #-}
171+pinnedNew :: Int -> IO MutByteArray
172+pinnedNew nbytes | nbytes < 0 =
173+ errorWithoutStackTrace "pinnedNew: size must be >= 0"
174+pinnedNew (I# nbytes) = IO $ \s ->
175+ case newPinnedByteArray# nbytes s of
176+ (# s', mbarr# #) ->
177+ let c = MutByteArray mbarr#
178+ in (# s', c #)
179+
180+{-# INLINE pinnedNewAlignedBytes #-}
181+pinnedNewAlignedBytes :: Int -> Int -> IO MutByteArray
182+pinnedNewAlignedBytes nbytes _align | nbytes < 0 =
183+ errorWithoutStackTrace "pinnedNewAlignedBytes: size must be >= 0"
184+pinnedNewAlignedBytes (I# nbytes) (I# align) = IO $ \s ->
185+ case newAlignedPinnedByteArray# nbytes align s of
186+ (# s', mbarr# #) ->
187+ let c = MutByteArray mbarr#
188+ in (# s', c #)
189+
190+{-# INLINE newBytesAs #-}
191+newBytesAs :: PinnedState -> Int -> IO MutByteArray
192+newBytesAs Unpinned = new
193+newBytesAs Pinned = pinnedNew
194+
195+-------------------------------------------------------------------------------
196+-- Copying
197+-------------------------------------------------------------------------------
198+
199+-- Note: Array copy is more efficient than streaming copy.
200+-- CopyMutableByteArray# translates to genMemcpy in GHC/CmmToAsm/X86/CodeGen.hs
201+-- glibc memcpy copies bytes/words/pages - unrolls the loops:
202+-- https://github.com/bminor/glibc/blob/4290aed05135ae4c0272006442d147f2155e70d7/string/memcpy.c
203+-- https://github.com/bminor/glibc/blob/4290aed05135ae4c0272006442d147f2155e70d7/string/wordcopy.c
204+
205+-- | Put a sub range of a source array into a subrange of a destination array.
206+-- This is not safe as it does not check the bounds of neither the src array
207+-- nor the destination array.
208+{-# INLINE putSliceUnsafe #-}
209+putSliceUnsafe ::
210+ MonadIO m
211+ => MutByteArray
212+ -> Int
213+ -> MutByteArray
214+ -> Int
215+ -> Int
216+ -> m ()
217+putSliceUnsafe src srcStartBytes dst dstStartBytes lenBytes = liftIO $ do
218+#ifdef DEBUG
219+ srcLen <- sizeOfMutableByteArray src
220+ dstLen <- sizeOfMutableByteArray dst
221+ when (srcLen - srcStartBytes < lenBytes)
222+ $ error $ "putSliceUnsafe: src overflow: start" ++ show srcStartBytes
223+ ++ " end " ++ show srcLen ++ " len " ++ show lenBytes
224+ when (dstLen - dstStartBytes < lenBytes)
225+ $ error $ "putSliceUnsafe: dst overflow: start" ++ show dstStartBytes
226+ ++ " end " ++ show dstLen ++ " len " ++ show lenBytes
227+#endif
228+ let !(I# srcStartBytes#) = srcStartBytes
229+ !(I# dstStartBytes#) = dstStartBytes
230+ !(I# lenBytes#) = lenBytes
231+ let arrS# = getMutableByteArray# src
232+ arrD# = getMutableByteArray# dst
233+ IO $ \s# -> (# copyMutableByteArray#
234+ arrS# srcStartBytes# arrD# dstStartBytes# lenBytes# s#
235+ , () #)
236+
237+-- | Unsafe as it does not check whether the start offset and length supplied
238+-- are valid inside the array.
239+{-# INLINE cloneSliceUnsafeAs #-}
240+cloneSliceUnsafeAs :: MonadIO m =>
241+ PinnedState -> Int -> Int -> MutByteArray -> m MutByteArray
242+cloneSliceUnsafeAs ps srcOff srcLen src =
243+ liftIO $ do
244+ mba <- newBytesAs ps srcLen
245+ putSliceUnsafe src srcOff mba 0 srcLen
246+ return mba
247+
248+-- | @cloneSliceUnsafe offset len arr@ clones a slice of the supplied array
249+-- starting at the given offset and equal to the given length.
250+{-# INLINE cloneSliceUnsafe #-}
251+cloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
252+cloneSliceUnsafe = cloneSliceUnsafeAs Unpinned
253+
254+-- | @pinnedCloneSliceUnsafe offset len arr@
255+{-# INLINE pinnedCloneSliceUnsafe #-}
256+pinnedCloneSliceUnsafe :: MonadIO m =>
257+ Int -> Int -> MutByteArray -> m MutByteArray
258+pinnedCloneSliceUnsafe = cloneSliceUnsafeAs Pinned
259+
260+-------------------------------------------------------------------------------
261+-- Pinning & Unpinning
262+-------------------------------------------------------------------------------
263+
264+-- | Return 'True' if the array is allocated in pinned memory.
265+{-# INLINE isPinned #-}
266+isPinned :: MutByteArray -> Bool
267+isPinned (MutByteArray arr#) =
268+ let pinnedInt = I# (isMutableByteArrayPinned# arr#)
269+ in pinnedInt /= 0
270+
271+
272+{-# INLINE cloneMutableArrayWith# #-}
273+cloneMutableArrayWith#
274+ :: (Int# -> State# RealWorld -> (# State# RealWorld
275+ , MutableByteArray# RealWorld #))
276+ -> MutableByteArray# RealWorld
277+ -> State# RealWorld
278+ -> (# State# RealWorld, MutableByteArray# RealWorld #)
279+cloneMutableArrayWith# alloc# arr# s# =
280+ case getSizeofMutableByteArray# arr# s# of
281+ (# s1#, i# #) ->
282+ case alloc# i# s1# of
283+ (# s2#, arr1# #) ->
284+ case copyMutableByteArray# arr# 0# arr1# 0# i# s2# of
285+ s3# -> (# s3#, arr1# #)
286+
287+-- | Return a copy of the array in pinned memory if unpinned, else return the
288+-- original array.
289+{-# INLINE pin #-}
290+pin :: MutByteArray -> IO MutByteArray
291+pin arr@(MutByteArray marr#) =
292+ if isPinned arr
293+ then return arr
294+ else
295+#ifdef DEBUG
296+ do
297+ -- XXX dump stack trace
298+ trace ("pin: Copying array") (return ())
299+#endif
300+ IO
301+ $ \s# ->
302+ case cloneMutableArrayWith# newPinnedByteArray# marr# s# of
303+ (# s1#, marr1# #) -> (# s1#, MutByteArray marr1# #)
304+
305+-- | Return a copy of the array in unpinned memory if pinned, else return the
306+-- original array.
307+{-# INLINE unpin #-}
308+unpin :: MutByteArray -> IO MutByteArray
309+unpin arr@(MutByteArray marr#) =
310+ if not (isPinned arr)
311+ then return arr
312+ else
313+#ifdef DEBUG
314+ do
315+ -- XXX dump stack trace
316+ trace ("unpin: Copying array") (return ())
317+#endif
318+ IO
319+ $ \s# ->
320+ case cloneMutableArrayWith# newByteArray# marr# s# of
321+ (# s1#, marr1# #) -> (# s1#, MutByteArray marr1# #)
core/src/Streamly/Internal/Data/Parser.hs+135−143
@@ -10,10 +10,16 @@
1010module Streamly.Internal.Data.Parser
1111 (
1212 -- * Setup
13+ -- | To execute the code examples provided in this module in ghci, please
14+ -- run the following commands first.
15+ --
1316 -- $setup
1417
18+ module Streamly.Internal.Data.Parser.Type
19+ --, module Streamly.Internal.Data.Parser.Tee
20+
1521 -- * Types
16− Parser (..)
22+ , Parser (..)
1723 , ParseError (..)
1824 , Step (..)
1925 , Initial (..)
@@ -25,19 +31,9 @@ module Streamly.Internal.Data.Parser
2531 -- * Accumulators
2632 , fromFold
2733 , fromFoldMaybe
28− , fromPure
29− , fromEffect
30− , die
31− , dieM
3234
3335 -- * Map on input
34− , lmap
35− , lmapM
3636 , postscan
37− , filter
38−
39− -- * Map on output
40− , rmapM
4137
4238 -- * Element parsers
4339 , peek
@@ -139,11 +135,6 @@ module Streamly.Internal.Data.Parser
139135
140136 -- Second order parsers (parsers using parsers)
141137 -- * Binary Combinators
142−
143− -- ** Sequential Applicative
144− , splitWith
145− , split_
146−
147138 {-
148139 -- ** Parallel Applicatives
149140 , teeWith
@@ -152,9 +143,6 @@ module Streamly.Internal.Data.Parser
152143 -- , teeTill -- like manyTill but parallel
153144 -}
154145
155− -- ** Sequential Alternative
156− , alt
157−
158146 {-
159147 -- ** Parallel Alternatives
160148 , shortest
@@ -165,7 +153,6 @@ module Streamly.Internal.Data.Parser
165153 -- * N-ary Combinators
166154 -- ** Sequential Collection
167155 , sequence
168− , concatMap
169156
170157 -- ** Sequential Repetition
171158 , count
@@ -266,18 +253,19 @@ import Streamly.Internal.Data.SVar.Type (defState)
266253import Streamly.Internal.Data.Either.Strict (Either'(..))
267254import Streamly.Internal.Data.Maybe.Strict (Maybe'(..))
268255import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
269−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
256+import Streamly.Internal.Data.Stream.Type (Stream)
270257
271258import qualified Data.Foldable as Foldable
272259import qualified Streamly.Internal.Data.Fold.Type as FL
273−import qualified Streamly.Internal.Data.Stream.StreamD.Type as D
274−import qualified Streamly.Internal.Data.Stream.StreamD.Generate as D
260+import qualified Streamly.Internal.Data.Stream.Type as D
261+import qualified Streamly.Internal.Data.Stream.Generate as D
262+
263+import Streamly.Internal.Data.Parser.Type
264+--import Streamly.Internal.Data.Parser.Tee -- It's empty
275265
276266import Prelude hiding
277267 (any, all, take, takeWhile, sequence, concatMap, maybe, either, span
278268 , zip, filter, dropWhile)
279−-- import Streamly.Internal.Data.Parser.ParserD.Tee
280−import Streamly.Internal.Data.Parser.ParserD.Type
281269
282270#include "DocTestDataParser.hs"
283271
@@ -296,7 +284,7 @@ import Streamly.Internal.Data.Parser.ParserD.Type
296284--
297285{-# INLINE toFold #-}
298286toFold :: Monad m => Parser a m b -> Fold m a b
299−toFold (Parser pstep pinitial pextract) = Fold step initial extract
287+toFold (Parser pstep pinitial pextract) = Fold step initial extract final
300288
301289 where
302290
@@ -324,7 +312,9 @@ toFold (Parser pstep pinitial pextract) = Fold step initial extract
324312 Done n _ -> derror n
325313 Error err -> eerror err
326314
327− extract st = do
315+ extract = error "toFold: parser cannot be used for scanning"
316+
317+ final st = do
328318 r <- pextract st
329319 case r of
330320 Done 0 b -> return b
@@ -342,7 +332,7 @@ toFold (Parser pstep pinitial pextract) = Fold step initial extract
342332--
343333{-# INLINE fromFold #-}
344334fromFold :: Monad m => Fold m a b -> Parser a m b
345−fromFold (Fold fstep finitial fextract) = Parser step initial extract
335+fromFold (Fold fstep finitial _ ffinal) = Parser step initial extract
346336
347337 where
348338
@@ -360,7 +350,7 @@ fromFold (Fold fstep finitial fextract) = Parser step initial extract
360350 FL.Partial s1 -> Partial 0 s1
361351 FL.Done b -> Done 0 b
362352
363− extract = fmap (Done 0) . fextract
353+ extract = fmap (Done 0) . ffinal
364354
365355-- | Convert a Maybe returning fold to an error returning parser. The first
366356-- argument is the error message that the parser would return when the fold
@@ -370,7 +360,7 @@ fromFold (Fold fstep finitial fextract) = Parser step initial extract
370360--
371361{-# INLINE fromFoldMaybe #-}
372362fromFoldMaybe :: Monad m => String -> Fold m a (Maybe b) -> Parser a m b
373−fromFoldMaybe errMsg (Fold fstep finitial fextract) =
363+fromFoldMaybe errMsg (Fold fstep finitial _ ffinal) =
374364 Parser step initial extract
375365
376366 where
@@ -396,7 +386,7 @@ fromFoldMaybe errMsg (Fold fstep finitial fextract) =
396386 Nothing -> Error errMsg
397387
398388 extract s = do
399− res <- fextract s
389+ res <- ffinal s
400390 case res of
401391 Just x -> return $ Done 0 x
402392 Nothing -> return $ Error errMsg
@@ -645,7 +635,7 @@ data Tuple'Fused a b = Tuple'Fused !a !b deriving Show
645635--
646636{-# INLINE takeBetween #-}
647637takeBetween :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
648−takeBetween low high (Fold fstep finitial fextract) =
638+takeBetween low high (Fold fstep finitial _ ffinal) =
649639
650640 Parser step initial (extract streamErr)
651641
@@ -696,9 +686,11 @@ takeBetween low high (Fold fstep finitial fextract) =
696686 in case res of
697687 FL.Partial s -> do
698688 let s1 = Tuple'Fused i1 s
699− if i1 < high
689+ if i1 < low
700690 then return $ Continue 0 s1
701− else extract foldErr s1
691+ else if i1 < high
692+ then return $ Partial 0 s1
693+ else fmap (Done 0) (ffinal s)
702694 FL.Done b ->
703695 return
704696 $ if i1 >= low
@@ -708,12 +700,12 @@ takeBetween low high (Fold fstep finitial fextract) =
708700 step (Tuple'Fused i s) a = fstep s a >>= snext i
709701
710702 extract f (Tuple'Fused i s)
711− | i >= low && i <= high = fmap (Done 0) (fextract s)
703+ | i >= low && i <= high = fmap (Done 0) (ffinal s)
712704 | otherwise = return $ Error (f i)
713705
714706 -- XXX Need to make Initial return type Step to deduplicate this
715707 iextract f (Tuple'Fused i s)
716− | i >= low && i <= high = fmap IDone (fextract s)
708+ | i >= low && i <= high = fmap IDone (ffinal s)
717709 | otherwise = return $ IError (f i)
718710
719711-- | Stops after taking exactly @n@ input elements.
@@ -730,7 +722,7 @@ takeBetween low high (Fold fstep finitial fextract) =
730722--
731723{-# INLINE takeEQ #-}
732724takeEQ :: Monad m => Int -> Fold m a b -> Parser a m b
733−takeEQ n (Fold fstep finitial fextract) = Parser step initial extract
725+takeEQ n (Fold fstep finitial _ ffinal) = Parser step initial extract
734726
735727 where
736728
@@ -740,7 +732,7 @@ takeEQ n (Fold fstep finitial fextract) = Parser step initial extract
740732 FL.Partial s ->
741733 if n > 0
742734 then return $ IPartial $ Tuple'Fused 1 s
743− else fmap IDone (fextract s)
735+ else fmap IDone (ffinal s)
744736 FL.Done b -> return $
745737 if n > 0
746738 then IError
@@ -764,7 +756,7 @@ takeEQ n (Fold fstep finitial fextract) = Parser step initial extract
764756 -- assert (n == i1)
765757 Done 0
766758 <$> case res of
767− FL.Partial s -> fextract s
759+ FL.Partial s -> ffinal s
768760 FL.Done b -> return b
769761
770762 extract (Tuple'Fused i _) =
@@ -796,7 +788,7 @@ data TakeGEState s =
796788--
797789{-# INLINE takeGE #-}
798790takeGE :: Monad m => Int -> Fold m a b -> Parser a m b
799−takeGE n (Fold fstep finitial fextract) = Parser step initial extract
791+takeGE n (Fold fstep finitial _ ffinal) = Parser step initial extract
800792
801793 where
802794
@@ -844,7 +836,7 @@ takeGE n (Fold fstep finitial fextract) = Parser step initial extract
844836 $ Error
845837 $ "takeGE: Expecting at least " ++ show n
846838 ++ " elements, input terminated on " ++ show (i - 1)
847− extract (TakeGEGE r) = fmap (Done 0) $ fextract r
839+ extract (TakeGEGE r) = fmap (Done 0) $ ffinal r
848840
849841-------------------------------------------------------------------------------
850842-- Conditional splitting
@@ -905,7 +897,7 @@ takeWhileP predicate (Parser pstep pinitial pextract) =
905897{-# INLINE takeWhile #-}
906898takeWhile :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
907899-- takeWhile cond f = takeWhileP cond (fromFold f)
908−takeWhile predicate (Fold fstep finitial fextract) =
900+takeWhile predicate (Fold fstep finitial _ ffinal) =
909901 Parser step initial extract
910902
911903 where
@@ -924,9 +916,9 @@ takeWhile predicate (Fold fstep finitial fextract) =
924916 $ case fres of
925917 FL.Partial s1 -> Partial 0 s1
926918 FL.Done b -> Done 0 b
927− else Done 1 <$> fextract s
919+ else Done 1 <$> ffinal s
928920
929− extract s = fmap (Done 0) (fextract s)
921+ extract s = fmap (Done 0) (ffinal s)
930922
931923{-
932924-- XXX This may not be composable because of the b argument. We can instead
@@ -946,7 +938,7 @@ takeWhile1 acc cond f = undefined
946938{-# INLINE takeWhile1 #-}
947939takeWhile1 :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
948940-- takeWhile1 cond f = takeWhileP cond (takeBetween 1 maxBound f)
949−takeWhile1 predicate (Fold fstep finitial fextract) =
941+takeWhile1 predicate (Fold fstep finitial _ ffinal) =
950942 Parser step initial extract
951943
952944 where
@@ -976,11 +968,11 @@ takeWhile1 predicate (Fold fstep finitial fextract) =
976968 if predicate a
977969 then process s a
978970 else do
979− b <- fextract s
971+ b <- ffinal s
980972 return $ Done 1 b
981973
982974 extract (Left' _) = return $ Error "takeWhile1: end of input"
983− extract (Right' s) = fmap (Done 0) (fextract s)
975+ extract (Right' s) = fmap (Done 0) (ffinal s)
984976
985977-- | Drain the input as long as the predicate succeeds, running the effects and
986978-- discarding the results.
@@ -1009,7 +1001,7 @@ takeFramedByGeneric :: Monad m =>
10091001 -> Maybe (a -> Bool) -- is frame end?
10101002 -> Fold m a b
10111003 -> Parser a m b
1012−takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
1004+takeFramedByGeneric esc begin end (Fold fstep finitial _ ffinal) =
10131005
10141006 Parser step initial extract
10151007
@@ -1041,20 +1033,20 @@ takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
10411033 if isEnd a
10421034 then
10431035 if n == 0
1044− then Done 0 <$> fextract s
1036+ then Done 0 <$> ffinal s
10451037 else process s a (n - 1)
10461038 else
10471039 let n1 = if isBegin a then n + 1 else n
10481040 in process s a n1
10491041 Nothing -> -- takeEndBy case
10501042 if isEnd a
1051− then Done 0 <$> fextract s
1043+ then Done 0 <$> ffinal s
10521044 else process s a n
10531045 Nothing -> -- takeStartBy case
10541046 case begin of
10551047 Just isBegin ->
10561048 if isBegin a
1057− then Done 0 <$> fextract s
1049+ then Done 0 <$> ffinal s
10581050 else process s a n
10591051 Nothing ->
10601052 error $ "takeFramedByGeneric: "
@@ -1079,7 +1071,7 @@ takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
10791071 case end of
10801072 Just isEnd ->
10811073 if isEnd a
1082− then Done 0 <$> fextract s
1074+ then Done 0 <$> ffinal s
10831075 else processCheckEsc s a 0
10841076 Nothing ->
10851077 error "Both begin and end frame predicate missing"
@@ -1094,7 +1086,7 @@ takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
10941086 case begin of
10951087 Just _ ->
10961088 case end of
1097− Nothing -> fmap (Done 0) $ fextract s
1089+ Nothing -> fmap (Done 0) $ ffinal s
10981090 Just _ -> err "takeFramedByGeneric: missing frame end"
10991091 Nothing -> err "takeFramedByGeneric: missing closing frame"
11001092 extract (FrameEscEsc _ _) = err "takeFramedByGeneric: trailing escape"
@@ -1133,7 +1125,7 @@ blockWithQuotes :: (Monad m, Eq a) =>
11331125 -> Fold m a b
11341126 -> Parser a m b
11351127blockWithQuotes isEsc isQuote bopen bclose
1136− (Fold fstep finitial fextract) =
1128+ (Fold fstep finitial _ ffinal) =
11371129 Parser step initial extract
11381130
11391131 where
@@ -1163,7 +1155,7 @@ blockWithQuotes isEsc isQuote bopen bclose
11631155 | a == bopen = process s a (BlockUnquoted (level + 1))
11641156 | a == bclose =
11651157 if level == 1
1166− then fmap (Done 0) (fextract s)
1158+ then fmap (Done 0) (ffinal s)
11671159 else process s a (BlockUnquoted (level - 1))
11681160 | isQuote a = process s a (BlockQuoted level)
11691161 | otherwise = process s a (BlockUnquoted level)
@@ -1177,7 +1169,7 @@ blockWithQuotes isEsc isQuote bopen bclose
11771169
11781170 err = return . Error
11791171
1180− extract (BlockInit s) = fmap (Done 0) $ fextract s
1172+ extract (BlockInit s) = fmap (Done 0) $ ffinal s
11811173 extract (BlockUnquoted level _) =
11821174 err $ "blockWithQuotes: finished at block nest level " ++ show level
11831175 extract (BlockQuoted level _) =
@@ -1312,7 +1304,7 @@ takeEitherSepBy _cond = undefined -- D.toParserK . D.takeEitherSepBy cond
13121304--
13131305{-# INLINE takeStartBy #-}
13141306takeStartBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
1315−takeStartBy cond (Fold fstep finitial fextract) =
1307+takeStartBy cond (Fold fstep finitial _ ffinal) =
13161308
13171309 Parser step initial extract
13181310
@@ -1340,10 +1332,10 @@ takeStartBy cond (Fold fstep finitial fextract) =
13401332 step (Right' s) a =
13411333 if not (cond a)
13421334 then process s a
1343− else Done 1 <$> fextract s
1335+ else Done 1 <$> ffinal s
13441336
1345− extract (Left' s) = fmap (Done 0) $ fextract s
1346− extract (Right' s) = fmap (Done 0) $ fextract s
1337+ extract (Left' s) = fmap (Done 0) $ ffinal s
1338+ extract (Right' s) = fmap (Done 0) $ ffinal s
13471339
13481340-- | Like 'takeStartBy' but drops the separator.
13491341--
@@ -1379,7 +1371,7 @@ takeFramedByEsc_ :: Monad m =>
13791371 (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
13801372-- takeFramedByEsc_ isEsc isEnd p =
13811373-- takeFramedByGeneric (Just isEsc) Nothing (Just isEnd) (toFold p)
1382−takeFramedByEsc_ isEsc isBegin isEnd (Fold fstep finitial fextract) =
1374+takeFramedByEsc_ isEsc isBegin isEnd (Fold fstep finitial _ ffinal ) =
13831375
13841376 Parser step initial extract
13851377
@@ -1415,7 +1407,7 @@ takeFramedByEsc_ isEsc isBegin isEnd (Fold fstep finitial fextract) =
14151407 in process s a n1
14161408 else
14171409 if n == 0
1418− then Done 0 <$> fextract s
1410+ then Done 0 <$> ffinal s
14191411 else process s a (n - 1)
14201412 step (FrameEscEsc s n) a = process s a n
14211413
@@ -1437,7 +1429,7 @@ takeFramedBy_ :: Monad m =>
14371429 (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
14381430-- takeFramedBy_ isBegin isEnd =
14391431-- takeFramedByGeneric (Just (const False)) (Just isBegin) (Just isEnd)
1440−takeFramedBy_ isBegin isEnd (Fold fstep finitial fextract) =
1432+takeFramedBy_ isBegin isEnd (Fold fstep finitial _ ffinal) =
14411433
14421434 Parser step initial extract
14431435
@@ -1467,7 +1459,7 @@ takeFramedBy_ isBegin isEnd (Fold fstep finitial fextract) =
14671459 | not (isEnd a) =
14681460 let n1 = if isBegin a then n + 1 else n
14691461 in process s a n1
1470− | n == 0 = Done 0 <$> fextract s
1462+ | n == 0 = Done 0 <$> ffinal s
14711463 | otherwise = process s a (n - 1)
14721464
14731465 err = return . Error
@@ -1502,7 +1494,7 @@ data WordByState s b = WBLeft !s | WBWord !s | WBRight !b
15021494--
15031495{-# INLINE wordBy #-}
15041496wordBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
1505−wordBy predicate (Fold fstep finitial fextract) = Parser step initial extract
1497+wordBy predicate (Fold fstep finitial _ ffinal) = Parser step initial extract
15061498
15071499 where
15081500
@@ -1529,7 +1521,7 @@ wordBy predicate (Fold fstep finitial fextract) = Parser step initial extract
15291521 if not (predicate a)
15301522 then worder s a
15311523 else do
1532− b <- fextract s
1524+ b <- ffinal s
15331525 return $ Partial 0 $ WBRight b
15341526 step (WBRight b) a =
15351527 return
@@ -1537,8 +1529,8 @@ wordBy predicate (Fold fstep finitial fextract) = Parser step initial extract
15371529 then Done 1 b
15381530 else Partial 0 $ WBRight b
15391531
1540− extract (WBLeft s) = fmap (Done 0) $ fextract s
1541− extract (WBWord s) = fmap (Done 0) $ fextract s
1532+ extract (WBLeft s) = fmap (Done 0) $ ffinal s
1533+ extract (WBWord s) = fmap (Done 0) $ ffinal s
15421534 extract (WBRight b) = return (Done 0 b)
15431535
15441536data WordFramedState s b =
@@ -1575,7 +1567,7 @@ wordFramedBy :: Monad m =>
15751567 -> Fold m a b
15761568 -> Parser a m b
15771569wordFramedBy isEsc isBegin isEnd isSep
1578− (Fold fstep finitial fextract) =
1570+ (Fold fstep finitial _ ffinal) =
15791571 Parser step initial extract
15801572
15811573 where
@@ -1606,7 +1598,7 @@ wordFramedBy isEsc isBegin isEnd isSep
16061598 step (WordFramedWord s n) a
16071599 | isEsc a = return $ Continue 0 $ WordFramedEsc s n
16081600 | n == 0 && isSep a = do
1609− b <- fextract s
1601+ b <- ffinal s
16101602 return $ Partial 0 $ WordFramedSkipPost b
16111603 | otherwise = do
16121604 -- We need to use different order for checking begin and end for
@@ -1639,10 +1631,10 @@ wordFramedBy isEsc isBegin isEnd isSep
16391631
16401632 err = return . Error
16411633
1642− extract (WordFramedSkipPre s) = fmap (Done 0) $ fextract s
1634+ extract (WordFramedSkipPre s) = fmap (Done 0) $ ffinal s
16431635 extract (WordFramedWord s n) =
16441636 if n == 0
1645− then fmap (Done 0) $ fextract s
1637+ then fmap (Done 0) $ ffinal s
16461638 else err "wordFramedBy: missing frame end"
16471639 extract (WordFramedEsc _ _) =
16481640 err "wordFramedBy: trailing escape"
@@ -1729,7 +1721,7 @@ wordWithQuotes :: (Monad m, Eq a) =>
17291721 -> Fold m a b
17301722 -> Parser a m b
17311723wordWithQuotes keepQuotes tr escChar toRight isSep
1732− (Fold fstep finitial fextract) =
1724+ (Fold fstep finitial _ ffinal) =
17331725 Parser step initial extract
17341726
17351727 where
@@ -1780,7 +1772,7 @@ wordWithQuotes keepQuotes tr escChar toRight isSep
17801772 step (WordUnquotedWord s) a
17811773 | isEsc a = return $ Continue 0 $ WordUnquotedEsc s
17821774 | isSep a = do
1783− b <- fextract s
1775+ b <- ffinal s
17841776 return $ Partial 0 $ WordQuotedSkipPost b
17851777 | otherwise = do
17861778 case toRight a of
@@ -1828,11 +1820,11 @@ wordWithQuotes keepQuotes tr escChar toRight isSep
18281820
18291821 err = return . Error
18301822
1831− extract (WordQuotedSkipPre s) = fmap (Done 0) $ fextract s
1832− extract (WordUnquotedWord s) = fmap (Done 0) $ fextract s
1823+ extract (WordQuotedSkipPre s) = fmap (Done 0) $ ffinal s
1824+ extract (WordUnquotedWord s) = fmap (Done 0) $ ffinal s
18331825 extract (WordQuotedWord s n _ _) =
18341826 if n == 0
1835− then fmap (Done 0) $ fextract s
1827+ then fmap (Done 0) $ ffinal s
18361828 else err "wordWithQuotes: missing frame end"
18371829 extract WordQuotedEsc {} =
18381830 err "wordWithQuotes: trailing escape"
@@ -1910,7 +1902,7 @@ data GroupByState a s
19101902--
19111903{-# INLINE groupBy #-}
19121904groupBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
1913−groupBy eq (Fold fstep finitial fextract) = Parser step initial extract
1905+groupBy eq (Fold fstep finitial _ ffinal) = Parser step initial extract
19141906
19151907 where
19161908
@@ -1933,10 +1925,10 @@ groupBy eq (Fold fstep finitial fextract) = Parser step initial extract
19331925 step (GroupByGrouping a0 s) a =
19341926 if eq a0 a
19351927 then grouper s a0 a
1936− else Done 1 <$> fextract s
1928+ else Done 1 <$> ffinal s
19371929
1938− extract (GroupByInit s) = fmap (Done 0) $ fextract s
1939− extract (GroupByGrouping _ s) = fmap (Done 0) $ fextract s
1930+ extract (GroupByInit s) = fmap (Done 0) $ ffinal s
1931+ extract (GroupByGrouping _ s) = fmap (Done 0) $ ffinal s
19401932
19411933-- | Unlike 'groupBy' this combinator performs a rolling comparison of two
19421934-- successive elements in the input stream. Assuming the input stream
@@ -1970,7 +1962,7 @@ groupBy eq (Fold fstep finitial fextract) = Parser step initial extract
19701962--
19711963{-# INLINE groupByRolling #-}
19721964groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
1973−groupByRolling eq (Fold fstep finitial fextract) = Parser step initial extract
1965+groupByRolling eq (Fold fstep finitial _ ffinal) = Parser step initial extract
19741966
19751967 where
19761968
@@ -1993,10 +1985,10 @@ groupByRolling eq (Fold fstep finitial fextract) = Parser step initial extract
19931985 step (GroupByGrouping a0 s) a =
19941986 if eq a0 a
19951987 then grouper s a
1996− else Done 1 <$> fextract s
1988+ else Done 1 <$> ffinal s
19971989
1998− extract (GroupByInit s) = fmap (Done 0) $ fextract s
1999− extract (GroupByGrouping _ s) = fmap (Done 0) $ fextract s
1990+ extract (GroupByInit s) = fmap (Done 0) $ ffinal s
1991+ extract (GroupByGrouping _ s) = fmap (Done 0) $ ffinal s
20001992
20011993{-# ANN type GroupByStatePair Fuse #-}
20021994data GroupByStatePair a s1 s2
@@ -2020,8 +2012,8 @@ groupByRollingEither :: Monad m =>
20202012 (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
20212013groupByRollingEither
20222014 eq
2023− (Fold fstep1 finitial1 fextract1)
2024− (Fold fstep2 finitial2 fextract2) = Parser step initial extract
2015+ (Fold fstep1 finitial1 _ ffinal1)
2016+ (Fold fstep2 finitial2 _ ffinal2) = Parser step initial extract
20252017
20262018 where
20272019
@@ -2080,21 +2072,21 @@ groupByRollingEither
20802072 step (GroupByGroupingPairL a0 s1 s2) a =
20812073 if not (eq a0 a)
20822074 then grouperL2 s1 s2 a
2083− else Done 1 . Left <$> fextract1 s1
2075+ else Done 1 . Left <$> ffinal1 s1
20842076
20852077 step (GroupByGroupingPairR a0 s1 s2) a =
20862078 if eq a0 a
20872079 then grouperR2 s1 s2 a
2088− else Done 1 . Right <$> fextract2 s2
2080+ else Done 1 . Right <$> ffinal2 s2
20892081
2090− extract (GroupByInitPair s1 _) = Done 0 . Left <$> fextract1 s1
2091− extract (GroupByGroupingPairL _ s1 _) = Done 0 . Left <$> fextract1 s1
2092− extract (GroupByGroupingPairR _ _ s2) = Done 0 . Right <$> fextract2 s2
2082+ extract (GroupByInitPair s1 _) = Done 0 . Left <$> ffinal1 s1
2083+ extract (GroupByGroupingPairL _ s1 _) = Done 0 . Left <$> ffinal1 s1
2084+ extract (GroupByGroupingPairR _ _ s2) = Done 0 . Right <$> ffinal2 s2
20932085 extract (GroupByGroupingPair a s1 _) = do
20942086 res <- fstep1 s1 a
20952087 case res of
20962088 FL.Done b -> return $ Done 0 (Left b)
2097− FL.Partial s11 -> Done 0 . Left <$> fextract1 s11
2089+ FL.Partial s11 -> Done 0 . Left <$> ffinal1 s11
20982090
20992091-- XXX use an Unfold instead of a list?
21002092-- XXX custom combinators for matching list, array and stream?
@@ -2247,7 +2239,7 @@ postscan = undefined
22472239{-# INLINE zipWithM #-}
22482240zipWithM :: Monad m =>
22492241 (a -> b -> m c) -> D.Stream m a -> Fold m c x -> Parser b m x
2250−zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
2242+zipWithM zf (D.Stream sstep state) (Fold fstep finitial _ ffinal) =
22512243 Parser step initial extract
22522244
22532245 where
@@ -2260,7 +2252,7 @@ zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
22602252 case r of
22612253 D.Yield x s -> return $ IPartial (Just' x, s, fs)
22622254 D.Stop -> do
2263− x <- fextract fs
2255+ x <- ffinal fs
22642256 return $ IDone x
22652257 -- Need Skip/Continue in initial to loop right here
22662258 D.Skip s -> return $ IPartial (Nothing', s, fs)
@@ -2275,7 +2267,7 @@ zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
22752267 case r of
22762268 D.Yield x1 s -> return $ Continue 0 (Just' x1, s, fs1)
22772269 D.Stop -> do
2278− x <- fextract fs1
2270+ x <- ffinal fs1
22792271 return $ Done 0 x
22802272 D.Skip s -> return $ Continue 1 (Nothing', s, fs1)
22812273 FL.Done x -> return $ Done 0 x
@@ -2290,7 +2282,7 @@ zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
22902282 return $ Continue 0 (Nothing', s, fs1)
22912283 FL.Done x -> return $ Done 0 x
22922284 D.Stop -> do
2293− x <- fextract fs
2285+ x <- ffinal fs
22942286 return $ Done 1 x
22952287 D.Skip s -> return $ Continue 1 (Nothing', s, fs)
22962288
@@ -2574,7 +2566,7 @@ deintercalateAll :: Monad m =>
25742566deintercalateAll
25752567 (Parser stepL initialL extractL)
25762568 (Parser stepR initialR _)
2577− (Fold fstep finitial fextract) = Parser step initial extract
2569+ (Fold fstep finitial _ ffinal) = Parser step initial extract
25782570
25792571 where
25802572
@@ -2645,9 +2637,9 @@ deintercalateAll
26452637 extractResult n fs r = do
26462638 res <- fstep fs r
26472639 case res of
2648− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
2640+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
26492641 FL.Done c -> return (Done n c)
2650− extract (DeintercalateAllInitL fs) = fmap (Done 0) $ fextract fs
2642+ extract (DeintercalateAllInitL fs) = fmap (Done 0) $ ffinal fs
26512643 extract (DeintercalateAllL fs sL) = do
26522644 r <- extractL sL
26532645 case r of
@@ -2655,7 +2647,7 @@ deintercalateAll
26552647 Error err -> return $ Error err
26562648 Continue n s -> return $ Continue n (DeintercalateAllL fs s)
26572649 Partial _ _ -> error "Partial in extract"
2658− extract (DeintercalateAllInitR fs) = fmap (Done 0) $ fextract fs
2650+ extract (DeintercalateAllInitR fs) = fmap (Done 0) $ ffinal fs
26592651 extract (DeintercalateAllR _ _) =
26602652 return $ Error "deintercalateAll: input ended at 'Right' value"
26612653
@@ -2698,7 +2690,7 @@ deintercalate :: Monad m =>
26982690deintercalate
26992691 (Parser stepL initialL extractL)
27002692 (Parser stepR initialR _)
2701− (Fold fstep finitial fextract) = Parser step initial extract
2693+ (Fold fstep finitial _ ffinal) = Parser step initial extract
27022694
27032695 where
27042696
@@ -2729,7 +2721,7 @@ deintercalate
27292721 Done n b ->
27302722 processL (fstep fs (Left b)) n DeintercalateInitR
27312723 Error _ -> do
2732− xs <- fextract fs
2724+ xs <- ffinal fs
27332725 return $ Done cnt1 xs
27342726
27352727 {-# INLINE processR #-}
@@ -2749,7 +2741,7 @@ deintercalate
27492741 Continue n s -> return $ Continue n (DeintercalateR (cnt1 - n) fs s)
27502742 Done n b -> processR (cnt1 - n) b fs n
27512743 Error _ -> do
2752− xs <- fextract fs
2744+ xs <- ffinal fs
27532745 return $ Done cnt1 xs
27542746
27552747 step (DeintercalateInitL fs) a = do
@@ -2784,17 +2776,17 @@ deintercalate
27842776 -- XXX We could have the fold accept pairs of (bR, bL)
27852777 FL.Done _ -> error "Fold terminated consuming partial input"
27862778 Error _ -> do
2787− xs <- fextract fs
2779+ xs <- ffinal fs
27882780 return $ Done cnt1 xs
27892781
27902782 {-# INLINE extractResult #-}
27912783 extractResult n fs r = do
27922784 res <- fstep fs r
27932785 case res of
2794− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
2786+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
27952787 FL.Done c -> return (Done n c)
27962788
2797− extract (DeintercalateInitL fs) = fmap (Done 0) $ fextract fs
2789+ extract (DeintercalateInitL fs) = fmap (Done 0) $ ffinal fs
27982790 extract (DeintercalateL cnt fs sL) = do
27992791 r <- extractL sL
28002792 case r of
@@ -2802,10 +2794,10 @@ deintercalate
28022794 Continue n s -> return $ Continue n (DeintercalateL (cnt - n) fs s)
28032795 Partial _ _ -> error "Partial in extract"
28042796 Error _ -> do
2805− xs <- fextract fs
2797+ xs <- ffinal fs
28062798 return $ Done cnt xs
2807− extract (DeintercalateInitR fs) = fmap (Done 0) $ fextract fs
2808− extract (DeintercalateR cnt fs _) = fmap (Done cnt) $ fextract fs
2799+ extract (DeintercalateInitR fs) = fmap (Done 0) $ ffinal fs
2800+ extract (DeintercalateR cnt fs _) = fmap (Done cnt) $ ffinal fs
28092801 extract (DeintercalateRL cnt bR fs sL) = do
28102802 r <- extractL sL
28112803 case r of
@@ -2817,7 +2809,7 @@ deintercalate
28172809 Continue n s -> return $ Continue n (DeintercalateRL (cnt - n) bR fs s)
28182810 Partial _ _ -> error "Partial in extract"
28192811 Error _ -> do
2820− xs <- fextract fs
2812+ xs <- ffinal fs
28212813 return $ Done cnt xs
28222814
28232815{-# ANN type Deintercalate1State Fuse #-}
@@ -2854,7 +2846,7 @@ deintercalate1 :: Monad m =>
28542846deintercalate1
28552847 (Parser stepL initialL extractL)
28562848 (Parser stepR initialR _)
2857− (Fold fstep finitial fextract) = Parser step initial extract
2849+ (Fold fstep finitial _ ffinal) = Parser step initial extract
28582850
28592851 where
28602852
@@ -2908,7 +2900,7 @@ deintercalate1
29082900 Continue n s -> return $ Continue n (Deintercalate1R (cnt1 - n) fs s)
29092901 Done n b -> processR (cnt1 - n) b fs n
29102902 Error _ -> do
2911− xs <- fextract fs
2903+ xs <- ffinal fs
29122904 return $ Done cnt1 xs
29132905
29142906 step (Deintercalate1InitL cnt fs sL) a = runStepInitL cnt fs sL a
@@ -2937,14 +2929,14 @@ deintercalate1
29372929 -- XXX We could have the fold accept pairs of (bR, bL)
29382930 FL.Done _ -> error "Fold terminated consuming partial input"
29392931 Error _ -> do
2940− xs <- fextract fs
2932+ xs <- ffinal fs
29412933 return $ Done cnt1 xs
29422934
29432935 {-# INLINE extractResult #-}
29442936 extractResult n fs r = do
29452937 res <- fstep fs r
29462938 case res of
2947− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
2939+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
29482940 FL.Done c -> return (Done n c)
29492941
29502942 extract (Deintercalate1InitL cnt fs sL) = do
@@ -2954,8 +2946,8 @@ deintercalate1
29542946 Continue n s -> return $ Continue n (Deintercalate1InitL (cnt - n) fs s)
29552947 Partial _ _ -> error "Partial in extract"
29562948 Error err -> return $ Error err
2957− extract (Deintercalate1InitR fs) = fmap (Done 0) $ fextract fs
2958− extract (Deintercalate1R cnt fs _) = fmap (Done cnt) $ fextract fs
2949+ extract (Deintercalate1InitR fs) = fmap (Done 0) $ ffinal fs
2950+ extract (Deintercalate1R cnt fs _) = fmap (Done cnt) $ ffinal fs
29592951 extract (Deintercalate1RL cnt bR fs sL) = do
29602952 r <- extractL sL
29612953 case r of
@@ -2967,7 +2959,7 @@ deintercalate1
29672959 Continue n s -> return $ Continue n (Deintercalate1RL (cnt - n) bR fs s)
29682960 Partial _ _ -> error "Partial in extract"
29692961 Error _ -> do
2970− xs <- fextract fs
2962+ xs <- ffinal fs
29712963 return $ Done cnt xs
29722964
29732965{-# ANN type SepByState Fuse #-}
@@ -3010,7 +3002,7 @@ sepBy :: Monad m =>
30103002sepBy
30113003 (Parser stepL initialL extractL)
30123004 (Parser stepR initialR _)
3013− (Fold fstep finitial fextract) = Parser step initial extract
3005+ (Fold fstep finitial _ ffinal) = Parser step initial extract
30143006
30153007 where
30163008
@@ -3041,7 +3033,7 @@ sepBy
30413033 Done n b ->
30423034 processL (fstep fs b) n SepByInitR
30433035 Error _ -> do
3044− xs <- fextract fs
3036+ xs <- ffinal fs
30453037 return $ Done cnt1 xs
30463038
30473039 {-# INLINE processR #-}
@@ -3061,7 +3053,7 @@ sepBy
30613053 Continue n s -> return $ Continue n (SepByR (cnt1 - n) fs s)
30623054 Done n _ -> processR (cnt1 - n) fs n
30633055 Error _ -> do
3064− xs <- fextract fs
3056+ xs <- ffinal fs
30653057 return $ Done cnt1 xs
30663058
30673059 step (SepByInitL fs) a = do
@@ -3083,10 +3075,10 @@ sepBy
30833075 extractResult n fs r = do
30843076 res <- fstep fs r
30853077 case res of
3086− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
3078+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
30873079 FL.Done c -> return (Done n c)
30883080
3089− extract (SepByInitL fs) = fmap (Done 0) $ fextract fs
3081+ extract (SepByInitL fs) = fmap (Done 0) $ ffinal fs
30903082 extract (SepByL cnt fs sL) = do
30913083 r <- extractL sL
30923084 case r of
@@ -3094,10 +3086,10 @@ sepBy
30943086 Continue n s -> return $ Continue n (SepByL (cnt - n) fs s)
30953087 Partial _ _ -> error "Partial in extract"
30963088 Error _ -> do
3097− xs <- fextract fs
3089+ xs <- ffinal fs
30983090 return $ Done cnt xs
3099− extract (SepByInitR fs) = fmap (Done 0) $ fextract fs
3100− extract (SepByR cnt fs _) = fmap (Done cnt) $ fextract fs
3091+ extract (SepByInitR fs) = fmap (Done 0) $ ffinal fs
3092+ extract (SepByR cnt fs _) = fmap (Done cnt) $ ffinal fs
31013093
31023094-- | Non-backtracking version of sepBy. Several times faster.
31033095{-# INLINE sepByAll #-}
@@ -3152,7 +3144,7 @@ sepBy1 :: Monad m =>
31523144sepBy1
31533145 (Parser stepL initialL extractL)
31543146 (Parser stepR initialR _)
3155− (Fold fstep finitial fextract) = Parser step initial extract
3147+ (Fold fstep finitial _ ffinal) = Parser step initial extract
31563148
31573149 where
31583150
@@ -3199,7 +3191,7 @@ sepBy1
31993191 Done n b ->
32003192 processL (fstep fs b) n SepBy1InitR
32013193 Error _ -> do
3202− xs <- fextract fs
3194+ xs <- ffinal fs
32033195 return $ Done cnt1 xs
32043196
32053197 {-# INLINE processR #-}
@@ -3219,7 +3211,7 @@ sepBy1
32193211 Continue n s -> return $ Continue n (SepBy1R (cnt1 - n) fs s)
32203212 Done n _ -> processR (cnt1 - n) fs n
32213213 Error _ -> do
3222− xs <- fextract fs
3214+ xs <- ffinal fs
32233215 return $ Done cnt1 xs
32243216
32253217 step (SepBy1InitL cnt fs sL) a = runStepInitL cnt fs sL a
@@ -3236,7 +3228,7 @@ sepBy1
32363228 extractResult n fs r = do
32373229 res <- fstep fs r
32383230 case res of
3239− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
3231+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
32403232 FL.Done c -> return (Done n c)
32413233
32423234 extract (SepBy1InitL cnt fs sL) = do
@@ -3253,10 +3245,10 @@ sepBy1
32533245 Continue n s -> return $ Continue n (SepBy1L (cnt - n) fs s)
32543246 Partial _ _ -> error "Partial in extract"
32553247 Error _ -> do
3256− xs <- fextract fs
3248+ xs <- ffinal fs
32573249 return $ Done cnt xs
3258− extract (SepBy1InitR fs) = fmap (Done 0) $ fextract fs
3259− extract (SepBy1R cnt fs _) = fmap (Done cnt) $ fextract fs
3250+ extract (SepBy1InitR fs) = fmap (Done 0) $ ffinal fs
3251+ extract (SepBy1R cnt fs _) = fmap (Done cnt) $ ffinal fs
32603252
32613253-------------------------------------------------------------------------------
32623254-- Interleaving a collection of parsers
@@ -3286,7 +3278,7 @@ roundRobin _ps _f = undefined
32863278{-# INLINE sequence #-}
32873279sequence :: Monad m =>
32883280 D.Stream m (Parser a m b) -> Fold m b c -> Parser a m c
3289−sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
3281+sequence (D.Stream sstep sstate) (Fold fstep finitial _ ffinal) =
32903282 Parser step initial extract
32913283
32923284 where
@@ -3304,7 +3296,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
33043296 case sres of
33053297 D.Yield p ss1 -> return $ Continue 1 (Just' p, ss1, fs)
33063298 D.Stop -> do
3307− c <- fextract fs
3299+ c <- ffinal fs
33083300 return $ Done 1 c
33093301 D.Skip ss1 -> return $ Continue 1 (Nothing', ss1, fs)
33103302
@@ -3340,7 +3332,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
33403332 FL.Done c -> return $ Done 1 c
33413333 IError err -> return $ Error err
33423334
3343− extract (Nothing', _, fs) = fmap (Done 0) $ fextract fs
3335+ extract (Nothing', _, fs) = fmap (Done 0) $ ffinal fs
33443336 extract (Just' (Parser pstep pinit pextr), ss, fs) = do
33453337 ps <- pinit
33463338 case ps of
@@ -3350,7 +3342,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
33503342 Done n b -> do
33513343 res <- fstep fs b
33523344 case res of
3353− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
3345+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
33543346 FL.Done c -> return (Done n c)
33553347 Error err -> return $ Error err
33563348 Continue n s -> return $ Continue n (Just' (Parser pstep (return (IPartial s)) pextr), ss, fs)
@@ -3358,7 +3350,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
33583350 IDone b -> do
33593351 fres <- fstep fs b
33603352 case fres of
3361− FL.Partial fs1 -> fmap (Done 0) $ fextract fs1
3353+ FL.Partial fs1 -> fmap (Done 0) $ ffinal fs1
33623354 FL.Done c -> return (Done 0 c)
33633355 IError err -> return $ Error err
33643356
@@ -3493,7 +3485,7 @@ manyTill :: Monad m
34933485 => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
34943486manyTill (Parser stepL initialL extractL)
34953487 (Parser stepR initialR _)
3496− (Fold fstep finitial fextract) =
3488+ (Fold fstep finitial _ ffinal) =
34973489 Parser step initial extract
34983490
34993491 where
@@ -3515,7 +3507,7 @@ manyTill (Parser stepL initialL extractL)
35153507 resR <- initialR
35163508 case resR of
35173509 IPartial sr -> return $ p (ManyTillR 0 fs sr)
3518− IDone _ -> d <$> fextract fs
3510+ IDone _ -> d <$> ffinal fs
35193511 IError _ -> scrutL fs p c d e
35203512
35213513 initial = do
@@ -3532,7 +3524,7 @@ manyTill (Parser stepL initialL extractL)
35323524 assertM(cnt + 1 - n >= 0)
35333525 return $ Continue n (ManyTillR (cnt + 1 - n) fs s)
35343526 Done n _ -> do
3535− b <- fextract fs
3527+ b <- ffinal fs
35363528 return $ Done n b
35373529 Error _ -> do
35383530 resL <- initialL
@@ -3571,12 +3563,12 @@ manyTill (Parser stepL initialL extractL)
35713563 Done n b -> do
35723564 r <- fstep fs b
35733565 case r of
3574− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
3566+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
35753567 FL.Done c -> return (Done n c)
35763568 Error err -> return $ Error err
35773569 Continue n s -> return $ Continue n (ManyTillL fs s)
35783570 Partial _ _ -> error "Partial in extract"
3579− extract (ManyTillR _ fs _) = fmap (Done 0) $ fextract fs
3571+ extract (ManyTillR _ fs _) = fmap (Done 0) $ ffinal fs
35803572
35813573-- | @manyThen f collect recover@ repeats the parser @collect@ on the input and
35823574-- collects the output in the supplied fold. If the the parser @collect@ fails,
core/src/Streamly/Internal/Data/Parser/ParserDK.hsdeleted−1544
@@ -1,1544 +0,0 @@
1−{-# OPTIONS_GHC -Wno-orphans #-}
2−
3−-- |
4−-- Module : Streamly.Internal.Data.Parser
5−-- Copyright : (c) 2020 Composewell Technologies
6−-- License : BSD-3-Clause
7−-- Maintainer : streamly@composewell.com
8−-- Stability : pre-release
9−-- Portability : GHC
10−--
11−-- Fast backtracking parsers with stream fusion and native streaming
12−-- capability.
13−--
14−-- 'Applicative' and 'Control.Applicative.Alternative' type class based
15−-- combinators from the
16−-- <http://hackage.haskell.org/package/parser-combinators parser-combinators>
17−-- package can also be used with the 'Parser' type. However, there are two
18−-- important differences between @parser-combinators@ and the equivalent ones
19−-- provided in this module in terms of performance:
20−--
21−-- 1) @parser-combinators@ use plain Haskell lists to collect the results, in a
22−-- strict Monad like IO, the results are necessarily buffered before they can
23−-- be consumed. This may not perform optimally in streaming applications
24−-- processing large amounts of data. Equivalent combinators in this module can
25−-- consume the results of parsing using a 'Fold', thus providing a scalability
26−-- and a composable consumer.
27−--
28−-- 2) Several combinators in this module can be many times faster because of
29−-- stream fusion. For example, 'Streamly.Internal.Data.Parser.many' combinator
30−-- in this module is much faster than the 'Control.Applicative.many' combinator
31−-- of 'Control.Applicative.Alternative' type class.
32−--
33−-- = Errors
34−--
35−-- Failing parsers in this module throw the 'D.ParseError' exception.
36−--
37−-- = Naming
38−--
39−-- As far as possible, try that the names of the combinators in this module are
40−-- consistent with:
41−--
42−-- * <https://hackage.haskell.org/package/base/docs/Text-ParserCombinators-ReadP.html base/Text.ParserCombinators.ReadP>
43−-- * <http://hackage.haskell.org/package/parser-combinators parser-combinators>
44−-- * <http://hackage.haskell.org/package/megaparsec megaparsec>
45−-- * <http://hackage.haskell.org/package/attoparsec attoparsec>
46−-- * <http://hackage.haskell.org/package/parsec parsec>
47−
48−module Streamly.Internal.Data.Parser
49− (
50− K.Parser (..)
51− , D.ParseError (..)
52− , D.Step (..)
53−
54− -- * Downgrade to Fold
55− , toFold
56−
57− -- First order parsers
58− -- * Accumulators
59− , fromFold
60− , fromFoldMaybe
61− , fromPure
62− , fromEffect
63− , die
64− , dieM
65−
66− -- * Map on input
67− , lmap
68− , lmapM
69− , filter
70−
71− -- * Map on output
72− , rmapM
73−
74− -- * Element parsers
75− , peek
76−
77− -- All of these can be expressed in terms of either
78− , one
79− , oneEq
80− , oneNotEq
81− , oneOf
82− , noneOf
83− , eof
84− , satisfy
85− , maybe
86− , either
87−
88− -- * Sequence parsers (tokenizers)
89− --
90− -- | Parsers chained in series, if one parser terminates the composition
91− -- terminates.
92−
93− , lookAhead
94−
95− -- ** By length
96− -- | Grab a sequence of input elements without inspecting them
97− , takeBetween
98− -- , take -- takeBetween 0 n
99− , takeEQ -- takeBetween n n
100− , takeGE -- takeBetween n maxBound
101− , takeP
102−
103− -- Grab a sequence of input elements by inspecting them
104− -- ** Exact match
105− , listEq
106− , listEqBy
107− , streamEqBy
108− , subsequenceBy
109−
110− -- ** By predicate
111− , takeWhileP
112− , takeWhile
113− -- $takeWhile
114− , takeWhile1
115− , dropWhile
116−
117− -- ** Separators
118− , takeEndBy
119− , takeEndBy_
120− , takeEndByEsc
121− -- , takeEndByEsc_
122− , takeStartBy
123− , takeStartBy_
124− , takeEitherSepBy
125− , wordBy
126− -- , wordByEsc
127−
128− -- ** By comparing
129− , groupBy
130− , groupByRolling
131− , groupByRollingEither
132−
133− -- ** Framing
134− -- , takeFramedBy
135− , takeFramedBy_
136− , takeFramedByEsc_
137− , takeFramedByGeneric
138− , wordFramedBy
139− , wordQuotedBy
140−
141− -- | Unimplemented
142− --
143− -- @
144− -- , prefixOf -- match any prefix of a given string
145− -- , suffixOf -- match any suffix of a given string
146− -- , infixOf -- match any substring of a given string
147− -- @
148−
149− -- Second order parsers (parsers using parsers)
150− -- * Binary Combinators
151−
152− -- ** Sequential Applicative
153− , splitWith
154− , split_
155−
156−{-
157− -- ** Parallel Applicatives
158− , teeWith
159− , teeWithFst
160− , teeWithMin
161− -- , teeTill -- like manyTill but parallel
162−-}
163−
164− -- ** Sequential Interleaving
165− -- Use two folds, run a primary parser, its rejected values go to the
166− -- secondary parser.
167− , deintercalate
168− -- , deintercalatePrefix
169− -- , deintercalateSuffix
170−
171− -- *** Special cases
172− -- | TODO: traditional implmentations of these may be of limited use. For
173− -- example, consider parsing lines separated by @\\r\\n@. The main parser
174− -- will have to detect and exclude the sequence @\\r\\n@ anyway so that we
175− -- can apply the "sep" parser.
176− --
177− -- We can instead implement these as special cases of deintercalate.
178− --
179− -- @
180− -- , endBy
181− -- , sepEndBy
182− -- , beginBy
183− -- , sepBeginBy
184− -- , sepAroundBy
185− -- @
186− , sepBy1
187− , sepBy
188−
189− -- ** Sequential Alternative
190− , alt
191−
192−{-
193− -- ** Parallel Alternatives
194− , shortest
195− , longest
196− -- , fastest
197−-}
198−
199− -- * N-ary Combinators
200− -- ** Sequential Collection
201− , concatSequence
202− , concatMap
203−
204− -- ** Sequential Repetition
205− , count
206− , countBetween
207−
208− , manyP
209− , many
210− , some
211− , manyTillP
212− , manyTill
213− , manyThen
214−
215− -- * Distribution
216− --
217− -- | A simple and stupid impl would be to just convert the stream to an
218− -- array and give the array reference to all consumers. The array can be
219− -- grown on demand by any consumer and truncated when nonbody needs it.
220−
221− -- ** Distribute to collection
222− -- ** Distribute to repetition
223−
224− -- ** Interleaved collection
225− -- |
226− --
227− -- 1. Round robin
228− -- 2. Priority based
229− , roundRobin
230−
231− -- ** Collection of Alternatives
232− -- | Unimplemented
233− --
234− -- @
235− -- , shortestN
236− -- , longestN
237− -- , fastestN -- first N successful in time
238− -- , choiceN -- first N successful in position
239− -- @
240− -- , choice -- first successful in position
241−
242− -- ** Repeated Alternatives
243− , retryMaxTotal
244− , retryMaxSuccessive
245− , retry
246−
247− -- * Deprecated
248− , next
249− )
250−where
251−
252−import Streamly.Internal.Data.Fold.Type (Fold(..))
253−import Streamly.Internal.Data.Parser.ParserK.Type (Parser)
254−
255−import qualified Data.Foldable as Foldable
256−import qualified Streamly.Internal.Data.Fold.Type as FL
257−import qualified Streamly.Internal.Data.Parser as D
258−import qualified Streamly.Internal.Data.Parser.ParserK.Type as K
259−
260−#ifdef USE_STREAMK
261−import Streamly.Internal.Data.StreamK (Stream)
262−import qualified Streamly.Internal.Data.StreamK as Stream
263−#else
264−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
265−import qualified Streamly.Internal.Data.Stream.StreamD.Type as Stream
266−#endif
267−
268−import Prelude hiding
269− ( any, all, dropWhile, take, takeWhile, sequence, concatMap, maybe, either
270− , filter )
271−
272−--
273−-- $setup
274−-- >>> :m
275−-- >>> import Prelude hiding (any, all, dropWhile, take, takeWhile, sequence, concatMap, maybe, either, filter)
276−-- >>> import Control.Applicative ((<|>))
277−-- >>> import Data.Char (isSpace)
278−-- >>> import qualified Data.Maybe as Maybe
279−-- >>> import qualified Data.Foldable as Foldable
280−-- >>> import qualified Streamly.Data.Stream as Stream
281−-- >>> import qualified Streamly.Internal.Data.Stream as Stream (parse, parseMany)
282−-- >>> import qualified Streamly.Internal.Data.Fold as Fold
283−-- >>> import qualified Streamly.Internal.Data.Parser as Parser
284−
285−-------------------------------------------------------------------------------
286−-- Downgrade a parser to a Fold
287−-------------------------------------------------------------------------------
288−--
289−-- | Make a 'Fold' from a 'Parser'. The fold just throws an exception if the
290−-- parser fails or tries to backtrack.
291−--
292−-- This can be useful in combinators that accept a Fold and we know that a
293−-- Parser cannot fail or failure exception is acceptable as there is no way to
294−-- recover.
295−--
296−-- /Pre-release/
297−--
298−{-# INLINE toFold #-}
299−toFold :: Monad m => Parser a m b -> Fold m a b
300−toFold p = D.toFold $ D.fromParserK p
301−
302−-------------------------------------------------------------------------------
303−-- Upgrade folds to parses
304−-------------------------------------------------------------------------------
305−--
306−-- | Make a 'Parser' from a 'Fold'.
307−--
308−{-# INLINE fromFold #-}
309−fromFold :: Monad m => Fold m a b -> Parser a m b
310−fromFold = D.toParserK . D.fromFold
311−
312−-- | Convert a Maybe returning fold to an error returning parser. The first
313−-- argument is the error message that the parser would return when the fold
314−-- returns Nothing.
315−--
316−-- /Pre-release/
317−--
318−{-# INLINE fromFoldMaybe #-}
319−fromFoldMaybe :: Monad m => String -> Fold m a (Maybe b) -> Parser a m b
320−fromFoldMaybe err = D.toParserK . D.fromFoldMaybe err
321−
322−-------------------------------------------------------------------------------
323−-- Terminating but not failing folds
324−-------------------------------------------------------------------------------
325−--
326−-- This is the dual of stream "fromPure".
327−--
328−-- | A parser that always yields a pure value without consuming any input.
329−--
330−{-# INLINE [3] fromPure #-}
331−fromPure :: Monad m => b -> Parser a m b
332−fromPure = D.toParserK . D.fromPure
333−{-# RULES "fromPure fallback to CPS" [2]
334− forall a. D.toParserK (D.fromPure a) = K.fromPure a #-}
335−
336−-- This is the dual of stream "fromEffect".
337−--
338−-- | A parser that always yields the result of an effectful action without
339−-- consuming any input.
340−--
341−{-# INLINE fromEffect #-}
342−fromEffect :: Monad m => m b -> Parser a m b
343−fromEffect = K.fromEffect -- D.toParserK . D.fromEffect
344−
345−-- This is the dual of "nil".
346−--
347−-- | A parser that always fails with an error message without consuming
348−-- any input.
349−--
350−{-# INLINE [3] die #-}
351−die :: Monad m => String -> Parser a m b
352−die = D.toParserK . D.die
353−{-# RULES "die fallback to CPS" [2]
354− forall a. D.toParserK (D.die a) = K.die a #-}
355−
356−-- This is the dual of "nilM".
357−--
358−-- | A parser that always fails with an effectful error message and without
359−-- consuming any input.
360−--
361−-- /Pre-release/
362−--
363−{-# INLINE dieM #-}
364−dieM :: Monad m => m String -> Parser a m b
365−dieM = D.toParserK . D.dieM
366−
367−-------------------------------------------------------------------------------
368−-- Map on input
369−-------------------------------------------------------------------------------
370−
371−-- | @lmap f parser@ maps the function @f@ on the input of the parser.
372−--
373−-- >>> Stream.parse (Parser.lmap (\x -> x * x) (Parser.fromFold Fold.sum)) (Stream.enumerateFromTo 1 100)
374−-- Right 338350
375−--
376−-- > lmap = Parser.lmapM return
377−--
378−{-# INLINE lmap #-}
379−lmap :: Monad m => (a -> b) -> Parser b m r -> Parser a m r
380−lmap f p = D.toParserK $ D.lmap f $ D.fromParserK p
381−
382−-- | @lmapM f parser@ maps the monadic function @f@ on the input of the parser.
383−--
384−{-# INLINE lmapM #-}
385−lmapM :: Monad m => (a -> m b) -> Parser b m r -> Parser a m r
386−lmapM f p = D.toParserK $ D.lmapM f $ D.fromParserK p
387−
388−-- | @rmapM f parser@ maps the monadic function @f@ on the output of the parser.
389−--
390−-- >>> rmap = fmap
391−{-# INLINE rmapM #-}
392−rmapM :: Monad m => (b -> m c) -> Parser a m b -> Parser a m c
393−rmapM f p = D.toParserK $ D.rmapM f $ D.fromParserK p
394−
395−-- | Include only those elements that pass a predicate.
396−--
397−-- >>> Stream.parse (Parser.filter (> 5) (Parser.fromFold Fold.sum)) $ Stream.fromList [1..10]
398−-- Right 40
399−--
400−{-# INLINE filter #-}
401−filter :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
402−filter f p = D.toParserK $ D.filter f $ D.fromParserK p
403−
404−-------------------------------------------------------------------------------
405−-- Failing Parsers
406−-------------------------------------------------------------------------------
407−
408−-- | Peek the head element of a stream, without consuming it. Fails if it
409−-- encounters end of input.
410−--
411−-- >>> Stream.parse ((,) <$> Parser.peek <*> Parser.satisfy (> 0)) $ Stream.fromList [1]
412−-- Right (1,1)
413−--
414−-- @
415−-- peek = lookAhead (satisfy True)
416−-- @
417−--
418−{-# INLINE peek #-}
419−peek :: Monad m => Parser a m a
420−peek = D.toParserK D.peek
421−
422−-- | Succeeds if we are at the end of input, fails otherwise.
423−--
424−-- >>> Stream.parse ((,) <$> Parser.satisfy (> 0) <*> Parser.eof) $ Stream.fromList [1]
425−-- Right (1,())
426−--
427−{-# INLINE eof #-}
428−eof :: Monad m => Parser a m ()
429−eof = D.toParserK D.eof
430−
431−-- | Returns the next element if it passes the predicate, fails otherwise.
432−--
433−-- >>> Stream.parse (Parser.satisfy (== 1)) $ Stream.fromList [1,0,1]
434−-- Right 1
435−--
436−-- >>> toMaybe f x = if f x then Just x else Nothing
437−-- >>> satisfy f = Parser.maybe (toMaybe f)
438−--
439−{-# INLINE satisfy #-}
440−satisfy :: Monad m => (a -> Bool) -> Parser a m a
441−satisfy = D.toParserK . D.satisfy
442−
443−-- | Consume one element from the head of the stream. Fails if it encounters
444−-- end of input.
445−--
446−-- >>> one = Parser.satisfy $ const True
447−--
448−{-# INLINE one #-}
449−one :: Monad m => Parser a m a
450−one = satisfy $ const True
451−
452−-- Alternate names: "only", "onlyThis".
453−
454−-- | Match a specific element.
455−--
456−-- >>> oneEq x = Parser.satisfy (== x)
457−--
458−{-# INLINE oneEq #-}
459−oneEq :: (Monad m, Eq a) => a -> Parser a m a
460−oneEq x = satisfy (== x)
461−
462−-- Alternate names: "exclude", "notThis".
463−
464−-- | Match anything other than the supplied element.
465−--
466−-- >>> oneNotEq x = Parser.satisfy (/= x)
467−--
468−{-# INLINE oneNotEq #-}
469−oneNotEq :: (Monad m, Eq a) => a -> Parser a m a
470−oneNotEq x = satisfy (/= x)
471−
472−-- | Match any one of the elements in the supplied list.
473−--
474−-- >>> oneOf xs = Parser.satisfy (`Foldable.elem` xs)
475−--
476−-- When performance matters a pattern matching predicate could be more
477−-- efficient than a 'Foldable' datatype:
478−--
479−-- @
480−-- let p x =
481−-- case x of
482−-- 'a' -> True
483−-- 'e' -> True
484−-- _ -> False
485−-- in satisfy p
486−-- @
487−--
488−-- GHC may use a binary search instead of linear search in the list.
489−-- Alternatively, you can also use an array instead of list for storage and
490−-- search.
491−--
492−{-# INLINE oneOf #-}
493−oneOf :: (Monad m, Eq a, Foldable f) => f a -> Parser a m a
494−oneOf xs = satisfy (`Foldable.elem` xs)
495−
496−-- | See performance notes in 'oneOf'.
497−--
498−-- >>> noneOf xs = Parser.satisfy (`Foldable.notElem` xs)
499−--
500−{-# INLINE noneOf #-}
501−noneOf :: (Monad m, Eq a, Foldable f) => f a -> Parser a m a
502−noneOf xs = satisfy (`Foldable.notElem` xs)
503−
504−-- | Return the next element of the input. Returns 'Nothing'
505−-- on end of input. Also known as 'head'.
506−--
507−-- /Pre-release/
508−--
509−{-# DEPRECATED next "Please use \"fromFold Fold.one\" instead" #-}
510−{-# INLINE next #-}
511−next :: Monad m => Parser a m (Maybe a)
512−next = D.toParserK D.next
513−
514−-- | Map a 'Maybe' returning function on the next element in the stream. The
515−-- parser fails if the function returns 'Nothing' otherwise returns the 'Just'
516−-- value.
517−--
518−-- >>> toEither = Maybe.maybe (Left "maybe: predicate failed") Right
519−-- >>> maybe f = Parser.either (toEither . f)
520−--
521−-- >>> maybe f = Parser.fromFoldMaybe "maybe: predicate failed" (Fold.maybe f)
522−--
523−-- /Pre-release/
524−--
525−{-# INLINE maybe #-}
526−maybe :: Monad m => (a -> Maybe b) -> Parser a m b
527−maybe = D.toParserK . D.maybe
528−
529−-- | Map an 'Either' returning function on the next element in the stream. If
530−-- the function returns 'Left err', the parser fails with the error message
531−-- @err@ otherwise returns the 'Right' value.
532−--
533−-- /Pre-release/
534−--
535−{-# INLINE either #-}
536−either :: Monad m => (a -> Either String b) -> Parser a m b
537−either = D.toParserK . D.either
538−
539−-------------------------------------------------------------------------------
540−-- Taking elements
541−-------------------------------------------------------------------------------
542−
543−-- | @takeBetween m n@ takes a minimum of @m@ and a maximum of @n@ input
544−-- elements and folds them using the supplied fold.
545−--
546−-- Stops after @n@ elements.
547−-- Fails if the stream ends before @m@ elements could be taken.
548−--
549−-- Examples: -
550−--
551−-- @
552−-- >>> :{
553−-- takeBetween' low high ls = Stream.parse prsr (Stream.fromList ls)
554−-- where prsr = Parser.takeBetween low high Fold.toList
555−-- :}
556−--
557−-- @
558−--
559−-- >>> takeBetween' 2 4 [1, 2, 3, 4, 5]
560−-- Right [1,2,3,4]
561−--
562−-- >>> takeBetween' 2 4 [1, 2]
563−-- Right [1,2]
564−--
565−-- >>> takeBetween' 2 4 [1]
566−-- Left (ParseError "takeBetween: Expecting alteast 2 elements, got 1")
567−--
568−-- >>> takeBetween' 0 0 [1, 2]
569−-- Right []
570−--
571−-- >>> takeBetween' 0 1 []
572−-- Right []
573−--
574−-- @takeBetween@ is the most general take operation, other take operations can
575−-- be defined in terms of takeBetween. For example:
576−--
577−-- >>> take n = Parser.takeBetween 0 n
578−-- >>> takeEQ n = Parser.takeBetween n n
579−-- >>> takeGE n = Parser.takeBetween n maxBound
580−--
581−-- /Pre-release/
582−--
583−{-# INLINE takeBetween #-}
584−takeBetween :: Monad m =>
585− Int -> Int -> Fold m a b -> Parser a m b
586−takeBetween m n = D.toParserK . D.takeBetween m n
587−
588−-- | Stops after taking exactly @n@ input elements.
589−--
590−-- * Stops - after consuming @n@ elements.
591−-- * Fails - if the stream or the collecting fold ends before it can collect
592−-- exactly @n@ elements.
593−--
594−-- >>> Stream.parse (Parser.takeEQ 4 Fold.toList) $ Stream.fromList [1,0,1]
595−-- Left (ParseError "takeEQ: Expecting exactly 4 elements, input terminated on 3")
596−--
597−{-# INLINE takeEQ #-}
598−takeEQ :: Monad m => Int -> Fold m a b -> Parser a m b
599−takeEQ n = D.toParserK . D.takeEQ n
600−
601−-- | Take at least @n@ input elements, but can collect more.
602−--
603−-- * Stops - when the collecting fold stops.
604−-- * Fails - if the stream or the collecting fold ends before producing @n@
605−-- elements.
606−--
607−-- >>> Stream.parse (Parser.takeGE 4 Fold.toList) $ Stream.fromList [1,0,1]
608−-- Left (ParseError "takeGE: Expecting at least 4 elements, input terminated on 3")
609−--
610−-- >>> Stream.parse (Parser.takeGE 4 Fold.toList) $ Stream.fromList [1,0,1,0,1]
611−-- Right [1,0,1,0,1]
612−--
613−-- /Pre-release/
614−--
615−{-# INLINE takeGE #-}
616−takeGE :: Monad m => Int -> Fold m a b -> Parser a m b
617−takeGE n = D.toParserK . D.takeGE n
618−
619−-------------------------------------------------------------------------------
620−-- Take until a condition
621−-------------------------------------------------------------------------------
622−
623−-- $takeWhile
624−-- Note: This is called @takeWhileP@ and @munch@ in some parser libraries.
625−
626−-- XXX We should perhaps use only takeWhileP and rename it to takeWhile.
627−--
628−-- | Like 'takeWhile' but uses a 'Parser' instead of a 'Fold' to collect the
629−-- input. The combinator stops when the condition fails or if the collecting
630−-- parser stops.
631−--
632−-- Other interesting parsers can be implemented in terms of this parser:
633−--
634−-- >>> takeWhile1 cond p = Parser.takeWhileP cond (Parser.takeBetween 1 maxBound p)
635−-- >>> takeWhileBetween cond m n p = Parser.takeWhileP cond (Parser.takeBetween m n p)
636−--
637−-- Stops: when the condition fails or the collecting parser stops.
638−-- Fails: when the collecting parser fails.
639−--
640−-- /Pre-release/
641−--
642−{-# INLINE takeWhileP #-}
643−takeWhileP :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
644−takeWhileP cond p = D.toParserK $ D.takeWhileP cond (D.fromParserK p)
645−
646−-- | Collect stream elements until an element fails the predicate. The element
647−-- on which the predicate fails is returned back to the input stream.
648−--
649−-- * Stops - when the predicate fails or the collecting fold stops.
650−-- * Fails - never.
651−--
652−-- >>> Stream.parse (Parser.takeWhile (== 0) Fold.toList) $ Stream.fromList [0,0,1,0,1]
653−-- Right [0,0]
654−--
655−-- >>> takeWhile cond f = Parser.takeWhileP cond (Parser.fromFold f)
656−--
657−-- We can implement a @breakOn@ using 'takeWhile':
658−--
659−-- @
660−-- breakOn p = takeWhile (not p)
661−-- @
662−--
663−{-# INLINE takeWhile #-}
664−takeWhile :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
665−takeWhile cond = D.toParserK . D.takeWhile cond
666−-- takeWhile cond f = takeWhileP cond (fromFold f)
667−
668−-- | Like 'takeWhile' but takes at least one element otherwise fails.
669−--
670−-- >>> takeWhile1 cond p = Parser.takeWhileP cond (Parser.takeBetween 1 maxBound p)
671−--
672−{-# INLINE takeWhile1 #-}
673−takeWhile1 :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
674−takeWhile1 cond = D.toParserK . D.takeWhile1 cond
675−-- takeWhile1 cond f = takeWhileP cond (takeBetween 1 maxBound f)
676−
677−-- | Drain the input as long as the predicate succeeds, running the effects and
678−-- discarding the results.
679−--
680−-- This is also called @skipWhile@ in some parsing libraries.
681−--
682−-- >>> dropWhile p = Parser.takeWhile p Fold.drain
683−--
684−{-# INLINE dropWhile #-}
685−dropWhile :: Monad m => (a -> Bool) -> Parser a m ()
686−dropWhile p = takeWhile p FL.drain
687−
688−-------------------------------------------------------------------------------
689−-- Separators
690−-------------------------------------------------------------------------------
691−
692−-- XXX We can remove Maybe from esc
693−{-# INLINE takeFramedByGeneric #-}
694−takeFramedByGeneric :: Monad m =>
695− Maybe (a -> Bool)
696− -> Maybe (a -> Bool)
697− -> Maybe (a -> Bool)
698− -> Fold m a b
699− -> Parser a m b
700−takeFramedByGeneric esc begin end f =
701− D.toParserK $ D.takeFramedByGeneric esc begin end f
702−
703−-- | @takeEndBy cond parser@ parses a token that ends by a separator chosen by
704−-- the supplied predicate. The separator is also taken with the token.
705−--
706−-- This can be combined with other parsers to implement other interesting
707−-- parsers as follows:
708−--
709−-- >>> takeEndByLE cond n p = Parser.takeEndBy cond (Parser.fromFold $ Fold.take n p)
710−-- >>> takeEndByBetween cond m n p = Parser.takeEndBy cond (Parser.takeBetween m n p)
711−--
712−-- >>> takeEndBy = Parser.takeEndByEsc (const False)
713−--
714−-- See also "Streamly.Data.Fold.takeEndBy". Unlike the fold, the collecting
715−-- parser in the takeEndBy parser can decide whether to fail or not if the
716−-- stream does not end with separator.
717−--
718−-- /Pre-release/
719−--
720−{-# INLINE takeEndBy #-}
721−takeEndBy :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
722−takeEndBy cond = D.toParserK . D.takeEndBy cond . D.fromParserK
723−-- takeEndBy = takeEndByEsc (const False)
724−
725−-- | Like 'takeEndBy' but the separator is dropped.
726−--
727−-- See also "Streamly.Data.Fold.takeEndBy_".
728−--
729−-- /Pre-release/
730−--
731−{-# INLINE takeEndBy_ #-}
732−takeEndBy_ :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
733−takeEndBy_ cond = D.toParserK . D.takeEndBy_ cond . D.fromParserK
734−{-
735−takeEndBy_ isEnd p =
736− takeFramedByGeneric Nothing Nothing (Just isEnd) (toFold p)
737−-}
738−
739−-- | Take either the separator or the token. Separator is a Left value and
740−-- token is Right value.
741−--
742−-- /Unimplemented/
743−{-# INLINE takeEitherSepBy #-}
744−takeEitherSepBy :: -- Monad m =>
745− (a -> Bool) -> Fold m (Either a b) c -> Parser a m c
746−takeEitherSepBy _cond = undefined -- D.toParserK . D.takeEitherSepBy cond
747−
748−-- | Parse a token that starts with an element chosen by the predicate. The
749−-- parser fails if the input does not start with the selected element.
750−--
751−-- * Stops - when the predicate succeeds in non-leading position.
752−-- * Fails - when the predicate fails in the leading position.
753−--
754−-- >>> splitWithPrefix p f = Stream.parseMany (Parser.takeStartBy p f)
755−--
756−-- Examples: -
757−--
758−-- >>> p = Parser.takeStartBy (== ',') Fold.toList
759−-- >>> leadingComma = Stream.parse p . Stream.fromList
760−-- >>> leadingComma "a,b"
761−-- Left (ParseError "takeStartBy: missing frame start")
762−-- ...
763−-- >>> leadingComma ",,"
764−-- Right ","
765−-- >>> leadingComma ",a,b"
766−-- Right ",a"
767−-- >>> leadingComma ""
768−-- Right ""
769−--
770−-- /Pre-release/
771−--
772−{-# INLINE takeStartBy #-}
773−takeStartBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
774−takeStartBy cond = D.toParserK . D.takeStartBy cond
775−
776−-- | Like 'takeStartBy' but drops the separator.
777−--
778−-- >>> takeStartBy_ isBegin = Parser.takeFramedByGeneric Nothing (Just isBegin) Nothing
779−--
780−{-# INLINE takeStartBy_ #-}
781−takeStartBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
782−takeStartBy_ isBegin = takeFramedByGeneric Nothing (Just isBegin) Nothing
783−
784−-------------------------------------------------------------------------------
785−-- Quoting and Escaping
786−-------------------------------------------------------------------------------
787−
788−-- | Like 'takeEndBy' but the separator elements can be escaped using an
789−-- escape char determined by the first predicate. The escape characters are
790−-- removed.
791−--
792−-- /pre-release/
793−{-# INLINE takeEndByEsc #-}
794−takeEndByEsc :: Monad m =>
795− (a -> Bool) -> (a -> Bool) -> Parser a m b -> Parser a m b
796−takeEndByEsc isEsc isEnd p =
797− D.toParserK $ D.takeEndByEsc isEsc isEnd (D.fromParserK p)
798−
799−-- | @takeFramedByEsc_ isEsc isBegin isEnd fold@ parses a token framed using a
800−-- begin and end predicate, and an escape character. The frame begin and end
801−-- characters lose their special meaning if preceded by the escape character.
802−--
803−-- Nested frames are allowed if begin and end markers are different, nested
804−-- frames must be balanced unless escaped, nested frame markers are emitted as
805−-- it is.
806−--
807−-- For example,
808−--
809−-- >>> p = Parser.takeFramedByEsc_ (== '\\') (== '{') (== '}') Fold.toList
810−-- >>> Stream.parse p $ Stream.fromList "{hello}"
811−-- Right "hello"
812−-- >>> Stream.parse p $ Stream.fromList "{hello {world}}"
813−-- Right "hello {world}"
814−-- >>> Stream.parse p $ Stream.fromList "{hello \\{world}"
815−-- Right "hello {world"
816−-- >>> Stream.parse p $ Stream.fromList "{hello {world}"
817−-- Left (ParseError "takeFramedByEsc_: missing frame end")
818−--
819−-- /Pre-release/
820−{-# INLINE takeFramedByEsc_ #-}
821−takeFramedByEsc_ :: Monad m =>
822− (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
823−takeFramedByEsc_ isEsc isBegin isEnd f =
824− D.toParserK $ D.takeFramedByEsc_ isEsc isBegin isEnd f
825−-- takeEndByEsc_ isEsc isEnd p =
826−-- takeFramedByGeneric (Just isEsc) Nothing (Just isEnd) (toFold p)
827−
828−-- | @takeFramedBy_ isBegin isEnd fold@ parses a token framed by a begin and an
829−-- end predicate.
830−--
831−-- >>> takeFramedBy_ = Parser.takeFramedByEsc_ (const False)
832−--
833−{-# INLINE takeFramedBy_ #-}
834−takeFramedBy_ :: Monad m =>
835− (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
836−takeFramedBy_ isBegin isEnd f = D.toParserK $ D.takeFramedBy_ isBegin isEnd f
837−-- takeFramedBy_ isBegin isEnd =
838−-- takeFramedByGeneric (Just (const False)) (Just isBegin) (Just isEnd)
839−
840−-------------------------------------------------------------------------------
841−-- Grouping and words
842−-------------------------------------------------------------------------------
843−
844−-- Note we can also get words using something like:
845−-- sepBy FL.toList (takeWhile (not . p) Fold.toList) (dropWhile p)
846−--
847−-- But that won't be as efficient and ergonomic.
848−--
849−-- | Like 'splitOn' but strips leading, trailing, and repeated separators.
850−-- Therefore, @".a..b."@ having '.' as the separator would be parsed as
851−-- @["a","b"]@. In other words, its like parsing words from whitespace
852−-- separated text.
853−--
854−-- * Stops - when it finds a word separator after a non-word element
855−-- * Fails - never.
856−--
857−-- >>> wordBy = Parser.wordFramedBy (const False) (const False) (const False)
858−--
859−-- @
860−-- S.wordsBy pred f = S.parseMany (PR.wordBy pred f)
861−-- @
862−--
863−{-# INLINE wordBy #-}
864−wordBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
865−wordBy f = D.toParserK . D.wordBy f
866−
867−-- | Like 'wordBy' but treats anything inside a pair of quotes as a single
868−-- word, the quotes can be escaped by an escape character. Recursive quotes
869−-- are possible if quote begin and end characters are different, quotes must be
870−-- balanced. Outermost quotes are stripped.
871−--
872−-- >>> braces = Parser.wordFramedBy (== '\\') (== '{') (== '}') isSpace Fold.toList
873−-- >>> Stream.parse braces $ Stream.fromList "{ab} cd"
874−-- Right "ab"
875−-- >>> Stream.parse braces $ Stream.fromList "{ab}{cd}"
876−-- Right "abcd"
877−-- >>> Stream.parse braces $ Stream.fromList "a{b} cd"
878−-- Right "ab"
879−-- >>> Stream.parse braces $ Stream.fromList "a{{b}} cd"
880−-- Right "a{b}"
881−--
882−-- >>> quotes = Parser.wordFramedBy (== '\\') (== '"') (== '"') isSpace Fold.toList
883−-- >>> Stream.parse quotes $ Stream.fromList "\"a\"\"b\""
884−-- Right "ab"
885−--
886−{-# INLINE wordFramedBy #-}
887−wordFramedBy :: Monad m =>
888− (a -> Bool) -- ^ Escape
889− -> (a -> Bool) -- ^ left quote
890− -> (a -> Bool) -- ^ right quote
891− -> (a -> Bool) -- ^ word seperator
892− -> Fold m a b
893− -> Parser a m b
894−wordFramedBy isEsc isBegin isEnd isSpc =
895− D.toParserK . D.wordFramedBy isEsc isBegin isEnd isSpc
896−
897−-- | Like 'wordFramedBy' but the closing quote is determined by the opening
898−-- quote. The first quote begin starts a quote that is closed by its
899−-- corresponding closing quote.
900−--
901−-- 'wordFramedBy' and 'wordQuotedBy' both allow multiple quote characters based
902−-- on the predicates but 'wordQuotedBy' always fixes the quote at the first
903−-- occurrence and then it is closed only by the corresponding closing quote.
904−-- Therefore, other quoting characters can be embedded inside it as normal
905−-- characters. On the other hand, 'wordFramedBy' would close the quote as soon
906−-- as it encounters any of the closing quotes.
907−--
908−-- >>> q = (`elem` ['"', '\''])
909−-- >>> p kQ = Parser.wordQuotedBy kQ (== '\\') q q id isSpace Fold.toList
910−--
911−-- >>> Stream.parse (p False) $ Stream.fromList "a\"b'c\";'d\"e'f ghi"
912−-- Right "ab'c;d\"ef"
913−--
914−-- >>> Stream.parse (p True) $ Stream.fromList "a\"b'c\";'d\"e'f ghi"
915−-- Right "a\"b'c\";'d\"e'f"
916−--
917−{-# INLINE wordQuotedBy #-}
918−wordQuotedBy :: (Monad m, Eq a) =>
919− Bool -- ^ keep the quotes in the output
920− -> (a -> Bool) -- ^ Escape
921− -> (a -> Bool) -- ^ left quote
922− -> (a -> Bool) -- ^ right quote
923− -> (a -> a) -- ^ get right quote from left quote
924− -> (a -> Bool) -- ^ word seperator
925− -> Fold m a b
926− -> Parser a m b
927−wordQuotedBy keepQuotes isEsc isBegin isEnd toRight isSpc =
928− D.toParserK . D.wordQuotedBy keepQuotes isEsc isBegin isEnd toRight isSpc
929−
930−-- | Given an input stream @[a,b,c,...]@ and a comparison function @cmp@, the
931−-- parser assigns the element @a@ to the first group, then if @a \`cmp` b@ is
932−-- 'True' @b@ is also assigned to the same group. If @a \`cmp` c@ is 'True'
933−-- then @c@ is also assigned to the same group and so on. When the comparison
934−-- fails the parser is terminated. Each group is folded using the 'Fold' @f@ and
935−-- the result of the fold is the result of the parser.
936−--
937−-- * Stops - when the comparison fails.
938−-- * Fails - never.
939−--
940−-- >>> :{
941−-- runGroupsBy eq =
942−-- Stream.fold Fold.toList
943−-- . Stream.parseMany (Parser.groupBy eq Fold.toList)
944−-- . Stream.fromList
945−-- :}
946−--
947−-- >>> runGroupsBy (<) []
948−-- []
949−--
950−-- >>> runGroupsBy (<) [1]
951−-- [Right [1]]
952−--
953−-- >>> runGroupsBy (<) [3, 5, 4, 1, 2, 0]
954−-- [Right [3,5,4],Right [1,2],Right [0]]
955−--
956−{-# INLINE groupBy #-}
957−groupBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
958−groupBy eq = D.toParserK . D.groupBy eq
959−
960−-- | Unlike 'groupBy' this combinator performs a rolling comparison of two
961−-- successive elements in the input stream. Assuming the input stream
962−-- is @[a,b,c,...]@ and the comparison function is @cmp@, the parser
963−-- first assigns the element @a@ to the first group, then if @a \`cmp` b@ is
964−-- 'True' @b@ is also assigned to the same group. If @b \`cmp` c@ is 'True'
965−-- then @c@ is also assigned to the same group and so on. When the comparison
966−-- fails the parser is terminated. Each group is folded using the 'Fold' @f@ and
967−-- the result of the fold is the result of the parser.
968−--
969−-- * Stops - when the comparison fails.
970−-- * Fails - never.
971−--
972−-- >>> :{
973−-- runGroupsByRolling eq =
974−-- Stream.fold Fold.toList
975−-- . Stream.parseMany (Parser.groupByRolling eq Fold.toList)
976−-- . Stream.fromList
977−-- :}
978−--
979−-- >>> runGroupsByRolling (<) []
980−-- []
981−--
982−-- >>> runGroupsByRolling (<) [1]
983−-- [Right [1]]
984−--
985−-- >>> runGroupsByRolling (<) [3, 5, 4, 1, 2, 0]
986−-- [Right [3,5],Right [4],Right [1,2],Right [0]]
987−--
988−-- /Pre-release/
989−--
990−{-# INLINE groupByRolling #-}
991−groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
992−groupByRolling eq = D.toParserK . D.groupByRolling eq
993−
994−-- | Like 'groupByRolling', but if the predicate is 'True' then collects using
995−-- the first fold as long as the predicate holds 'True', if the predicate is
996−-- 'False' collects using the second fold as long as it remains 'False'.
997−-- Returns 'Left' for the first case and 'Right' for the second case.
998−--
999−-- For example, if we want to detect sorted sequences in a stream, both
1000−-- ascending and descending cases we can use 'groupByRollingEither (<=)
1001−-- Fold.toList Fold.toList'.
1002−--
1003−-- /Pre-release/
1004−{-# INLINE groupByRollingEither #-}
1005−groupByRollingEither :: Monad m =>
1006− (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
1007−groupByRollingEither eq f1 = D.toParserK . D.groupByRollingEither eq f1
1008−
1009−-- | Like 'listEqBy' but uses a stream instead of a list and does not return
1010−-- the stream.
1011−--
1012−-- See also: "Streamly.Data.Stream.streamEqBy"
1013−--
1014−{-# INLINE streamEqBy #-}
1015−streamEqBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Parser a m ()
1016−streamEqBy cmp = D.toParserK . D.streamEqBy cmp
1017−
1018−-- | Match the given sequence of elements using the given comparison function.
1019−-- Returns the original sequence if successful.
1020−--
1021−-- Definition:
1022−--
1023−-- >>> listEqBy cmp xs = Parser.streamEqBy cmp (Stream.fromList xs) *> Parser.fromPure xs
1024−--
1025−-- Examples:
1026−--
1027−-- >>> Stream.parse (Parser.listEqBy (==) "string") $ Stream.fromList "string"
1028−-- Right "string"
1029−--
1030−-- >>> Stream.parse (Parser.listEqBy (==) "mismatch") $ Stream.fromList "match"
1031−-- Left (ParseError "streamEqBy: mismtach occurred")
1032−--
1033−{-# INLINE listEqBy #-}
1034−listEqBy :: Monad m => (a -> a -> Bool) -> [a] -> Parser a m [a]
1035−-- listEqBy cmp xs = D.toParserK (D.listEqBy cmp xs)
1036−listEqBy cmp xs = streamEqBy cmp (Stream.fromList xs) *> fromPure xs
1037−
1038−-- Rename to "list".
1039−-- | Match the input sequence with the supplied list and return it if
1040−-- successful.
1041−--
1042−-- >>> listEq = Parser.listEqBy (==)
1043−--
1044−{-# INLINE listEq #-}
1045−listEq :: (Monad m, Eq a) => [a] -> Parser a m [a]
1046−listEq = listEqBy (==)
1047−
1048−-- | Match if the input stream is a subsequence of the argument stream i.e. all
1049−-- the elements of the input stream occur, in order, in the argument stream.
1050−-- The elements do not have to occur consecutively. A sequence is considered a
1051−-- subsequence of itself.
1052−{-# INLINE subsequenceBy #-}
1053−subsequenceBy :: -- Monad m =>
1054− (a -> a -> Bool) -> Stream m a -> Parser a m ()
1055−subsequenceBy = undefined
1056−
1057−{-
1058−-- Should go in Data.Parser.Regex in streamly package so that it can depend on
1059−-- regex backends.
1060−{-# INLINE regexPosix #-}
1061−regexPosix :: -- Monad m =>
1062− Regex -> Parser m a (Maybe (Array (MatchOffset, MatchLength)))
1063−regexPosix = undefined
1064−
1065−{-# INLINE regexPCRE #-}
1066−regexPCRE :: -- Monad m =>
1067− Regex -> Parser m a (Maybe (Array (MatchOffset, MatchLength)))
1068−regexPCRE = undefined
1069−-}
1070−
1071−-------------------------------------------------------------------------------
1072−-- nested parsers
1073−-------------------------------------------------------------------------------
1074−
1075−-- | Sequential parser application. Apply two parsers sequentially to an input
1076−-- stream. The input is provided to the first parser, when it is done the
1077−-- remaining input is provided to the second parser. If both the parsers
1078−-- succeed their outputs are combined using the supplied function. The
1079−-- operation fails if any of the parsers fail.
1080−--
1081−-- Note: This is a parsing dual of appending streams using
1082−-- 'Streamly.Data.Stream.append', it splits the streams using two parsers and zips
1083−-- the results.
1084−--
1085−-- This implementation is strict in the second argument, therefore, the
1086−-- following will fail:
1087−--
1088−-- >>> Stream.parse (Parser.splitWith const (Parser.satisfy (> 0)) undefined) $ Stream.fromList [1]
1089−-- *** Exception: Prelude.undefined
1090−-- ...
1091−--
1092−-- Compare with 'Applicative' instance method '<*>'. This implementation allows
1093−-- stream fusion but has quadratic complexity. This can fuse with other
1094−-- operations and can be faster than 'Applicative' instance for small number
1095−-- (less than 8) of compositions.
1096−--
1097−-- Many combinators can be expressed using @splitWith@ and other parser
1098−-- primitives. Some common idioms are described below,
1099−--
1100−-- @
1101−-- span :: (a -> Bool) -> Fold m a b -> Fold m a b -> Parser a m b
1102−-- span pred f1 f2 = splitWith (,) ('takeWhile' pred f1) ('fromFold' f2)
1103−-- @
1104−--
1105−-- @
1106−-- spanBy :: (a -> a -> Bool) -> Fold m a b -> Fold m a b -> Parser a m b
1107−-- spanBy eq f1 f2 = splitWith (,) ('groupBy' eq f1) ('fromFold' f2)
1108−-- @
1109−--
1110−-- @
1111−-- spanByRolling :: (a -> a -> Bool) -> Fold m a b -> Fold m a b -> Parser a m b
1112−-- spanByRolling eq f1 f2 = splitWith (,) ('groupByRolling' eq f1) ('fromFold' f2)
1113−-- @
1114−--
1115−-- /Pre-release/
1116−--
1117−{-# INLINE splitWith #-}
1118−splitWith :: Monad m
1119− => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
1120−splitWith f p1 p2 =
1121− D.toParserK $ D.splitWith f (D.fromParserK p1) (D.fromParserK p2)
1122−
1123−-- | Sequential parser application ignoring the output of the first parser.
1124−-- Apply two parsers sequentially to an input stream. The input is provided to
1125−-- the first parser, when it is done the remaining input is provided to the
1126−-- second parser. The output of the parser is the output of the second parser.
1127−-- The operation fails if any of the parsers fail.
1128−--
1129−-- This implementation is strict in the second argument, therefore, the
1130−-- following will fail:
1131−--
1132−-- >>> Stream.parse (Parser.split_ (Parser.satisfy (> 0)) undefined) $ Stream.fromList [1]
1133−-- *** Exception: Prelude.undefined
1134−-- ...
1135−--
1136−-- Compare with 'Applicative' instance method '*>'. This implementation allows
1137−-- stream fusion but has quadratic complexity. This can fuse with other
1138−-- operations, and can be faster than 'Applicative' instance for small
1139−-- number (less than 8) of compositions.
1140−--
1141−-- /Pre-release/
1142−--
1143−{-# INLINE split_ #-}
1144−split_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
1145−split_ p1 p2 = D.toParserK $ D.split_ (D.fromParserK p1) (D.fromParserK p2)
1146−
1147−{-
1148−-- | @teeWith f p1 p2@ distributes its input to both @p1@ and @p2@ until both
1149−-- of them succeed or anyone of them fails and combines their output using @f@.
1150−-- The parser succeeds if both the parsers succeed.
1151−--
1152−-- /Pre-release/
1153−--
1154−{-# INLINE teeWith #-}
1155−teeWith :: Monad m
1156− => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
1157−teeWith f p1 p2 =
1158− D.toParserK $ D.teeWith f (D.fromParserK p1) (D.fromParserK p2)
1159−
1160−-- | Like 'teeWith' but ends parsing and zips the results, if available,
1161−-- whenever the first parser ends.
1162−--
1163−-- /Pre-release/
1164−--
1165−{-# INLINE teeWithFst #-}
1166−teeWithFst :: Monad m
1167− => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
1168−teeWithFst f p1 p2 =
1169− D.toParserK $ D.teeWithFst f (D.fromParserK p1) (D.fromParserK p2)
1170−
1171−-- | Like 'teeWith' but ends parsing and zips the results, if available,
1172−-- whenever any of the parsers ends or fails.
1173−--
1174−-- /Unimplemented/
1175−--
1176−{-# INLINE teeWithMin #-}
1177−teeWithMin :: Monad m
1178− => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
1179−teeWithMin f p1 p2 =
1180− D.toParserK $ D.teeWithMin f (D.fromParserK p1) (D.fromParserK p2)
1181−-}
1182−
1183−-- | Sequential alternative. Apply the input to the first parser and return the
1184−-- result if the parser succeeds. If the first parser fails then backtrack and
1185−-- apply the same input to the second parser and return the result.
1186−--
1187−-- Note: This implementation is not lazy in the second argument. The following
1188−-- will fail:
1189−--
1190−-- >>> Stream.parse (Parser.satisfy (> 0) `Parser.alt` undefined) $ Stream.fromList [1..10]
1191−-- Right 1
1192−--
1193−-- Compare with 'Alternative' instance method '<|>'. This implementation allows
1194−-- stream fusion but has quadratic complexity. This can fuse with other
1195−-- operations and can be much faster than 'Alternative' instance for small
1196−-- number (less than 8) of alternatives.
1197−--
1198−-- /Pre-release/
1199−--
1200−{-# INLINE alt #-}
1201−alt :: Monad m => Parser x m a -> Parser x m a -> Parser x m a
1202−alt p1 p2 = D.toParserK $ D.alt (D.fromParserK p1) (D.fromParserK p2)
1203−
1204−{-
1205−-- | Shortest alternative. Apply both parsers in parallel but choose the result
1206−-- from the one which consumed least input i.e. take the shortest succeeding
1207−-- parse.
1208−--
1209−-- /Pre-release/
1210−--
1211−{-# INLINE shortest #-}
1212−shortest :: Monad m
1213− => Parser x m a -> Parser x m a -> Parser x m a
1214−shortest p1 p2 = D.toParserK $ D.shortest (D.fromParserK p1) (D.fromParserK p2)
1215−
1216−-- | Longest alternative. Apply both parsers in parallel but choose the result
1217−-- from the one which consumed more input i.e. take the longest succeeding
1218−-- parse.
1219−--
1220−-- /Pre-release/
1221−--
1222−{-# INLINE longest #-}
1223−longest :: Monad m
1224− => Parser x m a -> Parser x m a -> Parser x m a
1225−longest p1 p2 = D.toParserK $ D.longest (D.fromParserK p1) (D.fromParserK p2)
1226−-}
1227−
1228−-- | Run a parser without consuming the input.
1229−--
1230−{-# INLINE lookAhead #-}
1231−lookAhead :: Monad m => Parser a m b -> Parser a m b
1232−lookAhead p = D.toParserK $ D.lookAhead $ D.fromParserK p
1233−
1234−-- | Takes at-most @n@ input elements.
1235−--
1236−-- * Stops - when the collecting parser stops.
1237−-- * Fails - when the collecting parser fails.
1238−--
1239−-- >>> Stream.parse (Parser.takeP 4 (Parser.takeEQ 2 Fold.toList)) $ Stream.fromList [1, 2, 3, 4, 5]
1240−-- Right [1,2]
1241−--
1242−-- >>> Stream.parse (Parser.takeP 4 (Parser.takeEQ 5 Fold.toList)) $ Stream.fromList [1, 2, 3, 4, 5]
1243−-- Left (ParseError "takeEQ: Expecting exactly 5 elements, input terminated on 4")
1244−--
1245−-- /Internal/
1246−{-# INLINE takeP #-}
1247−takeP :: Monad m => Int -> Parser a m b -> Parser a m b
1248−takeP i p = D.toParserK $ D.takeP i $ D.fromParserK p
1249−
1250−-------------------------------------------------------------------------------
1251−-- Sequential Collection
1252−-------------------------------------------------------------------------------
1253−--
1254−-- | @concatSequence f p@ collects sequential parses of parsers in a
1255−-- serial stream @p@ using the fold @f@. Fails if the input ends or any
1256−-- of the parsers fail.
1257−--
1258−-- An even more efficient implementation can use ParserD type Parser in
1259−-- the stream.
1260−--
1261−-- /Pre-release/
1262−--
1263−{-# INLINE concatSequence #-}
1264−concatSequence ::
1265− Monad m =>
1266− Fold m b c -> Stream m (Parser a m b) -> Parser a m c
1267−concatSequence f p =
1268− let sp = fmap D.fromParserK p
1269− in D.toParserK $ D.sequence sp f
1270−
1271−-- | Map a 'Parser' returning function on the result of a 'Parser'.
1272−--
1273−-- Compare with 'Monad' instance method '>>='. This implementation allows
1274−-- stream fusion but has quadratic complexity. This can fuse with other
1275−-- operations and can be much faster than 'Monad' instance for small number
1276−-- (less than 8) of compositions.
1277−--
1278−-- /Pre-release/
1279−--
1280−{-# INLINE concatMap #-}
1281−concatMap :: Monad m
1282− => (b -> Parser a m c) -> Parser a m b -> Parser a m c
1283−concatMap f p = D.toParserK $ D.concatMap (D.fromParserK . f) (D.fromParserK p)
1284−
1285−{-
1286−-------------------------------------------------------------------------------
1287−-- Alternative Collection
1288−-------------------------------------------------------------------------------
1289−--
1290−-- | @choice parsers@ applies the @parsers@ in order and returns the first
1291−-- successful parse.
1292−--
1293−-- This is same as 'asum' but more efficient.
1294−--
1295−-- /Broken/
1296−--
1297−{-# INLINE choice #-}
1298−choice ::
1299− (Functor t, Foldable t, Monad m) => t (Parser a m b) -> Parser a m b
1300−choice ps = D.toParserK $ D.choice $ D.fromParserK <$> ps
1301−-}
1302−
1303−-------------------------------------------------------------------------------
1304−-- Sequential Repetition
1305−-------------------------------------------------------------------------------
1306−
1307−-- | Like 'many' but uses a 'Parser' instead of a 'Fold' to collect the
1308−-- results. Parsing stops or fails if the collecting parser stops or fails.
1309−--
1310−-- /Unimplemented/
1311−--
1312−{-# INLINE manyP #-}
1313−manyP :: -- Monad m =>
1314− Parser a m b -> Parser b m c -> Parser a m c
1315−manyP _p _f = undefined -- D.toParserK $ D.manyP (D.fromParserK p) f
1316−
1317−-- | Collect zero or more parses. Apply the supplied parser repeatedly on the
1318−-- input stream and push the parse results to a downstream fold.
1319−--
1320−-- Stops: when the downstream fold stops or the parser fails.
1321−-- Fails: never, produces zero or more results.
1322−--
1323−-- >>> many = Parser.countBetween 0 maxBound
1324−--
1325−-- Compare with 'Control.Applicative.many'.
1326−--
1327−{-# INLINE many #-}
1328−many :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
1329−many p f = D.toParserK $ D.many (D.fromParserK p) f
1330−
1331−-- Note: many1 would perhaps be a better name for this and consistent with
1332−-- other names like takeWhile1. But we retain the name "some" for
1333−-- compatibility.
1334−--
1335−-- | Collect one or more parses. Apply the supplied parser repeatedly on the
1336−-- input stream and push the parse results to a downstream fold.
1337−--
1338−-- Stops: when the downstream fold stops or the parser fails.
1339−-- Fails: if it stops without producing a single result.
1340−--
1341−-- >>> some p f = Parser.manyP p (Parser.takeGE 1 f)
1342−-- >>> some = Parser.countBetween 1 maxBound
1343−--
1344−-- Compare with 'Control.Applicative.some'.
1345−--
1346−{-# INLINE some #-}
1347−some :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
1348−some p f = D.toParserK $ D.some (D.fromParserK p) f
1349−-- some p f = manyP p (takeGE 1 f)
1350−-- some = countBetween 1 maxBound
1351−
1352−-- | @countBetween m n f p@ collects between @m@ and @n@ sequential parses of
1353−-- parser @p@ using the fold @f@. Stop after collecting @n@ results. Fails if
1354−-- the input ends or the parser fails before @m@ results are collected.
1355−--
1356−-- >>> countBetween m n p f = Parser.manyP p (Parser.takeBetween m n f)
1357−--
1358−-- /Unimplemented/
1359−--
1360−{-# INLINE countBetween #-}
1361−countBetween :: -- Monad m =>
1362− Int -> Int -> Parser a m b -> Fold m b c -> Parser a m c
1363−countBetween _m _n _p = undefined
1364−-- countBetween m n p f = manyP p (takeBetween m n f)
1365−
1366−-- | @count n f p@ collects exactly @n@ sequential parses of parser @p@ using
1367−-- the fold @f@. Fails if the input ends or the parser fails before @n@
1368−-- results are collected.
1369−--
1370−-- >>> count n = Parser.countBetween n n
1371−-- >>> count n p f = Parser.manyP p (Parser.takeEQ n f)
1372−--
1373−-- /Unimplemented/
1374−--
1375−{-# INLINE count #-}
1376−count :: -- Monad m =>
1377− Int -> Parser a m b -> Fold m b c -> Parser a m c
1378−count n = countBetween n n
1379−-- count n p f = manyP p (takeEQ n f)
1380−
1381−-- | Like 'manyTill' but uses a 'Parser' to collect the results instead of a
1382−-- 'Fold'. Parsing stops or fails if the collecting parser stops or fails.
1383−--
1384−-- We can implemnent parsers like the following using 'manyTillP':
1385−--
1386−-- @
1387−-- countBetweenTill m n f p = manyTillP (takeBetween m n f) p
1388−-- @
1389−--
1390−-- /Unimplemented/
1391−--
1392−{-# INLINE manyTillP #-}
1393−manyTillP :: -- Monad m =>
1394− Parser a m b -> Parser a m x -> Parser b m c -> Parser a m c
1395−manyTillP _p1 _p2 _f = undefined
1396− -- D.toParserK $ D.manyTillP (D.fromParserK p1) (D.fromParserK p2) f
1397−
1398−-- | @manyTill chunking test f@ tries the parser @test@ on the input, if @test@
1399−-- fails it backtracks and tries @chunking@, after @chunking@ succeeds @test@ is
1400−-- tried again and so on. The parser stops when @test@ succeeds. The output of
1401−-- @test@ is discarded and the output of @chunking@ is accumulated by the
1402−-- supplied fold. The parser fails if @chunking@ fails.
1403−--
1404−-- Stops when the fold @f@ stops.
1405−--
1406−{-# INLINE manyTill #-}
1407−manyTill :: Monad m
1408− => Parser a m b -- ^ Chunking parser. Parses chunks of input.
1409− -> Parser a m x -- ^ Test parser. Parsing stops when this parser succeeds
1410− -- else backtract and run the chunking parser.
1411− -> Fold m b c -- ^ Folds the output of the chunking parser.
1412− -> Parser a m c
1413−manyTill collect test f =
1414− D.toParserK $ D.manyTill (D.fromParserK collect) (D.fromParserK test) f
1415−
1416−-- | @manyThen f collect recover@ repeats the parser @collect@ on the input and
1417−-- collects the output in the supplied fold. If the the parser @collect@ fails,
1418−-- parser @recover@ is run until it stops and then we start repeating the
1419−-- parser @collect@ again. The parser fails if the recovery parser fails.
1420−--
1421−-- For example, this can be used to find a key frame in a video stream after an
1422−-- error.
1423−--
1424−-- /Unimplemented/
1425−--
1426−{-# INLINE manyThen #-}
1427−manyThen :: -- (Foldable t, Monad m) =>
1428− Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
1429−manyThen _parser _recover _f = undefined
1430−
1431−-------------------------------------------------------------------------------
1432−-- Interleaving
1433−-------------------------------------------------------------------------------
1434−--
1435−-- To deinterleave we can chain two parsers one behind the other. The input is
1436−-- given to the first parser and the input definitively rejected by the first
1437−-- parser is given to the second parser.
1438−--
1439−-- We can either have the parsers themselves buffer the input or use the shared
1440−-- global buffer to hold it until none of the parsers need it. When the first
1441−-- parser returns Skip (i.e. rewind) we let the second parser consume the
1442−-- rejected input and when it is done we move the cursor forward to the first
1443−-- parser again. This will require a "move forward" command as well.
1444−--
1445−-- To implement grep we can use three parsers, one to find the pattern, one
1446−-- to store the context behind the pattern and one to store the context in
1447−-- front of the pattern. When a match occurs we need to emit the accumulator of
1448−-- all the three parsers. One parser can count the line numbers to provide the
1449−-- line number info.
1450−
1451−-- XXX rename this to intercalate
1452−-- | Apply two parsers alternately to an input stream. The input stream is
1453−-- considered an interleaving of two patterns. The two parsers represent the
1454−-- two patterns.
1455−--
1456−{-# INLINE deintercalate #-}
1457−deintercalate :: Monad m =>
1458− Parser a m x
1459− -> Parser a m y
1460− -> Fold m (Either x y) z
1461− -> Parser a m z
1462−deintercalate contentL contentR sink =
1463− D.toParserK
1464− $ D.deintercalate
1465− (D.fromParserK contentL) (D.fromParserK contentR) sink
1466−
1467−-- | Parse items separated by a separator parsed by the supplied parser. At
1468−-- least one item must be present for the parser to succeed.
1469−--
1470−-- Note that this can go in infinite loop if both the parsers fail on some
1471−-- input. Detection of that would make the implementation more complex.
1472−--
1473−{-# INLINE sepBy1 #-}
1474−sepBy1 :: Monad m =>
1475− Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
1476−sepBy1 p sep sink = do
1477− x <- p
1478− f <- fromEffect $ FL.reduce sink
1479− f1 <- fromEffect $ FL.snoc f x
1480− many (sep >> p) f1
1481−
1482−-- | Run the content parser first, when it is done, the separator parser is
1483−-- run, when it is done content parser is run again and so on. If none of the
1484−-- parsers consumes an input then parser returns a failure.
1485−--
1486−-- >>> sepBy p1 p2 sink = Parser.deintercalate p1 p2 (Fold.catLefts sink)
1487−-- >>> sepBy content sep sink = Parser.sepBy1 content sep sink <|> return mempty
1488−--
1489−{-# INLINE sepBy #-}
1490−sepBy :: Monad m =>
1491− Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
1492−sepBy content sep sink =
1493− D.toParserK $ D.sepBy (D.fromParserK content) (D.fromParserK sep) sink
1494−-- sepBy sink = deintercalate (FL.catLefts sink)
1495−
1496−-------------------------------------------------------------------------------
1497−-- Interleaving a collection of parsers
1498−-------------------------------------------------------------------------------
1499−--
1500−-- | Apply a collection of parsers to an input stream in a round robin fashion.
1501−-- Each parser is applied until it stops and then we repeat starting with the
1502−-- the first parser again.
1503−--
1504−-- /Unimplemented/
1505−--
1506−{-# INLINE roundRobin #-}
1507−roundRobin :: -- (Foldable t, Monad m) =>
1508− t (Parser a m b) -> Fold m b c -> Parser a m c
1509−roundRobin _ps _f = undefined
1510−
1511−-------------------------------------------------------------------------------
1512−-- Repeated Alternatives
1513−-------------------------------------------------------------------------------
1514−
1515−-- | Keep trying a parser up to a maximum of @n@ failures. When the parser
1516−-- fails the input consumed till now is dropped and the new instance is tried
1517−-- on the fresh input.
1518−--
1519−-- /Unimplemented/
1520−--
1521−{-# INLINE retryMaxTotal #-}
1522−retryMaxTotal :: -- (Monad m) =>
1523− Int -> Parser a m b -> Fold m b c -> Parser a m c
1524−retryMaxTotal _n _p _f = undefined
1525−
1526−-- | Like 'retryMaxTotal' but aborts after @n@ successive failures.
1527−--
1528−-- /Unimplemented/
1529−--
1530−{-# INLINE retryMaxSuccessive #-}
1531−retryMaxSuccessive :: -- (Monad m) =>
1532− Int -> Parser a m b -> Fold m b c -> Parser a m c
1533−retryMaxSuccessive _n _p _f = undefined
1534−
1535−-- | Keep trying a parser until it succeeds. When the parser fails the input
1536−-- consumed till now is dropped and the new instance is tried on the fresh
1537−-- input.
1538−--
1539−-- /Unimplemented/
1540−--
1541−{-# INLINE retry #-}
1542−retry :: -- (Monad m) =>
1543− Parser a m b -> Parser a m b
1544−retry _p = undefined
core/src/Streamly/Internal/Data/Parser/ParserD/Tee.hs → core/src/Streamly/Internal/Data/Parser/Tee.hsrenamed+1−1
@@ -24,7 +24,7 @@
2424-- some legit warnings, therefore, we have segregated only the code that uses
2525-- uni-pattern matches in this module.
2626
27−module Streamly.Internal.Data.Parser.ParserD.Tee
27+module Streamly.Internal.Data.Parser.Tee
2828 (
2929 {-
3030 -- Parallel zipped
core/src/Streamly/Internal/Data/Parser/ParserD/Type.hs → core/src/Streamly/Internal/Data/Parser/Type.hsrenamed+94−71
@@ -173,11 +173,8 @@
173173-- * <http://hackage.haskell.org/package/attoparsec attoparsec>
174174-- * <http://hackage.haskell.org/package/parsec parsec>
175175
176−module Streamly.Internal.Data.Parser.ParserD.Type
176+module Streamly.Internal.Data.Parser.Type
177177 (
178− -- * Setup
179− -- $setup
180−
181178 -- * Types
182179 Initial (..)
183180 , Step (..)
@@ -216,7 +213,10 @@ where
216213#include "inline.hs"
217214#include "assert.hs"
218215
219−import Control.Applicative (Alternative(..), liftA2)
216+#if !MIN_VERSION_base(4,18,0)
217+import Control.Applicative (liftA2)
218+#endif
219+import Control.Applicative (Alternative(..))
220220import Control.Exception (Exception(..))
221221-- import Control.Monad (MonadPlus(..), (>=>))
222222import Control.Monad ((>=>))
@@ -438,6 +438,7 @@ newtype ParseError = ParseError String
438438instance Exception ParseError where
439439 displayException (ParseError err) = err
440440
441+-- | Map a function on the result i.e. on @b@ in @Parser a m b@.
441442instance Functor m => Functor (Parser a m) where
442443 {-# INLINE fmap #-}
443444 fmap f (Parser step1 initial1 extract) =
@@ -504,26 +505,30 @@ data SeqParseState sl f sr = SeqParseL !sl | SeqParseR !f !sr
504505-- 'Streamly.Data.Stream.append', it splits the streams using two parsers and
505506-- zips the results.
506507
507−-- | Sequential parser application.
508+-- | Sequential parser application. Apply two parsers sequentially to an input
509+-- stream. The first parser runs and processes the input, the remaining input
510+-- is then passed to the second parser. If both parsers succeed, their outputs
511+-- are combined using the supplied function. If either parser fails, the
512+-- operation fails.
508513--
509−-- Apply two parsers sequentially to an input stream. The first parser runs and
510−-- processes the input, the remaining input is then passed to the second
511−-- parser. If both parsers succeed, their outputs are combined using the
512−-- supplied function. If either parser fails, the operation fails.
514+-- This combinator delivers high performance by stream fusion but it comes with
515+-- some limitations. For those cases use the 'Applicative' instance of
516+-- 'Streamly.Data.ParserK.ParserK'.
513517--
514−-- This implementation is strict in the second argument, therefore, the
515−-- following will fail:
518+-- CAVEAT 1: NO RECURSION. This function is strict in both arguments. As a
519+-- result, if a parser is defined recursively using this, it may cause an
520+-- infintie loop. The following example checks the strictness:
516521--
517−-- >>> Stream.parse (Parser.splitWith const (Parser.satisfy (> 0)) undefined) $ Stream.fromList [1]
522+-- >>> p = Parser.splitWith const (Parser.satisfy (> 0)) undefined
523+-- >>> Stream.parse p $ Stream.fromList [1]
518524-- *** Exception: Prelude.undefined
519525-- ...
520526--
521−-- Although this implementation allows stream fusion, it has quadratic
522−-- complexity, making it suitable only for a small number of compositions.
523−-- As a thumb rule use it for less than 8 compositions, use ParserK otherwise.
527+-- CAVEAT 2: QUADRATIC TIME COMPLEXITY. Static composition is fast due to
528+-- stream fusion, but it works well only for limited (e.g. up to 8)
529+-- compositions, use "Streamly.Data.ParserK" for larger compositions.
524530--
525−-- Below are some common idioms that can be expressed using 'splitWith' and
526−-- other parser primitives:
531+-- Below are some common idioms that can be expressed using 'splitWith':
527532--
528533-- >>> span p f1 f2 = Parser.splitWith (,) (Parser.takeWhile p f1) (Parser.fromFold f2)
529534-- >>> spanBy eq f1 f2 = Parser.splitWith (,) (Parser.groupBy eq f1) (Parser.fromFold f2)
@@ -612,7 +617,12 @@ splitWith func (Parser stepL initialL extractL)
612617-- We can use the compiler to automatically annotate accumulators, terminating
613618-- folds, non-failing parsers and failing parsers.
614619
615−-- | Works correctly only if both the parsers are guaranteed to never fail.
620+-- | Better performance 'splitWith' for non-failing parsers.
621+--
622+-- Does not work correctly for parsers that can fail.
623+--
624+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
625+--
616626{-# INLINE noErrorUnsafeSplitWith #-}
617627noErrorUnsafeSplitWith :: Monad m
618628 => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
@@ -683,6 +693,8 @@ data SeqAState sl sr = SeqAL !sl | SeqAR !sr
683693-- second parser. The output of the parser is the output of the second parser.
684694-- The operation fails if any of the parsers fail.
685695--
696+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
697+--
686698-- This implementation is strict in the second argument, therefore, the
687699-- following will fail:
688700--
@@ -690,10 +702,6 @@ data SeqAState sl sr = SeqAL !sl | SeqAR !sr
690702-- *** Exception: Prelude.undefined
691703-- ...
692704--
693−-- Although this implementation allows stream fusion, it has quadratic
694−-- complexity, making it suitable only for a small number of compositions.
695−-- As a thumb rule use it for less than 8 compositions, use ParserK otherwise.
696−--
697705-- /Pre-release/
698706--
699707{-# INLINE split_ #-}
@@ -749,7 +757,12 @@ split_ (Parser stepL initialL extractL) (Parser stepR initialR extractR) =
749757 Partial _ _ -> error "split_: Partial"
750758 Continue n s -> return $ Continue n (SeqAL s)
751759
752−-- For backtracking folds
760+-- | Better performance 'split_' for non-failing parsers.
761+--
762+-- Does not work correctly for parsers that can fail.
763+--
764+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
765+--
753766{-# INLINE noErrorUnsafeSplit_ #-}
754767noErrorUnsafeSplit_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
755768noErrorUnsafeSplit_
@@ -804,7 +817,11 @@ noErrorUnsafeSplit_
804817 Partial _ _ -> error "split_: Partial"
805818 Continue n s -> return $ Continue n (SeqAL s)
806819
807−-- | 'Applicative' form of 'splitWith'.
820+-- | READ THE CAVEATS in 'splitWith' before using this instance.
821+--
822+-- >>> pure = Parser.fromPure
823+-- >>> (<*>) = Parser.splitWith id
824+-- >>> (*>) = Parser.split_
808825instance Monad m => Applicative (Parser a m) where
809826 {-# INLINE pure #-}
810827 pure = fromPure
@@ -830,20 +847,26 @@ data AltParseState sl sr = AltParseL !Int !sl | AltParseR !sr
830847-- each subsequent alternative's input element has to go through, therefore, it
831848-- cannot scale to a large number of compositions
832849
833−-- | Sequential alternative. The input is first passed to the first parser, and
850+-- | Sequential alternative. The input is first passed to the first parser,
834851-- if it succeeds, the result is returned. However, if the first parser fails,
835−-- the parser driver backtracks and tries the same input on the second parser,
836−-- returning the result if it succeeds.
852+-- the parser driver backtracks and tries the same input on the second
853+-- (alternative) parser, returning the result if it succeeds.
854+--
855+-- This combinator delivers high performance by stream fusion but it comes with
856+-- some limitations. For those cases use the 'Alternative' instance of
857+-- 'Streamly.Data.ParserK.ParserK'.
837858--
838−-- Note: This implementation is not lazy in the second argument. The following
839−-- will fail:
859+-- CAVEAT 1: NO RECURSION. This function is strict in both arguments. As a
860+-- result, if a parser is defined recursively using this, it may cause an
861+-- infintie loop. The following example checks the strictness:
840862--
841−-- >> Stream.parse (Parser.satisfy (> 0) `Parser.alt` undefined) $ Stream.fromList [1..10]
863+-- >>> p = Parser.satisfy (> 0) `Parser.alt` undefined
864+-- >>> Stream.parse p $ Stream.fromList [1..10]
842865-- *** Exception: Prelude.undefined
843866--
844−-- Although this implementation allows stream fusion, it has quadratic
845−-- complexity, making it suitable only for a small number of compositions.
846−-- As a thumb rule use it for less than 8 compositions, use ParserK otherwise.
867+-- CAVEAT 2: QUADRATIC TIME COMPLEXITY. Static composition is fast due to
868+-- stream fusion, but it works well only for limited (e.g. up to 8)
869+-- compositions, use "Streamly.Data.ParserK" for larger compositions.
847870--
848871-- /Time Complexity:/ O(n^2) where n is the number of compositions.
849872--
@@ -924,7 +947,7 @@ data Fused3 a b c = Fused3 !a !b !c
924947--
925948{-# INLINE splitMany #-}
926949splitMany :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
927−splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
950+splitMany (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
928951 Parser step initial extract
929952
930953 where
@@ -940,7 +963,7 @@ splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
940963 IPartial ps -> return $ partial $ Fused3 ps 0 fs
941964 IDone pb ->
942965 runCollectorWith (handleCollect partial done) fs pb
943− IError _ -> done <$> fextract fs
966+ IError _ -> done <$> ffinal fs
944967 FL.Done fb -> return $ done fb
945968
946969 runCollectorWith cont fs pb = fstep fs pb >>= cont
@@ -964,19 +987,19 @@ splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
964987 assertM(cnt1 - n >= 0)
965988 fstep fs b >>= handleCollect (Partial n) (Done n)
966989 Error _ -> do
967− xs <- fextract fs
990+ xs <- ffinal fs
968991 return $ Done cnt xs
969992
970− extract (Fused3 _ 0 fs) = fmap (Done 0) (fextract fs)
993+ extract (Fused3 _ 0 fs) = fmap (Done 0) (ffinal fs)
971994 extract (Fused3 s cnt fs) = do
972995 r <- extract1 s
973996 case r of
974− Error _ -> fmap (Done cnt) (fextract fs)
997+ Error _ -> fmap (Done cnt) (ffinal fs)
975998 Done n b -> do
976999 assertM(n <= cnt)
9771000 fs1 <- fstep fs b
9781001 case fs1 of
979− FL.Partial s1 -> fmap (Done n) (fextract s1)
1002+ FL.Partial s1 -> fmap (Done n) (ffinal s1)
9801003 FL.Done b1 -> return (Done n b1)
9811004 Partial _ _ -> error "splitMany: Partial in extract"
9821005 Continue n s1 -> do
@@ -990,7 +1013,7 @@ splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
9901013--
9911014{-# INLINE splitManyPost #-}
9921015splitManyPost :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
993−splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
1016+splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
9941017 Parser step initial extract
9951018
9961019 where
@@ -1006,7 +1029,7 @@ splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10061029 IPartial ps -> return $ partial $ Fused3 ps 0 fs
10071030 IDone pb ->
10081031 runCollectorWith (handleCollect partial done) fs pb
1009− IError _ -> done <$> fextract fs
1032+ IError _ -> done <$> ffinal fs
10101033 FL.Done fb -> return $ done fb
10111034
10121035 runCollectorWith cont fs pb = fstep fs pb >>= cont
@@ -1028,18 +1051,18 @@ splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10281051 assertM(cnt1 - n >= 0)
10291052 fstep fs b >>= handleCollect (Partial n) (Done n)
10301053 Error _ -> do
1031− xs <- fextract fs
1054+ xs <- ffinal fs
10321055 return $ Done cnt1 xs
10331056
10341057 extract (Fused3 s cnt fs) = do
10351058 r <- extract1 s
10361059 case r of
1037− Error _ -> fmap (Done cnt) (fextract fs)
1060+ Error _ -> fmap (Done cnt) (ffinal fs)
10381061 Done n b -> do
10391062 assertM(n <= cnt)
10401063 fs1 <- fstep fs b
10411064 case fs1 of
1042− FL.Partial s1 -> fmap (Done n) (fextract s1)
1065+ FL.Partial s1 -> fmap (Done n) (ffinal s1)
10431066 FL.Done b1 -> return (Done n b1)
10441067 Partial _ _ -> error "splitMany: Partial in extract"
10451068 Continue n s1 -> do
@@ -1052,7 +1075,7 @@ splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10521075--
10531076{-# INLINE splitSome #-}
10541077splitSome :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
1055−splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
1078+splitSome (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
10561079 Parser step initial extract
10571080
10581081 where
@@ -1068,7 +1091,7 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10681091 IPartial ps -> return $ partial $ Fused3 ps 0 $ Right fs
10691092 IDone pb ->
10701093 runCollectorWith (handleCollect partial done) fs pb
1071− IError _ -> done <$> fextract fs
1094+ IError _ -> done <$> ffinal fs
10721095 FL.Done fb -> return $ done fb
10731096
10741097 runCollectorWith cont fs pb = fstep fs pb >>= cont
@@ -1118,7 +1141,7 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
11181141 Done n b -> do
11191142 assertM(cnt1 - n >= 0)
11201143 fstep fs b >>= handleCollect (Partial n) (Done n)
1121− Error _ -> Done cnt1 <$> fextract fs
1144+ Error _ -> Done cnt1 <$> ffinal fs
11221145
11231146 extract (Fused3 s cnt (Left fs)) = do
11241147 r <- extract1 s
@@ -1128,7 +1151,7 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
11281151 assertM(n <= cnt)
11291152 fs1 <- fstep fs b
11301153 case fs1 of
1131− FL.Partial s1 -> fmap (Done n) (fextract s1)
1154+ FL.Partial s1 -> fmap (Done n) (ffinal s1)
11321155 FL.Done b1 -> return (Done n b1)
11331156 Partial _ _ -> error "splitSome: Partial in extract"
11341157 Continue n s1 -> do
@@ -1137,12 +1160,12 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
11371160 extract (Fused3 s cnt (Right fs)) = do
11381161 r <- extract1 s
11391162 case r of
1140− Error _ -> fmap (Done cnt) (fextract fs)
1163+ Error _ -> fmap (Done cnt) (ffinal fs)
11411164 Done n b -> do
11421165 assertM(n <= cnt)
11431166 fs1 <- fstep fs b
11441167 case fs1 of
1145− FL.Partial s1 -> fmap (Done n) (fextract s1)
1168+ FL.Partial s1 -> fmap (Done n) (ffinal s1)
11461169 FL.Done b1 -> return (Done n b1)
11471170 Partial _ _ -> error "splitSome: Partial in extract"
11481171 Continue n s1 -> do
@@ -1173,23 +1196,12 @@ dieM err = Parser undefined (IError <$> err) undefined
11731196-- Note: With the direct style parser type, the list in "some" and "many" is
11741197-- accumulated strictly, it cannot be consumed lazily.
11751198
1176−-- | Sequential alternative. The input is first passed to the first parser, and
1177−-- if it succeeds, the result is returned. However, if the first parser fails,
1178−-- the parser driver backtracks and tries the same input on the second parser,
1179−-- returning the result if it succeeds.
1180−--
1181−-- Note: The implementation of '<|>' is not lazy in the second
1182−-- argument. The following code will fail:
1183−--
1184−-- >>> Stream.parse (Parser.satisfy (> 0) <|> undefined) $ Stream.fromList [1..10]
1185−-- *** Exception: Prelude.undefined
1186−-- ...
1199+-- | READ THE CAVEATS in 'alt' before using this instance.
11871200--
1188−-- WARNING! this is not suitable for large scale use. As a thumb rule stream
1189−-- fusion works well for less than 8 compositions of this operation, otherwise
1190−-- consider using 'ParserK'. Do not use recursive parser implementations based
1191−-- on this Alternative instance.
1192−
1201+-- >>> empty = Parser.die "empty"
1202+-- >>> (<|>) = Parser.alt
1203+-- >>> many = flip Parser.many Fold.toList
1204+-- >>> some = flip Parser.some Fold.toList
11931205instance Monad m => Alternative (Parser a m) where
11941206 {-# INLINE empty #-}
11951207 empty = die "empty"
@@ -1208,8 +1220,13 @@ data ConcatParseState sl m a b =
12081220 ConcatParseL !sl
12091221 | forall s. ConcatParseR (s -> a -> m (Step s b)) s (s -> m (Step s b))
12101222
1223+-- XXX Does it fuse completely? Check and update, it cannot fuse the
1224+-- dynamically generated parser.
1225+
12111226-- | Map a 'Parser' returning function on the result of a 'Parser'.
12121227--
1228+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
1229+--
12131230-- /Pre-release/
12141231--
12151232{-# INLINE concatMap #-}
@@ -1279,6 +1296,12 @@ concatMap func (Parser stepL initialL extractL) = Parser step initial extract
12791296 Partial _ _ -> error "concatMap: extract Partial"
12801297 Continue n s -> return $ Continue n (ConcatParseL s)
12811298
1299+-- | Better performance 'concatMap' for non-failing parsers.
1300+--
1301+-- Does not work correctly for parsers that can fail.
1302+--
1303+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
1304+--
12821305{-# INLINE noErrorUnsafeConcatMap #-}
12831306noErrorUnsafeConcatMap :: Monad m =>
12841307 (b -> Parser a m c) -> Parser a m b -> Parser a m c
@@ -1351,11 +1374,9 @@ noErrorUnsafeConcatMap func (Parser stepL initialL extractL) =
13511374-- for small number of compositions but for a scalable implementation we need a
13521375-- CPS version.
13531376
1354−-- | See documentation of 'Streamly.Internal.Data.Parser.ParserK.Type.Parser'.
1377+-- | READ THE CAVEATS in 'concatMap' before using this instance.
13551378--
1356−-- Although this implementation allows stream fusion, it has quadratic
1357−-- complexity, making it suitable only for a small number of compositions. As a
1358−-- thumb rule use it for less than 8 compositions, use 'ParserK' otherwise.
1379+-- >>> (>>=) = flip Parser.concatMap
13591380--
13601381instance Monad m => Monad (Parser a m) where
13611382 {-# INLINE return #-}
@@ -1367,6 +1388,7 @@ instance Monad m => Monad (Parser a m) where
13671388 {-# INLINE (>>) #-}
13681389 (>>) = (*>)
13691390
1391+-- | >>> fail = Parser.die
13701392instance Monad m => Fail.MonadFail (Parser a m) where
13711393 {-# INLINE fail #-}
13721394 fail = die
@@ -1382,6 +1404,7 @@ instance Monad m => MonadPlus (Parser a m) where
13821404 mplus = alt
13831405-}
13841406
1407+-- | >>> liftIO = Parser.fromEffect . liftIO
13851408instance (MonadIO m) => MonadIO (Parser a m) where
13861409 {-# INLINE liftIO #-}
13871410 liftIO = fromEffect . liftIO
core/src/Streamly/Internal/Data/ParserK.hsnew file+16
@@ -0,0 +1,16 @@
1+{-# LANGUAGE CPP #-}
2+-- |
3+-- Module : Streamly.Internal.Data.ParserK
4+-- Copyright : (c) 2020 Composewell Technologies
5+-- License : BSD-3-Clause
6+-- Maintainer : streamly@composewell.com
7+-- Stability : experimental
8+-- Portability : GHC
9+
10+module Streamly.Internal.Data.ParserK
11+ (
12+ module Streamly.Internal.Data.ParserK.Type
13+ )
14+where
15+
16+import Streamly.Internal.Data.ParserK.Type
core/src/Streamly/Internal/Data/Parser/ParserK/Type.hs → core/src/Streamly/Internal/Data/ParserK/Type.hsrenamed+288−83
@@ -21,13 +21,15 @@
2121-- <http://hackage.haskell.org/package/parser-combinators parser-combinators>
2222-- package can also be used with the 'ParserK' type.
2323
24−module Streamly.Internal.Data.Parser.ParserK.Type
24+module Streamly.Internal.Data.ParserK.Type
2525 (
2626 Step (..)
2727 , Input (..)
2828 , ParseResult (..)
2929 , ParserK (..)
30− , fromParser
30+ , adaptC
31+ , adapt
32+ , adaptCG
3133 -- , toParser
3234 , fromPure
3335 , fromEffect
@@ -39,7 +41,10 @@ where
3941#include "assert.hs"
4042#include "inline.hs"
4143
42−import Control.Applicative (Alternative(..), liftA2)
44+#if !MIN_VERSION_base(4,18,0)
45+import Control.Applicative (liftA2)
46+#endif
47+import Control.Applicative (Alternative(..))
4348import Control.Monad (MonadPlus(..), ap)
4449import Control.Monad.IO.Class (MonadIO, liftIO)
4550-- import Control.Monad.Trans.Class (MonadTrans(lift))
@@ -51,12 +56,24 @@ import Streamly.Internal.System.IO (unsafeInlineIO)
5156
5257import qualified Control.Monad.Fail as Fail
5358import qualified Streamly.Internal.Data.Array.Type as Array
54−import qualified Streamly.Internal.Data.Parser.ParserD.Type as ParserD
59+import qualified Streamly.Internal.Data.MutArray.Generic as GenArr
60+ ( getIndexUnsafeWith
61+ )
62+import qualified Streamly.Internal.Data.Array.Generic as GenArr
63+import qualified Streamly.Internal.Data.Parser.Type as ParserD
5564
5665-- Note: We cannot use an Array directly as input because we need to identify
5766-- the end of input case using None. We cannot do that using nil Array as nil
5867-- Arrays can be encountered in normal input as well.
59−data Input a = None | Chunk {-# UNPACK #-} !(Array a)
68+--
69+-- We could specialize the ParserK type to use an Array directly, that provides
70+-- some performance improvement. The best advantage of that is when we consume
71+-- one element at a time from the array. If we really want that perf
72+-- improvement we can use a special ParserK type with the following Input.
73+--
74+-- data Input a = None | Chunk {-# UNPACK #-} !(Array a)
75+--
76+data Input a = None | Chunk a
6077
6178-- | The intermediate result of running a parser step. The parser driver may
6279-- stop with a final result, pause with a continuation to resume, or fail with
@@ -72,9 +89,6 @@ data Step a m r =
7289 -- The Int is the current stream position index wrt to the start of the
7390 -- array.
7491 Done !Int r
75− -- XXX we can use a "resume" and a "stop" continuations instead of Maybe.
76− -- measure if that works any better.
77− -- Array a -> m (Step a m r), m (Step a m r)
7892 | Partial !Int (Input a -> m (Step a m r))
7993 | Continue !Int (Input a -> m (Step a m r))
8094 | Error !Int String
@@ -154,8 +168,8 @@ newtype ParserK a m b = MkParser
154168-------------------------------------------------------------------------------
155169
156170-- XXX rewrite this using ParserD, expose rmapM from ParserD.
157−-- | Maps a function over the output of the parser.
158−--
171+
172+-- | Map a function on the result i.e. on @b@ in @Parser a m b@.
159173instance Functor m => Functor (ParserK a m) where
160174 {-# INLINE fmap #-}
161175 fmap f parser = MkParser $ \k n st arr ->
@@ -167,7 +181,7 @@ instance Functor m => Functor (ParserK a m) where
167181-------------------------------------------------------------------------------
168182
169183-- This is the dual of stream "fromPure".
170−--
184+
171185-- | A parser that always yields a pure value without consuming any input.
172186--
173187-- /Pre-release/
@@ -185,9 +199,11 @@ fromEffect :: Monad m => m b -> ParserK a m b
185199fromEffect eff =
186200 MkParser $ \k n st arr -> eff >>= \b -> k (Success n b) st arr
187201
188−-- | 'Applicative' form of 'Streamly.Internal.Data.Parser.splitWith'. Note that
189−-- this operation does not fuse, use 'Streamly.Internal.Data.Parser.splitWith'
190−-- when fusion is important.
202+-- | @f \<$> p1 \<*> p2@ applies parsers p1 and p2 sequentially to an input
203+-- stream. The first parser runs and processes the input, the remaining input
204+-- is then passed to the second parser. If both parsers succeed, their outputs
205+-- are applied to the function @f@. If either parser fails, the operation
206+-- fails.
191207--
192208instance Monad m => Applicative (ParserK a m) where
193209 {-# INLINE pure #-}
@@ -229,48 +245,8 @@ instance Monad m => Applicative (ParserK a m) where
229245die :: String -> ParserK a m b
230246die err = MkParser (\k n st arr -> k (Failure n err) st arr)
231247
232−-- | Monad composition can be used for lookbehind parsers, we can make the
233−-- future parses depend on the previously parsed values.
234−--
235−-- If we have to parse "a9" or "9a" but not "99" or "aa" we can use the
236−-- following parser:
237−--
238−-- @
239−-- backtracking :: MonadCatch m => PR.Parser Char m String
240−-- backtracking =
241−-- sequence [PR.satisfy isDigit, PR.satisfy isAlpha]
242−-- '<|>'
243−-- sequence [PR.satisfy isAlpha, PR.satisfy isDigit]
244−-- @
245−--
246−-- We know that if the first parse resulted in a digit at the first place then
247−-- the second parse is going to fail. However, we waste that information and
248−-- parse the first character again in the second parse only to know that it is
249−-- not an alphabetic char. By using lookbehind in a 'Monad' composition we can
250−-- avoid redundant work:
251−--
252−-- @
253−-- data DigitOrAlpha = Digit Char | Alpha Char
254−--
255−-- lookbehind :: MonadCatch m => PR.Parser Char m String
256−-- lookbehind = do
257−-- x1 \<- Digit '<$>' PR.satisfy isDigit
258−-- '<|>' Alpha '<$>' PR.satisfy isAlpha
259−--
260−-- -- Note: the parse depends on what we parsed already
261−-- x2 <- case x1 of
262−-- Digit _ -> PR.satisfy isAlpha
263−-- Alpha _ -> PR.satisfy isDigit
264−--
265−-- return $ case x1 of
266−-- Digit x -> [x,x2]
267−-- Alpha x -> [x,x2]
268−-- @
269−--
270−-- See also 'Streamly.Internal.Data.Parser.concatMap'. This monad instance
271−-- does not fuse, use 'Streamly.Internal.Data.Parser.concatMap' when you need
272−-- fusion.
273−--
248+-- | Monad composition can be used for lookbehind parsers, we can dynamically
249+-- compose new parsers based on the results of the previously parsed values.
274250instance Monad m => Monad (ParserK a m) where
275251 {-# INLINE return #-}
276252 return = pure
@@ -302,18 +278,10 @@ instance MonadIO m => MonadIO (ParserK a m) where
302278-- Alternative
303279-------------------------------------------------------------------------------
304280
305−-- | 'Alternative' form of 'Streamly.Internal.Data.Parser.alt'. Backtrack and
306−-- run the second parser if the first one fails.
307−--
308−-- The "some" and "many" operations of alternative accumulate results in a pure
309−-- list which is not scalable and streaming. Instead use
310−-- 'Streamly.Internal.Data.Parser.some' and
311−-- 'Streamly.Internal.Data.Parser.many' for fusible operations with composable
312−-- accumulation of results.
313−--
314−-- See also 'Streamly.Internal.Data.Parser.alt'. This 'Alternative' instance
315−-- does not fuse, use 'Streamly.Internal.Data.Parser.alt' when you need
316−-- fusion.
281+-- | @p1 \<|> p2@ passes the input to parser p1, if it succeeds, the result is
282+-- returned. However, if p1 fails, the parser driver backtracks and tries the
283+-- same input on the alternative parser p2, returning the result if it
284+-- succeeds.
317285--
318286instance Monad m => Alternative (ParserK a m) where
319287 {-# INLINE empty #-}
@@ -365,18 +333,22 @@ instance MonadTrans (ParserK a) where
365333-- Convert ParserD to ParserK
366334-------------------------------------------------------------------------------
367335
368−{-# INLINE parseDToK #-}
369−parseDToK
336+--------------------------------------------------------------------------------
337+-- Chunked
338+--------------------------------------------------------------------------------
339+
340+{-# INLINE adaptCWith #-}
341+adaptCWith
370342 :: forall m a s b r. (Monad m, Unbox a)
371343 => (s -> a -> m (ParserD.Step s b))
372344 -> m (ParserD.Initial s b)
373345 -> (s -> m (ParserD.Step s b))
374− -> (ParseResult b -> Int -> Input a -> m (Step a m r))
346+ -> (ParseResult b -> Int -> Input (Array a) -> m (Step (Array a) m r))
375347 -> Int
376348 -> Int
377− -> Input a
378− -> m (Step a m r)
379−parseDToK pstep initial extract cont !offset0 !usedCount !input = do
349+ -> Input (Array a)
350+ -> m (Step (Array a) m r)
351+adaptCWith pstep initial extract cont !offset0 !usedCount !input = do
380352 res <- initial
381353 case res of
382354 ParserD.IPartial pst -> do
@@ -432,7 +404,7 @@ parseDToK pstep initial extract cont !offset0 !usedCount !input = do
432404 go !_ !cur !pst | cur >= end =
433405 onContinue ((end - start) `div` SIZE_OF(a)) pst
434406 go !_ !cur !pst = do
435− let !x = unsafeInlineIO $ peekByteIndex cur contents
407+ let !x = unsafeInlineIO $ peekAt cur contents
436408 pRes <- pstep pst x
437409 let elemSize = SIZE_OF(a)
438410 next = INDEX_NEXT(cur,a)
@@ -480,23 +452,258 @@ parseDToK pstep initial extract cont !offset0 !usedCount !input = do
480452 -- XXX It is called only when there is no input arr. So using 0
481453 -- as the position is correct?
482454 cont (Failure 0 err) count None
483− ParserD.Partial _ _ -> error "Bug: parseDToK Partial unreachable"
455+ ParserD.Partial _ _ -> error "Bug: adaptCWith Partial unreachable"
456+
457+ -- XXX Maybe we can use two separate continuations instead of using
458+ -- Just/Nothing cases here. That may help in avoiding the parseContJust
459+ -- function call.
460+ {-# INLINE parseCont #-}
461+ parseCont !cnt !pst (Chunk arr) = parseContChunk cnt 0 pst arr
462+ parseCont !cnt !pst None = parseContNothing cnt pst
463+
464+-- | Convert an element 'Parser' to a chunked 'ParserK'. A chunked parser is
465+-- more efficient than an element parser.
466+--
467+-- /Pre-release/
468+--
469+{-# INLINE_LATE adaptC #-}
470+adaptC :: (Monad m, Unbox a) => ParserD.Parser a m b -> ParserK (Array a) m b
471+adaptC (ParserD.Parser step initial extract) =
472+ MkParser $ adaptCWith step initial extract
473+
474+--------------------------------------------------------------------------------
475+-- Singular
476+--------------------------------------------------------------------------------
477+
478+{-# INLINE adaptWith #-}
479+adaptWith
480+ :: forall m a s b r. (Monad m)
481+ => (s -> a -> m (ParserD.Step s b))
482+ -> m (ParserD.Initial s b)
483+ -> (s -> m (ParserD.Step s b))
484+ -> (ParseResult b -> Int -> Input a -> m (Step a m r))
485+ -> Int
486+ -> Int
487+ -> Input a
488+ -> m (Step a m r)
489+adaptWith pstep initial extract cont !relPos !usedCount !input = do
490+ res <- initial
491+ case res of
492+ ParserD.IPartial pst -> do
493+ -- XXX can we come here with relPos 1?
494+ if relPos == 0
495+ then
496+ case input of
497+ Chunk arr -> parseContChunk usedCount pst arr
498+ None -> parseContNothing usedCount pst
499+ -- XXX Previous code was using Continue in this case
500+ else pure $ Partial relPos (parseCont usedCount pst)
501+ ParserD.IDone b -> cont (Success relPos b) usedCount input
502+ ParserD.IError err -> cont (Failure relPos err) usedCount input
503+
504+ where
505+
506+ -- XXX We can maintain an absolute position instead of relative that will
507+ -- help in reporting of error location in the stream.
508+ {-# NOINLINE parseContChunk #-}
509+ parseContChunk !count !state x = do
510+ go SPEC state
511+
512+ where
513+
514+ go !_ !pst = do
515+ pRes <- pstep pst x
516+ case pRes of
517+ ParserD.Done 0 b ->
518+ cont (Success 1 b) (count + 1) (Chunk x)
519+ ParserD.Done 1 b ->
520+ cont (Success 0 b) count (Chunk x)
521+ ParserD.Done n b ->
522+ cont (Success (1 - n) b) (count + 1 - n) (Chunk x)
523+ ParserD.Partial 0 pst1 ->
524+ pure $ Partial 1 (parseCont (count + 1) pst1)
525+ ParserD.Partial 1 pst1 ->
526+ -- XXX Since we got Partial, the driver should drop the
527+ -- buffer, we should call the driver here?
528+ go SPEC pst1
529+ ParserD.Partial n pst1 ->
530+ pure $ Partial (1 - n) (parseCont (count + 1 - n) pst1)
531+ ParserD.Continue 0 pst1 ->
532+ pure $ Continue 1 (parseCont (count + 1) pst1)
533+ ParserD.Continue 1 pst1 ->
534+ go SPEC pst1
535+ ParserD.Continue n pst1 ->
536+ pure $ Continue (1 - n) (parseCont (count + 1 - n) pst1)
537+ ParserD.Error err ->
538+ -- XXX fix undefined
539+ cont (Failure 0 err) count (Chunk x)
540+
541+ {-# NOINLINE parseContNothing #-}
542+ parseContNothing !count !pst = do
543+ r <- extract pst
544+ case r of
545+ -- IMPORTANT: the n here is from the byte stream parser, that means
546+ -- it is the backtrack element count and not the stream position
547+ -- index into the current input array.
548+ ParserD.Done n b ->
549+ assert (n >= 0)
550+ (cont (Success (- n) b) (count - n) None)
551+ ParserD.Continue n pst1 ->
552+ assert (n >= 0)
553+ (return $ Continue (- n) (parseCont (count - n) pst1))
554+ ParserD.Error err ->
555+ -- XXX It is called only when there is no input arr. So using 0
556+ -- as the position is correct?
557+ cont (Failure 0 err) count None
558+ ParserD.Partial _ _ -> error "Bug: adaptCWith Partial unreachable"
484559
485560 -- XXX Maybe we can use two separate continuations instead of using
486561 -- Just/Nothing cases here. That may help in avoiding the parseContJust
487562 -- function call.
563+ {-# INLINE parseCont #-}
564+ parseCont !cnt !pst (Chunk arr) = parseContChunk cnt pst arr
565+ parseCont !cnt !pst None = parseContNothing cnt pst
566+
567+-- | Convert a 'Parser' to 'ParserK'.
568+--
569+-- /Pre-release/
570+--
571+{-# INLINE_LATE adapt #-}
572+adapt :: Monad m => ParserD.Parser a m b -> ParserK a m b
573+adapt (ParserD.Parser step initial extract) =
574+ MkParser $ adaptWith step initial extract
575+
576+--------------------------------------------------------------------------------
577+-- Chunked Generic
578+--------------------------------------------------------------------------------
579+
580+{-# INLINE adaptCGWith #-}
581+adaptCGWith
582+ :: forall m a s b r. (Monad m)
583+ => (s -> a -> m (ParserD.Step s b))
584+ -> m (ParserD.Initial s b)
585+ -> (s -> m (ParserD.Step s b))
586+ -> (ParseResult b -> Int -> Input (GenArr.Array a) -> m (Step (GenArr.Array a) m r))
587+ -> Int
588+ -> Int
589+ -> Input (GenArr.Array a)
590+ -> m (Step (GenArr.Array a) m r)
591+adaptCGWith pstep initial extract cont !offset0 !usedCount !input = do
592+ res <- initial
593+ case res of
594+ ParserD.IPartial pst -> do
595+ case input of
596+ Chunk arr -> parseContChunk usedCount offset0 pst arr
597+ None -> parseContNothing usedCount pst
598+ ParserD.IDone b -> cont (Success offset0 b) usedCount input
599+ ParserD.IError err -> cont (Failure offset0 err) usedCount input
600+
601+ where
602+
603+ {-# NOINLINE parseContChunk #-}
604+ parseContChunk !count !offset !state arr@(GenArr.Array contents start len) = do
605+ if offset >= 0
606+ then go SPEC (start + offset) state
607+ else return $ Continue offset (parseCont count state)
608+
609+ where
610+
611+ {-# INLINE end #-}
612+ end = start + len
613+
614+ {-# INLINE onDone #-}
615+ onDone n b =
616+ assert (n <= GenArr.length arr)
617+ (cont (Success n b) (count + n - offset) (Chunk arr))
618+
619+ {-# INLINE callParseCont #-}
620+ callParseCont constr n pst1 =
621+ assert (n < 0 || n >= GenArr.length arr)
622+ (return $ constr n (parseCont (count + n - offset) pst1))
623+
624+ {-# INLINE onPartial #-}
625+ onPartial = callParseCont Partial
626+
627+ {-# INLINE onContinue #-}
628+ onContinue = callParseCont Continue
629+
630+ {-# INLINE onError #-}
631+ onError n err =
632+ cont (Failure n err) (count + n - offset) (Chunk arr)
633+
634+ {-# INLINE onBack #-}
635+ onBack offset1 constr pst = do
636+ let pos = offset1 - start
637+ in if pos >= 0
638+ then go SPEC offset1 pst
639+ else constr pos pst
640+
641+ go !_ !cur !pst | cur >= end =
642+ onContinue len pst
643+ go !_ !cur !pst = do
644+ let !x = unsafeInlineIO $ GenArr.getIndexUnsafeWith contents cur
645+ pRes <- pstep pst x
646+ let next = cur + 1
647+ back n = next - n
648+ curOff = cur - start
649+ nextOff = next - start
650+ -- The "n" here is stream position index wrt the array start, and
651+ -- not the backtrack count as returned by byte stream parsers.
652+ case pRes of
653+ ParserD.Done 0 b ->
654+ onDone nextOff b
655+ ParserD.Done 1 b ->
656+ onDone curOff b
657+ ParserD.Done n b ->
658+ onDone (back n - start) b
659+ ParserD.Partial 0 pst1 ->
660+ go SPEC next pst1
661+ ParserD.Partial 1 pst1 ->
662+ go SPEC cur pst1
663+ ParserD.Partial n pst1 ->
664+ onBack (back n) onPartial pst1
665+ ParserD.Continue 0 pst1 ->
666+ go SPEC next pst1
667+ ParserD.Continue 1 pst1 ->
668+ go SPEC cur pst1
669+ ParserD.Continue n pst1 ->
670+ onBack (back n) onContinue pst1
671+ ParserD.Error err ->
672+ onError curOff err
673+
674+ {-# NOINLINE parseContNothing #-}
675+ parseContNothing !count !pst = do
676+ r <- extract pst
677+ case r of
678+ -- IMPORTANT: the n here is from the byte stream parser, that means
679+ -- it is the backtrack element count and not the stream position
680+ -- index into the current input array.
681+ ParserD.Done n b ->
682+ assert (n >= 0)
683+ (cont (Success (- n) b) (count - n) None)
684+ ParserD.Continue n pst1 ->
685+ assert (n >= 0)
686+ (return $ Continue (- n) (parseCont (count - n) pst1))
687+ ParserD.Error err ->
688+ -- XXX It is called only when there is no input arr. So using 0
689+ -- as the position is correct?
690+ cont (Failure 0 err) count None
691+ ParserD.Partial _ _ -> error "Bug: adaptCGWith Partial unreachable"
692+
488693 {-# INLINE parseCont #-}
489694 parseCont !cnt !pst (Chunk arr) = parseContChunk cnt 0 pst arr
490695 parseCont !cnt !pst None = parseContNothing cnt pst
491696
492−-- | Convert a raw byte 'Parser' to a chunked 'ParserK'.
697+-- | A generic 'adaptC'. Similar to 'adaptC' but is not constrained to 'Unbox'
698+-- types.
493699--
494700-- /Pre-release/
495701--
496−{-# INLINE_LATE fromParser #-}
497−fromParser :: (Monad m, Unbox a) => ParserD.Parser a m b -> ParserK a m b
498−fromParser (ParserD.Parser step initial extract) =
499− MkParser $ parseDToK step initial extract
702+{-# INLINE_LATE adaptCG #-}
703+adaptCG ::
704+ Monad m => ParserD.Parser a m b -> ParserK (GenArr.Array a) m b
705+adaptCG (ParserD.Parser step initial extract) =
706+ MkParser $ adaptCGWith step initial extract
500707
501708{-
502709-------------------------------------------------------------------------------
@@ -538,10 +745,8 @@ toParser parser = ParserD.Parser step initial extract
538745 Partial _ cont1 -> extract cont1
539746 Continue n cont1 -> return $ ParserD.Continue n cont1
540747
541−#ifndef DISABLE_FUSION
542748{-# RULES "fromParser/toParser fusion" [2]
543749 forall s. toParser (fromParser s) = s #-}
544750{-# RULES "toParser/fromParser fusion" [2]
545751 forall s. fromParser (toParser s) = s #-}
546−#endif
547752-}
core/src/Streamly/Internal/Data/Pipe.hs+4−7
@@ -24,11 +24,10 @@
2424module Streamly.Internal.Data.Pipe
2525 (
2626 -- * Pipe Type
27− Pipe
27+ module Streamly.Internal.Data.Pipe.Type
2828
2929 -- * Pipes
3030 -- ** Mapping
31− , map
3231 , mapM
3332
3433 {-
@@ -112,10 +111,8 @@ module Streamly.Internal.Data.Pipe
112111 , groupsRollingBy
113112 -}
114113
115− -- * Composing Pipes
116− , tee
117− , zipWith
118− , compose
114+
115+
119116
120117 {-
121118 -- * Distributing
@@ -244,7 +241,7 @@ import Prelude
244241
245242-- import Streamly.Data.Fold.Types (Fold(..))
246243import Streamly.Internal.Data.Pipe.Type
247− (Pipe(..), PipeState(..), Step(..), zipWith, tee, map, compose)
244+
248245-- import Streamly.Internal.Data.Array.Type (Array)
249246-- import Streamly.Internal.Data.Ring (Ring)
250247-- import Streamly.Internal.Data.Stream (Stream)
core/src/Streamly/Internal/Data/Producer.hs+6−13
@@ -24,31 +24,24 @@
2424-- unecessary function calls can be avoided.
2525
2626module Streamly.Internal.Data.Producer
27− ( Producer (..)
27+ (
28+ module Streamly.Internal.Data.Producer.Source
29+ , module Streamly.Internal.Data.Producer.Type
2830
2931 -- * Converting
3032 , simplify
31−
32− -- * Producers
33− , nil
34− , nilM
35− , unfoldrM
3633 , fromStreamD
37− , fromList
38−
39− -- * Combinators
40− , NestedLoop (..)
41− , concat
4234 )
4335where
4436
4537#include "inline.hs"
4638
47−import Streamly.Internal.Data.Stream.StreamD.Step (Step(..))
48−import Streamly.Internal.Data.Stream.StreamD.Type (Stream(..))
39+import Streamly.Internal.Data.Stream.Step (Step(..))
40+import Streamly.Internal.Data.Stream.Type (Stream(..))
4941import Streamly.Internal.Data.SVar.Type (defState)
5042import Streamly.Internal.Data.Unfold.Type (Unfold(..))
5143
44+import Streamly.Internal.Data.Producer.Source
5245import Streamly.Internal.Data.Producer.Type
5346import Prelude hiding (concat)
5447
core/src/Streamly/Internal/Data/Producer/Source.hs+1−1
@@ -38,7 +38,7 @@ import GHC.Exts (SpecConstrAnnotation(..))
3838import GHC.Types (SPEC(..))
3939import Streamly.Internal.Data.Parser (ParseError(..), Step(..))
4040import Streamly.Internal.Data.Producer.Type (Producer(..))
41−import Streamly.Internal.Data.Stream.StreamD.Step (Step(..))
41+import Streamly.Internal.Data.Stream.Step (Step(..))
4242
4343import qualified Streamly.Internal.Data.Parser as ParserD
4444-- import qualified Streamly.Internal.Data.Parser.ParserK.Type as ParserK
core/src/Streamly/Internal/Data/Producer/Type.hs+1−1
@@ -33,7 +33,7 @@ where
3333#include "inline.hs"
3434
3535import Fusion.Plugin.Types (Fuse(..))
36−import Streamly.Internal.Data.Stream.StreamD.Step (Step(..))
36+import Streamly.Internal.Data.Stream.Step (Step(..))
3737import Prelude hiding (concat, map)
3838
3939------------------------------------------------------------------------------
core/src/Streamly/Internal/Data/Ring.hs+9−6
@@ -79,13 +79,13 @@ import Foreign.Storable
7979import Foreign.ForeignPtr (ForeignPtr, withForeignPtr, touchForeignPtr)
8080import Foreign.ForeignPtr.Unsafe (unsafeForeignPtrToPtr)
8181import Foreign.Ptr (plusPtr, minusPtr, castPtr)
82−import Streamly.Internal.Data.Unbox as Unboxed (Unbox(peekByteIndex))
82+import Streamly.Internal.Data.Unbox as Unboxed (Unbox(peekAt))
8383import GHC.ForeignPtr (mallocPlainForeignPtrAlignedBytes)
8484import GHC.Ptr (Ptr(..))
8585import Streamly.Internal.Data.MutArray.Type (MutArray)
8686import Streamly.Internal.Data.Fold.Type (Fold(..), Step(..), lmap)
87−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
88−import Streamly.Internal.Data.Stream.StreamD.Step (Step(..))
87+import Streamly.Internal.Data.Stream.Type (Stream)
88+import Streamly.Internal.Data.Stream.Step (Step(..))
8989import Streamly.Internal.Data.Unfold.Type (Unfold(..))
9090import Streamly.Internal.System.IO (unsafeInlineIO)
9191
@@ -411,7 +411,7 @@ unsafeEqArrayN Ring{..} rh A.Array{..} nBytes
411411
412412 check p i = do
413413 (relem :: Word8) <- peek p
414− aelem <- peekByteIndex i w8Contents
414+ aelem <- peekAt i w8Contents
415415 if relem == aelem
416416 then go (p `plusPtr` 1) (i + 1)
417417 else return False
@@ -445,7 +445,7 @@ unsafeEqArray Ring{..} rh A.Array{..} =
445445
446446 check p i = do
447447 (relem :: Word8) <- peek p
448− aelem <- peekByteIndex i w8Contents
448+ aelem <- peekAt i w8Contents
449449 if relem == aelem
450450 then go (p `plusPtr` 1) (i + 1)
451451 else return False
@@ -558,7 +558,8 @@ data Tuple4' a b c d = Tuple4' !a !b !c !d deriving Show
558558{-# INLINE slidingWindowWith #-}
559559slidingWindowWith :: forall m a b. (MonadIO m, Storable a, Unbox a)
560560 => Int -> Fold m ((a, Maybe a), m (MutArray a)) b -> Fold m a b
561−slidingWindowWith n (Fold step1 initial1 extract1) = Fold step initial extract
561+slidingWindowWith n (Fold step1 initial1 extract1 final1) =
562+ Fold step initial extract final
562563
563564 where
564565
@@ -601,6 +602,8 @@ slidingWindowWith n (Fold step1 initial1 extract1) = Fold step initial extract
601602
602603 extract (Tuple4' _ _ _ st) = extract1 st
603604
605+ final (Tuple4' _ _ _ st) = final1 st
606+
604607-- | @slidingWindow collector@ is an incremental sliding window
605608-- fold that does not require all the intermediate elements in a computation.
606609-- This maintains @n@ elements in the window, when a new element comes it slides
core/src/Streamly/Internal/Data/Ring/Generic.hs+49−28
@@ -20,24 +20,21 @@ module Streamly.Internal.Data.Ring.Generic
2020
2121 -- * Conversion
2222 , toMutArray
23+ , copyToMutArray
2324 , toStreamWith
2425 ) where
2526
2627#include "assert.hs"
2728
2829import Control.Monad.IO.Class (liftIO, MonadIO)
29−import Streamly.Internal.Data.Stream.StreamD.Type (Stream)
30+import Streamly.Internal.Data.Stream.Type (Stream)
3031import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
3132import Streamly.Internal.Data.Fold.Type (Fold(..))
32−import Streamly.Internal.Data.MutArray.Generic
33− ( MutArray(..)
34− , new
35− , uninit
36− , putIndexUnsafe
37− , putSliceUnsafe
38− )
39−-- import qualified Streamly.Internal.Data.Stream.StreamD.Type as Stream
33+import Streamly.Internal.Data.MutArray.Generic (MutArray(..))
34+
35+-- import qualified Streamly.Internal.Data.Stream.Type as Stream
4036import qualified Streamly.Internal.Data.Fold.Type as Fold
37+import qualified Streamly.Internal.Data.MutArray.Generic as MutArray
4138
4239-- XXX Use MutableArray rather than keeping a MutArray here.
4340data Ring a = Ring
@@ -58,8 +55,8 @@ data Ring a = Ring
5855{-# INLINE createRing #-}
5956createRing :: MonadIO m => Int -> m (Ring a)
6057createRing count = liftIO $ do
61− arr <- new count
62− arr1 <- uninit arr count
58+ arr <- MutArray.new count
59+ arr1 <- MutArray.uninit arr count
6360 return (Ring
6461 { ringArr = arr1
6562 , ringHead = 0
@@ -67,9 +64,12 @@ createRing count = liftIO $ do
6764 })
6865
6966
67+-- | Note that it is not safe to return a reference to the mutable Ring using a
68+-- scan as the Ring is continuously getting mutated. You could however copy out
69+-- the Ring.
7070{-# INLINE writeLastN #-}
7171writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
72−writeLastN n = Fold step initial extract
72+writeLastN n = Fold step initial extract extract
7373
7474 where
7575
@@ -101,7 +101,7 @@ unsafeInsertRingWith :: Ring a -> a -> IO Int
101101unsafeInsertRingWith Ring{..} x = do
102102 assertM(ringMax >= 1)
103103 assertM(ringHead < ringMax)
104− putIndexUnsafe ringHead ringArr x
104+ MutArray.putIndexUnsafe ringHead ringArr x
105105 let rh1 = ringHead + 1
106106 next = if rh1 == ringMax then 0 else rh1
107107 return next
@@ -124,24 +124,45 @@ seek adj rng@Ring{..}
124124
125125-- | @toMutArray rignHeadAdjustment lengthToRead ring@.
126126-- Convert the ring into a boxed mutable array. Note that the returned MutArray
127−-- may share the same underlying memory as the Ring.
127+-- shares the same underlying memory as the Ring, the user of this API needs to
128+-- ensure that the ring is not mutated during and after the conversion.
129+--
128130{-# INLINE toMutArray #-}
129131toMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
130−toMutArray adj n Ring{..} = do
131− let len = min ringMax n
132− let idx = mod (ringHead + adj) ringMax
133− end = idx + len
134− if end <= ringMax
135− then
136− -- putSliceUnsafe ringArr idx arr1 0 len
137− return $ ringArr { arrStart = idx, arrLen = len }
132+toMutArray adj n Ring{..} =
133+ -- XXX for empty Ring it will raise an Exception: divide by zero
134+ if ringMax <= 0
135+ then MutArray.nil
136+ else do
137+ let len = min ringMax n
138+ let idx = mod (ringHead + adj) ringMax
139+ end = idx + len
140+ if end <= ringMax
141+ then
142+ return $ ringArr { arrStart = idx, arrLen = len }
143+ else do
144+ -- XXX Just swap the elements in the existing ring and return the
145+ -- same array without reallocation.
146+ arr <- liftIO $ MutArray.new len
147+ arr1 <- MutArray.uninit arr len
148+ MutArray.putSliceUnsafe ringArr idx arr1 0 (ringMax - idx)
149+ MutArray.putSliceUnsafe ringArr 0 arr1 (ringMax - idx) (end - ringMax)
150+ return arr1
151+
152+-- | Copy out the mutable ring to a mutable Array.
153+{-# INLINE copyToMutArray #-}
154+copyToMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
155+copyToMutArray adj n Ring{..} = do
156+ if ringMax <= 0
157+ then MutArray.nil
138158 else do
139− -- XXX Just swap the elements in the existing ring and return the
140− -- same array without reallocation.
141− arr <- liftIO $ new len
142− arr1 <- uninit arr len
143− putSliceUnsafe ringArr idx arr1 0 (ringMax - idx)
144− putSliceUnsafe ringArr 0 arr1 (ringMax - idx) (end - ringMax)
159+ let len = min ringMax n
160+ let idx = mod (ringHead + adj) ringMax
161+ end = idx + len
162+ arr <- MutArray.new len
163+ arr1 <- MutArray.uninit arr len
164+ MutArray.putSliceUnsafe ringArr idx arr1 0 (ringMax - idx)
165+ MutArray.putSliceUnsafe ringArr 0 arr1 (ringMax - idx) (end - ringMax)
145166 return arr1
146167
147168-- This would be theoretically slower than toMutArray because of a branch
core/src/Streamly/Internal/Data/Serialize.hsdeleted−157
@@ -1,157 +0,0 @@
1−module Streamly.Internal.Data.Serialize
2− ( Size(..)
3− , Serialize(..)
4− ) where
5−
6−--------------------------------------------------------------------------------
7−-- Imports
8−--------------------------------------------------------------------------------
9−
10−import Control.Monad (void)
11−import Data.List (foldl')
12−import Data.Proxy (Proxy (..))
13−import Streamly.Internal.Data.Unbox (MutableByteArray(..))
14−
15−import GHC.Int (Int16(..), Int32(..), Int64(..), Int8(..))
16−import GHC.Word (Word16(..), Word32(..), Word64(..), Word8(..))
17−import GHC.Stable (StablePtr(..))
18−
19−import qualified Streamly.Internal.Data.Unbox as Unbox
20−
21−import GHC.Exts
22−
23−--------------------------------------------------------------------------------
24−-- Types
25−--------------------------------------------------------------------------------
26−
27−-- | Info about the length of a serializable type. Length can depend on the
28−-- value or can be independent.
29−data Size a
30− = VarSize (a -> Int)
31− | ConstSize !Int
32−
33−-- | A type implementing the 'Serialize' interface supplies operations for
34−-- reading and writing the type from and to a mutable byte array (an unboxed
35−-- representation of the type) in memory. The read operation 'deserialize'
36−-- deserializes the boxed type from the mutable byte array. The write operation
37−-- 'serialize' serializes the boxed type to the mutable byte array.
38−--
39−-- IMPORTANT: The serialized data's byte ordering is not normalized, which means
40−-- it remains the same as the machine's byte order where the function is
41−-- executed. Consequently, it may not be compatible across machines with
42−-- different byte ordering.
43−--
44−-- 'Serialize' contains enough information to serialize and deserialize variable
45−-- length types.
46−--
47−-- >>> import Streamly.Internal.Data.Serialize (Serialize(..), Size(..))
48−--
49−-- >>> :{
50−-- data Object = Object
51−-- { _varLen :: [Int]
52−-- , _constLen :: Int
53−-- }
54−-- :}
55−--
56−-- >>> :{
57−-- instance Serialize Object where
58−-- size =
59−-- case (size :: Size [Int], size :: Size Int) of
60−-- (VarSize f, ConstSize g) ->
61−-- VarSize $ \obj ->
62−-- f (_varLen obj) + g
63−-- _ -> error "size is not defined properly"
64−-- deserialize i arr = do
65−-- (i1, x0) <- deserialize i arr
66−-- (i2, x1) <- deserialize i1 arr
67−-- pure (i2, Object x0 x1)
68−-- serialize i arr (Object x0 x1) = do
69−-- i1 <- serialize i arr x0
70−-- i2 <- serialize i1 arr x1
71−-- pure i2
72−-- :}
73−--
74−class Serialize a where
75− -- | Get the 'Size', in bytes, reqired to store the serialized
76− -- representation of the type. Size cannot be zero.
77− size :: Size a
78−
79− -- We can implement the following functions without returning the `Int`
80− -- offset but that may require traversing the Haskell structure again to get
81− -- the size. Therefore, this is a performance optimization.
82−
83− -- | Deserialize a value from the given byte-index in the array. Returns a
84− -- tuple of the next byte-index and the deserialized value.
85− deserialize :: Int -> MutableByteArray -> IO (Int, a)
86−
87− -- | Write the serialized representation of the value in the array at the
88− -- given byte-index. Returns the next byte-index.
89− serialize :: Int -> MutableByteArray -> a -> IO Int
90−
91−--------------------------------------------------------------------------------
92−-- Instances
93−--------------------------------------------------------------------------------
94−
95−#define DERIVE_SERIALIZE_FROM_UNBOX(_type) \
96−instance Serialize _type where \
97−; size = ConstSize $ Unbox.sizeOf (Proxy :: Proxy _type) \
98−; deserialize off arr = \
99− Unbox.peekByteIndex off arr >>= \
100− \val -> let sz = Unbox.sizeOf (Proxy :: Proxy _type) \
101− in pure (off + sz, val) \
102−; serialize off arr val = \
103− Unbox.pokeByteIndex off arr val \
104− >> let sz = Unbox.sizeOf (Proxy :: Proxy _type) \
105− in pure (off + sz)
106−
107−DERIVE_SERIALIZE_FROM_UNBOX(Char)
108−DERIVE_SERIALIZE_FROM_UNBOX(Int8)
109−DERIVE_SERIALIZE_FROM_UNBOX(Int16)
110−DERIVE_SERIALIZE_FROM_UNBOX(Int32)
111−DERIVE_SERIALIZE_FROM_UNBOX(Int)
112−DERIVE_SERIALIZE_FROM_UNBOX(Int64)
113−DERIVE_SERIALIZE_FROM_UNBOX(Word)
114−DERIVE_SERIALIZE_FROM_UNBOX(Word8)
115−DERIVE_SERIALIZE_FROM_UNBOX(Word16)
116−DERIVE_SERIALIZE_FROM_UNBOX(Word32)
117−DERIVE_SERIALIZE_FROM_UNBOX(Word64)
118−DERIVE_SERIALIZE_FROM_UNBOX(Double)
119−DERIVE_SERIALIZE_FROM_UNBOX(Float)
120−DERIVE_SERIALIZE_FROM_UNBOX((StablePtr a))
121−DERIVE_SERIALIZE_FROM_UNBOX((Ptr a))
122−DERIVE_SERIALIZE_FROM_UNBOX((FunPtr a))
123−
124−instance forall a. Serialize a => Serialize [a] where
125−
126− {-# INLINE size #-}
127− size = VarSize $ \lst ->
128− case size :: Size a of
129− VarSize f ->
130− foldl'
131− (\acc x -> acc + f x)
132− (Unbox.sizeOf (Proxy :: Proxy Int))
133− lst
134− ConstSize sz ->
135− length lst
136− * sz
137− + Unbox.sizeOf (Proxy :: Proxy Int)
138−
139− {-# INLINE deserialize #-}
140− deserialize off arr = do
141− len <- Unbox.peekByteIndex off arr :: IO Int
142− let off1 = off + Unbox.sizeOf (Proxy :: Proxy Int)
143− let peekList buf o 0 = pure (o, reverse buf)
144− peekList buf o i = do
145− (o1, x) <- deserialize o arr
146− peekList (x:buf) o1 (i - 1)
147− peekList [] off1 len
148−
149− {-# INLINE serialize #-}
150− serialize off arr val = do
151− void $ serialize off arr (length val)
152− let off1 = off + Unbox.sizeOf (Proxy :: Proxy Int)
153− let pokeList o [] = pure o
154− pokeList o (x:xs) = do
155− o1 <- serialize o arr x
156− pokeList o1 xs
157− pokeList off1 val
core/src/Streamly/Internal/Data/Serialize/TH.hsnew file+530

This diff is too big to show here.

core/src/Streamly/Internal/Data/Serialize/TH/Bottom.hsnew file+475

This diff is too big to show here.

core/src/Streamly/Internal/Data/Serialize/TH/Common.hsnew file+70

This diff is too big to show here.

core/src/Streamly/Internal/Data/Serialize/TH/RecHeader.hsnew file+411

This diff is too big to show here.

core/src/Streamly/Internal/Data/Serialize/Type.hsnew file+317

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream.hs+20−20

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Container.hs → core/src/Streamly/Internal/Data/Stream/Container.hsrenamed+65−57

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Eliminate.hs → core/src/Streamly/Internal/Data/Stream/Eliminate.hsrenamed+39−33

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Exception.hs → core/src/Streamly/Internal/Data/Stream/Exception.hsrenamed+12−9

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Generate.hs → core/src/Streamly/Internal/Data/Stream/Generate.hsrenamed+5−15

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Lift.hs → core/src/Streamly/Internal/Data/Stream/Lift.hsrenamed+3−3

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/MutChunked.hsdeleted−323

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Nesting.hs → core/src/Streamly/Internal/Data/Stream/Nesting.hsrenamed+90−206

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Step.hs → core/src/Streamly/Internal/Data/Stream/Step.hsrenamed+2−2

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Top.hsdeleted−353

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/Top.hsnew file+480

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Transform.hs → core/src/Streamly/Internal/Data/Stream/Transform.hsrenamed+66−40

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Transformer.hs → core/src/Streamly/Internal/Data/Stream/Transformer.hsrenamed+5−4

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamD/Type.hs → core/src/Streamly/Internal/Data/Stream/Type.hsrenamed+164−33

This diff is too big to show here.

core/src/Streamly/Internal/Data/StreamK.hs+325−124

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamK/Alt.hs → core/src/Streamly/Internal/Data/StreamK/Alt.hsrenamed+1−1

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamK/Transformer.hs → core/src/Streamly/Internal/Data/StreamK/Transformer.hsrenamed+3−3

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/StreamK/Type.hs → core/src/Streamly/Internal/Data/StreamK/Type.hsrenamed+88−17

This diff is too big to show here.

core/src/Streamly/Internal/Data/Time/Clock.hs+3−3

This diff is too big to show here.

core/src/Streamly/Internal/Data/Time/Units.hs−3

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unbox.hs+195−260

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unbox/TH.hs+85−82

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unfold.hs+19−17

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unfold/Type.hs+1−1

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/File.hs+11−13

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/Handle.hs+23−30

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/Path.hsnew file+515

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/Parser.hs+3

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/Stream.hs+43−15

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/String.hs+7−7

This diff is too big to show here.

core/streamly-core.cabal+135−73

This diff is too big to show here.

docs/Developer/optimization-guidelines.md+12

This diff is too big to show here.

docs/Developer/paths.rst+25−2

This diff is too big to show here.

docs/User/HowTo/Compiling.md+5−3

This diff is too big to show here.

docs/User/Project/ApiChangelogs/0.9.0-0.10.0.txtnew file+424

This diff is too big to show here.

docs/User/Project/CONTRIBUTORS.md+8

This diff is too big to show here.

docs/User/Project/Changelog.md+25−10

This diff is too big to show here.

docs/User/Project/README.md+9−2

This diff is too big to show here.

docs/User/Project/Upgrading.md → docs/User/Project/Upgrading-0.8-to-0.9.mdrenamed+87−9

This diff is too big to show here.

docs/User/Tutorials/quick-overview.md+10−14

This diff is too big to show here.

docs/streamly-docs.cabal+1−1

This diff is too big to show here.

hie.yaml+2−2

This diff is too big to show here.

src/Streamly.hsdeleted−625

This diff is too big to show here.

src/Streamly/Data/Fold/Prelude.hs+31−13

This diff is too big to show here.

src/Streamly/Data/Fold/Tee.hs+1−1

This diff is too big to show here.

src/Streamly/Data/Stream/Concurrent.hsdeleted−194

This diff is too big to show here.

src/Streamly/Data/Stream/Exception.hsdeleted−24

This diff is too big to show here.

src/Streamly/Data/Stream/MkType.hs+41−10

This diff is too big to show here.

src/Streamly/Data/Stream/Prelude.hs+217−15

This diff is too big to show here.

src/Streamly/Data/Stream/Time.hsdeleted−37

This diff is too big to show here.

src/Streamly/Data/Unicode/Stream.hsdeleted−99

This diff is too big to show here.

src/Streamly/Internal/Data/Array/ArrayMacros.hdeleted−40

This diff is too big to show here.

src/Streamly/Internal/Data/Channel.hsnew file+19

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Channel/Dispatcher.hs → src/Streamly/Internal/Data/Channel/Dispatcher.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Channel/Types.hs → src/Streamly/Internal/Data/Channel/Types.hsrenamed+12−37

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Channel/Worker.hs → src/Streamly/Internal/Data/Channel/Worker.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Async.hs+5−2

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent.hs+1−44

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent/Channel.hs+29−10

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent/Channel/Type.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Prelude.hsnew file+19

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/SVar.hs+6−5

This diff is too big to show here.

src/Streamly/Internal/Data/IsMap/HashMap.hs+6

This diff is too big to show here.

src/Streamly/Internal/Data/SVar.hs+1

This diff is too big to show here.

src/Streamly/Internal/Data/SmallArray.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Ahead.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Async.hs+2−2

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/Common.hs → src/Streamly/Internal/Data/Stream/Common.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent.hs+51−42

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel.hs+17−14

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Append.hs+11−11

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Consumer.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Dispatcher.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Interleave.hs+9−9

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Operations.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Type.hs+4−4

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream.hs+6

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Combinators.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Common.hs+5−5

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Exception.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Expand.hs+7−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Generate.hs+5−5

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Reduce.hs+6−6

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Top.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Transform.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Type.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Exception/Lifted.hs → src/Streamly/Internal/Data/Stream/Lifted.hsrenamed+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/MkType.hs+33−34

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Parallel.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Prelude.hsnew file+19

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/SVar.hsnew file+18

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/SVar/Eliminate.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/SVar/Generate.hs+3−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Serial.hs+7−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Time.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Zip.hs+41−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Zip/Concurrent.hsdeleted−50

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/ZipAsync.hs+3−2

This diff is too big to show here.

src/Streamly/Internal/Data/Unfold/Prelude.hsnew file+17

This diff is too big to show here.

src/Streamly/Internal/Data/Unfold/SVar.hs+1

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event.hs+5−2

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Darwin.hs+5−5

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Linux.hs+15−9

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Windows.hs+11−8

This diff is too big to show here.

src/Streamly/Internal/FileSystem/FD.hs+11−10

This diff is too big to show here.

src/Streamly/Internal/Network/Inet/TCP.hs+81−46

This diff is too big to show here.

src/Streamly/Internal/Network/Socket.hs+32−29

This diff is too big to show here.

src/Streamly/Internal/System/IOVec.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Unicode/Char.hs+9−7

This diff is too big to show here.

src/Streamly/Memory/Array.hsdeleted−74

This diff is too big to show here.

src/Streamly/Network/Inet/TCP.hs+41−30

This diff is too big to show here.

src/Streamly/Network/Socket.hs+16−30

This diff is too big to show here.

streamly.cabal+84−62

This diff is too big to show here.

targets/Targets.hs+8−3

This diff is too big to show here.

test/Streamly/Test/Data/Array.hs+7−8

This diff is too big to show here.

test/Streamly/Test/Data/Array/Stream.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Fold.hs+7−6

This diff is too big to show here.

test/Streamly/Test/Data/Fold/Window.hs+7−8

This diff is too big to show here.

test/Streamly/Test/Data/MutArray.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Parser.hs+18−13

This diff is too big to show here.

test/Streamly/Test/Data/ParserK.hs+13−9

This diff is too big to show here.

test/Streamly/Test/Data/Ring/Unboxed.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Serialize.hs+251−13

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/CompatV0.hsnew file+46

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/CompatV1.hsnew file+48

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/TH.hsnew file+82

This diff is too big to show here.

test/Streamly/Test/Data/Stream/Concurrent.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Unbox.hs+125−26

This diff is too big to show here.

test/Streamly/Test/Data/Unbox/TH.hs+11−10

This diff is too big to show here.

test/Streamly/Test/FileSystem/Event/Common.hs+1−1

This diff is too big to show here.

test/Streamly/Test/FileSystem/Handle.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Prelude/Serial.hs+7−3

This diff is too big to show here.

test/Streamly/Test/Serialize/Serializable.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Unicode/Char.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Unicode/Parser.hs+6−16

This diff is too big to show here.

test/Streamly/Test/Unicode/Stream.hs+1−1

This diff is too big to show here.

test/lib/Streamly/Test/Prelude/Common.hs+9−2

This diff is too big to show here.

test/streamly-tests.cabal+38−11

This diff is too big to show here.