git.iko.soy/streamly
on master help sign in
swap
30c5a56271 ahead, 0 behind
235 files changed+12844−6551
since splitsnapshots
unifiedsplit
.circleci/config.yml+73−277 .github/workflows/haskell.yml+84−91 .hlint.ignore+5 benchmark/NanoBenchmarks.hs+1−1 benchmark/Streamly/Benchmark/Data/Array/CommonImports.hs+1−1 benchmark/Streamly/Benchmark/Data/Array/Stream.hs+3−3 benchmark/Streamly/Benchmark/Data/Fold.hs+28−4 benchmark/Streamly/Benchmark/Data/Fold/Window.hs+1−1 benchmark/Streamly/Benchmark/Data/MutArray.hs+1−1 benchmark/Streamly/Benchmark/Data/Parser.hs+7−6 benchmark/Streamly/Benchmark/Data/ParserK.hs+6−6 benchmark/Streamly/Benchmark/Data/Ring/Unboxed.hs+1−1 benchmark/Streamly/Benchmark/Data/Serialize.hs+89−53 benchmark/Streamly/Benchmark/Data/Serialize/RecCompatible.hs+24 benchmark/Streamly/Benchmark/Data/Serialize/RecNonCompatible.hs+21 benchmark/Streamly/Benchmark/Data/Serialize/TH.hs+76 benchmark/Streamly/Benchmark/Data/Stream/Common.hs+3−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentCommon.hs+3−3 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentEager.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentInterleaved.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentOrdered.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Eliminate.hs+3−1 benchmark/Streamly/Benchmark/Data/Stream/Exceptions.hs+12−3 benchmark/Streamly/Benchmark/Data/Stream/Expand.hs+9−1 benchmark/Streamly/Benchmark/Data/Stream/Generate.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Lift.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Reduce.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/Split.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/StreamD.hs+3−1 benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs+3−1 benchmark/Streamly/Benchmark/Data/Stream/StreamKAlt.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/ToStreamK.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/Transform.hs+2−2 benchmark/Streamly/Benchmark/Data/Unfold.hs+11−2 benchmark/Streamly/Benchmark/FileSystem/Handle.hs+1−1 benchmark/Streamly/Benchmark/FileSystem/Handle/Read.hs+8−9 benchmark/Streamly/Benchmark/FileSystem/Handle/ReadWrite.hs+9−10 benchmark/Streamly/Benchmark/Prelude/Adaptive.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Ahead.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Async.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Concurrent.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Merge.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Parallel.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Rate.hs+1−1 benchmark/Streamly/Benchmark/Prelude/WAsync.hs+1−1 benchmark/Streamly/Benchmark/Prelude/WSerial.hs+1−1 benchmark/Streamly/Benchmark/Prelude/ZipAsync.hs+1−1 benchmark/Streamly/Benchmark/Prelude/ZipSerial.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Char.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Parser.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Stream.hs+24−6 benchmark/Streamly/Benchmark/Unicode/Utf8.hs+1−1 benchmark/bench-runner/Main.hs+32−4 benchmark/lib/Streamly/Benchmark/Common.hs+1−1 benchmark/lib/Streamly/Benchmark/Common/Handle.hs+3−1 benchmark/lib/Streamly/Benchmark/Prelude.hs+1−1 benchmark/streamly-benchmarks.cabal+23−27 configure+9−9 configure.ac+1−1 core/configure+9−9 core/configure.ac+1−1 core/docs/ApiChangelogs/0.1.0-0.2.0.txt+1471 core/docs/ApiChangelogs/0.2.0-0.2.2.txt+306 core/docs/Changelog.md+58−16 core/src/DocTestDataArray.hs+3 core/src/DocTestDataParserK.hs+15 core/src/DocTestDataStream.hs+2 core/src/DocTestFileSystemHandle.hs+1−1 core/src/Streamly/Console/Stdio.hs+1−27 core/src/Streamly/Data/Array.hs+40−39 core/src/Streamly/Data/Array/Generic.hs+16−13 core/src/Streamly/Data/Fold.hs+150−118 core/src/Streamly/Data/MutArray.hs+18−12 core/src/Streamly/Data/MutArray/Generic.hs+8−3 core/src/Streamly/Data/MutByteArray.hs+106 core/src/Streamly/Data/Parser.hs+130−60 core/src/Streamly/Data/ParserK.hs+89−3 core/src/Streamly/Data/Serialize.hs−52 core/src/Streamly/Data/Stream.hs+131−36 core/src/Streamly/Data/StreamK.hs+2−1 core/src/Streamly/Data/Unfold.hs+2 core/src/Streamly/FileSystem/Handle.hs+7−30 core/src/Streamly/Internal/Data/Array.hs+608−135 core/src/Streamly/Internal/Data/Array/Generic.hs+22−6 core/src/Streamly/Internal/Data/Stream/Chunked.hs → core/src/Streamly/Internal/Data/Array/Stream.hs+74−586 core/src/Streamly/Internal/Data/Array/Type.hs+529−104 core/src/Streamly/Internal/Serialize/FromBytes.hs → core/src/Streamly/Internal/Data/Binary/Parser.hs+8−2 core/src/Streamly/Internal/Serialize/ToBytes.hs → core/src/Streamly/Internal/Data/Binary/Stream.hs+2−2 core/src/Streamly/Internal/Data/Builder.hs+11−9 core/src/Streamly/Internal/Data/Fold/Chunked.hs+18−15 core/src/Streamly/Internal/Data/Fold/Combinators.hs+81−137 core/src/Streamly/Internal/Data/Fold/Container.hs+73−28 core/src/Streamly/Internal/Data/Fold/Step.hs+1−1 core/src/Streamly/Internal/Data/Fold/Tee.hs+2 core/src/Streamly/Internal/Data/Fold/Type.hs+338−114 core/src/Streamly/Internal/Data/Fold/Window.hs+10−4 core/src/Streamly/Internal/Data/IORef/Unboxed.hs+22−19 core/src/Streamly/Internal/Data/IsMap.hs+8 core/src/Streamly/Internal/Data/MutArray.hs+154−33 core/src/Streamly/Internal/Data/MutArray/Generic.hs+65−30 core/src/Streamly/Internal/Data/MutArray/Stream.hs+145 core/src/Streamly/Internal/Data/MutArray/Type.hs+1303−436 core/src/Streamly/Internal/Data/MutByteArray.hs+251 core/src/Streamly/Internal/Data/MutByteArray/Type.hs+321 core/src/Streamly/Internal/Data/Parser.hs+126−119 core/src/Streamly/Internal/Data/Parser/Type.hs+93−70 core/src/Streamly/Internal/Data/ParserK/Type.hs+28−65 core/src/Streamly/Internal/Data/Ring.hs+7−4 core/src/Streamly/Internal/Data/Ring/Generic.hs+1−1 core/src/Streamly/Internal/Data/Serialize.hs−278 core/src/Streamly/Internal/Data/Serialize/TH.hs+337−247 core/src/Streamly/Internal/Data/Serialize/TH/Bottom.hs+475 core/src/Streamly/Internal/Data/Serialize/TH/Common.hs+70 core/src/Streamly/Internal/Data/Serialize/TH/RecHeader.hs+411 core/src/Streamly/Internal/Data/Serialize/Type.hs+317 core/src/Streamly/Internal/Data/Stream/Container.hs+56−48 core/src/Streamly/Internal/Data/Stream/Eliminate.hs+33−25 core/src/Streamly/Internal/Data/Stream/Exception.hs+9−6 core/src/Streamly/Internal/Data/Stream/Generate.hs+2−12 core/src/Streamly/Internal/Data/Stream/MutChunked.hs−323 core/src/Streamly/Internal/Data/Stream/Nesting.hs+86−202 core/src/Streamly/Internal/Data/Stream/Top.hs+243−116 core/src/Streamly/Internal/Data/Stream/Transform.hs+61−35 core/src/Streamly/Internal/Data/Stream/Transformer.hs+2−1 core/src/Streamly/Internal/Data/Stream/Type.hs+147−14 core/src/Streamly/Internal/Data/StreamK.hs+35−40 core/src/Streamly/Internal/Data/StreamK/Type.hs+82−11 core/src/Streamly/Internal/Data/Time/Units.hs−3 core/src/Streamly/Internal/Data/Unbox.hs+172−263 core/src/Streamly/Internal/Data/Unbox/TH.hs+76−85 core/src/Streamly/Internal/Data/Unfold.hs+15−13 core/src/Streamly/Internal/FileSystem/File.hs+11−13 core/src/Streamly/Internal/FileSystem/Handle.hs+21−28 core/src/Streamly/Internal/FileSystem/Path.hs+515 core/src/Streamly/Internal/Unicode/Parser.hs+3 core/src/Streamly/Internal/Unicode/Stream.hs+39−11 core/src/Streamly/Internal/Unicode/String.hs+7−7 core/streamly-core.cabal+102−37 docs/Developer/optimization-guidelines.md+12 docs/Developer/paths.rst+25−2 docs/User/HowTo/Compiling.md+5−3 docs/User/Project/ApiChangelogs/0.9.0-0.10.0.txt+424 docs/User/Project/CONTRIBUTORS.md+8 docs/User/Project/Changelog.md+25−10 docs/User/Project/README.md+9−2 docs/User/Project/Upgrading.md → docs/User/Project/Upgrading-0.8-to-0.9.md+87−9 docs/User/Tutorials/quick-overview.md+10−14 docs/streamly-docs.cabal+1−1 src/Streamly.hs−625 src/Streamly/Data/Fold/Prelude.hs+31−13 src/Streamly/Data/Fold/Tee.hs+1−1 src/Streamly/Data/Stream/Concurrent.hs−194 src/Streamly/Data/Stream/Exception.hs−24 src/Streamly/Data/Stream/MkType.hs+41−10 src/Streamly/Data/Stream/Prelude.hs+217−15 src/Streamly/Data/Stream/Time.hs−37 src/Streamly/Data/Unicode/Stream.hs−99 src/Streamly/Internal/Data/Array/ArrayMacros.h−40 src/Streamly/Internal/Data/Channel.hs+19 src/Streamly/Internal/Data/Stream/Channel/Dispatcher.hs → src/Streamly/Internal/Data/Channel/Dispatcher.hs+3−3 src/Streamly/Internal/Data/Stream/Channel/Types.hs → src/Streamly/Internal/Data/Channel/Types.hs+12−37 src/Streamly/Internal/Data/Stream/Channel/Worker.hs → src/Streamly/Internal/Data/Channel/Worker.hs+3−3 src/Streamly/Internal/Data/Fold/Async.hs+5−2 src/Streamly/Internal/Data/Fold/Concurrent.hs+1−44 src/Streamly/Internal/Data/Fold/Concurrent/Channel.hs+29−10 src/Streamly/Internal/Data/Fold/Concurrent/Channel/Type.hs+3−3 src/Streamly/Internal/Data/Fold/Prelude.hs+19 src/Streamly/Internal/Data/Fold/SVar.hs+6−5 src/Streamly/Internal/Data/IsMap/HashMap.hs+6 src/Streamly/Internal/Data/SVar.hs+1 src/Streamly/Internal/Data/SmallArray.hs+1−1 core/src/Streamly/Internal/Data/Stream/Common.hs → src/Streamly/Internal/Data/Stream/Common.hs+3−3 src/Streamly/Internal/Data/Stream/Concurrent.hs+50−39 src/Streamly/Internal/Data/Stream/Concurrent/Channel.hs+17−14 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Append.hs+10−10 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Consumer.hs+2−2 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Dispatcher.hs+3−3 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Interleave.hs+8−8 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Operations.hs+1−1 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Type.hs+3−3 src/Streamly/Internal/Data/Stream/IsStream.hs+6 src/Streamly/Internal/Data/Stream/IsStream/Common.hs+2−2 src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs+1−1 src/Streamly/Internal/Data/Stream/IsStream/Exception.hs+1−1 src/Streamly/Internal/Data/Stream/IsStream/Expand.hs+5 src/Streamly/Internal/Data/Stream/IsStream/Generate.hs+4−4 src/Streamly/Internal/Data/Stream/IsStream/Reduce.hs+4−4 src/Streamly/Internal/Data/Stream/IsStream/Top.hs+1−1 src/Streamly/Internal/Data/Stream/Exception/Lifted.hs → src/Streamly/Internal/Data/Stream/Lifted.hs+2−2 src/Streamly/Internal/Data/Stream/MkType.hs+33−34 src/Streamly/Internal/Data/Stream/Prelude.hs+19 src/Streamly/Internal/Data/Stream/SVar.hs+18 src/Streamly/Internal/Data/Stream/SVar/Generate.hs+2 src/Streamly/Internal/Data/Stream/Serial.hs+5 src/Streamly/Internal/Data/Stream/Time.hs+2−2 src/Streamly/Internal/Data/Stream/Zip.hs+39 src/Streamly/Internal/Data/Stream/Zip/Concurrent.hs−50 src/Streamly/Internal/Data/Stream/ZipAsync.hs+2−1 src/Streamly/Internal/Data/Unfold/Prelude.hs+17 src/Streamly/Internal/Data/Unfold/SVar.hs+1 src/Streamly/Internal/FileSystem/Event.hs+5−2 src/Streamly/Internal/FileSystem/Event/Darwin.hs+1−1 src/Streamly/Internal/FileSystem/Event/Linux.hs+15−9 src/Streamly/Internal/FileSystem/Event/Windows.hs+11−8 src/Streamly/Internal/FileSystem/FD.hs+10−9 src/Streamly/Internal/Network/Inet/TCP.hs+80−45 src/Streamly/Internal/Network/Socket.hs+29−26 src/Streamly/Internal/Unicode/Char.hs+9−7 src/Streamly/Memory/Array.hs−74 src/Streamly/Network/Inet/TCP.hs+41−30 src/Streamly/Network/Socket.hs+16−30 streamly.cabal+82−62 targets/Targets.hs+5−2 test/Streamly/Test/Data/Array.hs+3−3 test/Streamly/Test/Data/Array/Stream.hs+1−1 test/Streamly/Test/Data/Fold.hs+7−6 test/Streamly/Test/Data/MutArray.hs+1−1 test/Streamly/Test/Data/Parser.hs+12−6 test/Streamly/Test/Data/ParserK.hs+7−2 test/Streamly/Test/Data/Serialize.hs+201−22 test/Streamly/Test/Data/Serialize/CompatV0.hs+46 test/Streamly/Test/Data/Serialize/CompatV1.hs+48 test/Streamly/Test/Data/Serialize/TH.hs+2−2 test/Streamly/Test/Data/Stream/Concurrent.hs+1−1 test/Streamly/Test/Data/Unbox.hs+75−45 test/Streamly/Test/Data/Unbox/TH.hs+11−10 test/Streamly/Test/FileSystem/Event/Common.hs+1−1 test/Streamly/Test/FileSystem/Handle.hs+2−2 test/Streamly/Test/Prelude/Serial.hs+7−3 test/Streamly/Test/Serialize/Serializable.hs+2−2 test/Streamly/Test/Unicode/Char.hs+1−1 test/Streamly/Test/Unicode/Parser.hs+6−16 test/Streamly/Test/Unicode/Stream.hs+1−1 test/lib/Streamly/Test/Prelude/Common.hs+9−2 test/streamly-tests.cabal+29−12
.circleci/config.yml+73−277
@@ -1,7 +1,7 @@
11version: 2.1
22
33#-----------------------------------------------------------------------------
4−# packcheck-0.4.2
4+# packcheck-0.7.0
55# Packcheck global environment variables
66#-----------------------------------------------------------------------------
77
@@ -11,6 +11,8 @@ env: &env
1111 # Common options
1212 # ------------------------------------------------------------------------
1313 # GHC_OPTIONS: "-Werror"
14+ # For updating see: https://downloads.haskell.org/~ghcup/
15+ GHCUP_VERSION: 0.1.20.0
1416 CABAL_REINIT_CONFIG: "y"
1517 LC_ALL: "C.UTF-8"
1618
@@ -22,7 +24,6 @@ env: &env
2224 # DISABLE_DOCS: "y"
2325 # DISABLE_SDIST_BUILD: "y"
2426 # DISABLE_DIST_CHECKS: "y"
25− ENABLE_INSTALL: "y"
2627
2728 # ------------------------------------------------------------------------
2829 # stack options
@@ -38,15 +39,12 @@ env: &env
3839 # cabal options
3940 # ------------------------------------------------------------------------
4041 CABAL_CHECK_RELAX: "y"
41− CABAL_NO_SANDBOX: "y"
42− CABAL_HACKAGE_MIRROR: "hackage.haskell.org:http://hackage.fpcomplete.com"
4342 CABAL_BUILD_OPTIONS: "-j1 --flag limit-build-mem"
4443
4544 # ------------------------------------------------------------------------
4645 # Where to find the required tools
4746 # ------------------------------------------------------------------------
48− PATH: /opt/ghc/bin:/opt/ghcjs/bin:/sbin:/usr/sbin:/bin:/usr/bin
49− TOOLS_DIR: /opt
47+ PATH: /sbin:/usr/sbin:/bin:/usr/bin
5048
5149 # ------------------------------------------------------------------------
5250 # Location of packcheck.sh (the shell script invoked to perform CI tests ).
@@ -59,15 +57,15 @@ env: &env
5957 # If you have not committed packcheck.sh in your repo at PACKCHECK
6058 # then it is automatically pulled from this URL.
6159 PACKCHECK_GITHUB_URL: "https://raw.githubusercontent.com/composewell/packcheck"
62− PACKCHECK_GITHUB_COMMIT: "a302772c8f3791c477edd1764b82f8d1f5b22fdf"
60+ PACKCHECK_GITHUB_COMMIT: "f6cfd7b7a33b4bf6fd63cce6a3fb3d26f5031403"
6361
6462executors:
6563 amd64-executor:
6664 docker:
67− - image: ubuntu:xenial
65+ - image: ubuntu:latest
6866 x86-executor:
6967 docker:
70− - image: i386/ubuntu:eoan
68+ - image: i386/ubuntu:focal
7169
7270#-----------------------------------------------------------------------------
7371# Common utility stuff, not to be modified usually
@@ -79,38 +77,50 @@ preinstall: &preinstall
7977 # required for https/cache save and restore
8078 apt-get install -y ca-certificates
8179
82− # required to (re)generate the configure script
83− apt-get install -y autoconf
84−
85− # For ghc and cabal-install packages from hvr's ppa
80+ # Alternative way of installing ghc and cabal, directly from
81+ # haskell.org instead of using ghcup. NOTE: this is for Debian
82+ # only and is debian release specific.
8683 # gnupg is required for apt-key to work
87− apt-get install -y gnupg
88− apt-get install -y apt-transport-https
89− apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA3CBA3FFE22B574
90− apt-get install -y software-properties-common
91− add-apt-repository -y ppa:hvr/ghc
92− # echo "deb http://downloads.haskell.org/debian stretch main" >> /etc/apt/sources.list
93− echo "deb-src http://ppa.launchpad.net/hvr/ghc/ubuntu xenial main" >> /etc/apt/sources.list
94− apt-get update
84+ #apt-get install -y gnupg
85+ #apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA3CBA3FFE22B574
86+ #echo "deb http://downloads.haskell.org/debian buster main" >> /etc/apt/sources.list
87+ #apt-get update
9588
96− # required for packcheck
97− apt-get install -y curl
89+ # required for outbound https for stack and for stack setup
90+ apt-get install -y netbase xz-utils make
91+ apt-get install -y zlib1g-dev
9892
99− # required for circleci.
100− apt-get install -y libcurl4-gnutls-dev
101− apt-get install -y git
93+ # For ghcup to install ghc
94+ if test -n "$GHCUP_VERSION"
95+ then
96+ apt-get install -y gcc
97+ apt-get install -y g++
98+ fi
99+
100+ # libgmp required by ghc for linking
101+ apt-get install -y libgmp-dev
102102 apt-get install -y libtinfo-dev
103103
104− # required for outbound https for stack and for stack setup
105− apt-get install -y netbase xz-utils make
104+ # Required by cabal when git URL is specified in project file
105+ apt-get install -y git
106+
107+ # Required for and by packcheck
108+ apt-get install -y curl
106109
107−project-preinstall: &project-preinstall
108− run: |
109110 # Get packcheck if needed
110− CURL=$(which curl)
111− PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
112− if test ! -e "$PACKCHECK"; then $CURL -sL -o "$PACKCHECK" $PACKCHECK_URL; fi;
113− chmod +x $PACKCHECK
111+ if test ! -e "$PACKCHECK"
112+ then
113+ if test -z "$PACKCHECK_GITHUB_COMMIT"
114+ then
115+ die "PACKCHECK_GITHUB_COMMIT is not specified."
116+ fi
117+ PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
118+ curl --fail -sL -o "$PACKCHECK" $PACKCHECK_URL || exit 1
119+ chmod +x $PACKCHECK
120+ elif test ! -x "$PACKCHECK"
121+ then
122+ chmod +x $PACKCHECK
123+ fi
114124
115125restore: &restore
116126 # Needs to happen after installing ca-certificates
@@ -126,271 +136,71 @@ save: &save
126136 save_cache:
127137 key: v1-{{ .Environment.CIRCLE_JOB }}-{{ .Branch }}
128138 paths:
129− - ~/.cabal
130− - ~/.ghc
131139 - ~/.local
140+ - ~/.cabal
132141 - ~/.stack
142+ - ~/.ghcup
133143
134144#-----------------------------------------------------------------------------
135145# Build matrix
136146#-----------------------------------------------------------------------------
137147
138148jobs:
139− cabal-ghc-8_10_4:
149+ cabal-ghc-8_6_5:
140150 <<: *env
141151 executor: amd64-executor
142152 steps:
143− - *preinstall
144153 - checkout
145− - *project-preinstall
146154 - *restore
155+ - *preinstall
147156 - run:
148157 environment:
158+ GHCVER: "8.6.5"
159+ CABAL_VERSION: "3.6.2.0"
149160 CABAL_PROJECT: "cabal.project"
150− DISABLE_TEST: "yes"
151− DISABLE_BENCH: "yes"
152161 DISABLE_SDIST_BUILD: "yes"
162+ CABAL_BUILD_OPTIONS: "--flag debug --flag -opt"
153163 command: |
154− apt-get install -y ghc-8.10.4
155− apt-get install -y cabal-install-3.4
156164 bash -c "$PACKCHECK cabal"
157165 - *save
158− stack-ghc-8_8:
166+ cabal-ghc-9_8_1-docspec:
159167 <<: *env
160168 executor: amd64-executor
161169 steps:
162− - *preinstall
163170 - checkout
164− - *project-preinstall
165171 - *restore
166− - run:
167− name: GHC 8.8 + stack lts-16.12
168− environment:
169− BUILD: "stack"
170− RESOLVER: "lts-16.12"
171− SDIST_OPTIONS: "--ignore-check"
172− STACK_BUILD_OPTIONS: "--flag streamly-benchmarks:-opt"
173− command: |
174− apt-get build-dep -y ghc-8.8.4
175− apt-get install -y cabal-install-3.2
176− bash -c "$PACKCHECK $BUILD"
177− - *save
178− cabal-ghc-8_6_5:
179− <<: *env
180− executor: amd64-executor
181− steps:
182172 - *preinstall
183− - checkout
184− - *project-preinstall
185− - *restore
186− - run: |
187− apt-get install -y ghc-8.6.3
188− apt-get install -y cabal-install-2.4
189− bash -c "$PACKCHECK cabal-v2"
190− - *save
191− cabal-ghc-8_4_4:
192− <<: *env
193− executor: x86-executor
194− steps:
195− - *preinstall
196− - checkout
197− - *project-preinstall
198− - *restore
199− - run:
200− name: GHC 8.4.4 + x86 + debug
201− environment:
202− BUILD: "cabal-v2"
203− GHCVER: "8.4.4"
204− CABAL_BUILD_OPTIONS: "--flag debug --flag limit-build-mem"
205− command: |
206− apt-get install -y ghc-8.4.4
207− apt-get install -y cabal-install-3.2
208− bash -c "$PACKCHECK $BUILD"
209− - *save
210− cabal-ghc-8_2_2:
211− <<: *env
212− executor: amd64-executor
213− steps:
214− - *preinstall
215− - checkout
216− - *project-preinstall
217− - *restore
218173 - run:
219174 environment:
220− BUILD: "cabal-v2"
221− DISABLE_TEST: "yes"
222− DISABLE_BENCH: "yes"
223− DISABLE_DOCS: "yes"
224− DISABLE_SDIST_BUILD: "yes"
225− DISABLE_DIST_CHECKS: "yes"
175+ GHCVER: "9.8.1"
176+ CABAL_VERSION: "3.10.1.0"
177+ CABAL_PROJECT: "cabal.project.doctest"
178+ DISABLE_SDIST_BUILD: "y"
179+ DISABLE_TEST: "y"
180+ DISABLE_BENCH: "y"
181+ DISABLE_DOCS: "y"
182+ ENABLE_DOCSPEC: "y"
183+ DOCSPEC_URL: https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz
184+ DOCSPEC_OPTIONS: "--timeout 60 --check-properties --property-variables xs"
226185 command: |
227− apt-get install -y ghc-8.2.2
228− apt-get install -y cabal-install-3.2
229− bash -c "$PACKCHECK $BUILD"
230− - *save
231− cabal-ghc-8_0_2:
232− <<: *env
233− executor: amd64-executor
234− steps:
235− - *preinstall
236− - checkout
237− - *project-preinstall
238− - *restore
239− - run:
240− environment:
241− BUILD: "cabal-v2"
242− DISABLE_TEST: "yes"
243− DISABLE_BENCH: "yes"
244− DISABLE_DOCS: "yes"
245− DISABLE_SDIST_BUILD: "yes"
246− DISABLE_DIST_CHECKS: "yes"
247− command: |
248− apt-get install -y ghc-8.0.2
249− apt-get install -y cabal-install-3.2
250− bash -c "$PACKCHECK $BUILD"
251− - *save
252− cabal-ghc-7_10_3:
253− <<: *env
254− executor: amd64-executor
255− steps:
256− - *preinstall
257− - checkout
258− - *project-preinstall
259− - *restore
260− - run: |
261− apt-get install -y ghc-7.10.3
262− apt-get install -y cabal-install-2.4
263− bash -c "$PACKCHECK cabal-v2"
186+ bash -c "$PACKCHECK cabal"
264187 - *save
265− stack-ghc-8_4:
188+ hlint-trailing-spaces:
266189 <<: *env
267190 executor: amd64-executor
268191 steps:
269− - *preinstall
270192 - checkout
271− - *project-preinstall
272193 - *restore
273− - run: |
274− bash -c "$PACKCHECK stack RESOLVER=lts-12"
275− - *save
276− stack-ghc-8_2:
277− <<: *env
278− executor: amd64-executor
279− steps:
280194 - *preinstall
281− - checkout
282− - *project-preinstall
283− - *restore
284− - run: |
285− bash -c "$PACKCHECK stack RESOLVER=lts-11"
286− - *save
287− cabal-ghcjs-8_4:
288− <<: *env
289− executor: amd64-executor
290− steps:
291− - *preinstall
292− - checkout
293− - *project-preinstall
294− - *restore
295195 - run:
296− name: GHCJS 8.4+no-test+no-docs
196+ name: hlint and trailing spaces
297197 environment:
298− BUILD: cabal-v2
299− ENABLE_GHCJS: "yes"
300− DISABLE_TEST: "yes"
301− ENABLE_INSTALL: ""
302− DISABLE_DOCS: "yes"
303198 DISABLE_SDIST_BUILD: "yes"
304− command: |
305− add-apt-repository -y ppa:hvr/ghcjs
306− add-apt-repository -y 'deb https://deb.nodesource.com/node_11.x xenial main'
307− curl -sSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -
308− apt-get update
309− apt-get install -y cabal-install-3.2
310− apt-get install -y nodejs
311− apt-get install -y ghcjs-8.4
312− cabal user-config update -a "jobs: 1"
313− bash -c "$PACKCHECK $BUILD"
314− - *save
315− coveralls-ghc-8_2_2:
316− <<: *env
317− executor: amd64-executor
318− steps:
319− - *preinstall
320− - checkout
321− - *project-preinstall
322− - *restore
323− - run:
324− environment:
325− BUILD: cabal
326− COVERALLS_OPTIONS: "--repo-token=\"$REPO_TOKEN\" --coverage-mode=StrictlyFullLines --exclude-dir=test test"
327− GHC_OPTIONS: "-DCOVERAGE_BUILD"
328− command: |
329− apt-get install -y ghc-8.2.2
330− apt-get install -y cabal-install-2.4
331− bash -c "$PACKCHECK cabal-v1"
332− - *save
333− coveralls-ghc-8_8_3:
334− <<: *env
335− executor: amd64-executor
336− steps:
337− - *preinstall
338− - checkout
339− - *project-preinstall
340− - *restore
341− - run:
342− environment:
343− BUILD: cabal-v2
344− COVERALLS_OPTIONS: "--repo-token=\"$REPO_TOKEN\" --coverage-mode=StrictlyFullLines --exclude-dir=test"
345− CABAL_PROJECT: "cabal.project.coverage"
346− CABAL_BUILD_TARGETS: ""
347− command: |
348− apt-get install -y ghc-8.8.3
349− apt-get install -y cabal-install-3.0
350− bash -c "$PACKCHECK cabal-v2"
351− no_output_timeout: 25m
352− - *save
353− coveralls-ghc-8_10_2:
354− <<: *env
355− executor: amd64-executor
356− steps:
357− - *preinstall
358− - checkout
359− - *project-preinstall
360− - *restore
361− - run:
362− name: Build and test with inspection, coverage, Werror
363− command: |
364− apt-get install -y ghc-8.10.2
365− apt-get install -y cabal-install-3.2
366− # Run tests with coverage
367− cabal update
368− cabal user-config update -a "jobs: 1"
369− bin/test.sh --coverage --raw
370− # Upload results to coveralls.io
371− PATH=$HOME/.cabal/bin:$PATH
372− export PATH
373− which hpc-coveralls 2>/dev/null || cabal install --project-file cabal.project.hpc-coveralls hpc-coveralls
374− hpc-coveralls --repo-token="$REPO_TOKEN" --coverage-mode=StrictlyFullLines
375− no_output_timeout: 25m
376− - *save
377− hlint-src:
378− <<: *env
379− executor: amd64-executor
380− steps:
381− - *preinstall
382− - checkout
383− - *project-preinstall
384− - *restore
385− - run:
386− name: Hlint src
387− environment:
388− BUILD: hlint
199+ HLINT_VERSION: 3.6.1
389200 HLINT_OPTIONS: lint
390− DISABLE_SDIST_BUILD: "yes"
391201 HLINT_TARGETS: core/src src test benchmark
392202 command: |
393− bash -c "$PACKCHECK $BUILD" || exit 1
203+ bash -c "$PACKCHECK hlint" || exit 1
394204 echo "Checking trailing spaces..."
395205 count=$(find . -name "*.hs" -exec grep -H '\ $' {} \; | tee /dev/tty | wc -l)
396206 exit $count
@@ -400,23 +210,9 @@ workflows:
400210 version: 2
401211 build:
402212 jobs:
403− - cabal-ghc-8_10_4:
404− name: GHC 8.10.4 + sdist + no-test + no-bench
405− #- cabal-ghc-8.6.5
406− #- cabal-ghc-8_4_4:
407− # name: GHC 8.4.4 + x86 + debug
408− #- cabal-ghc-8_2_2:
409− # name: GHC 8.2.2 + no-test + no-bench + no-docs
410− #- cabal-ghc-8_0_2:
411− # name: GHC 8.0.2 + no-test + no-bench + no-docs
412− #- cabal-ghc-7.10.3
413− # - cabal-ghcjs-8_4:
414− # name: GHCJS 8.4 + no-test + no-docs
415− #- stack-ghc-8.4
416− #- stack-ghc-8.2
417− #- coveralls-ghc-8.2.2
418− #- coveralls-ghc-8.8.3
419− #- coveralls-ghc-8_10_2:
420− # name: GHC 8.10.2 + inspection + coverage + Werror
421− - hlint-src:
422− name: Hlint src + Trailing Spaces
213+ #- cabal-ghc-8_6_5:
214+ # name: 8.6.5-debug-unoptimized
215+ - cabal-ghc-9_8_1-docspec:
216+ name: ghc-9.8.1-docspec
217+ - hlint-trailing-spaces:
218+ name: hlint and trailing spaces
.github/workflows/haskell.yml+84−91
@@ -13,9 +13,15 @@ jobs:
1313 # packcheck environment variables
1414 LC_ALL: C.UTF-8
1515 BUILD: ${{ matrix.build }}
16+ # For updating see: https://downloads.haskell.org/~ghcup/
17+ GHCUP_VERSION: 0.1.20.0
18+ DOCSPEC_URL: https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz
19+ DOCSPEC_OPTIONS: "--timeout 60 --check-properties --property-variables xs"
20+ GHCUP_GHC_OPTIONS: ${{ matrix.ghcup_ghc_options }}
1621 GHCVER: ${{ matrix.ghc_version }}
1722 CABALVER: ${{ matrix.cabal_version }}
1823 DISABLE_DOCS: ${{ matrix.disable_docs }}
24+ ENABLE_DOCSPEC: ${{ matrix.enable_docspec }}
1925 DISABLE_TEST: ${{ matrix.disable_test }}
2026 DISABLE_BENCH: ${{ matrix.disable_bench }}
2127 DISABLE_DIST_CHECKS: ${{ matrix.disable_dist_checks }}
@@ -39,14 +45,18 @@ jobs:
3945 STACK_BUILD_OPTIONS: ${{ matrix.stack_build_options }}
4046
4147 # packcheck location and revision
42− PACKCHECK_LOCAL_PATH: "./packcheck.sh"
48+ PACKCHECK: "./packcheck.sh"
4349 PACKCHECK_GITHUB_URL: "https://raw.githubusercontent.com/composewell/packcheck"
44− PACKCHECK_GITHUB_COMMIT: "a302772c8f3791c477edd1764b82f8d1f5b22fdf"
50+ PACKCHECK_GITHUB_COMMIT: "v0.7.0"
4551
4652 # Pull token from "secrets" setting of the github repo
4753 COVERALLS_TOKEN: ${{ secrets.COVERALLS_TOKEN }}
4854 COVERAGE: ${{ matrix.coverage }}
49− DOCTEST: ${{ matrix.doctest }}
55+
56+ # hlint
57+ HLINT_VERSION: 3.6.1
58+ HLINT_OPTIONS: "lint"
59+ HLINT_TARGETS: "core/src src test benchmark"
5060
5161 # Subdir
5262 SUBDIR: ${{ matrix.subdir }}
@@ -58,8 +68,8 @@ jobs:
5868 matrix:
5969 # The order is important to optimize fail-fast.
6070 name:
61− - 9.4.4-Werror
62− - 9.2.7-docspec
71+ - 9.8.1-Werror
72+ # - 9.8.1-docspec
6373 # - 8.10.7-coverage
6474
6575 # Note: if cabal.project is not specified benchmarks and tests won't
@@ -68,51 +78,74 @@ jobs:
6878 include:
6979 - name: head
7080 ghc_version: head
81+ # The URL may change, to find a working URL go to https://gitlab.haskell.org/ghc/ghc/-/jobs/
82+ # Find a debian10/11/12 job, click on a passed/failed status, at the
83+ # end of the output you will find the tar.xz name, put that tar
84+ # name after "raw/", and put the job name after "job=".
85+ # Also see https://github.com/mpickering/ghc-artefact-nix/blob/master/gitlab-artifact.nix
86+ #
87+ # May also use ghcup for installing ghc head version, use the
88+ # version "LatestNightly", and the following config:
89+ # ghcup config add-release-channel https://ghc.gitlab.haskell.org/ghcup-metadata/ghcup-nightlies-0.0.7.yaml
90+ ghcup_ghc_options: "-u https://gitlab.haskell.org/ghc/ghc/-/jobs/artifacts/master/raw/ghc-x86_64-linux-deb10-int_native-validate.tar.xz?job=x86_64-linux-deb10-int_native-validate"
7191 runner: ubuntu-latest
7292 build: cabal
73− cabal_version: 3.8.1.0
93+ cabal_version: 3.10.2.0
7494 cabal_project: cabal.project.ghc-head
7595 disable_sdist_build: "y"
7696 ignore_error: true
77− - name: 9.6.1
78− ghc_version: 9.6.1
97+ # Note: use linux for warning build for convenient dev testing
98+ - name: 9.8.1-Werror
99+ ghc_version: 9.8.1
79100 runner: ubuntu-latest
80101 build: cabal
81102 cabal_version: 3.10.1.0
82103 disable_sdist_build: "y"
83− cabal_project: cabal.project
84− ignore_error: true
85− - name: 9.4.4-Werror
86− ghc_version: 9.4.4
87− runner: ubuntu-latest
88− build: cabal
89− cabal_version: 3.8.1.0
90− disable_sdist_build: "y"
91104 cabal_project: cabal.project.Werror
92105 ignore_error: false
93− - name: 9.4.4-fusion-inspection
94− ghc_version: 9.4.4
106+ #- name: 9.8.1-docspec
107+ # ghc_version: 9.8.1
108+ # runner: ubuntu-latest
109+ # build: cabal
110+ # cabal_version: 3.10.1.0
111+ # cabal_project: cabal.project.doctest
112+ # disable_test: "y"
113+ # disable_bench: "y"
114+ # disable_docs: "y"
115+ # enable_docspec: "y"
116+ # disable_sdist_build: "y"
117+ # ignore_error: false
118+ - name: 9.8.1-fusion-inspection
119+ ghc_version: 9.8.1
95120 runner: ubuntu-latest
96121 build: cabal
97− cabal_version: 3.8.1.0
122+ cabal_version: 3.10.1.0
98123 disable_sdist_build: "y"
99124 cabal_project: cabal.project
100125 cabal_build_options: "--flag fusion-plugin --flag inspection"
101126 ignore_error: false
102− - name: 9.2.7
103− ghc_version: 9.2.7
104− runner: ubuntu-latest
127+ - name: 9.6.3-macos
128+ ghc_version: 9.6.3
129+ runner: macos-latest
105130 build: cabal
131+ cabal_version: 3.10.1.0
132+ disable_sdist_build: "y"
106133 cabal_project: cabal.project
107− cabal_version: 3.6.2.0
134+ ignore_error: false
135+ - name: 9.4.7
136+ ghc_version: 9.4.7
137+ runner: ubuntu-latest
138+ build: cabal
139+ cabal_version: 3.8.1.0
108140 disable_sdist_build: "y"
141+ cabal_project: cabal.project
109142 ignore_error: false
110− - name: 9.2.7-docspec
111− ghc_version: 9.2.7
143+ - name: 9.2.8
144+ ghc_version: 9.2.8
112145 runner: ubuntu-latest
113− doctest: "y"
114− cabal_version: 3.6.2.0
146+ build: cabal
115147 cabal_project: cabal.project
148+ cabal_version: 3.6.2.0
116149 disable_sdist_build: "y"
117150 ignore_error: false
118151 - name: 9.0.2-streamly-sdist
@@ -138,14 +171,6 @@ jobs:
138171 stack_build_options: "--flag streamly-benchmarks:-opt"
139172 cabal_version: 3.6.2.0
140173 ignore_error: false
141− - name: 8.10.7-macos
142− ghc_version: 8.10.7
143− runner: macos-latest
144− build: cabal
145− cabal_version: 3.6.2.0
146− cabal_project: cabal.project
147− disable_sdist_build: "y"
148− ignore_error: false
149174 # - name: 8.10.7-coverage
150175 # ghc_version: 8.10.7
151176 # runner: ubuntu-latest
@@ -160,15 +185,6 @@ jobs:
160185 cabal_project: cabal.project
161186 disable_sdist_build: "y"
162187 ignore_error: false
163− - name: 8.6.5-no-fusion-unoptimized
164− ghc_version: 8.6.5
165− runner: ubuntu-latest
166− build: cabal
167− cabal_version: 3.6.2.0
168− cabal_project: cabal.project
169− cabal_build_options: "--flag no-fusion --flag -opt"
170− disable_sdist_build: "y"
171− ignore_error: false
172188 - name: 8.6.5-debug-unoptimized
173189 ghc_version: 8.6.5
174190 runner: ubuntu-latest
@@ -178,66 +194,44 @@ jobs:
178194 cabal_build_options: "--flag debug --flag -opt"
179195 disable_sdist_build: "y"
180196 ignore_error: false
197+ - name: hlint
198+ build: hlint
199+ runner: ubuntu-latest
200+ ignore_error: true
181201
182202 steps:
183203 - uses: actions/checkout@v2
184204
185205 - uses: actions/cache@v1
186− name: Cache ~/.cabal
206+ name: Cache common directories
187207 with:
188− path: ~/.cabal
208+ path: |
209+ ~/.local
210+ ~/.cabal
211+ ~/.stack
212+ ~/.ghcup
189213 # Bump the key version to clear the cache
190214 key: ${{ runner.os }}-${{ matrix.ghc_version }}-cabal-v2
191215
192216 - name: Download packcheck
193217 run: |
194− # Get packcheck if needed
195− CURL=$(which curl)
196− PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
197− if test ! -e "$PACKCHECK_LOCAL_PATH"; then $CURL -sL -o "$PACKCHECK_LOCAL_PATH" $PACKCHECK_URL; fi;
198− chmod +x $PACKCHECK_LOCAL_PATH
199−
200− os=$(uname -s -m)
201− case "$os" in
202− "Linux x86_64") GHCUP_ARCH="x86_64-linux" ;;
203− "Darwin x86_64") GHCUP_ARCH="x86_64-apple-darwin" ;;
204− *) echo "Unknown OS/Arch: $os"; exit 1;;
205− esac
206−
207− # Check available versions here: https://downloads.haskell.org/~ghcup/
208− GHCUP_VER=0.1.18.0
209− $CURL -sL -o ./ghcup https://downloads.haskell.org/~ghcup/$GHCUP_VER/${GHCUP_ARCH}-ghcup-$GHCUP_VER
210− chmod +x ./ghcup
211− if test "$GHCVER" = "head"
218+ if test ! -e "$PACKCHECK"
212219 then
213− # The URL may change, to find a working URL go to https://gitlab.haskell.org/ghc/ghc/-/jobs/
214− # Find a debian10 job, click on a passed/failed job, at the
215− # end of the output you will find the tar.xz name, put that after
216− # "raw/", and put the job name after "job=".
217− # Also see https://github.com/mpickering/ghc-artefact-nix/blob/master/gitlab-artifact.nix
218− ./ghcup install ghc -u https://gitlab.haskell.org/ghc/ghc/-/jobs/artifacts/master/raw/ghc-x86_64-linux-deb10-int_native-validate.tar.xz?job=x86_64-linux-deb10-int_native-validate head
219− else
220− ./ghcup install ghc $GHCVER
220+ if test -z "$PACKCHECK_GITHUB_COMMIT"
221+ then
222+ die "PACKCHECK_GITHUB_COMMIT is not specified."
223+ fi
224+ PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
225+ curl --fail -sL -o "$PACKCHECK" $PACKCHECK_URL || exit 1
226+ chmod +x $PACKCHECK
227+ elif test ! -x "$PACKCHECK"
228+ then
229+ chmod +x $PACKCHECK
221230 fi
222− ./ghcup install cabal $CABALVER
223− ./ghcup set ghc $GHCVER
224− ./ghcup set cabal $CABALVER
225231
226232 - name: Run tests
227233 run: |
228− if test -n "$DOCTEST"
229− then
230− mkdir -p $HOME/.cabal/bin
231− curl -sL https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz > cabal-docspec.xz
232− echo '0829bd034fba901cbcfe491d98ed8b28fd54f9cb5c91fa8e1ac62dc4413c9562 cabal-docspec.xz' | sha256sum -c -
233− xz -d < cabal-docspec.xz > $HOME/.cabal/bin/cabal-docspec
234− rm -f cabal-docspec.xz
235− chmod a+x $HOME/.cabal/bin/cabal-docspec
236− cabal update
237− cabal build --project-file cabal.project.doctest all
238− $HOME/.cabal/bin/cabal-docspec --version
239− $HOME/.cabal/bin/cabal-docspec --timeout 60 --check-properties --property-variables xs
240− elif test -n "$COVERAGE"
234+ if test -n "$COVERAGE"
241235 then
242236 # Run tests with coverage
243237 cabal update
@@ -250,16 +244,15 @@ jobs:
250244 # XXX Uncomment this and fix it properly later
251245 # hpc-coveralls --repo-token="$COVERALLS_TOKEN" --coverage-mode=StrictlyFullLines
252246 else
253− export TOOLS_DIR=/opt
254247 # /usr/local/opt/curl/bin for macOS
255248 export PATH=$HOME/.local/bin:$HOME/.ghcup/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/opt/curl/bin
256249 if test -n "$SUBDIR"
257250 then
258− mv "$PACKCHECK_LOCAL_PATH" "$SUBDIR/packcheck.sh"
251+ mv "$PACKCHECK" "$SUBDIR/packcheck.sh"
259252 # This is required as dist-newstyle will be created on the top level
260253 # directory as it is considered the working dir.
261254 rm cabal.project
262255 cd "$SUBDIR"
263256 fi
264− bash -c "$PACKCHECK_LOCAL_PATH $BUILD"
257+ bash -c "$PACKCHECK $BUILD"
265258 fi
.hlint.ignore+5
@@ -2,6 +2,9 @@ core/src/Streamly/Internal/Data/Stream/StreamK/Type.hs
22core/src/Streamly/Internal/Data/Pipe/Type.hs
33core/src/Streamly/Internal/Unicode/Stream.hs
44src/Streamly/Internal/Data/SmallArray/Type.hs
5+src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs
6+src/Streamly/Internal/Data/Stream/Serial.hs
7+src/Streamly/Internal/Data/Stream/Zip.hs
58test/Streamly/Test/Data/Array.hs
69test/Streamly/Test/Data/Parser.hs
710test/Streamly/Test/Data/ParserK.hs
@@ -10,6 +13,7 @@ test/Streamly/Test/Data/Unfold.hs
1013test/Streamly/Test/FileSystem/Event.hs
1114test/Streamly/Test/Prelude/Concurrent.hs
1215test/Streamly/Test/Data/Stream/Concurrent.hs
16+test/lib/Streamly/Test/Prelude/Common.hs
1317test/Streamly/Test/Prelude/Fold.hs
1418test/Streamly/Test/Prelude/Rate.hs
1519test/Streamly/Test/Prelude/Serial.hs
@@ -27,6 +31,7 @@ benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs
2731benchmark/Streamly/Benchmark/Data/Unfold.hs
2832benchmark/Streamly/Benchmark/FileSystem/Handle.hs
2933benchmark/Streamly/Benchmark/Prelude/Async.hs
34+benchmark/Streamly/Benchmark/Prelude/Concurrent.hs
3035benchmark/Streamly/Benchmark/Prelude/Merge.hs
3136benchmark/Streamly/Benchmark/Prelude/Parallel.hs
3237benchmark/Streamly/Benchmark/Prelude/Rate.hs
benchmark/NanoBenchmarks.hs+1−1
@@ -18,7 +18,7 @@ import qualified Streamly.Data.Fold as FL
1818import qualified Streamly.Internal.Data.Stream as S
1919import qualified Streamly.Internal.Data.Stream.IsStream as Internal
2020
21−import Gauge
21+import Test.Tasty.Bench
2222import System.Random
2323
2424maxValue :: Int
benchmark/Streamly/Benchmark/Data/Array/CommonImports.hs+1−1
@@ -9,7 +9,7 @@ import qualified Streamly.Data.Fold as Fold
99import qualified Streamly.Data.Stream as S
1010import qualified Streamly.Internal.Data.Stream as Stream
1111
12−import Gauge
12+import Test.Tasty.Bench
1313import Streamly.Benchmark.Common hiding (benchPureSrc)
1414import qualified Stream.Common as P
1515
benchmark/Streamly/Benchmark/Data/Array/Stream.hs+3−3
@@ -39,7 +39,7 @@ import Prelude hiding ()
3939
4040import qualified Streamly.Data.Stream as Stream
4141import qualified Streamly.Internal.Data.Array as Array
42−import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
42+import qualified Streamly.Internal.Data.Array.Stream as ArrayStream
4343import qualified Streamly.Internal.Data.Fold as Fold
4444import qualified Streamly.Internal.Data.Parser as Parser
4545import qualified Streamly.Internal.Data.Stream as Stream
@@ -47,13 +47,13 @@ import qualified Streamly.Internal.Data.StreamK as StreamK
4747import qualified Streamly.Internal.FileSystem.Handle as Handle
4848import qualified Streamly.Internal.Unicode.Stream as Unicode
4949
50−import Gauge hiding (env)
50+import Test.Tasty.Bench hiding (env)
5151import Streamly.Benchmark.Common
5252import Streamly.Benchmark.Common.Handle
5353import Control.Monad.IO.Class (MonadIO)
5454
5555#ifdef INSPECTION
56−import Streamly.Internal.Data.Unbox (Unbox)
56+import Streamly.Internal.Data.MutByteArray (Unbox)
5757import Streamly.Internal.Data.Stream (Step(..))
5858import Test.Inspection
5959#endif
benchmark/Streamly/Benchmark/Data/Fold.hs+28−4
@@ -12,6 +12,11 @@
1212{-# OPTIONS_GHC -fno-warn-warnings-deprecations #-}
1313{-# OPTIONS_GHC -Wno-orphans #-}
1414
15+#undef FUSION_CHECK
16+#ifdef FUSION_CHECK
17+{-# OPTIONS_GHC -ddump-simpl -ddump-to-file -dsuppress-all #-}
18+#endif
19+
1520module Main (main) where
1621
1722import Control.DeepSeq (NFData(..))
@@ -36,7 +41,7 @@ import qualified Streamly.Internal.Data.Unfold as Unfold
3641import qualified Streamly.Internal.Data.Pipe as Pipe
3742import qualified Streamly.Internal.Data.Stream as Stream
3843
39−import Gauge
44+import Test.Tasty.Bench
4045import Streamly.Benchmark.Common
4146import Prelude hiding (all, any, take, unzip, sequence_, filter)
4247
@@ -264,13 +269,13 @@ unzip = Stream.fold $ FL.lmap (\a -> (a, a)) (FL.unzip FL.sum FL.length)
264269{-# INLINE unzipWithFstM #-}
265270unzipWithFstM :: Monad m => Stream m Int -> m (Int, Int)
266271unzipWithFstM = do
267− let f = \a -> return (a + 1, a)
272+ let f a = return (a + 1, a)
268273 Stream.fold (FL.unzipWithFstM f FL.sum FL.length)
269274
270275{-# INLINE unzipWithMinM #-}
271276unzipWithMinM :: Monad m => Stream m Int -> m (Int, Int)
272277unzipWithMinM = do
273− let f = \a -> return (a + 1, a)
278+ let f a = return (a + 1, a)
274279 Stream.fold (FL.unzipWithMinM f FL.sum FL.length)
275280
276281-------------------------------------------------------------------------------
@@ -502,7 +507,9 @@ o_n_heap_serial value =
502507-------------------------------------------------------------------------------
503508
504509main :: IO ()
505−main = runWithCLIOpts defaultStreamSize allBenchmarks
510+main = do
511+#ifndef FUSION_CHECK
512+ runWithCLIOpts defaultStreamSize allBenchmarks
506513
507514 where
508515
@@ -515,3 +522,20 @@ main = runWithCLIOpts defaultStreamSize allBenchmarks
515522 , bgroup (o_n_space_prefix moduleName) (o_n_space_serial value)
516523 , bgroup (o_n_heap_prefix moduleName) (o_n_heap_serial value)
517524 ]
525+#else
526+ -- Enable FUSION_CHECK macro at the beginning of the file
527+ -- Enable one benchmark below, and run the benchmark
528+ -- Check the .dump-simpl output
529+ let value = 100000
530+ let input = source value 1
531+ let getKey buckets = (`mod` buckets)
532+ let getFold k =
533+ return $ case k of
534+ 0 -> FL.sum
535+ 1 -> FL.length
536+ _ -> FL.length
537+
538+ -- demuxToMap (getKey 64) (getFold . getKey 64) input
539+ toIntMapIO (getKey 64) input
540+ return ()
541+#endif
benchmark/Streamly/Benchmark/Data/Fold/Window.hs+1−1
@@ -13,7 +13,7 @@ import qualified Streamly.Internal.Data.Ring as Ring
1313import qualified Streamly.Internal.Data.Stream as Stream
1414
1515import Streamly.Benchmark.Common
16−import Gauge
16+import Test.Tasty.Bench
1717
1818{-# INLINE source #-}
1919source :: (Monad m, Num a, Stream.Enumerable a) =>
benchmark/Streamly/Benchmark/Data/MutArray.hs+1−1
@@ -49,7 +49,7 @@ import qualified Streamly.Internal.Data.MutArray as MArray
4949import qualified Streamly.Internal.Data.Fold as Fold
5050import qualified Streamly.Internal.Data.Stream as Stream
5151
52−import Gauge
52+import Test.Tasty.Bench
5353import Streamly.Benchmark.Common hiding (benchPureSrc)
5454
5555type Stream = MutArray
benchmark/Streamly/Benchmark/Data/Parser.hs+7−6
@@ -49,7 +49,7 @@ import qualified Streamly.Data.Stream as Stream
4949import qualified Streamly.Internal.Data.Producer as Producer
5050import qualified Streamly.Internal.Data.Stream as Stream
5151
52−import Gauge hiding (env)
52+import Test.Tasty.Bench hiding (env)
5353import Streamly.Benchmark.Common
5454import Streamly.Benchmark.Common.Handle
5555
@@ -348,7 +348,7 @@ split_ value =
348348-- PR.dropWhile (<= (value * 1 `div` 4)) *> PR.die "alt"
349349{-# INLINE takeWhileFail #-}
350350takeWhileFail :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
351−takeWhileFail predicate (Fold fstep finitial fextract) =
351+takeWhileFail predicate (Fold fstep finitial _ ffinal) =
352352 Parser step initial extract
353353
354354 where
@@ -369,7 +369,7 @@ takeWhileFail predicate (Fold fstep finitial fextract) =
369369 Fold.Done b -> Done 0 b
370370 else return $ Error "fail"
371371
372− extract s = fmap (Done 0) (fextract s)
372+ extract s = fmap (Done 0) (ffinal s)
373373
374374{-# INLINE alt2 #-}
375375alt2 :: Monad m
@@ -714,8 +714,6 @@ instance NFData ParseError where
714714o_1_space_serial :: Int -> [Benchmark]
715715o_1_space_serial value =
716716 [ benchIOSink value "takeBetween" $ takeBetween value
717− , benchIOSink value "takeEQ" $ takeEQ value
718− , benchIOSink value "takeGE" $ takeGE value
719717 , benchIOSink value "takeWhile" $ takeWhile value
720718 , benchIOSink value "takeWhileP" $ takeWhileP value
721719 , benchIOSink value "takeP" $ takeP value
@@ -812,8 +810,11 @@ o_1_space_serial_unfold bound arrays =
812810o_n_heap_serial :: Int -> [Benchmark]
813811o_n_heap_serial value =
814812 [
813+ benchIOSink value "takeEQ" $ takeEQ value
814+ , benchIOSink value "takeGE" $ takeGE value
815+
815816 -- lookahead benchmark holds the entire input till end
816− benchIOSink value "lookAhead" $ lookAhead value
817+ , benchIOSink value "lookAhead" $ lookAhead value
817818
818819 -- o-n-heap because of backtracking
819820 , benchIOSrc sourceEscapedFrames value "takeFramedByEsc_"
benchmark/Streamly/Benchmark/Data/ParserK.hs+6−6
@@ -48,7 +48,7 @@ import qualified Streamly.Data.Parser as PRD
4848import qualified Streamly.Internal.Data.ParserK as PR
4949import qualified Streamly.Internal.Data.StreamK as StreamK
5050
51−import Gauge
51+import Test.Tasty.Bench
5252import Streamly.Benchmark.Common
5353
5454-------------------------------------------------------------------------------
@@ -63,7 +63,7 @@ import Streamly.Benchmark.Common
6363#define PARSE_ELEM (Array Int)
6464#define CONSTRAINT_IO (MonadIO m, Unbox a)
6565#define CONSTRAINT (Monad m, Unbox a)
66−#define MODULE_NAME "ParserK.Chunked"
66+#define MODULE_NAME "Data.ParserK.Chunked"
6767
6868#endif
6969
@@ -75,7 +75,7 @@ import Streamly.Benchmark.Common
7575#define PARSE_ELEM (Array Int)
7676#define CONSTRAINT_IO (MonadIO m)
7777#define CONSTRAINT (Monad m)
78−#define MODULE_NAME "ParserK.Chunked.Generic"
78+#define MODULE_NAME "Data.ParserK.Chunked.Generic"
7979
8080#endif
8181
@@ -87,7 +87,7 @@ import Streamly.Benchmark.Common
8787#define PARSE_ELEM Int
8888#define CONSTRAINT_IO (MonadIO m)
8989#define CONSTRAINT (Monad m)
90−#define MODULE_NAME "ParserK"
90+#define MODULE_NAME "Data.ParserK"
9191
9292#endif
9393
@@ -213,7 +213,7 @@ sequence_ value =
213213
214214{-# INLINE takeWhileFailD #-}
215215takeWhileFailD :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
216−takeWhileFailD predicate (Fold fstep finitial fextract) =
216+takeWhileFailD predicate (Fold fstep finitial _ ffinal) =
217217 Parser step initial extract
218218
219219 where
@@ -234,7 +234,7 @@ takeWhileFailD predicate (Fold fstep finitial fextract) =
234234 Fold.Done b -> Done 0 b
235235 else return $ Error "fail"
236236
237− extract s = fmap (Done 0) (fextract s)
237+ extract s = fmap (Done 0) (ffinal s)
238238
239239{-# INLINE takeWhileFail #-}
240240takeWhileFail :: CONSTRAINT =>
benchmark/Streamly/Benchmark/Data/Ring/Unboxed.hs+1−1
@@ -16,7 +16,7 @@ import qualified Streamly.Internal.Data.Array as Array
1616import qualified Streamly.Internal.Data.Ring as Ring
1717import qualified Data.Foldable as P
1818
19−import Gauge
19+import Test.Tasty.Bench
2020import Streamly.Benchmark.Common
2121
2222import Prelude as P
benchmark/Streamly/Benchmark/Data/Serialize.hs+89−53
@@ -1,4 +1,5 @@
11{-# LANGUAGE TemplateHaskell #-}
2+{- HLINT ignore -}
23
34#undef FUSION_CHECK
45#ifdef FUSION_CHECK
@@ -18,43 +19,40 @@ module Main (main) where
1819-------------------------------------------------------------------------------
1920
2021import Control.DeepSeq (NFData(..), deepseq)
22+import Control.Monad (when, void)
2123import GHC.Generics (Generic)
2224import System.Random (randomRIO)
2325#ifndef USE_UNBOX
2426import Test.QuickCheck (Arbitrary, arbitrary)
2527#endif
2628
29+import Streamly.Internal.Data.MutByteArray
30+import qualified Streamly.Internal.Data.MutByteArray as MBA
2731#ifdef USE_UNBOX
2832import Data.Proxy (Proxy(..))
29−import Streamly.Internal.Data.Unbox
3033#else
3134import Control.DeepSeq (force)
3235import Test.QuickCheck (oneof, generate)
33−import Streamly.Internal.Data.Unbox (newBytes, MutableByteArray)
34−import Streamly.Internal.Data.Serialize hiding (encode)
3536#endif
3637
37−#ifdef USE_TH
38−#ifdef USE_UNBOX
39−import Streamly.Internal.Data.Unbox.TH
40−#else
41−import Streamly.Internal.Data.Serialize.TH
42−#endif
43−#endif
44−
45−import Gauge
38+import Test.Tasty.Bench
4639import Streamly.Benchmark.Common
4740
41+#ifndef USE_UNBOX
42+import Streamly.Benchmark.Data.Serialize.RecCompatible
43+import Streamly.Benchmark.Data.Serialize.RecNonCompatible
44+#endif
45+
4846#ifdef USE_UNBOX
4947#define SERIALIZE_CLASS Unbox
50−#define DERIVE_CLASS deriveUnbox
51−#define SERIALIZE_OP pokeByteIndex
52−#define DESERIALIZE_OP peekByteIndex
48+#define DERIVE_CLASS(typ) $(deriveUnbox [d|instance Unbox typ|])
49+#define SERIALIZE_OP pokeAt
50+#define DESERIALIZE_OP peekAt
5351#else
5452#define SERIALIZE_CLASS Serialize
55−#define DERIVE_CLASS deriveSerialize
56−#define SERIALIZE_OP serialize
57−#define DESERIALIZE_OP deserialize
53+#define DERIVE_CLASS(typ) $(deriveSerialize [d|instance Serialize typ|])
54+#define SERIALIZE_OP serializeAt
55+#define DESERIALIZE_OP deserializeAt
5856#endif
5957
6058-------------------------------------------------------------------------------
@@ -67,7 +65,7 @@ data Unit = Unit
6765#ifndef USE_TH
6866instance SERIALIZE_CLASS Unit
6967#else
70−$(DERIVE_CLASS ''Unit)
68+DERIVE_CLASS(Unit)
7169#endif
7270
7371instance NFData Unit where
@@ -82,7 +80,7 @@ data Sum2
8280#ifndef USE_TH
8381instance SERIALIZE_CLASS Sum2
8482#else
85−$(DERIVE_CLASS ''Sum2)
83+DERIVE_CLASS(Sum2)
8684#endif
8785
8886instance NFData Sum2 where
@@ -120,7 +118,7 @@ data Sum25
120118#ifndef USE_TH
121119instance SERIALIZE_CLASS Sum25
122120#else
123−$(DERIVE_CLASS ''Sum25)
121+DERIVE_CLASS(Sum25)
124122#endif
125123
126124instance NFData Sum25 where
@@ -159,7 +157,7 @@ data Product25
159157#ifndef USE_TH
160158instance SERIALIZE_CLASS Product25
161159#else
162−$(DERIVE_CLASS ''Product25)
160+DERIVE_CLASS(Product25)
163161#endif
164162
165163instance NFData Product25 where
@@ -239,7 +237,7 @@ data CustomDT1
239237#ifndef USE_TH
240238instance SERIALIZE_CLASS CustomDT1
241239#else
242−$(DERIVE_CLASS ''CustomDT1)
240+DERIVE_CLASS(CustomDT1)
243241#endif
244242
245243instance NFData CustomDT1 where
@@ -262,19 +260,20 @@ data BinTree a
262260#ifndef USE_TH
263261instance Serialize (BinTree a)
264262#else
265−$(deriveSerialize ''BinTree)
263+$(deriveSerialize [d|instance Serialize a => Serialize (BinTree a)|])
266264#endif
267265
268266instance NFData a => NFData (BinTree a) where
269267 {-# INLINE rnf #-}
270268 rnf (Leaf a) = rnf a `seq` ()
271− rnf (Tree l r) = rnf l `seq` rnf r `seq` ()
269+ rnf (Tree l r) = rnf l `seq` rnf r
272270
271+-- XXX This may not terminate, or could become really large.
273272instance Arbitrary a => Arbitrary (BinTree a) where
274273 arbitrary = oneof [Leaf <$> arbitrary, Tree <$> arbitrary <*> arbitrary]
275274
276275mkBinTree :: (Arbitrary a) => Int -> IO (BinTree a)
277−mkBinTree = go (generate $ arbitrary)
276+mkBinTree = go (generate arbitrary)
278277
279278 where
280279
@@ -292,37 +291,30 @@ getSize :: forall a. SERIALIZE_CLASS a => a -> Int
292291#ifdef USE_UNBOX
293292getSize _ = sizeOf (Proxy :: Proxy a)
294293#else
295−getSize val =
296− case size :: Size a of
297− Size f -> f 0 val
294+getSize = addSizeTo 0
298295#endif
299296
300297-------------------------------------------------------------------------------
301298-- Common helpers
302299-------------------------------------------------------------------------------
303300
301+{- HLINT ignore "Eta reduce" -}
304302-- Parts of "f" that are dependent on val will not be optimized out.
305303{-# INLINE loop #-}
306304loop :: Int -> (a -> IO b) -> a -> IO ()
307305loop count f val = go count val
308306 where
309307
310− go n x = do
311− if n > 0
312− then f x >> go (n-1) x
313− else return ()
308+ go n x = when (n > 0) $ f x >> go (n-1) x
314309
315310-- The first arg of "f" is the environment which is not threaded around in the
316311-- loop.
317312{-# INLINE loopWith #-}
318−loopWith :: Int -> (env -> a -> IO b) -> env -> a -> IO ()
313+loopWith :: Int -> (e -> a -> IO b) -> e -> a -> IO ()
319314loopWith count f e val = go count val
320315 where
321316
322− go n x = do
323− if n > 0
324− then f e x >> go (n-1) x
325− else return ()
317+ go n x = when (n > 0) $ f e x >> go (n-1) x
326318
327319benchSink :: NFData b => String -> Int -> (Int -> IO b) -> Benchmark
328320benchSink name times f = bench name (nfIO (randomRIO (times, times) >>= f))
@@ -333,7 +325,7 @@ benchSink name times f = bench name (nfIO (randomRIO (times, times) >>= f))
333325
334326{-
335327{-# INLINE pokeWithSize #-}
336−pokeWithSize :: SERIALIZE_CLASS a => MutableByteArray -> a -> IO ()
328+pokeWithSize :: SERIALIZE_CLASS a => MutByteArray -> a -> IO ()
337329pokeWithSize arr val = do
338330 let n = getSize val
339331 n `seq` SERIALIZE_OP 0 arr val >> return ()
@@ -342,27 +334,27 @@ pokeWithSize arr val = do
342334pokeTimesWithSize :: SERIALIZE_CLASS a => a -> Int -> IO ()
343335pokeTimesWithSize val times = do
344336 let n = getSize val
345− arr <- newBytes n
337+ arr <- MBA.new n
346338 loopWith times pokeWithSize arr val
347339-}
348340
349341{-# INLINE poke #-}
350−poke :: SERIALIZE_CLASS a => MutableByteArray -> a -> IO ()
351−poke arr val = SERIALIZE_OP 0 arr val >> return ()
342+poke :: SERIALIZE_CLASS a => MutByteArray -> a -> IO ()
343+poke arr val = void (SERIALIZE_OP 0 arr val)
352344
353345{-# INLINE pokeTimes #-}
354346pokeTimes :: SERIALIZE_CLASS a => a -> Int -> IO ()
355347pokeTimes val times = do
356348 let n = getSize val
357− arr <- newBytes n
349+ arr <- MBA.new n
358350 loopWith times poke arr val
359351
360352{-# INLINE encode #-}
361353encode :: SERIALIZE_CLASS a => a -> IO ()
362354encode val = do
363355 let n = getSize val
364− arr <- newBytes n
365− SERIALIZE_OP 0 arr val >> return ()
356+ arr <- MBA.new n
357+ void (SERIALIZE_OP 0 arr val)
366358
367359{-# INLINE encodeTimes #-}
368360encodeTimes :: SERIALIZE_CLASS a => a -> Int -> IO ()
@@ -370,7 +362,7 @@ encodeTimes val times = loop times encode val
370362
371363{-# INLINE peek #-}
372364peek :: forall a. (NFData a, SERIALIZE_CLASS a) =>
373− (a, Int) -> MutableByteArray -> IO ()
365+ (a, Int) -> MutByteArray -> IO ()
374366#ifdef USE_UNBOX
375367peek (_val, _) arr = do
376368 (val1 :: a) <- DESERIALIZE_OP 0 arr
@@ -397,7 +389,7 @@ peek (_val, n) arr = do
397389{-# INLINE peekTimes #-}
398390peekTimes :: (NFData a, SERIALIZE_CLASS a) => Int -> a -> Int -> IO ()
399391peekTimes n val times = do
400− arr <- newBytes n
392+ arr <- MBA.new n
401393 _ <- SERIALIZE_OP 0 arr val
402394 loopWith times peek (val, n) arr
403395
@@ -405,7 +397,7 @@ peekTimes n val times = do
405397trip :: forall a. (NFData a, SERIALIZE_CLASS a) => a -> IO ()
406398trip val = do
407399 let n = getSize val
408− arr <- newBytes n
400+ arr <- MBA.new n
409401 _ <- SERIALIZE_OP 0 arr val
410402#ifdef USE_UNBOX
411403 (val1 :: a) <- DESERIALIZE_OP 0 arr
@@ -476,6 +468,41 @@ benchVar gname f tInt lInt times =
476468 , let !n = getSize lInt
477469 in benchSink "list-int" times (f n lInt)
478470 ]
471+
472+{-
473+{-# INLINE benchGeneric #-}
474+benchGeneric
475+ :: (NFData b, Serialize b)
476+ => String
477+ -> String
478+ -> (forall a. (NFData a, Serialize a) =>
479+ Int -> a -> Int -> IO ())
480+ -> b
481+ -> Int
482+ -> Benchmark
483+benchGeneric gname valId f val times =
484+ bgroup gname
485+ [ let !n = getSize val
486+ in benchSink valId times (f n val)
487+ ]
488+-}
489+
490+{-# INLINE benchLargeRec #-}
491+benchLargeRec ::
492+ String
493+ -> (forall a. (NFData a, Serialize a) =>
494+ Int -> a -> Int -> IO ())
495+ -> RecNonCompatible
496+ -> RecCompatible
497+ -> Int
498+ -> Benchmark
499+benchLargeRec gname f recL recR times =
500+ bgroup gname
501+ [ let !n = getSize recL
502+ in benchSink "RecNonCompatible" times (f n recL)
503+ , let !n = getSize recR
504+ in benchSink "RecCompatible" times (f n recR)
505+ ]
479506#endif
480507
481508-- Times is scaled by the number of constructors to normalize
@@ -483,8 +510,9 @@ benchVar gname f tInt lInt times =
483510allBenchmarks :: Int -> [Benchmark]
484511allBenchmarks times =
485512#else
486−allBenchmarks :: BinTree Int -> [Int] -> Int -> [Benchmark]
487−allBenchmarks tInt lInt times =
513+allBenchmarks ::
514+ BinTree Int -> [Int] -> RecNonCompatible -> RecCompatible -> Int -> [Benchmark]
515+allBenchmarks tInt lInt recL recR times =
488516#endif
489517 [ bgroup "sizeOf"
490518 [
@@ -504,6 +532,10 @@ allBenchmarks tInt lInt times =
504532 , benchVar "encode" (const encodeTimes) tInt lInt 1
505533 , benchVar "peek" peekTimes tInt lInt 1
506534 , benchVar "roundtrip" (const roundtrip) tInt lInt 1
535+ , benchLargeRec "poke" (const pokeTimes) recL recR times
536+ , benchLargeRec "encode" (const encodeTimes) recL recR times
537+ , benchLargeRec "peek" peekTimes recL recR times
538+ , benchLargeRec "roundtrip" (const roundtrip) recL recR times
507539#endif
508540 ]
509541
@@ -519,18 +551,22 @@ main = do
519551 -- nodes 1 level = log_2 (100001/3) + 1 = 16
520552 !(tInt :: BinTree Int) <- force <$> mkBinTree 16
521553
554+ let !recL = valRecNonCompatible
555+ let !recR = valRecCompatible
556+
522557 -- Approximately 100000 constructors, assuming two constructors (Cons, Int)
523558 -- per element.
524559 let lInt = [1..50000 :: Int]
525560 let !len = length lInt -- evaluate the list
526561#endif
527562#ifndef FUSION_CHECK
528− -- This can take too much memory/CPU, need to restrict the test
529− -- runQC
530563#ifdef USE_UNBOX
531564 runWithCLIOpts defaultStreamSize allBenchmarks
532565#else
533− len `seq` runWithCLIOpts defaultStreamSize (allBenchmarks tInt lInt)
566+ len
567+ `seq` runWithCLIOpts
568+ defaultStreamSize
569+ (allBenchmarks tInt lInt recL recR)
534570#endif
535571#else
536572 -- Enable FUSION_CHECK macro at the beginning of the file
benchmark/Streamly/Benchmark/Data/Serialize/RecCompatible.hsnew file+24
@@ -0,0 +1,24 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Streamly.Benchmark.Data.Serialize.RecCompatible
4+ ( RecCompatible(..)
5+ , valRecCompatible
6+ ) where
7+
8+--------------------------------------------------------------------------------
9+-- Imports
10+--------------------------------------------------------------------------------
11+
12+import Streamly.Internal.Data.MutByteArray (Serialize)
13+import Streamly.Benchmark.Data.Serialize.TH (genLargeRecord)
14+
15+import qualified Streamly.Internal.Data.MutByteArray as Serialize
16+
17+--------------------------------------------------------------------------------
18+-- Code
19+--------------------------------------------------------------------------------
20+
21+$(genLargeRecord "RecCompatible" 50)
22+$(Serialize.deriveSerializeWith
23+ (Serialize.encodeRecordFields True)
24+ [d|instance Serialize RecCompatible|])
benchmark/Streamly/Benchmark/Data/Serialize/RecNonCompatible.hsnew file+21
@@ -0,0 +1,21 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Streamly.Benchmark.Data.Serialize.RecNonCompatible
4+ ( RecNonCompatible(..)
5+ , valRecNonCompatible
6+ ) where
7+
8+--------------------------------------------------------------------------------
9+-- Imports
10+--------------------------------------------------------------------------------
11+
12+import Streamly.Benchmark.Data.Serialize.TH (genLargeRecord)
13+
14+import qualified Streamly.Internal.Data.MutByteArray as Serialize
15+
16+--------------------------------------------------------------------------------
17+-- Code
18+--------------------------------------------------------------------------------
19+
20+$(genLargeRecord "RecNonCompatible" 50)
21+$(Serialize.deriveSerialize [d|instance Serialize.Serialize RecNonCompatible|])
benchmark/Streamly/Benchmark/Data/Serialize/TH.hsnew file+76
@@ -0,0 +1,76 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+module Streamly.Benchmark.Data.Serialize.TH (genLargeRecord) where
4+
5+--------------------------------------------------------------------------------
6+-- Imports
7+--------------------------------------------------------------------------------
8+
9+import Language.Haskell.TH
10+
11+import Streamly.Internal.Data.MutByteArray (makeI)
12+import Control.DeepSeq (NFData(..))
13+
14+--------------------------------------------------------------------------------
15+-- Large Record
16+--------------------------------------------------------------------------------
17+
18+genLargeRecord :: String -> Int -> Q [Dec]
19+genLargeRecord tyName numFields =
20+ sequence
21+ [ dataD
22+ (pure [])
23+ (mkName tyName)
24+ []
25+ Nothing
26+ [mkCon tyName]
27+ [derivClause Nothing [conT ''Eq, conT ''Show]]
28+ , mkValueSigDec
29+ , mkValueDec
30+ , nfDataInstance tyName
31+ ]
32+
33+ where
34+
35+ fieldTypeChoices = [conT ''()]
36+ chooseCycle i xs = xs !! (i `mod` length xs)
37+ nfDataInstance nm =
38+ instanceD
39+ (pure [])
40+ (appT (conT ''NFData) (conT (mkName nm)))
41+ [ funD
42+ 'rnf
43+ [ clause
44+ [ conP
45+ (mkName nm)
46+ (varP . makeI <$> [0 .. (numFields - 1)])
47+ ]
48+ (normalB
49+ (foldl
50+ (\b a -> [|rnf $(b) `seq` rnf $(a)|])
51+ [|()|]
52+ (varE . makeI <$> [0 .. (numFields - 1)])))
53+ []
54+ ]
55+ ]
56+ valueName = mkName $ "val" ++ tyName
57+ mkValueSigDec = sigD valueName [t|$(conT (mkName tyName))|]
58+ mkValueDec =
59+ funD
60+ valueName
61+ [ clause
62+ []
63+ (normalB
64+ (foldl
65+ (\b a -> [|$(b) $(a)|])
66+ (conE (mkName tyName))
67+ (conE '() <$ [0 .. (numFields - 1)])))
68+ []
69+ ]
70+ mkCon nm = recC (mkName nm) (mkField <$> [0 .. (numFields - 1)])
71+ mkField i =
72+ varBangType
73+ (mkName ("field" ++ show i))
74+ (bangType
75+ (bang noSourceUnpackedness noSourceStrictness)
76+ (chooseCycle i fieldTypeChoices))
benchmark/Streamly/Benchmark/Data/Stream/Common.hs+3−1
@@ -71,7 +71,9 @@ module Stream.Common
7171 )
7272where
7373
74+#if !MIN_VERSION_base(4,18,0)
7475import Control.Applicative (liftA2)
76+#endif
7577import Control.DeepSeq (NFData)
7678import Control.Exception (try)
7779import GHC.Exception (ErrorCall)
@@ -102,7 +104,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
102104#endif
103105#endif
104106
105−import Gauge
107+import Test.Tasty.Bench
106108import Prelude hiding (Foldable(..), mapM, replicate)
107109
108110#ifdef USE_STREAMK
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentCommon.hs+3−3
@@ -14,14 +14,14 @@ where
1414import Stream.Common
1515 (composeN, benchIO, benchIOSink, benchIOSrc, sourceUnfoldrM)
1616import Streamly.Data.Stream (Stream)
17−import Streamly.Internal.Data.Stream.Concurrent (MonadAsync, Config)
17+import Streamly.Internal.Data.Stream.Prelude (MonadAsync, Config)
1818
1919import qualified Data.List as List
2020import qualified Streamly.Data.Fold as Fold
2121import qualified Streamly.Data.Stream as Stream
22−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
22+import qualified Streamly.Internal.Data.Stream.Prelude as Async
2323
24−import Gauge
24+import Test.Tasty.Bench
2525import Prelude hiding (mapM)
2626import Streamly.Benchmark.Common
2727
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentEager.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
12+import qualified Streamly.Internal.Data.Stream.Prelude as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentEager"
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentInterleaved.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
12+import qualified Streamly.Internal.Data.Stream.Prelude as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentInterleaved"
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentOrdered.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
12+import qualified Streamly.Internal.Data.Stream.Prelude as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentOrdered"
benchmark/Streamly/Benchmark/Data/Stream/Eliminate.hs+3−1
@@ -57,7 +57,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
5757#endif
5858#endif
5959
60−import Gauge
60+import Test.Tasty.Bench
6161import Streamly.Benchmark.Common
6262import Prelude hiding (length, sum, or, and, any, all, notElem, elem, (!!),
6363 lookup, repeat, minimum, maximum, product, last, mapM_, init)
@@ -132,12 +132,14 @@ foldableAny :: Int -> Int -> Bool
132132foldableAny value n =
133133 Prelude.any (> (value + 1)) (sourceUnfoldr value n :: Stream Identity Int)
134134
135+{- HLINT ignore "Use all"-}
135136{-# INLINE foldableAnd #-}
136137foldableAnd :: Int -> Int -> Bool
137138foldableAnd value n =
138139 Prelude.and $ fmap
139140 (<= (value + 1)) (sourceUnfoldr value n :: Stream Identity Int)
140141
142+{- HLINT ignore "Use any"-}
141143{-# INLINE foldableOr #-}
142144foldableOr :: Int -> Int -> Bool
143145foldableOr value n =
benchmark/Streamly/Benchmark/Data/Stream/Exceptions.hs+12−3
@@ -36,21 +36,22 @@ import System.IO (Handle, hClose, hPutChar)
3636import qualified Streamly.FileSystem.Handle as FH
3737import qualified Streamly.Internal.FileSystem.Handle as IFH
3838import qualified Streamly.Internal.Data.Unfold as IUF
39−import qualified Streamly.Internal.Data.Unfold.Exception as IUF
39+import qualified Streamly.Internal.Data.Unfold.Prelude as IUF
4040
4141#ifdef USE_PRELUDE
4242import qualified Streamly.Internal.Data.Stream.IsStream as Stream
4343#else
4444import qualified Streamly.Internal.Data.Stream as Stream
45−import qualified Streamly.Internal.Data.Stream.Exception.Lifted as Stream
45+import qualified Streamly.Internal.Data.Stream.Prelude as Stream
4646#endif
4747
48−import Gauge hiding (env)
48+import Test.Tasty.Bench hiding (env)
4949import Prelude hiding (last, length)
5050import Streamly.Benchmark.Common
5151import Streamly.Benchmark.Common.Handle
5252
5353#ifdef INSPECTION
54+import Control.Monad.Catch (MonadCatch)
5455import Test.Inspection
5556
5657import qualified Streamly.Internal.Data.Stream as D
@@ -286,8 +287,12 @@ readChunksOnException inh devNull =
286287 in IUF.fold (IFH.writeChunks devNull) readEx inh
287288
288289#ifdef INSPECTION
290+#if __GLASGOW_HASKELL__ >= 906
291+inspect $ hasNoTypeClassesExcept 'readChunksOnException [''MonadCatch]
292+#else
289293inspect $ hasNoTypeClasses 'readChunksOnException
290294#endif
295+#endif
291296
292297-- | Send the file contents to /dev/null with exception handling
293298readChunksBracket_ :: Handle -> Handle -> IO ()
@@ -296,8 +301,12 @@ readChunksBracket_ inh devNull =
296301 in IUF.fold (IFH.writeChunks devNull) readEx inh
297302
298303#ifdef INSPECTION
304+#if __GLASGOW_HASKELL__ >= 906
305+inspect $ hasNoTypeClassesExcept 'readChunksBracket_ [''MonadCatch]
306+#else
299307inspect $ hasNoTypeClasses 'readChunksBracket_
300308#endif
309+#endif
301310
302311readChunksBracket :: Handle -> Handle -> IO ()
303312readChunksBracket inh devNull =
benchmark/Streamly/Benchmark/Data/Stream/Expand.hs+9−1
@@ -54,7 +54,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
5454#endif
5555#endif
5656
57−import Gauge
57+import Test.Tasty.Bench
5858import Stream.Common
5959import Streamly.Benchmark.Common
6060import Prelude hiding (concatMap)
@@ -198,7 +198,11 @@ concatMapPure outer inner n =
198198 (fromStream $ sourceUnfoldr outer n)
199199
200200#ifdef INSPECTION
201+#if __GLASGOW_HASKELL__ >= 906
202+inspect $ hasNoTypeClassesExcept 'concatMapPure [''Applicative]
203+#else
201204inspect $ hasNoTypeClasses 'concatMapPure
205+#endif
202206inspect $ 'concatMapPure `hasNoType` ''SPEC
203207#endif
204208
@@ -211,7 +215,11 @@ concatMapRepl outer inner n =
211215 (fromStream . S.replicate inner) (fromStream $ sourceUnfoldrM outer n)
212216
213217#ifdef INSPECTION
218+#if __GLASGOW_HASKELL__ >= 906
219+inspect $ hasNoTypeClassesExcept 'concatMapRepl [''Applicative]
220+#else
214221inspect $ hasNoTypeClasses 'concatMapRepl
222+#endif
215223inspect $ 'concatMapRepl `hasNoType` ''SPEC
216224#endif
217225
benchmark/Streamly/Benchmark/Data/Stream/Generate.hs+1−1
@@ -40,7 +40,7 @@ import qualified Stream.Common as Common
4040#endif
4141#endif
4242
43−import Gauge
43+import Test.Tasty.Bench
4444import Streamly.Benchmark.Common
4545import qualified Prelude
4646
benchmark/Streamly/Benchmark/Data/Stream/Lift.hs+1−1
@@ -37,7 +37,7 @@ import qualified Control.Monad.State.Strict as State
3737#endif
3838#endif
3939
40−import Gauge
40+import Test.Tasty.Bench
4141import Streamly.Benchmark.Common
4242import Prelude hiding (reverse, tail)
4343
benchmark/Streamly/Benchmark/Data/Stream/Reduce.hs+2−2
@@ -48,7 +48,7 @@ import Streamly.Internal.Data.Stream (Stream)
4848import qualified Streamly.Internal.Data.Stream as S
4949#ifndef USE_STREAMLY_CORE
5050import qualified Streamly.Data.Stream.Prelude as S
51−import qualified Streamly.Internal.Data.Stream.Time as S
51+import qualified Streamly.Internal.Data.Stream.Prelude as S
5252#endif
5353
5454#ifdef USE_STREAMK
@@ -61,7 +61,7 @@ import qualified Streamly.Internal.Data.Stream as K
6161
6262#endif
6363
64−import Gauge
64+import Test.Tasty.Bench
6565import Streamly.Benchmark.Common
6666import Stream.Common
6767import Prelude hiding (reverse, tail)
benchmark/Streamly/Benchmark/Data/Stream/Split.hs+1−1
@@ -35,7 +35,7 @@ import qualified Streamly.Internal.FileSystem.Handle as IFH
3535import qualified Streamly.Internal.Unicode.Stream as IUS
3636import qualified Streamly.Prelude as S
3737
38−import Gauge hiding (env)
38+import Test.Tasty.Bench hiding (env)
3939import Prelude hiding (last, length)
4040import Streamly.Benchmark.Common
4141import Streamly.Benchmark.Common.Handle
benchmark/Streamly/Benchmark/Data/Stream/StreamD.hs+3−1
@@ -22,10 +22,12 @@
2222
2323module Main (main) where
2424
25+#if !MIN_VERSION_base(4,18,0)
2526import Control.Applicative (liftA2)
27+#endif
2628import Control.Monad (when)
2729import Data.Maybe (isJust)
28−import Gauge (bench, nfIO, bgroup, Benchmark, defaultMain)
30+import Test.Tasty.Bench (bench, nfIO, bgroup, Benchmark, defaultMain)
2931import System.Random (randomRIO)
3032import Prelude hiding (tail, mapM_, foldl, last, map, mapM, concatMap, zip)
3133
benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs+3−1
@@ -21,7 +21,9 @@
2121
2222module Main (main) where
2323
24+#if !MIN_VERSION_base(4,18,0)
2425import Control.Applicative (liftA2)
26+#endif
2527import Control.Monad (when)
2628import Data.Maybe (isJust)
2729import System.Random (randomRIO)
@@ -35,7 +37,7 @@ import qualified Data.List as List
3537
3638import qualified Streamly.Internal.Data.StreamK as S
3739
38−import Gauge (bench, nfIO, bgroup, Benchmark, defaultMain)
40+import Test.Tasty.Bench (bench, nfIO, bgroup, Benchmark, defaultMain)
3941
4042import Streamly.Benchmark.Common
4143
benchmark/Streamly/Benchmark/Data/Stream/StreamKAlt.hs+2−2
@@ -23,7 +23,7 @@ import qualified Streamly.Internal.Data.StreamK.Alt as S
2323-- import qualified Streamly.Internal.Data.SVar as S
2424
2525import Streamly.Benchmark.Common
26−import Gauge (bgroup, Benchmark, defaultMain)
26+import Test.Tasty.Bench (bgroup, Benchmark, defaultMain)
2727
2828value :: Int
2929value = 100000
@@ -452,4 +452,4 @@ o_1_space =
452452 ]
453453
454454main :: IO ()
455−main = defaultMain $ concat [o_1_space]
455+main = defaultMain [o_1_space]
benchmark/Streamly/Benchmark/Data/Stream/ToStreamK.hs+2−2
@@ -37,7 +37,7 @@ import qualified Streamly.Internal.Data.StreamK as S
3737import qualified Streamly.Internal.Data.Stream as D
3838import qualified Streamly.Internal.Data.Fold as Fold
3939
40−import Gauge (bgroup, Benchmark, defaultMain)
40+import Test.Tasty.Bench (bgroup, Benchmark, defaultMain)
4141
4242import Streamly.Benchmark.Common
4343
@@ -120,7 +120,7 @@ fromFoldable streamLen n = S.fromFoldable [n..n+streamLen]
120120{-# INLINE fromFoldableM #-}
121121fromFoldableM :: Monad m => Int -> Int -> Stream m Int
122122fromFoldableM streamLen n =
123− Prelude.foldr S.consM S.nil (Prelude.fmap return [n..n+streamLen])
123+ Prelude.foldr (S.consM . return) S.nil [n .. n + streamLen]
124124
125125{-
126126{-# INLINABLE concatMapFoldableWith #-}
benchmark/Streamly/Benchmark/Data/Stream/Transform.hs+2−2
@@ -46,7 +46,7 @@ import Streamly.Internal.Data.Time.Units
4646import Streamly.Internal.Data.Stream (Stream)
4747import qualified Streamly.Internal.Data.Stream as Stream
4848#ifndef USE_STREAMLY_CORE
49−import qualified Streamly.Internal.Data.Stream.Time as Stream
49+import qualified Streamly.Internal.Data.Stream.Prelude as Stream
5050#endif
5151#ifdef USE_STREAMK
5252import Control.DeepSeq (NFData(..))
@@ -58,7 +58,7 @@ import qualified Streamly.Internal.Data.StreamK as StreamK
5858#endif
5959#endif
6060
61−import Gauge
61+import Test.Tasty.Bench
6262import Stream.Common hiding (scanl')
6363import Streamly.Benchmark.Common
6464import Prelude hiding (sequence, mapM)
benchmark/Streamly/Benchmark/Data/Unfold.hs+11−2
@@ -31,17 +31,18 @@ import qualified Prelude
3131import qualified Streamly.FileSystem.Handle as FH
3232import qualified Streamly.Internal.Data.Fold as FL
3333import qualified Streamly.Internal.Data.Unfold as UF
34−import qualified Streamly.Internal.Data.Unfold.Exception as UF
34+import qualified Streamly.Internal.Data.Unfold.Prelude as UF
3535import qualified Streamly.Internal.Data.Stream as S
3636import qualified Streamly.Internal.Data.Stream as D
3737import qualified Streamly.Internal.Data.StreamK as K
3838
39−import Gauge hiding (env)
39+import Test.Tasty.Bench hiding (env)
4040import Prelude hiding (take, filter, zipWith, map, mapM, takeWhile)
4141import Streamly.Benchmark.Common
4242import Streamly.Benchmark.Common.Handle
4343
4444#ifdef INSPECTION
45+import Control.Monad.Catch (MonadCatch)
4546import Test.Inspection
4647#endif
4748
@@ -776,7 +777,11 @@ readWriteFinally_Unfold inh devNull =
776777 in S.fold (FH.write devNull) $ S.unfold readEx inh
777778
778779#ifdef INSPECTION
780+#if __GLASGOW_HASKELL__ >= 906
781+inspect $ hasNoTypeClassesExcept 'readWriteFinally_Unfold [''MonadCatch]
782+#else
779783inspect $ hasNoTypeClasses 'readWriteFinally_Unfold
784+#endif
780785-- inspect $ 'readWriteFinallyUnfold `hasNoType` ''Step
781786#endif
782787
@@ -792,7 +797,11 @@ readWriteBracket_Unfold inh devNull =
792797 in S.fold (FH.write devNull) $ S.unfold readEx inh
793798
794799#ifdef INSPECTION
800+#if __GLASGOW_HASKELL__ >= 906
801+inspect $ hasNoTypeClassesExcept 'readWriteBracket_Unfold [''MonadCatch]
802+#else
795803inspect $ hasNoTypeClasses 'readWriteBracket_Unfold
804+#endif
796805-- inspect $ 'readWriteBracketUnfold `hasNoType` ''Step
797806#endif
798807
benchmark/Streamly/Benchmark/FileSystem/Handle.hs+1−1
@@ -22,7 +22,7 @@ import Streamly.Benchmark.Common.Handle (mkHandleBenchEnv)
2222import qualified Handle.ReadWrite as RW
2323import qualified Handle.Read as RO
2424
25−import Gauge hiding (env)
25+import Test.Tasty.Bench hiding (env)
2626import Prelude hiding (last, length)
2727import Streamly.Benchmark.Common
2828
benchmark/Streamly/Benchmark/FileSystem/Handle/Read.hs+8−9
@@ -33,7 +33,6 @@ import qualified Streamly.Data.Stream as Stream
3333import qualified Streamly.Data.Fold as Fold
3434import qualified Streamly.FileSystem.Handle as FH
3535import qualified Streamly.Internal.Data.Array as A
36−import qualified Streamly.Internal.Data.Array as AT
3736import qualified Streamly.Internal.Data.Fold as FL
3837import qualified Streamly.Internal.Data.Stream as IP
3938import qualified Streamly.Internal.FileSystem.Handle as IFH
@@ -41,14 +40,14 @@ import qualified Streamly.Internal.Unicode.Stream as IUS
4140import qualified Streamly.Data.Stream.Prelude as S
4241import qualified Streamly.Unicode.Stream as SS
4342
44−import Gauge hiding (env)
43+import Test.Tasty.Bench hiding (env)
4544import Prelude hiding (last, length)
4645import Streamly.Benchmark.Common.Handle
4746
4847#ifdef INSPECTION
4948import Streamly.Internal.Data.Stream (Step(..), FoldMany)
5049
51−import qualified Streamly.Internal.Data.MutArray as MA
50+import qualified Streamly.Internal.Data.MutArray as MutArray
5251import qualified Streamly.Internal.Data.Stream as D
5352import qualified Streamly.Internal.Data.Unfold as IUF
5453
@@ -69,7 +68,7 @@ readLast = S.fold Fold.last . S.unfold FH.reader
6968inspect $ hasNoTypeClasses 'readLast
7069inspect $ 'readLast `hasNoType` ''Step -- S.unfold
7170inspect $ 'readLast `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
72−inspect $ 'readLast `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
71+inspect $ 'readLast `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
7372#endif
7473
7574-- assert that flattenArrays constructors are not present
@@ -81,7 +80,7 @@ readCountBytes = S.fold Fold.length . S.unfold FH.reader
8180inspect $ hasNoTypeClasses 'readCountBytes
8281inspect $ 'readCountBytes `hasNoType` ''Step -- S.unfold
8382inspect $ 'readCountBytes `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
84−inspect $ 'readCountBytes `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
83+inspect $ 'readCountBytes `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
8584#endif
8685
8786-- | Count the number of lines in a file.
@@ -96,7 +95,7 @@ readCountLines =
9695inspect $ hasNoTypeClasses 'readCountLines
9796inspect $ 'readCountLines `hasNoType` ''Step
9897inspect $ 'readCountLines `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
99−inspect $ 'readCountLines `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
98+inspect $ 'readCountLines `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
10099#endif
101100
102101-- | Count the number of words in a file.
@@ -120,7 +119,7 @@ readSumBytes = S.fold Fold.sum . S.unfold FH.reader
120119inspect $ hasNoTypeClasses 'readSumBytes
121120inspect $ 'readSumBytes `hasNoType` ''Step
122121inspect $ 'readSumBytes `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
123−inspect $ 'readSumBytes `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
122+inspect $ 'readSumBytes `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
124123#endif
125124
126125-- XXX When we mark this with INLINE and we have two benchmarks using S.drain
@@ -236,13 +235,13 @@ groupsOf :: Int -> Handle -> IO Int
236235groupsOf n inh =
237236 -- writeNUnsafe gives 2.5x boost here over writeN.
238237 S.fold Fold.length
239− $ IP.groupsOf n (AT.writeNUnsafe n) (S.unfold FH.reader inh)
238+ $ IP.groupsOf n (A.writeNUnsafe n) (S.unfold FH.reader inh)
240239
241240#ifdef INSPECTION
242241inspect $ hasNoTypeClasses 'groupsOf
243242inspect $ 'groupsOf `hasNoType` ''Step
244243inspect $ 'groupsOf `hasNoType` ''FoldMany
245−inspect $ 'groupsOf `hasNoType` ''AT.ArrayUnsafe -- AT.writeNUnsafe
244+inspect $ 'groupsOf `hasNoType` ''MutArray.ArrayUnsafe -- AT.writeNUnsafe
246245 -- FH.read/A.read
247246inspect $ 'groupsOf `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
248247#endif
benchmark/Streamly/Benchmark/FileSystem/Handle/ReadWrite.hs+9−10
@@ -34,7 +34,7 @@ import qualified Streamly.Internal.FileSystem.Handle as IFH
3434import qualified Streamly.Data.Array as A
3535import qualified Streamly.Data.Stream.Prelude as S
3636
37−import Gauge hiding (env)
37+import Test.Tasty.Bench hiding (env)
3838import Streamly.Benchmark.Common.Handle
3939
4040#ifdef INSPECTION
@@ -42,8 +42,7 @@ import Streamly.Internal.Data.Stream (Step(..))
4242
4343import qualified Streamly.Internal.Data.Stream as D
4444import qualified Streamly.Internal.Data.Tuple.Strict as Strict
45−import qualified Streamly.Internal.Data.Stream.MutChunked as MAS
46−import qualified Streamly.Internal.Data.Array as AT
45+import qualified Streamly.Internal.Data.MutArray as MutArray
4746
4847import Test.Inspection
4948#endif
@@ -83,7 +82,7 @@ copyStream inh outh = S.fold (FH.write outh) (S.unfold FH.reader inh)
8382inspect $ hasNoTypeClasses 'copyStream
8483inspect $ 'copyStream `hasNoType` ''Step -- S.unfold
8584inspect $ 'copyStream `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
86−inspect $ 'copyStream `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
85+inspect $ 'copyStream `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
8786 -- FH.read/A.read
8887inspect $ 'copyStream `hasNoType` ''Strict.Tuple3' -- FH.write/chunksOf
8988#endif
@@ -109,8 +108,8 @@ readFromBytesNull inh devNull = IFH.putBytes devNull $ S.unfold FH.reader inh
109108#ifdef INSPECTION
110109inspect $ hasNoTypeClasses 'readFromBytesNull
111110inspect $ 'readFromBytesNull `hasNoType` ''Step
112−inspect $ 'readFromBytesNull `hasNoType` ''MAS.SpliceState
113−inspect $ 'readFromBytesNull `hasNoType` ''AT.ArrayUnsafe -- FH.fromBytes/S.chunksOf
111+inspect $ 'readFromBytesNull `hasNoType` ''MutArray.SpliceState
112+inspect $ 'readFromBytesNull `hasNoType` ''MutArray.ArrayUnsafe -- FH.fromBytes/S.chunksOf
114113inspect $ 'readFromBytesNull `hasNoType` ''D.FoldMany
115114#endif
116115
@@ -123,8 +122,8 @@ readWithFromBytesNull inh devNull =
123122#ifdef INSPECTION
124123inspect $ hasNoTypeClasses 'readWithFromBytesNull
125124inspect $ 'readWithFromBytesNull `hasNoType` ''Step
126−inspect $ 'readWithFromBytesNull `hasNoType` ''MAS.SpliceState
127−inspect $ 'readWithFromBytesNull `hasNoType` ''AT.ArrayUnsafe -- FH.fromBytes/S.chunksOf
125+inspect $ 'readWithFromBytesNull `hasNoType` ''MutArray.SpliceState
126+inspect $ 'readWithFromBytesNull `hasNoType` ''MutArray.ArrayUnsafe -- FH.fromBytes/S.chunksOf
128127inspect $ 'readWithFromBytesNull `hasNoType` ''D.FoldMany
129128#endif
130129
@@ -171,7 +170,7 @@ writeReadWith inh devNull = IUF.fold fld unf (defaultChunkSize, inh)
171170inspect $ hasNoTypeClasses 'writeReadWith
172171inspect $ 'writeReadWith `hasNoType` ''Step
173172inspect $ 'writeReadWith `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
174−inspect $ 'writeReadWith `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
173+inspect $ 'writeReadWith `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
175174 -- FH.read/A.read
176175#endif
177176
@@ -188,7 +187,7 @@ writeRead inh devNull = IUF.fold fld unf inh
188187inspect $ hasNoTypeClasses 'writeRead
189188inspect $ 'writeRead `hasNoType` ''Step
190189inspect $ 'writeRead `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
191−inspect $ 'writeRead `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
190+inspect $ 'writeRead `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
192191 -- FH.read/A.read
193192#endif
194193
benchmark/Streamly/Benchmark/Prelude/Adaptive.hs+1−1
@@ -10,7 +10,7 @@
1010import Control.Concurrent (threadDelay)
1111import Control.Monad (when)
1212import Control.Monad.IO.Class (liftIO)
13−import Gauge
13+import Test.Tasty.Bench
1414import Streamly.Prelude as S
1515import System.Random (randomRIO)
1616
benchmark/Streamly/Benchmark/Prelude/Ahead.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Gauge
18+import Test.Tasty.Bench
1919
2020moduleName :: String
2121moduleName = "Prelude.Ahead"
benchmark/Streamly/Benchmark/Prelude/Async.hs+1−1
@@ -16,7 +16,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream.Transform as Transform
1616import Streamly.Benchmark.Common
1717import Streamly.Benchmark.Prelude
1818
19−import Gauge
19+import Test.Tasty.Bench
2020
2121moduleName :: String
2222moduleName = "Prelude.Async"
benchmark/Streamly/Benchmark/Prelude/Concurrent.hs+1−1
@@ -15,7 +15,7 @@ import Streamly.Prelude
1515 , fromWAsync, parallel, fromParallel
1616 )
1717
18−import Gauge
18+import Test.Tasty.Bench
1919import qualified Streamly.Prelude as S
2020
2121-------------------------------------------------------------------------------
benchmark/Streamly/Benchmark/Prelude/Merge.hs+1−1
@@ -30,7 +30,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream as Stream
3030import Streamly.Benchmark.Common
3131import Streamly.Benchmark.Prelude
3232
33−import Gauge
33+import Test.Tasty.Bench
3434
3535#ifdef INSPECTION
3636import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Prelude/Parallel.hs+1−1
@@ -22,7 +22,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream as Internal
2222import Streamly.Benchmark.Common
2323import Streamly.Benchmark.Prelude
2424
25−import Gauge
25+import Test.Tasty.Bench
2626
2727moduleName :: String
2828moduleName = "Prelude.Parallel"
benchmark/Streamly/Benchmark/Prelude/Rate.hs+1−1
@@ -14,7 +14,7 @@ import qualified Streamly.Prelude as S
1414import Streamly.Benchmark.Common
1515import Streamly.Benchmark.Prelude
1616
17−import Gauge
17+import Test.Tasty.Bench
1818
1919moduleName :: String
2020moduleName = "Prelude.Rate"
benchmark/Streamly/Benchmark/Prelude/WAsync.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Gauge
18+import Test.Tasty.Bench
1919
2020moduleName :: String
2121moduleName = "Prelude.WAsync"
benchmark/Streamly/Benchmark/Prelude/WSerial.hs+1−1
@@ -29,7 +29,7 @@ import qualified Streamly.Internal.Data.Unfold as UF
2929import Streamly.Benchmark.Common
3030import Streamly.Benchmark.Prelude
3131
32−import Gauge
32+import Test.Tasty.Bench
3333
3434#ifdef INSPECTION
3535import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Prelude/ZipAsync.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Gauge
18+import Test.Tasty.Bench
1919
2020moduleName :: String
2121moduleName = "Prelude.ZipAsync"
benchmark/Streamly/Benchmark/Prelude/ZipSerial.hs+1−1
@@ -29,7 +29,7 @@ import qualified Streamly.Prelude as S
2929import Streamly.Benchmark.Common
3030import Streamly.Benchmark.Prelude hiding (sourceUnfoldrM)
3131
32−import Gauge
32+import Test.Tasty.Bench
3333
3434#ifdef INSPECTION
3535import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Unicode/Char.hs+1−1
@@ -16,7 +16,7 @@ import Control.DeepSeq (NFData(..))
1616import Streamly.Internal.Data.Array (Array)
1717import System.FilePath (dropExtensions, takeFileName)
1818import System.FilePath.Posix ((</>))
19−import Gauge.Main (Benchmark, bench, bgroup, defaultMain, env, nfIO)
19+import Test.Tasty.Bench (Benchmark, bench, bgroup, defaultMain, env, nfIO)
2020import Streamly.Internal.Unicode.Char
2121 ( NormalizationMode(NFC, NFD, NFKC, NFKD)
2222 , normalize
benchmark/Streamly/Benchmark/Unicode/Parser.hs+1−1
@@ -31,7 +31,7 @@ import qualified Streamly.Data.Array as Array
3131import qualified Streamly.Data.Stream as Stream
3232import qualified Streamly.Internal.Unicode.Parser as PRU
3333
34−import Gauge hiding (env)
34+import Test.Tasty.Bench hiding (env)
3535import Streamly.Benchmark.Common
3636
3737runParser :: Int -> (Stream IO Char -> IO a) -> IO ()
benchmark/Streamly/Benchmark/Unicode/Stream.hs+24−6
@@ -11,6 +11,11 @@
1111{-# LANGUAGE CPP #-}
1212{-# LANGUAGE ScopedTypeVariables #-}
1313
14+#undef FUSION_CHECK
15+#ifdef FUSION_CHECK
16+{-# OPTIONS_GHC -ddump-simpl -ddump-to-file -dsuppress-all #-}
17+#endif
18+
1419#ifdef __HADDOCK_VERSION__
1520#undef INSPECTION
1621#endif
@@ -33,16 +38,17 @@ import qualified Streamly.Internal.FileSystem.Handle as Handle
3338import qualified Streamly.Internal.Unicode.Array as UnicodeArr
3439import qualified Streamly.Internal.Unicode.Stream as Unicode
3540
36−import Gauge hiding (env)
41+import Test.Tasty.Bench hiding (env)
3742import Streamly.Benchmark.Common
3843import Streamly.Benchmark.Common.Handle
3944
4045#ifdef INSPECTION
41−import Streamly.Internal.Data.Unbox (Unbox)
46+import Streamly.Internal.Data.MutByteArray (Unbox)
4247import Streamly.Internal.Data.Stream (Step(..))
48+import qualified Streamly.Internal.Data.Array as Array
49+import qualified Streamly.Internal.Data.MutArray as MutArray
4350import qualified Streamly.Internal.Data.Fold as Fold
4451import qualified Streamly.Internal.Data.Tuple.Strict as Strict
45−import qualified Streamly.Internal.Data.Array as Array
4652
4753import Test.Inspection
4854#endif
@@ -218,7 +224,7 @@ inspect $ 'copyStreamLatin1' `hasNoType` ''Step
218224inspect $ 'copyStreamLatin1' `hasNoType` ''Unfold.ConcatState -- Handle.read/UF.many
219225
220226inspect $ 'copyStreamLatin1' `hasNoType` ''Fold.Step
221−inspect $ 'copyStreamLatin1' `hasNoType` ''Array.ArrayUnsafe -- Handle.write/writeNUnsafe
227+inspect $ 'copyStreamLatin1' `hasNoType` ''MutArray.ArrayUnsafe -- Handle.write/writeNUnsafe
222228 -- Handle.read/Array.read
223229inspect $ 'copyStreamLatin1' `hasNoType` ''Strict.Tuple3' -- Handle.write/chunksOf
224230#endif
@@ -239,7 +245,7 @@ inspect $ 'copyStreamLatin1 `hasNoType` ''Unfold.ConcatState -- Handle.read/UF.m
239245
240246inspect $ 'copyStreamLatin1 `hasNoType` ''Fold.ManyState
241247inspect $ 'copyStreamLatin1 `hasNoType` ''Fold.Step
242−inspect $ 'copyStreamLatin1 `hasNoType` ''Array.ArrayUnsafe -- Handle.write/writeNUnsafe
248+inspect $ 'copyStreamLatin1 `hasNoType` ''MutArray.ArrayUnsafe -- Handle.write/writeNUnsafe
243249 -- Handle.read/Array.read
244250inspect $ 'copyStreamLatin1 `hasNoType` ''Strict.Tuple3' -- Handle.write/chunksOf
245251#endif
@@ -280,7 +286,8 @@ _copyStreamUtf8'Fold :: Handle -> Handle -> IO ()
280286_copyStreamUtf8'Fold inh outh =
281287 Stream.fold (Handle.write outh)
282288 $ Unicode.encodeUtf8
283− $ Stream.foldMany Unicode.writeCharUtf8'
289+ $ Stream.catRights
290+ $ Stream.parseMany Unicode.writeCharUtf8'
284291 $ Stream.unfold Handle.reader inh
285292
286293{-# NOINLINE _copyStreamUtf8Parser #-}
@@ -317,6 +324,7 @@ o_1_space_decode_encode_read env =
317324
318325main :: IO ()
319326main = do
327+#ifndef FUSION_CHECK
320328 env <- mkHandleBenchEnv
321329 defaultMain (allBenchmarks env)
322330
@@ -329,3 +337,13 @@ main = do
329337 , o_1_space_decode_encode_read env
330338 ]
331339 ]
340+#else
341+ -- Enable FUSION_CHECK macro at the beginning of the file
342+ -- Enable one benchmark below, and run the benchmark
343+ -- Check the .dump-simpl output
344+ env <- mkHandleBenchEnv
345+ let mkHandles (RefHandles {bigInH = inh, outputH = outh}) = Handles inh outh
346+ (Handles inh outh) <- getHandles env mkHandles
347+ copyStreamLatin1' inh outh
348+ return ()
349+#endif
benchmark/Streamly/Benchmark/Unicode/Utf8.hs+1−1
@@ -13,7 +13,7 @@
1313import Data.Char (chr)
1414import Streamly.Internal.Unicode.Utf8 (Utf8)
1515
16−import Gauge hiding (env)
16+import Test.Tasty.Bench hiding (env)
1717import Streamly.Benchmark.Common
1818
1919import qualified Streamly.Internal.Unicode.Utf8 as Utf8
benchmark/bench-runner/Main.hs+32−4
@@ -16,9 +16,11 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
1616 benchName = drop 4 benchName0
1717
1818 general
19− | "o-1-sp" `isInfixOf` benchName = "-K36K -M16M"
19+ -- GHC-9.6 requires 32M heap vs 16M for GHC-9.4
20+ | "o-1-sp" `isInfixOf` benchName = "-K36K -M32M"
2021 | "o-n-h" `isInfixOf` benchName = "-K36K -M32M"
21− | "o-n-st" `isInfixOf` benchName = "-K1M -M16M"
22+ -- GHC-9.6 requires 32M heap vs 16M for GHC-9.4
23+ | "o-n-st" `isInfixOf` benchName = "-K1M -M32M"
2224 | "o-n-sp" `isInfixOf` benchName = "-K1M -M32M"
2325 | otherwise = ""
2426
@@ -27,6 +29,17 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
2729 | otherwise = ""
2830
2931 benchSpecific
32+ -- GHC-9.6 requires 64M, earlier it was 16M
33+ | "Data.Fold/o-n-heap.key-value.toHashMapIO (max buckets) sum" == benchName =
34+ "-M64M"
35+
36+ -- This is required only for the --long case because we allocate all
37+ -- the arrays upfront. depends on the size of the stream.
38+ | "Data.Parser/o-1-space" `isPrefixOf` benchName =
39+ "-M128M"
40+
41+ -----------------------------------------------------------------------
42+
3043 | "Prelude.Parallel/o-n-heap.mapping.mapM" == benchName = "-M256M"
3144 | "Prelude.Parallel/o-n-heap.monad-outer-product."
3245 `isPrefixOf` benchName = "-M256M"
@@ -57,6 +70,12 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
5770 -- GHC 9.4.4 requires 4M
5871 | "Data.Stream.StreamDK/o-n-space.traversable."
5972 `isPrefixOf` benchName = "-K4M"
73+ -- GHC-9.6 requires 64M, earlier it was 32M
74+ | "Data.Stream.StreamDK/o-n-heap.buffered.readsPrec pure streams" == benchName =
75+ "-M64M"
76+ -- GHC-9.6 requires 64M, earlier it was 32M
77+ | "Data.Stream.StreamDK/o-n-heap.buffered.showPrec Haskell lists" == benchName =
78+ "-M64M"
6079
6180 -----------------------------------------------------------------------
6281
@@ -74,6 +93,13 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
7493 | "Data.Stream/o-n-space.iterated."
7594 `isPrefixOf` benchName = "-K4M"
7695
96+ -- GHC-9.6 requires 64M, earlier it was 32M
97+ | "Data.Stream/o-n-heap.buffered.showPrec Haskell lists" == benchName =
98+ "-M64M"
99+ -- GHC-9.6 requires 64M, earlier it was 32M
100+ | "Data.Stream/o-n-heap.buffered.readsPrec pure streams" == benchName =
101+ "-M64M"
102+
77103 | "Data.Stream.ConcurrentEager/o-n-heap.monad-outer-product.toNullAp"
78104 `isPrefixOf` benchName = "-M1024M"
79105 | "Data.Stream.ConcurrentEager/o-1-space."
@@ -86,10 +112,12 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
86112
87113 | "Data.Array" `isPrefixOf` benchName
88114 && "/o-1-space.generation.read" `isSuffixOf` benchName = "-M32M"
115+ -- XXX GHC 9.6 onwards needs 64M, earlier it was 32M
89116 | "Data.Array" `isPrefixOf` benchName
90− && "/o-1-space.generation.show" `isSuffixOf` benchName = "-M32M"
117+ && "/o-1-space.generation.show" `isSuffixOf` benchName = "-M64M"
118+ -- XXX GHC 9.6 onwards needs 64M, earlier it was 32M
91119 | "Data.Array.Generic/o-1-space.transformationX4.map"
92− `isPrefixOf` benchName = "-M32M"
120+ `isPrefixOf` benchName = "-M64M"
93121
94122 -- XXX For --long option, need to check why so much heap is required.
95123 -- Note, if we remove the chunked stream module we need to keep the
benchmark/lib/Streamly/Benchmark/Common.hs+1−1
@@ -56,7 +56,7 @@ import System.Random (randomRIO)
5656import qualified Streamly.Internal.Data.Fold as Fold
5757import qualified Streamly.Internal.Data.Stream as S
5858
59−import Gauge
59+import Test.Tasty.Bench
6060
6161-------------------------------------------------------------------------------
6262-- Benchmark Prefixes
benchmark/lib/Streamly/Benchmark/Common/Handle.hs+3−1
@@ -30,6 +30,8 @@ module Streamly.Benchmark.Common.Handle
3030 , isSpace
3131 , isSp
3232 , mkHandleBenchEnv
33+ , Handles(..)
34+ , getHandles
3335 )
3436where
3537
@@ -46,7 +48,7 @@ import System.Process (callCommand)
4648
4749import Data.IORef
4850import Prelude hiding (last, length)
49−import Gauge hiding (env)
51+import Test.Tasty.Bench hiding (env)
5052
5153
5254scratchDir :: String
benchmark/lib/Streamly/Benchmark/Prelude.hs+1−1
@@ -82,7 +82,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream.Type as IsStream
8282import qualified Streamly.Internal.Data.Pipe as Pipe
8383import qualified Streamly.Internal.Data.Stream.Serial as Serial
8484
85−import Gauge
85+import Test.Tasty.Bench
8686import Streamly.Internal.Data.Time.Units
8787
8888-- Common polymorphic stream APIs used across multiple stream modules
benchmark/streamly-benchmarks.cabal+23−27
@@ -54,11 +54,6 @@ flag opt
5454 manual: True
5555 default: True
5656
57−flag use-gauge
58− description: Use gauge instead of tasty-bench for benchmarking
59− manual: True
60− default: False
61−
6257flag use-streamly-core
6358 description: Benchmark only core package
6459 manual: True
@@ -145,6 +140,10 @@ common compile-options
145140 -Wno-missing-kind-signatures
146141 -Wno-operator-whitespace
147142
143+ if impl(ghc >= 9.8)
144+ ghc-options:
145+ -Wno-missing-role-annotations
146+
148147 if flag(has-llvm)
149148 ghc-options: -fllvm
150149
@@ -176,43 +175,34 @@ common bench-depends
176175 -- Core libraries shipped with ghc, the min and max
177176 -- constraints of these libraries should match with
178177 -- the GHC versions we support
179− base >= 4.9 && < 4.19
180− , deepseq >= 1.4.1 && < 1.5
178+ base >= 4.9 && < 4.20
179+ , deepseq >= 1.4.1 && < 1.6
181180 , mtl >= 2.2 && < 2.3.2
182181
183182 -- other libraries
183+ , exceptions >= 0.8 && < 0.11
184184 , random >= 1.0 && < 2.0
185185 , transformers >= 0.4 && < 0.7
186− , containers >= 0.5 && < 0.7
186+ , containers >= 0.5 && < 0.8
187187 , hashable >= 1.3 && < 1.5
188188 , unordered-containers >= 0.2 && < 0.3
189189 , process >= 1.4 && < 1.7
190190 , directory >= 1.2.2 && < 1.4
191− , filepath >= 1.4.1 && < 1.5
192− , ghc-prim >= 0.4 && < 0.11
191+ , filepath >= 1.4.1 && < 1.6
192+ , ghc-prim >= 0.4 && < 0.12
193+ , tasty-bench >= 0.3 && < 0.4
194+ , tasty >= 1.4.1 && < 1.6
195+ , streamly-core
193196
194− if flag(use-streamly-core)
195− build-depends: streamly-core
196− else
197+ if !flag(use-streamly-core)
197198 build-depends: streamly
198− , streamly-core
199−
200− if flag(use-gauge)
201− build-depends: gauge >= 0.2.4 && < 0.3
202− else
203− build-depends: tasty-bench >= 0.3 && < 0.4
204− , tasty >= 1.4.1 && < 1.5
205− mixins: tasty-bench
206− (Test.Tasty.Bench as Gauge
207− , Test.Tasty.Bench as Gauge.Main
208− )
209199
210200 if flag(fusion-plugin) && !impl(ghcjs) && !impl(ghc < 8.6)
211201 build-depends:
212202 fusion-plugin >= 0.2 && < 0.3
213203 if flag(inspection)
214− build-depends: template-haskell >= 2.14 && < 2.21
215− , inspection-testing >= 0.4 && < 0.5
204+ build-depends: template-haskell >= 2.14 && < 2.22
205+ , inspection-testing >= 0.4 && < 0.6
216206 -- Array uses a Storable constraint in dev build making several inspection
217207 -- tests fail
218208 if flag(dev) && flag(inspection)
@@ -379,7 +369,13 @@ benchmark Data.Serialize
379369 hs-source-dirs: .
380370 cpp-options: -DUSE_TH
381371 main-is: Streamly/Benchmark/Data/Serialize.hs
382− build-depends: QuickCheck
372+ build-depends: QuickCheck, template-haskell
373+ other-modules:
374+ Streamly.Benchmark.Data.Serialize.TH
375+ Streamly.Benchmark.Data.Serialize.RecCompatible
376+ Streamly.Benchmark.Data.Serialize.RecNonCompatible
377+ if flag(limit-build-mem)
378+ ghc-options: +RTS -M1000M -RTS
383379
384380benchmark Data.Unfold
385381 import: bench-options
configure+9−9
@@ -1,6 +1,6 @@
11#! /bin/sh
22# Guess values for system-dependent variables and create Makefiles.
3−# Generated by GNU Autoconf 2.71 for streamly 0.10.0.
3+# Generated by GNU Autoconf 2.71 for streamly 0.10.1.
44#
55# Report bugs to <streamly@composewell.com>.
66#
@@ -610,8 +610,8 @@ MAKEFLAGS=
610610# Identity of this package.
611611PACKAGE_NAME='streamly'
612612PACKAGE_TARNAME='streamly'
613−PACKAGE_VERSION='0.10.0'
614−PACKAGE_STRING='streamly 0.10.0'
613+PACKAGE_VERSION='0.10.1'
614+PACKAGE_STRING='streamly 0.10.1'
615615PACKAGE_BUGREPORT='streamly@composewell.com'
616616PACKAGE_URL='https://streamly.composewell.com'
617617
@@ -1224,7 +1224,7 @@ if test "$ac_init_help" = "long"; then
12241224 # Omit some internal or obsolete options to make the list less imposing.
12251225 # This message is too long to be a string in the A/UX 3.1 sh.
12261226 cat <<_ACEOF
1227−\`configure' configures streamly 0.10.0 to adapt to many kinds of systems.
1227+\`configure' configures streamly 0.10.1 to adapt to many kinds of systems.
12281228
12291229Usage: $0 [OPTION]... [VAR=VALUE]...
12301230
@@ -1286,7 +1286,7 @@ fi
12861286
12871287if test -n "$ac_init_help"; then
12881288 case $ac_init_help in
1289− short | recursive ) echo "Configuration of streamly 0.10.0:";;
1289+ short | recursive ) echo "Configuration of streamly 0.10.1:";;
12901290 esac
12911291 cat <<\_ACEOF
12921292
@@ -1372,7 +1372,7 @@ fi
13721372test -n "$ac_init_help" && exit $ac_status
13731373if $ac_init_version; then
13741374 cat <<\_ACEOF
1375−streamly configure 0.10.0
1375+streamly configure 0.10.1
13761376generated by GNU Autoconf 2.71
13771377
13781378Copyright (C) 2021 Free Software Foundation, Inc.
@@ -1500,7 +1500,7 @@ cat >config.log <<_ACEOF
15001500This file contains any messages produced by compilers while
15011501running configure, to aid debugging if configure makes a mistake.
15021502
1503−It was created by streamly $as_me 0.10.0, which was
1503+It was created by streamly $as_me 0.10.1, which was
15041504generated by GNU Autoconf 2.71. Invocation command line was
15051505
15061506 $ $0$ac_configure_args_raw
@@ -3800,7 +3800,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
38003800# report actual input values of CONFIG_FILES etc. instead of their
38013801# values after options handling.
38023802ac_log="
3803−This file was extended by streamly $as_me 0.10.0, which was
3803+This file was extended by streamly $as_me 0.10.1, which was
38043804generated by GNU Autoconf 2.71. Invocation command line was
38053805
38063806 CONFIG_FILES = $CONFIG_FILES
@@ -3856,7 +3856,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\
38563856cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
38573857ac_cs_config='$ac_cs_config_escaped'
38583858ac_cs_version="\\
3859−streamly config.status 0.10.0
3859+streamly config.status 0.10.1
38603860configured by $0, generated by GNU Autoconf 2.71,
38613861 with options \\"\$ac_cs_config\\"
38623862
configure.ac+1−1
@@ -3,7 +3,7 @@
33# See https://www.gnu.org/software/autoconf/manual/autoconf.html for help on
44# the macros used in this file.
55
6−AC_INIT([streamly], [0.10.0], [streamly@composewell.com], [streamly], [https://streamly.composewell.com])
6+AC_INIT([streamly], [0.10.1], [streamly@composewell.com], [streamly], [https://streamly.composewell.com])
77
88# To suppress "WARNING: unrecognized options: --with-compiler"
99AC_ARG_WITH([compiler], [GHC])
core/configure+9−9
@@ -1,6 +1,6 @@
11#! /bin/sh
22# Guess values for system-dependent variables and create Makefiles.
3−# Generated by GNU Autoconf 2.71 for streamly-core 0.2.0.
3+# Generated by GNU Autoconf 2.71 for streamly-core 0.2.2.
44#
55# Report bugs to <streamly@composewell.com>.
66#
@@ -610,8 +610,8 @@ MAKEFLAGS=
610610# Identity of this package.
611611PACKAGE_NAME='streamly-core'
612612PACKAGE_TARNAME='streamly-core'
613−PACKAGE_VERSION='0.2.0'
614−PACKAGE_STRING='streamly-core 0.2.0'
613+PACKAGE_VERSION='0.2.2'
614+PACKAGE_STRING='streamly-core 0.2.2'
615615PACKAGE_BUGREPORT='streamly@composewell.com'
616616PACKAGE_URL='https://streamly.composewell.com'
617617
@@ -1256,7 +1256,7 @@ if test "$ac_init_help" = "long"; then
12561256 # Omit some internal or obsolete options to make the list less imposing.
12571257 # This message is too long to be a string in the A/UX 3.1 sh.
12581258 cat <<_ACEOF
1259−\`configure' configures streamly-core 0.2.0 to adapt to many kinds of systems.
1259+\`configure' configures streamly-core 0.2.2 to adapt to many kinds of systems.
12601260
12611261Usage: $0 [OPTION]... [VAR=VALUE]...
12621262
@@ -1318,7 +1318,7 @@ fi
13181318
13191319if test -n "$ac_init_help"; then
13201320 case $ac_init_help in
1321− short | recursive ) echo "Configuration of streamly-core 0.2.0:";;
1321+ short | recursive ) echo "Configuration of streamly-core 0.2.2:";;
13221322 esac
13231323 cat <<\_ACEOF
13241324
@@ -1404,7 +1404,7 @@ fi
14041404test -n "$ac_init_help" && exit $ac_status
14051405if $ac_init_version; then
14061406 cat <<\_ACEOF
1407−streamly-core configure 0.2.0
1407+streamly-core configure 0.2.2
14081408generated by GNU Autoconf 2.71
14091409
14101410Copyright (C) 2021 Free Software Foundation, Inc.
@@ -1622,7 +1622,7 @@ cat >config.log <<_ACEOF
16221622This file contains any messages produced by compilers while
16231623running configure, to aid debugging if configure makes a mistake.
16241624
1625−It was created by streamly-core $as_me 0.2.0, which was
1625+It was created by streamly-core $as_me 0.2.2, which was
16261626generated by GNU Autoconf 2.71. Invocation command line was
16271627
16281628 $ $0$ac_configure_args_raw
@@ -3832,7 +3832,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
38323832# report actual input values of CONFIG_FILES etc. instead of their
38333833# values after options handling.
38343834ac_log="
3835−This file was extended by streamly-core $as_me 0.2.0, which was
3835+This file was extended by streamly-core $as_me 0.2.2, which was
38363836generated by GNU Autoconf 2.71. Invocation command line was
38373837
38383838 CONFIG_FILES = $CONFIG_FILES
@@ -3888,7 +3888,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\
38883888cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
38893889ac_cs_config='$ac_cs_config_escaped'
38903890ac_cs_version="\\
3891−streamly-core config.status 0.2.0
3891+streamly-core config.status 0.2.2
38923892configured by $0, generated by GNU Autoconf 2.71,
38933893 with options \\"\$ac_cs_config\\"
38943894
core/configure.ac+1−1
@@ -3,7 +3,7 @@
33# See https://www.gnu.org/software/autoconf/manual/autoconf.html for help on
44# the macros used in this file.
55
6−AC_INIT([streamly-core], [0.2.0], [streamly@composewell.com], [streamly-core], [https://streamly.composewell.com])
6+AC_INIT([streamly-core], [0.2.2], [streamly@composewell.com], [streamly-core], [https://streamly.composewell.com])
77
88# To suppress "WARNING: unrecognized options: --with-compiler"
99AC_ARG_WITH([compiler], [GHC])
core/docs/ApiChangelogs/0.1.0-0.2.0.txtnew file+1471
@@ -0,0 +1,1471 @@
1+---------------------------------
2+API Annotations
3+---------------------------------
4+
5+[A] : Added
6+[R] : Removed
7+[C] : Changed
8+[O] : Old definition
9+[N] : New definition
10+[D] : Deprecated
11+
12+---------------------------------
13+API diff
14+---------------------------------
15+
16+[C] Streamly.Unicode.Parser
17+ [A] double :: Monad m => Parser Char m Double
18+[C] Streamly.FileSystem.Handle
19+ [A] readWith :: MonadIO m => Int -> Handle -> Stream m Word8
20+ [A] readChunksWith :: MonadIO m => Int -> Handle -> Stream m (Array Word8)
21+[C] Streamly.FileSystem.Dir
22+ [C] readEither
23+ [O] readEither :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
24+ [N] readEither :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
25+ [C] read
26+ [O] read :: MonadIO m => FilePath -> Stream m FilePath
27+ [N] read :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
28+[C] Streamly.Data.Unfold
29+ [A] class Enum a => Enumerable a
30+ [A] second :: b -> Unfold m (a, b) c -> Unfold m a c
31+ [A] first :: a -> Unfold m (a, b) c -> Unfold m b c
32+ [A] enumerateFromTo :: (Enumerable a, Monad m) => Unfold m (a, a) a
33+ [A] enumerateFromThenTo :: (Enumerable a, Monad m) => Unfold m (a, a, a) a
34+ [A] enumerateFromThen :: (Enumerable a, Monad m) => Unfold m (a, a) a
35+ [A] enumerateFrom :: (Enumerable a, Monad m) => Unfold m a a
36+[C] Streamly.Data.StreamK
37+ [C] parseChunks
38+ [O] parseChunks :: (Monad m, Unbox a) => ParserK a m b -> StreamK m (Array a) -> m (Either ParseError b)
39+ [N] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
40+ [C] parseBreakChunks
41+ [O] parseBreakChunks :: (Monad m, Unbox a) => ParserK a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
42+ [N] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
43+ [A] parseBreak :: forall m a b. Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
44+ [A] parse :: Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b)
45+ [A] handle :: (MonadCatch m, Exception e) => (e -> m (StreamK m a)) -> StreamK m a -> StreamK m a
46+ [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> StreamK m a) -> StreamK m a
47+[C] Streamly.Data.Stream
48+ [A] wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
49+ [A] splitOn :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
50+ [C] handle
51+ [O] handle :: (MonadCatch m, Exception e) => (e -> Stream m a) -> Stream m a -> Stream m a
52+ [N] handle :: (MonadCatch m, Exception e) => (e -> m (Stream m a)) -> Stream m a -> Stream m a
53+ [A] groupsOf :: Monad m => Int -> Fold m a b -> Stream m a -> Stream m b
54+[C] Streamly.Data.ParserK
55+ [D] fromParser :: (MonadIO m, Unbox a) => Parser a m b -> ParserK (Array a) m b
56+ [D] fromFold :: (MonadIO m, Unbox a) => Fold m a b -> ParserK (Array a) m b
57+ [A] adaptCG :: Monad m => Parser a m b -> ParserK (Array a) m b
58+ [A] adaptC :: (Monad m, Unbox a) => Parser a m b -> ParserK (Array a) m b
59+ [A] adapt :: Monad m => Parser a m b -> ParserK a m b
60+[C] Streamly.Data.Parser
61+ [A] groupByRollingEither :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
62+ [A] groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
63+[A] Streamly.Data.MutByteArray
64+ [A] class Unbox a
65+ [A] class Serialize a
66+ [A] SerializeConfig
67+ [A] MutByteArray
68+ [A] unpin :: MutByteArray -> IO MutByteArray
69+ [A] sizeOf :: (Unbox a, SizeOfRep (Rep a)) => Proxy a -> Int
70+ [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
71+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
72+ [A] pinnedNew :: Int -> IO MutByteArray
73+ [A] pin :: MutByteArray -> IO MutByteArray
74+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
75+ [A] new :: Int -> IO MutByteArray
76+ [A] isPinned :: MutByteArray -> Bool
77+ [A] inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
78+ [A] inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
79+ [A] inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
80+ [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
81+ [A] deriveUnbox :: Q [Dec] -> Q [Dec]
82+ [A] deriveSerializeWith :: (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
83+ [A] deriveSerialize :: Q [Dec] -> Q [Dec]
84+ [A] addSizeTo :: Serialize a => Int -> a -> Int
85+[C] Streamly.Data.MutArray.Generic
86+ [A] write :: MonadIO m => Fold m a (MutArray a)
87+ [A] readRev :: MonadIO m => MutArray a -> Stream m a
88+ [A] read :: MonadIO m => MutArray a -> Stream m a
89+ [A] putIndexUnsafe :: forall m a. MonadIO m => Int -> MutArray a -> a -> m ()
90+ [A] modifyIndexUnsafe :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
91+ [A] length :: MutArray a -> Int
92+ [A] getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
93+ [C] getIndex
94+ [O] getIndex :: MonadIO m => Int -> MutArray a -> m a
95+ [N] getIndex :: MonadIO m => Int -> MutArray a -> m (Maybe a)
96+ [A] fromListN :: MonadIO m => Int -> [a] -> m (MutArray a)
97+ [A] fromList :: MonadIO m => [a] -> m (MutArray a)
98+[C] Streamly.Data.MutArray
99+ [A] unpin :: MutArray a -> IO (MutArray a)
100+ [A] readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
101+ [A] read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
102+ [A] putIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
103+ [D] pokeByteIndex :: Unbox a => Int -> MutByteArray -> a -> IO ()
104+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
105+ [A] pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
106+ [A] pin :: MutArray a -> IO (MutArray a)
107+ [D] peekByteIndex :: Unbox a => Int -> MutByteArray -> IO a
108+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
109+ [D] newPinned :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
110+ [A] modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
111+ [A] modifyIndex :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
112+ [A] modify :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (a -> a) -> m ()
113+ [A] isPinned :: MutArray a -> Bool
114+ [A] getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
115+ [C] getIndex
116+ [O] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
117+ [N] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
118+[C] Streamly.Data.Array.Generic
119+ [A] toList :: Array a -> [a]
120+ [A] getIndex :: Int -> Array a -> Maybe a
121+[C] Streamly.Data.Array
122+ [A] unpin :: Array a -> IO (Array a)
123+ [D] pokeByteIndex :: Unbox a => Int -> MutByteArray -> a -> IO ()
124+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
125+ [A] pin :: Array a -> IO (Array a)
126+ [D] peekByteIndex :: Unbox a => Int -> MutByteArray -> IO a
127+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
128+ [A] isPinned :: Array a -> Bool
129+[C] Streamly.Console.Stdio
130+
131+---------------------------------
132+Internal API diff
133+---------------------------------
134+
135+[C] Streamly.Internal.Unicode.Stream
136+ [C] writeCharUtf8'
137+ [O] writeCharUtf8' :: Monad m => Fold m Word8 Char
138+ [N] writeCharUtf8' :: Monad m => Parser Word8 m Char
139+[C] Streamly.Internal.Unicode.Parser
140+ [A] number :: Monad m => Parser Char m (Integer, Int)
141+ [A] mkDouble :: Integer -> Int -> Double
142+ [A] doubleParser :: Monad m => Parser Char m (Int, Int)
143+ [C] double
144+ [O] double :: Parser Char m Double
145+ [N] double :: Monad m => Parser Char m Double
146+[R] Streamly.Internal.Serialize.ToBytes
147+[R] Streamly.Internal.Serialize.FromBytes
148+[C] Streamly.Internal.FileSystem.File
149+ [A] writeAppendWith :: (MonadIO m, MonadCatch m) => Int -> FilePath -> Stream m Word8 -> m ()
150+ [A] writeAppendChunks :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Array a) -> m ()
151+ [A] writeAppendArray :: FilePath -> Array a -> IO ()
152+ [A] writeAppend :: (MonadIO m, MonadCatch m) => FilePath -> Stream m Word8 -> m ()
153+ [R] appendWith :: (MonadIO m, MonadCatch m) => Int -> FilePath -> Stream m Word8 -> m ()
154+ [R] appendChunks :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Array a) -> m ()
155+ [R] appendArray :: FilePath -> Array a -> IO ()
156+ [R] append :: (MonadIO m, MonadCatch m) => FilePath -> Stream m Word8 -> m ()
157+[C] Streamly.Internal.FileSystem.Dir
158+ [C] reader
159+ [O] reader :: MonadIO m => Unfold m FilePath FilePath
160+ [N] reader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
161+ [C] readFiles
162+ [O] readFiles :: MonadIO m => FilePath -> Stream m FilePath
163+ [N] readFiles :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
164+ [C] readEitherPaths
165+ [O] readEitherPaths :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
166+ [N] readEitherPaths :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
167+ [C] readEither
168+ [O] readEither :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
169+ [N] readEither :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
170+ [C] readDirs
171+ [O] readDirs :: MonadIO m => FilePath -> Stream m FilePath
172+ [N] readDirs :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
173+ [C] read
174+ [O] read :: MonadIO m => FilePath -> Stream m FilePath
175+ [N] read :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
176+ [C] fileReader
177+ [O] fileReader :: MonadIO m => Unfold m FilePath FilePath
178+ [N] fileReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
179+ [C] eitherReaderPaths
180+ [O] eitherReaderPaths :: MonadIO m => Unfold m FilePath (Either FilePath FilePath)
181+ [N] eitherReaderPaths :: (MonadIO m, MonadCatch m) => Unfold m FilePath (Either FilePath FilePath)
182+ [C] eitherReader
183+ [O] eitherReader :: MonadIO m => Unfold m FilePath (Either FilePath FilePath)
184+ [N] eitherReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath (Either FilePath FilePath)
185+ [C] dirReader
186+ [O] dirReader :: MonadIO m => Unfold m FilePath FilePath
187+ [N] dirReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
188+[R] Streamly.Internal.Data.Unfold.Type
189+[R] Streamly.Internal.Data.Unfold.Enumeration
190+[C] Streamly.Internal.Data.Unfold
191+ [C] Unfold
192+ [A] Unfold :: (s -> m (Step s b)) -> (a -> m s) -> Unfold m a b
193+ [A] takeWhileMWithInput :: Monad m => (a -> b -> m Bool) -> Unfold m a b -> Unfold m a b
194+ [A] manyInterleave :: Monad m => Unfold m a b -> Unfold m c a -> Unfold m c b
195+ [A] enumerateFromStepIntegral :: (Monad m, Integral a) => Unfold m (a, a) a
196+ [A] crossApplySnd :: Unfold m a b -> Unfold m a c -> Unfold m a c
197+ [A] crossApplyFst :: Unfold m a b -> Unfold m a c -> Unfold m a b
198+ [A] concatMap :: Monad m => (b -> Unfold m a c) -> Unfold m a b -> Unfold m a c
199+[R] Streamly.Internal.Data.Unboxed
200+[C] Streamly.Internal.Data.Time.Units
201+ [R] Streamly.Internal.Data.Unboxed.Unbox
202+ [A] Streamly.Internal.Data.Unbox.Unbox
203+ [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.NanoSecond64
204+ [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.MilliSecond64
205+ [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.MicroSecond64
206+[R] Streamly.Internal.Data.Time.Clock.Type
207+[A] Streamly.Internal.Data.StreamK
208+ [A] CrossStreamK
209+ [A] (FixityR,6)
210+ [A] (FixityR,6)
211+ [A] (FixityR,6)
212+ [A] (FixityR,5)
213+ [A] (FixityR,5)
214+ [A] (FixityR,6)
215+ [A] (FixityR,5)
216+ [A] StreamK
217+ [A] MkStream :: (forall r. State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
218+ [A] zipWithM :: Monad m => (a -> b -> m c) -> StreamK m a -> StreamK m b -> StreamK m c
219+ [A] zipWith :: Monad m => (a -> b -> c) -> StreamK m a -> StreamK m b -> StreamK m c
220+ [A] unfoldrMWith :: Monad m => (m a -> StreamK m a -> StreamK m a) -> (b -> m (Maybe (a, b))) -> b -> StreamK m a
221+ [A] unfoldrM :: Monad m => (b -> m (Maybe (a, b))) -> b -> StreamK m a
222+ [A] unfoldr :: (b -> Maybe (a, b)) -> b -> StreamK m a
223+ [A] uncons :: Applicative m => StreamK m a -> m (Maybe (a, StreamK m a))
224+ [A] unShare :: StreamK m a -> StreamK m a
225+ [A] unCross :: CrossStreamK m a -> StreamK m a
226+ [A] toStream :: Applicative m => StreamK m a -> Stream m a
227+ [A] toList :: Monad m => StreamK m a -> m [a]
228+ [A] the :: (Eq a, Monad m) => StreamK m a -> m (Maybe a)
229+ [A] takeWhile :: (a -> Bool) -> StreamK m a -> StreamK m a
230+ [A] take :: Int -> StreamK m a -> StreamK m a
231+ [A] tail :: Applicative m => StreamK m a -> m (Maybe (StreamK m a))
232+ [A] sortBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> StreamK m a
233+ [A] sequence :: Monad m => StreamK m (m a) -> StreamK m a
234+ [A] scanlx' :: (x -> a -> x) -> x -> (x -> b) -> StreamK m a -> StreamK m b
235+ [A] scanl' :: (b -> a -> b) -> b -> StreamK m a -> StreamK m b
236+ [A] reverse :: StreamK m a -> StreamK m a
237+ [A] replicateMWith :: (m a -> StreamK m a -> StreamK m a) -> Int -> m a -> StreamK m a
238+ [A] replicateM :: Monad m => Int -> m a -> StreamK m a
239+ [A] replicate :: Int -> a -> StreamK m a
240+ [A] repeatMWith :: (m a -> t m a -> t m a) -> m a -> t m a
241+ [A] repeatM :: Monad m => m a -> StreamK m a
242+ [A] repeat :: a -> StreamK m a
243+ [A] parseDBreak :: Monad m => Parser a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
244+ [A] parseD :: Monad m => Parser a m b -> StreamK m a -> m (Either ParseError b)
245+ [A] parseChunksGeneric :: Monad m => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
246+ [A] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
247+ [A] parseBreakChunksGeneric :: forall m a b. Monad m => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
248+ [A] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
249+ [A] parseBreak :: forall m a b. Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
250+ [A] parse :: Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b)
251+ [A] null :: Monad m => StreamK m a -> m Bool
252+ [A] notElem :: (Monad m, Eq a) => a -> StreamK m a -> m Bool
253+ [A] nilM :: Applicative m => m b -> StreamK m a
254+ [A] nil :: StreamK m a
255+ [A] mkStream :: (forall r. State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
256+ [A] mkCross :: StreamK m a -> CrossStreamK m a
257+ [A] minimumBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> m (Maybe a)
258+ [A] minimum :: (Monad m, Ord a) => StreamK m a -> m (Maybe a)
259+ [A] mfix :: Monad m => (m a -> StreamK m a) -> StreamK m a
260+ [A] mergeMapWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m a -> StreamK m b
261+ [A] mergeIterateWith :: (StreamK m a -> StreamK m a -> StreamK m a) -> (a -> StreamK m a) -> StreamK m a -> StreamK m a
262+ [A] mergeByM :: Monad m => (a -> a -> m Ordering) -> StreamK m a -> StreamK m a -> StreamK m a
263+ [A] mergeBy :: (a -> a -> Ordering) -> StreamK m a -> StreamK m a -> StreamK m a
264+ [A] maximumBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> m (Maybe a)
265+ [A] maximum :: (Monad m, Ord a) => StreamK m a -> m (Maybe a)
266+ [A] mapMaybe :: (a -> Maybe b) -> StreamK m a -> StreamK m b
267+ [A] mapM_ :: Monad m => (a -> m b) -> StreamK m a -> m ()
268+ [A] mapMWith :: (m b -> StreamK m b -> StreamK m b) -> (a -> m b) -> StreamK m a -> StreamK m b
269+ [A] mapMSerial :: Monad m => (a -> m b) -> StreamK m a -> StreamK m b
270+ [A] mapM :: Monad m => (a -> m b) -> StreamK m a -> StreamK m b
271+ [A] map :: (a -> b) -> StreamK m a -> StreamK m b
272+ [A] lookup :: (Monad m, Eq a) => a -> StreamK m (a, b) -> m (Maybe b)
273+ [A] liftInner :: (Monad m, MonadTrans t, Monad (t m)) => StreamK m a -> StreamK (t m) a
274+ [A] last :: Monad m => StreamK m a -> m (Maybe a)
275+ [A] iterateMWith :: Monad m => (m a -> StreamK m a -> StreamK m a) -> (a -> m a) -> m a -> StreamK m a
276+ [A] iterateM :: Monad m => (a -> m a) -> m a -> StreamK m a
277+ [A] iterate :: (a -> a) -> a -> StreamK m a
278+ [A] intersperseM :: Monad m => m a -> StreamK m a -> StreamK m a
279+ [A] intersperse :: Monad m => a -> StreamK m a -> StreamK m a
280+ [A] interleaveMin :: StreamK m a -> StreamK m a -> StreamK m a
281+ [A] interleaveFst :: StreamK m a -> StreamK m a -> StreamK m a
282+ [A] interleave :: StreamK m a -> StreamK m a -> StreamK m a
283+ [A] insertBy :: (a -> a -> Ordering) -> a -> StreamK m a -> StreamK m a
284+ [A] init :: Applicative m => StreamK m a -> m (Maybe (StreamK m a))
285+ [A] hoist :: (Monad m, Monad n) => (forall x. m x -> n x) -> StreamK m a -> StreamK n a
286+ [A] head :: Monad m => StreamK m a -> m (Maybe a)
287+ [A] handle :: (MonadCatch m, Exception e) => (e -> m (StreamK m a)) -> StreamK m a -> StreamK m a
288+ [A] fromYieldK :: YieldK m a -> StreamK m a
289+ [A] fromStream :: Monad m => Stream m a -> StreamK m a
290+ [A] fromStopK :: StopK m -> StreamK m a
291+ [A] fromPure :: a -> StreamK m a
292+ [A] fromList :: [a] -> StreamK m a
293+ [A] fromIndicesMWith :: (m a -> StreamK m a -> StreamK m a) -> (Int -> m a) -> StreamK m a
294+ [A] fromIndicesM :: Monad m => (Int -> m a) -> StreamK m a
295+ [A] fromIndices :: (Int -> a) -> StreamK m a
296+ [A] fromFoldableM :: (Foldable f, Monad m) => f (m a) -> StreamK m a
297+ [A] fromFoldable :: Foldable f => f a -> StreamK m a
298+ [A] fromEffect :: Monad m => m a -> StreamK m a
299+ [A] foldrT :: (Monad m, Monad (s m), MonadTrans s) => (a -> s m b -> s m b) -> s m b -> StreamK m a -> s m b
300+ [A] foldrSShared :: (a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
301+ [A] foldrSM :: Monad m => (m a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
302+ [A] foldrS :: (a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
303+ [A] foldrM :: (a -> m b -> m b) -> m b -> StreamK m a -> m b
304+ [A] foldr1 :: Monad m => (a -> a -> a) -> StreamK m a -> m (Maybe a)
305+ [A] foldr :: Monad m => (a -> b -> b) -> b -> StreamK m a -> m b
306+ [A] foldlx' :: forall m a b x. Monad m => (x -> a -> x) -> x -> (x -> b) -> StreamK m a -> m b
307+ [A] foldlT :: (Monad m, Monad (s m), MonadTrans s) => (s m b -> a -> s m b) -> s m b -> StreamK m a -> s m b
308+ [A] foldlS :: (StreamK m b -> a -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
309+ [A] foldlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> StreamK m a -> m b
310+ [A] foldlM' :: Monad m => (b -> a -> m b) -> m b -> StreamK m a -> m b
311+ [A] foldl' :: Monad m => (b -> a -> b) -> b -> StreamK m a -> m b
312+ [A] foldStreamShared :: State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> StreamK m a -> m r
313+ [A] foldStream :: State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> StreamK m a -> m r
314+ [A] foldEither :: Monad m => Fold m a b -> StreamK m a -> m (Either (Fold m a b) (b, StreamK m a))
315+ [A] foldConcat :: Monad m => Producer m a b -> Fold m b c -> StreamK m a -> m (c, StreamK m a)
316+ [A] foldBreak :: Monad m => Fold m a b -> StreamK m a -> m (b, StreamK m a)
317+ [A] fold :: Monad m => Fold m a b -> StreamK m a -> m b
318+ [A] findM :: Monad m => (a -> m Bool) -> StreamK m a -> m (Maybe a)
319+ [A] findIndices :: (a -> Bool) -> StreamK m a -> StreamK m Int
320+ [A] find :: Monad m => (a -> Bool) -> StreamK m a -> m (Maybe a)
321+ [A] filter :: (a -> Bool) -> StreamK m a -> StreamK m a
322+ [A] evalStateT :: Monad m => m s -> StreamK (StateT s m) a -> StreamK m a
323+ [A] elem :: (Monad m, Eq a) => a -> StreamK m a -> m Bool
324+ [A] dropWhile :: (a -> Bool) -> StreamK m a -> StreamK m a
325+ [A] drop :: Int -> StreamK m a -> StreamK m a
326+ [A] drain :: Monad m => StreamK m a -> m ()
327+ [A] deleteBy :: (a -> a -> Bool) -> a -> StreamK m a -> StreamK m a
328+ [A] crossWith :: Monad m => (a -> b -> c) -> StreamK m a -> StreamK m b -> StreamK m c
329+ [A] crossApplyWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> StreamK m (a -> b) -> StreamK m a -> StreamK m b
330+ [A] crossApplySnd :: StreamK m a -> StreamK m b -> StreamK m b
331+ [A] crossApplyFst :: StreamK m a -> StreamK m b -> StreamK m a
332+ [A] crossApply :: StreamK m (a -> b) -> StreamK m a -> StreamK m b
333+ [A] cross :: Monad m => StreamK m a -> StreamK m b -> StreamK m (a, b)
334+ [A] consMBy :: Monad m => (StreamK m a -> StreamK m a -> StreamK m a) -> m a -> StreamK m a -> StreamK m a
335+ [A] consM :: Monad m => m a -> StreamK m a -> StreamK m a
336+ [A] consK :: YieldK m a -> StreamK m a -> StreamK m a
337+ [A] cons :: a -> StreamK m a -> StreamK m a
338+ [A] conjoin :: Monad m => StreamK m a -> StreamK m a -> StreamK m a
339+ [A] concatMapWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m a -> StreamK m b
340+ [A] concatMapEffect :: Monad m => (b -> StreamK m a) -> m b -> StreamK m a
341+ [A] concatMap :: (a -> StreamK m b) -> StreamK m a -> StreamK m b
342+ [A] concatIterateWith :: (StreamK m a -> StreamK m a -> StreamK m a) -> (a -> StreamK m a) -> StreamK m a -> StreamK m a
343+ [A] concatIterateScanWith :: Monad m => (StreamK m a -> StreamK m a -> StreamK m a) -> (b -> a -> m (b, StreamK m a)) -> m b -> StreamK m a -> StreamK m a
344+ [A] concatIterateLeftsWith :: b ~ Either a c => (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m b -> StreamK m b
345+ [A] concatEffect :: Monad m => m (StreamK m a) -> StreamK m a
346+ [A] buildSM :: Monad m => ((m a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a
347+ [A] buildS :: ((a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a
348+ [A] buildM :: Monad m => (forall r. (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
349+ [A] build :: forall m a. (forall b. (a -> b -> b) -> b -> b) -> StreamK m a
350+ [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> StreamK m a) -> StreamK m a
351+ [A] bindWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> StreamK m a -> (a -> StreamK m b) -> StreamK m b
352+ [A] before :: Monad m => m b -> StreamK m a -> StreamK m a
353+ [A] augmentSM :: Monad m => ((m a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a
354+ [A] augmentS :: ((a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a
355+ [A] append :: StreamK m a -> StreamK m a -> StreamK m a
356+ [A] any :: Monad m => (a -> Bool) -> StreamK m a -> m Bool
357+ [A] all :: Monad m => (a -> Bool) -> StreamK m a -> m Bool
358+ [A] (.:) :: a -> StreamK m a -> StreamK m a
359+ [A] (!!) :: Monad m => StreamK m a -> Int -> m (Maybe a)
360+[R] Streamly.Internal.Data.Stream.StreamK.Type
361+[R] Streamly.Internal.Data.Stream.StreamK.Transformer
362+[R] Streamly.Internal.Data.Stream.StreamK
363+[R] Streamly.Internal.Data.Stream.StreamD.Type
364+[R] Streamly.Internal.Data.Stream.StreamD.Transformer
365+[R] Streamly.Internal.Data.Stream.StreamD.Transform
366+[R] Streamly.Internal.Data.Stream.StreamD.Top
367+[R] Streamly.Internal.Data.Stream.StreamD.Step
368+[R] Streamly.Internal.Data.Stream.StreamD.Nesting
369+[R] Streamly.Internal.Data.Stream.StreamD.Lift
370+[R] Streamly.Internal.Data.Stream.StreamD.Generate
371+[R] Streamly.Internal.Data.Stream.StreamD.Exception
372+[R] Streamly.Internal.Data.Stream.StreamD.Eliminate
373+[R] Streamly.Internal.Data.Stream.StreamD.Container
374+[D] Streamly.Internal.Data.Stream.StreamD
375+[R] Streamly.Internal.Data.Stream.Common
376+[R] Streamly.Internal.Data.Stream.Chunked
377+[C] Streamly.Internal.Data.Stream
378+ [A] class Enum a => Enumerable a
379+ [A] Stream
380+ [A] UnStream :: (State StreamK m a -> s -> m (Step s a)) -> s -> Stream m a
381+ [A] Step
382+ [A] Yield :: a -> s -> Step s a
383+ [A] Stop :: Step s a
384+ [A] Skip :: s -> Step s a
385+ [A] InterleaveState
386+ [A] InterleaveSecondOnly :: s2 -> InterleaveState s1 s2
387+ [A] InterleaveSecond :: s1 -> s2 -> InterleaveState s1 s2
388+ [A] InterleaveFirstOnly :: s1 -> InterleaveState s1 s2
389+ [A] InterleaveFirst :: s1 -> s2 -> InterleaveState s1 s2
390+ [A] FoldManyPost
391+ [A] FoldManyPostYield :: b -> FoldManyPost s fs b a -> FoldManyPost s fs b a
392+ [A] FoldManyPostStart :: s -> FoldManyPost s fs b a
393+ [A] FoldManyPostLoop :: s -> fs -> FoldManyPost s fs b a
394+ [A] FoldManyPostDone :: FoldManyPost s fs b a
395+ [A] FoldMany
396+ [A] FoldManyYield :: b -> FoldMany s fs b a -> FoldMany s fs b a
397+ [A] FoldManyStart :: s -> FoldMany s fs b a
398+ [A] FoldManyLoop :: s -> fs -> FoldMany s fs b a
399+ [A] FoldManyFirst :: fs -> s -> FoldMany s fs b a
400+ [A] FoldManyDone :: FoldMany s fs b a
401+ [A] CrossStream
402+ [A] ConcatUnfoldInterleaveState
403+ [A] ConcatUnfoldInterleaveOuter :: o -> [i] -> ConcatUnfoldInterleaveState o i
404+ [A] ConcatUnfoldInterleaveInnerR :: [i] -> [i] -> ConcatUnfoldInterleaveState o i
405+ [A] ConcatUnfoldInterleaveInnerL :: [i] -> [i] -> ConcatUnfoldInterleaveState o i
406+ [A] ConcatUnfoldInterleaveInner :: o -> [i] -> ConcatUnfoldInterleaveState o i
407+ [A] ConcatMapUState
408+ [A] ConcatMapUOuter :: o -> ConcatMapUState o i
409+ [A] ConcatMapUInner :: o -> i -> ConcatMapUState o i
410+ [A] AppendState
411+ [A] AppendSecond :: s2 -> AppendState s1 s2
412+ [A] AppendFirst :: s1 -> AppendState s1 s2
413+ [A] pattern Stream :: (State StreamK m a -> s -> m (Step s a)) -> s -> Stream m a
414+ [A] zipWithM :: Monad m => (a -> b -> m c) -> Stream m a -> Stream m b -> Stream m c
415+ [A] zipWith :: Monad m => (a -> b -> c) -> Stream m a -> Stream m b -> Stream m c
416+ [A] wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
417+ [A] with :: Monad m => (Stream m a -> Stream m (s, a)) -> (((s, a) -> b) -> Stream m (s, a) -> Stream m (s, a)) -> ((s, a) -> b) -> Stream m a -> Stream m a
418+ [A] usingStateT :: Monad m => m s -> (Stream (StateT s m) a -> Stream (StateT s m) a) -> Stream m a -> Stream m a
419+ [A] usingReaderT :: Monad m => m r -> (Stream (ReaderT r m) a -> Stream (ReaderT r m) a) -> Stream m a -> Stream m a
420+ [A] uniqBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a
421+ [A] uniq :: (Eq a, Monad m) => Stream m a -> Stream m a
422+ [A] unionWithStreamGenericBy :: MonadIO m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
423+ [A] unionWithStreamAscBy :: (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
424+ [A] unfoldrM :: Monad m => (s -> m (Maybe (a, s))) -> s -> Stream m a
425+ [A] unfoldr :: Monad m => (s -> Maybe (a, s)) -> s -> Stream m a
426+ [A] unfoldRoundRobin :: Monad m => Unfold m a b -> Stream m a -> Stream m b
427+ [A] unfoldMany :: Monad m => Unfold m a b -> Stream m a -> Stream m b
428+ [A] unfoldIterateDfs :: Monad m => Unfold m a a -> Stream m a -> Stream m a
429+ [A] unfoldIterateBfsRev :: Monad m => Unfold m a a -> Stream m a -> Stream m a
430+ [A] unfoldIterateBfs :: Monad m => Unfold m a a -> Stream m a -> Stream m a
431+ [A] unfoldInterleave :: Monad m => Unfold m a b -> Stream m a -> Stream m b
432+ [A] unfold :: Applicative m => Unfold m a b -> a -> Stream m b
433+ [A] uncons :: Monad m => Stream m a -> m (Maybe (a, Stream m a))
434+ [A] unCross :: CrossStream m a -> Stream m a
435+ [A] transform :: Monad m => Pipe m a b -> Stream m a -> Stream m b
436+ [A] trace_ :: Monad m => m b -> Stream m a -> Stream m a
437+ [A] trace :: Monad m => (a -> m b) -> Stream m a -> Stream m a
438+ [A] toStreamK :: Monad m => Stream m a -> StreamK m a
439+ [A] toListRev :: Monad m => Stream m a -> m [a]
440+ [A] toList :: Monad m => Stream m a -> m [a]
441+ [A] timestamped :: MonadIO m => Stream m a -> Stream m (AbsTime, a)
442+ [A] timestampWith :: MonadIO m => Double -> Stream m a -> Stream m (AbsTime, a)
443+ [A] timesWith :: MonadIO m => Double -> Stream m (AbsTime, RelTime64)
444+ [A] times :: MonadIO m => Stream m (AbsTime, RelTime64)
445+ [A] timeout :: AbsTime -> t m ()
446+ [A] timeIndexed :: MonadIO m => Stream m a -> Stream m (RelTime64, a)
447+ [A] timeIndexWith :: MonadIO m => Double -> Stream m a -> Stream m (RelTime64, a)
448+ [A] the :: (Eq a, Monad m) => Stream m a -> m (Maybe a)
449+ [A] tapOffsetEvery :: Monad m => Int -> Int -> Fold m a b -> Stream m a -> Stream m a
450+ [A] tap :: Monad m => Fold m a b -> Stream m a -> Stream m a
451+ [A] takeWhileM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
452+ [A] takeWhileLast :: (a -> Bool) -> Stream m a -> Stream m a
453+ [A] takeWhileAround :: (a -> Bool) -> Stream m a -> Stream m a
454+ [A] takeWhile :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
455+ [A] takeEndByM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
456+ [A] takeEndBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
457+ [A] take :: Applicative m => Int -> Stream m a -> Stream m a
458+ [A] tail :: Monad m => Stream m a -> m (Maybe (Stream m a))
459+ [A] stripSuffixUnbox :: (MonadIO m, Eq a, Unbox a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
460+ [A] stripSuffix :: (Monad m, Eq a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
461+ [A] stripPrefix :: (Monad m, Eq a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
462+ [A] strideFromThen :: Monad m => Int -> Int -> Stream m a -> Stream m a
463+ [A] splitOnSuffixSeqAny :: [Array a] -> Fold m a b -> Stream m a -> Stream m b
464+ [A] splitOnSuffixSeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) => Bool -> Array a -> Fold m a b -> Stream m a -> Stream m b
465+ [A] splitOnSeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) => Array a -> Fold m a b -> Stream m a -> Stream m b
466+ [A] splitOnPrefix :: (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
467+ [A] splitOnAny :: [Array a] -> Fold m a b -> Stream m a -> Stream m b
468+ [A] splitOn :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
469+ [A] splitInnerBySuffix :: (Monad m, Eq (f a), Monoid (f a)) => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
470+ [A] splitInnerBy :: Monad m => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
471+ [A] slicesBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
472+ [A] sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
473+ [A] sequence :: Monad m => Stream m (m a) -> Stream m a
474+ [A] scanlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> Stream m b
475+ [A] scanlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> Stream m b
476+ [A] scanlMAfter' :: Monad m => (b -> a -> m b) -> m b -> (b -> m b) -> Stream m a -> Stream m b
477+ [A] scanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
478+ [A] scanlM :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
479+ [A] scanl1M' :: Monad m => (a -> a -> m a) -> Stream m a -> Stream m a
480+ [A] scanl1M :: Monad m => (a -> a -> m a) -> Stream m a -> Stream m a
481+ [A] scanl1' :: Monad m => (a -> a -> a) -> Stream m a -> Stream m a
482+ [A] scanl1 :: Monad m => (a -> a -> a) -> Stream m a -> Stream m a
483+ [A] scanl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
484+ [A] scanl :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
485+ [A] scanMaybe :: Monad m => Fold m a (Maybe b) -> Stream m a -> Stream m b
486+ [A] scanMany :: Monad m => Fold m a b -> Stream m a -> Stream m b
487+ [A] scan :: Monad m => Fold m a b -> Stream m a -> Stream m b
488+ [A] runStateT :: Monad m => m s -> Stream (StateT s m) a -> Stream m (s, a)
489+ [A] runReaderT :: Monad m => m s -> Stream (ReaderT s m) a -> Stream m a
490+ [A] runInnerWithState :: Monad m => (forall b. s -> t m b -> m (b, s)) -> m s -> Stream (t m) a -> Stream m (s, a)
491+ [A] runInnerWith :: Monad m => (forall b. t m b -> m b) -> Stream (t m) a -> Stream m a
492+ [A] roundRobin :: Monad m => Stream m a -> Stream m a -> Stream m a
493+ [A] rollingMapM :: Monad m => (Maybe a -> a -> m b) -> Stream m a -> Stream m b
494+ [A] rollingMap2 :: Monad m => (a -> a -> b) -> Stream m a -> Stream m b
495+ [A] rollingMap :: Monad m => (Maybe a -> a -> b) -> Stream m a -> Stream m b
496+ [A] reverseUnbox :: (MonadIO m, Unbox a) => Stream m a -> Stream m a
497+ [A] reverse :: Monad m => Stream m a -> Stream m a
498+ [A] replicateM :: Monad m => Int -> m a -> Stream m a
499+ [A] replicate :: Monad m => Int -> a -> Stream m a
500+ [A] repeated :: Stream m a -> Stream m a
501+ [A] repeatM :: Monad m => m a -> Stream m a
502+ [A] repeat :: Monad m => a -> Stream m a
503+ [A] relTimesWith :: MonadIO m => Double -> Stream m RelTime64
504+ [A] relTimes :: MonadIO m => Stream m RelTime64
505+ [A] refoldMany :: Monad m => Refold m x a b -> m x -> Stream m a -> Stream m b
506+ [A] refoldIterateM :: Monad m => Refold m b a b -> m b -> Stream m a -> Stream m b
507+ [A] reduceIterateBfs :: Monad m => (a -> a -> m a) -> Stream m a -> m (Maybe a)
508+ [A] reassembleBy :: Fold m a b -> (a -> a -> Int) -> Stream m a -> Stream m b
509+ [A] prune :: (a -> Bool) -> Stream m a -> Stream m a
510+ [A] prescanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
511+ [A] prescanl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
512+ [A] postscanlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> Stream m b
513+ [A] postscanlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> Stream m b
514+ [A] postscanlMAfter' :: Monad m => (b -> a -> m b) -> m b -> (b -> m b) -> Stream m a -> Stream m b
515+ [A] postscanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
516+ [A] postscanlM :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
517+ [A] postscanl' :: Monad m => (a -> b -> a) -> a -> Stream m b -> Stream m a
518+ [A] postscanl :: Monad m => (a -> b -> a) -> a -> Stream m b -> Stream m a
519+ [A] postscan :: Monad m => Fold m a b -> Stream m a -> Stream m b
520+ [A] parseSequence :: Stream m (Parser a m b) -> Stream m a -> Stream m b
521+ [A] parseManyTill :: Parser a m b -> Parser a m x -> Stream m a -> Stream m b
522+ [A] parseManyD :: Monad m => Parser a m b -> Stream m a -> Stream m (Either ParseError b)
523+ [A] parseMany :: Monad m => Parser a m b -> Stream m a -> Stream m (Either ParseError b)
524+ [A] parseIterateD :: Monad m => (b -> Parser a m b) -> b -> Stream m a -> Stream m (Either ParseError b)
525+ [A] parseIterate :: Monad m => (b -> Parser a m b) -> b -> Stream m a -> Stream m (Either ParseError b)
526+ [A] parseD :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b)
527+ [A] parseBreakD :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b, Stream m a)
528+ [A] parseBreak :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b, Stream m a)
529+ [A] parse :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b)
530+ [A] onException :: MonadCatch m => m b -> Stream m a -> Stream m a
531+ [A] null :: Monad m => Stream m a -> m Bool
532+ [A] nub :: (Monad m, Ord a) => Stream m a -> Stream m a
533+ [A] notElem :: (Monad m, Eq a) => a -> Stream m a -> m Bool
534+ [A] nilM :: Applicative m => m b -> Stream m a
535+ [A] nil :: Applicative m => Stream m a
536+ [A] morphInner :: Monad n => (forall x. m x -> n x) -> Stream m a -> Stream n a
537+ [A] mkCross :: Stream m a -> CrossStream m a
538+ [A] minimumBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> m (Maybe a)
539+ [A] minimum :: (Monad m, Ord a) => Stream m a -> m (Maybe a)
540+ [A] mergeMinBy :: (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
541+ [A] mergeFstBy :: (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
542+ [A] mergeByM :: Monad m => (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
543+ [A] mergeBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
544+ [A] maximumBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> m (Maybe a)
545+ [A] maximum :: (Monad m, Ord a) => Stream m a -> m (Maybe a)
546+ [A] mapMaybeM :: Monad m => (a -> m (Maybe b)) -> Stream m a -> Stream m b
547+ [A] mapMaybe :: Monad m => (a -> Maybe b) -> Stream m a -> Stream m b
548+ [A] mapM_ :: Monad m => (a -> m b) -> Stream m a -> m ()
549+ [A] mapM :: Monad m => (a -> m b) -> Stream m a -> Stream m b
550+ [A] map :: Monad m => (a -> b) -> Stream m a -> Stream m b
551+ [A] lookup :: (Monad m, Eq a) => a -> Stream m (a, b) -> m (Maybe b)
552+ [A] liftInnerWith :: Monad (t m) => (forall b. m b -> t m b) -> Stream m a -> Stream (t m) a
553+ [A] liftInner :: (Monad m, MonadTrans t, Monad (t m)) => Stream m a -> Stream (t m) a
554+ [A] last :: Monad m => Stream m a -> m (Maybe a)
555+ [A] joinOuterGeneric :: MonadIO m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (Maybe a, Maybe b)
556+ [A] joinOuterAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (Maybe a, Maybe b)
557+ [A] joinOuter :: (Ord k, MonadIO m) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, Maybe a, Maybe b)
558+ [A] joinLeftGeneric :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
559+ [A] joinLeftAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
560+ [A] joinLeft :: (Ord k, Monad m) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, Maybe b)
561+ [A] joinInnerGeneric :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, b)
562+ [A] joinInnerAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, b)
563+ [A] joinInner :: (Monad m, Ord k) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, b)
564+ [A] iterateM :: Monad m => (a -> m a) -> m a -> Stream m a
565+ [A] iterate :: Monad m => (a -> a) -> a -> Stream m a
566+ [A] isSuffixOfUnbox :: (MonadIO m, Eq a, Unbox a) => Stream m a -> Stream m a -> m Bool
567+ [A] isSuffixOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
568+ [A] isSubsequenceOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
569+ [A] isPrefixOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
570+ [A] isInfixOf :: (MonadIO m, Eq a, Enum a, Storable a, Unbox a) => Stream m a -> Stream m a -> m Bool
571+ [A] intersperseM_ :: Monad m => m b -> Stream m a -> Stream m a
572+ [A] intersperseMWith :: Int -> m a -> Stream m a -> Stream m a
573+ [A] intersperseMSuffix_ :: Monad m => m b -> Stream m a -> Stream m a
574+ [A] intersperseMSuffixWith :: forall m a. Monad m => Int -> m a -> Stream m a -> Stream m a
575+ [A] intersperseMSuffix :: forall m a. Monad m => m a -> Stream m a -> Stream m a
576+ [A] intersperseMPrefix_ :: Monad m => m b -> Stream m a -> Stream m a
577+ [A] intersperseM :: Monad m => m a -> Stream m a -> Stream m a
578+ [A] intersperse :: Monad m => a -> Stream m a -> Stream m a
579+ [A] intersectBySorted :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
580+ [A] interposeSuffixM :: Monad m => m c -> Unfold m b c -> Stream m b -> Stream m c
581+ [A] interposeSuffix :: Monad m => c -> Unfold m b c -> Stream m b -> Stream m c
582+ [A] interposeM :: Monad m => m c -> Unfold m b c -> Stream m b -> Stream m c
583+ [A] interpose :: Monad m => c -> Unfold m b c -> Stream m b -> Stream m c
584+ [A] interleaveMin :: Monad m => Stream m a -> Stream m a -> Stream m a
585+ [A] interleaveFstSuffix :: Monad m => Stream m a -> Stream m a -> Stream m a
586+ [A] interleaveFst :: Monad m => Stream m a -> Stream m a -> Stream m a
587+ [A] interleave :: Monad m => Stream m a -> Stream m a -> Stream m a
588+ [A] intercalateSuffix :: Monad m => Unfold m b c -> b -> Stream m b -> Stream m c
589+ [A] intercalate :: Monad m => Unfold m b c -> b -> Stream m b -> Stream m c
590+ [A] insertBy :: Monad m => (a -> a -> Ordering) -> a -> Stream m a -> Stream m a
591+ [A] indexedR :: Monad m => Int -> Stream m a -> Stream m (Int, a)
592+ [A] indexed :: Monad m => Stream m a -> Stream m (Int, a)
593+ [A] headElse :: Monad m => a -> Stream m a -> m a
594+ [A] head :: Monad m => Stream m a -> m (Maybe a)
595+ [A] handle :: (MonadCatch m, Exception e) => (e -> m (Stream m a)) -> Stream m a -> Stream m a
596+ [A] groupsWhile :: Monad m => (a -> a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
597+ [A] groupsRollingBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
598+ [A] groupsOf :: Monad m => Int -> Fold m a b -> Stream m a -> Stream m b
599+ [A] gintercalateSuffix :: Monad m => Unfold m a c -> Stream m a -> Unfold m b c -> Stream m b -> Stream m c
600+ [A] gintercalate :: Monad m => Unfold m a c -> Stream m a -> Unfold m b c -> Stream m b -> Stream m c
601+ [A] ghandle :: (MonadCatch m, Exception e) => (e -> Stream m a -> m (Stream m a)) -> Stream m a -> Stream m a
602+ [A] generateM :: Monad m => Int -> (Int -> m a) -> Stream m a
603+ [A] generate :: Monad m => Int -> (Int -> a) -> Stream m a
604+ [A] generalizeInner :: Monad m => Stream Identity a -> Stream m a
605+ [A] gbracket_ :: Monad m => m c -> (c -> m d) -> (c -> e -> Stream m b -> m (Stream m b)) -> (forall s. m s -> m (Either e s)) -> (c -> Stream m b) -> Stream m b
606+ [A] gbracket :: MonadIO m => IO c -> (c -> IO d1) -> (c -> e -> Stream m b -> IO (Stream m b)) -> (c -> IO d2) -> (forall s. m s -> m (Either e s)) -> (c -> Stream m b) -> Stream m b
607+ [A] fromStreamK :: Applicative m => StreamK m a -> Stream m a
608+ [A] fromPure :: Applicative m => a -> Stream m a
609+ [A] fromPtrN :: (Monad m, Storable a) => Int -> Ptr a -> Stream m a
610+ [A] fromPtr :: forall m a. (Monad m, Storable a) => Ptr a -> Stream m a
611+ [A] fromListM :: Monad m => [m a] -> Stream m a
612+ [A] fromList :: Applicative m => [a] -> Stream m a
613+ [A] fromIndicesM :: Monad m => (Int -> m a) -> Stream m a
614+ [A] fromIndices :: Monad m => (Int -> a) -> Stream m a
615+ [A] fromFoldableM :: (Monad m, Foldable f) => f (m a) -> Stream m a
616+ [A] fromFoldable :: (Monad m, Foldable f) => f a -> Stream m a
617+ [A] fromEffect :: Applicative m => m a -> Stream m a
618+ [A] fromByteStr# :: Monad m => Addr# -> Stream m Word8
619+ [A] foldrT :: (Monad m, Monad (t m), MonadTrans t) => (a -> t m b -> t m b) -> t m b -> Stream m a -> t m b
620+ [A] foldrS :: Monad m => (a -> Stream m b -> Stream m b) -> Stream m b -> Stream m a -> Stream m b
621+ [A] foldrMx :: Monad m => (a -> m x -> m x) -> m x -> (m x -> m b) -> Stream m a -> m b
622+ [A] foldrM :: Monad m => (a -> m b -> m b) -> m b -> Stream m a -> m b
623+ [A] foldr1 :: Monad m => (a -> a -> a) -> Stream m a -> m (Maybe a)
624+ [A] foldr :: Monad m => (a -> b -> b) -> b -> Stream m a -> m b
625+ [A] foldlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> m b
626+ [A] foldlT :: (Monad m, Monad (s m), MonadTrans s) => (s m b -> a -> s m b) -> s m b -> Stream m a -> s m b
627+ [A] foldlS :: Monad m => (Stream m b -> a -> Stream m b) -> Stream m b -> Stream m a -> Stream m b
628+ [A] foldlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> m b
629+ [A] foldlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> m b
630+ [A] foldl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> m b
631+ [A] foldSequence :: Stream m (Fold m a b) -> Stream m a -> Stream m b
632+ [A] foldManyPost :: Monad m => Fold m a b -> Stream m a -> Stream m b
633+ [A] foldMany :: Monad m => Fold m a b -> Stream m a -> Stream m b
634+ [A] foldIterateM :: Monad m => (b -> m (Fold m a b)) -> m b -> Stream m a -> Stream m b
635+ [A] foldIterateBfs :: Fold m a (Either a a) -> Stream m a -> m (Maybe a)
636+ [A] foldEither :: Monad m => Fold m a b -> Stream m a -> m (Either (Fold m a b) (b, Stream m a))
637+ [A] foldBreak :: Monad m => Fold m a b -> Stream m a -> m (b, Stream m a)
638+ [A] foldAddLazy :: Monad m => Fold m a b -> Stream m a -> Fold m a b
639+ [A] foldAdd :: Monad m => Fold m a b -> Stream m a -> m (Fold m a b)
640+ [A] fold :: Monad m => Fold m a b -> Stream m a -> m b
641+ [A] findM :: Monad m => (a -> m Bool) -> Stream m a -> m (Maybe a)
642+ [A] findIndices :: Monad m => (a -> Bool) -> Stream m a -> Stream m Int
643+ [A] find :: Monad m => (a -> Bool) -> Stream m a -> m (Maybe a)
644+ [A] finallyUnsafe :: MonadCatch m => m b -> Stream m a -> Stream m a
645+ [A] finallyIO :: (MonadIO m, MonadCatch m) => IO b -> Stream m a -> Stream m a
646+ [A] filterM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
647+ [A] filterInStreamGenericBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
648+ [A] filterInStreamAscBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
649+ [A] filter :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
650+ [A] evalStateT :: Monad m => m s -> Stream (StateT s m) a -> Stream m a
651+ [A] eqBy :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> m Bool
652+ [A] enumerateTo :: (Monad m, Bounded a, Enumerable a) => a -> Stream m a
653+ [A] enumerateFromToSmall :: (Monad m, Enum a) => a -> a -> Stream m a
654+ [A] enumerateFromToIntegral :: (Monad m, Integral a) => a -> a -> Stream m a
655+ [A] enumerateFromToFractional :: (Monad m, Fractional a, Ord a) => a -> a -> Stream m a
656+ [A] enumerateFromTo :: (Enumerable a, Monad m) => a -> a -> Stream m a
657+ [A] enumerateFromThenToSmall :: (Monad m, Enum a) => a -> a -> a -> Stream m a
658+ [A] enumerateFromThenToIntegral :: (Monad m, Integral a) => a -> a -> a -> Stream m a
659+ [A] enumerateFromThenToFractional :: (Monad m, Fractional a, Ord a) => a -> a -> a -> Stream m a
660+ [A] enumerateFromThenTo :: (Enumerable a, Monad m) => a -> a -> a -> Stream m a
661+ [A] enumerateFromThenSmallBounded :: (Monad m, Enumerable a, Bounded a) => a -> a -> Stream m a
662+ [A] enumerateFromThenNum :: (Monad m, Num a) => a -> a -> Stream m a
663+ [A] enumerateFromThenIntegral :: (Monad m, Integral a, Bounded a) => a -> a -> Stream m a
664+ [A] enumerateFromThenFractional :: (Monad m, Fractional a) => a -> a -> Stream m a
665+ [A] enumerateFromThen :: (Enumerable a, Monad m) => a -> a -> Stream m a
666+ [A] enumerateFromStepNum :: (Monad m, Num a) => a -> a -> Stream m a
667+ [A] enumerateFromStepIntegral :: (Integral a, Monad m) => a -> a -> Stream m a
668+ [A] enumerateFromNum :: (Monad m, Num a) => a -> Stream m a
669+ [A] enumerateFromIntegral :: (Monad m, Integral a, Bounded a) => a -> Stream m a
670+ [A] enumerateFromFractional :: (Monad m, Fractional a) => a -> Stream m a
671+ [A] enumerateFromBounded :: (Monad m, Enumerable a, Bounded a) => a -> Stream m a
672+ [A] enumerateFrom :: (Enumerable a, Monad m) => a -> Stream m a
673+ [A] enumerate :: (Monad m, Bounded a, Enumerable a) => Stream m a
674+ [A] elemIndices :: (Monad m, Eq a) => a -> Stream m a -> Stream m Int
675+ [A] elem :: (Monad m, Eq a) => a -> Stream m a -> m Bool
676+ [A] durations :: Double -> t m RelTime64
677+ [A] dropWhileM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
678+ [A] dropWhileLast :: (a -> Bool) -> Stream m a -> Stream m a
679+ [A] dropWhileAround :: (a -> Bool) -> Stream m a -> Stream m a
680+ [A] dropWhile :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
681+ [A] dropSuffix :: Stream m a -> Stream m a -> Stream m a
682+ [A] dropPrefix :: Stream m a -> Stream m a -> Stream m a
683+ [A] dropLast :: Int -> Stream m a -> Stream m a
684+ [A] dropInfix :: Stream m a -> Stream m a -> Stream m a
685+ [A] drop :: Monad m => Int -> Stream m a -> Stream m a
686+ [A] drain :: Monad m => Stream m a -> m ()
687+ [A] deleteInStreamGenericBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
688+ [A] deleteInStreamAscBy :: (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
689+ [A] deleteBy :: Monad m => (a -> a -> Bool) -> a -> Stream m a -> Stream m a
690+ [A] delayPre :: MonadIO m => Double -> Stream m a -> Stream m a
691+ [A] delayPost :: MonadIO m => Double -> Stream m a -> Stream m a
692+ [A] delay :: MonadIO m => Double -> Stream m a -> Stream m a
693+ [A] crossWith :: Monad m => (a -> b -> c) -> Stream m a -> Stream m b -> Stream m c
694+ [A] crossApplySnd :: Functor f => Stream f a -> Stream f b -> Stream f b
695+ [A] crossApplyFst :: Functor f => Stream f a -> Stream f b -> Stream f a
696+ [A] crossApply :: Functor f => Stream f (a -> b) -> Stream f a -> Stream f b
697+ [A] cross :: Monad m => Stream m a -> Stream m b -> Stream m (a, b)
698+ [A] consM :: Applicative m => m a -> Stream m a -> Stream m a
699+ [A] cons :: Applicative m => a -> Stream m a -> Stream m a
700+ [A] concatMapM :: Monad m => (a -> m (Stream m b)) -> Stream m a -> Stream m b
701+ [A] concatMap :: Monad m => (a -> Stream m b) -> Stream m a -> Stream m b
702+ [A] concatIterateScan :: Monad m => (b -> a -> m b) -> (b -> m (Maybe (b, Stream m a))) -> b -> Stream m a
703+ [A] concatIterateDfs :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
704+ [A] concatIterateBfsRev :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
705+ [A] concatIterateBfs :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
706+ [A] concatEffect :: Monad m => m (Stream m a) -> Stream m a
707+ [A] concat :: Monad m => Stream m (Stream m a) -> Stream m a
708+ [A] cmpBy :: Monad m => (a -> b -> Ordering) -> Stream m a -> Stream m b -> m Ordering
709+ [A] catRights :: Monad m => Stream m (Either a b) -> Stream m b
710+ [A] catMaybes :: Monad m => Stream m (Maybe a) -> Stream m a
711+ [A] catLefts :: Monad m => Stream m (Either a b) -> Stream m a
712+ [A] catEithers :: Monad m => Stream m (Either a a) -> Stream m a
713+ [A] bracketUnsafe :: MonadCatch m => m b -> (b -> m c) -> (b -> Stream m a) -> Stream m a
714+ [A] bracketIO3 :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> IO d) -> (b -> IO e) -> (b -> Stream m a) -> Stream m a
715+ [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> Stream m a) -> Stream m a
716+ [A] before :: Monad m => m b -> Stream m a -> Stream m a
717+ [A] append :: Monad m => Stream m a -> Stream m a -> Stream m a
718+ [A] any :: Monad m => (a -> Bool) -> Stream m a -> m Bool
719+ [A] all :: Monad m => (a -> Bool) -> Stream m a -> m Bool
720+ [A] afterUnsafe :: Monad m => m b -> Stream m a -> Stream m a
721+ [A] afterIO :: MonadIO m => IO b -> Stream m a -> Stream m a
722+ [A] absTimesWith :: MonadIO m => Double -> Stream m AbsTime
723+ [A] absTimes :: MonadIO m => Stream m AbsTime
724+ [A] (!!) :: Monad m => Stream m a -> Int -> m (Maybe a)
725+[R] Streamly.Internal.Data.Ring.Unboxed
726+[A] Streamly.Internal.Data.Ring.Generic
727+ [A] Ring
728+ [A] [ringMax] :: Ring a -> !Int
729+ [A] [ringHead] :: Ring a -> !Int
730+ [A] [ringArr] :: Ring a -> MutArray a
731+ [A] Ring :: MutArray a -> !Int -> !Int -> Ring a
732+ [A] writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
733+ [A] unsafeInsertRingWith :: Ring a -> a -> IO Int
734+ [A] toStreamWith :: Int -> Ring a -> Stream m a
735+ [A] toMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
736+ [A] seek :: MonadIO m => Int -> Ring a -> m (Ring a)
737+ [A] createRing :: MonadIO m => Int -> m (Ring a)
738+ [A] copyToMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
739+[C] Streamly.Internal.Data.Ring
740+ [C] Ring
741+ [A] [ringStart] :: Ring a -> {-# UNPACK #-} !ForeignPtr a
742+ [R] [ringMax] :: Ring a -> !Int
743+ [R] [ringHead] :: Ring a -> !Int
744+ [A] [ringBound] :: Ring a -> {-# UNPACK #-} !Ptr a
745+ [R] [ringArr] :: Ring a -> MutArray a
746+ [C] Ring
747+ [O] Ring :: MutArray a -> !Int -> !Int -> Ring a
748+ [N] Ring :: {-# UNPACK #-} !ForeignPtr a -> {-# UNPACK #-} !Ptr a -> Ring a
749+ [A] GHC.Show.Show
750+ [A] instance (GHC.Show.Show a, GHC.Show.Show b, GHC.Show.Show c, GHC.Show.Show d) => GHC.Show.Show (Streamly.Internal.Data.Ring.Tuple4' a b c d)
751+ [A] writeN :: Int -> Fold m a (Ring a)
752+ [R] writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
753+ [R] unsafeInsertRingWith :: Ring a -> a -> IO Int
754+ [A] unsafeInsert :: Storable a => Ring a -> Ptr a -> a -> IO (Ptr a)
755+ [A] unsafeFoldRingNM :: forall m a b. (MonadIO m, Storable a) => Int -> Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
756+ [A] unsafeFoldRingM :: forall m a b. (MonadIO m, Storable a) => Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
757+ [A] unsafeFoldRingFullM :: forall m a b. (MonadIO m, Storable a) => Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
758+ [A] unsafeFoldRing :: forall a b. Storable a => Ptr a -> (b -> a -> b) -> b -> Ring a -> b
759+ [A] unsafeEqArrayN :: Ring a -> Ptr a -> Array a -> Int -> Bool
760+ [A] unsafeEqArray :: Ring a -> Ptr a -> Array a -> Bool
761+ [R] toStreamWith :: Int -> Ring a -> Stream m a
762+ [R] toMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
763+ [A] startOf :: Ring a -> Ptr a
764+ [A] slidingWindowWith :: forall m a b. (MonadIO m, Storable a, Unbox a) => Int -> Fold m ((a, Maybe a), m (MutArray a)) b -> Fold m a b
765+ [A] slidingWindow :: forall m a b. (MonadIO m, Storable a, Unbox a) => Int -> Fold m (a, Maybe a) b -> Fold m a b
766+ [A] slide :: Ring a -> a -> m (Ring a)
767+ [R] seek :: MonadIO m => Int -> Ring a -> m (Ring a)
768+ [A] ringsOf :: Int -> Stream m a -> Stream m (MutArray a)
769+ [A] readRev :: Unfold m (MutArray a) a
770+ [A] read :: forall m a. (MonadIO m, Storable a) => Unfold m (Ring a, Ptr a, Int) a
771+ [A] putIndex :: Ring a -> Int -> a -> m ()
772+ [A] newRing :: Int -> m (Ring a)
773+ [A] new :: forall a. Storable a => Int -> IO (Ring a, Ptr a)
774+ [A] moveBy :: forall a. Storable a => Int -> Ring a -> Ptr a -> Ptr a
775+ [A] modifyIndex :: Ring a -> Int -> (a -> (a, b)) -> m b
776+ [A] length :: Ring a -> Int
777+ [A] getIndexUnsafe :: Ring a -> Int -> m a
778+ [A] getIndexRev :: Ring a -> Int -> m a
779+ [A] getIndex :: Ring a -> Int -> m a
780+ [A] fromArray :: MutArray a -> Ring a
781+ [R] createRing :: MonadIO m => Int -> m (Ring a)
782+ [A] castUnsafe :: Ring a -> Ring b
783+ [A] cast :: forall a b. Storable b => Ring a -> Maybe (Ring b)
784+ [A] bytesFree :: Ring a -> Int
785+ [A] byteLength :: Ring a -> Int
786+ [A] byteCapacity :: Ring a -> Int
787+ [A] asBytes :: Ring a -> Ring Word8
788+ [A] advance :: forall a. Storable a => Ring a -> Ptr a -> Ptr a
789+[R] Streamly.Internal.Data.Producer.Type
790+[R] Streamly.Internal.Data.Producer.Source
791+[C] Streamly.Internal.Data.Producer
792+ [A] Source
793+ [A] unread :: [b] -> Source a b -> Source a b
794+ [A] translate :: Functor m => (a -> c) -> (c -> a) -> Producer m c b -> Producer m a b
795+ [A] source :: Maybe a -> Source a b
796+ [A] producer :: Monad m => Producer m a b -> Producer m (Source a b) b
797+ [A] parseManyD :: Monad m => Parser a m b -> Producer m (Source x a) a -> Producer m (Source x a) (Either ParseError b)
798+ [A] parseMany :: Monad m => Parser a m b -> Producer m (Source x a) a -> Producer m (Source x a) (Either ParseError b)
799+ [A] parse :: Monad m => Parser a m b -> Producer m (Source s a) a -> Source s a -> m (Either ParseError b, Source s a)
800+ [A] lmap :: (a -> a) -> Producer m a b -> Producer m a b
801+ [A] isEmpty :: Source a b -> Bool
802+[R] Streamly.Internal.Data.Pipe.Type
803+[C] Streamly.Internal.Data.Pipe
804+ [A] Step
805+ [A] Yield :: a -> s -> Step s a
806+ [A] Continue :: s -> Step s a
807+ [A] PipeState
808+ [A] Produce :: s2 -> PipeState s1 s2
809+ [A] Consume :: s1 -> PipeState s1 s2
810+ [C] Pipe
811+ [A] Pipe :: (s1 -> a -> m (Step (PipeState s1 s2) b)) -> (s2 -> m (Step (PipeState s1 s2) b)) -> s1 -> Pipe m a b
812+[A] Streamly.Internal.Data.ParserK
813+ [A] Step
814+ [A] Partial :: !Int -> (Input a -> m (Step a m r)) -> Step a m r
815+ [A] Error :: !Int -> String -> Step a m r
816+ [A] Done :: !Int -> r -> Step a m r
817+ [A] Continue :: !Int -> (Input a -> m (Step a m r)) -> Step a m r
818+ [A] ParseResult
819+ [A] Success :: !Int -> !b -> ParseResult b
820+ [A] Failure :: !Int -> !String -> ParseResult b
821+ [A] Input
822+ [A] None :: Input a
823+ [A] Chunk :: a -> Input a
824+ [A] ParserK
825+ [A] [runParser] :: ParserK a m b -> forall r. (ParseResult b -> Int -> Input a -> m (Step a m r)) -> Int -> Int -> Input a -> m (Step a m r)
826+ [A] MkParser :: (forall r. (ParseResult b -> Int -> Input a -> m (Step a m r)) -> Int -> Int -> Input a -> m (Step a m r)) -> ParserK a m b
827+ [A] fromPure :: b -> ParserK a m b
828+ [A] fromEffect :: Monad m => m b -> ParserK a m b
829+ [A] die :: String -> ParserK a m b
830+ [A] adaptCG :: Monad m => Parser a m b -> ParserK (Array a) m b
831+ [A] adaptC :: (Monad m, Unbox a) => Parser a m b -> ParserK (Array a) m b
832+ [A] adapt :: Monad m => Parser a m b -> ParserK a m b
833+[R] Streamly.Internal.Data.Parser.ParserK.Type
834+[R] Streamly.Internal.Data.Parser.ParserD.Type
835+[R] Streamly.Internal.Data.Parser.ParserD.Tee
836+[R] Streamly.Internal.Data.Parser.ParserD
837+[C] Streamly.Internal.Data.Parser
838+ [A] Step
839+ [A] Partial :: !Int -> !s -> Step s b
840+ [A] Error :: !String -> Step s b
841+ [A] Done :: !Int -> !b -> Step s b
842+ [A] Continue :: !Int -> !s -> Step s b
843+ [A] Parser
844+ [A] Parser :: (s -> a -> m (Step s b)) -> m (Initial s b) -> (s -> m (Step s b)) -> Parser a m b
845+ [A] Initial
846+ [A] IPartial :: !s -> Initial s b
847+ [A] IError :: !String -> Initial s b
848+ [A] IDone :: !b -> Initial s b
849+ [A] GHC.Show.Show
850+ [A] instance (GHC.Show.Show a, GHC.Show.Show b) => GHC.Show.Show (Streamly.Internal.Data.Parser.Tuple'Fused a b)
851+ [A] ParseError
852+ [A] ParseError :: String -> ParseError
853+ [A] zipWithM :: Monad m => (a -> b -> m c) -> Stream m a -> Fold m c x -> Parser b m x
854+ [A] zip :: Monad m => Stream m a -> Fold m (a, b) x -> Parser b m x
855+ [A] wordWithQuotes :: (Monad m, Eq a) => Bool -> (a -> a -> Maybe a) -> a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
856+ [A] wordProcessQuotes :: (Monad m, Eq a) => a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
857+ [A] wordKeepQuotes :: (Monad m, Eq a) => a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
858+ [A] wordFramedBy :: Monad m => (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
859+ [A] wordBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
860+ [A] toFold :: Monad m => Parser a m b -> Fold m a b
861+ [A] takeWhileP :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
862+ [A] takeWhile1 :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
863+ [A] takeWhile :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
864+ [A] takeStartBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
865+ [A] takeStartBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
866+ [A] takeP :: Monad m => Int -> Parser a m b -> Parser a m b
867+ [A] takeGE :: Monad m => Int -> Fold m a b -> Parser a m b
868+ [A] takeFramedBy_ :: Monad m => (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
869+ [A] takeFramedByGeneric :: Monad m => Maybe (a -> Bool) -> Maybe (a -> Bool) -> Maybe (a -> Bool) -> Fold m a b -> Parser a m b
870+ [A] takeFramedByEsc_ :: Monad m => (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
871+ [A] takeEndBy_ :: (a -> Bool) -> Parser a m b -> Parser a m b
872+ [A] takeEndByEsc :: Monad m => (a -> Bool) -> (a -> Bool) -> Parser a m b -> Parser a m b
873+ [A] takeEndBy :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
874+ [A] takeEitherSepBy :: (a -> Bool) -> Fold m (Either a b) c -> Parser a m c
875+ [A] takeEQ :: Monad m => Int -> Fold m a b -> Parser a m b
876+ [A] takeBetween :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
877+ [A] subsequenceBy :: (a -> a -> Bool) -> Stream m a -> Parser a m ()
878+ [A] streamEqBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Parser a m ()
879+ [A] split_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
880+ [A] splitWith :: Monad m => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
881+ [A] splitSome :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
882+ [A] splitManyPost :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
883+ [A] splitMany :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
884+ [A] spanByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
885+ [A] spanBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
886+ [A] span :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
887+ [A] some :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
888+ [A] sequence :: Monad m => Stream m (Parser a m b) -> Fold m b c -> Parser a m c
889+ [A] sepByAll :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
890+ [A] sepBy1 :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
891+ [A] sepBy :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
892+ [A] satisfy :: Monad m => (a -> Bool) -> Parser a m a
893+ [A] sampleFromthen :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
894+ [A] roundRobin :: t (Parser a m b) -> Fold m b c -> Parser a m c
895+ [A] rmapM :: Monad m => (b -> m c) -> Parser a m b -> Parser a m c
896+ [A] retryMaxTotal :: Int -> Parser a m b -> Fold m b c -> Parser a m c
897+ [A] retryMaxSuccessive :: Int -> Parser a m b -> Fold m b c -> Parser a m c
898+ [A] retry :: Parser a m b -> Parser a m b
899+ [A] postscan :: Fold m a b -> Parser b m c -> Parser a m c
900+ [A] peek :: Monad m => Parser a m a
901+ [A] oneOf :: (Monad m, Eq a, Foldable f) => f a -> Parser a m a
902+ [A] oneNotEq :: (Monad m, Eq a) => a -> Parser a m a
903+ [A] oneEq :: (Monad m, Eq a) => a -> Parser a m a
904+ [A] one :: Monad m => Parser a m a
905+ [A] noneOf :: (Monad m, Eq a, Foldable f) => f a -> Parser a m a
906+ [A] noErrorUnsafeSplit_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
907+ [A] noErrorUnsafeSplitWith :: Monad m => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
908+ [A] noErrorUnsafeConcatMap :: Monad m => (b -> Parser a m c) -> Parser a m b -> Parser a m c
909+ [A] maybe :: Monad m => (a -> Maybe b) -> Parser a m b
910+ [A] manyTillP :: Parser a m b -> Parser a m x -> Parser b m c -> Parser a m c
911+ [A] manyTill :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
912+ [A] manyThen :: Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
913+ [A] manyP :: Parser a m b -> Parser b m c -> Parser a m c
914+ [A] many :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
915+ [A] makeIndexFilter :: (Fold m (s, a) b -> Parser a m b) -> (((s, a) -> Bool) -> Fold m (s, a) b -> Fold m (s, a) b) -> ((s, a) -> Bool) -> Fold m a b -> Parser a m b
916+ [A] lookAhead :: Monad m => Parser a m b -> Parser a m b
917+ [A] lmapM :: Monad m => (a -> m b) -> Parser b m r -> Parser a m r
918+ [A] lmap :: (a -> b) -> Parser b m r -> Parser a m r
919+ [A] listEqBy :: Monad m => (a -> a -> Bool) -> [a] -> Parser a m [a]
920+ [A] listEq :: (Monad m, Eq a) => [a] -> Parser a m [a]
921+ [A] indexed :: forall m a b. Monad m => Fold m (Int, a) b -> Parser a m b
922+ [A] groupByRollingEither :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
923+ [A] groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
924+ [A] groupBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
925+ [A] fromPure :: Monad m => b -> Parser a m b
926+ [A] fromFoldMaybe :: Monad m => String -> Fold m a (Maybe b) -> Parser a m b
927+ [A] fromFold :: Monad m => Fold m a b -> Parser a m b
928+ [A] fromEffect :: Monad m => m b -> Parser a m b
929+ [A] filter :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
930+ [A] extractStep :: Monad m => (s -> m (Step s1 b)) -> Step s b -> m (Step s1 b)
931+ [A] eof :: Monad m => Parser a m ()
932+ [A] either :: Monad m => (a -> Either String b) -> Parser a m b
933+ [A] dropWhile :: Monad m => (a -> Bool) -> Parser a m ()
934+ [A] dieM :: Monad m => m String -> Parser a m b
935+ [A] die :: Monad m => String -> Parser a m b
936+ [A] deintercalateAll :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
937+ [A] deintercalate1 :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
938+ [A] deintercalate :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
939+ [A] countBetween :: Int -> Int -> Parser a m b -> Fold m b c -> Parser a m c
940+ [A] count :: Int -> Parser a m b -> Fold m b c -> Parser a m c
941+ [A] concatMap :: Monad m => (b -> Parser a m c) -> Parser a m b -> Parser a m c
942+ [A] blockWithQuotes :: (Monad m, Eq a) => (a -> Bool) -> (a -> Bool) -> a -> a -> Fold m a b -> Parser a m b
943+ [A] bimapOverrideCount :: Int -> (s -> s1) -> (b -> b1) -> Step s b -> Step s1 b1
944+ [A] alt :: Monad m => Parser x m a -> Parser x m a -> Parser x m a
945+[A] Streamly.Internal.Data.MutByteArray
946+ [A] class Unbox a
947+ [A] class SizeOfRep (f :: Type -> Type)
948+ [A] class Serialize a
949+ [A] TypeOfType
950+ [A] UnitType :: Name -> TypeOfType
951+ [A] TheType :: SimpleDataCon -> TypeOfType
952+ [A] MultiType :: [SimpleDataCon] -> TypeOfType
953+ [A] SimpleDataCon
954+ [A] SimpleDataCon :: Name -> [Field] -> SimpleDataCon
955+ [A] SerializeConfig
956+ [A] [cfgRecordSyntaxWithHeader] :: SerializeConfig -> Bool
957+ [A] [cfgInlineSize] :: SerializeConfig -> Maybe Inline
958+ [A] [cfgInlineSerialize] :: SerializeConfig -> Maybe Inline
959+ [A] [cfgInlineDeserialize] :: SerializeConfig -> Maybe Inline
960+ [A] [cfgConstructorTagAsString] :: SerializeConfig -> Bool
961+ [A] SerializeConfig :: Maybe Inline -> Maybe Inline -> Maybe Inline -> Bool -> Bool -> SerializeConfig
962+ [A] PinnedState
963+ [A] Unpinned :: PinnedState
964+ [A] Pinned :: PinnedState
965+ [A] MutByteArray
966+ [A] MutByteArray :: MutableByteArray# RealWorld -> MutByteArray
967+ [A] DataType
968+ [A] [dtTvs] :: DataType -> [Name]
969+ [A] [dtName] :: DataType -> Name
970+ [A] [dtCxt] :: DataType -> Cxt
971+ [A] [dtCons] :: DataType -> [DataCon]
972+ [A] DataType :: Name -> [Name] -> Cxt -> [DataCon] -> DataType
973+ [A] DataCon
974+ [A] [dcTvs] :: DataCon -> [Name]
975+ [A] [dcName] :: DataCon -> Name
976+ [A] [dcFields] :: DataCon -> [(Maybe Name, Type)]
977+ [A] [dcCxt] :: DataCon -> Cxt
978+ [A] DataCon :: Name -> [Name] -> Cxt -> [(Maybe Name, Type)] -> DataCon
979+ [A] BoundedPtr
980+ [A] BoundedPtr :: MutByteArray -> Int -> Int -> BoundedPtr
981+ [A] Streamly.Internal.Data.Serialize.Type.Serialize
982+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize Streamly.Internal.Data.MutByteArray.LiftedInteger
983+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize GHC.Num.Integer.Integer
984+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize a => Streamly.Internal.Data.Serialize.Type.Serialize (GHC.Maybe.Maybe a)
985+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize (Data.Proxy.Proxy a)
986+ [A] instance (Streamly.Internal.Data.Serialize.Type.Serialize a, Streamly.Internal.Data.Serialize.Type.Serialize b) => Streamly.Internal.Data.Serialize.Type.Serialize (Data.Either.Either a b)
987+ [A] Peeker
988+ [A] Peeker :: Builder BoundedPtr IO a -> Peeker a
989+ [A] type Field = (Maybe Name, Type)
990+ [A] xorCmp :: [Word8] -> Name -> Name -> Q Exp
991+ [A] wListToString :: [Word8] -> String
992+ [A] w8_int :: Word8 -> Int
993+ [A] w32_int :: Word32 -> Int
994+ [A] unpin :: MutByteArray -> IO MutByteArray
995+ [A] typeOfType :: Type -> [DataCon] -> TypeOfType
996+ [A] skipByte :: Peeker ()
997+ [A] sizeOfRep :: SizeOfRep f => f x -> Int
998+ [A] sizeOfMutableByteArray :: MutByteArray -> IO Int
999+ [A] sizeOf :: (Unbox a, SizeOfRep (Rep a)) => Proxy a -> Int
1000+ [A] simplifyDataCon :: DataCon -> SimpleDataCon
1001+ [A] serializeW8List :: Name -> Name -> [Word8] -> Q Exp
1002+ [A] serializeConfig :: SerializeConfig
1003+ [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
1004+ [A] runPeeker :: Peeker a -> BoundedPtr -> IO a
1005+ [A] reifyDataType :: Name -> Q DataType
1006+ [A] readUnsafe :: Unbox a => Peeker a
1007+ [A] read :: Unbox a => Peeker a
1008+ [A] putSliceUnsafe :: MonadIO m => MutByteArray -> Int -> MutByteArray -> Int -> Int -> m ()
1009+ [A] pokeRep :: PokeRep f => f a -> BoundedPtr -> IO BoundedPtr
1010+ [A] pokeBoundedPtrUnsafe :: forall a. Unbox a => a -> BoundedPtr -> IO BoundedPtr
1011+ [A] pokeBoundedPtr :: forall a. Unbox a => a -> BoundedPtr -> IO BoundedPtr
1012+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
1013+ [A] pinnedNewAlignedBytes :: Int -> Int -> IO MutByteArray
1014+ [A] pinnedNew :: Int -> IO MutByteArray
1015+ [A] pinnedCloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
1016+ [A] pin :: MutByteArray -> IO MutByteArray
1017+ [A] peekRep :: PeekRep f => Peeker (f x)
1018+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
1019+ [A] openConstructor :: Name -> Int -> Q Pat
1020+ [A] nil :: MutByteArray
1021+ [A] newBytesAs :: PinnedState -> Int -> IO MutByteArray
1022+ [A] new :: Int -> IO MutByteArray
1023+ [A] mkSerializeExprFields :: Name -> [Field] -> Q Exp
1024+ [A] mkRecSizeOfExpr :: SimpleDataCon -> Q Exp
1025+ [A] mkRecSerializeExpr :: Name -> SimpleDataCon -> Q Exp
1026+ [A] mkRecDeserializeExpr :: Name -> Name -> Name -> SimpleDataCon -> Q Exp
1027+ [A] mkFieldName :: Int -> Name
1028+ [A] mkDeserializeKeysDec :: Name -> Name -> SimpleDataCon -> Q [Dec]
1029+ [A] mkDeserializeExprOne :: Name -> SimpleDataCon -> Q Exp
1030+ [A] matchConstructor :: Name -> Int -> Q Exp -> Q Match
1031+ [A] makeN :: Int -> Name
1032+ [A] makeI :: Int -> Name
1033+ [A] makeA :: Int -> Name
1034+ [A] litProxy :: Unbox a => Proxy a -> Q Exp
1035+ [A] litIntegral :: Integral a => a -> Q Exp
1036+ [A] isUnitType :: [DataCon] -> Bool
1037+ [A] isRecordSyntax :: SimpleDataCon -> Bool
1038+ [A] isPinned :: MutByteArray -> Bool
1039+ [A] int_w8 :: Int -> Word8
1040+ [A] int_w32 :: Int -> Word32
1041+ [A] inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
1042+ [A] inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
1043+ [A] inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
1044+ [A] getMutableByteArray# :: MutByteArray -> MutableByteArray# RealWorld
1045+ [A] genericSizeOf :: forall a. SizeOfRep (Rep a) => Proxy a -> Int
1046+ [A] genericPokeByteIndex :: (Generic a, PokeRep (Rep a)) => MutByteArray -> Int -> a -> IO ()
1047+ [A] genericPeekByteIndex :: (Generic a, PeekRep (Rep a)) => MutByteArray -> Int -> IO a
1048+ [A] errorUnsupported :: String -> a
1049+ [A] errorUnimplemented :: a
1050+ [A] encodeRecordFields :: Bool -> SerializeConfig -> SerializeConfig
1051+ [A] encodeConstrNames :: Bool -> SerializeConfig -> SerializeConfig
1052+ [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
1053+ [A] deriveUnbox :: Q [Dec] -> Q [Dec]
1054+ [A] deriveSerializeWith :: (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
1055+ [A] deriveSerialize :: Q [Dec] -> Q [Dec]
1056+ [A] conUpdateFuncDec :: Name -> [Field] -> Q [Dec]
1057+ [A] cloneSliceUnsafeAs :: MonadIO m => PinnedState -> Int -> Int -> MutByteArray -> m MutByteArray
1058+ [A] cloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
1059+ [A] c2w :: Char -> Word8
1060+ [A] asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
1061+ [A] addSizeTo :: Serialize a => Int -> a -> Int
1062+ [A] _x :: Name
1063+ [A] _val :: Name
1064+ [A] _tag :: Name
1065+ [A] _initialOffset :: Name
1066+ [A] _endOffset :: Name
1067+ [A] _arr :: Name
1068+ [A] _acc :: Name
1069+[A] Streamly.Internal.Data.MutArray.Stream
1070+ [A] SpliceState
1071+ [A] SpliceYielding :: arr -> SpliceState s arr -> SpliceState s arr
1072+ [A] SpliceInitial :: s -> SpliceState s arr
1073+ [A] SpliceFinish :: SpliceState s arr
1074+ [A] SpliceBuffering :: s -> arr -> SpliceState s arr
1075+ [A] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
1076+ [A] splitOn :: (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> Stream m (MutArray a)
1077+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1078+ [A] packArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1079+ [A] lpackArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
1080+ [A] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
1081+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1082+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1083+ [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
1084+ [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1085+ [A] compactEQ :: Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1086+ [A] compact :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1087+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1088+[A] Streamly.Internal.Data.MutArray.Generic
1089+ [A] MutArray
1090+ [A] [arrTrueLen] :: MutArray a -> {-# UNPACK #-} !Int
1091+ [A] [arrStart] :: MutArray a -> {-# UNPACK #-} !Int
1092+ [A] [arrLen] :: MutArray a -> {-# UNPACK #-} !Int
1093+ [A] [arrContents#] :: MutArray a -> MutableArray# RealWorld a
1094+ [A] MutArray :: MutableArray# RealWorld a -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> MutArray a
1095+ [A] writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
1096+ [A] writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
1097+ [A] writeN :: MonadIO m => Int -> Fold m a (MutArray a)
1098+ [A] write :: MonadIO m => Fold m a (MutArray a)
1099+ [A] uninit :: MonadIO m => MutArray a -> Int -> m (MutArray a)
1100+ [A] toStreamK :: MonadIO m => MutArray a -> StreamK m a
1101+ [A] toList :: MonadIO m => MutArray a -> m [a]
1102+ [A] strip :: MonadIO m => (a -> Bool) -> MutArray a -> m (MutArray a)
1103+ [A] snocWith :: MonadIO m => (Int -> Int) -> MutArray a -> a -> m (MutArray a)
1104+ [A] snocUnsafe :: MonadIO m => MutArray a -> a -> m (MutArray a)
1105+ [A] snoc :: MonadIO m => MutArray a -> a -> m (MutArray a)
1106+ [A] realloc :: MonadIO m => Int -> MutArray a -> m (MutArray a)
1107+ [A] reader :: MonadIO m => Unfold m (MutArray a) a
1108+ [A] readRev :: MonadIO m => MutArray a -> Stream m a
1109+ [A] read :: MonadIO m => MutArray a -> Stream m a
1110+ [A] putSliceUnsafe :: MonadIO m => MutArray a -> Int -> MutArray a -> Int -> Int -> m ()
1111+ [A] putIndices :: MonadIO m => MutArray a -> Fold m (Int, a) ()
1112+ [A] putIndexUnsafe :: forall m a. MonadIO m => Int -> MutArray a -> a -> m ()
1113+ [A] putIndex :: MonadIO m => Int -> MutArray a -> a -> m ()
1114+ [A] producerWith :: Monad m => (forall b. IO b -> m b) -> Producer m (MutArray a) a
1115+ [A] producer :: MonadIO m => Producer m (MutArray a) a
1116+ [A] nil :: MonadIO m => m (MutArray a)
1117+ [A] new :: MonadIO m => Int -> m (MutArray a)
1118+ [A] modifyIndexUnsafe :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
1119+ [A] modifyIndex :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
1120+ [A] length :: MutArray a -> Int
1121+ [A] getSliceUnsafe :: Int -> Int -> MutArray a -> MutArray a
1122+ [A] getSlice :: Int -> Int -> MutArray a -> MutArray a
1123+ [A] getIndexUnsafeWith :: MonadIO m => MutableArray# RealWorld a -> Int -> m a
1124+ [A] getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
1125+ [A] getIndex :: MonadIO m => Int -> MutArray a -> m (Maybe a)
1126+ [A] fromStreamN :: MonadIO m => Int -> Stream m a -> m (MutArray a)
1127+ [A] fromStream :: MonadIO m => Stream m a -> m (MutArray a)
1128+ [A] fromPureStream :: MonadIO m => Stream Identity a -> m (MutArray a)
1129+ [A] fromListN :: MonadIO m => Int -> [a] -> m (MutArray a)
1130+ [A] fromList :: MonadIO m => [a] -> m (MutArray a)
1131+ [A] eq :: (MonadIO m, Eq a) => MutArray a -> MutArray a -> m Bool
1132+ [A] cmp :: (MonadIO m, Ord a) => MutArray a -> MutArray a -> m Ordering
1133+ [A] clone :: MonadIO m => MutArray a -> m (MutArray a)
1134+ [A] chunksOf :: forall m a. MonadIO m => Int -> Stream m a -> Stream m (MutArray a)
1135+[A] Streamly.Internal.Data.MutArray
1136+ [A] MutByteArray
1137+ [A] MutArray
1138+ [A] [arrStart] :: MutArray a -> {-# UNPACK #-} !Int
1139+ [A] [arrEnd] :: MutArray a -> {-# UNPACK #-} !Int
1140+ [A] [arrContents] :: MutArray a -> {-# UNPACK #-} !MutByteArray
1141+ [A] [arrBound] :: MutArray a -> {-# UNPACK #-} !Int
1142+ [A] MutArray :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> MutArray a
1143+ [A] IORef
1144+ [A] ArrayUnsafe
1145+ [A] ArrayUnsafe :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> ArrayUnsafe a
1146+ [A] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1147+ [A] writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1148+ [A] writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1149+ [A] writeNWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1150+ [A] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1151+ [A] writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1152+ [A] writeIORef :: Unbox a => IORef a -> a -> IO ()
1153+ [A] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
1154+ [A] writeAppendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
1155+ [A] writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
1156+ [A] writeAppendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
1157+ [A] writeAppend :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
1158+ [A] write :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1159+ [A] unsafeSwapIndices :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> MutArray a -> m ()
1160+ [A] unpin :: MutArray a -> IO (MutArray a)
1161+ [A] toStreamKWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> StreamK m a
1162+ [A] toStreamKRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> StreamK m a
1163+ [A] toStreamKRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
1164+ [A] toStreamK :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
1165+ [A] toStreamDWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
1166+ [A] toStreamDRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
1167+ [A] toList :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m [a]
1168+ [A] swapIndices :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> MutArray a -> m ()
1169+ [A] strip :: forall a m. (Unbox a, MonadIO m) => (a -> Bool) -> MutArray a -> m (MutArray a)
1170+ [A] splitOn :: (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> Stream m (MutArray a)
1171+ [A] splitAt :: forall a. Unbox a => Int -> MutArray a -> (MutArray a, MutArray a)
1172+ [A] spliceWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int -> Int) -> MutArray a -> MutArray a -> m (MutArray a)
1173+ [A] spliceUnsafe :: MonadIO m => MutArray a -> MutArray a -> m (MutArray a)
1174+ [A] spliceExp :: (MonadIO m, Unbox a) => MutArray a -> MutArray a -> m (MutArray a)
1175+ [A] spliceCopy :: forall m a. MonadIO m => MutArray a -> MutArray a -> m (MutArray a)
1176+ [A] splice :: (MonadIO m, Unbox a) => MutArray a -> MutArray a -> m (MutArray a)
1177+ [A] snocWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> MutArray a -> a -> m (MutArray a)
1178+ [A] snocUnsafe :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1179+ [A] snocMay :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (Maybe (MutArray a))
1180+ [A] snocLinear :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1181+ [A] snoc :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1182+ [A] shuffleBy :: (a -> a -> m Bool) -> MutArray a -> MutArray a -> m ()
1183+ [A] roundUpToPower2 :: Int -> Int
1184+ [A] rightSize :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m (MutArray a)
1185+ [A] reverse :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m ()
1186+ [A] resizeExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1187+ [A] resize :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1188+ [A] realloc :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1189+ [A] readerRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Unfold m (MutArray a) a
1190+ [A] readerRev :: forall m a. (MonadIO m, Unbox a) => Unfold m (MutArray a) a
1191+ [A] reader :: forall m a. (MonadIO m, Unbox a) => Unfold m (MutArray a) a
1192+ [A] readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
1193+ [A] readIORef :: Unbox a => IORef a -> IO a
1194+ [A] read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
1195+ [A] putIndices :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Fold m (Int, a) ()
1196+ [A] putIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
1197+ [A] putIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
1198+ [A] producerWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Producer m (MutArray a) a
1199+ [A] producer :: forall m a. (MonadIO m, Unbox a) => Producer m (MutArray a) a
1200+ [A] pollIntIORef :: (MonadIO m, Unbox a) => IORef a -> Stream m a
1201+ [A] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1202+ [A] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (MutArray a)
1203+ [A] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1204+ [A] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1205+ [A] pinnedNewBytes :: MonadIO m => Int -> m (MutArray a)
1206+ [A] pinnedNewAligned :: (MonadIO m, Unbox a) => Int -> Int -> m (MutArray a)
1207+ [A] pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
1208+ [A] pinnedFromListN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1209+ [A] pinnedFromList :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1210+ [A] pinnedClone :: MonadIO m => MutArray a -> m (MutArray a)
1211+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1212+ [A] pin :: MutArray a -> IO (MutArray a)
1213+ [A] permute :: MutArray a -> m Bool
1214+ [A] partitionBy :: forall m a. (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> m (MutArray a, MutArray a)
1215+ [A] nil :: MutArray a
1216+ [A] newIORef :: forall a. Unbox a => a -> IO (IORef a)
1217+ [A] newArrayWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int -> m MutByteArray) -> Int -> Int -> m (MutArray a)
1218+ [A] new :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
1219+ [A] modifyIndices :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (Int -> a -> a) -> Fold m Int ()
1220+ [A] modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
1221+ [A] modifyIndex :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
1222+ [A] modifyIORef' :: Unbox a => IORef a -> (a -> a) -> IO ()
1223+ [A] modify :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (a -> a) -> m ()
1224+ [A] mergeBy :: Int -> (MutArray a -> MutArray a -> m ()) -> MutArray a -> m ()
1225+ [A] memcpy :: Ptr Word8 -> Ptr Word8 -> Int -> IO ()
1226+ [A] memcmp :: Ptr Word8 -> Ptr Word8 -> Int -> IO Bool
1227+ [A] length :: forall a. Unbox a => MutArray a -> Int
1228+ [A] isPinned :: MutArray a -> Bool
1229+ [A] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
1230+ [A] getSliceUnsafe :: forall a. Unbox a => Int -> Int -> MutArray a -> MutArray a
1231+ [A] getSlice :: forall a. Unbox a => Int -> Int -> MutArray a -> MutArray a
1232+ [A] getIndicesD :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
1233+ [A] getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1234+ [A] getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
1235+ [A] getIndexRev :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
1236+ [A] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
1237+ [A] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
1238+ [A] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
1239+ [A] fromStreamD :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
1240+ [A] fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
1241+ [A] fromPureStream :: (MonadIO m, Unbox a) => Stream Identity a -> m (MutArray a)
1242+ [A] fromListRevN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1243+ [A] fromListRev :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1244+ [A] fromListN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1245+ [A] fromList :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1246+ [A] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
1247+ [A] foldr :: (MonadIO m, Unbox a) => (a -> b -> b) -> b -> MutArray a -> m b
1248+ [A] foldl' :: (MonadIO m, Unbox a) => (b -> a -> b) -> b -> MutArray a -> m b
1249+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1250+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1251+ [A] divideBy :: Int -> (MutArray a -> m (MutArray a, MutArray a)) -> MutArray a -> m ()
1252+ [A] cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
1253+ [A] clone :: MonadIO m => MutArray a -> m (MutArray a)
1254+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1255+ [A] castUnsafe :: MutArray a -> MutArray b
1256+ [A] cast :: forall a b. Unbox b => MutArray a -> Maybe (MutArray b)
1257+ [A] c_memchr :: Ptr Word8 -> Word8 -> CSize -> IO (Ptr Word8)
1258+ [A] bytesFree :: MutArray a -> Int
1259+ [A] byteLength :: MutArray a -> Int
1260+ [A] byteCapacity :: MutArray a -> Int
1261+ [A] bubble :: (MonadIO m, Unbox a) => (a -> a -> Ordering) -> MutArray a -> m ()
1262+ [A] breakOn :: MonadIO m => Word8 -> MutArray Word8 -> m (MutArray Word8, Maybe (MutArray Word8))
1263+ [A] blockSize :: Int
1264+ [A] asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
1265+ [A] asBytes :: MutArray a -> MutArray Word8
1266+ [A] arrayChunkBytes :: Int
1267+ [A] allocBytesToElemCount :: Unbox a => a -> Int -> Int
1268+[C] Streamly.Internal.Data.IsMap
1269+ [A] mapTraverseWithKey :: (IsMap f, Applicative t) => (Key f -> a -> t b) -> f a -> t (f b)
1270+[R] Streamly.Internal.Data.IORef.Unboxed
1271+[R] Streamly.Internal.Data.Fold.Window
1272+[R] Streamly.Internal.Data.Fold.Type
1273+[R] Streamly.Internal.Data.Fold.Tee
1274+[R] Streamly.Internal.Data.Fold.Step
1275+[R] Streamly.Internal.Data.Fold.Container
1276+[D] Streamly.Internal.Data.Fold.Chunked
1277+[C] Streamly.Internal.Data.Fold
1278+ [A] ManyState
1279+ [C] Fold
1280+ [C] Fold
1281+ [O] Fold :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> Fold m a b
1282+ [N] Fold :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> (s -> m b) -> Fold m a b
1283+ [A] windowSumInt :: forall m a. (Monad m, Integral a) => Fold m (a, Maybe a) a
1284+ [A] windowSum :: forall m a. (Monad m, Num a) => Fold m (a, Maybe a) a
1285+ [A] windowRollingMapM :: Monad m => (Maybe a -> a -> m (Maybe b)) -> Fold m (a, Maybe a) (Maybe b)
1286+ [A] windowRollingMap :: Monad m => (Maybe a -> a -> Maybe b) -> Fold m (a, Maybe a) (Maybe b)
1287+ [A] windowRange :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe (a, a))
1288+ [A] windowPowerSumFrac :: (Monad m, Floating a) => a -> Fold m (a, Maybe a) a
1289+ [A] windowPowerSum :: (Monad m, Num a) => Int -> Fold m (a, Maybe a) a
1290+ [A] windowMinimum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
1291+ [A] windowMean :: forall m a. (Monad m, Fractional a) => Fold m (a, Maybe a) a
1292+ [A] windowMaximum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
1293+ [A] windowLmap :: (c -> a) -> Fold m (a, Maybe a) b -> Fold m (c, Maybe c) b
1294+ [A] windowLength :: (Monad m, Num b) => Fold m (a, Maybe a) b
1295+ [A] toSet :: (Monad m, Ord a) => Fold m a (Set a)
1296+ [A] toMapIO :: (MonadIO m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (Map k b)
1297+ [A] toMap :: (Monad m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (Map k b)
1298+ [A] toIntSet :: Monad m => Fold m Int IntSet
1299+ [A] toContainerIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (f b)
1300+ [A] toContainer :: (Monad m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (f b)
1301+ [A] nubInt :: Monad m => Fold m Int (Maybe Int)
1302+ [A] nub :: (Monad m, Ord a) => Fold m a (Maybe a)
1303+ [A] mapMStep :: Applicative m => (a -> m b) -> Step s a -> m (Step s b)
1304+ [A] kvToMap :: (Monad m, Ord k) => Fold m a b -> Fold m (k, a) (Map k b)
1305+ [A] frequency :: (Monad m, Ord a) => Fold m a (Map a Int)
1306+ [A] demuxToMapIO :: (MonadIO m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (Map k b)
1307+ [A] demuxToMap :: (Monad m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (Map k b)
1308+ [A] demuxToContainerIO :: (MonadIO m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
1309+ [A] demuxToContainer :: (Monad m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
1310+ [A] demuxKvToMap :: (Monad m, Ord k) => (k -> m (Fold m a b)) -> Fold m (k, a) (Map k b)
1311+ [A] demuxKvToContainer :: (Monad m, IsMap f, Traversable f) => (Key f -> m (Fold m a b)) -> Fold m (Key f, a) (f b)
1312+ [A] demuxIO :: (MonadIO m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (m (Map k b), Maybe (k, b))
1313+ [A] demuxGenericIO :: (MonadIO m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (m (f b), Maybe (Key f, b))
1314+ [A] demuxGeneric :: (Monad m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (m (f b), Maybe (Key f, b))
1315+ [A] demux :: (Monad m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (m (Map k b), Maybe (k, b))
1316+ [A] cumulative :: Fold m (a, Maybe a) b -> Fold m a b
1317+ [A] countDistinctInt :: Monad m => Fold m Int Int
1318+ [A] countDistinct :: (Monad m, Ord a) => Fold m a Int
1319+ [A] classifyIO :: (MonadIO m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
1320+ [A] classifyGenericIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
1321+ [A] classifyGeneric :: (Monad m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
1322+ [A] classify :: (Monad m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
1323+ [A] chainStepM :: Applicative m => (s1 -> m s2) -> (a -> m (Step s2 b)) -> Step s1 a -> m (Step s2 b)
1324+[C] Streamly.Internal.Data.Builder
1325+ [C] Builder
1326+ [C] Builder
1327+ [O] Builder :: (s -> m (s, a)) -> Builder s m a
1328+ [N] Builder :: (s -> m (a, s)) -> Builder s m a
1329+[A] Streamly.Internal.Data.Binary.Stream
1330+ [A] class ToBytes a
1331+ [A] word8 :: Applicative m => Word8 -> Stream m Word8
1332+ [A] word64le :: Monad m => Word64 -> Stream m Word8
1333+ [A] word64host :: Monad m => Word64 -> Stream m Word8
1334+ [A] word64be :: Monad m => Word64 -> Stream m Word8
1335+ [A] word32le :: Monad m => Word32 -> Stream m Word8
1336+ [A] word32be :: Monad m => Word32 -> Stream m Word8
1337+ [A] word16le :: Monad m => Word16 -> Stream m Word8
1338+ [A] word16be :: Monad m => Word16 -> Stream m Word8
1339+ [A] unit :: Applicative m => Stream m Word8
1340+ [A] toBytes :: ToBytes a => a -> Stream m Word8
1341+ [A] ordering :: Applicative m => Ordering -> Stream m Word8
1342+ [A] int8 :: Applicative m => Int8 -> Stream m Word8
1343+ [A] int64le :: Monad m => Int64 -> Stream m Word8
1344+ [A] int64be :: Monad m => Int64 -> Stream m Word8
1345+ [A] int32le :: Monad m => Int32 -> Stream m Word8
1346+ [A] int32be :: Monad m => Int32 -> Stream m Word8
1347+ [A] int16le :: Monad m => Int16 -> Stream m Word8
1348+ [A] int16be :: Monad m => Int16 -> Stream m Word8
1349+ [A] float32le :: Monad m => Float -> Stream m Word8
1350+ [A] float32be :: Monad m => Float -> Stream m Word8
1351+ [A] double64le :: Monad m => Double -> Stream m Word8
1352+ [A] double64be :: Monad m => Double -> Stream m Word8
1353+ [A] charUtf8 :: Monad m => Char -> Stream m Word8
1354+ [A] charLatin1 :: Applicative m => Char -> Stream m Word8
1355+ [A] bool :: Applicative m => Bool -> Stream m Word8
1356+[A] Streamly.Internal.Data.Binary.Parser
1357+ [A] class FromBytes a
1358+ [A] word8 :: Monad m => Parser Word8 m Word8
1359+ [A] word64le :: Monad m => Parser Word8 m Word64
1360+ [A] word64host :: MonadIO m => Parser Word8 m Word64
1361+ [A] word64be :: Monad m => Parser Word8 m Word64
1362+ [A] word32le :: Monad m => Parser Word8 m Word32
1363+ [A] word32be :: Monad m => Parser Word8 m Word32
1364+ [A] word16le :: Monad m => Parser Word8 m Word16
1365+ [A] word16be :: Monad m => Parser Word8 m Word16
1366+ [A] unit :: Monad m => Parser Word8 m ()
1367+ [A] ordering :: Monad m => Parser Word8 m Ordering
1368+ [A] int8 :: Monad m => Parser Word8 m Int8
1369+ [A] int64le :: Monad m => Parser Word8 m Int64
1370+ [A] int64be :: Monad m => Parser Word8 m Int64
1371+ [A] int32le :: Monad m => Parser Word8 m Int32
1372+ [A] int32be :: Monad m => Parser Word8 m Int32
1373+ [A] int16le :: Monad m => Parser Word8 m Int16
1374+ [A] int16be :: Monad m => Parser Word8 m Int16
1375+ [A] fromBytes :: FromBytes a => Parser Word8 m a
1376+ [A] float32le :: MonadIO m => Parser Word8 m Float
1377+ [A] float32be :: MonadIO m => Parser Word8 m Float
1378+ [A] eqWord8 :: Monad m => Word8 -> Parser Word8 m Word8
1379+ [A] double64le :: MonadIO m => Parser Word8 m Double
1380+ [A] double64be :: MonadIO m => Parser Word8 m Double
1381+ [A] charLatin1 :: Monad m => Parser Word8 m Char
1382+ [A] bool :: Monad m => Parser Word8 m Bool
1383+[R] Streamly.Internal.Data.Array.Type
1384+[A] Streamly.Internal.Data.Array.Stream
1385+ [A] unlines :: forall m a. (MonadIO m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1386+ [A] toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
1387+ [A] splitOnSuffix :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
1388+ [A] splitOn :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
1389+ [A] runArrayParserDBreak :: forall m a b. (MonadIO m, Unbox a) => Parser (Array a) m b -> Stream m (Array a) -> m (Either ParseError b, Stream m (Array a))
1390+ [A] runArrayFoldMany :: (Monad m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> StreamK m (Either ParseError b)
1391+ [A] runArrayFoldBreak :: (MonadIO m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1392+ [A] runArrayFold :: (MonadIO m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> m (Either ParseError b)
1393+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1394+ [A] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
1395+ [A] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1396+ [A] parseBreak :: (MonadIO m, Unbox a) => Parser a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1397+ [A] lpackArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
1398+ [A] interposeSuffix :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1399+ [A] interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1400+ [A] intercalateSuffix :: (Monad m, Unbox a) => Array a -> Stream m (Array a) -> Stream m a
1401+ [A] foldBreakD :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
1402+ [A] foldBreak :: (MonadIO m, Unbox a) => Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
1403+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1404+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1405+ [A] concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
1406+ [A] concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
1407+ [A] compact :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
1408+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1409+ [A] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
1410+[R] Streamly.Internal.Data.Array.Mut.Type
1411+[R] Streamly.Internal.Data.Array.Mut.Stream
1412+[R] Streamly.Internal.Data.Array.Mut
1413+[R] Streamly.Internal.Data.Array.Generic.Mut.Type
1414+[C] Streamly.Internal.Data.Array.Generic
1415+ [A] getIndex :: Int -> Array a -> Maybe a
1416+ [A] fromPureStream :: Stream Identity a -> Array a
1417+ [A] fromByteStr# :: Addr# -> Array Word8
1418+ [A] chunksOf :: forall m a. MonadIO m => Int -> Stream m a -> Stream m (Array a)
1419+[C] Streamly.Internal.Data.Array
1420+ [A] ArrayUnsafe
1421+ [A] ArrayUnsafe :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> ArrayUnsafe a
1422+ [C] Array
1423+ [A] [arrStart] :: Array a -> {-# UNPACK #-} !Int
1424+ [A] [arrEnd] :: Array a -> {-# UNPACK #-} !Int
1425+ [A] [arrContents] :: Array a -> {-# UNPACK #-} !MutByteArray
1426+ [A] Array :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> Array a
1427+ [A] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1428+ [A] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1429+ [R] writeNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
1430+ [A] unsafeMakePure :: Monad m => Fold IO a b -> Fold m a b
1431+ [A] unsafeIndexIO :: forall a. Unbox a => Int -> Array a -> IO a
1432+ [D] unsafeIndex :: forall a. Unbox a => Int -> Array a -> a
1433+ [A] unsafeFreezeWithShrink :: Unbox a => MutArray a -> Array a
1434+ [A] unpin :: Array a -> IO (Array a)
1435+ [A] toStreamKRev :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a
1436+ [A] toStreamK :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a
1437+ [A] toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
1438+ [A] toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
1439+ [A] splitAt :: Unbox a => Int -> Array a -> (Array a, Array a)
1440+ [A] splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
1441+ [A] serialize :: Serialize a => a -> Array Word8
1442+ [A] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1443+ [A] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
1444+ [A] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1445+ [A] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
1446+ [A] pinnedSerialize :: Serialize a => a -> Array Word8
1447+ [A] pinnedFromListN :: Unbox a => Int -> [a] -> Array a
1448+ [A] pinnedFromList :: Unbox a => [a] -> Array a
1449+ [A] pinnedClone :: MonadIO m => Array a -> m (Array a)
1450+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1451+ [A] pin :: Array a -> IO (Array a)
1452+ [A] nil :: Array a
1453+ [A] isPinned :: Array a -> Bool
1454+ [A] getIndexUnsafe :: forall a. Unbox a => Int -> Array a -> a
1455+ [A] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
1456+ [A] fromStreamD :: forall m a. (MonadIO m, Unbox a) => Stream m a -> m (Array a)
1457+ [A] fromPureStream :: Unbox a => Stream Identity a -> Array a
1458+ [A] fromListRevN :: Unbox a => Int -> [a] -> Array a
1459+ [A] fromListRev :: Unbox a => [a] -> Array a
1460+ [A] fromByteStr# :: Addr# -> Array Word8
1461+ [A] foldr :: Unbox a => (a -> b -> b) -> b -> Array a -> b
1462+ [A] foldl' :: forall a b. Unbox a => (b -> a -> b) -> b -> Array a -> b
1463+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1464+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1465+ [A] encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
1466+ [A] deserialize :: Serialize a => Array Word8 -> a
1467+ [A] clone :: MonadIO m => Array a -> m (Array a)
1468+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1469+ [A] byteLength :: Array a -> Int
1470+ [A] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
1471+ [A] breakOn :: MonadIO m => Word8 -> Array Word8 -> m (Array Word8, Maybe (Array Word8))
core/docs/ApiChangelogs/0.2.0-0.2.2.txtnew file+306
@@ -0,0 +1,306 @@
1+---------------------------------
2+API Annotations
3+---------------------------------
4+
5+[A] : Added
6+[R] : Removed
7+[C] : Changed
8+[O] : Old definition
9+[N] : New definition
10+[D] : Deprecated
11+
12+---------------------------------
13+API diff
14+---------------------------------
15+
16+[C] Streamly.Data.Stream
17+ [A] (FixityR,5)
18+ [A] (FixityR,5)
19+[C] Streamly.Data.MutArray.Generic
20+ [A] emptyOf :: MonadIO m => Int -> m (MutArray a)
21+ [A] createOf :: MonadIO m => Int -> Fold m a (MutArray a)
22+ [A] create :: MonadIO m => Fold m a (MutArray a)
23+[C] Streamly.Data.MutArray
24+ [A] pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
25+ [A] emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
26+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
27+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
28+ [A] appendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
29+ [A] append :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
30+[C] Streamly.Data.Array.Generic
31+ [A] createOf :: MonadIO m => Int -> Fold m a (Array a)
32+ [A] create :: MonadIO m => Fold m a (Array a)
33+[C] Streamly.Data.Array
34+ [A] class Serialize a
35+ [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
36+ [A] pinnedSerialize :: Serialize a => a -> Array Word8
37+ [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
38+ [A] deserialize :: Serialize a => Array Word8 -> a
39+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
40+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
41+ [A] addSizeTo :: Serialize a => Int -> a -> Int
42+
43+---------------------------------
44+Internal API diff
45+---------------------------------
46+
47+[C] Streamly.Internal.Unicode.Stream
48+ [A] encodeUtf16le' :: Stream m Char -> Stream m Word16
49+ [A] decodeUtf16le' :: Stream m Word16 -> Stream m Char
50+[A] Streamly.Internal.FileSystem.Path
51+ [A] class IsPath a
52+ [A] Rel
53+ [A] File
54+ [A] Dir
55+ [A] Abs
56+ [A] Streamly.Internal.FileSystem.Path.IsPath
57+ [A] instance Streamly.Internal.FileSystem.Path.IsPath Streamly.Internal.FileSystem.Path.Path
58+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel Streamly.Internal.FileSystem.Path.Path)
59+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path))
60+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path))
61+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path)
62+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path)
63+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs Streamly.Internal.FileSystem.Path.Path)
64+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path))
65+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path))
66+ [A] GHC.Show.Show
67+ [A] instance GHC.Show.Show Streamly.Internal.FileSystem.Path.PathException
68+ [A] GHC.Exception.Type.Exception
69+ [A] instance GHC.Exception.Type.Exception Streamly.Internal.FileSystem.Path.PathException
70+ [A] GHC.Classes.Eq
71+ [A] instance GHC.Classes.Eq Streamly.Internal.FileSystem.Path.PathException
72+ [A] Path
73+ [A] Path :: Array Word8 -> Path
74+ [A] toString :: Path -> [Char]
75+ [A] toPath :: IsPath a => a -> Path
76+ [A] toChunk :: Path -> Array Word8
77+ [A] toChars :: Monad m => Path -> Stream m Char
78+ [A] relfile :: QuasiQuoter
79+ [A] reldir :: QuasiQuoter
80+ [A] rel :: QuasiQuoter
81+ [A] primarySeparator :: Char
82+ [A] path :: QuasiQuoter
83+ [A] mkRelFile :: String -> Q Exp
84+ [A] mkRelDir :: String -> Q Exp
85+ [A] mkRel :: String -> Q Exp
86+ [A] mkPath :: String -> Q Exp
87+ [A] mkFile :: String -> Q Exp
88+ [A] mkDir :: String -> Q Exp
89+ [A] mkAbsFile :: String -> Q Exp
90+ [A] mkAbsDir :: String -> Q Exp
91+ [A] mkAbs :: String -> Q Exp
92+ [A] isSeparator :: Char -> Bool
93+ [A] fromString :: MonadThrow m => [Char] -> m Path
94+ [A] fromPathUnsafe :: IsPath a => Path -> a
95+ [A] fromPath :: (IsPath a, MonadThrow m) => Path -> m a
96+ [A] fromChunkUnsafe :: Array Word8 -> Path
97+ [A] fromChunk :: MonadThrow m => Array Word8 -> m Path
98+ [A] fromChars :: MonadThrow m => Stream Identity Char -> m Path
99+ [A] file :: QuasiQuoter
100+ [A] extendPath :: Path -> Path -> Path
101+ [A] extendDir :: (IsPath (a (Dir Path)), IsPath b, IsPath (a b)) => a (Dir Path) -> Rel b -> a b
102+ [A] dir :: QuasiQuoter
103+ [A] adaptPath :: (MonadThrow m, IsPath a, IsPath b) => a -> m b
104+ [A] absfile :: QuasiQuoter
105+ [A] absdir :: QuasiQuoter
106+ [A] abs :: QuasiQuoter
107+[C] Streamly.Internal.Data.Stream
108+ [A] (FixityR,5)
109+ [A] (FixityR,5)
110+ [C] splitInnerBySuffix
111+ [O] splitInnerBySuffix :: (Monad m, Eq (f a), Monoid (f a)) => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
112+ [N] splitInnerBySuffix :: Monad m => (f a -> Bool) -> (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
113+ [D] sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
114+ [A] indexOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
115+[C] Streamly.Internal.Data.MutByteArray
116+ [A] unsafePinnedAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
117+ [A] unsafeAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
118+ [D] nil :: MutByteArray
119+ [A] empty :: MutByteArray
120+ [D] asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
121+[D] Streamly.Internal.Data.MutArray.Stream
122+ [D] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
123+ [D] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
124+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
125+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
126+[C] Streamly.Internal.Data.MutArray.Generic
127+ [D] writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
128+ [D] writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
129+ [A] unsafeCreateOf :: MonadIO m => Int -> Fold m a (MutArray a)
130+ [A] emptyOf :: MonadIO m => Int -> m (MutArray a)
131+ [A] createWith :: MonadIO m => Int -> Fold m a (MutArray a)
132+ [A] createOf :: MonadIO m => Int -> Fold m a (MutArray a)
133+ [A] create :: MonadIO m => Fold m a (MutArray a)
134+[C] Streamly.Internal.Data.MutArray
135+ [A] SpliceState
136+ [A] SpliceYielding :: arr -> SpliceState s arr -> SpliceState s arr
137+ [A] SpliceInitial :: s -> SpliceState s arr
138+ [A] SpliceFinish :: SpliceState s arr
139+ [A] SpliceBuffering :: s -> arr -> SpliceState s arr
140+ [R] MutByteArray
141+ [D] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
142+ [D] writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
143+ [D] writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
144+ [D] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
145+ [D] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
146+ [D] writeAppendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
147+ [D] writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
148+ [A] unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
149+ [A] unsafePinnedAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
150+ [A] unsafeCreateOfWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
151+ [A] unsafeCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
152+ [A] unsafeAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
153+ [A] unsafeAppendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
154+ [A] toStreamWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
155+ [A] toStreamRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
156+ [R] toStreamDWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
157+ [R] toStreamDRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
158+ [A] slicerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
159+ [A] sliceIndexerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
160+ [A] revCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
161+ [D] resizeExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
162+ [D] resize :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
163+ [A] pokeSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
164+ [A] pokeAppendMay :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> a -> m (Maybe (MutArray Word8))
165+ [A] pokeAppend :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> a -> m (MutArray Word8)
166+ [D] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
167+ [D] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
168+ [D] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
169+ [D] pinnedNewBytes :: MonadIO m => Int -> m (MutArray a)
170+ [A] pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
171+ [A] pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
172+ [A] pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
173+ [A] pinnedCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
174+ [A] peekUnconsUnsafe :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> m (a, MutArray Word8)
175+ [A] peekUncons :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> m (Maybe a, MutArray Word8)
176+ [A] peekSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
177+ [A] pPinnedCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Parser (MutArray a) m (MutArray a)
178+ [A] pCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Parser (MutArray a) m (MutArray a)
179+ [D] nil :: MutArray a
180+ [A] lPinnedCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
181+ [A] lCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
182+ [A] indexReaderWith :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
183+ [A] indexReader :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
184+ [A] growExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
185+ [A] grow :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
186+ [D] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
187+ [R] getIndicesD :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
188+ [D] getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
189+ [D] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
190+ [A] fromStreamN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
191+ [D] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
192+ [D] fromStreamD :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
193+ [A] fromPureStreamN :: (MonadIO m, Unbox a) => Int -> Stream Identity a -> m (MutArray a)
194+ [A] fromPtrN :: MonadIO m => Int -> Ptr Word8 -> m (MutArray Word8)
195+ [A] fromChunksRealloced :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> m (MutArray a)
196+ [A] fromChunksK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
197+ [A] fromByteStr# :: MonadIO m => Addr# -> m (MutArray Word8)
198+ [D] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
199+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
200+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
201+ [A] fPinnedCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) (MutArray a)
202+ [A] fCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) (MutArray a)
203+ [A] emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
204+ [A] empty :: MutArray a
205+ [A] createWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
206+ [A] createOfWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
207+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
208+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
209+ [A] concatWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m (MutArray a) -> Stream m a
210+ [A] concatRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m (MutArray a) -> Stream m a
211+ [A] concatRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
212+ [A] concat :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
213+ [A] compactOnByteSuffix :: MonadIO m => Word8 -> Stream m (MutArray Word8) -> Stream m (MutArray Word8)
214+ [A] compactOnByte :: MonadIO m => Word8 -> Stream m (MutArray Word8) -> Stream m (MutArray Word8)
215+ [A] compactLeAs :: forall m a. (MonadIO m, Unbox a) => PinnedState -> Int -> Stream m (MutArray a) -> Stream m (MutArray a)
216+ [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
217+ [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
218+ [A] compactEQ :: Int -> Stream m (MutArray a) -> Stream m (MutArray a)
219+ [D] cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
220+ [A] byteEq :: MonadIO m => MutArray a -> MutArray a -> m Bool
221+ [A] byteCmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
222+ [A] buildChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
223+ [D] asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
224+ [A] appendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
225+ [A] appendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
226+ [A] append :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
227+[D] Streamly.Internal.Data.Array.Stream
228+ [C] interposeSuffix
229+ [O] interposeSuffix :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
230+ [N] interposeSuffix :: forall m a. (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
231+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
232+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
233+ [C] concatRev
234+ [O] concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
235+ [N] concatRev :: forall m a. (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
236+ [D] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
237+[C] Streamly.Internal.Data.Array.Generic
238+ [A] createOf :: MonadIO m => Int -> Fold m a (Array a)
239+ [A] create :: MonadIO m => Fold m a (Array a)
240+[C] Streamly.Internal.Data.Array
241+ [R] ArrayUnsafe
242+ [D] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
243+ [D] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
244+ [A] unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
245+ [A] unsafePinnedAsPtr :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
246+ [A] unsafeCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
247+ [D] toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
248+ [D] toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
249+ [C] splice
250+ [O] splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
251+ [N] splice :: MonadIO m => Array a -> Array a -> m (Array a)
252+ [A] slicerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Array a)
253+ [A] sliceIndexerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Int, Int)
254+ [D] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
255+ [D] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
256+ [D] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
257+ [D] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
258+ [A] pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
259+ [A] pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
260+ [A] pinnedCompactLE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
261+ [A] parseBreakChunksK :: forall m a b. (MonadIO m, Unbox a) => Parser a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
262+ [D] nil :: Array a
263+ [A] lPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
264+ [A] lCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
265+ [A] interposeSuffix :: forall m a. (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
266+ [A] interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
267+ [A] intercalateSuffix :: (Monad m, Unbox a) => Array a -> Stream m (Array a) -> Stream m a
268+ [A] indexReaderFromThenTo :: Unfold m (Int, Int, Int, Array a) a
269+ [A] indexReader :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
270+ [A] indexFinder :: (a -> Bool) -> Unfold Identity (Array a) Int
271+ [D] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Array a)
272+ [R] getIndicesFromThenTo :: Unfold m (Int, Int, Int, Array a) a
273+ [D] getIndices :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
274+ [D] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Int, Int)
275+ [D] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
276+ [D] fromStreamD :: forall m a. (MonadIO m, Unbox a) => Stream m a -> m (Array a)
277+ [A] fromPureStreamN :: Unbox a => Int -> Stream Identity a -> Array a
278+ [A] fromPtrN :: Int -> Ptr Word8 -> Array Word8
279+ [A] fromChunksK :: (MonadIO m, Unbox a) => StreamK m (Array a) -> m (Array a)
280+ [A] fromChunks :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
281+ [A] fromByteStr :: Ptr Word8 -> Array Word8
282+ [A] foldChunks :: (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m b
283+ [A] foldBreakChunksK :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
284+ [A] foldBreakChunks :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
285+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
286+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
287+ [C] findIndicesOf
288+ [O] findIndicesOf :: (a -> Bool) -> Unfold Identity (Array a) Int
289+ [N] findIndicesOf :: (a -> Bool) -> Array a -> Stream Identity Int
290+ [A] fPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
291+ [A] fCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
292+ [A] empty :: Array a
293+ [A] createWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
294+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
295+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
296+ [A] concatRev :: forall m a. (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
297+ [A] concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
298+ [A] compactOnByteSuffix :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
299+ [A] compactOnByte :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
300+ [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
301+ [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
302+ [A] byteEq :: Array a -> Array a -> Bool
303+ [A] byteCmp :: Array a -> Array a -> Ordering
304+ [A] buildChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
305+ [D] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
306+ [D] asPtrUnsafe :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
core/docs/Changelog.md+58−16
@@ -1,31 +1,73 @@
11# Changelog
22
3−## Unreleased
3+## 0.2.2 (Jan 2024)
44
5−* Arrays are now created unpinned by default, they were created pinned
6− earlier. During IO operations unpinned arrays are automatically copied
7− to pinned memory. When arrays are directly passed to IO operations
8− programmers can choose to create them pinned to avoid a copy. To
9− create pinned arrays, use the internal APIs with the `pinned*` prefix.
5+* Add fixities `infixr 5` for `cons` and `consM` functions.
6+* Fix a bug in Array `Eq` instance when the type is a sum type with
7+ differently sized constructors.
8+* lpackArraysChunksOf, compact, writeChunksWith, putChunksWith now take the
9+ buffer size in number of array elements instead of bytes.
1010
11−* Changed the signature of 'Streamly.Data.Stream.handle' to make the exception
12− handler monadic.
13−* Rethrow the exception promptly in bracketIO.
11+## 0.2.1 (Dec 2023)
1412
15−### Breaking
13+* Make the serialization of the unit constructor deterministic.
14+* Expose `pinnedSerialize` & `deserialize` via `Streamly.Data.Array`.
1615
17−* `ParserK` in `Streamly.Data.ParserK` is not specialized to arrays anymore. To
18− adapt to the new code, change `ParserK a m b` to `ParserK (Array a) m b` where
19− `Array` comes from `Streamly.Data.Array`.
16+## 0.2.0 (Nov 2023)
17+
18+See [0.1.0-0.2.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0-0.2.0.txt)
19+for a full list of API changes in this release. Only a few significant
20+changes are mentioned here.
21+
22+### Breaking Changes
23+
24+* `ParserK` in `Streamly.Data.ParserK` is not implicitly specialized
25+ to arrays anymore. To adapt to the new code, change `ParserK a m
26+ b` to `ParserK (Array a) m b` where the `Array` type comes from
27+ `Streamly.Data.Array`. This change also affected the signatures of
28+ `parseChunks` and `parseBreakChunks`.
29+* Changed the signature of 'Streamly.Data.Stream.handle' to make the
30+ exception handler monadic.
31+* Behavior change: Exceptions are now rethrown promptly in `bracketIO`.
32+
33+### Enhancements
34+
35+* __Serialization__: Added a `Streamly.Data.MutByteArray` module with a
36+ `Serialize` type class for fast binary serialization. The Data.Array
37+ module supplies the `serialize` and `deserialize` operations for arrays.
38+* __Unpinned Arrays__: Unboxed arrays are now created unpinned by default,
39+ they were created pinned earlier. During IO operations, unpinned arrays
40+ are automatically copied to pinned memory. When arrays are directly
41+ passed to IO operations programmers can choose to create them pinned to
42+ avoid a copy. To create pinned arrays, use the internal APIs with the
43+ `pinned*` prefix.
44+* StreamK now supports native exception handling routines (handle, bracketIO).
45+ Earlier we had to convert it to the `Stream` type for exception handling.
2046
2147### Deprecations
2248
23−* `Streamly.Data.MutArray.newUnpinned` is renamed to
24− `Streamly.Data.MutArray.pinnedNew`
49+See [0.1.0-0.2.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0-0.2.0.txt)
50+for a full list of deprecations.
51+
52+### Internal API Changes
53+
54+* Fold constructor has changed, added a `final` field to support
55+ finalization and cleanup of a chain of folds. The `extract` field is
56+ now used only for mapping the fold internal state to fold result for
57+ scanning purposes. If your fold does not require cleanup you can just use
58+ your existing `extract` function as `final` as well to adapt to this change.
59+* Many low level internal modules have been removed, they are entirely
60+ exported from higher level internal modules. If you were importing any
61+ of the missing low level modules then import the higher level modules instead.
62+* Internal module changes:
63+ * Streamly.Internal.Serialize.FromBytes -> Streamly.Internal.Data.Binary.Parser
64+ * Streamly.Internal.Serialize.ToBytes -> Streamly.Internal.Data.Binary.Stream
65+ * Streamly.Internal.Data.Unbox is now exported via Streamly.Internal.Data.Serialize
66+ * Streamly.Internal.Data.IORef.Unboxed is now exported via Streamly.Internal.Data.Serialize
2567
2668## 0.1.0 (March 2023)
2769
28−Also see [streamly-core-0.1.0 API Changelog](/core/docs/ApiChangelogs/0.1.0.txt) or
70+Also see [streamly-core-0.1.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0.txt) or
2971https://hackage.haskell.org/package/streamly-core-0.1.0/docs/docs/ApiChangelogs/0.1.0.txt
3072
3173`streamly` package is split into two packages, (1) `streamly-core` that
core/src/DocTestDataArray.hs+3
@@ -11,9 +11,12 @@
1111
1212>>> import qualified Streamly.Data.Array as Array
1313>>> import qualified Streamly.Data.Fold as Fold
14+>>> import qualified Streamly.Data.ParserK as ParserK
1415>>> import qualified Streamly.Data.Stream as Stream
16+>>> import qualified Streamly.Data.StreamK as StreamK
1517
1618For APIs that have not been released yet.
1719
1820>>> import qualified Streamly.Internal.Data.Array as Array
21+>>> import qualified Streamly.Internal.Data.Stream as Stream
1922-}
core/src/DocTestDataParserK.hsnew file+15
@@ -0,0 +1,15 @@
1+{- $setup
2+>>> :m
3+>>> import Control.Applicative ((<|>))
4+>>> import Data.Char (isDigit, isAlpha)
5+
6+>>> import Streamly.Data.Parser (Parser)
7+>>> import Streamly.Data.ParserK (ParserK)
8+
9+>>> import qualified Streamly.Data.Parser as Parser
10+>>> import qualified Streamly.Data.ParserK as ParserK
11+
12+For APIs that have not been released yet.
13+
14+>>> import qualified Streamly.Internal.Data.ParserK as ParserK
15+-}
core/src/DocTestDataStream.hs+2
@@ -6,6 +6,7 @@
66>>> import Control.Monad.IO.Class (MonadIO (liftIO))
77>>> import Control.Monad.Trans.Class (lift)
88>>> import Control.Monad.Trans.Identity (runIdentityT)
9+>>> import Data.Char (isSpace)
910>>> import Data.Either (fromLeft, fromRight, isLeft, isRight, either)
1011>>> import Data.Maybe (fromJust, isJust)
1112>>> import Data.Function (fix, (&))
@@ -32,6 +33,7 @@ For APIs that have not been released yet.
3233>>> import qualified Streamly.Internal.Data.Fold as Fold
3334>>> import qualified Streamly.Internal.Data.Parser as Parser
3435>>> import qualified Streamly.Internal.Data.Stream as Stream
36+>>> import qualified Streamly.Internal.Data.StreamK as StreamK
3537>>> import qualified Streamly.Internal.Data.Unfold as Unfold
3638>>> import qualified Streamly.Internal.FileSystem.Dir as Dir
3739-}
core/src/DocTestFileSystemHandle.hs+1−1
@@ -8,7 +8,7 @@
88
99For APIs that have not been released yet.
1010
11−>>> import qualified Streamly.Internal.Data.Array as Array (writeNUnsafe)
11+>>> import qualified Streamly.Internal.Data.Array as Array (unsafeCreateOf)
1212>>> import qualified Streamly.Internal.Data.Unfold as Unfold (first)
1313>>> import qualified Streamly.Internal.FileSystem.Handle as Handle
1414>>> import qualified Streamly.Internal.System.IO as IO (defaultChunkSize)
core/src/Streamly/Console/Stdio.hs+1−27
@@ -24,33 +24,7 @@ module Streamly.Console.Stdio
2424 -- * Write (stderr)
2525 , writeErr
2626 , writeErrChunks
27−
28− -- * Deprecated
29− , read
30− , readChunks
3127 )
3228where
3329
34−import Control.Monad.IO.Class (MonadIO(..))
35−import Data.Word (Word8)
36−import Streamly.Internal.Data.Array.Type (Array)
37−import Streamly.Internal.Data.Unfold (Unfold)
38−
39−import Streamly.Internal.Console.Stdio hiding (read, readChunks)
40−import Prelude hiding (read)
41−
42−-- Same as 'reader'
43−--
44−-- @since 0.8.0
45−{-# DEPRECATED read "Please use 'reader' instead" #-}
46−{-# INLINE read #-}
47−read :: MonadIO m => Unfold m () Word8
48−read = reader
49−
50−-- Same as 'chunkReader'
51−--
52−-- @since 0.8.0
53−{-# DEPRECATED readChunks "Please use 'chunkReader' instead" #-}
54−{-# INLINE readChunks #-}
55−readChunks :: MonadIO m => Unfold m () (Array Word8)
56−readChunks = chunkReader
30+import Streamly.Internal.Console.Stdio
core/src/Streamly/Data/Array.hs+40−39
@@ -31,28 +31,45 @@ module Streamly.Data.Array
3131 -- * The Array Type
3232 Array
3333
34+ -- * Pinning & Unpinning
35+ -- | Arrays are created unpinned by default unless pinned versions of
36+ -- creation APIs are used. Look for APIs with @pinned@ prefix in
37+ -- "Streamly.Internal.Data.Array" for some unreleased pinned creation APIs.
38+ -- If an array is to be sent to the OS without any further modification
39+ -- then it should be created pinned in the first place instead of pinning
40+ -- it later. Pinning an unpinned array has a copy overhead. OS interfacing
41+ -- APIs create a pinned array directly or convert an unpinned array to
42+ -- pinned array before sending it to the OS.
43+ , pin
44+ , unpin
45+ , isPinned
46+
3447 -- * Construction
3548 -- | When performance matters, the fastest way to generate an array is
36− -- 'writeN'. 'IsList' and 'IsString' instances can be
49+ -- 'createOf'. 'IsList' and 'IsString' instances can be
3750 -- used to conveniently construct arrays from literal values.
3851 -- 'OverloadedLists' extension or 'fromList' can be used to construct an
3952 -- array from a list literal. Similarly, 'OverloadedStrings' extension or
4053 -- 'fromList' can be used to construct an array from a string literal.
4154
42− -- Pure List APIs
55+ -- ** From Stream
56+ , createOf
57+ , create
58+ , writeLastN -- drop old (ring buffer)
59+
60+ -- ** From List
4361 , fromListN
4462 , fromList
4563
46− -- Monadic APIs
47− , writeN -- drop new
48− , write -- full buffer
49− , writeLastN -- drop old (ring buffer)
50−
51− -- * Conversion
64+ -- * To List
5265 -- 'GHC.Exts.toList' from "GHC.Exts" can be used to convert an array to a
5366 -- list.
5467 , toList
5568
69+ -- * To Stream
70+ , read
71+ , readRev
72+
5673 -- * Unfolds
5774 , reader
5875 , readerRev
@@ -66,25 +83,24 @@ module Streamly.Data.Array
6683 -- , (!!)
6784 , getIndex
6885
69− -- * Unbox Type Class
70− , Unbox (..)
86+ -- * Serialization
87+ , pinnedSerialize
88+ , deserialize
7189
72− -- * Pinning & Unpinning
73− , pin
74− , unpin
75− , isPinned
90+ -- * Re-exports
91+ , Unbox (..)
92+ , Serialize(..)
7693
7794 -- * Deprecated
78− , read
79− , readRev
95+ , writeN -- drop new
96+ , write -- full buffer
8097 )
8198where
8299
83100#include "inline.hs"
84101
85−import Streamly.Internal.Data.Array hiding (read, readRev)
86−import Streamly.Internal.Data.Unbox (Unbox (..))
87−import Streamly.Internal.Data.Unfold (Unfold)
102+import Streamly.Internal.Data.Array
103+import Streamly.Internal.Data.MutByteArray (Unbox(..), Serialize(..))
88104
89105import Prelude hiding (read, length)
90106
@@ -102,7 +118,7 @@ import Prelude hiding (read, length)
102118--
103119-- Convert array to stream, and fold the stream:
104120--
105−-- >>> fold f arr = Stream.unfold Array.reader arr & Stream.fold f
121+-- >>> fold f arr = Array.read arr & Stream.fold f
106122-- >>> fold Fold.sum (Array.fromList [1,2,3::Int])
107123-- 6
108124--
@@ -110,7 +126,7 @@ import Prelude hiding (read, length)
110126--
111127-- Convert array to stream, transform, and fold back to array:
112128--
113−-- >>> amap f arr = Stream.unfold Array.reader arr & fmap f & Stream.fold Array.write
129+-- >>> amap f arr = Array.read arr & fmap f & Stream.fold (Array.createOf (Array.length arr))
114130-- >>> amap (+1) (Array.fromList [1,2,3::Int])
115131-- fromList [2,3,4]
116132--
@@ -137,12 +153,11 @@ import Prelude hiding (read, length)
137153-- See the 'fromPureStream' unreleased API to generate an array from an
138154-- Identity stream safely without using MonadIO constraint.
139155--
140−--
141156-- Note that 'Identity' streams can be generalized to IO streams:
142157--
143−-- >>> pure = Stream.fromList [1,2,3] :: Stream Identity Int
144−-- >>> generally = Stream.morphInner (return . runIdentity)
145−-- >>> Stream.fold Array.write (generally pure :: Stream IO Int)
158+-- >>> stream = Stream.fromList [1,2,3] :: Stream Identity Int
159+-- >>> pure2IO s = Stream.morphInner (return . runIdentity) s :: Stream IO Int
160+-- >>> Stream.fold Array.create (pure2IO stream)
146161-- fromList [1,2,3]
147162--
148163-- == Programming Tips
@@ -150,17 +165,3 @@ import Prelude hiding (read, length)
150165-- This module is designed to be imported qualified:
151166--
152167-- >>> import qualified Streamly.Data.Array as Array
153−
154−-- | Same as 'reader'
155−--
156−{-# DEPRECATED read "Please use 'reader' instead" #-}
157−{-# INLINE_NORMAL read #-}
158−read :: (Monad m, Unbox a) => Unfold m (Array a) a
159−read = reader
160−
161−-- | Same as 'readerRev'
162−--
163−{-# DEPRECATED readRev "Please use 'readerRev' instead" #-}
164−{-# INLINE_NORMAL readRev #-}
165−readRev :: (Monad m, Unbox a) => Unfold m (Array a) a
166−readRev = readerRev
core/src/Streamly/Data/Array/Generic.hs+16−13
@@ -15,34 +15,37 @@ module Streamly.Data.Array.Generic
1515 ( Array
1616
1717 -- * Construction
18− , A.fromListN
19− , A.fromList
18+ , fromListN
19+ , fromList
2020
2121 -- MonadicAPIs
22− , A.writeN
23− , A.write
22+ , createOf
23+ , create
2424
2525 -- * Conversion
26− , A.toList
26+ , toList
2727
2828 -- * Streams
29− , A.read
30− , A.readRev
29+ , read
30+ , readRev
3131
3232 -- * Unfolds
33− , A.reader
33+ , reader
3434 -- , A.readerRev
3535
3636 -- * Random Access
37− , A.length
38− , A.getIndex
37+ , length
38+ , getIndex
3939
4040 -- -- * Folding Arrays
4141 -- , A.streamFold
4242 -- , A.fold
43+
44+ -- * Deprecated
45+ , writeN
46+ , write
4347 )
4448where
4549
46−import Streamly.Internal.Data.Array.Generic (Array)
47−
48−import qualified Streamly.Internal.Data.Array.Generic as A
50+import Streamly.Internal.Data.Array.Generic
51+import Prelude hiding (length, read)
core/src/Streamly/Data/Fold.hs+150−118
@@ -10,6 +10,156 @@
1010-- Fast, composable stream consumers with ability to terminate, supporting
1111-- stream fusion.
1212--
13+-- == Using Folds
14+--
15+-- This module provides elementary folds and fold combinators that can be used
16+-- to consume a stream of data and reduce it to a final value, or transform it
17+-- in a stateful manner using scans. A data stream can be reduced into a stream
18+-- of folded data elements by folding segments of the stream. Fold combinators
19+-- can be used to compose multiple folds in parallel or to create a pipeline of
20+-- folds such that the next fold consumes the result of the previous fold. To
21+-- run these folds on a stream see 'Streamly.Data.Stream.fold',
22+-- 'Streamly.Data.Stream.scan', 'Streamly.Data.Stream.postscan',
23+-- 'Streamly.Data.Stream.scanMaybe', 'Streamly.Data.Stream.foldMany' and other
24+-- operations accepting 'Fold' type as argument "Streamly.Data.Stream".
25+--
26+-- == Reducing a Stream
27+--
28+-- A 'Fold' is a consumer of a stream of values. A fold driver (such as
29+-- 'Streamly.Data.Stream.fold') initializes the fold @accumulator@, runs the
30+-- fold @step@ function in a loop, processing the input stream one element at a
31+-- time and accumulating the result. The loop continues until the fold
32+-- terminates, at which point the accumulated result is returned.
33+--
34+-- For example, a 'sum' Fold represents a stream consumer that adds the values
35+-- in the input stream:
36+--
37+-- >>> Stream.fold Fold.sum $ Stream.fromList [1..100]
38+-- 5050
39+--
40+-- Conceptually, a 'Fold' is a data type that mimics a strict left fold
41+-- ('Data.List.foldl'). The above example is similar to a left fold using
42+-- @(+)@ as the step and @0@ as the initial value of the accumulator:
43+--
44+-- >>> Data.List.foldl' (+) 0 [1..100]
45+-- 5050
46+--
47+-- 'Fold's have an early termination capability e.g. the 'one' fold terminates
48+-- after consuming one element:
49+--
50+-- >>> Stream.fold Fold.one $ Stream.fromList [1..]
51+-- Just 1
52+--
53+-- The above example is similar to the following right fold:
54+--
55+-- >>> Prelude.foldr (\x _ -> Just x) Nothing [1..]
56+-- Just 1
57+--
58+-- 'Fold's can be combined together using combinators. For example, to create a
59+-- fold that sums first two elements in a stream:
60+--
61+-- >>> sumTwo = Fold.take 2 Fold.sum
62+-- >>> Stream.fold sumTwo $ Stream.fromList [1..100]
63+-- 3
64+--
65+-- == Parallel Composition
66+--
67+-- Folds can be combined to run in parallel on the same input. For example, to
68+-- compute the average of numbers in a stream without going through the stream
69+-- twice:
70+--
71+-- >>> avg = Fold.teeWith (/) Fold.sum (fmap fromIntegral Fold.length)
72+-- >>> Stream.fold avg $ Stream.fromList [1.0..100.0]
73+-- 50.5
74+--
75+-- Folds can be combined so as to partition the input stream over multiple
76+-- folds. For example, to count even and odd numbers in a stream:
77+--
78+-- >>> split n = if even n then Left n else Right n
79+-- >>> stream = fmap split $ Stream.fromList [1..100]
80+-- >>> countEven = fmap (("Even " ++) . show) Fold.length
81+-- >>> countOdd = fmap (("Odd " ++) . show) Fold.length
82+-- >>> f = Fold.partition countEven countOdd
83+-- >>> Stream.fold f stream
84+-- ("Even 50","Odd 50")
85+--
86+-- == Sequential Composition
87+--
88+-- Terminating folds can be combined to parse the stream serially such that the
89+-- first fold consumes the input until it terminates and the second fold
90+-- consumes the rest of the input until it terminates:
91+--
92+-- >>> f = Fold.splitWith (,) (Fold.take 8 Fold.toList) (Fold.takeEndBy (== '\n') Fold.toList)
93+-- >>> Stream.fold f $ Stream.fromList "header: hello\n"
94+-- ("header: ","hello\n")
95+--
96+-- == Splitting a Stream
97+--
98+-- A 'Fold' can be applied repeatedly on a stream to transform it to a stream
99+-- of fold results. To split a stream on newlines:
100+--
101+-- >>> f = Fold.takeEndBy (== '\n') Fold.toList
102+-- >>> Stream.fold Fold.toList $ Stream.foldMany f $ Stream.fromList "Hello there!\nHow are you\n"
103+-- ["Hello there!\n","How are you\n"]
104+--
105+-- Similarly, we can split the input of a fold too:
106+--
107+-- >>> Stream.fold (Fold.many f Fold.toList) $ Stream.fromList "Hello there!\nHow are you\n"
108+-- ["Hello there!\n","How are you\n"]
109+--
110+-- == Folds vs. Streams
111+--
112+-- We can often use streams or folds to achieve the same goal. However, streams
113+-- are more efficient in composition of producers (e.g.
114+-- 'Data.Stream.append' or 'Data.Stream.mergeBy') whereas folds are
115+-- more efficient in composition of consumers (e.g. 'splitWith', 'partition'
116+-- or 'teeWith').
117+--
118+-- Streams are producers, transformations on streams happen on the output side:
119+--
120+-- >>> :{
121+-- f stream =
122+-- Stream.filter odd stream
123+-- & fmap (+1)
124+-- & Stream.fold Fold.sum
125+-- :}
126+--
127+-- >>> f $ Stream.fromList [1..100 :: Int]
128+-- 2550
129+--
130+-- Folds are stream consumers with an input stream and an output value, stream
131+-- transformations on folds happen on the input side:
132+--
133+-- >>> :{
134+-- f =
135+-- Fold.filter odd
136+-- $ Fold.lmap (+1)
137+-- $ Fold.sum
138+-- :}
139+--
140+-- >>> Stream.fold f $ Stream.fromList [1..100 :: Int]
141+-- 2550
142+--
143+-- Notice the similiarity in the definition of @f@ in both cases, the only
144+-- difference is the composition by @&@ vs @$@ and the use @lmap@ vs @map@, the
145+-- difference is due to output vs input side transformations.
146+--
147+-- == Fusion Limitations
148+--
149+-- Folds support stream fusion for generating loops comparable to the speed of
150+-- C. However, it has some limitations. For fusion to work, the folds must be
151+-- inlined, folds must be statically known and not generated dynamically, folds
152+-- should not be passed recursively.
153+--
154+-- Another limitation is due to the quadratic complexity causing slowdown when
155+-- too many nested compositions are used. Especially, the performance of the
156+-- Applicative instance and splitting operations (e.g. 'splitWith') degrades
157+-- quadratically (O(n^2)) when combined @n@ times, roughly 8 or less sequenced
158+-- operations are fine. For these cases folds can be converted to parsers and
159+-- then used as ParserK.
160+--
161+-- == Experimental APIs
162+--
13163-- Please refer to "Streamly.Internal.Data.Fold" for more functions that have
14164-- not yet been released.
15165
@@ -21,9 +171,6 @@ module Streamly.Data.Fold
21171 --
22172 -- $setup
23173
24− -- * Overview
25− -- $overview
26−
27174 -- * Running A Fold
28175 drive
29176 -- XXX Should we have a stream returning function in fold module?
@@ -258,121 +405,6 @@ import Streamly.Internal.Data.Fold
258405
259406#include "DocTestDataFold.hs"
260407
261−-- $overview
262−--
263−-- A 'Fold' is a consumer of a stream of values. A fold driver (such as
264−-- 'Streamly.Data.Stream.fold') initializes the fold @accumulator@, runs the
265−-- fold @step@ function in a loop, processing the input stream one element at a
266−-- time and accumulating the result. The loop continues until the fold
267−-- terminates, at which point the accumulated result is returned.
268−--
269−-- For example, a 'sum' Fold represents a stream consumer that adds the values
270−-- in the input stream:
271−--
272−-- >>> Stream.fold Fold.sum $ Stream.fromList [1..100]
273−-- 5050
274−--
275−-- Conceptually, a 'Fold' is a data type that mimics a strict left fold
276−-- ('Data.List.foldl'). The above example is similar to a left fold using
277−-- @(+)@ as the step and @0@ as the initial value of the accumulator:
278−--
279−-- >>> Data.List.foldl' (+) 0 [1..100]
280−-- 5050
281−--
282−-- 'Fold's have an early termination capability e.g. the 'one' fold terminates
283−-- after consuming one element:
284−--
285−-- >>> Stream.fold Fold.one $ Stream.fromList [1..]
286−-- Just 1
287−--
288−-- The above example is similar to the following right fold:
289−--
290−-- >>> Prelude.foldr (\x _ -> Just x) Nothing [1..]
291−-- Just 1
292−--
293−-- 'Fold's can be combined together using combinators. For example, to create a
294−-- fold that sums first two elements in a stream:
295−--
296−-- >>> sumTwo = Fold.take 2 Fold.sum
297−-- >>> Stream.fold sumTwo $ Stream.fromList [1..100]
298−-- 3
299−--
300−-- Folds can be combined to run in parallel on the same input. For example, to
301−-- compute the average of numbers in a stream without going through the stream
302−-- twice:
303−--
304−-- >>> avg = Fold.teeWith (/) Fold.sum (fmap fromIntegral Fold.length)
305−-- >>> Stream.fold avg $ Stream.fromList [1.0..100.0]
306−-- 50.5
307−--
308−-- Folds can be combined so as to partition the input stream over multiple
309−-- folds. For example, to count even and odd numbers in a stream:
310−--
311−-- >>> split n = if even n then Left n else Right n
312−-- >>> stream = fmap split $ Stream.fromList [1..100]
313−-- >>> countEven = fmap (("Even " ++) . show) Fold.length
314−-- >>> countOdd = fmap (("Odd " ++) . show) Fold.length
315−-- >>> f = Fold.partition countEven countOdd
316−-- >>> Stream.fold f stream
317−-- ("Even 50","Odd 50")
318−--
319−-- Terminating folds can be combined to parse the stream serially such that the
320−-- first fold consumes the input until it terminates and the second fold
321−-- consumes the rest of the input until it terminates:
322−--
323−-- >>> f = Fold.splitWith (,) (Fold.take 8 Fold.toList) (Fold.takeEndBy (== '\n') Fold.toList)
324−-- >>> Stream.fold f $ Stream.fromList "header: hello\n"
325−-- ("header: ","hello\n")
326−--
327−-- A 'Fold' can be applied repeatedly on a stream to transform it to a stream
328−-- of fold results. To split a stream on newlines:
329−--
330−-- >>> f = Fold.takeEndBy (== '\n') Fold.toList
331−-- >>> Stream.fold Fold.toList $ Stream.foldMany f $ Stream.fromList "Hello there!\nHow are you\n"
332−-- ["Hello there!\n","How are you\n"]
333−--
334−-- Similarly, we can split the input of a fold too:
335−--
336−-- >>> Stream.fold (Fold.many f Fold.toList) $ Stream.fromList "Hello there!\nHow are you\n"
337−-- ["Hello there!\n","How are you\n"]
338−--
339−-- = Folds vs. Streams
340−--
341−-- We can often use streams or folds to achieve the same goal. However, streams
342−-- are more efficient in composition of producers (e.g.
343−-- 'Data.Stream.append' or 'Data.Stream.mergeBy') whereas folds are
344−-- more efficient in composition of consumers (e.g. 'splitWith', 'partition'
345−-- or 'teeWith').
346−--
347−-- Streams are producers, transformations on streams happen on the output side:
348−--
349−-- >>> :{
350−-- f stream =
351−-- Stream.filter odd stream
352−-- & fmap (+1)
353−-- & Stream.fold Fold.sum
354−-- :}
355−--
356−-- >>> f $ Stream.fromList [1..100 :: Int]
357−-- 2550
358−--
359−-- Folds are stream consumers with an input stream and an output value, stream
360−-- transformations on folds happen on the input side:
361−--
362−-- >>> :{
363−-- f =
364−-- Fold.filter odd
365−-- $ Fold.lmap (+1)
366−-- $ Fold.sum
367−-- :}
368−--
369−-- >>> Stream.fold f $ Stream.fromList [1..100 :: Int]
370−-- 2550
371−--
372−-- Notice the similiarity in the definition of @f@ in both cases, the only
373−-- difference is the composition by @&@ vs @$@ and the use @lmap@ vs @map@, the
374−-- difference is due to output vs input side transformations.
375−
376408--------------------------------------------------------------------------------
377409-- Deprecated
378410--------------------------------------------------------------------------------
core/src/Streamly/Data/MutArray.hs+18−12
@@ -32,22 +32,27 @@ module Streamly.Data.MutArray
3232 -- * Construction
3333
3434 -- Uninitialized Arrays
35− , new
36− , pinnedNew
35+ , emptyOf
36+ , pinnedEmptyOf
3737
3838 -- From containers
3939 , fromListN
4040 , fromList
41− , writeN -- drop new
42− , write -- full buffer
41+ , createOf
42+ , create
4343 -- writeLastN
4444
45+ -- * Pinning & Unpinning
46+ , pin
47+ , unpin
48+ , isPinned
49+
4550 -- * Appending elements
4651 , snoc
4752
4853 -- * Appending streams
49− , writeAppendN -- XXX Rename to writeSnocN?
50− , writeAppend -- XXX Rename to writeSnoc?
54+ , appendN
55+ , append
5156
5257 -- * Inplace mutation
5358 , putIndex
@@ -78,16 +83,17 @@ module Streamly.Data.MutArray
7883 -- * Size
7984 , length
8085
81− -- * Unbox Type Class
86+ -- * Re-exports
8287 , Unbox (..)
8388
84− -- * Pinning & Unpinning
85− , pin
86− , unpin
87− , isPinned
88−
8989 -- * Deprecated
9090 , newPinned
91+ , new
92+ , pinnedNew
93+ , writeN
94+ , write
95+ , writeAppendN
96+ , writeAppend
9197 )
9298where
9399
core/src/Streamly/Data/MutArray/Generic.hs+8−3
@@ -23,11 +23,11 @@ module Streamly.Data.MutArray.Generic
2323 MutArray
2424
2525 -- * Construction
26− , new
26+ , emptyOf
2727 , fromListN
2828 , fromList
29− , writeN
30− , write
29+ , createOf
30+ , create
3131
3232 -- * Appending elements
3333 , snoc
@@ -56,6 +56,11 @@ module Streamly.Data.MutArray.Generic
5656
5757 -- * Size
5858 , length
59+
60+ -- * Deprecated
61+ , new
62+ , writeN
63+ , write
5964 )
6065where
6166
core/src/Streamly/Data/MutByteArray.hsnew file+106
@@ -0,0 +1,106 @@
1+-- |
2+-- Module : Streamly.Data.MutByteArray
3+-- Copyright : (c) 2023 Composewell Technologies
4+-- License : BSD3-3-Clause
5+-- Maintainer : streamly@composewell.com
6+-- Portability : GHC
7+--
8+-- A low level byte Array type 'MutByteArray', along with type classes 'Unbox'
9+-- and 'Serialize' for fast binary serialization and deserialization of Haskell
10+-- values. Serialization, deserialization performance is similar to, and in
11+-- some cases many times better than the store package. Conceptually, the
12+-- 'Serialize' type class works in the same way as store.
13+--
14+-- Serialize instances are configurable to use constructor names (see
15+-- 'Streamly.Internal.Data.MutByteArray.encodeConstrNames'), record field names (see
16+-- 'Streamly.Internal.Data.MutByteArray.encodeRecordFields') instead of binary
17+-- encoded values. This is an experimental feature which allows JSON like
18+-- properties with faster speed. For example, you can change the order of
19+-- constructors or record fields without affecting serialized value.
20+--
21+-- Higher level unboxed array modules "Streamly.Data.Array" and
22+-- "Streamly.Data.MutArray" are built on top of this module. Unboxed arrays are
23+-- essentially serialized Haskell values. Array modules provide higher level
24+-- serialization routines like 'Streamly.Internal.Data.Array.pinnedSerialize'
25+-- and 'Streamly.Internal.Data.Array.deserialize' from the
26+-- "Streamly.Internal.Data.Array" module.
27+--
28+-- == Mutable Byte Array
29+--
30+-- 'MutByteArray' is a primitive mutable array in the IO monad. 'Unbox' and
31+-- 'Serialize' type classes use this primitive array to serialize data to and
32+-- deserialize it from. This array is used to build higher level unboxed
33+-- array types 'Streamly.Data.MutArray.MutArray' and 'Streamly.Data.Array.Array'.
34+--
35+-- == Using Unbox
36+--
37+-- The 'Unbox' type class is simple and used to serialize non-recursive fixed
38+-- size data types. This type class is primarily used to implement unboxed
39+-- arrays. Unboxed arrays are just a sequence of serialized fixed length
40+-- Haskell data types. Instances of this type class can be derived using
41+-- 'Generic' or template haskell based deriving functions provided in this
42+-- module.
43+--
44+-- Writing a data type to an array using the array creation routines in
45+-- "Streamly.Data.Array" or "Streamly.Data.MutArray" (e.g. @writeN@ or
46+-- @fromListN@), serializes the type to the array. Similarly, reading the data
47+-- type from the array deserializes it. You can also serialize and deserialize
48+-- directly to and from a 'MutByteArray', using the type class methods.
49+--
50+-- == Using Serialize
51+--
52+-- The 'Serialize' type class is a superset of the 'Unbox' type class, it can
53+-- serialize variable length data types as well e.g. Haskell lists. Use
54+-- 'deriveSerialize' to derive the instances of the type class automatically
55+-- and then use the type class methods to serialize and deserialize to and from
56+-- a 'MutByteArray'.
57+--
58+-- See 'Streamly.Internal.Data.Array.pinnedSerialize' and
59+-- 'Streamly.Internal.Data.Array.deserialize' for 'Array' type based
60+-- serialization.
61+--
62+-- == Comparing serialized values
63+--
64+-- When using the `Unbox` type class the same value may result in differing
65+-- serialized bytes because of unused uninitialized data in case of sum types.
66+-- Therefore, byte comparison of serialized values is not reliable.
67+--
68+-- However, the 'Serialize' type class guarantees that the serialized values
69+-- are always exactly the same and byte comparison of serialized is reliable.
70+--
71+module Streamly.Data.MutByteArray
72+ (
73+
74+ -- * Mutable Byte Array
75+ -- | The standard way to read from or write to a 'MutByteArray' is by using
76+ -- the 'Unbox' or 'Serialize' type class methods.
77+ MutByteArray
78+ , isPinned
79+ , pin
80+ , unpin
81+ , new
82+ , pinnedNew
83+
84+ -- * Unbox
85+ , Unbox(..)
86+ , deriveUnbox
87+
88+ -- * Serialize
89+ , Serialize(..)
90+
91+ -- ** Instance Config
92+ , SerializeConfig
93+ , inlineAddSizeTo
94+ , inlineSerializeAt
95+ , inlineDeserializeAt
96+
97+ -- ** Instance Deriving
98+ , deriveSerialize
99+ , deriveSerializeWith
100+ ) where
101+
102+--------------------------------------------------------------------------------
103+-- Imports
104+--------------------------------------------------------------------------------
105+
106+import Streamly.Internal.Data.MutByteArray
core/src/Streamly/Data/Parser.hs+130−60
@@ -7,12 +7,137 @@
77-- Stability : pre-release
88-- Portability : GHC
99--
10−-- Fast, composable stream consumers with ability to terminate, backtrack and
11−-- fail, supporting stream fusion. Parsers are a natural extension of
12−-- "Streamly.Data.Fold". Parsers and folds can be interconverted.
10+-- Parsers are stream consumers like folds with the following differences:
1311--
14−-- Please refer to "Streamly.Internal.Data.Parser" for functions that have
15−-- not yet been released.
12+-- * folds cannot fail but parsers can fail and backtrack.
13+-- * folds can be composed as a Tee but parsers cannot.
14+-- * folds can be used for scanning but parsers cannot.
15+-- * folds can be converted to parsers.
16+--
17+-- This module implements parsers with stream fusion which compile to efficient
18+-- loops comparable to the speed of C.
19+--
20+-- == Using Parsers
21+--
22+-- This module provides elementary parsers and parser combinators that can be
23+-- used to parse a stream of data. Additionally, all the folds from the
24+-- "Streamly.Data.Fold" module can be converted to parsers using 'fromFold'.
25+-- All the parsing functionality provided by popular parsing libraries, and
26+-- more is available. Also see "Streamly.Unicode.Parser" module for Char stream
27+-- parsers.
28+--
29+-- A data stream can be transformed to a stream of parsed data elements. Parser
30+-- combinators can be used to create a pipeline of folds or parsers such that
31+-- the next fold or parser consumes the result of the previous parser. See
32+-- 'Streamly.Data.Stream.parse' and 'Streamly.Data.Stream.parseMany' to run
33+-- these parsers on a stream.
34+--
35+-- == Parser vs ParserK
36+--
37+-- There are two functionally equivalent parsing modules,
38+-- "Streamly.Data.Parser" (this module) and "Streamly.Data.ParserK". The latter
39+-- is a CPS based wrapper over the former, and can be used for parsing in
40+-- general. "Streamly.Data.Parser" enables stream fusion and should be
41+-- preferred over "Streamly.Data.ParserK" for high performance stream parsing
42+-- use cases. However, there are a few cases where this module is not
43+-- suitable and ParserK should be used instead.
44+--
45+-- For static fusion, parser combinators have to use strict pattern matching on
46+-- arguments of type Parser. This leads to infinte loop when a parser is
47+-- defined recursively, due to strict evaluation of the recursive call. For
48+-- example, the following implementation loops infinitely because of the
49+-- recursive use of parser @p@ in the @*>@ combinator:
50+--
51+-- >>> import Streamly.Data.Parser (Parser)
52+-- >>> import qualified Streamly.Data.Fold as Fold
53+-- >>> import qualified Streamly.Data.Parser as Parser
54+-- >>> import qualified Streamly.Data.Stream as Stream
55+-- >>> import Control.Applicative ((<|>))
56+--
57+-- >>> :{
58+-- >>> p :: Monad m => Parser Char m String
59+-- >>> p = Parser.satisfy (== '(') *> p <|> Parser.fromFold Fold.toList
60+-- >>> :}
61+--
62+-- Use ParserK when recursive use is required:
63+--
64+-- >>> import Streamly.Data.ParserK (ParserK)
65+-- >>> import qualified Streamly.Data.StreamK as StreamK
66+-- >>> import qualified Streamly.Internal.Data.StreamK as StreamK (parse)
67+-- >>> import qualified Streamly.Internal.Data.ParserK as ParserK (adapt)
68+--
69+-- >>> :{
70+-- >>> p :: Monad m => ParserK Char m String
71+-- >>> p = ParserK.adapt (Parser.satisfy (== '(')) *> p <|> ParserK.adapt (Parser.fromFold Fold.toList)
72+-- >>> :}
73+--
74+-- >>> StreamK.parse p $ StreamK.fromStream $ Stream.fromList "hello"
75+-- Right "hello"
76+--
77+-- For this reason Applicative, Alternative or Monad compositions with
78+-- recursion cannot be used with the 'Parser' type. Alternative type class based
79+-- operations like 'asum' and Alternative based generic parser combinators use
80+-- recursion. Similarly, Applicative type class based operations like
81+-- 'Prelude.sequence' use recursion. Custom implementations of many such
82+-- operations are provided in this module (e.g. 'some', 'many'), and those
83+-- should be used instead.
84+--
85+-- Another limitation of Parser type is due to the quadratic complexity causing
86+-- slowdown when too many nested compositions are used. Especially Applicative,
87+-- Monad, Alternative instances, and sequenced parsing operations (e.g. nested
88+-- 'one', and 'splitWith') degrade the performance quadratically (O(n^2)) when
89+-- combined @n@ times, roughly 8 or less sequenced parsers are fine. READ THE
90+-- DOCS OF APPLICATIVE, MONAD AND ALTERNATIVE INSTANCES.
91+--
92+-- == Streaming Parsers
93+--
94+-- With 'Streamly.Data.ParserK.ParserK' you can use the generic Alternative
95+-- type class based parsers from the
96+-- <https://hackage.haskell.org/package/parser-combinators parser-combinators>
97+-- library or similar. However, we recommend that you use the equivalent
98+-- functionality from this module for better performance and for streaming
99+-- behavior.
100+--
101+-- Firstly, the combinators in this module are faster due to stream fusion.
102+-- Secondly, these are streaming in nature as the results can be passed
103+-- directly to other stream consumers (folds or parsers). The Alternative type
104+-- class based parsers would end up buffering all the results in lists before
105+-- they can be consumed.
106+--
107+-- When recursion or heavy nesting is needed use ParserK.
108+--
109+-- == Error Reporting
110+--
111+-- These parsers do not report the error context (e.g. line number or column).
112+-- This may be supported in future.
113+--
114+-- == Monad Transformer Stack
115+--
116+-- 'MonadTrans' instance is not provided. If the 'Parser' type is the top most
117+-- layer (which should be the case almost always) you can just use 'fromEffect'
118+-- to execute the lower layer monad effects.
119+--
120+-- == Parser vs ParserK Implementation
121+--
122+-- The 'Parser' type represents a stream consumer by composing state as data
123+-- which enables stream fusion. Stream fusion generates a tight loop without
124+-- any constructor allocations between the stages, providing C like performance
125+-- for the loop. Stream fusion works when multiple functions are combined in a
126+-- pipeline statically. Therefore, the operations in this module must be
127+-- inlined and must not be used recursively to allow for stream fusion.
128+--
129+-- The 'ParserK' type represents a stream consumer by composing function calls,
130+-- therefore, a function call overhead is incurred at each composition. It is
131+-- quite fast in general but may be a few times slower than a fused parser.
132+-- However, it allows for scalable dynamic composition especially parsers can
133+-- be used in recursive calls. Using the 'ParserK' type operations like
134+-- 'splitWith' provide linear (O(n)) performance with respect to the number of
135+-- compositions.
136+--
137+-- == Experimental APIs
138+--
139+-- Please refer to "Streamly.Internal.Data.Parser" for functions that have not
140+-- yet been released.
16141--
17142module Streamly.Data.Parser
18143 (
@@ -22,9 +147,6 @@ module Streamly.Data.Parser
22147 --
23148 -- $setup
24149
25− -- * Overview
26− -- $overview
27−
28150 -- * Parser Type
29151 Parser
30152
@@ -123,55 +245,3 @@ import Streamly.Internal.Data.Parser
123245import Prelude hiding (dropWhile, takeWhile, filter)
124246
125247#include "DocTestDataParser.hs"
126−
127−-- $overview
128−--
129−-- Several combinators in this module can be many times faster than CPS based
130−-- parsers because of stream fusion. For example,
131−-- 'Streamly.Internal.Data.Parser.many' combinator in this module is much
132−-- faster than the 'Control.Applicative.many' combinator of
133−-- 'Control.Applicative.Alternative' type class used by CPS based parsers.
134−--
135−-- The use of 'Alternative' type class, in parsers has another drawback.
136−-- Alternative based parsers use plain Haskell lists to collect the results. In
137−-- a strict Monad like IO, the results are necessarily buffered before they can
138−-- be consumed. This may not perform optimally in streaming applications
139−-- processing large amounts of data. Equivalent combinators in this module can
140−-- consume the results of parsing using a 'Fold' or another parser, thus
141−-- providing a scalable and composable consumer.
142−--
143−-- Note that these parsers do not report the error context (e.g. line number or
144−-- column). This may be supported in future.
145−--
146−-- mtl instances are not provided. If the 'Parser' type is the top most layer
147−-- (which should be the case almost always) you can just use 'fromEffect' to
148−-- execute the lower layer monad effects.
149−--
150−-- == Performance Notes
151−--
152−-- The 'Parser' type represents a stream consumer by composing state as data
153−-- which enables stream fusion. Stream fusion generates a tight loop without
154−-- any constructor allocations between the stages, providing C like performance
155−-- for the loop. Stream fusion works when multiple functions are combined in a
156−-- pipeline statically. Therefore, the operations in this module must be
157−-- inlined and must not be used recursively to allow for stream fusion. Note
158−-- that operations like 'sequence', and 'asum' that compose pasrers using
159−-- recursion should be avoided with these parsers. You can use these with the
160−-- 'ParserK' module instead.
161−--
162−-- Using the 'Parser' type, parsing operations like 'one', 'splitWith' etc.
163−-- degrade quadratically (O(n^2)) when combined many times. If you need to
164−-- combine these operations, say more than 8 times in a single loop, then you
165−-- should consider using the continuation style parser type 'ParserK' instead.
166−-- Also, if you need to use these operations in a recursive loop you should use
167−-- 'ParserK' instead.
168−--
169−-- The 'ParserK' type represents a stream consumer by composing function calls,
170−-- therefore, a function call overhead is incurred at each composition. It is
171−-- quite fast in general but may be a few times slower than a fused parser.
172−-- However, it allows for scalable dynamic composition especially parsers can
173−-- be used in recursive calls. Using the 'ParserK' type operations like
174−-- 'splitWith' provide linear (O(n)) performance with respect to the number of
175−-- compositions..
176−--
177−-- 'Parser' and 'ParserK' types can be interconverted.
core/src/Streamly/Data/ParserK.hs+89−3
@@ -1,3 +1,4 @@
1+{-# LANGUAGE CPP #-}
12-- |
23-- Module : Streamly.Data.ParserK
34-- Copyright : (c) 2023 Composewell Technologies
@@ -6,19 +7,102 @@
67-- Stability : pre-release
78-- Portability : GHC
89--
9−-- Parsers using Continuation Passing Style (CPS). See notes in
10−-- "Streamly.Data.Parser" module to know when to use this module.
10+-- See the general notes about parsing in the "Streamly.Data.Parser" module.
11+-- This module implements a using Continuation Passing Style (CPS) wrapper over
12+-- the "Streamly.Data.Parser" module. It is as fast or faster than attoparsec.
1113--
12−-- To run a 'ParserK' use 'Streamly.Data.StreamK.parseChunks'.
14+-- == Parser vs ParserK
15+--
16+-- 'ParserK' is preferred over 'Streamly.Data.Parser.Parser' when extensive
17+-- applicative, alternative and monadic composition is required, or when
18+-- recursive or dynamic composition of parsers is required. The
19+-- 'Streamly.Data.Parser.Parser' type fuses statically and creates efficient
20+-- loops whereas 'ParserK' uses function call based composition and has
21+-- comparatively larger runtime overhead but it is better suited to the
22+-- specific use cases mentioned above. 'ParserK' also allows to efficient parse
23+-- a stream of arrays, it can also break the input stream into a parse result
24+-- and remaining stream so that the stream can be parsed independently in
25+-- segments.
26+--
27+-- == Using ParserK
28+--
29+-- All the parsers from the "Streamly.Data.Parser" module can be adapted to
30+-- ParserK using the 'Streamly.Data.ParserK.adaptC',
31+-- 'Streamly.Internal.Data.ParserK.adapt', and
32+-- 'Streamly.Internal.Data.ParserK.adaptCG' combinators.
33+--
34+-- 'Streamly.Data.StreamK.parseChunks' runs a parser on a stream of unboxed
35+-- arrays, this is the preferred and most efficient way to parse chunked input.
36+-- The more general 'Streamly.Data.StreamK.parseBreakChunks' function returns
37+-- the remaining stream as well along with the parse result. There are
38+-- 'Streamly.Internal.Data.StreamK.parseChunksGeneric',
39+-- 'Streamly.Internal.Data.StreamK.parseBreakChunksGeneric' as well to run
40+-- parsers on boxed arrays. 'Streamly.Internal.Data.StreamK.parse',
41+-- 'Streamly.Internal.Data.StreamK.parseBreak' run parsers on a stream of
42+-- individual elements instead of stream of arrays.
43+--
44+-- == Monadic Composition
45+--
46+-- Monad composition can be used for lookbehind parsers, we can dynamically
47+-- compose new parsers based on the results of the previously parsed values.
48+--
49+-- If we have to parse "a9" or "9a" but not "99" or "aa" we can use the
50+-- following non-monadic, backtracking parser:
51+--
52+-- >>> digits p1 p2 = ((:) <$> p1 <*> ((:) <$> p2 <*> pure []))
53+-- >>> :{
54+-- backtracking :: Monad m => ParserK Char m String
55+-- backtracking = ParserK.adapt $
56+-- digits (Parser.satisfy isDigit) (Parser.satisfy isAlpha)
57+-- <|>
58+-- digits (Parser.satisfy isAlpha) (Parser.satisfy isDigit)
59+-- :}
60+--
61+-- We know that if the first parse resulted in a digit at the first place then
62+-- the second parse is going to fail. However, we waste that information and
63+-- parse the first character again in the second parse only to know that it is
64+-- not an alphabetic char. By using lookbehind in a 'Monad' composition we can
65+-- avoid redundant work:
66+--
67+-- >>> data DigitOrAlpha = Digit Char | Alpha Char
68+--
69+-- >>> :{
70+-- lookbehind :: Monad m => ParserK Char m String
71+-- lookbehind = do
72+-- x1 <- ParserK.adapt $
73+-- Digit <$> Parser.satisfy isDigit
74+-- <|> Alpha <$> Parser.satisfy isAlpha
75+-- -- Note: the parse depends on what we parsed already
76+-- x2 <- ParserK.adapt $
77+-- case x1 of
78+-- Digit _ -> Parser.satisfy isAlpha
79+-- Alpha _ -> Parser.satisfy isDigit
80+-- return $ case x1 of
81+-- Digit x -> [x,x2]
82+-- Alpha x -> [x,x2]
83+-- :}
84+--
85+-- == Experimental APIs
86+--
87+-- Please refer to "Streamly.Internal.Data.ParserK" for functions that have
88+-- not yet been released.
1389--
1490module Streamly.Data.ParserK
1591 (
92+ -- * Setup
93+ -- | To execute the code examples provided in this module in ghci, please
94+ -- run the following commands first.
95+ --
96+ -- $setup
97+
1698 -- * Parser Type
1799 ParserK
18100
19101 -- * Parsers
20102 -- ** Conversions
103+ , adapt
21104 , adaptC
105+ , adaptCG
22106 -- , toParser
23107
24108 -- ** Without Input
@@ -41,6 +125,8 @@ import qualified Streamly.Internal.Data.Parser as ParserD
41125
42126import Streamly.Internal.Data.ParserK.Type
43127
128+#include "DocTestDataParserK.hs"
129+
44130{-# DEPRECATED fromFold "Please use \"ParserK.adaptC . Parser.fromFold\" instead." #-}
45131{-# INLINE fromFold #-}
46132fromFold :: (MonadIO m, Unbox a) => Fold m a b -> ParserK (Array a) m b
core/src/Streamly/Data/Serialize.hsdeleted−52
@@ -1,52 +0,0 @@
1−-- |
2−-- Module : Streamly.Data.Serialize
3−-- Copyright : (c) 2023 Composewell Technologies
4−-- License : BSD3-3-Clause
5−-- Maintainer : streamly@composewell.com
6−-- Portability : GHC
7−--
8−module Streamly.Data.Serialize
9− (
10−
11− -- * Types
12− MutableByteArray
13−
14− -- * Unbox
15− , Unbox(..)
16− , deriveUnbox
17− , deriveUnboxWith
18−
19− -- * Serialize
20− , Size(..)
21− , Serialize(..)
22− , deriveSerialize
23− , deriveSerializeWith
24−
25− -- * Encoding
26− , encode
27− , pinnedEncode
28−
29− -- * Decoding
30− , decode
31− ) where
32−
33−--------------------------------------------------------------------------------
34−-- Imports
35−--------------------------------------------------------------------------------
36−
37−import Streamly.Internal.Data.Serialize
38− ( Serialize(..)
39− , Size(..)
40− , decode
41− , encode
42− , pinnedEncode
43− )
44−import Streamly.Internal.Data.Serialize.TH
45− ( deriveSerialize
46− , deriveSerializeWith
47− )
48−import Streamly.Internal.Data.Unbox
49− ( MutableByteArray
50− , Unbox(..)
51− )
52−import Streamly.Internal.Data.Unbox.TH (deriveUnbox, deriveUnboxWith)
core/src/Streamly/Data/Stream.hs+131−36
@@ -70,6 +70,15 @@ module Streamly.Data.Stream
7070
7171 -- ** From Values
7272 -- | Generate a monadic stream from a seed value or values.
73+ --
74+ -- Non-empty pure streams can be cycled using repeat and unfold:
75+ --
76+ -- >>> cycle = Stream.unfoldMany Unfold.fromList . Stream.repeat
77+ --
78+ -- Non-empty StreamK can be cycled using append:
79+ --
80+ -- >>> cycle xs = let ys = xs `StreamK.append` ys in ys
81+ --
7382 , fromPure
7483 , fromEffect
7584 , repeat
@@ -212,21 +221,57 @@ module Streamly.Data.Stream
212221 -- operations like mapping a function over the stream.
213222 , foldrM
214223 , foldr
224+ -- foldr1
215225
216226 -- ** Specific Folds
217− -- | Usually you can use the folds in "Streamly.Data.Fold". However, some
218− -- folds that may be commonly used or may have an edge in performance in
219− -- some cases are provided here.
220− --
221− -- Useful idioms:
227+ -- | Streams are folded using folds in "Streamly.Data.Fold". Here are some
228+ -- idioms and equivalents of Data.List APIs using folds:
222229 --
223230 -- >>> foldlM' f a = Stream.fold (Fold.foldlM' f a)
224− -- >>> foldl1 f = Stream.fold (Fold.foldl1' f)
231+ -- >>> foldl1' f = Stream.fold (Fold.foldl1' f)
225232 -- >>> foldl' f a = Stream.fold (Fold.foldl' f a)
226233 -- >>> drain = Stream.fold Fold.drain
227234 -- >>> mapM_ f = Stream.fold (Fold.drainMapM f)
228235 -- >>> length = Stream.fold Fold.length
236+ -- >>> genericLength = Stream.fold Fold.lengthGeneric
229237 -- >>> head = Stream.fold Fold.one
238+ -- >>> last = Stream.fold Fold.latest
239+ -- >>> null = Stream.fold Fold.null
240+ -- >>> and = Stream.fold Fold.and
241+ -- >>> or = Stream.fold Fold.or
242+ -- >>> any p = Stream.fold (Fold.any p)
243+ -- >>> all p = Stream.fold (Fold.all p)
244+ -- >>> sum = Stream.fold Fold.sum
245+ -- >>> product = Stream.fold Fold.product
246+ -- >>> maximum = Stream.fold Fold.maximum
247+ -- >>> maximumBy cmp = Stream.fold (Fold.maximumBy cmp)
248+ -- >>> minimum = Stream.fold Fold.minimum
249+ -- >>> minimumBy cmp = Stream.fold (Fold.minimumBy cmp)
250+ -- >>> elem x = Stream.fold (Fold.elem x)
251+ -- >>> notElem x = Stream.fold (Fold.notElem x)
252+ -- >>> lookup x = Stream.fold (Fold.lookup x)
253+ -- >>> find p = Stream.fold (Fold.find p)
254+ -- >>> (!?) i = Stream.fold (Fold.index i)
255+ -- >>> genericIndex i = Stream.fold (Fold.indexGeneric i)
256+ -- >>> elemIndex x = Stream.fold (Fold.elemIndex x)
257+ -- >>> findIndex p = Stream.fold (Fold.findIndex p)
258+ --
259+ -- Some equivalents of Data.List APIs from the Stream module:
260+ --
261+ -- >>> head = fmap (fmap fst) . Stream.uncons
262+ -- >>> tail = fmap (fmap snd) . Stream.uncons
263+ -- >>> tail = Stream.tail -- unreleased API
264+ -- >>> init = Stream.init -- unreleased API
265+ --
266+ -- A Stream based toList fold implementation is provided below because it
267+ -- has a better performance compared to the fold.
268+
269+ -- Functions in Data.List, missing here:
270+ -- unsnoc = Stream.parseBreak (Parser.init Fold.toList)
271+ -- genericTake
272+ -- genericDrop
273+ -- genericSplitAt
274+ -- genericReplicate
230275 , toList
231276
232277 -- * Mapping
@@ -280,6 +325,15 @@ module Streamly.Data.Stream
280325 , scan
281326 , postscan
282327 -- XXX postscan1 can be implemented using Monoids or Refolds.
328+ -- The following scans from Data.List are not provided.
329+ -- XXX scanl
330+ -- XXX scanl1
331+ -- XXX scanr
332+ -- XXX scanr1
333+ -- XXX mapAccumL
334+ -- XXX mapAccumR
335+ -- XXX inits
336+ -- XXX tails
283337
284338 -- ** Specific scans
285339 -- Indexing can be considered as a special type of zipping where we zip a
@@ -288,6 +342,8 @@ module Streamly.Data.Stream
288342
289343 -- * Insertion
290344 -- | Add elements to the stream.
345+ --
346+ -- >>> insert = Stream.insertBy compare
291347
292348 -- Inserting elements is a special case of interleaving/merging streams.
293349 , insertBy
@@ -323,12 +379,15 @@ module Streamly.Data.Stream
323379 -- "Streamly.Internal.Data.Fold" can be used along with 'scanMaybe' to
324380 -- perform stateful filtering operations in general.
325381 --
326− -- Useful idioms:
382+ -- Idioms and equivalents of Data.List APIs:
327383 --
328384 -- >>> deleteBy cmp x = Stream.scanMaybe (Fold.deleteBy cmp x)
385+ -- >>> deleteBy = Stream.deleteBy -- unreleased API
386+ -- >>> delete = deleteBy (==)
329387 -- >>> findIndices p = Stream.scanMaybe (Fold.findIndices p)
330388 -- >>> elemIndices a = findIndices (== a)
331389 -- >>> uniq = Stream.scanMaybe (Fold.uniqBy (==))
390+ -- >>> partition p = Stream.fold (Fold.partition Fold.toList Fold.toList) . fmap (if p then Left else Right)
332391 , scanMaybe
333392 , take
334393 , takeWhile
@@ -336,6 +395,8 @@ module Streamly.Data.Stream
336395 , drop
337396 , dropWhile
338397 , dropWhileM
398+ -- XXX write to an array in reverse and then read in reverse
399+ -- > dropWhileEnd = reverse . dropWhile p . reverse
339400
340401 -- XXX These are available as scans in folds. We need to check the
341402 -- performance though. If these are common and we need convenient stream
@@ -372,6 +433,9 @@ module Streamly.Data.Stream
372433 -- >>> s = s1 `Stream.append` s2
373434
374435 -- ** Appending
436+ -- | Equivalent of Data.List append:
437+ --
438+ -- >>> (++) = Stream.append
375439 , append
376440
377441 -- ** Interleaving
@@ -382,8 +446,14 @@ module Streamly.Data.Stream
382446 , mergeByM
383447
384448 -- ** Zipping
449+ -- | Idioms and equivalents of Data.List APIs:
450+ --
451+ -- >>> zip = Stream.zipWith (,)
452+ -- >>> unzip = Stream.fold (Fold.unzip Fold.toList Fold.toList)
385453 , zipWith
386454 , zipWithM
455+ -- XXX zipWith3,4,5,6,7
456+ -- XXX unzip3,4,5,6,7
387457 -- , ZipStream (..)
388458
389459 -- ** Cross Product
@@ -416,48 +486,89 @@ module Streamly.Data.Stream
416486 -- One dimension loops are just a special case of nested loops. For
417487 -- example, 'concatMap' can degenerate to a simple map operation:
418488 --
419− -- > map f m = S.concatMap (\x -> S.fromPure (f x)) m
489+ -- >>> map f m = Stream.concatMap (\x -> Stream.fromPure (f x)) m
420490 --
421491 -- Similarly, 'concatMap' can perform filtering by mapping an element to a
422492 -- 'nil' stream:
423493 --
424− -- > filter p m = S.concatMap (\x -> if p x then S.fromPure x else S.nil) m
494+ -- >>> filter p m = Stream.concatMap (\x -> if p x then Stream.fromPure x else Stream.nil) m
425495 --
496+ -- Idioms and equivalents of Data.List APIs:
497+ --
498+ -- >>> concat = Stream.concatMap id
499+ -- >>> unlines = Stream.interposeSuffix '\n'
500+ -- >>> unlines = Stream.intercalateSuffix Unfold.fromList "\n"
501+ -- >>> unwords = Stream.interpose ' '
502+ -- >>> unwords = Stream.intercalate Unfold.fromList " "
426503
427504 , concatEffect
428505 , concatMap
429506 , concatMapM
430507
431508 -- * Repeated Fold
432− -- | Useful idioms:
509+ -- | Idioms and equivalents of Data.List APIs:
433510 --
434− -- >>> splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
435− -- >>> splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
436− -- >>> groupsBy eq f = Stream.parseMany (Parser.groupBy eq f)
437− -- >>> groupsByRolling eq f = Stream.parseMany (Parser.groupByRolling eq f)
438− -- >>> groupsOf n f = Stream.foldMany (Fold.take n f)
511+ -- >>> groupsOf n = Stream.foldMany (Fold.take n Fold.toList)
512+ -- >>> groupBy eq = Stream.groupsWhile eq Fold.toList
513+ -- >>> groupBy eq = Stream.parseMany (Parser.groupBy eq Fold.toList)
514+ -- >>> groupsByRolling eq = Stream.parseMany (Parser.groupByRolling eq Fold.toList)
515+ -- >>> groups = groupBy (==)
439516 , foldMany -- XXX Rename to foldRepeat
440517 , groupsOf
441518 , parseMany
442519
443520 -- * Splitting
521+ -- | Idioms and equivalents of Data.List APIs:
522+ --
523+ -- >>> splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
524+ -- >>> splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
525+ -- >>> lines = splitOnSuffix (== '\n')
526+ -- >>> words = Stream.wordsBy isSpace
527+ -- >>> splitAt n = Stream.fold (Fold.splitAt n Fold.toList Fold.toList)
528+ -- >>> span p = Parser.splitWith (,) (Parser.takeWhile p Fold.toList) (Parser.fromFold Fold.toList)
529+ -- >>> break p = span (not . p)
444530 , splitOn
445531 , wordsBy
446532
447533 -- * Buffered Operations
448534 -- | Operations that require buffering of the stream.
449535 -- Reverse is essentially a left fold followed by an unfold.
536+ --
537+ -- Idioms and equivalents of Data.List APIs:
538+ --
539+ -- >>> nub = Stream.fold Fold.toList . Stream.scanMaybe Fold.nub
540+ -- >>> nub = Stream.ordNub -- unreleased API
541+ -- >>> sortBy = StreamK.sortBy
542+ -- >>> sortOn f = StreamK.sortOn -- unreleased API
543+ -- >>> deleteFirstsBy = Stream.deleteFirstsBy -- unreleased
544+ -- >>> (\\) = Stream.deleteFirstsBy (==) -- unreleased
545+ -- >>> intersectBy = Stream.intersectBy -- unreleased
546+ -- >>> intersect = Stream.intersectBy (==) -- unreleased
547+ -- >>> unionBy = Stream.unionBy -- unreleased
548+ -- >>> union = Stream.unionBy (==) -- unreleased
549+ --
450550 , reverse
551+ -- XXX transpose: write the streams to arrays and then stream transposed.
552+ -- XXX subsequences
553+ -- XXX permutations
554+ -- , nub
555+ -- , ordNub
556+ -- , nubBy
451557
452558 -- * Multi-Stream folds
453559 -- | Operations that consume multiple streams at the same time.
454560 , eqBy
455561 , cmpBy
456562 , isPrefixOf
563+ , isInfixOf
564+ -- , isSuffixOf
565+ -- , isSuffixOfUnbox
457566 , isSubsequenceOf
458567
459568 -- trimming sequences
460− , stripPrefix
569+ -- , stripPrefix
570+ -- , stripSuffix
571+ -- , stripSuffixUnbox
461572
462573 -- Exceptions and resource management depend on the "exceptions" package
463574 -- XXX We can have IO Stream operations not depending on "exceptions"
@@ -511,25 +622,20 @@ module Streamly.Data.Stream
511622 , runReaderT
512623 , runStateT
513624
625+ -- XXX Arrays could be different types, therefore, this should be in
626+ -- specific array module. Or maybe we should abstract over array types.
514627 -- * Stream of Arrays
515628 , Array.chunksOf
516− , chunksSplitOn
517629 )
518630where
519631
520−import Control.Monad.IO.Class (MonadIO)
521−import Data.Word (Word8)
522−import Streamly.Data.Array (Array)
523632import Streamly.Internal.Data.Stream
524633import Prelude
525634 hiding (filter, drop, dropWhile, take, takeWhile, zipWith, foldr,
526− foldl, map, mapM, mapM_, sequence, all, any, sum, product, elem,
527− notElem, maximum, minimum, head, last, tail, length, null,
528− reverse, iterate, init, and, or, lookup, foldr1, (!!),
529− scanl, scanl1, repeat, replicate, concatMap, span)
635+ mapM, sequence, reverse, iterate, foldr1, repeat, replicate,
636+ concatMap)
530637
531638import qualified Streamly.Internal.Data.Array.Type as Array
532−import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
533639
534640#include "DocTestDataStream.hs"
535641
@@ -638,14 +744,3 @@ import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
638744--
639745-- 'Stream' and 'StreamK' types can be interconverted. See
640746-- "Streamly.Data.StreamK" module for conversion operations.
641−
642−-- | Split a stream of arrays on a given separator byte, dropping the separator
643−-- and coalescing all the arrays between two separators into a single array.
644−--
645−{-# INLINE chunksSplitOn #-}
646−chunksSplitOn
647− :: (MonadIO m)
648− => Word8
649− -> Stream m (Array Word8)
650− -> Stream m (Array Word8)
651−chunksSplitOn = ArrayStream.splitOn
core/src/Streamly/Data/StreamK.hs+2−1
@@ -95,7 +95,8 @@ module Streamly.Data.StreamK
9595 -- , foldBreak
9696
9797 -- ** Parsing
98− -- , parseBreak
98+ , parse
99+ , parseBreak
99100 , parseBreakChunks
100101 , parseChunks
101102
core/src/Streamly/Data/Unfold.hs+2
@@ -65,6 +65,8 @@ module Streamly.Data.Unfold
6565 -- ** Mapping on Input
6666 , lmap
6767 , lmapM
68+ , first
69+ , second
6870
6971 -- ** Mapping on Output
7072 , mapM
core/src/Streamly/FileSystem/Handle.hs+7−30
@@ -1,5 +1,4 @@
11{-# LANGUAGE CPP #-}
2−#include "inline.hs"
32
43-- |
54-- Module : Streamly.FileSystem.Handle
@@ -82,13 +81,13 @@ module Streamly.FileSystem.Handle
8281 -- position of the file handle. The stream ends as soon as EOF is
8382 -- encountered.
8483
85− -- -- *** Streams
86− -- , read
87− -- , readWith
88− -- , readChunks
89− -- , readChunksWith
84+ -- *** Streams
85+ , read
86+ , readWith
87+ , readChunks
88+ , readChunksWith
9089
91− -- -- *** Unfolds
90+ -- *** Unfolds
9291 , reader
9392 , readerWith
9493 , chunkReader
@@ -105,36 +104,14 @@ module Streamly.FileSystem.Handle
105104 , writeChunks
106105
107106 -- * Deprecated
108− , read
109107 , readWithBufferOf
110− , readChunks
111108 , readChunksWithBufferOf
112109 , writeChunksWithBufferOf
113110 , writeWithBufferOf
114111 )
115112where
116113
117−import Control.Monad.IO.Class (MonadIO(..))
118−import Data.Word (Word8)
119−import Streamly.Internal.Data.Array.Type (Array)
120−import Streamly.Internal.Data.Unfold.Type (Unfold)
121−import System.IO (Handle)
122−
123−import Streamly.Internal.FileSystem.Handle hiding (read, readChunks)
114+import Streamly.Internal.FileSystem.Handle
124115import Prelude hiding (read)
125116
126117#include "DocTestFileSystemHandle.hs"
127−
128−-- | Same as 'reader'
129−--
130−{-# DEPRECATED read "Please use 'reader' instead" #-}
131−{-# INLINE read #-}
132−read :: MonadIO m => Unfold m Handle Word8
133−read = reader
134−
135−-- | Same as 'chunkReader'
136−--
137−{-# DEPRECATED readChunks "Please use 'chunkReader' instead" #-}
138−{-# INLINE readChunks #-}
139−readChunks :: MonadIO m => Unfold m Handle (Array Word8)
140−readChunks = chunkReader
core/src/Streamly/Internal/Data/Array.hs+608−135
@@ -2,7 +2,6 @@
22-- |
33-- Module : Streamly.Internal.Data.Array
44-- Copyright : (c) 2019 Composewell Technologies
5−--
65-- License : BSD3
76-- Maintainer : streamly@composewell.com
87-- Stability : experimental
@@ -20,33 +19,21 @@ module Streamly.Internal.Data.Array
2019 module Streamly.Internal.Data.Array.Type
2120
2221 -- * Construction
23− -- Stream Folds
24− , fromStreamN
25− , fromStream
26−
2722 -- Monadic Folds
2823 , writeLastN
2924
30− -- * Elimination
31− -- ** Conversion
32−
33− -- ** Unfolds
34− , reader
35− , readerUnsafe
36− , producer -- experimental
37−
3825 -- * Random Access
3926 -- , (!!)
4027 , getIndex
4128 , getIndexRev
42− , last -- XXX getIndexLast?
43− , getIndices
44− , getIndicesFromThenTo
29+ , last -- XXX getLastIndex?
4530 -- , getIndicesFrom -- read from a given position to the end of file
4631 -- , getIndicesUpto -- read from beginning up to the given position
4732 -- , getIndicesFromTo
4833 -- , getIndicesFromRev -- read from a given position to the beginning of file
4934 -- , getIndicesUptoRev -- read from end to the given position in file
35+ , indexReader
36+ , indexReaderFromThenTo
5037
5138 -- * Size
5239 , null
@@ -54,6 +41,8 @@ module Streamly.Internal.Data.Array
5441 -- * Search
5542 , binarySearch
5643 , findIndicesOf
44+ -- getIndicesOf
45+ , indexFinder -- see splitOn
5746 -- , findIndexOf
5847 -- , find
5948
@@ -66,51 +55,89 @@ module Streamly.Internal.Data.Array
6655 -- * Subarrays
6756 , getSliceUnsafe
6857 -- , getSlice
69− , genSlicesFromLen
70− , getSlicesFromLen
58+ , sliceIndexerFromLen
59+ , slicerFromLen
7160 , splitOn
7261
7362 -- * Streaming Operations
7463 , streamTransform
7564
76− -- ** Folding
65+ -- * Folding
7766 , streamFold
7867 , fold
68+
69+ -- * Stream of Arrays
70+
71+ -- XXX these are probably not very useful to have in this module as we can
72+ -- express these idiomatically using streams.
73+ , interpose
74+ , interposeSuffix
75+ , intercalateSuffix
76+
77+ , compactLE
78+ , pinnedCompactLE
79+ , compactOnByte
80+ , compactOnByteSuffix
81+
82+ , foldBreakChunks
83+ , foldChunks
84+ , foldBreakChunksK
85+ , parseBreakChunksK
86+
87+ -- * Serialization
88+ , encodeAs
89+ , serialize
90+ , pinnedSerialize
91+ , deserialize
92+
93+ -- * Deprecated
94+ , genSlicesFromLen
95+ , getSlicesFromLen
96+ , getIndices
7997 )
8098where
8199
100+#include "assert.hs"
82101#include "inline.hs"
83102#include "ArrayMacros.h"
84103
85−import Control.Exception (assert)
86−import Control.Monad (when)
87104import Control.Monad.IO.Class (MonadIO(..))
105+-- import Data.Bifunctor (first)
106+-- import Data.Either (fromRight)
88107import Data.Functor.Identity (Identity)
89108import Data.Proxy (Proxy(..))
90109import Data.Word (Word8)
91110import Foreign.C.String (CString)
92111import Foreign.Ptr (castPtr)
93112import Foreign.Storable (Storable)
113+import GHC.Types (SPEC(..))
94114import Streamly.Internal.Data.Unbox (Unbox(..))
95115import Prelude hiding (length, null, last, map, (!!), read, concat)
96116
117+import Streamly.Internal.Data.MutByteArray.Type (PinnedState(..), MutByteArray)
118+import Streamly.Internal.Data.Serialize.Type (Serialize)
97119import Streamly.Internal.Data.Fold.Type (Fold(..))
98−import Streamly.Internal.Data.Producer.Type (Producer(..))
99−import Streamly.Internal.Data.Stream (Stream)
100−import Streamly.Internal.Data.Tuple.Strict (Tuple3Fused'(..))
120+import Streamly.Internal.Data.Parser (Parser(..), Initial(..), ParseError(..))
121+import Streamly.Internal.Data.Stream (Stream(..))
122+import Streamly.Internal.Data.StreamK (StreamK)
123+import Streamly.Internal.Data.SVar.Type (adaptState, defState)
124+import Streamly.Internal.Data.Tuple.Strict (Tuple'(..), Tuple3Fused'(..))
101125import Streamly.Internal.Data.Unfold.Type (Unfold(..))
102126import Streamly.Internal.System.IO (unsafeInlineIO)
103127
104−import qualified Streamly.Internal.Data.MutArray.Type as MA
128+import qualified Streamly.Internal.Data.Fold.Type as Fold
129+import qualified Streamly.Internal.Data.Serialize.Type as Serialize
130+import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
105131import qualified Streamly.Internal.Data.MutArray as MA
106−import qualified Streamly.Internal.Data.Array.Type as A
107132import qualified Streamly.Internal.Data.Fold as FL
108−import qualified Streamly.Internal.Data.Producer.Type as Producer
109−import qualified Streamly.Internal.Data.Producer as Producer
110133import qualified Streamly.Internal.Data.Ring as RB
134+import qualified Streamly.Internal.Data.Parser as Parser
135+-- import qualified Streamly.Internal.Data.ParserK as ParserK
111136import qualified Streamly.Internal.Data.Stream as D
112137import qualified Streamly.Internal.Data.Stream as Stream
138+import qualified Streamly.Internal.Data.StreamK as StreamK
113139import qualified Streamly.Internal.Data.Unfold as Unfold
140+import qualified Prelude
114141
115142import Streamly.Internal.Data.Array.Type
116143
@@ -139,82 +166,10 @@ import Streamly.Internal.Data.Array.Type
139166-- if we use one array for each chunk. This is still significant to add
140167-- pressure to GC.
141168
142−-------------------------------------------------------------------------------
143−-- Construction
144−-------------------------------------------------------------------------------
145−
146−-- | Create an 'Array' from the first N elements of a stream. The array is
147−-- allocated to size N, if the stream terminates before N elements then the
148−-- array may hold less than N elements.
149−--
150−-- /Pre-release/
151−{-# INLINE fromStreamN #-}
152−fromStreamN :: (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
153−fromStreamN n m = do
154− when (n < 0) $ error "writeN: negative write count specified"
155− A.fromStreamDN n m
156−
157−-- | Create an 'Array' from a stream. This is useful when we want to create a
158−-- single array from a stream of unknown size. 'writeN' is at least twice
159−-- as efficient when the size is already known.
160−--
161−-- Note that if the input stream is too large memory allocation for the array
162−-- may fail. When the stream size is not known, `chunksOf` followed by
163−-- processing of indvidual arrays in the resulting stream should be preferred.
164−--
165−-- /Pre-release/
166−{-# INLINE fromStream #-}
167−fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (Array a)
168−fromStream = Stream.fold A.write
169−-- write m = A.fromStreamD $ D.fromStreamK m
170−
171169-------------------------------------------------------------------------------
172170-- Elimination
173171-------------------------------------------------------------------------------
174172
175−{-# INLINE_NORMAL producer #-}
176−producer :: forall m a. (Monad m, Unbox a) => Producer m (Array a) a
177−producer =
178− Producer.translate A.unsafeThaw A.unsafeFreeze
179− $ MA.producerWith (return . unsafeInlineIO)
180−
181−-- | Unfold an array into a stream.
182−--
183−{-# INLINE_NORMAL reader #-}
184−reader :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
185−reader = Producer.simplify producer
186−
187−-- | Unfold an array into a stream, does not check the end of the array, the
188−-- user is responsible for terminating the stream within the array bounds. For
189−-- high performance application where the end condition can be determined by
190−-- a terminating fold.
191−--
192−-- Written in the hope that it may be faster than "read", however, in the case
193−-- for which this was written, "read" proves to be faster even though the core
194−-- generated with unsafeRead looks simpler.
195−--
196−-- /Pre-release/
197−--
198−{-# INLINE_NORMAL readerUnsafe #-}
199−readerUnsafe :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
200−readerUnsafe = Unfold step inject
201− where
202−
203− inject (Array contents start end) =
204− return (ArrayUnsafe contents end start)
205−
206− {-# INLINE_LATE step #-}
207− step (ArrayUnsafe contents end p) = do
208− -- unsafeInlineIO allows us to run this in Identity monad for pure
209− -- toList/foldr case which makes them much faster due to not
210− -- accumulating the list and fusing better with the pure consumers.
211− --
212− -- This should be safe as the array contents are guaranteed to be
213− -- evaluated/written to before we peek at them.
214− let !x = unsafeInlineIO $ peekByteIndex p contents
215− let !p1 = INDEX_NEXT(p,a)
216− return $ D.Yield x (ArrayUnsafe contents end p1)
217−
218173-- |
219174--
220175-- >>> null arr = Array.byteLength arr == 0
@@ -222,7 +177,7 @@ readerUnsafe = Unfold step inject
222177-- /Pre-release/
223178{-# INLINE null #-}
224179null :: Array a -> Bool
225−null arr = A.byteLength arr == 0
180+null arr = byteLength arr == 0
226181
227182-- | Like 'getIndex' but indexes the array in reverse from the end.
228183--
@@ -234,7 +189,7 @@ getIndexRev i arr =
234189 $ do
235190 let elemPtr = RINDEX_OF(arrEnd arr, i, a)
236191 if i >= 0 && elemPtr >= arrStart arr
237− then Just <$> peekByteIndex elemPtr (arrContents arr)
192+ then Just <$> peekAt elemPtr (arrContents arr)
238193 else return Nothing
239194
240195-- |
@@ -250,6 +205,8 @@ last = getIndexRev 0
250205-- Folds with Array as the container
251206-------------------------------------------------------------------------------
252207
208+-- XXX We should generate this from Ring.
209+
253210-- | @writeLastN n@ folds a maximum of @n@ elements from the end of the input
254211-- stream to an 'Array'.
255212--
@@ -258,7 +215,7 @@ writeLastN ::
258215 (Storable a, Unbox a, MonadIO m) => Int -> Fold m a (Array a)
259216writeLastN n
260217 | n <= 0 = fmap (const mempty) FL.drain
261− | otherwise = A.unsafeFreeze <$> Fold step initial done
218+ | otherwise = unsafeFreeze <$> Fold step initial done done
262219
263220 where
264221
@@ -271,11 +228,9 @@ writeLastN n
271228 in fmap f $ liftIO $ RB.new n
272229
273230 done (Tuple3Fused' rb rh i) = do
274− arr <- liftIO $ MA.new n
275− foldFunc i rh snoc' arr rb
276−
277− -- XXX We should write a read unfold for ring.
278− snoc' b a = liftIO $ MA.snocUnsafe b a
231+ arr <- MA.new n
232+ -- XXX We should write a read unfold for ring.
233+ foldFunc i rh MA.snocUnsafe arr rb
279234
280235 foldFunc i
281236 | i < n = RB.unsafeFoldRingM
@@ -301,19 +256,26 @@ binarySearch = undefined
301256-- compared to streams by using SIMD instructions.
302257-- We can also return a bit array instead.
303258
259+-- Can use SIMD.
260+
304261-- | Perform a linear search to find all the indices where a given element is
305262-- present in an array.
306263--
307264-- /Unimplemented/
308−findIndicesOf :: (a -> Bool) -> Unfold Identity (Array a) Int
309−findIndicesOf = undefined
265+indexFinder :: (a -> Bool) -> Unfold Identity (Array a) Int
266+indexFinder = undefined
267+
268+-- |
269+-- /Unimplemented/
270+findIndicesOf :: (a -> Bool) -> Array a -> Stream Identity Int
271+findIndicesOf p = Stream.unfold (indexFinder p)
310272
311273{-
312274findIndexOf :: (a -> Bool) -> Array a -> Maybe Int
313−findIndexOf p = Unfold.fold Fold.one . Stream.unfold (findIndicesOf p)
275+findIndexOf p = Unfold.fold Fold.one . Stream.unfold (indexFinder p)
314276
315277find :: (a -> Bool) -> Array a -> Bool
316−find = Unfold.fold Fold.null . Stream.unfold (findIndicesOf p)
278+find = Unfold.fold Fold.null . Stream.unfold (indexFinder p)
317279-}
318280
319281-------------------------------------------------------------------------------
@@ -355,29 +317,45 @@ splitOn :: (Monad m, Unbox a) =>
355317 (a -> Bool) -> Array a -> Stream m (Array a)
356318splitOn predicate arr =
357319 fmap (\(i, len) -> getSliceUnsafe i len arr)
358− $ D.sliceOnSuffix predicate (A.toStreamD arr)
320+ $ D.indexOnSuffix predicate (read arr)
321+
322+{-# INLINE sliceIndexerFromLen #-}
323+sliceIndexerFromLen :: forall m a. (Monad m, Unbox a)
324+ => Int -- ^ from index
325+ -> Int -- ^ length of the slice
326+ -> Unfold m (Array a) (Int, Int)
327+sliceIndexerFromLen from len =
328+ Unfold.lmap unsafeThaw (MA.sliceIndexerFromLen from len)
359329
330+{-# DEPRECATED genSlicesFromLen "Please use sliceIndexerFromLen instead." #-}
360331{-# INLINE genSlicesFromLen #-}
361332genSlicesFromLen :: forall m a. (Monad m, Unbox a)
362333 => Int -- ^ from index
363334 -> Int -- ^ length of the slice
364335 -> Unfold m (Array a) (Int, Int)
365−genSlicesFromLen from len =
366− Unfold.lmap A.unsafeThaw (MA.genSlicesFromLen from len)
336+genSlicesFromLen = sliceIndexerFromLen
367337
368338-- | Generate a stream of slices of specified length from an array, starting
369339-- from the supplied array index. The last slice may be shorter than the
370340-- requested length.
371341--
372342-- /Pre-release//
343+{-# INLINE slicerFromLen #-}
344+slicerFromLen :: forall m a. (Monad m, Unbox a)
345+ => Int -- ^ from index
346+ -> Int -- ^ length of the slice
347+ -> Unfold m (Array a) (Array a)
348+slicerFromLen from len =
349+ fmap unsafeFreeze
350+ $ Unfold.lmap unsafeThaw (MA.slicerFromLen from len)
351+
352+{-# DEPRECATED getSlicesFromLen "Please use slicerFromLen instead." #-}
373353{-# INLINE getSlicesFromLen #-}
374354getSlicesFromLen :: forall m a. (Monad m, Unbox a)
375355 => Int -- ^ from index
376356 -> Int -- ^ length of the slice
377357 -> Unfold m (Array a) (Array a)
378−getSlicesFromLen from len =
379− fmap A.unsafeFreeze
380− $ Unfold.lmap A.unsafeThaw (MA.getSlicesFromLen from len)
358+getSlicesFromLen = slicerFromLen
381359
382360-------------------------------------------------------------------------------
383361-- Random reads and writes
@@ -395,7 +373,7 @@ getIndex i arr =
395373 $ do
396374 let elemPtr = INDEX_OF(arrStart arr, i, a)
397375 if i >= 0 && INDEX_VALID(elemPtr, arrEnd arr, a)
398− then Just <$> peekByteIndex elemPtr (arrContents arr)
376+ then Just <$> peekAt elemPtr (arrContents arr)
399377 else return Nothing
400378
401379-- | Given a stream of array indices, read the elements on those indices from
@@ -407,19 +385,25 @@ getIndex i arr =
407385-- @
408386-- read =
409387-- let u = lmap (\arr -> (0, length arr - 1)) Unfold.enumerateFromTo
410−-- in Unfold.lmap f (getIndices arr)
388+-- in Unfold.lmap f (indexReader arr)
411389--
412390-- readRev =
413391-- let i = length arr - 1
414−-- in Unfold.lmap f (getIndicesFromThenTo i (i - 1) 0)
392+-- in Unfold.lmap f (indexReaderFromThenTo i (i - 1) 0)
415393-- @
416394--
417395-- /Pre-release/
396+{-# INLINE indexReader #-}
397+indexReader :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
398+indexReader m =
399+ let unf = MA.indexReaderWith (return . unsafeInlineIO) m
400+ in Unfold.lmap unsafeThaw unf
401+
402+-- XXX DO NOT REMOVE, change the signature to use Stream instead of unfold
403+{-# DEPRECATED getIndices "Please use getIndices instead." #-}
418404{-# INLINE getIndices #-}
419405getIndices :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
420−getIndices m =
421− let unf = MA.getIndicesD (return . unsafeInlineIO) m
422− in Unfold.lmap A.unsafeThaw unf
406+getIndices = indexReader
423407
424408-- | Unfolds @(from, then, to, array)@ generating a finite stream whose first
425409-- element is the array value from the index @from@ and the successive elements
@@ -435,9 +419,9 @@ getIndices m =
435419-- @
436420--
437421-- /Unimplemented/
438−{-# INLINE getIndicesFromThenTo #-}
439−getIndicesFromThenTo :: Unfold m (Int, Int, Int, Array a) a
440−getIndicesFromThenTo = undefined
422+{-# INLINE indexReaderFromThenTo #-}
423+indexReaderFromThenTo :: Unfold m (Int, Int, Int, Array a) a
424+indexReaderFromThenTo = undefined
441425
442426-------------------------------------------------------------------------------
443427-- Transform via stream operations
@@ -454,7 +438,7 @@ getIndicesFromThenTo = undefined
454438{-# INLINE runPipe #-}
455439runPipe :: (MonadIO m, Unbox a, Unbox b)
456440 => Pipe m a b -> Array a -> m (Array b)
457−runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (A.read arr)
441+runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (read arr)
458442-}
459443
460444-- XXX For transformations that cannot change the number of elements e.g. "map"
@@ -468,7 +452,7 @@ runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (A.read arr)
468452streamTransform :: forall m a b. (MonadIO m, Unbox a, Unbox b)
469453 => (Stream m a -> Stream m b) -> Array a -> m (Array b)
470454streamTransform f arr =
471− Stream.fold (A.writeWith (length arr)) $ f (A.read arr)
455+ Stream.fold (createWith (length arr)) $ f (read arr)
472456
473457-------------------------------------------------------------------------------
474458-- Casts
@@ -502,7 +486,7 @@ asBytes = castUnsafe
502486--
503487cast :: forall a b. (Unbox b) => Array a -> Maybe (Array b)
504488cast arr =
505− let len = A.byteLength arr
489+ let len = byteLength arr
506490 r = len `mod` SIZE_OF(b)
507491 in if r /= 0
508492 then Nothing
@@ -520,9 +504,9 @@ cast arr =
520504--
521505asCStringUnsafe :: Array a -> (CString -> IO b) -> IO b
522506asCStringUnsafe arr act = do
523− let arr1 = asBytes arr <> A.fromList [0]
524− -- asPtrUnsafe makes sure the array is pinned
525− asPtrUnsafe arr1 $ \ptr -> act (castPtr ptr)
507+ let arr1 = asBytes arr <> fromList [0]
508+ -- unsafePinnedAsPtr makes sure the array is pinned
509+ unsafePinnedAsPtr arr1 $ \ptr -> act (castPtr ptr)
526510
527511-------------------------------------------------------------------------------
528512-- Folds
@@ -535,11 +519,500 @@ asCStringUnsafe arr act = do
535519-- /Pre-release/
536520{-# INLINE fold #-}
537521fold :: forall m a b. (Monad m, Unbox a) => Fold m a b -> Array a -> m b
538−fold f arr = Stream.fold f (A.read arr)
522+fold f arr = Stream.fold f (read arr)
539523
540524-- | Fold an array using a stream fold operation.
541525--
542526-- /Pre-release/
543527{-# INLINE streamFold #-}
544528streamFold :: (Monad m, Unbox a) => (Stream m a -> m b) -> Array a -> m b
545−streamFold f arr = f (A.read arr)
529+streamFold f arr = f (read arr)
530+
531+--------------------------------------------------------------------------------
532+-- Serialization
533+--------------------------------------------------------------------------------
534+
535+{-# INLINE encodeAs #-}
536+encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
537+encodeAs ps a =
538+ unsafeInlineIO $ do
539+ let len = Serialize.addSizeTo 0 a
540+ mbarr <- MBA.newBytesAs ps len
541+ off <- Serialize.serializeAt 0 mbarr a
542+ assertM(len == off)
543+ pure $ Array mbarr 0 off
544+
545+-- |
546+-- Properties:
547+-- 1. Identity: @deserialize . serialize == id@
548+-- 2. Encoded equivalence: @serialize a == serialize a@
549+{-# INLINE serialize #-}
550+serialize :: Serialize a => a -> Array Word8
551+serialize = encodeAs Unpinned
552+
553+-- | Serialize a Haskell type to a pinned byte array. The array is allocated
554+-- using pinned memory so that it can be used directly in OS APIs for writing
555+-- to file or sending over the network.
556+--
557+-- Properties:
558+-- 1. Identity: @deserialize . pinnedSerialize == id@
559+-- 2. Encoded equivalence: @pinnedSerialize a == pinnedSerialize a@
560+{-# INLINE pinnedSerialize #-}
561+pinnedSerialize :: Serialize a => a -> Array Word8
562+pinnedSerialize = encodeAs Pinned
563+
564+-- | Decode a Haskell type from a byte array containing its serialized
565+-- representation.
566+{-# INLINE deserialize #-}
567+deserialize :: Serialize a => Array Word8 -> a
568+deserialize arr@(Array {..}) = unsafeInlineIO $ do
569+ let lenArr = length arr
570+ (off, val) <-
571+ Serialize.deserializeAt arrStart arrContents (arrStart + lenArr)
572+ assertM(off == arrStart + lenArr)
573+ pure val
574+
575+-------------------------------------------------------------------------------
576+-- Streams of Arrays
577+-------------------------------------------------------------------------------
578+
579+-- TODO: efficiently compare two streams of arrays. Two streams can have chunks
580+-- of different sizes, we can handle that in the stream comparison abstraction.
581+-- This could be useful e.g. to fast compare whether two files differ.
582+
583+-- | Insert the given element between arrays and flatten.
584+--
585+-- >>> interpose x = Stream.interpose x Array.reader
586+--
587+{-# INLINE interpose #-}
588+interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
589+interpose x = D.interpose x reader
590+
591+data FlattenState s =
592+ OuterLoop s
593+ | InnerLoop s !MutByteArray !Int !Int
594+
595+-- | Insert the given element after each array and flatten. This is similar to
596+-- unlines.
597+--
598+-- >>> interposeSuffix x = Stream.interposeSuffix x Array.reader
599+--
600+{-# INLINE_NORMAL interposeSuffix #-}
601+interposeSuffix :: forall m a. (Monad m, Unbox a)
602+ => a -> Stream m (Array a) -> Stream m a
603+-- This does not require MonadIO constraint.
604+-- interposeSuffix x = D.interposeSuffix x reader
605+interposeSuffix sep (D.Stream step state) = D.Stream step' (OuterLoop state)
606+
607+ where
608+
609+ {-# INLINE_LATE step' #-}
610+ step' gst (OuterLoop st) = do
611+ r <- step (adaptState gst) st
612+ return $ case r of
613+ D.Yield Array{..} s ->
614+ D.Skip (InnerLoop s arrContents arrStart arrEnd)
615+ D.Skip s -> D.Skip (OuterLoop s)
616+ D.Stop -> D.Stop
617+
618+ step' _ (InnerLoop st _ p end) | p == end =
619+ return $ D.Yield sep $ OuterLoop st
620+
621+ step' _ (InnerLoop st contents p end) = do
622+ x <- pure $ unsafeInlineIO $ peekAt p contents
623+ return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
624+
625+-- | Insert the given array after each array and flatten.
626+--
627+-- >>> intercalateSuffix = Stream.intercalateSuffix Array.reader
628+--
629+{-# INLINE intercalateSuffix #-}
630+intercalateSuffix :: (Monad m, Unbox a)
631+ => Array a -> Stream m (Array a) -> Stream m a
632+intercalateSuffix = D.intercalateSuffix reader
633+
634+-- | @compactLE n@ coalesces adjacent arrays in the input stream
635+-- only if the combined size would be less than or equal to n.
636+--
637+-- Generates unpinned arrays irrespective of the pinning status of input
638+-- arrays.
639+{-# INLINE_NORMAL compactLE #-}
640+compactLE :: (MonadIO m, Unbox a)
641+ => Int -> Stream m (Array a) -> Stream m (Array a)
642+compactLE n stream =
643+ D.map unsafeFreeze $ MA.compactLE n $ D.map unsafeThaw stream
644+
645+-- | Pinned version of 'compactLE'.
646+{-# INLINE_NORMAL pinnedCompactLE #-}
647+pinnedCompactLE :: (MonadIO m, Unbox a)
648+ => Int -> Stream m (Array a) -> Stream m (Array a)
649+pinnedCompactLE n stream =
650+ D.map unsafeFreeze $ MA.pinnedCompactLE n $ D.map unsafeThaw stream
651+
652+-- | Split a stream of arrays on a given separator byte, dropping the separator
653+-- and coalescing all the arrays between two separators into a single array.
654+--
655+{-# INLINE compactOnByte #-}
656+compactOnByte
657+ :: (MonadIO m)
658+ => Word8
659+ -> Stream m (Array Word8)
660+ -> Stream m (Array Word8)
661+compactOnByte byte =
662+ fmap unsafeFreeze . MA.compactOnByte byte . fmap unsafeThaw
663+
664+-- | Like 'compactOnByte' considers the separator in suffix position instead of
665+-- infix position.
666+{-# INLINE compactOnByteSuffix #-}
667+compactOnByteSuffix
668+ :: (MonadIO m)
669+ => Word8
670+ -> Stream m (Array Word8)
671+ -> Stream m (Array Word8)
672+compactOnByteSuffix byte =
673+ fmap unsafeFreeze . MA.compactOnByteSuffix byte . fmap unsafeThaw
674+
675+-------------------------------------------------------------------------------
676+-- Folding Streams of Arrays
677+-------------------------------------------------------------------------------
678+
679+-- XXX This should not be used for breaking a stream as the D.cons used in
680+-- reconstructing the stream could be very bad for performance. This can only
681+-- be useful in folding without breaking.
682+{-# INLINE_NORMAL foldBreakChunks #-}
683+foldBreakChunks :: forall m a b. (MonadIO m, Unbox a) =>
684+ Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
685+foldBreakChunks (Fold fstep initial _ final) stream@(Stream step state) = do
686+ res <- initial
687+ case res of
688+ Fold.Partial fs -> go SPEC state fs
689+ Fold.Done fb -> return $! (fb, stream)
690+
691+ where
692+
693+ {-# INLINE go #-}
694+ go !_ st !fs = do
695+ r <- step defState st
696+ case r of
697+ Stream.Yield (Array contents start end) s ->
698+ let fp = Tuple' end contents
699+ in goArray SPEC s fp start fs
700+ Stream.Skip s -> go SPEC s fs
701+ Stream.Stop -> do
702+ b <- final fs
703+ return (b, D.nil)
704+
705+ goArray !_ s (Tuple' end _) !cur !fs
706+ | cur == end = do
707+ go SPEC s fs
708+ goArray !_ st fp@(Tuple' end contents) !cur !fs = do
709+ x <- liftIO $ peekAt cur contents
710+ res <- fstep fs x
711+ let next = INDEX_NEXT(cur,a)
712+ case res of
713+ Fold.Done b -> do
714+ let arr = Array contents next end
715+ return $! (b, D.cons arr (D.Stream step st))
716+ Fold.Partial fs1 -> goArray SPEC st fp next fs1
717+
718+-- This may be more robust wrt fusion compared to unfoldMany?
719+
720+-- | Fold a stream of arrays using a 'Fold'. This is equivalent to the
721+-- following:
722+--
723+-- >>> foldChunks f = Stream.fold f . Stream.unfoldMany Array.reader
724+--
725+foldChunks :: (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m b
726+foldChunks f s = fmap fst (foldBreakChunks f s)
727+-- foldStream f = Stream.fold f . Stream.unfoldMany reader
728+
729+-- | Fold a stream of arrays using a 'Fold' and return the remaining stream.
730+--
731+-- The following alternative to this function allows composing the fold using
732+-- the parser Monad:
733+--
734+-- @
735+-- foldBreakStreamK f s =
736+-- fmap (first (fromRight undefined))
737+-- $ StreamK.parseBreakChunks (ParserK.adaptC (Parser.fromFold f)) s
738+-- @
739+--
740+-- We can compare perf and remove this one or define it in terms of that.
741+--
742+foldBreakChunksK :: forall m a b. (MonadIO m, Unbox a) =>
743+ Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
744+{-
745+foldBreakChunksK f s =
746+ fmap (first (fromRight undefined))
747+ $ StreamK.parseBreakChunks (ParserK.adaptC (Parser.fromFold f)) s
748+-}
749+foldBreakChunksK (Fold fstep initial _ final) stream = do
750+ res <- initial
751+ case res of
752+ Fold.Partial fs -> go fs stream
753+ Fold.Done fb -> return (fb, stream)
754+
755+ where
756+
757+ {-# INLINE go #-}
758+ go !fs st = do
759+ let stop = (, StreamK.nil) <$> final fs
760+ single a = yieldk a StreamK.nil
761+ yieldk (Array contents start end) r =
762+ let fp = Tuple' end contents
763+ in goArray fs r fp start
764+ in StreamK.foldStream defState yieldk single stop st
765+
766+ goArray !fs st (Tuple' end _) !cur
767+ | cur == end = do
768+ go fs st
769+ goArray !fs st fp@(Tuple' end contents) !cur = do
770+ x <- liftIO $ peekAt cur contents
771+ res <- fstep fs x
772+ let next = INDEX_NEXT(cur,a)
773+ case res of
774+ Fold.Done b -> do
775+ let arr = Array contents next end
776+ return $! (b, StreamK.cons arr st)
777+ Fold.Partial fs1 -> goArray fs1 st fp next
778+
779+{-
780+-- This can be generalized to any type provided it can be unfolded to a stream
781+-- and it can be combined using a semigroup operation.
782+--
783+{-# INLINE_NORMAL parseBreakD #-}
784+parseBreakD ::
785+ forall m a b. (MonadIO m, MonadThrow m, Unbox a)
786+ => PRD.Parser a m b
787+ -> D.Stream m (Array.Array a)
788+ -> m (b, D.Stream m (Array.Array a))
789+parseBreakD
790+ (PRD.Parser pstep initial extract) stream@(D.Stream step state) = do
791+
792+ res <- initial
793+ case res of
794+ PRD.IPartial s -> go SPEC state (List []) s
795+ PRD.IDone b -> return (b, stream)
796+ PRD.IError err -> throwM $ ParseError err
797+
798+ where
799+
800+ -- "backBuf" contains last few items in the stream that we may have to
801+ -- backtrack to.
802+ --
803+ -- XXX currently we are using a dumb list based approach for backtracking
804+ -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
805+ -- That will allow us more efficient random back and forth movement.
806+ go !_ st backBuf !pst = do
807+ r <- step defState st
808+ case r of
809+ D.Yield (Array contents start end) s ->
810+ gobuf SPEC s backBuf
811+ (Tuple' end contents) start pst
812+ D.Skip s -> go SPEC s backBuf pst
813+ D.Stop -> do
814+ b <- extract pst
815+ return (b, D.nil)
816+
817+ -- Use strictness on "cur" to keep it unboxed
818+ gobuf !_ s backBuf (Tuple' end _) !cur !pst
819+ | cur == end = do
820+ go SPEC s backBuf pst
821+ gobuf !_ s backBuf fp@(Tuple' end contents) !cur !pst = do
822+ x <- liftIO $ peekByteIndex contents cur
823+ pRes <- pstep pst x
824+ let next = INDEX_NEXT(cur,a)
825+ case pRes of
826+ PR.Partial 0 pst1 ->
827+ gobuf SPEC s (List []) fp next pst1
828+ PR.Partial n pst1 -> do
829+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
830+ let src0 = Prelude.take n (x:getList backBuf)
831+ arr0 = A.fromListN n (Prelude.reverse src0)
832+ arr1 = Array contents next end
833+ src = arr0 <> arr1
834+ let !(Array cont1 start end1) = src
835+ fp1 = Tuple' end1 cont1
836+ gobuf SPEC s (List []) fp1 start pst1
837+ PR.Continue 0 pst1 ->
838+ gobuf SPEC s (List (x:getList backBuf)) fp next pst1
839+ PR.Continue n pst1 -> do
840+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
841+ let (src0, buf1) = splitAt n (x:getList backBuf)
842+ arr0 = A.fromListN n (Prelude.reverse src0)
843+ arr1 = Array contents next end
844+ src = arr0 <> arr1
845+ let !(Array cont1 start end1) = src
846+ fp1 = Tuple' end1 cont1
847+ gobuf SPEC s (List buf1) fp1 start pst1
848+ PR.Done 0 b -> do
849+ let arr = Array contents next end
850+ return (b, D.cons arr (D.Stream step s))
851+ PR.Done n b -> do
852+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
853+ let src0 = Prelude.take n (x:getList backBuf)
854+ -- XXX create the array in reverse instead
855+ arr0 = A.fromListN n (Prelude.reverse src0)
856+ arr1 = Array contents next end
857+ -- XXX Use StreamK to avoid adding arbitrary layers of
858+ -- constructors every time.
859+ str = D.cons arr0 (D.cons arr1 (D.Stream step s))
860+ return (b, str)
861+ PR.Error err -> throwM $ ParseError err
862+-}
863+
864+-- | Parse an array stream using the supplied 'Parser'. Returns the parse
865+-- result and the unconsumed stream. Throws 'ParseError' if the parse fails.
866+--
867+-- The following alternative to this function allows composing the parser using
868+-- the parser Monad:
869+--
870+-- >>> parseBreakStreamK p = StreamK.parseBreakChunks (ParserK.adaptC p)
871+--
872+-- We can compare perf and remove this one or define it in terms of that.
873+--
874+-- /Internal/
875+--
876+{-# INLINE_NORMAL parseBreakChunksK #-}
877+parseBreakChunksK ::
878+ forall m a b. (MonadIO m, Unbox a)
879+ => Parser a m b
880+ -> StreamK m (Array a)
881+ -> m (Either ParseError b, StreamK m (Array a))
882+-- parseBreakStreamK p = StreamK.parseBreakChunks (ParserK.adaptC p)
883+parseBreakChunksK (Parser pstep initial extract) stream = do
884+ res <- initial
885+ case res of
886+ IPartial s -> go s stream []
887+ IDone b -> return (Right b, stream)
888+ IError err -> return (Left (ParseError err), stream)
889+
890+ where
891+
892+ -- "backBuf" contains last few items in the stream that we may have to
893+ -- backtrack to.
894+ --
895+ -- XXX currently we are using a dumb list based approach for backtracking
896+ -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
897+ -- That will allow us more efficient random back and forth movement.
898+ go !pst st backBuf = do
899+ let stop = goStop pst backBuf -- (, K.nil) <$> extract pst
900+ single a = yieldk a StreamK.nil
901+ yieldk arr r = goArray pst backBuf r arr
902+ in StreamK.foldStream defState yieldk single stop st
903+
904+ -- Use strictness on "cur" to keep it unboxed
905+ goArray !pst backBuf st (Array _ cur end) | cur == end = go pst st backBuf
906+ goArray !pst backBuf st (Array contents cur end) = do
907+ x <- liftIO $ peekAt cur contents
908+ pRes <- pstep pst x
909+ let next = INDEX_NEXT(cur,a)
910+ case pRes of
911+ Parser.Partial 0 s ->
912+ goArray s [] st (Array contents next end)
913+ Parser.Partial n s -> do
914+ assert (n <= Prelude.length (x:backBuf)) (return ())
915+ let src0 = Prelude.take n (x:backBuf)
916+ arr0 = fromListN n (Prelude.reverse src0)
917+ arr1 = Array contents next end
918+ src = arr0 <> arr1
919+ goArray s [] st src
920+ Parser.Continue 0 s ->
921+ goArray s (x:backBuf) st (Array contents next end)
922+ Parser.Continue n s -> do
923+ assert (n <= Prelude.length (x:backBuf)) (return ())
924+ let (src0, buf1) = Prelude.splitAt n (x:backBuf)
925+ arr0 = fromListN n (Prelude.reverse src0)
926+ arr1 = Array contents next end
927+ src = arr0 <> arr1
928+ goArray s buf1 st src
929+ Parser.Done 0 b -> do
930+ let arr = Array contents next end
931+ return (Right b, StreamK.cons arr st)
932+ Parser.Done n b -> do
933+ assert (n <= Prelude.length (x:backBuf)) (return ())
934+ let src0 = Prelude.take n (x:backBuf)
935+ -- XXX Use fromListRevN once implemented
936+ -- arr0 = A.fromListRevN n src0
937+ arr0 = fromListN n (Prelude.reverse src0)
938+ arr1 = Array contents next end
939+ str = StreamK.cons arr0 (StreamK.cons arr1 st)
940+ return (Right b, str)
941+ Parser.Error err -> do
942+ let n = Prelude.length backBuf
943+ arr0 = fromListN n (Prelude.reverse backBuf)
944+ arr1 = Array contents cur end
945+ str = StreamK.cons arr0 (StreamK.cons arr1 stream)
946+ return (Left (ParseError err), str)
947+
948+ -- This is a simplified goArray
949+ goExtract !pst backBuf (Array _ cur end)
950+ | cur == end = goStop pst backBuf
951+ goExtract !pst backBuf (Array contents cur end) = do
952+ x <- liftIO $ peekAt cur contents
953+ pRes <- pstep pst x
954+ let next = INDEX_NEXT(cur,a)
955+ case pRes of
956+ Parser.Partial 0 s ->
957+ goExtract s [] (Array contents next end)
958+ Parser.Partial n s -> do
959+ assert (n <= Prelude.length (x:backBuf)) (return ())
960+ let src0 = Prelude.take n (x:backBuf)
961+ arr0 = fromListN n (Prelude.reverse src0)
962+ arr1 = Array contents next end
963+ src = arr0 <> arr1
964+ goExtract s [] src
965+ Parser.Continue 0 s ->
966+ goExtract s backBuf (Array contents next end)
967+ Parser.Continue n s -> do
968+ assert (n <= Prelude.length (x:backBuf)) (return ())
969+ let (src0, buf1) = Prelude.splitAt n (x:backBuf)
970+ arr0 = fromListN n (Prelude.reverse src0)
971+ arr1 = Array contents next end
972+ src = arr0 <> arr1
973+ goExtract s buf1 src
974+ Parser.Done 0 b -> do
975+ let arr = Array contents next end
976+ return (Right b, StreamK.fromPure arr)
977+ Parser.Done n b -> do
978+ assert (n <= Prelude.length backBuf) (return ())
979+ let src0 = Prelude.take n backBuf
980+ -- XXX Use fromListRevN once implemented
981+ -- arr0 = A.fromListRevN n src0
982+ arr0 = fromListN n (Prelude.reverse src0)
983+ arr1 = Array contents next end
984+ str = StreamK.cons arr0 (StreamK.fromPure arr1)
985+ return (Right b, str)
986+ Parser.Error err -> do
987+ let n = Prelude.length backBuf
988+ arr0 = fromListN n (Prelude.reverse backBuf)
989+ arr1 = Array contents cur end
990+ str = StreamK.cons arr0 (StreamK.cons arr1 stream)
991+ return (Left (ParseError err), str)
992+
993+ -- This is a simplified goExtract
994+ {-# INLINE goStop #-}
995+ goStop !pst backBuf = do
996+ pRes <- extract pst
997+ case pRes of
998+ Parser.Partial _ _ -> error "Bug: parseBreak: Partial in extract"
999+ Parser.Continue 0 s ->
1000+ goStop s backBuf
1001+ Parser.Continue n s -> do
1002+ assert (n <= Prelude.length backBuf) (return ())
1003+ let (src0, buf1) = Prelude.splitAt n backBuf
1004+ arr = fromListN n (Prelude.reverse src0)
1005+ goExtract s buf1 arr
1006+ Parser.Done 0 b ->
1007+ return (Right b, StreamK.nil)
1008+ Parser.Done n b -> do
1009+ assert (n <= Prelude.length backBuf) (return ())
1010+ let src0 = Prelude.take n backBuf
1011+ -- XXX Use fromListRevN once implemented
1012+ -- arr0 = A.fromListRevN n src0
1013+ arr0 = fromListN n (Prelude.reverse src0)
1014+ return (Right b, StreamK.fromPure arr0)
1015+ Parser.Error err -> do
1016+ let n = Prelude.length backBuf
1017+ arr0 = fromListN n (Prelude.reverse backBuf)
1018+ return (Left (ParseError err), StreamK.fromPure arr0)
core/src/Streamly/Internal/Data/Array/Generic.hs+22−6
@@ -11,8 +11,8 @@ module Streamly.Internal.Data.Array.Generic
1111
1212 -- * Construction
1313 , nil
14− , writeN
15− , write
14+ , createOf
15+ , create
1616 , writeWith
1717 , writeLastN
1818
@@ -44,6 +44,10 @@ module Streamly.Internal.Data.Array.Generic
4444 , getIndex
4545 , getSliceUnsafe
4646 , strip
47+
48+ -- * Deprecated
49+ , writeN
50+ , write
4751 )
4852where
4953
@@ -104,21 +108,33 @@ nil = unsafePerformIO $ unsafeFreeze <$> MArray.nil
104108-- Construction - Folds
105109-------------------------------------------------------------------------------
106110
107−{-# INLINE_NORMAL writeN #-}
111+{-# INLINE_NORMAL createOf #-}
112+createOf :: MonadIO m => Int -> Fold m a (Array a)
113+createOf = fmap unsafeFreeze <$> MArray.createOf
114+
115+-- XXX Deprecate in major
116+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
117+{-# INLINE writeN #-}
108118writeN :: MonadIO m => Int -> Fold m a (Array a)
109−writeN = fmap unsafeFreeze <$> MArray.writeN
119+writeN = createOf
110120
111121{-# INLINE_NORMAL writeWith #-}
112122writeWith :: MonadIO m => Int -> Fold m a (Array a)
113−writeWith elemCount = unsafeFreeze <$> MArray.writeWith elemCount
123+writeWith elemCount = unsafeFreeze <$> MArray.createWith elemCount
114124
115125-- | Fold the whole input to a single array.
116126--
117127-- /Caution! Do not use this on infinite streams./
118128--
129+{-# INLINE create #-}
130+create :: MonadIO m => Fold m a (Array a)
131+create = fmap unsafeFreeze MArray.create
132+
133+-- XXX Deprecate in major
134+-- {-# DEPRECATED write "Please use create instead." #-}
119135{-# INLINE write #-}
120136write :: MonadIO m => Fold m a (Array a)
121−write = fmap unsafeFreeze MArray.write
137+write = create
122138
123139fromPureStream :: Stream Identity a -> Array a
124140fromPureStream x =
core/src/Streamly/Internal/Data/Stream/Chunked.hs → core/src/Streamly/Internal/Data/Array/Stream.hsrenamed+74−586
@@ -1,5 +1,6 @@
1+{-# OPTIONS_GHC -Wno-deprecations #-}
12-- |
2−-- Module : Streamly.Internal.Data.Stream.Chunked
3+-- Module : Streamly.Internal.Data.Array.Stream
34-- Copyright : (c) 2019 Composewell Technologies
45-- License : BSD3-3-Clause
56-- Maintainer : streamly@composewell.com
@@ -7,17 +8,25 @@
78--
89-- Combinators to efficiently manipulate streams of immutable arrays.
910--
10−module Streamly.Internal.Data.Stream.Chunked
11+-- We can either push these in the MutArray module with a "chunks" prefix or
12+-- keep this as a separate module and release it.
13+--
14+module Streamly.Internal.Data.Array.Stream
15+{-# DEPRECATED "Please use \"Streamly.Internal.Data.Array\" instead." #-}
1116 (
1217 -- * Creation
13− chunksOf
18+ Array.chunksOf
19+ , Array.pinnedChunksOf
20+ , Array.bufferChunks
1421
1522 -- * Flattening to elements
16− , concat
17− , concatRev
18− , interpose
19− , interposeSuffix
20− , intercalateSuffix
23+ , Array.concat
24+ , Array.flattenArrays
25+ , Array.concatRev
26+ , Array.flattenArraysRev
27+ , Array.interpose
28+ , Array.interposeSuffix
29+ , Array.intercalateSuffix
2130 , unlines
2231
2332 -- * Elimination
@@ -26,24 +35,27 @@ module Streamly.Internal.Data.Stream.Chunked
2635 -- can flatten the stream to byte stream and use that. But if we want the
2736 -- remaining stream to be a chunk stream then this could be handy. But it
2837 -- could also be implemented using parseBreak.
29− , foldBreak -- StreamK.foldBreakChunks
38+ , foldBreak
3039 , foldBreakD
31− -- The byte level parseBreak cannot work efficiently. Because the stream
32− -- will have to be a StreamK for backtracking, StreamK at byte level would
33− -- not be efficient.
34− , parseBreak -- StreamK.parseBreakChunks
40+ -- This is chunked parseBreak. A byte level parseBreak cannot work
41+ -- efficiently. Because the stream will have to be a StreamK for
42+ -- backtracking, StreamK at byte level would not be efficient.
43+ -- parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
44+ , parseBreak
3545 -- , parseBreakD
3646 -- , foldManyChunks
3747 -- , parseManyChunks
48+ , K.parseBreakChunks
49+ , K.parseChunks
3850
3951 -- ** Array Folds
40− -- XXX Use Parser.Chunked instead, need only chunkedParseBreak,
52+ -- XXX Use parseBreakChunks/parseChunks instead
4153 -- foldBreak can be implemented using parseBreak. Use StreamK.
4254 , runArrayFold
4355 , runArrayFoldBreak
4456 -- , parseArr
45− , runArrayParserDBreak -- StreamK.chunkedParseBreak
46− , runArrayFoldMany -- StreamK.chunkedParseMany
57+ , runArrayParserDBreak -- StreamK.parseBreakChunks
58+ , runArrayFoldMany
4759
4860 , toArray
4961
@@ -67,6 +79,8 @@ where
6779import Data.Bifunctor (second)
6880import Control.Exception (assert)
6981import Control.Monad.IO.Class (MonadIO(..))
82+-- import Data.Bifunctor (first)
83+-- import Data.Either (fromRight)
7084import Data.Proxy (Proxy(..))
7185import Data.Word (Word8)
7286import Streamly.Internal.Data.Unbox (Unbox(..))
@@ -82,142 +96,32 @@ import Streamly.Internal.Data.Parser (ParseError(..))
8296import Streamly.Internal.Data.Stream (Stream)
8397import Streamly.Internal.Data.StreamK (StreamK, fromStream, toStream)
8498import Streamly.Internal.Data.SVar.Type (adaptState, defState)
85−import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
8699
87−import qualified Streamly.Data.Fold as FL
88100import qualified Streamly.Internal.Data.Array as A
89101import qualified Streamly.Internal.Data.Array as Array
90−import qualified Streamly.Internal.Data.MutArray.Type as MA
91−import qualified Streamly.Internal.Data.Stream.MutChunked as AS
92−import qualified Streamly.Internal.Data.Fold.Type as FL (Fold(..), Step(..))
93102import qualified Streamly.Internal.Data.Parser as PR
94103import qualified Streamly.Internal.Data.Parser as PRD
95104 (Parser(..), Initial(..))
105+-- import qualified Streamly.Internal.Data.ParserK as ParserK
96106import qualified Streamly.Internal.Data.Stream as D
97107import qualified Streamly.Internal.Data.StreamK as K
98108
99109-- XXX Since these are immutable arrays MonadIO constraint can be removed from
100110-- most places.
101111
102−-------------------------------------------------------------------------------
103−-- Generation
104−-------------------------------------------------------------------------------
105−
106−-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
107−-- @n@ elements each.
108−--
109−-- > chunksOf n = Stream.groupsOf n (Array.writeN n)
110−--
111−-- /Pre-release/
112−{-# INLINE chunksOf #-}
113−chunksOf :: (MonadIO m, Unbox a)
114− => Int -> Stream m a -> Stream m (Array a)
115−chunksOf = A.chunksOf
116−
117−-------------------------------------------------------------------------------
118−-- Append
119−-------------------------------------------------------------------------------
120−
121−-- XXX efficiently compare two streams of arrays. Two streams can have chunks
122−-- of different sizes, we can handle that in the stream comparison abstraction.
123−-- This could be useful e.g. to fast compare whether two files differ.
124−
125−-- | Convert a stream of arrays into a stream of their elements.
126−--
127−-- Same as the following:
128−--
129−-- > concat = Stream.unfoldMany Array.read
130−--
131−-- @since 0.7.0
132−{-# INLINE concat #-}
133−concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
134−-- concat m = fromStreamD $ A.flattenArrays (toStreamD m)
135−-- concat m = fromStreamD $ D.concatMap A.toStreamD (toStreamD m)
136−concat = D.unfoldMany A.reader
137−
138−-- | Convert a stream of arrays into a stream of their elements reversing the
139−-- contents of each array before flattening.
140−--
141−-- > concatRev = Stream.unfoldMany Array.readerRev
142−--
143−-- @since 0.7.0
144−{-# INLINE concatRev #-}
145−concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
146−-- concatRev m = fromStreamD $ A.flattenArraysRev (toStreamD m)
147−concatRev = D.unfoldMany A.readerRev
148−
149112-------------------------------------------------------------------------------
150113-- Intersperse and append
151114-------------------------------------------------------------------------------
152115
153−-- | Flatten a stream of arrays after inserting the given element between
154−-- arrays.
155−--
156−-- /Pre-release/
157−{-# INLINE interpose #-}
158−interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
159−interpose x = D.interpose x A.reader
160−
161−{-# INLINE intercalateSuffix #-}
162−intercalateSuffix :: (Monad m, Unbox a)
163− => Array a -> Stream m (Array a) -> Stream m a
164−intercalateSuffix = D.intercalateSuffix A.reader
165−
166−-- | Flatten a stream of arrays appending the given element after each
167−-- array.
168−--
169−-- @since 0.7.0
170−{-# INLINE interposeSuffix #-}
171−interposeSuffix :: (Monad m, Unbox a)
172− => a -> Stream m (Array a) -> Stream m a
173−-- interposeSuffix x = fromStreamD . A.unlines x . toStreamD
174−interposeSuffix x = D.interposeSuffix x A.reader
175−
176−data FlattenState s =
177− OuterLoop s
178− | InnerLoop s !MA.MutableByteArray !Int !Int
179−
180−-- XXX This is a special case of interposeSuffix, can be removed.
181−-- XXX Remove monadIO constraint
182116{-# INLINE_NORMAL unlines #-}
183117unlines :: forall m a. (MonadIO m, Unbox a)
184118 => a -> D.Stream m (Array a) -> D.Stream m a
185−unlines sep (D.Stream step state) = D.Stream step' (OuterLoop state)
186− where
187− {-# INLINE_LATE step' #-}
188− step' gst (OuterLoop st) = do
189− r <- step (adaptState gst) st
190− return $ case r of
191− D.Yield Array{..} s ->
192− D.Skip (InnerLoop s arrContents arrStart arrEnd)
193− D.Skip s -> D.Skip (OuterLoop s)
194− D.Stop -> D.Stop
195−
196− step' _ (InnerLoop st _ p end) | p == end =
197− return $ D.Yield sep $ OuterLoop st
198−
199− step' _ (InnerLoop st contents p end) = do
200− x <- liftIO $ peekByteIndex p contents
201− return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
119+unlines = Array.interposeSuffix
202120
203121-------------------------------------------------------------------------------
204122-- Compact
205123-------------------------------------------------------------------------------
206124
207−-- XXX These would not be needed once we implement compactLEFold, see
208−-- module Streamly.Internal.Data.Stream.MutChunked
209−--
210−-- XXX Note that this thaws immutable arrays for appending, that may be
211−-- problematic if multiple users do the same thing, however, immutable arrays
212−-- would usually have no capacity to append, therefore, a copy will be forced
213−-- anyway. Confirm this. We can forcefully trim the array capacity before thaw
214−-- to ensure this.
215−{-# INLINE_NORMAL packArraysChunksOf #-}
216−packArraysChunksOf :: (MonadIO m, Unbox a)
217− => Int -> D.Stream m (Array a) -> D.Stream m (Array a)
218−packArraysChunksOf n str =
219− D.map A.unsafeFreeze $ AS.packArraysChunksOf n $ D.map A.unsafeThaw str
220−
221125-- XXX instead of writing two different versions of this operation, we should
222126-- write it as a pipe.
223127--
@@ -225,8 +129,7 @@ packArraysChunksOf n str =
225129{-# INLINE_NORMAL lpackArraysChunksOf #-}
226130lpackArraysChunksOf :: (MonadIO m, Unbox a)
227131 => Int -> Fold m (Array a) () -> Fold m (Array a) ()
228−lpackArraysChunksOf n fld =
229− FL.lmap A.unsafeThaw $ AS.lpackArraysChunksOf n (FL.lmap A.unsafeFreeze fld)
132+lpackArraysChunksOf = Array.lCompactGE
230133
231134-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
232135-- maximum specified size in bytes.
@@ -235,68 +138,19 @@ lpackArraysChunksOf n fld =
235138{-# INLINE compact #-}
236139compact :: (MonadIO m, Unbox a)
237140 => Int -> Stream m (Array a) -> Stream m (Array a)
238−compact = packArraysChunksOf
141+compact = Array.compactLE
239142
240−-------------------------------------------------------------------------------
241−-- Split
242−-------------------------------------------------------------------------------
243−
244−data SplitState s arr
245− = Initial s
246− | Buffering s arr
247− | Splitting s arr
248− | Yielding arr (SplitState s arr)
249− | Finishing
250−
251−-- | Split a stream of arrays on a given separator byte, dropping the separator
252−-- and coalescing all the arrays between two separators into a single array.
143+-- | Given a stream of arrays, splice them all together to generate a single
144+-- array. The stream must be /finite/.
253145--
254146-- @since 0.7.0
255−{-# INLINE_NORMAL _splitOn #-}
256−_splitOn
257− :: MonadIO m
258− => Word8
259− -> D.Stream m (Array Word8)
260− -> D.Stream m (Array Word8)
261−_splitOn byte (D.Stream step state) = D.Stream step' (Initial state)
262−
263− where
264−
265− {-# INLINE_LATE step' #-}
266− step' gst (Initial st) = do
267− r <- step gst st
268− case r of
269− D.Yield arr s -> do
270− (arr1, marr2) <- A.breakOn byte arr
271− return $ case marr2 of
272− Nothing -> D.Skip (Buffering s arr1)
273− Just arr2 -> D.Skip (Yielding arr1 (Splitting s arr2))
274− D.Skip s -> return $ D.Skip (Initial s)
275− D.Stop -> return D.Stop
276−
277− step' gst (Buffering st buf) = do
278− r <- step gst st
279− case r of
280− D.Yield arr s -> do
281− (arr1, marr2) <- A.breakOn byte arr
282− buf' <- A.splice buf arr1
283− return $ case marr2 of
284− Nothing -> D.Skip (Buffering s buf')
285− Just x -> D.Skip (Yielding buf' (Splitting s x))
286− D.Skip s -> return $ D.Skip (Buffering s buf)
287− D.Stop -> return $
288− if A.byteLength buf == 0
289− then D.Stop
290− else D.Skip (Yielding buf Finishing)
291−
292− step' _ (Splitting st buf) = do
293− (arr1, marr2) <- A.breakOn byte buf
294− return $ case marr2 of
295− Nothing -> D.Skip $ Buffering st arr1
296− Just arr2 -> D.Skip $ Yielding arr1 (Splitting st arr2)
147+{-# INLINE toArray #-}
148+toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
149+toArray = Array.fromChunks
297150
298− step' _ (Yielding arr next) = return $ D.Yield arr next
299− step' _ Finishing = return D.Stop
151+-------------------------------------------------------------------------------
152+-- Split
153+-------------------------------------------------------------------------------
300154
301155-- XXX Remove MonadIO constraint.
302156-- | Split a stream of arrays on a given separator byte, dropping the separator
@@ -309,7 +163,7 @@ splitOn
309163 => Word8
310164 -> Stream m (Array Word8)
311165 -> Stream m (Array Word8)
312−splitOn byte = D.splitInnerBy (A.breakOn byte) A.splice
166+splitOn = Array.compactOnByte
313167
314168{-# INLINE splitOnSuffix #-}
315169splitOnSuffix
@@ -317,8 +171,7 @@ splitOnSuffix
317171 => Word8
318172 -> Stream m (Array Word8)
319173 -> Stream m (Array Word8)
320−-- splitOn byte s = fromStreamD $ A.splitOn byte $ toStreamD s
321−splitOnSuffix byte = D.splitInnerBySuffix (A.breakOn byte) A.splice
174+splitOnSuffix = Array.compactOnByteSuffix
322175
323176-------------------------------------------------------------------------------
324177-- Elimination - Running folds
@@ -327,77 +180,24 @@ splitOnSuffix byte = D.splitInnerBySuffix (A.breakOn byte) A.splice
327180{-# INLINE_NORMAL foldBreakD #-}
328181foldBreakD :: forall m a b. (MonadIO m, Unbox a) =>
329182 Fold m a b -> D.Stream m (Array a) -> m (b, D.Stream m (Array a))
330−foldBreakD (FL.Fold fstep initial extract) stream@(D.Stream step state) = do
331− res <- initial
332− case res of
333− FL.Partial fs -> go SPEC state fs
334− FL.Done fb -> return $! (fb, stream)
335−
336− where
337−
338− {-# INLINE go #-}
339− go !_ st !fs = do
340− r <- step defState st
341− case r of
342− D.Yield (Array contents start end) s ->
343− let fp = Tuple' end contents
344− in goArray SPEC s fp start fs
345− D.Skip s -> go SPEC s fs
346− D.Stop -> do
347− b <- extract fs
348− return (b, D.nil)
349−
350− goArray !_ s (Tuple' end _) !cur !fs
351− | cur == end = do
352− go SPEC s fs
353− goArray !_ st fp@(Tuple' end contents) !cur !fs = do
354− x <- liftIO $ peekByteIndex cur contents
355− res <- fstep fs x
356− let next = INDEX_NEXT(cur,a)
357− case res of
358− FL.Done b -> do
359− let arr = Array contents next end
360− return $! (b, D.cons arr (D.Stream step st))
361− FL.Partial fs1 -> goArray SPEC st fp next fs1
362−
363−{-# INLINE_NORMAL foldBreakK #-}
364−foldBreakK :: forall m a b. (MonadIO m, Unbox a) =>
365− Fold m a b -> K.StreamK m (Array a) -> m (b, K.StreamK m (Array a))
366−foldBreakK (FL.Fold fstep initial extract) stream = do
367− res <- initial
368− case res of
369− FL.Partial fs -> go fs stream
370− FL.Done fb -> return (fb, stream)
371−
372− where
373−
374− {-# INLINE go #-}
375− go !fs st = do
376− let stop = (, K.nil) <$> extract fs
377− single a = yieldk a K.nil
378− yieldk (Array contents start end) r =
379− let fp = Tuple' end contents
380− in goArray fs r fp start
381− in K.foldStream defState yieldk single stop st
382−
383− goArray !fs st (Tuple' end _) !cur
384− | cur == end = do
385− go fs st
386− goArray !fs st fp@(Tuple' end contents) !cur = do
387− x <- liftIO $ peekByteIndex cur contents
388− res <- fstep fs x
389− let next = INDEX_NEXT(cur,a)
390− case res of
391− FL.Done b -> do
392− let arr = Array contents next end
393− return $! (b, K.cons arr st)
394− FL.Partial fs1 -> goArray fs1 st fp next
183+foldBreakD = Array.foldBreakChunks
395184
396185-- | Fold an array stream using the supplied 'Fold'. Returns the fold result
397186-- and the unconsumed stream.
398187--
399188-- > foldBreak f = runArrayFoldBreak (ChunkFold.fromFold f)
400189--
190+-- Instead of using this we can adapt the fold to ParserK and use
191+-- parseBreakChunks instead. ParserK allows composing using Monad as well.
192+--
193+-- @
194+-- foldBreak f s =
195+-- fmap (first (fromRight undefined))
196+-- $ K.parseBreakChunks (ParserK.adaptC (PR.fromFold f)) s
197+-- @
198+--
199+-- We can compare perf and remove this one or define it in terms of that.
200+--
401201-- /Internal/
402202--
403203{-# INLINE_NORMAL foldBreak #-}
@@ -406,14 +206,20 @@ foldBreak ::
406206 => Fold m a b
407207 -> StreamK m (A.Array a)
408208 -> m (b, StreamK m (A.Array a))
209+foldBreak = Array.foldBreakChunksK
210+--
409211-- foldBreak f s = fmap fromStreamD <$> foldBreakD f (toStreamD s)
410−foldBreak = foldBreakK
212+--
213+-- foldBreak f s =
214+-- fmap (first (fromRight undefined))
215+-- $ K.parseBreakChunks (ParserK.adaptC (PR.fromFold f)) s
216+--
411217-- If foldBreak performs better than runArrayFoldBreak we can use a rewrite
412218-- rule to rewrite runArrayFoldBreak to fold.
413219-- foldBreak f = runArrayFoldBreak (ChunkFold.fromFold f)
414220
415221-------------------------------------------------------------------------------
416−-- Fold to a single Array
222+-- Elimination - running element parsers
417223-------------------------------------------------------------------------------
418224
419225-- When we have to take an array partially, take the last part of the array.
@@ -459,341 +265,22 @@ splitAtArrayListRev n ls
459265 arr1 = Array contents end1 end
460266 in ([arr1], arr2:xs)
461267
462−-------------------------------------------------------------------------------
463−-- Fold to a single Array
464−-------------------------------------------------------------------------------
465−
466−{-
467−
468−-- XXX Both of these implementations of splicing seem to perform equally well.
469−-- We need to perform benchmarks over a range of sizes though.
470−
471−-- CAUTION! length must more than equal to lengths of all the arrays in the
472−-- stream.
473−{-# INLINE spliceArraysLenUnsafe #-}
474−spliceArraysLenUnsafe :: (MonadIO m, Unbox a)
475− => Int -> Stream m (MutArray a) -> m (MutArray a)
476−spliceArraysLenUnsafe len buffered = do
477− -- XXX The new array's pinned state should depend on the first element
478− -- of the stream. We should uncons the stream and do the required.
479− arr <- undefined
480− D.foldlM' MA.spliceUnsafe (return arr) buffered
481−
482−{-# INLINE _spliceArrays #-}
483−_spliceArrays :: (MonadIO m, Unbox a)
484− => Stream m (Array a) -> m (Array a)
485−_spliceArrays s = do
486− buffered <- D.foldr K.cons K.nil s
487− len <- K.fold FL.sum (fmap Array.length buffered)
488− -- XXX The new array's pinned state should depend on the first element
489− -- of the stream. We should uncons the stream and do the required.
490− arr <- undefined
491− final <- D.foldlM' writeArr (return arr) (toStream buffered)
492− return $ A.unsafeFreeze final
493−
494− where
495−
496− writeArr dst arr = MA.spliceUnsafe dst (A.unsafeThaw arr)
497−
498−{-# INLINE _spliceArraysBuffered #-}
499−_spliceArraysBuffered :: (MonadIO m, Unbox a)
500− => Stream m (Array a) -> m (Array a)
501−_spliceArraysBuffered s = do
502− buffered <- D.foldr K.cons K.nil s
503− len <- K.fold FL.sum (fmap Array.length buffered)
504− A.unsafeFreeze <$>
505− spliceArraysLenUnsafe len (fmap A.unsafeThaw (toStream buffered))
506−
507−-}
508−
509−{-# INLINE spliceArraysRealloced #-}
510−spliceArraysRealloced :: forall m a. (MonadIO m, Unbox a)
511− => Stream m (Array a) -> m (Array a)
512−spliceArraysRealloced s = do
513− res <- D.uncons s
514− case res of
515− Just (a, strm) -> do
516− arr <-
517− D.foldlM'
518− MA.spliceExp
519− (pure (A.unsafeThaw a))
520− (fmap A.unsafeThaw strm)
521− liftIO $ A.unsafeFreeze <$> MA.rightSize arr
522− Nothing -> pure A.nil
523−
524−-- XXX This should just be "fold A.write"
525−--
526−-- | Given a stream of arrays, splice them all together to generate a single
527−-- array. The stream must be /finite/.
528−--
529−-- @since 0.7.0
530−{-# INLINE toArray #-}
531−toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
532−toArray = spliceArraysRealloced
533−-- spliceArrays = _spliceArraysBuffered
534−
535−-- exponentially increasing sizes of the chunks upto the max limit.
536−-- XXX this will be easier to implement with parsers/terminating folds
537−-- With this we should be able to reduce the number of chunks/allocations.
538−-- The reallocation/copy based toArray can also be implemented using this.
539−--
540−{-
541−{-# INLINE toArraysInRange #-}
542−toArraysInRange :: (MonadIO m, Unbox a)
543− => Int -> Int -> Fold m (Array a) b -> Fold m a b
544−toArraysInRange low high (Fold step initial extract) =
545−-}
546−
547−{-
548−-- | Fold the input to a pure buffered stream (List) of arrays.
549−{-# INLINE _toArraysOf #-}
550−_toArraysOf :: (MonadIO m, Unbox a)
551− => Int -> Fold m a (Stream Identity (Array a))
552−_toArraysOf n = FL.groupsOf n (A.writeNF n) FL.toStream
553−-}
554−
555−-------------------------------------------------------------------------------
556−-- Elimination - running element parsers
557−-------------------------------------------------------------------------------
558−
559268-- GHC parser does not accept {-# ANN type [] NoSpecConstr #-}, so we need
560269-- to make a newtype.
561270{-# ANN type List NoSpecConstr #-}
562271newtype List a = List {getList :: [a]}
563272
564−{-
565−-- This can be generalized to any type provided it can be unfolded to a stream
566−-- and it can be combined using a semigroup operation.
567−--
568−-- XXX This should be written using CPS (as parseK) if we want it to scale wrt
569−-- to the number of times it can be called on the same stream.
570−{-# INLINE_NORMAL parseBreakD #-}
571−parseBreakD ::
572− forall m a b. (MonadIO m, MonadThrow m, Unbox a)
573− => PRD.Parser a m b
574− -> D.Stream m (Array.Array a)
575− -> m (b, D.Stream m (Array.Array a))
576−parseBreakD
577− (PRD.Parser pstep initial extract) stream@(D.Stream step state) = do
578−
579− res <- initial
580− case res of
581− PRD.IPartial s -> go SPEC state (List []) s
582− PRD.IDone b -> return (b, stream)
583− PRD.IError err -> throwM $ ParseError err
584−
585− where
586−
587− -- "backBuf" contains last few items in the stream that we may have to
588− -- backtrack to.
589− --
590− -- XXX currently we are using a dumb list based approach for backtracking
591− -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
592− -- That will allow us more efficient random back and forth movement.
593− go !_ st backBuf !pst = do
594− r <- step defState st
595− case r of
596− D.Yield (Array contents start end) s ->
597− gobuf SPEC s backBuf
598− (Tuple' end contents) start pst
599− D.Skip s -> go SPEC s backBuf pst
600− D.Stop -> do
601− b <- extract pst
602− return (b, D.nil)
603−
604− -- Use strictness on "cur" to keep it unboxed
605− gobuf !_ s backBuf (Tuple' end _) !cur !pst
606− | cur == end = do
607− go SPEC s backBuf pst
608− gobuf !_ s backBuf fp@(Tuple' end contents) !cur !pst = do
609− x <- liftIO $ peekByteIndex contents cur
610− pRes <- pstep pst x
611− let next = INDEX_NEXT(cur,a)
612− case pRes of
613− PR.Partial 0 pst1 ->
614− gobuf SPEC s (List []) fp next pst1
615− PR.Partial n pst1 -> do
616− assert (n <= Prelude.length (x:getList backBuf)) (return ())
617− let src0 = Prelude.take n (x:getList backBuf)
618− arr0 = A.fromListN n (Prelude.reverse src0)
619− arr1 = Array contents next end
620− src = arr0 <> arr1
621− let !(Array cont1 start end1) = src
622− fp1 = Tuple' end1 cont1
623− gobuf SPEC s (List []) fp1 start pst1
624− PR.Continue 0 pst1 ->
625− gobuf SPEC s (List (x:getList backBuf)) fp next pst1
626− PR.Continue n pst1 -> do
627− assert (n <= Prelude.length (x:getList backBuf)) (return ())
628− let (src0, buf1) = splitAt n (x:getList backBuf)
629− arr0 = A.fromListN n (Prelude.reverse src0)
630− arr1 = Array contents next end
631− src = arr0 <> arr1
632− let !(Array cont1 start end1) = src
633− fp1 = Tuple' end1 cont1
634− gobuf SPEC s (List buf1) fp1 start pst1
635− PR.Done 0 b -> do
636− let arr = Array contents next end
637− return (b, D.cons arr (D.Stream step s))
638− PR.Done n b -> do
639− assert (n <= Prelude.length (x:getList backBuf)) (return ())
640− let src0 = Prelude.take n (x:getList backBuf)
641− -- XXX create the array in reverse instead
642− arr0 = A.fromListN n (Prelude.reverse src0)
643− arr1 = Array contents next end
644− -- XXX Use StreamK to avoid adding arbitrary layers of
645− -- constructors every time.
646− str = D.cons arr0 (D.cons arr1 (D.Stream step s))
647− return (b, str)
648− PR.Error err -> throwM $ ParseError err
649−-}
650−
651−{-# INLINE_NORMAL parseBreakK #-}
652−parseBreakK ::
653− forall m a b. (MonadIO m, Unbox a)
654− => PRD.Parser a m b
655− -> K.StreamK m (Array.Array a)
656− -> m (Either ParseError b, K.StreamK m (Array.Array a))
657−parseBreakK (PRD.Parser pstep initial extract) stream = do
658− res <- initial
659− case res of
660− PRD.IPartial s -> go s stream []
661− PRD.IDone b -> return (Right b, stream)
662− PRD.IError err -> return (Left (ParseError err), stream)
663−
664− where
665−
666− -- "backBuf" contains last few items in the stream that we may have to
667− -- backtrack to.
668− --
669− -- XXX currently we are using a dumb list based approach for backtracking
670− -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
671− -- That will allow us more efficient random back and forth movement.
672− go !pst st backBuf = do
673− let stop = goStop pst backBuf -- (, K.nil) <$> extract pst
674− single a = yieldk a K.nil
675− yieldk arr r = goArray pst backBuf r arr
676− in K.foldStream defState yieldk single stop st
677−
678− -- Use strictness on "cur" to keep it unboxed
679− goArray !pst backBuf st (Array _ cur end) | cur == end = go pst st backBuf
680− goArray !pst backBuf st (Array contents cur end) = do
681− x <- liftIO $ peekByteIndex cur contents
682− pRes <- pstep pst x
683− let next = INDEX_NEXT(cur,a)
684− case pRes of
685− PR.Partial 0 s ->
686− goArray s [] st (Array contents next end)
687− PR.Partial n s -> do
688− assert (n <= Prelude.length (x:backBuf)) (return ())
689− let src0 = Prelude.take n (x:backBuf)
690− arr0 = A.fromListN n (Prelude.reverse src0)
691− arr1 = Array contents next end
692− src = arr0 <> arr1
693− goArray s [] st src
694− PR.Continue 0 s ->
695− goArray s (x:backBuf) st (Array contents next end)
696− PR.Continue n s -> do
697− assert (n <= Prelude.length (x:backBuf)) (return ())
698− let (src0, buf1) = splitAt n (x:backBuf)
699− arr0 = A.fromListN n (Prelude.reverse src0)
700− arr1 = Array contents next end
701− src = arr0 <> arr1
702− goArray s buf1 st src
703− PR.Done 0 b -> do
704− let arr = Array contents next end
705− return (Right b, K.cons arr st)
706− PR.Done n b -> do
707− assert (n <= Prelude.length (x:backBuf)) (return ())
708− let src0 = Prelude.take n (x:backBuf)
709− -- XXX Use fromListRevN once implemented
710− -- arr0 = A.fromListRevN n src0
711− arr0 = A.fromListN n (Prelude.reverse src0)
712− arr1 = Array contents next end
713− str = K.cons arr0 (K.cons arr1 st)
714− return (Right b, str)
715− PR.Error err -> do
716− let n = Prelude.length backBuf
717− arr0 = A.fromListN n (Prelude.reverse backBuf)
718− arr1 = Array contents cur end
719− str = K.cons arr0 (K.cons arr1 stream)
720− return (Left (ParseError err), str)
721−
722− -- This is a simplified goArray
723− goExtract !pst backBuf (Array _ cur end)
724− | cur == end = goStop pst backBuf
725− goExtract !pst backBuf (Array contents cur end) = do
726− x <- liftIO $ peekByteIndex cur contents
727− pRes <- pstep pst x
728− let next = INDEX_NEXT(cur,a)
729− case pRes of
730− PR.Partial 0 s ->
731− goExtract s [] (Array contents next end)
732− PR.Partial n s -> do
733− assert (n <= Prelude.length (x:backBuf)) (return ())
734− let src0 = Prelude.take n (x:backBuf)
735− arr0 = A.fromListN n (Prelude.reverse src0)
736− arr1 = Array contents next end
737− src = arr0 <> arr1
738− goExtract s [] src
739− PR.Continue 0 s ->
740− goExtract s backBuf (Array contents next end)
741− PR.Continue n s -> do
742− assert (n <= Prelude.length (x:backBuf)) (return ())
743− let (src0, buf1) = splitAt n (x:backBuf)
744− arr0 = A.fromListN n (Prelude.reverse src0)
745− arr1 = Array contents next end
746− src = arr0 <> arr1
747− goExtract s buf1 src
748− PR.Done 0 b -> do
749− let arr = Array contents next end
750− return (Right b, K.fromPure arr)
751− PR.Done n b -> do
752− assert (n <= Prelude.length backBuf) (return ())
753− let src0 = Prelude.take n backBuf
754− -- XXX Use fromListRevN once implemented
755− -- arr0 = A.fromListRevN n src0
756− arr0 = A.fromListN n (Prelude.reverse src0)
757− arr1 = Array contents next end
758− str = K.cons arr0 (K.fromPure arr1)
759− return (Right b, str)
760− PR.Error err -> do
761− let n = Prelude.length backBuf
762− arr0 = A.fromListN n (Prelude.reverse backBuf)
763− arr1 = Array contents cur end
764− str = K.cons arr0 (K.cons arr1 stream)
765− return (Left (ParseError err), str)
766−
767− -- This is a simplified goExtract
768− {-# INLINE goStop #-}
769− goStop !pst backBuf = do
770− pRes <- extract pst
771− case pRes of
772− PR.Partial _ _ -> error "Bug: parseBreak: Partial in extract"
773− PR.Continue 0 s ->
774− goStop s backBuf
775− PR.Continue n s -> do
776− assert (n <= Prelude.length backBuf) (return ())
777− let (src0, buf1) = splitAt n backBuf
778− arr = A.fromListN n (Prelude.reverse src0)
779− goExtract s buf1 arr
780− PR.Done 0 b ->
781− return (Right b, K.nil)
782− PR.Done n b -> do
783− assert (n <= Prelude.length backBuf) (return ())
784− let src0 = Prelude.take n backBuf
785− -- XXX Use fromListRevN once implemented
786− -- arr0 = A.fromListRevN n src0
787− arr0 = A.fromListN n (Prelude.reverse src0)
788− return (Right b, K.fromPure arr0)
789− PR.Error err -> do
790− let n = Prelude.length backBuf
791− arr0 = A.fromListN n (Prelude.reverse backBuf)
792− return (Left (ParseError err), K.fromPure arr0)
793−
794273-- | Parse an array stream using the supplied 'Parser'. Returns the parse
795274-- result and the unconsumed stream. Throws 'ParseError' if the parse fails.
796275--
276+-- >> parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
277+--
278+-- This is redundant and we can just use parseBreakChunks, as ParserK can be
279+-- composed using Monad. The only advantage of this is that we do not need to
280+-- adapt.
281+--
282+-- We can compare perf and remove this one or define it in terms of that.
283+--
797284-- /Internal/
798285--
799286{-# INLINE_NORMAL parseBreak #-}
@@ -806,7 +293,8 @@ parseBreak ::
806293parseBreak p s =
807294 fmap fromStreamD <$> parseBreakD (PRD.fromParserK p) (toStreamD s)
808295-}
809−parseBreak = parseBreakK
296+parseBreak = Array.parseBreakChunksK
297+-- parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
810298
811299-------------------------------------------------------------------------------
812300-- Elimination - Running Array Folds and parsers
core/src/Streamly/Internal/Data/Array/Type.hs+529−104
@@ -1,8 +1,11 @@
11{-# LANGUAGE CPP #-}
2+{-# LANGUAGE TypeFamilies #-}
3+-- Must come after TypeFamilies, otherwise it is re-enabled.
4+-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
5+{-# LANGUAGE NoMonoLocalBinds #-}
26-- |
37-- Module : Streamly.Internal.Data.Array.Type
48-- Copyright : (c) 2020 Composewell Technologies
5−--
69-- License : BSD3-3-Clause
710-- Maintainer : streamly@composewell.com
811-- Stability : experimental
@@ -10,82 +13,146 @@
1013--
1114-- See notes in "Streamly.Internal.Data.MutArray.Type"
1215--
16+
1317module Streamly.Internal.Data.Array.Type
1418 (
19+ -- ** Type
1520 -- $arrayNotes
1621 Array (..)
17− , asPtrUnsafe
18− , nil
1922
20− -- * Freezing and Thawing
23+ -- ** Conversion
24+ -- *** Mutable and Immutable
2125 , unsafeFreeze
2226 , unsafeFreezeWithShrink
2327 , unsafeThaw
2428
25− -- * Pinning and Unpinning
29+ -- *** Pinned and Unpinned
2630 , pin
2731 , unpin
2832 , isPinned
2933
30− -- * Construction
31− , splice
34+ -- *** Casting
35+ , unsafePinnedAsPtr
3236
33− , fromList
34− , pinnedFromList
37+ -- ** Construction
38+ , empty
39+
40+ -- *** Cloning
41+ -- XXX Why would we clone an immutable array?
42+ , clone
43+ , pinnedClone
44+
45+ -- *** Slicing
46+ -- | Get a subarray without copying
47+ , splitAt
48+ , breakOn -- XXX requires MonadIO
49+
50+ -- *** Stream Folds
51+ , unsafeMakePure
52+ , createOf
53+ , pinnedCreateOf
54+ , unsafeCreateOf
55+ , unsafePinnedCreateOf
56+ , create
57+ , pinnedCreate
58+ , createWith
59+
60+ -- *** From containers
3561 , fromListN
3662 , pinnedFromListN
37− , fromListRev
63+ , fromList
64+ , pinnedFromList
3865 , fromListRevN
39− , fromStreamDN
40− , fromStreamD
66+ , fromListRev
67+ , fromStreamN
68+ , fromStream
69+ , fromPureStreamN
4170 , fromPureStream
4271 , fromByteStr#
72+ , fromByteStr
73+ , fromPtrN
74+ , fromChunks
75+ , fromChunksK
4376
44− -- * Split
45− , breakOn
77+ -- ** Reading
4678
47− -- * Elimination
48− , unsafeIndexIO
49− , getIndexUnsafe
50− , byteLength
79+ -- *** Indexing
80+ , unsafeIndexIO -- XXX unsafeGetIndexIO
81+ , getIndexUnsafe -- XXX unsafeGetIndex
82+
83+ -- *** To Streams
84+ , read
85+ , readRev
86+ , toStreamK
87+ , toStreamKRev
88+
89+ -- *** To Containers
90+ , toList
91+
92+ -- *** Unfolds
93+ , producer -- experimental
94+ , readerUnsafe
95+ , reader
96+ , readerRev
97+
98+ -- *** Size
5199 , length
100+ , byteLength
52101
102+ -- ** Folding
53103 , foldl'
54104 , foldr
55− , splitAt
105+ , byteCmp
106+ , byteEq
107+
108+ -- ** Appending
109+ , splice -- XXX requires MonadIO
110+
111+ -- ** Streams of arrays
112+ -- *** Chunk
113+ -- | Group a stream into arrays.
114+ , chunksOf
115+ , pinnedChunksOf
116+ , buildChunks
56117
118+ -- *** Split
119+ -- | Split an array into slices.
120+
121+ -- *** Concat
122+ -- | Append the arrays in a stream to form a stream of elements.
123+ , concat
124+ , concatRev
125+
126+ -- *** Compact
127+ -- | Append the arrays in a stream to form a stream of larger arrays.
128+ , fCompactGE
129+ , fPinnedCompactGE
130+ , lCompactGE
131+ , lPinnedCompactGE
132+ , compactGE
133+
134+ -- ** Deprecated
135+ , asPtrUnsafe
136+ , unsafeIndex
137+ , bufferChunks
138+ , flattenArrays
139+ , flattenArraysRev
140+ , fromArrayStreamK
141+ , fromStreamDN
142+ , fromStreamD
57143 , toStreamD
58144 , toStreamDRev
59− , toStreamK
60− , toStreamKRev
61145 , toStream
62146 , toStreamRev
63− , read
64− , readRev
65− , readerRev
66− , toList
67−
68− -- * Folds
147+ , nil
69148 , writeWith
70149 , writeN
71150 , pinnedWriteN
72151 , writeNUnsafe
73152 , pinnedWriteNUnsafe
74− , MA.ArrayUnsafe (..)
75153 , pinnedWriteNAligned
76154 , write
77155 , pinnedWrite
78− , unsafeMakePure
79−
80− -- * Streams of arrays
81− , chunksOf
82− , pinnedChunksOf
83− , bufferChunks
84− , flattenArrays
85− , flattenArraysRev
86−
87− -- * Deprecated
88− , unsafeIndex
89156 )
90157where
91158
@@ -93,31 +160,36 @@ where
93160#include "inline.hs"
94161
95162import Control.Exception (assert)
96−import Control.Monad (replicateM)
163+import Control.Monad (replicateM, when)
97164import Control.Monad.IO.Class (MonadIO(..))
98165import Data.Functor.Identity (Identity(..))
166+import Data.Int (Int8, Int16, Int32, Int64)
99167import Data.Proxy (Proxy(..))
100−import Data.Word (Word8)
168+import Data.Word (Word8, Word16, Word32, Word64)
101169import GHC.Base (build)
102170import GHC.Exts (IsList, IsString(..), Addr#)
103171
104172import GHC.IO (unsafePerformIO)
105173import GHC.Ptr (Ptr(..))
106−import Streamly.Internal.Data.MutArray.Type (MutArray(..), MutableByteArray)
174+import Streamly.Internal.Data.Producer.Type (Producer(..))
175+import Streamly.Internal.Data.MutArray.Type (MutArray(..))
176+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
107177import Streamly.Internal.Data.Fold.Type (Fold(..))
108178import Streamly.Internal.Data.Stream.Type (Stream)
179+import Streamly.Internal.Data.StreamK.Type (StreamK)
109180import Streamly.Internal.Data.Unbox (Unbox(..))
110181import Streamly.Internal.Data.Unfold.Type (Unfold(..))
111182import Text.Read (readPrec)
112183
113−import Prelude hiding (Foldable(..), read, unlines, splitAt)
184+import Prelude hiding (Foldable(..), concat, read, unlines, splitAt)
114185
115186import qualified GHC.Exts as Exts
187+import qualified Streamly.Internal.Data.Fold.Type as Fold
116188import qualified Streamly.Internal.Data.MutArray.Type as MA
117189import qualified Streamly.Internal.Data.Stream.Type as D
118−import qualified Streamly.Internal.Data.Stream.Generate as D
119190import qualified Streamly.Internal.Data.StreamK.Type as K
120−import qualified Streamly.Internal.Data.Unbox as Unboxed
191+import qualified Streamly.Internal.Data.MutByteArray.Type as Unboxed
192+import qualified Streamly.Internal.Data.Producer as Producer
121193import qualified Streamly.Internal.Data.Unfold.Type as Unfold
122194import qualified Text.ParserCombinators.ReadPrec as ReadPrec
123195
@@ -160,7 +232,7 @@ data Array a =
160232#endif
161233 -- All offsets are in terms of bytes from the start of arraycontents
162234 Array
163− { arrContents :: {-# UNPACK #-} !MutableByteArray
235+ { arrContents :: {-# UNPACK #-} !MutByteArray
164236 , arrStart :: {-# UNPACK #-} !Int -- offset
165237 , arrEnd :: {-# UNPACK #-} !Int -- offset + len
166238 }
@@ -171,14 +243,20 @@ data Array a =
171243
172244-- | Use an @Array a@ as @Ptr a@.
173245--
174−-- See 'MA.asPtrUnsafe' in the Mutable array module for more details.
246+-- See 'MA.unsafePinnedAsPtr' in the Mutable array module for more details.
175247--
176248-- /Unsafe/
177249--
178250-- /Pre-release/
179251--
252+{-# INLINE unsafePinnedAsPtr #-}
253+unsafePinnedAsPtr :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
254+unsafePinnedAsPtr arr = MA.unsafePinnedAsPtr (unsafeThaw arr)
255+
256+{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
257+{-# INLINE asPtrUnsafe #-}
180258asPtrUnsafe :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
181−asPtrUnsafe arr = MA.asPtrUnsafe (unsafeThaw arr)
259+asPtrUnsafe = unsafePinnedAsPtr
182260
183261-------------------------------------------------------------------------------
184262-- Freezing and Thawing
@@ -225,18 +303,19 @@ unsafeThaw (Array ac as ae) = MutArray ac as ae ae
225303-- Pinning & Unpinning
226304-------------------------------------------------------------------------------
227305
228−-- | Copy the 'Array' to pinned memory if unpinned, else do nothing. The
229−-- overhead is a copy if the input array is unpinned.
306+-- | Return a copy of the 'Array' in pinned memory if unpinned, else return the
307+-- original array.
230308{-# INLINE pin #-}
231309pin :: Array a -> IO (Array a)
232310pin = fmap unsafeFreeze . MA.pin . unsafeThaw
233311
234−-- | Copy the 'Array' to unpinned memory if pinned, else do nothing. The
235−-- overhead is a copy if the input array is pinned.
312+-- | Return a copy of the 'Array' in unpinned memory if pinned, else return the
313+-- original array.
236314{-# INLINE unpin #-}
237315unpin :: Array a -> IO (Array a)
238316unpin = fmap unsafeFreeze . MA.unpin . unsafeThaw
239317
318+-- | Return 'True' if the array is allocated in pinned memory.
240319{-# INLINE isPinned #-}
241320isPinned :: Array a -> Bool
242321isPinned = MA.isPinned . unsafeThaw
@@ -245,11 +324,14 @@ isPinned = MA.isPinned . unsafeThaw
245324-- Construction
246325-------------------------------------------------------------------------------
247326
248−-- Splice two immutable arrays creating a new array.
327+-- | Copy two immutable arrays into a new array. If you want to splice more
328+-- than two arrays then this operation would be highly inefficient because it
329+-- would make a copy on every splice operation, instead use the
330+-- 'fromChunksK' operation to combine n immutable arrays.
249331{-# INLINE splice #-}
250−splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
332+splice :: MonadIO m => Array a -> Array a -> m (Array a)
251333splice arr1 arr2 =
252− unsafeFreeze <$> MA.splice (unsafeThaw arr1) (unsafeThaw arr2)
334+ unsafeFreeze <$> MA.spliceCopy (unsafeThaw arr1) (unsafeThaw arr2)
253335
254336-- | Create an 'Array' from the first N elements of a list. The array is
255337-- allocated to size N, if the list terminates before N elements then the
@@ -293,24 +375,83 @@ pinnedFromList xs = unsafePerformIO $ unsafeFreeze <$> MA.pinnedFromList xs
293375fromListRev :: Unbox a => [a] -> Array a
294376fromListRev xs = unsafePerformIO $ unsafeFreeze <$> MA.fromListRev xs
295377
296−{-# INLINE_NORMAL fromStreamDN #-}
378+-- | Create an 'Array' from the first N elements of a stream. The array is
379+-- allocated to size N, if the stream terminates before N elements then the
380+-- array may hold less than N elements.
381+--
382+-- >>> fromStreamN n = Stream.fold (Array.writeN n)
383+--
384+-- /Pre-release/
385+{-# INLINE_NORMAL fromStreamN #-}
386+fromStreamN :: (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
387+fromStreamN n m = do
388+ when (n < 0) $ error "writeN: negative write count specified"
389+ unsafeFreeze <$> MA.fromStreamN n m
390+-- fromStreamN n = D.fold (writeN n)
391+
392+{-# DEPRECATED fromStreamDN "Please use fromStreamN instead." #-}
297393fromStreamDN :: forall m a. (MonadIO m, Unbox a)
298394 => Int -> D.Stream m a -> m (Array a)
299−fromStreamDN limit str = unsafeFreeze <$> MA.fromStreamDN limit str
395+fromStreamDN = fromStreamN
300396
397+-- | Create an 'Array' from a stream. This is useful when we want to create a
398+-- single array from a stream of unknown size. 'writeN' is at least twice
399+-- as efficient when the size is already known.
400+--
401+-- >>> fromStream = Stream.fold Array.write
402+--
403+-- Note that if the input stream is too large memory allocation for the array
404+-- may fail. When the stream size is not known, `chunksOf` followed by
405+-- processing of indvidual arrays in the resulting stream should be preferred.
406+--
407+-- /Pre-release/
301408{-# INLINE_NORMAL fromStreamD #-}
409+fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (Array a)
410+fromStream = D.fold write
411+-- fromStreamD str = unsafeFreeze <$> MA.fromStream str
412+
413+{-# DEPRECATED fromStreamD "Please use fromStream instead." #-}
302414fromStreamD :: forall m a. (MonadIO m, Unbox a)
303415 => D.Stream m a -> m (Array a)
304−fromStreamD str = unsafeFreeze <$> MA.fromStreamD str
416+fromStreamD = fromStream
417+
418+-------------------------------------------------------------------------------
419+-- Cloning
420+-------------------------------------------------------------------------------
421+
422+{-# INLINE clone #-}
423+clone ::
424+ ( MonadIO m
425+#ifdef DEVBUILD
426+ , Unbox a
427+#endif
428+ )
429+ => Array a -> m (Array a)
430+clone = fmap unsafeFreeze . MA.clone . unsafeThaw
431+
432+{-# INLINE pinnedClone #-}
433+pinnedClone ::
434+ ( MonadIO m
435+#ifdef DEVBUILD
436+ , Unbox a
437+#endif
438+ )
439+ => Array a -> m (Array a)
440+pinnedClone = fmap unsafeFreeze . MA.pinnedClone . unsafeThaw
305441
306442-------------------------------------------------------------------------------
307443-- Streams of arrays
308444-------------------------------------------------------------------------------
309445
310−{-# INLINE bufferChunks #-}
446+{-# INLINE buildChunks #-}
447+buildChunks :: (MonadIO m, Unbox a) =>
448+ D.Stream m a -> m (K.StreamK m (Array a))
449+buildChunks m = D.foldr K.cons K.nil $ chunksOf defaultChunkSize m
450+
451+{-# DEPRECATED bufferChunks "Please use buildChunks instead." #-}
311452bufferChunks :: (MonadIO m, Unbox a) =>
312453 D.Stream m a -> m (K.StreamK m (Array a))
313−bufferChunks m = D.foldr K.cons K.nil $ chunksOf defaultChunkSize m
454+bufferChunks = buildChunks
314455
315456-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
316457-- @n@ elements each.
@@ -331,27 +472,100 @@ pinnedChunksOf :: forall m a. (MonadIO m, Unbox a)
331472 => Int -> D.Stream m a -> D.Stream m (Array a)
332473pinnedChunksOf n str = D.map unsafeFreeze $ MA.pinnedChunksOf n str
333474
334−-- | Use the "read" unfold instead.
335−--
336−-- @flattenArrays = unfoldMany read@
475+-- | Convert a stream of arrays into a stream of their elements.
337476--
338−-- We can try this if there are any fusion issues in the unfold.
477+-- >>> concat = Stream.unfoldMany Array.reader
339478--
340−{-# INLINE_NORMAL flattenArrays #-}
479+{-# INLINE_NORMAL concat #-}
480+concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
481+-- XXX this requires MonadIO whereas the unfoldMany version does not
482+concat = MA.concatWith (pure . unsafeInlineIO) . D.map unsafeThaw
483+-- concat = D.unfoldMany reader
484+
485+{-# DEPRECATED flattenArrays "Please use \"unfoldMany reader\" instead." #-}
486+{-# INLINE flattenArrays #-}
341487flattenArrays :: forall m a. (MonadIO m, Unbox a)
342488 => D.Stream m (Array a) -> D.Stream m a
343−flattenArrays = MA.flattenArrays . D.map unsafeThaw
489+flattenArrays = concat
344490
345−-- | Use the "readRev" unfold instead.
491+-- | Convert a stream of arrays into a stream of their elements reversing the
492+-- contents of each array before flattening.
346493--
347−-- @flattenArrays = unfoldMany readRev@
494+-- >>> concatRev = Stream.unfoldMany Array.readerRev
348495--
349−-- We can try this if there are any fusion issues in the unfold.
350−--
351−{-# INLINE_NORMAL flattenArraysRev #-}
496+{-# INLINE_NORMAL concatRev #-}
497+concatRev :: forall m a. (Monad m, Unbox a)
498+ => D.Stream m (Array a) -> D.Stream m a
499+-- XXX this requires MonadIO whereas the unfoldMany version does not
500+concatRev = MA.concatRevWith (pure . unsafeInlineIO) . D.map unsafeThaw
501+-- concatRev = D.unfoldMany readerRev
502+
503+{-# DEPRECATED flattenArraysRev "Please use \"unfoldMany readerRev\" instead." #-}
504+{-# INLINE flattenArraysRev #-}
352505flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
353506 => D.Stream m (Array a) -> D.Stream m a
354−flattenArraysRev = MA.flattenArraysRev . D.map unsafeThaw
507+flattenArraysRev = concatRev
508+
509+-------------------------------------------------------------------------------
510+-- Compact
511+-------------------------------------------------------------------------------
512+
513+-- XXX Note that this thaws immutable arrays for appending, that may be
514+-- problematic if multiple users do the same thing, however, thawed immutable
515+-- arrays would have no capacity to append, therefore, a copy will be forced
516+-- anyway.
517+
518+-- | Fold @fCompactGE n@ coalesces adjacent arrays in the input stream
519+-- until the size becomes greater than or equal to n.
520+--
521+-- Generates unpinned arrays irrespective of the pinning status of input
522+-- arrays.
523+{-# INLINE_NORMAL fCompactGE #-}
524+fCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
525+fCompactGE n = fmap unsafeFreeze $ Fold.lmap unsafeThaw $ MA.fCompactGE n
526+
527+-- | PInned version of 'fCompactGE'.
528+{-# INLINE_NORMAL fPinnedCompactGE #-}
529+fPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
530+fPinnedCompactGE n =
531+ fmap unsafeFreeze $ Fold.lmap unsafeThaw $ MA.fPinnedCompactGE n
532+
533+-- | @compactGE n stream@ coalesces adjacent arrays in the @stream@ until
534+-- the size becomes greater than or equal to @n@.
535+--
536+-- >>> compactGE n = Stream.foldMany (Array.fCompactGE n)
537+--
538+-- Generates unpinned arrays irrespective of the pinning status of input
539+-- arrays.
540+{-# INLINE compactGE #-}
541+compactGE ::
542+ (MonadIO m, Unbox a)
543+ => Int -> Stream m (Array a) -> Stream m (Array a)
544+compactGE n stream =
545+ D.map unsafeFreeze $ MA.compactGE n $ D.map unsafeThaw stream
546+
547+-- | Like 'compactGE' but for transforming folds instead of stream.
548+--
549+-- >>> lCompactGE n = Fold.many (Array.fCompactGE n)
550+--
551+-- Generates unpinned arrays irrespective of the pinning status of input
552+-- arrays.
553+{-# INLINE_NORMAL lCompactGE #-}
554+lCompactGE :: (MonadIO m, Unbox a)
555+ => Int -> Fold m (Array a) () -> Fold m (Array a) ()
556+lCompactGE n fld =
557+ Fold.lmap unsafeThaw $ MA.lCompactGE n (Fold.lmap unsafeFreeze fld)
558+
559+-- | Pinned version of 'lCompactGE'.
560+{-# INLINE_NORMAL lPinnedCompactGE #-}
561+lPinnedCompactGE :: (MonadIO m, Unbox a)
562+ => Int -> Fold m (Array a) () -> Fold m (Array a) ()
563+lPinnedCompactGE n fld =
564+ Fold.lmap unsafeThaw $ MA.lPinnedCompactGE n (Fold.lmap unsafeFreeze fld)
565+
566+-------------------------------------------------------------------------------
567+-- Splitting
568+-------------------------------------------------------------------------------
355569
356570-- Drops the separator byte
357571{-# INLINE breakOn #-}
@@ -395,24 +609,68 @@ byteLength = MA.byteLength . unsafeThaw
395609length :: Unbox a => Array a -> Int
396610length arr = MA.length (unsafeThaw arr)
397611
612+{-# INLINE_NORMAL producer #-}
613+producer :: forall m a. (Monad m, Unbox a) => Producer m (Array a) a
614+producer =
615+ Producer.translate unsafeThaw unsafeFreeze
616+ $ MA.producerWith (return . unsafeInlineIO)
617+
618+-- | Unfold an array into a stream.
619+--
620+{-# INLINE_NORMAL reader #-}
621+reader :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
622+reader = Producer.simplify producer
623+
624+-- | Unfold an array into a stream, does not check the end of the array, the
625+-- user is responsible for terminating the stream within the array bounds. For
626+-- high performance application where the end condition can be determined by
627+-- a terminating fold.
628+--
629+-- Written in the hope that it may be faster than "read", however, in the case
630+-- for which this was written, "read" proves to be faster even though the core
631+-- generated with unsafeRead looks simpler.
632+--
633+-- /Pre-release/
634+--
635+{-# INLINE_NORMAL readerUnsafe #-}
636+readerUnsafe :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
637+readerUnsafe = Unfold step inject
638+ where
639+
640+ inject (Array contents start end) =
641+ return (MA.ArrayUnsafe contents end start)
642+
643+ {-# INLINE_LATE step #-}
644+ step (MA.ArrayUnsafe contents end p) = do
645+ -- unsafeInlineIO allows us to run this in Identity monad for pure
646+ -- toList/foldr case which makes them much faster due to not
647+ -- accumulating the list and fusing better with the pure consumers.
648+ --
649+ -- This should be safe as the array contents are guaranteed to be
650+ -- evaluated/written to before we peek at them.
651+ let !x = unsafeInlineIO $ peekAt p contents
652+ let !p1 = INDEX_NEXT(p,a)
653+ return $ D.Yield x (MA.ArrayUnsafe contents end p1)
654+
398655-- | Unfold an array into a stream in reverse order.
399656--
400657{-# INLINE_NORMAL readerRev #-}
401658readerRev :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
402659readerRev = Unfold.lmap unsafeThaw $ MA.readerRevWith (return . unsafeInlineIO)
403660
661+{-# DEPRECATED toStreamD "Please use 'read' instead." #-}
404662{-# INLINE_NORMAL toStreamD #-}
405663toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> D.Stream m a
406−toStreamD arr = MA.toStreamDWith (return . unsafeInlineIO) (unsafeThaw arr)
664+toStreamD = read
407665
408666{-# INLINE toStreamK #-}
409667toStreamK :: forall m a. (Monad m, Unbox a) => Array a -> K.StreamK m a
410668toStreamK arr = MA.toStreamKWith (return . unsafeInlineIO) (unsafeThaw arr)
411669
670+{-# DEPRECATED toStreamDRev "Please use 'readRev' instead." #-}
412671{-# INLINE_NORMAL toStreamDRev #-}
413672toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> D.Stream m a
414−toStreamDRev arr =
415− MA.toStreamDRevWith (return . unsafeInlineIO) (unsafeThaw arr)
673+toStreamDRev = readRev
416674
417675{-# INLINE toStreamKRev #-}
418676toStreamKRev :: forall m a. (Monad m, Unbox a) => Array a -> K.StreamK m a
@@ -424,7 +682,7 @@ toStreamKRev arr =
424682-- /Pre-release/
425683{-# INLINE_EARLY read #-}
426684read :: (Monad m, Unbox a) => Array a -> Stream m a
427−read = toStreamD
685+read arr = MA.toStreamWith (return . unsafeInlineIO) (unsafeThaw arr)
428686
429687-- | Same as 'read'
430688--
@@ -441,7 +699,7 @@ toStream = read
441699-- /Pre-release/
442700{-# INLINE_EARLY readRev #-}
443701readRev :: (Monad m, Unbox a) => Array a -> Stream m a
444−readRev = toStreamDRev
702+readRev arr = MA.toStreamRevWith (return . unsafeInlineIO) (unsafeThaw arr)
445703
446704-- | Same as 'readRev'
447705--
@@ -465,6 +723,7 @@ foldr f z arr = runIdentity $ D.foldr f z $ toStreamD arr
465723-- | Create two slices of an array without copying the original array. The
466724-- specified index @i@ is the first index of the second slice.
467725--
726+{-# INLINE splitAt #-}
468727splitAt :: Unbox a => Int -> Array a -> (Array a, Array a)
469728splitAt i arr = (unsafeFreeze a, unsafeFreeze b)
470729 where
@@ -484,8 +743,8 @@ toListFB c n Array{..} = go arrStart
484743 -- accumulating the list and fusing better with the pure consumers.
485744 --
486745 -- This should be safe as the array contents are guaranteed to be
487− -- evaluated/written to before we peekByteIndex at them.
488− let !x = unsafeInlineIO $ peekByteIndex p arrContents
746+ -- evaluated/written to before we peekAt at them.
747+ let !x = unsafeInlineIO $ peekAt p arrContents
489748 in c x (go (INDEX_NEXT(p,a)))
490749
491750-- | Convert an 'Array' into a list.
@@ -498,17 +757,28 @@ toList s = build (\c n -> toListFB c n s)
498757-- Folds
499758-------------------------------------------------------------------------------
500759
501−-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
760+-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
502761-- 'Array'.
503762--
504−{-# INLINE_NORMAL writeN #-}
763+{-# INLINE_NORMAL createOf #-}
764+createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
765+createOf = fmap unsafeFreeze . MA.createOf
766+
767+-- XXX Deprecate in major
768+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
769+{-# INLINE writeN #-}
505770writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
506−writeN = fmap unsafeFreeze . MA.writeN
771+writeN = createOf
507772
508−-- | Like 'fromListN' but creates a pinned array.
509−{-# INLINE_NORMAL pinnedWriteN #-}
773+-- | Like 'createOf' but creates a pinned array.
774+{-# INLINE_NORMAL pinnedCreateOf #-}
775+pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
776+pinnedCreateOf = fmap unsafeFreeze . MA.pinnedCreateOf
777+
778+{-# DEPRECATED pinnedWriteN "Please use pinnedCreateOf instead." #-}
779+{-# INLINE pinnedWriteN #-}
510780pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
511−pinnedWriteN = fmap unsafeFreeze . MA.pinnedWriteN
781+pinnedWriteN = pinnedCreateOf
512782
513783-- | @pinnedWriteNAligned alignment n@ folds a maximum of @n@ elements from the input
514784-- stream to an 'Array' aligned to the given size.
@@ -516,44 +786,74 @@ pinnedWriteN = fmap unsafeFreeze . MA.pinnedWriteN
516786-- /Pre-release/
517787--
518788{-# INLINE_NORMAL pinnedWriteNAligned #-}
789+{-# DEPRECATED pinnedWriteNAligned "To be removed." #-}
519790pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a)
520791 => Int -> Int -> Fold m a (Array a)
521792pinnedWriteNAligned alignSize = fmap unsafeFreeze . MA.pinnedWriteNAligned alignSize
522793
523−-- | Like 'writeN' but does not check the array bounds when writing. The fold
794+-- | Like 'createOf' but does not check the array bounds when writing. The fold
524795-- driver must not call the step function more than 'n' times otherwise it will
525796-- corrupt the memory and crash. This function exists mainly because any
526797-- conditional in the step function blocks fusion causing 10x performance
527798-- slowdown.
528799--
529−{-# INLINE_NORMAL writeNUnsafe #-}
800+{-# INLINE_NORMAL unsafeCreateOf #-}
801+unsafeCreateOf :: forall m a. (MonadIO m, Unbox a)
802+ => Int -> Fold m a (Array a)
803+unsafeCreateOf n = unsafeFreeze <$> MA.unsafeCreateOf n
804+
805+{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
806+{-# INLINE writeNUnsafe #-}
530807writeNUnsafe :: forall m a. (MonadIO m, Unbox a)
531808 => Int -> Fold m a (Array a)
532−writeNUnsafe n = unsafeFreeze <$> MA.writeNUnsafe n
809+writeNUnsafe = unsafeCreateOf
533810
534−{-# INLINE_NORMAL pinnedWriteNUnsafe #-}
811+{-# INLINE_NORMAL unsafePinnedCreateOf #-}
812+unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a)
813+ => Int -> Fold m a (Array a)
814+unsafePinnedCreateOf n = unsafeFreeze <$> MA.unsafePinnedCreateOf n
815+
816+{-# DEPRECATED pinnedWriteNUnsafe "Please use unsafePinnedCreateOf instead." #-}
817+{-# INLINE pinnedWriteNUnsafe #-}
535818pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a)
536819 => Int -> Fold m a (Array a)
537−pinnedWriteNUnsafe n = unsafeFreeze <$> MA.pinnedWriteNUnsafe n
820+pinnedWriteNUnsafe = unsafePinnedCreateOf
538821
539−{-# INLINE_NORMAL writeWith #-}
822+{-# INLINE_NORMAL createWith #-}
823+createWith :: forall m a. (MonadIO m, Unbox a)
824+ => Int -> Fold m a (Array a)
825+-- createWith n = FL.rmapM spliceArrays $ toArraysOf n
826+createWith elemCount = unsafeFreeze <$> MA.createWith elemCount
827+
828+{-# DEPRECATED writeWith "Please use createWith instead." #-}
829+{-# INLINE writeWith #-}
540830writeWith :: forall m a. (MonadIO m, Unbox a)
541831 => Int -> Fold m a (Array a)
542−-- writeWith n = FL.rmapM spliceArrays $ toArraysOf n
543−writeWith elemCount = unsafeFreeze <$> MA.writeWith elemCount
832+writeWith = createWith
544833
545834-- | Fold the whole input to a single array.
546835--
547836-- /Caution! Do not use this on infinite streams./
548837--
838+{-# INLINE create #-}
839+create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
840+create = fmap unsafeFreeze MA.create
841+
842+-- XXX Deprecate in major
843+-- {-# DEPRECATED write "Please use create instead." #-}
549844{-# INLINE write #-}
550845write :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
551−write = fmap unsafeFreeze MA.write
846+write = create
552847
553−-- | Like 'write' but creates a pinned array.
848+-- | Like 'create' but creates a pinned array.
849+{-# INLINE pinnedCreate #-}
850+pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
851+pinnedCreate = fmap unsafeFreeze MA.pinnedCreate
852+
853+{-# DEPRECATED pinnedWrite "Please use pinnedCreate instead." #-}
554854{-# INLINE pinnedWrite #-}
555855pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
556−pinnedWrite = fmap unsafeFreeze MA.pinnedWrite
856+pinnedWrite = pinnedCreate
557857
558858-- | Fold "step" has a dependency on "initial", and each step is dependent on
559859-- the previous invocation of step due to state passing, finally extract
@@ -579,10 +879,16 @@ pinnedWrite = fmap unsafeFreeze MA.pinnedWrite
579879--
580880{-# INLINE unsafeMakePure #-}
581881unsafeMakePure :: Monad m => Fold IO a b -> Fold m a b
582−unsafeMakePure (Fold step initial extract) =
882+unsafeMakePure (Fold step initial extract final) =
583883 Fold (\x a -> return $! unsafeInlineIO (step x a))
584884 (return $! unsafePerformIO initial)
585885 (\s -> return $! unsafeInlineIO $ extract s)
886+ (\s -> return $! unsafeInlineIO $ final s)
887+
888+{-# INLINE fromPureStreamN #-}
889+fromPureStreamN :: Unbox a => Int -> Stream Identity a -> Array a
890+fromPureStreamN n x =
891+ unsafePerformIO $ fmap unsafeFreeze (MA.fromPureStreamN n x)
586892
587893-- | Convert a pure stream in Identity monad to an immutable array.
588894--
@@ -595,6 +901,19 @@ fromPureStream x = unsafePerformIO $ fmap (unsafeFreeze) (MA.fromPureStream x)
595901-- fromPureStream = runIdentity . D.fold (unsafeMakePure write)
596902-- fromPureStream = fromList . runIdentity . D.toList
597903
904+-- XXX This should be monadic.
905+
906+-- | Copy an immutable 'Ptr Word8' sequence into an array.
907+--
908+-- /Unsafe:/ The caller is responsible for safe addressing.
909+--
910+-- Note that this should be evaluated strictly to ensure that we do not hold
911+-- the reference to the pointer in a lazy thunk.
912+fromPtrN :: Int -> Ptr Word8 -> Array Word8
913+fromPtrN n addr = unsafePerformIO $ fmap unsafeFreeze (MA.fromPtrN n addr)
914+
915+-- XXX This should be monadic.
916+
598917-- | Copy a null terminated immutable 'Addr#' Word8 sequence into an array.
599918--
600919-- /Unsafe:/ The caller is responsible for safe addressing.
@@ -605,8 +924,46 @@ fromPureStream x = unsafePerformIO $ fmap (unsafeFreeze) (MA.fromPureStream x)
605924-- >>> Array.toList $ Array.fromByteStr# "\1\2\3\0"#
606925-- [1,2,3]
607926--
927+-- Note that this should be evaluated strictly to ensure that we do not hold
928+-- the reference to the pointer in a lazy thunk.
608929fromByteStr# :: Addr# -> Array Word8
609−fromByteStr# addr = fromPureStream (D.fromByteStr# addr)
930+fromByteStr# addr = unsafePerformIO $ fmap unsafeFreeze (MA.fromByteStr# addr)
931+
932+-- XXX This should be monadic.
933+
934+-- | Note that this should be evaluated strictly to ensure that we do not hold
935+-- the reference to the pointer in a lazy thunk.
936+fromByteStr :: Ptr Word8 -> Array Word8
937+fromByteStr (Ptr addr#) = fromByteStr# addr#
938+
939+-- XXX implement fromChunks/fromChunkList instead?
940+
941+-- | Convert an array stream to an array. Note that this requires peak memory
942+-- that is double the size of the array stream.
943+--
944+{-# INLINE fromChunksK #-}
945+fromChunksK :: (MonadIO m, Unbox a) => StreamK m (Array a) -> m (Array a)
946+fromChunksK stream =
947+ -- We buffer the entire stream and then allocate the target array of the
948+ -- same size, thus requiring double the memory.
949+ fmap unsafeFreeze $ MA.fromChunksK $ fmap unsafeThaw stream
950+
951+{-# DEPRECATED fromArrayStreamK "Please use fromChunksK instead." #-}
952+fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (Array a) -> m (Array a)
953+fromArrayStreamK = fromChunksK
954+
955+-- | Given a stream of arrays, splice them all together to generate a single
956+-- array. The stream must be /finite/.
957+--
958+{-# INLINE fromChunks #-}
959+fromChunks :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
960+fromChunks s =
961+ -- XXX Check which implementation is better
962+ -- This may also require double the memory as we double the space every
963+ -- time, when copying the last array we may have reallocated almost double
964+ -- the space required before we right size it.
965+ fmap unsafeFreeze $ MA.fromChunksRealloced (fmap unsafeThaw s)
966+ -- fromChunkStreamK $ D.toStreamK s
610967
611968-------------------------------------------------------------------------------
612969-- Instances
@@ -638,12 +995,66 @@ instance Unbox a => IsList (Array a) where
638995 {-# INLINE toList #-}
639996 toList = toList
640997
641−-- XXX we are assuming that Unboxed equality means element equality. This may
642−-- or may not be correct? arrcmp is 40% faster compared to stream equality.
643−instance (Unbox a, Eq a) => Eq (Array a) where
998+-- | Byte compare two arrays. Compare the length of the arrays. If the length
999+-- is equal, compare the lexicographical ordering of two underlying byte arrays
1000+-- otherwise return the result of length comparison.
1001+--
1002+-- /Unsafe/: Note that the 'Unbox' instance of sum types with constructors of
1003+-- different sizes may leave some memory uninitialized which can make byte
1004+-- comparison unreliable.
1005+--
1006+-- /Pre-release/
1007+{-# INLINE byteCmp #-}
1008+byteCmp :: Array a -> Array a -> Ordering
1009+byteCmp arr1 arr2 =
1010+ -- unsafePerformIO?
1011+ unsafeInlineIO $! unsafeThaw arr1 `MA.byteCmp` unsafeThaw arr2
1012+
1013+-- | Byte equality of two arrays.
1014+--
1015+-- >>> byteEq arr1 arr2 = (==) EQ $ Array.byteCmp arr1 arr2
1016+--
1017+-- /Unsafe/: See 'byteCmp'.
1018+{-# INLINE byteEq #-}
1019+byteEq :: Array a -> Array a -> Bool
1020+byteEq arr1 arr2 = (==) EQ $ byteCmp arr1 arr2
1021+
1022+#define MK_EQ_INSTANCE(typ) \
1023+instance {-# OVERLAPPING #-} Eq (Array typ) where { \
1024+; {-# INLINE (==) #-} \
1025+; (==) = byteEq \
1026+}
1027+
1028+MK_EQ_INSTANCE(Char)
1029+MK_EQ_INSTANCE(Word8)
1030+MK_EQ_INSTANCE(Word16)
1031+MK_EQ_INSTANCE(Word32)
1032+
1033+-- XXX The Word64 default instance should be as fast because we are comparing
1034+-- 64-bit at a time.
1035+MK_EQ_INSTANCE(Word64)
1036+MK_EQ_INSTANCE(Int)
1037+MK_EQ_INSTANCE(Int8)
1038+MK_EQ_INSTANCE(Int16)
1039+MK_EQ_INSTANCE(Int32)
1040+
1041+-- XXX The Int64 default instance should be as fast.
1042+MK_EQ_INSTANCE(Int64)
1043+
1044+-- | If the type allows a byte-by-byte comparison this instance can be
1045+-- overlapped by a more specific instance that uses 'byteCmp'. Byte comparison
1046+-- can be significantly faster.
1047+--
1048+instance {-# OVERLAPPABLE #-} (Unbox a, Eq a) => Eq (Array a) where
6441049 {-# INLINE (==) #-}
6451050 arr1 == arr2 =
646− (==) EQ $ unsafeInlineIO $! unsafeThaw arr1 `MA.cmp` unsafeThaw arr2
1051+ -- Does unboxed byte equality mean element equality?
1052+ -- XXX This is incorrect for sum types, as we may have some
1053+ -- uninitialized memory in that case. If we always initialize the
1054+ -- unused memory to zero we can use this.
1055+ -- Byte comparison is 40% faster compared to stream equality.
1056+ -- (==) EQ $ unsafeInlineIO $! unsafeThaw arr1 `MA.cmp` unsafeThaw arr2
1057+ (toStreamD arr1 :: Stream Identity a) == toStreamD arr2
6471058
6481059instance (Unbox a, Ord a) => Ord (Array a) where
6491060 {-# INLINE compare #-}
@@ -685,7 +1096,7 @@ _toStreamD_ size Array{..} = D.Stream step arrStart
6851096 {-# INLINE_LATE step #-}
6861097 step _ p | p == arrEnd = return D.Stop
6871098 step _ p = liftIO $ do
688− x <- peekByteIndex p arrContents
1099+ x <- peekAt p arrContents
6891100 return $ D.Yield x (p + size)
6901101
6911102{-
@@ -708,15 +1119,29 @@ instance Foldable Array where
7081119-- Semigroup and Monoid
7091120-------------------------------------------------------------------------------
7101121
1122+-- XXX Deprecate and remove the Semigroup and Monoid instances because of
1123+-- potential misuse chances.
1124+
1125+-- | This should not be used for combining many or N arrays as it would copy
1126+-- the two arrays everytime to a new array. For coalescing multiple arrays use
1127+-- 'fromChunksK' instead.
7111128instance Unbox a => Semigroup (Array a) where
7121129 arr1 <> arr2 = unsafePerformIO $ splice arr1 arr2
7131130
1131+empty ::
1132+#ifdef DEVBUILD
1133+ Unbox a =>
1134+#endif
1135+ Array a
1136+empty = Array Unboxed.empty 0 0
1137+
1138+{-# DEPRECATED nil "Please use empty instead." #-}
7141139nil ::
7151140#ifdef DEVBUILD
7161141 Unbox a =>
7171142#endif
7181143 Array a
719−nil = Array Unboxed.nil 0 0
1144+nil = empty
7201145
7211146instance Unbox a => Monoid (Array a) where
7221147 mempty = nil
core/src/Streamly/Internal/Serialize/FromBytes.hs → core/src/Streamly/Internal/Data/Binary/Parser.hsrenamed+8−2
@@ -1,13 +1,19 @@
11-- |
2−-- Module : Streamly.Internal.Serialize.FromBytes
2+-- Module : Streamly.Internal.Data.Binary.Parser
33-- Copyright : (c) 2020 Composewell Technologies
44-- License : BSD-3-Clause
55-- Maintainer : streamly@composewell.com
66-- Portability : GHC
77--
88-- Decode Haskell data types from byte streams.
9+--
10+-- It would be inefficient to use this to compose parsers for general algebraic
11+-- data types. For general deserialization of ADTs please use the Serialize
12+-- type class instances. The fastest way to deserialize byte streams
13+-- representing Haskell data types is to write them to arrays and deserialize
14+-- the array using the Serialize type class.
915
10−module Streamly.Internal.Serialize.FromBytes
16+module Streamly.Internal.Data.Binary.Parser
1117 (
1218 -- * Type class
1319 FromBytes (..)
core/src/Streamly/Internal/Serialize/ToBytes.hs → core/src/Streamly/Internal/Data/Binary/Stream.hsrenamed+2−2
@@ -1,5 +1,5 @@
11-- |
2−-- Module : Streamly.Internal.Serialize.ToBytes
2+-- Module : Streamly.Internal.Data.Binary.Stream
33-- Copyright : (c) 2022 Composewell Technologies
44-- License : BSD-3-Clause
55-- Maintainer : streamly@composewell.com
@@ -17,7 +17,7 @@
1717
1818-- XXX remove unit, bool, ordering, and the type class as well
1919
20−module Streamly.Internal.Serialize.ToBytes
20+module Streamly.Internal.Data.Binary.Stream
2121 (
2222 -- * Type class
2323 ToBytes (..)
core/src/Streamly/Internal/Data/Builder.hs+11−9
@@ -16,7 +16,9 @@ module Streamly.Internal.Data.Builder
1616 )
1717where
1818
19+#if !MIN_VERSION_base(4,18,0)
1920import Control.Applicative (liftA2)
21+#endif
2022
2123------------------------------------------------------------------------------
2224-- The Builder type
@@ -27,16 +29,16 @@ import Control.Applicative (liftA2)
2729-- or even a Fold. Unlike fold the step function is one-shot and not called in
2830-- a loop.
2931newtype Builder s m a =
30− Builder (s -> m (s, a))
32+ Builder (s -> m (a, s))
3133
3234-- | Maps a function on the output of the fold (the type @b@).
3335instance Functor m => Functor (Builder s m) where
3436 {-# INLINE fmap #-}
35− fmap f (Builder step1) = Builder (fmap (fmap f) . step1)
37+ fmap f (Builder step1) = Builder (fmap (\ (a, s) -> (f a, s)) . step1)
3638
3739{-# INLINE fromPure #-}
3840fromPure :: Applicative m => b -> Builder s m b
39−fromPure b = Builder (\s -> pure (s, b))
41+fromPure b = Builder (\s -> pure (b, s))
4042
4143-- | Chain the actions and zip the outputs.
4244{-# INLINE sequenceWith #-}
@@ -47,9 +49,9 @@ sequenceWith func (Builder stepL) (Builder stepR) = Builder step
4749 where
4850
4951 step s = do
50− (s1, x) <- stepL s
51− (s2, y) <- stepR s1
52− pure (s2, func x y)
52+ (x, s1) <- stepL s
53+ (y, s2) <- stepR s1
54+ pure (func x y, s2)
5355
5456instance Monad m => Applicative (Builder a m) where
5557 {-# INLINE pure #-}
@@ -74,7 +76,7 @@ instance Monad m => Monad (Builder a m) where
7476 where
7577
7678 step s = do
77− (s1, x) <- stepL s
79+ (x, s1) <- stepL s
7880 let Builder stepR = f x
79− (s2, y) <- stepR s1
80− pure (s2, y)
81+ (y, s2) <- stepR s1
82+ pure (y, s2)
core/src/Streamly/Internal/Data/Fold/Chunked.hs+18−15
@@ -6,7 +6,7 @@
66-- Stability : experimental
77-- Portability : GHC
88--
9−-- Use "Streamly.Data.Parser.Chunked" instead.
9+-- Use "Streamly.Data.Parser" instead.
1010--
1111-- Fold a stream of foreign arrays. @Fold m a b@ in this module works
1212-- on a stream of "Array a" and produces an output of type @b@.
@@ -19,18 +19,19 @@
1919-- folds in Data.Fold to correctly work on an array stream as if it is an
2020-- element stream. For example:
2121--
22−-- >>> import qualified Streamly.Data.Fold as Fold
23−-- >>> import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
24−-- >>> import qualified Streamly.Internal.Data.Fold.Chunked as ChunkFold
25−-- >>> import qualified Streamly.Data.Stream as Stream
26−-- >>> import qualified Streamly.Data.StreamK as StreamK
22+-- >> import qualified Streamly.Data.Fold as Fold
23+-- >> import qualified Streamly.Internal.Data.Array.Stream as ArrayStream
24+-- >> import qualified Streamly.Internal.Data.Fold.Chunked as ChunkFold
25+-- >> import qualified Streamly.Data.Stream as Stream
26+-- >> import qualified Streamly.Data.StreamK as StreamK
2727--
28−-- >>> f = ChunkFold.fromFold (Fold.take 7 Fold.toList)
29−-- >>> s = Stream.chunksOf 5 $ Stream.fromList "hello world"
30−-- >>> ArrayStream.runArrayFold f (StreamK.fromStream s)
28+-- >> f = ChunkFold.fromFold (Fold.take 7 Fold.toList)
29+-- >> s = Stream.chunksOf 5 $ Stream.fromList "hello world"
30+-- >> ArrayStream.runArrayFold f (StreamK.fromStream s)
3131-- Right "hello w"
3232--
3333module Streamly.Internal.Data.Fold.Chunked
34+ {-# DEPRECATED "Please use Streamly.Data.Parser instead." #-}
3435 (
3536 ChunkFold (..)
3637
@@ -58,14 +59,15 @@ where
5859
5960#include "ArrayMacros.h"
6061
62+#if !MIN_VERSION_base(4,18,0)
6163import Control.Applicative (liftA2)
64+#endif
6265import Control.Exception (assert)
6366import Control.Monad.IO.Class (MonadIO(..))
6467import Data.Bifunctor (first)
6568import Data.Proxy (Proxy(..))
6669import Streamly.Internal.Data.Unbox (Unbox(..))
6770import GHC.Types (SPEC(..))
68−import Streamly.Internal.Data.MutArray.Type (touch)
6971import Streamly.Internal.Data.Array (Array(..))
7072import Streamly.Internal.Data.Parser (Initial(..), Step(..))
7173import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
@@ -101,8 +103,8 @@ newtype ChunkFold m a b = ChunkFold (ParserD.Parser (Array a) m b)
101103{-# INLINE fromFold #-}
102104fromFold :: forall m a b. (MonadIO m, Unbox a) =>
103105 Fold.Fold m a b -> ChunkFold m a b
104−fromFold (Fold.Fold fstep finitial fextract) =
105− ChunkFold (ParserD.Parser step initial (fmap (Done 0) . fextract))
106+fromFold (Fold.Fold fstep finitial _ ffinal) =
107+ ChunkFold (ParserD.Parser step initial extract)
106108
107109 where
108110
@@ -122,7 +124,7 @@ fromFold (Fold.Fold fstep finitial fextract) =
122124 assert (cur == end) (return ())
123125 return $ Partial 0 fs
124126 goArray !_ !cur !fs = do
125− x <- liftIO $ peekByteIndex cur contents
127+ x <- liftIO $ peekAt cur contents
126128 res <- fstep fs x
127129 let elemSize = SIZE_OF(a)
128130 next = INDEX_NEXT(cur,a)
@@ -132,6 +134,8 @@ fromFold (Fold.Fold fstep finitial fextract) =
132134 Fold.Partial fs1 ->
133135 goArray SPEC next fs1
134136
137+ extract = fmap (Done 0) . ffinal
138+
135139-- | Convert an element 'ParserD.Parser' into an array stream fold. If the
136140-- parser fails the fold would throw an exception.
137141--
@@ -159,8 +163,7 @@ fromParserD (ParserD.Parser step1 initial1 extract1) =
159163 else return $ st (arrRem + n) fs1
160164
161165 goArray !_ !cur !fs = do
162− x <- liftIO $ peekByteIndex cur contents
163− liftIO $ touch contents
166+ x <- liftIO $ peekAt cur contents
164167 res <- step1 fs x
165168 let elemSize = SIZE_OF(a)
166169 next = INDEX_NEXT(cur,a)
core/src/Streamly/Internal/Data/Fold/Combinators.hs+81−137
@@ -30,8 +30,6 @@ module Streamly.Internal.Data.Fold.Combinators
3030 -- *** Reducers
3131 , drainMapM
3232 , the
33− , length
34− , lengthGeneric
3533 , mean
3634 , rollingHash
3735 , defaultSalt
@@ -166,8 +164,6 @@ module Streamly.Internal.Data.Fold.Combinators
166164 -- ** Trimming
167165
168166 -- By elements
169− , takeEndBy
170− , takeEndBy_
171167 , takeEndBySeq
172168 , takeEndBySeq_
173169 {-
@@ -443,8 +439,8 @@ trace f = lmapM (tracing f)
443439-- /Pre-release/
444440{-# INLINE transform #-}
445441transform :: Monad m => Pipe m a b -> Fold m b c -> Fold m a c
446−transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract) =
447− Fold step initial extract
442+transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract ffinal) =
443+ Fold step initial extract final
448444
449445 where
450446
@@ -477,10 +473,14 @@ transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract) =
477473
478474 extract (Tuple' _ fs) = fextract fs
479475
476+ final (Tuple' _ fs) = ffinal fs
477+
480478{-# INLINE scanWith #-}
481479scanWith :: Monad m => Bool -> Fold m a b -> Fold m b c -> Fold m a c
482−scanWith isMany (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
483− Fold step initial extract
480+scanWith isMany
481+ (Fold stepL initialL extractL finalL)
482+ (Fold stepR initialR extractR finalR) =
483+ Fold step initial extract final
484484
485485 where
486486
@@ -494,15 +494,14 @@ scanWith isMany (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
494494 Partial sR1 ->
495495 if isMany
496496 then runStep initialL sR1
497− else Done <$> extractR sR1
497+ else Done <$> finalR sR1
498498 Done bR -> return $ Done bR
499499 Partial sL -> do
500500 !b <- extractL sL
501501 rR <- stepR sR b
502− return
503− $ case rR of
504− Partial sR1 -> Partial (sL, sR1)
505− Done bR -> Done bR
502+ case rR of
503+ Partial sR1 -> return $ Partial (sL, sR1)
504+ Done bR -> finalL sL >> return (Done bR)
506505
507506 initial = do
508507 r <- initialR
@@ -514,6 +513,8 @@ scanWith isMany (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
514513
515514 extract = extractR . snd
516515
516+ final (sL, sR) = finalL sL *> finalR sR
517+
517518-- | Scan the input of a 'Fold' to change it in a stateful manner using another
518519-- 'Fold'. The scan stops as soon as the fold terminates.
519520--
@@ -565,7 +566,7 @@ deleteBy eq x0 = fmap extract $ foldl' step (Tuple' False Nothing)
565566--
566567{-# INLINE slide2 #-}
567568slide2 :: Monad m => Fold m (a, Maybe a) b -> Fold m a b
568−slide2 (Fold step1 initial1 extract1) = Fold step initial extract
569+slide2 (Fold step1 initial1 extract1 final1) = Fold step initial extract final
569570
570571 where
571572
@@ -577,6 +578,8 @@ slide2 (Fold step1 initial1 extract1) = Fold step initial extract
577578
578579 extract (Tuple' _ s) = extract1 s
579580
581+ final (Tuple' _ s) = final1 s
582+
580583-- | Return the latest unique element using the supplied comparison function.
581584-- Returns 'Nothing' if the current element is same as the last element
582585-- otherwise returns 'Just'.
@@ -702,30 +705,6 @@ the = foldt' step initial id
702705-- To Summary
703706------------------------------------------------------------------------------
704707
705−-- | Like 'length', except with a more general 'Num' return value
706−--
707−-- Definition:
708−--
709−-- >>> lengthGeneric = fmap getSum $ Fold.foldMap (Sum . const 1)
710−-- >>> lengthGeneric = Fold.foldl' (\n _ -> n + 1) 0
711−--
712−-- /Pre-release/
713−{-# INLINE lengthGeneric #-}
714−lengthGeneric :: (Monad m, Num b) => Fold m a b
715−lengthGeneric = foldl' (\n _ -> n + 1) 0
716−
717−-- | Determine the length of the input stream.
718−--
719−-- Definition:
720−--
721−-- >>> length = Fold.lengthGeneric
722−-- >>> length = fmap getSum $ Fold.foldMap (Sum . const 1)
723−--
724−{-# INLINE length #-}
725−length :: Monad m => Fold m a Int
726−length = lengthGeneric
727−
728−
729708-- | Determine the sum of all elements of a stream of numbers. Returns additive
730709-- identity (@0@) when the stream is empty. Note that this is not numerically
731710-- stable for floating point numbers.
@@ -932,7 +911,7 @@ rollingHashFirstN n = take n rollingHash
932911--
933912{-# INLINE rollingMapM #-}
934913rollingMapM :: Monad m => (Maybe a -> a -> m b) -> Fold m a b
935−rollingMapM f = Fold step initial extract
914+rollingMapM f = Fold step initial extract extract
936915
937916 where
938917
@@ -1173,7 +1152,8 @@ head = one
11731152-- /Pre-release/
11741153{-# INLINE findM #-}
11751154findM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1176−findM predicate = Fold step (return $ Partial ()) (const $ return Nothing)
1155+findM predicate =
1156+ Fold step (return $ Partial ()) extract extract
11771157
11781158 where
11791159
@@ -1184,6 +1164,8 @@ findM predicate = Fold step (return $ Partial ()) (const $ return Nothing)
11841164 else Partial ()
11851165 in f <$> predicate a
11861166
1167+ extract = const $ return Nothing
1168+
11871169-- | Returns the first element that satisfies the given predicate.
11881170--
11891171{-# INLINE find #-}
@@ -1426,7 +1408,7 @@ splitAt n fld = splitWith (,) (take n fld)
14261408
14271409{-# INLINE takingEndByM #-}
14281410takingEndByM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1429−takingEndByM p = Fold step initial (return . toMaybe)
1411+takingEndByM p = Fold step initial extract extract
14301412
14311413 where
14321414
@@ -1439,6 +1421,8 @@ takingEndByM p = Fold step initial (return . toMaybe)
14391421 then Done $ Just a
14401422 else Partial $ Just' a
14411423
1424+ extract = return . toMaybe
1425+
14421426-- |
14431427--
14441428-- >>> takingEndBy p = Fold.takingEndByM (return . p)
@@ -1449,7 +1433,7 @@ takingEndBy p = takingEndByM (return . p)
14491433
14501434{-# INLINE takingEndByM_ #-}
14511435takingEndByM_ :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1452−takingEndByM_ p = Fold step initial (return . toMaybe)
1436+takingEndByM_ p = Fold step initial extract extract
14531437
14541438 where
14551439
@@ -1462,6 +1446,8 @@ takingEndByM_ p = Fold step initial (return . toMaybe)
14621446 then Done Nothing
14631447 else Partial $ Just' a
14641448
1449+ extract = return . toMaybe
1450+
14651451-- |
14661452--
14671453-- >>> takingEndBy_ p = Fold.takingEndByM_ (return . p)
@@ -1472,7 +1458,7 @@ takingEndBy_ p = takingEndByM_ (return . p)
14721458
14731459{-# INLINE droppingWhileM #-}
14741460droppingWhileM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1475−droppingWhileM p = Fold step initial (return . toMaybe)
1461+droppingWhileM p = Fold step initial extract extract
14761462
14771463 where
14781464
@@ -1487,6 +1473,8 @@ droppingWhileM p = Fold step initial (return . toMaybe)
14871473 else Just' a
14881474 step _ a = return $ Partial $ Just' a
14891475
1476+ extract = return . toMaybe
1477+
14901478-- |
14911479-- >>> droppingWhile p = Fold.droppingWhileM (return . p)
14921480--
@@ -1494,70 +1482,6 @@ droppingWhileM p = Fold step initial (return . toMaybe)
14941482droppingWhile :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
14951483droppingWhile p = droppingWhileM (return . p)
14961484
1497−-- Note: Keep this consistent with S.splitOn. In fact we should eliminate
1498−-- S.splitOn in favor of the fold.
1499−--
1500−-- XXX Use Fold.many instead once it is fixed.
1501−-- > Stream.splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
1502−
1503−-- | Like 'takeEndBy' but drops the element on which the predicate succeeds.
1504−--
1505−-- Example:
1506−--
1507−-- >>> input = Stream.fromList "hello\nthere\n"
1508−-- >>> line = Fold.takeEndBy_ (== '\n') Fold.toList
1509−-- >>> Stream.fold line input
1510−-- "hello"
1511−--
1512−-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1513−-- ["hello","there"]
1514−--
1515−{-# INLINE takeEndBy_ #-}
1516−takeEndBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1517−-- takeEndBy_ predicate = scanMaybe (takingEndBy_ predicate)
1518−takeEndBy_ predicate (Fold fstep finitial fextract) =
1519− Fold step finitial fextract
1520−
1521− where
1522−
1523− step s a =
1524− if not (predicate a)
1525− then fstep s a
1526− else Done <$> fextract s
1527−
1528−-- Note:
1529−-- > Stream.splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
1530−
1531−-- | Take the input, stop when the predicate succeeds taking the succeeding
1532−-- element as well.
1533−--
1534−-- Example:
1535−--
1536−-- >>> input = Stream.fromList "hello\nthere\n"
1537−-- >>> line = Fold.takeEndBy (== '\n') Fold.toList
1538−-- >>> Stream.fold line input
1539−-- "hello\n"
1540−--
1541−-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1542−-- ["hello\n","there\n"]
1543−--
1544−{-# INLINE takeEndBy #-}
1545−takeEndBy :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1546−-- takeEndBy predicate = scanMaybe (takingEndBy predicate)
1547−takeEndBy predicate (Fold fstep finitial fextract) =
1548− Fold step finitial fextract
1549−
1550− where
1551−
1552− step s a = do
1553− res <- fstep s a
1554− if not (predicate a)
1555− then return res
1556− else do
1557− case res of
1558− Partial s1 -> Done <$> fextract s1
1559− Done b -> return $ Done b
1560−
15611485------------------------------------------------------------------------------
15621486-- Binary splitting on a separator
15631487------------------------------------------------------------------------------
@@ -1590,8 +1514,8 @@ takeEndBySeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) =>
15901514 Array.Array a
15911515 -> Fold m a b
15921516 -> Fold m a b
1593−takeEndBySeq patArr (Fold fstep finitial fextract) =
1594− Fold step initial extract
1517+takeEndBySeq patArr (Fold fstep finitial fextract ffinal) =
1518+ Fold step initial extract final
15951519
15961520 where
15971521
@@ -1604,7 +1528,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16041528 | patLen == 0 ->
16051529 -- XXX Should we match nothing or everything on empty
16061530 -- pattern?
1607− -- Done <$> fextract acc
1531+ -- Done <$> ffinal acc
16081532 return $ Partial $ SplitOnSeqEmpty acc
16091533 | patLen == 1 -> do
16101534 pat <- liftIO $ Array.unsafeIndexIO 0 patArr
@@ -1655,7 +1579,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16551579 case res of
16561580 Partial s1
16571581 | pat /= x -> return $ Partial $ SplitOnSeqSingle s1 pat
1658− | otherwise -> Done <$> fextract s1
1582+ | otherwise -> Done <$> ffinal s1
16591583 Done b -> return $ Done b
16601584 step (SplitOnSeqWord s idx wrd) x = do
16611585 res <- fstep s x
@@ -1664,7 +1588,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16641588 Partial s1
16651589 | idx == maxIndex -> do
16661590 if wrd1 .&. wordMask == wordPat
1667− then Done <$> fextract s1
1591+ then Done <$> ffinal s1
16681592 else return $ Partial $ SplitOnSeqWordLoop s1 wrd1
16691593 | otherwise ->
16701594 return $ Partial $ SplitOnSeqWord s1 (idx + 1) wrd1
@@ -1675,7 +1599,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16751599 case res of
16761600 Partial s1
16771601 | wrd1 .&. wordMask == wordPat ->
1678− Done <$> fextract s1
1602+ Done <$> ffinal s1
16791603 | otherwise ->
16801604 return $ Partial $ SplitOnSeqWordLoop s1 wrd1
16811605 Done b -> return $ Done b
@@ -1689,7 +1613,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16891613 let fld = Ring.unsafeFoldRing (Ring.ringBound rb)
16901614 let !ringHash = fld addCksum 0 rb
16911615 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1692− then Done <$> fextract s1
1616+ then Done <$> ffinal s1
16931617 else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
16941618 else
16951619 return $ Partial $ SplitOnSeqKR s1 (idx + 1) rb rh1
@@ -1702,11 +1626,11 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
17021626 rh1 <- liftIO $ Ring.unsafeInsert rb rh x
17031627 let ringHash = deltaCksum cksum old x
17041628 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1705− then Done <$> fextract s1
1629+ then Done <$> ffinal s1
17061630 else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
17071631 Done b -> return $ Done b
17081632
1709− extract state =
1633+ extractFunc fex state =
17101634 let st =
17111635 case state of
17121636 SplitOnSeqEmpty s -> s
@@ -1715,7 +1639,11 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
17151639 SplitOnSeqWordLoop s _ -> s
17161640 SplitOnSeqKR s _ _ _ -> s
17171641 SplitOnSeqKRLoop s _ _ _ -> s
1718− in fextract st
1642+ in fex st
1643+
1644+ extract state = extractFunc fextract state
1645+
1646+ final state = extractFunc ffinal state
17191647
17201648-- | Like 'takeEndBySeq' but discards the matched sequence.
17211649--
@@ -1726,8 +1654,8 @@ takeEndBySeq_ :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) =>
17261654 Array.Array a
17271655 -> Fold m a b
17281656 -> Fold m a b
1729−takeEndBySeq_ patArr (Fold fstep finitial fextract) =
1730− Fold step initial extract
1657+takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
1658+ Fold step initial extract final
17311659
17321660 where
17331661
@@ -1740,7 +1668,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
17401668 | patLen == 0 ->
17411669 -- XXX Should we match nothing or everything on empty
17421670 -- pattern?
1743− -- Done <$> fextract acc
1671+ -- Done <$> ffinal acc
17441672 return $ Partial $ SplitOnSeqEmpty acc
17451673 | patLen == 1 -> do
17461674 pat <- liftIO $ Array.unsafeIndexIO 0 patArr
@@ -1797,13 +1725,13 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
17971725 case res of
17981726 Partial s1 -> return $ Partial $ SplitOnSeqSingle s1 pat
17991727 Done b -> return $ Done b
1800− else Done <$> fextract s
1728+ else Done <$> ffinal s
18011729 step (SplitOnSeqWord s idx wrd) x = do
18021730 let wrd1 = addToWord wrd x
18031731 if idx == maxIndex
18041732 then do
18051733 if wrd1 .&. wordMask == wordPat
1806− then Done <$> fextract s
1734+ then Done <$> ffinal s
18071735 else return $ Partial $ SplitOnSeqWordLoop s wrd1
18081736 else return $ Partial $ SplitOnSeqWord s (idx + 1) wrd1
18091737 step (SplitOnSeqWordLoop s wrd) x = do
@@ -1814,7 +1742,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18141742 case res of
18151743 Partial s1
18161744 | wrd1 .&. wordMask == wordPat ->
1817− Done <$> fextract s1
1745+ Done <$> ffinal s1
18181746 | otherwise ->
18191747 return $ Partial $ SplitOnSeqWordLoop s1 wrd1
18201748 Done b -> return $ Done b
@@ -1825,7 +1753,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18251753 let fld = Ring.unsafeFoldRing (Ring.ringBound rb)
18261754 let !ringHash = fld addCksum 0 rb
18271755 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1828− then Done <$> fextract s
1756+ then Done <$> ffinal s
18291757 else return $ Partial $ SplitOnSeqKRLoop s ringHash rb rh1
18301758 else return $ Partial $ SplitOnSeqKR s (idx + 1) rb rh1
18311759 step (SplitOnSeqKRLoop s cksum rb rh) x = do
@@ -1836,7 +1764,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18361764 rh1 <- liftIO $ Ring.unsafeInsert rb rh x
18371765 let ringHash = deltaCksum cksum old x
18381766 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1839− then Done <$> fextract s1
1767+ then Done <$> ffinal s1
18401768 else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
18411769 Done b -> return $ Done b
18421770
@@ -1844,10 +1772,10 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18441772 -- terminates early inside extract, we may still have buffered data
18451773 -- remaining which will be lost if we do not communicate that to the
18461774 -- driver.
1847− extract state = do
1775+ extractFunc fex state = do
18481776 let consumeWord s n wrd = do
18491777 if n == 0
1850− then fextract s
1778+ then fex s
18511779 else do
18521780 let old = elemMask .&. (wrd `shiftR` (elemBits * (n - 1)))
18531781 r <- fstep s (toEnum $ fromIntegral old)
@@ -1857,7 +1785,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18571785
18581786 let consumeRing s n rb rh =
18591787 if n == 0
1860− then fextract s
1788+ then fex s
18611789 else do
18621790 old <- liftIO $ peek rh
18631791 let rh1 = Ring.advance rb rh
@@ -1867,13 +1795,17 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18671795 Done b -> return b
18681796
18691797 case state of
1870− SplitOnSeqEmpty s -> fextract s
1871− SplitOnSeqSingle s _ -> fextract s
1798+ SplitOnSeqEmpty s -> fex s
1799+ SplitOnSeqSingle s _ -> fex s
18721800 SplitOnSeqWord s idx wrd -> consumeWord s idx wrd
18731801 SplitOnSeqWordLoop s wrd -> consumeWord s patLen wrd
18741802 SplitOnSeqKR s idx rb _ -> consumeRing s idx rb (Ring.startOf rb)
18751803 SplitOnSeqKRLoop s _ rb rh -> consumeRing s patLen rb rh
18761804
1805+ extract state = extractFunc fextract state
1806+
1807+ final state = extractFunc ffinal state
1808+
18771809------------------------------------------------------------------------------
18781810-- Distributing
18791811------------------------------------------------------------------------------
@@ -2317,8 +2249,8 @@ toStreamRev = fmap StreamD.fromList toListRev
23172249-- /Pre-release/
23182250{-# INLINE unfoldMany #-}
23192251unfoldMany :: Monad m => Unfold m a b -> Fold m b c -> Fold m a c
2320−unfoldMany (Unfold ustep inject) (Fold fstep initial extract) =
2321− Fold consume initial extract
2252+unfoldMany (Unfold ustep inject) (Fold fstep initial extract final) =
2253+ Fold consume initial extract final
23222254
23232255 where
23242256
@@ -2345,7 +2277,7 @@ bottomBy :: (MonadIO m, Unbox a) =>
23452277 (a -> a -> Ordering)
23462278 -> Int
23472279 -> Fold m a (MutArray a)
2348−bottomBy cmp n = Fold step initial extract
2280+bottomBy cmp n = Fold step initial extract extract
23492281
23502282 where
23512283
@@ -2440,8 +2372,8 @@ intersperseWithQuotes
24402372 quote
24412373 esc
24422374 separator
2443− (Fold stepL initialL extractL)
2444− (Fold stepR initialR extractR) = Fold step initial extract
2375+ (Fold stepL initialL _ finalL)
2376+ (Fold stepR initialR extractR finalR) = Fold step initial extract final
24452377
24462378 where
24472379
@@ -2484,11 +2416,13 @@ intersperseWithQuotes
24842416 r <- stepL sL a
24852417 case r of
24862418 Partial s -> return $ Partial (nextState sR s)
2487− Done _ -> error "Collecting fold finished inside quote"
2419+ Done _ -> do
2420+ _ <- finalR sR
2421+ error "Collecting fold finished inside quote"
24882422
24892423 step (IntersperseQUnquoted sR sL) a
24902424 | a == separator = do
2491− b <- extractL sL
2425+ b <- finalL sL
24922426 collect IntersperseQUnquoted sR b
24932427 | a == quote = processQuoted a sL sR IntersperseQQuoted
24942428 | otherwise = process a sL sR IntersperseQUnquoted
@@ -2506,3 +2440,13 @@ intersperseWithQuotes
25062440 error "intersperseWithQuotes: finished inside quote"
25072441 extract (IntersperseQQuotedEsc _ _) =
25082442 error "intersperseWithQuotes: finished inside quote, at escape char"
2443+
2444+ final (IntersperseQUnquoted sR sL) = finalL sL *> finalR sR
2445+ final (IntersperseQQuoted sR sL) = do
2446+ _ <- finalR sR
2447+ _ <- finalL sL
2448+ error "intersperseWithQuotes: finished inside quote"
2449+ final (IntersperseQQuotedEsc sR sL) = do
2450+ _ <- finalR sR
2451+ _ <- finalL sL
2452+ error "intersperseWithQuotes: finished inside quote, at escape char"
core/src/Streamly/Internal/Data/Fold/Container.hs+73−28
@@ -270,14 +270,15 @@ demuxGeneric :: (Monad m, IsMap f, Traversable f) =>
270270 (a -> Key f)
271271 -> (a -> m (Fold m a b))
272272 -> Fold m a (m (f b), Maybe (Key f, b))
273−demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
273+demuxGeneric getKey getFold =
274+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
274275
275276 where
276277
277278 initial = return $ Tuple' IsMap.mapEmpty Nothing
278279
279280 {-# INLINE runFold #-}
280− runFold kv (Fold step1 initial1 extract1) (k, a) = do
281+ runFold kv (Fold step1 initial1 extract1 final1) (k, a) = do
281282 res <- initial1
282283 case res of
283284 Partial s -> do
@@ -285,7 +286,7 @@ demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
285286 return
286287 $ case res1 of
287288 Partial _ ->
288− let fld = Fold step1 (return res1) extract1
289+ let fld = Fold step1 (return res1) extract1 final1
289290 in Tuple' (IsMap.mapInsert k fld kv) Nothing
290291 Done b -> Tuple' (IsMap.mapDelete k kv) (Just (k, b))
291292 Done b -> return $ Tuple' kv (Just (k, b))
@@ -298,15 +299,25 @@ demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
298299 runFold kv fld (k, a)
299300 Just f -> runFold kv f (k, a)
300301
301− extract (Tuple' kv x) = (Prelude.mapM f kv, x)
302+ extract (Tuple' kv x) = return (Prelude.mapM f kv, x)
302303
303304 where
304305
305− f (Fold _ i e) = do
306+ f (Fold _ i e _) = do
306307 r <- i
307308 case r of
308309 Partial s -> e s
309− Done b -> return b
310+ _ -> error "demuxGeneric: unreachable code"
311+
312+ final (Tuple' kv x) = return (Prelude.mapM f kv, x)
313+
314+ where
315+
316+ f (Fold _ i _ fin) = do
317+ r <- i
318+ case r of
319+ Partial s -> fin s
320+ _ -> error "demuxGeneric: unreachable code"
310321
311322-- | @demux getKey getFold@: In a key value stream, fold values corresponding
312323-- to each key using a key specific fold. @getFold@ is invoked to generate a
@@ -348,35 +359,36 @@ demuxGenericIO :: (MonadIO m, IsMap f, Traversable f) =>
348359 (a -> Key f)
349360 -> (a -> m (Fold m a b))
350361 -> Fold m a (m (f b), Maybe (Key f, b))
351−demuxGenericIO getKey getFold = fmap extract $ foldlM' step initial
362+demuxGenericIO getKey getFold =
363+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
352364
353365 where
354366
355367 initial = return $ Tuple' IsMap.mapEmpty Nothing
356368
357369 {-# INLINE initFold #-}
358− initFold kv (Fold step1 initial1 extract1) (k, a) = do
370+ initFold kv (Fold step1 initial1 extract1 final1) (k, a) = do
359371 res <- initial1
360372 case res of
361373 Partial s -> do
362374 res1 <- step1 s a
363375 case res1 of
364376 Partial _ -> do
365− let fld = Fold step1 (return res1) extract1
377+ let fld = Fold step1 (return res1) extract1 final1
366378 ref <- liftIO $ newIORef fld
367379 return $ Tuple' (IsMap.mapInsert k ref kv) Nothing
368380 Done b -> return $ Tuple' kv (Just (k, b))
369381 Done b -> return $ Tuple' kv (Just (k, b))
370382
371383 {-# INLINE runFold #-}
372− runFold kv ref (Fold step1 initial1 extract1) (k, a) = do
384+ runFold kv ref (Fold step1 initial1 extract1 final1) (k, a) = do
373385 res <- initial1
374386 case res of
375387 Partial s -> do
376388 res1 <- step1 s a
377389 case res1 of
378390 Partial _ -> do
379− let fld = Fold step1 (return res1) extract1
391+ let fld = Fold step1 (return res1) extract1 final1
380392 liftIO $ writeIORef ref fld
381393 return $ Tuple' kv Nothing
382394 Done b ->
@@ -394,16 +406,27 @@ demuxGenericIO getKey getFold = fmap extract $ foldlM' step initial
394406 f <- liftIO $ readIORef ref
395407 runFold kv ref f (k, a)
396408
397− extract (Tuple' kv x) = (Prelude.mapM f kv, x)
409+ extract (Tuple' kv x) = return (Prelude.mapM f kv, x)
398410
399411 where
400412
401413 f ref = do
402− (Fold _ i e) <- liftIO $ readIORef ref
414+ Fold _ i e _ <- liftIO $ readIORef ref
403415 r <- i
404416 case r of
405417 Partial s -> e s
406− Done b -> return b
418+ _ -> error "demuxGenericIO: unreachable code"
419+
420+ final (Tuple' kv x) = return (Prelude.mapM f kv, x)
421+
422+ where
423+
424+ f ref = do
425+ Fold _ i _ fin <- liftIO $ readIORef ref
426+ r <- i
427+ case r of
428+ Partial s -> fin s
429+ _ -> error "demuxGenericIO: unreachable code"
407430
408431-- | This is specialized version of 'demux' that uses mutable IO cells as
409432-- fold accumulators for better performance.
@@ -418,6 +441,13 @@ demuxIO :: (MonadIO m, Ord k) =>
418441 -> Fold m a (m (Map k b), Maybe (k, b))
419442demuxIO = demuxGenericIO
420443
444+-- | Fold a key value stream to a key-value Map. If the same key appears
445+-- multiple times, only the last value is retained.
446+{-# INLINE kvToMapOverwriteGeneric #-}
447+kvToMapOverwriteGeneric :: (Monad m, IsMap f) => Fold m (Key f, a) (f a)
448+kvToMapOverwriteGeneric =
449+ foldl' (\kv (k, v) -> IsMap.mapInsert k v kv) IsMap.mapEmpty
450+
421451{-# INLINE demuxToContainer #-}
422452demuxToContainer :: (Monad m, IsMap f, Traversable f) =>
423453 (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
@@ -505,8 +535,8 @@ classifyGeneric :: (Monad m, IsMap f, Traversable f, Ord (Key f)) =>
505535 -- for that use case. We return an action because we want it to be lazy so
506536 -- that the downstream consumers can choose to process or discard it.
507537 (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
508−classifyGeneric f (Fold step1 initial1 extract1) =
509− fmap extract $ foldlM' step initial
538+classifyGeneric f (Fold step1 initial1 extract1 final1) =
539+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
510540
511541 where
512542
@@ -543,7 +573,16 @@ classifyGeneric f (Fold step1 initial1 extract1) =
543573 let kv1 = IsMap.mapDelete k kv
544574 in Tuple3' kv1 (Set.insert k set) (Just (k, b))
545575
546− extract (Tuple3' kv _ x) = (Prelude.mapM extract1 kv, x)
576+ extract (Tuple3' kv _ x) = return (Prelude.mapM extract1 kv, x)
577+
578+ final (Tuple3' kv set x) = return (IsMap.mapTraverseWithKey f1 kv, x)
579+
580+ where
581+
582+ f1 k s = do
583+ if Set.member k set
584+ then extract1 s
585+ else final1 s
547586
548587-- | Folds the values for each key using the supplied fold. When scanning, as
549588-- soon as the fold is complete, its result is available in the second
@@ -568,8 +607,8 @@ classify = classifyGeneric
568607{-# INLINE classifyGenericIO #-}
569608classifyGenericIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) =>
570609 (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
571−classifyGenericIO f (Fold step1 initial1 extract1) =
572− fmap extract $ foldlM' step initial
610+classifyGenericIO f (Fold step1 initial1 extract1 final1) =
611+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
573612
574613 where
575614
@@ -608,8 +647,21 @@ classifyGenericIO f (Fold step1 initial1 extract1) =
608647 in return
609648 $ Tuple3' kv1 (Set.insert k set) (Just (k, b))
610649
611− extract (Tuple3' kv _ x) =
612− (Prelude.mapM (\ref -> liftIO (readIORef ref) >>= extract1) kv, x)
650+ extract (Tuple3' kv _ x) = return (Prelude.mapM g kv, x)
651+
652+ where
653+
654+ g ref = liftIO (readIORef ref) >>= extract1
655+
656+ final (Tuple3' kv set x) = return (IsMap.mapTraverseWithKey g kv, x)
657+
658+ where
659+
660+ g k ref = do
661+ s <- liftIO $ readIORef ref
662+ if Set.member k set
663+ then extract1 s
664+ else final1 s
613665
614666-- | Same as classify except that it uses mutable IORef cells in the
615667-- Map providing better performance. Be aware that if this is used as a scan,
@@ -624,13 +676,6 @@ classifyIO :: (MonadIO m, Ord k) =>
624676 (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
625677classifyIO = classifyGenericIO
626678
627−-- | Fold a key value stream to a key-value Map. If the same key appears
628−-- multiple times, only the last value is retained.
629−{-# INLINE kvToMapOverwriteGeneric #-}
630−kvToMapOverwriteGeneric :: (Monad m, IsMap f) => Fold m (Key f, a) (f a)
631−kvToMapOverwriteGeneric =
632− foldl' (\kv (k, v) -> IsMap.mapInsert k v kv) IsMap.mapEmpty
633−
634679{-# INLINE toContainer #-}
635680toContainer :: (Monad m, IsMap f, Traversable f, Ord (Key f)) =>
636681 (a -> Key f) -> Fold m a b -> Fold m a (f b)
core/src/Streamly/Internal/Data/Fold/Step.hs+1−1
@@ -40,7 +40,7 @@ data Step s b
4040 = Partial !s
4141 | Done !b
4242
43−-- | 'first' maps over 'Partial' and 'second' maps over 'Done'.
43+-- | 'first' maps over the fold state and 'second' maps over the fold result.
4444--
4545instance Bifunctor Step where
4646 {-# INLINE bimap #-}
core/src/Streamly/Internal/Data/Fold/Tee.hs+2
@@ -16,7 +16,9 @@ module Streamly.Internal.Data.Fold.Tee
1616 )
1717where
1818
19+#if !MIN_VERSION_base(4,18,0)
1920import Control.Applicative (liftA2)
21+#endif
2022import Streamly.Internal.Data.Fold.Type (Fold)
2123
2224import qualified Streamly.Internal.Data.Fold.Type as Fold
core/src/Streamly/Internal/Data/Fold/Type.hs+338−114
@@ -367,6 +367,8 @@ module Streamly.Internal.Data.Fold.Type
367367 , toList
368368 , toStreamK
369369 , toStreamKRev
370+ , lengthGeneric
371+ , length
370372
371373 -- * Combinators
372374
@@ -391,6 +393,8 @@ module Streamly.Internal.Data.Fold.Type
391393 -- ** Trimming
392394 , take
393395 , taking
396+ , takeEndBy_
397+ , takeEndBy
394398 , dropping
395399
396400 -- ** Sequential application
@@ -442,8 +446,10 @@ where
442446
443447#include "inline.hs"
444448
449+#if !MIN_VERSION_base(4,18,0)
445450import Control.Applicative (liftA2)
446−import Control.Monad ((>=>))
451+#endif
452+import Control.Monad ((>=>), void)
447453import Data.Bifunctor (Bifunctor(..))
448454import Data.Either (fromLeft, fromRight, isLeft, isRight)
449455import Data.Functor.Identity (Identity(..))
@@ -469,23 +475,64 @@ import Streamly.Internal.Data.Fold.Step
469475-- The type @b@ is the accumulator of the writer. That's the reason the
470476-- default folds in various modules are called "write".
471477
472−-- | The type @Fold m a b@ having constructor @Fold step initial extract@
473−-- represents a fold over an input stream of values of type @a@ to a final
474−-- value of type @b@ in 'Monad' @m@.
475−--
476−-- The fold uses an intermediate state @s@ as accumulator, the type @s@ is
477−-- internal to the specific fold definition. The initial value of the fold
478−-- state @s@ is returned by @initial@. The @step@ function consumes an input
479−-- and either returns the final result @b@ if the fold is done or the next
480−-- intermediate state (see 'Step'). At any point the fold driver can extract
481−-- the result from the intermediate state using the @extract@ function.
478+-- An alternative to using an "extract" function is to use "Partial s b" style
479+-- partial value so that we always emit the output value and there is no need
480+-- to extract. Then extract can be used for cleanup purposes. But in this case
481+-- in some cases we may need a "Continue" constructor where an output value is
482+-- not available, this was implicit earlier. Also, "b" should be lazy here so
483+-- that we do not always compute it even if we do not need it.
484+--
485+-- Partial s b --> extract :: s -> b
486+-- Continue --> extract :: s -> Maybe b
487+--
488+-- But keeping 'b' lazy does not let the fold optimize well. It leads to
489+-- significant regressions in the key-value folds.
490+--
491+-- The "final" function complicates combinators that take other folds as
492+-- argument because we need to call their finalizers at right places. An
493+-- alternative to reduce this complexity where it is not required is to use a
494+-- separate type for bracketed folds but then we need to manage the complexity
495+-- of two different fold types.
496+
497+-- The "final" function could be (s -> m (Step s b)), like in parsers so that
498+-- it can be called in a loop to drain the fold.
499+
500+-- | The type @Fold m a b@ represents a consumer of an input stream of values
501+-- of type @a@ and returning a final value of type @b@ in 'Monad' @m@. The
502+-- constructor of a fold is @Fold step initial extract final@.
503+--
504+-- The fold uses an internal state of type @s@. The initial value of the state
505+-- @s@ is created by @initial@. This function is called once and only once
506+-- before the fold starts consuming input. Any resource allocation can be done
507+-- in this function.
508+--
509+-- The @step@ function is called on each input, it consumes an input and
510+-- returns the next intermediate state (see 'Step') or the final result @b@ if
511+-- the fold terminates.
512+--
513+-- If the fold is used as a scan, the @extract@ function is used by the scan
514+-- driver to map the current state @s@ of the fold to the fold result. Thus
515+-- @extract@ can be called multiple times. In some folds, where scanning does
516+-- not make sense, this function is left unimplemented; such folds cannot be
517+-- used as scans.
518+--
519+-- Before a fold terminates, @final@ is called once and only once (unless the
520+-- fold terminated in @initial@ itself). Any resources allocated by @initial@
521+-- can be released in @final@. In folds that do not require any cleanup
522+-- @extract@ and @final@ are typically the same.
523+--
524+-- When implementing fold combinators, care should be taken to cleanup any
525+-- state of the argument folds held by the fold by calling the respective
526+-- @final@ at all exit points of the fold. Also, @final@ should not be called
527+-- more than once. Note that if a fold terminates by 'Done' constructor, there
528+-- is no state to cleanup.
482529--
483530-- NOTE: The constructor is not yet released, smart constructors are provided
484531-- to create folds.
485532--
486533data Fold m a b =
487− -- | @Fold @ @ step @ @ initial @ @ extract@
488− forall s. Fold (s -> a -> m (Step s b)) (m (Step s b)) (s -> m b)
534+ -- | @Fold@ @step@ @initial@ @extract@ @final@
535+ forall s. Fold (s -> a -> m (Step s b)) (m (Step s b)) (s -> m b) (s -> m b)
489536
490537------------------------------------------------------------------------------
491538-- Mapping on the output
@@ -495,7 +542,8 @@ data Fold m a b =
495542--
496543{-# INLINE rmapM #-}
497544rmapM :: Monad m => (b -> m c) -> Fold m a b -> Fold m a c
498−rmapM f (Fold step initial extract) = Fold step1 initial1 (extract >=> f)
545+rmapM f (Fold step initial extract final) =
546+ Fold step1 initial1 (extract >=> f) (final >=> f)
499547
500548 where
501549
@@ -526,6 +574,7 @@ foldl' step initial =
526574 (\s a -> return $ Partial $ step s a)
527575 (return (Partial initial))
528576 return
577+ return
529578
530579-- | Make a fold from a left fold style monadic step function and initial value
531580-- of the accumulator.
@@ -540,7 +589,7 @@ foldl' step initial =
540589{-# INLINE foldlM' #-}
541590foldlM' :: Monad m => (b -> a -> m b) -> m b -> Fold m a b
542591foldlM' step initial =
543− Fold (\s a -> Partial <$> step s a) (Partial <$> initial) return
592+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) return return
544593
545594-- | Make a strict left fold, for non-empty streams, using first element as the
546595-- starting value. Returns Nothing if the stream is empty.
@@ -638,7 +687,11 @@ foldrM' g z =
638687{-# INLINE foldt' #-}
639688foldt' :: Monad m => (s -> a -> Step s b) -> Step s b -> (s -> b) -> Fold m a b
640689foldt' step initial extract =
641− Fold (\s a -> return $ step s a) (return initial) (return . extract)
690+ Fold
691+ (\s a -> return $ step s a)
692+ (return initial)
693+ (return . extract)
694+ (return . extract)
642695
643696-- | Make a terminating fold with an effectful step function and initial state,
644697-- and a state extraction function.
@@ -651,7 +704,7 @@ foldt' step initial extract =
651704--
652705{-# INLINE foldtM' #-}
653706foldtM' :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> Fold m a b
654−foldtM' = Fold
707+foldtM' step initial extract = Fold step initial extract extract
655708
656709------------------------------------------------------------------------------
657710-- Refold
@@ -665,7 +718,7 @@ foldtM' = Fold
665718-- /Internal/
666719fromRefold :: Refold m c a b -> c -> Fold m a b
667720fromRefold (Refold step inject extract) c =
668− Fold step (inject c) extract
721+ Fold step (inject c) extract extract
669722
670723------------------------------------------------------------------------------
671724-- Basic Folds
@@ -718,6 +771,29 @@ toStreamKRev = foldl' (flip K.cons) K.nil
718771toStreamK :: Monad m => Fold m a (K.StreamK n a)
719772toStreamK = foldr K.cons K.nil
720773
774+-- | Like 'length', except with a more general 'Num' return value
775+--
776+-- Definition:
777+--
778+-- >>> lengthGeneric = fmap getSum $ Fold.foldMap (Sum . const 1)
779+-- >>> lengthGeneric = Fold.foldl' (\n _ -> n + 1) 0
780+--
781+-- /Pre-release/
782+{-# INLINE lengthGeneric #-}
783+lengthGeneric :: (Monad m, Num b) => Fold m a b
784+lengthGeneric = foldl' (\n _ -> n + 1) 0
785+
786+-- | Determine the length of the input stream.
787+--
788+-- Definition:
789+--
790+-- >>> length = Fold.lengthGeneric
791+-- >>> length = fmap getSum $ Fold.foldMap (Sum . const 1)
792+--
793+{-# INLINE length #-}
794+length :: Monad m => Fold m a Int
795+length = lengthGeneric
796+
721797------------------------------------------------------------------------------
722798-- Instances
723799------------------------------------------------------------------------------
@@ -725,7 +801,8 @@ toStreamK = foldr K.cons K.nil
725801-- | Maps a function on the output of the fold (the type @b@).
726802instance Functor m => Functor (Fold m a) where
727803 {-# INLINE fmap #-}
728− fmap f (Fold step1 initial1 extract) = Fold step initial (fmap2 f extract)
804+ fmap f (Fold step1 initial1 extract final) =
805+ Fold step initial (fmap2 f extract) (fmap2 f final)
729806
730807 where
731808
@@ -760,7 +837,7 @@ instance Functor m => Functor (Fold m a) where
760837--
761838{-# INLINE fromPure #-}
762839fromPure :: Applicative m => b -> Fold m a b
763−fromPure b = Fold undefined (pure $ Done b) pure
840+fromPure b = Fold undefined (pure $ Done b) pure pure
764841
765842-- | Make a fold that yields the result of the supplied effectful action
766843-- without consuming any further input.
@@ -769,7 +846,7 @@ fromPure b = Fold undefined (pure $ Done b) pure
769846--
770847{-# INLINE fromEffect #-}
771848fromEffect :: Applicative m => m b -> Fold m a b
772−fromEffect b = Fold undefined (Done <$> b) pure
849+fromEffect b = Fold undefined (Done <$> b) pure pure
773850
774851{-# ANN type SeqFoldState Fuse #-}
775852data SeqFoldState sl f sr = SeqFoldL !sl | SeqFoldR !f !sr
@@ -794,16 +871,20 @@ data SeqFoldState sl f sr = SeqFoldL !sl | SeqFoldR !f !sr
794871-- complexity, because each composition adds a new branch that each subsequent
795872-- fold's input element has to traverse, therefore, it cannot scale to a large
796873-- number of compositions. After around 100 compositions the performance starts
797−-- dipping rapidly compared to a CPS style implementation. When you need
798−-- scaling use parser monad instead.
874+-- dipping rapidly compared to a CPS style implementation.
875+--
876+-- For larger number of compositions you can convert the fold to a parser and
877+-- use ParserK.
799878--
800879-- /Time: O(n^2) where n is the number of compositions./
801880--
802881{-# INLINE splitWith #-}
803882splitWith :: Monad m =>
804883 (a -> b -> c) -> Fold m x a -> Fold m x b -> Fold m x c
805−splitWith func (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
806− Fold step initial extract
884+splitWith func
885+ (Fold stepL initialL _ finalL)
886+ (Fold stepR initialR _ finalR) =
887+ Fold step initial extract final
807888
808889 where
809890
@@ -820,13 +901,18 @@ splitWith func (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
820901 step (SeqFoldL st) a = runL (stepL st a)
821902 step (SeqFoldR f st) a = runR (stepR st a) f
822903
823− extract (SeqFoldR f sR) = fmap f (extractR sR)
824− extract (SeqFoldL sL) = do
825− rL <- extractL sL
904+ -- XXX splitWith should not be used for scanning
905+ -- It would rarely make sense and resource tracking and cleanup would be
906+ -- expensive. especially when multiple splitWith are chained.
907+ extract _ = error "splitWith: cannot be used for scanning"
908+
909+ final (SeqFoldR f sR) = fmap f (finalR sR)
910+ final (SeqFoldL sL) = do
911+ rL <- finalL sL
826912 res <- initialR
827913 fmap (func rL)
828914 $ case res of
829− Partial sR -> extractR sR
915+ Partial sR -> finalR sR
830916 Done rR -> return rR
831917
832918{-# DEPRECATED serialWith "Please use \"splitWith\" instead" #-}
@@ -846,8 +932,8 @@ data SeqFoldState_ sl sr = SeqFoldL_ !sl | SeqFoldR_ !sr
846932--
847933{-# INLINE split_ #-}
848934split_ :: Monad m => Fold m x a -> Fold m x b -> Fold m x b
849−split_ (Fold stepL initialL _) (Fold stepR initialR extractR) =
850− Fold step initial extract
935+split_ (Fold stepL initialL _ finalL) (Fold stepR initialR _ finalR) =
936+ Fold step initial extract final
851937
852938 where
853939
@@ -870,11 +956,16 @@ split_ (Fold stepL initialL _) (Fold stepR initialR extractR) =
870956 resR <- stepR st a
871957 return $ first SeqFoldR_ resR
872958
873− extract (SeqFoldR_ sR) = extractR sR
874− extract (SeqFoldL_ _) = do
959+ -- XXX split_ should not be used for scanning
960+ -- See splitWith for more details.
961+ extract _ = error "split_: cannot be used for scanning"
962+
963+ final (SeqFoldR_ sR) = finalR sR
964+ final (SeqFoldL_ sL) = do
965+ _ <- finalL sL
875966 res <- initialR
876967 case res of
877− Partial sR -> extractR sR
968+ Partial sR -> finalR sR
878969 Done rR -> return rR
879970
880971-- | 'Applicative' form of 'splitWith'. Split the input serially over two
@@ -922,8 +1013,10 @@ data TeeState sL sR bL bR
9221013--
9231014{-# INLINE teeWith #-}
9241015teeWith :: Monad m => (a -> b -> c) -> Fold m x a -> Fold m x b -> Fold m x c
925−teeWith f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
926− Fold step initial extract
1016+teeWith f
1017+ (Fold stepL initialL extractL finalL)
1018+ (Fold stepR initialR extractR finalR) =
1019+ Fold step initial extract final
9271020
9281021 where
9291022
@@ -950,6 +1043,10 @@ teeWith f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
9501043 extract (TeeLeft bR sL) = (`f` bR) <$> extractL sL
9511044 extract (TeeRight bL sR) = f bL <$> extractR sR
9521045
1046+ final (TeeBoth sL sR) = f <$> finalL sL <*> finalR sR
1047+ final (TeeLeft bR sL) = (`f` bR) <$> finalL sL
1048+ final (TeeRight bL sR) = f bL <$> finalR sR
1049+
9531050{-# ANN type TeeFstState Fuse #-}
9541051data TeeFstState sL sR b
9551052 = TeeFstBoth !sL !sR
@@ -962,8 +1059,10 @@ data TeeFstState sL sR b
9621059{-# INLINE teeWithFst #-}
9631060teeWithFst :: Monad m =>
9641061 (b -> c -> d) -> Fold m a b -> Fold m a c -> Fold m a d
965−teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
966− Fold step initial extract
1062+teeWithFst f
1063+ (Fold stepL initialL extractL finalL)
1064+ (Fold stepR initialR extractR finalR) =
1065+ Fold step initial extract final
9671066
9681067 where
9691068
@@ -982,7 +1081,7 @@ teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
9821081 Done bl -> do
9831082 Done . f bl <$>
9841083 case resR of
985− Partial sr -> extractR sr
1084+ Partial sr -> finalR sr
9861085 Done br -> return br
9871086
9881087 initial = runBoth initialL initialR
@@ -993,6 +1092,9 @@ teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
9931092 extract (TeeFstBoth sL sR) = f <$> extractL sL <*> extractR sR
9941093 extract (TeeFstLeft bR sL) = (`f` bR) <$> extractL sL
9951094
1095+ final (TeeFstBoth sL sR) = f <$> finalL sL <*> finalR sR
1096+ final (TeeFstLeft bR sL) = (`f` bR) <$> finalL sL
1097+
9961098-- | Like 'teeWith' but terminates as soon as any one of the two folds
9971099-- terminates.
9981100--
@@ -1001,8 +1103,10 @@ teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10011103{-# INLINE teeWithMin #-}
10021104teeWithMin :: Monad m =>
10031105 (b -> c -> d) -> Fold m a b -> Fold m a c -> Fold m a d
1004−teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1005− Fold step initial extract
1106+teeWithMin f
1107+ (Fold stepL initialL extractL finalL)
1108+ (Fold stepR initialR extractR finalR) =
1109+ Fold step initial extract final
10061110
10071111 where
10081112
@@ -1014,12 +1118,12 @@ teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10141118 Partial sl -> do
10151119 case resR of
10161120 Partial sr -> return $ Partial $ Tuple' sl sr
1017− Done br -> Done . (`f` br) <$> extractL sl
1121+ Done br -> Done . (`f` br) <$> finalL sl
10181122
10191123 Done bl -> do
10201124 Done . f bl <$>
10211125 case resR of
1022− Partial sr -> extractR sr
1126+ Partial sr -> finalR sr
10231127 Done br -> return br
10241128
10251129 initial = runBoth initialL initialR
@@ -1028,6 +1132,8 @@ teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10281132
10291133 extract (Tuple' sL sR) = f <$> extractL sL <*> extractR sR
10301134
1135+ final (Tuple' sL sR) = f <$> finalL sL <*> finalR sR
1136+
10311137-- | Shortest alternative. Apply both folds in parallel but choose the result
10321138-- from the one which consumed least input i.e. take the shortest succeeding
10331139-- fold.
@@ -1039,8 +1145,8 @@ teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10391145--
10401146{-# INLINE shortest #-}
10411147shortest :: Monad m => Fold m x a -> Fold m x b -> Fold m x (Either a b)
1042−shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
1043− Fold step initial extract
1148+shortest (Fold stepL initialL extractL finalL) (Fold stepR initialR _ finalR) =
1149+ Fold step initial extract final
10441150
10451151 where
10461152
@@ -1048,10 +1154,16 @@ shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
10481154 runBoth actionL actionR = do
10491155 resL <- actionL
10501156 resR <- actionR
1051− return $
1052− case resL of
1053− Partial sL -> bimap (Tuple' sL) Right resR
1054− Done bL -> Done $ Left bL
1157+ case resL of
1158+ Partial sL ->
1159+ case resR of
1160+ Partial sR -> return $ Partial $ Tuple' sL sR
1161+ Done bR -> finalL sL >> return (Done (Right bR))
1162+ Done bL -> do
1163+ case resR of
1164+ Partial sR -> void (finalR sR)
1165+ Done _ -> return ()
1166+ return (Done (Left bL))
10551167
10561168 initial = runBoth initialL initialR
10571169
@@ -1059,6 +1171,8 @@ shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
10591171
10601172 extract (Tuple' sL _) = Left <$> extractL sL
10611173
1174+ final (Tuple' sL sR) = Left <$> finalL sL <* finalR sR
1175+
10621176{-# ANN type LongestState Fuse #-}
10631177data LongestState sL sR
10641178 = LongestBoth !sL !sR
@@ -1076,8 +1190,10 @@ data LongestState sL sR
10761190--
10771191{-# INLINE longest #-}
10781192longest :: Monad m => Fold m x a -> Fold m x b -> Fold m x (Either a b)
1079−longest (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1080− Fold step initial extract
1193+longest
1194+ (Fold stepL initialL _ finalL)
1195+ (Fold stepR initialR _ finalR) =
1196+ Fold step initial extract final
10811197
10821198 where
10831199
@@ -1100,14 +1216,18 @@ longest (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
11001216 step (LongestLeft sL) a = bimap LongestLeft Left <$> stepL sL a
11011217 step (LongestRight sR) a = bimap LongestRight Right <$> stepR sR a
11021218
1103− left sL = Left <$> extractL sL
1104− extract (LongestLeft sL) = left sL
1105− extract (LongestRight sR) = Right <$> extractR sR
1106− extract (LongestBoth sL _) = left sL
1219+ -- XXX Scan with this may not make sense as we cannot determine the longest
1220+ -- until one of them have exhausted.
1221+ extract _ = error $ "longest: scan is not allowed as longest cannot be "
1222+ ++ "determined until one fold has exhausted."
1223+
1224+ final (LongestLeft sL) = Left <$> finalL sL
1225+ final (LongestRight sR) = Right <$> finalR sR
1226+ final (LongestBoth sL sR) = Left <$> finalL sL <* finalR sR
11071227
1108−data ConcatMapState m sa a c
1109− = B !sa
1110− | forall s. C (s -> a -> m (Step s c)) !s (s -> m c)
1228+data ConcatMapState m sa a b c
1229+ = B !sa (sa -> m b)
1230+ | forall s. C (s -> a -> m (Step s c)) !s (s -> m c) (s -> m c)
11111231
11121232-- | Map a 'Fold' returning function on the result of a 'Fold' and run the
11131233-- returned fold. This operation can be used to express data dependencies
@@ -1130,43 +1250,47 @@ data ConcatMapState m sa a c
11301250--
11311251{-# INLINE concatMap #-}
11321252concatMap :: Monad m => (b -> Fold m a c) -> Fold m a b -> Fold m a c
1133−concatMap f (Fold stepa initiala extracta) = Fold stepc initialc extractc
1253+concatMap f (Fold stepa initiala _ finala) =
1254+ Fold stepc initialc extractc finalc
11341255 where
11351256 initialc = do
11361257 r <- initiala
11371258 case r of
1138− Partial s -> return $ Partial (B s)
1259+ Partial s -> return $ Partial (B s finala)
11391260 Done b -> initInnerFold (f b)
11401261
1141− stepc (B s) a = do
1262+ stepc (B s fin) a = do
11421263 r <- stepa s a
11431264 case r of
1144− Partial s1 -> return $ Partial (B s1)
1265+ Partial s1 -> return $ Partial (B s1 fin)
11451266 Done b -> initInnerFold (f b)
11461267
1147− stepc (C stepInner s extractInner) a = do
1268+ stepc (C stepInner s extractInner fin) a = do
11481269 r <- stepInner s a
11491270 return $ case r of
1150− Partial sc -> Partial (C stepInner sc extractInner)
1271+ Partial sc -> Partial (C stepInner sc extractInner fin)
11511272 Done c -> Done c
11521273
1153− extractc (B s) = do
1154− r <- extracta s
1155− initExtract (f r)
1156− extractc (C _ sInner extractInner) = extractInner sInner
1274+ -- XXX Cannot use for scanning
1275+ extractc _ = error "concatMap: cannot be used for scanning"
11571276
1158− initInnerFold (Fold step i e) = do
1277+ initInnerFold (Fold step i e fin) = do
11591278 r <- i
11601279 return $ case r of
1161− Partial s -> Partial (C step s e)
1280+ Partial s -> Partial (C step s e fin)
11621281 Done c -> Done c
11631282
1164− initExtract (Fold _ i e) = do
1283+ initFinalize (Fold _ i _ fin) = do
11651284 r <- i
11661285 case r of
1167− Partial s -> e s
1286+ Partial s -> fin s
11681287 Done c -> return c
11691288
1289+ finalc (B s fin) = do
1290+ r <- fin s
1291+ initFinalize (f r)
1292+ finalc (C _ sInner _ fin) = fin sInner
1293+
11701294------------------------------------------------------------------------------
11711295-- Mapping on input
11721296------------------------------------------------------------------------------
@@ -1185,7 +1309,7 @@ concatMap f (Fold stepa initiala extracta) = Fold stepc initialc extractc
11851309--
11861310{-# INLINE lmap #-}
11871311lmap :: (a -> b) -> Fold m b r -> Fold m a r
1188−lmap f (Fold step begin done) = Fold step' begin done
1312+lmap f (Fold step begin done final) = Fold step' begin done final
11891313 where
11901314 step' x a = step x (f a)
11911315
@@ -1193,7 +1317,7 @@ lmap f (Fold step begin done) = Fold step' begin done
11931317--
11941318{-# INLINE lmapM #-}
11951319lmapM :: Monad m => (a -> m b) -> Fold m b r -> Fold m a r
1196−lmapM f (Fold step begin done) = Fold step' begin done
1320+lmapM f (Fold step begin done final) = Fold step' begin done final
11971321 where
11981322 step' x a = f a >>= step x
11991323
@@ -1205,8 +1329,10 @@ lmapM f (Fold step begin done) = Fold step' begin done
12051329-- /Pre-release/
12061330{-# INLINE postscan #-}
12071331postscan :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1208−postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1209− Fold step initial extract
1332+postscan
1333+ (Fold stepL initialL extractL finalL)
1334+ (Fold stepR initialR extractR finalR) =
1335+ Fold step initial extract final
12101336
12111337 where
12121338
@@ -1217,30 +1343,32 @@ postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
12171343 Done bL -> do
12181344 rR <- stepR sR bL
12191345 case rR of
1220− Partial sR1 -> Done <$> extractR sR1
1346+ Partial sR1 -> Done <$> finalR sR1
12211347 Done bR -> return $ Done bR
12221348 Partial sL -> do
12231349 !b <- extractL sL
12241350 rR <- stepR sR b
1225− return
1226− $ case rR of
1227− Partial sR1 -> Partial (sL, sR1)
1228− Done bR -> Done bR
1351+ case rR of
1352+ Partial sR1 -> return $ Partial (sL, sR1)
1353+ Done bR -> finalL sL >> return (Done bR)
12291354
12301355 initial = do
1231− r <- initialR
1232− rL <- initialL
1233− case r of
1234− Partial sR ->
1356+ rR <- initialR
1357+ case rR of
1358+ Partial sR -> do
1359+ rL <- initialL
12351360 case rL of
1236− Done _ -> Done <$> extractR sR
1361+ Done _ -> Done <$> finalR sR
12371362 Partial sL -> return $ Partial (sL, sR)
12381363 Done b -> return $ Done b
12391364
1365+ -- XXX should use Tuple'
12401366 step (sL, sR) x = runStep (stepL sL x) sR
12411367
12421368 extract = extractR . snd
12431369
1370+ final (sL, sR) = finalL sL *> finalR sR
1371+
12441372------------------------------------------------------------------------------
12451373-- Filtering
12461374------------------------------------------------------------------------------
@@ -1253,7 +1381,7 @@ postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
12531381--
12541382{-# INLINE_NORMAL catMaybes #-}
12551383catMaybes :: Monad m => Fold m a b -> Fold m (Maybe a) b
1256−catMaybes (Fold step initial extract) = Fold step1 initial extract
1384+catMaybes (Fold step initial extract final) = Fold step1 initial extract final
12571385
12581386 where
12591387
@@ -1293,7 +1421,7 @@ filtering f = foldl' step Nothing
12931421{-# INLINE filter #-}
12941422filter :: Monad m => (a -> Bool) -> Fold m a r -> Fold m a r
12951423-- filter p = scanMaybe (filtering p)
1296−filter f (Fold step begin done) = Fold step' begin done
1424+filter f (Fold step begin extract final) = Fold step' begin extract final
12971425 where
12981426 step' x a = if f a then step x a else return $ Partial x
12991427
@@ -1304,7 +1432,7 @@ filter f (Fold step begin done) = Fold step' begin done
13041432--
13051433{-# INLINE filterM #-}
13061434filterM :: Monad m => (a -> m Bool) -> Fold m a r -> Fold m a r
1307−filterM f (Fold step begin done) = Fold step' begin done
1435+filterM f (Fold step begin extract final) = Fold step' begin extract final
13081436 where
13091437 step' x a = do
13101438 use <- f a
@@ -1393,7 +1521,7 @@ dropping n = foldt' step initial extract
13931521{-# INLINE take #-}
13941522take :: Monad m => Int -> Fold m a b -> Fold m a b
13951523-- take n = scanMaybe (taking n)
1396−take n (Fold fstep finitial fextract) = Fold step initial extract
1524+take n (Fold fstep finitial fextract ffinal) = Fold step initial extract final
13971525
13981526 where
13991527
@@ -1405,7 +1533,7 @@ take n (Fold fstep finitial fextract) = Fold step initial extract
14051533 s1 = Tuple'Fused i1 s
14061534 if i1 < n
14071535 then return $ Partial s1
1408− else Done <$> fextract s
1536+ else Done <$> ffinal s
14091537 Done b -> return $ Done b
14101538
14111539 initial = finitial >>= next (-1)
@@ -1414,6 +1542,72 @@ take n (Fold fstep finitial fextract) = Fold step initial extract
14141542
14151543 extract (Tuple'Fused _ r) = fextract r
14161544
1545+ final (Tuple'Fused _ r) = ffinal r
1546+
1547+-- Note: Keep this consistent with S.splitOn. In fact we should eliminate
1548+-- S.splitOn in favor of the fold.
1549+--
1550+-- XXX Use Fold.many instead once it is fixed.
1551+-- > Stream.splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
1552+
1553+-- | Like 'takeEndBy' but drops the element on which the predicate succeeds.
1554+--
1555+-- Example:
1556+--
1557+-- >>> input = Stream.fromList "hello\nthere\n"
1558+-- >>> line = Fold.takeEndBy_ (== '\n') Fold.toList
1559+-- >>> Stream.fold line input
1560+-- "hello"
1561+--
1562+-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1563+-- ["hello","there"]
1564+--
1565+{-# INLINE takeEndBy_ #-}
1566+takeEndBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1567+-- takeEndBy_ predicate = scanMaybe (takingEndBy_ predicate)
1568+takeEndBy_ predicate (Fold fstep finitial fextract ffinal) =
1569+ Fold step finitial fextract ffinal
1570+
1571+ where
1572+
1573+ step s a =
1574+ if not (predicate a)
1575+ then fstep s a
1576+ else Done <$> ffinal s
1577+
1578+-- Note:
1579+-- > Stream.splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
1580+
1581+-- | Take the input, stop when the predicate succeeds taking the succeeding
1582+-- element as well.
1583+--
1584+-- Example:
1585+--
1586+-- >>> input = Stream.fromList "hello\nthere\n"
1587+-- >>> line = Fold.takeEndBy (== '\n') Fold.toList
1588+-- >>> Stream.fold line input
1589+-- "hello\n"
1590+--
1591+-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1592+-- ["hello\n","there\n"]
1593+--
1594+{-# INLINE takeEndBy #-}
1595+takeEndBy :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1596+-- takeEndBy predicate = scanMaybe (takingEndBy predicate)
1597+takeEndBy predicate (Fold fstep finitial fextract ffinal) =
1598+ Fold step finitial fextract ffinal
1599+
1600+ where
1601+
1602+ step s a = do
1603+ res <- fstep s a
1604+ if not (predicate a)
1605+ then return res
1606+ else do
1607+ case res of
1608+ Partial s1 -> Done <$> ffinal s1
1609+ Done b -> return $ Done b
1610+
14171611------------------------------------------------------------------------------
14181612-- Nesting
14191613------------------------------------------------------------------------------
@@ -1431,8 +1625,8 @@ take n (Fold fstep finitial fextract) = Fold step initial extract
14311625-- /Pre-release/
14321626{-# INLINE duplicate #-}
14331627duplicate :: Monad m => Fold m a b -> Fold m a (Fold m a b)
1434−duplicate (Fold step1 initial1 extract1) =
1435− Fold step initial (\s -> pure $ Fold step1 (pure $ Partial s) extract1)
1628+duplicate (Fold step1 initial1 extract1 final1) =
1629+ Fold step initial extract final
14361630
14371631 where
14381632
@@ -1440,6 +1634,11 @@ duplicate (Fold step1 initial1 extract1) =
14401634
14411635 step s a = second fromPure <$> step1 s a
14421636
1637+ -- Scanning may be problematic due to multiple finalizations.
1638+ extract = error "duplicate: scanning may be problematic"
1639+
1640+ final s = pure $ Fold step1 (pure $ Partial s) extract1 final1
1641+
14431642-- If there were a finalize/flushing action in the stream type that would be
14441643-- equivalent to running initialize in Fold. But we do not have a flushing
14451644-- action in streams.
@@ -1451,9 +1650,9 @@ duplicate (Fold step1 initial1 extract1) =
14511650-- /Pre-release/
14521651{-# INLINE reduce #-}
14531652reduce :: Monad m => Fold m a b -> m (Fold m a b)
1454−reduce (Fold step initial extract) = do
1653+reduce (Fold step initial extract final) = do
14551654 i <- initial
1456− return $ Fold step (return i) extract
1655+ return $ Fold step (return i) extract final
14571656
14581657-- This is the dual of Stream @cons@.
14591658
@@ -1463,7 +1662,8 @@ reduce (Fold step initial extract) = do
14631662-- /Pre-release/
14641663{-# INLINE snoclM #-}
14651664snoclM :: Monad m => Fold m a b -> m a -> Fold m a b
1466−snoclM (Fold fstep finitial fextract) action = Fold fstep initial fextract
1665+snoclM (Fold fstep finitial fextract ffinal) action =
1666+ Fold fstep initial fextract ffinal
14671667
14681668 where
14691669
@@ -1490,7 +1690,8 @@ snoclM (Fold fstep finitial fextract) action = Fold fstep initial fextract
14901690{-# INLINE snocl #-}
14911691snocl :: Monad m => Fold m a b -> a -> Fold m a b
14921692-- snocl f = snoclM f . return
1493−snocl (Fold fstep finitial fextract) a = Fold fstep initial fextract
1693+snocl (Fold fstep finitial fextract ffinal) a =
1694+ Fold fstep initial fextract ffinal
14941695
14951696 where
14961697
@@ -1510,12 +1711,12 @@ snocl (Fold fstep finitial fextract) a = Fold fstep initial fextract
15101711-- /Pre-release/
15111712{-# INLINE snocM #-}
15121713snocM :: Monad m => Fold m a b -> m a -> m (Fold m a b)
1513−snocM (Fold step initial extract) action = do
1714+snocM (Fold step initial extract final) action = do
15141715 res <- initial
15151716 r <- case res of
15161717 Partial fs -> action >>= step fs
15171718 Done _ -> return res
1518− return $ Fold step (return r) extract
1719+ return $ Fold step (return r) extract final
15191720
15201721-- Definitions:
15211722--
@@ -1534,12 +1735,12 @@ snocM (Fold step initial extract) action = do
15341735-- /Pre-release/
15351736{-# INLINE snoc #-}
15361737snoc :: Monad m => Fold m a b -> a -> m (Fold m a b)
1537−snoc (Fold step initial extract) a = do
1738+snoc (Fold step initial extract final) a = do
15381739 res <- initial
15391740 r <- case res of
15401741 Partial fs -> step fs a
15411742 Done _ -> return res
1542− return $ Fold step (return r) extract
1743+ return $ Fold step (return r) extract final
15431744
15441745-- | Append a singleton value to the fold.
15451746--
@@ -1567,7 +1768,7 @@ addOne = flip snoc
15671768-- /Pre-release/
15681769{-# INLINE extractM #-}
15691770extractM :: Monad m => Fold m a b -> m b
1570−extractM (Fold _ initial extract) = do
1771+extractM (Fold _ initial extract _) = do
15711772 res <- initial
15721773 case res of
15731774 Partial fs -> extract fs
@@ -1576,14 +1777,15 @@ extractM (Fold _ initial extract) = do
15761777-- | Close a fold so that it does not accept any more input.
15771778{-# INLINE close #-}
15781779close :: Monad m => Fold m a b -> Fold m a b
1579−close (Fold _ initial1 extract1) = Fold undefined initial undefined
1780+close (Fold _ initial1 _ final1) =
1781+ Fold undefined initial undefined undefined
15801782
15811783 where
15821784
15831785 initial = do
15841786 res <- initial1
15851787 case res of
1586− Partial s -> Done <$> extract1 s
1788+ Partial s -> Done <$> final1 s
15871789 Done b -> return $ Done b
15881790
15891791-- Corresponds to the null check for streams.
@@ -1593,7 +1795,7 @@ close (Fold _ initial1 extract1) = Fold undefined initial undefined
15931795-- /Pre-release/
15941796{-# INLINE isClosed #-}
15951797isClosed :: Monad m => Fold m a b -> m Bool
1596−isClosed (Fold _ initial _) = do
1798+isClosed (Fold _ initial _ _) = do
15971799 res <- initial
15981800 return $ case res of
15991801 Partial _ -> False
@@ -1627,8 +1829,10 @@ data ManyState s1 s2
16271829--
16281830{-# INLINE many #-}
16291831many :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1630−many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
1631− Fold step initial extract
1832+many
1833+ (Fold sstep sinitial sextract sfinal)
1834+ (Fold cstep cinitial cextract cfinal) =
1835+ Fold step initial extract final
16321836
16331837 where
16341838
@@ -1677,6 +1881,13 @@ many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
16771881 Partial s -> cextract s
16781882 Done b -> return b
16791883
1884+ final (ManyFirst ss cs) = sfinal ss *> cfinal cs
1885+ final (ManyLoop ss cs) = do
1886+ cres <- sfinal ss >>= cstep cs
1887+ case cres of
1888+ Partial s -> cfinal s
1889+ Done b -> return b
1890+
16801891-- | Like many, but the "first" fold emits an output at the end even if no
16811892-- input is received.
16821893--
@@ -1686,8 +1897,10 @@ many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
16861897--
16871898{-# INLINE manyPost #-}
16881899manyPost :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1689−manyPost (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
1690− Fold step initial extract
1900+manyPost
1901+ (Fold sstep sinitial sextract sfinal)
1902+ (Fold cstep cinitial cextract cfinal) =
1903+ Fold step initial extract final
16911904
16921905 where
16931906
@@ -1723,6 +1936,12 @@ manyPost (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
17231936 Partial s -> cextract s
17241937 Done b -> return b
17251938
1939+ final (Tuple' ss cs) = do
1940+ cres <- sfinal ss >>= cstep cs
1941+ case cres of
1942+ Partial s -> cfinal s
1943+ Done b -> return b
1944+
17261945-- | @groupsOf n split collect@ repeatedly applies the @split@ fold to chunks
17271946-- of @n@ items in the input stream and supplies the result to the @collect@
17281947-- fold.
@@ -1751,7 +1970,10 @@ groupsOf n split = many (take n split)
17511970--
17521971{-# INLINE refoldMany #-}
17531972refoldMany :: Monad m => Fold m a b -> Refold m x b c -> Refold m x a c
1754−refoldMany (Fold sstep sinitial sextract) (Refold cstep cinject cextract) =
1973+refoldMany
1974+ (Fold sstep sinitial sextract _sfinal)
1975+ -- XXX We will need a "final" in refold as well
1976+ (Refold cstep cinject cextract) =
17551977 Refold step inject extract
17561978
17571979 where
@@ -1802,7 +2024,9 @@ data ConsumeManyState x cs ss = ConsumeMany x cs (Either ss ss)
18022024-- /Internal/
18032025{-# INLINE refoldMany1 #-}
18042026refoldMany1 :: Monad m => Refold m x a b -> Fold m b c -> Refold m x a c
1805−refoldMany1 (Refold sstep sinject sextract) (Fold cstep cinitial cextract) =
2027+refoldMany1
2028+ (Refold sstep sinject sextract)
2029+ (Fold cstep cinitial cextract _cfinal) =
18062030 Refold step inject extract
18072031
18082032 where
@@ -1853,7 +2077,7 @@ refoldMany1 (Refold sstep sinject sextract) (Fold cstep cinitial cextract) =
18532077{-# INLINE refold #-}
18542078refold :: Monad m => Refold m b a c -> Fold m a b -> Fold m a c
18552079refold (Refold step inject extract) f =
1856− Fold step (extractM f >>= inject) extract
2080+ Fold step (extractM f >>= inject) extract extract
18572081
18582082------------------------------------------------------------------------------
18592083-- morphInner
@@ -1863,8 +2087,8 @@ refold (Refold step inject extract) f =
18632087--
18642088-- /Pre-release/
18652089morphInner :: (forall x. m x -> n x) -> Fold m a b -> Fold n a b
1866−morphInner f (Fold step initial extract) =
1867− Fold (\x a -> f $ step x a) (f initial) (f . extract)
2090+morphInner f (Fold step initial extract final) =
2091+ Fold (\x a -> f $ step x a) (f initial) (f . extract) (f . final)
18682092
18692093-- | Adapt a pure fold to any monad.
18702094--
core/src/Streamly/Internal/Data/Fold/Window.hs+10−4
@@ -65,6 +65,8 @@ import Streamly.Internal.Data.Fold.Type (Fold(..), Step(..))
6565import Streamly.Internal.Data.Tuple.Strict
6666 (Tuple'(..), Tuple3Fused' (Tuple3Fused'))
6767
68+import Foreign.ForeignPtr.Unsafe (unsafeForeignPtrToPtr)
69+
6870import qualified Streamly.Internal.Data.Fold.Type as Fold
6971import qualified Streamly.Internal.Data.Ring as Ring
7072
@@ -148,7 +150,7 @@ windowRollingMap f = Fold.foldl' f1 initial
148150--
149151{-# INLINE windowSumInt #-}
150152windowSumInt :: forall m a. (Monad m, Integral a) => Fold m (a, Maybe a) a
151−windowSumInt = Fold step initial extract
153+windowSumInt = Fold step initial extract extract
152154
153155 where
154156
@@ -182,7 +184,7 @@ windowSumInt = Fold step initial extract
182184--
183185{-# INLINE windowSum #-}
184186windowSum :: forall m a. (Monad m, Num a) => Fold m (a, Maybe a) a
185−windowSum = Fold step initial extract
187+windowSum = Fold step initial extract extract
186188
187189 where
188190
@@ -267,7 +269,7 @@ windowPowerSumFrac p = windowLmap (** p) windowSum
267269--
268270{-# INLINE windowRange #-}
269271windowRange :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe (a, a))
270−windowRange n = Fold step initial extract
272+windowRange n = Fold step initial extract extract
271273
272274 where
273275
@@ -300,7 +302,11 @@ windowRange n = Fold step initial extract
300302 if i == 0
301303 then return Nothing
302304 else do
303− x <- liftIO $ peek rh
305+ -- Here we use "ringStart" over "ringHead" as "ringHead" will be
306+ -- uninitialized if the ring is not full.
307+ -- Using "unsafeForeignPtrToPtr" here is safe as we touch the ring
308+ -- again in "foldFunc".
309+ x <- liftIO $ peek (unsafeForeignPtrToPtr (Ring.ringStart rb))
304310 let accum (mn, mx) a = return (min mn a, max mx a)
305311 fmap Just $ foldFunc i rh accum (x, x) rb
306312
core/src/Streamly/Internal/Data/IORef/Unboxed.hs+22−19
@@ -23,34 +23,31 @@ module Streamly.Internal.Data.IORef.Unboxed
2323 (
2424 IORef
2525
26− -- * Construction
26+ -- Construction
2727 , newIORef
2828
29− -- * Write
29+ -- Write
3030 , writeIORef
3131 , modifyIORef'
3232
33− -- * Read
33+ -- Read
3434 , readIORef
35− , toStreamD
35+ , pollIntIORef
3636 )
3737where
3838
3939#include "inline.hs"
4040
41−import Data.Proxy (Proxy(..))
4241import Control.Monad.IO.Class (MonadIO(..))
43−import Streamly.Internal.Data.Unbox
44− ( MutableByteArray(..)
45− , Unbox(..)
46− , newBytes
47− , sizeOf
48− )
42+import Data.Proxy (Proxy(..))
43+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
44+import Streamly.Internal.Data.Unbox (Unbox(..), sizeOf)
4945
46+import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
5047import qualified Streamly.Internal.Data.Stream.Type as D
5148
5249-- | An 'IORef' holds a single 'Unbox'-able value.
53−newtype IORef a = IORef MutableByteArray
50+newtype IORef a = IORef MutByteArray
5451
5552-- | Create a new 'IORef'.
5653--
@@ -58,8 +55,8 @@ newtype IORef a = IORef MutableByteArray
5855{-# INLINE newIORef #-}
5956newIORef :: forall a. Unbox a => a -> IO (IORef a)
6057newIORef x = do
61− var <- newBytes (sizeOf (Proxy :: Proxy a))
62− pokeByteIndex 0 var x
58+ var <- MBA.new (sizeOf (Proxy :: Proxy a))
59+ pokeAt 0 var x
6360 return $ IORef var
6461
6562-- | Write a value to an 'IORef'.
@@ -67,14 +64,14 @@ newIORef x = do
6764-- /Pre-release/
6865{-# INLINE writeIORef #-}
6966writeIORef :: Unbox a => IORef a -> a -> IO ()
70−writeIORef (IORef var) = pokeByteIndex 0 var
67+writeIORef (IORef var) = pokeAt 0 var
7168
7269-- | Read a value from an 'IORef'.
7370--
7471-- /Pre-release/
7572{-# INLINE readIORef #-}
7673readIORef :: Unbox a => IORef a -> IO a
77−readIORef (IORef var) = peekByteIndex 0 var
74+readIORef (IORef var) = peekAt 0 var
7875
7976-- | Modify the value of an 'IORef' using a function with strict application.
8077--
@@ -87,10 +84,16 @@ modifyIORef' var g = do
8784
8885-- | Generate a stream by continuously reading the IORef.
8986--
87+-- This operation reads the IORef without any synchronization. It can be
88+-- assumed to be atomic because the IORef (MutableByteArray) is always aligned
89+-- to Int boundaries, we are assuming that compiler uses single instructions to
90+-- access the memory. It may read stale values though until caches are
91+-- synchronised in a multiprocessor architecture.
92+--
9093-- /Pre-release/
91−{-# INLINE_NORMAL toStreamD #-}
92−toStreamD :: (MonadIO m, Unbox a) => IORef a -> D.Stream m a
93−toStreamD var = D.Stream step ()
94+{-# INLINE_NORMAL pollIntIORef #-}
95+pollIntIORef :: (MonadIO m, Unbox a) => IORef a -> D.Stream m a
96+pollIntIORef var = D.Stream step ()
9497
9598 where
9699
core/src/Streamly/Internal/Data/IsMap.hs+8
@@ -1,3 +1,7 @@
1+{-# LANGUAGE TypeFamilies #-}
2+-- Must come after TypeFamilies, otherwise it is re-enabled.
3+-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
4+{-# LANGUAGE NoMonoLocalBinds #-}
15-- |
26-- Module : Streamly.Internal.Data.IsMap
37-- Copyright : (c) 2022 Composewell Technologies
@@ -28,6 +32,8 @@ class IsMap f where
2832 mapDelete :: Key f -> f a -> f a
2933 mapUnion :: f a -> f a -> f a
3034 mapNull :: f a -> Bool
35+ mapTraverseWithKey ::
36+ Applicative t => (Key f -> a -> t b) -> f a -> t (f b)
3137
3238instance Ord k => IsMap (Map k) where
3339 type Key (Map k) = k
@@ -39,6 +45,7 @@ instance Ord k => IsMap (Map k) where
3945 mapDelete = Map.delete
4046 mapUnion = Map.union
4147 mapNull = Map.null
48+ mapTraverseWithKey = Map.traverseWithKey
4249
4350instance IsMap IntMap.IntMap where
4451 type Key IntMap.IntMap = Int
@@ -50,3 +57,4 @@ instance IsMap IntMap.IntMap where
5057 mapDelete = IntMap.delete
5158 mapUnion = IntMap.union
5259 mapNull = IntMap.null
60+ mapTraverseWithKey = IntMap.traverseWithKey
core/src/Streamly/Internal/Data/MutArray.hs+154−33
@@ -8,79 +8,200 @@
88--
99module Streamly.Internal.Data.MutArray
1010 (
11+ -- * MutArray.Type module
1112 module Streamly.Internal.Data.MutArray.Type
12− , splitOn
13+ -- * MutArray module
14+ , sliceIndexerFromLen
15+ , slicerFromLen
16+ , compactLE
17+ , pinnedCompactLE
18+ , compactOnByte
19+ , compactOnByteSuffix
20+ -- * Unboxed IORef
21+ , module Streamly.Internal.Data.IORef.Unboxed
22+
23+ -- * Deprecated
1324 , genSlicesFromLen
1425 , getSlicesFromLen
15− , fromStream
1626 )
1727where
1828
1929#include "inline.hs"
2030
2131import Control.Monad.IO.Class (MonadIO(..))
32+import Data.Word (Word8)
33+import Streamly.Internal.Data.MutByteArray.Type (PinnedState(..))
34+import Streamly.Internal.Data.Stream.Type (Stream)
2235import Streamly.Internal.Data.Unbox (Unbox)
23−import Streamly.Internal.Data.Stream (Stream)
2436import Streamly.Internal.Data.Unfold.Type (Unfold(..))
2537
26−import qualified Streamly.Internal.Data.Stream as D
38+import qualified Streamly.Internal.Data.Stream.Type as Stream
39+import qualified Streamly.Internal.Data.Stream.Nesting as Stream
40+-- import qualified Streamly.Internal.Data.Stream.Transform as Stream
2741import qualified Streamly.Internal.Data.Unfold as Unfold
2842
2943import Prelude hiding (foldr, length, read, splitAt)
3044import Streamly.Internal.Data.MutArray.Type
31−
32−-- | Split the array into a stream of slices using a predicate. The element
33−-- matching the predicate is dropped.
34−--
35−-- /Pre-release/
36−{-# INLINE splitOn #-}
37−splitOn :: (MonadIO m, Unbox a) =>
38− (a -> Bool) -> MutArray a -> Stream m (MutArray a)
39−splitOn predicate arr =
40− fmap (\(i, len) -> getSliceUnsafe i len arr)
41− $ D.sliceOnSuffix predicate (read arr)
45+import Streamly.Internal.Data.IORef.Unboxed
4246
4347-- | Generate a stream of array slice descriptors ((index, len)) of specified
4448-- length from an array, starting from the supplied array index. The last slice
4549-- may be shorter than the requested length depending on the array length.
4650--
4751-- /Pre-release/
48−{-# INLINE genSlicesFromLen #-}
49−genSlicesFromLen :: forall m a. (Monad m, Unbox a)
52+{-# INLINE sliceIndexerFromLen #-}
53+sliceIndexerFromLen :: forall m a. (Monad m, Unbox a)
5054 => Int -- ^ from index
5155 -> Int -- ^ length of the slice
5256 -> Unfold m (MutArray a) (Int, Int)
53−genSlicesFromLen from len =
57+sliceIndexerFromLen from len =
5458 let fromThenTo n = (from, from + len, n - 1)
5559 mkSlice n i = return (i, min len (n - i))
5660 in Unfold.lmap length
5761 $ Unfold.mapM2 mkSlice
5862 $ Unfold.lmap fromThenTo Unfold.enumerateFromThenTo
5963
64+{-# DEPRECATED genSlicesFromLen "Please use sliceIndexerFromLen instead." #-}
65+genSlicesFromLen :: forall m a. (Monad m, Unbox a)
66+ => Int -- ^ from index
67+ -> Int -- ^ length of the slice
68+ -> Unfold m (MutArray a) (Int, Int)
69+genSlicesFromLen = sliceIndexerFromLen
70+
6071-- | Generate a stream of slices of specified length from an array, starting
6172-- from the supplied array index. The last slice may be shorter than the
6273-- requested length depending on the array length.
6374--
6475-- /Pre-release/
65−{-# INLINE getSlicesFromLen #-}
66−getSlicesFromLen :: forall m a. (Monad m, Unbox a)
76+{-# INLINE slicerFromLen #-}
77+slicerFromLen :: forall m a. (Monad m, Unbox a)
6778 => Int -- ^ from index
6879 -> Int -- ^ length of the slice
6980 -> Unfold m (MutArray a) (MutArray a)
70−getSlicesFromLen from len =
81+slicerFromLen from len =
7182 let mkSlice arr (i, n) = return $ getSliceUnsafe i n arr
72− in Unfold.mapM2 mkSlice (genSlicesFromLen from len)
83+ in Unfold.mapM2 mkSlice (sliceIndexerFromLen from len)
84+
85+{-# DEPRECATED getSlicesFromLen "Please use slicerFromLen instead." #-}
86+getSlicesFromLen :: forall m a. (Monad m, Unbox a)
87+ => Int -- ^ from index
88+ -> Int -- ^ length of the slice
89+ -> Unfold m (MutArray a) (MutArray a)
90+getSlicesFromLen = slicerFromLen
7391
74−-- | Create an 'Array' from a stream. This is useful when we want to create a
75−-- single array from a stream of unknown size. 'writeN' is at least twice
76−-- as efficient when the size is already known.
92+-------------------------------------------------------------------------------
93+-- Compacting Streams of Arrays
94+-------------------------------------------------------------------------------
95+
96+-- | @compactLE maxElems@ coalesces adjacent arrays in the input stream
97+-- only if the combined size would be less than or equal to @maxElems@
98+-- elements. Note that it won't split an array if the original array is already
99+-- larger than maxElems.
77100--
78−-- Note that if the input stream is too large memory allocation for the array
79−-- may fail. When the stream size is not known, `chunksOf` followed by
80−-- processing of indvidual arrays in the resulting stream should be preferred.
101+-- @maxElems@ must be greater than 0.
81102--
82−-- /Pre-release/
83−{-# INLINE fromStream #-}
84−fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
85−fromStream = fromStreamD
86−-- fromStream (Stream m) = P.fold write m
103+-- Generates unpinned arrays irrespective of the pinning status of input
104+-- arrays.
105+{-# INLINE compactLE #-}
106+compactLE :: (MonadIO m, Unbox a) =>
107+ Int -> Stream m (MutArray a) -> Stream m (MutArray a)
108+-- XXX compactLE can be moved to MutArray/Type if we are not using the parser
109+-- to implement it.
110+compactLE = compactLeAs Unpinned
111+-- The parser version turns out to be a little bit slower.
112+-- compactLE n = Stream.catRights . Stream.parseManyD (pCompactLE n)
113+
114+-- | Pinned version of 'compactLE'.
115+{-# INLINE pinnedCompactLE #-}
116+pinnedCompactLE :: forall m a. (MonadIO m, Unbox a)
117+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
118+pinnedCompactLE = compactLeAs Pinned
119+-- pinnedCompactLE n = Stream.catRights . Stream.parseManyD (pPinnedCompactLE n)
120+
121+data SplitState s arr
122+ = Initial s
123+ | Buffering s arr
124+ | Splitting s arr
125+ | Yielding arr (SplitState s arr)
126+ | Finishing
127+
128+-- | Split a stream of arrays on a given separator byte, dropping the separator
129+-- and coalescing all the arrays between two separators into a single array.
130+--
131+{-# INLINE_NORMAL _compactOnByteCustom #-}
132+_compactOnByteCustom
133+ :: MonadIO m
134+ => Word8
135+ -> Stream m (MutArray Word8)
136+ -> Stream m (MutArray Word8)
137+_compactOnByteCustom byte (Stream.Stream step state) =
138+ Stream.Stream step' (Initial state)
139+
140+ where
141+
142+ {-# INLINE_LATE step' #-}
143+ step' gst (Initial st) = do
144+ r <- step gst st
145+ case r of
146+ Stream.Yield arr s -> do
147+ (arr1, marr2) <- breakOn byte arr
148+ return $ case marr2 of
149+ Nothing -> Stream.Skip (Buffering s arr1)
150+ Just arr2 -> Stream.Skip (Yielding arr1 (Splitting s arr2))
151+ Stream.Skip s -> return $ Stream.Skip (Initial s)
152+ Stream.Stop -> return Stream.Stop
153+
154+ step' gst (Buffering st buf) = do
155+ r <- step gst st
156+ case r of
157+ Stream.Yield arr s -> do
158+ (arr1, marr2) <- breakOn byte arr
159+ -- XXX Use spliceExp instead and then rightSize?
160+ buf1 <- splice buf arr1
161+ return $ case marr2 of
162+ Nothing -> Stream.Skip (Buffering s buf1)
163+ Just x -> Stream.Skip (Yielding buf1 (Splitting s x))
164+ Stream.Skip s -> return $ Stream.Skip (Buffering s buf)
165+ Stream.Stop -> return $
166+ if byteLength buf == 0
167+ then Stream.Stop
168+ else Stream.Skip (Yielding buf Finishing)
169+
170+ step' _ (Splitting st buf) = do
171+ (arr1, marr2) <- breakOn byte buf
172+ return $ case marr2 of
173+ Nothing -> Stream.Skip $ Buffering st arr1
174+ Just arr2 -> Stream.Skip $ Yielding arr1 (Splitting st arr2)
175+
176+ step' _ (Yielding arr next) = return $ Stream.Yield arr next
177+ step' _ Finishing = return Stream.Stop
178+
179+-- XXX implement predicate based version of this
180+-- XXX Naming of predicate based vs custom version
181+
182+-- | Split a stream of arrays on a given separator byte, dropping the separator
183+-- and coalescing all the arrays between two separators into a single array.
184+--
185+{-# INLINE compactOnByte #-}
186+compactOnByte
187+ :: (MonadIO m)
188+ => Word8
189+ -> Stream m (MutArray Word8)
190+ -> Stream m (MutArray Word8)
191+-- XXX compare perf of custom vs idiomatic version
192+-- compactOnByte = _compactOnByteCustom
193+-- XXX use spliceExp and rightSize?
194+compactOnByte byte = Stream.splitInnerBy (breakOn byte) splice
195+
196+-- | Like 'compactOnByte' considers the separator in suffix position instead of
197+-- infix position.
198+{-# INLINE compactOnByteSuffix #-}
199+compactOnByteSuffix
200+ :: (MonadIO m)
201+ => Word8
202+ -> Stream m (MutArray Word8)
203+ -> Stream m (MutArray Word8)
204+compactOnByteSuffix byte =
205+ -- XXX use spliceExp and rightSize?
206+ Stream.splitInnerBySuffix
207+ (\arr -> byteLength arr == 0) (breakOn byte) splice
core/src/Streamly/Internal/Data/MutArray/Generic.hs+65−30
@@ -19,14 +19,14 @@ module Streamly.Internal.Data.MutArray.Generic
1919 , nil
2020
2121 -- *** Uninitialized Arrays
22− , new
22+ , emptyOf
2323 -- , newArrayWith
2424
2525 -- *** From streams
26− , writeNUnsafe
27− , writeN
28− , writeWith
29− , write
26+ , unsafeCreateOf
27+ , createOf
28+ , createWith -- createOfMin/createMin/createGE?
29+ , create
3030 , fromStreamN
3131 , fromStream
3232 , fromPureStream
@@ -160,6 +160,13 @@ module Streamly.Internal.Data.MutArray.Generic
160160 -- , appendSliceFrom
161161
162162 , clone
163+
164+ -- * Deprecated
165+ , new
166+ , writeNUnsafe
167+ , writeN
168+ , writeWith
169+ , write
163170 )
164171where
165172
@@ -236,13 +243,13 @@ bottomElement =
236243-- XXX Would be nice if GHC can provide something like newUninitializedArray# so
237244-- that we do not have to write undefined or error in the whole array.
238245
239−-- | @new count@ allocates a zero length array that can be extended to hold
246+-- | @emptyOf count@ allocates a zero length array that can be extended to hold
240247-- up to 'count' items without reallocating.
241248--
242249-- /Pre-release/
243−{-# INLINE new #-}
244−new :: MonadIO m => Int -> m (MutArray a)
245−new n@(I# n#) =
250+{-# INLINE emptyOf #-}
251+emptyOf :: MonadIO m => Int -> m (MutArray a)
252+emptyOf n@(I# n#) =
246253 liftIO
247254 $ IO
248255 $ \s# ->
@@ -251,6 +258,12 @@ new n@(I# n#) =
251258 let ma = MutArray arr# 0 0 n
252259 in (# s1#, ma #)
253260
261+-- XXX Deprecate in major
262+-- {-# DEPRECATED new "Please use emptyOf instead." #-}
263+{-# INLINE new #-}
264+new :: MonadIO m => Int -> m (MutArray a)
265+new = emptyOf
266+
254267-- XXX This could be pure?
255268
256269-- |
@@ -318,7 +331,7 @@ putIndices arr = FL.foldlM' step (return ())
318331
319332 where
320333
321− step () (i, x) = liftIO (putIndex i arr x)
334+ step () (i, x) = putIndex i arr x
322335
323336-- | Modify a given index of an array using a modifier function without checking
324337-- the bounds.
@@ -589,16 +602,16 @@ readRev arr@MutArray{..} =
589602arrayChunkSize :: Int
590603arrayChunkSize = 1024
591604
592−-- | Like 'writeN' but does not check the array bounds when writing. The fold
605+-- | Like 'createOf' but does not check the array bounds when writing. The fold
593606-- driver must not call the step function more than 'n' times otherwise it will
594607-- corrupt the memory and crash. This function exists mainly because any
595608-- conditional in the step function blocks fusion causing 10x performance
596609-- slowdown.
597610--
598611-- /Pre-release/
599−{-# INLINE_NORMAL writeNUnsafe #-}
600−writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
601−writeNUnsafe n = Fold step initial return
612+{-# INLINE_NORMAL unsafeCreateOf #-}
613+unsafeCreateOf :: MonadIO m => Int -> Fold m a (MutArray a)
614+unsafeCreateOf n = Fold step initial return return
602615
603616 where
604617
@@ -606,59 +619,81 @@ writeNUnsafe n = Fold step initial return
606619
607620 step arr x = FL.Partial <$> snocUnsafe arr x
608621
609−-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
622+{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
623+{-# INLINE writeNUnsafe #-}
624+writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
625+writeNUnsafe = unsafeCreateOf
626+
627+-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
610628-- 'Array'.
611629--
612−-- >>> writeN n = Fold.take n (MutArray.writeNUnsafe n)
630+-- >>> createOf n = Fold.take n (MutArray.unsafeCreateOf n)
613631--
614632-- /Pre-release/
615−{-# INLINE_NORMAL writeN #-}
633+{-# INLINE_NORMAL createOf #-}
634+createOf :: MonadIO m => Int -> Fold m a (MutArray a)
635+createOf n = FL.take n $ unsafeCreateOf n
636+
637+-- XXX Deprecate in major
638+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
639+{-# INLINE writeN #-}
616640writeN :: MonadIO m => Int -> Fold m a (MutArray a)
617−writeN n = FL.take n $ writeNUnsafe n
641+writeN = createOf
618642
619643-- >>> f n = MutArray.writeAppendWith (* 2) (MutArray.pinnedNew n)
620644-- >>> writeWith n = Fold.rmapM MutArray.rightSize (f n)
621645-- >>> writeWith n = Fold.rmapM MutArray.fromArrayStreamK (MutArray.writeChunks n)
622646
623−-- | @writeWith minCount@ folds the whole input to a single array. The array
647+-- | @createWith minCount@ folds the whole input to a single array. The array
624648-- starts at a size big enough to hold minCount elements, the size is doubled
625649-- every time the array needs to be grown.
626650--
627651-- /Caution! Do not use this on infinite streams./
628652--
629653-- /Pre-release/
630−{-# INLINE_NORMAL writeWith #-}
631−writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
654+{-# INLINE_NORMAL createWith #-}
655+createWith :: MonadIO m => Int -> Fold m a (MutArray a)
632656-- writeWith n = FL.rmapM rightSize $ writeAppendWith (* 2) (pinnedNew n)
633−writeWith elemCount = FL.rmapM extract $ FL.foldlM' step initial
657+createWith elemCount = FL.rmapM extract $ FL.foldlM' step initial
634658
635659 where
636660
637661 initial = do
638− when (elemCount < 0) $ error "writeWith: elemCount is negative"
639− liftIO $ new elemCount
662+ when (elemCount < 0) $ error "createWith: elemCount is negative"
663+ new elemCount
640664
641665 step arr@(MutArray _ start end bound) x
642666 | end == bound = do
643667 let oldSize = end - start
644668 newSize = max (oldSize * 2) 1
645− arr1 <- liftIO $ realloc newSize arr
669+ arr1 <- realloc newSize arr
646670 snocUnsafe arr1 x
647671 step arr x = snocUnsafe arr x
648672
649− -- extract = liftIO . rightSize
673+ -- extract = rightSize
650674 extract = return
651675
676+{-# DEPRECATED writeWith "Please use createWith instead." #-}
677+{-# INLINE writeWith #-}
678+writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
679+writeWith = createWith
680+
652681-- | Fold the whole input to a single array.
653682--
654−-- Same as 'writeWith' using an initial array size of 'arrayChunkSize' bytes
683+-- Same as 'createWith' using an initial array size of 'arrayChunkSize' bytes
655684-- rounded up to the element size.
656685--
657686-- /Caution! Do not use this on infinite streams./
658687--
688+{-# INLINE create #-}
689+create :: MonadIO m => Fold m a (MutArray a)
690+create = writeWith arrayChunkSize
691+
692+-- XXX Deprecate in major
693+-- {-# DEPRECATED write "Please use create instead." #-}
659694{-# INLINE write #-}
660695write :: MonadIO m => Fold m a (MutArray a)
661−write = writeWith arrayChunkSize
696+write = create
662697
663698-- | Create a 'MutArray' from the first @n@ elements of a stream. The
664699-- array is allocated to size @n@, if the stream terminates before @n@
@@ -683,7 +718,7 @@ fromList xs = fromStream $ D.fromList xs
683718{-# INLINABLE fromPureStream #-}
684719fromPureStream :: MonadIO m => Stream Identity a -> m (MutArray a)
685720fromPureStream xs =
686− liftIO $ D.fold write $ D.morphInner (return . runIdentity) xs
721+ D.fold write $ D.morphInner (return . runIdentity) xs
687722
688723-------------------------------------------------------------------------------
689724-- Chunking
@@ -815,7 +850,7 @@ putSliceUnsafe src srcStart dst dstStart len = liftIO $ do
815850
816851{-# INLINE clone #-}
817852clone :: MonadIO m => MutArray a -> m (MutArray a)
818−clone src = liftIO $ do
853+clone src = do
819854 let len = arrLen src
820855 dst <- new len
821856 putSliceUnsafe src 0 dst 0 len
core/src/Streamly/Internal/Data/MutArray/Stream.hsnew file+145
@@ -0,0 +1,145 @@
1+-- |
2+-- Module : Streamly.Internal.Data.MutArray.Stream
3+-- Copyright : (c) 2019 Composewell Technologies
4+-- License : BSD3-3-Clause
5+-- Maintainer : streamly@composewell.com
6+-- Stability : experimental
7+-- Portability : GHC
8+--
9+-- Combinators to efficiently manipulate streams of mutable arrays.
10+--
11+-- We can either push these in the MutArray module with a "chunks" prefix or
12+-- keep this as a separate module and release it.
13+--
14+module Streamly.Internal.Data.MutArray.Stream
15+{-# DEPRECATED "Please use \"Streamly.Internal.Data.MutArray\" instead." #-}
16+ (
17+ -- * Generation
18+ MArray.chunksOf
19+ , MArray.pinnedChunksOf
20+ , MArray.writeChunks -- chunksWrite?
21+ , MArray.splitOn -- chunksSplitOn
22+
23+ -- * Compaction
24+ , packArraysChunksOf
25+ , MArray.SpliceState (..)
26+ , lpackArraysChunksOf
27+ , compact -- chunksCompact
28+ , compactLE
29+ , compactEQ
30+ , compactGE
31+
32+ -- * Elimination
33+ , MArray.flattenArrays -- chunksConcat
34+ , MArray.flattenArraysRev -- chunksConcatRev
35+ , MArray.fromArrayStreamK -- chunksCoalesce
36+ )
37+where
38+
39+import Control.Monad.IO.Class (MonadIO(..))
40+import Streamly.Internal.Data.Unbox (Unbox)
41+import Streamly.Internal.Data.MutArray.Type (MutArray(..))
42+import Streamly.Internal.Data.Fold.Type (Fold(..))
43+import Streamly.Internal.Data.Parser (ParseError)
44+import Streamly.Internal.Data.Stream.Type (Stream)
45+
46+import qualified Streamly.Internal.Data.MutArray as MArray
47+import qualified Streamly.Internal.Data.Fold.Type as FL
48+import qualified Streamly.Internal.Data.Parser as ParserD
49+import qualified Streamly.Internal.Data.Stream as D
50+
51+-------------------------------------------------------------------------------
52+-- Compact
53+-------------------------------------------------------------------------------
54+
55+-- XXX This can be removed once compactLEFold/compactLE are implemented.
56+--
57+-- | This mutates the first array (if it has space) to append values from the
58+-- second one. This would work for immutable arrays as well because an
59+-- immutable array never has space so a new array is allocated instead of
60+-- mutating it.
61+--
62+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
63+-- maximum specified size. Note that if a single array is bigger than the
64+-- specified size we do not split it to fit. When we coalesce multiple arrays
65+-- if the size would exceed the specified size we do not coalesce therefore the
66+-- actual array size may be less than the specified chunk size.
67+--
68+-- @since 0.7.0
69+{-# INLINE packArraysChunksOf #-}
70+packArraysChunksOf :: (MonadIO m, Unbox a)
71+ => Int -> D.Stream m (MutArray a) -> D.Stream m (MutArray a)
72+packArraysChunksOf = MArray.compactLE
73+
74+-- XXX Remove this once compactLEFold is implemented
75+-- lpackArraysChunksOf = Fold.many compactLEFold
76+--
77+{-# INLINE lpackArraysChunksOf #-}
78+lpackArraysChunksOf :: (MonadIO m, Unbox a)
79+ => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
80+lpackArraysChunksOf = MArray.lCompactGE
81+
82+-- XXX Same as compactLE, to be removed once that is implemented.
83+--
84+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
85+-- maximum specified size in bytes.
86+--
87+-- /Internal/
88+{-# INLINE compact #-}
89+compact :: (MonadIO m, Unbox a)
90+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
91+compact = packArraysChunksOf
92+
93+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
94+-- maximum specified size. Note that if a single array is bigger than the
95+-- specified size we do not split it to fit. When we coalesce multiple arrays
96+-- if the size would exceed the specified size we do not coalesce therefore the
97+-- actual array size may be less than the specified chunk size.
98+--
99+-- /Internal/
100+{-# INLINE compactLEParserD #-}
101+compactLEParserD ::
102+ forall m a. (MonadIO m, Unbox a)
103+ => Int -> ParserD.Parser (MutArray a) m (MutArray a)
104+compactLEParserD = MArray.pCompactLE
105+
106+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
107+-- minimum specified size. Note that if all the arrays in the stream together
108+-- are smaller than the specified size the resulting array will be smaller than
109+-- the specified size. When we coalesce multiple arrays if the size would exceed
110+-- the specified size we stop coalescing further.
111+--
112+-- /Internal/
113+{-# INLINE compactGEFold #-}
114+compactGEFold ::
115+ forall m a. (MonadIO m, Unbox a)
116+ => Int -> FL.Fold m (MutArray a) (MutArray a)
117+compactGEFold = MArray.fCompactGE
118+
119+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
120+-- maximum specified size in bytes.
121+--
122+-- /Internal/
123+compactLE :: (MonadIO m, Unbox a) =>
124+ Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
125+compactLE n = D.parseManyD (compactLEParserD n)
126+
127+-- | Like 'compactLE' but generates arrays of exactly equal to the size
128+-- specified except for the last array in the stream which could be shorter.
129+--
130+-- /Unimplemented/
131+{-# INLINE compactEQ #-}
132+compactEQ :: -- (MonadIO m, Unbox a) =>
133+ Int -> Stream m (MutArray a) -> Stream m (MutArray a)
134+compactEQ _n _xs = undefined
135+ -- IsStream.fromStreamD $ D.foldMany (compactEQFold n) (IsStream.toStreamD xs)
136+
137+-- | Like 'compactLE' but generates arrays of size greater than or equal to the
138+-- specified except for the last array in the stream which could be shorter.
139+--
140+-- /Internal/
141+{-# INLINE compactGE #-}
142+compactGE ::
143+ (MonadIO m, Unbox a)
144+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
145+compactGE n = D.foldMany (compactGEFold n)
core/src/Streamly/Internal/Data/MutArray/Type.hs+1303−436
@@ -28,57 +28,80 @@
2828
2929module Streamly.Internal.Data.MutArray.Type
3030 (
31− -- * Type
31+ -- ** Type
3232 -- $arrayNotes
3333 MutArray (..)
34− , MutableByteArray
35− , touch
34+
35+ -- ** Conversion
36+ -- *** Pinned and Unpinned
3637 , pin
3738 , unpin
3839 , isPinned
3940
40− -- * Constructing and Writing
41− -- ** Construction
42− , nil
41+ -- ** Casting
42+ , cast
43+ , castUnsafe -- XXX unsafeCast
44+ , asBytes
45+ , unsafePinnedAsPtr
46+ , unsafeAsPtr
4347
44− -- *** Uninitialized Arrays
45− , pinnedNew
46− , pinnedNewBytes
47− , pinnedNewAligned
48− , new
49− , newArrayWith
48+ -- ** Construction
49+ , empty
50+
51+ -- *** New
52+ -- | New arrays are always empty arrays with some reserve capacity to
53+ -- extend the length without reallocating.
54+ , emptyOf
55+ , newArrayWith -- emptyAlignedWith
56+ , pinnedEmptyOf
57+ , pinnedNewAligned -- XXX not required
58+ -- , new -- uninitialized array of specified length
59+
60+ -- *** Cloning
61+ , clone
62+ , pinnedClone
63+
64+ -- *** Slicing
65+ -- | Get a subarray without copying
66+ , getSliceUnsafe -- XXX unsafeGetSlice
67+ , getSlice
68+ , splitAt -- XXX should be able to express using getSlice
69+ , breakOn
5070
51− -- *** From streams
71+ -- *** Stream Folds
5272 , ArrayUnsafe (..)
53− , writeNWithUnsafe
54− , writeNWith
55− , writeNUnsafe
56− , pinnedWriteNUnsafe
57− , writeN
58− , pinnedWriteN
59− , pinnedWriteNAligned
60−
61− , writeWith
62− , write
63− , pinnedWrite
64−
65− , writeRevN
66− -- , writeRev
67−
68− -- ** From containers
73+ , unsafeCreateOfWith
74+ , unsafeCreateOf
75+ , unsafePinnedCreateOf
76+ , pinnedCreateOf
77+ , createOfWith
78+ , createOf
79+ , revCreateOf
80+
81+ , pinnedCreate
82+ , createWith
83+ , create
84+ -- , revCreate
85+
86+ -- *** From containers
6987 , fromListN
7088 , pinnedFromListN
7189 , fromList
7290 , pinnedFromList
7391 , fromListRevN
7492 , fromListRev
75− , fromStreamDN
76− , fromStreamD
93+ , fromStreamN
94+ , fromStream
95+ , fromPureStreamN
7796 , fromPureStream
97+ , fromByteStr#
98+ , fromPtrN
99+ , fromChunksK
100+ , fromChunksRealloced -- fromSmallChunks
78101
79− -- * Random writes
102+ -- ** Random writes
80103 , putIndex
81− , putIndexUnsafe
104+ , putIndexUnsafe -- XXX unsafePutIndex
82105 , putIndices
83106 -- , putFromThenTo
84107 -- , putFrom -- start writing at the given position
@@ -86,78 +109,71 @@ module Streamly.Internal.Data.MutArray.Type
86109 -- , putFromTo
87110 -- , putFromRev
88111 -- , putUptoRev
89− , modifyIndexUnsafe
112+ , modifyIndexUnsafe -- XXX unsafeModifyIndex
90113 , modifyIndex
91114 , modifyIndices
92115 , modify
93116 , swapIndices
94117 , unsafeSwapIndices
95118
96− -- * Growing and Shrinking
97− -- Arrays grow only at the end, though it is possible to grow on both sides
98− -- and therefore have a cons as well as snoc. But that will require two
99− -- bounds in the array representation.
119+ -- ** Reading
100120
101− -- ** Appending elements
102− , snocWith
103− , snoc
104− , snocLinear
105− , snocMay
106− , snocUnsafe
107−
108− -- ** Appending streams
109− , writeAppendNUnsafe
110− , writeAppendN
111− , writeAppendWith
112− , writeAppend
113−
114− -- * Eliminating and Reading
115−
116− -- ** To streams
117− , reader
118− , readerRevWith
119− , readerRev
121+ -- *** Indexing
122+ , getIndex
123+ , getIndexUnsafe -- XXX unsafeGetIndex
124+ -- , getFromThenTo
125+ , getIndexRev -- getRevIndex?
126+ , indexReader
127+ , indexReaderWith
120128
121− -- ** To containers
122− , toStreamDWith
123− , toStreamDRevWith
124− , toStreamKWith
125− , toStreamKRevWith
129+ -- *** To Streams
126130 , read
127131 , readRev
132+ , toStreamWith
133+ , toStreamRevWith
128134 , toStreamK
135+ , toStreamKWith
129136 , toStreamKRev
137+ , toStreamKRevWith
138+
139+ -- *** To Containers
130140 , toList
131141
142+ -- *** Unfolds
132143 -- experimental
133144 , producerWith
134145 , producer
135146
136− -- ** Random reads
137− , getIndex
138− , getIndexUnsafe
139− , getIndices
140− , getIndicesD
141− -- , getFromThenTo
142− , getIndexRev
143−
144− -- * Memory Management
145− , blockSize
146− , arrayChunkBytes
147− , allocBytesToElemCount
148− , realloc
149− , resize
150− , resizeExp
151− , rightSize
147+ , reader
148+ , readerRevWith
149+ , readerRev
152150
153− -- * Size
151+ -- ** Size and Capacity
152+ -- *** Size
154153 , length
155154 , byteLength
155+
156+ -- *** Capacity
156157 -- , capacity
157158 , byteCapacity
158159 , bytesFree
159160
160− -- * In-place Mutation Algorithms
161+ -- *** Capacity Management
162+ , blockSize
163+ , arrayChunkBytes
164+ , allocBytesToElemCount
165+ , realloc -- this is "resize"
166+ , grow
167+ , growExp
168+ , rightSize
169+
170+ -- ** Folding
171+ , foldl'
172+ , foldr
173+ , byteCmp
174+ , byteEq
175+
176+ -- ** In-place Mutation Algorithms
161177 , strip
162178 , reverse
163179 , permute
@@ -167,59 +183,121 @@ module Streamly.Internal.Data.MutArray.Type
167183 , mergeBy
168184 , bubble
169185
170− -- * Casting
171− , cast
172− , castUnsafe
173− , asBytes
174− , asPtrUnsafe
175−
176− -- * Folding
177− , foldl'
178− , foldr
179− , cmp
180−
181− -- * Arrays of arrays
182− -- We can add dimensionality parameter to the array type to get
183− -- multidimensional arrays. Multidimensional arrays would just be a
184− -- convenience wrapper on top of single dimensional arrays.
185−
186− -- | Operations dealing with multiple arrays, streams of arrays or
187− -- multidimensional array representations.
186+ -- ** Growing and Shrinking
187+ -- | Arrays grow only at the end, though it is possible to grow on both
188+ -- sides and therefore have a cons as well as snoc. But that will require
189+ -- both lower and upper bound in the array representation.
188190
189− -- ** Construct from streams
190− , chunksOf
191− , pinnedChunksOf
192− , writeChunks
193−
194− -- ** Eliminate to streams
195− , flattenArrays
196− , flattenArraysRev
197− , fromArrayStreamK
191+ -- *** Appending elements
192+ , snocWith
193+ , snoc
194+ , snocLinear
195+ , snocMay
196+ , snocUnsafe -- XXX unsafeSnoc
198197
199− -- ** Construct from arrays
200− -- get chunks without copying
201− , getSliceUnsafe
202− , getSlice
203− -- , getSlicesFromLenN
204− , splitAt -- XXX should be able to express using getSlice
205− , breakOn
198+ -- *** Appending streams
199+ , unsafeAppendN
200+ , appendN
201+ , appendWith
202+ , append
206203
207− -- ** Appending arrays
204+ -- *** Appending arrays
208205 , spliceCopy
209206 , spliceWith
210207 , splice
211208 , spliceExp
212− , spliceUnsafe
213− , putSliceUnsafe
209+ , spliceUnsafe -- XXX unsafeSplice
214210 -- , putSlice
215211 -- , appendSlice
216212 -- , appendSliceFrom
217213
218− -- * Utilities
214+ -- ** Serialization using Unbox
215+ , pokeAppend
216+ , pokeAppendMay
217+ , pokeSkipUnsafe -- unsafePokeSkip
218+
219+ -- ** Deserialization using Unbox
220+ , peekUncons
221+ , peekUnconsUnsafe -- unsafePeekUncons
222+ , peekSkipUnsafe -- unsafePeekSkip
223+
224+ -- Arrays of arrays
225+ -- We can add dimensionality parameter to the array type to get
226+ -- multidimensional arrays. Multidimensional arrays would just be a
227+ -- convenience wrapper on top of single dimensional arrays.
228+
229+ -- ** Streams of Arrays
230+ -- *** Chunk
231+ -- | Group a stream into arrays.
232+ , chunksOf
233+ , pinnedChunksOf
234+ , buildChunks
235+
236+ -- *** Split
237+ -- | Split an array into slices.
238+
239+ -- , getSlicesFromLenN
240+ , splitOn
241+ -- , slicesOf
242+
243+ -- *** Concat
244+ -- | Append the arrays in a stream to form a stream of elements.
245+ , concatWith
246+ , concatRevWith
247+ , concat
248+ , concatRev
249+
250+ -- *** Compact
251+ -- | Append the arrays in a stream to form a stream of larger arrays.
252+ , SpliceState (..)
253+ , pCompactLE
254+ , pPinnedCompactLE
255+ , compactLeAs
256+ , fCompactGE
257+ , fPinnedCompactGE
258+ , lCompactGE
259+ , lPinnedCompactGE
260+ , compactGE
261+ , compactEQ
262+
263+ -- ** Utilities
219264 , roundUpToPower2
220265 , memcpy
221266 , memcmp
222267 , c_memchr
268+
269+ -- * Deprecated
270+ , asPtrUnsafe
271+ , writeChunks
272+ , flattenArrays
273+ , flattenArraysRev
274+ , fromArrayStreamK
275+ , fromStreamDN
276+ , fromStreamD
277+ , cmp
278+ , getIndices
279+ , getIndicesWith
280+ , resize
281+ , resizeExp
282+ , nil
283+ , new
284+ , pinnedNew
285+ , pinnedNewBytes
286+ , writeAppendNUnsafe
287+ , writeAppendN
288+ , writeAppendWith
289+ , writeAppend
290+ , writeNWithUnsafe
291+ , writeNWith
292+ , writeNUnsafe
293+ , pinnedWriteNUnsafe
294+ , writeN
295+ , pinnedWriteN
296+ , pinnedWriteNAligned -- XXX not required
297+ , writeWith
298+ , write
299+ , pinnedWrite
300+ , writeRevN
223301 )
224302where
225303
@@ -230,48 +308,52 @@ where
230308
231309import Control.Monad (when, void)
232310import Control.Monad.IO.Class (MonadIO(..))
311+import Data.Bifunctor (first)
233312import Data.Bits (shiftR, (.|.), (.&.))
234313import Data.Functor.Identity (Identity(..))
235314import Data.Proxy (Proxy(..))
236315import Data.Word (Word8)
237316import Foreign.C.Types (CSize(..), CInt(..))
238317import Foreign.Ptr (plusPtr, minusPtr, nullPtr)
239−import Streamly.Internal.Data.Unbox
240− ( MutableByteArray(..)
241− , Unbox(..)
318+import Streamly.Internal.Data.MutByteArray.Type
319+ ( MutByteArray(..)
242320 , PinnedState(..)
243321 , getMutableByteArray#
244− , touch
322+ , putSliceUnsafe
245323 )
324+import Streamly.Internal.Data.Unbox (Unbox(..))
246325import GHC.Base
247326 ( IO(..)
248327 , Int(..)
249− , byteArrayContents#
250328 , compareByteArrays#
251329 , copyMutableByteArray#
252330 )
253331import GHC.Base (noinline)
254−import GHC.Exts (unsafeCoerce#)
332+import GHC.Exts (unsafeCoerce#, Addr#)
255333import GHC.Ptr (Ptr(..))
256334
257335import Streamly.Internal.Data.Fold.Type (Fold(..))
258336import Streamly.Internal.Data.Producer.Type (Producer (..))
259337import Streamly.Internal.Data.Stream.Type (Stream)
338+import Streamly.Internal.Data.Parser.Type (Parser (..))
260339import Streamly.Internal.Data.StreamK.Type (StreamK)
261340import Streamly.Internal.Data.SVar.Type (adaptState, defState)
341+import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
262342import Streamly.Internal.Data.Unfold.Type (Unfold(..))
263343import Streamly.Internal.System.IO (arrayPayloadSize, defaultChunkSize)
264344
265345import qualified Streamly.Internal.Data.Fold.Type as FL
346+import qualified Streamly.Internal.Data.MutByteArray.Type as Unboxed
347+import qualified Streamly.Internal.Data.Parser.Type as Parser
348+-- import qualified Streamly.Internal.Data.Fold.Type as Fold
266349import qualified Streamly.Internal.Data.Producer as Producer
267350import qualified Streamly.Internal.Data.Stream.Type as D
268351import qualified Streamly.Internal.Data.Stream.Lift as D
269352import qualified Streamly.Internal.Data.StreamK.Type as K
270−import qualified Streamly.Internal.Data.Unbox as Unboxed
271353import qualified Prelude
272354
273355import Prelude hiding
274− (Foldable(..), read, unlines, splitAt, reverse, truncate)
356+ (Foldable(..), concat, read, unlines, splitAt, reverse, truncate)
275357
276358#include "DocTestDataMutArray.hs"
277359
@@ -279,6 +361,7 @@ import Prelude hiding
279361-- Foreign helpers
280362-------------------------------------------------------------------------------
281363
364+-- NOTE: Have to be "ccall unsafe" so that we can pass unpinned memory to these
282365foreign import ccall unsafe "string.h memcpy" c_memcpy
283366 :: Ptr Word8 -> Ptr Word8 -> CSize -> IO (Ptr Word8)
284367
@@ -288,6 +371,9 @@ foreign import ccall unsafe "string.h memchr" c_memchr
288371foreign import ccall unsafe "string.h memcmp" c_memcmp
289372 :: Ptr Word8 -> Ptr Word8 -> CSize -> IO CInt
290373
374+foreign import ccall unsafe "string.h strlen" c_strlen
375+ :: Ptr Word8 -> IO CSize
376+
291377-- | Given an 'Unboxed' type (unused first arg) and a number of bytes, return
292378-- how many elements of that type will completely fit in those bytes.
293379--
@@ -342,7 +428,7 @@ data MutArray a =
342428 -- The array is a range into arrContents. arrContents may be a superset of
343429 -- the slice represented by the array. All offsets are in bytes.
344430 MutArray
345− { arrContents :: {-# UNPACK #-} !MutableByteArray
431+ { arrContents :: {-# UNPACK #-} !MutByteArray
346432 , arrStart :: {-# UNPACK #-} !Int -- ^ index into arrContents
347433 , arrEnd :: {-# UNPACK #-} !Int -- ^ index into arrContents
348434 -- Represents the first invalid index of
@@ -354,22 +440,25 @@ data MutArray a =
354440-- Pinning & Unpinning
355441-------------------------------------------------------------------------------
356442
357−-- | Copy the 'MutArray' to pinned memory if unpinned, else do nothing. The
358−-- overhead is a copy if the input array is unpinned.
443+-- | Return a copy of the array in pinned memory if unpinned, else return the
444+-- original array.
359445{-# INLINE pin #-}
360446pin :: MutArray a -> IO (MutArray a)
361−pin arr@MutArray{..} = do
362− contents <- Unboxed.pin arrContents
363− return $ arr {arrContents = contents}
447+pin arr@MutArray{..} =
448+ if Unboxed.isPinned arrContents
449+ then pure arr
450+ else pinnedClone arr
364451
365−-- | Copy the 'MutArray' to unpinned memory if pinned, else do nothing. The
366−-- overhead is a copy if the input array is pinned.
452+-- | Return a copy of the array in unpinned memory if pinned, else return the
453+-- original array.
367454{-# INLINE unpin #-}
368455unpin :: MutArray a -> IO (MutArray a)
369−unpin arr@MutArray{..} = do
370− contents <- Unboxed.unpin arrContents
371− return $ arr {arrContents = contents}
456+unpin arr@MutArray{..} =
457+ if Unboxed.isPinned arrContents
458+ then clone arr
459+ else pure arr
372460
461+-- | Return 'True' if the array is allocated in pinned memory.
373462{-# INLINE isPinned #-}
374463isPinned :: MutArray a -> Bool
375464isPinned MutArray{..} = Unboxed.isPinned arrContents
@@ -390,6 +479,8 @@ isPinned MutArray{..} = Unboxed.isPinned arrContents
390479-- we need more than that. See stg_pinnedNewAlignedByteArrayzh and
391480-- allocatePinned in GHC source.
392481
482+-- XXX Rename to emptyAlignedWith, alignSize should be first arg.
483+
393484-- | @newArrayWith allocator alignment count@ allocates a new array of zero
394485-- length and with a capacity to hold @count@ elements, using @allocator
395486-- size alignment@ as the memory allocator function.
@@ -402,7 +493,7 @@ isPinned MutArray{..} = Unboxed.isPinned arrContents
402493-- /Pre-release/
403494{-# INLINE newArrayWith #-}
404495newArrayWith :: forall m a. (MonadIO m, Unbox a)
405− => (Int -> Int -> m MutableByteArray) -> Int -> Int -> m (MutArray a)
496+ => (Int -> Int -> m MutByteArray) -> Int -> Int -> m (MutArray a)
406497newArrayWith alloc alignSize count = do
407498 let size = max (count * SIZE_OF(a)) 0
408499 contents <- alloc size alignSize
@@ -413,12 +504,25 @@ newArrayWith alloc alignSize count = do
413504 , arrBound = size
414505 }
415506
507+-- For arrays "nil" sounds a bit odd. empty is better. The only problem with
508+-- empty is that it is also used by the Alternative type class. But assuming we
509+-- will mostly import the Array module qualified this should be fine.
510+
511+-- | Create an empty array.
512+empty ::
513+#ifdef DEVBUILD
514+ Unbox a =>
515+#endif
516+ MutArray a
517+empty = MutArray Unboxed.empty 0 0 0
518+
519+{-# DEPRECATED nil "Please use empty instead." #-}
416520nil ::
417521#ifdef DEVBUILD
418522 Unbox a =>
419523#endif
420524 MutArray a
421−nil = MutArray Unboxed.nil 0 0 0
525+nil = empty
422526
423527{-# INLINE newBytesAs #-}
424528newBytesAs :: MonadIO m =>
@@ -435,12 +539,13 @@ newBytesAs ps bytes = do
435539 , arrBound = bytes
436540 }
437541
438−-- | Allocates a pinned empty array that can hold 'count' items. The memory of
439−-- the array is uninitialized and the allocation is aligned as per the
440−-- 'Unboxed' instance of the type.
542+-- | Allocates a pinned empty array that with a reserved capacity of bytes.
543+-- The memory of the array is uninitialized and the allocation is aligned as
544+-- per the 'Unboxed' instance of the type.
441545--
442546-- /Pre-release/
443547{-# INLINE pinnedNewBytes #-}
548+{-# DEPRECATED pinnedNewBytes "Please use pinnedEmptyOf with appropriate calculation" #-}
444549pinnedNewBytes :: MonadIO m =>
445550#ifdef DEVBUILD
446551 Unbox a =>
@@ -465,21 +570,31 @@ newAs ps =
465570 (error "new: alignment is not used in unpinned arrays.")
466571
467572-- XXX can unaligned allocation be more efficient when alignment is not needed?
468−--
469−-- | Allocates an empty pinned array that can hold 'count' items. The memory of
470−-- the array is uninitialized and the allocation is aligned as per the 'Unboxed'
471−-- instance of the type.
472−--
573+
574+-- | Allocates a pinned array of zero length but growable to the specified
575+-- capacity without reallocation.
576+{-# INLINE pinnedEmptyOf #-}
577+pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
578+pinnedEmptyOf = newAs Pinned
579+
580+-- XXX Deprecate in major
581+-- {-# DEPRECATED pinnedNew "Please use pinnedEmptyOf instead." #-}
473582{-# INLINE pinnedNew #-}
474583pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
475−pinnedNew = newAs Pinned
584+pinnedNew = pinnedEmptyOf
476585
477−-- | Allocates an empty unpinned array that can hold 'count' items. The memory
478−-- of the array is uninitialized.
586+-- | Allocates an unpinned array of zero length but growable to the specified
587+-- capacity without reallocation.
479588--
589+{-# INLINE emptyOf #-}
590+emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
591+emptyOf = newAs Unpinned
592+
593+-- XXX Deprecate in major
594+-- {-# DEPRECATED new "Please use emptyOf instead." #-}
480595{-# INLINE new #-}
481596new :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
482−new = newAs Unpinned
597+new = emptyOf
483598
484599-------------------------------------------------------------------------------
485600-- Random writes
@@ -495,7 +610,7 @@ putIndexUnsafe :: forall m a. (MonadIO m, Unbox a)
495610putIndexUnsafe i MutArray{..} x = do
496611 let index = INDEX_OF(arrStart, i, a)
497612 assert (i >= 0 && INDEX_VALID(index, arrEnd, a)) (return ())
498− liftIO $ pokeByteIndex index arrContents x
613+ liftIO $ pokeAt index arrContents x
499614
500615invalidIndex :: String -> Int -> a
501616invalidIndex label i =
@@ -513,7 +628,7 @@ putIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
513628putIndex i MutArray{..} x = do
514629 let index = INDEX_OF(arrStart,i,a)
515630 if i >= 0 && INDEX_VALID(index,arrEnd,a)
516− then liftIO $ pokeByteIndex index arrContents x
631+ then liftIO $ pokeAt index arrContents x
517632 else invalidIndex "putIndex" i
518633
519634-- | Write an input stream of (index, value) pairs to an array. Throws an
@@ -527,7 +642,7 @@ putIndices arr = FL.foldlM' step (return ())
527642
528643 where
529644
530− step () (i, x) = liftIO (putIndex i arr x)
645+ step () (i, x) = putIndex i arr x
531646
532647-- | Modify a given index of an array using a modifier function.
533648--
@@ -539,9 +654,9 @@ modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) =>
539654modifyIndexUnsafe i MutArray{..} f = liftIO $ do
540655 let index = INDEX_OF(arrStart,i,a)
541656 assert (i >= 0 && INDEX_NEXT(index,a) <= arrEnd) (return ())
542− r <- peekByteIndex index arrContents
657+ r <- peekAt index arrContents
543658 let (x, res) = f r
544− pokeByteIndex index arrContents x
659+ pokeAt index arrContents x
545660 return res
546661
547662-- | Modify a given index of an array using a modifier function.
@@ -553,9 +668,9 @@ modifyIndex i MutArray{..} f = do
553668 let index = INDEX_OF(arrStart,i,a)
554669 if i >= 0 && INDEX_VALID(index,arrEnd,a)
555670 then liftIO $ do
556− r <- peekByteIndex index arrContents
671+ r <- peekAt index arrContents
557672 let (x, res) = f r
558− pokeByteIndex index arrContents x
673+ pokeAt index arrContents x
559674 return res
560675 else invalidIndex "modifyIndex" i
561676
@@ -589,8 +704,8 @@ modify MutArray{..} f = liftIO $
589704
590705 go i =
591706 when (INDEX_VALID(i,arrEnd,a)) $ do
592− r <- peekByteIndex i arrContents
593− pokeByteIndex i arrContents (f r)
707+ r <- peekAt i arrContents
708+ pokeAt i arrContents (f r)
594709 go (INDEX_NEXT(i,a))
595710
596711-- XXX We could specify the number of bytes to swap instead of Proxy. Need
@@ -599,15 +714,15 @@ modify MutArray{..} f = liftIO $
599714swapArrayByteIndices ::
600715 forall a. Unbox a
601716 => Proxy a
602− -> MutableByteArray
717+ -> MutByteArray
603718 -> Int
604719 -> Int
605720 -> IO ()
606721swapArrayByteIndices _ arrContents i1 i2 = do
607− r1 <- peekByteIndex i1 arrContents
608− r2 <- peekByteIndex i2 arrContents
609− pokeByteIndex i1 arrContents (r2 :: a)
610− pokeByteIndex i2 arrContents (r1 :: a)
722+ r1 <- peekAt i1 arrContents
723+ r2 <- peekAt i2 arrContents
724+ pokeAt i1 arrContents (r2 :: a)
725+ pokeAt i2 arrContents (r1 :: a)
611726
612727-- | Swap the elements at two indices without validating the indices.
613728--
@@ -723,26 +838,23 @@ arrayChunkBytes = 1024
723838roundDownTo :: Int -> Int -> Int
724839roundDownTo elemSize size = size - (size `mod` elemSize)
725840
726−-- XXX See if resizing can be implemented by reading the old array as a stream
727−-- and then using writeN to the new array.
728−--
729841-- NOTE: we are passing elemSize explicitly to avoid an Unboxed constraint.
730842-- Since this is not inlined Unboxed consrraint leads to dictionary passing
731843-- which complicates some inspection tests.
732844--
733−{-# NOINLINE reallocExplicit #-}
734−reallocExplicit :: Int -> Int -> MutArray a -> IO (MutArray a)
735−reallocExplicit elemSize newCapacityInBytes MutArray{..} = do
845+{-# NOINLINE reallocExplicitAs #-}
846+reallocExplicitAs :: PinnedState -> Int -> Int -> MutArray a -> IO (MutArray a)
847+reallocExplicitAs ps elemSize newCapacityInBytes MutArray{..} = do
736848 assertM(arrEnd <= arrBound)
737849
738850 -- Allocate new array
739851 let newCapMaxInBytes = roundUpLargeArray newCapacityInBytes
740852 contents <-
741− if Unboxed.isPinned arrContents
742− then Unboxed.pinnedNewBytes newCapMaxInBytes
743− else Unboxed.newBytes newCapMaxInBytes
744− let !(MutableByteArray mbarrFrom#) = arrContents
745− !(MutableByteArray mbarrTo#) = contents
853+ if ps == Pinned
854+ then Unboxed.pinnedNew newCapMaxInBytes
855+ else Unboxed.new newCapMaxInBytes
856+ let !(MutByteArray mbarrFrom#) = arrContents
857+ !(MutByteArray mbarrTo#) = contents
746858
747859 -- Copy old data
748860 let oldStart = arrStart
@@ -763,6 +875,10 @@ reallocExplicit elemSize newCapacityInBytes MutArray{..} = do
763875 , arrBound = newCapInBytes
764876 }
765877
878+-- XXX Should these be called reallocBytes etc?
879+-- XXX We may also need reallocAs to allocate as pinned/unpinned explicitly. In
880+-- fact clone/pinnedClone can be implemented using reallocAs.
881+
766882-- | @realloc newCapacity array@ reallocates the array to the specified
767883-- capacity in bytes.
768884--
@@ -774,7 +890,12 @@ reallocExplicit elemSize newCapacityInBytes MutArray{..} = do
774890-- If the original array is pinned, the newly allocated array is also pinned.
775891{-# INLINABLE realloc #-}
776892realloc :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
777−realloc bytes arr = liftIO $ reallocExplicit (SIZE_OF(a)) bytes arr
893+realloc bytes arr =
894+ let ps =
895+ if isPinned arr
896+ then Pinned
897+ else Unpinned
898+ in liftIO $ reallocExplicitAs ps (SIZE_OF(a)) bytes arr
778899
779900-- | @reallocWith label capSizer minIncrBytes array@. The label is used
780901-- in error messages and the capSizer is used to determine the capacity of the
@@ -797,14 +918,14 @@ reallocWith label capSizer minIncrBytes arr = do
797918 where
798919
799920 badSize newSize =
800− concat
921+ Prelude.concat
801922 [ label
802923 , ": new array size (in bytes) is less than required size "
803924 , show newSize
804925 , ". Please check the sizing function passed."
805926 ]
806927
807−-- | @resize newCapacity array@ changes the total capacity of the array so that
928+-- | @grow newCapacity array@ changes the total capacity of the array so that
808929-- it is enough to hold the specified number of elements. Nothing is done if
809930-- the specified capacity is less than the length of the array.
810931--
@@ -812,34 +933,46 @@ reallocWith label capSizer minIncrBytes arr = do
812933-- the block size (4K).
813934--
814935-- /Pre-release/
815−{-# INLINE resize #-}
816−resize :: forall m a. (MonadIO m, Unbox a) =>
936+{-# INLINE grow #-}
937+grow :: forall m a. (MonadIO m, Unbox a) =>
817938 Int -> MutArray a -> m (MutArray a)
818−resize nElems arr@MutArray{..} = do
939+grow nElems arr@MutArray{..} = do
819940 let req = SIZE_OF(a) * nElems
820− len = arrEnd - arrStart
821− if req < len
941+ cap = arrBound - arrStart
942+ if req < cap
822943 then return arr
823944 else realloc req arr
824945
825−-- | Like 'resize' but if the byte capacity is more than 'largeObjectThreshold'
826−-- then it is rounded up to the closest power of 2.
946+{-# DEPRECATED resize "Please use grow instead." #-}
947+{-# INLINE resize #-}
948+resize :: forall m a. (MonadIO m, Unbox a) =>
949+ Int -> MutArray a -> m (MutArray a)
950+resize = grow
951+
952+-- | Like 'grow' but if the requested byte capacity is more than
953+-- 'largeObjectThreshold' then it is rounded up to the closest power of 2.
827954--
828955-- /Pre-release/
829−{-# INLINE resizeExp #-}
830−resizeExp :: forall m a. (MonadIO m, Unbox a) =>
956+{-# INLINE growExp #-}
957+growExp :: forall m a. (MonadIO m, Unbox a) =>
831958 Int -> MutArray a -> m (MutArray a)
832−resizeExp nElems arr@MutArray{..} = do
959+growExp nElems arr@MutArray{..} = do
833960 let req = roundUpLargeArray (SIZE_OF(a) * nElems)
834961 req1 =
835962 if req > largeObjectThreshold
836963 then roundUpToPower2 req
837964 else req
838− len = arrEnd - arrStart
839− if req1 < len
965+ cap = arrBound - arrStart
966+ if req1 < cap
840967 then return arr
841968 else realloc req1 arr
842969
970+{-# DEPRECATED resizeExp "Please use growExp instead." #-}
971+{-# INLINE resizeExp #-}
972+resizeExp :: forall m a. (MonadIO m, Unbox a) =>
973+ Int -> MutArray a -> m (MutArray a)
974+resizeExp = growExp
975+
843976-- | Resize the allocated memory to drop any reserved free space at the end of
844977-- the array and reallocate it to reduce wastage.
845978--
@@ -884,7 +1017,7 @@ rightSize arr@MutArray{..} = do
8841017snocNewEnd :: (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m (MutArray a)
8851018snocNewEnd newEnd arr@MutArray{..} x = liftIO $ do
8861019 assert (newEnd <= arrBound) (return ())
887− pokeByteIndex arrEnd arrContents x
1020+ pokeAt arrEnd arrContents x
8881021 return $ arr {arrEnd = newEnd}
8891022
8901023-- | Really really unsafe, appends the element into the first array, may
@@ -904,7 +1037,7 @@ snocUnsafe arr@MutArray{..} = snocNewEnd (INDEX_NEXT(arrEnd,a)) arr
9041037{-# INLINE snocMay #-}
9051038snocMay :: forall m a. (MonadIO m, Unbox a) =>
9061039 MutArray a -> a -> m (Maybe (MutArray a))
907−snocMay arr@MutArray{..} x = liftIO $ do
1040+snocMay arr@MutArray{..} x = do
9081041 let newEnd = INDEX_NEXT(arrEnd,a)
9091042 if newEnd <= arrBound
9101043 then Just <$> snocNewEnd newEnd arr x
@@ -918,7 +1051,7 @@ snocWithRealloc :: forall m a. (MonadIO m, Unbox a) =>
9181051 -> a
9191052 -> m (MutArray a)
9201053snocWithRealloc sizer arr x = do
921− arr1 <- liftIO $ reallocWith "snocWith" sizer (SIZE_OF(a)) arr
1054+ arr1 <- reallocWith "snocWith" sizer (SIZE_OF(a)) arr
9221055 snocUnsafe arr1 x
9231056
9241057-- | @snocWith sizer arr elem@ mutates @arr@ to append @elem@. The length of
@@ -940,7 +1073,7 @@ snocWith :: forall m a. (MonadIO m, Unbox a) =>
9401073 -> MutArray a
9411074 -> a
9421075 -> m (MutArray a)
943−snocWith allocSize arr x = liftIO $ do
1076+snocWith allocSize arr x = do
9441077 let newEnd = INDEX_NEXT(arrEnd arr,a)
9451078 if newEnd <= arrBound arr
9461079 then snocNewEnd newEnd arr x
@@ -984,6 +1117,126 @@ snoc = snocWith f
9841117 then oldSize * 2
9851118 else roundUpToPower2 oldSize * 2
9861119
1120+-------------------------------------------------------------------------------
1121+-- Serialization/Deserialization
1122+-------------------------------------------------------------------------------
1123+
1124+{-# INLINE pokeNewEnd #-}
1125+pokeNewEnd :: (MonadIO m, Unbox a) =>
1126+ Int -> MutArray Word8 -> a -> m (MutArray Word8)
1127+pokeNewEnd newEnd arr@MutArray{..} x = liftIO $ do
1128+ assert (newEnd <= arrBound) (return ())
1129+ liftIO $ pokeAt arrEnd arrContents x
1130+ return $ arr {arrEnd = newEnd}
1131+
1132+-- | Really really unsafe, unboxes a Haskell type and appends the resulting
1133+-- bytes to the byte array, may cause silent data corruption or if you are
1134+-- lucky a segfault if the array does not have enough space to append the
1135+-- element.
1136+--
1137+-- /Internal/
1138+{-# INLINE pokeAppendUnsafe #-}
1139+pokeAppendUnsafe :: forall m a. (MonadIO m, Unbox a) =>
1140+ MutArray Word8 -> a -> m (MutArray Word8)
1141+pokeAppendUnsafe arr@MutArray{..} = pokeNewEnd (arrEnd + SIZE_OF(a)) arr
1142+
1143+-- | Skip the specified number of bytes in the array. The data in the skipped
1144+-- region remains uninitialzed.
1145+{-# INLINE pokeSkipUnsafe #-}
1146+pokeSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
1147+pokeSkipUnsafe n arr@MutArray{..} = do
1148+ let newEnd = arrEnd + n
1149+ in assert (newEnd <= arrBound) (arr {arrEnd = newEnd})
1150+
1151+-- | Like 'pokeAppend' but does not grow the array when pre-allocated array
1152+-- capacity becomes full.
1153+--
1154+-- /Internal/
1155+{-# INLINE pokeAppendMay #-}
1156+pokeAppendMay :: forall m a. (MonadIO m, Unbox a) =>
1157+ MutArray Word8 -> a -> m (Maybe (MutArray Word8))
1158+pokeAppendMay arr@MutArray{..} x = liftIO $ do
1159+ let newEnd = arrEnd + SIZE_OF(a)
1160+ if newEnd <= arrBound
1161+ then Just <$> pokeNewEnd newEnd arr x
1162+ else return Nothing
1163+
1164+{-# NOINLINE pokeWithRealloc #-}
1165+pokeWithRealloc :: forall m a. (MonadIO m, Unbox a) =>
1166+ (Int -> Int)
1167+ -> MutArray Word8
1168+ -> a
1169+ -> m (MutArray Word8)
1170+pokeWithRealloc sizer arr x = do
1171+ arr1 <- liftIO $ reallocWith "pokeWithRealloc" sizer (SIZE_OF(a)) arr
1172+ pokeAppendUnsafe arr1 x
1173+
1174+{-# INLINE pokeAppendWith #-}
1175+pokeAppendWith :: forall m a. (MonadIO m, Unbox a) =>
1176+ (Int -> Int)
1177+ -> MutArray Word8
1178+ -> a
1179+ -> m (MutArray Word8)
1180+pokeAppendWith allocSize arr x = liftIO $ do
1181+ let newEnd = arrEnd arr + SIZE_OF(a)
1182+ if newEnd <= arrBound arr
1183+ then pokeNewEnd newEnd arr x
1184+ else pokeWithRealloc allocSize arr x
1185+
1186+-- | Unbox a Haskell type and append the resulting bytes to a mutable byte
1187+-- array. The array is grown exponentially when more space is needed.
1188+--
1189+-- Definition:
1190+--
1191+-- >>> pokeAppend arr x = MutArray.castUnsafe <$> MutArray.snoc (MutArray.castUnsafe arr) x
1192+--
1193+{-# INLINE pokeAppend #-}
1194+pokeAppend :: forall m a. (MonadIO m, Unbox a) =>
1195+ MutArray Word8 -> a -> m (MutArray Word8)
1196+pokeAppend = pokeAppendWith f
1197+
1198+ where
1199+
1200+ f oldSize =
1201+ if isPower2 oldSize
1202+ then oldSize * 2
1203+ else roundUpToPower2 oldSize * 2
1204+
1205+-- | Really really unsafe, create a Haskell value from an unboxed byte array,
1206+-- does not check if the array is big enough, may return garbage or if you are
1207+-- lucky may cause a segfault.
1208+--
1209+-- /Internal/
1210+{-# INLINE peekUnconsUnsafe #-}
1211+peekUnconsUnsafe :: forall m a. (MonadIO m, Unbox a) =>
1212+ MutArray Word8 -> m (a, MutArray Word8)
1213+peekUnconsUnsafe MutArray{..} = do
1214+ let start1 = arrStart + SIZE_OF(a)
1215+ assert (start1 <= arrEnd) (return ())
1216+ liftIO $ do
1217+ r <- peekAt arrStart arrContents
1218+ return (r, MutArray arrContents start1 arrEnd arrBound)
1219+
1220+-- | Discard the specified number of bytes in the array.
1221+{-# INLINE peekSkipUnsafe #-}
1222+peekSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
1223+peekSkipUnsafe n MutArray{..} =
1224+ let start1 = arrStart + n
1225+ in assert (start1 <= arrEnd) (MutArray arrContents start1 arrEnd arrBound)
1226+
1227+-- | Create a Haskell value from its unboxed representation from the head of a
1228+-- byte array, return the value and the remaining array.
1229+{-# INLINE peekUncons #-}
1230+peekUncons :: forall m a. (MonadIO m, Unbox a) =>
1231+ MutArray Word8 -> m (Maybe a, MutArray Word8)
1232+peekUncons arr@MutArray{..} = do
1233+ let start1 = arrStart + SIZE_OF(a)
1234+ if start1 > arrEnd
1235+ then return (Nothing, arr)
1236+ else liftIO $ do
1237+ r <- peekAt arrStart arrContents
1238+ return (Just r, MutArray arrContents start1 arrEnd arrBound)
1239+
9871240-------------------------------------------------------------------------------
9881241-- Random reads
9891242-------------------------------------------------------------------------------
@@ -998,7 +1251,7 @@ getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
9981251getIndexUnsafe i MutArray{..} = do
9991252 let index = INDEX_OF(arrStart,i,a)
10001253 assert (i >= 0 && INDEX_VALID(index,arrEnd,a)) (return ())
1001− liftIO $ peekByteIndex index arrContents
1254+ liftIO $ peekAt index arrContents
10021255
10031256-- | /O(1)/ Lookup the element at the given index. Index starts from 0.
10041257--
@@ -1007,7 +1260,7 @@ getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
10071260getIndex i MutArray{..} = do
10081261 let index = INDEX_OF(arrStart,i,a)
10091262 if i >= 0 && INDEX_VALID(index,arrEnd,a)
1010− then liftIO $ Just <$> peekByteIndex index arrContents
1263+ then liftIO $ Just <$> peekAt index arrContents
10111264 else return Nothing
10121265
10131266-- | /O(1)/ Lookup the element at the given index from the end of the array.
@@ -1020,21 +1273,16 @@ getIndexRev :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
10201273getIndexRev i MutArray{..} = do
10211274 let index = RINDEX_OF(arrEnd,i,a)
10221275 if i >= 0 && index >= arrStart
1023− then liftIO $ peekByteIndex index arrContents
1276+ then liftIO $ peekAt index arrContents
10241277 else invalidIndex "getIndexRev" i
10251278
10261279data GetIndicesState contents start end st =
10271280 GetIndicesState contents start end st
10281281
1029−-- | Given an unfold that generates array indices, read the elements on those
1030−-- indices from the supplied MutArray. An error is thrown if an index is out of
1031−-- bounds.
1032−--
1033−-- /Pre-release/
1034−{-# INLINE getIndicesD #-}
1035−getIndicesD :: (Monad m, Unbox a) =>
1282+{-# INLINE indexReaderWith #-}
1283+indexReaderWith :: (Monad m, Unbox a) =>
10361284 (forall b. IO b -> m b) -> D.Stream m Int -> Unfold m (MutArray a) a
1037−getIndicesD liftio (D.Stream stepi sti) = Unfold step inject
1285+indexReaderWith liftio (D.Stream stepi sti) = Unfold step inject
10381286
10391287 where
10401288
@@ -1053,9 +1301,26 @@ getIndicesD liftio (D.Stream stepi sti) = Unfold step inject
10531301 D.Skip s -> return $ D.Skip (GetIndicesState contents start end s)
10541302 D.Stop -> return D.Stop
10551303
1304+{-# DEPRECATED getIndicesWith "Please use indexReaderWith instead." #-}
1305+{-# INLINE getIndicesWith #-}
1306+getIndicesWith :: (Monad m, Unbox a) =>
1307+ (forall b. IO b -> m b) -> D.Stream m Int -> Unfold m (MutArray a) a
1308+getIndicesWith = indexReaderWith
1309+
1310+-- | Given an unfold that generates array indices, read the elements on those
1311+-- indices from the supplied MutArray. An error is thrown if an index is out of
1312+-- bounds.
1313+--
1314+-- /Pre-release/
1315+{-# INLINE indexReader #-}
1316+indexReader :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1317+indexReader = indexReaderWith liftIO
1318+
1319+-- XXX DO NOT REMOVE, change the signature to use Stream instead of unfold
1320+{-# DEPRECATED getIndices "Please use indexReader instead." #-}
10561321{-# INLINE getIndices #-}
10571322getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1058−getIndices = getIndicesD liftIO
1323+getIndices = indexReader
10591324
10601325-------------------------------------------------------------------------------
10611326-- Subarrays
@@ -1160,7 +1425,7 @@ partitionBy f arr@MutArray{..} = liftIO $ do
11601425
11611426 -- Invariant low < high on entry, and on return as well
11621427 moveHigh low high = do
1163− h <- peekByteIndex high arrContents
1428+ h <- peekAt high arrContents
11641429 if f h
11651430 then
11661431 -- Correctly classified, continue the loop
@@ -1180,7 +1445,7 @@ partitionBy f arr@MutArray{..} = liftIO $ do
11801445 -- low <= high
11811446 -- Both low and high are valid locations within the array
11821447 go low high = do
1183− l <- peekByteIndex low arrContents
1448+ l <- peekAt low arrContents
11841449 if f l
11851450 then
11861451 -- low is wrongly classified
@@ -1191,8 +1456,8 @@ partitionBy f arr@MutArray{..} = liftIO $ do
11911456 case r of
11921457 Nothing -> return low
11931458 Just (high1, h) -> do -- low < high1
1194− pokeByteIndex low arrContents h
1195− pokeByteIndex high1 arrContents l
1459+ pokeAt low arrContents h
1460+ pokeAt high1 arrContents l
11961461 let low1 = INDEX_NEXT(low,a)
11971462 high2 = INDEX_PREV(high1,a)
11981463 if low1 <= high2
@@ -1326,7 +1591,7 @@ chunksOfAs ps n (D.Stream step state) =
13261591 r <- step (adaptState gst) st
13271592 case r of
13281593 D.Yield x s -> do
1329− liftIO $ pokeByteIndex end contents x
1594+ liftIO $ pokeAt end contents x
13301595 let end1 = INDEX_NEXT(end,a)
13311596 return $
13321597 if end1 >= bound
@@ -1345,10 +1610,12 @@ chunksOfAs ps n (D.Stream step state) =
13451610
13461611 step' _ GroupFinish = return D.Stop
13471612
1348−-- | @chunksOf n stream@ groups the input stream into a stream of
1349−-- arrays of size n.
1613+-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
1614+-- @n@ elements each.
13501615--
1351−-- @chunksOf n = StreamD.foldMany (MutArray.writeN n)@
1616+-- Same as the following but may be more efficient:
1617+--
1618+-- >>> chunksOf n = Stream.foldMany (MutArray.createOf n)
13521619--
13531620-- /Pre-release/
13541621{-# INLINE_NORMAL chunksOf #-}
@@ -1357,17 +1624,29 @@ chunksOf :: forall m a. (MonadIO m, Unbox a)
13571624-- XXX the idiomatic implementation leads to large regression in the D.reverse'
13581625-- benchmark. It seems it has difficulty producing optimized code when
13591626-- converting to StreamK. Investigate GHC optimizations.
1360−-- chunksOf n = D.foldMany (writeN n)
1627+-- chunksOf n = D.foldMany (createOf n)
13611628chunksOf = chunksOfAs Unpinned
13621629
13631630-- | Like 'chunksOf' but creates pinned arrays.
13641631{-# INLINE_NORMAL pinnedChunksOf #-}
13651632pinnedChunksOf :: forall m a. (MonadIO m, Unbox a)
13661633 => Int -> D.Stream m a -> D.Stream m (MutArray a)
1367−-- pinnedChunksOf n = D.foldMany (pinnedWriteN n)
1634+-- pinnedChunksOf n = D.foldMany (pinnedCreateOf n)
13681635pinnedChunksOf = chunksOfAs Pinned
13691636
1370−-- XXX This should take a PinnedState
1637+-- | When we are buffering a stream of unknown size into an array we do not
1638+-- know how much space to pre-allocate. So we start with the min size and emit
1639+-- the array then keep on doubling the size every time. Thus we do not need to
1640+-- guess the optimum chunk size.
1641+--
1642+-- We can incorporate this in chunksOfAs if the additional size parameter does
1643+-- not impact perf.
1644+--
1645+{-# INLINE _chunksOfRange #-}
1646+_chunksOfRange :: -- (MonadIO m, Unbox a) =>
1647+ PinnedState -> Int -> Int -> D.Stream m a -> D.Stream m (MutArray a)
1648+_chunksOfRange _ps _low _hi = undefined
1649+
13711650-- XXX buffer to a list instead?
13721651-- | Buffer the stream into arrays in memory.
13731652{-# INLINE arrayStreamKFromStreamDAs #-}
@@ -1385,16 +1664,10 @@ data FlattenState s contents a =
13851664 OuterLoop s
13861665 | InnerLoop s contents !Int !Int
13871666
1388−-- | Use the "reader" unfold instead.
1389−--
1390−-- @flattenArrays = unfoldMany reader@
1391−--
1392−-- We can try this if there are any fusion issues in the unfold.
1393−--
1394−{-# INLINE_NORMAL flattenArrays #-}
1395−flattenArrays :: forall m a. (MonadIO m, Unbox a)
1396− => D.Stream m (MutArray a) -> D.Stream m a
1397−flattenArrays (D.Stream step state) = D.Stream step' (OuterLoop state)
1667+{-# INLINE_NORMAL concatWith #-}
1668+concatWith :: forall m a. (Monad m, Unbox a)
1669+ => (forall b. IO b -> m b) -> D.Stream m (MutArray a) -> D.Stream m a
1670+concatWith liftio (D.Stream step state) = D.Stream step' (OuterLoop state)
13981671
13991672 where
14001673
@@ -1411,19 +1684,30 @@ flattenArrays (D.Stream step state) = D.Stream step' (OuterLoop state)
14111684 return $ D.Skip $ OuterLoop st
14121685
14131686 step' _ (InnerLoop st contents p end) = do
1414− x <- liftIO $ peekByteIndex p contents
1687+ x <- liftio $ peekAt p contents
14151688 return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
14161689
1417−-- | Use the "readerRev" unfold instead.
1690+-- | Use the "reader" unfold instead.
14181691--
1419−-- @flattenArrays = unfoldMany readerRev@
1692+-- @concat = unfoldMany reader@
14201693--
14211694-- We can try this if there are any fusion issues in the unfold.
14221695--
1423−{-# INLINE_NORMAL flattenArraysRev #-}
1424−flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
1696+{-# INLINE_NORMAL concat #-}
1697+concat :: forall m a. (MonadIO m, Unbox a)
1698+ => D.Stream m (MutArray a) -> D.Stream m a
1699+concat = concatWith liftIO
1700+
1701+{-# DEPRECATED flattenArrays "Please use \"unfoldMany reader\" instead." #-}
1702+{-# INLINE flattenArrays #-}
1703+flattenArrays :: forall m a. (MonadIO m, Unbox a)
14251704 => D.Stream m (MutArray a) -> D.Stream m a
1426−flattenArraysRev (D.Stream step state) = D.Stream step' (OuterLoop state)
1705+flattenArrays = concat
1706+
1707+{-# INLINE_NORMAL concatRevWith #-}
1708+concatRevWith :: forall m a. (Monad m, Unbox a)
1709+ => (forall b. IO b -> m b) -> D.Stream m (MutArray a) -> D.Stream m a
1710+concatRevWith liftio (D.Stream step state) = D.Stream step' (OuterLoop state)
14271711
14281712 where
14291713
@@ -1441,16 +1725,33 @@ flattenArraysRev (D.Stream step state) = D.Stream step' (OuterLoop state)
14411725 return $ D.Skip $ OuterLoop st
14421726
14431727 step' _ (InnerLoop st contents p start) = do
1444− x <- liftIO $ peekByteIndex p contents
1728+ x <- liftio $ peekAt p contents
14451729 let cur = INDEX_PREV(p,a)
14461730 return $ D.Yield x (InnerLoop st contents cur start)
14471731
1732+-- | Use the "readerRev" unfold instead.
1733+--
1734+-- @concat = unfoldMany readerRev@
1735+--
1736+-- We can try this if there are any fusion issues in the unfold.
1737+--
1738+{-# INLINE_NORMAL concatRev #-}
1739+concatRev :: forall m a. (MonadIO m, Unbox a)
1740+ => D.Stream m (MutArray a) -> D.Stream m a
1741+concatRev = concatRevWith liftIO
1742+
1743+{-# DEPRECATED flattenArraysRev "Please use \"unfoldMany readerRev\" instead." #-}
1744+{-# INLINE flattenArraysRev #-}
1745+flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
1746+ => D.Stream m (MutArray a) -> D.Stream m a
1747+flattenArraysRev = concatRev
1748+
14481749-------------------------------------------------------------------------------
14491750-- Unfolds
14501751-------------------------------------------------------------------------------
14511752
14521753data ArrayUnsafe a = ArrayUnsafe
1453− {-# UNPACK #-} !MutableByteArray -- contents
1754+ {-# UNPACK #-} !MutByteArray -- contents
14541755 {-# UNPACK #-} !Int -- index 1
14551756 {-# UNPACK #-} !Int -- index 2
14561757
@@ -1480,7 +1781,7 @@ producerWith liftio = Producer step (return . toArrayUnsafe) extract
14801781 -- few actions for correctness and execution order sanity. We want
14811782 -- the peek to occur right here and not lazily at some later point
14821783 -- because we want the peek to be ordered with respect to the touch.
1483− !x <- liftio $ peekByteIndex cur contents
1784+ !x <- liftio $ peekAt cur contents
14841785 return $ D.Yield x (ArrayUnsafe contents (INDEX_NEXT(cur,a)) end)
14851786
14861787 extract = return . fromArrayUnsafe
@@ -1511,7 +1812,7 @@ readerRevWith liftio = Unfold step inject
15111812 {-# INLINE_LATE step #-}
15121813 step (ArrayUnsafe _ start p) | p < start = return D.Stop
15131814 step (ArrayUnsafe contents start p) = do
1514− !x <- liftio $ peekByteIndex p contents
1815+ !x <- liftio $ peekAt p contents
15151816 return $ D.Yield x (ArrayUnsafe contents start (INDEX_PREV(p,a)))
15161817
15171818-- | Unfold an array into a stream in reverse order.
@@ -1542,7 +1843,7 @@ toListFB c n MutArray{..} = go arrStart
15421843 -- evaluated/written to before we peek at them.
15431844 -- XXX
15441845 let !x = unsafeInlineIO $ do
1545− r <- peekByteIndex arrContents p
1846+ r <- peekAt arrContents p
15461847 return r
15471848 in c x (go (PTR_NEXT(p,a)))
15481849-}
@@ -1559,21 +1860,21 @@ toList MutArray{..} = liftIO $ go arrStart
15591860
15601861 go p | assert (p <= arrEnd) (p == arrEnd) = return []
15611862 go p = do
1562− x <- peekByteIndex p arrContents
1863+ x <- peekAt p arrContents
15631864 (:) x <$> go (INDEX_NEXT(p,a))
15641865
1565−{-# INLINE_NORMAL toStreamDWith #-}
1566−toStreamDWith ::
1866+{-# INLINE_NORMAL toStreamWith #-}
1867+toStreamWith ::
15671868 forall m a. (Monad m, Unbox a)
15681869 => (forall b. IO b -> m b) -> MutArray a -> D.Stream m a
1569−toStreamDWith liftio MutArray{..} = D.Stream step arrStart
1870+toStreamWith liftio MutArray{..} = D.Stream step arrStart
15701871
15711872 where
15721873
15731874 {-# INLINE_LATE step #-}
15741875 step _ p | assert (p <= arrEnd) (p == arrEnd) = return D.Stop
15751876 step _ p = liftio $ do
1576− r <- peekByteIndex p arrContents
1877+ r <- peekAt p arrContents
15771878 return $ D.Yield r (INDEX_NEXT(p,a))
15781879
15791880-- | Convert a 'MutArray' into a stream.
@@ -1582,7 +1883,7 @@ toStreamDWith liftio MutArray{..} = D.Stream step arrStart
15821883--
15831884{-# INLINE_NORMAL read #-}
15841885read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> D.Stream m a
1585−read = toStreamDWith liftIO
1886+read = toStreamWith liftIO
15861887
15871888{-# INLINE toStreamKWith #-}
15881889toStreamKWith ::
@@ -1594,18 +1895,18 @@ toStreamKWith liftio MutArray{..} = go arrStart
15941895
15951896 go p | assert (p <= arrEnd) (p == arrEnd) = K.nil
15961897 | otherwise =
1597− let elemM = peekByteIndex p arrContents
1898+ let elemM = peekAt p arrContents
15981899 in liftio elemM `K.consM` go (INDEX_NEXT(p,a))
15991900
16001901{-# INLINE toStreamK #-}
16011902toStreamK :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
16021903toStreamK = toStreamKWith liftIO
16031904
1604−{-# INLINE_NORMAL toStreamDRevWith #-}
1605−toStreamDRevWith ::
1905+{-# INLINE_NORMAL toStreamRevWith #-}
1906+toStreamRevWith ::
16061907 forall m a. (Monad m, Unbox a)
16071908 => (forall b. IO b -> m b) -> MutArray a -> D.Stream m a
1608−toStreamDRevWith liftio MutArray{..} =
1909+toStreamRevWith liftio MutArray{..} =
16091910 let p = INDEX_PREV(arrEnd,a)
16101911 in D.Stream step p
16111912
@@ -1614,7 +1915,7 @@ toStreamDRevWith liftio MutArray{..} =
16141915 {-# INLINE_LATE step #-}
16151916 step _ p | p < arrStart = return D.Stop
16161917 step _ p = liftio $ do
1617− r <- peekByteIndex p arrContents
1918+ r <- peekAt p arrContents
16181919 return $ D.Yield r (INDEX_PREV(p,a))
16191920
16201921-- | Convert a 'MutArray' into a stream in reverse order.
@@ -1623,7 +1924,7 @@ toStreamDRevWith liftio MutArray{..} =
16231924--
16241925{-# INLINE_NORMAL readRev #-}
16251926readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> D.Stream m a
1626−readRev = toStreamDRevWith liftIO
1927+readRev = toStreamRevWith liftIO
16271928
16281929{-# INLINE toStreamKRevWith #-}
16291930toStreamKRevWith ::
@@ -1637,7 +1938,7 @@ toStreamKRevWith liftio MutArray {..} =
16371938
16381939 go p | p < arrStart = K.nil
16391940 | otherwise =
1640− let elemM = peekByteIndex p arrContents
1941+ let elemM = peekAt p arrContents
16411942 in liftio elemM `K.consM` go (INDEX_PREV(p,a))
16421943
16431944{-# INLINE toStreamKRev #-}
@@ -1672,7 +1973,7 @@ foldr f z arr = D.foldr f z $ read arr
16721973
16731974-- XXX Keep the bound intact to not lose any free space? Perf impact?
16741975
1675−-- | @writeAppendNUnsafe n arr@ appends up to @n@ input items to the supplied
1976+-- | @unsafeAppendN n arr@ appends up to @n@ input items to the supplied
16761977-- array.
16771978--
16781979-- Unsafe: Do not drive the fold beyond @n@ elements, it will lead to memory
@@ -1681,13 +1982,12 @@ foldr f z arr = D.foldr f z $ read arr
16811982-- Any free space left in the array after appending @n@ elements is lost.
16821983--
16831984-- /Internal/
1684−{-# INLINE_NORMAL writeAppendNUnsafe #-}
1685−writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) =>
1985+{-# INLINE_NORMAL unsafeAppendN #-}
1986+unsafeAppendN :: forall m a. (MonadIO m, Unbox a) =>
16861987 Int
16871988 -> m (MutArray a)
16881989 -> Fold m a (MutArray a)
1689−writeAppendNUnsafe n action =
1690− fmap fromArrayUnsafe $ FL.foldlM' step initial
1990+unsafeAppendN n action = fmap fromArrayUnsafe $ FL.foldlM' step initial
16911991
16921992 where
16931993
@@ -1700,38 +2000,59 @@ writeAppendNUnsafe n action =
17002000 -- otherwise we lose that space.
17012001 arr1 <-
17022002 if free < needed
1703− then noinline reallocWith "writeAppendNUnsafeWith" (+ needed) needed arr
2003+ then noinline reallocWith "unsafeAppendN" (+ needed) needed arr
17042004 else return arr
17052005 return $ toArrayUnsafe arr1
17062006
17072007 step (ArrayUnsafe contents start end) x = do
1708− liftIO $ pokeByteIndex end contents x
2008+ liftIO $ pokeAt end contents x
17092009 return $ ArrayUnsafe contents start (INDEX_NEXT(end,a))
17102010
2011+{-# DEPRECATED writeAppendNUnsafe "Please use unsafeAppendN instead." #-}
2012+{-# INLINE writeAppendNUnsafe #-}
2013+writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) =>
2014+ Int
2015+ -> m (MutArray a)
2016+ -> Fold m a (MutArray a)
2017+writeAppendNUnsafe = unsafeAppendN
2018+
17112019-- | Append @n@ elements to an existing array. Any free space left in the array
17122020-- after appending @n@ elements is lost.
17132021--
1714−-- >>> writeAppendN n initial = Fold.take n (MutArray.writeAppendNUnsafe n initial)
2022+-- >>> appendN n initial = Fold.take n (MutArray.unsafeAppendN n initial)
17152023--
1716−{-# INLINE_NORMAL writeAppendN #-}
2024+{-# INLINE_NORMAL appendN #-}
2025+appendN :: forall m a. (MonadIO m, Unbox a) =>
2026+ Int -> m (MutArray a) -> Fold m a (MutArray a)
2027+appendN n initial = FL.take n (unsafeAppendN n initial)
2028+
2029+-- XXX Deprecate in major
2030+-- {-# DEPRECATED writeAppendN "Please use appendN instead." #-}
2031+{-# INLINE writeAppendN #-}
17172032writeAppendN :: forall m a. (MonadIO m, Unbox a) =>
17182033 Int -> m (MutArray a) -> Fold m a (MutArray a)
1719−writeAppendN n initial = FL.take n (writeAppendNUnsafe n initial)
2034+writeAppendN = appendN
17202035
1721−-- | @writeAppendWith realloc action@ mutates the array generated by @action@ to
2036+-- | @appendWith realloc action@ mutates the array generated by @action@ to
17222037-- append the input stream. If there is no reserved space available in the
17232038-- array it is reallocated to a size in bytes determined by @realloc oldSize@,
17242039-- where @oldSize@ is the current size of the array in bytes.
17252040--
17262041-- Note that the returned array may be a mutated version of original array.
17272042--
1728−-- >>> writeAppendWith sizer = Fold.foldlM' (MutArray.snocWith sizer)
2043+-- >>> appendWith sizer = Fold.foldlM' (MutArray.snocWith sizer)
17292044--
17302045-- /Pre-release/
2046+{-# INLINE appendWith #-}
2047+appendWith :: forall m a. (MonadIO m, Unbox a) =>
2048+ (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
2049+appendWith sizer = FL.foldlM' (snocWith sizer)
2050+
2051+{-# DEPRECATED writeAppendWith "Please use appendWith instead." #-}
17312052{-# INLINE writeAppendWith #-}
17322053writeAppendWith :: forall m a. (MonadIO m, Unbox a) =>
17332054 (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
1734−writeAppendWith sizer = FL.foldlM' (snocWith sizer)
2055+writeAppendWith = appendWith
17352056
17362057-- | @append action@ mutates the array generated by @action@ to append the
17372058-- input stream. If there is no reserved space available in the array it is
@@ -1739,70 +2060,101 @@ writeAppendWith sizer = FL.foldlM' (snocWith sizer)
17392060--
17402061-- Note that the returned array may be a mutated version of original array.
17412062--
1742−-- >>> writeAppend = MutArray.writeAppendWith (* 2)
2063+-- >>> append = MutArray.appendWith (* 2)
17432064--
2065+{-# INLINE append #-}
2066+append :: forall m a. (MonadIO m, Unbox a) =>
2067+ m (MutArray a) -> Fold m a (MutArray a)
2068+append = appendWith (* 2)
2069+
2070+-- XXX Deprecate in major
2071+-- {-# DEPRECATED writeAppend "Please use append instead." #-}
17442072{-# INLINE writeAppend #-}
17452073writeAppend :: forall m a. (MonadIO m, Unbox a) =>
17462074 m (MutArray a) -> Fold m a (MutArray a)
1747−writeAppend = writeAppendWith (* 2)
2075+writeAppend = append
17482076
17492077-- XXX We can carry bound as well in the state to make sure we do not lose the
17502078-- remaining capacity. Need to check perf impact.
17512079--
1752−-- | Like 'writeNUnsafe' but takes a new array allocator @alloc size@ function
1753−-- as argument.
2080+-- | Like 'unsafeCreateOf' but takes a new array allocator @alloc size@
2081+-- function as argument.
17542082--
1755−-- >>> writeNWithUnsafe alloc n = MutArray.writeAppendNUnsafe (alloc n) n
2083+-- >>> unsafeCreateOfWith alloc n = MutArray.unsafeAppendN (alloc n) n
17562084--
17572085-- /Pre-release/
1758−{-# INLINE_NORMAL writeNWithUnsafe #-}
1759−writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a)
2086+{-# INLINE_NORMAL unsafeCreateOfWith #-}
2087+unsafeCreateOfWith :: forall m a. (MonadIO m, Unbox a)
17602088 => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1761−writeNWithUnsafe alloc n = fromArrayUnsafe <$> FL.foldlM' step initial
2089+unsafeCreateOfWith alloc n = fromArrayUnsafe <$> FL.foldlM' step initial
17622090
17632091 where
17642092
17652093 initial = toArrayUnsafe <$> alloc (max n 0)
17662094
17672095 step (ArrayUnsafe contents start end) x = do
1768− liftIO $ pokeByteIndex end contents x
2096+ liftIO $ pokeAt end contents x
17692097 return
17702098 $ ArrayUnsafe contents start (INDEX_NEXT(end,a))
17712099
2100+{-# DEPRECATED writeNWithUnsafe "Please use unsafeCreateOfWith instead." #-}
2101+{-# INLINE writeNWithUnsafe #-}
2102+writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a)
2103+ => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
2104+writeNWithUnsafe = unsafeCreateOfWith
2105+
17722106{-# INLINE_NORMAL writeNUnsafeAs #-}
17732107writeNUnsafeAs :: forall m a. (MonadIO m, Unbox a)
17742108 => PinnedState -> Int -> Fold m a (MutArray a)
1775−writeNUnsafeAs ps = writeNWithUnsafe (newAs ps)
2109+writeNUnsafeAs ps = unsafeCreateOfWith (newAs ps)
17762110
1777−-- | Like 'writeN' but does not check the array bounds when writing. The fold
2111+-- | Like 'createOf' but does not check the array bounds when writing. The fold
17782112-- driver must not call the step function more than 'n' times otherwise it will
17792113-- corrupt the memory and crash. This function exists mainly because any
17802114-- conditional in the step function blocks fusion causing 10x performance
17812115-- slowdown.
17822116--
1783−-- >>> writeNUnsafe = MutArray.writeNWithUnsafe MutArray.new
2117+-- >>> unsafeCreateOf = MutArray.unsafeCreateOfWith MutArray.emptyOf
17842118--
1785−{-# INLINE_NORMAL writeNUnsafe #-}
2119+{-# INLINE_NORMAL unsafeCreateOf #-}
2120+unsafeCreateOf :: forall m a. (MonadIO m, Unbox a)
2121+ => Int -> Fold m a (MutArray a)
2122+unsafeCreateOf = writeNUnsafeAs Unpinned
2123+
2124+{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
2125+{-# INLINE writeNUnsafe #-}
17862126writeNUnsafe :: forall m a. (MonadIO m, Unbox a)
17872127 => Int -> Fold m a (MutArray a)
1788−writeNUnsafe = writeNUnsafeAs Unpinned
2128+writeNUnsafe = unsafeCreateOf
2129+
2130+-- | Like 'unsafeCreateOf' but creates a pinned array.
2131+{-# INLINE_NORMAL unsafePinnedCreateOf #-}
2132+unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a)
2133+ => Int -> Fold m a (MutArray a)
2134+unsafePinnedCreateOf = writeNUnsafeAs Pinned
17892135
1790−-- | Like 'writeNUnsafe' but creates a pinned array.
1791−{-# INLINE_NORMAL pinnedWriteNUnsafe #-}
2136+{-# DEPRECATED pinnedWriteNUnsafe "Please use unsafePinnedCreateOf instead." #-}
2137+{-# INLINE pinnedWriteNUnsafe #-}
17922138pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a)
17932139 => Int -> Fold m a (MutArray a)
1794−pinnedWriteNUnsafe = writeNUnsafeAs Pinned
2140+pinnedWriteNUnsafe = unsafePinnedCreateOf
17952141
1796−-- | @writeNWith alloc n@ folds a maximum of @n@ elements into an array
2142+-- | @createOfWith alloc n@ folds a maximum of @n@ elements into an array
17972143-- allocated using the @alloc@ function.
17982144--
1799−-- >>> writeNWith alloc n = Fold.take n (MutArray.writeNWithUnsafe alloc n)
1800−-- >>> writeNWith alloc n = MutArray.writeAppendN (alloc n) n
2145+-- >>> createOfWith alloc n = Fold.take n (MutArray.unsafeCreateOfWith alloc n)
2146+-- >>> createOfWith alloc n = MutArray.appendN (alloc n) n
18012147--
1802−{-# INLINE_NORMAL writeNWith #-}
2148+{-# INLINE_NORMAL createOfWith #-}
2149+createOfWith :: forall m a. (MonadIO m, Unbox a)
2150+ => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
2151+createOfWith alloc n = FL.take n (unsafeCreateOfWith alloc n)
2152+
2153+-- {-# DEPRECATED writeNWith "Please use createOfWith instead." #-}
2154+{-# INLINE writeNWith #-}
18032155writeNWith :: forall m a. (MonadIO m, Unbox a)
18042156 => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1805−writeNWith alloc n = FL.take n (writeNWithUnsafe alloc n)
2157+writeNWith = createOfWith
18062158
18072159{-# INLINE_NORMAL writeNAs #-}
18082160writeNAs ::
@@ -1810,28 +2162,42 @@ writeNAs ::
18102162 => PinnedState
18112163 -> Int
18122164 -> Fold m a (MutArray a)
1813−writeNAs ps = writeNWith (newAs ps)
2165+writeNAs ps = createOfWith (newAs ps)
18142166
1815−-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
2167+-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
18162168-- 'MutArray'.
18172169--
1818−-- >>> writeN = MutArray.writeNWith MutArray.new
1819−-- >>> writeN n = Fold.take n (MutArray.writeNUnsafe n)
1820−-- >>> writeN n = MutArray.writeAppendN n (MutArray.new n)
2170+-- >>> createOf = MutArray.createOfWith MutArray.new
2171+-- >>> createOf n = Fold.take n (MutArray.unsafeCreateOf n)
2172+-- >>> createOf n = MutArray.appendN n (MutArray.emptyOf n)
18212173--
1822−{-# INLINE_NORMAL writeN #-}
2174+{-# INLINE_NORMAL createOf #-}
2175+createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
2176+createOf = writeNAs Unpinned
2177+
2178+-- XXX Deprecate in major
2179+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
2180+{-# INLINE writeN #-}
18232181writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1824−writeN = writeNAs Unpinned
2182+writeN = createOf
18252183
1826−-- | Like 'writeN' but creates a pinned array.
1827−{-# INLINE_NORMAL pinnedWriteN #-}
2184+-- | Like 'createOf' but creates a pinned array.
2185+{-# INLINE_NORMAL pinnedCreateOf #-}
2186+pinnedCreateOf ::
2187+ forall m a. (MonadIO m, Unbox a)
2188+ => Int
2189+ -> Fold m a (MutArray a)
2190+pinnedCreateOf = writeNAs Pinned
2191+
2192+{-# DEPRECATED pinnedWriteN "Please use pinnedCreateOf instead." #-}
2193+{-# INLINE pinnedWriteN #-}
18282194pinnedWriteN ::
18292195 forall m a. (MonadIO m, Unbox a)
18302196 => Int
18312197 -> Fold m a (MutArray a)
1832−pinnedWriteN = writeNAs Pinned
2198+pinnedWriteN = pinnedCreateOf
18332199
1834−-- | Like writeNWithUnsafe but writes the array in reverse order.
2200+-- | Like unsafeCreateOfWith but writes the array in reverse order.
18352201--
18362202-- /Internal/
18372203{-# INLINE_NORMAL writeRevNWithUnsafe #-}
@@ -1848,11 +2214,11 @@ writeRevNWithUnsafe alloc n = fromArrayUnsafe <$> FL.foldlM' step initial
18482214
18492215 step (ArrayUnsafe contents start end) x = do
18502216 let ptr = INDEX_PREV(start,a)
1851− liftIO $ pokeByteIndex ptr contents x
2217+ liftIO $ pokeAt ptr contents x
18522218 return
18532219 $ ArrayUnsafe contents ptr end
18542220
1855−-- | Like writeNWith but writes the array in reverse order.
2221+-- | Like createOfWith but writes the array in reverse order.
18562222--
18572223-- /Internal/
18582224{-# INLINE_NORMAL writeRevNWith #-}
@@ -1860,31 +2226,36 @@ writeRevNWith :: forall m a. (MonadIO m, Unbox a)
18602226 => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
18612227writeRevNWith alloc n = FL.take n (writeRevNWithUnsafe alloc n)
18622228
1863−-- | Like writeN but writes the array in reverse order.
2229+-- | Like 'createOf' but writes the array in reverse order.
18642230--
18652231-- /Pre-release/
1866−{-# INLINE_NORMAL writeRevN #-}
2232+{-# INLINE_NORMAL revCreateOf #-}
2233+revCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
2234+revCreateOf = writeRevNWith new
2235+
2236+{-# DEPRECATED writeRevN "Please use revCreateOf instead." #-}
2237+{-# INLINE writeRevN #-}
18672238writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1868−writeRevN = writeRevNWith new
2239+writeRevN = revCreateOf
18692240
18702241-- | @pinnedWriteNAligned align n@ folds a maximum of @n@ elements from the
18712242-- input stream to a 'MutArray' aligned to the given size.
18722243--
1873−-- >>> pinnedWriteNAligned align = MutArray.writeNWith (MutArray.pinnedNewAligned align)
1874−-- >>> pinnedWriteNAligned align n = MutArray.writeAppendN n (MutArray.pinnedNewAligned align n)
2244+-- >>> pinnedWriteNAligned align = MutArray.createOfWith (MutArray.pinnedNewAligned align)
2245+-- >>> pinnedWriteNAligned align n = MutArray.appendN n (MutArray.pinnedNewAligned align n)
18752246--
18762247-- /Pre-release/
18772248--
18782249{-# INLINE_NORMAL pinnedWriteNAligned #-}
18792250pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a)
18802251 => Int -> Int -> Fold m a (MutArray a)
1881−pinnedWriteNAligned align = writeNWith (pinnedNewAligned align)
2252+pinnedWriteNAligned align = createOfWith (pinnedNewAligned align)
18822253
18832254-- XXX Buffer to a list instead?
1884−--
2255+
18852256-- | Buffer a stream into a stream of arrays.
18862257--
1887−-- >>> writeChunks n = Fold.many (MutArray.writeN n) Fold.toStreamK
2258+-- >>> buildChunks n = Fold.many (MutArray.createOf n) Fold.toStreamK
18882259--
18892260-- Breaking an array into an array stream can be useful to consume a large
18902261-- array sequentially such that memory of the array is released incrementatlly.
@@ -1893,37 +2264,43 @@ pinnedWriteNAligned align = writeNWith (pinnedNewAligned align)
18932264--
18942265-- /Unimplemented/
18952266--
1896−{-# INLINE_NORMAL writeChunks #-}
2267+{-# INLINE_NORMAL buildChunks #-}
2268+buildChunks :: (MonadIO m, Unbox a) =>
2269+ Int -> Fold m a (StreamK n (MutArray a))
2270+buildChunks n = FL.many (createOf n) FL.toStreamK
2271+
2272+{-# DEPRECATED writeChunks "Please use buildChunks instead." #-}
2273+{-# INLINE writeChunks #-}
18972274writeChunks :: (MonadIO m, Unbox a) =>
18982275 Int -> Fold m a (StreamK n (MutArray a))
1899−writeChunks n = FL.many (writeN n) FL.toStreamK
2276+writeChunks = buildChunks
19002277
19012278{-# INLINE_NORMAL writeWithAs #-}
19022279writeWithAs :: forall m a. (MonadIO m, Unbox a)
19032280 => PinnedState -> Int -> Fold m a (MutArray a)
1904−-- writeWithAs ps n = FL.rmapM rightSize $ writeAppendWith (* 2) (newAs ps n)
2281+-- writeWithAs ps n = FL.rmapM rightSize $ appendWith (* 2) (newAs ps n)
19052282writeWithAs ps elemCount =
19062283 FL.rmapM extract $ FL.foldlM' step initial
19072284
19082285 where
19092286
19102287 initial = do
1911− when (elemCount < 0) $ error "writeWith: elemCount is negative"
1912− liftIO $ newAs ps elemCount
2288+ when (elemCount < 0) $ error "createWith: elemCount is negative"
2289+ newAs ps elemCount
19132290
19142291 step arr@(MutArray _ start end bound) x
19152292 | INDEX_NEXT(end,a) > bound = do
19162293 let oldSize = end - start
19172294 newSize = max (oldSize * 2) 1
1918− arr1 <- liftIO $ reallocExplicit (SIZE_OF(a)) newSize arr
2295+ arr1 <- liftIO $ reallocExplicitAs ps (SIZE_OF(a)) newSize arr
19192296 snocUnsafe arr1 x
19202297 step arr x = snocUnsafe arr x
19212298
19222299 extract = liftIO . rightSize
19232300
1924−-- XXX Compare writeWith with fromStreamD which uses an array of streams
1925−-- implementation. We can write this using writeChunks above if that is faster.
1926−-- If writeWith is faster then we should use that to implement
2301+-- XXX Compare createWith with fromStreamD which uses an array of streams
2302+-- implementation. We can write this using buildChunks above if that is faster.
2303+-- If createWith is faster then we should use that to implement
19272304-- fromStreamD.
19282305--
19292306-- XXX The realloc based implementation needs to make one extra copy if we use
@@ -1938,39 +2315,56 @@ writeWithAs ps elemCount =
19382315-- XXX check if GHC's memory allocator is efficient enough. We can try the C
19392316-- malloc to compare against.
19402317
1941−-- | @writeWith minCount@ folds the whole input to a single array. The array
2318+-- | @createWith minCount@ folds the whole input to a single array. The array
19422319-- starts at a size big enough to hold minCount elements, the size is doubled
19432320-- every time the array needs to be grown.
19442321--
19452322-- /Caution! Do not use this on infinite streams./
19462323--
1947−-- >>> f n = MutArray.writeAppendWith (* 2) (MutArray.new n)
1948−-- >>> writeWith n = Fold.rmapM MutArray.rightSize (f n)
1949−-- >>> writeWith n = Fold.rmapM MutArray.fromArrayStreamK (MutArray.writeChunks n)
2324+-- >>> f n = MutArray.appendWith (* 2) (MutArray.emptyOf n)
2325+-- >>> createWith n = Fold.rmapM MutArray.rightSize (f n)
2326+-- >>> createWith n = Fold.rmapM MutArray.fromChunksK (MutArray.buildChunks n)
19502327--
19512328-- /Pre-release/
1952−{-# INLINE_NORMAL writeWith #-}
2329+{-# INLINE_NORMAL createWith #-}
2330+createWith :: forall m a. (MonadIO m, Unbox a)
2331+ => Int -> Fold m a (MutArray a)
2332+-- createWith n = FL.rmapM rightSize $ appendWith (* 2) (emptyOf n)
2333+createWith = writeWithAs Unpinned
2334+
2335+{-# DEPRECATED writeWith "Please use createWith instead." #-}
2336+{-# INLINE writeWith #-}
19532337writeWith :: forall m a. (MonadIO m, Unbox a)
19542338 => Int -> Fold m a (MutArray a)
1955−-- writeWith n = FL.rmapM rightSize $ writeAppendWith (* 2) (new n)
1956−writeWith = writeWithAs Unpinned
2339+writeWith = createWith
19572340
19582341-- | Fold the whole input to a single array.
19592342--
1960−-- Same as 'writeWith' using an initial array size of 'arrayChunkBytes' bytes
2343+-- Same as 'createWith' using an initial array size of 'arrayChunkBytes' bytes
19612344-- rounded up to the element size.
19622345--
19632346-- /Caution! Do not use this on infinite streams./
19642347--
2348+{-# INLINE create #-}
2349+create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
2350+create = createWith (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
2351+
2352+-- XXX Deprecate in major
2353+-- {-# DEPRECATED write "Please use create instead." #-}
19652354{-# INLINE write #-}
19662355write :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1967−write = writeWith (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
2356+write = create
19682357
1969−-- | Like 'write' but creates a pinned array.
2358+-- | Like 'create' but creates a pinned array.
2359+{-# INLINE pinnedCreate #-}
2360+pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
2361+pinnedCreate =
2362+ writeWithAs Pinned (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
2363+
2364+{-# DEPRECATED pinnedWrite "Please use pinnedCreate instead." #-}
19702365{-# INLINE pinnedWrite #-}
19712366pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1972−pinnedWrite =
1973− writeWithAs Pinned (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
2367+pinnedWrite = pinnedCreate
19742368
19752369-------------------------------------------------------------------------------
19762370-- construct from streams, known size
@@ -1980,25 +2374,31 @@ pinnedWrite =
19802374fromStreamDNAs :: forall m a. (MonadIO m, Unbox a)
19812375 => PinnedState -> Int -> D.Stream m a -> m (MutArray a)
19822376fromStreamDNAs ps limit str = do
1983− (arr :: MutArray a) <- liftIO $ newAs ps limit
2377+ (arr :: MutArray a) <- newAs ps limit
19842378 end <- D.foldlM' (fwrite (arrContents arr)) (return $ arrEnd arr) $ D.take limit str
19852379 return $ arr {arrEnd = end}
19862380
19872381 where
19882382
19892383 fwrite arrContents ptr x = do
1990− liftIO $ pokeByteIndex ptr arrContents x
2384+ liftIO $ pokeAt ptr arrContents x
19912385 return $ INDEX_NEXT(ptr,a)
19922386
1993−-- | Use the 'writeN' fold instead.
2387+-- | Use the 'createOf' fold instead.
19942388--
1995−-- >>> fromStreamDN n = Stream.fold (MutArray.writeN n)
2389+-- >>> fromStreamN n = Stream.fold (MutArray.createOf n)
19962390--
1997−{-# INLINE_NORMAL fromStreamDN #-}
2391+{-# INLINE_NORMAL fromStreamN #-}
2392+fromStreamN :: forall m a. (MonadIO m, Unbox a)
2393+ => Int -> D.Stream m a -> m (MutArray a)
2394+-- fromStreamDN n = D.fold (createOf n)
2395+fromStreamN = fromStreamDNAs Unpinned
2396+
2397+{-# DEPRECATED fromStreamDN "Please use fromStreamN instead." #-}
2398+{-# INLINE fromStreamDN #-}
19982399fromStreamDN :: forall m a. (MonadIO m, Unbox a)
19992400 => Int -> D.Stream m a -> m (MutArray a)
2000−-- fromStreamDN n = D.fold (writeN n)
2001−fromStreamDN = fromStreamDNAs Unpinned
2401+fromStreamDN = fromStreamN
20022402
20032403-- | Create a 'MutArray' from the first N elements of a list. The array is
20042404-- allocated to size N, if the list terminates before N elements then the
@@ -2018,16 +2418,70 @@ pinnedFromListN n xs = fromStreamDNAs Pinned n $ D.fromList xs
20182418-- /Pre-release/
20192419{-# INLINE fromListRevN #-}
20202420fromListRevN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
2021−fromListRevN n xs = D.fold (writeRevN n) $ D.fromList xs
2421+fromListRevN n xs = D.fold (revCreateOf n) $ D.fromList xs
2422+
2423+-- | Convert a pure stream in Identity monad to a mutable array.
2424+{-# INLINABLE fromPureStreamN #-}
2425+fromPureStreamN :: (MonadIO m, Unbox a) =>
2426+ Int -> Stream Identity a -> m (MutArray a)
2427+fromPureStreamN n xs =
2428+ D.fold (createOf n) $ D.morphInner (return . runIdentity) xs
20222429
20232430-- | Convert a pure stream in Identity monad to a mutable array.
20242431{-# INLINABLE fromPureStream #-}
20252432fromPureStream :: (MonadIO m, Unbox a) => Stream Identity a -> m (MutArray a)
20262433fromPureStream xs =
2027− liftIO $ D.fold write $ D.morphInner (return . runIdentity) xs
2434+ D.fold create $ D.morphInner (return . runIdentity) xs
2435+
2436+{-# INLINABLE fromPtrN #-}
2437+fromPtrN :: MonadIO m => Int -> Ptr Word8 -> m (MutArray Word8)
2438+fromPtrN len addr = do
2439+ -- memcpy is better than stream copy when the size is known.
2440+ -- XXX We can implement a stream copy in a similar way by streaming Word64
2441+ -- first and then remaining Word8.
2442+ arr <- new len
2443+ _ <- unsafeAsPtr arr
2444+ (\ptr -> liftIO $ c_memcpy ptr addr (fromIntegral len))
2445+ return (arr {arrEnd = len})
2446+
2447+{-# INLINABLE fromByteStr# #-}
2448+fromByteStr# :: MonadIO m => Addr# -> m (MutArray Word8)
2449+fromByteStr# addr = do
2450+ -- It is better to count the size first and allocate exact space.
2451+ -- Also, memcpy is better than stream copy when the size is known.
2452+ -- C strlen compares 4 bytes at a time, so is better than the stream
2453+ -- version. https://github.com/bminor/glibc/blob/master/string/strlen.c
2454+ -- XXX We can possibly use a stream of Word64 to do the same.
2455+ -- fromByteStr# addr = fromPureStream (D.fromByteStr# addr)
2456+ len <- liftIO $ c_strlen (Ptr addr)
2457+ let lenInt = fromIntegral len
2458+ arr <- new lenInt
2459+ _ <- unsafeAsPtr arr (\ptr -> liftIO $ c_memcpy ptr (Ptr addr) len)
2460+ return (arr {arrEnd = lenInt})
20282461
20292462-------------------------------------------------------------------------------
2030−-- convert stream to a single array
2463+-- convert a stream of arrays to a single array by reallocating and copying
2464+-------------------------------------------------------------------------------
2465+
2466+-- XXX Both of these implementations of splicing seem to perform equally well.
2467+-- We need to perform benchmarks over a range of sizes though.
2468+
2469+-- | Also see 'fromChunksK'.
2470+{-# INLINE fromChunksRealloced #-}
2471+fromChunksRealloced :: forall m a. (MonadIO m, Unbox a)
2472+ => Stream m (MutArray a) -> m (MutArray a)
2473+fromChunksRealloced s = do
2474+ res <- D.uncons s
2475+ case res of
2476+ Just (a, strm) -> do
2477+ arr <- D.foldlM' spliceExp (pure a) strm
2478+ -- Reallocation is exponential so there may be 50% empty space in
2479+ -- worst case. One more reallocation to reclaim the space.
2480+ rightSize arr
2481+ Nothing -> pure nil
2482+
2483+-------------------------------------------------------------------------------
2484+-- convert a stream of arrays to a single array by buffering arrays first
20312485-------------------------------------------------------------------------------
20322486
20332487{-# INLINE arrayStreamKLength #-}
@@ -2037,34 +2491,72 @@ arrayStreamKLength as = K.foldl' (+) 0 (K.map length as)
20372491-- | Convert an array stream to an array. Note that this requires peak memory
20382492-- that is double the size of the array stream.
20392493--
2494+{-# INLINE fromChunkskAs #-}
2495+fromChunkskAs :: (Unbox a, MonadIO m) =>
2496+ PinnedState -> StreamK m (MutArray a) -> m (MutArray a)
2497+fromChunkskAs ps as = do
2498+ len <- arrayStreamKLength as
2499+ arr <- newAs ps len
2500+ -- XXX is StreamK fold faster or StreamD fold?
2501+ K.foldlM' spliceUnsafe (pure arr) as
2502+ -- fromStreamDN len $ D.unfoldMany reader $ D.fromStreamK as
2503+
2504+-- XXX Need to compare this with fromChunks and fromChunkList and keep the
2505+-- fastest or simplest one if all are equally fast.
2506+
2507+-- | Convert an array stream to an array. Note that this requires peak memory
2508+-- that is double the size of the array stream.
2509+--
2510+-- Also see 'fromChunksRealloced'.
2511+--
2512+{-# INLINE fromChunksK #-}
2513+fromChunksK :: (Unbox a, MonadIO m) =>
2514+ StreamK m (MutArray a) -> m (MutArray a)
2515+fromChunksK = fromChunkskAs Unpinned
2516+
2517+{-# DEPRECATED fromArrayStreamK "Please use fromChunksK instead." #-}
20402518{-# INLINE fromArrayStreamK #-}
20412519fromArrayStreamK :: (Unbox a, MonadIO m) =>
20422520 StreamK m (MutArray a) -> m (MutArray a)
2043−fromArrayStreamK as = do
2044− len <- arrayStreamKLength as
2045− fromStreamDN len $ D.unfoldMany reader $ D.fromStreamK as
2521+fromArrayStreamK = fromChunksK
20462522
20472523{-# INLINE fromStreamDAs #-}
20482524fromStreamDAs ::
20492525 (MonadIO m, Unbox a) => PinnedState -> D.Stream m a -> m (MutArray a)
2050−fromStreamDAs ps m = arrayStreamKFromStreamDAs ps m >>= fromArrayStreamK
2526+fromStreamDAs ps m =
2527+ arrayStreamKFromStreamDAs Unpinned m >>= fromChunkskAs ps
2528+
2529+-- | Create an 'Array' from a stream. This is useful when we want to create a
2530+-- single array from a stream of unknown size. 'createOf' is at least twice
2531+-- as efficient when the size is already known.
2532+--
2533+-- Note that if the input stream is too large memory allocation for the array
2534+-- may fail. When the stream size is not known, `chunksOf` followed by
2535+-- processing of indvidual arrays in the resulting stream should be preferred.
2536+--
2537+-- /Pre-release/
2538+{-# INLINE fromStream #-}
2539+fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
2540+fromStream = fromStreamDAs Unpinned
20512541
2542+-- fromStream (Stream m) = P.fold create m
20522543-- CAUTION: a very large number (millions) of arrays can degrade performance
20532544-- due to GC overhead because we need to buffer the arrays before we flatten
20542545-- all the arrays.
20552546--
2056−-- XXX Compare if this is faster or "fold write".
2547+-- XXX Compare if this is faster or "fold create".
20572548--
20582549-- | We could take the approach of doubling the memory allocation on each
20592550-- overflow. This would result in more or less the same amount of copying as in
20602551-- the chunking approach. However, if we have to shrink in the end then it may
20612552-- result in an extra copy of the entire data.
20622553--
2063−-- >>> fromStreamD = StreamD.fold MutArray.write
2554+-- >>> fromStreamD = StreamD.fold MutArray.create
20642555--
20652556{-# INLINE fromStreamD #-}
2557+{-# DEPRECATED fromStreamD "Please use fromStream instead." #-}
20662558fromStreamD :: (MonadIO m, Unbox a) => D.Stream m a -> m (MutArray a)
2067−fromStreamD = fromStreamDAs Unpinned
2559+fromStreamD = fromStream
20682560
20692561-- | Create a 'MutArray' from a list. The list must be of finite size.
20702562--
@@ -2086,24 +2578,52 @@ fromListRev :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
20862578fromListRev xs = fromListRevN (Prelude.length xs) xs
20872579
20882580-------------------------------------------------------------------------------
2089−-- Combining
2581+-- Cloning
20902582-------------------------------------------------------------------------------
20912583
2092−-- | Put a sub range of a source array into a subrange of a destination array.
2093−-- This is not safe as it does not check the bounds.
2094−{-# INLINE putSliceUnsafe #-}
2095−putSliceUnsafe :: MonadIO m => MutArray a -> Int -> MutArray a -> Int -> Int -> m ()
2096−putSliceUnsafe src srcStartBytes dst dstStartBytes lenBytes = liftIO $ do
2097− assertM(lenBytes <= arrBound dst - dstStartBytes)
2098− assertM(lenBytes <= arrEnd src - srcStartBytes)
2099− let !(I# srcStartBytes#) = srcStartBytes
2100− !(I# dstStartBytes#) = dstStartBytes
2101− !(I# lenBytes#) = lenBytes
2102− let arrS# = getMutableByteArray# (arrContents src)
2103− arrD# = getMutableByteArray# (arrContents dst)
2104− IO $ \s# -> (# copyMutableByteArray#
2105− arrS# srcStartBytes# arrD# dstStartBytes# lenBytes# s#
2106− , () #)
2584+-- Arrays are aligned on 64-bit boundaries. The fastest way to copy an array is
2585+-- to unsafeCast it to Word64, read it, write it to Word64 array and unsafeCast
2586+-- it again. We can use SIMD read/write as well.
2587+
2588+{-# INLINE cloneAs #-}
2589+cloneAs ::
2590+ ( MonadIO m
2591+#ifdef DEVBUILD
2592+ , Unbox a
2593+#endif
2594+ )
2595+ => PinnedState -> MutArray a -> m (MutArray a)
2596+cloneAs ps src =
2597+ do
2598+ let startSrc = arrStart src
2599+ srcLen = arrEnd src - startSrc
2600+ newArrContents <-
2601+ Unboxed.cloneSliceUnsafeAs ps startSrc srcLen (arrContents src)
2602+ return $ MutArray newArrContents 0 srcLen srcLen
2603+
2604+{-# INLINE clone #-}
2605+clone ::
2606+ ( MonadIO m
2607+#ifdef DEVBUILD
2608+ , Unbox a
2609+#endif
2610+ )
2611+ => MutArray a -> m (MutArray a)
2612+clone = cloneAs Unpinned
2613+
2614+{-# INLINE pinnedClone #-}
2615+pinnedClone ::
2616+ ( MonadIO m
2617+#ifdef DEVBUILD
2618+ , Unbox a
2619+#endif
2620+ )
2621+ => MutArray a -> m (MutArray a)
2622+pinnedClone = cloneAs Pinned
2623+
2624+-------------------------------------------------------------------------------
2625+-- Combining
2626+-------------------------------------------------------------------------------
21072627
21082628-- | Copy two arrays into a newly allocated array. If the first array is pinned
21092629-- the spliced array is also pinned.
@@ -2113,21 +2633,19 @@ spliceCopy :: forall m a. MonadIO m =>
21132633 Unbox a =>
21142634#endif
21152635 MutArray a -> MutArray a -> m (MutArray a)
2116−spliceCopy arr1 arr2 = liftIO $ do
2636+spliceCopy arr1 arr2 = do
21172637 let start1 = arrStart arr1
21182638 start2 = arrStart arr2
21192639 len1 = arrEnd arr1 - start1
21202640 len2 = arrEnd arr2 - start2
2121− let newLen = len1 + len2
2641+ let len = len1 + len2
21222642 newArrContents <-
21232643 if Unboxed.isPinned (arrContents arr1)
2124− then Unboxed.pinnedNewBytes newLen
2125− else Unboxed.newBytes newLen
2126− let len = len1 + len2
2127− newArr = MutArray newArrContents 0 len len
2128− putSliceUnsafe arr1 start1 newArr 0 len1
2129− putSliceUnsafe arr2 start2 newArr len1 len2
2130− return newArr
2644+ then liftIO $ Unboxed.pinnedNew len
2645+ else liftIO $ Unboxed.new len
2646+ putSliceUnsafe (arrContents arr1) start1 newArrContents 0 len1
2647+ putSliceUnsafe (arrContents arr2) start2 newArrContents len1 len2
2648+ return $ MutArray newArrContents 0 len len
21312649
21322650-- | Really really unsafe, appends the second array into the first array. If
21332651-- the first array does not have enough space it may cause silent data
@@ -2136,12 +2654,13 @@ spliceCopy arr1 arr2 = liftIO $ do
21362654spliceUnsafe :: MonadIO m =>
21372655 MutArray a -> MutArray a -> m (MutArray a)
21382656spliceUnsafe dst src =
2139− liftIO $ do
2657+ do
21402658 let startSrc = arrStart src
21412659 srcLen = arrEnd src - startSrc
21422660 endDst = arrEnd dst
21432661 assertM(endDst + srcLen <= arrBound dst)
2144− putSliceUnsafe src startSrc dst endDst srcLen
2662+ putSliceUnsafe
2663+ (arrContents src) startSrc (arrContents dst) endDst srcLen
21452664 return $ dst {arrEnd = endDst + srcLen}
21462665
21472666-- | @spliceWith sizer dst src@ mutates @dst@ to append @src@. If there is no
@@ -2157,7 +2676,7 @@ spliceWith :: forall m a. (MonadIO m, Unbox a) =>
21572676 (Int -> Int -> Int) -> MutArray a -> MutArray a -> m (MutArray a)
21582677spliceWith sizer dst@(MutArray _ start end bound) src = do
21592678{-
2160− let f = writeAppendWith (`sizer` byteLength src) (return dst)
2679+ let f = appendWith (`sizer` byteLength src) (return dst)
21612680 in D.fold f (toStreamD src)
21622681-}
21632682 assert (end <= bound) (return ())
@@ -2173,7 +2692,7 @@ spliceWith sizer dst@(MutArray _ start end bound) src = do
21732692 $ "splice: newSize is less than the total size "
21742693 ++ "of arrays being appended. Please check the "
21752694 ++ "sizer function passed."
2176− liftIO $ realloc newSizeInBytes dst
2695+ realloc newSizeInBytes dst
21772696 else return dst
21782697 spliceUnsafe dst1 src
21792698
@@ -2210,14 +2729,24 @@ spliceExp = spliceWith (\l1 l2 -> max (l1 * 2) (l1 + l2))
22102729-- Splitting
22112730-------------------------------------------------------------------------------
22122731
2732+-- | Generate a stream of array slices using a predicate. The array element
2733+-- matching the predicate is dropped.
2734+--
2735+-- /Pre-release/
2736+{-# INLINE splitOn #-}
2737+splitOn :: (MonadIO m, Unbox a) =>
2738+ (a -> Bool) -> MutArray a -> Stream m (MutArray a)
2739+splitOn predicate arr =
2740+ fmap (\(i, len) -> getSliceUnsafe i len arr)
2741+ $ D.indexOnSuffix predicate (read arr)
2742+
22132743-- | Drops the separator byte
22142744{-# INLINE breakOn #-}
22152745breakOn :: MonadIO m
22162746 => Word8 -> MutArray Word8 -> m (MutArray Word8, Maybe (MutArray Word8))
2217−breakOn sep arr@MutArray{..} = asPtrUnsafe arr $ \p -> liftIO $ do
2218− -- XXX Instead of using asPtrUnsafe (pinning memory) we can pass unlifted
2219− -- Addr# to memchr and it should be safe (from ghc 8.4).
2747+breakOn sep arr@MutArray{..} = unsafeAsPtr arr $ \p -> liftIO $ do
22202748 -- XXX We do not need memchr here, we can use a Haskell equivalent.
2749+ -- Need efficient stream based primitives that work on Word64.
22212750 loc <- c_memchr p sep (fromIntegral $ byteLength arr)
22222751 let sepIndex = loc `minusPtr` p
22232752 return $
@@ -2238,32 +2767,41 @@ breakOn sep arr@MutArray{..} = asPtrUnsafe arr $ \p -> liftIO $ do
22382767 }
22392768 )
22402769
2770+-- | Like 'splitAt' but does not check whether the index is valid.
2771+--
2772+{-# INLINE unsafeSplitAt #-}
2773+unsafeSplitAt :: forall a. Unbox a =>
2774+ Int -> MutArray a -> (MutArray a, MutArray a)
2775+unsafeSplitAt i MutArray{..} =
2776+ let off = i * SIZE_OF(a)
2777+ p = arrStart + off
2778+ in ( MutArray
2779+ { arrContents = arrContents
2780+ , arrStart = arrStart
2781+ , arrEnd = p
2782+ , arrBound = p
2783+ }
2784+ , MutArray
2785+ { arrContents = arrContents
2786+ , arrStart = p
2787+ , arrEnd = arrEnd
2788+ , arrBound = arrBound
2789+ }
2790+ )
2791+
22412792-- | Create two slices of an array without copying the original array. The
22422793-- specified index @i@ is the first index of the second slice.
22432794--
2795+{-# INLINE splitAt #-}
22442796splitAt :: forall a. Unbox a => Int -> MutArray a -> (MutArray a, MutArray a)
2245−splitAt i arr@MutArray{..} =
2797+splitAt i arr =
22462798 let maxIndex = length arr - 1
22472799 in if i < 0
22482800 then error "sliceAt: negative array index"
22492801 else if i > maxIndex
22502802 then error $ "sliceAt: specified array index " ++ show i
22512803 ++ " is beyond the maximum index " ++ show maxIndex
2252− else let off = i * SIZE_OF(a)
2253− p = arrStart + off
2254− in ( MutArray
2255− { arrContents = arrContents
2256− , arrStart = arrStart
2257− , arrEnd = p
2258− , arrBound = p
2259− }
2260− , MutArray
2261− { arrContents = arrContents
2262− , arrStart = p
2263− , arrEnd = arrEnd
2264− , arrBound = arrBound
2265− }
2266− )
2804+ else unsafeSplitAt i arr
22672805
22682806-------------------------------------------------------------------------------
22692807-- Casting
@@ -2313,13 +2851,16 @@ cast arr =
23132851-- guarantees that unsafe calls will be performed in the calling thread. Making
23142852-- it safe to pass heap-allocated objects to unsafe functions.
23152853
2316−-- Unsafe because of direct pointer operations. The user must ensure that they
2317−-- are writing within the legal bounds of the array. Should we just name it
2318−-- asPtr, the unsafety is implicit for any pointer operations. And we are safe
2319−-- from Haskell perspective because we will be pinning the memory.
2854+-- XXX Should we just name it asPtr, the unsafety is implicit for any pointer
2855+-- operations. And we are safe from Haskell perspective because we will be
2856+-- pinning the memory.
2857+--
2858+-- XXX we cannot pass the length of the ptr here as in some cases it may not be
2859+-- available e.g. a null terminated C string. However, we can create another
2860+-- flavor of the API e.g. asPtrN.
23202861
2321−-- | Use an @MutArray a@ as @Ptr a@. This is useful when we want to pass an array
2322−-- as a pointer to some operating system call or to a "safe" FFI call.
2862+-- | Use a @MutArray a@ as @Ptr a@. This is useful when we want to pass an
2863+-- array as a pointer to some operating system call or to a "safe" FFI call.
23232864--
23242865-- If the array is not pinned it is copied to pinned memory before passing it
23252866-- to the monadic action.
@@ -2329,56 +2870,382 @@ cast arr =
23292870-- opportunistically before this operation occurs, to avoid the cost of a copy
23302871-- if possible.
23312872--
2332−-- /Unsafe/
2873+-- /Unsafe/ because of direct pointer operations. The user must ensure that
2874+-- they are writing within the legal bounds of the array.
23332875--
23342876-- /Pre-release/
23352877--
2878+{-# INLINE unsafePinnedAsPtr #-}
2879+unsafePinnedAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
2880+unsafePinnedAsPtr arr f =
2881+ Unboxed.unsafePinnedAsPtr
2882+ (arrContents arr) (\ptr -> f (ptr `plusPtr` arrStart arr))
2883+
2884+{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
2885+{-# INLINE asPtrUnsafe #-}
23362886asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
2337−asPtrUnsafe arr f = do
2338− contents <- liftIO $ Unboxed.pin $ arrContents arr
2339− let !ptr = Ptr (byteArrayContents#
2340− (unsafeCoerce# (getMutableByteArray# contents)))
2887+asPtrUnsafe = unsafePinnedAsPtr
23412888
2342− r <- f (ptr `plusPtr` arrStart arr)
2343− liftIO $ touch contents
2344− return r
2889+{-# INLINE unsafeAsPtr #-}
2890+unsafeAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
2891+unsafeAsPtr arr f =
2892+ Unboxed.unsafeAsPtr
2893+ (arrContents arr) (\ptr -> f (ptr `plusPtr` arrStart arr))
23452894
23462895-------------------------------------------------------------------------------
23472896-- Equality
23482897-------------------------------------------------------------------------------
23492898
2350−-- | Compare the length of the arrays. If the length is equal, compare the
2351−-- lexicographical ordering of two underlying byte arrays otherwise return the
2352−-- result of length comparison.
2899+-- | Byte compare two arrays. Compare the length of the arrays. If the length
2900+-- is equal, compare the lexicographical ordering of two underlying byte arrays
2901+-- otherwise return the result of length comparison.
2902+--
2903+-- /Unsafe/: Note that the 'Unbox' instance of sum types with constructors of
2904+-- different sizes may leave some memory uninitialized which can make byte
2905+-- comparison unreliable.
23532906--
23542907-- /Pre-release/
2908+{-# INLINE byteCmp #-}
2909+byteCmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
2910+byteCmp arr1 arr2 = do
2911+ let marr1 = getMutableByteArray# (arrContents arr1)
2912+ marr2 = getMutableByteArray# (arrContents arr2)
2913+ !(I# st1#) = arrStart arr1
2914+ !(I# st2#) = arrStart arr2
2915+ !(I# len#) = byteLength arr1
2916+ case compare (byteLength arr1) (byteLength arr2) of
2917+ EQ -> do
2918+ r <- liftIO $ IO $ \s# ->
2919+ let res =
2920+ I#
2921+ (compareByteArrays#
2922+ (unsafeCoerce# marr1)
2923+ st1#
2924+ (unsafeCoerce# marr2)
2925+ st2#
2926+ len#)
2927+ in (# s#, res #)
2928+ return $ compare r 0
2929+ x -> return x
2930+
23552931{-# INLINE cmp #-}
2932+{-# DEPRECATED cmp "Please use byteCmp instead." #-}
23562933cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
2357−cmp arr1 arr2 =
2358− liftIO
2359− $ do
2360− let marr1 = getMutableByteArray# (arrContents arr1)
2361− marr2 = getMutableByteArray# (arrContents arr2)
2362− !(I# st1#) = arrStart arr1
2363− !(I# st2#) = arrStart arr2
2364− !(I# len#) = byteLength arr1
2365− case compare (byteLength arr1) (byteLength arr2) of
2366− EQ -> do
2367− r <- liftIO $ IO $ \s# ->
2368− let res =
2369− I#
2370− (compareByteArrays#
2371− (unsafeCoerce# marr1)
2372− st1#
2373− (unsafeCoerce# marr2)
2374− st2#
2375− len#)
2376− in (# s#, res #)
2377− return $ compare r 0
2378− x -> return x
2934+cmp = byteCmp
2935+
2936+-- | Byte equality of two arrays.
2937+--
2938+-- >>> byteEq arr1 arr2 = (==) EQ $ MArray.byteCmp arr1 arr2
2939+--
2940+-- /Unsafe/: See 'byteCmp'.
2941+{-# INLINE byteEq #-}
2942+byteEq :: MonadIO m => MutArray a -> MutArray a -> m Bool
2943+byteEq arr1 arr2 = fmap (EQ ==) $ byteCmp arr1 arr2
2944+
2945+-------------------------------------------------------------------------------
2946+-- Compact
2947+-------------------------------------------------------------------------------
2948+
2949+-- Note: LE versions avoid an extra copy compared to GE. LE parser trades
2950+-- backtracking one array in lieu of avoiding a copy. However, LE and GE both
2951+-- can leave some memory unused. They can split the last array to fit it
2952+-- exactly in the space.
2953+
2954+{-# INLINE_NORMAL pCompactLeAs #-}
2955+pCompactLeAs ::
2956+ forall m a. (MonadIO m, Unbox a)
2957+ => PinnedState -> Int -> Parser (MutArray a) m (MutArray a)
2958+pCompactLeAs ps maxElems = Parser step initial extract
2959+
2960+ where
2961+
2962+ maxBytes = maxElems * SIZE_OF(a)
2963+
2964+ functionName = "Streamly.Internal.Data.MutArray.pCompactLE"
2965+
2966+ initial =
2967+ return
2968+ $ if maxElems <= 0
2969+ then error
2970+ $ functionName
2971+ ++ ": the size of arrays ["
2972+ ++ show maxElems ++ "] must be a natural number"
2973+ else Parser.IPartial Nothing
2974+
2975+ step Nothing arr =
2976+ return
2977+ $ let len = byteLength arr
2978+ in if len >= maxBytes
2979+ then Parser.Done 0 arr
2980+ else Parser.Partial 0 (Just arr)
2981+ -- XXX Split the last array to use the space more compactly.
2982+ step (Just buf) arr =
2983+ let len = byteLength buf + byteLength arr
2984+ in if len > maxBytes
2985+ then return $ Parser.Done 1 buf
2986+ else do
2987+ buf1 <-
2988+ if byteCapacity buf < maxBytes
2989+ then liftIO $ reallocExplicitAs
2990+ ps (SIZE_OF(a)) maxBytes buf
2991+ else return buf
2992+ buf2 <- spliceUnsafe buf1 arr
2993+ return $ Parser.Partial 0 (Just buf2)
2994+
2995+ extract Nothing = return $ Parser.Done 0 nil
2996+ extract (Just buf) = return $ Parser.Done 0 buf
2997+
2998+-- | Parser @pCompactLE maxElems@ coalesces adjacent arrays in the input stream
2999+-- only if the combined size would be less than or equal to @maxElems@
3000+-- elements. Note that it won't split an array if the original array is already
3001+-- larger than maxElems.
3002+--
3003+-- @maxElems@ must be greater than 0.
3004+--
3005+-- Generates unpinned arrays irrespective of the pinning status of input
3006+-- arrays.
3007+--
3008+-- /Internal/
3009+{-# INLINE pCompactLE #-}
3010+pCompactLE ::
3011+ forall m a. (MonadIO m, Unbox a)
3012+ => Int -> Parser (MutArray a) m (MutArray a)
3013+pCompactLE = pCompactLeAs Unpinned
3014+
3015+-- | Pinned version of 'pCompactLE'.
3016+{-# INLINE pPinnedCompactLE #-}
3017+pPinnedCompactLE ::
3018+ forall m a. (MonadIO m, Unbox a)
3019+ => Int -> Parser (MutArray a) m (MutArray a)
3020+pPinnedCompactLE = pCompactLeAs Pinned
3021+
3022+data SpliceState s arr
3023+ = SpliceInitial s
3024+ | SpliceBuffering s arr
3025+ | SpliceYielding arr (SpliceState s arr)
3026+ | SpliceFinish
3027+
3028+-- This mutates the first array (if it has space) to append values from the
3029+-- second one. This would work for immutable arrays as well because an
3030+-- immutable array never has additional space so a new array is allocated
3031+-- instead of mutating it.
3032+
3033+{-# INLINE_NORMAL compactLeAs #-}
3034+compactLeAs :: forall m a. (MonadIO m, Unbox a)
3035+ => PinnedState -> Int -> D.Stream m (MutArray a) -> D.Stream m (MutArray a)
3036+compactLeAs ps maxElems (D.Stream step state) =
3037+ D.Stream step' (SpliceInitial state)
3038+
3039+ where
3040+
3041+ maxBytes = maxElems * SIZE_OF(a)
3042+
3043+ functionName = "Streamly.Internal.Data.MutArray.rCompactLE"
3044+
3045+ {-# INLINE_LATE step' #-}
3046+ step' gst (SpliceInitial st) = do
3047+ when (maxElems <= 0) $
3048+ -- XXX we can pass the module string from the higher level API
3049+ error $ functionName ++ ": the size of arrays [" ++ show maxElems
3050+ ++ "] must be a natural number"
3051+ r <- step gst st
3052+ case r of
3053+ D.Yield arr s -> return $
3054+ let len = byteLength arr
3055+ in if len >= maxBytes
3056+ then D.Skip (SpliceYielding arr (SpliceInitial s))
3057+ else D.Skip (SpliceBuffering s arr)
3058+ D.Skip s -> return $ D.Skip (SpliceInitial s)
3059+ D.Stop -> return D.Stop
3060+
3061+ -- XXX Split the last array to use the space more compactly.
3062+ step' gst (SpliceBuffering st buf) = do
3063+ r <- step gst st
3064+ case r of
3065+ D.Yield arr s -> do
3066+ let len = byteLength buf + byteLength arr
3067+ if len > maxBytes
3068+ then return $
3069+ D.Skip (SpliceYielding buf (SpliceBuffering s arr))
3070+ else do
3071+ buf1 <- if byteCapacity buf < maxBytes
3072+ then liftIO $ reallocExplicitAs
3073+ ps (SIZE_OF(a)) maxBytes buf
3074+ else return buf
3075+ buf2 <- spliceUnsafe buf1 arr
3076+ return $ D.Skip (SpliceBuffering s buf2)
3077+ D.Skip s -> return $ D.Skip (SpliceBuffering s buf)
3078+ D.Stop -> return $ D.Skip (SpliceYielding buf SpliceFinish)
3079+
3080+ step' _ SpliceFinish = return D.Stop
3081+
3082+ step' _ (SpliceYielding arr next) = return $ D.Yield arr next
3083+
3084+
3085+{-# INLINE_NORMAL fCompactGeAs #-}
3086+fCompactGeAs ::
3087+ forall m a. (MonadIO m, Unbox a)
3088+ => PinnedState -> Int -> FL.Fold m (MutArray a) (MutArray a)
3089+fCompactGeAs ps minElems = Fold step initial extract extract
3090+
3091+ where
3092+
3093+ minBytes = minElems * SIZE_OF(a)
3094+
3095+ functionName = "Streamly.Internal.Data.MutArray.fCompactGE"
3096+
3097+ initial =
3098+ return
3099+ $ if minElems < 0
3100+ then error
3101+ $ functionName
3102+ ++ ": the size of arrays ["
3103+ ++ show minElems ++ "] must be a natural number"
3104+ else FL.Partial Nothing
3105+
3106+ step Nothing arr =
3107+ return
3108+ $ let len = byteLength arr
3109+ in if len >= minBytes
3110+ then FL.Done arr
3111+ else FL.Partial (Just arr)
3112+ -- XXX Buffer arrays as a list to avoid copy and reallocations
3113+ step (Just buf) arr = do
3114+ let len = byteLength buf + byteLength arr
3115+ buf1 <-
3116+ if byteCapacity buf < len
3117+ then liftIO $ reallocExplicitAs
3118+ ps (SIZE_OF(a)) (max minBytes len) buf
3119+ else return buf
3120+ buf2 <- spliceUnsafe buf1 arr
3121+ if len >= minBytes
3122+ then return $ FL.Done buf2
3123+ else return $ FL.Partial (Just buf2)
3124+
3125+ extract Nothing = return nil
3126+ extract (Just buf) = return buf
3127+
3128+-- | Fold @fCompactGE minElems@ coalesces adjacent arrays in the input stream
3129+-- until the size becomes greater than or equal to @minElems@.
3130+--
3131+-- Generates unpinned arrays irrespective of the pinning status of input
3132+-- arrays.
3133+{-# INLINE fCompactGE #-}
3134+fCompactGE ::
3135+ forall m a. (MonadIO m, Unbox a)
3136+ => Int -> FL.Fold m (MutArray a) (MutArray a)
3137+fCompactGE = fCompactGeAs Unpinned
3138+
3139+-- | Pinned version of 'fCompactGE'.
3140+{-# INLINE fPinnedCompactGE #-}
3141+fPinnedCompactGE ::
3142+ forall m a. (MonadIO m, Unbox a)
3143+ => Int -> FL.Fold m (MutArray a) (MutArray a)
3144+fPinnedCompactGE = fCompactGeAs Pinned
3145+
3146+{-# INLINE_NORMAL lCompactGeAs #-}
3147+lCompactGeAs :: forall m a. (MonadIO m, Unbox a)
3148+ => PinnedState -> Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
3149+-- The fold version turns out to be a little bit slower.
3150+-- lCompactGeAs ps n = FL.many (fCompactGeAs ps n)
3151+lCompactGeAs ps minElems (Fold step1 initial1 _ final1) =
3152+ Fold step initial extract final
3153+
3154+ where
3155+
3156+ minBytes = minElems * SIZE_OF(a)
3157+
3158+ functionName = "Streamly.Internal.Data.MutArray.lCompactGE"
3159+
3160+ initial = do
3161+ when (minElems <= 0) $
3162+ -- XXX we can pass the module string from the higher level API
3163+ error $ functionName ++ ": the size of arrays ["
3164+ ++ show minElems ++ "] must be a natural number"
3165+
3166+ r <- initial1
3167+ return $ first (Tuple' Nothing) r
3168+
3169+ {-# INLINE runInner #-}
3170+ runInner len acc buf =
3171+ if len >= minBytes
3172+ then do
3173+ r <- step1 acc buf
3174+ case r of
3175+ FL.Done _ -> return $ FL.Done ()
3176+ FL.Partial s -> do
3177+ _ <- final1 s
3178+ res <- initial1
3179+ return $ first (Tuple' Nothing) res
3180+ else return $ FL.Partial $ Tuple' (Just buf) acc
3181+
3182+ step (Tuple' Nothing r1) arr =
3183+ runInner (byteLength arr) r1 arr
3184+
3185+ -- XXX Buffer arrays as a list to avoid copy and reallocations
3186+ step (Tuple' (Just buf) r1) arr = do
3187+ let len = byteLength buf + byteLength arr
3188+ buf1 <- if byteCapacity buf < len
3189+ then liftIO $ reallocExplicitAs
3190+ ps (SIZE_OF(a)) (max minBytes len) buf
3191+ else return buf
3192+ buf2 <- spliceUnsafe buf1 arr
3193+ runInner len r1 buf2
3194+
3195+ -- XXX Several folds do extract >=> final, therefore, we need to make final
3196+ -- return "m b" rather than using extract post it if we want extract to be
3197+ -- partial.
3198+ --
3199+ -- extract forces the pending buffer to be sent to the fold which is not
3200+ -- what we want.
3201+ extract _ = error "lCompactGE: not designed for scanning"
3202+
3203+ final (Tuple' Nothing r1) = final1 r1
3204+ final (Tuple' (Just buf) r1) = do
3205+ r <- step1 r1 buf
3206+ case r of
3207+ FL.Partial rr -> final1 rr
3208+ FL.Done _ -> return ()
3209+
3210+-- | Like 'compactGE' but for transforming folds instead of stream.
3211+--
3212+-- >>> lCompactGE n = Fold.many (MutArray.fCompactGE n)
3213+--
3214+-- Generates unpinned arrays irrespective of the pinning status of input
3215+-- arrays.
3216+{-# INLINE lCompactGE #-}
3217+lCompactGE :: forall m a. (MonadIO m, Unbox a)
3218+ => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
3219+lCompactGE = lCompactGeAs Unpinned
3220+
3221+-- | Pinned version of 'lCompactGE'.
3222+{-# INLINE lPinnedCompactGE #-}
3223+lPinnedCompactGE :: forall m a. (MonadIO m, Unbox a)
3224+ => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
3225+lPinnedCompactGE = lCompactGeAs Pinned
3226+
3227+-- | @compactGE n stream@ coalesces adjacent arrays in the @stream@ until
3228+-- the size becomes greater than or equal to @n@.
3229+--
3230+-- >>> compactGE n = Stream.foldMany (MutArray.fCompactGE n)
3231+--
3232+{-# INLINE compactGE #-}
3233+compactGE ::
3234+ (MonadIO m, Unbox a)
3235+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
3236+compactGE n = D.foldMany (fCompactGE n)
3237+
3238+-- | 'compactEQ n' coalesces adajacent arrays in the input stream to
3239+-- arrays of exact size @n@.
3240+--
3241+-- /Unimplemented/
3242+{-# INLINE compactEQ #-}
3243+compactEQ :: -- (MonadIO m, Unbox a) =>
3244+ Int -> Stream m (MutArray a) -> Stream m (MutArray a)
3245+compactEQ _n = undefined -- D.parseManyD (pCompactEQ n)
23793246
23803247-------------------------------------------------------------------------------
2381−-- NFData
3248+-- In-place mutation algorithms
23823249-------------------------------------------------------------------------------
23833250
23843251-- | Strip elements which match with predicate from both ends.
@@ -2407,7 +3274,7 @@ strip eq arr@MutArray{..} = liftIO $ do
24073274 getStart cur = do
24083275 if cur < arrEnd
24093276 then do
2410− r <- peekByteIndex cur arrContents
3277+ r <- peekAt cur arrContents
24113278 if eq r
24123279 then getStart (INDEX_NEXT(cur,a))
24133280 else return cur
@@ -2417,7 +3284,7 @@ strip eq arr@MutArray{..} = liftIO $ do
24173284 if cur > low
24183285 then do
24193286 let prev = INDEX_PREV(cur,a)
2420− r <- peekByteIndex prev arrContents
3287+ r <- peekAt prev arrContents
24213288 if eq r
24223289 then getLast prev low
24233290 else return cur
core/src/Streamly/Internal/Data/MutByteArray.hsnew file+251
@@ -0,0 +1,251 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+-- This is required as all the instances in this module are orphan instances.
4+{-# OPTIONS_GHC -fno-warn-orphans #-}
5+
6+-- |
7+-- Module : Streamly.Internal.Data.MutByteArray
8+-- Copyright : (c) 2023 Composewell Technologies
9+-- License : BSD3-3-Clause
10+-- Maintainer : streamly@composewell.com
11+-- Portability : GHC
12+--
13+
14+module Streamly.Internal.Data.MutByteArray
15+ (
16+ -- * MutByteArray
17+ module Streamly.Internal.Data.MutByteArray.Type
18+ -- * Unbox
19+ , module Streamly.Internal.Data.Unbox
20+ , module Streamly.Internal.Data.Unbox.TH
21+ -- * Serialize
22+ , module Streamly.Internal.Data.Serialize.Type
23+ -- * Serialize TH
24+ , module Streamly.Internal.Data.Serialize.TH
25+ ) where
26+
27+--------------------------------------------------------------------------------
28+-- Imports
29+--------------------------------------------------------------------------------
30+
31+import Data.Proxy (Proxy(..))
32+import Streamly.Internal.Data.Array (Array(..))
33+import GHC.Exts (Int(..), sizeofByteArray#, unsafeCoerce#)
34+import GHC.Word (Word8)
35+
36+#if __GLASGOW_HASKELL__ >= 900
37+import GHC.Num.Integer (Integer(..))
38+#else
39+import GHC.Integer.GMP.Internals (Integer(..), BigNat(..))
40+#endif
41+
42+import Streamly.Internal.Data.MutByteArray.Type
43+import Streamly.Internal.Data.Serialize.TH
44+import Streamly.Internal.Data.Serialize.Type
45+import Streamly.Internal.Data.Unbox
46+import Streamly.Internal.Data.Unbox.TH
47+
48+--------------------------------------------------------------------------------
49+-- Common instances
50+--------------------------------------------------------------------------------
51+
52+-- Note
53+-- ====
54+--
55+-- Even a non-functional change such as changing the order of constructors will
56+-- change the instance derivation.
57+--
58+-- This will not pose a problem if both, encode, and decode are done by the same
59+-- version of the application. There *might* be a problem if version that
60+-- encodes differs from the version that decodes.
61+--
62+-- We need to add some compatibility tests using different versions of
63+-- dependencies.
64+--
65+-- Although such chages for the most basic types won't happen we need to detect
66+-- if it ever happens.
67+--
68+-- Should we worry about these kind of changes and this kind of compatibility?
69+-- This is a problem for all types of derivations that depend on the order of
70+-- constructors, for example, Enum.
71+
72+-- Note on Windows build
73+-- =====================
74+--
75+-- On Windows, having template haskell splices here fail the build with the
76+-- following error:
77+--
78+-- @
79+-- addLibrarySearchPath: C:\... (Win32 error 3): The system cannot find the path specified.
80+-- @
81+--
82+-- The error might be irrelavant but having these splices triggers it. We should
83+-- either fix the problem or avoid the use to template haskell splices in this
84+-- file.
85+--
86+-- Similar issue: https://github.com/haskell/cabal/issues/4741
87+
88+-- $(Serialize.deriveSerialize ''Maybe)
89+instance Serialize a => Serialize (Maybe a) where
90+
91+ {-# INLINE addSizeTo #-}
92+ addSizeTo acc x =
93+ case x of
94+ Nothing -> (acc + 1)
95+ Just field0 -> (addSizeTo (acc + 1)) field0
96+
97+ {-# INLINE deserializeAt #-}
98+ deserializeAt initialOffset arr endOffset = do
99+ (i0, tag) <- ((deserializeAt initialOffset) arr) endOffset
100+ case tag :: Word8 of
101+ 0 -> pure (i0, Nothing)
102+ 1 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
103+ pure (i1, Just a0)
104+ _ -> error "Found invalid tag while peeking (Maybe a)"
105+
106+ {-# INLINE serializeAt #-}
107+ serializeAt initialOffset arr val =
108+ case val of
109+ Nothing -> do
110+ i0 <- ((serializeAt initialOffset) arr) (0 :: Word8)
111+ pure i0
112+ Just field0 -> do
113+ i0 <- ((serializeAt initialOffset) arr) (1 :: Word8)
114+ i1 <- ((serializeAt i0) arr) field0
115+ pure i1
116+
117+-- $(Serialize.deriveSerialize ''Either)
118+instance (Serialize a, Serialize b) => Serialize (Either a b) where
119+
120+ {-# INLINE addSizeTo #-}
121+ addSizeTo acc x =
122+ case x of
123+ Left field0 -> (addSizeTo (acc + 1)) field0
124+ Right field0 -> (addSizeTo (acc + 1)) field0
125+
126+ {-# INLINE deserializeAt #-}
127+ deserializeAt initialOffset arr endOffset = do
128+ (i0, tag) <- ((deserializeAt initialOffset) arr) endOffset
129+ case tag :: Word8 of
130+ 0 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
131+ pure (i1, Left a0)
132+ 1 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
133+ pure (i1, Right a0)
134+ _ -> error "Found invalid tag while peeking (Either a b)"
135+
136+ {-# INLINE serializeAt #-}
137+ serializeAt initialOffset arr val =
138+ case val of
139+ Left field0 -> do
140+ i0 <- ((serializeAt initialOffset) arr) (0 :: Word8)
141+ i1 <- ((serializeAt i0) arr) field0
142+ pure i1
143+ Right field0 -> do
144+ i0 <- ((serializeAt initialOffset) arr) (1 :: Word8)
145+ i1 <- ((serializeAt i0) arr) field0
146+ pure i1
147+
148+instance Serialize (Proxy a) where
149+
150+ {-# INLINE addSizeTo #-}
151+ addSizeTo acc _ = (acc + 1)
152+
153+ {-# INLINE deserializeAt #-}
154+ deserializeAt initialOffset _ _ = pure ((initialOffset + 1), Proxy)
155+
156+ {-# INLINE serializeAt #-}
157+ serializeAt initialOffset _ _ = pure (initialOffset + 1)
158+
159+--------------------------------------------------------------------------------
160+-- Integer
161+--------------------------------------------------------------------------------
162+
163+data LiftedInteger
164+ = LIS Int
165+ | LIP (Array Word)
166+ | LIN (Array Word)
167+
168+-- $(Serialize.deriveSerialize ''LiftedInteger)
169+instance Serialize LiftedInteger where
170+
171+ {-# INLINE addSizeTo #-}
172+ addSizeTo acc x =
173+ case x of
174+ LIS field0 -> (addSizeTo (acc + 1)) field0
175+ LIP field0 -> (addSizeTo (acc + 1)) field0
176+ LIN field0 -> (addSizeTo (acc + 1)) field0
177+
178+ {-# INLINE deserializeAt #-}
179+ deserializeAt initialOffset arr endOffset = do
180+ (i0, tag) <- ((deserializeAt initialOffset) arr) endOffset
181+ case tag :: Word8 of
182+ 0 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
183+ pure (i1, LIS a0)
184+ 1 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
185+ pure (i1, LIP a0)
186+ 2 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
187+ pure (i1, LIN a0)
188+ _ -> error "Found invalid tag while peeking (LiftedInteger)"
189+
190+ {-# INLINE serializeAt #-}
191+ serializeAt initialOffset arr val =
192+ case val of
193+ LIS field0 -> do
194+ i0 <- ((serializeAt initialOffset) arr) (0 :: Word8)
195+ i1 <- ((serializeAt i0) arr) field0
196+ pure i1
197+ LIP field0 -> do
198+ i0 <- ((serializeAt initialOffset) arr) (1 :: Word8)
199+ i1 <- ((serializeAt i0) arr) field0
200+ pure i1
201+ LIN field0 -> do
202+ i0 <- ((serializeAt initialOffset) arr) (2 :: Word8)
203+ i1 <- ((serializeAt i0) arr) field0
204+ pure i1
205+
206+#if __GLASGOW_HASKELL__ >= 900
207+
208+{-# INLINE liftInteger #-}
209+liftInteger :: Integer -> LiftedInteger
210+liftInteger (IS x) = LIS (I# x)
211+liftInteger (IP x) =
212+ LIP (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
213+liftInteger (IN x) =
214+ LIN (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
215+
216+{-# INLINE unliftInteger #-}
217+unliftInteger :: LiftedInteger -> Integer
218+unliftInteger (LIS (I# x)) = IS x
219+unliftInteger (LIP (Array (MutByteArray x) _ _)) = IP (unsafeCoerce# x)
220+unliftInteger (LIN (Array (MutByteArray x) _ _)) = IN (unsafeCoerce# x)
221+
222+#else
223+
224+{-# INLINE liftInteger #-}
225+liftInteger :: Integer -> LiftedInteger
226+liftInteger (S# x) = LIS (I# x)
227+liftInteger (Jp# (BN# x)) =
228+ LIP (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
229+liftInteger (Jn# (BN# x)) =
230+ LIN (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
231+
232+{-# INLINE unliftInteger #-}
233+unliftInteger :: LiftedInteger -> Integer
234+unliftInteger (LIS (I# x)) = S# x
235+unliftInteger (LIP (Array (MutByteArray x) _ _)) =
236+ Jp# (BN# (unsafeCoerce# x))
237+unliftInteger (LIN (Array (MutByteArray x) _ _)) =
238+ Jn# (BN# (unsafeCoerce# x))
239+
240+#endif
241+
242+instance Serialize Integer where
243+ {-# INLINE addSizeTo #-}
244+ addSizeTo i a = addSizeTo i (liftInteger a)
245+
246+ {-# INLINE deserializeAt #-}
247+ deserializeAt off arr end =
248+ fmap unliftInteger <$> deserializeAt off arr end
249+
250+ {-# INLINE serializeAt #-}
251+ serializeAt off arr val = serializeAt off arr (liftInteger val)
core/src/Streamly/Internal/Data/MutByteArray/Type.hsnew file+321
@@ -0,0 +1,321 @@
1+{-# LANGUAGE UnboxedTuples #-}
2+
3+-- |
4+-- Module : Streamly.Internal.Data.MutByteArray.Type
5+-- Copyright : (c) 2023 Composewell Technologies
6+-- License : BSD3-3-Clause
7+-- Maintainer : streamly@composewell.com
8+-- Stability : experimental
9+-- Portability : GHC
10+--
11+module Streamly.Internal.Data.MutByteArray.Type
12+ (
13+ -- ** MutByteArray
14+ MutByteArray(..)
15+ , MutableByteArray
16+ , getMutableByteArray# -- XXX getMutByteArray#
17+
18+ -- ** Pinning
19+ , PinnedState(..)
20+ , isPinned
21+ , pin
22+ , unpin
23+
24+ -- ** Allocation
25+ , empty
26+ , newBytesAs -- XXX should be removed
27+ , new
28+ , pinnedNew
29+ , pinnedNewAlignedBytes -- XXX should be removed
30+
31+ -- ** Access
32+ , sizeOfMutableByteArray -- XXX length
33+ , putSliceUnsafe
34+ , cloneSliceUnsafeAs
35+ , cloneSliceUnsafe
36+ , pinnedCloneSliceUnsafe
37+ , unsafePinnedAsPtr
38+ , unsafeAsPtr
39+
40+ -- ** Deprecated
41+ , asPtrUnsafe
42+ , nil
43+ ) where
44+
45+import Control.Monad.IO.Class (MonadIO(..))
46+#ifdef DEBUG
47+import Control.Monad (when)
48+import Debug.Trace (trace)
49+#endif
50+import GHC.Base (IO(..))
51+import System.IO.Unsafe (unsafePerformIO)
52+
53+import GHC.Exts
54+
55+--------------------------------------------------------------------------------
56+-- The ArrayContents type
57+--------------------------------------------------------------------------------
58+
59+data PinnedState
60+ = Pinned
61+ | Unpinned deriving (Show, Eq)
62+
63+-- XXX can use UnliftedNewtypes
64+
65+-- | A lifted mutable byte array type wrapping @MutableByteArray# RealWorld@.
66+-- This is a low level array used to back high level unboxed arrays and
67+-- serialized data.
68+data MutByteArray = MutByteArray (MutableByteArray# RealWorld)
69+
70+{-# DEPRECATED MutableByteArray "Please use MutByteArray instead" #-}
71+type MutableByteArray = MutByteArray
72+
73+{-# INLINE getMutableByteArray# #-}
74+getMutableByteArray# :: MutByteArray -> MutableByteArray# RealWorld
75+getMutableByteArray# (MutByteArray mbarr) = mbarr
76+
77+-- | Return the size of the array in bytes.
78+{-# INLINE sizeOfMutableByteArray #-}
79+sizeOfMutableByteArray :: MutByteArray -> IO Int
80+sizeOfMutableByteArray (MutByteArray arr) =
81+ IO $ \s ->
82+ case getSizeofMutableByteArray# arr s of
83+ (# s1, i #) -> (# s1, I# i #)
84+
85+{-# INLINE touch #-}
86+touch :: MutByteArray -> IO ()
87+touch (MutByteArray contents) =
88+ IO $ \s -> case touch# contents s of s' -> (# s', () #)
89+
90+-- XXX Some functions in this module are "IO" and others are "m", we need to
91+-- make it consistent.
92+
93+-- XXX We can provide another API for "unsafe" FFI calls passing an unlifted
94+-- pointer to the FFI call. For unsafe calls we do not need to pin the array.
95+-- We can pass an unlifted pointer to the FFI routine to avoid GC kicking in
96+-- before the pointer is wrapped.
97+--
98+-- From the GHC manual:
99+--
100+-- GHC, since version 8.4, guarantees that garbage collection will never occur
101+-- during an unsafe call, even in the bytecode interpreter, and further
102+-- guarantees that unsafe calls will be performed in the calling thread. Making
103+-- it safe to pass heap-allocated objects to unsafe functions.
104+
105+-- | Use a @MutByteArray@ as @Ptr a@. This is useful when we want to pass
106+-- an array as a pointer to some operating system call or to a "safe" FFI call.
107+--
108+-- If the array is not pinned it is copied to pinned memory before passing it
109+-- to the monadic action.
110+--
111+-- /Performance Notes:/ Forces a copy if the array is not pinned. It is advised
112+-- that the programmer keeps this in mind and creates a pinned array
113+-- opportunistically before this operation occurs, to avoid the cost of a copy
114+-- if possible.
115+--
116+-- /Unsafe/ because of direct pointer operations. The user must ensure that
117+-- they are writing within the legal bounds of the array.
118+--
119+-- /Pre-release/
120+--
121+{-# INLINE unsafePinnedAsPtr #-}
122+unsafePinnedAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
123+unsafePinnedAsPtr arr f = do
124+ contents <- liftIO $ pin arr
125+ let !ptr = Ptr (byteArrayContents#
126+ (unsafeCoerce# (getMutableByteArray# contents)))
127+ r <- f ptr
128+ liftIO $ touch contents
129+ return r
130+
131+{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
132+{-# INLINE asPtrUnsafe #-}
133+asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
134+asPtrUnsafe = unsafePinnedAsPtr
135+
136+-- | For use with unsafe FFI functions. Does not force pin the array memory.
137+{-# INLINE unsafeAsPtr #-}
138+unsafeAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
139+unsafeAsPtr arr f = do
140+ let !ptr = Ptr (byteArrayContents#
141+ (unsafeCoerce# (getMutableByteArray# arr)))
142+ r <- f ptr
143+ liftIO $ touch arr
144+ return r
145+
146+--------------------------------------------------------------------------------
147+-- Creation
148+--------------------------------------------------------------------------------
149+
150+{-# NOINLINE empty #-}
151+empty :: MutByteArray
152+empty = unsafePerformIO $ new 0
153+
154+{-# DEPRECATED nil "Please use empty instead" #-}
155+nil :: MutByteArray
156+nil = empty
157+
158+-- XXX add "newRounded" to round up the large size to the next page boundary
159+-- and return the allocated size.
160+{-# INLINE new #-}
161+new :: Int -> IO MutByteArray
162+new nbytes | nbytes < 0 =
163+ errorWithoutStackTrace "newByteArray: size must be >= 0"
164+new (I# nbytes) = IO $ \s ->
165+ case newByteArray# nbytes s of
166+ (# s', mbarr# #) ->
167+ let c = MutByteArray mbarr#
168+ in (# s', c #)
169+
170+{-# INLINE pinnedNew #-}
171+pinnedNew :: Int -> IO MutByteArray
172+pinnedNew nbytes | nbytes < 0 =
173+ errorWithoutStackTrace "pinnedNew: size must be >= 0"
174+pinnedNew (I# nbytes) = IO $ \s ->
175+ case newPinnedByteArray# nbytes s of
176+ (# s', mbarr# #) ->
177+ let c = MutByteArray mbarr#
178+ in (# s', c #)
179+
180+{-# INLINE pinnedNewAlignedBytes #-}
181+pinnedNewAlignedBytes :: Int -> Int -> IO MutByteArray
182+pinnedNewAlignedBytes nbytes _align | nbytes < 0 =
183+ errorWithoutStackTrace "pinnedNewAlignedBytes: size must be >= 0"
184+pinnedNewAlignedBytes (I# nbytes) (I# align) = IO $ \s ->
185+ case newAlignedPinnedByteArray# nbytes align s of
186+ (# s', mbarr# #) ->
187+ let c = MutByteArray mbarr#
188+ in (# s', c #)
189+
190+{-# INLINE newBytesAs #-}
191+newBytesAs :: PinnedState -> Int -> IO MutByteArray
192+newBytesAs Unpinned = new
193+newBytesAs Pinned = pinnedNew
194+
195+-------------------------------------------------------------------------------
196+-- Copying
197+-------------------------------------------------------------------------------
198+
199+-- Note: Array copy is more efficient than streaming copy.
200+-- CopyMutableByteArray# translates to genMemcpy in GHC/CmmToAsm/X86/CodeGen.hs
201+-- glibc memcpy copies bytes/words/pages - unrolls the loops:
202+-- https://github.com/bminor/glibc/blob/4290aed05135ae4c0272006442d147f2155e70d7/string/memcpy.c
203+-- https://github.com/bminor/glibc/blob/4290aed05135ae4c0272006442d147f2155e70d7/string/wordcopy.c
204+
205+-- | Put a sub range of a source array into a subrange of a destination array.
206+-- This is not safe as it does not check the bounds of neither the src array
207+-- nor the destination array.
208+{-# INLINE putSliceUnsafe #-}
209+putSliceUnsafe ::
210+ MonadIO m
211+ => MutByteArray
212+ -> Int
213+ -> MutByteArray
214+ -> Int
215+ -> Int
216+ -> m ()
217+putSliceUnsafe src srcStartBytes dst dstStartBytes lenBytes = liftIO $ do
218+#ifdef DEBUG
219+ srcLen <- sizeOfMutableByteArray src
220+ dstLen <- sizeOfMutableByteArray dst
221+ when (srcLen - srcStartBytes < lenBytes)
222+ $ error $ "putSliceUnsafe: src overflow: start" ++ show srcStartBytes
223+ ++ " end " ++ show srcLen ++ " len " ++ show lenBytes
224+ when (dstLen - dstStartBytes < lenBytes)
225+ $ error $ "putSliceUnsafe: dst overflow: start" ++ show dstStartBytes
226+ ++ " end " ++ show dstLen ++ " len " ++ show lenBytes
227+#endif
228+ let !(I# srcStartBytes#) = srcStartBytes
229+ !(I# dstStartBytes#) = dstStartBytes
230+ !(I# lenBytes#) = lenBytes
231+ let arrS# = getMutableByteArray# src
232+ arrD# = getMutableByteArray# dst
233+ IO $ \s# -> (# copyMutableByteArray#
234+ arrS# srcStartBytes# arrD# dstStartBytes# lenBytes# s#
235+ , () #)
236+
237+-- | Unsafe as it does not check whether the start offset and length supplied
238+-- are valid inside the array.
239+{-# INLINE cloneSliceUnsafeAs #-}
240+cloneSliceUnsafeAs :: MonadIO m =>
241+ PinnedState -> Int -> Int -> MutByteArray -> m MutByteArray
242+cloneSliceUnsafeAs ps srcOff srcLen src =
243+ liftIO $ do
244+ mba <- newBytesAs ps srcLen
245+ putSliceUnsafe src srcOff mba 0 srcLen
246+ return mba
247+
248+-- | @cloneSliceUnsafe offset len arr@ clones a slice of the supplied array
249+-- starting at the given offset and equal to the given length.
250+{-# INLINE cloneSliceUnsafe #-}
251+cloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
252+cloneSliceUnsafe = cloneSliceUnsafeAs Unpinned
253+
254+-- | @pinnedCloneSliceUnsafe offset len arr@
255+{-# INLINE pinnedCloneSliceUnsafe #-}
256+pinnedCloneSliceUnsafe :: MonadIO m =>
257+ Int -> Int -> MutByteArray -> m MutByteArray
258+pinnedCloneSliceUnsafe = cloneSliceUnsafeAs Pinned
259+
260+-------------------------------------------------------------------------------
261+-- Pinning & Unpinning
262+-------------------------------------------------------------------------------
263+
264+-- | Return 'True' if the array is allocated in pinned memory.
265+{-# INLINE isPinned #-}
266+isPinned :: MutByteArray -> Bool
267+isPinned (MutByteArray arr#) =
268+ let pinnedInt = I# (isMutableByteArrayPinned# arr#)
269+ in pinnedInt /= 0
270+
271+
272+{-# INLINE cloneMutableArrayWith# #-}
273+cloneMutableArrayWith#
274+ :: (Int# -> State# RealWorld -> (# State# RealWorld
275+ , MutableByteArray# RealWorld #))
276+ -> MutableByteArray# RealWorld
277+ -> State# RealWorld
278+ -> (# State# RealWorld, MutableByteArray# RealWorld #)
279+cloneMutableArrayWith# alloc# arr# s# =
280+ case getSizeofMutableByteArray# arr# s# of
281+ (# s1#, i# #) ->
282+ case alloc# i# s1# of
283+ (# s2#, arr1# #) ->
284+ case copyMutableByteArray# arr# 0# arr1# 0# i# s2# of
285+ s3# -> (# s3#, arr1# #)
286+
287+-- | Return a copy of the array in pinned memory if unpinned, else return the
288+-- original array.
289+{-# INLINE pin #-}
290+pin :: MutByteArray -> IO MutByteArray
291+pin arr@(MutByteArray marr#) =
292+ if isPinned arr
293+ then return arr
294+ else
295+#ifdef DEBUG
296+ do
297+ -- XXX dump stack trace
298+ trace ("pin: Copying array") (return ())
299+#endif
300+ IO
301+ $ \s# ->
302+ case cloneMutableArrayWith# newPinnedByteArray# marr# s# of
303+ (# s1#, marr1# #) -> (# s1#, MutByteArray marr1# #)
304+
305+-- | Return a copy of the array in unpinned memory if pinned, else return the
306+-- original array.
307+{-# INLINE unpin #-}
308+unpin :: MutByteArray -> IO MutByteArray
309+unpin arr@(MutByteArray marr#) =
310+ if not (isPinned arr)
311+ then return arr
312+ else
313+#ifdef DEBUG
314+ do
315+ -- XXX dump stack trace
316+ trace ("unpin: Copying array") (return ())
317+#endif
318+ IO
319+ $ \s# ->
320+ case cloneMutableArrayWith# newByteArray# marr# s# of
321+ (# s1#, marr1# #) -> (# s1#, MutByteArray marr1# #)
core/src/Streamly/Internal/Data/Parser.hs+126−119
@@ -10,10 +10,13 @@
1010module Streamly.Internal.Data.Parser
1111 (
1212 -- * Setup
13+ -- | To execute the code examples provided in this module in ghci, please
14+ -- run the following commands first.
15+ --
1316 -- $setup
1417
1518 module Streamly.Internal.Data.Parser.Type
16− --, module Streamly.Internal.Data.Parser.Tee
19+ --, module Streamly.Internal.Data.Parser.Tee
1720
1821 -- * Types
1922 , Parser (..)
@@ -281,7 +284,7 @@ import Prelude hiding
281284--
282285{-# INLINE toFold #-}
283286toFold :: Monad m => Parser a m b -> Fold m a b
284−toFold (Parser pstep pinitial pextract) = Fold step initial extract
287+toFold (Parser pstep pinitial pextract) = Fold step initial extract final
285288
286289 where
287290
@@ -309,7 +312,9 @@ toFold (Parser pstep pinitial pextract) = Fold step initial extract
309312 Done n _ -> derror n
310313 Error err -> eerror err
311314
312− extract st = do
315+ extract = error "toFold: parser cannot be used for scanning"
316+
317+ final st = do
313318 r <- pextract st
314319 case r of
315320 Done 0 b -> return b
@@ -327,7 +332,7 @@ toFold (Parser pstep pinitial pextract) = Fold step initial extract
327332--
328333{-# INLINE fromFold #-}
329334fromFold :: Monad m => Fold m a b -> Parser a m b
330−fromFold (Fold fstep finitial fextract) = Parser step initial extract
335+fromFold (Fold fstep finitial _ ffinal) = Parser step initial extract
331336
332337 where
333338
@@ -345,7 +350,7 @@ fromFold (Fold fstep finitial fextract) = Parser step initial extract
345350 FL.Partial s1 -> Partial 0 s1
346351 FL.Done b -> Done 0 b
347352
348− extract = fmap (Done 0) . fextract
353+ extract = fmap (Done 0) . ffinal
349354
350355-- | Convert a Maybe returning fold to an error returning parser. The first
351356-- argument is the error message that the parser would return when the fold
@@ -355,7 +360,7 @@ fromFold (Fold fstep finitial fextract) = Parser step initial extract
355360--
356361{-# INLINE fromFoldMaybe #-}
357362fromFoldMaybe :: Monad m => String -> Fold m a (Maybe b) -> Parser a m b
358−fromFoldMaybe errMsg (Fold fstep finitial fextract) =
363+fromFoldMaybe errMsg (Fold fstep finitial _ ffinal) =
359364 Parser step initial extract
360365
361366 where
@@ -381,7 +386,7 @@ fromFoldMaybe errMsg (Fold fstep finitial fextract) =
381386 Nothing -> Error errMsg
382387
383388 extract s = do
384− res <- fextract s
389+ res <- ffinal s
385390 case res of
386391 Just x -> return $ Done 0 x
387392 Nothing -> return $ Error errMsg
@@ -630,7 +635,7 @@ data Tuple'Fused a b = Tuple'Fused !a !b deriving Show
630635--
631636{-# INLINE takeBetween #-}
632637takeBetween :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
633−takeBetween low high (Fold fstep finitial fextract) =
638+takeBetween low high (Fold fstep finitial _ ffinal) =
634639
635640 Parser step initial (extract streamErr)
636641
@@ -681,9 +686,11 @@ takeBetween low high (Fold fstep finitial fextract) =
681686 in case res of
682687 FL.Partial s -> do
683688 let s1 = Tuple'Fused i1 s
684− if i1 < high
689+ if i1 < low
685690 then return $ Continue 0 s1
686− else extract foldErr s1
691+ else if i1 < high
692+ then return $ Partial 0 s1
693+ else fmap (Done 0) (ffinal s)
687694 FL.Done b ->
688695 return
689696 $ if i1 >= low
@@ -693,12 +700,12 @@ takeBetween low high (Fold fstep finitial fextract) =
693700 step (Tuple'Fused i s) a = fstep s a >>= snext i
694701
695702 extract f (Tuple'Fused i s)
696− | i >= low && i <= high = fmap (Done 0) (fextract s)
703+ | i >= low && i <= high = fmap (Done 0) (ffinal s)
697704 | otherwise = return $ Error (f i)
698705
699706 -- XXX Need to make Initial return type Step to deduplicate this
700707 iextract f (Tuple'Fused i s)
701− | i >= low && i <= high = fmap IDone (fextract s)
708+ | i >= low && i <= high = fmap IDone (ffinal s)
702709 | otherwise = return $ IError (f i)
703710
704711-- | Stops after taking exactly @n@ input elements.
@@ -715,7 +722,7 @@ takeBetween low high (Fold fstep finitial fextract) =
715722--
716723{-# INLINE takeEQ #-}
717724takeEQ :: Monad m => Int -> Fold m a b -> Parser a m b
718−takeEQ n (Fold fstep finitial fextract) = Parser step initial extract
725+takeEQ n (Fold fstep finitial _ ffinal) = Parser step initial extract
719726
720727 where
721728
@@ -725,7 +732,7 @@ takeEQ n (Fold fstep finitial fextract) = Parser step initial extract
725732 FL.Partial s ->
726733 if n > 0
727734 then return $ IPartial $ Tuple'Fused 1 s
728− else fmap IDone (fextract s)
735+ else fmap IDone (ffinal s)
729736 FL.Done b -> return $
730737 if n > 0
731738 then IError
@@ -749,7 +756,7 @@ takeEQ n (Fold fstep finitial fextract) = Parser step initial extract
749756 -- assert (n == i1)
750757 Done 0
751758 <$> case res of
752− FL.Partial s -> fextract s
759+ FL.Partial s -> ffinal s
753760 FL.Done b -> return b
754761
755762 extract (Tuple'Fused i _) =
@@ -781,7 +788,7 @@ data TakeGEState s =
781788--
782789{-# INLINE takeGE #-}
783790takeGE :: Monad m => Int -> Fold m a b -> Parser a m b
784−takeGE n (Fold fstep finitial fextract) = Parser step initial extract
791+takeGE n (Fold fstep finitial _ ffinal) = Parser step initial extract
785792
786793 where
787794
@@ -829,7 +836,7 @@ takeGE n (Fold fstep finitial fextract) = Parser step initial extract
829836 $ Error
830837 $ "takeGE: Expecting at least " ++ show n
831838 ++ " elements, input terminated on " ++ show (i - 1)
832− extract (TakeGEGE r) = fmap (Done 0) $ fextract r
839+ extract (TakeGEGE r) = fmap (Done 0) $ ffinal r
833840
834841-------------------------------------------------------------------------------
835842-- Conditional splitting
@@ -890,7 +897,7 @@ takeWhileP predicate (Parser pstep pinitial pextract) =
890897{-# INLINE takeWhile #-}
891898takeWhile :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
892899-- takeWhile cond f = takeWhileP cond (fromFold f)
893−takeWhile predicate (Fold fstep finitial fextract) =
900+takeWhile predicate (Fold fstep finitial _ ffinal) =
894901 Parser step initial extract
895902
896903 where
@@ -909,9 +916,9 @@ takeWhile predicate (Fold fstep finitial fextract) =
909916 $ case fres of
910917 FL.Partial s1 -> Partial 0 s1
911918 FL.Done b -> Done 0 b
912− else Done 1 <$> fextract s
919+ else Done 1 <$> ffinal s
913920
914− extract s = fmap (Done 0) (fextract s)
921+ extract s = fmap (Done 0) (ffinal s)
915922
916923{-
917924-- XXX This may not be composable because of the b argument. We can instead
@@ -931,7 +938,7 @@ takeWhile1 acc cond f = undefined
931938{-# INLINE takeWhile1 #-}
932939takeWhile1 :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
933940-- takeWhile1 cond f = takeWhileP cond (takeBetween 1 maxBound f)
934−takeWhile1 predicate (Fold fstep finitial fextract) =
941+takeWhile1 predicate (Fold fstep finitial _ ffinal) =
935942 Parser step initial extract
936943
937944 where
@@ -961,11 +968,11 @@ takeWhile1 predicate (Fold fstep finitial fextract) =
961968 if predicate a
962969 then process s a
963970 else do
964− b <- fextract s
971+ b <- ffinal s
965972 return $ Done 1 b
966973
967974 extract (Left' _) = return $ Error "takeWhile1: end of input"
968− extract (Right' s) = fmap (Done 0) (fextract s)
975+ extract (Right' s) = fmap (Done 0) (ffinal s)
969976
970977-- | Drain the input as long as the predicate succeeds, running the effects and
971978-- discarding the results.
@@ -994,7 +1001,7 @@ takeFramedByGeneric :: Monad m =>
9941001 -> Maybe (a -> Bool) -- is frame end?
9951002 -> Fold m a b
9961003 -> Parser a m b
997−takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
1004+takeFramedByGeneric esc begin end (Fold fstep finitial _ ffinal) =
9981005
9991006 Parser step initial extract
10001007
@@ -1026,20 +1033,20 @@ takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
10261033 if isEnd a
10271034 then
10281035 if n == 0
1029− then Done 0 <$> fextract s
1036+ then Done 0 <$> ffinal s
10301037 else process s a (n - 1)
10311038 else
10321039 let n1 = if isBegin a then n + 1 else n
10331040 in process s a n1
10341041 Nothing -> -- takeEndBy case
10351042 if isEnd a
1036− then Done 0 <$> fextract s
1043+ then Done 0 <$> ffinal s
10371044 else process s a n
10381045 Nothing -> -- takeStartBy case
10391046 case begin of
10401047 Just isBegin ->
10411048 if isBegin a
1042− then Done 0 <$> fextract s
1049+ then Done 0 <$> ffinal s
10431050 else process s a n
10441051 Nothing ->
10451052 error $ "takeFramedByGeneric: "
@@ -1064,7 +1071,7 @@ takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
10641071 case end of
10651072 Just isEnd ->
10661073 if isEnd a
1067− then Done 0 <$> fextract s
1074+ then Done 0 <$> ffinal s
10681075 else processCheckEsc s a 0
10691076 Nothing ->
10701077 error "Both begin and end frame predicate missing"
@@ -1079,7 +1086,7 @@ takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
10791086 case begin of
10801087 Just _ ->
10811088 case end of
1082− Nothing -> fmap (Done 0) $ fextract s
1089+ Nothing -> fmap (Done 0) $ ffinal s
10831090 Just _ -> err "takeFramedByGeneric: missing frame end"
10841091 Nothing -> err "takeFramedByGeneric: missing closing frame"
10851092 extract (FrameEscEsc _ _) = err "takeFramedByGeneric: trailing escape"
@@ -1118,7 +1125,7 @@ blockWithQuotes :: (Monad m, Eq a) =>
11181125 -> Fold m a b
11191126 -> Parser a m b
11201127blockWithQuotes isEsc isQuote bopen bclose
1121− (Fold fstep finitial fextract) =
1128+ (Fold fstep finitial _ ffinal) =
11221129 Parser step initial extract
11231130
11241131 where
@@ -1148,7 +1155,7 @@ blockWithQuotes isEsc isQuote bopen bclose
11481155 | a == bopen = process s a (BlockUnquoted (level + 1))
11491156 | a == bclose =
11501157 if level == 1
1151− then fmap (Done 0) (fextract s)
1158+ then fmap (Done 0) (ffinal s)
11521159 else process s a (BlockUnquoted (level - 1))
11531160 | isQuote a = process s a (BlockQuoted level)
11541161 | otherwise = process s a (BlockUnquoted level)
@@ -1162,7 +1169,7 @@ blockWithQuotes isEsc isQuote bopen bclose
11621169
11631170 err = return . Error
11641171
1165− extract (BlockInit s) = fmap (Done 0) $ fextract s
1172+ extract (BlockInit s) = fmap (Done 0) $ ffinal s
11661173 extract (BlockUnquoted level _) =
11671174 err $ "blockWithQuotes: finished at block nest level " ++ show level
11681175 extract (BlockQuoted level _) =
@@ -1297,7 +1304,7 @@ takeEitherSepBy _cond = undefined -- D.toParserK . D.takeEitherSepBy cond
12971304--
12981305{-# INLINE takeStartBy #-}
12991306takeStartBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
1300−takeStartBy cond (Fold fstep finitial fextract) =
1307+takeStartBy cond (Fold fstep finitial _ ffinal) =
13011308
13021309 Parser step initial extract
13031310
@@ -1325,10 +1332,10 @@ takeStartBy cond (Fold fstep finitial fextract) =
13251332 step (Right' s) a =
13261333 if not (cond a)
13271334 then process s a
1328− else Done 1 <$> fextract s
1335+ else Done 1 <$> ffinal s
13291336
1330− extract (Left' s) = fmap (Done 0) $ fextract s
1331− extract (Right' s) = fmap (Done 0) $ fextract s
1337+ extract (Left' s) = fmap (Done 0) $ ffinal s
1338+ extract (Right' s) = fmap (Done 0) $ ffinal s
13321339
13331340-- | Like 'takeStartBy' but drops the separator.
13341341--
@@ -1364,7 +1371,7 @@ takeFramedByEsc_ :: Monad m =>
13641371 (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
13651372-- takeFramedByEsc_ isEsc isEnd p =
13661373-- takeFramedByGeneric (Just isEsc) Nothing (Just isEnd) (toFold p)
1367−takeFramedByEsc_ isEsc isBegin isEnd (Fold fstep finitial fextract) =
1374+takeFramedByEsc_ isEsc isBegin isEnd (Fold fstep finitial _ ffinal ) =
13681375
13691376 Parser step initial extract
13701377
@@ -1400,7 +1407,7 @@ takeFramedByEsc_ isEsc isBegin isEnd (Fold fstep finitial fextract) =
14001407 in process s a n1
14011408 else
14021409 if n == 0
1403− then Done 0 <$> fextract s
1410+ then Done 0 <$> ffinal s
14041411 else process s a (n - 1)
14051412 step (FrameEscEsc s n) a = process s a n
14061413
@@ -1422,7 +1429,7 @@ takeFramedBy_ :: Monad m =>
14221429 (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
14231430-- takeFramedBy_ isBegin isEnd =
14241431-- takeFramedByGeneric (Just (const False)) (Just isBegin) (Just isEnd)
1425−takeFramedBy_ isBegin isEnd (Fold fstep finitial fextract) =
1432+takeFramedBy_ isBegin isEnd (Fold fstep finitial _ ffinal) =
14261433
14271434 Parser step initial extract
14281435
@@ -1452,7 +1459,7 @@ takeFramedBy_ isBegin isEnd (Fold fstep finitial fextract) =
14521459 | not (isEnd a) =
14531460 let n1 = if isBegin a then n + 1 else n
14541461 in process s a n1
1455− | n == 0 = Done 0 <$> fextract s
1462+ | n == 0 = Done 0 <$> ffinal s
14561463 | otherwise = process s a (n - 1)
14571464
14581465 err = return . Error
@@ -1487,7 +1494,7 @@ data WordByState s b = WBLeft !s | WBWord !s | WBRight !b
14871494--
14881495{-# INLINE wordBy #-}
14891496wordBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
1490−wordBy predicate (Fold fstep finitial fextract) = Parser step initial extract
1497+wordBy predicate (Fold fstep finitial _ ffinal) = Parser step initial extract
14911498
14921499 where
14931500
@@ -1514,7 +1521,7 @@ wordBy predicate (Fold fstep finitial fextract) = Parser step initial extract
15141521 if not (predicate a)
15151522 then worder s a
15161523 else do
1517− b <- fextract s
1524+ b <- ffinal s
15181525 return $ Partial 0 $ WBRight b
15191526 step (WBRight b) a =
15201527 return
@@ -1522,8 +1529,8 @@ wordBy predicate (Fold fstep finitial fextract) = Parser step initial extract
15221529 then Done 1 b
15231530 else Partial 0 $ WBRight b
15241531
1525− extract (WBLeft s) = fmap (Done 0) $ fextract s
1526− extract (WBWord s) = fmap (Done 0) $ fextract s
1532+ extract (WBLeft s) = fmap (Done 0) $ ffinal s
1533+ extract (WBWord s) = fmap (Done 0) $ ffinal s
15271534 extract (WBRight b) = return (Done 0 b)
15281535
15291536data WordFramedState s b =
@@ -1560,7 +1567,7 @@ wordFramedBy :: Monad m =>
15601567 -> Fold m a b
15611568 -> Parser a m b
15621569wordFramedBy isEsc isBegin isEnd isSep
1563− (Fold fstep finitial fextract) =
1570+ (Fold fstep finitial _ ffinal) =
15641571 Parser step initial extract
15651572
15661573 where
@@ -1591,7 +1598,7 @@ wordFramedBy isEsc isBegin isEnd isSep
15911598 step (WordFramedWord s n) a
15921599 | isEsc a = return $ Continue 0 $ WordFramedEsc s n
15931600 | n == 0 && isSep a = do
1594− b <- fextract s
1601+ b <- ffinal s
15951602 return $ Partial 0 $ WordFramedSkipPost b
15961603 | otherwise = do
15971604 -- We need to use different order for checking begin and end for
@@ -1624,10 +1631,10 @@ wordFramedBy isEsc isBegin isEnd isSep
16241631
16251632 err = return . Error
16261633
1627− extract (WordFramedSkipPre s) = fmap (Done 0) $ fextract s
1634+ extract (WordFramedSkipPre s) = fmap (Done 0) $ ffinal s
16281635 extract (WordFramedWord s n) =
16291636 if n == 0
1630− then fmap (Done 0) $ fextract s
1637+ then fmap (Done 0) $ ffinal s
16311638 else err "wordFramedBy: missing frame end"
16321639 extract (WordFramedEsc _ _) =
16331640 err "wordFramedBy: trailing escape"
@@ -1714,7 +1721,7 @@ wordWithQuotes :: (Monad m, Eq a) =>
17141721 -> Fold m a b
17151722 -> Parser a m b
17161723wordWithQuotes keepQuotes tr escChar toRight isSep
1717− (Fold fstep finitial fextract) =
1724+ (Fold fstep finitial _ ffinal) =
17181725 Parser step initial extract
17191726
17201727 where
@@ -1765,7 +1772,7 @@ wordWithQuotes keepQuotes tr escChar toRight isSep
17651772 step (WordUnquotedWord s) a
17661773 | isEsc a = return $ Continue 0 $ WordUnquotedEsc s
17671774 | isSep a = do
1768− b <- fextract s
1775+ b <- ffinal s
17691776 return $ Partial 0 $ WordQuotedSkipPost b
17701777 | otherwise = do
17711778 case toRight a of
@@ -1813,11 +1820,11 @@ wordWithQuotes keepQuotes tr escChar toRight isSep
18131820
18141821 err = return . Error
18151822
1816− extract (WordQuotedSkipPre s) = fmap (Done 0) $ fextract s
1817− extract (WordUnquotedWord s) = fmap (Done 0) $ fextract s
1823+ extract (WordQuotedSkipPre s) = fmap (Done 0) $ ffinal s
1824+ extract (WordUnquotedWord s) = fmap (Done 0) $ ffinal s
18181825 extract (WordQuotedWord s n _ _) =
18191826 if n == 0
1820− then fmap (Done 0) $ fextract s
1827+ then fmap (Done 0) $ ffinal s
18211828 else err "wordWithQuotes: missing frame end"
18221829 extract WordQuotedEsc {} =
18231830 err "wordWithQuotes: trailing escape"
@@ -1895,7 +1902,7 @@ data GroupByState a s
18951902--
18961903{-# INLINE groupBy #-}
18971904groupBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
1898−groupBy eq (Fold fstep finitial fextract) = Parser step initial extract
1905+groupBy eq (Fold fstep finitial _ ffinal) = Parser step initial extract
18991906
19001907 where
19011908
@@ -1918,10 +1925,10 @@ groupBy eq (Fold fstep finitial fextract) = Parser step initial extract
19181925 step (GroupByGrouping a0 s) a =
19191926 if eq a0 a
19201927 then grouper s a0 a
1921− else Done 1 <$> fextract s
1928+ else Done 1 <$> ffinal s
19221929
1923− extract (GroupByInit s) = fmap (Done 0) $ fextract s
1924− extract (GroupByGrouping _ s) = fmap (Done 0) $ fextract s
1930+ extract (GroupByInit s) = fmap (Done 0) $ ffinal s
1931+ extract (GroupByGrouping _ s) = fmap (Done 0) $ ffinal s
19251932
19261933-- | Unlike 'groupBy' this combinator performs a rolling comparison of two
19271934-- successive elements in the input stream. Assuming the input stream
@@ -1955,7 +1962,7 @@ groupBy eq (Fold fstep finitial fextract) = Parser step initial extract
19551962--
19561963{-# INLINE groupByRolling #-}
19571964groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
1958−groupByRolling eq (Fold fstep finitial fextract) = Parser step initial extract
1965+groupByRolling eq (Fold fstep finitial _ ffinal) = Parser step initial extract
19591966
19601967 where
19611968
@@ -1978,10 +1985,10 @@ groupByRolling eq (Fold fstep finitial fextract) = Parser step initial extract
19781985 step (GroupByGrouping a0 s) a =
19791986 if eq a0 a
19801987 then grouper s a
1981− else Done 1 <$> fextract s
1988+ else Done 1 <$> ffinal s
19821989
1983− extract (GroupByInit s) = fmap (Done 0) $ fextract s
1984− extract (GroupByGrouping _ s) = fmap (Done 0) $ fextract s
1990+ extract (GroupByInit s) = fmap (Done 0) $ ffinal s
1991+ extract (GroupByGrouping _ s) = fmap (Done 0) $ ffinal s
19851992
19861993{-# ANN type GroupByStatePair Fuse #-}
19871994data GroupByStatePair a s1 s2
@@ -2005,8 +2012,8 @@ groupByRollingEither :: Monad m =>
20052012 (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
20062013groupByRollingEither
20072014 eq
2008− (Fold fstep1 finitial1 fextract1)
2009− (Fold fstep2 finitial2 fextract2) = Parser step initial extract
2015+ (Fold fstep1 finitial1 _ ffinal1)
2016+ (Fold fstep2 finitial2 _ ffinal2) = Parser step initial extract
20102017
20112018 where
20122019
@@ -2065,21 +2072,21 @@ groupByRollingEither
20652072 step (GroupByGroupingPairL a0 s1 s2) a =
20662073 if not (eq a0 a)
20672074 then grouperL2 s1 s2 a
2068− else Done 1 . Left <$> fextract1 s1
2075+ else Done 1 . Left <$> ffinal1 s1
20692076
20702077 step (GroupByGroupingPairR a0 s1 s2) a =
20712078 if eq a0 a
20722079 then grouperR2 s1 s2 a
2073− else Done 1 . Right <$> fextract2 s2
2080+ else Done 1 . Right <$> ffinal2 s2
20742081
2075− extract (GroupByInitPair s1 _) = Done 0 . Left <$> fextract1 s1
2076− extract (GroupByGroupingPairL _ s1 _) = Done 0 . Left <$> fextract1 s1
2077− extract (GroupByGroupingPairR _ _ s2) = Done 0 . Right <$> fextract2 s2
2082+ extract (GroupByInitPair s1 _) = Done 0 . Left <$> ffinal1 s1
2083+ extract (GroupByGroupingPairL _ s1 _) = Done 0 . Left <$> ffinal1 s1
2084+ extract (GroupByGroupingPairR _ _ s2) = Done 0 . Right <$> ffinal2 s2
20782085 extract (GroupByGroupingPair a s1 _) = do
20792086 res <- fstep1 s1 a
20802087 case res of
20812088 FL.Done b -> return $ Done 0 (Left b)
2082− FL.Partial s11 -> Done 0 . Left <$> fextract1 s11
2089+ FL.Partial s11 -> Done 0 . Left <$> ffinal1 s11
20832090
20842091-- XXX use an Unfold instead of a list?
20852092-- XXX custom combinators for matching list, array and stream?
@@ -2232,7 +2239,7 @@ postscan = undefined
22322239{-# INLINE zipWithM #-}
22332240zipWithM :: Monad m =>
22342241 (a -> b -> m c) -> D.Stream m a -> Fold m c x -> Parser b m x
2235−zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
2242+zipWithM zf (D.Stream sstep state) (Fold fstep finitial _ ffinal) =
22362243 Parser step initial extract
22372244
22382245 where
@@ -2245,7 +2252,7 @@ zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
22452252 case r of
22462253 D.Yield x s -> return $ IPartial (Just' x, s, fs)
22472254 D.Stop -> do
2248− x <- fextract fs
2255+ x <- ffinal fs
22492256 return $ IDone x
22502257 -- Need Skip/Continue in initial to loop right here
22512258 D.Skip s -> return $ IPartial (Nothing', s, fs)
@@ -2260,7 +2267,7 @@ zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
22602267 case r of
22612268 D.Yield x1 s -> return $ Continue 0 (Just' x1, s, fs1)
22622269 D.Stop -> do
2263− x <- fextract fs1
2270+ x <- ffinal fs1
22642271 return $ Done 0 x
22652272 D.Skip s -> return $ Continue 1 (Nothing', s, fs1)
22662273 FL.Done x -> return $ Done 0 x
@@ -2275,7 +2282,7 @@ zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
22752282 return $ Continue 0 (Nothing', s, fs1)
22762283 FL.Done x -> return $ Done 0 x
22772284 D.Stop -> do
2278− x <- fextract fs
2285+ x <- ffinal fs
22792286 return $ Done 1 x
22802287 D.Skip s -> return $ Continue 1 (Nothing', s, fs)
22812288
@@ -2559,7 +2566,7 @@ deintercalateAll :: Monad m =>
25592566deintercalateAll
25602567 (Parser stepL initialL extractL)
25612568 (Parser stepR initialR _)
2562− (Fold fstep finitial fextract) = Parser step initial extract
2569+ (Fold fstep finitial _ ffinal) = Parser step initial extract
25632570
25642571 where
25652572
@@ -2630,9 +2637,9 @@ deintercalateAll
26302637 extractResult n fs r = do
26312638 res <- fstep fs r
26322639 case res of
2633− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
2640+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
26342641 FL.Done c -> return (Done n c)
2635− extract (DeintercalateAllInitL fs) = fmap (Done 0) $ fextract fs
2642+ extract (DeintercalateAllInitL fs) = fmap (Done 0) $ ffinal fs
26362643 extract (DeintercalateAllL fs sL) = do
26372644 r <- extractL sL
26382645 case r of
@@ -2640,7 +2647,7 @@ deintercalateAll
26402647 Error err -> return $ Error err
26412648 Continue n s -> return $ Continue n (DeintercalateAllL fs s)
26422649 Partial _ _ -> error "Partial in extract"
2643− extract (DeintercalateAllInitR fs) = fmap (Done 0) $ fextract fs
2650+ extract (DeintercalateAllInitR fs) = fmap (Done 0) $ ffinal fs
26442651 extract (DeintercalateAllR _ _) =
26452652 return $ Error "deintercalateAll: input ended at 'Right' value"
26462653
@@ -2683,7 +2690,7 @@ deintercalate :: Monad m =>
26832690deintercalate
26842691 (Parser stepL initialL extractL)
26852692 (Parser stepR initialR _)
2686− (Fold fstep finitial fextract) = Parser step initial extract
2693+ (Fold fstep finitial _ ffinal) = Parser step initial extract
26872694
26882695 where
26892696
@@ -2714,7 +2721,7 @@ deintercalate
27142721 Done n b ->
27152722 processL (fstep fs (Left b)) n DeintercalateInitR
27162723 Error _ -> do
2717− xs <- fextract fs
2724+ xs <- ffinal fs
27182725 return $ Done cnt1 xs
27192726
27202727 {-# INLINE processR #-}
@@ -2734,7 +2741,7 @@ deintercalate
27342741 Continue n s -> return $ Continue n (DeintercalateR (cnt1 - n) fs s)
27352742 Done n b -> processR (cnt1 - n) b fs n
27362743 Error _ -> do
2737− xs <- fextract fs
2744+ xs <- ffinal fs
27382745 return $ Done cnt1 xs
27392746
27402747 step (DeintercalateInitL fs) a = do
@@ -2769,17 +2776,17 @@ deintercalate
27692776 -- XXX We could have the fold accept pairs of (bR, bL)
27702777 FL.Done _ -> error "Fold terminated consuming partial input"
27712778 Error _ -> do
2772− xs <- fextract fs
2779+ xs <- ffinal fs
27732780 return $ Done cnt1 xs
27742781
27752782 {-# INLINE extractResult #-}
27762783 extractResult n fs r = do
27772784 res <- fstep fs r
27782785 case res of
2779− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
2786+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
27802787 FL.Done c -> return (Done n c)
27812788
2782− extract (DeintercalateInitL fs) = fmap (Done 0) $ fextract fs
2789+ extract (DeintercalateInitL fs) = fmap (Done 0) $ ffinal fs
27832790 extract (DeintercalateL cnt fs sL) = do
27842791 r <- extractL sL
27852792 case r of
@@ -2787,10 +2794,10 @@ deintercalate
27872794 Continue n s -> return $ Continue n (DeintercalateL (cnt - n) fs s)
27882795 Partial _ _ -> error "Partial in extract"
27892796 Error _ -> do
2790− xs <- fextract fs
2797+ xs <- ffinal fs
27912798 return $ Done cnt xs
2792− extract (DeintercalateInitR fs) = fmap (Done 0) $ fextract fs
2793− extract (DeintercalateR cnt fs _) = fmap (Done cnt) $ fextract fs
2799+ extract (DeintercalateInitR fs) = fmap (Done 0) $ ffinal fs
2800+ extract (DeintercalateR cnt fs _) = fmap (Done cnt) $ ffinal fs
27942801 extract (DeintercalateRL cnt bR fs sL) = do
27952802 r <- extractL sL
27962803 case r of
@@ -2802,7 +2809,7 @@ deintercalate
28022809 Continue n s -> return $ Continue n (DeintercalateRL (cnt - n) bR fs s)
28032810 Partial _ _ -> error "Partial in extract"
28042811 Error _ -> do
2805− xs <- fextract fs
2812+ xs <- ffinal fs
28062813 return $ Done cnt xs
28072814
28082815{-# ANN type Deintercalate1State Fuse #-}
@@ -2839,7 +2846,7 @@ deintercalate1 :: Monad m =>
28392846deintercalate1
28402847 (Parser stepL initialL extractL)
28412848 (Parser stepR initialR _)
2842− (Fold fstep finitial fextract) = Parser step initial extract
2849+ (Fold fstep finitial _ ffinal) = Parser step initial extract
28432850
28442851 where
28452852
@@ -2893,7 +2900,7 @@ deintercalate1
28932900 Continue n s -> return $ Continue n (Deintercalate1R (cnt1 - n) fs s)
28942901 Done n b -> processR (cnt1 - n) b fs n
28952902 Error _ -> do
2896− xs <- fextract fs
2903+ xs <- ffinal fs
28972904 return $ Done cnt1 xs
28982905
28992906 step (Deintercalate1InitL cnt fs sL) a = runStepInitL cnt fs sL a
@@ -2922,14 +2929,14 @@ deintercalate1
29222929 -- XXX We could have the fold accept pairs of (bR, bL)
29232930 FL.Done _ -> error "Fold terminated consuming partial input"
29242931 Error _ -> do
2925− xs <- fextract fs
2932+ xs <- ffinal fs
29262933 return $ Done cnt1 xs
29272934
29282935 {-# INLINE extractResult #-}
29292936 extractResult n fs r = do
29302937 res <- fstep fs r
29312938 case res of
2932− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
2939+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
29332940 FL.Done c -> return (Done n c)
29342941
29352942 extract (Deintercalate1InitL cnt fs sL) = do
@@ -2939,8 +2946,8 @@ deintercalate1
29392946 Continue n s -> return $ Continue n (Deintercalate1InitL (cnt - n) fs s)
29402947 Partial _ _ -> error "Partial in extract"
29412948 Error err -> return $ Error err
2942− extract (Deintercalate1InitR fs) = fmap (Done 0) $ fextract fs
2943− extract (Deintercalate1R cnt fs _) = fmap (Done cnt) $ fextract fs
2949+ extract (Deintercalate1InitR fs) = fmap (Done 0) $ ffinal fs
2950+ extract (Deintercalate1R cnt fs _) = fmap (Done cnt) $ ffinal fs
29442951 extract (Deintercalate1RL cnt bR fs sL) = do
29452952 r <- extractL sL
29462953 case r of
@@ -2952,7 +2959,7 @@ deintercalate1
29522959 Continue n s -> return $ Continue n (Deintercalate1RL (cnt - n) bR fs s)
29532960 Partial _ _ -> error "Partial in extract"
29542961 Error _ -> do
2955− xs <- fextract fs
2962+ xs <- ffinal fs
29562963 return $ Done cnt xs
29572964
29582965{-# ANN type SepByState Fuse #-}
@@ -2995,7 +3002,7 @@ sepBy :: Monad m =>
29953002sepBy
29963003 (Parser stepL initialL extractL)
29973004 (Parser stepR initialR _)
2998− (Fold fstep finitial fextract) = Parser step initial extract
3005+ (Fold fstep finitial _ ffinal) = Parser step initial extract
29993006
30003007 where
30013008
@@ -3026,7 +3033,7 @@ sepBy
30263033 Done n b ->
30273034 processL (fstep fs b) n SepByInitR
30283035 Error _ -> do
3029− xs <- fextract fs
3036+ xs <- ffinal fs
30303037 return $ Done cnt1 xs
30313038
30323039 {-# INLINE processR #-}
@@ -3046,7 +3053,7 @@ sepBy
30463053 Continue n s -> return $ Continue n (SepByR (cnt1 - n) fs s)
30473054 Done n _ -> processR (cnt1 - n) fs n
30483055 Error _ -> do
3049− xs <- fextract fs
3056+ xs <- ffinal fs
30503057 return $ Done cnt1 xs
30513058
30523059 step (SepByInitL fs) a = do
@@ -3068,10 +3075,10 @@ sepBy
30683075 extractResult n fs r = do
30693076 res <- fstep fs r
30703077 case res of
3071− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
3078+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
30723079 FL.Done c -> return (Done n c)
30733080
3074− extract (SepByInitL fs) = fmap (Done 0) $ fextract fs
3081+ extract (SepByInitL fs) = fmap (Done 0) $ ffinal fs
30753082 extract (SepByL cnt fs sL) = do
30763083 r <- extractL sL
30773084 case r of
@@ -3079,10 +3086,10 @@ sepBy
30793086 Continue n s -> return $ Continue n (SepByL (cnt - n) fs s)
30803087 Partial _ _ -> error "Partial in extract"
30813088 Error _ -> do
3082− xs <- fextract fs
3089+ xs <- ffinal fs
30833090 return $ Done cnt xs
3084− extract (SepByInitR fs) = fmap (Done 0) $ fextract fs
3085− extract (SepByR cnt fs _) = fmap (Done cnt) $ fextract fs
3091+ extract (SepByInitR fs) = fmap (Done 0) $ ffinal fs
3092+ extract (SepByR cnt fs _) = fmap (Done cnt) $ ffinal fs
30863093
30873094-- | Non-backtracking version of sepBy. Several times faster.
30883095{-# INLINE sepByAll #-}
@@ -3137,7 +3144,7 @@ sepBy1 :: Monad m =>
31373144sepBy1
31383145 (Parser stepL initialL extractL)
31393146 (Parser stepR initialR _)
3140− (Fold fstep finitial fextract) = Parser step initial extract
3147+ (Fold fstep finitial _ ffinal) = Parser step initial extract
31413148
31423149 where
31433150
@@ -3184,7 +3191,7 @@ sepBy1
31843191 Done n b ->
31853192 processL (fstep fs b) n SepBy1InitR
31863193 Error _ -> do
3187− xs <- fextract fs
3194+ xs <- ffinal fs
31883195 return $ Done cnt1 xs
31893196
31903197 {-# INLINE processR #-}
@@ -3204,7 +3211,7 @@ sepBy1
32043211 Continue n s -> return $ Continue n (SepBy1R (cnt1 - n) fs s)
32053212 Done n _ -> processR (cnt1 - n) fs n
32063213 Error _ -> do
3207− xs <- fextract fs
3214+ xs <- ffinal fs
32083215 return $ Done cnt1 xs
32093216
32103217 step (SepBy1InitL cnt fs sL) a = runStepInitL cnt fs sL a
@@ -3221,7 +3228,7 @@ sepBy1
32213228 extractResult n fs r = do
32223229 res <- fstep fs r
32233230 case res of
3224− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
3231+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
32253232 FL.Done c -> return (Done n c)
32263233
32273234 extract (SepBy1InitL cnt fs sL) = do
@@ -3238,10 +3245,10 @@ sepBy1
32383245 Continue n s -> return $ Continue n (SepBy1L (cnt - n) fs s)
32393246 Partial _ _ -> error "Partial in extract"
32403247 Error _ -> do
3241− xs <- fextract fs
3248+ xs <- ffinal fs
32423249 return $ Done cnt xs
3243− extract (SepBy1InitR fs) = fmap (Done 0) $ fextract fs
3244− extract (SepBy1R cnt fs _) = fmap (Done cnt) $ fextract fs
3250+ extract (SepBy1InitR fs) = fmap (Done 0) $ ffinal fs
3251+ extract (SepBy1R cnt fs _) = fmap (Done cnt) $ ffinal fs
32453252
32463253-------------------------------------------------------------------------------
32473254-- Interleaving a collection of parsers
@@ -3271,7 +3278,7 @@ roundRobin _ps _f = undefined
32713278{-# INLINE sequence #-}
32723279sequence :: Monad m =>
32733280 D.Stream m (Parser a m b) -> Fold m b c -> Parser a m c
3274−sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
3281+sequence (D.Stream sstep sstate) (Fold fstep finitial _ ffinal) =
32753282 Parser step initial extract
32763283
32773284 where
@@ -3289,7 +3296,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
32893296 case sres of
32903297 D.Yield p ss1 -> return $ Continue 1 (Just' p, ss1, fs)
32913298 D.Stop -> do
3292− c <- fextract fs
3299+ c <- ffinal fs
32933300 return $ Done 1 c
32943301 D.Skip ss1 -> return $ Continue 1 (Nothing', ss1, fs)
32953302
@@ -3325,7 +3332,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
33253332 FL.Done c -> return $ Done 1 c
33263333 IError err -> return $ Error err
33273334
3328− extract (Nothing', _, fs) = fmap (Done 0) $ fextract fs
3335+ extract (Nothing', _, fs) = fmap (Done 0) $ ffinal fs
33293336 extract (Just' (Parser pstep pinit pextr), ss, fs) = do
33303337 ps <- pinit
33313338 case ps of
@@ -3335,7 +3342,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
33353342 Done n b -> do
33363343 res <- fstep fs b
33373344 case res of
3338− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
3345+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
33393346 FL.Done c -> return (Done n c)
33403347 Error err -> return $ Error err
33413348 Continue n s -> return $ Continue n (Just' (Parser pstep (return (IPartial s)) pextr), ss, fs)
@@ -3343,7 +3350,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
33433350 IDone b -> do
33443351 fres <- fstep fs b
33453352 case fres of
3346− FL.Partial fs1 -> fmap (Done 0) $ fextract fs1
3353+ FL.Partial fs1 -> fmap (Done 0) $ ffinal fs1
33473354 FL.Done c -> return (Done 0 c)
33483355 IError err -> return $ Error err
33493356
@@ -3478,7 +3485,7 @@ manyTill :: Monad m
34783485 => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
34793486manyTill (Parser stepL initialL extractL)
34803487 (Parser stepR initialR _)
3481− (Fold fstep finitial fextract) =
3488+ (Fold fstep finitial _ ffinal) =
34823489 Parser step initial extract
34833490
34843491 where
@@ -3500,7 +3507,7 @@ manyTill (Parser stepL initialL extractL)
35003507 resR <- initialR
35013508 case resR of
35023509 IPartial sr -> return $ p (ManyTillR 0 fs sr)
3503− IDone _ -> d <$> fextract fs
3510+ IDone _ -> d <$> ffinal fs
35043511 IError _ -> scrutL fs p c d e
35053512
35063513 initial = do
@@ -3517,7 +3524,7 @@ manyTill (Parser stepL initialL extractL)
35173524 assertM(cnt + 1 - n >= 0)
35183525 return $ Continue n (ManyTillR (cnt + 1 - n) fs s)
35193526 Done n _ -> do
3520− b <- fextract fs
3527+ b <- ffinal fs
35213528 return $ Done n b
35223529 Error _ -> do
35233530 resL <- initialL
@@ -3556,12 +3563,12 @@ manyTill (Parser stepL initialL extractL)
35563563 Done n b -> do
35573564 r <- fstep fs b
35583565 case r of
3559− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
3566+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
35603567 FL.Done c -> return (Done n c)
35613568 Error err -> return $ Error err
35623569 Continue n s -> return $ Continue n (ManyTillL fs s)
35633570 Partial _ _ -> error "Partial in extract"
3564− extract (ManyTillR _ fs _) = fmap (Done 0) $ fextract fs
3571+ extract (ManyTillR _ fs _) = fmap (Done 0) $ ffinal fs
35653572
35663573-- | @manyThen f collect recover@ repeats the parser @collect@ on the input and
35673574-- collects the output in the supplied fold. If the the parser @collect@ fails,
core/src/Streamly/Internal/Data/Parser/Type.hs+93−70
@@ -175,9 +175,6 @@
175175
176176module Streamly.Internal.Data.Parser.Type
177177 (
178− -- * Setup
179− -- $setup
180−
181178 -- * Types
182179 Initial (..)
183180 , Step (..)
@@ -216,7 +213,10 @@ where
216213#include "inline.hs"
217214#include "assert.hs"
218215
219−import Control.Applicative (Alternative(..), liftA2)
216+#if !MIN_VERSION_base(4,18,0)
217+import Control.Applicative (liftA2)
218+#endif
219+import Control.Applicative (Alternative(..))
220220import Control.Exception (Exception(..))
221221-- import Control.Monad (MonadPlus(..), (>=>))
222222import Control.Monad ((>=>))
@@ -438,6 +438,7 @@ newtype ParseError = ParseError String
438438instance Exception ParseError where
439439 displayException (ParseError err) = err
440440
441+-- | Map a function on the result i.e. on @b@ in @Parser a m b@.
441442instance Functor m => Functor (Parser a m) where
442443 {-# INLINE fmap #-}
443444 fmap f (Parser step1 initial1 extract) =
@@ -504,26 +505,30 @@ data SeqParseState sl f sr = SeqParseL !sl | SeqParseR !f !sr
504505-- 'Streamly.Data.Stream.append', it splits the streams using two parsers and
505506-- zips the results.
506507
507−-- | Sequential parser application.
508+-- | Sequential parser application. Apply two parsers sequentially to an input
509+-- stream. The first parser runs and processes the input, the remaining input
510+-- is then passed to the second parser. If both parsers succeed, their outputs
511+-- are combined using the supplied function. If either parser fails, the
512+-- operation fails.
508513--
509−-- Apply two parsers sequentially to an input stream. The first parser runs and
510−-- processes the input, the remaining input is then passed to the second
511−-- parser. If both parsers succeed, their outputs are combined using the
512−-- supplied function. If either parser fails, the operation fails.
514+-- This combinator delivers high performance by stream fusion but it comes with
515+-- some limitations. For those cases use the 'Applicative' instance of
516+-- 'Streamly.Data.ParserK.ParserK'.
513517--
514−-- This implementation is strict in the second argument, therefore, the
515−-- following will fail:
518+-- CAVEAT 1: NO RECURSION. This function is strict in both arguments. As a
519+-- result, if a parser is defined recursively using this, it may cause an
520+-- infintie loop. The following example checks the strictness:
516521--
517−-- >>> Stream.parse (Parser.splitWith const (Parser.satisfy (> 0)) undefined) $ Stream.fromList [1]
522+-- >>> p = Parser.splitWith const (Parser.satisfy (> 0)) undefined
523+-- >>> Stream.parse p $ Stream.fromList [1]
518524-- *** Exception: Prelude.undefined
519525-- ...
520526--
521−-- Although this implementation allows stream fusion, it has quadratic
522−-- complexity, making it suitable only for a small number of compositions.
523−-- As a thumb rule use it for less than 8 compositions, use ParserK otherwise.
527+-- CAVEAT 2: QUADRATIC TIME COMPLEXITY. Static composition is fast due to
528+-- stream fusion, but it works well only for limited (e.g. up to 8)
529+-- compositions, use "Streamly.Data.ParserK" for larger compositions.
524530--
525−-- Below are some common idioms that can be expressed using 'splitWith' and
526−-- other parser primitives:
531+-- Below are some common idioms that can be expressed using 'splitWith':
527532--
528533-- >>> span p f1 f2 = Parser.splitWith (,) (Parser.takeWhile p f1) (Parser.fromFold f2)
529534-- >>> spanBy eq f1 f2 = Parser.splitWith (,) (Parser.groupBy eq f1) (Parser.fromFold f2)
@@ -612,7 +617,12 @@ splitWith func (Parser stepL initialL extractL)
612617-- We can use the compiler to automatically annotate accumulators, terminating
613618-- folds, non-failing parsers and failing parsers.
614619
615−-- | Works correctly only if both the parsers are guaranteed to never fail.
620+-- | Better performance 'splitWith' for non-failing parsers.
621+--
622+-- Does not work correctly for parsers that can fail.
623+--
624+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
625+--
616626{-# INLINE noErrorUnsafeSplitWith #-}
617627noErrorUnsafeSplitWith :: Monad m
618628 => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
@@ -683,6 +693,8 @@ data SeqAState sl sr = SeqAL !sl | SeqAR !sr
683693-- second parser. The output of the parser is the output of the second parser.
684694-- The operation fails if any of the parsers fail.
685695--
696+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
697+--
686698-- This implementation is strict in the second argument, therefore, the
687699-- following will fail:
688700--
@@ -690,10 +702,6 @@ data SeqAState sl sr = SeqAL !sl | SeqAR !sr
690702-- *** Exception: Prelude.undefined
691703-- ...
692704--
693−-- Although this implementation allows stream fusion, it has quadratic
694−-- complexity, making it suitable only for a small number of compositions.
695−-- As a thumb rule use it for less than 8 compositions, use ParserK otherwise.
696−--
697705-- /Pre-release/
698706--
699707{-# INLINE split_ #-}
@@ -749,7 +757,12 @@ split_ (Parser stepL initialL extractL) (Parser stepR initialR extractR) =
749757 Partial _ _ -> error "split_: Partial"
750758 Continue n s -> return $ Continue n (SeqAL s)
751759
752−-- For backtracking folds
760+-- | Better performance 'split_' for non-failing parsers.
761+--
762+-- Does not work correctly for parsers that can fail.
763+--
764+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
765+--
753766{-# INLINE noErrorUnsafeSplit_ #-}
754767noErrorUnsafeSplit_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
755768noErrorUnsafeSplit_
@@ -804,7 +817,11 @@ noErrorUnsafeSplit_
804817 Partial _ _ -> error "split_: Partial"
805818 Continue n s -> return $ Continue n (SeqAL s)
806819
807−-- | 'Applicative' form of 'splitWith'.
820+-- | READ THE CAVEATS in 'splitWith' before using this instance.
821+--
822+-- >>> pure = Parser.fromPure
823+-- >>> (<*>) = Parser.splitWith id
824+-- >>> (*>) = Parser.split_
808825instance Monad m => Applicative (Parser a m) where
809826 {-# INLINE pure #-}
810827 pure = fromPure
@@ -830,20 +847,26 @@ data AltParseState sl sr = AltParseL !Int !sl | AltParseR !sr
830847-- each subsequent alternative's input element has to go through, therefore, it
831848-- cannot scale to a large number of compositions
832849
833−-- | Sequential alternative. The input is first passed to the first parser, and
850+-- | Sequential alternative. The input is first passed to the first parser,
834851-- if it succeeds, the result is returned. However, if the first parser fails,
835−-- the parser driver backtracks and tries the same input on the second parser,
836−-- returning the result if it succeeds.
852+-- the parser driver backtracks and tries the same input on the second
853+-- (alternative) parser, returning the result if it succeeds.
854+--
855+-- This combinator delivers high performance by stream fusion but it comes with
856+-- some limitations. For those cases use the 'Alternative' instance of
857+-- 'Streamly.Data.ParserK.ParserK'.
837858--
838−-- Note: This implementation is not lazy in the second argument. The following
839−-- will fail:
859+-- CAVEAT 1: NO RECURSION. This function is strict in both arguments. As a
860+-- result, if a parser is defined recursively using this, it may cause an
861+-- infintie loop. The following example checks the strictness:
840862--
841−-- >> Stream.parse (Parser.satisfy (> 0) `Parser.alt` undefined) $ Stream.fromList [1..10]
863+-- >>> p = Parser.satisfy (> 0) `Parser.alt` undefined
864+-- >>> Stream.parse p $ Stream.fromList [1..10]
842865-- *** Exception: Prelude.undefined
843866--
844−-- Although this implementation allows stream fusion, it has quadratic
845−-- complexity, making it suitable only for a small number of compositions.
846−-- As a thumb rule use it for less than 8 compositions, use ParserK otherwise.
867+-- CAVEAT 2: QUADRATIC TIME COMPLEXITY. Static composition is fast due to
868+-- stream fusion, but it works well only for limited (e.g. up to 8)
869+-- compositions, use "Streamly.Data.ParserK" for larger compositions.
847870--
848871-- /Time Complexity:/ O(n^2) where n is the number of compositions.
849872--
@@ -924,7 +947,7 @@ data Fused3 a b c = Fused3 !a !b !c
924947--
925948{-# INLINE splitMany #-}
926949splitMany :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
927−splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
950+splitMany (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
928951 Parser step initial extract
929952
930953 where
@@ -940,7 +963,7 @@ splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
940963 IPartial ps -> return $ partial $ Fused3 ps 0 fs
941964 IDone pb ->
942965 runCollectorWith (handleCollect partial done) fs pb
943− IError _ -> done <$> fextract fs
966+ IError _ -> done <$> ffinal fs
944967 FL.Done fb -> return $ done fb
945968
946969 runCollectorWith cont fs pb = fstep fs pb >>= cont
@@ -964,19 +987,19 @@ splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
964987 assertM(cnt1 - n >= 0)
965988 fstep fs b >>= handleCollect (Partial n) (Done n)
966989 Error _ -> do
967− xs <- fextract fs
990+ xs <- ffinal fs
968991 return $ Done cnt xs
969992
970− extract (Fused3 _ 0 fs) = fmap (Done 0) (fextract fs)
993+ extract (Fused3 _ 0 fs) = fmap (Done 0) (ffinal fs)
971994 extract (Fused3 s cnt fs) = do
972995 r <- extract1 s
973996 case r of
974− Error _ -> fmap (Done cnt) (fextract fs)
997+ Error _ -> fmap (Done cnt) (ffinal fs)
975998 Done n b -> do
976999 assertM(n <= cnt)
9771000 fs1 <- fstep fs b
9781001 case fs1 of
979− FL.Partial s1 -> fmap (Done n) (fextract s1)
1002+ FL.Partial s1 -> fmap (Done n) (ffinal s1)
9801003 FL.Done b1 -> return (Done n b1)
9811004 Partial _ _ -> error "splitMany: Partial in extract"
9821005 Continue n s1 -> do
@@ -990,7 +1013,7 @@ splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
9901013--
9911014{-# INLINE splitManyPost #-}
9921015splitManyPost :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
993−splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
1016+splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
9941017 Parser step initial extract
9951018
9961019 where
@@ -1006,7 +1029,7 @@ splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10061029 IPartial ps -> return $ partial $ Fused3 ps 0 fs
10071030 IDone pb ->
10081031 runCollectorWith (handleCollect partial done) fs pb
1009− IError _ -> done <$> fextract fs
1032+ IError _ -> done <$> ffinal fs
10101033 FL.Done fb -> return $ done fb
10111034
10121035 runCollectorWith cont fs pb = fstep fs pb >>= cont
@@ -1028,18 +1051,18 @@ splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10281051 assertM(cnt1 - n >= 0)
10291052 fstep fs b >>= handleCollect (Partial n) (Done n)
10301053 Error _ -> do
1031− xs <- fextract fs
1054+ xs <- ffinal fs
10321055 return $ Done cnt1 xs
10331056
10341057 extract (Fused3 s cnt fs) = do
10351058 r <- extract1 s
10361059 case r of
1037− Error _ -> fmap (Done cnt) (fextract fs)
1060+ Error _ -> fmap (Done cnt) (ffinal fs)
10381061 Done n b -> do
10391062 assertM(n <= cnt)
10401063 fs1 <- fstep fs b
10411064 case fs1 of
1042− FL.Partial s1 -> fmap (Done n) (fextract s1)
1065+ FL.Partial s1 -> fmap (Done n) (ffinal s1)
10431066 FL.Done b1 -> return (Done n b1)
10441067 Partial _ _ -> error "splitMany: Partial in extract"
10451068 Continue n s1 -> do
@@ -1052,7 +1075,7 @@ splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10521075--
10531076{-# INLINE splitSome #-}
10541077splitSome :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
1055−splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
1078+splitSome (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
10561079 Parser step initial extract
10571080
10581081 where
@@ -1068,7 +1091,7 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10681091 IPartial ps -> return $ partial $ Fused3 ps 0 $ Right fs
10691092 IDone pb ->
10701093 runCollectorWith (handleCollect partial done) fs pb
1071− IError _ -> done <$> fextract fs
1094+ IError _ -> done <$> ffinal fs
10721095 FL.Done fb -> return $ done fb
10731096
10741097 runCollectorWith cont fs pb = fstep fs pb >>= cont
@@ -1118,7 +1141,7 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
11181141 Done n b -> do
11191142 assertM(cnt1 - n >= 0)
11201143 fstep fs b >>= handleCollect (Partial n) (Done n)
1121− Error _ -> Done cnt1 <$> fextract fs
1144+ Error _ -> Done cnt1 <$> ffinal fs
11221145
11231146 extract (Fused3 s cnt (Left fs)) = do
11241147 r <- extract1 s
@@ -1128,7 +1151,7 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
11281151 assertM(n <= cnt)
11291152 fs1 <- fstep fs b
11301153 case fs1 of
1131− FL.Partial s1 -> fmap (Done n) (fextract s1)
1154+ FL.Partial s1 -> fmap (Done n) (ffinal s1)
11321155 FL.Done b1 -> return (Done n b1)
11331156 Partial _ _ -> error "splitSome: Partial in extract"
11341157 Continue n s1 -> do
@@ -1137,12 +1160,12 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
11371160 extract (Fused3 s cnt (Right fs)) = do
11381161 r <- extract1 s
11391162 case r of
1140− Error _ -> fmap (Done cnt) (fextract fs)
1163+ Error _ -> fmap (Done cnt) (ffinal fs)
11411164 Done n b -> do
11421165 assertM(n <= cnt)
11431166 fs1 <- fstep fs b
11441167 case fs1 of
1145− FL.Partial s1 -> fmap (Done n) (fextract s1)
1168+ FL.Partial s1 -> fmap (Done n) (ffinal s1)
11461169 FL.Done b1 -> return (Done n b1)
11471170 Partial _ _ -> error "splitSome: Partial in extract"
11481171 Continue n s1 -> do
@@ -1173,23 +1196,12 @@ dieM err = Parser undefined (IError <$> err) undefined
11731196-- Note: With the direct style parser type, the list in "some" and "many" is
11741197-- accumulated strictly, it cannot be consumed lazily.
11751198
1176−-- | Sequential alternative. The input is first passed to the first parser, and
1177−-- if it succeeds, the result is returned. However, if the first parser fails,
1178−-- the parser driver backtracks and tries the same input on the second parser,
1179−-- returning the result if it succeeds.
1180−--
1181−-- Note: The implementation of '<|>' is not lazy in the second
1182−-- argument. The following code will fail:
1183−--
1184−-- >>> Stream.parse (Parser.satisfy (> 0) <|> undefined) $ Stream.fromList [1..10]
1185−-- *** Exception: Prelude.undefined
1186−-- ...
1199+-- | READ THE CAVEATS in 'alt' before using this instance.
11871200--
1188−-- WARNING! this is not suitable for large scale use. As a thumb rule stream
1189−-- fusion works well for less than 8 compositions of this operation, otherwise
1190−-- consider using 'ParserK'. Do not use recursive parser implementations based
1191−-- on this Alternative instance.
1192−
1201+-- >>> empty = Parser.die "empty"
1202+-- >>> (<|>) = Parser.alt
1203+-- >>> many = flip Parser.many Fold.toList
1204+-- >>> some = flip Parser.some Fold.toList
11931205instance Monad m => Alternative (Parser a m) where
11941206 {-# INLINE empty #-}
11951207 empty = die "empty"
@@ -1208,8 +1220,13 @@ data ConcatParseState sl m a b =
12081220 ConcatParseL !sl
12091221 | forall s. ConcatParseR (s -> a -> m (Step s b)) s (s -> m (Step s b))
12101222
1223+-- XXX Does it fuse completely? Check and update, it cannot fuse the
1224+-- dynamically generated parser.
1225+
12111226-- | Map a 'Parser' returning function on the result of a 'Parser'.
12121227--
1228+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
1229+--
12131230-- /Pre-release/
12141231--
12151232{-# INLINE concatMap #-}
@@ -1279,6 +1296,12 @@ concatMap func (Parser stepL initialL extractL) = Parser step initial extract
12791296 Partial _ _ -> error "concatMap: extract Partial"
12801297 Continue n s -> return $ Continue n (ConcatParseL s)
12811298
1299+-- | Better performance 'concatMap' for non-failing parsers.
1300+--
1301+-- Does not work correctly for parsers that can fail.
1302+--
1303+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
1304+--
12821305{-# INLINE noErrorUnsafeConcatMap #-}
12831306noErrorUnsafeConcatMap :: Monad m =>
12841307 (b -> Parser a m c) -> Parser a m b -> Parser a m c
@@ -1351,11 +1374,9 @@ noErrorUnsafeConcatMap func (Parser stepL initialL extractL) =
13511374-- for small number of compositions but for a scalable implementation we need a
13521375-- CPS version.
13531376
1354−-- | See documentation of 'Streamly.Internal.Data.Parser.ParserK.Type.Parser'.
1377+-- | READ THE CAVEATS in 'concatMap' before using this instance.
13551378--
1356−-- Although this implementation allows stream fusion, it has quadratic
1357−-- complexity, making it suitable only for a small number of compositions. As a
1358−-- thumb rule use it for less than 8 compositions, use 'ParserK' otherwise.
1379+-- >>> (>>=) = flip Parser.concatMap
13591380--
13601381instance Monad m => Monad (Parser a m) where
13611382 {-# INLINE return #-}
@@ -1367,6 +1388,7 @@ instance Monad m => Monad (Parser a m) where
13671388 {-# INLINE (>>) #-}
13681389 (>>) = (*>)
13691390
1391+-- | >>> fail = Parser.die
13701392instance Monad m => Fail.MonadFail (Parser a m) where
13711393 {-# INLINE fail #-}
13721394 fail = die
@@ -1382,6 +1404,7 @@ instance Monad m => MonadPlus (Parser a m) where
13821404 mplus = alt
13831405-}
13841406
1407+-- | >>> liftIO = Parser.fromEffect . liftIO
13851408instance (MonadIO m) => MonadIO (Parser a m) where
13861409 {-# INLINE liftIO #-}
13871410 liftIO = fromEffect . liftIO
core/src/Streamly/Internal/Data/ParserK/Type.hs+28−65
@@ -41,7 +41,10 @@ where
4141#include "assert.hs"
4242#include "inline.hs"
4343
44−import Control.Applicative (Alternative(..), liftA2)
44+#if !MIN_VERSION_base(4,18,0)
45+import Control.Applicative (liftA2)
46+#endif
47+import Control.Applicative (Alternative(..))
4548import Control.Monad (MonadPlus(..), ap)
4649import Control.Monad.IO.Class (MonadIO, liftIO)
4750-- import Control.Monad.Trans.Class (MonadTrans(lift))
@@ -165,8 +168,8 @@ newtype ParserK a m b = MkParser
165168-------------------------------------------------------------------------------
166169
167170-- XXX rewrite this using ParserD, expose rmapM from ParserD.
168−-- | Maps a function over the output of the parser.
169−--
171+
172+-- | Map a function on the result i.e. on @b@ in @Parser a m b@.
170173instance Functor m => Functor (ParserK a m) where
171174 {-# INLINE fmap #-}
172175 fmap f parser = MkParser $ \k n st arr ->
@@ -178,7 +181,7 @@ instance Functor m => Functor (ParserK a m) where
178181-------------------------------------------------------------------------------
179182
180183-- This is the dual of stream "fromPure".
181−--
184+
182185-- | A parser that always yields a pure value without consuming any input.
183186--
184187-- /Pre-release/
@@ -196,9 +199,11 @@ fromEffect :: Monad m => m b -> ParserK a m b
196199fromEffect eff =
197200 MkParser $ \k n st arr -> eff >>= \b -> k (Success n b) st arr
198201
199−-- | 'Applicative' form of 'Streamly.Internal.Data.Parser.splitWith'. Note that
200−-- this operation does not fuse, use 'Streamly.Internal.Data.Parser.splitWith'
201−-- when fusion is important.
202+-- | @f \<$> p1 \<*> p2@ applies parsers p1 and p2 sequentially to an input
203+-- stream. The first parser runs and processes the input, the remaining input
204+-- is then passed to the second parser. If both parsers succeed, their outputs
205+-- are applied to the function @f@. If either parser fails, the operation
206+-- fails.
202207--
203208instance Monad m => Applicative (ParserK a m) where
204209 {-# INLINE pure #-}
@@ -240,48 +245,8 @@ instance Monad m => Applicative (ParserK a m) where
240245die :: String -> ParserK a m b
241246die err = MkParser (\k n st arr -> k (Failure n err) st arr)
242247
243−-- | Monad composition can be used for lookbehind parsers, we can make the
244−-- future parses depend on the previously parsed values.
245−--
246−-- If we have to parse "a9" or "9a" but not "99" or "aa" we can use the
247−-- following parser:
248−--
249−-- @
250−-- backtracking :: MonadCatch m => PR.Parser Char m String
251−-- backtracking =
252−-- sequence [PR.satisfy isDigit, PR.satisfy isAlpha]
253−-- '<|>'
254−-- sequence [PR.satisfy isAlpha, PR.satisfy isDigit]
255−-- @
256−--
257−-- We know that if the first parse resulted in a digit at the first place then
258−-- the second parse is going to fail. However, we waste that information and
259−-- parse the first character again in the second parse only to know that it is
260−-- not an alphabetic char. By using lookbehind in a 'Monad' composition we can
261−-- avoid redundant work:
262−--
263−-- @
264−-- data DigitOrAlpha = Digit Char | Alpha Char
265−--
266−-- lookbehind :: MonadCatch m => PR.Parser Char m String
267−-- lookbehind = do
268−-- x1 \<- Digit '<$>' PR.satisfy isDigit
269−-- '<|>' Alpha '<$>' PR.satisfy isAlpha
270−--
271−-- -- Note: the parse depends on what we parsed already
272−-- x2 <- case x1 of
273−-- Digit _ -> PR.satisfy isAlpha
274−-- Alpha _ -> PR.satisfy isDigit
275−--
276−-- return $ case x1 of
277−-- Digit x -> [x,x2]
278−-- Alpha x -> [x,x2]
279−-- @
280−--
281−-- See also 'Streamly.Internal.Data.Parser.concatMap'. This monad instance
282−-- does not fuse, use 'Streamly.Internal.Data.Parser.concatMap' when you need
283−-- fusion.
284−--
248+-- | Monad composition can be used for lookbehind parsers, we can dynamically
249+-- compose new parsers based on the results of the previously parsed values.
285250instance Monad m => Monad (ParserK a m) where
286251 {-# INLINE return #-}
287252 return = pure
@@ -313,18 +278,10 @@ instance MonadIO m => MonadIO (ParserK a m) where
313278-- Alternative
314279-------------------------------------------------------------------------------
315280
316−-- | 'Alternative' form of 'Streamly.Internal.Data.Parser.alt'. Backtrack and
317−-- run the second parser if the first one fails.
318−--
319−-- The "some" and "many" operations of alternative accumulate results in a pure
320−-- list which is not scalable and streaming. Instead use
321−-- 'Streamly.Internal.Data.Parser.some' and
322−-- 'Streamly.Internal.Data.Parser.many' for fusible operations with composable
323−-- accumulation of results.
324−--
325−-- See also 'Streamly.Internal.Data.Parser.alt'. This 'Alternative' instance
326−-- does not fuse, use 'Streamly.Internal.Data.Parser.alt' when you need
327−-- fusion.
281+-- | @p1 \<|> p2@ passes the input to parser p1, if it succeeds, the result is
282+-- returned. However, if p1 fails, the parser driver backtracks and tries the
283+-- same input on the alternative parser p2, returning the result if it
284+-- succeeds.
328285--
329286instance Monad m => Alternative (ParserK a m) where
330287 {-# INLINE empty #-}
@@ -447,7 +404,7 @@ adaptCWith pstep initial extract cont !offset0 !usedCount !input = do
447404 go !_ !cur !pst | cur >= end =
448405 onContinue ((end - start) `div` SIZE_OF(a)) pst
449406 go !_ !cur !pst = do
450− let !x = unsafeInlineIO $ peekByteIndex cur contents
407+ let !x = unsafeInlineIO $ peekAt cur contents
451408 pRes <- pstep pst x
452409 let elemSize = SIZE_OF(a)
453410 next = INDEX_NEXT(cur,a)
@@ -504,7 +461,8 @@ adaptCWith pstep initial extract cont !offset0 !usedCount !input = do
504461 parseCont !cnt !pst (Chunk arr) = parseContChunk cnt 0 pst arr
505462 parseCont !cnt !pst None = parseContNothing cnt pst
506463
507−-- | Convert a raw byte 'Parser' to a chunked 'ParserK'.
464+-- | Convert an element 'Parser' to a chunked 'ParserK'. A chunked parser is
465+-- more efficient than an element parser.
508466--
509467-- /Pre-release/
510468--
@@ -606,7 +564,9 @@ adaptWith pstep initial extract cont !relPos !usedCount !input = do
606564 parseCont !cnt !pst (Chunk arr) = parseContChunk cnt pst arr
607565 parseCont !cnt !pst None = parseContNothing cnt pst
608566
609−-- | Similar to "adaptC" but for non-chunked input.
567+-- | Convert a 'Parser' to 'ParserK'.
568+--
569+-- /Pre-release/
610570--
611571{-# INLINE_LATE adapt #-}
612572adapt :: Monad m => ParserD.Parser a m b -> ParserK a m b
@@ -734,7 +694,10 @@ adaptCGWith pstep initial extract cont !offset0 !usedCount !input = do
734694 parseCont !cnt !pst (Chunk arr) = parseContChunk cnt 0 pst arr
735695 parseCont !cnt !pst None = parseContNothing cnt pst
736696
737−-- | Similar to "adaptC" but is not constrained.
697+-- | A generic 'adaptC'. Similar to 'adaptC' but is not constrained to 'Unbox'
698+-- types.
699+--
700+-- /Pre-release/
738701--
739702{-# INLINE_LATE adaptCG #-}
740703adaptCG ::
core/src/Streamly/Internal/Data/Ring.hs+7−4
@@ -79,7 +79,7 @@ import Foreign.Storable
7979import Foreign.ForeignPtr (ForeignPtr, withForeignPtr, touchForeignPtr)
8080import Foreign.ForeignPtr.Unsafe (unsafeForeignPtrToPtr)
8181import Foreign.Ptr (plusPtr, minusPtr, castPtr)
82−import Streamly.Internal.Data.Unbox as Unboxed (Unbox(peekByteIndex))
82+import Streamly.Internal.Data.Unbox as Unboxed (Unbox(peekAt))
8383import GHC.ForeignPtr (mallocPlainForeignPtrAlignedBytes)
8484import GHC.Ptr (Ptr(..))
8585import Streamly.Internal.Data.MutArray.Type (MutArray)
@@ -411,7 +411,7 @@ unsafeEqArrayN Ring{..} rh A.Array{..} nBytes
411411
412412 check p i = do
413413 (relem :: Word8) <- peek p
414− aelem <- peekByteIndex i w8Contents
414+ aelem <- peekAt i w8Contents
415415 if relem == aelem
416416 then go (p `plusPtr` 1) (i + 1)
417417 else return False
@@ -445,7 +445,7 @@ unsafeEqArray Ring{..} rh A.Array{..} =
445445
446446 check p i = do
447447 (relem :: Word8) <- peek p
448− aelem <- peekByteIndex i w8Contents
448+ aelem <- peekAt i w8Contents
449449 if relem == aelem
450450 then go (p `plusPtr` 1) (i + 1)
451451 else return False
@@ -558,7 +558,8 @@ data Tuple4' a b c d = Tuple4' !a !b !c !d deriving Show
558558{-# INLINE slidingWindowWith #-}
559559slidingWindowWith :: forall m a b. (MonadIO m, Storable a, Unbox a)
560560 => Int -> Fold m ((a, Maybe a), m (MutArray a)) b -> Fold m a b
561−slidingWindowWith n (Fold step1 initial1 extract1) = Fold step initial extract
561+slidingWindowWith n (Fold step1 initial1 extract1 final1) =
562+ Fold step initial extract final
562563
563564 where
564565
@@ -601,6 +602,8 @@ slidingWindowWith n (Fold step1 initial1 extract1) = Fold step initial extract
601602
602603 extract (Tuple4' _ _ _ st) = extract1 st
603604
605+ final (Tuple4' _ _ _ st) = final1 st
606+
604607-- | @slidingWindow collector@ is an incremental sliding window
605608-- fold that does not require all the intermediate elements in a computation.
606609-- This maintains @n@ elements in the window, when a new element comes it slides
core/src/Streamly/Internal/Data/Ring/Generic.hs+1−1
@@ -69,7 +69,7 @@ createRing count = liftIO $ do
6969-- the Ring.
7070{-# INLINE writeLastN #-}
7171writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
72−writeLastN n = Fold step initial extract
72+writeLastN n = Fold step initial extract extract
7373
7474 where
7575
core/src/Streamly/Internal/Data/Serialize.hsdeleted−278
@@ -1,278 +0,0 @@
1−-- |
2−-- Module : Streamly.Internal.Data.Serialize
3−-- Copyright : (c) 2023 Composewell Technologies
4−-- License : BSD3-3-Clause
5−-- Maintainer : streamly@composewell.com
6−-- Portability : GHC
7−--
8−
9−module Streamly.Internal.Data.Serialize
10− ( Size(..)
11− , Serialize(..)
12− , encode
13− , pinnedEncode
14− , decode
15− ) where
16−
17−--------------------------------------------------------------------------------
18−-- Imports
19−--------------------------------------------------------------------------------
20−
21−#include "assert.hs"
22−
23−#ifdef DEBUG
24−import Control.Exception (assert)
25−#endif
26−
27−import Control.Monad (void)
28−import Data.List (foldl')
29−import Data.Proxy (Proxy (..))
30−import Streamly.Internal.Data.Unbox
31− ( MutableByteArray(..)
32− , PinnedState(..)
33− , Unbox
34− )
35−import Streamly.Internal.Data.Array.Type (Array(..))
36−import Streamly.Internal.System.IO (unsafeInlineIO)
37−import GHC.Int (Int16(..), Int32(..), Int64(..), Int8(..))
38−import GHC.Word (Word16(..), Word32(..), Word64(..), Word8(..))
39−import GHC.Stable (StablePtr(..))
40−
41−import qualified Streamly.Internal.Data.Unbox as Unbox
42−import qualified Streamly.Internal.Data.Array as Array
43−
44−import GHC.Exts
45−
46−--------------------------------------------------------------------------------
47−-- Types
48−--------------------------------------------------------------------------------
49−
50−-- XXX Use (a -> Sum Int) instead, remove the Size type
51−
52−-- | A left fold step to fold a generic structure to its serializable size.
53−newtype Size a = Size (Int -> a -> Int) -- a left fold or Sum monoid
54−
55−-- | A type implementing the 'Serialize' interface supplies operations for
56−-- reading and writing the type from and to a mutable byte array (an unboxed
57−-- representation of the type) in memory. The read operation 'deserialize'
58−-- deserializes the boxed type from the mutable byte array. The write operation
59−-- 'serialize' serializes the boxed type to the mutable byte array.
60−--
61−-- IMPORTANT: The serialized data's byte ordering is not normalized, which means
62−-- it remains the same as the machine's byte order where the function is
63−-- executed. Consequently, it may not be compatible across machines with
64−-- different byte ordering.
65−--
66−-- 'Serialize' contains enough information to serialize and deserialize variable
67−-- length types.
68−--
69−-- >>> import Streamly.Internal.Data.Serialize (Serialize(..), Size(..))
70−--
71−-- >>> :{
72−-- data Object = Object
73−-- { _obj1 :: [Int]
74−-- , _obj2 :: Int
75−-- }
76−-- :}
77−--
78−-- >>> :{
79−-- instance Serialize Object where
80−-- size =
81−-- case (size :: Size [Int], size :: Size Int) of
82−-- (Size f, Size g) ->
83−-- Size $ \acc obj ->
84−-- acc + f 0 (_obj1 obj) + g 0 (_obj2 obj)
85−-- deserialize i arr len = do
86−-- (i1, x0) <- deserialize i arr len
87−-- (i2, x1) <- deserialize i1 arr len
88−-- pure (i2, Object x0 x1)
89−-- serialize i arr (Object x0 x1) = do
90−-- i1 <- serialize i arr x0
91−-- i2 <- serialize i1 arr x1
92−-- pure i2
93−-- :}
94−--
95−class Serialize a where
96− -- | Get the 'Size', in bytes, reqired to store the serialized
97− -- representation of the type. Size cannot be zero.
98− size :: Size a
99−
100− -- We can implement the following functions without returning the `Int`
101− -- offset but that may require traversing the Haskell structure again to get
102− -- the size. Therefore, this is a performance optimization.
103−
104− -- | @deserialize offset array arrayLen@ deserializes a value from the
105− -- given byte-index in the array. Returns a tuple of the next byte-index
106− -- and the deserialized value.
107− deserialize :: Int -> MutableByteArray -> Int -> IO (Int, a)
108−
109− -- | Write the serialized representation of the value in the array at the
110− -- given byte-index. Returns the next byte-index. This is an unsafe
111− -- operation, the programmer must ensure that the array has enough space
112− -- available in the array to serialize the value as determined by the
113− -- @size@ operation.
114− serialize :: Int -> MutableByteArray -> a -> IO Int
115−
116−--------------------------------------------------------------------------------
117−-- Instances
118−--------------------------------------------------------------------------------
119−
120−-- XXX We do not need to pass the end of data offset, we can just check the
121−-- mutable array end here to avoid a crash, and when we return from deserialize
122−-- we can check if the offset returned is beyond the bound or not.
123−--
124−#ifdef DEBUG
125−{-# INLINE checkBounds #-}
126−checkBounds :: String -> Int -> MutableByteArray -> IO ()
127−checkBounds _label _off _arr = do
128− sz <- sizeOfMutableByteArray _arr
129− if (_off > sz)
130− then error
131− $ _label
132− ++ ": accessing array at offset = "
133− ++ show (_off - 1)
134− ++ " max valid offset = " ++ show (sz - 1)
135− else return ()
136−#endif
137−
138−-- Note: Instead of passing around the size parameter, we can use
139−-- (sizeOfMutableByteArray arr) for checking the array bound, but that turns
140−-- out to be more expensive.
141−--
142−-- Another way to optimize this is to avoid the check for fixed size
143−-- structures. For fixed size structures we can do a check at the top level and
144−-- then use checkless deserialization using the Unbox type class. That will
145−-- require ConstSize and VarSize constructors in size. The programmer can
146−-- bundle all const size fields in a newtype to make serialization faster. This
147−-- can speed up the computation of size when serializing and checking size when
148−-- deserialing.
149−--
150−-- For variable size non-recursive structures a separate size validation method
151−-- could be used to validate the size before deserializing. "validate" can also
152−-- be used to collpase multiple chunks of arrays coming from network into a
153−-- single array for deserializing. But that can also be done by framing the
154−-- serialized value with a size header.
155−--
156−{-# INLINE deserializeUnsafe #-}
157−deserializeUnsafe :: forall a. Unbox a => Int -> MutableByteArray -> Int -> IO (Int, a)
158−deserializeUnsafe off arr sz =
159− let next = off + Unbox.sizeOf (Proxy :: Proxy a)
160− in do
161− -- Keep likely path in the straight branch.
162− if (next <= sz)
163− then Unbox.peekByteIndex off arr >>= \val -> pure (next, val)
164− else error
165− $ "deserialize: accessing array at offset = "
166− ++ show (next - 1)
167− ++ " max valid offset = " ++ show (sz - 1)
168−
169−{-# INLINE serializeUnsafe #-}
170−serializeUnsafe :: forall a. Unbox a => Int -> MutableByteArray -> a -> IO Int
171−serializeUnsafe off arr val =
172− let next = off + Unbox.sizeOf (Proxy :: Proxy a)
173− in do
174−#ifdef DEBUG
175− checkBounds "serialize" next arr
176−#endif
177− Unbox.pokeByteIndex off arr val
178− pure next
179−
180−#define DERIVE_SERIALIZE_FROM_UNBOX(_type) \
181−instance Serialize _type where \
182−; {-# INLINE size #-} \
183−; size = Size (\acc _ -> acc + Unbox.sizeOf (Proxy :: Proxy _type)) \
184−; {-# INLINE deserialize #-} \
185−; deserialize off arr end = deserializeUnsafe off arr end :: IO (Int, _type) \
186−; {-# INLINE serialize #-} \
187−; serialize = \
188− serializeUnsafe :: Int -> MutableByteArray -> _type -> IO Int
189−
190−DERIVE_SERIALIZE_FROM_UNBOX(Bool)
191−DERIVE_SERIALIZE_FROM_UNBOX(Char)
192−DERIVE_SERIALIZE_FROM_UNBOX(Int8)
193−DERIVE_SERIALIZE_FROM_UNBOX(Int16)
194−DERIVE_SERIALIZE_FROM_UNBOX(Int32)
195−DERIVE_SERIALIZE_FROM_UNBOX(Int)
196−DERIVE_SERIALIZE_FROM_UNBOX(Int64)
197−DERIVE_SERIALIZE_FROM_UNBOX(Word)
198−DERIVE_SERIALIZE_FROM_UNBOX(Word8)
199−DERIVE_SERIALIZE_FROM_UNBOX(Word16)
200−DERIVE_SERIALIZE_FROM_UNBOX(Word32)
201−DERIVE_SERIALIZE_FROM_UNBOX(Word64)
202−DERIVE_SERIALIZE_FROM_UNBOX(Double)
203−DERIVE_SERIALIZE_FROM_UNBOX(Float)
204−DERIVE_SERIALIZE_FROM_UNBOX((StablePtr a))
205−DERIVE_SERIALIZE_FROM_UNBOX((Ptr a))
206−DERIVE_SERIALIZE_FROM_UNBOX((FunPtr a))
207−
208−instance forall a. Serialize a => Serialize [a] where
209−
210− -- {-# INLINE size #-}
211− size = Size $ \acc xs ->
212− case size :: Size a of
213− Size f -> foldl' f (acc + (Unbox.sizeOf (Proxy :: Proxy Int))) xs
214−
215− -- Inlining this causes large compilation times for tests
216− {-# INLINABLE deserialize #-}
217− deserialize off arr sz = do
218− (off1, len) <- deserialize off arr sz :: IO (Int, Int)
219− let
220− peekList f o i | i >= 3 = do
221− -- Unfold the loop three times
222− (o1, x1) <- deserialize o arr sz
223− (o2, x2) <- deserialize o1 arr sz
224− (o3, x3) <- deserialize o2 arr sz
225− peekList (f . (\xs -> x1:x2:x3:xs)) o3 (i - 3)
226− peekList f o 0 = pure (o, f [])
227− peekList f o i = do
228− (o1, x) <- deserialize o arr sz
229− peekList (f . (x:)) o1 (i - 1)
230− peekList id off1 len
231−
232− -- Inlining this causes large compilation times for tests
233− {-# INLINABLE serialize #-}
234− serialize off arr val = do
235− void $ serialize off arr (length val)
236− let off1 = off + Unbox.sizeOf (Proxy :: Proxy Int)
237− let pokeList o [] = pure o
238− pokeList o (x:xs) = do
239− o1 <- serialize o arr x
240− pokeList o1 xs
241− pokeList off1 val
242−
243−--------------------------------------------------------------------------------
244−-- High level functions
245−--------------------------------------------------------------------------------
246−
247−{-# INLINE encodeAs #-}
248−encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
249−encodeAs ps a =
250− unsafeInlineIO $ do
251− let len =
252− case size :: Size a of
253− Size f -> f 0 a
254− -- We encode the length of the encoding as a header hence the 8 extra
255− -- bytes to encode Int64
256− mbarr <- Unbox.newBytesAs ps (8 + len)
257− off1 <- serialize 0 mbarr (fromIntegral len :: Int64)
258− off2 <- serialize off1 mbarr a
259− assertM(off2 == len + off1)
260− pure $ Array mbarr 0 len
261−
262−{-# INLINE encode #-}
263−encode :: Serialize a => a -> Array Word8
264−encode = encodeAs Unpinned
265−
266−{-# INLINE pinnedEncode #-}
267−pinnedEncode :: Serialize a => a -> Array Word8
268−pinnedEncode = encodeAs Pinned
269−
270−{-# INLINE decode #-}
271−decode :: Serialize a => Array Word8 -> a
272−decode arr@(Array {..}) = unsafeInlineIO $ do
273− let lenArr = Array.length arr
274− (off1, lenEncoding :: Int64) <- deserialize 0 arrContents lenArr
275− (off2, val) <- deserialize off1 arrContents lenArr
276− assertM(fromIntegral lenEncoding + off1 == off2)
277− assertM(lenArr == off2)
278− pure val
core/src/Streamly/Internal/Data/Serialize/TH.hs+337−247
@@ -7,10 +7,23 @@
77-- Maintainer : streamly@composewell.com
88-- Stability : experimental
99-- Portability : GHC
10−--
10+
11+-- XXX Instead of applying the product constructor in one go can we apply it
12+-- one at a time in case of too many arguments, compiler may not have to save
13+-- them in local vars.
14+
1115module Streamly.Internal.Data.Serialize.TH
12− ( deriveSerialize
16+ (
17+ -- Deriving
18+ deriveSerialize
1319 , deriveSerializeWith
20+
21+ -- Utilities
22+ , module Streamly.Internal.Data.Serialize.TH.Bottom
23+ -- ** Common
24+ , module Streamly.Internal.Data.Serialize.TH.Common
25+ -- ** RecHeader
26+ , module Streamly.Internal.Data.Serialize.TH.RecHeader
1427 ) where
1528
1629--------------------------------------------------------------------------------
@@ -22,67 +35,26 @@ import Data.Word (Word16, Word32, Word64, Word8)
2235
2336import Language.Haskell.TH
2437import Language.Haskell.TH.Syntax
25−import Streamly.Internal.Data.Serialize
38+import Streamly.Internal.Data.Serialize.Type
2639
2740import Streamly.Internal.Data.Unbox.TH
2841 ( DataCon(..)
2942 , DataType(..)
30− , appsT
31− , plainInstanceD
3243 , reifyDataType
3344 )
3445
35−--------------------------------------------------------------------------------
36−-- Helpers
37−--------------------------------------------------------------------------------
38−
39−type Field = (Maybe Name, Type)
40−
41−_x :: Name
42−_x = mkName "x"
43−
44−_acc :: Name
45−_acc = mkName "acc"
46−
47−_arr :: Name
48−_arr = mkName "arr"
49−
50−_tag :: Name
51−_tag = mkName "tag"
52−
53−_initialOffset :: Name
54−_initialOffset = mkName "initialOffset"
55−
56−_endOffset :: Name
57−_endOffset = mkName "endOffset"
46+import qualified Streamly.Internal.Data.Serialize.TH.RecHeader as RecHeader
5847
59−_val :: Name
60−_val = mkName "val"
61−
62−mkFieldName :: Int -> Name
63−mkFieldName i = mkName ("field" ++ show i)
64−
65−makeI :: Int -> Name
66−makeI i = mkName $ "i" ++ show i
67−
68−makeA :: Int -> Name
69−makeA i = mkName $ "a" ++ show i
48+import Streamly.Internal.Data.Serialize.TH.Bottom
49+import Streamly.Internal.Data.Serialize.TH.Common
50+import Streamly.Internal.Data.Serialize.TH.RecHeader
7051
7152--------------------------------------------------------------------------------
7253-- Domain specific helpers
7354--------------------------------------------------------------------------------
7455
75−matchConstructor :: Name -> Int -> Q Exp -> Q Match
76−matchConstructor cname numFields exp0 =
77− match
78− (conP cname (map varP (map mkFieldName [0 .. (numFields - 1)])))
79− (normalB exp0)
80− []
81−
8256exprGetSize :: Q Exp -> (Int, Type) -> Q Exp
83−exprGetSize acc (i, ty) =
84− [|case size :: Size $(pure ty) of
85− Size f -> f $(acc) $(varE (mkFieldName i))|]
57+exprGetSize acc (i, _) = [|addSizeTo $(acc) $(varE (mkFieldName i))|]
8658
8759getTagSize :: Int -> Int
8860getTagSize numConstructors
@@ -106,41 +78,63 @@ getTagType numConstructors
10678-- Size
10779--------------------------------------------------------------------------------
10880
109−isUnitType :: [DataCon] -> Bool
110−isUnitType [DataCon _ _ _ []] = True
111−isUnitType _ = False
81+getNameBaseLen :: Name -> Word8
82+getNameBaseLen cname =
83+ let x = length (nameBase cname)
84+ in if x > 63
85+ then error "Max Constructor Len: 63 characters"
86+ else fromIntegral x
11287
113−mkSizeOfExpr :: Type -> [DataCon] -> Q Exp
114−mkSizeOfExpr headTy constructors =
115− case constructors of
116− [] ->
117− [|error
118− ("Attempting to get size with no constructors (" ++
119− $(lift (pprint headTy)) ++ ")")|]
120− -- One constructor with no fields is a unit type. Size of a unit type is
121− -- 1. XXX Use isUnitType?
122− [constructor@(DataCon _ _ _ fields)] ->
123− case fields of
124− -- Unit type
125− [] ->
126− appE
127− (conE 'Size)
128− (lamE [varP _acc, wildP] [| $(varE _acc) + 1 |])
129− -- Product type
130− _ ->
131− appE
132− (conE 'Size)
133− (lamE
134− [varP _acc, varP _x]
135− (caseE (varE _x)
136− [matchCons (varE _acc) constructor]))
137− -- Sum type
138− _ -> sizeOfHeadDt
88+conEncLen :: Name -> Word8
89+conEncLen cname = getNameBaseLen cname + 1
90+
91+--------------------------------------------------------------------------------
92+-- Size
93+--------------------------------------------------------------------------------
94+
95+mkSizeOfExpr :: Bool -> Bool -> TypeOfType -> Q Exp
96+mkSizeOfExpr True False tyOfTy =
97+ case tyOfTy of
98+ UnitType cname ->
99+ lamE
100+ [varP _acc, wildP]
101+ [|$(varE _acc) + $(litIntegral (conEncLen cname))|]
102+ TheType con ->
103+ lamE
104+ [varP _acc, varP _x]
105+ (caseE (varE _x) [matchCons (varE _acc) con])
106+ MultiType constructors -> sizeOfHeadDt constructors
139107
140108 where
141109
142− tagSizeExp =
143− litE (IntegerL (fromIntegral (getTagSize (length constructors))))
110+ sizeOfFields acc fields =
111+ foldl' exprGetSize acc $ zip [0..] fields
112+
113+ matchCons acc (SimpleDataCon cname fields) =
114+ let a = litIntegral (conEncLen cname)
115+ b = sizeOfFields acc (map snd fields)
116+ expr = [|$(a) + $(b)|]
117+ in matchConstructor cname (length fields) expr
118+
119+ sizeOfHeadDt cons =
120+ let acc = [|$(varE _acc)|]
121+ in lamE
122+ [varP _acc, varP _x]
123+ (caseE (varE _x) (fmap (matchCons acc) cons))
124+
125+mkSizeOfExpr False False tyOfTy =
126+ case tyOfTy of
127+ UnitType _ -> lamE [varP _acc, wildP] [|$(varE _acc) + 1|]
128+ TheType con ->
129+ lamE
130+ [varP _acc, varP _x]
131+ (caseE (varE _x) [matchCons (varE _acc) con])
132+ MultiType constructors -> sizeOfHeadDt constructors
133+
134+ where
135+
136+ tagSizeExp numConstructors =
137+ litE (IntegerL (fromIntegral (getTagSize numConstructors)))
144138
145139 -- XXX fields of the same type can be folded together, will reduce the code
146140 -- size when there are many fields of the same type.
@@ -153,84 +147,106 @@ mkSizeOfExpr headTy constructors =
153147 sizeOfFields acc fields =
154148 foldl' exprGetSize acc $ zip [0..] fields
155149
156− matchCons acc (DataCon cname _ _ fields) =
150+ matchCons acc (SimpleDataCon cname fields) =
157151 let expr = sizeOfFields acc (map snd fields)
158152 in matchConstructor cname (length fields) expr
159153
160154 -- XXX We fix VarSize for simplicity. Should be changed later.
161− sizeOfHeadDt =
162− let acc = [|$(varE _acc) + $(tagSizeExp)|]
163− f =
164− (lamE
165− [varP _acc, varP _x]
166− (caseE (varE _x) (fmap (matchCons acc) constructors))
167− )
168− in appE (conE 'Size) f
155+ sizeOfHeadDt cons =
156+ let numCons = length cons
157+ acc = [|$(varE _acc) + $(tagSizeExp numCons)|]
158+ in lamE
159+ [varP _acc, varP _x]
160+ (caseE (varE _x) (fmap (matchCons acc) cons))
161+
162+mkSizeOfExpr False True (TheType con) = RecHeader.mkRecSizeOfExpr con
163+
164+mkSizeOfExpr _ _ _ = errorUnimplemented
165+
166+mkSizeDec :: SerializeConfig -> Type -> [DataCon] -> Q [Dec]
167+mkSizeDec (SerializeConfig {..}) headTy cons = do
168+ -- INLINE on sizeOf actually worsens some benchmarks, and improves none
169+ sizeOfMethod <-
170+ mkSizeOfExpr
171+ cfgConstructorTagAsString
172+ cfgRecordSyntaxWithHeader
173+ (typeOfType headTy cons)
174+ pure
175+ ( maybe
176+ []
177+ (\x -> [PragmaD (InlineP 'addSizeTo x FunLike AllPhases)])
178+ cfgInlineSize
179+ ++ [FunD 'addSizeTo [Clause [] (NormalB sizeOfMethod) []]]
180+ )
169181
170182--------------------------------------------------------------------------------
171183-- Peek
172184--------------------------------------------------------------------------------
173185
174−mkDeserializeExprOne :: DataCon -> Q Exp
175−mkDeserializeExprOne (DataCon cname _ _ fields) =
176− case fields of
177− -- Only tag is serialized for unit fields, no actual value
178− [] -> [|pure ($(varE (mkName "i0")), $(conE cname))|]
179− _ ->
180− doE
181− (concat
182− [ fmap makeBind [0 .. (numFields - 1)]
183− , [ noBindS
184− (appE
185− (varE 'pure)
186− (tupE
187− [ varE (makeI numFields)
188− , appsE
189− (conE cname :
190− (map (varE . makeA)
191− [0 .. (numFields - 1)]))
192− ]))
193− ]
194− ])
195− where
196− numFields = length fields
197− makeBind i =
198− bindS
199− (tupP [varP (makeI (i + 1)), varP (makeA i)])
200− [|deserialize $(varE (makeI i)) $(varE _arr) $(varE _endOffset)|]
186+mkDeserializeExpr :: Bool -> Bool -> Type -> TypeOfType -> Q Exp
187+mkDeserializeExpr True False headTy tyOfTy =
188+ case tyOfTy of
189+ UnitType cname -> deserializeConsExpr [SimpleDataCon cname []]
190+ TheType con -> deserializeConsExpr [con]
191+ MultiType cons -> deserializeConsExpr cons
201192
193+ where
202194
203−mkDeserializeExpr :: Type -> [DataCon] -> Q Exp
204−mkDeserializeExpr headTy cons =
205− case cons of
206− [] ->
195+ deserializeConsExpr cons = do
196+ conLen <- newName "conLen"
197+ off1 <- newName "off1"
198+ [|do ($(varP off1), $(varP conLen) :: Word8) <-
199+ deserializeAt
200+ $(varE _initialOffset)
201+ $(varE _arr)
202+ $(varE _endOffset)
203+ $(multiIfE (map (guardCon conLen off1) cons ++ [catchAll]))|]
204+
205+ catchAll =
206+ normalGE
207+ [|True|]
207208 [|error
208− ("Attempting to peek type with no constructors (" ++
209+ ("Found invalid tag while peeking (" ++
209210 $(lift (pprint headTy)) ++ ")")|]
211+
212+ guardCon conLen off con@(SimpleDataCon cname _) = do
213+ let lenCname = getNameBaseLen cname
214+ tag = map c2w (nameBase cname)
215+ normalGE
216+ [|($(litIntegral lenCname) == $(varE conLen))
217+ && $(xorCmp tag off _arr)|]
218+ [|let $(varP (makeI 0)) = $(varE off) + $(litIntegral lenCname)
219+ in $(mkDeserializeExprOne 'deserializeAt con)|]
220+
221+mkDeserializeExpr False False headTy tyOfTy =
222+ case tyOfTy of
210223 -- Unit constructor
211− [(DataCon cname _ _ [])] ->
224+ -- XXX Should we peek and check if the byte value is 0?
225+ UnitType cname ->
212226 [|pure ($(varE _initialOffset) + 1, $(conE cname))|]
213227 -- Product type
214− [con] ->
228+ TheType con ->
215229 letE
216230 [valD (varP (mkName "i0")) (normalB (varE _initialOffset)) []]
217− (mkDeserializeExprOne con)
231+ (mkDeserializeExprOne 'deserializeAt con)
218232 -- Sum type
219− _ ->
233+ MultiType cons -> do
234+ let lenCons = length cons
235+ tagType = getTagType lenCons
220236 doE
221237 [ bindS
222238 (tupP [varP (mkName "i0"), varP _tag])
223− [|deserialize $(varE _initialOffset) $(varE _arr) $(varE _endOffset)|]
239+ [|deserializeAt $(varE _initialOffset) $(varE _arr) $(varE _endOffset)|]
224240 , noBindS
225241 (caseE
226242 (sigE (varE _tag) (conT tagType))
227243 (map peekMatch (zip [0 ..] cons) ++ [peekErr]))
228244 ]
229245 where
230− lenCons = length cons
231− tagType = getTagType lenCons
232246 peekMatch (i, con) =
233− match (litP (IntegerL i)) (normalB (mkDeserializeExprOne con)) []
247+ match
248+ (litP (IntegerL i))
249+ (normalB (mkDeserializeExprOne 'deserializeAt con)) []
234250 peekErr =
235251 match
236252 wildP
@@ -241,44 +257,99 @@ mkDeserializeExpr headTy cons =
241257 $(lift (pprint headTy)) ++ ")")|])
242258 []
243259
260+mkDeserializeExpr False True _ (TheType con@(SimpleDataCon _ fields)) = do
261+ deserializeWithKeys <- newName "deserializeWithKeys"
262+ updateFunc <- newName "updateFunc"
263+ updateFuncDec <- RecHeader.conUpdateFuncDec updateFunc fields
264+ deserializeWithKeysDec <-
265+ RecHeader.mkDeserializeKeysDec deserializeWithKeys updateFunc con
266+ letE
267+ (pure <$> (deserializeWithKeysDec ++ updateFuncDec))
268+ (RecHeader.mkRecDeserializeExpr
269+ _initialOffset
270+ _endOffset
271+ deserializeWithKeys
272+ con)
273+
274+mkDeserializeExpr _ _ _ _ = errorUnimplemented
275+
276+mkDeserializeDec :: SerializeConfig -> Type -> [DataCon] -> Q [Dec]
277+mkDeserializeDec (SerializeConfig {..}) headTy cons = do
278+ peekMethod <-
279+ mkDeserializeExpr
280+ cfgConstructorTagAsString
281+ cfgRecordSyntaxWithHeader
282+ headTy
283+ (typeOfType headTy cons)
284+ pure
285+ ( maybe
286+ []
287+ (\x -> [PragmaD (InlineP 'deserializeAt x FunLike AllPhases)])
288+ cfgInlineDeserialize
289+ ++
290+ [ FunD
291+ 'deserializeAt
292+ [ Clause
293+ (if isUnitType cons && not cfgConstructorTagAsString
294+ then [VarP _initialOffset, WildP, WildP]
295+ else [VarP _initialOffset, VarP _arr, VarP _endOffset])
296+ (NormalB peekMethod)
297+ []
298+ ]
299+ ]
300+ )
301+
244302--------------------------------------------------------------------------------
245303-- Poke
246304--------------------------------------------------------------------------------
247305
248306mkSerializeExprTag :: Name -> Int -> Q Exp
249307mkSerializeExprTag tagType tagVal =
250− [|serialize
308+ [|serializeAt
251309 $(varE _initialOffset)
252310 $(varE _arr)
253311 $((sigE (litE (IntegerL (fromIntegral tagVal))) (conT tagType)))|]
254312
255−mkSerializeExprFields :: [Field] -> Q Exp
256−mkSerializeExprFields fields =
257− case fields of
258− -- Unit constructor, do nothing just tag is enough
259− [] -> [|pure ($(varE (mkName "i0")))|]
260− _ ->
261− doE
262− (fmap makeBind [0 .. (numFields - 1)] ++
263− [noBindS ([|pure $(varE (makeI numFields))|])])
264− where
265− numFields = length fields
266− makeBind i =
267− bindS
268− (varP (makeI (i + 1)))
269− [|serialize $(varE (makeI i)) $(varE _arr) $(varE (mkFieldName i))|]
270−
271−mkSerializeExpr :: Type -> [DataCon] -> Q Exp
272−mkSerializeExpr headTy cons =
273− case cons of
274− [] ->
275− [|error
276− ("Attempting to poke type with no constructors (" ++
277− $(lift (pprint headTy)) ++ ")")|]
313+mkSerializeExpr :: Bool -> Bool -> TypeOfType -> Q Exp
314+mkSerializeExpr True False tyOfTy =
315+ case tyOfTy of
316+ -- Unit type
317+ UnitType cname ->
318+ caseE
319+ (varE _val)
320+ [serializeDataCon (SimpleDataCon cname [])]
321+ -- Product type
322+ (TheType con) ->
323+ caseE
324+ (varE _val)
325+ [serializeDataCon con]
326+ -- Sum type
327+ (MultiType cons) ->
328+ caseE
329+ (varE _val)
330+ (map serializeDataCon cons)
331+
332+ where
333+
334+ serializeDataCon (SimpleDataCon cname fields) = do
335+ let tagLen8 = getNameBaseLen cname
336+ conEnc = tagLen8 : map c2w (nameBase cname)
337+ matchConstructor
338+ cname
339+ (length fields)
340+ (doE [ bindS
341+ (varP (mkName "i0"))
342+ (serializeW8List _initialOffset _arr conEnc)
343+ , noBindS (mkSerializeExprFields 'serializeAt fields)
344+ ])
345+
346+mkSerializeExpr False False tyOfTy =
347+ case tyOfTy of
278348 -- Unit type
279− [(DataCon _ _ _ [])] -> [|pure ($(varE _initialOffset) + 1)|]
349+ UnitType _ ->
350+ [|serializeAt $(varE _initialOffset) $(varE _arr) (0 :: Word8)|]
280351 -- Product type
281− [(DataCon cname _ _ fields)] ->
352+ (TheType (SimpleDataCon cname fields)) ->
282353 letE
283354 [valD (varP (mkName "i0")) (normalB (varE _initialOffset)) []]
284355 (caseE
@@ -286,25 +357,57 @@ mkSerializeExpr headTy cons =
286357 [ matchConstructor
287358 cname
288359 (length fields)
289− (mkSerializeExprFields fields)
360+ (mkSerializeExprFields 'serializeAt fields)
290361 ])
291362 -- Sum type
292− _ ->
363+ (MultiType cons) -> do
364+ let lenCons = length cons
365+ tagType = getTagType lenCons
293366 caseE
294367 (varE _val)
295− (map (\(tagVal, (DataCon cname _ _ fields)) ->
368+ (map (\(tagVal, (SimpleDataCon cname fields)) ->
296369 matchConstructor
297370 cname
298371 (length fields)
299372 (doE [ bindS
300373 (varP (mkName "i0"))
301374 (mkSerializeExprTag tagType tagVal)
302− , noBindS (mkSerializeExprFields fields)
375+ , noBindS
376+ (mkSerializeExprFields
377+ 'serializeAt
378+ fields)
303379 ]))
304380 (zip [0 ..] cons))
305− where
306− lenCons = length cons
307− tagType = getTagType lenCons
381+
382+mkSerializeExpr False True (TheType con) =
383+ RecHeader.mkRecSerializeExpr _initialOffset con
384+
385+mkSerializeExpr _ _ _ = errorUnimplemented
386+
387+mkSerializeDec :: SerializeConfig -> Type -> [DataCon] -> Q [Dec]
388+mkSerializeDec (SerializeConfig {..}) headTy cons = do
389+ pokeMethod <-
390+ mkSerializeExpr
391+ cfgConstructorTagAsString
392+ cfgRecordSyntaxWithHeader
393+ (typeOfType headTy cons)
394+ pure
395+ ( maybe
396+ []
397+ (\x -> [PragmaD (InlineP 'serializeAt x FunLike AllPhases)])
398+ cfgInlineSerialize
399+ ++
400+ [FunD
401+ 'serializeAt
402+ [ Clause
403+ (if isUnitType cons && not cfgConstructorTagAsString
404+ then [VarP _initialOffset, VarP _arr, WildP]
405+ else [VarP _initialOffset, VarP _arr, VarP _val])
406+ (NormalB pokeMethod)
407+ []
408+ ]
409+ ]
410+ )
308411
309412--------------------------------------------------------------------------------
310413-- Main
@@ -326,6 +429,7 @@ mkSerializeExpr headTy cons =
326429-- Usage:
327430-- @
328431-- $(deriveSerializeInternal
432+-- serializeConfig
329433-- [AppT (ConT ''Serialize) (VarT (mkName "b"))]
330434-- (AppT
331435-- (AppT (ConT ''CustomDataType) (VarT (mkName "a")))
@@ -339,102 +443,88 @@ mkSerializeExpr headTy cons =
339443-- [(Nothing, (ConT ''Bool)), (Nothing, (VarT (mkName "b")))]
340444-- ])
341445-- @
342−deriveSerializeInternal :: Cxt -> Type -> [DataCon] -> Q [Dec]
343−deriveSerializeInternal preds headTy cons = do
344− sizeOfMethod <- mkSizeOfExpr headTy cons
345− peekMethod <- mkDeserializeExpr headTy cons
346− pokeMethod <- mkSerializeExpr headTy cons
347− let methods =
348− -- INLINE on sizeOf actually worsens some benchmarks, and improves
349− -- none
350− [ -- PragmaD (InlineP 'size Inlinable FunLike AllPhases)
351− FunD 'size [Clause [] (NormalB sizeOfMethod) []]
352− , PragmaD (InlineP 'deserialize Inline FunLike AllPhases)
353− , FunD
354− 'deserialize
355− [ Clause
356− (if isUnitType cons
357− then [VarP _initialOffset, WildP, WildP]
358− else [VarP _initialOffset, VarP _arr, VarP _endOffset])
359− (NormalB peekMethod)
360− []
361− ]
362− , PragmaD (InlineP 'serialize Inline FunLike AllPhases)
363− , FunD
364− 'serialize
365− [ Clause
366− (if isUnitType cons
367− then [VarP _initialOffset, WildP, WildP]
368− else [VarP _initialOffset, VarP _arr, VarP _val])
369− (NormalB pokeMethod)
370− []
371− ]
372− ]
373− return [plainInstanceD preds (AppT (ConT ''Serialize) headTy) methods]
374−
375−-- | Template haskell helper to create instances of 'Serialize' automatically.
446+deriveSerializeInternal ::
447+ SerializeConfig -> Type -> [DataCon] -> ([Dec] -> Q [Dec]) -> Q [Dec]
448+deriveSerializeInternal conf headTy cons next = do
449+ sizeDec <- mkSizeDec conf headTy cons
450+ peekDec <- mkDeserializeDec conf headTy cons
451+ pokeDec <- mkSerializeDec conf headTy cons
452+ let methods = concat [sizeDec, peekDec, pokeDec]
453+ next methods
454+
455+-- | @deriveSerializeWith config-modifier instance-dec@ generates a template
456+-- Haskell splice consisting of a declaration of a 'Serialize' instance.
457+-- @instance-dec@ is a template Haskell declaration splice consisting of a
458+-- standard Haskell instance declaration without the type class methods (e.g.
459+-- @[d|instance Serialize a => Serialize (Maybe a)|]@).
376460--
377−-- Consider the datatype:
378−-- @
379−-- data CustomDataType a b c = ...
380−-- @
461+-- The type class methods for the given instance are generated according to the
462+-- supplied @config-modifier@ parameter. See 'SerializeConfig' for default
463+-- configuration settings.
381464--
382−-- Usage: @$(deriveSerialize ''CustomDataType)@
465+-- Usage:
383466--
384−-- Note: All type variables automatcally get an "Serialize" constraint.
385−-- The derived code will look like the following,
386467-- @
387−-- instance (Serialize a, Serialize b, Serialize c) => Serialize (CustomDataType a b c) where
388−-- ...
468+-- \$(deriveSerializeWith
469+-- ( inlineSerializeAt (Just NoInline)
470+-- . inlineDeserializeAt (Just NoInline)
471+-- )
472+-- [d|instance Serialize a => Serialize (Maybe a)|])
389473-- @
390−--
391−-- To control which type variables get the Serialize constraint, use
392−-- 'deriveSerializeWith'.
393−deriveSerialize :: Name -> Q [Dec]
394−deriveSerialize name = do
395− dt <- reifyDataType name
396− let preds = map (unboxPred . VarT) (dtTvs dt)
397− headTy = appsT (ConT name) (map VarT (dtTvs dt))
398− cons = dtCons dt
399− deriveSerializeInternal preds headTy cons
474+deriveSerializeWith ::
475+ (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
476+deriveSerializeWith modifier mDecs = do
477+ dec <- mDecs
478+ case dec of
479+ [InstanceD mo preds headTyWC []] -> do
480+ let headTy = unwrap dec headTyWC
481+ dt <- reifyDataType (getMainTypeName dec headTy)
482+ let cons = dtCons dt
483+ deriveSerializeInternal
484+ (modifier serializeConfig) headTy cons (next mo preds headTyWC)
485+ _ -> errorMessage dec
400486
401487 where
402488
403− unboxPred ty =
404−#if MIN_VERSION_template_haskell(2,10,0)
405− AppT (ConT ''Serialize) ty
406−#else
407− ClassP ''Serialize [ty]
408−#endif
489+ next mo preds headTyWC methods = pure [InstanceD mo preds headTyWC methods]
490+
491+ errorMessage dec =
492+ error $ unlines
493+ [ "Error: deriveSerializeWith:"
494+ , ""
495+ , ">> " ++ pprint dec
496+ , ""
497+ , "The supplied declaration is not a valid instance declaration."
498+ , "Provide a valid Haskell instance declaration without a body."
499+ , ""
500+ , "Examples:"
501+ , "instance Serialize (Proxy a)"
502+ , "instance Serialize a => Serialize (Identity a)"
503+ , "instance Serialize (TableT Identity)"
504+ ]
505+
506+ unwrap _ (AppT (ConT _) r) = r
507+ unwrap dec _ = errorMessage dec
508+
509+ getMainTypeName dec = go
510+
511+ where
409512
410−-- | Like 'deriveSerialize' but control which types variables get the 'Serialize'
411−-- constraint.
513+ go (ConT nm) = nm
514+ go (AppT l _) = go l
515+ go _ = errorMessage dec
516+
517+-- | Given an 'Serialize' instance declaration splice without the methods (e.g.
518+-- @[d|instance Serialize a => Serialize (Maybe a)|]@), generate an instance
519+-- declaration including all the type class method implementations.
412520--
413−-- Consider the datatype:
414−-- @
415−-- data CustomDataType a b c = ...
416−-- @
521+-- >>> deriveSerialize = deriveSerializeWith id
417522--
418−-- Usage: @$(deriveSerializeWith ["a", "c"] ''CustomDataType)@
523+-- Usage:
419524--
420525-- @
421−-- instance (Serialize a, Serialize c) => Serialize (CustomDataType a b c) where
422−-- ...
526+-- \$(deriveSerialize
527+-- [d|instance Serialize a => Serialize (Maybe a)|])
423528-- @
424−--
425−deriveSerializeWith :: [String] -> Name -> Q [Dec]
426−deriveSerializeWith vars name = do
427− dt <- reifyDataType name
428− let preds = map (unboxPred . VarT) (fmap mkName vars)
429− headTy = appsT (ConT name) (map VarT (dtTvs dt))
430− cons = dtCons dt
431− deriveSerializeInternal preds headTy cons
432−
433− where
434−
435− unboxPred ty =
436−#if MIN_VERSION_template_haskell(2,10,0)
437− AppT (ConT ''Serialize) ty
438−#else
439− ClassP ''Serialize [ty]
440−#endif
529+deriveSerialize :: Q [Dec] -> Q [Dec]
530+deriveSerialize = deriveSerializeWith id
core/src/Streamly/Internal/Data/Serialize/TH/Bottom.hsnew file+475
@@ -0,0 +1,475 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+-- |
4+-- Module : Streamly.Internal.Data.Serialize.TH.Bottom
5+-- Copyright : (c) 2023 Composewell Technologies
6+-- License : BSD3-3-Clause
7+-- Maintainer : streamly@composewell.com
8+-- Stability : experimental
9+-- Portability : GHC
10+--
11+module Streamly.Internal.Data.Serialize.TH.Bottom
12+ (
13+ -- ** Config
14+ SerializeConfig(..)
15+ , serializeConfig
16+ , inlineAddSizeTo
17+ , inlineSerializeAt
18+ , inlineDeserializeAt
19+ , encodeConstrNames
20+ , encodeRecordFields
21+
22+ -- ** Other Utilities
23+ , TypeOfType(..)
24+ , typeOfType
25+ , SimpleDataCon(..)
26+ , simplifyDataCon
27+ , Field
28+ , mkFieldName
29+ , isUnitType
30+ , isRecordSyntax
31+ , c2w
32+ , wListToString
33+ , xorCmp
34+ , serializeW8List
35+ , litIntegral
36+ , litProxy
37+ , matchConstructor
38+ , openConstructor
39+ , makeI
40+ , makeN
41+ , makeA
42+ , int_w8
43+ , int_w32
44+ , w32_int
45+ , w8_int
46+ , _acc
47+ , _arr
48+ , _endOffset
49+ , _initialOffset
50+ , _x
51+ , _tag
52+ , _val
53+ , errorUnsupported
54+ , errorUnimplemented
55+ ) where
56+
57+import Data.Maybe (isJust)
58+import Data.Char (chr, ord)
59+import Data.List (foldl')
60+import Data.Word (Word16, Word32, Word64, Word8)
61+import Data.Bits (Bits, (.|.), shiftL, zeroBits, xor)
62+import Streamly.Internal.System.IO (unsafeInlineIO)
63+import Streamly.Internal.Data.Unbox (Unbox)
64+import Data.Proxy (Proxy)
65+
66+import Language.Haskell.TH
67+import Streamly.Internal.Data.Serialize.Type
68+
69+import qualified Streamly.Internal.Data.Unbox as Unbox
70+
71+import Streamly.Internal.Data.Unbox.TH (DataCon(..))
72+
73+--------------------------------------------------------------------------------
74+-- Config
75+--------------------------------------------------------------------------------
76+
77+-- NOTE: 'Nothing' is not eqvivalant to 'Just Inlinable'. Ie. Having no inline
78+-- specific pragma and having an Inlinable pragma are different. Having an
79+-- Inlinable pragma makes GHC put the code in the interface file whereas having
80+-- no inline specific pragma let's GHC decide whether to put the code in
81+-- interface file or not.
82+
83+-- | Configuration to control how the 'Serialize' instance is generated. The
84+-- configuration is opaque and is modified by composing config modifier
85+-- functions, for example:
86+--
87+-- >>> (inlineSerializeAt (Just NoInline)) . (inlineSerializeAt (Just Inlinable))
88+--
89+-- The default configuration settings are:
90+--
91+-- * 'inlineAddSizeTo' Nothing
92+-- * 'inlineSerializeAt' (Just Inline)
93+-- * 'inlineDeserializeAt' (Just Inline)
94+--
95+-- The following experimental options are also available:
96+--
97+-- * 'encodeConstrNames' False
98+-- * 'encodeRecordFields' False
99+--
100+data SerializeConfig =
101+ SerializeConfig
102+ { cfgInlineSize :: Maybe Inline
103+ , cfgInlineSerialize :: Maybe Inline
104+ , cfgInlineDeserialize :: Maybe Inline
105+ , cfgConstructorTagAsString :: Bool
106+ , cfgRecordSyntaxWithHeader :: Bool
107+ }
108+
109+-- | How should we inline the 'addSizeTo' function? The default is 'Nothing'
110+-- which means left to the compiler. Forcing inline on @addSizeTo@ function
111+-- actually worsens some benchmarks and improves none.
112+inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
113+inlineAddSizeTo v cfg = cfg {cfgInlineSize = v}
114+
115+-- XXX Should we make the default Inlinable instead?
116+
117+-- | How should we inline the 'serialize' function? The default 'Just Inline'.
118+-- However, aggressive inlining can bloat the code and increase in compilation
119+-- times when there are big functions and too many nesting levels so you can
120+-- change it accordingly. A 'Nothing' value leaves the decision to the
121+-- compiler.
122+inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
123+inlineSerializeAt v cfg = cfg {cfgInlineSerialize = v}
124+
125+-- XXX Should we make the default Inlinable instead?
126+
127+-- | How should we inline the 'deserialize' function? See guidelines in
128+-- 'inlineSerializeAt'.
129+inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
130+inlineDeserializeAt v cfg = cfg {cfgInlineDeserialize = v}
131+
132+-- | __Experimental__
133+--
134+-- In sum types, use Latin-1 encoded original constructor names rather than
135+-- binary values to identify constructors. This option is not applicable to
136+-- product types.
137+--
138+-- This option enables the following behavior:
139+--
140+-- * __Reordering__: Order of the fields can be changed without affecting
141+-- serialization.
142+-- * __Addition__: If a field is added in the new version, the old version of
143+-- the data type can still be deserialized by the new version. The new value
144+-- would never occur in the old one.
145+-- * __Deletion__: If a field is deleted in the new version, deserialization
146+-- of the old version will result in an error. TBD: We can possibly designate a
147+-- catch-all case to handle this scenario.
148+--
149+-- Note that if you change a type, change the semantics of a type, or delete a
150+-- field and add a new field with the same name, deserialization of old data
151+-- may result in silent unexpected behavior.
152+--
153+-- This option has to be the same on both encoding and decoding side.
154+--
155+-- The default is 'False'.
156+--
157+encodeConstrNames :: Bool -> SerializeConfig -> SerializeConfig
158+encodeConstrNames v cfg = cfg {cfgConstructorTagAsString = v}
159+
160+-- XXX We can deserialize each field to Either, so if there is a
161+-- deserialization error in any field it can handled independently. Also, a
162+-- unique type/version identifier of the field (based on the versions of the
163+-- packages, full module name space + type identifier) can be serialized along
164+-- with the value for stricter compatibility, semantics checking. Or we can
165+-- store a type hash.
166+
167+-- | __Experimental__
168+--
169+-- In explicit record types, use Latin-1 encoded record field names rather than
170+-- binary values to identify the record fields. Note that this option is not
171+-- applicable to sum types. Also, it does not work on a product type which is
172+-- not a record, because there are no field names to begin with.
173+--
174+-- This option enables the following behavior:
175+--
176+-- * __Reordering__: Order of the fields can be changed without affecting
177+-- serialization.
178+-- * __Addition__: If a 'Maybe' type field is added in the new version, the old
179+-- version of the data type can still be deserialized by the new version, the
180+-- field value in the older version is assumed to be 'Nothing'. If any other
181+-- type of field is added, deserialization of the older version results in an
182+-- error but only when that field is actually accessed in the deserialized
183+-- record.
184+-- * __Deletion__: If a field is deleted in the new version and it is
185+-- encountered in a previously serialized version then the field is discarded.
186+--
187+-- This option has to be the same on both encoding and decoding side.
188+--
189+-- There is a constant performance overhead proportional to the total length of
190+-- the record field names and the number of record fields.
191+--
192+-- The default is 'False'.
193+--
194+encodeRecordFields :: Bool -> SerializeConfig -> SerializeConfig
195+encodeRecordFields v cfg = cfg {cfgRecordSyntaxWithHeader = v}
196+
197+serializeConfig :: SerializeConfig
198+serializeConfig =
199+ SerializeConfig
200+ { cfgInlineSize = Nothing
201+ , cfgInlineSerialize = Just Inline
202+ , cfgInlineDeserialize = Just Inline
203+ , cfgConstructorTagAsString = False
204+ , cfgRecordSyntaxWithHeader = False
205+ }
206+
207+--------------------------------------------------------------------------------
208+-- Helpers
209+--------------------------------------------------------------------------------
210+
211+type Field = (Maybe Name, Type)
212+
213+_x :: Name
214+_x = mkName "x"
215+
216+_acc :: Name
217+_acc = mkName "acc"
218+
219+_arr :: Name
220+_arr = mkName "arr"
221+
222+_tag :: Name
223+_tag = mkName "tag"
224+
225+_initialOffset :: Name
226+_initialOffset = mkName "initialOffset"
227+
228+_endOffset :: Name
229+_endOffset = mkName "endOffset"
230+
231+_val :: Name
232+_val = mkName "val"
233+
234+mkFieldName :: Int -> Name
235+mkFieldName i = mkName ("field" ++ show i)
236+
237+makeI :: Int -> Name
238+makeI i = mkName $ "i" ++ show i
239+
240+makeN :: Int -> Name
241+makeN i = mkName $ "n" ++ show i
242+
243+makeA :: Int -> Name
244+makeA i = mkName $ "a" ++ show i
245+
246+--------------------------------------------------------------------------------
247+-- Domain specific helpers
248+--------------------------------------------------------------------------------
249+
250+openConstructor :: Name -> Int -> Q Pat
251+openConstructor cname numFields =
252+ conP cname (map varP (map mkFieldName [0 .. (numFields - 1)]))
253+
254+matchConstructor :: Name -> Int -> Q Exp -> Q Match
255+matchConstructor cname numFields exp0 =
256+ match (openConstructor cname numFields) (normalB exp0) []
257+
258+--------------------------------------------------------------------------------
259+-- Constructor types
260+--------------------------------------------------------------------------------
261+
262+data SimpleDataCon =
263+ SimpleDataCon Name [Field]
264+ deriving (Eq)
265+
266+simplifyDataCon :: DataCon -> SimpleDataCon
267+simplifyDataCon (DataCon cname _ _ fields) = SimpleDataCon cname fields
268+
269+data TypeOfType
270+ = UnitType Name -- 1 constructor and 1 field
271+ | TheType SimpleDataCon -- 1 constructor and 1+ fields
272+ | MultiType [SimpleDataCon] -- 1+ constructors
273+ deriving (Eq)
274+
275+typeOfType :: Type -> [DataCon] -> TypeOfType
276+typeOfType headTy [] =
277+ error
278+ ("Attempting to get size with no constructors (" ++
279+ (pprint headTy) ++ ")")
280+typeOfType _ [DataCon cname _ _ []] = UnitType cname
281+typeOfType _ [con@(DataCon _ _ _ _)] = TheType $ simplifyDataCon con
282+typeOfType _ cons = MultiType $ map simplifyDataCon cons
283+
284+isUnitType :: [DataCon] -> Bool
285+isUnitType [DataCon _ _ _ []] = True
286+isUnitType _ = False
287+
288+isRecordSyntax :: SimpleDataCon -> Bool
289+isRecordSyntax (SimpleDataCon _ fields) = and (isJust . fst <$> fields)
290+
291+--------------------------------------------------------------------------------
292+-- Type casting
293+--------------------------------------------------------------------------------
294+
295+int_w8 :: Int -> Word8
296+int_w8 = fromIntegral
297+
298+int_w32 :: Int -> Word32
299+int_w32 = fromIntegral
300+
301+w8_w16 :: Word8 -> Word16
302+w8_w16 = fromIntegral
303+
304+w8_w32 :: Word8 -> Word32
305+w8_w32 = fromIntegral
306+
307+w8_w64 :: Word8 -> Word64
308+w8_w64 = fromIntegral
309+
310+w8_int :: Word8 -> Int
311+w8_int = fromIntegral
312+
313+w32_int :: Word32 -> Int
314+w32_int = fromIntegral
315+
316+c2w :: Char -> Word8
317+c2w = fromIntegral . ord
318+
319+wListToString :: [Word8] -> String
320+wListToString = fmap (chr . fromIntegral)
321+
322+--------------------------------------------------------------------------------
323+-- Bit manipulation
324+--------------------------------------------------------------------------------
325+
326+shiftAdd :: Bits a => (b -> a) -> [b] -> a
327+shiftAdd conv xs =
328+ foldl' (.|.) zeroBits $
329+ map (\(j, x) -> shiftL x (j * 8)) $ zip [0 ..] $ map conv xs
330+
331+-- Note: This only works in little endian machines
332+-- TODO:
333+-- Instead of generating this via TH can't we write it directly in Haskell and
334+-- use that? Creating one comparison function for each deserialization may be
335+-- too much code and may not be necessary.
336+-- Benchmark both the implementations and check.
337+xorCmp :: [Word8] -> Name -> Name -> Q Exp
338+xorCmp tag off arr =
339+ case tagLen of
340+ x | x < 2 -> [|$(go8 0) == zeroBits|]
341+ x | x < 4 -> [|$(go16 0) == zeroBits|]
342+ x | x < 8 -> [|$(go32 0) == zeroBits|]
343+ _ -> [|$(go64 0) == zeroBits|]
344+ where
345+ tagLen = length tag
346+ go8 i | i >= tagLen = [|zeroBits|]
347+ go8 i = do
348+ let wIntegral = litIntegral i
349+ [|xor (unsafeInlineIO
350+ (Unbox.peekAt
351+ ($(varE off) + $(litIntegral i))
352+ $(varE arr)))
353+ ($(wIntegral) :: Word8) .|.
354+ $(go8 (i + 1))|]
355+ go16 i
356+ | i >= tagLen = [|zeroBits|]
357+ go16 i
358+ | tagLen - i < 2 = go16 (tagLen - 2)
359+ go16 i = do
360+ let wIntegral =
361+ litIntegral
362+ (shiftAdd w8_w16 [tag !! i, tag !! (i + 1)] :: Word16)
363+ [|xor (unsafeInlineIO
364+ (Unbox.peekAt
365+ ($(varE off) + $(litIntegral i))
366+ $(varE arr)))
367+ ($(wIntegral) :: Word16) .|.
368+ $(go16 (i + 2))|]
369+ go32 i
370+ | i >= tagLen = [|zeroBits|]
371+ go32 i
372+ | tagLen - i < 4 = go32 (tagLen - 4)
373+ go32 i = do
374+ let wIntegral =
375+ litIntegral
376+ (shiftAdd
377+ w8_w32
378+ [ tag !! i
379+ , tag !! (i + 1)
380+ , tag !! (i + 2)
381+ , tag !! (i + 3)
382+ ] :: Word32)
383+ [|xor (unsafeInlineIO
384+ (Unbox.peekAt
385+ ($(varE off) + $(litIntegral i))
386+ $(varE arr)))
387+ ($(wIntegral) :: Word32) .|.
388+ $(go32 (i + 4))|]
389+ go64 i
390+ | i >= tagLen = [|zeroBits|]
391+ go64 i
392+ | tagLen - i < 8 = go64 (tagLen - 8)
393+ go64 i = do
394+ let wIntegral =
395+ litIntegral
396+ (shiftAdd
397+ w8_w64
398+ [ tag !! i
399+ , tag !! (i + 1)
400+ , tag !! (i + 2)
401+ , tag !! (i + 3)
402+ , tag !! (i + 4)
403+ , tag !! (i + 5)
404+ , tag !! (i + 6)
405+ , tag !! (i + 7)
406+ ])
407+ [|xor (unsafeInlineIO
408+ (Unbox.peekAt
409+ ($(varE off) + $(litIntegral i))
410+ $(varE arr)))
411+ ($(wIntegral) :: Word64) .|.
412+ $(go64 (i + 8))|]
413+
414+--------------------------------------------------------------------------------
415+-- Primitive serialization
416+--------------------------------------------------------------------------------
417+
418+-- TODO:
419+-- Will this be too much of a code bloat?
420+-- Loop with the loop body unrolled?
421+-- Serialize this in batches similar to batch comparision in xorCmp?
422+serializeW8List :: Name -> Name -> [Word8] -> Q Exp
423+serializeW8List off arr w8List = do
424+ [|let $(varP (makeN 0)) = $(varE off)
425+ in $(doE (fmap makeBind [0 .. (lenW8List - 1)] ++
426+ [noBindS ([|pure $(varE (makeN lenW8List))|])]))|]
427+
428+ where
429+
430+ lenW8List = length w8List
431+ makeBind i =
432+ bindS
433+ (varP (makeN (i + 1)))
434+ [|$(varE 'serializeAt)
435+ $(varE (makeN i))
436+ $(varE arr)
437+ ($(litIntegral (w8List !! i)) :: Word8)|]
438+
439+--------------------------------------------------------------------------------
440+-- TH Helpers
441+--------------------------------------------------------------------------------
442+
443+litIntegral :: Integral a => a -> Q Exp
444+litIntegral = litE . IntegerL . fromIntegral
445+
446+litProxy :: Unbox a => Proxy a -> Q Exp
447+litProxy = litE . IntegerL . fromIntegral . Unbox.sizeOf
448+
449+--------------------------------------------------------------------------------
450+-- Error codes
451+--------------------------------------------------------------------------------
452+
453+errorUnsupported :: String -> a
454+errorUnsupported err =
455+ error
456+ $ unlines
457+ [ "Unsupported"
458+ , "==========="
459+ , "This is improper use of the library."
460+ , "This case is unsupported."
461+ , "Please contact the developer if this case is of interest."
462+ , ""
463+ , "Message"
464+ , "-------"
465+ , err
466+ ]
467+
468+errorUnimplemented :: a
469+errorUnimplemented =
470+ error
471+ $ unlines
472+ [ "Unimplemented"
473+ , "============="
474+ , "Please contact the developer if this case is of interest."
475+ ]
core/src/Streamly/Internal/Data/Serialize/TH/Common.hsnew file+70
@@ -0,0 +1,70 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+-- |
4+-- Module : Streamly.Internal.Data.Serialize.TH.Common
5+-- Copyright : (c) 2023 Composewell Technologies
6+-- License : BSD3-3-Clause
7+-- Maintainer : streamly@composewell.com
8+-- Stability : experimental
9+-- Portability : GHC
10+--
11+module Streamly.Internal.Data.Serialize.TH.Common
12+ ( mkDeserializeExprOne
13+ , mkSerializeExprFields
14+ ) where
15+
16+--------------------------------------------------------------------------------
17+-- Imports
18+--------------------------------------------------------------------------------
19+
20+import Language.Haskell.TH
21+import Streamly.Internal.Data.Serialize.TH.Bottom
22+
23+--------------------------------------------------------------------------------
24+-- Code
25+--------------------------------------------------------------------------------
26+
27+mkDeserializeExprOne :: Name -> SimpleDataCon -> Q Exp
28+mkDeserializeExprOne peeker (SimpleDataCon cname fields) =
29+ case fields of
30+ -- Only tag is serialized for unit fields, no actual value
31+ [] -> [|pure ($(varE (mkName "i0")), $(conE cname))|]
32+ _ ->
33+ doE
34+ (concat
35+ [ fmap makeBind [0 .. (numFields - 1)]
36+ , [ noBindS
37+ (appE
38+ (varE 'pure)
39+ (tupE
40+ [ varE (makeI numFields)
41+ , appsE
42+ (conE cname :
43+ (map (varE . makeA)
44+ [0 .. (numFields - 1)]))
45+ ]))
46+ ]
47+ ])
48+ where
49+ numFields = length fields
50+ makeBind i =
51+ bindS
52+ (tupP [varP (makeI (i + 1)), varP (makeA i)])
53+ [|$(varE peeker) $(varE (makeI i)) $(varE _arr) $(varE _endOffset)|]
54+
55+mkSerializeExprFields :: Name -> [Field] -> Q Exp
56+mkSerializeExprFields poker fields =
57+ case fields of
58+ -- Unit constructor, do nothing just tag is enough
59+ [] -> [|pure ($(varE (mkName "i0")))|]
60+ _ ->
61+ doE
62+ (fmap makeBind [0 .. (numFields - 1)] ++
63+ [noBindS ([|pure $(varE (makeI numFields))|])])
64+ where
65+ numFields = length fields
66+ makeBind i =
67+ bindS
68+ (varP (makeI (i + 1)))
69+ [|$(varE poker)
70+ $(varE (makeI i)) $(varE _arr) $(varE (mkFieldName i))|]
core/src/Streamly/Internal/Data/Serialize/TH/RecHeader.hsnew file+411
@@ -0,0 +1,411 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+-- |
4+-- Module : Streamly.Internal.Data.Serialize.TH.RecHeader
5+-- Copyright : (c) 2023 Composewell Technologies
6+-- License : BSD3-3-Clause
7+-- Maintainer : streamly@composewell.com
8+-- Stability : experimental
9+-- Portability : GHC
10+--
11+module Streamly.Internal.Data.Serialize.TH.RecHeader
12+ ( mkRecSerializeExpr
13+ , mkRecDeserializeExpr
14+ , mkRecSizeOfExpr
15+ , conUpdateFuncDec
16+ , mkDeserializeKeysDec
17+ ) where
18+
19+--------------------------------------------------------------------------------
20+-- Imports
21+--------------------------------------------------------------------------------
22+
23+import Control.Monad (void)
24+import Data.List (foldl')
25+import Data.Word (Word32, Word8)
26+import Data.Maybe (fromJust)
27+import Language.Haskell.TH
28+import Streamly.Internal.Data.Serialize.Type (Serialize(..))
29+import Data.Foldable (foldlM)
30+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
31+import Data.Proxy (Proxy(..))
32+
33+import qualified Streamly.Internal.Data.Unbox as Unbox
34+
35+import Streamly.Internal.Data.Serialize.TH.Bottom
36+import Streamly.Internal.Data.Serialize.TH.Common
37+
38+--------------------------------------------------------------------------------
39+-- Notes
40+--------------------------------------------------------------------------------
41+
42+-- Compatibility Algorithm
43+-- =======================
44+--
45+-- The algorithm is written without any low level implementation details. See
46+-- the code for any low level implementation details.
47+--
48+-- Serialization:
49+-- --------------
50+--
51+-- To serialize the data,
52+--
53+-- * Get the list of keys for the record as @keyList@.
54+-- * Serialize the @keyList@.
55+-- * Serialize the @fields@ one-by-one after serializing the @keyList@.
56+--
57+-- Deserialization:
58+-- ----------------
59+--
60+-- To deserialize the data to type @T@,
61+--
62+-- __Checking for type match__:
63+--
64+-- * Get the list of keys for type @T@ as @targetKeyList@.
65+-- * Get the list of keys encoded as @encodedKeyList@.
66+-- * If @targetKeyList == encodedKeyList@ see the __Type Match__ section else
67+-- see the __No Type Match__ section.
68+--
69+-- __Type Match__:
70+--
71+-- * Decode the fields one-by-one and construct the type @T@ in the end.
72+--
73+-- __No Type Match__:
74+--
75+-- * Decode the list of keys encoded into @encodedKeyList@.
76+-- * Get the list of keys for type @T@ as @targetKeyList@.
77+-- * Loop through @encodedKeyList@ and start deserializing the encoded data.
78+-- * If the key is present in @encodedKeyList@ and not in @targetKeyList@
79+-- then skip parsing the corresponding value.
80+-- * If the key is present in @targetKeyList@ and not in @encodedKeyList@
81+-- then set the value for that key as @Nothing@.
82+-- * If the key is present in both @encodedKeyList@ and in @targetKeyList@
83+-- parse the value.
84+-- * Construct @T@ after parsing all the data.
85+
86+-- Developer Notes
87+-- ===============
88+--
89+-- * Record update syntax is not robust across language extensions and common
90+-- record plugins (like record-dot-processor, large-records, etc.).
91+
92+--------------------------------------------------------------------------------
93+-- Compact lists
94+--------------------------------------------------------------------------------
95+
96+-- Like haskell list but the maximum length of the list is 255
97+newtype CompactList a =
98+ CompactList
99+ { unCompactList :: [a]
100+ }
101+
102+-- We use 'Word8' to encode the length, hence the maximim number of elements in
103+-- the list is 255.
104+instance forall a. Serialize a => Serialize (CompactList a) where
105+
106+ -- {-# INLINE addSizeTo #-}
107+ addSizeTo acc (CompactList xs) =
108+ foldl' addSizeTo (acc + (Unbox.sizeOf (Proxy :: Proxy Word8))) xs
109+
110+ -- Inlining this causes large compilation times for tests
111+ {-# INLINABLE deserializeAt #-}
112+ deserializeAt off arr sz = do
113+ (off1, len8) <- deserializeAt off arr sz :: IO (Int, Word8)
114+ let len = w8_int len8
115+ peekList f o i | i >= 3 = do
116+ -- Unfold the loop three times
117+ (o1, x1) <- deserializeAt o arr sz
118+ (o2, x2) <- deserializeAt o1 arr sz
119+ (o3, x3) <- deserializeAt o2 arr sz
120+ peekList (f . (\xs -> x1:x2:x3:xs)) o3 (i - 3)
121+ peekList f o 0 = pure (o, f [])
122+ peekList f o i = do
123+ (o1, x) <- deserializeAt o arr sz
124+ peekList (f . (x:)) o1 (i - 1)
125+ (nextOff, lst) <- peekList id off1 len
126+ pure (nextOff, CompactList lst)
127+
128+ -- Inlining this causes large compilation times for tests
129+ {-# INLINABLE serializeAt #-}
130+ serializeAt off arr (CompactList val) = do
131+ void $ serializeAt off arr (int_w8 (length val) :: Word8)
132+ let off1 = off + Unbox.sizeOf (Proxy :: Proxy Word8)
133+ let pokeList o [] = pure o
134+ pokeList o (x:xs) = do
135+ o1 <- serializeAt o arr x
136+ pokeList o1 xs
137+ pokeList off1 val
138+
139+--------------------------------------------------------------------------------
140+-- Helpers
141+--------------------------------------------------------------------------------
142+
143+fieldToNameBase :: Field -> String
144+fieldToNameBase = nameBase . fromJust . fst
145+
146+isMaybeType :: Type -> Bool
147+isMaybeType (AppT (ConT m) _) = m == ''Maybe
148+isMaybeType _ = False
149+
150+--------------------------------------------------------------------------------
151+-- Size
152+--------------------------------------------------------------------------------
153+
154+-- We add 4 here because we use 'serializeWithSize' for serializing.
155+exprGetSize :: Q Exp -> (Int, Type) -> Q Exp
156+exprGetSize acc (i, _) =
157+ [|addSizeTo $(acc) $(varE (mkFieldName i)) + 4|]
158+
159+sizeOfHeader :: SimpleDataCon -> Int
160+sizeOfHeader (SimpleDataCon _ fields) =
161+ sizeForFinalOff + sizeForHeaderLength + sizeForNumFields
162+ + sum (map ((+ sizeForFieldLen) . length . fieldToNameBase) fields)
163+
164+ where
165+
166+ sizeForFinalOff = 4
167+ sizeForHeaderLength = 4 -- Max header length is (255 * (255 + 1) + 1) and
168+ -- hence 2 bytes is enough to store it. But we still
169+ -- use 4 bytes as using 2 bytes introduces
170+ -- regression.
171+ sizeForNumFields = 1 -- At max 255 fields in the record constructor
172+ sizeForFieldLen = 1 -- At max 255 letters in the key
173+
174+mkRecSizeOfExpr :: SimpleDataCon -> Q Exp
175+mkRecSizeOfExpr con = do
176+ n_acc <- newName "acc"
177+ n_x <- newName "x"
178+ (lamE
179+ [varP n_acc, varP n_x]
180+ [|$(litIntegral hlen) +
181+ $(caseE (varE n_x) [matchCons (varE n_acc) con])|])
182+
183+ where
184+
185+ hlen = sizeOfHeader con
186+ sizeOfFields acc fields = foldl' exprGetSize acc $ zip [0 ..] fields
187+ matchCons acc (SimpleDataCon cname fields) =
188+ let expr = sizeOfFields acc (map snd fields)
189+ in matchConstructor cname (length fields) expr
190+
191+--------------------------------------------------------------------------------
192+-- Header
193+--------------------------------------------------------------------------------
194+
195+headerValue :: SimpleDataCon -> [Word8]
196+headerValue (SimpleDataCon _ fields) =
197+ int_w8 numFields : concat (fmap lengthPrependedFieldEncoding fields)
198+
199+ where
200+
201+ -- Error out if the number of fields or the length of key is >= 256. We use
202+ -- Word8 for encoding the info and hence the max value is 255.
203+ numFields =
204+ let lenFields = length fields
205+ in if lenFields <= 255
206+ then lenFields
207+ else errorUnsupported
208+ "Number of fields in the record should be <= 255."
209+ lengthPrependedFieldEncoding field =
210+ let fEnc =
211+ let fEnc_ = map c2w (fieldToNameBase field)
212+ lenFEnc = length fEnc_
213+ in if lenFEnc <= 255
214+ then fEnc_
215+ else
216+ errorUnsupported
217+ "Length of any key should be <= 255."
218+ in (int_w8 (length fEnc)) : fEnc
219+
220+--------------------------------------------------------------------------------
221+-- Peek
222+--------------------------------------------------------------------------------
223+
224+-- Encoding the size is required if we want to skip the field without knowing
225+-- its type. We encode the size as 'Word32' hence there is a 4 bytes increase
226+-- in size.
227+{-# INLINE serializeWithSize #-}
228+serializeWithSize :: Serialize a => Int -> MutByteArray -> a -> IO Int
229+serializeWithSize off arr val = do
230+ off1 <- serializeAt (off + 4) arr val
231+ Unbox.pokeAt off arr (int_w32 (off1 - off - 4) :: Word32)
232+ pure off1
233+
234+mkRecSerializeExpr :: Name -> SimpleDataCon -> Q Exp
235+mkRecSerializeExpr initialOffset (con@(SimpleDataCon cname fields)) = do
236+ afterHLen <- newName "afterHLen"
237+ -- Encoding the header length is required.
238+ -- We first compare the header length encoded and the current header
239+ -- length. Only if the header lengths match, we compare the headers.
240+ [|do $(varP afterHLen) <-
241+ serializeAt
242+ ($(varE initialOffset) + 4)
243+ $(varE _arr)
244+ ($(litIntegral hlen) :: Word32)
245+ $(varP (makeI 0)) <- $(serializeW8List afterHLen _arr hval)
246+ let $(openConstructor cname (length fields)) = $(varE _val)
247+ finalOff <- $(mkSerializeExprFields 'serializeWithSize fields)
248+ Unbox.pokeAt
249+ $(varE initialOffset)
250+ $(varE _arr)
251+ ((fromIntegral :: Int -> Word32)
252+ (finalOff - $(varE initialOffset)))
253+ pure finalOff|]
254+
255+ where
256+
257+ hval = headerValue con
258+ hlen = length hval
259+
260+--------------------------------------------------------------------------------
261+-- Poke
262+--------------------------------------------------------------------------------
263+
264+{-# INLINE deserializeWithSize #-}
265+deserializeWithSize ::
266+ Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
267+deserializeWithSize off arr endOff = deserializeAt (off + 4) arr endOff
268+
269+conUpdateFuncDec :: Name -> [Field] -> Q [Dec]
270+conUpdateFuncDec funcName fields = do
271+ prevAcc <- newName "prevAcc"
272+ curOff <- newName "curOff"
273+ endOff <- newName "endOff"
274+ arr <- newName "arr"
275+ key <- newName "key"
276+ method <-
277+ (caseE
278+ (varE key)
279+ (concat
280+ [ map (matchField arr endOff (prevAcc, curOff)) fnames
281+ , [ match
282+ wildP
283+ (normalB
284+ [|do (valOff, valLen :: Word32) <-
285+ deserializeAt
286+ $(varE curOff)
287+ $(varE arr)
288+ $(varE endOff)
289+ pure
290+ ( $(varE prevAcc)
291+ , valOff + w32_int valLen)|])
292+ []
293+ ]
294+ ]))
295+ pure
296+ [ PragmaD (InlineP funcName NoInline FunLike AllPhases)
297+ , FunD
298+ funcName
299+ [ Clause
300+ [ VarP arr
301+ , VarP endOff
302+ , TupP [VarP prevAcc, VarP curOff]
303+ , VarP key
304+ ]
305+ (NormalB method)
306+ []
307+ ]
308+ ]
309+
310+ where
311+
312+ fnames = fmap (fromJust . fst) fields
313+ matchField :: Name -> Name -> (Name, Name) -> Name -> Q Match
314+ matchField arr endOff (acc, currOff) fname = do
315+ let fnameLit = StringL (nameBase fname)
316+ match
317+ (litP fnameLit)
318+ (normalB
319+ [|do (valOff, valLen :: Word32) <-
320+ deserializeAt
321+ $(varE currOff)
322+ $(varE arr)
323+ $(varE endOff)
324+ pure
325+ ( ($(litE fnameLit), $(varE currOff)) : $(varE acc)
326+ , valOff + w32_int valLen)|])
327+ []
328+
329+mkDeserializeKeysDec :: Name -> Name -> SimpleDataCon -> Q [Dec]
330+mkDeserializeKeysDec funcName updateFunc (SimpleDataCon cname fields) = do
331+ hOff <- newName "hOff"
332+ finalOff <- newName "finalOff"
333+ arr <- newName "arr"
334+ endOff <- newName "endOff"
335+ kvEncoded <- newName "kvEncoded"
336+ finalRec <- newName "finalRec"
337+ let deserializeFieldExpr (Just name, ty) = do
338+ let nameLit = litE (StringL (nameBase name))
339+ [|case lookup $(nameLit) $(varE kvEncoded) of
340+ Nothing -> $(emptyTy name ty)
341+ Just off -> do
342+ val <- deserializeWithSize off $(varE arr) $(varE endOff)
343+ pure $ snd val|]
344+ deserializeFieldExpr _ =
345+ errorUnsupported "The datatype should use record syntax."
346+ method <-
347+ [|do (dataOff, hlist :: CompactList (CompactList Word8)) <-
348+ deserializeAt $(varE hOff) $(varE arr) $(varE endOff)
349+ let keys = wListToString . unCompactList <$> unCompactList hlist
350+ ($(varP kvEncoded), _) <-
351+ foldlM
352+ ($(varE updateFunc) $(varE arr) $(varE endOff))
353+ ([], dataOff)
354+ keys
355+ $(varP finalRec) <-
356+ $(foldl
357+ (\acc i ->
358+ [|$(acc) <*>
359+ $(deserializeFieldExpr i)|])
360+ [|pure $(conE cname)|]
361+ fields)
362+ pure ($(varE finalOff), $(varE finalRec))|]
363+ pure
364+ [ PragmaD (InlineP funcName NoInline FunLike AllPhases)
365+ , FunD
366+ funcName
367+ [ Clause
368+ [ VarP hOff
369+ , VarP finalOff
370+ , VarP arr
371+ , VarP endOff
372+ ]
373+ (NormalB method)
374+ []
375+ ]
376+ ]
377+
378+ where
379+
380+ emptyTy k ty =
381+ if isMaybeType ty
382+ then [|pure Nothing|]
383+ else [|error $(litE (StringL (nameBase k ++ " is not found.")))|]
384+
385+
386+mkRecDeserializeExpr :: Name -> Name -> Name -> SimpleDataCon -> Q Exp
387+mkRecDeserializeExpr initialOff endOff deserializeWithKeys con = do
388+ hOff <- newName "hOff"
389+ let sizeForFinalOff = 4 -- Word32
390+ sizeForHeaderLength = 4 -- Word32
391+ sizePreData = sizeForFinalOff + sizeForHeaderLength + hlen
392+ [|do (hlenOff, encLen :: Word32) <-
393+ deserializeAt $(varE initialOff) $(varE _arr) $(varE endOff)
394+ ($(varP hOff), hlen1 :: Word32) <-
395+ deserializeAt hlenOff $(varE _arr) $(varE endOff)
396+ if (hlen1 == $(litIntegral hlen)) && $(xorCmp hval hOff _arr)
397+ then do
398+ let $(varP (makeI 0)) =
399+ $(varE initialOff) +
400+ $(litIntegral sizePreData)
401+ $(mkDeserializeExprOne 'deserializeWithSize con)
402+ else $(varE deserializeWithKeys)
403+ $(varE hOff)
404+ ($(varE initialOff) + w32_int encLen)
405+ $(varE _arr)
406+ $(varE endOff)|]
407+
408+ where
409+
410+ hval = headerValue con
411+ hlen = length hval
core/src/Streamly/Internal/Data/Serialize/Type.hsnew file+317
@@ -0,0 +1,317 @@
1+-- |
2+-- Module : Streamly.Internal.Data.Serialize.Type
3+-- Copyright : (c) 2023 Composewell Technologies
4+-- License : BSD3-3-Clause
5+-- Maintainer : streamly@composewell.com
6+-- Portability : GHC
7+--
8+
9+module Streamly.Internal.Data.Serialize.Type
10+ (
11+ Serialize(..)
12+ ) where
13+
14+--------------------------------------------------------------------------------
15+-- Imports
16+--------------------------------------------------------------------------------
17+
18+import Control.Monad (when)
19+import Data.List (foldl')
20+import Data.Proxy (Proxy (..))
21+import Streamly.Internal.Data.Unbox (Unbox)
22+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray(..))
23+import Streamly.Internal.Data.Array.Type (Array(..))
24+import GHC.Int (Int16(..), Int32(..), Int64(..), Int8(..))
25+import GHC.Word (Word16(..), Word32(..), Word64(..), Word8(..))
26+import GHC.Stable (StablePtr(..))
27+
28+import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
29+import qualified Streamly.Internal.Data.Unbox as Unbox
30+import qualified Streamly.Internal.Data.Array.Type as Array
31+import qualified Streamly.Internal.Data.MutArray as MutArray
32+
33+import GHC.Exts
34+
35+--------------------------------------------------------------------------------
36+-- Developer Note
37+--------------------------------------------------------------------------------
38+
39+-- IMPORTANT
40+-- =========
41+--
42+-- Don't ever serialize the absolute offsets in the encoding. Serialize length
43+-- instead. Absolute offsets are NOT stable.
44+--
45+-- They will only work if the start offset of the Array when encoding and
46+-- decoding is the same. This is almost never the case.
47+
48+--------------------------------------------------------------------------------
49+-- Types
50+--------------------------------------------------------------------------------
51+
52+-- | The 'Serialize' type class provides operations for serialization and
53+-- deserialization of general Haskell data types to and from their byte stream
54+-- representation.
55+--
56+-- Unlike 'Unbox', 'Serialize' uses variable length encoding, therefore, it can
57+-- serialize recursive and variable length data types like lists, or variable
58+-- length sum types where the length of the value may vary depending on a
59+-- particular constructor. For variable length data types the length is encoded
60+-- along with the data.
61+--
62+-- The 'deserializeAt' operation reads bytes from the mutable byte array and
63+-- builds a Haskell data type from these bytes, the number of bytes it reads
64+-- depends on the type and the encoded value it is reading. 'serializeAt'
65+-- operation converts a Haskell data type to its binary representation which
66+-- must consist of as many bytes as added by the @addSizeTo@ operation for that
67+-- value and then stores these bytes into the mutable byte array. The
68+-- programmer is expected to use the @addSizeTo@ operation and allocate an
69+-- array of sufficient length before calling 'serializeAt'.
70+--
71+-- IMPORTANT: The serialized data's byte ordering remains the same as the host
72+-- machine's byte order. Therefore, it can not be deserialized from host
73+-- machines with a different byte ordering.
74+--
75+-- Instances can be derived via Template Haskell, or written manually.
76+--
77+-- Here is an example, for deriving an instance of this type class using
78+-- template Haskell:
79+--
80+-- >>> :{
81+-- data Object = Object
82+-- { _obj1 :: [Int]
83+-- , _obj2 :: Int
84+-- }
85+-- :}
86+--
87+-- @
88+-- import Streamly.Data.MutByteArray (deriveSerialize)
89+-- \$(deriveSerialize [d|instance Serialize Object|])
90+-- @
91+--
92+-- See 'Streamly.Data.MutByteArray.deriveSerialize' and
93+-- 'Streamly.Data.MutByteArray.deriveSerializeWith' for more information on
94+-- deriving using Template Haskell.
95+--
96+-- Here is an example of a manual instance.
97+--
98+-- >>> import Streamly.Data.MutByteArray (Serialize(..))
99+--
100+-- >>> :{
101+-- instance Serialize Object where
102+-- addSizeTo acc obj = addSizeTo (addSizeTo acc (_obj1 obj)) (_obj2 obj)
103+-- deserializeAt i arr len = do
104+-- -- Check the array bounds before reading
105+-- (i1, x0) <- deserializeAt i arr len
106+-- (i2, x1) <- deserializeAt i1 arr len
107+-- pure (i2, Object x0 x1)
108+-- serializeAt i arr (Object x0 x1) = do
109+-- i1 <- serializeAt i arr x0
110+-- i2 <- serializeAt i1 arr x1
111+-- pure i2
112+-- :}
113+--
114+class Serialize a where
115+ -- XXX Use (a -> Sum Int) instead, remove the Size type
116+
117+ -- A left fold step to fold a generic structure to its serializable size.
118+ -- It is of the form @Int -> a -> Int@ because you can have tail-recursive
119+ -- traversal of the structures.
120+
121+ -- | @addSizeTo accum value@ returns @accum@ incremented by the size of the
122+ -- serialized representation of @value@ in bytes. Size cannot be zero. It
123+ -- should be at least 1 byte.
124+ addSizeTo :: Int -> a -> Int
125+
126+ -- We can implement the following functions without returning the `Int`
127+ -- offset but that may require traversing the Haskell structure again to get
128+ -- the size. Therefore, this is a performance optimization.
129+
130+ -- | @deserializeAt byte-offset array arrayLen@ deserializes a value from
131+ -- the given byte-offset in the array. Returns a tuple consisting of the
132+ -- next byte-offset and the deserialized value.
133+ --
134+ -- The arrayLen passed is the entire length of the input buffer. It is to
135+ -- be used to check if we would overflow the input buffer when
136+ -- deserializing.
137+ --
138+ -- Throws an exception if the operation would exceed the supplied arrayLen.
139+ deserializeAt :: Int -> MutByteArray -> Int -> IO (Int, a)
140+
141+ -- | @serializeAt byte-offset array value@ writes the serialized
142+ -- representation of the @value@ in the array at the given byte-offset.
143+ -- Returns the next byte-offset.
144+ --
145+ -- This is an unsafe operation, the programmer must ensure that the array
146+ -- has enough space available to serialize the value as determined by the
147+ -- @addSizeTo@ operation.
148+ serializeAt :: Int -> MutByteArray -> a -> IO Int
149+
150+--------------------------------------------------------------------------------
151+-- Instances
152+--------------------------------------------------------------------------------
153+
154+-- _size is the length from array start to the last accessed byte.
155+#ifdef DEBUG
156+{-# INLINE checkBounds #-}
157+checkBounds :: String -> Int -> MutByteArray -> IO ()
158+checkBounds _label _size _arr = do
159+ sz <- MBA.sizeOfMutableByteArray _arr
160+ if (_size > sz)
161+ then error
162+ $ _label
163+ ++ ": accessing array at offset = "
164+ ++ show (_size - 1)
165+ ++ " max valid offset = " ++ show (sz - 1)
166+ else return ()
167+#endif
168+
169+-- Note: Instead of passing around the size parameter, we can use
170+-- (sizeOfMutableByteArray arr) for checking the array bound, but that turns
171+-- out to be more expensive.
172+--
173+-- Another way to optimize this is to avoid the check for fixed size
174+-- structures. For fixed size structures we can do a check at the top level and
175+-- then use checkless deserialization using the Unbox type class. That will
176+-- require ConstSize and VarSize constructors in size. The programmer can
177+-- bundle all const size fields in a newtype to make serialization faster. This
178+-- can speed up the computation of size when serializing and checking size when
179+-- deserialing.
180+--
181+-- For variable size non-recursive structures a separate size validation method
182+-- could be used to validate the size before deserializing. "validate" can also
183+-- be used to collpase multiple chunks of arrays coming from network into a
184+-- single array for deserializing. But that can also be done by framing the
185+-- serialized value with a size header.
186+--
187+{-# INLINE deserializeChecked #-}
188+deserializeChecked :: forall a. Unbox a => Int -> MutByteArray -> Int -> IO (Int, a)
189+deserializeChecked off arr sz =
190+ let next = off + Unbox.sizeOf (Proxy :: Proxy a)
191+ in do
192+ -- Keep likely path in the straight branch.
193+ if (next <= sz)
194+ then Unbox.peekAt off arr >>= \val -> pure (next, val)
195+ else error
196+ $ "deserializeAt: accessing array at offset = "
197+ ++ show (next - 1)
198+ ++ " max valid offset = " ++ show (sz - 1)
199+
200+{-# INLINE serializeUnsafe #-}
201+serializeUnsafe :: forall a. Unbox a => Int -> MutByteArray -> a -> IO Int
202+serializeUnsafe off arr val =
203+ let next = off + Unbox.sizeOf (Proxy :: Proxy a)
204+ in do
205+#ifdef DEBUG
206+ checkBounds "serializeAt" next arr
207+#endif
208+ Unbox.pokeAt off arr val
209+ pure next
210+
211+#define DERIVE_SERIALIZE_FROM_UNBOX(_type) \
212+instance Serialize _type where \
213+; {-# INLINE addSizeTo #-} \
214+; addSizeTo acc _ = acc + Unbox.sizeOf (Proxy :: Proxy _type) \
215+; {-# INLINE deserializeAt #-} \
216+; deserializeAt off arr end = deserializeChecked off arr end :: IO (Int, _type) \
217+; {-# INLINE serializeAt #-} \
218+; serializeAt = \
219+ serializeUnsafe :: Int -> MutByteArray -> _type -> IO Int
220+
221+DERIVE_SERIALIZE_FROM_UNBOX(())
222+DERIVE_SERIALIZE_FROM_UNBOX(Bool)
223+DERIVE_SERIALIZE_FROM_UNBOX(Char)
224+DERIVE_SERIALIZE_FROM_UNBOX(Int8)
225+DERIVE_SERIALIZE_FROM_UNBOX(Int16)
226+DERIVE_SERIALIZE_FROM_UNBOX(Int32)
227+DERIVE_SERIALIZE_FROM_UNBOX(Int)
228+DERIVE_SERIALIZE_FROM_UNBOX(Int64)
229+DERIVE_SERIALIZE_FROM_UNBOX(Word)
230+DERIVE_SERIALIZE_FROM_UNBOX(Word8)
231+DERIVE_SERIALIZE_FROM_UNBOX(Word16)
232+DERIVE_SERIALIZE_FROM_UNBOX(Word32)
233+DERIVE_SERIALIZE_FROM_UNBOX(Word64)
234+DERIVE_SERIALIZE_FROM_UNBOX(Double)
235+DERIVE_SERIALIZE_FROM_UNBOX(Float)
236+DERIVE_SERIALIZE_FROM_UNBOX((StablePtr a))
237+DERIVE_SERIALIZE_FROM_UNBOX((Ptr a))
238+DERIVE_SERIALIZE_FROM_UNBOX((FunPtr a))
239+
240+instance forall a. Serialize a => Serialize [a] where
241+
242+ -- {-# INLINE addSizeTo #-}
243+ addSizeTo acc xs =
244+ foldl' addSizeTo (acc + (Unbox.sizeOf (Proxy :: Proxy Int))) xs
245+
246+ -- Inlining this causes large compilation times for tests
247+ {-# INLINABLE deserializeAt #-}
248+ deserializeAt off arr sz = do
249+ (off1, len64) <- deserializeAt off arr sz :: IO (Int, Int64)
250+ let len = (fromIntegral :: Int64 -> Int) len64
251+ peekList f o i | i >= 3 = do
252+ -- Unfold the loop three times
253+ (o1, x1) <- deserializeAt o arr sz
254+ (o2, x2) <- deserializeAt o1 arr sz
255+ (o3, x3) <- deserializeAt o2 arr sz
256+ peekList (f . (\xs -> x1:x2:x3:xs)) o3 (i - 3)
257+ peekList f o 0 = pure (o, f [])
258+ peekList f o i = do
259+ (o1, x) <- deserializeAt o arr sz
260+ peekList (f . (x:)) o1 (i - 1)
261+ peekList id off1 len
262+
263+ -- Inlining this causes large compilation times for tests
264+ {-# INLINABLE serializeAt #-}
265+ serializeAt off arr val = do
266+ let off1 = off + Unbox.sizeOf (Proxy :: Proxy Int64)
267+ let pokeList acc o [] =
268+ Unbox.pokeAt off arr (acc :: Int64) >> pure o
269+ pokeList acc o (x:xs) = do
270+ o1 <- serializeAt o arr x
271+ pokeList (acc + 1) o1 xs
272+ pokeList 0 off1 val
273+
274+instance
275+#ifdef DEVBUILD
276+ Unbox a =>
277+#endif
278+ Serialize (Array a) where
279+ {-# INLINE addSizeTo #-}
280+ addSizeTo i (Array {..}) = i + (arrEnd - arrStart) + 8
281+
282+ {-# INLINE deserializeAt #-}
283+ deserializeAt off arr len = do
284+ (off1, byteLen) <- deserializeAt off arr len :: IO (Int, Int)
285+ let off2 = off1 + byteLen
286+ when (off2 > len) $
287+ error
288+ $ "deserializeAt: accessing array at offset = "
289+ ++ show (off2 - 1)
290+ ++ " max valid offset = " ++ show (len - 1)
291+ -- XXX Use MutByteArray.cloneSliceUnsafe
292+ let slice = MutArray.MutArray arr off1 off2 off2
293+ newArr <- MutArray.clone slice
294+ pure (off2, Array.unsafeFreeze newArr)
295+
296+ {-# INLINE serializeAt #-}
297+ serializeAt off arr (Array {..}) = do
298+ let arrLen = arrEnd - arrStart
299+ off1 <- serializeAt off arr arrLen
300+ MBA.putSliceUnsafe arrContents arrStart arr off1 arrLen
301+ pure (off1 + arrLen)
302+
303+instance (Serialize a, Serialize b) => Serialize (a, b) where
304+
305+ {-# INLINE addSizeTo #-}
306+ addSizeTo acc (a, b) = addSizeTo (addSizeTo acc a) b
307+
308+ {-# INLINE serializeAt #-}
309+ serializeAt off arr (a, b) = do
310+ off1 <- serializeAt off arr a
311+ serializeAt off1 arr b
312+
313+ {-# INLINE deserializeAt #-}
314+ deserializeAt off arr end = do
315+ (off1, a) <- deserializeAt off arr end
316+ (off2, b) <- deserializeAt off1 arr end
317+ pure (off2, (a, b))
core/src/Streamly/Internal/Data/Stream/Container.hs+56−48
@@ -11,16 +11,17 @@
1111
1212module Streamly.Internal.Data.Stream.Container
1313 (
14− nub
14+ -- * Deduplication
15+ ordNub
1516
16− -- * Joins for unconstrained types
17− , joinLeftGeneric
18− , joinOuterGeneric
17+ -- * Joins
18+ , leftJoin
19+ , outerJoin
1920
20− -- * Joins with Ord constraint
21− , joinInner
22− , joinLeft
23− , joinOuter
21+ -- * Ord Joins
22+ , innerOrdJoin
23+ , leftOrdJoin
24+ , outerOrdJoin
2425 )
2526where
2627
@@ -47,12 +48,15 @@ import qualified Streamly.Internal.Data.Stream.Transformer as Stream
4748
4849#include "DocTestDataStream.hs"
4950
50−-- | The memory used is proportional to the number of unique elements in the
51−-- stream. If we want to limit the memory we can just use "take" to limit the
52−-- uniq elements in the stream.
53−{-# INLINE_NORMAL nub #-}
54−nub :: (Monad m, Ord a) => Stream m a -> Stream m a
55−nub (Stream step1 state1) = Stream step (Set.empty, state1)
51+-- | @nub@ specialized to 'Ord' types for better performance. Returns a
52+-- subsequence of the stream removing any duplicate elements.
53+--
54+-- The memory used is proportional to the number of unique elements in the
55+-- stream. One way to limit the memory is to use @take@ on the resulting
56+-- stream to limit the unique elements in the stream.
57+{-# INLINE_NORMAL ordNub #-}
58+ordNub :: (Monad m, Ord a) => Stream m a -> Stream m a
59+ordNub (Stream step1 state1) = Stream step (Set.empty, state1)
5660
5761 where
5862
@@ -78,8 +82,8 @@ toMap =
7882--
7983-- XXX An IntMap may be faster when the keys are Int.
8084-- XXX Use hashmap instead of map?
81−--
82−-- | Like 'joinInner' but uses a 'Map' for efficiency.
85+
86+-- | 'innerJoin' specialized to 'Ord' types for better performance.
8387--
8488-- If the input streams have duplicate keys, the behavior is undefined.
8589--
@@ -90,10 +94,10 @@ toMap =
9094-- Time: O(m + n)
9195--
9296-- /Pre-release/
93−{-# INLINE joinInner #-}
94−joinInner :: (Monad m, Ord k) =>
97+{-# INLINE innerOrdJoin #-}
98+innerOrdJoin :: (Monad m, Ord k) =>
9599 Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, b)
96−joinInner s1 s2 =
100+innerOrdJoin s1 s2 =
97101 Stream.concatEffect $ do
98102 km <- toMap s2
99103 pure $ Stream.mapMaybe (joinAB km) s1
@@ -106,32 +110,33 @@ joinInner s1 s2 =
106110 Nothing -> Nothing
107111
108112-- XXX We can do this concurrently.
113+-- XXX Check performance of StreamD vs StreamK
109114-- XXX If the second stream is sorted and passed as an Array or a seek capable
110115-- stream then we could use binary search if we have an Ord instance or
111116-- Ordering returning function. The time complexity would then become (m x log
112117-- n).
113118
114−-- XXX Check performance of StreamD vs StreamK
115−
116−-- | Like 'joinInner' but emit @(a, Just b)@, and additionally, for those @a@'s
117−-- that are not equal to any @b@ emit @(a, Nothing)@.
119+-- | Like 'innerJoin' but emits @(a, Just b)@ whenever a and b are equal, for
120+-- those @a@'s that are not equal to any @b@ emits @(a, Nothing)@.
118121--
119−-- The second stream is evaluated multiple times. If the stream is a
120−-- consume-once stream then the caller should cache it in an 'Data.Array.Array'
121−-- before calling this function. Caching may also improve performance if the
122−-- stream is expensive to evaluate.
122+-- This is a generalization of 'innerJoin' to include all elements from the
123+-- left stream and not just those which have an equal in the right stream. This
124+-- is not a commutative operation, the order of the stream arguments matters.
123125--
124−-- >>> joinRightGeneric eq = flip (Stream.joinLeftGeneric eq)
126+-- All the caveats mentioned in 'innerJoin' apply here as well. Right join is
127+-- not provided because it is just a flipped left join:
128+--
129+-- >>> rightJoin eq = flip (Stream.leftJoin eq)
125130--
126131-- Space: O(n) assuming the second stream is cached in memory.
127132--
128133-- Time: O(m x n)
129134--
130135-- /Unimplemented/
131−{-# INLINE joinLeftGeneric #-}
132−joinLeftGeneric :: Monad m =>
136+{-# INLINE leftJoin #-}
137+leftJoin :: Monad m =>
133138 (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
134−joinLeftGeneric eq s1 s2 = Stream.evalStateT (return False) $ unCross $ do
139+leftJoin eq s1 s2 = Stream.evalStateT (return False) $ unCross $ do
135140 a <- mkCross (Stream.liftInner s1)
136141 -- XXX should we use StreamD monad here?
137142 -- XXX Is there a better way to perform some action at the end of a loop
@@ -152,19 +157,17 @@ joinLeftGeneric eq s1 s2 = Stream.evalStateT (return False) $ unCross $ do
152157 else mkCross Stream.nil
153158 Nothing -> return (a, Nothing)
154159
155−-- XXX rename to joinLeftOrd?
156−
157−-- | A more efficient 'joinLeft' using a hashmap for efficiency.
160+-- | 'leftJoin' specialized to 'Ord' types for better performance.
158161--
159162-- Space: O(n)
160163--
161164-- Time: O(m + n)
162165--
163166-- /Pre-release/
164−{-# INLINE joinLeft #-}
165−joinLeft :: (Ord k, Monad m) =>
167+{-# INLINE leftOrdJoin #-}
168+leftOrdJoin :: (Ord k, Monad m) =>
166169 Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, Maybe b)
167−joinLeft s1 s2 =
170+leftOrdJoin s1 s2 =
168171 Stream.concatEffect $ do
169172 km <- toMap s2
170173 return $ fmap (joinAB km) s1
@@ -177,14 +180,19 @@ joinLeft s1 s2 =
177180 Nothing -> (k, a, Nothing)
178181
179182-- XXX We can do this concurrently.
183+-- XXX Check performance of StreamD vs StreamK cross operation.
180184
181−-- XXX Check performance of StreamD vs StreamK
182−
183−-- | Like 'joinLeft' but emits a @(Just a, Just b)@. Like 'joinLeft', for those
185+-- | Like 'leftJoin' but emits a @(Just a, Just b)@. Like 'leftJoin', for those
184186-- @a@'s that are not equal to any @b@ emit @(Just a, Nothing)@, but
185187-- additionally, for those @b@'s that are not equal to any @a@ emit @(Nothing,
186188-- Just b)@.
187189--
190+-- This is a generalization of left join to include all the elements from the
191+-- right stream as well, in other words it is a combination of left and right
192+-- joins. This is a commutative operation. The order of stream arguments can be
193+-- changed without affecting results, except for the ordering of elements in
194+-- the resulting tuple.
195+--
188196-- For space efficiency use the smaller stream as the second stream.
189197--
190198-- Space: O(n)
@@ -192,15 +200,15 @@ joinLeft s1 s2 =
192200-- Time: O(m x n)
193201--
194202-- /Pre-release/
195−{-# INLINE joinOuterGeneric #-}
196−joinOuterGeneric :: MonadIO m =>
203+{-# INLINE outerJoin #-}
204+outerJoin :: MonadIO m =>
197205 (a -> b -> Bool)
198206 -> Stream m a
199207 -> Stream m b
200208 -> Stream m (Maybe a, Maybe b)
201−joinOuterGeneric eq s1 s =
209+outerJoin eq s1 s2 =
202210 Stream.concatEffect $ do
203− inputArr <- Array.fromStream s
211+ inputArr <- Array.fromStream s2
204212 let len = Array.length inputArr
205213 foundArr <-
206214 Stream.fold
@@ -254,18 +262,18 @@ joinOuterGeneric eq s1 s =
254262-- a flag. At the end go through @Stream m b@ and find those that are not in that
255263-- hash to return (Nothing, b).
256264
257−-- | Like 'joinOuter' but uses a 'Map' for efficiency.
265+-- | 'outerJoin' specialized to 'Ord' types for better performance.
258266--
259267-- Space: O(m + n)
260268--
261269-- Time: O(m + n)
262270--
263271-- /Pre-release/
264−{-# INLINE joinOuter #-}
265−joinOuter ::
272+{-# INLINE outerOrdJoin #-}
273+outerOrdJoin ::
266274 (Ord k, MonadIO m) =>
267275 Stream m (k, a) -> Stream m (k, b) -> Stream m (k, Maybe a, Maybe b)
268−joinOuter s1 s2 =
276+outerOrdJoin s1 s2 =
269277 Stream.concatEffect $ do
270278 km1 <- kvFold s1
271279 km2 <- kvFold s2
core/src/Streamly/Internal/Data/Stream/Eliminate.hs+33−25
@@ -13,36 +13,24 @@
1313--
1414module Streamly.Internal.Data.Stream.Eliminate
1515 (
16− -- * Running a 'Fold'
17− fold
18−
19− -- -- * Running a 'Parser'
20− , parse
16+ -- * Running a Parser
17+ parse
2118 , parseD
2219 , parseBreak
2320 , parseBreakD
2421
25− -- * Stream Deconstruction
22+ -- * Deconstruction
2623 , uncons
2724
2825 -- * Right Folds
29− , foldrM
30− , foldr
31− , foldrMx
3226 , foldr1
3327
34− -- * Left Folds
35− , foldlM'
36− , foldl'
37− , foldlMx'
38− , foldlx'
39−
4028 -- * Specific Fold Functions
41− , drain
4229 , mapM_ -- Map and Fold
4330 , null
4431 , head
4532 , headElse
33+ , init
4634 , tail
4735 , last
4836 , elem
@@ -60,16 +48,9 @@ module Streamly.Internal.Data.Stream.Eliminate
6048 , the
6149
6250 -- * To containers
63− , toList
6451 , toListRev
6552
6653 -- * Multi-Stream Folds
67− -- ** Comparisons
68− -- | These should probably be expressed using zipping operations.
69− , eqBy
70− , cmpBy
71−
72− -- ** Substreams
7354 -- | These should probably be expressed using parsers.
7455 , isPrefixOf
7556 , isInfixOf
@@ -90,7 +71,7 @@ import Foreign.Storable (Storable)
9071import GHC.Exts (SpecConstrAnnotation(..))
9172import GHC.Types (SPEC(..))
9273import Streamly.Internal.Data.Parser (ParseError(..))
93−import Streamly.Internal.Data.SVar.Type (defState)
74+import Streamly.Internal.Data.SVar.Type (adaptState, defState)
9475import Streamly.Internal.Data.Unbox (Unbox)
9576
9677import Streamly.Internal.Data.Maybe.Strict (Maybe'(..))
@@ -105,7 +86,7 @@ import qualified Streamly.Internal.Data.Stream.Transform as StreamD
10586
10687import Prelude hiding
10788 ( Foldable(..), all, any, head, last, lookup, mapM, mapM_
108− , notElem, splitAt, tail, (!!))
89+ , notElem, splitAt, init, tail, (!!))
10990import Data.Foldable (length)
11091import Streamly.Internal.Data.Stream.Type
11192
@@ -391,9 +372,35 @@ head = foldrM (\x _ -> return (Just x)) (return Nothing)
391372headElse :: Monad m => a -> Stream m a -> m a
392373headElse a = foldrM (\x _ -> return x) (return a)
393374
375+{-# INLINE_NORMAL init #-}
376+init :: Monad m => Stream m a -> m (Maybe (Stream m a))
377+init stream = do
378+ r <- uncons stream
379+ case r of
380+ Nothing -> return Nothing
381+ Just (h, Stream step1 state1) ->
382+ return $ Just $ Stream step (h, state1)
383+
384+ where
385+
386+ step gst (a, s1) = do
387+ res <- step1 (adaptState gst) s1
388+ return $
389+ case res of
390+ Yield x s -> Yield a (x, s)
391+ Skip s -> Skip (a, s)
392+ Stop -> Stop
393+
394+-- | Same as:
395+--
396+-- >>> tail = fmap (fmap snd) . Stream.uncons
397+--
394398-- Does not fuse, has the same performance as the StreamK version.
399+--
395400{-# INLINE_NORMAL tail #-}
396401tail :: Monad m => Stream m a -> m (Maybe (Stream m a))
402+tail = fmap (fmap snd) . uncons
403+{-
397404tail (UnStream step state) = go SPEC state
398405 where
399406 go !_ st = do
@@ -402,6 +409,7 @@ tail (UnStream step state) = go SPEC state
402409 Yield _ s -> return (Just $ Stream step s)
403410 Skip s -> go SPEC s
404411 Stop -> return Nothing
412+-}
405413
406414-- XXX will it fuse? need custom impl?
407415{-# INLINE_NORMAL last #-}
core/src/Streamly/Internal/Data/Stream/Exception.hs+9−6
@@ -9,17 +9,20 @@
99
1010module Streamly.Internal.Data.Stream.Exception
1111 (
12− gbracket_
13− , gbracket
14− , before
15− , afterUnsafe
12+ -- * Resources
13+ before
1614 , afterIO
15+ , afterUnsafe
16+ , finallyIO
17+ , finallyUnsafe
18+ , gbracket_
19+ , gbracket
1720 , bracketUnsafe
1821 , bracketIO3
1922 , bracketIO
23+
24+ -- * Exceptions
2025 , onException
21− , finallyUnsafe
22− , finallyIO
2326 , ghandle
2427 , handle
2528 )
core/src/Streamly/Internal/Data/Stream/Generate.hs+2−12
@@ -16,20 +16,13 @@ module Streamly.Internal.Data.Stream.Generate
1616 (
1717 -- * Primitives
1818 nil
19− , nilM
2019 , cons
21− , consM
22−
23− -- * From 'Unfold'
24− , unfold
2520
2621 -- * Unfolding
2722 , unfoldr
2823 , unfoldrM
2924
3025 -- * From Values
31− , fromPure
32− , fromEffect
3326 , repeat
3427 , repeatM
3528 , replicate
@@ -95,7 +88,6 @@ module Streamly.Internal.Data.Stream.Generate
9588 -- * From Containers
9689 -- | Transform an input structure into a stream.
9790
98− , fromList
9991 , fromListM
10092 , fromFoldable
10193 , fromFoldableM
@@ -104,10 +96,6 @@ module Streamly.Internal.Data.Stream.Generate
10496 , fromPtr
10597 , fromPtrN
10698 , fromByteStr#
107−
108− -- * Conversions
109− , fromStreamK
110− , toStreamK
11199 )
112100where
113101
@@ -154,6 +142,8 @@ import Streamly.Internal.Data.Stream.Type
154142nil :: Applicative m => Stream m a
155143nil = Stream (\_ _ -> pure Stop) ()
156144
145+infixr 5 `cons`
146+
157147-- XXX implement in terms of consM?
158148-- cons x = consM (return x)
159149-- From an implementation perspective, StreamK.cons translates into a
core/src/Streamly/Internal/Data/Stream/MutChunked.hsdeleted−323
@@ -1,323 +0,0 @@
1−-- |
2−-- Module : Streamly.Internal.Data.Stream.MutChunked
3−-- Copyright : (c) 2019 Composewell Technologies
4−-- License : BSD3-3-Clause
5−-- Maintainer : streamly@composewell.com
6−-- Stability : experimental
7−-- Portability : GHC
8−--
9−-- Combinators to efficiently manipulate streams of mutable arrays.
10−--
11−module Streamly.Internal.Data.Stream.MutChunked
12− (
13− -- * Generation
14− chunksOf
15−
16− -- * Compaction
17− , packArraysChunksOf
18− , SpliceState (..)
19− , lpackArraysChunksOf
20− , compact
21− , compactLE
22− , compactEQ
23− , compactGE
24− )
25−where
26−
27−#include "inline.hs"
28−#include "ArrayMacros.h"
29−
30−import Control.Monad.IO.Class (MonadIO(..))
31−import Control.Monad (when)
32−import Data.Bifunctor (first)
33−import Data.Proxy (Proxy(..))
34−import Streamly.Internal.Data.Unbox (Unbox, sizeOf)
35−import Streamly.Internal.Data.MutArray.Type (MutArray(..))
36−import Streamly.Internal.Data.Fold.Type (Fold(..))
37−import Streamly.Internal.Data.Parser (ParseError)
38−import Streamly.Internal.Data.Stream.Type (Stream)
39−import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
40−
41−import qualified Streamly.Internal.Data.MutArray.Type as MArray
42−import qualified Streamly.Internal.Data.Fold.Type as FL
43−import qualified Streamly.Internal.Data.Parser as ParserD
44−import qualified Streamly.Internal.Data.Stream as D
45−
46−-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
47−-- @n@ elements each.
48−--
49−-- Same as the following but may be more efficient:
50−--
51−-- > chunksOf n = Stream.foldMany (MArray.writeN n)
52−--
53−-- /Pre-release/
54−{-# INLINE chunksOf #-}
55−chunksOf :: (MonadIO m, Unbox a)
56− => Int -> Stream m a -> Stream m (MutArray a)
57−chunksOf = MArray.chunksOf
58−
59−-------------------------------------------------------------------------------
60−-- Compact
61−-------------------------------------------------------------------------------
62−
63−data SpliceState s arr
64− = SpliceInitial s
65− | SpliceBuffering s arr
66− | SpliceYielding arr (SpliceState s arr)
67− | SpliceFinish
68−
69−-- XXX This can be removed once compactLEFold/compactLE are implemented.
70−--
71−-- | This mutates the first array (if it has space) to append values from the
72−-- second one. This would work for immutable arrays as well because an
73−-- immutable array never has space so a new array is allocated instead of
74−-- mutating it.
75−--
76−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
77−-- maximum specified size. Note that if a single array is bigger than the
78−-- specified size we do not split it to fit. When we coalesce multiple arrays
79−-- if the size would exceed the specified size we do not coalesce therefore the
80−-- actual array size may be less than the specified chunk size.
81−--
82−-- @since 0.7.0
83−{-# INLINE_NORMAL packArraysChunksOf #-}
84−packArraysChunksOf :: (MonadIO m, Unbox a)
85− => Int -> D.Stream m (MutArray a) -> D.Stream m (MutArray a)
86−packArraysChunksOf n (D.Stream step state) =
87− D.Stream step' (SpliceInitial state)
88−
89− where
90−
91− {-# INLINE_LATE step' #-}
92− step' gst (SpliceInitial st) = do
93− when (n <= 0) $
94− -- XXX we can pass the module string from the higher level API
95− error $ "Streamly.Internal.Data.MutArray.Type.packArraysChunksOf: the size of "
96− ++ "arrays [" ++ show n ++ "] must be a natural number"
97− r <- step gst st
98− case r of
99− D.Yield arr s -> return $
100− let len = MArray.byteLength arr
101− in if len >= n
102− then D.Skip (SpliceYielding arr (SpliceInitial s))
103− else D.Skip (SpliceBuffering s arr)
104− D.Skip s -> return $ D.Skip (SpliceInitial s)
105− D.Stop -> return D.Stop
106−
107− step' gst (SpliceBuffering st buf) = do
108− r <- step gst st
109− case r of
110− D.Yield arr s -> do
111− let len = MArray.byteLength buf + MArray.byteLength arr
112− if len > n
113− then return $
114− D.Skip (SpliceYielding buf (SpliceBuffering s arr))
115− else do
116− buf' <- if MArray.byteCapacity buf < n
117− then liftIO $ MArray.realloc n buf
118− else return buf
119− buf'' <- MArray.splice buf' arr
120− return $ D.Skip (SpliceBuffering s buf'')
121− D.Skip s -> return $ D.Skip (SpliceBuffering s buf)
122− D.Stop -> return $ D.Skip (SpliceYielding buf SpliceFinish)
123−
124− step' _ SpliceFinish = return D.Stop
125−
126− step' _ (SpliceYielding arr next) = return $ D.Yield arr next
127−
128−-- XXX Remove this once compactLEFold is implemented
129−-- lpackArraysChunksOf = Fold.many compactLEFold
130−--
131−{-# INLINE_NORMAL lpackArraysChunksOf #-}
132−lpackArraysChunksOf :: (MonadIO m, Unbox a)
133− => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
134−lpackArraysChunksOf n (Fold step1 initial1 extract1) =
135− Fold step initial extract
136−
137− where
138−
139− initial = do
140− when (n <= 0) $
141− -- XXX we can pass the module string from the higher level API
142− error $ "Streamly.Internal.Data.MutArray.Type.packArraysChunksOf: the size of "
143− ++ "arrays [" ++ show n ++ "] must be a natural number"
144−
145− r <- initial1
146− return $ first (Tuple' Nothing) r
147−
148− extract (Tuple' Nothing r1) = extract1 r1
149− extract (Tuple' (Just buf) r1) = do
150− r <- step1 r1 buf
151− case r of
152− FL.Partial rr -> extract1 rr
153− FL.Done _ -> return ()
154−
155− step (Tuple' Nothing r1) arr =
156− let len = MArray.byteLength arr
157− in if len >= n
158− then do
159− r <- step1 r1 arr
160− case r of
161− FL.Done _ -> return $ FL.Done ()
162− FL.Partial s -> do
163− extract1 s
164− res <- initial1
165− return $ first (Tuple' Nothing) res
166− else return $ FL.Partial $ Tuple' (Just arr) r1
167−
168− step (Tuple' (Just buf) r1) arr = do
169− let len = MArray.byteLength buf + MArray.byteLength arr
170− buf' <- if MArray.byteCapacity buf < len
171− then liftIO $ MArray.realloc (max n len) buf
172− else return buf
173− buf'' <- MArray.splice buf' arr
174−
175− -- XXX this is common in both the equations of step
176− if len >= n
177− then do
178− r <- step1 r1 buf''
179− case r of
180− FL.Done _ -> return $ FL.Done ()
181− FL.Partial s -> do
182− extract1 s
183− res <- initial1
184− return $ first (Tuple' Nothing) res
185− else return $ FL.Partial $ Tuple' (Just buf'') r1
186−
187−-- XXX Same as compactLE, to be removed once that is implemented.
188−--
189−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
190−-- maximum specified size in bytes.
191−--
192−-- /Internal/
193−{-# INLINE compact #-}
194−compact :: (MonadIO m, Unbox a)
195− => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
196−compact = packArraysChunksOf
197−
198−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
199−-- maximum specified size. Note that if a single array is bigger than the
200−-- specified size we do not split it to fit. When we coalesce multiple arrays
201−-- if the size would exceed the specified size we do not coalesce therefore the
202−-- actual array size may be less than the specified chunk size.
203−--
204−-- /Internal/
205−{-# INLINE_NORMAL compactLEParserD #-}
206−compactLEParserD ::
207− forall m a. (MonadIO m, Unbox a)
208− => Int -> ParserD.Parser (MutArray a) m (MutArray a)
209−compactLEParserD n = ParserD.Parser step initial extract
210−
211− where
212−
213− nBytes = n * SIZE_OF(a)
214−
215− initial =
216− return
217− $ if n <= 0
218− then error
219− $ functionPath
220− ++ ": the size of arrays ["
221− ++ show n ++ "] must be a natural number"
222− else ParserD.IPartial Nothing
223−
224− step Nothing arr =
225− return
226− $ let len = MArray.byteLength arr
227− in if len >= nBytes
228− then ParserD.Done 0 arr
229− else ParserD.Partial 0 (Just arr)
230− step (Just buf) arr =
231− let len = MArray.byteLength buf + MArray.byteLength arr
232− in if len > nBytes
233− then return $ ParserD.Done 1 buf
234− else do
235− buf1 <-
236− if MArray.byteCapacity buf < nBytes
237− then liftIO $ MArray.realloc nBytes buf
238− else return buf
239− buf2 <- MArray.splice buf1 arr
240− return $ ParserD.Partial 0 (Just buf2)
241−
242− extract Nothing = return $ ParserD.Done 0 MArray.nil
243− extract (Just buf) = return $ ParserD.Done 0 buf
244−
245− functionPath =
246− "Streamly.Internal.Data.Stream.MutChunked.compactLEParserD"
247−
248−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
249−-- minimum specified size. Note that if all the arrays in the stream together
250−-- are smaller than the specified size the resulting array will be smaller than
251−-- the specified size. When we coalesce multiple arrays if the size would exceed
252−-- the specified size we stop coalescing further.
253−--
254−-- /Internal/
255−{-# INLINE_NORMAL compactGEFold #-}
256−compactGEFold ::
257− forall m a. (MonadIO m, Unbox a)
258− => Int -> FL.Fold m (MutArray a) (MutArray a)
259−compactGEFold n = Fold step initial extract
260−
261− where
262−
263− nBytes = n * SIZE_OF(a)
264−
265− initial =
266− return
267− $ if n < 0
268− then error
269− $ functionPath
270− ++ ": the size of arrays ["
271− ++ show n ++ "] must be a natural number"
272− else FL.Partial Nothing
273−
274− step Nothing arr =
275− return
276− $ let len = MArray.byteLength arr
277− in if len >= nBytes
278− then FL.Done arr
279− else FL.Partial (Just arr)
280− step (Just buf) arr = do
281− let len = MArray.byteLength buf + MArray.byteLength arr
282− buf1 <-
283− if MArray.byteCapacity buf < len
284− then liftIO $ MArray.realloc (max len nBytes) buf
285− else return buf
286− buf2 <- MArray.splice buf1 arr
287− if len >= n
288− then return $ FL.Done buf2
289− else return $ FL.Partial (Just buf2)
290−
291− extract Nothing = return MArray.nil
292− extract (Just buf) = return buf
293−
294− functionPath =
295− "Streamly.Internal.Data.Stream.MutChunked.compactGEFold"
296−
297−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
298−-- maximum specified size in bytes.
299−--
300−-- /Internal/
301−compactLE :: (MonadIO m, Unbox a) =>
302− Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
303−compactLE n = D.parseManyD (compactLEParserD n)
304−
305−-- | Like 'compactLE' but generates arrays of exactly equal to the size
306−-- specified except for the last array in the stream which could be shorter.
307−--
308−-- /Unimplemented/
309−{-# INLINE compactEQ #-}
310−compactEQ :: -- (MonadIO m, Unbox a) =>
311− Int -> Stream m (MutArray a) -> Stream m (MutArray a)
312−compactEQ _n _xs = undefined
313− -- IsStream.fromStreamD $ D.foldMany (compactEQFold n) (IsStream.toStreamD xs)
314−
315−-- | Like 'compactLE' but generates arrays of size greater than or equal to the
316−-- specified except for the last array in the stream which could be shorter.
317−--
318−-- /Internal/
319−{-# INLINE compactGE #-}
320−compactGE ::
321− (MonadIO m, Unbox a)
322− => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
323−compactGE n = D.foldMany (compactGEFold n)
core/src/Streamly/Internal/Data/Stream/Nesting.hs+86−202
@@ -55,11 +55,6 @@ module Streamly.Internal.Data.Stream.Nesting
5555 -- yields an element. A special case of unfoldRoundRobin.
5656 , roundRobin -- interleaveFair?/ParallelFair
5757
58− -- *** Zipping
59− -- | Zip corresponding elements of two streams.
60− , zipWith
61− , zipWithM
62−
6358 -- *** Merging
6459 -- | Interleave elements from two streams based on a condition.
6560 , mergeBy
@@ -76,17 +71,10 @@ module Streamly.Internal.Data.Stream.Nesting
7671 -- unfoldMany: Unfold m a b -> t m a -> t m b
7772 -- @
7873
79− -- *** ConcatMap
80− -- | Generate streams by mapping a stream generator on each element of an
81− -- input stream, append the resulting streams and flatten.
82− , concatMap
83− , concatMapM
84−
8574 -- *** ConcatUnfold
8675 -- | Generate streams by using an unfold on each element of an input
8776 -- stream, append the resulting streams and flatten. A special case of
8877 -- gintercalate.
89− , unfoldMany
9078 , ConcatUnfoldInterleaveState (..)
9179 , unfoldInterleave
9280 , unfoldRoundRobin
@@ -118,11 +106,8 @@ module Streamly.Internal.Data.Stream.Nesting
118106
119107 -- ** Folding
120108 -- | Apply folds on a stream.
121− , foldMany
122− , refoldMany
123109 , foldSequence
124110 , foldIterateM
125− , refoldIterateM
126111
127112 -- ** Parsing
128113 -- | Parsing is opposite to flattening. 'parseMany' is dual to concatMap or
@@ -139,7 +124,6 @@ module Streamly.Internal.Data.Stream.Nesting
139124
140125 -- ** Grouping
141126 -- | Group segments of a stream and fold. Special case of parsing.
142− , groupsOf
143127 , groupsBy
144128 , groupsWhile
145129 , groupsRollingBy
@@ -147,9 +131,8 @@ module Streamly.Internal.Data.Stream.Nesting
147131 -- ** Splitting
148132 -- | A special case of parsing.
149133 , wordsBy
150− , splitOnSeq
151− , splitOnSuffixSeq
152− , sliceOnSuffix
134+ , splitOnSeq -- XXX splitOnSeg
135+ , splitOnSuffixSeq -- XXX splitOnSegSuffix, splitOnTrailer
153136
154137 -- XXX Implement these as folds or parsers instead.
155138 , splitOnSuffixSeqAny
@@ -158,9 +141,8 @@ module Streamly.Internal.Data.Stream.Nesting
158141
159142 -- * Transform (Nested Containers)
160143 -- | Opposite to compact in ArrayStream
161− , splitInnerBy
162− , splitInnerBySuffix
163− , intersectBySorted
144+ , splitInnerBy -- XXX innerSplitOn
145+ , splitInnerBySuffix -- XXX innerSplitOnSuffix
164146
165147 -- * Reduce By Streams
166148 , dropPrefix
@@ -182,12 +164,9 @@ import Fusion.Plugin.Types (Fuse(..))
182164import GHC.Types (SPEC(..))
183165
184166import Streamly.Internal.Data.Array.Type (Array(..))
185−import Streamly.Internal.Data.Fold.Step (Step(..))
186167import Streamly.Internal.Data.Fold.Type (Fold(..))
187168import Streamly.Internal.Data.Parser (ParseError(..))
188−import Streamly.Internal.Data.Refold.Type (Refold(..))
189169import Streamly.Internal.Data.SVar.Type (adaptState)
190−import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
191170import Streamly.Internal.Data.Unbox (Unbox, sizeOf)
192171import Streamly.Internal.Data.Unfold.Type (Unfold(..))
193172
@@ -613,49 +592,6 @@ mergeFstBy :: -- Monad m =>
613592mergeFstBy _f _m1 _m2 = undefined
614593 -- fromStreamK $ D.mergeFstBy f (toStreamD m1) (toStreamD m2)
615594
616−-------------------------------------------------------------------------------
617−-- Intersection of sorted streams
618−-------------------------------------------------------------------------------
619−
620−-- Assuming the streams are sorted in ascending order
621−{-# INLINE_NORMAL intersectBySorted #-}
622−intersectBySorted :: Monad m
623− => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
624−intersectBySorted cmp (Stream stepa ta) (Stream stepb tb) =
625− Stream step
626− ( ta -- left stream state
627− , tb -- right stream state
628− , Nothing -- left value
629− , Nothing -- right value
630− )
631−
632− where
633−
634− {-# INLINE_LATE step #-}
635− -- step 1, fetch the first value
636− step gst (sa, sb, Nothing, b) = do
637− r <- stepa gst sa
638− return $ case r of
639− Yield a sa' -> Skip (sa', sb, Just a, b) -- step 2/3
640− Skip sa' -> Skip (sa', sb, Nothing, b)
641− Stop -> Stop
642−
643− -- step 2, fetch the second value
644− step gst (sa, sb, a@(Just _), Nothing) = do
645− r <- stepb gst sb
646− return $ case r of
647− Yield b sb' -> Skip (sa, sb', a, Just b) -- step 3
648− Skip sb' -> Skip (sa, sb', a, Nothing)
649− Stop -> Stop
650−
651− -- step 3, compare the two values
652− step _ (sa, sb, Just a, Just b) = do
653− let res = cmp a b
654− return $ case res of
655− GT -> Skip (sa, sb, Just a, Nothing) -- step 2
656− LT -> Skip (sa, sb, Nothing, Just b) -- step 1
657− EQ -> Yield a (sa, sb, Nothing, Just b) -- step 1
658−
659595------------------------------------------------------------------------------
660596-- Combine N Streams - unfoldMany
661597------------------------------------------------------------------------------
@@ -1242,6 +1178,7 @@ foldSequence _f _m = undefined
12421178data FIterState s f m a b
12431179 = FIterInit s f
12441180 | forall fs. FIterStream s (fs -> a -> m (FL.Step fs b)) fs (fs -> m b)
1181+ (fs -> m b)
12451182 | FIterYield b (FIterState s f m a b)
12461183 | FIterStop
12471184
@@ -1269,101 +1206,30 @@ foldIterateM func seed0 (Stream step state) =
12691206 where
12701207
12711208 {-# INLINE iterStep #-}
1272− iterStep from st fstep extract = do
1209+ iterStep from st fstep extract final = do
12731210 res <- from
12741211 return
12751212 $ Skip
12761213 $ case res of
1277− FL.Partial fs -> FIterStream st fstep fs extract
1214+ FL.Partial fs -> FIterStream st fstep fs extract final
12781215 FL.Done fb -> FIterYield fb $ FIterInit st (return fb)
12791216
12801217 {-# INLINE_LATE stepOuter #-}
12811218 stepOuter _ (FIterInit st seed) = do
1282− (FL.Fold fstep initial extract) <- seed >>= func
1283− iterStep initial st fstep extract
1284− stepOuter gst (FIterStream st fstep fs extract) = do
1219+ (FL.Fold fstep initial extract final) <- seed >>= func
1220+ iterStep initial st fstep extract final
1221+ stepOuter gst (FIterStream st fstep fs extract final) = do
12851222 r <- step (adaptState gst) st
12861223 case r of
12871224 Yield x s -> do
1288− iterStep (fstep fs x) s fstep extract
1289− Skip s -> return $ Skip $ FIterStream s fstep fs extract
1225+ iterStep (fstep fs x) s fstep extract final
1226+ Skip s -> return $ Skip $ FIterStream s fstep fs extract final
12901227 Stop -> do
1291− b <- extract fs
1228+ b <- final fs
12921229 return $ Skip $ FIterYield b FIterStop
12931230 stepOuter _ (FIterYield a next) = return $ Yield a next
12941231 stepOuter _ FIterStop = return Stop
12951232
1296−{-# ANN type CIterState Fuse #-}
1297−data CIterState s f fs b
1298− = CIterInit s f
1299− | CIterConsume s fs
1300− | CIterYield b (CIterState s f fs b)
1301− | CIterStop
1302−
1303−-- | Like 'foldIterateM' but using the 'Refold' type instead. This could be
1304−-- much more efficient due to stream fusion.
1305−--
1306−-- /Internal/
1307−{-# INLINE_NORMAL refoldIterateM #-}
1308−refoldIterateM ::
1309− Monad m => Refold m b a b -> m b -> Stream m a -> Stream m b
1310−refoldIterateM (Refold fstep finject fextract) initial (Stream step state) =
1311− Stream stepOuter (CIterInit state initial)
1312−
1313− where
1314−
1315− {-# INLINE iterStep #-}
1316− iterStep st action = do
1317− res <- action
1318− return
1319− $ Skip
1320− $ case res of
1321− FL.Partial fs -> CIterConsume st fs
1322− FL.Done fb -> CIterYield fb $ CIterInit st (return fb)
1323−
1324− {-# INLINE_LATE stepOuter #-}
1325− stepOuter _ (CIterInit st action) = do
1326− iterStep st (action >>= finject)
1327− stepOuter gst (CIterConsume st fs) = do
1328− r <- step (adaptState gst) st
1329− case r of
1330− Yield x s -> iterStep s (fstep fs x)
1331− Skip s -> return $ Skip $ CIterConsume s fs
1332− Stop -> do
1333− b <- fextract fs
1334− return $ Skip $ CIterYield b CIterStop
1335− stepOuter _ (CIterYield a next) = return $ Yield a next
1336− stepOuter _ CIterStop = return Stop
1337−
1338−-- "n" elements at the end are dropped by the fold.
1339−{-# INLINE sliceBy #-}
1340−sliceBy :: Monad m => Fold m a Int -> Int -> Refold m (Int, Int) a (Int, Int)
1341−sliceBy (Fold step1 initial1 extract1) n = Refold step inject extract
1342−
1343− where
1344−
1345− inject (i, len) = do
1346− r <- initial1
1347− return $ case r of
1348− Partial s -> Partial $ Tuple' (i + len + n) s
1349− Done l -> Done (i, l)
1350−
1351− step (Tuple' i s) x = do
1352− r <- step1 s x
1353− return $ case r of
1354− Partial s1 -> Partial $ Tuple' i s1
1355− Done len -> Done (i, len)
1356−
1357− extract (Tuple' i s) = (i,) <$> extract1 s
1358−
1359−{-# INLINE sliceOnSuffix #-}
1360−sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
1361−sliceOnSuffix predicate =
1362− -- Scan the stream with the given refold
1363− refoldIterateM
1364− (sliceBy (FL.takeEndBy_ predicate FL.length) 1)
1365− (return (-1, 0))
1366−
13671233------------------------------------------------------------------------------
13681234-- Parsing
13691235------------------------------------------------------------------------------
@@ -1916,7 +1782,7 @@ groupsWhile :: Monad m
19161782{-
19171783groupsWhile eq fld = parseMany (PRD.groupBy eq fld)
19181784-}
1919−groupsWhile cmp (Fold fstep initial done) (Stream step state) =
1785+groupsWhile cmp (Fold fstep initial _ final) (Stream step state) =
19201786 Stream stepOuter (GroupingInit state)
19211787
19221788 where
@@ -1939,7 +1805,7 @@ groupsWhile cmp (Fold fstep initial done) (Stream step state) =
19391805 FL.Partial fs1 -> go SPEC x s fs1
19401806 FL.Done b -> return $ Yield b (GroupingInit s)
19411807 Skip s -> return $ Skip $ GroupingDo s fs
1942− Stop -> return Stop
1808+ Stop -> final fs >> return Stop
19431809
19441810 where
19451811
@@ -1954,10 +1820,12 @@ groupsWhile cmp (Fold fstep initial done) (Stream step state) =
19541820 FL.Partial fs1 -> go SPEC prev s fs1
19551821 FL.Done b -> return $ Yield b (GroupingInit s)
19561822 else do
1957− r <- done acc
1823+ r <- final acc
19581824 return $ Yield r (GroupingInitWith s x)
19591825 Skip s -> go SPEC prev s acc
1960− Stop -> done acc >>= \r -> return $ Yield r GroupingDone
1826+ Stop -> do
1827+ r <- final acc
1828+ return $ Yield r GroupingDone
19611829 stepOuter _ (GroupingInitWith st x) = do
19621830 res <- initial
19631831 return
@@ -1984,10 +1852,12 @@ groupsWhile cmp (Fold fstep initial done) (Stream step state) =
19841852 FL.Partial fs1 -> go SPEC s fs1
19851853 FL.Done b -> return $ Yield b (GroupingInit s)
19861854 else do
1987− r <- done acc
1855+ r <- final acc
19881856 return $ Yield r (GroupingInitWith s x)
19891857 Skip s -> go SPEC s acc
1990− Stop -> done acc >>= \r -> return $ Yield r GroupingDone
1858+ Stop -> do
1859+ r <- final acc
1860+ return $ Yield r GroupingDone
19911861 stepOuter _ (GroupingYield _ _) = error "groupsWhile: Unreachable"
19921862 stepOuter _ GroupingDone = return Stop
19931863
@@ -2009,7 +1879,7 @@ groupsRollingBy :: Monad m
20091879{-
20101880groupsRollingBy eq fld = parseMany (PRD.groupByRolling eq fld)
20111881-}
2012−groupsRollingBy cmp (Fold fstep initial done) (Stream step state) =
1882+groupsRollingBy cmp (Fold fstep initial _ final) (Stream step state) =
20131883 Stream stepOuter (GroupingInit state)
20141884
20151885 where
@@ -2032,7 +1902,7 @@ groupsRollingBy cmp (Fold fstep initial done) (Stream step state) =
20321902 FL.Partial fs1 -> go SPEC x s fs1
20331903 FL.Done fb -> return $ Yield fb (GroupingInit s)
20341904 Skip s -> return $ Skip $ GroupingDo s fs
2035− Stop -> return Stop
1905+ Stop -> final fs >> return Stop
20361906
20371907 where
20381908
@@ -2047,10 +1917,12 @@ groupsRollingBy cmp (Fold fstep initial done) (Stream step state) =
20471917 FL.Partial fs1 -> go SPEC x s fs1
20481918 FL.Done b -> return $ Yield b (GroupingInit s)
20491919 else do
2050− r <- done acc
1920+ r <- final acc
20511921 return $ Yield r (GroupingInitWith s x)
20521922 Skip s -> go SPEC prev s acc
2053− Stop -> done acc >>= \r -> return $ Yield r GroupingDone
1923+ Stop -> do
1924+ r <- final acc
1925+ return $ Yield r GroupingDone
20541926 stepOuter _ (GroupingInitWith st x) = do
20551927 res <- initial
20561928 return
@@ -2081,7 +1953,7 @@ groupsRollingBy cmp (Fold fstep initial done) (Stream step state) =
20811953 FL.Done b -> return $ Yield b (GroupingInit st)
20821954 else do
20831955 {-
2084− r <- done acc
1956+ r <- final acc
20851957 return $ Yield r (GroupingInitWith s x)
20861958 -}
20871959 -- The code above does not let groupBy fuse. We use the
@@ -2090,14 +1962,16 @@ groupsRollingBy cmp (Fold fstep initial done) (Stream step state) =
20901962 -- GroupingInitWith state here to help GHC with stream
20911963 -- fusion.
20921964 result <- initial
2093− r <- done acc
1965+ r <- final acc
20941966 return
20951967 $ Yield r
20961968 $ case result of
20971969 FL.Partial fsi -> GroupingDoWith s fsi x
20981970 FL.Done b -> GroupingYield b (GroupingInit s)
20991971 Skip s -> go SPEC prev s acc
2100− Stop -> done acc >>= \r -> return $ Yield r GroupingDone
1972+ Stop -> do
1973+ r <- final acc
1974+ return $ Yield r GroupingDone
21011975 stepOuter _ (GroupingYield r next) = return $ Yield r next
21021976 stepOuter _ GroupingDone = return Stop
21031977
@@ -2119,7 +1993,7 @@ data WordsByState st fs b
21191993
21201994{-# INLINE_NORMAL wordsBy #-}
21211995wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
2122−wordsBy predicate (Fold fstep initial done) (Stream step state) =
1996+wordsBy predicate (Fold fstep initial _ final) (Stream step state) =
21231997 Stream stepOuter (WordsByInit state)
21241998
21251999 where
@@ -2149,7 +2023,7 @@ wordsBy predicate (Fold fstep initial done) (Stream step state) =
21492023 FL.Partial fs1 -> go SPEC s fs1
21502024 FL.Done b -> return $ Yield b (WordsByInit s)
21512025 Skip s -> return $ Skip $ WordsByDo s fs
2152− Stop -> return Stop
2026+ Stop -> final fs >> return Stop
21532027
21542028 where
21552029
@@ -2160,7 +2034,7 @@ wordsBy predicate (Fold fstep initial done) (Stream step state) =
21602034 if predicate x
21612035 then do
21622036 {-
2163− r <- done acc
2037+ r <- final acc
21642038 return $ Yield r (WordsByInit s)
21652039 -}
21662040 -- The above code does not fuse well. Need to check why
@@ -2169,7 +2043,7 @@ wordsBy predicate (Fold fstep initial done) (Stream step state) =
21692043 -- state always, we directly go to WordsByDo state in
21702044 -- the common case of Partial.
21712045 resi <- initial
2172− r <- done acc
2046+ r <- final acc
21732047 return
21742048 $ Yield r
21752049 $ case resi of
@@ -2181,7 +2055,9 @@ wordsBy predicate (Fold fstep initial done) (Stream step state) =
21812055 FL.Partial fs1 -> go SPEC s fs1
21822056 FL.Done b -> return $ Yield b (WordsByInit s)
21832057 Skip s -> go SPEC s acc
2184− Stop -> done acc >>= \r -> return $ Yield r WordsByDone
2058+ Stop -> do
2059+ r <- final acc
2060+ return $ Yield r WordsByDone
21852061
21862062 stepOuter _ WordsByDone = return Stop
21872063
@@ -2239,7 +2115,7 @@ splitOnSeq
22392115 -> Fold m a b
22402116 -> Stream m a
22412117 -> Stream m b
2242−splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
2118+splitOnSeq patArr (Fold fstep initial _ final) (Stream step state) =
22432119 Stream stepOuter SplitOnSeqInit
22442120
22452121 where
@@ -2322,12 +2198,12 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
23222198 r <- fstep acc x
23232199 b1 <-
23242200 case r of
2325− FL.Partial acc1 -> done acc1
2201+ FL.Partial acc1 -> final acc1
23262202 FL.Done b -> return b
23272203 let jump c = SplitOnSeqEmpty c s
23282204 in yieldProceed jump b1
23292205 Skip s -> skip (SplitOnSeqEmpty acc s)
2330− Stop -> return Stop
2206+ Stop -> final acc >> return Stop
23312207
23322208 -----------------
23332209 -- Done
@@ -2345,7 +2221,7 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
23452221 Yield x s -> do
23462222 let jump c = SplitOnSeqSingle c s pat
23472223 if pat == x
2348− then done fs >>= yieldProceed jump
2224+ then final fs >>= yieldProceed jump
23492225 else do
23502226 r <- fstep fs x
23512227 case r of
@@ -2353,7 +2229,7 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
23532229 FL.Done b -> yieldProceed jump b
23542230 Skip s -> return $ Skip $ SplitOnSeqSingle fs s pat
23552231 Stop -> do
2356− r <- done fs
2232+ r <- final fs
23572233 return $ Skip $ SplitOnSeqYield r SplitOnSeqDone
23582234
23592235 ---------------------------
@@ -2361,7 +2237,7 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
23612237 ---------------------------
23622238
23632239 stepOuter _ (SplitOnSeqWordDone 0 fs _) = do
2364− r <- done fs
2240+ r <- final fs
23652241 skip $ SplitOnSeqYield r SplitOnSeqDone
23662242 stepOuter _ (SplitOnSeqWordDone n fs wrd) = do
23672243 let old = elemMask .&. (wrd `shiftR` (elemBits * (n - 1)))
@@ -2388,7 +2264,7 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
23882264 if wrd1 .&. wordMask == wordPat
23892265 then do
23902266 let jump c = SplitOnSeqWordInit c s
2391− done fs >>= yieldProceed jump
2267+ final fs >>= yieldProceed jump
23922268 else skip $ SplitOnSeqWordLoop wrd1 s fs
23932269 else go SPEC (idx + 1) wrd1 s
23942270 Skip s -> go SPEC idx wrd s
@@ -2396,7 +2272,7 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
23962272 if idx /= 0
23972273 then skip $ SplitOnSeqWordDone idx fs wrd
23982274 else do
2399− r <- done fs
2275+ r <- final fs
24002276 skip $ SplitOnSeqYield r SplitOnSeqDone
24012277
24022278 stepOuter gst (SplitOnSeqWordLoop wrd0 st0 fs0) =
@@ -2417,7 +2293,7 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
24172293 case r of
24182294 FL.Partial fs1 -> do
24192295 if wrd1 .&. wordMask == wordPat
2420− then done fs1 >>= yieldProceed jump
2296+ then final fs1 >>= yieldProceed jump
24212297 else go SPEC wrd1 s fs1
24222298 FL.Done b -> yieldProceed jump b
24232299 Skip s -> go SPEC wrd s fs
@@ -2502,14 +2378,14 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
25022378 stepOuter _ (SplitOnSeqKRCheck fs st rb rh) = do
25032379 if RB.unsafeEqArray rb rh patArr
25042380 then do
2505− r <- done fs
2381+ r <- final fs
25062382 let rst = RB.startOf rb
25072383 jump c = SplitOnSeqKRInit 0 c st rb rst
25082384 yieldProceed jump r
25092385 else skip $ SplitOnSeqKRLoop fs st rb rh patHash
25102386
25112387 stepOuter _ (SplitOnSeqKRDone 0 fs _ _) = do
2512− r <- done fs
2388+ r <- final fs
25132389 skip $ SplitOnSeqYield r SplitOnSeqDone
25142390 stepOuter _ (SplitOnSeqKRDone n fs rb rh) = do
25152391 old <- liftIO $ peek rh
@@ -2553,7 +2429,7 @@ splitOnSuffixSeq
25532429 -> Fold m a b
25542430 -> Stream m a
25552431 -> Stream m b
2556−splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
2432+splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state) =
25572433 Stream stepOuter SplitOnSuffixSeqInit
25582434
25592435 where
@@ -2593,7 +2469,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
25932469 r <- if withSep then fstep fs x else return $ FL.Partial fs
25942470 b1 <-
25952471 case r of
2596− FL.Partial fs1 -> done fs1
2472+ FL.Partial fs1 -> final fs1
25972473 FL.Done b -> return b
25982474 yieldProceed jump b1
25992475 else do
@@ -2656,11 +2532,11 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
26562532 r <- fstep acc x
26572533 b1 <-
26582534 case r of
2659− FL.Partial fs -> done fs
2535+ FL.Partial fs -> final fs
26602536 FL.Done b -> return b
26612537 yieldProceed jump b1
26622538 Skip s -> skip (SplitOnSuffixSeqEmpty acc s)
2663− Stop -> return Stop
2539+ Stop -> final acc >> return Stop
26642540
26652541 -----------------
26662542 -- Done
@@ -2677,7 +2553,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
26772553 case res of
26782554 Yield x s -> processYieldSingle pat x s fs
26792555 Skip s -> skip $ SplitOnSuffixSeqSingleInit fs s pat
2680− Stop -> return Stop
2556+ Stop -> final fs >> return Stop
26812557
26822558 stepOuter gst (SplitOnSuffixSeqSingle fs st pat) = do
26832559 res <- step (adaptState gst) st
@@ -2685,7 +2561,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
26852561 Yield x s -> processYieldSingle pat x s fs
26862562 Skip s -> skip $ SplitOnSuffixSeqSingle fs s pat
26872563 Stop -> do
2688− r <- done fs
2564+ r <- final fs
26892565 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
26902566
26912567 ---------------------------
@@ -2693,7 +2569,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
26932569 ---------------------------
26942570
26952571 stepOuter _ (SplitOnSuffixSeqWordDone 0 fs _) = do
2696− r <- done fs
2572+ r <- final fs
26972573 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
26982574 stepOuter _ (SplitOnSuffixSeqWordDone n fs wrd) = do
26992575 let old = elemMask .&. (wrd `shiftR` (elemBits * (n - 1)))
@@ -2716,7 +2592,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
27162592 let jump c = SplitOnSuffixSeqWordInit c s
27172593 yieldProceed jump b
27182594 Skip s -> skip (SplitOnSuffixSeqWordInit fs0 s)
2719− Stop -> return Stop
2595+ Stop -> final fs0 >> return Stop
27202596
27212597 where
27222598
@@ -2734,7 +2610,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
27342610 then go SPEC (idx + 1) wrd1 s fs1
27352611 else if wrd1 .&. wordMask /= wordPat
27362612 then skip $ SplitOnSuffixSeqWordLoop wrd1 s fs1
2737− else do done fs >>= yieldProceed jump
2613+ else do final fs >>= yieldProceed jump
27382614 FL.Done b -> yieldProceed jump b
27392615 Skip s -> go SPEC idx wrd s fs
27402616 Stop -> skip $ SplitOnSuffixSeqWordDone idx fs wrd
@@ -2760,16 +2636,16 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
27602636 case r of
27612637 FL.Partial fs1 ->
27622638 if wrd1 .&. wordMask == wordPat
2763− then done fs1 >>= yieldProceed jump
2639+ then final fs1 >>= yieldProceed jump
27642640 else go SPEC wrd1 s fs1
27652641 FL.Done b -> yieldProceed jump b
27662642 Skip s -> go SPEC wrd s fs
27672643 Stop ->
27682644 if wrd .&. wordMask == wordPat
2769− then return Stop
2645+ then final fs >> return Stop
27702646 else if withSep
27712647 then do
2772− r <- done fs
2648+ r <- final fs
27732649 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
27742650 else skip $ SplitOnSuffixSeqWordDone patLen fs wrd
27752651
@@ -2791,7 +2667,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
27912667 jump c = SplitOnSuffixSeqKRInit 0 c s rb rst
27922668 yieldProceed jump b
27932669 Skip s -> skip $ SplitOnSuffixSeqKRInit idx0 fs s rb rh0
2794− Stop -> return Stop
2670+ Stop -> final fs >> return Stop
27952671
27962672 stepOuter gst (SplitOnSuffixSeqKRInit1 fs0 st0 rb rh0) = do
27972673 go SPEC 1 rh0 st0 fs0
@@ -2823,10 +2699,10 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
28232699 Stop -> do
28242700 -- do not issue a blank segment when we end at pattern
28252701 if (idx == maxIndex) && RB.unsafeEqArray rb rh patArr
2826− then return Stop
2702+ then final fs >> return Stop
28272703 else if withSep
28282704 then do
2829− r <- done fs
2705+ r <- final fs
28302706 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
28312707 else skip $ SplitOnSuffixSeqKRDone idx fs rb (RB.startOf rb)
28322708
@@ -2855,24 +2731,24 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
28552731 Skip s -> go SPEC fs s rh cksum
28562732 Stop ->
28572733 if RB.unsafeEqArray rb rh patArr
2858− then return Stop
2734+ then final fs >> return Stop
28592735 else if withSep
28602736 then do
2861− r <- done fs
2737+ r <- final fs
28622738 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
28632739 else skip $ SplitOnSuffixSeqKRDone patLen fs rb rh
28642740
28652741 stepOuter _ (SplitOnSuffixSeqKRCheck fs st rb rh) = do
28662742 if RB.unsafeEqArray rb rh patArr
28672743 then do
2868− r <- done fs
2744+ r <- final fs
28692745 let rst = RB.startOf rb
28702746 jump c = SplitOnSuffixSeqKRInit 0 c st rb rst
28712747 yieldProceed jump r
28722748 else skip $ SplitOnSuffixSeqKRLoop fs st rb rh patHash
28732749
28742750 stepOuter _ (SplitOnSuffixSeqKRDone 0 fs _ _) = do
2875− r <- done fs
2751+ r <- final fs
28762752 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
28772753 stepOuter _ (SplitOnSuffixSeqKRDone n fs rb rh) = do
28782754 old <- liftIO $ peek rh
@@ -3054,12 +2930,13 @@ splitInnerBy splitter joiner (Stream step1 state1) =
30542930-- | Performs infix separator style splitting.
30552931{-# INLINE_NORMAL splitInnerBySuffix #-}
30562932splitInnerBySuffix
3057− :: (Monad m, Eq (f a), Monoid (f a))
3058− => (f a -> m (f a, Maybe (f a))) -- splitter
2933+ :: Monad m
2934+ => (f a -> Bool) -- isEmpty?
2935+ -> (f a -> m (f a, Maybe (f a))) -- splitter
30592936 -> (f a -> f a -> m (f a)) -- joiner
30602937 -> Stream m (f a)
30612938 -> Stream m (f a)
3062−splitInnerBySuffix splitter joiner (Stream step1 state1) =
2939+splitInnerBySuffix isEmpty splitter joiner (Stream step1 state1) =
30632940 Stream step (SplitInitial state1)
30642941
30652942 where
@@ -3086,10 +2963,11 @@ splitInnerBySuffix splitter joiner (Stream step1 state1) =
30862963 Nothing -> Skip (SplitBuffering s buf')
30872964 Just x2 -> Skip (SplitYielding buf' (SplitSplitting s x2))
30882965 Skip s -> return $ Skip (SplitBuffering s buf)
3089− Stop -> return $
3090− if buf == mempty
3091− then Stop
3092− else Skip (SplitYielding buf SplitFinishing)
2966+ Stop ->
2967+ return $
2968+ if isEmpty buf
2969+ then Stop
2970+ else Skip (SplitYielding buf SplitFinishing)
30932971
30942972 step _ (SplitSplitting st buf) = do
30952973 (x1, mx2) <- splitter buf
@@ -3108,6 +2986,8 @@ splitInnerBySuffix splitter joiner (Stream step1 state1) =
31082986--
31092987-- Space: @O(1)@
31102988--
2989+-- See also stripPrefix.
2990+--
31112991-- /Unimplemented/
31122992{-# INLINE dropPrefix #-}
31132993dropPrefix ::
@@ -3120,6 +3000,8 @@ dropPrefix = error "Not implemented yet!"
31203000--
31213001-- Space: @O(n)@ where n is the length of the infix.
31223002--
3003+-- See also stripInfix.
3004+--
31233005-- /Unimplemented/
31243006{-# INLINE dropInfix #-}
31253007dropInfix ::
@@ -3132,6 +3014,8 @@ dropInfix = error "Not implemented yet!"
31323014--
31333015-- Space: @O(n)@ where n is the length of the suffix.
31343016--
3017+-- See also stripSuffix.
3018+--
31353019-- /Unimplemented/
31363020{-# INLINE dropSuffix #-}
31373021dropSuffix ::
core/src/Streamly/Internal/Data/Stream/Top.hs+243−116
@@ -8,38 +8,73 @@
88-- Portability : GHC
99--
1010-- Top level module that can depend on all other lower level Stream modules.
11+--
12+-- Design notes:
13+--
14+-- The order of arguments in the join operations should ideally be opposite. It
15+-- should be such that the infinite stream is the last one. The transformation
16+-- should be on the last argument, so if you curry the functions with all other
17+-- arguments we get a @Stream -> Stream@ function. The first stream argument
18+-- may be considered as a config or modifier for the operation.
19+--
20+-- Benefit of changing the order is that we get a more intuitive Stream ->
21+-- Stream transformation after currying all other arguments. The inner loop
22+-- streams become arguments for the transformation, more like local modifiers
23+-- for the global outer stream as the last argument. Thus we can continue using
24+-- transformations on the outer stream in a composed pipeline. Otherwise we can
25+-- use flip to flip the order.
26+--
27+-- The fact that the inner stream can be used in the loop multiple times also
28+-- tells that this is not the real effectful stream, it is more like a pure
29+-- stream or an array. In fact we may consider using an Identity streams as
30+-- inner streams in which case these functions will not look nice.
31+--
32+-- Downsides:
33+--
34+-- * Maybe less intuitive to think about, because we usually think the first
35+-- stream as the outer loop and second as the inner.
36+-- * Zip and merge operations will continue using the opposite order.
37+-- * Need to change the order of cross, crossWith operations as well
38+-- * It will be inconsistent with Data.List. The functions cannot be used as
39+-- intuitive operators.
40+--
41+-- The choice is similar to concatMap vs bind. concatMap is pipeline
42+-- composition friendly but bind is user intuition friendly. Another option is
43+-- to have other functions with a different argument order e.g. flippedCross
44+-- instead of cross.
45+--
46+-- If we change the order we have to make sure that we have a consistent
47+-- convention for set-like and the cross join operations.
1148
1249module Streamly.Internal.Data.Stream.Top
1350 (
14− -- * Transformation
15− -- ** Sampling
51+ -- * Sampling
1652 -- | Value agnostic filtering.
1753 strideFromThen
1854
19− -- * Nesting
20− -- ** Set like operations
21− -- | These are not exactly set operations because streams are not
22− -- necessarily sets, they may have duplicated elements. These operations
23− -- are generic i.e. they work on streams of unconstrained types, therefore,
24− -- they have quadratic performance characterstics. For better performance
25− -- using Set structures see the Streamly.Internal.Data.Stream.Container
26− -- module.
27− , filterInStreamGenericBy
28− , deleteInStreamGenericBy
29− , unionWithStreamGenericBy
30−
31− -- ** Set like operations on sorted streams
32− , filterInStreamAscBy
33− , deleteInStreamAscBy
34− , unionWithStreamAscBy
35−
36− -- ** Join operations
37− , joinInnerGeneric
38−
39− -- * Joins on sorted stream
40− , joinInnerAscBy
41− , joinLeftAscBy
42− , joinOuterAscBy
55+ -- * Straight Joins
56+ -- | These are set-like operations but not exactly set operations because
57+ -- streams are not necessarily sets, they may have duplicated elements.
58+ -- These operations are generic i.e. they work on streams of unconstrained
59+ -- types, therefore, they have quadratic performance characterstics. For
60+ -- better performance using Set or Map structures see the
61+ -- Streamly.Internal.Data.Stream.Container module.
62+ , intersectBy
63+ , deleteFirstsBy
64+ , unionBy
65+
66+ -- Set like operations on sorted streams
67+ , sortedIntersectBy
68+ , sortedDeleteFirstsBy
69+ , sortedUnionBy
70+
71+ -- * Cross Joins
72+ , innerJoin
73+
74+ -- Joins on sorted stream
75+ , innerSortedJoin
76+ , leftSortedJoin
77+ , outerSortedJoin
4378 )
4479where
4580
@@ -48,8 +83,7 @@ where
4883import Control.Monad.IO.Class (MonadIO(..))
4984import Data.IORef (newIORef, readIORef, modifyIORef')
5085import Streamly.Internal.Data.Fold.Type (Fold)
51−import Streamly.Internal.Data.Stream.Common ()
52−import Streamly.Internal.Data.Stream.Type (Stream, cross)
86+import Streamly.Internal.Data.Stream.Type (Stream(..), Step(..), cross)
5387
5488import qualified Data.List as List
5589import qualified Streamly.Internal.Data.Fold as Fold
@@ -111,29 +145,40 @@ strideFromThen offset stride =
111145-- binary search if we have an Ord instance or Ordering returning function. The
112146-- time complexity would then become (m x log n).
113147
114−-- | Like 'cross' but emits only those tuples where @a == b@ using the
115−-- supplied equality predicate.
148+-- | Like 'cross' but emits only those tuples where @a == b@ using the supplied
149+-- equality predicate. This is essentially a @cross intersection@ of two
150+-- streams.
116151--
117152-- Definition:
118153--
119−-- >>> joinInnerGeneric eq s1 s2 = Stream.filter (\(a, b) -> a `eq` b) $ Stream.cross s1 s2
154+-- >>> innerJoin eq s1 s2 = Stream.filter (\(a, b) -> a `eq` b) $ Stream.cross s1 s2
155+--
156+-- The second (inner) stream must be finite. Moreover, it must be either pure
157+-- or capable of multiple evaluations. If not then the caller should cache it
158+-- in an 'Data.Array.Array', if the type does not have an 'Unbox' instance then
159+-- use the Generic 'Data.Array.Generic.Array'. Convert the array to stream
160+-- before calling this function. Caching may also improve performance if the
161+-- stream is expensive to evaluate.
120162--
121−-- You should almost always prefer @joinInnerOrd@ over 'joinInnerGeneric' if
122−-- possible. @joinInnerOrd@ is an order of magnitude faster but may take more
123−-- space for caching the second stream.
163+-- If you care about performance this function should be your last choice among
164+-- all inner joins. 'Streamly.Internal.Data.Unfold.innerJoin' is a much faster
165+-- fused alternative. 'innerSortedJoin' is a faster alternative when streams
166+-- are sorted. 'innerOrdJoin' is an order of magnitude faster alternative when
167+-- the type has an 'Ord' instance.
124168--
125−-- See 'Streamly.Internal.Data.Unfold.joinInnerGeneric' for a much faster fused
126−-- alternative.
169+-- Note: Conceptually, this is a commutative operation. Result includes all the
170+-- elements from the left and the right stream. The order of streams can be
171+-- changed without affecting results, except for the ordering within the tuple.
127172--
128173-- Time: O(m x n)
129174--
130175-- /Pre-release/
131−{-# INLINE joinInnerGeneric #-}
132−joinInnerGeneric :: Monad m =>
176+{-# INLINE innerJoin #-}
177+innerJoin :: Monad m =>
133178 (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, b)
134−joinInnerGeneric eq s1 s2 = Stream.filter (\(a, b) -> a `eq` b) $ cross s1 s2
179+innerJoin eq s1 s2 = Stream.filter (\(a, b) -> a `eq` b) $ cross s1 s2
135180{-
136−joinInnerGeneric eq s1 s2 = do
181+innerJoin eq s1 s2 = do
137182 -- ConcatMap works faster than bind
138183 Stream.concatMap (\a ->
139184 Stream.concatMap (\b ->
@@ -144,44 +189,44 @@ joinInnerGeneric eq s1 s2 = do
144189 ) s1
145190-}
146191
147−-- | A more efficient 'joinInner' for sorted streams.
192+-- | A more efficient 'innerJoin' for sorted streams.
148193--
149194-- Space: O(1)
150195--
151196-- Time: O(m + n)
152197--
153198-- /Unimplemented/
154−{-# INLINE joinInnerAscBy #-}
155−joinInnerAscBy ::
199+{-# INLINE innerSortedJoin #-}
200+innerSortedJoin ::
156201 (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, b)
157−joinInnerAscBy = undefined
202+innerSortedJoin = undefined
158203
159−-- | A more efficient 'joinLeft' for sorted streams.
204+-- | A more efficient 'leftJoin' for sorted streams.
160205--
161206-- Space: O(1)
162207--
163208-- Time: O(m + n)
164209--
165210-- /Unimplemented/
166−{-# INLINE joinLeftAscBy #-}
167−joinLeftAscBy :: -- Monad m =>
211+{-# INLINE leftSortedJoin #-}
212+leftSortedJoin :: -- Monad m =>
168213 (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
169−joinLeftAscBy _eq _s1 _s2 = undefined
214+leftSortedJoin _eq _s1 _s2 = undefined
170215
171−-- | A more efficient 'joinOuter' for sorted streams.
216+-- | A more efficient 'outerJoin' for sorted streams.
172217--
173218-- Space: O(1)
174219--
175220-- Time: O(m + n)
176221--
177222-- /Unimplemented/
178−{-# INLINE joinOuterAscBy #-}
179−joinOuterAscBy :: -- Monad m =>
223+{-# INLINE outerSortedJoin #-}
224+outerSortedJoin :: -- Monad m =>
180225 (a -> b -> Ordering)
181226 -> Stream m a
182227 -> Stream m b
183228 -> Stream m (Maybe a, Maybe b)
184−joinOuterAscBy _eq _s1 _s2 = undefined
229+outerSortedJoin _eq _s1 _s2 = undefined
185230
186231------------------------------------------------------------------------------
187232-- Set operations (special joins)
@@ -191,8 +236,8 @@ joinOuterAscBy _eq _s1 _s2 = undefined
191236-- the best would be to use an Array with linear search. If the second stream
192237-- is sorted we can also use a binary search, using Ord constraint.
193238
194−-- | Keep only those elements in the second stream that are present in the
195−-- first stream too. The first stream is folded to a container using the
239+-- | Keep only those elements in the first stream that are present in the
240+-- second stream too. The second stream is folded to a container using the
196241-- supplied fold and then the elements in the container are looked up using the
197242-- supplied lookup function.
198243--
@@ -207,23 +252,30 @@ filterStreamWith :: Monad m =>
207252filterStreamWith fld member s1 s2 =
208253 Stream.concatEffect
209254 $ do
210− xs <- Stream.fold fld s1
211− return $ Stream.filter (`member` xs) s2
255+ xs <- Stream.fold fld s2
256+ return $ Stream.filter (`member` xs) s1
212257
213−-- | 'filterInStreamGenericBy' retains only those elements in the second stream that
214−-- are present in the first stream.
215−--
216−-- >>> Stream.fold Fold.toList $ Stream.filterInStreamGenericBy (==) (Stream.fromList [1,2,2,4]) (Stream.fromList [2,1,1,3])
217−-- [2,1,1]
218−--
219−-- >>> Stream.fold Fold.toList $ Stream.filterInStreamGenericBy (==) (Stream.fromList [2,1,1,3]) (Stream.fromList [1,2,2,4])
220−-- [1,2,2]
258+-- XXX instead of folding the second stream to a list we could use it directly.
259+-- If the user wants they can generate the stream from an array and also call
260+-- uniq or nub on it. We can provide a convenience Stream -> Stream to cache
261+-- a finite stream in an array and serve it from the cache. The user can decide
262+-- what is best based on the context. They can also choose to use a boxed or
263+-- unboxed array for caching. To force caching we can make the second stream
264+-- monad type Identity. But that may be less flexible. One option is to use
265+-- cachedIntersectBy etc for automatic caching.
266+
267+-- | 'intersectBy' returns a subsequence of the first stream which intersects
268+-- with the second stream. Note that this is not a commutative operation unlike
269+-- a set intersection, because of duplicate elements in the stream the order of
270+-- the streams matters. This is similar to 'Data.List.intersectBy'. Note that
271+-- intersectBy is a special case of 'innerJoin'.
221272--
222−-- Similar to the list intersectBy operation but with the stream argument order
223−-- flipped.
273+-- >>> f s1 s2 = Stream.fold Fold.toList $ Stream.intersectBy (==) (Stream.fromList s1) (Stream.fromList s2)
274+-- >>> f [1,3,4,4,5]) [2,3,4,5,5]
275+-- [3,4,4,5]
224276--
225−-- The first stream must be finite and must not block. Second stream is
226−-- processed only after the first stream is fully realized.
277+-- First stream can be infinite, the second stream must be finite and must be
278+-- capable of multiple evaluations.
227279--
228280-- Space: O(n) where @n@ is the number of elements in the second stream.
229281--
@@ -231,46 +283,87 @@ filterStreamWith fld member s1 s2 =
231283-- @n@ is the number of elements in the second stream.
232284--
233285-- /Pre-release/
234−{-# INLINE filterInStreamGenericBy #-}
235−filterInStreamGenericBy :: Monad m =>
286+{-# INLINE intersectBy #-}
287+intersectBy :: Monad m =>
236288 (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
237−filterInStreamGenericBy eq =
289+intersectBy eq =
238290 -- XXX Use an (unboxed) array instead.
239291 filterStreamWith
240292 (Fold.scanMaybe (Fold.uniqBy eq) Fold.toListRev)
241293 (List.any . eq)
242294
243−-- | Like 'filterInStreamGenericBy' but assumes that the input streams are sorted in
295+-------------------------------------------------------------------------------
296+-- Intersection of sorted streams
297+-------------------------------------------------------------------------------
298+
299+-- XXX The sort order is not important as long both the streams have the same
300+-- sort order. We need to move only in one direction in each stream.
301+-- XXX Fix the argument order to use the same behavior as intersectBy.
302+
303+-- | Like 'intersectBy' but assumes that the input streams are sorted in
244304-- ascending order. To use it on streams sorted in descending order pass an
245305-- inverted comparison function returning GT for less than and LT for greater
246306-- than.
247307--
308+-- Both streams can be infinite.
309+--
248310-- Space: O(1)
249311--
250312-- Time: O(m+n)
251313--
252314-- /Pre-release/
253−{-# INLINE filterInStreamAscBy #-}
254−filterInStreamAscBy :: Monad m =>
315+{-# INLINE_NORMAL sortedIntersectBy #-}
316+sortedIntersectBy :: Monad m =>
255317 (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
256−filterInStreamAscBy eq s1 s2 = Stream.intersectBySorted eq s2 s1
318+sortedIntersectBy cmp (Stream stepa ta) (Stream stepb tb) =
319+ Stream step
320+ ( ta -- left stream state
321+ , tb -- right stream state
322+ , Nothing -- left value
323+ , Nothing -- right value
324+ )
257325
258−-- | Delete all elements of the first stream from the seconds stream. If an
259−-- element occurs multiple times in the first stream as many occurrences of it
260−-- are deleted from the second stream.
261−--
262−-- >>> Stream.fold Fold.toList $ Stream.deleteInStreamGenericBy (==) (Stream.fromList [1,2,3]) (Stream.fromList [1,2,2])
263−-- [2]
326+ where
327+
328+ {-# INLINE_LATE step #-}
329+ -- step 1, fetch the first value
330+ step gst (sa, sb, Nothing, b) = do
331+ r <- stepa gst sa
332+ return $ case r of
333+ Yield a sa' -> Skip (sa', sb, Just a, b) -- step 2/3
334+ Skip sa' -> Skip (sa', sb, Nothing, b)
335+ Stop -> Stop
336+
337+ -- step 2, fetch the second value
338+ step gst (sa, sb, a@(Just _), Nothing) = do
339+ r <- stepb gst sb
340+ return $ case r of
341+ Yield b sb' -> Skip (sa, sb', a, Just b) -- step 3
342+ Skip sb' -> Skip (sa, sb', a, Nothing)
343+ Stop -> Stop
344+
345+ -- step 3, compare the two values
346+ step _ (sa, sb, Just a, Just b) = do
347+ let res = cmp a b
348+ return $ case res of
349+ GT -> Skip (sa, sb, Just a, Nothing) -- step 2
350+ LT -> Skip (sa, sb, Nothing, Just b) -- step 1
351+ EQ -> Yield a (sa, sb, Nothing, Just b) -- step 1
352+
353+-- | Returns a subsequence of the first stream, deleting first occurrences of
354+-- those elements that are present in the second stream. Note that this is not
355+-- a commutative operation. This is similar to the 'Data.List.deleteFirstsBy'.
264356--
265−-- The following laws hold:
357+-- >>> f xs ys = Stream.fold Fold.toList $ Stream.deleteFirstsBy (==) (Stream.fromList xs) (Stream.fromList ys)
358+-- >>> f [1,2,2,3,3,5] [1,2,2,3,4]
359+-- [2,3,5]
266360--
267−-- > deleteInStreamGenericBy (==) s1 (s1 `append` s2) === s2
268−-- > deleteInStreamGenericBy (==) s1 (s1 `interleave` s2) === s2
361+-- The following holds:
269362--
270−-- Same as the list 'Data.List.//' operation but with argument order flipped.
363+-- > deleteFirstsBy (==) (Stream.nub s2 `append` s1) s2 === s1
364+-- > deleteFirstsBy (==) (Stream.nub s2 `interleave` s1) s2 === s1
271365--
272−-- The first stream must be finite and must not block. Second stream is
273−-- processed only after the first stream is fully realized.
366+-- First stream can be infinite, second stream must be finite.
274367--
275368-- Space: O(m) where @m@ is the number of elements in the first stream.
276369--
@@ -278,58 +371,83 @@ filterInStreamAscBy eq s1 s2 = Stream.intersectBySorted eq s2 s1
278371-- @n@ is the number of elements in the second stream.
279372--
280373-- /Pre-release/
281−{-# INLINE deleteInStreamGenericBy #-}
282−deleteInStreamGenericBy :: Monad m =>
374+{-# INLINE deleteFirstsBy #-}
375+deleteFirstsBy :: Monad m =>
283376 (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
284−deleteInStreamGenericBy eq s1 s2 =
285− Stream.concatEffect
286− $ do
287− -- This may work well if s1 is small
288− -- If s1 is big we can go through s1, deleting elements from s2 and
289− -- not emitting an element if it was successfully deleted from s2.
290− -- we will need a deleteBy that can return whether the element was
291− -- deleted or not.
292− xs <- Stream.fold Fold.toList s2
293− let f = Fold.foldl' (flip (List.deleteBy eq)) xs
294− fmap Stream.fromList $ Stream.fold f s1
377+deleteFirstsBy eq s2 s1 =
378+ -- XXX s2 can be a sorted mutable array and we can use binary
379+ -- search to find. Mark the element deleted, count the deletions
380+ -- and reconsolidate the array when a min number of elements is
381+ -- deleted.
295382
296−-- | A more efficient 'deleteInStreamGenericBy' for streams sorted in ascending order.
383+ -- XXX Use StreamK or list as second argument instead of Stream to avoid
384+ -- concatEffect?
385+ Stream.concatEffect $ do
386+ xs <- Stream.toList s1
387+ -- It reverses the list but that is fine.
388+ let del x =
389+ List.foldl' (\(ys,res) y ->
390+ if x `eq` y
391+ then (ys, True)
392+ else (x:ys, res)) ([], False)
393+ g (ys,_) x =
394+ let (ys1, deleted) = del x ys
395+ in if deleted
396+ then (ys1, Nothing)
397+ else (ys1, Just x)
398+ in return
399+ $ Stream.catMaybes
400+ $ fmap snd
401+ $ Stream.postscanl' g (xs, Nothing) s2
402+
403+-- | A more efficient 'deleteFirstsBy' for streams sorted in ascending order.
404+--
405+-- Both streams can be infinite.
297406--
298407-- Space: O(1)
299408--
300409-- /Unimplemented/
301−{-# INLINE deleteInStreamAscBy #-}
302−deleteInStreamAscBy :: -- (Monad m) =>
410+{-# INLINE sortedDeleteFirstsBy #-}
411+sortedDeleteFirstsBy :: -- (Monad m) =>
303412 (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
304−deleteInStreamAscBy _eq _s1 _s2 = undefined
413+sortedDeleteFirstsBy _eq _s1 _s2 = undefined
305414
306415-- XXX Remove the MonadIO constraint. We can just cache one stream and then
307416-- implement using differenceEqBy.
308417
309−-- | This essentially appends to the second stream all the occurrences of
310−-- elements in the first stream that are not already present in the second
311−-- stream.
418+-- | Returns the first stream appended with those unique elements from the
419+-- second stream that are not already present in the first stream. Note that
420+-- this is not a commutative operation unlike a set union, argument order
421+-- matters. The behavior is similar to the 'Data.List.unionBy'.
312422--
313423-- Equivalent to the following except that @s2@ is evaluated only once:
314424--
315−-- >>> unionWithStreamGenericBy eq s1 s2 = s2 `Stream.append` (Stream.deleteInStreamGenericBy eq s2 s1)
425+-- >>> unionBy eq s1 s2 = s1 `Stream.append` Stream.deleteFirstsBy eq s1 (Stream.nub s2)
316426--
317427-- Example:
318428--
319−-- >>> Stream.fold Fold.toList $ Stream.unionWithStreamGenericBy (==) (Stream.fromList [1,1,2,3]) (Stream.fromList [1,2,2,4])
429+-- >>> f s1 s2 = Stream.fold Fold.toList $ Stream.unionBy (==) (Stream.fromList s1) (Stream.fromList s2)
430+-- >>> f [1,2,2,4] [1,1,2,3,3]
320431-- [1,2,2,4,3]
321432--
433+-- First stream can be infinite, but second stream must be finite. Note that if
434+-- the first stream is infinite the union means just the first stream. Thus
435+-- union is useful only when both streams are finite. See 'sortedUnionBy' where
436+-- union can work on infinite streams if they are sorted.
437+--
322438-- Space: O(n)
323439--
324440-- Time: O(m x n)
325441--
326442-- /Pre-release/
327−{-# INLINE unionWithStreamGenericBy #-}
328−unionWithStreamGenericBy :: MonadIO m =>
443+{-# INLINE unionBy #-}
444+unionBy :: MonadIO m =>
329445 (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
330−unionWithStreamGenericBy eq s1 s2 =
446+unionBy eq s2 s1 =
331447 Stream.concatEffect
332448 $ do
449+ -- XXX use a rewrite rule such that if a list converted to stream
450+ -- is passed to unionBy then this becomes an identity operation.
333451 xs <- Stream.fold Fold.toList s1
334452 -- XXX we can use postscanlMAfter' instead of IORef
335453 ref <- liftIO $ newIORef $! List.nubBy eq xs
@@ -342,12 +460,21 @@ unionWithStreamGenericBy eq s1 s2 =
342460 return $ Stream.fromList xs1
343461 return $ Stream.mapM f s2 `Stream.append` s3
344462
345−-- | A more efficient 'unionWithStreamGenericBy' for sorted streams.
463+-- | A more efficient 'unionBy' for sorted streams.
464+--
465+-- Note that the behavior is different from 'unionBy'. In 'unionBy' we append
466+-- the unique elements from second stream only after exhausting the first one
467+-- whereas in sorted streams we can determine unique elements early even when
468+-- we are going through the first stream. Thus the result is an interleaving of
469+-- the two streams, merging those elements from the second stream that are not
470+-- present in the first.
346471--
347472-- Space: O(1)
348473--
474+-- Both streams can be infinite.
475+--
349476-- /Unimplemented/
350−{-# INLINE unionWithStreamAscBy #-}
351−unionWithStreamAscBy :: -- (Monad m) =>
477+{-# INLINE sortedUnionBy #-}
478+sortedUnionBy :: -- (Monad m) =>
352479 (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
353−unionWithStreamAscBy _eq _s1 _s2 = undefined
480+sortedUnionBy _eq _s1 _s2 = undefined
core/src/Streamly/Internal/Data/Stream/Transform.hs+61−35
@@ -21,8 +21,6 @@ module Streamly.Internal.Data.Stream.Transform
2121
2222 -- * Mapping
2323 -- | Stateless one-to-one maps.
24− , map
25− , mapM
2624 , sequence
2725
2826 -- * Mapping Effects
@@ -75,7 +73,7 @@ module Streamly.Internal.Data.Stream.Transform
7573 , scanMaybe
7674 , filter
7775 , filterM
78− , deleteBy
76+ , deleteBy -- deleteOnceBy?
7977 , uniqBy
8078 , uniq
8179 , prune
@@ -83,9 +81,8 @@ module Streamly.Internal.Data.Stream.Transform
8381
8482 -- * Trimming
8583 -- | Produce a subset of the stream trimmed at ends.
86− , take
87− , takeWhile
88− , takeWhileM
84+ , initNonEmpty
85+ , tailNonEmpty
8986 , takeWhileLast
9087 , takeWhileAround
9188 , drop
@@ -132,7 +129,6 @@ module Streamly.Internal.Data.Stream.Transform
132129 -- * Searching
133130 , findIndices
134131 , elemIndices
135− , slicesBy
136132
137133 -- * Rolling map
138134 -- | Map using the previous element.
@@ -300,7 +296,7 @@ data TapState fs st a
300296--
301297{-# INLINE tap #-}
302298tap :: Monad m => Fold m a b -> Stream m a -> Stream m a
303−tap (Fold fstep initial extract) (Stream step state) = Stream step' TapInit
299+tap (Fold fstep initial _ final) (Stream step state) = Stream step' TapInit
304300
305301 where
306302
@@ -323,7 +319,7 @@ tap (Fold fstep initial extract) (Stream step state) = Stream step' TapInit
323319 FL.Done _ -> TapDone s
324320 Skip s -> return $ Skip (Tapping acc s)
325321 Stop -> do
326− void $ extract acc
322+ void $ final acc
327323 return Stop
328324 step' gst (TapDone st) = do
329325 r <- step gst st
@@ -342,7 +338,7 @@ data TapOffState fs s a
342338{-# INLINE_NORMAL tapOffsetEvery #-}
343339tapOffsetEvery :: Monad m
344340 => Int -> Int -> Fold m a b -> Stream m a -> Stream m a
345−tapOffsetEvery offset n (Fold fstep initial extract) (Stream step state) =
341+tapOffsetEvery offset n (Fold fstep initial _ final) (Stream step state) =
346342 Stream step' TapOffInit
347343
348344 where
@@ -372,7 +368,7 @@ tapOffsetEvery offset n (Fold fstep initial extract) (Stream step state) =
372368 return $ Yield x next
373369 Skip s -> return $ Skip (TapOffTapping acc s count)
374370 Stop -> do
375− void $ extract acc
371+ void $ final acc
376372 return Stop
377373 step' gst (TapOffDone st) = do
378374 r <- step gst st
@@ -437,7 +433,7 @@ data ScanState s f = ScanInit s | ScanDo s !f | ScanDone
437433--
438434{-# INLINE_NORMAL postscan #-}
439435postscan :: Monad m => FL.Fold m a b -> Stream m a -> Stream m b
440−postscan (FL.Fold fstep initial extract) (Stream sstep state) =
436+postscan (FL.Fold fstep initial extract final) (Stream sstep state) =
441437 Stream step (ScanInit state)
442438
443439 where
@@ -460,13 +456,13 @@ postscan (FL.Fold fstep initial extract) (Stream sstep state) =
460456 return $ Yield b $ ScanDo s fs1
461457 FL.Done b -> return $ Yield b ScanDone
462458 Skip s -> return $ Skip $ ScanDo s fs
463− Stop -> return Stop
459+ Stop -> final fs >> return Stop
464460 step _ ScanDone = return Stop
465461
466462{-# INLINE scanWith #-}
467463scanWith :: Monad m
468464 => Bool -> Fold m a b -> Stream m a -> Stream m b
469−scanWith restart (Fold fstep initial extract) (Stream sstep state) =
465+scanWith restart (Fold fstep initial extract final) (Stream sstep state) =
470466 Stream step (ScanInit state)
471467
472468 where
@@ -489,7 +485,7 @@ scanWith restart (Fold fstep initial extract) (Stream sstep state) =
489485 case res of
490486 Yield x s -> runStep s (fstep fs x)
491487 Skip s -> return $ Skip $ ScanDo s fs
492− Stop -> return Stop
488+ Stop -> final fs >> return Stop
493489 step _ ScanDone = return Stop
494490
495491-- XXX It may be useful to have a version of scan where we can keep the
@@ -1074,6 +1070,55 @@ repeated = undefined
10741070-- Trimming
10751071------------------------------------------------------------------------------
10761072
1073+-- | init for non-empty streams, fails for empty stream case.
1074+--
1075+{-# INLINE initNonEmpty #-}
1076+initNonEmpty :: Monad m => Stream m a -> Stream m a
1077+initNonEmpty (Stream step1 state1) = Stream step (Nothing, state1)
1078+
1079+ where
1080+
1081+ step gst (Nothing, s1) = do
1082+ r <- step1 (adaptState gst) s1
1083+ return $
1084+ case r of
1085+ Yield x s -> Skip (Just x, s)
1086+ Skip s -> Skip (Nothing, s)
1087+ Stop -> error "initNonEmpty: empty Stream"
1088+
1089+ step gst (Just a, s1) = do
1090+ r <- step1 (adaptState gst) s1
1091+ return $
1092+ case r of
1093+ Yield x s -> Yield a (Just x, s)
1094+ Skip s -> Skip (Just a, s)
1095+ Stop -> Stop
1096+
1097+-- | tail for non-empty streams, fails for empty stream case.
1098+--
1099+-- See also 'tail' for a non-partial version of this function..
1100+{-# INLINE tailNonEmpty #-}
1101+tailNonEmpty :: Monad m => Stream m a -> Stream m a
1102+tailNonEmpty (Stream step1 state1) = Stream step (Nothing, state1)
1103+
1104+ where
1105+
1106+ step gst (Nothing, s1) = do
1107+ r <- step1 (adaptState gst) s1
1108+ return $
1109+ case r of
1110+ Yield x s -> Skip (Just x, s)
1111+ Skip s -> Skip (Nothing, s)
1112+ Stop -> error "tailNonEmpty: empty Stream"
1113+
1114+ step gst (Just a, s1) = do
1115+ r <- step1 (adaptState gst) s1
1116+ return $
1117+ case r of
1118+ Yield x s -> Yield x (Just x, s)
1119+ Skip s -> Skip (Just a, s)
1120+ Stop -> Stop
1121+
10771122-- | Take all consecutive elements at the end of the stream for which the
10781123-- predicate is true.
10791124--
@@ -1537,7 +1582,7 @@ reverse m = Stream step Nothing
15371582{-# INLINE reverseUnbox #-}
15381583reverseUnbox :: (MonadIO m, Unbox a) => Stream m a -> Stream m a
15391584reverseUnbox =
1540− A.flattenArraysRev -- unfoldMany A.readRev
1585+ A.concatRev -- unfoldMany A.readerRev
15411586 . fromStreamK
15421587 . K.reverse
15431588 . toStreamK
@@ -1709,25 +1754,6 @@ findIndices p (Stream step state) = Stream step' (state, 0)
17091754elemIndices :: (Monad m, Eq a) => a -> Stream m a -> Stream m Int
17101755elemIndices a = findIndices (== a)
17111756
1712−{-# INLINE_NORMAL slicesBy #-}
1713−slicesBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
1714−slicesBy p (Stream step1 state1) = Stream step (Just (state1, 0, 0))
1715−
1716− where
1717−
1718− {-# INLINE_LATE step #-}
1719− step gst (Just (st, i, len)) = i `seq` len `seq` do
1720− r <- step1 (adaptState gst) st
1721− return
1722− $ case r of
1723− Yield x s ->
1724− if p x
1725− then Yield (i, len + 1) (Just (s, i + len + 1, 0))
1726− else Skip (Just (s, i, len + 1))
1727− Skip s -> Skip (Just (s, i, len))
1728− Stop -> if len == 0 then Stop else Yield (i, len) Nothing
1729− step _ Nothing = return Stop
1730−
17311757------------------------------------------------------------------------------
17321758-- Rolling map
17331759------------------------------------------------------------------------------
core/src/Streamly/Internal/Data/Stream/Transformer.hs+2−1
@@ -11,10 +11,11 @@
1111
1212module Streamly.Internal.Data.Stream.Transformer
1313 (
14+ -- * Fold to Transformer Monad
1415 foldlT
1516 , foldrT
1617
17− -- * Transform Inner Monad
18+ -- * Inner Monad Operations
1819 , liftInner
1920 , runReaderT
2021 , usingReaderT
core/src/Streamly/Internal/Data/Stream/Type.hs+147−14

This diff is too big to show here.

core/src/Streamly/Internal/Data/StreamK.hs+35−40

This diff is too big to show here.

core/src/Streamly/Internal/Data/StreamK/Type.hs+82−11

This diff is too big to show here.

core/src/Streamly/Internal/Data/Time/Units.hs−3

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unbox.hs+172−263

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unbox/TH.hs+76−85

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unfold.hs+15−13

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/File.hs+11−13

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/Handle.hs+21−28

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/Path.hsnew file+515

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/Parser.hs+3

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/Stream.hs+39−11

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/String.hs+7−7

This diff is too big to show here.

core/streamly-core.cabal+102−37

This diff is too big to show here.

docs/Developer/optimization-guidelines.md+12

This diff is too big to show here.

docs/Developer/paths.rst+25−2

This diff is too big to show here.

docs/User/HowTo/Compiling.md+5−3

This diff is too big to show here.

docs/User/Project/ApiChangelogs/0.9.0-0.10.0.txtnew file+424

This diff is too big to show here.

docs/User/Project/CONTRIBUTORS.md+8

This diff is too big to show here.

docs/User/Project/Changelog.md+25−10

This diff is too big to show here.

docs/User/Project/README.md+9−2

This diff is too big to show here.

docs/User/Project/Upgrading.md → docs/User/Project/Upgrading-0.8-to-0.9.mdrenamed+87−9

This diff is too big to show here.

docs/User/Tutorials/quick-overview.md+10−14

This diff is too big to show here.

docs/streamly-docs.cabal+1−1

This diff is too big to show here.

src/Streamly.hsdeleted−625

This diff is too big to show here.

src/Streamly/Data/Fold/Prelude.hs+31−13

This diff is too big to show here.

src/Streamly/Data/Fold/Tee.hs+1−1

This diff is too big to show here.

src/Streamly/Data/Stream/Concurrent.hsdeleted−194

This diff is too big to show here.

src/Streamly/Data/Stream/Exception.hsdeleted−24

This diff is too big to show here.

src/Streamly/Data/Stream/MkType.hs+41−10

This diff is too big to show here.

src/Streamly/Data/Stream/Prelude.hs+217−15

This diff is too big to show here.

src/Streamly/Data/Stream/Time.hsdeleted−37

This diff is too big to show here.

src/Streamly/Data/Unicode/Stream.hsdeleted−99

This diff is too big to show here.

src/Streamly/Internal/Data/Array/ArrayMacros.hdeleted−40

This diff is too big to show here.

src/Streamly/Internal/Data/Channel.hsnew file+19

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Channel/Dispatcher.hs → src/Streamly/Internal/Data/Channel/Dispatcher.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Channel/Types.hs → src/Streamly/Internal/Data/Channel/Types.hsrenamed+12−37

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Channel/Worker.hs → src/Streamly/Internal/Data/Channel/Worker.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Async.hs+5−2

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent.hs+1−44

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent/Channel.hs+29−10

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent/Channel/Type.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Prelude.hsnew file+19

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/SVar.hs+6−5

This diff is too big to show here.

src/Streamly/Internal/Data/IsMap/HashMap.hs+6

This diff is too big to show here.

src/Streamly/Internal/Data/SVar.hs+1

This diff is too big to show here.

src/Streamly/Internal/Data/SmallArray.hs+1−1

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/Common.hs → src/Streamly/Internal/Data/Stream/Common.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent.hs+50−39

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel.hs+17−14

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Append.hs+10−10

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Consumer.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Dispatcher.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Interleave.hs+8−8

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Operations.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Type.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream.hs+6

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Common.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Exception.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Expand.hs+5

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Generate.hs+4−4

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Reduce.hs+4−4

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Top.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Exception/Lifted.hs → src/Streamly/Internal/Data/Stream/Lifted.hsrenamed+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/MkType.hs+33−34

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Prelude.hsnew file+19

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/SVar.hsnew file+18

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/SVar/Generate.hs+2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Serial.hs+5

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Time.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Zip.hs+39

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Zip/Concurrent.hsdeleted−50

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/ZipAsync.hs+2−1

This diff is too big to show here.

src/Streamly/Internal/Data/Unfold/Prelude.hsnew file+17

This diff is too big to show here.

src/Streamly/Internal/Data/Unfold/SVar.hs+1

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event.hs+5−2

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Darwin.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Linux.hs+15−9

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Windows.hs+11−8

This diff is too big to show here.

src/Streamly/Internal/FileSystem/FD.hs+10−9

This diff is too big to show here.

src/Streamly/Internal/Network/Inet/TCP.hs+80−45

This diff is too big to show here.

src/Streamly/Internal/Network/Socket.hs+29−26

This diff is too big to show here.

src/Streamly/Internal/Unicode/Char.hs+9−7

This diff is too big to show here.

src/Streamly/Memory/Array.hsdeleted−74

This diff is too big to show here.

src/Streamly/Network/Inet/TCP.hs+41−30

This diff is too big to show here.

src/Streamly/Network/Socket.hs+16−30

This diff is too big to show here.

streamly.cabal+82−62

This diff is too big to show here.

targets/Targets.hs+5−2

This diff is too big to show here.

test/Streamly/Test/Data/Array.hs+3−3

This diff is too big to show here.

test/Streamly/Test/Data/Array/Stream.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Fold.hs+7−6

This diff is too big to show here.

test/Streamly/Test/Data/MutArray.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Parser.hs+12−6

This diff is too big to show here.

test/Streamly/Test/Data/ParserK.hs+7−2

This diff is too big to show here.

test/Streamly/Test/Data/Serialize.hs+201−22

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/CompatV0.hsnew file+46

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/CompatV1.hsnew file+48

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/TH.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Data/Stream/Concurrent.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Unbox.hs+75−45

This diff is too big to show here.

test/Streamly/Test/Data/Unbox/TH.hs+11−10

This diff is too big to show here.

test/Streamly/Test/FileSystem/Event/Common.hs+1−1

This diff is too big to show here.

test/Streamly/Test/FileSystem/Handle.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Prelude/Serial.hs+7−3

This diff is too big to show here.

test/Streamly/Test/Serialize/Serializable.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Unicode/Char.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Unicode/Parser.hs+6−16

This diff is too big to show here.

test/Streamly/Test/Unicode/Stream.hs+1−1

This diff is too big to show here.

test/lib/Streamly/Test/Prelude/Common.hs+9−2

This diff is too big to show here.

test/streamly-tests.cabal+29−12

This diff is too big to show here.