git.iko.soy/streamly
on master help sign in
swap
30c5a56201 ahead, 0 behind
227 files changed+10871−6352
since splitsnapshots
unifiedsplit
.circleci/config.yml+73−277 .github/workflows/haskell.yml+61−60 .hlint.ignore+5 benchmark/NanoBenchmarks.hs+1−1 benchmark/Streamly/Benchmark/Data/Array/CommonImports.hs+1−1 benchmark/Streamly/Benchmark/Data/Array/Stream.hs+3−3 benchmark/Streamly/Benchmark/Data/Fold.hs+3−3 benchmark/Streamly/Benchmark/Data/Fold/Window.hs+1−1 benchmark/Streamly/Benchmark/Data/MutArray.hs+1−1 benchmark/Streamly/Benchmark/Data/Parser.hs+3−3 benchmark/Streamly/Benchmark/Data/ParserK.hs+3−3 benchmark/Streamly/Benchmark/Data/Ring/Unboxed.hs+1−1 benchmark/Streamly/Benchmark/Data/Serialize.hs+34−48 benchmark/Streamly/Benchmark/Data/Serialize/RecCompatible.hs+3−3 benchmark/Streamly/Benchmark/Data/Serialize/RecNonCompatible.hs+2−2 benchmark/Streamly/Benchmark/Data/Serialize/TH.hs+4−4 benchmark/Streamly/Benchmark/Data/Stream/Common.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentCommon.hs+3−3 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentEager.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentInterleaved.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentOrdered.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Eliminate.hs+3−1 benchmark/Streamly/Benchmark/Data/Stream/Exceptions.hs+3−3 benchmark/Streamly/Benchmark/Data/Stream/Expand.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Generate.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Lift.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Reduce.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/Split.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/StreamD.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/StreamKAlt.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/ToStreamK.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/Transform.hs+2−2 benchmark/Streamly/Benchmark/Data/Unfold.hs+2−2 benchmark/Streamly/Benchmark/FileSystem/Handle.hs+1−1 benchmark/Streamly/Benchmark/FileSystem/Handle/Read.hs+8−9 benchmark/Streamly/Benchmark/FileSystem/Handle/ReadWrite.hs+9−10 benchmark/Streamly/Benchmark/Prelude/Adaptive.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Ahead.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Async.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Concurrent.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Merge.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Parallel.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Rate.hs+1−1 benchmark/Streamly/Benchmark/Prelude/WAsync.hs+1−1 benchmark/Streamly/Benchmark/Prelude/WSerial.hs+1−1 benchmark/Streamly/Benchmark/Prelude/ZipAsync.hs+1−1 benchmark/Streamly/Benchmark/Prelude/ZipSerial.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Char.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Parser.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Stream.hs+24−6 benchmark/Streamly/Benchmark/Unicode/Utf8.hs+1−1 benchmark/lib/Streamly/Benchmark/Common.hs+1−1 benchmark/lib/Streamly/Benchmark/Common/Handle.hs+3−1 benchmark/lib/Streamly/Benchmark/Prelude.hs+1−1 benchmark/streamly-benchmarks.cabal+5−20 configure+9−9 configure.ac+1−1 core/configure+9−9 core/configure.ac+1−1 core/docs/ApiChangelogs/0.1.0-0.2.0.txt+1471 core/docs/ApiChangelogs/0.2.0-0.2.2.txt+306 core/docs/Changelog.md+58−16 core/src/DocTestDataArray.hs+3 core/src/DocTestDataParserK.hs+15 core/src/DocTestDataStream.hs+2 core/src/DocTestFileSystemHandle.hs+1−1 core/src/Streamly/Console/Stdio.hs+1−27 core/src/Streamly/Data/Array.hs+40−39 core/src/Streamly/Data/Array/Generic.hs+16−13 core/src/Streamly/Data/Fold.hs+150−118 core/src/Streamly/Data/MutArray.hs+18−12 core/src/Streamly/Data/MutArray/Generic.hs+8−3 core/src/Streamly/Data/MutByteArray.hs+106 core/src/Streamly/Data/Parser.hs+130−60 core/src/Streamly/Data/ParserK.hs+89−3 core/src/Streamly/Data/Serialize.hs−50 core/src/Streamly/Data/Stream.hs+131−36 core/src/Streamly/Data/StreamK.hs+2−1 core/src/Streamly/Data/Unfold.hs+2 core/src/Streamly/FileSystem/Handle.hs+7−30 core/src/Streamly/Internal/Data/Array.hs+608−135 core/src/Streamly/Internal/Data/Array/Generic.hs+22−6 core/src/Streamly/Internal/Data/Stream/Chunked.hs → core/src/Streamly/Internal/Data/Array/Stream.hs+74−586 core/src/Streamly/Internal/Data/Array/Type.hs+517−109 core/src/Streamly/Internal/Serialize/FromBytes.hs → core/src/Streamly/Internal/Data/Binary/Parser.hs+8−2 core/src/Streamly/Internal/Serialize/ToBytes.hs → core/src/Streamly/Internal/Data/Binary/Stream.hs+2−2 core/src/Streamly/Internal/Data/Fold/Chunked.hs+16−15 core/src/Streamly/Internal/Data/Fold/Combinators.hs+81−137 core/src/Streamly/Internal/Data/Fold/Container.hs+73−28 core/src/Streamly/Internal/Data/Fold/Step.hs+1−1 core/src/Streamly/Internal/Data/Fold/Type.hs+336−114 core/src/Streamly/Internal/Data/Fold/Window.hs+10−4 core/src/Streamly/Internal/Data/IORef/Unboxed.hs+22−19 core/src/Streamly/Internal/Data/IsMap.hs+8 core/src/Streamly/Internal/Data/MutArray.hs+154−33 core/src/Streamly/Internal/Data/MutArray/Generic.hs+65−30 core/src/Streamly/Internal/Data/MutArray/Stream.hs+145 core/src/Streamly/Internal/Data/MutArray/Type.hs+1271−417 core/src/Streamly/Internal/Data/MutByteArray.hs+251 core/src/Streamly/Internal/Data/MutByteArray/Type.hs+321 core/src/Streamly/Internal/Data/Parser.hs+122−117 core/src/Streamly/Internal/Data/Parser/Type.hs+89−69 core/src/Streamly/Internal/Data/ParserK/Type.hs+24−64 core/src/Streamly/Internal/Data/Ring.hs+7−4 core/src/Streamly/Internal/Data/Ring/Generic.hs+1−1 core/src/Streamly/Internal/Data/Serialize.hs−122 core/src/Streamly/Internal/Data/Serialize/TH.hs+113−96 core/src/Streamly/Internal/Data/Serialize/TH/Bottom.hs+143−48 core/src/Streamly/Internal/Data/Serialize/TH/RecHeader.hs+37−37 core/src/Streamly/Internal/Data/Serialize/Type.hs+140−137 core/src/Streamly/Internal/Data/Stream/Container.hs+56−48 core/src/Streamly/Internal/Data/Stream/Eliminate.hs+33−25 core/src/Streamly/Internal/Data/Stream/Exception.hs+9−6 core/src/Streamly/Internal/Data/Stream/Generate.hs+2−12 core/src/Streamly/Internal/Data/Stream/MutChunked.hs−323 core/src/Streamly/Internal/Data/Stream/Nesting.hs+86−202 core/src/Streamly/Internal/Data/Stream/Top.hs+243−116 core/src/Streamly/Internal/Data/Stream/Transform.hs+61−35 core/src/Streamly/Internal/Data/Stream/Transformer.hs+2−1 core/src/Streamly/Internal/Data/Stream/Type.hs+145−14 core/src/Streamly/Internal/Data/StreamK.hs+35−40 core/src/Streamly/Internal/Data/StreamK/Type.hs+80−11 core/src/Streamly/Internal/Data/Time/Units.hs−4 core/src/Streamly/Internal/Data/Unbox.hs+165−289 core/src/Streamly/Internal/Data/Unbox/TH.hs+74−74 core/src/Streamly/Internal/Data/Unfold.hs+15−13 core/src/Streamly/Internal/FileSystem/File.hs+11−13 core/src/Streamly/Internal/FileSystem/Handle.hs+21−28 core/src/Streamly/Internal/FileSystem/Path.hs+515 core/src/Streamly/Internal/Unicode/Parser.hs+3 core/src/Streamly/Internal/Unicode/Stream.hs+39−11 core/src/Streamly/Internal/Unicode/String.hs+7−7 core/streamly-core.cabal+83−35 docs/Developer/optimization-guidelines.md+12 docs/Developer/paths.rst+25−2 docs/User/Project/ApiChangelogs/0.9.0-0.10.0.txt+424 docs/User/Project/CONTRIBUTORS.md+8 docs/User/Project/Changelog.md+25−10 docs/User/Project/README.md+9−2 docs/User/Project/Upgrading.md → docs/User/Project/Upgrading-0.8-to-0.9.md+87−9 docs/User/Tutorials/quick-overview.md+10−14 src/Streamly.hs−625 src/Streamly/Data/Fold/Prelude.hs+31−13 src/Streamly/Data/Fold/Tee.hs+1−1 src/Streamly/Data/Stream/Concurrent.hs−194 src/Streamly/Data/Stream/Exception.hs−25 src/Streamly/Data/Stream/MkType.hs+41−10 src/Streamly/Data/Stream/Prelude.hs+217−15 src/Streamly/Data/Stream/Time.hs−37 src/Streamly/Data/Unicode/Stream.hs−99 src/Streamly/Internal/Data/Array/ArrayMacros.h−40 src/Streamly/Internal/Data/Channel.hs+19 src/Streamly/Internal/Data/Stream/Channel/Dispatcher.hs → src/Streamly/Internal/Data/Channel/Dispatcher.hs+3−3 src/Streamly/Internal/Data/Stream/Channel/Types.hs → src/Streamly/Internal/Data/Channel/Types.hs+12−37 src/Streamly/Internal/Data/Stream/Channel/Worker.hs → src/Streamly/Internal/Data/Channel/Worker.hs+3−3 src/Streamly/Internal/Data/Fold/Async.hs+5−2 src/Streamly/Internal/Data/Fold/Concurrent.hs+1−44 src/Streamly/Internal/Data/Fold/Concurrent/Channel.hs+29−10 src/Streamly/Internal/Data/Fold/Concurrent/Channel/Type.hs+3−3 src/Streamly/Internal/Data/Fold/Prelude.hs+19 src/Streamly/Internal/Data/Fold/SVar.hs+6−5 src/Streamly/Internal/Data/IsMap/HashMap.hs+6 src/Streamly/Internal/Data/SVar.hs+1 src/Streamly/Internal/Data/SmallArray.hs+1−1 core/src/Streamly/Internal/Data/Stream/Common.hs → src/Streamly/Internal/Data/Stream/Common.hs+3−3 src/Streamly/Internal/Data/Stream/Concurrent.hs+50−39 src/Streamly/Internal/Data/Stream/Concurrent/Channel.hs+17−14 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Append.hs+10−10 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Consumer.hs+2−2 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Dispatcher.hs+3−3 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Interleave.hs+8−8 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Operations.hs+1−1 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Type.hs+3−3 src/Streamly/Internal/Data/Stream/IsStream.hs+6 src/Streamly/Internal/Data/Stream/IsStream/Common.hs+2−2 src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs+1−1 src/Streamly/Internal/Data/Stream/IsStream/Exception.hs+1−1 src/Streamly/Internal/Data/Stream/IsStream/Expand.hs+5 src/Streamly/Internal/Data/Stream/IsStream/Generate.hs+4−4 src/Streamly/Internal/Data/Stream/IsStream/Reduce.hs+4−4 src/Streamly/Internal/Data/Stream/IsStream/Top.hs+1−1 src/Streamly/Internal/Data/Stream/Exception/Lifted.hs → src/Streamly/Internal/Data/Stream/Lifted.hs+2−2 src/Streamly/Internal/Data/Stream/MkType.hs+33−34 src/Streamly/Internal/Data/Stream/Prelude.hs+19 src/Streamly/Internal/Data/Stream/SVar.hs+18 src/Streamly/Internal/Data/Stream/Serial.hs+5 src/Streamly/Internal/Data/Stream/Time.hs+2−2 src/Streamly/Internal/Data/Stream/Zip.hs+37 src/Streamly/Internal/Data/Stream/Zip/Concurrent.hs−50 src/Streamly/Internal/Data/Stream/ZipAsync.hs+2−1 src/Streamly/Internal/Data/Unfold/Prelude.hs+17 src/Streamly/Internal/Data/Unfold/SVar.hs+1 src/Streamly/Internal/FileSystem/Event.hs+3 src/Streamly/Internal/FileSystem/Event/Darwin.hs+1−1 src/Streamly/Internal/FileSystem/Event/Linux.hs+2−2 src/Streamly/Internal/FileSystem/Event/Windows.hs+2−2 src/Streamly/Internal/FileSystem/FD.hs+7−8 src/Streamly/Internal/Network/Inet/TCP.hs+76−44 src/Streamly/Internal/Network/Socket.hs+17−14 src/Streamly/Internal/Unicode/Char.hs+3−3 src/Streamly/Memory/Array.hs−74 src/Streamly/Network/Inet/TCP.hs+41−30 src/Streamly/Network/Socket.hs+16−30 streamly.cabal+69−56 targets/Targets.hs+4−1 test/Streamly/Test/Data/Array.hs+3−3 test/Streamly/Test/Data/Array/Stream.hs+1−1 test/Streamly/Test/Data/Fold.hs+7−6 test/Streamly/Test/Data/MutArray.hs+1−1 test/Streamly/Test/Data/Parser.hs+8−6 test/Streamly/Test/Data/ParserK.hs+2−2 test/Streamly/Test/Data/Serialize.hs+49−20 test/Streamly/Test/Data/Serialize/CompatV0.hs+4−4 test/Streamly/Test/Data/Serialize/CompatV1.hs+4−4 test/Streamly/Test/Data/Serialize/TH.hs+2−2 test/Streamly/Test/Data/Stream/Concurrent.hs+1−1 test/Streamly/Test/Data/Unbox.hs+47−50 test/Streamly/Test/Data/Unbox/TH.hs+11−10 test/Streamly/Test/FileSystem/Event/Common.hs+1−1 test/Streamly/Test/FileSystem/Handle.hs+2−2 test/Streamly/Test/Prelude/Serial.hs+3−3 test/Streamly/Test/Serialize/Serializable.hs+2−2 test/Streamly/Test/Unicode/Char.hs+1−1 test/Streamly/Test/Unicode/Stream.hs+1−1 test/lib/Streamly/Test/Prelude/Common.hs+1−1 test/streamly-tests.cabal+8−7
.circleci/config.yml+73−277
@@ -1,7 +1,7 @@
11version: 2.1
22
33#-----------------------------------------------------------------------------
4−# packcheck-0.4.2
4+# packcheck-0.7.0
55# Packcheck global environment variables
66#-----------------------------------------------------------------------------
77
@@ -11,6 +11,8 @@ env: &env
1111 # Common options
1212 # ------------------------------------------------------------------------
1313 # GHC_OPTIONS: "-Werror"
14+ # For updating see: https://downloads.haskell.org/~ghcup/
15+ GHCUP_VERSION: 0.1.20.0
1416 CABAL_REINIT_CONFIG: "y"
1517 LC_ALL: "C.UTF-8"
1618
@@ -22,7 +24,6 @@ env: &env
2224 # DISABLE_DOCS: "y"
2325 # DISABLE_SDIST_BUILD: "y"
2426 # DISABLE_DIST_CHECKS: "y"
25− ENABLE_INSTALL: "y"
2627
2728 # ------------------------------------------------------------------------
2829 # stack options
@@ -38,15 +39,12 @@ env: &env
3839 # cabal options
3940 # ------------------------------------------------------------------------
4041 CABAL_CHECK_RELAX: "y"
41− CABAL_NO_SANDBOX: "y"
42− CABAL_HACKAGE_MIRROR: "hackage.haskell.org:http://hackage.fpcomplete.com"
4342 CABAL_BUILD_OPTIONS: "-j1 --flag limit-build-mem"
4443
4544 # ------------------------------------------------------------------------
4645 # Where to find the required tools
4746 # ------------------------------------------------------------------------
48− PATH: /opt/ghc/bin:/opt/ghcjs/bin:/sbin:/usr/sbin:/bin:/usr/bin
49− TOOLS_DIR: /opt
47+ PATH: /sbin:/usr/sbin:/bin:/usr/bin
5048
5149 # ------------------------------------------------------------------------
5250 # Location of packcheck.sh (the shell script invoked to perform CI tests ).
@@ -59,15 +57,15 @@ env: &env
5957 # If you have not committed packcheck.sh in your repo at PACKCHECK
6058 # then it is automatically pulled from this URL.
6159 PACKCHECK_GITHUB_URL: "https://raw.githubusercontent.com/composewell/packcheck"
62− PACKCHECK_GITHUB_COMMIT: "a302772c8f3791c477edd1764b82f8d1f5b22fdf"
60+ PACKCHECK_GITHUB_COMMIT: "f6cfd7b7a33b4bf6fd63cce6a3fb3d26f5031403"
6361
6462executors:
6563 amd64-executor:
6664 docker:
67− - image: ubuntu:xenial
65+ - image: ubuntu:latest
6866 x86-executor:
6967 docker:
70− - image: i386/ubuntu:eoan
68+ - image: i386/ubuntu:focal
7169
7270#-----------------------------------------------------------------------------
7371# Common utility stuff, not to be modified usually
@@ -79,38 +77,50 @@ preinstall: &preinstall
7977 # required for https/cache save and restore
8078 apt-get install -y ca-certificates
8179
82− # required to (re)generate the configure script
83− apt-get install -y autoconf
84−
85− # For ghc and cabal-install packages from hvr's ppa
80+ # Alternative way of installing ghc and cabal, directly from
81+ # haskell.org instead of using ghcup. NOTE: this is for Debian
82+ # only and is debian release specific.
8683 # gnupg is required for apt-key to work
87− apt-get install -y gnupg
88− apt-get install -y apt-transport-https
89− apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA3CBA3FFE22B574
90− apt-get install -y software-properties-common
91− add-apt-repository -y ppa:hvr/ghc
92− # echo "deb http://downloads.haskell.org/debian stretch main" >> /etc/apt/sources.list
93− echo "deb-src http://ppa.launchpad.net/hvr/ghc/ubuntu xenial main" >> /etc/apt/sources.list
94− apt-get update
84+ #apt-get install -y gnupg
85+ #apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA3CBA3FFE22B574
86+ #echo "deb http://downloads.haskell.org/debian buster main" >> /etc/apt/sources.list
87+ #apt-get update
9588
96− # required for packcheck
97− apt-get install -y curl
89+ # required for outbound https for stack and for stack setup
90+ apt-get install -y netbase xz-utils make
91+ apt-get install -y zlib1g-dev
9892
99− # required for circleci.
100− apt-get install -y libcurl4-gnutls-dev
101− apt-get install -y git
93+ # For ghcup to install ghc
94+ if test -n "$GHCUP_VERSION"
95+ then
96+ apt-get install -y gcc
97+ apt-get install -y g++
98+ fi
99+
100+ # libgmp required by ghc for linking
101+ apt-get install -y libgmp-dev
102102 apt-get install -y libtinfo-dev
103103
104− # required for outbound https for stack and for stack setup
105− apt-get install -y netbase xz-utils make
104+ # Required by cabal when git URL is specified in project file
105+ apt-get install -y git
106+
107+ # Required for and by packcheck
108+ apt-get install -y curl
106109
107−project-preinstall: &project-preinstall
108− run: |
109110 # Get packcheck if needed
110− CURL=$(which curl)
111− PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
112− if test ! -e "$PACKCHECK"; then $CURL -sL -o "$PACKCHECK" $PACKCHECK_URL; fi;
113− chmod +x $PACKCHECK
111+ if test ! -e "$PACKCHECK"
112+ then
113+ if test -z "$PACKCHECK_GITHUB_COMMIT"
114+ then
115+ die "PACKCHECK_GITHUB_COMMIT is not specified."
116+ fi
117+ PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
118+ curl --fail -sL -o "$PACKCHECK" $PACKCHECK_URL || exit 1
119+ chmod +x $PACKCHECK
120+ elif test ! -x "$PACKCHECK"
121+ then
122+ chmod +x $PACKCHECK
123+ fi
114124
115125restore: &restore
116126 # Needs to happen after installing ca-certificates
@@ -126,271 +136,71 @@ save: &save
126136 save_cache:
127137 key: v1-{{ .Environment.CIRCLE_JOB }}-{{ .Branch }}
128138 paths:
129− - ~/.cabal
130− - ~/.ghc
131139 - ~/.local
140+ - ~/.cabal
132141 - ~/.stack
142+ - ~/.ghcup
133143
134144#-----------------------------------------------------------------------------
135145# Build matrix
136146#-----------------------------------------------------------------------------
137147
138148jobs:
139− cabal-ghc-8_10_4:
149+ cabal-ghc-8_6_5:
140150 <<: *env
141151 executor: amd64-executor
142152 steps:
143− - *preinstall
144153 - checkout
145− - *project-preinstall
146154 - *restore
155+ - *preinstall
147156 - run:
148157 environment:
158+ GHCVER: "8.6.5"
159+ CABAL_VERSION: "3.6.2.0"
149160 CABAL_PROJECT: "cabal.project"
150− DISABLE_TEST: "yes"
151− DISABLE_BENCH: "yes"
152161 DISABLE_SDIST_BUILD: "yes"
162+ CABAL_BUILD_OPTIONS: "--flag debug --flag -opt"
153163 command: |
154− apt-get install -y ghc-8.10.4
155− apt-get install -y cabal-install-3.4
156164 bash -c "$PACKCHECK cabal"
157165 - *save
158− stack-ghc-8_8:
166+ cabal-ghc-9_8_1-docspec:
159167 <<: *env
160168 executor: amd64-executor
161169 steps:
162− - *preinstall
163170 - checkout
164− - *project-preinstall
165171 - *restore
166− - run:
167− name: GHC 8.8 + stack lts-16.12
168− environment:
169− BUILD: "stack"
170− RESOLVER: "lts-16.12"
171− SDIST_OPTIONS: "--ignore-check"
172− STACK_BUILD_OPTIONS: "--flag streamly-benchmarks:-opt"
173− command: |
174− apt-get build-dep -y ghc-8.8.4
175− apt-get install -y cabal-install-3.2
176− bash -c "$PACKCHECK $BUILD"
177− - *save
178− cabal-ghc-8_6_5:
179− <<: *env
180− executor: amd64-executor
181− steps:
182172 - *preinstall
183− - checkout
184− - *project-preinstall
185− - *restore
186− - run: |
187− apt-get install -y ghc-8.6.3
188− apt-get install -y cabal-install-2.4
189− bash -c "$PACKCHECK cabal-v2"
190− - *save
191− cabal-ghc-8_4_4:
192− <<: *env
193− executor: x86-executor
194− steps:
195− - *preinstall
196− - checkout
197− - *project-preinstall
198− - *restore
199− - run:
200− name: GHC 8.4.4 + x86 + debug
201− environment:
202− BUILD: "cabal-v2"
203− GHCVER: "8.4.4"
204− CABAL_BUILD_OPTIONS: "--flag debug --flag limit-build-mem"
205− command: |
206− apt-get install -y ghc-8.4.4
207− apt-get install -y cabal-install-3.2
208− bash -c "$PACKCHECK $BUILD"
209− - *save
210− cabal-ghc-8_2_2:
211− <<: *env
212− executor: amd64-executor
213− steps:
214− - *preinstall
215− - checkout
216− - *project-preinstall
217− - *restore
218173 - run:
219174 environment:
220− BUILD: "cabal-v2"
221− DISABLE_TEST: "yes"
222− DISABLE_BENCH: "yes"
223− DISABLE_DOCS: "yes"
224− DISABLE_SDIST_BUILD: "yes"
225− DISABLE_DIST_CHECKS: "yes"
175+ GHCVER: "9.8.1"
176+ CABAL_VERSION: "3.10.1.0"
177+ CABAL_PROJECT: "cabal.project.doctest"
178+ DISABLE_SDIST_BUILD: "y"
179+ DISABLE_TEST: "y"
180+ DISABLE_BENCH: "y"
181+ DISABLE_DOCS: "y"
182+ ENABLE_DOCSPEC: "y"
183+ DOCSPEC_URL: https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz
184+ DOCSPEC_OPTIONS: "--timeout 60 --check-properties --property-variables xs"
226185 command: |
227− apt-get install -y ghc-8.2.2
228− apt-get install -y cabal-install-3.2
229− bash -c "$PACKCHECK $BUILD"
230− - *save
231− cabal-ghc-8_0_2:
232− <<: *env
233− executor: amd64-executor
234− steps:
235− - *preinstall
236− - checkout
237− - *project-preinstall
238− - *restore
239− - run:
240− environment:
241− BUILD: "cabal-v2"
242− DISABLE_TEST: "yes"
243− DISABLE_BENCH: "yes"
244− DISABLE_DOCS: "yes"
245− DISABLE_SDIST_BUILD: "yes"
246− DISABLE_DIST_CHECKS: "yes"
247− command: |
248− apt-get install -y ghc-8.0.2
249− apt-get install -y cabal-install-3.2
250− bash -c "$PACKCHECK $BUILD"
251− - *save
252− cabal-ghc-7_10_3:
253− <<: *env
254− executor: amd64-executor
255− steps:
256− - *preinstall
257− - checkout
258− - *project-preinstall
259− - *restore
260− - run: |
261− apt-get install -y ghc-7.10.3
262− apt-get install -y cabal-install-2.4
263− bash -c "$PACKCHECK cabal-v2"
186+ bash -c "$PACKCHECK cabal"
264187 - *save
265− stack-ghc-8_4:
188+ hlint-trailing-spaces:
266189 <<: *env
267190 executor: amd64-executor
268191 steps:
269− - *preinstall
270192 - checkout
271− - *project-preinstall
272193 - *restore
273− - run: |
274− bash -c "$PACKCHECK stack RESOLVER=lts-12"
275− - *save
276− stack-ghc-8_2:
277− <<: *env
278− executor: amd64-executor
279− steps:
280194 - *preinstall
281− - checkout
282− - *project-preinstall
283− - *restore
284− - run: |
285− bash -c "$PACKCHECK stack RESOLVER=lts-11"
286− - *save
287− cabal-ghcjs-8_4:
288− <<: *env
289− executor: amd64-executor
290− steps:
291− - *preinstall
292− - checkout
293− - *project-preinstall
294− - *restore
295195 - run:
296− name: GHCJS 8.4+no-test+no-docs
196+ name: hlint and trailing spaces
297197 environment:
298− BUILD: cabal-v2
299− ENABLE_GHCJS: "yes"
300− DISABLE_TEST: "yes"
301− ENABLE_INSTALL: ""
302− DISABLE_DOCS: "yes"
303198 DISABLE_SDIST_BUILD: "yes"
304− command: |
305− add-apt-repository -y ppa:hvr/ghcjs
306− add-apt-repository -y 'deb https://deb.nodesource.com/node_11.x xenial main'
307− curl -sSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -
308− apt-get update
309− apt-get install -y cabal-install-3.2
310− apt-get install -y nodejs
311− apt-get install -y ghcjs-8.4
312− cabal user-config update -a "jobs: 1"
313− bash -c "$PACKCHECK $BUILD"
314− - *save
315− coveralls-ghc-8_2_2:
316− <<: *env
317− executor: amd64-executor
318− steps:
319− - *preinstall
320− - checkout
321− - *project-preinstall
322− - *restore
323− - run:
324− environment:
325− BUILD: cabal
326− COVERALLS_OPTIONS: "--repo-token=\"$REPO_TOKEN\" --coverage-mode=StrictlyFullLines --exclude-dir=test test"
327− GHC_OPTIONS: "-DCOVERAGE_BUILD"
328− command: |
329− apt-get install -y ghc-8.2.2
330− apt-get install -y cabal-install-2.4
331− bash -c "$PACKCHECK cabal-v1"
332− - *save
333− coveralls-ghc-8_8_3:
334− <<: *env
335− executor: amd64-executor
336− steps:
337− - *preinstall
338− - checkout
339− - *project-preinstall
340− - *restore
341− - run:
342− environment:
343− BUILD: cabal-v2
344− COVERALLS_OPTIONS: "--repo-token=\"$REPO_TOKEN\" --coverage-mode=StrictlyFullLines --exclude-dir=test"
345− CABAL_PROJECT: "cabal.project.coverage"
346− CABAL_BUILD_TARGETS: ""
347− command: |
348− apt-get install -y ghc-8.8.3
349− apt-get install -y cabal-install-3.0
350− bash -c "$PACKCHECK cabal-v2"
351− no_output_timeout: 25m
352− - *save
353− coveralls-ghc-8_10_2:
354− <<: *env
355− executor: amd64-executor
356− steps:
357− - *preinstall
358− - checkout
359− - *project-preinstall
360− - *restore
361− - run:
362− name: Build and test with inspection, coverage, Werror
363− command: |
364− apt-get install -y ghc-8.10.2
365− apt-get install -y cabal-install-3.2
366− # Run tests with coverage
367− cabal update
368− cabal user-config update -a "jobs: 1"
369− bin/test.sh --coverage --raw
370− # Upload results to coveralls.io
371− PATH=$HOME/.cabal/bin:$PATH
372− export PATH
373− which hpc-coveralls 2>/dev/null || cabal install --project-file cabal.project.hpc-coveralls hpc-coveralls
374− hpc-coveralls --repo-token="$REPO_TOKEN" --coverage-mode=StrictlyFullLines
375− no_output_timeout: 25m
376− - *save
377− hlint-src:
378− <<: *env
379− executor: amd64-executor
380− steps:
381− - *preinstall
382− - checkout
383− - *project-preinstall
384− - *restore
385− - run:
386− name: Hlint src
387− environment:
388− BUILD: hlint
199+ HLINT_VERSION: 3.6.1
389200 HLINT_OPTIONS: lint
390− DISABLE_SDIST_BUILD: "yes"
391201 HLINT_TARGETS: core/src src test benchmark
392202 command: |
393− bash -c "$PACKCHECK $BUILD" || exit 1
203+ bash -c "$PACKCHECK hlint" || exit 1
394204 echo "Checking trailing spaces..."
395205 count=$(find . -name "*.hs" -exec grep -H '\ $' {} \; | tee /dev/tty | wc -l)
396206 exit $count
@@ -400,23 +210,9 @@ workflows:
400210 version: 2
401211 build:
402212 jobs:
403− - cabal-ghc-8_10_4:
404− name: GHC 8.10.4 + sdist + no-test + no-bench
405− #- cabal-ghc-8.6.5
406− #- cabal-ghc-8_4_4:
407− # name: GHC 8.4.4 + x86 + debug
408− #- cabal-ghc-8_2_2:
409− # name: GHC 8.2.2 + no-test + no-bench + no-docs
410− #- cabal-ghc-8_0_2:
411− # name: GHC 8.0.2 + no-test + no-bench + no-docs
412− #- cabal-ghc-7.10.3
413− # - cabal-ghcjs-8_4:
414− # name: GHCJS 8.4 + no-test + no-docs
415− #- stack-ghc-8.4
416− #- stack-ghc-8.2
417− #- coveralls-ghc-8.2.2
418− #- coveralls-ghc-8.8.3
419− #- coveralls-ghc-8_10_2:
420− # name: GHC 8.10.2 + inspection + coverage + Werror
421− - hlint-src:
422− name: Hlint src + Trailing Spaces
213+ #- cabal-ghc-8_6_5:
214+ # name: 8.6.5-debug-unoptimized
215+ - cabal-ghc-9_8_1-docspec:
216+ name: ghc-9.8.1-docspec
217+ - hlint-trailing-spaces:
218+ name: hlint and trailing spaces
.github/workflows/haskell.yml+61−60
@@ -13,9 +13,15 @@ jobs:
1313 # packcheck environment variables
1414 LC_ALL: C.UTF-8
1515 BUILD: ${{ matrix.build }}
16+ # For updating see: https://downloads.haskell.org/~ghcup/
17+ GHCUP_VERSION: 0.1.20.0
18+ DOCSPEC_URL: https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz
19+ DOCSPEC_OPTIONS: "--timeout 60 --check-properties --property-variables xs"
20+ GHCUP_GHC_OPTIONS: ${{ matrix.ghcup_ghc_options }}
1621 GHCVER: ${{ matrix.ghc_version }}
1722 CABALVER: ${{ matrix.cabal_version }}
1823 DISABLE_DOCS: ${{ matrix.disable_docs }}
24+ ENABLE_DOCSPEC: ${{ matrix.enable_docspec }}
1925 DISABLE_TEST: ${{ matrix.disable_test }}
2026 DISABLE_BENCH: ${{ matrix.disable_bench }}
2127 DISABLE_DIST_CHECKS: ${{ matrix.disable_dist_checks }}
@@ -39,14 +45,18 @@ jobs:
3945 STACK_BUILD_OPTIONS: ${{ matrix.stack_build_options }}
4046
4147 # packcheck location and revision
42− PACKCHECK_LOCAL_PATH: "./packcheck.sh"
48+ PACKCHECK: "./packcheck.sh"
4349 PACKCHECK_GITHUB_URL: "https://raw.githubusercontent.com/composewell/packcheck"
44− PACKCHECK_GITHUB_COMMIT: "a302772c8f3791c477edd1764b82f8d1f5b22fdf"
50+ PACKCHECK_GITHUB_COMMIT: "v0.7.0"
4551
4652 # Pull token from "secrets" setting of the github repo
4753 COVERALLS_TOKEN: ${{ secrets.COVERALLS_TOKEN }}
4854 COVERAGE: ${{ matrix.coverage }}
49− DOCTEST: ${{ matrix.doctest }}
55+
56+ # hlint
57+ HLINT_VERSION: 3.6.1
58+ HLINT_OPTIONS: "lint"
59+ HLINT_TARGETS: "core/src src test benchmark"
5060
5161 # Subdir
5262 SUBDIR: ${{ matrix.subdir }}
@@ -59,7 +69,7 @@ jobs:
5969 # The order is important to optimize fail-fast.
6070 name:
6171 - 9.8.1-Werror
62− - 9.2.8-docspec
72+ # - 9.8.1-docspec
6373 # - 8.10.7-coverage
6474
6575 # Note: if cabal.project is not specified benchmarks and tests won't
@@ -68,9 +78,19 @@ jobs:
6878 include:
6979 - name: head
7080 ghc_version: head
81+ # The URL may change, to find a working URL go to https://gitlab.haskell.org/ghc/ghc/-/jobs/
82+ # Find a debian10/11/12 job, click on a passed/failed status, at the
83+ # end of the output you will find the tar.xz name, put that tar
84+ # name after "raw/", and put the job name after "job=".
85+ # Also see https://github.com/mpickering/ghc-artefact-nix/blob/master/gitlab-artifact.nix
86+ #
87+ # May also use ghcup for installing ghc head version, use the
88+ # version "LatestNightly", and the following config:
89+ # ghcup config add-release-channel https://ghc.gitlab.haskell.org/ghcup-metadata/ghcup-nightlies-0.0.7.yaml
90+ ghcup_ghc_options: "-u https://gitlab.haskell.org/ghc/ghc/-/jobs/artifacts/master/raw/ghc-x86_64-linux-deb10-int_native-validate.tar.xz?job=x86_64-linux-deb10-int_native-validate"
7191 runner: ubuntu-latest
7292 build: cabal
73− cabal_version: 3.10.1.0
93+ cabal_version: 3.10.2.0
7494 cabal_project: cabal.project.ghc-head
7595 disable_sdist_build: "y"
7696 ignore_error: true
@@ -83,6 +103,18 @@ jobs:
83103 disable_sdist_build: "y"
84104 cabal_project: cabal.project.Werror
85105 ignore_error: false
106+ #- name: 9.8.1-docspec
107+ # ghc_version: 9.8.1
108+ # runner: ubuntu-latest
109+ # build: cabal
110+ # cabal_version: 3.10.1.0
111+ # cabal_project: cabal.project.doctest
112+ # disable_test: "y"
113+ # disable_bench: "y"
114+ # disable_docs: "y"
115+ # enable_docspec: "y"
116+ # disable_sdist_build: "y"
117+ # ignore_error: false
86118 - name: 9.8.1-fusion-inspection
87119 ghc_version: 9.8.1
88120 runner: ubuntu-latest
@@ -116,14 +148,6 @@ jobs:
116148 cabal_version: 3.6.2.0
117149 disable_sdist_build: "y"
118150 ignore_error: false
119− - name: 9.2.8-docspec
120− ghc_version: 9.2.8
121− runner: ubuntu-latest
122− doctest: "y"
123− cabal_version: 3.6.2.0
124− cabal_project: cabal.project
125− disable_sdist_build: "y"
126− ignore_error: false
127151 - name: 9.0.2-streamly-sdist
128152 ghc_version: 9.0.2
129153 runner: ubuntu-latest
@@ -170,66 +194,44 @@ jobs:
170194 cabal_build_options: "--flag debug --flag -opt"
171195 disable_sdist_build: "y"
172196 ignore_error: false
197+ - name: hlint
198+ build: hlint
199+ runner: ubuntu-latest
200+ ignore_error: true
173201
174202 steps:
175203 - uses: actions/checkout@v2
176204
177205 - uses: actions/cache@v1
178− name: Cache ~/.cabal
206+ name: Cache common directories
179207 with:
180− path: ~/.cabal
208+ path: |
209+ ~/.local
210+ ~/.cabal
211+ ~/.stack
212+ ~/.ghcup
181213 # Bump the key version to clear the cache
182214 key: ${{ runner.os }}-${{ matrix.ghc_version }}-cabal-v2
183215
184216 - name: Download packcheck
185217 run: |
186− # Get packcheck if needed
187− CURL=$(which curl)
188− PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
189− if test ! -e "$PACKCHECK_LOCAL_PATH"; then $CURL -sL -o "$PACKCHECK_LOCAL_PATH" $PACKCHECK_URL; fi;
190− chmod +x $PACKCHECK_LOCAL_PATH
191−
192− os=$(uname -s -m)
193− case "$os" in
194− "Linux x86_64") GHCUP_ARCH="x86_64-linux" ;;
195− "Darwin x86_64") GHCUP_ARCH="x86_64-apple-darwin" ;;
196− *) echo "Unknown OS/Arch: $os"; exit 1;;
197− esac
198−
199− # Check available versions here: https://downloads.haskell.org/~ghcup/
200− GHCUP_VER=0.1.18.0
201− $CURL -sL -o ./ghcup https://downloads.haskell.org/~ghcup/$GHCUP_VER/${GHCUP_ARCH}-ghcup-$GHCUP_VER
202− chmod +x ./ghcup
203− if test "$GHCVER" = "head"
218+ if test ! -e "$PACKCHECK"
204219 then
205− # The URL may change, to find a working URL go to https://gitlab.haskell.org/ghc/ghc/-/jobs/
206− # Find a debian10 job, click on a passed/failed job, at the
207− # end of the output you will find the tar.xz name, put that after
208− # "raw/", and put the job name after "job=".
209− # Also see https://github.com/mpickering/ghc-artefact-nix/blob/master/gitlab-artifact.nix
210− ./ghcup install ghc -u https://gitlab.haskell.org/ghc/ghc/-/jobs/artifacts/master/raw/ghc-x86_64-linux-deb10-int_native-validate.tar.xz?job=x86_64-linux-deb10-int_native-validate head
211− else
212− ./ghcup install ghc $GHCVER
220+ if test -z "$PACKCHECK_GITHUB_COMMIT"
221+ then
222+ die "PACKCHECK_GITHUB_COMMIT is not specified."
223+ fi
224+ PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
225+ curl --fail -sL -o "$PACKCHECK" $PACKCHECK_URL || exit 1
226+ chmod +x $PACKCHECK
227+ elif test ! -x "$PACKCHECK"
228+ then
229+ chmod +x $PACKCHECK
213230 fi
214− ./ghcup install cabal $CABALVER
215− ./ghcup set ghc $GHCVER
216− ./ghcup set cabal $CABALVER
217231
218232 - name: Run tests
219233 run: |
220− if test -n "$DOCTEST"
221− then
222− mkdir -p $HOME/.cabal/bin
223− curl -sL https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz > cabal-docspec.xz
224− echo '0829bd034fba901cbcfe491d98ed8b28fd54f9cb5c91fa8e1ac62dc4413c9562 cabal-docspec.xz' | sha256sum -c -
225− xz -d < cabal-docspec.xz > $HOME/.cabal/bin/cabal-docspec
226− rm -f cabal-docspec.xz
227− chmod a+x $HOME/.cabal/bin/cabal-docspec
228− cabal update
229− cabal build --project-file cabal.project.doctest all
230− $HOME/.cabal/bin/cabal-docspec --version
231− $HOME/.cabal/bin/cabal-docspec --timeout 60 --check-properties --property-variables xs
232− elif test -n "$COVERAGE"
234+ if test -n "$COVERAGE"
233235 then
234236 # Run tests with coverage
235237 cabal update
@@ -242,16 +244,15 @@ jobs:
242244 # XXX Uncomment this and fix it properly later
243245 # hpc-coveralls --repo-token="$COVERALLS_TOKEN" --coverage-mode=StrictlyFullLines
244246 else
245− export TOOLS_DIR=/opt
246247 # /usr/local/opt/curl/bin for macOS
247248 export PATH=$HOME/.local/bin:$HOME/.ghcup/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/opt/curl/bin
248249 if test -n "$SUBDIR"
249250 then
250− mv "$PACKCHECK_LOCAL_PATH" "$SUBDIR/packcheck.sh"
251+ mv "$PACKCHECK" "$SUBDIR/packcheck.sh"
251252 # This is required as dist-newstyle will be created on the top level
252253 # directory as it is considered the working dir.
253254 rm cabal.project
254255 cd "$SUBDIR"
255256 fi
256− bash -c "$PACKCHECK_LOCAL_PATH $BUILD"
257+ bash -c "$PACKCHECK $BUILD"
257258 fi
.hlint.ignore+5
@@ -2,6 +2,9 @@ core/src/Streamly/Internal/Data/Stream/StreamK/Type.hs
22core/src/Streamly/Internal/Data/Pipe/Type.hs
33core/src/Streamly/Internal/Unicode/Stream.hs
44src/Streamly/Internal/Data/SmallArray/Type.hs
5+src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs
6+src/Streamly/Internal/Data/Stream/Serial.hs
7+src/Streamly/Internal/Data/Stream/Zip.hs
58test/Streamly/Test/Data/Array.hs
69test/Streamly/Test/Data/Parser.hs
710test/Streamly/Test/Data/ParserK.hs
@@ -10,6 +13,7 @@ test/Streamly/Test/Data/Unfold.hs
1013test/Streamly/Test/FileSystem/Event.hs
1114test/Streamly/Test/Prelude/Concurrent.hs
1215test/Streamly/Test/Data/Stream/Concurrent.hs
16+test/lib/Streamly/Test/Prelude/Common.hs
1317test/Streamly/Test/Prelude/Fold.hs
1418test/Streamly/Test/Prelude/Rate.hs
1519test/Streamly/Test/Prelude/Serial.hs
@@ -27,6 +31,7 @@ benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs
2731benchmark/Streamly/Benchmark/Data/Unfold.hs
2832benchmark/Streamly/Benchmark/FileSystem/Handle.hs
2933benchmark/Streamly/Benchmark/Prelude/Async.hs
34+benchmark/Streamly/Benchmark/Prelude/Concurrent.hs
3035benchmark/Streamly/Benchmark/Prelude/Merge.hs
3136benchmark/Streamly/Benchmark/Prelude/Parallel.hs
3237benchmark/Streamly/Benchmark/Prelude/Rate.hs
benchmark/NanoBenchmarks.hs+1−1
@@ -18,7 +18,7 @@ import qualified Streamly.Data.Fold as FL
1818import qualified Streamly.Internal.Data.Stream as S
1919import qualified Streamly.Internal.Data.Stream.IsStream as Internal
2020
21−import Gauge
21+import Test.Tasty.Bench
2222import System.Random
2323
2424maxValue :: Int
benchmark/Streamly/Benchmark/Data/Array/CommonImports.hs+1−1
@@ -9,7 +9,7 @@ import qualified Streamly.Data.Fold as Fold
99import qualified Streamly.Data.Stream as S
1010import qualified Streamly.Internal.Data.Stream as Stream
1111
12−import Gauge
12+import Test.Tasty.Bench
1313import Streamly.Benchmark.Common hiding (benchPureSrc)
1414import qualified Stream.Common as P
1515
benchmark/Streamly/Benchmark/Data/Array/Stream.hs+3−3
@@ -39,7 +39,7 @@ import Prelude hiding ()
3939
4040import qualified Streamly.Data.Stream as Stream
4141import qualified Streamly.Internal.Data.Array as Array
42−import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
42+import qualified Streamly.Internal.Data.Array.Stream as ArrayStream
4343import qualified Streamly.Internal.Data.Fold as Fold
4444import qualified Streamly.Internal.Data.Parser as Parser
4545import qualified Streamly.Internal.Data.Stream as Stream
@@ -47,13 +47,13 @@ import qualified Streamly.Internal.Data.StreamK as StreamK
4747import qualified Streamly.Internal.FileSystem.Handle as Handle
4848import qualified Streamly.Internal.Unicode.Stream as Unicode
4949
50−import Gauge hiding (env)
50+import Test.Tasty.Bench hiding (env)
5151import Streamly.Benchmark.Common
5252import Streamly.Benchmark.Common.Handle
5353import Control.Monad.IO.Class (MonadIO)
5454
5555#ifdef INSPECTION
56−import Streamly.Internal.Data.Unbox (Unbox)
56+import Streamly.Internal.Data.MutByteArray (Unbox)
5757import Streamly.Internal.Data.Stream (Step(..))
5858import Test.Inspection
5959#endif
benchmark/Streamly/Benchmark/Data/Fold.hs+3−3
@@ -41,7 +41,7 @@ import qualified Streamly.Internal.Data.Unfold as Unfold
4141import qualified Streamly.Internal.Data.Pipe as Pipe
4242import qualified Streamly.Internal.Data.Stream as Stream
4343
44−import Gauge
44+import Test.Tasty.Bench
4545import Streamly.Benchmark.Common
4646import Prelude hiding (all, any, take, unzip, sequence_, filter)
4747
@@ -269,13 +269,13 @@ unzip = Stream.fold $ FL.lmap (\a -> (a, a)) (FL.unzip FL.sum FL.length)
269269{-# INLINE unzipWithFstM #-}
270270unzipWithFstM :: Monad m => Stream m Int -> m (Int, Int)
271271unzipWithFstM = do
272− let f = \a -> return (a + 1, a)
272+ let f a = return (a + 1, a)
273273 Stream.fold (FL.unzipWithFstM f FL.sum FL.length)
274274
275275{-# INLINE unzipWithMinM #-}
276276unzipWithMinM :: Monad m => Stream m Int -> m (Int, Int)
277277unzipWithMinM = do
278− let f = \a -> return (a + 1, a)
278+ let f a = return (a + 1, a)
279279 Stream.fold (FL.unzipWithMinM f FL.sum FL.length)
280280
281281-------------------------------------------------------------------------------
benchmark/Streamly/Benchmark/Data/Fold/Window.hs+1−1
@@ -13,7 +13,7 @@ import qualified Streamly.Internal.Data.Ring as Ring
1313import qualified Streamly.Internal.Data.Stream as Stream
1414
1515import Streamly.Benchmark.Common
16−import Gauge
16+import Test.Tasty.Bench
1717
1818{-# INLINE source #-}
1919source :: (Monad m, Num a, Stream.Enumerable a) =>
benchmark/Streamly/Benchmark/Data/MutArray.hs+1−1
@@ -49,7 +49,7 @@ import qualified Streamly.Internal.Data.MutArray as MArray
4949import qualified Streamly.Internal.Data.Fold as Fold
5050import qualified Streamly.Internal.Data.Stream as Stream
5151
52−import Gauge
52+import Test.Tasty.Bench
5353import Streamly.Benchmark.Common hiding (benchPureSrc)
5454
5555type Stream = MutArray
benchmark/Streamly/Benchmark/Data/Parser.hs+3−3
@@ -49,7 +49,7 @@ import qualified Streamly.Data.Stream as Stream
4949import qualified Streamly.Internal.Data.Producer as Producer
5050import qualified Streamly.Internal.Data.Stream as Stream
5151
52−import Gauge hiding (env)
52+import Test.Tasty.Bench hiding (env)
5353import Streamly.Benchmark.Common
5454import Streamly.Benchmark.Common.Handle
5555
@@ -348,7 +348,7 @@ split_ value =
348348-- PR.dropWhile (<= (value * 1 `div` 4)) *> PR.die "alt"
349349{-# INLINE takeWhileFail #-}
350350takeWhileFail :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
351−takeWhileFail predicate (Fold fstep finitial fextract) =
351+takeWhileFail predicate (Fold fstep finitial _ ffinal) =
352352 Parser step initial extract
353353
354354 where
@@ -369,7 +369,7 @@ takeWhileFail predicate (Fold fstep finitial fextract) =
369369 Fold.Done b -> Done 0 b
370370 else return $ Error "fail"
371371
372− extract s = fmap (Done 0) (fextract s)
372+ extract s = fmap (Done 0) (ffinal s)
373373
374374{-# INLINE alt2 #-}
375375alt2 :: Monad m
benchmark/Streamly/Benchmark/Data/ParserK.hs+3−3
@@ -48,7 +48,7 @@ import qualified Streamly.Data.Parser as PRD
4848import qualified Streamly.Internal.Data.ParserK as PR
4949import qualified Streamly.Internal.Data.StreamK as StreamK
5050
51−import Gauge
51+import Test.Tasty.Bench
5252import Streamly.Benchmark.Common
5353
5454-------------------------------------------------------------------------------
@@ -213,7 +213,7 @@ sequence_ value =
213213
214214{-# INLINE takeWhileFailD #-}
215215takeWhileFailD :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
216−takeWhileFailD predicate (Fold fstep finitial fextract) =
216+takeWhileFailD predicate (Fold fstep finitial _ ffinal) =
217217 Parser step initial extract
218218
219219 where
@@ -234,7 +234,7 @@ takeWhileFailD predicate (Fold fstep finitial fextract) =
234234 Fold.Done b -> Done 0 b
235235 else return $ Error "fail"
236236
237− extract s = fmap (Done 0) (fextract s)
237+ extract s = fmap (Done 0) (ffinal s)
238238
239239{-# INLINE takeWhileFail #-}
240240takeWhileFail :: CONSTRAINT =>
benchmark/Streamly/Benchmark/Data/Ring/Unboxed.hs+1−1
@@ -16,7 +16,7 @@ import qualified Streamly.Internal.Data.Array as Array
1616import qualified Streamly.Internal.Data.Ring as Ring
1717import qualified Data.Foldable as P
1818
19−import Gauge
19+import Test.Tasty.Bench
2020import Streamly.Benchmark.Common
2121
2222import Prelude as P
benchmark/Streamly/Benchmark/Data/Serialize.hs+34−48
@@ -1,4 +1,5 @@
11{-# LANGUAGE TemplateHaskell #-}
2+{- HLINT ignore -}
23
34#undef FUSION_CHECK
45#ifdef FUSION_CHECK
@@ -18,31 +19,23 @@ module Main (main) where
1819-------------------------------------------------------------------------------
1920
2021import Control.DeepSeq (NFData(..), deepseq)
22+import Control.Monad (when, void)
2123import GHC.Generics (Generic)
2224import System.Random (randomRIO)
2325#ifndef USE_UNBOX
2426import Test.QuickCheck (Arbitrary, arbitrary)
2527#endif
2628
29+import Streamly.Internal.Data.MutByteArray
30+import qualified Streamly.Internal.Data.MutByteArray as MBA
2731#ifdef USE_UNBOX
2832import Data.Proxy (Proxy(..))
29−import Streamly.Internal.Data.Unbox
3033#else
3134import Control.DeepSeq (force)
3235import Test.QuickCheck (oneof, generate)
33−import Streamly.Internal.Data.Unbox (newBytes, MutableByteArray)
34−import Streamly.Internal.Data.Serialize hiding (encode)
3536#endif
3637
37−#ifdef USE_TH
38−#ifdef USE_UNBOX
39−import Streamly.Internal.Data.Unbox.TH
40−#else
41−import Streamly.Internal.Data.Serialize.TH
42−#endif
43−#endif
44−
45−import Gauge
38+import Test.Tasty.Bench
4639import Streamly.Benchmark.Common
4740
4841#ifndef USE_UNBOX
@@ -52,14 +45,14 @@ import Streamly.Benchmark.Data.Serialize.RecNonCompatible
5245
5346#ifdef USE_UNBOX
5447#define SERIALIZE_CLASS Unbox
55−#define DERIVE_CLASS deriveUnbox
56−#define SERIALIZE_OP pokeByteIndex
57−#define DESERIALIZE_OP peekByteIndex
48+#define DERIVE_CLASS(typ) $(deriveUnbox [d|instance Unbox typ|])
49+#define SERIALIZE_OP pokeAt
50+#define DESERIALIZE_OP peekAt
5851#else
5952#define SERIALIZE_CLASS Serialize
60−#define DERIVE_CLASS deriveSerialize
61−#define SERIALIZE_OP serialize
62−#define DESERIALIZE_OP deserialize
53+#define DERIVE_CLASS(typ) $(deriveSerialize [d|instance Serialize typ|])
54+#define SERIALIZE_OP serializeAt
55+#define DESERIALIZE_OP deserializeAt
6356#endif
6457
6558-------------------------------------------------------------------------------
@@ -72,7 +65,7 @@ data Unit = Unit
7265#ifndef USE_TH
7366instance SERIALIZE_CLASS Unit
7467#else
75−$(DERIVE_CLASS ''Unit)
68+DERIVE_CLASS(Unit)
7669#endif
7770
7871instance NFData Unit where
@@ -87,7 +80,7 @@ data Sum2
8780#ifndef USE_TH
8881instance SERIALIZE_CLASS Sum2
8982#else
90−$(DERIVE_CLASS ''Sum2)
83+DERIVE_CLASS(Sum2)
9184#endif
9285
9386instance NFData Sum2 where
@@ -125,7 +118,7 @@ data Sum25
125118#ifndef USE_TH
126119instance SERIALIZE_CLASS Sum25
127120#else
128−$(DERIVE_CLASS ''Sum25)
121+DERIVE_CLASS(Sum25)
129122#endif
130123
131124instance NFData Sum25 where
@@ -164,7 +157,7 @@ data Product25
164157#ifndef USE_TH
165158instance SERIALIZE_CLASS Product25
166159#else
167−$(DERIVE_CLASS ''Product25)
160+DERIVE_CLASS(Product25)
168161#endif
169162
170163instance NFData Product25 where
@@ -244,7 +237,7 @@ data CustomDT1
244237#ifndef USE_TH
245238instance SERIALIZE_CLASS CustomDT1
246239#else
247−$(DERIVE_CLASS ''CustomDT1)
240+DERIVE_CLASS(CustomDT1)
248241#endif
249242
250243instance NFData CustomDT1 where
@@ -267,19 +260,20 @@ data BinTree a
267260#ifndef USE_TH
268261instance Serialize (BinTree a)
269262#else
270−$(deriveSerialize ''BinTree)
263+$(deriveSerialize [d|instance Serialize a => Serialize (BinTree a)|])
271264#endif
272265
273266instance NFData a => NFData (BinTree a) where
274267 {-# INLINE rnf #-}
275268 rnf (Leaf a) = rnf a `seq` ()
276− rnf (Tree l r) = rnf l `seq` rnf r `seq` ()
269+ rnf (Tree l r) = rnf l `seq` rnf r
277270
271+-- XXX This may not terminate, or could become really large.
278272instance Arbitrary a => Arbitrary (BinTree a) where
279273 arbitrary = oneof [Leaf <$> arbitrary, Tree <$> arbitrary <*> arbitrary]
280274
281275mkBinTree :: (Arbitrary a) => Int -> IO (BinTree a)
282−mkBinTree = go (generate $ arbitrary)
276+mkBinTree = go (generate arbitrary)
283277
284278 where
285279
@@ -297,23 +291,21 @@ getSize :: forall a. SERIALIZE_CLASS a => a -> Int
297291#ifdef USE_UNBOX
298292getSize _ = sizeOf (Proxy :: Proxy a)
299293#else
300−getSize = size 0
294+getSize = addSizeTo 0
301295#endif
302296
303297-------------------------------------------------------------------------------
304298-- Common helpers
305299-------------------------------------------------------------------------------
306300
301+{- HLINT ignore "Eta reduce" -}
307302-- Parts of "f" that are dependent on val will not be optimized out.
308303{-# INLINE loop #-}
309304loop :: Int -> (a -> IO b) -> a -> IO ()
310305loop count f val = go count val
311306 where
312307
313− go n x = do
314− if n > 0
315− then f x >> go (n-1) x
316− else return ()
308+ go n x = when (n > 0) $ f x >> go (n-1) x
317309
318310-- The first arg of "f" is the environment which is not threaded around in the
319311-- loop.
@@ -322,10 +314,7 @@ loopWith :: Int -> (e -> a -> IO b) -> e -> a -> IO ()
322314loopWith count f e val = go count val
323315 where
324316
325− go n x = do
326− if n > 0
327− then f e x >> go (n-1) x
328− else return ()
317+ go n x = when (n > 0) $ f e x >> go (n-1) x
329318
330319benchSink :: NFData b => String -> Int -> (Int -> IO b) -> Benchmark
331320benchSink name times f = bench name (nfIO (randomRIO (times, times) >>= f))
@@ -336,7 +325,7 @@ benchSink name times f = bench name (nfIO (randomRIO (times, times) >>= f))
336325
337326{-
338327{-# INLINE pokeWithSize #-}
339−pokeWithSize :: SERIALIZE_CLASS a => MutableByteArray -> a -> IO ()
328+pokeWithSize :: SERIALIZE_CLASS a => MutByteArray -> a -> IO ()
340329pokeWithSize arr val = do
341330 let n = getSize val
342331 n `seq` SERIALIZE_OP 0 arr val >> return ()
@@ -345,27 +334,27 @@ pokeWithSize arr val = do
345334pokeTimesWithSize :: SERIALIZE_CLASS a => a -> Int -> IO ()
346335pokeTimesWithSize val times = do
347336 let n = getSize val
348− arr <- newBytes n
337+ arr <- MBA.new n
349338 loopWith times pokeWithSize arr val
350339-}
351340
352341{-# INLINE poke #-}
353−poke :: SERIALIZE_CLASS a => MutableByteArray -> a -> IO ()
354−poke arr val = SERIALIZE_OP 0 arr val >> return ()
342+poke :: SERIALIZE_CLASS a => MutByteArray -> a -> IO ()
343+poke arr val = void (SERIALIZE_OP 0 arr val)
355344
356345{-# INLINE pokeTimes #-}
357346pokeTimes :: SERIALIZE_CLASS a => a -> Int -> IO ()
358347pokeTimes val times = do
359348 let n = getSize val
360− arr <- newBytes n
349+ arr <- MBA.new n
361350 loopWith times poke arr val
362351
363352{-# INLINE encode #-}
364353encode :: SERIALIZE_CLASS a => a -> IO ()
365354encode val = do
366355 let n = getSize val
367− arr <- newBytes n
368− SERIALIZE_OP 0 arr val >> return ()
356+ arr <- MBA.new n
357+ void (SERIALIZE_OP 0 arr val)
369358
370359{-# INLINE encodeTimes #-}
371360encodeTimes :: SERIALIZE_CLASS a => a -> Int -> IO ()
@@ -373,7 +362,7 @@ encodeTimes val times = loop times encode val
373362
374363{-# INLINE peek #-}
375364peek :: forall a. (NFData a, SERIALIZE_CLASS a) =>
376− (a, Int) -> MutableByteArray -> IO ()
365+ (a, Int) -> MutByteArray -> IO ()
377366#ifdef USE_UNBOX
378367peek (_val, _) arr = do
379368 (val1 :: a) <- DESERIALIZE_OP 0 arr
@@ -400,7 +389,7 @@ peek (_val, n) arr = do
400389{-# INLINE peekTimes #-}
401390peekTimes :: (NFData a, SERIALIZE_CLASS a) => Int -> a -> Int -> IO ()
402391peekTimes n val times = do
403− arr <- newBytes n
392+ arr <- MBA.new n
404393 _ <- SERIALIZE_OP 0 arr val
405394 loopWith times peek (val, n) arr
406395
@@ -408,7 +397,7 @@ peekTimes n val times = do
408397trip :: forall a. (NFData a, SERIALIZE_CLASS a) => a -> IO ()
409398trip val = do
410399 let n = getSize val
411− arr <- newBytes n
400+ arr <- MBA.new n
412401 _ <- SERIALIZE_OP 0 arr val
413402#ifdef USE_UNBOX
414403 (val1 :: a) <- DESERIALIZE_OP 0 arr
@@ -571,8 +560,6 @@ main = do
571560 let !len = length lInt -- evaluate the list
572561#endif
573562#ifndef FUSION_CHECK
574− -- This can take too much memory/CPU, need to restrict the test
575− -- runQC
576563#ifdef USE_UNBOX
577564 runWithCLIOpts defaultStreamSize allBenchmarks
578565#else
@@ -580,7 +567,6 @@ main = do
580567 `seq` runWithCLIOpts
581568 defaultStreamSize
582569 (allBenchmarks tInt lInt recL recR)
583−
584570#endif
585571#else
586572 -- Enable FUSION_CHECK macro at the beginning of the file
benchmark/Streamly/Benchmark/Data/Serialize/RecCompatible.hs+3−3
@@ -9,10 +9,10 @@ module Streamly.Benchmark.Data.Serialize.RecCompatible
99-- Imports
1010--------------------------------------------------------------------------------
1111
12−import Streamly.Internal.Data.Serialize (Serialize)
12+import Streamly.Internal.Data.MutByteArray (Serialize)
1313import Streamly.Benchmark.Data.Serialize.TH (genLargeRecord)
1414
15−import qualified Streamly.Internal.Data.Serialize.TH as Serialize
15+import qualified Streamly.Internal.Data.MutByteArray as Serialize
1616
1717--------------------------------------------------------------------------------
1818-- Code
@@ -20,5 +20,5 @@ import qualified Streamly.Internal.Data.Serialize.TH as Serialize
2020
2121$(genLargeRecord "RecCompatible" 50)
2222$(Serialize.deriveSerializeWith
23− (Serialize.defaultConfig {Serialize.recordSyntaxWithHeader = True})
23+ (Serialize.encodeRecordFields True)
2424 [d|instance Serialize RecCompatible|])
benchmark/Streamly/Benchmark/Data/Serialize/RecNonCompatible.hs+2−2
@@ -11,11 +11,11 @@ module Streamly.Benchmark.Data.Serialize.RecNonCompatible
1111
1212import Streamly.Benchmark.Data.Serialize.TH (genLargeRecord)
1313
14−import qualified Streamly.Internal.Data.Serialize.TH as Serialize
14+import qualified Streamly.Internal.Data.MutByteArray as Serialize
1515
1616--------------------------------------------------------------------------------
1717-- Code
1818--------------------------------------------------------------------------------
1919
2020$(genLargeRecord "RecNonCompatible" 50)
21−$(Serialize.deriveSerialize ''RecNonCompatible)
21+$(Serialize.deriveSerialize [d|instance Serialize.Serialize RecNonCompatible|])
benchmark/Streamly/Benchmark/Data/Serialize/TH.hs+4−4
@@ -8,7 +8,7 @@ module Streamly.Benchmark.Data.Serialize.TH (genLargeRecord) where
88
99import Language.Haskell.TH
1010
11−import Streamly.Internal.Data.Serialize.TH.Bottom (makeI)
11+import Streamly.Internal.Data.MutByteArray (makeI)
1212import Control.DeepSeq (NFData(..))
1313
1414--------------------------------------------------------------------------------
@@ -18,7 +18,7 @@ import Control.DeepSeq (NFData(..))
1818genLargeRecord :: String -> Int -> Q [Dec]
1919genLargeRecord tyName numFields =
2020 sequence
21− ([ dataD
21+ [ dataD
2222 (pure [])
2323 (mkName tyName)
2424 []
@@ -28,7 +28,7 @@ genLargeRecord tyName numFields =
2828 , mkValueSigDec
2929 , mkValueDec
3030 , nfDataInstance tyName
31− ])
31+ ]
3232
3333 where
3434
@@ -64,7 +64,7 @@ genLargeRecord tyName numFields =
6464 (foldl
6565 (\b a -> [|$(b) $(a)|])
6666 (conE (mkName tyName))
67− (const (conE '()) <$> [0 .. (numFields - 1)])))
67+ (conE '() <$ [0 .. (numFields - 1)])))
6868 []
6969 ]
7070 mkCon nm = recC (mkName nm) (mkField <$> [0 .. (numFields - 1)])
benchmark/Streamly/Benchmark/Data/Stream/Common.hs+1−1
@@ -104,7 +104,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
104104#endif
105105#endif
106106
107−import Gauge
107+import Test.Tasty.Bench
108108import Prelude hiding (Foldable(..), mapM, replicate)
109109
110110#ifdef USE_STREAMK
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentCommon.hs+3−3
@@ -14,14 +14,14 @@ where
1414import Stream.Common
1515 (composeN, benchIO, benchIOSink, benchIOSrc, sourceUnfoldrM)
1616import Streamly.Data.Stream (Stream)
17−import Streamly.Internal.Data.Stream.Concurrent (MonadAsync, Config)
17+import Streamly.Internal.Data.Stream.Prelude (MonadAsync, Config)
1818
1919import qualified Data.List as List
2020import qualified Streamly.Data.Fold as Fold
2121import qualified Streamly.Data.Stream as Stream
22−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
22+import qualified Streamly.Internal.Data.Stream.Prelude as Async
2323
24−import Gauge
24+import Test.Tasty.Bench
2525import Prelude hiding (mapM)
2626import Streamly.Benchmark.Common
2727
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentEager.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
12+import qualified Streamly.Internal.Data.Stream.Prelude as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentEager"
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentInterleaved.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
12+import qualified Streamly.Internal.Data.Stream.Prelude as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentInterleaved"
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentOrdered.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Concurrent as Async
12+import qualified Streamly.Internal.Data.Stream.Prelude as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentOrdered"
benchmark/Streamly/Benchmark/Data/Stream/Eliminate.hs+3−1
@@ -57,7 +57,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
5757#endif
5858#endif
5959
60−import Gauge
60+import Test.Tasty.Bench
6161import Streamly.Benchmark.Common
6262import Prelude hiding (length, sum, or, and, any, all, notElem, elem, (!!),
6363 lookup, repeat, minimum, maximum, product, last, mapM_, init)
@@ -132,12 +132,14 @@ foldableAny :: Int -> Int -> Bool
132132foldableAny value n =
133133 Prelude.any (> (value + 1)) (sourceUnfoldr value n :: Stream Identity Int)
134134
135+{- HLINT ignore "Use all"-}
135136{-# INLINE foldableAnd #-}
136137foldableAnd :: Int -> Int -> Bool
137138foldableAnd value n =
138139 Prelude.and $ fmap
139140 (<= (value + 1)) (sourceUnfoldr value n :: Stream Identity Int)
140141
142+{- HLINT ignore "Use any"-}
141143{-# INLINE foldableOr #-}
142144foldableOr :: Int -> Int -> Bool
143145foldableOr value n =
benchmark/Streamly/Benchmark/Data/Stream/Exceptions.hs+3−3
@@ -36,16 +36,16 @@ import System.IO (Handle, hClose, hPutChar)
3636import qualified Streamly.FileSystem.Handle as FH
3737import qualified Streamly.Internal.FileSystem.Handle as IFH
3838import qualified Streamly.Internal.Data.Unfold as IUF
39−import qualified Streamly.Internal.Data.Unfold.Exception as IUF
39+import qualified Streamly.Internal.Data.Unfold.Prelude as IUF
4040
4141#ifdef USE_PRELUDE
4242import qualified Streamly.Internal.Data.Stream.IsStream as Stream
4343#else
4444import qualified Streamly.Internal.Data.Stream as Stream
45−import qualified Streamly.Internal.Data.Stream.Exception.Lifted as Stream
45+import qualified Streamly.Internal.Data.Stream.Prelude as Stream
4646#endif
4747
48−import Gauge hiding (env)
48+import Test.Tasty.Bench hiding (env)
4949import Prelude hiding (last, length)
5050import Streamly.Benchmark.Common
5151import Streamly.Benchmark.Common.Handle
benchmark/Streamly/Benchmark/Data/Stream/Expand.hs+1−1
@@ -54,7 +54,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
5454#endif
5555#endif
5656
57−import Gauge
57+import Test.Tasty.Bench
5858import Stream.Common
5959import Streamly.Benchmark.Common
6060import Prelude hiding (concatMap)
benchmark/Streamly/Benchmark/Data/Stream/Generate.hs+1−1
@@ -40,7 +40,7 @@ import qualified Stream.Common as Common
4040#endif
4141#endif
4242
43−import Gauge
43+import Test.Tasty.Bench
4444import Streamly.Benchmark.Common
4545import qualified Prelude
4646
benchmark/Streamly/Benchmark/Data/Stream/Lift.hs+1−1
@@ -37,7 +37,7 @@ import qualified Control.Monad.State.Strict as State
3737#endif
3838#endif
3939
40−import Gauge
40+import Test.Tasty.Bench
4141import Streamly.Benchmark.Common
4242import Prelude hiding (reverse, tail)
4343
benchmark/Streamly/Benchmark/Data/Stream/Reduce.hs+2−2
@@ -48,7 +48,7 @@ import Streamly.Internal.Data.Stream (Stream)
4848import qualified Streamly.Internal.Data.Stream as S
4949#ifndef USE_STREAMLY_CORE
5050import qualified Streamly.Data.Stream.Prelude as S
51−import qualified Streamly.Internal.Data.Stream.Time as S
51+import qualified Streamly.Internal.Data.Stream.Prelude as S
5252#endif
5353
5454#ifdef USE_STREAMK
@@ -61,7 +61,7 @@ import qualified Streamly.Internal.Data.Stream as K
6161
6262#endif
6363
64−import Gauge
64+import Test.Tasty.Bench
6565import Streamly.Benchmark.Common
6666import Stream.Common
6767import Prelude hiding (reverse, tail)
benchmark/Streamly/Benchmark/Data/Stream/Split.hs+1−1
@@ -35,7 +35,7 @@ import qualified Streamly.Internal.FileSystem.Handle as IFH
3535import qualified Streamly.Internal.Unicode.Stream as IUS
3636import qualified Streamly.Prelude as S
3737
38−import Gauge hiding (env)
38+import Test.Tasty.Bench hiding (env)
3939import Prelude hiding (last, length)
4040import Streamly.Benchmark.Common
4141import Streamly.Benchmark.Common.Handle
benchmark/Streamly/Benchmark/Data/Stream/StreamD.hs+1−1
@@ -27,7 +27,7 @@ import Control.Applicative (liftA2)
2727#endif
2828import Control.Monad (when)
2929import Data.Maybe (isJust)
30−import Gauge (bench, nfIO, bgroup, Benchmark, defaultMain)
30+import Test.Tasty.Bench (bench, nfIO, bgroup, Benchmark, defaultMain)
3131import System.Random (randomRIO)
3232import Prelude hiding (tail, mapM_, foldl, last, map, mapM, concatMap, zip)
3333
benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs+1−1
@@ -37,7 +37,7 @@ import qualified Data.List as List
3737
3838import qualified Streamly.Internal.Data.StreamK as S
3939
40−import Gauge (bench, nfIO, bgroup, Benchmark, defaultMain)
40+import Test.Tasty.Bench (bench, nfIO, bgroup, Benchmark, defaultMain)
4141
4242import Streamly.Benchmark.Common
4343
benchmark/Streamly/Benchmark/Data/Stream/StreamKAlt.hs+2−2
@@ -23,7 +23,7 @@ import qualified Streamly.Internal.Data.StreamK.Alt as S
2323-- import qualified Streamly.Internal.Data.SVar as S
2424
2525import Streamly.Benchmark.Common
26−import Gauge (bgroup, Benchmark, defaultMain)
26+import Test.Tasty.Bench (bgroup, Benchmark, defaultMain)
2727
2828value :: Int
2929value = 100000
@@ -452,4 +452,4 @@ o_1_space =
452452 ]
453453
454454main :: IO ()
455−main = defaultMain $ concat [o_1_space]
455+main = defaultMain [o_1_space]
benchmark/Streamly/Benchmark/Data/Stream/ToStreamK.hs+2−2
@@ -37,7 +37,7 @@ import qualified Streamly.Internal.Data.StreamK as S
3737import qualified Streamly.Internal.Data.Stream as D
3838import qualified Streamly.Internal.Data.Fold as Fold
3939
40−import Gauge (bgroup, Benchmark, defaultMain)
40+import Test.Tasty.Bench (bgroup, Benchmark, defaultMain)
4141
4242import Streamly.Benchmark.Common
4343
@@ -120,7 +120,7 @@ fromFoldable streamLen n = S.fromFoldable [n..n+streamLen]
120120{-# INLINE fromFoldableM #-}
121121fromFoldableM :: Monad m => Int -> Int -> Stream m Int
122122fromFoldableM streamLen n =
123− Prelude.foldr S.consM S.nil (Prelude.fmap return [n..n+streamLen])
123+ Prelude.foldr (S.consM . return) S.nil [n .. n + streamLen]
124124
125125{-
126126{-# INLINABLE concatMapFoldableWith #-}
benchmark/Streamly/Benchmark/Data/Stream/Transform.hs+2−2
@@ -46,7 +46,7 @@ import Streamly.Internal.Data.Time.Units
4646import Streamly.Internal.Data.Stream (Stream)
4747import qualified Streamly.Internal.Data.Stream as Stream
4848#ifndef USE_STREAMLY_CORE
49−import qualified Streamly.Internal.Data.Stream.Time as Stream
49+import qualified Streamly.Internal.Data.Stream.Prelude as Stream
5050#endif
5151#ifdef USE_STREAMK
5252import Control.DeepSeq (NFData(..))
@@ -58,7 +58,7 @@ import qualified Streamly.Internal.Data.StreamK as StreamK
5858#endif
5959#endif
6060
61−import Gauge
61+import Test.Tasty.Bench
6262import Stream.Common hiding (scanl')
6363import Streamly.Benchmark.Common
6464import Prelude hiding (sequence, mapM)
benchmark/Streamly/Benchmark/Data/Unfold.hs+2−2
@@ -31,12 +31,12 @@ import qualified Prelude
3131import qualified Streamly.FileSystem.Handle as FH
3232import qualified Streamly.Internal.Data.Fold as FL
3333import qualified Streamly.Internal.Data.Unfold as UF
34−import qualified Streamly.Internal.Data.Unfold.Exception as UF
34+import qualified Streamly.Internal.Data.Unfold.Prelude as UF
3535import qualified Streamly.Internal.Data.Stream as S
3636import qualified Streamly.Internal.Data.Stream as D
3737import qualified Streamly.Internal.Data.StreamK as K
3838
39−import Gauge hiding (env)
39+import Test.Tasty.Bench hiding (env)
4040import Prelude hiding (take, filter, zipWith, map, mapM, takeWhile)
4141import Streamly.Benchmark.Common
4242import Streamly.Benchmark.Common.Handle
benchmark/Streamly/Benchmark/FileSystem/Handle.hs+1−1
@@ -22,7 +22,7 @@ import Streamly.Benchmark.Common.Handle (mkHandleBenchEnv)
2222import qualified Handle.ReadWrite as RW
2323import qualified Handle.Read as RO
2424
25−import Gauge hiding (env)
25+import Test.Tasty.Bench hiding (env)
2626import Prelude hiding (last, length)
2727import Streamly.Benchmark.Common
2828
benchmark/Streamly/Benchmark/FileSystem/Handle/Read.hs+8−9
@@ -33,7 +33,6 @@ import qualified Streamly.Data.Stream as Stream
3333import qualified Streamly.Data.Fold as Fold
3434import qualified Streamly.FileSystem.Handle as FH
3535import qualified Streamly.Internal.Data.Array as A
36−import qualified Streamly.Internal.Data.Array as AT
3736import qualified Streamly.Internal.Data.Fold as FL
3837import qualified Streamly.Internal.Data.Stream as IP
3938import qualified Streamly.Internal.FileSystem.Handle as IFH
@@ -41,14 +40,14 @@ import qualified Streamly.Internal.Unicode.Stream as IUS
4140import qualified Streamly.Data.Stream.Prelude as S
4241import qualified Streamly.Unicode.Stream as SS
4342
44−import Gauge hiding (env)
43+import Test.Tasty.Bench hiding (env)
4544import Prelude hiding (last, length)
4645import Streamly.Benchmark.Common.Handle
4746
4847#ifdef INSPECTION
4948import Streamly.Internal.Data.Stream (Step(..), FoldMany)
5049
51−import qualified Streamly.Internal.Data.MutArray as MA
50+import qualified Streamly.Internal.Data.MutArray as MutArray
5251import qualified Streamly.Internal.Data.Stream as D
5352import qualified Streamly.Internal.Data.Unfold as IUF
5453
@@ -69,7 +68,7 @@ readLast = S.fold Fold.last . S.unfold FH.reader
6968inspect $ hasNoTypeClasses 'readLast
7069inspect $ 'readLast `hasNoType` ''Step -- S.unfold
7170inspect $ 'readLast `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
72−inspect $ 'readLast `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
71+inspect $ 'readLast `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
7372#endif
7473
7574-- assert that flattenArrays constructors are not present
@@ -81,7 +80,7 @@ readCountBytes = S.fold Fold.length . S.unfold FH.reader
8180inspect $ hasNoTypeClasses 'readCountBytes
8281inspect $ 'readCountBytes `hasNoType` ''Step -- S.unfold
8382inspect $ 'readCountBytes `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
84−inspect $ 'readCountBytes `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
83+inspect $ 'readCountBytes `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
8584#endif
8685
8786-- | Count the number of lines in a file.
@@ -96,7 +95,7 @@ readCountLines =
9695inspect $ hasNoTypeClasses 'readCountLines
9796inspect $ 'readCountLines `hasNoType` ''Step
9897inspect $ 'readCountLines `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
99−inspect $ 'readCountLines `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
98+inspect $ 'readCountLines `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
10099#endif
101100
102101-- | Count the number of words in a file.
@@ -120,7 +119,7 @@ readSumBytes = S.fold Fold.sum . S.unfold FH.reader
120119inspect $ hasNoTypeClasses 'readSumBytes
121120inspect $ 'readSumBytes `hasNoType` ''Step
122121inspect $ 'readSumBytes `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
123−inspect $ 'readSumBytes `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
122+inspect $ 'readSumBytes `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
124123#endif
125124
126125-- XXX When we mark this with INLINE and we have two benchmarks using S.drain
@@ -236,13 +235,13 @@ groupsOf :: Int -> Handle -> IO Int
236235groupsOf n inh =
237236 -- writeNUnsafe gives 2.5x boost here over writeN.
238237 S.fold Fold.length
239− $ IP.groupsOf n (AT.writeNUnsafe n) (S.unfold FH.reader inh)
238+ $ IP.groupsOf n (A.writeNUnsafe n) (S.unfold FH.reader inh)
240239
241240#ifdef INSPECTION
242241inspect $ hasNoTypeClasses 'groupsOf
243242inspect $ 'groupsOf `hasNoType` ''Step
244243inspect $ 'groupsOf `hasNoType` ''FoldMany
245−inspect $ 'groupsOf `hasNoType` ''AT.ArrayUnsafe -- AT.writeNUnsafe
244+inspect $ 'groupsOf `hasNoType` ''MutArray.ArrayUnsafe -- AT.writeNUnsafe
246245 -- FH.read/A.read
247246inspect $ 'groupsOf `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
248247#endif
benchmark/Streamly/Benchmark/FileSystem/Handle/ReadWrite.hs+9−10
@@ -34,7 +34,7 @@ import qualified Streamly.Internal.FileSystem.Handle as IFH
3434import qualified Streamly.Data.Array as A
3535import qualified Streamly.Data.Stream.Prelude as S
3636
37−import Gauge hiding (env)
37+import Test.Tasty.Bench hiding (env)
3838import Streamly.Benchmark.Common.Handle
3939
4040#ifdef INSPECTION
@@ -42,8 +42,7 @@ import Streamly.Internal.Data.Stream (Step(..))
4242
4343import qualified Streamly.Internal.Data.Stream as D
4444import qualified Streamly.Internal.Data.Tuple.Strict as Strict
45−import qualified Streamly.Internal.Data.Stream.MutChunked as MAS
46−import qualified Streamly.Internal.Data.Array as AT
45+import qualified Streamly.Internal.Data.MutArray as MutArray
4746
4847import Test.Inspection
4948#endif
@@ -83,7 +82,7 @@ copyStream inh outh = S.fold (FH.write outh) (S.unfold FH.reader inh)
8382inspect $ hasNoTypeClasses 'copyStream
8483inspect $ 'copyStream `hasNoType` ''Step -- S.unfold
8584inspect $ 'copyStream `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
86−inspect $ 'copyStream `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
85+inspect $ 'copyStream `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
8786 -- FH.read/A.read
8887inspect $ 'copyStream `hasNoType` ''Strict.Tuple3' -- FH.write/chunksOf
8988#endif
@@ -109,8 +108,8 @@ readFromBytesNull inh devNull = IFH.putBytes devNull $ S.unfold FH.reader inh
109108#ifdef INSPECTION
110109inspect $ hasNoTypeClasses 'readFromBytesNull
111110inspect $ 'readFromBytesNull `hasNoType` ''Step
112−inspect $ 'readFromBytesNull `hasNoType` ''MAS.SpliceState
113−inspect $ 'readFromBytesNull `hasNoType` ''AT.ArrayUnsafe -- FH.fromBytes/S.chunksOf
111+inspect $ 'readFromBytesNull `hasNoType` ''MutArray.SpliceState
112+inspect $ 'readFromBytesNull `hasNoType` ''MutArray.ArrayUnsafe -- FH.fromBytes/S.chunksOf
114113inspect $ 'readFromBytesNull `hasNoType` ''D.FoldMany
115114#endif
116115
@@ -123,8 +122,8 @@ readWithFromBytesNull inh devNull =
123122#ifdef INSPECTION
124123inspect $ hasNoTypeClasses 'readWithFromBytesNull
125124inspect $ 'readWithFromBytesNull `hasNoType` ''Step
126−inspect $ 'readWithFromBytesNull `hasNoType` ''MAS.SpliceState
127−inspect $ 'readWithFromBytesNull `hasNoType` ''AT.ArrayUnsafe -- FH.fromBytes/S.chunksOf
125+inspect $ 'readWithFromBytesNull `hasNoType` ''MutArray.SpliceState
126+inspect $ 'readWithFromBytesNull `hasNoType` ''MutArray.ArrayUnsafe -- FH.fromBytes/S.chunksOf
128127inspect $ 'readWithFromBytesNull `hasNoType` ''D.FoldMany
129128#endif
130129
@@ -171,7 +170,7 @@ writeReadWith inh devNull = IUF.fold fld unf (defaultChunkSize, inh)
171170inspect $ hasNoTypeClasses 'writeReadWith
172171inspect $ 'writeReadWith `hasNoType` ''Step
173172inspect $ 'writeReadWith `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
174−inspect $ 'writeReadWith `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
173+inspect $ 'writeReadWith `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
175174 -- FH.read/A.read
176175#endif
177176
@@ -188,7 +187,7 @@ writeRead inh devNull = IUF.fold fld unf inh
188187inspect $ hasNoTypeClasses 'writeRead
189188inspect $ 'writeRead `hasNoType` ''Step
190189inspect $ 'writeRead `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
191−inspect $ 'writeRead `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
190+inspect $ 'writeRead `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
192191 -- FH.read/A.read
193192#endif
194193
benchmark/Streamly/Benchmark/Prelude/Adaptive.hs+1−1
@@ -10,7 +10,7 @@
1010import Control.Concurrent (threadDelay)
1111import Control.Monad (when)
1212import Control.Monad.IO.Class (liftIO)
13−import Gauge
13+import Test.Tasty.Bench
1414import Streamly.Prelude as S
1515import System.Random (randomRIO)
1616
benchmark/Streamly/Benchmark/Prelude/Ahead.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Gauge
18+import Test.Tasty.Bench
1919
2020moduleName :: String
2121moduleName = "Prelude.Ahead"
benchmark/Streamly/Benchmark/Prelude/Async.hs+1−1
@@ -16,7 +16,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream.Transform as Transform
1616import Streamly.Benchmark.Common
1717import Streamly.Benchmark.Prelude
1818
19−import Gauge
19+import Test.Tasty.Bench
2020
2121moduleName :: String
2222moduleName = "Prelude.Async"
benchmark/Streamly/Benchmark/Prelude/Concurrent.hs+1−1
@@ -15,7 +15,7 @@ import Streamly.Prelude
1515 , fromWAsync, parallel, fromParallel
1616 )
1717
18−import Gauge
18+import Test.Tasty.Bench
1919import qualified Streamly.Prelude as S
2020
2121-------------------------------------------------------------------------------
benchmark/Streamly/Benchmark/Prelude/Merge.hs+1−1
@@ -30,7 +30,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream as Stream
3030import Streamly.Benchmark.Common
3131import Streamly.Benchmark.Prelude
3232
33−import Gauge
33+import Test.Tasty.Bench
3434
3535#ifdef INSPECTION
3636import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Prelude/Parallel.hs+1−1
@@ -22,7 +22,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream as Internal
2222import Streamly.Benchmark.Common
2323import Streamly.Benchmark.Prelude
2424
25−import Gauge
25+import Test.Tasty.Bench
2626
2727moduleName :: String
2828moduleName = "Prelude.Parallel"
benchmark/Streamly/Benchmark/Prelude/Rate.hs+1−1
@@ -14,7 +14,7 @@ import qualified Streamly.Prelude as S
1414import Streamly.Benchmark.Common
1515import Streamly.Benchmark.Prelude
1616
17−import Gauge
17+import Test.Tasty.Bench
1818
1919moduleName :: String
2020moduleName = "Prelude.Rate"
benchmark/Streamly/Benchmark/Prelude/WAsync.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Gauge
18+import Test.Tasty.Bench
1919
2020moduleName :: String
2121moduleName = "Prelude.WAsync"
benchmark/Streamly/Benchmark/Prelude/WSerial.hs+1−1
@@ -29,7 +29,7 @@ import qualified Streamly.Internal.Data.Unfold as UF
2929import Streamly.Benchmark.Common
3030import Streamly.Benchmark.Prelude
3131
32−import Gauge
32+import Test.Tasty.Bench
3333
3434#ifdef INSPECTION
3535import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Prelude/ZipAsync.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Gauge
18+import Test.Tasty.Bench
1919
2020moduleName :: String
2121moduleName = "Prelude.ZipAsync"
benchmark/Streamly/Benchmark/Prelude/ZipSerial.hs+1−1
@@ -29,7 +29,7 @@ import qualified Streamly.Prelude as S
2929import Streamly.Benchmark.Common
3030import Streamly.Benchmark.Prelude hiding (sourceUnfoldrM)
3131
32−import Gauge
32+import Test.Tasty.Bench
3333
3434#ifdef INSPECTION
3535import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Unicode/Char.hs+1−1
@@ -16,7 +16,7 @@ import Control.DeepSeq (NFData(..))
1616import Streamly.Internal.Data.Array (Array)
1717import System.FilePath (dropExtensions, takeFileName)
1818import System.FilePath.Posix ((</>))
19−import Gauge.Main (Benchmark, bench, bgroup, defaultMain, env, nfIO)
19+import Test.Tasty.Bench (Benchmark, bench, bgroup, defaultMain, env, nfIO)
2020import Streamly.Internal.Unicode.Char
2121 ( NormalizationMode(NFC, NFD, NFKC, NFKD)
2222 , normalize
benchmark/Streamly/Benchmark/Unicode/Parser.hs+1−1
@@ -31,7 +31,7 @@ import qualified Streamly.Data.Array as Array
3131import qualified Streamly.Data.Stream as Stream
3232import qualified Streamly.Internal.Unicode.Parser as PRU
3333
34−import Gauge hiding (env)
34+import Test.Tasty.Bench hiding (env)
3535import Streamly.Benchmark.Common
3636
3737runParser :: Int -> (Stream IO Char -> IO a) -> IO ()
benchmark/Streamly/Benchmark/Unicode/Stream.hs+24−6
@@ -11,6 +11,11 @@
1111{-# LANGUAGE CPP #-}
1212{-# LANGUAGE ScopedTypeVariables #-}
1313
14+#undef FUSION_CHECK
15+#ifdef FUSION_CHECK
16+{-# OPTIONS_GHC -ddump-simpl -ddump-to-file -dsuppress-all #-}
17+#endif
18+
1419#ifdef __HADDOCK_VERSION__
1520#undef INSPECTION
1621#endif
@@ -33,16 +38,17 @@ import qualified Streamly.Internal.FileSystem.Handle as Handle
3338import qualified Streamly.Internal.Unicode.Array as UnicodeArr
3439import qualified Streamly.Internal.Unicode.Stream as Unicode
3540
36−import Gauge hiding (env)
41+import Test.Tasty.Bench hiding (env)
3742import Streamly.Benchmark.Common
3843import Streamly.Benchmark.Common.Handle
3944
4045#ifdef INSPECTION
41−import Streamly.Internal.Data.Unbox (Unbox)
46+import Streamly.Internal.Data.MutByteArray (Unbox)
4247import Streamly.Internal.Data.Stream (Step(..))
48+import qualified Streamly.Internal.Data.Array as Array
49+import qualified Streamly.Internal.Data.MutArray as MutArray
4350import qualified Streamly.Internal.Data.Fold as Fold
4451import qualified Streamly.Internal.Data.Tuple.Strict as Strict
45−import qualified Streamly.Internal.Data.Array as Array
4652
4753import Test.Inspection
4854#endif
@@ -218,7 +224,7 @@ inspect $ 'copyStreamLatin1' `hasNoType` ''Step
218224inspect $ 'copyStreamLatin1' `hasNoType` ''Unfold.ConcatState -- Handle.read/UF.many
219225
220226inspect $ 'copyStreamLatin1' `hasNoType` ''Fold.Step
221−inspect $ 'copyStreamLatin1' `hasNoType` ''Array.ArrayUnsafe -- Handle.write/writeNUnsafe
227+inspect $ 'copyStreamLatin1' `hasNoType` ''MutArray.ArrayUnsafe -- Handle.write/writeNUnsafe
222228 -- Handle.read/Array.read
223229inspect $ 'copyStreamLatin1' `hasNoType` ''Strict.Tuple3' -- Handle.write/chunksOf
224230#endif
@@ -239,7 +245,7 @@ inspect $ 'copyStreamLatin1 `hasNoType` ''Unfold.ConcatState -- Handle.read/UF.m
239245
240246inspect $ 'copyStreamLatin1 `hasNoType` ''Fold.ManyState
241247inspect $ 'copyStreamLatin1 `hasNoType` ''Fold.Step
242−inspect $ 'copyStreamLatin1 `hasNoType` ''Array.ArrayUnsafe -- Handle.write/writeNUnsafe
248+inspect $ 'copyStreamLatin1 `hasNoType` ''MutArray.ArrayUnsafe -- Handle.write/writeNUnsafe
243249 -- Handle.read/Array.read
244250inspect $ 'copyStreamLatin1 `hasNoType` ''Strict.Tuple3' -- Handle.write/chunksOf
245251#endif
@@ -280,7 +286,8 @@ _copyStreamUtf8'Fold :: Handle -> Handle -> IO ()
280286_copyStreamUtf8'Fold inh outh =
281287 Stream.fold (Handle.write outh)
282288 $ Unicode.encodeUtf8
283− $ Stream.foldMany Unicode.writeCharUtf8'
289+ $ Stream.catRights
290+ $ Stream.parseMany Unicode.writeCharUtf8'
284291 $ Stream.unfold Handle.reader inh
285292
286293{-# NOINLINE _copyStreamUtf8Parser #-}
@@ -317,6 +324,7 @@ o_1_space_decode_encode_read env =
317324
318325main :: IO ()
319326main = do
327+#ifndef FUSION_CHECK
320328 env <- mkHandleBenchEnv
321329 defaultMain (allBenchmarks env)
322330
@@ -329,3 +337,13 @@ main = do
329337 , o_1_space_decode_encode_read env
330338 ]
331339 ]
340+#else
341+ -- Enable FUSION_CHECK macro at the beginning of the file
342+ -- Enable one benchmark below, and run the benchmark
343+ -- Check the .dump-simpl output
344+ env <- mkHandleBenchEnv
345+ let mkHandles (RefHandles {bigInH = inh, outputH = outh}) = Handles inh outh
346+ (Handles inh outh) <- getHandles env mkHandles
347+ copyStreamLatin1' inh outh
348+ return ()
349+#endif
benchmark/Streamly/Benchmark/Unicode/Utf8.hs+1−1
@@ -13,7 +13,7 @@
1313import Data.Char (chr)
1414import Streamly.Internal.Unicode.Utf8 (Utf8)
1515
16−import Gauge hiding (env)
16+import Test.Tasty.Bench hiding (env)
1717import Streamly.Benchmark.Common
1818
1919import qualified Streamly.Internal.Unicode.Utf8 as Utf8
benchmark/lib/Streamly/Benchmark/Common.hs+1−1
@@ -56,7 +56,7 @@ import System.Random (randomRIO)
5656import qualified Streamly.Internal.Data.Fold as Fold
5757import qualified Streamly.Internal.Data.Stream as S
5858
59−import Gauge
59+import Test.Tasty.Bench
6060
6161-------------------------------------------------------------------------------
6262-- Benchmark Prefixes
benchmark/lib/Streamly/Benchmark/Common/Handle.hs+3−1
@@ -30,6 +30,8 @@ module Streamly.Benchmark.Common.Handle
3030 , isSpace
3131 , isSp
3232 , mkHandleBenchEnv
33+ , Handles(..)
34+ , getHandles
3335 )
3436where
3537
@@ -46,7 +48,7 @@ import System.Process (callCommand)
4648
4749import Data.IORef
4850import Prelude hiding (last, length)
49−import Gauge hiding (env)
51+import Test.Tasty.Bench hiding (env)
5052
5153
5254scratchDir :: String
benchmark/lib/Streamly/Benchmark/Prelude.hs+1−1
@@ -82,7 +82,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream.Type as IsStream
8282import qualified Streamly.Internal.Data.Pipe as Pipe
8383import qualified Streamly.Internal.Data.Stream.Serial as Serial
8484
85−import Gauge
85+import Test.Tasty.Bench
8686import Streamly.Internal.Data.Time.Units
8787
8888-- Common polymorphic stream APIs used across multiple stream modules
benchmark/streamly-benchmarks.cabal+5−20
@@ -54,11 +54,6 @@ flag opt
5454 manual: True
5555 default: True
5656
57−flag use-gauge
58− description: Use gauge instead of tasty-bench for benchmarking
59− manual: True
60− default: False
61−
6257flag use-streamly-core
6358 description: Benchmark only core package
6459 manual: True
@@ -193,24 +188,14 @@ common bench-depends
193188 , unordered-containers >= 0.2 && < 0.3
194189 , process >= 1.4 && < 1.7
195190 , directory >= 1.2.2 && < 1.4
196− , filepath >= 1.4.1 && < 1.5
191+ , filepath >= 1.4.1 && < 1.6
197192 , ghc-prim >= 0.4 && < 0.12
193+ , tasty-bench >= 0.3 && < 0.4
194+ , tasty >= 1.4.1 && < 1.6
195+ , streamly-core
198196
199− if flag(use-streamly-core)
200− build-depends: streamly-core
201− else
197+ if !flag(use-streamly-core)
202198 build-depends: streamly
203− , streamly-core
204−
205− if flag(use-gauge)
206− build-depends: gauge >= 0.2.4 && < 0.3
207− else
208− build-depends: tasty-bench >= 0.3 && < 0.4
209− , tasty >= 1.4.1 && < 1.6
210− mixins: tasty-bench
211− (Test.Tasty.Bench as Gauge
212− , Test.Tasty.Bench as Gauge.Main
213− )
214199
215200 if flag(fusion-plugin) && !impl(ghcjs) && !impl(ghc < 8.6)
216201 build-depends:
configure+9−9
@@ -1,6 +1,6 @@
11#! /bin/sh
22# Guess values for system-dependent variables and create Makefiles.
3−# Generated by GNU Autoconf 2.71 for streamly 0.10.0.
3+# Generated by GNU Autoconf 2.71 for streamly 0.10.1.
44#
55# Report bugs to <streamly@composewell.com>.
66#
@@ -610,8 +610,8 @@ MAKEFLAGS=
610610# Identity of this package.
611611PACKAGE_NAME='streamly'
612612PACKAGE_TARNAME='streamly'
613−PACKAGE_VERSION='0.10.0'
614−PACKAGE_STRING='streamly 0.10.0'
613+PACKAGE_VERSION='0.10.1'
614+PACKAGE_STRING='streamly 0.10.1'
615615PACKAGE_BUGREPORT='streamly@composewell.com'
616616PACKAGE_URL='https://streamly.composewell.com'
617617
@@ -1224,7 +1224,7 @@ if test "$ac_init_help" = "long"; then
12241224 # Omit some internal or obsolete options to make the list less imposing.
12251225 # This message is too long to be a string in the A/UX 3.1 sh.
12261226 cat <<_ACEOF
1227−\`configure' configures streamly 0.10.0 to adapt to many kinds of systems.
1227+\`configure' configures streamly 0.10.1 to adapt to many kinds of systems.
12281228
12291229Usage: $0 [OPTION]... [VAR=VALUE]...
12301230
@@ -1286,7 +1286,7 @@ fi
12861286
12871287if test -n "$ac_init_help"; then
12881288 case $ac_init_help in
1289− short | recursive ) echo "Configuration of streamly 0.10.0:";;
1289+ short | recursive ) echo "Configuration of streamly 0.10.1:";;
12901290 esac
12911291 cat <<\_ACEOF
12921292
@@ -1372,7 +1372,7 @@ fi
13721372test -n "$ac_init_help" && exit $ac_status
13731373if $ac_init_version; then
13741374 cat <<\_ACEOF
1375−streamly configure 0.10.0
1375+streamly configure 0.10.1
13761376generated by GNU Autoconf 2.71
13771377
13781378Copyright (C) 2021 Free Software Foundation, Inc.
@@ -1500,7 +1500,7 @@ cat >config.log <<_ACEOF
15001500This file contains any messages produced by compilers while
15011501running configure, to aid debugging if configure makes a mistake.
15021502
1503−It was created by streamly $as_me 0.10.0, which was
1503+It was created by streamly $as_me 0.10.1, which was
15041504generated by GNU Autoconf 2.71. Invocation command line was
15051505
15061506 $ $0$ac_configure_args_raw
@@ -3800,7 +3800,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
38003800# report actual input values of CONFIG_FILES etc. instead of their
38013801# values after options handling.
38023802ac_log="
3803−This file was extended by streamly $as_me 0.10.0, which was
3803+This file was extended by streamly $as_me 0.10.1, which was
38043804generated by GNU Autoconf 2.71. Invocation command line was
38053805
38063806 CONFIG_FILES = $CONFIG_FILES
@@ -3856,7 +3856,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\
38563856cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
38573857ac_cs_config='$ac_cs_config_escaped'
38583858ac_cs_version="\\
3859−streamly config.status 0.10.0
3859+streamly config.status 0.10.1
38603860configured by $0, generated by GNU Autoconf 2.71,
38613861 with options \\"\$ac_cs_config\\"
38623862
configure.ac+1−1
@@ -3,7 +3,7 @@
33# See https://www.gnu.org/software/autoconf/manual/autoconf.html for help on
44# the macros used in this file.
55
6−AC_INIT([streamly], [0.10.0], [streamly@composewell.com], [streamly], [https://streamly.composewell.com])
6+AC_INIT([streamly], [0.10.1], [streamly@composewell.com], [streamly], [https://streamly.composewell.com])
77
88# To suppress "WARNING: unrecognized options: --with-compiler"
99AC_ARG_WITH([compiler], [GHC])
core/configure+9−9
@@ -1,6 +1,6 @@
11#! /bin/sh
22# Guess values for system-dependent variables and create Makefiles.
3−# Generated by GNU Autoconf 2.71 for streamly-core 0.2.0.
3+# Generated by GNU Autoconf 2.71 for streamly-core 0.2.2.
44#
55# Report bugs to <streamly@composewell.com>.
66#
@@ -610,8 +610,8 @@ MAKEFLAGS=
610610# Identity of this package.
611611PACKAGE_NAME='streamly-core'
612612PACKAGE_TARNAME='streamly-core'
613−PACKAGE_VERSION='0.2.0'
614−PACKAGE_STRING='streamly-core 0.2.0'
613+PACKAGE_VERSION='0.2.2'
614+PACKAGE_STRING='streamly-core 0.2.2'
615615PACKAGE_BUGREPORT='streamly@composewell.com'
616616PACKAGE_URL='https://streamly.composewell.com'
617617
@@ -1256,7 +1256,7 @@ if test "$ac_init_help" = "long"; then
12561256 # Omit some internal or obsolete options to make the list less imposing.
12571257 # This message is too long to be a string in the A/UX 3.1 sh.
12581258 cat <<_ACEOF
1259−\`configure' configures streamly-core 0.2.0 to adapt to many kinds of systems.
1259+\`configure' configures streamly-core 0.2.2 to adapt to many kinds of systems.
12601260
12611261Usage: $0 [OPTION]... [VAR=VALUE]...
12621262
@@ -1318,7 +1318,7 @@ fi
13181318
13191319if test -n "$ac_init_help"; then
13201320 case $ac_init_help in
1321− short | recursive ) echo "Configuration of streamly-core 0.2.0:";;
1321+ short | recursive ) echo "Configuration of streamly-core 0.2.2:";;
13221322 esac
13231323 cat <<\_ACEOF
13241324
@@ -1404,7 +1404,7 @@ fi
14041404test -n "$ac_init_help" && exit $ac_status
14051405if $ac_init_version; then
14061406 cat <<\_ACEOF
1407−streamly-core configure 0.2.0
1407+streamly-core configure 0.2.2
14081408generated by GNU Autoconf 2.71
14091409
14101410Copyright (C) 2021 Free Software Foundation, Inc.
@@ -1622,7 +1622,7 @@ cat >config.log <<_ACEOF
16221622This file contains any messages produced by compilers while
16231623running configure, to aid debugging if configure makes a mistake.
16241624
1625−It was created by streamly-core $as_me 0.2.0, which was
1625+It was created by streamly-core $as_me 0.2.2, which was
16261626generated by GNU Autoconf 2.71. Invocation command line was
16271627
16281628 $ $0$ac_configure_args_raw
@@ -3832,7 +3832,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
38323832# report actual input values of CONFIG_FILES etc. instead of their
38333833# values after options handling.
38343834ac_log="
3835−This file was extended by streamly-core $as_me 0.2.0, which was
3835+This file was extended by streamly-core $as_me 0.2.2, which was
38363836generated by GNU Autoconf 2.71. Invocation command line was
38373837
38383838 CONFIG_FILES = $CONFIG_FILES
@@ -3888,7 +3888,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\
38883888cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
38893889ac_cs_config='$ac_cs_config_escaped'
38903890ac_cs_version="\\
3891−streamly-core config.status 0.2.0
3891+streamly-core config.status 0.2.2
38923892configured by $0, generated by GNU Autoconf 2.71,
38933893 with options \\"\$ac_cs_config\\"
38943894
core/configure.ac+1−1
@@ -3,7 +3,7 @@
33# See https://www.gnu.org/software/autoconf/manual/autoconf.html for help on
44# the macros used in this file.
55
6−AC_INIT([streamly-core], [0.2.0], [streamly@composewell.com], [streamly-core], [https://streamly.composewell.com])
6+AC_INIT([streamly-core], [0.2.2], [streamly@composewell.com], [streamly-core], [https://streamly.composewell.com])
77
88# To suppress "WARNING: unrecognized options: --with-compiler"
99AC_ARG_WITH([compiler], [GHC])
core/docs/ApiChangelogs/0.1.0-0.2.0.txtnew file+1471
@@ -0,0 +1,1471 @@
1+---------------------------------
2+API Annotations
3+---------------------------------
4+
5+[A] : Added
6+[R] : Removed
7+[C] : Changed
8+[O] : Old definition
9+[N] : New definition
10+[D] : Deprecated
11+
12+---------------------------------
13+API diff
14+---------------------------------
15+
16+[C] Streamly.Unicode.Parser
17+ [A] double :: Monad m => Parser Char m Double
18+[C] Streamly.FileSystem.Handle
19+ [A] readWith :: MonadIO m => Int -> Handle -> Stream m Word8
20+ [A] readChunksWith :: MonadIO m => Int -> Handle -> Stream m (Array Word8)
21+[C] Streamly.FileSystem.Dir
22+ [C] readEither
23+ [O] readEither :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
24+ [N] readEither :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
25+ [C] read
26+ [O] read :: MonadIO m => FilePath -> Stream m FilePath
27+ [N] read :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
28+[C] Streamly.Data.Unfold
29+ [A] class Enum a => Enumerable a
30+ [A] second :: b -> Unfold m (a, b) c -> Unfold m a c
31+ [A] first :: a -> Unfold m (a, b) c -> Unfold m b c
32+ [A] enumerateFromTo :: (Enumerable a, Monad m) => Unfold m (a, a) a
33+ [A] enumerateFromThenTo :: (Enumerable a, Monad m) => Unfold m (a, a, a) a
34+ [A] enumerateFromThen :: (Enumerable a, Monad m) => Unfold m (a, a) a
35+ [A] enumerateFrom :: (Enumerable a, Monad m) => Unfold m a a
36+[C] Streamly.Data.StreamK
37+ [C] parseChunks
38+ [O] parseChunks :: (Monad m, Unbox a) => ParserK a m b -> StreamK m (Array a) -> m (Either ParseError b)
39+ [N] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
40+ [C] parseBreakChunks
41+ [O] parseBreakChunks :: (Monad m, Unbox a) => ParserK a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
42+ [N] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
43+ [A] parseBreak :: forall m a b. Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
44+ [A] parse :: Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b)
45+ [A] handle :: (MonadCatch m, Exception e) => (e -> m (StreamK m a)) -> StreamK m a -> StreamK m a
46+ [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> StreamK m a) -> StreamK m a
47+[C] Streamly.Data.Stream
48+ [A] wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
49+ [A] splitOn :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
50+ [C] handle
51+ [O] handle :: (MonadCatch m, Exception e) => (e -> Stream m a) -> Stream m a -> Stream m a
52+ [N] handle :: (MonadCatch m, Exception e) => (e -> m (Stream m a)) -> Stream m a -> Stream m a
53+ [A] groupsOf :: Monad m => Int -> Fold m a b -> Stream m a -> Stream m b
54+[C] Streamly.Data.ParserK
55+ [D] fromParser :: (MonadIO m, Unbox a) => Parser a m b -> ParserK (Array a) m b
56+ [D] fromFold :: (MonadIO m, Unbox a) => Fold m a b -> ParserK (Array a) m b
57+ [A] adaptCG :: Monad m => Parser a m b -> ParserK (Array a) m b
58+ [A] adaptC :: (Monad m, Unbox a) => Parser a m b -> ParserK (Array a) m b
59+ [A] adapt :: Monad m => Parser a m b -> ParserK a m b
60+[C] Streamly.Data.Parser
61+ [A] groupByRollingEither :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
62+ [A] groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
63+[A] Streamly.Data.MutByteArray
64+ [A] class Unbox a
65+ [A] class Serialize a
66+ [A] SerializeConfig
67+ [A] MutByteArray
68+ [A] unpin :: MutByteArray -> IO MutByteArray
69+ [A] sizeOf :: (Unbox a, SizeOfRep (Rep a)) => Proxy a -> Int
70+ [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
71+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
72+ [A] pinnedNew :: Int -> IO MutByteArray
73+ [A] pin :: MutByteArray -> IO MutByteArray
74+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
75+ [A] new :: Int -> IO MutByteArray
76+ [A] isPinned :: MutByteArray -> Bool
77+ [A] inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
78+ [A] inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
79+ [A] inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
80+ [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
81+ [A] deriveUnbox :: Q [Dec] -> Q [Dec]
82+ [A] deriveSerializeWith :: (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
83+ [A] deriveSerialize :: Q [Dec] -> Q [Dec]
84+ [A] addSizeTo :: Serialize a => Int -> a -> Int
85+[C] Streamly.Data.MutArray.Generic
86+ [A] write :: MonadIO m => Fold m a (MutArray a)
87+ [A] readRev :: MonadIO m => MutArray a -> Stream m a
88+ [A] read :: MonadIO m => MutArray a -> Stream m a
89+ [A] putIndexUnsafe :: forall m a. MonadIO m => Int -> MutArray a -> a -> m ()
90+ [A] modifyIndexUnsafe :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
91+ [A] length :: MutArray a -> Int
92+ [A] getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
93+ [C] getIndex
94+ [O] getIndex :: MonadIO m => Int -> MutArray a -> m a
95+ [N] getIndex :: MonadIO m => Int -> MutArray a -> m (Maybe a)
96+ [A] fromListN :: MonadIO m => Int -> [a] -> m (MutArray a)
97+ [A] fromList :: MonadIO m => [a] -> m (MutArray a)
98+[C] Streamly.Data.MutArray
99+ [A] unpin :: MutArray a -> IO (MutArray a)
100+ [A] readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
101+ [A] read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
102+ [A] putIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
103+ [D] pokeByteIndex :: Unbox a => Int -> MutByteArray -> a -> IO ()
104+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
105+ [A] pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
106+ [A] pin :: MutArray a -> IO (MutArray a)
107+ [D] peekByteIndex :: Unbox a => Int -> MutByteArray -> IO a
108+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
109+ [D] newPinned :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
110+ [A] modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
111+ [A] modifyIndex :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
112+ [A] modify :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (a -> a) -> m ()
113+ [A] isPinned :: MutArray a -> Bool
114+ [A] getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
115+ [C] getIndex
116+ [O] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
117+ [N] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
118+[C] Streamly.Data.Array.Generic
119+ [A] toList :: Array a -> [a]
120+ [A] getIndex :: Int -> Array a -> Maybe a
121+[C] Streamly.Data.Array
122+ [A] unpin :: Array a -> IO (Array a)
123+ [D] pokeByteIndex :: Unbox a => Int -> MutByteArray -> a -> IO ()
124+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
125+ [A] pin :: Array a -> IO (Array a)
126+ [D] peekByteIndex :: Unbox a => Int -> MutByteArray -> IO a
127+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
128+ [A] isPinned :: Array a -> Bool
129+[C] Streamly.Console.Stdio
130+
131+---------------------------------
132+Internal API diff
133+---------------------------------
134+
135+[C] Streamly.Internal.Unicode.Stream
136+ [C] writeCharUtf8'
137+ [O] writeCharUtf8' :: Monad m => Fold m Word8 Char
138+ [N] writeCharUtf8' :: Monad m => Parser Word8 m Char
139+[C] Streamly.Internal.Unicode.Parser
140+ [A] number :: Monad m => Parser Char m (Integer, Int)
141+ [A] mkDouble :: Integer -> Int -> Double
142+ [A] doubleParser :: Monad m => Parser Char m (Int, Int)
143+ [C] double
144+ [O] double :: Parser Char m Double
145+ [N] double :: Monad m => Parser Char m Double
146+[R] Streamly.Internal.Serialize.ToBytes
147+[R] Streamly.Internal.Serialize.FromBytes
148+[C] Streamly.Internal.FileSystem.File
149+ [A] writeAppendWith :: (MonadIO m, MonadCatch m) => Int -> FilePath -> Stream m Word8 -> m ()
150+ [A] writeAppendChunks :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Array a) -> m ()
151+ [A] writeAppendArray :: FilePath -> Array a -> IO ()
152+ [A] writeAppend :: (MonadIO m, MonadCatch m) => FilePath -> Stream m Word8 -> m ()
153+ [R] appendWith :: (MonadIO m, MonadCatch m) => Int -> FilePath -> Stream m Word8 -> m ()
154+ [R] appendChunks :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Array a) -> m ()
155+ [R] appendArray :: FilePath -> Array a -> IO ()
156+ [R] append :: (MonadIO m, MonadCatch m) => FilePath -> Stream m Word8 -> m ()
157+[C] Streamly.Internal.FileSystem.Dir
158+ [C] reader
159+ [O] reader :: MonadIO m => Unfold m FilePath FilePath
160+ [N] reader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
161+ [C] readFiles
162+ [O] readFiles :: MonadIO m => FilePath -> Stream m FilePath
163+ [N] readFiles :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
164+ [C] readEitherPaths
165+ [O] readEitherPaths :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
166+ [N] readEitherPaths :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
167+ [C] readEither
168+ [O] readEither :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
169+ [N] readEither :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
170+ [C] readDirs
171+ [O] readDirs :: MonadIO m => FilePath -> Stream m FilePath
172+ [N] readDirs :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
173+ [C] read
174+ [O] read :: MonadIO m => FilePath -> Stream m FilePath
175+ [N] read :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
176+ [C] fileReader
177+ [O] fileReader :: MonadIO m => Unfold m FilePath FilePath
178+ [N] fileReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
179+ [C] eitherReaderPaths
180+ [O] eitherReaderPaths :: MonadIO m => Unfold m FilePath (Either FilePath FilePath)
181+ [N] eitherReaderPaths :: (MonadIO m, MonadCatch m) => Unfold m FilePath (Either FilePath FilePath)
182+ [C] eitherReader
183+ [O] eitherReader :: MonadIO m => Unfold m FilePath (Either FilePath FilePath)
184+ [N] eitherReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath (Either FilePath FilePath)
185+ [C] dirReader
186+ [O] dirReader :: MonadIO m => Unfold m FilePath FilePath
187+ [N] dirReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
188+[R] Streamly.Internal.Data.Unfold.Type
189+[R] Streamly.Internal.Data.Unfold.Enumeration
190+[C] Streamly.Internal.Data.Unfold
191+ [C] Unfold
192+ [A] Unfold :: (s -> m (Step s b)) -> (a -> m s) -> Unfold m a b
193+ [A] takeWhileMWithInput :: Monad m => (a -> b -> m Bool) -> Unfold m a b -> Unfold m a b
194+ [A] manyInterleave :: Monad m => Unfold m a b -> Unfold m c a -> Unfold m c b
195+ [A] enumerateFromStepIntegral :: (Monad m, Integral a) => Unfold m (a, a) a
196+ [A] crossApplySnd :: Unfold m a b -> Unfold m a c -> Unfold m a c
197+ [A] crossApplyFst :: Unfold m a b -> Unfold m a c -> Unfold m a b
198+ [A] concatMap :: Monad m => (b -> Unfold m a c) -> Unfold m a b -> Unfold m a c
199+[R] Streamly.Internal.Data.Unboxed
200+[C] Streamly.Internal.Data.Time.Units
201+ [R] Streamly.Internal.Data.Unboxed.Unbox
202+ [A] Streamly.Internal.Data.Unbox.Unbox
203+ [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.NanoSecond64
204+ [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.MilliSecond64
205+ [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.MicroSecond64
206+[R] Streamly.Internal.Data.Time.Clock.Type
207+[A] Streamly.Internal.Data.StreamK
208+ [A] CrossStreamK
209+ [A] (FixityR,6)
210+ [A] (FixityR,6)
211+ [A] (FixityR,6)
212+ [A] (FixityR,5)
213+ [A] (FixityR,5)
214+ [A] (FixityR,6)
215+ [A] (FixityR,5)
216+ [A] StreamK
217+ [A] MkStream :: (forall r. State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
218+ [A] zipWithM :: Monad m => (a -> b -> m c) -> StreamK m a -> StreamK m b -> StreamK m c
219+ [A] zipWith :: Monad m => (a -> b -> c) -> StreamK m a -> StreamK m b -> StreamK m c
220+ [A] unfoldrMWith :: Monad m => (m a -> StreamK m a -> StreamK m a) -> (b -> m (Maybe (a, b))) -> b -> StreamK m a
221+ [A] unfoldrM :: Monad m => (b -> m (Maybe (a, b))) -> b -> StreamK m a
222+ [A] unfoldr :: (b -> Maybe (a, b)) -> b -> StreamK m a
223+ [A] uncons :: Applicative m => StreamK m a -> m (Maybe (a, StreamK m a))
224+ [A] unShare :: StreamK m a -> StreamK m a
225+ [A] unCross :: CrossStreamK m a -> StreamK m a
226+ [A] toStream :: Applicative m => StreamK m a -> Stream m a
227+ [A] toList :: Monad m => StreamK m a -> m [a]
228+ [A] the :: (Eq a, Monad m) => StreamK m a -> m (Maybe a)
229+ [A] takeWhile :: (a -> Bool) -> StreamK m a -> StreamK m a
230+ [A] take :: Int -> StreamK m a -> StreamK m a
231+ [A] tail :: Applicative m => StreamK m a -> m (Maybe (StreamK m a))
232+ [A] sortBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> StreamK m a
233+ [A] sequence :: Monad m => StreamK m (m a) -> StreamK m a
234+ [A] scanlx' :: (x -> a -> x) -> x -> (x -> b) -> StreamK m a -> StreamK m b
235+ [A] scanl' :: (b -> a -> b) -> b -> StreamK m a -> StreamK m b
236+ [A] reverse :: StreamK m a -> StreamK m a
237+ [A] replicateMWith :: (m a -> StreamK m a -> StreamK m a) -> Int -> m a -> StreamK m a
238+ [A] replicateM :: Monad m => Int -> m a -> StreamK m a
239+ [A] replicate :: Int -> a -> StreamK m a
240+ [A] repeatMWith :: (m a -> t m a -> t m a) -> m a -> t m a
241+ [A] repeatM :: Monad m => m a -> StreamK m a
242+ [A] repeat :: a -> StreamK m a
243+ [A] parseDBreak :: Monad m => Parser a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
244+ [A] parseD :: Monad m => Parser a m b -> StreamK m a -> m (Either ParseError b)
245+ [A] parseChunksGeneric :: Monad m => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
246+ [A] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
247+ [A] parseBreakChunksGeneric :: forall m a b. Monad m => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
248+ [A] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
249+ [A] parseBreak :: forall m a b. Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
250+ [A] parse :: Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b)
251+ [A] null :: Monad m => StreamK m a -> m Bool
252+ [A] notElem :: (Monad m, Eq a) => a -> StreamK m a -> m Bool
253+ [A] nilM :: Applicative m => m b -> StreamK m a
254+ [A] nil :: StreamK m a
255+ [A] mkStream :: (forall r. State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
256+ [A] mkCross :: StreamK m a -> CrossStreamK m a
257+ [A] minimumBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> m (Maybe a)
258+ [A] minimum :: (Monad m, Ord a) => StreamK m a -> m (Maybe a)
259+ [A] mfix :: Monad m => (m a -> StreamK m a) -> StreamK m a
260+ [A] mergeMapWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m a -> StreamK m b
261+ [A] mergeIterateWith :: (StreamK m a -> StreamK m a -> StreamK m a) -> (a -> StreamK m a) -> StreamK m a -> StreamK m a
262+ [A] mergeByM :: Monad m => (a -> a -> m Ordering) -> StreamK m a -> StreamK m a -> StreamK m a
263+ [A] mergeBy :: (a -> a -> Ordering) -> StreamK m a -> StreamK m a -> StreamK m a
264+ [A] maximumBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> m (Maybe a)
265+ [A] maximum :: (Monad m, Ord a) => StreamK m a -> m (Maybe a)
266+ [A] mapMaybe :: (a -> Maybe b) -> StreamK m a -> StreamK m b
267+ [A] mapM_ :: Monad m => (a -> m b) -> StreamK m a -> m ()
268+ [A] mapMWith :: (m b -> StreamK m b -> StreamK m b) -> (a -> m b) -> StreamK m a -> StreamK m b
269+ [A] mapMSerial :: Monad m => (a -> m b) -> StreamK m a -> StreamK m b
270+ [A] mapM :: Monad m => (a -> m b) -> StreamK m a -> StreamK m b
271+ [A] map :: (a -> b) -> StreamK m a -> StreamK m b
272+ [A] lookup :: (Monad m, Eq a) => a -> StreamK m (a, b) -> m (Maybe b)
273+ [A] liftInner :: (Monad m, MonadTrans t, Monad (t m)) => StreamK m a -> StreamK (t m) a
274+ [A] last :: Monad m => StreamK m a -> m (Maybe a)
275+ [A] iterateMWith :: Monad m => (m a -> StreamK m a -> StreamK m a) -> (a -> m a) -> m a -> StreamK m a
276+ [A] iterateM :: Monad m => (a -> m a) -> m a -> StreamK m a
277+ [A] iterate :: (a -> a) -> a -> StreamK m a
278+ [A] intersperseM :: Monad m => m a -> StreamK m a -> StreamK m a
279+ [A] intersperse :: Monad m => a -> StreamK m a -> StreamK m a
280+ [A] interleaveMin :: StreamK m a -> StreamK m a -> StreamK m a
281+ [A] interleaveFst :: StreamK m a -> StreamK m a -> StreamK m a
282+ [A] interleave :: StreamK m a -> StreamK m a -> StreamK m a
283+ [A] insertBy :: (a -> a -> Ordering) -> a -> StreamK m a -> StreamK m a
284+ [A] init :: Applicative m => StreamK m a -> m (Maybe (StreamK m a))
285+ [A] hoist :: (Monad m, Monad n) => (forall x. m x -> n x) -> StreamK m a -> StreamK n a
286+ [A] head :: Monad m => StreamK m a -> m (Maybe a)
287+ [A] handle :: (MonadCatch m, Exception e) => (e -> m (StreamK m a)) -> StreamK m a -> StreamK m a
288+ [A] fromYieldK :: YieldK m a -> StreamK m a
289+ [A] fromStream :: Monad m => Stream m a -> StreamK m a
290+ [A] fromStopK :: StopK m -> StreamK m a
291+ [A] fromPure :: a -> StreamK m a
292+ [A] fromList :: [a] -> StreamK m a
293+ [A] fromIndicesMWith :: (m a -> StreamK m a -> StreamK m a) -> (Int -> m a) -> StreamK m a
294+ [A] fromIndicesM :: Monad m => (Int -> m a) -> StreamK m a
295+ [A] fromIndices :: (Int -> a) -> StreamK m a
296+ [A] fromFoldableM :: (Foldable f, Monad m) => f (m a) -> StreamK m a
297+ [A] fromFoldable :: Foldable f => f a -> StreamK m a
298+ [A] fromEffect :: Monad m => m a -> StreamK m a
299+ [A] foldrT :: (Monad m, Monad (s m), MonadTrans s) => (a -> s m b -> s m b) -> s m b -> StreamK m a -> s m b
300+ [A] foldrSShared :: (a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
301+ [A] foldrSM :: Monad m => (m a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
302+ [A] foldrS :: (a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
303+ [A] foldrM :: (a -> m b -> m b) -> m b -> StreamK m a -> m b
304+ [A] foldr1 :: Monad m => (a -> a -> a) -> StreamK m a -> m (Maybe a)
305+ [A] foldr :: Monad m => (a -> b -> b) -> b -> StreamK m a -> m b
306+ [A] foldlx' :: forall m a b x. Monad m => (x -> a -> x) -> x -> (x -> b) -> StreamK m a -> m b
307+ [A] foldlT :: (Monad m, Monad (s m), MonadTrans s) => (s m b -> a -> s m b) -> s m b -> StreamK m a -> s m b
308+ [A] foldlS :: (StreamK m b -> a -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
309+ [A] foldlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> StreamK m a -> m b
310+ [A] foldlM' :: Monad m => (b -> a -> m b) -> m b -> StreamK m a -> m b
311+ [A] foldl' :: Monad m => (b -> a -> b) -> b -> StreamK m a -> m b
312+ [A] foldStreamShared :: State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> StreamK m a -> m r
313+ [A] foldStream :: State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> StreamK m a -> m r
314+ [A] foldEither :: Monad m => Fold m a b -> StreamK m a -> m (Either (Fold m a b) (b, StreamK m a))
315+ [A] foldConcat :: Monad m => Producer m a b -> Fold m b c -> StreamK m a -> m (c, StreamK m a)
316+ [A] foldBreak :: Monad m => Fold m a b -> StreamK m a -> m (b, StreamK m a)
317+ [A] fold :: Monad m => Fold m a b -> StreamK m a -> m b
318+ [A] findM :: Monad m => (a -> m Bool) -> StreamK m a -> m (Maybe a)
319+ [A] findIndices :: (a -> Bool) -> StreamK m a -> StreamK m Int
320+ [A] find :: Monad m => (a -> Bool) -> StreamK m a -> m (Maybe a)
321+ [A] filter :: (a -> Bool) -> StreamK m a -> StreamK m a
322+ [A] evalStateT :: Monad m => m s -> StreamK (StateT s m) a -> StreamK m a
323+ [A] elem :: (Monad m, Eq a) => a -> StreamK m a -> m Bool
324+ [A] dropWhile :: (a -> Bool) -> StreamK m a -> StreamK m a
325+ [A] drop :: Int -> StreamK m a -> StreamK m a
326+ [A] drain :: Monad m => StreamK m a -> m ()
327+ [A] deleteBy :: (a -> a -> Bool) -> a -> StreamK m a -> StreamK m a
328+ [A] crossWith :: Monad m => (a -> b -> c) -> StreamK m a -> StreamK m b -> StreamK m c
329+ [A] crossApplyWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> StreamK m (a -> b) -> StreamK m a -> StreamK m b
330+ [A] crossApplySnd :: StreamK m a -> StreamK m b -> StreamK m b
331+ [A] crossApplyFst :: StreamK m a -> StreamK m b -> StreamK m a
332+ [A] crossApply :: StreamK m (a -> b) -> StreamK m a -> StreamK m b
333+ [A] cross :: Monad m => StreamK m a -> StreamK m b -> StreamK m (a, b)
334+ [A] consMBy :: Monad m => (StreamK m a -> StreamK m a -> StreamK m a) -> m a -> StreamK m a -> StreamK m a
335+ [A] consM :: Monad m => m a -> StreamK m a -> StreamK m a
336+ [A] consK :: YieldK m a -> StreamK m a -> StreamK m a
337+ [A] cons :: a -> StreamK m a -> StreamK m a
338+ [A] conjoin :: Monad m => StreamK m a -> StreamK m a -> StreamK m a
339+ [A] concatMapWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m a -> StreamK m b
340+ [A] concatMapEffect :: Monad m => (b -> StreamK m a) -> m b -> StreamK m a
341+ [A] concatMap :: (a -> StreamK m b) -> StreamK m a -> StreamK m b
342+ [A] concatIterateWith :: (StreamK m a -> StreamK m a -> StreamK m a) -> (a -> StreamK m a) -> StreamK m a -> StreamK m a
343+ [A] concatIterateScanWith :: Monad m => (StreamK m a -> StreamK m a -> StreamK m a) -> (b -> a -> m (b, StreamK m a)) -> m b -> StreamK m a -> StreamK m a
344+ [A] concatIterateLeftsWith :: b ~ Either a c => (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m b -> StreamK m b
345+ [A] concatEffect :: Monad m => m (StreamK m a) -> StreamK m a
346+ [A] buildSM :: Monad m => ((m a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a
347+ [A] buildS :: ((a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a
348+ [A] buildM :: Monad m => (forall r. (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
349+ [A] build :: forall m a. (forall b. (a -> b -> b) -> b -> b) -> StreamK m a
350+ [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> StreamK m a) -> StreamK m a
351+ [A] bindWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> StreamK m a -> (a -> StreamK m b) -> StreamK m b
352+ [A] before :: Monad m => m b -> StreamK m a -> StreamK m a
353+ [A] augmentSM :: Monad m => ((m a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a
354+ [A] augmentS :: ((a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a
355+ [A] append :: StreamK m a -> StreamK m a -> StreamK m a
356+ [A] any :: Monad m => (a -> Bool) -> StreamK m a -> m Bool
357+ [A] all :: Monad m => (a -> Bool) -> StreamK m a -> m Bool
358+ [A] (.:) :: a -> StreamK m a -> StreamK m a
359+ [A] (!!) :: Monad m => StreamK m a -> Int -> m (Maybe a)
360+[R] Streamly.Internal.Data.Stream.StreamK.Type
361+[R] Streamly.Internal.Data.Stream.StreamK.Transformer
362+[R] Streamly.Internal.Data.Stream.StreamK
363+[R] Streamly.Internal.Data.Stream.StreamD.Type
364+[R] Streamly.Internal.Data.Stream.StreamD.Transformer
365+[R] Streamly.Internal.Data.Stream.StreamD.Transform
366+[R] Streamly.Internal.Data.Stream.StreamD.Top
367+[R] Streamly.Internal.Data.Stream.StreamD.Step
368+[R] Streamly.Internal.Data.Stream.StreamD.Nesting
369+[R] Streamly.Internal.Data.Stream.StreamD.Lift
370+[R] Streamly.Internal.Data.Stream.StreamD.Generate
371+[R] Streamly.Internal.Data.Stream.StreamD.Exception
372+[R] Streamly.Internal.Data.Stream.StreamD.Eliminate
373+[R] Streamly.Internal.Data.Stream.StreamD.Container
374+[D] Streamly.Internal.Data.Stream.StreamD
375+[R] Streamly.Internal.Data.Stream.Common
376+[R] Streamly.Internal.Data.Stream.Chunked
377+[C] Streamly.Internal.Data.Stream
378+ [A] class Enum a => Enumerable a
379+ [A] Stream
380+ [A] UnStream :: (State StreamK m a -> s -> m (Step s a)) -> s -> Stream m a
381+ [A] Step
382+ [A] Yield :: a -> s -> Step s a
383+ [A] Stop :: Step s a
384+ [A] Skip :: s -> Step s a
385+ [A] InterleaveState
386+ [A] InterleaveSecondOnly :: s2 -> InterleaveState s1 s2
387+ [A] InterleaveSecond :: s1 -> s2 -> InterleaveState s1 s2
388+ [A] InterleaveFirstOnly :: s1 -> InterleaveState s1 s2
389+ [A] InterleaveFirst :: s1 -> s2 -> InterleaveState s1 s2
390+ [A] FoldManyPost
391+ [A] FoldManyPostYield :: b -> FoldManyPost s fs b a -> FoldManyPost s fs b a
392+ [A] FoldManyPostStart :: s -> FoldManyPost s fs b a
393+ [A] FoldManyPostLoop :: s -> fs -> FoldManyPost s fs b a
394+ [A] FoldManyPostDone :: FoldManyPost s fs b a
395+ [A] FoldMany
396+ [A] FoldManyYield :: b -> FoldMany s fs b a -> FoldMany s fs b a
397+ [A] FoldManyStart :: s -> FoldMany s fs b a
398+ [A] FoldManyLoop :: s -> fs -> FoldMany s fs b a
399+ [A] FoldManyFirst :: fs -> s -> FoldMany s fs b a
400+ [A] FoldManyDone :: FoldMany s fs b a
401+ [A] CrossStream
402+ [A] ConcatUnfoldInterleaveState
403+ [A] ConcatUnfoldInterleaveOuter :: o -> [i] -> ConcatUnfoldInterleaveState o i
404+ [A] ConcatUnfoldInterleaveInnerR :: [i] -> [i] -> ConcatUnfoldInterleaveState o i
405+ [A] ConcatUnfoldInterleaveInnerL :: [i] -> [i] -> ConcatUnfoldInterleaveState o i
406+ [A] ConcatUnfoldInterleaveInner :: o -> [i] -> ConcatUnfoldInterleaveState o i
407+ [A] ConcatMapUState
408+ [A] ConcatMapUOuter :: o -> ConcatMapUState o i
409+ [A] ConcatMapUInner :: o -> i -> ConcatMapUState o i
410+ [A] AppendState
411+ [A] AppendSecond :: s2 -> AppendState s1 s2
412+ [A] AppendFirst :: s1 -> AppendState s1 s2
413+ [A] pattern Stream :: (State StreamK m a -> s -> m (Step s a)) -> s -> Stream m a
414+ [A] zipWithM :: Monad m => (a -> b -> m c) -> Stream m a -> Stream m b -> Stream m c
415+ [A] zipWith :: Monad m => (a -> b -> c) -> Stream m a -> Stream m b -> Stream m c
416+ [A] wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
417+ [A] with :: Monad m => (Stream m a -> Stream m (s, a)) -> (((s, a) -> b) -> Stream m (s, a) -> Stream m (s, a)) -> ((s, a) -> b) -> Stream m a -> Stream m a
418+ [A] usingStateT :: Monad m => m s -> (Stream (StateT s m) a -> Stream (StateT s m) a) -> Stream m a -> Stream m a
419+ [A] usingReaderT :: Monad m => m r -> (Stream (ReaderT r m) a -> Stream (ReaderT r m) a) -> Stream m a -> Stream m a
420+ [A] uniqBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a
421+ [A] uniq :: (Eq a, Monad m) => Stream m a -> Stream m a
422+ [A] unionWithStreamGenericBy :: MonadIO m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
423+ [A] unionWithStreamAscBy :: (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
424+ [A] unfoldrM :: Monad m => (s -> m (Maybe (a, s))) -> s -> Stream m a
425+ [A] unfoldr :: Monad m => (s -> Maybe (a, s)) -> s -> Stream m a
426+ [A] unfoldRoundRobin :: Monad m => Unfold m a b -> Stream m a -> Stream m b
427+ [A] unfoldMany :: Monad m => Unfold m a b -> Stream m a -> Stream m b
428+ [A] unfoldIterateDfs :: Monad m => Unfold m a a -> Stream m a -> Stream m a
429+ [A] unfoldIterateBfsRev :: Monad m => Unfold m a a -> Stream m a -> Stream m a
430+ [A] unfoldIterateBfs :: Monad m => Unfold m a a -> Stream m a -> Stream m a
431+ [A] unfoldInterleave :: Monad m => Unfold m a b -> Stream m a -> Stream m b
432+ [A] unfold :: Applicative m => Unfold m a b -> a -> Stream m b
433+ [A] uncons :: Monad m => Stream m a -> m (Maybe (a, Stream m a))
434+ [A] unCross :: CrossStream m a -> Stream m a
435+ [A] transform :: Monad m => Pipe m a b -> Stream m a -> Stream m b
436+ [A] trace_ :: Monad m => m b -> Stream m a -> Stream m a
437+ [A] trace :: Monad m => (a -> m b) -> Stream m a -> Stream m a
438+ [A] toStreamK :: Monad m => Stream m a -> StreamK m a
439+ [A] toListRev :: Monad m => Stream m a -> m [a]
440+ [A] toList :: Monad m => Stream m a -> m [a]
441+ [A] timestamped :: MonadIO m => Stream m a -> Stream m (AbsTime, a)
442+ [A] timestampWith :: MonadIO m => Double -> Stream m a -> Stream m (AbsTime, a)
443+ [A] timesWith :: MonadIO m => Double -> Stream m (AbsTime, RelTime64)
444+ [A] times :: MonadIO m => Stream m (AbsTime, RelTime64)
445+ [A] timeout :: AbsTime -> t m ()
446+ [A] timeIndexed :: MonadIO m => Stream m a -> Stream m (RelTime64, a)
447+ [A] timeIndexWith :: MonadIO m => Double -> Stream m a -> Stream m (RelTime64, a)
448+ [A] the :: (Eq a, Monad m) => Stream m a -> m (Maybe a)
449+ [A] tapOffsetEvery :: Monad m => Int -> Int -> Fold m a b -> Stream m a -> Stream m a
450+ [A] tap :: Monad m => Fold m a b -> Stream m a -> Stream m a
451+ [A] takeWhileM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
452+ [A] takeWhileLast :: (a -> Bool) -> Stream m a -> Stream m a
453+ [A] takeWhileAround :: (a -> Bool) -> Stream m a -> Stream m a
454+ [A] takeWhile :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
455+ [A] takeEndByM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
456+ [A] takeEndBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
457+ [A] take :: Applicative m => Int -> Stream m a -> Stream m a
458+ [A] tail :: Monad m => Stream m a -> m (Maybe (Stream m a))
459+ [A] stripSuffixUnbox :: (MonadIO m, Eq a, Unbox a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
460+ [A] stripSuffix :: (Monad m, Eq a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
461+ [A] stripPrefix :: (Monad m, Eq a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
462+ [A] strideFromThen :: Monad m => Int -> Int -> Stream m a -> Stream m a
463+ [A] splitOnSuffixSeqAny :: [Array a] -> Fold m a b -> Stream m a -> Stream m b
464+ [A] splitOnSuffixSeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) => Bool -> Array a -> Fold m a b -> Stream m a -> Stream m b
465+ [A] splitOnSeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) => Array a -> Fold m a b -> Stream m a -> Stream m b
466+ [A] splitOnPrefix :: (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
467+ [A] splitOnAny :: [Array a] -> Fold m a b -> Stream m a -> Stream m b
468+ [A] splitOn :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
469+ [A] splitInnerBySuffix :: (Monad m, Eq (f a), Monoid (f a)) => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
470+ [A] splitInnerBy :: Monad m => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
471+ [A] slicesBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
472+ [A] sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
473+ [A] sequence :: Monad m => Stream m (m a) -> Stream m a
474+ [A] scanlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> Stream m b
475+ [A] scanlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> Stream m b
476+ [A] scanlMAfter' :: Monad m => (b -> a -> m b) -> m b -> (b -> m b) -> Stream m a -> Stream m b
477+ [A] scanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
478+ [A] scanlM :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
479+ [A] scanl1M' :: Monad m => (a -> a -> m a) -> Stream m a -> Stream m a
480+ [A] scanl1M :: Monad m => (a -> a -> m a) -> Stream m a -> Stream m a
481+ [A] scanl1' :: Monad m => (a -> a -> a) -> Stream m a -> Stream m a
482+ [A] scanl1 :: Monad m => (a -> a -> a) -> Stream m a -> Stream m a
483+ [A] scanl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
484+ [A] scanl :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
485+ [A] scanMaybe :: Monad m => Fold m a (Maybe b) -> Stream m a -> Stream m b
486+ [A] scanMany :: Monad m => Fold m a b -> Stream m a -> Stream m b
487+ [A] scan :: Monad m => Fold m a b -> Stream m a -> Stream m b
488+ [A] runStateT :: Monad m => m s -> Stream (StateT s m) a -> Stream m (s, a)
489+ [A] runReaderT :: Monad m => m s -> Stream (ReaderT s m) a -> Stream m a
490+ [A] runInnerWithState :: Monad m => (forall b. s -> t m b -> m (b, s)) -> m s -> Stream (t m) a -> Stream m (s, a)
491+ [A] runInnerWith :: Monad m => (forall b. t m b -> m b) -> Stream (t m) a -> Stream m a
492+ [A] roundRobin :: Monad m => Stream m a -> Stream m a -> Stream m a
493+ [A] rollingMapM :: Monad m => (Maybe a -> a -> m b) -> Stream m a -> Stream m b
494+ [A] rollingMap2 :: Monad m => (a -> a -> b) -> Stream m a -> Stream m b
495+ [A] rollingMap :: Monad m => (Maybe a -> a -> b) -> Stream m a -> Stream m b
496+ [A] reverseUnbox :: (MonadIO m, Unbox a) => Stream m a -> Stream m a
497+ [A] reverse :: Monad m => Stream m a -> Stream m a
498+ [A] replicateM :: Monad m => Int -> m a -> Stream m a
499+ [A] replicate :: Monad m => Int -> a -> Stream m a
500+ [A] repeated :: Stream m a -> Stream m a
501+ [A] repeatM :: Monad m => m a -> Stream m a
502+ [A] repeat :: Monad m => a -> Stream m a
503+ [A] relTimesWith :: MonadIO m => Double -> Stream m RelTime64
504+ [A] relTimes :: MonadIO m => Stream m RelTime64
505+ [A] refoldMany :: Monad m => Refold m x a b -> m x -> Stream m a -> Stream m b
506+ [A] refoldIterateM :: Monad m => Refold m b a b -> m b -> Stream m a -> Stream m b
507+ [A] reduceIterateBfs :: Monad m => (a -> a -> m a) -> Stream m a -> m (Maybe a)
508+ [A] reassembleBy :: Fold m a b -> (a -> a -> Int) -> Stream m a -> Stream m b
509+ [A] prune :: (a -> Bool) -> Stream m a -> Stream m a
510+ [A] prescanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
511+ [A] prescanl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
512+ [A] postscanlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> Stream m b
513+ [A] postscanlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> Stream m b
514+ [A] postscanlMAfter' :: Monad m => (b -> a -> m b) -> m b -> (b -> m b) -> Stream m a -> Stream m b
515+ [A] postscanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
516+ [A] postscanlM :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
517+ [A] postscanl' :: Monad m => (a -> b -> a) -> a -> Stream m b -> Stream m a
518+ [A] postscanl :: Monad m => (a -> b -> a) -> a -> Stream m b -> Stream m a
519+ [A] postscan :: Monad m => Fold m a b -> Stream m a -> Stream m b
520+ [A] parseSequence :: Stream m (Parser a m b) -> Stream m a -> Stream m b
521+ [A] parseManyTill :: Parser a m b -> Parser a m x -> Stream m a -> Stream m b
522+ [A] parseManyD :: Monad m => Parser a m b -> Stream m a -> Stream m (Either ParseError b)
523+ [A] parseMany :: Monad m => Parser a m b -> Stream m a -> Stream m (Either ParseError b)
524+ [A] parseIterateD :: Monad m => (b -> Parser a m b) -> b -> Stream m a -> Stream m (Either ParseError b)
525+ [A] parseIterate :: Monad m => (b -> Parser a m b) -> b -> Stream m a -> Stream m (Either ParseError b)
526+ [A] parseD :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b)
527+ [A] parseBreakD :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b, Stream m a)
528+ [A] parseBreak :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b, Stream m a)
529+ [A] parse :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b)
530+ [A] onException :: MonadCatch m => m b -> Stream m a -> Stream m a
531+ [A] null :: Monad m => Stream m a -> m Bool
532+ [A] nub :: (Monad m, Ord a) => Stream m a -> Stream m a
533+ [A] notElem :: (Monad m, Eq a) => a -> Stream m a -> m Bool
534+ [A] nilM :: Applicative m => m b -> Stream m a
535+ [A] nil :: Applicative m => Stream m a
536+ [A] morphInner :: Monad n => (forall x. m x -> n x) -> Stream m a -> Stream n a
537+ [A] mkCross :: Stream m a -> CrossStream m a
538+ [A] minimumBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> m (Maybe a)
539+ [A] minimum :: (Monad m, Ord a) => Stream m a -> m (Maybe a)
540+ [A] mergeMinBy :: (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
541+ [A] mergeFstBy :: (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
542+ [A] mergeByM :: Monad m => (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
543+ [A] mergeBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
544+ [A] maximumBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> m (Maybe a)
545+ [A] maximum :: (Monad m, Ord a) => Stream m a -> m (Maybe a)
546+ [A] mapMaybeM :: Monad m => (a -> m (Maybe b)) -> Stream m a -> Stream m b
547+ [A] mapMaybe :: Monad m => (a -> Maybe b) -> Stream m a -> Stream m b
548+ [A] mapM_ :: Monad m => (a -> m b) -> Stream m a -> m ()
549+ [A] mapM :: Monad m => (a -> m b) -> Stream m a -> Stream m b
550+ [A] map :: Monad m => (a -> b) -> Stream m a -> Stream m b
551+ [A] lookup :: (Monad m, Eq a) => a -> Stream m (a, b) -> m (Maybe b)
552+ [A] liftInnerWith :: Monad (t m) => (forall b. m b -> t m b) -> Stream m a -> Stream (t m) a
553+ [A] liftInner :: (Monad m, MonadTrans t, Monad (t m)) => Stream m a -> Stream (t m) a
554+ [A] last :: Monad m => Stream m a -> m (Maybe a)
555+ [A] joinOuterGeneric :: MonadIO m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (Maybe a, Maybe b)
556+ [A] joinOuterAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (Maybe a, Maybe b)
557+ [A] joinOuter :: (Ord k, MonadIO m) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, Maybe a, Maybe b)
558+ [A] joinLeftGeneric :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
559+ [A] joinLeftAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
560+ [A] joinLeft :: (Ord k, Monad m) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, Maybe b)
561+ [A] joinInnerGeneric :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, b)
562+ [A] joinInnerAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, b)
563+ [A] joinInner :: (Monad m, Ord k) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, b)
564+ [A] iterateM :: Monad m => (a -> m a) -> m a -> Stream m a
565+ [A] iterate :: Monad m => (a -> a) -> a -> Stream m a
566+ [A] isSuffixOfUnbox :: (MonadIO m, Eq a, Unbox a) => Stream m a -> Stream m a -> m Bool
567+ [A] isSuffixOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
568+ [A] isSubsequenceOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
569+ [A] isPrefixOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
570+ [A] isInfixOf :: (MonadIO m, Eq a, Enum a, Storable a, Unbox a) => Stream m a -> Stream m a -> m Bool
571+ [A] intersperseM_ :: Monad m => m b -> Stream m a -> Stream m a
572+ [A] intersperseMWith :: Int -> m a -> Stream m a -> Stream m a
573+ [A] intersperseMSuffix_ :: Monad m => m b -> Stream m a -> Stream m a
574+ [A] intersperseMSuffixWith :: forall m a. Monad m => Int -> m a -> Stream m a -> Stream m a
575+ [A] intersperseMSuffix :: forall m a. Monad m => m a -> Stream m a -> Stream m a
576+ [A] intersperseMPrefix_ :: Monad m => m b -> Stream m a -> Stream m a
577+ [A] intersperseM :: Monad m => m a -> Stream m a -> Stream m a
578+ [A] intersperse :: Monad m => a -> Stream m a -> Stream m a
579+ [A] intersectBySorted :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
580+ [A] interposeSuffixM :: Monad m => m c -> Unfold m b c -> Stream m b -> Stream m c
581+ [A] interposeSuffix :: Monad m => c -> Unfold m b c -> Stream m b -> Stream m c
582+ [A] interposeM :: Monad m => m c -> Unfold m b c -> Stream m b -> Stream m c
583+ [A] interpose :: Monad m => c -> Unfold m b c -> Stream m b -> Stream m c
584+ [A] interleaveMin :: Monad m => Stream m a -> Stream m a -> Stream m a
585+ [A] interleaveFstSuffix :: Monad m => Stream m a -> Stream m a -> Stream m a
586+ [A] interleaveFst :: Monad m => Stream m a -> Stream m a -> Stream m a
587+ [A] interleave :: Monad m => Stream m a -> Stream m a -> Stream m a
588+ [A] intercalateSuffix :: Monad m => Unfold m b c -> b -> Stream m b -> Stream m c
589+ [A] intercalate :: Monad m => Unfold m b c -> b -> Stream m b -> Stream m c
590+ [A] insertBy :: Monad m => (a -> a -> Ordering) -> a -> Stream m a -> Stream m a
591+ [A] indexedR :: Monad m => Int -> Stream m a -> Stream m (Int, a)
592+ [A] indexed :: Monad m => Stream m a -> Stream m (Int, a)
593+ [A] headElse :: Monad m => a -> Stream m a -> m a
594+ [A] head :: Monad m => Stream m a -> m (Maybe a)
595+ [A] handle :: (MonadCatch m, Exception e) => (e -> m (Stream m a)) -> Stream m a -> Stream m a
596+ [A] groupsWhile :: Monad m => (a -> a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
597+ [A] groupsRollingBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
598+ [A] groupsOf :: Monad m => Int -> Fold m a b -> Stream m a -> Stream m b
599+ [A] gintercalateSuffix :: Monad m => Unfold m a c -> Stream m a -> Unfold m b c -> Stream m b -> Stream m c
600+ [A] gintercalate :: Monad m => Unfold m a c -> Stream m a -> Unfold m b c -> Stream m b -> Stream m c
601+ [A] ghandle :: (MonadCatch m, Exception e) => (e -> Stream m a -> m (Stream m a)) -> Stream m a -> Stream m a
602+ [A] generateM :: Monad m => Int -> (Int -> m a) -> Stream m a
603+ [A] generate :: Monad m => Int -> (Int -> a) -> Stream m a
604+ [A] generalizeInner :: Monad m => Stream Identity a -> Stream m a
605+ [A] gbracket_ :: Monad m => m c -> (c -> m d) -> (c -> e -> Stream m b -> m (Stream m b)) -> (forall s. m s -> m (Either e s)) -> (c -> Stream m b) -> Stream m b
606+ [A] gbracket :: MonadIO m => IO c -> (c -> IO d1) -> (c -> e -> Stream m b -> IO (Stream m b)) -> (c -> IO d2) -> (forall s. m s -> m (Either e s)) -> (c -> Stream m b) -> Stream m b
607+ [A] fromStreamK :: Applicative m => StreamK m a -> Stream m a
608+ [A] fromPure :: Applicative m => a -> Stream m a
609+ [A] fromPtrN :: (Monad m, Storable a) => Int -> Ptr a -> Stream m a
610+ [A] fromPtr :: forall m a. (Monad m, Storable a) => Ptr a -> Stream m a
611+ [A] fromListM :: Monad m => [m a] -> Stream m a
612+ [A] fromList :: Applicative m => [a] -> Stream m a
613+ [A] fromIndicesM :: Monad m => (Int -> m a) -> Stream m a
614+ [A] fromIndices :: Monad m => (Int -> a) -> Stream m a
615+ [A] fromFoldableM :: (Monad m, Foldable f) => f (m a) -> Stream m a
616+ [A] fromFoldable :: (Monad m, Foldable f) => f a -> Stream m a
617+ [A] fromEffect :: Applicative m => m a -> Stream m a
618+ [A] fromByteStr# :: Monad m => Addr# -> Stream m Word8
619+ [A] foldrT :: (Monad m, Monad (t m), MonadTrans t) => (a -> t m b -> t m b) -> t m b -> Stream m a -> t m b
620+ [A] foldrS :: Monad m => (a -> Stream m b -> Stream m b) -> Stream m b -> Stream m a -> Stream m b
621+ [A] foldrMx :: Monad m => (a -> m x -> m x) -> m x -> (m x -> m b) -> Stream m a -> m b
622+ [A] foldrM :: Monad m => (a -> m b -> m b) -> m b -> Stream m a -> m b
623+ [A] foldr1 :: Monad m => (a -> a -> a) -> Stream m a -> m (Maybe a)
624+ [A] foldr :: Monad m => (a -> b -> b) -> b -> Stream m a -> m b
625+ [A] foldlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> m b
626+ [A] foldlT :: (Monad m, Monad (s m), MonadTrans s) => (s m b -> a -> s m b) -> s m b -> Stream m a -> s m b
627+ [A] foldlS :: Monad m => (Stream m b -> a -> Stream m b) -> Stream m b -> Stream m a -> Stream m b
628+ [A] foldlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> m b
629+ [A] foldlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> m b
630+ [A] foldl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> m b
631+ [A] foldSequence :: Stream m (Fold m a b) -> Stream m a -> Stream m b
632+ [A] foldManyPost :: Monad m => Fold m a b -> Stream m a -> Stream m b
633+ [A] foldMany :: Monad m => Fold m a b -> Stream m a -> Stream m b
634+ [A] foldIterateM :: Monad m => (b -> m (Fold m a b)) -> m b -> Stream m a -> Stream m b
635+ [A] foldIterateBfs :: Fold m a (Either a a) -> Stream m a -> m (Maybe a)
636+ [A] foldEither :: Monad m => Fold m a b -> Stream m a -> m (Either (Fold m a b) (b, Stream m a))
637+ [A] foldBreak :: Monad m => Fold m a b -> Stream m a -> m (b, Stream m a)
638+ [A] foldAddLazy :: Monad m => Fold m a b -> Stream m a -> Fold m a b
639+ [A] foldAdd :: Monad m => Fold m a b -> Stream m a -> m (Fold m a b)
640+ [A] fold :: Monad m => Fold m a b -> Stream m a -> m b
641+ [A] findM :: Monad m => (a -> m Bool) -> Stream m a -> m (Maybe a)
642+ [A] findIndices :: Monad m => (a -> Bool) -> Stream m a -> Stream m Int
643+ [A] find :: Monad m => (a -> Bool) -> Stream m a -> m (Maybe a)
644+ [A] finallyUnsafe :: MonadCatch m => m b -> Stream m a -> Stream m a
645+ [A] finallyIO :: (MonadIO m, MonadCatch m) => IO b -> Stream m a -> Stream m a
646+ [A] filterM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
647+ [A] filterInStreamGenericBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
648+ [A] filterInStreamAscBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
649+ [A] filter :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
650+ [A] evalStateT :: Monad m => m s -> Stream (StateT s m) a -> Stream m a
651+ [A] eqBy :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> m Bool
652+ [A] enumerateTo :: (Monad m, Bounded a, Enumerable a) => a -> Stream m a
653+ [A] enumerateFromToSmall :: (Monad m, Enum a) => a -> a -> Stream m a
654+ [A] enumerateFromToIntegral :: (Monad m, Integral a) => a -> a -> Stream m a
655+ [A] enumerateFromToFractional :: (Monad m, Fractional a, Ord a) => a -> a -> Stream m a
656+ [A] enumerateFromTo :: (Enumerable a, Monad m) => a -> a -> Stream m a
657+ [A] enumerateFromThenToSmall :: (Monad m, Enum a) => a -> a -> a -> Stream m a
658+ [A] enumerateFromThenToIntegral :: (Monad m, Integral a) => a -> a -> a -> Stream m a
659+ [A] enumerateFromThenToFractional :: (Monad m, Fractional a, Ord a) => a -> a -> a -> Stream m a
660+ [A] enumerateFromThenTo :: (Enumerable a, Monad m) => a -> a -> a -> Stream m a
661+ [A] enumerateFromThenSmallBounded :: (Monad m, Enumerable a, Bounded a) => a -> a -> Stream m a
662+ [A] enumerateFromThenNum :: (Monad m, Num a) => a -> a -> Stream m a
663+ [A] enumerateFromThenIntegral :: (Monad m, Integral a, Bounded a) => a -> a -> Stream m a
664+ [A] enumerateFromThenFractional :: (Monad m, Fractional a) => a -> a -> Stream m a
665+ [A] enumerateFromThen :: (Enumerable a, Monad m) => a -> a -> Stream m a
666+ [A] enumerateFromStepNum :: (Monad m, Num a) => a -> a -> Stream m a
667+ [A] enumerateFromStepIntegral :: (Integral a, Monad m) => a -> a -> Stream m a
668+ [A] enumerateFromNum :: (Monad m, Num a) => a -> Stream m a
669+ [A] enumerateFromIntegral :: (Monad m, Integral a, Bounded a) => a -> Stream m a
670+ [A] enumerateFromFractional :: (Monad m, Fractional a) => a -> Stream m a
671+ [A] enumerateFromBounded :: (Monad m, Enumerable a, Bounded a) => a -> Stream m a
672+ [A] enumerateFrom :: (Enumerable a, Monad m) => a -> Stream m a
673+ [A] enumerate :: (Monad m, Bounded a, Enumerable a) => Stream m a
674+ [A] elemIndices :: (Monad m, Eq a) => a -> Stream m a -> Stream m Int
675+ [A] elem :: (Monad m, Eq a) => a -> Stream m a -> m Bool
676+ [A] durations :: Double -> t m RelTime64
677+ [A] dropWhileM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
678+ [A] dropWhileLast :: (a -> Bool) -> Stream m a -> Stream m a
679+ [A] dropWhileAround :: (a -> Bool) -> Stream m a -> Stream m a
680+ [A] dropWhile :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
681+ [A] dropSuffix :: Stream m a -> Stream m a -> Stream m a
682+ [A] dropPrefix :: Stream m a -> Stream m a -> Stream m a
683+ [A] dropLast :: Int -> Stream m a -> Stream m a
684+ [A] dropInfix :: Stream m a -> Stream m a -> Stream m a
685+ [A] drop :: Monad m => Int -> Stream m a -> Stream m a
686+ [A] drain :: Monad m => Stream m a -> m ()
687+ [A] deleteInStreamGenericBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
688+ [A] deleteInStreamAscBy :: (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
689+ [A] deleteBy :: Monad m => (a -> a -> Bool) -> a -> Stream m a -> Stream m a
690+ [A] delayPre :: MonadIO m => Double -> Stream m a -> Stream m a
691+ [A] delayPost :: MonadIO m => Double -> Stream m a -> Stream m a
692+ [A] delay :: MonadIO m => Double -> Stream m a -> Stream m a
693+ [A] crossWith :: Monad m => (a -> b -> c) -> Stream m a -> Stream m b -> Stream m c
694+ [A] crossApplySnd :: Functor f => Stream f a -> Stream f b -> Stream f b
695+ [A] crossApplyFst :: Functor f => Stream f a -> Stream f b -> Stream f a
696+ [A] crossApply :: Functor f => Stream f (a -> b) -> Stream f a -> Stream f b
697+ [A] cross :: Monad m => Stream m a -> Stream m b -> Stream m (a, b)
698+ [A] consM :: Applicative m => m a -> Stream m a -> Stream m a
699+ [A] cons :: Applicative m => a -> Stream m a -> Stream m a
700+ [A] concatMapM :: Monad m => (a -> m (Stream m b)) -> Stream m a -> Stream m b
701+ [A] concatMap :: Monad m => (a -> Stream m b) -> Stream m a -> Stream m b
702+ [A] concatIterateScan :: Monad m => (b -> a -> m b) -> (b -> m (Maybe (b, Stream m a))) -> b -> Stream m a
703+ [A] concatIterateDfs :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
704+ [A] concatIterateBfsRev :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
705+ [A] concatIterateBfs :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
706+ [A] concatEffect :: Monad m => m (Stream m a) -> Stream m a
707+ [A] concat :: Monad m => Stream m (Stream m a) -> Stream m a
708+ [A] cmpBy :: Monad m => (a -> b -> Ordering) -> Stream m a -> Stream m b -> m Ordering
709+ [A] catRights :: Monad m => Stream m (Either a b) -> Stream m b
710+ [A] catMaybes :: Monad m => Stream m (Maybe a) -> Stream m a
711+ [A] catLefts :: Monad m => Stream m (Either a b) -> Stream m a
712+ [A] catEithers :: Monad m => Stream m (Either a a) -> Stream m a
713+ [A] bracketUnsafe :: MonadCatch m => m b -> (b -> m c) -> (b -> Stream m a) -> Stream m a
714+ [A] bracketIO3 :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> IO d) -> (b -> IO e) -> (b -> Stream m a) -> Stream m a
715+ [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> Stream m a) -> Stream m a
716+ [A] before :: Monad m => m b -> Stream m a -> Stream m a
717+ [A] append :: Monad m => Stream m a -> Stream m a -> Stream m a
718+ [A] any :: Monad m => (a -> Bool) -> Stream m a -> m Bool
719+ [A] all :: Monad m => (a -> Bool) -> Stream m a -> m Bool
720+ [A] afterUnsafe :: Monad m => m b -> Stream m a -> Stream m a
721+ [A] afterIO :: MonadIO m => IO b -> Stream m a -> Stream m a
722+ [A] absTimesWith :: MonadIO m => Double -> Stream m AbsTime
723+ [A] absTimes :: MonadIO m => Stream m AbsTime
724+ [A] (!!) :: Monad m => Stream m a -> Int -> m (Maybe a)
725+[R] Streamly.Internal.Data.Ring.Unboxed
726+[A] Streamly.Internal.Data.Ring.Generic
727+ [A] Ring
728+ [A] [ringMax] :: Ring a -> !Int
729+ [A] [ringHead] :: Ring a -> !Int
730+ [A] [ringArr] :: Ring a -> MutArray a
731+ [A] Ring :: MutArray a -> !Int -> !Int -> Ring a
732+ [A] writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
733+ [A] unsafeInsertRingWith :: Ring a -> a -> IO Int
734+ [A] toStreamWith :: Int -> Ring a -> Stream m a
735+ [A] toMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
736+ [A] seek :: MonadIO m => Int -> Ring a -> m (Ring a)
737+ [A] createRing :: MonadIO m => Int -> m (Ring a)
738+ [A] copyToMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
739+[C] Streamly.Internal.Data.Ring
740+ [C] Ring
741+ [A] [ringStart] :: Ring a -> {-# UNPACK #-} !ForeignPtr a
742+ [R] [ringMax] :: Ring a -> !Int
743+ [R] [ringHead] :: Ring a -> !Int
744+ [A] [ringBound] :: Ring a -> {-# UNPACK #-} !Ptr a
745+ [R] [ringArr] :: Ring a -> MutArray a
746+ [C] Ring
747+ [O] Ring :: MutArray a -> !Int -> !Int -> Ring a
748+ [N] Ring :: {-# UNPACK #-} !ForeignPtr a -> {-# UNPACK #-} !Ptr a -> Ring a
749+ [A] GHC.Show.Show
750+ [A] instance (GHC.Show.Show a, GHC.Show.Show b, GHC.Show.Show c, GHC.Show.Show d) => GHC.Show.Show (Streamly.Internal.Data.Ring.Tuple4' a b c d)
751+ [A] writeN :: Int -> Fold m a (Ring a)
752+ [R] writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
753+ [R] unsafeInsertRingWith :: Ring a -> a -> IO Int
754+ [A] unsafeInsert :: Storable a => Ring a -> Ptr a -> a -> IO (Ptr a)
755+ [A] unsafeFoldRingNM :: forall m a b. (MonadIO m, Storable a) => Int -> Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
756+ [A] unsafeFoldRingM :: forall m a b. (MonadIO m, Storable a) => Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
757+ [A] unsafeFoldRingFullM :: forall m a b. (MonadIO m, Storable a) => Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
758+ [A] unsafeFoldRing :: forall a b. Storable a => Ptr a -> (b -> a -> b) -> b -> Ring a -> b
759+ [A] unsafeEqArrayN :: Ring a -> Ptr a -> Array a -> Int -> Bool
760+ [A] unsafeEqArray :: Ring a -> Ptr a -> Array a -> Bool
761+ [R] toStreamWith :: Int -> Ring a -> Stream m a
762+ [R] toMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
763+ [A] startOf :: Ring a -> Ptr a
764+ [A] slidingWindowWith :: forall m a b. (MonadIO m, Storable a, Unbox a) => Int -> Fold m ((a, Maybe a), m (MutArray a)) b -> Fold m a b
765+ [A] slidingWindow :: forall m a b. (MonadIO m, Storable a, Unbox a) => Int -> Fold m (a, Maybe a) b -> Fold m a b
766+ [A] slide :: Ring a -> a -> m (Ring a)
767+ [R] seek :: MonadIO m => Int -> Ring a -> m (Ring a)
768+ [A] ringsOf :: Int -> Stream m a -> Stream m (MutArray a)
769+ [A] readRev :: Unfold m (MutArray a) a
770+ [A] read :: forall m a. (MonadIO m, Storable a) => Unfold m (Ring a, Ptr a, Int) a
771+ [A] putIndex :: Ring a -> Int -> a -> m ()
772+ [A] newRing :: Int -> m (Ring a)
773+ [A] new :: forall a. Storable a => Int -> IO (Ring a, Ptr a)
774+ [A] moveBy :: forall a. Storable a => Int -> Ring a -> Ptr a -> Ptr a
775+ [A] modifyIndex :: Ring a -> Int -> (a -> (a, b)) -> m b
776+ [A] length :: Ring a -> Int
777+ [A] getIndexUnsafe :: Ring a -> Int -> m a
778+ [A] getIndexRev :: Ring a -> Int -> m a
779+ [A] getIndex :: Ring a -> Int -> m a
780+ [A] fromArray :: MutArray a -> Ring a
781+ [R] createRing :: MonadIO m => Int -> m (Ring a)
782+ [A] castUnsafe :: Ring a -> Ring b
783+ [A] cast :: forall a b. Storable b => Ring a -> Maybe (Ring b)
784+ [A] bytesFree :: Ring a -> Int
785+ [A] byteLength :: Ring a -> Int
786+ [A] byteCapacity :: Ring a -> Int
787+ [A] asBytes :: Ring a -> Ring Word8
788+ [A] advance :: forall a. Storable a => Ring a -> Ptr a -> Ptr a
789+[R] Streamly.Internal.Data.Producer.Type
790+[R] Streamly.Internal.Data.Producer.Source
791+[C] Streamly.Internal.Data.Producer
792+ [A] Source
793+ [A] unread :: [b] -> Source a b -> Source a b
794+ [A] translate :: Functor m => (a -> c) -> (c -> a) -> Producer m c b -> Producer m a b
795+ [A] source :: Maybe a -> Source a b
796+ [A] producer :: Monad m => Producer m a b -> Producer m (Source a b) b
797+ [A] parseManyD :: Monad m => Parser a m b -> Producer m (Source x a) a -> Producer m (Source x a) (Either ParseError b)
798+ [A] parseMany :: Monad m => Parser a m b -> Producer m (Source x a) a -> Producer m (Source x a) (Either ParseError b)
799+ [A] parse :: Monad m => Parser a m b -> Producer m (Source s a) a -> Source s a -> m (Either ParseError b, Source s a)
800+ [A] lmap :: (a -> a) -> Producer m a b -> Producer m a b
801+ [A] isEmpty :: Source a b -> Bool
802+[R] Streamly.Internal.Data.Pipe.Type
803+[C] Streamly.Internal.Data.Pipe
804+ [A] Step
805+ [A] Yield :: a -> s -> Step s a
806+ [A] Continue :: s -> Step s a
807+ [A] PipeState
808+ [A] Produce :: s2 -> PipeState s1 s2
809+ [A] Consume :: s1 -> PipeState s1 s2
810+ [C] Pipe
811+ [A] Pipe :: (s1 -> a -> m (Step (PipeState s1 s2) b)) -> (s2 -> m (Step (PipeState s1 s2) b)) -> s1 -> Pipe m a b
812+[A] Streamly.Internal.Data.ParserK
813+ [A] Step
814+ [A] Partial :: !Int -> (Input a -> m (Step a m r)) -> Step a m r
815+ [A] Error :: !Int -> String -> Step a m r
816+ [A] Done :: !Int -> r -> Step a m r
817+ [A] Continue :: !Int -> (Input a -> m (Step a m r)) -> Step a m r
818+ [A] ParseResult
819+ [A] Success :: !Int -> !b -> ParseResult b
820+ [A] Failure :: !Int -> !String -> ParseResult b
821+ [A] Input
822+ [A] None :: Input a
823+ [A] Chunk :: a -> Input a
824+ [A] ParserK
825+ [A] [runParser] :: ParserK a m b -> forall r. (ParseResult b -> Int -> Input a -> m (Step a m r)) -> Int -> Int -> Input a -> m (Step a m r)
826+ [A] MkParser :: (forall r. (ParseResult b -> Int -> Input a -> m (Step a m r)) -> Int -> Int -> Input a -> m (Step a m r)) -> ParserK a m b
827+ [A] fromPure :: b -> ParserK a m b
828+ [A] fromEffect :: Monad m => m b -> ParserK a m b
829+ [A] die :: String -> ParserK a m b
830+ [A] adaptCG :: Monad m => Parser a m b -> ParserK (Array a) m b
831+ [A] adaptC :: (Monad m, Unbox a) => Parser a m b -> ParserK (Array a) m b
832+ [A] adapt :: Monad m => Parser a m b -> ParserK a m b
833+[R] Streamly.Internal.Data.Parser.ParserK.Type
834+[R] Streamly.Internal.Data.Parser.ParserD.Type
835+[R] Streamly.Internal.Data.Parser.ParserD.Tee
836+[R] Streamly.Internal.Data.Parser.ParserD
837+[C] Streamly.Internal.Data.Parser
838+ [A] Step
839+ [A] Partial :: !Int -> !s -> Step s b
840+ [A] Error :: !String -> Step s b
841+ [A] Done :: !Int -> !b -> Step s b
842+ [A] Continue :: !Int -> !s -> Step s b
843+ [A] Parser
844+ [A] Parser :: (s -> a -> m (Step s b)) -> m (Initial s b) -> (s -> m (Step s b)) -> Parser a m b
845+ [A] Initial
846+ [A] IPartial :: !s -> Initial s b
847+ [A] IError :: !String -> Initial s b
848+ [A] IDone :: !b -> Initial s b
849+ [A] GHC.Show.Show
850+ [A] instance (GHC.Show.Show a, GHC.Show.Show b) => GHC.Show.Show (Streamly.Internal.Data.Parser.Tuple'Fused a b)
851+ [A] ParseError
852+ [A] ParseError :: String -> ParseError
853+ [A] zipWithM :: Monad m => (a -> b -> m c) -> Stream m a -> Fold m c x -> Parser b m x
854+ [A] zip :: Monad m => Stream m a -> Fold m (a, b) x -> Parser b m x
855+ [A] wordWithQuotes :: (Monad m, Eq a) => Bool -> (a -> a -> Maybe a) -> a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
856+ [A] wordProcessQuotes :: (Monad m, Eq a) => a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
857+ [A] wordKeepQuotes :: (Monad m, Eq a) => a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
858+ [A] wordFramedBy :: Monad m => (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
859+ [A] wordBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
860+ [A] toFold :: Monad m => Parser a m b -> Fold m a b
861+ [A] takeWhileP :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
862+ [A] takeWhile1 :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
863+ [A] takeWhile :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
864+ [A] takeStartBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
865+ [A] takeStartBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
866+ [A] takeP :: Monad m => Int -> Parser a m b -> Parser a m b
867+ [A] takeGE :: Monad m => Int -> Fold m a b -> Parser a m b
868+ [A] takeFramedBy_ :: Monad m => (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
869+ [A] takeFramedByGeneric :: Monad m => Maybe (a -> Bool) -> Maybe (a -> Bool) -> Maybe (a -> Bool) -> Fold m a b -> Parser a m b
870+ [A] takeFramedByEsc_ :: Monad m => (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
871+ [A] takeEndBy_ :: (a -> Bool) -> Parser a m b -> Parser a m b
872+ [A] takeEndByEsc :: Monad m => (a -> Bool) -> (a -> Bool) -> Parser a m b -> Parser a m b
873+ [A] takeEndBy :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
874+ [A] takeEitherSepBy :: (a -> Bool) -> Fold m (Either a b) c -> Parser a m c
875+ [A] takeEQ :: Monad m => Int -> Fold m a b -> Parser a m b
876+ [A] takeBetween :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
877+ [A] subsequenceBy :: (a -> a -> Bool) -> Stream m a -> Parser a m ()
878+ [A] streamEqBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Parser a m ()
879+ [A] split_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
880+ [A] splitWith :: Monad m => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
881+ [A] splitSome :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
882+ [A] splitManyPost :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
883+ [A] splitMany :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
884+ [A] spanByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
885+ [A] spanBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
886+ [A] span :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
887+ [A] some :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
888+ [A] sequence :: Monad m => Stream m (Parser a m b) -> Fold m b c -> Parser a m c
889+ [A] sepByAll :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
890+ [A] sepBy1 :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
891+ [A] sepBy :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
892+ [A] satisfy :: Monad m => (a -> Bool) -> Parser a m a
893+ [A] sampleFromthen :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
894+ [A] roundRobin :: t (Parser a m b) -> Fold m b c -> Parser a m c
895+ [A] rmapM :: Monad m => (b -> m c) -> Parser a m b -> Parser a m c
896+ [A] retryMaxTotal :: Int -> Parser a m b -> Fold m b c -> Parser a m c
897+ [A] retryMaxSuccessive :: Int -> Parser a m b -> Fold m b c -> Parser a m c
898+ [A] retry :: Parser a m b -> Parser a m b
899+ [A] postscan :: Fold m a b -> Parser b m c -> Parser a m c
900+ [A] peek :: Monad m => Parser a m a
901+ [A] oneOf :: (Monad m, Eq a, Foldable f) => f a -> Parser a m a
902+ [A] oneNotEq :: (Monad m, Eq a) => a -> Parser a m a
903+ [A] oneEq :: (Monad m, Eq a) => a -> Parser a m a
904+ [A] one :: Monad m => Parser a m a
905+ [A] noneOf :: (Monad m, Eq a, Foldable f) => f a -> Parser a m a
906+ [A] noErrorUnsafeSplit_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
907+ [A] noErrorUnsafeSplitWith :: Monad m => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
908+ [A] noErrorUnsafeConcatMap :: Monad m => (b -> Parser a m c) -> Parser a m b -> Parser a m c
909+ [A] maybe :: Monad m => (a -> Maybe b) -> Parser a m b
910+ [A] manyTillP :: Parser a m b -> Parser a m x -> Parser b m c -> Parser a m c
911+ [A] manyTill :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
912+ [A] manyThen :: Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
913+ [A] manyP :: Parser a m b -> Parser b m c -> Parser a m c
914+ [A] many :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
915+ [A] makeIndexFilter :: (Fold m (s, a) b -> Parser a m b) -> (((s, a) -> Bool) -> Fold m (s, a) b -> Fold m (s, a) b) -> ((s, a) -> Bool) -> Fold m a b -> Parser a m b
916+ [A] lookAhead :: Monad m => Parser a m b -> Parser a m b
917+ [A] lmapM :: Monad m => (a -> m b) -> Parser b m r -> Parser a m r
918+ [A] lmap :: (a -> b) -> Parser b m r -> Parser a m r
919+ [A] listEqBy :: Monad m => (a -> a -> Bool) -> [a] -> Parser a m [a]
920+ [A] listEq :: (Monad m, Eq a) => [a] -> Parser a m [a]
921+ [A] indexed :: forall m a b. Monad m => Fold m (Int, a) b -> Parser a m b
922+ [A] groupByRollingEither :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
923+ [A] groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
924+ [A] groupBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
925+ [A] fromPure :: Monad m => b -> Parser a m b
926+ [A] fromFoldMaybe :: Monad m => String -> Fold m a (Maybe b) -> Parser a m b
927+ [A] fromFold :: Monad m => Fold m a b -> Parser a m b
928+ [A] fromEffect :: Monad m => m b -> Parser a m b
929+ [A] filter :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
930+ [A] extractStep :: Monad m => (s -> m (Step s1 b)) -> Step s b -> m (Step s1 b)
931+ [A] eof :: Monad m => Parser a m ()
932+ [A] either :: Monad m => (a -> Either String b) -> Parser a m b
933+ [A] dropWhile :: Monad m => (a -> Bool) -> Parser a m ()
934+ [A] dieM :: Monad m => m String -> Parser a m b
935+ [A] die :: Monad m => String -> Parser a m b
936+ [A] deintercalateAll :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
937+ [A] deintercalate1 :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
938+ [A] deintercalate :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
939+ [A] countBetween :: Int -> Int -> Parser a m b -> Fold m b c -> Parser a m c
940+ [A] count :: Int -> Parser a m b -> Fold m b c -> Parser a m c
941+ [A] concatMap :: Monad m => (b -> Parser a m c) -> Parser a m b -> Parser a m c
942+ [A] blockWithQuotes :: (Monad m, Eq a) => (a -> Bool) -> (a -> Bool) -> a -> a -> Fold m a b -> Parser a m b
943+ [A] bimapOverrideCount :: Int -> (s -> s1) -> (b -> b1) -> Step s b -> Step s1 b1
944+ [A] alt :: Monad m => Parser x m a -> Parser x m a -> Parser x m a
945+[A] Streamly.Internal.Data.MutByteArray
946+ [A] class Unbox a
947+ [A] class SizeOfRep (f :: Type -> Type)
948+ [A] class Serialize a
949+ [A] TypeOfType
950+ [A] UnitType :: Name -> TypeOfType
951+ [A] TheType :: SimpleDataCon -> TypeOfType
952+ [A] MultiType :: [SimpleDataCon] -> TypeOfType
953+ [A] SimpleDataCon
954+ [A] SimpleDataCon :: Name -> [Field] -> SimpleDataCon
955+ [A] SerializeConfig
956+ [A] [cfgRecordSyntaxWithHeader] :: SerializeConfig -> Bool
957+ [A] [cfgInlineSize] :: SerializeConfig -> Maybe Inline
958+ [A] [cfgInlineSerialize] :: SerializeConfig -> Maybe Inline
959+ [A] [cfgInlineDeserialize] :: SerializeConfig -> Maybe Inline
960+ [A] [cfgConstructorTagAsString] :: SerializeConfig -> Bool
961+ [A] SerializeConfig :: Maybe Inline -> Maybe Inline -> Maybe Inline -> Bool -> Bool -> SerializeConfig
962+ [A] PinnedState
963+ [A] Unpinned :: PinnedState
964+ [A] Pinned :: PinnedState
965+ [A] MutByteArray
966+ [A] MutByteArray :: MutableByteArray# RealWorld -> MutByteArray
967+ [A] DataType
968+ [A] [dtTvs] :: DataType -> [Name]
969+ [A] [dtName] :: DataType -> Name
970+ [A] [dtCxt] :: DataType -> Cxt
971+ [A] [dtCons] :: DataType -> [DataCon]
972+ [A] DataType :: Name -> [Name] -> Cxt -> [DataCon] -> DataType
973+ [A] DataCon
974+ [A] [dcTvs] :: DataCon -> [Name]
975+ [A] [dcName] :: DataCon -> Name
976+ [A] [dcFields] :: DataCon -> [(Maybe Name, Type)]
977+ [A] [dcCxt] :: DataCon -> Cxt
978+ [A] DataCon :: Name -> [Name] -> Cxt -> [(Maybe Name, Type)] -> DataCon
979+ [A] BoundedPtr
980+ [A] BoundedPtr :: MutByteArray -> Int -> Int -> BoundedPtr
981+ [A] Streamly.Internal.Data.Serialize.Type.Serialize
982+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize Streamly.Internal.Data.MutByteArray.LiftedInteger
983+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize GHC.Num.Integer.Integer
984+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize a => Streamly.Internal.Data.Serialize.Type.Serialize (GHC.Maybe.Maybe a)
985+ [A] instance Streamly.Internal.Data.Serialize.Type.Serialize (Data.Proxy.Proxy a)
986+ [A] instance (Streamly.Internal.Data.Serialize.Type.Serialize a, Streamly.Internal.Data.Serialize.Type.Serialize b) => Streamly.Internal.Data.Serialize.Type.Serialize (Data.Either.Either a b)
987+ [A] Peeker
988+ [A] Peeker :: Builder BoundedPtr IO a -> Peeker a
989+ [A] type Field = (Maybe Name, Type)
990+ [A] xorCmp :: [Word8] -> Name -> Name -> Q Exp
991+ [A] wListToString :: [Word8] -> String
992+ [A] w8_int :: Word8 -> Int
993+ [A] w32_int :: Word32 -> Int
994+ [A] unpin :: MutByteArray -> IO MutByteArray
995+ [A] typeOfType :: Type -> [DataCon] -> TypeOfType
996+ [A] skipByte :: Peeker ()
997+ [A] sizeOfRep :: SizeOfRep f => f x -> Int
998+ [A] sizeOfMutableByteArray :: MutByteArray -> IO Int
999+ [A] sizeOf :: (Unbox a, SizeOfRep (Rep a)) => Proxy a -> Int
1000+ [A] simplifyDataCon :: DataCon -> SimpleDataCon
1001+ [A] serializeW8List :: Name -> Name -> [Word8] -> Q Exp
1002+ [A] serializeConfig :: SerializeConfig
1003+ [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
1004+ [A] runPeeker :: Peeker a -> BoundedPtr -> IO a
1005+ [A] reifyDataType :: Name -> Q DataType
1006+ [A] readUnsafe :: Unbox a => Peeker a
1007+ [A] read :: Unbox a => Peeker a
1008+ [A] putSliceUnsafe :: MonadIO m => MutByteArray -> Int -> MutByteArray -> Int -> Int -> m ()
1009+ [A] pokeRep :: PokeRep f => f a -> BoundedPtr -> IO BoundedPtr
1010+ [A] pokeBoundedPtrUnsafe :: forall a. Unbox a => a -> BoundedPtr -> IO BoundedPtr
1011+ [A] pokeBoundedPtr :: forall a. Unbox a => a -> BoundedPtr -> IO BoundedPtr
1012+ [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
1013+ [A] pinnedNewAlignedBytes :: Int -> Int -> IO MutByteArray
1014+ [A] pinnedNew :: Int -> IO MutByteArray
1015+ [A] pinnedCloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
1016+ [A] pin :: MutByteArray -> IO MutByteArray
1017+ [A] peekRep :: PeekRep f => Peeker (f x)
1018+ [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
1019+ [A] openConstructor :: Name -> Int -> Q Pat
1020+ [A] nil :: MutByteArray
1021+ [A] newBytesAs :: PinnedState -> Int -> IO MutByteArray
1022+ [A] new :: Int -> IO MutByteArray
1023+ [A] mkSerializeExprFields :: Name -> [Field] -> Q Exp
1024+ [A] mkRecSizeOfExpr :: SimpleDataCon -> Q Exp
1025+ [A] mkRecSerializeExpr :: Name -> SimpleDataCon -> Q Exp
1026+ [A] mkRecDeserializeExpr :: Name -> Name -> Name -> SimpleDataCon -> Q Exp
1027+ [A] mkFieldName :: Int -> Name
1028+ [A] mkDeserializeKeysDec :: Name -> Name -> SimpleDataCon -> Q [Dec]
1029+ [A] mkDeserializeExprOne :: Name -> SimpleDataCon -> Q Exp
1030+ [A] matchConstructor :: Name -> Int -> Q Exp -> Q Match
1031+ [A] makeN :: Int -> Name
1032+ [A] makeI :: Int -> Name
1033+ [A] makeA :: Int -> Name
1034+ [A] litProxy :: Unbox a => Proxy a -> Q Exp
1035+ [A] litIntegral :: Integral a => a -> Q Exp
1036+ [A] isUnitType :: [DataCon] -> Bool
1037+ [A] isRecordSyntax :: SimpleDataCon -> Bool
1038+ [A] isPinned :: MutByteArray -> Bool
1039+ [A] int_w8 :: Int -> Word8
1040+ [A] int_w32 :: Int -> Word32
1041+ [A] inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
1042+ [A] inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
1043+ [A] inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
1044+ [A] getMutableByteArray# :: MutByteArray -> MutableByteArray# RealWorld
1045+ [A] genericSizeOf :: forall a. SizeOfRep (Rep a) => Proxy a -> Int
1046+ [A] genericPokeByteIndex :: (Generic a, PokeRep (Rep a)) => MutByteArray -> Int -> a -> IO ()
1047+ [A] genericPeekByteIndex :: (Generic a, PeekRep (Rep a)) => MutByteArray -> Int -> IO a
1048+ [A] errorUnsupported :: String -> a
1049+ [A] errorUnimplemented :: a
1050+ [A] encodeRecordFields :: Bool -> SerializeConfig -> SerializeConfig
1051+ [A] encodeConstrNames :: Bool -> SerializeConfig -> SerializeConfig
1052+ [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
1053+ [A] deriveUnbox :: Q [Dec] -> Q [Dec]
1054+ [A] deriveSerializeWith :: (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
1055+ [A] deriveSerialize :: Q [Dec] -> Q [Dec]
1056+ [A] conUpdateFuncDec :: Name -> [Field] -> Q [Dec]
1057+ [A] cloneSliceUnsafeAs :: MonadIO m => PinnedState -> Int -> Int -> MutByteArray -> m MutByteArray
1058+ [A] cloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
1059+ [A] c2w :: Char -> Word8
1060+ [A] asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
1061+ [A] addSizeTo :: Serialize a => Int -> a -> Int
1062+ [A] _x :: Name
1063+ [A] _val :: Name
1064+ [A] _tag :: Name
1065+ [A] _initialOffset :: Name
1066+ [A] _endOffset :: Name
1067+ [A] _arr :: Name
1068+ [A] _acc :: Name
1069+[A] Streamly.Internal.Data.MutArray.Stream
1070+ [A] SpliceState
1071+ [A] SpliceYielding :: arr -> SpliceState s arr -> SpliceState s arr
1072+ [A] SpliceInitial :: s -> SpliceState s arr
1073+ [A] SpliceFinish :: SpliceState s arr
1074+ [A] SpliceBuffering :: s -> arr -> SpliceState s arr
1075+ [A] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
1076+ [A] splitOn :: (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> Stream m (MutArray a)
1077+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1078+ [A] packArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1079+ [A] lpackArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
1080+ [A] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
1081+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1082+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1083+ [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
1084+ [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1085+ [A] compactEQ :: Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1086+ [A] compact :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1087+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1088+[A] Streamly.Internal.Data.MutArray.Generic
1089+ [A] MutArray
1090+ [A] [arrTrueLen] :: MutArray a -> {-# UNPACK #-} !Int
1091+ [A] [arrStart] :: MutArray a -> {-# UNPACK #-} !Int
1092+ [A] [arrLen] :: MutArray a -> {-# UNPACK #-} !Int
1093+ [A] [arrContents#] :: MutArray a -> MutableArray# RealWorld a
1094+ [A] MutArray :: MutableArray# RealWorld a -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> MutArray a
1095+ [A] writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
1096+ [A] writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
1097+ [A] writeN :: MonadIO m => Int -> Fold m a (MutArray a)
1098+ [A] write :: MonadIO m => Fold m a (MutArray a)
1099+ [A] uninit :: MonadIO m => MutArray a -> Int -> m (MutArray a)
1100+ [A] toStreamK :: MonadIO m => MutArray a -> StreamK m a
1101+ [A] toList :: MonadIO m => MutArray a -> m [a]
1102+ [A] strip :: MonadIO m => (a -> Bool) -> MutArray a -> m (MutArray a)
1103+ [A] snocWith :: MonadIO m => (Int -> Int) -> MutArray a -> a -> m (MutArray a)
1104+ [A] snocUnsafe :: MonadIO m => MutArray a -> a -> m (MutArray a)
1105+ [A] snoc :: MonadIO m => MutArray a -> a -> m (MutArray a)
1106+ [A] realloc :: MonadIO m => Int -> MutArray a -> m (MutArray a)
1107+ [A] reader :: MonadIO m => Unfold m (MutArray a) a
1108+ [A] readRev :: MonadIO m => MutArray a -> Stream m a
1109+ [A] read :: MonadIO m => MutArray a -> Stream m a
1110+ [A] putSliceUnsafe :: MonadIO m => MutArray a -> Int -> MutArray a -> Int -> Int -> m ()
1111+ [A] putIndices :: MonadIO m => MutArray a -> Fold m (Int, a) ()
1112+ [A] putIndexUnsafe :: forall m a. MonadIO m => Int -> MutArray a -> a -> m ()
1113+ [A] putIndex :: MonadIO m => Int -> MutArray a -> a -> m ()
1114+ [A] producerWith :: Monad m => (forall b. IO b -> m b) -> Producer m (MutArray a) a
1115+ [A] producer :: MonadIO m => Producer m (MutArray a) a
1116+ [A] nil :: MonadIO m => m (MutArray a)
1117+ [A] new :: MonadIO m => Int -> m (MutArray a)
1118+ [A] modifyIndexUnsafe :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
1119+ [A] modifyIndex :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
1120+ [A] length :: MutArray a -> Int
1121+ [A] getSliceUnsafe :: Int -> Int -> MutArray a -> MutArray a
1122+ [A] getSlice :: Int -> Int -> MutArray a -> MutArray a
1123+ [A] getIndexUnsafeWith :: MonadIO m => MutableArray# RealWorld a -> Int -> m a
1124+ [A] getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
1125+ [A] getIndex :: MonadIO m => Int -> MutArray a -> m (Maybe a)
1126+ [A] fromStreamN :: MonadIO m => Int -> Stream m a -> m (MutArray a)
1127+ [A] fromStream :: MonadIO m => Stream m a -> m (MutArray a)
1128+ [A] fromPureStream :: MonadIO m => Stream Identity a -> m (MutArray a)
1129+ [A] fromListN :: MonadIO m => Int -> [a] -> m (MutArray a)
1130+ [A] fromList :: MonadIO m => [a] -> m (MutArray a)
1131+ [A] eq :: (MonadIO m, Eq a) => MutArray a -> MutArray a -> m Bool
1132+ [A] cmp :: (MonadIO m, Ord a) => MutArray a -> MutArray a -> m Ordering
1133+ [A] clone :: MonadIO m => MutArray a -> m (MutArray a)
1134+ [A] chunksOf :: forall m a. MonadIO m => Int -> Stream m a -> Stream m (MutArray a)
1135+[A] Streamly.Internal.Data.MutArray
1136+ [A] MutByteArray
1137+ [A] MutArray
1138+ [A] [arrStart] :: MutArray a -> {-# UNPACK #-} !Int
1139+ [A] [arrEnd] :: MutArray a -> {-# UNPACK #-} !Int
1140+ [A] [arrContents] :: MutArray a -> {-# UNPACK #-} !MutByteArray
1141+ [A] [arrBound] :: MutArray a -> {-# UNPACK #-} !Int
1142+ [A] MutArray :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> MutArray a
1143+ [A] IORef
1144+ [A] ArrayUnsafe
1145+ [A] ArrayUnsafe :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> ArrayUnsafe a
1146+ [A] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1147+ [A] writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1148+ [A] writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1149+ [A] writeNWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1150+ [A] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1151+ [A] writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1152+ [A] writeIORef :: Unbox a => IORef a -> a -> IO ()
1153+ [A] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
1154+ [A] writeAppendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
1155+ [A] writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
1156+ [A] writeAppendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
1157+ [A] writeAppend :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
1158+ [A] write :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1159+ [A] unsafeSwapIndices :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> MutArray a -> m ()
1160+ [A] unpin :: MutArray a -> IO (MutArray a)
1161+ [A] toStreamKWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> StreamK m a
1162+ [A] toStreamKRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> StreamK m a
1163+ [A] toStreamKRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
1164+ [A] toStreamK :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
1165+ [A] toStreamDWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
1166+ [A] toStreamDRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
1167+ [A] toList :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m [a]
1168+ [A] swapIndices :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> MutArray a -> m ()
1169+ [A] strip :: forall a m. (Unbox a, MonadIO m) => (a -> Bool) -> MutArray a -> m (MutArray a)
1170+ [A] splitOn :: (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> Stream m (MutArray a)
1171+ [A] splitAt :: forall a. Unbox a => Int -> MutArray a -> (MutArray a, MutArray a)
1172+ [A] spliceWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int -> Int) -> MutArray a -> MutArray a -> m (MutArray a)
1173+ [A] spliceUnsafe :: MonadIO m => MutArray a -> MutArray a -> m (MutArray a)
1174+ [A] spliceExp :: (MonadIO m, Unbox a) => MutArray a -> MutArray a -> m (MutArray a)
1175+ [A] spliceCopy :: forall m a. MonadIO m => MutArray a -> MutArray a -> m (MutArray a)
1176+ [A] splice :: (MonadIO m, Unbox a) => MutArray a -> MutArray a -> m (MutArray a)
1177+ [A] snocWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> MutArray a -> a -> m (MutArray a)
1178+ [A] snocUnsafe :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1179+ [A] snocMay :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (Maybe (MutArray a))
1180+ [A] snocLinear :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1181+ [A] snoc :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1182+ [A] shuffleBy :: (a -> a -> m Bool) -> MutArray a -> MutArray a -> m ()
1183+ [A] roundUpToPower2 :: Int -> Int
1184+ [A] rightSize :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m (MutArray a)
1185+ [A] reverse :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m ()
1186+ [A] resizeExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1187+ [A] resize :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1188+ [A] realloc :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1189+ [A] readerRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Unfold m (MutArray a) a
1190+ [A] readerRev :: forall m a. (MonadIO m, Unbox a) => Unfold m (MutArray a) a
1191+ [A] reader :: forall m a. (MonadIO m, Unbox a) => Unfold m (MutArray a) a
1192+ [A] readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
1193+ [A] readIORef :: Unbox a => IORef a -> IO a
1194+ [A] read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
1195+ [A] putIndices :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Fold m (Int, a) ()
1196+ [A] putIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
1197+ [A] putIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
1198+ [A] producerWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Producer m (MutArray a) a
1199+ [A] producer :: forall m a. (MonadIO m, Unbox a) => Producer m (MutArray a) a
1200+ [A] pollIntIORef :: (MonadIO m, Unbox a) => IORef a -> Stream m a
1201+ [A] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1202+ [A] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (MutArray a)
1203+ [A] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1204+ [A] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1205+ [A] pinnedNewBytes :: MonadIO m => Int -> m (MutArray a)
1206+ [A] pinnedNewAligned :: (MonadIO m, Unbox a) => Int -> Int -> m (MutArray a)
1207+ [A] pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
1208+ [A] pinnedFromListN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1209+ [A] pinnedFromList :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1210+ [A] pinnedClone :: MonadIO m => MutArray a -> m (MutArray a)
1211+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1212+ [A] pin :: MutArray a -> IO (MutArray a)
1213+ [A] permute :: MutArray a -> m Bool
1214+ [A] partitionBy :: forall m a. (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> m (MutArray a, MutArray a)
1215+ [A] nil :: MutArray a
1216+ [A] newIORef :: forall a. Unbox a => a -> IO (IORef a)
1217+ [A] newArrayWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int -> m MutByteArray) -> Int -> Int -> m (MutArray a)
1218+ [A] new :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
1219+ [A] modifyIndices :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (Int -> a -> a) -> Fold m Int ()
1220+ [A] modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
1221+ [A] modifyIndex :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
1222+ [A] modifyIORef' :: Unbox a => IORef a -> (a -> a) -> IO ()
1223+ [A] modify :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (a -> a) -> m ()
1224+ [A] mergeBy :: Int -> (MutArray a -> MutArray a -> m ()) -> MutArray a -> m ()
1225+ [A] memcpy :: Ptr Word8 -> Ptr Word8 -> Int -> IO ()
1226+ [A] memcmp :: Ptr Word8 -> Ptr Word8 -> Int -> IO Bool
1227+ [A] length :: forall a. Unbox a => MutArray a -> Int
1228+ [A] isPinned :: MutArray a -> Bool
1229+ [A] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
1230+ [A] getSliceUnsafe :: forall a. Unbox a => Int -> Int -> MutArray a -> MutArray a
1231+ [A] getSlice :: forall a. Unbox a => Int -> Int -> MutArray a -> MutArray a
1232+ [A] getIndicesD :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
1233+ [A] getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1234+ [A] getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
1235+ [A] getIndexRev :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
1236+ [A] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
1237+ [A] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
1238+ [A] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
1239+ [A] fromStreamD :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
1240+ [A] fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
1241+ [A] fromPureStream :: (MonadIO m, Unbox a) => Stream Identity a -> m (MutArray a)
1242+ [A] fromListRevN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1243+ [A] fromListRev :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1244+ [A] fromListN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1245+ [A] fromList :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1246+ [A] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
1247+ [A] foldr :: (MonadIO m, Unbox a) => (a -> b -> b) -> b -> MutArray a -> m b
1248+ [A] foldl' :: (MonadIO m, Unbox a) => (b -> a -> b) -> b -> MutArray a -> m b
1249+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1250+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1251+ [A] divideBy :: Int -> (MutArray a -> m (MutArray a, MutArray a)) -> MutArray a -> m ()
1252+ [A] cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
1253+ [A] clone :: MonadIO m => MutArray a -> m (MutArray a)
1254+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1255+ [A] castUnsafe :: MutArray a -> MutArray b
1256+ [A] cast :: forall a b. Unbox b => MutArray a -> Maybe (MutArray b)
1257+ [A] c_memchr :: Ptr Word8 -> Word8 -> CSize -> IO (Ptr Word8)
1258+ [A] bytesFree :: MutArray a -> Int
1259+ [A] byteLength :: MutArray a -> Int
1260+ [A] byteCapacity :: MutArray a -> Int
1261+ [A] bubble :: (MonadIO m, Unbox a) => (a -> a -> Ordering) -> MutArray a -> m ()
1262+ [A] breakOn :: MonadIO m => Word8 -> MutArray Word8 -> m (MutArray Word8, Maybe (MutArray Word8))
1263+ [A] blockSize :: Int
1264+ [A] asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
1265+ [A] asBytes :: MutArray a -> MutArray Word8
1266+ [A] arrayChunkBytes :: Int
1267+ [A] allocBytesToElemCount :: Unbox a => a -> Int -> Int
1268+[C] Streamly.Internal.Data.IsMap
1269+ [A] mapTraverseWithKey :: (IsMap f, Applicative t) => (Key f -> a -> t b) -> f a -> t (f b)
1270+[R] Streamly.Internal.Data.IORef.Unboxed
1271+[R] Streamly.Internal.Data.Fold.Window
1272+[R] Streamly.Internal.Data.Fold.Type
1273+[R] Streamly.Internal.Data.Fold.Tee
1274+[R] Streamly.Internal.Data.Fold.Step
1275+[R] Streamly.Internal.Data.Fold.Container
1276+[D] Streamly.Internal.Data.Fold.Chunked
1277+[C] Streamly.Internal.Data.Fold
1278+ [A] ManyState
1279+ [C] Fold
1280+ [C] Fold
1281+ [O] Fold :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> Fold m a b
1282+ [N] Fold :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> (s -> m b) -> Fold m a b
1283+ [A] windowSumInt :: forall m a. (Monad m, Integral a) => Fold m (a, Maybe a) a
1284+ [A] windowSum :: forall m a. (Monad m, Num a) => Fold m (a, Maybe a) a
1285+ [A] windowRollingMapM :: Monad m => (Maybe a -> a -> m (Maybe b)) -> Fold m (a, Maybe a) (Maybe b)
1286+ [A] windowRollingMap :: Monad m => (Maybe a -> a -> Maybe b) -> Fold m (a, Maybe a) (Maybe b)
1287+ [A] windowRange :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe (a, a))
1288+ [A] windowPowerSumFrac :: (Monad m, Floating a) => a -> Fold m (a, Maybe a) a
1289+ [A] windowPowerSum :: (Monad m, Num a) => Int -> Fold m (a, Maybe a) a
1290+ [A] windowMinimum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
1291+ [A] windowMean :: forall m a. (Monad m, Fractional a) => Fold m (a, Maybe a) a
1292+ [A] windowMaximum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
1293+ [A] windowLmap :: (c -> a) -> Fold m (a, Maybe a) b -> Fold m (c, Maybe c) b
1294+ [A] windowLength :: (Monad m, Num b) => Fold m (a, Maybe a) b
1295+ [A] toSet :: (Monad m, Ord a) => Fold m a (Set a)
1296+ [A] toMapIO :: (MonadIO m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (Map k b)
1297+ [A] toMap :: (Monad m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (Map k b)
1298+ [A] toIntSet :: Monad m => Fold m Int IntSet
1299+ [A] toContainerIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (f b)
1300+ [A] toContainer :: (Monad m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (f b)
1301+ [A] nubInt :: Monad m => Fold m Int (Maybe Int)
1302+ [A] nub :: (Monad m, Ord a) => Fold m a (Maybe a)
1303+ [A] mapMStep :: Applicative m => (a -> m b) -> Step s a -> m (Step s b)
1304+ [A] kvToMap :: (Monad m, Ord k) => Fold m a b -> Fold m (k, a) (Map k b)
1305+ [A] frequency :: (Monad m, Ord a) => Fold m a (Map a Int)
1306+ [A] demuxToMapIO :: (MonadIO m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (Map k b)
1307+ [A] demuxToMap :: (Monad m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (Map k b)
1308+ [A] demuxToContainerIO :: (MonadIO m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
1309+ [A] demuxToContainer :: (Monad m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
1310+ [A] demuxKvToMap :: (Monad m, Ord k) => (k -> m (Fold m a b)) -> Fold m (k, a) (Map k b)
1311+ [A] demuxKvToContainer :: (Monad m, IsMap f, Traversable f) => (Key f -> m (Fold m a b)) -> Fold m (Key f, a) (f b)
1312+ [A] demuxIO :: (MonadIO m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (m (Map k b), Maybe (k, b))
1313+ [A] demuxGenericIO :: (MonadIO m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (m (f b), Maybe (Key f, b))
1314+ [A] demuxGeneric :: (Monad m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (m (f b), Maybe (Key f, b))
1315+ [A] demux :: (Monad m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (m (Map k b), Maybe (k, b))
1316+ [A] cumulative :: Fold m (a, Maybe a) b -> Fold m a b
1317+ [A] countDistinctInt :: Monad m => Fold m Int Int
1318+ [A] countDistinct :: (Monad m, Ord a) => Fold m a Int
1319+ [A] classifyIO :: (MonadIO m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
1320+ [A] classifyGenericIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
1321+ [A] classifyGeneric :: (Monad m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
1322+ [A] classify :: (Monad m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
1323+ [A] chainStepM :: Applicative m => (s1 -> m s2) -> (a -> m (Step s2 b)) -> Step s1 a -> m (Step s2 b)
1324+[C] Streamly.Internal.Data.Builder
1325+ [C] Builder
1326+ [C] Builder
1327+ [O] Builder :: (s -> m (s, a)) -> Builder s m a
1328+ [N] Builder :: (s -> m (a, s)) -> Builder s m a
1329+[A] Streamly.Internal.Data.Binary.Stream
1330+ [A] class ToBytes a
1331+ [A] word8 :: Applicative m => Word8 -> Stream m Word8
1332+ [A] word64le :: Monad m => Word64 -> Stream m Word8
1333+ [A] word64host :: Monad m => Word64 -> Stream m Word8
1334+ [A] word64be :: Monad m => Word64 -> Stream m Word8
1335+ [A] word32le :: Monad m => Word32 -> Stream m Word8
1336+ [A] word32be :: Monad m => Word32 -> Stream m Word8
1337+ [A] word16le :: Monad m => Word16 -> Stream m Word8
1338+ [A] word16be :: Monad m => Word16 -> Stream m Word8
1339+ [A] unit :: Applicative m => Stream m Word8
1340+ [A] toBytes :: ToBytes a => a -> Stream m Word8
1341+ [A] ordering :: Applicative m => Ordering -> Stream m Word8
1342+ [A] int8 :: Applicative m => Int8 -> Stream m Word8
1343+ [A] int64le :: Monad m => Int64 -> Stream m Word8
1344+ [A] int64be :: Monad m => Int64 -> Stream m Word8
1345+ [A] int32le :: Monad m => Int32 -> Stream m Word8
1346+ [A] int32be :: Monad m => Int32 -> Stream m Word8
1347+ [A] int16le :: Monad m => Int16 -> Stream m Word8
1348+ [A] int16be :: Monad m => Int16 -> Stream m Word8
1349+ [A] float32le :: Monad m => Float -> Stream m Word8
1350+ [A] float32be :: Monad m => Float -> Stream m Word8
1351+ [A] double64le :: Monad m => Double -> Stream m Word8
1352+ [A] double64be :: Monad m => Double -> Stream m Word8
1353+ [A] charUtf8 :: Monad m => Char -> Stream m Word8
1354+ [A] charLatin1 :: Applicative m => Char -> Stream m Word8
1355+ [A] bool :: Applicative m => Bool -> Stream m Word8
1356+[A] Streamly.Internal.Data.Binary.Parser
1357+ [A] class FromBytes a
1358+ [A] word8 :: Monad m => Parser Word8 m Word8
1359+ [A] word64le :: Monad m => Parser Word8 m Word64
1360+ [A] word64host :: MonadIO m => Parser Word8 m Word64
1361+ [A] word64be :: Monad m => Parser Word8 m Word64
1362+ [A] word32le :: Monad m => Parser Word8 m Word32
1363+ [A] word32be :: Monad m => Parser Word8 m Word32
1364+ [A] word16le :: Monad m => Parser Word8 m Word16
1365+ [A] word16be :: Monad m => Parser Word8 m Word16
1366+ [A] unit :: Monad m => Parser Word8 m ()
1367+ [A] ordering :: Monad m => Parser Word8 m Ordering
1368+ [A] int8 :: Monad m => Parser Word8 m Int8
1369+ [A] int64le :: Monad m => Parser Word8 m Int64
1370+ [A] int64be :: Monad m => Parser Word8 m Int64
1371+ [A] int32le :: Monad m => Parser Word8 m Int32
1372+ [A] int32be :: Monad m => Parser Word8 m Int32
1373+ [A] int16le :: Monad m => Parser Word8 m Int16
1374+ [A] int16be :: Monad m => Parser Word8 m Int16
1375+ [A] fromBytes :: FromBytes a => Parser Word8 m a
1376+ [A] float32le :: MonadIO m => Parser Word8 m Float
1377+ [A] float32be :: MonadIO m => Parser Word8 m Float
1378+ [A] eqWord8 :: Monad m => Word8 -> Parser Word8 m Word8
1379+ [A] double64le :: MonadIO m => Parser Word8 m Double
1380+ [A] double64be :: MonadIO m => Parser Word8 m Double
1381+ [A] charLatin1 :: Monad m => Parser Word8 m Char
1382+ [A] bool :: Monad m => Parser Word8 m Bool
1383+[R] Streamly.Internal.Data.Array.Type
1384+[A] Streamly.Internal.Data.Array.Stream
1385+ [A] unlines :: forall m a. (MonadIO m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1386+ [A] toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
1387+ [A] splitOnSuffix :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
1388+ [A] splitOn :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
1389+ [A] runArrayParserDBreak :: forall m a b. (MonadIO m, Unbox a) => Parser (Array a) m b -> Stream m (Array a) -> m (Either ParseError b, Stream m (Array a))
1390+ [A] runArrayFoldMany :: (Monad m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> StreamK m (Either ParseError b)
1391+ [A] runArrayFoldBreak :: (MonadIO m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1392+ [A] runArrayFold :: (MonadIO m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> m (Either ParseError b)
1393+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1394+ [A] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
1395+ [A] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1396+ [A] parseBreak :: (MonadIO m, Unbox a) => Parser a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1397+ [A] lpackArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
1398+ [A] interposeSuffix :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1399+ [A] interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1400+ [A] intercalateSuffix :: (Monad m, Unbox a) => Array a -> Stream m (Array a) -> Stream m a
1401+ [A] foldBreakD :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
1402+ [A] foldBreak :: (MonadIO m, Unbox a) => Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
1403+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1404+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1405+ [A] concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
1406+ [A] concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
1407+ [A] compact :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
1408+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1409+ [A] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
1410+[R] Streamly.Internal.Data.Array.Mut.Type
1411+[R] Streamly.Internal.Data.Array.Mut.Stream
1412+[R] Streamly.Internal.Data.Array.Mut
1413+[R] Streamly.Internal.Data.Array.Generic.Mut.Type
1414+[C] Streamly.Internal.Data.Array.Generic
1415+ [A] getIndex :: Int -> Array a -> Maybe a
1416+ [A] fromPureStream :: Stream Identity a -> Array a
1417+ [A] fromByteStr# :: Addr# -> Array Word8
1418+ [A] chunksOf :: forall m a. MonadIO m => Int -> Stream m a -> Stream m (Array a)
1419+[C] Streamly.Internal.Data.Array
1420+ [A] ArrayUnsafe
1421+ [A] ArrayUnsafe :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> ArrayUnsafe a
1422+ [C] Array
1423+ [A] [arrStart] :: Array a -> {-# UNPACK #-} !Int
1424+ [A] [arrEnd] :: Array a -> {-# UNPACK #-} !Int
1425+ [A] [arrContents] :: Array a -> {-# UNPACK #-} !MutByteArray
1426+ [A] Array :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> Array a
1427+ [A] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1428+ [A] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1429+ [R] writeNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
1430+ [A] unsafeMakePure :: Monad m => Fold IO a b -> Fold m a b
1431+ [A] unsafeIndexIO :: forall a. Unbox a => Int -> Array a -> IO a
1432+ [D] unsafeIndex :: forall a. Unbox a => Int -> Array a -> a
1433+ [A] unsafeFreezeWithShrink :: Unbox a => MutArray a -> Array a
1434+ [A] unpin :: Array a -> IO (Array a)
1435+ [A] toStreamKRev :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a
1436+ [A] toStreamK :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a
1437+ [A] toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
1438+ [A] toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
1439+ [A] splitAt :: Unbox a => Int -> Array a -> (Array a, Array a)
1440+ [A] splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
1441+ [A] serialize :: Serialize a => a -> Array Word8
1442+ [A] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1443+ [A] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
1444+ [A] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1445+ [A] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
1446+ [A] pinnedSerialize :: Serialize a => a -> Array Word8
1447+ [A] pinnedFromListN :: Unbox a => Int -> [a] -> Array a
1448+ [A] pinnedFromList :: Unbox a => [a] -> Array a
1449+ [A] pinnedClone :: MonadIO m => Array a -> m (Array a)
1450+ [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1451+ [A] pin :: Array a -> IO (Array a)
1452+ [A] nil :: Array a
1453+ [A] isPinned :: Array a -> Bool
1454+ [A] getIndexUnsafe :: forall a. Unbox a => Int -> Array a -> a
1455+ [A] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
1456+ [A] fromStreamD :: forall m a. (MonadIO m, Unbox a) => Stream m a -> m (Array a)
1457+ [A] fromPureStream :: Unbox a => Stream Identity a -> Array a
1458+ [A] fromListRevN :: Unbox a => Int -> [a] -> Array a
1459+ [A] fromListRev :: Unbox a => [a] -> Array a
1460+ [A] fromByteStr# :: Addr# -> Array Word8
1461+ [A] foldr :: Unbox a => (a -> b -> b) -> b -> Array a -> b
1462+ [A] foldl' :: forall a b. Unbox a => (b -> a -> b) -> b -> Array a -> b
1463+ [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1464+ [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1465+ [A] encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
1466+ [A] deserialize :: Serialize a => Array Word8 -> a
1467+ [A] clone :: MonadIO m => Array a -> m (Array a)
1468+ [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1469+ [A] byteLength :: Array a -> Int
1470+ [A] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
1471+ [A] breakOn :: MonadIO m => Word8 -> Array Word8 -> m (Array Word8, Maybe (Array Word8))
core/docs/ApiChangelogs/0.2.0-0.2.2.txtnew file+306
@@ -0,0 +1,306 @@
1+---------------------------------
2+API Annotations
3+---------------------------------
4+
5+[A] : Added
6+[R] : Removed
7+[C] : Changed
8+[O] : Old definition
9+[N] : New definition
10+[D] : Deprecated
11+
12+---------------------------------
13+API diff
14+---------------------------------
15+
16+[C] Streamly.Data.Stream
17+ [A] (FixityR,5)
18+ [A] (FixityR,5)
19+[C] Streamly.Data.MutArray.Generic
20+ [A] emptyOf :: MonadIO m => Int -> m (MutArray a)
21+ [A] createOf :: MonadIO m => Int -> Fold m a (MutArray a)
22+ [A] create :: MonadIO m => Fold m a (MutArray a)
23+[C] Streamly.Data.MutArray
24+ [A] pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
25+ [A] emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
26+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
27+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
28+ [A] appendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
29+ [A] append :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
30+[C] Streamly.Data.Array.Generic
31+ [A] createOf :: MonadIO m => Int -> Fold m a (Array a)
32+ [A] create :: MonadIO m => Fold m a (Array a)
33+[C] Streamly.Data.Array
34+ [A] class Serialize a
35+ [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
36+ [A] pinnedSerialize :: Serialize a => a -> Array Word8
37+ [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
38+ [A] deserialize :: Serialize a => Array Word8 -> a
39+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
40+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
41+ [A] addSizeTo :: Serialize a => Int -> a -> Int
42+
43+---------------------------------
44+Internal API diff
45+---------------------------------
46+
47+[C] Streamly.Internal.Unicode.Stream
48+ [A] encodeUtf16le' :: Stream m Char -> Stream m Word16
49+ [A] decodeUtf16le' :: Stream m Word16 -> Stream m Char
50+[A] Streamly.Internal.FileSystem.Path
51+ [A] class IsPath a
52+ [A] Rel
53+ [A] File
54+ [A] Dir
55+ [A] Abs
56+ [A] Streamly.Internal.FileSystem.Path.IsPath
57+ [A] instance Streamly.Internal.FileSystem.Path.IsPath Streamly.Internal.FileSystem.Path.Path
58+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel Streamly.Internal.FileSystem.Path.Path)
59+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path))
60+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path))
61+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path)
62+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path)
63+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs Streamly.Internal.FileSystem.Path.Path)
64+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path))
65+ [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path))
66+ [A] GHC.Show.Show
67+ [A] instance GHC.Show.Show Streamly.Internal.FileSystem.Path.PathException
68+ [A] GHC.Exception.Type.Exception
69+ [A] instance GHC.Exception.Type.Exception Streamly.Internal.FileSystem.Path.PathException
70+ [A] GHC.Classes.Eq
71+ [A] instance GHC.Classes.Eq Streamly.Internal.FileSystem.Path.PathException
72+ [A] Path
73+ [A] Path :: Array Word8 -> Path
74+ [A] toString :: Path -> [Char]
75+ [A] toPath :: IsPath a => a -> Path
76+ [A] toChunk :: Path -> Array Word8
77+ [A] toChars :: Monad m => Path -> Stream m Char
78+ [A] relfile :: QuasiQuoter
79+ [A] reldir :: QuasiQuoter
80+ [A] rel :: QuasiQuoter
81+ [A] primarySeparator :: Char
82+ [A] path :: QuasiQuoter
83+ [A] mkRelFile :: String -> Q Exp
84+ [A] mkRelDir :: String -> Q Exp
85+ [A] mkRel :: String -> Q Exp
86+ [A] mkPath :: String -> Q Exp
87+ [A] mkFile :: String -> Q Exp
88+ [A] mkDir :: String -> Q Exp
89+ [A] mkAbsFile :: String -> Q Exp
90+ [A] mkAbsDir :: String -> Q Exp
91+ [A] mkAbs :: String -> Q Exp
92+ [A] isSeparator :: Char -> Bool
93+ [A] fromString :: MonadThrow m => [Char] -> m Path
94+ [A] fromPathUnsafe :: IsPath a => Path -> a
95+ [A] fromPath :: (IsPath a, MonadThrow m) => Path -> m a
96+ [A] fromChunkUnsafe :: Array Word8 -> Path
97+ [A] fromChunk :: MonadThrow m => Array Word8 -> m Path
98+ [A] fromChars :: MonadThrow m => Stream Identity Char -> m Path
99+ [A] file :: QuasiQuoter
100+ [A] extendPath :: Path -> Path -> Path
101+ [A] extendDir :: (IsPath (a (Dir Path)), IsPath b, IsPath (a b)) => a (Dir Path) -> Rel b -> a b
102+ [A] dir :: QuasiQuoter
103+ [A] adaptPath :: (MonadThrow m, IsPath a, IsPath b) => a -> m b
104+ [A] absfile :: QuasiQuoter
105+ [A] absdir :: QuasiQuoter
106+ [A] abs :: QuasiQuoter
107+[C] Streamly.Internal.Data.Stream
108+ [A] (FixityR,5)
109+ [A] (FixityR,5)
110+ [C] splitInnerBySuffix
111+ [O] splitInnerBySuffix :: (Monad m, Eq (f a), Monoid (f a)) => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
112+ [N] splitInnerBySuffix :: Monad m => (f a -> Bool) -> (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
113+ [D] sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
114+ [A] indexOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
115+[C] Streamly.Internal.Data.MutByteArray
116+ [A] unsafePinnedAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
117+ [A] unsafeAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
118+ [D] nil :: MutByteArray
119+ [A] empty :: MutByteArray
120+ [D] asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
121+[D] Streamly.Internal.Data.MutArray.Stream
122+ [D] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
123+ [D] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
124+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
125+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
126+[C] Streamly.Internal.Data.MutArray.Generic
127+ [D] writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
128+ [D] writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
129+ [A] unsafeCreateOf :: MonadIO m => Int -> Fold m a (MutArray a)
130+ [A] emptyOf :: MonadIO m => Int -> m (MutArray a)
131+ [A] createWith :: MonadIO m => Int -> Fold m a (MutArray a)
132+ [A] createOf :: MonadIO m => Int -> Fold m a (MutArray a)
133+ [A] create :: MonadIO m => Fold m a (MutArray a)
134+[C] Streamly.Internal.Data.MutArray
135+ [A] SpliceState
136+ [A] SpliceYielding :: arr -> SpliceState s arr -> SpliceState s arr
137+ [A] SpliceInitial :: s -> SpliceState s arr
138+ [A] SpliceFinish :: SpliceState s arr
139+ [A] SpliceBuffering :: s -> arr -> SpliceState s arr
140+ [R] MutByteArray
141+ [D] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
142+ [D] writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
143+ [D] writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
144+ [D] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
145+ [D] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
146+ [D] writeAppendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
147+ [D] writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
148+ [A] unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
149+ [A] unsafePinnedAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
150+ [A] unsafeCreateOfWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
151+ [A] unsafeCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
152+ [A] unsafeAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
153+ [A] unsafeAppendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
154+ [A] toStreamWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
155+ [A] toStreamRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
156+ [R] toStreamDWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
157+ [R] toStreamDRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
158+ [A] slicerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
159+ [A] sliceIndexerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
160+ [A] revCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
161+ [D] resizeExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
162+ [D] resize :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
163+ [A] pokeSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
164+ [A] pokeAppendMay :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> a -> m (Maybe (MutArray Word8))
165+ [A] pokeAppend :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> a -> m (MutArray Word8)
166+ [D] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
167+ [D] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
168+ [D] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
169+ [D] pinnedNewBytes :: MonadIO m => Int -> m (MutArray a)
170+ [A] pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
171+ [A] pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
172+ [A] pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
173+ [A] pinnedCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
174+ [A] peekUnconsUnsafe :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> m (a, MutArray Word8)
175+ [A] peekUncons :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> m (Maybe a, MutArray Word8)
176+ [A] peekSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
177+ [A] pPinnedCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Parser (MutArray a) m (MutArray a)
178+ [A] pCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Parser (MutArray a) m (MutArray a)
179+ [D] nil :: MutArray a
180+ [A] lPinnedCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
181+ [A] lCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
182+ [A] indexReaderWith :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
183+ [A] indexReader :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
184+ [A] growExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
185+ [A] grow :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
186+ [D] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
187+ [R] getIndicesD :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
188+ [D] getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
189+ [D] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
190+ [A] fromStreamN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
191+ [D] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
192+ [D] fromStreamD :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
193+ [A] fromPureStreamN :: (MonadIO m, Unbox a) => Int -> Stream Identity a -> m (MutArray a)
194+ [A] fromPtrN :: MonadIO m => Int -> Ptr Word8 -> m (MutArray Word8)
195+ [A] fromChunksRealloced :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> m (MutArray a)
196+ [A] fromChunksK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
197+ [A] fromByteStr# :: MonadIO m => Addr# -> m (MutArray Word8)
198+ [D] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
199+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
200+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
201+ [A] fPinnedCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) (MutArray a)
202+ [A] fCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) (MutArray a)
203+ [A] emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
204+ [A] empty :: MutArray a
205+ [A] createWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
206+ [A] createOfWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
207+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
208+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
209+ [A] concatWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m (MutArray a) -> Stream m a
210+ [A] concatRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m (MutArray a) -> Stream m a
211+ [A] concatRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
212+ [A] concat :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
213+ [A] compactOnByteSuffix :: MonadIO m => Word8 -> Stream m (MutArray Word8) -> Stream m (MutArray Word8)
214+ [A] compactOnByte :: MonadIO m => Word8 -> Stream m (MutArray Word8) -> Stream m (MutArray Word8)
215+ [A] compactLeAs :: forall m a. (MonadIO m, Unbox a) => PinnedState -> Int -> Stream m (MutArray a) -> Stream m (MutArray a)
216+ [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
217+ [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
218+ [A] compactEQ :: Int -> Stream m (MutArray a) -> Stream m (MutArray a)
219+ [D] cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
220+ [A] byteEq :: MonadIO m => MutArray a -> MutArray a -> m Bool
221+ [A] byteCmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
222+ [A] buildChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
223+ [D] asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
224+ [A] appendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
225+ [A] appendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
226+ [A] append :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
227+[D] Streamly.Internal.Data.Array.Stream
228+ [C] interposeSuffix
229+ [O] interposeSuffix :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
230+ [N] interposeSuffix :: forall m a. (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
231+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
232+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
233+ [C] concatRev
234+ [O] concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
235+ [N] concatRev :: forall m a. (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
236+ [D] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
237+[C] Streamly.Internal.Data.Array.Generic
238+ [A] createOf :: MonadIO m => Int -> Fold m a (Array a)
239+ [A] create :: MonadIO m => Fold m a (Array a)
240+[C] Streamly.Internal.Data.Array
241+ [R] ArrayUnsafe
242+ [D] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
243+ [D] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
244+ [A] unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
245+ [A] unsafePinnedAsPtr :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
246+ [A] unsafeCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
247+ [D] toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
248+ [D] toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
249+ [C] splice
250+ [O] splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
251+ [N] splice :: MonadIO m => Array a -> Array a -> m (Array a)
252+ [A] slicerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Array a)
253+ [A] sliceIndexerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Int, Int)
254+ [D] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
255+ [D] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
256+ [D] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
257+ [D] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
258+ [A] pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
259+ [A] pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
260+ [A] pinnedCompactLE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
261+ [A] parseBreakChunksK :: forall m a b. (MonadIO m, Unbox a) => Parser a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
262+ [D] nil :: Array a
263+ [A] lPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
264+ [A] lCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
265+ [A] interposeSuffix :: forall m a. (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
266+ [A] interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
267+ [A] intercalateSuffix :: (Monad m, Unbox a) => Array a -> Stream m (Array a) -> Stream m a
268+ [A] indexReaderFromThenTo :: Unfold m (Int, Int, Int, Array a) a
269+ [A] indexReader :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
270+ [A] indexFinder :: (a -> Bool) -> Unfold Identity (Array a) Int
271+ [D] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Array a)
272+ [R] getIndicesFromThenTo :: Unfold m (Int, Int, Int, Array a) a
273+ [D] getIndices :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
274+ [D] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Int, Int)
275+ [D] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
276+ [D] fromStreamD :: forall m a. (MonadIO m, Unbox a) => Stream m a -> m (Array a)
277+ [A] fromPureStreamN :: Unbox a => Int -> Stream Identity a -> Array a
278+ [A] fromPtrN :: Int -> Ptr Word8 -> Array Word8
279+ [A] fromChunksK :: (MonadIO m, Unbox a) => StreamK m (Array a) -> m (Array a)
280+ [A] fromChunks :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
281+ [A] fromByteStr :: Ptr Word8 -> Array Word8
282+ [A] foldChunks :: (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m b
283+ [A] foldBreakChunksK :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
284+ [A] foldBreakChunks :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
285+ [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
286+ [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
287+ [C] findIndicesOf
288+ [O] findIndicesOf :: (a -> Bool) -> Unfold Identity (Array a) Int
289+ [N] findIndicesOf :: (a -> Bool) -> Array a -> Stream Identity Int
290+ [A] fPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
291+ [A] fCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
292+ [A] empty :: Array a
293+ [A] createWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
294+ [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
295+ [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
296+ [A] concatRev :: forall m a. (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
297+ [A] concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
298+ [A] compactOnByteSuffix :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
299+ [A] compactOnByte :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
300+ [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
301+ [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
302+ [A] byteEq :: Array a -> Array a -> Bool
303+ [A] byteCmp :: Array a -> Array a -> Ordering
304+ [A] buildChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
305+ [D] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
306+ [D] asPtrUnsafe :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
core/docs/Changelog.md+58−16
@@ -1,31 +1,73 @@
11# Changelog
22
3−## Unreleased
3+## 0.2.2 (Jan 2024)
44
5−* Arrays are now created unpinned by default, they were created pinned
6− earlier. During IO operations unpinned arrays are automatically copied
7− to pinned memory. When arrays are directly passed to IO operations
8− programmers can choose to create them pinned to avoid a copy. To
9− create pinned arrays, use the internal APIs with the `pinned*` prefix.
5+* Add fixities `infixr 5` for `cons` and `consM` functions.
6+* Fix a bug in Array `Eq` instance when the type is a sum type with
7+ differently sized constructors.
8+* lpackArraysChunksOf, compact, writeChunksWith, putChunksWith now take the
9+ buffer size in number of array elements instead of bytes.
1010
11−* Changed the signature of 'Streamly.Data.Stream.handle' to make the exception
12− handler monadic.
13−* Rethrow the exception promptly in bracketIO.
11+## 0.2.1 (Dec 2023)
1412
15−### Breaking
13+* Make the serialization of the unit constructor deterministic.
14+* Expose `pinnedSerialize` & `deserialize` via `Streamly.Data.Array`.
1615
17−* `ParserK` in `Streamly.Data.ParserK` is not specialized to arrays anymore. To
18− adapt to the new code, change `ParserK a m b` to `ParserK (Array a) m b` where
19− `Array` comes from `Streamly.Data.Array`.
16+## 0.2.0 (Nov 2023)
17+
18+See [0.1.0-0.2.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0-0.2.0.txt)
19+for a full list of API changes in this release. Only a few significant
20+changes are mentioned here.
21+
22+### Breaking Changes
23+
24+* `ParserK` in `Streamly.Data.ParserK` is not implicitly specialized
25+ to arrays anymore. To adapt to the new code, change `ParserK a m
26+ b` to `ParserK (Array a) m b` where the `Array` type comes from
27+ `Streamly.Data.Array`. This change also affected the signatures of
28+ `parseChunks` and `parseBreakChunks`.
29+* Changed the signature of 'Streamly.Data.Stream.handle' to make the
30+ exception handler monadic.
31+* Behavior change: Exceptions are now rethrown promptly in `bracketIO`.
32+
33+### Enhancements
34+
35+* __Serialization__: Added a `Streamly.Data.MutByteArray` module with a
36+ `Serialize` type class for fast binary serialization. The Data.Array
37+ module supplies the `serialize` and `deserialize` operations for arrays.
38+* __Unpinned Arrays__: Unboxed arrays are now created unpinned by default,
39+ they were created pinned earlier. During IO operations, unpinned arrays
40+ are automatically copied to pinned memory. When arrays are directly
41+ passed to IO operations programmers can choose to create them pinned to
42+ avoid a copy. To create pinned arrays, use the internal APIs with the
43+ `pinned*` prefix.
44+* StreamK now supports native exception handling routines (handle, bracketIO).
45+ Earlier we had to convert it to the `Stream` type for exception handling.
2046
2147### Deprecations
2248
23−* `Streamly.Data.MutArray.newUnpinned` is renamed to
24− `Streamly.Data.MutArray.pinnedNew`
49+See [0.1.0-0.2.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0-0.2.0.txt)
50+for a full list of deprecations.
51+
52+### Internal API Changes
53+
54+* Fold constructor has changed, added a `final` field to support
55+ finalization and cleanup of a chain of folds. The `extract` field is
56+ now used only for mapping the fold internal state to fold result for
57+ scanning purposes. If your fold does not require cleanup you can just use
58+ your existing `extract` function as `final` as well to adapt to this change.
59+* Many low level internal modules have been removed, they are entirely
60+ exported from higher level internal modules. If you were importing any
61+ of the missing low level modules then import the higher level modules instead.
62+* Internal module changes:
63+ * Streamly.Internal.Serialize.FromBytes -> Streamly.Internal.Data.Binary.Parser
64+ * Streamly.Internal.Serialize.ToBytes -> Streamly.Internal.Data.Binary.Stream
65+ * Streamly.Internal.Data.Unbox is now exported via Streamly.Internal.Data.Serialize
66+ * Streamly.Internal.Data.IORef.Unboxed is now exported via Streamly.Internal.Data.Serialize
2567
2668## 0.1.0 (March 2023)
2769
28−Also see [streamly-core-0.1.0 API Changelog](/core/docs/ApiChangelogs/0.1.0.txt) or
70+Also see [streamly-core-0.1.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0.txt) or
2971https://hackage.haskell.org/package/streamly-core-0.1.0/docs/docs/ApiChangelogs/0.1.0.txt
3072
3173`streamly` package is split into two packages, (1) `streamly-core` that
core/src/DocTestDataArray.hs+3
@@ -11,9 +11,12 @@
1111
1212>>> import qualified Streamly.Data.Array as Array
1313>>> import qualified Streamly.Data.Fold as Fold
14+>>> import qualified Streamly.Data.ParserK as ParserK
1415>>> import qualified Streamly.Data.Stream as Stream
16+>>> import qualified Streamly.Data.StreamK as StreamK
1517
1618For APIs that have not been released yet.
1719
1820>>> import qualified Streamly.Internal.Data.Array as Array
21+>>> import qualified Streamly.Internal.Data.Stream as Stream
1922-}
core/src/DocTestDataParserK.hsnew file+15
@@ -0,0 +1,15 @@
1+{- $setup
2+>>> :m
3+>>> import Control.Applicative ((<|>))
4+>>> import Data.Char (isDigit, isAlpha)
5+
6+>>> import Streamly.Data.Parser (Parser)
7+>>> import Streamly.Data.ParserK (ParserK)
8+
9+>>> import qualified Streamly.Data.Parser as Parser
10+>>> import qualified Streamly.Data.ParserK as ParserK
11+
12+For APIs that have not been released yet.
13+
14+>>> import qualified Streamly.Internal.Data.ParserK as ParserK
15+-}
core/src/DocTestDataStream.hs+2
@@ -6,6 +6,7 @@
66>>> import Control.Monad.IO.Class (MonadIO (liftIO))
77>>> import Control.Monad.Trans.Class (lift)
88>>> import Control.Monad.Trans.Identity (runIdentityT)
9+>>> import Data.Char (isSpace)
910>>> import Data.Either (fromLeft, fromRight, isLeft, isRight, either)
1011>>> import Data.Maybe (fromJust, isJust)
1112>>> import Data.Function (fix, (&))
@@ -32,6 +33,7 @@ For APIs that have not been released yet.
3233>>> import qualified Streamly.Internal.Data.Fold as Fold
3334>>> import qualified Streamly.Internal.Data.Parser as Parser
3435>>> import qualified Streamly.Internal.Data.Stream as Stream
36+>>> import qualified Streamly.Internal.Data.StreamK as StreamK
3537>>> import qualified Streamly.Internal.Data.Unfold as Unfold
3638>>> import qualified Streamly.Internal.FileSystem.Dir as Dir
3739-}
core/src/DocTestFileSystemHandle.hs+1−1
@@ -8,7 +8,7 @@
88
99For APIs that have not been released yet.
1010
11−>>> import qualified Streamly.Internal.Data.Array as Array (writeNUnsafe)
11+>>> import qualified Streamly.Internal.Data.Array as Array (unsafeCreateOf)
1212>>> import qualified Streamly.Internal.Data.Unfold as Unfold (first)
1313>>> import qualified Streamly.Internal.FileSystem.Handle as Handle
1414>>> import qualified Streamly.Internal.System.IO as IO (defaultChunkSize)
core/src/Streamly/Console/Stdio.hs+1−27
@@ -24,33 +24,7 @@ module Streamly.Console.Stdio
2424 -- * Write (stderr)
2525 , writeErr
2626 , writeErrChunks
27−
28− -- * Deprecated
29− , read
30− , readChunks
3127 )
3228where
3329
34−import Control.Monad.IO.Class (MonadIO(..))
35−import Data.Word (Word8)
36−import Streamly.Internal.Data.Array.Type (Array)
37−import Streamly.Internal.Data.Unfold (Unfold)
38−
39−import Streamly.Internal.Console.Stdio hiding (read, readChunks)
40−import Prelude hiding (read)
41−
42−-- Same as 'reader'
43−--
44−-- @since 0.8.0
45−{-# DEPRECATED read "Please use 'reader' instead" #-}
46−{-# INLINE read #-}
47−read :: MonadIO m => Unfold m () Word8
48−read = reader
49−
50−-- Same as 'chunkReader'
51−--
52−-- @since 0.8.0
53−{-# DEPRECATED readChunks "Please use 'chunkReader' instead" #-}
54−{-# INLINE readChunks #-}
55−readChunks :: MonadIO m => Unfold m () (Array Word8)
56−readChunks = chunkReader
30+import Streamly.Internal.Console.Stdio
core/src/Streamly/Data/Array.hs+40−39
@@ -31,28 +31,45 @@ module Streamly.Data.Array
3131 -- * The Array Type
3232 Array
3333
34+ -- * Pinning & Unpinning
35+ -- | Arrays are created unpinned by default unless pinned versions of
36+ -- creation APIs are used. Look for APIs with @pinned@ prefix in
37+ -- "Streamly.Internal.Data.Array" for some unreleased pinned creation APIs.
38+ -- If an array is to be sent to the OS without any further modification
39+ -- then it should be created pinned in the first place instead of pinning
40+ -- it later. Pinning an unpinned array has a copy overhead. OS interfacing
41+ -- APIs create a pinned array directly or convert an unpinned array to
42+ -- pinned array before sending it to the OS.
43+ , pin
44+ , unpin
45+ , isPinned
46+
3447 -- * Construction
3548 -- | When performance matters, the fastest way to generate an array is
36− -- 'writeN'. 'IsList' and 'IsString' instances can be
49+ -- 'createOf'. 'IsList' and 'IsString' instances can be
3750 -- used to conveniently construct arrays from literal values.
3851 -- 'OverloadedLists' extension or 'fromList' can be used to construct an
3952 -- array from a list literal. Similarly, 'OverloadedStrings' extension or
4053 -- 'fromList' can be used to construct an array from a string literal.
4154
42− -- Pure List APIs
55+ -- ** From Stream
56+ , createOf
57+ , create
58+ , writeLastN -- drop old (ring buffer)
59+
60+ -- ** From List
4361 , fromListN
4462 , fromList
4563
46− -- Monadic APIs
47− , writeN -- drop new
48− , write -- full buffer
49− , writeLastN -- drop old (ring buffer)
50−
51− -- * Conversion
64+ -- * To List
5265 -- 'GHC.Exts.toList' from "GHC.Exts" can be used to convert an array to a
5366 -- list.
5467 , toList
5568
69+ -- * To Stream
70+ , read
71+ , readRev
72+
5673 -- * Unfolds
5774 , reader
5875 , readerRev
@@ -66,25 +83,24 @@ module Streamly.Data.Array
6683 -- , (!!)
6784 , getIndex
6885
69− -- * Unbox Type Class
70− , Unbox (..)
86+ -- * Serialization
87+ , pinnedSerialize
88+ , deserialize
7189
72− -- * Pinning & Unpinning
73− , pin
74− , unpin
75− , isPinned
90+ -- * Re-exports
91+ , Unbox (..)
92+ , Serialize(..)
7693
7794 -- * Deprecated
78− , read
79− , readRev
95+ , writeN -- drop new
96+ , write -- full buffer
8097 )
8198where
8299
83100#include "inline.hs"
84101
85−import Streamly.Internal.Data.Array hiding (read, readRev)
86−import Streamly.Internal.Data.Unbox (Unbox (..))
87−import Streamly.Internal.Data.Unfold (Unfold)
102+import Streamly.Internal.Data.Array
103+import Streamly.Internal.Data.MutByteArray (Unbox(..), Serialize(..))
88104
89105import Prelude hiding (read, length)
90106
@@ -102,7 +118,7 @@ import Prelude hiding (read, length)
102118--
103119-- Convert array to stream, and fold the stream:
104120--
105−-- >>> fold f arr = Stream.unfold Array.reader arr & Stream.fold f
121+-- >>> fold f arr = Array.read arr & Stream.fold f
106122-- >>> fold Fold.sum (Array.fromList [1,2,3::Int])
107123-- 6
108124--
@@ -110,7 +126,7 @@ import Prelude hiding (read, length)
110126--
111127-- Convert array to stream, transform, and fold back to array:
112128--
113−-- >>> amap f arr = Stream.unfold Array.reader arr & fmap f & Stream.fold Array.write
129+-- >>> amap f arr = Array.read arr & fmap f & Stream.fold (Array.createOf (Array.length arr))
114130-- >>> amap (+1) (Array.fromList [1,2,3::Int])
115131-- fromList [2,3,4]
116132--
@@ -137,12 +153,11 @@ import Prelude hiding (read, length)
137153-- See the 'fromPureStream' unreleased API to generate an array from an
138154-- Identity stream safely without using MonadIO constraint.
139155--
140−--
141156-- Note that 'Identity' streams can be generalized to IO streams:
142157--
143−-- >>> pure = Stream.fromList [1,2,3] :: Stream Identity Int
144−-- >>> generally = Stream.morphInner (return . runIdentity)
145−-- >>> Stream.fold Array.write (generally pure :: Stream IO Int)
158+-- >>> stream = Stream.fromList [1,2,3] :: Stream Identity Int
159+-- >>> pure2IO s = Stream.morphInner (return . runIdentity) s :: Stream IO Int
160+-- >>> Stream.fold Array.create (pure2IO stream)
146161-- fromList [1,2,3]
147162--
148163-- == Programming Tips
@@ -150,17 +165,3 @@ import Prelude hiding (read, length)
150165-- This module is designed to be imported qualified:
151166--
152167-- >>> import qualified Streamly.Data.Array as Array
153−
154−-- | Same as 'reader'
155−--
156−{-# DEPRECATED read "Please use 'reader' instead" #-}
157−{-# INLINE_NORMAL read #-}
158−read :: (Monad m, Unbox a) => Unfold m (Array a) a
159−read = reader
160−
161−-- | Same as 'readerRev'
162−--
163−{-# DEPRECATED readRev "Please use 'readerRev' instead" #-}
164−{-# INLINE_NORMAL readRev #-}
165−readRev :: (Monad m, Unbox a) => Unfold m (Array a) a
166−readRev = readerRev
core/src/Streamly/Data/Array/Generic.hs+16−13
@@ -15,34 +15,37 @@ module Streamly.Data.Array.Generic
1515 ( Array
1616
1717 -- * Construction
18− , A.fromListN
19− , A.fromList
18+ , fromListN
19+ , fromList
2020
2121 -- MonadicAPIs
22− , A.writeN
23− , A.write
22+ , createOf
23+ , create
2424
2525 -- * Conversion
26− , A.toList
26+ , toList
2727
2828 -- * Streams
29− , A.read
30− , A.readRev
29+ , read
30+ , readRev
3131
3232 -- * Unfolds
33− , A.reader
33+ , reader
3434 -- , A.readerRev
3535
3636 -- * Random Access
37− , A.length
38− , A.getIndex
37+ , length
38+ , getIndex
3939
4040 -- -- * Folding Arrays
4141 -- , A.streamFold
4242 -- , A.fold
43+
44+ -- * Deprecated
45+ , writeN
46+ , write
4347 )
4448where
4549
46−import Streamly.Internal.Data.Array.Generic (Array)
47−
48−import qualified Streamly.Internal.Data.Array.Generic as A
50+import Streamly.Internal.Data.Array.Generic
51+import Prelude hiding (length, read)
core/src/Streamly/Data/Fold.hs+150−118
@@ -10,6 +10,156 @@
1010-- Fast, composable stream consumers with ability to terminate, supporting
1111-- stream fusion.
1212--
13+-- == Using Folds
14+--
15+-- This module provides elementary folds and fold combinators that can be used
16+-- to consume a stream of data and reduce it to a final value, or transform it
17+-- in a stateful manner using scans. A data stream can be reduced into a stream
18+-- of folded data elements by folding segments of the stream. Fold combinators
19+-- can be used to compose multiple folds in parallel or to create a pipeline of
20+-- folds such that the next fold consumes the result of the previous fold. To
21+-- run these folds on a stream see 'Streamly.Data.Stream.fold',
22+-- 'Streamly.Data.Stream.scan', 'Streamly.Data.Stream.postscan',
23+-- 'Streamly.Data.Stream.scanMaybe', 'Streamly.Data.Stream.foldMany' and other
24+-- operations accepting 'Fold' type as argument "Streamly.Data.Stream".
25+--
26+-- == Reducing a Stream
27+--
28+-- A 'Fold' is a consumer of a stream of values. A fold driver (such as
29+-- 'Streamly.Data.Stream.fold') initializes the fold @accumulator@, runs the
30+-- fold @step@ function in a loop, processing the input stream one element at a
31+-- time and accumulating the result. The loop continues until the fold
32+-- terminates, at which point the accumulated result is returned.
33+--
34+-- For example, a 'sum' Fold represents a stream consumer that adds the values
35+-- in the input stream:
36+--
37+-- >>> Stream.fold Fold.sum $ Stream.fromList [1..100]
38+-- 5050
39+--
40+-- Conceptually, a 'Fold' is a data type that mimics a strict left fold
41+-- ('Data.List.foldl'). The above example is similar to a left fold using
42+-- @(+)@ as the step and @0@ as the initial value of the accumulator:
43+--
44+-- >>> Data.List.foldl' (+) 0 [1..100]
45+-- 5050
46+--
47+-- 'Fold's have an early termination capability e.g. the 'one' fold terminates
48+-- after consuming one element:
49+--
50+-- >>> Stream.fold Fold.one $ Stream.fromList [1..]
51+-- Just 1
52+--
53+-- The above example is similar to the following right fold:
54+--
55+-- >>> Prelude.foldr (\x _ -> Just x) Nothing [1..]
56+-- Just 1
57+--
58+-- 'Fold's can be combined together using combinators. For example, to create a
59+-- fold that sums first two elements in a stream:
60+--
61+-- >>> sumTwo = Fold.take 2 Fold.sum
62+-- >>> Stream.fold sumTwo $ Stream.fromList [1..100]
63+-- 3
64+--
65+-- == Parallel Composition
66+--
67+-- Folds can be combined to run in parallel on the same input. For example, to
68+-- compute the average of numbers in a stream without going through the stream
69+-- twice:
70+--
71+-- >>> avg = Fold.teeWith (/) Fold.sum (fmap fromIntegral Fold.length)
72+-- >>> Stream.fold avg $ Stream.fromList [1.0..100.0]
73+-- 50.5
74+--
75+-- Folds can be combined so as to partition the input stream over multiple
76+-- folds. For example, to count even and odd numbers in a stream:
77+--
78+-- >>> split n = if even n then Left n else Right n
79+-- >>> stream = fmap split $ Stream.fromList [1..100]
80+-- >>> countEven = fmap (("Even " ++) . show) Fold.length
81+-- >>> countOdd = fmap (("Odd " ++) . show) Fold.length
82+-- >>> f = Fold.partition countEven countOdd
83+-- >>> Stream.fold f stream
84+-- ("Even 50","Odd 50")
85+--
86+-- == Sequential Composition
87+--
88+-- Terminating folds can be combined to parse the stream serially such that the
89+-- first fold consumes the input until it terminates and the second fold
90+-- consumes the rest of the input until it terminates:
91+--
92+-- >>> f = Fold.splitWith (,) (Fold.take 8 Fold.toList) (Fold.takeEndBy (== '\n') Fold.toList)
93+-- >>> Stream.fold f $ Stream.fromList "header: hello\n"
94+-- ("header: ","hello\n")
95+--
96+-- == Splitting a Stream
97+--
98+-- A 'Fold' can be applied repeatedly on a stream to transform it to a stream
99+-- of fold results. To split a stream on newlines:
100+--
101+-- >>> f = Fold.takeEndBy (== '\n') Fold.toList
102+-- >>> Stream.fold Fold.toList $ Stream.foldMany f $ Stream.fromList "Hello there!\nHow are you\n"
103+-- ["Hello there!\n","How are you\n"]
104+--
105+-- Similarly, we can split the input of a fold too:
106+--
107+-- >>> Stream.fold (Fold.many f Fold.toList) $ Stream.fromList "Hello there!\nHow are you\n"
108+-- ["Hello there!\n","How are you\n"]
109+--
110+-- == Folds vs. Streams
111+--
112+-- We can often use streams or folds to achieve the same goal. However, streams
113+-- are more efficient in composition of producers (e.g.
114+-- 'Data.Stream.append' or 'Data.Stream.mergeBy') whereas folds are
115+-- more efficient in composition of consumers (e.g. 'splitWith', 'partition'
116+-- or 'teeWith').
117+--
118+-- Streams are producers, transformations on streams happen on the output side:
119+--
120+-- >>> :{
121+-- f stream =
122+-- Stream.filter odd stream
123+-- & fmap (+1)
124+-- & Stream.fold Fold.sum
125+-- :}
126+--
127+-- >>> f $ Stream.fromList [1..100 :: Int]
128+-- 2550
129+--
130+-- Folds are stream consumers with an input stream and an output value, stream
131+-- transformations on folds happen on the input side:
132+--
133+-- >>> :{
134+-- f =
135+-- Fold.filter odd
136+-- $ Fold.lmap (+1)
137+-- $ Fold.sum
138+-- :}
139+--
140+-- >>> Stream.fold f $ Stream.fromList [1..100 :: Int]
141+-- 2550
142+--
143+-- Notice the similiarity in the definition of @f@ in both cases, the only
144+-- difference is the composition by @&@ vs @$@ and the use @lmap@ vs @map@, the
145+-- difference is due to output vs input side transformations.
146+--
147+-- == Fusion Limitations
148+--
149+-- Folds support stream fusion for generating loops comparable to the speed of
150+-- C. However, it has some limitations. For fusion to work, the folds must be
151+-- inlined, folds must be statically known and not generated dynamically, folds
152+-- should not be passed recursively.
153+--
154+-- Another limitation is due to the quadratic complexity causing slowdown when
155+-- too many nested compositions are used. Especially, the performance of the
156+-- Applicative instance and splitting operations (e.g. 'splitWith') degrades
157+-- quadratically (O(n^2)) when combined @n@ times, roughly 8 or less sequenced
158+-- operations are fine. For these cases folds can be converted to parsers and
159+-- then used as ParserK.
160+--
161+-- == Experimental APIs
162+--
13163-- Please refer to "Streamly.Internal.Data.Fold" for more functions that have
14164-- not yet been released.
15165
@@ -21,9 +171,6 @@ module Streamly.Data.Fold
21171 --
22172 -- $setup
23173
24− -- * Overview
25− -- $overview
26−
27174 -- * Running A Fold
28175 drive
29176 -- XXX Should we have a stream returning function in fold module?
@@ -258,121 +405,6 @@ import Streamly.Internal.Data.Fold
258405
259406#include "DocTestDataFold.hs"
260407
261−-- $overview
262−--
263−-- A 'Fold' is a consumer of a stream of values. A fold driver (such as
264−-- 'Streamly.Data.Stream.fold') initializes the fold @accumulator@, runs the
265−-- fold @step@ function in a loop, processing the input stream one element at a
266−-- time and accumulating the result. The loop continues until the fold
267−-- terminates, at which point the accumulated result is returned.
268−--
269−-- For example, a 'sum' Fold represents a stream consumer that adds the values
270−-- in the input stream:
271−--
272−-- >>> Stream.fold Fold.sum $ Stream.fromList [1..100]
273−-- 5050
274−--
275−-- Conceptually, a 'Fold' is a data type that mimics a strict left fold
276−-- ('Data.List.foldl'). The above example is similar to a left fold using
277−-- @(+)@ as the step and @0@ as the initial value of the accumulator:
278−--
279−-- >>> Data.List.foldl' (+) 0 [1..100]
280−-- 5050
281−--
282−-- 'Fold's have an early termination capability e.g. the 'one' fold terminates
283−-- after consuming one element:
284−--
285−-- >>> Stream.fold Fold.one $ Stream.fromList [1..]
286−-- Just 1
287−--
288−-- The above example is similar to the following right fold:
289−--
290−-- >>> Prelude.foldr (\x _ -> Just x) Nothing [1..]
291−-- Just 1
292−--
293−-- 'Fold's can be combined together using combinators. For example, to create a
294−-- fold that sums first two elements in a stream:
295−--
296−-- >>> sumTwo = Fold.take 2 Fold.sum
297−-- >>> Stream.fold sumTwo $ Stream.fromList [1..100]
298−-- 3
299−--
300−-- Folds can be combined to run in parallel on the same input. For example, to
301−-- compute the average of numbers in a stream without going through the stream
302−-- twice:
303−--
304−-- >>> avg = Fold.teeWith (/) Fold.sum (fmap fromIntegral Fold.length)
305−-- >>> Stream.fold avg $ Stream.fromList [1.0..100.0]
306−-- 50.5
307−--
308−-- Folds can be combined so as to partition the input stream over multiple
309−-- folds. For example, to count even and odd numbers in a stream:
310−--
311−-- >>> split n = if even n then Left n else Right n
312−-- >>> stream = fmap split $ Stream.fromList [1..100]
313−-- >>> countEven = fmap (("Even " ++) . show) Fold.length
314−-- >>> countOdd = fmap (("Odd " ++) . show) Fold.length
315−-- >>> f = Fold.partition countEven countOdd
316−-- >>> Stream.fold f stream
317−-- ("Even 50","Odd 50")
318−--
319−-- Terminating folds can be combined to parse the stream serially such that the
320−-- first fold consumes the input until it terminates and the second fold
321−-- consumes the rest of the input until it terminates:
322−--
323−-- >>> f = Fold.splitWith (,) (Fold.take 8 Fold.toList) (Fold.takeEndBy (== '\n') Fold.toList)
324−-- >>> Stream.fold f $ Stream.fromList "header: hello\n"
325−-- ("header: ","hello\n")
326−--
327−-- A 'Fold' can be applied repeatedly on a stream to transform it to a stream
328−-- of fold results. To split a stream on newlines:
329−--
330−-- >>> f = Fold.takeEndBy (== '\n') Fold.toList
331−-- >>> Stream.fold Fold.toList $ Stream.foldMany f $ Stream.fromList "Hello there!\nHow are you\n"
332−-- ["Hello there!\n","How are you\n"]
333−--
334−-- Similarly, we can split the input of a fold too:
335−--
336−-- >>> Stream.fold (Fold.many f Fold.toList) $ Stream.fromList "Hello there!\nHow are you\n"
337−-- ["Hello there!\n","How are you\n"]
338−--
339−-- = Folds vs. Streams
340−--
341−-- We can often use streams or folds to achieve the same goal. However, streams
342−-- are more efficient in composition of producers (e.g.
343−-- 'Data.Stream.append' or 'Data.Stream.mergeBy') whereas folds are
344−-- more efficient in composition of consumers (e.g. 'splitWith', 'partition'
345−-- or 'teeWith').
346−--
347−-- Streams are producers, transformations on streams happen on the output side:
348−--
349−-- >>> :{
350−-- f stream =
351−-- Stream.filter odd stream
352−-- & fmap (+1)
353−-- & Stream.fold Fold.sum
354−-- :}
355−--
356−-- >>> f $ Stream.fromList [1..100 :: Int]
357−-- 2550
358−--
359−-- Folds are stream consumers with an input stream and an output value, stream
360−-- transformations on folds happen on the input side:
361−--
362−-- >>> :{
363−-- f =
364−-- Fold.filter odd
365−-- $ Fold.lmap (+1)
366−-- $ Fold.sum
367−-- :}
368−--
369−-- >>> Stream.fold f $ Stream.fromList [1..100 :: Int]
370−-- 2550
371−--
372−-- Notice the similiarity in the definition of @f@ in both cases, the only
373−-- difference is the composition by @&@ vs @$@ and the use @lmap@ vs @map@, the
374−-- difference is due to output vs input side transformations.
375−
376408--------------------------------------------------------------------------------
377409-- Deprecated
378410--------------------------------------------------------------------------------
core/src/Streamly/Data/MutArray.hs+18−12
@@ -32,22 +32,27 @@ module Streamly.Data.MutArray
3232 -- * Construction
3333
3434 -- Uninitialized Arrays
35− , new
36− , pinnedNew
35+ , emptyOf
36+ , pinnedEmptyOf
3737
3838 -- From containers
3939 , fromListN
4040 , fromList
41− , writeN -- drop new
42− , write -- full buffer
41+ , createOf
42+ , create
4343 -- writeLastN
4444
45+ -- * Pinning & Unpinning
46+ , pin
47+ , unpin
48+ , isPinned
49+
4550 -- * Appending elements
4651 , snoc
4752
4853 -- * Appending streams
49− , writeAppendN -- XXX Rename to writeSnocN?
50− , writeAppend -- XXX Rename to writeSnoc?
54+ , appendN
55+ , append
5156
5257 -- * Inplace mutation
5358 , putIndex
@@ -78,16 +83,17 @@ module Streamly.Data.MutArray
7883 -- * Size
7984 , length
8085
81− -- * Unbox Type Class
86+ -- * Re-exports
8287 , Unbox (..)
8388
84− -- * Pinning & Unpinning
85− , pin
86− , unpin
87− , isPinned
88−
8989 -- * Deprecated
9090 , newPinned
91+ , new
92+ , pinnedNew
93+ , writeN
94+ , write
95+ , writeAppendN
96+ , writeAppend
9197 )
9298where
9399
core/src/Streamly/Data/MutArray/Generic.hs+8−3
@@ -23,11 +23,11 @@ module Streamly.Data.MutArray.Generic
2323 MutArray
2424
2525 -- * Construction
26− , new
26+ , emptyOf
2727 , fromListN
2828 , fromList
29− , writeN
30− , write
29+ , createOf
30+ , create
3131
3232 -- * Appending elements
3333 , snoc
@@ -56,6 +56,11 @@ module Streamly.Data.MutArray.Generic
5656
5757 -- * Size
5858 , length
59+
60+ -- * Deprecated
61+ , new
62+ , writeN
63+ , write
5964 )
6065where
6166
core/src/Streamly/Data/MutByteArray.hsnew file+106
@@ -0,0 +1,106 @@
1+-- |
2+-- Module : Streamly.Data.MutByteArray
3+-- Copyright : (c) 2023 Composewell Technologies
4+-- License : BSD3-3-Clause
5+-- Maintainer : streamly@composewell.com
6+-- Portability : GHC
7+--
8+-- A low level byte Array type 'MutByteArray', along with type classes 'Unbox'
9+-- and 'Serialize' for fast binary serialization and deserialization of Haskell
10+-- values. Serialization, deserialization performance is similar to, and in
11+-- some cases many times better than the store package. Conceptually, the
12+-- 'Serialize' type class works in the same way as store.
13+--
14+-- Serialize instances are configurable to use constructor names (see
15+-- 'Streamly.Internal.Data.MutByteArray.encodeConstrNames'), record field names (see
16+-- 'Streamly.Internal.Data.MutByteArray.encodeRecordFields') instead of binary
17+-- encoded values. This is an experimental feature which allows JSON like
18+-- properties with faster speed. For example, you can change the order of
19+-- constructors or record fields without affecting serialized value.
20+--
21+-- Higher level unboxed array modules "Streamly.Data.Array" and
22+-- "Streamly.Data.MutArray" are built on top of this module. Unboxed arrays are
23+-- essentially serialized Haskell values. Array modules provide higher level
24+-- serialization routines like 'Streamly.Internal.Data.Array.pinnedSerialize'
25+-- and 'Streamly.Internal.Data.Array.deserialize' from the
26+-- "Streamly.Internal.Data.Array" module.
27+--
28+-- == Mutable Byte Array
29+--
30+-- 'MutByteArray' is a primitive mutable array in the IO monad. 'Unbox' and
31+-- 'Serialize' type classes use this primitive array to serialize data to and
32+-- deserialize it from. This array is used to build higher level unboxed
33+-- array types 'Streamly.Data.MutArray.MutArray' and 'Streamly.Data.Array.Array'.
34+--
35+-- == Using Unbox
36+--
37+-- The 'Unbox' type class is simple and used to serialize non-recursive fixed
38+-- size data types. This type class is primarily used to implement unboxed
39+-- arrays. Unboxed arrays are just a sequence of serialized fixed length
40+-- Haskell data types. Instances of this type class can be derived using
41+-- 'Generic' or template haskell based deriving functions provided in this
42+-- module.
43+--
44+-- Writing a data type to an array using the array creation routines in
45+-- "Streamly.Data.Array" or "Streamly.Data.MutArray" (e.g. @writeN@ or
46+-- @fromListN@), serializes the type to the array. Similarly, reading the data
47+-- type from the array deserializes it. You can also serialize and deserialize
48+-- directly to and from a 'MutByteArray', using the type class methods.
49+--
50+-- == Using Serialize
51+--
52+-- The 'Serialize' type class is a superset of the 'Unbox' type class, it can
53+-- serialize variable length data types as well e.g. Haskell lists. Use
54+-- 'deriveSerialize' to derive the instances of the type class automatically
55+-- and then use the type class methods to serialize and deserialize to and from
56+-- a 'MutByteArray'.
57+--
58+-- See 'Streamly.Internal.Data.Array.pinnedSerialize' and
59+-- 'Streamly.Internal.Data.Array.deserialize' for 'Array' type based
60+-- serialization.
61+--
62+-- == Comparing serialized values
63+--
64+-- When using the `Unbox` type class the same value may result in differing
65+-- serialized bytes because of unused uninitialized data in case of sum types.
66+-- Therefore, byte comparison of serialized values is not reliable.
67+--
68+-- However, the 'Serialize' type class guarantees that the serialized values
69+-- are always exactly the same and byte comparison of serialized is reliable.
70+--
71+module Streamly.Data.MutByteArray
72+ (
73+
74+ -- * Mutable Byte Array
75+ -- | The standard way to read from or write to a 'MutByteArray' is by using
76+ -- the 'Unbox' or 'Serialize' type class methods.
77+ MutByteArray
78+ , isPinned
79+ , pin
80+ , unpin
81+ , new
82+ , pinnedNew
83+
84+ -- * Unbox
85+ , Unbox(..)
86+ , deriveUnbox
87+
88+ -- * Serialize
89+ , Serialize(..)
90+
91+ -- ** Instance Config
92+ , SerializeConfig
93+ , inlineAddSizeTo
94+ , inlineSerializeAt
95+ , inlineDeserializeAt
96+
97+ -- ** Instance Deriving
98+ , deriveSerialize
99+ , deriveSerializeWith
100+ ) where
101+
102+--------------------------------------------------------------------------------
103+-- Imports
104+--------------------------------------------------------------------------------
105+
106+import Streamly.Internal.Data.MutByteArray
core/src/Streamly/Data/Parser.hs+130−60
@@ -7,12 +7,137 @@
77-- Stability : pre-release
88-- Portability : GHC
99--
10−-- Fast, composable stream consumers with ability to terminate, backtrack and
11−-- fail, supporting stream fusion. Parsers are a natural extension of
12−-- "Streamly.Data.Fold". Parsers and folds can be interconverted.
10+-- Parsers are stream consumers like folds with the following differences:
1311--
14−-- Please refer to "Streamly.Internal.Data.Parser" for functions that have
15−-- not yet been released.
12+-- * folds cannot fail but parsers can fail and backtrack.
13+-- * folds can be composed as a Tee but parsers cannot.
14+-- * folds can be used for scanning but parsers cannot.
15+-- * folds can be converted to parsers.
16+--
17+-- This module implements parsers with stream fusion which compile to efficient
18+-- loops comparable to the speed of C.
19+--
20+-- == Using Parsers
21+--
22+-- This module provides elementary parsers and parser combinators that can be
23+-- used to parse a stream of data. Additionally, all the folds from the
24+-- "Streamly.Data.Fold" module can be converted to parsers using 'fromFold'.
25+-- All the parsing functionality provided by popular parsing libraries, and
26+-- more is available. Also see "Streamly.Unicode.Parser" module for Char stream
27+-- parsers.
28+--
29+-- A data stream can be transformed to a stream of parsed data elements. Parser
30+-- combinators can be used to create a pipeline of folds or parsers such that
31+-- the next fold or parser consumes the result of the previous parser. See
32+-- 'Streamly.Data.Stream.parse' and 'Streamly.Data.Stream.parseMany' to run
33+-- these parsers on a stream.
34+--
35+-- == Parser vs ParserK
36+--
37+-- There are two functionally equivalent parsing modules,
38+-- "Streamly.Data.Parser" (this module) and "Streamly.Data.ParserK". The latter
39+-- is a CPS based wrapper over the former, and can be used for parsing in
40+-- general. "Streamly.Data.Parser" enables stream fusion and should be
41+-- preferred over "Streamly.Data.ParserK" for high performance stream parsing
42+-- use cases. However, there are a few cases where this module is not
43+-- suitable and ParserK should be used instead.
44+--
45+-- For static fusion, parser combinators have to use strict pattern matching on
46+-- arguments of type Parser. This leads to infinte loop when a parser is
47+-- defined recursively, due to strict evaluation of the recursive call. For
48+-- example, the following implementation loops infinitely because of the
49+-- recursive use of parser @p@ in the @*>@ combinator:
50+--
51+-- >>> import Streamly.Data.Parser (Parser)
52+-- >>> import qualified Streamly.Data.Fold as Fold
53+-- >>> import qualified Streamly.Data.Parser as Parser
54+-- >>> import qualified Streamly.Data.Stream as Stream
55+-- >>> import Control.Applicative ((<|>))
56+--
57+-- >>> :{
58+-- >>> p :: Monad m => Parser Char m String
59+-- >>> p = Parser.satisfy (== '(') *> p <|> Parser.fromFold Fold.toList
60+-- >>> :}
61+--
62+-- Use ParserK when recursive use is required:
63+--
64+-- >>> import Streamly.Data.ParserK (ParserK)
65+-- >>> import qualified Streamly.Data.StreamK as StreamK
66+-- >>> import qualified Streamly.Internal.Data.StreamK as StreamK (parse)
67+-- >>> import qualified Streamly.Internal.Data.ParserK as ParserK (adapt)
68+--
69+-- >>> :{
70+-- >>> p :: Monad m => ParserK Char m String
71+-- >>> p = ParserK.adapt (Parser.satisfy (== '(')) *> p <|> ParserK.adapt (Parser.fromFold Fold.toList)
72+-- >>> :}
73+--
74+-- >>> StreamK.parse p $ StreamK.fromStream $ Stream.fromList "hello"
75+-- Right "hello"
76+--
77+-- For this reason Applicative, Alternative or Monad compositions with
78+-- recursion cannot be used with the 'Parser' type. Alternative type class based
79+-- operations like 'asum' and Alternative based generic parser combinators use
80+-- recursion. Similarly, Applicative type class based operations like
81+-- 'Prelude.sequence' use recursion. Custom implementations of many such
82+-- operations are provided in this module (e.g. 'some', 'many'), and those
83+-- should be used instead.
84+--
85+-- Another limitation of Parser type is due to the quadratic complexity causing
86+-- slowdown when too many nested compositions are used. Especially Applicative,
87+-- Monad, Alternative instances, and sequenced parsing operations (e.g. nested
88+-- 'one', and 'splitWith') degrade the performance quadratically (O(n^2)) when
89+-- combined @n@ times, roughly 8 or less sequenced parsers are fine. READ THE
90+-- DOCS OF APPLICATIVE, MONAD AND ALTERNATIVE INSTANCES.
91+--
92+-- == Streaming Parsers
93+--
94+-- With 'Streamly.Data.ParserK.ParserK' you can use the generic Alternative
95+-- type class based parsers from the
96+-- <https://hackage.haskell.org/package/parser-combinators parser-combinators>
97+-- library or similar. However, we recommend that you use the equivalent
98+-- functionality from this module for better performance and for streaming
99+-- behavior.
100+--
101+-- Firstly, the combinators in this module are faster due to stream fusion.
102+-- Secondly, these are streaming in nature as the results can be passed
103+-- directly to other stream consumers (folds or parsers). The Alternative type
104+-- class based parsers would end up buffering all the results in lists before
105+-- they can be consumed.
106+--
107+-- When recursion or heavy nesting is needed use ParserK.
108+--
109+-- == Error Reporting
110+--
111+-- These parsers do not report the error context (e.g. line number or column).
112+-- This may be supported in future.
113+--
114+-- == Monad Transformer Stack
115+--
116+-- 'MonadTrans' instance is not provided. If the 'Parser' type is the top most
117+-- layer (which should be the case almost always) you can just use 'fromEffect'
118+-- to execute the lower layer monad effects.
119+--
120+-- == Parser vs ParserK Implementation
121+--
122+-- The 'Parser' type represents a stream consumer by composing state as data
123+-- which enables stream fusion. Stream fusion generates a tight loop without
124+-- any constructor allocations between the stages, providing C like performance
125+-- for the loop. Stream fusion works when multiple functions are combined in a
126+-- pipeline statically. Therefore, the operations in this module must be
127+-- inlined and must not be used recursively to allow for stream fusion.
128+--
129+-- The 'ParserK' type represents a stream consumer by composing function calls,
130+-- therefore, a function call overhead is incurred at each composition. It is
131+-- quite fast in general but may be a few times slower than a fused parser.
132+-- However, it allows for scalable dynamic composition especially parsers can
133+-- be used in recursive calls. Using the 'ParserK' type operations like
134+-- 'splitWith' provide linear (O(n)) performance with respect to the number of
135+-- compositions.
136+--
137+-- == Experimental APIs
138+--
139+-- Please refer to "Streamly.Internal.Data.Parser" for functions that have not
140+-- yet been released.
16141--
17142module Streamly.Data.Parser
18143 (
@@ -22,9 +147,6 @@ module Streamly.Data.Parser
22147 --
23148 -- $setup
24149
25− -- * Overview
26− -- $overview
27−
28150 -- * Parser Type
29151 Parser
30152
@@ -123,55 +245,3 @@ import Streamly.Internal.Data.Parser
123245import Prelude hiding (dropWhile, takeWhile, filter)
124246
125247#include "DocTestDataParser.hs"
126−
127−-- $overview
128−--
129−-- Several combinators in this module can be many times faster than CPS based
130−-- parsers because of stream fusion. For example,
131−-- 'Streamly.Internal.Data.Parser.many' combinator in this module is much
132−-- faster than the 'Control.Applicative.many' combinator of
133−-- 'Control.Applicative.Alternative' type class used by CPS based parsers.
134−--
135−-- The use of 'Alternative' type class, in parsers has another drawback.
136−-- Alternative based parsers use plain Haskell lists to collect the results. In
137−-- a strict Monad like IO, the results are necessarily buffered before they can
138−-- be consumed. This may not perform optimally in streaming applications
139−-- processing large amounts of data. Equivalent combinators in this module can
140−-- consume the results of parsing using a 'Fold' or another parser, thus
141−-- providing a scalable and composable consumer.
142−--
143−-- Note that these parsers do not report the error context (e.g. line number or
144−-- column). This may be supported in future.
145−--
146−-- mtl instances are not provided. If the 'Parser' type is the top most layer
147−-- (which should be the case almost always) you can just use 'fromEffect' to
148−-- execute the lower layer monad effects.
149−--
150−-- == Performance Notes
151−--
152−-- The 'Parser' type represents a stream consumer by composing state as data
153−-- which enables stream fusion. Stream fusion generates a tight loop without
154−-- any constructor allocations between the stages, providing C like performance
155−-- for the loop. Stream fusion works when multiple functions are combined in a
156−-- pipeline statically. Therefore, the operations in this module must be
157−-- inlined and must not be used recursively to allow for stream fusion. Note
158−-- that operations like 'sequence', and 'asum' that compose pasrers using
159−-- recursion should be avoided with these parsers. You can use these with the
160−-- 'ParserK' module instead.
161−--
162−-- Using the 'Parser' type, parsing operations like 'one', 'splitWith' etc.
163−-- degrade quadratically (O(n^2)) when combined many times. If you need to
164−-- combine these operations, say more than 8 times in a single loop, then you
165−-- should consider using the continuation style parser type 'ParserK' instead.
166−-- Also, if you need to use these operations in a recursive loop you should use
167−-- 'ParserK' instead.
168−--
169−-- The 'ParserK' type represents a stream consumer by composing function calls,
170−-- therefore, a function call overhead is incurred at each composition. It is
171−-- quite fast in general but may be a few times slower than a fused parser.
172−-- However, it allows for scalable dynamic composition especially parsers can
173−-- be used in recursive calls. Using the 'ParserK' type operations like
174−-- 'splitWith' provide linear (O(n)) performance with respect to the number of
175−-- compositions..
176−--
177−-- 'Parser' and 'ParserK' types can be interconverted.
core/src/Streamly/Data/ParserK.hs+89−3
@@ -1,3 +1,4 @@
1+{-# LANGUAGE CPP #-}
12-- |
23-- Module : Streamly.Data.ParserK
34-- Copyright : (c) 2023 Composewell Technologies
@@ -6,19 +7,102 @@
67-- Stability : pre-release
78-- Portability : GHC
89--
9−-- Parsers using Continuation Passing Style (CPS). See notes in
10−-- "Streamly.Data.Parser" module to know when to use this module.
10+-- See the general notes about parsing in the "Streamly.Data.Parser" module.
11+-- This module implements a using Continuation Passing Style (CPS) wrapper over
12+-- the "Streamly.Data.Parser" module. It is as fast or faster than attoparsec.
1113--
12−-- To run a 'ParserK' use 'Streamly.Data.StreamK.parseChunks'.
14+-- == Parser vs ParserK
15+--
16+-- 'ParserK' is preferred over 'Streamly.Data.Parser.Parser' when extensive
17+-- applicative, alternative and monadic composition is required, or when
18+-- recursive or dynamic composition of parsers is required. The
19+-- 'Streamly.Data.Parser.Parser' type fuses statically and creates efficient
20+-- loops whereas 'ParserK' uses function call based composition and has
21+-- comparatively larger runtime overhead but it is better suited to the
22+-- specific use cases mentioned above. 'ParserK' also allows to efficient parse
23+-- a stream of arrays, it can also break the input stream into a parse result
24+-- and remaining stream so that the stream can be parsed independently in
25+-- segments.
26+--
27+-- == Using ParserK
28+--
29+-- All the parsers from the "Streamly.Data.Parser" module can be adapted to
30+-- ParserK using the 'Streamly.Data.ParserK.adaptC',
31+-- 'Streamly.Internal.Data.ParserK.adapt', and
32+-- 'Streamly.Internal.Data.ParserK.adaptCG' combinators.
33+--
34+-- 'Streamly.Data.StreamK.parseChunks' runs a parser on a stream of unboxed
35+-- arrays, this is the preferred and most efficient way to parse chunked input.
36+-- The more general 'Streamly.Data.StreamK.parseBreakChunks' function returns
37+-- the remaining stream as well along with the parse result. There are
38+-- 'Streamly.Internal.Data.StreamK.parseChunksGeneric',
39+-- 'Streamly.Internal.Data.StreamK.parseBreakChunksGeneric' as well to run
40+-- parsers on boxed arrays. 'Streamly.Internal.Data.StreamK.parse',
41+-- 'Streamly.Internal.Data.StreamK.parseBreak' run parsers on a stream of
42+-- individual elements instead of stream of arrays.
43+--
44+-- == Monadic Composition
45+--
46+-- Monad composition can be used for lookbehind parsers, we can dynamically
47+-- compose new parsers based on the results of the previously parsed values.
48+--
49+-- If we have to parse "a9" or "9a" but not "99" or "aa" we can use the
50+-- following non-monadic, backtracking parser:
51+--
52+-- >>> digits p1 p2 = ((:) <$> p1 <*> ((:) <$> p2 <*> pure []))
53+-- >>> :{
54+-- backtracking :: Monad m => ParserK Char m String
55+-- backtracking = ParserK.adapt $
56+-- digits (Parser.satisfy isDigit) (Parser.satisfy isAlpha)
57+-- <|>
58+-- digits (Parser.satisfy isAlpha) (Parser.satisfy isDigit)
59+-- :}
60+--
61+-- We know that if the first parse resulted in a digit at the first place then
62+-- the second parse is going to fail. However, we waste that information and
63+-- parse the first character again in the second parse only to know that it is
64+-- not an alphabetic char. By using lookbehind in a 'Monad' composition we can
65+-- avoid redundant work:
66+--
67+-- >>> data DigitOrAlpha = Digit Char | Alpha Char
68+--
69+-- >>> :{
70+-- lookbehind :: Monad m => ParserK Char m String
71+-- lookbehind = do
72+-- x1 <- ParserK.adapt $
73+-- Digit <$> Parser.satisfy isDigit
74+-- <|> Alpha <$> Parser.satisfy isAlpha
75+-- -- Note: the parse depends on what we parsed already
76+-- x2 <- ParserK.adapt $
77+-- case x1 of
78+-- Digit _ -> Parser.satisfy isAlpha
79+-- Alpha _ -> Parser.satisfy isDigit
80+-- return $ case x1 of
81+-- Digit x -> [x,x2]
82+-- Alpha x -> [x,x2]
83+-- :}
84+--
85+-- == Experimental APIs
86+--
87+-- Please refer to "Streamly.Internal.Data.ParserK" for functions that have
88+-- not yet been released.
1389--
1490module Streamly.Data.ParserK
1591 (
92+ -- * Setup
93+ -- | To execute the code examples provided in this module in ghci, please
94+ -- run the following commands first.
95+ --
96+ -- $setup
97+
1698 -- * Parser Type
1799 ParserK
18100
19101 -- * Parsers
20102 -- ** Conversions
103+ , adapt
21104 , adaptC
105+ , adaptCG
22106 -- , toParser
23107
24108 -- ** Without Input
@@ -41,6 +125,8 @@ import qualified Streamly.Internal.Data.Parser as ParserD
41125
42126import Streamly.Internal.Data.ParserK.Type
43127
128+#include "DocTestDataParserK.hs"
129+
44130{-# DEPRECATED fromFold "Please use \"ParserK.adaptC . Parser.fromFold\" instead." #-}
45131{-# INLINE fromFold #-}
46132fromFold :: (MonadIO m, Unbox a) => Fold m a b -> ParserK (Array a) m b
core/src/Streamly/Data/Serialize.hsdeleted−50
@@ -1,50 +0,0 @@
1−-- |
2−-- Module : Streamly.Data.Serialize
3−-- Copyright : (c) 2023 Composewell Technologies
4−-- License : BSD3-3-Clause
5−-- Maintainer : streamly@composewell.com
6−-- Portability : GHC
7−--
8−module Streamly.Data.Serialize
9− (
10−
11− -- * Types
12− MutableByteArray
13−
14− -- * Unbox
15− , Unbox(..)
16− , deriveUnbox
17− , deriveUnboxWith
18−
19− -- * Serialize
20− , Serialize(..)
21− , deriveSerialize
22− , deriveSerializeWith
23−
24− -- * Encoding
25− , encode
26− , pinnedEncode
27−
28− -- * Decoding
29− , decode
30− ) where
31−
32−--------------------------------------------------------------------------------
33−-- Imports
34−--------------------------------------------------------------------------------
35−
36−import Streamly.Internal.Data.Serialize
37− ( Serialize(..)
38− , decode
39− , encode
40− , pinnedEncode
41− )
42−import Streamly.Internal.Data.Serialize.TH
43− ( deriveSerialize
44− , deriveSerializeWith
45− )
46−import Streamly.Internal.Data.Unbox
47− ( MutableByteArray
48− , Unbox(..)
49− )
50−import Streamly.Internal.Data.Unbox.TH (deriveUnbox, deriveUnboxWith)
core/src/Streamly/Data/Stream.hs+131−36
@@ -70,6 +70,15 @@ module Streamly.Data.Stream
7070
7171 -- ** From Values
7272 -- | Generate a monadic stream from a seed value or values.
73+ --
74+ -- Non-empty pure streams can be cycled using repeat and unfold:
75+ --
76+ -- >>> cycle = Stream.unfoldMany Unfold.fromList . Stream.repeat
77+ --
78+ -- Non-empty StreamK can be cycled using append:
79+ --
80+ -- >>> cycle xs = let ys = xs `StreamK.append` ys in ys
81+ --
7382 , fromPure
7483 , fromEffect
7584 , repeat
@@ -212,21 +221,57 @@ module Streamly.Data.Stream
212221 -- operations like mapping a function over the stream.
213222 , foldrM
214223 , foldr
224+ -- foldr1
215225
216226 -- ** Specific Folds
217− -- | Usually you can use the folds in "Streamly.Data.Fold". However, some
218− -- folds that may be commonly used or may have an edge in performance in
219− -- some cases are provided here.
220− --
221− -- Useful idioms:
227+ -- | Streams are folded using folds in "Streamly.Data.Fold". Here are some
228+ -- idioms and equivalents of Data.List APIs using folds:
222229 --
223230 -- >>> foldlM' f a = Stream.fold (Fold.foldlM' f a)
224− -- >>> foldl1 f = Stream.fold (Fold.foldl1' f)
231+ -- >>> foldl1' f = Stream.fold (Fold.foldl1' f)
225232 -- >>> foldl' f a = Stream.fold (Fold.foldl' f a)
226233 -- >>> drain = Stream.fold Fold.drain
227234 -- >>> mapM_ f = Stream.fold (Fold.drainMapM f)
228235 -- >>> length = Stream.fold Fold.length
236+ -- >>> genericLength = Stream.fold Fold.lengthGeneric
229237 -- >>> head = Stream.fold Fold.one
238+ -- >>> last = Stream.fold Fold.latest
239+ -- >>> null = Stream.fold Fold.null
240+ -- >>> and = Stream.fold Fold.and
241+ -- >>> or = Stream.fold Fold.or
242+ -- >>> any p = Stream.fold (Fold.any p)
243+ -- >>> all p = Stream.fold (Fold.all p)
244+ -- >>> sum = Stream.fold Fold.sum
245+ -- >>> product = Stream.fold Fold.product
246+ -- >>> maximum = Stream.fold Fold.maximum
247+ -- >>> maximumBy cmp = Stream.fold (Fold.maximumBy cmp)
248+ -- >>> minimum = Stream.fold Fold.minimum
249+ -- >>> minimumBy cmp = Stream.fold (Fold.minimumBy cmp)
250+ -- >>> elem x = Stream.fold (Fold.elem x)
251+ -- >>> notElem x = Stream.fold (Fold.notElem x)
252+ -- >>> lookup x = Stream.fold (Fold.lookup x)
253+ -- >>> find p = Stream.fold (Fold.find p)
254+ -- >>> (!?) i = Stream.fold (Fold.index i)
255+ -- >>> genericIndex i = Stream.fold (Fold.indexGeneric i)
256+ -- >>> elemIndex x = Stream.fold (Fold.elemIndex x)
257+ -- >>> findIndex p = Stream.fold (Fold.findIndex p)
258+ --
259+ -- Some equivalents of Data.List APIs from the Stream module:
260+ --
261+ -- >>> head = fmap (fmap fst) . Stream.uncons
262+ -- >>> tail = fmap (fmap snd) . Stream.uncons
263+ -- >>> tail = Stream.tail -- unreleased API
264+ -- >>> init = Stream.init -- unreleased API
265+ --
266+ -- A Stream based toList fold implementation is provided below because it
267+ -- has a better performance compared to the fold.
268+
269+ -- Functions in Data.List, missing here:
270+ -- unsnoc = Stream.parseBreak (Parser.init Fold.toList)
271+ -- genericTake
272+ -- genericDrop
273+ -- genericSplitAt
274+ -- genericReplicate
230275 , toList
231276
232277 -- * Mapping
@@ -280,6 +325,15 @@ module Streamly.Data.Stream
280325 , scan
281326 , postscan
282327 -- XXX postscan1 can be implemented using Monoids or Refolds.
328+ -- The following scans from Data.List are not provided.
329+ -- XXX scanl
330+ -- XXX scanl1
331+ -- XXX scanr
332+ -- XXX scanr1
333+ -- XXX mapAccumL
334+ -- XXX mapAccumR
335+ -- XXX inits
336+ -- XXX tails
283337
284338 -- ** Specific scans
285339 -- Indexing can be considered as a special type of zipping where we zip a
@@ -288,6 +342,8 @@ module Streamly.Data.Stream
288342
289343 -- * Insertion
290344 -- | Add elements to the stream.
345+ --
346+ -- >>> insert = Stream.insertBy compare
291347
292348 -- Inserting elements is a special case of interleaving/merging streams.
293349 , insertBy
@@ -323,12 +379,15 @@ module Streamly.Data.Stream
323379 -- "Streamly.Internal.Data.Fold" can be used along with 'scanMaybe' to
324380 -- perform stateful filtering operations in general.
325381 --
326− -- Useful idioms:
382+ -- Idioms and equivalents of Data.List APIs:
327383 --
328384 -- >>> deleteBy cmp x = Stream.scanMaybe (Fold.deleteBy cmp x)
385+ -- >>> deleteBy = Stream.deleteBy -- unreleased API
386+ -- >>> delete = deleteBy (==)
329387 -- >>> findIndices p = Stream.scanMaybe (Fold.findIndices p)
330388 -- >>> elemIndices a = findIndices (== a)
331389 -- >>> uniq = Stream.scanMaybe (Fold.uniqBy (==))
390+ -- >>> partition p = Stream.fold (Fold.partition Fold.toList Fold.toList) . fmap (if p then Left else Right)
332391 , scanMaybe
333392 , take
334393 , takeWhile
@@ -336,6 +395,8 @@ module Streamly.Data.Stream
336395 , drop
337396 , dropWhile
338397 , dropWhileM
398+ -- XXX write to an array in reverse and then read in reverse
399+ -- > dropWhileEnd = reverse . dropWhile p . reverse
339400
340401 -- XXX These are available as scans in folds. We need to check the
341402 -- performance though. If these are common and we need convenient stream
@@ -372,6 +433,9 @@ module Streamly.Data.Stream
372433 -- >>> s = s1 `Stream.append` s2
373434
374435 -- ** Appending
436+ -- | Equivalent of Data.List append:
437+ --
438+ -- >>> (++) = Stream.append
375439 , append
376440
377441 -- ** Interleaving
@@ -382,8 +446,14 @@ module Streamly.Data.Stream
382446 , mergeByM
383447
384448 -- ** Zipping
449+ -- | Idioms and equivalents of Data.List APIs:
450+ --
451+ -- >>> zip = Stream.zipWith (,)
452+ -- >>> unzip = Stream.fold (Fold.unzip Fold.toList Fold.toList)
385453 , zipWith
386454 , zipWithM
455+ -- XXX zipWith3,4,5,6,7
456+ -- XXX unzip3,4,5,6,7
387457 -- , ZipStream (..)
388458
389459 -- ** Cross Product
@@ -416,48 +486,89 @@ module Streamly.Data.Stream
416486 -- One dimension loops are just a special case of nested loops. For
417487 -- example, 'concatMap' can degenerate to a simple map operation:
418488 --
419− -- > map f m = S.concatMap (\x -> S.fromPure (f x)) m
489+ -- >>> map f m = Stream.concatMap (\x -> Stream.fromPure (f x)) m
420490 --
421491 -- Similarly, 'concatMap' can perform filtering by mapping an element to a
422492 -- 'nil' stream:
423493 --
424− -- > filter p m = S.concatMap (\x -> if p x then S.fromPure x else S.nil) m
494+ -- >>> filter p m = Stream.concatMap (\x -> if p x then Stream.fromPure x else Stream.nil) m
425495 --
496+ -- Idioms and equivalents of Data.List APIs:
497+ --
498+ -- >>> concat = Stream.concatMap id
499+ -- >>> unlines = Stream.interposeSuffix '\n'
500+ -- >>> unlines = Stream.intercalateSuffix Unfold.fromList "\n"
501+ -- >>> unwords = Stream.interpose ' '
502+ -- >>> unwords = Stream.intercalate Unfold.fromList " "
426503
427504 , concatEffect
428505 , concatMap
429506 , concatMapM
430507
431508 -- * Repeated Fold
432− -- | Useful idioms:
509+ -- | Idioms and equivalents of Data.List APIs:
433510 --
434− -- >>> splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
435− -- >>> splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
436− -- >>> groupsBy eq f = Stream.parseMany (Parser.groupBy eq f)
437− -- >>> groupsByRolling eq f = Stream.parseMany (Parser.groupByRolling eq f)
438− -- >>> groupsOf n f = Stream.foldMany (Fold.take n f)
511+ -- >>> groupsOf n = Stream.foldMany (Fold.take n Fold.toList)
512+ -- >>> groupBy eq = Stream.groupsWhile eq Fold.toList
513+ -- >>> groupBy eq = Stream.parseMany (Parser.groupBy eq Fold.toList)
514+ -- >>> groupsByRolling eq = Stream.parseMany (Parser.groupByRolling eq Fold.toList)
515+ -- >>> groups = groupBy (==)
439516 , foldMany -- XXX Rename to foldRepeat
440517 , groupsOf
441518 , parseMany
442519
443520 -- * Splitting
521+ -- | Idioms and equivalents of Data.List APIs:
522+ --
523+ -- >>> splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
524+ -- >>> splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
525+ -- >>> lines = splitOnSuffix (== '\n')
526+ -- >>> words = Stream.wordsBy isSpace
527+ -- >>> splitAt n = Stream.fold (Fold.splitAt n Fold.toList Fold.toList)
528+ -- >>> span p = Parser.splitWith (,) (Parser.takeWhile p Fold.toList) (Parser.fromFold Fold.toList)
529+ -- >>> break p = span (not . p)
444530 , splitOn
445531 , wordsBy
446532
447533 -- * Buffered Operations
448534 -- | Operations that require buffering of the stream.
449535 -- Reverse is essentially a left fold followed by an unfold.
536+ --
537+ -- Idioms and equivalents of Data.List APIs:
538+ --
539+ -- >>> nub = Stream.fold Fold.toList . Stream.scanMaybe Fold.nub
540+ -- >>> nub = Stream.ordNub -- unreleased API
541+ -- >>> sortBy = StreamK.sortBy
542+ -- >>> sortOn f = StreamK.sortOn -- unreleased API
543+ -- >>> deleteFirstsBy = Stream.deleteFirstsBy -- unreleased
544+ -- >>> (\\) = Stream.deleteFirstsBy (==) -- unreleased
545+ -- >>> intersectBy = Stream.intersectBy -- unreleased
546+ -- >>> intersect = Stream.intersectBy (==) -- unreleased
547+ -- >>> unionBy = Stream.unionBy -- unreleased
548+ -- >>> union = Stream.unionBy (==) -- unreleased
549+ --
450550 , reverse
551+ -- XXX transpose: write the streams to arrays and then stream transposed.
552+ -- XXX subsequences
553+ -- XXX permutations
554+ -- , nub
555+ -- , ordNub
556+ -- , nubBy
451557
452558 -- * Multi-Stream folds
453559 -- | Operations that consume multiple streams at the same time.
454560 , eqBy
455561 , cmpBy
456562 , isPrefixOf
563+ , isInfixOf
564+ -- , isSuffixOf
565+ -- , isSuffixOfUnbox
457566 , isSubsequenceOf
458567
459568 -- trimming sequences
460− , stripPrefix
569+ -- , stripPrefix
570+ -- , stripSuffix
571+ -- , stripSuffixUnbox
461572
462573 -- Exceptions and resource management depend on the "exceptions" package
463574 -- XXX We can have IO Stream operations not depending on "exceptions"
@@ -511,25 +622,20 @@ module Streamly.Data.Stream
511622 , runReaderT
512623 , runStateT
513624
625+ -- XXX Arrays could be different types, therefore, this should be in
626+ -- specific array module. Or maybe we should abstract over array types.
514627 -- * Stream of Arrays
515628 , Array.chunksOf
516− , chunksSplitOn
517629 )
518630where
519631
520−import Control.Monad.IO.Class (MonadIO)
521−import Data.Word (Word8)
522−import Streamly.Data.Array (Array)
523632import Streamly.Internal.Data.Stream
524633import Prelude
525634 hiding (filter, drop, dropWhile, take, takeWhile, zipWith, foldr,
526− foldl, map, mapM, mapM_, sequence, all, any, sum, product, elem,
527− notElem, maximum, minimum, head, last, tail, length, null,
528− reverse, iterate, init, and, or, lookup, foldr1, (!!),
529− scanl, scanl1, repeat, replicate, concatMap, span)
635+ mapM, sequence, reverse, iterate, foldr1, repeat, replicate,
636+ concatMap)
530637
531638import qualified Streamly.Internal.Data.Array.Type as Array
532−import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
533639
534640#include "DocTestDataStream.hs"
535641
@@ -638,14 +744,3 @@ import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
638744--
639745-- 'Stream' and 'StreamK' types can be interconverted. See
640746-- "Streamly.Data.StreamK" module for conversion operations.
641−
642−-- | Split a stream of arrays on a given separator byte, dropping the separator
643−-- and coalescing all the arrays between two separators into a single array.
644−--
645−{-# INLINE chunksSplitOn #-}
646−chunksSplitOn
647− :: (MonadIO m)
648− => Word8
649− -> Stream m (Array Word8)
650− -> Stream m (Array Word8)
651−chunksSplitOn = ArrayStream.splitOn
core/src/Streamly/Data/StreamK.hs+2−1
@@ -95,7 +95,8 @@ module Streamly.Data.StreamK
9595 -- , foldBreak
9696
9797 -- ** Parsing
98− -- , parseBreak
98+ , parse
99+ , parseBreak
99100 , parseBreakChunks
100101 , parseChunks
101102
core/src/Streamly/Data/Unfold.hs+2
@@ -65,6 +65,8 @@ module Streamly.Data.Unfold
6565 -- ** Mapping on Input
6666 , lmap
6767 , lmapM
68+ , first
69+ , second
6870
6971 -- ** Mapping on Output
7072 , mapM
core/src/Streamly/FileSystem/Handle.hs+7−30
@@ -1,5 +1,4 @@
11{-# LANGUAGE CPP #-}
2−#include "inline.hs"
32
43-- |
54-- Module : Streamly.FileSystem.Handle
@@ -82,13 +81,13 @@ module Streamly.FileSystem.Handle
8281 -- position of the file handle. The stream ends as soon as EOF is
8382 -- encountered.
8483
85− -- -- *** Streams
86− -- , read
87− -- , readWith
88− -- , readChunks
89− -- , readChunksWith
84+ -- *** Streams
85+ , read
86+ , readWith
87+ , readChunks
88+ , readChunksWith
9089
91− -- -- *** Unfolds
90+ -- *** Unfolds
9291 , reader
9392 , readerWith
9493 , chunkReader
@@ -105,36 +104,14 @@ module Streamly.FileSystem.Handle
105104 , writeChunks
106105
107106 -- * Deprecated
108− , read
109107 , readWithBufferOf
110− , readChunks
111108 , readChunksWithBufferOf
112109 , writeChunksWithBufferOf
113110 , writeWithBufferOf
114111 )
115112where
116113
117−import Control.Monad.IO.Class (MonadIO(..))
118−import Data.Word (Word8)
119−import Streamly.Internal.Data.Array.Type (Array)
120−import Streamly.Internal.Data.Unfold.Type (Unfold)
121−import System.IO (Handle)
122−
123−import Streamly.Internal.FileSystem.Handle hiding (read, readChunks)
114+import Streamly.Internal.FileSystem.Handle
124115import Prelude hiding (read)
125116
126117#include "DocTestFileSystemHandle.hs"
127−
128−-- | Same as 'reader'
129−--
130−{-# DEPRECATED read "Please use 'reader' instead" #-}
131−{-# INLINE read #-}
132−read :: MonadIO m => Unfold m Handle Word8
133−read = reader
134−
135−-- | Same as 'chunkReader'
136−--
137−{-# DEPRECATED readChunks "Please use 'chunkReader' instead" #-}
138−{-# INLINE readChunks #-}
139−readChunks :: MonadIO m => Unfold m Handle (Array Word8)
140−readChunks = chunkReader
core/src/Streamly/Internal/Data/Array.hs+608−135
@@ -2,7 +2,6 @@
22-- |
33-- Module : Streamly.Internal.Data.Array
44-- Copyright : (c) 2019 Composewell Technologies
5−--
65-- License : BSD3
76-- Maintainer : streamly@composewell.com
87-- Stability : experimental
@@ -20,33 +19,21 @@ module Streamly.Internal.Data.Array
2019 module Streamly.Internal.Data.Array.Type
2120
2221 -- * Construction
23− -- Stream Folds
24− , fromStreamN
25− , fromStream
26−
2722 -- Monadic Folds
2823 , writeLastN
2924
30− -- * Elimination
31− -- ** Conversion
32−
33− -- ** Unfolds
34− , reader
35− , readerUnsafe
36− , producer -- experimental
37−
3825 -- * Random Access
3926 -- , (!!)
4027 , getIndex
4128 , getIndexRev
42− , last -- XXX getIndexLast?
43− , getIndices
44− , getIndicesFromThenTo
29+ , last -- XXX getLastIndex?
4530 -- , getIndicesFrom -- read from a given position to the end of file
4631 -- , getIndicesUpto -- read from beginning up to the given position
4732 -- , getIndicesFromTo
4833 -- , getIndicesFromRev -- read from a given position to the beginning of file
4934 -- , getIndicesUptoRev -- read from end to the given position in file
35+ , indexReader
36+ , indexReaderFromThenTo
5037
5138 -- * Size
5239 , null
@@ -54,6 +41,8 @@ module Streamly.Internal.Data.Array
5441 -- * Search
5542 , binarySearch
5643 , findIndicesOf
44+ -- getIndicesOf
45+ , indexFinder -- see splitOn
5746 -- , findIndexOf
5847 -- , find
5948
@@ -66,51 +55,89 @@ module Streamly.Internal.Data.Array
6655 -- * Subarrays
6756 , getSliceUnsafe
6857 -- , getSlice
69− , genSlicesFromLen
70− , getSlicesFromLen
58+ , sliceIndexerFromLen
59+ , slicerFromLen
7160 , splitOn
7261
7362 -- * Streaming Operations
7463 , streamTransform
7564
76− -- ** Folding
65+ -- * Folding
7766 , streamFold
7867 , fold
68+
69+ -- * Stream of Arrays
70+
71+ -- XXX these are probably not very useful to have in this module as we can
72+ -- express these idiomatically using streams.
73+ , interpose
74+ , interposeSuffix
75+ , intercalateSuffix
76+
77+ , compactLE
78+ , pinnedCompactLE
79+ , compactOnByte
80+ , compactOnByteSuffix
81+
82+ , foldBreakChunks
83+ , foldChunks
84+ , foldBreakChunksK
85+ , parseBreakChunksK
86+
87+ -- * Serialization
88+ , encodeAs
89+ , serialize
90+ , pinnedSerialize
91+ , deserialize
92+
93+ -- * Deprecated
94+ , genSlicesFromLen
95+ , getSlicesFromLen
96+ , getIndices
7997 )
8098where
8199
100+#include "assert.hs"
82101#include "inline.hs"
83102#include "ArrayMacros.h"
84103
85−import Control.Exception (assert)
86−import Control.Monad (when)
87104import Control.Monad.IO.Class (MonadIO(..))
105+-- import Data.Bifunctor (first)
106+-- import Data.Either (fromRight)
88107import Data.Functor.Identity (Identity)
89108import Data.Proxy (Proxy(..))
90109import Data.Word (Word8)
91110import Foreign.C.String (CString)
92111import Foreign.Ptr (castPtr)
93112import Foreign.Storable (Storable)
113+import GHC.Types (SPEC(..))
94114import Streamly.Internal.Data.Unbox (Unbox(..))
95115import Prelude hiding (length, null, last, map, (!!), read, concat)
96116
117+import Streamly.Internal.Data.MutByteArray.Type (PinnedState(..), MutByteArray)
118+import Streamly.Internal.Data.Serialize.Type (Serialize)
97119import Streamly.Internal.Data.Fold.Type (Fold(..))
98−import Streamly.Internal.Data.Producer.Type (Producer(..))
99−import Streamly.Internal.Data.Stream (Stream)
100−import Streamly.Internal.Data.Tuple.Strict (Tuple3Fused'(..))
120+import Streamly.Internal.Data.Parser (Parser(..), Initial(..), ParseError(..))
121+import Streamly.Internal.Data.Stream (Stream(..))
122+import Streamly.Internal.Data.StreamK (StreamK)
123+import Streamly.Internal.Data.SVar.Type (adaptState, defState)
124+import Streamly.Internal.Data.Tuple.Strict (Tuple'(..), Tuple3Fused'(..))
101125import Streamly.Internal.Data.Unfold.Type (Unfold(..))
102126import Streamly.Internal.System.IO (unsafeInlineIO)
103127
104−import qualified Streamly.Internal.Data.MutArray.Type as MA
128+import qualified Streamly.Internal.Data.Fold.Type as Fold
129+import qualified Streamly.Internal.Data.Serialize.Type as Serialize
130+import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
105131import qualified Streamly.Internal.Data.MutArray as MA
106−import qualified Streamly.Internal.Data.Array.Type as A
107132import qualified Streamly.Internal.Data.Fold as FL
108−import qualified Streamly.Internal.Data.Producer.Type as Producer
109−import qualified Streamly.Internal.Data.Producer as Producer
110133import qualified Streamly.Internal.Data.Ring as RB
134+import qualified Streamly.Internal.Data.Parser as Parser
135+-- import qualified Streamly.Internal.Data.ParserK as ParserK
111136import qualified Streamly.Internal.Data.Stream as D
112137import qualified Streamly.Internal.Data.Stream as Stream
138+import qualified Streamly.Internal.Data.StreamK as StreamK
113139import qualified Streamly.Internal.Data.Unfold as Unfold
140+import qualified Prelude
114141
115142import Streamly.Internal.Data.Array.Type
116143
@@ -139,82 +166,10 @@ import Streamly.Internal.Data.Array.Type
139166-- if we use one array for each chunk. This is still significant to add
140167-- pressure to GC.
141168
142−-------------------------------------------------------------------------------
143−-- Construction
144−-------------------------------------------------------------------------------
145−
146−-- | Create an 'Array' from the first N elements of a stream. The array is
147−-- allocated to size N, if the stream terminates before N elements then the
148−-- array may hold less than N elements.
149−--
150−-- /Pre-release/
151−{-# INLINE fromStreamN #-}
152−fromStreamN :: (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
153−fromStreamN n m = do
154− when (n < 0) $ error "writeN: negative write count specified"
155− A.fromStreamDN n m
156−
157−-- | Create an 'Array' from a stream. This is useful when we want to create a
158−-- single array from a stream of unknown size. 'writeN' is at least twice
159−-- as efficient when the size is already known.
160−--
161−-- Note that if the input stream is too large memory allocation for the array
162−-- may fail. When the stream size is not known, `chunksOf` followed by
163−-- processing of indvidual arrays in the resulting stream should be preferred.
164−--
165−-- /Pre-release/
166−{-# INLINE fromStream #-}
167−fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (Array a)
168−fromStream = Stream.fold A.write
169−-- write m = A.fromStreamD $ D.fromStreamK m
170−
171169-------------------------------------------------------------------------------
172170-- Elimination
173171-------------------------------------------------------------------------------
174172
175−{-# INLINE_NORMAL producer #-}
176−producer :: forall m a. (Monad m, Unbox a) => Producer m (Array a) a
177−producer =
178− Producer.translate A.unsafeThaw A.unsafeFreeze
179− $ MA.producerWith (return . unsafeInlineIO)
180−
181−-- | Unfold an array into a stream.
182−--
183−{-# INLINE_NORMAL reader #-}
184−reader :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
185−reader = Producer.simplify producer
186−
187−-- | Unfold an array into a stream, does not check the end of the array, the
188−-- user is responsible for terminating the stream within the array bounds. For
189−-- high performance application where the end condition can be determined by
190−-- a terminating fold.
191−--
192−-- Written in the hope that it may be faster than "read", however, in the case
193−-- for which this was written, "read" proves to be faster even though the core
194−-- generated with unsafeRead looks simpler.
195−--
196−-- /Pre-release/
197−--
198−{-# INLINE_NORMAL readerUnsafe #-}
199−readerUnsafe :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
200−readerUnsafe = Unfold step inject
201− where
202−
203− inject (Array contents start end) =
204− return (ArrayUnsafe contents end start)
205−
206− {-# INLINE_LATE step #-}
207− step (ArrayUnsafe contents end p) = do
208− -- unsafeInlineIO allows us to run this in Identity monad for pure
209− -- toList/foldr case which makes them much faster due to not
210− -- accumulating the list and fusing better with the pure consumers.
211− --
212− -- This should be safe as the array contents are guaranteed to be
213− -- evaluated/written to before we peek at them.
214− let !x = unsafeInlineIO $ peekByteIndex p contents
215− let !p1 = INDEX_NEXT(p,a)
216− return $ D.Yield x (ArrayUnsafe contents end p1)
217−
218173-- |
219174--
220175-- >>> null arr = Array.byteLength arr == 0
@@ -222,7 +177,7 @@ readerUnsafe = Unfold step inject
222177-- /Pre-release/
223178{-# INLINE null #-}
224179null :: Array a -> Bool
225−null arr = A.byteLength arr == 0
180+null arr = byteLength arr == 0
226181
227182-- | Like 'getIndex' but indexes the array in reverse from the end.
228183--
@@ -234,7 +189,7 @@ getIndexRev i arr =
234189 $ do
235190 let elemPtr = RINDEX_OF(arrEnd arr, i, a)
236191 if i >= 0 && elemPtr >= arrStart arr
237− then Just <$> peekByteIndex elemPtr (arrContents arr)
192+ then Just <$> peekAt elemPtr (arrContents arr)
238193 else return Nothing
239194
240195-- |
@@ -250,6 +205,8 @@ last = getIndexRev 0
250205-- Folds with Array as the container
251206-------------------------------------------------------------------------------
252207
208+-- XXX We should generate this from Ring.
209+
253210-- | @writeLastN n@ folds a maximum of @n@ elements from the end of the input
254211-- stream to an 'Array'.
255212--
@@ -258,7 +215,7 @@ writeLastN ::
258215 (Storable a, Unbox a, MonadIO m) => Int -> Fold m a (Array a)
259216writeLastN n
260217 | n <= 0 = fmap (const mempty) FL.drain
261− | otherwise = A.unsafeFreeze <$> Fold step initial done
218+ | otherwise = unsafeFreeze <$> Fold step initial done done
262219
263220 where
264221
@@ -271,11 +228,9 @@ writeLastN n
271228 in fmap f $ liftIO $ RB.new n
272229
273230 done (Tuple3Fused' rb rh i) = do
274− arr <- liftIO $ MA.new n
275− foldFunc i rh snoc' arr rb
276−
277− -- XXX We should write a read unfold for ring.
278− snoc' b a = liftIO $ MA.snocUnsafe b a
231+ arr <- MA.new n
232+ -- XXX We should write a read unfold for ring.
233+ foldFunc i rh MA.snocUnsafe arr rb
279234
280235 foldFunc i
281236 | i < n = RB.unsafeFoldRingM
@@ -301,19 +256,26 @@ binarySearch = undefined
301256-- compared to streams by using SIMD instructions.
302257-- We can also return a bit array instead.
303258
259+-- Can use SIMD.
260+
304261-- | Perform a linear search to find all the indices where a given element is
305262-- present in an array.
306263--
307264-- /Unimplemented/
308−findIndicesOf :: (a -> Bool) -> Unfold Identity (Array a) Int
309−findIndicesOf = undefined
265+indexFinder :: (a -> Bool) -> Unfold Identity (Array a) Int
266+indexFinder = undefined
267+
268+-- |
269+-- /Unimplemented/
270+findIndicesOf :: (a -> Bool) -> Array a -> Stream Identity Int
271+findIndicesOf p = Stream.unfold (indexFinder p)
310272
311273{-
312274findIndexOf :: (a -> Bool) -> Array a -> Maybe Int
313−findIndexOf p = Unfold.fold Fold.one . Stream.unfold (findIndicesOf p)
275+findIndexOf p = Unfold.fold Fold.one . Stream.unfold (indexFinder p)
314276
315277find :: (a -> Bool) -> Array a -> Bool
316−find = Unfold.fold Fold.null . Stream.unfold (findIndicesOf p)
278+find = Unfold.fold Fold.null . Stream.unfold (indexFinder p)
317279-}
318280
319281-------------------------------------------------------------------------------
@@ -355,29 +317,45 @@ splitOn :: (Monad m, Unbox a) =>
355317 (a -> Bool) -> Array a -> Stream m (Array a)
356318splitOn predicate arr =
357319 fmap (\(i, len) -> getSliceUnsafe i len arr)
358− $ D.sliceOnSuffix predicate (A.toStreamD arr)
320+ $ D.indexOnSuffix predicate (read arr)
321+
322+{-# INLINE sliceIndexerFromLen #-}
323+sliceIndexerFromLen :: forall m a. (Monad m, Unbox a)
324+ => Int -- ^ from index
325+ -> Int -- ^ length of the slice
326+ -> Unfold m (Array a) (Int, Int)
327+sliceIndexerFromLen from len =
328+ Unfold.lmap unsafeThaw (MA.sliceIndexerFromLen from len)
359329
330+{-# DEPRECATED genSlicesFromLen "Please use sliceIndexerFromLen instead." #-}
360331{-# INLINE genSlicesFromLen #-}
361332genSlicesFromLen :: forall m a. (Monad m, Unbox a)
362333 => Int -- ^ from index
363334 -> Int -- ^ length of the slice
364335 -> Unfold m (Array a) (Int, Int)
365−genSlicesFromLen from len =
366− Unfold.lmap A.unsafeThaw (MA.genSlicesFromLen from len)
336+genSlicesFromLen = sliceIndexerFromLen
367337
368338-- | Generate a stream of slices of specified length from an array, starting
369339-- from the supplied array index. The last slice may be shorter than the
370340-- requested length.
371341--
372342-- /Pre-release//
343+{-# INLINE slicerFromLen #-}
344+slicerFromLen :: forall m a. (Monad m, Unbox a)
345+ => Int -- ^ from index
346+ -> Int -- ^ length of the slice
347+ -> Unfold m (Array a) (Array a)
348+slicerFromLen from len =
349+ fmap unsafeFreeze
350+ $ Unfold.lmap unsafeThaw (MA.slicerFromLen from len)
351+
352+{-# DEPRECATED getSlicesFromLen "Please use slicerFromLen instead." #-}
373353{-# INLINE getSlicesFromLen #-}
374354getSlicesFromLen :: forall m a. (Monad m, Unbox a)
375355 => Int -- ^ from index
376356 -> Int -- ^ length of the slice
377357 -> Unfold m (Array a) (Array a)
378−getSlicesFromLen from len =
379− fmap A.unsafeFreeze
380− $ Unfold.lmap A.unsafeThaw (MA.getSlicesFromLen from len)
358+getSlicesFromLen = slicerFromLen
381359
382360-------------------------------------------------------------------------------
383361-- Random reads and writes
@@ -395,7 +373,7 @@ getIndex i arr =
395373 $ do
396374 let elemPtr = INDEX_OF(arrStart arr, i, a)
397375 if i >= 0 && INDEX_VALID(elemPtr, arrEnd arr, a)
398− then Just <$> peekByteIndex elemPtr (arrContents arr)
376+ then Just <$> peekAt elemPtr (arrContents arr)
399377 else return Nothing
400378
401379-- | Given a stream of array indices, read the elements on those indices from
@@ -407,19 +385,25 @@ getIndex i arr =
407385-- @
408386-- read =
409387-- let u = lmap (\arr -> (0, length arr - 1)) Unfold.enumerateFromTo
410−-- in Unfold.lmap f (getIndices arr)
388+-- in Unfold.lmap f (indexReader arr)
411389--
412390-- readRev =
413391-- let i = length arr - 1
414−-- in Unfold.lmap f (getIndicesFromThenTo i (i - 1) 0)
392+-- in Unfold.lmap f (indexReaderFromThenTo i (i - 1) 0)
415393-- @
416394--
417395-- /Pre-release/
396+{-# INLINE indexReader #-}
397+indexReader :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
398+indexReader m =
399+ let unf = MA.indexReaderWith (return . unsafeInlineIO) m
400+ in Unfold.lmap unsafeThaw unf
401+
402+-- XXX DO NOT REMOVE, change the signature to use Stream instead of unfold
403+{-# DEPRECATED getIndices "Please use getIndices instead." #-}
418404{-# INLINE getIndices #-}
419405getIndices :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
420−getIndices m =
421− let unf = MA.getIndicesD (return . unsafeInlineIO) m
422− in Unfold.lmap A.unsafeThaw unf
406+getIndices = indexReader
423407
424408-- | Unfolds @(from, then, to, array)@ generating a finite stream whose first
425409-- element is the array value from the index @from@ and the successive elements
@@ -435,9 +419,9 @@ getIndices m =
435419-- @
436420--
437421-- /Unimplemented/
438−{-# INLINE getIndicesFromThenTo #-}
439−getIndicesFromThenTo :: Unfold m (Int, Int, Int, Array a) a
440−getIndicesFromThenTo = undefined
422+{-# INLINE indexReaderFromThenTo #-}
423+indexReaderFromThenTo :: Unfold m (Int, Int, Int, Array a) a
424+indexReaderFromThenTo = undefined
441425
442426-------------------------------------------------------------------------------
443427-- Transform via stream operations
@@ -454,7 +438,7 @@ getIndicesFromThenTo = undefined
454438{-# INLINE runPipe #-}
455439runPipe :: (MonadIO m, Unbox a, Unbox b)
456440 => Pipe m a b -> Array a -> m (Array b)
457−runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (A.read arr)
441+runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (read arr)
458442-}
459443
460444-- XXX For transformations that cannot change the number of elements e.g. "map"
@@ -468,7 +452,7 @@ runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (A.read arr)
468452streamTransform :: forall m a b. (MonadIO m, Unbox a, Unbox b)
469453 => (Stream m a -> Stream m b) -> Array a -> m (Array b)
470454streamTransform f arr =
471− Stream.fold (A.writeWith (length arr)) $ f (A.read arr)
455+ Stream.fold (createWith (length arr)) $ f (read arr)
472456
473457-------------------------------------------------------------------------------
474458-- Casts
@@ -502,7 +486,7 @@ asBytes = castUnsafe
502486--
503487cast :: forall a b. (Unbox b) => Array a -> Maybe (Array b)
504488cast arr =
505− let len = A.byteLength arr
489+ let len = byteLength arr
506490 r = len `mod` SIZE_OF(b)
507491 in if r /= 0
508492 then Nothing
@@ -520,9 +504,9 @@ cast arr =
520504--
521505asCStringUnsafe :: Array a -> (CString -> IO b) -> IO b
522506asCStringUnsafe arr act = do
523− let arr1 = asBytes arr <> A.fromList [0]
524− -- asPtrUnsafe makes sure the array is pinned
525− asPtrUnsafe arr1 $ \ptr -> act (castPtr ptr)
507+ let arr1 = asBytes arr <> fromList [0]
508+ -- unsafePinnedAsPtr makes sure the array is pinned
509+ unsafePinnedAsPtr arr1 $ \ptr -> act (castPtr ptr)
526510
527511-------------------------------------------------------------------------------
528512-- Folds
@@ -535,11 +519,500 @@ asCStringUnsafe arr act = do
535519-- /Pre-release/
536520{-# INLINE fold #-}
537521fold :: forall m a b. (Monad m, Unbox a) => Fold m a b -> Array a -> m b
538−fold f arr = Stream.fold f (A.read arr)
522+fold f arr = Stream.fold f (read arr)
539523
540524-- | Fold an array using a stream fold operation.
541525--
542526-- /Pre-release/
543527{-# INLINE streamFold #-}
544528streamFold :: (Monad m, Unbox a) => (Stream m a -> m b) -> Array a -> m b
545−streamFold f arr = f (A.read arr)
529+streamFold f arr = f (read arr)
530+
531+--------------------------------------------------------------------------------
532+-- Serialization
533+--------------------------------------------------------------------------------
534+
535+{-# INLINE encodeAs #-}
536+encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
537+encodeAs ps a =
538+ unsafeInlineIO $ do
539+ let len = Serialize.addSizeTo 0 a
540+ mbarr <- MBA.newBytesAs ps len
541+ off <- Serialize.serializeAt 0 mbarr a
542+ assertM(len == off)
543+ pure $ Array mbarr 0 off
544+
545+-- |
546+-- Properties:
547+-- 1. Identity: @deserialize . serialize == id@
548+-- 2. Encoded equivalence: @serialize a == serialize a@
549+{-# INLINE serialize #-}
550+serialize :: Serialize a => a -> Array Word8
551+serialize = encodeAs Unpinned
552+
553+-- | Serialize a Haskell type to a pinned byte array. The array is allocated
554+-- using pinned memory so that it can be used directly in OS APIs for writing
555+-- to file or sending over the network.
556+--
557+-- Properties:
558+-- 1. Identity: @deserialize . pinnedSerialize == id@
559+-- 2. Encoded equivalence: @pinnedSerialize a == pinnedSerialize a@
560+{-# INLINE pinnedSerialize #-}
561+pinnedSerialize :: Serialize a => a -> Array Word8
562+pinnedSerialize = encodeAs Pinned
563+
564+-- | Decode a Haskell type from a byte array containing its serialized
565+-- representation.
566+{-# INLINE deserialize #-}
567+deserialize :: Serialize a => Array Word8 -> a
568+deserialize arr@(Array {..}) = unsafeInlineIO $ do
569+ let lenArr = length arr
570+ (off, val) <-
571+ Serialize.deserializeAt arrStart arrContents (arrStart + lenArr)
572+ assertM(off == arrStart + lenArr)
573+ pure val
574+
575+-------------------------------------------------------------------------------
576+-- Streams of Arrays
577+-------------------------------------------------------------------------------
578+
579+-- TODO: efficiently compare two streams of arrays. Two streams can have chunks
580+-- of different sizes, we can handle that in the stream comparison abstraction.
581+-- This could be useful e.g. to fast compare whether two files differ.
582+
583+-- | Insert the given element between arrays and flatten.
584+--
585+-- >>> interpose x = Stream.interpose x Array.reader
586+--
587+{-# INLINE interpose #-}
588+interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
589+interpose x = D.interpose x reader
590+
591+data FlattenState s =
592+ OuterLoop s
593+ | InnerLoop s !MutByteArray !Int !Int
594+
595+-- | Insert the given element after each array and flatten. This is similar to
596+-- unlines.
597+--
598+-- >>> interposeSuffix x = Stream.interposeSuffix x Array.reader
599+--
600+{-# INLINE_NORMAL interposeSuffix #-}
601+interposeSuffix :: forall m a. (Monad m, Unbox a)
602+ => a -> Stream m (Array a) -> Stream m a
603+-- This does not require MonadIO constraint.
604+-- interposeSuffix x = D.interposeSuffix x reader
605+interposeSuffix sep (D.Stream step state) = D.Stream step' (OuterLoop state)
606+
607+ where
608+
609+ {-# INLINE_LATE step' #-}
610+ step' gst (OuterLoop st) = do
611+ r <- step (adaptState gst) st
612+ return $ case r of
613+ D.Yield Array{..} s ->
614+ D.Skip (InnerLoop s arrContents arrStart arrEnd)
615+ D.Skip s -> D.Skip (OuterLoop s)
616+ D.Stop -> D.Stop
617+
618+ step' _ (InnerLoop st _ p end) | p == end =
619+ return $ D.Yield sep $ OuterLoop st
620+
621+ step' _ (InnerLoop st contents p end) = do
622+ x <- pure $ unsafeInlineIO $ peekAt p contents
623+ return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
624+
625+-- | Insert the given array after each array and flatten.
626+--
627+-- >>> intercalateSuffix = Stream.intercalateSuffix Array.reader
628+--
629+{-# INLINE intercalateSuffix #-}
630+intercalateSuffix :: (Monad m, Unbox a)
631+ => Array a -> Stream m (Array a) -> Stream m a
632+intercalateSuffix = D.intercalateSuffix reader
633+
634+-- | @compactLE n@ coalesces adjacent arrays in the input stream
635+-- only if the combined size would be less than or equal to n.
636+--
637+-- Generates unpinned arrays irrespective of the pinning status of input
638+-- arrays.
639+{-# INLINE_NORMAL compactLE #-}
640+compactLE :: (MonadIO m, Unbox a)
641+ => Int -> Stream m (Array a) -> Stream m (Array a)
642+compactLE n stream =
643+ D.map unsafeFreeze $ MA.compactLE n $ D.map unsafeThaw stream
644+
645+-- | Pinned version of 'compactLE'.
646+{-# INLINE_NORMAL pinnedCompactLE #-}
647+pinnedCompactLE :: (MonadIO m, Unbox a)
648+ => Int -> Stream m (Array a) -> Stream m (Array a)
649+pinnedCompactLE n stream =
650+ D.map unsafeFreeze $ MA.pinnedCompactLE n $ D.map unsafeThaw stream
651+
652+-- | Split a stream of arrays on a given separator byte, dropping the separator
653+-- and coalescing all the arrays between two separators into a single array.
654+--
655+{-# INLINE compactOnByte #-}
656+compactOnByte
657+ :: (MonadIO m)
658+ => Word8
659+ -> Stream m (Array Word8)
660+ -> Stream m (Array Word8)
661+compactOnByte byte =
662+ fmap unsafeFreeze . MA.compactOnByte byte . fmap unsafeThaw
663+
664+-- | Like 'compactOnByte' considers the separator in suffix position instead of
665+-- infix position.
666+{-# INLINE compactOnByteSuffix #-}
667+compactOnByteSuffix
668+ :: (MonadIO m)
669+ => Word8
670+ -> Stream m (Array Word8)
671+ -> Stream m (Array Word8)
672+compactOnByteSuffix byte =
673+ fmap unsafeFreeze . MA.compactOnByteSuffix byte . fmap unsafeThaw
674+
675+-------------------------------------------------------------------------------
676+-- Folding Streams of Arrays
677+-------------------------------------------------------------------------------
678+
679+-- XXX This should not be used for breaking a stream as the D.cons used in
680+-- reconstructing the stream could be very bad for performance. This can only
681+-- be useful in folding without breaking.
682+{-# INLINE_NORMAL foldBreakChunks #-}
683+foldBreakChunks :: forall m a b. (MonadIO m, Unbox a) =>
684+ Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
685+foldBreakChunks (Fold fstep initial _ final) stream@(Stream step state) = do
686+ res <- initial
687+ case res of
688+ Fold.Partial fs -> go SPEC state fs
689+ Fold.Done fb -> return $! (fb, stream)
690+
691+ where
692+
693+ {-# INLINE go #-}
694+ go !_ st !fs = do
695+ r <- step defState st
696+ case r of
697+ Stream.Yield (Array contents start end) s ->
698+ let fp = Tuple' end contents
699+ in goArray SPEC s fp start fs
700+ Stream.Skip s -> go SPEC s fs
701+ Stream.Stop -> do
702+ b <- final fs
703+ return (b, D.nil)
704+
705+ goArray !_ s (Tuple' end _) !cur !fs
706+ | cur == end = do
707+ go SPEC s fs
708+ goArray !_ st fp@(Tuple' end contents) !cur !fs = do
709+ x <- liftIO $ peekAt cur contents
710+ res <- fstep fs x
711+ let next = INDEX_NEXT(cur,a)
712+ case res of
713+ Fold.Done b -> do
714+ let arr = Array contents next end
715+ return $! (b, D.cons arr (D.Stream step st))
716+ Fold.Partial fs1 -> goArray SPEC st fp next fs1
717+
718+-- This may be more robust wrt fusion compared to unfoldMany?
719+
720+-- | Fold a stream of arrays using a 'Fold'. This is equivalent to the
721+-- following:
722+--
723+-- >>> foldChunks f = Stream.fold f . Stream.unfoldMany Array.reader
724+--
725+foldChunks :: (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m b
726+foldChunks f s = fmap fst (foldBreakChunks f s)
727+-- foldStream f = Stream.fold f . Stream.unfoldMany reader
728+
729+-- | Fold a stream of arrays using a 'Fold' and return the remaining stream.
730+--
731+-- The following alternative to this function allows composing the fold using
732+-- the parser Monad:
733+--
734+-- @
735+-- foldBreakStreamK f s =
736+-- fmap (first (fromRight undefined))
737+-- $ StreamK.parseBreakChunks (ParserK.adaptC (Parser.fromFold f)) s
738+-- @
739+--
740+-- We can compare perf and remove this one or define it in terms of that.
741+--
742+foldBreakChunksK :: forall m a b. (MonadIO m, Unbox a) =>
743+ Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
744+{-
745+foldBreakChunksK f s =
746+ fmap (first (fromRight undefined))
747+ $ StreamK.parseBreakChunks (ParserK.adaptC (Parser.fromFold f)) s
748+-}
749+foldBreakChunksK (Fold fstep initial _ final) stream = do
750+ res <- initial
751+ case res of
752+ Fold.Partial fs -> go fs stream
753+ Fold.Done fb -> return (fb, stream)
754+
755+ where
756+
757+ {-# INLINE go #-}
758+ go !fs st = do
759+ let stop = (, StreamK.nil) <$> final fs
760+ single a = yieldk a StreamK.nil
761+ yieldk (Array contents start end) r =
762+ let fp = Tuple' end contents
763+ in goArray fs r fp start
764+ in StreamK.foldStream defState yieldk single stop st
765+
766+ goArray !fs st (Tuple' end _) !cur
767+ | cur == end = do
768+ go fs st
769+ goArray !fs st fp@(Tuple' end contents) !cur = do
770+ x <- liftIO $ peekAt cur contents
771+ res <- fstep fs x
772+ let next = INDEX_NEXT(cur,a)
773+ case res of
774+ Fold.Done b -> do
775+ let arr = Array contents next end
776+ return $! (b, StreamK.cons arr st)
777+ Fold.Partial fs1 -> goArray fs1 st fp next
778+
779+{-
780+-- This can be generalized to any type provided it can be unfolded to a stream
781+-- and it can be combined using a semigroup operation.
782+--
783+{-# INLINE_NORMAL parseBreakD #-}
784+parseBreakD ::
785+ forall m a b. (MonadIO m, MonadThrow m, Unbox a)
786+ => PRD.Parser a m b
787+ -> D.Stream m (Array.Array a)
788+ -> m (b, D.Stream m (Array.Array a))
789+parseBreakD
790+ (PRD.Parser pstep initial extract) stream@(D.Stream step state) = do
791+
792+ res <- initial
793+ case res of
794+ PRD.IPartial s -> go SPEC state (List []) s
795+ PRD.IDone b -> return (b, stream)
796+ PRD.IError err -> throwM $ ParseError err
797+
798+ where
799+
800+ -- "backBuf" contains last few items in the stream that we may have to
801+ -- backtrack to.
802+ --
803+ -- XXX currently we are using a dumb list based approach for backtracking
804+ -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
805+ -- That will allow us more efficient random back and forth movement.
806+ go !_ st backBuf !pst = do
807+ r <- step defState st
808+ case r of
809+ D.Yield (Array contents start end) s ->
810+ gobuf SPEC s backBuf
811+ (Tuple' end contents) start pst
812+ D.Skip s -> go SPEC s backBuf pst
813+ D.Stop -> do
814+ b <- extract pst
815+ return (b, D.nil)
816+
817+ -- Use strictness on "cur" to keep it unboxed
818+ gobuf !_ s backBuf (Tuple' end _) !cur !pst
819+ | cur == end = do
820+ go SPEC s backBuf pst
821+ gobuf !_ s backBuf fp@(Tuple' end contents) !cur !pst = do
822+ x <- liftIO $ peekByteIndex contents cur
823+ pRes <- pstep pst x
824+ let next = INDEX_NEXT(cur,a)
825+ case pRes of
826+ PR.Partial 0 pst1 ->
827+ gobuf SPEC s (List []) fp next pst1
828+ PR.Partial n pst1 -> do
829+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
830+ let src0 = Prelude.take n (x:getList backBuf)
831+ arr0 = A.fromListN n (Prelude.reverse src0)
832+ arr1 = Array contents next end
833+ src = arr0 <> arr1
834+ let !(Array cont1 start end1) = src
835+ fp1 = Tuple' end1 cont1
836+ gobuf SPEC s (List []) fp1 start pst1
837+ PR.Continue 0 pst1 ->
838+ gobuf SPEC s (List (x:getList backBuf)) fp next pst1
839+ PR.Continue n pst1 -> do
840+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
841+ let (src0, buf1) = splitAt n (x:getList backBuf)
842+ arr0 = A.fromListN n (Prelude.reverse src0)
843+ arr1 = Array contents next end
844+ src = arr0 <> arr1
845+ let !(Array cont1 start end1) = src
846+ fp1 = Tuple' end1 cont1
847+ gobuf SPEC s (List buf1) fp1 start pst1
848+ PR.Done 0 b -> do
849+ let arr = Array contents next end
850+ return (b, D.cons arr (D.Stream step s))
851+ PR.Done n b -> do
852+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
853+ let src0 = Prelude.take n (x:getList backBuf)
854+ -- XXX create the array in reverse instead
855+ arr0 = A.fromListN n (Prelude.reverse src0)
856+ arr1 = Array contents next end
857+ -- XXX Use StreamK to avoid adding arbitrary layers of
858+ -- constructors every time.
859+ str = D.cons arr0 (D.cons arr1 (D.Stream step s))
860+ return (b, str)
861+ PR.Error err -> throwM $ ParseError err
862+-}
863+
864+-- | Parse an array stream using the supplied 'Parser'. Returns the parse
865+-- result and the unconsumed stream. Throws 'ParseError' if the parse fails.
866+--
867+-- The following alternative to this function allows composing the parser using
868+-- the parser Monad:
869+--
870+-- >>> parseBreakStreamK p = StreamK.parseBreakChunks (ParserK.adaptC p)
871+--
872+-- We can compare perf and remove this one or define it in terms of that.
873+--
874+-- /Internal/
875+--
876+{-# INLINE_NORMAL parseBreakChunksK #-}
877+parseBreakChunksK ::
878+ forall m a b. (MonadIO m, Unbox a)
879+ => Parser a m b
880+ -> StreamK m (Array a)
881+ -> m (Either ParseError b, StreamK m (Array a))
882+-- parseBreakStreamK p = StreamK.parseBreakChunks (ParserK.adaptC p)
883+parseBreakChunksK (Parser pstep initial extract) stream = do
884+ res <- initial
885+ case res of
886+ IPartial s -> go s stream []
887+ IDone b -> return (Right b, stream)
888+ IError err -> return (Left (ParseError err), stream)
889+
890+ where
891+
892+ -- "backBuf" contains last few items in the stream that we may have to
893+ -- backtrack to.
894+ --
895+ -- XXX currently we are using a dumb list based approach for backtracking
896+ -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
897+ -- That will allow us more efficient random back and forth movement.
898+ go !pst st backBuf = do
899+ let stop = goStop pst backBuf -- (, K.nil) <$> extract pst
900+ single a = yieldk a StreamK.nil
901+ yieldk arr r = goArray pst backBuf r arr
902+ in StreamK.foldStream defState yieldk single stop st
903+
904+ -- Use strictness on "cur" to keep it unboxed
905+ goArray !pst backBuf st (Array _ cur end) | cur == end = go pst st backBuf
906+ goArray !pst backBuf st (Array contents cur end) = do
907+ x <- liftIO $ peekAt cur contents
908+ pRes <- pstep pst x
909+ let next = INDEX_NEXT(cur,a)
910+ case pRes of
911+ Parser.Partial 0 s ->
912+ goArray s [] st (Array contents next end)
913+ Parser.Partial n s -> do
914+ assert (n <= Prelude.length (x:backBuf)) (return ())
915+ let src0 = Prelude.take n (x:backBuf)
916+ arr0 = fromListN n (Prelude.reverse src0)
917+ arr1 = Array contents next end
918+ src = arr0 <> arr1
919+ goArray s [] st src
920+ Parser.Continue 0 s ->
921+ goArray s (x:backBuf) st (Array contents next end)
922+ Parser.Continue n s -> do
923+ assert (n <= Prelude.length (x:backBuf)) (return ())
924+ let (src0, buf1) = Prelude.splitAt n (x:backBuf)
925+ arr0 = fromListN n (Prelude.reverse src0)
926+ arr1 = Array contents next end
927+ src = arr0 <> arr1
928+ goArray s buf1 st src
929+ Parser.Done 0 b -> do
930+ let arr = Array contents next end
931+ return (Right b, StreamK.cons arr st)
932+ Parser.Done n b -> do
933+ assert (n <= Prelude.length (x:backBuf)) (return ())
934+ let src0 = Prelude.take n (x:backBuf)
935+ -- XXX Use fromListRevN once implemented
936+ -- arr0 = A.fromListRevN n src0
937+ arr0 = fromListN n (Prelude.reverse src0)
938+ arr1 = Array contents next end
939+ str = StreamK.cons arr0 (StreamK.cons arr1 st)
940+ return (Right b, str)
941+ Parser.Error err -> do
942+ let n = Prelude.length backBuf
943+ arr0 = fromListN n (Prelude.reverse backBuf)
944+ arr1 = Array contents cur end
945+ str = StreamK.cons arr0 (StreamK.cons arr1 stream)
946+ return (Left (ParseError err), str)
947+
948+ -- This is a simplified goArray
949+ goExtract !pst backBuf (Array _ cur end)
950+ | cur == end = goStop pst backBuf
951+ goExtract !pst backBuf (Array contents cur end) = do
952+ x <- liftIO $ peekAt cur contents
953+ pRes <- pstep pst x
954+ let next = INDEX_NEXT(cur,a)
955+ case pRes of
956+ Parser.Partial 0 s ->
957+ goExtract s [] (Array contents next end)
958+ Parser.Partial n s -> do
959+ assert (n <= Prelude.length (x:backBuf)) (return ())
960+ let src0 = Prelude.take n (x:backBuf)
961+ arr0 = fromListN n (Prelude.reverse src0)
962+ arr1 = Array contents next end
963+ src = arr0 <> arr1
964+ goExtract s [] src
965+ Parser.Continue 0 s ->
966+ goExtract s backBuf (Array contents next end)
967+ Parser.Continue n s -> do
968+ assert (n <= Prelude.length (x:backBuf)) (return ())
969+ let (src0, buf1) = Prelude.splitAt n (x:backBuf)
970+ arr0 = fromListN n (Prelude.reverse src0)
971+ arr1 = Array contents next end
972+ src = arr0 <> arr1
973+ goExtract s buf1 src
974+ Parser.Done 0 b -> do
975+ let arr = Array contents next end
976+ return (Right b, StreamK.fromPure arr)
977+ Parser.Done n b -> do
978+ assert (n <= Prelude.length backBuf) (return ())
979+ let src0 = Prelude.take n backBuf
980+ -- XXX Use fromListRevN once implemented
981+ -- arr0 = A.fromListRevN n src0
982+ arr0 = fromListN n (Prelude.reverse src0)
983+ arr1 = Array contents next end
984+ str = StreamK.cons arr0 (StreamK.fromPure arr1)
985+ return (Right b, str)
986+ Parser.Error err -> do
987+ let n = Prelude.length backBuf
988+ arr0 = fromListN n (Prelude.reverse backBuf)
989+ arr1 = Array contents cur end
990+ str = StreamK.cons arr0 (StreamK.cons arr1 stream)
991+ return (Left (ParseError err), str)
992+
993+ -- This is a simplified goExtract
994+ {-# INLINE goStop #-}
995+ goStop !pst backBuf = do
996+ pRes <- extract pst
997+ case pRes of
998+ Parser.Partial _ _ -> error "Bug: parseBreak: Partial in extract"
999+ Parser.Continue 0 s ->
1000+ goStop s backBuf
1001+ Parser.Continue n s -> do
1002+ assert (n <= Prelude.length backBuf) (return ())
1003+ let (src0, buf1) = Prelude.splitAt n backBuf
1004+ arr = fromListN n (Prelude.reverse src0)
1005+ goExtract s buf1 arr
1006+ Parser.Done 0 b ->
1007+ return (Right b, StreamK.nil)
1008+ Parser.Done n b -> do
1009+ assert (n <= Prelude.length backBuf) (return ())
1010+ let src0 = Prelude.take n backBuf
1011+ -- XXX Use fromListRevN once implemented
1012+ -- arr0 = A.fromListRevN n src0
1013+ arr0 = fromListN n (Prelude.reverse src0)
1014+ return (Right b, StreamK.fromPure arr0)
1015+ Parser.Error err -> do
1016+ let n = Prelude.length backBuf
1017+ arr0 = fromListN n (Prelude.reverse backBuf)
1018+ return (Left (ParseError err), StreamK.fromPure arr0)
core/src/Streamly/Internal/Data/Array/Generic.hs+22−6
@@ -11,8 +11,8 @@ module Streamly.Internal.Data.Array.Generic
1111
1212 -- * Construction
1313 , nil
14− , writeN
15− , write
14+ , createOf
15+ , create
1616 , writeWith
1717 , writeLastN
1818
@@ -44,6 +44,10 @@ module Streamly.Internal.Data.Array.Generic
4444 , getIndex
4545 , getSliceUnsafe
4646 , strip
47+
48+ -- * Deprecated
49+ , writeN
50+ , write
4751 )
4852where
4953
@@ -104,21 +108,33 @@ nil = unsafePerformIO $ unsafeFreeze <$> MArray.nil
104108-- Construction - Folds
105109-------------------------------------------------------------------------------
106110
107−{-# INLINE_NORMAL writeN #-}
111+{-# INLINE_NORMAL createOf #-}
112+createOf :: MonadIO m => Int -> Fold m a (Array a)
113+createOf = fmap unsafeFreeze <$> MArray.createOf
114+
115+-- XXX Deprecate in major
116+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
117+{-# INLINE writeN #-}
108118writeN :: MonadIO m => Int -> Fold m a (Array a)
109−writeN = fmap unsafeFreeze <$> MArray.writeN
119+writeN = createOf
110120
111121{-# INLINE_NORMAL writeWith #-}
112122writeWith :: MonadIO m => Int -> Fold m a (Array a)
113−writeWith elemCount = unsafeFreeze <$> MArray.writeWith elemCount
123+writeWith elemCount = unsafeFreeze <$> MArray.createWith elemCount
114124
115125-- | Fold the whole input to a single array.
116126--
117127-- /Caution! Do not use this on infinite streams./
118128--
129+{-# INLINE create #-}
130+create :: MonadIO m => Fold m a (Array a)
131+create = fmap unsafeFreeze MArray.create
132+
133+-- XXX Deprecate in major
134+-- {-# DEPRECATED write "Please use create instead." #-}
119135{-# INLINE write #-}
120136write :: MonadIO m => Fold m a (Array a)
121−write = fmap unsafeFreeze MArray.write
137+write = create
122138
123139fromPureStream :: Stream Identity a -> Array a
124140fromPureStream x =
core/src/Streamly/Internal/Data/Stream/Chunked.hs → core/src/Streamly/Internal/Data/Array/Stream.hsrenamed+74−586
@@ -1,5 +1,6 @@
1+{-# OPTIONS_GHC -Wno-deprecations #-}
12-- |
2−-- Module : Streamly.Internal.Data.Stream.Chunked
3+-- Module : Streamly.Internal.Data.Array.Stream
34-- Copyright : (c) 2019 Composewell Technologies
45-- License : BSD3-3-Clause
56-- Maintainer : streamly@composewell.com
@@ -7,17 +8,25 @@
78--
89-- Combinators to efficiently manipulate streams of immutable arrays.
910--
10−module Streamly.Internal.Data.Stream.Chunked
11+-- We can either push these in the MutArray module with a "chunks" prefix or
12+-- keep this as a separate module and release it.
13+--
14+module Streamly.Internal.Data.Array.Stream
15+{-# DEPRECATED "Please use \"Streamly.Internal.Data.Array\" instead." #-}
1116 (
1217 -- * Creation
13− chunksOf
18+ Array.chunksOf
19+ , Array.pinnedChunksOf
20+ , Array.bufferChunks
1421
1522 -- * Flattening to elements
16− , concat
17− , concatRev
18− , interpose
19− , interposeSuffix
20− , intercalateSuffix
23+ , Array.concat
24+ , Array.flattenArrays
25+ , Array.concatRev
26+ , Array.flattenArraysRev
27+ , Array.interpose
28+ , Array.interposeSuffix
29+ , Array.intercalateSuffix
2130 , unlines
2231
2332 -- * Elimination
@@ -26,24 +35,27 @@ module Streamly.Internal.Data.Stream.Chunked
2635 -- can flatten the stream to byte stream and use that. But if we want the
2736 -- remaining stream to be a chunk stream then this could be handy. But it
2837 -- could also be implemented using parseBreak.
29− , foldBreak -- StreamK.foldBreakChunks
38+ , foldBreak
3039 , foldBreakD
31− -- The byte level parseBreak cannot work efficiently. Because the stream
32− -- will have to be a StreamK for backtracking, StreamK at byte level would
33− -- not be efficient.
34− , parseBreak -- StreamK.parseBreakChunks
40+ -- This is chunked parseBreak. A byte level parseBreak cannot work
41+ -- efficiently. Because the stream will have to be a StreamK for
42+ -- backtracking, StreamK at byte level would not be efficient.
43+ -- parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
44+ , parseBreak
3545 -- , parseBreakD
3646 -- , foldManyChunks
3747 -- , parseManyChunks
48+ , K.parseBreakChunks
49+ , K.parseChunks
3850
3951 -- ** Array Folds
40− -- XXX Use Parser.Chunked instead, need only chunkedParseBreak,
52+ -- XXX Use parseBreakChunks/parseChunks instead
4153 -- foldBreak can be implemented using parseBreak. Use StreamK.
4254 , runArrayFold
4355 , runArrayFoldBreak
4456 -- , parseArr
45− , runArrayParserDBreak -- StreamK.chunkedParseBreak
46− , runArrayFoldMany -- StreamK.chunkedParseMany
57+ , runArrayParserDBreak -- StreamK.parseBreakChunks
58+ , runArrayFoldMany
4759
4860 , toArray
4961
@@ -67,6 +79,8 @@ where
6779import Data.Bifunctor (second)
6880import Control.Exception (assert)
6981import Control.Monad.IO.Class (MonadIO(..))
82+-- import Data.Bifunctor (first)
83+-- import Data.Either (fromRight)
7084import Data.Proxy (Proxy(..))
7185import Data.Word (Word8)
7286import Streamly.Internal.Data.Unbox (Unbox(..))
@@ -82,142 +96,32 @@ import Streamly.Internal.Data.Parser (ParseError(..))
8296import Streamly.Internal.Data.Stream (Stream)
8397import Streamly.Internal.Data.StreamK (StreamK, fromStream, toStream)
8498import Streamly.Internal.Data.SVar.Type (adaptState, defState)
85−import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
8699
87−import qualified Streamly.Data.Fold as FL
88100import qualified Streamly.Internal.Data.Array as A
89101import qualified Streamly.Internal.Data.Array as Array
90−import qualified Streamly.Internal.Data.MutArray.Type as MA
91−import qualified Streamly.Internal.Data.Stream.MutChunked as AS
92−import qualified Streamly.Internal.Data.Fold.Type as FL (Fold(..), Step(..))
93102import qualified Streamly.Internal.Data.Parser as PR
94103import qualified Streamly.Internal.Data.Parser as PRD
95104 (Parser(..), Initial(..))
105+-- import qualified Streamly.Internal.Data.ParserK as ParserK
96106import qualified Streamly.Internal.Data.Stream as D
97107import qualified Streamly.Internal.Data.StreamK as K
98108
99109-- XXX Since these are immutable arrays MonadIO constraint can be removed from
100110-- most places.
101111
102−-------------------------------------------------------------------------------
103−-- Generation
104−-------------------------------------------------------------------------------
105−
106−-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
107−-- @n@ elements each.
108−--
109−-- > chunksOf n = Stream.groupsOf n (Array.writeN n)
110−--
111−-- /Pre-release/
112−{-# INLINE chunksOf #-}
113−chunksOf :: (MonadIO m, Unbox a)
114− => Int -> Stream m a -> Stream m (Array a)
115−chunksOf = A.chunksOf
116−
117−-------------------------------------------------------------------------------
118−-- Append
119−-------------------------------------------------------------------------------
120−
121−-- XXX efficiently compare two streams of arrays. Two streams can have chunks
122−-- of different sizes, we can handle that in the stream comparison abstraction.
123−-- This could be useful e.g. to fast compare whether two files differ.
124−
125−-- | Convert a stream of arrays into a stream of their elements.
126−--
127−-- Same as the following:
128−--
129−-- > concat = Stream.unfoldMany Array.read
130−--
131−-- @since 0.7.0
132−{-# INLINE concat #-}
133−concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
134−-- concat m = fromStreamD $ A.flattenArrays (toStreamD m)
135−-- concat m = fromStreamD $ D.concatMap A.toStreamD (toStreamD m)
136−concat = D.unfoldMany A.reader
137−
138−-- | Convert a stream of arrays into a stream of their elements reversing the
139−-- contents of each array before flattening.
140−--
141−-- > concatRev = Stream.unfoldMany Array.readerRev
142−--
143−-- @since 0.7.0
144−{-# INLINE concatRev #-}
145−concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
146−-- concatRev m = fromStreamD $ A.flattenArraysRev (toStreamD m)
147−concatRev = D.unfoldMany A.readerRev
148−
149112-------------------------------------------------------------------------------
150113-- Intersperse and append
151114-------------------------------------------------------------------------------
152115
153−-- | Flatten a stream of arrays after inserting the given element between
154−-- arrays.
155−--
156−-- /Pre-release/
157−{-# INLINE interpose #-}
158−interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
159−interpose x = D.interpose x A.reader
160−
161−{-# INLINE intercalateSuffix #-}
162−intercalateSuffix :: (Monad m, Unbox a)
163− => Array a -> Stream m (Array a) -> Stream m a
164−intercalateSuffix = D.intercalateSuffix A.reader
165−
166−-- | Flatten a stream of arrays appending the given element after each
167−-- array.
168−--
169−-- @since 0.7.0
170−{-# INLINE interposeSuffix #-}
171−interposeSuffix :: (Monad m, Unbox a)
172− => a -> Stream m (Array a) -> Stream m a
173−-- interposeSuffix x = fromStreamD . A.unlines x . toStreamD
174−interposeSuffix x = D.interposeSuffix x A.reader
175−
176−data FlattenState s =
177− OuterLoop s
178− | InnerLoop s !MA.MutableByteArray !Int !Int
179−
180−-- XXX This is a special case of interposeSuffix, can be removed.
181−-- XXX Remove monadIO constraint
182116{-# INLINE_NORMAL unlines #-}
183117unlines :: forall m a. (MonadIO m, Unbox a)
184118 => a -> D.Stream m (Array a) -> D.Stream m a
185−unlines sep (D.Stream step state) = D.Stream step' (OuterLoop state)
186− where
187− {-# INLINE_LATE step' #-}
188− step' gst (OuterLoop st) = do
189− r <- step (adaptState gst) st
190− return $ case r of
191− D.Yield Array{..} s ->
192− D.Skip (InnerLoop s arrContents arrStart arrEnd)
193− D.Skip s -> D.Skip (OuterLoop s)
194− D.Stop -> D.Stop
195−
196− step' _ (InnerLoop st _ p end) | p == end =
197− return $ D.Yield sep $ OuterLoop st
198−
199− step' _ (InnerLoop st contents p end) = do
200− x <- liftIO $ peekByteIndex p contents
201− return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
119+unlines = Array.interposeSuffix
202120
203121-------------------------------------------------------------------------------
204122-- Compact
205123-------------------------------------------------------------------------------
206124
207−-- XXX These would not be needed once we implement compactLEFold, see
208−-- module Streamly.Internal.Data.Stream.MutChunked
209−--
210−-- XXX Note that this thaws immutable arrays for appending, that may be
211−-- problematic if multiple users do the same thing, however, immutable arrays
212−-- would usually have no capacity to append, therefore, a copy will be forced
213−-- anyway. Confirm this. We can forcefully trim the array capacity before thaw
214−-- to ensure this.
215−{-# INLINE_NORMAL packArraysChunksOf #-}
216−packArraysChunksOf :: (MonadIO m, Unbox a)
217− => Int -> D.Stream m (Array a) -> D.Stream m (Array a)
218−packArraysChunksOf n str =
219− D.map A.unsafeFreeze $ AS.packArraysChunksOf n $ D.map A.unsafeThaw str
220−
221125-- XXX instead of writing two different versions of this operation, we should
222126-- write it as a pipe.
223127--
@@ -225,8 +129,7 @@ packArraysChunksOf n str =
225129{-# INLINE_NORMAL lpackArraysChunksOf #-}
226130lpackArraysChunksOf :: (MonadIO m, Unbox a)
227131 => Int -> Fold m (Array a) () -> Fold m (Array a) ()
228−lpackArraysChunksOf n fld =
229− FL.lmap A.unsafeThaw $ AS.lpackArraysChunksOf n (FL.lmap A.unsafeFreeze fld)
132+lpackArraysChunksOf = Array.lCompactGE
230133
231134-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
232135-- maximum specified size in bytes.
@@ -235,68 +138,19 @@ lpackArraysChunksOf n fld =
235138{-# INLINE compact #-}
236139compact :: (MonadIO m, Unbox a)
237140 => Int -> Stream m (Array a) -> Stream m (Array a)
238−compact = packArraysChunksOf
141+compact = Array.compactLE
239142
240−-------------------------------------------------------------------------------
241−-- Split
242−-------------------------------------------------------------------------------
243−
244−data SplitState s arr
245− = Initial s
246− | Buffering s arr
247− | Splitting s arr
248− | Yielding arr (SplitState s arr)
249− | Finishing
250−
251−-- | Split a stream of arrays on a given separator byte, dropping the separator
252−-- and coalescing all the arrays between two separators into a single array.
143+-- | Given a stream of arrays, splice them all together to generate a single
144+-- array. The stream must be /finite/.
253145--
254146-- @since 0.7.0
255−{-# INLINE_NORMAL _splitOn #-}
256−_splitOn
257− :: MonadIO m
258− => Word8
259− -> D.Stream m (Array Word8)
260− -> D.Stream m (Array Word8)
261−_splitOn byte (D.Stream step state) = D.Stream step' (Initial state)
262−
263− where
264−
265− {-# INLINE_LATE step' #-}
266− step' gst (Initial st) = do
267− r <- step gst st
268− case r of
269− D.Yield arr s -> do
270− (arr1, marr2) <- A.breakOn byte arr
271− return $ case marr2 of
272− Nothing -> D.Skip (Buffering s arr1)
273− Just arr2 -> D.Skip (Yielding arr1 (Splitting s arr2))
274− D.Skip s -> return $ D.Skip (Initial s)
275− D.Stop -> return D.Stop
276−
277− step' gst (Buffering st buf) = do
278− r <- step gst st
279− case r of
280− D.Yield arr s -> do
281− (arr1, marr2) <- A.breakOn byte arr
282− buf' <- A.splice buf arr1
283− return $ case marr2 of
284− Nothing -> D.Skip (Buffering s buf')
285− Just x -> D.Skip (Yielding buf' (Splitting s x))
286− D.Skip s -> return $ D.Skip (Buffering s buf)
287− D.Stop -> return $
288− if A.byteLength buf == 0
289− then D.Stop
290− else D.Skip (Yielding buf Finishing)
291−
292− step' _ (Splitting st buf) = do
293− (arr1, marr2) <- A.breakOn byte buf
294− return $ case marr2 of
295− Nothing -> D.Skip $ Buffering st arr1
296− Just arr2 -> D.Skip $ Yielding arr1 (Splitting st arr2)
147+{-# INLINE toArray #-}
148+toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
149+toArray = Array.fromChunks
297150
298− step' _ (Yielding arr next) = return $ D.Yield arr next
299− step' _ Finishing = return D.Stop
151+-------------------------------------------------------------------------------
152+-- Split
153+-------------------------------------------------------------------------------
300154
301155-- XXX Remove MonadIO constraint.
302156-- | Split a stream of arrays on a given separator byte, dropping the separator
@@ -309,7 +163,7 @@ splitOn
309163 => Word8
310164 -> Stream m (Array Word8)
311165 -> Stream m (Array Word8)
312−splitOn byte = D.splitInnerBy (A.breakOn byte) A.splice
166+splitOn = Array.compactOnByte
313167
314168{-# INLINE splitOnSuffix #-}
315169splitOnSuffix
@@ -317,8 +171,7 @@ splitOnSuffix
317171 => Word8
318172 -> Stream m (Array Word8)
319173 -> Stream m (Array Word8)
320−-- splitOn byte s = fromStreamD $ A.splitOn byte $ toStreamD s
321−splitOnSuffix byte = D.splitInnerBySuffix (A.breakOn byte) A.splice
174+splitOnSuffix = Array.compactOnByteSuffix
322175
323176-------------------------------------------------------------------------------
324177-- Elimination - Running folds
@@ -327,77 +180,24 @@ splitOnSuffix byte = D.splitInnerBySuffix (A.breakOn byte) A.splice
327180{-# INLINE_NORMAL foldBreakD #-}
328181foldBreakD :: forall m a b. (MonadIO m, Unbox a) =>
329182 Fold m a b -> D.Stream m (Array a) -> m (b, D.Stream m (Array a))
330−foldBreakD (FL.Fold fstep initial extract) stream@(D.Stream step state) = do
331− res <- initial
332− case res of
333− FL.Partial fs -> go SPEC state fs
334− FL.Done fb -> return $! (fb, stream)
335−
336− where
337−
338− {-# INLINE go #-}
339− go !_ st !fs = do
340− r <- step defState st
341− case r of
342− D.Yield (Array contents start end) s ->
343− let fp = Tuple' end contents
344− in goArray SPEC s fp start fs
345− D.Skip s -> go SPEC s fs
346− D.Stop -> do
347− b <- extract fs
348− return (b, D.nil)
349−
350− goArray !_ s (Tuple' end _) !cur !fs
351− | cur == end = do
352− go SPEC s fs
353− goArray !_ st fp@(Tuple' end contents) !cur !fs = do
354− x <- liftIO $ peekByteIndex cur contents
355− res <- fstep fs x
356− let next = INDEX_NEXT(cur,a)
357− case res of
358− FL.Done b -> do
359− let arr = Array contents next end
360− return $! (b, D.cons arr (D.Stream step st))
361− FL.Partial fs1 -> goArray SPEC st fp next fs1
362−
363−{-# INLINE_NORMAL foldBreakK #-}
364−foldBreakK :: forall m a b. (MonadIO m, Unbox a) =>
365− Fold m a b -> K.StreamK m (Array a) -> m (b, K.StreamK m (Array a))
366−foldBreakK (FL.Fold fstep initial extract) stream = do
367− res <- initial
368− case res of
369− FL.Partial fs -> go fs stream
370− FL.Done fb -> return (fb, stream)
371−
372− where
373−
374− {-# INLINE go #-}
375− go !fs st = do
376− let stop = (, K.nil) <$> extract fs
377− single a = yieldk a K.nil
378− yieldk (Array contents start end) r =
379− let fp = Tuple' end contents
380− in goArray fs r fp start
381− in K.foldStream defState yieldk single stop st
382−
383− goArray !fs st (Tuple' end _) !cur
384− | cur == end = do
385− go fs st
386− goArray !fs st fp@(Tuple' end contents) !cur = do
387− x <- liftIO $ peekByteIndex cur contents
388− res <- fstep fs x
389− let next = INDEX_NEXT(cur,a)
390− case res of
391− FL.Done b -> do
392− let arr = Array contents next end
393− return $! (b, K.cons arr st)
394− FL.Partial fs1 -> goArray fs1 st fp next
183+foldBreakD = Array.foldBreakChunks
395184
396185-- | Fold an array stream using the supplied 'Fold'. Returns the fold result
397186-- and the unconsumed stream.
398187--
399188-- > foldBreak f = runArrayFoldBreak (ChunkFold.fromFold f)
400189--
190+-- Instead of using this we can adapt the fold to ParserK and use
191+-- parseBreakChunks instead. ParserK allows composing using Monad as well.
192+--
193+-- @
194+-- foldBreak f s =
195+-- fmap (first (fromRight undefined))
196+-- $ K.parseBreakChunks (ParserK.adaptC (PR.fromFold f)) s
197+-- @
198+--
199+-- We can compare perf and remove this one or define it in terms of that.
200+--
401201-- /Internal/
402202--
403203{-# INLINE_NORMAL foldBreak #-}
@@ -406,14 +206,20 @@ foldBreak ::
406206 => Fold m a b
407207 -> StreamK m (A.Array a)
408208 -> m (b, StreamK m (A.Array a))
209+foldBreak = Array.foldBreakChunksK
210+--
409211-- foldBreak f s = fmap fromStreamD <$> foldBreakD f (toStreamD s)
410−foldBreak = foldBreakK
212+--
213+-- foldBreak f s =
214+-- fmap (first (fromRight undefined))
215+-- $ K.parseBreakChunks (ParserK.adaptC (PR.fromFold f)) s
216+--
411217-- If foldBreak performs better than runArrayFoldBreak we can use a rewrite
412218-- rule to rewrite runArrayFoldBreak to fold.
413219-- foldBreak f = runArrayFoldBreak (ChunkFold.fromFold f)
414220
415221-------------------------------------------------------------------------------
416−-- Fold to a single Array
222+-- Elimination - running element parsers
417223-------------------------------------------------------------------------------
418224
419225-- When we have to take an array partially, take the last part of the array.
@@ -459,341 +265,22 @@ splitAtArrayListRev n ls
459265 arr1 = Array contents end1 end
460266 in ([arr1], arr2:xs)
461267
462−-------------------------------------------------------------------------------
463−-- Fold to a single Array
464−-------------------------------------------------------------------------------
465−
466−{-
467−
468−-- XXX Both of these implementations of splicing seem to perform equally well.
469−-- We need to perform benchmarks over a range of sizes though.
470−
471−-- CAUTION! length must more than equal to lengths of all the arrays in the
472−-- stream.
473−{-# INLINE spliceArraysLenUnsafe #-}
474−spliceArraysLenUnsafe :: (MonadIO m, Unbox a)
475− => Int -> Stream m (MutArray a) -> m (MutArray a)
476−spliceArraysLenUnsafe len buffered = do
477− -- XXX The new array's pinned state should depend on the first element
478− -- of the stream. We should uncons the stream and do the required.
479− arr <- undefined
480− D.foldlM' MA.spliceUnsafe (return arr) buffered
481−
482−{-# INLINE _spliceArrays #-}
483−_spliceArrays :: (MonadIO m, Unbox a)
484− => Stream m (Array a) -> m (Array a)
485−_spliceArrays s = do
486− buffered <- D.foldr K.cons K.nil s
487− len <- K.fold FL.sum (fmap Array.length buffered)
488− -- XXX The new array's pinned state should depend on the first element
489− -- of the stream. We should uncons the stream and do the required.
490− arr <- undefined
491− final <- D.foldlM' writeArr (return arr) (toStream buffered)
492− return $ A.unsafeFreeze final
493−
494− where
495−
496− writeArr dst arr = MA.spliceUnsafe dst (A.unsafeThaw arr)
497−
498−{-# INLINE _spliceArraysBuffered #-}
499−_spliceArraysBuffered :: (MonadIO m, Unbox a)
500− => Stream m (Array a) -> m (Array a)
501−_spliceArraysBuffered s = do
502− buffered <- D.foldr K.cons K.nil s
503− len <- K.fold FL.sum (fmap Array.length buffered)
504− A.unsafeFreeze <$>
505− spliceArraysLenUnsafe len (fmap A.unsafeThaw (toStream buffered))
506−
507−-}
508−
509−{-# INLINE spliceArraysRealloced #-}
510−spliceArraysRealloced :: forall m a. (MonadIO m, Unbox a)
511− => Stream m (Array a) -> m (Array a)
512−spliceArraysRealloced s = do
513− res <- D.uncons s
514− case res of
515− Just (a, strm) -> do
516− arr <-
517− D.foldlM'
518− MA.spliceExp
519− (pure (A.unsafeThaw a))
520− (fmap A.unsafeThaw strm)
521− liftIO $ A.unsafeFreeze <$> MA.rightSize arr
522− Nothing -> pure A.nil
523−
524−-- XXX This should just be "fold A.write"
525−--
526−-- | Given a stream of arrays, splice them all together to generate a single
527−-- array. The stream must be /finite/.
528−--
529−-- @since 0.7.0
530−{-# INLINE toArray #-}
531−toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
532−toArray = spliceArraysRealloced
533−-- spliceArrays = _spliceArraysBuffered
534−
535−-- exponentially increasing sizes of the chunks upto the max limit.
536−-- XXX this will be easier to implement with parsers/terminating folds
537−-- With this we should be able to reduce the number of chunks/allocations.
538−-- The reallocation/copy based toArray can also be implemented using this.
539−--
540−{-
541−{-# INLINE toArraysInRange #-}
542−toArraysInRange :: (MonadIO m, Unbox a)
543− => Int -> Int -> Fold m (Array a) b -> Fold m a b
544−toArraysInRange low high (Fold step initial extract) =
545−-}
546−
547−{-
548−-- | Fold the input to a pure buffered stream (List) of arrays.
549−{-# INLINE _toArraysOf #-}
550−_toArraysOf :: (MonadIO m, Unbox a)
551− => Int -> Fold m a (Stream Identity (Array a))
552−_toArraysOf n = FL.groupsOf n (A.writeNF n) FL.toStream
553−-}
554−
555−-------------------------------------------------------------------------------
556−-- Elimination - running element parsers
557−-------------------------------------------------------------------------------
558−
559268-- GHC parser does not accept {-# ANN type [] NoSpecConstr #-}, so we need
560269-- to make a newtype.
561270{-# ANN type List NoSpecConstr #-}
562271newtype List a = List {getList :: [a]}
563272
564−{-
565−-- This can be generalized to any type provided it can be unfolded to a stream
566−-- and it can be combined using a semigroup operation.
567−--
568−-- XXX This should be written using CPS (as parseK) if we want it to scale wrt
569−-- to the number of times it can be called on the same stream.
570−{-# INLINE_NORMAL parseBreakD #-}
571−parseBreakD ::
572− forall m a b. (MonadIO m, MonadThrow m, Unbox a)
573− => PRD.Parser a m b
574− -> D.Stream m (Array.Array a)
575− -> m (b, D.Stream m (Array.Array a))
576−parseBreakD
577− (PRD.Parser pstep initial extract) stream@(D.Stream step state) = do
578−
579− res <- initial
580− case res of
581− PRD.IPartial s -> go SPEC state (List []) s
582− PRD.IDone b -> return (b, stream)
583− PRD.IError err -> throwM $ ParseError err
584−
585− where
586−
587− -- "backBuf" contains last few items in the stream that we may have to
588− -- backtrack to.
589− --
590− -- XXX currently we are using a dumb list based approach for backtracking
591− -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
592− -- That will allow us more efficient random back and forth movement.
593− go !_ st backBuf !pst = do
594− r <- step defState st
595− case r of
596− D.Yield (Array contents start end) s ->
597− gobuf SPEC s backBuf
598− (Tuple' end contents) start pst
599− D.Skip s -> go SPEC s backBuf pst
600− D.Stop -> do
601− b <- extract pst
602− return (b, D.nil)
603−
604− -- Use strictness on "cur" to keep it unboxed
605− gobuf !_ s backBuf (Tuple' end _) !cur !pst
606− | cur == end = do
607− go SPEC s backBuf pst
608− gobuf !_ s backBuf fp@(Tuple' end contents) !cur !pst = do
609− x <- liftIO $ peekByteIndex contents cur
610− pRes <- pstep pst x
611− let next = INDEX_NEXT(cur,a)
612− case pRes of
613− PR.Partial 0 pst1 ->
614− gobuf SPEC s (List []) fp next pst1
615− PR.Partial n pst1 -> do
616− assert (n <= Prelude.length (x:getList backBuf)) (return ())
617− let src0 = Prelude.take n (x:getList backBuf)
618− arr0 = A.fromListN n (Prelude.reverse src0)
619− arr1 = Array contents next end
620− src = arr0 <> arr1
621− let !(Array cont1 start end1) = src
622− fp1 = Tuple' end1 cont1
623− gobuf SPEC s (List []) fp1 start pst1
624− PR.Continue 0 pst1 ->
625− gobuf SPEC s (List (x:getList backBuf)) fp next pst1
626− PR.Continue n pst1 -> do
627− assert (n <= Prelude.length (x:getList backBuf)) (return ())
628− let (src0, buf1) = splitAt n (x:getList backBuf)
629− arr0 = A.fromListN n (Prelude.reverse src0)
630− arr1 = Array contents next end
631− src = arr0 <> arr1
632− let !(Array cont1 start end1) = src
633− fp1 = Tuple' end1 cont1
634− gobuf SPEC s (List buf1) fp1 start pst1
635− PR.Done 0 b -> do
636− let arr = Array contents next end
637− return (b, D.cons arr (D.Stream step s))
638− PR.Done n b -> do
639− assert (n <= Prelude.length (x:getList backBuf)) (return ())
640− let src0 = Prelude.take n (x:getList backBuf)
641− -- XXX create the array in reverse instead
642− arr0 = A.fromListN n (Prelude.reverse src0)
643− arr1 = Array contents next end
644− -- XXX Use StreamK to avoid adding arbitrary layers of
645− -- constructors every time.
646− str = D.cons arr0 (D.cons arr1 (D.Stream step s))
647− return (b, str)
648− PR.Error err -> throwM $ ParseError err
649−-}
650−
651−{-# INLINE_NORMAL parseBreakK #-}
652−parseBreakK ::
653− forall m a b. (MonadIO m, Unbox a)
654− => PRD.Parser a m b
655− -> K.StreamK m (Array.Array a)
656− -> m (Either ParseError b, K.StreamK m (Array.Array a))
657−parseBreakK (PRD.Parser pstep initial extract) stream = do
658− res <- initial
659− case res of
660− PRD.IPartial s -> go s stream []
661− PRD.IDone b -> return (Right b, stream)
662− PRD.IError err -> return (Left (ParseError err), stream)
663−
664− where
665−
666− -- "backBuf" contains last few items in the stream that we may have to
667− -- backtrack to.
668− --
669− -- XXX currently we are using a dumb list based approach for backtracking
670− -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
671− -- That will allow us more efficient random back and forth movement.
672− go !pst st backBuf = do
673− let stop = goStop pst backBuf -- (, K.nil) <$> extract pst
674− single a = yieldk a K.nil
675− yieldk arr r = goArray pst backBuf r arr
676− in K.foldStream defState yieldk single stop st
677−
678− -- Use strictness on "cur" to keep it unboxed
679− goArray !pst backBuf st (Array _ cur end) | cur == end = go pst st backBuf
680− goArray !pst backBuf st (Array contents cur end) = do
681− x <- liftIO $ peekByteIndex cur contents
682− pRes <- pstep pst x
683− let next = INDEX_NEXT(cur,a)
684− case pRes of
685− PR.Partial 0 s ->
686− goArray s [] st (Array contents next end)
687− PR.Partial n s -> do
688− assert (n <= Prelude.length (x:backBuf)) (return ())
689− let src0 = Prelude.take n (x:backBuf)
690− arr0 = A.fromListN n (Prelude.reverse src0)
691− arr1 = Array contents next end
692− src = arr0 <> arr1
693− goArray s [] st src
694− PR.Continue 0 s ->
695− goArray s (x:backBuf) st (Array contents next end)
696− PR.Continue n s -> do
697− assert (n <= Prelude.length (x:backBuf)) (return ())
698− let (src0, buf1) = splitAt n (x:backBuf)
699− arr0 = A.fromListN n (Prelude.reverse src0)
700− arr1 = Array contents next end
701− src = arr0 <> arr1
702− goArray s buf1 st src
703− PR.Done 0 b -> do
704− let arr = Array contents next end
705− return (Right b, K.cons arr st)
706− PR.Done n b -> do
707− assert (n <= Prelude.length (x:backBuf)) (return ())
708− let src0 = Prelude.take n (x:backBuf)
709− -- XXX Use fromListRevN once implemented
710− -- arr0 = A.fromListRevN n src0
711− arr0 = A.fromListN n (Prelude.reverse src0)
712− arr1 = Array contents next end
713− str = K.cons arr0 (K.cons arr1 st)
714− return (Right b, str)
715− PR.Error err -> do
716− let n = Prelude.length backBuf
717− arr0 = A.fromListN n (Prelude.reverse backBuf)
718− arr1 = Array contents cur end
719− str = K.cons arr0 (K.cons arr1 stream)
720− return (Left (ParseError err), str)
721−
722− -- This is a simplified goArray
723− goExtract !pst backBuf (Array _ cur end)
724− | cur == end = goStop pst backBuf
725− goExtract !pst backBuf (Array contents cur end) = do
726− x <- liftIO $ peekByteIndex cur contents
727− pRes <- pstep pst x
728− let next = INDEX_NEXT(cur,a)
729− case pRes of
730− PR.Partial 0 s ->
731− goExtract s [] (Array contents next end)
732− PR.Partial n s -> do
733− assert (n <= Prelude.length (x:backBuf)) (return ())
734− let src0 = Prelude.take n (x:backBuf)
735− arr0 = A.fromListN n (Prelude.reverse src0)
736− arr1 = Array contents next end
737− src = arr0 <> arr1
738− goExtract s [] src
739− PR.Continue 0 s ->
740− goExtract s backBuf (Array contents next end)
741− PR.Continue n s -> do
742− assert (n <= Prelude.length (x:backBuf)) (return ())
743− let (src0, buf1) = splitAt n (x:backBuf)
744− arr0 = A.fromListN n (Prelude.reverse src0)
745− arr1 = Array contents next end
746− src = arr0 <> arr1
747− goExtract s buf1 src
748− PR.Done 0 b -> do
749− let arr = Array contents next end
750− return (Right b, K.fromPure arr)
751− PR.Done n b -> do
752− assert (n <= Prelude.length backBuf) (return ())
753− let src0 = Prelude.take n backBuf
754− -- XXX Use fromListRevN once implemented
755− -- arr0 = A.fromListRevN n src0
756− arr0 = A.fromListN n (Prelude.reverse src0)
757− arr1 = Array contents next end
758− str = K.cons arr0 (K.fromPure arr1)
759− return (Right b, str)
760− PR.Error err -> do
761− let n = Prelude.length backBuf
762− arr0 = A.fromListN n (Prelude.reverse backBuf)
763− arr1 = Array contents cur end
764− str = K.cons arr0 (K.cons arr1 stream)
765− return (Left (ParseError err), str)
766−
767− -- This is a simplified goExtract
768− {-# INLINE goStop #-}
769− goStop !pst backBuf = do
770− pRes <- extract pst
771− case pRes of
772− PR.Partial _ _ -> error "Bug: parseBreak: Partial in extract"
773− PR.Continue 0 s ->
774− goStop s backBuf
775− PR.Continue n s -> do
776− assert (n <= Prelude.length backBuf) (return ())
777− let (src0, buf1) = splitAt n backBuf
778− arr = A.fromListN n (Prelude.reverse src0)
779− goExtract s buf1 arr
780− PR.Done 0 b ->
781− return (Right b, K.nil)
782− PR.Done n b -> do
783− assert (n <= Prelude.length backBuf) (return ())
784− let src0 = Prelude.take n backBuf
785− -- XXX Use fromListRevN once implemented
786− -- arr0 = A.fromListRevN n src0
787− arr0 = A.fromListN n (Prelude.reverse src0)
788− return (Right b, K.fromPure arr0)
789− PR.Error err -> do
790− let n = Prelude.length backBuf
791− arr0 = A.fromListN n (Prelude.reverse backBuf)
792− return (Left (ParseError err), K.fromPure arr0)
793−
794273-- | Parse an array stream using the supplied 'Parser'. Returns the parse
795274-- result and the unconsumed stream. Throws 'ParseError' if the parse fails.
796275--
276+-- >> parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
277+--
278+-- This is redundant and we can just use parseBreakChunks, as ParserK can be
279+-- composed using Monad. The only advantage of this is that we do not need to
280+-- adapt.
281+--
282+-- We can compare perf and remove this one or define it in terms of that.
283+--
797284-- /Internal/
798285--
799286{-# INLINE_NORMAL parseBreak #-}
@@ -806,7 +293,8 @@ parseBreak ::
806293parseBreak p s =
807294 fmap fromStreamD <$> parseBreakD (PRD.fromParserK p) (toStreamD s)
808295-}
809−parseBreak = parseBreakK
296+parseBreak = Array.parseBreakChunksK
297+-- parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
810298
811299-------------------------------------------------------------------------------
812300-- Elimination - Running Array Folds and parsers
core/src/Streamly/Internal/Data/Array/Type.hs+517−109
@@ -1,8 +1,11 @@
11{-# LANGUAGE CPP #-}
2+{-# LANGUAGE TypeFamilies #-}
3+-- Must come after TypeFamilies, otherwise it is re-enabled.
4+-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
5+{-# LANGUAGE NoMonoLocalBinds #-}
26-- |
37-- Module : Streamly.Internal.Data.Array.Type
48-- Copyright : (c) 2020 Composewell Technologies
5−--
69-- License : BSD3-3-Clause
710-- Maintainer : streamly@composewell.com
811-- Stability : experimental
@@ -10,86 +13,146 @@
1013--
1114-- See notes in "Streamly.Internal.Data.MutArray.Type"
1215--
16+
1317module Streamly.Internal.Data.Array.Type
1418 (
19+ -- ** Type
1520 -- $arrayNotes
1621 Array (..)
17− , asPtrUnsafe
18− , nil
1922
20− -- * Freezing and Thawing
23+ -- ** Conversion
24+ -- *** Mutable and Immutable
2125 , unsafeFreeze
2226 , unsafeFreezeWithShrink
2327 , unsafeThaw
2428
25− -- * Pinning and Unpinning
29+ -- *** Pinned and Unpinned
2630 , pin
2731 , unpin
2832 , isPinned
2933
30− -- * Construction
31− , splice
34+ -- *** Casting
35+ , unsafePinnedAsPtr
3236
33− , fromList
34− , pinnedFromList
37+ -- ** Construction
38+ , empty
39+
40+ -- *** Cloning
41+ -- XXX Why would we clone an immutable array?
42+ , clone
43+ , pinnedClone
44+
45+ -- *** Slicing
46+ -- | Get a subarray without copying
47+ , splitAt
48+ , breakOn -- XXX requires MonadIO
49+
50+ -- *** Stream Folds
51+ , unsafeMakePure
52+ , createOf
53+ , pinnedCreateOf
54+ , unsafeCreateOf
55+ , unsafePinnedCreateOf
56+ , create
57+ , pinnedCreate
58+ , createWith
59+
60+ -- *** From containers
3561 , fromListN
3662 , pinnedFromListN
37− , fromListRev
63+ , fromList
64+ , pinnedFromList
3865 , fromListRevN
39− , fromStreamDN
40− , fromStreamD
66+ , fromListRev
67+ , fromStreamN
68+ , fromStream
69+ , fromPureStreamN
4170 , fromPureStream
4271 , fromByteStr#
72+ , fromByteStr
73+ , fromPtrN
74+ , fromChunks
75+ , fromChunksK
4376
44− -- * Split
45− , breakOn
77+ -- ** Reading
4678
47− -- ** Cloning arrays
48− , clone
49− , pinnedClone
79+ -- *** Indexing
80+ , unsafeIndexIO -- XXX unsafeGetIndexIO
81+ , getIndexUnsafe -- XXX unsafeGetIndex
5082
51− -- * Elimination
52− , unsafeIndexIO
53− , getIndexUnsafe
54− , byteLength
83+ -- *** To Streams
84+ , read
85+ , readRev
86+ , toStreamK
87+ , toStreamKRev
88+
89+ -- *** To Containers
90+ , toList
91+
92+ -- *** Unfolds
93+ , producer -- experimental
94+ , readerUnsafe
95+ , reader
96+ , readerRev
97+
98+ -- *** Size
5599 , length
100+ , byteLength
56101
102+ -- ** Folding
57103 , foldl'
58104 , foldr
59− , splitAt
105+ , byteCmp
106+ , byteEq
107+
108+ -- ** Appending
109+ , splice -- XXX requires MonadIO
110+
111+ -- ** Streams of arrays
112+ -- *** Chunk
113+ -- | Group a stream into arrays.
114+ , chunksOf
115+ , pinnedChunksOf
116+ , buildChunks
60117
118+ -- *** Split
119+ -- | Split an array into slices.
120+
121+ -- *** Concat
122+ -- | Append the arrays in a stream to form a stream of elements.
123+ , concat
124+ , concatRev
125+
126+ -- *** Compact
127+ -- | Append the arrays in a stream to form a stream of larger arrays.
128+ , fCompactGE
129+ , fPinnedCompactGE
130+ , lCompactGE
131+ , lPinnedCompactGE
132+ , compactGE
133+
134+ -- ** Deprecated
135+ , asPtrUnsafe
136+ , unsafeIndex
137+ , bufferChunks
138+ , flattenArrays
139+ , flattenArraysRev
140+ , fromArrayStreamK
141+ , fromStreamDN
142+ , fromStreamD
61143 , toStreamD
62144 , toStreamDRev
63− , toStreamK
64− , toStreamKRev
65145 , toStream
66146 , toStreamRev
67− , read
68− , readRev
69− , readerRev
70− , toList
71−
72− -- * Folds
147+ , nil
73148 , writeWith
74149 , writeN
75150 , pinnedWriteN
76151 , writeNUnsafe
77152 , pinnedWriteNUnsafe
78− , MA.ArrayUnsafe (..)
79153 , pinnedWriteNAligned
80154 , write
81155 , pinnedWrite
82− , unsafeMakePure
83−
84− -- * Streams of arrays
85− , chunksOf
86− , pinnedChunksOf
87− , bufferChunks
88− , flattenArrays
89− , flattenArraysRev
90−
91− -- * Deprecated
92− , unsafeIndex
93156 )
94157where
95158
@@ -97,31 +160,36 @@ where
97160#include "inline.hs"
98161
99162import Control.Exception (assert)
100−import Control.Monad (replicateM)
163+import Control.Monad (replicateM, when)
101164import Control.Monad.IO.Class (MonadIO(..))
102165import Data.Functor.Identity (Identity(..))
166+import Data.Int (Int8, Int16, Int32, Int64)
103167import Data.Proxy (Proxy(..))
104−import Data.Word (Word8)
168+import Data.Word (Word8, Word16, Word32, Word64)
105169import GHC.Base (build)
106170import GHC.Exts (IsList, IsString(..), Addr#)
107171
108172import GHC.IO (unsafePerformIO)
109173import GHC.Ptr (Ptr(..))
110−import Streamly.Internal.Data.MutArray.Type (MutArray(..), MutableByteArray)
174+import Streamly.Internal.Data.Producer.Type (Producer(..))
175+import Streamly.Internal.Data.MutArray.Type (MutArray(..))
176+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
111177import Streamly.Internal.Data.Fold.Type (Fold(..))
112178import Streamly.Internal.Data.Stream.Type (Stream)
179+import Streamly.Internal.Data.StreamK.Type (StreamK)
113180import Streamly.Internal.Data.Unbox (Unbox(..))
114181import Streamly.Internal.Data.Unfold.Type (Unfold(..))
115182import Text.Read (readPrec)
116183
117−import Prelude hiding (Foldable(..), read, unlines, splitAt)
184+import Prelude hiding (Foldable(..), concat, read, unlines, splitAt)
118185
119186import qualified GHC.Exts as Exts
187+import qualified Streamly.Internal.Data.Fold.Type as Fold
120188import qualified Streamly.Internal.Data.MutArray.Type as MA
121189import qualified Streamly.Internal.Data.Stream.Type as D
122−import qualified Streamly.Internal.Data.Stream.Generate as D
123190import qualified Streamly.Internal.Data.StreamK.Type as K
124−import qualified Streamly.Internal.Data.Unbox as Unboxed
191+import qualified Streamly.Internal.Data.MutByteArray.Type as Unboxed
192+import qualified Streamly.Internal.Data.Producer as Producer
125193import qualified Streamly.Internal.Data.Unfold.Type as Unfold
126194import qualified Text.ParserCombinators.ReadPrec as ReadPrec
127195
@@ -164,7 +232,7 @@ data Array a =
164232#endif
165233 -- All offsets are in terms of bytes from the start of arraycontents
166234 Array
167− { arrContents :: {-# UNPACK #-} !MutableByteArray
235+ { arrContents :: {-# UNPACK #-} !MutByteArray
168236 , arrStart :: {-# UNPACK #-} !Int -- offset
169237 , arrEnd :: {-# UNPACK #-} !Int -- offset + len
170238 }
@@ -175,15 +243,20 @@ data Array a =
175243
176244-- | Use an @Array a@ as @Ptr a@.
177245--
178−-- See 'MA.asPtrUnsafe' in the Mutable array module for more details.
246+-- See 'MA.unsafePinnedAsPtr' in the Mutable array module for more details.
179247--
180248-- /Unsafe/
181249--
182250-- /Pre-release/
183251--
252+{-# INLINE unsafePinnedAsPtr #-}
253+unsafePinnedAsPtr :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
254+unsafePinnedAsPtr arr = MA.unsafePinnedAsPtr (unsafeThaw arr)
255+
256+{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
184257{-# INLINE asPtrUnsafe #-}
185258asPtrUnsafe :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
186−asPtrUnsafe arr = MA.asPtrUnsafe (unsafeThaw arr)
259+asPtrUnsafe = unsafePinnedAsPtr
187260
188261-------------------------------------------------------------------------------
189262-- Freezing and Thawing
@@ -230,18 +303,19 @@ unsafeThaw (Array ac as ae) = MutArray ac as ae ae
230303-- Pinning & Unpinning
231304-------------------------------------------------------------------------------
232305
233−-- | Copy the 'Array' to pinned memory if unpinned, else do nothing. The
234−-- overhead is a copy if the input array is unpinned.
306+-- | Return a copy of the 'Array' in pinned memory if unpinned, else return the
307+-- original array.
235308{-# INLINE pin #-}
236309pin :: Array a -> IO (Array a)
237310pin = fmap unsafeFreeze . MA.pin . unsafeThaw
238311
239−-- | Copy the 'Array' to unpinned memory if pinned, else do nothing. The
240−-- overhead is a copy if the input array is pinned.
312+-- | Return a copy of the 'Array' in unpinned memory if pinned, else return the
313+-- original array.
241314{-# INLINE unpin #-}
242315unpin :: Array a -> IO (Array a)
243316unpin = fmap unsafeFreeze . MA.unpin . unsafeThaw
244317
318+-- | Return 'True' if the array is allocated in pinned memory.
245319{-# INLINE isPinned #-}
246320isPinned :: Array a -> Bool
247321isPinned = MA.isPinned . unsafeThaw
@@ -250,11 +324,14 @@ isPinned = MA.isPinned . unsafeThaw
250324-- Construction
251325-------------------------------------------------------------------------------
252326
253−-- Splice two immutable arrays creating a new array.
327+-- | Copy two immutable arrays into a new array. If you want to splice more
328+-- than two arrays then this operation would be highly inefficient because it
329+-- would make a copy on every splice operation, instead use the
330+-- 'fromChunksK' operation to combine n immutable arrays.
254331{-# INLINE splice #-}
255−splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
332+splice :: MonadIO m => Array a -> Array a -> m (Array a)
256333splice arr1 arr2 =
257− unsafeFreeze <$> MA.splice (unsafeThaw arr1) (unsafeThaw arr2)
334+ unsafeFreeze <$> MA.spliceCopy (unsafeThaw arr1) (unsafeThaw arr2)
258335
259336-- | Create an 'Array' from the first N elements of a list. The array is
260337-- allocated to size N, if the list terminates before N elements then the
@@ -298,36 +375,83 @@ pinnedFromList xs = unsafePerformIO $ unsafeFreeze <$> MA.pinnedFromList xs
298375fromListRev :: Unbox a => [a] -> Array a
299376fromListRev xs = unsafePerformIO $ unsafeFreeze <$> MA.fromListRev xs
300377
301−{-# INLINE_NORMAL fromStreamDN #-}
378+-- | Create an 'Array' from the first N elements of a stream. The array is
379+-- allocated to size N, if the stream terminates before N elements then the
380+-- array may hold less than N elements.
381+--
382+-- >>> fromStreamN n = Stream.fold (Array.writeN n)
383+--
384+-- /Pre-release/
385+{-# INLINE_NORMAL fromStreamN #-}
386+fromStreamN :: (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
387+fromStreamN n m = do
388+ when (n < 0) $ error "writeN: negative write count specified"
389+ unsafeFreeze <$> MA.fromStreamN n m
390+-- fromStreamN n = D.fold (writeN n)
391+
392+{-# DEPRECATED fromStreamDN "Please use fromStreamN instead." #-}
302393fromStreamDN :: forall m a. (MonadIO m, Unbox a)
303394 => Int -> D.Stream m a -> m (Array a)
304−fromStreamDN limit str = unsafeFreeze <$> MA.fromStreamDN limit str
395+fromStreamDN = fromStreamN
305396
397+-- | Create an 'Array' from a stream. This is useful when we want to create a
398+-- single array from a stream of unknown size. 'writeN' is at least twice
399+-- as efficient when the size is already known.
400+--
401+-- >>> fromStream = Stream.fold Array.write
402+--
403+-- Note that if the input stream is too large memory allocation for the array
404+-- may fail. When the stream size is not known, `chunksOf` followed by
405+-- processing of indvidual arrays in the resulting stream should be preferred.
406+--
407+-- /Pre-release/
306408{-# INLINE_NORMAL fromStreamD #-}
409+fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (Array a)
410+fromStream = D.fold write
411+-- fromStreamD str = unsafeFreeze <$> MA.fromStream str
412+
413+{-# DEPRECATED fromStreamD "Please use fromStream instead." #-}
307414fromStreamD :: forall m a. (MonadIO m, Unbox a)
308415 => D.Stream m a -> m (Array a)
309−fromStreamD str = unsafeFreeze <$> MA.fromStreamD str
416+fromStreamD = fromStream
310417
311418-------------------------------------------------------------------------------
312419-- Cloning
313420-------------------------------------------------------------------------------
314421
315422{-# INLINE clone #-}
316−clone :: MonadIO m => Array a -> m (Array a)
423+clone ::
424+ ( MonadIO m
425+#ifdef DEVBUILD
426+ , Unbox a
427+#endif
428+ )
429+ => Array a -> m (Array a)
317430clone = fmap unsafeFreeze . MA.clone . unsafeThaw
318431
319432{-# INLINE pinnedClone #-}
320−pinnedClone :: MonadIO m => Array a -> m (Array a)
433+pinnedClone ::
434+ ( MonadIO m
435+#ifdef DEVBUILD
436+ , Unbox a
437+#endif
438+ )
439+ => Array a -> m (Array a)
321440pinnedClone = fmap unsafeFreeze . MA.pinnedClone . unsafeThaw
322441
323442-------------------------------------------------------------------------------
324443-- Streams of arrays
325444-------------------------------------------------------------------------------
326445
327−{-# INLINE bufferChunks #-}
446+{-# INLINE buildChunks #-}
447+buildChunks :: (MonadIO m, Unbox a) =>
448+ D.Stream m a -> m (K.StreamK m (Array a))
449+buildChunks m = D.foldr K.cons K.nil $ chunksOf defaultChunkSize m
450+
451+{-# DEPRECATED bufferChunks "Please use buildChunks instead." #-}
328452bufferChunks :: (MonadIO m, Unbox a) =>
329453 D.Stream m a -> m (K.StreamK m (Array a))
330−bufferChunks m = D.foldr K.cons K.nil $ chunksOf defaultChunkSize m
454+bufferChunks = buildChunks
331455
332456-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
333457-- @n@ elements each.
@@ -348,27 +472,100 @@ pinnedChunksOf :: forall m a. (MonadIO m, Unbox a)
348472 => Int -> D.Stream m a -> D.Stream m (Array a)
349473pinnedChunksOf n str = D.map unsafeFreeze $ MA.pinnedChunksOf n str
350474
351−-- | Use the "read" unfold instead.
475+-- | Convert a stream of arrays into a stream of their elements.
352476--
353−-- @flattenArrays = unfoldMany read@
477+-- >>> concat = Stream.unfoldMany Array.reader
354478--
355−-- We can try this if there are any fusion issues in the unfold.
356−--
357−{-# INLINE_NORMAL flattenArrays #-}
479+{-# INLINE_NORMAL concat #-}
480+concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
481+-- XXX this requires MonadIO whereas the unfoldMany version does not
482+concat = MA.concatWith (pure . unsafeInlineIO) . D.map unsafeThaw
483+-- concat = D.unfoldMany reader
484+
485+{-# DEPRECATED flattenArrays "Please use \"unfoldMany reader\" instead." #-}
486+{-# INLINE flattenArrays #-}
358487flattenArrays :: forall m a. (MonadIO m, Unbox a)
359488 => D.Stream m (Array a) -> D.Stream m a
360−flattenArrays = MA.flattenArrays . D.map unsafeThaw
489+flattenArrays = concat
361490
362−-- | Use the "readRev" unfold instead.
363−--
364−-- @flattenArrays = unfoldMany readRev@
491+-- | Convert a stream of arrays into a stream of their elements reversing the
492+-- contents of each array before flattening.
365493--
366−-- We can try this if there are any fusion issues in the unfold.
494+-- >>> concatRev = Stream.unfoldMany Array.readerRev
367495--
368−{-# INLINE_NORMAL flattenArraysRev #-}
496+{-# INLINE_NORMAL concatRev #-}
497+concatRev :: forall m a. (Monad m, Unbox a)
498+ => D.Stream m (Array a) -> D.Stream m a
499+-- XXX this requires MonadIO whereas the unfoldMany version does not
500+concatRev = MA.concatRevWith (pure . unsafeInlineIO) . D.map unsafeThaw
501+-- concatRev = D.unfoldMany readerRev
502+
503+{-# DEPRECATED flattenArraysRev "Please use \"unfoldMany readerRev\" instead." #-}
504+{-# INLINE flattenArraysRev #-}
369505flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
370506 => D.Stream m (Array a) -> D.Stream m a
371−flattenArraysRev = MA.flattenArraysRev . D.map unsafeThaw
507+flattenArraysRev = concatRev
508+
509+-------------------------------------------------------------------------------
510+-- Compact
511+-------------------------------------------------------------------------------
512+
513+-- XXX Note that this thaws immutable arrays for appending, that may be
514+-- problematic if multiple users do the same thing, however, thawed immutable
515+-- arrays would have no capacity to append, therefore, a copy will be forced
516+-- anyway.
517+
518+-- | Fold @fCompactGE n@ coalesces adjacent arrays in the input stream
519+-- until the size becomes greater than or equal to n.
520+--
521+-- Generates unpinned arrays irrespective of the pinning status of input
522+-- arrays.
523+{-# INLINE_NORMAL fCompactGE #-}
524+fCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
525+fCompactGE n = fmap unsafeFreeze $ Fold.lmap unsafeThaw $ MA.fCompactGE n
526+
527+-- | PInned version of 'fCompactGE'.
528+{-# INLINE_NORMAL fPinnedCompactGE #-}
529+fPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
530+fPinnedCompactGE n =
531+ fmap unsafeFreeze $ Fold.lmap unsafeThaw $ MA.fPinnedCompactGE n
532+
533+-- | @compactGE n stream@ coalesces adjacent arrays in the @stream@ until
534+-- the size becomes greater than or equal to @n@.
535+--
536+-- >>> compactGE n = Stream.foldMany (Array.fCompactGE n)
537+--
538+-- Generates unpinned arrays irrespective of the pinning status of input
539+-- arrays.
540+{-# INLINE compactGE #-}
541+compactGE ::
542+ (MonadIO m, Unbox a)
543+ => Int -> Stream m (Array a) -> Stream m (Array a)
544+compactGE n stream =
545+ D.map unsafeFreeze $ MA.compactGE n $ D.map unsafeThaw stream
546+
547+-- | Like 'compactGE' but for transforming folds instead of stream.
548+--
549+-- >>> lCompactGE n = Fold.many (Array.fCompactGE n)
550+--
551+-- Generates unpinned arrays irrespective of the pinning status of input
552+-- arrays.
553+{-# INLINE_NORMAL lCompactGE #-}
554+lCompactGE :: (MonadIO m, Unbox a)
555+ => Int -> Fold m (Array a) () -> Fold m (Array a) ()
556+lCompactGE n fld =
557+ Fold.lmap unsafeThaw $ MA.lCompactGE n (Fold.lmap unsafeFreeze fld)
558+
559+-- | Pinned version of 'lCompactGE'.
560+{-# INLINE_NORMAL lPinnedCompactGE #-}
561+lPinnedCompactGE :: (MonadIO m, Unbox a)
562+ => Int -> Fold m (Array a) () -> Fold m (Array a) ()
563+lPinnedCompactGE n fld =
564+ Fold.lmap unsafeThaw $ MA.lPinnedCompactGE n (Fold.lmap unsafeFreeze fld)
565+
566+-------------------------------------------------------------------------------
567+-- Splitting
568+-------------------------------------------------------------------------------
372569
373570-- Drops the separator byte
374571{-# INLINE breakOn #-}
@@ -412,24 +609,68 @@ byteLength = MA.byteLength . unsafeThaw
412609length :: Unbox a => Array a -> Int
413610length arr = MA.length (unsafeThaw arr)
414611
612+{-# INLINE_NORMAL producer #-}
613+producer :: forall m a. (Monad m, Unbox a) => Producer m (Array a) a
614+producer =
615+ Producer.translate unsafeThaw unsafeFreeze
616+ $ MA.producerWith (return . unsafeInlineIO)
617+
618+-- | Unfold an array into a stream.
619+--
620+{-# INLINE_NORMAL reader #-}
621+reader :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
622+reader = Producer.simplify producer
623+
624+-- | Unfold an array into a stream, does not check the end of the array, the
625+-- user is responsible for terminating the stream within the array bounds. For
626+-- high performance application where the end condition can be determined by
627+-- a terminating fold.
628+--
629+-- Written in the hope that it may be faster than "read", however, in the case
630+-- for which this was written, "read" proves to be faster even though the core
631+-- generated with unsafeRead looks simpler.
632+--
633+-- /Pre-release/
634+--
635+{-# INLINE_NORMAL readerUnsafe #-}
636+readerUnsafe :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
637+readerUnsafe = Unfold step inject
638+ where
639+
640+ inject (Array contents start end) =
641+ return (MA.ArrayUnsafe contents end start)
642+
643+ {-# INLINE_LATE step #-}
644+ step (MA.ArrayUnsafe contents end p) = do
645+ -- unsafeInlineIO allows us to run this in Identity monad for pure
646+ -- toList/foldr case which makes them much faster due to not
647+ -- accumulating the list and fusing better with the pure consumers.
648+ --
649+ -- This should be safe as the array contents are guaranteed to be
650+ -- evaluated/written to before we peek at them.
651+ let !x = unsafeInlineIO $ peekAt p contents
652+ let !p1 = INDEX_NEXT(p,a)
653+ return $ D.Yield x (MA.ArrayUnsafe contents end p1)
654+
415655-- | Unfold an array into a stream in reverse order.
416656--
417657{-# INLINE_NORMAL readerRev #-}
418658readerRev :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
419659readerRev = Unfold.lmap unsafeThaw $ MA.readerRevWith (return . unsafeInlineIO)
420660
661+{-# DEPRECATED toStreamD "Please use 'read' instead." #-}
421662{-# INLINE_NORMAL toStreamD #-}
422663toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> D.Stream m a
423−toStreamD arr = MA.toStreamDWith (return . unsafeInlineIO) (unsafeThaw arr)
664+toStreamD = read
424665
425666{-# INLINE toStreamK #-}
426667toStreamK :: forall m a. (Monad m, Unbox a) => Array a -> K.StreamK m a
427668toStreamK arr = MA.toStreamKWith (return . unsafeInlineIO) (unsafeThaw arr)
428669
670+{-# DEPRECATED toStreamDRev "Please use 'readRev' instead." #-}
429671{-# INLINE_NORMAL toStreamDRev #-}
430672toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> D.Stream m a
431−toStreamDRev arr =
432− MA.toStreamDRevWith (return . unsafeInlineIO) (unsafeThaw arr)
673+toStreamDRev = readRev
433674
434675{-# INLINE toStreamKRev #-}
435676toStreamKRev :: forall m a. (Monad m, Unbox a) => Array a -> K.StreamK m a
@@ -441,7 +682,7 @@ toStreamKRev arr =
441682-- /Pre-release/
442683{-# INLINE_EARLY read #-}
443684read :: (Monad m, Unbox a) => Array a -> Stream m a
444−read = toStreamD
685+read arr = MA.toStreamWith (return . unsafeInlineIO) (unsafeThaw arr)
445686
446687-- | Same as 'read'
447688--
@@ -458,7 +699,7 @@ toStream = read
458699-- /Pre-release/
459700{-# INLINE_EARLY readRev #-}
460701readRev :: (Monad m, Unbox a) => Array a -> Stream m a
461−readRev = toStreamDRev
702+readRev arr = MA.toStreamRevWith (return . unsafeInlineIO) (unsafeThaw arr)
462703
463704-- | Same as 'readRev'
464705--
@@ -482,6 +723,7 @@ foldr f z arr = runIdentity $ D.foldr f z $ toStreamD arr
482723-- | Create two slices of an array without copying the original array. The
483724-- specified index @i@ is the first index of the second slice.
484725--
726+{-# INLINE splitAt #-}
485727splitAt :: Unbox a => Int -> Array a -> (Array a, Array a)
486728splitAt i arr = (unsafeFreeze a, unsafeFreeze b)
487729 where
@@ -501,8 +743,8 @@ toListFB c n Array{..} = go arrStart
501743 -- accumulating the list and fusing better with the pure consumers.
502744 --
503745 -- This should be safe as the array contents are guaranteed to be
504− -- evaluated/written to before we peekByteIndex at them.
505− let !x = unsafeInlineIO $ peekByteIndex p arrContents
746+ -- evaluated/written to before we peekAt at them.
747+ let !x = unsafeInlineIO $ peekAt p arrContents
506748 in c x (go (INDEX_NEXT(p,a)))
507749
508750-- | Convert an 'Array' into a list.
@@ -515,17 +757,28 @@ toList s = build (\c n -> toListFB c n s)
515757-- Folds
516758-------------------------------------------------------------------------------
517759
518−-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
760+-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
519761-- 'Array'.
520762--
521−{-# INLINE_NORMAL writeN #-}
763+{-# INLINE_NORMAL createOf #-}
764+createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
765+createOf = fmap unsafeFreeze . MA.createOf
766+
767+-- XXX Deprecate in major
768+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
769+{-# INLINE writeN #-}
522770writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
523−writeN = fmap unsafeFreeze . MA.writeN
771+writeN = createOf
524772
525−-- | Like 'fromListN' but creates a pinned array.
526−{-# INLINE_NORMAL pinnedWriteN #-}
773+-- | Like 'createOf' but creates a pinned array.
774+{-# INLINE_NORMAL pinnedCreateOf #-}
775+pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
776+pinnedCreateOf = fmap unsafeFreeze . MA.pinnedCreateOf
777+
778+{-# DEPRECATED pinnedWriteN "Please use pinnedCreateOf instead." #-}
779+{-# INLINE pinnedWriteN #-}
527780pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
528−pinnedWriteN = fmap unsafeFreeze . MA.pinnedWriteN
781+pinnedWriteN = pinnedCreateOf
529782
530783-- | @pinnedWriteNAligned alignment n@ folds a maximum of @n@ elements from the input
531784-- stream to an 'Array' aligned to the given size.
@@ -533,44 +786,74 @@ pinnedWriteN = fmap unsafeFreeze . MA.pinnedWriteN
533786-- /Pre-release/
534787--
535788{-# INLINE_NORMAL pinnedWriteNAligned #-}
789+{-# DEPRECATED pinnedWriteNAligned "To be removed." #-}
536790pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a)
537791 => Int -> Int -> Fold m a (Array a)
538792pinnedWriteNAligned alignSize = fmap unsafeFreeze . MA.pinnedWriteNAligned alignSize
539793
540−-- | Like 'writeN' but does not check the array bounds when writing. The fold
794+-- | Like 'createOf' but does not check the array bounds when writing. The fold
541795-- driver must not call the step function more than 'n' times otherwise it will
542796-- corrupt the memory and crash. This function exists mainly because any
543797-- conditional in the step function blocks fusion causing 10x performance
544798-- slowdown.
545799--
546−{-# INLINE_NORMAL writeNUnsafe #-}
800+{-# INLINE_NORMAL unsafeCreateOf #-}
801+unsafeCreateOf :: forall m a. (MonadIO m, Unbox a)
802+ => Int -> Fold m a (Array a)
803+unsafeCreateOf n = unsafeFreeze <$> MA.unsafeCreateOf n
804+
805+{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
806+{-# INLINE writeNUnsafe #-}
547807writeNUnsafe :: forall m a. (MonadIO m, Unbox a)
548808 => Int -> Fold m a (Array a)
549−writeNUnsafe n = unsafeFreeze <$> MA.writeNUnsafe n
809+writeNUnsafe = unsafeCreateOf
550810
551−{-# INLINE_NORMAL pinnedWriteNUnsafe #-}
811+{-# INLINE_NORMAL unsafePinnedCreateOf #-}
812+unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a)
813+ => Int -> Fold m a (Array a)
814+unsafePinnedCreateOf n = unsafeFreeze <$> MA.unsafePinnedCreateOf n
815+
816+{-# DEPRECATED pinnedWriteNUnsafe "Please use unsafePinnedCreateOf instead." #-}
817+{-# INLINE pinnedWriteNUnsafe #-}
552818pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a)
553819 => Int -> Fold m a (Array a)
554−pinnedWriteNUnsafe n = unsafeFreeze <$> MA.pinnedWriteNUnsafe n
820+pinnedWriteNUnsafe = unsafePinnedCreateOf
821+
822+{-# INLINE_NORMAL createWith #-}
823+createWith :: forall m a. (MonadIO m, Unbox a)
824+ => Int -> Fold m a (Array a)
825+-- createWith n = FL.rmapM spliceArrays $ toArraysOf n
826+createWith elemCount = unsafeFreeze <$> MA.createWith elemCount
555827
556−{-# INLINE_NORMAL writeWith #-}
828+{-# DEPRECATED writeWith "Please use createWith instead." #-}
829+{-# INLINE writeWith #-}
557830writeWith :: forall m a. (MonadIO m, Unbox a)
558831 => Int -> Fold m a (Array a)
559−-- writeWith n = FL.rmapM spliceArrays $ toArraysOf n
560−writeWith elemCount = unsafeFreeze <$> MA.writeWith elemCount
832+writeWith = createWith
561833
562834-- | Fold the whole input to a single array.
563835--
564836-- /Caution! Do not use this on infinite streams./
565837--
838+{-# INLINE create #-}
839+create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
840+create = fmap unsafeFreeze MA.create
841+
842+-- XXX Deprecate in major
843+-- {-# DEPRECATED write "Please use create instead." #-}
566844{-# INLINE write #-}
567845write :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
568−write = fmap unsafeFreeze MA.write
846+write = create
569847
570−-- | Like 'write' but creates a pinned array.
848+-- | Like 'create' but creates a pinned array.
849+{-# INLINE pinnedCreate #-}
850+pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
851+pinnedCreate = fmap unsafeFreeze MA.pinnedCreate
852+
853+{-# DEPRECATED pinnedWrite "Please use pinnedCreate instead." #-}
571854{-# INLINE pinnedWrite #-}
572855pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
573−pinnedWrite = fmap unsafeFreeze MA.pinnedWrite
856+pinnedWrite = pinnedCreate
574857
575858-- | Fold "step" has a dependency on "initial", and each step is dependent on
576859-- the previous invocation of step due to state passing, finally extract
@@ -596,10 +879,16 @@ pinnedWrite = fmap unsafeFreeze MA.pinnedWrite
596879--
597880{-# INLINE unsafeMakePure #-}
598881unsafeMakePure :: Monad m => Fold IO a b -> Fold m a b
599−unsafeMakePure (Fold step initial extract) =
882+unsafeMakePure (Fold step initial extract final) =
600883 Fold (\x a -> return $! unsafeInlineIO (step x a))
601884 (return $! unsafePerformIO initial)
602885 (\s -> return $! unsafeInlineIO $ extract s)
886+ (\s -> return $! unsafeInlineIO $ final s)
887+
888+{-# INLINE fromPureStreamN #-}
889+fromPureStreamN :: Unbox a => Int -> Stream Identity a -> Array a
890+fromPureStreamN n x =
891+ unsafePerformIO $ fmap unsafeFreeze (MA.fromPureStreamN n x)
603892
604893-- | Convert a pure stream in Identity monad to an immutable array.
605894--
@@ -612,6 +901,19 @@ fromPureStream x = unsafePerformIO $ fmap (unsafeFreeze) (MA.fromPureStream x)
612901-- fromPureStream = runIdentity . D.fold (unsafeMakePure write)
613902-- fromPureStream = fromList . runIdentity . D.toList
614903
904+-- XXX This should be monadic.
905+
906+-- | Copy an immutable 'Ptr Word8' sequence into an array.
907+--
908+-- /Unsafe:/ The caller is responsible for safe addressing.
909+--
910+-- Note that this should be evaluated strictly to ensure that we do not hold
911+-- the reference to the pointer in a lazy thunk.
912+fromPtrN :: Int -> Ptr Word8 -> Array Word8
913+fromPtrN n addr = unsafePerformIO $ fmap unsafeFreeze (MA.fromPtrN n addr)
914+
915+-- XXX This should be monadic.
916+
615917-- | Copy a null terminated immutable 'Addr#' Word8 sequence into an array.
616918--
617919-- /Unsafe:/ The caller is responsible for safe addressing.
@@ -622,8 +924,46 @@ fromPureStream x = unsafePerformIO $ fmap (unsafeFreeze) (MA.fromPureStream x)
622924-- >>> Array.toList $ Array.fromByteStr# "\1\2\3\0"#
623925-- [1,2,3]
624926--
927+-- Note that this should be evaluated strictly to ensure that we do not hold
928+-- the reference to the pointer in a lazy thunk.
625929fromByteStr# :: Addr# -> Array Word8
626−fromByteStr# addr = fromPureStream (D.fromByteStr# addr)
930+fromByteStr# addr = unsafePerformIO $ fmap unsafeFreeze (MA.fromByteStr# addr)
931+
932+-- XXX This should be monadic.
933+
934+-- | Note that this should be evaluated strictly to ensure that we do not hold
935+-- the reference to the pointer in a lazy thunk.
936+fromByteStr :: Ptr Word8 -> Array Word8
937+fromByteStr (Ptr addr#) = fromByteStr# addr#
938+
939+-- XXX implement fromChunks/fromChunkList instead?
940+
941+-- | Convert an array stream to an array. Note that this requires peak memory
942+-- that is double the size of the array stream.
943+--
944+{-# INLINE fromChunksK #-}
945+fromChunksK :: (MonadIO m, Unbox a) => StreamK m (Array a) -> m (Array a)
946+fromChunksK stream =
947+ -- We buffer the entire stream and then allocate the target array of the
948+ -- same size, thus requiring double the memory.
949+ fmap unsafeFreeze $ MA.fromChunksK $ fmap unsafeThaw stream
950+
951+{-# DEPRECATED fromArrayStreamK "Please use fromChunksK instead." #-}
952+fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (Array a) -> m (Array a)
953+fromArrayStreamK = fromChunksK
954+
955+-- | Given a stream of arrays, splice them all together to generate a single
956+-- array. The stream must be /finite/.
957+--
958+{-# INLINE fromChunks #-}
959+fromChunks :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
960+fromChunks s =
961+ -- XXX Check which implementation is better
962+ -- This may also require double the memory as we double the space every
963+ -- time, when copying the last array we may have reallocated almost double
964+ -- the space required before we right size it.
965+ fmap unsafeFreeze $ MA.fromChunksRealloced (fmap unsafeThaw s)
966+ -- fromChunkStreamK $ D.toStreamK s
627967
628968-------------------------------------------------------------------------------
629969-- Instances
@@ -655,12 +995,66 @@ instance Unbox a => IsList (Array a) where
655995 {-# INLINE toList #-}
656996 toList = toList
657997
658−-- XXX we are assuming that Unboxed equality means element equality. This may
659−-- or may not be correct? arrcmp is 40% faster compared to stream equality.
660−instance (Unbox a, Eq a) => Eq (Array a) where
998+-- | Byte compare two arrays. Compare the length of the arrays. If the length
999+-- is equal, compare the lexicographical ordering of two underlying byte arrays
1000+-- otherwise return the result of length comparison.
1001+--
1002+-- /Unsafe/: Note that the 'Unbox' instance of sum types with constructors of
1003+-- different sizes may leave some memory uninitialized which can make byte
1004+-- comparison unreliable.
1005+--
1006+-- /Pre-release/
1007+{-# INLINE byteCmp #-}
1008+byteCmp :: Array a -> Array a -> Ordering
1009+byteCmp arr1 arr2 =
1010+ -- unsafePerformIO?
1011+ unsafeInlineIO $! unsafeThaw arr1 `MA.byteCmp` unsafeThaw arr2
1012+
1013+-- | Byte equality of two arrays.
1014+--
1015+-- >>> byteEq arr1 arr2 = (==) EQ $ Array.byteCmp arr1 arr2
1016+--
1017+-- /Unsafe/: See 'byteCmp'.
1018+{-# INLINE byteEq #-}
1019+byteEq :: Array a -> Array a -> Bool
1020+byteEq arr1 arr2 = (==) EQ $ byteCmp arr1 arr2
1021+
1022+#define MK_EQ_INSTANCE(typ) \
1023+instance {-# OVERLAPPING #-} Eq (Array typ) where { \
1024+; {-# INLINE (==) #-} \
1025+; (==) = byteEq \
1026+}
1027+
1028+MK_EQ_INSTANCE(Char)
1029+MK_EQ_INSTANCE(Word8)
1030+MK_EQ_INSTANCE(Word16)
1031+MK_EQ_INSTANCE(Word32)
1032+
1033+-- XXX The Word64 default instance should be as fast because we are comparing
1034+-- 64-bit at a time.
1035+MK_EQ_INSTANCE(Word64)
1036+MK_EQ_INSTANCE(Int)
1037+MK_EQ_INSTANCE(Int8)
1038+MK_EQ_INSTANCE(Int16)
1039+MK_EQ_INSTANCE(Int32)
1040+
1041+-- XXX The Int64 default instance should be as fast.
1042+MK_EQ_INSTANCE(Int64)
1043+
1044+-- | If the type allows a byte-by-byte comparison this instance can be
1045+-- overlapped by a more specific instance that uses 'byteCmp'. Byte comparison
1046+-- can be significantly faster.
1047+--
1048+instance {-# OVERLAPPABLE #-} (Unbox a, Eq a) => Eq (Array a) where
6611049 {-# INLINE (==) #-}
6621050 arr1 == arr2 =
663− (==) EQ $ unsafeInlineIO $! unsafeThaw arr1 `MA.cmp` unsafeThaw arr2
1051+ -- Does unboxed byte equality mean element equality?
1052+ -- XXX This is incorrect for sum types, as we may have some
1053+ -- uninitialized memory in that case. If we always initialize the
1054+ -- unused memory to zero we can use this.
1055+ -- Byte comparison is 40% faster compared to stream equality.
1056+ -- (==) EQ $ unsafeInlineIO $! unsafeThaw arr1 `MA.cmp` unsafeThaw arr2
1057+ (toStreamD arr1 :: Stream Identity a) == toStreamD arr2
6641058
6651059instance (Unbox a, Ord a) => Ord (Array a) where
6661060 {-# INLINE compare #-}
@@ -702,7 +1096,7 @@ _toStreamD_ size Array{..} = D.Stream step arrStart
7021096 {-# INLINE_LATE step #-}
7031097 step _ p | p == arrEnd = return D.Stop
7041098 step _ p = liftIO $ do
705− x <- peekByteIndex p arrContents
1099+ x <- peekAt p arrContents
7061100 return $ D.Yield x (p + size)
7071101
7081102{-
@@ -725,15 +1119,29 @@ instance Foldable Array where
7251119-- Semigroup and Monoid
7261120-------------------------------------------------------------------------------
7271121
1122+-- XXX Deprecate and remove the Semigroup and Monoid instances because of
1123+-- potential misuse chances.
1124+
1125+-- | This should not be used for combining many or N arrays as it would copy
1126+-- the two arrays everytime to a new array. For coalescing multiple arrays use
1127+-- 'fromChunksK' instead.
7281128instance Unbox a => Semigroup (Array a) where
7291129 arr1 <> arr2 = unsafePerformIO $ splice arr1 arr2
7301130
1131+empty ::
1132+#ifdef DEVBUILD
1133+ Unbox a =>
1134+#endif
1135+ Array a
1136+empty = Array Unboxed.empty 0 0
1137+
1138+{-# DEPRECATED nil "Please use empty instead." #-}
7311139nil ::
7321140#ifdef DEVBUILD
7331141 Unbox a =>
7341142#endif
7351143 Array a
736−nil = Array Unboxed.nil 0 0
1144+nil = empty
7371145
7381146instance Unbox a => Monoid (Array a) where
7391147 mempty = nil
core/src/Streamly/Internal/Serialize/FromBytes.hs → core/src/Streamly/Internal/Data/Binary/Parser.hsrenamed+8−2
@@ -1,13 +1,19 @@
11-- |
2−-- Module : Streamly.Internal.Serialize.FromBytes
2+-- Module : Streamly.Internal.Data.Binary.Parser
33-- Copyright : (c) 2020 Composewell Technologies
44-- License : BSD-3-Clause
55-- Maintainer : streamly@composewell.com
66-- Portability : GHC
77--
88-- Decode Haskell data types from byte streams.
9+--
10+-- It would be inefficient to use this to compose parsers for general algebraic
11+-- data types. For general deserialization of ADTs please use the Serialize
12+-- type class instances. The fastest way to deserialize byte streams
13+-- representing Haskell data types is to write them to arrays and deserialize
14+-- the array using the Serialize type class.
915
10−module Streamly.Internal.Serialize.FromBytes
16+module Streamly.Internal.Data.Binary.Parser
1117 (
1218 -- * Type class
1319 FromBytes (..)
core/src/Streamly/Internal/Serialize/ToBytes.hs → core/src/Streamly/Internal/Data/Binary/Stream.hsrenamed+2−2
@@ -1,5 +1,5 @@
11-- |
2−-- Module : Streamly.Internal.Serialize.ToBytes
2+-- Module : Streamly.Internal.Data.Binary.Stream
33-- Copyright : (c) 2022 Composewell Technologies
44-- License : BSD-3-Clause
55-- Maintainer : streamly@composewell.com
@@ -17,7 +17,7 @@
1717
1818-- XXX remove unit, bool, ordering, and the type class as well
1919
20−module Streamly.Internal.Serialize.ToBytes
20+module Streamly.Internal.Data.Binary.Stream
2121 (
2222 -- * Type class
2323 ToBytes (..)
core/src/Streamly/Internal/Data/Fold/Chunked.hs+16−15
@@ -6,7 +6,7 @@
66-- Stability : experimental
77-- Portability : GHC
88--
9−-- Use "Streamly.Data.Parser.Chunked" instead.
9+-- Use "Streamly.Data.Parser" instead.
1010--
1111-- Fold a stream of foreign arrays. @Fold m a b@ in this module works
1212-- on a stream of "Array a" and produces an output of type @b@.
@@ -19,18 +19,19 @@
1919-- folds in Data.Fold to correctly work on an array stream as if it is an
2020-- element stream. For example:
2121--
22−-- >>> import qualified Streamly.Data.Fold as Fold
23−-- >>> import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
24−-- >>> import qualified Streamly.Internal.Data.Fold.Chunked as ChunkFold
25−-- >>> import qualified Streamly.Data.Stream as Stream
26−-- >>> import qualified Streamly.Data.StreamK as StreamK
22+-- >> import qualified Streamly.Data.Fold as Fold
23+-- >> import qualified Streamly.Internal.Data.Array.Stream as ArrayStream
24+-- >> import qualified Streamly.Internal.Data.Fold.Chunked as ChunkFold
25+-- >> import qualified Streamly.Data.Stream as Stream
26+-- >> import qualified Streamly.Data.StreamK as StreamK
2727--
28−-- >>> f = ChunkFold.fromFold (Fold.take 7 Fold.toList)
29−-- >>> s = Stream.chunksOf 5 $ Stream.fromList "hello world"
30−-- >>> ArrayStream.runArrayFold f (StreamK.fromStream s)
28+-- >> f = ChunkFold.fromFold (Fold.take 7 Fold.toList)
29+-- >> s = Stream.chunksOf 5 $ Stream.fromList "hello world"
30+-- >> ArrayStream.runArrayFold f (StreamK.fromStream s)
3131-- Right "hello w"
3232--
3333module Streamly.Internal.Data.Fold.Chunked
34+ {-# DEPRECATED "Please use Streamly.Data.Parser instead." #-}
3435 (
3536 ChunkFold (..)
3637
@@ -67,7 +68,6 @@ import Data.Bifunctor (first)
6768import Data.Proxy (Proxy(..))
6869import Streamly.Internal.Data.Unbox (Unbox(..))
6970import GHC.Types (SPEC(..))
70−import Streamly.Internal.Data.MutArray.Type (touch)
7171import Streamly.Internal.Data.Array (Array(..))
7272import Streamly.Internal.Data.Parser (Initial(..), Step(..))
7373import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
@@ -103,8 +103,8 @@ newtype ChunkFold m a b = ChunkFold (ParserD.Parser (Array a) m b)
103103{-# INLINE fromFold #-}
104104fromFold :: forall m a b. (MonadIO m, Unbox a) =>
105105 Fold.Fold m a b -> ChunkFold m a b
106−fromFold (Fold.Fold fstep finitial fextract) =
107− ChunkFold (ParserD.Parser step initial (fmap (Done 0) . fextract))
106+fromFold (Fold.Fold fstep finitial _ ffinal) =
107+ ChunkFold (ParserD.Parser step initial extract)
108108
109109 where
110110
@@ -124,7 +124,7 @@ fromFold (Fold.Fold fstep finitial fextract) =
124124 assert (cur == end) (return ())
125125 return $ Partial 0 fs
126126 goArray !_ !cur !fs = do
127− x <- liftIO $ peekByteIndex cur contents
127+ x <- liftIO $ peekAt cur contents
128128 res <- fstep fs x
129129 let elemSize = SIZE_OF(a)
130130 next = INDEX_NEXT(cur,a)
@@ -134,6 +134,8 @@ fromFold (Fold.Fold fstep finitial fextract) =
134134 Fold.Partial fs1 ->
135135 goArray SPEC next fs1
136136
137+ extract = fmap (Done 0) . ffinal
138+
137139-- | Convert an element 'ParserD.Parser' into an array stream fold. If the
138140-- parser fails the fold would throw an exception.
139141--
@@ -161,8 +163,7 @@ fromParserD (ParserD.Parser step1 initial1 extract1) =
161163 else return $ st (arrRem + n) fs1
162164
163165 goArray !_ !cur !fs = do
164− x <- liftIO $ peekByteIndex cur contents
165− liftIO $ touch contents
166+ x <- liftIO $ peekAt cur contents
166167 res <- step1 fs x
167168 let elemSize = SIZE_OF(a)
168169 next = INDEX_NEXT(cur,a)
core/src/Streamly/Internal/Data/Fold/Combinators.hs+81−137
@@ -30,8 +30,6 @@ module Streamly.Internal.Data.Fold.Combinators
3030 -- *** Reducers
3131 , drainMapM
3232 , the
33− , length
34− , lengthGeneric
3533 , mean
3634 , rollingHash
3735 , defaultSalt
@@ -166,8 +164,6 @@ module Streamly.Internal.Data.Fold.Combinators
166164 -- ** Trimming
167165
168166 -- By elements
169− , takeEndBy
170− , takeEndBy_
171167 , takeEndBySeq
172168 , takeEndBySeq_
173169 {-
@@ -443,8 +439,8 @@ trace f = lmapM (tracing f)
443439-- /Pre-release/
444440{-# INLINE transform #-}
445441transform :: Monad m => Pipe m a b -> Fold m b c -> Fold m a c
446−transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract) =
447− Fold step initial extract
442+transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract ffinal) =
443+ Fold step initial extract final
448444
449445 where
450446
@@ -477,10 +473,14 @@ transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract) =
477473
478474 extract (Tuple' _ fs) = fextract fs
479475
476+ final (Tuple' _ fs) = ffinal fs
477+
480478{-# INLINE scanWith #-}
481479scanWith :: Monad m => Bool -> Fold m a b -> Fold m b c -> Fold m a c
482−scanWith isMany (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
483− Fold step initial extract
480+scanWith isMany
481+ (Fold stepL initialL extractL finalL)
482+ (Fold stepR initialR extractR finalR) =
483+ Fold step initial extract final
484484
485485 where
486486
@@ -494,15 +494,14 @@ scanWith isMany (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
494494 Partial sR1 ->
495495 if isMany
496496 then runStep initialL sR1
497− else Done <$> extractR sR1
497+ else Done <$> finalR sR1
498498 Done bR -> return $ Done bR
499499 Partial sL -> do
500500 !b <- extractL sL
501501 rR <- stepR sR b
502− return
503− $ case rR of
504− Partial sR1 -> Partial (sL, sR1)
505− Done bR -> Done bR
502+ case rR of
503+ Partial sR1 -> return $ Partial (sL, sR1)
504+ Done bR -> finalL sL >> return (Done bR)
506505
507506 initial = do
508507 r <- initialR
@@ -514,6 +513,8 @@ scanWith isMany (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
514513
515514 extract = extractR . snd
516515
516+ final (sL, sR) = finalL sL *> finalR sR
517+
517518-- | Scan the input of a 'Fold' to change it in a stateful manner using another
518519-- 'Fold'. The scan stops as soon as the fold terminates.
519520--
@@ -565,7 +566,7 @@ deleteBy eq x0 = fmap extract $ foldl' step (Tuple' False Nothing)
565566--
566567{-# INLINE slide2 #-}
567568slide2 :: Monad m => Fold m (a, Maybe a) b -> Fold m a b
568−slide2 (Fold step1 initial1 extract1) = Fold step initial extract
569+slide2 (Fold step1 initial1 extract1 final1) = Fold step initial extract final
569570
570571 where
571572
@@ -577,6 +578,8 @@ slide2 (Fold step1 initial1 extract1) = Fold step initial extract
577578
578579 extract (Tuple' _ s) = extract1 s
579580
581+ final (Tuple' _ s) = final1 s
582+
580583-- | Return the latest unique element using the supplied comparison function.
581584-- Returns 'Nothing' if the current element is same as the last element
582585-- otherwise returns 'Just'.
@@ -702,30 +705,6 @@ the = foldt' step initial id
702705-- To Summary
703706------------------------------------------------------------------------------
704707
705−-- | Like 'length', except with a more general 'Num' return value
706−--
707−-- Definition:
708−--
709−-- >>> lengthGeneric = fmap getSum $ Fold.foldMap (Sum . const 1)
710−-- >>> lengthGeneric = Fold.foldl' (\n _ -> n + 1) 0
711−--
712−-- /Pre-release/
713−{-# INLINE lengthGeneric #-}
714−lengthGeneric :: (Monad m, Num b) => Fold m a b
715−lengthGeneric = foldl' (\n _ -> n + 1) 0
716−
717−-- | Determine the length of the input stream.
718−--
719−-- Definition:
720−--
721−-- >>> length = Fold.lengthGeneric
722−-- >>> length = fmap getSum $ Fold.foldMap (Sum . const 1)
723−--
724−{-# INLINE length #-}
725−length :: Monad m => Fold m a Int
726−length = lengthGeneric
727−
728−
729708-- | Determine the sum of all elements of a stream of numbers. Returns additive
730709-- identity (@0@) when the stream is empty. Note that this is not numerically
731710-- stable for floating point numbers.
@@ -932,7 +911,7 @@ rollingHashFirstN n = take n rollingHash
932911--
933912{-# INLINE rollingMapM #-}
934913rollingMapM :: Monad m => (Maybe a -> a -> m b) -> Fold m a b
935−rollingMapM f = Fold step initial extract
914+rollingMapM f = Fold step initial extract extract
936915
937916 where
938917
@@ -1173,7 +1152,8 @@ head = one
11731152-- /Pre-release/
11741153{-# INLINE findM #-}
11751154findM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1176−findM predicate = Fold step (return $ Partial ()) (const $ return Nothing)
1155+findM predicate =
1156+ Fold step (return $ Partial ()) extract extract
11771157
11781158 where
11791159
@@ -1184,6 +1164,8 @@ findM predicate = Fold step (return $ Partial ()) (const $ return Nothing)
11841164 else Partial ()
11851165 in f <$> predicate a
11861166
1167+ extract = const $ return Nothing
1168+
11871169-- | Returns the first element that satisfies the given predicate.
11881170--
11891171{-# INLINE find #-}
@@ -1426,7 +1408,7 @@ splitAt n fld = splitWith (,) (take n fld)
14261408
14271409{-# INLINE takingEndByM #-}
14281410takingEndByM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1429−takingEndByM p = Fold step initial (return . toMaybe)
1411+takingEndByM p = Fold step initial extract extract
14301412
14311413 where
14321414
@@ -1439,6 +1421,8 @@ takingEndByM p = Fold step initial (return . toMaybe)
14391421 then Done $ Just a
14401422 else Partial $ Just' a
14411423
1424+ extract = return . toMaybe
1425+
14421426-- |
14431427--
14441428-- >>> takingEndBy p = Fold.takingEndByM (return . p)
@@ -1449,7 +1433,7 @@ takingEndBy p = takingEndByM (return . p)
14491433
14501434{-# INLINE takingEndByM_ #-}
14511435takingEndByM_ :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1452−takingEndByM_ p = Fold step initial (return . toMaybe)
1436+takingEndByM_ p = Fold step initial extract extract
14531437
14541438 where
14551439
@@ -1462,6 +1446,8 @@ takingEndByM_ p = Fold step initial (return . toMaybe)
14621446 then Done Nothing
14631447 else Partial $ Just' a
14641448
1449+ extract = return . toMaybe
1450+
14651451-- |
14661452--
14671453-- >>> takingEndBy_ p = Fold.takingEndByM_ (return . p)
@@ -1472,7 +1458,7 @@ takingEndBy_ p = takingEndByM_ (return . p)
14721458
14731459{-# INLINE droppingWhileM #-}
14741460droppingWhileM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1475−droppingWhileM p = Fold step initial (return . toMaybe)
1461+droppingWhileM p = Fold step initial extract extract
14761462
14771463 where
14781464
@@ -1487,6 +1473,8 @@ droppingWhileM p = Fold step initial (return . toMaybe)
14871473 else Just' a
14881474 step _ a = return $ Partial $ Just' a
14891475
1476+ extract = return . toMaybe
1477+
14901478-- |
14911479-- >>> droppingWhile p = Fold.droppingWhileM (return . p)
14921480--
@@ -1494,70 +1482,6 @@ droppingWhileM p = Fold step initial (return . toMaybe)
14941482droppingWhile :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
14951483droppingWhile p = droppingWhileM (return . p)
14961484
1497−-- Note: Keep this consistent with S.splitOn. In fact we should eliminate
1498−-- S.splitOn in favor of the fold.
1499−--
1500−-- XXX Use Fold.many instead once it is fixed.
1501−-- > Stream.splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
1502−
1503−-- | Like 'takeEndBy' but drops the element on which the predicate succeeds.
1504−--
1505−-- Example:
1506−--
1507−-- >>> input = Stream.fromList "hello\nthere\n"
1508−-- >>> line = Fold.takeEndBy_ (== '\n') Fold.toList
1509−-- >>> Stream.fold line input
1510−-- "hello"
1511−--
1512−-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1513−-- ["hello","there"]
1514−--
1515−{-# INLINE takeEndBy_ #-}
1516−takeEndBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1517−-- takeEndBy_ predicate = scanMaybe (takingEndBy_ predicate)
1518−takeEndBy_ predicate (Fold fstep finitial fextract) =
1519− Fold step finitial fextract
1520−
1521− where
1522−
1523− step s a =
1524− if not (predicate a)
1525− then fstep s a
1526− else Done <$> fextract s
1527−
1528−-- Note:
1529−-- > Stream.splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
1530−
1531−-- | Take the input, stop when the predicate succeeds taking the succeeding
1532−-- element as well.
1533−--
1534−-- Example:
1535−--
1536−-- >>> input = Stream.fromList "hello\nthere\n"
1537−-- >>> line = Fold.takeEndBy (== '\n') Fold.toList
1538−-- >>> Stream.fold line input
1539−-- "hello\n"
1540−--
1541−-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1542−-- ["hello\n","there\n"]
1543−--
1544−{-# INLINE takeEndBy #-}
1545−takeEndBy :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1546−-- takeEndBy predicate = scanMaybe (takingEndBy predicate)
1547−takeEndBy predicate (Fold fstep finitial fextract) =
1548− Fold step finitial fextract
1549−
1550− where
1551−
1552− step s a = do
1553− res <- fstep s a
1554− if not (predicate a)
1555− then return res
1556− else do
1557− case res of
1558− Partial s1 -> Done <$> fextract s1
1559− Done b -> return $ Done b
1560−
15611485------------------------------------------------------------------------------
15621486-- Binary splitting on a separator
15631487------------------------------------------------------------------------------
@@ -1590,8 +1514,8 @@ takeEndBySeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) =>
15901514 Array.Array a
15911515 -> Fold m a b
15921516 -> Fold m a b
1593−takeEndBySeq patArr (Fold fstep finitial fextract) =
1594− Fold step initial extract
1517+takeEndBySeq patArr (Fold fstep finitial fextract ffinal) =
1518+ Fold step initial extract final
15951519
15961520 where
15971521
@@ -1604,7 +1528,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16041528 | patLen == 0 ->
16051529 -- XXX Should we match nothing or everything on empty
16061530 -- pattern?
1607− -- Done <$> fextract acc
1531+ -- Done <$> ffinal acc
16081532 return $ Partial $ SplitOnSeqEmpty acc
16091533 | patLen == 1 -> do
16101534 pat <- liftIO $ Array.unsafeIndexIO 0 patArr
@@ -1655,7 +1579,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16551579 case res of
16561580 Partial s1
16571581 | pat /= x -> return $ Partial $ SplitOnSeqSingle s1 pat
1658− | otherwise -> Done <$> fextract s1
1582+ | otherwise -> Done <$> ffinal s1
16591583 Done b -> return $ Done b
16601584 step (SplitOnSeqWord s idx wrd) x = do
16611585 res <- fstep s x
@@ -1664,7 +1588,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16641588 Partial s1
16651589 | idx == maxIndex -> do
16661590 if wrd1 .&. wordMask == wordPat
1667− then Done <$> fextract s1
1591+ then Done <$> ffinal s1
16681592 else return $ Partial $ SplitOnSeqWordLoop s1 wrd1
16691593 | otherwise ->
16701594 return $ Partial $ SplitOnSeqWord s1 (idx + 1) wrd1
@@ -1675,7 +1599,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16751599 case res of
16761600 Partial s1
16771601 | wrd1 .&. wordMask == wordPat ->
1678− Done <$> fextract s1
1602+ Done <$> ffinal s1
16791603 | otherwise ->
16801604 return $ Partial $ SplitOnSeqWordLoop s1 wrd1
16811605 Done b -> return $ Done b
@@ -1689,7 +1613,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
16891613 let fld = Ring.unsafeFoldRing (Ring.ringBound rb)
16901614 let !ringHash = fld addCksum 0 rb
16911615 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1692− then Done <$> fextract s1
1616+ then Done <$> ffinal s1
16931617 else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
16941618 else
16951619 return $ Partial $ SplitOnSeqKR s1 (idx + 1) rb rh1
@@ -1702,11 +1626,11 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
17021626 rh1 <- liftIO $ Ring.unsafeInsert rb rh x
17031627 let ringHash = deltaCksum cksum old x
17041628 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1705− then Done <$> fextract s1
1629+ then Done <$> ffinal s1
17061630 else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
17071631 Done b -> return $ Done b
17081632
1709− extract state =
1633+ extractFunc fex state =
17101634 let st =
17111635 case state of
17121636 SplitOnSeqEmpty s -> s
@@ -1715,7 +1639,11 @@ takeEndBySeq patArr (Fold fstep finitial fextract) =
17151639 SplitOnSeqWordLoop s _ -> s
17161640 SplitOnSeqKR s _ _ _ -> s
17171641 SplitOnSeqKRLoop s _ _ _ -> s
1718− in fextract st
1642+ in fex st
1643+
1644+ extract state = extractFunc fextract state
1645+
1646+ final state = extractFunc ffinal state
17191647
17201648-- | Like 'takeEndBySeq' but discards the matched sequence.
17211649--
@@ -1726,8 +1654,8 @@ takeEndBySeq_ :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) =>
17261654 Array.Array a
17271655 -> Fold m a b
17281656 -> Fold m a b
1729−takeEndBySeq_ patArr (Fold fstep finitial fextract) =
1730− Fold step initial extract
1657+takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
1658+ Fold step initial extract final
17311659
17321660 where
17331661
@@ -1740,7 +1668,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
17401668 | patLen == 0 ->
17411669 -- XXX Should we match nothing or everything on empty
17421670 -- pattern?
1743− -- Done <$> fextract acc
1671+ -- Done <$> ffinal acc
17441672 return $ Partial $ SplitOnSeqEmpty acc
17451673 | patLen == 1 -> do
17461674 pat <- liftIO $ Array.unsafeIndexIO 0 patArr
@@ -1797,13 +1725,13 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
17971725 case res of
17981726 Partial s1 -> return $ Partial $ SplitOnSeqSingle s1 pat
17991727 Done b -> return $ Done b
1800− else Done <$> fextract s
1728+ else Done <$> ffinal s
18011729 step (SplitOnSeqWord s idx wrd) x = do
18021730 let wrd1 = addToWord wrd x
18031731 if idx == maxIndex
18041732 then do
18051733 if wrd1 .&. wordMask == wordPat
1806− then Done <$> fextract s
1734+ then Done <$> ffinal s
18071735 else return $ Partial $ SplitOnSeqWordLoop s wrd1
18081736 else return $ Partial $ SplitOnSeqWord s (idx + 1) wrd1
18091737 step (SplitOnSeqWordLoop s wrd) x = do
@@ -1814,7 +1742,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18141742 case res of
18151743 Partial s1
18161744 | wrd1 .&. wordMask == wordPat ->
1817− Done <$> fextract s1
1745+ Done <$> ffinal s1
18181746 | otherwise ->
18191747 return $ Partial $ SplitOnSeqWordLoop s1 wrd1
18201748 Done b -> return $ Done b
@@ -1825,7 +1753,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18251753 let fld = Ring.unsafeFoldRing (Ring.ringBound rb)
18261754 let !ringHash = fld addCksum 0 rb
18271755 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1828− then Done <$> fextract s
1756+ then Done <$> ffinal s
18291757 else return $ Partial $ SplitOnSeqKRLoop s ringHash rb rh1
18301758 else return $ Partial $ SplitOnSeqKR s (idx + 1) rb rh1
18311759 step (SplitOnSeqKRLoop s cksum rb rh) x = do
@@ -1836,7 +1764,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18361764 rh1 <- liftIO $ Ring.unsafeInsert rb rh x
18371765 let ringHash = deltaCksum cksum old x
18381766 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1839− then Done <$> fextract s1
1767+ then Done <$> ffinal s1
18401768 else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
18411769 Done b -> return $ Done b
18421770
@@ -1844,10 +1772,10 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18441772 -- terminates early inside extract, we may still have buffered data
18451773 -- remaining which will be lost if we do not communicate that to the
18461774 -- driver.
1847− extract state = do
1775+ extractFunc fex state = do
18481776 let consumeWord s n wrd = do
18491777 if n == 0
1850− then fextract s
1778+ then fex s
18511779 else do
18521780 let old = elemMask .&. (wrd `shiftR` (elemBits * (n - 1)))
18531781 r <- fstep s (toEnum $ fromIntegral old)
@@ -1857,7 +1785,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18571785
18581786 let consumeRing s n rb rh =
18591787 if n == 0
1860− then fextract s
1788+ then fex s
18611789 else do
18621790 old <- liftIO $ peek rh
18631791 let rh1 = Ring.advance rb rh
@@ -1867,13 +1795,17 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract) =
18671795 Done b -> return b
18681796
18691797 case state of
1870− SplitOnSeqEmpty s -> fextract s
1871− SplitOnSeqSingle s _ -> fextract s
1798+ SplitOnSeqEmpty s -> fex s
1799+ SplitOnSeqSingle s _ -> fex s
18721800 SplitOnSeqWord s idx wrd -> consumeWord s idx wrd
18731801 SplitOnSeqWordLoop s wrd -> consumeWord s patLen wrd
18741802 SplitOnSeqKR s idx rb _ -> consumeRing s idx rb (Ring.startOf rb)
18751803 SplitOnSeqKRLoop s _ rb rh -> consumeRing s patLen rb rh
18761804
1805+ extract state = extractFunc fextract state
1806+
1807+ final state = extractFunc ffinal state
1808+
18771809------------------------------------------------------------------------------
18781810-- Distributing
18791811------------------------------------------------------------------------------
@@ -2317,8 +2249,8 @@ toStreamRev = fmap StreamD.fromList toListRev
23172249-- /Pre-release/
23182250{-# INLINE unfoldMany #-}
23192251unfoldMany :: Monad m => Unfold m a b -> Fold m b c -> Fold m a c
2320−unfoldMany (Unfold ustep inject) (Fold fstep initial extract) =
2321− Fold consume initial extract
2252+unfoldMany (Unfold ustep inject) (Fold fstep initial extract final) =
2253+ Fold consume initial extract final
23222254
23232255 where
23242256
@@ -2345,7 +2277,7 @@ bottomBy :: (MonadIO m, Unbox a) =>
23452277 (a -> a -> Ordering)
23462278 -> Int
23472279 -> Fold m a (MutArray a)
2348−bottomBy cmp n = Fold step initial extract
2280+bottomBy cmp n = Fold step initial extract extract
23492281
23502282 where
23512283
@@ -2440,8 +2372,8 @@ intersperseWithQuotes
24402372 quote
24412373 esc
24422374 separator
2443− (Fold stepL initialL extractL)
2444− (Fold stepR initialR extractR) = Fold step initial extract
2375+ (Fold stepL initialL _ finalL)
2376+ (Fold stepR initialR extractR finalR) = Fold step initial extract final
24452377
24462378 where
24472379
@@ -2484,11 +2416,13 @@ intersperseWithQuotes
24842416 r <- stepL sL a
24852417 case r of
24862418 Partial s -> return $ Partial (nextState sR s)
2487− Done _ -> error "Collecting fold finished inside quote"
2419+ Done _ -> do
2420+ _ <- finalR sR
2421+ error "Collecting fold finished inside quote"
24882422
24892423 step (IntersperseQUnquoted sR sL) a
24902424 | a == separator = do
2491− b <- extractL sL
2425+ b <- finalL sL
24922426 collect IntersperseQUnquoted sR b
24932427 | a == quote = processQuoted a sL sR IntersperseQQuoted
24942428 | otherwise = process a sL sR IntersperseQUnquoted
@@ -2506,3 +2440,13 @@ intersperseWithQuotes
25062440 error "intersperseWithQuotes: finished inside quote"
25072441 extract (IntersperseQQuotedEsc _ _) =
25082442 error "intersperseWithQuotes: finished inside quote, at escape char"
2443+
2444+ final (IntersperseQUnquoted sR sL) = finalL sL *> finalR sR
2445+ final (IntersperseQQuoted sR sL) = do
2446+ _ <- finalR sR
2447+ _ <- finalL sL
2448+ error "intersperseWithQuotes: finished inside quote"
2449+ final (IntersperseQQuotedEsc sR sL) = do
2450+ _ <- finalR sR
2451+ _ <- finalL sL
2452+ error "intersperseWithQuotes: finished inside quote, at escape char"
core/src/Streamly/Internal/Data/Fold/Container.hs+73−28
@@ -270,14 +270,15 @@ demuxGeneric :: (Monad m, IsMap f, Traversable f) =>
270270 (a -> Key f)
271271 -> (a -> m (Fold m a b))
272272 -> Fold m a (m (f b), Maybe (Key f, b))
273−demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
273+demuxGeneric getKey getFold =
274+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
274275
275276 where
276277
277278 initial = return $ Tuple' IsMap.mapEmpty Nothing
278279
279280 {-# INLINE runFold #-}
280− runFold kv (Fold step1 initial1 extract1) (k, a) = do
281+ runFold kv (Fold step1 initial1 extract1 final1) (k, a) = do
281282 res <- initial1
282283 case res of
283284 Partial s -> do
@@ -285,7 +286,7 @@ demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
285286 return
286287 $ case res1 of
287288 Partial _ ->
288− let fld = Fold step1 (return res1) extract1
289+ let fld = Fold step1 (return res1) extract1 final1
289290 in Tuple' (IsMap.mapInsert k fld kv) Nothing
290291 Done b -> Tuple' (IsMap.mapDelete k kv) (Just (k, b))
291292 Done b -> return $ Tuple' kv (Just (k, b))
@@ -298,15 +299,25 @@ demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
298299 runFold kv fld (k, a)
299300 Just f -> runFold kv f (k, a)
300301
301− extract (Tuple' kv x) = (Prelude.mapM f kv, x)
302+ extract (Tuple' kv x) = return (Prelude.mapM f kv, x)
302303
303304 where
304305
305− f (Fold _ i e) = do
306+ f (Fold _ i e _) = do
306307 r <- i
307308 case r of
308309 Partial s -> e s
309− Done b -> return b
310+ _ -> error "demuxGeneric: unreachable code"
311+
312+ final (Tuple' kv x) = return (Prelude.mapM f kv, x)
313+
314+ where
315+
316+ f (Fold _ i _ fin) = do
317+ r <- i
318+ case r of
319+ Partial s -> fin s
320+ _ -> error "demuxGeneric: unreachable code"
310321
311322-- | @demux getKey getFold@: In a key value stream, fold values corresponding
312323-- to each key using a key specific fold. @getFold@ is invoked to generate a
@@ -348,35 +359,36 @@ demuxGenericIO :: (MonadIO m, IsMap f, Traversable f) =>
348359 (a -> Key f)
349360 -> (a -> m (Fold m a b))
350361 -> Fold m a (m (f b), Maybe (Key f, b))
351−demuxGenericIO getKey getFold = fmap extract $ foldlM' step initial
362+demuxGenericIO getKey getFold =
363+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
352364
353365 where
354366
355367 initial = return $ Tuple' IsMap.mapEmpty Nothing
356368
357369 {-# INLINE initFold #-}
358− initFold kv (Fold step1 initial1 extract1) (k, a) = do
370+ initFold kv (Fold step1 initial1 extract1 final1) (k, a) = do
359371 res <- initial1
360372 case res of
361373 Partial s -> do
362374 res1 <- step1 s a
363375 case res1 of
364376 Partial _ -> do
365− let fld = Fold step1 (return res1) extract1
377+ let fld = Fold step1 (return res1) extract1 final1
366378 ref <- liftIO $ newIORef fld
367379 return $ Tuple' (IsMap.mapInsert k ref kv) Nothing
368380 Done b -> return $ Tuple' kv (Just (k, b))
369381 Done b -> return $ Tuple' kv (Just (k, b))
370382
371383 {-# INLINE runFold #-}
372− runFold kv ref (Fold step1 initial1 extract1) (k, a) = do
384+ runFold kv ref (Fold step1 initial1 extract1 final1) (k, a) = do
373385 res <- initial1
374386 case res of
375387 Partial s -> do
376388 res1 <- step1 s a
377389 case res1 of
378390 Partial _ -> do
379− let fld = Fold step1 (return res1) extract1
391+ let fld = Fold step1 (return res1) extract1 final1
380392 liftIO $ writeIORef ref fld
381393 return $ Tuple' kv Nothing
382394 Done b ->
@@ -394,16 +406,27 @@ demuxGenericIO getKey getFold = fmap extract $ foldlM' step initial
394406 f <- liftIO $ readIORef ref
395407 runFold kv ref f (k, a)
396408
397− extract (Tuple' kv x) = (Prelude.mapM f kv, x)
409+ extract (Tuple' kv x) = return (Prelude.mapM f kv, x)
398410
399411 where
400412
401413 f ref = do
402− (Fold _ i e) <- liftIO $ readIORef ref
414+ Fold _ i e _ <- liftIO $ readIORef ref
403415 r <- i
404416 case r of
405417 Partial s -> e s
406− Done b -> return b
418+ _ -> error "demuxGenericIO: unreachable code"
419+
420+ final (Tuple' kv x) = return (Prelude.mapM f kv, x)
421+
422+ where
423+
424+ f ref = do
425+ Fold _ i _ fin <- liftIO $ readIORef ref
426+ r <- i
427+ case r of
428+ Partial s -> fin s
429+ _ -> error "demuxGenericIO: unreachable code"
407430
408431-- | This is specialized version of 'demux' that uses mutable IO cells as
409432-- fold accumulators for better performance.
@@ -418,6 +441,13 @@ demuxIO :: (MonadIO m, Ord k) =>
418441 -> Fold m a (m (Map k b), Maybe (k, b))
419442demuxIO = demuxGenericIO
420443
444+-- | Fold a key value stream to a key-value Map. If the same key appears
445+-- multiple times, only the last value is retained.
446+{-# INLINE kvToMapOverwriteGeneric #-}
447+kvToMapOverwriteGeneric :: (Monad m, IsMap f) => Fold m (Key f, a) (f a)
448+kvToMapOverwriteGeneric =
449+ foldl' (\kv (k, v) -> IsMap.mapInsert k v kv) IsMap.mapEmpty
450+
421451{-# INLINE demuxToContainer #-}
422452demuxToContainer :: (Monad m, IsMap f, Traversable f) =>
423453 (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
@@ -505,8 +535,8 @@ classifyGeneric :: (Monad m, IsMap f, Traversable f, Ord (Key f)) =>
505535 -- for that use case. We return an action because we want it to be lazy so
506536 -- that the downstream consumers can choose to process or discard it.
507537 (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
508−classifyGeneric f (Fold step1 initial1 extract1) =
509− fmap extract $ foldlM' step initial
538+classifyGeneric f (Fold step1 initial1 extract1 final1) =
539+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
510540
511541 where
512542
@@ -543,7 +573,16 @@ classifyGeneric f (Fold step1 initial1 extract1) =
543573 let kv1 = IsMap.mapDelete k kv
544574 in Tuple3' kv1 (Set.insert k set) (Just (k, b))
545575
546− extract (Tuple3' kv _ x) = (Prelude.mapM extract1 kv, x)
576+ extract (Tuple3' kv _ x) = return (Prelude.mapM extract1 kv, x)
577+
578+ final (Tuple3' kv set x) = return (IsMap.mapTraverseWithKey f1 kv, x)
579+
580+ where
581+
582+ f1 k s = do
583+ if Set.member k set
584+ then extract1 s
585+ else final1 s
547586
548587-- | Folds the values for each key using the supplied fold. When scanning, as
549588-- soon as the fold is complete, its result is available in the second
@@ -568,8 +607,8 @@ classify = classifyGeneric
568607{-# INLINE classifyGenericIO #-}
569608classifyGenericIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) =>
570609 (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
571−classifyGenericIO f (Fold step1 initial1 extract1) =
572− fmap extract $ foldlM' step initial
610+classifyGenericIO f (Fold step1 initial1 extract1 final1) =
611+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
573612
574613 where
575614
@@ -608,8 +647,21 @@ classifyGenericIO f (Fold step1 initial1 extract1) =
608647 in return
609648 $ Tuple3' kv1 (Set.insert k set) (Just (k, b))
610649
611− extract (Tuple3' kv _ x) =
612− (Prelude.mapM (\ref -> liftIO (readIORef ref) >>= extract1) kv, x)
650+ extract (Tuple3' kv _ x) = return (Prelude.mapM g kv, x)
651+
652+ where
653+
654+ g ref = liftIO (readIORef ref) >>= extract1
655+
656+ final (Tuple3' kv set x) = return (IsMap.mapTraverseWithKey g kv, x)
657+
658+ where
659+
660+ g k ref = do
661+ s <- liftIO $ readIORef ref
662+ if Set.member k set
663+ then extract1 s
664+ else final1 s
613665
614666-- | Same as classify except that it uses mutable IORef cells in the
615667-- Map providing better performance. Be aware that if this is used as a scan,
@@ -624,13 +676,6 @@ classifyIO :: (MonadIO m, Ord k) =>
624676 (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
625677classifyIO = classifyGenericIO
626678
627−-- | Fold a key value stream to a key-value Map. If the same key appears
628−-- multiple times, only the last value is retained.
629−{-# INLINE kvToMapOverwriteGeneric #-}
630−kvToMapOverwriteGeneric :: (Monad m, IsMap f) => Fold m (Key f, a) (f a)
631−kvToMapOverwriteGeneric =
632− foldl' (\kv (k, v) -> IsMap.mapInsert k v kv) IsMap.mapEmpty
633−
634679{-# INLINE toContainer #-}
635680toContainer :: (Monad m, IsMap f, Traversable f, Ord (Key f)) =>
636681 (a -> Key f) -> Fold m a b -> Fold m a (f b)
core/src/Streamly/Internal/Data/Fold/Step.hs+1−1
@@ -40,7 +40,7 @@ data Step s b
4040 = Partial !s
4141 | Done !b
4242
43−-- | 'first' maps over 'Partial' and 'second' maps over 'Done'.
43+-- | 'first' maps over the fold state and 'second' maps over the fold result.
4444--
4545instance Bifunctor Step where
4646 {-# INLINE bimap #-}
core/src/Streamly/Internal/Data/Fold/Type.hs+336−114
@@ -367,6 +367,8 @@ module Streamly.Internal.Data.Fold.Type
367367 , toList
368368 , toStreamK
369369 , toStreamKRev
370+ , lengthGeneric
371+ , length
370372
371373 -- * Combinators
372374
@@ -391,6 +393,8 @@ module Streamly.Internal.Data.Fold.Type
391393 -- ** Trimming
392394 , take
393395 , taking
396+ , takeEndBy_
397+ , takeEndBy
394398 , dropping
395399
396400 -- ** Sequential application
@@ -445,7 +449,7 @@ where
445449#if !MIN_VERSION_base(4,18,0)
446450import Control.Applicative (liftA2)
447451#endif
448−import Control.Monad ((>=>))
452+import Control.Monad ((>=>), void)
449453import Data.Bifunctor (Bifunctor(..))
450454import Data.Either (fromLeft, fromRight, isLeft, isRight)
451455import Data.Functor.Identity (Identity(..))
@@ -471,23 +475,64 @@ import Streamly.Internal.Data.Fold.Step
471475-- The type @b@ is the accumulator of the writer. That's the reason the
472476-- default folds in various modules are called "write".
473477
474−-- | The type @Fold m a b@ having constructor @Fold step initial extract@
475−-- represents a fold over an input stream of values of type @a@ to a final
476−-- value of type @b@ in 'Monad' @m@.
477−--
478−-- The fold uses an intermediate state @s@ as accumulator, the type @s@ is
479−-- internal to the specific fold definition. The initial value of the fold
480−-- state @s@ is returned by @initial@. The @step@ function consumes an input
481−-- and either returns the final result @b@ if the fold is done or the next
482−-- intermediate state (see 'Step'). At any point the fold driver can extract
483−-- the result from the intermediate state using the @extract@ function.
478+-- An alternative to using an "extract" function is to use "Partial s b" style
479+-- partial value so that we always emit the output value and there is no need
480+-- to extract. Then extract can be used for cleanup purposes. But in this case
481+-- in some cases we may need a "Continue" constructor where an output value is
482+-- not available, this was implicit earlier. Also, "b" should be lazy here so
483+-- that we do not always compute it even if we do not need it.
484+--
485+-- Partial s b --> extract :: s -> b
486+-- Continue --> extract :: s -> Maybe b
487+--
488+-- But keeping 'b' lazy does not let the fold optimize well. It leads to
489+-- significant regressions in the key-value folds.
490+--
491+-- The "final" function complicates combinators that take other folds as
492+-- argument because we need to call their finalizers at right places. An
493+-- alternative to reduce this complexity where it is not required is to use a
494+-- separate type for bracketed folds but then we need to manage the complexity
495+-- of two different fold types.
496+
497+-- The "final" function could be (s -> m (Step s b)), like in parsers so that
498+-- it can be called in a loop to drain the fold.
499+
500+-- | The type @Fold m a b@ represents a consumer of an input stream of values
501+-- of type @a@ and returning a final value of type @b@ in 'Monad' @m@. The
502+-- constructor of a fold is @Fold step initial extract final@.
503+--
504+-- The fold uses an internal state of type @s@. The initial value of the state
505+-- @s@ is created by @initial@. This function is called once and only once
506+-- before the fold starts consuming input. Any resource allocation can be done
507+-- in this function.
508+--
509+-- The @step@ function is called on each input, it consumes an input and
510+-- returns the next intermediate state (see 'Step') or the final result @b@ if
511+-- the fold terminates.
512+--
513+-- If the fold is used as a scan, the @extract@ function is used by the scan
514+-- driver to map the current state @s@ of the fold to the fold result. Thus
515+-- @extract@ can be called multiple times. In some folds, where scanning does
516+-- not make sense, this function is left unimplemented; such folds cannot be
517+-- used as scans.
518+--
519+-- Before a fold terminates, @final@ is called once and only once (unless the
520+-- fold terminated in @initial@ itself). Any resources allocated by @initial@
521+-- can be released in @final@. In folds that do not require any cleanup
522+-- @extract@ and @final@ are typically the same.
523+--
524+-- When implementing fold combinators, care should be taken to cleanup any
525+-- state of the argument folds held by the fold by calling the respective
526+-- @final@ at all exit points of the fold. Also, @final@ should not be called
527+-- more than once. Note that if a fold terminates by 'Done' constructor, there
528+-- is no state to cleanup.
484529--
485530-- NOTE: The constructor is not yet released, smart constructors are provided
486531-- to create folds.
487532--
488533data Fold m a b =
489− -- | @Fold @ @ step @ @ initial @ @ extract@
490− forall s. Fold (s -> a -> m (Step s b)) (m (Step s b)) (s -> m b)
534+ -- | @Fold@ @step@ @initial@ @extract@ @final@
535+ forall s. Fold (s -> a -> m (Step s b)) (m (Step s b)) (s -> m b) (s -> m b)
491536
492537------------------------------------------------------------------------------
493538-- Mapping on the output
@@ -497,7 +542,8 @@ data Fold m a b =
497542--
498543{-# INLINE rmapM #-}
499544rmapM :: Monad m => (b -> m c) -> Fold m a b -> Fold m a c
500−rmapM f (Fold step initial extract) = Fold step1 initial1 (extract >=> f)
545+rmapM f (Fold step initial extract final) =
546+ Fold step1 initial1 (extract >=> f) (final >=> f)
501547
502548 where
503549
@@ -528,6 +574,7 @@ foldl' step initial =
528574 (\s a -> return $ Partial $ step s a)
529575 (return (Partial initial))
530576 return
577+ return
531578
532579-- | Make a fold from a left fold style monadic step function and initial value
533580-- of the accumulator.
@@ -542,7 +589,7 @@ foldl' step initial =
542589{-# INLINE foldlM' #-}
543590foldlM' :: Monad m => (b -> a -> m b) -> m b -> Fold m a b
544591foldlM' step initial =
545− Fold (\s a -> Partial <$> step s a) (Partial <$> initial) return
592+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) return return
546593
547594-- | Make a strict left fold, for non-empty streams, using first element as the
548595-- starting value. Returns Nothing if the stream is empty.
@@ -640,7 +687,11 @@ foldrM' g z =
640687{-# INLINE foldt' #-}
641688foldt' :: Monad m => (s -> a -> Step s b) -> Step s b -> (s -> b) -> Fold m a b
642689foldt' step initial extract =
643− Fold (\s a -> return $ step s a) (return initial) (return . extract)
690+ Fold
691+ (\s a -> return $ step s a)
692+ (return initial)
693+ (return . extract)
694+ (return . extract)
644695
645696-- | Make a terminating fold with an effectful step function and initial state,
646697-- and a state extraction function.
@@ -653,7 +704,7 @@ foldt' step initial extract =
653704--
654705{-# INLINE foldtM' #-}
655706foldtM' :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> Fold m a b
656−foldtM' = Fold
707+foldtM' step initial extract = Fold step initial extract extract
657708
658709------------------------------------------------------------------------------
659710-- Refold
@@ -667,7 +718,7 @@ foldtM' = Fold
667718-- /Internal/
668719fromRefold :: Refold m c a b -> c -> Fold m a b
669720fromRefold (Refold step inject extract) c =
670− Fold step (inject c) extract
721+ Fold step (inject c) extract extract
671722
672723------------------------------------------------------------------------------
673724-- Basic Folds
@@ -720,6 +771,29 @@ toStreamKRev = foldl' (flip K.cons) K.nil
720771toStreamK :: Monad m => Fold m a (K.StreamK n a)
721772toStreamK = foldr K.cons K.nil
722773
774+-- | Like 'length', except with a more general 'Num' return value
775+--
776+-- Definition:
777+--
778+-- >>> lengthGeneric = fmap getSum $ Fold.foldMap (Sum . const 1)
779+-- >>> lengthGeneric = Fold.foldl' (\n _ -> n + 1) 0
780+--
781+-- /Pre-release/
782+{-# INLINE lengthGeneric #-}
783+lengthGeneric :: (Monad m, Num b) => Fold m a b
784+lengthGeneric = foldl' (\n _ -> n + 1) 0
785+
786+-- | Determine the length of the input stream.
787+--
788+-- Definition:
789+--
790+-- >>> length = Fold.lengthGeneric
791+-- >>> length = fmap getSum $ Fold.foldMap (Sum . const 1)
792+--
793+{-# INLINE length #-}
794+length :: Monad m => Fold m a Int
795+length = lengthGeneric
796+
723797------------------------------------------------------------------------------
724798-- Instances
725799------------------------------------------------------------------------------
@@ -727,7 +801,8 @@ toStreamK = foldr K.cons K.nil
727801-- | Maps a function on the output of the fold (the type @b@).
728802instance Functor m => Functor (Fold m a) where
729803 {-# INLINE fmap #-}
730− fmap f (Fold step1 initial1 extract) = Fold step initial (fmap2 f extract)
804+ fmap f (Fold step1 initial1 extract final) =
805+ Fold step initial (fmap2 f extract) (fmap2 f final)
731806
732807 where
733808
@@ -762,7 +837,7 @@ instance Functor m => Functor (Fold m a) where
762837--
763838{-# INLINE fromPure #-}
764839fromPure :: Applicative m => b -> Fold m a b
765−fromPure b = Fold undefined (pure $ Done b) pure
840+fromPure b = Fold undefined (pure $ Done b) pure pure
766841
767842-- | Make a fold that yields the result of the supplied effectful action
768843-- without consuming any further input.
@@ -771,7 +846,7 @@ fromPure b = Fold undefined (pure $ Done b) pure
771846--
772847{-# INLINE fromEffect #-}
773848fromEffect :: Applicative m => m b -> Fold m a b
774−fromEffect b = Fold undefined (Done <$> b) pure
849+fromEffect b = Fold undefined (Done <$> b) pure pure
775850
776851{-# ANN type SeqFoldState Fuse #-}
777852data SeqFoldState sl f sr = SeqFoldL !sl | SeqFoldR !f !sr
@@ -796,16 +871,20 @@ data SeqFoldState sl f sr = SeqFoldL !sl | SeqFoldR !f !sr
796871-- complexity, because each composition adds a new branch that each subsequent
797872-- fold's input element has to traverse, therefore, it cannot scale to a large
798873-- number of compositions. After around 100 compositions the performance starts
799−-- dipping rapidly compared to a CPS style implementation. When you need
800−-- scaling use parser monad instead.
874+-- dipping rapidly compared to a CPS style implementation.
875+--
876+-- For larger number of compositions you can convert the fold to a parser and
877+-- use ParserK.
801878--
802879-- /Time: O(n^2) where n is the number of compositions./
803880--
804881{-# INLINE splitWith #-}
805882splitWith :: Monad m =>
806883 (a -> b -> c) -> Fold m x a -> Fold m x b -> Fold m x c
807−splitWith func (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
808− Fold step initial extract
884+splitWith func
885+ (Fold stepL initialL _ finalL)
886+ (Fold stepR initialR _ finalR) =
887+ Fold step initial extract final
809888
810889 where
811890
@@ -822,13 +901,18 @@ splitWith func (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
822901 step (SeqFoldL st) a = runL (stepL st a)
823902 step (SeqFoldR f st) a = runR (stepR st a) f
824903
825− extract (SeqFoldR f sR) = fmap f (extractR sR)
826− extract (SeqFoldL sL) = do
827− rL <- extractL sL
904+ -- XXX splitWith should not be used for scanning
905+ -- It would rarely make sense and resource tracking and cleanup would be
906+ -- expensive. especially when multiple splitWith are chained.
907+ extract _ = error "splitWith: cannot be used for scanning"
908+
909+ final (SeqFoldR f sR) = fmap f (finalR sR)
910+ final (SeqFoldL sL) = do
911+ rL <- finalL sL
828912 res <- initialR
829913 fmap (func rL)
830914 $ case res of
831− Partial sR -> extractR sR
915+ Partial sR -> finalR sR
832916 Done rR -> return rR
833917
834918{-# DEPRECATED serialWith "Please use \"splitWith\" instead" #-}
@@ -848,8 +932,8 @@ data SeqFoldState_ sl sr = SeqFoldL_ !sl | SeqFoldR_ !sr
848932--
849933{-# INLINE split_ #-}
850934split_ :: Monad m => Fold m x a -> Fold m x b -> Fold m x b
851−split_ (Fold stepL initialL _) (Fold stepR initialR extractR) =
852− Fold step initial extract
935+split_ (Fold stepL initialL _ finalL) (Fold stepR initialR _ finalR) =
936+ Fold step initial extract final
853937
854938 where
855939
@@ -872,11 +956,16 @@ split_ (Fold stepL initialL _) (Fold stepR initialR extractR) =
872956 resR <- stepR st a
873957 return $ first SeqFoldR_ resR
874958
875− extract (SeqFoldR_ sR) = extractR sR
876− extract (SeqFoldL_ _) = do
959+ -- XXX split_ should not be used for scanning
960+ -- See splitWith for more details.
961+ extract _ = error "split_: cannot be used for scanning"
962+
963+ final (SeqFoldR_ sR) = finalR sR
964+ final (SeqFoldL_ sL) = do
965+ _ <- finalL sL
877966 res <- initialR
878967 case res of
879− Partial sR -> extractR sR
968+ Partial sR -> finalR sR
880969 Done rR -> return rR
881970
882971-- | 'Applicative' form of 'splitWith'. Split the input serially over two
@@ -924,8 +1013,10 @@ data TeeState sL sR bL bR
9241013--
9251014{-# INLINE teeWith #-}
9261015teeWith :: Monad m => (a -> b -> c) -> Fold m x a -> Fold m x b -> Fold m x c
927−teeWith f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
928− Fold step initial extract
1016+teeWith f
1017+ (Fold stepL initialL extractL finalL)
1018+ (Fold stepR initialR extractR finalR) =
1019+ Fold step initial extract final
9291020
9301021 where
9311022
@@ -952,6 +1043,10 @@ teeWith f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
9521043 extract (TeeLeft bR sL) = (`f` bR) <$> extractL sL
9531044 extract (TeeRight bL sR) = f bL <$> extractR sR
9541045
1046+ final (TeeBoth sL sR) = f <$> finalL sL <*> finalR sR
1047+ final (TeeLeft bR sL) = (`f` bR) <$> finalL sL
1048+ final (TeeRight bL sR) = f bL <$> finalR sR
1049+
9551050{-# ANN type TeeFstState Fuse #-}
9561051data TeeFstState sL sR b
9571052 = TeeFstBoth !sL !sR
@@ -964,8 +1059,10 @@ data TeeFstState sL sR b
9641059{-# INLINE teeWithFst #-}
9651060teeWithFst :: Monad m =>
9661061 (b -> c -> d) -> Fold m a b -> Fold m a c -> Fold m a d
967−teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
968− Fold step initial extract
1062+teeWithFst f
1063+ (Fold stepL initialL extractL finalL)
1064+ (Fold stepR initialR extractR finalR) =
1065+ Fold step initial extract final
9691066
9701067 where
9711068
@@ -984,7 +1081,7 @@ teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
9841081 Done bl -> do
9851082 Done . f bl <$>
9861083 case resR of
987− Partial sr -> extractR sr
1084+ Partial sr -> finalR sr
9881085 Done br -> return br
9891086
9901087 initial = runBoth initialL initialR
@@ -995,6 +1092,9 @@ teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
9951092 extract (TeeFstBoth sL sR) = f <$> extractL sL <*> extractR sR
9961093 extract (TeeFstLeft bR sL) = (`f` bR) <$> extractL sL
9971094
1095+ final (TeeFstBoth sL sR) = f <$> finalL sL <*> finalR sR
1096+ final (TeeFstLeft bR sL) = (`f` bR) <$> finalL sL
1097+
9981098-- | Like 'teeWith' but terminates as soon as any one of the two folds
9991099-- terminates.
10001100--
@@ -1003,8 +1103,10 @@ teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10031103{-# INLINE teeWithMin #-}
10041104teeWithMin :: Monad m =>
10051105 (b -> c -> d) -> Fold m a b -> Fold m a c -> Fold m a d
1006−teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1007− Fold step initial extract
1106+teeWithMin f
1107+ (Fold stepL initialL extractL finalL)
1108+ (Fold stepR initialR extractR finalR) =
1109+ Fold step initial extract final
10081110
10091111 where
10101112
@@ -1016,12 +1118,12 @@ teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10161118 Partial sl -> do
10171119 case resR of
10181120 Partial sr -> return $ Partial $ Tuple' sl sr
1019− Done br -> Done . (`f` br) <$> extractL sl
1121+ Done br -> Done . (`f` br) <$> finalL sl
10201122
10211123 Done bl -> do
10221124 Done . f bl <$>
10231125 case resR of
1024− Partial sr -> extractR sr
1126+ Partial sr -> finalR sr
10251127 Done br -> return br
10261128
10271129 initial = runBoth initialL initialR
@@ -1030,6 +1132,8 @@ teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10301132
10311133 extract (Tuple' sL sR) = f <$> extractL sL <*> extractR sR
10321134
1135+ final (Tuple' sL sR) = f <$> finalL sL <*> finalR sR
1136+
10331137-- | Shortest alternative. Apply both folds in parallel but choose the result
10341138-- from the one which consumed least input i.e. take the shortest succeeding
10351139-- fold.
@@ -1041,8 +1145,8 @@ teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
10411145--
10421146{-# INLINE shortest #-}
10431147shortest :: Monad m => Fold m x a -> Fold m x b -> Fold m x (Either a b)
1044−shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
1045− Fold step initial extract
1148+shortest (Fold stepL initialL extractL finalL) (Fold stepR initialR _ finalR) =
1149+ Fold step initial extract final
10461150
10471151 where
10481152
@@ -1050,10 +1154,16 @@ shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
10501154 runBoth actionL actionR = do
10511155 resL <- actionL
10521156 resR <- actionR
1053− return $
1054− case resL of
1055− Partial sL -> bimap (Tuple' sL) Right resR
1056− Done bL -> Done $ Left bL
1157+ case resL of
1158+ Partial sL ->
1159+ case resR of
1160+ Partial sR -> return $ Partial $ Tuple' sL sR
1161+ Done bR -> finalL sL >> return (Done (Right bR))
1162+ Done bL -> do
1163+ case resR of
1164+ Partial sR -> void (finalR sR)
1165+ Done _ -> return ()
1166+ return (Done (Left bL))
10571167
10581168 initial = runBoth initialL initialR
10591169
@@ -1061,6 +1171,8 @@ shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
10611171
10621172 extract (Tuple' sL _) = Left <$> extractL sL
10631173
1174+ final (Tuple' sL sR) = Left <$> finalL sL <* finalR sR
1175+
10641176{-# ANN type LongestState Fuse #-}
10651177data LongestState sL sR
10661178 = LongestBoth !sL !sR
@@ -1078,8 +1190,10 @@ data LongestState sL sR
10781190--
10791191{-# INLINE longest #-}
10801192longest :: Monad m => Fold m x a -> Fold m x b -> Fold m x (Either a b)
1081−longest (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1082− Fold step initial extract
1193+longest
1194+ (Fold stepL initialL _ finalL)
1195+ (Fold stepR initialR _ finalR) =
1196+ Fold step initial extract final
10831197
10841198 where
10851199
@@ -1102,14 +1216,18 @@ longest (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
11021216 step (LongestLeft sL) a = bimap LongestLeft Left <$> stepL sL a
11031217 step (LongestRight sR) a = bimap LongestRight Right <$> stepR sR a
11041218
1105− left sL = Left <$> extractL sL
1106− extract (LongestLeft sL) = left sL
1107− extract (LongestRight sR) = Right <$> extractR sR
1108− extract (LongestBoth sL _) = left sL
1219+ -- XXX Scan with this may not make sense as we cannot determine the longest
1220+ -- until one of them have exhausted.
1221+ extract _ = error $ "longest: scan is not allowed as longest cannot be "
1222+ ++ "determined until one fold has exhausted."
1223+
1224+ final (LongestLeft sL) = Left <$> finalL sL
1225+ final (LongestRight sR) = Right <$> finalR sR
1226+ final (LongestBoth sL sR) = Left <$> finalL sL <* finalR sR
11091227
1110−data ConcatMapState m sa a c
1111− = B !sa
1112− | forall s. C (s -> a -> m (Step s c)) !s (s -> m c)
1228+data ConcatMapState m sa a b c
1229+ = B !sa (sa -> m b)
1230+ | forall s. C (s -> a -> m (Step s c)) !s (s -> m c) (s -> m c)
11131231
11141232-- | Map a 'Fold' returning function on the result of a 'Fold' and run the
11151233-- returned fold. This operation can be used to express data dependencies
@@ -1132,43 +1250,47 @@ data ConcatMapState m sa a c
11321250--
11331251{-# INLINE concatMap #-}
11341252concatMap :: Monad m => (b -> Fold m a c) -> Fold m a b -> Fold m a c
1135−concatMap f (Fold stepa initiala extracta) = Fold stepc initialc extractc
1253+concatMap f (Fold stepa initiala _ finala) =
1254+ Fold stepc initialc extractc finalc
11361255 where
11371256 initialc = do
11381257 r <- initiala
11391258 case r of
1140− Partial s -> return $ Partial (B s)
1259+ Partial s -> return $ Partial (B s finala)
11411260 Done b -> initInnerFold (f b)
11421261
1143− stepc (B s) a = do
1262+ stepc (B s fin) a = do
11441263 r <- stepa s a
11451264 case r of
1146− Partial s1 -> return $ Partial (B s1)
1265+ Partial s1 -> return $ Partial (B s1 fin)
11471266 Done b -> initInnerFold (f b)
11481267
1149− stepc (C stepInner s extractInner) a = do
1268+ stepc (C stepInner s extractInner fin) a = do
11501269 r <- stepInner s a
11511270 return $ case r of
1152− Partial sc -> Partial (C stepInner sc extractInner)
1271+ Partial sc -> Partial (C stepInner sc extractInner fin)
11531272 Done c -> Done c
11541273
1155− extractc (B s) = do
1156− r <- extracta s
1157− initExtract (f r)
1158− extractc (C _ sInner extractInner) = extractInner sInner
1274+ -- XXX Cannot use for scanning
1275+ extractc _ = error "concatMap: cannot be used for scanning"
11591276
1160− initInnerFold (Fold step i e) = do
1277+ initInnerFold (Fold step i e fin) = do
11611278 r <- i
11621279 return $ case r of
1163− Partial s -> Partial (C step s e)
1280+ Partial s -> Partial (C step s e fin)
11641281 Done c -> Done c
11651282
1166− initExtract (Fold _ i e) = do
1283+ initFinalize (Fold _ i _ fin) = do
11671284 r <- i
11681285 case r of
1169− Partial s -> e s
1286+ Partial s -> fin s
11701287 Done c -> return c
11711288
1289+ finalc (B s fin) = do
1290+ r <- fin s
1291+ initFinalize (f r)
1292+ finalc (C _ sInner _ fin) = fin sInner
1293+
11721294------------------------------------------------------------------------------
11731295-- Mapping on input
11741296------------------------------------------------------------------------------
@@ -1187,7 +1309,7 @@ concatMap f (Fold stepa initiala extracta) = Fold stepc initialc extractc
11871309--
11881310{-# INLINE lmap #-}
11891311lmap :: (a -> b) -> Fold m b r -> Fold m a r
1190−lmap f (Fold step begin done) = Fold step' begin done
1312+lmap f (Fold step begin done final) = Fold step' begin done final
11911313 where
11921314 step' x a = step x (f a)
11931315
@@ -1195,7 +1317,7 @@ lmap f (Fold step begin done) = Fold step' begin done
11951317--
11961318{-# INLINE lmapM #-}
11971319lmapM :: Monad m => (a -> m b) -> Fold m b r -> Fold m a r
1198−lmapM f (Fold step begin done) = Fold step' begin done
1320+lmapM f (Fold step begin done final) = Fold step' begin done final
11991321 where
12001322 step' x a = f a >>= step x
12011323
@@ -1207,8 +1329,10 @@ lmapM f (Fold step begin done) = Fold step' begin done
12071329-- /Pre-release/
12081330{-# INLINE postscan #-}
12091331postscan :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1210−postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1211− Fold step initial extract
1332+postscan
1333+ (Fold stepL initialL extractL finalL)
1334+ (Fold stepR initialR extractR finalR) =
1335+ Fold step initial extract final
12121336
12131337 where
12141338
@@ -1219,30 +1343,32 @@ postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
12191343 Done bL -> do
12201344 rR <- stepR sR bL
12211345 case rR of
1222− Partial sR1 -> Done <$> extractR sR1
1346+ Partial sR1 -> Done <$> finalR sR1
12231347 Done bR -> return $ Done bR
12241348 Partial sL -> do
12251349 !b <- extractL sL
12261350 rR <- stepR sR b
1227− return
1228− $ case rR of
1229− Partial sR1 -> Partial (sL, sR1)
1230− Done bR -> Done bR
1351+ case rR of
1352+ Partial sR1 -> return $ Partial (sL, sR1)
1353+ Done bR -> finalL sL >> return (Done bR)
12311354
12321355 initial = do
1233− r <- initialR
1234− rL <- initialL
1235− case r of
1236− Partial sR ->
1356+ rR <- initialR
1357+ case rR of
1358+ Partial sR -> do
1359+ rL <- initialL
12371360 case rL of
1238− Done _ -> Done <$> extractR sR
1361+ Done _ -> Done <$> finalR sR
12391362 Partial sL -> return $ Partial (sL, sR)
12401363 Done b -> return $ Done b
12411364
1365+ -- XXX should use Tuple'
12421366 step (sL, sR) x = runStep (stepL sL x) sR
12431367
12441368 extract = extractR . snd
12451369
1370+ final (sL, sR) = finalL sL *> finalR sR
1371+
12461372------------------------------------------------------------------------------
12471373-- Filtering
12481374------------------------------------------------------------------------------
@@ -1255,7 +1381,7 @@ postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
12551381--
12561382{-# INLINE_NORMAL catMaybes #-}
12571383catMaybes :: Monad m => Fold m a b -> Fold m (Maybe a) b
1258−catMaybes (Fold step initial extract) = Fold step1 initial extract
1384+catMaybes (Fold step initial extract final) = Fold step1 initial extract final
12591385
12601386 where
12611387
@@ -1295,7 +1421,7 @@ filtering f = foldl' step Nothing
12951421{-# INLINE filter #-}
12961422filter :: Monad m => (a -> Bool) -> Fold m a r -> Fold m a r
12971423-- filter p = scanMaybe (filtering p)
1298−filter f (Fold step begin done) = Fold step' begin done
1424+filter f (Fold step begin extract final) = Fold step' begin extract final
12991425 where
13001426 step' x a = if f a then step x a else return $ Partial x
13011427
@@ -1306,7 +1432,7 @@ filter f (Fold step begin done) = Fold step' begin done
13061432--
13071433{-# INLINE filterM #-}
13081434filterM :: Monad m => (a -> m Bool) -> Fold m a r -> Fold m a r
1309−filterM f (Fold step begin done) = Fold step' begin done
1435+filterM f (Fold step begin extract final) = Fold step' begin extract final
13101436 where
13111437 step' x a = do
13121438 use <- f a
@@ -1395,7 +1521,7 @@ dropping n = foldt' step initial extract
13951521{-# INLINE take #-}
13961522take :: Monad m => Int -> Fold m a b -> Fold m a b
13971523-- take n = scanMaybe (taking n)
1398−take n (Fold fstep finitial fextract) = Fold step initial extract
1524+take n (Fold fstep finitial fextract ffinal) = Fold step initial extract final
13991525
14001526 where
14011527
@@ -1407,7 +1533,7 @@ take n (Fold fstep finitial fextract) = Fold step initial extract
14071533 s1 = Tuple'Fused i1 s
14081534 if i1 < n
14091535 then return $ Partial s1
1410− else Done <$> fextract s
1536+ else Done <$> ffinal s
14111537 Done b -> return $ Done b
14121538
14131539 initial = finitial >>= next (-1)
@@ -1416,6 +1542,72 @@ take n (Fold fstep finitial fextract) = Fold step initial extract
14161542
14171543 extract (Tuple'Fused _ r) = fextract r
14181544
1545+ final (Tuple'Fused _ r) = ffinal r
1546+
1547+-- Note: Keep this consistent with S.splitOn. In fact we should eliminate
1548+-- S.splitOn in favor of the fold.
1549+--
1550+-- XXX Use Fold.many instead once it is fixed.
1551+-- > Stream.splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
1552+
1553+-- | Like 'takeEndBy' but drops the element on which the predicate succeeds.
1554+--
1555+-- Example:
1556+--
1557+-- >>> input = Stream.fromList "hello\nthere\n"
1558+-- >>> line = Fold.takeEndBy_ (== '\n') Fold.toList
1559+-- >>> Stream.fold line input
1560+-- "hello"
1561+--
1562+-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1563+-- ["hello","there"]
1564+--
1565+{-# INLINE takeEndBy_ #-}
1566+takeEndBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1567+-- takeEndBy_ predicate = scanMaybe (takingEndBy_ predicate)
1568+takeEndBy_ predicate (Fold fstep finitial fextract ffinal) =
1569+ Fold step finitial fextract ffinal
1570+
1571+ where
1572+
1573+ step s a =
1574+ if not (predicate a)
1575+ then fstep s a
1576+ else Done <$> ffinal s
1577+
1578+-- Note:
1579+-- > Stream.splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
1580+
1581+-- | Take the input, stop when the predicate succeeds taking the succeeding
1582+-- element as well.
1583+--
1584+-- Example:
1585+--
1586+-- >>> input = Stream.fromList "hello\nthere\n"
1587+-- >>> line = Fold.takeEndBy (== '\n') Fold.toList
1588+-- >>> Stream.fold line input
1589+-- "hello\n"
1590+--
1591+-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1592+-- ["hello\n","there\n"]
1593+--
1594+{-# INLINE takeEndBy #-}
1595+takeEndBy :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1596+-- takeEndBy predicate = scanMaybe (takingEndBy predicate)
1597+takeEndBy predicate (Fold fstep finitial fextract ffinal) =
1598+ Fold step finitial fextract ffinal
1599+
1600+ where
1601+
1602+ step s a = do
1603+ res <- fstep s a
1604+ if not (predicate a)
1605+ then return res
1606+ else do
1607+ case res of
1608+ Partial s1 -> Done <$> ffinal s1
1609+ Done b -> return $ Done b
1610+
14191611------------------------------------------------------------------------------
14201612-- Nesting
14211613------------------------------------------------------------------------------
@@ -1433,8 +1625,8 @@ take n (Fold fstep finitial fextract) = Fold step initial extract
14331625-- /Pre-release/
14341626{-# INLINE duplicate #-}
14351627duplicate :: Monad m => Fold m a b -> Fold m a (Fold m a b)
1436−duplicate (Fold step1 initial1 extract1) =
1437− Fold step initial (\s -> pure $ Fold step1 (pure $ Partial s) extract1)
1628+duplicate (Fold step1 initial1 extract1 final1) =
1629+ Fold step initial extract final
14381630
14391631 where
14401632
@@ -1442,6 +1634,11 @@ duplicate (Fold step1 initial1 extract1) =
14421634
14431635 step s a = second fromPure <$> step1 s a
14441636
1637+ -- Scanning may be problematic due to multiple finalizations.
1638+ extract = error "duplicate: scanning may be problematic"
1639+
1640+ final s = pure $ Fold step1 (pure $ Partial s) extract1 final1
1641+
14451642-- If there were a finalize/flushing action in the stream type that would be
14461643-- equivalent to running initialize in Fold. But we do not have a flushing
14471644-- action in streams.
@@ -1453,9 +1650,9 @@ duplicate (Fold step1 initial1 extract1) =
14531650-- /Pre-release/
14541651{-# INLINE reduce #-}
14551652reduce :: Monad m => Fold m a b -> m (Fold m a b)
1456−reduce (Fold step initial extract) = do
1653+reduce (Fold step initial extract final) = do
14571654 i <- initial
1458− return $ Fold step (return i) extract
1655+ return $ Fold step (return i) extract final
14591656
14601657-- This is the dual of Stream @cons@.
14611658
@@ -1465,7 +1662,8 @@ reduce (Fold step initial extract) = do
14651662-- /Pre-release/
14661663{-# INLINE snoclM #-}
14671664snoclM :: Monad m => Fold m a b -> m a -> Fold m a b
1468−snoclM (Fold fstep finitial fextract) action = Fold fstep initial fextract
1665+snoclM (Fold fstep finitial fextract ffinal) action =
1666+ Fold fstep initial fextract ffinal
14691667
14701668 where
14711669
@@ -1492,7 +1690,8 @@ snoclM (Fold fstep finitial fextract) action = Fold fstep initial fextract
14921690{-# INLINE snocl #-}
14931691snocl :: Monad m => Fold m a b -> a -> Fold m a b
14941692-- snocl f = snoclM f . return
1495−snocl (Fold fstep finitial fextract) a = Fold fstep initial fextract
1693+snocl (Fold fstep finitial fextract ffinal) a =
1694+ Fold fstep initial fextract ffinal
14961695
14971696 where
14981697
@@ -1512,12 +1711,12 @@ snocl (Fold fstep finitial fextract) a = Fold fstep initial fextract
15121711-- /Pre-release/
15131712{-# INLINE snocM #-}
15141713snocM :: Monad m => Fold m a b -> m a -> m (Fold m a b)
1515−snocM (Fold step initial extract) action = do
1714+snocM (Fold step initial extract final) action = do
15161715 res <- initial
15171716 r <- case res of
15181717 Partial fs -> action >>= step fs
15191718 Done _ -> return res
1520− return $ Fold step (return r) extract
1719+ return $ Fold step (return r) extract final
15211720
15221721-- Definitions:
15231722--
@@ -1536,12 +1735,12 @@ snocM (Fold step initial extract) action = do
15361735-- /Pre-release/
15371736{-# INLINE snoc #-}
15381737snoc :: Monad m => Fold m a b -> a -> m (Fold m a b)
1539−snoc (Fold step initial extract) a = do
1738+snoc (Fold step initial extract final) a = do
15401739 res <- initial
15411740 r <- case res of
15421741 Partial fs -> step fs a
15431742 Done _ -> return res
1544− return $ Fold step (return r) extract
1743+ return $ Fold step (return r) extract final
15451744
15461745-- | Append a singleton value to the fold.
15471746--
@@ -1569,7 +1768,7 @@ addOne = flip snoc
15691768-- /Pre-release/
15701769{-# INLINE extractM #-}
15711770extractM :: Monad m => Fold m a b -> m b
1572−extractM (Fold _ initial extract) = do
1771+extractM (Fold _ initial extract _) = do
15731772 res <- initial
15741773 case res of
15751774 Partial fs -> extract fs
@@ -1578,14 +1777,15 @@ extractM (Fold _ initial extract) = do
15781777-- | Close a fold so that it does not accept any more input.
15791778{-# INLINE close #-}
15801779close :: Monad m => Fold m a b -> Fold m a b
1581−close (Fold _ initial1 extract1) = Fold undefined initial undefined
1780+close (Fold _ initial1 _ final1) =
1781+ Fold undefined initial undefined undefined
15821782
15831783 where
15841784
15851785 initial = do
15861786 res <- initial1
15871787 case res of
1588− Partial s -> Done <$> extract1 s
1788+ Partial s -> Done <$> final1 s
15891789 Done b -> return $ Done b
15901790
15911791-- Corresponds to the null check for streams.
@@ -1595,7 +1795,7 @@ close (Fold _ initial1 extract1) = Fold undefined initial undefined
15951795-- /Pre-release/
15961796{-# INLINE isClosed #-}
15971797isClosed :: Monad m => Fold m a b -> m Bool
1598−isClosed (Fold _ initial _) = do
1798+isClosed (Fold _ initial _ _) = do
15991799 res <- initial
16001800 return $ case res of
16011801 Partial _ -> False
@@ -1629,8 +1829,10 @@ data ManyState s1 s2
16291829--
16301830{-# INLINE many #-}
16311831many :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1632−many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
1633− Fold step initial extract
1832+many
1833+ (Fold sstep sinitial sextract sfinal)
1834+ (Fold cstep cinitial cextract cfinal) =
1835+ Fold step initial extract final
16341836
16351837 where
16361838
@@ -1679,6 +1881,13 @@ many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
16791881 Partial s -> cextract s
16801882 Done b -> return b
16811883
1884+ final (ManyFirst ss cs) = sfinal ss *> cfinal cs
1885+ final (ManyLoop ss cs) = do
1886+ cres <- sfinal ss >>= cstep cs
1887+ case cres of
1888+ Partial s -> cfinal s
1889+ Done b -> return b
1890+
16821891-- | Like many, but the "first" fold emits an output at the end even if no
16831892-- input is received.
16841893--
@@ -1688,8 +1897,10 @@ many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
16881897--
16891898{-# INLINE manyPost #-}
16901899manyPost :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1691−manyPost (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
1692− Fold step initial extract
1900+manyPost
1901+ (Fold sstep sinitial sextract sfinal)
1902+ (Fold cstep cinitial cextract cfinal) =
1903+ Fold step initial extract final
16931904
16941905 where
16951906
@@ -1725,6 +1936,12 @@ manyPost (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
17251936 Partial s -> cextract s
17261937 Done b -> return b
17271938
1939+ final (Tuple' ss cs) = do
1940+ cres <- sfinal ss >>= cstep cs
1941+ case cres of
1942+ Partial s -> cfinal s
1943+ Done b -> return b
1944+
17281945-- | @groupsOf n split collect@ repeatedly applies the @split@ fold to chunks
17291946-- of @n@ items in the input stream and supplies the result to the @collect@
17301947-- fold.
@@ -1753,7 +1970,10 @@ groupsOf n split = many (take n split)
17531970--
17541971{-# INLINE refoldMany #-}
17551972refoldMany :: Monad m => Fold m a b -> Refold m x b c -> Refold m x a c
1756−refoldMany (Fold sstep sinitial sextract) (Refold cstep cinject cextract) =
1973+refoldMany
1974+ (Fold sstep sinitial sextract _sfinal)
1975+ -- XXX We will need a "final" in refold as well
1976+ (Refold cstep cinject cextract) =
17571977 Refold step inject extract
17581978
17591979 where
@@ -1804,7 +2024,9 @@ data ConsumeManyState x cs ss = ConsumeMany x cs (Either ss ss)
18042024-- /Internal/
18052025{-# INLINE refoldMany1 #-}
18062026refoldMany1 :: Monad m => Refold m x a b -> Fold m b c -> Refold m x a c
1807−refoldMany1 (Refold sstep sinject sextract) (Fold cstep cinitial cextract) =
2027+refoldMany1
2028+ (Refold sstep sinject sextract)
2029+ (Fold cstep cinitial cextract _cfinal) =
18082030 Refold step inject extract
18092031
18102032 where
@@ -1855,7 +2077,7 @@ refoldMany1 (Refold sstep sinject sextract) (Fold cstep cinitial cextract) =
18552077{-# INLINE refold #-}
18562078refold :: Monad m => Refold m b a c -> Fold m a b -> Fold m a c
18572079refold (Refold step inject extract) f =
1858− Fold step (extractM f >>= inject) extract
2080+ Fold step (extractM f >>= inject) extract extract
18592081
18602082------------------------------------------------------------------------------
18612083-- morphInner
@@ -1865,8 +2087,8 @@ refold (Refold step inject extract) f =
18652087--
18662088-- /Pre-release/
18672089morphInner :: (forall x. m x -> n x) -> Fold m a b -> Fold n a b
1868−morphInner f (Fold step initial extract) =
1869− Fold (\x a -> f $ step x a) (f initial) (f . extract)
2090+morphInner f (Fold step initial extract final) =
2091+ Fold (\x a -> f $ step x a) (f initial) (f . extract) (f . final)
18702092
18712093-- | Adapt a pure fold to any monad.
18722094--
core/src/Streamly/Internal/Data/Fold/Window.hs+10−4
@@ -65,6 +65,8 @@ import Streamly.Internal.Data.Fold.Type (Fold(..), Step(..))
6565import Streamly.Internal.Data.Tuple.Strict
6666 (Tuple'(..), Tuple3Fused' (Tuple3Fused'))
6767
68+import Foreign.ForeignPtr.Unsafe (unsafeForeignPtrToPtr)
69+
6870import qualified Streamly.Internal.Data.Fold.Type as Fold
6971import qualified Streamly.Internal.Data.Ring as Ring
7072
@@ -148,7 +150,7 @@ windowRollingMap f = Fold.foldl' f1 initial
148150--
149151{-# INLINE windowSumInt #-}
150152windowSumInt :: forall m a. (Monad m, Integral a) => Fold m (a, Maybe a) a
151−windowSumInt = Fold step initial extract
153+windowSumInt = Fold step initial extract extract
152154
153155 where
154156
@@ -182,7 +184,7 @@ windowSumInt = Fold step initial extract
182184--
183185{-# INLINE windowSum #-}
184186windowSum :: forall m a. (Monad m, Num a) => Fold m (a, Maybe a) a
185−windowSum = Fold step initial extract
187+windowSum = Fold step initial extract extract
186188
187189 where
188190
@@ -267,7 +269,7 @@ windowPowerSumFrac p = windowLmap (** p) windowSum
267269--
268270{-# INLINE windowRange #-}
269271windowRange :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe (a, a))
270−windowRange n = Fold step initial extract
272+windowRange n = Fold step initial extract extract
271273
272274 where
273275
@@ -300,7 +302,11 @@ windowRange n = Fold step initial extract
300302 if i == 0
301303 then return Nothing
302304 else do
303− x <- liftIO $ peek rh
305+ -- Here we use "ringStart" over "ringHead" as "ringHead" will be
306+ -- uninitialized if the ring is not full.
307+ -- Using "unsafeForeignPtrToPtr" here is safe as we touch the ring
308+ -- again in "foldFunc".
309+ x <- liftIO $ peek (unsafeForeignPtrToPtr (Ring.ringStart rb))
304310 let accum (mn, mx) a = return (min mn a, max mx a)
305311 fmap Just $ foldFunc i rh accum (x, x) rb
306312
core/src/Streamly/Internal/Data/IORef/Unboxed.hs+22−19
@@ -23,34 +23,31 @@ module Streamly.Internal.Data.IORef.Unboxed
2323 (
2424 IORef
2525
26− -- * Construction
26+ -- Construction
2727 , newIORef
2828
29− -- * Write
29+ -- Write
3030 , writeIORef
3131 , modifyIORef'
3232
33− -- * Read
33+ -- Read
3434 , readIORef
35− , toStreamD
35+ , pollIntIORef
3636 )
3737where
3838
3939#include "inline.hs"
4040
41−import Data.Proxy (Proxy(..))
4241import Control.Monad.IO.Class (MonadIO(..))
43−import Streamly.Internal.Data.Unbox
44− ( MutableByteArray(..)
45− , Unbox(..)
46− , newBytes
47− , sizeOf
48− )
42+import Data.Proxy (Proxy(..))
43+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
44+import Streamly.Internal.Data.Unbox (Unbox(..), sizeOf)
4945
46+import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
5047import qualified Streamly.Internal.Data.Stream.Type as D
5148
5249-- | An 'IORef' holds a single 'Unbox'-able value.
53−newtype IORef a = IORef MutableByteArray
50+newtype IORef a = IORef MutByteArray
5451
5552-- | Create a new 'IORef'.
5653--
@@ -58,8 +55,8 @@ newtype IORef a = IORef MutableByteArray
5855{-# INLINE newIORef #-}
5956newIORef :: forall a. Unbox a => a -> IO (IORef a)
6057newIORef x = do
61− var <- newBytes (sizeOf (Proxy :: Proxy a))
62− pokeByteIndex 0 var x
58+ var <- MBA.new (sizeOf (Proxy :: Proxy a))
59+ pokeAt 0 var x
6360 return $ IORef var
6461
6562-- | Write a value to an 'IORef'.
@@ -67,14 +64,14 @@ newIORef x = do
6764-- /Pre-release/
6865{-# INLINE writeIORef #-}
6966writeIORef :: Unbox a => IORef a -> a -> IO ()
70−writeIORef (IORef var) = pokeByteIndex 0 var
67+writeIORef (IORef var) = pokeAt 0 var
7168
7269-- | Read a value from an 'IORef'.
7370--
7471-- /Pre-release/
7572{-# INLINE readIORef #-}
7673readIORef :: Unbox a => IORef a -> IO a
77−readIORef (IORef var) = peekByteIndex 0 var
74+readIORef (IORef var) = peekAt 0 var
7875
7976-- | Modify the value of an 'IORef' using a function with strict application.
8077--
@@ -87,10 +84,16 @@ modifyIORef' var g = do
8784
8885-- | Generate a stream by continuously reading the IORef.
8986--
87+-- This operation reads the IORef without any synchronization. It can be
88+-- assumed to be atomic because the IORef (MutableByteArray) is always aligned
89+-- to Int boundaries, we are assuming that compiler uses single instructions to
90+-- access the memory. It may read stale values though until caches are
91+-- synchronised in a multiprocessor architecture.
92+--
9093-- /Pre-release/
91−{-# INLINE_NORMAL toStreamD #-}
92−toStreamD :: (MonadIO m, Unbox a) => IORef a -> D.Stream m a
93−toStreamD var = D.Stream step ()
94+{-# INLINE_NORMAL pollIntIORef #-}
95+pollIntIORef :: (MonadIO m, Unbox a) => IORef a -> D.Stream m a
96+pollIntIORef var = D.Stream step ()
9497
9598 where
9699
core/src/Streamly/Internal/Data/IsMap.hs+8
@@ -1,3 +1,7 @@
1+{-# LANGUAGE TypeFamilies #-}
2+-- Must come after TypeFamilies, otherwise it is re-enabled.
3+-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
4+{-# LANGUAGE NoMonoLocalBinds #-}
15-- |
26-- Module : Streamly.Internal.Data.IsMap
37-- Copyright : (c) 2022 Composewell Technologies
@@ -28,6 +32,8 @@ class IsMap f where
2832 mapDelete :: Key f -> f a -> f a
2933 mapUnion :: f a -> f a -> f a
3034 mapNull :: f a -> Bool
35+ mapTraverseWithKey ::
36+ Applicative t => (Key f -> a -> t b) -> f a -> t (f b)
3137
3238instance Ord k => IsMap (Map k) where
3339 type Key (Map k) = k
@@ -39,6 +45,7 @@ instance Ord k => IsMap (Map k) where
3945 mapDelete = Map.delete
4046 mapUnion = Map.union
4147 mapNull = Map.null
48+ mapTraverseWithKey = Map.traverseWithKey
4249
4350instance IsMap IntMap.IntMap where
4451 type Key IntMap.IntMap = Int
@@ -50,3 +57,4 @@ instance IsMap IntMap.IntMap where
5057 mapDelete = IntMap.delete
5158 mapUnion = IntMap.union
5259 mapNull = IntMap.null
60+ mapTraverseWithKey = IntMap.traverseWithKey
core/src/Streamly/Internal/Data/MutArray.hs+154−33
@@ -8,79 +8,200 @@
88--
99module Streamly.Internal.Data.MutArray
1010 (
11+ -- * MutArray.Type module
1112 module Streamly.Internal.Data.MutArray.Type
12− , splitOn
13+ -- * MutArray module
14+ , sliceIndexerFromLen
15+ , slicerFromLen
16+ , compactLE
17+ , pinnedCompactLE
18+ , compactOnByte
19+ , compactOnByteSuffix
20+ -- * Unboxed IORef
21+ , module Streamly.Internal.Data.IORef.Unboxed
22+
23+ -- * Deprecated
1324 , genSlicesFromLen
1425 , getSlicesFromLen
15− , fromStream
1626 )
1727where
1828
1929#include "inline.hs"
2030
2131import Control.Monad.IO.Class (MonadIO(..))
32+import Data.Word (Word8)
33+import Streamly.Internal.Data.MutByteArray.Type (PinnedState(..))
34+import Streamly.Internal.Data.Stream.Type (Stream)
2235import Streamly.Internal.Data.Unbox (Unbox)
23−import Streamly.Internal.Data.Stream (Stream)
2436import Streamly.Internal.Data.Unfold.Type (Unfold(..))
2537
26−import qualified Streamly.Internal.Data.Stream as D
38+import qualified Streamly.Internal.Data.Stream.Type as Stream
39+import qualified Streamly.Internal.Data.Stream.Nesting as Stream
40+-- import qualified Streamly.Internal.Data.Stream.Transform as Stream
2741import qualified Streamly.Internal.Data.Unfold as Unfold
2842
2943import Prelude hiding (foldr, length, read, splitAt)
3044import Streamly.Internal.Data.MutArray.Type
31−
32−-- | Split the array into a stream of slices using a predicate. The element
33−-- matching the predicate is dropped.
34−--
35−-- /Pre-release/
36−{-# INLINE splitOn #-}
37−splitOn :: (MonadIO m, Unbox a) =>
38− (a -> Bool) -> MutArray a -> Stream m (MutArray a)
39−splitOn predicate arr =
40− fmap (\(i, len) -> getSliceUnsafe i len arr)
41− $ D.sliceOnSuffix predicate (read arr)
45+import Streamly.Internal.Data.IORef.Unboxed
4246
4347-- | Generate a stream of array slice descriptors ((index, len)) of specified
4448-- length from an array, starting from the supplied array index. The last slice
4549-- may be shorter than the requested length depending on the array length.
4650--
4751-- /Pre-release/
48−{-# INLINE genSlicesFromLen #-}
49−genSlicesFromLen :: forall m a. (Monad m, Unbox a)
52+{-# INLINE sliceIndexerFromLen #-}
53+sliceIndexerFromLen :: forall m a. (Monad m, Unbox a)
5054 => Int -- ^ from index
5155 -> Int -- ^ length of the slice
5256 -> Unfold m (MutArray a) (Int, Int)
53−genSlicesFromLen from len =
57+sliceIndexerFromLen from len =
5458 let fromThenTo n = (from, from + len, n - 1)
5559 mkSlice n i = return (i, min len (n - i))
5660 in Unfold.lmap length
5761 $ Unfold.mapM2 mkSlice
5862 $ Unfold.lmap fromThenTo Unfold.enumerateFromThenTo
5963
64+{-# DEPRECATED genSlicesFromLen "Please use sliceIndexerFromLen instead." #-}
65+genSlicesFromLen :: forall m a. (Monad m, Unbox a)
66+ => Int -- ^ from index
67+ -> Int -- ^ length of the slice
68+ -> Unfold m (MutArray a) (Int, Int)
69+genSlicesFromLen = sliceIndexerFromLen
70+
6071-- | Generate a stream of slices of specified length from an array, starting
6172-- from the supplied array index. The last slice may be shorter than the
6273-- requested length depending on the array length.
6374--
6475-- /Pre-release/
65−{-# INLINE getSlicesFromLen #-}
66−getSlicesFromLen :: forall m a. (Monad m, Unbox a)
76+{-# INLINE slicerFromLen #-}
77+slicerFromLen :: forall m a. (Monad m, Unbox a)
6778 => Int -- ^ from index
6879 -> Int -- ^ length of the slice
6980 -> Unfold m (MutArray a) (MutArray a)
70−getSlicesFromLen from len =
81+slicerFromLen from len =
7182 let mkSlice arr (i, n) = return $ getSliceUnsafe i n arr
72− in Unfold.mapM2 mkSlice (genSlicesFromLen from len)
83+ in Unfold.mapM2 mkSlice (sliceIndexerFromLen from len)
84+
85+{-# DEPRECATED getSlicesFromLen "Please use slicerFromLen instead." #-}
86+getSlicesFromLen :: forall m a. (Monad m, Unbox a)
87+ => Int -- ^ from index
88+ -> Int -- ^ length of the slice
89+ -> Unfold m (MutArray a) (MutArray a)
90+getSlicesFromLen = slicerFromLen
7391
74−-- | Create an 'Array' from a stream. This is useful when we want to create a
75−-- single array from a stream of unknown size. 'writeN' is at least twice
76−-- as efficient when the size is already known.
92+-------------------------------------------------------------------------------
93+-- Compacting Streams of Arrays
94+-------------------------------------------------------------------------------
95+
96+-- | @compactLE maxElems@ coalesces adjacent arrays in the input stream
97+-- only if the combined size would be less than or equal to @maxElems@
98+-- elements. Note that it won't split an array if the original array is already
99+-- larger than maxElems.
77100--
78−-- Note that if the input stream is too large memory allocation for the array
79−-- may fail. When the stream size is not known, `chunksOf` followed by
80−-- processing of indvidual arrays in the resulting stream should be preferred.
101+-- @maxElems@ must be greater than 0.
81102--
82−-- /Pre-release/
83−{-# INLINE fromStream #-}
84−fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
85−fromStream = fromStreamD
86−-- fromStream (Stream m) = P.fold write m
103+-- Generates unpinned arrays irrespective of the pinning status of input
104+-- arrays.
105+{-# INLINE compactLE #-}
106+compactLE :: (MonadIO m, Unbox a) =>
107+ Int -> Stream m (MutArray a) -> Stream m (MutArray a)
108+-- XXX compactLE can be moved to MutArray/Type if we are not using the parser
109+-- to implement it.
110+compactLE = compactLeAs Unpinned
111+-- The parser version turns out to be a little bit slower.
112+-- compactLE n = Stream.catRights . Stream.parseManyD (pCompactLE n)
113+
114+-- | Pinned version of 'compactLE'.
115+{-# INLINE pinnedCompactLE #-}
116+pinnedCompactLE :: forall m a. (MonadIO m, Unbox a)
117+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
118+pinnedCompactLE = compactLeAs Pinned
119+-- pinnedCompactLE n = Stream.catRights . Stream.parseManyD (pPinnedCompactLE n)
120+
121+data SplitState s arr
122+ = Initial s
123+ | Buffering s arr
124+ | Splitting s arr
125+ | Yielding arr (SplitState s arr)
126+ | Finishing
127+
128+-- | Split a stream of arrays on a given separator byte, dropping the separator
129+-- and coalescing all the arrays between two separators into a single array.
130+--
131+{-# INLINE_NORMAL _compactOnByteCustom #-}
132+_compactOnByteCustom
133+ :: MonadIO m
134+ => Word8
135+ -> Stream m (MutArray Word8)
136+ -> Stream m (MutArray Word8)
137+_compactOnByteCustom byte (Stream.Stream step state) =
138+ Stream.Stream step' (Initial state)
139+
140+ where
141+
142+ {-# INLINE_LATE step' #-}
143+ step' gst (Initial st) = do
144+ r <- step gst st
145+ case r of
146+ Stream.Yield arr s -> do
147+ (arr1, marr2) <- breakOn byte arr
148+ return $ case marr2 of
149+ Nothing -> Stream.Skip (Buffering s arr1)
150+ Just arr2 -> Stream.Skip (Yielding arr1 (Splitting s arr2))
151+ Stream.Skip s -> return $ Stream.Skip (Initial s)
152+ Stream.Stop -> return Stream.Stop
153+
154+ step' gst (Buffering st buf) = do
155+ r <- step gst st
156+ case r of
157+ Stream.Yield arr s -> do
158+ (arr1, marr2) <- breakOn byte arr
159+ -- XXX Use spliceExp instead and then rightSize?
160+ buf1 <- splice buf arr1
161+ return $ case marr2 of
162+ Nothing -> Stream.Skip (Buffering s buf1)
163+ Just x -> Stream.Skip (Yielding buf1 (Splitting s x))
164+ Stream.Skip s -> return $ Stream.Skip (Buffering s buf)
165+ Stream.Stop -> return $
166+ if byteLength buf == 0
167+ then Stream.Stop
168+ else Stream.Skip (Yielding buf Finishing)
169+
170+ step' _ (Splitting st buf) = do
171+ (arr1, marr2) <- breakOn byte buf
172+ return $ case marr2 of
173+ Nothing -> Stream.Skip $ Buffering st arr1
174+ Just arr2 -> Stream.Skip $ Yielding arr1 (Splitting st arr2)
175+
176+ step' _ (Yielding arr next) = return $ Stream.Yield arr next
177+ step' _ Finishing = return Stream.Stop
178+
179+-- XXX implement predicate based version of this
180+-- XXX Naming of predicate based vs custom version
181+
182+-- | Split a stream of arrays on a given separator byte, dropping the separator
183+-- and coalescing all the arrays between two separators into a single array.
184+--
185+{-# INLINE compactOnByte #-}
186+compactOnByte
187+ :: (MonadIO m)
188+ => Word8
189+ -> Stream m (MutArray Word8)
190+ -> Stream m (MutArray Word8)
191+-- XXX compare perf of custom vs idiomatic version
192+-- compactOnByte = _compactOnByteCustom
193+-- XXX use spliceExp and rightSize?
194+compactOnByte byte = Stream.splitInnerBy (breakOn byte) splice
195+
196+-- | Like 'compactOnByte' considers the separator in suffix position instead of
197+-- infix position.
198+{-# INLINE compactOnByteSuffix #-}
199+compactOnByteSuffix
200+ :: (MonadIO m)
201+ => Word8
202+ -> Stream m (MutArray Word8)
203+ -> Stream m (MutArray Word8)
204+compactOnByteSuffix byte =
205+ -- XXX use spliceExp and rightSize?
206+ Stream.splitInnerBySuffix
207+ (\arr -> byteLength arr == 0) (breakOn byte) splice
core/src/Streamly/Internal/Data/MutArray/Generic.hs+65−30
@@ -19,14 +19,14 @@ module Streamly.Internal.Data.MutArray.Generic
1919 , nil
2020
2121 -- *** Uninitialized Arrays
22− , new
22+ , emptyOf
2323 -- , newArrayWith
2424
2525 -- *** From streams
26− , writeNUnsafe
27− , writeN
28− , writeWith
29− , write
26+ , unsafeCreateOf
27+ , createOf
28+ , createWith -- createOfMin/createMin/createGE?
29+ , create
3030 , fromStreamN
3131 , fromStream
3232 , fromPureStream
@@ -160,6 +160,13 @@ module Streamly.Internal.Data.MutArray.Generic
160160 -- , appendSliceFrom
161161
162162 , clone
163+
164+ -- * Deprecated
165+ , new
166+ , writeNUnsafe
167+ , writeN
168+ , writeWith
169+ , write
163170 )
164171where
165172
@@ -236,13 +243,13 @@ bottomElement =
236243-- XXX Would be nice if GHC can provide something like newUninitializedArray# so
237244-- that we do not have to write undefined or error in the whole array.
238245
239−-- | @new count@ allocates a zero length array that can be extended to hold
246+-- | @emptyOf count@ allocates a zero length array that can be extended to hold
240247-- up to 'count' items without reallocating.
241248--
242249-- /Pre-release/
243−{-# INLINE new #-}
244−new :: MonadIO m => Int -> m (MutArray a)
245−new n@(I# n#) =
250+{-# INLINE emptyOf #-}
251+emptyOf :: MonadIO m => Int -> m (MutArray a)
252+emptyOf n@(I# n#) =
246253 liftIO
247254 $ IO
248255 $ \s# ->
@@ -251,6 +258,12 @@ new n@(I# n#) =
251258 let ma = MutArray arr# 0 0 n
252259 in (# s1#, ma #)
253260
261+-- XXX Deprecate in major
262+-- {-# DEPRECATED new "Please use emptyOf instead." #-}
263+{-# INLINE new #-}
264+new :: MonadIO m => Int -> m (MutArray a)
265+new = emptyOf
266+
254267-- XXX This could be pure?
255268
256269-- |
@@ -318,7 +331,7 @@ putIndices arr = FL.foldlM' step (return ())
318331
319332 where
320333
321− step () (i, x) = liftIO (putIndex i arr x)
334+ step () (i, x) = putIndex i arr x
322335
323336-- | Modify a given index of an array using a modifier function without checking
324337-- the bounds.
@@ -589,16 +602,16 @@ readRev arr@MutArray{..} =
589602arrayChunkSize :: Int
590603arrayChunkSize = 1024
591604
592−-- | Like 'writeN' but does not check the array bounds when writing. The fold
605+-- | Like 'createOf' but does not check the array bounds when writing. The fold
593606-- driver must not call the step function more than 'n' times otherwise it will
594607-- corrupt the memory and crash. This function exists mainly because any
595608-- conditional in the step function blocks fusion causing 10x performance
596609-- slowdown.
597610--
598611-- /Pre-release/
599−{-# INLINE_NORMAL writeNUnsafe #-}
600−writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
601−writeNUnsafe n = Fold step initial return
612+{-# INLINE_NORMAL unsafeCreateOf #-}
613+unsafeCreateOf :: MonadIO m => Int -> Fold m a (MutArray a)
614+unsafeCreateOf n = Fold step initial return return
602615
603616 where
604617
@@ -606,59 +619,81 @@ writeNUnsafe n = Fold step initial return
606619
607620 step arr x = FL.Partial <$> snocUnsafe arr x
608621
609−-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
622+{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
623+{-# INLINE writeNUnsafe #-}
624+writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
625+writeNUnsafe = unsafeCreateOf
626+
627+-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
610628-- 'Array'.
611629--
612−-- >>> writeN n = Fold.take n (MutArray.writeNUnsafe n)
630+-- >>> createOf n = Fold.take n (MutArray.unsafeCreateOf n)
613631--
614632-- /Pre-release/
615−{-# INLINE_NORMAL writeN #-}
633+{-# INLINE_NORMAL createOf #-}
634+createOf :: MonadIO m => Int -> Fold m a (MutArray a)
635+createOf n = FL.take n $ unsafeCreateOf n
636+
637+-- XXX Deprecate in major
638+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
639+{-# INLINE writeN #-}
616640writeN :: MonadIO m => Int -> Fold m a (MutArray a)
617−writeN n = FL.take n $ writeNUnsafe n
641+writeN = createOf
618642
619643-- >>> f n = MutArray.writeAppendWith (* 2) (MutArray.pinnedNew n)
620644-- >>> writeWith n = Fold.rmapM MutArray.rightSize (f n)
621645-- >>> writeWith n = Fold.rmapM MutArray.fromArrayStreamK (MutArray.writeChunks n)
622646
623−-- | @writeWith minCount@ folds the whole input to a single array. The array
647+-- | @createWith minCount@ folds the whole input to a single array. The array
624648-- starts at a size big enough to hold minCount elements, the size is doubled
625649-- every time the array needs to be grown.
626650--
627651-- /Caution! Do not use this on infinite streams./
628652--
629653-- /Pre-release/
630−{-# INLINE_NORMAL writeWith #-}
631−writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
654+{-# INLINE_NORMAL createWith #-}
655+createWith :: MonadIO m => Int -> Fold m a (MutArray a)
632656-- writeWith n = FL.rmapM rightSize $ writeAppendWith (* 2) (pinnedNew n)
633−writeWith elemCount = FL.rmapM extract $ FL.foldlM' step initial
657+createWith elemCount = FL.rmapM extract $ FL.foldlM' step initial
634658
635659 where
636660
637661 initial = do
638− when (elemCount < 0) $ error "writeWith: elemCount is negative"
639− liftIO $ new elemCount
662+ when (elemCount < 0) $ error "createWith: elemCount is negative"
663+ new elemCount
640664
641665 step arr@(MutArray _ start end bound) x
642666 | end == bound = do
643667 let oldSize = end - start
644668 newSize = max (oldSize * 2) 1
645− arr1 <- liftIO $ realloc newSize arr
669+ arr1 <- realloc newSize arr
646670 snocUnsafe arr1 x
647671 step arr x = snocUnsafe arr x
648672
649− -- extract = liftIO . rightSize
673+ -- extract = rightSize
650674 extract = return
651675
676+{-# DEPRECATED writeWith "Please use createWith instead." #-}
677+{-# INLINE writeWith #-}
678+writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
679+writeWith = createWith
680+
652681-- | Fold the whole input to a single array.
653682--
654−-- Same as 'writeWith' using an initial array size of 'arrayChunkSize' bytes
683+-- Same as 'createWith' using an initial array size of 'arrayChunkSize' bytes
655684-- rounded up to the element size.
656685--
657686-- /Caution! Do not use this on infinite streams./
658687--
688+{-# INLINE create #-}
689+create :: MonadIO m => Fold m a (MutArray a)
690+create = writeWith arrayChunkSize
691+
692+-- XXX Deprecate in major
693+-- {-# DEPRECATED write "Please use create instead." #-}
659694{-# INLINE write #-}
660695write :: MonadIO m => Fold m a (MutArray a)
661−write = writeWith arrayChunkSize
696+write = create
662697
663698-- | Create a 'MutArray' from the first @n@ elements of a stream. The
664699-- array is allocated to size @n@, if the stream terminates before @n@
@@ -683,7 +718,7 @@ fromList xs = fromStream $ D.fromList xs
683718{-# INLINABLE fromPureStream #-}
684719fromPureStream :: MonadIO m => Stream Identity a -> m (MutArray a)
685720fromPureStream xs =
686− liftIO $ D.fold write $ D.morphInner (return . runIdentity) xs
721+ D.fold write $ D.morphInner (return . runIdentity) xs
687722
688723-------------------------------------------------------------------------------
689724-- Chunking
@@ -815,7 +850,7 @@ putSliceUnsafe src srcStart dst dstStart len = liftIO $ do
815850
816851{-# INLINE clone #-}
817852clone :: MonadIO m => MutArray a -> m (MutArray a)
818−clone src = liftIO $ do
853+clone src = do
819854 let len = arrLen src
820855 dst <- new len
821856 putSliceUnsafe src 0 dst 0 len
core/src/Streamly/Internal/Data/MutArray/Stream.hsnew file+145
@@ -0,0 +1,145 @@
1+-- |
2+-- Module : Streamly.Internal.Data.MutArray.Stream
3+-- Copyright : (c) 2019 Composewell Technologies
4+-- License : BSD3-3-Clause
5+-- Maintainer : streamly@composewell.com
6+-- Stability : experimental
7+-- Portability : GHC
8+--
9+-- Combinators to efficiently manipulate streams of mutable arrays.
10+--
11+-- We can either push these in the MutArray module with a "chunks" prefix or
12+-- keep this as a separate module and release it.
13+--
14+module Streamly.Internal.Data.MutArray.Stream
15+{-# DEPRECATED "Please use \"Streamly.Internal.Data.MutArray\" instead." #-}
16+ (
17+ -- * Generation
18+ MArray.chunksOf
19+ , MArray.pinnedChunksOf
20+ , MArray.writeChunks -- chunksWrite?
21+ , MArray.splitOn -- chunksSplitOn
22+
23+ -- * Compaction
24+ , packArraysChunksOf
25+ , MArray.SpliceState (..)
26+ , lpackArraysChunksOf
27+ , compact -- chunksCompact
28+ , compactLE
29+ , compactEQ
30+ , compactGE
31+
32+ -- * Elimination
33+ , MArray.flattenArrays -- chunksConcat
34+ , MArray.flattenArraysRev -- chunksConcatRev
35+ , MArray.fromArrayStreamK -- chunksCoalesce
36+ )
37+where
38+
39+import Control.Monad.IO.Class (MonadIO(..))
40+import Streamly.Internal.Data.Unbox (Unbox)
41+import Streamly.Internal.Data.MutArray.Type (MutArray(..))
42+import Streamly.Internal.Data.Fold.Type (Fold(..))
43+import Streamly.Internal.Data.Parser (ParseError)
44+import Streamly.Internal.Data.Stream.Type (Stream)
45+
46+import qualified Streamly.Internal.Data.MutArray as MArray
47+import qualified Streamly.Internal.Data.Fold.Type as FL
48+import qualified Streamly.Internal.Data.Parser as ParserD
49+import qualified Streamly.Internal.Data.Stream as D
50+
51+-------------------------------------------------------------------------------
52+-- Compact
53+-------------------------------------------------------------------------------
54+
55+-- XXX This can be removed once compactLEFold/compactLE are implemented.
56+--
57+-- | This mutates the first array (if it has space) to append values from the
58+-- second one. This would work for immutable arrays as well because an
59+-- immutable array never has space so a new array is allocated instead of
60+-- mutating it.
61+--
62+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
63+-- maximum specified size. Note that if a single array is bigger than the
64+-- specified size we do not split it to fit. When we coalesce multiple arrays
65+-- if the size would exceed the specified size we do not coalesce therefore the
66+-- actual array size may be less than the specified chunk size.
67+--
68+-- @since 0.7.0
69+{-# INLINE packArraysChunksOf #-}
70+packArraysChunksOf :: (MonadIO m, Unbox a)
71+ => Int -> D.Stream m (MutArray a) -> D.Stream m (MutArray a)
72+packArraysChunksOf = MArray.compactLE
73+
74+-- XXX Remove this once compactLEFold is implemented
75+-- lpackArraysChunksOf = Fold.many compactLEFold
76+--
77+{-# INLINE lpackArraysChunksOf #-}
78+lpackArraysChunksOf :: (MonadIO m, Unbox a)
79+ => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
80+lpackArraysChunksOf = MArray.lCompactGE
81+
82+-- XXX Same as compactLE, to be removed once that is implemented.
83+--
84+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
85+-- maximum specified size in bytes.
86+--
87+-- /Internal/
88+{-# INLINE compact #-}
89+compact :: (MonadIO m, Unbox a)
90+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
91+compact = packArraysChunksOf
92+
93+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
94+-- maximum specified size. Note that if a single array is bigger than the
95+-- specified size we do not split it to fit. When we coalesce multiple arrays
96+-- if the size would exceed the specified size we do not coalesce therefore the
97+-- actual array size may be less than the specified chunk size.
98+--
99+-- /Internal/
100+{-# INLINE compactLEParserD #-}
101+compactLEParserD ::
102+ forall m a. (MonadIO m, Unbox a)
103+ => Int -> ParserD.Parser (MutArray a) m (MutArray a)
104+compactLEParserD = MArray.pCompactLE
105+
106+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
107+-- minimum specified size. Note that if all the arrays in the stream together
108+-- are smaller than the specified size the resulting array will be smaller than
109+-- the specified size. When we coalesce multiple arrays if the size would exceed
110+-- the specified size we stop coalescing further.
111+--
112+-- /Internal/
113+{-# INLINE compactGEFold #-}
114+compactGEFold ::
115+ forall m a. (MonadIO m, Unbox a)
116+ => Int -> FL.Fold m (MutArray a) (MutArray a)
117+compactGEFold = MArray.fCompactGE
118+
119+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
120+-- maximum specified size in bytes.
121+--
122+-- /Internal/
123+compactLE :: (MonadIO m, Unbox a) =>
124+ Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
125+compactLE n = D.parseManyD (compactLEParserD n)
126+
127+-- | Like 'compactLE' but generates arrays of exactly equal to the size
128+-- specified except for the last array in the stream which could be shorter.
129+--
130+-- /Unimplemented/
131+{-# INLINE compactEQ #-}
132+compactEQ :: -- (MonadIO m, Unbox a) =>
133+ Int -> Stream m (MutArray a) -> Stream m (MutArray a)
134+compactEQ _n _xs = undefined
135+ -- IsStream.fromStreamD $ D.foldMany (compactEQFold n) (IsStream.toStreamD xs)
136+
137+-- | Like 'compactLE' but generates arrays of size greater than or equal to the
138+-- specified except for the last array in the stream which could be shorter.
139+--
140+-- /Internal/
141+{-# INLINE compactGE #-}
142+compactGE ::
143+ (MonadIO m, Unbox a)
144+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
145+compactGE n = D.foldMany (compactGEFold n)
core/src/Streamly/Internal/Data/MutArray/Type.hs+1271−417
@@ -28,57 +28,80 @@
2828
2929module Streamly.Internal.Data.MutArray.Type
3030 (
31− -- * Type
31+ -- ** Type
3232 -- $arrayNotes
3333 MutArray (..)
34− , MutableByteArray
35− , touch
34+
35+ -- ** Conversion
36+ -- *** Pinned and Unpinned
3637 , pin
3738 , unpin
3839 , isPinned
3940
40− -- * Constructing and Writing
41+ -- ** Casting
42+ , cast
43+ , castUnsafe -- XXX unsafeCast
44+ , asBytes
45+ , unsafePinnedAsPtr
46+ , unsafeAsPtr
47+
4148 -- ** Construction
42− , nil
49+ , empty
50+
51+ -- *** New
52+ -- | New arrays are always empty arrays with some reserve capacity to
53+ -- extend the length without reallocating.
54+ , emptyOf
55+ , newArrayWith -- emptyAlignedWith
56+ , pinnedEmptyOf
57+ , pinnedNewAligned -- XXX not required
58+ -- , new -- uninitialized array of specified length
59+
60+ -- *** Cloning
61+ , clone
62+ , pinnedClone
4363
44− -- *** Uninitialized Arrays
45− , pinnedNew
46− , pinnedNewBytes
47− , pinnedNewAligned
48− , new
49− , newArrayWith
64+ -- *** Slicing
65+ -- | Get a subarray without copying
66+ , getSliceUnsafe -- XXX unsafeGetSlice
67+ , getSlice
68+ , splitAt -- XXX should be able to express using getSlice
69+ , breakOn
5070
51− -- *** From streams
71+ -- *** Stream Folds
5272 , ArrayUnsafe (..)
53− , writeNWithUnsafe
54− , writeNWith
55− , writeNUnsafe
56− , pinnedWriteNUnsafe
57− , writeN
58− , pinnedWriteN
59− , pinnedWriteNAligned
60−
61− , writeWith
62− , write
63− , pinnedWrite
64−
65− , writeRevN
66− -- , writeRev
67−
68− -- ** From containers
73+ , unsafeCreateOfWith
74+ , unsafeCreateOf
75+ , unsafePinnedCreateOf
76+ , pinnedCreateOf
77+ , createOfWith
78+ , createOf
79+ , revCreateOf
80+
81+ , pinnedCreate
82+ , createWith
83+ , create
84+ -- , revCreate
85+
86+ -- *** From containers
6987 , fromListN
7088 , pinnedFromListN
7189 , fromList
7290 , pinnedFromList
7391 , fromListRevN
7492 , fromListRev
75− , fromStreamDN
76− , fromStreamD
93+ , fromStreamN
94+ , fromStream
95+ , fromPureStreamN
7796 , fromPureStream
97+ , fromByteStr#
98+ , fromPtrN
99+ , fromChunksK
100+ , fromChunksRealloced -- fromSmallChunks
78101
79− -- * Random writes
102+ -- ** Random writes
80103 , putIndex
81− , putIndexUnsafe
104+ , putIndexUnsafe -- XXX unsafePutIndex
82105 , putIndices
83106 -- , putFromThenTo
84107 -- , putFrom -- start writing at the given position
@@ -86,78 +109,71 @@ module Streamly.Internal.Data.MutArray.Type
86109 -- , putFromTo
87110 -- , putFromRev
88111 -- , putUptoRev
89− , modifyIndexUnsafe
112+ , modifyIndexUnsafe -- XXX unsafeModifyIndex
90113 , modifyIndex
91114 , modifyIndices
92115 , modify
93116 , swapIndices
94117 , unsafeSwapIndices
95118
96− -- * Growing and Shrinking
97− -- Arrays grow only at the end, though it is possible to grow on both sides
98− -- and therefore have a cons as well as snoc. But that will require two
99− -- bounds in the array representation.
100−
101− -- ** Appending elements
102− , snocWith
103− , snoc
104− , snocLinear
105− , snocMay
106− , snocUnsafe
107−
108− -- ** Appending streams
109− , writeAppendNUnsafe
110− , writeAppendN
111− , writeAppendWith
112− , writeAppend
119+ -- ** Reading
113120
114− -- * Eliminating and Reading
115−
116− -- ** To streams
117− , reader
118− , readerRevWith
119− , readerRev
121+ -- *** Indexing
122+ , getIndex
123+ , getIndexUnsafe -- XXX unsafeGetIndex
124+ -- , getFromThenTo
125+ , getIndexRev -- getRevIndex?
126+ , indexReader
127+ , indexReaderWith
120128
121− -- ** To containers
122− , toStreamDWith
123− , toStreamDRevWith
124− , toStreamKWith
125− , toStreamKRevWith
129+ -- *** To Streams
126130 , read
127131 , readRev
132+ , toStreamWith
133+ , toStreamRevWith
128134 , toStreamK
135+ , toStreamKWith
129136 , toStreamKRev
137+ , toStreamKRevWith
138+
139+ -- *** To Containers
130140 , toList
131141
142+ -- *** Unfolds
132143 -- experimental
133144 , producerWith
134145 , producer
135146
136− -- ** Random reads
137− , getIndex
138− , getIndexUnsafe
139− , getIndices
140− , getIndicesD
141− -- , getFromThenTo
142− , getIndexRev
143−
144− -- * Memory Management
145− , blockSize
146− , arrayChunkBytes
147− , allocBytesToElemCount
148− , realloc
149− , resize
150− , resizeExp
151− , rightSize
147+ , reader
148+ , readerRevWith
149+ , readerRev
152150
153− -- * Size
151+ -- ** Size and Capacity
152+ -- *** Size
154153 , length
155154 , byteLength
155+
156+ -- *** Capacity
156157 -- , capacity
157158 , byteCapacity
158159 , bytesFree
159160
160− -- * In-place Mutation Algorithms
161+ -- *** Capacity Management
162+ , blockSize
163+ , arrayChunkBytes
164+ , allocBytesToElemCount
165+ , realloc -- this is "resize"
166+ , grow
167+ , growExp
168+ , rightSize
169+
170+ -- ** Folding
171+ , foldl'
172+ , foldr
173+ , byteCmp
174+ , byteEq
175+
176+ -- ** In-place Mutation Algorithms
161177 , strip
162178 , reverse
163179 , permute
@@ -167,62 +183,121 @@ module Streamly.Internal.Data.MutArray.Type
167183 , mergeBy
168184 , bubble
169185
170− -- * Casting
171− , cast
172− , castUnsafe
173− , asBytes
174− , asPtrUnsafe
175−
176− -- * Folding
177− , foldl'
178− , foldr
179− , cmp
180−
181− -- * Arrays of arrays
182− -- We can add dimensionality parameter to the array type to get
183− -- multidimensional arrays. Multidimensional arrays would just be a
184− -- convenience wrapper on top of single dimensional arrays.
186+ -- ** Growing and Shrinking
187+ -- | Arrays grow only at the end, though it is possible to grow on both
188+ -- sides and therefore have a cons as well as snoc. But that will require
189+ -- both lower and upper bound in the array representation.
185190
186− -- | Operations dealing with multiple arrays, streams of arrays or
187− -- multidimensional array representations.
188−
189− -- ** Construct from streams
190− , chunksOf
191− , pinnedChunksOf
192− , writeChunks
193−
194− -- ** Eliminate to streams
195− , flattenArrays
196− , flattenArraysRev
197− , fromArrayStreamK
198−
199− -- ** Construct from arrays
200− -- get chunks without copying
201− , getSliceUnsafe
202− , getSlice
203− -- , getSlicesFromLenN
204− , splitAt -- XXX should be able to express using getSlice
205− , breakOn
191+ -- *** Appending elements
192+ , snocWith
193+ , snoc
194+ , snocLinear
195+ , snocMay
196+ , snocUnsafe -- XXX unsafeSnoc
206197
207− -- ** Cloning arrays
208− , clone
209− , pinnedClone
198+ -- *** Appending streams
199+ , unsafeAppendN
200+ , appendN
201+ , appendWith
202+ , append
210203
211− -- ** Appending arrays
204+ -- *** Appending arrays
212205 , spliceCopy
213206 , spliceWith
214207 , splice
215208 , spliceExp
216− , spliceUnsafe
209+ , spliceUnsafe -- XXX unsafeSplice
217210 -- , putSlice
218211 -- , appendSlice
219212 -- , appendSliceFrom
220213
221− -- * Utilities
214+ -- ** Serialization using Unbox
215+ , pokeAppend
216+ , pokeAppendMay
217+ , pokeSkipUnsafe -- unsafePokeSkip
218+
219+ -- ** Deserialization using Unbox
220+ , peekUncons
221+ , peekUnconsUnsafe -- unsafePeekUncons
222+ , peekSkipUnsafe -- unsafePeekSkip
223+
224+ -- Arrays of arrays
225+ -- We can add dimensionality parameter to the array type to get
226+ -- multidimensional arrays. Multidimensional arrays would just be a
227+ -- convenience wrapper on top of single dimensional arrays.
228+
229+ -- ** Streams of Arrays
230+ -- *** Chunk
231+ -- | Group a stream into arrays.
232+ , chunksOf
233+ , pinnedChunksOf
234+ , buildChunks
235+
236+ -- *** Split
237+ -- | Split an array into slices.
238+
239+ -- , getSlicesFromLenN
240+ , splitOn
241+ -- , slicesOf
242+
243+ -- *** Concat
244+ -- | Append the arrays in a stream to form a stream of elements.
245+ , concatWith
246+ , concatRevWith
247+ , concat
248+ , concatRev
249+
250+ -- *** Compact
251+ -- | Append the arrays in a stream to form a stream of larger arrays.
252+ , SpliceState (..)
253+ , pCompactLE
254+ , pPinnedCompactLE
255+ , compactLeAs
256+ , fCompactGE
257+ , fPinnedCompactGE
258+ , lCompactGE
259+ , lPinnedCompactGE
260+ , compactGE
261+ , compactEQ
262+
263+ -- ** Utilities
222264 , roundUpToPower2
223265 , memcpy
224266 , memcmp
225267 , c_memchr
268+
269+ -- * Deprecated
270+ , asPtrUnsafe
271+ , writeChunks
272+ , flattenArrays
273+ , flattenArraysRev
274+ , fromArrayStreamK
275+ , fromStreamDN
276+ , fromStreamD
277+ , cmp
278+ , getIndices
279+ , getIndicesWith
280+ , resize
281+ , resizeExp
282+ , nil
283+ , new
284+ , pinnedNew
285+ , pinnedNewBytes
286+ , writeAppendNUnsafe
287+ , writeAppendN
288+ , writeAppendWith
289+ , writeAppend
290+ , writeNWithUnsafe
291+ , writeNWith
292+ , writeNUnsafe
293+ , pinnedWriteNUnsafe
294+ , writeN
295+ , pinnedWriteN
296+ , pinnedWriteNAligned -- XXX not required
297+ , writeWith
298+ , write
299+ , pinnedWrite
300+ , writeRevN
226301 )
227302where
228303
@@ -233,49 +308,52 @@ where
233308
234309import Control.Monad (when, void)
235310import Control.Monad.IO.Class (MonadIO(..))
311+import Data.Bifunctor (first)
236312import Data.Bits (shiftR, (.|.), (.&.))
237313import Data.Functor.Identity (Identity(..))
238314import Data.Proxy (Proxy(..))
239315import Data.Word (Word8)
240316import Foreign.C.Types (CSize(..), CInt(..))
241317import Foreign.Ptr (plusPtr, minusPtr, nullPtr)
242−import Streamly.Internal.Data.Unbox
243− ( MutableByteArray(..)
244− , Unbox(..)
318+import Streamly.Internal.Data.MutByteArray.Type
319+ ( MutByteArray(..)
245320 , PinnedState(..)
246321 , getMutableByteArray#
247− , touch
248322 , putSliceUnsafe
249323 )
324+import Streamly.Internal.Data.Unbox (Unbox(..))
250325import GHC.Base
251326 ( IO(..)
252327 , Int(..)
253− , byteArrayContents#
254328 , compareByteArrays#
255329 , copyMutableByteArray#
256330 )
257331import GHC.Base (noinline)
258−import GHC.Exts (unsafeCoerce#)
332+import GHC.Exts (unsafeCoerce#, Addr#)
259333import GHC.Ptr (Ptr(..))
260334
261335import Streamly.Internal.Data.Fold.Type (Fold(..))
262336import Streamly.Internal.Data.Producer.Type (Producer (..))
263337import Streamly.Internal.Data.Stream.Type (Stream)
338+import Streamly.Internal.Data.Parser.Type (Parser (..))
264339import Streamly.Internal.Data.StreamK.Type (StreamK)
265340import Streamly.Internal.Data.SVar.Type (adaptState, defState)
341+import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
266342import Streamly.Internal.Data.Unfold.Type (Unfold(..))
267343import Streamly.Internal.System.IO (arrayPayloadSize, defaultChunkSize)
268344
269345import qualified Streamly.Internal.Data.Fold.Type as FL
346+import qualified Streamly.Internal.Data.MutByteArray.Type as Unboxed
347+import qualified Streamly.Internal.Data.Parser.Type as Parser
348+-- import qualified Streamly.Internal.Data.Fold.Type as Fold
270349import qualified Streamly.Internal.Data.Producer as Producer
271350import qualified Streamly.Internal.Data.Stream.Type as D
272351import qualified Streamly.Internal.Data.Stream.Lift as D
273352import qualified Streamly.Internal.Data.StreamK.Type as K
274−import qualified Streamly.Internal.Data.Unbox as Unboxed
275353import qualified Prelude
276354
277355import Prelude hiding
278− (Foldable(..), read, unlines, splitAt, reverse, truncate)
356+ (Foldable(..), concat, read, unlines, splitAt, reverse, truncate)
279357
280358#include "DocTestDataMutArray.hs"
281359
@@ -283,6 +361,7 @@ import Prelude hiding
283361-- Foreign helpers
284362-------------------------------------------------------------------------------
285363
364+-- NOTE: Have to be "ccall unsafe" so that we can pass unpinned memory to these
286365foreign import ccall unsafe "string.h memcpy" c_memcpy
287366 :: Ptr Word8 -> Ptr Word8 -> CSize -> IO (Ptr Word8)
288367
@@ -292,6 +371,9 @@ foreign import ccall unsafe "string.h memchr" c_memchr
292371foreign import ccall unsafe "string.h memcmp" c_memcmp
293372 :: Ptr Word8 -> Ptr Word8 -> CSize -> IO CInt
294373
374+foreign import ccall unsafe "string.h strlen" c_strlen
375+ :: Ptr Word8 -> IO CSize
376+
295377-- | Given an 'Unboxed' type (unused first arg) and a number of bytes, return
296378-- how many elements of that type will completely fit in those bytes.
297379--
@@ -346,7 +428,7 @@ data MutArray a =
346428 -- The array is a range into arrContents. arrContents may be a superset of
347429 -- the slice represented by the array. All offsets are in bytes.
348430 MutArray
349− { arrContents :: {-# UNPACK #-} !MutableByteArray
431+ { arrContents :: {-# UNPACK #-} !MutByteArray
350432 , arrStart :: {-# UNPACK #-} !Int -- ^ index into arrContents
351433 , arrEnd :: {-# UNPACK #-} !Int -- ^ index into arrContents
352434 -- Represents the first invalid index of
@@ -358,8 +440,8 @@ data MutArray a =
358440-- Pinning & Unpinning
359441-------------------------------------------------------------------------------
360442
361−-- | Copy the 'MutArray' to pinned memory if unpinned, else do nothing. The
362−-- overhead is a copy if the input array is unpinned.
443+-- | Return a copy of the array in pinned memory if unpinned, else return the
444+-- original array.
363445{-# INLINE pin #-}
364446pin :: MutArray a -> IO (MutArray a)
365447pin arr@MutArray{..} =
@@ -367,8 +449,8 @@ pin arr@MutArray{..} =
367449 then pure arr
368450 else pinnedClone arr
369451
370−-- | Copy the 'MutArray' to unpinned memory if pinned, else do nothing. The
371−-- overhead is a copy if the input array is pinned.
452+-- | Return a copy of the array in unpinned memory if pinned, else return the
453+-- original array.
372454{-# INLINE unpin #-}
373455unpin :: MutArray a -> IO (MutArray a)
374456unpin arr@MutArray{..} =
@@ -376,6 +458,7 @@ unpin arr@MutArray{..} =
376458 then clone arr
377459 else pure arr
378460
461+-- | Return 'True' if the array is allocated in pinned memory.
379462{-# INLINE isPinned #-}
380463isPinned :: MutArray a -> Bool
381464isPinned MutArray{..} = Unboxed.isPinned arrContents
@@ -396,6 +479,8 @@ isPinned MutArray{..} = Unboxed.isPinned arrContents
396479-- we need more than that. See stg_pinnedNewAlignedByteArrayzh and
397480-- allocatePinned in GHC source.
398481
482+-- XXX Rename to emptyAlignedWith, alignSize should be first arg.
483+
399484-- | @newArrayWith allocator alignment count@ allocates a new array of zero
400485-- length and with a capacity to hold @count@ elements, using @allocator
401486-- size alignment@ as the memory allocator function.
@@ -408,7 +493,7 @@ isPinned MutArray{..} = Unboxed.isPinned arrContents
408493-- /Pre-release/
409494{-# INLINE newArrayWith #-}
410495newArrayWith :: forall m a. (MonadIO m, Unbox a)
411− => (Int -> Int -> m MutableByteArray) -> Int -> Int -> m (MutArray a)
496+ => (Int -> Int -> m MutByteArray) -> Int -> Int -> m (MutArray a)
412497newArrayWith alloc alignSize count = do
413498 let size = max (count * SIZE_OF(a)) 0
414499 contents <- alloc size alignSize
@@ -419,12 +504,25 @@ newArrayWith alloc alignSize count = do
419504 , arrBound = size
420505 }
421506
507+-- For arrays "nil" sounds a bit odd. empty is better. The only problem with
508+-- empty is that it is also used by the Alternative type class. But assuming we
509+-- will mostly import the Array module qualified this should be fine.
510+
511+-- | Create an empty array.
512+empty ::
513+#ifdef DEVBUILD
514+ Unbox a =>
515+#endif
516+ MutArray a
517+empty = MutArray Unboxed.empty 0 0 0
518+
519+{-# DEPRECATED nil "Please use empty instead." #-}
422520nil ::
423521#ifdef DEVBUILD
424522 Unbox a =>
425523#endif
426524 MutArray a
427−nil = MutArray Unboxed.nil 0 0 0
525+nil = empty
428526
429527{-# INLINE newBytesAs #-}
430528newBytesAs :: MonadIO m =>
@@ -441,12 +539,13 @@ newBytesAs ps bytes = do
441539 , arrBound = bytes
442540 }
443541
444−-- | Allocates a pinned empty array that can hold 'count' items. The memory of
445−-- the array is uninitialized and the allocation is aligned as per the
446−-- 'Unboxed' instance of the type.
542+-- | Allocates a pinned empty array that with a reserved capacity of bytes.
543+-- The memory of the array is uninitialized and the allocation is aligned as
544+-- per the 'Unboxed' instance of the type.
447545--
448546-- /Pre-release/
449547{-# INLINE pinnedNewBytes #-}
548+{-# DEPRECATED pinnedNewBytes "Please use pinnedEmptyOf with appropriate calculation" #-}
450549pinnedNewBytes :: MonadIO m =>
451550#ifdef DEVBUILD
452551 Unbox a =>
@@ -471,21 +570,31 @@ newAs ps =
471570 (error "new: alignment is not used in unpinned arrays.")
472571
473572-- XXX can unaligned allocation be more efficient when alignment is not needed?
474−--
475−-- | Allocates an empty pinned array that can hold 'count' items. The memory of
476−-- the array is uninitialized and the allocation is aligned as per the 'Unboxed'
477−-- instance of the type.
478−--
573+
574+-- | Allocates a pinned array of zero length but growable to the specified
575+-- capacity without reallocation.
576+{-# INLINE pinnedEmptyOf #-}
577+pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
578+pinnedEmptyOf = newAs Pinned
579+
580+-- XXX Deprecate in major
581+-- {-# DEPRECATED pinnedNew "Please use pinnedEmptyOf instead." #-}
479582{-# INLINE pinnedNew #-}
480583pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
481−pinnedNew = newAs Pinned
584+pinnedNew = pinnedEmptyOf
482585
483−-- | Allocates an empty unpinned array that can hold 'count' items. The memory
484−-- of the array is uninitialized.
586+-- | Allocates an unpinned array of zero length but growable to the specified
587+-- capacity without reallocation.
485588--
589+{-# INLINE emptyOf #-}
590+emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
591+emptyOf = newAs Unpinned
592+
593+-- XXX Deprecate in major
594+-- {-# DEPRECATED new "Please use emptyOf instead." #-}
486595{-# INLINE new #-}
487596new :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
488−new = newAs Unpinned
597+new = emptyOf
489598
490599-------------------------------------------------------------------------------
491600-- Random writes
@@ -501,7 +610,7 @@ putIndexUnsafe :: forall m a. (MonadIO m, Unbox a)
501610putIndexUnsafe i MutArray{..} x = do
502611 let index = INDEX_OF(arrStart, i, a)
503612 assert (i >= 0 && INDEX_VALID(index, arrEnd, a)) (return ())
504− liftIO $ pokeByteIndex index arrContents x
613+ liftIO $ pokeAt index arrContents x
505614
506615invalidIndex :: String -> Int -> a
507616invalidIndex label i =
@@ -519,7 +628,7 @@ putIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
519628putIndex i MutArray{..} x = do
520629 let index = INDEX_OF(arrStart,i,a)
521630 if i >= 0 && INDEX_VALID(index,arrEnd,a)
522− then liftIO $ pokeByteIndex index arrContents x
631+ then liftIO $ pokeAt index arrContents x
523632 else invalidIndex "putIndex" i
524633
525634-- | Write an input stream of (index, value) pairs to an array. Throws an
@@ -533,7 +642,7 @@ putIndices arr = FL.foldlM' step (return ())
533642
534643 where
535644
536− step () (i, x) = liftIO (putIndex i arr x)
645+ step () (i, x) = putIndex i arr x
537646
538647-- | Modify a given index of an array using a modifier function.
539648--
@@ -545,9 +654,9 @@ modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) =>
545654modifyIndexUnsafe i MutArray{..} f = liftIO $ do
546655 let index = INDEX_OF(arrStart,i,a)
547656 assert (i >= 0 && INDEX_NEXT(index,a) <= arrEnd) (return ())
548− r <- peekByteIndex index arrContents
657+ r <- peekAt index arrContents
549658 let (x, res) = f r
550− pokeByteIndex index arrContents x
659+ pokeAt index arrContents x
551660 return res
552661
553662-- | Modify a given index of an array using a modifier function.
@@ -559,9 +668,9 @@ modifyIndex i MutArray{..} f = do
559668 let index = INDEX_OF(arrStart,i,a)
560669 if i >= 0 && INDEX_VALID(index,arrEnd,a)
561670 then liftIO $ do
562− r <- peekByteIndex index arrContents
671+ r <- peekAt index arrContents
563672 let (x, res) = f r
564− pokeByteIndex index arrContents x
673+ pokeAt index arrContents x
565674 return res
566675 else invalidIndex "modifyIndex" i
567676
@@ -595,8 +704,8 @@ modify MutArray{..} f = liftIO $
595704
596705 go i =
597706 when (INDEX_VALID(i,arrEnd,a)) $ do
598− r <- peekByteIndex i arrContents
599− pokeByteIndex i arrContents (f r)
707+ r <- peekAt i arrContents
708+ pokeAt i arrContents (f r)
600709 go (INDEX_NEXT(i,a))
601710
602711-- XXX We could specify the number of bytes to swap instead of Proxy. Need
@@ -605,15 +714,15 @@ modify MutArray{..} f = liftIO $
605714swapArrayByteIndices ::
606715 forall a. Unbox a
607716 => Proxy a
608− -> MutableByteArray
717+ -> MutByteArray
609718 -> Int
610719 -> Int
611720 -> IO ()
612721swapArrayByteIndices _ arrContents i1 i2 = do
613− r1 <- peekByteIndex i1 arrContents
614− r2 <- peekByteIndex i2 arrContents
615− pokeByteIndex i1 arrContents (r2 :: a)
616− pokeByteIndex i2 arrContents (r1 :: a)
722+ r1 <- peekAt i1 arrContents
723+ r2 <- peekAt i2 arrContents
724+ pokeAt i1 arrContents (r2 :: a)
725+ pokeAt i2 arrContents (r1 :: a)
617726
618727-- | Swap the elements at two indices without validating the indices.
619728--
@@ -729,26 +838,23 @@ arrayChunkBytes = 1024
729838roundDownTo :: Int -> Int -> Int
730839roundDownTo elemSize size = size - (size `mod` elemSize)
731840
732−-- XXX See if resizing can be implemented by reading the old array as a stream
733−-- and then using writeN to the new array.
734−--
735841-- NOTE: we are passing elemSize explicitly to avoid an Unboxed constraint.
736842-- Since this is not inlined Unboxed consrraint leads to dictionary passing
737843-- which complicates some inspection tests.
738844--
739−{-# NOINLINE reallocExplicit #-}
740−reallocExplicit :: Int -> Int -> MutArray a -> IO (MutArray a)
741−reallocExplicit elemSize newCapacityInBytes MutArray{..} = do
845+{-# NOINLINE reallocExplicitAs #-}
846+reallocExplicitAs :: PinnedState -> Int -> Int -> MutArray a -> IO (MutArray a)
847+reallocExplicitAs ps elemSize newCapacityInBytes MutArray{..} = do
742848 assertM(arrEnd <= arrBound)
743849
744850 -- Allocate new array
745851 let newCapMaxInBytes = roundUpLargeArray newCapacityInBytes
746852 contents <-
747− if Unboxed.isPinned arrContents
748− then Unboxed.pinnedNewBytes newCapMaxInBytes
749− else Unboxed.newBytes newCapMaxInBytes
750− let !(MutableByteArray mbarrFrom#) = arrContents
751− !(MutableByteArray mbarrTo#) = contents
853+ if ps == Pinned
854+ then Unboxed.pinnedNew newCapMaxInBytes
855+ else Unboxed.new newCapMaxInBytes
856+ let !(MutByteArray mbarrFrom#) = arrContents
857+ !(MutByteArray mbarrTo#) = contents
752858
753859 -- Copy old data
754860 let oldStart = arrStart
@@ -769,6 +875,10 @@ reallocExplicit elemSize newCapacityInBytes MutArray{..} = do
769875 , arrBound = newCapInBytes
770876 }
771877
878+-- XXX Should these be called reallocBytes etc?
879+-- XXX We may also need reallocAs to allocate as pinned/unpinned explicitly. In
880+-- fact clone/pinnedClone can be implemented using reallocAs.
881+
772882-- | @realloc newCapacity array@ reallocates the array to the specified
773883-- capacity in bytes.
774884--
@@ -780,7 +890,12 @@ reallocExplicit elemSize newCapacityInBytes MutArray{..} = do
780890-- If the original array is pinned, the newly allocated array is also pinned.
781891{-# INLINABLE realloc #-}
782892realloc :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
783−realloc bytes arr = liftIO $ reallocExplicit (SIZE_OF(a)) bytes arr
893+realloc bytes arr =
894+ let ps =
895+ if isPinned arr
896+ then Pinned
897+ else Unpinned
898+ in liftIO $ reallocExplicitAs ps (SIZE_OF(a)) bytes arr
784899
785900-- | @reallocWith label capSizer minIncrBytes array@. The label is used
786901-- in error messages and the capSizer is used to determine the capacity of the
@@ -803,14 +918,14 @@ reallocWith label capSizer minIncrBytes arr = do
803918 where
804919
805920 badSize newSize =
806− concat
921+ Prelude.concat
807922 [ label
808923 , ": new array size (in bytes) is less than required size "
809924 , show newSize
810925 , ". Please check the sizing function passed."
811926 ]
812927
813−-- | @resize newCapacity array@ changes the total capacity of the array so that
928+-- | @grow newCapacity array@ changes the total capacity of the array so that
814929-- it is enough to hold the specified number of elements. Nothing is done if
815930-- the specified capacity is less than the length of the array.
816931--
@@ -818,34 +933,46 @@ reallocWith label capSizer minIncrBytes arr = do
818933-- the block size (4K).
819934--
820935-- /Pre-release/
821−{-# INLINE resize #-}
822−resize :: forall m a. (MonadIO m, Unbox a) =>
936+{-# INLINE grow #-}
937+grow :: forall m a. (MonadIO m, Unbox a) =>
823938 Int -> MutArray a -> m (MutArray a)
824−resize nElems arr@MutArray{..} = do
939+grow nElems arr@MutArray{..} = do
825940 let req = SIZE_OF(a) * nElems
826− len = arrEnd - arrStart
827− if req < len
941+ cap = arrBound - arrStart
942+ if req < cap
828943 then return arr
829944 else realloc req arr
830945
831−-- | Like 'resize' but if the byte capacity is more than 'largeObjectThreshold'
832−-- then it is rounded up to the closest power of 2.
946+{-# DEPRECATED resize "Please use grow instead." #-}
947+{-# INLINE resize #-}
948+resize :: forall m a. (MonadIO m, Unbox a) =>
949+ Int -> MutArray a -> m (MutArray a)
950+resize = grow
951+
952+-- | Like 'grow' but if the requested byte capacity is more than
953+-- 'largeObjectThreshold' then it is rounded up to the closest power of 2.
833954--
834955-- /Pre-release/
835−{-# INLINE resizeExp #-}
836−resizeExp :: forall m a. (MonadIO m, Unbox a) =>
956+{-# INLINE growExp #-}
957+growExp :: forall m a. (MonadIO m, Unbox a) =>
837958 Int -> MutArray a -> m (MutArray a)
838−resizeExp nElems arr@MutArray{..} = do
959+growExp nElems arr@MutArray{..} = do
839960 let req = roundUpLargeArray (SIZE_OF(a) * nElems)
840961 req1 =
841962 if req > largeObjectThreshold
842963 then roundUpToPower2 req
843964 else req
844− len = arrEnd - arrStart
845− if req1 < len
965+ cap = arrBound - arrStart
966+ if req1 < cap
846967 then return arr
847968 else realloc req1 arr
848969
970+{-# DEPRECATED resizeExp "Please use growExp instead." #-}
971+{-# INLINE resizeExp #-}
972+resizeExp :: forall m a. (MonadIO m, Unbox a) =>
973+ Int -> MutArray a -> m (MutArray a)
974+resizeExp = growExp
975+
849976-- | Resize the allocated memory to drop any reserved free space at the end of
850977-- the array and reallocate it to reduce wastage.
851978--
@@ -890,7 +1017,7 @@ rightSize arr@MutArray{..} = do
8901017snocNewEnd :: (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m (MutArray a)
8911018snocNewEnd newEnd arr@MutArray{..} x = liftIO $ do
8921019 assert (newEnd <= arrBound) (return ())
893− pokeByteIndex arrEnd arrContents x
1020+ pokeAt arrEnd arrContents x
8941021 return $ arr {arrEnd = newEnd}
8951022
8961023-- | Really really unsafe, appends the element into the first array, may
@@ -910,7 +1037,7 @@ snocUnsafe arr@MutArray{..} = snocNewEnd (INDEX_NEXT(arrEnd,a)) arr
9101037{-# INLINE snocMay #-}
9111038snocMay :: forall m a. (MonadIO m, Unbox a) =>
9121039 MutArray a -> a -> m (Maybe (MutArray a))
913−snocMay arr@MutArray{..} x = liftIO $ do
1040+snocMay arr@MutArray{..} x = do
9141041 let newEnd = INDEX_NEXT(arrEnd,a)
9151042 if newEnd <= arrBound
9161043 then Just <$> snocNewEnd newEnd arr x
@@ -924,7 +1051,7 @@ snocWithRealloc :: forall m a. (MonadIO m, Unbox a) =>
9241051 -> a
9251052 -> m (MutArray a)
9261053snocWithRealloc sizer arr x = do
927− arr1 <- liftIO $ reallocWith "snocWith" sizer (SIZE_OF(a)) arr
1054+ arr1 <- reallocWith "snocWith" sizer (SIZE_OF(a)) arr
9281055 snocUnsafe arr1 x
9291056
9301057-- | @snocWith sizer arr elem@ mutates @arr@ to append @elem@. The length of
@@ -946,7 +1073,7 @@ snocWith :: forall m a. (MonadIO m, Unbox a) =>
9461073 -> MutArray a
9471074 -> a
9481075 -> m (MutArray a)
949−snocWith allocSize arr x = liftIO $ do
1076+snocWith allocSize arr x = do
9501077 let newEnd = INDEX_NEXT(arrEnd arr,a)
9511078 if newEnd <= arrBound arr
9521079 then snocNewEnd newEnd arr x
@@ -990,6 +1117,126 @@ snoc = snocWith f
9901117 then oldSize * 2
9911118 else roundUpToPower2 oldSize * 2
9921119
1120+-------------------------------------------------------------------------------
1121+-- Serialization/Deserialization
1122+-------------------------------------------------------------------------------
1123+
1124+{-# INLINE pokeNewEnd #-}
1125+pokeNewEnd :: (MonadIO m, Unbox a) =>
1126+ Int -> MutArray Word8 -> a -> m (MutArray Word8)
1127+pokeNewEnd newEnd arr@MutArray{..} x = liftIO $ do
1128+ assert (newEnd <= arrBound) (return ())
1129+ liftIO $ pokeAt arrEnd arrContents x
1130+ return $ arr {arrEnd = newEnd}
1131+
1132+-- | Really really unsafe, unboxes a Haskell type and appends the resulting
1133+-- bytes to the byte array, may cause silent data corruption or if you are
1134+-- lucky a segfault if the array does not have enough space to append the
1135+-- element.
1136+--
1137+-- /Internal/
1138+{-# INLINE pokeAppendUnsafe #-}
1139+pokeAppendUnsafe :: forall m a. (MonadIO m, Unbox a) =>
1140+ MutArray Word8 -> a -> m (MutArray Word8)
1141+pokeAppendUnsafe arr@MutArray{..} = pokeNewEnd (arrEnd + SIZE_OF(a)) arr
1142+
1143+-- | Skip the specified number of bytes in the array. The data in the skipped
1144+-- region remains uninitialzed.
1145+{-# INLINE pokeSkipUnsafe #-}
1146+pokeSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
1147+pokeSkipUnsafe n arr@MutArray{..} = do
1148+ let newEnd = arrEnd + n
1149+ in assert (newEnd <= arrBound) (arr {arrEnd = newEnd})
1150+
1151+-- | Like 'pokeAppend' but does not grow the array when pre-allocated array
1152+-- capacity becomes full.
1153+--
1154+-- /Internal/
1155+{-# INLINE pokeAppendMay #-}
1156+pokeAppendMay :: forall m a. (MonadIO m, Unbox a) =>
1157+ MutArray Word8 -> a -> m (Maybe (MutArray Word8))
1158+pokeAppendMay arr@MutArray{..} x = liftIO $ do
1159+ let newEnd = arrEnd + SIZE_OF(a)
1160+ if newEnd <= arrBound
1161+ then Just <$> pokeNewEnd newEnd arr x
1162+ else return Nothing
1163+
1164+{-# NOINLINE pokeWithRealloc #-}
1165+pokeWithRealloc :: forall m a. (MonadIO m, Unbox a) =>
1166+ (Int -> Int)
1167+ -> MutArray Word8
1168+ -> a
1169+ -> m (MutArray Word8)
1170+pokeWithRealloc sizer arr x = do
1171+ arr1 <- liftIO $ reallocWith "pokeWithRealloc" sizer (SIZE_OF(a)) arr
1172+ pokeAppendUnsafe arr1 x
1173+
1174+{-# INLINE pokeAppendWith #-}
1175+pokeAppendWith :: forall m a. (MonadIO m, Unbox a) =>
1176+ (Int -> Int)
1177+ -> MutArray Word8
1178+ -> a
1179+ -> m (MutArray Word8)
1180+pokeAppendWith allocSize arr x = liftIO $ do
1181+ let newEnd = arrEnd arr + SIZE_OF(a)
1182+ if newEnd <= arrBound arr
1183+ then pokeNewEnd newEnd arr x
1184+ else pokeWithRealloc allocSize arr x
1185+
1186+-- | Unbox a Haskell type and append the resulting bytes to a mutable byte
1187+-- array. The array is grown exponentially when more space is needed.
1188+--
1189+-- Definition:
1190+--
1191+-- >>> pokeAppend arr x = MutArray.castUnsafe <$> MutArray.snoc (MutArray.castUnsafe arr) x
1192+--
1193+{-# INLINE pokeAppend #-}
1194+pokeAppend :: forall m a. (MonadIO m, Unbox a) =>
1195+ MutArray Word8 -> a -> m (MutArray Word8)
1196+pokeAppend = pokeAppendWith f
1197+
1198+ where
1199+
1200+ f oldSize =
1201+ if isPower2 oldSize
1202+ then oldSize * 2
1203+ else roundUpToPower2 oldSize * 2
1204+
1205+-- | Really really unsafe, create a Haskell value from an unboxed byte array,
1206+-- does not check if the array is big enough, may return garbage or if you are
1207+-- lucky may cause a segfault.
1208+--
1209+-- /Internal/
1210+{-# INLINE peekUnconsUnsafe #-}
1211+peekUnconsUnsafe :: forall m a. (MonadIO m, Unbox a) =>
1212+ MutArray Word8 -> m (a, MutArray Word8)
1213+peekUnconsUnsafe MutArray{..} = do
1214+ let start1 = arrStart + SIZE_OF(a)
1215+ assert (start1 <= arrEnd) (return ())
1216+ liftIO $ do
1217+ r <- peekAt arrStart arrContents
1218+ return (r, MutArray arrContents start1 arrEnd arrBound)
1219+
1220+-- | Discard the specified number of bytes in the array.
1221+{-# INLINE peekSkipUnsafe #-}
1222+peekSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
1223+peekSkipUnsafe n MutArray{..} =
1224+ let start1 = arrStart + n
1225+ in assert (start1 <= arrEnd) (MutArray arrContents start1 arrEnd arrBound)
1226+
1227+-- | Create a Haskell value from its unboxed representation from the head of a
1228+-- byte array, return the value and the remaining array.
1229+{-# INLINE peekUncons #-}
1230+peekUncons :: forall m a. (MonadIO m, Unbox a) =>
1231+ MutArray Word8 -> m (Maybe a, MutArray Word8)
1232+peekUncons arr@MutArray{..} = do
1233+ let start1 = arrStart + SIZE_OF(a)
1234+ if start1 > arrEnd
1235+ then return (Nothing, arr)
1236+ else liftIO $ do
1237+ r <- peekAt arrStart arrContents
1238+ return (Just r, MutArray arrContents start1 arrEnd arrBound)
1239+
9931240-------------------------------------------------------------------------------
9941241-- Random reads
9951242-------------------------------------------------------------------------------
@@ -1004,7 +1251,7 @@ getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
10041251getIndexUnsafe i MutArray{..} = do
10051252 let index = INDEX_OF(arrStart,i,a)
10061253 assert (i >= 0 && INDEX_VALID(index,arrEnd,a)) (return ())
1007− liftIO $ peekByteIndex index arrContents
1254+ liftIO $ peekAt index arrContents
10081255
10091256-- | /O(1)/ Lookup the element at the given index. Index starts from 0.
10101257--
@@ -1013,7 +1260,7 @@ getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
10131260getIndex i MutArray{..} = do
10141261 let index = INDEX_OF(arrStart,i,a)
10151262 if i >= 0 && INDEX_VALID(index,arrEnd,a)
1016− then liftIO $ Just <$> peekByteIndex index arrContents
1263+ then liftIO $ Just <$> peekAt index arrContents
10171264 else return Nothing
10181265
10191266-- | /O(1)/ Lookup the element at the given index from the end of the array.
@@ -1026,21 +1273,16 @@ getIndexRev :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
10261273getIndexRev i MutArray{..} = do
10271274 let index = RINDEX_OF(arrEnd,i,a)
10281275 if i >= 0 && index >= arrStart
1029− then liftIO $ peekByteIndex index arrContents
1276+ then liftIO $ peekAt index arrContents
10301277 else invalidIndex "getIndexRev" i
10311278
10321279data GetIndicesState contents start end st =
10331280 GetIndicesState contents start end st
10341281
1035−-- | Given an unfold that generates array indices, read the elements on those
1036−-- indices from the supplied MutArray. An error is thrown if an index is out of
1037−-- bounds.
1038−--
1039−-- /Pre-release/
1040−{-# INLINE getIndicesD #-}
1041−getIndicesD :: (Monad m, Unbox a) =>
1282+{-# INLINE indexReaderWith #-}
1283+indexReaderWith :: (Monad m, Unbox a) =>
10421284 (forall b. IO b -> m b) -> D.Stream m Int -> Unfold m (MutArray a) a
1043−getIndicesD liftio (D.Stream stepi sti) = Unfold step inject
1285+indexReaderWith liftio (D.Stream stepi sti) = Unfold step inject
10441286
10451287 where
10461288
@@ -1059,9 +1301,26 @@ getIndicesD liftio (D.Stream stepi sti) = Unfold step inject
10591301 D.Skip s -> return $ D.Skip (GetIndicesState contents start end s)
10601302 D.Stop -> return D.Stop
10611303
1304+{-# DEPRECATED getIndicesWith "Please use indexReaderWith instead." #-}
1305+{-# INLINE getIndicesWith #-}
1306+getIndicesWith :: (Monad m, Unbox a) =>
1307+ (forall b. IO b -> m b) -> D.Stream m Int -> Unfold m (MutArray a) a
1308+getIndicesWith = indexReaderWith
1309+
1310+-- | Given an unfold that generates array indices, read the elements on those
1311+-- indices from the supplied MutArray. An error is thrown if an index is out of
1312+-- bounds.
1313+--
1314+-- /Pre-release/
1315+{-# INLINE indexReader #-}
1316+indexReader :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1317+indexReader = indexReaderWith liftIO
1318+
1319+-- XXX DO NOT REMOVE, change the signature to use Stream instead of unfold
1320+{-# DEPRECATED getIndices "Please use indexReader instead." #-}
10621321{-# INLINE getIndices #-}
10631322getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1064−getIndices = getIndicesD liftIO
1323+getIndices = indexReader
10651324
10661325-------------------------------------------------------------------------------
10671326-- Subarrays
@@ -1166,7 +1425,7 @@ partitionBy f arr@MutArray{..} = liftIO $ do
11661425
11671426 -- Invariant low < high on entry, and on return as well
11681427 moveHigh low high = do
1169− h <- peekByteIndex high arrContents
1428+ h <- peekAt high arrContents
11701429 if f h
11711430 then
11721431 -- Correctly classified, continue the loop
@@ -1186,7 +1445,7 @@ partitionBy f arr@MutArray{..} = liftIO $ do
11861445 -- low <= high
11871446 -- Both low and high are valid locations within the array
11881447 go low high = do
1189− l <- peekByteIndex low arrContents
1448+ l <- peekAt low arrContents
11901449 if f l
11911450 then
11921451 -- low is wrongly classified
@@ -1197,8 +1456,8 @@ partitionBy f arr@MutArray{..} = liftIO $ do
11971456 case r of
11981457 Nothing -> return low
11991458 Just (high1, h) -> do -- low < high1
1200− pokeByteIndex low arrContents h
1201− pokeByteIndex high1 arrContents l
1459+ pokeAt low arrContents h
1460+ pokeAt high1 arrContents l
12021461 let low1 = INDEX_NEXT(low,a)
12031462 high2 = INDEX_PREV(high1,a)
12041463 if low1 <= high2
@@ -1332,7 +1591,7 @@ chunksOfAs ps n (D.Stream step state) =
13321591 r <- step (adaptState gst) st
13331592 case r of
13341593 D.Yield x s -> do
1335− liftIO $ pokeByteIndex end contents x
1594+ liftIO $ pokeAt end contents x
13361595 let end1 = INDEX_NEXT(end,a)
13371596 return $
13381597 if end1 >= bound
@@ -1351,10 +1610,12 @@ chunksOfAs ps n (D.Stream step state) =
13511610
13521611 step' _ GroupFinish = return D.Stop
13531612
1354−-- | @chunksOf n stream@ groups the input stream into a stream of
1355−-- arrays of size n.
1613+-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
1614+-- @n@ elements each.
1615+--
1616+-- Same as the following but may be more efficient:
13561617--
1357−-- @chunksOf n = StreamD.foldMany (MutArray.writeN n)@
1618+-- >>> chunksOf n = Stream.foldMany (MutArray.createOf n)
13581619--
13591620-- /Pre-release/
13601621{-# INLINE_NORMAL chunksOf #-}
@@ -1363,17 +1624,29 @@ chunksOf :: forall m a. (MonadIO m, Unbox a)
13631624-- XXX the idiomatic implementation leads to large regression in the D.reverse'
13641625-- benchmark. It seems it has difficulty producing optimized code when
13651626-- converting to StreamK. Investigate GHC optimizations.
1366−-- chunksOf n = D.foldMany (writeN n)
1627+-- chunksOf n = D.foldMany (createOf n)
13671628chunksOf = chunksOfAs Unpinned
13681629
13691630-- | Like 'chunksOf' but creates pinned arrays.
13701631{-# INLINE_NORMAL pinnedChunksOf #-}
13711632pinnedChunksOf :: forall m a. (MonadIO m, Unbox a)
13721633 => Int -> D.Stream m a -> D.Stream m (MutArray a)
1373−-- pinnedChunksOf n = D.foldMany (pinnedWriteN n)
1634+-- pinnedChunksOf n = D.foldMany (pinnedCreateOf n)
13741635pinnedChunksOf = chunksOfAs Pinned
13751636
1376−-- XXX This should take a PinnedState
1637+-- | When we are buffering a stream of unknown size into an array we do not
1638+-- know how much space to pre-allocate. So we start with the min size and emit
1639+-- the array then keep on doubling the size every time. Thus we do not need to
1640+-- guess the optimum chunk size.
1641+--
1642+-- We can incorporate this in chunksOfAs if the additional size parameter does
1643+-- not impact perf.
1644+--
1645+{-# INLINE _chunksOfRange #-}
1646+_chunksOfRange :: -- (MonadIO m, Unbox a) =>
1647+ PinnedState -> Int -> Int -> D.Stream m a -> D.Stream m (MutArray a)
1648+_chunksOfRange _ps _low _hi = undefined
1649+
13771650-- XXX buffer to a list instead?
13781651-- | Buffer the stream into arrays in memory.
13791652{-# INLINE arrayStreamKFromStreamDAs #-}
@@ -1391,16 +1664,10 @@ data FlattenState s contents a =
13911664 OuterLoop s
13921665 | InnerLoop s contents !Int !Int
13931666
1394−-- | Use the "reader" unfold instead.
1395−--
1396−-- @flattenArrays = unfoldMany reader@
1397−--
1398−-- We can try this if there are any fusion issues in the unfold.
1399−--
1400−{-# INLINE_NORMAL flattenArrays #-}
1401−flattenArrays :: forall m a. (MonadIO m, Unbox a)
1402− => D.Stream m (MutArray a) -> D.Stream m a
1403−flattenArrays (D.Stream step state) = D.Stream step' (OuterLoop state)
1667+{-# INLINE_NORMAL concatWith #-}
1668+concatWith :: forall m a. (Monad m, Unbox a)
1669+ => (forall b. IO b -> m b) -> D.Stream m (MutArray a) -> D.Stream m a
1670+concatWith liftio (D.Stream step state) = D.Stream step' (OuterLoop state)
14041671
14051672 where
14061673
@@ -1417,19 +1684,30 @@ flattenArrays (D.Stream step state) = D.Stream step' (OuterLoop state)
14171684 return $ D.Skip $ OuterLoop st
14181685
14191686 step' _ (InnerLoop st contents p end) = do
1420− x <- liftIO $ peekByteIndex p contents
1687+ x <- liftio $ peekAt p contents
14211688 return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
14221689
1423−-- | Use the "readerRev" unfold instead.
1690+-- | Use the "reader" unfold instead.
14241691--
1425−-- @flattenArrays = unfoldMany readerRev@
1692+-- @concat = unfoldMany reader@
14261693--
14271694-- We can try this if there are any fusion issues in the unfold.
14281695--
1429−{-# INLINE_NORMAL flattenArraysRev #-}
1430−flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
1696+{-# INLINE_NORMAL concat #-}
1697+concat :: forall m a. (MonadIO m, Unbox a)
14311698 => D.Stream m (MutArray a) -> D.Stream m a
1432−flattenArraysRev (D.Stream step state) = D.Stream step' (OuterLoop state)
1699+concat = concatWith liftIO
1700+
1701+{-# DEPRECATED flattenArrays "Please use \"unfoldMany reader\" instead." #-}
1702+{-# INLINE flattenArrays #-}
1703+flattenArrays :: forall m a. (MonadIO m, Unbox a)
1704+ => D.Stream m (MutArray a) -> D.Stream m a
1705+flattenArrays = concat
1706+
1707+{-# INLINE_NORMAL concatRevWith #-}
1708+concatRevWith :: forall m a. (Monad m, Unbox a)
1709+ => (forall b. IO b -> m b) -> D.Stream m (MutArray a) -> D.Stream m a
1710+concatRevWith liftio (D.Stream step state) = D.Stream step' (OuterLoop state)
14331711
14341712 where
14351713
@@ -1447,16 +1725,33 @@ flattenArraysRev (D.Stream step state) = D.Stream step' (OuterLoop state)
14471725 return $ D.Skip $ OuterLoop st
14481726
14491727 step' _ (InnerLoop st contents p start) = do
1450− x <- liftIO $ peekByteIndex p contents
1728+ x <- liftio $ peekAt p contents
14511729 let cur = INDEX_PREV(p,a)
14521730 return $ D.Yield x (InnerLoop st contents cur start)
14531731
1732+-- | Use the "readerRev" unfold instead.
1733+--
1734+-- @concat = unfoldMany readerRev@
1735+--
1736+-- We can try this if there are any fusion issues in the unfold.
1737+--
1738+{-# INLINE_NORMAL concatRev #-}
1739+concatRev :: forall m a. (MonadIO m, Unbox a)
1740+ => D.Stream m (MutArray a) -> D.Stream m a
1741+concatRev = concatRevWith liftIO
1742+
1743+{-# DEPRECATED flattenArraysRev "Please use \"unfoldMany readerRev\" instead." #-}
1744+{-# INLINE flattenArraysRev #-}
1745+flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
1746+ => D.Stream m (MutArray a) -> D.Stream m a
1747+flattenArraysRev = concatRev
1748+
14541749-------------------------------------------------------------------------------
14551750-- Unfolds
14561751-------------------------------------------------------------------------------
14571752
14581753data ArrayUnsafe a = ArrayUnsafe
1459− {-# UNPACK #-} !MutableByteArray -- contents
1754+ {-# UNPACK #-} !MutByteArray -- contents
14601755 {-# UNPACK #-} !Int -- index 1
14611756 {-# UNPACK #-} !Int -- index 2
14621757
@@ -1486,7 +1781,7 @@ producerWith liftio = Producer step (return . toArrayUnsafe) extract
14861781 -- few actions for correctness and execution order sanity. We want
14871782 -- the peek to occur right here and not lazily at some later point
14881783 -- because we want the peek to be ordered with respect to the touch.
1489− !x <- liftio $ peekByteIndex cur contents
1784+ !x <- liftio $ peekAt cur contents
14901785 return $ D.Yield x (ArrayUnsafe contents (INDEX_NEXT(cur,a)) end)
14911786
14921787 extract = return . fromArrayUnsafe
@@ -1517,7 +1812,7 @@ readerRevWith liftio = Unfold step inject
15171812 {-# INLINE_LATE step #-}
15181813 step (ArrayUnsafe _ start p) | p < start = return D.Stop
15191814 step (ArrayUnsafe contents start p) = do
1520− !x <- liftio $ peekByteIndex p contents
1815+ !x <- liftio $ peekAt p contents
15211816 return $ D.Yield x (ArrayUnsafe contents start (INDEX_PREV(p,a)))
15221817
15231818-- | Unfold an array into a stream in reverse order.
@@ -1548,7 +1843,7 @@ toListFB c n MutArray{..} = go arrStart
15481843 -- evaluated/written to before we peek at them.
15491844 -- XXX
15501845 let !x = unsafeInlineIO $ do
1551− r <- peekByteIndex arrContents p
1846+ r <- peekAt arrContents p
15521847 return r
15531848 in c x (go (PTR_NEXT(p,a)))
15541849-}
@@ -1565,21 +1860,21 @@ toList MutArray{..} = liftIO $ go arrStart
15651860
15661861 go p | assert (p <= arrEnd) (p == arrEnd) = return []
15671862 go p = do
1568− x <- peekByteIndex p arrContents
1863+ x <- peekAt p arrContents
15691864 (:) x <$> go (INDEX_NEXT(p,a))
15701865
1571−{-# INLINE_NORMAL toStreamDWith #-}
1572−toStreamDWith ::
1866+{-# INLINE_NORMAL toStreamWith #-}
1867+toStreamWith ::
15731868 forall m a. (Monad m, Unbox a)
15741869 => (forall b. IO b -> m b) -> MutArray a -> D.Stream m a
1575−toStreamDWith liftio MutArray{..} = D.Stream step arrStart
1870+toStreamWith liftio MutArray{..} = D.Stream step arrStart
15761871
15771872 where
15781873
15791874 {-# INLINE_LATE step #-}
15801875 step _ p | assert (p <= arrEnd) (p == arrEnd) = return D.Stop
15811876 step _ p = liftio $ do
1582− r <- peekByteIndex p arrContents
1877+ r <- peekAt p arrContents
15831878 return $ D.Yield r (INDEX_NEXT(p,a))
15841879
15851880-- | Convert a 'MutArray' into a stream.
@@ -1588,7 +1883,7 @@ toStreamDWith liftio MutArray{..} = D.Stream step arrStart
15881883--
15891884{-# INLINE_NORMAL read #-}
15901885read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> D.Stream m a
1591−read = toStreamDWith liftIO
1886+read = toStreamWith liftIO
15921887
15931888{-# INLINE toStreamKWith #-}
15941889toStreamKWith ::
@@ -1600,18 +1895,18 @@ toStreamKWith liftio MutArray{..} = go arrStart
16001895
16011896 go p | assert (p <= arrEnd) (p == arrEnd) = K.nil
16021897 | otherwise =
1603− let elemM = peekByteIndex p arrContents
1898+ let elemM = peekAt p arrContents
16041899 in liftio elemM `K.consM` go (INDEX_NEXT(p,a))
16051900
16061901{-# INLINE toStreamK #-}
16071902toStreamK :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
16081903toStreamK = toStreamKWith liftIO
16091904
1610−{-# INLINE_NORMAL toStreamDRevWith #-}
1611−toStreamDRevWith ::
1905+{-# INLINE_NORMAL toStreamRevWith #-}
1906+toStreamRevWith ::
16121907 forall m a. (Monad m, Unbox a)
16131908 => (forall b. IO b -> m b) -> MutArray a -> D.Stream m a
1614−toStreamDRevWith liftio MutArray{..} =
1909+toStreamRevWith liftio MutArray{..} =
16151910 let p = INDEX_PREV(arrEnd,a)
16161911 in D.Stream step p
16171912
@@ -1620,7 +1915,7 @@ toStreamDRevWith liftio MutArray{..} =
16201915 {-# INLINE_LATE step #-}
16211916 step _ p | p < arrStart = return D.Stop
16221917 step _ p = liftio $ do
1623− r <- peekByteIndex p arrContents
1918+ r <- peekAt p arrContents
16241919 return $ D.Yield r (INDEX_PREV(p,a))
16251920
16261921-- | Convert a 'MutArray' into a stream in reverse order.
@@ -1629,7 +1924,7 @@ toStreamDRevWith liftio MutArray{..} =
16291924--
16301925{-# INLINE_NORMAL readRev #-}
16311926readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> D.Stream m a
1632−readRev = toStreamDRevWith liftIO
1927+readRev = toStreamRevWith liftIO
16331928
16341929{-# INLINE toStreamKRevWith #-}
16351930toStreamKRevWith ::
@@ -1643,7 +1938,7 @@ toStreamKRevWith liftio MutArray {..} =
16431938
16441939 go p | p < arrStart = K.nil
16451940 | otherwise =
1646− let elemM = peekByteIndex p arrContents
1941+ let elemM = peekAt p arrContents
16471942 in liftio elemM `K.consM` go (INDEX_PREV(p,a))
16481943
16491944{-# INLINE toStreamKRev #-}
@@ -1678,7 +1973,7 @@ foldr f z arr = D.foldr f z $ read arr
16781973
16791974-- XXX Keep the bound intact to not lose any free space? Perf impact?
16801975
1681−-- | @writeAppendNUnsafe n arr@ appends up to @n@ input items to the supplied
1976+-- | @unsafeAppendN n arr@ appends up to @n@ input items to the supplied
16821977-- array.
16831978--
16841979-- Unsafe: Do not drive the fold beyond @n@ elements, it will lead to memory
@@ -1687,13 +1982,12 @@ foldr f z arr = D.foldr f z $ read arr
16871982-- Any free space left in the array after appending @n@ elements is lost.
16881983--
16891984-- /Internal/
1690−{-# INLINE_NORMAL writeAppendNUnsafe #-}
1691−writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) =>
1985+{-# INLINE_NORMAL unsafeAppendN #-}
1986+unsafeAppendN :: forall m a. (MonadIO m, Unbox a) =>
16921987 Int
16931988 -> m (MutArray a)
16941989 -> Fold m a (MutArray a)
1695−writeAppendNUnsafe n action =
1696− fmap fromArrayUnsafe $ FL.foldlM' step initial
1990+unsafeAppendN n action = fmap fromArrayUnsafe $ FL.foldlM' step initial
16971991
16981992 where
16991993
@@ -1706,38 +2000,59 @@ writeAppendNUnsafe n action =
17062000 -- otherwise we lose that space.
17072001 arr1 <-
17082002 if free < needed
1709− then noinline reallocWith "writeAppendNUnsafeWith" (+ needed) needed arr
2003+ then noinline reallocWith "unsafeAppendN" (+ needed) needed arr
17102004 else return arr
17112005 return $ toArrayUnsafe arr1
17122006
17132007 step (ArrayUnsafe contents start end) x = do
1714− liftIO $ pokeByteIndex end contents x
2008+ liftIO $ pokeAt end contents x
17152009 return $ ArrayUnsafe contents start (INDEX_NEXT(end,a))
17162010
2011+{-# DEPRECATED writeAppendNUnsafe "Please use unsafeAppendN instead." #-}
2012+{-# INLINE writeAppendNUnsafe #-}
2013+writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) =>
2014+ Int
2015+ -> m (MutArray a)
2016+ -> Fold m a (MutArray a)
2017+writeAppendNUnsafe = unsafeAppendN
2018+
17172019-- | Append @n@ elements to an existing array. Any free space left in the array
17182020-- after appending @n@ elements is lost.
17192021--
1720−-- >>> writeAppendN n initial = Fold.take n (MutArray.writeAppendNUnsafe n initial)
2022+-- >>> appendN n initial = Fold.take n (MutArray.unsafeAppendN n initial)
17212023--
1722−{-# INLINE_NORMAL writeAppendN #-}
2024+{-# INLINE_NORMAL appendN #-}
2025+appendN :: forall m a. (MonadIO m, Unbox a) =>
2026+ Int -> m (MutArray a) -> Fold m a (MutArray a)
2027+appendN n initial = FL.take n (unsafeAppendN n initial)
2028+
2029+-- XXX Deprecate in major
2030+-- {-# DEPRECATED writeAppendN "Please use appendN instead." #-}
2031+{-# INLINE writeAppendN #-}
17232032writeAppendN :: forall m a. (MonadIO m, Unbox a) =>
17242033 Int -> m (MutArray a) -> Fold m a (MutArray a)
1725−writeAppendN n initial = FL.take n (writeAppendNUnsafe n initial)
2034+writeAppendN = appendN
17262035
1727−-- | @writeAppendWith realloc action@ mutates the array generated by @action@ to
2036+-- | @appendWith realloc action@ mutates the array generated by @action@ to
17282037-- append the input stream. If there is no reserved space available in the
17292038-- array it is reallocated to a size in bytes determined by @realloc oldSize@,
17302039-- where @oldSize@ is the current size of the array in bytes.
17312040--
17322041-- Note that the returned array may be a mutated version of original array.
17332042--
1734−-- >>> writeAppendWith sizer = Fold.foldlM' (MutArray.snocWith sizer)
2043+-- >>> appendWith sizer = Fold.foldlM' (MutArray.snocWith sizer)
17352044--
17362045-- /Pre-release/
2046+{-# INLINE appendWith #-}
2047+appendWith :: forall m a. (MonadIO m, Unbox a) =>
2048+ (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
2049+appendWith sizer = FL.foldlM' (snocWith sizer)
2050+
2051+{-# DEPRECATED writeAppendWith "Please use appendWith instead." #-}
17372052{-# INLINE writeAppendWith #-}
17382053writeAppendWith :: forall m a. (MonadIO m, Unbox a) =>
17392054 (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
1740−writeAppendWith sizer = FL.foldlM' (snocWith sizer)
2055+writeAppendWith = appendWith
17412056
17422057-- | @append action@ mutates the array generated by @action@ to append the
17432058-- input stream. If there is no reserved space available in the array it is
@@ -1745,70 +2060,101 @@ writeAppendWith sizer = FL.foldlM' (snocWith sizer)
17452060--
17462061-- Note that the returned array may be a mutated version of original array.
17472062--
1748−-- >>> writeAppend = MutArray.writeAppendWith (* 2)
2063+-- >>> append = MutArray.appendWith (* 2)
17492064--
2065+{-# INLINE append #-}
2066+append :: forall m a. (MonadIO m, Unbox a) =>
2067+ m (MutArray a) -> Fold m a (MutArray a)
2068+append = appendWith (* 2)
2069+
2070+-- XXX Deprecate in major
2071+-- {-# DEPRECATED writeAppend "Please use append instead." #-}
17502072{-# INLINE writeAppend #-}
17512073writeAppend :: forall m a. (MonadIO m, Unbox a) =>
17522074 m (MutArray a) -> Fold m a (MutArray a)
1753−writeAppend = writeAppendWith (* 2)
2075+writeAppend = append
17542076
17552077-- XXX We can carry bound as well in the state to make sure we do not lose the
17562078-- remaining capacity. Need to check perf impact.
17572079--
1758−-- | Like 'writeNUnsafe' but takes a new array allocator @alloc size@ function
1759−-- as argument.
2080+-- | Like 'unsafeCreateOf' but takes a new array allocator @alloc size@
2081+-- function as argument.
17602082--
1761−-- >>> writeNWithUnsafe alloc n = MutArray.writeAppendNUnsafe (alloc n) n
2083+-- >>> unsafeCreateOfWith alloc n = MutArray.unsafeAppendN (alloc n) n
17622084--
17632085-- /Pre-release/
1764−{-# INLINE_NORMAL writeNWithUnsafe #-}
1765−writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a)
2086+{-# INLINE_NORMAL unsafeCreateOfWith #-}
2087+unsafeCreateOfWith :: forall m a. (MonadIO m, Unbox a)
17662088 => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1767−writeNWithUnsafe alloc n = fromArrayUnsafe <$> FL.foldlM' step initial
2089+unsafeCreateOfWith alloc n = fromArrayUnsafe <$> FL.foldlM' step initial
17682090
17692091 where
17702092
17712093 initial = toArrayUnsafe <$> alloc (max n 0)
17722094
17732095 step (ArrayUnsafe contents start end) x = do
1774− liftIO $ pokeByteIndex end contents x
2096+ liftIO $ pokeAt end contents x
17752097 return
17762098 $ ArrayUnsafe contents start (INDEX_NEXT(end,a))
17772099
2100+{-# DEPRECATED writeNWithUnsafe "Please use unsafeCreateOfWith instead." #-}
2101+{-# INLINE writeNWithUnsafe #-}
2102+writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a)
2103+ => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
2104+writeNWithUnsafe = unsafeCreateOfWith
2105+
17782106{-# INLINE_NORMAL writeNUnsafeAs #-}
17792107writeNUnsafeAs :: forall m a. (MonadIO m, Unbox a)
17802108 => PinnedState -> Int -> Fold m a (MutArray a)
1781−writeNUnsafeAs ps = writeNWithUnsafe (newAs ps)
2109+writeNUnsafeAs ps = unsafeCreateOfWith (newAs ps)
17822110
1783−-- | Like 'writeN' but does not check the array bounds when writing. The fold
2111+-- | Like 'createOf' but does not check the array bounds when writing. The fold
17842112-- driver must not call the step function more than 'n' times otherwise it will
17852113-- corrupt the memory and crash. This function exists mainly because any
17862114-- conditional in the step function blocks fusion causing 10x performance
17872115-- slowdown.
17882116--
1789−-- >>> writeNUnsafe = MutArray.writeNWithUnsafe MutArray.new
2117+-- >>> unsafeCreateOf = MutArray.unsafeCreateOfWith MutArray.emptyOf
17902118--
1791−{-# INLINE_NORMAL writeNUnsafe #-}
2119+{-# INLINE_NORMAL unsafeCreateOf #-}
2120+unsafeCreateOf :: forall m a. (MonadIO m, Unbox a)
2121+ => Int -> Fold m a (MutArray a)
2122+unsafeCreateOf = writeNUnsafeAs Unpinned
2123+
2124+{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
2125+{-# INLINE writeNUnsafe #-}
17922126writeNUnsafe :: forall m a. (MonadIO m, Unbox a)
17932127 => Int -> Fold m a (MutArray a)
1794−writeNUnsafe = writeNUnsafeAs Unpinned
2128+writeNUnsafe = unsafeCreateOf
17952129
1796−-- | Like 'writeNUnsafe' but creates a pinned array.
1797−{-# INLINE_NORMAL pinnedWriteNUnsafe #-}
2130+-- | Like 'unsafeCreateOf' but creates a pinned array.
2131+{-# INLINE_NORMAL unsafePinnedCreateOf #-}
2132+unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a)
2133+ => Int -> Fold m a (MutArray a)
2134+unsafePinnedCreateOf = writeNUnsafeAs Pinned
2135+
2136+{-# DEPRECATED pinnedWriteNUnsafe "Please use unsafePinnedCreateOf instead." #-}
2137+{-# INLINE pinnedWriteNUnsafe #-}
17982138pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a)
17992139 => Int -> Fold m a (MutArray a)
1800−pinnedWriteNUnsafe = writeNUnsafeAs Pinned
2140+pinnedWriteNUnsafe = unsafePinnedCreateOf
18012141
1802−-- | @writeNWith alloc n@ folds a maximum of @n@ elements into an array
2142+-- | @createOfWith alloc n@ folds a maximum of @n@ elements into an array
18032143-- allocated using the @alloc@ function.
18042144--
1805−-- >>> writeNWith alloc n = Fold.take n (MutArray.writeNWithUnsafe alloc n)
1806−-- >>> writeNWith alloc n = MutArray.writeAppendN (alloc n) n
2145+-- >>> createOfWith alloc n = Fold.take n (MutArray.unsafeCreateOfWith alloc n)
2146+-- >>> createOfWith alloc n = MutArray.appendN (alloc n) n
18072147--
1808−{-# INLINE_NORMAL writeNWith #-}
2148+{-# INLINE_NORMAL createOfWith #-}
2149+createOfWith :: forall m a. (MonadIO m, Unbox a)
2150+ => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
2151+createOfWith alloc n = FL.take n (unsafeCreateOfWith alloc n)
2152+
2153+-- {-# DEPRECATED writeNWith "Please use createOfWith instead." #-}
2154+{-# INLINE writeNWith #-}
18092155writeNWith :: forall m a. (MonadIO m, Unbox a)
18102156 => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1811−writeNWith alloc n = FL.take n (writeNWithUnsafe alloc n)
2157+writeNWith = createOfWith
18122158
18132159{-# INLINE_NORMAL writeNAs #-}
18142160writeNAs ::
@@ -1816,28 +2162,42 @@ writeNAs ::
18162162 => PinnedState
18172163 -> Int
18182164 -> Fold m a (MutArray a)
1819−writeNAs ps = writeNWith (newAs ps)
2165+writeNAs ps = createOfWith (newAs ps)
18202166
1821−-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
2167+-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
18222168-- 'MutArray'.
18232169--
1824−-- >>> writeN = MutArray.writeNWith MutArray.new
1825−-- >>> writeN n = Fold.take n (MutArray.writeNUnsafe n)
1826−-- >>> writeN n = MutArray.writeAppendN n (MutArray.new n)
2170+-- >>> createOf = MutArray.createOfWith MutArray.new
2171+-- >>> createOf n = Fold.take n (MutArray.unsafeCreateOf n)
2172+-- >>> createOf n = MutArray.appendN n (MutArray.emptyOf n)
18272173--
1828−{-# INLINE_NORMAL writeN #-}
2174+{-# INLINE_NORMAL createOf #-}
2175+createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
2176+createOf = writeNAs Unpinned
2177+
2178+-- XXX Deprecate in major
2179+-- {-# DEPRECATED writeN "Please use createOf instead." #-}
2180+{-# INLINE writeN #-}
18292181writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1830−writeN = writeNAs Unpinned
2182+writeN = createOf
2183+
2184+-- | Like 'createOf' but creates a pinned array.
2185+{-# INLINE_NORMAL pinnedCreateOf #-}
2186+pinnedCreateOf ::
2187+ forall m a. (MonadIO m, Unbox a)
2188+ => Int
2189+ -> Fold m a (MutArray a)
2190+pinnedCreateOf = writeNAs Pinned
18312191
1832−-- | Like 'writeN' but creates a pinned array.
1833−{-# INLINE_NORMAL pinnedWriteN #-}
2192+{-# DEPRECATED pinnedWriteN "Please use pinnedCreateOf instead." #-}
2193+{-# INLINE pinnedWriteN #-}
18342194pinnedWriteN ::
18352195 forall m a. (MonadIO m, Unbox a)
18362196 => Int
18372197 -> Fold m a (MutArray a)
1838−pinnedWriteN = writeNAs Pinned
2198+pinnedWriteN = pinnedCreateOf
18392199
1840−-- | Like writeNWithUnsafe but writes the array in reverse order.
2200+-- | Like unsafeCreateOfWith but writes the array in reverse order.
18412201--
18422202-- /Internal/
18432203{-# INLINE_NORMAL writeRevNWithUnsafe #-}
@@ -1854,11 +2214,11 @@ writeRevNWithUnsafe alloc n = fromArrayUnsafe <$> FL.foldlM' step initial
18542214
18552215 step (ArrayUnsafe contents start end) x = do
18562216 let ptr = INDEX_PREV(start,a)
1857− liftIO $ pokeByteIndex ptr contents x
2217+ liftIO $ pokeAt ptr contents x
18582218 return
18592219 $ ArrayUnsafe contents ptr end
18602220
1861−-- | Like writeNWith but writes the array in reverse order.
2221+-- | Like createOfWith but writes the array in reverse order.
18622222--
18632223-- /Internal/
18642224{-# INLINE_NORMAL writeRevNWith #-}
@@ -1866,31 +2226,36 @@ writeRevNWith :: forall m a. (MonadIO m, Unbox a)
18662226 => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
18672227writeRevNWith alloc n = FL.take n (writeRevNWithUnsafe alloc n)
18682228
1869−-- | Like writeN but writes the array in reverse order.
2229+-- | Like 'createOf' but writes the array in reverse order.
18702230--
18712231-- /Pre-release/
1872−{-# INLINE_NORMAL writeRevN #-}
2232+{-# INLINE_NORMAL revCreateOf #-}
2233+revCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
2234+revCreateOf = writeRevNWith new
2235+
2236+{-# DEPRECATED writeRevN "Please use revCreateOf instead." #-}
2237+{-# INLINE writeRevN #-}
18732238writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1874−writeRevN = writeRevNWith new
2239+writeRevN = revCreateOf
18752240
18762241-- | @pinnedWriteNAligned align n@ folds a maximum of @n@ elements from the
18772242-- input stream to a 'MutArray' aligned to the given size.
18782243--
1879−-- >>> pinnedWriteNAligned align = MutArray.writeNWith (MutArray.pinnedNewAligned align)
1880−-- >>> pinnedWriteNAligned align n = MutArray.writeAppendN n (MutArray.pinnedNewAligned align n)
2244+-- >>> pinnedWriteNAligned align = MutArray.createOfWith (MutArray.pinnedNewAligned align)
2245+-- >>> pinnedWriteNAligned align n = MutArray.appendN n (MutArray.pinnedNewAligned align n)
18812246--
18822247-- /Pre-release/
18832248--
18842249{-# INLINE_NORMAL pinnedWriteNAligned #-}
18852250pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a)
18862251 => Int -> Int -> Fold m a (MutArray a)
1887−pinnedWriteNAligned align = writeNWith (pinnedNewAligned align)
2252+pinnedWriteNAligned align = createOfWith (pinnedNewAligned align)
18882253
18892254-- XXX Buffer to a list instead?
1890−--
2255+
18912256-- | Buffer a stream into a stream of arrays.
18922257--
1893−-- >>> writeChunks n = Fold.many (MutArray.writeN n) Fold.toStreamK
2258+-- >>> buildChunks n = Fold.many (MutArray.createOf n) Fold.toStreamK
18942259--
18952260-- Breaking an array into an array stream can be useful to consume a large
18962261-- array sequentially such that memory of the array is released incrementatlly.
@@ -1899,37 +2264,43 @@ pinnedWriteNAligned align = writeNWith (pinnedNewAligned align)
18992264--
19002265-- /Unimplemented/
19012266--
1902−{-# INLINE_NORMAL writeChunks #-}
2267+{-# INLINE_NORMAL buildChunks #-}
2268+buildChunks :: (MonadIO m, Unbox a) =>
2269+ Int -> Fold m a (StreamK n (MutArray a))
2270+buildChunks n = FL.many (createOf n) FL.toStreamK
2271+
2272+{-# DEPRECATED writeChunks "Please use buildChunks instead." #-}
2273+{-# INLINE writeChunks #-}
19032274writeChunks :: (MonadIO m, Unbox a) =>
19042275 Int -> Fold m a (StreamK n (MutArray a))
1905−writeChunks n = FL.many (writeN n) FL.toStreamK
2276+writeChunks = buildChunks
19062277
19072278{-# INLINE_NORMAL writeWithAs #-}
19082279writeWithAs :: forall m a. (MonadIO m, Unbox a)
19092280 => PinnedState -> Int -> Fold m a (MutArray a)
1910−-- writeWithAs ps n = FL.rmapM rightSize $ writeAppendWith (* 2) (newAs ps n)
2281+-- writeWithAs ps n = FL.rmapM rightSize $ appendWith (* 2) (newAs ps n)
19112282writeWithAs ps elemCount =
19122283 FL.rmapM extract $ FL.foldlM' step initial
19132284
19142285 where
19152286
19162287 initial = do
1917− when (elemCount < 0) $ error "writeWith: elemCount is negative"
1918− liftIO $ newAs ps elemCount
2288+ when (elemCount < 0) $ error "createWith: elemCount is negative"
2289+ newAs ps elemCount
19192290
19202291 step arr@(MutArray _ start end bound) x
19212292 | INDEX_NEXT(end,a) > bound = do
19222293 let oldSize = end - start
19232294 newSize = max (oldSize * 2) 1
1924− arr1 <- liftIO $ reallocExplicit (SIZE_OF(a)) newSize arr
2295+ arr1 <- liftIO $ reallocExplicitAs ps (SIZE_OF(a)) newSize arr
19252296 snocUnsafe arr1 x
19262297 step arr x = snocUnsafe arr x
19272298
19282299 extract = liftIO . rightSize
19292300
1930−-- XXX Compare writeWith with fromStreamD which uses an array of streams
1931−-- implementation. We can write this using writeChunks above if that is faster.
1932−-- If writeWith is faster then we should use that to implement
2301+-- XXX Compare createWith with fromStreamD which uses an array of streams
2302+-- implementation. We can write this using buildChunks above if that is faster.
2303+-- If createWith is faster then we should use that to implement
19332304-- fromStreamD.
19342305--
19352306-- XXX The realloc based implementation needs to make one extra copy if we use
@@ -1944,39 +2315,56 @@ writeWithAs ps elemCount =
19442315-- XXX check if GHC's memory allocator is efficient enough. We can try the C
19452316-- malloc to compare against.
19462317
1947−-- | @writeWith minCount@ folds the whole input to a single array. The array
2318+-- | @createWith minCount@ folds the whole input to a single array. The array
19482319-- starts at a size big enough to hold minCount elements, the size is doubled
19492320-- every time the array needs to be grown.
19502321--
19512322-- /Caution! Do not use this on infinite streams./
19522323--
1953−-- >>> f n = MutArray.writeAppendWith (* 2) (MutArray.new n)
1954−-- >>> writeWith n = Fold.rmapM MutArray.rightSize (f n)
1955−-- >>> writeWith n = Fold.rmapM MutArray.fromArrayStreamK (MutArray.writeChunks n)
2324+-- >>> f n = MutArray.appendWith (* 2) (MutArray.emptyOf n)
2325+-- >>> createWith n = Fold.rmapM MutArray.rightSize (f n)
2326+-- >>> createWith n = Fold.rmapM MutArray.fromChunksK (MutArray.buildChunks n)
19562327--
19572328-- /Pre-release/
1958−{-# INLINE_NORMAL writeWith #-}
2329+{-# INLINE_NORMAL createWith #-}
2330+createWith :: forall m a. (MonadIO m, Unbox a)
2331+ => Int -> Fold m a (MutArray a)
2332+-- createWith n = FL.rmapM rightSize $ appendWith (* 2) (emptyOf n)
2333+createWith = writeWithAs Unpinned
2334+
2335+{-# DEPRECATED writeWith "Please use createWith instead." #-}
2336+{-# INLINE writeWith #-}
19592337writeWith :: forall m a. (MonadIO m, Unbox a)
19602338 => Int -> Fold m a (MutArray a)
1961−-- writeWith n = FL.rmapM rightSize $ writeAppendWith (* 2) (new n)
1962−writeWith = writeWithAs Unpinned
2339+writeWith = createWith
19632340
19642341-- | Fold the whole input to a single array.
19652342--
1966−-- Same as 'writeWith' using an initial array size of 'arrayChunkBytes' bytes
2343+-- Same as 'createWith' using an initial array size of 'arrayChunkBytes' bytes
19672344-- rounded up to the element size.
19682345--
19692346-- /Caution! Do not use this on infinite streams./
19702347--
2348+{-# INLINE create #-}
2349+create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
2350+create = createWith (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
2351+
2352+-- XXX Deprecate in major
2353+-- {-# DEPRECATED write "Please use create instead." #-}
19712354{-# INLINE write #-}
19722355write :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1973−write = writeWith (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
2356+write = create
2357+
2358+-- | Like 'create' but creates a pinned array.
2359+{-# INLINE pinnedCreate #-}
2360+pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
2361+pinnedCreate =
2362+ writeWithAs Pinned (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
19742363
1975−-- | Like 'write' but creates a pinned array.
2364+{-# DEPRECATED pinnedWrite "Please use pinnedCreate instead." #-}
19762365{-# INLINE pinnedWrite #-}
19772366pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1978−pinnedWrite =
1979− writeWithAs Pinned (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
2367+pinnedWrite = pinnedCreate
19802368
19812369-------------------------------------------------------------------------------
19822370-- construct from streams, known size
@@ -1986,25 +2374,31 @@ pinnedWrite =
19862374fromStreamDNAs :: forall m a. (MonadIO m, Unbox a)
19872375 => PinnedState -> Int -> D.Stream m a -> m (MutArray a)
19882376fromStreamDNAs ps limit str = do
1989− (arr :: MutArray a) <- liftIO $ newAs ps limit
2377+ (arr :: MutArray a) <- newAs ps limit
19902378 end <- D.foldlM' (fwrite (arrContents arr)) (return $ arrEnd arr) $ D.take limit str
19912379 return $ arr {arrEnd = end}
19922380
19932381 where
19942382
19952383 fwrite arrContents ptr x = do
1996− liftIO $ pokeByteIndex ptr arrContents x
2384+ liftIO $ pokeAt ptr arrContents x
19972385 return $ INDEX_NEXT(ptr,a)
19982386
1999−-- | Use the 'writeN' fold instead.
2387+-- | Use the 'createOf' fold instead.
20002388--
2001−-- >>> fromStreamDN n = Stream.fold (MutArray.writeN n)
2389+-- >>> fromStreamN n = Stream.fold (MutArray.createOf n)
20022390--
2003−{-# INLINE_NORMAL fromStreamDN #-}
2391+{-# INLINE_NORMAL fromStreamN #-}
2392+fromStreamN :: forall m a. (MonadIO m, Unbox a)
2393+ => Int -> D.Stream m a -> m (MutArray a)
2394+-- fromStreamDN n = D.fold (createOf n)
2395+fromStreamN = fromStreamDNAs Unpinned
2396+
2397+{-# DEPRECATED fromStreamDN "Please use fromStreamN instead." #-}
2398+{-# INLINE fromStreamDN #-}
20042399fromStreamDN :: forall m a. (MonadIO m, Unbox a)
20052400 => Int -> D.Stream m a -> m (MutArray a)
2006−-- fromStreamDN n = D.fold (writeN n)
2007−fromStreamDN = fromStreamDNAs Unpinned
2401+fromStreamDN = fromStreamN
20082402
20092403-- | Create a 'MutArray' from the first N elements of a list. The array is
20102404-- allocated to size N, if the list terminates before N elements then the
@@ -2024,16 +2418,70 @@ pinnedFromListN n xs = fromStreamDNAs Pinned n $ D.fromList xs
20242418-- /Pre-release/
20252419{-# INLINE fromListRevN #-}
20262420fromListRevN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
2027−fromListRevN n xs = D.fold (writeRevN n) $ D.fromList xs
2421+fromListRevN n xs = D.fold (revCreateOf n) $ D.fromList xs
2422+
2423+-- | Convert a pure stream in Identity monad to a mutable array.
2424+{-# INLINABLE fromPureStreamN #-}
2425+fromPureStreamN :: (MonadIO m, Unbox a) =>
2426+ Int -> Stream Identity a -> m (MutArray a)
2427+fromPureStreamN n xs =
2428+ D.fold (createOf n) $ D.morphInner (return . runIdentity) xs
20282429
20292430-- | Convert a pure stream in Identity monad to a mutable array.
20302431{-# INLINABLE fromPureStream #-}
20312432fromPureStream :: (MonadIO m, Unbox a) => Stream Identity a -> m (MutArray a)
20322433fromPureStream xs =
2033− liftIO $ D.fold write $ D.morphInner (return . runIdentity) xs
2434+ D.fold create $ D.morphInner (return . runIdentity) xs
2435+
2436+{-# INLINABLE fromPtrN #-}
2437+fromPtrN :: MonadIO m => Int -> Ptr Word8 -> m (MutArray Word8)
2438+fromPtrN len addr = do
2439+ -- memcpy is better than stream copy when the size is known.
2440+ -- XXX We can implement a stream copy in a similar way by streaming Word64
2441+ -- first and then remaining Word8.
2442+ arr <- new len
2443+ _ <- unsafeAsPtr arr
2444+ (\ptr -> liftIO $ c_memcpy ptr addr (fromIntegral len))
2445+ return (arr {arrEnd = len})
2446+
2447+{-# INLINABLE fromByteStr# #-}
2448+fromByteStr# :: MonadIO m => Addr# -> m (MutArray Word8)
2449+fromByteStr# addr = do
2450+ -- It is better to count the size first and allocate exact space.
2451+ -- Also, memcpy is better than stream copy when the size is known.
2452+ -- C strlen compares 4 bytes at a time, so is better than the stream
2453+ -- version. https://github.com/bminor/glibc/blob/master/string/strlen.c
2454+ -- XXX We can possibly use a stream of Word64 to do the same.
2455+ -- fromByteStr# addr = fromPureStream (D.fromByteStr# addr)
2456+ len <- liftIO $ c_strlen (Ptr addr)
2457+ let lenInt = fromIntegral len
2458+ arr <- new lenInt
2459+ _ <- unsafeAsPtr arr (\ptr -> liftIO $ c_memcpy ptr (Ptr addr) len)
2460+ return (arr {arrEnd = lenInt})
20342461
20352462-------------------------------------------------------------------------------
2036−-- convert stream to a single array
2463+-- convert a stream of arrays to a single array by reallocating and copying
2464+-------------------------------------------------------------------------------
2465+
2466+-- XXX Both of these implementations of splicing seem to perform equally well.
2467+-- We need to perform benchmarks over a range of sizes though.
2468+
2469+-- | Also see 'fromChunksK'.
2470+{-# INLINE fromChunksRealloced #-}
2471+fromChunksRealloced :: forall m a. (MonadIO m, Unbox a)
2472+ => Stream m (MutArray a) -> m (MutArray a)
2473+fromChunksRealloced s = do
2474+ res <- D.uncons s
2475+ case res of
2476+ Just (a, strm) -> do
2477+ arr <- D.foldlM' spliceExp (pure a) strm
2478+ -- Reallocation is exponential so there may be 50% empty space in
2479+ -- worst case. One more reallocation to reclaim the space.
2480+ rightSize arr
2481+ Nothing -> pure nil
2482+
2483+-------------------------------------------------------------------------------
2484+-- convert a stream of arrays to a single array by buffering arrays first
20372485-------------------------------------------------------------------------------
20382486
20392487{-# INLINE arrayStreamKLength #-}
@@ -2043,34 +2491,72 @@ arrayStreamKLength as = K.foldl' (+) 0 (K.map length as)
20432491-- | Convert an array stream to an array. Note that this requires peak memory
20442492-- that is double the size of the array stream.
20452493--
2494+{-# INLINE fromChunkskAs #-}
2495+fromChunkskAs :: (Unbox a, MonadIO m) =>
2496+ PinnedState -> StreamK m (MutArray a) -> m (MutArray a)
2497+fromChunkskAs ps as = do
2498+ len <- arrayStreamKLength as
2499+ arr <- newAs ps len
2500+ -- XXX is StreamK fold faster or StreamD fold?
2501+ K.foldlM' spliceUnsafe (pure arr) as
2502+ -- fromStreamDN len $ D.unfoldMany reader $ D.fromStreamK as
2503+
2504+-- XXX Need to compare this with fromChunks and fromChunkList and keep the
2505+-- fastest or simplest one if all are equally fast.
2506+
2507+-- | Convert an array stream to an array. Note that this requires peak memory
2508+-- that is double the size of the array stream.
2509+--
2510+-- Also see 'fromChunksRealloced'.
2511+--
2512+{-# INLINE fromChunksK #-}
2513+fromChunksK :: (Unbox a, MonadIO m) =>
2514+ StreamK m (MutArray a) -> m (MutArray a)
2515+fromChunksK = fromChunkskAs Unpinned
2516+
2517+{-# DEPRECATED fromArrayStreamK "Please use fromChunksK instead." #-}
20462518{-# INLINE fromArrayStreamK #-}
20472519fromArrayStreamK :: (Unbox a, MonadIO m) =>
20482520 StreamK m (MutArray a) -> m (MutArray a)
2049−fromArrayStreamK as = do
2050− len <- arrayStreamKLength as
2051− fromStreamDN len $ D.unfoldMany reader $ D.fromStreamK as
2521+fromArrayStreamK = fromChunksK
20522522
20532523{-# INLINE fromStreamDAs #-}
20542524fromStreamDAs ::
20552525 (MonadIO m, Unbox a) => PinnedState -> D.Stream m a -> m (MutArray a)
2056−fromStreamDAs ps m = arrayStreamKFromStreamDAs ps m >>= fromArrayStreamK
2526+fromStreamDAs ps m =
2527+ arrayStreamKFromStreamDAs Unpinned m >>= fromChunkskAs ps
20572528
2529+-- | Create an 'Array' from a stream. This is useful when we want to create a
2530+-- single array from a stream of unknown size. 'createOf' is at least twice
2531+-- as efficient when the size is already known.
2532+--
2533+-- Note that if the input stream is too large memory allocation for the array
2534+-- may fail. When the stream size is not known, `chunksOf` followed by
2535+-- processing of indvidual arrays in the resulting stream should be preferred.
2536+--
2537+-- /Pre-release/
2538+{-# INLINE fromStream #-}
2539+fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
2540+fromStream = fromStreamDAs Unpinned
2541+
2542+-- fromStream (Stream m) = P.fold create m
20582543-- CAUTION: a very large number (millions) of arrays can degrade performance
20592544-- due to GC overhead because we need to buffer the arrays before we flatten
20602545-- all the arrays.
20612546--
2062−-- XXX Compare if this is faster or "fold write".
2547+-- XXX Compare if this is faster or "fold create".
20632548--
20642549-- | We could take the approach of doubling the memory allocation on each
20652550-- overflow. This would result in more or less the same amount of copying as in
20662551-- the chunking approach. However, if we have to shrink in the end then it may
20672552-- result in an extra copy of the entire data.
20682553--
2069−-- >>> fromStreamD = StreamD.fold MutArray.write
2554+-- >>> fromStreamD = StreamD.fold MutArray.create
20702555--
20712556{-# INLINE fromStreamD #-}
2557+{-# DEPRECATED fromStreamD "Please use fromStream instead." #-}
20722558fromStreamD :: (MonadIO m, Unbox a) => D.Stream m a -> m (MutArray a)
2073−fromStreamD = fromStreamDAs Unpinned
2559+fromStreamD = fromStream
20742560
20752561-- | Create a 'MutArray' from a list. The list must be of finite size.
20762562--
@@ -2095,22 +2581,44 @@ fromListRev xs = fromListRevN (Prelude.length xs) xs
20952581-- Cloning
20962582-------------------------------------------------------------------------------
20972583
2584+-- Arrays are aligned on 64-bit boundaries. The fastest way to copy an array is
2585+-- to unsafeCast it to Word64, read it, write it to Word64 array and unsafeCast
2586+-- it again. We can use SIMD read/write as well.
2587+
20982588{-# INLINE cloneAs #-}
2099−cloneAs :: MonadIO m => PinnedState -> MutArray a -> m (MutArray a)
2589+cloneAs ::
2590+ ( MonadIO m
2591+#ifdef DEVBUILD
2592+ , Unbox a
2593+#endif
2594+ )
2595+ => PinnedState -> MutArray a -> m (MutArray a)
21002596cloneAs ps src =
2101− liftIO $ do
2597+ do
21022598 let startSrc = arrStart src
21032599 srcLen = arrEnd src - startSrc
2104− newArrContents <- Unboxed.newBytesAs ps srcLen
2105− putSliceUnsafe (arrContents src) startSrc newArrContents 0 srcLen
2600+ newArrContents <-
2601+ Unboxed.cloneSliceUnsafeAs ps startSrc srcLen (arrContents src)
21062602 return $ MutArray newArrContents 0 srcLen srcLen
21072603
21082604{-# INLINE clone #-}
2109−clone :: MonadIO m => MutArray a -> m (MutArray a)
2605+clone ::
2606+ ( MonadIO m
2607+#ifdef DEVBUILD
2608+ , Unbox a
2609+#endif
2610+ )
2611+ => MutArray a -> m (MutArray a)
21102612clone = cloneAs Unpinned
21112613
21122614{-# INLINE pinnedClone #-}
2113−pinnedClone :: MonadIO m => MutArray a -> m (MutArray a)
2615+pinnedClone ::
2616+ ( MonadIO m
2617+#ifdef DEVBUILD
2618+ , Unbox a
2619+#endif
2620+ )
2621+ => MutArray a -> m (MutArray a)
21142622pinnedClone = cloneAs Pinned
21152623
21162624-------------------------------------------------------------------------------
@@ -2125,17 +2633,16 @@ spliceCopy :: forall m a. MonadIO m =>
21252633 Unbox a =>
21262634#endif
21272635 MutArray a -> MutArray a -> m (MutArray a)
2128−spliceCopy arr1 arr2 = liftIO $ do
2636+spliceCopy arr1 arr2 = do
21292637 let start1 = arrStart arr1
21302638 start2 = arrStart arr2
21312639 len1 = arrEnd arr1 - start1
21322640 len2 = arrEnd arr2 - start2
2133− let newLen = len1 + len2
2641+ let len = len1 + len2
21342642 newArrContents <-
21352643 if Unboxed.isPinned (arrContents arr1)
2136− then Unboxed.pinnedNewBytes newLen
2137− else Unboxed.newBytes newLen
2138− let len = len1 + len2
2644+ then liftIO $ Unboxed.pinnedNew len
2645+ else liftIO $ Unboxed.new len
21392646 putSliceUnsafe (arrContents arr1) start1 newArrContents 0 len1
21402647 putSliceUnsafe (arrContents arr2) start2 newArrContents len1 len2
21412648 return $ MutArray newArrContents 0 len len
@@ -2147,7 +2654,7 @@ spliceCopy arr1 arr2 = liftIO $ do
21472654spliceUnsafe :: MonadIO m =>
21482655 MutArray a -> MutArray a -> m (MutArray a)
21492656spliceUnsafe dst src =
2150− liftIO $ do
2657+ do
21512658 let startSrc = arrStart src
21522659 srcLen = arrEnd src - startSrc
21532660 endDst = arrEnd dst
@@ -2169,7 +2676,7 @@ spliceWith :: forall m a. (MonadIO m, Unbox a) =>
21692676 (Int -> Int -> Int) -> MutArray a -> MutArray a -> m (MutArray a)
21702677spliceWith sizer dst@(MutArray _ start end bound) src = do
21712678{-
2172− let f = writeAppendWith (`sizer` byteLength src) (return dst)
2679+ let f = appendWith (`sizer` byteLength src) (return dst)
21732680 in D.fold f (toStreamD src)
21742681-}
21752682 assert (end <= bound) (return ())
@@ -2185,7 +2692,7 @@ spliceWith sizer dst@(MutArray _ start end bound) src = do
21852692 $ "splice: newSize is less than the total size "
21862693 ++ "of arrays being appended. Please check the "
21872694 ++ "sizer function passed."
2188− liftIO $ realloc newSizeInBytes dst
2695+ realloc newSizeInBytes dst
21892696 else return dst
21902697 spliceUnsafe dst1 src
21912698
@@ -2222,14 +2729,24 @@ spliceExp = spliceWith (\l1 l2 -> max (l1 * 2) (l1 + l2))
22222729-- Splitting
22232730-------------------------------------------------------------------------------
22242731
2732+-- | Generate a stream of array slices using a predicate. The array element
2733+-- matching the predicate is dropped.
2734+--
2735+-- /Pre-release/
2736+{-# INLINE splitOn #-}
2737+splitOn :: (MonadIO m, Unbox a) =>
2738+ (a -> Bool) -> MutArray a -> Stream m (MutArray a)
2739+splitOn predicate arr =
2740+ fmap (\(i, len) -> getSliceUnsafe i len arr)
2741+ $ D.indexOnSuffix predicate (read arr)
2742+
22252743-- | Drops the separator byte
22262744{-# INLINE breakOn #-}
22272745breakOn :: MonadIO m
22282746 => Word8 -> MutArray Word8 -> m (MutArray Word8, Maybe (MutArray Word8))
2229−breakOn sep arr@MutArray{..} = asPtrUnsafe arr $ \p -> liftIO $ do
2230− -- XXX Instead of using asPtrUnsafe (pinning memory) we can pass unlifted
2231− -- Addr# to memchr and it should be safe (from ghc 8.4).
2747+breakOn sep arr@MutArray{..} = unsafeAsPtr arr $ \p -> liftIO $ do
22322748 -- XXX We do not need memchr here, we can use a Haskell equivalent.
2749+ -- Need efficient stream based primitives that work on Word64.
22332750 loc <- c_memchr p sep (fromIntegral $ byteLength arr)
22342751 let sepIndex = loc `minusPtr` p
22352752 return $
@@ -2250,32 +2767,41 @@ breakOn sep arr@MutArray{..} = asPtrUnsafe arr $ \p -> liftIO $ do
22502767 }
22512768 )
22522769
2770+-- | Like 'splitAt' but does not check whether the index is valid.
2771+--
2772+{-# INLINE unsafeSplitAt #-}
2773+unsafeSplitAt :: forall a. Unbox a =>
2774+ Int -> MutArray a -> (MutArray a, MutArray a)
2775+unsafeSplitAt i MutArray{..} =
2776+ let off = i * SIZE_OF(a)
2777+ p = arrStart + off
2778+ in ( MutArray
2779+ { arrContents = arrContents
2780+ , arrStart = arrStart
2781+ , arrEnd = p
2782+ , arrBound = p
2783+ }
2784+ , MutArray
2785+ { arrContents = arrContents
2786+ , arrStart = p
2787+ , arrEnd = arrEnd
2788+ , arrBound = arrBound
2789+ }
2790+ )
2791+
22532792-- | Create two slices of an array without copying the original array. The
22542793-- specified index @i@ is the first index of the second slice.
22552794--
2795+{-# INLINE splitAt #-}
22562796splitAt :: forall a. Unbox a => Int -> MutArray a -> (MutArray a, MutArray a)
2257−splitAt i arr@MutArray{..} =
2797+splitAt i arr =
22582798 let maxIndex = length arr - 1
22592799 in if i < 0
22602800 then error "sliceAt: negative array index"
22612801 else if i > maxIndex
22622802 then error $ "sliceAt: specified array index " ++ show i
22632803 ++ " is beyond the maximum index " ++ show maxIndex
2264− else let off = i * SIZE_OF(a)
2265− p = arrStart + off
2266− in ( MutArray
2267− { arrContents = arrContents
2268− , arrStart = arrStart
2269− , arrEnd = p
2270− , arrBound = p
2271− }
2272− , MutArray
2273− { arrContents = arrContents
2274− , arrStart = p
2275− , arrEnd = arrEnd
2276− , arrBound = arrBound
2277− }
2278− )
2804+ else unsafeSplitAt i arr
22792805
22802806-------------------------------------------------------------------------------
22812807-- Casting
@@ -2325,13 +2851,16 @@ cast arr =
23252851-- guarantees that unsafe calls will be performed in the calling thread. Making
23262852-- it safe to pass heap-allocated objects to unsafe functions.
23272853
2328−-- Unsafe because of direct pointer operations. The user must ensure that they
2329−-- are writing within the legal bounds of the array. Should we just name it
2330−-- asPtr, the unsafety is implicit for any pointer operations. And we are safe
2331−-- from Haskell perspective because we will be pinning the memory.
2854+-- XXX Should we just name it asPtr, the unsafety is implicit for any pointer
2855+-- operations. And we are safe from Haskell perspective because we will be
2856+-- pinning the memory.
2857+--
2858+-- XXX we cannot pass the length of the ptr here as in some cases it may not be
2859+-- available e.g. a null terminated C string. However, we can create another
2860+-- flavor of the API e.g. asPtrN.
23322861
2333−-- | Use an @MutArray a@ as @Ptr a@. This is useful when we want to pass an array
2334−-- as a pointer to some operating system call or to a "safe" FFI call.
2862+-- | Use a @MutArray a@ as @Ptr a@. This is useful when we want to pass an
2863+-- array as a pointer to some operating system call or to a "safe" FFI call.
23352864--
23362865-- If the array is not pinned it is copied to pinned memory before passing it
23372866-- to the monadic action.
@@ -2341,57 +2870,382 @@ cast arr =
23412870-- opportunistically before this operation occurs, to avoid the cost of a copy
23422871-- if possible.
23432872--
2344−-- /Unsafe/
2873+-- /Unsafe/ because of direct pointer operations. The user must ensure that
2874+-- they are writing within the legal bounds of the array.
23452875--
23462876-- /Pre-release/
23472877--
2878+{-# INLINE unsafePinnedAsPtr #-}
2879+unsafePinnedAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
2880+unsafePinnedAsPtr arr f =
2881+ Unboxed.unsafePinnedAsPtr
2882+ (arrContents arr) (\ptr -> f (ptr `plusPtr` arrStart arr))
2883+
2884+{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
23482885{-# INLINE asPtrUnsafe #-}
23492886asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
2350−asPtrUnsafe arr f = do
2351− contents <- liftIO $ Unboxed.pin $ arrContents arr
2352− let !ptr = Ptr (byteArrayContents#
2353− (unsafeCoerce# (getMutableByteArray# contents)))
2887+asPtrUnsafe = unsafePinnedAsPtr
23542888
2355− r <- f (ptr `plusPtr` arrStart arr)
2356− liftIO $ touch contents
2357− return r
2889+{-# INLINE unsafeAsPtr #-}
2890+unsafeAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
2891+unsafeAsPtr arr f =
2892+ Unboxed.unsafeAsPtr
2893+ (arrContents arr) (\ptr -> f (ptr `plusPtr` arrStart arr))
23582894
23592895-------------------------------------------------------------------------------
23602896-- Equality
23612897-------------------------------------------------------------------------------
23622898
2363−-- | Compare the length of the arrays. If the length is equal, compare the
2364−-- lexicographical ordering of two underlying byte arrays otherwise return the
2365−-- result of length comparison.
2899+-- | Byte compare two arrays. Compare the length of the arrays. If the length
2900+-- is equal, compare the lexicographical ordering of two underlying byte arrays
2901+-- otherwise return the result of length comparison.
2902+--
2903+-- /Unsafe/: Note that the 'Unbox' instance of sum types with constructors of
2904+-- different sizes may leave some memory uninitialized which can make byte
2905+-- comparison unreliable.
23662906--
23672907-- /Pre-release/
2908+{-# INLINE byteCmp #-}
2909+byteCmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
2910+byteCmp arr1 arr2 = do
2911+ let marr1 = getMutableByteArray# (arrContents arr1)
2912+ marr2 = getMutableByteArray# (arrContents arr2)
2913+ !(I# st1#) = arrStart arr1
2914+ !(I# st2#) = arrStart arr2
2915+ !(I# len#) = byteLength arr1
2916+ case compare (byteLength arr1) (byteLength arr2) of
2917+ EQ -> do
2918+ r <- liftIO $ IO $ \s# ->
2919+ let res =
2920+ I#
2921+ (compareByteArrays#
2922+ (unsafeCoerce# marr1)
2923+ st1#
2924+ (unsafeCoerce# marr2)
2925+ st2#
2926+ len#)
2927+ in (# s#, res #)
2928+ return $ compare r 0
2929+ x -> return x
2930+
23682931{-# INLINE cmp #-}
2932+{-# DEPRECATED cmp "Please use byteCmp instead." #-}
23692933cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
2370−cmp arr1 arr2 =
2371− liftIO
2372− $ do
2373− let marr1 = getMutableByteArray# (arrContents arr1)
2374− marr2 = getMutableByteArray# (arrContents arr2)
2375− !(I# st1#) = arrStart arr1
2376− !(I# st2#) = arrStart arr2
2377− !(I# len#) = byteLength arr1
2378− case compare (byteLength arr1) (byteLength arr2) of
2379− EQ -> do
2380− r <- liftIO $ IO $ \s# ->
2381− let res =
2382− I#
2383− (compareByteArrays#
2384− (unsafeCoerce# marr1)
2385− st1#
2386− (unsafeCoerce# marr2)
2387− st2#
2388− len#)
2389− in (# s#, res #)
2390− return $ compare r 0
2391− x -> return x
2934+cmp = byteCmp
2935+
2936+-- | Byte equality of two arrays.
2937+--
2938+-- >>> byteEq arr1 arr2 = (==) EQ $ MArray.byteCmp arr1 arr2
2939+--
2940+-- /Unsafe/: See 'byteCmp'.
2941+{-# INLINE byteEq #-}
2942+byteEq :: MonadIO m => MutArray a -> MutArray a -> m Bool
2943+byteEq arr1 arr2 = fmap (EQ ==) $ byteCmp arr1 arr2
2944+
2945+-------------------------------------------------------------------------------
2946+-- Compact
2947+-------------------------------------------------------------------------------
2948+
2949+-- Note: LE versions avoid an extra copy compared to GE. LE parser trades
2950+-- backtracking one array in lieu of avoiding a copy. However, LE and GE both
2951+-- can leave some memory unused. They can split the last array to fit it
2952+-- exactly in the space.
2953+
2954+{-# INLINE_NORMAL pCompactLeAs #-}
2955+pCompactLeAs ::
2956+ forall m a. (MonadIO m, Unbox a)
2957+ => PinnedState -> Int -> Parser (MutArray a) m (MutArray a)
2958+pCompactLeAs ps maxElems = Parser step initial extract
2959+
2960+ where
2961+
2962+ maxBytes = maxElems * SIZE_OF(a)
2963+
2964+ functionName = "Streamly.Internal.Data.MutArray.pCompactLE"
2965+
2966+ initial =
2967+ return
2968+ $ if maxElems <= 0
2969+ then error
2970+ $ functionName
2971+ ++ ": the size of arrays ["
2972+ ++ show maxElems ++ "] must be a natural number"
2973+ else Parser.IPartial Nothing
2974+
2975+ step Nothing arr =
2976+ return
2977+ $ let len = byteLength arr
2978+ in if len >= maxBytes
2979+ then Parser.Done 0 arr
2980+ else Parser.Partial 0 (Just arr)
2981+ -- XXX Split the last array to use the space more compactly.
2982+ step (Just buf) arr =
2983+ let len = byteLength buf + byteLength arr
2984+ in if len > maxBytes
2985+ then return $ Parser.Done 1 buf
2986+ else do
2987+ buf1 <-
2988+ if byteCapacity buf < maxBytes
2989+ then liftIO $ reallocExplicitAs
2990+ ps (SIZE_OF(a)) maxBytes buf
2991+ else return buf
2992+ buf2 <- spliceUnsafe buf1 arr
2993+ return $ Parser.Partial 0 (Just buf2)
2994+
2995+ extract Nothing = return $ Parser.Done 0 nil
2996+ extract (Just buf) = return $ Parser.Done 0 buf
2997+
2998+-- | Parser @pCompactLE maxElems@ coalesces adjacent arrays in the input stream
2999+-- only if the combined size would be less than or equal to @maxElems@
3000+-- elements. Note that it won't split an array if the original array is already
3001+-- larger than maxElems.
3002+--
3003+-- @maxElems@ must be greater than 0.
3004+--
3005+-- Generates unpinned arrays irrespective of the pinning status of input
3006+-- arrays.
3007+--
3008+-- /Internal/
3009+{-# INLINE pCompactLE #-}
3010+pCompactLE ::
3011+ forall m a. (MonadIO m, Unbox a)
3012+ => Int -> Parser (MutArray a) m (MutArray a)
3013+pCompactLE = pCompactLeAs Unpinned
3014+
3015+-- | Pinned version of 'pCompactLE'.
3016+{-# INLINE pPinnedCompactLE #-}
3017+pPinnedCompactLE ::
3018+ forall m a. (MonadIO m, Unbox a)
3019+ => Int -> Parser (MutArray a) m (MutArray a)
3020+pPinnedCompactLE = pCompactLeAs Pinned
3021+
3022+data SpliceState s arr
3023+ = SpliceInitial s
3024+ | SpliceBuffering s arr
3025+ | SpliceYielding arr (SpliceState s arr)
3026+ | SpliceFinish
3027+
3028+-- This mutates the first array (if it has space) to append values from the
3029+-- second one. This would work for immutable arrays as well because an
3030+-- immutable array never has additional space so a new array is allocated
3031+-- instead of mutating it.
3032+
3033+{-# INLINE_NORMAL compactLeAs #-}
3034+compactLeAs :: forall m a. (MonadIO m, Unbox a)
3035+ => PinnedState -> Int -> D.Stream m (MutArray a) -> D.Stream m (MutArray a)
3036+compactLeAs ps maxElems (D.Stream step state) =
3037+ D.Stream step' (SpliceInitial state)
3038+
3039+ where
3040+
3041+ maxBytes = maxElems * SIZE_OF(a)
3042+
3043+ functionName = "Streamly.Internal.Data.MutArray.rCompactLE"
3044+
3045+ {-# INLINE_LATE step' #-}
3046+ step' gst (SpliceInitial st) = do
3047+ when (maxElems <= 0) $
3048+ -- XXX we can pass the module string from the higher level API
3049+ error $ functionName ++ ": the size of arrays [" ++ show maxElems
3050+ ++ "] must be a natural number"
3051+ r <- step gst st
3052+ case r of
3053+ D.Yield arr s -> return $
3054+ let len = byteLength arr
3055+ in if len >= maxBytes
3056+ then D.Skip (SpliceYielding arr (SpliceInitial s))
3057+ else D.Skip (SpliceBuffering s arr)
3058+ D.Skip s -> return $ D.Skip (SpliceInitial s)
3059+ D.Stop -> return D.Stop
3060+
3061+ -- XXX Split the last array to use the space more compactly.
3062+ step' gst (SpliceBuffering st buf) = do
3063+ r <- step gst st
3064+ case r of
3065+ D.Yield arr s -> do
3066+ let len = byteLength buf + byteLength arr
3067+ if len > maxBytes
3068+ then return $
3069+ D.Skip (SpliceYielding buf (SpliceBuffering s arr))
3070+ else do
3071+ buf1 <- if byteCapacity buf < maxBytes
3072+ then liftIO $ reallocExplicitAs
3073+ ps (SIZE_OF(a)) maxBytes buf
3074+ else return buf
3075+ buf2 <- spliceUnsafe buf1 arr
3076+ return $ D.Skip (SpliceBuffering s buf2)
3077+ D.Skip s -> return $ D.Skip (SpliceBuffering s buf)
3078+ D.Stop -> return $ D.Skip (SpliceYielding buf SpliceFinish)
3079+
3080+ step' _ SpliceFinish = return D.Stop
3081+
3082+ step' _ (SpliceYielding arr next) = return $ D.Yield arr next
3083+
3084+
3085+{-# INLINE_NORMAL fCompactGeAs #-}
3086+fCompactGeAs ::
3087+ forall m a. (MonadIO m, Unbox a)
3088+ => PinnedState -> Int -> FL.Fold m (MutArray a) (MutArray a)
3089+fCompactGeAs ps minElems = Fold step initial extract extract
3090+
3091+ where
3092+
3093+ minBytes = minElems * SIZE_OF(a)
3094+
3095+ functionName = "Streamly.Internal.Data.MutArray.fCompactGE"
3096+
3097+ initial =
3098+ return
3099+ $ if minElems < 0
3100+ then error
3101+ $ functionName
3102+ ++ ": the size of arrays ["
3103+ ++ show minElems ++ "] must be a natural number"
3104+ else FL.Partial Nothing
3105+
3106+ step Nothing arr =
3107+ return
3108+ $ let len = byteLength arr
3109+ in if len >= minBytes
3110+ then FL.Done arr
3111+ else FL.Partial (Just arr)
3112+ -- XXX Buffer arrays as a list to avoid copy and reallocations
3113+ step (Just buf) arr = do
3114+ let len = byteLength buf + byteLength arr
3115+ buf1 <-
3116+ if byteCapacity buf < len
3117+ then liftIO $ reallocExplicitAs
3118+ ps (SIZE_OF(a)) (max minBytes len) buf
3119+ else return buf
3120+ buf2 <- spliceUnsafe buf1 arr
3121+ if len >= minBytes
3122+ then return $ FL.Done buf2
3123+ else return $ FL.Partial (Just buf2)
3124+
3125+ extract Nothing = return nil
3126+ extract (Just buf) = return buf
3127+
3128+-- | Fold @fCompactGE minElems@ coalesces adjacent arrays in the input stream
3129+-- until the size becomes greater than or equal to @minElems@.
3130+--
3131+-- Generates unpinned arrays irrespective of the pinning status of input
3132+-- arrays.
3133+{-# INLINE fCompactGE #-}
3134+fCompactGE ::
3135+ forall m a. (MonadIO m, Unbox a)
3136+ => Int -> FL.Fold m (MutArray a) (MutArray a)
3137+fCompactGE = fCompactGeAs Unpinned
3138+
3139+-- | Pinned version of 'fCompactGE'.
3140+{-# INLINE fPinnedCompactGE #-}
3141+fPinnedCompactGE ::
3142+ forall m a. (MonadIO m, Unbox a)
3143+ => Int -> FL.Fold m (MutArray a) (MutArray a)
3144+fPinnedCompactGE = fCompactGeAs Pinned
3145+
3146+{-# INLINE_NORMAL lCompactGeAs #-}
3147+lCompactGeAs :: forall m a. (MonadIO m, Unbox a)
3148+ => PinnedState -> Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
3149+-- The fold version turns out to be a little bit slower.
3150+-- lCompactGeAs ps n = FL.many (fCompactGeAs ps n)
3151+lCompactGeAs ps minElems (Fold step1 initial1 _ final1) =
3152+ Fold step initial extract final
3153+
3154+ where
3155+
3156+ minBytes = minElems * SIZE_OF(a)
3157+
3158+ functionName = "Streamly.Internal.Data.MutArray.lCompactGE"
3159+
3160+ initial = do
3161+ when (minElems <= 0) $
3162+ -- XXX we can pass the module string from the higher level API
3163+ error $ functionName ++ ": the size of arrays ["
3164+ ++ show minElems ++ "] must be a natural number"
3165+
3166+ r <- initial1
3167+ return $ first (Tuple' Nothing) r
3168+
3169+ {-# INLINE runInner #-}
3170+ runInner len acc buf =
3171+ if len >= minBytes
3172+ then do
3173+ r <- step1 acc buf
3174+ case r of
3175+ FL.Done _ -> return $ FL.Done ()
3176+ FL.Partial s -> do
3177+ _ <- final1 s
3178+ res <- initial1
3179+ return $ first (Tuple' Nothing) res
3180+ else return $ FL.Partial $ Tuple' (Just buf) acc
3181+
3182+ step (Tuple' Nothing r1) arr =
3183+ runInner (byteLength arr) r1 arr
3184+
3185+ -- XXX Buffer arrays as a list to avoid copy and reallocations
3186+ step (Tuple' (Just buf) r1) arr = do
3187+ let len = byteLength buf + byteLength arr
3188+ buf1 <- if byteCapacity buf < len
3189+ then liftIO $ reallocExplicitAs
3190+ ps (SIZE_OF(a)) (max minBytes len) buf
3191+ else return buf
3192+ buf2 <- spliceUnsafe buf1 arr
3193+ runInner len r1 buf2
3194+
3195+ -- XXX Several folds do extract >=> final, therefore, we need to make final
3196+ -- return "m b" rather than using extract post it if we want extract to be
3197+ -- partial.
3198+ --
3199+ -- extract forces the pending buffer to be sent to the fold which is not
3200+ -- what we want.
3201+ extract _ = error "lCompactGE: not designed for scanning"
3202+
3203+ final (Tuple' Nothing r1) = final1 r1
3204+ final (Tuple' (Just buf) r1) = do
3205+ r <- step1 r1 buf
3206+ case r of
3207+ FL.Partial rr -> final1 rr
3208+ FL.Done _ -> return ()
3209+
3210+-- | Like 'compactGE' but for transforming folds instead of stream.
3211+--
3212+-- >>> lCompactGE n = Fold.many (MutArray.fCompactGE n)
3213+--
3214+-- Generates unpinned arrays irrespective of the pinning status of input
3215+-- arrays.
3216+{-# INLINE lCompactGE #-}
3217+lCompactGE :: forall m a. (MonadIO m, Unbox a)
3218+ => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
3219+lCompactGE = lCompactGeAs Unpinned
3220+
3221+-- | Pinned version of 'lCompactGE'.
3222+{-# INLINE lPinnedCompactGE #-}
3223+lPinnedCompactGE :: forall m a. (MonadIO m, Unbox a)
3224+ => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
3225+lPinnedCompactGE = lCompactGeAs Pinned
3226+
3227+-- | @compactGE n stream@ coalesces adjacent arrays in the @stream@ until
3228+-- the size becomes greater than or equal to @n@.
3229+--
3230+-- >>> compactGE n = Stream.foldMany (MutArray.fCompactGE n)
3231+--
3232+{-# INLINE compactGE #-}
3233+compactGE ::
3234+ (MonadIO m, Unbox a)
3235+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
3236+compactGE n = D.foldMany (fCompactGE n)
3237+
3238+-- | 'compactEQ n' coalesces adajacent arrays in the input stream to
3239+-- arrays of exact size @n@.
3240+--
3241+-- /Unimplemented/
3242+{-# INLINE compactEQ #-}
3243+compactEQ :: -- (MonadIO m, Unbox a) =>
3244+ Int -> Stream m (MutArray a) -> Stream m (MutArray a)
3245+compactEQ _n = undefined -- D.parseManyD (pCompactEQ n)
23923246
23933247-------------------------------------------------------------------------------
2394−-- NFData
3248+-- In-place mutation algorithms
23953249-------------------------------------------------------------------------------
23963250
23973251-- | Strip elements which match with predicate from both ends.
@@ -2420,7 +3274,7 @@ strip eq arr@MutArray{..} = liftIO $ do
24203274 getStart cur = do
24213275 if cur < arrEnd
24223276 then do
2423− r <- peekByteIndex cur arrContents
3277+ r <- peekAt cur arrContents
24243278 if eq r
24253279 then getStart (INDEX_NEXT(cur,a))
24263280 else return cur
@@ -2430,7 +3284,7 @@ strip eq arr@MutArray{..} = liftIO $ do
24303284 if cur > low
24313285 then do
24323286 let prev = INDEX_PREV(cur,a)
2433− r <- peekByteIndex prev arrContents
3287+ r <- peekAt prev arrContents
24343288 if eq r
24353289 then getLast prev low
24363290 else return cur
core/src/Streamly/Internal/Data/MutByteArray.hsnew file+251
@@ -0,0 +1,251 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+-- This is required as all the instances in this module are orphan instances.
4+{-# OPTIONS_GHC -fno-warn-orphans #-}
5+
6+-- |
7+-- Module : Streamly.Internal.Data.MutByteArray
8+-- Copyright : (c) 2023 Composewell Technologies
9+-- License : BSD3-3-Clause
10+-- Maintainer : streamly@composewell.com
11+-- Portability : GHC
12+--
13+
14+module Streamly.Internal.Data.MutByteArray
15+ (
16+ -- * MutByteArray
17+ module Streamly.Internal.Data.MutByteArray.Type
18+ -- * Unbox
19+ , module Streamly.Internal.Data.Unbox
20+ , module Streamly.Internal.Data.Unbox.TH
21+ -- * Serialize
22+ , module Streamly.Internal.Data.Serialize.Type
23+ -- * Serialize TH
24+ , module Streamly.Internal.Data.Serialize.TH
25+ ) where
26+
27+--------------------------------------------------------------------------------
28+-- Imports
29+--------------------------------------------------------------------------------
30+
31+import Data.Proxy (Proxy(..))
32+import Streamly.Internal.Data.Array (Array(..))
33+import GHC.Exts (Int(..), sizeofByteArray#, unsafeCoerce#)
34+import GHC.Word (Word8)
35+
36+#if __GLASGOW_HASKELL__ >= 900
37+import GHC.Num.Integer (Integer(..))
38+#else
39+import GHC.Integer.GMP.Internals (Integer(..), BigNat(..))
40+#endif
41+
42+import Streamly.Internal.Data.MutByteArray.Type
43+import Streamly.Internal.Data.Serialize.TH
44+import Streamly.Internal.Data.Serialize.Type
45+import Streamly.Internal.Data.Unbox
46+import Streamly.Internal.Data.Unbox.TH
47+
48+--------------------------------------------------------------------------------
49+-- Common instances
50+--------------------------------------------------------------------------------
51+
52+-- Note
53+-- ====
54+--
55+-- Even a non-functional change such as changing the order of constructors will
56+-- change the instance derivation.
57+--
58+-- This will not pose a problem if both, encode, and decode are done by the same
59+-- version of the application. There *might* be a problem if version that
60+-- encodes differs from the version that decodes.
61+--
62+-- We need to add some compatibility tests using different versions of
63+-- dependencies.
64+--
65+-- Although such chages for the most basic types won't happen we need to detect
66+-- if it ever happens.
67+--
68+-- Should we worry about these kind of changes and this kind of compatibility?
69+-- This is a problem for all types of derivations that depend on the order of
70+-- constructors, for example, Enum.
71+
72+-- Note on Windows build
73+-- =====================
74+--
75+-- On Windows, having template haskell splices here fail the build with the
76+-- following error:
77+--
78+-- @
79+-- addLibrarySearchPath: C:\... (Win32 error 3): The system cannot find the path specified.
80+-- @
81+--
82+-- The error might be irrelavant but having these splices triggers it. We should
83+-- either fix the problem or avoid the use to template haskell splices in this
84+-- file.
85+--
86+-- Similar issue: https://github.com/haskell/cabal/issues/4741
87+
88+-- $(Serialize.deriveSerialize ''Maybe)
89+instance Serialize a => Serialize (Maybe a) where
90+
91+ {-# INLINE addSizeTo #-}
92+ addSizeTo acc x =
93+ case x of
94+ Nothing -> (acc + 1)
95+ Just field0 -> (addSizeTo (acc + 1)) field0
96+
97+ {-# INLINE deserializeAt #-}
98+ deserializeAt initialOffset arr endOffset = do
99+ (i0, tag) <- ((deserializeAt initialOffset) arr) endOffset
100+ case tag :: Word8 of
101+ 0 -> pure (i0, Nothing)
102+ 1 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
103+ pure (i1, Just a0)
104+ _ -> error "Found invalid tag while peeking (Maybe a)"
105+
106+ {-# INLINE serializeAt #-}
107+ serializeAt initialOffset arr val =
108+ case val of
109+ Nothing -> do
110+ i0 <- ((serializeAt initialOffset) arr) (0 :: Word8)
111+ pure i0
112+ Just field0 -> do
113+ i0 <- ((serializeAt initialOffset) arr) (1 :: Word8)
114+ i1 <- ((serializeAt i0) arr) field0
115+ pure i1
116+
117+-- $(Serialize.deriveSerialize ''Either)
118+instance (Serialize a, Serialize b) => Serialize (Either a b) where
119+
120+ {-# INLINE addSizeTo #-}
121+ addSizeTo acc x =
122+ case x of
123+ Left field0 -> (addSizeTo (acc + 1)) field0
124+ Right field0 -> (addSizeTo (acc + 1)) field0
125+
126+ {-# INLINE deserializeAt #-}
127+ deserializeAt initialOffset arr endOffset = do
128+ (i0, tag) <- ((deserializeAt initialOffset) arr) endOffset
129+ case tag :: Word8 of
130+ 0 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
131+ pure (i1, Left a0)
132+ 1 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
133+ pure (i1, Right a0)
134+ _ -> error "Found invalid tag while peeking (Either a b)"
135+
136+ {-# INLINE serializeAt #-}
137+ serializeAt initialOffset arr val =
138+ case val of
139+ Left field0 -> do
140+ i0 <- ((serializeAt initialOffset) arr) (0 :: Word8)
141+ i1 <- ((serializeAt i0) arr) field0
142+ pure i1
143+ Right field0 -> do
144+ i0 <- ((serializeAt initialOffset) arr) (1 :: Word8)
145+ i1 <- ((serializeAt i0) arr) field0
146+ pure i1
147+
148+instance Serialize (Proxy a) where
149+
150+ {-# INLINE addSizeTo #-}
151+ addSizeTo acc _ = (acc + 1)
152+
153+ {-# INLINE deserializeAt #-}
154+ deserializeAt initialOffset _ _ = pure ((initialOffset + 1), Proxy)
155+
156+ {-# INLINE serializeAt #-}
157+ serializeAt initialOffset _ _ = pure (initialOffset + 1)
158+
159+--------------------------------------------------------------------------------
160+-- Integer
161+--------------------------------------------------------------------------------
162+
163+data LiftedInteger
164+ = LIS Int
165+ | LIP (Array Word)
166+ | LIN (Array Word)
167+
168+-- $(Serialize.deriveSerialize ''LiftedInteger)
169+instance Serialize LiftedInteger where
170+
171+ {-# INLINE addSizeTo #-}
172+ addSizeTo acc x =
173+ case x of
174+ LIS field0 -> (addSizeTo (acc + 1)) field0
175+ LIP field0 -> (addSizeTo (acc + 1)) field0
176+ LIN field0 -> (addSizeTo (acc + 1)) field0
177+
178+ {-# INLINE deserializeAt #-}
179+ deserializeAt initialOffset arr endOffset = do
180+ (i0, tag) <- ((deserializeAt initialOffset) arr) endOffset
181+ case tag :: Word8 of
182+ 0 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
183+ pure (i1, LIS a0)
184+ 1 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
185+ pure (i1, LIP a0)
186+ 2 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
187+ pure (i1, LIN a0)
188+ _ -> error "Found invalid tag while peeking (LiftedInteger)"
189+
190+ {-# INLINE serializeAt #-}
191+ serializeAt initialOffset arr val =
192+ case val of
193+ LIS field0 -> do
194+ i0 <- ((serializeAt initialOffset) arr) (0 :: Word8)
195+ i1 <- ((serializeAt i0) arr) field0
196+ pure i1
197+ LIP field0 -> do
198+ i0 <- ((serializeAt initialOffset) arr) (1 :: Word8)
199+ i1 <- ((serializeAt i0) arr) field0
200+ pure i1
201+ LIN field0 -> do
202+ i0 <- ((serializeAt initialOffset) arr) (2 :: Word8)
203+ i1 <- ((serializeAt i0) arr) field0
204+ pure i1
205+
206+#if __GLASGOW_HASKELL__ >= 900
207+
208+{-# INLINE liftInteger #-}
209+liftInteger :: Integer -> LiftedInteger
210+liftInteger (IS x) = LIS (I# x)
211+liftInteger (IP x) =
212+ LIP (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
213+liftInteger (IN x) =
214+ LIN (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
215+
216+{-# INLINE unliftInteger #-}
217+unliftInteger :: LiftedInteger -> Integer
218+unliftInteger (LIS (I# x)) = IS x
219+unliftInteger (LIP (Array (MutByteArray x) _ _)) = IP (unsafeCoerce# x)
220+unliftInteger (LIN (Array (MutByteArray x) _ _)) = IN (unsafeCoerce# x)
221+
222+#else
223+
224+{-# INLINE liftInteger #-}
225+liftInteger :: Integer -> LiftedInteger
226+liftInteger (S# x) = LIS (I# x)
227+liftInteger (Jp# (BN# x)) =
228+ LIP (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
229+liftInteger (Jn# (BN# x)) =
230+ LIN (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
231+
232+{-# INLINE unliftInteger #-}
233+unliftInteger :: LiftedInteger -> Integer
234+unliftInteger (LIS (I# x)) = S# x
235+unliftInteger (LIP (Array (MutByteArray x) _ _)) =
236+ Jp# (BN# (unsafeCoerce# x))
237+unliftInteger (LIN (Array (MutByteArray x) _ _)) =
238+ Jn# (BN# (unsafeCoerce# x))
239+
240+#endif
241+
242+instance Serialize Integer where
243+ {-# INLINE addSizeTo #-}
244+ addSizeTo i a = addSizeTo i (liftInteger a)
245+
246+ {-# INLINE deserializeAt #-}
247+ deserializeAt off arr end =
248+ fmap unliftInteger <$> deserializeAt off arr end
249+
250+ {-# INLINE serializeAt #-}
251+ serializeAt off arr val = serializeAt off arr (liftInteger val)
core/src/Streamly/Internal/Data/MutByteArray/Type.hsnew file+321
@@ -0,0 +1,321 @@
1+{-# LANGUAGE UnboxedTuples #-}
2+
3+-- |
4+-- Module : Streamly.Internal.Data.MutByteArray.Type
5+-- Copyright : (c) 2023 Composewell Technologies
6+-- License : BSD3-3-Clause
7+-- Maintainer : streamly@composewell.com
8+-- Stability : experimental
9+-- Portability : GHC
10+--
11+module Streamly.Internal.Data.MutByteArray.Type
12+ (
13+ -- ** MutByteArray
14+ MutByteArray(..)
15+ , MutableByteArray
16+ , getMutableByteArray# -- XXX getMutByteArray#
17+
18+ -- ** Pinning
19+ , PinnedState(..)
20+ , isPinned
21+ , pin
22+ , unpin
23+
24+ -- ** Allocation
25+ , empty
26+ , newBytesAs -- XXX should be removed
27+ , new
28+ , pinnedNew
29+ , pinnedNewAlignedBytes -- XXX should be removed
30+
31+ -- ** Access
32+ , sizeOfMutableByteArray -- XXX length
33+ , putSliceUnsafe
34+ , cloneSliceUnsafeAs
35+ , cloneSliceUnsafe
36+ , pinnedCloneSliceUnsafe
37+ , unsafePinnedAsPtr
38+ , unsafeAsPtr
39+
40+ -- ** Deprecated
41+ , asPtrUnsafe
42+ , nil
43+ ) where
44+
45+import Control.Monad.IO.Class (MonadIO(..))
46+#ifdef DEBUG
47+import Control.Monad (when)
48+import Debug.Trace (trace)
49+#endif
50+import GHC.Base (IO(..))
51+import System.IO.Unsafe (unsafePerformIO)
52+
53+import GHC.Exts
54+
55+--------------------------------------------------------------------------------
56+-- The ArrayContents type
57+--------------------------------------------------------------------------------
58+
59+data PinnedState
60+ = Pinned
61+ | Unpinned deriving (Show, Eq)
62+
63+-- XXX can use UnliftedNewtypes
64+
65+-- | A lifted mutable byte array type wrapping @MutableByteArray# RealWorld@.
66+-- This is a low level array used to back high level unboxed arrays and
67+-- serialized data.
68+data MutByteArray = MutByteArray (MutableByteArray# RealWorld)
69+
70+{-# DEPRECATED MutableByteArray "Please use MutByteArray instead" #-}
71+type MutableByteArray = MutByteArray
72+
73+{-# INLINE getMutableByteArray# #-}
74+getMutableByteArray# :: MutByteArray -> MutableByteArray# RealWorld
75+getMutableByteArray# (MutByteArray mbarr) = mbarr
76+
77+-- | Return the size of the array in bytes.
78+{-# INLINE sizeOfMutableByteArray #-}
79+sizeOfMutableByteArray :: MutByteArray -> IO Int
80+sizeOfMutableByteArray (MutByteArray arr) =
81+ IO $ \s ->
82+ case getSizeofMutableByteArray# arr s of
83+ (# s1, i #) -> (# s1, I# i #)
84+
85+{-# INLINE touch #-}
86+touch :: MutByteArray -> IO ()
87+touch (MutByteArray contents) =
88+ IO $ \s -> case touch# contents s of s' -> (# s', () #)
89+
90+-- XXX Some functions in this module are "IO" and others are "m", we need to
91+-- make it consistent.
92+
93+-- XXX We can provide another API for "unsafe" FFI calls passing an unlifted
94+-- pointer to the FFI call. For unsafe calls we do not need to pin the array.
95+-- We can pass an unlifted pointer to the FFI routine to avoid GC kicking in
96+-- before the pointer is wrapped.
97+--
98+-- From the GHC manual:
99+--
100+-- GHC, since version 8.4, guarantees that garbage collection will never occur
101+-- during an unsafe call, even in the bytecode interpreter, and further
102+-- guarantees that unsafe calls will be performed in the calling thread. Making
103+-- it safe to pass heap-allocated objects to unsafe functions.
104+
105+-- | Use a @MutByteArray@ as @Ptr a@. This is useful when we want to pass
106+-- an array as a pointer to some operating system call or to a "safe" FFI call.
107+--
108+-- If the array is not pinned it is copied to pinned memory before passing it
109+-- to the monadic action.
110+--
111+-- /Performance Notes:/ Forces a copy if the array is not pinned. It is advised
112+-- that the programmer keeps this in mind and creates a pinned array
113+-- opportunistically before this operation occurs, to avoid the cost of a copy
114+-- if possible.
115+--
116+-- /Unsafe/ because of direct pointer operations. The user must ensure that
117+-- they are writing within the legal bounds of the array.
118+--
119+-- /Pre-release/
120+--
121+{-# INLINE unsafePinnedAsPtr #-}
122+unsafePinnedAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
123+unsafePinnedAsPtr arr f = do
124+ contents <- liftIO $ pin arr
125+ let !ptr = Ptr (byteArrayContents#
126+ (unsafeCoerce# (getMutableByteArray# contents)))
127+ r <- f ptr
128+ liftIO $ touch contents
129+ return r
130+
131+{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
132+{-# INLINE asPtrUnsafe #-}
133+asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
134+asPtrUnsafe = unsafePinnedAsPtr
135+
136+-- | For use with unsafe FFI functions. Does not force pin the array memory.
137+{-# INLINE unsafeAsPtr #-}
138+unsafeAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
139+unsafeAsPtr arr f = do
140+ let !ptr = Ptr (byteArrayContents#
141+ (unsafeCoerce# (getMutableByteArray# arr)))
142+ r <- f ptr
143+ liftIO $ touch arr
144+ return r
145+
146+--------------------------------------------------------------------------------
147+-- Creation
148+--------------------------------------------------------------------------------
149+
150+{-# NOINLINE empty #-}
151+empty :: MutByteArray
152+empty = unsafePerformIO $ new 0
153+
154+{-# DEPRECATED nil "Please use empty instead" #-}
155+nil :: MutByteArray
156+nil = empty
157+
158+-- XXX add "newRounded" to round up the large size to the next page boundary
159+-- and return the allocated size.
160+{-# INLINE new #-}
161+new :: Int -> IO MutByteArray
162+new nbytes | nbytes < 0 =
163+ errorWithoutStackTrace "newByteArray: size must be >= 0"
164+new (I# nbytes) = IO $ \s ->
165+ case newByteArray# nbytes s of
166+ (# s', mbarr# #) ->
167+ let c = MutByteArray mbarr#
168+ in (# s', c #)
169+
170+{-# INLINE pinnedNew #-}
171+pinnedNew :: Int -> IO MutByteArray
172+pinnedNew nbytes | nbytes < 0 =
173+ errorWithoutStackTrace "pinnedNew: size must be >= 0"
174+pinnedNew (I# nbytes) = IO $ \s ->
175+ case newPinnedByteArray# nbytes s of
176+ (# s', mbarr# #) ->
177+ let c = MutByteArray mbarr#
178+ in (# s', c #)
179+
180+{-# INLINE pinnedNewAlignedBytes #-}
181+pinnedNewAlignedBytes :: Int -> Int -> IO MutByteArray
182+pinnedNewAlignedBytes nbytes _align | nbytes < 0 =
183+ errorWithoutStackTrace "pinnedNewAlignedBytes: size must be >= 0"
184+pinnedNewAlignedBytes (I# nbytes) (I# align) = IO $ \s ->
185+ case newAlignedPinnedByteArray# nbytes align s of
186+ (# s', mbarr# #) ->
187+ let c = MutByteArray mbarr#
188+ in (# s', c #)
189+
190+{-# INLINE newBytesAs #-}
191+newBytesAs :: PinnedState -> Int -> IO MutByteArray
192+newBytesAs Unpinned = new
193+newBytesAs Pinned = pinnedNew
194+
195+-------------------------------------------------------------------------------
196+-- Copying
197+-------------------------------------------------------------------------------
198+
199+-- Note: Array copy is more efficient than streaming copy.
200+-- CopyMutableByteArray# translates to genMemcpy in GHC/CmmToAsm/X86/CodeGen.hs
201+-- glibc memcpy copies bytes/words/pages - unrolls the loops:
202+-- https://github.com/bminor/glibc/blob/4290aed05135ae4c0272006442d147f2155e70d7/string/memcpy.c
203+-- https://github.com/bminor/glibc/blob/4290aed05135ae4c0272006442d147f2155e70d7/string/wordcopy.c
204+
205+-- | Put a sub range of a source array into a subrange of a destination array.
206+-- This is not safe as it does not check the bounds of neither the src array
207+-- nor the destination array.
208+{-# INLINE putSliceUnsafe #-}
209+putSliceUnsafe ::
210+ MonadIO m
211+ => MutByteArray
212+ -> Int
213+ -> MutByteArray
214+ -> Int
215+ -> Int
216+ -> m ()
217+putSliceUnsafe src srcStartBytes dst dstStartBytes lenBytes = liftIO $ do
218+#ifdef DEBUG
219+ srcLen <- sizeOfMutableByteArray src
220+ dstLen <- sizeOfMutableByteArray dst
221+ when (srcLen - srcStartBytes < lenBytes)
222+ $ error $ "putSliceUnsafe: src overflow: start" ++ show srcStartBytes
223+ ++ " end " ++ show srcLen ++ " len " ++ show lenBytes
224+ when (dstLen - dstStartBytes < lenBytes)
225+ $ error $ "putSliceUnsafe: dst overflow: start" ++ show dstStartBytes
226+ ++ " end " ++ show dstLen ++ " len " ++ show lenBytes
227+#endif
228+ let !(I# srcStartBytes#) = srcStartBytes
229+ !(I# dstStartBytes#) = dstStartBytes
230+ !(I# lenBytes#) = lenBytes
231+ let arrS# = getMutableByteArray# src
232+ arrD# = getMutableByteArray# dst
233+ IO $ \s# -> (# copyMutableByteArray#
234+ arrS# srcStartBytes# arrD# dstStartBytes# lenBytes# s#
235+ , () #)
236+
237+-- | Unsafe as it does not check whether the start offset and length supplied
238+-- are valid inside the array.
239+{-# INLINE cloneSliceUnsafeAs #-}
240+cloneSliceUnsafeAs :: MonadIO m =>
241+ PinnedState -> Int -> Int -> MutByteArray -> m MutByteArray
242+cloneSliceUnsafeAs ps srcOff srcLen src =
243+ liftIO $ do
244+ mba <- newBytesAs ps srcLen
245+ putSliceUnsafe src srcOff mba 0 srcLen
246+ return mba
247+
248+-- | @cloneSliceUnsafe offset len arr@ clones a slice of the supplied array
249+-- starting at the given offset and equal to the given length.
250+{-# INLINE cloneSliceUnsafe #-}
251+cloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
252+cloneSliceUnsafe = cloneSliceUnsafeAs Unpinned
253+
254+-- | @pinnedCloneSliceUnsafe offset len arr@
255+{-# INLINE pinnedCloneSliceUnsafe #-}
256+pinnedCloneSliceUnsafe :: MonadIO m =>
257+ Int -> Int -> MutByteArray -> m MutByteArray
258+pinnedCloneSliceUnsafe = cloneSliceUnsafeAs Pinned
259+
260+-------------------------------------------------------------------------------
261+-- Pinning & Unpinning
262+-------------------------------------------------------------------------------
263+
264+-- | Return 'True' if the array is allocated in pinned memory.
265+{-# INLINE isPinned #-}
266+isPinned :: MutByteArray -> Bool
267+isPinned (MutByteArray arr#) =
268+ let pinnedInt = I# (isMutableByteArrayPinned# arr#)
269+ in pinnedInt /= 0
270+
271+
272+{-# INLINE cloneMutableArrayWith# #-}
273+cloneMutableArrayWith#
274+ :: (Int# -> State# RealWorld -> (# State# RealWorld
275+ , MutableByteArray# RealWorld #))
276+ -> MutableByteArray# RealWorld
277+ -> State# RealWorld
278+ -> (# State# RealWorld, MutableByteArray# RealWorld #)
279+cloneMutableArrayWith# alloc# arr# s# =
280+ case getSizeofMutableByteArray# arr# s# of
281+ (# s1#, i# #) ->
282+ case alloc# i# s1# of
283+ (# s2#, arr1# #) ->
284+ case copyMutableByteArray# arr# 0# arr1# 0# i# s2# of
285+ s3# -> (# s3#, arr1# #)
286+
287+-- | Return a copy of the array in pinned memory if unpinned, else return the
288+-- original array.
289+{-# INLINE pin #-}
290+pin :: MutByteArray -> IO MutByteArray
291+pin arr@(MutByteArray marr#) =
292+ if isPinned arr
293+ then return arr
294+ else
295+#ifdef DEBUG
296+ do
297+ -- XXX dump stack trace
298+ trace ("pin: Copying array") (return ())
299+#endif
300+ IO
301+ $ \s# ->
302+ case cloneMutableArrayWith# newPinnedByteArray# marr# s# of
303+ (# s1#, marr1# #) -> (# s1#, MutByteArray marr1# #)
304+
305+-- | Return a copy of the array in unpinned memory if pinned, else return the
306+-- original array.
307+{-# INLINE unpin #-}
308+unpin :: MutByteArray -> IO MutByteArray
309+unpin arr@(MutByteArray marr#) =
310+ if not (isPinned arr)
311+ then return arr
312+ else
313+#ifdef DEBUG
314+ do
315+ -- XXX dump stack trace
316+ trace ("unpin: Copying array") (return ())
317+#endif
318+ IO
319+ $ \s# ->
320+ case cloneMutableArrayWith# newByteArray# marr# s# of
321+ (# s1#, marr1# #) -> (# s1#, MutByteArray marr1# #)
core/src/Streamly/Internal/Data/Parser.hs+122−117
@@ -10,6 +10,9 @@
1010module Streamly.Internal.Data.Parser
1111 (
1212 -- * Setup
13+ -- | To execute the code examples provided in this module in ghci, please
14+ -- run the following commands first.
15+ --
1316 -- $setup
1417
1518 module Streamly.Internal.Data.Parser.Type
@@ -281,7 +284,7 @@ import Prelude hiding
281284--
282285{-# INLINE toFold #-}
283286toFold :: Monad m => Parser a m b -> Fold m a b
284−toFold (Parser pstep pinitial pextract) = Fold step initial extract
287+toFold (Parser pstep pinitial pextract) = Fold step initial extract final
285288
286289 where
287290
@@ -309,7 +312,9 @@ toFold (Parser pstep pinitial pextract) = Fold step initial extract
309312 Done n _ -> derror n
310313 Error err -> eerror err
311314
312− extract st = do
315+ extract = error "toFold: parser cannot be used for scanning"
316+
317+ final st = do
313318 r <- pextract st
314319 case r of
315320 Done 0 b -> return b
@@ -327,7 +332,7 @@ toFold (Parser pstep pinitial pextract) = Fold step initial extract
327332--
328333{-# INLINE fromFold #-}
329334fromFold :: Monad m => Fold m a b -> Parser a m b
330−fromFold (Fold fstep finitial fextract) = Parser step initial extract
335+fromFold (Fold fstep finitial _ ffinal) = Parser step initial extract
331336
332337 where
333338
@@ -345,7 +350,7 @@ fromFold (Fold fstep finitial fextract) = Parser step initial extract
345350 FL.Partial s1 -> Partial 0 s1
346351 FL.Done b -> Done 0 b
347352
348− extract = fmap (Done 0) . fextract
353+ extract = fmap (Done 0) . ffinal
349354
350355-- | Convert a Maybe returning fold to an error returning parser. The first
351356-- argument is the error message that the parser would return when the fold
@@ -355,7 +360,7 @@ fromFold (Fold fstep finitial fextract) = Parser step initial extract
355360--
356361{-# INLINE fromFoldMaybe #-}
357362fromFoldMaybe :: Monad m => String -> Fold m a (Maybe b) -> Parser a m b
358−fromFoldMaybe errMsg (Fold fstep finitial fextract) =
363+fromFoldMaybe errMsg (Fold fstep finitial _ ffinal) =
359364 Parser step initial extract
360365
361366 where
@@ -381,7 +386,7 @@ fromFoldMaybe errMsg (Fold fstep finitial fextract) =
381386 Nothing -> Error errMsg
382387
383388 extract s = do
384− res <- fextract s
389+ res <- ffinal s
385390 case res of
386391 Just x -> return $ Done 0 x
387392 Nothing -> return $ Error errMsg
@@ -630,7 +635,7 @@ data Tuple'Fused a b = Tuple'Fused !a !b deriving Show
630635--
631636{-# INLINE takeBetween #-}
632637takeBetween :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
633−takeBetween low high (Fold fstep finitial fextract) =
638+takeBetween low high (Fold fstep finitial _ ffinal) =
634639
635640 Parser step initial (extract streamErr)
636641
@@ -685,7 +690,7 @@ takeBetween low high (Fold fstep finitial fextract) =
685690 then return $ Continue 0 s1
686691 else if i1 < high
687692 then return $ Partial 0 s1
688− else fmap (Done 0) (fextract s)
693+ else fmap (Done 0) (ffinal s)
689694 FL.Done b ->
690695 return
691696 $ if i1 >= low
@@ -695,12 +700,12 @@ takeBetween low high (Fold fstep finitial fextract) =
695700 step (Tuple'Fused i s) a = fstep s a >>= snext i
696701
697702 extract f (Tuple'Fused i s)
698− | i >= low && i <= high = fmap (Done 0) (fextract s)
703+ | i >= low && i <= high = fmap (Done 0) (ffinal s)
699704 | otherwise = return $ Error (f i)
700705
701706 -- XXX Need to make Initial return type Step to deduplicate this
702707 iextract f (Tuple'Fused i s)
703− | i >= low && i <= high = fmap IDone (fextract s)
708+ | i >= low && i <= high = fmap IDone (ffinal s)
704709 | otherwise = return $ IError (f i)
705710
706711-- | Stops after taking exactly @n@ input elements.
@@ -717,7 +722,7 @@ takeBetween low high (Fold fstep finitial fextract) =
717722--
718723{-# INLINE takeEQ #-}
719724takeEQ :: Monad m => Int -> Fold m a b -> Parser a m b
720−takeEQ n (Fold fstep finitial fextract) = Parser step initial extract
725+takeEQ n (Fold fstep finitial _ ffinal) = Parser step initial extract
721726
722727 where
723728
@@ -727,7 +732,7 @@ takeEQ n (Fold fstep finitial fextract) = Parser step initial extract
727732 FL.Partial s ->
728733 if n > 0
729734 then return $ IPartial $ Tuple'Fused 1 s
730− else fmap IDone (fextract s)
735+ else fmap IDone (ffinal s)
731736 FL.Done b -> return $
732737 if n > 0
733738 then IError
@@ -751,7 +756,7 @@ takeEQ n (Fold fstep finitial fextract) = Parser step initial extract
751756 -- assert (n == i1)
752757 Done 0
753758 <$> case res of
754− FL.Partial s -> fextract s
759+ FL.Partial s -> ffinal s
755760 FL.Done b -> return b
756761
757762 extract (Tuple'Fused i _) =
@@ -783,7 +788,7 @@ data TakeGEState s =
783788--
784789{-# INLINE takeGE #-}
785790takeGE :: Monad m => Int -> Fold m a b -> Parser a m b
786−takeGE n (Fold fstep finitial fextract) = Parser step initial extract
791+takeGE n (Fold fstep finitial _ ffinal) = Parser step initial extract
787792
788793 where
789794
@@ -831,7 +836,7 @@ takeGE n (Fold fstep finitial fextract) = Parser step initial extract
831836 $ Error
832837 $ "takeGE: Expecting at least " ++ show n
833838 ++ " elements, input terminated on " ++ show (i - 1)
834− extract (TakeGEGE r) = fmap (Done 0) $ fextract r
839+ extract (TakeGEGE r) = fmap (Done 0) $ ffinal r
835840
836841-------------------------------------------------------------------------------
837842-- Conditional splitting
@@ -892,7 +897,7 @@ takeWhileP predicate (Parser pstep pinitial pextract) =
892897{-# INLINE takeWhile #-}
893898takeWhile :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
894899-- takeWhile cond f = takeWhileP cond (fromFold f)
895−takeWhile predicate (Fold fstep finitial fextract) =
900+takeWhile predicate (Fold fstep finitial _ ffinal) =
896901 Parser step initial extract
897902
898903 where
@@ -911,9 +916,9 @@ takeWhile predicate (Fold fstep finitial fextract) =
911916 $ case fres of
912917 FL.Partial s1 -> Partial 0 s1
913918 FL.Done b -> Done 0 b
914− else Done 1 <$> fextract s
919+ else Done 1 <$> ffinal s
915920
916− extract s = fmap (Done 0) (fextract s)
921+ extract s = fmap (Done 0) (ffinal s)
917922
918923{-
919924-- XXX This may not be composable because of the b argument. We can instead
@@ -933,7 +938,7 @@ takeWhile1 acc cond f = undefined
933938{-# INLINE takeWhile1 #-}
934939takeWhile1 :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
935940-- takeWhile1 cond f = takeWhileP cond (takeBetween 1 maxBound f)
936−takeWhile1 predicate (Fold fstep finitial fextract) =
941+takeWhile1 predicate (Fold fstep finitial _ ffinal) =
937942 Parser step initial extract
938943
939944 where
@@ -963,11 +968,11 @@ takeWhile1 predicate (Fold fstep finitial fextract) =
963968 if predicate a
964969 then process s a
965970 else do
966− b <- fextract s
971+ b <- ffinal s
967972 return $ Done 1 b
968973
969974 extract (Left' _) = return $ Error "takeWhile1: end of input"
970− extract (Right' s) = fmap (Done 0) (fextract s)
975+ extract (Right' s) = fmap (Done 0) (ffinal s)
971976
972977-- | Drain the input as long as the predicate succeeds, running the effects and
973978-- discarding the results.
@@ -996,7 +1001,7 @@ takeFramedByGeneric :: Monad m =>
9961001 -> Maybe (a -> Bool) -- is frame end?
9971002 -> Fold m a b
9981003 -> Parser a m b
999−takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
1004+takeFramedByGeneric esc begin end (Fold fstep finitial _ ffinal) =
10001005
10011006 Parser step initial extract
10021007
@@ -1028,20 +1033,20 @@ takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
10281033 if isEnd a
10291034 then
10301035 if n == 0
1031− then Done 0 <$> fextract s
1036+ then Done 0 <$> ffinal s
10321037 else process s a (n - 1)
10331038 else
10341039 let n1 = if isBegin a then n + 1 else n
10351040 in process s a n1
10361041 Nothing -> -- takeEndBy case
10371042 if isEnd a
1038− then Done 0 <$> fextract s
1043+ then Done 0 <$> ffinal s
10391044 else process s a n
10401045 Nothing -> -- takeStartBy case
10411046 case begin of
10421047 Just isBegin ->
10431048 if isBegin a
1044− then Done 0 <$> fextract s
1049+ then Done 0 <$> ffinal s
10451050 else process s a n
10461051 Nothing ->
10471052 error $ "takeFramedByGeneric: "
@@ -1066,7 +1071,7 @@ takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
10661071 case end of
10671072 Just isEnd ->
10681073 if isEnd a
1069− then Done 0 <$> fextract s
1074+ then Done 0 <$> ffinal s
10701075 else processCheckEsc s a 0
10711076 Nothing ->
10721077 error "Both begin and end frame predicate missing"
@@ -1081,7 +1086,7 @@ takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
10811086 case begin of
10821087 Just _ ->
10831088 case end of
1084− Nothing -> fmap (Done 0) $ fextract s
1089+ Nothing -> fmap (Done 0) $ ffinal s
10851090 Just _ -> err "takeFramedByGeneric: missing frame end"
10861091 Nothing -> err "takeFramedByGeneric: missing closing frame"
10871092 extract (FrameEscEsc _ _) = err "takeFramedByGeneric: trailing escape"
@@ -1120,7 +1125,7 @@ blockWithQuotes :: (Monad m, Eq a) =>
11201125 -> Fold m a b
11211126 -> Parser a m b
11221127blockWithQuotes isEsc isQuote bopen bclose
1123− (Fold fstep finitial fextract) =
1128+ (Fold fstep finitial _ ffinal) =
11241129 Parser step initial extract
11251130
11261131 where
@@ -1150,7 +1155,7 @@ blockWithQuotes isEsc isQuote bopen bclose
11501155 | a == bopen = process s a (BlockUnquoted (level + 1))
11511156 | a == bclose =
11521157 if level == 1
1153− then fmap (Done 0) (fextract s)
1158+ then fmap (Done 0) (ffinal s)
11541159 else process s a (BlockUnquoted (level - 1))
11551160 | isQuote a = process s a (BlockQuoted level)
11561161 | otherwise = process s a (BlockUnquoted level)
@@ -1164,7 +1169,7 @@ blockWithQuotes isEsc isQuote bopen bclose
11641169
11651170 err = return . Error
11661171
1167− extract (BlockInit s) = fmap (Done 0) $ fextract s
1172+ extract (BlockInit s) = fmap (Done 0) $ ffinal s
11681173 extract (BlockUnquoted level _) =
11691174 err $ "blockWithQuotes: finished at block nest level " ++ show level
11701175 extract (BlockQuoted level _) =
@@ -1299,7 +1304,7 @@ takeEitherSepBy _cond = undefined -- D.toParserK . D.takeEitherSepBy cond
12991304--
13001305{-# INLINE takeStartBy #-}
13011306takeStartBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
1302−takeStartBy cond (Fold fstep finitial fextract) =
1307+takeStartBy cond (Fold fstep finitial _ ffinal) =
13031308
13041309 Parser step initial extract
13051310
@@ -1327,10 +1332,10 @@ takeStartBy cond (Fold fstep finitial fextract) =
13271332 step (Right' s) a =
13281333 if not (cond a)
13291334 then process s a
1330− else Done 1 <$> fextract s
1335+ else Done 1 <$> ffinal s
13311336
1332− extract (Left' s) = fmap (Done 0) $ fextract s
1333− extract (Right' s) = fmap (Done 0) $ fextract s
1337+ extract (Left' s) = fmap (Done 0) $ ffinal s
1338+ extract (Right' s) = fmap (Done 0) $ ffinal s
13341339
13351340-- | Like 'takeStartBy' but drops the separator.
13361341--
@@ -1366,7 +1371,7 @@ takeFramedByEsc_ :: Monad m =>
13661371 (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
13671372-- takeFramedByEsc_ isEsc isEnd p =
13681373-- takeFramedByGeneric (Just isEsc) Nothing (Just isEnd) (toFold p)
1369−takeFramedByEsc_ isEsc isBegin isEnd (Fold fstep finitial fextract) =
1374+takeFramedByEsc_ isEsc isBegin isEnd (Fold fstep finitial _ ffinal ) =
13701375
13711376 Parser step initial extract
13721377
@@ -1402,7 +1407,7 @@ takeFramedByEsc_ isEsc isBegin isEnd (Fold fstep finitial fextract) =
14021407 in process s a n1
14031408 else
14041409 if n == 0
1405− then Done 0 <$> fextract s
1410+ then Done 0 <$> ffinal s
14061411 else process s a (n - 1)
14071412 step (FrameEscEsc s n) a = process s a n
14081413
@@ -1424,7 +1429,7 @@ takeFramedBy_ :: Monad m =>
14241429 (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
14251430-- takeFramedBy_ isBegin isEnd =
14261431-- takeFramedByGeneric (Just (const False)) (Just isBegin) (Just isEnd)
1427−takeFramedBy_ isBegin isEnd (Fold fstep finitial fextract) =
1432+takeFramedBy_ isBegin isEnd (Fold fstep finitial _ ffinal) =
14281433
14291434 Parser step initial extract
14301435
@@ -1454,7 +1459,7 @@ takeFramedBy_ isBegin isEnd (Fold fstep finitial fextract) =
14541459 | not (isEnd a) =
14551460 let n1 = if isBegin a then n + 1 else n
14561461 in process s a n1
1457− | n == 0 = Done 0 <$> fextract s
1462+ | n == 0 = Done 0 <$> ffinal s
14581463 | otherwise = process s a (n - 1)
14591464
14601465 err = return . Error
@@ -1489,7 +1494,7 @@ data WordByState s b = WBLeft !s | WBWord !s | WBRight !b
14891494--
14901495{-# INLINE wordBy #-}
14911496wordBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
1492−wordBy predicate (Fold fstep finitial fextract) = Parser step initial extract
1497+wordBy predicate (Fold fstep finitial _ ffinal) = Parser step initial extract
14931498
14941499 where
14951500
@@ -1516,7 +1521,7 @@ wordBy predicate (Fold fstep finitial fextract) = Parser step initial extract
15161521 if not (predicate a)
15171522 then worder s a
15181523 else do
1519− b <- fextract s
1524+ b <- ffinal s
15201525 return $ Partial 0 $ WBRight b
15211526 step (WBRight b) a =
15221527 return
@@ -1524,8 +1529,8 @@ wordBy predicate (Fold fstep finitial fextract) = Parser step initial extract
15241529 then Done 1 b
15251530 else Partial 0 $ WBRight b
15261531
1527− extract (WBLeft s) = fmap (Done 0) $ fextract s
1528− extract (WBWord s) = fmap (Done 0) $ fextract s
1532+ extract (WBLeft s) = fmap (Done 0) $ ffinal s
1533+ extract (WBWord s) = fmap (Done 0) $ ffinal s
15291534 extract (WBRight b) = return (Done 0 b)
15301535
15311536data WordFramedState s b =
@@ -1562,7 +1567,7 @@ wordFramedBy :: Monad m =>
15621567 -> Fold m a b
15631568 -> Parser a m b
15641569wordFramedBy isEsc isBegin isEnd isSep
1565− (Fold fstep finitial fextract) =
1570+ (Fold fstep finitial _ ffinal) =
15661571 Parser step initial extract
15671572
15681573 where
@@ -1593,7 +1598,7 @@ wordFramedBy isEsc isBegin isEnd isSep
15931598 step (WordFramedWord s n) a
15941599 | isEsc a = return $ Continue 0 $ WordFramedEsc s n
15951600 | n == 0 && isSep a = do
1596− b <- fextract s
1601+ b <- ffinal s
15971602 return $ Partial 0 $ WordFramedSkipPost b
15981603 | otherwise = do
15991604 -- We need to use different order for checking begin and end for
@@ -1626,10 +1631,10 @@ wordFramedBy isEsc isBegin isEnd isSep
16261631
16271632 err = return . Error
16281633
1629− extract (WordFramedSkipPre s) = fmap (Done 0) $ fextract s
1634+ extract (WordFramedSkipPre s) = fmap (Done 0) $ ffinal s
16301635 extract (WordFramedWord s n) =
16311636 if n == 0
1632− then fmap (Done 0) $ fextract s
1637+ then fmap (Done 0) $ ffinal s
16331638 else err "wordFramedBy: missing frame end"
16341639 extract (WordFramedEsc _ _) =
16351640 err "wordFramedBy: trailing escape"
@@ -1716,7 +1721,7 @@ wordWithQuotes :: (Monad m, Eq a) =>
17161721 -> Fold m a b
17171722 -> Parser a m b
17181723wordWithQuotes keepQuotes tr escChar toRight isSep
1719− (Fold fstep finitial fextract) =
1724+ (Fold fstep finitial _ ffinal) =
17201725 Parser step initial extract
17211726
17221727 where
@@ -1767,7 +1772,7 @@ wordWithQuotes keepQuotes tr escChar toRight isSep
17671772 step (WordUnquotedWord s) a
17681773 | isEsc a = return $ Continue 0 $ WordUnquotedEsc s
17691774 | isSep a = do
1770− b <- fextract s
1775+ b <- ffinal s
17711776 return $ Partial 0 $ WordQuotedSkipPost b
17721777 | otherwise = do
17731778 case toRight a of
@@ -1815,11 +1820,11 @@ wordWithQuotes keepQuotes tr escChar toRight isSep
18151820
18161821 err = return . Error
18171822
1818− extract (WordQuotedSkipPre s) = fmap (Done 0) $ fextract s
1819− extract (WordUnquotedWord s) = fmap (Done 0) $ fextract s
1823+ extract (WordQuotedSkipPre s) = fmap (Done 0) $ ffinal s
1824+ extract (WordUnquotedWord s) = fmap (Done 0) $ ffinal s
18201825 extract (WordQuotedWord s n _ _) =
18211826 if n == 0
1822− then fmap (Done 0) $ fextract s
1827+ then fmap (Done 0) $ ffinal s
18231828 else err "wordWithQuotes: missing frame end"
18241829 extract WordQuotedEsc {} =
18251830 err "wordWithQuotes: trailing escape"
@@ -1897,7 +1902,7 @@ data GroupByState a s
18971902--
18981903{-# INLINE groupBy #-}
18991904groupBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
1900−groupBy eq (Fold fstep finitial fextract) = Parser step initial extract
1905+groupBy eq (Fold fstep finitial _ ffinal) = Parser step initial extract
19011906
19021907 where
19031908
@@ -1920,10 +1925,10 @@ groupBy eq (Fold fstep finitial fextract) = Parser step initial extract
19201925 step (GroupByGrouping a0 s) a =
19211926 if eq a0 a
19221927 then grouper s a0 a
1923− else Done 1 <$> fextract s
1928+ else Done 1 <$> ffinal s
19241929
1925− extract (GroupByInit s) = fmap (Done 0) $ fextract s
1926− extract (GroupByGrouping _ s) = fmap (Done 0) $ fextract s
1930+ extract (GroupByInit s) = fmap (Done 0) $ ffinal s
1931+ extract (GroupByGrouping _ s) = fmap (Done 0) $ ffinal s
19271932
19281933-- | Unlike 'groupBy' this combinator performs a rolling comparison of two
19291934-- successive elements in the input stream. Assuming the input stream
@@ -1957,7 +1962,7 @@ groupBy eq (Fold fstep finitial fextract) = Parser step initial extract
19571962--
19581963{-# INLINE groupByRolling #-}
19591964groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
1960−groupByRolling eq (Fold fstep finitial fextract) = Parser step initial extract
1965+groupByRolling eq (Fold fstep finitial _ ffinal) = Parser step initial extract
19611966
19621967 where
19631968
@@ -1980,10 +1985,10 @@ groupByRolling eq (Fold fstep finitial fextract) = Parser step initial extract
19801985 step (GroupByGrouping a0 s) a =
19811986 if eq a0 a
19821987 then grouper s a
1983− else Done 1 <$> fextract s
1988+ else Done 1 <$> ffinal s
19841989
1985− extract (GroupByInit s) = fmap (Done 0) $ fextract s
1986− extract (GroupByGrouping _ s) = fmap (Done 0) $ fextract s
1990+ extract (GroupByInit s) = fmap (Done 0) $ ffinal s
1991+ extract (GroupByGrouping _ s) = fmap (Done 0) $ ffinal s
19871992
19881993{-# ANN type GroupByStatePair Fuse #-}
19891994data GroupByStatePair a s1 s2
@@ -2007,8 +2012,8 @@ groupByRollingEither :: Monad m =>
20072012 (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
20082013groupByRollingEither
20092014 eq
2010− (Fold fstep1 finitial1 fextract1)
2011− (Fold fstep2 finitial2 fextract2) = Parser step initial extract
2015+ (Fold fstep1 finitial1 _ ffinal1)
2016+ (Fold fstep2 finitial2 _ ffinal2) = Parser step initial extract
20122017
20132018 where
20142019
@@ -2067,21 +2072,21 @@ groupByRollingEither
20672072 step (GroupByGroupingPairL a0 s1 s2) a =
20682073 if not (eq a0 a)
20692074 then grouperL2 s1 s2 a
2070− else Done 1 . Left <$> fextract1 s1
2075+ else Done 1 . Left <$> ffinal1 s1
20712076
20722077 step (GroupByGroupingPairR a0 s1 s2) a =
20732078 if eq a0 a
20742079 then grouperR2 s1 s2 a
2075− else Done 1 . Right <$> fextract2 s2
2080+ else Done 1 . Right <$> ffinal2 s2
20762081
2077− extract (GroupByInitPair s1 _) = Done 0 . Left <$> fextract1 s1
2078− extract (GroupByGroupingPairL _ s1 _) = Done 0 . Left <$> fextract1 s1
2079− extract (GroupByGroupingPairR _ _ s2) = Done 0 . Right <$> fextract2 s2
2082+ extract (GroupByInitPair s1 _) = Done 0 . Left <$> ffinal1 s1
2083+ extract (GroupByGroupingPairL _ s1 _) = Done 0 . Left <$> ffinal1 s1
2084+ extract (GroupByGroupingPairR _ _ s2) = Done 0 . Right <$> ffinal2 s2
20802085 extract (GroupByGroupingPair a s1 _) = do
20812086 res <- fstep1 s1 a
20822087 case res of
20832088 FL.Done b -> return $ Done 0 (Left b)
2084− FL.Partial s11 -> Done 0 . Left <$> fextract1 s11
2089+ FL.Partial s11 -> Done 0 . Left <$> ffinal1 s11
20852090
20862091-- XXX use an Unfold instead of a list?
20872092-- XXX custom combinators for matching list, array and stream?
@@ -2234,7 +2239,7 @@ postscan = undefined
22342239{-# INLINE zipWithM #-}
22352240zipWithM :: Monad m =>
22362241 (a -> b -> m c) -> D.Stream m a -> Fold m c x -> Parser b m x
2237−zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
2242+zipWithM zf (D.Stream sstep state) (Fold fstep finitial _ ffinal) =
22382243 Parser step initial extract
22392244
22402245 where
@@ -2247,7 +2252,7 @@ zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
22472252 case r of
22482253 D.Yield x s -> return $ IPartial (Just' x, s, fs)
22492254 D.Stop -> do
2250− x <- fextract fs
2255+ x <- ffinal fs
22512256 return $ IDone x
22522257 -- Need Skip/Continue in initial to loop right here
22532258 D.Skip s -> return $ IPartial (Nothing', s, fs)
@@ -2262,7 +2267,7 @@ zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
22622267 case r of
22632268 D.Yield x1 s -> return $ Continue 0 (Just' x1, s, fs1)
22642269 D.Stop -> do
2265− x <- fextract fs1
2270+ x <- ffinal fs1
22662271 return $ Done 0 x
22672272 D.Skip s -> return $ Continue 1 (Nothing', s, fs1)
22682273 FL.Done x -> return $ Done 0 x
@@ -2277,7 +2282,7 @@ zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
22772282 return $ Continue 0 (Nothing', s, fs1)
22782283 FL.Done x -> return $ Done 0 x
22792284 D.Stop -> do
2280− x <- fextract fs
2285+ x <- ffinal fs
22812286 return $ Done 1 x
22822287 D.Skip s -> return $ Continue 1 (Nothing', s, fs)
22832288
@@ -2561,7 +2566,7 @@ deintercalateAll :: Monad m =>
25612566deintercalateAll
25622567 (Parser stepL initialL extractL)
25632568 (Parser stepR initialR _)
2564− (Fold fstep finitial fextract) = Parser step initial extract
2569+ (Fold fstep finitial _ ffinal) = Parser step initial extract
25652570
25662571 where
25672572
@@ -2632,9 +2637,9 @@ deintercalateAll
26322637 extractResult n fs r = do
26332638 res <- fstep fs r
26342639 case res of
2635− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
2640+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
26362641 FL.Done c -> return (Done n c)
2637− extract (DeintercalateAllInitL fs) = fmap (Done 0) $ fextract fs
2642+ extract (DeintercalateAllInitL fs) = fmap (Done 0) $ ffinal fs
26382643 extract (DeintercalateAllL fs sL) = do
26392644 r <- extractL sL
26402645 case r of
@@ -2642,7 +2647,7 @@ deintercalateAll
26422647 Error err -> return $ Error err
26432648 Continue n s -> return $ Continue n (DeintercalateAllL fs s)
26442649 Partial _ _ -> error "Partial in extract"
2645− extract (DeintercalateAllInitR fs) = fmap (Done 0) $ fextract fs
2650+ extract (DeintercalateAllInitR fs) = fmap (Done 0) $ ffinal fs
26462651 extract (DeintercalateAllR _ _) =
26472652 return $ Error "deintercalateAll: input ended at 'Right' value"
26482653
@@ -2685,7 +2690,7 @@ deintercalate :: Monad m =>
26852690deintercalate
26862691 (Parser stepL initialL extractL)
26872692 (Parser stepR initialR _)
2688− (Fold fstep finitial fextract) = Parser step initial extract
2693+ (Fold fstep finitial _ ffinal) = Parser step initial extract
26892694
26902695 where
26912696
@@ -2716,7 +2721,7 @@ deintercalate
27162721 Done n b ->
27172722 processL (fstep fs (Left b)) n DeintercalateInitR
27182723 Error _ -> do
2719− xs <- fextract fs
2724+ xs <- ffinal fs
27202725 return $ Done cnt1 xs
27212726
27222727 {-# INLINE processR #-}
@@ -2736,7 +2741,7 @@ deintercalate
27362741 Continue n s -> return $ Continue n (DeintercalateR (cnt1 - n) fs s)
27372742 Done n b -> processR (cnt1 - n) b fs n
27382743 Error _ -> do
2739− xs <- fextract fs
2744+ xs <- ffinal fs
27402745 return $ Done cnt1 xs
27412746
27422747 step (DeintercalateInitL fs) a = do
@@ -2771,17 +2776,17 @@ deintercalate
27712776 -- XXX We could have the fold accept pairs of (bR, bL)
27722777 FL.Done _ -> error "Fold terminated consuming partial input"
27732778 Error _ -> do
2774− xs <- fextract fs
2779+ xs <- ffinal fs
27752780 return $ Done cnt1 xs
27762781
27772782 {-# INLINE extractResult #-}
27782783 extractResult n fs r = do
27792784 res <- fstep fs r
27802785 case res of
2781− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
2786+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
27822787 FL.Done c -> return (Done n c)
27832788
2784− extract (DeintercalateInitL fs) = fmap (Done 0) $ fextract fs
2789+ extract (DeintercalateInitL fs) = fmap (Done 0) $ ffinal fs
27852790 extract (DeintercalateL cnt fs sL) = do
27862791 r <- extractL sL
27872792 case r of
@@ -2789,10 +2794,10 @@ deintercalate
27892794 Continue n s -> return $ Continue n (DeintercalateL (cnt - n) fs s)
27902795 Partial _ _ -> error "Partial in extract"
27912796 Error _ -> do
2792− xs <- fextract fs
2797+ xs <- ffinal fs
27932798 return $ Done cnt xs
2794− extract (DeintercalateInitR fs) = fmap (Done 0) $ fextract fs
2795− extract (DeintercalateR cnt fs _) = fmap (Done cnt) $ fextract fs
2799+ extract (DeintercalateInitR fs) = fmap (Done 0) $ ffinal fs
2800+ extract (DeintercalateR cnt fs _) = fmap (Done cnt) $ ffinal fs
27962801 extract (DeintercalateRL cnt bR fs sL) = do
27972802 r <- extractL sL
27982803 case r of
@@ -2804,7 +2809,7 @@ deintercalate
28042809 Continue n s -> return $ Continue n (DeintercalateRL (cnt - n) bR fs s)
28052810 Partial _ _ -> error "Partial in extract"
28062811 Error _ -> do
2807− xs <- fextract fs
2812+ xs <- ffinal fs
28082813 return $ Done cnt xs
28092814
28102815{-# ANN type Deintercalate1State Fuse #-}
@@ -2841,7 +2846,7 @@ deintercalate1 :: Monad m =>
28412846deintercalate1
28422847 (Parser stepL initialL extractL)
28432848 (Parser stepR initialR _)
2844− (Fold fstep finitial fextract) = Parser step initial extract
2849+ (Fold fstep finitial _ ffinal) = Parser step initial extract
28452850
28462851 where
28472852
@@ -2895,7 +2900,7 @@ deintercalate1
28952900 Continue n s -> return $ Continue n (Deintercalate1R (cnt1 - n) fs s)
28962901 Done n b -> processR (cnt1 - n) b fs n
28972902 Error _ -> do
2898− xs <- fextract fs
2903+ xs <- ffinal fs
28992904 return $ Done cnt1 xs
29002905
29012906 step (Deintercalate1InitL cnt fs sL) a = runStepInitL cnt fs sL a
@@ -2924,14 +2929,14 @@ deintercalate1
29242929 -- XXX We could have the fold accept pairs of (bR, bL)
29252930 FL.Done _ -> error "Fold terminated consuming partial input"
29262931 Error _ -> do
2927− xs <- fextract fs
2932+ xs <- ffinal fs
29282933 return $ Done cnt1 xs
29292934
29302935 {-# INLINE extractResult #-}
29312936 extractResult n fs r = do
29322937 res <- fstep fs r
29332938 case res of
2934− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
2939+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
29352940 FL.Done c -> return (Done n c)
29362941
29372942 extract (Deintercalate1InitL cnt fs sL) = do
@@ -2941,8 +2946,8 @@ deintercalate1
29412946 Continue n s -> return $ Continue n (Deintercalate1InitL (cnt - n) fs s)
29422947 Partial _ _ -> error "Partial in extract"
29432948 Error err -> return $ Error err
2944− extract (Deintercalate1InitR fs) = fmap (Done 0) $ fextract fs
2945− extract (Deintercalate1R cnt fs _) = fmap (Done cnt) $ fextract fs
2949+ extract (Deintercalate1InitR fs) = fmap (Done 0) $ ffinal fs
2950+ extract (Deintercalate1R cnt fs _) = fmap (Done cnt) $ ffinal fs
29462951 extract (Deintercalate1RL cnt bR fs sL) = do
29472952 r <- extractL sL
29482953 case r of
@@ -2954,7 +2959,7 @@ deintercalate1
29542959 Continue n s -> return $ Continue n (Deintercalate1RL (cnt - n) bR fs s)
29552960 Partial _ _ -> error "Partial in extract"
29562961 Error _ -> do
2957− xs <- fextract fs
2962+ xs <- ffinal fs
29582963 return $ Done cnt xs
29592964
29602965{-# ANN type SepByState Fuse #-}
@@ -2997,7 +3002,7 @@ sepBy :: Monad m =>
29973002sepBy
29983003 (Parser stepL initialL extractL)
29993004 (Parser stepR initialR _)
3000− (Fold fstep finitial fextract) = Parser step initial extract
3005+ (Fold fstep finitial _ ffinal) = Parser step initial extract
30013006
30023007 where
30033008
@@ -3028,7 +3033,7 @@ sepBy
30283033 Done n b ->
30293034 processL (fstep fs b) n SepByInitR
30303035 Error _ -> do
3031− xs <- fextract fs
3036+ xs <- ffinal fs
30323037 return $ Done cnt1 xs
30333038
30343039 {-# INLINE processR #-}
@@ -3048,7 +3053,7 @@ sepBy
30483053 Continue n s -> return $ Continue n (SepByR (cnt1 - n) fs s)
30493054 Done n _ -> processR (cnt1 - n) fs n
30503055 Error _ -> do
3051− xs <- fextract fs
3056+ xs <- ffinal fs
30523057 return $ Done cnt1 xs
30533058
30543059 step (SepByInitL fs) a = do
@@ -3070,10 +3075,10 @@ sepBy
30703075 extractResult n fs r = do
30713076 res <- fstep fs r
30723077 case res of
3073− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
3078+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
30743079 FL.Done c -> return (Done n c)
30753080
3076− extract (SepByInitL fs) = fmap (Done 0) $ fextract fs
3081+ extract (SepByInitL fs) = fmap (Done 0) $ ffinal fs
30773082 extract (SepByL cnt fs sL) = do
30783083 r <- extractL sL
30793084 case r of
@@ -3081,10 +3086,10 @@ sepBy
30813086 Continue n s -> return $ Continue n (SepByL (cnt - n) fs s)
30823087 Partial _ _ -> error "Partial in extract"
30833088 Error _ -> do
3084− xs <- fextract fs
3089+ xs <- ffinal fs
30853090 return $ Done cnt xs
3086− extract (SepByInitR fs) = fmap (Done 0) $ fextract fs
3087− extract (SepByR cnt fs _) = fmap (Done cnt) $ fextract fs
3091+ extract (SepByInitR fs) = fmap (Done 0) $ ffinal fs
3092+ extract (SepByR cnt fs _) = fmap (Done cnt) $ ffinal fs
30883093
30893094-- | Non-backtracking version of sepBy. Several times faster.
30903095{-# INLINE sepByAll #-}
@@ -3139,7 +3144,7 @@ sepBy1 :: Monad m =>
31393144sepBy1
31403145 (Parser stepL initialL extractL)
31413146 (Parser stepR initialR _)
3142− (Fold fstep finitial fextract) = Parser step initial extract
3147+ (Fold fstep finitial _ ffinal) = Parser step initial extract
31433148
31443149 where
31453150
@@ -3186,7 +3191,7 @@ sepBy1
31863191 Done n b ->
31873192 processL (fstep fs b) n SepBy1InitR
31883193 Error _ -> do
3189− xs <- fextract fs
3194+ xs <- ffinal fs
31903195 return $ Done cnt1 xs
31913196
31923197 {-# INLINE processR #-}
@@ -3206,7 +3211,7 @@ sepBy1
32063211 Continue n s -> return $ Continue n (SepBy1R (cnt1 - n) fs s)
32073212 Done n _ -> processR (cnt1 - n) fs n
32083213 Error _ -> do
3209− xs <- fextract fs
3214+ xs <- ffinal fs
32103215 return $ Done cnt1 xs
32113216
32123217 step (SepBy1InitL cnt fs sL) a = runStepInitL cnt fs sL a
@@ -3223,7 +3228,7 @@ sepBy1
32233228 extractResult n fs r = do
32243229 res <- fstep fs r
32253230 case res of
3226− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
3231+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
32273232 FL.Done c -> return (Done n c)
32283233
32293234 extract (SepBy1InitL cnt fs sL) = do
@@ -3240,10 +3245,10 @@ sepBy1
32403245 Continue n s -> return $ Continue n (SepBy1L (cnt - n) fs s)
32413246 Partial _ _ -> error "Partial in extract"
32423247 Error _ -> do
3243− xs <- fextract fs
3248+ xs <- ffinal fs
32443249 return $ Done cnt xs
3245− extract (SepBy1InitR fs) = fmap (Done 0) $ fextract fs
3246− extract (SepBy1R cnt fs _) = fmap (Done cnt) $ fextract fs
3250+ extract (SepBy1InitR fs) = fmap (Done 0) $ ffinal fs
3251+ extract (SepBy1R cnt fs _) = fmap (Done cnt) $ ffinal fs
32473252
32483253-------------------------------------------------------------------------------
32493254-- Interleaving a collection of parsers
@@ -3273,7 +3278,7 @@ roundRobin _ps _f = undefined
32733278{-# INLINE sequence #-}
32743279sequence :: Monad m =>
32753280 D.Stream m (Parser a m b) -> Fold m b c -> Parser a m c
3276−sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
3281+sequence (D.Stream sstep sstate) (Fold fstep finitial _ ffinal) =
32773282 Parser step initial extract
32783283
32793284 where
@@ -3291,7 +3296,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
32913296 case sres of
32923297 D.Yield p ss1 -> return $ Continue 1 (Just' p, ss1, fs)
32933298 D.Stop -> do
3294− c <- fextract fs
3299+ c <- ffinal fs
32953300 return $ Done 1 c
32963301 D.Skip ss1 -> return $ Continue 1 (Nothing', ss1, fs)
32973302
@@ -3327,7 +3332,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
33273332 FL.Done c -> return $ Done 1 c
33283333 IError err -> return $ Error err
33293334
3330− extract (Nothing', _, fs) = fmap (Done 0) $ fextract fs
3335+ extract (Nothing', _, fs) = fmap (Done 0) $ ffinal fs
33313336 extract (Just' (Parser pstep pinit pextr), ss, fs) = do
33323337 ps <- pinit
33333338 case ps of
@@ -3337,7 +3342,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
33373342 Done n b -> do
33383343 res <- fstep fs b
33393344 case res of
3340− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
3345+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
33413346 FL.Done c -> return (Done n c)
33423347 Error err -> return $ Error err
33433348 Continue n s -> return $ Continue n (Just' (Parser pstep (return (IPartial s)) pextr), ss, fs)
@@ -3345,7 +3350,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
33453350 IDone b -> do
33463351 fres <- fstep fs b
33473352 case fres of
3348− FL.Partial fs1 -> fmap (Done 0) $ fextract fs1
3353+ FL.Partial fs1 -> fmap (Done 0) $ ffinal fs1
33493354 FL.Done c -> return (Done 0 c)
33503355 IError err -> return $ Error err
33513356
@@ -3480,7 +3485,7 @@ manyTill :: Monad m
34803485 => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
34813486manyTill (Parser stepL initialL extractL)
34823487 (Parser stepR initialR _)
3483− (Fold fstep finitial fextract) =
3488+ (Fold fstep finitial _ ffinal) =
34843489 Parser step initial extract
34853490
34863491 where
@@ -3502,7 +3507,7 @@ manyTill (Parser stepL initialL extractL)
35023507 resR <- initialR
35033508 case resR of
35043509 IPartial sr -> return $ p (ManyTillR 0 fs sr)
3505− IDone _ -> d <$> fextract fs
3510+ IDone _ -> d <$> ffinal fs
35063511 IError _ -> scrutL fs p c d e
35073512
35083513 initial = do
@@ -3519,7 +3524,7 @@ manyTill (Parser stepL initialL extractL)
35193524 assertM(cnt + 1 - n >= 0)
35203525 return $ Continue n (ManyTillR (cnt + 1 - n) fs s)
35213526 Done n _ -> do
3522− b <- fextract fs
3527+ b <- ffinal fs
35233528 return $ Done n b
35243529 Error _ -> do
35253530 resL <- initialL
@@ -3558,12 +3563,12 @@ manyTill (Parser stepL initialL extractL)
35583563 Done n b -> do
35593564 r <- fstep fs b
35603565 case r of
3561− FL.Partial fs1 -> fmap (Done n) $ fextract fs1
3566+ FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
35623567 FL.Done c -> return (Done n c)
35633568 Error err -> return $ Error err
35643569 Continue n s -> return $ Continue n (ManyTillL fs s)
35653570 Partial _ _ -> error "Partial in extract"
3566− extract (ManyTillR _ fs _) = fmap (Done 0) $ fextract fs
3571+ extract (ManyTillR _ fs _) = fmap (Done 0) $ ffinal fs
35673572
35683573-- | @manyThen f collect recover@ repeats the parser @collect@ on the input and
35693574-- collects the output in the supplied fold. If the the parser @collect@ fails,
core/src/Streamly/Internal/Data/Parser/Type.hs+89−69
@@ -175,9 +175,6 @@
175175
176176module Streamly.Internal.Data.Parser.Type
177177 (
178− -- * Setup
179− -- $setup
180−
181178 -- * Types
182179 Initial (..)
183180 , Step (..)
@@ -441,6 +438,7 @@ newtype ParseError = ParseError String
441438instance Exception ParseError where
442439 displayException (ParseError err) = err
443440
441+-- | Map a function on the result i.e. on @b@ in @Parser a m b@.
444442instance Functor m => Functor (Parser a m) where
445443 {-# INLINE fmap #-}
446444 fmap f (Parser step1 initial1 extract) =
@@ -507,26 +505,30 @@ data SeqParseState sl f sr = SeqParseL !sl | SeqParseR !f !sr
507505-- 'Streamly.Data.Stream.append', it splits the streams using two parsers and
508506-- zips the results.
509507
510−-- | Sequential parser application.
508+-- | Sequential parser application. Apply two parsers sequentially to an input
509+-- stream. The first parser runs and processes the input, the remaining input
510+-- is then passed to the second parser. If both parsers succeed, their outputs
511+-- are combined using the supplied function. If either parser fails, the
512+-- operation fails.
511513--
512−-- Apply two parsers sequentially to an input stream. The first parser runs and
513−-- processes the input, the remaining input is then passed to the second
514−-- parser. If both parsers succeed, their outputs are combined using the
515−-- supplied function. If either parser fails, the operation fails.
514+-- This combinator delivers high performance by stream fusion but it comes with
515+-- some limitations. For those cases use the 'Applicative' instance of
516+-- 'Streamly.Data.ParserK.ParserK'.
516517--
517−-- This implementation is strict in the second argument, therefore, the
518−-- following will fail:
518+-- CAVEAT 1: NO RECURSION. This function is strict in both arguments. As a
519+-- result, if a parser is defined recursively using this, it may cause an
520+-- infintie loop. The following example checks the strictness:
519521--
520−-- >>> Stream.parse (Parser.splitWith const (Parser.satisfy (> 0)) undefined) $ Stream.fromList [1]
522+-- >>> p = Parser.splitWith const (Parser.satisfy (> 0)) undefined
523+-- >>> Stream.parse p $ Stream.fromList [1]
521524-- *** Exception: Prelude.undefined
522525-- ...
523526--
524−-- Although this implementation allows stream fusion, it has quadratic
525−-- complexity, making it suitable only for a small number of compositions.
526−-- As a thumb rule use it for less than 8 compositions, use ParserK otherwise.
527+-- CAVEAT 2: QUADRATIC TIME COMPLEXITY. Static composition is fast due to
528+-- stream fusion, but it works well only for limited (e.g. up to 8)
529+-- compositions, use "Streamly.Data.ParserK" for larger compositions.
527530--
528−-- Below are some common idioms that can be expressed using 'splitWith' and
529−-- other parser primitives:
531+-- Below are some common idioms that can be expressed using 'splitWith':
530532--
531533-- >>> span p f1 f2 = Parser.splitWith (,) (Parser.takeWhile p f1) (Parser.fromFold f2)
532534-- >>> spanBy eq f1 f2 = Parser.splitWith (,) (Parser.groupBy eq f1) (Parser.fromFold f2)
@@ -615,7 +617,12 @@ splitWith func (Parser stepL initialL extractL)
615617-- We can use the compiler to automatically annotate accumulators, terminating
616618-- folds, non-failing parsers and failing parsers.
617619
618−-- | Works correctly only if both the parsers are guaranteed to never fail.
620+-- | Better performance 'splitWith' for non-failing parsers.
621+--
622+-- Does not work correctly for parsers that can fail.
623+--
624+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
625+--
619626{-# INLINE noErrorUnsafeSplitWith #-}
620627noErrorUnsafeSplitWith :: Monad m
621628 => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
@@ -686,6 +693,8 @@ data SeqAState sl sr = SeqAL !sl | SeqAR !sr
686693-- second parser. The output of the parser is the output of the second parser.
687694-- The operation fails if any of the parsers fail.
688695--
696+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
697+--
689698-- This implementation is strict in the second argument, therefore, the
690699-- following will fail:
691700--
@@ -693,10 +702,6 @@ data SeqAState sl sr = SeqAL !sl | SeqAR !sr
693702-- *** Exception: Prelude.undefined
694703-- ...
695704--
696−-- Although this implementation allows stream fusion, it has quadratic
697−-- complexity, making it suitable only for a small number of compositions.
698−-- As a thumb rule use it for less than 8 compositions, use ParserK otherwise.
699−--
700705-- /Pre-release/
701706--
702707{-# INLINE split_ #-}
@@ -752,7 +757,12 @@ split_ (Parser stepL initialL extractL) (Parser stepR initialR extractR) =
752757 Partial _ _ -> error "split_: Partial"
753758 Continue n s -> return $ Continue n (SeqAL s)
754759
755−-- For backtracking folds
760+-- | Better performance 'split_' for non-failing parsers.
761+--
762+-- Does not work correctly for parsers that can fail.
763+--
764+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
765+--
756766{-# INLINE noErrorUnsafeSplit_ #-}
757767noErrorUnsafeSplit_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
758768noErrorUnsafeSplit_
@@ -807,7 +817,11 @@ noErrorUnsafeSplit_
807817 Partial _ _ -> error "split_: Partial"
808818 Continue n s -> return $ Continue n (SeqAL s)
809819
810−-- | 'Applicative' form of 'splitWith'.
820+-- | READ THE CAVEATS in 'splitWith' before using this instance.
821+--
822+-- >>> pure = Parser.fromPure
823+-- >>> (<*>) = Parser.splitWith id
824+-- >>> (*>) = Parser.split_
811825instance Monad m => Applicative (Parser a m) where
812826 {-# INLINE pure #-}
813827 pure = fromPure
@@ -833,20 +847,26 @@ data AltParseState sl sr = AltParseL !Int !sl | AltParseR !sr
833847-- each subsequent alternative's input element has to go through, therefore, it
834848-- cannot scale to a large number of compositions
835849
836−-- | Sequential alternative. The input is first passed to the first parser, and
850+-- | Sequential alternative. The input is first passed to the first parser,
837851-- if it succeeds, the result is returned. However, if the first parser fails,
838−-- the parser driver backtracks and tries the same input on the second parser,
839−-- returning the result if it succeeds.
852+-- the parser driver backtracks and tries the same input on the second
853+-- (alternative) parser, returning the result if it succeeds.
854+--
855+-- This combinator delivers high performance by stream fusion but it comes with
856+-- some limitations. For those cases use the 'Alternative' instance of
857+-- 'Streamly.Data.ParserK.ParserK'.
840858--
841−-- Note: This implementation is not lazy in the second argument. The following
842−-- will fail:
859+-- CAVEAT 1: NO RECURSION. This function is strict in both arguments. As a
860+-- result, if a parser is defined recursively using this, it may cause an
861+-- infintie loop. The following example checks the strictness:
843862--
844−-- >> Stream.parse (Parser.satisfy (> 0) `Parser.alt` undefined) $ Stream.fromList [1..10]
863+-- >>> p = Parser.satisfy (> 0) `Parser.alt` undefined
864+-- >>> Stream.parse p $ Stream.fromList [1..10]
845865-- *** Exception: Prelude.undefined
846866--
847−-- Although this implementation allows stream fusion, it has quadratic
848−-- complexity, making it suitable only for a small number of compositions.
849−-- As a thumb rule use it for less than 8 compositions, use ParserK otherwise.
867+-- CAVEAT 2: QUADRATIC TIME COMPLEXITY. Static composition is fast due to
868+-- stream fusion, but it works well only for limited (e.g. up to 8)
869+-- compositions, use "Streamly.Data.ParserK" for larger compositions.
850870--
851871-- /Time Complexity:/ O(n^2) where n is the number of compositions.
852872--
@@ -927,7 +947,7 @@ data Fused3 a b c = Fused3 !a !b !c
927947--
928948{-# INLINE splitMany #-}
929949splitMany :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
930−splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
950+splitMany (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
931951 Parser step initial extract
932952
933953 where
@@ -943,7 +963,7 @@ splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
943963 IPartial ps -> return $ partial $ Fused3 ps 0 fs
944964 IDone pb ->
945965 runCollectorWith (handleCollect partial done) fs pb
946− IError _ -> done <$> fextract fs
966+ IError _ -> done <$> ffinal fs
947967 FL.Done fb -> return $ done fb
948968
949969 runCollectorWith cont fs pb = fstep fs pb >>= cont
@@ -967,19 +987,19 @@ splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
967987 assertM(cnt1 - n >= 0)
968988 fstep fs b >>= handleCollect (Partial n) (Done n)
969989 Error _ -> do
970− xs <- fextract fs
990+ xs <- ffinal fs
971991 return $ Done cnt xs
972992
973− extract (Fused3 _ 0 fs) = fmap (Done 0) (fextract fs)
993+ extract (Fused3 _ 0 fs) = fmap (Done 0) (ffinal fs)
974994 extract (Fused3 s cnt fs) = do
975995 r <- extract1 s
976996 case r of
977− Error _ -> fmap (Done cnt) (fextract fs)
997+ Error _ -> fmap (Done cnt) (ffinal fs)
978998 Done n b -> do
979999 assertM(n <= cnt)
9801000 fs1 <- fstep fs b
9811001 case fs1 of
982− FL.Partial s1 -> fmap (Done n) (fextract s1)
1002+ FL.Partial s1 -> fmap (Done n) (ffinal s1)
9831003 FL.Done b1 -> return (Done n b1)
9841004 Partial _ _ -> error "splitMany: Partial in extract"
9851005 Continue n s1 -> do
@@ -993,7 +1013,7 @@ splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
9931013--
9941014{-# INLINE splitManyPost #-}
9951015splitManyPost :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
996−splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
1016+splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
9971017 Parser step initial extract
9981018
9991019 where
@@ -1009,7 +1029,7 @@ splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10091029 IPartial ps -> return $ partial $ Fused3 ps 0 fs
10101030 IDone pb ->
10111031 runCollectorWith (handleCollect partial done) fs pb
1012− IError _ -> done <$> fextract fs
1032+ IError _ -> done <$> ffinal fs
10131033 FL.Done fb -> return $ done fb
10141034
10151035 runCollectorWith cont fs pb = fstep fs pb >>= cont
@@ -1031,18 +1051,18 @@ splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10311051 assertM(cnt1 - n >= 0)
10321052 fstep fs b >>= handleCollect (Partial n) (Done n)
10331053 Error _ -> do
1034− xs <- fextract fs
1054+ xs <- ffinal fs
10351055 return $ Done cnt1 xs
10361056
10371057 extract (Fused3 s cnt fs) = do
10381058 r <- extract1 s
10391059 case r of
1040− Error _ -> fmap (Done cnt) (fextract fs)
1060+ Error _ -> fmap (Done cnt) (ffinal fs)
10411061 Done n b -> do
10421062 assertM(n <= cnt)
10431063 fs1 <- fstep fs b
10441064 case fs1 of
1045− FL.Partial s1 -> fmap (Done n) (fextract s1)
1065+ FL.Partial s1 -> fmap (Done n) (ffinal s1)
10461066 FL.Done b1 -> return (Done n b1)
10471067 Partial _ _ -> error "splitMany: Partial in extract"
10481068 Continue n s1 -> do
@@ -1055,7 +1075,7 @@ splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10551075--
10561076{-# INLINE splitSome #-}
10571077splitSome :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
1058−splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
1078+splitSome (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
10591079 Parser step initial extract
10601080
10611081 where
@@ -1071,7 +1091,7 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10711091 IPartial ps -> return $ partial $ Fused3 ps 0 $ Right fs
10721092 IDone pb ->
10731093 runCollectorWith (handleCollect partial done) fs pb
1074− IError _ -> done <$> fextract fs
1094+ IError _ -> done <$> ffinal fs
10751095 FL.Done fb -> return $ done fb
10761096
10771097 runCollectorWith cont fs pb = fstep fs pb >>= cont
@@ -1121,7 +1141,7 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
11211141 Done n b -> do
11221142 assertM(cnt1 - n >= 0)
11231143 fstep fs b >>= handleCollect (Partial n) (Done n)
1124− Error _ -> Done cnt1 <$> fextract fs
1144+ Error _ -> Done cnt1 <$> ffinal fs
11251145
11261146 extract (Fused3 s cnt (Left fs)) = do
11271147 r <- extract1 s
@@ -1131,7 +1151,7 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
11311151 assertM(n <= cnt)
11321152 fs1 <- fstep fs b
11331153 case fs1 of
1134− FL.Partial s1 -> fmap (Done n) (fextract s1)
1154+ FL.Partial s1 -> fmap (Done n) (ffinal s1)
11351155 FL.Done b1 -> return (Done n b1)
11361156 Partial _ _ -> error "splitSome: Partial in extract"
11371157 Continue n s1 -> do
@@ -1140,12 +1160,12 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
11401160 extract (Fused3 s cnt (Right fs)) = do
11411161 r <- extract1 s
11421162 case r of
1143− Error _ -> fmap (Done cnt) (fextract fs)
1163+ Error _ -> fmap (Done cnt) (ffinal fs)
11441164 Done n b -> do
11451165 assertM(n <= cnt)
11461166 fs1 <- fstep fs b
11471167 case fs1 of
1148− FL.Partial s1 -> fmap (Done n) (fextract s1)
1168+ FL.Partial s1 -> fmap (Done n) (ffinal s1)
11491169 FL.Done b1 -> return (Done n b1)
11501170 Partial _ _ -> error "splitSome: Partial in extract"
11511171 Continue n s1 -> do
@@ -1176,23 +1196,12 @@ dieM err = Parser undefined (IError <$> err) undefined
11761196-- Note: With the direct style parser type, the list in "some" and "many" is
11771197-- accumulated strictly, it cannot be consumed lazily.
11781198
1179−-- | Sequential alternative. The input is first passed to the first parser, and
1180−-- if it succeeds, the result is returned. However, if the first parser fails,
1181−-- the parser driver backtracks and tries the same input on the second parser,
1182−-- returning the result if it succeeds.
1183−--
1184−-- Note: The implementation of '<|>' is not lazy in the second
1185−-- argument. The following code will fail:
1186−--
1187−-- >>> Stream.parse (Parser.satisfy (> 0) <|> undefined) $ Stream.fromList [1..10]
1188−-- *** Exception: Prelude.undefined
1189−-- ...
1199+-- | READ THE CAVEATS in 'alt' before using this instance.
11901200--
1191−-- WARNING! this is not suitable for large scale use. As a thumb rule stream
1192−-- fusion works well for less than 8 compositions of this operation, otherwise
1193−-- consider using 'ParserK'. Do not use recursive parser implementations based
1194−-- on this Alternative instance.
1195−
1201+-- >>> empty = Parser.die "empty"
1202+-- >>> (<|>) = Parser.alt
1203+-- >>> many = flip Parser.many Fold.toList
1204+-- >>> some = flip Parser.some Fold.toList
11961205instance Monad m => Alternative (Parser a m) where
11971206 {-# INLINE empty #-}
11981207 empty = die "empty"
@@ -1211,8 +1220,13 @@ data ConcatParseState sl m a b =
12111220 ConcatParseL !sl
12121221 | forall s. ConcatParseR (s -> a -> m (Step s b)) s (s -> m (Step s b))
12131222
1223+-- XXX Does it fuse completely? Check and update, it cannot fuse the
1224+-- dynamically generated parser.
1225+
12141226-- | Map a 'Parser' returning function on the result of a 'Parser'.
12151227--
1228+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
1229+--
12161230-- /Pre-release/
12171231--
12181232{-# INLINE concatMap #-}
@@ -1282,6 +1296,12 @@ concatMap func (Parser stepL initialL extractL) = Parser step initial extract
12821296 Partial _ _ -> error "concatMap: extract Partial"
12831297 Continue n s -> return $ Continue n (ConcatParseL s)
12841298
1299+-- | Better performance 'concatMap' for non-failing parsers.
1300+--
1301+-- Does not work correctly for parsers that can fail.
1302+--
1303+-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
1304+--
12851305{-# INLINE noErrorUnsafeConcatMap #-}
12861306noErrorUnsafeConcatMap :: Monad m =>
12871307 (b -> Parser a m c) -> Parser a m b -> Parser a m c
@@ -1354,11 +1374,9 @@ noErrorUnsafeConcatMap func (Parser stepL initialL extractL) =
13541374-- for small number of compositions but for a scalable implementation we need a
13551375-- CPS version.
13561376
1357−-- | See documentation of 'Streamly.Internal.Data.Parser.ParserK.Type.Parser'.
1377+-- | READ THE CAVEATS in 'concatMap' before using this instance.
13581378--
1359−-- Although this implementation allows stream fusion, it has quadratic
1360−-- complexity, making it suitable only for a small number of compositions. As a
1361−-- thumb rule use it for less than 8 compositions, use 'ParserK' otherwise.
1379+-- >>> (>>=) = flip Parser.concatMap
13621380--
13631381instance Monad m => Monad (Parser a m) where
13641382 {-# INLINE return #-}
@@ -1370,6 +1388,7 @@ instance Monad m => Monad (Parser a m) where
13701388 {-# INLINE (>>) #-}
13711389 (>>) = (*>)
13721390
1391+-- | >>> fail = Parser.die
13731392instance Monad m => Fail.MonadFail (Parser a m) where
13741393 {-# INLINE fail #-}
13751394 fail = die
@@ -1385,6 +1404,7 @@ instance Monad m => MonadPlus (Parser a m) where
13851404 mplus = alt
13861405-}
13871406
1407+-- | >>> liftIO = Parser.fromEffect . liftIO
13881408instance (MonadIO m) => MonadIO (Parser a m) where
13891409 {-# INLINE liftIO #-}
13901410 liftIO = fromEffect . liftIO
core/src/Streamly/Internal/Data/ParserK/Type.hs+24−64
@@ -168,8 +168,8 @@ newtype ParserK a m b = MkParser
168168-------------------------------------------------------------------------------
169169
170170-- XXX rewrite this using ParserD, expose rmapM from ParserD.
171−-- | Maps a function over the output of the parser.
172−--
171+
172+-- | Map a function on the result i.e. on @b@ in @Parser a m b@.
173173instance Functor m => Functor (ParserK a m) where
174174 {-# INLINE fmap #-}
175175 fmap f parser = MkParser $ \k n st arr ->
@@ -181,7 +181,7 @@ instance Functor m => Functor (ParserK a m) where
181181-------------------------------------------------------------------------------
182182
183183-- This is the dual of stream "fromPure".
184−--
184+
185185-- | A parser that always yields a pure value without consuming any input.
186186--
187187-- /Pre-release/
@@ -199,9 +199,11 @@ fromEffect :: Monad m => m b -> ParserK a m b
199199fromEffect eff =
200200 MkParser $ \k n st arr -> eff >>= \b -> k (Success n b) st arr
201201
202−-- | 'Applicative' form of 'Streamly.Internal.Data.Parser.splitWith'. Note that
203−-- this operation does not fuse, use 'Streamly.Internal.Data.Parser.splitWith'
204−-- when fusion is important.
202+-- | @f \<$> p1 \<*> p2@ applies parsers p1 and p2 sequentially to an input
203+-- stream. The first parser runs and processes the input, the remaining input
204+-- is then passed to the second parser. If both parsers succeed, their outputs
205+-- are applied to the function @f@. If either parser fails, the operation
206+-- fails.
205207--
206208instance Monad m => Applicative (ParserK a m) where
207209 {-# INLINE pure #-}
@@ -243,48 +245,8 @@ instance Monad m => Applicative (ParserK a m) where
243245die :: String -> ParserK a m b
244246die err = MkParser (\k n st arr -> k (Failure n err) st arr)
245247
246−-- | Monad composition can be used for lookbehind parsers, we can make the
247−-- future parses depend on the previously parsed values.
248−--
249−-- If we have to parse "a9" or "9a" but not "99" or "aa" we can use the
250−-- following parser:
251−--
252−-- @
253−-- backtracking :: MonadCatch m => PR.Parser Char m String
254−-- backtracking =
255−-- sequence [PR.satisfy isDigit, PR.satisfy isAlpha]
256−-- '<|>'
257−-- sequence [PR.satisfy isAlpha, PR.satisfy isDigit]
258−-- @
259−--
260−-- We know that if the first parse resulted in a digit at the first place then
261−-- the second parse is going to fail. However, we waste that information and
262−-- parse the first character again in the second parse only to know that it is
263−-- not an alphabetic char. By using lookbehind in a 'Monad' composition we can
264−-- avoid redundant work:
265−--
266−-- @
267−-- data DigitOrAlpha = Digit Char | Alpha Char
268−--
269−-- lookbehind :: MonadCatch m => PR.Parser Char m String
270−-- lookbehind = do
271−-- x1 \<- Digit '<$>' PR.satisfy isDigit
272−-- '<|>' Alpha '<$>' PR.satisfy isAlpha
273−--
274−-- -- Note: the parse depends on what we parsed already
275−-- x2 <- case x1 of
276−-- Digit _ -> PR.satisfy isAlpha
277−-- Alpha _ -> PR.satisfy isDigit
278−--
279−-- return $ case x1 of
280−-- Digit x -> [x,x2]
281−-- Alpha x -> [x,x2]
282−-- @
283−--
284−-- See also 'Streamly.Internal.Data.Parser.concatMap'. This monad instance
285−-- does not fuse, use 'Streamly.Internal.Data.Parser.concatMap' when you need
286−-- fusion.
287−--
248+-- | Monad composition can be used for lookbehind parsers, we can dynamically
249+-- compose new parsers based on the results of the previously parsed values.
288250instance Monad m => Monad (ParserK a m) where
289251 {-# INLINE return #-}
290252 return = pure
@@ -316,18 +278,10 @@ instance MonadIO m => MonadIO (ParserK a m) where
316278-- Alternative
317279-------------------------------------------------------------------------------
318280
319−-- | 'Alternative' form of 'Streamly.Internal.Data.Parser.alt'. Backtrack and
320−-- run the second parser if the first one fails.
321−--
322−-- The "some" and "many" operations of alternative accumulate results in a pure
323−-- list which is not scalable and streaming. Instead use
324−-- 'Streamly.Internal.Data.Parser.some' and
325−-- 'Streamly.Internal.Data.Parser.many' for fusible operations with composable
326−-- accumulation of results.
327−--
328−-- See also 'Streamly.Internal.Data.Parser.alt'. This 'Alternative' instance
329−-- does not fuse, use 'Streamly.Internal.Data.Parser.alt' when you need
330−-- fusion.
281+-- | @p1 \<|> p2@ passes the input to parser p1, if it succeeds, the result is
282+-- returned. However, if p1 fails, the parser driver backtracks and tries the
283+-- same input on the alternative parser p2, returning the result if it
284+-- succeeds.
331285--
332286instance Monad m => Alternative (ParserK a m) where
333287 {-# INLINE empty #-}
@@ -450,7 +404,7 @@ adaptCWith pstep initial extract cont !offset0 !usedCount !input = do
450404 go !_ !cur !pst | cur >= end =
451405 onContinue ((end - start) `div` SIZE_OF(a)) pst
452406 go !_ !cur !pst = do
453− let !x = unsafeInlineIO $ peekByteIndex cur contents
407+ let !x = unsafeInlineIO $ peekAt cur contents
454408 pRes <- pstep pst x
455409 let elemSize = SIZE_OF(a)
456410 next = INDEX_NEXT(cur,a)
@@ -507,7 +461,8 @@ adaptCWith pstep initial extract cont !offset0 !usedCount !input = do
507461 parseCont !cnt !pst (Chunk arr) = parseContChunk cnt 0 pst arr
508462 parseCont !cnt !pst None = parseContNothing cnt pst
509463
510−-- | Convert a raw byte 'Parser' to a chunked 'ParserK'.
464+-- | Convert an element 'Parser' to a chunked 'ParserK'. A chunked parser is
465+-- more efficient than an element parser.
511466--
512467-- /Pre-release/
513468--
@@ -609,7 +564,9 @@ adaptWith pstep initial extract cont !relPos !usedCount !input = do
609564 parseCont !cnt !pst (Chunk arr) = parseContChunk cnt pst arr
610565 parseCont !cnt !pst None = parseContNothing cnt pst
611566
612−-- | Similar to "adaptC" but for non-chunked input.
567+-- | Convert a 'Parser' to 'ParserK'.
568+--
569+-- /Pre-release/
613570--
614571{-# INLINE_LATE adapt #-}
615572adapt :: Monad m => ParserD.Parser a m b -> ParserK a m b
@@ -737,7 +694,10 @@ adaptCGWith pstep initial extract cont !offset0 !usedCount !input = do
737694 parseCont !cnt !pst (Chunk arr) = parseContChunk cnt 0 pst arr
738695 parseCont !cnt !pst None = parseContNothing cnt pst
739696
740−-- | Similar to "adaptC" but is not constrained.
697+-- | A generic 'adaptC'. Similar to 'adaptC' but is not constrained to 'Unbox'
698+-- types.
699+--
700+-- /Pre-release/
741701--
742702{-# INLINE_LATE adaptCG #-}
743703adaptCG ::
core/src/Streamly/Internal/Data/Ring.hs+7−4
@@ -79,7 +79,7 @@ import Foreign.Storable
7979import Foreign.ForeignPtr (ForeignPtr, withForeignPtr, touchForeignPtr)
8080import Foreign.ForeignPtr.Unsafe (unsafeForeignPtrToPtr)
8181import Foreign.Ptr (plusPtr, minusPtr, castPtr)
82−import Streamly.Internal.Data.Unbox as Unboxed (Unbox(peekByteIndex))
82+import Streamly.Internal.Data.Unbox as Unboxed (Unbox(peekAt))
8383import GHC.ForeignPtr (mallocPlainForeignPtrAlignedBytes)
8484import GHC.Ptr (Ptr(..))
8585import Streamly.Internal.Data.MutArray.Type (MutArray)
@@ -411,7 +411,7 @@ unsafeEqArrayN Ring{..} rh A.Array{..} nBytes
411411
412412 check p i = do
413413 (relem :: Word8) <- peek p
414− aelem <- peekByteIndex i w8Contents
414+ aelem <- peekAt i w8Contents
415415 if relem == aelem
416416 then go (p `plusPtr` 1) (i + 1)
417417 else return False
@@ -445,7 +445,7 @@ unsafeEqArray Ring{..} rh A.Array{..} =
445445
446446 check p i = do
447447 (relem :: Word8) <- peek p
448− aelem <- peekByteIndex i w8Contents
448+ aelem <- peekAt i w8Contents
449449 if relem == aelem
450450 then go (p `plusPtr` 1) (i + 1)
451451 else return False
@@ -558,7 +558,8 @@ data Tuple4' a b c d = Tuple4' !a !b !c !d deriving Show
558558{-# INLINE slidingWindowWith #-}
559559slidingWindowWith :: forall m a b. (MonadIO m, Storable a, Unbox a)
560560 => Int -> Fold m ((a, Maybe a), m (MutArray a)) b -> Fold m a b
561−slidingWindowWith n (Fold step1 initial1 extract1) = Fold step initial extract
561+slidingWindowWith n (Fold step1 initial1 extract1 final1) =
562+ Fold step initial extract final
562563
563564 where
564565
@@ -601,6 +602,8 @@ slidingWindowWith n (Fold step1 initial1 extract1) = Fold step initial extract
601602
602603 extract (Tuple4' _ _ _ st) = extract1 st
603604
605+ final (Tuple4' _ _ _ st) = final1 st
606+
604607-- | @slidingWindow collector@ is an incremental sliding window
605608-- fold that does not require all the intermediate elements in a computation.
606609-- This maintains @n@ elements in the window, when a new element comes it slides
core/src/Streamly/Internal/Data/Ring/Generic.hs+1−1
@@ -69,7 +69,7 @@ createRing count = liftIO $ do
6969-- the Ring.
7070{-# INLINE writeLastN #-}
7171writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
72−writeLastN n = Fold step initial extract
72+writeLastN n = Fold step initial extract extract
7373
7474 where
7575
core/src/Streamly/Internal/Data/Serialize.hsdeleted−122
@@ -1,122 +0,0 @@
1−{-# LANGUAGE TemplateHaskell #-}
2−
3−-- This is required as all the instances in this module are orphan instances.
4−{-# OPTIONS_GHC -fno-warn-orphans #-}
5−
6−-- |
7−-- Module : Streamly.Internal.Data.Serialize
8−-- Copyright : (c) 2023 Composewell Technologies
9−-- License : BSD3-3-Clause
10−-- Maintainer : streamly@composewell.com
11−-- Portability : GHC
12−--
13−
14−module Streamly.Internal.Data.Serialize
15− ( module Streamly.Internal.Data.Serialize.Type
16− ) where
17−
18−--------------------------------------------------------------------------------
19−-- Imports
20−--------------------------------------------------------------------------------
21−
22−import Data.Proxy (Proxy)
23−import Streamly.Internal.Data.Array (Array(..))
24−import Streamly.Internal.Data.Unbox (MutableByteArray(..))
25−import GHC.Exts (Int(..), sizeofByteArray#, unsafeCoerce#)
26−
27−import qualified Streamly.Internal.Data.Serialize.TH as Serialize
28−
29−#if __GLASGOW_HASKELL__ >= 900
30−import GHC.Num.Integer (Integer(..))
31−#else
32−import GHC.Integer.GMP.Internals (Integer(..), BigNat(..))
33−#endif
34−
35−import Streamly.Internal.Data.Serialize.Type
36−
37−--------------------------------------------------------------------------------
38−-- Common instances
39−--------------------------------------------------------------------------------
40−
41−-- Note
42−-- ====
43−--
44−-- Even a non-functional change such as changing the order of constructors will
45−-- change the instance derivation.
46−--
47−-- This will not pose a problem if both, encode, and decode are done by the same
48−-- version of the application. There *might* be a problem if version that
49−-- encodes differs from the version that decodes.
50−--
51−-- We need to add some compatibility tests using different versions of
52−-- dependencies.
53−--
54−-- Although such chages for the most basic types won't happen we need to detect
55−-- if it ever happens.
56−--
57−-- Should we worry about these kind of changes and this kind of compatibility?
58−-- This is a problem for all types of derivations that depend on the order of
59−-- constructors, for example, Enum.
60−
61−$(Serialize.deriveSerialize ''Maybe)
62−$(Serialize.deriveSerialize ''Either)
63−$(Serialize.deriveSerializeWith
64− Serialize.defaultConfig
65− [d|instance Serialize (Proxy a)|])
66−
67−--------------------------------------------------------------------------------
68−-- Integer
69−--------------------------------------------------------------------------------
70−
71−data LiftedInteger
72− = LIS Int
73− | LIP (Array Word)
74− | LIN (Array Word)
75−
76−$(Serialize.deriveSerialize ''LiftedInteger)
77−
78−#if __GLASGOW_HASKELL__ >= 900
79−
80−{-# INLINE liftInteger #-}
81−liftInteger :: Integer -> LiftedInteger
82−liftInteger (IS x) = LIS (I# x)
83−liftInteger (IP x) =
84− LIP (Array (MutableByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
85−liftInteger (IN x) =
86− LIN (Array (MutableByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
87−
88−{-# INLINE unliftInteger #-}
89−unliftInteger :: LiftedInteger -> Integer
90−unliftInteger (LIS (I# x)) = IS x
91−unliftInteger (LIP (Array (MutableByteArray x) _ _)) = IP (unsafeCoerce# x)
92−unliftInteger (LIN (Array (MutableByteArray x) _ _)) = IN (unsafeCoerce# x)
93−
94−#else
95−
96−{-# INLINE liftInteger #-}
97−liftInteger :: Integer -> LiftedInteger
98−liftInteger (S# x) = LIS (I# x)
99−liftInteger (Jp# (BN# x)) =
100− LIP (Array (MutableByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
101−liftInteger (Jn# (BN# x)) =
102− LIN (Array (MutableByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
103−
104−{-# INLINE unliftInteger #-}
105−unliftInteger :: LiftedInteger -> Integer
106−unliftInteger (LIS (I# x)) = S# x
107−unliftInteger (LIP (Array (MutableByteArray x) _ _)) =
108− Jp# (BN# (unsafeCoerce# x))
109−unliftInteger (LIN (Array (MutableByteArray x) _ _)) =
110− Jn# (BN# (unsafeCoerce# x))
111−
112−#endif
113−
114−instance Serialize Integer where
115− {-# INLINE size #-}
116− size i a = size i (liftInteger a)
117−
118− {-# INLINE deserialize #-}
119− deserialize off arr end = fmap unliftInteger <$> deserialize off arr end
120−
121− {-# INLINE serialize #-}
122− serialize off arr val = serialize off arr (liftInteger val)
core/src/Streamly/Internal/Data/Serialize/TH.hs+113−96
@@ -7,12 +7,23 @@
77-- Maintainer : streamly@composewell.com
88-- Stability : experimental
99-- Portability : GHC
10−--
10+
11+-- XXX Instead of applying the product constructor in one go can we apply it
12+-- one at a time in case of too many arguments, compiler may not have to save
13+-- them in local vars.
14+
1115module Streamly.Internal.Data.Serialize.TH
12− ( deriveSerialize
13− , Config(..)
14− , defaultConfig
16+ (
17+ -- Deriving
18+ deriveSerialize
1519 , deriveSerializeWith
20+
21+ -- Utilities
22+ , module Streamly.Internal.Data.Serialize.TH.Bottom
23+ -- ** Common
24+ , module Streamly.Internal.Data.Serialize.TH.Common
25+ -- ** RecHeader
26+ , module Streamly.Internal.Data.Serialize.TH.RecHeader
1627 ) where
1728
1829--------------------------------------------------------------------------------
@@ -29,7 +40,6 @@ import Streamly.Internal.Data.Serialize.Type
2940import Streamly.Internal.Data.Unbox.TH
3041 ( DataCon(..)
3142 , DataType(..)
32− , appsT
3343 , reifyDataType
3444 )
3545
@@ -37,13 +47,14 @@ import qualified Streamly.Internal.Data.Serialize.TH.RecHeader as RecHeader
3747
3848import Streamly.Internal.Data.Serialize.TH.Bottom
3949import Streamly.Internal.Data.Serialize.TH.Common
50+import Streamly.Internal.Data.Serialize.TH.RecHeader
4051
4152--------------------------------------------------------------------------------
4253-- Domain specific helpers
4354--------------------------------------------------------------------------------
4455
4556exprGetSize :: Q Exp -> (Int, Type) -> Q Exp
46−exprGetSize acc (i, _) = [|size $(acc) $(varE (mkFieldName i))|]
57+exprGetSize acc (i, _) = [|addSizeTo $(acc) $(varE (mkFieldName i))|]
4758
4859getTagSize :: Int -> Int
4960getTagSize numConstructors
@@ -148,22 +159,25 @@ mkSizeOfExpr False False tyOfTy =
148159 [varP _acc, varP _x]
149160 (caseE (varE _x) (fmap (matchCons acc) cons))
150161
151−mkSizeOfExpr False True (TheType con) = RecHeader.mkSizeOfExpr con
162+mkSizeOfExpr False True (TheType con) = RecHeader.mkRecSizeOfExpr con
152163
153164mkSizeOfExpr _ _ _ = errorUnimplemented
154165
155−mkSizeDec :: Config -> Type -> [DataCon] -> Q [Dec]
156−mkSizeDec (Config {..}) headTy cons = do
166+mkSizeDec :: SerializeConfig -> Type -> [DataCon] -> Q [Dec]
167+mkSizeDec (SerializeConfig {..}) headTy cons = do
157168 -- INLINE on sizeOf actually worsens some benchmarks, and improves none
158169 sizeOfMethod <-
159170 mkSizeOfExpr
160− constructorTagAsString
161− recordSyntaxWithHeader
171+ cfgConstructorTagAsString
172+ cfgRecordSyntaxWithHeader
162173 (typeOfType headTy cons)
163174 pure
164− [ PragmaD (InlineP 'size inlineSize FunLike AllPhases)
165− , FunD 'size [Clause [] (NormalB sizeOfMethod) []]
166− ]
175+ ( maybe
176+ []
177+ (\x -> [PragmaD (InlineP 'addSizeTo x FunLike AllPhases)])
178+ cfgInlineSize
179+ ++ [FunD 'addSizeTo [Clause [] (NormalB sizeOfMethod) []]]
180+ )
167181
168182--------------------------------------------------------------------------------
169183-- Peek
@@ -182,7 +196,7 @@ mkDeserializeExpr True False headTy tyOfTy =
182196 conLen <- newName "conLen"
183197 off1 <- newName "off1"
184198 [|do ($(varP off1), $(varP conLen) :: Word8) <-
185− deserialize
199+ deserializeAt
186200 $(varE _initialOffset)
187201 $(varE _arr)
188202 $(varE _endOffset)
@@ -202,18 +216,19 @@ mkDeserializeExpr True False headTy tyOfTy =
202216 [|($(litIntegral lenCname) == $(varE conLen))
203217 && $(xorCmp tag off _arr)|]
204218 [|let $(varP (makeI 0)) = $(varE off) + $(litIntegral lenCname)
205− in $(mkDeserializeExprOne 'deserialize con)|]
219+ in $(mkDeserializeExprOne 'deserializeAt con)|]
206220
207221mkDeserializeExpr False False headTy tyOfTy =
208222 case tyOfTy of
209223 -- Unit constructor
224+ -- XXX Should we peek and check if the byte value is 0?
210225 UnitType cname ->
211226 [|pure ($(varE _initialOffset) + 1, $(conE cname))|]
212227 -- Product type
213228 TheType con ->
214229 letE
215230 [valD (varP (mkName "i0")) (normalB (varE _initialOffset)) []]
216− (mkDeserializeExprOne 'deserialize con)
231+ (mkDeserializeExprOne 'deserializeAt con)
217232 -- Sum type
218233 MultiType cons -> do
219234 let lenCons = length cons
@@ -221,7 +236,7 @@ mkDeserializeExpr False False headTy tyOfTy =
221236 doE
222237 [ bindS
223238 (tupP [varP (mkName "i0"), varP _tag])
224− [|deserialize $(varE _initialOffset) $(varE _arr) $(varE _endOffset)|]
239+ [|deserializeAt $(varE _initialOffset) $(varE _arr) $(varE _endOffset)|]
225240 , noBindS
226241 (caseE
227242 (sigE (varE _tag) (conT tagType))
@@ -231,7 +246,7 @@ mkDeserializeExpr False False headTy tyOfTy =
231246 peekMatch (i, con) =
232247 match
233248 (litP (IntegerL i))
234− (normalB (mkDeserializeExprOne 'deserialize con)) []
249+ (normalB (mkDeserializeExprOne 'deserializeAt con)) []
235250 peekErr =
236251 match
237252 wildP
@@ -250,7 +265,7 @@ mkDeserializeExpr False True _ (TheType con@(SimpleDataCon _ fields)) = do
250265 RecHeader.mkDeserializeKeysDec deserializeWithKeys updateFunc con
251266 letE
252267 (pure <$> (deserializeWithKeysDec ++ updateFuncDec))
253− (RecHeader.mkDeserializeExpr
268+ (RecHeader.mkRecDeserializeExpr
254269 _initialOffset
255270 _endOffset
256271 deserializeWithKeys
@@ -258,26 +273,31 @@ mkDeserializeExpr False True _ (TheType con@(SimpleDataCon _ fields)) = do
258273
259274mkDeserializeExpr _ _ _ _ = errorUnimplemented
260275
261−mkDeserializeDec :: Config -> Type -> [DataCon] -> Q [Dec]
262−mkDeserializeDec (Config {..}) headTy cons = do
276+mkDeserializeDec :: SerializeConfig -> Type -> [DataCon] -> Q [Dec]
277+mkDeserializeDec (SerializeConfig {..}) headTy cons = do
263278 peekMethod <-
264279 mkDeserializeExpr
265− constructorTagAsString
266− recordSyntaxWithHeader
280+ cfgConstructorTagAsString
281+ cfgRecordSyntaxWithHeader
267282 headTy
268283 (typeOfType headTy cons)
269284 pure
270− [ PragmaD (InlineP 'deserialize inlineDeserialize FunLike AllPhases)
271− , FunD
272− 'deserialize
285+ ( maybe
286+ []
287+ (\x -> [PragmaD (InlineP 'deserializeAt x FunLike AllPhases)])
288+ cfgInlineDeserialize
289+ ++
290+ [ FunD
291+ 'deserializeAt
273292 [ Clause
274− (if isUnitType cons && not constructorTagAsString
293+ (if isUnitType cons && not cfgConstructorTagAsString
275294 then [VarP _initialOffset, WildP, WildP]
276295 else [VarP _initialOffset, VarP _arr, VarP _endOffset])
277296 (NormalB peekMethod)
278297 []
279298 ]
280− ]
299+ ]
300+ )
281301
282302--------------------------------------------------------------------------------
283303-- Poke
@@ -285,7 +305,7 @@ mkDeserializeDec (Config {..}) headTy cons = do
285305
286306mkSerializeExprTag :: Name -> Int -> Q Exp
287307mkSerializeExprTag tagType tagVal =
288− [|serialize
308+ [|serializeAt
289309 $(varE _initialOffset)
290310 $(varE _arr)
291311 $((sigE (litE (IntegerL (fromIntegral tagVal))) (conT tagType)))|]
@@ -320,13 +340,14 @@ mkSerializeExpr True False tyOfTy =
320340 (doE [ bindS
321341 (varP (mkName "i0"))
322342 (serializeW8List _initialOffset _arr conEnc)
323− , noBindS (mkSerializeExprFields 'serialize fields)
343+ , noBindS (mkSerializeExprFields 'serializeAt fields)
324344 ])
325345
326346mkSerializeExpr False False tyOfTy =
327347 case tyOfTy of
328348 -- Unit type
329− UnitType _ -> [|pure ($(varE _initialOffset) + 1)|]
349+ UnitType _ ->
350+ [|serializeAt $(varE _initialOffset) $(varE _arr) (0 :: Word8)|]
330351 -- Product type
331352 (TheType (SimpleDataCon cname fields)) ->
332353 letE
@@ -336,7 +357,7 @@ mkSerializeExpr False False tyOfTy =
336357 [ matchConstructor
337358 cname
338359 (length fields)
339− (mkSerializeExprFields 'serialize fields)
360+ (mkSerializeExprFields 'serializeAt fields)
340361 ])
341362 -- Sum type
342363 (MultiType cons) -> do
@@ -353,35 +374,40 @@ mkSerializeExpr False False tyOfTy =
353374 (mkSerializeExprTag tagType tagVal)
354375 , noBindS
355376 (mkSerializeExprFields
356− 'serialize
377+ 'serializeAt
357378 fields)
358379 ]))
359380 (zip [0 ..] cons))
360381
361382mkSerializeExpr False True (TheType con) =
362− RecHeader.mkSerializeExpr _initialOffset con
383+ RecHeader.mkRecSerializeExpr _initialOffset con
363384
364385mkSerializeExpr _ _ _ = errorUnimplemented
365386
366−mkSerializeDec :: Config -> Type -> [DataCon] -> Q [Dec]
367−mkSerializeDec (Config {..}) headTy cons = do
387+mkSerializeDec :: SerializeConfig -> Type -> [DataCon] -> Q [Dec]
388+mkSerializeDec (SerializeConfig {..}) headTy cons = do
368389 pokeMethod <-
369390 mkSerializeExpr
370− constructorTagAsString
371− recordSyntaxWithHeader
391+ cfgConstructorTagAsString
392+ cfgRecordSyntaxWithHeader
372393 (typeOfType headTy cons)
373394 pure
374− [ PragmaD (InlineP 'serialize inlineSerialize FunLike AllPhases)
375− , FunD
376− 'serialize
377− [ Clause
378− (if isUnitType cons && not constructorTagAsString
379− then [VarP _initialOffset, WildP, WildP]
380− else [VarP _initialOffset, VarP _arr, VarP _val])
381− (NormalB pokeMethod)
382− []
383− ]
384− ]
395+ ( maybe
396+ []
397+ (\x -> [PragmaD (InlineP 'serializeAt x FunLike AllPhases)])
398+ cfgInlineSerialize
399+ ++
400+ [FunD
401+ 'serializeAt
402+ [ Clause
403+ (if isUnitType cons && not cfgConstructorTagAsString
404+ then [VarP _initialOffset, VarP _arr, WildP]
405+ else [VarP _initialOffset, VarP _arr, VarP _val])
406+ (NormalB pokeMethod)
407+ []
408+ ]
409+ ]
410+ )
385411
386412--------------------------------------------------------------------------------
387413-- Main
@@ -403,7 +429,7 @@ mkSerializeDec (Config {..}) headTy cons = do
403429-- Usage:
404430-- @
405431-- $(deriveSerializeInternal
406−-- defaultConfig
432+-- serializeConfig
407433-- [AppT (ConT ''Serialize) (VarT (mkName "b"))]
408434-- (AppT
409435-- (AppT (ConT ''CustomDataType) (VarT (mkName "a")))
@@ -418,7 +444,7 @@ mkSerializeDec (Config {..}) headTy cons = do
418444-- ])
419445-- @
420446deriveSerializeInternal ::
421− Config -> Type -> [DataCon] -> ([Dec] -> Q [Dec]) -> Q [Dec]
447+ SerializeConfig -> Type -> [DataCon] -> ([Dec] -> Q [Dec]) -> Q [Dec]
422448deriveSerializeInternal conf headTy cons next = do
423449 sizeDec <- mkSizeDec conf headTy cons
424450 peekDec <- mkDeserializeDec conf headTy cons
@@ -426,30 +452,41 @@ deriveSerializeInternal conf headTy cons next = do
426452 let methods = concat [sizeDec, peekDec, pokeDec]
427453 next methods
428454
429−-- | Similar to 'deriveSerialize' but take a 'Config' to control how the
430−-- instance is generated.
455+-- | @deriveSerializeWith config-modifier instance-dec@ generates a template
456+-- Haskell splice consisting of a declaration of a 'Serialize' instance.
457+-- @instance-dec@ is a template Haskell declaration splice consisting of a
458+-- standard Haskell instance declaration without the type class methods (e.g.
459+-- @[d|instance Serialize a => Serialize (Maybe a)|]@).
460+--
461+-- The type class methods for the given instance are generated according to the
462+-- supplied @config-modifier@ parameter. See 'SerializeConfig' for default
463+-- configuration settings.
431464--
432465-- Usage:
466+--
433467-- @
434−-- $(deriveSerializeWith
435−-- config
468+-- \$(deriveSerializeWith
469+-- ( inlineSerializeAt (Just NoInline)
470+-- . inlineDeserializeAt (Just NoInline)
471+-- )
436472-- [d|instance Serialize a => Serialize (Maybe a)|])
437473-- @
438−deriveSerializeWith :: Config -> Q [Dec] -> Q [Dec]
439−deriveSerializeWith conf mDecs = do
474+deriveSerializeWith ::
475+ (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
476+deriveSerializeWith modifier mDecs = do
440477 dec <- mDecs
441478 case dec of
442− [inst@(InstanceD _ _ headTyWC [])] -> do
479+ [InstanceD mo preds headTyWC []] -> do
443480 let headTy = unwrap dec headTyWC
444481 dt <- reifyDataType (getMainTypeName dec headTy)
445482 let cons = dtCons dt
446− deriveSerializeInternal conf headTy cons (next inst)
447− _ -> errorUnsupported (errorMessage dec)
448− where
483+ deriveSerializeInternal
484+ (modifier serializeConfig) headTy cons (next mo preds headTyWC)
485+ _ -> errorMessage dec
486+
487+ where
449488
450− next (InstanceD mo preds headTyWC []) methods =
451− pure [InstanceD mo preds headTyWC methods]
452− next dec _ = errorUnsupported (errorMessage dec)
489+ next mo preds headTyWC methods = pure [InstanceD mo preds headTyWC methods]
453490
454491 errorMessage dec =
455492 error $ unlines
@@ -457,8 +494,8 @@ deriveSerializeWith conf mDecs = do
457494 , ""
458495 , ">> " ++ pprint dec
459496 , ""
460− , "The above is not a valid instance declaration."
461− , "Any haskell instance declaration without a body is valid."
497+ , "The supplied declaration is not a valid instance declaration."
498+ , "Provide a valid Haskell instance declaration without a body."
462499 , ""
463500 , "Examples:"
464501 , "instance Serialize (Proxy a)"
@@ -477,37 +514,17 @@ deriveSerializeWith conf mDecs = do
477514 go (AppT l _) = go l
478515 go _ = errorMessage dec
479516
480−
481−-- | Template haskell helper to create instances of 'Serialize' automatically.
517+-- | Given an 'Serialize' instance declaration splice without the methods (e.g.
518+-- @[d|instance Serialize a => Serialize (Maybe a)|]@), generate an instance
519+-- declaration including all the type class method implementations.
482520--
483−-- Consider the datatype:
484−-- @
485−-- data CustomDataType a b c = ...
486−-- @
521+-- >>> deriveSerialize = deriveSerializeWith id
487522--
488−-- Usage: @$(deriveSerialize ''CustomDataType)@
523+-- Usage:
489524--
490−-- Note: All type variables automatcally get an "Serialize" constraint.
491−-- The derived code will look like the following,
492525-- @
493−-- instance (Serialize a, Serialize b, Serialize c) => Serialize (CustomDataType a b c) where
494−-- ...
526+-- \$(deriveSerialize
527+-- [d|instance Serialize a => Serialize (Maybe a)|])
495528-- @
496−--
497−-- To control which type variables don't get the Serialize constraint, use
498−-- 'deriveSerializeWith'.
499−deriveSerialize :: Name -> Q [Dec]
500−deriveSerialize name = do
501− dt <- reifyDataType name
502− let preds = map (unboxPred . VarT) (dtTvs dt)
503− headTy = appsT (ConT name) (map VarT (dtTvs dt))
504− cons = dtCons dt
505− deriveSerializeInternal defaultConfig headTy cons (next preds headTy)
506−
507−
508− where
509−
510− next preds headTy methods =
511− pure [InstanceD Nothing preds (AppT (ConT ''Serialize) headTy) methods]
512−
513− unboxPred ty = AppT (ConT ''Serialize) ty
529+deriveSerialize :: Q [Dec] -> Q [Dec]
530+deriveSerialize = deriveSerializeWith id
core/src/Streamly/Internal/Data/Serialize/TH/Bottom.hs+143−48
@@ -9,8 +9,17 @@
99-- Portability : GHC
1010--
1111module Streamly.Internal.Data.Serialize.TH.Bottom
12− ( Config(..)
13− , defaultConfig
12+ (
13+ -- ** Config
14+ SerializeConfig(..)
15+ , serializeConfig
16+ , inlineAddSizeTo
17+ , inlineSerializeAt
18+ , inlineDeserializeAt
19+ , encodeConstrNames
20+ , encodeRecordFields
21+
22+ -- ** Other Utilities
1423 , TypeOfType(..)
1524 , typeOfType
1625 , SimpleDataCon(..)
@@ -65,48 +74,134 @@ import Streamly.Internal.Data.Unbox.TH (DataCon(..))
6574-- Config
6675--------------------------------------------------------------------------------
6776
68−-- | Config to control how the 'Serialize' instance is generated.
69−data Config =
70− Config
71− { inlineSize :: Inline
72− -- ^ Inline value for 'size'. Default is Inline.
73− , inlineSerialize :: Inline
74− -- ^ Inline value for 'serialize'. Default is Inline.
75− , inlineDeserialize :: Inline
76− -- ^ Inline value for 'deserialize'. Default is Inline.
77− , constructorTagAsString :: Bool
78− -- ^ __Experimental__
79− --
80− -- If True, encode constructors using the constructor names as Latin-1
81− -- byte sequence.
82− , recordSyntaxWithHeader :: Bool
83− -- ^ __Experimental__
84− --
85− -- If True, encode the keys of the record as a header and then
86− -- serialize the data. Multiple constructors are not supported with
87− -- @recordSyntaxWithHeader@ enabled.
88− --
89− -- __Performance Notes:__
90− --
91− -- There is a constant regression proportional to
92− -- @sum (map length keyList) + length keyList@ where @keyList@ is the
93− -- list of keys of that record as strings.
94− --
95− -- As an example, @keyList@ of,
96− -- @
97− -- data RecordType = RecordType { field0 :: Int, field2 :: Char }
98− -- @
99− -- is @["field0", "field1"]@
77+-- NOTE: 'Nothing' is not eqvivalant to 'Just Inlinable'. Ie. Having no inline
78+-- specific pragma and having an Inlinable pragma are different. Having an
79+-- Inlinable pragma makes GHC put the code in the interface file whereas having
80+-- no inline specific pragma let's GHC decide whether to put the code in
81+-- interface file or not.
82+
83+-- | Configuration to control how the 'Serialize' instance is generated. The
84+-- configuration is opaque and is modified by composing config modifier
85+-- functions, for example:
86+--
87+-- >>> (inlineSerializeAt (Just NoInline)) . (inlineSerializeAt (Just Inlinable))
88+--
89+-- The default configuration settings are:
90+--
91+-- * 'inlineAddSizeTo' Nothing
92+-- * 'inlineSerializeAt' (Just Inline)
93+-- * 'inlineDeserializeAt' (Just Inline)
94+--
95+-- The following experimental options are also available:
96+--
97+-- * 'encodeConstrNames' False
98+-- * 'encodeRecordFields' False
99+--
100+data SerializeConfig =
101+ SerializeConfig
102+ { cfgInlineSize :: Maybe Inline
103+ , cfgInlineSerialize :: Maybe Inline
104+ , cfgInlineDeserialize :: Maybe Inline
105+ , cfgConstructorTagAsString :: Bool
106+ , cfgRecordSyntaxWithHeader :: Bool
100107 }
101108
102−defaultConfig :: Config
103−defaultConfig =
104− Config
105− { inlineSize = Inline
106− , inlineSerialize = Inline
107− , inlineDeserialize = Inline
108− , constructorTagAsString = False
109− , recordSyntaxWithHeader = False
109+-- | How should we inline the 'addSizeTo' function? The default is 'Nothing'
110+-- which means left to the compiler. Forcing inline on @addSizeTo@ function
111+-- actually worsens some benchmarks and improves none.
112+inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
113+inlineAddSizeTo v cfg = cfg {cfgInlineSize = v}
114+
115+-- XXX Should we make the default Inlinable instead?
116+
117+-- | How should we inline the 'serialize' function? The default 'Just Inline'.
118+-- However, aggressive inlining can bloat the code and increase in compilation
119+-- times when there are big functions and too many nesting levels so you can
120+-- change it accordingly. A 'Nothing' value leaves the decision to the
121+-- compiler.
122+inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
123+inlineSerializeAt v cfg = cfg {cfgInlineSerialize = v}
124+
125+-- XXX Should we make the default Inlinable instead?
126+
127+-- | How should we inline the 'deserialize' function? See guidelines in
128+-- 'inlineSerializeAt'.
129+inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
130+inlineDeserializeAt v cfg = cfg {cfgInlineDeserialize = v}
131+
132+-- | __Experimental__
133+--
134+-- In sum types, use Latin-1 encoded original constructor names rather than
135+-- binary values to identify constructors. This option is not applicable to
136+-- product types.
137+--
138+-- This option enables the following behavior:
139+--
140+-- * __Reordering__: Order of the fields can be changed without affecting
141+-- serialization.
142+-- * __Addition__: If a field is added in the new version, the old version of
143+-- the data type can still be deserialized by the new version. The new value
144+-- would never occur in the old one.
145+-- * __Deletion__: If a field is deleted in the new version, deserialization
146+-- of the old version will result in an error. TBD: We can possibly designate a
147+-- catch-all case to handle this scenario.
148+--
149+-- Note that if you change a type, change the semantics of a type, or delete a
150+-- field and add a new field with the same name, deserialization of old data
151+-- may result in silent unexpected behavior.
152+--
153+-- This option has to be the same on both encoding and decoding side.
154+--
155+-- The default is 'False'.
156+--
157+encodeConstrNames :: Bool -> SerializeConfig -> SerializeConfig
158+encodeConstrNames v cfg = cfg {cfgConstructorTagAsString = v}
159+
160+-- XXX We can deserialize each field to Either, so if there is a
161+-- deserialization error in any field it can handled independently. Also, a
162+-- unique type/version identifier of the field (based on the versions of the
163+-- packages, full module name space + type identifier) can be serialized along
164+-- with the value for stricter compatibility, semantics checking. Or we can
165+-- store a type hash.
166+
167+-- | __Experimental__
168+--
169+-- In explicit record types, use Latin-1 encoded record field names rather than
170+-- binary values to identify the record fields. Note that this option is not
171+-- applicable to sum types. Also, it does not work on a product type which is
172+-- not a record, because there are no field names to begin with.
173+--
174+-- This option enables the following behavior:
175+--
176+-- * __Reordering__: Order of the fields can be changed without affecting
177+-- serialization.
178+-- * __Addition__: If a 'Maybe' type field is added in the new version, the old
179+-- version of the data type can still be deserialized by the new version, the
180+-- field value in the older version is assumed to be 'Nothing'. If any other
181+-- type of field is added, deserialization of the older version results in an
182+-- error but only when that field is actually accessed in the deserialized
183+-- record.
184+-- * __Deletion__: If a field is deleted in the new version and it is
185+-- encountered in a previously serialized version then the field is discarded.
186+--
187+-- This option has to be the same on both encoding and decoding side.
188+--
189+-- There is a constant performance overhead proportional to the total length of
190+-- the record field names and the number of record fields.
191+--
192+-- The default is 'False'.
193+--
194+encodeRecordFields :: Bool -> SerializeConfig -> SerializeConfig
195+encodeRecordFields v cfg = cfg {cfgRecordSyntaxWithHeader = v}
196+
197+serializeConfig :: SerializeConfig
198+serializeConfig =
199+ SerializeConfig
200+ { cfgInlineSize = Nothing
201+ , cfgInlineSerialize = Just Inline
202+ , cfgInlineDeserialize = Just Inline
203+ , cfgConstructorTagAsString = False
204+ , cfgRecordSyntaxWithHeader = False
110205 }
111206
112207--------------------------------------------------------------------------------
@@ -252,7 +347,7 @@ xorCmp tag off arr =
252347 go8 i = do
253348 let wIntegral = litIntegral i
254349 [|xor (unsafeInlineIO
255− (Unbox.peekByteIndex
350+ (Unbox.peekAt
256351 ($(varE off) + $(litIntegral i))
257352 $(varE arr)))
258353 ($(wIntegral) :: Word8) .|.
@@ -266,7 +361,7 @@ xorCmp tag off arr =
266361 litIntegral
267362 (shiftAdd w8_w16 [tag !! i, tag !! (i + 1)] :: Word16)
268363 [|xor (unsafeInlineIO
269− (Unbox.peekByteIndex
364+ (Unbox.peekAt
270365 ($(varE off) + $(litIntegral i))
271366 $(varE arr)))
272367 ($(wIntegral) :: Word16) .|.
@@ -286,7 +381,7 @@ xorCmp tag off arr =
286381 , tag !! (i + 3)
287382 ] :: Word32)
288383 [|xor (unsafeInlineIO
289− (Unbox.peekByteIndex
384+ (Unbox.peekAt
290385 ($(varE off) + $(litIntegral i))
291386 $(varE arr)))
292387 ($(wIntegral) :: Word32) .|.
@@ -310,7 +405,7 @@ xorCmp tag off arr =
310405 , tag !! (i + 7)
311406 ])
312407 [|xor (unsafeInlineIO
313− (Unbox.peekByteIndex
408+ (Unbox.peekAt
314409 ($(varE off) + $(litIntegral i))
315410 $(varE arr)))
316411 ($(wIntegral) :: Word64) .|.
@@ -336,7 +431,7 @@ serializeW8List off arr w8List = do
336431 makeBind i =
337432 bindS
338433 (varP (makeN (i + 1)))
339− [|$(varE 'serialize)
434+ [|$(varE 'serializeAt)
340435 $(varE (makeN i))
341436 $(varE arr)
342437 ($(litIntegral (w8List !! i)) :: Word8)|]
@@ -361,7 +456,7 @@ errorUnsupported err =
361456 $ unlines
362457 [ "Unsupported"
363458 , "==========="
364− , "There is improper use of the library."
459+ , "This is improper use of the library."
365460 , "This case is unsupported."
366461 , "Please contact the developer if this case is of interest."
367462 , ""
core/src/Streamly/Internal/Data/Serialize/TH/RecHeader.hs+37−37
@@ -9,9 +9,9 @@
99-- Portability : GHC
1010--
1111module Streamly.Internal.Data.Serialize.TH.RecHeader
12− ( mkSerializeExpr
13− , mkDeserializeExpr
14− , mkSizeOfExpr
12+ ( mkRecSerializeExpr
13+ , mkRecDeserializeExpr
14+ , mkRecSizeOfExpr
1515 , conUpdateFuncDec
1616 , mkDeserializeKeysDec
1717 ) where
@@ -27,7 +27,7 @@ import Data.Maybe (fromJust)
2727import Language.Haskell.TH
2828import Streamly.Internal.Data.Serialize.Type (Serialize(..))
2929import Data.Foldable (foldlM)
30−import Streamly.Internal.Data.Unbox (MutableByteArray)
30+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
3131import Data.Proxy (Proxy(..))
3232
3333import qualified Streamly.Internal.Data.Unbox as Unbox
@@ -103,36 +103,36 @@ newtype CompactList a =
103103-- the list is 255.
104104instance forall a. Serialize a => Serialize (CompactList a) where
105105
106− -- {-# INLINE size #-}
107− size acc (CompactList xs) =
108− foldl' size (acc + (Unbox.sizeOf (Proxy :: Proxy Word8))) xs
106+ -- {-# INLINE addSizeTo #-}
107+ addSizeTo acc (CompactList xs) =
108+ foldl' addSizeTo (acc + (Unbox.sizeOf (Proxy :: Proxy Word8))) xs
109109
110110 -- Inlining this causes large compilation times for tests
111− {-# INLINABLE deserialize #-}
112− deserialize off arr sz = do
113− (off1, len8) <- deserialize off arr sz :: IO (Int, Word8)
111+ {-# INLINABLE deserializeAt #-}
112+ deserializeAt off arr sz = do
113+ (off1, len8) <- deserializeAt off arr sz :: IO (Int, Word8)
114114 let len = w8_int len8
115115 peekList f o i | i >= 3 = do
116116 -- Unfold the loop three times
117− (o1, x1) <- deserialize o arr sz
118− (o2, x2) <- deserialize o1 arr sz
119− (o3, x3) <- deserialize o2 arr sz
117+ (o1, x1) <- deserializeAt o arr sz
118+ (o2, x2) <- deserializeAt o1 arr sz
119+ (o3, x3) <- deserializeAt o2 arr sz
120120 peekList (f . (\xs -> x1:x2:x3:xs)) o3 (i - 3)
121121 peekList f o 0 = pure (o, f [])
122122 peekList f o i = do
123− (o1, x) <- deserialize o arr sz
123+ (o1, x) <- deserializeAt o arr sz
124124 peekList (f . (x:)) o1 (i - 1)
125125 (nextOff, lst) <- peekList id off1 len
126126 pure (nextOff, CompactList lst)
127127
128128 -- Inlining this causes large compilation times for tests
129− {-# INLINABLE serialize #-}
130− serialize off arr (CompactList val) = do
131− void $ serialize off arr (int_w8 (length val) :: Word8)
129+ {-# INLINABLE serializeAt #-}
130+ serializeAt off arr (CompactList val) = do
131+ void $ serializeAt off arr (int_w8 (length val) :: Word8)
132132 let off1 = off + Unbox.sizeOf (Proxy :: Proxy Word8)
133133 let pokeList o [] = pure o
134134 pokeList o (x:xs) = do
135− o1 <- serialize o arr x
135+ o1 <- serializeAt o arr x
136136 pokeList o1 xs
137137 pokeList off1 val
138138
@@ -154,7 +154,7 @@ isMaybeType _ = False
154154-- We add 4 here because we use 'serializeWithSize' for serializing.
155155exprGetSize :: Q Exp -> (Int, Type) -> Q Exp
156156exprGetSize acc (i, _) =
157− [|size $(acc) $(varE (mkFieldName i)) + 4|]
157+ [|addSizeTo $(acc) $(varE (mkFieldName i)) + 4|]
158158
159159sizeOfHeader :: SimpleDataCon -> Int
160160sizeOfHeader (SimpleDataCon _ fields) =
@@ -171,8 +171,8 @@ sizeOfHeader (SimpleDataCon _ fields) =
171171 sizeForNumFields = 1 -- At max 255 fields in the record constructor
172172 sizeForFieldLen = 1 -- At max 255 letters in the key
173173
174−mkSizeOfExpr :: SimpleDataCon -> Q Exp
175−mkSizeOfExpr con = do
174+mkRecSizeOfExpr :: SimpleDataCon -> Q Exp
175+mkRecSizeOfExpr con = do
176176 n_acc <- newName "acc"
177177 n_x <- newName "x"
178178 (lamE
@@ -225,27 +225,27 @@ headerValue (SimpleDataCon _ fields) =
225225-- its type. We encode the size as 'Word32' hence there is a 4 bytes increase
226226-- in size.
227227{-# INLINE serializeWithSize #-}
228−serializeWithSize :: Serialize a => Int -> MutableByteArray -> a -> IO Int
228+serializeWithSize :: Serialize a => Int -> MutByteArray -> a -> IO Int
229229serializeWithSize off arr val = do
230− off1 <- serialize (off + 4) arr val
231− Unbox.pokeByteIndex off arr (int_w32 (off1 - off - 4) :: Word32)
230+ off1 <- serializeAt (off + 4) arr val
231+ Unbox.pokeAt off arr (int_w32 (off1 - off - 4) :: Word32)
232232 pure off1
233233
234−mkSerializeExpr :: Name -> SimpleDataCon -> Q Exp
235−mkSerializeExpr initialOffset (con@(SimpleDataCon cname fields)) = do
234+mkRecSerializeExpr :: Name -> SimpleDataCon -> Q Exp
235+mkRecSerializeExpr initialOffset (con@(SimpleDataCon cname fields)) = do
236236 afterHLen <- newName "afterHLen"
237237 -- Encoding the header length is required.
238238 -- We first compare the header length encoded and the current header
239239 -- length. Only if the header lengths match, we compare the headers.
240240 [|do $(varP afterHLen) <-
241− serialize
241+ serializeAt
242242 ($(varE initialOffset) + 4)
243243 $(varE _arr)
244244 ($(litIntegral hlen) :: Word32)
245245 $(varP (makeI 0)) <- $(serializeW8List afterHLen _arr hval)
246246 let $(openConstructor cname (length fields)) = $(varE _val)
247247 finalOff <- $(mkSerializeExprFields 'serializeWithSize fields)
248− Unbox.pokeByteIndex
248+ Unbox.pokeAt
249249 $(varE initialOffset)
250250 $(varE _arr)
251251 ((fromIntegral :: Int -> Word32)
@@ -263,8 +263,8 @@ mkSerializeExpr initialOffset (con@(SimpleDataCon cname fields)) = do
263263
264264{-# INLINE deserializeWithSize #-}
265265deserializeWithSize ::
266− Serialize a => Int -> MutableByteArray -> Int -> IO (Int, a)
267−deserializeWithSize off arr endOff = deserialize (off + 4) arr endOff
266+ Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
267+deserializeWithSize off arr endOff = deserializeAt (off + 4) arr endOff
268268
269269conUpdateFuncDec :: Name -> [Field] -> Q [Dec]
270270conUpdateFuncDec funcName fields = do
@@ -282,7 +282,7 @@ conUpdateFuncDec funcName fields = do
282282 wildP
283283 (normalB
284284 [|do (valOff, valLen :: Word32) <-
285− deserialize
285+ deserializeAt
286286 $(varE curOff)
287287 $(varE arr)
288288 $(varE endOff)
@@ -317,7 +317,7 @@ conUpdateFuncDec funcName fields = do
317317 (litP fnameLit)
318318 (normalB
319319 [|do (valOff, valLen :: Word32) <-
320− deserialize
320+ deserializeAt
321321 $(varE currOff)
322322 $(varE arr)
323323 $(varE endOff)
@@ -345,7 +345,7 @@ mkDeserializeKeysDec funcName updateFunc (SimpleDataCon cname fields) = do
345345 errorUnsupported "The datatype should use record syntax."
346346 method <-
347347 [|do (dataOff, hlist :: CompactList (CompactList Word8)) <-
348− deserialize $(varE hOff) $(varE arr) $(varE endOff)
348+ deserializeAt $(varE hOff) $(varE arr) $(varE endOff)
349349 let keys = wListToString . unCompactList <$> unCompactList hlist
350350 ($(varP kvEncoded), _) <-
351351 foldlM
@@ -383,16 +383,16 @@ mkDeserializeKeysDec funcName updateFunc (SimpleDataCon cname fields) = do
383383 else [|error $(litE (StringL (nameBase k ++ " is not found.")))|]
384384
385385
386−mkDeserializeExpr :: Name -> Name -> Name -> SimpleDataCon -> Q Exp
387−mkDeserializeExpr initialOff endOff deserializeWithKeys con = do
386+mkRecDeserializeExpr :: Name -> Name -> Name -> SimpleDataCon -> Q Exp
387+mkRecDeserializeExpr initialOff endOff deserializeWithKeys con = do
388388 hOff <- newName "hOff"
389389 let sizeForFinalOff = 4 -- Word32
390390 sizeForHeaderLength = 4 -- Word32
391391 sizePreData = sizeForFinalOff + sizeForHeaderLength + hlen
392392 [|do (hlenOff, encLen :: Word32) <-
393− deserialize $(varE initialOff) $(varE _arr) $(varE endOff)
393+ deserializeAt $(varE initialOff) $(varE _arr) $(varE endOff)
394394 ($(varP hOff), hlen1 :: Word32) <-
395− deserialize hlenOff $(varE _arr) $(varE endOff)
395+ deserializeAt hlenOff $(varE _arr) $(varE endOff)
396396 if (hlen1 == $(litIntegral hlen)) && $(xorCmp hval hOff _arr)
397397 then do
398398 let $(varP (makeI 0)) =
core/src/Streamly/Internal/Data/Serialize/Type.hs+140−137
@@ -7,37 +7,27 @@
77--
88
99module Streamly.Internal.Data.Serialize.Type
10− ( Serialize(..)
11− , encode
12− , pinnedEncode
13− , decode
10+ (
11+ Serialize(..)
1412 ) where
1513
1614--------------------------------------------------------------------------------
1715-- Imports
1816--------------------------------------------------------------------------------
1917
20−#include "assert.hs"
21−
22−#ifdef DEBUG
23−import Control.Exception (assert)
24−#endif
25−
18+import Control.Monad (when)
2619import Data.List (foldl')
2720import Data.Proxy (Proxy (..))
28−import Streamly.Internal.Data.Unbox
29− ( MutableByteArray(..)
30− , PinnedState(..)
31− , Unbox
32− )
21+import Streamly.Internal.Data.Unbox (Unbox)
22+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray(..))
3323import Streamly.Internal.Data.Array.Type (Array(..))
34−import Streamly.Internal.System.IO (unsafeInlineIO)
3524import GHC.Int (Int16(..), Int32(..), Int64(..), Int8(..))
3625import GHC.Word (Word16(..), Word32(..), Word64(..), Word8(..))
3726import GHC.Stable (StablePtr(..))
3827
28+import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
3929import qualified Streamly.Internal.Data.Unbox as Unbox
40−import qualified Streamly.Internal.Data.Array as Array
30+import qualified Streamly.Internal.Data.Array.Type as Array
4131import qualified Streamly.Internal.Data.MutArray as MutArray
4232
4333import GHC.Exts
@@ -59,21 +49,33 @@ import GHC.Exts
5949-- Types
6050--------------------------------------------------------------------------------
6151
62−-- | A type implementing the 'Serialize' interface supplies operations for
63−-- reading and writing the type from and to a mutable byte array (an unboxed
64−-- representation of the type) in memory. The read operation 'deserialize'
65−-- deserializes the boxed type from the mutable byte array. The write operation
66−-- 'serialize' serializes the boxed type to the mutable byte array.
52+-- | The 'Serialize' type class provides operations for serialization and
53+-- deserialization of general Haskell data types to and from their byte stream
54+-- representation.
55+--
56+-- Unlike 'Unbox', 'Serialize' uses variable length encoding, therefore, it can
57+-- serialize recursive and variable length data types like lists, or variable
58+-- length sum types where the length of the value may vary depending on a
59+-- particular constructor. For variable length data types the length is encoded
60+-- along with the data.
6761--
68−-- IMPORTANT: The serialized data's byte ordering is not normalized, which means
69−-- it remains the same as the machine's byte order where the function is
70−-- executed. Consequently, it may not be compatible across machines with
71−-- different byte ordering.
62+-- The 'deserializeAt' operation reads bytes from the mutable byte array and
63+-- builds a Haskell data type from these bytes, the number of bytes it reads
64+-- depends on the type and the encoded value it is reading. 'serializeAt'
65+-- operation converts a Haskell data type to its binary representation which
66+-- must consist of as many bytes as added by the @addSizeTo@ operation for that
67+-- value and then stores these bytes into the mutable byte array. The
68+-- programmer is expected to use the @addSizeTo@ operation and allocate an
69+-- array of sufficient length before calling 'serializeAt'.
7270--
73−-- 'Serialize' contains enough information to serialize and deserialize variable
74−-- length types.
71+-- IMPORTANT: The serialized data's byte ordering remains the same as the host
72+-- machine's byte order. Therefore, it can not be deserialized from host
73+-- machines with a different byte ordering.
7574--
76−-- >>> import Streamly.Internal.Data.Serialize.Type (Serialize(..))
75+-- Instances can be derived via Template Haskell, or written manually.
76+--
77+-- Here is an example, for deriving an instance of this type class using
78+-- template Haskell:
7779--
7880-- >>> :{
7981-- data Object = Object
@@ -82,16 +84,30 @@ import GHC.Exts
8284-- }
8385-- :}
8486--
87+-- @
88+-- import Streamly.Data.MutByteArray (deriveSerialize)
89+-- \$(deriveSerialize [d|instance Serialize Object|])
90+-- @
91+--
92+-- See 'Streamly.Data.MutByteArray.deriveSerialize' and
93+-- 'Streamly.Data.MutByteArray.deriveSerializeWith' for more information on
94+-- deriving using Template Haskell.
95+--
96+-- Here is an example of a manual instance.
97+--
98+-- >>> import Streamly.Data.MutByteArray (Serialize(..))
99+--
85100-- >>> :{
86101-- instance Serialize Object where
87−-- size acc obj = size (size acc (_obj1 obj)) (_obj2 obj)
88−-- deserialize i arr len = do
89−-- (i1, x0) <- deserialize i arr len
90−-- (i2, x1) <- deserialize i1 arr len
102+-- addSizeTo acc obj = addSizeTo (addSizeTo acc (_obj1 obj)) (_obj2 obj)
103+-- deserializeAt i arr len = do
104+-- -- Check the array bounds before reading
105+-- (i1, x0) <- deserializeAt i arr len
106+-- (i2, x1) <- deserializeAt i1 arr len
91107-- pure (i2, Object x0 x1)
92−-- serialize i arr (Object x0 x1) = do
93−-- i1 <- serialize i arr x0
94−-- i2 <- serialize i1 arr x1
108+-- serializeAt i arr (Object x0 x1) = do
109+-- i1 <- serializeAt i arr x0
110+-- i2 <- serializeAt i1 arr x1
95111-- pure i2
96112-- :}
97113--
@@ -102,44 +118,50 @@ class Serialize a where
102118 -- It is of the form @Int -> a -> Int@ because you can have tail-recursive
103119 -- traversal of the structures.
104120
105− -- | Get the size, in bytes, reqired to store the serialized
106− -- representation of the type. Size cannot be zero.
107− size :: Int -> a -> Int
121+ -- | @addSizeTo accum value@ returns @accum@ incremented by the size of the
122+ -- serialized representation of @value@ in bytes. Size cannot be zero. It
123+ -- should be at least 1 byte.
124+ addSizeTo :: Int -> a -> Int
108125
109126 -- We can implement the following functions without returning the `Int`
110127 -- offset but that may require traversing the Haskell structure again to get
111128 -- the size. Therefore, this is a performance optimization.
112129
113− -- | @deserialize offset array arrayLen@ deserializes a value from the
114− -- given byte-index in the array. Returns a tuple of the next byte-index
115− -- and the deserialized value.
116− deserialize :: Int -> MutableByteArray -> Int -> IO (Int, a)
117−
118− -- | Write the serialized representation of the value in the array at the
119− -- given byte-index. Returns the next byte-index. This is an unsafe
120− -- operation, the programmer must ensure that the array has enough space
121− -- available in the array to serialize the value as determined by the
122− -- @size@ operation.
123− serialize :: Int -> MutableByteArray -> a -> IO Int
130+ -- | @deserializeAt byte-offset array arrayLen@ deserializes a value from
131+ -- the given byte-offset in the array. Returns a tuple consisting of the
132+ -- next byte-offset and the deserialized value.
133+ --
134+ -- The arrayLen passed is the entire length of the input buffer. It is to
135+ -- be used to check if we would overflow the input buffer when
136+ -- deserializing.
137+ --
138+ -- Throws an exception if the operation would exceed the supplied arrayLen.
139+ deserializeAt :: Int -> MutByteArray -> Int -> IO (Int, a)
140+
141+ -- | @serializeAt byte-offset array value@ writes the serialized
142+ -- representation of the @value@ in the array at the given byte-offset.
143+ -- Returns the next byte-offset.
144+ --
145+ -- This is an unsafe operation, the programmer must ensure that the array
146+ -- has enough space available to serialize the value as determined by the
147+ -- @addSizeTo@ operation.
148+ serializeAt :: Int -> MutByteArray -> a -> IO Int
124149
125150--------------------------------------------------------------------------------
126151-- Instances
127152--------------------------------------------------------------------------------
128153
129−-- XXX We do not need to pass the end of data offset, we can just check the
130−-- mutable array end here to avoid a crash, and when we return from deserialize
131−-- we can check if the offset returned is beyond the bound or not.
132−--
154+-- _size is the length from array start to the last accessed byte.
133155#ifdef DEBUG
134156{-# INLINE checkBounds #-}
135−checkBounds :: String -> Int -> MutableByteArray -> IO ()
136−checkBounds _label _off _arr = do
137− sz <- sizeOfMutableByteArray _arr
138− if (_off > sz)
157+checkBounds :: String -> Int -> MutByteArray -> IO ()
158+checkBounds _label _size _arr = do
159+ sz <- MBA.sizeOfMutableByteArray _arr
160+ if (_size > sz)
139161 then error
140162 $ _label
141163 ++ ": accessing array at offset = "
142− ++ show (_off - 1)
164+ ++ show (_size - 1)
143165 ++ " max valid offset = " ++ show (sz - 1)
144166 else return ()
145167#endif
@@ -162,39 +184,39 @@ checkBounds _label _off _arr = do
162184-- single array for deserializing. But that can also be done by framing the
163185-- serialized value with a size header.
164186--
165−{-# INLINE deserializeUnsafe #-}
166−deserializeUnsafe :: forall a. Unbox a => Int -> MutableByteArray -> Int -> IO (Int, a)
167−deserializeUnsafe off arr sz =
187+{-# INLINE deserializeChecked #-}
188+deserializeChecked :: forall a. Unbox a => Int -> MutByteArray -> Int -> IO (Int, a)
189+deserializeChecked off arr sz =
168190 let next = off + Unbox.sizeOf (Proxy :: Proxy a)
169191 in do
170192 -- Keep likely path in the straight branch.
171193 if (next <= sz)
172− then Unbox.peekByteIndex off arr >>= \val -> pure (next, val)
194+ then Unbox.peekAt off arr >>= \val -> pure (next, val)
173195 else error
174− $ "deserialize: accessing array at offset = "
196+ $ "deserializeAt: accessing array at offset = "
175197 ++ show (next - 1)
176198 ++ " max valid offset = " ++ show (sz - 1)
177199
178200{-# INLINE serializeUnsafe #-}
179−serializeUnsafe :: forall a. Unbox a => Int -> MutableByteArray -> a -> IO Int
201+serializeUnsafe :: forall a. Unbox a => Int -> MutByteArray -> a -> IO Int
180202serializeUnsafe off arr val =
181203 let next = off + Unbox.sizeOf (Proxy :: Proxy a)
182204 in do
183205#ifdef DEBUG
184− checkBounds "serialize" next arr
206+ checkBounds "serializeAt" next arr
185207#endif
186− Unbox.pokeByteIndex off arr val
208+ Unbox.pokeAt off arr val
187209 pure next
188210
189211#define DERIVE_SERIALIZE_FROM_UNBOX(_type) \
190212instance Serialize _type where \
191−; {-# INLINE size #-} \
192−; size acc _ = acc + Unbox.sizeOf (Proxy :: Proxy _type) \
193−; {-# INLINE deserialize #-} \
194−; deserialize off arr end = deserializeUnsafe off arr end :: IO (Int, _type) \
195−; {-# INLINE serialize #-} \
196−; serialize = \
197− serializeUnsafe :: Int -> MutableByteArray -> _type -> IO Int
213+; {-# INLINE addSizeTo #-} \
214+; addSizeTo acc _ = acc + Unbox.sizeOf (Proxy :: Proxy _type) \
215+; {-# INLINE deserializeAt #-} \
216+; deserializeAt off arr end = deserializeChecked off arr end :: IO (Int, _type) \
217+; {-# INLINE serializeAt #-} \
218+; serializeAt = \
219+ serializeUnsafe :: Int -> MutByteArray -> _type -> IO Int
198220
199221DERIVE_SERIALIZE_FROM_UNBOX(())
200222DERIVE_SERIALIZE_FROM_UNBOX(Bool)
@@ -217,98 +239,79 @@ DERIVE_SERIALIZE_FROM_UNBOX((FunPtr a))
217239
218240instance forall a. Serialize a => Serialize [a] where
219241
220− -- {-# INLINE size #-}
221− size acc xs = foldl' size (acc + (Unbox.sizeOf (Proxy :: Proxy Int))) xs
242+ -- {-# INLINE addSizeTo #-}
243+ addSizeTo acc xs =
244+ foldl' addSizeTo (acc + (Unbox.sizeOf (Proxy :: Proxy Int))) xs
222245
223246 -- Inlining this causes large compilation times for tests
224− {-# INLINABLE deserialize #-}
225− deserialize off arr sz = do
226− (off1, len64) <- deserialize off arr sz :: IO (Int, Int64)
247+ {-# INLINABLE deserializeAt #-}
248+ deserializeAt off arr sz = do
249+ (off1, len64) <- deserializeAt off arr sz :: IO (Int, Int64)
227250 let len = (fromIntegral :: Int64 -> Int) len64
228251 peekList f o i | i >= 3 = do
229252 -- Unfold the loop three times
230− (o1, x1) <- deserialize o arr sz
231− (o2, x2) <- deserialize o1 arr sz
232− (o3, x3) <- deserialize o2 arr sz
253+ (o1, x1) <- deserializeAt o arr sz
254+ (o2, x2) <- deserializeAt o1 arr sz
255+ (o3, x3) <- deserializeAt o2 arr sz
233256 peekList (f . (\xs -> x1:x2:x3:xs)) o3 (i - 3)
234257 peekList f o 0 = pure (o, f [])
235258 peekList f o i = do
236− (o1, x) <- deserialize o arr sz
259+ (o1, x) <- deserializeAt o arr sz
237260 peekList (f . (x:)) o1 (i - 1)
238261 peekList id off1 len
239262
240263 -- Inlining this causes large compilation times for tests
241− {-# INLINABLE serialize #-}
242− serialize off arr val = do
264+ {-# INLINABLE serializeAt #-}
265+ serializeAt off arr val = do
243266 let off1 = off + Unbox.sizeOf (Proxy :: Proxy Int64)
244267 let pokeList acc o [] =
245− Unbox.pokeByteIndex off arr (acc :: Int64) >> pure o
268+ Unbox.pokeAt off arr (acc :: Int64) >> pure o
246269 pokeList acc o (x:xs) = do
247− o1 <- serialize o arr x
270+ o1 <- serializeAt o arr x
248271 pokeList (acc + 1) o1 xs
249272 pokeList 0 off1 val
250273
251−instance Serialize (Array a) where
252− {-# INLINE size #-}
253− size i (Array {..}) = i + (arrEnd - arrStart) + 8
274+instance
275+#ifdef DEVBUILD
276+ Unbox a =>
277+#endif
278+ Serialize (Array a) where
279+ {-# INLINE addSizeTo #-}
280+ addSizeTo i (Array {..}) = i + (arrEnd - arrStart) + 8
254281
255− {-# INLINE deserialize #-}
256− deserialize off arr end = do
257− (off1, byteLen) <- deserialize off arr end :: IO (Int, Int)
282+ {-# INLINE deserializeAt #-}
283+ deserializeAt off arr len = do
284+ (off1, byteLen) <- deserializeAt off arr len :: IO (Int, Int)
258285 let off2 = off1 + byteLen
286+ when (off2 > len) $
287+ error
288+ $ "deserializeAt: accessing array at offset = "
289+ ++ show (off2 - 1)
290+ ++ " max valid offset = " ++ show (len - 1)
291+ -- XXX Use MutByteArray.cloneSliceUnsafe
259292 let slice = MutArray.MutArray arr off1 off2 off2
260293 newArr <- MutArray.clone slice
261294 pure (off2, Array.unsafeFreeze newArr)
262295
263− {-# INLINE serialize #-}
264− serialize off arr (Array {..}) = do
296+ {-# INLINE serializeAt #-}
297+ serializeAt off arr (Array {..}) = do
265298 let arrLen = arrEnd - arrStart
266− off1 <- serialize off arr arrLen
267− Unbox.putSliceUnsafe arrContents arrStart arr off1 arrLen
299+ off1 <- serializeAt off arr arrLen
300+ MBA.putSliceUnsafe arrContents arrStart arr off1 arrLen
268301 pure (off1 + arrLen)
269302
270303instance (Serialize a, Serialize b) => Serialize (a, b) where
271304
272− {-# INLINE size #-}
273− size acc (a, b) = size (size acc a) b
305+ {-# INLINE addSizeTo #-}
306+ addSizeTo acc (a, b) = addSizeTo (addSizeTo acc a) b
274307
275− {-# INLINE serialize #-}
276− serialize off arr (a, b) = do
277− off1 <- serialize off arr a
278− serialize off1 arr b
308+ {-# INLINE serializeAt #-}
309+ serializeAt off arr (a, b) = do
310+ off1 <- serializeAt off arr a
311+ serializeAt off1 arr b
279312
280− {-# INLINE deserialize #-}
281− deserialize off arr end = do
282− (off1, a) <- deserialize off arr end
283− (off2, b) <- deserialize off1 arr end
313+ {-# INLINE deserializeAt #-}
314+ deserializeAt off arr end = do
315+ (off1, a) <- deserializeAt off arr end
316+ (off2, b) <- deserializeAt off1 arr end
284317 pure (off2, (a, b))
285−
286−--------------------------------------------------------------------------------
287−-- High level functions
288−--------------------------------------------------------------------------------
289−
290−{-# INLINE encodeAs #-}
291−encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
292−encodeAs ps a =
293− unsafeInlineIO $ do
294− let len = size 0 a
295− mbarr <- Unbox.newBytesAs ps len
296− off <- serialize 0 mbarr a
297− assertM(len == off)
298− pure $ Array mbarr 0 off
299−
300−{-# INLINE encode #-}
301−encode :: Serialize a => a -> Array Word8
302−encode = encodeAs Unpinned
303−
304−{-# INLINE pinnedEncode #-}
305−pinnedEncode :: Serialize a => a -> Array Word8
306−pinnedEncode = encodeAs Pinned
307−
308−{-# INLINE decode #-}
309−decode :: Serialize a => Array Word8 -> a
310−decode arr@(Array {..}) = unsafeInlineIO $ do
311− let lenArr = Array.length arr
312− (off, val) <- deserialize arrStart arrContents (arrStart + lenArr)
313− assertM(off == arrStart + lenArr)
314− pure val
core/src/Streamly/Internal/Data/Stream/Container.hs+56−48
@@ -11,16 +11,17 @@
1111
1212module Streamly.Internal.Data.Stream.Container
1313 (
14− nub
14+ -- * Deduplication
15+ ordNub
1516
16− -- * Joins for unconstrained types
17− , joinLeftGeneric
18− , joinOuterGeneric
17+ -- * Joins
18+ , leftJoin
19+ , outerJoin
1920
20− -- * Joins with Ord constraint
21− , joinInner
22− , joinLeft
23− , joinOuter
21+ -- * Ord Joins
22+ , innerOrdJoin
23+ , leftOrdJoin
24+ , outerOrdJoin
2425 )
2526where
2627
@@ -47,12 +48,15 @@ import qualified Streamly.Internal.Data.Stream.Transformer as Stream
4748
4849#include "DocTestDataStream.hs"
4950
50−-- | The memory used is proportional to the number of unique elements in the
51−-- stream. If we want to limit the memory we can just use "take" to limit the
52−-- uniq elements in the stream.
53−{-# INLINE_NORMAL nub #-}
54−nub :: (Monad m, Ord a) => Stream m a -> Stream m a
55−nub (Stream step1 state1) = Stream step (Set.empty, state1)
51+-- | @nub@ specialized to 'Ord' types for better performance. Returns a
52+-- subsequence of the stream removing any duplicate elements.
53+--
54+-- The memory used is proportional to the number of unique elements in the
55+-- stream. One way to limit the memory is to use @take@ on the resulting
56+-- stream to limit the unique elements in the stream.
57+{-# INLINE_NORMAL ordNub #-}
58+ordNub :: (Monad m, Ord a) => Stream m a -> Stream m a
59+ordNub (Stream step1 state1) = Stream step (Set.empty, state1)
5660
5761 where
5862
@@ -78,8 +82,8 @@ toMap =
7882--
7983-- XXX An IntMap may be faster when the keys are Int.
8084-- XXX Use hashmap instead of map?
81−--
82−-- | Like 'joinInner' but uses a 'Map' for efficiency.
85+
86+-- | 'innerJoin' specialized to 'Ord' types for better performance.
8387--
8488-- If the input streams have duplicate keys, the behavior is undefined.
8589--
@@ -90,10 +94,10 @@ toMap =
9094-- Time: O(m + n)
9195--
9296-- /Pre-release/
93−{-# INLINE joinInner #-}
94−joinInner :: (Monad m, Ord k) =>
97+{-# INLINE innerOrdJoin #-}
98+innerOrdJoin :: (Monad m, Ord k) =>
9599 Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, b)
96−joinInner s1 s2 =
100+innerOrdJoin s1 s2 =
97101 Stream.concatEffect $ do
98102 km <- toMap s2
99103 pure $ Stream.mapMaybe (joinAB km) s1
@@ -106,32 +110,33 @@ joinInner s1 s2 =
106110 Nothing -> Nothing
107111
108112-- XXX We can do this concurrently.
113+-- XXX Check performance of StreamD vs StreamK
109114-- XXX If the second stream is sorted and passed as an Array or a seek capable
110115-- stream then we could use binary search if we have an Ord instance or
111116-- Ordering returning function. The time complexity would then become (m x log
112117-- n).
113118
114−-- XXX Check performance of StreamD vs StreamK
115−
116−-- | Like 'joinInner' but emit @(a, Just b)@, and additionally, for those @a@'s
117−-- that are not equal to any @b@ emit @(a, Nothing)@.
119+-- | Like 'innerJoin' but emits @(a, Just b)@ whenever a and b are equal, for
120+-- those @a@'s that are not equal to any @b@ emits @(a, Nothing)@.
118121--
119−-- The second stream is evaluated multiple times. If the stream is a
120−-- consume-once stream then the caller should cache it in an 'Data.Array.Array'
121−-- before calling this function. Caching may also improve performance if the
122−-- stream is expensive to evaluate.
122+-- This is a generalization of 'innerJoin' to include all elements from the
123+-- left stream and not just those which have an equal in the right stream. This
124+-- is not a commutative operation, the order of the stream arguments matters.
123125--
124−-- >>> joinRightGeneric eq = flip (Stream.joinLeftGeneric eq)
126+-- All the caveats mentioned in 'innerJoin' apply here as well. Right join is
127+-- not provided because it is just a flipped left join:
128+--
129+-- >>> rightJoin eq = flip (Stream.leftJoin eq)
125130--
126131-- Space: O(n) assuming the second stream is cached in memory.
127132--
128133-- Time: O(m x n)
129134--
130135-- /Unimplemented/
131−{-# INLINE joinLeftGeneric #-}
132−joinLeftGeneric :: Monad m =>
136+{-# INLINE leftJoin #-}
137+leftJoin :: Monad m =>
133138 (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
134−joinLeftGeneric eq s1 s2 = Stream.evalStateT (return False) $ unCross $ do
139+leftJoin eq s1 s2 = Stream.evalStateT (return False) $ unCross $ do
135140 a <- mkCross (Stream.liftInner s1)
136141 -- XXX should we use StreamD monad here?
137142 -- XXX Is there a better way to perform some action at the end of a loop
@@ -152,19 +157,17 @@ joinLeftGeneric eq s1 s2 = Stream.evalStateT (return False) $ unCross $ do
152157 else mkCross Stream.nil
153158 Nothing -> return (a, Nothing)
154159
155−-- XXX rename to joinLeftOrd?
156−
157−-- | A more efficient 'joinLeft' using a hashmap for efficiency.
160+-- | 'leftJoin' specialized to 'Ord' types for better performance.
158161--
159162-- Space: O(n)
160163--
161164-- Time: O(m + n)
162165--
163166-- /Pre-release/
164−{-# INLINE joinLeft #-}
165−joinLeft :: (Ord k, Monad m) =>
167+{-# INLINE leftOrdJoin #-}
168+leftOrdJoin :: (Ord k, Monad m) =>
166169 Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, Maybe b)
167−joinLeft s1 s2 =
170+leftOrdJoin s1 s2 =
168171 Stream.concatEffect $ do
169172 km <- toMap s2
170173 return $ fmap (joinAB km) s1
@@ -177,14 +180,19 @@ joinLeft s1 s2 =
177180 Nothing -> (k, a, Nothing)
178181
179182-- XXX We can do this concurrently.
183+-- XXX Check performance of StreamD vs StreamK cross operation.
180184
181−-- XXX Check performance of StreamD vs StreamK
182−
183−-- | Like 'joinLeft' but emits a @(Just a, Just b)@. Like 'joinLeft', for those
185+-- | Like 'leftJoin' but emits a @(Just a, Just b)@. Like 'leftJoin', for those
184186-- @a@'s that are not equal to any @b@ emit @(Just a, Nothing)@, but
185187-- additionally, for those @b@'s that are not equal to any @a@ emit @(Nothing,
186188-- Just b)@.
187189--
190+-- This is a generalization of left join to include all the elements from the
191+-- right stream as well, in other words it is a combination of left and right
192+-- joins. This is a commutative operation. The order of stream arguments can be
193+-- changed without affecting results, except for the ordering of elements in
194+-- the resulting tuple.
195+--
188196-- For space efficiency use the smaller stream as the second stream.
189197--
190198-- Space: O(n)
@@ -192,15 +200,15 @@ joinLeft s1 s2 =
192200-- Time: O(m x n)
193201--
194202-- /Pre-release/
195−{-# INLINE joinOuterGeneric #-}
196−joinOuterGeneric :: MonadIO m =>
203+{-# INLINE outerJoin #-}
204+outerJoin :: MonadIO m =>
197205 (a -> b -> Bool)
198206 -> Stream m a
199207 -> Stream m b
200208 -> Stream m (Maybe a, Maybe b)
201−joinOuterGeneric eq s1 s =
209+outerJoin eq s1 s2 =
202210 Stream.concatEffect $ do
203− inputArr <- Array.fromStream s
211+ inputArr <- Array.fromStream s2
204212 let len = Array.length inputArr
205213 foundArr <-
206214 Stream.fold
@@ -254,18 +262,18 @@ joinOuterGeneric eq s1 s =
254262-- a flag. At the end go through @Stream m b@ and find those that are not in that
255263-- hash to return (Nothing, b).
256264
257−-- | Like 'joinOuter' but uses a 'Map' for efficiency.
265+-- | 'outerJoin' specialized to 'Ord' types for better performance.
258266--
259267-- Space: O(m + n)
260268--
261269-- Time: O(m + n)
262270--
263271-- /Pre-release/
264−{-# INLINE joinOuter #-}
265−joinOuter ::
272+{-# INLINE outerOrdJoin #-}
273+outerOrdJoin ::
266274 (Ord k, MonadIO m) =>
267275 Stream m (k, a) -> Stream m (k, b) -> Stream m (k, Maybe a, Maybe b)
268−joinOuter s1 s2 =
276+outerOrdJoin s1 s2 =
269277 Stream.concatEffect $ do
270278 km1 <- kvFold s1
271279 km2 <- kvFold s2
core/src/Streamly/Internal/Data/Stream/Eliminate.hs+33−25
@@ -13,36 +13,24 @@
1313--
1414module Streamly.Internal.Data.Stream.Eliminate
1515 (
16− -- * Running a 'Fold'
17− fold
18−
19− -- -- * Running a 'Parser'
20− , parse
16+ -- * Running a Parser
17+ parse
2118 , parseD
2219 , parseBreak
2320 , parseBreakD
2421
25− -- * Stream Deconstruction
22+ -- * Deconstruction
2623 , uncons
2724
2825 -- * Right Folds
29− , foldrM
30− , foldr
31− , foldrMx
3226 , foldr1
3327
34− -- * Left Folds
35− , foldlM'
36− , foldl'
37− , foldlMx'
38− , foldlx'
39−
4028 -- * Specific Fold Functions
41− , drain
4229 , mapM_ -- Map and Fold
4330 , null
4431 , head
4532 , headElse
33+ , init
4634 , tail
4735 , last
4836 , elem
@@ -60,16 +48,9 @@ module Streamly.Internal.Data.Stream.Eliminate
6048 , the
6149
6250 -- * To containers
63− , toList
6451 , toListRev
6552
6653 -- * Multi-Stream Folds
67− -- ** Comparisons
68− -- | These should probably be expressed using zipping operations.
69− , eqBy
70− , cmpBy
71−
72− -- ** Substreams
7354 -- | These should probably be expressed using parsers.
7455 , isPrefixOf
7556 , isInfixOf
@@ -90,7 +71,7 @@ import Foreign.Storable (Storable)
9071import GHC.Exts (SpecConstrAnnotation(..))
9172import GHC.Types (SPEC(..))
9273import Streamly.Internal.Data.Parser (ParseError(..))
93−import Streamly.Internal.Data.SVar.Type (defState)
74+import Streamly.Internal.Data.SVar.Type (adaptState, defState)
9475import Streamly.Internal.Data.Unbox (Unbox)
9576
9677import Streamly.Internal.Data.Maybe.Strict (Maybe'(..))
@@ -105,7 +86,7 @@ import qualified Streamly.Internal.Data.Stream.Transform as StreamD
10586
10687import Prelude hiding
10788 ( Foldable(..), all, any, head, last, lookup, mapM, mapM_
108− , notElem, splitAt, tail, (!!))
89+ , notElem, splitAt, init, tail, (!!))
10990import Data.Foldable (length)
11091import Streamly.Internal.Data.Stream.Type
11192
@@ -391,9 +372,35 @@ head = foldrM (\x _ -> return (Just x)) (return Nothing)
391372headElse :: Monad m => a -> Stream m a -> m a
392373headElse a = foldrM (\x _ -> return x) (return a)
393374
375+{-# INLINE_NORMAL init #-}
376+init :: Monad m => Stream m a -> m (Maybe (Stream m a))
377+init stream = do
378+ r <- uncons stream
379+ case r of
380+ Nothing -> return Nothing
381+ Just (h, Stream step1 state1) ->
382+ return $ Just $ Stream step (h, state1)
383+
384+ where
385+
386+ step gst (a, s1) = do
387+ res <- step1 (adaptState gst) s1
388+ return $
389+ case res of
390+ Yield x s -> Yield a (x, s)
391+ Skip s -> Skip (a, s)
392+ Stop -> Stop
393+
394+-- | Same as:
395+--
396+-- >>> tail = fmap (fmap snd) . Stream.uncons
397+--
394398-- Does not fuse, has the same performance as the StreamK version.
399+--
395400{-# INLINE_NORMAL tail #-}
396401tail :: Monad m => Stream m a -> m (Maybe (Stream m a))
402+tail = fmap (fmap snd) . uncons
403+{-
397404tail (UnStream step state) = go SPEC state
398405 where
399406 go !_ st = do
@@ -402,6 +409,7 @@ tail (UnStream step state) = go SPEC state
402409 Yield _ s -> return (Just $ Stream step s)
403410 Skip s -> go SPEC s
404411 Stop -> return Nothing
412+-}
405413
406414-- XXX will it fuse? need custom impl?
407415{-# INLINE_NORMAL last #-}
core/src/Streamly/Internal/Data/Stream/Exception.hs+9−6
@@ -9,17 +9,20 @@
99
1010module Streamly.Internal.Data.Stream.Exception
1111 (
12− gbracket_
13− , gbracket
14− , before
15− , afterUnsafe
12+ -- * Resources
13+ before
1614 , afterIO
15+ , afterUnsafe
16+ , finallyIO
17+ , finallyUnsafe
18+ , gbracket_
19+ , gbracket
1720 , bracketUnsafe
1821 , bracketIO3
1922 , bracketIO
23+
24+ -- * Exceptions
2025 , onException
21− , finallyUnsafe
22− , finallyIO
2326 , ghandle
2427 , handle
2528 )
core/src/Streamly/Internal/Data/Stream/Generate.hs+2−12
@@ -16,20 +16,13 @@ module Streamly.Internal.Data.Stream.Generate
1616 (
1717 -- * Primitives
1818 nil
19− , nilM
2019 , cons
21− , consM
22−
23− -- * From 'Unfold'
24− , unfold
2520
2621 -- * Unfolding
2722 , unfoldr
2823 , unfoldrM
2924
3025 -- * From Values
31− , fromPure
32− , fromEffect
3326 , repeat
3427 , repeatM
3528 , replicate
@@ -95,7 +88,6 @@ module Streamly.Internal.Data.Stream.Generate
9588 -- * From Containers
9689 -- | Transform an input structure into a stream.
9790
98− , fromList
9991 , fromListM
10092 , fromFoldable
10193 , fromFoldableM
@@ -104,10 +96,6 @@ module Streamly.Internal.Data.Stream.Generate
10496 , fromPtr
10597 , fromPtrN
10698 , fromByteStr#
107−
108− -- * Conversions
109− , fromStreamK
110− , toStreamK
11199 )
112100where
113101
@@ -154,6 +142,8 @@ import Streamly.Internal.Data.Stream.Type
154142nil :: Applicative m => Stream m a
155143nil = Stream (\_ _ -> pure Stop) ()
156144
145+infixr 5 `cons`
146+
157147-- XXX implement in terms of consM?
158148-- cons x = consM (return x)
159149-- From an implementation perspective, StreamK.cons translates into a
core/src/Streamly/Internal/Data/Stream/MutChunked.hsdeleted−323
@@ -1,323 +0,0 @@
1−-- |
2−-- Module : Streamly.Internal.Data.Stream.MutChunked
3−-- Copyright : (c) 2019 Composewell Technologies
4−-- License : BSD3-3-Clause
5−-- Maintainer : streamly@composewell.com
6−-- Stability : experimental
7−-- Portability : GHC
8−--
9−-- Combinators to efficiently manipulate streams of mutable arrays.
10−--
11−module Streamly.Internal.Data.Stream.MutChunked
12− (
13− -- * Generation
14− chunksOf
15−
16− -- * Compaction
17− , packArraysChunksOf
18− , SpliceState (..)
19− , lpackArraysChunksOf
20− , compact
21− , compactLE
22− , compactEQ
23− , compactGE
24− )
25−where
26−
27−#include "inline.hs"
28−#include "ArrayMacros.h"
29−
30−import Control.Monad.IO.Class (MonadIO(..))
31−import Control.Monad (when)
32−import Data.Bifunctor (first)
33−import Data.Proxy (Proxy(..))
34−import Streamly.Internal.Data.Unbox (Unbox, sizeOf)
35−import Streamly.Internal.Data.MutArray.Type (MutArray(..))
36−import Streamly.Internal.Data.Fold.Type (Fold(..))
37−import Streamly.Internal.Data.Parser (ParseError)
38−import Streamly.Internal.Data.Stream.Type (Stream)
39−import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
40−
41−import qualified Streamly.Internal.Data.MutArray.Type as MArray
42−import qualified Streamly.Internal.Data.Fold.Type as FL
43−import qualified Streamly.Internal.Data.Parser as ParserD
44−import qualified Streamly.Internal.Data.Stream as D
45−
46−-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
47−-- @n@ elements each.
48−--
49−-- Same as the following but may be more efficient:
50−--
51−-- > chunksOf n = Stream.foldMany (MArray.writeN n)
52−--
53−-- /Pre-release/
54−{-# INLINE chunksOf #-}
55−chunksOf :: (MonadIO m, Unbox a)
56− => Int -> Stream m a -> Stream m (MutArray a)
57−chunksOf = MArray.chunksOf
58−
59−-------------------------------------------------------------------------------
60−-- Compact
61−-------------------------------------------------------------------------------
62−
63−data SpliceState s arr
64− = SpliceInitial s
65− | SpliceBuffering s arr
66− | SpliceYielding arr (SpliceState s arr)
67− | SpliceFinish
68−
69−-- XXX This can be removed once compactLEFold/compactLE are implemented.
70−--
71−-- | This mutates the first array (if it has space) to append values from the
72−-- second one. This would work for immutable arrays as well because an
73−-- immutable array never has space so a new array is allocated instead of
74−-- mutating it.
75−--
76−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
77−-- maximum specified size. Note that if a single array is bigger than the
78−-- specified size we do not split it to fit. When we coalesce multiple arrays
79−-- if the size would exceed the specified size we do not coalesce therefore the
80−-- actual array size may be less than the specified chunk size.
81−--
82−-- @since 0.7.0
83−{-# INLINE_NORMAL packArraysChunksOf #-}
84−packArraysChunksOf :: (MonadIO m, Unbox a)
85− => Int -> D.Stream m (MutArray a) -> D.Stream m (MutArray a)
86−packArraysChunksOf n (D.Stream step state) =
87− D.Stream step' (SpliceInitial state)
88−
89− where
90−
91− {-# INLINE_LATE step' #-}
92− step' gst (SpliceInitial st) = do
93− when (n <= 0) $
94− -- XXX we can pass the module string from the higher level API
95− error $ "Streamly.Internal.Data.MutArray.Type.packArraysChunksOf: the size of "
96− ++ "arrays [" ++ show n ++ "] must be a natural number"
97− r <- step gst st
98− case r of
99− D.Yield arr s -> return $
100− let len = MArray.byteLength arr
101− in if len >= n
102− then D.Skip (SpliceYielding arr (SpliceInitial s))
103− else D.Skip (SpliceBuffering s arr)
104− D.Skip s -> return $ D.Skip (SpliceInitial s)
105− D.Stop -> return D.Stop
106−
107− step' gst (SpliceBuffering st buf) = do
108− r <- step gst st
109− case r of
110− D.Yield arr s -> do
111− let len = MArray.byteLength buf + MArray.byteLength arr
112− if len > n
113− then return $
114− D.Skip (SpliceYielding buf (SpliceBuffering s arr))
115− else do
116− buf' <- if MArray.byteCapacity buf < n
117− then liftIO $ MArray.realloc n buf
118− else return buf
119− buf'' <- MArray.splice buf' arr
120− return $ D.Skip (SpliceBuffering s buf'')
121− D.Skip s -> return $ D.Skip (SpliceBuffering s buf)
122− D.Stop -> return $ D.Skip (SpliceYielding buf SpliceFinish)
123−
124− step' _ SpliceFinish = return D.Stop
125−
126− step' _ (SpliceYielding arr next) = return $ D.Yield arr next
127−
128−-- XXX Remove this once compactLEFold is implemented
129−-- lpackArraysChunksOf = Fold.many compactLEFold
130−--
131−{-# INLINE_NORMAL lpackArraysChunksOf #-}
132−lpackArraysChunksOf :: (MonadIO m, Unbox a)
133− => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
134−lpackArraysChunksOf n (Fold step1 initial1 extract1) =
135− Fold step initial extract
136−
137− where
138−
139− initial = do
140− when (n <= 0) $
141− -- XXX we can pass the module string from the higher level API
142− error $ "Streamly.Internal.Data.MutArray.Type.packArraysChunksOf: the size of "
143− ++ "arrays [" ++ show n ++ "] must be a natural number"
144−
145− r <- initial1
146− return $ first (Tuple' Nothing) r
147−
148− extract (Tuple' Nothing r1) = extract1 r1
149− extract (Tuple' (Just buf) r1) = do
150− r <- step1 r1 buf
151− case r of
152− FL.Partial rr -> extract1 rr
153− FL.Done _ -> return ()
154−
155− step (Tuple' Nothing r1) arr =
156− let len = MArray.byteLength arr
157− in if len >= n
158− then do
159− r <- step1 r1 arr
160− case r of
161− FL.Done _ -> return $ FL.Done ()
162− FL.Partial s -> do
163− extract1 s
164− res <- initial1
165− return $ first (Tuple' Nothing) res
166− else return $ FL.Partial $ Tuple' (Just arr) r1
167−
168− step (Tuple' (Just buf) r1) arr = do
169− let len = MArray.byteLength buf + MArray.byteLength arr
170− buf' <- if MArray.byteCapacity buf < len
171− then liftIO $ MArray.realloc (max n len) buf
172− else return buf
173− buf'' <- MArray.splice buf' arr
174−
175− -- XXX this is common in both the equations of step
176− if len >= n
177− then do
178− r <- step1 r1 buf''
179− case r of
180− FL.Done _ -> return $ FL.Done ()
181− FL.Partial s -> do
182− extract1 s
183− res <- initial1
184− return $ first (Tuple' Nothing) res
185− else return $ FL.Partial $ Tuple' (Just buf'') r1
186−
187−-- XXX Same as compactLE, to be removed once that is implemented.
188−--
189−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
190−-- maximum specified size in bytes.
191−--
192−-- /Internal/
193−{-# INLINE compact #-}
194−compact :: (MonadIO m, Unbox a)
195− => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
196−compact = packArraysChunksOf
197−
198−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
199−-- maximum specified size. Note that if a single array is bigger than the
200−-- specified size we do not split it to fit. When we coalesce multiple arrays
201−-- if the size would exceed the specified size we do not coalesce therefore the
202−-- actual array size may be less than the specified chunk size.
203−--
204−-- /Internal/
205−{-# INLINE_NORMAL compactLEParserD #-}
206−compactLEParserD ::
207− forall m a. (MonadIO m, Unbox a)
208− => Int -> ParserD.Parser (MutArray a) m (MutArray a)
209−compactLEParserD n = ParserD.Parser step initial extract
210−
211− where
212−
213− nBytes = n * SIZE_OF(a)
214−
215− initial =
216− return
217− $ if n <= 0
218− then error
219− $ functionPath
220− ++ ": the size of arrays ["
221− ++ show n ++ "] must be a natural number"
222− else ParserD.IPartial Nothing
223−
224− step Nothing arr =
225− return
226− $ let len = MArray.byteLength arr
227− in if len >= nBytes
228− then ParserD.Done 0 arr
229− else ParserD.Partial 0 (Just arr)
230− step (Just buf) arr =
231− let len = MArray.byteLength buf + MArray.byteLength arr
232− in if len > nBytes
233− then return $ ParserD.Done 1 buf
234− else do
235− buf1 <-
236− if MArray.byteCapacity buf < nBytes
237− then liftIO $ MArray.realloc nBytes buf
238− else return buf
239− buf2 <- MArray.splice buf1 arr
240− return $ ParserD.Partial 0 (Just buf2)
241−
242− extract Nothing = return $ ParserD.Done 0 MArray.nil
243− extract (Just buf) = return $ ParserD.Done 0 buf
244−
245− functionPath =
246− "Streamly.Internal.Data.Stream.MutChunked.compactLEParserD"
247−
248−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
249−-- minimum specified size. Note that if all the arrays in the stream together
250−-- are smaller than the specified size the resulting array will be smaller than
251−-- the specified size. When we coalesce multiple arrays if the size would exceed
252−-- the specified size we stop coalescing further.
253−--
254−-- /Internal/
255−{-# INLINE_NORMAL compactGEFold #-}
256−compactGEFold ::
257− forall m a. (MonadIO m, Unbox a)
258− => Int -> FL.Fold m (MutArray a) (MutArray a)
259−compactGEFold n = Fold step initial extract
260−
261− where
262−
263− nBytes = n * SIZE_OF(a)
264−
265− initial =
266− return
267− $ if n < 0
268− then error
269− $ functionPath
270− ++ ": the size of arrays ["
271− ++ show n ++ "] must be a natural number"
272− else FL.Partial Nothing
273−
274− step Nothing arr =
275− return
276− $ let len = MArray.byteLength arr
277− in if len >= nBytes
278− then FL.Done arr
279− else FL.Partial (Just arr)
280− step (Just buf) arr = do
281− let len = MArray.byteLength buf + MArray.byteLength arr
282− buf1 <-
283− if MArray.byteCapacity buf < len
284− then liftIO $ MArray.realloc (max len nBytes) buf
285− else return buf
286− buf2 <- MArray.splice buf1 arr
287− if len >= n
288− then return $ FL.Done buf2
289− else return $ FL.Partial (Just buf2)
290−
291− extract Nothing = return MArray.nil
292− extract (Just buf) = return buf
293−
294− functionPath =
295− "Streamly.Internal.Data.Stream.MutChunked.compactGEFold"
296−
297−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
298−-- maximum specified size in bytes.
299−--
300−-- /Internal/
301−compactLE :: (MonadIO m, Unbox a) =>
302− Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
303−compactLE n = D.parseManyD (compactLEParserD n)
304−
305−-- | Like 'compactLE' but generates arrays of exactly equal to the size
306−-- specified except for the last array in the stream which could be shorter.
307−--
308−-- /Unimplemented/
309−{-# INLINE compactEQ #-}
310−compactEQ :: -- (MonadIO m, Unbox a) =>
311− Int -> Stream m (MutArray a) -> Stream m (MutArray a)
312−compactEQ _n _xs = undefined
313− -- IsStream.fromStreamD $ D.foldMany (compactEQFold n) (IsStream.toStreamD xs)
314−
315−-- | Like 'compactLE' but generates arrays of size greater than or equal to the
316−-- specified except for the last array in the stream which could be shorter.
317−--
318−-- /Internal/
319−{-# INLINE compactGE #-}
320−compactGE ::
321− (MonadIO m, Unbox a)
322− => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
323−compactGE n = D.foldMany (compactGEFold n)
core/src/Streamly/Internal/Data/Stream/Nesting.hs+86−202
@@ -55,11 +55,6 @@ module Streamly.Internal.Data.Stream.Nesting
5555 -- yields an element. A special case of unfoldRoundRobin.
5656 , roundRobin -- interleaveFair?/ParallelFair
5757
58− -- *** Zipping
59− -- | Zip corresponding elements of two streams.
60− , zipWith
61− , zipWithM
62−
6358 -- *** Merging
6459 -- | Interleave elements from two streams based on a condition.
6560 , mergeBy
@@ -76,17 +71,10 @@ module Streamly.Internal.Data.Stream.Nesting
7671 -- unfoldMany: Unfold m a b -> t m a -> t m b
7772 -- @
7873
79− -- *** ConcatMap
80− -- | Generate streams by mapping a stream generator on each element of an
81− -- input stream, append the resulting streams and flatten.
82− , concatMap
83− , concatMapM
84−
8574 -- *** ConcatUnfold
8675 -- | Generate streams by using an unfold on each element of an input
8776 -- stream, append the resulting streams and flatten. A special case of
8877 -- gintercalate.
89− , unfoldMany
9078 , ConcatUnfoldInterleaveState (..)
9179 , unfoldInterleave
9280 , unfoldRoundRobin
@@ -118,11 +106,8 @@ module Streamly.Internal.Data.Stream.Nesting
118106
119107 -- ** Folding
120108 -- | Apply folds on a stream.
121− , foldMany
122− , refoldMany
123109 , foldSequence
124110 , foldIterateM
125− , refoldIterateM
126111
127112 -- ** Parsing
128113 -- | Parsing is opposite to flattening. 'parseMany' is dual to concatMap or
@@ -139,7 +124,6 @@ module Streamly.Internal.Data.Stream.Nesting
139124
140125 -- ** Grouping
141126 -- | Group segments of a stream and fold. Special case of parsing.
142− , groupsOf
143127 , groupsBy
144128 , groupsWhile
145129 , groupsRollingBy
@@ -147,9 +131,8 @@ module Streamly.Internal.Data.Stream.Nesting
147131 -- ** Splitting
148132 -- | A special case of parsing.
149133 , wordsBy
150− , splitOnSeq
151− , splitOnSuffixSeq
152− , sliceOnSuffix
134+ , splitOnSeq -- XXX splitOnSeg
135+ , splitOnSuffixSeq -- XXX splitOnSegSuffix, splitOnTrailer
153136
154137 -- XXX Implement these as folds or parsers instead.
155138 , splitOnSuffixSeqAny
@@ -158,9 +141,8 @@ module Streamly.Internal.Data.Stream.Nesting
158141
159142 -- * Transform (Nested Containers)
160143 -- | Opposite to compact in ArrayStream
161− , splitInnerBy
162− , splitInnerBySuffix
163− , intersectBySorted
144+ , splitInnerBy -- XXX innerSplitOn
145+ , splitInnerBySuffix -- XXX innerSplitOnSuffix
164146
165147 -- * Reduce By Streams
166148 , dropPrefix
@@ -182,12 +164,9 @@ import Fusion.Plugin.Types (Fuse(..))
182164import GHC.Types (SPEC(..))
183165
184166import Streamly.Internal.Data.Array.Type (Array(..))
185−import Streamly.Internal.Data.Fold.Step (Step(..))
186167import Streamly.Internal.Data.Fold.Type (Fold(..))
187168import Streamly.Internal.Data.Parser (ParseError(..))
188−import Streamly.Internal.Data.Refold.Type (Refold(..))
189169import Streamly.Internal.Data.SVar.Type (adaptState)
190−import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
191170import Streamly.Internal.Data.Unbox (Unbox, sizeOf)
192171import Streamly.Internal.Data.Unfold.Type (Unfold(..))
193172
@@ -613,49 +592,6 @@ mergeFstBy :: -- Monad m =>
613592mergeFstBy _f _m1 _m2 = undefined
614593 -- fromStreamK $ D.mergeFstBy f (toStreamD m1) (toStreamD m2)
615594
616−-------------------------------------------------------------------------------
617−-- Intersection of sorted streams
618−-------------------------------------------------------------------------------
619−
620−-- Assuming the streams are sorted in ascending order
621−{-# INLINE_NORMAL intersectBySorted #-}
622−intersectBySorted :: Monad m
623− => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
624−intersectBySorted cmp (Stream stepa ta) (Stream stepb tb) =
625− Stream step
626− ( ta -- left stream state
627− , tb -- right stream state
628− , Nothing -- left value
629− , Nothing -- right value
630− )
631−
632− where
633−
634− {-# INLINE_LATE step #-}
635− -- step 1, fetch the first value
636− step gst (sa, sb, Nothing, b) = do
637− r <- stepa gst sa
638− return $ case r of
639− Yield a sa' -> Skip (sa', sb, Just a, b) -- step 2/3
640− Skip sa' -> Skip (sa', sb, Nothing, b)
641− Stop -> Stop
642−
643− -- step 2, fetch the second value
644− step gst (sa, sb, a@(Just _), Nothing) = do
645− r <- stepb gst sb
646− return $ case r of
647− Yield b sb' -> Skip (sa, sb', a, Just b) -- step 3
648− Skip sb' -> Skip (sa, sb', a, Nothing)
649− Stop -> Stop
650−
651− -- step 3, compare the two values
652− step _ (sa, sb, Just a, Just b) = do
653− let res = cmp a b
654− return $ case res of
655− GT -> Skip (sa, sb, Just a, Nothing) -- step 2
656− LT -> Skip (sa, sb, Nothing, Just b) -- step 1
657− EQ -> Yield a (sa, sb, Nothing, Just b) -- step 1
658−
659595------------------------------------------------------------------------------
660596-- Combine N Streams - unfoldMany
661597------------------------------------------------------------------------------
@@ -1242,6 +1178,7 @@ foldSequence _f _m = undefined
12421178data FIterState s f m a b
12431179 = FIterInit s f
12441180 | forall fs. FIterStream s (fs -> a -> m (FL.Step fs b)) fs (fs -> m b)
1181+ (fs -> m b)
12451182 | FIterYield b (FIterState s f m a b)
12461183 | FIterStop
12471184
@@ -1269,101 +1206,30 @@ foldIterateM func seed0 (Stream step state) =
12691206 where
12701207
12711208 {-# INLINE iterStep #-}
1272− iterStep from st fstep extract = do
1209+ iterStep from st fstep extract final = do
12731210 res <- from
12741211 return
12751212 $ Skip
12761213 $ case res of
1277− FL.Partial fs -> FIterStream st fstep fs extract
1214+ FL.Partial fs -> FIterStream st fstep fs extract final
12781215 FL.Done fb -> FIterYield fb $ FIterInit st (return fb)
12791216
12801217 {-# INLINE_LATE stepOuter #-}
12811218 stepOuter _ (FIterInit st seed) = do
1282− (FL.Fold fstep initial extract) <- seed >>= func
1283− iterStep initial st fstep extract
1284− stepOuter gst (FIterStream st fstep fs extract) = do
1219+ (FL.Fold fstep initial extract final) <- seed >>= func
1220+ iterStep initial st fstep extract final
1221+ stepOuter gst (FIterStream st fstep fs extract final) = do
12851222 r <- step (adaptState gst) st
12861223 case r of
12871224 Yield x s -> do
1288− iterStep (fstep fs x) s fstep extract
1289− Skip s -> return $ Skip $ FIterStream s fstep fs extract
1225+ iterStep (fstep fs x) s fstep extract final
1226+ Skip s -> return $ Skip $ FIterStream s fstep fs extract final
12901227 Stop -> do
1291− b <- extract fs
1228+ b <- final fs
12921229 return $ Skip $ FIterYield b FIterStop
12931230 stepOuter _ (FIterYield a next) = return $ Yield a next
12941231 stepOuter _ FIterStop = return Stop
12951232
1296−{-# ANN type CIterState Fuse #-}
1297−data CIterState s f fs b
1298− = CIterInit s f
1299− | CIterConsume s fs
1300− | CIterYield b (CIterState s f fs b)
1301− | CIterStop
1302−
1303−-- | Like 'foldIterateM' but using the 'Refold' type instead. This could be
1304−-- much more efficient due to stream fusion.
1305−--
1306−-- /Internal/
1307−{-# INLINE_NORMAL refoldIterateM #-}
1308−refoldIterateM ::
1309− Monad m => Refold m b a b -> m b -> Stream m a -> Stream m b
1310−refoldIterateM (Refold fstep finject fextract) initial (Stream step state) =
1311− Stream stepOuter (CIterInit state initial)
1312−
1313− where
1314−
1315− {-# INLINE iterStep #-}
1316− iterStep st action = do
1317− res <- action
1318− return
1319− $ Skip
1320− $ case res of
1321− FL.Partial fs -> CIterConsume st fs
1322− FL.Done fb -> CIterYield fb $ CIterInit st (return fb)
1323−
1324− {-# INLINE_LATE stepOuter #-}
1325− stepOuter _ (CIterInit st action) = do
1326− iterStep st (action >>= finject)
1327− stepOuter gst (CIterConsume st fs) = do
1328− r <- step (adaptState gst) st
1329− case r of
1330− Yield x s -> iterStep s (fstep fs x)
1331− Skip s -> return $ Skip $ CIterConsume s fs
1332− Stop -> do
1333− b <- fextract fs
1334− return $ Skip $ CIterYield b CIterStop
1335− stepOuter _ (CIterYield a next) = return $ Yield a next
1336− stepOuter _ CIterStop = return Stop
1337−
1338−-- "n" elements at the end are dropped by the fold.
1339−{-# INLINE sliceBy #-}
1340−sliceBy :: Monad m => Fold m a Int -> Int -> Refold m (Int, Int) a (Int, Int)
1341−sliceBy (Fold step1 initial1 extract1) n = Refold step inject extract
1342−
1343− where
1344−
1345− inject (i, len) = do
1346− r <- initial1
1347− return $ case r of
1348− Partial s -> Partial $ Tuple' (i + len + n) s
1349− Done l -> Done (i, l)
1350−
1351− step (Tuple' i s) x = do
1352− r <- step1 s x
1353− return $ case r of
1354− Partial s1 -> Partial $ Tuple' i s1
1355− Done len -> Done (i, len)
1356−
1357− extract (Tuple' i s) = (i,) <$> extract1 s
1358−
1359−{-# INLINE sliceOnSuffix #-}
1360−sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
1361−sliceOnSuffix predicate =
1362− -- Scan the stream with the given refold
1363− refoldIterateM
1364− (sliceBy (FL.takeEndBy_ predicate FL.length) 1)
1365− (return (-1, 0))
1366−
13671233------------------------------------------------------------------------------
13681234-- Parsing
13691235------------------------------------------------------------------------------
@@ -1916,7 +1782,7 @@ groupsWhile :: Monad m
19161782{-
19171783groupsWhile eq fld = parseMany (PRD.groupBy eq fld)
19181784-}
1919−groupsWhile cmp (Fold fstep initial done) (Stream step state) =
1785+groupsWhile cmp (Fold fstep initial _ final) (Stream step state) =
19201786 Stream stepOuter (GroupingInit state)
19211787
19221788 where
@@ -1939,7 +1805,7 @@ groupsWhile cmp (Fold fstep initial done) (Stream step state) =
19391805 FL.Partial fs1 -> go SPEC x s fs1
19401806 FL.Done b -> return $ Yield b (GroupingInit s)
19411807 Skip s -> return $ Skip $ GroupingDo s fs
1942− Stop -> return Stop
1808+ Stop -> final fs >> return Stop
19431809
19441810 where
19451811
@@ -1954,10 +1820,12 @@ groupsWhile cmp (Fold fstep initial done) (Stream step state) =
19541820 FL.Partial fs1 -> go SPEC prev s fs1
19551821 FL.Done b -> return $ Yield b (GroupingInit s)
19561822 else do
1957− r <- done acc
1823+ r <- final acc
19581824 return $ Yield r (GroupingInitWith s x)
19591825 Skip s -> go SPEC prev s acc
1960− Stop -> done acc >>= \r -> return $ Yield r GroupingDone
1826+ Stop -> do
1827+ r <- final acc
1828+ return $ Yield r GroupingDone
19611829 stepOuter _ (GroupingInitWith st x) = do
19621830 res <- initial
19631831 return
@@ -1984,10 +1852,12 @@ groupsWhile cmp (Fold fstep initial done) (Stream step state) =
19841852 FL.Partial fs1 -> go SPEC s fs1
19851853 FL.Done b -> return $ Yield b (GroupingInit s)
19861854 else do
1987− r <- done acc
1855+ r <- final acc
19881856 return $ Yield r (GroupingInitWith s x)
19891857 Skip s -> go SPEC s acc
1990− Stop -> done acc >>= \r -> return $ Yield r GroupingDone
1858+ Stop -> do
1859+ r <- final acc
1860+ return $ Yield r GroupingDone
19911861 stepOuter _ (GroupingYield _ _) = error "groupsWhile: Unreachable"
19921862 stepOuter _ GroupingDone = return Stop
19931863
@@ -2009,7 +1879,7 @@ groupsRollingBy :: Monad m
20091879{-
20101880groupsRollingBy eq fld = parseMany (PRD.groupByRolling eq fld)
20111881-}
2012−groupsRollingBy cmp (Fold fstep initial done) (Stream step state) =
1882+groupsRollingBy cmp (Fold fstep initial _ final) (Stream step state) =
20131883 Stream stepOuter (GroupingInit state)
20141884
20151885 where
@@ -2032,7 +1902,7 @@ groupsRollingBy cmp (Fold fstep initial done) (Stream step state) =
20321902 FL.Partial fs1 -> go SPEC x s fs1
20331903 FL.Done fb -> return $ Yield fb (GroupingInit s)
20341904 Skip s -> return $ Skip $ GroupingDo s fs
2035− Stop -> return Stop
1905+ Stop -> final fs >> return Stop
20361906
20371907 where
20381908
@@ -2047,10 +1917,12 @@ groupsRollingBy cmp (Fold fstep initial done) (Stream step state) =
20471917 FL.Partial fs1 -> go SPEC x s fs1
20481918 FL.Done b -> return $ Yield b (GroupingInit s)
20491919 else do
2050− r <- done acc
1920+ r <- final acc
20511921 return $ Yield r (GroupingInitWith s x)
20521922 Skip s -> go SPEC prev s acc
2053− Stop -> done acc >>= \r -> return $ Yield r GroupingDone
1923+ Stop -> do
1924+ r <- final acc
1925+ return $ Yield r GroupingDone
20541926 stepOuter _ (GroupingInitWith st x) = do
20551927 res <- initial
20561928 return
@@ -2081,7 +1953,7 @@ groupsRollingBy cmp (Fold fstep initial done) (Stream step state) =
20811953 FL.Done b -> return $ Yield b (GroupingInit st)
20821954 else do
20831955 {-
2084− r <- done acc
1956+ r <- final acc
20851957 return $ Yield r (GroupingInitWith s x)
20861958 -}
20871959 -- The code above does not let groupBy fuse. We use the
@@ -2090,14 +1962,16 @@ groupsRollingBy cmp (Fold fstep initial done) (Stream step state) =
20901962 -- GroupingInitWith state here to help GHC with stream
20911963 -- fusion.
20921964 result <- initial
2093− r <- done acc
1965+ r <- final acc
20941966 return
20951967 $ Yield r
20961968 $ case result of
20971969 FL.Partial fsi -> GroupingDoWith s fsi x
20981970 FL.Done b -> GroupingYield b (GroupingInit s)
20991971 Skip s -> go SPEC prev s acc
2100− Stop -> done acc >>= \r -> return $ Yield r GroupingDone
1972+ Stop -> do
1973+ r <- final acc
1974+ return $ Yield r GroupingDone
21011975 stepOuter _ (GroupingYield r next) = return $ Yield r next
21021976 stepOuter _ GroupingDone = return Stop
21031977
@@ -2119,7 +1993,7 @@ data WordsByState st fs b
21191993
21201994{-# INLINE_NORMAL wordsBy #-}
21211995wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
2122−wordsBy predicate (Fold fstep initial done) (Stream step state) =
1996+wordsBy predicate (Fold fstep initial _ final) (Stream step state) =
21231997 Stream stepOuter (WordsByInit state)
21241998
21251999 where
@@ -2149,7 +2023,7 @@ wordsBy predicate (Fold fstep initial done) (Stream step state) =
21492023 FL.Partial fs1 -> go SPEC s fs1
21502024 FL.Done b -> return $ Yield b (WordsByInit s)
21512025 Skip s -> return $ Skip $ WordsByDo s fs
2152− Stop -> return Stop
2026+ Stop -> final fs >> return Stop
21532027
21542028 where
21552029
@@ -2160,7 +2034,7 @@ wordsBy predicate (Fold fstep initial done) (Stream step state) =
21602034 if predicate x
21612035 then do
21622036 {-
2163− r <- done acc
2037+ r <- final acc
21642038 return $ Yield r (WordsByInit s)
21652039 -}
21662040 -- The above code does not fuse well. Need to check why
@@ -2169,7 +2043,7 @@ wordsBy predicate (Fold fstep initial done) (Stream step state) =
21692043 -- state always, we directly go to WordsByDo state in
21702044 -- the common case of Partial.
21712045 resi <- initial
2172− r <- done acc
2046+ r <- final acc
21732047 return
21742048 $ Yield r
21752049 $ case resi of
@@ -2181,7 +2055,9 @@ wordsBy predicate (Fold fstep initial done) (Stream step state) =
21812055 FL.Partial fs1 -> go SPEC s fs1
21822056 FL.Done b -> return $ Yield b (WordsByInit s)
21832057 Skip s -> go SPEC s acc
2184− Stop -> done acc >>= \r -> return $ Yield r WordsByDone
2058+ Stop -> do
2059+ r <- final acc
2060+ return $ Yield r WordsByDone
21852061
21862062 stepOuter _ WordsByDone = return Stop
21872063
@@ -2239,7 +2115,7 @@ splitOnSeq
22392115 -> Fold m a b
22402116 -> Stream m a
22412117 -> Stream m b
2242−splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
2118+splitOnSeq patArr (Fold fstep initial _ final) (Stream step state) =
22432119 Stream stepOuter SplitOnSeqInit
22442120
22452121 where
@@ -2322,12 +2198,12 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
23222198 r <- fstep acc x
23232199 b1 <-
23242200 case r of
2325− FL.Partial acc1 -> done acc1
2201+ FL.Partial acc1 -> final acc1
23262202 FL.Done b -> return b
23272203 let jump c = SplitOnSeqEmpty c s
23282204 in yieldProceed jump b1
23292205 Skip s -> skip (SplitOnSeqEmpty acc s)
2330− Stop -> return Stop
2206+ Stop -> final acc >> return Stop
23312207
23322208 -----------------
23332209 -- Done
@@ -2345,7 +2221,7 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
23452221 Yield x s -> do
23462222 let jump c = SplitOnSeqSingle c s pat
23472223 if pat == x
2348− then done fs >>= yieldProceed jump
2224+ then final fs >>= yieldProceed jump
23492225 else do
23502226 r <- fstep fs x
23512227 case r of
@@ -2353,7 +2229,7 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
23532229 FL.Done b -> yieldProceed jump b
23542230 Skip s -> return $ Skip $ SplitOnSeqSingle fs s pat
23552231 Stop -> do
2356− r <- done fs
2232+ r <- final fs
23572233 return $ Skip $ SplitOnSeqYield r SplitOnSeqDone
23582234
23592235 ---------------------------
@@ -2361,7 +2237,7 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
23612237 ---------------------------
23622238
23632239 stepOuter _ (SplitOnSeqWordDone 0 fs _) = do
2364− r <- done fs
2240+ r <- final fs
23652241 skip $ SplitOnSeqYield r SplitOnSeqDone
23662242 stepOuter _ (SplitOnSeqWordDone n fs wrd) = do
23672243 let old = elemMask .&. (wrd `shiftR` (elemBits * (n - 1)))
@@ -2388,7 +2264,7 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
23882264 if wrd1 .&. wordMask == wordPat
23892265 then do
23902266 let jump c = SplitOnSeqWordInit c s
2391− done fs >>= yieldProceed jump
2267+ final fs >>= yieldProceed jump
23922268 else skip $ SplitOnSeqWordLoop wrd1 s fs
23932269 else go SPEC (idx + 1) wrd1 s
23942270 Skip s -> go SPEC idx wrd s
@@ -2396,7 +2272,7 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
23962272 if idx /= 0
23972273 then skip $ SplitOnSeqWordDone idx fs wrd
23982274 else do
2399− r <- done fs
2275+ r <- final fs
24002276 skip $ SplitOnSeqYield r SplitOnSeqDone
24012277
24022278 stepOuter gst (SplitOnSeqWordLoop wrd0 st0 fs0) =
@@ -2417,7 +2293,7 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
24172293 case r of
24182294 FL.Partial fs1 -> do
24192295 if wrd1 .&. wordMask == wordPat
2420− then done fs1 >>= yieldProceed jump
2296+ then final fs1 >>= yieldProceed jump
24212297 else go SPEC wrd1 s fs1
24222298 FL.Done b -> yieldProceed jump b
24232299 Skip s -> go SPEC wrd s fs
@@ -2502,14 +2378,14 @@ splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
25022378 stepOuter _ (SplitOnSeqKRCheck fs st rb rh) = do
25032379 if RB.unsafeEqArray rb rh patArr
25042380 then do
2505− r <- done fs
2381+ r <- final fs
25062382 let rst = RB.startOf rb
25072383 jump c = SplitOnSeqKRInit 0 c st rb rst
25082384 yieldProceed jump r
25092385 else skip $ SplitOnSeqKRLoop fs st rb rh patHash
25102386
25112387 stepOuter _ (SplitOnSeqKRDone 0 fs _ _) = do
2512− r <- done fs
2388+ r <- final fs
25132389 skip $ SplitOnSeqYield r SplitOnSeqDone
25142390 stepOuter _ (SplitOnSeqKRDone n fs rb rh) = do
25152391 old <- liftIO $ peek rh
@@ -2553,7 +2429,7 @@ splitOnSuffixSeq
25532429 -> Fold m a b
25542430 -> Stream m a
25552431 -> Stream m b
2556−splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
2432+splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state) =
25572433 Stream stepOuter SplitOnSuffixSeqInit
25582434
25592435 where
@@ -2593,7 +2469,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
25932469 r <- if withSep then fstep fs x else return $ FL.Partial fs
25942470 b1 <-
25952471 case r of
2596− FL.Partial fs1 -> done fs1
2472+ FL.Partial fs1 -> final fs1
25972473 FL.Done b -> return b
25982474 yieldProceed jump b1
25992475 else do
@@ -2656,11 +2532,11 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
26562532 r <- fstep acc x
26572533 b1 <-
26582534 case r of
2659− FL.Partial fs -> done fs
2535+ FL.Partial fs -> final fs
26602536 FL.Done b -> return b
26612537 yieldProceed jump b1
26622538 Skip s -> skip (SplitOnSuffixSeqEmpty acc s)
2663− Stop -> return Stop
2539+ Stop -> final acc >> return Stop
26642540
26652541 -----------------
26662542 -- Done
@@ -2677,7 +2553,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
26772553 case res of
26782554 Yield x s -> processYieldSingle pat x s fs
26792555 Skip s -> skip $ SplitOnSuffixSeqSingleInit fs s pat
2680− Stop -> return Stop
2556+ Stop -> final fs >> return Stop
26812557
26822558 stepOuter gst (SplitOnSuffixSeqSingle fs st pat) = do
26832559 res <- step (adaptState gst) st
@@ -2685,7 +2561,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
26852561 Yield x s -> processYieldSingle pat x s fs
26862562 Skip s -> skip $ SplitOnSuffixSeqSingle fs s pat
26872563 Stop -> do
2688− r <- done fs
2564+ r <- final fs
26892565 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
26902566
26912567 ---------------------------
@@ -2693,7 +2569,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
26932569 ---------------------------
26942570
26952571 stepOuter _ (SplitOnSuffixSeqWordDone 0 fs _) = do
2696− r <- done fs
2572+ r <- final fs
26972573 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
26982574 stepOuter _ (SplitOnSuffixSeqWordDone n fs wrd) = do
26992575 let old = elemMask .&. (wrd `shiftR` (elemBits * (n - 1)))
@@ -2716,7 +2592,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
27162592 let jump c = SplitOnSuffixSeqWordInit c s
27172593 yieldProceed jump b
27182594 Skip s -> skip (SplitOnSuffixSeqWordInit fs0 s)
2719− Stop -> return Stop
2595+ Stop -> final fs0 >> return Stop
27202596
27212597 where
27222598
@@ -2734,7 +2610,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
27342610 then go SPEC (idx + 1) wrd1 s fs1
27352611 else if wrd1 .&. wordMask /= wordPat
27362612 then skip $ SplitOnSuffixSeqWordLoop wrd1 s fs1
2737− else do done fs >>= yieldProceed jump
2613+ else do final fs >>= yieldProceed jump
27382614 FL.Done b -> yieldProceed jump b
27392615 Skip s -> go SPEC idx wrd s fs
27402616 Stop -> skip $ SplitOnSuffixSeqWordDone idx fs wrd
@@ -2760,16 +2636,16 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
27602636 case r of
27612637 FL.Partial fs1 ->
27622638 if wrd1 .&. wordMask == wordPat
2763− then done fs1 >>= yieldProceed jump
2639+ then final fs1 >>= yieldProceed jump
27642640 else go SPEC wrd1 s fs1
27652641 FL.Done b -> yieldProceed jump b
27662642 Skip s -> go SPEC wrd s fs
27672643 Stop ->
27682644 if wrd .&. wordMask == wordPat
2769− then return Stop
2645+ then final fs >> return Stop
27702646 else if withSep
27712647 then do
2772− r <- done fs
2648+ r <- final fs
27732649 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
27742650 else skip $ SplitOnSuffixSeqWordDone patLen fs wrd
27752651
@@ -2791,7 +2667,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
27912667 jump c = SplitOnSuffixSeqKRInit 0 c s rb rst
27922668 yieldProceed jump b
27932669 Skip s -> skip $ SplitOnSuffixSeqKRInit idx0 fs s rb rh0
2794− Stop -> return Stop
2670+ Stop -> final fs >> return Stop
27952671
27962672 stepOuter gst (SplitOnSuffixSeqKRInit1 fs0 st0 rb rh0) = do
27972673 go SPEC 1 rh0 st0 fs0
@@ -2823,10 +2699,10 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
28232699 Stop -> do
28242700 -- do not issue a blank segment when we end at pattern
28252701 if (idx == maxIndex) && RB.unsafeEqArray rb rh patArr
2826− then return Stop
2702+ then final fs >> return Stop
28272703 else if withSep
28282704 then do
2829− r <- done fs
2705+ r <- final fs
28302706 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
28312707 else skip $ SplitOnSuffixSeqKRDone idx fs rb (RB.startOf rb)
28322708
@@ -2855,24 +2731,24 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
28552731 Skip s -> go SPEC fs s rh cksum
28562732 Stop ->
28572733 if RB.unsafeEqArray rb rh patArr
2858− then return Stop
2734+ then final fs >> return Stop
28592735 else if withSep
28602736 then do
2861− r <- done fs
2737+ r <- final fs
28622738 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
28632739 else skip $ SplitOnSuffixSeqKRDone patLen fs rb rh
28642740
28652741 stepOuter _ (SplitOnSuffixSeqKRCheck fs st rb rh) = do
28662742 if RB.unsafeEqArray rb rh patArr
28672743 then do
2868− r <- done fs
2744+ r <- final fs
28692745 let rst = RB.startOf rb
28702746 jump c = SplitOnSuffixSeqKRInit 0 c st rb rst
28712747 yieldProceed jump r
28722748 else skip $ SplitOnSuffixSeqKRLoop fs st rb rh patHash
28732749
28742750 stepOuter _ (SplitOnSuffixSeqKRDone 0 fs _ _) = do
2875− r <- done fs
2751+ r <- final fs
28762752 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
28772753 stepOuter _ (SplitOnSuffixSeqKRDone n fs rb rh) = do
28782754 old <- liftIO $ peek rh
@@ -3054,12 +2930,13 @@ splitInnerBy splitter joiner (Stream step1 state1) =
30542930-- | Performs infix separator style splitting.
30552931{-# INLINE_NORMAL splitInnerBySuffix #-}
30562932splitInnerBySuffix
3057− :: (Monad m, Eq (f a), Monoid (f a))
3058− => (f a -> m (f a, Maybe (f a))) -- splitter
2933+ :: Monad m
2934+ => (f a -> Bool) -- isEmpty?
2935+ -> (f a -> m (f a, Maybe (f a))) -- splitter
30592936 -> (f a -> f a -> m (f a)) -- joiner
30602937 -> Stream m (f a)
30612938 -> Stream m (f a)
3062−splitInnerBySuffix splitter joiner (Stream step1 state1) =
2939+splitInnerBySuffix isEmpty splitter joiner (Stream step1 state1) =
30632940 Stream step (SplitInitial state1)
30642941
30652942 where
@@ -3086,10 +2963,11 @@ splitInnerBySuffix splitter joiner (Stream step1 state1) =
30862963 Nothing -> Skip (SplitBuffering s buf')
30872964 Just x2 -> Skip (SplitYielding buf' (SplitSplitting s x2))
30882965 Skip s -> return $ Skip (SplitBuffering s buf)
3089− Stop -> return $
3090− if buf == mempty
3091− then Stop
3092− else Skip (SplitYielding buf SplitFinishing)
2966+ Stop ->
2967+ return $
2968+ if isEmpty buf
2969+ then Stop
2970+ else Skip (SplitYielding buf SplitFinishing)
30932971
30942972 step _ (SplitSplitting st buf) = do
30952973 (x1, mx2) <- splitter buf
@@ -3108,6 +2986,8 @@ splitInnerBySuffix splitter joiner (Stream step1 state1) =
31082986--
31092987-- Space: @O(1)@
31102988--
2989+-- See also stripPrefix.
2990+--
31112991-- /Unimplemented/
31122992{-# INLINE dropPrefix #-}
31132993dropPrefix ::
@@ -3120,6 +3000,8 @@ dropPrefix = error "Not implemented yet!"
31203000--
31213001-- Space: @O(n)@ where n is the length of the infix.
31223002--
3003+-- See also stripInfix.
3004+--
31233005-- /Unimplemented/
31243006{-# INLINE dropInfix #-}
31253007dropInfix ::
@@ -3132,6 +3014,8 @@ dropInfix = error "Not implemented yet!"
31323014--
31333015-- Space: @O(n)@ where n is the length of the suffix.
31343016--
3017+-- See also stripSuffix.
3018+--
31353019-- /Unimplemented/
31363020{-# INLINE dropSuffix #-}
31373021dropSuffix ::
core/src/Streamly/Internal/Data/Stream/Top.hs+243−116
@@ -8,38 +8,73 @@
88-- Portability : GHC
99--
1010-- Top level module that can depend on all other lower level Stream modules.
11+--
12+-- Design notes:
13+--
14+-- The order of arguments in the join operations should ideally be opposite. It
15+-- should be such that the infinite stream is the last one. The transformation
16+-- should be on the last argument, so if you curry the functions with all other
17+-- arguments we get a @Stream -> Stream@ function. The first stream argument
18+-- may be considered as a config or modifier for the operation.
19+--
20+-- Benefit of changing the order is that we get a more intuitive Stream ->
21+-- Stream transformation after currying all other arguments. The inner loop
22+-- streams become arguments for the transformation, more like local modifiers
23+-- for the global outer stream as the last argument. Thus we can continue using
24+-- transformations on the outer stream in a composed pipeline. Otherwise we can
25+-- use flip to flip the order.
26+--
27+-- The fact that the inner stream can be used in the loop multiple times also
28+-- tells that this is not the real effectful stream, it is more like a pure
29+-- stream or an array. In fact we may consider using an Identity streams as
30+-- inner streams in which case these functions will not look nice.
31+--
32+-- Downsides:
33+--
34+-- * Maybe less intuitive to think about, because we usually think the first
35+-- stream as the outer loop and second as the inner.
36+-- * Zip and merge operations will continue using the opposite order.
37+-- * Need to change the order of cross, crossWith operations as well
38+-- * It will be inconsistent with Data.List. The functions cannot be used as
39+-- intuitive operators.
40+--
41+-- The choice is similar to concatMap vs bind. concatMap is pipeline
42+-- composition friendly but bind is user intuition friendly. Another option is
43+-- to have other functions with a different argument order e.g. flippedCross
44+-- instead of cross.
45+--
46+-- If we change the order we have to make sure that we have a consistent
47+-- convention for set-like and the cross join operations.
1148
1249module Streamly.Internal.Data.Stream.Top
1350 (
14− -- * Transformation
15− -- ** Sampling
51+ -- * Sampling
1652 -- | Value agnostic filtering.
1753 strideFromThen
1854
19− -- * Nesting
20− -- ** Set like operations
21− -- | These are not exactly set operations because streams are not
22− -- necessarily sets, they may have duplicated elements. These operations
23− -- are generic i.e. they work on streams of unconstrained types, therefore,
24− -- they have quadratic performance characterstics. For better performance
25− -- using Set structures see the Streamly.Internal.Data.Stream.Container
26− -- module.
27− , filterInStreamGenericBy
28− , deleteInStreamGenericBy
29− , unionWithStreamGenericBy
30−
31− -- ** Set like operations on sorted streams
32− , filterInStreamAscBy
33− , deleteInStreamAscBy
34− , unionWithStreamAscBy
35−
36− -- ** Join operations
37− , joinInnerGeneric
38−
39− -- * Joins on sorted stream
40− , joinInnerAscBy
41− , joinLeftAscBy
42− , joinOuterAscBy
55+ -- * Straight Joins
56+ -- | These are set-like operations but not exactly set operations because
57+ -- streams are not necessarily sets, they may have duplicated elements.
58+ -- These operations are generic i.e. they work on streams of unconstrained
59+ -- types, therefore, they have quadratic performance characterstics. For
60+ -- better performance using Set or Map structures see the
61+ -- Streamly.Internal.Data.Stream.Container module.
62+ , intersectBy
63+ , deleteFirstsBy
64+ , unionBy
65+
66+ -- Set like operations on sorted streams
67+ , sortedIntersectBy
68+ , sortedDeleteFirstsBy
69+ , sortedUnionBy
70+
71+ -- * Cross Joins
72+ , innerJoin
73+
74+ -- Joins on sorted stream
75+ , innerSortedJoin
76+ , leftSortedJoin
77+ , outerSortedJoin
4378 )
4479where
4580
@@ -48,8 +83,7 @@ where
4883import Control.Monad.IO.Class (MonadIO(..))
4984import Data.IORef (newIORef, readIORef, modifyIORef')
5085import Streamly.Internal.Data.Fold.Type (Fold)
51−import Streamly.Internal.Data.Stream.Common ()
52−import Streamly.Internal.Data.Stream.Type (Stream, cross)
86+import Streamly.Internal.Data.Stream.Type (Stream(..), Step(..), cross)
5387
5488import qualified Data.List as List
5589import qualified Streamly.Internal.Data.Fold as Fold
@@ -111,29 +145,40 @@ strideFromThen offset stride =
111145-- binary search if we have an Ord instance or Ordering returning function. The
112146-- time complexity would then become (m x log n).
113147
114−-- | Like 'cross' but emits only those tuples where @a == b@ using the
115−-- supplied equality predicate.
148+-- | Like 'cross' but emits only those tuples where @a == b@ using the supplied
149+-- equality predicate. This is essentially a @cross intersection@ of two
150+-- streams.
116151--
117152-- Definition:
118153--
119−-- >>> joinInnerGeneric eq s1 s2 = Stream.filter (\(a, b) -> a `eq` b) $ Stream.cross s1 s2
154+-- >>> innerJoin eq s1 s2 = Stream.filter (\(a, b) -> a `eq` b) $ Stream.cross s1 s2
155+--
156+-- The second (inner) stream must be finite. Moreover, it must be either pure
157+-- or capable of multiple evaluations. If not then the caller should cache it
158+-- in an 'Data.Array.Array', if the type does not have an 'Unbox' instance then
159+-- use the Generic 'Data.Array.Generic.Array'. Convert the array to stream
160+-- before calling this function. Caching may also improve performance if the
161+-- stream is expensive to evaluate.
120162--
121−-- You should almost always prefer @joinInnerOrd@ over 'joinInnerGeneric' if
122−-- possible. @joinInnerOrd@ is an order of magnitude faster but may take more
123−-- space for caching the second stream.
163+-- If you care about performance this function should be your last choice among
164+-- all inner joins. 'Streamly.Internal.Data.Unfold.innerJoin' is a much faster
165+-- fused alternative. 'innerSortedJoin' is a faster alternative when streams
166+-- are sorted. 'innerOrdJoin' is an order of magnitude faster alternative when
167+-- the type has an 'Ord' instance.
124168--
125−-- See 'Streamly.Internal.Data.Unfold.joinInnerGeneric' for a much faster fused
126−-- alternative.
169+-- Note: Conceptually, this is a commutative operation. Result includes all the
170+-- elements from the left and the right stream. The order of streams can be
171+-- changed without affecting results, except for the ordering within the tuple.
127172--
128173-- Time: O(m x n)
129174--
130175-- /Pre-release/
131−{-# INLINE joinInnerGeneric #-}
132−joinInnerGeneric :: Monad m =>
176+{-# INLINE innerJoin #-}
177+innerJoin :: Monad m =>
133178 (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, b)
134−joinInnerGeneric eq s1 s2 = Stream.filter (\(a, b) -> a `eq` b) $ cross s1 s2
179+innerJoin eq s1 s2 = Stream.filter (\(a, b) -> a `eq` b) $ cross s1 s2
135180{-
136−joinInnerGeneric eq s1 s2 = do
181+innerJoin eq s1 s2 = do
137182 -- ConcatMap works faster than bind
138183 Stream.concatMap (\a ->
139184 Stream.concatMap (\b ->
@@ -144,44 +189,44 @@ joinInnerGeneric eq s1 s2 = do
144189 ) s1
145190-}
146191
147−-- | A more efficient 'joinInner' for sorted streams.
192+-- | A more efficient 'innerJoin' for sorted streams.
148193--
149194-- Space: O(1)
150195--
151196-- Time: O(m + n)
152197--
153198-- /Unimplemented/
154−{-# INLINE joinInnerAscBy #-}
155−joinInnerAscBy ::
199+{-# INLINE innerSortedJoin #-}
200+innerSortedJoin ::
156201 (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, b)
157−joinInnerAscBy = undefined
202+innerSortedJoin = undefined
158203
159−-- | A more efficient 'joinLeft' for sorted streams.
204+-- | A more efficient 'leftJoin' for sorted streams.
160205--
161206-- Space: O(1)
162207--
163208-- Time: O(m + n)
164209--
165210-- /Unimplemented/
166−{-# INLINE joinLeftAscBy #-}
167−joinLeftAscBy :: -- Monad m =>
211+{-# INLINE leftSortedJoin #-}
212+leftSortedJoin :: -- Monad m =>
168213 (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
169−joinLeftAscBy _eq _s1 _s2 = undefined
214+leftSortedJoin _eq _s1 _s2 = undefined
170215
171−-- | A more efficient 'joinOuter' for sorted streams.
216+-- | A more efficient 'outerJoin' for sorted streams.
172217--
173218-- Space: O(1)
174219--
175220-- Time: O(m + n)
176221--
177222-- /Unimplemented/
178−{-# INLINE joinOuterAscBy #-}
179−joinOuterAscBy :: -- Monad m =>
223+{-# INLINE outerSortedJoin #-}
224+outerSortedJoin :: -- Monad m =>
180225 (a -> b -> Ordering)
181226 -> Stream m a
182227 -> Stream m b
183228 -> Stream m (Maybe a, Maybe b)
184−joinOuterAscBy _eq _s1 _s2 = undefined
229+outerSortedJoin _eq _s1 _s2 = undefined
185230
186231------------------------------------------------------------------------------
187232-- Set operations (special joins)
@@ -191,8 +236,8 @@ joinOuterAscBy _eq _s1 _s2 = undefined
191236-- the best would be to use an Array with linear search. If the second stream
192237-- is sorted we can also use a binary search, using Ord constraint.
193238
194−-- | Keep only those elements in the second stream that are present in the
195−-- first stream too. The first stream is folded to a container using the
239+-- | Keep only those elements in the first stream that are present in the
240+-- second stream too. The second stream is folded to a container using the
196241-- supplied fold and then the elements in the container are looked up using the
197242-- supplied lookup function.
198243--
@@ -207,23 +252,30 @@ filterStreamWith :: Monad m =>
207252filterStreamWith fld member s1 s2 =
208253 Stream.concatEffect
209254 $ do
210− xs <- Stream.fold fld s1
211− return $ Stream.filter (`member` xs) s2
255+ xs <- Stream.fold fld s2
256+ return $ Stream.filter (`member` xs) s1
212257
213−-- | 'filterInStreamGenericBy' retains only those elements in the second stream that
214−-- are present in the first stream.
215−--
216−-- >>> Stream.fold Fold.toList $ Stream.filterInStreamGenericBy (==) (Stream.fromList [1,2,2,4]) (Stream.fromList [2,1,1,3])
217−-- [2,1,1]
218−--
219−-- >>> Stream.fold Fold.toList $ Stream.filterInStreamGenericBy (==) (Stream.fromList [2,1,1,3]) (Stream.fromList [1,2,2,4])
220−-- [1,2,2]
258+-- XXX instead of folding the second stream to a list we could use it directly.
259+-- If the user wants they can generate the stream from an array and also call
260+-- uniq or nub on it. We can provide a convenience Stream -> Stream to cache
261+-- a finite stream in an array and serve it from the cache. The user can decide
262+-- what is best based on the context. They can also choose to use a boxed or
263+-- unboxed array for caching. To force caching we can make the second stream
264+-- monad type Identity. But that may be less flexible. One option is to use
265+-- cachedIntersectBy etc for automatic caching.
266+
267+-- | 'intersectBy' returns a subsequence of the first stream which intersects
268+-- with the second stream. Note that this is not a commutative operation unlike
269+-- a set intersection, because of duplicate elements in the stream the order of
270+-- the streams matters. This is similar to 'Data.List.intersectBy'. Note that
271+-- intersectBy is a special case of 'innerJoin'.
221272--
222−-- Similar to the list intersectBy operation but with the stream argument order
223−-- flipped.
273+-- >>> f s1 s2 = Stream.fold Fold.toList $ Stream.intersectBy (==) (Stream.fromList s1) (Stream.fromList s2)
274+-- >>> f [1,3,4,4,5]) [2,3,4,5,5]
275+-- [3,4,4,5]
224276--
225−-- The first stream must be finite and must not block. Second stream is
226−-- processed only after the first stream is fully realized.
277+-- First stream can be infinite, the second stream must be finite and must be
278+-- capable of multiple evaluations.
227279--
228280-- Space: O(n) where @n@ is the number of elements in the second stream.
229281--
@@ -231,46 +283,87 @@ filterStreamWith fld member s1 s2 =
231283-- @n@ is the number of elements in the second stream.
232284--
233285-- /Pre-release/
234−{-# INLINE filterInStreamGenericBy #-}
235−filterInStreamGenericBy :: Monad m =>
286+{-# INLINE intersectBy #-}
287+intersectBy :: Monad m =>
236288 (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
237−filterInStreamGenericBy eq =
289+intersectBy eq =
238290 -- XXX Use an (unboxed) array instead.
239291 filterStreamWith
240292 (Fold.scanMaybe (Fold.uniqBy eq) Fold.toListRev)
241293 (List.any . eq)
242294
243−-- | Like 'filterInStreamGenericBy' but assumes that the input streams are sorted in
295+-------------------------------------------------------------------------------
296+-- Intersection of sorted streams
297+-------------------------------------------------------------------------------
298+
299+-- XXX The sort order is not important as long both the streams have the same
300+-- sort order. We need to move only in one direction in each stream.
301+-- XXX Fix the argument order to use the same behavior as intersectBy.
302+
303+-- | Like 'intersectBy' but assumes that the input streams are sorted in
244304-- ascending order. To use it on streams sorted in descending order pass an
245305-- inverted comparison function returning GT for less than and LT for greater
246306-- than.
247307--
308+-- Both streams can be infinite.
309+--
248310-- Space: O(1)
249311--
250312-- Time: O(m+n)
251313--
252314-- /Pre-release/
253−{-# INLINE filterInStreamAscBy #-}
254−filterInStreamAscBy :: Monad m =>
315+{-# INLINE_NORMAL sortedIntersectBy #-}
316+sortedIntersectBy :: Monad m =>
255317 (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
256−filterInStreamAscBy eq s1 s2 = Stream.intersectBySorted eq s2 s1
318+sortedIntersectBy cmp (Stream stepa ta) (Stream stepb tb) =
319+ Stream step
320+ ( ta -- left stream state
321+ , tb -- right stream state
322+ , Nothing -- left value
323+ , Nothing -- right value
324+ )
257325
258−-- | Delete all elements of the first stream from the seconds stream. If an
259−-- element occurs multiple times in the first stream as many occurrences of it
260−-- are deleted from the second stream.
261−--
262−-- >>> Stream.fold Fold.toList $ Stream.deleteInStreamGenericBy (==) (Stream.fromList [1,2,3]) (Stream.fromList [1,2,2])
263−-- [2]
326+ where
327+
328+ {-# INLINE_LATE step #-}
329+ -- step 1, fetch the first value
330+ step gst (sa, sb, Nothing, b) = do
331+ r <- stepa gst sa
332+ return $ case r of
333+ Yield a sa' -> Skip (sa', sb, Just a, b) -- step 2/3
334+ Skip sa' -> Skip (sa', sb, Nothing, b)
335+ Stop -> Stop
336+
337+ -- step 2, fetch the second value
338+ step gst (sa, sb, a@(Just _), Nothing) = do
339+ r <- stepb gst sb
340+ return $ case r of
341+ Yield b sb' -> Skip (sa, sb', a, Just b) -- step 3
342+ Skip sb' -> Skip (sa, sb', a, Nothing)
343+ Stop -> Stop
344+
345+ -- step 3, compare the two values
346+ step _ (sa, sb, Just a, Just b) = do
347+ let res = cmp a b
348+ return $ case res of
349+ GT -> Skip (sa, sb, Just a, Nothing) -- step 2
350+ LT -> Skip (sa, sb, Nothing, Just b) -- step 1
351+ EQ -> Yield a (sa, sb, Nothing, Just b) -- step 1
352+
353+-- | Returns a subsequence of the first stream, deleting first occurrences of
354+-- those elements that are present in the second stream. Note that this is not
355+-- a commutative operation. This is similar to the 'Data.List.deleteFirstsBy'.
264356--
265−-- The following laws hold:
357+-- >>> f xs ys = Stream.fold Fold.toList $ Stream.deleteFirstsBy (==) (Stream.fromList xs) (Stream.fromList ys)
358+-- >>> f [1,2,2,3,3,5] [1,2,2,3,4]
359+-- [2,3,5]
266360--
267−-- > deleteInStreamGenericBy (==) s1 (s1 `append` s2) === s2
268−-- > deleteInStreamGenericBy (==) s1 (s1 `interleave` s2) === s2
361+-- The following holds:
269362--
270−-- Same as the list 'Data.List.//' operation but with argument order flipped.
363+-- > deleteFirstsBy (==) (Stream.nub s2 `append` s1) s2 === s1
364+-- > deleteFirstsBy (==) (Stream.nub s2 `interleave` s1) s2 === s1
271365--
272−-- The first stream must be finite and must not block. Second stream is
273−-- processed only after the first stream is fully realized.
366+-- First stream can be infinite, second stream must be finite.
274367--
275368-- Space: O(m) where @m@ is the number of elements in the first stream.
276369--
@@ -278,58 +371,83 @@ filterInStreamAscBy eq s1 s2 = Stream.intersectBySorted eq s2 s1
278371-- @n@ is the number of elements in the second stream.
279372--
280373-- /Pre-release/
281−{-# INLINE deleteInStreamGenericBy #-}
282−deleteInStreamGenericBy :: Monad m =>
374+{-# INLINE deleteFirstsBy #-}
375+deleteFirstsBy :: Monad m =>
283376 (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
284−deleteInStreamGenericBy eq s1 s2 =
285− Stream.concatEffect
286− $ do
287− -- This may work well if s1 is small
288− -- If s1 is big we can go through s1, deleting elements from s2 and
289− -- not emitting an element if it was successfully deleted from s2.
290− -- we will need a deleteBy that can return whether the element was
291− -- deleted or not.
292− xs <- Stream.fold Fold.toList s2
293− let f = Fold.foldl' (flip (List.deleteBy eq)) xs
294− fmap Stream.fromList $ Stream.fold f s1
377+deleteFirstsBy eq s2 s1 =
378+ -- XXX s2 can be a sorted mutable array and we can use binary
379+ -- search to find. Mark the element deleted, count the deletions
380+ -- and reconsolidate the array when a min number of elements is
381+ -- deleted.
295382
296−-- | A more efficient 'deleteInStreamGenericBy' for streams sorted in ascending order.
383+ -- XXX Use StreamK or list as second argument instead of Stream to avoid
384+ -- concatEffect?
385+ Stream.concatEffect $ do
386+ xs <- Stream.toList s1
387+ -- It reverses the list but that is fine.
388+ let del x =
389+ List.foldl' (\(ys,res) y ->
390+ if x `eq` y
391+ then (ys, True)
392+ else (x:ys, res)) ([], False)
393+ g (ys,_) x =
394+ let (ys1, deleted) = del x ys
395+ in if deleted
396+ then (ys1, Nothing)
397+ else (ys1, Just x)
398+ in return
399+ $ Stream.catMaybes
400+ $ fmap snd
401+ $ Stream.postscanl' g (xs, Nothing) s2
402+
403+-- | A more efficient 'deleteFirstsBy' for streams sorted in ascending order.
404+--
405+-- Both streams can be infinite.
297406--
298407-- Space: O(1)
299408--
300409-- /Unimplemented/
301−{-# INLINE deleteInStreamAscBy #-}
302−deleteInStreamAscBy :: -- (Monad m) =>
410+{-# INLINE sortedDeleteFirstsBy #-}
411+sortedDeleteFirstsBy :: -- (Monad m) =>
303412 (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
304−deleteInStreamAscBy _eq _s1 _s2 = undefined
413+sortedDeleteFirstsBy _eq _s1 _s2 = undefined
305414
306415-- XXX Remove the MonadIO constraint. We can just cache one stream and then
307416-- implement using differenceEqBy.
308417
309−-- | This essentially appends to the second stream all the occurrences of
310−-- elements in the first stream that are not already present in the second
311−-- stream.
418+-- | Returns the first stream appended with those unique elements from the
419+-- second stream that are not already present in the first stream. Note that
420+-- this is not a commutative operation unlike a set union, argument order
421+-- matters. The behavior is similar to the 'Data.List.unionBy'.
312422--
313423-- Equivalent to the following except that @s2@ is evaluated only once:
314424--
315−-- >>> unionWithStreamGenericBy eq s1 s2 = s2 `Stream.append` (Stream.deleteInStreamGenericBy eq s2 s1)
425+-- >>> unionBy eq s1 s2 = s1 `Stream.append` Stream.deleteFirstsBy eq s1 (Stream.nub s2)
316426--
317427-- Example:
318428--
319−-- >>> Stream.fold Fold.toList $ Stream.unionWithStreamGenericBy (==) (Stream.fromList [1,1,2,3]) (Stream.fromList [1,2,2,4])
429+-- >>> f s1 s2 = Stream.fold Fold.toList $ Stream.unionBy (==) (Stream.fromList s1) (Stream.fromList s2)
430+-- >>> f [1,2,2,4] [1,1,2,3,3]
320431-- [1,2,2,4,3]
321432--
433+-- First stream can be infinite, but second stream must be finite. Note that if
434+-- the first stream is infinite the union means just the first stream. Thus
435+-- union is useful only when both streams are finite. See 'sortedUnionBy' where
436+-- union can work on infinite streams if they are sorted.
437+--
322438-- Space: O(n)
323439--
324440-- Time: O(m x n)
325441--
326442-- /Pre-release/
327−{-# INLINE unionWithStreamGenericBy #-}
328−unionWithStreamGenericBy :: MonadIO m =>
443+{-# INLINE unionBy #-}
444+unionBy :: MonadIO m =>
329445 (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
330−unionWithStreamGenericBy eq s1 s2 =
446+unionBy eq s2 s1 =
331447 Stream.concatEffect
332448 $ do
449+ -- XXX use a rewrite rule such that if a list converted to stream
450+ -- is passed to unionBy then this becomes an identity operation.
333451 xs <- Stream.fold Fold.toList s1
334452 -- XXX we can use postscanlMAfter' instead of IORef
335453 ref <- liftIO $ newIORef $! List.nubBy eq xs
@@ -342,12 +460,21 @@ unionWithStreamGenericBy eq s1 s2 =
342460 return $ Stream.fromList xs1
343461 return $ Stream.mapM f s2 `Stream.append` s3
344462
345−-- | A more efficient 'unionWithStreamGenericBy' for sorted streams.
463+-- | A more efficient 'unionBy' for sorted streams.
464+--
465+-- Note that the behavior is different from 'unionBy'. In 'unionBy' we append
466+-- the unique elements from second stream only after exhausting the first one
467+-- whereas in sorted streams we can determine unique elements early even when
468+-- we are going through the first stream. Thus the result is an interleaving of
469+-- the two streams, merging those elements from the second stream that are not
470+-- present in the first.
346471--
347472-- Space: O(1)
348473--
474+-- Both streams can be infinite.
475+--
349476-- /Unimplemented/
350−{-# INLINE unionWithStreamAscBy #-}
351−unionWithStreamAscBy :: -- (Monad m) =>
477+{-# INLINE sortedUnionBy #-}
478+sortedUnionBy :: -- (Monad m) =>
352479 (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
353−unionWithStreamAscBy _eq _s1 _s2 = undefined
480+sortedUnionBy _eq _s1 _s2 = undefined
core/src/Streamly/Internal/Data/Stream/Transform.hs+61−35
@@ -21,8 +21,6 @@ module Streamly.Internal.Data.Stream.Transform
2121
2222 -- * Mapping
2323 -- | Stateless one-to-one maps.
24− , map
25− , mapM
2624 , sequence
2725
2826 -- * Mapping Effects
@@ -75,7 +73,7 @@ module Streamly.Internal.Data.Stream.Transform
7573 , scanMaybe
7674 , filter
7775 , filterM
78− , deleteBy
76+ , deleteBy -- deleteOnceBy?
7977 , uniqBy
8078 , uniq
8179 , prune
@@ -83,9 +81,8 @@ module Streamly.Internal.Data.Stream.Transform
8381
8482 -- * Trimming
8583 -- | Produce a subset of the stream trimmed at ends.
86− , take
87− , takeWhile
88− , takeWhileM
84+ , initNonEmpty
85+ , tailNonEmpty
8986 , takeWhileLast
9087 , takeWhileAround
9188 , drop
@@ -132,7 +129,6 @@ module Streamly.Internal.Data.Stream.Transform
132129 -- * Searching
133130 , findIndices
134131 , elemIndices
135− , slicesBy
136132
137133 -- * Rolling map
138134 -- | Map using the previous element.
@@ -300,7 +296,7 @@ data TapState fs st a
300296--
301297{-# INLINE tap #-}
302298tap :: Monad m => Fold m a b -> Stream m a -> Stream m a
303−tap (Fold fstep initial extract) (Stream step state) = Stream step' TapInit
299+tap (Fold fstep initial _ final) (Stream step state) = Stream step' TapInit
304300
305301 where
306302
@@ -323,7 +319,7 @@ tap (Fold fstep initial extract) (Stream step state) = Stream step' TapInit
323319 FL.Done _ -> TapDone s
324320 Skip s -> return $ Skip (Tapping acc s)
325321 Stop -> do
326− void $ extract acc
322+ void $ final acc
327323 return Stop
328324 step' gst (TapDone st) = do
329325 r <- step gst st
@@ -342,7 +338,7 @@ data TapOffState fs s a
342338{-# INLINE_NORMAL tapOffsetEvery #-}
343339tapOffsetEvery :: Monad m
344340 => Int -> Int -> Fold m a b -> Stream m a -> Stream m a
345−tapOffsetEvery offset n (Fold fstep initial extract) (Stream step state) =
341+tapOffsetEvery offset n (Fold fstep initial _ final) (Stream step state) =
346342 Stream step' TapOffInit
347343
348344 where
@@ -372,7 +368,7 @@ tapOffsetEvery offset n (Fold fstep initial extract) (Stream step state) =
372368 return $ Yield x next
373369 Skip s -> return $ Skip (TapOffTapping acc s count)
374370 Stop -> do
375− void $ extract acc
371+ void $ final acc
376372 return Stop
377373 step' gst (TapOffDone st) = do
378374 r <- step gst st
@@ -437,7 +433,7 @@ data ScanState s f = ScanInit s | ScanDo s !f | ScanDone
437433--
438434{-# INLINE_NORMAL postscan #-}
439435postscan :: Monad m => FL.Fold m a b -> Stream m a -> Stream m b
440−postscan (FL.Fold fstep initial extract) (Stream sstep state) =
436+postscan (FL.Fold fstep initial extract final) (Stream sstep state) =
441437 Stream step (ScanInit state)
442438
443439 where
@@ -460,13 +456,13 @@ postscan (FL.Fold fstep initial extract) (Stream sstep state) =
460456 return $ Yield b $ ScanDo s fs1
461457 FL.Done b -> return $ Yield b ScanDone
462458 Skip s -> return $ Skip $ ScanDo s fs
463− Stop -> return Stop
459+ Stop -> final fs >> return Stop
464460 step _ ScanDone = return Stop
465461
466462{-# INLINE scanWith #-}
467463scanWith :: Monad m
468464 => Bool -> Fold m a b -> Stream m a -> Stream m b
469−scanWith restart (Fold fstep initial extract) (Stream sstep state) =
465+scanWith restart (Fold fstep initial extract final) (Stream sstep state) =
470466 Stream step (ScanInit state)
471467
472468 where
@@ -489,7 +485,7 @@ scanWith restart (Fold fstep initial extract) (Stream sstep state) =
489485 case res of
490486 Yield x s -> runStep s (fstep fs x)
491487 Skip s -> return $ Skip $ ScanDo s fs
492− Stop -> return Stop
488+ Stop -> final fs >> return Stop
493489 step _ ScanDone = return Stop
494490
495491-- XXX It may be useful to have a version of scan where we can keep the
@@ -1074,6 +1070,55 @@ repeated = undefined
10741070-- Trimming
10751071------------------------------------------------------------------------------
10761072
1073+-- | init for non-empty streams, fails for empty stream case.
1074+--
1075+{-# INLINE initNonEmpty #-}
1076+initNonEmpty :: Monad m => Stream m a -> Stream m a
1077+initNonEmpty (Stream step1 state1) = Stream step (Nothing, state1)
1078+
1079+ where
1080+
1081+ step gst (Nothing, s1) = do
1082+ r <- step1 (adaptState gst) s1
1083+ return $
1084+ case r of
1085+ Yield x s -> Skip (Just x, s)
1086+ Skip s -> Skip (Nothing, s)
1087+ Stop -> error "initNonEmpty: empty Stream"
1088+
1089+ step gst (Just a, s1) = do
1090+ r <- step1 (adaptState gst) s1
1091+ return $
1092+ case r of
1093+ Yield x s -> Yield a (Just x, s)
1094+ Skip s -> Skip (Just a, s)
1095+ Stop -> Stop
1096+
1097+-- | tail for non-empty streams, fails for empty stream case.
1098+--
1099+-- See also 'tail' for a non-partial version of this function..
1100+{-# INLINE tailNonEmpty #-}
1101+tailNonEmpty :: Monad m => Stream m a -> Stream m a
1102+tailNonEmpty (Stream step1 state1) = Stream step (Nothing, state1)
1103+
1104+ where
1105+
1106+ step gst (Nothing, s1) = do
1107+ r <- step1 (adaptState gst) s1
1108+ return $
1109+ case r of
1110+ Yield x s -> Skip (Just x, s)
1111+ Skip s -> Skip (Nothing, s)
1112+ Stop -> error "tailNonEmpty: empty Stream"
1113+
1114+ step gst (Just a, s1) = do
1115+ r <- step1 (adaptState gst) s1
1116+ return $
1117+ case r of
1118+ Yield x s -> Yield x (Just x, s)
1119+ Skip s -> Skip (Just a, s)
1120+ Stop -> Stop
1121+
10771122-- | Take all consecutive elements at the end of the stream for which the
10781123-- predicate is true.
10791124--
@@ -1537,7 +1582,7 @@ reverse m = Stream step Nothing
15371582{-# INLINE reverseUnbox #-}
15381583reverseUnbox :: (MonadIO m, Unbox a) => Stream m a -> Stream m a
15391584reverseUnbox =
1540− A.flattenArraysRev -- unfoldMany A.readRev
1585+ A.concatRev -- unfoldMany A.readerRev
15411586 . fromStreamK
15421587 . K.reverse
15431588 . toStreamK
@@ -1709,25 +1754,6 @@ findIndices p (Stream step state) = Stream step' (state, 0)
17091754elemIndices :: (Monad m, Eq a) => a -> Stream m a -> Stream m Int
17101755elemIndices a = findIndices (== a)
17111756
1712−{-# INLINE_NORMAL slicesBy #-}
1713−slicesBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
1714−slicesBy p (Stream step1 state1) = Stream step (Just (state1, 0, 0))
1715−
1716− where
1717−
1718− {-# INLINE_LATE step #-}
1719− step gst (Just (st, i, len)) = i `seq` len `seq` do
1720− r <- step1 (adaptState gst) st
1721− return
1722− $ case r of
1723− Yield x s ->
1724− if p x
1725− then Yield (i, len + 1) (Just (s, i + len + 1, 0))
1726− else Skip (Just (s, i, len + 1))
1727− Skip s -> Skip (Just (s, i, len))
1728− Stop -> if len == 0 then Stop else Yield (i, len) Nothing
1729− step _ Nothing = return Stop
1730−
17311757------------------------------------------------------------------------------
17321758-- Rolling map
17331759------------------------------------------------------------------------------
core/src/Streamly/Internal/Data/Stream/Transformer.hs+2−1
@@ -11,10 +11,11 @@
1111
1212module Streamly.Internal.Data.Stream.Transformer
1313 (
14+ -- * Fold to Transformer Monad
1415 foldlT
1516 , foldrT
1617
17− -- * Transform Inner Monad
18+ -- * Inner Monad Operations
1819 , liftInner
1920 , runReaderT
2021 , usingReaderT
core/src/Streamly/Internal/Data/Stream/Type.hs+145−14
@@ -1,5 +1,11 @@
11{-# LANGUAGE CPP #-}
2+{-# LANGUAGE PatternSynonyms #-}
3+{-# LANGUAGE TypeFamilies #-}
4+-- Must come after TypeFamilies, otherwise it is re-enabled.
5+-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
6+{-# LANGUAGE NoMonoLocalBinds #-}
27{-# LANGUAGE UndecidableInstances #-}
8+{-# LANGUAGE ViewPatterns #-}
39
410-- |
511-- Module : Streamly.Internal.Data.Stream.Type
@@ -16,18 +22,18 @@
1622
1723module Streamly.Internal.Data.Stream.Type
1824 (
19− -- * The stream type
25+ -- * Type
2026 Step (..)
2127 -- XXX UnStream is exported to avoid a performance issue in some
2228 -- combinators if we use the pattern synonym "Stream".
2329 , Stream (Stream, UnStream)
2430
25− -- * CrossStream type wrapper
31+ -- * CrossStream
2632 , CrossStream
2733 , unCross
2834 , mkCross
2935
30− -- * Conversion to StreamK
36+ -- * To StreamK
3137 , fromStreamK
3238 , toStreamK
3339
@@ -85,6 +91,7 @@ module Streamly.Internal.Data.Stream.Type
8591
8692 -- * Combining Two Streams
8793 -- ** Zipping
94+ -- | Zip corresponding elements of two streams.
8895 , zipWithM
8996 , zipWith
9097
@@ -100,6 +107,8 @@ module Streamly.Internal.Data.Stream.Type
100107 , unfoldMany
101108
102109 -- * Concat
110+ -- | Generate streams by mapping a stream generator on each element of an
111+ -- input stream, append the resulting streams and flatten.
103112 , concatEffect
104113 , concatMap
105114 , concatMapM
@@ -123,14 +132,22 @@ module Streamly.Internal.Data.Stream.Type
123132 , foldManyPost
124133 , groupsOf
125134 , refoldMany
135+ , refoldIterateM
126136
127137 -- * Fold Iterate
128138 , reduceIterateBfs
129139 , foldIterateBfs
130140
141+ -- * Splitting
142+ , indexOnSuffix
143+
131144 -- * Multi-stream folds
145+ -- | These should probably be expressed using zipping operations.
132146 , eqBy
133147 , cmpBy
148+
149+ -- * Deprecated
150+ , sliceOnSuffix
134151 )
135152where
136153
@@ -162,6 +179,7 @@ import Streamly.Internal.Data.Maybe.Strict (Maybe'(..), toMaybe)
162179import Streamly.Internal.Data.Refold.Type (Refold(..))
163180import Streamly.Internal.Data.Stream.Step (Step (..))
164181import Streamly.Internal.Data.SVar.Type (State, adaptState, defState)
182+import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
165183import Streamly.Internal.Data.Unfold.Type (Unfold(..))
166184
167185import qualified Streamly.Internal.Data.Fold.Type as FL hiding (foldr)
@@ -212,6 +230,9 @@ pattern Stream step state <- (unShare -> UnStream step state)
212230nilM :: Applicative m => m b -> Stream m a
213231nilM m = Stream (\_ _ -> m $> Stop) ()
214232
233+
234+infixr 5 `consM`
235+
215236-- | Like 'cons' but fuses an effect instead of a pure value.
216237{-# INLINE_NORMAL consM #-}
217238consM :: Applicative m => m a -> Stream m a -> Stream m a
@@ -398,7 +419,7 @@ toStreamK (Stream step state) = go state
398419{-# INLINE_NORMAL foldEither #-}
399420foldEither :: Monad m =>
400421 Fold m a b -> Stream m a -> m (Either (Fold m a b) (b, Stream m a))
401−foldEither (Fold fstep begin done) (UnStream step state) = do
422+foldEither (Fold fstep begin done final) (UnStream step state) = do
402423 res <- begin
403424 case res of
404425 FL.Partial fs -> go SPEC fs state
@@ -416,7 +437,9 @@ foldEither (Fold fstep begin done) (UnStream step state) = do
416437 FL.Done b -> return $! Right (b, Stream step s)
417438 FL.Partial fs1 -> go SPEC fs1 s
418439 Skip s -> go SPEC fs s
419− Stop -> return $! Left (Fold fstep (return $ FL.Partial fs) done)
440+ Stop ->
441+ let f = Fold fstep (return $ FL.Partial fs) done final
442+ in return $! Left f
420443
421444-- | Like 'fold' but also returns the remaining stream. The resulting stream
422445-- would be 'Stream.nil' if the stream finished before the fold.
@@ -427,12 +450,12 @@ foldBreak fld strm = do
427450 r <- foldEither fld strm
428451 case r of
429452 Right res -> return res
430− Left (Fold _ initial extract) -> do
453+ Left (Fold _ initial _ final) -> do
431454 res <- initial
432455 case res of
433456 FL.Done _ -> error "foldBreak: unreachable state"
434457 FL.Partial s -> do
435− b <- extract s
458+ b <- final s
436459 return (b, nil)
437460
438461 where
@@ -475,8 +498,8 @@ fold fld strm = do
475498--
476499{-# INLINE_NORMAL foldAddLazy #-}
477500foldAddLazy :: Monad m => Fold m a b -> Stream m a -> Fold m a b
478−foldAddLazy (Fold fstep finitial fextract) (Stream sstep state) =
479− Fold fstep initial fextract
501+foldAddLazy (Fold fstep finitial fextract ffinal) (Stream sstep state) =
502+ Fold fstep initial fextract ffinal
480503
481504 where
482505
@@ -549,8 +572,12 @@ foldAdd f =
549572-- >>> Stream.foldrM step (return False) s
550573-- True
551574--
575+-- >>> import Control.Monad (join)
576+-- >>> foldrM f z = join $ Stream.foldr f z
577+--
552578{-# INLINE_NORMAL foldrM #-}
553579foldrM :: Monad m => (a -> m b -> m b) -> m b -> Stream m a -> m b
580+-- foldrM f z = join . Streamly.Internal.Data.Stream.StreamD.Type.foldr f z
554581foldrM f z (Stream step state) = go SPEC state
555582 where
556583 {-# INLINE_LATE go #-}
@@ -1796,7 +1823,7 @@ data FoldManyPost s fs b a
17961823--
17971824{-# INLINE_NORMAL foldManyPost #-}
17981825foldManyPost :: Monad m => Fold m a b -> Stream m a -> Stream m b
1799−foldManyPost (Fold fstep initial extract) (Stream step state) =
1826+foldManyPost (Fold fstep initial _ final) (Stream step state) =
18001827 Stream step' (FoldManyPostStart state)
18011828
18021829 where
@@ -1824,7 +1851,7 @@ foldManyPost (Fold fstep initial extract) (Stream step state) =
18241851 Yield x s -> consume x s fs
18251852 Skip s -> return $ Skip (FoldManyPostLoop s fs)
18261853 Stop -> do
1827− b <- extract fs
1854+ b <- final fs
18281855 return $ Skip (FoldManyPostYield b FoldManyPostDone)
18291856 step' _ (FoldManyPostYield b next) = return $ Yield b next
18301857 step' _ FoldManyPostDone = return Stop
@@ -1868,7 +1895,7 @@ data FoldMany s fs b a
18681895--
18691896{-# INLINE_NORMAL foldMany #-}
18701897foldMany :: Monad m => Fold m a b -> Stream m a -> Stream m b
1871−foldMany (Fold fstep initial extract) (Stream step state) =
1898+foldMany (Fold fstep initial _ final) (Stream step state) =
18721899 Stream step' (FoldManyStart state)
18731900
18741901 where
@@ -1895,14 +1922,14 @@ foldMany (Fold fstep initial extract) (Stream step state) =
18951922 case r of
18961923 Yield x s -> consume x s fs
18971924 Skip s -> return $ Skip (FoldManyFirst fs s)
1898− Stop -> return Stop
1925+ Stop -> final fs >> return Stop
18991926 step' gst (FoldManyLoop st fs) = do
19001927 r <- step (adaptState gst) st
19011928 case r of
19021929 Yield x s -> consume x s fs
19031930 Skip s -> return $ Skip (FoldManyLoop s fs)
19041931 Stop -> do
1905− b <- extract fs
1932+ b <- final fs
19061933 return $ Skip (FoldManyYield b FoldManyDone)
19071934 step' _ (FoldManyYield b next) = return $ Yield b next
19081935 step' _ FoldManyDone = return Stop
@@ -1969,6 +1996,110 @@ refoldMany (Refold fstep inject extract) action (Stream step state) =
19691996 step' _ (FoldManyYield b next) = return $ Yield b next
19701997 step' _ FoldManyDone = return Stop
19711998
1999+{-# ANN type CIterState Fuse #-}
2000+data CIterState s f fs b
2001+ = CIterInit s f
2002+ | CIterConsume s fs
2003+ | CIterYield b (CIterState s f fs b)
2004+ | CIterStop
2005+
2006+-- | Like 'foldIterateM' but using the 'Refold' type instead. This could be
2007+-- much more efficient due to stream fusion.
2008+--
2009+-- /Internal/
2010+{-# INLINE_NORMAL refoldIterateM #-}
2011+refoldIterateM ::
2012+ Monad m => Refold m b a b -> m b -> Stream m a -> Stream m b
2013+refoldIterateM (Refold fstep finject fextract) initial (Stream step state) =
2014+ Stream stepOuter (CIterInit state initial)
2015+
2016+ where
2017+
2018+ {-# INLINE iterStep #-}
2019+ iterStep st action = do
2020+ res <- action
2021+ return
2022+ $ Skip
2023+ $ case res of
2024+ FL.Partial fs -> CIterConsume st fs
2025+ FL.Done fb -> CIterYield fb $ CIterInit st (return fb)
2026+
2027+ {-# INLINE_LATE stepOuter #-}
2028+ stepOuter _ (CIterInit st action) = do
2029+ iterStep st (action >>= finject)
2030+ stepOuter gst (CIterConsume st fs) = do
2031+ r <- step (adaptState gst) st
2032+ case r of
2033+ Yield x s -> iterStep s (fstep fs x)
2034+ Skip s -> return $ Skip $ CIterConsume s fs
2035+ Stop -> do
2036+ b <- fextract fs
2037+ return $ Skip $ CIterYield b CIterStop
2038+ stepOuter _ (CIterYield a next) = return $ Yield a next
2039+ stepOuter _ CIterStop = return Stop
2040+
2041+-- | The refold @indexerBy f n@ takes an (index, len) tuple as initial input,
2042+-- and returns @(index + len + n, b)@ as output where @b@ is the output of the
2043+-- fold.
2044+{-# INLINE indexerBy #-}
2045+indexerBy :: Monad m =>
2046+ Fold m a Int -> Int -> Refold m (Int, Int) a (Int, Int)
2047+indexerBy (Fold step1 initial1 extract1 _final) n =
2048+ Refold step inject extract
2049+
2050+ where
2051+
2052+ inject (i, len) = do
2053+ r <- initial1
2054+ return $ case r of
2055+ FL.Partial s -> FL.Partial $ Tuple' (i + len + n) s
2056+ FL.Done l -> FL.Done (i, l)
2057+
2058+ step (Tuple' i s) x = do
2059+ r <- step1 s x
2060+ return $ case r of
2061+ FL.Partial s1 -> FL.Partial $ Tuple' i s1
2062+ FL.Done len -> FL.Done (i, len)
2063+
2064+ extract (Tuple' i s) = (i,) <$> extract1 s
2065+
2066+-- | Like 'splitOnSuffix' but generates a stream of (index, len) tuples marking
2067+-- the places where the predicate matches in the stream.
2068+--
2069+-- /Pre-release/
2070+{-# INLINE indexOnSuffix #-}
2071+indexOnSuffix :: Monad m =>
2072+ (a -> Bool) -> Stream m a -> Stream m (Int, Int)
2073+indexOnSuffix predicate =
2074+ -- Scan the stream with the given refold
2075+ refoldIterateM
2076+ (indexerBy (FL.takeEndBy_ predicate FL.length) 1)
2077+ (return (-1, 0))
2078+
2079+-- Alternate implementation
2080+{-# INLINE_NORMAL _indexOnSuffix #-}
2081+_indexOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
2082+_indexOnSuffix p (Stream step1 state1) = Stream step (Just (state1, 0, 0))
2083+
2084+ where
2085+
2086+ {-# INLINE_LATE step #-}
2087+ step gst (Just (st, i, len)) = i `seq` len `seq` do
2088+ r <- step1 (adaptState gst) st
2089+ return
2090+ $ case r of
2091+ Yield x s ->
2092+ if p x
2093+ then Yield (i, len + 1) (Just (s, i + len + 1, 0))
2094+ else Skip (Just (s, i, len + 1))
2095+ Skip s -> Skip (Just (s, i, len))
2096+ Stop -> if len == 0 then Stop else Yield (i, len) Nothing
2097+ step _ Nothing = return Stop
2098+
2099+{-# DEPRECATED sliceOnSuffix "Please use indexOnSuffix instead." #-}
2100+sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
2101+sliceOnSuffix = indexOnSuffix
2102+
19722103------------------------------------------------------------------------------
19732104-- Stream with a cross product style monad instance
19742105------------------------------------------------------------------------------
core/src/Streamly/Internal/Data/StreamK.hs+35−40
@@ -16,16 +16,15 @@ module Streamly.Internal.Data.StreamK
1616 --
1717 -- $setup
1818
19− -- * The stream type
2019 module Streamly.Internal.Data.StreamK.Type
20+ -- * Transformer
2121 , module Streamly.Internal.Data.StreamK.Transformer
2222
23− , StreamK(..)
23+ -- * From containers
2424 , fromList
2525 , fromStream
26− , toStream
2726
28− -- ** Specialized Generation
27+ -- * Specialized Generation
2928 , repeatM
3029 , replicate
3130 , replicateM
@@ -37,8 +36,6 @@ module Streamly.Internal.Data.StreamK
3736 -- * Elimination
3837 -- ** General Folds
3938 , foldr1
40− , foldlM'
41− , foldlMx'
4239 , fold
4340 , foldBreak
4441 , foldEither
@@ -69,13 +66,13 @@ module Streamly.Internal.Data.StreamK
6966 , find
7067 , (!!)
7168
69+ -- ** To Containers
70+ , toList
71+ , toStream
72+
7273 -- ** Map and Fold
7374 , mapM_
7475
75− -- ** Conversions
76− , toList
77− , hoist
78−
7976 -- * Transformation
8077 -- ** By folding (scans)
8178 , scanl'
@@ -102,6 +99,7 @@ module Streamly.Internal.Data.StreamK
10299
103100 -- ** Reordering
104101 , sortBy
102+ , sortOn
105103
106104 -- ** Map and Filter
107105 , mapMaybe
@@ -117,6 +115,9 @@ module Streamly.Internal.Data.StreamK
117115 -- ** Transformation comprehensions
118116 , the
119117
118+ -- ** Transforming Inner Monad
119+ , hoist
120+
120121 -- * Exceptions
121122 , handle
122123
@@ -133,6 +134,7 @@ import Control.Exception (mask_, Exception)
133134import Control.Monad (void, join)
134135import Control.Monad.Catch (MonadCatch)
135136import Control.Monad.IO.Class (MonadIO(..))
137+import Data.Ord (comparing)
136138import Data.Proxy (Proxy(..))
137139import GHC.Types (SPEC(..))
138140import Streamly.Internal.Data.Array.Type (Array(..))
@@ -306,19 +308,6 @@ foldr1 step m = do
306308 yieldk a r = fmap (step p) (go a r)
307309 in foldStream defState yieldk single stp m1
308310
309−-- XXX replace the recursive "go" with explicit continuations.
310−-- | Like 'foldx', but with a monadic step function.
311−{-# INLINABLE foldlMx' #-}
312−foldlMx' :: Monad m
313− => (x -> a -> m x) -> m x -> (x -> m b) -> StreamK m a -> m b
314−foldlMx' step begin done = go begin
315− where
316− go !acc m1 =
317− let stop = acc >>= done
318− single a = acc >>= \b -> step b a >>= done
319− yieldk a r = acc >>= \b -> step b a >>= \x -> go (return x) r
320− in foldStream defState yieldk single stop m1
321−
322311-- | Fold a stream using the supplied left 'Fold' and reducing the resulting
323312-- expression strictly at each step. The behavior is similar to 'foldl''. A
324313-- 'Fold' can terminate early without consuming the full stream. See the
@@ -336,7 +325,7 @@ foldlMx' step begin done = go begin
336325--
337326{-# INLINABLE fold #-}
338327fold :: Monad m => FL.Fold m a b -> StreamK m a -> m b
339−fold (FL.Fold step begin done) m = do
328+fold (FL.Fold step begin _ final) m = do
340329 res <- begin
341330 case res of
342331 FL.Partial fs -> go fs m
@@ -344,10 +333,10 @@ fold (FL.Fold step begin done) m = do
344333
345334 where
346335 go !acc m1 =
347− let stop = done acc
336+ let stop = final acc
348337 single a = step acc a
349338 >>= \case
350− FL.Partial s -> done s
339+ FL.Partial s -> final s
351340 FL.Done b1 -> return b1
352341 yieldk a r = step acc a
353342 >>= \case
@@ -365,7 +354,7 @@ fold (FL.Fold step begin done) m = do
365354{-# INLINE foldEither #-}
366355foldEither :: Monad m =>
367356 Fold m a b -> StreamK m a -> m (Either (Fold m a b) (b, StreamK m a))
368−foldEither (FL.Fold step begin done) m = do
357+foldEither (FL.Fold step begin done final) m = do
369358 res <- begin
370359 case res of
371360 FL.Partial fs -> go fs m
@@ -374,12 +363,15 @@ foldEither (FL.Fold step begin done) m = do
374363 where
375364
376365 go !acc m1 =
377− let stop = return $ Left (Fold step (return $ FL.Partial acc) done)
366+ let stop =
367+ let f = Fold step (return $ FL.Partial acc) done final
368+ in return $ Left f
378369 single a =
379370 step acc a
380371 >>= \case
381372 FL.Partial s ->
382− return $ Left (Fold step (return $ FL.Partial s) done)
373+ let f = Fold step (return $ FL.Partial s) done final
374+ in return $ Left f
383375 FL.Done b1 -> return $ Right (b1, nil)
384376 yieldk a r =
385377 step acc a
@@ -397,12 +389,12 @@ foldBreak fld strm = do
397389 r <- foldEither fld strm
398390 case r of
399391 Right res -> return res
400− Left (Fold _ initial extract) -> do
392+ Left (Fold _ initial _ final) -> do
401393 res <- initial
402394 case res of
403395 FL.Done _ -> error "foldBreak: unreachable state"
404396 FL.Partial s -> do
405− b <- extract s
397+ b <- final s
406398 return (b, nil)
407399
408400-- XXX Array folds can be implemented using this.
@@ -421,7 +413,7 @@ foldConcat :: Monad m =>
421413 Producer m a b -> Fold m b c -> StreamK m a -> m (c, StreamK m a)
422414foldConcat
423415 (Producer pstep pinject pextract)
424− (Fold fstep begin done)
416+ (Fold fstep begin _ final)
425417 stream = do
426418
427419 res <- begin
@@ -433,14 +425,14 @@ foldConcat
433425
434426 go !acc m1 = do
435427 let stop = do
436− r <- done acc
428+ r <- final acc
437429 return (r, nil)
438430 single a = do
439431 st <- pinject a
440432 res <- go1 SPEC acc st
441433 case res of
442434 Left fs -> do
443− r <- done fs
435+ r <- final fs
444436 return (r, nil)
445437 Right (b, s) -> do
446438 x <- pextract s
@@ -467,11 +459,6 @@ foldConcat
467459 Stream.Skip s -> go1 SPEC fs s
468460 Stream.Stop -> return $ Left fs
469461
470−-- | Like 'foldl'' but with a monadic step function.
471−{-# INLINE foldlM' #-}
472−foldlM' :: Monad m => (b -> a -> m b) -> m b -> StreamK m a -> m b
473−foldlM' step begin = foldlMx' step begin return
474−
475462------------------------------------------------------------------------------
476463-- Specialized folds
477464------------------------------------------------------------------------------
@@ -1328,7 +1315,8 @@ parserDone (ParserK.Failure n e) _ _ = pure $ ParserK.Error n e
13281315-- using parseBreakChunks. We can also use parseBreak as an alternative to the
13291316-- monad instance of ParserD.
13301317
1331−-- | Run a 'ParserK' over a chunked 'StreamK' and return the rest of the Stream.
1318+-- | Run a 'ParserK' over a chunked 'StreamK' and return the parse result and
1319+-- the remaining Stream.
13321320{-# INLINE_NORMAL parseBreakChunks #-}
13331321parseBreakChunks
13341322 :: (Monad m, Unbox a)
@@ -1733,3 +1721,10 @@ sortBy cmp =
17331721 . Stream.catRights -- its a non-failing backtracking parser
17341722 . Stream.parseMany (fmap (either id id) p)
17351723 . Stream.fromStreamK
1724+
1725+{-# INLINE sortOn #-}
1726+sortOn :: (Monad m, Ord b) => (a -> b) -> StreamK m a -> StreamK m a
1727+sortOn f =
1728+ fmap snd
1729+ . sortBy (comparing fst)
1730+ . fmap (\x -> let y = f x in y `seq` (y, x))
core/src/Streamly/Internal/Data/StreamK/Type.hs+80−11
@@ -1,4 +1,8 @@
11{-# LANGUAGE CPP #-}
2+{-# LANGUAGE TypeFamilies #-}
3+-- Must come after TypeFamilies, otherwise it is re-enabled.
4+-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
5+{-# LANGUAGE NoMonoLocalBinds #-}
26{-# LANGUAGE UndecidableInstances #-}
37-- |
48-- Module : Streamly.Internal.Data.StreamK.Type
@@ -84,6 +88,8 @@ module Streamly.Internal.Data.StreamK.Type
8488 -- ** Strict Left Folds
8589 , Streamly.Internal.Data.StreamK.Type.foldl'
8690 , foldlx'
91+ , foldlMx'
92+ , foldlM'
8793
8894 -- ** Lazy Right Folds
8995 , Streamly.Internal.Data.StreamK.Type.foldr
@@ -91,8 +97,11 @@ module Streamly.Internal.Data.StreamK.Type
9197 -- ** Specific Folds
9298 , drain
9399 , null
100+ , headNonEmpty
94101 , tail
102+ , tailNonEmpty
95103 , init
104+ , initNonEmpty
96105
97106 -- * Mapping
98107 , map
@@ -817,6 +826,24 @@ foldlx' step begin done m = get $ go m begin
817826foldl' :: Monad m => (b -> a -> b) -> b -> StreamK m a -> m b
818827foldl' step begin = foldlx' step begin id
819828
829+-- XXX replace the recursive "go" with explicit continuations.
830+-- | Like 'foldx', but with a monadic step function.
831+{-# INLINABLE foldlMx' #-}
832+foldlMx' :: Monad m
833+ => (x -> a -> m x) -> m x -> (x -> m b) -> StreamK m a -> m b
834+foldlMx' step begin done = go begin
835+ where
836+ go !acc m1 =
837+ let stop = acc >>= done
838+ single a = acc >>= \b -> step b a >>= done
839+ yieldk a r = acc >>= \b -> step b a >>= \x -> go (return x) r
840+ in foldStream defState yieldk single stop m1
841+
842+-- | Like 'foldl'' but with a monadic step function.
843+{-# INLINE foldlM' #-}
844+foldlM' :: Monad m => (b -> a -> m b) -> m b -> StreamK m a -> m b
845+foldlM' step begin = foldlMx' step begin return
846+
820847------------------------------------------------------------------------------
821848-- Specialized folds
822849------------------------------------------------------------------------------
@@ -1787,14 +1814,42 @@ iterateMWith cns step = go
17871814 !next <- s
17881815 foldStreamShared st stp sng yld (return next `cns` go (step next))
17891816
1790−{-# INLINE headPartial #-}
1791−headPartial :: Monad m => StreamK m a -> m a
1792−headPartial = foldrM (\x _ -> return x) (error "head of nil")
1817+-- | head for non-empty streams, fails for empty stream case.
1818+--
1819+{-# INLINE headNonEmpty #-}
1820+headNonEmpty :: Monad m => StreamK m a -> m a
1821+headNonEmpty = foldrM (\x _ -> return x) (error "headNonEmpty: empty stream")
1822+
1823+-- | init for non-empty streams, fails for empty stream case.
1824+--
1825+-- See also 'init' for a non-partial version of this function..
1826+{-# INLINE initNonEmpty #-}
1827+initNonEmpty :: Stream m a -> Stream m a
1828+initNonEmpty = go0
1829+
1830+ where
1831+
1832+ go0 m = mkStream $ \st yld sng stp ->
1833+ let stop = error "initNonEmpty: Empty Stream."
1834+ single _ = stp
1835+ yieldk a r = foldStream st yld sng stp (go1 a r)
1836+ in foldStream st yieldk single stop m
1837+
1838+ go1 a r = mkStream $ \st yld sng stp ->
1839+ let stop = stp
1840+ single _ = sng a
1841+ yieldk a1 r1 = yld a (go1 a1 r1)
1842+ in foldStream st yieldk single stop r
17931843
1794−{-# INLINE tailPartial #-}
1795−tailPartial :: StreamK m a -> StreamK m a
1796−tailPartial m = mkStream $ \st yld sng stp ->
1797− let stop = error "tail of nil"
1844+-- | tail for non-empty streams, fails for empty stream case.
1845+--
1846+-- See also 'tail' for a non-partial version of this function..
1847+--
1848+-- Note: this is same as "drop 1" with error on empty stream.
1849+{-# INLINE tailNonEmpty #-}
1850+tailNonEmpty :: StreamK m a -> StreamK m a
1851+tailNonEmpty m = mkStream $ \st yld sng stp ->
1852+ let stop = error "tailNonEmpty: empty stream"
17981853 single _ = stp
17991854 yieldk _ r = foldStream st yld sng stp r
18001855 in foldStream st yieldk single stop m
@@ -1859,10 +1914,10 @@ mfix f = mkStream $ \st yld sng stp ->
18591914 where
18601915
18611916 -- fix the head element of the stream
1862− xs = fix (f . headPartial)
1917+ xs = fix (f . headNonEmpty)
18631918
18641919 -- now fix the tail recursively
1865− ys = mfix (tailPartial . f)
1920+ ys = mfix (tailNonEmpty . f)
18661921
18671922-------------------------------------------------------------------------------
18681923-- Conversions
@@ -1893,6 +1948,15 @@ uncons m =
18931948 yieldk a r = pure (Just (a, r))
18941949 in foldStream defState yieldk single stop m
18951950
1951+-- Note that this is not a StreamK -> StreamK because then we cannot handle the
1952+-- empty stream case without making this a partial function.
1953+--
1954+-- See tailNonEmpty as well above.
1955+
1956+-- | Same as:
1957+--
1958+-- >>> tail = fmap (fmap snd) StreamK.uncons
1959+--
18961960{-# INLINE tail #-}
18971961tail :: Applicative m => StreamK m a -> m (Maybe (StreamK m a))
18981962tail =
@@ -1901,9 +1965,14 @@ tail =
19011965 yieldk _ r = pure $ Just r
19021966 in foldStream defState yieldk single stop
19031967
1904−-- | Extract all but the last element of the stream, if any.
1968+-- Note that this is not a StreamK -> StreamK because then we cannot handle the
1969+-- empty stream case without making this a partial function.
19051970--
1906−-- Note: This will end up buffering the entire stream.
1971+-- XXX How do we implement unsnoc? Make StreamK a monad and return the
1972+-- remaining stream as a result value in the monad?
1973+
1974+-- | Extract all but the last element of the stream, if any. This will end up
1975+-- evaluating the last element as well to find out that it is last.
19071976--
19081977-- /Pre-release/
19091978{-# INLINE init #-}
core/src/Streamly/Internal/Data/Time/Units.hs−4
@@ -1,7 +1,3 @@
1−{-# LANGUAGE DataKinds #-}
2−{-# LANGUAGE PolyKinds #-}
3−{-# LANGUAGE UnboxedTuples #-}
4−
51-- |
62-- Module : Streamly.Internal.Data.Time.Units
73-- Copyright : (c) 2019 Composewell Technologies
core/src/Streamly/Internal/Data/Unbox.hs+165−289
@@ -1,5 +1,8 @@
11{-# LANGUAGE DataKinds #-}
2−{-# LANGUAGE DefaultSignatures #-}
2+{-# LANGUAGE TypeFamilies #-}
3+-- Must come after TypeFamilies, otherwise it is re-enabled.
4+-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
5+{-# LANGUAGE NoMonoLocalBinds #-}
36{-# LANGUAGE UnboxedTuples #-}
47{-# LANGUAGE UndecidableInstances #-}
58
@@ -12,49 +15,34 @@
1215-- Portability : GHC
1316--
1417module Streamly.Internal.Data.Unbox
15− ( Unbox(..)
16− , PinnedState(..)
17− , MutableByteArray(..)
18− , touch
19− , getMutableByteArray#
20− , sizeOfMutableByteArray
21− , isPinned
22− , pin
23− , unpin
24− , newBytesAs
25− , newBytes
26− , pinnedNewBytes
27− , pinnedNewAlignedBytes
28− , nil
29− , putSliceUnsafe
30−
31− -- * Type Parser and Builder
32− , BoundedPtr (..)
18+ (
19+ -- ** Unbox type class
20+ Unbox(..)
3321
22+ -- ** Peek and poke utilities
23+ , BoundedPtr (..)
24+ -- Peek
3425 , Peeker (..)
3526 , read
3627 , readUnsafe
3728 , skipByte
3829 , runPeeker
39−
30+ -- Poke
4031 , pokeBoundedPtrUnsafe
4132 , pokeBoundedPtr
4233
43− -- * Generic Unbox instances
44− , genericSizeOf
45− , genericPeekByteIndex
46− , genericPokeByteIndex
47−
48− -- Classess used for generic deriving.
34+ -- ** Generic Deriving
4935 , PeekRep(..)
5036 , PokeRep(..)
5137 , SizeOfRep(..)
38+ , genericSizeOf
39+ , genericPeekByteIndex
40+ , genericPokeByteIndex
5241 ) where
5342
5443#include "MachDeps.h"
5544#include "ArrayMacros.h"
5645
57−import Control.Monad.IO.Class (MonadIO(..))
5846import Control.Monad (void, when)
5947import Data.Complex (Complex((:+)))
6048import Data.Functor ((<&>))
@@ -62,9 +50,6 @@ import Data.Functor.Const (Const(..))
6250import Data.Functor.Identity (Identity(..))
6351import Data.Kind (Type)
6452import Data.Proxy (Proxy (..))
65−#ifdef DEBUG
66−import Debug.Trace (trace)
67−#endif
6853import Foreign.Ptr (IntPtr(..), WordPtr(..))
6954import GHC.Base (IO(..))
7055import GHC.Fingerprint.Type (Fingerprint(..))
@@ -76,172 +61,16 @@ import GHC.Word (Word16(..), Word32(..), Word64(..), Word8(..))
7661import GHC.RTS.Flags (IoSubSystem(..))
7762#endif
7863import Streamly.Internal.Data.Builder (Builder (..))
79−import System.IO.Unsafe (unsafePerformIO)
8064
8165import GHC.Generics
8266import GHC.Exts
8367import GHC.TypeLits
8468import Prelude hiding (read)
8569
86−--------------------------------------------------------------------------------
87−-- The ArrayContents type
88−--------------------------------------------------------------------------------
89−
90−data PinnedState
91− = Pinned
92− | Unpinned
93−
94−-- XXX can use UnliftedNewtypes
95−data MutableByteArray = MutableByteArray (MutableByteArray# RealWorld)
96−
97−{-# INLINE getMutableByteArray# #-}
98−getMutableByteArray# :: MutableByteArray -> MutableByteArray# RealWorld
99−getMutableByteArray# (MutableByteArray mbarr) = mbarr
100−
101−{-# INLINE touch #-}
102−touch :: MutableByteArray -> IO ()
103−touch (MutableByteArray contents) =
104− IO $ \s -> case touch# contents s of s' -> (# s', () #)
105−
106−-- | Return the size of the array in bytes.
107−{-# INLINE sizeOfMutableByteArray #-}
108−sizeOfMutableByteArray :: MutableByteArray -> IO Int
109−sizeOfMutableByteArray (MutableByteArray arr) =
110− IO $ \s ->
111− case getSizeofMutableByteArray# arr s of
112− (# s1, i #) -> (# s1, I# i #)
113−
114−--------------------------------------------------------------------------------
115−-- Creation
116−--------------------------------------------------------------------------------
117−
118−{-# NOINLINE nil #-}
119−nil :: MutableByteArray
120−nil = unsafePerformIO $ newBytes 0
121−
122−{-# INLINE newBytes #-}
123−newBytes :: Int -> IO MutableByteArray
124−newBytes nbytes | nbytes < 0 =
125− errorWithoutStackTrace "newBytes: size must be >= 0"
126−newBytes (I# nbytes) = IO $ \s ->
127− case newByteArray# nbytes s of
128− (# s', mbarr# #) ->
129− let c = MutableByteArray mbarr#
130− in (# s', c #)
131−
132−{-# INLINE pinnedNewBytes #-}
133−pinnedNewBytes :: Int -> IO MutableByteArray
134−pinnedNewBytes nbytes | nbytes < 0 =
135− errorWithoutStackTrace "pinnedNewBytes: size must be >= 0"
136−pinnedNewBytes (I# nbytes) = IO $ \s ->
137− case newPinnedByteArray# nbytes s of
138− (# s', mbarr# #) ->
139− let c = MutableByteArray mbarr#
140− in (# s', c #)
141−
142−{-# INLINE pinnedNewAlignedBytes #-}
143−pinnedNewAlignedBytes :: Int -> Int -> IO MutableByteArray
144−pinnedNewAlignedBytes nbytes _align | nbytes < 0 =
145− errorWithoutStackTrace "pinnedNewAlignedBytes: size must be >= 0"
146−pinnedNewAlignedBytes (I# nbytes) (I# align) = IO $ \s ->
147− case newAlignedPinnedByteArray# nbytes align s of
148− (# s', mbarr# #) ->
149− let c = MutableByteArray mbarr#
150− in (# s', c #)
151−
152−{-# INLINE newBytesAs #-}
153−newBytesAs :: PinnedState -> Int -> IO MutableByteArray
154−newBytesAs Unpinned = newBytes
155−newBytesAs Pinned = pinnedNewBytes
156−
157−-------------------------------------------------------------------------------
158−-- Copying
159−-------------------------------------------------------------------------------
160−
161−-- | Put a sub range of a source array into a subrange of a destination array.
162−-- This is not safe as it does not check the bounds.
163−{-# INLINE putSliceUnsafe #-}
164−putSliceUnsafe ::
165− MonadIO m
166− => MutableByteArray
167− -> Int
168− -> MutableByteArray
169− -> Int
170− -> Int
171− -> m ()
172−putSliceUnsafe src srcStartBytes dst dstStartBytes lenBytes = liftIO $ do
70+import Streamly.Internal.Data.MutByteArray.Type (MutByteArray(..))
17371#ifdef DEBUG
174− srcLen <- sizeOfMutableByteArray src
175− dstLen <- sizeOfMutableByteArray src
176− when (srcLen - srcStartBytes < lenBytes) $ error "SRC: Insufficient length."
177− when (dstLen - dstStartBytes < lenBytes) $ error "DST: Insufficient length."
72+import Streamly.Internal.Data.MutByteArray.Type (sizeOfMutableByteArray)
17873#endif
179− let !(I# srcStartBytes#) = srcStartBytes
180− !(I# dstStartBytes#) = dstStartBytes
181− !(I# lenBytes#) = lenBytes
182− let arrS# = getMutableByteArray# src
183− arrD# = getMutableByteArray# dst
184− IO $ \s# -> (# copyMutableByteArray#
185− arrS# srcStartBytes# arrD# dstStartBytes# lenBytes# s#
186− , () #)
187−
188−-------------------------------------------------------------------------------
189−-- Pinning & Unpinning
190−-------------------------------------------------------------------------------
191−
192−{-# INLINE isPinned #-}
193−isPinned :: MutableByteArray -> Bool
194−isPinned (MutableByteArray arr#) =
195− let pinnedInt = I# (isMutableByteArrayPinned# arr#)
196− in pinnedInt /= 0
197−
198−
199−{-# INLINE cloneMutableArrayWith# #-}
200−cloneMutableArrayWith#
201− :: (Int# -> State# RealWorld -> (# State# RealWorld
202− , MutableByteArray# RealWorld #))
203− -> MutableByteArray# RealWorld
204− -> State# RealWorld
205− -> (# State# RealWorld, MutableByteArray# RealWorld #)
206−cloneMutableArrayWith# alloc# arr# s# =
207− case getSizeofMutableByteArray# arr# s# of
208− (# s1#, i# #) ->
209− case alloc# i# s1# of
210− (# s2#, arr1# #) ->
211− case copyMutableByteArray# arr# 0# arr1# 0# i# s2# of
212− s3# -> (# s3#, arr1# #)
213−
214−{-# INLINE pin #-}
215−pin :: MutableByteArray -> IO MutableByteArray
216−pin arr@(MutableByteArray marr#) =
217− if isPinned arr
218− then return arr
219− else
220−#ifdef DEBUG
221− do
222− -- XXX dump stack trace
223− trace ("pin: Copying array") (return ())
224−#endif
225− IO
226− $ \s# ->
227− case cloneMutableArrayWith# newPinnedByteArray# marr# s# of
228− (# s1#, marr1# #) -> (# s1#, MutableByteArray marr1# #)
229−
230−{-# INLINE unpin #-}
231−unpin :: MutableByteArray -> IO MutableByteArray
232−unpin arr@(MutableByteArray marr#) =
233− if not (isPinned arr)
234− then return arr
235− else
236−#ifdef DEBUG
237− do
238− -- XXX dump stack trace
239− trace ("unpin: Copying array") (return ())
240−#endif
241− IO
242− $ \s# ->
243− case cloneMutableArrayWith# newByteArray# marr# s# of
244− (# s1#, marr1# #) -> (# s1#, MutableByteArray marr1# #)
24574
24675--------------------------------------------------------------------------------
24776-- The Unbox type class
@@ -327,23 +156,37 @@ unpin arr@(MutableByteArray marr#) =
327156-- size fields at each nesting level, aggregating the size upwards.
328157
329158-- | The 'Unbox' type class provides operations for serialization (unboxing)
330−-- and deserialization (boxing) of a fixed-length, non-recursive Haskell data
331−-- type to and from its unboxed byte representation in a mutable byte array.
159+-- and deserialization (boxing) of fixed-length, non-recursive Haskell data
160+-- types to and from their byte stream representation.
161+--
332162-- Unbox uses fixed size encoding, therefore, size is independent of the value,
333−-- it can be determined solely by the type. This makes types with 'Unbox'
334−-- instances suitable for storing in arrays. Note that sum types may have
335−-- multiple constructors of different sizes, the size of the sum type is
163+-- it must be determined solely by the type. This restriction makes types with
164+-- 'Unbox' instances suitable for storing in arrays. Note that sum types may
165+-- have multiple constructors of different sizes, the size of a sum type is
336166-- computed as the maximum required by any constructor.
337167--
338−-- The 'peekByteIndex' read operation converts a Haskell type from its unboxed
339−-- representation stored in a mutable byte array, while the 'pokeByteIndex'
340−-- write operation serializes a Haskell data type to its byte representation in
341−-- the mutable byte array. These operations do not check the bounds of the
342−-- array, the user of the type class is expected to check the bounds before
343−-- peeking or poking.
344−--
345−-- Instances can be derived via Generics or Template Haskell. Note that the
346−-- data type must be non-recursive.
168+-- The 'peekAt' operation reads as many bytes from the mutable byte
169+-- array as the @size@ of the data type and builds a Haskell data type from
170+-- these bytes. 'pokeAt' operation converts a Haskell data type to its
171+-- binary representation which consists of @size@ bytes and then stores
172+-- these bytes into the mutable byte array. These operations do not check the
173+-- bounds of the array, the user of the type class is expected to check the
174+-- bounds before peeking or poking.
175+--
176+-- IMPORTANT: The serialized data's byte ordering remains the same as the host
177+-- machine's byte order. Therefore, it can not be deserialized from host
178+-- machines with a different byte ordering.
179+--
180+-- Instances can be derived via Generics, Template Haskell, or written
181+-- manually. Note that the data type must be non-recursive. WARNING! Generic
182+-- and Template Haskell deriving, both hang for recursive data types. Deriving
183+-- via Generics is more convenient but Template Haskell should be preferred
184+-- over Generics for the following reasons:
185+--
186+-- 1. Instances derived via Template Haskell provide better and more reliable
187+-- performance.
188+-- 2. Generic deriving allows only 256 fields or constructor tags whereas
189+-- template Haskell has no limit.
347190--
348191-- Here is an example, for deriving an instance of this type class using
349192-- generics:
@@ -356,113 +199,136 @@ unpin arr@(MutableByteArray marr#) =
356199-- } deriving Generic
357200-- :}
358201--
202+-- >>> import Streamly.Data.MutByteArray (Unbox(..))
203+-- >>> instance Unbox Object
204+--
359205-- To derive the instance via Template Haskell:
360206--
361207-- @
362−-- import Streamly.Data.Serialize (deriveUnbox)
363−-- $(deriveUnbox ''Object)
208+-- import Streamly.Data.MutByteArray (deriveUnbox)
209+-- \$(deriveUnbox [d|instance Unbox Object|])
364210-- @
365211--
366−-- See 'Streamly.Data.Serialize.deriveUnbox' and
367−-- 'Streamly.Data.Serialize.deriveUnboxWith' for more information on deriving
212+-- See 'Streamly.Data.MutByteArray.deriveUnbox' for more information on deriving
368213-- using Template Haskell.
369214--
370−-- Deriving via Template Haskell should be preferred for the following reasons:
371−-- 1. Instances derived via Template Haskell have more performant routines.
372−-- 2. Template Haskell deriving can handle (maxBound :: Word64) number of
373−-- constructors whereas the Generic deriving can only handle 256.
374−--
375−-- WARNING! Generic and Template Haskell deriving, both hang for recursive data
376−-- types.
377−--
378−-- >>> import Streamly.Data.Array (Unbox(..))
379−-- >>> instance Unbox Object
380−--
381215-- If you want to write the instance manually:
382216--
383217-- >>> :{
384218-- instance Unbox Object where
385219-- sizeOf _ = 16
386−-- peekByteIndex i arr = do
220+-- peekAt i arr = do
387221-- -- Check the array bounds
388−-- x0 <- peekByteIndex i arr
389−-- x1 <- peekByteIndex (i + 8) arr
222+-- x0 <- peekAt i arr
223+-- x1 <- peekAt (i + 8) arr
390224-- return $ Object x0 x1
391−-- pokeByteIndex i arr (Object x0 x1) = do
225+-- pokeAt i arr (Object x0 x1) = do
392226-- -- Check the array bounds
393−-- pokeByteIndex i arr x0
394−-- pokeByteIndex (i + 8) arr x1
227+-- pokeAt i arr x0
228+-- pokeAt (i + 8) arr x1
395229-- :}
396230--
397231class Unbox a where
398− -- | Get the size. Size cannot be zero.
232+ -- | Get the size. Size cannot be zero, should be at least 1 byte.
399233 sizeOf :: Proxy a -> Int
400234
401235 {-# INLINE sizeOf #-}
402236 default sizeOf :: (SizeOfRep (Rep a)) => Proxy a -> Int
403237 sizeOf = genericSizeOf
404238
405− -- | Read an element of type "a" from a MutableByteArray given the byte
406− -- index.
239+ -- | @peekAt byte-offset array@ reads an element of type @a@ from the
240+ -- the given the byte offset in the array.
407241 --
408242 -- IMPORTANT: The implementation of this interface may not check the bounds
409243 -- of the array, the caller must not assume that.
410− peekByteIndex :: Int -> MutableByteArray -> IO a
244+ peekAt :: Int -> MutByteArray -> IO a
245+
246+ {-# INLINE peekAt #-}
247+ default peekAt :: (Generic a, PeekRep (Rep a)) =>
248+ Int -> MutByteArray -> IO a
249+ peekAt i arr = genericPeekByteIndex arr i
411250
412− {-# INLINE peekByteIndex #-}
413− default peekByteIndex :: (Generic a, PeekRep (Rep a)) =>
414− Int -> MutableByteArray -> IO a
415− peekByteIndex i arr = genericPeekByteIndex arr i
251+ peekByteIndex :: Int -> MutByteArray -> IO a
252+ peekByteIndex = peekAt
416253
417− -- | Write an element of type "a" to a MutableByteArray given the byte
418− -- index.
254+ -- | @pokeAt byte-offset array@ writes an element of type @a@ to the
255+ -- the given the byte offset in the array.
419256 --
420257 -- IMPORTANT: The implementation of this interface may not check the bounds
421258 -- of the array, the caller must not assume that.
422− pokeByteIndex :: Int -> MutableByteArray -> a -> IO ()
259+ pokeAt :: Int -> MutByteArray -> a -> IO ()
260+
261+ pokeByteIndex :: Int -> MutByteArray -> a -> IO ()
262+ pokeByteIndex = pokeAt
423263
424− {-# INLINE pokeByteIndex #-}
425− default pokeByteIndex :: (Generic a, PokeRep (Rep a)) =>
426− Int -> MutableByteArray -> a -> IO ()
427− pokeByteIndex i arr = genericPokeByteIndex arr i
264+ {-# INLINE pokeAt #-}
265+ default pokeAt :: (Generic a, PokeRep (Rep a)) =>
266+ Int -> MutByteArray -> a -> IO ()
267+ pokeAt i arr = genericPokeByteIndex arr i
428268
429−-- XXX Add asserts to check bounds
269+{-# DEPRECATED peekByteIndex "Use peekAt." #-}
270+{-# DEPRECATED pokeByteIndex "Use pokeAt." #-}
271+
272+-- _size is the length from array start to the last accessed byte.
273+{-# INLINE checkBounds #-}
274+checkBounds :: String -> Int -> MutByteArray -> IO ()
275+checkBounds _label _size _arr = do
276+#ifdef DEBUG
277+ sz <- sizeOfMutableByteArray _arr
278+ if (_size > sz)
279+ then error
280+ $ _label
281+ ++ ": accessing array at offset = "
282+ ++ show (_size - 1)
283+ ++ " max valid offset = " ++ show (sz - 1)
284+ else return ()
285+#else
286+ return ()
287+#endif
430288
431289#define DERIVE_UNBOXED(_type, _constructor, _readArray, _writeArray, _sizeOf) \
432−instance Unbox _type where { \
433−; {-# INLINE peekByteIndex #-} \
434−; peekByteIndex (I# n) (MutableByteArray mbarr) = IO $ \s -> \
435− case _readArray mbarr n s of \
436− { (# s1, i #) -> (# s1, _constructor i #) } \
437−; {-# INLINE pokeByteIndex #-} \
438−; pokeByteIndex (I# n) (MutableByteArray mbarr) (_constructor val) = \
439− IO $ \s -> (# _writeArray mbarr n val s, () #) \
440−; {-# INLINE sizeOf #-} \
441−; sizeOf _ = _sizeOf \
290+instance Unbox _type where { \
291+; {-# INLINE peekAt #-} \
292+; peekAt off@(I# n) arr@(MutByteArray mbarr) = \
293+ checkBounds "peek" (off + sizeOf (Proxy :: Proxy _type)) arr \
294+ >> (IO $ \s -> \
295+ case _readArray mbarr n s of \
296+ { (# s1, i #) -> (# s1, _constructor i #) }) \
297+; {-# INLINE pokeAt #-} \
298+; pokeAt off@(I# n) arr@(MutByteArray mbarr) (_constructor val) = \
299+ checkBounds "poke" (off + sizeOf (Proxy :: Proxy _type)) arr \
300+ >> (IO $ \s -> (# _writeArray mbarr n val s, () #)) \
301+; {-# INLINE sizeOf #-} \
302+; sizeOf _ = _sizeOf \
442303}
443304
444305#define DERIVE_WRAPPED_UNBOX(_constraint, _type, _constructor, _innerType) \
445306instance _constraint Unbox _type where \
446−; {-# INLINE peekByteIndex #-} \
447−; peekByteIndex i arr = _constructor <$> peekByteIndex i arr \
448−; {-# INLINE pokeByteIndex #-} \
449−; pokeByteIndex i arr (_constructor a) = pokeByteIndex i arr a \
307+; {-# INLINE peekAt #-} \
308+; peekAt i arr = \
309+ checkBounds "peek" (i + sizeOf (Proxy :: Proxy _type)) arr \
310+ >> _constructor <$> peekAt i arr \
311+; {-# INLINE pokeAt #-} \
312+; pokeAt i arr (_constructor a) = \
313+ checkBounds "poke" (i + sizeOf (Proxy :: Proxy _type)) arr \
314+ >> pokeAt i arr a \
450315; {-# INLINE sizeOf #-} \
451316; sizeOf _ = SIZE_OF(_innerType)
452317
453−#define DERIVE_BINARY_UNBOX(_constraint, _type, _constructor, _innerType) \
454−instance _constraint Unbox _type where { \
455−; {-# INLINE peekByteIndex #-} \
456−; peekByteIndex i arr = \
457− peekByteIndex i arr >>= \
458− (\p1 -> peekByteIndex (i + SIZE_OF(_innerType)) arr \
459− <&> _constructor p1) \
460−; {-# INLINE pokeByteIndex #-} \
461−; pokeByteIndex i arr (_constructor p1 p2) = \
462− pokeByteIndex i arr p1 >> \
463− pokeByteIndex (i + SIZE_OF(_innerType)) arr p2 \
464−; {-# INLINE sizeOf #-} \
465−; sizeOf _ = 2 * SIZE_OF(_innerType) \
318+#define DERIVE_BINARY_UNBOX(_constraint, _type, _constructor, _innerType) \
319+instance _constraint Unbox _type where { \
320+; {-# INLINE peekAt #-} \
321+; peekAt i arr = \
322+ checkBounds "peek" (i + sizeOf (Proxy :: Proxy _type)) arr >> \
323+ peekAt i arr >>= \
324+ (\p1 -> peekAt (i + SIZE_OF(_innerType)) arr <&> _constructor p1) \
325+; {-# INLINE pokeAt #-} \
326+; pokeAt i arr (_constructor p1 p2) = \
327+ checkBounds "poke" (i + sizeOf (Proxy :: Proxy _type)) arr >> \
328+ pokeAt i arr p1 >> \
329+ pokeAt (i + SIZE_OF(_innerType)) arr p2 \
330+; {-# INLINE sizeOf #-} \
331+; sizeOf _ = 2 * SIZE_OF(_innerType) \
466332}
467333
468334-------------------------------------------------------------------------------
@@ -582,11 +448,13 @@ DERIVE_BINARY_UNBOX(,Fingerprint,Fingerprint,Word64)
582448
583449instance Unbox () where
584450
585− {-# INLINE peekByteIndex #-}
586− peekByteIndex _ _ = return ()
451+ {-# INLINE peekAt #-}
452+ peekAt i arr =
453+ checkBounds "peek ()" (i + sizeOf (Proxy :: Proxy ())) arr >> return ()
587454
588− {-# INLINE pokeByteIndex #-}
589− pokeByteIndex _ _ _ = return ()
455+ {-# INLINE pokeAt #-}
456+ pokeAt i arr _ =
457+ checkBounds "poke ()" (i + sizeOf (Proxy :: Proxy ())) arr >> return ()
590458
591459 {-# INLINE sizeOf #-}
592460 sizeOf _ = 1
@@ -594,11 +462,17 @@ instance Unbox () where
594462#if MIN_VERSION_base(4,15,0)
595463instance Unbox IoSubSystem where
596464
597− {-# INLINE peekByteIndex #-}
598− peekByteIndex i arr = toEnum <$> peekByteIndex i arr
465+ {-# INLINE peekAt #-}
466+ peekAt i arr =
467+ checkBounds
468+ "peek IoSubSystem" (i + sizeOf (Proxy :: Proxy IoSubSystem)) arr
469+ >> toEnum <$> peekAt i arr
599470
600− {-# INLINE pokeByteIndex #-}
601− pokeByteIndex i arr a = pokeByteIndex i arr (fromEnum a)
471+ {-# INLINE pokeAt #-}
472+ pokeAt i arr a =
473+ checkBounds
474+ "poke IoSubSystem" (i + sizeOf (Proxy :: Proxy IoSubSystem)) arr
475+ >> pokeAt i arr (fromEnum a)
602476
603477 {-# INLINE sizeOf #-}
604478 sizeOf _ = sizeOf (Proxy :: Proxy Int)
@@ -606,16 +480,18 @@ instance Unbox IoSubSystem where
606480
607481instance Unbox Bool where
608482
609− {-# INLINE peekByteIndex #-}
610− peekByteIndex i arr = do
611− res <- peekByteIndex i arr
483+ {-# INLINE peekAt #-}
484+ peekAt i arr = do
485+ checkBounds "peek Bool" (i + sizeOf (Proxy :: Proxy Bool)) arr
486+ res <- peekAt i arr
612487 return $ res /= (0 :: Int8)
613488
614− {-# INLINE pokeByteIndex #-}
615− pokeByteIndex i arr a =
616− if a
617− then pokeByteIndex i arr (1 :: Int8)
618− else pokeByteIndex i arr (0 :: Int8)
489+ {-# INLINE pokeAt #-}
490+ pokeAt i arr a =
491+ checkBounds "poke Bool" (i + sizeOf (Proxy :: Proxy Bool)) arr
492+ >> if a
493+ then pokeAt i arr (1 :: Int8)
494+ else pokeAt i arr (0 :: Int8)
619495
620496 {-# INLINE sizeOf #-}
621497 sizeOf _ = 1
@@ -630,7 +506,7 @@ instance Unbox Bool where
630506-- cheaper to just get the bound using the size of the array whenever needed?
631507data BoundedPtr =
632508 BoundedPtr
633− MutableByteArray -- byte array
509+ MutByteArray -- byte array
634510 Int -- current pos
635511 Int -- position after end
636512
@@ -658,7 +534,7 @@ readUnsafe = Peeker (Builder step)
658534 ++ show next
659535 ++ " end = " ++ show end
660536#endif
661− r <- peekByteIndex pos arr
537+ r <- peekAt pos arr
662538 return (r, BoundedPtr arr next end)
663539
664540{-# INLINE read #-}
@@ -675,7 +551,7 @@ read = Peeker (Builder step)
675551 $ error $ "read: reading beyond limit. next = "
676552 ++ show next
677553 ++ " end = " ++ show end
678− r <- peekByteIndex pos arr
554+ r <- peekAt pos arr
679555 return (r, BoundedPtr arr next end)
680556
681557{-# INLINE skipByte #-}
@@ -720,7 +596,7 @@ pokeBoundedPtrUnsafe a (BoundedPtr arr pos end) = do
720596 ++ show next
721597 ++ " end = " ++ show end
722598#endif
723− pokeByteIndex pos arr a
599+ pokeAt pos arr a
724600 return (BoundedPtr arr next end)
725601
726602{-# INLINE pokeBoundedPtr #-}
@@ -728,7 +604,7 @@ pokeBoundedPtr :: forall a. Unbox a => a -> BoundedPtr -> IO BoundedPtr
728604pokeBoundedPtr a (BoundedPtr arr pos end) = do
729605 let next = pos + sizeOf (Proxy :: Proxy a)
730606 when (next > end) $ error "pokeBoundedPtr writing beyond limit"
731− pokeByteIndex pos arr a
607+ pokeAt pos arr a
732608 return (BoundedPtr arr next end)
733609
734610--------------------------------------------------------------------------------
@@ -915,7 +791,7 @@ genericPokeObj a ptr = void $ genericPokeObject a ptr
915791
916792{-# INLINE genericPokeByteIndex #-}
917793genericPokeByteIndex :: (Generic a, PokeRep (Rep a)) =>
918− MutableByteArray -> Int -> a -> IO ()
794+ MutByteArray -> Int -> a -> IO ()
919795genericPokeByteIndex arr index x = do
920796 -- XXX Should we use unsafe poke?
921797#ifdef DEBUG
@@ -1021,7 +897,7 @@ genericPeekBoundedPtr = runPeeker genericPeeker
1021897
1022898{-# INLINE genericPeekByteIndex #-}
1023899genericPeekByteIndex :: (Generic a, PeekRep (Rep a)) =>
1024− MutableByteArray -> Int -> IO a
900+ MutByteArray -> Int -> IO a
1025901genericPeekByteIndex arr index = do
1026902 -- XXX Should we use unsafe peek?
1027903#ifdef DEBUG
core/src/Streamly/Internal/Data/Unbox/TH.hs+74−74
@@ -10,13 +10,11 @@
1010--
1111module Streamly.Internal.Data.Unbox.TH
1212 ( deriveUnbox
13− , deriveUnboxWith
1413
1514 -- th-helpers
1615 , DataCon(..)
1716 , DataType(..)
1817 , reifyDataType
19− , appsT
2018 ) where
2119
2220--------------------------------------------------------------------------------
@@ -25,6 +23,7 @@ module Streamly.Internal.Data.Unbox.TH
2523
2624import Data.Word (Word16, Word32, Word64, Word8)
2725import Data.Proxy (Proxy(..))
26+import Data.List (elemIndex)
2827
2928import Language.Haskell.TH
3029import Language.Haskell.TH.Syntax
@@ -69,10 +68,6 @@ tvName = elimTV id (\n _ -> n)
6968tyVarBndrName :: TyVarBndr_ flag -> Name
7069tyVarBndrName = tvName
7170
72−appsT :: Type -> [Type] -> Type
73−appsT x [] = x
74−appsT x (y:xs) = appsT (AppT x y) xs
75−
7671-- | Simplified info about a 'DataD'. Omits deriving, strictness,
7772-- kind info, and whether it's @data@ or @newtype@.
7873data DataType = DataType
@@ -246,6 +241,7 @@ mkSizeOfExpr headTy constructors =
246241mkPeekExprOne :: Int -> DataCon -> Q Exp
247242mkPeekExprOne tagSize (DataCon cname _ _ fields) =
248243 case fields of
244+ -- XXX Should we peek and check if the byte value is 0?
249245 [] -> [|pure $(conE cname)|]
250246 _ ->
251247 letE
@@ -257,7 +253,7 @@ mkPeekExprOne tagSize (DataCon cname _ _ fields) =
257253
258254 where
259255
260− peekField i = [|peekByteIndex $(varE (mkOffsetName i)) $(varE _arr)|]
256+ peekField i = [|peekAt $(varE (mkOffsetName i)) $(varE _arr)|]
261257
262258mkPeekExpr :: Type -> [DataCon] -> Q Exp
263259mkPeekExpr headTy cons =
@@ -271,7 +267,7 @@ mkPeekExpr headTy cons =
271267 doE
272268 [ bindS
273269 (varP _tag)
274− [|peekByteIndex $(varE _initialOffset) $(varE _arr)|]
270+ [|peekAt $(varE _initialOffset) $(varE _arr)|]
275271 , noBindS
276272 (caseE
277273 (sigE (varE _tag) (conT tagType))
@@ -304,7 +300,7 @@ mkPokeExprTag tagType tagVal = pokeTag
304300 where
305301
306302 pokeTag =
307− [|pokeByteIndex
303+ [|pokeAt
308304 $(varE _initialOffset)
309305 $(varE _arr)
310306 $((sigE (litE (IntegerL (fromIntegral tagVal))) (conT tagType)))|]
@@ -322,7 +318,7 @@ mkPokeExprFields tagSize fields = do
322318
323319 numFields = length fields
324320 pokeField i =
325− [|pokeByteIndex
321+ [|pokeAt
326322 $(varE (mkOffsetName i))
327323 $(varE _arr)
328324 $(varE (mkFieldName i))|]
@@ -341,7 +337,9 @@ mkPokeExpr headTy cons =
341337 [|error
342338 ("Attempting to poke type with no constructors (" ++
343339 $(lift (pprint headTy)) ++ ")")|]
344− [(DataCon _ _ _ [])] -> [|pure ()|]
340+ -- XXX We don't gaurentee encoded equivalilty for Unbox. Does it still
341+ -- make sense to encode a default value for unit constructor?
342+ [(DataCon _ _ _ [])] -> [|pure ()|] -- mkPokeExprTag ''Word8 0
345343 [(DataCon cname _ _ fields)] ->
346344 caseE
347345 (varE _val)
@@ -397,8 +395,8 @@ mkPokeExpr headTy cons =
397395-- [(Nothing, (ConT ''Bool)), (Nothing, (VarT (mkName "b")))]
398396-- ])
399397-- @
400−deriveUnboxInternal :: Cxt -> Type -> [DataCon] -> Q [Dec]
401−deriveUnboxInternal preds headTy cons = do
398+deriveUnboxInternal :: Type -> [DataCon] -> ([Dec] -> Q [Dec]) -> Q [Dec]
399+deriveUnboxInternal headTy cons mkDec = do
402400 sizeOfMethod <- mkSizeOfExpr headTy cons
403401 peekMethod <- mkPeekExpr headTy cons
404402 pokeMethod <- mkPokeExpr headTy cons
@@ -407,9 +405,9 @@ deriveUnboxInternal preds headTy cons = do
407405 -- none
408406 [ -- PragmaD (InlineP 'sizeOf Inline FunLike AllPhases)
409407 FunD 'sizeOf [Clause [WildP] (NormalB sizeOfMethod) []]
410− , PragmaD (InlineP 'peekByteIndex Inline FunLike AllPhases)
408+ , PragmaD (InlineP 'peekAt Inline FunLike AllPhases)
411409 , FunD
412− 'peekByteIndex
410+ 'peekAt
413411 [ Clause
414412 (if isUnitType cons
415413 then [WildP, WildP]
@@ -417,9 +415,9 @@ deriveUnboxInternal preds headTy cons = do
417415 (NormalB peekMethod)
418416 []
419417 ]
420− , PragmaD (InlineP 'pokeByteIndex Inline FunLike AllPhases)
418+ , PragmaD (InlineP 'pokeAt Inline FunLike AllPhases)
421419 , FunD
422− 'pokeByteIndex
420+ 'pokeAt
423421 [ Clause
424422 (if isUnitType cons
425423 then [WildP, WildP, WildP]
@@ -428,71 +426,73 @@ deriveUnboxInternal preds headTy cons = do
428426 []
429427 ]
430428 ]
431− return [InstanceD Nothing preds (AppT (ConT ''Unbox) headTy) methods]
429+ mkDec methods
432430
433−-- | Template haskell helper to create instances of 'Unbox' automatically.
434−--
435−-- Consider the datatype:
436−-- @
437−-- data CustomDataType a b c = ...
438−-- @
431+-- | Given an 'Unbox' instance declaration splice without the methods (e.g.
432+-- @[d|instance Unbox a => Unbox (Maybe a)|]@), generate an instance
433+-- declaration including all the type class method implementations.
439434--
440−-- Usage: @$(deriveUnbox ''CustomDataType)@
435+-- Usage:
441436--
442−-- Note: All type variables automatcally get an "Unbox" constraint.
443−-- The derived code will look like the following,
444437-- @
445−-- instance (Unbox a, Unbox b, Unbox c) => Unbox (CustomDataType a b c) where
446−-- ...
438+-- \$(deriveUnbox [d|instance Unbox a => Unbox (Maybe a)|])
447439-- @
448−--
449−-- To control which type variables get the Unbox constraint, use
450−-- 'deriveUnboxWith'.
451−deriveUnbox :: Name -> Q [Dec]
452−deriveUnbox name = do
453− dt <- reifyDataType name
454− let preds = map (unboxPred . VarT) (dtTvs dt)
455− headTy = appsT (ConT name) (map VarT (dtTvs dt))
456− cons = dtCons dt
457− deriveUnboxInternal preds headTy cons
440+deriveUnbox :: Q [Dec] -> Q [Dec]
441+deriveUnbox mDecs = do
442+ dec <- mDecs
443+ case dec of
444+ [InstanceD mo preds headTyWC []] -> do
445+ let headTy = unwrap dec headTyWC
446+ (mainTyName, subs) = getMainTypeName dec headTy
447+ dt <- reifyDataType mainTyName
448+ let tyVars = dtTvs dt
449+ mapper = mapperWith (VarT <$> tyVars) subs
450+ cons = map (modifyConVariables mapper) (dtCons dt)
451+ deriveUnboxInternal headTy cons (mkInst mo preds headTyWC)
452+ _ -> errorMessage dec
458453
459454 where
460455
461− unboxPred ty =
462−#if MIN_VERSION_template_haskell(2,10,0)
463− AppT (ConT ''Unbox) ty
464−#else
465− ClassP ''Unbox [ty]
466−#endif
456+ mapperWith l1 l2 a =
457+ case elemIndex a l1 of
458+ Nothing -> a
459+ -- XXX Capture this case and give a relavant error.
460+ Just i -> l2 !! i
461+
462+ mapType f (AppT t1 t2) = AppT (mapType f t1) (mapType f t2)
463+ mapType f (InfixT t1 n t2) = InfixT (mapType f t1) n (mapType f t2)
464+ mapType f (UInfixT t1 n t2) = UInfixT (mapType f t1) n (mapType f t2)
465+ mapType f (ParensT t) = ParensT (mapType f t)
466+ mapType f v = f v
467+
468+ modifyConVariables f con =
469+ con { dcFields = map (\(a, b) -> (a, mapType f b)) (dcFields con) }
470+
471+ mkInst mo preds headTyWC methods =
472+ pure [InstanceD mo preds headTyWC methods]
473+
474+ errorMessage dec =
475+ error $ unlines
476+ [ "Error: deriveUnbox:"
477+ , ""
478+ , ">> " ++ pprint dec
479+ , ""
480+ , "The supplied declaration not a valid instance declaration."
481+ , "Provide a valid Haskell instance declaration without a body."
482+ , ""
483+ , "Examples:"
484+ , "instance Unbox (Proxy a)"
485+ , "instance Unbox a => Unbox (Identity a)"
486+ , "instance Unbox (TableT Identity)"
487+ ]
467488
468−-- | Like 'deriveUnbox' but control which type variables get the 'Unbox'
469−-- constraint.
470−--
471−-- Consider the datatype:
472−-- @
473−-- data CustomDataType a b c = ...
474−-- @
475−--
476−-- Usage: @$(deriveUnboxWith ["a", "c"] ''CustomDataType)@
477−--
478−-- @
479−-- instance (Unbox a, Unbox c) => Unbox (CustomDataType a b c) where
480−-- ...
481−-- @
482−--
483−deriveUnboxWith :: [String] -> Name -> Q [Dec]
484−deriveUnboxWith vars name = do
485− dt <- reifyDataType name
486− let preds = map (unboxPred . VarT) (fmap mkName vars)
487− headTy = appsT (ConT name) (map VarT (dtTvs dt))
488− cons = dtCons dt
489− deriveUnboxInternal preds headTy cons
489+ unwrap _ (AppT (ConT _) r) = r
490+ unwrap dec _ = errorMessage dec
490491
491− where
492+ getMainTypeName dec = go []
492493
493− unboxPred ty =
494−#if MIN_VERSION_template_haskell(2,10,0)
495− AppT (ConT ''Unbox) ty
496−#else
497− ClassP ''Unbox [ty]
498−#endif
494+ where
495+
496+ go xs (ConT nm) = (nm, xs)
497+ go xs (AppT l r) = go (r:xs) l
498+ go _ _ = errorMessage dec
core/src/Streamly/Internal/Data/Unfold.hs+15−13
@@ -82,7 +82,7 @@ module Streamly.Internal.Data.Unfold
8282 , dropWhileM
8383
8484 -- ** Cross product
85− , joinInnerGeneric
85+ , innerJoin
8686
8787 -- ** Resource Management
8888 -- | 'bracket' is the most general resource management operation, all other
@@ -205,7 +205,7 @@ swap = lmap Tuple.swap
205205--
206206{-# INLINE_NORMAL fold #-}
207207fold :: Monad m => Fold m b c -> Unfold m a b -> a -> m c
208−fold (Fold fstep initial extract) (Unfold ustep inject) a = do
208+fold (Fold fstep initial _ final) (Unfold ustep inject) a = do
209209 res <- initial
210210 case res of
211211 FL.Partial x -> inject a >>= go SPEC x
@@ -223,7 +223,7 @@ fold (Fold fstep initial extract) (Unfold ustep inject) a = do
223223 FL.Partial fs1 -> go SPEC fs1 s
224224 FL.Done c -> return c
225225 Skip s -> go SPEC fs s
226− Stop -> extract fs
226+ Stop -> final fs
227227
228228-- {-# ANN type FoldMany Fuse #-}
229229data FoldMany s fs b a
@@ -238,7 +238,7 @@ data FoldMany s fs b a
238238-- /Pre-release/
239239{-# INLINE_NORMAL foldMany #-}
240240foldMany :: Monad m => Fold m b c -> Unfold m a b -> Unfold m a c
241−foldMany (Fold fstep initial extract) (Unfold ustep inject1) =
241+foldMany (Fold fstep initial _ final) (Unfold ustep inject1) =
242242 Unfold step inject
243243
244244 where
@@ -269,14 +269,14 @@ foldMany (Fold fstep initial extract) (Unfold ustep inject1) =
269269 case r of
270270 Yield x s -> consume x s fs
271271 Skip s -> return $ Skip (FoldManyFirst fs s)
272− Stop -> return Stop
272+ Stop -> final fs >> return Stop
273273 step (FoldManyLoop st fs) = do
274274 r <- ustep st
275275 case r of
276276 Yield x s -> consume x s fs
277277 Skip s -> return $ Skip (FoldManyLoop s fs)
278278 Stop -> do
279− b <- extract fs
279+ b <- final fs
280280 return $ Skip (FoldManyYield b FoldManyDone)
281281 step (FoldManyYield b next) = return $ Yield b next
282282 step FoldManyDone = return Stop
@@ -317,7 +317,7 @@ either (Unfold stepL injectL) (Unfold stepR injectR) = Unfold step inject
317317-- /Pre-release/
318318{-# INLINE_NORMAL postscan #-}
319319postscan :: Monad m => Fold m b c -> Unfold m a b -> Unfold m a c
320−postscan (Fold stepF initial extract) (Unfold stepU injectU) =
320+postscan (Fold stepF initial extract final) (Unfold stepU injectU) =
321321 Unfold step inject
322322
323323 where
@@ -340,7 +340,7 @@ postscan (Fold stepF initial extract) (Unfold stepU injectU) =
340340 v <- extract fs1
341341 return $ Yield v (Just (fs1, s))
342342 Skip s -> return $ Skip (Just (fs, s))
343− Stop -> return Stop
343+ Stop -> final fs >> return Stop
344344
345345 step Nothing = return Stop
346346
@@ -348,7 +348,7 @@ data ScanState s f = ScanInit s | ScanDo s !f | ScanDone
348348
349349{-# INLINE_NORMAL scanWith #-}
350350scanWith :: Monad m => Bool -> Fold m b c -> Unfold m a b -> Unfold m a c
351−scanWith restart (Fold fstep initial extract) (Unfold stepU injectU) =
351+scanWith restart (Fold fstep initial extract final) (Unfold stepU injectU) =
352352 Unfold step inject
353353
354354 where
@@ -373,7 +373,7 @@ scanWith restart (Fold fstep initial extract) (Unfold stepU injectU) =
373373 case res of
374374 Yield x s -> runStep s (fstep fs x)
375375 Skip s -> return $ Skip $ ScanDo s fs
376− Stop -> return Stop
376+ Stop -> final fs >> return Stop
377377 step ScanDone = return Stop
378378
379379-- | Scan the output of an 'Unfold' to change it in a stateful manner.
@@ -677,10 +677,12 @@ dropWhileM f (Unfold step inject) = Unfold step' inject'
677677dropWhile :: Monad m => (b -> Bool) -> Unfold m a b -> Unfold m a b
678678dropWhile f = dropWhileM (return . f)
679679
680−{-# INLINE_NORMAL joinInnerGeneric #-}
681−joinInnerGeneric :: Monad m =>
680+-- | Cross intersection of two unfolds. See
681+-- 'Streamly.Internal.Data.Stream.innerJoin' for more details.
682+{-# INLINE_NORMAL innerJoin #-}
683+innerJoin :: Monad m =>
682684 (b -> c -> Bool) -> Unfold m a b -> Unfold m a c -> Unfold m a (b, c)
683−joinInnerGeneric eq s1 s2 = filter (\(a, b) -> a `eq` b) $ cross s1 s2
685+innerJoin eq s1 s2 = filter (\(a, b) -> a `eq` b) $ cross s1 s2
684686
685687------------------------------------------------------------------------------
686688-- Exceptions
core/src/Streamly/Internal/FileSystem/File.hs+11−13
@@ -103,15 +103,14 @@ import qualified Control.Monad.Catch as MC
103103import qualified System.IO as SIO
104104
105105import Streamly.Data.Fold (groupsOf, drain)
106−import Streamly.Internal.Data.Array.Type (Array(..), pinnedWriteNUnsafe)
106+import Streamly.Internal.Data.Array.Type (Array(..))
107107import Streamly.Internal.Data.Fold.Type (Fold(..))
108108import Streamly.Data.Stream (Stream)
109−import Streamly.Internal.Data.Unbox (Unbox)
110109import Streamly.Internal.Data.Unfold.Type (Unfold(..))
111110-- import Streamly.String (encodeUtf8, decodeUtf8, foldLines)
112111import Streamly.Internal.System.IO (defaultChunkSize)
113112
114−import qualified Streamly.Data.Array as A
113+import qualified Streamly.Internal.Data.Array as A
115114import qualified Streamly.Data.Stream as S
116115import qualified Streamly.Data.Unfold as UF
117116import qualified Streamly.Internal.Data.Array.Type as IA (pinnedChunksOf)
@@ -339,10 +338,6 @@ readWithBufferOf = readerWith
339338reader :: (MonadIO m, MonadCatch m) => Unfold m FilePath Word8
340339reader = UF.many A.reader (usingFile FH.chunkReader)
341340
342−{-# INLINE concatChunks #-}
343−concatChunks :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
344−concatChunks = S.unfoldMany A.reader
345−
346341-- | Generate a stream of bytes from a file specified by path. The stream ends
347342-- when EOF is encountered. File is locked using multiple reader and single
348343-- writer locking mode.
@@ -351,7 +346,7 @@ concatChunks = S.unfoldMany A.reader
351346--
352347{-# INLINE read #-}
353348read :: (MonadIO m, MonadCatch m) => FilePath -> Stream m Word8
354−read file = concatChunks $ withFile file ReadMode FH.readChunks
349+read file = A.concat $ withFile file ReadMode FH.readChunks
355350
356351{-# DEPRECATED toBytes "Please use 'read' instead" #-}
357352{-# INLINE toBytes #-}
@@ -438,7 +433,7 @@ write = toHandleWith A.defaultChunkSize
438433{-# INLINE writeChunks #-}
439434writeChunks :: (MonadIO m, MonadCatch m)
440435 => FilePath -> Fold m (Array a) ()
441−writeChunks path = Fold step initial extract
436+writeChunks path = Fold step initial extract final
442437 where
443438 initial = do
444439 h <- liftIO (openFile path WriteMode)
@@ -448,12 +443,15 @@ writeChunks path = Fold step initial extract
448443 step (fld, h) x = do
449444 r <- FL.snoc fld x `MC.onException` liftIO (hClose h)
450445 return $ FL.Partial (r, h)
451− extract (Fold _ initial1 extract1, h) = do
446+
447+ extract _ = return ()
448+
449+ final (Fold _ initial1 _ final1, h) = do
452450 liftIO $ hClose h
453451 res <- initial1
454452 case res of
455− FL.Partial fs -> extract1 fs
456− FL.Done fb -> return fb
453+ FL.Partial fs -> final1 fs
454+ FL.Done () -> return ()
457455
458456-- | @writeWith chunkSize handle@ writes the input stream to @handle@.
459457-- Bytes in the input stream are collected into a buffer until we have a chunk
@@ -464,7 +462,7 @@ writeChunks path = Fold step initial extract
464462writeWith :: (MonadIO m, MonadCatch m)
465463 => Int -> FilePath -> Fold m Word8 ()
466464writeWith n path =
467− groupsOf n (pinnedWriteNUnsafe n) (writeChunks path)
465+ groupsOf n (A.unsafePinnedCreateOf n) (writeChunks path)
468466
469467{-# DEPRECATED writeWithBufferOf "Please use 'writeWith' instead" #-}
470468{-# INLINE writeWithBufferOf #-}
core/src/Streamly/Internal/FileSystem/Handle.hs+21−28

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/Path.hsnew file+515

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/Parser.hs+3

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/Stream.hs+39−11

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/String.hs+7−7

This diff is too big to show here.

core/streamly-core.cabal+83−35

This diff is too big to show here.

docs/Developer/optimization-guidelines.md+12

This diff is too big to show here.

docs/Developer/paths.rst+25−2

This diff is too big to show here.

docs/User/Project/ApiChangelogs/0.9.0-0.10.0.txtnew file+424

This diff is too big to show here.

docs/User/Project/CONTRIBUTORS.md+8

This diff is too big to show here.

docs/User/Project/Changelog.md+25−10

This diff is too big to show here.

docs/User/Project/README.md+9−2

This diff is too big to show here.

docs/User/Project/Upgrading.md → docs/User/Project/Upgrading-0.8-to-0.9.mdrenamed+87−9

This diff is too big to show here.

docs/User/Tutorials/quick-overview.md+10−14

This diff is too big to show here.

src/Streamly.hsdeleted−625

This diff is too big to show here.

src/Streamly/Data/Fold/Prelude.hs+31−13

This diff is too big to show here.

src/Streamly/Data/Fold/Tee.hs+1−1

This diff is too big to show here.

src/Streamly/Data/Stream/Concurrent.hsdeleted−194

This diff is too big to show here.

src/Streamly/Data/Stream/Exception.hsdeleted−25

This diff is too big to show here.

src/Streamly/Data/Stream/MkType.hs+41−10

This diff is too big to show here.

src/Streamly/Data/Stream/Prelude.hs+217−15

This diff is too big to show here.

src/Streamly/Data/Stream/Time.hsdeleted−37

This diff is too big to show here.

src/Streamly/Data/Unicode/Stream.hsdeleted−99

This diff is too big to show here.

src/Streamly/Internal/Data/Array/ArrayMacros.hdeleted−40

This diff is too big to show here.

src/Streamly/Internal/Data/Channel.hsnew file+19

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Channel/Dispatcher.hs → src/Streamly/Internal/Data/Channel/Dispatcher.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Channel/Types.hs → src/Streamly/Internal/Data/Channel/Types.hsrenamed+12−37

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Channel/Worker.hs → src/Streamly/Internal/Data/Channel/Worker.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Async.hs+5−2

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent.hs+1−44

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent/Channel.hs+29−10

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent/Channel/Type.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Prelude.hsnew file+19

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/SVar.hs+6−5

This diff is too big to show here.

src/Streamly/Internal/Data/IsMap/HashMap.hs+6

This diff is too big to show here.

src/Streamly/Internal/Data/SVar.hs+1

This diff is too big to show here.

src/Streamly/Internal/Data/SmallArray.hs+1−1

This diff is too big to show here.

core/src/Streamly/Internal/Data/Stream/Common.hs → src/Streamly/Internal/Data/Stream/Common.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent.hs+50−39

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel.hs+17−14

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Append.hs+10−10

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Consumer.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Dispatcher.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Interleave.hs+8−8

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Operations.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Type.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream.hs+6

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Common.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Exception.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Expand.hs+5

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Generate.hs+4−4

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Reduce.hs+4−4

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Top.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Exception/Lifted.hs → src/Streamly/Internal/Data/Stream/Lifted.hsrenamed+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/MkType.hs+33−34

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Prelude.hsnew file+19

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/SVar.hsnew file+18

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Serial.hs+5

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Time.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Zip.hs+37

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Zip/Concurrent.hsdeleted−50

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/ZipAsync.hs+2−1

This diff is too big to show here.

src/Streamly/Internal/Data/Unfold/Prelude.hsnew file+17

This diff is too big to show here.

src/Streamly/Internal/Data/Unfold/SVar.hs+1

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event.hs+3

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Darwin.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Linux.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Windows.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/FileSystem/FD.hs+7−8

This diff is too big to show here.

src/Streamly/Internal/Network/Inet/TCP.hs+76−44

This diff is too big to show here.

src/Streamly/Internal/Network/Socket.hs+17−14

This diff is too big to show here.

src/Streamly/Internal/Unicode/Char.hs+3−3

This diff is too big to show here.

src/Streamly/Memory/Array.hsdeleted−74

This diff is too big to show here.

src/Streamly/Network/Inet/TCP.hs+41−30

This diff is too big to show here.

src/Streamly/Network/Socket.hs+16−30

This diff is too big to show here.

streamly.cabal+69−56

This diff is too big to show here.

targets/Targets.hs+4−1

This diff is too big to show here.

test/Streamly/Test/Data/Array.hs+3−3

This diff is too big to show here.

test/Streamly/Test/Data/Array/Stream.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Fold.hs+7−6

This diff is too big to show here.

test/Streamly/Test/Data/MutArray.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Parser.hs+8−6

This diff is too big to show here.

test/Streamly/Test/Data/ParserK.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Data/Serialize.hs+49−20

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/CompatV0.hs+4−4

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/CompatV1.hs+4−4

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/TH.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Data/Stream/Concurrent.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Unbox.hs+47−50

This diff is too big to show here.

test/Streamly/Test/Data/Unbox/TH.hs+11−10

This diff is too big to show here.

test/Streamly/Test/FileSystem/Event/Common.hs+1−1

This diff is too big to show here.

test/Streamly/Test/FileSystem/Handle.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Prelude/Serial.hs+3−3

This diff is too big to show here.

test/Streamly/Test/Serialize/Serializable.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Unicode/Char.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Unicode/Stream.hs+1−1

This diff is too big to show here.

test/lib/Streamly/Test/Prelude/Common.hs+1−1

This diff is too big to show here.

test/streamly-tests.cabal+8−7

This diff is too big to show here.