git.iko.soy/streamly
on a309bf7 help sign in
30c5a56
swap
0 ahead, 224 behind
233 files changed+6472−11549
since splitsnapshots
unifiedsplit
.circleci/config.yml+277−73 .github/workflows/haskell.yml+91−84 .hlint.ignore−5 benchmark/NanoBenchmarks.hs+1−1 benchmark/Streamly/Benchmark/Data/Array/CommonImports.hs+1−1 benchmark/Streamly/Benchmark/Data/Array/Stream.hs+3−3 benchmark/Streamly/Benchmark/Data/Fold.hs+3−3 benchmark/Streamly/Benchmark/Data/Fold/Window.hs+1−1 benchmark/Streamly/Benchmark/Data/MutArray.hs+1−1 benchmark/Streamly/Benchmark/Data/Parser.hs+6−7 benchmark/Streamly/Benchmark/Data/ParserK.hs+6−6 benchmark/Streamly/Benchmark/Data/Ring/Unboxed.hs+1−1 benchmark/Streamly/Benchmark/Data/Serialize.hs+51−89 benchmark/Streamly/Benchmark/Data/Serialize/RecCompatible.hs−24 benchmark/Streamly/Benchmark/Data/Serialize/RecNonCompatible.hs−21 benchmark/Streamly/Benchmark/Data/Serialize/TH.hs−76 benchmark/Streamly/Benchmark/Data/Stream/Common.hs+1−3 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentCommon.hs+3−3 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentEager.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentInterleaved.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/ConcurrentOrdered.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Eliminate.hs+1−3 benchmark/Streamly/Benchmark/Data/Stream/Exceptions.hs+3−12 benchmark/Streamly/Benchmark/Data/Stream/Expand.hs+1−9 benchmark/Streamly/Benchmark/Data/Stream/Generate.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Lift.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/Reduce.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/Split.hs+1−1 benchmark/Streamly/Benchmark/Data/Stream/StreamD.hs+1−3 benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs+1−3 benchmark/Streamly/Benchmark/Data/Stream/StreamKAlt.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/ToStreamK.hs+2−2 benchmark/Streamly/Benchmark/Data/Stream/Transform.hs+2−2 benchmark/Streamly/Benchmark/Data/Unfold.hs+2−11 benchmark/Streamly/Benchmark/FileSystem/Handle.hs+1−1 benchmark/Streamly/Benchmark/FileSystem/Handle/Read.hs+9−8 benchmark/Streamly/Benchmark/FileSystem/Handle/ReadWrite.hs+10−9 benchmark/Streamly/Benchmark/Prelude/Adaptive.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Ahead.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Async.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Concurrent.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Merge.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Parallel.hs+1−1 benchmark/Streamly/Benchmark/Prelude/Rate.hs+1−1 benchmark/Streamly/Benchmark/Prelude/WAsync.hs+1−1 benchmark/Streamly/Benchmark/Prelude/WSerial.hs+1−1 benchmark/Streamly/Benchmark/Prelude/ZipAsync.hs+1−1 benchmark/Streamly/Benchmark/Prelude/ZipSerial.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Char.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Parser.hs+1−1 benchmark/Streamly/Benchmark/Unicode/Stream.hs+6−24 benchmark/Streamly/Benchmark/Unicode/Utf8.hs+1−1 benchmark/bench-runner/Main.hs+4−32 benchmark/lib/Streamly/Benchmark/Common.hs+1−1 benchmark/lib/Streamly/Benchmark/Common/Handle.hs+1−3 benchmark/lib/Streamly/Benchmark/Prelude.hs+1−1 benchmark/streamly-benchmarks.cabal+27−23 configure+9−9 configure.ac+1−1 core/configure+9−9 core/configure.ac+1−1 core/docs/ApiChangelogs/0.1.0-0.2.0.txt−1471 core/docs/ApiChangelogs/0.2.0-0.2.2.txt−306 core/docs/Changelog.md+16−58 core/src/DocTestDataArray.hs−3 core/src/DocTestDataParserK.hs−15 core/src/DocTestDataStream.hs−2 core/src/DocTestFileSystemHandle.hs+1−1 core/src/Streamly/Console/Stdio.hs+27−1 core/src/Streamly/Data/Array.hs+39−40 core/src/Streamly/Data/Array/Generic.hs+13−16 core/src/Streamly/Data/Fold.hs+118−150 core/src/Streamly/Data/MutArray.hs+12−18 core/src/Streamly/Data/MutArray/Generic.hs+3−8 core/src/Streamly/Data/MutByteArray.hs−106 core/src/Streamly/Data/Parser.hs+60−130 core/src/Streamly/Data/ParserK.hs+3−89 core/src/Streamly/Data/Serialize.hs+50 core/src/Streamly/Data/Stream.hs+36−131 core/src/Streamly/Data/StreamK.hs+1−2 core/src/Streamly/Data/Unfold.hs−2 core/src/Streamly/FileSystem/Handle.hs+30−7 core/src/Streamly/Internal/Data/Array.hs+135−608 core/src/Streamly/Internal/Data/Array/Generic.hs+6−22 core/src/Streamly/Internal/Data/Array/Type.hs+109−518 core/src/Streamly/Internal/Data/Builder.hs−2 core/src/Streamly/Internal/Data/Fold/Chunked.hs+15−18 core/src/Streamly/Internal/Data/Fold/Combinators.hs+137−81 core/src/Streamly/Internal/Data/Fold/Container.hs+28−73 core/src/Streamly/Internal/Data/Fold/Step.hs+1−1 core/src/Streamly/Internal/Data/Fold/Tee.hs−2 core/src/Streamly/Internal/Data/Fold/Type.hs+114−338 core/src/Streamly/Internal/Data/Fold/Window.hs+4−10 core/src/Streamly/Internal/Data/IORef/Unboxed.hs+19−22 core/src/Streamly/Internal/Data/IsMap.hs−8 core/src/Streamly/Internal/Data/MutArray.hs+33−154 core/src/Streamly/Internal/Data/MutArray/Generic.hs+30−65 core/src/Streamly/Internal/Data/MutArray/Stream.hs−145 core/src/Streamly/Internal/Data/MutArray/Type.hs+417−1272 core/src/Streamly/Internal/Data/MutByteArray.hs−251 core/src/Streamly/Internal/Data/MutByteArray/Type.hs−321 core/src/Streamly/Internal/Data/Parser.hs+119−126 core/src/Streamly/Internal/Data/Parser/Type.hs+70−93 core/src/Streamly/Internal/Data/ParserK/Type.hs+65−28 core/src/Streamly/Internal/Data/Ring.hs+4−7 core/src/Streamly/Internal/Data/Ring/Generic.hs+1−1 core/src/Streamly/Internal/Data/Serialize.hs+122 core/src/Streamly/Internal/Data/Serialize/TH.hs+111−118 core/src/Streamly/Internal/Data/Serialize/TH/Bottom.hs+39−153 core/src/Streamly/Internal/Data/Serialize/TH/RecHeader.hs+98−181 core/src/Streamly/Internal/Data/Serialize/Type.hs+123−142 core/src/Streamly/Internal/Data/Array/Stream.hs → core/src/Streamly/Internal/Data/Stream/Chunked.hs+586−74 src/Streamly/Internal/Data/Stream/Common.hs → core/src/Streamly/Internal/Data/Stream/Common.hs+3−3 core/src/Streamly/Internal/Data/Stream/Container.hs+48−56 core/src/Streamly/Internal/Data/Stream/Eliminate.hs+25−33 core/src/Streamly/Internal/Data/Stream/Exception.hs+6−9 core/src/Streamly/Internal/Data/Stream/Generate.hs+12−2 core/src/Streamly/Internal/Data/Stream/MutChunked.hs+323 core/src/Streamly/Internal/Data/Stream/Nesting.hs+202−86 core/src/Streamly/Internal/Data/Stream/Top.hs+116−243 core/src/Streamly/Internal/Data/Stream/Transform.hs+35−61 core/src/Streamly/Internal/Data/Stream/Transformer.hs+1−2 core/src/Streamly/Internal/Data/Stream/Type.hs+14−147 core/src/Streamly/Internal/Data/StreamK.hs+40−35 core/src/Streamly/Internal/Data/StreamK/Type.hs+11−82 core/src/Streamly/Internal/Data/Time/Units.hs+3 core/src/Streamly/Internal/Data/Unbox.hs+289−165 core/src/Streamly/Internal/Data/Unbox/TH.hs+74−74 core/src/Streamly/Internal/Data/Unfold.hs+13−15 core/src/Streamly/Internal/FileSystem/File.hs+13−11 core/src/Streamly/Internal/FileSystem/Handle.hs+28−21 core/src/Streamly/Internal/FileSystem/Path.hs−515 core/src/Streamly/Internal/Data/Binary/Parser.hs → core/src/Streamly/Internal/Serialize/FromBytes.hs+2−8 core/src/Streamly/Internal/Data/Binary/Stream.hs → core/src/Streamly/Internal/Serialize/ToBytes.hs+2−2 core/src/Streamly/Internal/Unicode/Parser.hs−3 core/src/Streamly/Internal/Unicode/Stream.hs+11−39 core/src/Streamly/Internal/Unicode/String.hs+7−7 core/streamly-core.cabal+41−95 docs/Developer/optimization-guidelines.md−12 docs/Developer/paths.rst+2−25 docs/User/HowTo/Compiling.md+3−5 docs/User/Project/ApiChangelogs/0.9.0-0.10.0.txt−424 docs/User/Project/CONTRIBUTORS.md−8 docs/User/Project/Changelog.md+10−25 docs/User/Project/README.md+2−9 docs/User/Project/Upgrading-0.8-to-0.9.md → docs/User/Project/Upgrading.md+9−87 docs/User/Tutorials/quick-overview.md+14−10 docs/streamly-docs.cabal+1−1 src/Streamly.hs+625 src/Streamly/Data/Fold/Prelude.hs+13−31 src/Streamly/Data/Fold/Tee.hs+1−1 src/Streamly/Data/Stream/Concurrent.hs+194 src/Streamly/Data/Stream/Exception.hs+25 src/Streamly/Data/Stream/MkType.hs+10−41 src/Streamly/Data/Stream/Prelude.hs+15−217 src/Streamly/Data/Stream/Time.hs+37 src/Streamly/Data/Unicode/Stream.hs+99 src/Streamly/Internal/Data/Array/ArrayMacros.h+40 src/Streamly/Internal/Data/Channel.hs−19 src/Streamly/Internal/Data/Fold/Async.hs+2−5 src/Streamly/Internal/Data/Fold/Concurrent.hs+44−1 src/Streamly/Internal/Data/Fold/Concurrent/Channel.hs+10−29 src/Streamly/Internal/Data/Fold/Concurrent/Channel/Type.hs+3−3 src/Streamly/Internal/Data/Fold/Prelude.hs−19 src/Streamly/Internal/Data/Fold/SVar.hs+5−6 src/Streamly/Internal/Data/IsMap/HashMap.hs−6 src/Streamly/Internal/Data/SVar.hs−1 src/Streamly/Internal/Data/SmallArray.hs+1−1 src/Streamly/Internal/Data/Channel/Dispatcher.hs → src/Streamly/Internal/Data/Stream/Channel/Dispatcher.hs+3−3 src/Streamly/Internal/Data/Channel/Types.hs → src/Streamly/Internal/Data/Stream/Channel/Types.hs+37−12 src/Streamly/Internal/Data/Channel/Worker.hs → src/Streamly/Internal/Data/Stream/Channel/Worker.hs+3−3 src/Streamly/Internal/Data/Stream/Concurrent.hs+39−50 src/Streamly/Internal/Data/Stream/Concurrent/Channel.hs+14−17 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Append.hs+10−10 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Consumer.hs+2−2 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Dispatcher.hs+3−3 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Interleave.hs+8−8 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Operations.hs+1−1 src/Streamly/Internal/Data/Stream/Concurrent/Channel/Type.hs+3−3 src/Streamly/Internal/Data/Stream/Lifted.hs → src/Streamly/Internal/Data/Stream/Exception/Lifted.hs+2−2 src/Streamly/Internal/Data/Stream/IsStream.hs−6 src/Streamly/Internal/Data/Stream/IsStream/Common.hs+2−2 src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs+1−1 src/Streamly/Internal/Data/Stream/IsStream/Exception.hs+1−1 src/Streamly/Internal/Data/Stream/IsStream/Expand.hs−5 src/Streamly/Internal/Data/Stream/IsStream/Generate.hs+4−4 src/Streamly/Internal/Data/Stream/IsStream/Reduce.hs+4−4 src/Streamly/Internal/Data/Stream/IsStream/Top.hs+1−1 src/Streamly/Internal/Data/Stream/MkType.hs+34−33 src/Streamly/Internal/Data/Stream/Prelude.hs−19 src/Streamly/Internal/Data/Stream/SVar.hs−18 src/Streamly/Internal/Data/Stream/SVar/Generate.hs−2 src/Streamly/Internal/Data/Stream/Serial.hs−5 src/Streamly/Internal/Data/Stream/Time.hs+2−2 src/Streamly/Internal/Data/Stream/Zip.hs−39 src/Streamly/Internal/Data/Stream/Zip/Concurrent.hs+50 src/Streamly/Internal/Data/Stream/ZipAsync.hs+1−2 src/Streamly/Internal/Data/Unfold/Prelude.hs−17 src/Streamly/Internal/Data/Unfold/SVar.hs−1 src/Streamly/Internal/FileSystem/Event.hs−3 src/Streamly/Internal/FileSystem/Event/Darwin.hs+1−1 src/Streamly/Internal/FileSystem/Event/Linux.hs+2−2 src/Streamly/Internal/FileSystem/Event/Windows.hs+2−2 src/Streamly/Internal/FileSystem/FD.hs+8−7 src/Streamly/Internal/Network/Inet/TCP.hs+44−76 src/Streamly/Internal/Network/Socket.hs+14−17 src/Streamly/Internal/Unicode/Char.hs+3−7 src/Streamly/Memory/Array.hs+74 src/Streamly/Network/Inet/TCP.hs+30−41 src/Streamly/Network/Socket.hs+30−16 streamly.cabal+62−82 targets/Targets.hs+2−5 test/Streamly/Test/Data/Array.hs+3−3 test/Streamly/Test/Data/Array/Stream.hs+1−1 test/Streamly/Test/Data/Fold.hs+6−7 test/Streamly/Test/Data/MutArray.hs+1−1 test/Streamly/Test/Data/Parser.hs+6−12 test/Streamly/Test/Data/ParserK.hs+2−7 test/Streamly/Test/Data/Serialize.hs+26−102 test/Streamly/Test/Data/Serialize/CompatV0.hs−46 test/Streamly/Test/Data/Serialize/CompatV1.hs−48 test/Streamly/Test/Data/Serialize/TH.hs+2−2 test/Streamly/Test/Data/Stream/Concurrent.hs+1−1 test/Streamly/Test/Data/Unbox.hs+50−47 test/Streamly/Test/Data/Unbox/TH.hs+10−11 test/Streamly/Test/FileSystem/Event/Common.hs+1−1 test/Streamly/Test/FileSystem/Handle.hs+2−2 test/Streamly/Test/Prelude/Serial.hs+3−7 test/Streamly/Test/Serialize/Serializable.hs+2−2 test/Streamly/Test/Unicode/Char.hs+1−1 test/Streamly/Test/Unicode/Stream.hs+1−1 test/lib/Streamly/Test/Prelude/Common.hs+2−9 test/streamly-tests.cabal+13−20
.circleci/config.yml+277−73
@@ -1,7 +1,7 @@
11version: 2.1
22
33#-----------------------------------------------------------------------------
4−# packcheck-0.7.0
4+# packcheck-0.4.2
55# Packcheck global environment variables
66#-----------------------------------------------------------------------------
77
@@ -11,8 +11,6 @@ env: &env
1111 # Common options
1212 # ------------------------------------------------------------------------
1313 # GHC_OPTIONS: "-Werror"
14− # For updating see: https://downloads.haskell.org/~ghcup/
15− GHCUP_VERSION: 0.1.20.0
1614 CABAL_REINIT_CONFIG: "y"
1715 LC_ALL: "C.UTF-8"
1816
@@ -24,6 +22,7 @@ env: &env
2422 # DISABLE_DOCS: "y"
2523 # DISABLE_SDIST_BUILD: "y"
2624 # DISABLE_DIST_CHECKS: "y"
25+ ENABLE_INSTALL: "y"
2726
2827 # ------------------------------------------------------------------------
2928 # stack options
@@ -39,12 +38,15 @@ env: &env
3938 # cabal options
4039 # ------------------------------------------------------------------------
4140 CABAL_CHECK_RELAX: "y"
41+ CABAL_NO_SANDBOX: "y"
42+ CABAL_HACKAGE_MIRROR: "hackage.haskell.org:http://hackage.fpcomplete.com"
4243 CABAL_BUILD_OPTIONS: "-j1 --flag limit-build-mem"
4344
4445 # ------------------------------------------------------------------------
4546 # Where to find the required tools
4647 # ------------------------------------------------------------------------
47− PATH: /sbin:/usr/sbin:/bin:/usr/bin
48+ PATH: /opt/ghc/bin:/opt/ghcjs/bin:/sbin:/usr/sbin:/bin:/usr/bin
49+ TOOLS_DIR: /opt
4850
4951 # ------------------------------------------------------------------------
5052 # Location of packcheck.sh (the shell script invoked to perform CI tests ).
@@ -57,15 +59,15 @@ env: &env
5759 # If you have not committed packcheck.sh in your repo at PACKCHECK
5860 # then it is automatically pulled from this URL.
5961 PACKCHECK_GITHUB_URL: "https://raw.githubusercontent.com/composewell/packcheck"
60− PACKCHECK_GITHUB_COMMIT: "f6cfd7b7a33b4bf6fd63cce6a3fb3d26f5031403"
62+ PACKCHECK_GITHUB_COMMIT: "a302772c8f3791c477edd1764b82f8d1f5b22fdf"
6163
6264executors:
6365 amd64-executor:
6466 docker:
65− - image: ubuntu:latest
67+ - image: ubuntu:xenial
6668 x86-executor:
6769 docker:
68− - image: i386/ubuntu:focal
70+ - image: i386/ubuntu:eoan
6971
7072#-----------------------------------------------------------------------------
7173# Common utility stuff, not to be modified usually
@@ -77,50 +79,38 @@ preinstall: &preinstall
7779 # required for https/cache save and restore
7880 apt-get install -y ca-certificates
7981
80− # Alternative way of installing ghc and cabal, directly from
81− # haskell.org instead of using ghcup. NOTE: this is for Debian
82− # only and is debian release specific.
83− # gnupg is required for apt-key to work
84− #apt-get install -y gnupg
85− #apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA3CBA3FFE22B574
86− #echo "deb http://downloads.haskell.org/debian buster main" >> /etc/apt/sources.list
87− #apt-get update
88−
89− # required for outbound https for stack and for stack setup
90− apt-get install -y netbase xz-utils make
91− apt-get install -y zlib1g-dev
82+ # required to (re)generate the configure script
83+ apt-get install -y autoconf
9284
93− # For ghcup to install ghc
94− if test -n "$GHCUP_VERSION"
95− then
96− apt-get install -y gcc
97− apt-get install -y g++
98− fi
85+ # For ghc and cabal-install packages from hvr's ppa
86+ # gnupg is required for apt-key to work
87+ apt-get install -y gnupg
88+ apt-get install -y apt-transport-https
89+ apt-key adv --keyserver keyserver.ubuntu.com --recv-keys BA3CBA3FFE22B574
90+ apt-get install -y software-properties-common
91+ add-apt-repository -y ppa:hvr/ghc
92+ # echo "deb http://downloads.haskell.org/debian stretch main" >> /etc/apt/sources.list
93+ echo "deb-src http://ppa.launchpad.net/hvr/ghc/ubuntu xenial main" >> /etc/apt/sources.list
94+ apt-get update
9995
100− # libgmp required by ghc for linking
101− apt-get install -y libgmp-dev
102− apt-get install -y libtinfo-dev
96+ # required for packcheck
97+ apt-get install -y curl
10398
104− # Required by cabal when git URL is specified in project file
99+ # required for circleci.
100+ apt-get install -y libcurl4-gnutls-dev
105101 apt-get install -y git
102+ apt-get install -y libtinfo-dev
106103
107− # Required for and by packcheck
108− apt-get install -y curl
104+ # required for outbound https for stack and for stack setup
105+ apt-get install -y netbase xz-utils make
109106
107+project-preinstall: &project-preinstall
108+ run: |
110109 # Get packcheck if needed
111− if test ! -e "$PACKCHECK"
112− then
113− if test -z "$PACKCHECK_GITHUB_COMMIT"
114− then
115− die "PACKCHECK_GITHUB_COMMIT is not specified."
116− fi
117− PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
118− curl --fail -sL -o "$PACKCHECK" $PACKCHECK_URL || exit 1
119− chmod +x $PACKCHECK
120− elif test ! -x "$PACKCHECK"
121− then
122− chmod +x $PACKCHECK
123− fi
110+ CURL=$(which curl)
111+ PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
112+ if test ! -e "$PACKCHECK"; then $CURL -sL -o "$PACKCHECK" $PACKCHECK_URL; fi;
113+ chmod +x $PACKCHECK
124114
125115restore: &restore
126116 # Needs to happen after installing ca-certificates
@@ -136,71 +126,271 @@ save: &save
136126 save_cache:
137127 key: v1-{{ .Environment.CIRCLE_JOB }}-{{ .Branch }}
138128 paths:
139− - ~/.local
140129 - ~/.cabal
130+ - ~/.ghc
131+ - ~/.local
141132 - ~/.stack
142− - ~/.ghcup
143133
144134#-----------------------------------------------------------------------------
145135# Build matrix
146136#-----------------------------------------------------------------------------
147137
148138jobs:
149− cabal-ghc-8_6_5:
139+ cabal-ghc-8_10_4:
150140 <<: *env
151141 executor: amd64-executor
152142 steps:
143+ - *preinstall
153144 - checkout
145+ - *project-preinstall
154146 - *restore
155− - *preinstall
156147 - run:
157148 environment:
158− GHCVER: "8.6.5"
159− CABAL_VERSION: "3.6.2.0"
160149 CABAL_PROJECT: "cabal.project"
150+ DISABLE_TEST: "yes"
151+ DISABLE_BENCH: "yes"
161152 DISABLE_SDIST_BUILD: "yes"
162− CABAL_BUILD_OPTIONS: "--flag debug --flag -opt"
163153 command: |
154+ apt-get install -y ghc-8.10.4
155+ apt-get install -y cabal-install-3.4
164156 bash -c "$PACKCHECK cabal"
165157 - *save
166− cabal-ghc-9_8_1-docspec:
158+ stack-ghc-8_8:
167159 <<: *env
168160 executor: amd64-executor
169161 steps:
162+ - *preinstall
170163 - checkout
164+ - *project-preinstall
171165 - *restore
166+ - run:
167+ name: GHC 8.8 + stack lts-16.12
168+ environment:
169+ BUILD: "stack"
170+ RESOLVER: "lts-16.12"
171+ SDIST_OPTIONS: "--ignore-check"
172+ STACK_BUILD_OPTIONS: "--flag streamly-benchmarks:-opt"
173+ command: |
174+ apt-get build-dep -y ghc-8.8.4
175+ apt-get install -y cabal-install-3.2
176+ bash -c "$PACKCHECK $BUILD"
177+ - *save
178+ cabal-ghc-8_6_5:
179+ <<: *env
180+ executor: amd64-executor
181+ steps:
172182 - *preinstall
183+ - checkout
184+ - *project-preinstall
185+ - *restore
186+ - run: |
187+ apt-get install -y ghc-8.6.3
188+ apt-get install -y cabal-install-2.4
189+ bash -c "$PACKCHECK cabal-v2"
190+ - *save
191+ cabal-ghc-8_4_4:
192+ <<: *env
193+ executor: x86-executor
194+ steps:
195+ - *preinstall
196+ - checkout
197+ - *project-preinstall
198+ - *restore
199+ - run:
200+ name: GHC 8.4.4 + x86 + debug
201+ environment:
202+ BUILD: "cabal-v2"
203+ GHCVER: "8.4.4"
204+ CABAL_BUILD_OPTIONS: "--flag debug --flag limit-build-mem"
205+ command: |
206+ apt-get install -y ghc-8.4.4
207+ apt-get install -y cabal-install-3.2
208+ bash -c "$PACKCHECK $BUILD"
209+ - *save
210+ cabal-ghc-8_2_2:
211+ <<: *env
212+ executor: amd64-executor
213+ steps:
214+ - *preinstall
215+ - checkout
216+ - *project-preinstall
217+ - *restore
173218 - run:
174219 environment:
175− GHCVER: "9.8.1"
176− CABAL_VERSION: "3.10.1.0"
177− CABAL_PROJECT: "cabal.project.doctest"
178− DISABLE_SDIST_BUILD: "y"
179− DISABLE_TEST: "y"
180− DISABLE_BENCH: "y"
181− DISABLE_DOCS: "y"
182− ENABLE_DOCSPEC: "y"
183− DOCSPEC_URL: https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz
184− DOCSPEC_OPTIONS: "--timeout 60 --check-properties --property-variables xs"
220+ BUILD: "cabal-v2"
221+ DISABLE_TEST: "yes"
222+ DISABLE_BENCH: "yes"
223+ DISABLE_DOCS: "yes"
224+ DISABLE_SDIST_BUILD: "yes"
225+ DISABLE_DIST_CHECKS: "yes"
185226 command: |
186− bash -c "$PACKCHECK cabal"
227+ apt-get install -y ghc-8.2.2
228+ apt-get install -y cabal-install-3.2
229+ bash -c "$PACKCHECK $BUILD"
187230 - *save
188− hlint-trailing-spaces:
231+ cabal-ghc-8_0_2:
189232 <<: *env
190233 executor: amd64-executor
191234 steps:
235+ - *preinstall
192236 - checkout
237+ - *project-preinstall
193238 - *restore
239+ - run:
240+ environment:
241+ BUILD: "cabal-v2"
242+ DISABLE_TEST: "yes"
243+ DISABLE_BENCH: "yes"
244+ DISABLE_DOCS: "yes"
245+ DISABLE_SDIST_BUILD: "yes"
246+ DISABLE_DIST_CHECKS: "yes"
247+ command: |
248+ apt-get install -y ghc-8.0.2
249+ apt-get install -y cabal-install-3.2
250+ bash -c "$PACKCHECK $BUILD"
251+ - *save
252+ cabal-ghc-7_10_3:
253+ <<: *env
254+ executor: amd64-executor
255+ steps:
194256 - *preinstall
257+ - checkout
258+ - *project-preinstall
259+ - *restore
260+ - run: |
261+ apt-get install -y ghc-7.10.3
262+ apt-get install -y cabal-install-2.4
263+ bash -c "$PACKCHECK cabal-v2"
264+ - *save
265+ stack-ghc-8_4:
266+ <<: *env
267+ executor: amd64-executor
268+ steps:
269+ - *preinstall
270+ - checkout
271+ - *project-preinstall
272+ - *restore
273+ - run: |
274+ bash -c "$PACKCHECK stack RESOLVER=lts-12"
275+ - *save
276+ stack-ghc-8_2:
277+ <<: *env
278+ executor: amd64-executor
279+ steps:
280+ - *preinstall
281+ - checkout
282+ - *project-preinstall
283+ - *restore
284+ - run: |
285+ bash -c "$PACKCHECK stack RESOLVER=lts-11"
286+ - *save
287+ cabal-ghcjs-8_4:
288+ <<: *env
289+ executor: amd64-executor
290+ steps:
291+ - *preinstall
292+ - checkout
293+ - *project-preinstall
294+ - *restore
195295 - run:
196− name: hlint and trailing spaces
296+ name: GHCJS 8.4+no-test+no-docs
197297 environment:
298+ BUILD: cabal-v2
299+ ENABLE_GHCJS: "yes"
300+ DISABLE_TEST: "yes"
301+ ENABLE_INSTALL: ""
302+ DISABLE_DOCS: "yes"
198303 DISABLE_SDIST_BUILD: "yes"
199− HLINT_VERSION: 3.6.1
304+ command: |
305+ add-apt-repository -y ppa:hvr/ghcjs
306+ add-apt-repository -y 'deb https://deb.nodesource.com/node_11.x xenial main'
307+ curl -sSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | apt-key add -
308+ apt-get update
309+ apt-get install -y cabal-install-3.2
310+ apt-get install -y nodejs
311+ apt-get install -y ghcjs-8.4
312+ cabal user-config update -a "jobs: 1"
313+ bash -c "$PACKCHECK $BUILD"
314+ - *save
315+ coveralls-ghc-8_2_2:
316+ <<: *env
317+ executor: amd64-executor
318+ steps:
319+ - *preinstall
320+ - checkout
321+ - *project-preinstall
322+ - *restore
323+ - run:
324+ environment:
325+ BUILD: cabal
326+ COVERALLS_OPTIONS: "--repo-token=\"$REPO_TOKEN\" --coverage-mode=StrictlyFullLines --exclude-dir=test test"
327+ GHC_OPTIONS: "-DCOVERAGE_BUILD"
328+ command: |
329+ apt-get install -y ghc-8.2.2
330+ apt-get install -y cabal-install-2.4
331+ bash -c "$PACKCHECK cabal-v1"
332+ - *save
333+ coveralls-ghc-8_8_3:
334+ <<: *env
335+ executor: amd64-executor
336+ steps:
337+ - *preinstall
338+ - checkout
339+ - *project-preinstall
340+ - *restore
341+ - run:
342+ environment:
343+ BUILD: cabal-v2
344+ COVERALLS_OPTIONS: "--repo-token=\"$REPO_TOKEN\" --coverage-mode=StrictlyFullLines --exclude-dir=test"
345+ CABAL_PROJECT: "cabal.project.coverage"
346+ CABAL_BUILD_TARGETS: ""
347+ command: |
348+ apt-get install -y ghc-8.8.3
349+ apt-get install -y cabal-install-3.0
350+ bash -c "$PACKCHECK cabal-v2"
351+ no_output_timeout: 25m
352+ - *save
353+ coveralls-ghc-8_10_2:
354+ <<: *env
355+ executor: amd64-executor
356+ steps:
357+ - *preinstall
358+ - checkout
359+ - *project-preinstall
360+ - *restore
361+ - run:
362+ name: Build and test with inspection, coverage, Werror
363+ command: |
364+ apt-get install -y ghc-8.10.2
365+ apt-get install -y cabal-install-3.2
366+ # Run tests with coverage
367+ cabal update
368+ cabal user-config update -a "jobs: 1"
369+ bin/test.sh --coverage --raw
370+ # Upload results to coveralls.io
371+ PATH=$HOME/.cabal/bin:$PATH
372+ export PATH
373+ which hpc-coveralls 2>/dev/null || cabal install --project-file cabal.project.hpc-coveralls hpc-coveralls
374+ hpc-coveralls --repo-token="$REPO_TOKEN" --coverage-mode=StrictlyFullLines
375+ no_output_timeout: 25m
376+ - *save
377+ hlint-src:
378+ <<: *env
379+ executor: amd64-executor
380+ steps:
381+ - *preinstall
382+ - checkout
383+ - *project-preinstall
384+ - *restore
385+ - run:
386+ name: Hlint src
387+ environment:
388+ BUILD: hlint
200389 HLINT_OPTIONS: lint
390+ DISABLE_SDIST_BUILD: "yes"
201391 HLINT_TARGETS: core/src src test benchmark
202392 command: |
203− bash -c "$PACKCHECK hlint" || exit 1
393+ bash -c "$PACKCHECK $BUILD" || exit 1
204394 echo "Checking trailing spaces..."
205395 count=$(find . -name "*.hs" -exec grep -H '\ $' {} \; | tee /dev/tty | wc -l)
206396 exit $count
@@ -210,9 +400,23 @@ workflows:
210400 version: 2
211401 build:
212402 jobs:
213− #- cabal-ghc-8_6_5:
214− # name: 8.6.5-debug-unoptimized
215− - cabal-ghc-9_8_1-docspec:
216− name: ghc-9.8.1-docspec
217− - hlint-trailing-spaces:
218− name: hlint and trailing spaces
403+ - cabal-ghc-8_10_4:
404+ name: GHC 8.10.4 + sdist + no-test + no-bench
405+ #- cabal-ghc-8.6.5
406+ #- cabal-ghc-8_4_4:
407+ # name: GHC 8.4.4 + x86 + debug
408+ #- cabal-ghc-8_2_2:
409+ # name: GHC 8.2.2 + no-test + no-bench + no-docs
410+ #- cabal-ghc-8_0_2:
411+ # name: GHC 8.0.2 + no-test + no-bench + no-docs
412+ #- cabal-ghc-7.10.3
413+ # - cabal-ghcjs-8_4:
414+ # name: GHCJS 8.4 + no-test + no-docs
415+ #- stack-ghc-8.4
416+ #- stack-ghc-8.2
417+ #- coveralls-ghc-8.2.2
418+ #- coveralls-ghc-8.8.3
419+ #- coveralls-ghc-8_10_2:
420+ # name: GHC 8.10.2 + inspection + coverage + Werror
421+ - hlint-src:
422+ name: Hlint src + Trailing Spaces
.github/workflows/haskell.yml+91−84
@@ -13,15 +13,9 @@ jobs:
1313 # packcheck environment variables
1414 LC_ALL: C.UTF-8
1515 BUILD: ${{ matrix.build }}
16− # For updating see: https://downloads.haskell.org/~ghcup/
17− GHCUP_VERSION: 0.1.20.0
18− DOCSPEC_URL: https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz
19− DOCSPEC_OPTIONS: "--timeout 60 --check-properties --property-variables xs"
20− GHCUP_GHC_OPTIONS: ${{ matrix.ghcup_ghc_options }}
2116 GHCVER: ${{ matrix.ghc_version }}
2217 CABALVER: ${{ matrix.cabal_version }}
2318 DISABLE_DOCS: ${{ matrix.disable_docs }}
24− ENABLE_DOCSPEC: ${{ matrix.enable_docspec }}
2519 DISABLE_TEST: ${{ matrix.disable_test }}
2620 DISABLE_BENCH: ${{ matrix.disable_bench }}
2721 DISABLE_DIST_CHECKS: ${{ matrix.disable_dist_checks }}
@@ -45,18 +39,14 @@ jobs:
4539 STACK_BUILD_OPTIONS: ${{ matrix.stack_build_options }}
4640
4741 # packcheck location and revision
48− PACKCHECK: "./packcheck.sh"
42+ PACKCHECK_LOCAL_PATH: "./packcheck.sh"
4943 PACKCHECK_GITHUB_URL: "https://raw.githubusercontent.com/composewell/packcheck"
50− PACKCHECK_GITHUB_COMMIT: "v0.7.0"
44+ PACKCHECK_GITHUB_COMMIT: "a302772c8f3791c477edd1764b82f8d1f5b22fdf"
5145
5246 # Pull token from "secrets" setting of the github repo
5347 COVERALLS_TOKEN: ${{ secrets.COVERALLS_TOKEN }}
5448 COVERAGE: ${{ matrix.coverage }}
55−
56− # hlint
57− HLINT_VERSION: 3.6.1
58− HLINT_OPTIONS: "lint"
59− HLINT_TARGETS: "core/src src test benchmark"
49+ DOCTEST: ${{ matrix.doctest }}
6050
6151 # Subdir
6252 SUBDIR: ${{ matrix.subdir }}
@@ -68,8 +58,8 @@ jobs:
6858 matrix:
6959 # The order is important to optimize fail-fast.
7060 name:
71− - 9.8.1-Werror
72− # - 9.8.1-docspec
61+ - 9.4.4-Werror
62+ - 9.2.7-docspec
7363 # - 8.10.7-coverage
7464
7565 # Note: if cabal.project is not specified benchmarks and tests won't
@@ -78,76 +68,53 @@ jobs:
7868 include:
7969 - name: head
8070 ghc_version: head
81− # The URL may change, to find a working URL go to https://gitlab.haskell.org/ghc/ghc/-/jobs/
82− # Find a debian10/11/12 job, click on a passed/failed status, at the
83− # end of the output you will find the tar.xz name, put that tar
84− # name after "raw/", and put the job name after "job=".
85− # Also see https://github.com/mpickering/ghc-artefact-nix/blob/master/gitlab-artifact.nix
86− #
87− # May also use ghcup for installing ghc head version, use the
88− # version "LatestNightly", and the following config:
89− # ghcup config add-release-channel https://ghc.gitlab.haskell.org/ghcup-metadata/ghcup-nightlies-0.0.7.yaml
90− ghcup_ghc_options: "-u https://gitlab.haskell.org/ghc/ghc/-/jobs/artifacts/master/raw/ghc-x86_64-linux-deb10-int_native-validate.tar.xz?job=x86_64-linux-deb10-int_native-validate"
9171 runner: ubuntu-latest
9272 build: cabal
93− cabal_version: 3.10.2.0
73+ cabal_version: 3.8.1.0
9474 cabal_project: cabal.project.ghc-head
9575 disable_sdist_build: "y"
9676 ignore_error: true
97− # Note: use linux for warning build for convenient dev testing
98− - name: 9.8.1-Werror
99− ghc_version: 9.8.1
100− runner: ubuntu-latest
101− build: cabal
102− cabal_version: 3.10.1.0
103− disable_sdist_build: "y"
104− cabal_project: cabal.project.Werror
105− ignore_error: false
106− #- name: 9.8.1-docspec
107− # ghc_version: 9.8.1
108− # runner: ubuntu-latest
109− # build: cabal
110− # cabal_version: 3.10.1.0
111− # cabal_project: cabal.project.doctest
112− # disable_test: "y"
113− # disable_bench: "y"
114− # disable_docs: "y"
115− # enable_docspec: "y"
116− # disable_sdist_build: "y"
117− # ignore_error: false
118− - name: 9.8.1-fusion-inspection
119− ghc_version: 9.8.1
77+ - name: 9.6.1
78+ ghc_version: 9.6.1
12079 runner: ubuntu-latest
12180 build: cabal
12281 cabal_version: 3.10.1.0
12382 disable_sdist_build: "y"
12483 cabal_project: cabal.project
125− cabal_build_options: "--flag fusion-plugin --flag inspection"
126− ignore_error: false
127− - name: 9.6.3-macos
128− ghc_version: 9.6.3
129− runner: macos-latest
84+ ignore_error: true
85+ - name: 9.4.4-Werror
86+ ghc_version: 9.4.4
87+ runner: ubuntu-latest
13088 build: cabal
131− cabal_version: 3.10.1.0
89+ cabal_version: 3.8.1.0
13290 disable_sdist_build: "y"
133− cabal_project: cabal.project
91+ cabal_project: cabal.project.Werror
13492 ignore_error: false
135− - name: 9.4.7
136− ghc_version: 9.4.7
93+ - name: 9.4.4-fusion-inspection
94+ ghc_version: 9.4.4
13795 runner: ubuntu-latest
13896 build: cabal
13997 cabal_version: 3.8.1.0
14098 disable_sdist_build: "y"
14199 cabal_project: cabal.project
100+ cabal_build_options: "--flag fusion-plugin --flag inspection"
142101 ignore_error: false
143− - name: 9.2.8
144− ghc_version: 9.2.8
102+ - name: 9.2.7
103+ ghc_version: 9.2.7
145104 runner: ubuntu-latest
146105 build: cabal
147106 cabal_project: cabal.project
148107 cabal_version: 3.6.2.0
149108 disable_sdist_build: "y"
150109 ignore_error: false
110+ - name: 9.2.7-docspec
111+ ghc_version: 9.2.7
112+ runner: ubuntu-latest
113+ doctest: "y"
114+ cabal_version: 3.6.2.0
115+ cabal_project: cabal.project
116+ disable_sdist_build: "y"
117+ ignore_error: false
151118 - name: 9.0.2-streamly-sdist
152119 ghc_version: 9.0.2
153120 runner: ubuntu-latest
@@ -171,6 +138,14 @@ jobs:
171138 stack_build_options: "--flag streamly-benchmarks:-opt"
172139 cabal_version: 3.6.2.0
173140 ignore_error: false
141+ - name: 8.10.7-macos
142+ ghc_version: 8.10.7
143+ runner: macos-latest
144+ build: cabal
145+ cabal_version: 3.6.2.0
146+ cabal_project: cabal.project
147+ disable_sdist_build: "y"
148+ ignore_error: false
174149 # - name: 8.10.7-coverage
175150 # ghc_version: 8.10.7
176151 # runner: ubuntu-latest
@@ -185,6 +160,15 @@ jobs:
185160 cabal_project: cabal.project
186161 disable_sdist_build: "y"
187162 ignore_error: false
163+ - name: 8.6.5-no-fusion-unoptimized
164+ ghc_version: 8.6.5
165+ runner: ubuntu-latest
166+ build: cabal
167+ cabal_version: 3.6.2.0
168+ cabal_project: cabal.project
169+ cabal_build_options: "--flag no-fusion --flag -opt"
170+ disable_sdist_build: "y"
171+ ignore_error: false
188172 - name: 8.6.5-debug-unoptimized
189173 ghc_version: 8.6.5
190174 runner: ubuntu-latest
@@ -194,44 +178,66 @@ jobs:
194178 cabal_build_options: "--flag debug --flag -opt"
195179 disable_sdist_build: "y"
196180 ignore_error: false
197− - name: hlint
198− build: hlint
199− runner: ubuntu-latest
200− ignore_error: true
201181
202182 steps:
203183 - uses: actions/checkout@v2
204184
205185 - uses: actions/cache@v1
206− name: Cache common directories
186+ name: Cache ~/.cabal
207187 with:
208− path: |
209− ~/.local
210− ~/.cabal
211− ~/.stack
212− ~/.ghcup
188+ path: ~/.cabal
213189 # Bump the key version to clear the cache
214190 key: ${{ runner.os }}-${{ matrix.ghc_version }}-cabal-v2
215191
216192 - name: Download packcheck
217193 run: |
218− if test ! -e "$PACKCHECK"
219− then
220− if test -z "$PACKCHECK_GITHUB_COMMIT"
221− then
222− die "PACKCHECK_GITHUB_COMMIT is not specified."
223− fi
224− PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
225− curl --fail -sL -o "$PACKCHECK" $PACKCHECK_URL || exit 1
226− chmod +x $PACKCHECK
227− elif test ! -x "$PACKCHECK"
194+ # Get packcheck if needed
195+ CURL=$(which curl)
196+ PACKCHECK_URL=${PACKCHECK_GITHUB_URL}/${PACKCHECK_GITHUB_COMMIT}/packcheck.sh
197+ if test ! -e "$PACKCHECK_LOCAL_PATH"; then $CURL -sL -o "$PACKCHECK_LOCAL_PATH" $PACKCHECK_URL; fi;
198+ chmod +x $PACKCHECK_LOCAL_PATH
199+
200+ os=$(uname -s -m)
201+ case "$os" in
202+ "Linux x86_64") GHCUP_ARCH="x86_64-linux" ;;
203+ "Darwin x86_64") GHCUP_ARCH="x86_64-apple-darwin" ;;
204+ *) echo "Unknown OS/Arch: $os"; exit 1;;
205+ esac
206+
207+ # Check available versions here: https://downloads.haskell.org/~ghcup/
208+ GHCUP_VER=0.1.18.0
209+ $CURL -sL -o ./ghcup https://downloads.haskell.org/~ghcup/$GHCUP_VER/${GHCUP_ARCH}-ghcup-$GHCUP_VER
210+ chmod +x ./ghcup
211+ if test "$GHCVER" = "head"
228212 then
229− chmod +x $PACKCHECK
213+ # The URL may change, to find a working URL go to https://gitlab.haskell.org/ghc/ghc/-/jobs/
214+ # Find a debian10 job, click on a passed/failed job, at the
215+ # end of the output you will find the tar.xz name, put that after
216+ # "raw/", and put the job name after "job=".
217+ # Also see https://github.com/mpickering/ghc-artefact-nix/blob/master/gitlab-artifact.nix
218+ ./ghcup install ghc -u https://gitlab.haskell.org/ghc/ghc/-/jobs/artifacts/master/raw/ghc-x86_64-linux-deb10-int_native-validate.tar.xz?job=x86_64-linux-deb10-int_native-validate head
219+ else
220+ ./ghcup install ghc $GHCVER
230221 fi
222+ ./ghcup install cabal $CABALVER
223+ ./ghcup set ghc $GHCVER
224+ ./ghcup set cabal $CABALVER
231225
232226 - name: Run tests
233227 run: |
234− if test -n "$COVERAGE"
228+ if test -n "$DOCTEST"
229+ then
230+ mkdir -p $HOME/.cabal/bin
231+ curl -sL https://github.com/phadej/cabal-extras/releases/download/cabal-docspec-0.0.0.20210111/cabal-docspec-0.0.0.20210111.xz > cabal-docspec.xz
232+ echo '0829bd034fba901cbcfe491d98ed8b28fd54f9cb5c91fa8e1ac62dc4413c9562 cabal-docspec.xz' | sha256sum -c -
233+ xz -d < cabal-docspec.xz > $HOME/.cabal/bin/cabal-docspec
234+ rm -f cabal-docspec.xz
235+ chmod a+x $HOME/.cabal/bin/cabal-docspec
236+ cabal update
237+ cabal build --project-file cabal.project.doctest all
238+ $HOME/.cabal/bin/cabal-docspec --version
239+ $HOME/.cabal/bin/cabal-docspec --timeout 60 --check-properties --property-variables xs
240+ elif test -n "$COVERAGE"
235241 then
236242 # Run tests with coverage
237243 cabal update
@@ -244,15 +250,16 @@ jobs:
244250 # XXX Uncomment this and fix it properly later
245251 # hpc-coveralls --repo-token="$COVERALLS_TOKEN" --coverage-mode=StrictlyFullLines
246252 else
253+ export TOOLS_DIR=/opt
247254 # /usr/local/opt/curl/bin for macOS
248255 export PATH=$HOME/.local/bin:$HOME/.ghcup/bin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/opt/curl/bin
249256 if test -n "$SUBDIR"
250257 then
251− mv "$PACKCHECK" "$SUBDIR/packcheck.sh"
258+ mv "$PACKCHECK_LOCAL_PATH" "$SUBDIR/packcheck.sh"
252259 # This is required as dist-newstyle will be created on the top level
253260 # directory as it is considered the working dir.
254261 rm cabal.project
255262 cd "$SUBDIR"
256263 fi
257− bash -c "$PACKCHECK $BUILD"
264+ bash -c "$PACKCHECK_LOCAL_PATH $BUILD"
258265 fi
.hlint.ignore−5
@@ -2,9 +2,6 @@ core/src/Streamly/Internal/Data/Stream/StreamK/Type.hs
22core/src/Streamly/Internal/Data/Pipe/Type.hs
33core/src/Streamly/Internal/Unicode/Stream.hs
44src/Streamly/Internal/Data/SmallArray/Type.hs
5−src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs
6−src/Streamly/Internal/Data/Stream/Serial.hs
7−src/Streamly/Internal/Data/Stream/Zip.hs
85test/Streamly/Test/Data/Array.hs
96test/Streamly/Test/Data/Parser.hs
107test/Streamly/Test/Data/ParserK.hs
@@ -13,7 +10,6 @@ test/Streamly/Test/Data/Unfold.hs
1310test/Streamly/Test/FileSystem/Event.hs
1411test/Streamly/Test/Prelude/Concurrent.hs
1512test/Streamly/Test/Data/Stream/Concurrent.hs
16−test/lib/Streamly/Test/Prelude/Common.hs
1713test/Streamly/Test/Prelude/Fold.hs
1814test/Streamly/Test/Prelude/Rate.hs
1915test/Streamly/Test/Prelude/Serial.hs
@@ -31,7 +27,6 @@ benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs
3127benchmark/Streamly/Benchmark/Data/Unfold.hs
3228benchmark/Streamly/Benchmark/FileSystem/Handle.hs
3329benchmark/Streamly/Benchmark/Prelude/Async.hs
34−benchmark/Streamly/Benchmark/Prelude/Concurrent.hs
3530benchmark/Streamly/Benchmark/Prelude/Merge.hs
3631benchmark/Streamly/Benchmark/Prelude/Parallel.hs
3732benchmark/Streamly/Benchmark/Prelude/Rate.hs
benchmark/NanoBenchmarks.hs+1−1
@@ -18,7 +18,7 @@ import qualified Streamly.Data.Fold as FL
1818import qualified Streamly.Internal.Data.Stream as S
1919import qualified Streamly.Internal.Data.Stream.IsStream as Internal
2020
21−import Test.Tasty.Bench
21+import Gauge
2222import System.Random
2323
2424maxValue :: Int
benchmark/Streamly/Benchmark/Data/Array/CommonImports.hs+1−1
@@ -9,7 +9,7 @@ import qualified Streamly.Data.Fold as Fold
99import qualified Streamly.Data.Stream as S
1010import qualified Streamly.Internal.Data.Stream as Stream
1111
12−import Test.Tasty.Bench
12+import Gauge
1313import Streamly.Benchmark.Common hiding (benchPureSrc)
1414import qualified Stream.Common as P
1515
benchmark/Streamly/Benchmark/Data/Array/Stream.hs+3−3
@@ -39,7 +39,7 @@ import Prelude hiding ()
3939
4040import qualified Streamly.Data.Stream as Stream
4141import qualified Streamly.Internal.Data.Array as Array
42−import qualified Streamly.Internal.Data.Array.Stream as ArrayStream
42+import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
4343import qualified Streamly.Internal.Data.Fold as Fold
4444import qualified Streamly.Internal.Data.Parser as Parser
4545import qualified Streamly.Internal.Data.Stream as Stream
@@ -47,13 +47,13 @@ import qualified Streamly.Internal.Data.StreamK as StreamK
4747import qualified Streamly.Internal.FileSystem.Handle as Handle
4848import qualified Streamly.Internal.Unicode.Stream as Unicode
4949
50−import Test.Tasty.Bench hiding (env)
50+import Gauge hiding (env)
5151import Streamly.Benchmark.Common
5252import Streamly.Benchmark.Common.Handle
5353import Control.Monad.IO.Class (MonadIO)
5454
5555#ifdef INSPECTION
56−import Streamly.Internal.Data.MutByteArray (Unbox)
56+import Streamly.Internal.Data.Unbox (Unbox)
5757import Streamly.Internal.Data.Stream (Step(..))
5858import Test.Inspection
5959#endif
benchmark/Streamly/Benchmark/Data/Fold.hs+3−3
@@ -41,7 +41,7 @@ import qualified Streamly.Internal.Data.Unfold as Unfold
4141import qualified Streamly.Internal.Data.Pipe as Pipe
4242import qualified Streamly.Internal.Data.Stream as Stream
4343
44−import Test.Tasty.Bench
44+import Gauge
4545import Streamly.Benchmark.Common
4646import Prelude hiding (all, any, take, unzip, sequence_, filter)
4747
@@ -269,13 +269,13 @@ unzip = Stream.fold $ FL.lmap (\a -> (a, a)) (FL.unzip FL.sum FL.length)
269269{-# INLINE unzipWithFstM #-}
270270unzipWithFstM :: Monad m => Stream m Int -> m (Int, Int)
271271unzipWithFstM = do
272− let f a = return (a + 1, a)
272+ let f = \a -> return (a + 1, a)
273273 Stream.fold (FL.unzipWithFstM f FL.sum FL.length)
274274
275275{-# INLINE unzipWithMinM #-}
276276unzipWithMinM :: Monad m => Stream m Int -> m (Int, Int)
277277unzipWithMinM = do
278− let f a = return (a + 1, a)
278+ let f = \a -> return (a + 1, a)
279279 Stream.fold (FL.unzipWithMinM f FL.sum FL.length)
280280
281281-------------------------------------------------------------------------------
benchmark/Streamly/Benchmark/Data/Fold/Window.hs+1−1
@@ -13,7 +13,7 @@ import qualified Streamly.Internal.Data.Ring as Ring
1313import qualified Streamly.Internal.Data.Stream as Stream
1414
1515import Streamly.Benchmark.Common
16−import Test.Tasty.Bench
16+import Gauge
1717
1818{-# INLINE source #-}
1919source :: (Monad m, Num a, Stream.Enumerable a) =>
benchmark/Streamly/Benchmark/Data/MutArray.hs+1−1
@@ -49,7 +49,7 @@ import qualified Streamly.Internal.Data.MutArray as MArray
4949import qualified Streamly.Internal.Data.Fold as Fold
5050import qualified Streamly.Internal.Data.Stream as Stream
5151
52−import Test.Tasty.Bench
52+import Gauge
5353import Streamly.Benchmark.Common hiding (benchPureSrc)
5454
5555type Stream = MutArray
benchmark/Streamly/Benchmark/Data/Parser.hs+6−7
@@ -49,7 +49,7 @@ import qualified Streamly.Data.Stream as Stream
4949import qualified Streamly.Internal.Data.Producer as Producer
5050import qualified Streamly.Internal.Data.Stream as Stream
5151
52−import Test.Tasty.Bench hiding (env)
52+import Gauge hiding (env)
5353import Streamly.Benchmark.Common
5454import Streamly.Benchmark.Common.Handle
5555
@@ -348,7 +348,7 @@ split_ value =
348348-- PR.dropWhile (<= (value * 1 `div` 4)) *> PR.die "alt"
349349{-# INLINE takeWhileFail #-}
350350takeWhileFail :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
351−takeWhileFail predicate (Fold fstep finitial _ ffinal) =
351+takeWhileFail predicate (Fold fstep finitial fextract) =
352352 Parser step initial extract
353353
354354 where
@@ -369,7 +369,7 @@ takeWhileFail predicate (Fold fstep finitial _ ffinal) =
369369 Fold.Done b -> Done 0 b
370370 else return $ Error "fail"
371371
372− extract s = fmap (Done 0) (ffinal s)
372+ extract s = fmap (Done 0) (fextract s)
373373
374374{-# INLINE alt2 #-}
375375alt2 :: Monad m
@@ -714,6 +714,8 @@ instance NFData ParseError where
714714o_1_space_serial :: Int -> [Benchmark]
715715o_1_space_serial value =
716716 [ benchIOSink value "takeBetween" $ takeBetween value
717+ , benchIOSink value "takeEQ" $ takeEQ value
718+ , benchIOSink value "takeGE" $ takeGE value
717719 , benchIOSink value "takeWhile" $ takeWhile value
718720 , benchIOSink value "takeWhileP" $ takeWhileP value
719721 , benchIOSink value "takeP" $ takeP value
@@ -810,11 +812,8 @@ o_1_space_serial_unfold bound arrays =
810812o_n_heap_serial :: Int -> [Benchmark]
811813o_n_heap_serial value =
812814 [
813− benchIOSink value "takeEQ" $ takeEQ value
814− , benchIOSink value "takeGE" $ takeGE value
815−
816815 -- lookahead benchmark holds the entire input till end
817− , benchIOSink value "lookAhead" $ lookAhead value
816+ benchIOSink value "lookAhead" $ lookAhead value
818817
819818 -- o-n-heap because of backtracking
820819 , benchIOSrc sourceEscapedFrames value "takeFramedByEsc_"
benchmark/Streamly/Benchmark/Data/ParserK.hs+6−6
@@ -48,7 +48,7 @@ import qualified Streamly.Data.Parser as PRD
4848import qualified Streamly.Internal.Data.ParserK as PR
4949import qualified Streamly.Internal.Data.StreamK as StreamK
5050
51−import Test.Tasty.Bench
51+import Gauge
5252import Streamly.Benchmark.Common
5353
5454-------------------------------------------------------------------------------
@@ -63,7 +63,7 @@ import Streamly.Benchmark.Common
6363#define PARSE_ELEM (Array Int)
6464#define CONSTRAINT_IO (MonadIO m, Unbox a)
6565#define CONSTRAINT (Monad m, Unbox a)
66−#define MODULE_NAME "Data.ParserK.Chunked"
66+#define MODULE_NAME "ParserK.Chunked"
6767
6868#endif
6969
@@ -75,7 +75,7 @@ import Streamly.Benchmark.Common
7575#define PARSE_ELEM (Array Int)
7676#define CONSTRAINT_IO (MonadIO m)
7777#define CONSTRAINT (Monad m)
78−#define MODULE_NAME "Data.ParserK.Chunked.Generic"
78+#define MODULE_NAME "ParserK.Chunked.Generic"
7979
8080#endif
8181
@@ -87,7 +87,7 @@ import Streamly.Benchmark.Common
8787#define PARSE_ELEM Int
8888#define CONSTRAINT_IO (MonadIO m)
8989#define CONSTRAINT (Monad m)
90−#define MODULE_NAME "Data.ParserK"
90+#define MODULE_NAME "ParserK"
9191
9292#endif
9393
@@ -213,7 +213,7 @@ sequence_ value =
213213
214214{-# INLINE takeWhileFailD #-}
215215takeWhileFailD :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
216−takeWhileFailD predicate (Fold fstep finitial _ ffinal) =
216+takeWhileFailD predicate (Fold fstep finitial fextract) =
217217 Parser step initial extract
218218
219219 where
@@ -234,7 +234,7 @@ takeWhileFailD predicate (Fold fstep finitial _ ffinal) =
234234 Fold.Done b -> Done 0 b
235235 else return $ Error "fail"
236236
237− extract s = fmap (Done 0) (ffinal s)
237+ extract s = fmap (Done 0) (fextract s)
238238
239239{-# INLINE takeWhileFail #-}
240240takeWhileFail :: CONSTRAINT =>
benchmark/Streamly/Benchmark/Data/Ring/Unboxed.hs+1−1
@@ -16,7 +16,7 @@ import qualified Streamly.Internal.Data.Array as Array
1616import qualified Streamly.Internal.Data.Ring as Ring
1717import qualified Data.Foldable as P
1818
19−import Test.Tasty.Bench
19+import Gauge
2020import Streamly.Benchmark.Common
2121
2222import Prelude as P
benchmark/Streamly/Benchmark/Data/Serialize.hs+51−89
@@ -1,5 +1,4 @@
11{-# LANGUAGE TemplateHaskell #-}
2−{- HLINT ignore -}
32
43#undef FUSION_CHECK
54#ifdef FUSION_CHECK
@@ -19,40 +18,43 @@ module Main (main) where
1918-------------------------------------------------------------------------------
2019
2120import Control.DeepSeq (NFData(..), deepseq)
22−import Control.Monad (when, void)
2321import GHC.Generics (Generic)
2422import System.Random (randomRIO)
2523#ifndef USE_UNBOX
2624import Test.QuickCheck (Arbitrary, arbitrary)
2725#endif
2826
29−import Streamly.Internal.Data.MutByteArray
30−import qualified Streamly.Internal.Data.MutByteArray as MBA
3127#ifdef USE_UNBOX
3228import Data.Proxy (Proxy(..))
29+import Streamly.Internal.Data.Unbox
3330#else
3431import Control.DeepSeq (force)
3532import Test.QuickCheck (oneof, generate)
33+import Streamly.Internal.Data.Unbox (newBytes, MutableByteArray)
34+import Streamly.Internal.Data.Serialize hiding (encode)
3635#endif
3736
38−import Test.Tasty.Bench
39−import Streamly.Benchmark.Common
40−
41−#ifndef USE_UNBOX
42−import Streamly.Benchmark.Data.Serialize.RecCompatible
43−import Streamly.Benchmark.Data.Serialize.RecNonCompatible
37+#ifdef USE_TH
38+#ifdef USE_UNBOX
39+import Streamly.Internal.Data.Unbox.TH
40+#else
41+import Streamly.Internal.Data.Serialize.TH
42+#endif
4443#endif
4544
45+import Gauge
46+import Streamly.Benchmark.Common
47+
4648#ifdef USE_UNBOX
4749#define SERIALIZE_CLASS Unbox
48−#define DERIVE_CLASS(typ) $(deriveUnbox [d|instance Unbox typ|])
49−#define SERIALIZE_OP pokeAt
50−#define DESERIALIZE_OP peekAt
50+#define DERIVE_CLASS deriveUnbox
51+#define SERIALIZE_OP pokeByteIndex
52+#define DESERIALIZE_OP peekByteIndex
5153#else
5254#define SERIALIZE_CLASS Serialize
53−#define DERIVE_CLASS(typ) $(deriveSerialize [d|instance Serialize typ|])
54−#define SERIALIZE_OP serializeAt
55−#define DESERIALIZE_OP deserializeAt
55+#define DERIVE_CLASS deriveSerialize
56+#define SERIALIZE_OP serialize
57+#define DESERIALIZE_OP deserialize
5658#endif
5759
5860-------------------------------------------------------------------------------
@@ -65,7 +67,7 @@ data Unit = Unit
6567#ifndef USE_TH
6668instance SERIALIZE_CLASS Unit
6769#else
68−DERIVE_CLASS(Unit)
70+$(DERIVE_CLASS ''Unit)
6971#endif
7072
7173instance NFData Unit where
@@ -80,7 +82,7 @@ data Sum2
8082#ifndef USE_TH
8183instance SERIALIZE_CLASS Sum2
8284#else
83−DERIVE_CLASS(Sum2)
85+$(DERIVE_CLASS ''Sum2)
8486#endif
8587
8688instance NFData Sum2 where
@@ -118,7 +120,7 @@ data Sum25
118120#ifndef USE_TH
119121instance SERIALIZE_CLASS Sum25
120122#else
121−DERIVE_CLASS(Sum25)
123+$(DERIVE_CLASS ''Sum25)
122124#endif
123125
124126instance NFData Sum25 where
@@ -157,7 +159,7 @@ data Product25
157159#ifndef USE_TH
158160instance SERIALIZE_CLASS Product25
159161#else
160−DERIVE_CLASS(Product25)
162+$(DERIVE_CLASS ''Product25)
161163#endif
162164
163165instance NFData Product25 where
@@ -237,7 +239,7 @@ data CustomDT1
237239#ifndef USE_TH
238240instance SERIALIZE_CLASS CustomDT1
239241#else
240−DERIVE_CLASS(CustomDT1)
242+$(DERIVE_CLASS ''CustomDT1)
241243#endif
242244
243245instance NFData CustomDT1 where
@@ -260,20 +262,19 @@ data BinTree a
260262#ifndef USE_TH
261263instance Serialize (BinTree a)
262264#else
263−$(deriveSerialize [d|instance Serialize a => Serialize (BinTree a)|])
265+$(deriveSerialize ''BinTree)
264266#endif
265267
266268instance NFData a => NFData (BinTree a) where
267269 {-# INLINE rnf #-}
268270 rnf (Leaf a) = rnf a `seq` ()
269− rnf (Tree l r) = rnf l `seq` rnf r
271+ rnf (Tree l r) = rnf l `seq` rnf r `seq` ()
270272
271−-- XXX This may not terminate, or could become really large.
272273instance Arbitrary a => Arbitrary (BinTree a) where
273274 arbitrary = oneof [Leaf <$> arbitrary, Tree <$> arbitrary <*> arbitrary]
274275
275276mkBinTree :: (Arbitrary a) => Int -> IO (BinTree a)
276−mkBinTree = go (generate arbitrary)
277+mkBinTree = go (generate $ arbitrary)
277278
278279 where
279280
@@ -291,30 +292,35 @@ getSize :: forall a. SERIALIZE_CLASS a => a -> Int
291292#ifdef USE_UNBOX
292293getSize _ = sizeOf (Proxy :: Proxy a)
293294#else
294−getSize = addSizeTo 0
295+getSize = size 0
295296#endif
296297
297298-------------------------------------------------------------------------------
298299-- Common helpers
299300-------------------------------------------------------------------------------
300301
301−{- HLINT ignore "Eta reduce" -}
302302-- Parts of "f" that are dependent on val will not be optimized out.
303303{-# INLINE loop #-}
304304loop :: Int -> (a -> IO b) -> a -> IO ()
305305loop count f val = go count val
306306 where
307307
308− go n x = when (n > 0) $ f x >> go (n-1) x
308+ go n x = do
309+ if n > 0
310+ then f x >> go (n-1) x
311+ else return ()
309312
310313-- The first arg of "f" is the environment which is not threaded around in the
311314-- loop.
312315{-# INLINE loopWith #-}
313−loopWith :: Int -> (e -> a -> IO b) -> e -> a -> IO ()
316+loopWith :: Int -> (env -> a -> IO b) -> env -> a -> IO ()
314317loopWith count f e val = go count val
315318 where
316319
317− go n x = when (n > 0) $ f e x >> go (n-1) x
320+ go n x = do
321+ if n > 0
322+ then f e x >> go (n-1) x
323+ else return ()
318324
319325benchSink :: NFData b => String -> Int -> (Int -> IO b) -> Benchmark
320326benchSink name times f = bench name (nfIO (randomRIO (times, times) >>= f))
@@ -325,7 +331,7 @@ benchSink name times f = bench name (nfIO (randomRIO (times, times) >>= f))
325331
326332{-
327333{-# INLINE pokeWithSize #-}
328−pokeWithSize :: SERIALIZE_CLASS a => MutByteArray -> a -> IO ()
334+pokeWithSize :: SERIALIZE_CLASS a => MutableByteArray -> a -> IO ()
329335pokeWithSize arr val = do
330336 let n = getSize val
331337 n `seq` SERIALIZE_OP 0 arr val >> return ()
@@ -334,27 +340,27 @@ pokeWithSize arr val = do
334340pokeTimesWithSize :: SERIALIZE_CLASS a => a -> Int -> IO ()
335341pokeTimesWithSize val times = do
336342 let n = getSize val
337− arr <- MBA.new n
343+ arr <- newBytes n
338344 loopWith times pokeWithSize arr val
339345-}
340346
341347{-# INLINE poke #-}
342−poke :: SERIALIZE_CLASS a => MutByteArray -> a -> IO ()
343−poke arr val = void (SERIALIZE_OP 0 arr val)
348+poke :: SERIALIZE_CLASS a => MutableByteArray -> a -> IO ()
349+poke arr val = SERIALIZE_OP 0 arr val >> return ()
344350
345351{-# INLINE pokeTimes #-}
346352pokeTimes :: SERIALIZE_CLASS a => a -> Int -> IO ()
347353pokeTimes val times = do
348354 let n = getSize val
349− arr <- MBA.new n
355+ arr <- newBytes n
350356 loopWith times poke arr val
351357
352358{-# INLINE encode #-}
353359encode :: SERIALIZE_CLASS a => a -> IO ()
354360encode val = do
355361 let n = getSize val
356− arr <- MBA.new n
357− void (SERIALIZE_OP 0 arr val)
362+ arr <- newBytes n
363+ SERIALIZE_OP 0 arr val >> return ()
358364
359365{-# INLINE encodeTimes #-}
360366encodeTimes :: SERIALIZE_CLASS a => a -> Int -> IO ()
@@ -362,7 +368,7 @@ encodeTimes val times = loop times encode val
362368
363369{-# INLINE peek #-}
364370peek :: forall a. (NFData a, SERIALIZE_CLASS a) =>
365− (a, Int) -> MutByteArray -> IO ()
371+ (a, Int) -> MutableByteArray -> IO ()
366372#ifdef USE_UNBOX
367373peek (_val, _) arr = do
368374 (val1 :: a) <- DESERIALIZE_OP 0 arr
@@ -389,7 +395,7 @@ peek (_val, n) arr = do
389395{-# INLINE peekTimes #-}
390396peekTimes :: (NFData a, SERIALIZE_CLASS a) => Int -> a -> Int -> IO ()
391397peekTimes n val times = do
392− arr <- MBA.new n
398+ arr <- newBytes n
393399 _ <- SERIALIZE_OP 0 arr val
394400 loopWith times peek (val, n) arr
395401
@@ -397,7 +403,7 @@ peekTimes n val times = do
397403trip :: forall a. (NFData a, SERIALIZE_CLASS a) => a -> IO ()
398404trip val = do
399405 let n = getSize val
400− arr <- MBA.new n
406+ arr <- newBytes n
401407 _ <- SERIALIZE_OP 0 arr val
402408#ifdef USE_UNBOX
403409 (val1 :: a) <- DESERIALIZE_OP 0 arr
@@ -468,41 +474,6 @@ benchVar gname f tInt lInt times =
468474 , let !n = getSize lInt
469475 in benchSink "list-int" times (f n lInt)
470476 ]
471−
472−{-
473−{-# INLINE benchGeneric #-}
474−benchGeneric
475− :: (NFData b, Serialize b)
476− => String
477− -> String
478− -> (forall a. (NFData a, Serialize a) =>
479− Int -> a -> Int -> IO ())
480− -> b
481− -> Int
482− -> Benchmark
483−benchGeneric gname valId f val times =
484− bgroup gname
485− [ let !n = getSize val
486− in benchSink valId times (f n val)
487− ]
488−-}
489−
490−{-# INLINE benchLargeRec #-}
491−benchLargeRec ::
492− String
493− -> (forall a. (NFData a, Serialize a) =>
494− Int -> a -> Int -> IO ())
495− -> RecNonCompatible
496− -> RecCompatible
497− -> Int
498− -> Benchmark
499−benchLargeRec gname f recL recR times =
500− bgroup gname
501− [ let !n = getSize recL
502− in benchSink "RecNonCompatible" times (f n recL)
503− , let !n = getSize recR
504− in benchSink "RecCompatible" times (f n recR)
505− ]
506477#endif
507478
508479-- Times is scaled by the number of constructors to normalize
@@ -510,9 +481,8 @@ benchLargeRec gname f recL recR times =
510481allBenchmarks :: Int -> [Benchmark]
511482allBenchmarks times =
512483#else
513−allBenchmarks ::
514− BinTree Int -> [Int] -> RecNonCompatible -> RecCompatible -> Int -> [Benchmark]
515−allBenchmarks tInt lInt recL recR times =
484+allBenchmarks :: BinTree Int -> [Int] -> Int -> [Benchmark]
485+allBenchmarks tInt lInt times =
516486#endif
517487 [ bgroup "sizeOf"
518488 [
@@ -532,10 +502,6 @@ allBenchmarks tInt lInt recL recR times =
532502 , benchVar "encode" (const encodeTimes) tInt lInt 1
533503 , benchVar "peek" peekTimes tInt lInt 1
534504 , benchVar "roundtrip" (const roundtrip) tInt lInt 1
535− , benchLargeRec "poke" (const pokeTimes) recL recR times
536− , benchLargeRec "encode" (const encodeTimes) recL recR times
537− , benchLargeRec "peek" peekTimes recL recR times
538− , benchLargeRec "roundtrip" (const roundtrip) recL recR times
539505#endif
540506 ]
541507
@@ -551,22 +517,18 @@ main = do
551517 -- nodes 1 level = log_2 (100001/3) + 1 = 16
552518 !(tInt :: BinTree Int) <- force <$> mkBinTree 16
553519
554− let !recL = valRecNonCompatible
555− let !recR = valRecCompatible
556−
557520 -- Approximately 100000 constructors, assuming two constructors (Cons, Int)
558521 -- per element.
559522 let lInt = [1..50000 :: Int]
560523 let !len = length lInt -- evaluate the list
561524#endif
562525#ifndef FUSION_CHECK
526+ -- This can take too much memory/CPU, need to restrict the test
527+ -- runQC
563528#ifdef USE_UNBOX
564529 runWithCLIOpts defaultStreamSize allBenchmarks
565530#else
566− len
567− `seq` runWithCLIOpts
568− defaultStreamSize
569− (allBenchmarks tInt lInt recL recR)
531+ len `seq` runWithCLIOpts defaultStreamSize (allBenchmarks tInt lInt)
570532#endif
571533#else
572534 -- Enable FUSION_CHECK macro at the beginning of the file
benchmark/Streamly/Benchmark/Data/Serialize/RecCompatible.hsdeleted−24
@@ -1,24 +0,0 @@
1−{-# LANGUAGE TemplateHaskell #-}
2−
3−module Streamly.Benchmark.Data.Serialize.RecCompatible
4− ( RecCompatible(..)
5− , valRecCompatible
6− ) where
7−
8−--------------------------------------------------------------------------------
9−-- Imports
10−--------------------------------------------------------------------------------
11−
12−import Streamly.Internal.Data.MutByteArray (Serialize)
13−import Streamly.Benchmark.Data.Serialize.TH (genLargeRecord)
14−
15−import qualified Streamly.Internal.Data.MutByteArray as Serialize
16−
17−--------------------------------------------------------------------------------
18−-- Code
19−--------------------------------------------------------------------------------
20−
21−$(genLargeRecord "RecCompatible" 50)
22−$(Serialize.deriveSerializeWith
23− (Serialize.encodeRecordFields True)
24− [d|instance Serialize RecCompatible|])
benchmark/Streamly/Benchmark/Data/Serialize/RecNonCompatible.hsdeleted−21
@@ -1,21 +0,0 @@
1−{-# LANGUAGE TemplateHaskell #-}
2−
3−module Streamly.Benchmark.Data.Serialize.RecNonCompatible
4− ( RecNonCompatible(..)
5− , valRecNonCompatible
6− ) where
7−
8−--------------------------------------------------------------------------------
9−-- Imports
10−--------------------------------------------------------------------------------
11−
12−import Streamly.Benchmark.Data.Serialize.TH (genLargeRecord)
13−
14−import qualified Streamly.Internal.Data.MutByteArray as Serialize
15−
16−--------------------------------------------------------------------------------
17−-- Code
18−--------------------------------------------------------------------------------
19−
20−$(genLargeRecord "RecNonCompatible" 50)
21−$(Serialize.deriveSerialize [d|instance Serialize.Serialize RecNonCompatible|])
benchmark/Streamly/Benchmark/Data/Serialize/TH.hsdeleted−76
@@ -1,76 +0,0 @@
1−{-# LANGUAGE TemplateHaskell #-}
2−
3−module Streamly.Benchmark.Data.Serialize.TH (genLargeRecord) where
4−
5−--------------------------------------------------------------------------------
6−-- Imports
7−--------------------------------------------------------------------------------
8−
9−import Language.Haskell.TH
10−
11−import Streamly.Internal.Data.MutByteArray (makeI)
12−import Control.DeepSeq (NFData(..))
13−
14−--------------------------------------------------------------------------------
15−-- Large Record
16−--------------------------------------------------------------------------------
17−
18−genLargeRecord :: String -> Int -> Q [Dec]
19−genLargeRecord tyName numFields =
20− sequence
21− [ dataD
22− (pure [])
23− (mkName tyName)
24− []
25− Nothing
26− [mkCon tyName]
27− [derivClause Nothing [conT ''Eq, conT ''Show]]
28− , mkValueSigDec
29− , mkValueDec
30− , nfDataInstance tyName
31− ]
32−
33− where
34−
35− fieldTypeChoices = [conT ''()]
36− chooseCycle i xs = xs !! (i `mod` length xs)
37− nfDataInstance nm =
38− instanceD
39− (pure [])
40− (appT (conT ''NFData) (conT (mkName nm)))
41− [ funD
42− 'rnf
43− [ clause
44− [ conP
45− (mkName nm)
46− (varP . makeI <$> [0 .. (numFields - 1)])
47− ]
48− (normalB
49− (foldl
50− (\b a -> [|rnf $(b) `seq` rnf $(a)|])
51− [|()|]
52− (varE . makeI <$> [0 .. (numFields - 1)])))
53− []
54− ]
55− ]
56− valueName = mkName $ "val" ++ tyName
57− mkValueSigDec = sigD valueName [t|$(conT (mkName tyName))|]
58− mkValueDec =
59− funD
60− valueName
61− [ clause
62− []
63− (normalB
64− (foldl
65− (\b a -> [|$(b) $(a)|])
66− (conE (mkName tyName))
67− (conE '() <$ [0 .. (numFields - 1)])))
68− []
69− ]
70− mkCon nm = recC (mkName nm) (mkField <$> [0 .. (numFields - 1)])
71− mkField i =
72− varBangType
73− (mkName ("field" ++ show i))
74− (bangType
75− (bang noSourceUnpackedness noSourceStrictness)
76− (chooseCycle i fieldTypeChoices))
benchmark/Streamly/Benchmark/Data/Stream/Common.hs+1−3
@@ -71,9 +71,7 @@ module Stream.Common
7171 )
7272where
7373
74−#if !MIN_VERSION_base(4,18,0)
7574import Control.Applicative (liftA2)
76−#endif
7775import Control.DeepSeq (NFData)
7876import Control.Exception (try)
7977import GHC.Exception (ErrorCall)
@@ -104,7 +102,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
104102#endif
105103#endif
106104
107−import Test.Tasty.Bench
105+import Gauge
108106import Prelude hiding (Foldable(..), mapM, replicate)
109107
110108#ifdef USE_STREAMK
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentCommon.hs+3−3
@@ -14,14 +14,14 @@ where
1414import Stream.Common
1515 (composeN, benchIO, benchIOSink, benchIOSrc, sourceUnfoldrM)
1616import Streamly.Data.Stream (Stream)
17−import Streamly.Internal.Data.Stream.Prelude (MonadAsync, Config)
17+import Streamly.Internal.Data.Stream.Concurrent (MonadAsync, Config)
1818
1919import qualified Data.List as List
2020import qualified Streamly.Data.Fold as Fold
2121import qualified Streamly.Data.Stream as Stream
22−import qualified Streamly.Internal.Data.Stream.Prelude as Async
22+import qualified Streamly.Internal.Data.Stream.Concurrent as Async
2323
24−import Test.Tasty.Bench
24+import Gauge
2525import Prelude hiding (mapM)
2626import Streamly.Benchmark.Common
2727
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentEager.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Prelude as Async
12+import qualified Streamly.Internal.Data.Stream.Concurrent as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentEager"
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentInterleaved.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Prelude as Async
12+import qualified Streamly.Internal.Data.Stream.Concurrent as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentInterleaved"
benchmark/Streamly/Benchmark/Data/Stream/ConcurrentOrdered.hs+1−1
@@ -9,7 +9,7 @@
99import Stream.ConcurrentCommon (allBenchmarks)
1010import Streamly.Benchmark.Common (runWithCLIOpts, defaultStreamSize)
1111
12−import qualified Streamly.Internal.Data.Stream.Prelude as Async
12+import qualified Streamly.Internal.Data.Stream.Concurrent as Async
1313
1414moduleName :: String
1515moduleName = "Data.Stream.ConcurrentOrdered"
benchmark/Streamly/Benchmark/Data/Stream/Eliminate.hs+1−3
@@ -57,7 +57,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
5757#endif
5858#endif
5959
60−import Test.Tasty.Bench
60+import Gauge
6161import Streamly.Benchmark.Common
6262import Prelude hiding (length, sum, or, and, any, all, notElem, elem, (!!),
6363 lookup, repeat, minimum, maximum, product, last, mapM_, init)
@@ -132,14 +132,12 @@ foldableAny :: Int -> Int -> Bool
132132foldableAny value n =
133133 Prelude.any (> (value + 1)) (sourceUnfoldr value n :: Stream Identity Int)
134134
135−{- HLINT ignore "Use all"-}
136135{-# INLINE foldableAnd #-}
137136foldableAnd :: Int -> Int -> Bool
138137foldableAnd value n =
139138 Prelude.and $ fmap
140139 (<= (value + 1)) (sourceUnfoldr value n :: Stream Identity Int)
141140
142−{- HLINT ignore "Use any"-}
143141{-# INLINE foldableOr #-}
144142foldableOr :: Int -> Int -> Bool
145143foldableOr value n =
benchmark/Streamly/Benchmark/Data/Stream/Exceptions.hs+3−12
@@ -36,22 +36,21 @@ import System.IO (Handle, hClose, hPutChar)
3636import qualified Streamly.FileSystem.Handle as FH
3737import qualified Streamly.Internal.FileSystem.Handle as IFH
3838import qualified Streamly.Internal.Data.Unfold as IUF
39−import qualified Streamly.Internal.Data.Unfold.Prelude as IUF
39+import qualified Streamly.Internal.Data.Unfold.Exception as IUF
4040
4141#ifdef USE_PRELUDE
4242import qualified Streamly.Internal.Data.Stream.IsStream as Stream
4343#else
4444import qualified Streamly.Internal.Data.Stream as Stream
45−import qualified Streamly.Internal.Data.Stream.Prelude as Stream
45+import qualified Streamly.Internal.Data.Stream.Exception.Lifted as Stream
4646#endif
4747
48−import Test.Tasty.Bench hiding (env)
48+import Gauge hiding (env)
4949import Prelude hiding (last, length)
5050import Streamly.Benchmark.Common
5151import Streamly.Benchmark.Common.Handle
5252
5353#ifdef INSPECTION
54−import Control.Monad.Catch (MonadCatch)
5554import Test.Inspection
5655
5756import qualified Streamly.Internal.Data.Stream as D
@@ -287,12 +286,8 @@ readChunksOnException inh devNull =
287286 in IUF.fold (IFH.writeChunks devNull) readEx inh
288287
289288#ifdef INSPECTION
290−#if __GLASGOW_HASKELL__ >= 906
291−inspect $ hasNoTypeClassesExcept 'readChunksOnException [''MonadCatch]
292−#else
293289inspect $ hasNoTypeClasses 'readChunksOnException
294290#endif
295−#endif
296291
297292-- | Send the file contents to /dev/null with exception handling
298293readChunksBracket_ :: Handle -> Handle -> IO ()
@@ -301,12 +296,8 @@ readChunksBracket_ inh devNull =
301296 in IUF.fold (IFH.writeChunks devNull) readEx inh
302297
303298#ifdef INSPECTION
304−#if __GLASGOW_HASKELL__ >= 906
305−inspect $ hasNoTypeClassesExcept 'readChunksBracket_ [''MonadCatch]
306−#else
307299inspect $ hasNoTypeClasses 'readChunksBracket_
308300#endif
309−#endif
310301
311302readChunksBracket :: Handle -> Handle -> IO ()
312303readChunksBracket inh devNull =
benchmark/Streamly/Benchmark/Data/Stream/Expand.hs+1−9
@@ -54,7 +54,7 @@ import qualified Streamly.Internal.Data.Stream as StreamK
5454#endif
5555#endif
5656
57−import Test.Tasty.Bench
57+import Gauge
5858import Stream.Common
5959import Streamly.Benchmark.Common
6060import Prelude hiding (concatMap)
@@ -198,11 +198,7 @@ concatMapPure outer inner n =
198198 (fromStream $ sourceUnfoldr outer n)
199199
200200#ifdef INSPECTION
201−#if __GLASGOW_HASKELL__ >= 906
202−inspect $ hasNoTypeClassesExcept 'concatMapPure [''Applicative]
203−#else
204201inspect $ hasNoTypeClasses 'concatMapPure
205−#endif
206202inspect $ 'concatMapPure `hasNoType` ''SPEC
207203#endif
208204
@@ -215,11 +211,7 @@ concatMapRepl outer inner n =
215211 (fromStream . S.replicate inner) (fromStream $ sourceUnfoldrM outer n)
216212
217213#ifdef INSPECTION
218−#if __GLASGOW_HASKELL__ >= 906
219−inspect $ hasNoTypeClassesExcept 'concatMapRepl [''Applicative]
220−#else
221214inspect $ hasNoTypeClasses 'concatMapRepl
222−#endif
223215inspect $ 'concatMapRepl `hasNoType` ''SPEC
224216#endif
225217
benchmark/Streamly/Benchmark/Data/Stream/Generate.hs+1−1
@@ -40,7 +40,7 @@ import qualified Stream.Common as Common
4040#endif
4141#endif
4242
43−import Test.Tasty.Bench
43+import Gauge
4444import Streamly.Benchmark.Common
4545import qualified Prelude
4646
benchmark/Streamly/Benchmark/Data/Stream/Lift.hs+1−1
@@ -37,7 +37,7 @@ import qualified Control.Monad.State.Strict as State
3737#endif
3838#endif
3939
40−import Test.Tasty.Bench
40+import Gauge
4141import Streamly.Benchmark.Common
4242import Prelude hiding (reverse, tail)
4343
benchmark/Streamly/Benchmark/Data/Stream/Reduce.hs+2−2
@@ -48,7 +48,7 @@ import Streamly.Internal.Data.Stream (Stream)
4848import qualified Streamly.Internal.Data.Stream as S
4949#ifndef USE_STREAMLY_CORE
5050import qualified Streamly.Data.Stream.Prelude as S
51−import qualified Streamly.Internal.Data.Stream.Prelude as S
51+import qualified Streamly.Internal.Data.Stream.Time as S
5252#endif
5353
5454#ifdef USE_STREAMK
@@ -61,7 +61,7 @@ import qualified Streamly.Internal.Data.Stream as K
6161
6262#endif
6363
64−import Test.Tasty.Bench
64+import Gauge
6565import Streamly.Benchmark.Common
6666import Stream.Common
6767import Prelude hiding (reverse, tail)
benchmark/Streamly/Benchmark/Data/Stream/Split.hs+1−1
@@ -35,7 +35,7 @@ import qualified Streamly.Internal.FileSystem.Handle as IFH
3535import qualified Streamly.Internal.Unicode.Stream as IUS
3636import qualified Streamly.Prelude as S
3737
38−import Test.Tasty.Bench hiding (env)
38+import Gauge hiding (env)
3939import Prelude hiding (last, length)
4040import Streamly.Benchmark.Common
4141import Streamly.Benchmark.Common.Handle
benchmark/Streamly/Benchmark/Data/Stream/StreamD.hs+1−3
@@ -22,12 +22,10 @@
2222
2323module Main (main) where
2424
25−#if !MIN_VERSION_base(4,18,0)
2625import Control.Applicative (liftA2)
27−#endif
2826import Control.Monad (when)
2927import Data.Maybe (isJust)
30−import Test.Tasty.Bench (bench, nfIO, bgroup, Benchmark, defaultMain)
28+import Gauge (bench, nfIO, bgroup, Benchmark, defaultMain)
3129import System.Random (randomRIO)
3230import Prelude hiding (tail, mapM_, foldl, last, map, mapM, concatMap, zip)
3331
benchmark/Streamly/Benchmark/Data/Stream/StreamK.hs+1−3
@@ -21,9 +21,7 @@
2121
2222module Main (main) where
2323
24−#if !MIN_VERSION_base(4,18,0)
2524import Control.Applicative (liftA2)
26−#endif
2725import Control.Monad (when)
2826import Data.Maybe (isJust)
2927import System.Random (randomRIO)
@@ -37,7 +35,7 @@ import qualified Data.List as List
3735
3836import qualified Streamly.Internal.Data.StreamK as S
3937
40−import Test.Tasty.Bench (bench, nfIO, bgroup, Benchmark, defaultMain)
38+import Gauge (bench, nfIO, bgroup, Benchmark, defaultMain)
4139
4240import Streamly.Benchmark.Common
4341
benchmark/Streamly/Benchmark/Data/Stream/StreamKAlt.hs+2−2
@@ -23,7 +23,7 @@ import qualified Streamly.Internal.Data.StreamK.Alt as S
2323-- import qualified Streamly.Internal.Data.SVar as S
2424
2525import Streamly.Benchmark.Common
26−import Test.Tasty.Bench (bgroup, Benchmark, defaultMain)
26+import Gauge (bgroup, Benchmark, defaultMain)
2727
2828value :: Int
2929value = 100000
@@ -452,4 +452,4 @@ o_1_space =
452452 ]
453453
454454main :: IO ()
455−main = defaultMain [o_1_space]
455+main = defaultMain $ concat [o_1_space]
benchmark/Streamly/Benchmark/Data/Stream/ToStreamK.hs+2−2
@@ -37,7 +37,7 @@ import qualified Streamly.Internal.Data.StreamK as S
3737import qualified Streamly.Internal.Data.Stream as D
3838import qualified Streamly.Internal.Data.Fold as Fold
3939
40−import Test.Tasty.Bench (bgroup, Benchmark, defaultMain)
40+import Gauge (bgroup, Benchmark, defaultMain)
4141
4242import Streamly.Benchmark.Common
4343
@@ -120,7 +120,7 @@ fromFoldable streamLen n = S.fromFoldable [n..n+streamLen]
120120{-# INLINE fromFoldableM #-}
121121fromFoldableM :: Monad m => Int -> Int -> Stream m Int
122122fromFoldableM streamLen n =
123− Prelude.foldr (S.consM . return) S.nil [n .. n + streamLen]
123+ Prelude.foldr S.consM S.nil (Prelude.fmap return [n..n+streamLen])
124124
125125{-
126126{-# INLINABLE concatMapFoldableWith #-}
benchmark/Streamly/Benchmark/Data/Stream/Transform.hs+2−2
@@ -46,7 +46,7 @@ import Streamly.Internal.Data.Time.Units
4646import Streamly.Internal.Data.Stream (Stream)
4747import qualified Streamly.Internal.Data.Stream as Stream
4848#ifndef USE_STREAMLY_CORE
49−import qualified Streamly.Internal.Data.Stream.Prelude as Stream
49+import qualified Streamly.Internal.Data.Stream.Time as Stream
5050#endif
5151#ifdef USE_STREAMK
5252import Control.DeepSeq (NFData(..))
@@ -58,7 +58,7 @@ import qualified Streamly.Internal.Data.StreamK as StreamK
5858#endif
5959#endif
6060
61−import Test.Tasty.Bench
61+import Gauge
6262import Stream.Common hiding (scanl')
6363import Streamly.Benchmark.Common
6464import Prelude hiding (sequence, mapM)
benchmark/Streamly/Benchmark/Data/Unfold.hs+2−11
@@ -31,18 +31,17 @@ import qualified Prelude
3131import qualified Streamly.FileSystem.Handle as FH
3232import qualified Streamly.Internal.Data.Fold as FL
3333import qualified Streamly.Internal.Data.Unfold as UF
34−import qualified Streamly.Internal.Data.Unfold.Prelude as UF
34+import qualified Streamly.Internal.Data.Unfold.Exception as UF
3535import qualified Streamly.Internal.Data.Stream as S
3636import qualified Streamly.Internal.Data.Stream as D
3737import qualified Streamly.Internal.Data.StreamK as K
3838
39−import Test.Tasty.Bench hiding (env)
39+import Gauge hiding (env)
4040import Prelude hiding (take, filter, zipWith, map, mapM, takeWhile)
4141import Streamly.Benchmark.Common
4242import Streamly.Benchmark.Common.Handle
4343
4444#ifdef INSPECTION
45−import Control.Monad.Catch (MonadCatch)
4645import Test.Inspection
4746#endif
4847
@@ -777,11 +776,7 @@ readWriteFinally_Unfold inh devNull =
777776 in S.fold (FH.write devNull) $ S.unfold readEx inh
778777
779778#ifdef INSPECTION
780−#if __GLASGOW_HASKELL__ >= 906
781−inspect $ hasNoTypeClassesExcept 'readWriteFinally_Unfold [''MonadCatch]
782−#else
783779inspect $ hasNoTypeClasses 'readWriteFinally_Unfold
784−#endif
785780-- inspect $ 'readWriteFinallyUnfold `hasNoType` ''Step
786781#endif
787782
@@ -797,11 +792,7 @@ readWriteBracket_Unfold inh devNull =
797792 in S.fold (FH.write devNull) $ S.unfold readEx inh
798793
799794#ifdef INSPECTION
800−#if __GLASGOW_HASKELL__ >= 906
801−inspect $ hasNoTypeClassesExcept 'readWriteBracket_Unfold [''MonadCatch]
802−#else
803795inspect $ hasNoTypeClasses 'readWriteBracket_Unfold
804−#endif
805796-- inspect $ 'readWriteBracketUnfold `hasNoType` ''Step
806797#endif
807798
benchmark/Streamly/Benchmark/FileSystem/Handle.hs+1−1
@@ -22,7 +22,7 @@ import Streamly.Benchmark.Common.Handle (mkHandleBenchEnv)
2222import qualified Handle.ReadWrite as RW
2323import qualified Handle.Read as RO
2424
25−import Test.Tasty.Bench hiding (env)
25+import Gauge hiding (env)
2626import Prelude hiding (last, length)
2727import Streamly.Benchmark.Common
2828
benchmark/Streamly/Benchmark/FileSystem/Handle/Read.hs+9−8
@@ -33,6 +33,7 @@ import qualified Streamly.Data.Stream as Stream
3333import qualified Streamly.Data.Fold as Fold
3434import qualified Streamly.FileSystem.Handle as FH
3535import qualified Streamly.Internal.Data.Array as A
36+import qualified Streamly.Internal.Data.Array as AT
3637import qualified Streamly.Internal.Data.Fold as FL
3738import qualified Streamly.Internal.Data.Stream as IP
3839import qualified Streamly.Internal.FileSystem.Handle as IFH
@@ -40,14 +41,14 @@ import qualified Streamly.Internal.Unicode.Stream as IUS
4041import qualified Streamly.Data.Stream.Prelude as S
4142import qualified Streamly.Unicode.Stream as SS
4243
43−import Test.Tasty.Bench hiding (env)
44+import Gauge hiding (env)
4445import Prelude hiding (last, length)
4546import Streamly.Benchmark.Common.Handle
4647
4748#ifdef INSPECTION
4849import Streamly.Internal.Data.Stream (Step(..), FoldMany)
4950
50−import qualified Streamly.Internal.Data.MutArray as MutArray
51+import qualified Streamly.Internal.Data.MutArray as MA
5152import qualified Streamly.Internal.Data.Stream as D
5253import qualified Streamly.Internal.Data.Unfold as IUF
5354
@@ -68,7 +69,7 @@ readLast = S.fold Fold.last . S.unfold FH.reader
6869inspect $ hasNoTypeClasses 'readLast
6970inspect $ 'readLast `hasNoType` ''Step -- S.unfold
7071inspect $ 'readLast `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
71−inspect $ 'readLast `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
72+inspect $ 'readLast `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
7273#endif
7374
7475-- assert that flattenArrays constructors are not present
@@ -80,7 +81,7 @@ readCountBytes = S.fold Fold.length . S.unfold FH.reader
8081inspect $ hasNoTypeClasses 'readCountBytes
8182inspect $ 'readCountBytes `hasNoType` ''Step -- S.unfold
8283inspect $ 'readCountBytes `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
83−inspect $ 'readCountBytes `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
84+inspect $ 'readCountBytes `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
8485#endif
8586
8687-- | Count the number of lines in a file.
@@ -95,7 +96,7 @@ readCountLines =
9596inspect $ hasNoTypeClasses 'readCountLines
9697inspect $ 'readCountLines `hasNoType` ''Step
9798inspect $ 'readCountLines `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
98−inspect $ 'readCountLines `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
99+inspect $ 'readCountLines `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
99100#endif
100101
101102-- | Count the number of words in a file.
@@ -119,7 +120,7 @@ readSumBytes = S.fold Fold.sum . S.unfold FH.reader
119120inspect $ hasNoTypeClasses 'readSumBytes
120121inspect $ 'readSumBytes `hasNoType` ''Step
121122inspect $ 'readSumBytes `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
122−inspect $ 'readSumBytes `hasNoType` ''MutArray.ArrayUnsafe -- FH.read/A.read
123+inspect $ 'readSumBytes `hasNoType` ''MA.ArrayUnsafe -- FH.read/A.read
123124#endif
124125
125126-- XXX When we mark this with INLINE and we have two benchmarks using S.drain
@@ -235,13 +236,13 @@ groupsOf :: Int -> Handle -> IO Int
235236groupsOf n inh =
236237 -- writeNUnsafe gives 2.5x boost here over writeN.
237238 S.fold Fold.length
238− $ IP.groupsOf n (A.writeNUnsafe n) (S.unfold FH.reader inh)
239+ $ IP.groupsOf n (AT.writeNUnsafe n) (S.unfold FH.reader inh)
239240
240241#ifdef INSPECTION
241242inspect $ hasNoTypeClasses 'groupsOf
242243inspect $ 'groupsOf `hasNoType` ''Step
243244inspect $ 'groupsOf `hasNoType` ''FoldMany
244−inspect $ 'groupsOf `hasNoType` ''MutArray.ArrayUnsafe -- AT.writeNUnsafe
245+inspect $ 'groupsOf `hasNoType` ''AT.ArrayUnsafe -- AT.writeNUnsafe
245246 -- FH.read/A.read
246247inspect $ 'groupsOf `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
247248#endif
benchmark/Streamly/Benchmark/FileSystem/Handle/ReadWrite.hs+10−9
@@ -34,7 +34,7 @@ import qualified Streamly.Internal.FileSystem.Handle as IFH
3434import qualified Streamly.Data.Array as A
3535import qualified Streamly.Data.Stream.Prelude as S
3636
37−import Test.Tasty.Bench hiding (env)
37+import Gauge hiding (env)
3838import Streamly.Benchmark.Common.Handle
3939
4040#ifdef INSPECTION
@@ -42,7 +42,8 @@ import Streamly.Internal.Data.Stream (Step(..))
4242
4343import qualified Streamly.Internal.Data.Stream as D
4444import qualified Streamly.Internal.Data.Tuple.Strict as Strict
45−import qualified Streamly.Internal.Data.MutArray as MutArray
45+import qualified Streamly.Internal.Data.Stream.MutChunked as MAS
46+import qualified Streamly.Internal.Data.Array as AT
4647
4748import Test.Inspection
4849#endif
@@ -82,7 +83,7 @@ copyStream inh outh = S.fold (FH.write outh) (S.unfold FH.reader inh)
8283inspect $ hasNoTypeClasses 'copyStream
8384inspect $ 'copyStream `hasNoType` ''Step -- S.unfold
8485inspect $ 'copyStream `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
85−inspect $ 'copyStream `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
86+inspect $ 'copyStream `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
8687 -- FH.read/A.read
8788inspect $ 'copyStream `hasNoType` ''Strict.Tuple3' -- FH.write/chunksOf
8889#endif
@@ -108,8 +109,8 @@ readFromBytesNull inh devNull = IFH.putBytes devNull $ S.unfold FH.reader inh
108109#ifdef INSPECTION
109110inspect $ hasNoTypeClasses 'readFromBytesNull
110111inspect $ 'readFromBytesNull `hasNoType` ''Step
111−inspect $ 'readFromBytesNull `hasNoType` ''MutArray.SpliceState
112−inspect $ 'readFromBytesNull `hasNoType` ''MutArray.ArrayUnsafe -- FH.fromBytes/S.chunksOf
112+inspect $ 'readFromBytesNull `hasNoType` ''MAS.SpliceState
113+inspect $ 'readFromBytesNull `hasNoType` ''AT.ArrayUnsafe -- FH.fromBytes/S.chunksOf
113114inspect $ 'readFromBytesNull `hasNoType` ''D.FoldMany
114115#endif
115116
@@ -122,8 +123,8 @@ readWithFromBytesNull inh devNull =
122123#ifdef INSPECTION
123124inspect $ hasNoTypeClasses 'readWithFromBytesNull
124125inspect $ 'readWithFromBytesNull `hasNoType` ''Step
125−inspect $ 'readWithFromBytesNull `hasNoType` ''MutArray.SpliceState
126−inspect $ 'readWithFromBytesNull `hasNoType` ''MutArray.ArrayUnsafe -- FH.fromBytes/S.chunksOf
126+inspect $ 'readWithFromBytesNull `hasNoType` ''MAS.SpliceState
127+inspect $ 'readWithFromBytesNull `hasNoType` ''AT.ArrayUnsafe -- FH.fromBytes/S.chunksOf
127128inspect $ 'readWithFromBytesNull `hasNoType` ''D.FoldMany
128129#endif
129130
@@ -170,7 +171,7 @@ writeReadWith inh devNull = IUF.fold fld unf (defaultChunkSize, inh)
170171inspect $ hasNoTypeClasses 'writeReadWith
171172inspect $ 'writeReadWith `hasNoType` ''Step
172173inspect $ 'writeReadWith `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
173−inspect $ 'writeReadWith `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
174+inspect $ 'writeReadWith `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
174175 -- FH.read/A.read
175176#endif
176177
@@ -187,7 +188,7 @@ writeRead inh devNull = IUF.fold fld unf inh
187188inspect $ hasNoTypeClasses 'writeRead
188189inspect $ 'writeRead `hasNoType` ''Step
189190inspect $ 'writeRead `hasNoType` ''IUF.ConcatState -- FH.read/UF.many
190−inspect $ 'writeRead `hasNoType` ''MutArray.ArrayUnsafe -- FH.write/writeNUnsafe
191+inspect $ 'writeRead `hasNoType` ''AT.ArrayUnsafe -- FH.write/writeNUnsafe
191192 -- FH.read/A.read
192193#endif
193194
benchmark/Streamly/Benchmark/Prelude/Adaptive.hs+1−1
@@ -10,7 +10,7 @@
1010import Control.Concurrent (threadDelay)
1111import Control.Monad (when)
1212import Control.Monad.IO.Class (liftIO)
13−import Test.Tasty.Bench
13+import Gauge
1414import Streamly.Prelude as S
1515import System.Random (randomRIO)
1616
benchmark/Streamly/Benchmark/Prelude/Ahead.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Test.Tasty.Bench
18+import Gauge
1919
2020moduleName :: String
2121moduleName = "Prelude.Ahead"
benchmark/Streamly/Benchmark/Prelude/Async.hs+1−1
@@ -16,7 +16,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream.Transform as Transform
1616import Streamly.Benchmark.Common
1717import Streamly.Benchmark.Prelude
1818
19−import Test.Tasty.Bench
19+import Gauge
2020
2121moduleName :: String
2222moduleName = "Prelude.Async"
benchmark/Streamly/Benchmark/Prelude/Concurrent.hs+1−1
@@ -15,7 +15,7 @@ import Streamly.Prelude
1515 , fromWAsync, parallel, fromParallel
1616 )
1717
18−import Test.Tasty.Bench
18+import Gauge
1919import qualified Streamly.Prelude as S
2020
2121-------------------------------------------------------------------------------
benchmark/Streamly/Benchmark/Prelude/Merge.hs+1−1
@@ -30,7 +30,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream as Stream
3030import Streamly.Benchmark.Common
3131import Streamly.Benchmark.Prelude
3232
33−import Test.Tasty.Bench
33+import Gauge
3434
3535#ifdef INSPECTION
3636import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Prelude/Parallel.hs+1−1
@@ -22,7 +22,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream as Internal
2222import Streamly.Benchmark.Common
2323import Streamly.Benchmark.Prelude
2424
25−import Test.Tasty.Bench
25+import Gauge
2626
2727moduleName :: String
2828moduleName = "Prelude.Parallel"
benchmark/Streamly/Benchmark/Prelude/Rate.hs+1−1
@@ -14,7 +14,7 @@ import qualified Streamly.Prelude as S
1414import Streamly.Benchmark.Common
1515import Streamly.Benchmark.Prelude
1616
17−import Test.Tasty.Bench
17+import Gauge
1818
1919moduleName :: String
2020moduleName = "Prelude.Rate"
benchmark/Streamly/Benchmark/Prelude/WAsync.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Test.Tasty.Bench
18+import Gauge
1919
2020moduleName :: String
2121moduleName = "Prelude.WAsync"
benchmark/Streamly/Benchmark/Prelude/WSerial.hs+1−1
@@ -29,7 +29,7 @@ import qualified Streamly.Internal.Data.Unfold as UF
2929import Streamly.Benchmark.Common
3030import Streamly.Benchmark.Prelude
3131
32−import Test.Tasty.Bench
32+import Gauge
3333
3434#ifdef INSPECTION
3535import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Prelude/ZipAsync.hs+1−1
@@ -15,7 +15,7 @@ import qualified Streamly.Prelude as S
1515import Streamly.Benchmark.Common
1616import Streamly.Benchmark.Prelude
1717
18−import Test.Tasty.Bench
18+import Gauge
1919
2020moduleName :: String
2121moduleName = "Prelude.ZipAsync"
benchmark/Streamly/Benchmark/Prelude/ZipSerial.hs+1−1
@@ -29,7 +29,7 @@ import qualified Streamly.Prelude as S
2929import Streamly.Benchmark.Common
3030import Streamly.Benchmark.Prelude hiding (sourceUnfoldrM)
3131
32−import Test.Tasty.Bench
32+import Gauge
3333
3434#ifdef INSPECTION
3535import GHC.Types (SPEC(..))
benchmark/Streamly/Benchmark/Unicode/Char.hs+1−1
@@ -16,7 +16,7 @@ import Control.DeepSeq (NFData(..))
1616import Streamly.Internal.Data.Array (Array)
1717import System.FilePath (dropExtensions, takeFileName)
1818import System.FilePath.Posix ((</>))
19−import Test.Tasty.Bench (Benchmark, bench, bgroup, defaultMain, env, nfIO)
19+import Gauge.Main (Benchmark, bench, bgroup, defaultMain, env, nfIO)
2020import Streamly.Internal.Unicode.Char
2121 ( NormalizationMode(NFC, NFD, NFKC, NFKD)
2222 , normalize
benchmark/Streamly/Benchmark/Unicode/Parser.hs+1−1
@@ -31,7 +31,7 @@ import qualified Streamly.Data.Array as Array
3131import qualified Streamly.Data.Stream as Stream
3232import qualified Streamly.Internal.Unicode.Parser as PRU
3333
34−import Test.Tasty.Bench hiding (env)
34+import Gauge hiding (env)
3535import Streamly.Benchmark.Common
3636
3737runParser :: Int -> (Stream IO Char -> IO a) -> IO ()
benchmark/Streamly/Benchmark/Unicode/Stream.hs+6−24
@@ -11,11 +11,6 @@
1111{-# LANGUAGE CPP #-}
1212{-# LANGUAGE ScopedTypeVariables #-}
1313
14−#undef FUSION_CHECK
15−#ifdef FUSION_CHECK
16−{-# OPTIONS_GHC -ddump-simpl -ddump-to-file -dsuppress-all #-}
17−#endif
18−
1914#ifdef __HADDOCK_VERSION__
2015#undef INSPECTION
2116#endif
@@ -38,17 +33,16 @@ import qualified Streamly.Internal.FileSystem.Handle as Handle
3833import qualified Streamly.Internal.Unicode.Array as UnicodeArr
3934import qualified Streamly.Internal.Unicode.Stream as Unicode
4035
41−import Test.Tasty.Bench hiding (env)
36+import Gauge hiding (env)
4237import Streamly.Benchmark.Common
4338import Streamly.Benchmark.Common.Handle
4439
4540#ifdef INSPECTION
46−import Streamly.Internal.Data.MutByteArray (Unbox)
41+import Streamly.Internal.Data.Unbox (Unbox)
4742import Streamly.Internal.Data.Stream (Step(..))
48−import qualified Streamly.Internal.Data.Array as Array
49−import qualified Streamly.Internal.Data.MutArray as MutArray
5043import qualified Streamly.Internal.Data.Fold as Fold
5144import qualified Streamly.Internal.Data.Tuple.Strict as Strict
45+import qualified Streamly.Internal.Data.Array as Array
5246
5347import Test.Inspection
5448#endif
@@ -224,7 +218,7 @@ inspect $ 'copyStreamLatin1' `hasNoType` ''Step
224218inspect $ 'copyStreamLatin1' `hasNoType` ''Unfold.ConcatState -- Handle.read/UF.many
225219
226220inspect $ 'copyStreamLatin1' `hasNoType` ''Fold.Step
227−inspect $ 'copyStreamLatin1' `hasNoType` ''MutArray.ArrayUnsafe -- Handle.write/writeNUnsafe
221+inspect $ 'copyStreamLatin1' `hasNoType` ''Array.ArrayUnsafe -- Handle.write/writeNUnsafe
228222 -- Handle.read/Array.read
229223inspect $ 'copyStreamLatin1' `hasNoType` ''Strict.Tuple3' -- Handle.write/chunksOf
230224#endif
@@ -245,7 +239,7 @@ inspect $ 'copyStreamLatin1 `hasNoType` ''Unfold.ConcatState -- Handle.read/UF.m
245239
246240inspect $ 'copyStreamLatin1 `hasNoType` ''Fold.ManyState
247241inspect $ 'copyStreamLatin1 `hasNoType` ''Fold.Step
248−inspect $ 'copyStreamLatin1 `hasNoType` ''MutArray.ArrayUnsafe -- Handle.write/writeNUnsafe
242+inspect $ 'copyStreamLatin1 `hasNoType` ''Array.ArrayUnsafe -- Handle.write/writeNUnsafe
249243 -- Handle.read/Array.read
250244inspect $ 'copyStreamLatin1 `hasNoType` ''Strict.Tuple3' -- Handle.write/chunksOf
251245#endif
@@ -286,8 +280,7 @@ _copyStreamUtf8'Fold :: Handle -> Handle -> IO ()
286280_copyStreamUtf8'Fold inh outh =
287281 Stream.fold (Handle.write outh)
288282 $ Unicode.encodeUtf8
289− $ Stream.catRights
290− $ Stream.parseMany Unicode.writeCharUtf8'
283+ $ Stream.foldMany Unicode.writeCharUtf8'
291284 $ Stream.unfold Handle.reader inh
292285
293286{-# NOINLINE _copyStreamUtf8Parser #-}
@@ -324,7 +317,6 @@ o_1_space_decode_encode_read env =
324317
325318main :: IO ()
326319main = do
327−#ifndef FUSION_CHECK
328320 env <- mkHandleBenchEnv
329321 defaultMain (allBenchmarks env)
330322
@@ -337,13 +329,3 @@ main = do
337329 , o_1_space_decode_encode_read env
338330 ]
339331 ]
340−#else
341− -- Enable FUSION_CHECK macro at the beginning of the file
342− -- Enable one benchmark below, and run the benchmark
343− -- Check the .dump-simpl output
344− env <- mkHandleBenchEnv
345− let mkHandles (RefHandles {bigInH = inh, outputH = outh}) = Handles inh outh
346− (Handles inh outh) <- getHandles env mkHandles
347− copyStreamLatin1' inh outh
348− return ()
349−#endif
benchmark/Streamly/Benchmark/Unicode/Utf8.hs+1−1
@@ -13,7 +13,7 @@
1313import Data.Char (chr)
1414import Streamly.Internal.Unicode.Utf8 (Utf8)
1515
16−import Test.Tasty.Bench hiding (env)
16+import Gauge hiding (env)
1717import Streamly.Benchmark.Common
1818
1919import qualified Streamly.Internal.Unicode.Utf8 as Utf8
benchmark/bench-runner/Main.hs+4−32
@@ -16,11 +16,9 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
1616 benchName = drop 4 benchName0
1717
1818 general
19− -- GHC-9.6 requires 32M heap vs 16M for GHC-9.4
20− | "o-1-sp" `isInfixOf` benchName = "-K36K -M32M"
19+ | "o-1-sp" `isInfixOf` benchName = "-K36K -M16M"
2120 | "o-n-h" `isInfixOf` benchName = "-K36K -M32M"
22− -- GHC-9.6 requires 32M heap vs 16M for GHC-9.4
23− | "o-n-st" `isInfixOf` benchName = "-K1M -M32M"
21+ | "o-n-st" `isInfixOf` benchName = "-K1M -M16M"
2422 | "o-n-sp" `isInfixOf` benchName = "-K1M -M32M"
2523 | otherwise = ""
2624
@@ -29,17 +27,6 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
2927 | otherwise = ""
3028
3129 benchSpecific
32− -- GHC-9.6 requires 64M, earlier it was 16M
33− | "Data.Fold/o-n-heap.key-value.toHashMapIO (max buckets) sum" == benchName =
34− "-M64M"
35−
36− -- This is required only for the --long case because we allocate all
37− -- the arrays upfront. depends on the size of the stream.
38− | "Data.Parser/o-1-space" `isPrefixOf` benchName =
39− "-M128M"
40−
41− -----------------------------------------------------------------------
42−
4330 | "Prelude.Parallel/o-n-heap.mapping.mapM" == benchName = "-M256M"
4431 | "Prelude.Parallel/o-n-heap.monad-outer-product."
4532 `isPrefixOf` benchName = "-M256M"
@@ -70,12 +57,6 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
7057 -- GHC 9.4.4 requires 4M
7158 | "Data.Stream.StreamDK/o-n-space.traversable."
7259 `isPrefixOf` benchName = "-K4M"
73− -- GHC-9.6 requires 64M, earlier it was 32M
74− | "Data.Stream.StreamDK/o-n-heap.buffered.readsPrec pure streams" == benchName =
75− "-M64M"
76− -- GHC-9.6 requires 64M, earlier it was 32M
77− | "Data.Stream.StreamDK/o-n-heap.buffered.showPrec Haskell lists" == benchName =
78− "-M64M"
7960
8061 -----------------------------------------------------------------------
8162
@@ -93,13 +74,6 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
9374 | "Data.Stream/o-n-space.iterated."
9475 `isPrefixOf` benchName = "-K4M"
9576
96− -- GHC-9.6 requires 64M, earlier it was 32M
97− | "Data.Stream/o-n-heap.buffered.showPrec Haskell lists" == benchName =
98− "-M64M"
99− -- GHC-9.6 requires 64M, earlier it was 32M
100− | "Data.Stream/o-n-heap.buffered.readsPrec pure streams" == benchName =
101− "-M64M"
102−
10377 | "Data.Stream.ConcurrentEager/o-n-heap.monad-outer-product.toNullAp"
10478 `isPrefixOf` benchName = "-M1024M"
10579 | "Data.Stream.ConcurrentEager/o-1-space."
@@ -112,12 +86,10 @@ rtsOpts exeName benchName0 = unwords [general, exeSpecific, benchSpecific]
11286
11387 | "Data.Array" `isPrefixOf` benchName
11488 && "/o-1-space.generation.read" `isSuffixOf` benchName = "-M32M"
115− -- XXX GHC 9.6 onwards needs 64M, earlier it was 32M
11689 | "Data.Array" `isPrefixOf` benchName
117− && "/o-1-space.generation.show" `isSuffixOf` benchName = "-M64M"
118− -- XXX GHC 9.6 onwards needs 64M, earlier it was 32M
90+ && "/o-1-space.generation.show" `isSuffixOf` benchName = "-M32M"
11991 | "Data.Array.Generic/o-1-space.transformationX4.map"
120− `isPrefixOf` benchName = "-M64M"
92+ `isPrefixOf` benchName = "-M32M"
12193
12294 -- XXX For --long option, need to check why so much heap is required.
12395 -- Note, if we remove the chunked stream module we need to keep the
benchmark/lib/Streamly/Benchmark/Common.hs+1−1
@@ -56,7 +56,7 @@ import System.Random (randomRIO)
5656import qualified Streamly.Internal.Data.Fold as Fold
5757import qualified Streamly.Internal.Data.Stream as S
5858
59−import Test.Tasty.Bench
59+import Gauge
6060
6161-------------------------------------------------------------------------------
6262-- Benchmark Prefixes
benchmark/lib/Streamly/Benchmark/Common/Handle.hs+1−3
@@ -30,8 +30,6 @@ module Streamly.Benchmark.Common.Handle
3030 , isSpace
3131 , isSp
3232 , mkHandleBenchEnv
33− , Handles(..)
34− , getHandles
3533 )
3634where
3735
@@ -48,7 +46,7 @@ import System.Process (callCommand)
4846
4947import Data.IORef
5048import Prelude hiding (last, length)
51−import Test.Tasty.Bench hiding (env)
49+import Gauge hiding (env)
5250
5351
5452scratchDir :: String
benchmark/lib/Streamly/Benchmark/Prelude.hs+1−1
@@ -82,7 +82,7 @@ import qualified Streamly.Internal.Data.Stream.IsStream.Type as IsStream
8282import qualified Streamly.Internal.Data.Pipe as Pipe
8383import qualified Streamly.Internal.Data.Stream.Serial as Serial
8484
85−import Test.Tasty.Bench
85+import Gauge
8686import Streamly.Internal.Data.Time.Units
8787
8888-- Common polymorphic stream APIs used across multiple stream modules
benchmark/streamly-benchmarks.cabal+27−23
@@ -54,6 +54,11 @@ flag opt
5454 manual: True
5555 default: True
5656
57+flag use-gauge
58+ description: Use gauge instead of tasty-bench for benchmarking
59+ manual: True
60+ default: False
61+
5762flag use-streamly-core
5863 description: Benchmark only core package
5964 manual: True
@@ -140,10 +145,6 @@ common compile-options
140145 -Wno-missing-kind-signatures
141146 -Wno-operator-whitespace
142147
143− if impl(ghc >= 9.8)
144− ghc-options:
145− -Wno-missing-role-annotations
146−
147148 if flag(has-llvm)
148149 ghc-options: -fllvm
149150
@@ -175,34 +176,43 @@ common bench-depends
175176 -- Core libraries shipped with ghc, the min and max
176177 -- constraints of these libraries should match with
177178 -- the GHC versions we support
178− base >= 4.9 && < 4.20
179− , deepseq >= 1.4.1 && < 1.6
179+ base >= 4.9 && < 4.19
180+ , deepseq >= 1.4.1 && < 1.5
180181 , mtl >= 2.2 && < 2.3.2
181182
182183 -- other libraries
183− , exceptions >= 0.8 && < 0.11
184184 , random >= 1.0 && < 2.0
185185 , transformers >= 0.4 && < 0.7
186− , containers >= 0.5 && < 0.8
186+ , containers >= 0.5 && < 0.7
187187 , hashable >= 1.3 && < 1.5
188188 , unordered-containers >= 0.2 && < 0.3
189189 , process >= 1.4 && < 1.7
190190 , directory >= 1.2.2 && < 1.4
191− , filepath >= 1.4.1 && < 1.6
192− , ghc-prim >= 0.4 && < 0.12
193− , tasty-bench >= 0.3 && < 0.4
194− , tasty >= 1.4.1 && < 1.6
195− , streamly-core
191+ , filepath >= 1.4.1 && < 1.5
192+ , ghc-prim >= 0.4 && < 0.11
196193
197− if !flag(use-streamly-core)
194+ if flag(use-streamly-core)
195+ build-depends: streamly-core
196+ else
198197 build-depends: streamly
198+ , streamly-core
199+
200+ if flag(use-gauge)
201+ build-depends: gauge >= 0.2.4 && < 0.3
202+ else
203+ build-depends: tasty-bench >= 0.3 && < 0.4
204+ , tasty >= 1.4.1 && < 1.5
205+ mixins: tasty-bench
206+ (Test.Tasty.Bench as Gauge
207+ , Test.Tasty.Bench as Gauge.Main
208+ )
199209
200210 if flag(fusion-plugin) && !impl(ghcjs) && !impl(ghc < 8.6)
201211 build-depends:
202212 fusion-plugin >= 0.2 && < 0.3
203213 if flag(inspection)
204− build-depends: template-haskell >= 2.14 && < 2.22
205− , inspection-testing >= 0.4 && < 0.6
214+ build-depends: template-haskell >= 2.14 && < 2.21
215+ , inspection-testing >= 0.4 && < 0.5
206216 -- Array uses a Storable constraint in dev build making several inspection
207217 -- tests fail
208218 if flag(dev) && flag(inspection)
@@ -369,13 +379,7 @@ benchmark Data.Serialize
369379 hs-source-dirs: .
370380 cpp-options: -DUSE_TH
371381 main-is: Streamly/Benchmark/Data/Serialize.hs
372− build-depends: QuickCheck, template-haskell
373− other-modules:
374− Streamly.Benchmark.Data.Serialize.TH
375− Streamly.Benchmark.Data.Serialize.RecCompatible
376− Streamly.Benchmark.Data.Serialize.RecNonCompatible
377− if flag(limit-build-mem)
378− ghc-options: +RTS -M1000M -RTS
382+ build-depends: QuickCheck
379383
380384benchmark Data.Unfold
381385 import: bench-options
configure+9−9
@@ -1,6 +1,6 @@
11#! /bin/sh
22# Guess values for system-dependent variables and create Makefiles.
3−# Generated by GNU Autoconf 2.71 for streamly 0.10.1.
3+# Generated by GNU Autoconf 2.71 for streamly 0.10.0.
44#
55# Report bugs to <streamly@composewell.com>.
66#
@@ -610,8 +610,8 @@ MAKEFLAGS=
610610# Identity of this package.
611611PACKAGE_NAME='streamly'
612612PACKAGE_TARNAME='streamly'
613−PACKAGE_VERSION='0.10.1'
614−PACKAGE_STRING='streamly 0.10.1'
613+PACKAGE_VERSION='0.10.0'
614+PACKAGE_STRING='streamly 0.10.0'
615615PACKAGE_BUGREPORT='streamly@composewell.com'
616616PACKAGE_URL='https://streamly.composewell.com'
617617
@@ -1224,7 +1224,7 @@ if test "$ac_init_help" = "long"; then
12241224 # Omit some internal or obsolete options to make the list less imposing.
12251225 # This message is too long to be a string in the A/UX 3.1 sh.
12261226 cat <<_ACEOF
1227−\`configure' configures streamly 0.10.1 to adapt to many kinds of systems.
1227+\`configure' configures streamly 0.10.0 to adapt to many kinds of systems.
12281228
12291229Usage: $0 [OPTION]... [VAR=VALUE]...
12301230
@@ -1286,7 +1286,7 @@ fi
12861286
12871287if test -n "$ac_init_help"; then
12881288 case $ac_init_help in
1289− short | recursive ) echo "Configuration of streamly 0.10.1:";;
1289+ short | recursive ) echo "Configuration of streamly 0.10.0:";;
12901290 esac
12911291 cat <<\_ACEOF
12921292
@@ -1372,7 +1372,7 @@ fi
13721372test -n "$ac_init_help" && exit $ac_status
13731373if $ac_init_version; then
13741374 cat <<\_ACEOF
1375−streamly configure 0.10.1
1375+streamly configure 0.10.0
13761376generated by GNU Autoconf 2.71
13771377
13781378Copyright (C) 2021 Free Software Foundation, Inc.
@@ -1500,7 +1500,7 @@ cat >config.log <<_ACEOF
15001500This file contains any messages produced by compilers while
15011501running configure, to aid debugging if configure makes a mistake.
15021502
1503−It was created by streamly $as_me 0.10.1, which was
1503+It was created by streamly $as_me 0.10.0, which was
15041504generated by GNU Autoconf 2.71. Invocation command line was
15051505
15061506 $ $0$ac_configure_args_raw
@@ -3800,7 +3800,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
38003800# report actual input values of CONFIG_FILES etc. instead of their
38013801# values after options handling.
38023802ac_log="
3803−This file was extended by streamly $as_me 0.10.1, which was
3803+This file was extended by streamly $as_me 0.10.0, which was
38043804generated by GNU Autoconf 2.71. Invocation command line was
38053805
38063806 CONFIG_FILES = $CONFIG_FILES
@@ -3856,7 +3856,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\
38563856cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
38573857ac_cs_config='$ac_cs_config_escaped'
38583858ac_cs_version="\\
3859−streamly config.status 0.10.1
3859+streamly config.status 0.10.0
38603860configured by $0, generated by GNU Autoconf 2.71,
38613861 with options \\"\$ac_cs_config\\"
38623862
configure.ac+1−1
@@ -3,7 +3,7 @@
33# See https://www.gnu.org/software/autoconf/manual/autoconf.html for help on
44# the macros used in this file.
55
6−AC_INIT([streamly], [0.10.1], [streamly@composewell.com], [streamly], [https://streamly.composewell.com])
6+AC_INIT([streamly], [0.10.0], [streamly@composewell.com], [streamly], [https://streamly.composewell.com])
77
88# To suppress "WARNING: unrecognized options: --with-compiler"
99AC_ARG_WITH([compiler], [GHC])
core/configure+9−9
@@ -1,6 +1,6 @@
11#! /bin/sh
22# Guess values for system-dependent variables and create Makefiles.
3−# Generated by GNU Autoconf 2.71 for streamly-core 0.2.2.
3+# Generated by GNU Autoconf 2.71 for streamly-core 0.2.0.
44#
55# Report bugs to <streamly@composewell.com>.
66#
@@ -610,8 +610,8 @@ MAKEFLAGS=
610610# Identity of this package.
611611PACKAGE_NAME='streamly-core'
612612PACKAGE_TARNAME='streamly-core'
613−PACKAGE_VERSION='0.2.2'
614−PACKAGE_STRING='streamly-core 0.2.2'
613+PACKAGE_VERSION='0.2.0'
614+PACKAGE_STRING='streamly-core 0.2.0'
615615PACKAGE_BUGREPORT='streamly@composewell.com'
616616PACKAGE_URL='https://streamly.composewell.com'
617617
@@ -1256,7 +1256,7 @@ if test "$ac_init_help" = "long"; then
12561256 # Omit some internal or obsolete options to make the list less imposing.
12571257 # This message is too long to be a string in the A/UX 3.1 sh.
12581258 cat <<_ACEOF
1259−\`configure' configures streamly-core 0.2.2 to adapt to many kinds of systems.
1259+\`configure' configures streamly-core 0.2.0 to adapt to many kinds of systems.
12601260
12611261Usage: $0 [OPTION]... [VAR=VALUE]...
12621262
@@ -1318,7 +1318,7 @@ fi
13181318
13191319if test -n "$ac_init_help"; then
13201320 case $ac_init_help in
1321− short | recursive ) echo "Configuration of streamly-core 0.2.2:";;
1321+ short | recursive ) echo "Configuration of streamly-core 0.2.0:";;
13221322 esac
13231323 cat <<\_ACEOF
13241324
@@ -1404,7 +1404,7 @@ fi
14041404test -n "$ac_init_help" && exit $ac_status
14051405if $ac_init_version; then
14061406 cat <<\_ACEOF
1407−streamly-core configure 0.2.2
1407+streamly-core configure 0.2.0
14081408generated by GNU Autoconf 2.71
14091409
14101410Copyright (C) 2021 Free Software Foundation, Inc.
@@ -1622,7 +1622,7 @@ cat >config.log <<_ACEOF
16221622This file contains any messages produced by compilers while
16231623running configure, to aid debugging if configure makes a mistake.
16241624
1625−It was created by streamly-core $as_me 0.2.2, which was
1625+It was created by streamly-core $as_me 0.2.0, which was
16261626generated by GNU Autoconf 2.71. Invocation command line was
16271627
16281628 $ $0$ac_configure_args_raw
@@ -3832,7 +3832,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
38323832# report actual input values of CONFIG_FILES etc. instead of their
38333833# values after options handling.
38343834ac_log="
3835−This file was extended by streamly-core $as_me 0.2.2, which was
3835+This file was extended by streamly-core $as_me 0.2.0, which was
38363836generated by GNU Autoconf 2.71. Invocation command line was
38373837
38383838 CONFIG_FILES = $CONFIG_FILES
@@ -3888,7 +3888,7 @@ ac_cs_config_escaped=`printf "%s\n" "$ac_cs_config" | sed "s/^ //; s/'/'\\\\\\\\
38883888cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
38893889ac_cs_config='$ac_cs_config_escaped'
38903890ac_cs_version="\\
3891−streamly-core config.status 0.2.2
3891+streamly-core config.status 0.2.0
38923892configured by $0, generated by GNU Autoconf 2.71,
38933893 with options \\"\$ac_cs_config\\"
38943894
core/configure.ac+1−1
@@ -3,7 +3,7 @@
33# See https://www.gnu.org/software/autoconf/manual/autoconf.html for help on
44# the macros used in this file.
55
6−AC_INIT([streamly-core], [0.2.2], [streamly@composewell.com], [streamly-core], [https://streamly.composewell.com])
6+AC_INIT([streamly-core], [0.2.0], [streamly@composewell.com], [streamly-core], [https://streamly.composewell.com])
77
88# To suppress "WARNING: unrecognized options: --with-compiler"
99AC_ARG_WITH([compiler], [GHC])
core/docs/ApiChangelogs/0.1.0-0.2.0.txtdeleted−1471
@@ -1,1471 +0,0 @@
1−---------------------------------
2−API Annotations
3−---------------------------------
4−
5−[A] : Added
6−[R] : Removed
7−[C] : Changed
8−[O] : Old definition
9−[N] : New definition
10−[D] : Deprecated
11−
12−---------------------------------
13−API diff
14−---------------------------------
15−
16−[C] Streamly.Unicode.Parser
17− [A] double :: Monad m => Parser Char m Double
18−[C] Streamly.FileSystem.Handle
19− [A] readWith :: MonadIO m => Int -> Handle -> Stream m Word8
20− [A] readChunksWith :: MonadIO m => Int -> Handle -> Stream m (Array Word8)
21−[C] Streamly.FileSystem.Dir
22− [C] readEither
23− [O] readEither :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
24− [N] readEither :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
25− [C] read
26− [O] read :: MonadIO m => FilePath -> Stream m FilePath
27− [N] read :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
28−[C] Streamly.Data.Unfold
29− [A] class Enum a => Enumerable a
30− [A] second :: b -> Unfold m (a, b) c -> Unfold m a c
31− [A] first :: a -> Unfold m (a, b) c -> Unfold m b c
32− [A] enumerateFromTo :: (Enumerable a, Monad m) => Unfold m (a, a) a
33− [A] enumerateFromThenTo :: (Enumerable a, Monad m) => Unfold m (a, a, a) a
34− [A] enumerateFromThen :: (Enumerable a, Monad m) => Unfold m (a, a) a
35− [A] enumerateFrom :: (Enumerable a, Monad m) => Unfold m a a
36−[C] Streamly.Data.StreamK
37− [C] parseChunks
38− [O] parseChunks :: (Monad m, Unbox a) => ParserK a m b -> StreamK m (Array a) -> m (Either ParseError b)
39− [N] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
40− [C] parseBreakChunks
41− [O] parseBreakChunks :: (Monad m, Unbox a) => ParserK a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
42− [N] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
43− [A] parseBreak :: forall m a b. Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
44− [A] parse :: Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b)
45− [A] handle :: (MonadCatch m, Exception e) => (e -> m (StreamK m a)) -> StreamK m a -> StreamK m a
46− [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> StreamK m a) -> StreamK m a
47−[C] Streamly.Data.Stream
48− [A] wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
49− [A] splitOn :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
50− [C] handle
51− [O] handle :: (MonadCatch m, Exception e) => (e -> Stream m a) -> Stream m a -> Stream m a
52− [N] handle :: (MonadCatch m, Exception e) => (e -> m (Stream m a)) -> Stream m a -> Stream m a
53− [A] groupsOf :: Monad m => Int -> Fold m a b -> Stream m a -> Stream m b
54−[C] Streamly.Data.ParserK
55− [D] fromParser :: (MonadIO m, Unbox a) => Parser a m b -> ParserK (Array a) m b
56− [D] fromFold :: (MonadIO m, Unbox a) => Fold m a b -> ParserK (Array a) m b
57− [A] adaptCG :: Monad m => Parser a m b -> ParserK (Array a) m b
58− [A] adaptC :: (Monad m, Unbox a) => Parser a m b -> ParserK (Array a) m b
59− [A] adapt :: Monad m => Parser a m b -> ParserK a m b
60−[C] Streamly.Data.Parser
61− [A] groupByRollingEither :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
62− [A] groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
63−[A] Streamly.Data.MutByteArray
64− [A] class Unbox a
65− [A] class Serialize a
66− [A] SerializeConfig
67− [A] MutByteArray
68− [A] unpin :: MutByteArray -> IO MutByteArray
69− [A] sizeOf :: (Unbox a, SizeOfRep (Rep a)) => Proxy a -> Int
70− [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
71− [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
72− [A] pinnedNew :: Int -> IO MutByteArray
73− [A] pin :: MutByteArray -> IO MutByteArray
74− [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
75− [A] new :: Int -> IO MutByteArray
76− [A] isPinned :: MutByteArray -> Bool
77− [A] inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
78− [A] inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
79− [A] inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
80− [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
81− [A] deriveUnbox :: Q [Dec] -> Q [Dec]
82− [A] deriveSerializeWith :: (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
83− [A] deriveSerialize :: Q [Dec] -> Q [Dec]
84− [A] addSizeTo :: Serialize a => Int -> a -> Int
85−[C] Streamly.Data.MutArray.Generic
86− [A] write :: MonadIO m => Fold m a (MutArray a)
87− [A] readRev :: MonadIO m => MutArray a -> Stream m a
88− [A] read :: MonadIO m => MutArray a -> Stream m a
89− [A] putIndexUnsafe :: forall m a. MonadIO m => Int -> MutArray a -> a -> m ()
90− [A] modifyIndexUnsafe :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
91− [A] length :: MutArray a -> Int
92− [A] getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
93− [C] getIndex
94− [O] getIndex :: MonadIO m => Int -> MutArray a -> m a
95− [N] getIndex :: MonadIO m => Int -> MutArray a -> m (Maybe a)
96− [A] fromListN :: MonadIO m => Int -> [a] -> m (MutArray a)
97− [A] fromList :: MonadIO m => [a] -> m (MutArray a)
98−[C] Streamly.Data.MutArray
99− [A] unpin :: MutArray a -> IO (MutArray a)
100− [A] readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
101− [A] read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
102− [A] putIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
103− [D] pokeByteIndex :: Unbox a => Int -> MutByteArray -> a -> IO ()
104− [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
105− [A] pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
106− [A] pin :: MutArray a -> IO (MutArray a)
107− [D] peekByteIndex :: Unbox a => Int -> MutByteArray -> IO a
108− [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
109− [D] newPinned :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
110− [A] modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
111− [A] modifyIndex :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
112− [A] modify :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (a -> a) -> m ()
113− [A] isPinned :: MutArray a -> Bool
114− [A] getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
115− [C] getIndex
116− [O] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
117− [N] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
118−[C] Streamly.Data.Array.Generic
119− [A] toList :: Array a -> [a]
120− [A] getIndex :: Int -> Array a -> Maybe a
121−[C] Streamly.Data.Array
122− [A] unpin :: Array a -> IO (Array a)
123− [D] pokeByteIndex :: Unbox a => Int -> MutByteArray -> a -> IO ()
124− [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
125− [A] pin :: Array a -> IO (Array a)
126− [D] peekByteIndex :: Unbox a => Int -> MutByteArray -> IO a
127− [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
128− [A] isPinned :: Array a -> Bool
129−[C] Streamly.Console.Stdio
130−
131−---------------------------------
132−Internal API diff
133−---------------------------------
134−
135−[C] Streamly.Internal.Unicode.Stream
136− [C] writeCharUtf8'
137− [O] writeCharUtf8' :: Monad m => Fold m Word8 Char
138− [N] writeCharUtf8' :: Monad m => Parser Word8 m Char
139−[C] Streamly.Internal.Unicode.Parser
140− [A] number :: Monad m => Parser Char m (Integer, Int)
141− [A] mkDouble :: Integer -> Int -> Double
142− [A] doubleParser :: Monad m => Parser Char m (Int, Int)
143− [C] double
144− [O] double :: Parser Char m Double
145− [N] double :: Monad m => Parser Char m Double
146−[R] Streamly.Internal.Serialize.ToBytes
147−[R] Streamly.Internal.Serialize.FromBytes
148−[C] Streamly.Internal.FileSystem.File
149− [A] writeAppendWith :: (MonadIO m, MonadCatch m) => Int -> FilePath -> Stream m Word8 -> m ()
150− [A] writeAppendChunks :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Array a) -> m ()
151− [A] writeAppendArray :: FilePath -> Array a -> IO ()
152− [A] writeAppend :: (MonadIO m, MonadCatch m) => FilePath -> Stream m Word8 -> m ()
153− [R] appendWith :: (MonadIO m, MonadCatch m) => Int -> FilePath -> Stream m Word8 -> m ()
154− [R] appendChunks :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Array a) -> m ()
155− [R] appendArray :: FilePath -> Array a -> IO ()
156− [R] append :: (MonadIO m, MonadCatch m) => FilePath -> Stream m Word8 -> m ()
157−[C] Streamly.Internal.FileSystem.Dir
158− [C] reader
159− [O] reader :: MonadIO m => Unfold m FilePath FilePath
160− [N] reader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
161− [C] readFiles
162− [O] readFiles :: MonadIO m => FilePath -> Stream m FilePath
163− [N] readFiles :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
164− [C] readEitherPaths
165− [O] readEitherPaths :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
166− [N] readEitherPaths :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
167− [C] readEither
168− [O] readEither :: MonadIO m => FilePath -> Stream m (Either FilePath FilePath)
169− [N] readEither :: (MonadIO m, MonadCatch m) => FilePath -> Stream m (Either FilePath FilePath)
170− [C] readDirs
171− [O] readDirs :: MonadIO m => FilePath -> Stream m FilePath
172− [N] readDirs :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
173− [C] read
174− [O] read :: MonadIO m => FilePath -> Stream m FilePath
175− [N] read :: (MonadIO m, MonadCatch m) => FilePath -> Stream m FilePath
176− [C] fileReader
177− [O] fileReader :: MonadIO m => Unfold m FilePath FilePath
178− [N] fileReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
179− [C] eitherReaderPaths
180− [O] eitherReaderPaths :: MonadIO m => Unfold m FilePath (Either FilePath FilePath)
181− [N] eitherReaderPaths :: (MonadIO m, MonadCatch m) => Unfold m FilePath (Either FilePath FilePath)
182− [C] eitherReader
183− [O] eitherReader :: MonadIO m => Unfold m FilePath (Either FilePath FilePath)
184− [N] eitherReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath (Either FilePath FilePath)
185− [C] dirReader
186− [O] dirReader :: MonadIO m => Unfold m FilePath FilePath
187− [N] dirReader :: (MonadIO m, MonadCatch m) => Unfold m FilePath FilePath
188−[R] Streamly.Internal.Data.Unfold.Type
189−[R] Streamly.Internal.Data.Unfold.Enumeration
190−[C] Streamly.Internal.Data.Unfold
191− [C] Unfold
192− [A] Unfold :: (s -> m (Step s b)) -> (a -> m s) -> Unfold m a b
193− [A] takeWhileMWithInput :: Monad m => (a -> b -> m Bool) -> Unfold m a b -> Unfold m a b
194− [A] manyInterleave :: Monad m => Unfold m a b -> Unfold m c a -> Unfold m c b
195− [A] enumerateFromStepIntegral :: (Monad m, Integral a) => Unfold m (a, a) a
196− [A] crossApplySnd :: Unfold m a b -> Unfold m a c -> Unfold m a c
197− [A] crossApplyFst :: Unfold m a b -> Unfold m a c -> Unfold m a b
198− [A] concatMap :: Monad m => (b -> Unfold m a c) -> Unfold m a b -> Unfold m a c
199−[R] Streamly.Internal.Data.Unboxed
200−[C] Streamly.Internal.Data.Time.Units
201− [R] Streamly.Internal.Data.Unboxed.Unbox
202− [A] Streamly.Internal.Data.Unbox.Unbox
203− [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.NanoSecond64
204− [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.MilliSecond64
205− [A] instance Streamly.Internal.Data.Unbox.Unbox Streamly.Internal.Data.Time.Units.MicroSecond64
206−[R] Streamly.Internal.Data.Time.Clock.Type
207−[A] Streamly.Internal.Data.StreamK
208− [A] CrossStreamK
209− [A] (FixityR,6)
210− [A] (FixityR,6)
211− [A] (FixityR,6)
212− [A] (FixityR,5)
213− [A] (FixityR,5)
214− [A] (FixityR,6)
215− [A] (FixityR,5)
216− [A] StreamK
217− [A] MkStream :: (forall r. State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
218− [A] zipWithM :: Monad m => (a -> b -> m c) -> StreamK m a -> StreamK m b -> StreamK m c
219− [A] zipWith :: Monad m => (a -> b -> c) -> StreamK m a -> StreamK m b -> StreamK m c
220− [A] unfoldrMWith :: Monad m => (m a -> StreamK m a -> StreamK m a) -> (b -> m (Maybe (a, b))) -> b -> StreamK m a
221− [A] unfoldrM :: Monad m => (b -> m (Maybe (a, b))) -> b -> StreamK m a
222− [A] unfoldr :: (b -> Maybe (a, b)) -> b -> StreamK m a
223− [A] uncons :: Applicative m => StreamK m a -> m (Maybe (a, StreamK m a))
224− [A] unShare :: StreamK m a -> StreamK m a
225− [A] unCross :: CrossStreamK m a -> StreamK m a
226− [A] toStream :: Applicative m => StreamK m a -> Stream m a
227− [A] toList :: Monad m => StreamK m a -> m [a]
228− [A] the :: (Eq a, Monad m) => StreamK m a -> m (Maybe a)
229− [A] takeWhile :: (a -> Bool) -> StreamK m a -> StreamK m a
230− [A] take :: Int -> StreamK m a -> StreamK m a
231− [A] tail :: Applicative m => StreamK m a -> m (Maybe (StreamK m a))
232− [A] sortBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> StreamK m a
233− [A] sequence :: Monad m => StreamK m (m a) -> StreamK m a
234− [A] scanlx' :: (x -> a -> x) -> x -> (x -> b) -> StreamK m a -> StreamK m b
235− [A] scanl' :: (b -> a -> b) -> b -> StreamK m a -> StreamK m b
236− [A] reverse :: StreamK m a -> StreamK m a
237− [A] replicateMWith :: (m a -> StreamK m a -> StreamK m a) -> Int -> m a -> StreamK m a
238− [A] replicateM :: Monad m => Int -> m a -> StreamK m a
239− [A] replicate :: Int -> a -> StreamK m a
240− [A] repeatMWith :: (m a -> t m a -> t m a) -> m a -> t m a
241− [A] repeatM :: Monad m => m a -> StreamK m a
242− [A] repeat :: a -> StreamK m a
243− [A] parseDBreak :: Monad m => Parser a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
244− [A] parseD :: Monad m => Parser a m b -> StreamK m a -> m (Either ParseError b)
245− [A] parseChunksGeneric :: Monad m => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
246− [A] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
247− [A] parseBreakChunksGeneric :: forall m a b. Monad m => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
248− [A] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
249− [A] parseBreak :: forall m a b. Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b, StreamK m a)
250− [A] parse :: Monad m => ParserK a m b -> StreamK m a -> m (Either ParseError b)
251− [A] null :: Monad m => StreamK m a -> m Bool
252− [A] notElem :: (Monad m, Eq a) => a -> StreamK m a -> m Bool
253− [A] nilM :: Applicative m => m b -> StreamK m a
254− [A] nil :: StreamK m a
255− [A] mkStream :: (forall r. State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
256− [A] mkCross :: StreamK m a -> CrossStreamK m a
257− [A] minimumBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> m (Maybe a)
258− [A] minimum :: (Monad m, Ord a) => StreamK m a -> m (Maybe a)
259− [A] mfix :: Monad m => (m a -> StreamK m a) -> StreamK m a
260− [A] mergeMapWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m a -> StreamK m b
261− [A] mergeIterateWith :: (StreamK m a -> StreamK m a -> StreamK m a) -> (a -> StreamK m a) -> StreamK m a -> StreamK m a
262− [A] mergeByM :: Monad m => (a -> a -> m Ordering) -> StreamK m a -> StreamK m a -> StreamK m a
263− [A] mergeBy :: (a -> a -> Ordering) -> StreamK m a -> StreamK m a -> StreamK m a
264− [A] maximumBy :: Monad m => (a -> a -> Ordering) -> StreamK m a -> m (Maybe a)
265− [A] maximum :: (Monad m, Ord a) => StreamK m a -> m (Maybe a)
266− [A] mapMaybe :: (a -> Maybe b) -> StreamK m a -> StreamK m b
267− [A] mapM_ :: Monad m => (a -> m b) -> StreamK m a -> m ()
268− [A] mapMWith :: (m b -> StreamK m b -> StreamK m b) -> (a -> m b) -> StreamK m a -> StreamK m b
269− [A] mapMSerial :: Monad m => (a -> m b) -> StreamK m a -> StreamK m b
270− [A] mapM :: Monad m => (a -> m b) -> StreamK m a -> StreamK m b
271− [A] map :: (a -> b) -> StreamK m a -> StreamK m b
272− [A] lookup :: (Monad m, Eq a) => a -> StreamK m (a, b) -> m (Maybe b)
273− [A] liftInner :: (Monad m, MonadTrans t, Monad (t m)) => StreamK m a -> StreamK (t m) a
274− [A] last :: Monad m => StreamK m a -> m (Maybe a)
275− [A] iterateMWith :: Monad m => (m a -> StreamK m a -> StreamK m a) -> (a -> m a) -> m a -> StreamK m a
276− [A] iterateM :: Monad m => (a -> m a) -> m a -> StreamK m a
277− [A] iterate :: (a -> a) -> a -> StreamK m a
278− [A] intersperseM :: Monad m => m a -> StreamK m a -> StreamK m a
279− [A] intersperse :: Monad m => a -> StreamK m a -> StreamK m a
280− [A] interleaveMin :: StreamK m a -> StreamK m a -> StreamK m a
281− [A] interleaveFst :: StreamK m a -> StreamK m a -> StreamK m a
282− [A] interleave :: StreamK m a -> StreamK m a -> StreamK m a
283− [A] insertBy :: (a -> a -> Ordering) -> a -> StreamK m a -> StreamK m a
284− [A] init :: Applicative m => StreamK m a -> m (Maybe (StreamK m a))
285− [A] hoist :: (Monad m, Monad n) => (forall x. m x -> n x) -> StreamK m a -> StreamK n a
286− [A] head :: Monad m => StreamK m a -> m (Maybe a)
287− [A] handle :: (MonadCatch m, Exception e) => (e -> m (StreamK m a)) -> StreamK m a -> StreamK m a
288− [A] fromYieldK :: YieldK m a -> StreamK m a
289− [A] fromStream :: Monad m => Stream m a -> StreamK m a
290− [A] fromStopK :: StopK m -> StreamK m a
291− [A] fromPure :: a -> StreamK m a
292− [A] fromList :: [a] -> StreamK m a
293− [A] fromIndicesMWith :: (m a -> StreamK m a -> StreamK m a) -> (Int -> m a) -> StreamK m a
294− [A] fromIndicesM :: Monad m => (Int -> m a) -> StreamK m a
295− [A] fromIndices :: (Int -> a) -> StreamK m a
296− [A] fromFoldableM :: (Foldable f, Monad m) => f (m a) -> StreamK m a
297− [A] fromFoldable :: Foldable f => f a -> StreamK m a
298− [A] fromEffect :: Monad m => m a -> StreamK m a
299− [A] foldrT :: (Monad m, Monad (s m), MonadTrans s) => (a -> s m b -> s m b) -> s m b -> StreamK m a -> s m b
300− [A] foldrSShared :: (a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
301− [A] foldrSM :: Monad m => (m a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
302− [A] foldrS :: (a -> StreamK m b -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
303− [A] foldrM :: (a -> m b -> m b) -> m b -> StreamK m a -> m b
304− [A] foldr1 :: Monad m => (a -> a -> a) -> StreamK m a -> m (Maybe a)
305− [A] foldr :: Monad m => (a -> b -> b) -> b -> StreamK m a -> m b
306− [A] foldlx' :: forall m a b x. Monad m => (x -> a -> x) -> x -> (x -> b) -> StreamK m a -> m b
307− [A] foldlT :: (Monad m, Monad (s m), MonadTrans s) => (s m b -> a -> s m b) -> s m b -> StreamK m a -> s m b
308− [A] foldlS :: (StreamK m b -> a -> StreamK m b) -> StreamK m b -> StreamK m a -> StreamK m b
309− [A] foldlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> StreamK m a -> m b
310− [A] foldlM' :: Monad m => (b -> a -> m b) -> m b -> StreamK m a -> m b
311− [A] foldl' :: Monad m => (b -> a -> b) -> b -> StreamK m a -> m b
312− [A] foldStreamShared :: State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> StreamK m a -> m r
313− [A] foldStream :: State StreamK m a -> (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> StreamK m a -> m r
314− [A] foldEither :: Monad m => Fold m a b -> StreamK m a -> m (Either (Fold m a b) (b, StreamK m a))
315− [A] foldConcat :: Monad m => Producer m a b -> Fold m b c -> StreamK m a -> m (c, StreamK m a)
316− [A] foldBreak :: Monad m => Fold m a b -> StreamK m a -> m (b, StreamK m a)
317− [A] fold :: Monad m => Fold m a b -> StreamK m a -> m b
318− [A] findM :: Monad m => (a -> m Bool) -> StreamK m a -> m (Maybe a)
319− [A] findIndices :: (a -> Bool) -> StreamK m a -> StreamK m Int
320− [A] find :: Monad m => (a -> Bool) -> StreamK m a -> m (Maybe a)
321− [A] filter :: (a -> Bool) -> StreamK m a -> StreamK m a
322− [A] evalStateT :: Monad m => m s -> StreamK (StateT s m) a -> StreamK m a
323− [A] elem :: (Monad m, Eq a) => a -> StreamK m a -> m Bool
324− [A] dropWhile :: (a -> Bool) -> StreamK m a -> StreamK m a
325− [A] drop :: Int -> StreamK m a -> StreamK m a
326− [A] drain :: Monad m => StreamK m a -> m ()
327− [A] deleteBy :: (a -> a -> Bool) -> a -> StreamK m a -> StreamK m a
328− [A] crossWith :: Monad m => (a -> b -> c) -> StreamK m a -> StreamK m b -> StreamK m c
329− [A] crossApplyWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> StreamK m (a -> b) -> StreamK m a -> StreamK m b
330− [A] crossApplySnd :: StreamK m a -> StreamK m b -> StreamK m b
331− [A] crossApplyFst :: StreamK m a -> StreamK m b -> StreamK m a
332− [A] crossApply :: StreamK m (a -> b) -> StreamK m a -> StreamK m b
333− [A] cross :: Monad m => StreamK m a -> StreamK m b -> StreamK m (a, b)
334− [A] consMBy :: Monad m => (StreamK m a -> StreamK m a -> StreamK m a) -> m a -> StreamK m a -> StreamK m a
335− [A] consM :: Monad m => m a -> StreamK m a -> StreamK m a
336− [A] consK :: YieldK m a -> StreamK m a -> StreamK m a
337− [A] cons :: a -> StreamK m a -> StreamK m a
338− [A] conjoin :: Monad m => StreamK m a -> StreamK m a -> StreamK m a
339− [A] concatMapWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m a -> StreamK m b
340− [A] concatMapEffect :: Monad m => (b -> StreamK m a) -> m b -> StreamK m a
341− [A] concatMap :: (a -> StreamK m b) -> StreamK m a -> StreamK m b
342− [A] concatIterateWith :: (StreamK m a -> StreamK m a -> StreamK m a) -> (a -> StreamK m a) -> StreamK m a -> StreamK m a
343− [A] concatIterateScanWith :: Monad m => (StreamK m a -> StreamK m a -> StreamK m a) -> (b -> a -> m (b, StreamK m a)) -> m b -> StreamK m a -> StreamK m a
344− [A] concatIterateLeftsWith :: b ~ Either a c => (StreamK m b -> StreamK m b -> StreamK m b) -> (a -> StreamK m b) -> StreamK m b -> StreamK m b
345− [A] concatEffect :: Monad m => m (StreamK m a) -> StreamK m a
346− [A] buildSM :: Monad m => ((m a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a
347− [A] buildS :: ((a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a
348− [A] buildM :: Monad m => (forall r. (a -> StreamK m a -> m r) -> (a -> m r) -> m r -> m r) -> StreamK m a
349− [A] build :: forall m a. (forall b. (a -> b -> b) -> b -> b) -> StreamK m a
350− [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> StreamK m a) -> StreamK m a
351− [A] bindWith :: (StreamK m b -> StreamK m b -> StreamK m b) -> StreamK m a -> (a -> StreamK m b) -> StreamK m b
352− [A] before :: Monad m => m b -> StreamK m a -> StreamK m a
353− [A] augmentSM :: Monad m => ((m a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a
354− [A] augmentS :: ((a -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a) -> StreamK m a -> StreamK m a
355− [A] append :: StreamK m a -> StreamK m a -> StreamK m a
356− [A] any :: Monad m => (a -> Bool) -> StreamK m a -> m Bool
357− [A] all :: Monad m => (a -> Bool) -> StreamK m a -> m Bool
358− [A] (.:) :: a -> StreamK m a -> StreamK m a
359− [A] (!!) :: Monad m => StreamK m a -> Int -> m (Maybe a)
360−[R] Streamly.Internal.Data.Stream.StreamK.Type
361−[R] Streamly.Internal.Data.Stream.StreamK.Transformer
362−[R] Streamly.Internal.Data.Stream.StreamK
363−[R] Streamly.Internal.Data.Stream.StreamD.Type
364−[R] Streamly.Internal.Data.Stream.StreamD.Transformer
365−[R] Streamly.Internal.Data.Stream.StreamD.Transform
366−[R] Streamly.Internal.Data.Stream.StreamD.Top
367−[R] Streamly.Internal.Data.Stream.StreamD.Step
368−[R] Streamly.Internal.Data.Stream.StreamD.Nesting
369−[R] Streamly.Internal.Data.Stream.StreamD.Lift
370−[R] Streamly.Internal.Data.Stream.StreamD.Generate
371−[R] Streamly.Internal.Data.Stream.StreamD.Exception
372−[R] Streamly.Internal.Data.Stream.StreamD.Eliminate
373−[R] Streamly.Internal.Data.Stream.StreamD.Container
374−[D] Streamly.Internal.Data.Stream.StreamD
375−[R] Streamly.Internal.Data.Stream.Common
376−[R] Streamly.Internal.Data.Stream.Chunked
377−[C] Streamly.Internal.Data.Stream
378− [A] class Enum a => Enumerable a
379− [A] Stream
380− [A] UnStream :: (State StreamK m a -> s -> m (Step s a)) -> s -> Stream m a
381− [A] Step
382− [A] Yield :: a -> s -> Step s a
383− [A] Stop :: Step s a
384− [A] Skip :: s -> Step s a
385− [A] InterleaveState
386− [A] InterleaveSecondOnly :: s2 -> InterleaveState s1 s2
387− [A] InterleaveSecond :: s1 -> s2 -> InterleaveState s1 s2
388− [A] InterleaveFirstOnly :: s1 -> InterleaveState s1 s2
389− [A] InterleaveFirst :: s1 -> s2 -> InterleaveState s1 s2
390− [A] FoldManyPost
391− [A] FoldManyPostYield :: b -> FoldManyPost s fs b a -> FoldManyPost s fs b a
392− [A] FoldManyPostStart :: s -> FoldManyPost s fs b a
393− [A] FoldManyPostLoop :: s -> fs -> FoldManyPost s fs b a
394− [A] FoldManyPostDone :: FoldManyPost s fs b a
395− [A] FoldMany
396− [A] FoldManyYield :: b -> FoldMany s fs b a -> FoldMany s fs b a
397− [A] FoldManyStart :: s -> FoldMany s fs b a
398− [A] FoldManyLoop :: s -> fs -> FoldMany s fs b a
399− [A] FoldManyFirst :: fs -> s -> FoldMany s fs b a
400− [A] FoldManyDone :: FoldMany s fs b a
401− [A] CrossStream
402− [A] ConcatUnfoldInterleaveState
403− [A] ConcatUnfoldInterleaveOuter :: o -> [i] -> ConcatUnfoldInterleaveState o i
404− [A] ConcatUnfoldInterleaveInnerR :: [i] -> [i] -> ConcatUnfoldInterleaveState o i
405− [A] ConcatUnfoldInterleaveInnerL :: [i] -> [i] -> ConcatUnfoldInterleaveState o i
406− [A] ConcatUnfoldInterleaveInner :: o -> [i] -> ConcatUnfoldInterleaveState o i
407− [A] ConcatMapUState
408− [A] ConcatMapUOuter :: o -> ConcatMapUState o i
409− [A] ConcatMapUInner :: o -> i -> ConcatMapUState o i
410− [A] AppendState
411− [A] AppendSecond :: s2 -> AppendState s1 s2
412− [A] AppendFirst :: s1 -> AppendState s1 s2
413− [A] pattern Stream :: (State StreamK m a -> s -> m (Step s a)) -> s -> Stream m a
414− [A] zipWithM :: Monad m => (a -> b -> m c) -> Stream m a -> Stream m b -> Stream m c
415− [A] zipWith :: Monad m => (a -> b -> c) -> Stream m a -> Stream m b -> Stream m c
416− [A] wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
417− [A] with :: Monad m => (Stream m a -> Stream m (s, a)) -> (((s, a) -> b) -> Stream m (s, a) -> Stream m (s, a)) -> ((s, a) -> b) -> Stream m a -> Stream m a
418− [A] usingStateT :: Monad m => m s -> (Stream (StateT s m) a -> Stream (StateT s m) a) -> Stream m a -> Stream m a
419− [A] usingReaderT :: Monad m => m r -> (Stream (ReaderT r m) a -> Stream (ReaderT r m) a) -> Stream m a -> Stream m a
420− [A] uniqBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a
421− [A] uniq :: (Eq a, Monad m) => Stream m a -> Stream m a
422− [A] unionWithStreamGenericBy :: MonadIO m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
423− [A] unionWithStreamAscBy :: (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
424− [A] unfoldrM :: Monad m => (s -> m (Maybe (a, s))) -> s -> Stream m a
425− [A] unfoldr :: Monad m => (s -> Maybe (a, s)) -> s -> Stream m a
426− [A] unfoldRoundRobin :: Monad m => Unfold m a b -> Stream m a -> Stream m b
427− [A] unfoldMany :: Monad m => Unfold m a b -> Stream m a -> Stream m b
428− [A] unfoldIterateDfs :: Monad m => Unfold m a a -> Stream m a -> Stream m a
429− [A] unfoldIterateBfsRev :: Monad m => Unfold m a a -> Stream m a -> Stream m a
430− [A] unfoldIterateBfs :: Monad m => Unfold m a a -> Stream m a -> Stream m a
431− [A] unfoldInterleave :: Monad m => Unfold m a b -> Stream m a -> Stream m b
432− [A] unfold :: Applicative m => Unfold m a b -> a -> Stream m b
433− [A] uncons :: Monad m => Stream m a -> m (Maybe (a, Stream m a))
434− [A] unCross :: CrossStream m a -> Stream m a
435− [A] transform :: Monad m => Pipe m a b -> Stream m a -> Stream m b
436− [A] trace_ :: Monad m => m b -> Stream m a -> Stream m a
437− [A] trace :: Monad m => (a -> m b) -> Stream m a -> Stream m a
438− [A] toStreamK :: Monad m => Stream m a -> StreamK m a
439− [A] toListRev :: Monad m => Stream m a -> m [a]
440− [A] toList :: Monad m => Stream m a -> m [a]
441− [A] timestamped :: MonadIO m => Stream m a -> Stream m (AbsTime, a)
442− [A] timestampWith :: MonadIO m => Double -> Stream m a -> Stream m (AbsTime, a)
443− [A] timesWith :: MonadIO m => Double -> Stream m (AbsTime, RelTime64)
444− [A] times :: MonadIO m => Stream m (AbsTime, RelTime64)
445− [A] timeout :: AbsTime -> t m ()
446− [A] timeIndexed :: MonadIO m => Stream m a -> Stream m (RelTime64, a)
447− [A] timeIndexWith :: MonadIO m => Double -> Stream m a -> Stream m (RelTime64, a)
448− [A] the :: (Eq a, Monad m) => Stream m a -> m (Maybe a)
449− [A] tapOffsetEvery :: Monad m => Int -> Int -> Fold m a b -> Stream m a -> Stream m a
450− [A] tap :: Monad m => Fold m a b -> Stream m a -> Stream m a
451− [A] takeWhileM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
452− [A] takeWhileLast :: (a -> Bool) -> Stream m a -> Stream m a
453− [A] takeWhileAround :: (a -> Bool) -> Stream m a -> Stream m a
454− [A] takeWhile :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
455− [A] takeEndByM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
456− [A] takeEndBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
457− [A] take :: Applicative m => Int -> Stream m a -> Stream m a
458− [A] tail :: Monad m => Stream m a -> m (Maybe (Stream m a))
459− [A] stripSuffixUnbox :: (MonadIO m, Eq a, Unbox a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
460− [A] stripSuffix :: (Monad m, Eq a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
461− [A] stripPrefix :: (Monad m, Eq a) => Stream m a -> Stream m a -> m (Maybe (Stream m a))
462− [A] strideFromThen :: Monad m => Int -> Int -> Stream m a -> Stream m a
463− [A] splitOnSuffixSeqAny :: [Array a] -> Fold m a b -> Stream m a -> Stream m b
464− [A] splitOnSuffixSeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) => Bool -> Array a -> Fold m a b -> Stream m a -> Stream m b
465− [A] splitOnSeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) => Array a -> Fold m a b -> Stream m a -> Stream m b
466− [A] splitOnPrefix :: (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
467− [A] splitOnAny :: [Array a] -> Fold m a b -> Stream m a -> Stream m b
468− [A] splitOn :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
469− [A] splitInnerBySuffix :: (Monad m, Eq (f a), Monoid (f a)) => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
470− [A] splitInnerBy :: Monad m => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
471− [A] slicesBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
472− [A] sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
473− [A] sequence :: Monad m => Stream m (m a) -> Stream m a
474− [A] scanlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> Stream m b
475− [A] scanlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> Stream m b
476− [A] scanlMAfter' :: Monad m => (b -> a -> m b) -> m b -> (b -> m b) -> Stream m a -> Stream m b
477− [A] scanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
478− [A] scanlM :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
479− [A] scanl1M' :: Monad m => (a -> a -> m a) -> Stream m a -> Stream m a
480− [A] scanl1M :: Monad m => (a -> a -> m a) -> Stream m a -> Stream m a
481− [A] scanl1' :: Monad m => (a -> a -> a) -> Stream m a -> Stream m a
482− [A] scanl1 :: Monad m => (a -> a -> a) -> Stream m a -> Stream m a
483− [A] scanl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
484− [A] scanl :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
485− [A] scanMaybe :: Monad m => Fold m a (Maybe b) -> Stream m a -> Stream m b
486− [A] scanMany :: Monad m => Fold m a b -> Stream m a -> Stream m b
487− [A] scan :: Monad m => Fold m a b -> Stream m a -> Stream m b
488− [A] runStateT :: Monad m => m s -> Stream (StateT s m) a -> Stream m (s, a)
489− [A] runReaderT :: Monad m => m s -> Stream (ReaderT s m) a -> Stream m a
490− [A] runInnerWithState :: Monad m => (forall b. s -> t m b -> m (b, s)) -> m s -> Stream (t m) a -> Stream m (s, a)
491− [A] runInnerWith :: Monad m => (forall b. t m b -> m b) -> Stream (t m) a -> Stream m a
492− [A] roundRobin :: Monad m => Stream m a -> Stream m a -> Stream m a
493− [A] rollingMapM :: Monad m => (Maybe a -> a -> m b) -> Stream m a -> Stream m b
494− [A] rollingMap2 :: Monad m => (a -> a -> b) -> Stream m a -> Stream m b
495− [A] rollingMap :: Monad m => (Maybe a -> a -> b) -> Stream m a -> Stream m b
496− [A] reverseUnbox :: (MonadIO m, Unbox a) => Stream m a -> Stream m a
497− [A] reverse :: Monad m => Stream m a -> Stream m a
498− [A] replicateM :: Monad m => Int -> m a -> Stream m a
499− [A] replicate :: Monad m => Int -> a -> Stream m a
500− [A] repeated :: Stream m a -> Stream m a
501− [A] repeatM :: Monad m => m a -> Stream m a
502− [A] repeat :: Monad m => a -> Stream m a
503− [A] relTimesWith :: MonadIO m => Double -> Stream m RelTime64
504− [A] relTimes :: MonadIO m => Stream m RelTime64
505− [A] refoldMany :: Monad m => Refold m x a b -> m x -> Stream m a -> Stream m b
506− [A] refoldIterateM :: Monad m => Refold m b a b -> m b -> Stream m a -> Stream m b
507− [A] reduceIterateBfs :: Monad m => (a -> a -> m a) -> Stream m a -> m (Maybe a)
508− [A] reassembleBy :: Fold m a b -> (a -> a -> Int) -> Stream m a -> Stream m b
509− [A] prune :: (a -> Bool) -> Stream m a -> Stream m a
510− [A] prescanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
511− [A] prescanl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> Stream m b
512− [A] postscanlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> Stream m b
513− [A] postscanlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> Stream m b
514− [A] postscanlMAfter' :: Monad m => (b -> a -> m b) -> m b -> (b -> m b) -> Stream m a -> Stream m b
515− [A] postscanlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
516− [A] postscanlM :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> Stream m b
517− [A] postscanl' :: Monad m => (a -> b -> a) -> a -> Stream m b -> Stream m a
518− [A] postscanl :: Monad m => (a -> b -> a) -> a -> Stream m b -> Stream m a
519− [A] postscan :: Monad m => Fold m a b -> Stream m a -> Stream m b
520− [A] parseSequence :: Stream m (Parser a m b) -> Stream m a -> Stream m b
521− [A] parseManyTill :: Parser a m b -> Parser a m x -> Stream m a -> Stream m b
522− [A] parseManyD :: Monad m => Parser a m b -> Stream m a -> Stream m (Either ParseError b)
523− [A] parseMany :: Monad m => Parser a m b -> Stream m a -> Stream m (Either ParseError b)
524− [A] parseIterateD :: Monad m => (b -> Parser a m b) -> b -> Stream m a -> Stream m (Either ParseError b)
525− [A] parseIterate :: Monad m => (b -> Parser a m b) -> b -> Stream m a -> Stream m (Either ParseError b)
526− [A] parseD :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b)
527− [A] parseBreakD :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b, Stream m a)
528− [A] parseBreak :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b, Stream m a)
529− [A] parse :: Monad m => Parser a m b -> Stream m a -> m (Either ParseError b)
530− [A] onException :: MonadCatch m => m b -> Stream m a -> Stream m a
531− [A] null :: Monad m => Stream m a -> m Bool
532− [A] nub :: (Monad m, Ord a) => Stream m a -> Stream m a
533− [A] notElem :: (Monad m, Eq a) => a -> Stream m a -> m Bool
534− [A] nilM :: Applicative m => m b -> Stream m a
535− [A] nil :: Applicative m => Stream m a
536− [A] morphInner :: Monad n => (forall x. m x -> n x) -> Stream m a -> Stream n a
537− [A] mkCross :: Stream m a -> CrossStream m a
538− [A] minimumBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> m (Maybe a)
539− [A] minimum :: (Monad m, Ord a) => Stream m a -> m (Maybe a)
540− [A] mergeMinBy :: (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
541− [A] mergeFstBy :: (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
542− [A] mergeByM :: Monad m => (a -> a -> m Ordering) -> Stream m a -> Stream m a -> Stream m a
543− [A] mergeBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
544− [A] maximumBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> m (Maybe a)
545− [A] maximum :: (Monad m, Ord a) => Stream m a -> m (Maybe a)
546− [A] mapMaybeM :: Monad m => (a -> m (Maybe b)) -> Stream m a -> Stream m b
547− [A] mapMaybe :: Monad m => (a -> Maybe b) -> Stream m a -> Stream m b
548− [A] mapM_ :: Monad m => (a -> m b) -> Stream m a -> m ()
549− [A] mapM :: Monad m => (a -> m b) -> Stream m a -> Stream m b
550− [A] map :: Monad m => (a -> b) -> Stream m a -> Stream m b
551− [A] lookup :: (Monad m, Eq a) => a -> Stream m (a, b) -> m (Maybe b)
552− [A] liftInnerWith :: Monad (t m) => (forall b. m b -> t m b) -> Stream m a -> Stream (t m) a
553− [A] liftInner :: (Monad m, MonadTrans t, Monad (t m)) => Stream m a -> Stream (t m) a
554− [A] last :: Monad m => Stream m a -> m (Maybe a)
555− [A] joinOuterGeneric :: MonadIO m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (Maybe a, Maybe b)
556− [A] joinOuterAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (Maybe a, Maybe b)
557− [A] joinOuter :: (Ord k, MonadIO m) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, Maybe a, Maybe b)
558− [A] joinLeftGeneric :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
559− [A] joinLeftAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
560− [A] joinLeft :: (Ord k, Monad m) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, Maybe b)
561− [A] joinInnerGeneric :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, b)
562− [A] joinInnerAscBy :: (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, b)
563− [A] joinInner :: (Monad m, Ord k) => Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, b)
564− [A] iterateM :: Monad m => (a -> m a) -> m a -> Stream m a
565− [A] iterate :: Monad m => (a -> a) -> a -> Stream m a
566− [A] isSuffixOfUnbox :: (MonadIO m, Eq a, Unbox a) => Stream m a -> Stream m a -> m Bool
567− [A] isSuffixOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
568− [A] isSubsequenceOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
569− [A] isPrefixOf :: (Monad m, Eq a) => Stream m a -> Stream m a -> m Bool
570− [A] isInfixOf :: (MonadIO m, Eq a, Enum a, Storable a, Unbox a) => Stream m a -> Stream m a -> m Bool
571− [A] intersperseM_ :: Monad m => m b -> Stream m a -> Stream m a
572− [A] intersperseMWith :: Int -> m a -> Stream m a -> Stream m a
573− [A] intersperseMSuffix_ :: Monad m => m b -> Stream m a -> Stream m a
574− [A] intersperseMSuffixWith :: forall m a. Monad m => Int -> m a -> Stream m a -> Stream m a
575− [A] intersperseMSuffix :: forall m a. Monad m => m a -> Stream m a -> Stream m a
576− [A] intersperseMPrefix_ :: Monad m => m b -> Stream m a -> Stream m a
577− [A] intersperseM :: Monad m => m a -> Stream m a -> Stream m a
578− [A] intersperse :: Monad m => a -> Stream m a -> Stream m a
579− [A] intersectBySorted :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
580− [A] interposeSuffixM :: Monad m => m c -> Unfold m b c -> Stream m b -> Stream m c
581− [A] interposeSuffix :: Monad m => c -> Unfold m b c -> Stream m b -> Stream m c
582− [A] interposeM :: Monad m => m c -> Unfold m b c -> Stream m b -> Stream m c
583− [A] interpose :: Monad m => c -> Unfold m b c -> Stream m b -> Stream m c
584− [A] interleaveMin :: Monad m => Stream m a -> Stream m a -> Stream m a
585− [A] interleaveFstSuffix :: Monad m => Stream m a -> Stream m a -> Stream m a
586− [A] interleaveFst :: Monad m => Stream m a -> Stream m a -> Stream m a
587− [A] interleave :: Monad m => Stream m a -> Stream m a -> Stream m a
588− [A] intercalateSuffix :: Monad m => Unfold m b c -> b -> Stream m b -> Stream m c
589− [A] intercalate :: Monad m => Unfold m b c -> b -> Stream m b -> Stream m c
590− [A] insertBy :: Monad m => (a -> a -> Ordering) -> a -> Stream m a -> Stream m a
591− [A] indexedR :: Monad m => Int -> Stream m a -> Stream m (Int, a)
592− [A] indexed :: Monad m => Stream m a -> Stream m (Int, a)
593− [A] headElse :: Monad m => a -> Stream m a -> m a
594− [A] head :: Monad m => Stream m a -> m (Maybe a)
595− [A] handle :: (MonadCatch m, Exception e) => (e -> m (Stream m a)) -> Stream m a -> Stream m a
596− [A] groupsWhile :: Monad m => (a -> a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
597− [A] groupsRollingBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
598− [A] groupsOf :: Monad m => Int -> Fold m a b -> Stream m a -> Stream m b
599− [A] gintercalateSuffix :: Monad m => Unfold m a c -> Stream m a -> Unfold m b c -> Stream m b -> Stream m c
600− [A] gintercalate :: Monad m => Unfold m a c -> Stream m a -> Unfold m b c -> Stream m b -> Stream m c
601− [A] ghandle :: (MonadCatch m, Exception e) => (e -> Stream m a -> m (Stream m a)) -> Stream m a -> Stream m a
602− [A] generateM :: Monad m => Int -> (Int -> m a) -> Stream m a
603− [A] generate :: Monad m => Int -> (Int -> a) -> Stream m a
604− [A] generalizeInner :: Monad m => Stream Identity a -> Stream m a
605− [A] gbracket_ :: Monad m => m c -> (c -> m d) -> (c -> e -> Stream m b -> m (Stream m b)) -> (forall s. m s -> m (Either e s)) -> (c -> Stream m b) -> Stream m b
606− [A] gbracket :: MonadIO m => IO c -> (c -> IO d1) -> (c -> e -> Stream m b -> IO (Stream m b)) -> (c -> IO d2) -> (forall s. m s -> m (Either e s)) -> (c -> Stream m b) -> Stream m b
607− [A] fromStreamK :: Applicative m => StreamK m a -> Stream m a
608− [A] fromPure :: Applicative m => a -> Stream m a
609− [A] fromPtrN :: (Monad m, Storable a) => Int -> Ptr a -> Stream m a
610− [A] fromPtr :: forall m a. (Monad m, Storable a) => Ptr a -> Stream m a
611− [A] fromListM :: Monad m => [m a] -> Stream m a
612− [A] fromList :: Applicative m => [a] -> Stream m a
613− [A] fromIndicesM :: Monad m => (Int -> m a) -> Stream m a
614− [A] fromIndices :: Monad m => (Int -> a) -> Stream m a
615− [A] fromFoldableM :: (Monad m, Foldable f) => f (m a) -> Stream m a
616− [A] fromFoldable :: (Monad m, Foldable f) => f a -> Stream m a
617− [A] fromEffect :: Applicative m => m a -> Stream m a
618− [A] fromByteStr# :: Monad m => Addr# -> Stream m Word8
619− [A] foldrT :: (Monad m, Monad (t m), MonadTrans t) => (a -> t m b -> t m b) -> t m b -> Stream m a -> t m b
620− [A] foldrS :: Monad m => (a -> Stream m b -> Stream m b) -> Stream m b -> Stream m a -> Stream m b
621− [A] foldrMx :: Monad m => (a -> m x -> m x) -> m x -> (m x -> m b) -> Stream m a -> m b
622− [A] foldrM :: Monad m => (a -> m b -> m b) -> m b -> Stream m a -> m b
623− [A] foldr1 :: Monad m => (a -> a -> a) -> Stream m a -> m (Maybe a)
624− [A] foldr :: Monad m => (a -> b -> b) -> b -> Stream m a -> m b
625− [A] foldlx' :: Monad m => (x -> a -> x) -> x -> (x -> b) -> Stream m a -> m b
626− [A] foldlT :: (Monad m, Monad (s m), MonadTrans s) => (s m b -> a -> s m b) -> s m b -> Stream m a -> s m b
627− [A] foldlS :: Monad m => (Stream m b -> a -> Stream m b) -> Stream m b -> Stream m a -> Stream m b
628− [A] foldlMx' :: Monad m => (x -> a -> m x) -> m x -> (x -> m b) -> Stream m a -> m b
629− [A] foldlM' :: Monad m => (b -> a -> m b) -> m b -> Stream m a -> m b
630− [A] foldl' :: Monad m => (b -> a -> b) -> b -> Stream m a -> m b
631− [A] foldSequence :: Stream m (Fold m a b) -> Stream m a -> Stream m b
632− [A] foldManyPost :: Monad m => Fold m a b -> Stream m a -> Stream m b
633− [A] foldMany :: Monad m => Fold m a b -> Stream m a -> Stream m b
634− [A] foldIterateM :: Monad m => (b -> m (Fold m a b)) -> m b -> Stream m a -> Stream m b
635− [A] foldIterateBfs :: Fold m a (Either a a) -> Stream m a -> m (Maybe a)
636− [A] foldEither :: Monad m => Fold m a b -> Stream m a -> m (Either (Fold m a b) (b, Stream m a))
637− [A] foldBreak :: Monad m => Fold m a b -> Stream m a -> m (b, Stream m a)
638− [A] foldAddLazy :: Monad m => Fold m a b -> Stream m a -> Fold m a b
639− [A] foldAdd :: Monad m => Fold m a b -> Stream m a -> m (Fold m a b)
640− [A] fold :: Monad m => Fold m a b -> Stream m a -> m b
641− [A] findM :: Monad m => (a -> m Bool) -> Stream m a -> m (Maybe a)
642− [A] findIndices :: Monad m => (a -> Bool) -> Stream m a -> Stream m Int
643− [A] find :: Monad m => (a -> Bool) -> Stream m a -> m (Maybe a)
644− [A] finallyUnsafe :: MonadCatch m => m b -> Stream m a -> Stream m a
645− [A] finallyIO :: (MonadIO m, MonadCatch m) => IO b -> Stream m a -> Stream m a
646− [A] filterM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
647− [A] filterInStreamGenericBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
648− [A] filterInStreamAscBy :: Monad m => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
649− [A] filter :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
650− [A] evalStateT :: Monad m => m s -> Stream (StateT s m) a -> Stream m a
651− [A] eqBy :: Monad m => (a -> b -> Bool) -> Stream m a -> Stream m b -> m Bool
652− [A] enumerateTo :: (Monad m, Bounded a, Enumerable a) => a -> Stream m a
653− [A] enumerateFromToSmall :: (Monad m, Enum a) => a -> a -> Stream m a
654− [A] enumerateFromToIntegral :: (Monad m, Integral a) => a -> a -> Stream m a
655− [A] enumerateFromToFractional :: (Monad m, Fractional a, Ord a) => a -> a -> Stream m a
656− [A] enumerateFromTo :: (Enumerable a, Monad m) => a -> a -> Stream m a
657− [A] enumerateFromThenToSmall :: (Monad m, Enum a) => a -> a -> a -> Stream m a
658− [A] enumerateFromThenToIntegral :: (Monad m, Integral a) => a -> a -> a -> Stream m a
659− [A] enumerateFromThenToFractional :: (Monad m, Fractional a, Ord a) => a -> a -> a -> Stream m a
660− [A] enumerateFromThenTo :: (Enumerable a, Monad m) => a -> a -> a -> Stream m a
661− [A] enumerateFromThenSmallBounded :: (Monad m, Enumerable a, Bounded a) => a -> a -> Stream m a
662− [A] enumerateFromThenNum :: (Monad m, Num a) => a -> a -> Stream m a
663− [A] enumerateFromThenIntegral :: (Monad m, Integral a, Bounded a) => a -> a -> Stream m a
664− [A] enumerateFromThenFractional :: (Monad m, Fractional a) => a -> a -> Stream m a
665− [A] enumerateFromThen :: (Enumerable a, Monad m) => a -> a -> Stream m a
666− [A] enumerateFromStepNum :: (Monad m, Num a) => a -> a -> Stream m a
667− [A] enumerateFromStepIntegral :: (Integral a, Monad m) => a -> a -> Stream m a
668− [A] enumerateFromNum :: (Monad m, Num a) => a -> Stream m a
669− [A] enumerateFromIntegral :: (Monad m, Integral a, Bounded a) => a -> Stream m a
670− [A] enumerateFromFractional :: (Monad m, Fractional a) => a -> Stream m a
671− [A] enumerateFromBounded :: (Monad m, Enumerable a, Bounded a) => a -> Stream m a
672− [A] enumerateFrom :: (Enumerable a, Monad m) => a -> Stream m a
673− [A] enumerate :: (Monad m, Bounded a, Enumerable a) => Stream m a
674− [A] elemIndices :: (Monad m, Eq a) => a -> Stream m a -> Stream m Int
675− [A] elem :: (Monad m, Eq a) => a -> Stream m a -> m Bool
676− [A] durations :: Double -> t m RelTime64
677− [A] dropWhileM :: Monad m => (a -> m Bool) -> Stream m a -> Stream m a
678− [A] dropWhileLast :: (a -> Bool) -> Stream m a -> Stream m a
679− [A] dropWhileAround :: (a -> Bool) -> Stream m a -> Stream m a
680− [A] dropWhile :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
681− [A] dropSuffix :: Stream m a -> Stream m a -> Stream m a
682− [A] dropPrefix :: Stream m a -> Stream m a -> Stream m a
683− [A] dropLast :: Int -> Stream m a -> Stream m a
684− [A] dropInfix :: Stream m a -> Stream m a -> Stream m a
685− [A] drop :: Monad m => Int -> Stream m a -> Stream m a
686− [A] drain :: Monad m => Stream m a -> m ()
687− [A] deleteInStreamGenericBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
688− [A] deleteInStreamAscBy :: (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
689− [A] deleteBy :: Monad m => (a -> a -> Bool) -> a -> Stream m a -> Stream m a
690− [A] delayPre :: MonadIO m => Double -> Stream m a -> Stream m a
691− [A] delayPost :: MonadIO m => Double -> Stream m a -> Stream m a
692− [A] delay :: MonadIO m => Double -> Stream m a -> Stream m a
693− [A] crossWith :: Monad m => (a -> b -> c) -> Stream m a -> Stream m b -> Stream m c
694− [A] crossApplySnd :: Functor f => Stream f a -> Stream f b -> Stream f b
695− [A] crossApplyFst :: Functor f => Stream f a -> Stream f b -> Stream f a
696− [A] crossApply :: Functor f => Stream f (a -> b) -> Stream f a -> Stream f b
697− [A] cross :: Monad m => Stream m a -> Stream m b -> Stream m (a, b)
698− [A] consM :: Applicative m => m a -> Stream m a -> Stream m a
699− [A] cons :: Applicative m => a -> Stream m a -> Stream m a
700− [A] concatMapM :: Monad m => (a -> m (Stream m b)) -> Stream m a -> Stream m b
701− [A] concatMap :: Monad m => (a -> Stream m b) -> Stream m a -> Stream m b
702− [A] concatIterateScan :: Monad m => (b -> a -> m b) -> (b -> m (Maybe (b, Stream m a))) -> b -> Stream m a
703− [A] concatIterateDfs :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
704− [A] concatIterateBfsRev :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
705− [A] concatIterateBfs :: Monad m => (a -> Maybe (Stream m a)) -> Stream m a -> Stream m a
706− [A] concatEffect :: Monad m => m (Stream m a) -> Stream m a
707− [A] concat :: Monad m => Stream m (Stream m a) -> Stream m a
708− [A] cmpBy :: Monad m => (a -> b -> Ordering) -> Stream m a -> Stream m b -> m Ordering
709− [A] catRights :: Monad m => Stream m (Either a b) -> Stream m b
710− [A] catMaybes :: Monad m => Stream m (Maybe a) -> Stream m a
711− [A] catLefts :: Monad m => Stream m (Either a b) -> Stream m a
712− [A] catEithers :: Monad m => Stream m (Either a a) -> Stream m a
713− [A] bracketUnsafe :: MonadCatch m => m b -> (b -> m c) -> (b -> Stream m a) -> Stream m a
714− [A] bracketIO3 :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> IO d) -> (b -> IO e) -> (b -> Stream m a) -> Stream m a
715− [A] bracketIO :: (MonadIO m, MonadCatch m) => IO b -> (b -> IO c) -> (b -> Stream m a) -> Stream m a
716− [A] before :: Monad m => m b -> Stream m a -> Stream m a
717− [A] append :: Monad m => Stream m a -> Stream m a -> Stream m a
718− [A] any :: Monad m => (a -> Bool) -> Stream m a -> m Bool
719− [A] all :: Monad m => (a -> Bool) -> Stream m a -> m Bool
720− [A] afterUnsafe :: Monad m => m b -> Stream m a -> Stream m a
721− [A] afterIO :: MonadIO m => IO b -> Stream m a -> Stream m a
722− [A] absTimesWith :: MonadIO m => Double -> Stream m AbsTime
723− [A] absTimes :: MonadIO m => Stream m AbsTime
724− [A] (!!) :: Monad m => Stream m a -> Int -> m (Maybe a)
725−[R] Streamly.Internal.Data.Ring.Unboxed
726−[A] Streamly.Internal.Data.Ring.Generic
727− [A] Ring
728− [A] [ringMax] :: Ring a -> !Int
729− [A] [ringHead] :: Ring a -> !Int
730− [A] [ringArr] :: Ring a -> MutArray a
731− [A] Ring :: MutArray a -> !Int -> !Int -> Ring a
732− [A] writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
733− [A] unsafeInsertRingWith :: Ring a -> a -> IO Int
734− [A] toStreamWith :: Int -> Ring a -> Stream m a
735− [A] toMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
736− [A] seek :: MonadIO m => Int -> Ring a -> m (Ring a)
737− [A] createRing :: MonadIO m => Int -> m (Ring a)
738− [A] copyToMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
739−[C] Streamly.Internal.Data.Ring
740− [C] Ring
741− [A] [ringStart] :: Ring a -> {-# UNPACK #-} !ForeignPtr a
742− [R] [ringMax] :: Ring a -> !Int
743− [R] [ringHead] :: Ring a -> !Int
744− [A] [ringBound] :: Ring a -> {-# UNPACK #-} !Ptr a
745− [R] [ringArr] :: Ring a -> MutArray a
746− [C] Ring
747− [O] Ring :: MutArray a -> !Int -> !Int -> Ring a
748− [N] Ring :: {-# UNPACK #-} !ForeignPtr a -> {-# UNPACK #-} !Ptr a -> Ring a
749− [A] GHC.Show.Show
750− [A] instance (GHC.Show.Show a, GHC.Show.Show b, GHC.Show.Show c, GHC.Show.Show d) => GHC.Show.Show (Streamly.Internal.Data.Ring.Tuple4' a b c d)
751− [A] writeN :: Int -> Fold m a (Ring a)
752− [R] writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
753− [R] unsafeInsertRingWith :: Ring a -> a -> IO Int
754− [A] unsafeInsert :: Storable a => Ring a -> Ptr a -> a -> IO (Ptr a)
755− [A] unsafeFoldRingNM :: forall m a b. (MonadIO m, Storable a) => Int -> Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
756− [A] unsafeFoldRingM :: forall m a b. (MonadIO m, Storable a) => Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
757− [A] unsafeFoldRingFullM :: forall m a b. (MonadIO m, Storable a) => Ptr a -> (b -> a -> m b) -> b -> Ring a -> m b
758− [A] unsafeFoldRing :: forall a b. Storable a => Ptr a -> (b -> a -> b) -> b -> Ring a -> b
759− [A] unsafeEqArrayN :: Ring a -> Ptr a -> Array a -> Int -> Bool
760− [A] unsafeEqArray :: Ring a -> Ptr a -> Array a -> Bool
761− [R] toStreamWith :: Int -> Ring a -> Stream m a
762− [R] toMutArray :: MonadIO m => Int -> Int -> Ring a -> m (MutArray a)
763− [A] startOf :: Ring a -> Ptr a
764− [A] slidingWindowWith :: forall m a b. (MonadIO m, Storable a, Unbox a) => Int -> Fold m ((a, Maybe a), m (MutArray a)) b -> Fold m a b
765− [A] slidingWindow :: forall m a b. (MonadIO m, Storable a, Unbox a) => Int -> Fold m (a, Maybe a) b -> Fold m a b
766− [A] slide :: Ring a -> a -> m (Ring a)
767− [R] seek :: MonadIO m => Int -> Ring a -> m (Ring a)
768− [A] ringsOf :: Int -> Stream m a -> Stream m (MutArray a)
769− [A] readRev :: Unfold m (MutArray a) a
770− [A] read :: forall m a. (MonadIO m, Storable a) => Unfold m (Ring a, Ptr a, Int) a
771− [A] putIndex :: Ring a -> Int -> a -> m ()
772− [A] newRing :: Int -> m (Ring a)
773− [A] new :: forall a. Storable a => Int -> IO (Ring a, Ptr a)
774− [A] moveBy :: forall a. Storable a => Int -> Ring a -> Ptr a -> Ptr a
775− [A] modifyIndex :: Ring a -> Int -> (a -> (a, b)) -> m b
776− [A] length :: Ring a -> Int
777− [A] getIndexUnsafe :: Ring a -> Int -> m a
778− [A] getIndexRev :: Ring a -> Int -> m a
779− [A] getIndex :: Ring a -> Int -> m a
780− [A] fromArray :: MutArray a -> Ring a
781− [R] createRing :: MonadIO m => Int -> m (Ring a)
782− [A] castUnsafe :: Ring a -> Ring b
783− [A] cast :: forall a b. Storable b => Ring a -> Maybe (Ring b)
784− [A] bytesFree :: Ring a -> Int
785− [A] byteLength :: Ring a -> Int
786− [A] byteCapacity :: Ring a -> Int
787− [A] asBytes :: Ring a -> Ring Word8
788− [A] advance :: forall a. Storable a => Ring a -> Ptr a -> Ptr a
789−[R] Streamly.Internal.Data.Producer.Type
790−[R] Streamly.Internal.Data.Producer.Source
791−[C] Streamly.Internal.Data.Producer
792− [A] Source
793− [A] unread :: [b] -> Source a b -> Source a b
794− [A] translate :: Functor m => (a -> c) -> (c -> a) -> Producer m c b -> Producer m a b
795− [A] source :: Maybe a -> Source a b
796− [A] producer :: Monad m => Producer m a b -> Producer m (Source a b) b
797− [A] parseManyD :: Monad m => Parser a m b -> Producer m (Source x a) a -> Producer m (Source x a) (Either ParseError b)
798− [A] parseMany :: Monad m => Parser a m b -> Producer m (Source x a) a -> Producer m (Source x a) (Either ParseError b)
799− [A] parse :: Monad m => Parser a m b -> Producer m (Source s a) a -> Source s a -> m (Either ParseError b, Source s a)
800− [A] lmap :: (a -> a) -> Producer m a b -> Producer m a b
801− [A] isEmpty :: Source a b -> Bool
802−[R] Streamly.Internal.Data.Pipe.Type
803−[C] Streamly.Internal.Data.Pipe
804− [A] Step
805− [A] Yield :: a -> s -> Step s a
806− [A] Continue :: s -> Step s a
807− [A] PipeState
808− [A] Produce :: s2 -> PipeState s1 s2
809− [A] Consume :: s1 -> PipeState s1 s2
810− [C] Pipe
811− [A] Pipe :: (s1 -> a -> m (Step (PipeState s1 s2) b)) -> (s2 -> m (Step (PipeState s1 s2) b)) -> s1 -> Pipe m a b
812−[A] Streamly.Internal.Data.ParserK
813− [A] Step
814− [A] Partial :: !Int -> (Input a -> m (Step a m r)) -> Step a m r
815− [A] Error :: !Int -> String -> Step a m r
816− [A] Done :: !Int -> r -> Step a m r
817− [A] Continue :: !Int -> (Input a -> m (Step a m r)) -> Step a m r
818− [A] ParseResult
819− [A] Success :: !Int -> !b -> ParseResult b
820− [A] Failure :: !Int -> !String -> ParseResult b
821− [A] Input
822− [A] None :: Input a
823− [A] Chunk :: a -> Input a
824− [A] ParserK
825− [A] [runParser] :: ParserK a m b -> forall r. (ParseResult b -> Int -> Input a -> m (Step a m r)) -> Int -> Int -> Input a -> m (Step a m r)
826− [A] MkParser :: (forall r. (ParseResult b -> Int -> Input a -> m (Step a m r)) -> Int -> Int -> Input a -> m (Step a m r)) -> ParserK a m b
827− [A] fromPure :: b -> ParserK a m b
828− [A] fromEffect :: Monad m => m b -> ParserK a m b
829− [A] die :: String -> ParserK a m b
830− [A] adaptCG :: Monad m => Parser a m b -> ParserK (Array a) m b
831− [A] adaptC :: (Monad m, Unbox a) => Parser a m b -> ParserK (Array a) m b
832− [A] adapt :: Monad m => Parser a m b -> ParserK a m b
833−[R] Streamly.Internal.Data.Parser.ParserK.Type
834−[R] Streamly.Internal.Data.Parser.ParserD.Type
835−[R] Streamly.Internal.Data.Parser.ParserD.Tee
836−[R] Streamly.Internal.Data.Parser.ParserD
837−[C] Streamly.Internal.Data.Parser
838− [A] Step
839− [A] Partial :: !Int -> !s -> Step s b
840− [A] Error :: !String -> Step s b
841− [A] Done :: !Int -> !b -> Step s b
842− [A] Continue :: !Int -> !s -> Step s b
843− [A] Parser
844− [A] Parser :: (s -> a -> m (Step s b)) -> m (Initial s b) -> (s -> m (Step s b)) -> Parser a m b
845− [A] Initial
846− [A] IPartial :: !s -> Initial s b
847− [A] IError :: !String -> Initial s b
848− [A] IDone :: !b -> Initial s b
849− [A] GHC.Show.Show
850− [A] instance (GHC.Show.Show a, GHC.Show.Show b) => GHC.Show.Show (Streamly.Internal.Data.Parser.Tuple'Fused a b)
851− [A] ParseError
852− [A] ParseError :: String -> ParseError
853− [A] zipWithM :: Monad m => (a -> b -> m c) -> Stream m a -> Fold m c x -> Parser b m x
854− [A] zip :: Monad m => Stream m a -> Fold m (a, b) x -> Parser b m x
855− [A] wordWithQuotes :: (Monad m, Eq a) => Bool -> (a -> a -> Maybe a) -> a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
856− [A] wordProcessQuotes :: (Monad m, Eq a) => a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
857− [A] wordKeepQuotes :: (Monad m, Eq a) => a -> (a -> Maybe a) -> (a -> Bool) -> Fold m a b -> Parser a m b
858− [A] wordFramedBy :: Monad m => (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
859− [A] wordBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
860− [A] toFold :: Monad m => Parser a m b -> Fold m a b
861− [A] takeWhileP :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
862− [A] takeWhile1 :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
863− [A] takeWhile :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
864− [A] takeStartBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
865− [A] takeStartBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
866− [A] takeP :: Monad m => Int -> Parser a m b -> Parser a m b
867− [A] takeGE :: Monad m => Int -> Fold m a b -> Parser a m b
868− [A] takeFramedBy_ :: Monad m => (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
869− [A] takeFramedByGeneric :: Monad m => Maybe (a -> Bool) -> Maybe (a -> Bool) -> Maybe (a -> Bool) -> Fold m a b -> Parser a m b
870− [A] takeFramedByEsc_ :: Monad m => (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
871− [A] takeEndBy_ :: (a -> Bool) -> Parser a m b -> Parser a m b
872− [A] takeEndByEsc :: Monad m => (a -> Bool) -> (a -> Bool) -> Parser a m b -> Parser a m b
873− [A] takeEndBy :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
874− [A] takeEitherSepBy :: (a -> Bool) -> Fold m (Either a b) c -> Parser a m c
875− [A] takeEQ :: Monad m => Int -> Fold m a b -> Parser a m b
876− [A] takeBetween :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
877− [A] subsequenceBy :: (a -> a -> Bool) -> Stream m a -> Parser a m ()
878− [A] streamEqBy :: Monad m => (a -> a -> Bool) -> Stream m a -> Parser a m ()
879− [A] split_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
880− [A] splitWith :: Monad m => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
881− [A] splitSome :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
882− [A] splitManyPost :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
883− [A] splitMany :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
884− [A] spanByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
885− [A] spanBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
886− [A] span :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (b, c)
887− [A] some :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
888− [A] sequence :: Monad m => Stream m (Parser a m b) -> Fold m b c -> Parser a m c
889− [A] sepByAll :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
890− [A] sepBy1 :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
891− [A] sepBy :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
892− [A] satisfy :: Monad m => (a -> Bool) -> Parser a m a
893− [A] sampleFromthen :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
894− [A] roundRobin :: t (Parser a m b) -> Fold m b c -> Parser a m c
895− [A] rmapM :: Monad m => (b -> m c) -> Parser a m b -> Parser a m c
896− [A] retryMaxTotal :: Int -> Parser a m b -> Fold m b c -> Parser a m c
897− [A] retryMaxSuccessive :: Int -> Parser a m b -> Fold m b c -> Parser a m c
898− [A] retry :: Parser a m b -> Parser a m b
899− [A] postscan :: Fold m a b -> Parser b m c -> Parser a m c
900− [A] peek :: Monad m => Parser a m a
901− [A] oneOf :: (Monad m, Eq a, Foldable f) => f a -> Parser a m a
902− [A] oneNotEq :: (Monad m, Eq a) => a -> Parser a m a
903− [A] oneEq :: (Monad m, Eq a) => a -> Parser a m a
904− [A] one :: Monad m => Parser a m a
905− [A] noneOf :: (Monad m, Eq a, Foldable f) => f a -> Parser a m a
906− [A] noErrorUnsafeSplit_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
907− [A] noErrorUnsafeSplitWith :: Monad m => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
908− [A] noErrorUnsafeConcatMap :: Monad m => (b -> Parser a m c) -> Parser a m b -> Parser a m c
909− [A] maybe :: Monad m => (a -> Maybe b) -> Parser a m b
910− [A] manyTillP :: Parser a m b -> Parser a m x -> Parser b m c -> Parser a m c
911− [A] manyTill :: Monad m => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
912− [A] manyThen :: Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
913− [A] manyP :: Parser a m b -> Parser b m c -> Parser a m c
914− [A] many :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
915− [A] makeIndexFilter :: (Fold m (s, a) b -> Parser a m b) -> (((s, a) -> Bool) -> Fold m (s, a) b -> Fold m (s, a) b) -> ((s, a) -> Bool) -> Fold m a b -> Parser a m b
916− [A] lookAhead :: Monad m => Parser a m b -> Parser a m b
917− [A] lmapM :: Monad m => (a -> m b) -> Parser b m r -> Parser a m r
918− [A] lmap :: (a -> b) -> Parser b m r -> Parser a m r
919− [A] listEqBy :: Monad m => (a -> a -> Bool) -> [a] -> Parser a m [a]
920− [A] listEq :: (Monad m, Eq a) => [a] -> Parser a m [a]
921− [A] indexed :: forall m a b. Monad m => Fold m (Int, a) b -> Parser a m b
922− [A] groupByRollingEither :: Monad m => (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
923− [A] groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
924− [A] groupBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
925− [A] fromPure :: Monad m => b -> Parser a m b
926− [A] fromFoldMaybe :: Monad m => String -> Fold m a (Maybe b) -> Parser a m b
927− [A] fromFold :: Monad m => Fold m a b -> Parser a m b
928− [A] fromEffect :: Monad m => m b -> Parser a m b
929− [A] filter :: Monad m => (a -> Bool) -> Parser a m b -> Parser a m b
930− [A] extractStep :: Monad m => (s -> m (Step s1 b)) -> Step s b -> m (Step s1 b)
931− [A] eof :: Monad m => Parser a m ()
932− [A] either :: Monad m => (a -> Either String b) -> Parser a m b
933− [A] dropWhile :: Monad m => (a -> Bool) -> Parser a m ()
934− [A] dieM :: Monad m => m String -> Parser a m b
935− [A] die :: Monad m => String -> Parser a m b
936− [A] deintercalateAll :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
937− [A] deintercalate1 :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
938− [A] deintercalate :: Monad m => Parser a m x -> Parser a m y -> Fold m (Either x y) z -> Parser a m z
939− [A] countBetween :: Int -> Int -> Parser a m b -> Fold m b c -> Parser a m c
940− [A] count :: Int -> Parser a m b -> Fold m b c -> Parser a m c
941− [A] concatMap :: Monad m => (b -> Parser a m c) -> Parser a m b -> Parser a m c
942− [A] blockWithQuotes :: (Monad m, Eq a) => (a -> Bool) -> (a -> Bool) -> a -> a -> Fold m a b -> Parser a m b
943− [A] bimapOverrideCount :: Int -> (s -> s1) -> (b -> b1) -> Step s b -> Step s1 b1
944− [A] alt :: Monad m => Parser x m a -> Parser x m a -> Parser x m a
945−[A] Streamly.Internal.Data.MutByteArray
946− [A] class Unbox a
947− [A] class SizeOfRep (f :: Type -> Type)
948− [A] class Serialize a
949− [A] TypeOfType
950− [A] UnitType :: Name -> TypeOfType
951− [A] TheType :: SimpleDataCon -> TypeOfType
952− [A] MultiType :: [SimpleDataCon] -> TypeOfType
953− [A] SimpleDataCon
954− [A] SimpleDataCon :: Name -> [Field] -> SimpleDataCon
955− [A] SerializeConfig
956− [A] [cfgRecordSyntaxWithHeader] :: SerializeConfig -> Bool
957− [A] [cfgInlineSize] :: SerializeConfig -> Maybe Inline
958− [A] [cfgInlineSerialize] :: SerializeConfig -> Maybe Inline
959− [A] [cfgInlineDeserialize] :: SerializeConfig -> Maybe Inline
960− [A] [cfgConstructorTagAsString] :: SerializeConfig -> Bool
961− [A] SerializeConfig :: Maybe Inline -> Maybe Inline -> Maybe Inline -> Bool -> Bool -> SerializeConfig
962− [A] PinnedState
963− [A] Unpinned :: PinnedState
964− [A] Pinned :: PinnedState
965− [A] MutByteArray
966− [A] MutByteArray :: MutableByteArray# RealWorld -> MutByteArray
967− [A] DataType
968− [A] [dtTvs] :: DataType -> [Name]
969− [A] [dtName] :: DataType -> Name
970− [A] [dtCxt] :: DataType -> Cxt
971− [A] [dtCons] :: DataType -> [DataCon]
972− [A] DataType :: Name -> [Name] -> Cxt -> [DataCon] -> DataType
973− [A] DataCon
974− [A] [dcTvs] :: DataCon -> [Name]
975− [A] [dcName] :: DataCon -> Name
976− [A] [dcFields] :: DataCon -> [(Maybe Name, Type)]
977− [A] [dcCxt] :: DataCon -> Cxt
978− [A] DataCon :: Name -> [Name] -> Cxt -> [(Maybe Name, Type)] -> DataCon
979− [A] BoundedPtr
980− [A] BoundedPtr :: MutByteArray -> Int -> Int -> BoundedPtr
981− [A] Streamly.Internal.Data.Serialize.Type.Serialize
982− [A] instance Streamly.Internal.Data.Serialize.Type.Serialize Streamly.Internal.Data.MutByteArray.LiftedInteger
983− [A] instance Streamly.Internal.Data.Serialize.Type.Serialize GHC.Num.Integer.Integer
984− [A] instance Streamly.Internal.Data.Serialize.Type.Serialize a => Streamly.Internal.Data.Serialize.Type.Serialize (GHC.Maybe.Maybe a)
985− [A] instance Streamly.Internal.Data.Serialize.Type.Serialize (Data.Proxy.Proxy a)
986− [A] instance (Streamly.Internal.Data.Serialize.Type.Serialize a, Streamly.Internal.Data.Serialize.Type.Serialize b) => Streamly.Internal.Data.Serialize.Type.Serialize (Data.Either.Either a b)
987− [A] Peeker
988− [A] Peeker :: Builder BoundedPtr IO a -> Peeker a
989− [A] type Field = (Maybe Name, Type)
990− [A] xorCmp :: [Word8] -> Name -> Name -> Q Exp
991− [A] wListToString :: [Word8] -> String
992− [A] w8_int :: Word8 -> Int
993− [A] w32_int :: Word32 -> Int
994− [A] unpin :: MutByteArray -> IO MutByteArray
995− [A] typeOfType :: Type -> [DataCon] -> TypeOfType
996− [A] skipByte :: Peeker ()
997− [A] sizeOfRep :: SizeOfRep f => f x -> Int
998− [A] sizeOfMutableByteArray :: MutByteArray -> IO Int
999− [A] sizeOf :: (Unbox a, SizeOfRep (Rep a)) => Proxy a -> Int
1000− [A] simplifyDataCon :: DataCon -> SimpleDataCon
1001− [A] serializeW8List :: Name -> Name -> [Word8] -> Q Exp
1002− [A] serializeConfig :: SerializeConfig
1003− [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
1004− [A] runPeeker :: Peeker a -> BoundedPtr -> IO a
1005− [A] reifyDataType :: Name -> Q DataType
1006− [A] readUnsafe :: Unbox a => Peeker a
1007− [A] read :: Unbox a => Peeker a
1008− [A] putSliceUnsafe :: MonadIO m => MutByteArray -> Int -> MutByteArray -> Int -> Int -> m ()
1009− [A] pokeRep :: PokeRep f => f a -> BoundedPtr -> IO BoundedPtr
1010− [A] pokeBoundedPtrUnsafe :: forall a. Unbox a => a -> BoundedPtr -> IO BoundedPtr
1011− [A] pokeBoundedPtr :: forall a. Unbox a => a -> BoundedPtr -> IO BoundedPtr
1012− [A] pokeAt :: (Unbox a, Generic a, PokeRep (Rep a)) => Int -> MutByteArray -> a -> IO ()
1013− [A] pinnedNewAlignedBytes :: Int -> Int -> IO MutByteArray
1014− [A] pinnedNew :: Int -> IO MutByteArray
1015− [A] pinnedCloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
1016− [A] pin :: MutByteArray -> IO MutByteArray
1017− [A] peekRep :: PeekRep f => Peeker (f x)
1018− [A] peekAt :: (Unbox a, Generic a, PeekRep (Rep a)) => Int -> MutByteArray -> IO a
1019− [A] openConstructor :: Name -> Int -> Q Pat
1020− [A] nil :: MutByteArray
1021− [A] newBytesAs :: PinnedState -> Int -> IO MutByteArray
1022− [A] new :: Int -> IO MutByteArray
1023− [A] mkSerializeExprFields :: Name -> [Field] -> Q Exp
1024− [A] mkRecSizeOfExpr :: SimpleDataCon -> Q Exp
1025− [A] mkRecSerializeExpr :: Name -> SimpleDataCon -> Q Exp
1026− [A] mkRecDeserializeExpr :: Name -> Name -> Name -> SimpleDataCon -> Q Exp
1027− [A] mkFieldName :: Int -> Name
1028− [A] mkDeserializeKeysDec :: Name -> Name -> SimpleDataCon -> Q [Dec]
1029− [A] mkDeserializeExprOne :: Name -> SimpleDataCon -> Q Exp
1030− [A] matchConstructor :: Name -> Int -> Q Exp -> Q Match
1031− [A] makeN :: Int -> Name
1032− [A] makeI :: Int -> Name
1033− [A] makeA :: Int -> Name
1034− [A] litProxy :: Unbox a => Proxy a -> Q Exp
1035− [A] litIntegral :: Integral a => a -> Q Exp
1036− [A] isUnitType :: [DataCon] -> Bool
1037− [A] isRecordSyntax :: SimpleDataCon -> Bool
1038− [A] isPinned :: MutByteArray -> Bool
1039− [A] int_w8 :: Int -> Word8
1040− [A] int_w32 :: Int -> Word32
1041− [A] inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
1042− [A] inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
1043− [A] inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
1044− [A] getMutableByteArray# :: MutByteArray -> MutableByteArray# RealWorld
1045− [A] genericSizeOf :: forall a. SizeOfRep (Rep a) => Proxy a -> Int
1046− [A] genericPokeByteIndex :: (Generic a, PokeRep (Rep a)) => MutByteArray -> Int -> a -> IO ()
1047− [A] genericPeekByteIndex :: (Generic a, PeekRep (Rep a)) => MutByteArray -> Int -> IO a
1048− [A] errorUnsupported :: String -> a
1049− [A] errorUnimplemented :: a
1050− [A] encodeRecordFields :: Bool -> SerializeConfig -> SerializeConfig
1051− [A] encodeConstrNames :: Bool -> SerializeConfig -> SerializeConfig
1052− [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
1053− [A] deriveUnbox :: Q [Dec] -> Q [Dec]
1054− [A] deriveSerializeWith :: (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
1055− [A] deriveSerialize :: Q [Dec] -> Q [Dec]
1056− [A] conUpdateFuncDec :: Name -> [Field] -> Q [Dec]
1057− [A] cloneSliceUnsafeAs :: MonadIO m => PinnedState -> Int -> Int -> MutByteArray -> m MutByteArray
1058− [A] cloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
1059− [A] c2w :: Char -> Word8
1060− [A] asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
1061− [A] addSizeTo :: Serialize a => Int -> a -> Int
1062− [A] _x :: Name
1063− [A] _val :: Name
1064− [A] _tag :: Name
1065− [A] _initialOffset :: Name
1066− [A] _endOffset :: Name
1067− [A] _arr :: Name
1068− [A] _acc :: Name
1069−[A] Streamly.Internal.Data.MutArray.Stream
1070− [A] SpliceState
1071− [A] SpliceYielding :: arr -> SpliceState s arr -> SpliceState s arr
1072− [A] SpliceInitial :: s -> SpliceState s arr
1073− [A] SpliceFinish :: SpliceState s arr
1074− [A] SpliceBuffering :: s -> arr -> SpliceState s arr
1075− [A] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
1076− [A] splitOn :: (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> Stream m (MutArray a)
1077− [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1078− [A] packArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1079− [A] lpackArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
1080− [A] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
1081− [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1082− [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1083− [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
1084− [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1085− [A] compactEQ :: Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1086− [A] compact :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
1087− [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1088−[A] Streamly.Internal.Data.MutArray.Generic
1089− [A] MutArray
1090− [A] [arrTrueLen] :: MutArray a -> {-# UNPACK #-} !Int
1091− [A] [arrStart] :: MutArray a -> {-# UNPACK #-} !Int
1092− [A] [arrLen] :: MutArray a -> {-# UNPACK #-} !Int
1093− [A] [arrContents#] :: MutArray a -> MutableArray# RealWorld a
1094− [A] MutArray :: MutableArray# RealWorld a -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> MutArray a
1095− [A] writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
1096− [A] writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
1097− [A] writeN :: MonadIO m => Int -> Fold m a (MutArray a)
1098− [A] write :: MonadIO m => Fold m a (MutArray a)
1099− [A] uninit :: MonadIO m => MutArray a -> Int -> m (MutArray a)
1100− [A] toStreamK :: MonadIO m => MutArray a -> StreamK m a
1101− [A] toList :: MonadIO m => MutArray a -> m [a]
1102− [A] strip :: MonadIO m => (a -> Bool) -> MutArray a -> m (MutArray a)
1103− [A] snocWith :: MonadIO m => (Int -> Int) -> MutArray a -> a -> m (MutArray a)
1104− [A] snocUnsafe :: MonadIO m => MutArray a -> a -> m (MutArray a)
1105− [A] snoc :: MonadIO m => MutArray a -> a -> m (MutArray a)
1106− [A] realloc :: MonadIO m => Int -> MutArray a -> m (MutArray a)
1107− [A] reader :: MonadIO m => Unfold m (MutArray a) a
1108− [A] readRev :: MonadIO m => MutArray a -> Stream m a
1109− [A] read :: MonadIO m => MutArray a -> Stream m a
1110− [A] putSliceUnsafe :: MonadIO m => MutArray a -> Int -> MutArray a -> Int -> Int -> m ()
1111− [A] putIndices :: MonadIO m => MutArray a -> Fold m (Int, a) ()
1112− [A] putIndexUnsafe :: forall m a. MonadIO m => Int -> MutArray a -> a -> m ()
1113− [A] putIndex :: MonadIO m => Int -> MutArray a -> a -> m ()
1114− [A] producerWith :: Monad m => (forall b. IO b -> m b) -> Producer m (MutArray a) a
1115− [A] producer :: MonadIO m => Producer m (MutArray a) a
1116− [A] nil :: MonadIO m => m (MutArray a)
1117− [A] new :: MonadIO m => Int -> m (MutArray a)
1118− [A] modifyIndexUnsafe :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
1119− [A] modifyIndex :: MonadIO m => Int -> MutArray a -> (a -> (a, b)) -> m b
1120− [A] length :: MutArray a -> Int
1121− [A] getSliceUnsafe :: Int -> Int -> MutArray a -> MutArray a
1122− [A] getSlice :: Int -> Int -> MutArray a -> MutArray a
1123− [A] getIndexUnsafeWith :: MonadIO m => MutableArray# RealWorld a -> Int -> m a
1124− [A] getIndexUnsafe :: MonadIO m => Int -> MutArray a -> m a
1125− [A] getIndex :: MonadIO m => Int -> MutArray a -> m (Maybe a)
1126− [A] fromStreamN :: MonadIO m => Int -> Stream m a -> m (MutArray a)
1127− [A] fromStream :: MonadIO m => Stream m a -> m (MutArray a)
1128− [A] fromPureStream :: MonadIO m => Stream Identity a -> m (MutArray a)
1129− [A] fromListN :: MonadIO m => Int -> [a] -> m (MutArray a)
1130− [A] fromList :: MonadIO m => [a] -> m (MutArray a)
1131− [A] eq :: (MonadIO m, Eq a) => MutArray a -> MutArray a -> m Bool
1132− [A] cmp :: (MonadIO m, Ord a) => MutArray a -> MutArray a -> m Ordering
1133− [A] clone :: MonadIO m => MutArray a -> m (MutArray a)
1134− [A] chunksOf :: forall m a. MonadIO m => Int -> Stream m a -> Stream m (MutArray a)
1135−[A] Streamly.Internal.Data.MutArray
1136− [A] MutByteArray
1137− [A] MutArray
1138− [A] [arrStart] :: MutArray a -> {-# UNPACK #-} !Int
1139− [A] [arrEnd] :: MutArray a -> {-# UNPACK #-} !Int
1140− [A] [arrContents] :: MutArray a -> {-# UNPACK #-} !MutByteArray
1141− [A] [arrBound] :: MutArray a -> {-# UNPACK #-} !Int
1142− [A] MutArray :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> MutArray a
1143− [A] IORef
1144− [A] ArrayUnsafe
1145− [A] ArrayUnsafe :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> ArrayUnsafe a
1146− [A] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1147− [A] writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1148− [A] writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1149− [A] writeNWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
1150− [A] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1151− [A] writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1152− [A] writeIORef :: Unbox a => IORef a -> a -> IO ()
1153− [A] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
1154− [A] writeAppendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
1155− [A] writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
1156− [A] writeAppendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
1157− [A] writeAppend :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
1158− [A] write :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1159− [A] unsafeSwapIndices :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> MutArray a -> m ()
1160− [A] unpin :: MutArray a -> IO (MutArray a)
1161− [A] toStreamKWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> StreamK m a
1162− [A] toStreamKRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> StreamK m a
1163− [A] toStreamKRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
1164− [A] toStreamK :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
1165− [A] toStreamDWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
1166− [A] toStreamDRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
1167− [A] toList :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m [a]
1168− [A] swapIndices :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> MutArray a -> m ()
1169− [A] strip :: forall a m. (Unbox a, MonadIO m) => (a -> Bool) -> MutArray a -> m (MutArray a)
1170− [A] splitOn :: (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> Stream m (MutArray a)
1171− [A] splitAt :: forall a. Unbox a => Int -> MutArray a -> (MutArray a, MutArray a)
1172− [A] spliceWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int -> Int) -> MutArray a -> MutArray a -> m (MutArray a)
1173− [A] spliceUnsafe :: MonadIO m => MutArray a -> MutArray a -> m (MutArray a)
1174− [A] spliceExp :: (MonadIO m, Unbox a) => MutArray a -> MutArray a -> m (MutArray a)
1175− [A] spliceCopy :: forall m a. MonadIO m => MutArray a -> MutArray a -> m (MutArray a)
1176− [A] splice :: (MonadIO m, Unbox a) => MutArray a -> MutArray a -> m (MutArray a)
1177− [A] snocWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> MutArray a -> a -> m (MutArray a)
1178− [A] snocUnsafe :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1179− [A] snocMay :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (Maybe (MutArray a))
1180− [A] snocLinear :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1181− [A] snoc :: forall m a. (MonadIO m, Unbox a) => MutArray a -> a -> m (MutArray a)
1182− [A] shuffleBy :: (a -> a -> m Bool) -> MutArray a -> MutArray a -> m ()
1183− [A] roundUpToPower2 :: Int -> Int
1184− [A] rightSize :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m (MutArray a)
1185− [A] reverse :: forall m a. (MonadIO m, Unbox a) => MutArray a -> m ()
1186− [A] resizeExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1187− [A] resize :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1188− [A] realloc :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
1189− [A] readerRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Unfold m (MutArray a) a
1190− [A] readerRev :: forall m a. (MonadIO m, Unbox a) => Unfold m (MutArray a) a
1191− [A] reader :: forall m a. (MonadIO m, Unbox a) => Unfold m (MutArray a) a
1192− [A] readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
1193− [A] readIORef :: Unbox a => IORef a -> IO a
1194− [A] read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Stream m a
1195− [A] putIndices :: forall m a. (MonadIO m, Unbox a) => MutArray a -> Fold m (Int, a) ()
1196− [A] putIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
1197− [A] putIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
1198− [A] producerWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Producer m (MutArray a) a
1199− [A] producer :: forall m a. (MonadIO m, Unbox a) => Producer m (MutArray a) a
1200− [A] pollIntIORef :: (MonadIO m, Unbox a) => IORef a -> Stream m a
1201− [A] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1202− [A] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (MutArray a)
1203− [A] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
1204− [A] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
1205− [A] pinnedNewBytes :: MonadIO m => Int -> m (MutArray a)
1206− [A] pinnedNewAligned :: (MonadIO m, Unbox a) => Int -> Int -> m (MutArray a)
1207− [A] pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
1208− [A] pinnedFromListN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1209− [A] pinnedFromList :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1210− [A] pinnedClone :: MonadIO m => MutArray a -> m (MutArray a)
1211− [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1212− [A] pin :: MutArray a -> IO (MutArray a)
1213− [A] permute :: MutArray a -> m Bool
1214− [A] partitionBy :: forall m a. (MonadIO m, Unbox a) => (a -> Bool) -> MutArray a -> m (MutArray a, MutArray a)
1215− [A] nil :: MutArray a
1216− [A] newIORef :: forall a. Unbox a => a -> IO (IORef a)
1217− [A] newArrayWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int -> m MutByteArray) -> Int -> Int -> m (MutArray a)
1218− [A] new :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
1219− [A] modifyIndices :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (Int -> a -> a) -> Fold m Int ()
1220− [A] modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
1221− [A] modifyIndex :: forall m a b. (MonadIO m, Unbox a) => Int -> MutArray a -> (a -> (a, b)) -> m b
1222− [A] modifyIORef' :: Unbox a => IORef a -> (a -> a) -> IO ()
1223− [A] modify :: forall m a. (MonadIO m, Unbox a) => MutArray a -> (a -> a) -> m ()
1224− [A] mergeBy :: Int -> (MutArray a -> MutArray a -> m ()) -> MutArray a -> m ()
1225− [A] memcpy :: Ptr Word8 -> Ptr Word8 -> Int -> IO ()
1226− [A] memcmp :: Ptr Word8 -> Ptr Word8 -> Int -> IO Bool
1227− [A] length :: forall a. Unbox a => MutArray a -> Int
1228− [A] isPinned :: MutArray a -> Bool
1229− [A] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
1230− [A] getSliceUnsafe :: forall a. Unbox a => Int -> Int -> MutArray a -> MutArray a
1231− [A] getSlice :: forall a. Unbox a => Int -> Int -> MutArray a -> MutArray a
1232− [A] getIndicesD :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
1233− [A] getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1234− [A] getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
1235− [A] getIndexRev :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
1236− [A] getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
1237− [A] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
1238− [A] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
1239− [A] fromStreamD :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
1240− [A] fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
1241− [A] fromPureStream :: (MonadIO m, Unbox a) => Stream Identity a -> m (MutArray a)
1242− [A] fromListRevN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1243− [A] fromListRev :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1244− [A] fromListN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
1245− [A] fromList :: (MonadIO m, Unbox a) => [a] -> m (MutArray a)
1246− [A] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
1247− [A] foldr :: (MonadIO m, Unbox a) => (a -> b -> b) -> b -> MutArray a -> m b
1248− [A] foldl' :: (MonadIO m, Unbox a) => (b -> a -> b) -> b -> MutArray a -> m b
1249− [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1250− [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
1251− [A] divideBy :: Int -> (MutArray a -> m (MutArray a, MutArray a)) -> MutArray a -> m ()
1252− [A] cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
1253− [A] clone :: MonadIO m => MutArray a -> m (MutArray a)
1254− [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (MutArray a)
1255− [A] castUnsafe :: MutArray a -> MutArray b
1256− [A] cast :: forall a b. Unbox b => MutArray a -> Maybe (MutArray b)
1257− [A] c_memchr :: Ptr Word8 -> Word8 -> CSize -> IO (Ptr Word8)
1258− [A] bytesFree :: MutArray a -> Int
1259− [A] byteLength :: MutArray a -> Int
1260− [A] byteCapacity :: MutArray a -> Int
1261− [A] bubble :: (MonadIO m, Unbox a) => (a -> a -> Ordering) -> MutArray a -> m ()
1262− [A] breakOn :: MonadIO m => Word8 -> MutArray Word8 -> m (MutArray Word8, Maybe (MutArray Word8))
1263− [A] blockSize :: Int
1264− [A] asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
1265− [A] asBytes :: MutArray a -> MutArray Word8
1266− [A] arrayChunkBytes :: Int
1267− [A] allocBytesToElemCount :: Unbox a => a -> Int -> Int
1268−[C] Streamly.Internal.Data.IsMap
1269− [A] mapTraverseWithKey :: (IsMap f, Applicative t) => (Key f -> a -> t b) -> f a -> t (f b)
1270−[R] Streamly.Internal.Data.IORef.Unboxed
1271−[R] Streamly.Internal.Data.Fold.Window
1272−[R] Streamly.Internal.Data.Fold.Type
1273−[R] Streamly.Internal.Data.Fold.Tee
1274−[R] Streamly.Internal.Data.Fold.Step
1275−[R] Streamly.Internal.Data.Fold.Container
1276−[D] Streamly.Internal.Data.Fold.Chunked
1277−[C] Streamly.Internal.Data.Fold
1278− [A] ManyState
1279− [C] Fold
1280− [C] Fold
1281− [O] Fold :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> Fold m a b
1282− [N] Fold :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> (s -> m b) -> Fold m a b
1283− [A] windowSumInt :: forall m a. (Monad m, Integral a) => Fold m (a, Maybe a) a
1284− [A] windowSum :: forall m a. (Monad m, Num a) => Fold m (a, Maybe a) a
1285− [A] windowRollingMapM :: Monad m => (Maybe a -> a -> m (Maybe b)) -> Fold m (a, Maybe a) (Maybe b)
1286− [A] windowRollingMap :: Monad m => (Maybe a -> a -> Maybe b) -> Fold m (a, Maybe a) (Maybe b)
1287− [A] windowRange :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe (a, a))
1288− [A] windowPowerSumFrac :: (Monad m, Floating a) => a -> Fold m (a, Maybe a) a
1289− [A] windowPowerSum :: (Monad m, Num a) => Int -> Fold m (a, Maybe a) a
1290− [A] windowMinimum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
1291− [A] windowMean :: forall m a. (Monad m, Fractional a) => Fold m (a, Maybe a) a
1292− [A] windowMaximum :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe a)
1293− [A] windowLmap :: (c -> a) -> Fold m (a, Maybe a) b -> Fold m (c, Maybe c) b
1294− [A] windowLength :: (Monad m, Num b) => Fold m (a, Maybe a) b
1295− [A] toSet :: (Monad m, Ord a) => Fold m a (Set a)
1296− [A] toMapIO :: (MonadIO m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (Map k b)
1297− [A] toMap :: (Monad m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (Map k b)
1298− [A] toIntSet :: Monad m => Fold m Int IntSet
1299− [A] toContainerIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (f b)
1300− [A] toContainer :: (Monad m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (f b)
1301− [A] nubInt :: Monad m => Fold m Int (Maybe Int)
1302− [A] nub :: (Monad m, Ord a) => Fold m a (Maybe a)
1303− [A] mapMStep :: Applicative m => (a -> m b) -> Step s a -> m (Step s b)
1304− [A] kvToMap :: (Monad m, Ord k) => Fold m a b -> Fold m (k, a) (Map k b)
1305− [A] frequency :: (Monad m, Ord a) => Fold m a (Map a Int)
1306− [A] demuxToMapIO :: (MonadIO m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (Map k b)
1307− [A] demuxToMap :: (Monad m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (Map k b)
1308− [A] demuxToContainerIO :: (MonadIO m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
1309− [A] demuxToContainer :: (Monad m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
1310− [A] demuxKvToMap :: (Monad m, Ord k) => (k -> m (Fold m a b)) -> Fold m (k, a) (Map k b)
1311− [A] demuxKvToContainer :: (Monad m, IsMap f, Traversable f) => (Key f -> m (Fold m a b)) -> Fold m (Key f, a) (f b)
1312− [A] demuxIO :: (MonadIO m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (m (Map k b), Maybe (k, b))
1313− [A] demuxGenericIO :: (MonadIO m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (m (f b), Maybe (Key f, b))
1314− [A] demuxGeneric :: (Monad m, IsMap f, Traversable f) => (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (m (f b), Maybe (Key f, b))
1315− [A] demux :: (Monad m, Ord k) => (a -> k) -> (a -> m (Fold m a b)) -> Fold m a (m (Map k b), Maybe (k, b))
1316− [A] cumulative :: Fold m (a, Maybe a) b -> Fold m a b
1317− [A] countDistinctInt :: Monad m => Fold m Int Int
1318− [A] countDistinct :: (Monad m, Ord a) => Fold m a Int
1319− [A] classifyIO :: (MonadIO m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
1320− [A] classifyGenericIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
1321− [A] classifyGeneric :: (Monad m, IsMap f, Traversable f, Ord (Key f)) => (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
1322− [A] classify :: (Monad m, Ord k) => (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
1323− [A] chainStepM :: Applicative m => (s1 -> m s2) -> (a -> m (Step s2 b)) -> Step s1 a -> m (Step s2 b)
1324−[C] Streamly.Internal.Data.Builder
1325− [C] Builder
1326− [C] Builder
1327− [O] Builder :: (s -> m (s, a)) -> Builder s m a
1328− [N] Builder :: (s -> m (a, s)) -> Builder s m a
1329−[A] Streamly.Internal.Data.Binary.Stream
1330− [A] class ToBytes a
1331− [A] word8 :: Applicative m => Word8 -> Stream m Word8
1332− [A] word64le :: Monad m => Word64 -> Stream m Word8
1333− [A] word64host :: Monad m => Word64 -> Stream m Word8
1334− [A] word64be :: Monad m => Word64 -> Stream m Word8
1335− [A] word32le :: Monad m => Word32 -> Stream m Word8
1336− [A] word32be :: Monad m => Word32 -> Stream m Word8
1337− [A] word16le :: Monad m => Word16 -> Stream m Word8
1338− [A] word16be :: Monad m => Word16 -> Stream m Word8
1339− [A] unit :: Applicative m => Stream m Word8
1340− [A] toBytes :: ToBytes a => a -> Stream m Word8
1341− [A] ordering :: Applicative m => Ordering -> Stream m Word8
1342− [A] int8 :: Applicative m => Int8 -> Stream m Word8
1343− [A] int64le :: Monad m => Int64 -> Stream m Word8
1344− [A] int64be :: Monad m => Int64 -> Stream m Word8
1345− [A] int32le :: Monad m => Int32 -> Stream m Word8
1346− [A] int32be :: Monad m => Int32 -> Stream m Word8
1347− [A] int16le :: Monad m => Int16 -> Stream m Word8
1348− [A] int16be :: Monad m => Int16 -> Stream m Word8
1349− [A] float32le :: Monad m => Float -> Stream m Word8
1350− [A] float32be :: Monad m => Float -> Stream m Word8
1351− [A] double64le :: Monad m => Double -> Stream m Word8
1352− [A] double64be :: Monad m => Double -> Stream m Word8
1353− [A] charUtf8 :: Monad m => Char -> Stream m Word8
1354− [A] charLatin1 :: Applicative m => Char -> Stream m Word8
1355− [A] bool :: Applicative m => Bool -> Stream m Word8
1356−[A] Streamly.Internal.Data.Binary.Parser
1357− [A] class FromBytes a
1358− [A] word8 :: Monad m => Parser Word8 m Word8
1359− [A] word64le :: Monad m => Parser Word8 m Word64
1360− [A] word64host :: MonadIO m => Parser Word8 m Word64
1361− [A] word64be :: Monad m => Parser Word8 m Word64
1362− [A] word32le :: Monad m => Parser Word8 m Word32
1363− [A] word32be :: Monad m => Parser Word8 m Word32
1364− [A] word16le :: Monad m => Parser Word8 m Word16
1365− [A] word16be :: Monad m => Parser Word8 m Word16
1366− [A] unit :: Monad m => Parser Word8 m ()
1367− [A] ordering :: Monad m => Parser Word8 m Ordering
1368− [A] int8 :: Monad m => Parser Word8 m Int8
1369− [A] int64le :: Monad m => Parser Word8 m Int64
1370− [A] int64be :: Monad m => Parser Word8 m Int64
1371− [A] int32le :: Monad m => Parser Word8 m Int32
1372− [A] int32be :: Monad m => Parser Word8 m Int32
1373− [A] int16le :: Monad m => Parser Word8 m Int16
1374− [A] int16be :: Monad m => Parser Word8 m Int16
1375− [A] fromBytes :: FromBytes a => Parser Word8 m a
1376− [A] float32le :: MonadIO m => Parser Word8 m Float
1377− [A] float32be :: MonadIO m => Parser Word8 m Float
1378− [A] eqWord8 :: Monad m => Word8 -> Parser Word8 m Word8
1379− [A] double64le :: MonadIO m => Parser Word8 m Double
1380− [A] double64be :: MonadIO m => Parser Word8 m Double
1381− [A] charLatin1 :: Monad m => Parser Word8 m Char
1382− [A] bool :: Monad m => Parser Word8 m Bool
1383−[R] Streamly.Internal.Data.Array.Type
1384−[A] Streamly.Internal.Data.Array.Stream
1385− [A] unlines :: forall m a. (MonadIO m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1386− [A] toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
1387− [A] splitOnSuffix :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
1388− [A] splitOn :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
1389− [A] runArrayParserDBreak :: forall m a b. (MonadIO m, Unbox a) => Parser (Array a) m b -> Stream m (Array a) -> m (Either ParseError b, Stream m (Array a))
1390− [A] runArrayFoldMany :: (Monad m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> StreamK m (Either ParseError b)
1391− [A] runArrayFoldBreak :: (MonadIO m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1392− [A] runArrayFold :: (MonadIO m, Unbox a) => ChunkFold m a b -> StreamK m (Array a) -> m (Either ParseError b)
1393− [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1394− [A] parseChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b)
1395− [A] parseBreakChunks :: (Monad m, Unbox a) => ParserK (Array a) m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1396− [A] parseBreak :: (MonadIO m, Unbox a) => Parser a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
1397− [A] lpackArraysChunksOf :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
1398− [A] interposeSuffix :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1399− [A] interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
1400− [A] intercalateSuffix :: (Monad m, Unbox a) => Array a -> Stream m (Array a) -> Stream m a
1401− [A] foldBreakD :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
1402− [A] foldBreak :: (MonadIO m, Unbox a) => Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
1403− [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1404− [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1405− [A] concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
1406− [A] concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
1407− [A] compact :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
1408− [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1409− [A] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
1410−[R] Streamly.Internal.Data.Array.Mut.Type
1411−[R] Streamly.Internal.Data.Array.Mut.Stream
1412−[R] Streamly.Internal.Data.Array.Mut
1413−[R] Streamly.Internal.Data.Array.Generic.Mut.Type
1414−[C] Streamly.Internal.Data.Array.Generic
1415− [A] getIndex :: Int -> Array a -> Maybe a
1416− [A] fromPureStream :: Stream Identity a -> Array a
1417− [A] fromByteStr# :: Addr# -> Array Word8
1418− [A] chunksOf :: forall m a. MonadIO m => Int -> Stream m a -> Stream m (Array a)
1419−[C] Streamly.Internal.Data.Array
1420− [A] ArrayUnsafe
1421− [A] ArrayUnsafe :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> ArrayUnsafe a
1422− [C] Array
1423− [A] [arrStart] :: Array a -> {-# UNPACK #-} !Int
1424− [A] [arrEnd] :: Array a -> {-# UNPACK #-} !Int
1425− [A] [arrContents] :: Array a -> {-# UNPACK #-} !MutByteArray
1426− [A] Array :: {-# UNPACK #-} !MutByteArray -> {-# UNPACK #-} !Int -> {-# UNPACK #-} !Int -> Array a
1427− [A] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1428− [A] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1429− [R] writeNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
1430− [A] unsafeMakePure :: Monad m => Fold IO a b -> Fold m a b
1431− [A] unsafeIndexIO :: forall a. Unbox a => Int -> Array a -> IO a
1432− [D] unsafeIndex :: forall a. Unbox a => Int -> Array a -> a
1433− [A] unsafeFreezeWithShrink :: Unbox a => MutArray a -> Array a
1434− [A] unpin :: Array a -> IO (Array a)
1435− [A] toStreamKRev :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a
1436− [A] toStreamK :: forall m a. (Monad m, Unbox a) => Array a -> StreamK m a
1437− [A] toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
1438− [A] toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
1439− [A] splitAt :: Unbox a => Int -> Array a -> (Array a, Array a)
1440− [A] splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
1441− [A] serialize :: Serialize a => a -> Array Word8
1442− [A] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1443− [A] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
1444− [A] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
1445− [A] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
1446− [A] pinnedSerialize :: Serialize a => a -> Array Word8
1447− [A] pinnedFromListN :: Unbox a => Int -> [a] -> Array a
1448− [A] pinnedFromList :: Unbox a => [a] -> Array a
1449− [A] pinnedClone :: MonadIO m => Array a -> m (Array a)
1450− [A] pinnedChunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1451− [A] pin :: Array a -> IO (Array a)
1452− [A] nil :: Array a
1453− [A] isPinned :: Array a -> Bool
1454− [A] getIndexUnsafe :: forall a. Unbox a => Int -> Array a -> a
1455− [A] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
1456− [A] fromStreamD :: forall m a. (MonadIO m, Unbox a) => Stream m a -> m (Array a)
1457− [A] fromPureStream :: Unbox a => Stream Identity a -> Array a
1458− [A] fromListRevN :: Unbox a => Int -> [a] -> Array a
1459− [A] fromListRev :: Unbox a => [a] -> Array a
1460− [A] fromByteStr# :: Addr# -> Array Word8
1461− [A] foldr :: Unbox a => (a -> b -> b) -> b -> Array a -> b
1462− [A] foldl' :: forall a b. Unbox a => (b -> a -> b) -> b -> Array a -> b
1463− [A] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1464− [A] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
1465− [A] encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
1466− [A] deserialize :: Serialize a => Array Word8 -> a
1467− [A] clone :: MonadIO m => Array a -> m (Array a)
1468− [A] chunksOf :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> Stream m (Array a)
1469− [A] byteLength :: Array a -> Int
1470− [A] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
1471− [A] breakOn :: MonadIO m => Word8 -> Array Word8 -> m (Array Word8, Maybe (Array Word8))
core/docs/ApiChangelogs/0.2.0-0.2.2.txtdeleted−306
@@ -1,306 +0,0 @@
1−---------------------------------
2−API Annotations
3−---------------------------------
4−
5−[A] : Added
6−[R] : Removed
7−[C] : Changed
8−[O] : Old definition
9−[N] : New definition
10−[D] : Deprecated
11−
12−---------------------------------
13−API diff
14−---------------------------------
15−
16−[C] Streamly.Data.Stream
17− [A] (FixityR,5)
18− [A] (FixityR,5)
19−[C] Streamly.Data.MutArray.Generic
20− [A] emptyOf :: MonadIO m => Int -> m (MutArray a)
21− [A] createOf :: MonadIO m => Int -> Fold m a (MutArray a)
22− [A] create :: MonadIO m => Fold m a (MutArray a)
23−[C] Streamly.Data.MutArray
24− [A] pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
25− [A] emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
26− [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
27− [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
28− [A] appendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
29− [A] append :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
30−[C] Streamly.Data.Array.Generic
31− [A] createOf :: MonadIO m => Int -> Fold m a (Array a)
32− [A] create :: MonadIO m => Fold m a (Array a)
33−[C] Streamly.Data.Array
34− [A] class Serialize a
35− [A] serializeAt :: Serialize a => Int -> MutByteArray -> a -> IO Int
36− [A] pinnedSerialize :: Serialize a => a -> Array Word8
37− [A] deserializeAt :: Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
38− [A] deserialize :: Serialize a => Array Word8 -> a
39− [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
40− [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
41− [A] addSizeTo :: Serialize a => Int -> a -> Int
42−
43−---------------------------------
44−Internal API diff
45−---------------------------------
46−
47−[C] Streamly.Internal.Unicode.Stream
48− [A] encodeUtf16le' :: Stream m Char -> Stream m Word16
49− [A] decodeUtf16le' :: Stream m Word16 -> Stream m Char
50−[A] Streamly.Internal.FileSystem.Path
51− [A] class IsPath a
52− [A] Rel
53− [A] File
54− [A] Dir
55− [A] Abs
56− [A] Streamly.Internal.FileSystem.Path.IsPath
57− [A] instance Streamly.Internal.FileSystem.Path.IsPath Streamly.Internal.FileSystem.Path.Path
58− [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel Streamly.Internal.FileSystem.Path.Path)
59− [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path))
60− [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Rel (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path))
61− [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path)
62− [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path)
63− [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs Streamly.Internal.FileSystem.Path.Path)
64− [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs (Streamly.Internal.FileSystem.Path.File Streamly.Internal.FileSystem.Path.Path))
65− [A] instance Streamly.Internal.FileSystem.Path.IsPath (Streamly.Internal.FileSystem.Path.Abs (Streamly.Internal.FileSystem.Path.Dir Streamly.Internal.FileSystem.Path.Path))
66− [A] GHC.Show.Show
67− [A] instance GHC.Show.Show Streamly.Internal.FileSystem.Path.PathException
68− [A] GHC.Exception.Type.Exception
69− [A] instance GHC.Exception.Type.Exception Streamly.Internal.FileSystem.Path.PathException
70− [A] GHC.Classes.Eq
71− [A] instance GHC.Classes.Eq Streamly.Internal.FileSystem.Path.PathException
72− [A] Path
73− [A] Path :: Array Word8 -> Path
74− [A] toString :: Path -> [Char]
75− [A] toPath :: IsPath a => a -> Path
76− [A] toChunk :: Path -> Array Word8
77− [A] toChars :: Monad m => Path -> Stream m Char
78− [A] relfile :: QuasiQuoter
79− [A] reldir :: QuasiQuoter
80− [A] rel :: QuasiQuoter
81− [A] primarySeparator :: Char
82− [A] path :: QuasiQuoter
83− [A] mkRelFile :: String -> Q Exp
84− [A] mkRelDir :: String -> Q Exp
85− [A] mkRel :: String -> Q Exp
86− [A] mkPath :: String -> Q Exp
87− [A] mkFile :: String -> Q Exp
88− [A] mkDir :: String -> Q Exp
89− [A] mkAbsFile :: String -> Q Exp
90− [A] mkAbsDir :: String -> Q Exp
91− [A] mkAbs :: String -> Q Exp
92− [A] isSeparator :: Char -> Bool
93− [A] fromString :: MonadThrow m => [Char] -> m Path
94− [A] fromPathUnsafe :: IsPath a => Path -> a
95− [A] fromPath :: (IsPath a, MonadThrow m) => Path -> m a
96− [A] fromChunkUnsafe :: Array Word8 -> Path
97− [A] fromChunk :: MonadThrow m => Array Word8 -> m Path
98− [A] fromChars :: MonadThrow m => Stream Identity Char -> m Path
99− [A] file :: QuasiQuoter
100− [A] extendPath :: Path -> Path -> Path
101− [A] extendDir :: (IsPath (a (Dir Path)), IsPath b, IsPath (a b)) => a (Dir Path) -> Rel b -> a b
102− [A] dir :: QuasiQuoter
103− [A] adaptPath :: (MonadThrow m, IsPath a, IsPath b) => a -> m b
104− [A] absfile :: QuasiQuoter
105− [A] absdir :: QuasiQuoter
106− [A] abs :: QuasiQuoter
107−[C] Streamly.Internal.Data.Stream
108− [A] (FixityR,5)
109− [A] (FixityR,5)
110− [C] splitInnerBySuffix
111− [O] splitInnerBySuffix :: (Monad m, Eq (f a), Monoid (f a)) => (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
112− [N] splitInnerBySuffix :: Monad m => (f a -> Bool) -> (f a -> m (f a, Maybe (f a))) -> (f a -> f a -> m (f a)) -> Stream m (f a) -> Stream m (f a)
113− [D] sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
114− [A] indexOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
115−[C] Streamly.Internal.Data.MutByteArray
116− [A] unsafePinnedAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
117− [A] unsafeAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
118− [D] nil :: MutByteArray
119− [A] empty :: MutByteArray
120− [D] asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
121−[D] Streamly.Internal.Data.MutArray.Stream
122− [D] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
123− [D] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
124− [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
125− [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
126−[C] Streamly.Internal.Data.MutArray.Generic
127− [D] writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
128− [D] writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
129− [A] unsafeCreateOf :: MonadIO m => Int -> Fold m a (MutArray a)
130− [A] emptyOf :: MonadIO m => Int -> m (MutArray a)
131− [A] createWith :: MonadIO m => Int -> Fold m a (MutArray a)
132− [A] createOf :: MonadIO m => Int -> Fold m a (MutArray a)
133− [A] create :: MonadIO m => Fold m a (MutArray a)
134−[C] Streamly.Internal.Data.MutArray
135− [A] SpliceState
136− [A] SpliceYielding :: arr -> SpliceState s arr -> SpliceState s arr
137− [A] SpliceInitial :: s -> SpliceState s arr
138− [A] SpliceFinish :: SpliceState s arr
139− [A] SpliceBuffering :: s -> arr -> SpliceState s arr
140− [R] MutByteArray
141− [D] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
142− [D] writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
143− [D] writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
144− [D] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
145− [D] writeChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
146− [D] writeAppendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
147− [D] writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
148− [A] unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
149− [A] unsafePinnedAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
150− [A] unsafeCreateOfWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
151− [A] unsafeCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
152− [A] unsafeAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
153− [A] unsafeAppendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
154− [A] toStreamWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
155− [A] toStreamRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
156− [R] toStreamDWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
157− [R] toStreamDRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> MutArray a -> Stream m a
158− [A] slicerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
159− [A] sliceIndexerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
160− [A] revCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
161− [D] resizeExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
162− [D] resize :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
163− [A] pokeSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
164− [A] pokeAppendMay :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> a -> m (Maybe (MutArray Word8))
165− [A] pokeAppend :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> a -> m (MutArray Word8)
166− [D] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
167− [D] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
168− [D] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
169− [D] pinnedNewBytes :: MonadIO m => Int -> m (MutArray a)
170− [A] pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
171− [A] pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
172− [A] pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
173− [A] pinnedCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
174− [A] peekUnconsUnsafe :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> m (a, MutArray Word8)
175− [A] peekUncons :: forall m a. (MonadIO m, Unbox a) => MutArray Word8 -> m (Maybe a, MutArray Word8)
176− [A] peekSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
177− [A] pPinnedCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Parser (MutArray a) m (MutArray a)
178− [A] pCompactLE :: forall m a. (MonadIO m, Unbox a) => Int -> Parser (MutArray a) m (MutArray a)
179− [D] nil :: MutArray a
180− [A] lPinnedCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
181− [A] lCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
182− [A] indexReaderWith :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
183− [A] indexReader :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
184− [A] growExp :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
185− [A] grow :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
186− [D] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (MutArray a)
187− [R] getIndicesD :: (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m Int -> Unfold m (MutArray a) a
188− [D] getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
189− [D] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (MutArray a) (Int, Int)
190− [A] fromStreamN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
191− [D] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (MutArray a)
192− [D] fromStreamD :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
193− [A] fromPureStreamN :: (MonadIO m, Unbox a) => Int -> Stream Identity a -> m (MutArray a)
194− [A] fromPtrN :: MonadIO m => Int -> Ptr Word8 -> m (MutArray Word8)
195− [A] fromChunksRealloced :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> m (MutArray a)
196− [A] fromChunksK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
197− [A] fromByteStr# :: MonadIO m => Addr# -> m (MutArray Word8)
198− [D] fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (MutArray a) -> m (MutArray a)
199− [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
200− [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
201− [A] fPinnedCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) (MutArray a)
202− [A] fCompactGE :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m (MutArray a) (MutArray a)
203− [A] emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
204− [A] empty :: MutArray a
205− [A] createWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
206− [A] createOfWith :: forall m a. (MonadIO m, Unbox a) => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
207− [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
208− [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
209− [A] concatWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m (MutArray a) -> Stream m a
210− [A] concatRevWith :: forall m a. (Monad m, Unbox a) => (forall b. IO b -> m b) -> Stream m (MutArray a) -> Stream m a
211− [A] concatRev :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
212− [A] concat :: forall m a. (MonadIO m, Unbox a) => Stream m (MutArray a) -> Stream m a
213− [A] compactOnByteSuffix :: MonadIO m => Word8 -> Stream m (MutArray Word8) -> Stream m (MutArray Word8)
214− [A] compactOnByte :: MonadIO m => Word8 -> Stream m (MutArray Word8) -> Stream m (MutArray Word8)
215− [A] compactLeAs :: forall m a. (MonadIO m, Unbox a) => PinnedState -> Int -> Stream m (MutArray a) -> Stream m (MutArray a)
216− [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
217− [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
218− [A] compactEQ :: Int -> Stream m (MutArray a) -> Stream m (MutArray a)
219− [D] cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
220− [A] byteEq :: MonadIO m => MutArray a -> MutArray a -> m Bool
221− [A] byteCmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
222− [A] buildChunks :: (MonadIO m, Unbox a) => Int -> Fold m a (StreamK n (MutArray a))
223− [D] asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
224− [A] appendWith :: forall m a. (MonadIO m, Unbox a) => (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
225− [A] appendN :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a) -> Fold m a (MutArray a)
226− [A] append :: forall m a. (MonadIO m, Unbox a) => m (MutArray a) -> Fold m a (MutArray a)
227−[D] Streamly.Internal.Data.Array.Stream
228− [C] interposeSuffix
229− [O] interposeSuffix :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
230− [N] interposeSuffix :: forall m a. (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
231− [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
232− [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
233− [C] concatRev
234− [O] concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
235− [N] concatRev :: forall m a. (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
236− [D] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
237−[C] Streamly.Internal.Data.Array.Generic
238− [A] createOf :: MonadIO m => Int -> Fold m a (Array a)
239− [A] create :: MonadIO m => Fold m a (Array a)
240−[C] Streamly.Internal.Data.Array
241− [R] ArrayUnsafe
242− [D] writeWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
243− [D] writeNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
244− [A] unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
245− [A] unsafePinnedAsPtr :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
246− [A] unsafeCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
247− [D] toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
248− [D] toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> Stream m a
249− [C] splice
250− [O] splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
251− [N] splice :: MonadIO m => Array a -> Array a -> m (Array a)
252− [A] slicerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Array a)
253− [A] sliceIndexerFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Int, Int)
254− [D] pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
255− [D] pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a) => Int -> Int -> Fold m a (Array a)
256− [D] pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
257− [D] pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
258− [A] pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
259− [A] pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
260− [A] pinnedCompactLE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
261− [A] parseBreakChunksK :: forall m a b. (MonadIO m, Unbox a) => Parser a m b -> StreamK m (Array a) -> m (Either ParseError b, StreamK m (Array a))
262− [D] nil :: Array a
263− [A] lPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
264− [A] lCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) () -> Fold m (Array a) ()
265− [A] interposeSuffix :: forall m a. (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
266− [A] interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
267− [A] intercalateSuffix :: (Monad m, Unbox a) => Array a -> Stream m (Array a) -> Stream m a
268− [A] indexReaderFromThenTo :: Unfold m (Int, Int, Int, Array a) a
269− [A] indexReader :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
270− [A] indexFinder :: (a -> Bool) -> Unfold Identity (Array a) Int
271− [D] getSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Array a)
272− [R] getIndicesFromThenTo :: Unfold m (Int, Int, Int, Array a) a
273− [D] getIndices :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
274− [D] genSlicesFromLen :: forall m a. (Monad m, Unbox a) => Int -> Int -> Unfold m (Array a) (Int, Int)
275− [D] fromStreamDN :: forall m a. (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
276− [D] fromStreamD :: forall m a. (MonadIO m, Unbox a) => Stream m a -> m (Array a)
277− [A] fromPureStreamN :: Unbox a => Int -> Stream Identity a -> Array a
278− [A] fromPtrN :: Int -> Ptr Word8 -> Array Word8
279− [A] fromChunksK :: (MonadIO m, Unbox a) => StreamK m (Array a) -> m (Array a)
280− [A] fromChunks :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
281− [A] fromByteStr :: Ptr Word8 -> Array Word8
282− [A] foldChunks :: (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m b
283− [A] foldBreakChunksK :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
284− [A] foldBreakChunks :: forall m a b. (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
285− [D] flattenArraysRev :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
286− [D] flattenArrays :: forall m a. (MonadIO m, Unbox a) => Stream m (Array a) -> Stream m a
287− [C] findIndicesOf
288− [O] findIndicesOf :: (a -> Bool) -> Unfold Identity (Array a) Int
289− [N] findIndicesOf :: (a -> Bool) -> Array a -> Stream Identity Int
290− [A] fPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
291− [A] fCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
292− [A] empty :: Array a
293− [A] createWith :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
294− [A] createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
295− [A] create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
296− [A] concatRev :: forall m a. (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
297− [A] concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
298− [A] compactOnByteSuffix :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
299− [A] compactOnByte :: MonadIO m => Word8 -> Stream m (Array Word8) -> Stream m (Array Word8)
300− [A] compactLE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
301− [A] compactGE :: (MonadIO m, Unbox a) => Int -> Stream m (Array a) -> Stream m (Array a)
302− [A] byteEq :: Array a -> Array a -> Bool
303− [A] byteCmp :: Array a -> Array a -> Ordering
304− [A] buildChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
305− [D] bufferChunks :: (MonadIO m, Unbox a) => Stream m a -> m (StreamK m (Array a))
306− [D] asPtrUnsafe :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
core/docs/Changelog.md+16−58
@@ -1,73 +1,31 @@
11# Changelog
22
3−## 0.2.2 (Jan 2024)
3+## Unreleased
44
5−* Add fixities `infixr 5` for `cons` and `consM` functions.
6−* Fix a bug in Array `Eq` instance when the type is a sum type with
7− differently sized constructors.
8−* lpackArraysChunksOf, compact, writeChunksWith, putChunksWith now take the
9− buffer size in number of array elements instead of bytes.
5+* Arrays are now created unpinned by default, they were created pinned
6+ earlier. During IO operations unpinned arrays are automatically copied
7+ to pinned memory. When arrays are directly passed to IO operations
8+ programmers can choose to create them pinned to avoid a copy. To
9+ create pinned arrays, use the internal APIs with the `pinned*` prefix.
1010
11−## 0.2.1 (Dec 2023)
11+* Changed the signature of 'Streamly.Data.Stream.handle' to make the exception
12+ handler monadic.
13+* Rethrow the exception promptly in bracketIO.
1214
13−* Make the serialization of the unit constructor deterministic.
14−* Expose `pinnedSerialize` & `deserialize` via `Streamly.Data.Array`.
15+### Breaking
1516
16−## 0.2.0 (Nov 2023)
17−
18−See [0.1.0-0.2.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0-0.2.0.txt)
19−for a full list of API changes in this release. Only a few significant
20−changes are mentioned here.
21−
22−### Breaking Changes
23−
24−* `ParserK` in `Streamly.Data.ParserK` is not implicitly specialized
25− to arrays anymore. To adapt to the new code, change `ParserK a m
26− b` to `ParserK (Array a) m b` where the `Array` type comes from
27− `Streamly.Data.Array`. This change also affected the signatures of
28− `parseChunks` and `parseBreakChunks`.
29−* Changed the signature of 'Streamly.Data.Stream.handle' to make the
30− exception handler monadic.
31−* Behavior change: Exceptions are now rethrown promptly in `bracketIO`.
32−
33−### Enhancements
34−
35−* __Serialization__: Added a `Streamly.Data.MutByteArray` module with a
36− `Serialize` type class for fast binary serialization. The Data.Array
37− module supplies the `serialize` and `deserialize` operations for arrays.
38−* __Unpinned Arrays__: Unboxed arrays are now created unpinned by default,
39− they were created pinned earlier. During IO operations, unpinned arrays
40− are automatically copied to pinned memory. When arrays are directly
41− passed to IO operations programmers can choose to create them pinned to
42− avoid a copy. To create pinned arrays, use the internal APIs with the
43− `pinned*` prefix.
44−* StreamK now supports native exception handling routines (handle, bracketIO).
45− Earlier we had to convert it to the `Stream` type for exception handling.
17+* `ParserK` in `Streamly.Data.ParserK` is not specialized to arrays anymore. To
18+ adapt to the new code, change `ParserK a m b` to `ParserK (Array a) m b` where
19+ `Array` comes from `Streamly.Data.Array`.
4620
4721### Deprecations
4822
49−See [0.1.0-0.2.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0-0.2.0.txt)
50−for a full list of deprecations.
51−
52−### Internal API Changes
53−
54−* Fold constructor has changed, added a `final` field to support
55− finalization and cleanup of a chain of folds. The `extract` field is
56− now used only for mapping the fold internal state to fold result for
57− scanning purposes. If your fold does not require cleanup you can just use
58− your existing `extract` function as `final` as well to adapt to this change.
59−* Many low level internal modules have been removed, they are entirely
60− exported from higher level internal modules. If you were importing any
61− of the missing low level modules then import the higher level modules instead.
62−* Internal module changes:
63− * Streamly.Internal.Serialize.FromBytes -> Streamly.Internal.Data.Binary.Parser
64− * Streamly.Internal.Serialize.ToBytes -> Streamly.Internal.Data.Binary.Stream
65− * Streamly.Internal.Data.Unbox is now exported via Streamly.Internal.Data.Serialize
66− * Streamly.Internal.Data.IORef.Unboxed is now exported via Streamly.Internal.Data.Serialize
23+* `Streamly.Data.MutArray.newUnpinned` is renamed to
24+ `Streamly.Data.MutArray.pinnedNew`
6725
6826## 0.1.0 (March 2023)
6927
70−Also see [streamly-core-0.1.0 API Changelog](https://github.com/composewell/streamly/blob/streamly-0.10.0/core/docs/ApiChangelogs/0.1.0.txt) or
28+Also see [streamly-core-0.1.0 API Changelog](/core/docs/ApiChangelogs/0.1.0.txt) or
7129https://hackage.haskell.org/package/streamly-core-0.1.0/docs/docs/ApiChangelogs/0.1.0.txt
7230
7331`streamly` package is split into two packages, (1) `streamly-core` that
core/src/DocTestDataArray.hs−3
@@ -11,12 +11,9 @@
1111
1212>>> import qualified Streamly.Data.Array as Array
1313>>> import qualified Streamly.Data.Fold as Fold
14−>>> import qualified Streamly.Data.ParserK as ParserK
1514>>> import qualified Streamly.Data.Stream as Stream
16−>>> import qualified Streamly.Data.StreamK as StreamK
1715
1816For APIs that have not been released yet.
1917
2018>>> import qualified Streamly.Internal.Data.Array as Array
21−>>> import qualified Streamly.Internal.Data.Stream as Stream
2219-}
core/src/DocTestDataParserK.hsdeleted−15
@@ -1,15 +0,0 @@
1−{- $setup
2−>>> :m
3−>>> import Control.Applicative ((<|>))
4−>>> import Data.Char (isDigit, isAlpha)
5−
6−>>> import Streamly.Data.Parser (Parser)
7−>>> import Streamly.Data.ParserK (ParserK)
8−
9−>>> import qualified Streamly.Data.Parser as Parser
10−>>> import qualified Streamly.Data.ParserK as ParserK
11−
12−For APIs that have not been released yet.
13−
14−>>> import qualified Streamly.Internal.Data.ParserK as ParserK
15−-}
core/src/DocTestDataStream.hs−2
@@ -6,7 +6,6 @@
66>>> import Control.Monad.IO.Class (MonadIO (liftIO))
77>>> import Control.Monad.Trans.Class (lift)
88>>> import Control.Monad.Trans.Identity (runIdentityT)
9−>>> import Data.Char (isSpace)
109>>> import Data.Either (fromLeft, fromRight, isLeft, isRight, either)
1110>>> import Data.Maybe (fromJust, isJust)
1211>>> import Data.Function (fix, (&))
@@ -33,7 +32,6 @@ For APIs that have not been released yet.
3332>>> import qualified Streamly.Internal.Data.Fold as Fold
3433>>> import qualified Streamly.Internal.Data.Parser as Parser
3534>>> import qualified Streamly.Internal.Data.Stream as Stream
36−>>> import qualified Streamly.Internal.Data.StreamK as StreamK
3735>>> import qualified Streamly.Internal.Data.Unfold as Unfold
3836>>> import qualified Streamly.Internal.FileSystem.Dir as Dir
3937-}
core/src/DocTestFileSystemHandle.hs+1−1
@@ -8,7 +8,7 @@
88
99For APIs that have not been released yet.
1010
11−>>> import qualified Streamly.Internal.Data.Array as Array (unsafeCreateOf)
11+>>> import qualified Streamly.Internal.Data.Array as Array (writeNUnsafe)
1212>>> import qualified Streamly.Internal.Data.Unfold as Unfold (first)
1313>>> import qualified Streamly.Internal.FileSystem.Handle as Handle
1414>>> import qualified Streamly.Internal.System.IO as IO (defaultChunkSize)
core/src/Streamly/Console/Stdio.hs+27−1
@@ -24,7 +24,33 @@ module Streamly.Console.Stdio
2424 -- * Write (stderr)
2525 , writeErr
2626 , writeErrChunks
27+
28+ -- * Deprecated
29+ , read
30+ , readChunks
2731 )
2832where
2933
30−import Streamly.Internal.Console.Stdio
34+import Control.Monad.IO.Class (MonadIO(..))
35+import Data.Word (Word8)
36+import Streamly.Internal.Data.Array.Type (Array)
37+import Streamly.Internal.Data.Unfold (Unfold)
38+
39+import Streamly.Internal.Console.Stdio hiding (read, readChunks)
40+import Prelude hiding (read)
41+
42+-- Same as 'reader'
43+--
44+-- @since 0.8.0
45+{-# DEPRECATED read "Please use 'reader' instead" #-}
46+{-# INLINE read #-}
47+read :: MonadIO m => Unfold m () Word8
48+read = reader
49+
50+-- Same as 'chunkReader'
51+--
52+-- @since 0.8.0
53+{-# DEPRECATED readChunks "Please use 'chunkReader' instead" #-}
54+{-# INLINE readChunks #-}
55+readChunks :: MonadIO m => Unfold m () (Array Word8)
56+readChunks = chunkReader
core/src/Streamly/Data/Array.hs+39−40
@@ -31,45 +31,28 @@ module Streamly.Data.Array
3131 -- * The Array Type
3232 Array
3333
34− -- * Pinning & Unpinning
35− -- | Arrays are created unpinned by default unless pinned versions of
36− -- creation APIs are used. Look for APIs with @pinned@ prefix in
37− -- "Streamly.Internal.Data.Array" for some unreleased pinned creation APIs.
38− -- If an array is to be sent to the OS without any further modification
39− -- then it should be created pinned in the first place instead of pinning
40− -- it later. Pinning an unpinned array has a copy overhead. OS interfacing
41− -- APIs create a pinned array directly or convert an unpinned array to
42− -- pinned array before sending it to the OS.
43− , pin
44− , unpin
45− , isPinned
46−
4734 -- * Construction
4835 -- | When performance matters, the fastest way to generate an array is
49− -- 'createOf'. 'IsList' and 'IsString' instances can be
36+ -- 'writeN'. 'IsList' and 'IsString' instances can be
5037 -- used to conveniently construct arrays from literal values.
5138 -- 'OverloadedLists' extension or 'fromList' can be used to construct an
5239 -- array from a list literal. Similarly, 'OverloadedStrings' extension or
5340 -- 'fromList' can be used to construct an array from a string literal.
5441
55− -- ** From Stream
56− , createOf
57− , create
58− , writeLastN -- drop old (ring buffer)
59−
60− -- ** From List
42+ -- Pure List APIs
6143 , fromListN
6244 , fromList
6345
64− -- * To List
46+ -- Monadic APIs
47+ , writeN -- drop new
48+ , write -- full buffer
49+ , writeLastN -- drop old (ring buffer)
50+
51+ -- * Conversion
6552 -- 'GHC.Exts.toList' from "GHC.Exts" can be used to convert an array to a
6653 -- list.
6754 , toList
6855
69− -- * To Stream
70− , read
71− , readRev
72−
7356 -- * Unfolds
7457 , reader
7558 , readerRev
@@ -83,24 +66,25 @@ module Streamly.Data.Array
8366 -- , (!!)
8467 , getIndex
8568
86− -- * Serialization
87− , pinnedSerialize
88− , deserialize
89−
90− -- * Re-exports
69+ -- * Unbox Type Class
9170 , Unbox (..)
92− , Serialize(..)
71+
72+ -- * Pinning & Unpinning
73+ , pin
74+ , unpin
75+ , isPinned
9376
9477 -- * Deprecated
95− , writeN -- drop new
96− , write -- full buffer
78+ , read
79+ , readRev
9780 )
9881where
9982
10083#include "inline.hs"
10184
102−import Streamly.Internal.Data.Array
103−import Streamly.Internal.Data.MutByteArray (Unbox(..), Serialize(..))
85+import Streamly.Internal.Data.Array hiding (read, readRev)
86+import Streamly.Internal.Data.Unbox (Unbox (..))
87+import Streamly.Internal.Data.Unfold (Unfold)
10488
10589import Prelude hiding (read, length)
10690
@@ -118,7 +102,7 @@ import Prelude hiding (read, length)
118102--
119103-- Convert array to stream, and fold the stream:
120104--
121−-- >>> fold f arr = Array.read arr & Stream.fold f
105+-- >>> fold f arr = Stream.unfold Array.reader arr & Stream.fold f
122106-- >>> fold Fold.sum (Array.fromList [1,2,3::Int])
123107-- 6
124108--
@@ -126,7 +110,7 @@ import Prelude hiding (read, length)
126110--
127111-- Convert array to stream, transform, and fold back to array:
128112--
129−-- >>> amap f arr = Array.read arr & fmap f & Stream.fold (Array.createOf (Array.length arr))
113+-- >>> amap f arr = Stream.unfold Array.reader arr & fmap f & Stream.fold Array.write
130114-- >>> amap (+1) (Array.fromList [1,2,3::Int])
131115-- fromList [2,3,4]
132116--
@@ -153,11 +137,12 @@ import Prelude hiding (read, length)
153137-- See the 'fromPureStream' unreleased API to generate an array from an
154138-- Identity stream safely without using MonadIO constraint.
155139--
140+--
156141-- Note that 'Identity' streams can be generalized to IO streams:
157142--
158−-- >>> stream = Stream.fromList [1,2,3] :: Stream Identity Int
159−-- >>> pure2IO s = Stream.morphInner (return . runIdentity) s :: Stream IO Int
160−-- >>> Stream.fold Array.create (pure2IO stream)
143+-- >>> pure = Stream.fromList [1,2,3] :: Stream Identity Int
144+-- >>> generally = Stream.morphInner (return . runIdentity)
145+-- >>> Stream.fold Array.write (generally pure :: Stream IO Int)
161146-- fromList [1,2,3]
162147--
163148-- == Programming Tips
@@ -165,3 +150,17 @@ import Prelude hiding (read, length)
165150-- This module is designed to be imported qualified:
166151--
167152-- >>> import qualified Streamly.Data.Array as Array
153+
154+-- | Same as 'reader'
155+--
156+{-# DEPRECATED read "Please use 'reader' instead" #-}
157+{-# INLINE_NORMAL read #-}
158+read :: (Monad m, Unbox a) => Unfold m (Array a) a
159+read = reader
160+
161+-- | Same as 'readerRev'
162+--
163+{-# DEPRECATED readRev "Please use 'readerRev' instead" #-}
164+{-# INLINE_NORMAL readRev #-}
165+readRev :: (Monad m, Unbox a) => Unfold m (Array a) a
166+readRev = readerRev
core/src/Streamly/Data/Array/Generic.hs+13−16
@@ -15,37 +15,34 @@ module Streamly.Data.Array.Generic
1515 ( Array
1616
1717 -- * Construction
18− , fromListN
19− , fromList
18+ , A.fromListN
19+ , A.fromList
2020
2121 -- MonadicAPIs
22− , createOf
23− , create
22+ , A.writeN
23+ , A.write
2424
2525 -- * Conversion
26− , toList
26+ , A.toList
2727
2828 -- * Streams
29− , read
30− , readRev
29+ , A.read
30+ , A.readRev
3131
3232 -- * Unfolds
33− , reader
33+ , A.reader
3434 -- , A.readerRev
3535
3636 -- * Random Access
37− , length
38− , getIndex
37+ , A.length
38+ , A.getIndex
3939
4040 -- -- * Folding Arrays
4141 -- , A.streamFold
4242 -- , A.fold
43−
44− -- * Deprecated
45− , writeN
46− , write
4743 )
4844where
4945
50−import Streamly.Internal.Data.Array.Generic
51−import Prelude hiding (length, read)
46+import Streamly.Internal.Data.Array.Generic (Array)
47+
48+import qualified Streamly.Internal.Data.Array.Generic as A
core/src/Streamly/Data/Fold.hs+118−150
@@ -10,156 +10,6 @@
1010-- Fast, composable stream consumers with ability to terminate, supporting
1111-- stream fusion.
1212--
13−-- == Using Folds
14−--
15−-- This module provides elementary folds and fold combinators that can be used
16−-- to consume a stream of data and reduce it to a final value, or transform it
17−-- in a stateful manner using scans. A data stream can be reduced into a stream
18−-- of folded data elements by folding segments of the stream. Fold combinators
19−-- can be used to compose multiple folds in parallel or to create a pipeline of
20−-- folds such that the next fold consumes the result of the previous fold. To
21−-- run these folds on a stream see 'Streamly.Data.Stream.fold',
22−-- 'Streamly.Data.Stream.scan', 'Streamly.Data.Stream.postscan',
23−-- 'Streamly.Data.Stream.scanMaybe', 'Streamly.Data.Stream.foldMany' and other
24−-- operations accepting 'Fold' type as argument "Streamly.Data.Stream".
25−--
26−-- == Reducing a Stream
27−--
28−-- A 'Fold' is a consumer of a stream of values. A fold driver (such as
29−-- 'Streamly.Data.Stream.fold') initializes the fold @accumulator@, runs the
30−-- fold @step@ function in a loop, processing the input stream one element at a
31−-- time and accumulating the result. The loop continues until the fold
32−-- terminates, at which point the accumulated result is returned.
33−--
34−-- For example, a 'sum' Fold represents a stream consumer that adds the values
35−-- in the input stream:
36−--
37−-- >>> Stream.fold Fold.sum $ Stream.fromList [1..100]
38−-- 5050
39−--
40−-- Conceptually, a 'Fold' is a data type that mimics a strict left fold
41−-- ('Data.List.foldl'). The above example is similar to a left fold using
42−-- @(+)@ as the step and @0@ as the initial value of the accumulator:
43−--
44−-- >>> Data.List.foldl' (+) 0 [1..100]
45−-- 5050
46−--
47−-- 'Fold's have an early termination capability e.g. the 'one' fold terminates
48−-- after consuming one element:
49−--
50−-- >>> Stream.fold Fold.one $ Stream.fromList [1..]
51−-- Just 1
52−--
53−-- The above example is similar to the following right fold:
54−--
55−-- >>> Prelude.foldr (\x _ -> Just x) Nothing [1..]
56−-- Just 1
57−--
58−-- 'Fold's can be combined together using combinators. For example, to create a
59−-- fold that sums first two elements in a stream:
60−--
61−-- >>> sumTwo = Fold.take 2 Fold.sum
62−-- >>> Stream.fold sumTwo $ Stream.fromList [1..100]
63−-- 3
64−--
65−-- == Parallel Composition
66−--
67−-- Folds can be combined to run in parallel on the same input. For example, to
68−-- compute the average of numbers in a stream without going through the stream
69−-- twice:
70−--
71−-- >>> avg = Fold.teeWith (/) Fold.sum (fmap fromIntegral Fold.length)
72−-- >>> Stream.fold avg $ Stream.fromList [1.0..100.0]
73−-- 50.5
74−--
75−-- Folds can be combined so as to partition the input stream over multiple
76−-- folds. For example, to count even and odd numbers in a stream:
77−--
78−-- >>> split n = if even n then Left n else Right n
79−-- >>> stream = fmap split $ Stream.fromList [1..100]
80−-- >>> countEven = fmap (("Even " ++) . show) Fold.length
81−-- >>> countOdd = fmap (("Odd " ++) . show) Fold.length
82−-- >>> f = Fold.partition countEven countOdd
83−-- >>> Stream.fold f stream
84−-- ("Even 50","Odd 50")
85−--
86−-- == Sequential Composition
87−--
88−-- Terminating folds can be combined to parse the stream serially such that the
89−-- first fold consumes the input until it terminates and the second fold
90−-- consumes the rest of the input until it terminates:
91−--
92−-- >>> f = Fold.splitWith (,) (Fold.take 8 Fold.toList) (Fold.takeEndBy (== '\n') Fold.toList)
93−-- >>> Stream.fold f $ Stream.fromList "header: hello\n"
94−-- ("header: ","hello\n")
95−--
96−-- == Splitting a Stream
97−--
98−-- A 'Fold' can be applied repeatedly on a stream to transform it to a stream
99−-- of fold results. To split a stream on newlines:
100−--
101−-- >>> f = Fold.takeEndBy (== '\n') Fold.toList
102−-- >>> Stream.fold Fold.toList $ Stream.foldMany f $ Stream.fromList "Hello there!\nHow are you\n"
103−-- ["Hello there!\n","How are you\n"]
104−--
105−-- Similarly, we can split the input of a fold too:
106−--
107−-- >>> Stream.fold (Fold.many f Fold.toList) $ Stream.fromList "Hello there!\nHow are you\n"
108−-- ["Hello there!\n","How are you\n"]
109−--
110−-- == Folds vs. Streams
111−--
112−-- We can often use streams or folds to achieve the same goal. However, streams
113−-- are more efficient in composition of producers (e.g.
114−-- 'Data.Stream.append' or 'Data.Stream.mergeBy') whereas folds are
115−-- more efficient in composition of consumers (e.g. 'splitWith', 'partition'
116−-- or 'teeWith').
117−--
118−-- Streams are producers, transformations on streams happen on the output side:
119−--
120−-- >>> :{
121−-- f stream =
122−-- Stream.filter odd stream
123−-- & fmap (+1)
124−-- & Stream.fold Fold.sum
125−-- :}
126−--
127−-- >>> f $ Stream.fromList [1..100 :: Int]
128−-- 2550
129−--
130−-- Folds are stream consumers with an input stream and an output value, stream
131−-- transformations on folds happen on the input side:
132−--
133−-- >>> :{
134−-- f =
135−-- Fold.filter odd
136−-- $ Fold.lmap (+1)
137−-- $ Fold.sum
138−-- :}
139−--
140−-- >>> Stream.fold f $ Stream.fromList [1..100 :: Int]
141−-- 2550
142−--
143−-- Notice the similiarity in the definition of @f@ in both cases, the only
144−-- difference is the composition by @&@ vs @$@ and the use @lmap@ vs @map@, the
145−-- difference is due to output vs input side transformations.
146−--
147−-- == Fusion Limitations
148−--
149−-- Folds support stream fusion for generating loops comparable to the speed of
150−-- C. However, it has some limitations. For fusion to work, the folds must be
151−-- inlined, folds must be statically known and not generated dynamically, folds
152−-- should not be passed recursively.
153−--
154−-- Another limitation is due to the quadratic complexity causing slowdown when
155−-- too many nested compositions are used. Especially, the performance of the
156−-- Applicative instance and splitting operations (e.g. 'splitWith') degrades
157−-- quadratically (O(n^2)) when combined @n@ times, roughly 8 or less sequenced
158−-- operations are fine. For these cases folds can be converted to parsers and
159−-- then used as ParserK.
160−--
161−-- == Experimental APIs
162−--
16313-- Please refer to "Streamly.Internal.Data.Fold" for more functions that have
16414-- not yet been released.
16515
@@ -171,6 +21,9 @@ module Streamly.Data.Fold
17121 --
17222 -- $setup
17323
24+ -- * Overview
25+ -- $overview
26+
17427 -- * Running A Fold
17528 drive
17629 -- XXX Should we have a stream returning function in fold module?
@@ -405,6 +258,121 @@ import Streamly.Internal.Data.Fold
405258
406259#include "DocTestDataFold.hs"
407260
261+-- $overview
262+--
263+-- A 'Fold' is a consumer of a stream of values. A fold driver (such as
264+-- 'Streamly.Data.Stream.fold') initializes the fold @accumulator@, runs the
265+-- fold @step@ function in a loop, processing the input stream one element at a
266+-- time and accumulating the result. The loop continues until the fold
267+-- terminates, at which point the accumulated result is returned.
268+--
269+-- For example, a 'sum' Fold represents a stream consumer that adds the values
270+-- in the input stream:
271+--
272+-- >>> Stream.fold Fold.sum $ Stream.fromList [1..100]
273+-- 5050
274+--
275+-- Conceptually, a 'Fold' is a data type that mimics a strict left fold
276+-- ('Data.List.foldl'). The above example is similar to a left fold using
277+-- @(+)@ as the step and @0@ as the initial value of the accumulator:
278+--
279+-- >>> Data.List.foldl' (+) 0 [1..100]
280+-- 5050
281+--
282+-- 'Fold's have an early termination capability e.g. the 'one' fold terminates
283+-- after consuming one element:
284+--
285+-- >>> Stream.fold Fold.one $ Stream.fromList [1..]
286+-- Just 1
287+--
288+-- The above example is similar to the following right fold:
289+--
290+-- >>> Prelude.foldr (\x _ -> Just x) Nothing [1..]
291+-- Just 1
292+--
293+-- 'Fold's can be combined together using combinators. For example, to create a
294+-- fold that sums first two elements in a stream:
295+--
296+-- >>> sumTwo = Fold.take 2 Fold.sum
297+-- >>> Stream.fold sumTwo $ Stream.fromList [1..100]
298+-- 3
299+--
300+-- Folds can be combined to run in parallel on the same input. For example, to
301+-- compute the average of numbers in a stream without going through the stream
302+-- twice:
303+--
304+-- >>> avg = Fold.teeWith (/) Fold.sum (fmap fromIntegral Fold.length)
305+-- >>> Stream.fold avg $ Stream.fromList [1.0..100.0]
306+-- 50.5
307+--
308+-- Folds can be combined so as to partition the input stream over multiple
309+-- folds. For example, to count even and odd numbers in a stream:
310+--
311+-- >>> split n = if even n then Left n else Right n
312+-- >>> stream = fmap split $ Stream.fromList [1..100]
313+-- >>> countEven = fmap (("Even " ++) . show) Fold.length
314+-- >>> countOdd = fmap (("Odd " ++) . show) Fold.length
315+-- >>> f = Fold.partition countEven countOdd
316+-- >>> Stream.fold f stream
317+-- ("Even 50","Odd 50")
318+--
319+-- Terminating folds can be combined to parse the stream serially such that the
320+-- first fold consumes the input until it terminates and the second fold
321+-- consumes the rest of the input until it terminates:
322+--
323+-- >>> f = Fold.splitWith (,) (Fold.take 8 Fold.toList) (Fold.takeEndBy (== '\n') Fold.toList)
324+-- >>> Stream.fold f $ Stream.fromList "header: hello\n"
325+-- ("header: ","hello\n")
326+--
327+-- A 'Fold' can be applied repeatedly on a stream to transform it to a stream
328+-- of fold results. To split a stream on newlines:
329+--
330+-- >>> f = Fold.takeEndBy (== '\n') Fold.toList
331+-- >>> Stream.fold Fold.toList $ Stream.foldMany f $ Stream.fromList "Hello there!\nHow are you\n"
332+-- ["Hello there!\n","How are you\n"]
333+--
334+-- Similarly, we can split the input of a fold too:
335+--
336+-- >>> Stream.fold (Fold.many f Fold.toList) $ Stream.fromList "Hello there!\nHow are you\n"
337+-- ["Hello there!\n","How are you\n"]
338+--
339+-- = Folds vs. Streams
340+--
341+-- We can often use streams or folds to achieve the same goal. However, streams
342+-- are more efficient in composition of producers (e.g.
343+-- 'Data.Stream.append' or 'Data.Stream.mergeBy') whereas folds are
344+-- more efficient in composition of consumers (e.g. 'splitWith', 'partition'
345+-- or 'teeWith').
346+--
347+-- Streams are producers, transformations on streams happen on the output side:
348+--
349+-- >>> :{
350+-- f stream =
351+-- Stream.filter odd stream
352+-- & fmap (+1)
353+-- & Stream.fold Fold.sum
354+-- :}
355+--
356+-- >>> f $ Stream.fromList [1..100 :: Int]
357+-- 2550
358+--
359+-- Folds are stream consumers with an input stream and an output value, stream
360+-- transformations on folds happen on the input side:
361+--
362+-- >>> :{
363+-- f =
364+-- Fold.filter odd
365+-- $ Fold.lmap (+1)
366+-- $ Fold.sum
367+-- :}
368+--
369+-- >>> Stream.fold f $ Stream.fromList [1..100 :: Int]
370+-- 2550
371+--
372+-- Notice the similiarity in the definition of @f@ in both cases, the only
373+-- difference is the composition by @&@ vs @$@ and the use @lmap@ vs @map@, the
374+-- difference is due to output vs input side transformations.
375+
408376--------------------------------------------------------------------------------
409377-- Deprecated
410378--------------------------------------------------------------------------------
core/src/Streamly/Data/MutArray.hs+12−18
@@ -32,27 +32,22 @@ module Streamly.Data.MutArray
3232 -- * Construction
3333
3434 -- Uninitialized Arrays
35− , emptyOf
36− , pinnedEmptyOf
35+ , new
36+ , pinnedNew
3737
3838 -- From containers
3939 , fromListN
4040 , fromList
41− , createOf
42− , create
41+ , writeN -- drop new
42+ , write -- full buffer
4343 -- writeLastN
4444
45− -- * Pinning & Unpinning
46− , pin
47− , unpin
48− , isPinned
49−
5045 -- * Appending elements
5146 , snoc
5247
5348 -- * Appending streams
54− , appendN
55− , append
49+ , writeAppendN -- XXX Rename to writeSnocN?
50+ , writeAppend -- XXX Rename to writeSnoc?
5651
5752 -- * Inplace mutation
5853 , putIndex
@@ -83,17 +78,16 @@ module Streamly.Data.MutArray
8378 -- * Size
8479 , length
8580
86− -- * Re-exports
81+ -- * Unbox Type Class
8782 , Unbox (..)
8883
84+ -- * Pinning & Unpinning
85+ , pin
86+ , unpin
87+ , isPinned
88+
8989 -- * Deprecated
9090 , newPinned
91− , new
92− , pinnedNew
93− , writeN
94− , write
95− , writeAppendN
96− , writeAppend
9791 )
9892where
9993
core/src/Streamly/Data/MutArray/Generic.hs+3−8
@@ -23,11 +23,11 @@ module Streamly.Data.MutArray.Generic
2323 MutArray
2424
2525 -- * Construction
26− , emptyOf
26+ , new
2727 , fromListN
2828 , fromList
29− , createOf
30− , create
29+ , writeN
30+ , write
3131
3232 -- * Appending elements
3333 , snoc
@@ -56,11 +56,6 @@ module Streamly.Data.MutArray.Generic
5656
5757 -- * Size
5858 , length
59−
60− -- * Deprecated
61− , new
62− , writeN
63− , write
6459 )
6560where
6661
core/src/Streamly/Data/MutByteArray.hsdeleted−106
@@ -1,106 +0,0 @@
1−-- |
2−-- Module : Streamly.Data.MutByteArray
3−-- Copyright : (c) 2023 Composewell Technologies
4−-- License : BSD3-3-Clause
5−-- Maintainer : streamly@composewell.com
6−-- Portability : GHC
7−--
8−-- A low level byte Array type 'MutByteArray', along with type classes 'Unbox'
9−-- and 'Serialize' for fast binary serialization and deserialization of Haskell
10−-- values. Serialization, deserialization performance is similar to, and in
11−-- some cases many times better than the store package. Conceptually, the
12−-- 'Serialize' type class works in the same way as store.
13−--
14−-- Serialize instances are configurable to use constructor names (see
15−-- 'Streamly.Internal.Data.MutByteArray.encodeConstrNames'), record field names (see
16−-- 'Streamly.Internal.Data.MutByteArray.encodeRecordFields') instead of binary
17−-- encoded values. This is an experimental feature which allows JSON like
18−-- properties with faster speed. For example, you can change the order of
19−-- constructors or record fields without affecting serialized value.
20−--
21−-- Higher level unboxed array modules "Streamly.Data.Array" and
22−-- "Streamly.Data.MutArray" are built on top of this module. Unboxed arrays are
23−-- essentially serialized Haskell values. Array modules provide higher level
24−-- serialization routines like 'Streamly.Internal.Data.Array.pinnedSerialize'
25−-- and 'Streamly.Internal.Data.Array.deserialize' from the
26−-- "Streamly.Internal.Data.Array" module.
27−--
28−-- == Mutable Byte Array
29−--
30−-- 'MutByteArray' is a primitive mutable array in the IO monad. 'Unbox' and
31−-- 'Serialize' type classes use this primitive array to serialize data to and
32−-- deserialize it from. This array is used to build higher level unboxed
33−-- array types 'Streamly.Data.MutArray.MutArray' and 'Streamly.Data.Array.Array'.
34−--
35−-- == Using Unbox
36−--
37−-- The 'Unbox' type class is simple and used to serialize non-recursive fixed
38−-- size data types. This type class is primarily used to implement unboxed
39−-- arrays. Unboxed arrays are just a sequence of serialized fixed length
40−-- Haskell data types. Instances of this type class can be derived using
41−-- 'Generic' or template haskell based deriving functions provided in this
42−-- module.
43−--
44−-- Writing a data type to an array using the array creation routines in
45−-- "Streamly.Data.Array" or "Streamly.Data.MutArray" (e.g. @writeN@ or
46−-- @fromListN@), serializes the type to the array. Similarly, reading the data
47−-- type from the array deserializes it. You can also serialize and deserialize
48−-- directly to and from a 'MutByteArray', using the type class methods.
49−--
50−-- == Using Serialize
51−--
52−-- The 'Serialize' type class is a superset of the 'Unbox' type class, it can
53−-- serialize variable length data types as well e.g. Haskell lists. Use
54−-- 'deriveSerialize' to derive the instances of the type class automatically
55−-- and then use the type class methods to serialize and deserialize to and from
56−-- a 'MutByteArray'.
57−--
58−-- See 'Streamly.Internal.Data.Array.pinnedSerialize' and
59−-- 'Streamly.Internal.Data.Array.deserialize' for 'Array' type based
60−-- serialization.
61−--
62−-- == Comparing serialized values
63−--
64−-- When using the `Unbox` type class the same value may result in differing
65−-- serialized bytes because of unused uninitialized data in case of sum types.
66−-- Therefore, byte comparison of serialized values is not reliable.
67−--
68−-- However, the 'Serialize' type class guarantees that the serialized values
69−-- are always exactly the same and byte comparison of serialized is reliable.
70−--
71−module Streamly.Data.MutByteArray
72− (
73−
74− -- * Mutable Byte Array
75− -- | The standard way to read from or write to a 'MutByteArray' is by using
76− -- the 'Unbox' or 'Serialize' type class methods.
77− MutByteArray
78− , isPinned
79− , pin
80− , unpin
81− , new
82− , pinnedNew
83−
84− -- * Unbox
85− , Unbox(..)
86− , deriveUnbox
87−
88− -- * Serialize
89− , Serialize(..)
90−
91− -- ** Instance Config
92− , SerializeConfig
93− , inlineAddSizeTo
94− , inlineSerializeAt
95− , inlineDeserializeAt
96−
97− -- ** Instance Deriving
98− , deriveSerialize
99− , deriveSerializeWith
100− ) where
101−
102−--------------------------------------------------------------------------------
103−-- Imports
104−--------------------------------------------------------------------------------
105−
106−import Streamly.Internal.Data.MutByteArray
core/src/Streamly/Data/Parser.hs+60−130
@@ -7,137 +7,12 @@
77-- Stability : pre-release
88-- Portability : GHC
99--
10−-- Parsers are stream consumers like folds with the following differences:
10+-- Fast, composable stream consumers with ability to terminate, backtrack and
11+-- fail, supporting stream fusion. Parsers are a natural extension of
12+-- "Streamly.Data.Fold". Parsers and folds can be interconverted.
1113--
12−-- * folds cannot fail but parsers can fail and backtrack.
13−-- * folds can be composed as a Tee but parsers cannot.
14−-- * folds can be used for scanning but parsers cannot.
15−-- * folds can be converted to parsers.
16−--
17−-- This module implements parsers with stream fusion which compile to efficient
18−-- loops comparable to the speed of C.
19−--
20−-- == Using Parsers
21−--
22−-- This module provides elementary parsers and parser combinators that can be
23−-- used to parse a stream of data. Additionally, all the folds from the
24−-- "Streamly.Data.Fold" module can be converted to parsers using 'fromFold'.
25−-- All the parsing functionality provided by popular parsing libraries, and
26−-- more is available. Also see "Streamly.Unicode.Parser" module for Char stream
27−-- parsers.
28−--
29−-- A data stream can be transformed to a stream of parsed data elements. Parser
30−-- combinators can be used to create a pipeline of folds or parsers such that
31−-- the next fold or parser consumes the result of the previous parser. See
32−-- 'Streamly.Data.Stream.parse' and 'Streamly.Data.Stream.parseMany' to run
33−-- these parsers on a stream.
34−--
35−-- == Parser vs ParserK
36−--
37−-- There are two functionally equivalent parsing modules,
38−-- "Streamly.Data.Parser" (this module) and "Streamly.Data.ParserK". The latter
39−-- is a CPS based wrapper over the former, and can be used for parsing in
40−-- general. "Streamly.Data.Parser" enables stream fusion and should be
41−-- preferred over "Streamly.Data.ParserK" for high performance stream parsing
42−-- use cases. However, there are a few cases where this module is not
43−-- suitable and ParserK should be used instead.
44−--
45−-- For static fusion, parser combinators have to use strict pattern matching on
46−-- arguments of type Parser. This leads to infinte loop when a parser is
47−-- defined recursively, due to strict evaluation of the recursive call. For
48−-- example, the following implementation loops infinitely because of the
49−-- recursive use of parser @p@ in the @*>@ combinator:
50−--
51−-- >>> import Streamly.Data.Parser (Parser)
52−-- >>> import qualified Streamly.Data.Fold as Fold
53−-- >>> import qualified Streamly.Data.Parser as Parser
54−-- >>> import qualified Streamly.Data.Stream as Stream
55−-- >>> import Control.Applicative ((<|>))
56−--
57−-- >>> :{
58−-- >>> p :: Monad m => Parser Char m String
59−-- >>> p = Parser.satisfy (== '(') *> p <|> Parser.fromFold Fold.toList
60−-- >>> :}
61−--
62−-- Use ParserK when recursive use is required:
63−--
64−-- >>> import Streamly.Data.ParserK (ParserK)
65−-- >>> import qualified Streamly.Data.StreamK as StreamK
66−-- >>> import qualified Streamly.Internal.Data.StreamK as StreamK (parse)
67−-- >>> import qualified Streamly.Internal.Data.ParserK as ParserK (adapt)
68−--
69−-- >>> :{
70−-- >>> p :: Monad m => ParserK Char m String
71−-- >>> p = ParserK.adapt (Parser.satisfy (== '(')) *> p <|> ParserK.adapt (Parser.fromFold Fold.toList)
72−-- >>> :}
73−--
74−-- >>> StreamK.parse p $ StreamK.fromStream $ Stream.fromList "hello"
75−-- Right "hello"
76−--
77−-- For this reason Applicative, Alternative or Monad compositions with
78−-- recursion cannot be used with the 'Parser' type. Alternative type class based
79−-- operations like 'asum' and Alternative based generic parser combinators use
80−-- recursion. Similarly, Applicative type class based operations like
81−-- 'Prelude.sequence' use recursion. Custom implementations of many such
82−-- operations are provided in this module (e.g. 'some', 'many'), and those
83−-- should be used instead.
84−--
85−-- Another limitation of Parser type is due to the quadratic complexity causing
86−-- slowdown when too many nested compositions are used. Especially Applicative,
87−-- Monad, Alternative instances, and sequenced parsing operations (e.g. nested
88−-- 'one', and 'splitWith') degrade the performance quadratically (O(n^2)) when
89−-- combined @n@ times, roughly 8 or less sequenced parsers are fine. READ THE
90−-- DOCS OF APPLICATIVE, MONAD AND ALTERNATIVE INSTANCES.
91−--
92−-- == Streaming Parsers
93−--
94−-- With 'Streamly.Data.ParserK.ParserK' you can use the generic Alternative
95−-- type class based parsers from the
96−-- <https://hackage.haskell.org/package/parser-combinators parser-combinators>
97−-- library or similar. However, we recommend that you use the equivalent
98−-- functionality from this module for better performance and for streaming
99−-- behavior.
100−--
101−-- Firstly, the combinators in this module are faster due to stream fusion.
102−-- Secondly, these are streaming in nature as the results can be passed
103−-- directly to other stream consumers (folds or parsers). The Alternative type
104−-- class based parsers would end up buffering all the results in lists before
105−-- they can be consumed.
106−--
107−-- When recursion or heavy nesting is needed use ParserK.
108−--
109−-- == Error Reporting
110−--
111−-- These parsers do not report the error context (e.g. line number or column).
112−-- This may be supported in future.
113−--
114−-- == Monad Transformer Stack
115−--
116−-- 'MonadTrans' instance is not provided. If the 'Parser' type is the top most
117−-- layer (which should be the case almost always) you can just use 'fromEffect'
118−-- to execute the lower layer monad effects.
119−--
120−-- == Parser vs ParserK Implementation
121−--
122−-- The 'Parser' type represents a stream consumer by composing state as data
123−-- which enables stream fusion. Stream fusion generates a tight loop without
124−-- any constructor allocations between the stages, providing C like performance
125−-- for the loop. Stream fusion works when multiple functions are combined in a
126−-- pipeline statically. Therefore, the operations in this module must be
127−-- inlined and must not be used recursively to allow for stream fusion.
128−--
129−-- The 'ParserK' type represents a stream consumer by composing function calls,
130−-- therefore, a function call overhead is incurred at each composition. It is
131−-- quite fast in general but may be a few times slower than a fused parser.
132−-- However, it allows for scalable dynamic composition especially parsers can
133−-- be used in recursive calls. Using the 'ParserK' type operations like
134−-- 'splitWith' provide linear (O(n)) performance with respect to the number of
135−-- compositions.
136−--
137−-- == Experimental APIs
138−--
139−-- Please refer to "Streamly.Internal.Data.Parser" for functions that have not
140−-- yet been released.
14+-- Please refer to "Streamly.Internal.Data.Parser" for functions that have
15+-- not yet been released.
14116--
14217module Streamly.Data.Parser
14318 (
@@ -147,6 +22,9 @@ module Streamly.Data.Parser
14722 --
14823 -- $setup
14924
25+ -- * Overview
26+ -- $overview
27+
15028 -- * Parser Type
15129 Parser
15230
@@ -245,3 +123,55 @@ import Streamly.Internal.Data.Parser
245123import Prelude hiding (dropWhile, takeWhile, filter)
246124
247125#include "DocTestDataParser.hs"
126+
127+-- $overview
128+--
129+-- Several combinators in this module can be many times faster than CPS based
130+-- parsers because of stream fusion. For example,
131+-- 'Streamly.Internal.Data.Parser.many' combinator in this module is much
132+-- faster than the 'Control.Applicative.many' combinator of
133+-- 'Control.Applicative.Alternative' type class used by CPS based parsers.
134+--
135+-- The use of 'Alternative' type class, in parsers has another drawback.
136+-- Alternative based parsers use plain Haskell lists to collect the results. In
137+-- a strict Monad like IO, the results are necessarily buffered before they can
138+-- be consumed. This may not perform optimally in streaming applications
139+-- processing large amounts of data. Equivalent combinators in this module can
140+-- consume the results of parsing using a 'Fold' or another parser, thus
141+-- providing a scalable and composable consumer.
142+--
143+-- Note that these parsers do not report the error context (e.g. line number or
144+-- column). This may be supported in future.
145+--
146+-- mtl instances are not provided. If the 'Parser' type is the top most layer
147+-- (which should be the case almost always) you can just use 'fromEffect' to
148+-- execute the lower layer monad effects.
149+--
150+-- == Performance Notes
151+--
152+-- The 'Parser' type represents a stream consumer by composing state as data
153+-- which enables stream fusion. Stream fusion generates a tight loop without
154+-- any constructor allocations between the stages, providing C like performance
155+-- for the loop. Stream fusion works when multiple functions are combined in a
156+-- pipeline statically. Therefore, the operations in this module must be
157+-- inlined and must not be used recursively to allow for stream fusion. Note
158+-- that operations like 'sequence', and 'asum' that compose pasrers using
159+-- recursion should be avoided with these parsers. You can use these with the
160+-- 'ParserK' module instead.
161+--
162+-- Using the 'Parser' type, parsing operations like 'one', 'splitWith' etc.
163+-- degrade quadratically (O(n^2)) when combined many times. If you need to
164+-- combine these operations, say more than 8 times in a single loop, then you
165+-- should consider using the continuation style parser type 'ParserK' instead.
166+-- Also, if you need to use these operations in a recursive loop you should use
167+-- 'ParserK' instead.
168+--
169+-- The 'ParserK' type represents a stream consumer by composing function calls,
170+-- therefore, a function call overhead is incurred at each composition. It is
171+-- quite fast in general but may be a few times slower than a fused parser.
172+-- However, it allows for scalable dynamic composition especially parsers can
173+-- be used in recursive calls. Using the 'ParserK' type operations like
174+-- 'splitWith' provide linear (O(n)) performance with respect to the number of
175+-- compositions..
176+--
177+-- 'Parser' and 'ParserK' types can be interconverted.
core/src/Streamly/Data/ParserK.hs+3−89
@@ -1,4 +1,3 @@
1−{-# LANGUAGE CPP #-}
21-- |
32-- Module : Streamly.Data.ParserK
43-- Copyright : (c) 2023 Composewell Technologies
@@ -7,102 +6,19 @@
76-- Stability : pre-release
87-- Portability : GHC
98--
10−-- See the general notes about parsing in the "Streamly.Data.Parser" module.
11−-- This module implements a using Continuation Passing Style (CPS) wrapper over
12−-- the "Streamly.Data.Parser" module. It is as fast or faster than attoparsec.
9+-- Parsers using Continuation Passing Style (CPS). See notes in
10+-- "Streamly.Data.Parser" module to know when to use this module.
1311--
14−-- == Parser vs ParserK
15−--
16−-- 'ParserK' is preferred over 'Streamly.Data.Parser.Parser' when extensive
17−-- applicative, alternative and monadic composition is required, or when
18−-- recursive or dynamic composition of parsers is required. The
19−-- 'Streamly.Data.Parser.Parser' type fuses statically and creates efficient
20−-- loops whereas 'ParserK' uses function call based composition and has
21−-- comparatively larger runtime overhead but it is better suited to the
22−-- specific use cases mentioned above. 'ParserK' also allows to efficient parse
23−-- a stream of arrays, it can also break the input stream into a parse result
24−-- and remaining stream so that the stream can be parsed independently in
25−-- segments.
26−--
27−-- == Using ParserK
28−--
29−-- All the parsers from the "Streamly.Data.Parser" module can be adapted to
30−-- ParserK using the 'Streamly.Data.ParserK.adaptC',
31−-- 'Streamly.Internal.Data.ParserK.adapt', and
32−-- 'Streamly.Internal.Data.ParserK.adaptCG' combinators.
33−--
34−-- 'Streamly.Data.StreamK.parseChunks' runs a parser on a stream of unboxed
35−-- arrays, this is the preferred and most efficient way to parse chunked input.
36−-- The more general 'Streamly.Data.StreamK.parseBreakChunks' function returns
37−-- the remaining stream as well along with the parse result. There are
38−-- 'Streamly.Internal.Data.StreamK.parseChunksGeneric',
39−-- 'Streamly.Internal.Data.StreamK.parseBreakChunksGeneric' as well to run
40−-- parsers on boxed arrays. 'Streamly.Internal.Data.StreamK.parse',
41−-- 'Streamly.Internal.Data.StreamK.parseBreak' run parsers on a stream of
42−-- individual elements instead of stream of arrays.
43−--
44−-- == Monadic Composition
45−--
46−-- Monad composition can be used for lookbehind parsers, we can dynamically
47−-- compose new parsers based on the results of the previously parsed values.
48−--
49−-- If we have to parse "a9" or "9a" but not "99" or "aa" we can use the
50−-- following non-monadic, backtracking parser:
51−--
52−-- >>> digits p1 p2 = ((:) <$> p1 <*> ((:) <$> p2 <*> pure []))
53−-- >>> :{
54−-- backtracking :: Monad m => ParserK Char m String
55−-- backtracking = ParserK.adapt $
56−-- digits (Parser.satisfy isDigit) (Parser.satisfy isAlpha)
57−-- <|>
58−-- digits (Parser.satisfy isAlpha) (Parser.satisfy isDigit)
59−-- :}
60−--
61−-- We know that if the first parse resulted in a digit at the first place then
62−-- the second parse is going to fail. However, we waste that information and
63−-- parse the first character again in the second parse only to know that it is
64−-- not an alphabetic char. By using lookbehind in a 'Monad' composition we can
65−-- avoid redundant work:
66−--
67−-- >>> data DigitOrAlpha = Digit Char | Alpha Char
68−--
69−-- >>> :{
70−-- lookbehind :: Monad m => ParserK Char m String
71−-- lookbehind = do
72−-- x1 <- ParserK.adapt $
73−-- Digit <$> Parser.satisfy isDigit
74−-- <|> Alpha <$> Parser.satisfy isAlpha
75−-- -- Note: the parse depends on what we parsed already
76−-- x2 <- ParserK.adapt $
77−-- case x1 of
78−-- Digit _ -> Parser.satisfy isAlpha
79−-- Alpha _ -> Parser.satisfy isDigit
80−-- return $ case x1 of
81−-- Digit x -> [x,x2]
82−-- Alpha x -> [x,x2]
83−-- :}
84−--
85−-- == Experimental APIs
86−--
87−-- Please refer to "Streamly.Internal.Data.ParserK" for functions that have
88−-- not yet been released.
12+-- To run a 'ParserK' use 'Streamly.Data.StreamK.parseChunks'.
8913--
9014module Streamly.Data.ParserK
9115 (
92− -- * Setup
93− -- | To execute the code examples provided in this module in ghci, please
94− -- run the following commands first.
95− --
96− -- $setup
97−
9816 -- * Parser Type
9917 ParserK
10018
10119 -- * Parsers
10220 -- ** Conversions
103− , adapt
10421 , adaptC
105− , adaptCG
10622 -- , toParser
10723
10824 -- ** Without Input
@@ -125,8 +41,6 @@ import qualified Streamly.Internal.Data.Parser as ParserD
12541
12642import Streamly.Internal.Data.ParserK.Type
12743
128−#include "DocTestDataParserK.hs"
129−
13044{-# DEPRECATED fromFold "Please use \"ParserK.adaptC . Parser.fromFold\" instead." #-}
13145{-# INLINE fromFold #-}
13246fromFold :: (MonadIO m, Unbox a) => Fold m a b -> ParserK (Array a) m b
core/src/Streamly/Data/Serialize.hsnew file+50
@@ -0,0 +1,50 @@
1+-- |
2+-- Module : Streamly.Data.Serialize
3+-- Copyright : (c) 2023 Composewell Technologies
4+-- License : BSD3-3-Clause
5+-- Maintainer : streamly@composewell.com
6+-- Portability : GHC
7+--
8+module Streamly.Data.Serialize
9+ (
10+
11+ -- * Types
12+ MutableByteArray
13+
14+ -- * Unbox
15+ , Unbox(..)
16+ , deriveUnbox
17+ , deriveUnboxWith
18+
19+ -- * Serialize
20+ , Serialize(..)
21+ , deriveSerialize
22+ , deriveSerializeWith
23+
24+ -- * Encoding
25+ , encode
26+ , pinnedEncode
27+
28+ -- * Decoding
29+ , decode
30+ ) where
31+
32+--------------------------------------------------------------------------------
33+-- Imports
34+--------------------------------------------------------------------------------
35+
36+import Streamly.Internal.Data.Serialize
37+ ( Serialize(..)
38+ , decode
39+ , encode
40+ , pinnedEncode
41+ )
42+import Streamly.Internal.Data.Serialize.TH
43+ ( deriveSerialize
44+ , deriveSerializeWith
45+ )
46+import Streamly.Internal.Data.Unbox
47+ ( MutableByteArray
48+ , Unbox(..)
49+ )
50+import Streamly.Internal.Data.Unbox.TH (deriveUnbox, deriveUnboxWith)
core/src/Streamly/Data/Stream.hs+36−131
@@ -70,15 +70,6 @@ module Streamly.Data.Stream
7070
7171 -- ** From Values
7272 -- | Generate a monadic stream from a seed value or values.
73− --
74− -- Non-empty pure streams can be cycled using repeat and unfold:
75− --
76− -- >>> cycle = Stream.unfoldMany Unfold.fromList . Stream.repeat
77− --
78− -- Non-empty StreamK can be cycled using append:
79− --
80− -- >>> cycle xs = let ys = xs `StreamK.append` ys in ys
81− --
8273 , fromPure
8374 , fromEffect
8475 , repeat
@@ -221,57 +212,21 @@ module Streamly.Data.Stream
221212 -- operations like mapping a function over the stream.
222213 , foldrM
223214 , foldr
224− -- foldr1
225215
226216 -- ** Specific Folds
227− -- | Streams are folded using folds in "Streamly.Data.Fold". Here are some
228− -- idioms and equivalents of Data.List APIs using folds:
217+ -- | Usually you can use the folds in "Streamly.Data.Fold". However, some
218+ -- folds that may be commonly used or may have an edge in performance in
219+ -- some cases are provided here.
220+ --
221+ -- Useful idioms:
229222 --
230223 -- >>> foldlM' f a = Stream.fold (Fold.foldlM' f a)
231− -- >>> foldl1' f = Stream.fold (Fold.foldl1' f)
224+ -- >>> foldl1 f = Stream.fold (Fold.foldl1' f)
232225 -- >>> foldl' f a = Stream.fold (Fold.foldl' f a)
233226 -- >>> drain = Stream.fold Fold.drain
234227 -- >>> mapM_ f = Stream.fold (Fold.drainMapM f)
235228 -- >>> length = Stream.fold Fold.length
236− -- >>> genericLength = Stream.fold Fold.lengthGeneric
237229 -- >>> head = Stream.fold Fold.one
238− -- >>> last = Stream.fold Fold.latest
239− -- >>> null = Stream.fold Fold.null
240− -- >>> and = Stream.fold Fold.and
241− -- >>> or = Stream.fold Fold.or
242− -- >>> any p = Stream.fold (Fold.any p)
243− -- >>> all p = Stream.fold (Fold.all p)
244− -- >>> sum = Stream.fold Fold.sum
245− -- >>> product = Stream.fold Fold.product
246− -- >>> maximum = Stream.fold Fold.maximum
247− -- >>> maximumBy cmp = Stream.fold (Fold.maximumBy cmp)
248− -- >>> minimum = Stream.fold Fold.minimum
249− -- >>> minimumBy cmp = Stream.fold (Fold.minimumBy cmp)
250− -- >>> elem x = Stream.fold (Fold.elem x)
251− -- >>> notElem x = Stream.fold (Fold.notElem x)
252− -- >>> lookup x = Stream.fold (Fold.lookup x)
253− -- >>> find p = Stream.fold (Fold.find p)
254− -- >>> (!?) i = Stream.fold (Fold.index i)
255− -- >>> genericIndex i = Stream.fold (Fold.indexGeneric i)
256− -- >>> elemIndex x = Stream.fold (Fold.elemIndex x)
257− -- >>> findIndex p = Stream.fold (Fold.findIndex p)
258− --
259− -- Some equivalents of Data.List APIs from the Stream module:
260− --
261− -- >>> head = fmap (fmap fst) . Stream.uncons
262− -- >>> tail = fmap (fmap snd) . Stream.uncons
263− -- >>> tail = Stream.tail -- unreleased API
264− -- >>> init = Stream.init -- unreleased API
265− --
266− -- A Stream based toList fold implementation is provided below because it
267− -- has a better performance compared to the fold.
268−
269− -- Functions in Data.List, missing here:
270− -- unsnoc = Stream.parseBreak (Parser.init Fold.toList)
271− -- genericTake
272− -- genericDrop
273− -- genericSplitAt
274− -- genericReplicate
275230 , toList
276231
277232 -- * Mapping
@@ -325,15 +280,6 @@ module Streamly.Data.Stream
325280 , scan
326281 , postscan
327282 -- XXX postscan1 can be implemented using Monoids or Refolds.
328− -- The following scans from Data.List are not provided.
329− -- XXX scanl
330− -- XXX scanl1
331− -- XXX scanr
332− -- XXX scanr1
333− -- XXX mapAccumL
334− -- XXX mapAccumR
335− -- XXX inits
336− -- XXX tails
337283
338284 -- ** Specific scans
339285 -- Indexing can be considered as a special type of zipping where we zip a
@@ -342,8 +288,6 @@ module Streamly.Data.Stream
342288
343289 -- * Insertion
344290 -- | Add elements to the stream.
345− --
346− -- >>> insert = Stream.insertBy compare
347291
348292 -- Inserting elements is a special case of interleaving/merging streams.
349293 , insertBy
@@ -379,15 +323,12 @@ module Streamly.Data.Stream
379323 -- "Streamly.Internal.Data.Fold" can be used along with 'scanMaybe' to
380324 -- perform stateful filtering operations in general.
381325 --
382− -- Idioms and equivalents of Data.List APIs:
326+ -- Useful idioms:
383327 --
384328 -- >>> deleteBy cmp x = Stream.scanMaybe (Fold.deleteBy cmp x)
385− -- >>> deleteBy = Stream.deleteBy -- unreleased API
386− -- >>> delete = deleteBy (==)
387329 -- >>> findIndices p = Stream.scanMaybe (Fold.findIndices p)
388330 -- >>> elemIndices a = findIndices (== a)
389331 -- >>> uniq = Stream.scanMaybe (Fold.uniqBy (==))
390− -- >>> partition p = Stream.fold (Fold.partition Fold.toList Fold.toList) . fmap (if p then Left else Right)
391332 , scanMaybe
392333 , take
393334 , takeWhile
@@ -395,8 +336,6 @@ module Streamly.Data.Stream
395336 , drop
396337 , dropWhile
397338 , dropWhileM
398− -- XXX write to an array in reverse and then read in reverse
399− -- > dropWhileEnd = reverse . dropWhile p . reverse
400339
401340 -- XXX These are available as scans in folds. We need to check the
402341 -- performance though. If these are common and we need convenient stream
@@ -433,9 +372,6 @@ module Streamly.Data.Stream
433372 -- >>> s = s1 `Stream.append` s2
434373
435374 -- ** Appending
436− -- | Equivalent of Data.List append:
437− --
438− -- >>> (++) = Stream.append
439375 , append
440376
441377 -- ** Interleaving
@@ -446,14 +382,8 @@ module Streamly.Data.Stream
446382 , mergeByM
447383
448384 -- ** Zipping
449− -- | Idioms and equivalents of Data.List APIs:
450− --
451− -- >>> zip = Stream.zipWith (,)
452− -- >>> unzip = Stream.fold (Fold.unzip Fold.toList Fold.toList)
453385 , zipWith
454386 , zipWithM
455− -- XXX zipWith3,4,5,6,7
456− -- XXX unzip3,4,5,6,7
457387 -- , ZipStream (..)
458388
459389 -- ** Cross Product
@@ -486,89 +416,48 @@ module Streamly.Data.Stream
486416 -- One dimension loops are just a special case of nested loops. For
487417 -- example, 'concatMap' can degenerate to a simple map operation:
488418 --
489− -- >>> map f m = Stream.concatMap (\x -> Stream.fromPure (f x)) m
419+ -- > map f m = S.concatMap (\x -> S.fromPure (f x)) m
490420 --
491421 -- Similarly, 'concatMap' can perform filtering by mapping an element to a
492422 -- 'nil' stream:
493423 --
494− -- >>> filter p m = Stream.concatMap (\x -> if p x then Stream.fromPure x else Stream.nil) m
424+ -- > filter p m = S.concatMap (\x -> if p x then S.fromPure x else S.nil) m
495425 --
496− -- Idioms and equivalents of Data.List APIs:
497− --
498− -- >>> concat = Stream.concatMap id
499− -- >>> unlines = Stream.interposeSuffix '\n'
500− -- >>> unlines = Stream.intercalateSuffix Unfold.fromList "\n"
501− -- >>> unwords = Stream.interpose ' '
502− -- >>> unwords = Stream.intercalate Unfold.fromList " "
503426
504427 , concatEffect
505428 , concatMap
506429 , concatMapM
507430
508431 -- * Repeated Fold
509− -- | Idioms and equivalents of Data.List APIs:
432+ -- | Useful idioms:
510433 --
511− -- >>> groupsOf n = Stream.foldMany (Fold.take n Fold.toList)
512− -- >>> groupBy eq = Stream.groupsWhile eq Fold.toList
513− -- >>> groupBy eq = Stream.parseMany (Parser.groupBy eq Fold.toList)
514− -- >>> groupsByRolling eq = Stream.parseMany (Parser.groupByRolling eq Fold.toList)
515− -- >>> groups = groupBy (==)
434+ -- >>> splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
435+ -- >>> splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
436+ -- >>> groupsBy eq f = Stream.parseMany (Parser.groupBy eq f)
437+ -- >>> groupsByRolling eq f = Stream.parseMany (Parser.groupByRolling eq f)
438+ -- >>> groupsOf n f = Stream.foldMany (Fold.take n f)
516439 , foldMany -- XXX Rename to foldRepeat
517440 , groupsOf
518441 , parseMany
519442
520443 -- * Splitting
521− -- | Idioms and equivalents of Data.List APIs:
522− --
523− -- >>> splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
524− -- >>> splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
525− -- >>> lines = splitOnSuffix (== '\n')
526− -- >>> words = Stream.wordsBy isSpace
527− -- >>> splitAt n = Stream.fold (Fold.splitAt n Fold.toList Fold.toList)
528− -- >>> span p = Parser.splitWith (,) (Parser.takeWhile p Fold.toList) (Parser.fromFold Fold.toList)
529− -- >>> break p = span (not . p)
530444 , splitOn
531445 , wordsBy
532446
533447 -- * Buffered Operations
534448 -- | Operations that require buffering of the stream.
535449 -- Reverse is essentially a left fold followed by an unfold.
536− --
537− -- Idioms and equivalents of Data.List APIs:
538− --
539− -- >>> nub = Stream.fold Fold.toList . Stream.scanMaybe Fold.nub
540− -- >>> nub = Stream.ordNub -- unreleased API
541− -- >>> sortBy = StreamK.sortBy
542− -- >>> sortOn f = StreamK.sortOn -- unreleased API
543− -- >>> deleteFirstsBy = Stream.deleteFirstsBy -- unreleased
544− -- >>> (\\) = Stream.deleteFirstsBy (==) -- unreleased
545− -- >>> intersectBy = Stream.intersectBy -- unreleased
546− -- >>> intersect = Stream.intersectBy (==) -- unreleased
547− -- >>> unionBy = Stream.unionBy -- unreleased
548− -- >>> union = Stream.unionBy (==) -- unreleased
549− --
550450 , reverse
551− -- XXX transpose: write the streams to arrays and then stream transposed.
552− -- XXX subsequences
553− -- XXX permutations
554− -- , nub
555− -- , ordNub
556− -- , nubBy
557451
558452 -- * Multi-Stream folds
559453 -- | Operations that consume multiple streams at the same time.
560454 , eqBy
561455 , cmpBy
562456 , isPrefixOf
563− , isInfixOf
564− -- , isSuffixOf
565− -- , isSuffixOfUnbox
566457 , isSubsequenceOf
567458
568459 -- trimming sequences
569− -- , stripPrefix
570− -- , stripSuffix
571− -- , stripSuffixUnbox
460+ , stripPrefix
572461
573462 -- Exceptions and resource management depend on the "exceptions" package
574463 -- XXX We can have IO Stream operations not depending on "exceptions"
@@ -622,20 +511,25 @@ module Streamly.Data.Stream
622511 , runReaderT
623512 , runStateT
624513
625− -- XXX Arrays could be different types, therefore, this should be in
626− -- specific array module. Or maybe we should abstract over array types.
627514 -- * Stream of Arrays
628515 , Array.chunksOf
516+ , chunksSplitOn
629517 )
630518where
631519
520+import Control.Monad.IO.Class (MonadIO)
521+import Data.Word (Word8)
522+import Streamly.Data.Array (Array)
632523import Streamly.Internal.Data.Stream
633524import Prelude
634525 hiding (filter, drop, dropWhile, take, takeWhile, zipWith, foldr,
635− mapM, sequence, reverse, iterate, foldr1, repeat, replicate,
636− concatMap)
526+ foldl, map, mapM, mapM_, sequence, all, any, sum, product, elem,
527+ notElem, maximum, minimum, head, last, tail, length, null,
528+ reverse, iterate, init, and, or, lookup, foldr1, (!!),
529+ scanl, scanl1, repeat, replicate, concatMap, span)
637530
638531import qualified Streamly.Internal.Data.Array.Type as Array
532+import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
639533
640534#include "DocTestDataStream.hs"
641535
@@ -744,3 +638,14 @@ import qualified Streamly.Internal.Data.Array.Type as Array
744638--
745639-- 'Stream' and 'StreamK' types can be interconverted. See
746640-- "Streamly.Data.StreamK" module for conversion operations.
641+
642+-- | Split a stream of arrays on a given separator byte, dropping the separator
643+-- and coalescing all the arrays between two separators into a single array.
644+--
645+{-# INLINE chunksSplitOn #-}
646+chunksSplitOn
647+ :: (MonadIO m)
648+ => Word8
649+ -> Stream m (Array Word8)
650+ -> Stream m (Array Word8)
651+chunksSplitOn = ArrayStream.splitOn
core/src/Streamly/Data/StreamK.hs+1−2
@@ -95,8 +95,7 @@ module Streamly.Data.StreamK
9595 -- , foldBreak
9696
9797 -- ** Parsing
98− , parse
99− , parseBreak
98+ -- , parseBreak
10099 , parseBreakChunks
101100 , parseChunks
102101
core/src/Streamly/Data/Unfold.hs−2
@@ -65,8 +65,6 @@ module Streamly.Data.Unfold
6565 -- ** Mapping on Input
6666 , lmap
6767 , lmapM
68− , first
69− , second
7068
7169 -- ** Mapping on Output
7270 , mapM
core/src/Streamly/FileSystem/Handle.hs+30−7
@@ -1,4 +1,5 @@
11{-# LANGUAGE CPP #-}
2+#include "inline.hs"
23
34-- |
45-- Module : Streamly.FileSystem.Handle
@@ -81,13 +82,13 @@ module Streamly.FileSystem.Handle
8182 -- position of the file handle. The stream ends as soon as EOF is
8283 -- encountered.
8384
84− -- *** Streams
85− , read
86− , readWith
87− , readChunks
88− , readChunksWith
85+ -- -- *** Streams
86+ -- , read
87+ -- , readWith
88+ -- , readChunks
89+ -- , readChunksWith
8990
90− -- *** Unfolds
91+ -- -- *** Unfolds
9192 , reader
9293 , readerWith
9394 , chunkReader
@@ -104,14 +105,36 @@ module Streamly.FileSystem.Handle
104105 , writeChunks
105106
106107 -- * Deprecated
108+ , read
107109 , readWithBufferOf
110+ , readChunks
108111 , readChunksWithBufferOf
109112 , writeChunksWithBufferOf
110113 , writeWithBufferOf
111114 )
112115where
113116
114−import Streamly.Internal.FileSystem.Handle
117+import Control.Monad.IO.Class (MonadIO(..))
118+import Data.Word (Word8)
119+import Streamly.Internal.Data.Array.Type (Array)
120+import Streamly.Internal.Data.Unfold.Type (Unfold)
121+import System.IO (Handle)
122+
123+import Streamly.Internal.FileSystem.Handle hiding (read, readChunks)
115124import Prelude hiding (read)
116125
117126#include "DocTestFileSystemHandle.hs"
127+
128+-- | Same as 'reader'
129+--
130+{-# DEPRECATED read "Please use 'reader' instead" #-}
131+{-# INLINE read #-}
132+read :: MonadIO m => Unfold m Handle Word8
133+read = reader
134+
135+-- | Same as 'chunkReader'
136+--
137+{-# DEPRECATED readChunks "Please use 'chunkReader' instead" #-}
138+{-# INLINE readChunks #-}
139+readChunks :: MonadIO m => Unfold m Handle (Array Word8)
140+readChunks = chunkReader
core/src/Streamly/Internal/Data/Array.hs+135−608
@@ -2,6 +2,7 @@
22-- |
33-- Module : Streamly.Internal.Data.Array
44-- Copyright : (c) 2019 Composewell Technologies
5+--
56-- License : BSD3
67-- Maintainer : streamly@composewell.com
78-- Stability : experimental
@@ -19,21 +20,33 @@ module Streamly.Internal.Data.Array
1920 module Streamly.Internal.Data.Array.Type
2021
2122 -- * Construction
23+ -- Stream Folds
24+ , fromStreamN
25+ , fromStream
26+
2227 -- Monadic Folds
2328 , writeLastN
2429
30+ -- * Elimination
31+ -- ** Conversion
32+
33+ -- ** Unfolds
34+ , reader
35+ , readerUnsafe
36+ , producer -- experimental
37+
2538 -- * Random Access
2639 -- , (!!)
2740 , getIndex
2841 , getIndexRev
29− , last -- XXX getLastIndex?
42+ , last -- XXX getIndexLast?
43+ , getIndices
44+ , getIndicesFromThenTo
3045 -- , getIndicesFrom -- read from a given position to the end of file
3146 -- , getIndicesUpto -- read from beginning up to the given position
3247 -- , getIndicesFromTo
3348 -- , getIndicesFromRev -- read from a given position to the beginning of file
3449 -- , getIndicesUptoRev -- read from end to the given position in file
35− , indexReader
36− , indexReaderFromThenTo
3750
3851 -- * Size
3952 , null
@@ -41,8 +54,6 @@ module Streamly.Internal.Data.Array
4154 -- * Search
4255 , binarySearch
4356 , findIndicesOf
44− -- getIndicesOf
45− , indexFinder -- see splitOn
4657 -- , findIndexOf
4758 -- , find
4859
@@ -55,89 +66,51 @@ module Streamly.Internal.Data.Array
5566 -- * Subarrays
5667 , getSliceUnsafe
5768 -- , getSlice
58− , sliceIndexerFromLen
59− , slicerFromLen
69+ , genSlicesFromLen
70+ , getSlicesFromLen
6071 , splitOn
6172
6273 -- * Streaming Operations
6374 , streamTransform
6475
65− -- * Folding
76+ -- ** Folding
6677 , streamFold
6778 , fold
68−
69− -- * Stream of Arrays
70−
71− -- XXX these are probably not very useful to have in this module as we can
72− -- express these idiomatically using streams.
73− , interpose
74− , interposeSuffix
75− , intercalateSuffix
76−
77− , compactLE
78− , pinnedCompactLE
79− , compactOnByte
80− , compactOnByteSuffix
81−
82− , foldBreakChunks
83− , foldChunks
84− , foldBreakChunksK
85− , parseBreakChunksK
86−
87− -- * Serialization
88− , encodeAs
89− , serialize
90− , pinnedSerialize
91− , deserialize
92−
93− -- * Deprecated
94− , genSlicesFromLen
95− , getSlicesFromLen
96− , getIndices
9779 )
9880where
9981
100−#include "assert.hs"
10182#include "inline.hs"
10283#include "ArrayMacros.h"
10384
85+import Control.Exception (assert)
86+import Control.Monad (when)
10487import Control.Monad.IO.Class (MonadIO(..))
105−-- import Data.Bifunctor (first)
106−-- import Data.Either (fromRight)
10788import Data.Functor.Identity (Identity)
10889import Data.Proxy (Proxy(..))
10990import Data.Word (Word8)
11091import Foreign.C.String (CString)
11192import Foreign.Ptr (castPtr)
11293import Foreign.Storable (Storable)
113−import GHC.Types (SPEC(..))
11494import Streamly.Internal.Data.Unbox (Unbox(..))
11595import Prelude hiding (length, null, last, map, (!!), read, concat)
11696
117−import Streamly.Internal.Data.MutByteArray.Type (PinnedState(..), MutByteArray)
118−import Streamly.Internal.Data.Serialize.Type (Serialize)
11997import Streamly.Internal.Data.Fold.Type (Fold(..))
120−import Streamly.Internal.Data.Parser (Parser(..), Initial(..), ParseError(..))
121−import Streamly.Internal.Data.Stream (Stream(..))
122−import Streamly.Internal.Data.StreamK (StreamK)
123−import Streamly.Internal.Data.SVar.Type (adaptState, defState)
124−import Streamly.Internal.Data.Tuple.Strict (Tuple'(..), Tuple3Fused'(..))
98+import Streamly.Internal.Data.Producer.Type (Producer(..))
99+import Streamly.Internal.Data.Stream (Stream)
100+import Streamly.Internal.Data.Tuple.Strict (Tuple3Fused'(..))
125101import Streamly.Internal.Data.Unfold.Type (Unfold(..))
126102import Streamly.Internal.System.IO (unsafeInlineIO)
127103
128−import qualified Streamly.Internal.Data.Fold.Type as Fold
129−import qualified Streamly.Internal.Data.Serialize.Type as Serialize
130−import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
104+import qualified Streamly.Internal.Data.MutArray.Type as MA
131105import qualified Streamly.Internal.Data.MutArray as MA
106+import qualified Streamly.Internal.Data.Array.Type as A
132107import qualified Streamly.Internal.Data.Fold as FL
108+import qualified Streamly.Internal.Data.Producer.Type as Producer
109+import qualified Streamly.Internal.Data.Producer as Producer
133110import qualified Streamly.Internal.Data.Ring as RB
134−import qualified Streamly.Internal.Data.Parser as Parser
135−-- import qualified Streamly.Internal.Data.ParserK as ParserK
136111import qualified Streamly.Internal.Data.Stream as D
137112import qualified Streamly.Internal.Data.Stream as Stream
138−import qualified Streamly.Internal.Data.StreamK as StreamK
139113import qualified Streamly.Internal.Data.Unfold as Unfold
140−import qualified Prelude
141114
142115import Streamly.Internal.Data.Array.Type
143116
@@ -166,10 +139,82 @@ import Streamly.Internal.Data.Array.Type
166139-- if we use one array for each chunk. This is still significant to add
167140-- pressure to GC.
168141
142+-------------------------------------------------------------------------------
143+-- Construction
144+-------------------------------------------------------------------------------
145+
146+-- | Create an 'Array' from the first N elements of a stream. The array is
147+-- allocated to size N, if the stream terminates before N elements then the
148+-- array may hold less than N elements.
149+--
150+-- /Pre-release/
151+{-# INLINE fromStreamN #-}
152+fromStreamN :: (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
153+fromStreamN n m = do
154+ when (n < 0) $ error "writeN: negative write count specified"
155+ A.fromStreamDN n m
156+
157+-- | Create an 'Array' from a stream. This is useful when we want to create a
158+-- single array from a stream of unknown size. 'writeN' is at least twice
159+-- as efficient when the size is already known.
160+--
161+-- Note that if the input stream is too large memory allocation for the array
162+-- may fail. When the stream size is not known, `chunksOf` followed by
163+-- processing of indvidual arrays in the resulting stream should be preferred.
164+--
165+-- /Pre-release/
166+{-# INLINE fromStream #-}
167+fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (Array a)
168+fromStream = Stream.fold A.write
169+-- write m = A.fromStreamD $ D.fromStreamK m
170+
169171-------------------------------------------------------------------------------
170172-- Elimination
171173-------------------------------------------------------------------------------
172174
175+{-# INLINE_NORMAL producer #-}
176+producer :: forall m a. (Monad m, Unbox a) => Producer m (Array a) a
177+producer =
178+ Producer.translate A.unsafeThaw A.unsafeFreeze
179+ $ MA.producerWith (return . unsafeInlineIO)
180+
181+-- | Unfold an array into a stream.
182+--
183+{-# INLINE_NORMAL reader #-}
184+reader :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
185+reader = Producer.simplify producer
186+
187+-- | Unfold an array into a stream, does not check the end of the array, the
188+-- user is responsible for terminating the stream within the array bounds. For
189+-- high performance application where the end condition can be determined by
190+-- a terminating fold.
191+--
192+-- Written in the hope that it may be faster than "read", however, in the case
193+-- for which this was written, "read" proves to be faster even though the core
194+-- generated with unsafeRead looks simpler.
195+--
196+-- /Pre-release/
197+--
198+{-# INLINE_NORMAL readerUnsafe #-}
199+readerUnsafe :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
200+readerUnsafe = Unfold step inject
201+ where
202+
203+ inject (Array contents start end) =
204+ return (ArrayUnsafe contents end start)
205+
206+ {-# INLINE_LATE step #-}
207+ step (ArrayUnsafe contents end p) = do
208+ -- unsafeInlineIO allows us to run this in Identity monad for pure
209+ -- toList/foldr case which makes them much faster due to not
210+ -- accumulating the list and fusing better with the pure consumers.
211+ --
212+ -- This should be safe as the array contents are guaranteed to be
213+ -- evaluated/written to before we peek at them.
214+ let !x = unsafeInlineIO $ peekByteIndex p contents
215+ let !p1 = INDEX_NEXT(p,a)
216+ return $ D.Yield x (ArrayUnsafe contents end p1)
217+
173218-- |
174219--
175220-- >>> null arr = Array.byteLength arr == 0
@@ -177,7 +222,7 @@ import Streamly.Internal.Data.Array.Type
177222-- /Pre-release/
178223{-# INLINE null #-}
179224null :: Array a -> Bool
180−null arr = byteLength arr == 0
225+null arr = A.byteLength arr == 0
181226
182227-- | Like 'getIndex' but indexes the array in reverse from the end.
183228--
@@ -189,7 +234,7 @@ getIndexRev i arr =
189234 $ do
190235 let elemPtr = RINDEX_OF(arrEnd arr, i, a)
191236 if i >= 0 && elemPtr >= arrStart arr
192− then Just <$> peekAt elemPtr (arrContents arr)
237+ then Just <$> peekByteIndex elemPtr (arrContents arr)
193238 else return Nothing
194239
195240-- |
@@ -205,8 +250,6 @@ last = getIndexRev 0
205250-- Folds with Array as the container
206251-------------------------------------------------------------------------------
207252
208−-- XXX We should generate this from Ring.
209−
210253-- | @writeLastN n@ folds a maximum of @n@ elements from the end of the input
211254-- stream to an 'Array'.
212255--
@@ -215,7 +258,7 @@ writeLastN ::
215258 (Storable a, Unbox a, MonadIO m) => Int -> Fold m a (Array a)
216259writeLastN n
217260 | n <= 0 = fmap (const mempty) FL.drain
218− | otherwise = unsafeFreeze <$> Fold step initial done done
261+ | otherwise = A.unsafeFreeze <$> Fold step initial done
219262
220263 where
221264
@@ -228,9 +271,11 @@ writeLastN n
228271 in fmap f $ liftIO $ RB.new n
229272
230273 done (Tuple3Fused' rb rh i) = do
231− arr <- MA.new n
232− -- XXX We should write a read unfold for ring.
233− foldFunc i rh MA.snocUnsafe arr rb
274+ arr <- liftIO $ MA.new n
275+ foldFunc i rh snoc' arr rb
276+
277+ -- XXX We should write a read unfold for ring.
278+ snoc' b a = liftIO $ MA.snocUnsafe b a
234279
235280 foldFunc i
236281 | i < n = RB.unsafeFoldRingM
@@ -256,26 +301,19 @@ binarySearch = undefined
256301-- compared to streams by using SIMD instructions.
257302-- We can also return a bit array instead.
258303
259−-- Can use SIMD.
260−
261304-- | Perform a linear search to find all the indices where a given element is
262305-- present in an array.
263306--
264307-- /Unimplemented/
265−indexFinder :: (a -> Bool) -> Unfold Identity (Array a) Int
266−indexFinder = undefined
267−
268−-- |
269−-- /Unimplemented/
270−findIndicesOf :: (a -> Bool) -> Array a -> Stream Identity Int
271−findIndicesOf p = Stream.unfold (indexFinder p)
308+findIndicesOf :: (a -> Bool) -> Unfold Identity (Array a) Int
309+findIndicesOf = undefined
272310
273311{-
274312findIndexOf :: (a -> Bool) -> Array a -> Maybe Int
275−findIndexOf p = Unfold.fold Fold.one . Stream.unfold (indexFinder p)
313+findIndexOf p = Unfold.fold Fold.one . Stream.unfold (findIndicesOf p)
276314
277315find :: (a -> Bool) -> Array a -> Bool
278−find = Unfold.fold Fold.null . Stream.unfold (indexFinder p)
316+find = Unfold.fold Fold.null . Stream.unfold (findIndicesOf p)
279317-}
280318
281319-------------------------------------------------------------------------------
@@ -317,45 +355,29 @@ splitOn :: (Monad m, Unbox a) =>
317355 (a -> Bool) -> Array a -> Stream m (Array a)
318356splitOn predicate arr =
319357 fmap (\(i, len) -> getSliceUnsafe i len arr)
320− $ D.indexOnSuffix predicate (read arr)
321−
322−{-# INLINE sliceIndexerFromLen #-}
323−sliceIndexerFromLen :: forall m a. (Monad m, Unbox a)
324− => Int -- ^ from index
325− -> Int -- ^ length of the slice
326− -> Unfold m (Array a) (Int, Int)
327−sliceIndexerFromLen from len =
328− Unfold.lmap unsafeThaw (MA.sliceIndexerFromLen from len)
358+ $ D.sliceOnSuffix predicate (A.toStreamD arr)
329359
330−{-# DEPRECATED genSlicesFromLen "Please use sliceIndexerFromLen instead." #-}
331360{-# INLINE genSlicesFromLen #-}
332361genSlicesFromLen :: forall m a. (Monad m, Unbox a)
333362 => Int -- ^ from index
334363 -> Int -- ^ length of the slice
335364 -> Unfold m (Array a) (Int, Int)
336−genSlicesFromLen = sliceIndexerFromLen
365+genSlicesFromLen from len =
366+ Unfold.lmap A.unsafeThaw (MA.genSlicesFromLen from len)
337367
338368-- | Generate a stream of slices of specified length from an array, starting
339369-- from the supplied array index. The last slice may be shorter than the
340370-- requested length.
341371--
342372-- /Pre-release//
343−{-# INLINE slicerFromLen #-}
344−slicerFromLen :: forall m a. (Monad m, Unbox a)
345− => Int -- ^ from index
346− -> Int -- ^ length of the slice
347− -> Unfold m (Array a) (Array a)
348−slicerFromLen from len =
349− fmap unsafeFreeze
350− $ Unfold.lmap unsafeThaw (MA.slicerFromLen from len)
351−
352−{-# DEPRECATED getSlicesFromLen "Please use slicerFromLen instead." #-}
353373{-# INLINE getSlicesFromLen #-}
354374getSlicesFromLen :: forall m a. (Monad m, Unbox a)
355375 => Int -- ^ from index
356376 -> Int -- ^ length of the slice
357377 -> Unfold m (Array a) (Array a)
358−getSlicesFromLen = slicerFromLen
378+getSlicesFromLen from len =
379+ fmap A.unsafeFreeze
380+ $ Unfold.lmap A.unsafeThaw (MA.getSlicesFromLen from len)
359381
360382-------------------------------------------------------------------------------
361383-- Random reads and writes
@@ -373,7 +395,7 @@ getIndex i arr =
373395 $ do
374396 let elemPtr = INDEX_OF(arrStart arr, i, a)
375397 if i >= 0 && INDEX_VALID(elemPtr, arrEnd arr, a)
376− then Just <$> peekAt elemPtr (arrContents arr)
398+ then Just <$> peekByteIndex elemPtr (arrContents arr)
377399 else return Nothing
378400
379401-- | Given a stream of array indices, read the elements on those indices from
@@ -385,25 +407,19 @@ getIndex i arr =
385407-- @
386408-- read =
387409-- let u = lmap (\arr -> (0, length arr - 1)) Unfold.enumerateFromTo
388−-- in Unfold.lmap f (indexReader arr)
410+-- in Unfold.lmap f (getIndices arr)
389411--
390412-- readRev =
391413-- let i = length arr - 1
392−-- in Unfold.lmap f (indexReaderFromThenTo i (i - 1) 0)
414+-- in Unfold.lmap f (getIndicesFromThenTo i (i - 1) 0)
393415-- @
394416--
395417-- /Pre-release/
396−{-# INLINE indexReader #-}
397−indexReader :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
398−indexReader m =
399− let unf = MA.indexReaderWith (return . unsafeInlineIO) m
400− in Unfold.lmap unsafeThaw unf
401−
402−-- XXX DO NOT REMOVE, change the signature to use Stream instead of unfold
403−{-# DEPRECATED getIndices "Please use getIndices instead." #-}
404418{-# INLINE getIndices #-}
405419getIndices :: (Monad m, Unbox a) => Stream m Int -> Unfold m (Array a) a
406−getIndices = indexReader
420+getIndices m =
421+ let unf = MA.getIndicesD (return . unsafeInlineIO) m
422+ in Unfold.lmap A.unsafeThaw unf
407423
408424-- | Unfolds @(from, then, to, array)@ generating a finite stream whose first
409425-- element is the array value from the index @from@ and the successive elements
@@ -419,9 +435,9 @@ getIndices = indexReader
419435-- @
420436--
421437-- /Unimplemented/
422−{-# INLINE indexReaderFromThenTo #-}
423−indexReaderFromThenTo :: Unfold m (Int, Int, Int, Array a) a
424−indexReaderFromThenTo = undefined
438+{-# INLINE getIndicesFromThenTo #-}
439+getIndicesFromThenTo :: Unfold m (Int, Int, Int, Array a) a
440+getIndicesFromThenTo = undefined
425441
426442-------------------------------------------------------------------------------
427443-- Transform via stream operations
@@ -438,7 +454,7 @@ indexReaderFromThenTo = undefined
438454{-# INLINE runPipe #-}
439455runPipe :: (MonadIO m, Unbox a, Unbox b)
440456 => Pipe m a b -> Array a -> m (Array b)
441−runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (read arr)
457+runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (A.read arr)
442458-}
443459
444460-- XXX For transformations that cannot change the number of elements e.g. "map"
@@ -452,7 +468,7 @@ runPipe f arr = P.runPipe (toArrayMinChunk (length arr)) $ f (read arr)
452468streamTransform :: forall m a b. (MonadIO m, Unbox a, Unbox b)
453469 => (Stream m a -> Stream m b) -> Array a -> m (Array b)
454470streamTransform f arr =
455− Stream.fold (createWith (length arr)) $ f (read arr)
471+ Stream.fold (A.writeWith (length arr)) $ f (A.read arr)
456472
457473-------------------------------------------------------------------------------
458474-- Casts
@@ -486,7 +502,7 @@ asBytes = castUnsafe
486502--
487503cast :: forall a b. (Unbox b) => Array a -> Maybe (Array b)
488504cast arr =
489− let len = byteLength arr
505+ let len = A.byteLength arr
490506 r = len `mod` SIZE_OF(b)
491507 in if r /= 0
492508 then Nothing
@@ -504,9 +520,9 @@ cast arr =
504520--
505521asCStringUnsafe :: Array a -> (CString -> IO b) -> IO b
506522asCStringUnsafe arr act = do
507− let arr1 = asBytes arr <> fromList [0]
508− -- unsafePinnedAsPtr makes sure the array is pinned
509− unsafePinnedAsPtr arr1 $ \ptr -> act (castPtr ptr)
523+ let arr1 = asBytes arr <> A.fromList [0]
524+ -- asPtrUnsafe makes sure the array is pinned
525+ asPtrUnsafe arr1 $ \ptr -> act (castPtr ptr)
510526
511527-------------------------------------------------------------------------------
512528-- Folds
@@ -519,500 +535,11 @@ asCStringUnsafe arr act = do
519535-- /Pre-release/
520536{-# INLINE fold #-}
521537fold :: forall m a b. (Monad m, Unbox a) => Fold m a b -> Array a -> m b
522−fold f arr = Stream.fold f (read arr)
538+fold f arr = Stream.fold f (A.read arr)
523539
524540-- | Fold an array using a stream fold operation.
525541--
526542-- /Pre-release/
527543{-# INLINE streamFold #-}
528544streamFold :: (Monad m, Unbox a) => (Stream m a -> m b) -> Array a -> m b
529−streamFold f arr = f (read arr)
530−
531−--------------------------------------------------------------------------------
532−-- Serialization
533−--------------------------------------------------------------------------------
534−
535−{-# INLINE encodeAs #-}
536−encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
537−encodeAs ps a =
538− unsafeInlineIO $ do
539− let len = Serialize.addSizeTo 0 a
540− mbarr <- MBA.newBytesAs ps len
541− off <- Serialize.serializeAt 0 mbarr a
542− assertM(len == off)
543− pure $ Array mbarr 0 off
544−
545−-- |
546−-- Properties:
547−-- 1. Identity: @deserialize . serialize == id@
548−-- 2. Encoded equivalence: @serialize a == serialize a@
549−{-# INLINE serialize #-}
550−serialize :: Serialize a => a -> Array Word8
551−serialize = encodeAs Unpinned
552−
553−-- | Serialize a Haskell type to a pinned byte array. The array is allocated
554−-- using pinned memory so that it can be used directly in OS APIs for writing
555−-- to file or sending over the network.
556−--
557−-- Properties:
558−-- 1. Identity: @deserialize . pinnedSerialize == id@
559−-- 2. Encoded equivalence: @pinnedSerialize a == pinnedSerialize a@
560−{-# INLINE pinnedSerialize #-}
561−pinnedSerialize :: Serialize a => a -> Array Word8
562−pinnedSerialize = encodeAs Pinned
563−
564−-- | Decode a Haskell type from a byte array containing its serialized
565−-- representation.
566−{-# INLINE deserialize #-}
567−deserialize :: Serialize a => Array Word8 -> a
568−deserialize arr@(Array {..}) = unsafeInlineIO $ do
569− let lenArr = length arr
570− (off, val) <-
571− Serialize.deserializeAt arrStart arrContents (arrStart + lenArr)
572− assertM(off == arrStart + lenArr)
573− pure val
574−
575−-------------------------------------------------------------------------------
576−-- Streams of Arrays
577−-------------------------------------------------------------------------------
578−
579−-- TODO: efficiently compare two streams of arrays. Two streams can have chunks
580−-- of different sizes, we can handle that in the stream comparison abstraction.
581−-- This could be useful e.g. to fast compare whether two files differ.
582−
583−-- | Insert the given element between arrays and flatten.
584−--
585−-- >>> interpose x = Stream.interpose x Array.reader
586−--
587−{-# INLINE interpose #-}
588−interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
589−interpose x = D.interpose x reader
590−
591−data FlattenState s =
592− OuterLoop s
593− | InnerLoop s !MutByteArray !Int !Int
594−
595−-- | Insert the given element after each array and flatten. This is similar to
596−-- unlines.
597−--
598−-- >>> interposeSuffix x = Stream.interposeSuffix x Array.reader
599−--
600−{-# INLINE_NORMAL interposeSuffix #-}
601−interposeSuffix :: forall m a. (Monad m, Unbox a)
602− => a -> Stream m (Array a) -> Stream m a
603−-- This does not require MonadIO constraint.
604−-- interposeSuffix x = D.interposeSuffix x reader
605−interposeSuffix sep (D.Stream step state) = D.Stream step' (OuterLoop state)
606−
607− where
608−
609− {-# INLINE_LATE step' #-}
610− step' gst (OuterLoop st) = do
611− r <- step (adaptState gst) st
612− return $ case r of
613− D.Yield Array{..} s ->
614− D.Skip (InnerLoop s arrContents arrStart arrEnd)
615− D.Skip s -> D.Skip (OuterLoop s)
616− D.Stop -> D.Stop
617−
618− step' _ (InnerLoop st _ p end) | p == end =
619− return $ D.Yield sep $ OuterLoop st
620−
621− step' _ (InnerLoop st contents p end) = do
622− x <- pure $ unsafeInlineIO $ peekAt p contents
623− return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
624−
625−-- | Insert the given array after each array and flatten.
626−--
627−-- >>> intercalateSuffix = Stream.intercalateSuffix Array.reader
628−--
629−{-# INLINE intercalateSuffix #-}
630−intercalateSuffix :: (Monad m, Unbox a)
631− => Array a -> Stream m (Array a) -> Stream m a
632−intercalateSuffix = D.intercalateSuffix reader
633−
634−-- | @compactLE n@ coalesces adjacent arrays in the input stream
635−-- only if the combined size would be less than or equal to n.
636−--
637−-- Generates unpinned arrays irrespective of the pinning status of input
638−-- arrays.
639−{-# INLINE_NORMAL compactLE #-}
640−compactLE :: (MonadIO m, Unbox a)
641− => Int -> Stream m (Array a) -> Stream m (Array a)
642−compactLE n stream =
643− D.map unsafeFreeze $ MA.compactLE n $ D.map unsafeThaw stream
644−
645−-- | Pinned version of 'compactLE'.
646−{-# INLINE_NORMAL pinnedCompactLE #-}
647−pinnedCompactLE :: (MonadIO m, Unbox a)
648− => Int -> Stream m (Array a) -> Stream m (Array a)
649−pinnedCompactLE n stream =
650− D.map unsafeFreeze $ MA.pinnedCompactLE n $ D.map unsafeThaw stream
651−
652−-- | Split a stream of arrays on a given separator byte, dropping the separator
653−-- and coalescing all the arrays between two separators into a single array.
654−--
655−{-# INLINE compactOnByte #-}
656−compactOnByte
657− :: (MonadIO m)
658− => Word8
659− -> Stream m (Array Word8)
660− -> Stream m (Array Word8)
661−compactOnByte byte =
662− fmap unsafeFreeze . MA.compactOnByte byte . fmap unsafeThaw
663−
664−-- | Like 'compactOnByte' considers the separator in suffix position instead of
665−-- infix position.
666−{-# INLINE compactOnByteSuffix #-}
667−compactOnByteSuffix
668− :: (MonadIO m)
669− => Word8
670− -> Stream m (Array Word8)
671− -> Stream m (Array Word8)
672−compactOnByteSuffix byte =
673− fmap unsafeFreeze . MA.compactOnByteSuffix byte . fmap unsafeThaw
674−
675−-------------------------------------------------------------------------------
676−-- Folding Streams of Arrays
677−-------------------------------------------------------------------------------
678−
679−-- XXX This should not be used for breaking a stream as the D.cons used in
680−-- reconstructing the stream could be very bad for performance. This can only
681−-- be useful in folding without breaking.
682−{-# INLINE_NORMAL foldBreakChunks #-}
683−foldBreakChunks :: forall m a b. (MonadIO m, Unbox a) =>
684− Fold m a b -> Stream m (Array a) -> m (b, Stream m (Array a))
685−foldBreakChunks (Fold fstep initial _ final) stream@(Stream step state) = do
686− res <- initial
687− case res of
688− Fold.Partial fs -> go SPEC state fs
689− Fold.Done fb -> return $! (fb, stream)
690−
691− where
692−
693− {-# INLINE go #-}
694− go !_ st !fs = do
695− r <- step defState st
696− case r of
697− Stream.Yield (Array contents start end) s ->
698− let fp = Tuple' end contents
699− in goArray SPEC s fp start fs
700− Stream.Skip s -> go SPEC s fs
701− Stream.Stop -> do
702− b <- final fs
703− return (b, D.nil)
704−
705− goArray !_ s (Tuple' end _) !cur !fs
706− | cur == end = do
707− go SPEC s fs
708− goArray !_ st fp@(Tuple' end contents) !cur !fs = do
709− x <- liftIO $ peekAt cur contents
710− res <- fstep fs x
711− let next = INDEX_NEXT(cur,a)
712− case res of
713− Fold.Done b -> do
714− let arr = Array contents next end
715− return $! (b, D.cons arr (D.Stream step st))
716− Fold.Partial fs1 -> goArray SPEC st fp next fs1
717−
718−-- This may be more robust wrt fusion compared to unfoldMany?
719−
720−-- | Fold a stream of arrays using a 'Fold'. This is equivalent to the
721−-- following:
722−--
723−-- >>> foldChunks f = Stream.fold f . Stream.unfoldMany Array.reader
724−--
725−foldChunks :: (MonadIO m, Unbox a) => Fold m a b -> Stream m (Array a) -> m b
726−foldChunks f s = fmap fst (foldBreakChunks f s)
727−-- foldStream f = Stream.fold f . Stream.unfoldMany reader
728−
729−-- | Fold a stream of arrays using a 'Fold' and return the remaining stream.
730−--
731−-- The following alternative to this function allows composing the fold using
732−-- the parser Monad:
733−--
734−-- @
735−-- foldBreakStreamK f s =
736−-- fmap (first (fromRight undefined))
737−-- $ StreamK.parseBreakChunks (ParserK.adaptC (Parser.fromFold f)) s
738−-- @
739−--
740−-- We can compare perf and remove this one or define it in terms of that.
741−--
742−foldBreakChunksK :: forall m a b. (MonadIO m, Unbox a) =>
743− Fold m a b -> StreamK m (Array a) -> m (b, StreamK m (Array a))
744−{-
745−foldBreakChunksK f s =
746− fmap (first (fromRight undefined))
747− $ StreamK.parseBreakChunks (ParserK.adaptC (Parser.fromFold f)) s
748−-}
749−foldBreakChunksK (Fold fstep initial _ final) stream = do
750− res <- initial
751− case res of
752− Fold.Partial fs -> go fs stream
753− Fold.Done fb -> return (fb, stream)
754−
755− where
756−
757− {-# INLINE go #-}
758− go !fs st = do
759− let stop = (, StreamK.nil) <$> final fs
760− single a = yieldk a StreamK.nil
761− yieldk (Array contents start end) r =
762− let fp = Tuple' end contents
763− in goArray fs r fp start
764− in StreamK.foldStream defState yieldk single stop st
765−
766− goArray !fs st (Tuple' end _) !cur
767− | cur == end = do
768− go fs st
769− goArray !fs st fp@(Tuple' end contents) !cur = do
770− x <- liftIO $ peekAt cur contents
771− res <- fstep fs x
772− let next = INDEX_NEXT(cur,a)
773− case res of
774− Fold.Done b -> do
775− let arr = Array contents next end
776− return $! (b, StreamK.cons arr st)
777− Fold.Partial fs1 -> goArray fs1 st fp next
778−
779−{-
780−-- This can be generalized to any type provided it can be unfolded to a stream
781−-- and it can be combined using a semigroup operation.
782−--
783−{-# INLINE_NORMAL parseBreakD #-}
784−parseBreakD ::
785− forall m a b. (MonadIO m, MonadThrow m, Unbox a)
786− => PRD.Parser a m b
787− -> D.Stream m (Array.Array a)
788− -> m (b, D.Stream m (Array.Array a))
789−parseBreakD
790− (PRD.Parser pstep initial extract) stream@(D.Stream step state) = do
791−
792− res <- initial
793− case res of
794− PRD.IPartial s -> go SPEC state (List []) s
795− PRD.IDone b -> return (b, stream)
796− PRD.IError err -> throwM $ ParseError err
797−
798− where
799−
800− -- "backBuf" contains last few items in the stream that we may have to
801− -- backtrack to.
802− --
803− -- XXX currently we are using a dumb list based approach for backtracking
804− -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
805− -- That will allow us more efficient random back and forth movement.
806− go !_ st backBuf !pst = do
807− r <- step defState st
808− case r of
809− D.Yield (Array contents start end) s ->
810− gobuf SPEC s backBuf
811− (Tuple' end contents) start pst
812− D.Skip s -> go SPEC s backBuf pst
813− D.Stop -> do
814− b <- extract pst
815− return (b, D.nil)
816−
817− -- Use strictness on "cur" to keep it unboxed
818− gobuf !_ s backBuf (Tuple' end _) !cur !pst
819− | cur == end = do
820− go SPEC s backBuf pst
821− gobuf !_ s backBuf fp@(Tuple' end contents) !cur !pst = do
822− x <- liftIO $ peekByteIndex contents cur
823− pRes <- pstep pst x
824− let next = INDEX_NEXT(cur,a)
825− case pRes of
826− PR.Partial 0 pst1 ->
827− gobuf SPEC s (List []) fp next pst1
828− PR.Partial n pst1 -> do
829− assert (n <= Prelude.length (x:getList backBuf)) (return ())
830− let src0 = Prelude.take n (x:getList backBuf)
831− arr0 = A.fromListN n (Prelude.reverse src0)
832− arr1 = Array contents next end
833− src = arr0 <> arr1
834− let !(Array cont1 start end1) = src
835− fp1 = Tuple' end1 cont1
836− gobuf SPEC s (List []) fp1 start pst1
837− PR.Continue 0 pst1 ->
838− gobuf SPEC s (List (x:getList backBuf)) fp next pst1
839− PR.Continue n pst1 -> do
840− assert (n <= Prelude.length (x:getList backBuf)) (return ())
841− let (src0, buf1) = splitAt n (x:getList backBuf)
842− arr0 = A.fromListN n (Prelude.reverse src0)
843− arr1 = Array contents next end
844− src = arr0 <> arr1
845− let !(Array cont1 start end1) = src
846− fp1 = Tuple' end1 cont1
847− gobuf SPEC s (List buf1) fp1 start pst1
848− PR.Done 0 b -> do
849− let arr = Array contents next end
850− return (b, D.cons arr (D.Stream step s))
851− PR.Done n b -> do
852− assert (n <= Prelude.length (x:getList backBuf)) (return ())
853− let src0 = Prelude.take n (x:getList backBuf)
854− -- XXX create the array in reverse instead
855− arr0 = A.fromListN n (Prelude.reverse src0)
856− arr1 = Array contents next end
857− -- XXX Use StreamK to avoid adding arbitrary layers of
858− -- constructors every time.
859− str = D.cons arr0 (D.cons arr1 (D.Stream step s))
860− return (b, str)
861− PR.Error err -> throwM $ ParseError err
862−-}
863−
864−-- | Parse an array stream using the supplied 'Parser'. Returns the parse
865−-- result and the unconsumed stream. Throws 'ParseError' if the parse fails.
866−--
867−-- The following alternative to this function allows composing the parser using
868−-- the parser Monad:
869−--
870−-- >>> parseBreakStreamK p = StreamK.parseBreakChunks (ParserK.adaptC p)
871−--
872−-- We can compare perf and remove this one or define it in terms of that.
873−--
874−-- /Internal/
875−--
876−{-# INLINE_NORMAL parseBreakChunksK #-}
877−parseBreakChunksK ::
878− forall m a b. (MonadIO m, Unbox a)
879− => Parser a m b
880− -> StreamK m (Array a)
881− -> m (Either ParseError b, StreamK m (Array a))
882−-- parseBreakStreamK p = StreamK.parseBreakChunks (ParserK.adaptC p)
883−parseBreakChunksK (Parser pstep initial extract) stream = do
884− res <- initial
885− case res of
886− IPartial s -> go s stream []
887− IDone b -> return (Right b, stream)
888− IError err -> return (Left (ParseError err), stream)
889−
890− where
891−
892− -- "backBuf" contains last few items in the stream that we may have to
893− -- backtrack to.
894− --
895− -- XXX currently we are using a dumb list based approach for backtracking
896− -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
897− -- That will allow us more efficient random back and forth movement.
898− go !pst st backBuf = do
899− let stop = goStop pst backBuf -- (, K.nil) <$> extract pst
900− single a = yieldk a StreamK.nil
901− yieldk arr r = goArray pst backBuf r arr
902− in StreamK.foldStream defState yieldk single stop st
903−
904− -- Use strictness on "cur" to keep it unboxed
905− goArray !pst backBuf st (Array _ cur end) | cur == end = go pst st backBuf
906− goArray !pst backBuf st (Array contents cur end) = do
907− x <- liftIO $ peekAt cur contents
908− pRes <- pstep pst x
909− let next = INDEX_NEXT(cur,a)
910− case pRes of
911− Parser.Partial 0 s ->
912− goArray s [] st (Array contents next end)
913− Parser.Partial n s -> do
914− assert (n <= Prelude.length (x:backBuf)) (return ())
915− let src0 = Prelude.take n (x:backBuf)
916− arr0 = fromListN n (Prelude.reverse src0)
917− arr1 = Array contents next end
918− src = arr0 <> arr1
919− goArray s [] st src
920− Parser.Continue 0 s ->
921− goArray s (x:backBuf) st (Array contents next end)
922− Parser.Continue n s -> do
923− assert (n <= Prelude.length (x:backBuf)) (return ())
924− let (src0, buf1) = Prelude.splitAt n (x:backBuf)
925− arr0 = fromListN n (Prelude.reverse src0)
926− arr1 = Array contents next end
927− src = arr0 <> arr1
928− goArray s buf1 st src
929− Parser.Done 0 b -> do
930− let arr = Array contents next end
931− return (Right b, StreamK.cons arr st)
932− Parser.Done n b -> do
933− assert (n <= Prelude.length (x:backBuf)) (return ())
934− let src0 = Prelude.take n (x:backBuf)
935− -- XXX Use fromListRevN once implemented
936− -- arr0 = A.fromListRevN n src0
937− arr0 = fromListN n (Prelude.reverse src0)
938− arr1 = Array contents next end
939− str = StreamK.cons arr0 (StreamK.cons arr1 st)
940− return (Right b, str)
941− Parser.Error err -> do
942− let n = Prelude.length backBuf
943− arr0 = fromListN n (Prelude.reverse backBuf)
944− arr1 = Array contents cur end
945− str = StreamK.cons arr0 (StreamK.cons arr1 stream)
946− return (Left (ParseError err), str)
947−
948− -- This is a simplified goArray
949− goExtract !pst backBuf (Array _ cur end)
950− | cur == end = goStop pst backBuf
951− goExtract !pst backBuf (Array contents cur end) = do
952− x <- liftIO $ peekAt cur contents
953− pRes <- pstep pst x
954− let next = INDEX_NEXT(cur,a)
955− case pRes of
956− Parser.Partial 0 s ->
957− goExtract s [] (Array contents next end)
958− Parser.Partial n s -> do
959− assert (n <= Prelude.length (x:backBuf)) (return ())
960− let src0 = Prelude.take n (x:backBuf)
961− arr0 = fromListN n (Prelude.reverse src0)
962− arr1 = Array contents next end
963− src = arr0 <> arr1
964− goExtract s [] src
965− Parser.Continue 0 s ->
966− goExtract s backBuf (Array contents next end)
967− Parser.Continue n s -> do
968− assert (n <= Prelude.length (x:backBuf)) (return ())
969− let (src0, buf1) = Prelude.splitAt n (x:backBuf)
970− arr0 = fromListN n (Prelude.reverse src0)
971− arr1 = Array contents next end
972− src = arr0 <> arr1
973− goExtract s buf1 src
974− Parser.Done 0 b -> do
975− let arr = Array contents next end
976− return (Right b, StreamK.fromPure arr)
977− Parser.Done n b -> do
978− assert (n <= Prelude.length backBuf) (return ())
979− let src0 = Prelude.take n backBuf
980− -- XXX Use fromListRevN once implemented
981− -- arr0 = A.fromListRevN n src0
982− arr0 = fromListN n (Prelude.reverse src0)
983− arr1 = Array contents next end
984− str = StreamK.cons arr0 (StreamK.fromPure arr1)
985− return (Right b, str)
986− Parser.Error err -> do
987− let n = Prelude.length backBuf
988− arr0 = fromListN n (Prelude.reverse backBuf)
989− arr1 = Array contents cur end
990− str = StreamK.cons arr0 (StreamK.cons arr1 stream)
991− return (Left (ParseError err), str)
992−
993− -- This is a simplified goExtract
994− {-# INLINE goStop #-}
995− goStop !pst backBuf = do
996− pRes <- extract pst
997− case pRes of
998− Parser.Partial _ _ -> error "Bug: parseBreak: Partial in extract"
999− Parser.Continue 0 s ->
1000− goStop s backBuf
1001− Parser.Continue n s -> do
1002− assert (n <= Prelude.length backBuf) (return ())
1003− let (src0, buf1) = Prelude.splitAt n backBuf
1004− arr = fromListN n (Prelude.reverse src0)
1005− goExtract s buf1 arr
1006− Parser.Done 0 b ->
1007− return (Right b, StreamK.nil)
1008− Parser.Done n b -> do
1009− assert (n <= Prelude.length backBuf) (return ())
1010− let src0 = Prelude.take n backBuf
1011− -- XXX Use fromListRevN once implemented
1012− -- arr0 = A.fromListRevN n src0
1013− arr0 = fromListN n (Prelude.reverse src0)
1014− return (Right b, StreamK.fromPure arr0)
1015− Parser.Error err -> do
1016− let n = Prelude.length backBuf
1017− arr0 = fromListN n (Prelude.reverse backBuf)
1018− return (Left (ParseError err), StreamK.fromPure arr0)
545+streamFold f arr = f (A.read arr)
core/src/Streamly/Internal/Data/Array/Generic.hs+6−22
@@ -11,8 +11,8 @@ module Streamly.Internal.Data.Array.Generic
1111
1212 -- * Construction
1313 , nil
14− , createOf
15− , create
14+ , writeN
15+ , write
1616 , writeWith
1717 , writeLastN
1818
@@ -44,10 +44,6 @@ module Streamly.Internal.Data.Array.Generic
4444 , getIndex
4545 , getSliceUnsafe
4646 , strip
47−
48− -- * Deprecated
49− , writeN
50− , write
5147 )
5248where
5349
@@ -108,33 +104,21 @@ nil = unsafePerformIO $ unsafeFreeze <$> MArray.nil
108104-- Construction - Folds
109105-------------------------------------------------------------------------------
110106
111−{-# INLINE_NORMAL createOf #-}
112−createOf :: MonadIO m => Int -> Fold m a (Array a)
113−createOf = fmap unsafeFreeze <$> MArray.createOf
114−
115−-- XXX Deprecate in major
116−-- {-# DEPRECATED writeN "Please use createOf instead." #-}
117−{-# INLINE writeN #-}
107+{-# INLINE_NORMAL writeN #-}
118108writeN :: MonadIO m => Int -> Fold m a (Array a)
119−writeN = createOf
109+writeN = fmap unsafeFreeze <$> MArray.writeN
120110
121111{-# INLINE_NORMAL writeWith #-}
122112writeWith :: MonadIO m => Int -> Fold m a (Array a)
123−writeWith elemCount = unsafeFreeze <$> MArray.createWith elemCount
113+writeWith elemCount = unsafeFreeze <$> MArray.writeWith elemCount
124114
125115-- | Fold the whole input to a single array.
126116--
127117-- /Caution! Do not use this on infinite streams./
128118--
129−{-# INLINE create #-}
130−create :: MonadIO m => Fold m a (Array a)
131−create = fmap unsafeFreeze MArray.create
132−
133−-- XXX Deprecate in major
134−-- {-# DEPRECATED write "Please use create instead." #-}
135119{-# INLINE write #-}
136120write :: MonadIO m => Fold m a (Array a)
137−write = create
121+write = fmap unsafeFreeze MArray.write
138122
139123fromPureStream :: Stream Identity a -> Array a
140124fromPureStream x =
core/src/Streamly/Internal/Data/Array/Type.hs+109−518
@@ -1,11 +1,8 @@
11{-# LANGUAGE CPP #-}
2−{-# LANGUAGE TypeFamilies #-}
3−-- Must come after TypeFamilies, otherwise it is re-enabled.
4−-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
5−{-# LANGUAGE NoMonoLocalBinds #-}
62-- |
73-- Module : Streamly.Internal.Data.Array.Type
84-- Copyright : (c) 2020 Composewell Technologies
5+--
96-- License : BSD3-3-Clause
107-- Maintainer : streamly@composewell.com
118-- Stability : experimental
@@ -13,146 +10,86 @@
1310--
1411-- See notes in "Streamly.Internal.Data.MutArray.Type"
1512--
16−
1713module Streamly.Internal.Data.Array.Type
1814 (
19− -- ** Type
2015 -- $arrayNotes
2116 Array (..)
17+ , asPtrUnsafe
18+ , nil
2219
23− -- ** Conversion
24− -- *** Mutable and Immutable
20+ -- * Freezing and Thawing
2521 , unsafeFreeze
2622 , unsafeFreezeWithShrink
2723 , unsafeThaw
2824
29− -- *** Pinned and Unpinned
25+ -- * Pinning and Unpinning
3026 , pin
3127 , unpin
3228 , isPinned
3329
34− -- *** Casting
35− , unsafePinnedAsPtr
36−
37− -- ** Construction
38− , empty
39−
40− -- *** Cloning
41− -- XXX Why would we clone an immutable array?
42− , clone
43− , pinnedClone
44−
45− -- *** Slicing
46− -- | Get a subarray without copying
47− , splitAt
48− , breakOn -- XXX requires MonadIO
30+ -- * Construction
31+ , splice
4932
50− -- *** Stream Folds
51− , unsafeMakePure
52− , createOf
53− , pinnedCreateOf
54− , unsafeCreateOf
55− , unsafePinnedCreateOf
56− , create
57− , pinnedCreate
58− , createWith
59−
60− -- *** From containers
61− , fromListN
62− , pinnedFromListN
6333 , fromList
6434 , pinnedFromList
65− , fromListRevN
35+ , fromListN
36+ , pinnedFromListN
6637 , fromListRev
67− , fromStreamN
68− , fromStream
69− , fromPureStreamN
38+ , fromListRevN
39+ , fromStreamDN
40+ , fromStreamD
7041 , fromPureStream
7142 , fromByteStr#
72− , fromByteStr
73− , fromPtrN
74− , fromChunks
75− , fromChunksK
7643
77− -- ** Reading
44+ -- * Split
45+ , breakOn
7846
79− -- *** Indexing
80− , unsafeIndexIO -- XXX unsafeGetIndexIO
81− , getIndexUnsafe -- XXX unsafeGetIndex
82−
83− -- *** To Streams
84− , read
85− , readRev
86− , toStreamK
87− , toStreamKRev
88−
89− -- *** To Containers
90− , toList
91−
92− -- *** Unfolds
93− , producer -- experimental
94− , readerUnsafe
95− , reader
96− , readerRev
47+ -- ** Cloning arrays
48+ , clone
49+ , pinnedClone
9750
98− -- *** Size
99− , length
51+ -- * Elimination
52+ , unsafeIndexIO
53+ , getIndexUnsafe
10054 , byteLength
55+ , length
10156
102− -- ** Folding
10357 , foldl'
10458 , foldr
105− , byteCmp
106− , byteEq
107−
108− -- ** Appending
109− , splice -- XXX requires MonadIO
110−
111− -- ** Streams of arrays
112− -- *** Chunk
113− -- | Group a stream into arrays.
114− , chunksOf
115− , pinnedChunksOf
116− , buildChunks
117−
118− -- *** Split
119− -- | Split an array into slices.
120−
121− -- *** Concat
122− -- | Append the arrays in a stream to form a stream of elements.
123− , concat
124− , concatRev
125−
126− -- *** Compact
127− -- | Append the arrays in a stream to form a stream of larger arrays.
128− , fCompactGE
129− , fPinnedCompactGE
130− , lCompactGE
131− , lPinnedCompactGE
132− , compactGE
59+ , splitAt
13360
134− -- ** Deprecated
135− , asPtrUnsafe
136− , unsafeIndex
137− , bufferChunks
138− , flattenArrays
139− , flattenArraysRev
140− , fromArrayStreamK
141− , fromStreamDN
142− , fromStreamD
14361 , toStreamD
14462 , toStreamDRev
63+ , toStreamK
64+ , toStreamKRev
14565 , toStream
14666 , toStreamRev
147− , nil
67+ , read
68+ , readRev
69+ , readerRev
70+ , toList
71+
72+ -- * Folds
14873 , writeWith
14974 , writeN
15075 , pinnedWriteN
15176 , writeNUnsafe
15277 , pinnedWriteNUnsafe
78+ , MA.ArrayUnsafe (..)
15379 , pinnedWriteNAligned
15480 , write
15581 , pinnedWrite
82+ , unsafeMakePure
83+
84+ -- * Streams of arrays
85+ , chunksOf
86+ , pinnedChunksOf
87+ , bufferChunks
88+ , flattenArrays
89+ , flattenArraysRev
90+
91+ -- * Deprecated
92+ , unsafeIndex
15693 )
15794where
15895
@@ -160,36 +97,31 @@ where
16097#include "inline.hs"
16198
16299import Control.Exception (assert)
163−import Control.Monad (replicateM, when)
100+import Control.Monad (replicateM)
164101import Control.Monad.IO.Class (MonadIO(..))
165102import Data.Functor.Identity (Identity(..))
166−import Data.Int (Int8, Int16, Int32, Int64)
167103import Data.Proxy (Proxy(..))
168−import Data.Word (Word8, Word16, Word32, Word64)
104+import Data.Word (Word8)
169105import GHC.Base (build)
170106import GHC.Exts (IsList, IsString(..), Addr#)
171107
172108import GHC.IO (unsafePerformIO)
173109import GHC.Ptr (Ptr(..))
174−import Streamly.Internal.Data.Producer.Type (Producer(..))
175−import Streamly.Internal.Data.MutArray.Type (MutArray(..))
176−import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
110+import Streamly.Internal.Data.MutArray.Type (MutArray(..), MutableByteArray)
177111import Streamly.Internal.Data.Fold.Type (Fold(..))
178112import Streamly.Internal.Data.Stream.Type (Stream)
179−import Streamly.Internal.Data.StreamK.Type (StreamK)
180113import Streamly.Internal.Data.Unbox (Unbox(..))
181114import Streamly.Internal.Data.Unfold.Type (Unfold(..))
182115import Text.Read (readPrec)
183116
184−import Prelude hiding (Foldable(..), concat, read, unlines, splitAt)
117+import Prelude hiding (Foldable(..), read, unlines, splitAt)
185118
186119import qualified GHC.Exts as Exts
187−import qualified Streamly.Internal.Data.Fold.Type as Fold
188120import qualified Streamly.Internal.Data.MutArray.Type as MA
189121import qualified Streamly.Internal.Data.Stream.Type as D
122+import qualified Streamly.Internal.Data.Stream.Generate as D
190123import qualified Streamly.Internal.Data.StreamK.Type as K
191−import qualified Streamly.Internal.Data.MutByteArray.Type as Unboxed
192−import qualified Streamly.Internal.Data.Producer as Producer
124+import qualified Streamly.Internal.Data.Unbox as Unboxed
193125import qualified Streamly.Internal.Data.Unfold.Type as Unfold
194126import qualified Text.ParserCombinators.ReadPrec as ReadPrec
195127
@@ -232,7 +164,7 @@ data Array a =
232164#endif
233165 -- All offsets are in terms of bytes from the start of arraycontents
234166 Array
235− { arrContents :: {-# UNPACK #-} !MutByteArray
167+ { arrContents :: {-# UNPACK #-} !MutableByteArray
236168 , arrStart :: {-# UNPACK #-} !Int -- offset
237169 , arrEnd :: {-# UNPACK #-} !Int -- offset + len
238170 }
@@ -243,20 +175,14 @@ data Array a =
243175
244176-- | Use an @Array a@ as @Ptr a@.
245177--
246−-- See 'MA.unsafePinnedAsPtr' in the Mutable array module for more details.
178+-- See 'MA.asPtrUnsafe' in the Mutable array module for more details.
247179--
248180-- /Unsafe/
249181--
250182-- /Pre-release/
251183--
252−{-# INLINE unsafePinnedAsPtr #-}
253−unsafePinnedAsPtr :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
254−unsafePinnedAsPtr arr = MA.unsafePinnedAsPtr (unsafeThaw arr)
255−
256−{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
257−{-# INLINE asPtrUnsafe #-}
258184asPtrUnsafe :: MonadIO m => Array a -> (Ptr a -> m b) -> m b
259−asPtrUnsafe = unsafePinnedAsPtr
185+asPtrUnsafe arr = MA.asPtrUnsafe (unsafeThaw arr)
260186
261187-------------------------------------------------------------------------------
262188-- Freezing and Thawing
@@ -303,19 +229,18 @@ unsafeThaw (Array ac as ae) = MutArray ac as ae ae
303229-- Pinning & Unpinning
304230-------------------------------------------------------------------------------
305231
306−-- | Return a copy of the 'Array' in pinned memory if unpinned, else return the
307−-- original array.
232+-- | Copy the 'Array' to pinned memory if unpinned, else do nothing. The
233+-- overhead is a copy if the input array is unpinned.
308234{-# INLINE pin #-}
309235pin :: Array a -> IO (Array a)
310236pin = fmap unsafeFreeze . MA.pin . unsafeThaw
311237
312−-- | Return a copy of the 'Array' in unpinned memory if pinned, else return the
313−-- original array.
238+-- | Copy the 'Array' to unpinned memory if pinned, else do nothing. The
239+-- overhead is a copy if the input array is pinned.
314240{-# INLINE unpin #-}
315241unpin :: Array a -> IO (Array a)
316242unpin = fmap unsafeFreeze . MA.unpin . unsafeThaw
317243
318−-- | Return 'True' if the array is allocated in pinned memory.
319244{-# INLINE isPinned #-}
320245isPinned :: Array a -> Bool
321246isPinned = MA.isPinned . unsafeThaw
@@ -324,14 +249,11 @@ isPinned = MA.isPinned . unsafeThaw
324249-- Construction
325250-------------------------------------------------------------------------------
326251
327−-- | Copy two immutable arrays into a new array. If you want to splice more
328−-- than two arrays then this operation would be highly inefficient because it
329−-- would make a copy on every splice operation, instead use the
330−-- 'fromChunksK' operation to combine n immutable arrays.
252+-- Splice two immutable arrays creating a new array.
331253{-# INLINE splice #-}
332−splice :: MonadIO m => Array a -> Array a -> m (Array a)
254+splice :: (MonadIO m, Unbox a) => Array a -> Array a -> m (Array a)
333255splice arr1 arr2 =
334− unsafeFreeze <$> MA.spliceCopy (unsafeThaw arr1) (unsafeThaw arr2)
256+ unsafeFreeze <$> MA.splice (unsafeThaw arr1) (unsafeThaw arr2)
335257
336258-- | Create an 'Array' from the first N elements of a list. The array is
337259-- allocated to size N, if the list terminates before N elements then the
@@ -375,83 +297,36 @@ pinnedFromList xs = unsafePerformIO $ unsafeFreeze <$> MA.pinnedFromList xs
375297fromListRev :: Unbox a => [a] -> Array a
376298fromListRev xs = unsafePerformIO $ unsafeFreeze <$> MA.fromListRev xs
377299
378−-- | Create an 'Array' from the first N elements of a stream. The array is
379−-- allocated to size N, if the stream terminates before N elements then the
380−-- array may hold less than N elements.
381−--
382−-- >>> fromStreamN n = Stream.fold (Array.writeN n)
383−--
384−-- /Pre-release/
385−{-# INLINE_NORMAL fromStreamN #-}
386−fromStreamN :: (MonadIO m, Unbox a) => Int -> Stream m a -> m (Array a)
387−fromStreamN n m = do
388− when (n < 0) $ error "writeN: negative write count specified"
389− unsafeFreeze <$> MA.fromStreamN n m
390−-- fromStreamN n = D.fold (writeN n)
391−
392−{-# DEPRECATED fromStreamDN "Please use fromStreamN instead." #-}
300+{-# INLINE_NORMAL fromStreamDN #-}
393301fromStreamDN :: forall m a. (MonadIO m, Unbox a)
394302 => Int -> D.Stream m a -> m (Array a)
395−fromStreamDN = fromStreamN
303+fromStreamDN limit str = unsafeFreeze <$> MA.fromStreamDN limit str
396304
397−-- | Create an 'Array' from a stream. This is useful when we want to create a
398−-- single array from a stream of unknown size. 'writeN' is at least twice
399−-- as efficient when the size is already known.
400−--
401−-- >>> fromStream = Stream.fold Array.write
402−--
403−-- Note that if the input stream is too large memory allocation for the array
404−-- may fail. When the stream size is not known, `chunksOf` followed by
405−-- processing of indvidual arrays in the resulting stream should be preferred.
406−--
407−-- /Pre-release/
408305{-# INLINE_NORMAL fromStreamD #-}
409−fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (Array a)
410−fromStream = D.fold write
411−-- fromStreamD str = unsafeFreeze <$> MA.fromStream str
412−
413−{-# DEPRECATED fromStreamD "Please use fromStream instead." #-}
414306fromStreamD :: forall m a. (MonadIO m, Unbox a)
415307 => D.Stream m a -> m (Array a)
416−fromStreamD = fromStream
308+fromStreamD str = unsafeFreeze <$> MA.fromStreamD str
417309
418310-------------------------------------------------------------------------------
419311-- Cloning
420312-------------------------------------------------------------------------------
421313
422314{-# INLINE clone #-}
423−clone ::
424− ( MonadIO m
425−#ifdef DEVBUILD
426− , Unbox a
427−#endif
428− )
429− => Array a -> m (Array a)
315+clone :: MonadIO m => Array a -> m (Array a)
430316clone = fmap unsafeFreeze . MA.clone . unsafeThaw
431317
432318{-# INLINE pinnedClone #-}
433−pinnedClone ::
434− ( MonadIO m
435−#ifdef DEVBUILD
436− , Unbox a
437−#endif
438− )
439− => Array a -> m (Array a)
319+pinnedClone :: MonadIO m => Array a -> m (Array a)
440320pinnedClone = fmap unsafeFreeze . MA.pinnedClone . unsafeThaw
441321
442322-------------------------------------------------------------------------------
443323-- Streams of arrays
444324-------------------------------------------------------------------------------
445325
446−{-# INLINE buildChunks #-}
447−buildChunks :: (MonadIO m, Unbox a) =>
448− D.Stream m a -> m (K.StreamK m (Array a))
449−buildChunks m = D.foldr K.cons K.nil $ chunksOf defaultChunkSize m
450−
451−{-# DEPRECATED bufferChunks "Please use buildChunks instead." #-}
326+{-# INLINE bufferChunks #-}
452327bufferChunks :: (MonadIO m, Unbox a) =>
453328 D.Stream m a -> m (K.StreamK m (Array a))
454−bufferChunks = buildChunks
329+bufferChunks m = D.foldr K.cons K.nil $ chunksOf defaultChunkSize m
455330
456331-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
457332-- @n@ elements each.
@@ -472,100 +347,27 @@ pinnedChunksOf :: forall m a. (MonadIO m, Unbox a)
472347 => Int -> D.Stream m a -> D.Stream m (Array a)
473348pinnedChunksOf n str = D.map unsafeFreeze $ MA.pinnedChunksOf n str
474349
475−-- | Convert a stream of arrays into a stream of their elements.
350+-- | Use the "read" unfold instead.
476351--
477−-- >>> concat = Stream.unfoldMany Array.reader
352+-- @flattenArrays = unfoldMany read@
478353--
479−{-# INLINE_NORMAL concat #-}
480−concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
481−-- XXX this requires MonadIO whereas the unfoldMany version does not
482−concat = MA.concatWith (pure . unsafeInlineIO) . D.map unsafeThaw
483−-- concat = D.unfoldMany reader
484−
485−{-# DEPRECATED flattenArrays "Please use \"unfoldMany reader\" instead." #-}
486−{-# INLINE flattenArrays #-}
354+-- We can try this if there are any fusion issues in the unfold.
355+--
356+{-# INLINE_NORMAL flattenArrays #-}
487357flattenArrays :: forall m a. (MonadIO m, Unbox a)
488358 => D.Stream m (Array a) -> D.Stream m a
489−flattenArrays = concat
359+flattenArrays = MA.flattenArrays . D.map unsafeThaw
490360
491−-- | Convert a stream of arrays into a stream of their elements reversing the
492−-- contents of each array before flattening.
361+-- | Use the "readRev" unfold instead.
493362--
494−-- >>> concatRev = Stream.unfoldMany Array.readerRev
363+-- @flattenArrays = unfoldMany readRev@
495364--
496−{-# INLINE_NORMAL concatRev #-}
497−concatRev :: forall m a. (Monad m, Unbox a)
498− => D.Stream m (Array a) -> D.Stream m a
499−-- XXX this requires MonadIO whereas the unfoldMany version does not
500−concatRev = MA.concatRevWith (pure . unsafeInlineIO) . D.map unsafeThaw
501−-- concatRev = D.unfoldMany readerRev
502−
503−{-# DEPRECATED flattenArraysRev "Please use \"unfoldMany readerRev\" instead." #-}
504−{-# INLINE flattenArraysRev #-}
365+-- We can try this if there are any fusion issues in the unfold.
366+--
367+{-# INLINE_NORMAL flattenArraysRev #-}
505368flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
506369 => D.Stream m (Array a) -> D.Stream m a
507−flattenArraysRev = concatRev
508−
509−-------------------------------------------------------------------------------
510−-- Compact
511−-------------------------------------------------------------------------------
512−
513−-- XXX Note that this thaws immutable arrays for appending, that may be
514−-- problematic if multiple users do the same thing, however, thawed immutable
515−-- arrays would have no capacity to append, therefore, a copy will be forced
516−-- anyway.
517−
518−-- | Fold @fCompactGE n@ coalesces adjacent arrays in the input stream
519−-- until the size becomes greater than or equal to n.
520−--
521−-- Generates unpinned arrays irrespective of the pinning status of input
522−-- arrays.
523−{-# INLINE_NORMAL fCompactGE #-}
524−fCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
525−fCompactGE n = fmap unsafeFreeze $ Fold.lmap unsafeThaw $ MA.fCompactGE n
526−
527−-- | PInned version of 'fCompactGE'.
528−{-# INLINE_NORMAL fPinnedCompactGE #-}
529−fPinnedCompactGE :: (MonadIO m, Unbox a) => Int -> Fold m (Array a) (Array a)
530−fPinnedCompactGE n =
531− fmap unsafeFreeze $ Fold.lmap unsafeThaw $ MA.fPinnedCompactGE n
532−
533−-- | @compactGE n stream@ coalesces adjacent arrays in the @stream@ until
534−-- the size becomes greater than or equal to @n@.
535−--
536−-- >>> compactGE n = Stream.foldMany (Array.fCompactGE n)
537−--
538−-- Generates unpinned arrays irrespective of the pinning status of input
539−-- arrays.
540−{-# INLINE compactGE #-}
541−compactGE ::
542− (MonadIO m, Unbox a)
543− => Int -> Stream m (Array a) -> Stream m (Array a)
544−compactGE n stream =
545− D.map unsafeFreeze $ MA.compactGE n $ D.map unsafeThaw stream
546−
547−-- | Like 'compactGE' but for transforming folds instead of stream.
548−--
549−-- >>> lCompactGE n = Fold.many (Array.fCompactGE n)
550−--
551−-- Generates unpinned arrays irrespective of the pinning status of input
552−-- arrays.
553−{-# INLINE_NORMAL lCompactGE #-}
554−lCompactGE :: (MonadIO m, Unbox a)
555− => Int -> Fold m (Array a) () -> Fold m (Array a) ()
556−lCompactGE n fld =
557− Fold.lmap unsafeThaw $ MA.lCompactGE n (Fold.lmap unsafeFreeze fld)
558−
559−-- | Pinned version of 'lCompactGE'.
560−{-# INLINE_NORMAL lPinnedCompactGE #-}
561−lPinnedCompactGE :: (MonadIO m, Unbox a)
562− => Int -> Fold m (Array a) () -> Fold m (Array a) ()
563−lPinnedCompactGE n fld =
564− Fold.lmap unsafeThaw $ MA.lPinnedCompactGE n (Fold.lmap unsafeFreeze fld)
565−
566−-------------------------------------------------------------------------------
567−-- Splitting
568−-------------------------------------------------------------------------------
370+flattenArraysRev = MA.flattenArraysRev . D.map unsafeThaw
569371
570372-- Drops the separator byte
571373{-# INLINE breakOn #-}
@@ -609,68 +411,24 @@ byteLength = MA.byteLength . unsafeThaw
609411length :: Unbox a => Array a -> Int
610412length arr = MA.length (unsafeThaw arr)
611413
612−{-# INLINE_NORMAL producer #-}
613−producer :: forall m a. (Monad m, Unbox a) => Producer m (Array a) a
614−producer =
615− Producer.translate unsafeThaw unsafeFreeze
616− $ MA.producerWith (return . unsafeInlineIO)
617−
618−-- | Unfold an array into a stream.
619−--
620−{-# INLINE_NORMAL reader #-}
621−reader :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
622−reader = Producer.simplify producer
623−
624−-- | Unfold an array into a stream, does not check the end of the array, the
625−-- user is responsible for terminating the stream within the array bounds. For
626−-- high performance application where the end condition can be determined by
627−-- a terminating fold.
628−--
629−-- Written in the hope that it may be faster than "read", however, in the case
630−-- for which this was written, "read" proves to be faster even though the core
631−-- generated with unsafeRead looks simpler.
632−--
633−-- /Pre-release/
634−--
635−{-# INLINE_NORMAL readerUnsafe #-}
636−readerUnsafe :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
637−readerUnsafe = Unfold step inject
638− where
639−
640− inject (Array contents start end) =
641− return (MA.ArrayUnsafe contents end start)
642−
643− {-# INLINE_LATE step #-}
644− step (MA.ArrayUnsafe contents end p) = do
645− -- unsafeInlineIO allows us to run this in Identity monad for pure
646− -- toList/foldr case which makes them much faster due to not
647− -- accumulating the list and fusing better with the pure consumers.
648− --
649− -- This should be safe as the array contents are guaranteed to be
650− -- evaluated/written to before we peek at them.
651− let !x = unsafeInlineIO $ peekAt p contents
652− let !p1 = INDEX_NEXT(p,a)
653− return $ D.Yield x (MA.ArrayUnsafe contents end p1)
654−
655414-- | Unfold an array into a stream in reverse order.
656415--
657416{-# INLINE_NORMAL readerRev #-}
658417readerRev :: forall m a. (Monad m, Unbox a) => Unfold m (Array a) a
659418readerRev = Unfold.lmap unsafeThaw $ MA.readerRevWith (return . unsafeInlineIO)
660419
661−{-# DEPRECATED toStreamD "Please use 'read' instead." #-}
662420{-# INLINE_NORMAL toStreamD #-}
663421toStreamD :: forall m a. (Monad m, Unbox a) => Array a -> D.Stream m a
664−toStreamD = read
422+toStreamD arr = MA.toStreamDWith (return . unsafeInlineIO) (unsafeThaw arr)
665423
666424{-# INLINE toStreamK #-}
667425toStreamK :: forall m a. (Monad m, Unbox a) => Array a -> K.StreamK m a
668426toStreamK arr = MA.toStreamKWith (return . unsafeInlineIO) (unsafeThaw arr)
669427
670−{-# DEPRECATED toStreamDRev "Please use 'readRev' instead." #-}
671428{-# INLINE_NORMAL toStreamDRev #-}
672429toStreamDRev :: forall m a. (Monad m, Unbox a) => Array a -> D.Stream m a
673−toStreamDRev = readRev
430+toStreamDRev arr =
431+ MA.toStreamDRevWith (return . unsafeInlineIO) (unsafeThaw arr)
674432
675433{-# INLINE toStreamKRev #-}
676434toStreamKRev :: forall m a. (Monad m, Unbox a) => Array a -> K.StreamK m a
@@ -682,7 +440,7 @@ toStreamKRev arr =
682440-- /Pre-release/
683441{-# INLINE_EARLY read #-}
684442read :: (Monad m, Unbox a) => Array a -> Stream m a
685−read arr = MA.toStreamWith (return . unsafeInlineIO) (unsafeThaw arr)
443+read = toStreamD
686444
687445-- | Same as 'read'
688446--
@@ -699,7 +457,7 @@ toStream = read
699457-- /Pre-release/
700458{-# INLINE_EARLY readRev #-}
701459readRev :: (Monad m, Unbox a) => Array a -> Stream m a
702−readRev arr = MA.toStreamRevWith (return . unsafeInlineIO) (unsafeThaw arr)
460+readRev = toStreamDRev
703461
704462-- | Same as 'readRev'
705463--
@@ -723,7 +481,6 @@ foldr f z arr = runIdentity $ D.foldr f z $ toStreamD arr
723481-- | Create two slices of an array without copying the original array. The
724482-- specified index @i@ is the first index of the second slice.
725483--
726−{-# INLINE splitAt #-}
727484splitAt :: Unbox a => Int -> Array a -> (Array a, Array a)
728485splitAt i arr = (unsafeFreeze a, unsafeFreeze b)
729486 where
@@ -743,8 +500,8 @@ toListFB c n Array{..} = go arrStart
743500 -- accumulating the list and fusing better with the pure consumers.
744501 --
745502 -- This should be safe as the array contents are guaranteed to be
746− -- evaluated/written to before we peekAt at them.
747− let !x = unsafeInlineIO $ peekAt p arrContents
503+ -- evaluated/written to before we peekByteIndex at them.
504+ let !x = unsafeInlineIO $ peekByteIndex p arrContents
748505 in c x (go (INDEX_NEXT(p,a)))
749506
750507-- | Convert an 'Array' into a list.
@@ -757,28 +514,17 @@ toList s = build (\c n -> toListFB c n s)
757514-- Folds
758515-------------------------------------------------------------------------------
759516
760−-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
517+-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
761518-- 'Array'.
762519--
763−{-# INLINE_NORMAL createOf #-}
764−createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
765−createOf = fmap unsafeFreeze . MA.createOf
766−
767−-- XXX Deprecate in major
768−-- {-# DEPRECATED writeN "Please use createOf instead." #-}
769−{-# INLINE writeN #-}
520+{-# INLINE_NORMAL writeN #-}
770521writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
771−writeN = createOf
522+writeN = fmap unsafeFreeze . MA.writeN
772523
773−-- | Like 'createOf' but creates a pinned array.
774−{-# INLINE_NORMAL pinnedCreateOf #-}
775−pinnedCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
776−pinnedCreateOf = fmap unsafeFreeze . MA.pinnedCreateOf
777−
778−{-# DEPRECATED pinnedWriteN "Please use pinnedCreateOf instead." #-}
779−{-# INLINE pinnedWriteN #-}
524+-- | Like 'fromListN' but creates a pinned array.
525+{-# INLINE_NORMAL pinnedWriteN #-}
780526pinnedWriteN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (Array a)
781−pinnedWriteN = pinnedCreateOf
527+pinnedWriteN = fmap unsafeFreeze . MA.pinnedWriteN
782528
783529-- | @pinnedWriteNAligned alignment n@ folds a maximum of @n@ elements from the input
784530-- stream to an 'Array' aligned to the given size.
@@ -786,74 +532,44 @@ pinnedWriteN = pinnedCreateOf
786532-- /Pre-release/
787533--
788534{-# INLINE_NORMAL pinnedWriteNAligned #-}
789−{-# DEPRECATED pinnedWriteNAligned "To be removed." #-}
790535pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a)
791536 => Int -> Int -> Fold m a (Array a)
792537pinnedWriteNAligned alignSize = fmap unsafeFreeze . MA.pinnedWriteNAligned alignSize
793538
794−-- | Like 'createOf' but does not check the array bounds when writing. The fold
539+-- | Like 'writeN' but does not check the array bounds when writing. The fold
795540-- driver must not call the step function more than 'n' times otherwise it will
796541-- corrupt the memory and crash. This function exists mainly because any
797542-- conditional in the step function blocks fusion causing 10x performance
798543-- slowdown.
799544--
800−{-# INLINE_NORMAL unsafeCreateOf #-}
801−unsafeCreateOf :: forall m a. (MonadIO m, Unbox a)
802− => Int -> Fold m a (Array a)
803−unsafeCreateOf n = unsafeFreeze <$> MA.unsafeCreateOf n
804−
805−{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
806−{-# INLINE writeNUnsafe #-}
545+{-# INLINE_NORMAL writeNUnsafe #-}
807546writeNUnsafe :: forall m a. (MonadIO m, Unbox a)
808547 => Int -> Fold m a (Array a)
809−writeNUnsafe = unsafeCreateOf
548+writeNUnsafe n = unsafeFreeze <$> MA.writeNUnsafe n
810549
811−{-# INLINE_NORMAL unsafePinnedCreateOf #-}
812−unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a)
813− => Int -> Fold m a (Array a)
814−unsafePinnedCreateOf n = unsafeFreeze <$> MA.unsafePinnedCreateOf n
815−
816−{-# DEPRECATED pinnedWriteNUnsafe "Please use unsafePinnedCreateOf instead." #-}
817−{-# INLINE pinnedWriteNUnsafe #-}
550+{-# INLINE_NORMAL pinnedWriteNUnsafe #-}
818551pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a)
819552 => Int -> Fold m a (Array a)
820−pinnedWriteNUnsafe = unsafePinnedCreateOf
821−
822−{-# INLINE_NORMAL createWith #-}
823−createWith :: forall m a. (MonadIO m, Unbox a)
824− => Int -> Fold m a (Array a)
825−-- createWith n = FL.rmapM spliceArrays $ toArraysOf n
826−createWith elemCount = unsafeFreeze <$> MA.createWith elemCount
553+pinnedWriteNUnsafe n = unsafeFreeze <$> MA.pinnedWriteNUnsafe n
827554
828−{-# DEPRECATED writeWith "Please use createWith instead." #-}
829−{-# INLINE writeWith #-}
555+{-# INLINE_NORMAL writeWith #-}
830556writeWith :: forall m a. (MonadIO m, Unbox a)
831557 => Int -> Fold m a (Array a)
832−writeWith = createWith
558+-- writeWith n = FL.rmapM spliceArrays $ toArraysOf n
559+writeWith elemCount = unsafeFreeze <$> MA.writeWith elemCount
833560
834561-- | Fold the whole input to a single array.
835562--
836563-- /Caution! Do not use this on infinite streams./
837564--
838−{-# INLINE create #-}
839−create :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
840−create = fmap unsafeFreeze MA.create
841−
842−-- XXX Deprecate in major
843−-- {-# DEPRECATED write "Please use create instead." #-}
844565{-# INLINE write #-}
845566write :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
846−write = create
567+write = fmap unsafeFreeze MA.write
847568
848−-- | Like 'create' but creates a pinned array.
849−{-# INLINE pinnedCreate #-}
850−pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
851−pinnedCreate = fmap unsafeFreeze MA.pinnedCreate
852−
853−{-# DEPRECATED pinnedWrite "Please use pinnedCreate instead." #-}
569+-- | Like 'write' but creates a pinned array.
854570{-# INLINE pinnedWrite #-}
855571pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (Array a)
856−pinnedWrite = pinnedCreate
572+pinnedWrite = fmap unsafeFreeze MA.pinnedWrite
857573
858574-- | Fold "step" has a dependency on "initial", and each step is dependent on
859575-- the previous invocation of step due to state passing, finally extract
@@ -879,16 +595,10 @@ pinnedWrite = pinnedCreate
879595--
880596{-# INLINE unsafeMakePure #-}
881597unsafeMakePure :: Monad m => Fold IO a b -> Fold m a b
882−unsafeMakePure (Fold step initial extract final) =
598+unsafeMakePure (Fold step initial extract) =
883599 Fold (\x a -> return $! unsafeInlineIO (step x a))
884600 (return $! unsafePerformIO initial)
885601 (\s -> return $! unsafeInlineIO $ extract s)
886− (\s -> return $! unsafeInlineIO $ final s)
887−
888−{-# INLINE fromPureStreamN #-}
889−fromPureStreamN :: Unbox a => Int -> Stream Identity a -> Array a
890−fromPureStreamN n x =
891− unsafePerformIO $ fmap unsafeFreeze (MA.fromPureStreamN n x)
892602
893603-- | Convert a pure stream in Identity monad to an immutable array.
894604--
@@ -901,19 +611,6 @@ fromPureStream x = unsafePerformIO $ fmap (unsafeFreeze) (MA.fromPureStream x)
901611-- fromPureStream = runIdentity . D.fold (unsafeMakePure write)
902612-- fromPureStream = fromList . runIdentity . D.toList
903613
904−-- XXX This should be monadic.
905−
906−-- | Copy an immutable 'Ptr Word8' sequence into an array.
907−--
908−-- /Unsafe:/ The caller is responsible for safe addressing.
909−--
910−-- Note that this should be evaluated strictly to ensure that we do not hold
911−-- the reference to the pointer in a lazy thunk.
912−fromPtrN :: Int -> Ptr Word8 -> Array Word8
913−fromPtrN n addr = unsafePerformIO $ fmap unsafeFreeze (MA.fromPtrN n addr)
914−
915−-- XXX This should be monadic.
916−
917614-- | Copy a null terminated immutable 'Addr#' Word8 sequence into an array.
918615--
919616-- /Unsafe:/ The caller is responsible for safe addressing.
@@ -924,46 +621,8 @@ fromPtrN n addr = unsafePerformIO $ fmap unsafeFreeze (MA.fromPtrN n addr)
924621-- >>> Array.toList $ Array.fromByteStr# "\1\2\3\0"#
925622-- [1,2,3]
926623--
927−-- Note that this should be evaluated strictly to ensure that we do not hold
928−-- the reference to the pointer in a lazy thunk.
929624fromByteStr# :: Addr# -> Array Word8
930−fromByteStr# addr = unsafePerformIO $ fmap unsafeFreeze (MA.fromByteStr# addr)
931−
932−-- XXX This should be monadic.
933−
934−-- | Note that this should be evaluated strictly to ensure that we do not hold
935−-- the reference to the pointer in a lazy thunk.
936−fromByteStr :: Ptr Word8 -> Array Word8
937−fromByteStr (Ptr addr#) = fromByteStr# addr#
938−
939−-- XXX implement fromChunks/fromChunkList instead?
940−
941−-- | Convert an array stream to an array. Note that this requires peak memory
942−-- that is double the size of the array stream.
943−--
944−{-# INLINE fromChunksK #-}
945−fromChunksK :: (MonadIO m, Unbox a) => StreamK m (Array a) -> m (Array a)
946−fromChunksK stream =
947− -- We buffer the entire stream and then allocate the target array of the
948− -- same size, thus requiring double the memory.
949− fmap unsafeFreeze $ MA.fromChunksK $ fmap unsafeThaw stream
950−
951−{-# DEPRECATED fromArrayStreamK "Please use fromChunksK instead." #-}
952−fromArrayStreamK :: (Unbox a, MonadIO m) => StreamK m (Array a) -> m (Array a)
953−fromArrayStreamK = fromChunksK
954−
955−-- | Given a stream of arrays, splice them all together to generate a single
956−-- array. The stream must be /finite/.
957−--
958−{-# INLINE fromChunks #-}
959−fromChunks :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
960−fromChunks s =
961− -- XXX Check which implementation is better
962− -- This may also require double the memory as we double the space every
963− -- time, when copying the last array we may have reallocated almost double
964− -- the space required before we right size it.
965− fmap unsafeFreeze $ MA.fromChunksRealloced (fmap unsafeThaw s)
966− -- fromChunkStreamK $ D.toStreamK s
625+fromByteStr# addr = fromPureStream (D.fromByteStr# addr)
967626
968627-------------------------------------------------------------------------------
969628-- Instances
@@ -995,66 +654,12 @@ instance Unbox a => IsList (Array a) where
995654 {-# INLINE toList #-}
996655 toList = toList
997656
998−-- | Byte compare two arrays. Compare the length of the arrays. If the length
999−-- is equal, compare the lexicographical ordering of two underlying byte arrays
1000−-- otherwise return the result of length comparison.
1001−--
1002−-- /Unsafe/: Note that the 'Unbox' instance of sum types with constructors of
1003−-- different sizes may leave some memory uninitialized which can make byte
1004−-- comparison unreliable.
1005−--
1006−-- /Pre-release/
1007−{-# INLINE byteCmp #-}
1008−byteCmp :: Array a -> Array a -> Ordering
1009−byteCmp arr1 arr2 =
1010− -- unsafePerformIO?
1011− unsafeInlineIO $! unsafeThaw arr1 `MA.byteCmp` unsafeThaw arr2
1012−
1013−-- | Byte equality of two arrays.
1014−--
1015−-- >>> byteEq arr1 arr2 = (==) EQ $ Array.byteCmp arr1 arr2
1016−--
1017−-- /Unsafe/: See 'byteCmp'.
1018−{-# INLINE byteEq #-}
1019−byteEq :: Array a -> Array a -> Bool
1020−byteEq arr1 arr2 = (==) EQ $ byteCmp arr1 arr2
1021−
1022−#define MK_EQ_INSTANCE(typ) \
1023−instance {-# OVERLAPPING #-} Eq (Array typ) where { \
1024−; {-# INLINE (==) #-} \
1025−; (==) = byteEq \
1026−}
1027−
1028−MK_EQ_INSTANCE(Char)
1029−MK_EQ_INSTANCE(Word8)
1030−MK_EQ_INSTANCE(Word16)
1031−MK_EQ_INSTANCE(Word32)
1032−
1033−-- XXX The Word64 default instance should be as fast because we are comparing
1034−-- 64-bit at a time.
1035−MK_EQ_INSTANCE(Word64)
1036−MK_EQ_INSTANCE(Int)
1037−MK_EQ_INSTANCE(Int8)
1038−MK_EQ_INSTANCE(Int16)
1039−MK_EQ_INSTANCE(Int32)
1040−
1041−-- XXX The Int64 default instance should be as fast.
1042−MK_EQ_INSTANCE(Int64)
1043−
1044−-- | If the type allows a byte-by-byte comparison this instance can be
1045−-- overlapped by a more specific instance that uses 'byteCmp'. Byte comparison
1046−-- can be significantly faster.
1047−--
1048−instance {-# OVERLAPPABLE #-} (Unbox a, Eq a) => Eq (Array a) where
657+-- XXX we are assuming that Unboxed equality means element equality. This may
658+-- or may not be correct? arrcmp is 40% faster compared to stream equality.
659+instance (Unbox a, Eq a) => Eq (Array a) where
1049660 {-# INLINE (==) #-}
1050661 arr1 == arr2 =
1051− -- Does unboxed byte equality mean element equality?
1052− -- XXX This is incorrect for sum types, as we may have some
1053− -- uninitialized memory in that case. If we always initialize the
1054− -- unused memory to zero we can use this.
1055− -- Byte comparison is 40% faster compared to stream equality.
1056− -- (==) EQ $ unsafeInlineIO $! unsafeThaw arr1 `MA.cmp` unsafeThaw arr2
1057− (toStreamD arr1 :: Stream Identity a) == toStreamD arr2
662+ (==) EQ $ unsafeInlineIO $! unsafeThaw arr1 `MA.cmp` unsafeThaw arr2
1058663
1059664instance (Unbox a, Ord a) => Ord (Array a) where
1060665 {-# INLINE compare #-}
@@ -1096,7 +701,7 @@ _toStreamD_ size Array{..} = D.Stream step arrStart
1096701 {-# INLINE_LATE step #-}
1097702 step _ p | p == arrEnd = return D.Stop
1098703 step _ p = liftIO $ do
1099− x <- peekAt p arrContents
704+ x <- peekByteIndex p arrContents
1100705 return $ D.Yield x (p + size)
1101706
1102707{-
@@ -1119,29 +724,15 @@ instance Foldable Array where
1119724-- Semigroup and Monoid
1120725-------------------------------------------------------------------------------
1121726
1122−-- XXX Deprecate and remove the Semigroup and Monoid instances because of
1123−-- potential misuse chances.
1124−
1125−-- | This should not be used for combining many or N arrays as it would copy
1126−-- the two arrays everytime to a new array. For coalescing multiple arrays use
1127−-- 'fromChunksK' instead.
1128727instance Unbox a => Semigroup (Array a) where
1129728 arr1 <> arr2 = unsafePerformIO $ splice arr1 arr2
1130729
1131−empty ::
1132−#ifdef DEVBUILD
1133− Unbox a =>
1134−#endif
1135− Array a
1136−empty = Array Unboxed.empty 0 0
1137−
1138−{-# DEPRECATED nil "Please use empty instead." #-}
1139730nil ::
1140731#ifdef DEVBUILD
1141732 Unbox a =>
1142733#endif
1143734 Array a
1144−nil = empty
735+nil = Array Unboxed.nil 0 0
1145736
1146737instance Unbox a => Monoid (Array a) where
1147738 mempty = nil
core/src/Streamly/Internal/Data/Builder.hs−2
@@ -16,9 +16,7 @@ module Streamly.Internal.Data.Builder
1616 )
1717where
1818
19−#if !MIN_VERSION_base(4,18,0)
2019import Control.Applicative (liftA2)
21−#endif
2220
2321------------------------------------------------------------------------------
2422-- The Builder type
core/src/Streamly/Internal/Data/Fold/Chunked.hs+15−18
@@ -6,7 +6,7 @@
66-- Stability : experimental
77-- Portability : GHC
88--
9−-- Use "Streamly.Data.Parser" instead.
9+-- Use "Streamly.Data.Parser.Chunked" instead.
1010--
1111-- Fold a stream of foreign arrays. @Fold m a b@ in this module works
1212-- on a stream of "Array a" and produces an output of type @b@.
@@ -19,19 +19,18 @@
1919-- folds in Data.Fold to correctly work on an array stream as if it is an
2020-- element stream. For example:
2121--
22−-- >> import qualified Streamly.Data.Fold as Fold
23−-- >> import qualified Streamly.Internal.Data.Array.Stream as ArrayStream
24−-- >> import qualified Streamly.Internal.Data.Fold.Chunked as ChunkFold
25−-- >> import qualified Streamly.Data.Stream as Stream
26−-- >> import qualified Streamly.Data.StreamK as StreamK
22+-- >>> import qualified Streamly.Data.Fold as Fold
23+-- >>> import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
24+-- >>> import qualified Streamly.Internal.Data.Fold.Chunked as ChunkFold
25+-- >>> import qualified Streamly.Data.Stream as Stream
26+-- >>> import qualified Streamly.Data.StreamK as StreamK
2727--
28−-- >> f = ChunkFold.fromFold (Fold.take 7 Fold.toList)
29−-- >> s = Stream.chunksOf 5 $ Stream.fromList "hello world"
30−-- >> ArrayStream.runArrayFold f (StreamK.fromStream s)
28+-- >>> f = ChunkFold.fromFold (Fold.take 7 Fold.toList)
29+-- >>> s = Stream.chunksOf 5 $ Stream.fromList "hello world"
30+-- >>> ArrayStream.runArrayFold f (StreamK.fromStream s)
3131-- Right "hello w"
3232--
3333module Streamly.Internal.Data.Fold.Chunked
34− {-# DEPRECATED "Please use Streamly.Data.Parser instead." #-}
3534 (
3635 ChunkFold (..)
3736
@@ -59,15 +58,14 @@ where
5958
6059#include "ArrayMacros.h"
6160
62−#if !MIN_VERSION_base(4,18,0)
6361import Control.Applicative (liftA2)
64−#endif
6562import Control.Exception (assert)
6663import Control.Monad.IO.Class (MonadIO(..))
6764import Data.Bifunctor (first)
6865import Data.Proxy (Proxy(..))
6966import Streamly.Internal.Data.Unbox (Unbox(..))
7067import GHC.Types (SPEC(..))
68+import Streamly.Internal.Data.MutArray.Type (touch)
7169import Streamly.Internal.Data.Array (Array(..))
7270import Streamly.Internal.Data.Parser (Initial(..), Step(..))
7371import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
@@ -103,8 +101,8 @@ newtype ChunkFold m a b = ChunkFold (ParserD.Parser (Array a) m b)
103101{-# INLINE fromFold #-}
104102fromFold :: forall m a b. (MonadIO m, Unbox a) =>
105103 Fold.Fold m a b -> ChunkFold m a b
106−fromFold (Fold.Fold fstep finitial _ ffinal) =
107− ChunkFold (ParserD.Parser step initial extract)
104+fromFold (Fold.Fold fstep finitial fextract) =
105+ ChunkFold (ParserD.Parser step initial (fmap (Done 0) . fextract))
108106
109107 where
110108
@@ -124,7 +122,7 @@ fromFold (Fold.Fold fstep finitial _ ffinal) =
124122 assert (cur == end) (return ())
125123 return $ Partial 0 fs
126124 goArray !_ !cur !fs = do
127− x <- liftIO $ peekAt cur contents
125+ x <- liftIO $ peekByteIndex cur contents
128126 res <- fstep fs x
129127 let elemSize = SIZE_OF(a)
130128 next = INDEX_NEXT(cur,a)
@@ -134,8 +132,6 @@ fromFold (Fold.Fold fstep finitial _ ffinal) =
134132 Fold.Partial fs1 ->
135133 goArray SPEC next fs1
136134
137− extract = fmap (Done 0) . ffinal
138−
139135-- | Convert an element 'ParserD.Parser' into an array stream fold. If the
140136-- parser fails the fold would throw an exception.
141137--
@@ -163,7 +159,8 @@ fromParserD (ParserD.Parser step1 initial1 extract1) =
163159 else return $ st (arrRem + n) fs1
164160
165161 goArray !_ !cur !fs = do
166− x <- liftIO $ peekAt cur contents
162+ x <- liftIO $ peekByteIndex cur contents
163+ liftIO $ touch contents
167164 res <- step1 fs x
168165 let elemSize = SIZE_OF(a)
169166 next = INDEX_NEXT(cur,a)
core/src/Streamly/Internal/Data/Fold/Combinators.hs+137−81
@@ -30,6 +30,8 @@ module Streamly.Internal.Data.Fold.Combinators
3030 -- *** Reducers
3131 , drainMapM
3232 , the
33+ , length
34+ , lengthGeneric
3335 , mean
3436 , rollingHash
3537 , defaultSalt
@@ -164,6 +166,8 @@ module Streamly.Internal.Data.Fold.Combinators
164166 -- ** Trimming
165167
166168 -- By elements
169+ , takeEndBy
170+ , takeEndBy_
167171 , takeEndBySeq
168172 , takeEndBySeq_
169173 {-
@@ -439,8 +443,8 @@ trace f = lmapM (tracing f)
439443-- /Pre-release/
440444{-# INLINE transform #-}
441445transform :: Monad m => Pipe m a b -> Fold m b c -> Fold m a c
442−transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract ffinal) =
443− Fold step initial extract final
446+transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract) =
447+ Fold step initial extract
444448
445449 where
446450
@@ -473,14 +477,10 @@ transform (Pipe pstep1 pstep2 pinitial) (Fold fstep finitial fextract ffinal) =
473477
474478 extract (Tuple' _ fs) = fextract fs
475479
476− final (Tuple' _ fs) = ffinal fs
477−
478480{-# INLINE scanWith #-}
479481scanWith :: Monad m => Bool -> Fold m a b -> Fold m b c -> Fold m a c
480−scanWith isMany
481− (Fold stepL initialL extractL finalL)
482− (Fold stepR initialR extractR finalR) =
483− Fold step initial extract final
482+scanWith isMany (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
483+ Fold step initial extract
484484
485485 where
486486
@@ -494,14 +494,15 @@ scanWith isMany
494494 Partial sR1 ->
495495 if isMany
496496 then runStep initialL sR1
497− else Done <$> finalR sR1
497+ else Done <$> extractR sR1
498498 Done bR -> return $ Done bR
499499 Partial sL -> do
500500 !b <- extractL sL
501501 rR <- stepR sR b
502− case rR of
503− Partial sR1 -> return $ Partial (sL, sR1)
504− Done bR -> finalL sL >> return (Done bR)
502+ return
503+ $ case rR of
504+ Partial sR1 -> Partial (sL, sR1)
505+ Done bR -> Done bR
505506
506507 initial = do
507508 r <- initialR
@@ -513,8 +514,6 @@ scanWith isMany
513514
514515 extract = extractR . snd
515516
516− final (sL, sR) = finalL sL *> finalR sR
517−
518517-- | Scan the input of a 'Fold' to change it in a stateful manner using another
519518-- 'Fold'. The scan stops as soon as the fold terminates.
520519--
@@ -566,7 +565,7 @@ deleteBy eq x0 = fmap extract $ foldl' step (Tuple' False Nothing)
566565--
567566{-# INLINE slide2 #-}
568567slide2 :: Monad m => Fold m (a, Maybe a) b -> Fold m a b
569−slide2 (Fold step1 initial1 extract1 final1) = Fold step initial extract final
568+slide2 (Fold step1 initial1 extract1) = Fold step initial extract
570569
571570 where
572571
@@ -578,8 +577,6 @@ slide2 (Fold step1 initial1 extract1 final1) = Fold step initial extract final
578577
579578 extract (Tuple' _ s) = extract1 s
580579
581− final (Tuple' _ s) = final1 s
582−
583580-- | Return the latest unique element using the supplied comparison function.
584581-- Returns 'Nothing' if the current element is same as the last element
585582-- otherwise returns 'Just'.
@@ -705,6 +702,30 @@ the = foldt' step initial id
705702-- To Summary
706703------------------------------------------------------------------------------
707704
705+-- | Like 'length', except with a more general 'Num' return value
706+--
707+-- Definition:
708+--
709+-- >>> lengthGeneric = fmap getSum $ Fold.foldMap (Sum . const 1)
710+-- >>> lengthGeneric = Fold.foldl' (\n _ -> n + 1) 0
711+--
712+-- /Pre-release/
713+{-# INLINE lengthGeneric #-}
714+lengthGeneric :: (Monad m, Num b) => Fold m a b
715+lengthGeneric = foldl' (\n _ -> n + 1) 0
716+
717+-- | Determine the length of the input stream.
718+--
719+-- Definition:
720+--
721+-- >>> length = Fold.lengthGeneric
722+-- >>> length = fmap getSum $ Fold.foldMap (Sum . const 1)
723+--
724+{-# INLINE length #-}
725+length :: Monad m => Fold m a Int
726+length = lengthGeneric
727+
728+
708729-- | Determine the sum of all elements of a stream of numbers. Returns additive
709730-- identity (@0@) when the stream is empty. Note that this is not numerically
710731-- stable for floating point numbers.
@@ -911,7 +932,7 @@ rollingHashFirstN n = take n rollingHash
911932--
912933{-# INLINE rollingMapM #-}
913934rollingMapM :: Monad m => (Maybe a -> a -> m b) -> Fold m a b
914−rollingMapM f = Fold step initial extract extract
935+rollingMapM f = Fold step initial extract
915936
916937 where
917938
@@ -1152,8 +1173,7 @@ head = one
11521173-- /Pre-release/
11531174{-# INLINE findM #-}
11541175findM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1155−findM predicate =
1156− Fold step (return $ Partial ()) extract extract
1176+findM predicate = Fold step (return $ Partial ()) (const $ return Nothing)
11571177
11581178 where
11591179
@@ -1164,8 +1184,6 @@ findM predicate =
11641184 else Partial ()
11651185 in f <$> predicate a
11661186
1167− extract = const $ return Nothing
1168−
11691187-- | Returns the first element that satisfies the given predicate.
11701188--
11711189{-# INLINE find #-}
@@ -1408,7 +1426,7 @@ splitAt n fld = splitWith (,) (take n fld)
14081426
14091427{-# INLINE takingEndByM #-}
14101428takingEndByM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1411−takingEndByM p = Fold step initial extract extract
1429+takingEndByM p = Fold step initial (return . toMaybe)
14121430
14131431 where
14141432
@@ -1421,8 +1439,6 @@ takingEndByM p = Fold step initial extract extract
14211439 then Done $ Just a
14221440 else Partial $ Just' a
14231441
1424− extract = return . toMaybe
1425−
14261442-- |
14271443--
14281444-- >>> takingEndBy p = Fold.takingEndByM (return . p)
@@ -1433,7 +1449,7 @@ takingEndBy p = takingEndByM (return . p)
14331449
14341450{-# INLINE takingEndByM_ #-}
14351451takingEndByM_ :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1436−takingEndByM_ p = Fold step initial extract extract
1452+takingEndByM_ p = Fold step initial (return . toMaybe)
14371453
14381454 where
14391455
@@ -1446,8 +1462,6 @@ takingEndByM_ p = Fold step initial extract extract
14461462 then Done Nothing
14471463 else Partial $ Just' a
14481464
1449− extract = return . toMaybe
1450−
14511465-- |
14521466--
14531467-- >>> takingEndBy_ p = Fold.takingEndByM_ (return . p)
@@ -1458,7 +1472,7 @@ takingEndBy_ p = takingEndByM_ (return . p)
14581472
14591473{-# INLINE droppingWhileM #-}
14601474droppingWhileM :: Monad m => (a -> m Bool) -> Fold m a (Maybe a)
1461−droppingWhileM p = Fold step initial extract extract
1475+droppingWhileM p = Fold step initial (return . toMaybe)
14621476
14631477 where
14641478
@@ -1473,8 +1487,6 @@ droppingWhileM p = Fold step initial extract extract
14731487 else Just' a
14741488 step _ a = return $ Partial $ Just' a
14751489
1476− extract = return . toMaybe
1477−
14781490-- |
14791491-- >>> droppingWhile p = Fold.droppingWhileM (return . p)
14801492--
@@ -1482,6 +1494,70 @@ droppingWhileM p = Fold step initial extract extract
14821494droppingWhile :: Monad m => (a -> Bool) -> Fold m a (Maybe a)
14831495droppingWhile p = droppingWhileM (return . p)
14841496
1497+-- Note: Keep this consistent with S.splitOn. In fact we should eliminate
1498+-- S.splitOn in favor of the fold.
1499+--
1500+-- XXX Use Fold.many instead once it is fixed.
1501+-- > Stream.splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
1502+
1503+-- | Like 'takeEndBy' but drops the element on which the predicate succeeds.
1504+--
1505+-- Example:
1506+--
1507+-- >>> input = Stream.fromList "hello\nthere\n"
1508+-- >>> line = Fold.takeEndBy_ (== '\n') Fold.toList
1509+-- >>> Stream.fold line input
1510+-- "hello"
1511+--
1512+-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1513+-- ["hello","there"]
1514+--
1515+{-# INLINE takeEndBy_ #-}
1516+takeEndBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1517+-- takeEndBy_ predicate = scanMaybe (takingEndBy_ predicate)
1518+takeEndBy_ predicate (Fold fstep finitial fextract) =
1519+ Fold step finitial fextract
1520+
1521+ where
1522+
1523+ step s a =
1524+ if not (predicate a)
1525+ then fstep s a
1526+ else Done <$> fextract s
1527+
1528+-- Note:
1529+-- > Stream.splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
1530+
1531+-- | Take the input, stop when the predicate succeeds taking the succeeding
1532+-- element as well.
1533+--
1534+-- Example:
1535+--
1536+-- >>> input = Stream.fromList "hello\nthere\n"
1537+-- >>> line = Fold.takeEndBy (== '\n') Fold.toList
1538+-- >>> Stream.fold line input
1539+-- "hello\n"
1540+--
1541+-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1542+-- ["hello\n","there\n"]
1543+--
1544+{-# INLINE takeEndBy #-}
1545+takeEndBy :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1546+-- takeEndBy predicate = scanMaybe (takingEndBy predicate)
1547+takeEndBy predicate (Fold fstep finitial fextract) =
1548+ Fold step finitial fextract
1549+
1550+ where
1551+
1552+ step s a = do
1553+ res <- fstep s a
1554+ if not (predicate a)
1555+ then return res
1556+ else do
1557+ case res of
1558+ Partial s1 -> Done <$> fextract s1
1559+ Done b -> return $ Done b
1560+
14851561------------------------------------------------------------------------------
14861562-- Binary splitting on a separator
14871563------------------------------------------------------------------------------
@@ -1514,8 +1590,8 @@ takeEndBySeq :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) =>
15141590 Array.Array a
15151591 -> Fold m a b
15161592 -> Fold m a b
1517−takeEndBySeq patArr (Fold fstep finitial fextract ffinal) =
1518− Fold step initial extract final
1593+takeEndBySeq patArr (Fold fstep finitial fextract) =
1594+ Fold step initial extract
15191595
15201596 where
15211597
@@ -1528,7 +1604,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract ffinal) =
15281604 | patLen == 0 ->
15291605 -- XXX Should we match nothing or everything on empty
15301606 -- pattern?
1531− -- Done <$> ffinal acc
1607+ -- Done <$> fextract acc
15321608 return $ Partial $ SplitOnSeqEmpty acc
15331609 | patLen == 1 -> do
15341610 pat <- liftIO $ Array.unsafeIndexIO 0 patArr
@@ -1579,7 +1655,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract ffinal) =
15791655 case res of
15801656 Partial s1
15811657 | pat /= x -> return $ Partial $ SplitOnSeqSingle s1 pat
1582− | otherwise -> Done <$> ffinal s1
1658+ | otherwise -> Done <$> fextract s1
15831659 Done b -> return $ Done b
15841660 step (SplitOnSeqWord s idx wrd) x = do
15851661 res <- fstep s x
@@ -1588,7 +1664,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract ffinal) =
15881664 Partial s1
15891665 | idx == maxIndex -> do
15901666 if wrd1 .&. wordMask == wordPat
1591− then Done <$> ffinal s1
1667+ then Done <$> fextract s1
15921668 else return $ Partial $ SplitOnSeqWordLoop s1 wrd1
15931669 | otherwise ->
15941670 return $ Partial $ SplitOnSeqWord s1 (idx + 1) wrd1
@@ -1599,7 +1675,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract ffinal) =
15991675 case res of
16001676 Partial s1
16011677 | wrd1 .&. wordMask == wordPat ->
1602− Done <$> ffinal s1
1678+ Done <$> fextract s1
16031679 | otherwise ->
16041680 return $ Partial $ SplitOnSeqWordLoop s1 wrd1
16051681 Done b -> return $ Done b
@@ -1613,7 +1689,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract ffinal) =
16131689 let fld = Ring.unsafeFoldRing (Ring.ringBound rb)
16141690 let !ringHash = fld addCksum 0 rb
16151691 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1616− then Done <$> ffinal s1
1692+ then Done <$> fextract s1
16171693 else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
16181694 else
16191695 return $ Partial $ SplitOnSeqKR s1 (idx + 1) rb rh1
@@ -1626,11 +1702,11 @@ takeEndBySeq patArr (Fold fstep finitial fextract ffinal) =
16261702 rh1 <- liftIO $ Ring.unsafeInsert rb rh x
16271703 let ringHash = deltaCksum cksum old x
16281704 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1629− then Done <$> ffinal s1
1705+ then Done <$> fextract s1
16301706 else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
16311707 Done b -> return $ Done b
16321708
1633− extractFunc fex state =
1709+ extract state =
16341710 let st =
16351711 case state of
16361712 SplitOnSeqEmpty s -> s
@@ -1639,11 +1715,7 @@ takeEndBySeq patArr (Fold fstep finitial fextract ffinal) =
16391715 SplitOnSeqWordLoop s _ -> s
16401716 SplitOnSeqKR s _ _ _ -> s
16411717 SplitOnSeqKRLoop s _ _ _ -> s
1642− in fex st
1643−
1644− extract state = extractFunc fextract state
1645−
1646− final state = extractFunc ffinal state
1718+ in fextract st
16471719
16481720-- | Like 'takeEndBySeq' but discards the matched sequence.
16491721--
@@ -1654,8 +1726,8 @@ takeEndBySeq_ :: forall m a b. (MonadIO m, Storable a, Unbox a, Enum a, Eq a) =>
16541726 Array.Array a
16551727 -> Fold m a b
16561728 -> Fold m a b
1657−takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
1658− Fold step initial extract final
1729+takeEndBySeq_ patArr (Fold fstep finitial fextract) =
1730+ Fold step initial extract
16591731
16601732 where
16611733
@@ -1668,7 +1740,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
16681740 | patLen == 0 ->
16691741 -- XXX Should we match nothing or everything on empty
16701742 -- pattern?
1671− -- Done <$> ffinal acc
1743+ -- Done <$> fextract acc
16721744 return $ Partial $ SplitOnSeqEmpty acc
16731745 | patLen == 1 -> do
16741746 pat <- liftIO $ Array.unsafeIndexIO 0 patArr
@@ -1725,13 +1797,13 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
17251797 case res of
17261798 Partial s1 -> return $ Partial $ SplitOnSeqSingle s1 pat
17271799 Done b -> return $ Done b
1728− else Done <$> ffinal s
1800+ else Done <$> fextract s
17291801 step (SplitOnSeqWord s idx wrd) x = do
17301802 let wrd1 = addToWord wrd x
17311803 if idx == maxIndex
17321804 then do
17331805 if wrd1 .&. wordMask == wordPat
1734− then Done <$> ffinal s
1806+ then Done <$> fextract s
17351807 else return $ Partial $ SplitOnSeqWordLoop s wrd1
17361808 else return $ Partial $ SplitOnSeqWord s (idx + 1) wrd1
17371809 step (SplitOnSeqWordLoop s wrd) x = do
@@ -1742,7 +1814,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
17421814 case res of
17431815 Partial s1
17441816 | wrd1 .&. wordMask == wordPat ->
1745− Done <$> ffinal s1
1817+ Done <$> fextract s1
17461818 | otherwise ->
17471819 return $ Partial $ SplitOnSeqWordLoop s1 wrd1
17481820 Done b -> return $ Done b
@@ -1753,7 +1825,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
17531825 let fld = Ring.unsafeFoldRing (Ring.ringBound rb)
17541826 let !ringHash = fld addCksum 0 rb
17551827 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1756− then Done <$> ffinal s
1828+ then Done <$> fextract s
17571829 else return $ Partial $ SplitOnSeqKRLoop s ringHash rb rh1
17581830 else return $ Partial $ SplitOnSeqKR s (idx + 1) rb rh1
17591831 step (SplitOnSeqKRLoop s cksum rb rh) x = do
@@ -1764,7 +1836,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
17641836 rh1 <- liftIO $ Ring.unsafeInsert rb rh x
17651837 let ringHash = deltaCksum cksum old x
17661838 if ringHash == patHash && Ring.unsafeEqArray rb rh1 patArr
1767− then Done <$> ffinal s1
1839+ then Done <$> fextract s1
17681840 else return $ Partial $ SplitOnSeqKRLoop s1 ringHash rb rh1
17691841 Done b -> return $ Done b
17701842
@@ -1772,10 +1844,10 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
17721844 -- terminates early inside extract, we may still have buffered data
17731845 -- remaining which will be lost if we do not communicate that to the
17741846 -- driver.
1775− extractFunc fex state = do
1847+ extract state = do
17761848 let consumeWord s n wrd = do
17771849 if n == 0
1778− then fex s
1850+ then fextract s
17791851 else do
17801852 let old = elemMask .&. (wrd `shiftR` (elemBits * (n - 1)))
17811853 r <- fstep s (toEnum $ fromIntegral old)
@@ -1785,7 +1857,7 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
17851857
17861858 let consumeRing s n rb rh =
17871859 if n == 0
1788− then fex s
1860+ then fextract s
17891861 else do
17901862 old <- liftIO $ peek rh
17911863 let rh1 = Ring.advance rb rh
@@ -1795,17 +1867,13 @@ takeEndBySeq_ patArr (Fold fstep finitial fextract ffinal) =
17951867 Done b -> return b
17961868
17971869 case state of
1798− SplitOnSeqEmpty s -> fex s
1799− SplitOnSeqSingle s _ -> fex s
1870+ SplitOnSeqEmpty s -> fextract s
1871+ SplitOnSeqSingle s _ -> fextract s
18001872 SplitOnSeqWord s idx wrd -> consumeWord s idx wrd
18011873 SplitOnSeqWordLoop s wrd -> consumeWord s patLen wrd
18021874 SplitOnSeqKR s idx rb _ -> consumeRing s idx rb (Ring.startOf rb)
18031875 SplitOnSeqKRLoop s _ rb rh -> consumeRing s patLen rb rh
18041876
1805− extract state = extractFunc fextract state
1806−
1807− final state = extractFunc ffinal state
1808−
18091877------------------------------------------------------------------------------
18101878-- Distributing
18111879------------------------------------------------------------------------------
@@ -2249,8 +2317,8 @@ toStreamRev = fmap StreamD.fromList toListRev
22492317-- /Pre-release/
22502318{-# INLINE unfoldMany #-}
22512319unfoldMany :: Monad m => Unfold m a b -> Fold m b c -> Fold m a c
2252−unfoldMany (Unfold ustep inject) (Fold fstep initial extract final) =
2253− Fold consume initial extract final
2320+unfoldMany (Unfold ustep inject) (Fold fstep initial extract) =
2321+ Fold consume initial extract
22542322
22552323 where
22562324
@@ -2277,7 +2345,7 @@ bottomBy :: (MonadIO m, Unbox a) =>
22772345 (a -> a -> Ordering)
22782346 -> Int
22792347 -> Fold m a (MutArray a)
2280−bottomBy cmp n = Fold step initial extract extract
2348+bottomBy cmp n = Fold step initial extract
22812349
22822350 where
22832351
@@ -2372,8 +2440,8 @@ intersperseWithQuotes
23722440 quote
23732441 esc
23742442 separator
2375− (Fold stepL initialL _ finalL)
2376− (Fold stepR initialR extractR finalR) = Fold step initial extract final
2443+ (Fold stepL initialL extractL)
2444+ (Fold stepR initialR extractR) = Fold step initial extract
23772445
23782446 where
23792447
@@ -2416,13 +2484,11 @@ intersperseWithQuotes
24162484 r <- stepL sL a
24172485 case r of
24182486 Partial s -> return $ Partial (nextState sR s)
2419− Done _ -> do
2420− _ <- finalR sR
2421− error "Collecting fold finished inside quote"
2487+ Done _ -> error "Collecting fold finished inside quote"
24222488
24232489 step (IntersperseQUnquoted sR sL) a
24242490 | a == separator = do
2425− b <- finalL sL
2491+ b <- extractL sL
24262492 collect IntersperseQUnquoted sR b
24272493 | a == quote = processQuoted a sL sR IntersperseQQuoted
24282494 | otherwise = process a sL sR IntersperseQUnquoted
@@ -2440,13 +2506,3 @@ intersperseWithQuotes
24402506 error "intersperseWithQuotes: finished inside quote"
24412507 extract (IntersperseQQuotedEsc _ _) =
24422508 error "intersperseWithQuotes: finished inside quote, at escape char"
2443−
2444− final (IntersperseQUnquoted sR sL) = finalL sL *> finalR sR
2445− final (IntersperseQQuoted sR sL) = do
2446− _ <- finalR sR
2447− _ <- finalL sL
2448− error "intersperseWithQuotes: finished inside quote"
2449− final (IntersperseQQuotedEsc sR sL) = do
2450− _ <- finalR sR
2451− _ <- finalL sL
2452− error "intersperseWithQuotes: finished inside quote, at escape char"
core/src/Streamly/Internal/Data/Fold/Container.hs+28−73
@@ -270,15 +270,14 @@ demuxGeneric :: (Monad m, IsMap f, Traversable f) =>
270270 (a -> Key f)
271271 -> (a -> m (Fold m a b))
272272 -> Fold m a (m (f b), Maybe (Key f, b))
273−demuxGeneric getKey getFold =
274− Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
273+demuxGeneric getKey getFold = fmap extract $ foldlM' step initial
275274
276275 where
277276
278277 initial = return $ Tuple' IsMap.mapEmpty Nothing
279278
280279 {-# INLINE runFold #-}
281− runFold kv (Fold step1 initial1 extract1 final1) (k, a) = do
280+ runFold kv (Fold step1 initial1 extract1) (k, a) = do
282281 res <- initial1
283282 case res of
284283 Partial s -> do
@@ -286,7 +285,7 @@ demuxGeneric getKey getFold =
286285 return
287286 $ case res1 of
288287 Partial _ ->
289− let fld = Fold step1 (return res1) extract1 final1
288+ let fld = Fold step1 (return res1) extract1
290289 in Tuple' (IsMap.mapInsert k fld kv) Nothing
291290 Done b -> Tuple' (IsMap.mapDelete k kv) (Just (k, b))
292291 Done b -> return $ Tuple' kv (Just (k, b))
@@ -299,25 +298,15 @@ demuxGeneric getKey getFold =
299298 runFold kv fld (k, a)
300299 Just f -> runFold kv f (k, a)
301300
302− extract (Tuple' kv x) = return (Prelude.mapM f kv, x)
301+ extract (Tuple' kv x) = (Prelude.mapM f kv, x)
303302
304303 where
305304
306− f (Fold _ i e _) = do
305+ f (Fold _ i e) = do
307306 r <- i
308307 case r of
309308 Partial s -> e s
310− _ -> error "demuxGeneric: unreachable code"
311−
312− final (Tuple' kv x) = return (Prelude.mapM f kv, x)
313−
314− where
315−
316− f (Fold _ i _ fin) = do
317− r <- i
318− case r of
319− Partial s -> fin s
320− _ -> error "demuxGeneric: unreachable code"
309+ Done b -> return b
321310
322311-- | @demux getKey getFold@: In a key value stream, fold values corresponding
323312-- to each key using a key specific fold. @getFold@ is invoked to generate a
@@ -359,36 +348,35 @@ demuxGenericIO :: (MonadIO m, IsMap f, Traversable f) =>
359348 (a -> Key f)
360349 -> (a -> m (Fold m a b))
361350 -> Fold m a (m (f b), Maybe (Key f, b))
362−demuxGenericIO getKey getFold =
363− Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
351+demuxGenericIO getKey getFold = fmap extract $ foldlM' step initial
364352
365353 where
366354
367355 initial = return $ Tuple' IsMap.mapEmpty Nothing
368356
369357 {-# INLINE initFold #-}
370− initFold kv (Fold step1 initial1 extract1 final1) (k, a) = do
358+ initFold kv (Fold step1 initial1 extract1) (k, a) = do
371359 res <- initial1
372360 case res of
373361 Partial s -> do
374362 res1 <- step1 s a
375363 case res1 of
376364 Partial _ -> do
377− let fld = Fold step1 (return res1) extract1 final1
365+ let fld = Fold step1 (return res1) extract1
378366 ref <- liftIO $ newIORef fld
379367 return $ Tuple' (IsMap.mapInsert k ref kv) Nothing
380368 Done b -> return $ Tuple' kv (Just (k, b))
381369 Done b -> return $ Tuple' kv (Just (k, b))
382370
383371 {-# INLINE runFold #-}
384− runFold kv ref (Fold step1 initial1 extract1 final1) (k, a) = do
372+ runFold kv ref (Fold step1 initial1 extract1) (k, a) = do
385373 res <- initial1
386374 case res of
387375 Partial s -> do
388376 res1 <- step1 s a
389377 case res1 of
390378 Partial _ -> do
391− let fld = Fold step1 (return res1) extract1 final1
379+ let fld = Fold step1 (return res1) extract1
392380 liftIO $ writeIORef ref fld
393381 return $ Tuple' kv Nothing
394382 Done b ->
@@ -406,27 +394,16 @@ demuxGenericIO getKey getFold =
406394 f <- liftIO $ readIORef ref
407395 runFold kv ref f (k, a)
408396
409− extract (Tuple' kv x) = return (Prelude.mapM f kv, x)
397+ extract (Tuple' kv x) = (Prelude.mapM f kv, x)
410398
411399 where
412400
413401 f ref = do
414− Fold _ i e _ <- liftIO $ readIORef ref
402+ (Fold _ i e) <- liftIO $ readIORef ref
415403 r <- i
416404 case r of
417405 Partial s -> e s
418− _ -> error "demuxGenericIO: unreachable code"
419−
420− final (Tuple' kv x) = return (Prelude.mapM f kv, x)
421−
422− where
423−
424− f ref = do
425− Fold _ i _ fin <- liftIO $ readIORef ref
426− r <- i
427− case r of
428− Partial s -> fin s
429− _ -> error "demuxGenericIO: unreachable code"
406+ Done b -> return b
430407
431408-- | This is specialized version of 'demux' that uses mutable IO cells as
432409-- fold accumulators for better performance.
@@ -441,13 +418,6 @@ demuxIO :: (MonadIO m, Ord k) =>
441418 -> Fold m a (m (Map k b), Maybe (k, b))
442419demuxIO = demuxGenericIO
443420
444−-- | Fold a key value stream to a key-value Map. If the same key appears
445−-- multiple times, only the last value is retained.
446−{-# INLINE kvToMapOverwriteGeneric #-}
447−kvToMapOverwriteGeneric :: (Monad m, IsMap f) => Fold m (Key f, a) (f a)
448−kvToMapOverwriteGeneric =
449− foldl' (\kv (k, v) -> IsMap.mapInsert k v kv) IsMap.mapEmpty
450−
451421{-# INLINE demuxToContainer #-}
452422demuxToContainer :: (Monad m, IsMap f, Traversable f) =>
453423 (a -> Key f) -> (a -> m (Fold m a b)) -> Fold m a (f b)
@@ -535,8 +505,8 @@ classifyGeneric :: (Monad m, IsMap f, Traversable f, Ord (Key f)) =>
535505 -- for that use case. We return an action because we want it to be lazy so
536506 -- that the downstream consumers can choose to process or discard it.
537507 (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
538−classifyGeneric f (Fold step1 initial1 extract1 final1) =
539− Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
508+classifyGeneric f (Fold step1 initial1 extract1) =
509+ fmap extract $ foldlM' step initial
540510
541511 where
542512
@@ -573,16 +543,7 @@ classifyGeneric f (Fold step1 initial1 extract1 final1) =
573543 let kv1 = IsMap.mapDelete k kv
574544 in Tuple3' kv1 (Set.insert k set) (Just (k, b))
575545
576− extract (Tuple3' kv _ x) = return (Prelude.mapM extract1 kv, x)
577−
578− final (Tuple3' kv set x) = return (IsMap.mapTraverseWithKey f1 kv, x)
579−
580− where
581−
582− f1 k s = do
583− if Set.member k set
584− then extract1 s
585− else final1 s
546+ extract (Tuple3' kv _ x) = (Prelude.mapM extract1 kv, x)
586547
587548-- | Folds the values for each key using the supplied fold. When scanning, as
588549-- soon as the fold is complete, its result is available in the second
@@ -607,8 +568,8 @@ classify = classifyGeneric
607568{-# INLINE classifyGenericIO #-}
608569classifyGenericIO :: (MonadIO m, IsMap f, Traversable f, Ord (Key f)) =>
609570 (a -> Key f) -> Fold m a b -> Fold m a (m (f b), Maybe (Key f, b))
610−classifyGenericIO f (Fold step1 initial1 extract1 final1) =
611− Fold (\s a -> Partial <$> step s a) (Partial <$> initial) extract final
571+classifyGenericIO f (Fold step1 initial1 extract1) =
572+ fmap extract $ foldlM' step initial
612573
613574 where
614575
@@ -647,21 +608,8 @@ classifyGenericIO f (Fold step1 initial1 extract1 final1) =
647608 in return
648609 $ Tuple3' kv1 (Set.insert k set) (Just (k, b))
649610
650− extract (Tuple3' kv _ x) = return (Prelude.mapM g kv, x)
651−
652− where
653−
654− g ref = liftIO (readIORef ref) >>= extract1
655−
656− final (Tuple3' kv set x) = return (IsMap.mapTraverseWithKey g kv, x)
657−
658− where
659−
660− g k ref = do
661− s <- liftIO $ readIORef ref
662− if Set.member k set
663− then extract1 s
664− else final1 s
611+ extract (Tuple3' kv _ x) =
612+ (Prelude.mapM (\ref -> liftIO (readIORef ref) >>= extract1) kv, x)
665613
666614-- | Same as classify except that it uses mutable IORef cells in the
667615-- Map providing better performance. Be aware that if this is used as a scan,
@@ -676,6 +624,13 @@ classifyIO :: (MonadIO m, Ord k) =>
676624 (a -> k) -> Fold m a b -> Fold m a (m (Map k b), Maybe (k, b))
677625classifyIO = classifyGenericIO
678626
627+-- | Fold a key value stream to a key-value Map. If the same key appears
628+-- multiple times, only the last value is retained.
629+{-# INLINE kvToMapOverwriteGeneric #-}
630+kvToMapOverwriteGeneric :: (Monad m, IsMap f) => Fold m (Key f, a) (f a)
631+kvToMapOverwriteGeneric =
632+ foldl' (\kv (k, v) -> IsMap.mapInsert k v kv) IsMap.mapEmpty
633+
679634{-# INLINE toContainer #-}
680635toContainer :: (Monad m, IsMap f, Traversable f, Ord (Key f)) =>
681636 (a -> Key f) -> Fold m a b -> Fold m a (f b)
core/src/Streamly/Internal/Data/Fold/Step.hs+1−1
@@ -40,7 +40,7 @@ data Step s b
4040 = Partial !s
4141 | Done !b
4242
43−-- | 'first' maps over the fold state and 'second' maps over the fold result.
43+-- | 'first' maps over 'Partial' and 'second' maps over 'Done'.
4444--
4545instance Bifunctor Step where
4646 {-# INLINE bimap #-}
core/src/Streamly/Internal/Data/Fold/Tee.hs−2
@@ -16,9 +16,7 @@ module Streamly.Internal.Data.Fold.Tee
1616 )
1717where
1818
19−#if !MIN_VERSION_base(4,18,0)
2019import Control.Applicative (liftA2)
21−#endif
2220import Streamly.Internal.Data.Fold.Type (Fold)
2321
2422import qualified Streamly.Internal.Data.Fold.Type as Fold
core/src/Streamly/Internal/Data/Fold/Type.hs+114−338
@@ -367,8 +367,6 @@ module Streamly.Internal.Data.Fold.Type
367367 , toList
368368 , toStreamK
369369 , toStreamKRev
370− , lengthGeneric
371− , length
372370
373371 -- * Combinators
374372
@@ -393,8 +391,6 @@ module Streamly.Internal.Data.Fold.Type
393391 -- ** Trimming
394392 , take
395393 , taking
396− , takeEndBy_
397− , takeEndBy
398394 , dropping
399395
400396 -- ** Sequential application
@@ -446,10 +442,8 @@ where
446442
447443#include "inline.hs"
448444
449−#if !MIN_VERSION_base(4,18,0)
450445import Control.Applicative (liftA2)
451−#endif
452−import Control.Monad ((>=>), void)
446+import Control.Monad ((>=>))
453447import Data.Bifunctor (Bifunctor(..))
454448import Data.Either (fromLeft, fromRight, isLeft, isRight)
455449import Data.Functor.Identity (Identity(..))
@@ -475,64 +469,23 @@ import Streamly.Internal.Data.Fold.Step
475469-- The type @b@ is the accumulator of the writer. That's the reason the
476470-- default folds in various modules are called "write".
477471
478−-- An alternative to using an "extract" function is to use "Partial s b" style
479−-- partial value so that we always emit the output value and there is no need
480−-- to extract. Then extract can be used for cleanup purposes. But in this case
481−-- in some cases we may need a "Continue" constructor where an output value is
482−-- not available, this was implicit earlier. Also, "b" should be lazy here so
483−-- that we do not always compute it even if we do not need it.
484−--
485−-- Partial s b --> extract :: s -> b
486−-- Continue --> extract :: s -> Maybe b
487−--
488−-- But keeping 'b' lazy does not let the fold optimize well. It leads to
489−-- significant regressions in the key-value folds.
490−--
491−-- The "final" function complicates combinators that take other folds as
492−-- argument because we need to call their finalizers at right places. An
493−-- alternative to reduce this complexity where it is not required is to use a
494−-- separate type for bracketed folds but then we need to manage the complexity
495−-- of two different fold types.
496−
497−-- The "final" function could be (s -> m (Step s b)), like in parsers so that
498−-- it can be called in a loop to drain the fold.
499−
500−-- | The type @Fold m a b@ represents a consumer of an input stream of values
501−-- of type @a@ and returning a final value of type @b@ in 'Monad' @m@. The
502−-- constructor of a fold is @Fold step initial extract final@.
503−--
504−-- The fold uses an internal state of type @s@. The initial value of the state
505−-- @s@ is created by @initial@. This function is called once and only once
506−-- before the fold starts consuming input. Any resource allocation can be done
507−-- in this function.
508−--
509−-- The @step@ function is called on each input, it consumes an input and
510−-- returns the next intermediate state (see 'Step') or the final result @b@ if
511−-- the fold terminates.
512−--
513−-- If the fold is used as a scan, the @extract@ function is used by the scan
514−-- driver to map the current state @s@ of the fold to the fold result. Thus
515−-- @extract@ can be called multiple times. In some folds, where scanning does
516−-- not make sense, this function is left unimplemented; such folds cannot be
517−-- used as scans.
518−--
519−-- Before a fold terminates, @final@ is called once and only once (unless the
520−-- fold terminated in @initial@ itself). Any resources allocated by @initial@
521−-- can be released in @final@. In folds that do not require any cleanup
522−-- @extract@ and @final@ are typically the same.
523−--
524−-- When implementing fold combinators, care should be taken to cleanup any
525−-- state of the argument folds held by the fold by calling the respective
526−-- @final@ at all exit points of the fold. Also, @final@ should not be called
527−-- more than once. Note that if a fold terminates by 'Done' constructor, there
528−-- is no state to cleanup.
472+-- | The type @Fold m a b@ having constructor @Fold step initial extract@
473+-- represents a fold over an input stream of values of type @a@ to a final
474+-- value of type @b@ in 'Monad' @m@.
475+--
476+-- The fold uses an intermediate state @s@ as accumulator, the type @s@ is
477+-- internal to the specific fold definition. The initial value of the fold
478+-- state @s@ is returned by @initial@. The @step@ function consumes an input
479+-- and either returns the final result @b@ if the fold is done or the next
480+-- intermediate state (see 'Step'). At any point the fold driver can extract
481+-- the result from the intermediate state using the @extract@ function.
529482--
530483-- NOTE: The constructor is not yet released, smart constructors are provided
531484-- to create folds.
532485--
533486data Fold m a b =
534− -- | @Fold@ @step@ @initial@ @extract@ @final@
535− forall s. Fold (s -> a -> m (Step s b)) (m (Step s b)) (s -> m b) (s -> m b)
487+ -- | @Fold @ @ step @ @ initial @ @ extract@
488+ forall s. Fold (s -> a -> m (Step s b)) (m (Step s b)) (s -> m b)
536489
537490------------------------------------------------------------------------------
538491-- Mapping on the output
@@ -542,8 +495,7 @@ data Fold m a b =
542495--
543496{-# INLINE rmapM #-}
544497rmapM :: Monad m => (b -> m c) -> Fold m a b -> Fold m a c
545−rmapM f (Fold step initial extract final) =
546− Fold step1 initial1 (extract >=> f) (final >=> f)
498+rmapM f (Fold step initial extract) = Fold step1 initial1 (extract >=> f)
547499
548500 where
549501
@@ -574,7 +526,6 @@ foldl' step initial =
574526 (\s a -> return $ Partial $ step s a)
575527 (return (Partial initial))
576528 return
577− return
578529
579530-- | Make a fold from a left fold style monadic step function and initial value
580531-- of the accumulator.
@@ -589,7 +540,7 @@ foldl' step initial =
589540{-# INLINE foldlM' #-}
590541foldlM' :: Monad m => (b -> a -> m b) -> m b -> Fold m a b
591542foldlM' step initial =
592− Fold (\s a -> Partial <$> step s a) (Partial <$> initial) return return
543+ Fold (\s a -> Partial <$> step s a) (Partial <$> initial) return
593544
594545-- | Make a strict left fold, for non-empty streams, using first element as the
595546-- starting value. Returns Nothing if the stream is empty.
@@ -687,11 +638,7 @@ foldrM' g z =
687638{-# INLINE foldt' #-}
688639foldt' :: Monad m => (s -> a -> Step s b) -> Step s b -> (s -> b) -> Fold m a b
689640foldt' step initial extract =
690− Fold
691− (\s a -> return $ step s a)
692− (return initial)
693− (return . extract)
694− (return . extract)
641+ Fold (\s a -> return $ step s a) (return initial) (return . extract)
695642
696643-- | Make a terminating fold with an effectful step function and initial state,
697644-- and a state extraction function.
@@ -704,7 +651,7 @@ foldt' step initial extract =
704651--
705652{-# INLINE foldtM' #-}
706653foldtM' :: (s -> a -> m (Step s b)) -> m (Step s b) -> (s -> m b) -> Fold m a b
707−foldtM' step initial extract = Fold step initial extract extract
654+foldtM' = Fold
708655
709656------------------------------------------------------------------------------
710657-- Refold
@@ -718,7 +665,7 @@ foldtM' step initial extract = Fold step initial extract extract
718665-- /Internal/
719666fromRefold :: Refold m c a b -> c -> Fold m a b
720667fromRefold (Refold step inject extract) c =
721− Fold step (inject c) extract extract
668+ Fold step (inject c) extract
722669
723670------------------------------------------------------------------------------
724671-- Basic Folds
@@ -771,29 +718,6 @@ toStreamKRev = foldl' (flip K.cons) K.nil
771718toStreamK :: Monad m => Fold m a (K.StreamK n a)
772719toStreamK = foldr K.cons K.nil
773720
774−-- | Like 'length', except with a more general 'Num' return value
775−--
776−-- Definition:
777−--
778−-- >>> lengthGeneric = fmap getSum $ Fold.foldMap (Sum . const 1)
779−-- >>> lengthGeneric = Fold.foldl' (\n _ -> n + 1) 0
780−--
781−-- /Pre-release/
782−{-# INLINE lengthGeneric #-}
783−lengthGeneric :: (Monad m, Num b) => Fold m a b
784−lengthGeneric = foldl' (\n _ -> n + 1) 0
785−
786−-- | Determine the length of the input stream.
787−--
788−-- Definition:
789−--
790−-- >>> length = Fold.lengthGeneric
791−-- >>> length = fmap getSum $ Fold.foldMap (Sum . const 1)
792−--
793−{-# INLINE length #-}
794−length :: Monad m => Fold m a Int
795−length = lengthGeneric
796−
797721------------------------------------------------------------------------------
798722-- Instances
799723------------------------------------------------------------------------------
@@ -801,8 +725,7 @@ length = lengthGeneric
801725-- | Maps a function on the output of the fold (the type @b@).
802726instance Functor m => Functor (Fold m a) where
803727 {-# INLINE fmap #-}
804− fmap f (Fold step1 initial1 extract final) =
805− Fold step initial (fmap2 f extract) (fmap2 f final)
728+ fmap f (Fold step1 initial1 extract) = Fold step initial (fmap2 f extract)
806729
807730 where
808731
@@ -837,7 +760,7 @@ instance Functor m => Functor (Fold m a) where
837760--
838761{-# INLINE fromPure #-}
839762fromPure :: Applicative m => b -> Fold m a b
840−fromPure b = Fold undefined (pure $ Done b) pure pure
763+fromPure b = Fold undefined (pure $ Done b) pure
841764
842765-- | Make a fold that yields the result of the supplied effectful action
843766-- without consuming any further input.
@@ -846,7 +769,7 @@ fromPure b = Fold undefined (pure $ Done b) pure pure
846769--
847770{-# INLINE fromEffect #-}
848771fromEffect :: Applicative m => m b -> Fold m a b
849−fromEffect b = Fold undefined (Done <$> b) pure pure
772+fromEffect b = Fold undefined (Done <$> b) pure
850773
851774{-# ANN type SeqFoldState Fuse #-}
852775data SeqFoldState sl f sr = SeqFoldL !sl | SeqFoldR !f !sr
@@ -871,20 +794,16 @@ data SeqFoldState sl f sr = SeqFoldL !sl | SeqFoldR !f !sr
871794-- complexity, because each composition adds a new branch that each subsequent
872795-- fold's input element has to traverse, therefore, it cannot scale to a large
873796-- number of compositions. After around 100 compositions the performance starts
874−-- dipping rapidly compared to a CPS style implementation.
875−--
876−-- For larger number of compositions you can convert the fold to a parser and
877−-- use ParserK.
797+-- dipping rapidly compared to a CPS style implementation. When you need
798+-- scaling use parser monad instead.
878799--
879800-- /Time: O(n^2) where n is the number of compositions./
880801--
881802{-# INLINE splitWith #-}
882803splitWith :: Monad m =>
883804 (a -> b -> c) -> Fold m x a -> Fold m x b -> Fold m x c
884−splitWith func
885− (Fold stepL initialL _ finalL)
886− (Fold stepR initialR _ finalR) =
887− Fold step initial extract final
805+splitWith func (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
806+ Fold step initial extract
888807
889808 where
890809
@@ -901,18 +820,13 @@ splitWith func
901820 step (SeqFoldL st) a = runL (stepL st a)
902821 step (SeqFoldR f st) a = runR (stepR st a) f
903822
904− -- XXX splitWith should not be used for scanning
905− -- It would rarely make sense and resource tracking and cleanup would be
906− -- expensive. especially when multiple splitWith are chained.
907− extract _ = error "splitWith: cannot be used for scanning"
908−
909− final (SeqFoldR f sR) = fmap f (finalR sR)
910− final (SeqFoldL sL) = do
911− rL <- finalL sL
823+ extract (SeqFoldR f sR) = fmap f (extractR sR)
824+ extract (SeqFoldL sL) = do
825+ rL <- extractL sL
912826 res <- initialR
913827 fmap (func rL)
914828 $ case res of
915− Partial sR -> finalR sR
829+ Partial sR -> extractR sR
916830 Done rR -> return rR
917831
918832{-# DEPRECATED serialWith "Please use \"splitWith\" instead" #-}
@@ -932,8 +846,8 @@ data SeqFoldState_ sl sr = SeqFoldL_ !sl | SeqFoldR_ !sr
932846--
933847{-# INLINE split_ #-}
934848split_ :: Monad m => Fold m x a -> Fold m x b -> Fold m x b
935−split_ (Fold stepL initialL _ finalL) (Fold stepR initialR _ finalR) =
936− Fold step initial extract final
849+split_ (Fold stepL initialL _) (Fold stepR initialR extractR) =
850+ Fold step initial extract
937851
938852 where
939853
@@ -956,16 +870,11 @@ split_ (Fold stepL initialL _ finalL) (Fold stepR initialR _ finalR) =
956870 resR <- stepR st a
957871 return $ first SeqFoldR_ resR
958872
959− -- XXX split_ should not be used for scanning
960− -- See splitWith for more details.
961− extract _ = error "split_: cannot be used for scanning"
962−
963− final (SeqFoldR_ sR) = finalR sR
964− final (SeqFoldL_ sL) = do
965− _ <- finalL sL
873+ extract (SeqFoldR_ sR) = extractR sR
874+ extract (SeqFoldL_ _) = do
966875 res <- initialR
967876 case res of
968− Partial sR -> finalR sR
877+ Partial sR -> extractR sR
969878 Done rR -> return rR
970879
971880-- | 'Applicative' form of 'splitWith'. Split the input serially over two
@@ -1013,10 +922,8 @@ data TeeState sL sR bL bR
1013922--
1014923{-# INLINE teeWith #-}
1015924teeWith :: Monad m => (a -> b -> c) -> Fold m x a -> Fold m x b -> Fold m x c
1016−teeWith f
1017− (Fold stepL initialL extractL finalL)
1018− (Fold stepR initialR extractR finalR) =
1019− Fold step initial extract final
925+teeWith f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
926+ Fold step initial extract
1020927
1021928 where
1022929
@@ -1043,10 +950,6 @@ teeWith f
1043950 extract (TeeLeft bR sL) = (`f` bR) <$> extractL sL
1044951 extract (TeeRight bL sR) = f bL <$> extractR sR
1045952
1046− final (TeeBoth sL sR) = f <$> finalL sL <*> finalR sR
1047− final (TeeLeft bR sL) = (`f` bR) <$> finalL sL
1048− final (TeeRight bL sR) = f bL <$> finalR sR
1049−
1050953{-# ANN type TeeFstState Fuse #-}
1051954data TeeFstState sL sR b
1052955 = TeeFstBoth !sL !sR
@@ -1059,10 +962,8 @@ data TeeFstState sL sR b
1059962{-# INLINE teeWithFst #-}
1060963teeWithFst :: Monad m =>
1061964 (b -> c -> d) -> Fold m a b -> Fold m a c -> Fold m a d
1062−teeWithFst f
1063− (Fold stepL initialL extractL finalL)
1064− (Fold stepR initialR extractR finalR) =
1065− Fold step initial extract final
965+teeWithFst f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
966+ Fold step initial extract
1066967
1067968 where
1068969
@@ -1081,7 +982,7 @@ teeWithFst f
1081982 Done bl -> do
1082983 Done . f bl <$>
1083984 case resR of
1084− Partial sr -> finalR sr
985+ Partial sr -> extractR sr
1085986 Done br -> return br
1086987
1087988 initial = runBoth initialL initialR
@@ -1092,9 +993,6 @@ teeWithFst f
1092993 extract (TeeFstBoth sL sR) = f <$> extractL sL <*> extractR sR
1093994 extract (TeeFstLeft bR sL) = (`f` bR) <$> extractL sL
1094995
1095− final (TeeFstBoth sL sR) = f <$> finalL sL <*> finalR sR
1096− final (TeeFstLeft bR sL) = (`f` bR) <$> finalL sL
1097−
1098996-- | Like 'teeWith' but terminates as soon as any one of the two folds
1099997-- terminates.
1100998--
@@ -1103,10 +1001,8 @@ teeWithFst f
11031001{-# INLINE teeWithMin #-}
11041002teeWithMin :: Monad m =>
11051003 (b -> c -> d) -> Fold m a b -> Fold m a c -> Fold m a d
1106−teeWithMin f
1107− (Fold stepL initialL extractL finalL)
1108− (Fold stepR initialR extractR finalR) =
1109− Fold step initial extract final
1004+teeWithMin f (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1005+ Fold step initial extract
11101006
11111007 where
11121008
@@ -1118,12 +1014,12 @@ teeWithMin f
11181014 Partial sl -> do
11191015 case resR of
11201016 Partial sr -> return $ Partial $ Tuple' sl sr
1121− Done br -> Done . (`f` br) <$> finalL sl
1017+ Done br -> Done . (`f` br) <$> extractL sl
11221018
11231019 Done bl -> do
11241020 Done . f bl <$>
11251021 case resR of
1126− Partial sr -> finalR sr
1022+ Partial sr -> extractR sr
11271023 Done br -> return br
11281024
11291025 initial = runBoth initialL initialR
@@ -1132,8 +1028,6 @@ teeWithMin f
11321028
11331029 extract (Tuple' sL sR) = f <$> extractL sL <*> extractR sR
11341030
1135− final (Tuple' sL sR) = f <$> finalL sL <*> finalR sR
1136−
11371031-- | Shortest alternative. Apply both folds in parallel but choose the result
11381032-- from the one which consumed least input i.e. take the shortest succeeding
11391033-- fold.
@@ -1145,8 +1039,8 @@ teeWithMin f
11451039--
11461040{-# INLINE shortest #-}
11471041shortest :: Monad m => Fold m x a -> Fold m x b -> Fold m x (Either a b)
1148−shortest (Fold stepL initialL extractL finalL) (Fold stepR initialR _ finalR) =
1149− Fold step initial extract final
1042+shortest (Fold stepL initialL extractL) (Fold stepR initialR _) =
1043+ Fold step initial extract
11501044
11511045 where
11521046
@@ -1154,16 +1048,10 @@ shortest (Fold stepL initialL extractL finalL) (Fold stepR initialR _ finalR) =
11541048 runBoth actionL actionR = do
11551049 resL <- actionL
11561050 resR <- actionR
1157− case resL of
1158− Partial sL ->
1159− case resR of
1160− Partial sR -> return $ Partial $ Tuple' sL sR
1161− Done bR -> finalL sL >> return (Done (Right bR))
1162− Done bL -> do
1163− case resR of
1164− Partial sR -> void (finalR sR)
1165− Done _ -> return ()
1166− return (Done (Left bL))
1051+ return $
1052+ case resL of
1053+ Partial sL -> bimap (Tuple' sL) Right resR
1054+ Done bL -> Done $ Left bL
11671055
11681056 initial = runBoth initialL initialR
11691057
@@ -1171,8 +1059,6 @@ shortest (Fold stepL initialL extractL finalL) (Fold stepR initialR _ finalR) =
11711059
11721060 extract (Tuple' sL _) = Left <$> extractL sL
11731061
1174− final (Tuple' sL sR) = Left <$> finalL sL <* finalR sR
1175−
11761062{-# ANN type LongestState Fuse #-}
11771063data LongestState sL sR
11781064 = LongestBoth !sL !sR
@@ -1190,10 +1076,8 @@ data LongestState sL sR
11901076--
11911077{-# INLINE longest #-}
11921078longest :: Monad m => Fold m x a -> Fold m x b -> Fold m x (Either a b)
1193−longest
1194− (Fold stepL initialL _ finalL)
1195− (Fold stepR initialR _ finalR) =
1196− Fold step initial extract final
1079+longest (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1080+ Fold step initial extract
11971081
11981082 where
11991083
@@ -1216,18 +1100,14 @@ longest
12161100 step (LongestLeft sL) a = bimap LongestLeft Left <$> stepL sL a
12171101 step (LongestRight sR) a = bimap LongestRight Right <$> stepR sR a
12181102
1219− -- XXX Scan with this may not make sense as we cannot determine the longest
1220− -- until one of them have exhausted.
1221− extract _ = error $ "longest: scan is not allowed as longest cannot be "
1222− ++ "determined until one fold has exhausted."
1223−
1224− final (LongestLeft sL) = Left <$> finalL sL
1225− final (LongestRight sR) = Right <$> finalR sR
1226− final (LongestBoth sL sR) = Left <$> finalL sL <* finalR sR
1103+ left sL = Left <$> extractL sL
1104+ extract (LongestLeft sL) = left sL
1105+ extract (LongestRight sR) = Right <$> extractR sR
1106+ extract (LongestBoth sL _) = left sL
12271107
1228−data ConcatMapState m sa a b c
1229− = B !sa (sa -> m b)
1230− | forall s. C (s -> a -> m (Step s c)) !s (s -> m c) (s -> m c)
1108+data ConcatMapState m sa a c
1109+ = B !sa
1110+ | forall s. C (s -> a -> m (Step s c)) !s (s -> m c)
12311111
12321112-- | Map a 'Fold' returning function on the result of a 'Fold' and run the
12331113-- returned fold. This operation can be used to express data dependencies
@@ -1250,47 +1130,43 @@ data ConcatMapState m sa a b c
12501130--
12511131{-# INLINE concatMap #-}
12521132concatMap :: Monad m => (b -> Fold m a c) -> Fold m a b -> Fold m a c
1253−concatMap f (Fold stepa initiala _ finala) =
1254− Fold stepc initialc extractc finalc
1133+concatMap f (Fold stepa initiala extracta) = Fold stepc initialc extractc
12551134 where
12561135 initialc = do
12571136 r <- initiala
12581137 case r of
1259− Partial s -> return $ Partial (B s finala)
1138+ Partial s -> return $ Partial (B s)
12601139 Done b -> initInnerFold (f b)
12611140
1262− stepc (B s fin) a = do
1141+ stepc (B s) a = do
12631142 r <- stepa s a
12641143 case r of
1265− Partial s1 -> return $ Partial (B s1 fin)
1144+ Partial s1 -> return $ Partial (B s1)
12661145 Done b -> initInnerFold (f b)
12671146
1268− stepc (C stepInner s extractInner fin) a = do
1147+ stepc (C stepInner s extractInner) a = do
12691148 r <- stepInner s a
12701149 return $ case r of
1271− Partial sc -> Partial (C stepInner sc extractInner fin)
1150+ Partial sc -> Partial (C stepInner sc extractInner)
12721151 Done c -> Done c
12731152
1274− -- XXX Cannot use for scanning
1275− extractc _ = error "concatMap: cannot be used for scanning"
1153+ extractc (B s) = do
1154+ r <- extracta s
1155+ initExtract (f r)
1156+ extractc (C _ sInner extractInner) = extractInner sInner
12761157
1277− initInnerFold (Fold step i e fin) = do
1158+ initInnerFold (Fold step i e) = do
12781159 r <- i
12791160 return $ case r of
1280− Partial s -> Partial (C step s e fin)
1161+ Partial s -> Partial (C step s e)
12811162 Done c -> Done c
12821163
1283− initFinalize (Fold _ i _ fin) = do
1164+ initExtract (Fold _ i e) = do
12841165 r <- i
12851166 case r of
1286− Partial s -> fin s
1167+ Partial s -> e s
12871168 Done c -> return c
12881169
1289− finalc (B s fin) = do
1290− r <- fin s
1291− initFinalize (f r)
1292− finalc (C _ sInner _ fin) = fin sInner
1293−
12941170------------------------------------------------------------------------------
12951171-- Mapping on input
12961172------------------------------------------------------------------------------
@@ -1309,7 +1185,7 @@ concatMap f (Fold stepa initiala _ finala) =
13091185--
13101186{-# INLINE lmap #-}
13111187lmap :: (a -> b) -> Fold m b r -> Fold m a r
1312−lmap f (Fold step begin done final) = Fold step' begin done final
1188+lmap f (Fold step begin done) = Fold step' begin done
13131189 where
13141190 step' x a = step x (f a)
13151191
@@ -1317,7 +1193,7 @@ lmap f (Fold step begin done final) = Fold step' begin done final
13171193--
13181194{-# INLINE lmapM #-}
13191195lmapM :: Monad m => (a -> m b) -> Fold m b r -> Fold m a r
1320−lmapM f (Fold step begin done final) = Fold step' begin done final
1196+lmapM f (Fold step begin done) = Fold step' begin done
13211197 where
13221198 step' x a = f a >>= step x
13231199
@@ -1329,10 +1205,8 @@ lmapM f (Fold step begin done final) = Fold step' begin done final
13291205-- /Pre-release/
13301206{-# INLINE postscan #-}
13311207postscan :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1332−postscan
1333− (Fold stepL initialL extractL finalL)
1334− (Fold stepR initialR extractR finalR) =
1335− Fold step initial extract final
1208+postscan (Fold stepL initialL extractL) (Fold stepR initialR extractR) =
1209+ Fold step initial extract
13361210
13371211 where
13381212
@@ -1343,32 +1217,30 @@ postscan
13431217 Done bL -> do
13441218 rR <- stepR sR bL
13451219 case rR of
1346− Partial sR1 -> Done <$> finalR sR1
1220+ Partial sR1 -> Done <$> extractR sR1
13471221 Done bR -> return $ Done bR
13481222 Partial sL -> do
13491223 !b <- extractL sL
13501224 rR <- stepR sR b
1351− case rR of
1352− Partial sR1 -> return $ Partial (sL, sR1)
1353− Done bR -> finalL sL >> return (Done bR)
1225+ return
1226+ $ case rR of
1227+ Partial sR1 -> Partial (sL, sR1)
1228+ Done bR -> Done bR
13541229
13551230 initial = do
1356− rR <- initialR
1357− case rR of
1358− Partial sR -> do
1359− rL <- initialL
1231+ r <- initialR
1232+ rL <- initialL
1233+ case r of
1234+ Partial sR ->
13601235 case rL of
1361− Done _ -> Done <$> finalR sR
1236+ Done _ -> Done <$> extractR sR
13621237 Partial sL -> return $ Partial (sL, sR)
13631238 Done b -> return $ Done b
13641239
1365− -- XXX should use Tuple'
13661240 step (sL, sR) x = runStep (stepL sL x) sR
13671241
13681242 extract = extractR . snd
13691243
1370− final (sL, sR) = finalL sL *> finalR sR
1371−
13721244------------------------------------------------------------------------------
13731245-- Filtering
13741246------------------------------------------------------------------------------
@@ -1381,7 +1253,7 @@ postscan
13811253--
13821254{-# INLINE_NORMAL catMaybes #-}
13831255catMaybes :: Monad m => Fold m a b -> Fold m (Maybe a) b
1384−catMaybes (Fold step initial extract final) = Fold step1 initial extract final
1256+catMaybes (Fold step initial extract) = Fold step1 initial extract
13851257
13861258 where
13871259
@@ -1421,7 +1293,7 @@ filtering f = foldl' step Nothing
14211293{-# INLINE filter #-}
14221294filter :: Monad m => (a -> Bool) -> Fold m a r -> Fold m a r
14231295-- filter p = scanMaybe (filtering p)
1424−filter f (Fold step begin extract final) = Fold step' begin extract final
1296+filter f (Fold step begin done) = Fold step' begin done
14251297 where
14261298 step' x a = if f a then step x a else return $ Partial x
14271299
@@ -1432,7 +1304,7 @@ filter f (Fold step begin extract final) = Fold step' begin extract final
14321304--
14331305{-# INLINE filterM #-}
14341306filterM :: Monad m => (a -> m Bool) -> Fold m a r -> Fold m a r
1435−filterM f (Fold step begin extract final) = Fold step' begin extract final
1307+filterM f (Fold step begin done) = Fold step' begin done
14361308 where
14371309 step' x a = do
14381310 use <- f a
@@ -1521,7 +1393,7 @@ dropping n = foldt' step initial extract
15211393{-# INLINE take #-}
15221394take :: Monad m => Int -> Fold m a b -> Fold m a b
15231395-- take n = scanMaybe (taking n)
1524−take n (Fold fstep finitial fextract ffinal) = Fold step initial extract final
1396+take n (Fold fstep finitial fextract) = Fold step initial extract
15251397
15261398 where
15271399
@@ -1533,7 +1405,7 @@ take n (Fold fstep finitial fextract ffinal) = Fold step initial extract final
15331405 s1 = Tuple'Fused i1 s
15341406 if i1 < n
15351407 then return $ Partial s1
1536− else Done <$> ffinal s
1408+ else Done <$> fextract s
15371409 Done b -> return $ Done b
15381410
15391411 initial = finitial >>= next (-1)
@@ -1542,72 +1414,6 @@ take n (Fold fstep finitial fextract ffinal) = Fold step initial extract final
15421414
15431415 extract (Tuple'Fused _ r) = fextract r
15441416
1545− final (Tuple'Fused _ r) = ffinal r
1546−
1547−-- Note: Keep this consistent with S.splitOn. In fact we should eliminate
1548−-- S.splitOn in favor of the fold.
1549−--
1550−-- XXX Use Fold.many instead once it is fixed.
1551−-- > Stream.splitOnSuffix p f = Stream.foldMany (Fold.takeEndBy_ p f)
1552−
1553−-- | Like 'takeEndBy' but drops the element on which the predicate succeeds.
1554−--
1555−-- Example:
1556−--
1557−-- >>> input = Stream.fromList "hello\nthere\n"
1558−-- >>> line = Fold.takeEndBy_ (== '\n') Fold.toList
1559−-- >>> Stream.fold line input
1560−-- "hello"
1561−--
1562−-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1563−-- ["hello","there"]
1564−--
1565−{-# INLINE takeEndBy_ #-}
1566−takeEndBy_ :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1567−-- takeEndBy_ predicate = scanMaybe (takingEndBy_ predicate)
1568−takeEndBy_ predicate (Fold fstep finitial fextract ffinal) =
1569− Fold step finitial fextract ffinal
1570−
1571− where
1572−
1573− step s a =
1574− if not (predicate a)
1575− then fstep s a
1576− else Done <$> ffinal s
1577−
1578−-- Note:
1579−-- > Stream.splitWithSuffix p f = Stream.foldMany (Fold.takeEndBy p f)
1580−
1581−-- | Take the input, stop when the predicate succeeds taking the succeeding
1582−-- element as well.
1583−--
1584−-- Example:
1585−--
1586−-- >>> input = Stream.fromList "hello\nthere\n"
1587−-- >>> line = Fold.takeEndBy (== '\n') Fold.toList
1588−-- >>> Stream.fold line input
1589−-- "hello\n"
1590−--
1591−-- >>> Stream.fold Fold.toList $ Stream.foldMany line input
1592−-- ["hello\n","there\n"]
1593−--
1594−{-# INLINE takeEndBy #-}
1595−takeEndBy :: Monad m => (a -> Bool) -> Fold m a b -> Fold m a b
1596−-- takeEndBy predicate = scanMaybe (takingEndBy predicate)
1597−takeEndBy predicate (Fold fstep finitial fextract ffinal) =
1598− Fold step finitial fextract ffinal
1599−
1600− where
1601−
1602− step s a = do
1603− res <- fstep s a
1604− if not (predicate a)
1605− then return res
1606− else do
1607− case res of
1608− Partial s1 -> Done <$> ffinal s1
1609− Done b -> return $ Done b
1610−
16111417------------------------------------------------------------------------------
16121418-- Nesting
16131419------------------------------------------------------------------------------
@@ -1625,8 +1431,8 @@ takeEndBy predicate (Fold fstep finitial fextract ffinal) =
16251431-- /Pre-release/
16261432{-# INLINE duplicate #-}
16271433duplicate :: Monad m => Fold m a b -> Fold m a (Fold m a b)
1628−duplicate (Fold step1 initial1 extract1 final1) =
1629− Fold step initial extract final
1434+duplicate (Fold step1 initial1 extract1) =
1435+ Fold step initial (\s -> pure $ Fold step1 (pure $ Partial s) extract1)
16301436
16311437 where
16321438
@@ -1634,11 +1440,6 @@ duplicate (Fold step1 initial1 extract1 final1) =
16341440
16351441 step s a = second fromPure <$> step1 s a
16361442
1637− -- Scanning may be problematic due to multiple finalizations.
1638− extract = error "duplicate: scanning may be problematic"
1639−
1640− final s = pure $ Fold step1 (pure $ Partial s) extract1 final1
1641−
16421443-- If there were a finalize/flushing action in the stream type that would be
16431444-- equivalent to running initialize in Fold. But we do not have a flushing
16441445-- action in streams.
@@ -1650,9 +1451,9 @@ duplicate (Fold step1 initial1 extract1 final1) =
16501451-- /Pre-release/
16511452{-# INLINE reduce #-}
16521453reduce :: Monad m => Fold m a b -> m (Fold m a b)
1653−reduce (Fold step initial extract final) = do
1454+reduce (Fold step initial extract) = do
16541455 i <- initial
1655− return $ Fold step (return i) extract final
1456+ return $ Fold step (return i) extract
16561457
16571458-- This is the dual of Stream @cons@.
16581459
@@ -1662,8 +1463,7 @@ reduce (Fold step initial extract final) = do
16621463-- /Pre-release/
16631464{-# INLINE snoclM #-}
16641465snoclM :: Monad m => Fold m a b -> m a -> Fold m a b
1665−snoclM (Fold fstep finitial fextract ffinal) action =
1666− Fold fstep initial fextract ffinal
1466+snoclM (Fold fstep finitial fextract) action = Fold fstep initial fextract
16671467
16681468 where
16691469
@@ -1690,8 +1490,7 @@ snoclM (Fold fstep finitial fextract ffinal) action =
16901490{-# INLINE snocl #-}
16911491snocl :: Monad m => Fold m a b -> a -> Fold m a b
16921492-- snocl f = snoclM f . return
1693−snocl (Fold fstep finitial fextract ffinal) a =
1694− Fold fstep initial fextract ffinal
1493+snocl (Fold fstep finitial fextract) a = Fold fstep initial fextract
16951494
16961495 where
16971496
@@ -1711,12 +1510,12 @@ snocl (Fold fstep finitial fextract ffinal) a =
17111510-- /Pre-release/
17121511{-# INLINE snocM #-}
17131512snocM :: Monad m => Fold m a b -> m a -> m (Fold m a b)
1714−snocM (Fold step initial extract final) action = do
1513+snocM (Fold step initial extract) action = do
17151514 res <- initial
17161515 r <- case res of
17171516 Partial fs -> action >>= step fs
17181517 Done _ -> return res
1719− return $ Fold step (return r) extract final
1518+ return $ Fold step (return r) extract
17201519
17211520-- Definitions:
17221521--
@@ -1735,12 +1534,12 @@ snocM (Fold step initial extract final) action = do
17351534-- /Pre-release/
17361535{-# INLINE snoc #-}
17371536snoc :: Monad m => Fold m a b -> a -> m (Fold m a b)
1738−snoc (Fold step initial extract final) a = do
1537+snoc (Fold step initial extract) a = do
17391538 res <- initial
17401539 r <- case res of
17411540 Partial fs -> step fs a
17421541 Done _ -> return res
1743− return $ Fold step (return r) extract final
1542+ return $ Fold step (return r) extract
17441543
17451544-- | Append a singleton value to the fold.
17461545--
@@ -1768,7 +1567,7 @@ addOne = flip snoc
17681567-- /Pre-release/
17691568{-# INLINE extractM #-}
17701569extractM :: Monad m => Fold m a b -> m b
1771−extractM (Fold _ initial extract _) = do
1570+extractM (Fold _ initial extract) = do
17721571 res <- initial
17731572 case res of
17741573 Partial fs -> extract fs
@@ -1777,15 +1576,14 @@ extractM (Fold _ initial extract _) = do
17771576-- | Close a fold so that it does not accept any more input.
17781577{-# INLINE close #-}
17791578close :: Monad m => Fold m a b -> Fold m a b
1780−close (Fold _ initial1 _ final1) =
1781− Fold undefined initial undefined undefined
1579+close (Fold _ initial1 extract1) = Fold undefined initial undefined
17821580
17831581 where
17841582
17851583 initial = do
17861584 res <- initial1
17871585 case res of
1788− Partial s -> Done <$> final1 s
1586+ Partial s -> Done <$> extract1 s
17891587 Done b -> return $ Done b
17901588
17911589-- Corresponds to the null check for streams.
@@ -1795,7 +1593,7 @@ close (Fold _ initial1 _ final1) =
17951593-- /Pre-release/
17961594{-# INLINE isClosed #-}
17971595isClosed :: Monad m => Fold m a b -> m Bool
1798−isClosed (Fold _ initial _ _) = do
1596+isClosed (Fold _ initial _) = do
17991597 res <- initial
18001598 return $ case res of
18011599 Partial _ -> False
@@ -1829,10 +1627,8 @@ data ManyState s1 s2
18291627--
18301628{-# INLINE many #-}
18311629many :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1832−many
1833− (Fold sstep sinitial sextract sfinal)
1834− (Fold cstep cinitial cextract cfinal) =
1835− Fold step initial extract final
1630+many (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
1631+ Fold step initial extract
18361632
18371633 where
18381634
@@ -1881,13 +1677,6 @@ many
18811677 Partial s -> cextract s
18821678 Done b -> return b
18831679
1884− final (ManyFirst ss cs) = sfinal ss *> cfinal cs
1885− final (ManyLoop ss cs) = do
1886− cres <- sfinal ss >>= cstep cs
1887− case cres of
1888− Partial s -> cfinal s
1889− Done b -> return b
1890−
18911680-- | Like many, but the "first" fold emits an output at the end even if no
18921681-- input is received.
18931682--
@@ -1897,10 +1686,8 @@ many
18971686--
18981687{-# INLINE manyPost #-}
18991688manyPost :: Monad m => Fold m a b -> Fold m b c -> Fold m a c
1900−manyPost
1901− (Fold sstep sinitial sextract sfinal)
1902− (Fold cstep cinitial cextract cfinal) =
1903− Fold step initial extract final
1689+manyPost (Fold sstep sinitial sextract) (Fold cstep cinitial cextract) =
1690+ Fold step initial extract
19041691
19051692 where
19061693
@@ -1936,12 +1723,6 @@ manyPost
19361723 Partial s -> cextract s
19371724 Done b -> return b
19381725
1939− final (Tuple' ss cs) = do
1940− cres <- sfinal ss >>= cstep cs
1941− case cres of
1942− Partial s -> cfinal s
1943− Done b -> return b
1944−
19451726-- | @groupsOf n split collect@ repeatedly applies the @split@ fold to chunks
19461727-- of @n@ items in the input stream and supplies the result to the @collect@
19471728-- fold.
@@ -1970,10 +1751,7 @@ groupsOf n split = many (take n split)
19701751--
19711752{-# INLINE refoldMany #-}
19721753refoldMany :: Monad m => Fold m a b -> Refold m x b c -> Refold m x a c
1973−refoldMany
1974− (Fold sstep sinitial sextract _sfinal)
1975− -- XXX We will need a "final" in refold as well
1976− (Refold cstep cinject cextract) =
1754+refoldMany (Fold sstep sinitial sextract) (Refold cstep cinject cextract) =
19771755 Refold step inject extract
19781756
19791757 where
@@ -2024,9 +1802,7 @@ data ConsumeManyState x cs ss = ConsumeMany x cs (Either ss ss)
20241802-- /Internal/
20251803{-# INLINE refoldMany1 #-}
20261804refoldMany1 :: Monad m => Refold m x a b -> Fold m b c -> Refold m x a c
2027−refoldMany1
2028− (Refold sstep sinject sextract)
2029− (Fold cstep cinitial cextract _cfinal) =
1805+refoldMany1 (Refold sstep sinject sextract) (Fold cstep cinitial cextract) =
20301806 Refold step inject extract
20311807
20321808 where
@@ -2077,7 +1853,7 @@ refoldMany1
20771853{-# INLINE refold #-}
20781854refold :: Monad m => Refold m b a c -> Fold m a b -> Fold m a c
20791855refold (Refold step inject extract) f =
2080− Fold step (extractM f >>= inject) extract extract
1856+ Fold step (extractM f >>= inject) extract
20811857
20821858------------------------------------------------------------------------------
20831859-- morphInner
@@ -2087,8 +1863,8 @@ refold (Refold step inject extract) f =
20871863--
20881864-- /Pre-release/
20891865morphInner :: (forall x. m x -> n x) -> Fold m a b -> Fold n a b
2090−morphInner f (Fold step initial extract final) =
2091− Fold (\x a -> f $ step x a) (f initial) (f . extract) (f . final)
1866+morphInner f (Fold step initial extract) =
1867+ Fold (\x a -> f $ step x a) (f initial) (f . extract)
20921868
20931869-- | Adapt a pure fold to any monad.
20941870--
core/src/Streamly/Internal/Data/Fold/Window.hs+4−10
@@ -65,8 +65,6 @@ import Streamly.Internal.Data.Fold.Type (Fold(..), Step(..))
6565import Streamly.Internal.Data.Tuple.Strict
6666 (Tuple'(..), Tuple3Fused' (Tuple3Fused'))
6767
68−import Foreign.ForeignPtr.Unsafe (unsafeForeignPtrToPtr)
69−
7068import qualified Streamly.Internal.Data.Fold.Type as Fold
7169import qualified Streamly.Internal.Data.Ring as Ring
7270
@@ -150,7 +148,7 @@ windowRollingMap f = Fold.foldl' f1 initial
150148--
151149{-# INLINE windowSumInt #-}
152150windowSumInt :: forall m a. (Monad m, Integral a) => Fold m (a, Maybe a) a
153−windowSumInt = Fold step initial extract extract
151+windowSumInt = Fold step initial extract
154152
155153 where
156154
@@ -184,7 +182,7 @@ windowSumInt = Fold step initial extract extract
184182--
185183{-# INLINE windowSum #-}
186184windowSum :: forall m a. (Monad m, Num a) => Fold m (a, Maybe a) a
187−windowSum = Fold step initial extract extract
185+windowSum = Fold step initial extract
188186
189187 where
190188
@@ -269,7 +267,7 @@ windowPowerSumFrac p = windowLmap (** p) windowSum
269267--
270268{-# INLINE windowRange #-}
271269windowRange :: (MonadIO m, Storable a, Ord a) => Int -> Fold m a (Maybe (a, a))
272−windowRange n = Fold step initial extract extract
270+windowRange n = Fold step initial extract
273271
274272 where
275273
@@ -302,11 +300,7 @@ windowRange n = Fold step initial extract extract
302300 if i == 0
303301 then return Nothing
304302 else do
305− -- Here we use "ringStart" over "ringHead" as "ringHead" will be
306− -- uninitialized if the ring is not full.
307− -- Using "unsafeForeignPtrToPtr" here is safe as we touch the ring
308− -- again in "foldFunc".
309− x <- liftIO $ peek (unsafeForeignPtrToPtr (Ring.ringStart rb))
303+ x <- liftIO $ peek rh
310304 let accum (mn, mx) a = return (min mn a, max mx a)
311305 fmap Just $ foldFunc i rh accum (x, x) rb
312306
core/src/Streamly/Internal/Data/IORef/Unboxed.hs+19−22
@@ -23,31 +23,34 @@ module Streamly.Internal.Data.IORef.Unboxed
2323 (
2424 IORef
2525
26− -- Construction
26+ -- * Construction
2727 , newIORef
2828
29− -- Write
29+ -- * Write
3030 , writeIORef
3131 , modifyIORef'
3232
33− -- Read
33+ -- * Read
3434 , readIORef
35− , pollIntIORef
35+ , toStreamD
3636 )
3737where
3838
3939#include "inline.hs"
4040
41−import Control.Monad.IO.Class (MonadIO(..))
4241import Data.Proxy (Proxy(..))
43−import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
44−import Streamly.Internal.Data.Unbox (Unbox(..), sizeOf)
42+import Control.Monad.IO.Class (MonadIO(..))
43+import Streamly.Internal.Data.Unbox
44+ ( MutableByteArray(..)
45+ , Unbox(..)
46+ , newBytes
47+ , sizeOf
48+ )
4549
46−import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
4750import qualified Streamly.Internal.Data.Stream.Type as D
4851
4952-- | An 'IORef' holds a single 'Unbox'-able value.
50−newtype IORef a = IORef MutByteArray
53+newtype IORef a = IORef MutableByteArray
5154
5255-- | Create a new 'IORef'.
5356--
@@ -55,8 +58,8 @@ newtype IORef a = IORef MutByteArray
5558{-# INLINE newIORef #-}
5659newIORef :: forall a. Unbox a => a -> IO (IORef a)
5760newIORef x = do
58− var <- MBA.new (sizeOf (Proxy :: Proxy a))
59− pokeAt 0 var x
61+ var <- newBytes (sizeOf (Proxy :: Proxy a))
62+ pokeByteIndex 0 var x
6063 return $ IORef var
6164
6265-- | Write a value to an 'IORef'.
@@ -64,14 +67,14 @@ newIORef x = do
6467-- /Pre-release/
6568{-# INLINE writeIORef #-}
6669writeIORef :: Unbox a => IORef a -> a -> IO ()
67−writeIORef (IORef var) = pokeAt 0 var
70+writeIORef (IORef var) = pokeByteIndex 0 var
6871
6972-- | Read a value from an 'IORef'.
7073--
7174-- /Pre-release/
7275{-# INLINE readIORef #-}
7376readIORef :: Unbox a => IORef a -> IO a
74−readIORef (IORef var) = peekAt 0 var
77+readIORef (IORef var) = peekByteIndex 0 var
7578
7679-- | Modify the value of an 'IORef' using a function with strict application.
7780--
@@ -84,16 +87,10 @@ modifyIORef' var g = do
8487
8588-- | Generate a stream by continuously reading the IORef.
8689--
87−-- This operation reads the IORef without any synchronization. It can be
88−-- assumed to be atomic because the IORef (MutableByteArray) is always aligned
89−-- to Int boundaries, we are assuming that compiler uses single instructions to
90−-- access the memory. It may read stale values though until caches are
91−-- synchronised in a multiprocessor architecture.
92−--
9390-- /Pre-release/
94−{-# INLINE_NORMAL pollIntIORef #-}
95−pollIntIORef :: (MonadIO m, Unbox a) => IORef a -> D.Stream m a
96−pollIntIORef var = D.Stream step ()
91+{-# INLINE_NORMAL toStreamD #-}
92+toStreamD :: (MonadIO m, Unbox a) => IORef a -> D.Stream m a
93+toStreamD var = D.Stream step ()
9794
9895 where
9996
core/src/Streamly/Internal/Data/IsMap.hs−8
@@ -1,7 +1,3 @@
1−{-# LANGUAGE TypeFamilies #-}
2−-- Must come after TypeFamilies, otherwise it is re-enabled.
3−-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
4−{-# LANGUAGE NoMonoLocalBinds #-}
51-- |
62-- Module : Streamly.Internal.Data.IsMap
73-- Copyright : (c) 2022 Composewell Technologies
@@ -32,8 +28,6 @@ class IsMap f where
3228 mapDelete :: Key f -> f a -> f a
3329 mapUnion :: f a -> f a -> f a
3430 mapNull :: f a -> Bool
35− mapTraverseWithKey ::
36− Applicative t => (Key f -> a -> t b) -> f a -> t (f b)
3731
3832instance Ord k => IsMap (Map k) where
3933 type Key (Map k) = k
@@ -45,7 +39,6 @@ instance Ord k => IsMap (Map k) where
4539 mapDelete = Map.delete
4640 mapUnion = Map.union
4741 mapNull = Map.null
48− mapTraverseWithKey = Map.traverseWithKey
4942
5043instance IsMap IntMap.IntMap where
5144 type Key IntMap.IntMap = Int
@@ -57,4 +50,3 @@ instance IsMap IntMap.IntMap where
5750 mapDelete = IntMap.delete
5851 mapUnion = IntMap.union
5952 mapNull = IntMap.null
60− mapTraverseWithKey = IntMap.traverseWithKey
core/src/Streamly/Internal/Data/MutArray.hs+33−154
@@ -8,200 +8,79 @@
88--
99module Streamly.Internal.Data.MutArray
1010 (
11− -- * MutArray.Type module
1211 module Streamly.Internal.Data.MutArray.Type
13− -- * MutArray module
14− , sliceIndexerFromLen
15− , slicerFromLen
16− , compactLE
17− , pinnedCompactLE
18− , compactOnByte
19− , compactOnByteSuffix
20− -- * Unboxed IORef
21− , module Streamly.Internal.Data.IORef.Unboxed
22−
23− -- * Deprecated
12+ , splitOn
2413 , genSlicesFromLen
2514 , getSlicesFromLen
15+ , fromStream
2616 )
2717where
2818
2919#include "inline.hs"
3020
3121import Control.Monad.IO.Class (MonadIO(..))
32−import Data.Word (Word8)
33−import Streamly.Internal.Data.MutByteArray.Type (PinnedState(..))
34−import Streamly.Internal.Data.Stream.Type (Stream)
3522import Streamly.Internal.Data.Unbox (Unbox)
23+import Streamly.Internal.Data.Stream (Stream)
3624import Streamly.Internal.Data.Unfold.Type (Unfold(..))
3725
38−import qualified Streamly.Internal.Data.Stream.Type as Stream
39−import qualified Streamly.Internal.Data.Stream.Nesting as Stream
40−-- import qualified Streamly.Internal.Data.Stream.Transform as Stream
26+import qualified Streamly.Internal.Data.Stream as D
4127import qualified Streamly.Internal.Data.Unfold as Unfold
4228
4329import Prelude hiding (foldr, length, read, splitAt)
4430import Streamly.Internal.Data.MutArray.Type
45−import Streamly.Internal.Data.IORef.Unboxed
31+
32+-- | Split the array into a stream of slices using a predicate. The element
33+-- matching the predicate is dropped.
34+--
35+-- /Pre-release/
36+{-# INLINE splitOn #-}
37+splitOn :: (MonadIO m, Unbox a) =>
38+ (a -> Bool) -> MutArray a -> Stream m (MutArray a)
39+splitOn predicate arr =
40+ fmap (\(i, len) -> getSliceUnsafe i len arr)
41+ $ D.sliceOnSuffix predicate (read arr)
4642
4743-- | Generate a stream of array slice descriptors ((index, len)) of specified
4844-- length from an array, starting from the supplied array index. The last slice
4945-- may be shorter than the requested length depending on the array length.
5046--
5147-- /Pre-release/
52−{-# INLINE sliceIndexerFromLen #-}
53−sliceIndexerFromLen :: forall m a. (Monad m, Unbox a)
48+{-# INLINE genSlicesFromLen #-}
49+genSlicesFromLen :: forall m a. (Monad m, Unbox a)
5450 => Int -- ^ from index
5551 -> Int -- ^ length of the slice
5652 -> Unfold m (MutArray a) (Int, Int)
57−sliceIndexerFromLen from len =
53+genSlicesFromLen from len =
5854 let fromThenTo n = (from, from + len, n - 1)
5955 mkSlice n i = return (i, min len (n - i))
6056 in Unfold.lmap length
6157 $ Unfold.mapM2 mkSlice
6258 $ Unfold.lmap fromThenTo Unfold.enumerateFromThenTo
6359
64−{-# DEPRECATED genSlicesFromLen "Please use sliceIndexerFromLen instead." #-}
65−genSlicesFromLen :: forall m a. (Monad m, Unbox a)
66− => Int -- ^ from index
67− -> Int -- ^ length of the slice
68− -> Unfold m (MutArray a) (Int, Int)
69−genSlicesFromLen = sliceIndexerFromLen
70−
7160-- | Generate a stream of slices of specified length from an array, starting
7261-- from the supplied array index. The last slice may be shorter than the
7362-- requested length depending on the array length.
7463--
7564-- /Pre-release/
76−{-# INLINE slicerFromLen #-}
77−slicerFromLen :: forall m a. (Monad m, Unbox a)
78− => Int -- ^ from index
79− -> Int -- ^ length of the slice
80− -> Unfold m (MutArray a) (MutArray a)
81−slicerFromLen from len =
82− let mkSlice arr (i, n) = return $ getSliceUnsafe i n arr
83− in Unfold.mapM2 mkSlice (sliceIndexerFromLen from len)
84−
85−{-# DEPRECATED getSlicesFromLen "Please use slicerFromLen instead." #-}
65+{-# INLINE getSlicesFromLen #-}
8666getSlicesFromLen :: forall m a. (Monad m, Unbox a)
8767 => Int -- ^ from index
8868 -> Int -- ^ length of the slice
8969 -> Unfold m (MutArray a) (MutArray a)
90−getSlicesFromLen = slicerFromLen
91−
92−-------------------------------------------------------------------------------
93−-- Compacting Streams of Arrays
94−-------------------------------------------------------------------------------
70+getSlicesFromLen from len =
71+ let mkSlice arr (i, n) = return $ getSliceUnsafe i n arr
72+ in Unfold.mapM2 mkSlice (genSlicesFromLen from len)
9573
96−-- | @compactLE maxElems@ coalesces adjacent arrays in the input stream
97−-- only if the combined size would be less than or equal to @maxElems@
98−-- elements. Note that it won't split an array if the original array is already
99−-- larger than maxElems.
100−--
101−-- @maxElems@ must be greater than 0.
74+-- | Create an 'Array' from a stream. This is useful when we want to create a
75+-- single array from a stream of unknown size. 'writeN' is at least twice
76+-- as efficient when the size is already known.
10277--
103−-- Generates unpinned arrays irrespective of the pinning status of input
104−-- arrays.
105−{-# INLINE compactLE #-}
106−compactLE :: (MonadIO m, Unbox a) =>
107− Int -> Stream m (MutArray a) -> Stream m (MutArray a)
108−-- XXX compactLE can be moved to MutArray/Type if we are not using the parser
109−-- to implement it.
110−compactLE = compactLeAs Unpinned
111−-- The parser version turns out to be a little bit slower.
112−-- compactLE n = Stream.catRights . Stream.parseManyD (pCompactLE n)
113−
114−-- | Pinned version of 'compactLE'.
115−{-# INLINE pinnedCompactLE #-}
116−pinnedCompactLE :: forall m a. (MonadIO m, Unbox a)
117− => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
118−pinnedCompactLE = compactLeAs Pinned
119−-- pinnedCompactLE n = Stream.catRights . Stream.parseManyD (pPinnedCompactLE n)
120−
121−data SplitState s arr
122− = Initial s
123− | Buffering s arr
124− | Splitting s arr
125− | Yielding arr (SplitState s arr)
126− | Finishing
127−
128−-- | Split a stream of arrays on a given separator byte, dropping the separator
129−-- and coalescing all the arrays between two separators into a single array.
78+-- Note that if the input stream is too large memory allocation for the array
79+-- may fail. When the stream size is not known, `chunksOf` followed by
80+-- processing of indvidual arrays in the resulting stream should be preferred.
13081--
131−{-# INLINE_NORMAL _compactOnByteCustom #-}
132−_compactOnByteCustom
133− :: MonadIO m
134− => Word8
135− -> Stream m (MutArray Word8)
136− -> Stream m (MutArray Word8)
137−_compactOnByteCustom byte (Stream.Stream step state) =
138− Stream.Stream step' (Initial state)
139−
140− where
141−
142− {-# INLINE_LATE step' #-}
143− step' gst (Initial st) = do
144− r <- step gst st
145− case r of
146− Stream.Yield arr s -> do
147− (arr1, marr2) <- breakOn byte arr
148− return $ case marr2 of
149− Nothing -> Stream.Skip (Buffering s arr1)
150− Just arr2 -> Stream.Skip (Yielding arr1 (Splitting s arr2))
151− Stream.Skip s -> return $ Stream.Skip (Initial s)
152− Stream.Stop -> return Stream.Stop
153−
154− step' gst (Buffering st buf) = do
155− r <- step gst st
156− case r of
157− Stream.Yield arr s -> do
158− (arr1, marr2) <- breakOn byte arr
159− -- XXX Use spliceExp instead and then rightSize?
160− buf1 <- splice buf arr1
161− return $ case marr2 of
162− Nothing -> Stream.Skip (Buffering s buf1)
163− Just x -> Stream.Skip (Yielding buf1 (Splitting s x))
164− Stream.Skip s -> return $ Stream.Skip (Buffering s buf)
165− Stream.Stop -> return $
166− if byteLength buf == 0
167− then Stream.Stop
168− else Stream.Skip (Yielding buf Finishing)
169−
170− step' _ (Splitting st buf) = do
171− (arr1, marr2) <- breakOn byte buf
172− return $ case marr2 of
173− Nothing -> Stream.Skip $ Buffering st arr1
174− Just arr2 -> Stream.Skip $ Yielding arr1 (Splitting st arr2)
175−
176− step' _ (Yielding arr next) = return $ Stream.Yield arr next
177− step' _ Finishing = return Stream.Stop
178−
179−-- XXX implement predicate based version of this
180−-- XXX Naming of predicate based vs custom version
181−
182−-- | Split a stream of arrays on a given separator byte, dropping the separator
183−-- and coalescing all the arrays between two separators into a single array.
184−--
185−{-# INLINE compactOnByte #-}
186−compactOnByte
187− :: (MonadIO m)
188− => Word8
189− -> Stream m (MutArray Word8)
190− -> Stream m (MutArray Word8)
191−-- XXX compare perf of custom vs idiomatic version
192−-- compactOnByte = _compactOnByteCustom
193−-- XXX use spliceExp and rightSize?
194−compactOnByte byte = Stream.splitInnerBy (breakOn byte) splice
195−
196−-- | Like 'compactOnByte' considers the separator in suffix position instead of
197−-- infix position.
198−{-# INLINE compactOnByteSuffix #-}
199−compactOnByteSuffix
200− :: (MonadIO m)
201− => Word8
202− -> Stream m (MutArray Word8)
203− -> Stream m (MutArray Word8)
204−compactOnByteSuffix byte =
205− -- XXX use spliceExp and rightSize?
206− Stream.splitInnerBySuffix
207− (\arr -> byteLength arr == 0) (breakOn byte) splice
82+-- /Pre-release/
83+{-# INLINE fromStream #-}
84+fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
85+fromStream = fromStreamD
86+-- fromStream (Stream m) = P.fold write m
core/src/Streamly/Internal/Data/MutArray/Generic.hs+30−65
@@ -19,14 +19,14 @@ module Streamly.Internal.Data.MutArray.Generic
1919 , nil
2020
2121 -- *** Uninitialized Arrays
22− , emptyOf
22+ , new
2323 -- , newArrayWith
2424
2525 -- *** From streams
26− , unsafeCreateOf
27− , createOf
28− , createWith -- createOfMin/createMin/createGE?
29− , create
26+ , writeNUnsafe
27+ , writeN
28+ , writeWith
29+ , write
3030 , fromStreamN
3131 , fromStream
3232 , fromPureStream
@@ -160,13 +160,6 @@ module Streamly.Internal.Data.MutArray.Generic
160160 -- , appendSliceFrom
161161
162162 , clone
163−
164− -- * Deprecated
165− , new
166− , writeNUnsafe
167− , writeN
168− , writeWith
169− , write
170163 )
171164where
172165
@@ -243,13 +236,13 @@ bottomElement =
243236-- XXX Would be nice if GHC can provide something like newUninitializedArray# so
244237-- that we do not have to write undefined or error in the whole array.
245238
246−-- | @emptyOf count@ allocates a zero length array that can be extended to hold
239+-- | @new count@ allocates a zero length array that can be extended to hold
247240-- up to 'count' items without reallocating.
248241--
249242-- /Pre-release/
250−{-# INLINE emptyOf #-}
251−emptyOf :: MonadIO m => Int -> m (MutArray a)
252−emptyOf n@(I# n#) =
243+{-# INLINE new #-}
244+new :: MonadIO m => Int -> m (MutArray a)
245+new n@(I# n#) =
253246 liftIO
254247 $ IO
255248 $ \s# ->
@@ -258,12 +251,6 @@ emptyOf n@(I# n#) =
258251 let ma = MutArray arr# 0 0 n
259252 in (# s1#, ma #)
260253
261−-- XXX Deprecate in major
262−-- {-# DEPRECATED new "Please use emptyOf instead." #-}
263−{-# INLINE new #-}
264−new :: MonadIO m => Int -> m (MutArray a)
265−new = emptyOf
266−
267254-- XXX This could be pure?
268255
269256-- |
@@ -331,7 +318,7 @@ putIndices arr = FL.foldlM' step (return ())
331318
332319 where
333320
334− step () (i, x) = putIndex i arr x
321+ step () (i, x) = liftIO (putIndex i arr x)
335322
336323-- | Modify a given index of an array using a modifier function without checking
337324-- the bounds.
@@ -602,16 +589,16 @@ readRev arr@MutArray{..} =
602589arrayChunkSize :: Int
603590arrayChunkSize = 1024
604591
605−-- | Like 'createOf' but does not check the array bounds when writing. The fold
592+-- | Like 'writeN' but does not check the array bounds when writing. The fold
606593-- driver must not call the step function more than 'n' times otherwise it will
607594-- corrupt the memory and crash. This function exists mainly because any
608595-- conditional in the step function blocks fusion causing 10x performance
609596-- slowdown.
610597--
611598-- /Pre-release/
612−{-# INLINE_NORMAL unsafeCreateOf #-}
613−unsafeCreateOf :: MonadIO m => Int -> Fold m a (MutArray a)
614−unsafeCreateOf n = Fold step initial return return
599+{-# INLINE_NORMAL writeNUnsafe #-}
600+writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
601+writeNUnsafe n = Fold step initial return
615602
616603 where
617604
@@ -619,81 +606,59 @@ unsafeCreateOf n = Fold step initial return return
619606
620607 step arr x = FL.Partial <$> snocUnsafe arr x
621608
622−{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
623−{-# INLINE writeNUnsafe #-}
624−writeNUnsafe :: MonadIO m => Int -> Fold m a (MutArray a)
625−writeNUnsafe = unsafeCreateOf
626−
627−-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
609+-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
628610-- 'Array'.
629611--
630−-- >>> createOf n = Fold.take n (MutArray.unsafeCreateOf n)
612+-- >>> writeN n = Fold.take n (MutArray.writeNUnsafe n)
631613--
632614-- /Pre-release/
633−{-# INLINE_NORMAL createOf #-}
634−createOf :: MonadIO m => Int -> Fold m a (MutArray a)
635−createOf n = FL.take n $ unsafeCreateOf n
636−
637−-- XXX Deprecate in major
638−-- {-# DEPRECATED writeN "Please use createOf instead." #-}
639−{-# INLINE writeN #-}
615+{-# INLINE_NORMAL writeN #-}
640616writeN :: MonadIO m => Int -> Fold m a (MutArray a)
641−writeN = createOf
617+writeN n = FL.take n $ writeNUnsafe n
642618
643619-- >>> f n = MutArray.writeAppendWith (* 2) (MutArray.pinnedNew n)
644620-- >>> writeWith n = Fold.rmapM MutArray.rightSize (f n)
645621-- >>> writeWith n = Fold.rmapM MutArray.fromArrayStreamK (MutArray.writeChunks n)
646622
647−-- | @createWith minCount@ folds the whole input to a single array. The array
623+-- | @writeWith minCount@ folds the whole input to a single array. The array
648624-- starts at a size big enough to hold minCount elements, the size is doubled
649625-- every time the array needs to be grown.
650626--
651627-- /Caution! Do not use this on infinite streams./
652628--
653629-- /Pre-release/
654−{-# INLINE_NORMAL createWith #-}
655−createWith :: MonadIO m => Int -> Fold m a (MutArray a)
630+{-# INLINE_NORMAL writeWith #-}
631+writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
656632-- writeWith n = FL.rmapM rightSize $ writeAppendWith (* 2) (pinnedNew n)
657−createWith elemCount = FL.rmapM extract $ FL.foldlM' step initial
633+writeWith elemCount = FL.rmapM extract $ FL.foldlM' step initial
658634
659635 where
660636
661637 initial = do
662− when (elemCount < 0) $ error "createWith: elemCount is negative"
663− new elemCount
638+ when (elemCount < 0) $ error "writeWith: elemCount is negative"
639+ liftIO $ new elemCount
664640
665641 step arr@(MutArray _ start end bound) x
666642 | end == bound = do
667643 let oldSize = end - start
668644 newSize = max (oldSize * 2) 1
669− arr1 <- realloc newSize arr
645+ arr1 <- liftIO $ realloc newSize arr
670646 snocUnsafe arr1 x
671647 step arr x = snocUnsafe arr x
672648
673− -- extract = rightSize
649+ -- extract = liftIO . rightSize
674650 extract = return
675651
676−{-# DEPRECATED writeWith "Please use createWith instead." #-}
677−{-# INLINE writeWith #-}
678−writeWith :: MonadIO m => Int -> Fold m a (MutArray a)
679−writeWith = createWith
680−
681652-- | Fold the whole input to a single array.
682653--
683−-- Same as 'createWith' using an initial array size of 'arrayChunkSize' bytes
654+-- Same as 'writeWith' using an initial array size of 'arrayChunkSize' bytes
684655-- rounded up to the element size.
685656--
686657-- /Caution! Do not use this on infinite streams./
687658--
688−{-# INLINE create #-}
689−create :: MonadIO m => Fold m a (MutArray a)
690−create = writeWith arrayChunkSize
691−
692−-- XXX Deprecate in major
693−-- {-# DEPRECATED write "Please use create instead." #-}
694659{-# INLINE write #-}
695660write :: MonadIO m => Fold m a (MutArray a)
696−write = create
661+write = writeWith arrayChunkSize
697662
698663-- | Create a 'MutArray' from the first @n@ elements of a stream. The
699664-- array is allocated to size @n@, if the stream terminates before @n@
@@ -718,7 +683,7 @@ fromList xs = fromStream $ D.fromList xs
718683{-# INLINABLE fromPureStream #-}
719684fromPureStream :: MonadIO m => Stream Identity a -> m (MutArray a)
720685fromPureStream xs =
721− D.fold write $ D.morphInner (return . runIdentity) xs
686+ liftIO $ D.fold write $ D.morphInner (return . runIdentity) xs
722687
723688-------------------------------------------------------------------------------
724689-- Chunking
@@ -850,7 +815,7 @@ putSliceUnsafe src srcStart dst dstStart len = liftIO $ do
850815
851816{-# INLINE clone #-}
852817clone :: MonadIO m => MutArray a -> m (MutArray a)
853−clone src = do
818+clone src = liftIO $ do
854819 let len = arrLen src
855820 dst <- new len
856821 putSliceUnsafe src 0 dst 0 len
core/src/Streamly/Internal/Data/MutArray/Stream.hsdeleted−145
@@ -1,145 +0,0 @@
1−-- |
2−-- Module : Streamly.Internal.Data.MutArray.Stream
3−-- Copyright : (c) 2019 Composewell Technologies
4−-- License : BSD3-3-Clause
5−-- Maintainer : streamly@composewell.com
6−-- Stability : experimental
7−-- Portability : GHC
8−--
9−-- Combinators to efficiently manipulate streams of mutable arrays.
10−--
11−-- We can either push these in the MutArray module with a "chunks" prefix or
12−-- keep this as a separate module and release it.
13−--
14−module Streamly.Internal.Data.MutArray.Stream
15−{-# DEPRECATED "Please use \"Streamly.Internal.Data.MutArray\" instead." #-}
16− (
17− -- * Generation
18− MArray.chunksOf
19− , MArray.pinnedChunksOf
20− , MArray.writeChunks -- chunksWrite?
21− , MArray.splitOn -- chunksSplitOn
22−
23− -- * Compaction
24− , packArraysChunksOf
25− , MArray.SpliceState (..)
26− , lpackArraysChunksOf
27− , compact -- chunksCompact
28− , compactLE
29− , compactEQ
30− , compactGE
31−
32− -- * Elimination
33− , MArray.flattenArrays -- chunksConcat
34− , MArray.flattenArraysRev -- chunksConcatRev
35− , MArray.fromArrayStreamK -- chunksCoalesce
36− )
37−where
38−
39−import Control.Monad.IO.Class (MonadIO(..))
40−import Streamly.Internal.Data.Unbox (Unbox)
41−import Streamly.Internal.Data.MutArray.Type (MutArray(..))
42−import Streamly.Internal.Data.Fold.Type (Fold(..))
43−import Streamly.Internal.Data.Parser (ParseError)
44−import Streamly.Internal.Data.Stream.Type (Stream)
45−
46−import qualified Streamly.Internal.Data.MutArray as MArray
47−import qualified Streamly.Internal.Data.Fold.Type as FL
48−import qualified Streamly.Internal.Data.Parser as ParserD
49−import qualified Streamly.Internal.Data.Stream as D
50−
51−-------------------------------------------------------------------------------
52−-- Compact
53−-------------------------------------------------------------------------------
54−
55−-- XXX This can be removed once compactLEFold/compactLE are implemented.
56−--
57−-- | This mutates the first array (if it has space) to append values from the
58−-- second one. This would work for immutable arrays as well because an
59−-- immutable array never has space so a new array is allocated instead of
60−-- mutating it.
61−--
62−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
63−-- maximum specified size. Note that if a single array is bigger than the
64−-- specified size we do not split it to fit. When we coalesce multiple arrays
65−-- if the size would exceed the specified size we do not coalesce therefore the
66−-- actual array size may be less than the specified chunk size.
67−--
68−-- @since 0.7.0
69−{-# INLINE packArraysChunksOf #-}
70−packArraysChunksOf :: (MonadIO m, Unbox a)
71− => Int -> D.Stream m (MutArray a) -> D.Stream m (MutArray a)
72−packArraysChunksOf = MArray.compactLE
73−
74−-- XXX Remove this once compactLEFold is implemented
75−-- lpackArraysChunksOf = Fold.many compactLEFold
76−--
77−{-# INLINE lpackArraysChunksOf #-}
78−lpackArraysChunksOf :: (MonadIO m, Unbox a)
79− => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
80−lpackArraysChunksOf = MArray.lCompactGE
81−
82−-- XXX Same as compactLE, to be removed once that is implemented.
83−--
84−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
85−-- maximum specified size in bytes.
86−--
87−-- /Internal/
88−{-# INLINE compact #-}
89−compact :: (MonadIO m, Unbox a)
90− => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
91−compact = packArraysChunksOf
92−
93−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
94−-- maximum specified size. Note that if a single array is bigger than the
95−-- specified size we do not split it to fit. When we coalesce multiple arrays
96−-- if the size would exceed the specified size we do not coalesce therefore the
97−-- actual array size may be less than the specified chunk size.
98−--
99−-- /Internal/
100−{-# INLINE compactLEParserD #-}
101−compactLEParserD ::
102− forall m a. (MonadIO m, Unbox a)
103− => Int -> ParserD.Parser (MutArray a) m (MutArray a)
104−compactLEParserD = MArray.pCompactLE
105−
106−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
107−-- minimum specified size. Note that if all the arrays in the stream together
108−-- are smaller than the specified size the resulting array will be smaller than
109−-- the specified size. When we coalesce multiple arrays if the size would exceed
110−-- the specified size we stop coalescing further.
111−--
112−-- /Internal/
113−{-# INLINE compactGEFold #-}
114−compactGEFold ::
115− forall m a. (MonadIO m, Unbox a)
116− => Int -> FL.Fold m (MutArray a) (MutArray a)
117−compactGEFold = MArray.fCompactGE
118−
119−-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
120−-- maximum specified size in bytes.
121−--
122−-- /Internal/
123−compactLE :: (MonadIO m, Unbox a) =>
124− Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
125−compactLE n = D.parseManyD (compactLEParserD n)
126−
127−-- | Like 'compactLE' but generates arrays of exactly equal to the size
128−-- specified except for the last array in the stream which could be shorter.
129−--
130−-- /Unimplemented/
131−{-# INLINE compactEQ #-}
132−compactEQ :: -- (MonadIO m, Unbox a) =>
133− Int -> Stream m (MutArray a) -> Stream m (MutArray a)
134−compactEQ _n _xs = undefined
135− -- IsStream.fromStreamD $ D.foldMany (compactEQFold n) (IsStream.toStreamD xs)
136−
137−-- | Like 'compactLE' but generates arrays of size greater than or equal to the
138−-- specified except for the last array in the stream which could be shorter.
139−--
140−-- /Internal/
141−{-# INLINE compactGE #-}
142−compactGE ::
143− (MonadIO m, Unbox a)
144− => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
145−compactGE n = D.foldMany (compactGEFold n)
core/src/Streamly/Internal/Data/MutArray/Type.hs+417−1272
@@ -28,80 +28,57 @@
2828
2929module Streamly.Internal.Data.MutArray.Type
3030 (
31− -- ** Type
31+ -- * Type
3232 -- $arrayNotes
3333 MutArray (..)
34−
35− -- ** Conversion
36− -- *** Pinned and Unpinned
34+ , MutableByteArray
35+ , touch
3736 , pin
3837 , unpin
3938 , isPinned
4039
41− -- ** Casting
42− , cast
43− , castUnsafe -- XXX unsafeCast
44− , asBytes
45− , unsafePinnedAsPtr
46− , unsafeAsPtr
47−
40+ -- * Constructing and Writing
4841 -- ** Construction
49− , empty
50−
51− -- *** New
52− -- | New arrays are always empty arrays with some reserve capacity to
53− -- extend the length without reallocating.
54− , emptyOf
55− , newArrayWith -- emptyAlignedWith
56− , pinnedEmptyOf
57− , pinnedNewAligned -- XXX not required
58− -- , new -- uninitialized array of specified length
59−
60− -- *** Cloning
61− , clone
62− , pinnedClone
42+ , nil
6343
64− -- *** Slicing
65− -- | Get a subarray without copying
66− , getSliceUnsafe -- XXX unsafeGetSlice
67− , getSlice
68− , splitAt -- XXX should be able to express using getSlice
69− , breakOn
44+ -- *** Uninitialized Arrays
45+ , pinnedNew
46+ , pinnedNewBytes
47+ , pinnedNewAligned
48+ , new
49+ , newArrayWith
7050
71− -- *** Stream Folds
51+ -- *** From streams
7252 , ArrayUnsafe (..)
73− , unsafeCreateOfWith
74− , unsafeCreateOf
75− , unsafePinnedCreateOf
76− , pinnedCreateOf
77− , createOfWith
78− , createOf
79− , revCreateOf
80−
81− , pinnedCreate
82− , createWith
83− , create
84− -- , revCreate
85−
86− -- *** From containers
53+ , writeNWithUnsafe
54+ , writeNWith
55+ , writeNUnsafe
56+ , pinnedWriteNUnsafe
57+ , writeN
58+ , pinnedWriteN
59+ , pinnedWriteNAligned
60+
61+ , writeWith
62+ , write
63+ , pinnedWrite
64+
65+ , writeRevN
66+ -- , writeRev
67+
68+ -- ** From containers
8769 , fromListN
8870 , pinnedFromListN
8971 , fromList
9072 , pinnedFromList
9173 , fromListRevN
9274 , fromListRev
93− , fromStreamN
94− , fromStream
95− , fromPureStreamN
75+ , fromStreamDN
76+ , fromStreamD
9677 , fromPureStream
97− , fromByteStr#
98− , fromPtrN
99− , fromChunksK
100− , fromChunksRealloced -- fromSmallChunks
10178
102− -- ** Random writes
79+ -- * Random writes
10380 , putIndex
104− , putIndexUnsafe -- XXX unsafePutIndex
81+ , putIndexUnsafe
10582 , putIndices
10683 -- , putFromThenTo
10784 -- , putFrom -- start writing at the given position
@@ -109,71 +86,78 @@ module Streamly.Internal.Data.MutArray.Type
10986 -- , putFromTo
11087 -- , putFromRev
11188 -- , putUptoRev
112− , modifyIndexUnsafe -- XXX unsafeModifyIndex
89+ , modifyIndexUnsafe
11390 , modifyIndex
11491 , modifyIndices
11592 , modify
11693 , swapIndices
11794 , unsafeSwapIndices
11895
119− -- ** Reading
96+ -- * Growing and Shrinking
97+ -- Arrays grow only at the end, though it is possible to grow on both sides
98+ -- and therefore have a cons as well as snoc. But that will require two
99+ -- bounds in the array representation.
120100
121− -- *** Indexing
122− , getIndex
123− , getIndexUnsafe -- XXX unsafeGetIndex
124− -- , getFromThenTo
125− , getIndexRev -- getRevIndex?
126− , indexReader
127− , indexReaderWith
101+ -- ** Appending elements
102+ , snocWith
103+ , snoc
104+ , snocLinear
105+ , snocMay
106+ , snocUnsafe
128107
129− -- *** To Streams
108+ -- ** Appending streams
109+ , writeAppendNUnsafe
110+ , writeAppendN
111+ , writeAppendWith
112+ , writeAppend
113+
114+ -- * Eliminating and Reading
115+
116+ -- ** To streams
117+ , reader
118+ , readerRevWith
119+ , readerRev
120+
121+ -- ** To containers
122+ , toStreamDWith
123+ , toStreamDRevWith
124+ , toStreamKWith
125+ , toStreamKRevWith
130126 , read
131127 , readRev
132− , toStreamWith
133− , toStreamRevWith
134128 , toStreamK
135− , toStreamKWith
136129 , toStreamKRev
137− , toStreamKRevWith
138−
139− -- *** To Containers
140130 , toList
141131
142− -- *** Unfolds
143132 -- experimental
144133 , producerWith
145134 , producer
146135
147− , reader
148− , readerRevWith
149− , readerRev
150−
151− -- ** Size and Capacity
152− -- *** Size
153− , length
154− , byteLength
155−
156− -- *** Capacity
157− -- , capacity
158− , byteCapacity
159− , bytesFree
136+ -- ** Random reads
137+ , getIndex
138+ , getIndexUnsafe
139+ , getIndices
140+ , getIndicesD
141+ -- , getFromThenTo
142+ , getIndexRev
160143
161− -- *** Capacity Management
144+ -- * Memory Management
162145 , blockSize
163146 , arrayChunkBytes
164147 , allocBytesToElemCount
165− , realloc -- this is "resize"
166− , grow
167− , growExp
148+ , realloc
149+ , resize
150+ , resizeExp
168151 , rightSize
169152
170− -- ** Folding
171− , foldl'
172− , foldr
173− , byteCmp
174− , byteEq
153+ -- * Size
154+ , length
155+ , byteLength
156+ -- , capacity
157+ , byteCapacity
158+ , bytesFree
175159
176− -- ** In-place Mutation Algorithms
160+ -- * In-place Mutation Algorithms
177161 , strip
178162 , reverse
179163 , permute
@@ -183,121 +167,62 @@ module Streamly.Internal.Data.MutArray.Type
183167 , mergeBy
184168 , bubble
185169
186− -- ** Growing and Shrinking
187− -- | Arrays grow only at the end, though it is possible to grow on both
188− -- sides and therefore have a cons as well as snoc. But that will require
189− -- both lower and upper bound in the array representation.
190−
191− -- *** Appending elements
192− , snocWith
193− , snoc
194− , snocLinear
195− , snocMay
196− , snocUnsafe -- XXX unsafeSnoc
197−
198− -- *** Appending streams
199− , unsafeAppendN
200− , appendN
201− , appendWith
202− , append
203−
204− -- *** Appending arrays
205− , spliceCopy
206− , spliceWith
207− , splice
208− , spliceExp
209− , spliceUnsafe -- XXX unsafeSplice
210− -- , putSlice
211− -- , appendSlice
212− -- , appendSliceFrom
213−
214− -- ** Serialization using Unbox
215− , pokeAppend
216− , pokeAppendMay
217− , pokeSkipUnsafe -- unsafePokeSkip
170+ -- * Casting
171+ , cast
172+ , castUnsafe
173+ , asBytes
174+ , asPtrUnsafe
218175
219− -- ** Deserialization using Unbox
220− , peekUncons
221− , peekUnconsUnsafe -- unsafePeekUncons
222− , peekSkipUnsafe -- unsafePeekSkip
176+ -- * Folding
177+ , foldl'
178+ , foldr
179+ , cmp
223180
224− -- Arrays of arrays
181+ -- * Arrays of arrays
225182 -- We can add dimensionality parameter to the array type to get
226183 -- multidimensional arrays. Multidimensional arrays would just be a
227184 -- convenience wrapper on top of single dimensional arrays.
228185
229− -- ** Streams of Arrays
230− -- *** Chunk
231− -- | Group a stream into arrays.
186+ -- | Operations dealing with multiple arrays, streams of arrays or
187+ -- multidimensional array representations.
188+
189+ -- ** Construct from streams
232190 , chunksOf
233191 , pinnedChunksOf
234− , buildChunks
192+ , writeChunks
235193
236− -- *** Split
237− -- | Split an array into slices.
194+ -- ** Eliminate to streams
195+ , flattenArrays
196+ , flattenArraysRev
197+ , fromArrayStreamK
238198
199+ -- ** Construct from arrays
200+ -- get chunks without copying
201+ , getSliceUnsafe
202+ , getSlice
239203 -- , getSlicesFromLenN
240− , splitOn
241− -- , slicesOf
242−
243− -- *** Concat
244− -- | Append the arrays in a stream to form a stream of elements.
245− , concatWith
246− , concatRevWith
247− , concat
248− , concatRev
249−
250− -- *** Compact
251− -- | Append the arrays in a stream to form a stream of larger arrays.
252− , SpliceState (..)
253− , pCompactLE
254− , pPinnedCompactLE
255− , compactLeAs
256− , fCompactGE
257− , fPinnedCompactGE
258− , lCompactGE
259− , lPinnedCompactGE
260− , compactGE
261− , compactEQ
262−
263− -- ** Utilities
204+ , splitAt -- XXX should be able to express using getSlice
205+ , breakOn
206+
207+ -- ** Cloning arrays
208+ , clone
209+ , pinnedClone
210+
211+ -- ** Appending arrays
212+ , spliceCopy
213+ , spliceWith
214+ , splice
215+ , spliceExp
216+ , spliceUnsafe
217+ -- , putSlice
218+ -- , appendSlice
219+ -- , appendSliceFrom
220+
221+ -- * Utilities
264222 , roundUpToPower2
265223 , memcpy
266224 , memcmp
267225 , c_memchr
268−
269− -- * Deprecated
270− , asPtrUnsafe
271− , writeChunks
272− , flattenArrays
273− , flattenArraysRev
274− , fromArrayStreamK
275− , fromStreamDN
276− , fromStreamD
277− , cmp
278− , getIndices
279− , getIndicesWith
280− , resize
281− , resizeExp
282− , nil
283− , new
284− , pinnedNew
285− , pinnedNewBytes
286− , writeAppendNUnsafe
287− , writeAppendN
288− , writeAppendWith
289− , writeAppend
290− , writeNWithUnsafe
291− , writeNWith
292− , writeNUnsafe
293− , pinnedWriteNUnsafe
294− , writeN
295− , pinnedWriteN
296− , pinnedWriteNAligned -- XXX not required
297− , writeWith
298− , write
299− , pinnedWrite
300− , writeRevN
301226 )
302227where
303228
@@ -308,52 +233,49 @@ where
308233
309234import Control.Monad (when, void)
310235import Control.Monad.IO.Class (MonadIO(..))
311−import Data.Bifunctor (first)
312236import Data.Bits (shiftR, (.|.), (.&.))
313237import Data.Functor.Identity (Identity(..))
314238import Data.Proxy (Proxy(..))
315239import Data.Word (Word8)
316240import Foreign.C.Types (CSize(..), CInt(..))
317241import Foreign.Ptr (plusPtr, minusPtr, nullPtr)
318−import Streamly.Internal.Data.MutByteArray.Type
319− ( MutByteArray(..)
242+import Streamly.Internal.Data.Unbox
243+ ( MutableByteArray(..)
244+ , Unbox(..)
320245 , PinnedState(..)
321246 , getMutableByteArray#
247+ , touch
322248 , putSliceUnsafe
323249 )
324−import Streamly.Internal.Data.Unbox (Unbox(..))
325250import GHC.Base
326251 ( IO(..)
327252 , Int(..)
253+ , byteArrayContents#
328254 , compareByteArrays#
329255 , copyMutableByteArray#
330256 )
331257import GHC.Base (noinline)
332−import GHC.Exts (unsafeCoerce#, Addr#)
258+import GHC.Exts (unsafeCoerce#)
333259import GHC.Ptr (Ptr(..))
334260
335261import Streamly.Internal.Data.Fold.Type (Fold(..))
336262import Streamly.Internal.Data.Producer.Type (Producer (..))
337263import Streamly.Internal.Data.Stream.Type (Stream)
338−import Streamly.Internal.Data.Parser.Type (Parser (..))
339264import Streamly.Internal.Data.StreamK.Type (StreamK)
340265import Streamly.Internal.Data.SVar.Type (adaptState, defState)
341−import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
342266import Streamly.Internal.Data.Unfold.Type (Unfold(..))
343267import Streamly.Internal.System.IO (arrayPayloadSize, defaultChunkSize)
344268
345269import qualified Streamly.Internal.Data.Fold.Type as FL
346−import qualified Streamly.Internal.Data.MutByteArray.Type as Unboxed
347−import qualified Streamly.Internal.Data.Parser.Type as Parser
348−-- import qualified Streamly.Internal.Data.Fold.Type as Fold
349270import qualified Streamly.Internal.Data.Producer as Producer
350271import qualified Streamly.Internal.Data.Stream.Type as D
351272import qualified Streamly.Internal.Data.Stream.Lift as D
352273import qualified Streamly.Internal.Data.StreamK.Type as K
274+import qualified Streamly.Internal.Data.Unbox as Unboxed
353275import qualified Prelude
354276
355277import Prelude hiding
356− (Foldable(..), concat, read, unlines, splitAt, reverse, truncate)
278+ (Foldable(..), read, unlines, splitAt, reverse, truncate)
357279
358280#include "DocTestDataMutArray.hs"
359281
@@ -361,7 +283,6 @@ import Prelude hiding
361283-- Foreign helpers
362284-------------------------------------------------------------------------------
363285
364−-- NOTE: Have to be "ccall unsafe" so that we can pass unpinned memory to these
365286foreign import ccall unsafe "string.h memcpy" c_memcpy
366287 :: Ptr Word8 -> Ptr Word8 -> CSize -> IO (Ptr Word8)
367288
@@ -371,9 +292,6 @@ foreign import ccall unsafe "string.h memchr" c_memchr
371292foreign import ccall unsafe "string.h memcmp" c_memcmp
372293 :: Ptr Word8 -> Ptr Word8 -> CSize -> IO CInt
373294
374−foreign import ccall unsafe "string.h strlen" c_strlen
375− :: Ptr Word8 -> IO CSize
376−
377295-- | Given an 'Unboxed' type (unused first arg) and a number of bytes, return
378296-- how many elements of that type will completely fit in those bytes.
379297--
@@ -428,7 +346,7 @@ data MutArray a =
428346 -- The array is a range into arrContents. arrContents may be a superset of
429347 -- the slice represented by the array. All offsets are in bytes.
430348 MutArray
431− { arrContents :: {-# UNPACK #-} !MutByteArray
349+ { arrContents :: {-# UNPACK #-} !MutableByteArray
432350 , arrStart :: {-# UNPACK #-} !Int -- ^ index into arrContents
433351 , arrEnd :: {-# UNPACK #-} !Int -- ^ index into arrContents
434352 -- Represents the first invalid index of
@@ -440,8 +358,8 @@ data MutArray a =
440358-- Pinning & Unpinning
441359-------------------------------------------------------------------------------
442360
443−-- | Return a copy of the array in pinned memory if unpinned, else return the
444−-- original array.
361+-- | Copy the 'MutArray' to pinned memory if unpinned, else do nothing. The
362+-- overhead is a copy if the input array is unpinned.
445363{-# INLINE pin #-}
446364pin :: MutArray a -> IO (MutArray a)
447365pin arr@MutArray{..} =
@@ -449,8 +367,8 @@ pin arr@MutArray{..} =
449367 then pure arr
450368 else pinnedClone arr
451369
452−-- | Return a copy of the array in unpinned memory if pinned, else return the
453−-- original array.
370+-- | Copy the 'MutArray' to unpinned memory if pinned, else do nothing. The
371+-- overhead is a copy if the input array is pinned.
454372{-# INLINE unpin #-}
455373unpin :: MutArray a -> IO (MutArray a)
456374unpin arr@MutArray{..} =
@@ -458,7 +376,6 @@ unpin arr@MutArray{..} =
458376 then clone arr
459377 else pure arr
460378
461−-- | Return 'True' if the array is allocated in pinned memory.
462379{-# INLINE isPinned #-}
463380isPinned :: MutArray a -> Bool
464381isPinned MutArray{..} = Unboxed.isPinned arrContents
@@ -479,8 +396,6 @@ isPinned MutArray{..} = Unboxed.isPinned arrContents
479396-- we need more than that. See stg_pinnedNewAlignedByteArrayzh and
480397-- allocatePinned in GHC source.
481398
482−-- XXX Rename to emptyAlignedWith, alignSize should be first arg.
483−
484399-- | @newArrayWith allocator alignment count@ allocates a new array of zero
485400-- length and with a capacity to hold @count@ elements, using @allocator
486401-- size alignment@ as the memory allocator function.
@@ -493,7 +408,7 @@ isPinned MutArray{..} = Unboxed.isPinned arrContents
493408-- /Pre-release/
494409{-# INLINE newArrayWith #-}
495410newArrayWith :: forall m a. (MonadIO m, Unbox a)
496− => (Int -> Int -> m MutByteArray) -> Int -> Int -> m (MutArray a)
411+ => (Int -> Int -> m MutableByteArray) -> Int -> Int -> m (MutArray a)
497412newArrayWith alloc alignSize count = do
498413 let size = max (count * SIZE_OF(a)) 0
499414 contents <- alloc size alignSize
@@ -504,25 +419,12 @@ newArrayWith alloc alignSize count = do
504419 , arrBound = size
505420 }
506421
507−-- For arrays "nil" sounds a bit odd. empty is better. The only problem with
508−-- empty is that it is also used by the Alternative type class. But assuming we
509−-- will mostly import the Array module qualified this should be fine.
510−
511−-- | Create an empty array.
512−empty ::
513−#ifdef DEVBUILD
514− Unbox a =>
515−#endif
516− MutArray a
517−empty = MutArray Unboxed.empty 0 0 0
518−
519−{-# DEPRECATED nil "Please use empty instead." #-}
520422nil ::
521423#ifdef DEVBUILD
522424 Unbox a =>
523425#endif
524426 MutArray a
525−nil = empty
427+nil = MutArray Unboxed.nil 0 0 0
526428
527429{-# INLINE newBytesAs #-}
528430newBytesAs :: MonadIO m =>
@@ -539,13 +441,12 @@ newBytesAs ps bytes = do
539441 , arrBound = bytes
540442 }
541443
542−-- | Allocates a pinned empty array that with a reserved capacity of bytes.
543−-- The memory of the array is uninitialized and the allocation is aligned as
544−-- per the 'Unboxed' instance of the type.
444+-- | Allocates a pinned empty array that can hold 'count' items. The memory of
445+-- the array is uninitialized and the allocation is aligned as per the
446+-- 'Unboxed' instance of the type.
545447--
546448-- /Pre-release/
547449{-# INLINE pinnedNewBytes #-}
548−{-# DEPRECATED pinnedNewBytes "Please use pinnedEmptyOf with appropriate calculation" #-}
549450pinnedNewBytes :: MonadIO m =>
550451#ifdef DEVBUILD
551452 Unbox a =>
@@ -570,31 +471,21 @@ newAs ps =
570471 (error "new: alignment is not used in unpinned arrays.")
571472
572473-- XXX can unaligned allocation be more efficient when alignment is not needed?
573−
574−-- | Allocates a pinned array of zero length but growable to the specified
575−-- capacity without reallocation.
576−{-# INLINE pinnedEmptyOf #-}
577−pinnedEmptyOf :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
578−pinnedEmptyOf = newAs Pinned
579−
580−-- XXX Deprecate in major
581−-- {-# DEPRECATED pinnedNew "Please use pinnedEmptyOf instead." #-}
474+--
475+-- | Allocates an empty pinned array that can hold 'count' items. The memory of
476+-- the array is uninitialized and the allocation is aligned as per the 'Unboxed'
477+-- instance of the type.
478+--
582479{-# INLINE pinnedNew #-}
583480pinnedNew :: forall m a. (MonadIO m, Unbox a) => Int -> m (MutArray a)
584−pinnedNew = pinnedEmptyOf
481+pinnedNew = newAs Pinned
585482
586−-- | Allocates an unpinned array of zero length but growable to the specified
587−-- capacity without reallocation.
483+-- | Allocates an empty unpinned array that can hold 'count' items. The memory
484+-- of the array is uninitialized.
588485--
589−{-# INLINE emptyOf #-}
590−emptyOf :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
591−emptyOf = newAs Unpinned
592−
593−-- XXX Deprecate in major
594−-- {-# DEPRECATED new "Please use emptyOf instead." #-}
595486{-# INLINE new #-}
596487new :: (MonadIO m, Unbox a) => Int -> m (MutArray a)
597−new = emptyOf
488+new = newAs Unpinned
598489
599490-------------------------------------------------------------------------------
600491-- Random writes
@@ -610,7 +501,7 @@ putIndexUnsafe :: forall m a. (MonadIO m, Unbox a)
610501putIndexUnsafe i MutArray{..} x = do
611502 let index = INDEX_OF(arrStart, i, a)
612503 assert (i >= 0 && INDEX_VALID(index, arrEnd, a)) (return ())
613− liftIO $ pokeAt index arrContents x
504+ liftIO $ pokeByteIndex index arrContents x
614505
615506invalidIndex :: String -> Int -> a
616507invalidIndex label i =
@@ -628,7 +519,7 @@ putIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m ()
628519putIndex i MutArray{..} x = do
629520 let index = INDEX_OF(arrStart,i,a)
630521 if i >= 0 && INDEX_VALID(index,arrEnd,a)
631− then liftIO $ pokeAt index arrContents x
522+ then liftIO $ pokeByteIndex index arrContents x
632523 else invalidIndex "putIndex" i
633524
634525-- | Write an input stream of (index, value) pairs to an array. Throws an
@@ -642,7 +533,7 @@ putIndices arr = FL.foldlM' step (return ())
642533
643534 where
644535
645− step () (i, x) = putIndex i arr x
536+ step () (i, x) = liftIO (putIndex i arr x)
646537
647538-- | Modify a given index of an array using a modifier function.
648539--
@@ -654,9 +545,9 @@ modifyIndexUnsafe :: forall m a b. (MonadIO m, Unbox a) =>
654545modifyIndexUnsafe i MutArray{..} f = liftIO $ do
655546 let index = INDEX_OF(arrStart,i,a)
656547 assert (i >= 0 && INDEX_NEXT(index,a) <= arrEnd) (return ())
657− r <- peekAt index arrContents
548+ r <- peekByteIndex index arrContents
658549 let (x, res) = f r
659− pokeAt index arrContents x
550+ pokeByteIndex index arrContents x
660551 return res
661552
662553-- | Modify a given index of an array using a modifier function.
@@ -668,9 +559,9 @@ modifyIndex i MutArray{..} f = do
668559 let index = INDEX_OF(arrStart,i,a)
669560 if i >= 0 && INDEX_VALID(index,arrEnd,a)
670561 then liftIO $ do
671− r <- peekAt index arrContents
562+ r <- peekByteIndex index arrContents
672563 let (x, res) = f r
673− pokeAt index arrContents x
564+ pokeByteIndex index arrContents x
674565 return res
675566 else invalidIndex "modifyIndex" i
676567
@@ -704,8 +595,8 @@ modify MutArray{..} f = liftIO $
704595
705596 go i =
706597 when (INDEX_VALID(i,arrEnd,a)) $ do
707− r <- peekAt i arrContents
708− pokeAt i arrContents (f r)
598+ r <- peekByteIndex i arrContents
599+ pokeByteIndex i arrContents (f r)
709600 go (INDEX_NEXT(i,a))
710601
711602-- XXX We could specify the number of bytes to swap instead of Proxy. Need
@@ -714,15 +605,15 @@ modify MutArray{..} f = liftIO $
714605swapArrayByteIndices ::
715606 forall a. Unbox a
716607 => Proxy a
717− -> MutByteArray
608+ -> MutableByteArray
718609 -> Int
719610 -> Int
720611 -> IO ()
721612swapArrayByteIndices _ arrContents i1 i2 = do
722− r1 <- peekAt i1 arrContents
723− r2 <- peekAt i2 arrContents
724− pokeAt i1 arrContents (r2 :: a)
725− pokeAt i2 arrContents (r1 :: a)
613+ r1 <- peekByteIndex i1 arrContents
614+ r2 <- peekByteIndex i2 arrContents
615+ pokeByteIndex i1 arrContents (r2 :: a)
616+ pokeByteIndex i2 arrContents (r1 :: a)
726617
727618-- | Swap the elements at two indices without validating the indices.
728619--
@@ -838,23 +729,26 @@ arrayChunkBytes = 1024
838729roundDownTo :: Int -> Int -> Int
839730roundDownTo elemSize size = size - (size `mod` elemSize)
840731
732+-- XXX See if resizing can be implemented by reading the old array as a stream
733+-- and then using writeN to the new array.
734+--
841735-- NOTE: we are passing elemSize explicitly to avoid an Unboxed constraint.
842736-- Since this is not inlined Unboxed consrraint leads to dictionary passing
843737-- which complicates some inspection tests.
844738--
845−{-# NOINLINE reallocExplicitAs #-}
846−reallocExplicitAs :: PinnedState -> Int -> Int -> MutArray a -> IO (MutArray a)
847−reallocExplicitAs ps elemSize newCapacityInBytes MutArray{..} = do
739+{-# NOINLINE reallocExplicit #-}
740+reallocExplicit :: Int -> Int -> MutArray a -> IO (MutArray a)
741+reallocExplicit elemSize newCapacityInBytes MutArray{..} = do
848742 assertM(arrEnd <= arrBound)
849743
850744 -- Allocate new array
851745 let newCapMaxInBytes = roundUpLargeArray newCapacityInBytes
852746 contents <-
853− if ps == Pinned
854− then Unboxed.pinnedNew newCapMaxInBytes
855− else Unboxed.new newCapMaxInBytes
856− let !(MutByteArray mbarrFrom#) = arrContents
857− !(MutByteArray mbarrTo#) = contents
747+ if Unboxed.isPinned arrContents
748+ then Unboxed.pinnedNewBytes newCapMaxInBytes
749+ else Unboxed.newBytes newCapMaxInBytes
750+ let !(MutableByteArray mbarrFrom#) = arrContents
751+ !(MutableByteArray mbarrTo#) = contents
858752
859753 -- Copy old data
860754 let oldStart = arrStart
@@ -875,10 +769,6 @@ reallocExplicitAs ps elemSize newCapacityInBytes MutArray{..} = do
875769 , arrBound = newCapInBytes
876770 }
877771
878−-- XXX Should these be called reallocBytes etc?
879−-- XXX We may also need reallocAs to allocate as pinned/unpinned explicitly. In
880−-- fact clone/pinnedClone can be implemented using reallocAs.
881−
882772-- | @realloc newCapacity array@ reallocates the array to the specified
883773-- capacity in bytes.
884774--
@@ -890,12 +780,7 @@ reallocExplicitAs ps elemSize newCapacityInBytes MutArray{..} = do
890780-- If the original array is pinned, the newly allocated array is also pinned.
891781{-# INLINABLE realloc #-}
892782realloc :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (MutArray a)
893−realloc bytes arr =
894− let ps =
895− if isPinned arr
896− then Pinned
897− else Unpinned
898− in liftIO $ reallocExplicitAs ps (SIZE_OF(a)) bytes arr
783+realloc bytes arr = liftIO $ reallocExplicit (SIZE_OF(a)) bytes arr
899784
900785-- | @reallocWith label capSizer minIncrBytes array@. The label is used
901786-- in error messages and the capSizer is used to determine the capacity of the
@@ -918,14 +803,14 @@ reallocWith label capSizer minIncrBytes arr = do
918803 where
919804
920805 badSize newSize =
921− Prelude.concat
806+ concat
922807 [ label
923808 , ": new array size (in bytes) is less than required size "
924809 , show newSize
925810 , ". Please check the sizing function passed."
926811 ]
927812
928−-- | @grow newCapacity array@ changes the total capacity of the array so that
813+-- | @resize newCapacity array@ changes the total capacity of the array so that
929814-- it is enough to hold the specified number of elements. Nothing is done if
930815-- the specified capacity is less than the length of the array.
931816--
@@ -933,46 +818,34 @@ reallocWith label capSizer minIncrBytes arr = do
933818-- the block size (4K).
934819--
935820-- /Pre-release/
936−{-# INLINE grow #-}
937−grow :: forall m a. (MonadIO m, Unbox a) =>
821+{-# INLINE resize #-}
822+resize :: forall m a. (MonadIO m, Unbox a) =>
938823 Int -> MutArray a -> m (MutArray a)
939−grow nElems arr@MutArray{..} = do
824+resize nElems arr@MutArray{..} = do
940825 let req = SIZE_OF(a) * nElems
941− cap = arrBound - arrStart
942− if req < cap
826+ len = arrEnd - arrStart
827+ if req < len
943828 then return arr
944829 else realloc req arr
945830
946−{-# DEPRECATED resize "Please use grow instead." #-}
947−{-# INLINE resize #-}
948−resize :: forall m a. (MonadIO m, Unbox a) =>
949− Int -> MutArray a -> m (MutArray a)
950−resize = grow
951−
952−-- | Like 'grow' but if the requested byte capacity is more than
953−-- 'largeObjectThreshold' then it is rounded up to the closest power of 2.
831+-- | Like 'resize' but if the byte capacity is more than 'largeObjectThreshold'
832+-- then it is rounded up to the closest power of 2.
954833--
955834-- /Pre-release/
956−{-# INLINE growExp #-}
957−growExp :: forall m a. (MonadIO m, Unbox a) =>
835+{-# INLINE resizeExp #-}
836+resizeExp :: forall m a. (MonadIO m, Unbox a) =>
958837 Int -> MutArray a -> m (MutArray a)
959−growExp nElems arr@MutArray{..} = do
838+resizeExp nElems arr@MutArray{..} = do
960839 let req = roundUpLargeArray (SIZE_OF(a) * nElems)
961840 req1 =
962841 if req > largeObjectThreshold
963842 then roundUpToPower2 req
964843 else req
965− cap = arrBound - arrStart
966− if req1 < cap
844+ len = arrEnd - arrStart
845+ if req1 < len
967846 then return arr
968847 else realloc req1 arr
969848
970−{-# DEPRECATED resizeExp "Please use growExp instead." #-}
971−{-# INLINE resizeExp #-}
972−resizeExp :: forall m a. (MonadIO m, Unbox a) =>
973− Int -> MutArray a -> m (MutArray a)
974−resizeExp = growExp
975−
976849-- | Resize the allocated memory to drop any reserved free space at the end of
977850-- the array and reallocate it to reduce wastage.
978851--
@@ -1017,7 +890,7 @@ rightSize arr@MutArray{..} = do
1017890snocNewEnd :: (MonadIO m, Unbox a) => Int -> MutArray a -> a -> m (MutArray a)
1018891snocNewEnd newEnd arr@MutArray{..} x = liftIO $ do
1019892 assert (newEnd <= arrBound) (return ())
1020− pokeAt arrEnd arrContents x
893+ pokeByteIndex arrEnd arrContents x
1021894 return $ arr {arrEnd = newEnd}
1022895
1023896-- | Really really unsafe, appends the element into the first array, may
@@ -1037,7 +910,7 @@ snocUnsafe arr@MutArray{..} = snocNewEnd (INDEX_NEXT(arrEnd,a)) arr
1037910{-# INLINE snocMay #-}
1038911snocMay :: forall m a. (MonadIO m, Unbox a) =>
1039912 MutArray a -> a -> m (Maybe (MutArray a))
1040−snocMay arr@MutArray{..} x = do
913+snocMay arr@MutArray{..} x = liftIO $ do
1041914 let newEnd = INDEX_NEXT(arrEnd,a)
1042915 if newEnd <= arrBound
1043916 then Just <$> snocNewEnd newEnd arr x
@@ -1051,7 +924,7 @@ snocWithRealloc :: forall m a. (MonadIO m, Unbox a) =>
1051924 -> a
1052925 -> m (MutArray a)
1053926snocWithRealloc sizer arr x = do
1054− arr1 <- reallocWith "snocWith" sizer (SIZE_OF(a)) arr
927+ arr1 <- liftIO $ reallocWith "snocWith" sizer (SIZE_OF(a)) arr
1055928 snocUnsafe arr1 x
1056929
1057930-- | @snocWith sizer arr elem@ mutates @arr@ to append @elem@. The length of
@@ -1073,7 +946,7 @@ snocWith :: forall m a. (MonadIO m, Unbox a) =>
1073946 -> MutArray a
1074947 -> a
1075948 -> m (MutArray a)
1076−snocWith allocSize arr x = do
949+snocWith allocSize arr x = liftIO $ do
1077950 let newEnd = INDEX_NEXT(arrEnd arr,a)
1078951 if newEnd <= arrBound arr
1079952 then snocNewEnd newEnd arr x
@@ -1117,126 +990,6 @@ snoc = snocWith f
1117990 then oldSize * 2
1118991 else roundUpToPower2 oldSize * 2
1119992
1120−-------------------------------------------------------------------------------
1121−-- Serialization/Deserialization
1122−-------------------------------------------------------------------------------
1123−
1124−{-# INLINE pokeNewEnd #-}
1125−pokeNewEnd :: (MonadIO m, Unbox a) =>
1126− Int -> MutArray Word8 -> a -> m (MutArray Word8)
1127−pokeNewEnd newEnd arr@MutArray{..} x = liftIO $ do
1128− assert (newEnd <= arrBound) (return ())
1129− liftIO $ pokeAt arrEnd arrContents x
1130− return $ arr {arrEnd = newEnd}
1131−
1132−-- | Really really unsafe, unboxes a Haskell type and appends the resulting
1133−-- bytes to the byte array, may cause silent data corruption or if you are
1134−-- lucky a segfault if the array does not have enough space to append the
1135−-- element.
1136−--
1137−-- /Internal/
1138−{-# INLINE pokeAppendUnsafe #-}
1139−pokeAppendUnsafe :: forall m a. (MonadIO m, Unbox a) =>
1140− MutArray Word8 -> a -> m (MutArray Word8)
1141−pokeAppendUnsafe arr@MutArray{..} = pokeNewEnd (arrEnd + SIZE_OF(a)) arr
1142−
1143−-- | Skip the specified number of bytes in the array. The data in the skipped
1144−-- region remains uninitialzed.
1145−{-# INLINE pokeSkipUnsafe #-}
1146−pokeSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
1147−pokeSkipUnsafe n arr@MutArray{..} = do
1148− let newEnd = arrEnd + n
1149− in assert (newEnd <= arrBound) (arr {arrEnd = newEnd})
1150−
1151−-- | Like 'pokeAppend' but does not grow the array when pre-allocated array
1152−-- capacity becomes full.
1153−--
1154−-- /Internal/
1155−{-# INLINE pokeAppendMay #-}
1156−pokeAppendMay :: forall m a. (MonadIO m, Unbox a) =>
1157− MutArray Word8 -> a -> m (Maybe (MutArray Word8))
1158−pokeAppendMay arr@MutArray{..} x = liftIO $ do
1159− let newEnd = arrEnd + SIZE_OF(a)
1160− if newEnd <= arrBound
1161− then Just <$> pokeNewEnd newEnd arr x
1162− else return Nothing
1163−
1164−{-# NOINLINE pokeWithRealloc #-}
1165−pokeWithRealloc :: forall m a. (MonadIO m, Unbox a) =>
1166− (Int -> Int)
1167− -> MutArray Word8
1168− -> a
1169− -> m (MutArray Word8)
1170−pokeWithRealloc sizer arr x = do
1171− arr1 <- liftIO $ reallocWith "pokeWithRealloc" sizer (SIZE_OF(a)) arr
1172− pokeAppendUnsafe arr1 x
1173−
1174−{-# INLINE pokeAppendWith #-}
1175−pokeAppendWith :: forall m a. (MonadIO m, Unbox a) =>
1176− (Int -> Int)
1177− -> MutArray Word8
1178− -> a
1179− -> m (MutArray Word8)
1180−pokeAppendWith allocSize arr x = liftIO $ do
1181− let newEnd = arrEnd arr + SIZE_OF(a)
1182− if newEnd <= arrBound arr
1183− then pokeNewEnd newEnd arr x
1184− else pokeWithRealloc allocSize arr x
1185−
1186−-- | Unbox a Haskell type and append the resulting bytes to a mutable byte
1187−-- array. The array is grown exponentially when more space is needed.
1188−--
1189−-- Definition:
1190−--
1191−-- >>> pokeAppend arr x = MutArray.castUnsafe <$> MutArray.snoc (MutArray.castUnsafe arr) x
1192−--
1193−{-# INLINE pokeAppend #-}
1194−pokeAppend :: forall m a. (MonadIO m, Unbox a) =>
1195− MutArray Word8 -> a -> m (MutArray Word8)
1196−pokeAppend = pokeAppendWith f
1197−
1198− where
1199−
1200− f oldSize =
1201− if isPower2 oldSize
1202− then oldSize * 2
1203− else roundUpToPower2 oldSize * 2
1204−
1205−-- | Really really unsafe, create a Haskell value from an unboxed byte array,
1206−-- does not check if the array is big enough, may return garbage or if you are
1207−-- lucky may cause a segfault.
1208−--
1209−-- /Internal/
1210−{-# INLINE peekUnconsUnsafe #-}
1211−peekUnconsUnsafe :: forall m a. (MonadIO m, Unbox a) =>
1212− MutArray Word8 -> m (a, MutArray Word8)
1213−peekUnconsUnsafe MutArray{..} = do
1214− let start1 = arrStart + SIZE_OF(a)
1215− assert (start1 <= arrEnd) (return ())
1216− liftIO $ do
1217− r <- peekAt arrStart arrContents
1218− return (r, MutArray arrContents start1 arrEnd arrBound)
1219−
1220−-- | Discard the specified number of bytes in the array.
1221−{-# INLINE peekSkipUnsafe #-}
1222−peekSkipUnsafe :: Int -> MutArray Word8 -> MutArray Word8
1223−peekSkipUnsafe n MutArray{..} =
1224− let start1 = arrStart + n
1225− in assert (start1 <= arrEnd) (MutArray arrContents start1 arrEnd arrBound)
1226−
1227−-- | Create a Haskell value from its unboxed representation from the head of a
1228−-- byte array, return the value and the remaining array.
1229−{-# INLINE peekUncons #-}
1230−peekUncons :: forall m a. (MonadIO m, Unbox a) =>
1231− MutArray Word8 -> m (Maybe a, MutArray Word8)
1232−peekUncons arr@MutArray{..} = do
1233− let start1 = arrStart + SIZE_OF(a)
1234− if start1 > arrEnd
1235− then return (Nothing, arr)
1236− else liftIO $ do
1237− r <- peekAt arrStart arrContents
1238− return (Just r, MutArray arrContents start1 arrEnd arrBound)
1239−
1240993-------------------------------------------------------------------------------
1241994-- Random reads
1242995-------------------------------------------------------------------------------
@@ -1251,7 +1004,7 @@ getIndexUnsafe :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
12511004getIndexUnsafe i MutArray{..} = do
12521005 let index = INDEX_OF(arrStart,i,a)
12531006 assert (i >= 0 && INDEX_VALID(index,arrEnd,a)) (return ())
1254− liftIO $ peekAt index arrContents
1007+ liftIO $ peekByteIndex index arrContents
12551008
12561009-- | /O(1)/ Lookup the element at the given index. Index starts from 0.
12571010--
@@ -1260,7 +1013,7 @@ getIndex :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m (Maybe a)
12601013getIndex i MutArray{..} = do
12611014 let index = INDEX_OF(arrStart,i,a)
12621015 if i >= 0 && INDEX_VALID(index,arrEnd,a)
1263− then liftIO $ Just <$> peekAt index arrContents
1016+ then liftIO $ Just <$> peekByteIndex index arrContents
12641017 else return Nothing
12651018
12661019-- | /O(1)/ Lookup the element at the given index from the end of the array.
@@ -1273,16 +1026,21 @@ getIndexRev :: forall m a. (MonadIO m, Unbox a) => Int -> MutArray a -> m a
12731026getIndexRev i MutArray{..} = do
12741027 let index = RINDEX_OF(arrEnd,i,a)
12751028 if i >= 0 && index >= arrStart
1276− then liftIO $ peekAt index arrContents
1029+ then liftIO $ peekByteIndex index arrContents
12771030 else invalidIndex "getIndexRev" i
12781031
12791032data GetIndicesState contents start end st =
12801033 GetIndicesState contents start end st
12811034
1282−{-# INLINE indexReaderWith #-}
1283−indexReaderWith :: (Monad m, Unbox a) =>
1035+-- | Given an unfold that generates array indices, read the elements on those
1036+-- indices from the supplied MutArray. An error is thrown if an index is out of
1037+-- bounds.
1038+--
1039+-- /Pre-release/
1040+{-# INLINE getIndicesD #-}
1041+getIndicesD :: (Monad m, Unbox a) =>
12841042 (forall b. IO b -> m b) -> D.Stream m Int -> Unfold m (MutArray a) a
1285−indexReaderWith liftio (D.Stream stepi sti) = Unfold step inject
1043+getIndicesD liftio (D.Stream stepi sti) = Unfold step inject
12861044
12871045 where
12881046
@@ -1301,26 +1059,9 @@ indexReaderWith liftio (D.Stream stepi sti) = Unfold step inject
13011059 D.Skip s -> return $ D.Skip (GetIndicesState contents start end s)
13021060 D.Stop -> return D.Stop
13031061
1304−{-# DEPRECATED getIndicesWith "Please use indexReaderWith instead." #-}
1305−{-# INLINE getIndicesWith #-}
1306−getIndicesWith :: (Monad m, Unbox a) =>
1307− (forall b. IO b -> m b) -> D.Stream m Int -> Unfold m (MutArray a) a
1308−getIndicesWith = indexReaderWith
1309−
1310−-- | Given an unfold that generates array indices, read the elements on those
1311−-- indices from the supplied MutArray. An error is thrown if an index is out of
1312−-- bounds.
1313−--
1314−-- /Pre-release/
1315−{-# INLINE indexReader #-}
1316−indexReader :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1317−indexReader = indexReaderWith liftIO
1318−
1319−-- XXX DO NOT REMOVE, change the signature to use Stream instead of unfold
1320−{-# DEPRECATED getIndices "Please use indexReader instead." #-}
13211062{-# INLINE getIndices #-}
13221063getIndices :: (MonadIO m, Unbox a) => Stream m Int -> Unfold m (MutArray a) a
1323−getIndices = indexReader
1064+getIndices = getIndicesD liftIO
13241065
13251066-------------------------------------------------------------------------------
13261067-- Subarrays
@@ -1425,7 +1166,7 @@ partitionBy f arr@MutArray{..} = liftIO $ do
14251166
14261167 -- Invariant low < high on entry, and on return as well
14271168 moveHigh low high = do
1428− h <- peekAt high arrContents
1169+ h <- peekByteIndex high arrContents
14291170 if f h
14301171 then
14311172 -- Correctly classified, continue the loop
@@ -1445,7 +1186,7 @@ partitionBy f arr@MutArray{..} = liftIO $ do
14451186 -- low <= high
14461187 -- Both low and high are valid locations within the array
14471188 go low high = do
1448− l <- peekAt low arrContents
1189+ l <- peekByteIndex low arrContents
14491190 if f l
14501191 then
14511192 -- low is wrongly classified
@@ -1456,8 +1197,8 @@ partitionBy f arr@MutArray{..} = liftIO $ do
14561197 case r of
14571198 Nothing -> return low
14581199 Just (high1, h) -> do -- low < high1
1459− pokeAt low arrContents h
1460− pokeAt high1 arrContents l
1200+ pokeByteIndex low arrContents h
1201+ pokeByteIndex high1 arrContents l
14611202 let low1 = INDEX_NEXT(low,a)
14621203 high2 = INDEX_PREV(high1,a)
14631204 if low1 <= high2
@@ -1591,7 +1332,7 @@ chunksOfAs ps n (D.Stream step state) =
15911332 r <- step (adaptState gst) st
15921333 case r of
15931334 D.Yield x s -> do
1594− liftIO $ pokeAt end contents x
1335+ liftIO $ pokeByteIndex end contents x
15951336 let end1 = INDEX_NEXT(end,a)
15961337 return $
15971338 if end1 >= bound
@@ -1610,12 +1351,10 @@ chunksOfAs ps n (D.Stream step state) =
16101351
16111352 step' _ GroupFinish = return D.Stop
16121353
1613−-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
1614−-- @n@ elements each.
1615−--
1616−-- Same as the following but may be more efficient:
1354+-- | @chunksOf n stream@ groups the input stream into a stream of
1355+-- arrays of size n.
16171356--
1618−-- >>> chunksOf n = Stream.foldMany (MutArray.createOf n)
1357+-- @chunksOf n = StreamD.foldMany (MutArray.writeN n)@
16191358--
16201359-- /Pre-release/
16211360{-# INLINE_NORMAL chunksOf #-}
@@ -1624,29 +1363,17 @@ chunksOf :: forall m a. (MonadIO m, Unbox a)
16241363-- XXX the idiomatic implementation leads to large regression in the D.reverse'
16251364-- benchmark. It seems it has difficulty producing optimized code when
16261365-- converting to StreamK. Investigate GHC optimizations.
1627−-- chunksOf n = D.foldMany (createOf n)
1366+-- chunksOf n = D.foldMany (writeN n)
16281367chunksOf = chunksOfAs Unpinned
16291368
16301369-- | Like 'chunksOf' but creates pinned arrays.
16311370{-# INLINE_NORMAL pinnedChunksOf #-}
16321371pinnedChunksOf :: forall m a. (MonadIO m, Unbox a)
16331372 => Int -> D.Stream m a -> D.Stream m (MutArray a)
1634−-- pinnedChunksOf n = D.foldMany (pinnedCreateOf n)
1373+-- pinnedChunksOf n = D.foldMany (pinnedWriteN n)
16351374pinnedChunksOf = chunksOfAs Pinned
16361375
1637−-- | When we are buffering a stream of unknown size into an array we do not
1638−-- know how much space to pre-allocate. So we start with the min size and emit
1639−-- the array then keep on doubling the size every time. Thus we do not need to
1640−-- guess the optimum chunk size.
1641−--
1642−-- We can incorporate this in chunksOfAs if the additional size parameter does
1643−-- not impact perf.
1644−--
1645−{-# INLINE _chunksOfRange #-}
1646−_chunksOfRange :: -- (MonadIO m, Unbox a) =>
1647− PinnedState -> Int -> Int -> D.Stream m a -> D.Stream m (MutArray a)
1648−_chunksOfRange _ps _low _hi = undefined
1649−
1376+-- XXX This should take a PinnedState
16501377-- XXX buffer to a list instead?
16511378-- | Buffer the stream into arrays in memory.
16521379{-# INLINE arrayStreamKFromStreamDAs #-}
@@ -1664,10 +1391,16 @@ data FlattenState s contents a =
16641391 OuterLoop s
16651392 | InnerLoop s contents !Int !Int
16661393
1667−{-# INLINE_NORMAL concatWith #-}
1668−concatWith :: forall m a. (Monad m, Unbox a)
1669− => (forall b. IO b -> m b) -> D.Stream m (MutArray a) -> D.Stream m a
1670−concatWith liftio (D.Stream step state) = D.Stream step' (OuterLoop state)
1394+-- | Use the "reader" unfold instead.
1395+--
1396+-- @flattenArrays = unfoldMany reader@
1397+--
1398+-- We can try this if there are any fusion issues in the unfold.
1399+--
1400+{-# INLINE_NORMAL flattenArrays #-}
1401+flattenArrays :: forall m a. (MonadIO m, Unbox a)
1402+ => D.Stream m (MutArray a) -> D.Stream m a
1403+flattenArrays (D.Stream step state) = D.Stream step' (OuterLoop state)
16711404
16721405 where
16731406
@@ -1684,30 +1417,19 @@ concatWith liftio (D.Stream step state) = D.Stream step' (OuterLoop state)
16841417 return $ D.Skip $ OuterLoop st
16851418
16861419 step' _ (InnerLoop st contents p end) = do
1687− x <- liftio $ peekAt p contents
1420+ x <- liftIO $ peekByteIndex p contents
16881421 return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
16891422
1690−-- | Use the "reader" unfold instead.
1423+-- | Use the "readerRev" unfold instead.
16911424--
1692−-- @concat = unfoldMany reader@
1425+-- @flattenArrays = unfoldMany readerRev@
16931426--
16941427-- We can try this if there are any fusion issues in the unfold.
16951428--
1696−{-# INLINE_NORMAL concat #-}
1697−concat :: forall m a. (MonadIO m, Unbox a)
1698− => D.Stream m (MutArray a) -> D.Stream m a
1699−concat = concatWith liftIO
1700−
1701−{-# DEPRECATED flattenArrays "Please use \"unfoldMany reader\" instead." #-}
1702−{-# INLINE flattenArrays #-}
1703−flattenArrays :: forall m a. (MonadIO m, Unbox a)
1429+{-# INLINE_NORMAL flattenArraysRev #-}
1430+flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
17041431 => D.Stream m (MutArray a) -> D.Stream m a
1705−flattenArrays = concat
1706−
1707−{-# INLINE_NORMAL concatRevWith #-}
1708−concatRevWith :: forall m a. (Monad m, Unbox a)
1709− => (forall b. IO b -> m b) -> D.Stream m (MutArray a) -> D.Stream m a
1710−concatRevWith liftio (D.Stream step state) = D.Stream step' (OuterLoop state)
1432+flattenArraysRev (D.Stream step state) = D.Stream step' (OuterLoop state)
17111433
17121434 where
17131435
@@ -1725,33 +1447,16 @@ concatRevWith liftio (D.Stream step state) = D.Stream step' (OuterLoop state)
17251447 return $ D.Skip $ OuterLoop st
17261448
17271449 step' _ (InnerLoop st contents p start) = do
1728− x <- liftio $ peekAt p contents
1450+ x <- liftIO $ peekByteIndex p contents
17291451 let cur = INDEX_PREV(p,a)
17301452 return $ D.Yield x (InnerLoop st contents cur start)
17311453
1732−-- | Use the "readerRev" unfold instead.
1733−--
1734−-- @concat = unfoldMany readerRev@
1735−--
1736−-- We can try this if there are any fusion issues in the unfold.
1737−--
1738−{-# INLINE_NORMAL concatRev #-}
1739−concatRev :: forall m a. (MonadIO m, Unbox a)
1740− => D.Stream m (MutArray a) -> D.Stream m a
1741−concatRev = concatRevWith liftIO
1742−
1743−{-# DEPRECATED flattenArraysRev "Please use \"unfoldMany readerRev\" instead." #-}
1744−{-# INLINE flattenArraysRev #-}
1745−flattenArraysRev :: forall m a. (MonadIO m, Unbox a)
1746− => D.Stream m (MutArray a) -> D.Stream m a
1747−flattenArraysRev = concatRev
1748−
17491454-------------------------------------------------------------------------------
17501455-- Unfolds
17511456-------------------------------------------------------------------------------
17521457
17531458data ArrayUnsafe a = ArrayUnsafe
1754− {-# UNPACK #-} !MutByteArray -- contents
1459+ {-# UNPACK #-} !MutableByteArray -- contents
17551460 {-# UNPACK #-} !Int -- index 1
17561461 {-# UNPACK #-} !Int -- index 2
17571462
@@ -1781,7 +1486,7 @@ producerWith liftio = Producer step (return . toArrayUnsafe) extract
17811486 -- few actions for correctness and execution order sanity. We want
17821487 -- the peek to occur right here and not lazily at some later point
17831488 -- because we want the peek to be ordered with respect to the touch.
1784− !x <- liftio $ peekAt cur contents
1489+ !x <- liftio $ peekByteIndex cur contents
17851490 return $ D.Yield x (ArrayUnsafe contents (INDEX_NEXT(cur,a)) end)
17861491
17871492 extract = return . fromArrayUnsafe
@@ -1812,7 +1517,7 @@ readerRevWith liftio = Unfold step inject
18121517 {-# INLINE_LATE step #-}
18131518 step (ArrayUnsafe _ start p) | p < start = return D.Stop
18141519 step (ArrayUnsafe contents start p) = do
1815− !x <- liftio $ peekAt p contents
1520+ !x <- liftio $ peekByteIndex p contents
18161521 return $ D.Yield x (ArrayUnsafe contents start (INDEX_PREV(p,a)))
18171522
18181523-- | Unfold an array into a stream in reverse order.
@@ -1843,7 +1548,7 @@ toListFB c n MutArray{..} = go arrStart
18431548 -- evaluated/written to before we peek at them.
18441549 -- XXX
18451550 let !x = unsafeInlineIO $ do
1846− r <- peekAt arrContents p
1551+ r <- peekByteIndex arrContents p
18471552 return r
18481553 in c x (go (PTR_NEXT(p,a)))
18491554-}
@@ -1860,21 +1565,21 @@ toList MutArray{..} = liftIO $ go arrStart
18601565
18611566 go p | assert (p <= arrEnd) (p == arrEnd) = return []
18621567 go p = do
1863− x <- peekAt p arrContents
1568+ x <- peekByteIndex p arrContents
18641569 (:) x <$> go (INDEX_NEXT(p,a))
18651570
1866−{-# INLINE_NORMAL toStreamWith #-}
1867−toStreamWith ::
1571+{-# INLINE_NORMAL toStreamDWith #-}
1572+toStreamDWith ::
18681573 forall m a. (Monad m, Unbox a)
18691574 => (forall b. IO b -> m b) -> MutArray a -> D.Stream m a
1870−toStreamWith liftio MutArray{..} = D.Stream step arrStart
1575+toStreamDWith liftio MutArray{..} = D.Stream step arrStart
18711576
18721577 where
18731578
18741579 {-# INLINE_LATE step #-}
18751580 step _ p | assert (p <= arrEnd) (p == arrEnd) = return D.Stop
18761581 step _ p = liftio $ do
1877− r <- peekAt p arrContents
1582+ r <- peekByteIndex p arrContents
18781583 return $ D.Yield r (INDEX_NEXT(p,a))
18791584
18801585-- | Convert a 'MutArray' into a stream.
@@ -1883,7 +1588,7 @@ toStreamWith liftio MutArray{..} = D.Stream step arrStart
18831588--
18841589{-# INLINE_NORMAL read #-}
18851590read :: forall m a. (MonadIO m, Unbox a) => MutArray a -> D.Stream m a
1886−read = toStreamWith liftIO
1591+read = toStreamDWith liftIO
18871592
18881593{-# INLINE toStreamKWith #-}
18891594toStreamKWith ::
@@ -1895,18 +1600,18 @@ toStreamKWith liftio MutArray{..} = go arrStart
18951600
18961601 go p | assert (p <= arrEnd) (p == arrEnd) = K.nil
18971602 | otherwise =
1898− let elemM = peekAt p arrContents
1603+ let elemM = peekByteIndex p arrContents
18991604 in liftio elemM `K.consM` go (INDEX_NEXT(p,a))
19001605
19011606{-# INLINE toStreamK #-}
19021607toStreamK :: forall m a. (MonadIO m, Unbox a) => MutArray a -> StreamK m a
19031608toStreamK = toStreamKWith liftIO
19041609
1905−{-# INLINE_NORMAL toStreamRevWith #-}
1906−toStreamRevWith ::
1610+{-# INLINE_NORMAL toStreamDRevWith #-}
1611+toStreamDRevWith ::
19071612 forall m a. (Monad m, Unbox a)
19081613 => (forall b. IO b -> m b) -> MutArray a -> D.Stream m a
1909−toStreamRevWith liftio MutArray{..} =
1614+toStreamDRevWith liftio MutArray{..} =
19101615 let p = INDEX_PREV(arrEnd,a)
19111616 in D.Stream step p
19121617
@@ -1915,7 +1620,7 @@ toStreamRevWith liftio MutArray{..} =
19151620 {-# INLINE_LATE step #-}
19161621 step _ p | p < arrStart = return D.Stop
19171622 step _ p = liftio $ do
1918− r <- peekAt p arrContents
1623+ r <- peekByteIndex p arrContents
19191624 return $ D.Yield r (INDEX_PREV(p,a))
19201625
19211626-- | Convert a 'MutArray' into a stream in reverse order.
@@ -1924,7 +1629,7 @@ toStreamRevWith liftio MutArray{..} =
19241629--
19251630{-# INLINE_NORMAL readRev #-}
19261631readRev :: forall m a. (MonadIO m, Unbox a) => MutArray a -> D.Stream m a
1927−readRev = toStreamRevWith liftIO
1632+readRev = toStreamDRevWith liftIO
19281633
19291634{-# INLINE toStreamKRevWith #-}
19301635toStreamKRevWith ::
@@ -1938,7 +1643,7 @@ toStreamKRevWith liftio MutArray {..} =
19381643
19391644 go p | p < arrStart = K.nil
19401645 | otherwise =
1941− let elemM = peekAt p arrContents
1646+ let elemM = peekByteIndex p arrContents
19421647 in liftio elemM `K.consM` go (INDEX_PREV(p,a))
19431648
19441649{-# INLINE toStreamKRev #-}
@@ -1973,7 +1678,7 @@ foldr f z arr = D.foldr f z $ read arr
19731678
19741679-- XXX Keep the bound intact to not lose any free space? Perf impact?
19751680
1976−-- | @unsafeAppendN n arr@ appends up to @n@ input items to the supplied
1681+-- | @writeAppendNUnsafe n arr@ appends up to @n@ input items to the supplied
19771682-- array.
19781683--
19791684-- Unsafe: Do not drive the fold beyond @n@ elements, it will lead to memory
@@ -1982,12 +1687,13 @@ foldr f z arr = D.foldr f z $ read arr
19821687-- Any free space left in the array after appending @n@ elements is lost.
19831688--
19841689-- /Internal/
1985−{-# INLINE_NORMAL unsafeAppendN #-}
1986−unsafeAppendN :: forall m a. (MonadIO m, Unbox a) =>
1690+{-# INLINE_NORMAL writeAppendNUnsafe #-}
1691+writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) =>
19871692 Int
19881693 -> m (MutArray a)
19891694 -> Fold m a (MutArray a)
1990−unsafeAppendN n action = fmap fromArrayUnsafe $ FL.foldlM' step initial
1695+writeAppendNUnsafe n action =
1696+ fmap fromArrayUnsafe $ FL.foldlM' step initial
19911697
19921698 where
19931699
@@ -2000,59 +1706,38 @@ unsafeAppendN n action = fmap fromArrayUnsafe $ FL.foldlM' step initial
20001706 -- otherwise we lose that space.
20011707 arr1 <-
20021708 if free < needed
2003− then noinline reallocWith "unsafeAppendN" (+ needed) needed arr
1709+ then noinline reallocWith "writeAppendNUnsafeWith" (+ needed) needed arr
20041710 else return arr
20051711 return $ toArrayUnsafe arr1
20061712
20071713 step (ArrayUnsafe contents start end) x = do
2008− liftIO $ pokeAt end contents x
1714+ liftIO $ pokeByteIndex end contents x
20091715 return $ ArrayUnsafe contents start (INDEX_NEXT(end,a))
20101716
2011−{-# DEPRECATED writeAppendNUnsafe "Please use unsafeAppendN instead." #-}
2012−{-# INLINE writeAppendNUnsafe #-}
2013−writeAppendNUnsafe :: forall m a. (MonadIO m, Unbox a) =>
2014− Int
2015− -> m (MutArray a)
2016− -> Fold m a (MutArray a)
2017−writeAppendNUnsafe = unsafeAppendN
2018−
20191717-- | Append @n@ elements to an existing array. Any free space left in the array
20201718-- after appending @n@ elements is lost.
20211719--
2022−-- >>> appendN n initial = Fold.take n (MutArray.unsafeAppendN n initial)
1720+-- >>> writeAppendN n initial = Fold.take n (MutArray.writeAppendNUnsafe n initial)
20231721--
2024−{-# INLINE_NORMAL appendN #-}
2025−appendN :: forall m a. (MonadIO m, Unbox a) =>
2026− Int -> m (MutArray a) -> Fold m a (MutArray a)
2027−appendN n initial = FL.take n (unsafeAppendN n initial)
2028−
2029−-- XXX Deprecate in major
2030−-- {-# DEPRECATED writeAppendN "Please use appendN instead." #-}
2031−{-# INLINE writeAppendN #-}
1722+{-# INLINE_NORMAL writeAppendN #-}
20321723writeAppendN :: forall m a. (MonadIO m, Unbox a) =>
20331724 Int -> m (MutArray a) -> Fold m a (MutArray a)
2034−writeAppendN = appendN
1725+writeAppendN n initial = FL.take n (writeAppendNUnsafe n initial)
20351726
2036−-- | @appendWith realloc action@ mutates the array generated by @action@ to
1727+-- | @writeAppendWith realloc action@ mutates the array generated by @action@ to
20371728-- append the input stream. If there is no reserved space available in the
20381729-- array it is reallocated to a size in bytes determined by @realloc oldSize@,
20391730-- where @oldSize@ is the current size of the array in bytes.
20401731--
20411732-- Note that the returned array may be a mutated version of original array.
20421733--
2043−-- >>> appendWith sizer = Fold.foldlM' (MutArray.snocWith sizer)
1734+-- >>> writeAppendWith sizer = Fold.foldlM' (MutArray.snocWith sizer)
20441735--
20451736-- /Pre-release/
2046−{-# INLINE appendWith #-}
2047−appendWith :: forall m a. (MonadIO m, Unbox a) =>
2048− (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
2049−appendWith sizer = FL.foldlM' (snocWith sizer)
2050−
2051−{-# DEPRECATED writeAppendWith "Please use appendWith instead." #-}
20521737{-# INLINE writeAppendWith #-}
20531738writeAppendWith :: forall m a. (MonadIO m, Unbox a) =>
20541739 (Int -> Int) -> m (MutArray a) -> Fold m a (MutArray a)
2055−writeAppendWith = appendWith
1740+writeAppendWith sizer = FL.foldlM' (snocWith sizer)
20561741
20571742-- | @append action@ mutates the array generated by @action@ to append the
20581743-- input stream. If there is no reserved space available in the array it is
@@ -2060,101 +1745,70 @@ writeAppendWith = appendWith
20601745--
20611746-- Note that the returned array may be a mutated version of original array.
20621747--
2063−-- >>> append = MutArray.appendWith (* 2)
1748+-- >>> writeAppend = MutArray.writeAppendWith (* 2)
20641749--
2065−{-# INLINE append #-}
2066−append :: forall m a. (MonadIO m, Unbox a) =>
2067− m (MutArray a) -> Fold m a (MutArray a)
2068−append = appendWith (* 2)
2069−
2070−-- XXX Deprecate in major
2071−-- {-# DEPRECATED writeAppend "Please use append instead." #-}
20721750{-# INLINE writeAppend #-}
20731751writeAppend :: forall m a. (MonadIO m, Unbox a) =>
20741752 m (MutArray a) -> Fold m a (MutArray a)
2075−writeAppend = append
1753+writeAppend = writeAppendWith (* 2)
20761754
20771755-- XXX We can carry bound as well in the state to make sure we do not lose the
20781756-- remaining capacity. Need to check perf impact.
20791757--
2080−-- | Like 'unsafeCreateOf' but takes a new array allocator @alloc size@
2081−-- function as argument.
1758+-- | Like 'writeNUnsafe' but takes a new array allocator @alloc size@ function
1759+-- as argument.
20821760--
2083−-- >>> unsafeCreateOfWith alloc n = MutArray.unsafeAppendN (alloc n) n
1761+-- >>> writeNWithUnsafe alloc n = MutArray.writeAppendNUnsafe (alloc n) n
20841762--
20851763-- /Pre-release/
2086−{-# INLINE_NORMAL unsafeCreateOfWith #-}
2087−unsafeCreateOfWith :: forall m a. (MonadIO m, Unbox a)
1764+{-# INLINE_NORMAL writeNWithUnsafe #-}
1765+writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a)
20881766 => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
2089−unsafeCreateOfWith alloc n = fromArrayUnsafe <$> FL.foldlM' step initial
1767+writeNWithUnsafe alloc n = fromArrayUnsafe <$> FL.foldlM' step initial
20901768
20911769 where
20921770
20931771 initial = toArrayUnsafe <$> alloc (max n 0)
20941772
20951773 step (ArrayUnsafe contents start end) x = do
2096− liftIO $ pokeAt end contents x
1774+ liftIO $ pokeByteIndex end contents x
20971775 return
20981776 $ ArrayUnsafe contents start (INDEX_NEXT(end,a))
20991777
2100−{-# DEPRECATED writeNWithUnsafe "Please use unsafeCreateOfWith instead." #-}
2101−{-# INLINE writeNWithUnsafe #-}
2102−writeNWithUnsafe :: forall m a. (MonadIO m, Unbox a)
2103− => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
2104−writeNWithUnsafe = unsafeCreateOfWith
2105−
21061778{-# INLINE_NORMAL writeNUnsafeAs #-}
21071779writeNUnsafeAs :: forall m a. (MonadIO m, Unbox a)
21081780 => PinnedState -> Int -> Fold m a (MutArray a)
2109−writeNUnsafeAs ps = unsafeCreateOfWith (newAs ps)
1781+writeNUnsafeAs ps = writeNWithUnsafe (newAs ps)
21101782
2111−-- | Like 'createOf' but does not check the array bounds when writing. The fold
1783+-- | Like 'writeN' but does not check the array bounds when writing. The fold
21121784-- driver must not call the step function more than 'n' times otherwise it will
21131785-- corrupt the memory and crash. This function exists mainly because any
21141786-- conditional in the step function blocks fusion causing 10x performance
21151787-- slowdown.
21161788--
2117−-- >>> unsafeCreateOf = MutArray.unsafeCreateOfWith MutArray.emptyOf
1789+-- >>> writeNUnsafe = MutArray.writeNWithUnsafe MutArray.new
21181790--
2119−{-# INLINE_NORMAL unsafeCreateOf #-}
2120−unsafeCreateOf :: forall m a. (MonadIO m, Unbox a)
2121− => Int -> Fold m a (MutArray a)
2122−unsafeCreateOf = writeNUnsafeAs Unpinned
2123−
2124−{-# DEPRECATED writeNUnsafe "Please use unsafeCreateOf instead." #-}
2125−{-# INLINE writeNUnsafe #-}
1791+{-# INLINE_NORMAL writeNUnsafe #-}
21261792writeNUnsafe :: forall m a. (MonadIO m, Unbox a)
21271793 => Int -> Fold m a (MutArray a)
2128−writeNUnsafe = unsafeCreateOf
1794+writeNUnsafe = writeNUnsafeAs Unpinned
21291795
2130−-- | Like 'unsafeCreateOf' but creates a pinned array.
2131−{-# INLINE_NORMAL unsafePinnedCreateOf #-}
2132−unsafePinnedCreateOf :: forall m a. (MonadIO m, Unbox a)
2133− => Int -> Fold m a (MutArray a)
2134−unsafePinnedCreateOf = writeNUnsafeAs Pinned
2135−
2136−{-# DEPRECATED pinnedWriteNUnsafe "Please use unsafePinnedCreateOf instead." #-}
2137−{-# INLINE pinnedWriteNUnsafe #-}
1796+-- | Like 'writeNUnsafe' but creates a pinned array.
1797+{-# INLINE_NORMAL pinnedWriteNUnsafe #-}
21381798pinnedWriteNUnsafe :: forall m a. (MonadIO m, Unbox a)
21391799 => Int -> Fold m a (MutArray a)
2140−pinnedWriteNUnsafe = unsafePinnedCreateOf
1800+pinnedWriteNUnsafe = writeNUnsafeAs Pinned
21411801
2142−-- | @createOfWith alloc n@ folds a maximum of @n@ elements into an array
1802+-- | @writeNWith alloc n@ folds a maximum of @n@ elements into an array
21431803-- allocated using the @alloc@ function.
21441804--
2145−-- >>> createOfWith alloc n = Fold.take n (MutArray.unsafeCreateOfWith alloc n)
2146−-- >>> createOfWith alloc n = MutArray.appendN (alloc n) n
1805+-- >>> writeNWith alloc n = Fold.take n (MutArray.writeNWithUnsafe alloc n)
1806+-- >>> writeNWith alloc n = MutArray.writeAppendN (alloc n) n
21471807--
2148−{-# INLINE_NORMAL createOfWith #-}
2149−createOfWith :: forall m a. (MonadIO m, Unbox a)
2150− => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
2151−createOfWith alloc n = FL.take n (unsafeCreateOfWith alloc n)
2152−
2153−-- {-# DEPRECATED writeNWith "Please use createOfWith instead." #-}
2154−{-# INLINE writeNWith #-}
1808+{-# INLINE_NORMAL writeNWith #-}
21551809writeNWith :: forall m a. (MonadIO m, Unbox a)
21561810 => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
2157−writeNWith = createOfWith
1811+writeNWith alloc n = FL.take n (writeNWithUnsafe alloc n)
21581812
21591813{-# INLINE_NORMAL writeNAs #-}
21601814writeNAs ::
@@ -2162,42 +1816,28 @@ writeNAs ::
21621816 => PinnedState
21631817 -> Int
21641818 -> Fold m a (MutArray a)
2165−writeNAs ps = createOfWith (newAs ps)
1819+writeNAs ps = writeNWith (newAs ps)
21661820
2167−-- | @createOf n@ folds a maximum of @n@ elements from the input stream to an
1821+-- | @writeN n@ folds a maximum of @n@ elements from the input stream to an
21681822-- 'MutArray'.
21691823--
2170−-- >>> createOf = MutArray.createOfWith MutArray.new
2171−-- >>> createOf n = Fold.take n (MutArray.unsafeCreateOf n)
2172−-- >>> createOf n = MutArray.appendN n (MutArray.emptyOf n)
1824+-- >>> writeN = MutArray.writeNWith MutArray.new
1825+-- >>> writeN n = Fold.take n (MutArray.writeNUnsafe n)
1826+-- >>> writeN n = MutArray.writeAppendN n (MutArray.new n)
21731827--
2174−{-# INLINE_NORMAL createOf #-}
2175−createOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
2176−createOf = writeNAs Unpinned
2177−
2178−-- XXX Deprecate in major
2179−-- {-# DEPRECATED writeN "Please use createOf instead." #-}
2180−{-# INLINE writeN #-}
1828+{-# INLINE_NORMAL writeN #-}
21811829writeN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
2182−writeN = createOf
2183−
2184−-- | Like 'createOf' but creates a pinned array.
2185−{-# INLINE_NORMAL pinnedCreateOf #-}
2186−pinnedCreateOf ::
2187− forall m a. (MonadIO m, Unbox a)
2188− => Int
2189− -> Fold m a (MutArray a)
2190−pinnedCreateOf = writeNAs Pinned
1830+writeN = writeNAs Unpinned
21911831
2192−{-# DEPRECATED pinnedWriteN "Please use pinnedCreateOf instead." #-}
2193−{-# INLINE pinnedWriteN #-}
1832+-- | Like 'writeN' but creates a pinned array.
1833+{-# INLINE_NORMAL pinnedWriteN #-}
21941834pinnedWriteN ::
21951835 forall m a. (MonadIO m, Unbox a)
21961836 => Int
21971837 -> Fold m a (MutArray a)
2198−pinnedWriteN = pinnedCreateOf
1838+pinnedWriteN = writeNAs Pinned
21991839
2200−-- | Like unsafeCreateOfWith but writes the array in reverse order.
1840+-- | Like writeNWithUnsafe but writes the array in reverse order.
22011841--
22021842-- /Internal/
22031843{-# INLINE_NORMAL writeRevNWithUnsafe #-}
@@ -2214,11 +1854,11 @@ writeRevNWithUnsafe alloc n = fromArrayUnsafe <$> FL.foldlM' step initial
22141854
22151855 step (ArrayUnsafe contents start end) x = do
22161856 let ptr = INDEX_PREV(start,a)
2217− liftIO $ pokeAt ptr contents x
1857+ liftIO $ pokeByteIndex ptr contents x
22181858 return
22191859 $ ArrayUnsafe contents ptr end
22201860
2221−-- | Like createOfWith but writes the array in reverse order.
1861+-- | Like writeNWith but writes the array in reverse order.
22221862--
22231863-- /Internal/
22241864{-# INLINE_NORMAL writeRevNWith #-}
@@ -2226,36 +1866,31 @@ writeRevNWith :: forall m a. (MonadIO m, Unbox a)
22261866 => (Int -> m (MutArray a)) -> Int -> Fold m a (MutArray a)
22271867writeRevNWith alloc n = FL.take n (writeRevNWithUnsafe alloc n)
22281868
2229−-- | Like 'createOf' but writes the array in reverse order.
1869+-- | Like writeN but writes the array in reverse order.
22301870--
22311871-- /Pre-release/
2232−{-# INLINE_NORMAL revCreateOf #-}
2233−revCreateOf :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
2234−revCreateOf = writeRevNWith new
2235−
2236−{-# DEPRECATED writeRevN "Please use revCreateOf instead." #-}
2237−{-# INLINE writeRevN #-}
1872+{-# INLINE_NORMAL writeRevN #-}
22381873writeRevN :: forall m a. (MonadIO m, Unbox a) => Int -> Fold m a (MutArray a)
2239−writeRevN = revCreateOf
1874+writeRevN = writeRevNWith new
22401875
22411876-- | @pinnedWriteNAligned align n@ folds a maximum of @n@ elements from the
22421877-- input stream to a 'MutArray' aligned to the given size.
22431878--
2244−-- >>> pinnedWriteNAligned align = MutArray.createOfWith (MutArray.pinnedNewAligned align)
2245−-- >>> pinnedWriteNAligned align n = MutArray.appendN n (MutArray.pinnedNewAligned align n)
1879+-- >>> pinnedWriteNAligned align = MutArray.writeNWith (MutArray.pinnedNewAligned align)
1880+-- >>> pinnedWriteNAligned align n = MutArray.writeAppendN n (MutArray.pinnedNewAligned align n)
22461881--
22471882-- /Pre-release/
22481883--
22491884{-# INLINE_NORMAL pinnedWriteNAligned #-}
22501885pinnedWriteNAligned :: forall m a. (MonadIO m, Unbox a)
22511886 => Int -> Int -> Fold m a (MutArray a)
2252−pinnedWriteNAligned align = createOfWith (pinnedNewAligned align)
1887+pinnedWriteNAligned align = writeNWith (pinnedNewAligned align)
22531888
22541889-- XXX Buffer to a list instead?
2255−
1890+--
22561891-- | Buffer a stream into a stream of arrays.
22571892--
2258−-- >>> buildChunks n = Fold.many (MutArray.createOf n) Fold.toStreamK
1893+-- >>> writeChunks n = Fold.many (MutArray.writeN n) Fold.toStreamK
22591894--
22601895-- Breaking an array into an array stream can be useful to consume a large
22611896-- array sequentially such that memory of the array is released incrementatlly.
@@ -2264,43 +1899,37 @@ pinnedWriteNAligned align = createOfWith (pinnedNewAligned align)
22641899--
22651900-- /Unimplemented/
22661901--
2267−{-# INLINE_NORMAL buildChunks #-}
2268−buildChunks :: (MonadIO m, Unbox a) =>
2269− Int -> Fold m a (StreamK n (MutArray a))
2270−buildChunks n = FL.many (createOf n) FL.toStreamK
2271−
2272−{-# DEPRECATED writeChunks "Please use buildChunks instead." #-}
2273−{-# INLINE writeChunks #-}
1902+{-# INLINE_NORMAL writeChunks #-}
22741903writeChunks :: (MonadIO m, Unbox a) =>
22751904 Int -> Fold m a (StreamK n (MutArray a))
2276−writeChunks = buildChunks
1905+writeChunks n = FL.many (writeN n) FL.toStreamK
22771906
22781907{-# INLINE_NORMAL writeWithAs #-}
22791908writeWithAs :: forall m a. (MonadIO m, Unbox a)
22801909 => PinnedState -> Int -> Fold m a (MutArray a)
2281−-- writeWithAs ps n = FL.rmapM rightSize $ appendWith (* 2) (newAs ps n)
1910+-- writeWithAs ps n = FL.rmapM rightSize $ writeAppendWith (* 2) (newAs ps n)
22821911writeWithAs ps elemCount =
22831912 FL.rmapM extract $ FL.foldlM' step initial
22841913
22851914 where
22861915
22871916 initial = do
2288− when (elemCount < 0) $ error "createWith: elemCount is negative"
2289− newAs ps elemCount
1917+ when (elemCount < 0) $ error "writeWith: elemCount is negative"
1918+ liftIO $ newAs ps elemCount
22901919
22911920 step arr@(MutArray _ start end bound) x
22921921 | INDEX_NEXT(end,a) > bound = do
22931922 let oldSize = end - start
22941923 newSize = max (oldSize * 2) 1
2295− arr1 <- liftIO $ reallocExplicitAs ps (SIZE_OF(a)) newSize arr
1924+ arr1 <- liftIO $ reallocExplicit (SIZE_OF(a)) newSize arr
22961925 snocUnsafe arr1 x
22971926 step arr x = snocUnsafe arr x
22981927
22991928 extract = liftIO . rightSize
23001929
2301−-- XXX Compare createWith with fromStreamD which uses an array of streams
2302−-- implementation. We can write this using buildChunks above if that is faster.
2303−-- If createWith is faster then we should use that to implement
1930+-- XXX Compare writeWith with fromStreamD which uses an array of streams
1931+-- implementation. We can write this using writeChunks above if that is faster.
1932+-- If writeWith is faster then we should use that to implement
23041933-- fromStreamD.
23051934--
23061935-- XXX The realloc based implementation needs to make one extra copy if we use
@@ -2315,56 +1944,39 @@ writeWithAs ps elemCount =
23151944-- XXX check if GHC's memory allocator is efficient enough. We can try the C
23161945-- malloc to compare against.
23171946
2318−-- | @createWith minCount@ folds the whole input to a single array. The array
1947+-- | @writeWith minCount@ folds the whole input to a single array. The array
23191948-- starts at a size big enough to hold minCount elements, the size is doubled
23201949-- every time the array needs to be grown.
23211950--
23221951-- /Caution! Do not use this on infinite streams./
23231952--
2324−-- >>> f n = MutArray.appendWith (* 2) (MutArray.emptyOf n)
2325−-- >>> createWith n = Fold.rmapM MutArray.rightSize (f n)
2326−-- >>> createWith n = Fold.rmapM MutArray.fromChunksK (MutArray.buildChunks n)
1953+-- >>> f n = MutArray.writeAppendWith (* 2) (MutArray.new n)
1954+-- >>> writeWith n = Fold.rmapM MutArray.rightSize (f n)
1955+-- >>> writeWith n = Fold.rmapM MutArray.fromArrayStreamK (MutArray.writeChunks n)
23271956--
23281957-- /Pre-release/
2329−{-# INLINE_NORMAL createWith #-}
2330−createWith :: forall m a. (MonadIO m, Unbox a)
2331− => Int -> Fold m a (MutArray a)
2332−-- createWith n = FL.rmapM rightSize $ appendWith (* 2) (emptyOf n)
2333−createWith = writeWithAs Unpinned
2334−
2335−{-# DEPRECATED writeWith "Please use createWith instead." #-}
2336−{-# INLINE writeWith #-}
1958+{-# INLINE_NORMAL writeWith #-}
23371959writeWith :: forall m a. (MonadIO m, Unbox a)
23381960 => Int -> Fold m a (MutArray a)
2339−writeWith = createWith
1961+-- writeWith n = FL.rmapM rightSize $ writeAppendWith (* 2) (new n)
1962+writeWith = writeWithAs Unpinned
23401963
23411964-- | Fold the whole input to a single array.
23421965--
2343−-- Same as 'createWith' using an initial array size of 'arrayChunkBytes' bytes
1966+-- Same as 'writeWith' using an initial array size of 'arrayChunkBytes' bytes
23441967-- rounded up to the element size.
23451968--
23461969-- /Caution! Do not use this on infinite streams./
23471970--
2348−{-# INLINE create #-}
2349−create :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
2350−create = createWith (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
2351−
2352−-- XXX Deprecate in major
2353−-- {-# DEPRECATED write "Please use create instead." #-}
23541971{-# INLINE write #-}
23551972write :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
2356−write = create
2357−
2358−-- | Like 'create' but creates a pinned array.
2359−{-# INLINE pinnedCreate #-}
2360−pinnedCreate :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
2361−pinnedCreate =
2362− writeWithAs Pinned (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
1973+write = writeWith (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
23631974
2364−{-# DEPRECATED pinnedWrite "Please use pinnedCreate instead." #-}
1975+-- | Like 'write' but creates a pinned array.
23651976{-# INLINE pinnedWrite #-}
23661977pinnedWrite :: forall m a. (MonadIO m, Unbox a) => Fold m a (MutArray a)
2367−pinnedWrite = pinnedCreate
1978+pinnedWrite =
1979+ writeWithAs Pinned (allocBytesToElemCount (undefined :: a) arrayChunkBytes)
23681980
23691981-------------------------------------------------------------------------------
23701982-- construct from streams, known size
@@ -2374,31 +1986,25 @@ pinnedWrite = pinnedCreate
23741986fromStreamDNAs :: forall m a. (MonadIO m, Unbox a)
23751987 => PinnedState -> Int -> D.Stream m a -> m (MutArray a)
23761988fromStreamDNAs ps limit str = do
2377− (arr :: MutArray a) <- newAs ps limit
1989+ (arr :: MutArray a) <- liftIO $ newAs ps limit
23781990 end <- D.foldlM' (fwrite (arrContents arr)) (return $ arrEnd arr) $ D.take limit str
23791991 return $ arr {arrEnd = end}
23801992
23811993 where
23821994
23831995 fwrite arrContents ptr x = do
2384− liftIO $ pokeAt ptr arrContents x
1996+ liftIO $ pokeByteIndex ptr arrContents x
23851997 return $ INDEX_NEXT(ptr,a)
23861998
2387−-- | Use the 'createOf' fold instead.
1999+-- | Use the 'writeN' fold instead.
23882000--
2389−-- >>> fromStreamN n = Stream.fold (MutArray.createOf n)
2001+-- >>> fromStreamDN n = Stream.fold (MutArray.writeN n)
23902002--
2391−{-# INLINE_NORMAL fromStreamN #-}
2392−fromStreamN :: forall m a. (MonadIO m, Unbox a)
2393− => Int -> D.Stream m a -> m (MutArray a)
2394−-- fromStreamDN n = D.fold (createOf n)
2395−fromStreamN = fromStreamDNAs Unpinned
2396−
2397−{-# DEPRECATED fromStreamDN "Please use fromStreamN instead." #-}
2398−{-# INLINE fromStreamDN #-}
2003+{-# INLINE_NORMAL fromStreamDN #-}
23992004fromStreamDN :: forall m a. (MonadIO m, Unbox a)
24002005 => Int -> D.Stream m a -> m (MutArray a)
2401−fromStreamDN = fromStreamN
2006+-- fromStreamDN n = D.fold (writeN n)
2007+fromStreamDN = fromStreamDNAs Unpinned
24022008
24032009-- | Create a 'MutArray' from the first N elements of a list. The array is
24042010-- allocated to size N, if the list terminates before N elements then the
@@ -2418,70 +2024,16 @@ pinnedFromListN n xs = fromStreamDNAs Pinned n $ D.fromList xs
24182024-- /Pre-release/
24192025{-# INLINE fromListRevN #-}
24202026fromListRevN :: (MonadIO m, Unbox a) => Int -> [a] -> m (MutArray a)
2421−fromListRevN n xs = D.fold (revCreateOf n) $ D.fromList xs
2422−
2423−-- | Convert a pure stream in Identity monad to a mutable array.
2424−{-# INLINABLE fromPureStreamN #-}
2425−fromPureStreamN :: (MonadIO m, Unbox a) =>
2426− Int -> Stream Identity a -> m (MutArray a)
2427−fromPureStreamN n xs =
2428− D.fold (createOf n) $ D.morphInner (return . runIdentity) xs
2027+fromListRevN n xs = D.fold (writeRevN n) $ D.fromList xs
24292028
24302029-- | Convert a pure stream in Identity monad to a mutable array.
24312030{-# INLINABLE fromPureStream #-}
24322031fromPureStream :: (MonadIO m, Unbox a) => Stream Identity a -> m (MutArray a)
24332032fromPureStream xs =
2434− D.fold create $ D.morphInner (return . runIdentity) xs
2435−
2436−{-# INLINABLE fromPtrN #-}
2437−fromPtrN :: MonadIO m => Int -> Ptr Word8 -> m (MutArray Word8)
2438−fromPtrN len addr = do
2439− -- memcpy is better than stream copy when the size is known.
2440− -- XXX We can implement a stream copy in a similar way by streaming Word64
2441− -- first and then remaining Word8.
2442− arr <- new len
2443− _ <- unsafeAsPtr arr
2444− (\ptr -> liftIO $ c_memcpy ptr addr (fromIntegral len))
2445− return (arr {arrEnd = len})
2446−
2447−{-# INLINABLE fromByteStr# #-}
2448−fromByteStr# :: MonadIO m => Addr# -> m (MutArray Word8)
2449−fromByteStr# addr = do
2450− -- It is better to count the size first and allocate exact space.
2451− -- Also, memcpy is better than stream copy when the size is known.
2452− -- C strlen compares 4 bytes at a time, so is better than the stream
2453− -- version. https://github.com/bminor/glibc/blob/master/string/strlen.c
2454− -- XXX We can possibly use a stream of Word64 to do the same.
2455− -- fromByteStr# addr = fromPureStream (D.fromByteStr# addr)
2456− len <- liftIO $ c_strlen (Ptr addr)
2457− let lenInt = fromIntegral len
2458− arr <- new lenInt
2459− _ <- unsafeAsPtr arr (\ptr -> liftIO $ c_memcpy ptr (Ptr addr) len)
2460− return (arr {arrEnd = lenInt})
2033+ liftIO $ D.fold write $ D.morphInner (return . runIdentity) xs
24612034
24622035-------------------------------------------------------------------------------
2463−-- convert a stream of arrays to a single array by reallocating and copying
2464−-------------------------------------------------------------------------------
2465−
2466−-- XXX Both of these implementations of splicing seem to perform equally well.
2467−-- We need to perform benchmarks over a range of sizes though.
2468−
2469−-- | Also see 'fromChunksK'.
2470−{-# INLINE fromChunksRealloced #-}
2471−fromChunksRealloced :: forall m a. (MonadIO m, Unbox a)
2472− => Stream m (MutArray a) -> m (MutArray a)
2473−fromChunksRealloced s = do
2474− res <- D.uncons s
2475− case res of
2476− Just (a, strm) -> do
2477− arr <- D.foldlM' spliceExp (pure a) strm
2478− -- Reallocation is exponential so there may be 50% empty space in
2479− -- worst case. One more reallocation to reclaim the space.
2480− rightSize arr
2481− Nothing -> pure nil
2482−
2483−-------------------------------------------------------------------------------
2484−-- convert a stream of arrays to a single array by buffering arrays first
2036+-- convert stream to a single array
24852037-------------------------------------------------------------------------------
24862038
24872039{-# INLINE arrayStreamKLength #-}
@@ -2491,72 +2043,34 @@ arrayStreamKLength as = K.foldl' (+) 0 (K.map length as)
24912043-- | Convert an array stream to an array. Note that this requires peak memory
24922044-- that is double the size of the array stream.
24932045--
2494−{-# INLINE fromChunkskAs #-}
2495−fromChunkskAs :: (Unbox a, MonadIO m) =>
2496− PinnedState -> StreamK m (MutArray a) -> m (MutArray a)
2497−fromChunkskAs ps as = do
2498− len <- arrayStreamKLength as
2499− arr <- newAs ps len
2500− -- XXX is StreamK fold faster or StreamD fold?
2501− K.foldlM' spliceUnsafe (pure arr) as
2502− -- fromStreamDN len $ D.unfoldMany reader $ D.fromStreamK as
2503−
2504−-- XXX Need to compare this with fromChunks and fromChunkList and keep the
2505−-- fastest or simplest one if all are equally fast.
2506−
2507−-- | Convert an array stream to an array. Note that this requires peak memory
2508−-- that is double the size of the array stream.
2509−--
2510−-- Also see 'fromChunksRealloced'.
2511−--
2512−{-# INLINE fromChunksK #-}
2513−fromChunksK :: (Unbox a, MonadIO m) =>
2514− StreamK m (MutArray a) -> m (MutArray a)
2515−fromChunksK = fromChunkskAs Unpinned
2516−
2517−{-# DEPRECATED fromArrayStreamK "Please use fromChunksK instead." #-}
25182046{-# INLINE fromArrayStreamK #-}
25192047fromArrayStreamK :: (Unbox a, MonadIO m) =>
25202048 StreamK m (MutArray a) -> m (MutArray a)
2521−fromArrayStreamK = fromChunksK
2049+fromArrayStreamK as = do
2050+ len <- arrayStreamKLength as
2051+ fromStreamDN len $ D.unfoldMany reader $ D.fromStreamK as
25222052
25232053{-# INLINE fromStreamDAs #-}
25242054fromStreamDAs ::
25252055 (MonadIO m, Unbox a) => PinnedState -> D.Stream m a -> m (MutArray a)
2526−fromStreamDAs ps m =
2527− arrayStreamKFromStreamDAs Unpinned m >>= fromChunkskAs ps
2056+fromStreamDAs ps m = arrayStreamKFromStreamDAs ps m >>= fromArrayStreamK
25282057
2529−-- | Create an 'Array' from a stream. This is useful when we want to create a
2530−-- single array from a stream of unknown size. 'createOf' is at least twice
2531−-- as efficient when the size is already known.
2532−--
2533−-- Note that if the input stream is too large memory allocation for the array
2534−-- may fail. When the stream size is not known, `chunksOf` followed by
2535−-- processing of indvidual arrays in the resulting stream should be preferred.
2536−--
2537−-- /Pre-release/
2538−{-# INLINE fromStream #-}
2539−fromStream :: (MonadIO m, Unbox a) => Stream m a -> m (MutArray a)
2540−fromStream = fromStreamDAs Unpinned
2541−
2542−-- fromStream (Stream m) = P.fold create m
25432058-- CAUTION: a very large number (millions) of arrays can degrade performance
25442059-- due to GC overhead because we need to buffer the arrays before we flatten
25452060-- all the arrays.
25462061--
2547−-- XXX Compare if this is faster or "fold create".
2062+-- XXX Compare if this is faster or "fold write".
25482063--
25492064-- | We could take the approach of doubling the memory allocation on each
25502065-- overflow. This would result in more or less the same amount of copying as in
25512066-- the chunking approach. However, if we have to shrink in the end then it may
25522067-- result in an extra copy of the entire data.
25532068--
2554−-- >>> fromStreamD = StreamD.fold MutArray.create
2069+-- >>> fromStreamD = StreamD.fold MutArray.write
25552070--
25562071{-# INLINE fromStreamD #-}
2557−{-# DEPRECATED fromStreamD "Please use fromStream instead." #-}
25582072fromStreamD :: (MonadIO m, Unbox a) => D.Stream m a -> m (MutArray a)
2559−fromStreamD = fromStream
2073+fromStreamD = fromStreamDAs Unpinned
25602074
25612075-- | Create a 'MutArray' from a list. The list must be of finite size.
25622076--
@@ -2581,44 +2095,22 @@ fromListRev xs = fromListRevN (Prelude.length xs) xs
25812095-- Cloning
25822096-------------------------------------------------------------------------------
25832097
2584−-- Arrays are aligned on 64-bit boundaries. The fastest way to copy an array is
2585−-- to unsafeCast it to Word64, read it, write it to Word64 array and unsafeCast
2586−-- it again. We can use SIMD read/write as well.
2587−
25882098{-# INLINE cloneAs #-}
2589−cloneAs ::
2590− ( MonadIO m
2591−#ifdef DEVBUILD
2592− , Unbox a
2593−#endif
2594− )
2595− => PinnedState -> MutArray a -> m (MutArray a)
2099+cloneAs :: MonadIO m => PinnedState -> MutArray a -> m (MutArray a)
25962100cloneAs ps src =
2597− do
2101+ liftIO $ do
25982102 let startSrc = arrStart src
25992103 srcLen = arrEnd src - startSrc
2600− newArrContents <-
2601− Unboxed.cloneSliceUnsafeAs ps startSrc srcLen (arrContents src)
2104+ newArrContents <- Unboxed.newBytesAs ps srcLen
2105+ putSliceUnsafe (arrContents src) startSrc newArrContents 0 srcLen
26022106 return $ MutArray newArrContents 0 srcLen srcLen
26032107
26042108{-# INLINE clone #-}
2605−clone ::
2606− ( MonadIO m
2607−#ifdef DEVBUILD
2608− , Unbox a
2609−#endif
2610− )
2611− => MutArray a -> m (MutArray a)
2109+clone :: MonadIO m => MutArray a -> m (MutArray a)
26122110clone = cloneAs Unpinned
26132111
26142112{-# INLINE pinnedClone #-}
2615−pinnedClone ::
2616− ( MonadIO m
2617−#ifdef DEVBUILD
2618− , Unbox a
2619−#endif
2620− )
2621− => MutArray a -> m (MutArray a)
2113+pinnedClone :: MonadIO m => MutArray a -> m (MutArray a)
26222114pinnedClone = cloneAs Pinned
26232115
26242116-------------------------------------------------------------------------------
@@ -2633,16 +2125,17 @@ spliceCopy :: forall m a. MonadIO m =>
26332125 Unbox a =>
26342126#endif
26352127 MutArray a -> MutArray a -> m (MutArray a)
2636−spliceCopy arr1 arr2 = do
2128+spliceCopy arr1 arr2 = liftIO $ do
26372129 let start1 = arrStart arr1
26382130 start2 = arrStart arr2
26392131 len1 = arrEnd arr1 - start1
26402132 len2 = arrEnd arr2 - start2
2641− let len = len1 + len2
2133+ let newLen = len1 + len2
26422134 newArrContents <-
26432135 if Unboxed.isPinned (arrContents arr1)
2644− then liftIO $ Unboxed.pinnedNew len
2645− else liftIO $ Unboxed.new len
2136+ then Unboxed.pinnedNewBytes newLen
2137+ else Unboxed.newBytes newLen
2138+ let len = len1 + len2
26462139 putSliceUnsafe (arrContents arr1) start1 newArrContents 0 len1
26472140 putSliceUnsafe (arrContents arr2) start2 newArrContents len1 len2
26482141 return $ MutArray newArrContents 0 len len
@@ -2654,7 +2147,7 @@ spliceCopy arr1 arr2 = do
26542147spliceUnsafe :: MonadIO m =>
26552148 MutArray a -> MutArray a -> m (MutArray a)
26562149spliceUnsafe dst src =
2657− do
2150+ liftIO $ do
26582151 let startSrc = arrStart src
26592152 srcLen = arrEnd src - startSrc
26602153 endDst = arrEnd dst
@@ -2676,7 +2169,7 @@ spliceWith :: forall m a. (MonadIO m, Unbox a) =>
26762169 (Int -> Int -> Int) -> MutArray a -> MutArray a -> m (MutArray a)
26772170spliceWith sizer dst@(MutArray _ start end bound) src = do
26782171{-
2679− let f = appendWith (`sizer` byteLength src) (return dst)
2172+ let f = writeAppendWith (`sizer` byteLength src) (return dst)
26802173 in D.fold f (toStreamD src)
26812174-}
26822175 assert (end <= bound) (return ())
@@ -2692,7 +2185,7 @@ spliceWith sizer dst@(MutArray _ start end bound) src = do
26922185 $ "splice: newSize is less than the total size "
26932186 ++ "of arrays being appended. Please check the "
26942187 ++ "sizer function passed."
2695− realloc newSizeInBytes dst
2188+ liftIO $ realloc newSizeInBytes dst
26962189 else return dst
26972190 spliceUnsafe dst1 src
26982191
@@ -2729,24 +2222,14 @@ spliceExp = spliceWith (\l1 l2 -> max (l1 * 2) (l1 + l2))
27292222-- Splitting
27302223-------------------------------------------------------------------------------
27312224
2732−-- | Generate a stream of array slices using a predicate. The array element
2733−-- matching the predicate is dropped.
2734−--
2735−-- /Pre-release/
2736−{-# INLINE splitOn #-}
2737−splitOn :: (MonadIO m, Unbox a) =>
2738− (a -> Bool) -> MutArray a -> Stream m (MutArray a)
2739−splitOn predicate arr =
2740− fmap (\(i, len) -> getSliceUnsafe i len arr)
2741− $ D.indexOnSuffix predicate (read arr)
2742−
27432225-- | Drops the separator byte
27442226{-# INLINE breakOn #-}
27452227breakOn :: MonadIO m
27462228 => Word8 -> MutArray Word8 -> m (MutArray Word8, Maybe (MutArray Word8))
2747−breakOn sep arr@MutArray{..} = unsafeAsPtr arr $ \p -> liftIO $ do
2229+breakOn sep arr@MutArray{..} = asPtrUnsafe arr $ \p -> liftIO $ do
2230+ -- XXX Instead of using asPtrUnsafe (pinning memory) we can pass unlifted
2231+ -- Addr# to memchr and it should be safe (from ghc 8.4).
27482232 -- XXX We do not need memchr here, we can use a Haskell equivalent.
2749− -- Need efficient stream based primitives that work on Word64.
27502233 loc <- c_memchr p sep (fromIntegral $ byteLength arr)
27512234 let sepIndex = loc `minusPtr` p
27522235 return $
@@ -2767,41 +2250,32 @@ breakOn sep arr@MutArray{..} = unsafeAsPtr arr $ \p -> liftIO $ do
27672250 }
27682251 )
27692252
2770−-- | Like 'splitAt' but does not check whether the index is valid.
2771−--
2772−{-# INLINE unsafeSplitAt #-}
2773−unsafeSplitAt :: forall a. Unbox a =>
2774− Int -> MutArray a -> (MutArray a, MutArray a)
2775−unsafeSplitAt i MutArray{..} =
2776− let off = i * SIZE_OF(a)
2777− p = arrStart + off
2778− in ( MutArray
2779− { arrContents = arrContents
2780− , arrStart = arrStart
2781− , arrEnd = p
2782− , arrBound = p
2783− }
2784− , MutArray
2785− { arrContents = arrContents
2786− , arrStart = p
2787− , arrEnd = arrEnd
2788− , arrBound = arrBound
2789− }
2790− )
2791−
27922253-- | Create two slices of an array without copying the original array. The
27932254-- specified index @i@ is the first index of the second slice.
27942255--
2795−{-# INLINE splitAt #-}
27962256splitAt :: forall a. Unbox a => Int -> MutArray a -> (MutArray a, MutArray a)
2797−splitAt i arr =
2257+splitAt i arr@MutArray{..} =
27982258 let maxIndex = length arr - 1
27992259 in if i < 0
28002260 then error "sliceAt: negative array index"
28012261 else if i > maxIndex
28022262 then error $ "sliceAt: specified array index " ++ show i
28032263 ++ " is beyond the maximum index " ++ show maxIndex
2804− else unsafeSplitAt i arr
2264+ else let off = i * SIZE_OF(a)
2265+ p = arrStart + off
2266+ in ( MutArray
2267+ { arrContents = arrContents
2268+ , arrStart = arrStart
2269+ , arrEnd = p
2270+ , arrBound = p
2271+ }
2272+ , MutArray
2273+ { arrContents = arrContents
2274+ , arrStart = p
2275+ , arrEnd = arrEnd
2276+ , arrBound = arrBound
2277+ }
2278+ )
28052279
28062280-------------------------------------------------------------------------------
28072281-- Casting
@@ -2851,16 +2325,13 @@ cast arr =
28512325-- guarantees that unsafe calls will be performed in the calling thread. Making
28522326-- it safe to pass heap-allocated objects to unsafe functions.
28532327
2854−-- XXX Should we just name it asPtr, the unsafety is implicit for any pointer
2855−-- operations. And we are safe from Haskell perspective because we will be
2856−-- pinning the memory.
2857−--
2858−-- XXX we cannot pass the length of the ptr here as in some cases it may not be
2859−-- available e.g. a null terminated C string. However, we can create another
2860−-- flavor of the API e.g. asPtrN.
2328+-- Unsafe because of direct pointer operations. The user must ensure that they
2329+-- are writing within the legal bounds of the array. Should we just name it
2330+-- asPtr, the unsafety is implicit for any pointer operations. And we are safe
2331+-- from Haskell perspective because we will be pinning the memory.
28612332
2862−-- | Use a @MutArray a@ as @Ptr a@. This is useful when we want to pass an
2863−-- array as a pointer to some operating system call or to a "safe" FFI call.
2333+-- | Use an @MutArray a@ as @Ptr a@. This is useful when we want to pass an array
2334+-- as a pointer to some operating system call or to a "safe" FFI call.
28642335--
28652336-- If the array is not pinned it is copied to pinned memory before passing it
28662337-- to the monadic action.
@@ -2870,382 +2341,56 @@ cast arr =
28702341-- opportunistically before this operation occurs, to avoid the cost of a copy
28712342-- if possible.
28722343--
2873−-- /Unsafe/ because of direct pointer operations. The user must ensure that
2874−-- they are writing within the legal bounds of the array.
2344+-- /Unsafe/
28752345--
28762346-- /Pre-release/
28772347--
2878−{-# INLINE unsafePinnedAsPtr #-}
2879−unsafePinnedAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
2880−unsafePinnedAsPtr arr f =
2881− Unboxed.unsafePinnedAsPtr
2882− (arrContents arr) (\ptr -> f (ptr `plusPtr` arrStart arr))
2883−
2884−{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
2885−{-# INLINE asPtrUnsafe #-}
28862348asPtrUnsafe :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
2887−asPtrUnsafe = unsafePinnedAsPtr
2349+asPtrUnsafe arr f = do
2350+ contents <- liftIO $ Unboxed.pin $ arrContents arr
2351+ let !ptr = Ptr (byteArrayContents#
2352+ (unsafeCoerce# (getMutableByteArray# contents)))
28882353
2889−{-# INLINE unsafeAsPtr #-}
2890−unsafeAsPtr :: MonadIO m => MutArray a -> (Ptr a -> m b) -> m b
2891−unsafeAsPtr arr f =
2892− Unboxed.unsafeAsPtr
2893− (arrContents arr) (\ptr -> f (ptr `plusPtr` arrStart arr))
2354+ r <- f (ptr `plusPtr` arrStart arr)
2355+ liftIO $ touch contents
2356+ return r
28942357
28952358-------------------------------------------------------------------------------
28962359-- Equality
28972360-------------------------------------------------------------------------------
28982361
2899−-- | Byte compare two arrays. Compare the length of the arrays. If the length
2900−-- is equal, compare the lexicographical ordering of two underlying byte arrays
2901−-- otherwise return the result of length comparison.
2902−--
2903−-- /Unsafe/: Note that the 'Unbox' instance of sum types with constructors of
2904−-- different sizes may leave some memory uninitialized which can make byte
2905−-- comparison unreliable.
2362+-- | Compare the length of the arrays. If the length is equal, compare the
2363+-- lexicographical ordering of two underlying byte arrays otherwise return the
2364+-- result of length comparison.
29062365--
29072366-- /Pre-release/
2908−{-# INLINE byteCmp #-}
2909−byteCmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
2910−byteCmp arr1 arr2 = do
2911− let marr1 = getMutableByteArray# (arrContents arr1)
2912− marr2 = getMutableByteArray# (arrContents arr2)
2913− !(I# st1#) = arrStart arr1
2914− !(I# st2#) = arrStart arr2
2915− !(I# len#) = byteLength arr1
2916− case compare (byteLength arr1) (byteLength arr2) of
2917− EQ -> do
2918− r <- liftIO $ IO $ \s# ->
2919− let res =
2920− I#
2921− (compareByteArrays#
2922− (unsafeCoerce# marr1)
2923− st1#
2924− (unsafeCoerce# marr2)
2925− st2#
2926− len#)
2927− in (# s#, res #)
2928− return $ compare r 0
2929− x -> return x
2930−
29312367{-# INLINE cmp #-}
2932−{-# DEPRECATED cmp "Please use byteCmp instead." #-}
29332368cmp :: MonadIO m => MutArray a -> MutArray a -> m Ordering
2934−cmp = byteCmp
2935−
2936−-- | Byte equality of two arrays.
2937−--
2938−-- >>> byteEq arr1 arr2 = (==) EQ $ MArray.byteCmp arr1 arr2
2939−--
2940−-- /Unsafe/: See 'byteCmp'.
2941−{-# INLINE byteEq #-}
2942−byteEq :: MonadIO m => MutArray a -> MutArray a -> m Bool
2943−byteEq arr1 arr2 = fmap (EQ ==) $ byteCmp arr1 arr2
2944−
2945−-------------------------------------------------------------------------------
2946−-- Compact
2947−-------------------------------------------------------------------------------
2948−
2949−-- Note: LE versions avoid an extra copy compared to GE. LE parser trades
2950−-- backtracking one array in lieu of avoiding a copy. However, LE and GE both
2951−-- can leave some memory unused. They can split the last array to fit it
2952−-- exactly in the space.
2953−
2954−{-# INLINE_NORMAL pCompactLeAs #-}
2955−pCompactLeAs ::
2956− forall m a. (MonadIO m, Unbox a)
2957− => PinnedState -> Int -> Parser (MutArray a) m (MutArray a)
2958−pCompactLeAs ps maxElems = Parser step initial extract
2959−
2960− where
2961−
2962− maxBytes = maxElems * SIZE_OF(a)
2963−
2964− functionName = "Streamly.Internal.Data.MutArray.pCompactLE"
2965−
2966− initial =
2967− return
2968− $ if maxElems <= 0
2969− then error
2970− $ functionName
2971− ++ ": the size of arrays ["
2972− ++ show maxElems ++ "] must be a natural number"
2973− else Parser.IPartial Nothing
2974−
2975− step Nothing arr =
2976− return
2977− $ let len = byteLength arr
2978− in if len >= maxBytes
2979− then Parser.Done 0 arr
2980− else Parser.Partial 0 (Just arr)
2981− -- XXX Split the last array to use the space more compactly.
2982− step (Just buf) arr =
2983− let len = byteLength buf + byteLength arr
2984− in if len > maxBytes
2985− then return $ Parser.Done 1 buf
2986− else do
2987− buf1 <-
2988− if byteCapacity buf < maxBytes
2989− then liftIO $ reallocExplicitAs
2990− ps (SIZE_OF(a)) maxBytes buf
2991− else return buf
2992− buf2 <- spliceUnsafe buf1 arr
2993− return $ Parser.Partial 0 (Just buf2)
2994−
2995− extract Nothing = return $ Parser.Done 0 nil
2996− extract (Just buf) = return $ Parser.Done 0 buf
2997−
2998−-- | Parser @pCompactLE maxElems@ coalesces adjacent arrays in the input stream
2999−-- only if the combined size would be less than or equal to @maxElems@
3000−-- elements. Note that it won't split an array if the original array is already
3001−-- larger than maxElems.
3002−--
3003−-- @maxElems@ must be greater than 0.
3004−--
3005−-- Generates unpinned arrays irrespective of the pinning status of input
3006−-- arrays.
3007−--
3008−-- /Internal/
3009−{-# INLINE pCompactLE #-}
3010−pCompactLE ::
3011− forall m a. (MonadIO m, Unbox a)
3012− => Int -> Parser (MutArray a) m (MutArray a)
3013−pCompactLE = pCompactLeAs Unpinned
3014−
3015−-- | Pinned version of 'pCompactLE'.
3016−{-# INLINE pPinnedCompactLE #-}
3017−pPinnedCompactLE ::
3018− forall m a. (MonadIO m, Unbox a)
3019− => Int -> Parser (MutArray a) m (MutArray a)
3020−pPinnedCompactLE = pCompactLeAs Pinned
3021−
3022−data SpliceState s arr
3023− = SpliceInitial s
3024− | SpliceBuffering s arr
3025− | SpliceYielding arr (SpliceState s arr)
3026− | SpliceFinish
3027−
3028−-- This mutates the first array (if it has space) to append values from the
3029−-- second one. This would work for immutable arrays as well because an
3030−-- immutable array never has additional space so a new array is allocated
3031−-- instead of mutating it.
3032−
3033−{-# INLINE_NORMAL compactLeAs #-}
3034−compactLeAs :: forall m a. (MonadIO m, Unbox a)
3035− => PinnedState -> Int -> D.Stream m (MutArray a) -> D.Stream m (MutArray a)
3036−compactLeAs ps maxElems (D.Stream step state) =
3037− D.Stream step' (SpliceInitial state)
3038−
3039− where
3040−
3041− maxBytes = maxElems * SIZE_OF(a)
3042−
3043− functionName = "Streamly.Internal.Data.MutArray.rCompactLE"
3044−
3045− {-# INLINE_LATE step' #-}
3046− step' gst (SpliceInitial st) = do
3047− when (maxElems <= 0) $
3048− -- XXX we can pass the module string from the higher level API
3049− error $ functionName ++ ": the size of arrays [" ++ show maxElems
3050− ++ "] must be a natural number"
3051− r <- step gst st
3052− case r of
3053− D.Yield arr s -> return $
3054− let len = byteLength arr
3055− in if len >= maxBytes
3056− then D.Skip (SpliceYielding arr (SpliceInitial s))
3057− else D.Skip (SpliceBuffering s arr)
3058− D.Skip s -> return $ D.Skip (SpliceInitial s)
3059− D.Stop -> return D.Stop
3060−
3061− -- XXX Split the last array to use the space more compactly.
3062− step' gst (SpliceBuffering st buf) = do
3063− r <- step gst st
3064− case r of
3065− D.Yield arr s -> do
3066− let len = byteLength buf + byteLength arr
3067− if len > maxBytes
3068− then return $
3069− D.Skip (SpliceYielding buf (SpliceBuffering s arr))
3070− else do
3071− buf1 <- if byteCapacity buf < maxBytes
3072− then liftIO $ reallocExplicitAs
3073− ps (SIZE_OF(a)) maxBytes buf
3074− else return buf
3075− buf2 <- spliceUnsafe buf1 arr
3076− return $ D.Skip (SpliceBuffering s buf2)
3077− D.Skip s -> return $ D.Skip (SpliceBuffering s buf)
3078− D.Stop -> return $ D.Skip (SpliceYielding buf SpliceFinish)
3079−
3080− step' _ SpliceFinish = return D.Stop
3081−
3082− step' _ (SpliceYielding arr next) = return $ D.Yield arr next
3083−
3084−
3085−{-# INLINE_NORMAL fCompactGeAs #-}
3086−fCompactGeAs ::
3087− forall m a. (MonadIO m, Unbox a)
3088− => PinnedState -> Int -> FL.Fold m (MutArray a) (MutArray a)
3089−fCompactGeAs ps minElems = Fold step initial extract extract
3090−
3091− where
3092−
3093− minBytes = minElems * SIZE_OF(a)
3094−
3095− functionName = "Streamly.Internal.Data.MutArray.fCompactGE"
3096−
3097− initial =
3098− return
3099− $ if minElems < 0
3100− then error
3101− $ functionName
3102− ++ ": the size of arrays ["
3103− ++ show minElems ++ "] must be a natural number"
3104− else FL.Partial Nothing
3105−
3106− step Nothing arr =
3107− return
3108− $ let len = byteLength arr
3109− in if len >= minBytes
3110− then FL.Done arr
3111− else FL.Partial (Just arr)
3112− -- XXX Buffer arrays as a list to avoid copy and reallocations
3113− step (Just buf) arr = do
3114− let len = byteLength buf + byteLength arr
3115− buf1 <-
3116− if byteCapacity buf < len
3117− then liftIO $ reallocExplicitAs
3118− ps (SIZE_OF(a)) (max minBytes len) buf
3119− else return buf
3120− buf2 <- spliceUnsafe buf1 arr
3121− if len >= minBytes
3122− then return $ FL.Done buf2
3123− else return $ FL.Partial (Just buf2)
3124−
3125− extract Nothing = return nil
3126− extract (Just buf) = return buf
3127−
3128−-- | Fold @fCompactGE minElems@ coalesces adjacent arrays in the input stream
3129−-- until the size becomes greater than or equal to @minElems@.
3130−--
3131−-- Generates unpinned arrays irrespective of the pinning status of input
3132−-- arrays.
3133−{-# INLINE fCompactGE #-}
3134−fCompactGE ::
3135− forall m a. (MonadIO m, Unbox a)
3136− => Int -> FL.Fold m (MutArray a) (MutArray a)
3137−fCompactGE = fCompactGeAs Unpinned
3138−
3139−-- | Pinned version of 'fCompactGE'.
3140−{-# INLINE fPinnedCompactGE #-}
3141−fPinnedCompactGE ::
3142− forall m a. (MonadIO m, Unbox a)
3143− => Int -> FL.Fold m (MutArray a) (MutArray a)
3144−fPinnedCompactGE = fCompactGeAs Pinned
3145−
3146−{-# INLINE_NORMAL lCompactGeAs #-}
3147−lCompactGeAs :: forall m a. (MonadIO m, Unbox a)
3148− => PinnedState -> Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
3149−-- The fold version turns out to be a little bit slower.
3150−-- lCompactGeAs ps n = FL.many (fCompactGeAs ps n)
3151−lCompactGeAs ps minElems (Fold step1 initial1 _ final1) =
3152− Fold step initial extract final
3153−
3154− where
3155−
3156− minBytes = minElems * SIZE_OF(a)
3157−
3158− functionName = "Streamly.Internal.Data.MutArray.lCompactGE"
3159−
3160− initial = do
3161− when (minElems <= 0) $
3162− -- XXX we can pass the module string from the higher level API
3163− error $ functionName ++ ": the size of arrays ["
3164− ++ show minElems ++ "] must be a natural number"
3165−
3166− r <- initial1
3167− return $ first (Tuple' Nothing) r
3168−
3169− {-# INLINE runInner #-}
3170− runInner len acc buf =
3171− if len >= minBytes
3172− then do
3173− r <- step1 acc buf
3174− case r of
3175− FL.Done _ -> return $ FL.Done ()
3176− FL.Partial s -> do
3177− _ <- final1 s
3178− res <- initial1
3179− return $ first (Tuple' Nothing) res
3180− else return $ FL.Partial $ Tuple' (Just buf) acc
3181−
3182− step (Tuple' Nothing r1) arr =
3183− runInner (byteLength arr) r1 arr
3184−
3185− -- XXX Buffer arrays as a list to avoid copy and reallocations
3186− step (Tuple' (Just buf) r1) arr = do
3187− let len = byteLength buf + byteLength arr
3188− buf1 <- if byteCapacity buf < len
3189− then liftIO $ reallocExplicitAs
3190− ps (SIZE_OF(a)) (max minBytes len) buf
3191− else return buf
3192− buf2 <- spliceUnsafe buf1 arr
3193− runInner len r1 buf2
3194−
3195− -- XXX Several folds do extract >=> final, therefore, we need to make final
3196− -- return "m b" rather than using extract post it if we want extract to be
3197− -- partial.
3198− --
3199− -- extract forces the pending buffer to be sent to the fold which is not
3200− -- what we want.
3201− extract _ = error "lCompactGE: not designed for scanning"
3202−
3203− final (Tuple' Nothing r1) = final1 r1
3204− final (Tuple' (Just buf) r1) = do
3205− r <- step1 r1 buf
3206− case r of
3207− FL.Partial rr -> final1 rr
3208− FL.Done _ -> return ()
3209−
3210−-- | Like 'compactGE' but for transforming folds instead of stream.
3211−--
3212−-- >>> lCompactGE n = Fold.many (MutArray.fCompactGE n)
3213−--
3214−-- Generates unpinned arrays irrespective of the pinning status of input
3215−-- arrays.
3216−{-# INLINE lCompactGE #-}
3217−lCompactGE :: forall m a. (MonadIO m, Unbox a)
3218− => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
3219−lCompactGE = lCompactGeAs Unpinned
3220−
3221−-- | Pinned version of 'lCompactGE'.
3222−{-# INLINE lPinnedCompactGE #-}
3223−lPinnedCompactGE :: forall m a. (MonadIO m, Unbox a)
3224− => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
3225−lPinnedCompactGE = lCompactGeAs Pinned
3226−
3227−-- | @compactGE n stream@ coalesces adjacent arrays in the @stream@ until
3228−-- the size becomes greater than or equal to @n@.
3229−--
3230−-- >>> compactGE n = Stream.foldMany (MutArray.fCompactGE n)
3231−--
3232−{-# INLINE compactGE #-}
3233−compactGE ::
3234− (MonadIO m, Unbox a)
3235− => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
3236−compactGE n = D.foldMany (fCompactGE n)
3237−
3238−-- | 'compactEQ n' coalesces adajacent arrays in the input stream to
3239−-- arrays of exact size @n@.
3240−--
3241−-- /Unimplemented/
3242−{-# INLINE compactEQ #-}
3243−compactEQ :: -- (MonadIO m, Unbox a) =>
3244− Int -> Stream m (MutArray a) -> Stream m (MutArray a)
3245−compactEQ _n = undefined -- D.parseManyD (pCompactEQ n)
2369+cmp arr1 arr2 =
2370+ liftIO
2371+ $ do
2372+ let marr1 = getMutableByteArray# (arrContents arr1)
2373+ marr2 = getMutableByteArray# (arrContents arr2)
2374+ !(I# st1#) = arrStart arr1
2375+ !(I# st2#) = arrStart arr2
2376+ !(I# len#) = byteLength arr1
2377+ case compare (byteLength arr1) (byteLength arr2) of
2378+ EQ -> do
2379+ r <- liftIO $ IO $ \s# ->
2380+ let res =
2381+ I#
2382+ (compareByteArrays#
2383+ (unsafeCoerce# marr1)
2384+ st1#
2385+ (unsafeCoerce# marr2)
2386+ st2#
2387+ len#)
2388+ in (# s#, res #)
2389+ return $ compare r 0
2390+ x -> return x
32462391
32472392-------------------------------------------------------------------------------
3248−-- In-place mutation algorithms
2393+-- NFData
32492394-------------------------------------------------------------------------------
32502395
32512396-- | Strip elements which match with predicate from both ends.
@@ -3274,7 +2419,7 @@ strip eq arr@MutArray{..} = liftIO $ do
32742419 getStart cur = do
32752420 if cur < arrEnd
32762421 then do
3277− r <- peekAt cur arrContents
2422+ r <- peekByteIndex cur arrContents
32782423 if eq r
32792424 then getStart (INDEX_NEXT(cur,a))
32802425 else return cur
@@ -3284,7 +2429,7 @@ strip eq arr@MutArray{..} = liftIO $ do
32842429 if cur > low
32852430 then do
32862431 let prev = INDEX_PREV(cur,a)
3287− r <- peekAt prev arrContents
2432+ r <- peekByteIndex prev arrContents
32882433 if eq r
32892434 then getLast prev low
32902435 else return cur
core/src/Streamly/Internal/Data/MutByteArray.hsdeleted−251
@@ -1,251 +0,0 @@
1−{-# LANGUAGE TemplateHaskell #-}
2−
3−-- This is required as all the instances in this module are orphan instances.
4−{-# OPTIONS_GHC -fno-warn-orphans #-}
5−
6−-- |
7−-- Module : Streamly.Internal.Data.MutByteArray
8−-- Copyright : (c) 2023 Composewell Technologies
9−-- License : BSD3-3-Clause
10−-- Maintainer : streamly@composewell.com
11−-- Portability : GHC
12−--
13−
14−module Streamly.Internal.Data.MutByteArray
15− (
16− -- * MutByteArray
17− module Streamly.Internal.Data.MutByteArray.Type
18− -- * Unbox
19− , module Streamly.Internal.Data.Unbox
20− , module Streamly.Internal.Data.Unbox.TH
21− -- * Serialize
22− , module Streamly.Internal.Data.Serialize.Type
23− -- * Serialize TH
24− , module Streamly.Internal.Data.Serialize.TH
25− ) where
26−
27−--------------------------------------------------------------------------------
28−-- Imports
29−--------------------------------------------------------------------------------
30−
31−import Data.Proxy (Proxy(..))
32−import Streamly.Internal.Data.Array (Array(..))
33−import GHC.Exts (Int(..), sizeofByteArray#, unsafeCoerce#)
34−import GHC.Word (Word8)
35−
36−#if __GLASGOW_HASKELL__ >= 900
37−import GHC.Num.Integer (Integer(..))
38−#else
39−import GHC.Integer.GMP.Internals (Integer(..), BigNat(..))
40−#endif
41−
42−import Streamly.Internal.Data.MutByteArray.Type
43−import Streamly.Internal.Data.Serialize.TH
44−import Streamly.Internal.Data.Serialize.Type
45−import Streamly.Internal.Data.Unbox
46−import Streamly.Internal.Data.Unbox.TH
47−
48−--------------------------------------------------------------------------------
49−-- Common instances
50−--------------------------------------------------------------------------------
51−
52−-- Note
53−-- ====
54−--
55−-- Even a non-functional change such as changing the order of constructors will
56−-- change the instance derivation.
57−--
58−-- This will not pose a problem if both, encode, and decode are done by the same
59−-- version of the application. There *might* be a problem if version that
60−-- encodes differs from the version that decodes.
61−--
62−-- We need to add some compatibility tests using different versions of
63−-- dependencies.
64−--
65−-- Although such chages for the most basic types won't happen we need to detect
66−-- if it ever happens.
67−--
68−-- Should we worry about these kind of changes and this kind of compatibility?
69−-- This is a problem for all types of derivations that depend on the order of
70−-- constructors, for example, Enum.
71−
72−-- Note on Windows build
73−-- =====================
74−--
75−-- On Windows, having template haskell splices here fail the build with the
76−-- following error:
77−--
78−-- @
79−-- addLibrarySearchPath: C:\... (Win32 error 3): The system cannot find the path specified.
80−-- @
81−--
82−-- The error might be irrelavant but having these splices triggers it. We should
83−-- either fix the problem or avoid the use to template haskell splices in this
84−-- file.
85−--
86−-- Similar issue: https://github.com/haskell/cabal/issues/4741
87−
88−-- $(Serialize.deriveSerialize ''Maybe)
89−instance Serialize a => Serialize (Maybe a) where
90−
91− {-# INLINE addSizeTo #-}
92− addSizeTo acc x =
93− case x of
94− Nothing -> (acc + 1)
95− Just field0 -> (addSizeTo (acc + 1)) field0
96−
97− {-# INLINE deserializeAt #-}
98− deserializeAt initialOffset arr endOffset = do
99− (i0, tag) <- ((deserializeAt initialOffset) arr) endOffset
100− case tag :: Word8 of
101− 0 -> pure (i0, Nothing)
102− 1 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
103− pure (i1, Just a0)
104− _ -> error "Found invalid tag while peeking (Maybe a)"
105−
106− {-# INLINE serializeAt #-}
107− serializeAt initialOffset arr val =
108− case val of
109− Nothing -> do
110− i0 <- ((serializeAt initialOffset) arr) (0 :: Word8)
111− pure i0
112− Just field0 -> do
113− i0 <- ((serializeAt initialOffset) arr) (1 :: Word8)
114− i1 <- ((serializeAt i0) arr) field0
115− pure i1
116−
117−-- $(Serialize.deriveSerialize ''Either)
118−instance (Serialize a, Serialize b) => Serialize (Either a b) where
119−
120− {-# INLINE addSizeTo #-}
121− addSizeTo acc x =
122− case x of
123− Left field0 -> (addSizeTo (acc + 1)) field0
124− Right field0 -> (addSizeTo (acc + 1)) field0
125−
126− {-# INLINE deserializeAt #-}
127− deserializeAt initialOffset arr endOffset = do
128− (i0, tag) <- ((deserializeAt initialOffset) arr) endOffset
129− case tag :: Word8 of
130− 0 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
131− pure (i1, Left a0)
132− 1 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
133− pure (i1, Right a0)
134− _ -> error "Found invalid tag while peeking (Either a b)"
135−
136− {-# INLINE serializeAt #-}
137− serializeAt initialOffset arr val =
138− case val of
139− Left field0 -> do
140− i0 <- ((serializeAt initialOffset) arr) (0 :: Word8)
141− i1 <- ((serializeAt i0) arr) field0
142− pure i1
143− Right field0 -> do
144− i0 <- ((serializeAt initialOffset) arr) (1 :: Word8)
145− i1 <- ((serializeAt i0) arr) field0
146− pure i1
147−
148−instance Serialize (Proxy a) where
149−
150− {-# INLINE addSizeTo #-}
151− addSizeTo acc _ = (acc + 1)
152−
153− {-# INLINE deserializeAt #-}
154− deserializeAt initialOffset _ _ = pure ((initialOffset + 1), Proxy)
155−
156− {-# INLINE serializeAt #-}
157− serializeAt initialOffset _ _ = pure (initialOffset + 1)
158−
159−--------------------------------------------------------------------------------
160−-- Integer
161−--------------------------------------------------------------------------------
162−
163−data LiftedInteger
164− = LIS Int
165− | LIP (Array Word)
166− | LIN (Array Word)
167−
168−-- $(Serialize.deriveSerialize ''LiftedInteger)
169−instance Serialize LiftedInteger where
170−
171− {-# INLINE addSizeTo #-}
172− addSizeTo acc x =
173− case x of
174− LIS field0 -> (addSizeTo (acc + 1)) field0
175− LIP field0 -> (addSizeTo (acc + 1)) field0
176− LIN field0 -> (addSizeTo (acc + 1)) field0
177−
178− {-# INLINE deserializeAt #-}
179− deserializeAt initialOffset arr endOffset = do
180− (i0, tag) <- ((deserializeAt initialOffset) arr) endOffset
181− case tag :: Word8 of
182− 0 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
183− pure (i1, LIS a0)
184− 1 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
185− pure (i1, LIP a0)
186− 2 -> do (i1, a0) <- ((deserializeAt i0) arr) endOffset
187− pure (i1, LIN a0)
188− _ -> error "Found invalid tag while peeking (LiftedInteger)"
189−
190− {-# INLINE serializeAt #-}
191− serializeAt initialOffset arr val =
192− case val of
193− LIS field0 -> do
194− i0 <- ((serializeAt initialOffset) arr) (0 :: Word8)
195− i1 <- ((serializeAt i0) arr) field0
196− pure i1
197− LIP field0 -> do
198− i0 <- ((serializeAt initialOffset) arr) (1 :: Word8)
199− i1 <- ((serializeAt i0) arr) field0
200− pure i1
201− LIN field0 -> do
202− i0 <- ((serializeAt initialOffset) arr) (2 :: Word8)
203− i1 <- ((serializeAt i0) arr) field0
204− pure i1
205−
206−#if __GLASGOW_HASKELL__ >= 900
207−
208−{-# INLINE liftInteger #-}
209−liftInteger :: Integer -> LiftedInteger
210−liftInteger (IS x) = LIS (I# x)
211−liftInteger (IP x) =
212− LIP (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
213−liftInteger (IN x) =
214− LIN (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
215−
216−{-# INLINE unliftInteger #-}
217−unliftInteger :: LiftedInteger -> Integer
218−unliftInteger (LIS (I# x)) = IS x
219−unliftInteger (LIP (Array (MutByteArray x) _ _)) = IP (unsafeCoerce# x)
220−unliftInteger (LIN (Array (MutByteArray x) _ _)) = IN (unsafeCoerce# x)
221−
222−#else
223−
224−{-# INLINE liftInteger #-}
225−liftInteger :: Integer -> LiftedInteger
226−liftInteger (S# x) = LIS (I# x)
227−liftInteger (Jp# (BN# x)) =
228− LIP (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
229−liftInteger (Jn# (BN# x)) =
230− LIN (Array (MutByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
231−
232−{-# INLINE unliftInteger #-}
233−unliftInteger :: LiftedInteger -> Integer
234−unliftInteger (LIS (I# x)) = S# x
235−unliftInteger (LIP (Array (MutByteArray x) _ _)) =
236− Jp# (BN# (unsafeCoerce# x))
237−unliftInteger (LIN (Array (MutByteArray x) _ _)) =
238− Jn# (BN# (unsafeCoerce# x))
239−
240−#endif
241−
242−instance Serialize Integer where
243− {-# INLINE addSizeTo #-}
244− addSizeTo i a = addSizeTo i (liftInteger a)
245−
246− {-# INLINE deserializeAt #-}
247− deserializeAt off arr end =
248− fmap unliftInteger <$> deserializeAt off arr end
249−
250− {-# INLINE serializeAt #-}
251− serializeAt off arr val = serializeAt off arr (liftInteger val)
core/src/Streamly/Internal/Data/MutByteArray/Type.hsdeleted−321
@@ -1,321 +0,0 @@
1−{-# LANGUAGE UnboxedTuples #-}
2−
3−-- |
4−-- Module : Streamly.Internal.Data.MutByteArray.Type
5−-- Copyright : (c) 2023 Composewell Technologies
6−-- License : BSD3-3-Clause
7−-- Maintainer : streamly@composewell.com
8−-- Stability : experimental
9−-- Portability : GHC
10−--
11−module Streamly.Internal.Data.MutByteArray.Type
12− (
13− -- ** MutByteArray
14− MutByteArray(..)
15− , MutableByteArray
16− , getMutableByteArray# -- XXX getMutByteArray#
17−
18− -- ** Pinning
19− , PinnedState(..)
20− , isPinned
21− , pin
22− , unpin
23−
24− -- ** Allocation
25− , empty
26− , newBytesAs -- XXX should be removed
27− , new
28− , pinnedNew
29− , pinnedNewAlignedBytes -- XXX should be removed
30−
31− -- ** Access
32− , sizeOfMutableByteArray -- XXX length
33− , putSliceUnsafe
34− , cloneSliceUnsafeAs
35− , cloneSliceUnsafe
36− , pinnedCloneSliceUnsafe
37− , unsafePinnedAsPtr
38− , unsafeAsPtr
39−
40− -- ** Deprecated
41− , asPtrUnsafe
42− , nil
43− ) where
44−
45−import Control.Monad.IO.Class (MonadIO(..))
46−#ifdef DEBUG
47−import Control.Monad (when)
48−import Debug.Trace (trace)
49−#endif
50−import GHC.Base (IO(..))
51−import System.IO.Unsafe (unsafePerformIO)
52−
53−import GHC.Exts
54−
55−--------------------------------------------------------------------------------
56−-- The ArrayContents type
57−--------------------------------------------------------------------------------
58−
59−data PinnedState
60− = Pinned
61− | Unpinned deriving (Show, Eq)
62−
63−-- XXX can use UnliftedNewtypes
64−
65−-- | A lifted mutable byte array type wrapping @MutableByteArray# RealWorld@.
66−-- This is a low level array used to back high level unboxed arrays and
67−-- serialized data.
68−data MutByteArray = MutByteArray (MutableByteArray# RealWorld)
69−
70−{-# DEPRECATED MutableByteArray "Please use MutByteArray instead" #-}
71−type MutableByteArray = MutByteArray
72−
73−{-# INLINE getMutableByteArray# #-}
74−getMutableByteArray# :: MutByteArray -> MutableByteArray# RealWorld
75−getMutableByteArray# (MutByteArray mbarr) = mbarr
76−
77−-- | Return the size of the array in bytes.
78−{-# INLINE sizeOfMutableByteArray #-}
79−sizeOfMutableByteArray :: MutByteArray -> IO Int
80−sizeOfMutableByteArray (MutByteArray arr) =
81− IO $ \s ->
82− case getSizeofMutableByteArray# arr s of
83− (# s1, i #) -> (# s1, I# i #)
84−
85−{-# INLINE touch #-}
86−touch :: MutByteArray -> IO ()
87−touch (MutByteArray contents) =
88− IO $ \s -> case touch# contents s of s' -> (# s', () #)
89−
90−-- XXX Some functions in this module are "IO" and others are "m", we need to
91−-- make it consistent.
92−
93−-- XXX We can provide another API for "unsafe" FFI calls passing an unlifted
94−-- pointer to the FFI call. For unsafe calls we do not need to pin the array.
95−-- We can pass an unlifted pointer to the FFI routine to avoid GC kicking in
96−-- before the pointer is wrapped.
97−--
98−-- From the GHC manual:
99−--
100−-- GHC, since version 8.4, guarantees that garbage collection will never occur
101−-- during an unsafe call, even in the bytecode interpreter, and further
102−-- guarantees that unsafe calls will be performed in the calling thread. Making
103−-- it safe to pass heap-allocated objects to unsafe functions.
104−
105−-- | Use a @MutByteArray@ as @Ptr a@. This is useful when we want to pass
106−-- an array as a pointer to some operating system call or to a "safe" FFI call.
107−--
108−-- If the array is not pinned it is copied to pinned memory before passing it
109−-- to the monadic action.
110−--
111−-- /Performance Notes:/ Forces a copy if the array is not pinned. It is advised
112−-- that the programmer keeps this in mind and creates a pinned array
113−-- opportunistically before this operation occurs, to avoid the cost of a copy
114−-- if possible.
115−--
116−-- /Unsafe/ because of direct pointer operations. The user must ensure that
117−-- they are writing within the legal bounds of the array.
118−--
119−-- /Pre-release/
120−--
121−{-# INLINE unsafePinnedAsPtr #-}
122−unsafePinnedAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
123−unsafePinnedAsPtr arr f = do
124− contents <- liftIO $ pin arr
125− let !ptr = Ptr (byteArrayContents#
126− (unsafeCoerce# (getMutableByteArray# contents)))
127− r <- f ptr
128− liftIO $ touch contents
129− return r
130−
131−{-# DEPRECATED asPtrUnsafe "Please use unsafePinnedAsPtr instead." #-}
132−{-# INLINE asPtrUnsafe #-}
133−asPtrUnsafe :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
134−asPtrUnsafe = unsafePinnedAsPtr
135−
136−-- | For use with unsafe FFI functions. Does not force pin the array memory.
137−{-# INLINE unsafeAsPtr #-}
138−unsafeAsPtr :: MonadIO m => MutByteArray -> (Ptr a -> m b) -> m b
139−unsafeAsPtr arr f = do
140− let !ptr = Ptr (byteArrayContents#
141− (unsafeCoerce# (getMutableByteArray# arr)))
142− r <- f ptr
143− liftIO $ touch arr
144− return r
145−
146−--------------------------------------------------------------------------------
147−-- Creation
148−--------------------------------------------------------------------------------
149−
150−{-# NOINLINE empty #-}
151−empty :: MutByteArray
152−empty = unsafePerformIO $ new 0
153−
154−{-# DEPRECATED nil "Please use empty instead" #-}
155−nil :: MutByteArray
156−nil = empty
157−
158−-- XXX add "newRounded" to round up the large size to the next page boundary
159−-- and return the allocated size.
160−{-# INLINE new #-}
161−new :: Int -> IO MutByteArray
162−new nbytes | nbytes < 0 =
163− errorWithoutStackTrace "newByteArray: size must be >= 0"
164−new (I# nbytes) = IO $ \s ->
165− case newByteArray# nbytes s of
166− (# s', mbarr# #) ->
167− let c = MutByteArray mbarr#
168− in (# s', c #)
169−
170−{-# INLINE pinnedNew #-}
171−pinnedNew :: Int -> IO MutByteArray
172−pinnedNew nbytes | nbytes < 0 =
173− errorWithoutStackTrace "pinnedNew: size must be >= 0"
174−pinnedNew (I# nbytes) = IO $ \s ->
175− case newPinnedByteArray# nbytes s of
176− (# s', mbarr# #) ->
177− let c = MutByteArray mbarr#
178− in (# s', c #)
179−
180−{-# INLINE pinnedNewAlignedBytes #-}
181−pinnedNewAlignedBytes :: Int -> Int -> IO MutByteArray
182−pinnedNewAlignedBytes nbytes _align | nbytes < 0 =
183− errorWithoutStackTrace "pinnedNewAlignedBytes: size must be >= 0"
184−pinnedNewAlignedBytes (I# nbytes) (I# align) = IO $ \s ->
185− case newAlignedPinnedByteArray# nbytes align s of
186− (# s', mbarr# #) ->
187− let c = MutByteArray mbarr#
188− in (# s', c #)
189−
190−{-# INLINE newBytesAs #-}
191−newBytesAs :: PinnedState -> Int -> IO MutByteArray
192−newBytesAs Unpinned = new
193−newBytesAs Pinned = pinnedNew
194−
195−-------------------------------------------------------------------------------
196−-- Copying
197−-------------------------------------------------------------------------------
198−
199−-- Note: Array copy is more efficient than streaming copy.
200−-- CopyMutableByteArray# translates to genMemcpy in GHC/CmmToAsm/X86/CodeGen.hs
201−-- glibc memcpy copies bytes/words/pages - unrolls the loops:
202−-- https://github.com/bminor/glibc/blob/4290aed05135ae4c0272006442d147f2155e70d7/string/memcpy.c
203−-- https://github.com/bminor/glibc/blob/4290aed05135ae4c0272006442d147f2155e70d7/string/wordcopy.c
204−
205−-- | Put a sub range of a source array into a subrange of a destination array.
206−-- This is not safe as it does not check the bounds of neither the src array
207−-- nor the destination array.
208−{-# INLINE putSliceUnsafe #-}
209−putSliceUnsafe ::
210− MonadIO m
211− => MutByteArray
212− -> Int
213− -> MutByteArray
214− -> Int
215− -> Int
216− -> m ()
217−putSliceUnsafe src srcStartBytes dst dstStartBytes lenBytes = liftIO $ do
218−#ifdef DEBUG
219− srcLen <- sizeOfMutableByteArray src
220− dstLen <- sizeOfMutableByteArray dst
221− when (srcLen - srcStartBytes < lenBytes)
222− $ error $ "putSliceUnsafe: src overflow: start" ++ show srcStartBytes
223− ++ " end " ++ show srcLen ++ " len " ++ show lenBytes
224− when (dstLen - dstStartBytes < lenBytes)
225− $ error $ "putSliceUnsafe: dst overflow: start" ++ show dstStartBytes
226− ++ " end " ++ show dstLen ++ " len " ++ show lenBytes
227−#endif
228− let !(I# srcStartBytes#) = srcStartBytes
229− !(I# dstStartBytes#) = dstStartBytes
230− !(I# lenBytes#) = lenBytes
231− let arrS# = getMutableByteArray# src
232− arrD# = getMutableByteArray# dst
233− IO $ \s# -> (# copyMutableByteArray#
234− arrS# srcStartBytes# arrD# dstStartBytes# lenBytes# s#
235− , () #)
236−
237−-- | Unsafe as it does not check whether the start offset and length supplied
238−-- are valid inside the array.
239−{-# INLINE cloneSliceUnsafeAs #-}
240−cloneSliceUnsafeAs :: MonadIO m =>
241− PinnedState -> Int -> Int -> MutByteArray -> m MutByteArray
242−cloneSliceUnsafeAs ps srcOff srcLen src =
243− liftIO $ do
244− mba <- newBytesAs ps srcLen
245− putSliceUnsafe src srcOff mba 0 srcLen
246− return mba
247−
248−-- | @cloneSliceUnsafe offset len arr@ clones a slice of the supplied array
249−-- starting at the given offset and equal to the given length.
250−{-# INLINE cloneSliceUnsafe #-}
251−cloneSliceUnsafe :: MonadIO m => Int -> Int -> MutByteArray -> m MutByteArray
252−cloneSliceUnsafe = cloneSliceUnsafeAs Unpinned
253−
254−-- | @pinnedCloneSliceUnsafe offset len arr@
255−{-# INLINE pinnedCloneSliceUnsafe #-}
256−pinnedCloneSliceUnsafe :: MonadIO m =>
257− Int -> Int -> MutByteArray -> m MutByteArray
258−pinnedCloneSliceUnsafe = cloneSliceUnsafeAs Pinned
259−
260−-------------------------------------------------------------------------------
261−-- Pinning & Unpinning
262−-------------------------------------------------------------------------------
263−
264−-- | Return 'True' if the array is allocated in pinned memory.
265−{-# INLINE isPinned #-}
266−isPinned :: MutByteArray -> Bool
267−isPinned (MutByteArray arr#) =
268− let pinnedInt = I# (isMutableByteArrayPinned# arr#)
269− in pinnedInt /= 0
270−
271−
272−{-# INLINE cloneMutableArrayWith# #-}
273−cloneMutableArrayWith#
274− :: (Int# -> State# RealWorld -> (# State# RealWorld
275− , MutableByteArray# RealWorld #))
276− -> MutableByteArray# RealWorld
277− -> State# RealWorld
278− -> (# State# RealWorld, MutableByteArray# RealWorld #)
279−cloneMutableArrayWith# alloc# arr# s# =
280− case getSizeofMutableByteArray# arr# s# of
281− (# s1#, i# #) ->
282− case alloc# i# s1# of
283− (# s2#, arr1# #) ->
284− case copyMutableByteArray# arr# 0# arr1# 0# i# s2# of
285− s3# -> (# s3#, arr1# #)
286−
287−-- | Return a copy of the array in pinned memory if unpinned, else return the
288−-- original array.
289−{-# INLINE pin #-}
290−pin :: MutByteArray -> IO MutByteArray
291−pin arr@(MutByteArray marr#) =
292− if isPinned arr
293− then return arr
294− else
295−#ifdef DEBUG
296− do
297− -- XXX dump stack trace
298− trace ("pin: Copying array") (return ())
299−#endif
300− IO
301− $ \s# ->
302− case cloneMutableArrayWith# newPinnedByteArray# marr# s# of
303− (# s1#, marr1# #) -> (# s1#, MutByteArray marr1# #)
304−
305−-- | Return a copy of the array in unpinned memory if pinned, else return the
306−-- original array.
307−{-# INLINE unpin #-}
308−unpin :: MutByteArray -> IO MutByteArray
309−unpin arr@(MutByteArray marr#) =
310− if not (isPinned arr)
311− then return arr
312− else
313−#ifdef DEBUG
314− do
315− -- XXX dump stack trace
316− trace ("unpin: Copying array") (return ())
317−#endif
318− IO
319− $ \s# ->
320− case cloneMutableArrayWith# newByteArray# marr# s# of
321− (# s1#, marr1# #) -> (# s1#, MutByteArray marr1# #)
core/src/Streamly/Internal/Data/Parser.hs+119−126
@@ -10,13 +10,10 @@
1010module Streamly.Internal.Data.Parser
1111 (
1212 -- * Setup
13− -- | To execute the code examples provided in this module in ghci, please
14− -- run the following commands first.
15− --
1613 -- $setup
1714
1815 module Streamly.Internal.Data.Parser.Type
19− --, module Streamly.Internal.Data.Parser.Tee
16+ --, module Streamly.Internal.Data.Parser.Tee
2017
2118 -- * Types
2219 , Parser (..)
@@ -284,7 +281,7 @@ import Prelude hiding
284281--
285282{-# INLINE toFold #-}
286283toFold :: Monad m => Parser a m b -> Fold m a b
287−toFold (Parser pstep pinitial pextract) = Fold step initial extract final
284+toFold (Parser pstep pinitial pextract) = Fold step initial extract
288285
289286 where
290287
@@ -312,9 +309,7 @@ toFold (Parser pstep pinitial pextract) = Fold step initial extract final
312309 Done n _ -> derror n
313310 Error err -> eerror err
314311
315− extract = error "toFold: parser cannot be used for scanning"
316−
317− final st = do
312+ extract st = do
318313 r <- pextract st
319314 case r of
320315 Done 0 b -> return b
@@ -332,7 +327,7 @@ toFold (Parser pstep pinitial pextract) = Fold step initial extract final
332327--
333328{-# INLINE fromFold #-}
334329fromFold :: Monad m => Fold m a b -> Parser a m b
335−fromFold (Fold fstep finitial _ ffinal) = Parser step initial extract
330+fromFold (Fold fstep finitial fextract) = Parser step initial extract
336331
337332 where
338333
@@ -350,7 +345,7 @@ fromFold (Fold fstep finitial _ ffinal) = Parser step initial extract
350345 FL.Partial s1 -> Partial 0 s1
351346 FL.Done b -> Done 0 b
352347
353− extract = fmap (Done 0) . ffinal
348+ extract = fmap (Done 0) . fextract
354349
355350-- | Convert a Maybe returning fold to an error returning parser. The first
356351-- argument is the error message that the parser would return when the fold
@@ -360,7 +355,7 @@ fromFold (Fold fstep finitial _ ffinal) = Parser step initial extract
360355--
361356{-# INLINE fromFoldMaybe #-}
362357fromFoldMaybe :: Monad m => String -> Fold m a (Maybe b) -> Parser a m b
363−fromFoldMaybe errMsg (Fold fstep finitial _ ffinal) =
358+fromFoldMaybe errMsg (Fold fstep finitial fextract) =
364359 Parser step initial extract
365360
366361 where
@@ -386,7 +381,7 @@ fromFoldMaybe errMsg (Fold fstep finitial _ ffinal) =
386381 Nothing -> Error errMsg
387382
388383 extract s = do
389− res <- ffinal s
384+ res <- fextract s
390385 case res of
391386 Just x -> return $ Done 0 x
392387 Nothing -> return $ Error errMsg
@@ -635,7 +630,7 @@ data Tuple'Fused a b = Tuple'Fused !a !b deriving Show
635630--
636631{-# INLINE takeBetween #-}
637632takeBetween :: Monad m => Int -> Int -> Fold m a b -> Parser a m b
638−takeBetween low high (Fold fstep finitial _ ffinal) =
633+takeBetween low high (Fold fstep finitial fextract) =
639634
640635 Parser step initial (extract streamErr)
641636
@@ -686,11 +681,9 @@ takeBetween low high (Fold fstep finitial _ ffinal) =
686681 in case res of
687682 FL.Partial s -> do
688683 let s1 = Tuple'Fused i1 s
689− if i1 < low
684+ if i1 < high
690685 then return $ Continue 0 s1
691− else if i1 < high
692− then return $ Partial 0 s1
693− else fmap (Done 0) (ffinal s)
686+ else extract foldErr s1
694687 FL.Done b ->
695688 return
696689 $ if i1 >= low
@@ -700,12 +693,12 @@ takeBetween low high (Fold fstep finitial _ ffinal) =
700693 step (Tuple'Fused i s) a = fstep s a >>= snext i
701694
702695 extract f (Tuple'Fused i s)
703− | i >= low && i <= high = fmap (Done 0) (ffinal s)
696+ | i >= low && i <= high = fmap (Done 0) (fextract s)
704697 | otherwise = return $ Error (f i)
705698
706699 -- XXX Need to make Initial return type Step to deduplicate this
707700 iextract f (Tuple'Fused i s)
708− | i >= low && i <= high = fmap IDone (ffinal s)
701+ | i >= low && i <= high = fmap IDone (fextract s)
709702 | otherwise = return $ IError (f i)
710703
711704-- | Stops after taking exactly @n@ input elements.
@@ -722,7 +715,7 @@ takeBetween low high (Fold fstep finitial _ ffinal) =
722715--
723716{-# INLINE takeEQ #-}
724717takeEQ :: Monad m => Int -> Fold m a b -> Parser a m b
725−takeEQ n (Fold fstep finitial _ ffinal) = Parser step initial extract
718+takeEQ n (Fold fstep finitial fextract) = Parser step initial extract
726719
727720 where
728721
@@ -732,7 +725,7 @@ takeEQ n (Fold fstep finitial _ ffinal) = Parser step initial extract
732725 FL.Partial s ->
733726 if n > 0
734727 then return $ IPartial $ Tuple'Fused 1 s
735− else fmap IDone (ffinal s)
728+ else fmap IDone (fextract s)
736729 FL.Done b -> return $
737730 if n > 0
738731 then IError
@@ -756,7 +749,7 @@ takeEQ n (Fold fstep finitial _ ffinal) = Parser step initial extract
756749 -- assert (n == i1)
757750 Done 0
758751 <$> case res of
759− FL.Partial s -> ffinal s
752+ FL.Partial s -> fextract s
760753 FL.Done b -> return b
761754
762755 extract (Tuple'Fused i _) =
@@ -788,7 +781,7 @@ data TakeGEState s =
788781--
789782{-# INLINE takeGE #-}
790783takeGE :: Monad m => Int -> Fold m a b -> Parser a m b
791−takeGE n (Fold fstep finitial _ ffinal) = Parser step initial extract
784+takeGE n (Fold fstep finitial fextract) = Parser step initial extract
792785
793786 where
794787
@@ -836,7 +829,7 @@ takeGE n (Fold fstep finitial _ ffinal) = Parser step initial extract
836829 $ Error
837830 $ "takeGE: Expecting at least " ++ show n
838831 ++ " elements, input terminated on " ++ show (i - 1)
839− extract (TakeGEGE r) = fmap (Done 0) $ ffinal r
832+ extract (TakeGEGE r) = fmap (Done 0) $ fextract r
840833
841834-------------------------------------------------------------------------------
842835-- Conditional splitting
@@ -897,7 +890,7 @@ takeWhileP predicate (Parser pstep pinitial pextract) =
897890{-# INLINE takeWhile #-}
898891takeWhile :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
899892-- takeWhile cond f = takeWhileP cond (fromFold f)
900−takeWhile predicate (Fold fstep finitial _ ffinal) =
893+takeWhile predicate (Fold fstep finitial fextract) =
901894 Parser step initial extract
902895
903896 where
@@ -916,9 +909,9 @@ takeWhile predicate (Fold fstep finitial _ ffinal) =
916909 $ case fres of
917910 FL.Partial s1 -> Partial 0 s1
918911 FL.Done b -> Done 0 b
919− else Done 1 <$> ffinal s
912+ else Done 1 <$> fextract s
920913
921− extract s = fmap (Done 0) (ffinal s)
914+ extract s = fmap (Done 0) (fextract s)
922915
923916{-
924917-- XXX This may not be composable because of the b argument. We can instead
@@ -938,7 +931,7 @@ takeWhile1 acc cond f = undefined
938931{-# INLINE takeWhile1 #-}
939932takeWhile1 :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
940933-- takeWhile1 cond f = takeWhileP cond (takeBetween 1 maxBound f)
941−takeWhile1 predicate (Fold fstep finitial _ ffinal) =
934+takeWhile1 predicate (Fold fstep finitial fextract) =
942935 Parser step initial extract
943936
944937 where
@@ -968,11 +961,11 @@ takeWhile1 predicate (Fold fstep finitial _ ffinal) =
968961 if predicate a
969962 then process s a
970963 else do
971− b <- ffinal s
964+ b <- fextract s
972965 return $ Done 1 b
973966
974967 extract (Left' _) = return $ Error "takeWhile1: end of input"
975− extract (Right' s) = fmap (Done 0) (ffinal s)
968+ extract (Right' s) = fmap (Done 0) (fextract s)
976969
977970-- | Drain the input as long as the predicate succeeds, running the effects and
978971-- discarding the results.
@@ -1001,7 +994,7 @@ takeFramedByGeneric :: Monad m =>
1001994 -> Maybe (a -> Bool) -- is frame end?
1002995 -> Fold m a b
1003996 -> Parser a m b
1004−takeFramedByGeneric esc begin end (Fold fstep finitial _ ffinal) =
997+takeFramedByGeneric esc begin end (Fold fstep finitial fextract) =
1005998
1006999 Parser step initial extract
10071000
@@ -1033,20 +1026,20 @@ takeFramedByGeneric esc begin end (Fold fstep finitial _ ffinal) =
10331026 if isEnd a
10341027 then
10351028 if n == 0
1036− then Done 0 <$> ffinal s
1029+ then Done 0 <$> fextract s
10371030 else process s a (n - 1)
10381031 else
10391032 let n1 = if isBegin a then n + 1 else n
10401033 in process s a n1
10411034 Nothing -> -- takeEndBy case
10421035 if isEnd a
1043− then Done 0 <$> ffinal s
1036+ then Done 0 <$> fextract s
10441037 else process s a n
10451038 Nothing -> -- takeStartBy case
10461039 case begin of
10471040 Just isBegin ->
10481041 if isBegin a
1049− then Done 0 <$> ffinal s
1042+ then Done 0 <$> fextract s
10501043 else process s a n
10511044 Nothing ->
10521045 error $ "takeFramedByGeneric: "
@@ -1071,7 +1064,7 @@ takeFramedByGeneric esc begin end (Fold fstep finitial _ ffinal) =
10711064 case end of
10721065 Just isEnd ->
10731066 if isEnd a
1074− then Done 0 <$> ffinal s
1067+ then Done 0 <$> fextract s
10751068 else processCheckEsc s a 0
10761069 Nothing ->
10771070 error "Both begin and end frame predicate missing"
@@ -1086,7 +1079,7 @@ takeFramedByGeneric esc begin end (Fold fstep finitial _ ffinal) =
10861079 case begin of
10871080 Just _ ->
10881081 case end of
1089− Nothing -> fmap (Done 0) $ ffinal s
1082+ Nothing -> fmap (Done 0) $ fextract s
10901083 Just _ -> err "takeFramedByGeneric: missing frame end"
10911084 Nothing -> err "takeFramedByGeneric: missing closing frame"
10921085 extract (FrameEscEsc _ _) = err "takeFramedByGeneric: trailing escape"
@@ -1125,7 +1118,7 @@ blockWithQuotes :: (Monad m, Eq a) =>
11251118 -> Fold m a b
11261119 -> Parser a m b
11271120blockWithQuotes isEsc isQuote bopen bclose
1128− (Fold fstep finitial _ ffinal) =
1121+ (Fold fstep finitial fextract) =
11291122 Parser step initial extract
11301123
11311124 where
@@ -1155,7 +1148,7 @@ blockWithQuotes isEsc isQuote bopen bclose
11551148 | a == bopen = process s a (BlockUnquoted (level + 1))
11561149 | a == bclose =
11571150 if level == 1
1158− then fmap (Done 0) (ffinal s)
1151+ then fmap (Done 0) (fextract s)
11591152 else process s a (BlockUnquoted (level - 1))
11601153 | isQuote a = process s a (BlockQuoted level)
11611154 | otherwise = process s a (BlockUnquoted level)
@@ -1169,7 +1162,7 @@ blockWithQuotes isEsc isQuote bopen bclose
11691162
11701163 err = return . Error
11711164
1172− extract (BlockInit s) = fmap (Done 0) $ ffinal s
1165+ extract (BlockInit s) = fmap (Done 0) $ fextract s
11731166 extract (BlockUnquoted level _) =
11741167 err $ "blockWithQuotes: finished at block nest level " ++ show level
11751168 extract (BlockQuoted level _) =
@@ -1304,7 +1297,7 @@ takeEitherSepBy _cond = undefined -- D.toParserK . D.takeEitherSepBy cond
13041297--
13051298{-# INLINE takeStartBy #-}
13061299takeStartBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
1307−takeStartBy cond (Fold fstep finitial _ ffinal) =
1300+takeStartBy cond (Fold fstep finitial fextract) =
13081301
13091302 Parser step initial extract
13101303
@@ -1332,10 +1325,10 @@ takeStartBy cond (Fold fstep finitial _ ffinal) =
13321325 step (Right' s) a =
13331326 if not (cond a)
13341327 then process s a
1335− else Done 1 <$> ffinal s
1328+ else Done 1 <$> fextract s
13361329
1337− extract (Left' s) = fmap (Done 0) $ ffinal s
1338− extract (Right' s) = fmap (Done 0) $ ffinal s
1330+ extract (Left' s) = fmap (Done 0) $ fextract s
1331+ extract (Right' s) = fmap (Done 0) $ fextract s
13391332
13401333-- | Like 'takeStartBy' but drops the separator.
13411334--
@@ -1371,7 +1364,7 @@ takeFramedByEsc_ :: Monad m =>
13711364 (a -> Bool) -> (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
13721365-- takeFramedByEsc_ isEsc isEnd p =
13731366-- takeFramedByGeneric (Just isEsc) Nothing (Just isEnd) (toFold p)
1374−takeFramedByEsc_ isEsc isBegin isEnd (Fold fstep finitial _ ffinal ) =
1367+takeFramedByEsc_ isEsc isBegin isEnd (Fold fstep finitial fextract) =
13751368
13761369 Parser step initial extract
13771370
@@ -1407,7 +1400,7 @@ takeFramedByEsc_ isEsc isBegin isEnd (Fold fstep finitial _ ffinal ) =
14071400 in process s a n1
14081401 else
14091402 if n == 0
1410− then Done 0 <$> ffinal s
1403+ then Done 0 <$> fextract s
14111404 else process s a (n - 1)
14121405 step (FrameEscEsc s n) a = process s a n
14131406
@@ -1429,7 +1422,7 @@ takeFramedBy_ :: Monad m =>
14291422 (a -> Bool) -> (a -> Bool) -> Fold m a b -> Parser a m b
14301423-- takeFramedBy_ isBegin isEnd =
14311424-- takeFramedByGeneric (Just (const False)) (Just isBegin) (Just isEnd)
1432−takeFramedBy_ isBegin isEnd (Fold fstep finitial _ ffinal) =
1425+takeFramedBy_ isBegin isEnd (Fold fstep finitial fextract) =
14331426
14341427 Parser step initial extract
14351428
@@ -1459,7 +1452,7 @@ takeFramedBy_ isBegin isEnd (Fold fstep finitial _ ffinal) =
14591452 | not (isEnd a) =
14601453 let n1 = if isBegin a then n + 1 else n
14611454 in process s a n1
1462− | n == 0 = Done 0 <$> ffinal s
1455+ | n == 0 = Done 0 <$> fextract s
14631456 | otherwise = process s a (n - 1)
14641457
14651458 err = return . Error
@@ -1494,7 +1487,7 @@ data WordByState s b = WBLeft !s | WBWord !s | WBRight !b
14941487--
14951488{-# INLINE wordBy #-}
14961489wordBy :: Monad m => (a -> Bool) -> Fold m a b -> Parser a m b
1497−wordBy predicate (Fold fstep finitial _ ffinal) = Parser step initial extract
1490+wordBy predicate (Fold fstep finitial fextract) = Parser step initial extract
14981491
14991492 where
15001493
@@ -1521,7 +1514,7 @@ wordBy predicate (Fold fstep finitial _ ffinal) = Parser step initial extract
15211514 if not (predicate a)
15221515 then worder s a
15231516 else do
1524− b <- ffinal s
1517+ b <- fextract s
15251518 return $ Partial 0 $ WBRight b
15261519 step (WBRight b) a =
15271520 return
@@ -1529,8 +1522,8 @@ wordBy predicate (Fold fstep finitial _ ffinal) = Parser step initial extract
15291522 then Done 1 b
15301523 else Partial 0 $ WBRight b
15311524
1532− extract (WBLeft s) = fmap (Done 0) $ ffinal s
1533− extract (WBWord s) = fmap (Done 0) $ ffinal s
1525+ extract (WBLeft s) = fmap (Done 0) $ fextract s
1526+ extract (WBWord s) = fmap (Done 0) $ fextract s
15341527 extract (WBRight b) = return (Done 0 b)
15351528
15361529data WordFramedState s b =
@@ -1567,7 +1560,7 @@ wordFramedBy :: Monad m =>
15671560 -> Fold m a b
15681561 -> Parser a m b
15691562wordFramedBy isEsc isBegin isEnd isSep
1570− (Fold fstep finitial _ ffinal) =
1563+ (Fold fstep finitial fextract) =
15711564 Parser step initial extract
15721565
15731566 where
@@ -1598,7 +1591,7 @@ wordFramedBy isEsc isBegin isEnd isSep
15981591 step (WordFramedWord s n) a
15991592 | isEsc a = return $ Continue 0 $ WordFramedEsc s n
16001593 | n == 0 && isSep a = do
1601− b <- ffinal s
1594+ b <- fextract s
16021595 return $ Partial 0 $ WordFramedSkipPost b
16031596 | otherwise = do
16041597 -- We need to use different order for checking begin and end for
@@ -1631,10 +1624,10 @@ wordFramedBy isEsc isBegin isEnd isSep
16311624
16321625 err = return . Error
16331626
1634− extract (WordFramedSkipPre s) = fmap (Done 0) $ ffinal s
1627+ extract (WordFramedSkipPre s) = fmap (Done 0) $ fextract s
16351628 extract (WordFramedWord s n) =
16361629 if n == 0
1637− then fmap (Done 0) $ ffinal s
1630+ then fmap (Done 0) $ fextract s
16381631 else err "wordFramedBy: missing frame end"
16391632 extract (WordFramedEsc _ _) =
16401633 err "wordFramedBy: trailing escape"
@@ -1721,7 +1714,7 @@ wordWithQuotes :: (Monad m, Eq a) =>
17211714 -> Fold m a b
17221715 -> Parser a m b
17231716wordWithQuotes keepQuotes tr escChar toRight isSep
1724− (Fold fstep finitial _ ffinal) =
1717+ (Fold fstep finitial fextract) =
17251718 Parser step initial extract
17261719
17271720 where
@@ -1772,7 +1765,7 @@ wordWithQuotes keepQuotes tr escChar toRight isSep
17721765 step (WordUnquotedWord s) a
17731766 | isEsc a = return $ Continue 0 $ WordUnquotedEsc s
17741767 | isSep a = do
1775− b <- ffinal s
1768+ b <- fextract s
17761769 return $ Partial 0 $ WordQuotedSkipPost b
17771770 | otherwise = do
17781771 case toRight a of
@@ -1820,11 +1813,11 @@ wordWithQuotes keepQuotes tr escChar toRight isSep
18201813
18211814 err = return . Error
18221815
1823− extract (WordQuotedSkipPre s) = fmap (Done 0) $ ffinal s
1824− extract (WordUnquotedWord s) = fmap (Done 0) $ ffinal s
1816+ extract (WordQuotedSkipPre s) = fmap (Done 0) $ fextract s
1817+ extract (WordUnquotedWord s) = fmap (Done 0) $ fextract s
18251818 extract (WordQuotedWord s n _ _) =
18261819 if n == 0
1827− then fmap (Done 0) $ ffinal s
1820+ then fmap (Done 0) $ fextract s
18281821 else err "wordWithQuotes: missing frame end"
18291822 extract WordQuotedEsc {} =
18301823 err "wordWithQuotes: trailing escape"
@@ -1902,7 +1895,7 @@ data GroupByState a s
19021895--
19031896{-# INLINE groupBy #-}
19041897groupBy :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
1905−groupBy eq (Fold fstep finitial _ ffinal) = Parser step initial extract
1898+groupBy eq (Fold fstep finitial fextract) = Parser step initial extract
19061899
19071900 where
19081901
@@ -1925,10 +1918,10 @@ groupBy eq (Fold fstep finitial _ ffinal) = Parser step initial extract
19251918 step (GroupByGrouping a0 s) a =
19261919 if eq a0 a
19271920 then grouper s a0 a
1928− else Done 1 <$> ffinal s
1921+ else Done 1 <$> fextract s
19291922
1930− extract (GroupByInit s) = fmap (Done 0) $ ffinal s
1931− extract (GroupByGrouping _ s) = fmap (Done 0) $ ffinal s
1923+ extract (GroupByInit s) = fmap (Done 0) $ fextract s
1924+ extract (GroupByGrouping _ s) = fmap (Done 0) $ fextract s
19321925
19331926-- | Unlike 'groupBy' this combinator performs a rolling comparison of two
19341927-- successive elements in the input stream. Assuming the input stream
@@ -1962,7 +1955,7 @@ groupBy eq (Fold fstep finitial _ ffinal) = Parser step initial extract
19621955--
19631956{-# INLINE groupByRolling #-}
19641957groupByRolling :: Monad m => (a -> a -> Bool) -> Fold m a b -> Parser a m b
1965−groupByRolling eq (Fold fstep finitial _ ffinal) = Parser step initial extract
1958+groupByRolling eq (Fold fstep finitial fextract) = Parser step initial extract
19661959
19671960 where
19681961
@@ -1985,10 +1978,10 @@ groupByRolling eq (Fold fstep finitial _ ffinal) = Parser step initial extract
19851978 step (GroupByGrouping a0 s) a =
19861979 if eq a0 a
19871980 then grouper s a
1988− else Done 1 <$> ffinal s
1981+ else Done 1 <$> fextract s
19891982
1990− extract (GroupByInit s) = fmap (Done 0) $ ffinal s
1991− extract (GroupByGrouping _ s) = fmap (Done 0) $ ffinal s
1983+ extract (GroupByInit s) = fmap (Done 0) $ fextract s
1984+ extract (GroupByGrouping _ s) = fmap (Done 0) $ fextract s
19921985
19931986{-# ANN type GroupByStatePair Fuse #-}
19941987data GroupByStatePair a s1 s2
@@ -2012,8 +2005,8 @@ groupByRollingEither :: Monad m =>
20122005 (a -> a -> Bool) -> Fold m a b -> Fold m a c -> Parser a m (Either b c)
20132006groupByRollingEither
20142007 eq
2015− (Fold fstep1 finitial1 _ ffinal1)
2016− (Fold fstep2 finitial2 _ ffinal2) = Parser step initial extract
2008+ (Fold fstep1 finitial1 fextract1)
2009+ (Fold fstep2 finitial2 fextract2) = Parser step initial extract
20172010
20182011 where
20192012
@@ -2072,21 +2065,21 @@ groupByRollingEither
20722065 step (GroupByGroupingPairL a0 s1 s2) a =
20732066 if not (eq a0 a)
20742067 then grouperL2 s1 s2 a
2075− else Done 1 . Left <$> ffinal1 s1
2068+ else Done 1 . Left <$> fextract1 s1
20762069
20772070 step (GroupByGroupingPairR a0 s1 s2) a =
20782071 if eq a0 a
20792072 then grouperR2 s1 s2 a
2080− else Done 1 . Right <$> ffinal2 s2
2073+ else Done 1 . Right <$> fextract2 s2
20812074
2082− extract (GroupByInitPair s1 _) = Done 0 . Left <$> ffinal1 s1
2083− extract (GroupByGroupingPairL _ s1 _) = Done 0 . Left <$> ffinal1 s1
2084− extract (GroupByGroupingPairR _ _ s2) = Done 0 . Right <$> ffinal2 s2
2075+ extract (GroupByInitPair s1 _) = Done 0 . Left <$> fextract1 s1
2076+ extract (GroupByGroupingPairL _ s1 _) = Done 0 . Left <$> fextract1 s1
2077+ extract (GroupByGroupingPairR _ _ s2) = Done 0 . Right <$> fextract2 s2
20852078 extract (GroupByGroupingPair a s1 _) = do
20862079 res <- fstep1 s1 a
20872080 case res of
20882081 FL.Done b -> return $ Done 0 (Left b)
2089− FL.Partial s11 -> Done 0 . Left <$> ffinal1 s11
2082+ FL.Partial s11 -> Done 0 . Left <$> fextract1 s11
20902083
20912084-- XXX use an Unfold instead of a list?
20922085-- XXX custom combinators for matching list, array and stream?
@@ -2239,7 +2232,7 @@ postscan = undefined
22392232{-# INLINE zipWithM #-}
22402233zipWithM :: Monad m =>
22412234 (a -> b -> m c) -> D.Stream m a -> Fold m c x -> Parser b m x
2242−zipWithM zf (D.Stream sstep state) (Fold fstep finitial _ ffinal) =
2235+zipWithM zf (D.Stream sstep state) (Fold fstep finitial fextract) =
22432236 Parser step initial extract
22442237
22452238 where
@@ -2252,7 +2245,7 @@ zipWithM zf (D.Stream sstep state) (Fold fstep finitial _ ffinal) =
22522245 case r of
22532246 D.Yield x s -> return $ IPartial (Just' x, s, fs)
22542247 D.Stop -> do
2255− x <- ffinal fs
2248+ x <- fextract fs
22562249 return $ IDone x
22572250 -- Need Skip/Continue in initial to loop right here
22582251 D.Skip s -> return $ IPartial (Nothing', s, fs)
@@ -2267,7 +2260,7 @@ zipWithM zf (D.Stream sstep state) (Fold fstep finitial _ ffinal) =
22672260 case r of
22682261 D.Yield x1 s -> return $ Continue 0 (Just' x1, s, fs1)
22692262 D.Stop -> do
2270− x <- ffinal fs1
2263+ x <- fextract fs1
22712264 return $ Done 0 x
22722265 D.Skip s -> return $ Continue 1 (Nothing', s, fs1)
22732266 FL.Done x -> return $ Done 0 x
@@ -2282,7 +2275,7 @@ zipWithM zf (D.Stream sstep state) (Fold fstep finitial _ ffinal) =
22822275 return $ Continue 0 (Nothing', s, fs1)
22832276 FL.Done x -> return $ Done 0 x
22842277 D.Stop -> do
2285− x <- ffinal fs
2278+ x <- fextract fs
22862279 return $ Done 1 x
22872280 D.Skip s -> return $ Continue 1 (Nothing', s, fs)
22882281
@@ -2566,7 +2559,7 @@ deintercalateAll :: Monad m =>
25662559deintercalateAll
25672560 (Parser stepL initialL extractL)
25682561 (Parser stepR initialR _)
2569− (Fold fstep finitial _ ffinal) = Parser step initial extract
2562+ (Fold fstep finitial fextract) = Parser step initial extract
25702563
25712564 where
25722565
@@ -2637,9 +2630,9 @@ deintercalateAll
26372630 extractResult n fs r = do
26382631 res <- fstep fs r
26392632 case res of
2640− FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
2633+ FL.Partial fs1 -> fmap (Done n) $ fextract fs1
26412634 FL.Done c -> return (Done n c)
2642− extract (DeintercalateAllInitL fs) = fmap (Done 0) $ ffinal fs
2635+ extract (DeintercalateAllInitL fs) = fmap (Done 0) $ fextract fs
26432636 extract (DeintercalateAllL fs sL) = do
26442637 r <- extractL sL
26452638 case r of
@@ -2647,7 +2640,7 @@ deintercalateAll
26472640 Error err -> return $ Error err
26482641 Continue n s -> return $ Continue n (DeintercalateAllL fs s)
26492642 Partial _ _ -> error "Partial in extract"
2650− extract (DeintercalateAllInitR fs) = fmap (Done 0) $ ffinal fs
2643+ extract (DeintercalateAllInitR fs) = fmap (Done 0) $ fextract fs
26512644 extract (DeintercalateAllR _ _) =
26522645 return $ Error "deintercalateAll: input ended at 'Right' value"
26532646
@@ -2690,7 +2683,7 @@ deintercalate :: Monad m =>
26902683deintercalate
26912684 (Parser stepL initialL extractL)
26922685 (Parser stepR initialR _)
2693− (Fold fstep finitial _ ffinal) = Parser step initial extract
2686+ (Fold fstep finitial fextract) = Parser step initial extract
26942687
26952688 where
26962689
@@ -2721,7 +2714,7 @@ deintercalate
27212714 Done n b ->
27222715 processL (fstep fs (Left b)) n DeintercalateInitR
27232716 Error _ -> do
2724− xs <- ffinal fs
2717+ xs <- fextract fs
27252718 return $ Done cnt1 xs
27262719
27272720 {-# INLINE processR #-}
@@ -2741,7 +2734,7 @@ deintercalate
27412734 Continue n s -> return $ Continue n (DeintercalateR (cnt1 - n) fs s)
27422735 Done n b -> processR (cnt1 - n) b fs n
27432736 Error _ -> do
2744− xs <- ffinal fs
2737+ xs <- fextract fs
27452738 return $ Done cnt1 xs
27462739
27472740 step (DeintercalateInitL fs) a = do
@@ -2776,17 +2769,17 @@ deintercalate
27762769 -- XXX We could have the fold accept pairs of (bR, bL)
27772770 FL.Done _ -> error "Fold terminated consuming partial input"
27782771 Error _ -> do
2779− xs <- ffinal fs
2772+ xs <- fextract fs
27802773 return $ Done cnt1 xs
27812774
27822775 {-# INLINE extractResult #-}
27832776 extractResult n fs r = do
27842777 res <- fstep fs r
27852778 case res of
2786− FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
2779+ FL.Partial fs1 -> fmap (Done n) $ fextract fs1
27872780 FL.Done c -> return (Done n c)
27882781
2789− extract (DeintercalateInitL fs) = fmap (Done 0) $ ffinal fs
2782+ extract (DeintercalateInitL fs) = fmap (Done 0) $ fextract fs
27902783 extract (DeintercalateL cnt fs sL) = do
27912784 r <- extractL sL
27922785 case r of
@@ -2794,10 +2787,10 @@ deintercalate
27942787 Continue n s -> return $ Continue n (DeintercalateL (cnt - n) fs s)
27952788 Partial _ _ -> error "Partial in extract"
27962789 Error _ -> do
2797− xs <- ffinal fs
2790+ xs <- fextract fs
27982791 return $ Done cnt xs
2799− extract (DeintercalateInitR fs) = fmap (Done 0) $ ffinal fs
2800− extract (DeintercalateR cnt fs _) = fmap (Done cnt) $ ffinal fs
2792+ extract (DeintercalateInitR fs) = fmap (Done 0) $ fextract fs
2793+ extract (DeintercalateR cnt fs _) = fmap (Done cnt) $ fextract fs
28012794 extract (DeintercalateRL cnt bR fs sL) = do
28022795 r <- extractL sL
28032796 case r of
@@ -2809,7 +2802,7 @@ deintercalate
28092802 Continue n s -> return $ Continue n (DeintercalateRL (cnt - n) bR fs s)
28102803 Partial _ _ -> error "Partial in extract"
28112804 Error _ -> do
2812− xs <- ffinal fs
2805+ xs <- fextract fs
28132806 return $ Done cnt xs
28142807
28152808{-# ANN type Deintercalate1State Fuse #-}
@@ -2846,7 +2839,7 @@ deintercalate1 :: Monad m =>
28462839deintercalate1
28472840 (Parser stepL initialL extractL)
28482841 (Parser stepR initialR _)
2849− (Fold fstep finitial _ ffinal) = Parser step initial extract
2842+ (Fold fstep finitial fextract) = Parser step initial extract
28502843
28512844 where
28522845
@@ -2900,7 +2893,7 @@ deintercalate1
29002893 Continue n s -> return $ Continue n (Deintercalate1R (cnt1 - n) fs s)
29012894 Done n b -> processR (cnt1 - n) b fs n
29022895 Error _ -> do
2903− xs <- ffinal fs
2896+ xs <- fextract fs
29042897 return $ Done cnt1 xs
29052898
29062899 step (Deintercalate1InitL cnt fs sL) a = runStepInitL cnt fs sL a
@@ -2929,14 +2922,14 @@ deintercalate1
29292922 -- XXX We could have the fold accept pairs of (bR, bL)
29302923 FL.Done _ -> error "Fold terminated consuming partial input"
29312924 Error _ -> do
2932− xs <- ffinal fs
2925+ xs <- fextract fs
29332926 return $ Done cnt1 xs
29342927
29352928 {-# INLINE extractResult #-}
29362929 extractResult n fs r = do
29372930 res <- fstep fs r
29382931 case res of
2939− FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
2932+ FL.Partial fs1 -> fmap (Done n) $ fextract fs1
29402933 FL.Done c -> return (Done n c)
29412934
29422935 extract (Deintercalate1InitL cnt fs sL) = do
@@ -2946,8 +2939,8 @@ deintercalate1
29462939 Continue n s -> return $ Continue n (Deintercalate1InitL (cnt - n) fs s)
29472940 Partial _ _ -> error "Partial in extract"
29482941 Error err -> return $ Error err
2949− extract (Deintercalate1InitR fs) = fmap (Done 0) $ ffinal fs
2950− extract (Deintercalate1R cnt fs _) = fmap (Done cnt) $ ffinal fs
2942+ extract (Deintercalate1InitR fs) = fmap (Done 0) $ fextract fs
2943+ extract (Deintercalate1R cnt fs _) = fmap (Done cnt) $ fextract fs
29512944 extract (Deintercalate1RL cnt bR fs sL) = do
29522945 r <- extractL sL
29532946 case r of
@@ -2959,7 +2952,7 @@ deintercalate1
29592952 Continue n s -> return $ Continue n (Deintercalate1RL (cnt - n) bR fs s)
29602953 Partial _ _ -> error "Partial in extract"
29612954 Error _ -> do
2962− xs <- ffinal fs
2955+ xs <- fextract fs
29632956 return $ Done cnt xs
29642957
29652958{-# ANN type SepByState Fuse #-}
@@ -3002,7 +2995,7 @@ sepBy :: Monad m =>
30022995sepBy
30032996 (Parser stepL initialL extractL)
30042997 (Parser stepR initialR _)
3005− (Fold fstep finitial _ ffinal) = Parser step initial extract
2998+ (Fold fstep finitial fextract) = Parser step initial extract
30062999
30073000 where
30083001
@@ -3033,7 +3026,7 @@ sepBy
30333026 Done n b ->
30343027 processL (fstep fs b) n SepByInitR
30353028 Error _ -> do
3036− xs <- ffinal fs
3029+ xs <- fextract fs
30373030 return $ Done cnt1 xs
30383031
30393032 {-# INLINE processR #-}
@@ -3053,7 +3046,7 @@ sepBy
30533046 Continue n s -> return $ Continue n (SepByR (cnt1 - n) fs s)
30543047 Done n _ -> processR (cnt1 - n) fs n
30553048 Error _ -> do
3056− xs <- ffinal fs
3049+ xs <- fextract fs
30573050 return $ Done cnt1 xs
30583051
30593052 step (SepByInitL fs) a = do
@@ -3075,10 +3068,10 @@ sepBy
30753068 extractResult n fs r = do
30763069 res <- fstep fs r
30773070 case res of
3078− FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
3071+ FL.Partial fs1 -> fmap (Done n) $ fextract fs1
30793072 FL.Done c -> return (Done n c)
30803073
3081− extract (SepByInitL fs) = fmap (Done 0) $ ffinal fs
3074+ extract (SepByInitL fs) = fmap (Done 0) $ fextract fs
30823075 extract (SepByL cnt fs sL) = do
30833076 r <- extractL sL
30843077 case r of
@@ -3086,10 +3079,10 @@ sepBy
30863079 Continue n s -> return $ Continue n (SepByL (cnt - n) fs s)
30873080 Partial _ _ -> error "Partial in extract"
30883081 Error _ -> do
3089− xs <- ffinal fs
3082+ xs <- fextract fs
30903083 return $ Done cnt xs
3091− extract (SepByInitR fs) = fmap (Done 0) $ ffinal fs
3092− extract (SepByR cnt fs _) = fmap (Done cnt) $ ffinal fs
3084+ extract (SepByInitR fs) = fmap (Done 0) $ fextract fs
3085+ extract (SepByR cnt fs _) = fmap (Done cnt) $ fextract fs
30933086
30943087-- | Non-backtracking version of sepBy. Several times faster.
30953088{-# INLINE sepByAll #-}
@@ -3144,7 +3137,7 @@ sepBy1 :: Monad m =>
31443137sepBy1
31453138 (Parser stepL initialL extractL)
31463139 (Parser stepR initialR _)
3147− (Fold fstep finitial _ ffinal) = Parser step initial extract
3140+ (Fold fstep finitial fextract) = Parser step initial extract
31483141
31493142 where
31503143
@@ -3191,7 +3184,7 @@ sepBy1
31913184 Done n b ->
31923185 processL (fstep fs b) n SepBy1InitR
31933186 Error _ -> do
3194− xs <- ffinal fs
3187+ xs <- fextract fs
31953188 return $ Done cnt1 xs
31963189
31973190 {-# INLINE processR #-}
@@ -3211,7 +3204,7 @@ sepBy1
32113204 Continue n s -> return $ Continue n (SepBy1R (cnt1 - n) fs s)
32123205 Done n _ -> processR (cnt1 - n) fs n
32133206 Error _ -> do
3214− xs <- ffinal fs
3207+ xs <- fextract fs
32153208 return $ Done cnt1 xs
32163209
32173210 step (SepBy1InitL cnt fs sL) a = runStepInitL cnt fs sL a
@@ -3228,7 +3221,7 @@ sepBy1
32283221 extractResult n fs r = do
32293222 res <- fstep fs r
32303223 case res of
3231− FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
3224+ FL.Partial fs1 -> fmap (Done n) $ fextract fs1
32323225 FL.Done c -> return (Done n c)
32333226
32343227 extract (SepBy1InitL cnt fs sL) = do
@@ -3245,10 +3238,10 @@ sepBy1
32453238 Continue n s -> return $ Continue n (SepBy1L (cnt - n) fs s)
32463239 Partial _ _ -> error "Partial in extract"
32473240 Error _ -> do
3248− xs <- ffinal fs
3241+ xs <- fextract fs
32493242 return $ Done cnt xs
3250− extract (SepBy1InitR fs) = fmap (Done 0) $ ffinal fs
3251− extract (SepBy1R cnt fs _) = fmap (Done cnt) $ ffinal fs
3243+ extract (SepBy1InitR fs) = fmap (Done 0) $ fextract fs
3244+ extract (SepBy1R cnt fs _) = fmap (Done cnt) $ fextract fs
32523245
32533246-------------------------------------------------------------------------------
32543247-- Interleaving a collection of parsers
@@ -3278,7 +3271,7 @@ roundRobin _ps _f = undefined
32783271{-# INLINE sequence #-}
32793272sequence :: Monad m =>
32803273 D.Stream m (Parser a m b) -> Fold m b c -> Parser a m c
3281−sequence (D.Stream sstep sstate) (Fold fstep finitial _ ffinal) =
3274+sequence (D.Stream sstep sstate) (Fold fstep finitial fextract) =
32823275 Parser step initial extract
32833276
32843277 where
@@ -3296,7 +3289,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial _ ffinal) =
32963289 case sres of
32973290 D.Yield p ss1 -> return $ Continue 1 (Just' p, ss1, fs)
32983291 D.Stop -> do
3299− c <- ffinal fs
3292+ c <- fextract fs
33003293 return $ Done 1 c
33013294 D.Skip ss1 -> return $ Continue 1 (Nothing', ss1, fs)
33023295
@@ -3332,7 +3325,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial _ ffinal) =
33323325 FL.Done c -> return $ Done 1 c
33333326 IError err -> return $ Error err
33343327
3335− extract (Nothing', _, fs) = fmap (Done 0) $ ffinal fs
3328+ extract (Nothing', _, fs) = fmap (Done 0) $ fextract fs
33363329 extract (Just' (Parser pstep pinit pextr), ss, fs) = do
33373330 ps <- pinit
33383331 case ps of
@@ -3342,7 +3335,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial _ ffinal) =
33423335 Done n b -> do
33433336 res <- fstep fs b
33443337 case res of
3345− FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
3338+ FL.Partial fs1 -> fmap (Done n) $ fextract fs1
33463339 FL.Done c -> return (Done n c)
33473340 Error err -> return $ Error err
33483341 Continue n s -> return $ Continue n (Just' (Parser pstep (return (IPartial s)) pextr), ss, fs)
@@ -3350,7 +3343,7 @@ sequence (D.Stream sstep sstate) (Fold fstep finitial _ ffinal) =
33503343 IDone b -> do
33513344 fres <- fstep fs b
33523345 case fres of
3353− FL.Partial fs1 -> fmap (Done 0) $ ffinal fs1
3346+ FL.Partial fs1 -> fmap (Done 0) $ fextract fs1
33543347 FL.Done c -> return (Done 0 c)
33553348 IError err -> return $ Error err
33563349
@@ -3485,7 +3478,7 @@ manyTill :: Monad m
34853478 => Parser a m b -> Parser a m x -> Fold m b c -> Parser a m c
34863479manyTill (Parser stepL initialL extractL)
34873480 (Parser stepR initialR _)
3488− (Fold fstep finitial _ ffinal) =
3481+ (Fold fstep finitial fextract) =
34893482 Parser step initial extract
34903483
34913484 where
@@ -3507,7 +3500,7 @@ manyTill (Parser stepL initialL extractL)
35073500 resR <- initialR
35083501 case resR of
35093502 IPartial sr -> return $ p (ManyTillR 0 fs sr)
3510− IDone _ -> d <$> ffinal fs
3503+ IDone _ -> d <$> fextract fs
35113504 IError _ -> scrutL fs p c d e
35123505
35133506 initial = do
@@ -3524,7 +3517,7 @@ manyTill (Parser stepL initialL extractL)
35243517 assertM(cnt + 1 - n >= 0)
35253518 return $ Continue n (ManyTillR (cnt + 1 - n) fs s)
35263519 Done n _ -> do
3527− b <- ffinal fs
3520+ b <- fextract fs
35283521 return $ Done n b
35293522 Error _ -> do
35303523 resL <- initialL
@@ -3563,12 +3556,12 @@ manyTill (Parser stepL initialL extractL)
35633556 Done n b -> do
35643557 r <- fstep fs b
35653558 case r of
3566− FL.Partial fs1 -> fmap (Done n) $ ffinal fs1
3559+ FL.Partial fs1 -> fmap (Done n) $ fextract fs1
35673560 FL.Done c -> return (Done n c)
35683561 Error err -> return $ Error err
35693562 Continue n s -> return $ Continue n (ManyTillL fs s)
35703563 Partial _ _ -> error "Partial in extract"
3571− extract (ManyTillR _ fs _) = fmap (Done 0) $ ffinal fs
3564+ extract (ManyTillR _ fs _) = fmap (Done 0) $ fextract fs
35723565
35733566-- | @manyThen f collect recover@ repeats the parser @collect@ on the input and
35743567-- collects the output in the supplied fold. If the the parser @collect@ fails,
core/src/Streamly/Internal/Data/Parser/Type.hs+70−93
@@ -175,6 +175,9 @@
175175
176176module Streamly.Internal.Data.Parser.Type
177177 (
178+ -- * Setup
179+ -- $setup
180+
178181 -- * Types
179182 Initial (..)
180183 , Step (..)
@@ -213,10 +216,7 @@ where
213216#include "inline.hs"
214217#include "assert.hs"
215218
216−#if !MIN_VERSION_base(4,18,0)
217−import Control.Applicative (liftA2)
218−#endif
219−import Control.Applicative (Alternative(..))
219+import Control.Applicative (Alternative(..), liftA2)
220220import Control.Exception (Exception(..))
221221-- import Control.Monad (MonadPlus(..), (>=>))
222222import Control.Monad ((>=>))
@@ -438,7 +438,6 @@ newtype ParseError = ParseError String
438438instance Exception ParseError where
439439 displayException (ParseError err) = err
440440
441−-- | Map a function on the result i.e. on @b@ in @Parser a m b@.
442441instance Functor m => Functor (Parser a m) where
443442 {-# INLINE fmap #-}
444443 fmap f (Parser step1 initial1 extract) =
@@ -505,30 +504,26 @@ data SeqParseState sl f sr = SeqParseL !sl | SeqParseR !f !sr
505504-- 'Streamly.Data.Stream.append', it splits the streams using two parsers and
506505-- zips the results.
507506
508−-- | Sequential parser application. Apply two parsers sequentially to an input
509−-- stream. The first parser runs and processes the input, the remaining input
510−-- is then passed to the second parser. If both parsers succeed, their outputs
511−-- are combined using the supplied function. If either parser fails, the
512−-- operation fails.
507+-- | Sequential parser application.
513508--
514−-- This combinator delivers high performance by stream fusion but it comes with
515−-- some limitations. For those cases use the 'Applicative' instance of
516−-- 'Streamly.Data.ParserK.ParserK'.
509+-- Apply two parsers sequentially to an input stream. The first parser runs and
510+-- processes the input, the remaining input is then passed to the second
511+-- parser. If both parsers succeed, their outputs are combined using the
512+-- supplied function. If either parser fails, the operation fails.
517513--
518−-- CAVEAT 1: NO RECURSION. This function is strict in both arguments. As a
519−-- result, if a parser is defined recursively using this, it may cause an
520−-- infintie loop. The following example checks the strictness:
514+-- This implementation is strict in the second argument, therefore, the
515+-- following will fail:
521516--
522−-- >>> p = Parser.splitWith const (Parser.satisfy (> 0)) undefined
523−-- >>> Stream.parse p $ Stream.fromList [1]
517+-- >>> Stream.parse (Parser.splitWith const (Parser.satisfy (> 0)) undefined) $ Stream.fromList [1]
524518-- *** Exception: Prelude.undefined
525519-- ...
526520--
527−-- CAVEAT 2: QUADRATIC TIME COMPLEXITY. Static composition is fast due to
528−-- stream fusion, but it works well only for limited (e.g. up to 8)
529−-- compositions, use "Streamly.Data.ParserK" for larger compositions.
521+-- Although this implementation allows stream fusion, it has quadratic
522+-- complexity, making it suitable only for a small number of compositions.
523+-- As a thumb rule use it for less than 8 compositions, use ParserK otherwise.
530524--
531−-- Below are some common idioms that can be expressed using 'splitWith':
525+-- Below are some common idioms that can be expressed using 'splitWith' and
526+-- other parser primitives:
532527--
533528-- >>> span p f1 f2 = Parser.splitWith (,) (Parser.takeWhile p f1) (Parser.fromFold f2)
534529-- >>> spanBy eq f1 f2 = Parser.splitWith (,) (Parser.groupBy eq f1) (Parser.fromFold f2)
@@ -617,12 +612,7 @@ splitWith func (Parser stepL initialL extractL)
617612-- We can use the compiler to automatically annotate accumulators, terminating
618613-- folds, non-failing parsers and failing parsers.
619614
620−-- | Better performance 'splitWith' for non-failing parsers.
621−--
622−-- Does not work correctly for parsers that can fail.
623−--
624−-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
625−--
615+-- | Works correctly only if both the parsers are guaranteed to never fail.
626616{-# INLINE noErrorUnsafeSplitWith #-}
627617noErrorUnsafeSplitWith :: Monad m
628618 => (a -> b -> c) -> Parser x m a -> Parser x m b -> Parser x m c
@@ -693,8 +683,6 @@ data SeqAState sl sr = SeqAL !sl | SeqAR !sr
693683-- second parser. The output of the parser is the output of the second parser.
694684-- The operation fails if any of the parsers fail.
695685--
696−-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
697−--
698686-- This implementation is strict in the second argument, therefore, the
699687-- following will fail:
700688--
@@ -702,6 +690,10 @@ data SeqAState sl sr = SeqAL !sl | SeqAR !sr
702690-- *** Exception: Prelude.undefined
703691-- ...
704692--
693+-- Although this implementation allows stream fusion, it has quadratic
694+-- complexity, making it suitable only for a small number of compositions.
695+-- As a thumb rule use it for less than 8 compositions, use ParserK otherwise.
696+--
705697-- /Pre-release/
706698--
707699{-# INLINE split_ #-}
@@ -757,12 +749,7 @@ split_ (Parser stepL initialL extractL) (Parser stepR initialR extractR) =
757749 Partial _ _ -> error "split_: Partial"
758750 Continue n s -> return $ Continue n (SeqAL s)
759751
760−-- | Better performance 'split_' for non-failing parsers.
761−--
762−-- Does not work correctly for parsers that can fail.
763−--
764−-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
765−--
752+-- For backtracking folds
766753{-# INLINE noErrorUnsafeSplit_ #-}
767754noErrorUnsafeSplit_ :: Monad m => Parser x m a -> Parser x m b -> Parser x m b
768755noErrorUnsafeSplit_
@@ -817,11 +804,7 @@ noErrorUnsafeSplit_
817804 Partial _ _ -> error "split_: Partial"
818805 Continue n s -> return $ Continue n (SeqAL s)
819806
820−-- | READ THE CAVEATS in 'splitWith' before using this instance.
821−--
822−-- >>> pure = Parser.fromPure
823−-- >>> (<*>) = Parser.splitWith id
824−-- >>> (*>) = Parser.split_
807+-- | 'Applicative' form of 'splitWith'.
825808instance Monad m => Applicative (Parser a m) where
826809 {-# INLINE pure #-}
827810 pure = fromPure
@@ -847,26 +830,20 @@ data AltParseState sl sr = AltParseL !Int !sl | AltParseR !sr
847830-- each subsequent alternative's input element has to go through, therefore, it
848831-- cannot scale to a large number of compositions
849832
850−-- | Sequential alternative. The input is first passed to the first parser,
833+-- | Sequential alternative. The input is first passed to the first parser, and
851834-- if it succeeds, the result is returned. However, if the first parser fails,
852−-- the parser driver backtracks and tries the same input on the second
853−-- (alternative) parser, returning the result if it succeeds.
854−--
855−-- This combinator delivers high performance by stream fusion but it comes with
856−-- some limitations. For those cases use the 'Alternative' instance of
857−-- 'Streamly.Data.ParserK.ParserK'.
835+-- the parser driver backtracks and tries the same input on the second parser,
836+-- returning the result if it succeeds.
858837--
859−-- CAVEAT 1: NO RECURSION. This function is strict in both arguments. As a
860−-- result, if a parser is defined recursively using this, it may cause an
861−-- infintie loop. The following example checks the strictness:
838+-- Note: This implementation is not lazy in the second argument. The following
839+-- will fail:
862840--
863−-- >>> p = Parser.satisfy (> 0) `Parser.alt` undefined
864−-- >>> Stream.parse p $ Stream.fromList [1..10]
841+-- >> Stream.parse (Parser.satisfy (> 0) `Parser.alt` undefined) $ Stream.fromList [1..10]
865842-- *** Exception: Prelude.undefined
866843--
867−-- CAVEAT 2: QUADRATIC TIME COMPLEXITY. Static composition is fast due to
868−-- stream fusion, but it works well only for limited (e.g. up to 8)
869−-- compositions, use "Streamly.Data.ParserK" for larger compositions.
844+-- Although this implementation allows stream fusion, it has quadratic
845+-- complexity, making it suitable only for a small number of compositions.
846+-- As a thumb rule use it for less than 8 compositions, use ParserK otherwise.
870847--
871848-- /Time Complexity:/ O(n^2) where n is the number of compositions.
872849--
@@ -947,7 +924,7 @@ data Fused3 a b c = Fused3 !a !b !c
947924--
948925{-# INLINE splitMany #-}
949926splitMany :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
950−splitMany (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
927+splitMany (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
951928 Parser step initial extract
952929
953930 where
@@ -963,7 +940,7 @@ splitMany (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
963940 IPartial ps -> return $ partial $ Fused3 ps 0 fs
964941 IDone pb ->
965942 runCollectorWith (handleCollect partial done) fs pb
966− IError _ -> done <$> ffinal fs
943+ IError _ -> done <$> fextract fs
967944 FL.Done fb -> return $ done fb
968945
969946 runCollectorWith cont fs pb = fstep fs pb >>= cont
@@ -987,19 +964,19 @@ splitMany (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
987964 assertM(cnt1 - n >= 0)
988965 fstep fs b >>= handleCollect (Partial n) (Done n)
989966 Error _ -> do
990− xs <- ffinal fs
967+ xs <- fextract fs
991968 return $ Done cnt xs
992969
993− extract (Fused3 _ 0 fs) = fmap (Done 0) (ffinal fs)
970+ extract (Fused3 _ 0 fs) = fmap (Done 0) (fextract fs)
994971 extract (Fused3 s cnt fs) = do
995972 r <- extract1 s
996973 case r of
997− Error _ -> fmap (Done cnt) (ffinal fs)
974+ Error _ -> fmap (Done cnt) (fextract fs)
998975 Done n b -> do
999976 assertM(n <= cnt)
1000977 fs1 <- fstep fs b
1001978 case fs1 of
1002− FL.Partial s1 -> fmap (Done n) (ffinal s1)
979+ FL.Partial s1 -> fmap (Done n) (fextract s1)
1003980 FL.Done b1 -> return (Done n b1)
1004981 Partial _ _ -> error "splitMany: Partial in extract"
1005982 Continue n s1 -> do
@@ -1013,7 +990,7 @@ splitMany (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
1013990--
1014991{-# INLINE splitManyPost #-}
1015992splitManyPost :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
1016−splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
993+splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
1017994 Parser step initial extract
1018995
1019996 where
@@ -1029,7 +1006,7 @@ splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
10291006 IPartial ps -> return $ partial $ Fused3 ps 0 fs
10301007 IDone pb ->
10311008 runCollectorWith (handleCollect partial done) fs pb
1032− IError _ -> done <$> ffinal fs
1009+ IError _ -> done <$> fextract fs
10331010 FL.Done fb -> return $ done fb
10341011
10351012 runCollectorWith cont fs pb = fstep fs pb >>= cont
@@ -1051,18 +1028,18 @@ splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
10511028 assertM(cnt1 - n >= 0)
10521029 fstep fs b >>= handleCollect (Partial n) (Done n)
10531030 Error _ -> do
1054− xs <- ffinal fs
1031+ xs <- fextract fs
10551032 return $ Done cnt1 xs
10561033
10571034 extract (Fused3 s cnt fs) = do
10581035 r <- extract1 s
10591036 case r of
1060− Error _ -> fmap (Done cnt) (ffinal fs)
1037+ Error _ -> fmap (Done cnt) (fextract fs)
10611038 Done n b -> do
10621039 assertM(n <= cnt)
10631040 fs1 <- fstep fs b
10641041 case fs1 of
1065− FL.Partial s1 -> fmap (Done n) (ffinal s1)
1042+ FL.Partial s1 -> fmap (Done n) (fextract s1)
10661043 FL.Done b1 -> return (Done n b1)
10671044 Partial _ _ -> error "splitMany: Partial in extract"
10681045 Continue n s1 -> do
@@ -1075,7 +1052,7 @@ splitManyPost (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
10751052--
10761053{-# INLINE splitSome #-}
10771054splitSome :: Monad m => Parser a m b -> Fold m b c -> Parser a m c
1078−splitSome (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
1055+splitSome (Parser step1 initial1 extract1) (Fold fstep finitial fextract) =
10791056 Parser step initial extract
10801057
10811058 where
@@ -1091,7 +1068,7 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
10911068 IPartial ps -> return $ partial $ Fused3 ps 0 $ Right fs
10921069 IDone pb ->
10931070 runCollectorWith (handleCollect partial done) fs pb
1094− IError _ -> done <$> ffinal fs
1071+ IError _ -> done <$> fextract fs
10951072 FL.Done fb -> return $ done fb
10961073
10971074 runCollectorWith cont fs pb = fstep fs pb >>= cont
@@ -1141,7 +1118,7 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
11411118 Done n b -> do
11421119 assertM(cnt1 - n >= 0)
11431120 fstep fs b >>= handleCollect (Partial n) (Done n)
1144− Error _ -> Done cnt1 <$> ffinal fs
1121+ Error _ -> Done cnt1 <$> fextract fs
11451122
11461123 extract (Fused3 s cnt (Left fs)) = do
11471124 r <- extract1 s
@@ -1151,7 +1128,7 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
11511128 assertM(n <= cnt)
11521129 fs1 <- fstep fs b
11531130 case fs1 of
1154− FL.Partial s1 -> fmap (Done n) (ffinal s1)
1131+ FL.Partial s1 -> fmap (Done n) (fextract s1)
11551132 FL.Done b1 -> return (Done n b1)
11561133 Partial _ _ -> error "splitSome: Partial in extract"
11571134 Continue n s1 -> do
@@ -1160,12 +1137,12 @@ splitSome (Parser step1 initial1 extract1) (Fold fstep finitial _ ffinal) =
11601137 extract (Fused3 s cnt (Right fs)) = do
11611138 r <- extract1 s
11621139 case r of
1163− Error _ -> fmap (Done cnt) (ffinal fs)
1140+ Error _ -> fmap (Done cnt) (fextract fs)
11641141 Done n b -> do
11651142 assertM(n <= cnt)
11661143 fs1 <- fstep fs b
11671144 case fs1 of
1168− FL.Partial s1 -> fmap (Done n) (ffinal s1)
1145+ FL.Partial s1 -> fmap (Done n) (fextract s1)
11691146 FL.Done b1 -> return (Done n b1)
11701147 Partial _ _ -> error "splitSome: Partial in extract"
11711148 Continue n s1 -> do
@@ -1196,12 +1173,23 @@ dieM err = Parser undefined (IError <$> err) undefined
11961173-- Note: With the direct style parser type, the list in "some" and "many" is
11971174-- accumulated strictly, it cannot be consumed lazily.
11981175
1199−-- | READ THE CAVEATS in 'alt' before using this instance.
1176+-- | Sequential alternative. The input is first passed to the first parser, and
1177+-- if it succeeds, the result is returned. However, if the first parser fails,
1178+-- the parser driver backtracks and tries the same input on the second parser,
1179+-- returning the result if it succeeds.
1180+--
1181+-- Note: The implementation of '<|>' is not lazy in the second
1182+-- argument. The following code will fail:
1183+--
1184+-- >>> Stream.parse (Parser.satisfy (> 0) <|> undefined) $ Stream.fromList [1..10]
1185+-- *** Exception: Prelude.undefined
1186+-- ...
12001187--
1201−-- >>> empty = Parser.die "empty"
1202−-- >>> (<|>) = Parser.alt
1203−-- >>> many = flip Parser.many Fold.toList
1204−-- >>> some = flip Parser.some Fold.toList
1188+-- WARNING! this is not suitable for large scale use. As a thumb rule stream
1189+-- fusion works well for less than 8 compositions of this operation, otherwise
1190+-- consider using 'ParserK'. Do not use recursive parser implementations based
1191+-- on this Alternative instance.
1192+
12051193instance Monad m => Alternative (Parser a m) where
12061194 {-# INLINE empty #-}
12071195 empty = die "empty"
@@ -1220,13 +1208,8 @@ data ConcatParseState sl m a b =
12201208 ConcatParseL !sl
12211209 | forall s. ConcatParseR (s -> a -> m (Step s b)) s (s -> m (Step s b))
12221210
1223−-- XXX Does it fuse completely? Check and update, it cannot fuse the
1224−-- dynamically generated parser.
1225−
12261211-- | Map a 'Parser' returning function on the result of a 'Parser'.
12271212--
1228−-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
1229−--
12301213-- /Pre-release/
12311214--
12321215{-# INLINE concatMap #-}
@@ -1296,12 +1279,6 @@ concatMap func (Parser stepL initialL extractL) = Parser step initial extract
12961279 Partial _ _ -> error "concatMap: extract Partial"
12971280 Continue n s -> return $ Continue n (ConcatParseL s)
12981281
1299−-- | Better performance 'concatMap' for non-failing parsers.
1300−--
1301−-- Does not work correctly for parsers that can fail.
1302−--
1303−-- ALL THE CAVEATS IN 'splitWith' APPLY HERE AS WELL.
1304−--
13051282{-# INLINE noErrorUnsafeConcatMap #-}
13061283noErrorUnsafeConcatMap :: Monad m =>
13071284 (b -> Parser a m c) -> Parser a m b -> Parser a m c
@@ -1374,9 +1351,11 @@ noErrorUnsafeConcatMap func (Parser stepL initialL extractL) =
13741351-- for small number of compositions but for a scalable implementation we need a
13751352-- CPS version.
13761353
1377−-- | READ THE CAVEATS in 'concatMap' before using this instance.
1354+-- | See documentation of 'Streamly.Internal.Data.Parser.ParserK.Type.Parser'.
13781355--
1379−-- >>> (>>=) = flip Parser.concatMap
1356+-- Although this implementation allows stream fusion, it has quadratic
1357+-- complexity, making it suitable only for a small number of compositions. As a
1358+-- thumb rule use it for less than 8 compositions, use 'ParserK' otherwise.
13801359--
13811360instance Monad m => Monad (Parser a m) where
13821361 {-# INLINE return #-}
@@ -1388,7 +1367,6 @@ instance Monad m => Monad (Parser a m) where
13881367 {-# INLINE (>>) #-}
13891368 (>>) = (*>)
13901369
1391−-- | >>> fail = Parser.die
13921370instance Monad m => Fail.MonadFail (Parser a m) where
13931371 {-# INLINE fail #-}
13941372 fail = die
@@ -1404,7 +1382,6 @@ instance Monad m => MonadPlus (Parser a m) where
14041382 mplus = alt
14051383-}
14061384
1407−-- | >>> liftIO = Parser.fromEffect . liftIO
14081385instance (MonadIO m) => MonadIO (Parser a m) where
14091386 {-# INLINE liftIO #-}
14101387 liftIO = fromEffect . liftIO
core/src/Streamly/Internal/Data/ParserK/Type.hs+65−28
@@ -41,10 +41,7 @@ where
4141#include "assert.hs"
4242#include "inline.hs"
4343
44−#if !MIN_VERSION_base(4,18,0)
45−import Control.Applicative (liftA2)
46−#endif
47−import Control.Applicative (Alternative(..))
44+import Control.Applicative (Alternative(..), liftA2)
4845import Control.Monad (MonadPlus(..), ap)
4946import Control.Monad.IO.Class (MonadIO, liftIO)
5047-- import Control.Monad.Trans.Class (MonadTrans(lift))
@@ -168,8 +165,8 @@ newtype ParserK a m b = MkParser
168165-------------------------------------------------------------------------------
169166
170167-- XXX rewrite this using ParserD, expose rmapM from ParserD.
171−
172−-- | Map a function on the result i.e. on @b@ in @Parser a m b@.
168+-- | Maps a function over the output of the parser.
169+--
173170instance Functor m => Functor (ParserK a m) where
174171 {-# INLINE fmap #-}
175172 fmap f parser = MkParser $ \k n st arr ->
@@ -181,7 +178,7 @@ instance Functor m => Functor (ParserK a m) where
181178-------------------------------------------------------------------------------
182179
183180-- This is the dual of stream "fromPure".
184−
181+--
185182-- | A parser that always yields a pure value without consuming any input.
186183--
187184-- /Pre-release/
@@ -199,11 +196,9 @@ fromEffect :: Monad m => m b -> ParserK a m b
199196fromEffect eff =
200197 MkParser $ \k n st arr -> eff >>= \b -> k (Success n b) st arr
201198
202−-- | @f \<$> p1 \<*> p2@ applies parsers p1 and p2 sequentially to an input
203−-- stream. The first parser runs and processes the input, the remaining input
204−-- is then passed to the second parser. If both parsers succeed, their outputs
205−-- are applied to the function @f@. If either parser fails, the operation
206−-- fails.
199+-- | 'Applicative' form of 'Streamly.Internal.Data.Parser.splitWith'. Note that
200+-- this operation does not fuse, use 'Streamly.Internal.Data.Parser.splitWith'
201+-- when fusion is important.
207202--
208203instance Monad m => Applicative (ParserK a m) where
209204 {-# INLINE pure #-}
@@ -245,8 +240,48 @@ instance Monad m => Applicative (ParserK a m) where
245240die :: String -> ParserK a m b
246241die err = MkParser (\k n st arr -> k (Failure n err) st arr)
247242
248−-- | Monad composition can be used for lookbehind parsers, we can dynamically
249−-- compose new parsers based on the results of the previously parsed values.
243+-- | Monad composition can be used for lookbehind parsers, we can make the
244+-- future parses depend on the previously parsed values.
245+--
246+-- If we have to parse "a9" or "9a" but not "99" or "aa" we can use the
247+-- following parser:
248+--
249+-- @
250+-- backtracking :: MonadCatch m => PR.Parser Char m String
251+-- backtracking =
252+-- sequence [PR.satisfy isDigit, PR.satisfy isAlpha]
253+-- '<|>'
254+-- sequence [PR.satisfy isAlpha, PR.satisfy isDigit]
255+-- @
256+--
257+-- We know that if the first parse resulted in a digit at the first place then
258+-- the second parse is going to fail. However, we waste that information and
259+-- parse the first character again in the second parse only to know that it is
260+-- not an alphabetic char. By using lookbehind in a 'Monad' composition we can
261+-- avoid redundant work:
262+--
263+-- @
264+-- data DigitOrAlpha = Digit Char | Alpha Char
265+--
266+-- lookbehind :: MonadCatch m => PR.Parser Char m String
267+-- lookbehind = do
268+-- x1 \<- Digit '<$>' PR.satisfy isDigit
269+-- '<|>' Alpha '<$>' PR.satisfy isAlpha
270+--
271+-- -- Note: the parse depends on what we parsed already
272+-- x2 <- case x1 of
273+-- Digit _ -> PR.satisfy isAlpha
274+-- Alpha _ -> PR.satisfy isDigit
275+--
276+-- return $ case x1 of
277+-- Digit x -> [x,x2]
278+-- Alpha x -> [x,x2]
279+-- @
280+--
281+-- See also 'Streamly.Internal.Data.Parser.concatMap'. This monad instance
282+-- does not fuse, use 'Streamly.Internal.Data.Parser.concatMap' when you need
283+-- fusion.
284+--
250285instance Monad m => Monad (ParserK a m) where
251286 {-# INLINE return #-}
252287 return = pure
@@ -278,10 +313,18 @@ instance MonadIO m => MonadIO (ParserK a m) where
278313-- Alternative
279314-------------------------------------------------------------------------------
280315
281−-- | @p1 \<|> p2@ passes the input to parser p1, if it succeeds, the result is
282−-- returned. However, if p1 fails, the parser driver backtracks and tries the
283−-- same input on the alternative parser p2, returning the result if it
284−-- succeeds.
316+-- | 'Alternative' form of 'Streamly.Internal.Data.Parser.alt'. Backtrack and
317+-- run the second parser if the first one fails.
318+--
319+-- The "some" and "many" operations of alternative accumulate results in a pure
320+-- list which is not scalable and streaming. Instead use
321+-- 'Streamly.Internal.Data.Parser.some' and
322+-- 'Streamly.Internal.Data.Parser.many' for fusible operations with composable
323+-- accumulation of results.
324+--
325+-- See also 'Streamly.Internal.Data.Parser.alt'. This 'Alternative' instance
326+-- does not fuse, use 'Streamly.Internal.Data.Parser.alt' when you need
327+-- fusion.
285328--
286329instance Monad m => Alternative (ParserK a m) where
287330 {-# INLINE empty #-}
@@ -404,7 +447,7 @@ adaptCWith pstep initial extract cont !offset0 !usedCount !input = do
404447 go !_ !cur !pst | cur >= end =
405448 onContinue ((end - start) `div` SIZE_OF(a)) pst
406449 go !_ !cur !pst = do
407− let !x = unsafeInlineIO $ peekAt cur contents
450+ let !x = unsafeInlineIO $ peekByteIndex cur contents
408451 pRes <- pstep pst x
409452 let elemSize = SIZE_OF(a)
410453 next = INDEX_NEXT(cur,a)
@@ -461,8 +504,7 @@ adaptCWith pstep initial extract cont !offset0 !usedCount !input = do
461504 parseCont !cnt !pst (Chunk arr) = parseContChunk cnt 0 pst arr
462505 parseCont !cnt !pst None = parseContNothing cnt pst
463506
464−-- | Convert an element 'Parser' to a chunked 'ParserK'. A chunked parser is
465−-- more efficient than an element parser.
507+-- | Convert a raw byte 'Parser' to a chunked 'ParserK'.
466508--
467509-- /Pre-release/
468510--
@@ -564,9 +606,7 @@ adaptWith pstep initial extract cont !relPos !usedCount !input = do
564606 parseCont !cnt !pst (Chunk arr) = parseContChunk cnt pst arr
565607 parseCont !cnt !pst None = parseContNothing cnt pst
566608
567−-- | Convert a 'Parser' to 'ParserK'.
568−--
569−-- /Pre-release/
609+-- | Similar to "adaptC" but for non-chunked input.
570610--
571611{-# INLINE_LATE adapt #-}
572612adapt :: Monad m => ParserD.Parser a m b -> ParserK a m b
@@ -694,10 +734,7 @@ adaptCGWith pstep initial extract cont !offset0 !usedCount !input = do
694734 parseCont !cnt !pst (Chunk arr) = parseContChunk cnt 0 pst arr
695735 parseCont !cnt !pst None = parseContNothing cnt pst
696736
697−-- | A generic 'adaptC'. Similar to 'adaptC' but is not constrained to 'Unbox'
698−-- types.
699−--
700−-- /Pre-release/
737+-- | Similar to "adaptC" but is not constrained.
701738--
702739{-# INLINE_LATE adaptCG #-}
703740adaptCG ::
core/src/Streamly/Internal/Data/Ring.hs+4−7
@@ -79,7 +79,7 @@ import Foreign.Storable
7979import Foreign.ForeignPtr (ForeignPtr, withForeignPtr, touchForeignPtr)
8080import Foreign.ForeignPtr.Unsafe (unsafeForeignPtrToPtr)
8181import Foreign.Ptr (plusPtr, minusPtr, castPtr)
82−import Streamly.Internal.Data.Unbox as Unboxed (Unbox(peekAt))
82+import Streamly.Internal.Data.Unbox as Unboxed (Unbox(peekByteIndex))
8383import GHC.ForeignPtr (mallocPlainForeignPtrAlignedBytes)
8484import GHC.Ptr (Ptr(..))
8585import Streamly.Internal.Data.MutArray.Type (MutArray)
@@ -411,7 +411,7 @@ unsafeEqArrayN Ring{..} rh A.Array{..} nBytes
411411
412412 check p i = do
413413 (relem :: Word8) <- peek p
414− aelem <- peekAt i w8Contents
414+ aelem <- peekByteIndex i w8Contents
415415 if relem == aelem
416416 then go (p `plusPtr` 1) (i + 1)
417417 else return False
@@ -445,7 +445,7 @@ unsafeEqArray Ring{..} rh A.Array{..} =
445445
446446 check p i = do
447447 (relem :: Word8) <- peek p
448− aelem <- peekAt i w8Contents
448+ aelem <- peekByteIndex i w8Contents
449449 if relem == aelem
450450 then go (p `plusPtr` 1) (i + 1)
451451 else return False
@@ -558,8 +558,7 @@ data Tuple4' a b c d = Tuple4' !a !b !c !d deriving Show
558558{-# INLINE slidingWindowWith #-}
559559slidingWindowWith :: forall m a b. (MonadIO m, Storable a, Unbox a)
560560 => Int -> Fold m ((a, Maybe a), m (MutArray a)) b -> Fold m a b
561−slidingWindowWith n (Fold step1 initial1 extract1 final1) =
562− Fold step initial extract final
561+slidingWindowWith n (Fold step1 initial1 extract1) = Fold step initial extract
563562
564563 where
565564
@@ -602,8 +601,6 @@ slidingWindowWith n (Fold step1 initial1 extract1 final1) =
602601
603602 extract (Tuple4' _ _ _ st) = extract1 st
604603
605− final (Tuple4' _ _ _ st) = final1 st
606−
607604-- | @slidingWindow collector@ is an incremental sliding window
608605-- fold that does not require all the intermediate elements in a computation.
609606-- This maintains @n@ elements in the window, when a new element comes it slides
core/src/Streamly/Internal/Data/Ring/Generic.hs+1−1
@@ -69,7 +69,7 @@ createRing count = liftIO $ do
6969-- the Ring.
7070{-# INLINE writeLastN #-}
7171writeLastN :: MonadIO m => Int -> Fold m a (Ring a)
72−writeLastN n = Fold step initial extract extract
72+writeLastN n = Fold step initial extract
7373
7474 where
7575
core/src/Streamly/Internal/Data/Serialize.hsnew file+122
@@ -0,0 +1,122 @@
1+{-# LANGUAGE TemplateHaskell #-}
2+
3+-- This is required as all the instances in this module are orphan instances.
4+{-# OPTIONS_GHC -fno-warn-orphans #-}
5+
6+-- |
7+-- Module : Streamly.Internal.Data.Serialize
8+-- Copyright : (c) 2023 Composewell Technologies
9+-- License : BSD3-3-Clause
10+-- Maintainer : streamly@composewell.com
11+-- Portability : GHC
12+--
13+
14+module Streamly.Internal.Data.Serialize
15+ ( module Streamly.Internal.Data.Serialize.Type
16+ ) where
17+
18+--------------------------------------------------------------------------------
19+-- Imports
20+--------------------------------------------------------------------------------
21+
22+import Data.Proxy (Proxy)
23+import Streamly.Internal.Data.Array (Array(..))
24+import Streamly.Internal.Data.Unbox (MutableByteArray(..))
25+import GHC.Exts (Int(..), sizeofByteArray#, unsafeCoerce#)
26+
27+import qualified Streamly.Internal.Data.Serialize.TH as Serialize
28+
29+#if __GLASGOW_HASKELL__ >= 900
30+import GHC.Num.Integer (Integer(..))
31+#else
32+import GHC.Integer.GMP.Internals (Integer(..), BigNat(..))
33+#endif
34+
35+import Streamly.Internal.Data.Serialize.Type
36+
37+--------------------------------------------------------------------------------
38+-- Common instances
39+--------------------------------------------------------------------------------
40+
41+-- Note
42+-- ====
43+--
44+-- Even a non-functional change such as changing the order of constructors will
45+-- change the instance derivation.
46+--
47+-- This will not pose a problem if both, encode, and decode are done by the same
48+-- version of the application. There *might* be a problem if version that
49+-- encodes differs from the version that decodes.
50+--
51+-- We need to add some compatibility tests using different versions of
52+-- dependencies.
53+--
54+-- Although such chages for the most basic types won't happen we need to detect
55+-- if it ever happens.
56+--
57+-- Should we worry about these kind of changes and this kind of compatibility?
58+-- This is a problem for all types of derivations that depend on the order of
59+-- constructors, for example, Enum.
60+
61+$(Serialize.deriveSerialize ''Maybe)
62+$(Serialize.deriveSerialize ''Either)
63+$(Serialize.deriveSerializeWith
64+ Serialize.defaultConfig
65+ [d|instance Serialize (Proxy a)|])
66+
67+--------------------------------------------------------------------------------
68+-- Integer
69+--------------------------------------------------------------------------------
70+
71+data LiftedInteger
72+ = LIS Int
73+ | LIP (Array Word)
74+ | LIN (Array Word)
75+
76+$(Serialize.deriveSerialize ''LiftedInteger)
77+
78+#if __GLASGOW_HASKELL__ >= 900
79+
80+{-# INLINE liftInteger #-}
81+liftInteger :: Integer -> LiftedInteger
82+liftInteger (IS x) = LIS (I# x)
83+liftInteger (IP x) =
84+ LIP (Array (MutableByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
85+liftInteger (IN x) =
86+ LIN (Array (MutableByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
87+
88+{-# INLINE unliftInteger #-}
89+unliftInteger :: LiftedInteger -> Integer
90+unliftInteger (LIS (I# x)) = IS x
91+unliftInteger (LIP (Array (MutableByteArray x) _ _)) = IP (unsafeCoerce# x)
92+unliftInteger (LIN (Array (MutableByteArray x) _ _)) = IN (unsafeCoerce# x)
93+
94+#else
95+
96+{-# INLINE liftInteger #-}
97+liftInteger :: Integer -> LiftedInteger
98+liftInteger (S# x) = LIS (I# x)
99+liftInteger (Jp# (BN# x)) =
100+ LIP (Array (MutableByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
101+liftInteger (Jn# (BN# x)) =
102+ LIN (Array (MutableByteArray (unsafeCoerce# x)) 0 (I# (sizeofByteArray# x)))
103+
104+{-# INLINE unliftInteger #-}
105+unliftInteger :: LiftedInteger -> Integer
106+unliftInteger (LIS (I# x)) = S# x
107+unliftInteger (LIP (Array (MutableByteArray x) _ _)) =
108+ Jp# (BN# (unsafeCoerce# x))
109+unliftInteger (LIN (Array (MutableByteArray x) _ _)) =
110+ Jn# (BN# (unsafeCoerce# x))
111+
112+#endif
113+
114+instance Serialize Integer where
115+ {-# INLINE size #-}
116+ size i a = size i (liftInteger a)
117+
118+ {-# INLINE deserialize #-}
119+ deserialize off arr end = fmap unliftInteger <$> deserialize off arr end
120+
121+ {-# INLINE serialize #-}
122+ serialize off arr val = serialize off arr (liftInteger val)
core/src/Streamly/Internal/Data/Serialize/TH.hs+111−118
@@ -7,23 +7,12 @@
77-- Maintainer : streamly@composewell.com
88-- Stability : experimental
99-- Portability : GHC
10−
11−-- XXX Instead of applying the product constructor in one go can we apply it
12−-- one at a time in case of too many arguments, compiler may not have to save
13−-- them in local vars.
14−
10+--
1511module Streamly.Internal.Data.Serialize.TH
16− (
17− -- Deriving
18− deriveSerialize
12+ ( deriveSerialize
13+ , Config(..)
14+ , defaultConfig
1915 , deriveSerializeWith
20−
21− -- Utilities
22− , module Streamly.Internal.Data.Serialize.TH.Bottom
23− -- ** Common
24− , module Streamly.Internal.Data.Serialize.TH.Common
25− -- ** RecHeader
26− , module Streamly.Internal.Data.Serialize.TH.RecHeader
2716 ) where
2817
2918--------------------------------------------------------------------------------
@@ -40,6 +29,7 @@ import Streamly.Internal.Data.Serialize.Type
4029import Streamly.Internal.Data.Unbox.TH
4130 ( DataCon(..)
4231 , DataType(..)
32+ , appsT
4333 , reifyDataType
4434 )
4535
@@ -47,14 +37,13 @@ import qualified Streamly.Internal.Data.Serialize.TH.RecHeader as RecHeader
4737
4838import Streamly.Internal.Data.Serialize.TH.Bottom
4939import Streamly.Internal.Data.Serialize.TH.Common
50−import Streamly.Internal.Data.Serialize.TH.RecHeader
5140
5241--------------------------------------------------------------------------------
5342-- Domain specific helpers
5443--------------------------------------------------------------------------------
5544
5645exprGetSize :: Q Exp -> (Int, Type) -> Q Exp
57−exprGetSize acc (i, _) = [|addSizeTo $(acc) $(varE (mkFieldName i))|]
46+exprGetSize acc (i, _) = [|size $(acc) $(varE (mkFieldName i))|]
5847
5948getTagSize :: Int -> Int
6049getTagSize numConstructors
@@ -159,25 +148,22 @@ mkSizeOfExpr False False tyOfTy =
159148 [varP _acc, varP _x]
160149 (caseE (varE _x) (fmap (matchCons acc) cons))
161150
162−mkSizeOfExpr False True (TheType con) = RecHeader.mkRecSizeOfExpr con
151+mkSizeOfExpr False True (TheType con) = RecHeader.mkSizeOfExpr con
163152
164153mkSizeOfExpr _ _ _ = errorUnimplemented
165154
166−mkSizeDec :: SerializeConfig -> Type -> [DataCon] -> Q [Dec]
167−mkSizeDec (SerializeConfig {..}) headTy cons = do
155+mkSizeDec :: Config -> Type -> [DataCon] -> Q [Dec]
156+mkSizeDec (Config {..}) headTy cons = do
168157 -- INLINE on sizeOf actually worsens some benchmarks, and improves none
169158 sizeOfMethod <-
170159 mkSizeOfExpr
171− cfgConstructorTagAsString
172− cfgRecordSyntaxWithHeader
160+ constructorTagAsString
161+ recordSyntaxWithHeader
173162 (typeOfType headTy cons)
174163 pure
175− ( maybe
176− []
177− (\x -> [PragmaD (InlineP 'addSizeTo x FunLike AllPhases)])
178− cfgInlineSize
179− ++ [FunD 'addSizeTo [Clause [] (NormalB sizeOfMethod) []]]
180− )
164+ [ PragmaD (InlineP 'size inlineSize FunLike AllPhases)
165+ , FunD 'size [Clause [] (NormalB sizeOfMethod) []]
166+ ]
181167
182168--------------------------------------------------------------------------------
183169-- Peek
@@ -196,7 +182,7 @@ mkDeserializeExpr True False headTy tyOfTy =
196182 conLen <- newName "conLen"
197183 off1 <- newName "off1"
198184 [|do ($(varP off1), $(varP conLen) :: Word8) <-
199− deserializeAt
185+ deserialize
200186 $(varE _initialOffset)
201187 $(varE _arr)
202188 $(varE _endOffset)
@@ -216,19 +202,18 @@ mkDeserializeExpr True False headTy tyOfTy =
216202 [|($(litIntegral lenCname) == $(varE conLen))
217203 && $(xorCmp tag off _arr)|]
218204 [|let $(varP (makeI 0)) = $(varE off) + $(litIntegral lenCname)
219− in $(mkDeserializeExprOne 'deserializeAt con)|]
205+ in $(mkDeserializeExprOne 'deserialize con)|]
220206
221207mkDeserializeExpr False False headTy tyOfTy =
222208 case tyOfTy of
223209 -- Unit constructor
224− -- XXX Should we peek and check if the byte value is 0?
225210 UnitType cname ->
226211 [|pure ($(varE _initialOffset) + 1, $(conE cname))|]
227212 -- Product type
228213 TheType con ->
229214 letE
230215 [valD (varP (mkName "i0")) (normalB (varE _initialOffset)) []]
231− (mkDeserializeExprOne 'deserializeAt con)
216+ (mkDeserializeExprOne 'deserialize con)
232217 -- Sum type
233218 MultiType cons -> do
234219 let lenCons = length cons
@@ -236,7 +221,7 @@ mkDeserializeExpr False False headTy tyOfTy =
236221 doE
237222 [ bindS
238223 (tupP [varP (mkName "i0"), varP _tag])
239− [|deserializeAt $(varE _initialOffset) $(varE _arr) $(varE _endOffset)|]
224+ [|deserialize $(varE _initialOffset) $(varE _arr) $(varE _endOffset)|]
240225 , noBindS
241226 (caseE
242227 (sigE (varE _tag) (conT tagType))
@@ -246,7 +231,7 @@ mkDeserializeExpr False False headTy tyOfTy =
246231 peekMatch (i, con) =
247232 match
248233 (litP (IntegerL i))
249− (normalB (mkDeserializeExprOne 'deserializeAt con)) []
234+ (normalB (mkDeserializeExprOne 'deserialize con)) []
250235 peekErr =
251236 match
252237 wildP
@@ -257,15 +242,25 @@ mkDeserializeExpr False False headTy tyOfTy =
257242 $(lift (pprint headTy)) ++ ")")|])
258243 []
259244
260−mkDeserializeExpr False True _ (TheType con@(SimpleDataCon _ fields)) = do
245+mkDeserializeExpr False True headTy (TheType con@(SimpleDataCon _ fields)) = do
261246 deserializeWithKeys <- newName "deserializeWithKeys"
262247 updateFunc <- newName "updateFunc"
263− updateFuncDec <- RecHeader.conUpdateFuncDec updateFunc fields
264− deserializeWithKeysDec <-
265− RecHeader.mkDeserializeKeysDec deserializeWithKeys updateFunc con
248+ hOff <- newName "hOff"
249+ finalOff <- newName "finalOff"
250+ deserializeWithKeysMethod <-
251+ RecHeader.mkDeserializeKeysExpr
252+ hOff finalOff _arr _endOffset updateFunc con
253+ updateFuncDec <- RecHeader.conUpdateFuncExpr updateFunc headTy fields
266254 letE
267− (pure <$> (deserializeWithKeysDec ++ updateFuncDec))
268− (RecHeader.mkRecDeserializeExpr
255+ ([ pure $ FunD
256+ deserializeWithKeys
257+ [ Clause
258+ [VarP hOff, VarP finalOff]
259+ (NormalB deserializeWithKeysMethod)
260+ []
261+ ]
262+ ] ++ (pure <$> updateFuncDec))
263+ (RecHeader.mkDeserializeExpr
269264 _initialOffset
270265 _endOffset
271266 deserializeWithKeys
@@ -273,31 +268,26 @@ mkDeserializeExpr False True _ (TheType con@(SimpleDataCon _ fields)) = do
273268
274269mkDeserializeExpr _ _ _ _ = errorUnimplemented
275270
276−mkDeserializeDec :: SerializeConfig -> Type -> [DataCon] -> Q [Dec]
277−mkDeserializeDec (SerializeConfig {..}) headTy cons = do
271+mkDeserializeDec :: Config -> Type -> [DataCon] -> Q [Dec]
272+mkDeserializeDec (Config {..}) headTy cons = do
278273 peekMethod <-
279274 mkDeserializeExpr
280− cfgConstructorTagAsString
281− cfgRecordSyntaxWithHeader
275+ constructorTagAsString
276+ recordSyntaxWithHeader
282277 headTy
283278 (typeOfType headTy cons)
284279 pure
285− ( maybe
286− []
287− (\x -> [PragmaD (InlineP 'deserializeAt x FunLike AllPhases)])
288− cfgInlineDeserialize
289− ++
290− [ FunD
291− 'deserializeAt
280+ [ PragmaD (InlineP 'deserialize inlineDeserialize FunLike AllPhases)
281+ , FunD
282+ 'deserialize
292283 [ Clause
293− (if isUnitType cons && not cfgConstructorTagAsString
284+ (if isUnitType cons && not constructorTagAsString
294285 then [VarP _initialOffset, WildP, WildP]
295286 else [VarP _initialOffset, VarP _arr, VarP _endOffset])
296287 (NormalB peekMethod)
297288 []
298289 ]
299− ]
300− )
290+ ]
301291
302292--------------------------------------------------------------------------------
303293-- Poke
@@ -305,7 +295,7 @@ mkDeserializeDec (SerializeConfig {..}) headTy cons = do
305295
306296mkSerializeExprTag :: Name -> Int -> Q Exp
307297mkSerializeExprTag tagType tagVal =
308− [|serializeAt
298+ [|serialize
309299 $(varE _initialOffset)
310300 $(varE _arr)
311301 $((sigE (litE (IntegerL (fromIntegral tagVal))) (conT tagType)))|]
@@ -340,14 +330,13 @@ mkSerializeExpr True False tyOfTy =
340330 (doE [ bindS
341331 (varP (mkName "i0"))
342332 (serializeW8List _initialOffset _arr conEnc)
343− , noBindS (mkSerializeExprFields 'serializeAt fields)
333+ , noBindS (mkSerializeExprFields 'serialize fields)
344334 ])
345335
346336mkSerializeExpr False False tyOfTy =
347337 case tyOfTy of
348338 -- Unit type
349− UnitType _ ->
350− [|serializeAt $(varE _initialOffset) $(varE _arr) (0 :: Word8)|]
339+ UnitType _ -> [|pure ($(varE _initialOffset) + 1)|]
351340 -- Product type
352341 (TheType (SimpleDataCon cname fields)) ->
353342 letE
@@ -357,7 +346,7 @@ mkSerializeExpr False False tyOfTy =
357346 [ matchConstructor
358347 cname
359348 (length fields)
360− (mkSerializeExprFields 'serializeAt fields)
349+ (mkSerializeExprFields 'serialize fields)
361350 ])
362351 -- Sum type
363352 (MultiType cons) -> do
@@ -374,40 +363,35 @@ mkSerializeExpr False False tyOfTy =
374363 (mkSerializeExprTag tagType tagVal)
375364 , noBindS
376365 (mkSerializeExprFields
377− 'serializeAt
366+ 'serialize
378367 fields)
379368 ]))
380369 (zip [0 ..] cons))
381370
382371mkSerializeExpr False True (TheType con) =
383− RecHeader.mkRecSerializeExpr _initialOffset con
372+ RecHeader.mkSerializeExpr _initialOffset con
384373
385374mkSerializeExpr _ _ _ = errorUnimplemented
386375
387−mkSerializeDec :: SerializeConfig -> Type -> [DataCon] -> Q [Dec]
388−mkSerializeDec (SerializeConfig {..}) headTy cons = do
376+mkSerializeDec :: Config -> Type -> [DataCon] -> Q [Dec]
377+mkSerializeDec (Config {..}) headTy cons = do
389378 pokeMethod <-
390379 mkSerializeExpr
391− cfgConstructorTagAsString
392− cfgRecordSyntaxWithHeader
380+ constructorTagAsString
381+ recordSyntaxWithHeader
393382 (typeOfType headTy cons)
394383 pure
395− ( maybe
396− []
397− (\x -> [PragmaD (InlineP 'serializeAt x FunLike AllPhases)])
398− cfgInlineSerialize
399− ++
400− [FunD
401− 'serializeAt
402− [ Clause
403− (if isUnitType cons && not cfgConstructorTagAsString
404− then [VarP _initialOffset, VarP _arr, WildP]
405− else [VarP _initialOffset, VarP _arr, VarP _val])
406− (NormalB pokeMethod)
407− []
408− ]
409− ]
410− )
384+ [ PragmaD (InlineP 'serialize inlineSerialize FunLike AllPhases)
385+ , FunD
386+ 'serialize
387+ [ Clause
388+ (if isUnitType cons && not constructorTagAsString
389+ then [VarP _initialOffset, WildP, WildP]
390+ else [VarP _initialOffset, VarP _arr, VarP _val])
391+ (NormalB pokeMethod)
392+ []
393+ ]
394+ ]
411395
412396--------------------------------------------------------------------------------
413397-- Main
@@ -429,7 +413,7 @@ mkSerializeDec (SerializeConfig {..}) headTy cons = do
429413-- Usage:
430414-- @
431415-- $(deriveSerializeInternal
432−-- serializeConfig
416+-- defaultConfig
433417-- [AppT (ConT ''Serialize) (VarT (mkName "b"))]
434418-- (AppT
435419-- (AppT (ConT ''CustomDataType) (VarT (mkName "a")))
@@ -444,7 +428,7 @@ mkSerializeDec (SerializeConfig {..}) headTy cons = do
444428-- ])
445429-- @
446430deriveSerializeInternal ::
447− SerializeConfig -> Type -> [DataCon] -> ([Dec] -> Q [Dec]) -> Q [Dec]
431+ Config -> Type -> [DataCon] -> ([Dec] -> Q [Dec]) -> Q [Dec]
448432deriveSerializeInternal conf headTy cons next = do
449433 sizeDec <- mkSizeDec conf headTy cons
450434 peekDec <- mkDeserializeDec conf headTy cons
@@ -452,41 +436,30 @@ deriveSerializeInternal conf headTy cons next = do
452436 let methods = concat [sizeDec, peekDec, pokeDec]
453437 next methods
454438
455−-- | @deriveSerializeWith config-modifier instance-dec@ generates a template
456−-- Haskell splice consisting of a declaration of a 'Serialize' instance.
457−-- @instance-dec@ is a template Haskell declaration splice consisting of a
458−-- standard Haskell instance declaration without the type class methods (e.g.
459−-- @[d|instance Serialize a => Serialize (Maybe a)|]@).
460−--
461−-- The type class methods for the given instance are generated according to the
462−-- supplied @config-modifier@ parameter. See 'SerializeConfig' for default
463−-- configuration settings.
439+-- | Similar to 'deriveSerialize' but take a 'Config' to control how the
440+-- instance is generated.
464441--
465442-- Usage:
466−--
467443-- @
468−-- \$(deriveSerializeWith
469−-- ( inlineSerializeAt (Just NoInline)
470−-- . inlineDeserializeAt (Just NoInline)
471−-- )
444+-- $(deriveSerializeWith
445+-- config
472446-- [d|instance Serialize a => Serialize (Maybe a)|])
473447-- @
474−deriveSerializeWith ::
475− (SerializeConfig -> SerializeConfig) -> Q [Dec] -> Q [Dec]
476−deriveSerializeWith modifier mDecs = do
448+deriveSerializeWith :: Config -> Q [Dec] -> Q [Dec]
449+deriveSerializeWith conf mDecs = do
477450 dec <- mDecs
478451 case dec of
479− [InstanceD mo preds headTyWC []] -> do
452+ [inst@(InstanceD _ _ headTyWC [])] -> do
480453 let headTy = unwrap dec headTyWC
481454 dt <- reifyDataType (getMainTypeName dec headTy)
482455 let cons = dtCons dt
483− deriveSerializeInternal
484− (modifier serializeConfig) headTy cons (next mo preds headTyWC)
485− _ -> errorMessage dec
486−
487− where
456+ deriveSerializeInternal conf headTy cons (next inst)
457+ _ -> errorUnsupported
458+ where
488459
489− next mo preds headTyWC methods = pure [InstanceD mo preds headTyWC methods]
460+ next (InstanceD mo preds headTyWC []) methods =
461+ pure [InstanceD mo preds headTyWC methods]
462+ next _ _ = errorUnsupported
490463
491464 errorMessage dec =
492465 error $ unlines
@@ -494,8 +467,8 @@ deriveSerializeWith modifier mDecs = do
494467 , ""
495468 , ">> " ++ pprint dec
496469 , ""
497− , "The supplied declaration is not a valid instance declaration."
498− , "Provide a valid Haskell instance declaration without a body."
470+ , "The above is not a valid instance declaration."
471+ , "Any haskell instance declaration without a body is valid."
499472 , ""
500473 , "Examples:"
501474 , "instance Serialize (Proxy a)"
@@ -514,17 +487,37 @@ deriveSerializeWith modifier mDecs = do
514487 go (AppT l _) = go l
515488 go _ = errorMessage dec
516489
517−-- | Given an 'Serialize' instance declaration splice without the methods (e.g.
518−-- @[d|instance Serialize a => Serialize (Maybe a)|]@), generate an instance
519−-- declaration including all the type class method implementations.
490+
491+-- | Template haskell helper to create instances of 'Serialize' automatically.
520492--
521−-- >>> deriveSerialize = deriveSerializeWith id
493+-- Consider the datatype:
494+-- @
495+-- data CustomDataType a b c = ...
496+-- @
522497--
523−-- Usage:
498+-- Usage: @$(deriveSerialize ''CustomDataType)@
524499--
500+-- Note: All type variables automatcally get an "Serialize" constraint.
501+-- The derived code will look like the following,
525502-- @
526−-- \$(deriveSerialize
527−-- [d|instance Serialize a => Serialize (Maybe a)|])
503+-- instance (Serialize a, Serialize b, Serialize c) => Serialize (CustomDataType a b c) where
504+-- ...
528505-- @
529−deriveSerialize :: Q [Dec] -> Q [Dec]
530−deriveSerialize = deriveSerializeWith id
506+--
507+-- To control which type variables don't get the Serialize constraint, use
508+-- 'deriveSerializeWith'.
509+deriveSerialize :: Name -> Q [Dec]
510+deriveSerialize name = do
511+ dt <- reifyDataType name
512+ let preds = map (unboxPred . VarT) (dtTvs dt)
513+ headTy = appsT (ConT name) (map VarT (dtTvs dt))
514+ cons = dtCons dt
515+ deriveSerializeInternal defaultConfig headTy cons (next preds headTy)
516+
517+
518+ where
519+
520+ next preds headTy methods =
521+ pure [InstanceD Nothing preds (AppT (ConT ''Serialize) headTy) methods]
522+
523+ unboxPred ty = AppT (ConT ''Serialize) ty
core/src/Streamly/Internal/Data/Serialize/TH/Bottom.hs+39−153
@@ -9,17 +9,8 @@
99-- Portability : GHC
1010--
1111module Streamly.Internal.Data.Serialize.TH.Bottom
12− (
13− -- ** Config
14− SerializeConfig(..)
15− , serializeConfig
16− , inlineAddSizeTo
17− , inlineSerializeAt
18− , inlineDeserializeAt
19− , encodeConstrNames
20− , encodeRecordFields
21−
22− -- ** Other Utilities
12+ ( Config(..)
13+ , defaultConfig
2314 , TypeOfType(..)
2415 , typeOfType
2516 , SimpleDataCon(..)
@@ -74,134 +65,35 @@ import Streamly.Internal.Data.Unbox.TH (DataCon(..))
7465-- Config
7566--------------------------------------------------------------------------------
7667
77−-- NOTE: 'Nothing' is not eqvivalant to 'Just Inlinable'. Ie. Having no inline
78−-- specific pragma and having an Inlinable pragma are different. Having an
79−-- Inlinable pragma makes GHC put the code in the interface file whereas having
80−-- no inline specific pragma let's GHC decide whether to put the code in
81−-- interface file or not.
82−
83−-- | Configuration to control how the 'Serialize' instance is generated. The
84−-- configuration is opaque and is modified by composing config modifier
85−-- functions, for example:
86−--
87−-- >>> (inlineSerializeAt (Just NoInline)) . (inlineSerializeAt (Just Inlinable))
88−--
89−-- The default configuration settings are:
90−--
91−-- * 'inlineAddSizeTo' Nothing
92−-- * 'inlineSerializeAt' (Just Inline)
93−-- * 'inlineDeserializeAt' (Just Inline)
94−--
95−-- The following experimental options are also available:
96−--
97−-- * 'encodeConstrNames' False
98−-- * 'encodeRecordFields' False
99−--
100−data SerializeConfig =
101− SerializeConfig
102− { cfgInlineSize :: Maybe Inline
103− , cfgInlineSerialize :: Maybe Inline
104− , cfgInlineDeserialize :: Maybe Inline
105− , cfgConstructorTagAsString :: Bool
106− , cfgRecordSyntaxWithHeader :: Bool
68+-- | Config to control how the 'Serialize' instance is generated.
69+data Config =
70+ Config
71+ { inlineSize :: Inline
72+ -- ^ Inline value for 'size'. Default is Inline.
73+ , inlineSerialize :: Inline
74+ -- ^ Inline value for 'serialize'. Default is Inline.
75+ , inlineDeserialize :: Inline
76+ -- ^ Inline value for 'deserialize'. Default is Inline.
77+ , constructorTagAsString :: Bool
78+ -- ^ __Experimental__
79+ --
80+ -- If True, encode constructors using the constructor names as Latin-1
81+ -- byte sequence.
82+ , recordSyntaxWithHeader :: Bool
83+ -- ^ __Experimental__
84+ --
85+ -- If True, encode the keys of the record as a header and then
86+ -- serialize the data.
10787 }
10888
109−-- | How should we inline the 'addSizeTo' function? The default is 'Nothing'
110−-- which means left to the compiler. Forcing inline on @addSizeTo@ function
111−-- actually worsens some benchmarks and improves none.
112−inlineAddSizeTo :: Maybe Inline -> SerializeConfig -> SerializeConfig
113−inlineAddSizeTo v cfg = cfg {cfgInlineSize = v}
114−
115−-- XXX Should we make the default Inlinable instead?
116−
117−-- | How should we inline the 'serialize' function? The default 'Just Inline'.
118−-- However, aggressive inlining can bloat the code and increase in compilation
119−-- times when there are big functions and too many nesting levels so you can
120−-- change it accordingly. A 'Nothing' value leaves the decision to the
121−-- compiler.
122−inlineSerializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
123−inlineSerializeAt v cfg = cfg {cfgInlineSerialize = v}
124−
125−-- XXX Should we make the default Inlinable instead?
126−
127−-- | How should we inline the 'deserialize' function? See guidelines in
128−-- 'inlineSerializeAt'.
129−inlineDeserializeAt :: Maybe Inline -> SerializeConfig -> SerializeConfig
130−inlineDeserializeAt v cfg = cfg {cfgInlineDeserialize = v}
131−
132−-- | __Experimental__
133−--
134−-- In sum types, use Latin-1 encoded original constructor names rather than
135−-- binary values to identify constructors. This option is not applicable to
136−-- product types.
137−--
138−-- This option enables the following behavior:
139−--
140−-- * __Reordering__: Order of the fields can be changed without affecting
141−-- serialization.
142−-- * __Addition__: If a field is added in the new version, the old version of
143−-- the data type can still be deserialized by the new version. The new value
144−-- would never occur in the old one.
145−-- * __Deletion__: If a field is deleted in the new version, deserialization
146−-- of the old version will result in an error. TBD: We can possibly designate a
147−-- catch-all case to handle this scenario.
148−--
149−-- Note that if you change a type, change the semantics of a type, or delete a
150−-- field and add a new field with the same name, deserialization of old data
151−-- may result in silent unexpected behavior.
152−--
153−-- This option has to be the same on both encoding and decoding side.
154−--
155−-- The default is 'False'.
156−--
157−encodeConstrNames :: Bool -> SerializeConfig -> SerializeConfig
158−encodeConstrNames v cfg = cfg {cfgConstructorTagAsString = v}
159−
160−-- XXX We can deserialize each field to Either, so if there is a
161−-- deserialization error in any field it can handled independently. Also, a
162−-- unique type/version identifier of the field (based on the versions of the
163−-- packages, full module name space + type identifier) can be serialized along
164−-- with the value for stricter compatibility, semantics checking. Or we can
165−-- store a type hash.
166−
167−-- | __Experimental__
168−--
169−-- In explicit record types, use Latin-1 encoded record field names rather than
170−-- binary values to identify the record fields. Note that this option is not
171−-- applicable to sum types. Also, it does not work on a product type which is
172−-- not a record, because there are no field names to begin with.
173−--
174−-- This option enables the following behavior:
175−--
176−-- * __Reordering__: Order of the fields can be changed without affecting
177−-- serialization.
178−-- * __Addition__: If a 'Maybe' type field is added in the new version, the old
179−-- version of the data type can still be deserialized by the new version, the
180−-- field value in the older version is assumed to be 'Nothing'. If any other
181−-- type of field is added, deserialization of the older version results in an
182−-- error but only when that field is actually accessed in the deserialized
183−-- record.
184−-- * __Deletion__: If a field is deleted in the new version and it is
185−-- encountered in a previously serialized version then the field is discarded.
186−--
187−-- This option has to be the same on both encoding and decoding side.
188−--
189−-- There is a constant performance overhead proportional to the total length of
190−-- the record field names and the number of record fields.
191−--
192−-- The default is 'False'.
193−--
194−encodeRecordFields :: Bool -> SerializeConfig -> SerializeConfig
195−encodeRecordFields v cfg = cfg {cfgRecordSyntaxWithHeader = v}
196−
197−serializeConfig :: SerializeConfig
198−serializeConfig =
199− SerializeConfig
200− { cfgInlineSize = Nothing
201− , cfgInlineSerialize = Just Inline
202− , cfgInlineDeserialize = Just Inline
203− , cfgConstructorTagAsString = False
204− , cfgRecordSyntaxWithHeader = False
89+defaultConfig :: Config
90+defaultConfig =
91+ Config
92+ { inlineSize = Inline
93+ , inlineSerialize = Inline
94+ , inlineDeserialize = Inline
95+ , constructorTagAsString = False
96+ , recordSyntaxWithHeader = False
20597 }
20698
20799--------------------------------------------------------------------------------
@@ -347,7 +239,7 @@ xorCmp tag off arr =
347239 go8 i = do
348240 let wIntegral = litIntegral i
349241 [|xor (unsafeInlineIO
350− (Unbox.peekAt
242+ (Unbox.peekByteIndex
351243 ($(varE off) + $(litIntegral i))
352244 $(varE arr)))
353245 ($(wIntegral) :: Word8) .|.
@@ -361,7 +253,7 @@ xorCmp tag off arr =
361253 litIntegral
362254 (shiftAdd w8_w16 [tag !! i, tag !! (i + 1)] :: Word16)
363255 [|xor (unsafeInlineIO
364− (Unbox.peekAt
256+ (Unbox.peekByteIndex
365257 ($(varE off) + $(litIntegral i))
366258 $(varE arr)))
367259 ($(wIntegral) :: Word16) .|.
@@ -381,7 +273,7 @@ xorCmp tag off arr =
381273 , tag !! (i + 3)
382274 ] :: Word32)
383275 [|xor (unsafeInlineIO
384− (Unbox.peekAt
276+ (Unbox.peekByteIndex
385277 ($(varE off) + $(litIntegral i))
386278 $(varE arr)))
387279 ($(wIntegral) :: Word32) .|.
@@ -405,7 +297,7 @@ xorCmp tag off arr =
405297 , tag !! (i + 7)
406298 ])
407299 [|xor (unsafeInlineIO
408− (Unbox.peekAt
300+ (Unbox.peekByteIndex
409301 ($(varE off) + $(litIntegral i))
410302 $(varE arr)))
411303 ($(wIntegral) :: Word64) .|.
@@ -431,7 +323,7 @@ serializeW8List off arr w8List = do
431323 makeBind i =
432324 bindS
433325 (varP (makeN (i + 1)))
434− [|$(varE 'serializeAt)
326+ [|$(varE 'serialize)
435327 $(varE (makeN i))
436328 $(varE arr)
437329 ($(litIntegral (w8List !! i)) :: Word8)|]
@@ -450,26 +342,20 @@ litProxy = litE . IntegerL . fromIntegral . Unbox.sizeOf
450342-- Error codes
451343--------------------------------------------------------------------------------
452344
453−errorUnsupported :: String -> a
454−errorUnsupported err =
345+errorUnsupported :: a
346+errorUnsupported =
455347 error
456348 $ unlines
457− [ "Unsupported"
458− , "==========="
459− , "This is improper use of the library."
349+ [ "Unsupported."
350+ , "There is improper use of the library."
460351 , "This case is unsupported."
461352 , "Please contact the developer if this case is of interest."
462− , ""
463− , "Message"
464− , "-------"
465− , err
466353 ]
467354
468355errorUnimplemented :: a
469356errorUnimplemented =
470357 error
471358 $ unlines
472− [ "Unimplemented"
473− , "============="
359+ [ "Unimplemented."
474360 , "Please contact the developer if this case is of interest."
475361 ]
core/src/Streamly/Internal/Data/Serialize/TH/RecHeader.hs+98−181
@@ -9,11 +9,11 @@
99-- Portability : GHC
1010--
1111module Streamly.Internal.Data.Serialize.TH.RecHeader
12− ( mkRecSerializeExpr
13− , mkRecDeserializeExpr
14− , mkRecSizeOfExpr
15− , conUpdateFuncDec
16− , mkDeserializeKeysDec
12+ ( mkSerializeExpr
13+ , mkDeserializeExpr
14+ , mkSizeOfExpr
15+ , conUpdateFuncExpr
16+ , mkDeserializeKeysExpr
1717 ) where
1818
1919--------------------------------------------------------------------------------
@@ -22,12 +22,12 @@ module Streamly.Internal.Data.Serialize.TH.RecHeader
2222
2323import Control.Monad (void)
2424import Data.List (foldl')
25−import Data.Word (Word32, Word8)
25+import Data.Word (Word16, Word32, Word8)
2626import Data.Maybe (fromJust)
2727import Language.Haskell.TH
2828import Streamly.Internal.Data.Serialize.Type (Serialize(..))
2929import Data.Foldable (foldlM)
30−import Streamly.Internal.Data.MutByteArray.Type (MutByteArray)
30+import Streamly.Internal.Data.Unbox (MutableByteArray)
3131import Data.Proxy (Proxy(..))
3232
3333import qualified Streamly.Internal.Data.Unbox as Unbox
@@ -39,55 +39,9 @@ import Streamly.Internal.Data.Serialize.TH.Common
3939-- Notes
4040--------------------------------------------------------------------------------
4141
42−-- Compatibility Algorithm
43−-- =======================
44−--
45−-- The algorithm is written without any low level implementation details. See
46−-- the code for any low level implementation details.
47−--
48−-- Serialization:
49−-- --------------
50−--
51−-- To serialize the data,
52−--
53−-- * Get the list of keys for the record as @keyList@.
54−-- * Serialize the @keyList@.
55−-- * Serialize the @fields@ one-by-one after serializing the @keyList@.
56−--
57−-- Deserialization:
58−-- ----------------
59−--
60−-- To deserialize the data to type @T@,
61−--
62−-- __Checking for type match__:
63−--
64−-- * Get the list of keys for type @T@ as @targetKeyList@.
65−-- * Get the list of keys encoded as @encodedKeyList@.
66−-- * If @targetKeyList == encodedKeyList@ see the __Type Match__ section else
67−-- see the __No Type Match__ section.
68−--
69−-- __Type Match__:
70−--
71−-- * Decode the fields one-by-one and construct the type @T@ in the end.
72−--
73−-- __No Type Match__:
74−--
75−-- * Decode the list of keys encoded into @encodedKeyList@.
76−-- * Get the list of keys for type @T@ as @targetKeyList@.
77−-- * Loop through @encodedKeyList@ and start deserializing the encoded data.
78−-- * If the key is present in @encodedKeyList@ and not in @targetKeyList@
79−-- then skip parsing the corresponding value.
80−-- * If the key is present in @targetKeyList@ and not in @encodedKeyList@
81−-- then set the value for that key as @Nothing@.
82−-- * If the key is present in both @encodedKeyList@ and in @targetKeyList@
83−-- parse the value.
84−-- * Construct @T@ after parsing all the data.
85−
86−-- Developer Notes
87−-- ===============
88−--
89−-- * Record update syntax is not robust across language extensions and common
90−-- record plugins (like record-dot-processor, large-records, etc.).
42+-- Encoding the header length is required to check if we want to compare the
43+-- header. We should not compare the headers if the lengths of the headers are
44+-- not equal as this might lead to a runtime error.
9145
9246--------------------------------------------------------------------------------
9347-- Compact lists
@@ -103,36 +57,36 @@ newtype CompactList a =
10357-- the list is 255.
10458instance forall a. Serialize a => Serialize (CompactList a) where
10559
106− -- {-# INLINE addSizeTo #-}
107− addSizeTo acc (CompactList xs) =
108− foldl' addSizeTo (acc + (Unbox.sizeOf (Proxy :: Proxy Word8))) xs
60+ -- {-# INLINE size #-}
61+ size acc (CompactList xs) =
62+ foldl' size (acc + (Unbox.sizeOf (Proxy :: Proxy Word8))) xs
10963
11064 -- Inlining this causes large compilation times for tests
111− {-# INLINABLE deserializeAt #-}
112− deserializeAt off arr sz = do
113− (off1, len8) <- deserializeAt off arr sz :: IO (Int, Word8)
65+ {-# INLINABLE deserialize #-}
66+ deserialize off arr sz = do
67+ (off1, len8) <- deserialize off arr sz :: IO (Int, Word8)
11468 let len = w8_int len8
11569 peekList f o i | i >= 3 = do
11670 -- Unfold the loop three times
117− (o1, x1) <- deserializeAt o arr sz
118− (o2, x2) <- deserializeAt o1 arr sz
119− (o3, x3) <- deserializeAt o2 arr sz
71+ (o1, x1) <- deserialize o arr sz
72+ (o2, x2) <- deserialize o1 arr sz
73+ (o3, x3) <- deserialize o2 arr sz
12074 peekList (f . (\xs -> x1:x2:x3:xs)) o3 (i - 3)
12175 peekList f o 0 = pure (o, f [])
12276 peekList f o i = do
123− (o1, x) <- deserializeAt o arr sz
77+ (o1, x) <- deserialize o arr sz
12478 peekList (f . (x:)) o1 (i - 1)
12579 (nextOff, lst) <- peekList id off1 len
12680 pure (nextOff, CompactList lst)
12781
12882 -- Inlining this causes large compilation times for tests
129− {-# INLINABLE serializeAt #-}
130− serializeAt off arr (CompactList val) = do
131− void $ serializeAt off arr (int_w8 (length val) :: Word8)
83+ {-# INLINABLE serialize #-}
84+ serialize off arr (CompactList val) = do
85+ void $ serialize off arr (int_w8 (length val) :: Word8)
13286 let off1 = off + Unbox.sizeOf (Proxy :: Proxy Word8)
13387 let pokeList o [] = pure o
13488 pokeList o (x:xs) = do
135− o1 <- serializeAt o arr x
89+ o1 <- serialize o arr x
13690 pokeList o1 xs
13791 pokeList off1 val
13892
@@ -154,7 +108,7 @@ isMaybeType _ = False
154108-- We add 4 here because we use 'serializeWithSize' for serializing.
155109exprGetSize :: Q Exp -> (Int, Type) -> Q Exp
156110exprGetSize acc (i, _) =
157− [|addSizeTo $(acc) $(varE (mkFieldName i)) + 4|]
111+ [|size $(acc) $(varE (mkFieldName i)) + 4|]
158112
159113sizeOfHeader :: SimpleDataCon -> Int
160114sizeOfHeader (SimpleDataCon _ fields) =
@@ -164,15 +118,13 @@ sizeOfHeader (SimpleDataCon _ fields) =
164118 where
165119
166120 sizeForFinalOff = 4
167− sizeForHeaderLength = 4 -- Max header length is (255 * (255 + 1) + 1) and
168− -- hence 2 bytes is enough to store it. But we still
169− -- use 4 bytes as using 2 bytes introduces
170− -- regression.
121+ sizeForHeaderLength = 2 -- Max header length is (255 * (255 + 1) + 1) and
122+ -- hence 2 bytes is enough to store it.
171123 sizeForNumFields = 1 -- At max 255 fields in the record constructor
172124 sizeForFieldLen = 1 -- At max 255 letters in the key
173125
174−mkRecSizeOfExpr :: SimpleDataCon -> Q Exp
175−mkRecSizeOfExpr con = do
126+mkSizeOfExpr :: SimpleDataCon -> Q Exp
127+mkSizeOfExpr con = do
176128 n_acc <- newName "acc"
177129 n_x <- newName "x"
178130 (lamE
@@ -205,47 +157,40 @@ headerValue (SimpleDataCon _ fields) =
205157 in if lenFields <= 255
206158 then lenFields
207159 else errorUnsupported
208− "Number of fields in the record should be <= 255."
209160 lengthPrependedFieldEncoding field =
210161 let fEnc =
211162 let fEnc_ = map c2w (fieldToNameBase field)
212163 lenFEnc = length fEnc_
213164 in if lenFEnc <= 255
214165 then fEnc_
215− else
216− errorUnsupported
217− "Length of any key should be <= 255."
166+ else errorUnsupported
218167 in (int_w8 (length fEnc)) : fEnc
219168
220169--------------------------------------------------------------------------------
221170-- Peek
222171--------------------------------------------------------------------------------
223172
224−-- Encoding the size is required if we want to skip the field without knowing
225−-- its type. We encode the size as 'Word32' hence there is a 4 bytes increase
226−-- in size.
173+-- Encoding the size is required if we want to skip the field.
174+-- We encode the size as 'Word32' hence there is a 4 bytes increase in size.
227175{-# INLINE serializeWithSize #-}
228−serializeWithSize :: Serialize a => Int -> MutByteArray -> a -> IO Int
176+serializeWithSize :: Serialize a => Int -> MutableByteArray -> a -> IO Int
229177serializeWithSize off arr val = do
230− off1 <- serializeAt (off + 4) arr val
231− Unbox.pokeAt off arr (int_w32 (off1 - off - 4) :: Word32)
178+ off1 <- serialize (off + 4) arr val
179+ Unbox.pokeByteIndex off arr (int_w32 (off1 - off - 4) :: Word32)
232180 pure off1
233181
234−mkRecSerializeExpr :: Name -> SimpleDataCon -> Q Exp
235−mkRecSerializeExpr initialOffset (con@(SimpleDataCon cname fields)) = do
182+mkSerializeExpr :: Name -> SimpleDataCon -> Q Exp
183+mkSerializeExpr initialOffset (con@(SimpleDataCon cname fields)) = do
236184 afterHLen <- newName "afterHLen"
237− -- Encoding the header length is required.
238− -- We first compare the header length encoded and the current header
239− -- length. Only if the header lengths match, we compare the headers.
240185 [|do $(varP afterHLen) <-
241− serializeAt
186+ serialize
242187 ($(varE initialOffset) + 4)
243188 $(varE _arr)
244− ($(litIntegral hlen) :: Word32)
189+ ($(litIntegral hlen) :: Word16)
245190 $(varP (makeI 0)) <- $(serializeW8List afterHLen _arr hval)
246191 let $(openConstructor cname (length fields)) = $(varE _val)
247192 finalOff <- $(mkSerializeExprFields 'serializeWithSize fields)
248− Unbox.pokeAt
193+ Unbox.pokeByteIndex
249194 $(varE initialOffset)
250195 $(varE _arr)
251196 ((fromIntegral :: Int -> Word32)
@@ -263,12 +208,12 @@ mkRecSerializeExpr initialOffset (con@(SimpleDataCon cname fields)) = do
263208
264209{-# INLINE deserializeWithSize #-}
265210deserializeWithSize ::
266− Serialize a => Int -> MutByteArray -> Int -> IO (Int, a)
267−deserializeWithSize off arr endOff = deserializeAt (off + 4) arr endOff
211+ Serialize a => Int -> MutableByteArray -> Int -> IO (Int, a)
212+deserializeWithSize off arr endOff = deserialize (off + 4) arr endOff
268213
269−conUpdateFuncDec :: Name -> [Field] -> Q [Dec]
270−conUpdateFuncDec funcName fields = do
271− prevAcc <- newName "prevAcc"
214+conUpdateFuncExpr :: Name -> Type -> [Field] -> Q [Dec]
215+conUpdateFuncExpr funcName headTy fields = do
216+ prevRec <- newName "prevRec"
272217 curOff <- newName "curOff"
273218 endOff <- newName "endOff"
274219 arr <- newName "arr"
@@ -277,17 +222,17 @@ conUpdateFuncDec funcName fields = do
277222 (caseE
278223 (varE key)
279224 (concat
280− [ map (matchField arr endOff (prevAcc, curOff)) fnames
225+ [ map (matchField arr endOff (prevRec, curOff)) fnames
281226 , [ match
282227 wildP
283228 (normalB
284229 [|do (valOff, valLen :: Word32) <-
285− deserializeAt
230+ deserialize
286231 $(varE curOff)
287232 $(varE arr)
288233 $(varE endOff)
289234 pure
290− ( $(varE prevAcc)
235+ ( $(varE prevRec)
291236 , valOff + w32_int valLen)|])
292237 []
293238 ]
@@ -299,7 +244,7 @@ conUpdateFuncDec funcName fields = do
299244 [ Clause
300245 [ VarP arr
301246 , VarP endOff
302− , TupP [VarP prevAcc, VarP curOff]
247+ , TupP [VarP prevRec, VarP curOff]
303248 , VarP key
304249 ]
305250 (NormalB method)
@@ -311,99 +256,71 @@ conUpdateFuncDec funcName fields = do
311256
312257 fnames = fmap (fromJust . fst) fields
313258 matchField :: Name -> Name -> (Name, Name) -> Name -> Q Match
314− matchField arr endOff (acc, currOff) fname = do
315− let fnameLit = StringL (nameBase fname)
259+ matchField arr endOff (prevRec, currOff) fname = do
260+ val <- newName "val"
261+ newOff <- newName "newOff"
316262 match
317− (litP fnameLit)
263+ (litP (StringL (nameBase fname)))
318264 (normalB
319− [|do (valOff, valLen :: Word32) <-
320− deserializeAt
321− $(varE currOff)
322− $(varE arr)
323− $(varE endOff)
265+ [|do ($(varP newOff), $(varP val)) <-
266+ deserializeWithSize
267+ $(varE currOff)
268+ $(varE arr)
269+ $(varE endOff)
324270 pure
325− ( ($(litE fnameLit), $(varE currOff)) : $(varE acc)
326− , valOff + w32_int valLen)|])
271+ ( $(recUpdE (varE prevRec) [pure (fname, VarE val)])
272+ :: $(pure headTy)
273+ , $(varE newOff))|])
327274 []
328275
329−mkDeserializeKeysDec :: Name -> Name -> SimpleDataCon -> Q [Dec]
330−mkDeserializeKeysDec funcName updateFunc (SimpleDataCon cname fields) = do
331− hOff <- newName "hOff"
332− finalOff <- newName "finalOff"
333− arr <- newName "arr"
334− endOff <- newName "endOff"
335− kvEncoded <- newName "kvEncoded"
336− finalRec <- newName "finalRec"
337− let deserializeFieldExpr (Just name, ty) = do
338− let nameLit = litE (StringL (nameBase name))
339− [|case lookup $(nameLit) $(varE kvEncoded) of
340− Nothing -> $(emptyTy name ty)
341− Just off -> do
342− val <- deserializeWithSize off $(varE arr) $(varE endOff)
343− pure $ snd val|]
344− deserializeFieldExpr _ =
345− errorUnsupported "The datatype should use record syntax."
346− method <-
347− [|do (dataOff, hlist :: CompactList (CompactList Word8)) <-
348− deserializeAt $(varE hOff) $(varE arr) $(varE endOff)
349− let keys = wListToString . unCompactList <$> unCompactList hlist
350− ($(varP kvEncoded), _) <-
351− foldlM
352− ($(varE updateFunc) $(varE arr) $(varE endOff))
353− ([], dataOff)
354− keys
355− $(varP finalRec) <-
356− $(foldl
357− (\acc i ->
358− [|$(acc) <*>
359− $(deserializeFieldExpr i)|])
360− [|pure $(conE cname)|]
361− fields)
362− pure ($(varE finalOff), $(varE finalRec))|]
363− pure
364− [ PragmaD (InlineP funcName NoInline FunLike AllPhases)
365− , FunD
366− funcName
367− [ Clause
368− [ VarP hOff
369− , VarP finalOff
370− , VarP arr
371− , VarP endOff
372− ]
373− (NormalB method)
374− []
375− ]
376− ]
276+mkDeserializeKeysExpr ::
277+ Name -> Name -> Name -> Name -> Name -> SimpleDataCon -> Q Exp
278+mkDeserializeKeysExpr
279+ hOff finalOff arr endOff updateFunc (SimpleDataCon cname fields) = do
280+
281+ [|do (dataOff, hlist :: CompactList (CompactList Word8)) <-
282+ deserialize $(varE hOff) $(varE arr) $(varE endOff)
283+ let keys = wListToString . unCompactList <$> unCompactList hlist
284+ (finalRec, _) <-
285+ foldlM
286+ ($(varE updateFunc) $(varE arr) $(varE endOff))
287+ ($(emptyConExpr), dataOff)
288+ keys
289+ pure ($(varE finalOff), finalRec)|]
377290
378291 where
379292
380293 emptyTy k ty =
381294 if isMaybeType ty
382− then [|pure Nothing|]
295+ then [|Nothing|]
383296 else [|error $(litE (StringL (nameBase k ++ " is not found.")))|]
297+ fieldToEmptyRecC (Just name, ty) = (name, ) <$> emptyTy name ty
298+ fieldToEmptyRecC _ = errorUnsupported
299+ emptyConExpr = recConE cname (fmap fieldToEmptyRecC fields)
384300
385−
386−mkRecDeserializeExpr :: Name -> Name -> Name -> SimpleDataCon -> Q Exp
387−mkRecDeserializeExpr initialOff endOff deserializeWithKeys con = do
301+mkDeserializeExpr :: Name -> Name -> Name -> SimpleDataCon -> Q Exp
302+mkDeserializeExpr initialOff endOff deserializeWithKeys con = do
388303 hOff <- newName "hOff"
389304 let sizeForFinalOff = 4 -- Word32
390− sizeForHeaderLength = 4 -- Word32
305+ sizeForHeaderLength = 2 -- Word16
391306 sizePreData = sizeForFinalOff + sizeForHeaderLength + hlen
392307 [|do (hlenOff, encLen :: Word32) <-
393− deserializeAt $(varE initialOff) $(varE _arr) $(varE endOff)
394− ($(varP hOff), hlen1 :: Word32) <-
395− deserializeAt hlenOff $(varE _arr) $(varE endOff)
396− if (hlen1 == $(litIntegral hlen)) && $(xorCmp hval hOff _arr)
397− then do
398− let $(varP (makeI 0)) =
399− $(varE initialOff) +
400− $(litIntegral sizePreData)
401− $(mkDeserializeExprOne 'deserializeWithSize con)
402− else $(varE deserializeWithKeys)
403− $(varE hOff)
404− ($(varE initialOff) + w32_int encLen)
405− $(varE _arr)
406− $(varE endOff)|]
308+ deserialize $(varE initialOff) $(varE _arr) $(varE endOff)
309+ ($(varP hOff), hlen1 :: Word16) <-
310+ deserialize hlenOff $(varE _arr) $(varE endOff)
311+ if hlen1 == $(litIntegral hlen)
312+ then if $(xorCmp hval hOff _arr)
313+ then do
314+ let $(varP (makeI 0)) =
315+ $(varE initialOff) +
316+ $(litIntegral sizePreData)
317+ $(mkDeserializeExprOne 'deserializeWithSize con)
318+ else $(varE deserializeWithKeys)
319+ $(varE hOff)
320+ ($(varE initialOff) + w32_int encLen)
321+ else $(varE deserializeWithKeys)
322+ $(varE hOff)
323+ ($(varE initialOff) + w32_int encLen)|]
407324
408325 where
409326
core/src/Streamly/Internal/Data/Serialize/Type.hs+123−142
@@ -7,27 +7,37 @@
77--
88
99module Streamly.Internal.Data.Serialize.Type
10− (
11− Serialize(..)
10+ ( Serialize(..)
11+ , encode
12+ , pinnedEncode
13+ , decode
1214 ) where
1315
1416--------------------------------------------------------------------------------
1517-- Imports
1618--------------------------------------------------------------------------------
1719
18−import Control.Monad (when)
20+#include "assert.hs"
21+
22+#ifdef DEBUG
23+import Control.Exception (assert)
24+#endif
25+
1926import Data.List (foldl')
2027import Data.Proxy (Proxy (..))
21−import Streamly.Internal.Data.Unbox (Unbox)
22−import Streamly.Internal.Data.MutByteArray.Type (MutByteArray(..))
28+import Streamly.Internal.Data.Unbox
29+ ( MutableByteArray(..)
30+ , PinnedState(..)
31+ , Unbox
32+ )
2333import Streamly.Internal.Data.Array.Type (Array(..))
34+import Streamly.Internal.System.IO (unsafeInlineIO)
2435import GHC.Int (Int16(..), Int32(..), Int64(..), Int8(..))
2536import GHC.Word (Word16(..), Word32(..), Word64(..), Word8(..))
2637import GHC.Stable (StablePtr(..))
2738
28−import qualified Streamly.Internal.Data.MutByteArray.Type as MBA
2939import qualified Streamly.Internal.Data.Unbox as Unbox
30−import qualified Streamly.Internal.Data.Array.Type as Array
40+import qualified Streamly.Internal.Data.Array as Array
3141import qualified Streamly.Internal.Data.MutArray as MutArray
3242
3343import GHC.Exts
@@ -49,33 +59,21 @@ import GHC.Exts
4959-- Types
5060--------------------------------------------------------------------------------
5161
52−-- | The 'Serialize' type class provides operations for serialization and
53−-- deserialization of general Haskell data types to and from their byte stream
54−-- representation.
62+-- | A type implementing the 'Serialize' interface supplies operations for
63+-- reading and writing the type from and to a mutable byte array (an unboxed
64+-- representation of the type) in memory. The read operation 'deserialize'
65+-- deserializes the boxed type from the mutable byte array. The write operation
66+-- 'serialize' serializes the boxed type to the mutable byte array.
5567--
56−-- Unlike 'Unbox', 'Serialize' uses variable length encoding, therefore, it can
57−-- serialize recursive and variable length data types like lists, or variable
58−-- length sum types where the length of the value may vary depending on a
59−-- particular constructor. For variable length data types the length is encoded
60−-- along with the data.
68+-- IMPORTANT: The serialized data's byte ordering is not normalized, which means
69+-- it remains the same as the machine's byte order where the function is
70+-- executed. Consequently, it may not be compatible across machines with
71+-- different byte ordering.
6172--
62−-- The 'deserializeAt' operation reads bytes from the mutable byte array and
63−-- builds a Haskell data type from these bytes, the number of bytes it reads
64−-- depends on the type and the encoded value it is reading. 'serializeAt'
65−-- operation converts a Haskell data type to its binary representation which
66−-- must consist of as many bytes as added by the @addSizeTo@ operation for that
67−-- value and then stores these bytes into the mutable byte array. The
68−-- programmer is expected to use the @addSizeTo@ operation and allocate an
69−-- array of sufficient length before calling 'serializeAt'.
73+-- 'Serialize' contains enough information to serialize and deserialize variable
74+-- length types.
7075--
71−-- IMPORTANT: The serialized data's byte ordering remains the same as the host
72−-- machine's byte order. Therefore, it can not be deserialized from host
73−-- machines with a different byte ordering.
74−--
75−-- Instances can be derived via Template Haskell, or written manually.
76−--
77−-- Here is an example, for deriving an instance of this type class using
78−-- template Haskell:
76+-- >>> import Streamly.Internal.Data.Serialize.Type (Serialize(..))
7977--
8078-- >>> :{
8179-- data Object = Object
@@ -84,30 +82,16 @@ import GHC.Exts
8482-- }
8583-- :}
8684--
87−-- @
88−-- import Streamly.Data.MutByteArray (deriveSerialize)
89−-- \$(deriveSerialize [d|instance Serialize Object|])
90−-- @
91−--
92−-- See 'Streamly.Data.MutByteArray.deriveSerialize' and
93−-- 'Streamly.Data.MutByteArray.deriveSerializeWith' for more information on
94−-- deriving using Template Haskell.
95−--
96−-- Here is an example of a manual instance.
97−--
98−-- >>> import Streamly.Data.MutByteArray (Serialize(..))
99−--
10085-- >>> :{
10186-- instance Serialize Object where
102−-- addSizeTo acc obj = addSizeTo (addSizeTo acc (_obj1 obj)) (_obj2 obj)
103−-- deserializeAt i arr len = do
104−-- -- Check the array bounds before reading
105−-- (i1, x0) <- deserializeAt i arr len
106−-- (i2, x1) <- deserializeAt i1 arr len
87+-- size acc obj = size (size acc (_obj1 obj)) (_obj2 obj)
88+-- deserialize i arr len = do
89+-- (i1, x0) <- deserialize i arr len
90+-- (i2, x1) <- deserialize i1 arr len
10791-- pure (i2, Object x0 x1)
108−-- serializeAt i arr (Object x0 x1) = do
109−-- i1 <- serializeAt i arr x0
110−-- i2 <- serializeAt i1 arr x1
92+-- serialize i arr (Object x0 x1) = do
93+-- i1 <- serialize i arr x0
94+-- i2 <- serialize i1 arr x1
11195-- pure i2
11296-- :}
11397--
@@ -118,50 +102,44 @@ class Serialize a where
118102 -- It is of the form @Int -> a -> Int@ because you can have tail-recursive
119103 -- traversal of the structures.
120104
121− -- | @addSizeTo accum value@ returns @accum@ incremented by the size of the
122− -- serialized representation of @value@ in bytes. Size cannot be zero. It
123− -- should be at least 1 byte.
124− addSizeTo :: Int -> a -> Int
105+ -- | Get the size, in bytes, reqired to store the serialized
106+ -- representation of the type. Size cannot be zero.
107+ size :: Int -> a -> Int
125108
126109 -- We can implement the following functions without returning the `Int`
127110 -- offset but that may require traversing the Haskell structure again to get
128111 -- the size. Therefore, this is a performance optimization.
129112
130− -- | @deserializeAt byte-offset array arrayLen@ deserializes a value from
131− -- the given byte-offset in the array. Returns a tuple consisting of the
132− -- next byte-offset and the deserialized value.
133− --
134− -- The arrayLen passed is the entire length of the input buffer. It is to
135− -- be used to check if we would overflow the input buffer when
136− -- deserializing.
137− --
138− -- Throws an exception if the operation would exceed the supplied arrayLen.
139− deserializeAt :: Int -> MutByteArray -> Int -> IO (Int, a)
140−
141− -- | @serializeAt byte-offset array value@ writes the serialized
142− -- representation of the @value@ in the array at the given byte-offset.
143− -- Returns the next byte-offset.
144− --
145− -- This is an unsafe operation, the programmer must ensure that the array
146− -- has enough space available to serialize the value as determined by the
147− -- @addSizeTo@ operation.
148− serializeAt :: Int -> MutByteArray -> a -> IO Int
113+ -- | @deserialize offset array arrayLen@ deserializes a value from the
114+ -- given byte-index in the array. Returns a tuple of the next byte-index
115+ -- and the deserialized value.
116+ deserialize :: Int -> MutableByteArray -> Int -> IO (Int, a)
117+
118+ -- | Write the serialized representation of the value in the array at the
119+ -- given byte-index. Returns the next byte-index. This is an unsafe
120+ -- operation, the programmer must ensure that the array has enough space
121+ -- available in the array to serialize the value as determined by the
122+ -- @size@ operation.
123+ serialize :: Int -> MutableByteArray -> a -> IO Int
149124
150125--------------------------------------------------------------------------------
151126-- Instances
152127--------------------------------------------------------------------------------
153128
154−-- _size is the length from array start to the last accessed byte.
129+-- XXX We do not need to pass the end of data offset, we can just check the
130+-- mutable array end here to avoid a crash, and when we return from deserialize
131+-- we can check if the offset returned is beyond the bound or not.
132+--
155133#ifdef DEBUG
156134{-# INLINE checkBounds #-}
157−checkBounds :: String -> Int -> MutByteArray -> IO ()
158−checkBounds _label _size _arr = do
159− sz <- MBA.sizeOfMutableByteArray _arr
160− if (_size > sz)
135+checkBounds :: String -> Int -> MutableByteArray -> IO ()
136+checkBounds _label _off _arr = do
137+ sz <- sizeOfMutableByteArray _arr
138+ if (_off > sz)
161139 then error
162140 $ _label
163141 ++ ": accessing array at offset = "
164− ++ show (_size - 1)
142+ ++ show (_off - 1)
165143 ++ " max valid offset = " ++ show (sz - 1)
166144 else return ()
167145#endif
@@ -184,39 +162,39 @@ checkBounds _label _size _arr = do
184162-- single array for deserializing. But that can also be done by framing the
185163-- serialized value with a size header.
186164--
187−{-# INLINE deserializeChecked #-}
188−deserializeChecked :: forall a. Unbox a => Int -> MutByteArray -> Int -> IO (Int, a)
189−deserializeChecked off arr sz =
165+{-# INLINE deserializeUnsafe #-}
166+deserializeUnsafe :: forall a. Unbox a => Int -> MutableByteArray -> Int -> IO (Int, a)
167+deserializeUnsafe off arr sz =
190168 let next = off + Unbox.sizeOf (Proxy :: Proxy a)
191169 in do
192170 -- Keep likely path in the straight branch.
193171 if (next <= sz)
194− then Unbox.peekAt off arr >>= \val -> pure (next, val)
172+ then Unbox.peekByteIndex off arr >>= \val -> pure (next, val)
195173 else error
196− $ "deserializeAt: accessing array at offset = "
174+ $ "deserialize: accessing array at offset = "
197175 ++ show (next - 1)
198176 ++ " max valid offset = " ++ show (sz - 1)
199177
200178{-# INLINE serializeUnsafe #-}
201−serializeUnsafe :: forall a. Unbox a => Int -> MutByteArray -> a -> IO Int
179+serializeUnsafe :: forall a. Unbox a => Int -> MutableByteArray -> a -> IO Int
202180serializeUnsafe off arr val =
203181 let next = off + Unbox.sizeOf (Proxy :: Proxy a)
204182 in do
205183#ifdef DEBUG
206− checkBounds "serializeAt" next arr
184+ checkBounds "serialize" next arr
207185#endif
208− Unbox.pokeAt off arr val
186+ Unbox.pokeByteIndex off arr val
209187 pure next
210188
211189#define DERIVE_SERIALIZE_FROM_UNBOX(_type) \
212190instance Serialize _type where \
213−; {-# INLINE addSizeTo #-} \
214−; addSizeTo acc _ = acc + Unbox.sizeOf (Proxy :: Proxy _type) \
215−; {-# INLINE deserializeAt #-} \
216−; deserializeAt off arr end = deserializeChecked off arr end :: IO (Int, _type) \
217−; {-# INLINE serializeAt #-} \
218−; serializeAt = \
219− serializeUnsafe :: Int -> MutByteArray -> _type -> IO Int
191+; {-# INLINE size #-} \
192+; size acc _ = acc + Unbox.sizeOf (Proxy :: Proxy _type) \
193+; {-# INLINE deserialize #-} \
194+; deserialize off arr end = deserializeUnsafe off arr end :: IO (Int, _type) \
195+; {-# INLINE serialize #-} \
196+; serialize = \
197+ serializeUnsafe :: Int -> MutableByteArray -> _type -> IO Int
220198
221199DERIVE_SERIALIZE_FROM_UNBOX(())
222200DERIVE_SERIALIZE_FROM_UNBOX(Bool)
@@ -239,79 +217,82 @@ DERIVE_SERIALIZE_FROM_UNBOX((FunPtr a))
239217
240218instance forall a. Serialize a => Serialize [a] where
241219
242− -- {-# INLINE addSizeTo #-}
243− addSizeTo acc xs =
244− foldl' addSizeTo (acc + (Unbox.sizeOf (Proxy :: Proxy Int))) xs
220+ -- {-# INLINE size #-}
221+ size acc xs = foldl' size (acc + (Unbox.sizeOf (Proxy :: Proxy Int))) xs
245222
246223 -- Inlining this causes large compilation times for tests
247− {-# INLINABLE deserializeAt #-}
248− deserializeAt off arr sz = do
249− (off1, len64) <- deserializeAt off arr sz :: IO (Int, Int64)
224+ {-# INLINABLE deserialize #-}
225+ deserialize off arr sz = do
226+ (off1, len64) <- deserialize off arr sz :: IO (Int, Int64)
250227 let len = (fromIntegral :: Int64 -> Int) len64
251228 peekList f o i | i >= 3 = do
252229 -- Unfold the loop three times
253− (o1, x1) <- deserializeAt o arr sz
254− (o2, x2) <- deserializeAt o1 arr sz
255− (o3, x3) <- deserializeAt o2 arr sz
230+ (o1, x1) <- deserialize o arr sz
231+ (o2, x2) <- deserialize o1 arr sz
232+ (o3, x3) <- deserialize o2 arr sz
256233 peekList (f . (\xs -> x1:x2:x3:xs)) o3 (i - 3)
257234 peekList f o 0 = pure (o, f [])
258235 peekList f o i = do
259− (o1, x) <- deserializeAt o arr sz
236+ (o1, x) <- deserialize o arr sz
260237 peekList (f . (x:)) o1 (i - 1)
261238 peekList id off1 len
262239
263240 -- Inlining this causes large compilation times for tests
264− {-# INLINABLE serializeAt #-}
265− serializeAt off arr val = do
241+ {-# INLINABLE serialize #-}
242+ serialize off arr val = do
266243 let off1 = off + Unbox.sizeOf (Proxy :: Proxy Int64)
267244 let pokeList acc o [] =
268− Unbox.pokeAt off arr (acc :: Int64) >> pure o
245+ Unbox.pokeByteIndex off arr (acc :: Int64) >> pure o
269246 pokeList acc o (x:xs) = do
270− o1 <- serializeAt o arr x
247+ o1 <- serialize o arr x
271248 pokeList (acc + 1) o1 xs
272249 pokeList 0 off1 val
273250
274−instance
275−#ifdef DEVBUILD
276− Unbox a =>
277−#endif
278− Serialize (Array a) where
279− {-# INLINE addSizeTo #-}
280− addSizeTo i (Array {..}) = i + (arrEnd - arrStart) + 8
251+instance Serialize (Array a) where
252+ {-# INLINE size #-}
253+ size i (Array {..}) = i + (arrEnd - arrStart) + 8
281254
282− {-# INLINE deserializeAt #-}
283− deserializeAt off arr len = do
284− (off1, byteLen) <- deserializeAt off arr len :: IO (Int, Int)
255+ {-# INLINE deserialize #-}
256+ deserialize off arr end = do
257+ (off1, byteLen) <- deserialize off arr end :: IO (Int, Int)
285258 let off2 = off1 + byteLen
286− when (off2 > len) $
287− error
288− $ "deserializeAt: accessing array at offset = "
289− ++ show (off2 - 1)
290− ++ " max valid offset = " ++ show (len - 1)
291− -- XXX Use MutByteArray.cloneSliceUnsafe
292259 let slice = MutArray.MutArray arr off1 off2 off2
293260 newArr <- MutArray.clone slice
294261 pure (off2, Array.unsafeFreeze newArr)
295262
296− {-# INLINE serializeAt #-}
297− serializeAt off arr (Array {..}) = do
263+ {-# INLINE serialize #-}
264+ serialize off arr (Array {..}) = do
298265 let arrLen = arrEnd - arrStart
299− off1 <- serializeAt off arr arrLen
300− MBA.putSliceUnsafe arrContents arrStart arr off1 arrLen
266+ off1 <- serialize off arr arrLen
267+ Unbox.putSliceUnsafe arrContents arrStart arr off1 arrLen
301268 pure (off1 + arrLen)
302269
303−instance (Serialize a, Serialize b) => Serialize (a, b) where
270+--------------------------------------------------------------------------------
271+-- High level functions
272+--------------------------------------------------------------------------------
273+
274+{-# INLINE encodeAs #-}
275+encodeAs :: forall a. Serialize a => PinnedState -> a -> Array Word8
276+encodeAs ps a =
277+ unsafeInlineIO $ do
278+ let len = size 0 a
279+ mbarr <- Unbox.newBytesAs ps len
280+ off <- serialize 0 mbarr a
281+ assertM(len == off)
282+ pure $ Array mbarr 0 off
304283
305− {-# INLINE addSizeTo #-}
306− addSizeTo acc (a, b) = addSizeTo (addSizeTo acc a) b
284+{-# INLINE encode #-}
285+encode :: Serialize a => a -> Array Word8
286+encode = encodeAs Unpinned
307287
308− {-# INLINE serializeAt #-}
309− serializeAt off arr (a, b) = do
310− off1 <- serializeAt off arr a
311− serializeAt off1 arr b
288+{-# INLINE pinnedEncode #-}
289+pinnedEncode :: Serialize a => a -> Array Word8
290+pinnedEncode = encodeAs Pinned
312291
313− {-# INLINE deserializeAt #-}
314− deserializeAt off arr end = do
315− (off1, a) <- deserializeAt off arr end
316− (off2, b) <- deserializeAt off1 arr end
317− pure (off2, (a, b))
292+{-# INLINE decode #-}
293+decode :: Serialize a => Array Word8 -> a
294+decode arr@(Array {..}) = unsafeInlineIO $ do
295+ let lenArr = Array.length arr
296+ (off, val) <- deserialize arrStart arrContents (arrStart + lenArr)
297+ assertM(off == arrStart + lenArr)
298+ pure val
core/src/Streamly/Internal/Data/Array/Stream.hs → core/src/Streamly/Internal/Data/Stream/Chunked.hsrenamed+586−74
@@ -1,6 +1,5 @@
1−{-# OPTIONS_GHC -Wno-deprecations #-}
21-- |
3−-- Module : Streamly.Internal.Data.Array.Stream
2+-- Module : Streamly.Internal.Data.Stream.Chunked
43-- Copyright : (c) 2019 Composewell Technologies
54-- License : BSD3-3-Clause
65-- Maintainer : streamly@composewell.com
@@ -8,25 +7,17 @@
87--
98-- Combinators to efficiently manipulate streams of immutable arrays.
109--
11−-- We can either push these in the MutArray module with a "chunks" prefix or
12−-- keep this as a separate module and release it.
13−--
14−module Streamly.Internal.Data.Array.Stream
15−{-# DEPRECATED "Please use \"Streamly.Internal.Data.Array\" instead." #-}
10+module Streamly.Internal.Data.Stream.Chunked
1611 (
1712 -- * Creation
18− Array.chunksOf
19− , Array.pinnedChunksOf
20− , Array.bufferChunks
13+ chunksOf
2114
2215 -- * Flattening to elements
23− , Array.concat
24− , Array.flattenArrays
25− , Array.concatRev
26− , Array.flattenArraysRev
27− , Array.interpose
28− , Array.interposeSuffix
29− , Array.intercalateSuffix
16+ , concat
17+ , concatRev
18+ , interpose
19+ , interposeSuffix
20+ , intercalateSuffix
3021 , unlines
3122
3223 -- * Elimination
@@ -35,27 +26,24 @@ module Streamly.Internal.Data.Array.Stream
3526 -- can flatten the stream to byte stream and use that. But if we want the
3627 -- remaining stream to be a chunk stream then this could be handy. But it
3728 -- could also be implemented using parseBreak.
38− , foldBreak
29+ , foldBreak -- StreamK.foldBreakChunks
3930 , foldBreakD
40− -- This is chunked parseBreak. A byte level parseBreak cannot work
41− -- efficiently. Because the stream will have to be a StreamK for
42− -- backtracking, StreamK at byte level would not be efficient.
43− -- parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
44− , parseBreak
31+ -- The byte level parseBreak cannot work efficiently. Because the stream
32+ -- will have to be a StreamK for backtracking, StreamK at byte level would
33+ -- not be efficient.
34+ , parseBreak -- StreamK.parseBreakChunks
4535 -- , parseBreakD
4636 -- , foldManyChunks
4737 -- , parseManyChunks
48− , K.parseBreakChunks
49− , K.parseChunks
5038
5139 -- ** Array Folds
52− -- XXX Use parseBreakChunks/parseChunks instead
40+ -- XXX Use Parser.Chunked instead, need only chunkedParseBreak,
5341 -- foldBreak can be implemented using parseBreak. Use StreamK.
5442 , runArrayFold
5543 , runArrayFoldBreak
5644 -- , parseArr
57− , runArrayParserDBreak -- StreamK.parseBreakChunks
58− , runArrayFoldMany
45+ , runArrayParserDBreak -- StreamK.chunkedParseBreak
46+ , runArrayFoldMany -- StreamK.chunkedParseMany
5947
6048 , toArray
6149
@@ -79,8 +67,6 @@ where
7967import Data.Bifunctor (second)
8068import Control.Exception (assert)
8169import Control.Monad.IO.Class (MonadIO(..))
82−-- import Data.Bifunctor (first)
83−-- import Data.Either (fromRight)
8470import Data.Proxy (Proxy(..))
8571import Data.Word (Word8)
8672import Streamly.Internal.Data.Unbox (Unbox(..))
@@ -96,32 +82,142 @@ import Streamly.Internal.Data.Parser (ParseError(..))
9682import Streamly.Internal.Data.Stream (Stream)
9783import Streamly.Internal.Data.StreamK (StreamK, fromStream, toStream)
9884import Streamly.Internal.Data.SVar.Type (adaptState, defState)
85+import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
9986
87+import qualified Streamly.Data.Fold as FL
10088import qualified Streamly.Internal.Data.Array as A
10189import qualified Streamly.Internal.Data.Array as Array
90+import qualified Streamly.Internal.Data.MutArray.Type as MA
91+import qualified Streamly.Internal.Data.Stream.MutChunked as AS
92+import qualified Streamly.Internal.Data.Fold.Type as FL (Fold(..), Step(..))
10293import qualified Streamly.Internal.Data.Parser as PR
10394import qualified Streamly.Internal.Data.Parser as PRD
10495 (Parser(..), Initial(..))
105−-- import qualified Streamly.Internal.Data.ParserK as ParserK
10696import qualified Streamly.Internal.Data.Stream as D
10797import qualified Streamly.Internal.Data.StreamK as K
10898
10999-- XXX Since these are immutable arrays MonadIO constraint can be removed from
110100-- most places.
111101
102+-------------------------------------------------------------------------------
103+-- Generation
104+-------------------------------------------------------------------------------
105+
106+-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
107+-- @n@ elements each.
108+--
109+-- > chunksOf n = Stream.groupsOf n (Array.writeN n)
110+--
111+-- /Pre-release/
112+{-# INLINE chunksOf #-}
113+chunksOf :: (MonadIO m, Unbox a)
114+ => Int -> Stream m a -> Stream m (Array a)
115+chunksOf = A.chunksOf
116+
117+-------------------------------------------------------------------------------
118+-- Append
119+-------------------------------------------------------------------------------
120+
121+-- XXX efficiently compare two streams of arrays. Two streams can have chunks
122+-- of different sizes, we can handle that in the stream comparison abstraction.
123+-- This could be useful e.g. to fast compare whether two files differ.
124+
125+-- | Convert a stream of arrays into a stream of their elements.
126+--
127+-- Same as the following:
128+--
129+-- > concat = Stream.unfoldMany Array.read
130+--
131+-- @since 0.7.0
132+{-# INLINE concat #-}
133+concat :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
134+-- concat m = fromStreamD $ A.flattenArrays (toStreamD m)
135+-- concat m = fromStreamD $ D.concatMap A.toStreamD (toStreamD m)
136+concat = D.unfoldMany A.reader
137+
138+-- | Convert a stream of arrays into a stream of their elements reversing the
139+-- contents of each array before flattening.
140+--
141+-- > concatRev = Stream.unfoldMany Array.readerRev
142+--
143+-- @since 0.7.0
144+{-# INLINE concatRev #-}
145+concatRev :: (Monad m, Unbox a) => Stream m (Array a) -> Stream m a
146+-- concatRev m = fromStreamD $ A.flattenArraysRev (toStreamD m)
147+concatRev = D.unfoldMany A.readerRev
148+
112149-------------------------------------------------------------------------------
113150-- Intersperse and append
114151-------------------------------------------------------------------------------
115152
153+-- | Flatten a stream of arrays after inserting the given element between
154+-- arrays.
155+--
156+-- /Pre-release/
157+{-# INLINE interpose #-}
158+interpose :: (Monad m, Unbox a) => a -> Stream m (Array a) -> Stream m a
159+interpose x = D.interpose x A.reader
160+
161+{-# INLINE intercalateSuffix #-}
162+intercalateSuffix :: (Monad m, Unbox a)
163+ => Array a -> Stream m (Array a) -> Stream m a
164+intercalateSuffix = D.intercalateSuffix A.reader
165+
166+-- | Flatten a stream of arrays appending the given element after each
167+-- array.
168+--
169+-- @since 0.7.0
170+{-# INLINE interposeSuffix #-}
171+interposeSuffix :: (Monad m, Unbox a)
172+ => a -> Stream m (Array a) -> Stream m a
173+-- interposeSuffix x = fromStreamD . A.unlines x . toStreamD
174+interposeSuffix x = D.interposeSuffix x A.reader
175+
176+data FlattenState s =
177+ OuterLoop s
178+ | InnerLoop s !MA.MutableByteArray !Int !Int
179+
180+-- XXX This is a special case of interposeSuffix, can be removed.
181+-- XXX Remove monadIO constraint
116182{-# INLINE_NORMAL unlines #-}
117183unlines :: forall m a. (MonadIO m, Unbox a)
118184 => a -> D.Stream m (Array a) -> D.Stream m a
119−unlines = Array.interposeSuffix
185+unlines sep (D.Stream step state) = D.Stream step' (OuterLoop state)
186+ where
187+ {-# INLINE_LATE step' #-}
188+ step' gst (OuterLoop st) = do
189+ r <- step (adaptState gst) st
190+ return $ case r of
191+ D.Yield Array{..} s ->
192+ D.Skip (InnerLoop s arrContents arrStart arrEnd)
193+ D.Skip s -> D.Skip (OuterLoop s)
194+ D.Stop -> D.Stop
195+
196+ step' _ (InnerLoop st _ p end) | p == end =
197+ return $ D.Yield sep $ OuterLoop st
198+
199+ step' _ (InnerLoop st contents p end) = do
200+ x <- liftIO $ peekByteIndex p contents
201+ return $ D.Yield x (InnerLoop st contents (INDEX_NEXT(p,a)) end)
120202
121203-------------------------------------------------------------------------------
122204-- Compact
123205-------------------------------------------------------------------------------
124206
207+-- XXX These would not be needed once we implement compactLEFold, see
208+-- module Streamly.Internal.Data.Stream.MutChunked
209+--
210+-- XXX Note that this thaws immutable arrays for appending, that may be
211+-- problematic if multiple users do the same thing, however, immutable arrays
212+-- would usually have no capacity to append, therefore, a copy will be forced
213+-- anyway. Confirm this. We can forcefully trim the array capacity before thaw
214+-- to ensure this.
215+{-# INLINE_NORMAL packArraysChunksOf #-}
216+packArraysChunksOf :: (MonadIO m, Unbox a)
217+ => Int -> D.Stream m (Array a) -> D.Stream m (Array a)
218+packArraysChunksOf n str =
219+ D.map A.unsafeFreeze $ AS.packArraysChunksOf n $ D.map A.unsafeThaw str
220+
125221-- XXX instead of writing two different versions of this operation, we should
126222-- write it as a pipe.
127223--
@@ -129,7 +225,8 @@ unlines = Array.interposeSuffix
129225{-# INLINE_NORMAL lpackArraysChunksOf #-}
130226lpackArraysChunksOf :: (MonadIO m, Unbox a)
131227 => Int -> Fold m (Array a) () -> Fold m (Array a) ()
132−lpackArraysChunksOf = Array.lCompactGE
228+lpackArraysChunksOf n fld =
229+ FL.lmap A.unsafeThaw $ AS.lpackArraysChunksOf n (FL.lmap A.unsafeFreeze fld)
133230
134231-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
135232-- maximum specified size in bytes.
@@ -138,20 +235,69 @@ lpackArraysChunksOf = Array.lCompactGE
138235{-# INLINE compact #-}
139236compact :: (MonadIO m, Unbox a)
140237 => Int -> Stream m (Array a) -> Stream m (Array a)
141−compact = Array.compactLE
142−
143−-- | Given a stream of arrays, splice them all together to generate a single
144−-- array. The stream must be /finite/.
145−--
146−-- @since 0.7.0
147−{-# INLINE toArray #-}
148−toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
149−toArray = Array.fromChunks
238+compact = packArraysChunksOf
150239
151240-------------------------------------------------------------------------------
152241-- Split
153242-------------------------------------------------------------------------------
154243
244+data SplitState s arr
245+ = Initial s
246+ | Buffering s arr
247+ | Splitting s arr
248+ | Yielding arr (SplitState s arr)
249+ | Finishing
250+
251+-- | Split a stream of arrays on a given separator byte, dropping the separator
252+-- and coalescing all the arrays between two separators into a single array.
253+--
254+-- @since 0.7.0
255+{-# INLINE_NORMAL _splitOn #-}
256+_splitOn
257+ :: MonadIO m
258+ => Word8
259+ -> D.Stream m (Array Word8)
260+ -> D.Stream m (Array Word8)
261+_splitOn byte (D.Stream step state) = D.Stream step' (Initial state)
262+
263+ where
264+
265+ {-# INLINE_LATE step' #-}
266+ step' gst (Initial st) = do
267+ r <- step gst st
268+ case r of
269+ D.Yield arr s -> do
270+ (arr1, marr2) <- A.breakOn byte arr
271+ return $ case marr2 of
272+ Nothing -> D.Skip (Buffering s arr1)
273+ Just arr2 -> D.Skip (Yielding arr1 (Splitting s arr2))
274+ D.Skip s -> return $ D.Skip (Initial s)
275+ D.Stop -> return D.Stop
276+
277+ step' gst (Buffering st buf) = do
278+ r <- step gst st
279+ case r of
280+ D.Yield arr s -> do
281+ (arr1, marr2) <- A.breakOn byte arr
282+ buf' <- A.splice buf arr1
283+ return $ case marr2 of
284+ Nothing -> D.Skip (Buffering s buf')
285+ Just x -> D.Skip (Yielding buf' (Splitting s x))
286+ D.Skip s -> return $ D.Skip (Buffering s buf)
287+ D.Stop -> return $
288+ if A.byteLength buf == 0
289+ then D.Stop
290+ else D.Skip (Yielding buf Finishing)
291+
292+ step' _ (Splitting st buf) = do
293+ (arr1, marr2) <- A.breakOn byte buf
294+ return $ case marr2 of
295+ Nothing -> D.Skip $ Buffering st arr1
296+ Just arr2 -> D.Skip $ Yielding arr1 (Splitting st arr2)
297+
298+ step' _ (Yielding arr next) = return $ D.Yield arr next
299+ step' _ Finishing = return D.Stop
300+
155301-- XXX Remove MonadIO constraint.
156302-- | Split a stream of arrays on a given separator byte, dropping the separator
157303-- and coalescing all the arrays between two separators into a single array.
@@ -163,7 +309,7 @@ splitOn
163309 => Word8
164310 -> Stream m (Array Word8)
165311 -> Stream m (Array Word8)
166−splitOn = Array.compactOnByte
312+splitOn byte = D.splitInnerBy (A.breakOn byte) A.splice
167313
168314{-# INLINE splitOnSuffix #-}
169315splitOnSuffix
@@ -171,7 +317,8 @@ splitOnSuffix
171317 => Word8
172318 -> Stream m (Array Word8)
173319 -> Stream m (Array Word8)
174−splitOnSuffix = Array.compactOnByteSuffix
320+-- splitOn byte s = fromStreamD $ A.splitOn byte $ toStreamD s
321+splitOnSuffix byte = D.splitInnerBySuffix (A.breakOn byte) A.splice
175322
176323-------------------------------------------------------------------------------
177324-- Elimination - Running folds
@@ -180,24 +327,77 @@ splitOnSuffix = Array.compactOnByteSuffix
180327{-# INLINE_NORMAL foldBreakD #-}
181328foldBreakD :: forall m a b. (MonadIO m, Unbox a) =>
182329 Fold m a b -> D.Stream m (Array a) -> m (b, D.Stream m (Array a))
183−foldBreakD = Array.foldBreakChunks
330+foldBreakD (FL.Fold fstep initial extract) stream@(D.Stream step state) = do
331+ res <- initial
332+ case res of
333+ FL.Partial fs -> go SPEC state fs
334+ FL.Done fb -> return $! (fb, stream)
335+
336+ where
337+
338+ {-# INLINE go #-}
339+ go !_ st !fs = do
340+ r <- step defState st
341+ case r of
342+ D.Yield (Array contents start end) s ->
343+ let fp = Tuple' end contents
344+ in goArray SPEC s fp start fs
345+ D.Skip s -> go SPEC s fs
346+ D.Stop -> do
347+ b <- extract fs
348+ return (b, D.nil)
349+
350+ goArray !_ s (Tuple' end _) !cur !fs
351+ | cur == end = do
352+ go SPEC s fs
353+ goArray !_ st fp@(Tuple' end contents) !cur !fs = do
354+ x <- liftIO $ peekByteIndex cur contents
355+ res <- fstep fs x
356+ let next = INDEX_NEXT(cur,a)
357+ case res of
358+ FL.Done b -> do
359+ let arr = Array contents next end
360+ return $! (b, D.cons arr (D.Stream step st))
361+ FL.Partial fs1 -> goArray SPEC st fp next fs1
362+
363+{-# INLINE_NORMAL foldBreakK #-}
364+foldBreakK :: forall m a b. (MonadIO m, Unbox a) =>
365+ Fold m a b -> K.StreamK m (Array a) -> m (b, K.StreamK m (Array a))
366+foldBreakK (FL.Fold fstep initial extract) stream = do
367+ res <- initial
368+ case res of
369+ FL.Partial fs -> go fs stream
370+ FL.Done fb -> return (fb, stream)
371+
372+ where
373+
374+ {-# INLINE go #-}
375+ go !fs st = do
376+ let stop = (, K.nil) <$> extract fs
377+ single a = yieldk a K.nil
378+ yieldk (Array contents start end) r =
379+ let fp = Tuple' end contents
380+ in goArray fs r fp start
381+ in K.foldStream defState yieldk single stop st
382+
383+ goArray !fs st (Tuple' end _) !cur
384+ | cur == end = do
385+ go fs st
386+ goArray !fs st fp@(Tuple' end contents) !cur = do
387+ x <- liftIO $ peekByteIndex cur contents
388+ res <- fstep fs x
389+ let next = INDEX_NEXT(cur,a)
390+ case res of
391+ FL.Done b -> do
392+ let arr = Array contents next end
393+ return $! (b, K.cons arr st)
394+ FL.Partial fs1 -> goArray fs1 st fp next
184395
185396-- | Fold an array stream using the supplied 'Fold'. Returns the fold result
186397-- and the unconsumed stream.
187398--
188399-- > foldBreak f = runArrayFoldBreak (ChunkFold.fromFold f)
189400--
190−-- Instead of using this we can adapt the fold to ParserK and use
191−-- parseBreakChunks instead. ParserK allows composing using Monad as well.
192−--
193−-- @
194−-- foldBreak f s =
195−-- fmap (first (fromRight undefined))
196−-- $ K.parseBreakChunks (ParserK.adaptC (PR.fromFold f)) s
197−-- @
198−--
199−-- We can compare perf and remove this one or define it in terms of that.
200−--
201401-- /Internal/
202402--
203403{-# INLINE_NORMAL foldBreak #-}
@@ -206,20 +406,14 @@ foldBreak ::
206406 => Fold m a b
207407 -> StreamK m (A.Array a)
208408 -> m (b, StreamK m (A.Array a))
209−foldBreak = Array.foldBreakChunksK
210−--
211409-- foldBreak f s = fmap fromStreamD <$> foldBreakD f (toStreamD s)
212−--
213−-- foldBreak f s =
214−-- fmap (first (fromRight undefined))
215−-- $ K.parseBreakChunks (ParserK.adaptC (PR.fromFold f)) s
216−--
410+foldBreak = foldBreakK
217411-- If foldBreak performs better than runArrayFoldBreak we can use a rewrite
218412-- rule to rewrite runArrayFoldBreak to fold.
219413-- foldBreak f = runArrayFoldBreak (ChunkFold.fromFold f)
220414
221415-------------------------------------------------------------------------------
222−-- Elimination - running element parsers
416+-- Fold to a single Array
223417-------------------------------------------------------------------------------
224418
225419-- When we have to take an array partially, take the last part of the array.
@@ -265,22 +459,341 @@ splitAtArrayListRev n ls
265459 arr1 = Array contents end1 end
266460 in ([arr1], arr2:xs)
267461
462+-------------------------------------------------------------------------------
463+-- Fold to a single Array
464+-------------------------------------------------------------------------------
465+
466+{-
467+
468+-- XXX Both of these implementations of splicing seem to perform equally well.
469+-- We need to perform benchmarks over a range of sizes though.
470+
471+-- CAUTION! length must more than equal to lengths of all the arrays in the
472+-- stream.
473+{-# INLINE spliceArraysLenUnsafe #-}
474+spliceArraysLenUnsafe :: (MonadIO m, Unbox a)
475+ => Int -> Stream m (MutArray a) -> m (MutArray a)
476+spliceArraysLenUnsafe len buffered = do
477+ -- XXX The new array's pinned state should depend on the first element
478+ -- of the stream. We should uncons the stream and do the required.
479+ arr <- undefined
480+ D.foldlM' MA.spliceUnsafe (return arr) buffered
481+
482+{-# INLINE _spliceArrays #-}
483+_spliceArrays :: (MonadIO m, Unbox a)
484+ => Stream m (Array a) -> m (Array a)
485+_spliceArrays s = do
486+ buffered <- D.foldr K.cons K.nil s
487+ len <- K.fold FL.sum (fmap Array.length buffered)
488+ -- XXX The new array's pinned state should depend on the first element
489+ -- of the stream. We should uncons the stream and do the required.
490+ arr <- undefined
491+ final <- D.foldlM' writeArr (return arr) (toStream buffered)
492+ return $ A.unsafeFreeze final
493+
494+ where
495+
496+ writeArr dst arr = MA.spliceUnsafe dst (A.unsafeThaw arr)
497+
498+{-# INLINE _spliceArraysBuffered #-}
499+_spliceArraysBuffered :: (MonadIO m, Unbox a)
500+ => Stream m (Array a) -> m (Array a)
501+_spliceArraysBuffered s = do
502+ buffered <- D.foldr K.cons K.nil s
503+ len <- K.fold FL.sum (fmap Array.length buffered)
504+ A.unsafeFreeze <$>
505+ spliceArraysLenUnsafe len (fmap A.unsafeThaw (toStream buffered))
506+
507+-}
508+
509+{-# INLINE spliceArraysRealloced #-}
510+spliceArraysRealloced :: forall m a. (MonadIO m, Unbox a)
511+ => Stream m (Array a) -> m (Array a)
512+spliceArraysRealloced s = do
513+ res <- D.uncons s
514+ case res of
515+ Just (a, strm) -> do
516+ arr <-
517+ D.foldlM'
518+ MA.spliceExp
519+ (pure (A.unsafeThaw a))
520+ (fmap A.unsafeThaw strm)
521+ liftIO $ A.unsafeFreeze <$> MA.rightSize arr
522+ Nothing -> pure A.nil
523+
524+-- XXX This should just be "fold A.write"
525+--
526+-- | Given a stream of arrays, splice them all together to generate a single
527+-- array. The stream must be /finite/.
528+--
529+-- @since 0.7.0
530+{-# INLINE toArray #-}
531+toArray :: (MonadIO m, Unbox a) => Stream m (Array a) -> m (Array a)
532+toArray = spliceArraysRealloced
533+-- spliceArrays = _spliceArraysBuffered
534+
535+-- exponentially increasing sizes of the chunks upto the max limit.
536+-- XXX this will be easier to implement with parsers/terminating folds
537+-- With this we should be able to reduce the number of chunks/allocations.
538+-- The reallocation/copy based toArray can also be implemented using this.
539+--
540+{-
541+{-# INLINE toArraysInRange #-}
542+toArraysInRange :: (MonadIO m, Unbox a)
543+ => Int -> Int -> Fold m (Array a) b -> Fold m a b
544+toArraysInRange low high (Fold step initial extract) =
545+-}
546+
547+{-
548+-- | Fold the input to a pure buffered stream (List) of arrays.
549+{-# INLINE _toArraysOf #-}
550+_toArraysOf :: (MonadIO m, Unbox a)
551+ => Int -> Fold m a (Stream Identity (Array a))
552+_toArraysOf n = FL.groupsOf n (A.writeNF n) FL.toStream
553+-}
554+
555+-------------------------------------------------------------------------------
556+-- Elimination - running element parsers
557+-------------------------------------------------------------------------------
558+
268559-- GHC parser does not accept {-# ANN type [] NoSpecConstr #-}, so we need
269560-- to make a newtype.
270561{-# ANN type List NoSpecConstr #-}
271562newtype List a = List {getList :: [a]}
272563
564+{-
565+-- This can be generalized to any type provided it can be unfolded to a stream
566+-- and it can be combined using a semigroup operation.
567+--
568+-- XXX This should be written using CPS (as parseK) if we want it to scale wrt
569+-- to the number of times it can be called on the same stream.
570+{-# INLINE_NORMAL parseBreakD #-}
571+parseBreakD ::
572+ forall m a b. (MonadIO m, MonadThrow m, Unbox a)
573+ => PRD.Parser a m b
574+ -> D.Stream m (Array.Array a)
575+ -> m (b, D.Stream m (Array.Array a))
576+parseBreakD
577+ (PRD.Parser pstep initial extract) stream@(D.Stream step state) = do
578+
579+ res <- initial
580+ case res of
581+ PRD.IPartial s -> go SPEC state (List []) s
582+ PRD.IDone b -> return (b, stream)
583+ PRD.IError err -> throwM $ ParseError err
584+
585+ where
586+
587+ -- "backBuf" contains last few items in the stream that we may have to
588+ -- backtrack to.
589+ --
590+ -- XXX currently we are using a dumb list based approach for backtracking
591+ -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
592+ -- That will allow us more efficient random back and forth movement.
593+ go !_ st backBuf !pst = do
594+ r <- step defState st
595+ case r of
596+ D.Yield (Array contents start end) s ->
597+ gobuf SPEC s backBuf
598+ (Tuple' end contents) start pst
599+ D.Skip s -> go SPEC s backBuf pst
600+ D.Stop -> do
601+ b <- extract pst
602+ return (b, D.nil)
603+
604+ -- Use strictness on "cur" to keep it unboxed
605+ gobuf !_ s backBuf (Tuple' end _) !cur !pst
606+ | cur == end = do
607+ go SPEC s backBuf pst
608+ gobuf !_ s backBuf fp@(Tuple' end contents) !cur !pst = do
609+ x <- liftIO $ peekByteIndex contents cur
610+ pRes <- pstep pst x
611+ let next = INDEX_NEXT(cur,a)
612+ case pRes of
613+ PR.Partial 0 pst1 ->
614+ gobuf SPEC s (List []) fp next pst1
615+ PR.Partial n pst1 -> do
616+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
617+ let src0 = Prelude.take n (x:getList backBuf)
618+ arr0 = A.fromListN n (Prelude.reverse src0)
619+ arr1 = Array contents next end
620+ src = arr0 <> arr1
621+ let !(Array cont1 start end1) = src
622+ fp1 = Tuple' end1 cont1
623+ gobuf SPEC s (List []) fp1 start pst1
624+ PR.Continue 0 pst1 ->
625+ gobuf SPEC s (List (x:getList backBuf)) fp next pst1
626+ PR.Continue n pst1 -> do
627+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
628+ let (src0, buf1) = splitAt n (x:getList backBuf)
629+ arr0 = A.fromListN n (Prelude.reverse src0)
630+ arr1 = Array contents next end
631+ src = arr0 <> arr1
632+ let !(Array cont1 start end1) = src
633+ fp1 = Tuple' end1 cont1
634+ gobuf SPEC s (List buf1) fp1 start pst1
635+ PR.Done 0 b -> do
636+ let arr = Array contents next end
637+ return (b, D.cons arr (D.Stream step s))
638+ PR.Done n b -> do
639+ assert (n <= Prelude.length (x:getList backBuf)) (return ())
640+ let src0 = Prelude.take n (x:getList backBuf)
641+ -- XXX create the array in reverse instead
642+ arr0 = A.fromListN n (Prelude.reverse src0)
643+ arr1 = Array contents next end
644+ -- XXX Use StreamK to avoid adding arbitrary layers of
645+ -- constructors every time.
646+ str = D.cons arr0 (D.cons arr1 (D.Stream step s))
647+ return (b, str)
648+ PR.Error err -> throwM $ ParseError err
649+-}
650+
651+{-# INLINE_NORMAL parseBreakK #-}
652+parseBreakK ::
653+ forall m a b. (MonadIO m, Unbox a)
654+ => PRD.Parser a m b
655+ -> K.StreamK m (Array.Array a)
656+ -> m (Either ParseError b, K.StreamK m (Array.Array a))
657+parseBreakK (PRD.Parser pstep initial extract) stream = do
658+ res <- initial
659+ case res of
660+ PRD.IPartial s -> go s stream []
661+ PRD.IDone b -> return (Right b, stream)
662+ PRD.IError err -> return (Left (ParseError err), stream)
663+
664+ where
665+
666+ -- "backBuf" contains last few items in the stream that we may have to
667+ -- backtrack to.
668+ --
669+ -- XXX currently we are using a dumb list based approach for backtracking
670+ -- buffer. This can be replaced by a sliding/ring buffer using Data.Array.
671+ -- That will allow us more efficient random back and forth movement.
672+ go !pst st backBuf = do
673+ let stop = goStop pst backBuf -- (, K.nil) <$> extract pst
674+ single a = yieldk a K.nil
675+ yieldk arr r = goArray pst backBuf r arr
676+ in K.foldStream defState yieldk single stop st
677+
678+ -- Use strictness on "cur" to keep it unboxed
679+ goArray !pst backBuf st (Array _ cur end) | cur == end = go pst st backBuf
680+ goArray !pst backBuf st (Array contents cur end) = do
681+ x <- liftIO $ peekByteIndex cur contents
682+ pRes <- pstep pst x
683+ let next = INDEX_NEXT(cur,a)
684+ case pRes of
685+ PR.Partial 0 s ->
686+ goArray s [] st (Array contents next end)
687+ PR.Partial n s -> do
688+ assert (n <= Prelude.length (x:backBuf)) (return ())
689+ let src0 = Prelude.take n (x:backBuf)
690+ arr0 = A.fromListN n (Prelude.reverse src0)
691+ arr1 = Array contents next end
692+ src = arr0 <> arr1
693+ goArray s [] st src
694+ PR.Continue 0 s ->
695+ goArray s (x:backBuf) st (Array contents next end)
696+ PR.Continue n s -> do
697+ assert (n <= Prelude.length (x:backBuf)) (return ())
698+ let (src0, buf1) = splitAt n (x:backBuf)
699+ arr0 = A.fromListN n (Prelude.reverse src0)
700+ arr1 = Array contents next end
701+ src = arr0 <> arr1
702+ goArray s buf1 st src
703+ PR.Done 0 b -> do
704+ let arr = Array contents next end
705+ return (Right b, K.cons arr st)
706+ PR.Done n b -> do
707+ assert (n <= Prelude.length (x:backBuf)) (return ())
708+ let src0 = Prelude.take n (x:backBuf)
709+ -- XXX Use fromListRevN once implemented
710+ -- arr0 = A.fromListRevN n src0
711+ arr0 = A.fromListN n (Prelude.reverse src0)
712+ arr1 = Array contents next end
713+ str = K.cons arr0 (K.cons arr1 st)
714+ return (Right b, str)
715+ PR.Error err -> do
716+ let n = Prelude.length backBuf
717+ arr0 = A.fromListN n (Prelude.reverse backBuf)
718+ arr1 = Array contents cur end
719+ str = K.cons arr0 (K.cons arr1 stream)
720+ return (Left (ParseError err), str)
721+
722+ -- This is a simplified goArray
723+ goExtract !pst backBuf (Array _ cur end)
724+ | cur == end = goStop pst backBuf
725+ goExtract !pst backBuf (Array contents cur end) = do
726+ x <- liftIO $ peekByteIndex cur contents
727+ pRes <- pstep pst x
728+ let next = INDEX_NEXT(cur,a)
729+ case pRes of
730+ PR.Partial 0 s ->
731+ goExtract s [] (Array contents next end)
732+ PR.Partial n s -> do
733+ assert (n <= Prelude.length (x:backBuf)) (return ())
734+ let src0 = Prelude.take n (x:backBuf)
735+ arr0 = A.fromListN n (Prelude.reverse src0)
736+ arr1 = Array contents next end
737+ src = arr0 <> arr1
738+ goExtract s [] src
739+ PR.Continue 0 s ->
740+ goExtract s backBuf (Array contents next end)
741+ PR.Continue n s -> do
742+ assert (n <= Prelude.length (x:backBuf)) (return ())
743+ let (src0, buf1) = splitAt n (x:backBuf)
744+ arr0 = A.fromListN n (Prelude.reverse src0)
745+ arr1 = Array contents next end
746+ src = arr0 <> arr1
747+ goExtract s buf1 src
748+ PR.Done 0 b -> do
749+ let arr = Array contents next end
750+ return (Right b, K.fromPure arr)
751+ PR.Done n b -> do
752+ assert (n <= Prelude.length backBuf) (return ())
753+ let src0 = Prelude.take n backBuf
754+ -- XXX Use fromListRevN once implemented
755+ -- arr0 = A.fromListRevN n src0
756+ arr0 = A.fromListN n (Prelude.reverse src0)
757+ arr1 = Array contents next end
758+ str = K.cons arr0 (K.fromPure arr1)
759+ return (Right b, str)
760+ PR.Error err -> do
761+ let n = Prelude.length backBuf
762+ arr0 = A.fromListN n (Prelude.reverse backBuf)
763+ arr1 = Array contents cur end
764+ str = K.cons arr0 (K.cons arr1 stream)
765+ return (Left (ParseError err), str)
766+
767+ -- This is a simplified goExtract
768+ {-# INLINE goStop #-}
769+ goStop !pst backBuf = do
770+ pRes <- extract pst
771+ case pRes of
772+ PR.Partial _ _ -> error "Bug: parseBreak: Partial in extract"
773+ PR.Continue 0 s ->
774+ goStop s backBuf
775+ PR.Continue n s -> do
776+ assert (n <= Prelude.length backBuf) (return ())
777+ let (src0, buf1) = splitAt n backBuf
778+ arr = A.fromListN n (Prelude.reverse src0)
779+ goExtract s buf1 arr
780+ PR.Done 0 b ->
781+ return (Right b, K.nil)
782+ PR.Done n b -> do
783+ assert (n <= Prelude.length backBuf) (return ())
784+ let src0 = Prelude.take n backBuf
785+ -- XXX Use fromListRevN once implemented
786+ -- arr0 = A.fromListRevN n src0
787+ arr0 = A.fromListN n (Prelude.reverse src0)
788+ return (Right b, K.fromPure arr0)
789+ PR.Error err -> do
790+ let n = Prelude.length backBuf
791+ arr0 = A.fromListN n (Prelude.reverse backBuf)
792+ return (Left (ParseError err), K.fromPure arr0)
793+
273794-- | Parse an array stream using the supplied 'Parser'. Returns the parse
274795-- result and the unconsumed stream. Throws 'ParseError' if the parse fails.
275796--
276−-- >> parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
277−--
278−-- This is redundant and we can just use parseBreakChunks, as ParserK can be
279−-- composed using Monad. The only advantage of this is that we do not need to
280−-- adapt.
281−--
282−-- We can compare perf and remove this one or define it in terms of that.
283−--
284797-- /Internal/
285798--
286799{-# INLINE_NORMAL parseBreak #-}
@@ -293,8 +806,7 @@ parseBreak ::
293806parseBreak p s =
294807 fmap fromStreamD <$> parseBreakD (PRD.fromParserK p) (toStreamD s)
295808-}
296−parseBreak = Array.parseBreakChunksK
297−-- parseBreak p = K.parseBreakChunks (ParserK.adaptC p)
809+parseBreak = parseBreakK
298810
299811-------------------------------------------------------------------------------
300812-- Elimination - Running Array Folds and parsers
src/Streamly/Internal/Data/Stream/Common.hs → core/src/Streamly/Internal/Data/Stream/Common.hsrenamed+3−3
@@ -30,10 +30,10 @@ where
3030
3131#include "inline.hs"
3232
33−import Streamly.Data.Fold (Fold)
33+import Streamly.Internal.Data.Fold.Type (Fold (..))
3434
35−import qualified Streamly.Data.StreamK as K
36−import qualified Streamly.Internal.Data.Stream as D
35+import qualified Streamly.Internal.Data.StreamK.Type as K
36+import qualified Streamly.Internal.Data.Stream.Type as D
3737
3838import Prelude hiding (Foldable(..), repeat)
3939
core/src/Streamly/Internal/Data/Stream/Container.hs+48−56
@@ -11,17 +11,16 @@
1111
1212module Streamly.Internal.Data.Stream.Container
1313 (
14− -- * Deduplication
15− ordNub
14+ nub
1615
17− -- * Joins
18− , leftJoin
19− , outerJoin
16+ -- * Joins for unconstrained types
17+ , joinLeftGeneric
18+ , joinOuterGeneric
2019
21− -- * Ord Joins
22− , innerOrdJoin
23− , leftOrdJoin
24− , outerOrdJoin
20+ -- * Joins with Ord constraint
21+ , joinInner
22+ , joinLeft
23+ , joinOuter
2524 )
2625where
2726
@@ -48,15 +47,12 @@ import qualified Streamly.Internal.Data.Stream.Transformer as Stream
4847
4948#include "DocTestDataStream.hs"
5049
51−-- | @nub@ specialized to 'Ord' types for better performance. Returns a
52−-- subsequence of the stream removing any duplicate elements.
53−--
54−-- The memory used is proportional to the number of unique elements in the
55−-- stream. One way to limit the memory is to use @take@ on the resulting
56−-- stream to limit the unique elements in the stream.
57−{-# INLINE_NORMAL ordNub #-}
58−ordNub :: (Monad m, Ord a) => Stream m a -> Stream m a
59−ordNub (Stream step1 state1) = Stream step (Set.empty, state1)
50+-- | The memory used is proportional to the number of unique elements in the
51+-- stream. If we want to limit the memory we can just use "take" to limit the
52+-- uniq elements in the stream.
53+{-# INLINE_NORMAL nub #-}
54+nub :: (Monad m, Ord a) => Stream m a -> Stream m a
55+nub (Stream step1 state1) = Stream step (Set.empty, state1)
6056
6157 where
6258
@@ -82,8 +78,8 @@ toMap =
8278--
8379-- XXX An IntMap may be faster when the keys are Int.
8480-- XXX Use hashmap instead of map?
85−
86−-- | 'innerJoin' specialized to 'Ord' types for better performance.
81+--
82+-- | Like 'joinInner' but uses a 'Map' for efficiency.
8783--
8884-- If the input streams have duplicate keys, the behavior is undefined.
8985--
@@ -94,10 +90,10 @@ toMap =
9490-- Time: O(m + n)
9591--
9692-- /Pre-release/
97−{-# INLINE innerOrdJoin #-}
98−innerOrdJoin :: (Monad m, Ord k) =>
93+{-# INLINE joinInner #-}
94+joinInner :: (Monad m, Ord k) =>
9995 Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, b)
100−innerOrdJoin s1 s2 =
96+joinInner s1 s2 =
10197 Stream.concatEffect $ do
10298 km <- toMap s2
10399 pure $ Stream.mapMaybe (joinAB km) s1
@@ -110,33 +106,32 @@ innerOrdJoin s1 s2 =
110106 Nothing -> Nothing
111107
112108-- XXX We can do this concurrently.
113−-- XXX Check performance of StreamD vs StreamK
114109-- XXX If the second stream is sorted and passed as an Array or a seek capable
115110-- stream then we could use binary search if we have an Ord instance or
116111-- Ordering returning function. The time complexity would then become (m x log
117112-- n).
118113
119−-- | Like 'innerJoin' but emits @(a, Just b)@ whenever a and b are equal, for
120−-- those @a@'s that are not equal to any @b@ emits @(a, Nothing)@.
121−--
122−-- This is a generalization of 'innerJoin' to include all elements from the
123−-- left stream and not just those which have an equal in the right stream. This
124−-- is not a commutative operation, the order of the stream arguments matters.
114+-- XXX Check performance of StreamD vs StreamK
115+
116+-- | Like 'joinInner' but emit @(a, Just b)@, and additionally, for those @a@'s
117+-- that are not equal to any @b@ emit @(a, Nothing)@.
125118--
126−-- All the caveats mentioned in 'innerJoin' apply here as well. Right join is
127−-- not provided because it is just a flipped left join:
119+-- The second stream is evaluated multiple times. If the stream is a
120+-- consume-once stream then the caller should cache it in an 'Data.Array.Array'
121+-- before calling this function. Caching may also improve performance if the
122+-- stream is expensive to evaluate.
128123--
129−-- >>> rightJoin eq = flip (Stream.leftJoin eq)
124+-- >>> joinRightGeneric eq = flip (Stream.joinLeftGeneric eq)
130125--
131126-- Space: O(n) assuming the second stream is cached in memory.
132127--
133128-- Time: O(m x n)
134129--
135130-- /Unimplemented/
136−{-# INLINE leftJoin #-}
137−leftJoin :: Monad m =>
131+{-# INLINE joinLeftGeneric #-}
132+joinLeftGeneric :: Monad m =>
138133 (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
139−leftJoin eq s1 s2 = Stream.evalStateT (return False) $ unCross $ do
134+joinLeftGeneric eq s1 s2 = Stream.evalStateT (return False) $ unCross $ do
140135 a <- mkCross (Stream.liftInner s1)
141136 -- XXX should we use StreamD monad here?
142137 -- XXX Is there a better way to perform some action at the end of a loop
@@ -157,17 +152,19 @@ leftJoin eq s1 s2 = Stream.evalStateT (return False) $ unCross $ do
157152 else mkCross Stream.nil
158153 Nothing -> return (a, Nothing)
159154
160−-- | 'leftJoin' specialized to 'Ord' types for better performance.
155+-- XXX rename to joinLeftOrd?
156+
157+-- | A more efficient 'joinLeft' using a hashmap for efficiency.
161158--
162159-- Space: O(n)
163160--
164161-- Time: O(m + n)
165162--
166163-- /Pre-release/
167−{-# INLINE leftOrdJoin #-}
168−leftOrdJoin :: (Ord k, Monad m) =>
164+{-# INLINE joinLeft #-}
165+joinLeft :: (Ord k, Monad m) =>
169166 Stream m (k, a) -> Stream m (k, b) -> Stream m (k, a, Maybe b)
170−leftOrdJoin s1 s2 =
167+joinLeft s1 s2 =
171168 Stream.concatEffect $ do
172169 km <- toMap s2
173170 return $ fmap (joinAB km) s1
@@ -180,19 +177,14 @@ leftOrdJoin s1 s2 =
180177 Nothing -> (k, a, Nothing)
181178
182179-- XXX We can do this concurrently.
183−-- XXX Check performance of StreamD vs StreamK cross operation.
184180
185−-- | Like 'leftJoin' but emits a @(Just a, Just b)@. Like 'leftJoin', for those
181+-- XXX Check performance of StreamD vs StreamK
182+
183+-- | Like 'joinLeft' but emits a @(Just a, Just b)@. Like 'joinLeft', for those
186184-- @a@'s that are not equal to any @b@ emit @(Just a, Nothing)@, but
187185-- additionally, for those @b@'s that are not equal to any @a@ emit @(Nothing,
188186-- Just b)@.
189187--
190−-- This is a generalization of left join to include all the elements from the
191−-- right stream as well, in other words it is a combination of left and right
192−-- joins. This is a commutative operation. The order of stream arguments can be
193−-- changed without affecting results, except for the ordering of elements in
194−-- the resulting tuple.
195−--
196188-- For space efficiency use the smaller stream as the second stream.
197189--
198190-- Space: O(n)
@@ -200,15 +192,15 @@ leftOrdJoin s1 s2 =
200192-- Time: O(m x n)
201193--
202194-- /Pre-release/
203−{-# INLINE outerJoin #-}
204−outerJoin :: MonadIO m =>
195+{-# INLINE joinOuterGeneric #-}
196+joinOuterGeneric :: MonadIO m =>
205197 (a -> b -> Bool)
206198 -> Stream m a
207199 -> Stream m b
208200 -> Stream m (Maybe a, Maybe b)
209−outerJoin eq s1 s2 =
201+joinOuterGeneric eq s1 s =
210202 Stream.concatEffect $ do
211− inputArr <- Array.fromStream s2
203+ inputArr <- Array.fromStream s
212204 let len = Array.length inputArr
213205 foundArr <-
214206 Stream.fold
@@ -262,18 +254,18 @@ outerJoin eq s1 s2 =
262254-- a flag. At the end go through @Stream m b@ and find those that are not in that
263255-- hash to return (Nothing, b).
264256
265−-- | 'outerJoin' specialized to 'Ord' types for better performance.
257+-- | Like 'joinOuter' but uses a 'Map' for efficiency.
266258--
267259-- Space: O(m + n)
268260--
269261-- Time: O(m + n)
270262--
271263-- /Pre-release/
272−{-# INLINE outerOrdJoin #-}
273−outerOrdJoin ::
264+{-# INLINE joinOuter #-}
265+joinOuter ::
274266 (Ord k, MonadIO m) =>
275267 Stream m (k, a) -> Stream m (k, b) -> Stream m (k, Maybe a, Maybe b)
276−outerOrdJoin s1 s2 =
268+joinOuter s1 s2 =
277269 Stream.concatEffect $ do
278270 km1 <- kvFold s1
279271 km2 <- kvFold s2
core/src/Streamly/Internal/Data/Stream/Eliminate.hs+25−33
@@ -13,24 +13,36 @@
1313--
1414module Streamly.Internal.Data.Stream.Eliminate
1515 (
16− -- * Running a Parser
17− parse
16+ -- * Running a 'Fold'
17+ fold
18+
19+ -- -- * Running a 'Parser'
20+ , parse
1821 , parseD
1922 , parseBreak
2023 , parseBreakD
2124
22− -- * Deconstruction
25+ -- * Stream Deconstruction
2326 , uncons
2427
2528 -- * Right Folds
29+ , foldrM
30+ , foldr
31+ , foldrMx
2632 , foldr1
2733
34+ -- * Left Folds
35+ , foldlM'
36+ , foldl'
37+ , foldlMx'
38+ , foldlx'
39+
2840 -- * Specific Fold Functions
41+ , drain
2942 , mapM_ -- Map and Fold
3043 , null
3144 , head
3245 , headElse
33− , init
3446 , tail
3547 , last
3648 , elem
@@ -48,9 +60,16 @@ module Streamly.Internal.Data.Stream.Eliminate
4860 , the
4961
5062 -- * To containers
63+ , toList
5164 , toListRev
5265
5366 -- * Multi-Stream Folds
67+ -- ** Comparisons
68+ -- | These should probably be expressed using zipping operations.
69+ , eqBy
70+ , cmpBy
71+
72+ -- ** Substreams
5473 -- | These should probably be expressed using parsers.
5574 , isPrefixOf
5675 , isInfixOf
@@ -71,7 +90,7 @@ import Foreign.Storable (Storable)
7190import GHC.Exts (SpecConstrAnnotation(..))
7291import GHC.Types (SPEC(..))
7392import Streamly.Internal.Data.Parser (ParseError(..))
74−import Streamly.Internal.Data.SVar.Type (adaptState, defState)
93+import Streamly.Internal.Data.SVar.Type (defState)
7594import Streamly.Internal.Data.Unbox (Unbox)
7695
7796import Streamly.Internal.Data.Maybe.Strict (Maybe'(..))
@@ -86,7 +105,7 @@ import qualified Streamly.Internal.Data.Stream.Transform as StreamD
86105
87106import Prelude hiding
88107 ( Foldable(..), all, any, head, last, lookup, mapM, mapM_
89− , notElem, splitAt, init, tail, (!!))
108+ , notElem, splitAt, tail, (!!))
90109import Data.Foldable (length)
91110import Streamly.Internal.Data.Stream.Type
92111
@@ -372,35 +391,9 @@ head = foldrM (\x _ -> return (Just x)) (return Nothing)
372391headElse :: Monad m => a -> Stream m a -> m a
373392headElse a = foldrM (\x _ -> return x) (return a)
374393
375−{-# INLINE_NORMAL init #-}
376−init :: Monad m => Stream m a -> m (Maybe (Stream m a))
377−init stream = do
378− r <- uncons stream
379− case r of
380− Nothing -> return Nothing
381− Just (h, Stream step1 state1) ->
382− return $ Just $ Stream step (h, state1)
383−
384− where
385−
386− step gst (a, s1) = do
387− res <- step1 (adaptState gst) s1
388− return $
389− case res of
390− Yield x s -> Yield a (x, s)
391− Skip s -> Skip (a, s)
392− Stop -> Stop
393−
394−-- | Same as:
395−--
396−-- >>> tail = fmap (fmap snd) . Stream.uncons
397−--
398394-- Does not fuse, has the same performance as the StreamK version.
399−--
400395{-# INLINE_NORMAL tail #-}
401396tail :: Monad m => Stream m a -> m (Maybe (Stream m a))
402−tail = fmap (fmap snd) . uncons
403−{-
404397tail (UnStream step state) = go SPEC state
405398 where
406399 go !_ st = do
@@ -409,7 +402,6 @@ tail (UnStream step state) = go SPEC state
409402 Yield _ s -> return (Just $ Stream step s)
410403 Skip s -> go SPEC s
411404 Stop -> return Nothing
412−-}
413405
414406-- XXX will it fuse? need custom impl?
415407{-# INLINE_NORMAL last #-}
core/src/Streamly/Internal/Data/Stream/Exception.hs+6−9
@@ -9,20 +9,17 @@
99
1010module Streamly.Internal.Data.Stream.Exception
1111 (
12− -- * Resources
13− before
14− , afterIO
15− , afterUnsafe
16− , finallyIO
17− , finallyUnsafe
18− , gbracket_
12+ gbracket_
1913 , gbracket
14+ , before
15+ , afterUnsafe
16+ , afterIO
2017 , bracketUnsafe
2118 , bracketIO3
2219 , bracketIO
23−
24− -- * Exceptions
2520 , onException
21+ , finallyUnsafe
22+ , finallyIO
2623 , ghandle
2724 , handle
2825 )
core/src/Streamly/Internal/Data/Stream/Generate.hs+12−2
@@ -16,13 +16,20 @@ module Streamly.Internal.Data.Stream.Generate
1616 (
1717 -- * Primitives
1818 nil
19+ , nilM
1920 , cons
21+ , consM
22+
23+ -- * From 'Unfold'
24+ , unfold
2025
2126 -- * Unfolding
2227 , unfoldr
2328 , unfoldrM
2429
2530 -- * From Values
31+ , fromPure
32+ , fromEffect
2633 , repeat
2734 , repeatM
2835 , replicate
@@ -88,6 +95,7 @@ module Streamly.Internal.Data.Stream.Generate
8895 -- * From Containers
8996 -- | Transform an input structure into a stream.
9097
98+ , fromList
9199 , fromListM
92100 , fromFoldable
93101 , fromFoldableM
@@ -96,6 +104,10 @@ module Streamly.Internal.Data.Stream.Generate
96104 , fromPtr
97105 , fromPtrN
98106 , fromByteStr#
107+
108+ -- * Conversions
109+ , fromStreamK
110+ , toStreamK
99111 )
100112where
101113
@@ -142,8 +154,6 @@ import Streamly.Internal.Data.Stream.Type
142154nil :: Applicative m => Stream m a
143155nil = Stream (\_ _ -> pure Stop) ()
144156
145−infixr 5 `cons`
146−
147157-- XXX implement in terms of consM?
148158-- cons x = consM (return x)
149159-- From an implementation perspective, StreamK.cons translates into a
core/src/Streamly/Internal/Data/Stream/MutChunked.hsnew file+323
@@ -0,0 +1,323 @@
1+-- |
2+-- Module : Streamly.Internal.Data.Stream.MutChunked
3+-- Copyright : (c) 2019 Composewell Technologies
4+-- License : BSD3-3-Clause
5+-- Maintainer : streamly@composewell.com
6+-- Stability : experimental
7+-- Portability : GHC
8+--
9+-- Combinators to efficiently manipulate streams of mutable arrays.
10+--
11+module Streamly.Internal.Data.Stream.MutChunked
12+ (
13+ -- * Generation
14+ chunksOf
15+
16+ -- * Compaction
17+ , packArraysChunksOf
18+ , SpliceState (..)
19+ , lpackArraysChunksOf
20+ , compact
21+ , compactLE
22+ , compactEQ
23+ , compactGE
24+ )
25+where
26+
27+#include "inline.hs"
28+#include "ArrayMacros.h"
29+
30+import Control.Monad.IO.Class (MonadIO(..))
31+import Control.Monad (when)
32+import Data.Bifunctor (first)
33+import Data.Proxy (Proxy(..))
34+import Streamly.Internal.Data.Unbox (Unbox, sizeOf)
35+import Streamly.Internal.Data.MutArray.Type (MutArray(..))
36+import Streamly.Internal.Data.Fold.Type (Fold(..))
37+import Streamly.Internal.Data.Parser (ParseError)
38+import Streamly.Internal.Data.Stream.Type (Stream)
39+import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
40+
41+import qualified Streamly.Internal.Data.MutArray.Type as MArray
42+import qualified Streamly.Internal.Data.Fold.Type as FL
43+import qualified Streamly.Internal.Data.Parser as ParserD
44+import qualified Streamly.Internal.Data.Stream as D
45+
46+-- | @chunksOf n stream@ groups the elements in the input stream into arrays of
47+-- @n@ elements each.
48+--
49+-- Same as the following but may be more efficient:
50+--
51+-- > chunksOf n = Stream.foldMany (MArray.writeN n)
52+--
53+-- /Pre-release/
54+{-# INLINE chunksOf #-}
55+chunksOf :: (MonadIO m, Unbox a)
56+ => Int -> Stream m a -> Stream m (MutArray a)
57+chunksOf = MArray.chunksOf
58+
59+-------------------------------------------------------------------------------
60+-- Compact
61+-------------------------------------------------------------------------------
62+
63+data SpliceState s arr
64+ = SpliceInitial s
65+ | SpliceBuffering s arr
66+ | SpliceYielding arr (SpliceState s arr)
67+ | SpliceFinish
68+
69+-- XXX This can be removed once compactLEFold/compactLE are implemented.
70+--
71+-- | This mutates the first array (if it has space) to append values from the
72+-- second one. This would work for immutable arrays as well because an
73+-- immutable array never has space so a new array is allocated instead of
74+-- mutating it.
75+--
76+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
77+-- maximum specified size. Note that if a single array is bigger than the
78+-- specified size we do not split it to fit. When we coalesce multiple arrays
79+-- if the size would exceed the specified size we do not coalesce therefore the
80+-- actual array size may be less than the specified chunk size.
81+--
82+-- @since 0.7.0
83+{-# INLINE_NORMAL packArraysChunksOf #-}
84+packArraysChunksOf :: (MonadIO m, Unbox a)
85+ => Int -> D.Stream m (MutArray a) -> D.Stream m (MutArray a)
86+packArraysChunksOf n (D.Stream step state) =
87+ D.Stream step' (SpliceInitial state)
88+
89+ where
90+
91+ {-# INLINE_LATE step' #-}
92+ step' gst (SpliceInitial st) = do
93+ when (n <= 0) $
94+ -- XXX we can pass the module string from the higher level API
95+ error $ "Streamly.Internal.Data.MutArray.Type.packArraysChunksOf: the size of "
96+ ++ "arrays [" ++ show n ++ "] must be a natural number"
97+ r <- step gst st
98+ case r of
99+ D.Yield arr s -> return $
100+ let len = MArray.byteLength arr
101+ in if len >= n
102+ then D.Skip (SpliceYielding arr (SpliceInitial s))
103+ else D.Skip (SpliceBuffering s arr)
104+ D.Skip s -> return $ D.Skip (SpliceInitial s)
105+ D.Stop -> return D.Stop
106+
107+ step' gst (SpliceBuffering st buf) = do
108+ r <- step gst st
109+ case r of
110+ D.Yield arr s -> do
111+ let len = MArray.byteLength buf + MArray.byteLength arr
112+ if len > n
113+ then return $
114+ D.Skip (SpliceYielding buf (SpliceBuffering s arr))
115+ else do
116+ buf' <- if MArray.byteCapacity buf < n
117+ then liftIO $ MArray.realloc n buf
118+ else return buf
119+ buf'' <- MArray.splice buf' arr
120+ return $ D.Skip (SpliceBuffering s buf'')
121+ D.Skip s -> return $ D.Skip (SpliceBuffering s buf)
122+ D.Stop -> return $ D.Skip (SpliceYielding buf SpliceFinish)
123+
124+ step' _ SpliceFinish = return D.Stop
125+
126+ step' _ (SpliceYielding arr next) = return $ D.Yield arr next
127+
128+-- XXX Remove this once compactLEFold is implemented
129+-- lpackArraysChunksOf = Fold.many compactLEFold
130+--
131+{-# INLINE_NORMAL lpackArraysChunksOf #-}
132+lpackArraysChunksOf :: (MonadIO m, Unbox a)
133+ => Int -> Fold m (MutArray a) () -> Fold m (MutArray a) ()
134+lpackArraysChunksOf n (Fold step1 initial1 extract1) =
135+ Fold step initial extract
136+
137+ where
138+
139+ initial = do
140+ when (n <= 0) $
141+ -- XXX we can pass the module string from the higher level API
142+ error $ "Streamly.Internal.Data.MutArray.Type.packArraysChunksOf: the size of "
143+ ++ "arrays [" ++ show n ++ "] must be a natural number"
144+
145+ r <- initial1
146+ return $ first (Tuple' Nothing) r
147+
148+ extract (Tuple' Nothing r1) = extract1 r1
149+ extract (Tuple' (Just buf) r1) = do
150+ r <- step1 r1 buf
151+ case r of
152+ FL.Partial rr -> extract1 rr
153+ FL.Done _ -> return ()
154+
155+ step (Tuple' Nothing r1) arr =
156+ let len = MArray.byteLength arr
157+ in if len >= n
158+ then do
159+ r <- step1 r1 arr
160+ case r of
161+ FL.Done _ -> return $ FL.Done ()
162+ FL.Partial s -> do
163+ extract1 s
164+ res <- initial1
165+ return $ first (Tuple' Nothing) res
166+ else return $ FL.Partial $ Tuple' (Just arr) r1
167+
168+ step (Tuple' (Just buf) r1) arr = do
169+ let len = MArray.byteLength buf + MArray.byteLength arr
170+ buf' <- if MArray.byteCapacity buf < len
171+ then liftIO $ MArray.realloc (max n len) buf
172+ else return buf
173+ buf'' <- MArray.splice buf' arr
174+
175+ -- XXX this is common in both the equations of step
176+ if len >= n
177+ then do
178+ r <- step1 r1 buf''
179+ case r of
180+ FL.Done _ -> return $ FL.Done ()
181+ FL.Partial s -> do
182+ extract1 s
183+ res <- initial1
184+ return $ first (Tuple' Nothing) res
185+ else return $ FL.Partial $ Tuple' (Just buf'') r1
186+
187+-- XXX Same as compactLE, to be removed once that is implemented.
188+--
189+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
190+-- maximum specified size in bytes.
191+--
192+-- /Internal/
193+{-# INLINE compact #-}
194+compact :: (MonadIO m, Unbox a)
195+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
196+compact = packArraysChunksOf
197+
198+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
199+-- maximum specified size. Note that if a single array is bigger than the
200+-- specified size we do not split it to fit. When we coalesce multiple arrays
201+-- if the size would exceed the specified size we do not coalesce therefore the
202+-- actual array size may be less than the specified chunk size.
203+--
204+-- /Internal/
205+{-# INLINE_NORMAL compactLEParserD #-}
206+compactLEParserD ::
207+ forall m a. (MonadIO m, Unbox a)
208+ => Int -> ParserD.Parser (MutArray a) m (MutArray a)
209+compactLEParserD n = ParserD.Parser step initial extract
210+
211+ where
212+
213+ nBytes = n * SIZE_OF(a)
214+
215+ initial =
216+ return
217+ $ if n <= 0
218+ then error
219+ $ functionPath
220+ ++ ": the size of arrays ["
221+ ++ show n ++ "] must be a natural number"
222+ else ParserD.IPartial Nothing
223+
224+ step Nothing arr =
225+ return
226+ $ let len = MArray.byteLength arr
227+ in if len >= nBytes
228+ then ParserD.Done 0 arr
229+ else ParserD.Partial 0 (Just arr)
230+ step (Just buf) arr =
231+ let len = MArray.byteLength buf + MArray.byteLength arr
232+ in if len > nBytes
233+ then return $ ParserD.Done 1 buf
234+ else do
235+ buf1 <-
236+ if MArray.byteCapacity buf < nBytes
237+ then liftIO $ MArray.realloc nBytes buf
238+ else return buf
239+ buf2 <- MArray.splice buf1 arr
240+ return $ ParserD.Partial 0 (Just buf2)
241+
242+ extract Nothing = return $ ParserD.Done 0 MArray.nil
243+ extract (Just buf) = return $ ParserD.Done 0 buf
244+
245+ functionPath =
246+ "Streamly.Internal.Data.Stream.MutChunked.compactLEParserD"
247+
248+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
249+-- minimum specified size. Note that if all the arrays in the stream together
250+-- are smaller than the specified size the resulting array will be smaller than
251+-- the specified size. When we coalesce multiple arrays if the size would exceed
252+-- the specified size we stop coalescing further.
253+--
254+-- /Internal/
255+{-# INLINE_NORMAL compactGEFold #-}
256+compactGEFold ::
257+ forall m a. (MonadIO m, Unbox a)
258+ => Int -> FL.Fold m (MutArray a) (MutArray a)
259+compactGEFold n = Fold step initial extract
260+
261+ where
262+
263+ nBytes = n * SIZE_OF(a)
264+
265+ initial =
266+ return
267+ $ if n < 0
268+ then error
269+ $ functionPath
270+ ++ ": the size of arrays ["
271+ ++ show n ++ "] must be a natural number"
272+ else FL.Partial Nothing
273+
274+ step Nothing arr =
275+ return
276+ $ let len = MArray.byteLength arr
277+ in if len >= nBytes
278+ then FL.Done arr
279+ else FL.Partial (Just arr)
280+ step (Just buf) arr = do
281+ let len = MArray.byteLength buf + MArray.byteLength arr
282+ buf1 <-
283+ if MArray.byteCapacity buf < len
284+ then liftIO $ MArray.realloc (max len nBytes) buf
285+ else return buf
286+ buf2 <- MArray.splice buf1 arr
287+ if len >= n
288+ then return $ FL.Done buf2
289+ else return $ FL.Partial (Just buf2)
290+
291+ extract Nothing = return MArray.nil
292+ extract (Just buf) = return buf
293+
294+ functionPath =
295+ "Streamly.Internal.Data.Stream.MutChunked.compactGEFold"
296+
297+-- | Coalesce adjacent arrays in incoming stream to form bigger arrays of a
298+-- maximum specified size in bytes.
299+--
300+-- /Internal/
301+compactLE :: (MonadIO m, Unbox a) =>
302+ Int -> Stream m (MutArray a) -> Stream m (Either ParseError (MutArray a))
303+compactLE n = D.parseManyD (compactLEParserD n)
304+
305+-- | Like 'compactLE' but generates arrays of exactly equal to the size
306+-- specified except for the last array in the stream which could be shorter.
307+--
308+-- /Unimplemented/
309+{-# INLINE compactEQ #-}
310+compactEQ :: -- (MonadIO m, Unbox a) =>
311+ Int -> Stream m (MutArray a) -> Stream m (MutArray a)
312+compactEQ _n _xs = undefined
313+ -- IsStream.fromStreamD $ D.foldMany (compactEQFold n) (IsStream.toStreamD xs)
314+
315+-- | Like 'compactLE' but generates arrays of size greater than or equal to the
316+-- specified except for the last array in the stream which could be shorter.
317+--
318+-- /Internal/
319+{-# INLINE compactGE #-}
320+compactGE ::
321+ (MonadIO m, Unbox a)
322+ => Int -> Stream m (MutArray a) -> Stream m (MutArray a)
323+compactGE n = D.foldMany (compactGEFold n)
core/src/Streamly/Internal/Data/Stream/Nesting.hs+202−86
@@ -55,6 +55,11 @@ module Streamly.Internal.Data.Stream.Nesting
5555 -- yields an element. A special case of unfoldRoundRobin.
5656 , roundRobin -- interleaveFair?/ParallelFair
5757
58+ -- *** Zipping
59+ -- | Zip corresponding elements of two streams.
60+ , zipWith
61+ , zipWithM
62+
5863 -- *** Merging
5964 -- | Interleave elements from two streams based on a condition.
6065 , mergeBy
@@ -71,10 +76,17 @@ module Streamly.Internal.Data.Stream.Nesting
7176 -- unfoldMany: Unfold m a b -> t m a -> t m b
7277 -- @
7378
79+ -- *** ConcatMap
80+ -- | Generate streams by mapping a stream generator on each element of an
81+ -- input stream, append the resulting streams and flatten.
82+ , concatMap
83+ , concatMapM
84+
7485 -- *** ConcatUnfold
7586 -- | Generate streams by using an unfold on each element of an input
7687 -- stream, append the resulting streams and flatten. A special case of
7788 -- gintercalate.
89+ , unfoldMany
7890 , ConcatUnfoldInterleaveState (..)
7991 , unfoldInterleave
8092 , unfoldRoundRobin
@@ -106,8 +118,11 @@ module Streamly.Internal.Data.Stream.Nesting
106118
107119 -- ** Folding
108120 -- | Apply folds on a stream.
121+ , foldMany
122+ , refoldMany
109123 , foldSequence
110124 , foldIterateM
125+ , refoldIterateM
111126
112127 -- ** Parsing
113128 -- | Parsing is opposite to flattening. 'parseMany' is dual to concatMap or
@@ -124,6 +139,7 @@ module Streamly.Internal.Data.Stream.Nesting
124139
125140 -- ** Grouping
126141 -- | Group segments of a stream and fold. Special case of parsing.
142+ , groupsOf
127143 , groupsBy
128144 , groupsWhile
129145 , groupsRollingBy
@@ -131,8 +147,9 @@ module Streamly.Internal.Data.Stream.Nesting
131147 -- ** Splitting
132148 -- | A special case of parsing.
133149 , wordsBy
134− , splitOnSeq -- XXX splitOnSeg
135− , splitOnSuffixSeq -- XXX splitOnSegSuffix, splitOnTrailer
150+ , splitOnSeq
151+ , splitOnSuffixSeq
152+ , sliceOnSuffix
136153
137154 -- XXX Implement these as folds or parsers instead.
138155 , splitOnSuffixSeqAny
@@ -141,8 +158,9 @@ module Streamly.Internal.Data.Stream.Nesting
141158
142159 -- * Transform (Nested Containers)
143160 -- | Opposite to compact in ArrayStream
144− , splitInnerBy -- XXX innerSplitOn
145− , splitInnerBySuffix -- XXX innerSplitOnSuffix
161+ , splitInnerBy
162+ , splitInnerBySuffix
163+ , intersectBySorted
146164
147165 -- * Reduce By Streams
148166 , dropPrefix
@@ -164,9 +182,12 @@ import Fusion.Plugin.Types (Fuse(..))
164182import GHC.Types (SPEC(..))
165183
166184import Streamly.Internal.Data.Array.Type (Array(..))
185+import Streamly.Internal.Data.Fold.Step (Step(..))
167186import Streamly.Internal.Data.Fold.Type (Fold(..))
168187import Streamly.Internal.Data.Parser (ParseError(..))
188+import Streamly.Internal.Data.Refold.Type (Refold(..))
169189import Streamly.Internal.Data.SVar.Type (adaptState)
190+import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
170191import Streamly.Internal.Data.Unbox (Unbox, sizeOf)
171192import Streamly.Internal.Data.Unfold.Type (Unfold(..))
172193
@@ -592,6 +613,49 @@ mergeFstBy :: -- Monad m =>
592613mergeFstBy _f _m1 _m2 = undefined
593614 -- fromStreamK $ D.mergeFstBy f (toStreamD m1) (toStreamD m2)
594615
616+-------------------------------------------------------------------------------
617+-- Intersection of sorted streams
618+-------------------------------------------------------------------------------
619+
620+-- Assuming the streams are sorted in ascending order
621+{-# INLINE_NORMAL intersectBySorted #-}
622+intersectBySorted :: Monad m
623+ => (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
624+intersectBySorted cmp (Stream stepa ta) (Stream stepb tb) =
625+ Stream step
626+ ( ta -- left stream state
627+ , tb -- right stream state
628+ , Nothing -- left value
629+ , Nothing -- right value
630+ )
631+
632+ where
633+
634+ {-# INLINE_LATE step #-}
635+ -- step 1, fetch the first value
636+ step gst (sa, sb, Nothing, b) = do
637+ r <- stepa gst sa
638+ return $ case r of
639+ Yield a sa' -> Skip (sa', sb, Just a, b) -- step 2/3
640+ Skip sa' -> Skip (sa', sb, Nothing, b)
641+ Stop -> Stop
642+
643+ -- step 2, fetch the second value
644+ step gst (sa, sb, a@(Just _), Nothing) = do
645+ r <- stepb gst sb
646+ return $ case r of
647+ Yield b sb' -> Skip (sa, sb', a, Just b) -- step 3
648+ Skip sb' -> Skip (sa, sb', a, Nothing)
649+ Stop -> Stop
650+
651+ -- step 3, compare the two values
652+ step _ (sa, sb, Just a, Just b) = do
653+ let res = cmp a b
654+ return $ case res of
655+ GT -> Skip (sa, sb, Just a, Nothing) -- step 2
656+ LT -> Skip (sa, sb, Nothing, Just b) -- step 1
657+ EQ -> Yield a (sa, sb, Nothing, Just b) -- step 1
658+
595659------------------------------------------------------------------------------
596660-- Combine N Streams - unfoldMany
597661------------------------------------------------------------------------------
@@ -1178,7 +1242,6 @@ foldSequence _f _m = undefined
11781242data FIterState s f m a b
11791243 = FIterInit s f
11801244 | forall fs. FIterStream s (fs -> a -> m (FL.Step fs b)) fs (fs -> m b)
1181− (fs -> m b)
11821245 | FIterYield b (FIterState s f m a b)
11831246 | FIterStop
11841247
@@ -1206,30 +1269,101 @@ foldIterateM func seed0 (Stream step state) =
12061269 where
12071270
12081271 {-# INLINE iterStep #-}
1209− iterStep from st fstep extract final = do
1272+ iterStep from st fstep extract = do
12101273 res <- from
12111274 return
12121275 $ Skip
12131276 $ case res of
1214− FL.Partial fs -> FIterStream st fstep fs extract final
1277+ FL.Partial fs -> FIterStream st fstep fs extract
12151278 FL.Done fb -> FIterYield fb $ FIterInit st (return fb)
12161279
12171280 {-# INLINE_LATE stepOuter #-}
12181281 stepOuter _ (FIterInit st seed) = do
1219− (FL.Fold fstep initial extract final) <- seed >>= func
1220− iterStep initial st fstep extract final
1221− stepOuter gst (FIterStream st fstep fs extract final) = do
1282+ (FL.Fold fstep initial extract) <- seed >>= func
1283+ iterStep initial st fstep extract
1284+ stepOuter gst (FIterStream st fstep fs extract) = do
12221285 r <- step (adaptState gst) st
12231286 case r of
12241287 Yield x s -> do
1225− iterStep (fstep fs x) s fstep extract final
1226− Skip s -> return $ Skip $ FIterStream s fstep fs extract final
1288+ iterStep (fstep fs x) s fstep extract
1289+ Skip s -> return $ Skip $ FIterStream s fstep fs extract
12271290 Stop -> do
1228− b <- final fs
1291+ b <- extract fs
12291292 return $ Skip $ FIterYield b FIterStop
12301293 stepOuter _ (FIterYield a next) = return $ Yield a next
12311294 stepOuter _ FIterStop = return Stop
12321295
1296+{-# ANN type CIterState Fuse #-}
1297+data CIterState s f fs b
1298+ = CIterInit s f
1299+ | CIterConsume s fs
1300+ | CIterYield b (CIterState s f fs b)
1301+ | CIterStop
1302+
1303+-- | Like 'foldIterateM' but using the 'Refold' type instead. This could be
1304+-- much more efficient due to stream fusion.
1305+--
1306+-- /Internal/
1307+{-# INLINE_NORMAL refoldIterateM #-}
1308+refoldIterateM ::
1309+ Monad m => Refold m b a b -> m b -> Stream m a -> Stream m b
1310+refoldIterateM (Refold fstep finject fextract) initial (Stream step state) =
1311+ Stream stepOuter (CIterInit state initial)
1312+
1313+ where
1314+
1315+ {-# INLINE iterStep #-}
1316+ iterStep st action = do
1317+ res <- action
1318+ return
1319+ $ Skip
1320+ $ case res of
1321+ FL.Partial fs -> CIterConsume st fs
1322+ FL.Done fb -> CIterYield fb $ CIterInit st (return fb)
1323+
1324+ {-# INLINE_LATE stepOuter #-}
1325+ stepOuter _ (CIterInit st action) = do
1326+ iterStep st (action >>= finject)
1327+ stepOuter gst (CIterConsume st fs) = do
1328+ r <- step (adaptState gst) st
1329+ case r of
1330+ Yield x s -> iterStep s (fstep fs x)
1331+ Skip s -> return $ Skip $ CIterConsume s fs
1332+ Stop -> do
1333+ b <- fextract fs
1334+ return $ Skip $ CIterYield b CIterStop
1335+ stepOuter _ (CIterYield a next) = return $ Yield a next
1336+ stepOuter _ CIterStop = return Stop
1337+
1338+-- "n" elements at the end are dropped by the fold.
1339+{-# INLINE sliceBy #-}
1340+sliceBy :: Monad m => Fold m a Int -> Int -> Refold m (Int, Int) a (Int, Int)
1341+sliceBy (Fold step1 initial1 extract1) n = Refold step inject extract
1342+
1343+ where
1344+
1345+ inject (i, len) = do
1346+ r <- initial1
1347+ return $ case r of
1348+ Partial s -> Partial $ Tuple' (i + len + n) s
1349+ Done l -> Done (i, l)
1350+
1351+ step (Tuple' i s) x = do
1352+ r <- step1 s x
1353+ return $ case r of
1354+ Partial s1 -> Partial $ Tuple' i s1
1355+ Done len -> Done (i, len)
1356+
1357+ extract (Tuple' i s) = (i,) <$> extract1 s
1358+
1359+{-# INLINE sliceOnSuffix #-}
1360+sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
1361+sliceOnSuffix predicate =
1362+ -- Scan the stream with the given refold
1363+ refoldIterateM
1364+ (sliceBy (FL.takeEndBy_ predicate FL.length) 1)
1365+ (return (-1, 0))
1366+
12331367------------------------------------------------------------------------------
12341368-- Parsing
12351369------------------------------------------------------------------------------
@@ -1782,7 +1916,7 @@ groupsWhile :: Monad m
17821916{-
17831917groupsWhile eq fld = parseMany (PRD.groupBy eq fld)
17841918-}
1785−groupsWhile cmp (Fold fstep initial _ final) (Stream step state) =
1919+groupsWhile cmp (Fold fstep initial done) (Stream step state) =
17861920 Stream stepOuter (GroupingInit state)
17871921
17881922 where
@@ -1805,7 +1939,7 @@ groupsWhile cmp (Fold fstep initial _ final) (Stream step state) =
18051939 FL.Partial fs1 -> go SPEC x s fs1
18061940 FL.Done b -> return $ Yield b (GroupingInit s)
18071941 Skip s -> return $ Skip $ GroupingDo s fs
1808− Stop -> final fs >> return Stop
1942+ Stop -> return Stop
18091943
18101944 where
18111945
@@ -1820,12 +1954,10 @@ groupsWhile cmp (Fold fstep initial _ final) (Stream step state) =
18201954 FL.Partial fs1 -> go SPEC prev s fs1
18211955 FL.Done b -> return $ Yield b (GroupingInit s)
18221956 else do
1823− r <- final acc
1957+ r <- done acc
18241958 return $ Yield r (GroupingInitWith s x)
18251959 Skip s -> go SPEC prev s acc
1826− Stop -> do
1827− r <- final acc
1828− return $ Yield r GroupingDone
1960+ Stop -> done acc >>= \r -> return $ Yield r GroupingDone
18291961 stepOuter _ (GroupingInitWith st x) = do
18301962 res <- initial
18311963 return
@@ -1852,12 +1984,10 @@ groupsWhile cmp (Fold fstep initial _ final) (Stream step state) =
18521984 FL.Partial fs1 -> go SPEC s fs1
18531985 FL.Done b -> return $ Yield b (GroupingInit s)
18541986 else do
1855− r <- final acc
1987+ r <- done acc
18561988 return $ Yield r (GroupingInitWith s x)
18571989 Skip s -> go SPEC s acc
1858− Stop -> do
1859− r <- final acc
1860− return $ Yield r GroupingDone
1990+ Stop -> done acc >>= \r -> return $ Yield r GroupingDone
18611991 stepOuter _ (GroupingYield _ _) = error "groupsWhile: Unreachable"
18621992 stepOuter _ GroupingDone = return Stop
18631993
@@ -1879,7 +2009,7 @@ groupsRollingBy :: Monad m
18792009{-
18802010groupsRollingBy eq fld = parseMany (PRD.groupByRolling eq fld)
18812011-}
1882−groupsRollingBy cmp (Fold fstep initial _ final) (Stream step state) =
2012+groupsRollingBy cmp (Fold fstep initial done) (Stream step state) =
18832013 Stream stepOuter (GroupingInit state)
18842014
18852015 where
@@ -1902,7 +2032,7 @@ groupsRollingBy cmp (Fold fstep initial _ final) (Stream step state) =
19022032 FL.Partial fs1 -> go SPEC x s fs1
19032033 FL.Done fb -> return $ Yield fb (GroupingInit s)
19042034 Skip s -> return $ Skip $ GroupingDo s fs
1905− Stop -> final fs >> return Stop
2035+ Stop -> return Stop
19062036
19072037 where
19082038
@@ -1917,12 +2047,10 @@ groupsRollingBy cmp (Fold fstep initial _ final) (Stream step state) =
19172047 FL.Partial fs1 -> go SPEC x s fs1
19182048 FL.Done b -> return $ Yield b (GroupingInit s)
19192049 else do
1920− r <- final acc
2050+ r <- done acc
19212051 return $ Yield r (GroupingInitWith s x)
19222052 Skip s -> go SPEC prev s acc
1923− Stop -> do
1924− r <- final acc
1925− return $ Yield r GroupingDone
2053+ Stop -> done acc >>= \r -> return $ Yield r GroupingDone
19262054 stepOuter _ (GroupingInitWith st x) = do
19272055 res <- initial
19282056 return
@@ -1953,7 +2081,7 @@ groupsRollingBy cmp (Fold fstep initial _ final) (Stream step state) =
19532081 FL.Done b -> return $ Yield b (GroupingInit st)
19542082 else do
19552083 {-
1956− r <- final acc
2084+ r <- done acc
19572085 return $ Yield r (GroupingInitWith s x)
19582086 -}
19592087 -- The code above does not let groupBy fuse. We use the
@@ -1962,16 +2090,14 @@ groupsRollingBy cmp (Fold fstep initial _ final) (Stream step state) =
19622090 -- GroupingInitWith state here to help GHC with stream
19632091 -- fusion.
19642092 result <- initial
1965− r <- final acc
2093+ r <- done acc
19662094 return
19672095 $ Yield r
19682096 $ case result of
19692097 FL.Partial fsi -> GroupingDoWith s fsi x
19702098 FL.Done b -> GroupingYield b (GroupingInit s)
19712099 Skip s -> go SPEC prev s acc
1972− Stop -> do
1973− r <- final acc
1974− return $ Yield r GroupingDone
2100+ Stop -> done acc >>= \r -> return $ Yield r GroupingDone
19752101 stepOuter _ (GroupingYield r next) = return $ Yield r next
19762102 stepOuter _ GroupingDone = return Stop
19772103
@@ -1993,7 +2119,7 @@ data WordsByState st fs b
19932119
19942120{-# INLINE_NORMAL wordsBy #-}
19952121wordsBy :: Monad m => (a -> Bool) -> Fold m a b -> Stream m a -> Stream m b
1996−wordsBy predicate (Fold fstep initial _ final) (Stream step state) =
2122+wordsBy predicate (Fold fstep initial done) (Stream step state) =
19972123 Stream stepOuter (WordsByInit state)
19982124
19992125 where
@@ -2023,7 +2149,7 @@ wordsBy predicate (Fold fstep initial _ final) (Stream step state) =
20232149 FL.Partial fs1 -> go SPEC s fs1
20242150 FL.Done b -> return $ Yield b (WordsByInit s)
20252151 Skip s -> return $ Skip $ WordsByDo s fs
2026− Stop -> final fs >> return Stop
2152+ Stop -> return Stop
20272153
20282154 where
20292155
@@ -2034,7 +2160,7 @@ wordsBy predicate (Fold fstep initial _ final) (Stream step state) =
20342160 if predicate x
20352161 then do
20362162 {-
2037− r <- final acc
2163+ r <- done acc
20382164 return $ Yield r (WordsByInit s)
20392165 -}
20402166 -- The above code does not fuse well. Need to check why
@@ -2043,7 +2169,7 @@ wordsBy predicate (Fold fstep initial _ final) (Stream step state) =
20432169 -- state always, we directly go to WordsByDo state in
20442170 -- the common case of Partial.
20452171 resi <- initial
2046− r <- final acc
2172+ r <- done acc
20472173 return
20482174 $ Yield r
20492175 $ case resi of
@@ -2055,9 +2181,7 @@ wordsBy predicate (Fold fstep initial _ final) (Stream step state) =
20552181 FL.Partial fs1 -> go SPEC s fs1
20562182 FL.Done b -> return $ Yield b (WordsByInit s)
20572183 Skip s -> go SPEC s acc
2058− Stop -> do
2059− r <- final acc
2060− return $ Yield r WordsByDone
2184+ Stop -> done acc >>= \r -> return $ Yield r WordsByDone
20612185
20622186 stepOuter _ WordsByDone = return Stop
20632187
@@ -2115,7 +2239,7 @@ splitOnSeq
21152239 -> Fold m a b
21162240 -> Stream m a
21172241 -> Stream m b
2118−splitOnSeq patArr (Fold fstep initial _ final) (Stream step state) =
2242+splitOnSeq patArr (Fold fstep initial done) (Stream step state) =
21192243 Stream stepOuter SplitOnSeqInit
21202244
21212245 where
@@ -2198,12 +2322,12 @@ splitOnSeq patArr (Fold fstep initial _ final) (Stream step state) =
21982322 r <- fstep acc x
21992323 b1 <-
22002324 case r of
2201− FL.Partial acc1 -> final acc1
2325+ FL.Partial acc1 -> done acc1
22022326 FL.Done b -> return b
22032327 let jump c = SplitOnSeqEmpty c s
22042328 in yieldProceed jump b1
22052329 Skip s -> skip (SplitOnSeqEmpty acc s)
2206− Stop -> final acc >> return Stop
2330+ Stop -> return Stop
22072331
22082332 -----------------
22092333 -- Done
@@ -2221,7 +2345,7 @@ splitOnSeq patArr (Fold fstep initial _ final) (Stream step state) =
22212345 Yield x s -> do
22222346 let jump c = SplitOnSeqSingle c s pat
22232347 if pat == x
2224− then final fs >>= yieldProceed jump
2348+ then done fs >>= yieldProceed jump
22252349 else do
22262350 r <- fstep fs x
22272351 case r of
@@ -2229,7 +2353,7 @@ splitOnSeq patArr (Fold fstep initial _ final) (Stream step state) =
22292353 FL.Done b -> yieldProceed jump b
22302354 Skip s -> return $ Skip $ SplitOnSeqSingle fs s pat
22312355 Stop -> do
2232− r <- final fs
2356+ r <- done fs
22332357 return $ Skip $ SplitOnSeqYield r SplitOnSeqDone
22342358
22352359 ---------------------------
@@ -2237,7 +2361,7 @@ splitOnSeq patArr (Fold fstep initial _ final) (Stream step state) =
22372361 ---------------------------
22382362
22392363 stepOuter _ (SplitOnSeqWordDone 0 fs _) = do
2240− r <- final fs
2364+ r <- done fs
22412365 skip $ SplitOnSeqYield r SplitOnSeqDone
22422366 stepOuter _ (SplitOnSeqWordDone n fs wrd) = do
22432367 let old = elemMask .&. (wrd `shiftR` (elemBits * (n - 1)))
@@ -2264,7 +2388,7 @@ splitOnSeq patArr (Fold fstep initial _ final) (Stream step state) =
22642388 if wrd1 .&. wordMask == wordPat
22652389 then do
22662390 let jump c = SplitOnSeqWordInit c s
2267− final fs >>= yieldProceed jump
2391+ done fs >>= yieldProceed jump
22682392 else skip $ SplitOnSeqWordLoop wrd1 s fs
22692393 else go SPEC (idx + 1) wrd1 s
22702394 Skip s -> go SPEC idx wrd s
@@ -2272,7 +2396,7 @@ splitOnSeq patArr (Fold fstep initial _ final) (Stream step state) =
22722396 if idx /= 0
22732397 then skip $ SplitOnSeqWordDone idx fs wrd
22742398 else do
2275− r <- final fs
2399+ r <- done fs
22762400 skip $ SplitOnSeqYield r SplitOnSeqDone
22772401
22782402 stepOuter gst (SplitOnSeqWordLoop wrd0 st0 fs0) =
@@ -2293,7 +2417,7 @@ splitOnSeq patArr (Fold fstep initial _ final) (Stream step state) =
22932417 case r of
22942418 FL.Partial fs1 -> do
22952419 if wrd1 .&. wordMask == wordPat
2296− then final fs1 >>= yieldProceed jump
2420+ then done fs1 >>= yieldProceed jump
22972421 else go SPEC wrd1 s fs1
22982422 FL.Done b -> yieldProceed jump b
22992423 Skip s -> go SPEC wrd s fs
@@ -2378,14 +2502,14 @@ splitOnSeq patArr (Fold fstep initial _ final) (Stream step state) =
23782502 stepOuter _ (SplitOnSeqKRCheck fs st rb rh) = do
23792503 if RB.unsafeEqArray rb rh patArr
23802504 then do
2381− r <- final fs
2505+ r <- done fs
23822506 let rst = RB.startOf rb
23832507 jump c = SplitOnSeqKRInit 0 c st rb rst
23842508 yieldProceed jump r
23852509 else skip $ SplitOnSeqKRLoop fs st rb rh patHash
23862510
23872511 stepOuter _ (SplitOnSeqKRDone 0 fs _ _) = do
2388− r <- final fs
2512+ r <- done fs
23892513 skip $ SplitOnSeqYield r SplitOnSeqDone
23902514 stepOuter _ (SplitOnSeqKRDone n fs rb rh) = do
23912515 old <- liftIO $ peek rh
@@ -2429,7 +2553,7 @@ splitOnSuffixSeq
24292553 -> Fold m a b
24302554 -> Stream m a
24312555 -> Stream m b
2432−splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state) =
2556+splitOnSuffixSeq withSep patArr (Fold fstep initial done) (Stream step state) =
24332557 Stream stepOuter SplitOnSuffixSeqInit
24342558
24352559 where
@@ -2469,7 +2593,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state)
24692593 r <- if withSep then fstep fs x else return $ FL.Partial fs
24702594 b1 <-
24712595 case r of
2472− FL.Partial fs1 -> final fs1
2596+ FL.Partial fs1 -> done fs1
24732597 FL.Done b -> return b
24742598 yieldProceed jump b1
24752599 else do
@@ -2532,11 +2656,11 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state)
25322656 r <- fstep acc x
25332657 b1 <-
25342658 case r of
2535− FL.Partial fs -> final fs
2659+ FL.Partial fs -> done fs
25362660 FL.Done b -> return b
25372661 yieldProceed jump b1
25382662 Skip s -> skip (SplitOnSuffixSeqEmpty acc s)
2539− Stop -> final acc >> return Stop
2663+ Stop -> return Stop
25402664
25412665 -----------------
25422666 -- Done
@@ -2553,7 +2677,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state)
25532677 case res of
25542678 Yield x s -> processYieldSingle pat x s fs
25552679 Skip s -> skip $ SplitOnSuffixSeqSingleInit fs s pat
2556− Stop -> final fs >> return Stop
2680+ Stop -> return Stop
25572681
25582682 stepOuter gst (SplitOnSuffixSeqSingle fs st pat) = do
25592683 res <- step (adaptState gst) st
@@ -2561,7 +2685,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state)
25612685 Yield x s -> processYieldSingle pat x s fs
25622686 Skip s -> skip $ SplitOnSuffixSeqSingle fs s pat
25632687 Stop -> do
2564− r <- final fs
2688+ r <- done fs
25652689 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
25662690
25672691 ---------------------------
@@ -2569,7 +2693,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state)
25692693 ---------------------------
25702694
25712695 stepOuter _ (SplitOnSuffixSeqWordDone 0 fs _) = do
2572− r <- final fs
2696+ r <- done fs
25732697 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
25742698 stepOuter _ (SplitOnSuffixSeqWordDone n fs wrd) = do
25752699 let old = elemMask .&. (wrd `shiftR` (elemBits * (n - 1)))
@@ -2592,7 +2716,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state)
25922716 let jump c = SplitOnSuffixSeqWordInit c s
25932717 yieldProceed jump b
25942718 Skip s -> skip (SplitOnSuffixSeqWordInit fs0 s)
2595− Stop -> final fs0 >> return Stop
2719+ Stop -> return Stop
25962720
25972721 where
25982722
@@ -2610,7 +2734,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state)
26102734 then go SPEC (idx + 1) wrd1 s fs1
26112735 else if wrd1 .&. wordMask /= wordPat
26122736 then skip $ SplitOnSuffixSeqWordLoop wrd1 s fs1
2613− else do final fs >>= yieldProceed jump
2737+ else do done fs >>= yieldProceed jump
26142738 FL.Done b -> yieldProceed jump b
26152739 Skip s -> go SPEC idx wrd s fs
26162740 Stop -> skip $ SplitOnSuffixSeqWordDone idx fs wrd
@@ -2636,16 +2760,16 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state)
26362760 case r of
26372761 FL.Partial fs1 ->
26382762 if wrd1 .&. wordMask == wordPat
2639− then final fs1 >>= yieldProceed jump
2763+ then done fs1 >>= yieldProceed jump
26402764 else go SPEC wrd1 s fs1
26412765 FL.Done b -> yieldProceed jump b
26422766 Skip s -> go SPEC wrd s fs
26432767 Stop ->
26442768 if wrd .&. wordMask == wordPat
2645− then final fs >> return Stop
2769+ then return Stop
26462770 else if withSep
26472771 then do
2648− r <- final fs
2772+ r <- done fs
26492773 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
26502774 else skip $ SplitOnSuffixSeqWordDone patLen fs wrd
26512775
@@ -2667,7 +2791,7 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state)
26672791 jump c = SplitOnSuffixSeqKRInit 0 c s rb rst
26682792 yieldProceed jump b
26692793 Skip s -> skip $ SplitOnSuffixSeqKRInit idx0 fs s rb rh0
2670− Stop -> final fs >> return Stop
2794+ Stop -> return Stop
26712795
26722796 stepOuter gst (SplitOnSuffixSeqKRInit1 fs0 st0 rb rh0) = do
26732797 go SPEC 1 rh0 st0 fs0
@@ -2699,10 +2823,10 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state)
26992823 Stop -> do
27002824 -- do not issue a blank segment when we end at pattern
27012825 if (idx == maxIndex) && RB.unsafeEqArray rb rh patArr
2702− then final fs >> return Stop
2826+ then return Stop
27032827 else if withSep
27042828 then do
2705− r <- final fs
2829+ r <- done fs
27062830 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
27072831 else skip $ SplitOnSuffixSeqKRDone idx fs rb (RB.startOf rb)
27082832
@@ -2731,24 +2855,24 @@ splitOnSuffixSeq withSep patArr (Fold fstep initial _ final) (Stream step state)
27312855 Skip s -> go SPEC fs s rh cksum
27322856 Stop ->
27332857 if RB.unsafeEqArray rb rh patArr
2734− then final fs >> return Stop
2858+ then return Stop
27352859 else if withSep
27362860 then do
2737− r <- final fs
2861+ r <- done fs
27382862 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
27392863 else skip $ SplitOnSuffixSeqKRDone patLen fs rb rh
27402864
27412865 stepOuter _ (SplitOnSuffixSeqKRCheck fs st rb rh) = do
27422866 if RB.unsafeEqArray rb rh patArr
27432867 then do
2744− r <- final fs
2868+ r <- done fs
27452869 let rst = RB.startOf rb
27462870 jump c = SplitOnSuffixSeqKRInit 0 c st rb rst
27472871 yieldProceed jump r
27482872 else skip $ SplitOnSuffixSeqKRLoop fs st rb rh patHash
27492873
27502874 stepOuter _ (SplitOnSuffixSeqKRDone 0 fs _ _) = do
2751− r <- final fs
2875+ r <- done fs
27522876 skip $ SplitOnSuffixSeqYield r SplitOnSuffixSeqDone
27532877 stepOuter _ (SplitOnSuffixSeqKRDone n fs rb rh) = do
27542878 old <- liftIO $ peek rh
@@ -2930,13 +3054,12 @@ splitInnerBy splitter joiner (Stream step1 state1) =
29303054-- | Performs infix separator style splitting.
29313055{-# INLINE_NORMAL splitInnerBySuffix #-}
29323056splitInnerBySuffix
2933− :: Monad m
2934− => (f a -> Bool) -- isEmpty?
2935− -> (f a -> m (f a, Maybe (f a))) -- splitter
3057+ :: (Monad m, Eq (f a), Monoid (f a))
3058+ => (f a -> m (f a, Maybe (f a))) -- splitter
29363059 -> (f a -> f a -> m (f a)) -- joiner
29373060 -> Stream m (f a)
29383061 -> Stream m (f a)
2939−splitInnerBySuffix isEmpty splitter joiner (Stream step1 state1) =
3062+splitInnerBySuffix splitter joiner (Stream step1 state1) =
29403063 Stream step (SplitInitial state1)
29413064
29423065 where
@@ -2963,11 +3086,10 @@ splitInnerBySuffix isEmpty splitter joiner (Stream step1 state1) =
29633086 Nothing -> Skip (SplitBuffering s buf')
29643087 Just x2 -> Skip (SplitYielding buf' (SplitSplitting s x2))
29653088 Skip s -> return $ Skip (SplitBuffering s buf)
2966− Stop ->
2967− return $
2968− if isEmpty buf
2969− then Stop
2970− else Skip (SplitYielding buf SplitFinishing)
3089+ Stop -> return $
3090+ if buf == mempty
3091+ then Stop
3092+ else Skip (SplitYielding buf SplitFinishing)
29713093
29723094 step _ (SplitSplitting st buf) = do
29733095 (x1, mx2) <- splitter buf
@@ -2986,8 +3108,6 @@ splitInnerBySuffix isEmpty splitter joiner (Stream step1 state1) =
29863108--
29873109-- Space: @O(1)@
29883110--
2989−-- See also stripPrefix.
2990−--
29913111-- /Unimplemented/
29923112{-# INLINE dropPrefix #-}
29933113dropPrefix ::
@@ -3000,8 +3120,6 @@ dropPrefix = error "Not implemented yet!"
30003120--
30013121-- Space: @O(n)@ where n is the length of the infix.
30023122--
3003−-- See also stripInfix.
3004−--
30053123-- /Unimplemented/
30063124{-# INLINE dropInfix #-}
30073125dropInfix ::
@@ -3014,8 +3132,6 @@ dropInfix = error "Not implemented yet!"
30143132--
30153133-- Space: @O(n)@ where n is the length of the suffix.
30163134--
3017−-- See also stripSuffix.
3018−--
30193135-- /Unimplemented/
30203136{-# INLINE dropSuffix #-}
30213137dropSuffix ::
core/src/Streamly/Internal/Data/Stream/Top.hs+116−243
@@ -8,73 +8,38 @@
88-- Portability : GHC
99--
1010-- Top level module that can depend on all other lower level Stream modules.
11−--
12−-- Design notes:
13−--
14−-- The order of arguments in the join operations should ideally be opposite. It
15−-- should be such that the infinite stream is the last one. The transformation
16−-- should be on the last argument, so if you curry the functions with all other
17−-- arguments we get a @Stream -> Stream@ function. The first stream argument
18−-- may be considered as a config or modifier for the operation.
19−--
20−-- Benefit of changing the order is that we get a more intuitive Stream ->
21−-- Stream transformation after currying all other arguments. The inner loop
22−-- streams become arguments for the transformation, more like local modifiers
23−-- for the global outer stream as the last argument. Thus we can continue using
24−-- transformations on the outer stream in a composed pipeline. Otherwise we can
25−-- use flip to flip the order.
26−--
27−-- The fact that the inner stream can be used in the loop multiple times also
28−-- tells that this is not the real effectful stream, it is more like a pure
29−-- stream or an array. In fact we may consider using an Identity streams as
30−-- inner streams in which case these functions will not look nice.
31−--
32−-- Downsides:
33−--
34−-- * Maybe less intuitive to think about, because we usually think the first
35−-- stream as the outer loop and second as the inner.
36−-- * Zip and merge operations will continue using the opposite order.
37−-- * Need to change the order of cross, crossWith operations as well
38−-- * It will be inconsistent with Data.List. The functions cannot be used as
39−-- intuitive operators.
40−--
41−-- The choice is similar to concatMap vs bind. concatMap is pipeline
42−-- composition friendly but bind is user intuition friendly. Another option is
43−-- to have other functions with a different argument order e.g. flippedCross
44−-- instead of cross.
45−--
46−-- If we change the order we have to make sure that we have a consistent
47−-- convention for set-like and the cross join operations.
4811
4912module Streamly.Internal.Data.Stream.Top
5013 (
51− -- * Sampling
14+ -- * Transformation
15+ -- ** Sampling
5216 -- | Value agnostic filtering.
5317 strideFromThen
5418
55− -- * Straight Joins
56− -- | These are set-like operations but not exactly set operations because
57− -- streams are not necessarily sets, they may have duplicated elements.
58− -- These operations are generic i.e. they work on streams of unconstrained
59− -- types, therefore, they have quadratic performance characterstics. For
60− -- better performance using Set or Map structures see the
61− -- Streamly.Internal.Data.Stream.Container module.
62− , intersectBy
63− , deleteFirstsBy
64− , unionBy
65−
66− -- Set like operations on sorted streams
67− , sortedIntersectBy
68− , sortedDeleteFirstsBy
69− , sortedUnionBy
70−
71− -- * Cross Joins
72− , innerJoin
73−
74− -- Joins on sorted stream
75− , innerSortedJoin
76− , leftSortedJoin
77− , outerSortedJoin
19+ -- * Nesting
20+ -- ** Set like operations
21+ -- | These are not exactly set operations because streams are not
22+ -- necessarily sets, they may have duplicated elements. These operations
23+ -- are generic i.e. they work on streams of unconstrained types, therefore,
24+ -- they have quadratic performance characterstics. For better performance
25+ -- using Set structures see the Streamly.Internal.Data.Stream.Container
26+ -- module.
27+ , filterInStreamGenericBy
28+ , deleteInStreamGenericBy
29+ , unionWithStreamGenericBy
30+
31+ -- ** Set like operations on sorted streams
32+ , filterInStreamAscBy
33+ , deleteInStreamAscBy
34+ , unionWithStreamAscBy
35+
36+ -- ** Join operations
37+ , joinInnerGeneric
38+
39+ -- * Joins on sorted stream
40+ , joinInnerAscBy
41+ , joinLeftAscBy
42+ , joinOuterAscBy
7843 )
7944where
8045
@@ -83,7 +48,8 @@ where
8348import Control.Monad.IO.Class (MonadIO(..))
8449import Data.IORef (newIORef, readIORef, modifyIORef')
8550import Streamly.Internal.Data.Fold.Type (Fold)
86−import Streamly.Internal.Data.Stream.Type (Stream(..), Step(..), cross)
51+import Streamly.Internal.Data.Stream.Common ()
52+import Streamly.Internal.Data.Stream.Type (Stream, cross)
8753
8854import qualified Data.List as List
8955import qualified Streamly.Internal.Data.Fold as Fold
@@ -145,40 +111,29 @@ strideFromThen offset stride =
145111-- binary search if we have an Ord instance or Ordering returning function. The
146112-- time complexity would then become (m x log n).
147113
148−-- | Like 'cross' but emits only those tuples where @a == b@ using the supplied
149−-- equality predicate. This is essentially a @cross intersection@ of two
150−-- streams.
114+-- | Like 'cross' but emits only those tuples where @a == b@ using the
115+-- supplied equality predicate.
151116--
152117-- Definition:
153118--
154−-- >>> innerJoin eq s1 s2 = Stream.filter (\(a, b) -> a `eq` b) $ Stream.cross s1 s2
155−--
156−-- The second (inner) stream must be finite. Moreover, it must be either pure
157−-- or capable of multiple evaluations. If not then the caller should cache it
158−-- in an 'Data.Array.Array', if the type does not have an 'Unbox' instance then
159−-- use the Generic 'Data.Array.Generic.Array'. Convert the array to stream
160−-- before calling this function. Caching may also improve performance if the
161−-- stream is expensive to evaluate.
119+-- >>> joinInnerGeneric eq s1 s2 = Stream.filter (\(a, b) -> a `eq` b) $ Stream.cross s1 s2
162120--
163−-- If you care about performance this function should be your last choice among
164−-- all inner joins. 'Streamly.Internal.Data.Unfold.innerJoin' is a much faster
165−-- fused alternative. 'innerSortedJoin' is a faster alternative when streams
166−-- are sorted. 'innerOrdJoin' is an order of magnitude faster alternative when
167−-- the type has an 'Ord' instance.
121+-- You should almost always prefer @joinInnerOrd@ over 'joinInnerGeneric' if
122+-- possible. @joinInnerOrd@ is an order of magnitude faster but may take more
123+-- space for caching the second stream.
168124--
169−-- Note: Conceptually, this is a commutative operation. Result includes all the
170−-- elements from the left and the right stream. The order of streams can be
171−-- changed without affecting results, except for the ordering within the tuple.
125+-- See 'Streamly.Internal.Data.Unfold.joinInnerGeneric' for a much faster fused
126+-- alternative.
172127--
173128-- Time: O(m x n)
174129--
175130-- /Pre-release/
176−{-# INLINE innerJoin #-}
177−innerJoin :: Monad m =>
131+{-# INLINE joinInnerGeneric #-}
132+joinInnerGeneric :: Monad m =>
178133 (a -> b -> Bool) -> Stream m a -> Stream m b -> Stream m (a, b)
179−innerJoin eq s1 s2 = Stream.filter (\(a, b) -> a `eq` b) $ cross s1 s2
134+joinInnerGeneric eq s1 s2 = Stream.filter (\(a, b) -> a `eq` b) $ cross s1 s2
180135{-
181−innerJoin eq s1 s2 = do
136+joinInnerGeneric eq s1 s2 = do
182137 -- ConcatMap works faster than bind
183138 Stream.concatMap (\a ->
184139 Stream.concatMap (\b ->
@@ -189,44 +144,44 @@ innerJoin eq s1 s2 = do
189144 ) s1
190145-}
191146
192−-- | A more efficient 'innerJoin' for sorted streams.
147+-- | A more efficient 'joinInner' for sorted streams.
193148--
194149-- Space: O(1)
195150--
196151-- Time: O(m + n)
197152--
198153-- /Unimplemented/
199−{-# INLINE innerSortedJoin #-}
200−innerSortedJoin ::
154+{-# INLINE joinInnerAscBy #-}
155+joinInnerAscBy ::
201156 (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, b)
202−innerSortedJoin = undefined
157+joinInnerAscBy = undefined
203158
204−-- | A more efficient 'leftJoin' for sorted streams.
159+-- | A more efficient 'joinLeft' for sorted streams.
205160--
206161-- Space: O(1)
207162--
208163-- Time: O(m + n)
209164--
210165-- /Unimplemented/
211−{-# INLINE leftSortedJoin #-}
212−leftSortedJoin :: -- Monad m =>
166+{-# INLINE joinLeftAscBy #-}
167+joinLeftAscBy :: -- Monad m =>
213168 (a -> b -> Ordering) -> Stream m a -> Stream m b -> Stream m (a, Maybe b)
214−leftSortedJoin _eq _s1 _s2 = undefined
169+joinLeftAscBy _eq _s1 _s2 = undefined
215170
216−-- | A more efficient 'outerJoin' for sorted streams.
171+-- | A more efficient 'joinOuter' for sorted streams.
217172--
218173-- Space: O(1)
219174--
220175-- Time: O(m + n)
221176--
222177-- /Unimplemented/
223−{-# INLINE outerSortedJoin #-}
224−outerSortedJoin :: -- Monad m =>
178+{-# INLINE joinOuterAscBy #-}
179+joinOuterAscBy :: -- Monad m =>
225180 (a -> b -> Ordering)
226181 -> Stream m a
227182 -> Stream m b
228183 -> Stream m (Maybe a, Maybe b)
229−outerSortedJoin _eq _s1 _s2 = undefined
184+joinOuterAscBy _eq _s1 _s2 = undefined
230185
231186------------------------------------------------------------------------------
232187-- Set operations (special joins)
@@ -236,8 +191,8 @@ outerSortedJoin _eq _s1 _s2 = undefined
236191-- the best would be to use an Array with linear search. If the second stream
237192-- is sorted we can also use a binary search, using Ord constraint.
238193
239−-- | Keep only those elements in the first stream that are present in the
240−-- second stream too. The second stream is folded to a container using the
194+-- | Keep only those elements in the second stream that are present in the
195+-- first stream too. The first stream is folded to a container using the
241196-- supplied fold and then the elements in the container are looked up using the
242197-- supplied lookup function.
243198--
@@ -252,30 +207,23 @@ filterStreamWith :: Monad m =>
252207filterStreamWith fld member s1 s2 =
253208 Stream.concatEffect
254209 $ do
255− xs <- Stream.fold fld s2
256− return $ Stream.filter (`member` xs) s1
257−
258−-- XXX instead of folding the second stream to a list we could use it directly.
259−-- If the user wants they can generate the stream from an array and also call
260−-- uniq or nub on it. We can provide a convenience Stream -> Stream to cache
261−-- a finite stream in an array and serve it from the cache. The user can decide
262−-- what is best based on the context. They can also choose to use a boxed or
263−-- unboxed array for caching. To force caching we can make the second stream
264−-- monad type Identity. But that may be less flexible. One option is to use
265−-- cachedIntersectBy etc for automatic caching.
210+ xs <- Stream.fold fld s1
211+ return $ Stream.filter (`member` xs) s2
266212
267−-- | 'intersectBy' returns a subsequence of the first stream which intersects
268−-- with the second stream. Note that this is not a commutative operation unlike
269−-- a set intersection, because of duplicate elements in the stream the order of
270−-- the streams matters. This is similar to 'Data.List.intersectBy'. Note that
271−-- intersectBy is a special case of 'innerJoin'.
213+-- | 'filterInStreamGenericBy' retains only those elements in the second stream that
214+-- are present in the first stream.
272215--
273−-- >>> f s1 s2 = Stream.fold Fold.toList $ Stream.intersectBy (==) (Stream.fromList s1) (Stream.fromList s2)
274−-- >>> f [1,3,4,4,5]) [2,3,4,5,5]
275−-- [3,4,4,5]
216+-- >>> Stream.fold Fold.toList $ Stream.filterInStreamGenericBy (==) (Stream.fromList [1,2,2,4]) (Stream.fromList [2,1,1,3])
217+-- [2,1,1]
276218--
277−-- First stream can be infinite, the second stream must be finite and must be
278−-- capable of multiple evaluations.
219+-- >>> Stream.fold Fold.toList $ Stream.filterInStreamGenericBy (==) (Stream.fromList [2,1,1,3]) (Stream.fromList [1,2,2,4])
220+-- [1,2,2]
221+--
222+-- Similar to the list intersectBy operation but with the stream argument order
223+-- flipped.
224+--
225+-- The first stream must be finite and must not block. Second stream is
226+-- processed only after the first stream is fully realized.
279227--
280228-- Space: O(n) where @n@ is the number of elements in the second stream.
281229--
@@ -283,87 +231,46 @@ filterStreamWith fld member s1 s2 =
283231-- @n@ is the number of elements in the second stream.
284232--
285233-- /Pre-release/
286−{-# INLINE intersectBy #-}
287−intersectBy :: Monad m =>
234+{-# INLINE filterInStreamGenericBy #-}
235+filterInStreamGenericBy :: Monad m =>
288236 (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
289−intersectBy eq =
237+filterInStreamGenericBy eq =
290238 -- XXX Use an (unboxed) array instead.
291239 filterStreamWith
292240 (Fold.scanMaybe (Fold.uniqBy eq) Fold.toListRev)
293241 (List.any . eq)
294242
295−-------------------------------------------------------------------------------
296−-- Intersection of sorted streams
297−-------------------------------------------------------------------------------
298−
299−-- XXX The sort order is not important as long both the streams have the same
300−-- sort order. We need to move only in one direction in each stream.
301−-- XXX Fix the argument order to use the same behavior as intersectBy.
302−
303−-- | Like 'intersectBy' but assumes that the input streams are sorted in
243+-- | Like 'filterInStreamGenericBy' but assumes that the input streams are sorted in
304244-- ascending order. To use it on streams sorted in descending order pass an
305245-- inverted comparison function returning GT for less than and LT for greater
306246-- than.
307247--
308−-- Both streams can be infinite.
309−--
310248-- Space: O(1)
311249--
312250-- Time: O(m+n)
313251--
314252-- /Pre-release/
315−{-# INLINE_NORMAL sortedIntersectBy #-}
316−sortedIntersectBy :: Monad m =>
253+{-# INLINE filterInStreamAscBy #-}
254+filterInStreamAscBy :: Monad m =>
317255 (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
318−sortedIntersectBy cmp (Stream stepa ta) (Stream stepb tb) =
319− Stream step
320− ( ta -- left stream state
321− , tb -- right stream state
322− , Nothing -- left value
323− , Nothing -- right value
324− )
325−
326− where
256+filterInStreamAscBy eq s1 s2 = Stream.intersectBySorted eq s2 s1
327257
328− {-# INLINE_LATE step #-}
329− -- step 1, fetch the first value
330− step gst (sa, sb, Nothing, b) = do
331− r <- stepa gst sa
332− return $ case r of
333− Yield a sa' -> Skip (sa', sb, Just a, b) -- step 2/3
334− Skip sa' -> Skip (sa', sb, Nothing, b)
335− Stop -> Stop
336−
337− -- step 2, fetch the second value
338− step gst (sa, sb, a@(Just _), Nothing) = do
339− r <- stepb gst sb
340− return $ case r of
341− Yield b sb' -> Skip (sa, sb', a, Just b) -- step 3
342− Skip sb' -> Skip (sa, sb', a, Nothing)
343− Stop -> Stop
344−
345− -- step 3, compare the two values
346− step _ (sa, sb, Just a, Just b) = do
347− let res = cmp a b
348− return $ case res of
349− GT -> Skip (sa, sb, Just a, Nothing) -- step 2
350− LT -> Skip (sa, sb, Nothing, Just b) -- step 1
351− EQ -> Yield a (sa, sb, Nothing, Just b) -- step 1
352−
353−-- | Returns a subsequence of the first stream, deleting first occurrences of
354−-- those elements that are present in the second stream. Note that this is not
355−-- a commutative operation. This is similar to the 'Data.List.deleteFirstsBy'.
258+-- | Delete all elements of the first stream from the seconds stream. If an
259+-- element occurs multiple times in the first stream as many occurrences of it
260+-- are deleted from the second stream.
356261--
357−-- >>> f xs ys = Stream.fold Fold.toList $ Stream.deleteFirstsBy (==) (Stream.fromList xs) (Stream.fromList ys)
358−-- >>> f [1,2,2,3,3,5] [1,2,2,3,4]
359−-- [2,3,5]
262+-- >>> Stream.fold Fold.toList $ Stream.deleteInStreamGenericBy (==) (Stream.fromList [1,2,3]) (Stream.fromList [1,2,2])
263+-- [2]
360264--
361−-- The following holds:
265+-- The following laws hold:
362266--
363−-- > deleteFirstsBy (==) (Stream.nub s2 `append` s1) s2 === s1
364−-- > deleteFirstsBy (==) (Stream.nub s2 `interleave` s1) s2 === s1
267+-- > deleteInStreamGenericBy (==) s1 (s1 `append` s2) === s2
268+-- > deleteInStreamGenericBy (==) s1 (s1 `interleave` s2) === s2
365269--
366−-- First stream can be infinite, second stream must be finite.
270+-- Same as the list 'Data.List.//' operation but with argument order flipped.
271+--
272+-- The first stream must be finite and must not block. Second stream is
273+-- processed only after the first stream is fully realized.
367274--
368275-- Space: O(m) where @m@ is the number of elements in the first stream.
369276--
@@ -371,83 +278,58 @@ sortedIntersectBy cmp (Stream stepa ta) (Stream stepb tb) =
371278-- @n@ is the number of elements in the second stream.
372279--
373280-- /Pre-release/
374−{-# INLINE deleteFirstsBy #-}
375−deleteFirstsBy :: Monad m =>
281+{-# INLINE deleteInStreamGenericBy #-}
282+deleteInStreamGenericBy :: Monad m =>
376283 (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
377−deleteFirstsBy eq s2 s1 =
378− -- XXX s2 can be a sorted mutable array and we can use binary
379− -- search to find. Mark the element deleted, count the deletions
380− -- and reconsolidate the array when a min number of elements is
381− -- deleted.
382−
383− -- XXX Use StreamK or list as second argument instead of Stream to avoid
384− -- concatEffect?
385− Stream.concatEffect $ do
386− xs <- Stream.toList s1
387− -- It reverses the list but that is fine.
388− let del x =
389− List.foldl' (\(ys,res) y ->
390− if x `eq` y
391− then (ys, True)
392− else (x:ys, res)) ([], False)
393− g (ys,_) x =
394− let (ys1, deleted) = del x ys
395− in if deleted
396− then (ys1, Nothing)
397− else (ys1, Just x)
398− in return
399− $ Stream.catMaybes
400− $ fmap snd
401− $ Stream.postscanl' g (xs, Nothing) s2
284+deleteInStreamGenericBy eq s1 s2 =
285+ Stream.concatEffect
286+ $ do
287+ -- This may work well if s1 is small
288+ -- If s1 is big we can go through s1, deleting elements from s2 and
289+ -- not emitting an element if it was successfully deleted from s2.
290+ -- we will need a deleteBy that can return whether the element was
291+ -- deleted or not.
292+ xs <- Stream.fold Fold.toList s2
293+ let f = Fold.foldl' (flip (List.deleteBy eq)) xs
294+ fmap Stream.fromList $ Stream.fold f s1
402295
403−-- | A more efficient 'deleteFirstsBy' for streams sorted in ascending order.
404−--
405−-- Both streams can be infinite.
296+-- | A more efficient 'deleteInStreamGenericBy' for streams sorted in ascending order.
406297--
407298-- Space: O(1)
408299--
409300-- /Unimplemented/
410−{-# INLINE sortedDeleteFirstsBy #-}
411−sortedDeleteFirstsBy :: -- (Monad m) =>
301+{-# INLINE deleteInStreamAscBy #-}
302+deleteInStreamAscBy :: -- (Monad m) =>
412303 (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
413−sortedDeleteFirstsBy _eq _s1 _s2 = undefined
304+deleteInStreamAscBy _eq _s1 _s2 = undefined
414305
415306-- XXX Remove the MonadIO constraint. We can just cache one stream and then
416307-- implement using differenceEqBy.
417308
418−-- | Returns the first stream appended with those unique elements from the
419−-- second stream that are not already present in the first stream. Note that
420−-- this is not a commutative operation unlike a set union, argument order
421−-- matters. The behavior is similar to the 'Data.List.unionBy'.
309+-- | This essentially appends to the second stream all the occurrences of
310+-- elements in the first stream that are not already present in the second
311+-- stream.
422312--
423313-- Equivalent to the following except that @s2@ is evaluated only once:
424314--
425−-- >>> unionBy eq s1 s2 = s1 `Stream.append` Stream.deleteFirstsBy eq s1 (Stream.nub s2)
315+-- >>> unionWithStreamGenericBy eq s1 s2 = s2 `Stream.append` (Stream.deleteInStreamGenericBy eq s2 s1)
426316--
427317-- Example:
428318--
429−-- >>> f s1 s2 = Stream.fold Fold.toList $ Stream.unionBy (==) (Stream.fromList s1) (Stream.fromList s2)
430−-- >>> f [1,2,2,4] [1,1,2,3,3]
319+-- >>> Stream.fold Fold.toList $ Stream.unionWithStreamGenericBy (==) (Stream.fromList [1,1,2,3]) (Stream.fromList [1,2,2,4])
431320-- [1,2,2,4,3]
432321--
433−-- First stream can be infinite, but second stream must be finite. Note that if
434−-- the first stream is infinite the union means just the first stream. Thus
435−-- union is useful only when both streams are finite. See 'sortedUnionBy' where
436−-- union can work on infinite streams if they are sorted.
437−--
438322-- Space: O(n)
439323--
440324-- Time: O(m x n)
441325--
442326-- /Pre-release/
443−{-# INLINE unionBy #-}
444−unionBy :: MonadIO m =>
327+{-# INLINE unionWithStreamGenericBy #-}
328+unionWithStreamGenericBy :: MonadIO m =>
445329 (a -> a -> Bool) -> Stream m a -> Stream m a -> Stream m a
446−unionBy eq s2 s1 =
330+unionWithStreamGenericBy eq s1 s2 =
447331 Stream.concatEffect
448332 $ do
449− -- XXX use a rewrite rule such that if a list converted to stream
450− -- is passed to unionBy then this becomes an identity operation.
451333 xs <- Stream.fold Fold.toList s1
452334 -- XXX we can use postscanlMAfter' instead of IORef
453335 ref <- liftIO $ newIORef $! List.nubBy eq xs
@@ -460,21 +342,12 @@ unionBy eq s2 s1 =
460342 return $ Stream.fromList xs1
461343 return $ Stream.mapM f s2 `Stream.append` s3
462344
463−-- | A more efficient 'unionBy' for sorted streams.
464−--
465−-- Note that the behavior is different from 'unionBy'. In 'unionBy' we append
466−-- the unique elements from second stream only after exhausting the first one
467−-- whereas in sorted streams we can determine unique elements early even when
468−-- we are going through the first stream. Thus the result is an interleaving of
469−-- the two streams, merging those elements from the second stream that are not
470−-- present in the first.
345+-- | A more efficient 'unionWithStreamGenericBy' for sorted streams.
471346--
472347-- Space: O(1)
473348--
474−-- Both streams can be infinite.
475−--
476349-- /Unimplemented/
477−{-# INLINE sortedUnionBy #-}
478−sortedUnionBy :: -- (Monad m) =>
350+{-# INLINE unionWithStreamAscBy #-}
351+unionWithStreamAscBy :: -- (Monad m) =>
479352 (a -> a -> Ordering) -> Stream m a -> Stream m a -> Stream m a
480−sortedUnionBy _eq _s1 _s2 = undefined
353+unionWithStreamAscBy _eq _s1 _s2 = undefined
core/src/Streamly/Internal/Data/Stream/Transform.hs+35−61
@@ -21,6 +21,8 @@ module Streamly.Internal.Data.Stream.Transform
2121
2222 -- * Mapping
2323 -- | Stateless one-to-one maps.
24+ , map
25+ , mapM
2426 , sequence
2527
2628 -- * Mapping Effects
@@ -73,7 +75,7 @@ module Streamly.Internal.Data.Stream.Transform
7375 , scanMaybe
7476 , filter
7577 , filterM
76− , deleteBy -- deleteOnceBy?
78+ , deleteBy
7779 , uniqBy
7880 , uniq
7981 , prune
@@ -81,8 +83,9 @@ module Streamly.Internal.Data.Stream.Transform
8183
8284 -- * Trimming
8385 -- | Produce a subset of the stream trimmed at ends.
84− , initNonEmpty
85− , tailNonEmpty
86+ , take
87+ , takeWhile
88+ , takeWhileM
8689 , takeWhileLast
8790 , takeWhileAround
8891 , drop
@@ -129,6 +132,7 @@ module Streamly.Internal.Data.Stream.Transform
129132 -- * Searching
130133 , findIndices
131134 , elemIndices
135+ , slicesBy
132136
133137 -- * Rolling map
134138 -- | Map using the previous element.
@@ -296,7 +300,7 @@ data TapState fs st a
296300--
297301{-# INLINE tap #-}
298302tap :: Monad m => Fold m a b -> Stream m a -> Stream m a
299−tap (Fold fstep initial _ final) (Stream step state) = Stream step' TapInit
303+tap (Fold fstep initial extract) (Stream step state) = Stream step' TapInit
300304
301305 where
302306
@@ -319,7 +323,7 @@ tap (Fold fstep initial _ final) (Stream step state) = Stream step' TapInit
319323 FL.Done _ -> TapDone s
320324 Skip s -> return $ Skip (Tapping acc s)
321325 Stop -> do
322− void $ final acc
326+ void $ extract acc
323327 return Stop
324328 step' gst (TapDone st) = do
325329 r <- step gst st
@@ -338,7 +342,7 @@ data TapOffState fs s a
338342{-# INLINE_NORMAL tapOffsetEvery #-}
339343tapOffsetEvery :: Monad m
340344 => Int -> Int -> Fold m a b -> Stream m a -> Stream m a
341−tapOffsetEvery offset n (Fold fstep initial _ final) (Stream step state) =
345+tapOffsetEvery offset n (Fold fstep initial extract) (Stream step state) =
342346 Stream step' TapOffInit
343347
344348 where
@@ -368,7 +372,7 @@ tapOffsetEvery offset n (Fold fstep initial _ final) (Stream step state) =
368372 return $ Yield x next
369373 Skip s -> return $ Skip (TapOffTapping acc s count)
370374 Stop -> do
371− void $ final acc
375+ void $ extract acc
372376 return Stop
373377 step' gst (TapOffDone st) = do
374378 r <- step gst st
@@ -433,7 +437,7 @@ data ScanState s f = ScanInit s | ScanDo s !f | ScanDone
433437--
434438{-# INLINE_NORMAL postscan #-}
435439postscan :: Monad m => FL.Fold m a b -> Stream m a -> Stream m b
436−postscan (FL.Fold fstep initial extract final) (Stream sstep state) =
440+postscan (FL.Fold fstep initial extract) (Stream sstep state) =
437441 Stream step (ScanInit state)
438442
439443 where
@@ -456,13 +460,13 @@ postscan (FL.Fold fstep initial extract final) (Stream sstep state) =
456460 return $ Yield b $ ScanDo s fs1
457461 FL.Done b -> return $ Yield b ScanDone
458462 Skip s -> return $ Skip $ ScanDo s fs
459− Stop -> final fs >> return Stop
463+ Stop -> return Stop
460464 step _ ScanDone = return Stop
461465
462466{-# INLINE scanWith #-}
463467scanWith :: Monad m
464468 => Bool -> Fold m a b -> Stream m a -> Stream m b
465−scanWith restart (Fold fstep initial extract final) (Stream sstep state) =
469+scanWith restart (Fold fstep initial extract) (Stream sstep state) =
466470 Stream step (ScanInit state)
467471
468472 where
@@ -485,7 +489,7 @@ scanWith restart (Fold fstep initial extract final) (Stream sstep state) =
485489 case res of
486490 Yield x s -> runStep s (fstep fs x)
487491 Skip s -> return $ Skip $ ScanDo s fs
488− Stop -> final fs >> return Stop
492+ Stop -> return Stop
489493 step _ ScanDone = return Stop
490494
491495-- XXX It may be useful to have a version of scan where we can keep the
@@ -1070,55 +1074,6 @@ repeated = undefined
10701074-- Trimming
10711075------------------------------------------------------------------------------
10721076
1073−-- | init for non-empty streams, fails for empty stream case.
1074−--
1075−{-# INLINE initNonEmpty #-}
1076−initNonEmpty :: Monad m => Stream m a -> Stream m a
1077−initNonEmpty (Stream step1 state1) = Stream step (Nothing, state1)
1078−
1079− where
1080−
1081− step gst (Nothing, s1) = do
1082− r <- step1 (adaptState gst) s1
1083− return $
1084− case r of
1085− Yield x s -> Skip (Just x, s)
1086− Skip s -> Skip (Nothing, s)
1087− Stop -> error "initNonEmpty: empty Stream"
1088−
1089− step gst (Just a, s1) = do
1090− r <- step1 (adaptState gst) s1
1091− return $
1092− case r of
1093− Yield x s -> Yield a (Just x, s)
1094− Skip s -> Skip (Just a, s)
1095− Stop -> Stop
1096−
1097−-- | tail for non-empty streams, fails for empty stream case.
1098−--
1099−-- See also 'tail' for a non-partial version of this function..
1100−{-# INLINE tailNonEmpty #-}
1101−tailNonEmpty :: Monad m => Stream m a -> Stream m a
1102−tailNonEmpty (Stream step1 state1) = Stream step (Nothing, state1)
1103−
1104− where
1105−
1106− step gst (Nothing, s1) = do
1107− r <- step1 (adaptState gst) s1
1108− return $
1109− case r of
1110− Yield x s -> Skip (Just x, s)
1111− Skip s -> Skip (Nothing, s)
1112− Stop -> error "tailNonEmpty: empty Stream"
1113−
1114− step gst (Just a, s1) = do
1115− r <- step1 (adaptState gst) s1
1116− return $
1117− case r of
1118− Yield x s -> Yield x (Just x, s)
1119− Skip s -> Skip (Just a, s)
1120− Stop -> Stop
1121−
11221077-- | Take all consecutive elements at the end of the stream for which the
11231078-- predicate is true.
11241079--
@@ -1582,7 +1537,7 @@ reverse m = Stream step Nothing
15821537{-# INLINE reverseUnbox #-}
15831538reverseUnbox :: (MonadIO m, Unbox a) => Stream m a -> Stream m a
15841539reverseUnbox =
1585− A.concatRev -- unfoldMany A.readerRev
1540+ A.flattenArraysRev -- unfoldMany A.readRev
15861541 . fromStreamK
15871542 . K.reverse
15881543 . toStreamK
@@ -1754,6 +1709,25 @@ findIndices p (Stream step state) = Stream step' (state, 0)
17541709elemIndices :: (Monad m, Eq a) => a -> Stream m a -> Stream m Int
17551710elemIndices a = findIndices (== a)
17561711
1712+{-# INLINE_NORMAL slicesBy #-}
1713+slicesBy :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
1714+slicesBy p (Stream step1 state1) = Stream step (Just (state1, 0, 0))
1715+
1716+ where
1717+
1718+ {-# INLINE_LATE step #-}
1719+ step gst (Just (st, i, len)) = i `seq` len `seq` do
1720+ r <- step1 (adaptState gst) st
1721+ return
1722+ $ case r of
1723+ Yield x s ->
1724+ if p x
1725+ then Yield (i, len + 1) (Just (s, i + len + 1, 0))
1726+ else Skip (Just (s, i, len + 1))
1727+ Skip s -> Skip (Just (s, i, len))
1728+ Stop -> if len == 0 then Stop else Yield (i, len) Nothing
1729+ step _ Nothing = return Stop
1730+
17571731------------------------------------------------------------------------------
17581732-- Rolling map
17591733------------------------------------------------------------------------------
core/src/Streamly/Internal/Data/Stream/Transformer.hs+1−2
@@ -11,11 +11,10 @@
1111
1212module Streamly.Internal.Data.Stream.Transformer
1313 (
14− -- * Fold to Transformer Monad
1514 foldlT
1615 , foldrT
1716
18− -- * Inner Monad Operations
17+ -- * Transform Inner Monad
1918 , liftInner
2019 , runReaderT
2120 , usingReaderT
core/src/Streamly/Internal/Data/Stream/Type.hs+14−147
@@ -1,11 +1,5 @@
11{-# LANGUAGE CPP #-}
2−{-# LANGUAGE PatternSynonyms #-}
3−{-# LANGUAGE TypeFamilies #-}
4−-- Must come after TypeFamilies, otherwise it is re-enabled.
5−-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
6−{-# LANGUAGE NoMonoLocalBinds #-}
72{-# LANGUAGE UndecidableInstances #-}
8−{-# LANGUAGE ViewPatterns #-}
93
104-- |
115-- Module : Streamly.Internal.Data.Stream.Type
@@ -22,18 +16,18 @@
2216
2317module Streamly.Internal.Data.Stream.Type
2418 (
25− -- * Type
19+ -- * The stream type
2620 Step (..)
2721 -- XXX UnStream is exported to avoid a performance issue in some
2822 -- combinators if we use the pattern synonym "Stream".
2923 , Stream (Stream, UnStream)
3024
31− -- * CrossStream
25+ -- * CrossStream type wrapper
3226 , CrossStream
3327 , unCross
3428 , mkCross
3529
36− -- * To StreamK
30+ -- * Conversion to StreamK
3731 , fromStreamK
3832 , toStreamK
3933
@@ -91,7 +85,6 @@ module Streamly.Internal.Data.Stream.Type
9185
9286 -- * Combining Two Streams
9387 -- ** Zipping
94− -- | Zip corresponding elements of two streams.
9588 , zipWithM
9689 , zipWith
9790
@@ -107,8 +100,6 @@ module Streamly.Internal.Data.Stream.Type
107100 , unfoldMany
108101
109102 -- * Concat
110− -- | Generate streams by mapping a stream generator on each element of an
111− -- input stream, append the resulting streams and flatten.
112103 , concatEffect
113104 , concatMap
114105 , concatMapM
@@ -132,30 +123,20 @@ module Streamly.Internal.Data.Stream.Type
132123 , foldManyPost
133124 , groupsOf
134125 , refoldMany
135− , refoldIterateM
136126
137127 -- * Fold Iterate
138128 , reduceIterateBfs
139129 , foldIterateBfs
140130
141− -- * Splitting
142− , indexOnSuffix
143−
144131 -- * Multi-stream folds
145− -- | These should probably be expressed using zipping operations.
146132 , eqBy
147133 , cmpBy
148−
149− -- * Deprecated
150− , sliceOnSuffix
151134 )
152135where
153136
154137#include "inline.hs"
155138
156−#if !MIN_VERSION_base(4,18,0)
157139import Control.Applicative (liftA2)
158−#endif
159140import Control.Monad.Catch (MonadThrow, throwM)
160141import Control.Monad.Trans.Class (MonadTrans(lift))
161142import Control.Monad.IO.Class (MonadIO(..))
@@ -179,7 +160,6 @@ import Streamly.Internal.Data.Maybe.Strict (Maybe'(..), toMaybe)
179160import Streamly.Internal.Data.Refold.Type (Refold(..))
180161import Streamly.Internal.Data.Stream.Step (Step (..))
181162import Streamly.Internal.Data.SVar.Type (State, adaptState, defState)
182−import Streamly.Internal.Data.Tuple.Strict (Tuple'(..))
183163import Streamly.Internal.Data.Unfold.Type (Unfold(..))
184164
185165import qualified Streamly.Internal.Data.Fold.Type as FL hiding (foldr)
@@ -230,9 +210,6 @@ pattern Stream step state <- (unShare -> UnStream step state)
230210nilM :: Applicative m => m b -> Stream m a
231211nilM m = Stream (\_ _ -> m $> Stop) ()
232212
233−
234−infixr 5 `consM`
235−
236213-- | Like 'cons' but fuses an effect instead of a pure value.
237214{-# INLINE_NORMAL consM #-}
238215consM :: Applicative m => m a -> Stream m a -> Stream m a
@@ -419,7 +396,7 @@ toStreamK (Stream step state) = go state
419396{-# INLINE_NORMAL foldEither #-}
420397foldEither :: Monad m =>
421398 Fold m a b -> Stream m a -> m (Either (Fold m a b) (b, Stream m a))
422−foldEither (Fold fstep begin done final) (UnStream step state) = do
399+foldEither (Fold fstep begin done) (UnStream step state) = do
423400 res <- begin
424401 case res of
425402 FL.Partial fs -> go SPEC fs state
@@ -437,9 +414,7 @@ foldEither (Fold fstep begin done final) (UnStream step state) = do
437414 FL.Done b -> return $! Right (b, Stream step s)
438415 FL.Partial fs1 -> go SPEC fs1 s
439416 Skip s -> go SPEC fs s
440− Stop ->
441− let f = Fold fstep (return $ FL.Partial fs) done final
442− in return $! Left f
417+ Stop -> return $! Left (Fold fstep (return $ FL.Partial fs) done)
443418
444419-- | Like 'fold' but also returns the remaining stream. The resulting stream
445420-- would be 'Stream.nil' if the stream finished before the fold.
@@ -450,12 +425,12 @@ foldBreak fld strm = do
450425 r <- foldEither fld strm
451426 case r of
452427 Right res -> return res
453− Left (Fold _ initial _ final) -> do
428+ Left (Fold _ initial extract) -> do
454429 res <- initial
455430 case res of
456431 FL.Done _ -> error "foldBreak: unreachable state"
457432 FL.Partial s -> do
458− b <- final s
433+ b <- extract s
459434 return (b, nil)
460435
461436 where
@@ -498,8 +473,8 @@ fold fld strm = do
498473--
499474{-# INLINE_NORMAL foldAddLazy #-}
500475foldAddLazy :: Monad m => Fold m a b -> Stream m a -> Fold m a b
501−foldAddLazy (Fold fstep finitial fextract ffinal) (Stream sstep state) =
502− Fold fstep initial fextract ffinal
476+foldAddLazy (Fold fstep finitial fextract) (Stream sstep state) =
477+ Fold fstep initial fextract
503478
504479 where
505480
@@ -572,12 +547,8 @@ foldAdd f =
572547-- >>> Stream.foldrM step (return False) s
573548-- True
574549--
575−-- >>> import Control.Monad (join)
576−-- >>> foldrM f z = join $ Stream.foldr f z
577−--
578550{-# INLINE_NORMAL foldrM #-}
579551foldrM :: Monad m => (a -> m b -> m b) -> m b -> Stream m a -> m b
580−-- foldrM f z = join . Streamly.Internal.Data.Stream.StreamD.Type.foldr f z
581552foldrM f z (Stream step state) = go SPEC state
582553 where
583554 {-# INLINE_LATE go #-}
@@ -1823,7 +1794,7 @@ data FoldManyPost s fs b a
18231794--
18241795{-# INLINE_NORMAL foldManyPost #-}
18251796foldManyPost :: Monad m => Fold m a b -> Stream m a -> Stream m b
1826−foldManyPost (Fold fstep initial _ final) (Stream step state) =
1797+foldManyPost (Fold fstep initial extract) (Stream step state) =
18271798 Stream step' (FoldManyPostStart state)
18281799
18291800 where
@@ -1851,7 +1822,7 @@ foldManyPost (Fold fstep initial _ final) (Stream step state) =
18511822 Yield x s -> consume x s fs
18521823 Skip s -> return $ Skip (FoldManyPostLoop s fs)
18531824 Stop -> do
1854− b <- final fs
1825+ b <- extract fs
18551826 return $ Skip (FoldManyPostYield b FoldManyPostDone)
18561827 step' _ (FoldManyPostYield b next) = return $ Yield b next
18571828 step' _ FoldManyPostDone = return Stop
@@ -1895,7 +1866,7 @@ data FoldMany s fs b a
18951866--
18961867{-# INLINE_NORMAL foldMany #-}
18971868foldMany :: Monad m => Fold m a b -> Stream m a -> Stream m b
1898−foldMany (Fold fstep initial _ final) (Stream step state) =
1869+foldMany (Fold fstep initial extract) (Stream step state) =
18991870 Stream step' (FoldManyStart state)
19001871
19011872 where
@@ -1922,14 +1893,14 @@ foldMany (Fold fstep initial _ final) (Stream step state) =
19221893 case r of
19231894 Yield x s -> consume x s fs
19241895 Skip s -> return $ Skip (FoldManyFirst fs s)
1925− Stop -> final fs >> return Stop
1896+ Stop -> return Stop
19261897 step' gst (FoldManyLoop st fs) = do
19271898 r <- step (adaptState gst) st
19281899 case r of
19291900 Yield x s -> consume x s fs
19301901 Skip s -> return $ Skip (FoldManyLoop s fs)
19311902 Stop -> do
1932− b <- final fs
1903+ b <- extract fs
19331904 return $ Skip (FoldManyYield b FoldManyDone)
19341905 step' _ (FoldManyYield b next) = return $ Yield b next
19351906 step' _ FoldManyDone = return Stop
@@ -1996,110 +1967,6 @@ refoldMany (Refold fstep inject extract) action (Stream step state) =
19961967 step' _ (FoldManyYield b next) = return $ Yield b next
19971968 step' _ FoldManyDone = return Stop
19981969
1999−{-# ANN type CIterState Fuse #-}
2000−data CIterState s f fs b
2001− = CIterInit s f
2002− | CIterConsume s fs
2003− | CIterYield b (CIterState s f fs b)
2004− | CIterStop
2005−
2006−-- | Like 'foldIterateM' but using the 'Refold' type instead. This could be
2007−-- much more efficient due to stream fusion.
2008−--
2009−-- /Internal/
2010−{-# INLINE_NORMAL refoldIterateM #-}
2011−refoldIterateM ::
2012− Monad m => Refold m b a b -> m b -> Stream m a -> Stream m b
2013−refoldIterateM (Refold fstep finject fextract) initial (Stream step state) =
2014− Stream stepOuter (CIterInit state initial)
2015−
2016− where
2017−
2018− {-# INLINE iterStep #-}
2019− iterStep st action = do
2020− res <- action
2021− return
2022− $ Skip
2023− $ case res of
2024− FL.Partial fs -> CIterConsume st fs
2025− FL.Done fb -> CIterYield fb $ CIterInit st (return fb)
2026−
2027− {-# INLINE_LATE stepOuter #-}
2028− stepOuter _ (CIterInit st action) = do
2029− iterStep st (action >>= finject)
2030− stepOuter gst (CIterConsume st fs) = do
2031− r <- step (adaptState gst) st
2032− case r of
2033− Yield x s -> iterStep s (fstep fs x)
2034− Skip s -> return $ Skip $ CIterConsume s fs
2035− Stop -> do
2036− b <- fextract fs
2037− return $ Skip $ CIterYield b CIterStop
2038− stepOuter _ (CIterYield a next) = return $ Yield a next
2039− stepOuter _ CIterStop = return Stop
2040−
2041−-- | The refold @indexerBy f n@ takes an (index, len) tuple as initial input,
2042−-- and returns @(index + len + n, b)@ as output where @b@ is the output of the
2043−-- fold.
2044−{-# INLINE indexerBy #-}
2045−indexerBy :: Monad m =>
2046− Fold m a Int -> Int -> Refold m (Int, Int) a (Int, Int)
2047−indexerBy (Fold step1 initial1 extract1 _final) n =
2048− Refold step inject extract
2049−
2050− where
2051−
2052− inject (i, len) = do
2053− r <- initial1
2054− return $ case r of
2055− FL.Partial s -> FL.Partial $ Tuple' (i + len + n) s
2056− FL.Done l -> FL.Done (i, l)
2057−
2058− step (Tuple' i s) x = do
2059− r <- step1 s x
2060− return $ case r of
2061− FL.Partial s1 -> FL.Partial $ Tuple' i s1
2062− FL.Done len -> FL.Done (i, len)
2063−
2064− extract (Tuple' i s) = (i,) <$> extract1 s
2065−
2066−-- | Like 'splitOnSuffix' but generates a stream of (index, len) tuples marking
2067−-- the places where the predicate matches in the stream.
2068−--
2069−-- /Pre-release/
2070−{-# INLINE indexOnSuffix #-}
2071−indexOnSuffix :: Monad m =>
2072− (a -> Bool) -> Stream m a -> Stream m (Int, Int)
2073−indexOnSuffix predicate =
2074− -- Scan the stream with the given refold
2075− refoldIterateM
2076− (indexerBy (FL.takeEndBy_ predicate FL.length) 1)
2077− (return (-1, 0))
2078−
2079−-- Alternate implementation
2080−{-# INLINE_NORMAL _indexOnSuffix #-}
2081−_indexOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
2082−_indexOnSuffix p (Stream step1 state1) = Stream step (Just (state1, 0, 0))
2083−
2084− where
2085−
2086− {-# INLINE_LATE step #-}
2087− step gst (Just (st, i, len)) = i `seq` len `seq` do
2088− r <- step1 (adaptState gst) st
2089− return
2090− $ case r of
2091− Yield x s ->
2092− if p x
2093− then Yield (i, len + 1) (Just (s, i + len + 1, 0))
2094− else Skip (Just (s, i, len + 1))
2095− Skip s -> Skip (Just (s, i, len))
2096− Stop -> if len == 0 then Stop else Yield (i, len) Nothing
2097− step _ Nothing = return Stop
2098−
2099−{-# DEPRECATED sliceOnSuffix "Please use indexOnSuffix instead." #-}
2100−sliceOnSuffix :: Monad m => (a -> Bool) -> Stream m a -> Stream m (Int, Int)
2101−sliceOnSuffix = indexOnSuffix
2102−
21031970------------------------------------------------------------------------------
21041971-- Stream with a cross product style monad instance
21051972------------------------------------------------------------------------------
core/src/Streamly/Internal/Data/StreamK.hs+40−35
@@ -16,15 +16,16 @@ module Streamly.Internal.Data.StreamK
1616 --
1717 -- $setup
1818
19+ -- * The stream type
1920 module Streamly.Internal.Data.StreamK.Type
20− -- * Transformer
2121 , module Streamly.Internal.Data.StreamK.Transformer
2222
23− -- * From containers
23+ , StreamK(..)
2424 , fromList
2525 , fromStream
26+ , toStream
2627
27− -- * Specialized Generation
28+ -- ** Specialized Generation
2829 , repeatM
2930 , replicate
3031 , replicateM
@@ -36,6 +37,8 @@ module Streamly.Internal.Data.StreamK
3637 -- * Elimination
3738 -- ** General Folds
3839 , foldr1
40+ , foldlM'
41+ , foldlMx'
3942 , fold
4043 , foldBreak
4144 , foldEither
@@ -66,13 +69,13 @@ module Streamly.Internal.Data.StreamK
6669 , find
6770 , (!!)
6871
69− -- ** To Containers
70− , toList
71− , toStream
72−
7372 -- ** Map and Fold
7473 , mapM_
7574
75+ -- ** Conversions
76+ , toList
77+ , hoist
78+
7679 -- * Transformation
7780 -- ** By folding (scans)
7881 , scanl'
@@ -99,7 +102,6 @@ module Streamly.Internal.Data.StreamK
99102
100103 -- ** Reordering
101104 , sortBy
102− , sortOn
103105
104106 -- ** Map and Filter
105107 , mapMaybe
@@ -115,9 +117,6 @@ module Streamly.Internal.Data.StreamK
115117 -- ** Transformation comprehensions
116118 , the
117119
118− -- ** Transforming Inner Monad
119− , hoist
120−
121120 -- * Exceptions
122121 , handle
123122
@@ -134,7 +133,6 @@ import Control.Exception (mask_, Exception)
134133import Control.Monad (void, join)
135134import Control.Monad.Catch (MonadCatch)
136135import Control.Monad.IO.Class (MonadIO(..))
137−import Data.Ord (comparing)
138136import Data.Proxy (Proxy(..))
139137import GHC.Types (SPEC(..))
140138import Streamly.Internal.Data.Array.Type (Array(..))
@@ -308,6 +306,19 @@ foldr1 step m = do
308306 yieldk a r = fmap (step p) (go a r)
309307 in foldStream defState yieldk single stp m1
310308
309+-- XXX replace the recursive "go" with explicit continuations.
310+-- | Like 'foldx', but with a monadic step function.
311+{-# INLINABLE foldlMx' #-}
312+foldlMx' :: Monad m
313+ => (x -> a -> m x) -> m x -> (x -> m b) -> StreamK m a -> m b
314+foldlMx' step begin done = go begin
315+ where
316+ go !acc m1 =
317+ let stop = acc >>= done
318+ single a = acc >>= \b -> step b a >>= done
319+ yieldk a r = acc >>= \b -> step b a >>= \x -> go (return x) r
320+ in foldStream defState yieldk single stop m1
321+
311322-- | Fold a stream using the supplied left 'Fold' and reducing the resulting
312323-- expression strictly at each step. The behavior is similar to 'foldl''. A
313324-- 'Fold' can terminate early without consuming the full stream. See the
@@ -325,7 +336,7 @@ foldr1 step m = do
325336--
326337{-# INLINABLE fold #-}
327338fold :: Monad m => FL.Fold m a b -> StreamK m a -> m b
328−fold (FL.Fold step begin _ final) m = do
339+fold (FL.Fold step begin done) m = do
329340 res <- begin
330341 case res of
331342 FL.Partial fs -> go fs m
@@ -333,10 +344,10 @@ fold (FL.Fold step begin _ final) m = do
333344
334345 where
335346 go !acc m1 =
336− let stop = final acc
347+ let stop = done acc
337348 single a = step acc a
338349 >>= \case
339− FL.Partial s -> final s
350+ FL.Partial s -> done s
340351 FL.Done b1 -> return b1
341352 yieldk a r = step acc a
342353 >>= \case
@@ -354,7 +365,7 @@ fold (FL.Fold step begin _ final) m = do
354365{-# INLINE foldEither #-}
355366foldEither :: Monad m =>
356367 Fold m a b -> StreamK m a -> m (Either (Fold m a b) (b, StreamK m a))
357−foldEither (FL.Fold step begin done final) m = do
368+foldEither (FL.Fold step begin done) m = do
358369 res <- begin
359370 case res of
360371 FL.Partial fs -> go fs m
@@ -363,15 +374,12 @@ foldEither (FL.Fold step begin done final) m = do
363374 where
364375
365376 go !acc m1 =
366− let stop =
367− let f = Fold step (return $ FL.Partial acc) done final
368− in return $ Left f
377+ let stop = return $ Left (Fold step (return $ FL.Partial acc) done)
369378 single a =
370379 step acc a
371380 >>= \case
372381 FL.Partial s ->
373− let f = Fold step (return $ FL.Partial s) done final
374− in return $ Left f
382+ return $ Left (Fold step (return $ FL.Partial s) done)
375383 FL.Done b1 -> return $ Right (b1, nil)
376384 yieldk a r =
377385 step acc a
@@ -389,12 +397,12 @@ foldBreak fld strm = do
389397 r <- foldEither fld strm
390398 case r of
391399 Right res -> return res
392− Left (Fold _ initial _ final) -> do
400+ Left (Fold _ initial extract) -> do
393401 res <- initial
394402 case res of
395403 FL.Done _ -> error "foldBreak: unreachable state"
396404 FL.Partial s -> do
397− b <- final s
405+ b <- extract s
398406 return (b, nil)
399407
400408-- XXX Array folds can be implemented using this.
@@ -413,7 +421,7 @@ foldConcat :: Monad m =>
413421 Producer m a b -> Fold m b c -> StreamK m a -> m (c, StreamK m a)
414422foldConcat
415423 (Producer pstep pinject pextract)
416− (Fold fstep begin _ final)
424+ (Fold fstep begin done)
417425 stream = do
418426
419427 res <- begin
@@ -425,14 +433,14 @@ foldConcat
425433
426434 go !acc m1 = do
427435 let stop = do
428− r <- final acc
436+ r <- done acc
429437 return (r, nil)
430438 single a = do
431439 st <- pinject a
432440 res <- go1 SPEC acc st
433441 case res of
434442 Left fs -> do
435− r <- final fs
443+ r <- done fs
436444 return (r, nil)
437445 Right (b, s) -> do
438446 x <- pextract s
@@ -459,6 +467,11 @@ foldConcat
459467 Stream.Skip s -> go1 SPEC fs s
460468 Stream.Stop -> return $ Left fs
461469
470+-- | Like 'foldl'' but with a monadic step function.
471+{-# INLINE foldlM' #-}
472+foldlM' :: Monad m => (b -> a -> m b) -> m b -> StreamK m a -> m b
473+foldlM' step begin = foldlMx' step begin return
474+
462475------------------------------------------------------------------------------
463476-- Specialized folds
464477------------------------------------------------------------------------------
@@ -1315,8 +1328,7 @@ parserDone (ParserK.Failure n e) _ _ = pure $ ParserK.Error n e
13151328-- using parseBreakChunks. We can also use parseBreak as an alternative to the
13161329-- monad instance of ParserD.
13171330
1318−-- | Run a 'ParserK' over a chunked 'StreamK' and return the parse result and
1319−-- the remaining Stream.
1331+-- | Run a 'ParserK' over a chunked 'StreamK' and return the rest of the Stream.
13201332{-# INLINE_NORMAL parseBreakChunks #-}
13211333parseBreakChunks
13221334 :: (Monad m, Unbox a)
@@ -1721,10 +1733,3 @@ sortBy cmp =
17211733 . Stream.catRights -- its a non-failing backtracking parser
17221734 . Stream.parseMany (fmap (either id id) p)
17231735 . Stream.fromStreamK
1724−
1725−{-# INLINE sortOn #-}
1726−sortOn :: (Monad m, Ord b) => (a -> b) -> StreamK m a -> StreamK m a
1727−sortOn f =
1728− fmap snd
1729− . sortBy (comparing fst)
1730− . fmap (\x -> let y = f x in y `seq` (y, x))
core/src/Streamly/Internal/Data/StreamK/Type.hs+11−82
@@ -1,8 +1,4 @@
11{-# LANGUAGE CPP #-}
2−{-# LANGUAGE TypeFamilies #-}
3−-- Must come after TypeFamilies, otherwise it is re-enabled.
4−-- MonoLocalBinds enabled by TypeFamilies causes perf regressions in general.
5−{-# LANGUAGE NoMonoLocalBinds #-}
62{-# LANGUAGE UndecidableInstances #-}
73-- |
84-- Module : Streamly.Internal.Data.StreamK.Type
@@ -88,8 +84,6 @@ module Streamly.Internal.Data.StreamK.Type
8884 -- ** Strict Left Folds
8985 , Streamly.Internal.Data.StreamK.Type.foldl'
9086 , foldlx'
91− , foldlMx'
92− , foldlM'
9387
9488 -- ** Lazy Right Folds
9589 , Streamly.Internal.Data.StreamK.Type.foldr
@@ -97,11 +91,8 @@ module Streamly.Internal.Data.StreamK.Type
9791 -- ** Specific Folds
9892 , drain
9993 , null
100− , headNonEmpty
10194 , tail
102− , tailNonEmpty
10395 , init
104− , initNonEmpty
10596
10697 -- * Mapping
10798 , map
@@ -153,9 +144,7 @@ where
153144import Control.Monad ((>=>))
154145import Control.Monad.Catch (MonadThrow, throwM)
155146import Control.Monad.Trans.Class (MonadTrans(lift))
156−#if !MIN_VERSION_base(4,18,0)
157147import Control.Applicative (liftA2)
158−#endif
159148import Control.Monad.IO.Class (MonadIO(..))
160149import Data.Foldable (Foldable(foldl'), fold, foldr)
161150import Data.Function (fix)
@@ -826,24 +815,6 @@ foldlx' step begin done m = get $ go m begin
826815foldl' :: Monad m => (b -> a -> b) -> b -> StreamK m a -> m b
827816foldl' step begin = foldlx' step begin id
828817
829−-- XXX replace the recursive "go" with explicit continuations.
830−-- | Like 'foldx', but with a monadic step function.
831−{-# INLINABLE foldlMx' #-}
832−foldlMx' :: Monad m
833− => (x -> a -> m x) -> m x -> (x -> m b) -> StreamK m a -> m b
834−foldlMx' step begin done = go begin
835− where
836− go !acc m1 =
837− let stop = acc >>= done
838− single a = acc >>= \b -> step b a >>= done
839− yieldk a r = acc >>= \b -> step b a >>= \x -> go (return x) r
840− in foldStream defState yieldk single stop m1
841−
842−-- | Like 'foldl'' but with a monadic step function.
843−{-# INLINE foldlM' #-}
844−foldlM' :: Monad m => (b -> a -> m b) -> m b -> StreamK m a -> m b
845−foldlM' step begin = foldlMx' step begin return
846−
847818------------------------------------------------------------------------------
848819-- Specialized folds
849820------------------------------------------------------------------------------
@@ -1814,42 +1785,14 @@ iterateMWith cns step = go
18141785 !next <- s
18151786 foldStreamShared st stp sng yld (return next `cns` go (step next))
18161787
1817−-- | head for non-empty streams, fails for empty stream case.
1818−--
1819−{-# INLINE headNonEmpty #-}
1820−headNonEmpty :: Monad m => StreamK m a -> m a
1821−headNonEmpty = foldrM (\x _ -> return x) (error "headNonEmpty: empty stream")
1822−
1823−-- | init for non-empty streams, fails for empty stream case.
1824−--
1825−-- See also 'init' for a non-partial version of this function..
1826−{-# INLINE initNonEmpty #-}
1827−initNonEmpty :: Stream m a -> Stream m a
1828−initNonEmpty = go0
1829−
1830− where
1831−
1832− go0 m = mkStream $ \st yld sng stp ->
1833− let stop = error "initNonEmpty: Empty Stream."
1834− single _ = stp
1835− yieldk a r = foldStream st yld sng stp (go1 a r)
1836− in foldStream st yieldk single stop m
1837−
1838− go1 a r = mkStream $ \st yld sng stp ->
1839− let stop = stp
1840− single _ = sng a
1841− yieldk a1 r1 = yld a (go1 a1 r1)
1842− in foldStream st yieldk single stop r
1788+{-# INLINE headPartial #-}
1789+headPartial :: Monad m => StreamK m a -> m a
1790+headPartial = foldrM (\x _ -> return x) (error "head of nil")
18431791
1844−-- | tail for non-empty streams, fails for empty stream case.
1845−--
1846−-- See also 'tail' for a non-partial version of this function..
1847−--
1848−-- Note: this is same as "drop 1" with error on empty stream.
1849−{-# INLINE tailNonEmpty #-}
1850−tailNonEmpty :: StreamK m a -> StreamK m a
1851−tailNonEmpty m = mkStream $ \st yld sng stp ->
1852− let stop = error "tailNonEmpty: empty stream"
1792+{-# INLINE tailPartial #-}
1793+tailPartial :: StreamK m a -> StreamK m a
1794+tailPartial m = mkStream $ \st yld sng stp ->
1795+ let stop = error "tail of nil"
18531796 single _ = stp
18541797 yieldk _ r = foldStream st yld sng stp r
18551798 in foldStream st yieldk single stop m
@@ -1914,10 +1857,10 @@ mfix f = mkStream $ \st yld sng stp ->
19141857 where
19151858
19161859 -- fix the head element of the stream
1917− xs = fix (f . headNonEmpty)
1860+ xs = fix (f . headPartial)
19181861
19191862 -- now fix the tail recursively
1920− ys = mfix (tailNonEmpty . f)
1863+ ys = mfix (tailPartial . f)
19211864
19221865-------------------------------------------------------------------------------
19231866-- Conversions
@@ -1948,15 +1891,6 @@ uncons m =
19481891 yieldk a r = pure (Just (a, r))
19491892 in foldStream defState yieldk single stop m
19501893
1951−-- Note that this is not a StreamK -> StreamK because then we cannot handle the
1952−-- empty stream case without making this a partial function.
1953−--
1954−-- See tailNonEmpty as well above.
1955−
1956−-- | Same as:
1957−--
1958−-- >>> tail = fmap (fmap snd) StreamK.uncons
1959−--
19601894{-# INLINE tail #-}
19611895tail :: Applicative m => StreamK m a -> m (Maybe (StreamK m a))
19621896tail =
@@ -1965,14 +1899,9 @@ tail =
19651899 yieldk _ r = pure $ Just r
19661900 in foldStream defState yieldk single stop
19671901
1968−-- Note that this is not a StreamK -> StreamK because then we cannot handle the
1969−-- empty stream case without making this a partial function.
1902+-- | Extract all but the last element of the stream, if any.
19701903--
1971−-- XXX How do we implement unsnoc? Make StreamK a monad and return the
1972−-- remaining stream as a result value in the monad?
1973−
1974−-- | Extract all but the last element of the stream, if any. This will end up
1975−-- evaluating the last element as well to find out that it is last.
1904+-- Note: This will end up buffering the entire stream.
19761905--
19771906-- /Pre-release/
19781907{-# INLINE init #-}
core/src/Streamly/Internal/Data/Time/Units.hs+3
@@ -1,3 +1,6 @@
1+{-# LANGUAGE TypeInType #-}
2+{-# LANGUAGE UnboxedTuples #-}
3+
14-- |
25-- Module : Streamly.Internal.Data.Time.Units
36-- Copyright : (c) 2019 Composewell Technologies
core/src/Streamly/Internal/Data/Unbox.hs+289−165

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unbox/TH.hs+74−74

This diff is too big to show here.

core/src/Streamly/Internal/Data/Unfold.hs+13−15

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/File.hs+13−11

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/Handle.hs+28−21

This diff is too big to show here.

core/src/Streamly/Internal/FileSystem/Path.hsdeleted−515

This diff is too big to show here.

core/src/Streamly/Internal/Data/Binary/Parser.hs → core/src/Streamly/Internal/Serialize/FromBytes.hsrenamed+2−8

This diff is too big to show here.

core/src/Streamly/Internal/Data/Binary/Stream.hs → core/src/Streamly/Internal/Serialize/ToBytes.hsrenamed+2−2

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/Parser.hs−3

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/Stream.hs+11−39

This diff is too big to show here.

core/src/Streamly/Internal/Unicode/String.hs+7−7

This diff is too big to show here.

core/streamly-core.cabal+41−95

This diff is too big to show here.

docs/Developer/optimization-guidelines.md−12

This diff is too big to show here.

docs/Developer/paths.rst+2−25

This diff is too big to show here.

docs/User/HowTo/Compiling.md+3−5

This diff is too big to show here.

docs/User/Project/ApiChangelogs/0.9.0-0.10.0.txtdeleted−424

This diff is too big to show here.

docs/User/Project/CONTRIBUTORS.md−8

This diff is too big to show here.

docs/User/Project/Changelog.md+10−25

This diff is too big to show here.

docs/User/Project/README.md+2−9

This diff is too big to show here.

docs/User/Project/Upgrading-0.8-to-0.9.md → docs/User/Project/Upgrading.mdrenamed+9−87

This diff is too big to show here.

docs/User/Tutorials/quick-overview.md+14−10

This diff is too big to show here.

docs/streamly-docs.cabal+1−1

This diff is too big to show here.

src/Streamly.hsnew file+625

This diff is too big to show here.

src/Streamly/Data/Fold/Prelude.hs+13−31

This diff is too big to show here.

src/Streamly/Data/Fold/Tee.hs+1−1

This diff is too big to show here.

src/Streamly/Data/Stream/Concurrent.hsnew file+194

This diff is too big to show here.

src/Streamly/Data/Stream/Exception.hsnew file+25

This diff is too big to show here.

src/Streamly/Data/Stream/MkType.hs+10−41

This diff is too big to show here.

src/Streamly/Data/Stream/Prelude.hs+15−217

This diff is too big to show here.

src/Streamly/Data/Stream/Time.hsnew file+37

This diff is too big to show here.

src/Streamly/Data/Unicode/Stream.hsnew file+99

This diff is too big to show here.

src/Streamly/Internal/Data/Array/ArrayMacros.hnew file+40

This diff is too big to show here.

src/Streamly/Internal/Data/Channel.hsdeleted−19

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Async.hs+2−5

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent.hs+44−1

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent/Channel.hs+10−29

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Concurrent/Channel/Type.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/Prelude.hsdeleted−19

This diff is too big to show here.

src/Streamly/Internal/Data/Fold/SVar.hs+5−6

This diff is too big to show here.

src/Streamly/Internal/Data/IsMap/HashMap.hs−6

This diff is too big to show here.

src/Streamly/Internal/Data/SVar.hs−1

This diff is too big to show here.

src/Streamly/Internal/Data/SmallArray.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Channel/Dispatcher.hs → src/Streamly/Internal/Data/Stream/Channel/Dispatcher.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Channel/Types.hs → src/Streamly/Internal/Data/Stream/Channel/Types.hsrenamed+37−12

This diff is too big to show here.

src/Streamly/Internal/Data/Channel/Worker.hs → src/Streamly/Internal/Data/Stream/Channel/Worker.hsrenamed+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent.hs+39−50

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel.hs+14−17

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Append.hs+10−10

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Consumer.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Dispatcher.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Interleave.hs+8−8

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Operations.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Concurrent/Channel/Type.hs+3−3

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Lifted.hs → src/Streamly/Internal/Data/Stream/Exception/Lifted.hsrenamed+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream.hs−6

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Common.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Eliminate.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Exception.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Expand.hs−5

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Generate.hs+4−4

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Reduce.hs+4−4

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/IsStream/Top.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/MkType.hs+34−33

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Prelude.hsdeleted−19

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/SVar.hsdeleted−18

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/SVar/Generate.hs−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Serial.hs−5

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Time.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Zip.hs−39

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/Zip/Concurrent.hsnew file+50

This diff is too big to show here.

src/Streamly/Internal/Data/Stream/ZipAsync.hs+1−2

This diff is too big to show here.

src/Streamly/Internal/Data/Unfold/Prelude.hsdeleted−17

This diff is too big to show here.

src/Streamly/Internal/Data/Unfold/SVar.hs−1

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event.hs−3

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Darwin.hs+1−1

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Linux.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/FileSystem/Event/Windows.hs+2−2

This diff is too big to show here.

src/Streamly/Internal/FileSystem/FD.hs+8−7

This diff is too big to show here.

src/Streamly/Internal/Network/Inet/TCP.hs+44−76

This diff is too big to show here.

src/Streamly/Internal/Network/Socket.hs+14−17

This diff is too big to show here.

src/Streamly/Internal/Unicode/Char.hs+3−7

This diff is too big to show here.

src/Streamly/Memory/Array.hsnew file+74

This diff is too big to show here.

src/Streamly/Network/Inet/TCP.hs+30−41

This diff is too big to show here.

src/Streamly/Network/Socket.hs+30−16

This diff is too big to show here.

streamly.cabal+62−82

This diff is too big to show here.

targets/Targets.hs+2−5

This diff is too big to show here.

test/Streamly/Test/Data/Array.hs+3−3

This diff is too big to show here.

test/Streamly/Test/Data/Array/Stream.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Fold.hs+6−7

This diff is too big to show here.

test/Streamly/Test/Data/MutArray.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Parser.hs+6−12

This diff is too big to show here.

test/Streamly/Test/Data/ParserK.hs+2−7

This diff is too big to show here.

test/Streamly/Test/Data/Serialize.hs+26−102

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/CompatV0.hsdeleted−46

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/CompatV1.hsdeleted−48

This diff is too big to show here.

test/Streamly/Test/Data/Serialize/TH.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Data/Stream/Concurrent.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Data/Unbox.hs+50−47

This diff is too big to show here.

test/Streamly/Test/Data/Unbox/TH.hs+10−11

This diff is too big to show here.

test/Streamly/Test/FileSystem/Event/Common.hs+1−1

This diff is too big to show here.

test/Streamly/Test/FileSystem/Handle.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Prelude/Serial.hs+3−7

This diff is too big to show here.

test/Streamly/Test/Serialize/Serializable.hs+2−2

This diff is too big to show here.

test/Streamly/Test/Unicode/Char.hs+1−1

This diff is too big to show here.

test/Streamly/Test/Unicode/Stream.hs+1−1

This diff is too big to show here.

test/lib/Streamly/Test/Prelude/Common.hs+2−9

This diff is too big to show here.

test/streamly-tests.cabal+13−20

This diff is too big to show here.