@@ -0,0 +1,452 @@
1+{-# OPTIONS_GHC -Wno-orphans #-}
2+
3+#include "inline.hs"
4+
5+-- |
6+-- Module : Streamly.Data.Stream
7+-- Copyright : (c) 2017 Composewell Technologies
8+--
9+-- License : BSD3
10+-- Maintainer : streamly@composewell.com
11+-- Stability : released
12+-- Portability : GHC
13+--
14+-- To run examples in this module:
15+--
16+-- >>> import qualified Streamly.Data.Fold as Fold
17+-- >>> import qualified Streamly.Data.Stream as Stream
18+--
19+-- We will add some more imports in the examples as needed.
20+--
21+-- For effectful streams we will use the following IO action that blocks for
22+-- @n@ seconds:
23+--
24+-- >>> import Control.Concurrent (threadDelay)
25+-- >>> :{
26+-- delay n = do
27+-- threadDelay (n * 1000000) -- sleep for n seconds
28+-- putStrLn (show n ++ " sec") -- print "n sec"
29+-- return n -- IO Int
30+-- :}
31+--
32+-- >>> delay 1
33+-- 1 sec
34+-- 1
35+--
36+-- = Overview
37+--
38+-- Streamly is a framework for modular data flow based programming and
39+-- declarative concurrency. Powerful stream fusion framework in streamly
40+-- allows high performance combinatorial programming even when using byte level
41+-- streams. Streamly API is similar to Haskell lists.
42+--
43+-- Streams can be constructed
44+-- like lists, except that they use 'nil' instead of '[]' and 'cons' instead of
45+-- ':'.
46+--
47+-- `cons` constructs a pure stream which is more or less the same as a list:
48+--
49+-- >>> import Streamly.Data.Stream (Stream, cons, consM, nil)
50+-- >>> stream = 1 `cons` 2 `cons` nil :: Stream IO Int
51+-- >>> Stream.fold Fold.toList stream -- IO [Int]
52+-- [1,2]
53+--
54+-- 'consM' constructs a stream from effectful actions:
55+--
56+-- >>> stream = delay 1 `consM` delay 2 `consM` nil
57+-- >>> Stream.fold Fold.toList stream
58+-- 1 sec
59+-- 2 sec
60+-- [1,2]
61+--
62+-- == Console Echo Program
63+--
64+-- In the following example, 'repeatM' generates an infinite stream of 'String'
65+-- by repeatedly performing the 'getLine' IO action. 'mapM' then applies
66+-- 'putStrLn' on each element in the stream converting it to stream of '()'.
67+-- Finally, 'drain' folds the stream to IO discarding the () values, thus
68+-- producing only effects.
69+--
70+-- >>> import Data.Function ((&))
71+--
72+-- @
73+-- > :{
74+-- Stream.repeatM getLine -- Stream IO String
75+-- & Stream.mapM putStrLn -- Stream IO ()
76+-- & Stream.fold Fold.drain -- IO ()
77+-- :}
78+-- @
79+--
80+-- This is a console echo program. It is an example of a declarative loop
81+-- written using streaming combinators. Compare it with an imperative @while@
82+-- loop.
83+--
84+-- Hopefully, this gives you an idea how we can program declaratively by
85+-- representing loops using streams. In this module, you can find all
86+-- "Data.List" like functions and many more powerful combinators to perform
87+-- common programming tasks. Also see "Streamly.Internal.Data.Stream.IsStream"
88+-- module for many more @Pre-release@ combinators. See the
89+-- <https://github.com/composewell/streamly-examples> repository for many more
90+-- real world examples of stream programming.
91+-- == Combining two streams
92+--
93+-- Two streams can be combined to form a single stream in various interesting
94+-- ways. 'serial' (append), 'wSerial' (interleave), 'ahead' (concurrent,
95+-- ordered append), 'async' (lazy concurrent, unordered append) , 'wAsync'
96+-- (lazy concurrent, unordered interleave), 'parallel' (strict concurrent
97+-- merge), 'zipWith', 'zipAsyncWith' (concurrent zip), 'mergeBy',
98+-- 'mergeAsyncBy' (concurrent merge) are some ways of combining two streams.
99+--
100+-- For example, the `parallel` combinator schedules both the streams
101+-- concurrently.
102+--
103+-- >>> stream1 = Stream.mapM id $ Stream.fromList [delay 3, delay 4]
104+-- >>> stream2 = Stream.mapM id $ Stream.fromList [delay 1, delay 2]
105+-- >>> Stream.fold Fold.toList $ stream1 `parallel` stream2
106+-- ...
107+--
108+-- We can chain the operations to combine more than two streams:
109+--
110+-- >>> stream3 = Stream.mapM id $ Stream.fromList [delay 1, delay 2]
111+-- >>> Stream.fold Fold.toList $ stream1 `parallel` stream2 `parallel` stream3
112+-- ...
113+--
114+-- Concurrent generation ('consM') and concurrent merging of streams is the
115+-- fundamental basis of all concurrency in streamly.
116+--
117+-- == Combining many streams
118+--
119+-- The 'concatMapWith' combinator can be used to generalize the two stream
120+-- combining combinators to @n@ streams. For example, we can use
121+-- @concatMapWith parallel@ to read concurrently from all incoming network
122+-- connections and combine the input streams into a single output stream:
123+--
124+-- @
125+-- import qualified Streamly.Network.Inet.TCP as TCP
126+-- import qualified Streamly.Network.Socket as Socket
127+--
128+-- Stream.unfold TCP.acceptOnPort 8090
129+-- & Stream.concatMapWith Stream.parallel (Stream.unfold Socket.read)
130+-- @
131+--
132+-- See the @streamly-examples@ repository for a full working example.
133+--
134+-- == Semigroup Instance
135+--
136+-- Earlier we distinguished stream types based on the execution behavior of
137+-- actions within a stream. Stream types are also distinguished based on how
138+-- actions from different streams are scheduled for execution when two streams
139+-- are combined together.
140+--
141+
142+-- For 'Stream', '<>' has an appending behavior i.e. it executes the actions
143+-- from the second stream after executing actions from the first stream:
144+--
145+-- >>> stream1 = Stream.mapM id $ Stream.fromList [delay 1, delay 2]
146+-- >>> stream2 = Stream.mapM id $ Stream.fromList [delay 3, delay 4]
147+-- >>> Stream.fold Fold.toList $ stream1 <> stream2
148+-- 1 sec
149+-- 2 sec
150+-- 3 sec
151+-- 4 sec
152+-- [1,2,3,4]
153+--
154+-- By default, folds like 'drain' force the stream type to be 'Stream', so
155+-- 'replicateM' in the following code runs serially, and takes 10 seconds:
156+--
157+-- >>> Stream.fold Fold.drain $ Stream.replicateM 10 $ delay 1
158+-- ...
159+--
160+module Streamly.Data.Stream
161+ (
162+ Stream
163+ -- * Construction
164+ -- | Functions ending in the general shape @b -> Stream m a@.
165+ --
166+ -- ** Primitives
167+ -- | Primitives to construct a stream from pure values or monadic actions.
168+ -- All other stream construction and generation combinators described later
169+ -- can be expressed in terms of these primitives. However, the special
170+ -- versions provided in this module can be much more efficient in most
171+ -- cases. Users can create custom combinators using these primitives.
172+ , nil
173+ , cons
174+ , consM
175+
176+ -- ** Unfolding
177+ -- | Generalized way of generating a stream efficiently.
178+ , unfold
179+ , unfoldr
180+ , unfoldrM
181+
182+ -- ** From Values
183+ -- | Generate a monadic stream from a seed value or values.
184+ , fromPure
185+ , fromEffect
186+ , repeat
187+ , replicate
188+
189+ -- Note: Using enumeration functions e.g. 'Prelude.enumFromThen' turns out
190+ -- to be slightly faster than the idioms like @[from, then..]@.
191+ --
192+ -- ** Enumeration
193+ -- | We can use the 'Enum' type class to enumerate a type producing a list
194+ -- and then convert it to a stream:
195+ --
196+ -- @
197+ -- 'fromList' $ 'Prelude.enumFromThen' from then
198+ -- @
199+ --
200+ -- However, this is not particularly efficient.
201+ -- The 'Enumerable' type class provides corresponding functions that
202+ -- generate a stream instead of a list, efficiently.
203+ , Enumerable (..)
204+ , enumerate
205+ , enumerateTo
206+
207+ -- ** Iteration
208+ , iterate
209+ , iterateM
210+
211+ -- ** From Containers
212+ -- | Convert an input structure, container or source into a stream. All of
213+ -- these can be expressed in terms of primitives.
214+ , fromList
215+ , fromFoldable
216+
217+ -- * Elimination
218+ -- | Functions ending in the general shape @Stream m a -> m b@
219+ --
220+ -- ** Running a 'Fold'
221+ -- $runningfolds
222+ , fold
223+
224+ -- ** Deconstruction
225+ -- | Functions ending in the general shape @Stream m a -> m (b, Stream m a)@
226+ , uncons
227+
228+ -- ** General Folds
229+-- | In imperative terms a fold can be considered as a loop over the stream
230+-- that reduces the stream to a single value.
231+-- Left and right folds both use a fold function @f@ and an identity element
232+-- @z@ (@zero@) to deconstruct a recursive data structure and reconstruct a
233+-- new data structure. The new structure may be a recursive construction (a
234+-- container) or a non-recursive single value reduction of the original
235+-- structure.
236+--
237+-- Both right and left folds are mathematical duals of each other, they are
238+-- functionally equivalent. Operationally, a left fold on a left associated
239+-- structure behaves exactly in the same way as a right fold on a right
240+-- associated structure. Similarly, a left fold on a right associated structure
241+-- behaves in the same way as a right fold on a left associated structure.
242+-- However, the behavior of a right fold on a right associated structure is
243+-- operationally different (even though functionally equivalent) than a left
244+-- fold on the same structure.
245+--
246+-- On right associated structures like Haskell @cons@ lists or Streamly
247+-- streams, a lazy right fold is naturally suitable for lazy recursive
248+-- reconstruction of a new structure, while a strict left fold is naturally
249+-- suitable for efficient reduction. In right folds control is in the hand of
250+-- the @puller@ whereas in left folds the control is in the hand of the
251+-- @pusher@.
252+--
253+-- The behavior of right and left folds are described in detail in the
254+-- individual fold's documentation. To illustrate the two folds for right
255+-- associated @cons@ lists:
256+--
257+-- > foldr :: (a -> b -> b) -> b -> [a] -> b
258+-- > foldr f z [] = z
259+-- > foldr f z (x:xs) = x `f` foldr f z xs
260+-- >
261+-- > foldl :: (b -> a -> b) -> b -> [a] -> b
262+-- > foldl f z [] = z
263+-- > foldl f z (x:xs) = foldl f (z `f` x) xs
264+--
265+-- @foldr@ is conceptually equivalent to:
266+--
267+-- > foldr f z [] = z
268+-- > foldr f z [x] = f x z
269+-- > foldr f z xs = foldr f (foldr f z (tail xs)) [head xs]
270+--
271+-- @foldl@ is conceptually equivalent to:
272+--
273+-- > foldl f z [] = z
274+-- > foldl f z [x] = f z x
275+-- > foldl f z xs = foldl f (foldl f z (init xs)) [last xs]
276+--
277+-- Left and right folds are duals of each other.
278+--
279+-- @
280+-- foldr f z xs = foldl (flip f) z (reverse xs)
281+-- foldl f z xs = foldr (flip f) z (reverse xs)
282+-- @
283+--
284+-- More generally:
285+--
286+-- @
287+-- foldr f z xs = foldl g id xs z where g k x = k . f x
288+-- foldl f z xs = foldr g id xs z where g x k = k . flip f x
289+-- @
290+--
291+
292+-- As a general rule, foldr cannot have state and foldl cannot have control.
293+
294+-- NOTE: Folds are inherently serial as each step needs to use the result of
295+-- the previous step. However, it is possible to fold parts of the stream in
296+-- parallel and then combine the results using a monoid.
297+
298+ -- *** Right Folds
299+ -- $rightfolds
300+ , foldrM
301+ , foldr
302+
303+ -- ** Multi-Stream folds
304+ , eqBy
305+ , cmpBy
306+ , isPrefixOf
307+ , isSubsequenceOf
308+
309+ -- trimming sequences
310+ , stripPrefix
311+
312+ -- * Transformation
313+ -- ** Mapping
314+ -- | In imperative terms a map operation can be considered as a loop over
315+ -- the stream that transforms the stream into another stream by performing
316+ -- an operation on each element of the stream.
317+ --
318+ -- 'map' is the least powerful transformation operation with strictest
319+ -- guarantees. A map, (1) is a stateless loop which means that no state is
320+ -- allowed to be carried from one iteration to another, therefore,
321+ -- operations on different elements are guaranteed to not affect each
322+ -- other, (2) is a strictly one-to-one transformation of stream elements
323+ -- which means it guarantees that no elements can be added or removed from
324+ -- the stream, it can merely transform them.
325+ , sequence
326+ , mapM
327+
328+ -- ** Mapping Side Effects
329+ , trace
330+ , tap
331+ , delay
332+
333+ -- ** Scanning By 'Fold'
334+ , scan
335+ , postscan
336+
337+ -- ** Filtering
338+ -- | Remove some elements from the stream based on a predicate. In
339+ -- imperative terms a filter over a stream corresponds to a loop with a
340+ -- @continue@ clause for the cases when the predicate fails.
341+ , deleteBy
342+ , filter
343+ , filterM
344+ , uniq
345+
346+ -- ** Trimming
347+ -- | Take or remove elements from one or both ends of a stream.
348+ , take
349+ , takeWhile
350+ , takeWhileM
351+ , drop
352+ , dropWhile
353+ , dropWhileM
354+
355+ -- ** Inserting Elements
356+ -- | Inserting elements is a special case of interleaving/merging streams.
357+ , insertBy
358+ , intersperseM
359+ , intersperse
360+
361+ -- ** Reordering Elements
362+ , reverse
363+
364+ -- ** Indexing
365+ -- | Indexing can be considered as a special type of zipping where we zip a
366+ -- stream with an index stream.
367+ , indexed
368+ , indexedR
369+
370+ -- ** Searching
371+ -- | Finding the presence or location of an element, a sequence of elements
372+ -- or another stream within a stream.
373+
374+ -- ** Searching Elements
375+ , findIndices
376+ , elemIndices
377+
378+ -- ** Maybe Streams
379+ , mapMaybe
380+ , mapMaybeM
381+
382+ -- ** PairWise combinators
383+ -- | These functions have O(n^2) append performance when used linearly e.g.
384+ -- using 'concatMapWith'. However, they can be combined pair wise using
385+ -- 'Streamly.Internal.Data.Stream.IsStream.Expand.concatPairsWith' to give
386+ -- O(n * log n) complexity.
387+ , mergeBy
388+ , mergeByM
389+ , zipWith
390+ , zipWithM
391+
392+ -- ** Nested Unfolds
393+ , unfoldMany
394+ , intercalate
395+ , intercalateSuffix
396+
397+ -- ** Nested Streams
398+ -- | Stream operations like map and filter represent loop processing in
399+ -- imperative programming terms. Similarly, the imperative concept of
400+ -- nested loops are represented by streams of streams. The 'concatMap'
401+ -- operation represents nested looping.
402+ -- A 'concatMap' operation loops over the input stream and then for each
403+ -- element of the input stream generates another stream and then loops over
404+ -- that inner stream as well producing effects and generating a single
405+ -- output stream.
406+ -- The 'Monad' instances of different stream types provide a more
407+ -- convenient way of writing nested loops. Note that the monad bind
408+ -- operation is just @flip concatMap@.
409+ --
410+ -- One dimension loops are just a special case of nested loops. For
411+ -- example, 'concatMap' can degenerate to a simple map operation:
412+ --
413+ -- > map f m = S.concatMap (\x -> S.fromPure (f x)) m
414+ --
415+ -- Similarly, 'concatMap' can perform filtering by mapping an element to a
416+ -- 'nil' stream:
417+ --
418+ -- > filter p m = S.concatMap (\x -> if p x then S.fromPure x else S.nil) m
419+ --
420+ , concatMapWith
421+ , concatMap
422+ , concatMapM
423+
424+ -- * Nested Folds
425+ , foldMany
426+
427+ -- * Exceptions
428+ -- | Most of these combinators inhibit stream fusion, therefore, when
429+ -- possible, they should be called in an outer loop to mitigate the cost.
430+ -- For example, instead of calling them on a stream of chars call them on a
431+ -- stream of arrays before flattening it to a stream of chars.
432+ , before
433+ , after
434+ , bracket
435+ , onException
436+ , finally
437+ , handle
438+
439+ -- * Lifting Inner Monad
440+ , liftInner
441+ , runReaderT
442+ , runStateT
443+ )
444+where
445+
446+import Streamly.Internal.Data.Stream
447+import Prelude
448+ hiding (filter, drop, dropWhile, take, takeWhile, zipWith, foldr,
449+ foldl, map, mapM, mapM_, sequence, all, any, sum, product, elem,
450+ notElem, maximum, minimum, head, last, tail, length, null,
451+ reverse, iterate, init, and, or, lookup, foldr1, (!!),
452+ scanl, scanl1, repeat, replicate, concatMap, span)