2{-# LANGUAGE RankNTypes #-}
3{-# LANGUAGE FlexibleContexts #-}
5#if __GLASGOW_HASKELL__ >= 800
6{-# OPTIONS_GHC -Wno-orphans #-}
9#include "Streams/inline.hs"
12-- Module : Streamly.Prelude
13-- Copyright : (c) 2017 Harendra Kumar
16-- Maintainer : harendra.kumar@gmail.com
17-- Stability : experimental
20-- This module is designed to be imported qualified:
23-- import qualified Streamly.Prelude as S
26-- Functions with the suffix @M@ are general functions that work on monadic
27-- arguments. The corresponding functions without the suffix @M@ work on pure
28-- arguments and can in general be derived from their monadic versions but are
29-- provided for convenience and for consistency with other pure APIs in the
32-- In many cases, short definitions of the combinators are provided in the
33-- documentation for illustration. The actual implementation may differ for
34-- performance reasons.
36-- Functions having a 'MonadAsync' constraint work concurrently when used with
37-- appropriate stream type combinator. Please be careful to not use 'parallely'
38-- with infinite streams.
40-- Deconstruction and folds accept a 'SerialT' type instead of a polymorphic
41-- type to ensure that streams always have a concrete monomorphic type by
42-- default, reducing type errors. In case you want to use any other type of
43-- stream you can use one of the type combinators provided in the "Streamly"
44-- module to convert the stream type.
46module Streamly.Prelude
50 -- | Primitives to construct a stream from pure values or monadic actions.
51 -- All other stream construction and generation combinators described later
52 -- can be expressed in terms of these primitives. However, the special
53 -- versions provided in this module can be much more efficient in most
54 -- cases. Users can create custom combinators using these primitives.
64 -- | Generate a monadic stream from a seed value or values.
72 -- Note: Using enumeration functions e.g. 'Prelude.enumFromThen' turns out
73 -- to be slightly faster than the idioms like @[from, then..]@.
76 -- | We can use the 'Enum' type class to enumerate a type producing a list
77 -- and then convert it to a stream:
80 -- 'fromList' $ 'Prelude.enumFromThen' from then
83 -- However, this is not particularly efficient.
84 -- The 'Enumerable' type class provides corresponding functions that
85 -- generate a stream instead of a list, efficiently.
92 -- | Generate a monadic stream from a seed value and a generator function.
100 -- ** From Containers
101 -- | Convert an input structure, container or source into a stream. All of
102 -- these can be expressed in terms of primitives.
111 -- | It is easy to express all the folds in terms of the 'uncons' primitive,
112 -- however the specific implementations provided later are generally more
120-- | In imperative terms a fold can be considered as a loop over the stream
121-- that reduces the stream to a single value.
122-- Left and right folds use a fold function @f@ and an identity element @z@
123-- (@zero@) to recursively deconstruct a structure and then combine and reduce
124-- the values or transform and reconstruct a new container.
126-- In general, a right fold is suitable for transforming and reconstructing a
127-- right associated structure (e.g. cons lists and streamly streams) and a left
128-- fold is suitable for reducing a right associated structure. The behavior of
129-- right and left folds are described in detail in the individual fold's
130-- documentation. To illustrate the two folds for cons lists:
132-- > foldr :: (a -> b -> b) -> b -> [a] -> b
134-- > foldr f z (x:xs) = x `f` foldr f z xs
136-- > foldl :: (b -> a -> b) -> b -> [a] -> b
138-- > foldl f z (x:xs) = foldl f (z `f` x) xs
140-- @foldr@ is conceptually equivalent to:
143-- > foldr f z [x] = f x z
144-- > foldr f z xs = foldr f (foldr f z (tail xs)) [head xs]
146-- @foldl@ is conceptually equivalent to:
149-- > foldl f z [x] = f z x
150-- > foldl f z xs = foldl f (foldl f z (init xs)) [last xs]
152-- Left and right folds are duals of each other.
155-- foldr f z xs = foldl (flip f) z (reverse xs)
156-- foldl f z xs = foldr (flip f) z (reverse xs)
162-- foldr f z xs = foldl g id xs z where g k x = k . f x
163-- foldl f z xs = foldr g id xs z where g x k = k . flip f x
167-- As a general rule, foldr cannot have state and foldl cannot have control.
169-- NOTE: Folds are inherently serial as each step needs to use the result of
170-- the previous step. However, it is possible to fold parts of the stream in
171-- parallel and then combine the results using a monoid.
186 -- ** Composable Left Folds
192 -- | Folds that are guaranteed to evaluate the whole stream.
194 -- -- ** To Summary (Full Folds)
195 -- -- | Folds that summarize the stream to a single value.
203 -- -- ** To Summary (Maybe) (Full Folds)
204 -- -- | Folds that summarize a non-empty stream to a 'Just' value and return
205 -- 'Nothing' for an empty stream.
211 -- , toListRev -- experimental
215 -- | Folds that generate a lazy structure. Note that the generated
216 -- structure may not be lazy if the underlying monad is strict.
218 -- -- ** To Containers (Full Folds)
219 -- -- | Convert or divert a stream into an output structure, container or
224 -- | Folds that may terminate before evaluating the whole stream. These
225 -- folds strictly evaluate the stream until the result is determined.
227 -- -- ** To Elements (Partial Folds)
231 -- -- | Folds that extract selected elements of a stream or their properties.
240 -- -- ** To Boolean (Partial Folds)
241 -- -- | Folds that summarize the stream to a boolean value.
250 -- ** Multi-Stream folds
258 -- trimming sequences
268 -- | In imperative terms a map operation can be considered as a loop over
269 -- the stream that transforms the stream into another stream by performing
270 -- an operation on each element of the stream.
272 -- 'map' is the least powerful transformation operation with strictest
273 -- guarantees. A map, (1) is a stateless loop which means that no state is
274 -- allowed to be carried from one iteration to another, therefore,
275 -- operations on different elements are guaranteed to not affect each
276 -- other, (2) is a strictly one-to-one transformation of stream elements
277 -- which means it guarantees that no elements can be added or removed from
278 -- the stream, it can merely transform them.
290 -- | A scan is more powerful than map. While a 'map' is a stateless loop, a
291 -- @scan@ is a stateful loop which means that a state can be shared across
292 -- all the loop iterations, therefore, future iterations can be impacted by
293 -- the state changes made by the past iterations. A scan yields the state
294 -- of the loop after each iteration. Like a map, a @postscan@ or @prescan@
295 -- does not add or remove elements in the stream, it just transforms them.
296 -- However, a @scan@ adds one extra element to the stream.
298 -- A left associative scan, also known as a prefix sum, can be thought of
299 -- as a stream transformation consisting of left folds of all prefixes of a
300 -- stream. Another way of thinking about it is that it streams all the
301 -- intermediate values of the accumulator while applying a left fold on the
302 -- input stream. A right associative scan, on the other hand, can be
303 -- thought of as a stream consisting of right folds of all the suffixes of
306 -- The following equations hold for lists:
308 -- > scanl f z xs == map (foldl f z) $ inits xs
309 -- > scanr f z xs == map (foldr f z) $ tails xs
312 -- > scanl (+) 0 [1,2,3,4]
317 -- 0 + 1 + 2 + 3 + 4 = 10
319 -- > scanr (+) 0 [1,2,3,4]
320 -- 1 + 2 + 3 + 4 + 0 = 10
327 -- Left and right scans are duals:
329 -- > scanr f z xs == reverse $ scanl (flip f) z (reverse xs)
330 -- > scanl f z xs == reverse $ scanr (flip f) z (reverse xs)
332 -- A scan is a stateful map i.e. a combination of map and fold:
334 -- > map f xs = tail $ scanl (\_ x -> f x) z xs
335 -- > map f xs = reverse $ head $ scanr (\_ x -> f x) z xs
337 -- > foldl f z xs = last $ scanl f z xs
338 -- > foldr f z xs = head $ scanr f z xs
350 -- ** Scan Using Fold
365 -- | Remove some elements from the stream based on a predicate. In
366 -- imperative terms a filter over a stream corresponds to a loop with a
367 -- @continue@ clause for the cases when the predicate fails.
372 -- ** Mapping Filters
373 -- | Mapping along with filtering
378 -- ** Deleting Elements
379 -- | Deleting elements is a special case of de-interleaving streams.
382 -- , uniqBy -- by predicate e.g. to remove duplicate "/" in a path
383 -- , uniqOn -- to remove duplicate sequences
384 -- , pruneBy -- dropAround + uniqBy - like words
386 -- ** Inserting Elements
387 -- | Inserting elements is a special case of interleaving/merging streams.
393 -- , intersperseBySpan
394 -- , intersperseByIndices -- using an index function/stream
395 -- , intersperseByTime
396 -- , intersperseByEvent
398 -- -- * Inserting Streams in Streams
403 -- | Indexing can be considered as a special type of zipping where we zip a
404 -- stream with an index stream.
408 -- , timestampedR -- timer
410 -- ** Reordering Elements
415 -- | Take or remove elements from one or both ends of a stream.
431 -- , splitAt -- spanN
432 -- , splitIn -- sessionN
435 -- , span -- spanWhile
436 -- , break -- breakBefore
444 -- breakOnSeq/breakOnArray -- on a fixed sequence
445 -- breakOnStream -- on a stream
448 -- | Streams can be sliced into segments in space or in time. We use the
449 -- term @chunk@ to refer to a spatial length of the stream (spatial window)
450 -- and the term @session@ to refer to a length in time (time window).
452 -- In imperative terms, grouped folding can be considered as a nested loop
453 -- where we loop over the stream to group elements and then loop over
454 -- individual groups to fold them to a single value that is yielded in the
463 -- | Finding the presence or location of an element, a sequence of elements
464 -- or another stream within a stream.
466 -- -- ** Searching Elements
470 -- -- *** Searching Sequences
471 -- , seqIndices -- search a sequence in the stream
473 -- -- *** Searching Multiple Sequences
474 -- , seqIndices -- search a sequence in the stream
476 -- -- ** Searching Streams
477 -- -- | Finding a stream within another stream.
480 -- | In general we can express splitting in terms of parser combinators.
481 -- These are some common use functions for convenience and efficiency.
482 -- While parsers can fail these functions are designed to transform a
483 -- stream without failure with a predefined behavior for all cases.
485 -- In general, there are three possible ways of combining stream segments
486 -- with a separator. The separator could be prefixed to each segment,
487 -- suffixed to each segment, or it could be infixed between segments.
488 -- 'intersperse' and 'intercalate' operations are examples of infixed
489 -- combining whereas 'unlines' is an example of suffixed combining. When we
490 -- split a stream with separators we can split in three different ways,
491 -- each being an opposite of the three ways of combining.
493 -- Splitting may keep the separator or drop it. Depending on how we split,
494 -- the separator may be kept attached to the stream segments in prefix or
495 -- suffix position or as a separate element in infix position. Combinators
496 -- like 'splitOn' that use @On@ in their names drop the separator and
497 -- combinators that use 'With' in their names keep the separator. When a
498 -- segment is missing it is considered as empty, therefore, we never
499 -- encounter an error in parsing.
501 -- -- ** Splitting By Elements
509 , wordsBy -- strip, compact and split
511 -- -- *** Splitting By Sequences
513 -- , splitOnSuffixSeq
514 -- , splitOnPrefixSeq
516 -- Keeping the delimiters
518 -- , splitBySeqSuffix
519 -- , splitBySeqPrefix
522 -- Splitting By Multiple Sequences
524 -- , splitOnAnySuffixSeq
525 -- , splitOnAnyPrefixSeq
527 -- -- ** Splitting By Streams
528 -- -- | Splitting a stream using another stream as separator.
531 -- | Splitting a stream by combining multiple contiguous elements into
532 -- groups using some criterion.
538 -- * Windowed Classification
539 -- | Split the stream into windows or chunks in space or time. Each window
540 -- can be associated with a key, all events associated with a particular
541 -- key in the window can be folded to a single result. The stream is split
542 -- into windows of specified size, the window can be terminated early if
543 -- the closing flag is specified in the input stream.
545 -- The term "chunk" is used for a space window and the term "session" is
546 -- used for a time window.
548 -- ** Tumbling Windows
549 -- | A new window starts after the previous window is finished.
550 -- , classifyChunksOf
551 -- , classifySessionsOf
553 -- ** Keep Alive Windows
554 -- | The window size is extended if an event arrives within the specified
555 -- window size. This can represent sessions with idle or inactive timeout.
556 -- , classifyKeepAliveChunks
557 -- , classifyKeepAliveSessions
560 -- ** Sliding Windows
561 -- | A new window starts after the specified slide from the previous
562 -- window. Therefore windows can overlap.
563 , classifySlidingChunks
564 , classifySlidingSessions
566 -- ** Sliding Window Buffers
567 -- , slidingChunkBuffer
568 -- , slidingSessionBuffer
571 -- * Combining Streams
572 -- | New streams can be constructed by appending, merging or zipping
576 -- | Streams form a 'Semigroup' and a 'Monoid' under the append
577 -- operation. Appending can be considered as a generalization of the `cons`
578 -- operation to consing a stream to a stream.
582 -- -------Stream m a------|-------Stream m a------|=>----Stream m a---
587 -- >> S.toList $ S.fromList [1,2] \<> S.fromList [3,4]
589 -- >> S.toList $ fold $ [S.fromList [1,2], S.fromList [3,4]]
594 -- | Streams form a commutative semigroup under the merge
595 -- operation. Merging can be considered as a generalization of inserting an
596 -- element in a stream to interleaving a stream with another stream.
600 -- -------Stream m a------|
601 -- |=>----Stream m a---
602 -- -------Stream m a------|
616 -- -------Stream m a------|
617 -- |=>----Stream m c---
618 -- -------Stream m b------|
626 -- ** Folding Containers of Streams
627 -- | These are variants of standard 'Foldable' fold functions that use a
628 -- polymorphic stream sum operation (e.g. 'async' or 'wSerial') to fold a
629 -- finite container of streams. Note that these are just special cases of
630 -- the more general 'concatMapWith' operation.
636 -- ** Folding Streams of Streams
637 -- | Stream operations like map and filter represent loop processing in
638 -- imperative programming terms. Similarly, the imperative concept of
639 -- nested loops are represented by streams of streams. The 'concatMap'
640 -- operation represents nested looping.
641 -- A 'concatMap' operation loops over the input stream and then for each
642 -- element of the input stream generates another stream and then loops over
643 -- that inner stream as well producing effects and generating a single
645 -- The 'Monad' instances of different stream types provide a more
646 -- convenient way of writing nested loops. Note that the monad bind
647 -- operation is just @flip concatMap@.
649 -- One dimension loops are just a special case of nested loops. For
650 -- example, 'concatMap' can degenerate to a simple map operation:
652 -- > map f m = S.concatMap (\x -> S.yield (f x)) m
654 -- Similarly, 'concatMap' can perform filtering by mapping an element to a
657 -- > filter p m = S.concatMap (\x -> if p x then S.yield x else S.nil) m
660 -- XXX add stateful concatMapWith
690 hiding (filter, drop, dropWhile, take, takeWhile, zipWith, foldr,
691 foldl, map, mapM, mapM_, sequence, all, any, sum, product, elem,
692 notElem, maximum, minimum, head, last, tail, length, null,
693 reverse, iterate, init, and, or, lookup, foldr1, (!!),
694 scanl, scanl1, repeat, replicate, concatMap, span)
696import Streamly.Prelude.Internal
700-- Let's take a closer look at the @foldr@ definition for lists, as given
704-- foldr f z (x:xs) = x \`f` foldr f z xs
707-- @foldr@ invokes the fold step function @f@ as @f x (foldr f z xs)@. At each
708-- invocation of @f@, @foldr@ gives us the next element in the input container
709-- @x@ and a recursive expression @foldr f z xs@ representing the yet unbuilt
710-- (lazy thunk) part of the output.
712-- When @f x xs@ is lazy in @xs@ it can consume the input one element at a time
713-- in FIFO order to build a lazy output expression. For example,
715-- > f x remaining = show x : remaining
717-- @take 2 $ foldr f [] (1:2:undefined)@ would consume the input lazily on
718-- demand, consuming only first two elements and resulting in ["1", "2"]. @f@
719-- can terminate recursion by not evaluating the @remaining@ part:
721-- > f 2 remaining = show 2 : []
722-- > f x remaining = show x : remaining
724-- @f@ would terminate recursion whenever it sees element @2@ in the input.
725-- Therefore, @take 2 $ foldr f [] (1:2:undefined)@ would work without any
728-- On the other hand, if @f a b@ is strict in @b@ it would end up consuming the
729-- whole input right away and expanding the recursive expression @b@ (i.e.
730-- @foldr f z xs@) fully before it yields an output expression, resulting in
731-- the following /right associated expression/:
734-- foldr f z xs == x1 \`f` (x2 \`f` ...(xn \`f` z))
739-- > f x remaining = x + remaining
741-- With this definition, @foldr f 0 [1..1000]@, would recurse completely until
742-- it reaches the terminating case @... `f` (1000 `f` 0)@, and then
743-- start reducing the whole expression from right to left, therefore, consuming
744-- the input elements in LIFO order. Thus, such an evaluation would require
745-- memory proportional to the size of input. Try out @foldr (+) 0 (map (\\x ->
746-- trace (show x) x) [1..10])@.
748-- Notice the order of the arguments to the step function @f a b@. It follows
749-- the order of @a@ and @b@ in the right associative recursive expression
750-- generated by expanding @a \`f` b@.
752-- A right fold is a pull fold, the step function is the puller, it can pull
753-- more data from the input container by using its second argument in the
754-- output expression or terminate pulling by not using it. As a corollary:
756-- 1. a step function which is lazy in its second argument (usually functions
757-- or constructors that build a lazy structure e.g. @(:)@) can pull lazily on
759-- 2. a step function strict in its second argument (usually reducers e.g.
760-- (+)) would end up pulling all of its input and buffer it in memory before
761-- potentially reducing it.
763-- A right fold is suitable for lazy reconstructions e.g. transformation,
764-- mapping, filtering of /right associated input structures/ (e.g. cons lists).
765-- Whereas a left fold is suitable for reductions (e.g. summing a stream of
766-- numbers) of right associated structures. Note that these roles will reverse
767-- for left associated structures (e.g. snoc lists). Most of our observations
768-- here assume right associated structures, lists being the canonical example.
770-- 1. A lazy FIFO style pull using a right fold allows pulling a potentially
771-- /infinite/ input stream lazily, perform transformations on it, and
772-- reconstruct a new structure without having to buffer the whole structure. In
773-- contrast, a left fold would buffer the entire structure before the
774-- reconstructed structure can be consumed.
775-- 2. Even if buffering the entire input structure is ok, we need to keep in
776-- mind that a right fold reconstructs structures in a FIFO style, whereas a
777-- left fold reconstructs in a LIFO style, thereby reversing the order of
779-- 3. A right fold has termination control and therefore can terminate early
780-- without going throught the entire input, a left fold cannot terminate
781-- without consuming all of its input. For example, a right fold
782-- implementation of 'or' can terminate as soon as it finds the first 'True'
783-- element, whereas a left fold would necessarily go through the entire input
784-- irrespective of that.
785-- 4. Reduction (e.g. using (+) on a stream of numbers) using a right fold
786-- occurs in a LIFO style, which means that the entire input gets buffered
787-- before reduction starts. Whereas with a strict left fold reductions occur
788-- incrementally in FIFO style. Therefore, a strict left fold is more suitable
794-- Note that the observations below about the behavior of a left fold assume
795-- that we are working on a right associated structure like cons lists and
796-- streamly streams. If we are working on a left associated structure (e.g.
797-- snoc lists) the roles of right and left folds would reverse.
799-- Let's take a closer look at the @foldl@ definition for lists given above:
802-- foldl f z (x:xs) = foldl f (z \`f` x) xs
805-- @foldl@ calls itself recursively, in each call it invokes @f@ as @f z x@
806-- providing it with the result accumulated till now @z@ (the state) and the
807-- next element from the input container. First call to @f@ is supplied with
808-- the initial value of the accumulator @z@ and each subsequent call uses the
809-- output of the previous call to @f z x@.
811-- >> foldl' (+) 0 [1,2,3]
814-- The recursive call at the head of the output expression is bound to be
815-- evaluated until recursion terminates, therefore, a left fold always
816-- /consumes the whole input container/. The following would result in an
817-- error, even though the fold is not using the values at all:
819-- >> foldl' (\_ _ -> 0) 0 (1:undefined)
820-- > *** Exception: Prelude.undefined
822-- As @foldl@ recurses, it builds the left associated expression shown below.
823-- Notice, the order of the arguments to the step function @f b a@. It follows
824-- the left associative recursive expression generated by expanding @b \`f` a@.
827-- foldl f z xs == (((z \`f` x1) \`f` x2) ...) \`f` xn
831-- The strict left fold @foldl'@ forces the reduction of its argument @z \`f`
832-- x@ before using it, therefore it never builds the whole expression in
833-- memory. Thus, @z \`f` x1@ would get reduced to @z1@ and then @z1 \`f` x2@
834-- would get reduced to @z2@ and so on, incrementally reducing the expression
835-- from left to right as it recurses, consuming the input in FIFO order. Try
836-- out @foldl' (+) 0 (map (\\x -> trace (show x) x) [1..10])@ to see how it
837-- works. For example:
840-- > S.foldl' (+) 0 $ S.fromList [1,2,3,4]
851-- However, @foldl'@ evaluates the accumulator only to WHNF. It may further
852-- help if the step function uses a strict data structure as accumulator to
853-- improve performance and to keep the expression fully reduced at all times
856-- A left fold can also build a new structure instead of reducing one if a
857-- constructor is used as a fold step. However, it may not be very useful
858-- because it will consume the whole input and construct the new structure in
859-- memory before we can consume it. Thus the whole structure gets buffered in
860-- memory. When the list constructor is used it would build a new list in
861-- reverse (LIFO) order:
864-- > S.foldl' (flip (:)) [] $ S.fromList [1,2,3,4]
868-- A left fold is a push fold. The producer pushes its contents to the step
869-- function of the fold. The step function therefore has no control to stop the
870-- input, it can only discard it if it does not need it. We can also consider a
871-- left fold as a state machine where the state is store in the accumulator,
872-- the state can be modified based on new inputs that are pushed to the fold.
874-- In general, a strict left fold is a reducing fold, whereas a right fold is a
875-- constructing fold. A strict left fold reduces in a FIFO order whereas it
876-- constructs in a LIFO order, and vice-versa for the right fold. See the
877-- documentation of 'foldrM' for a discussion on where a left or right fold is
880-- To perform a left fold lazily without having to consume all the input one
881-- can use @scanl@ to stream the intermediate results of the fold and consume
882-- the resulting stream lazily.
887-- "Streamly.Fold" module defines composable left folds which can be combined
888-- together in many interesting ways. Those folds can be run using 'runFold'.
889-- The following two ways of folding are equivalent in functionality and
892-- >>> S.runFold FL.sum (S.enumerateFromTo 1 100)
894-- >>> S.sum (S.enumerateFromTo 1 100)
897-- However, left folds cannot terminate early even if it does not need to
898-- consume more input to determine the result. Therefore, the performance is
899-- equivalent only for full folds like 'sum' and 'length'. For partial folds
900-- like 'head' or 'any' the the folds defined in this module may be much more
901-- efficient because they are implemented as right folds that terminate as soon
902-- as we get the result. Note that when a full fold is composed with a partial
903-- fold in parallel the performance is not impacted as we anyway have to
904-- consume the whole stream due to the full fold.
906-- >>> S.head (1 `S.cons` undefined)
908-- >>> S.runFold FL.head (1 `S.cons` undefined)
909-- *** Exception: Prelude.undefined
911-- However, we can wrap the fold in a scan to convert it into a lazy stream of
912-- fold steps. We can then terminate the stream whenever we want. For example,
914-- >>> S.toList $ S.take 1 $ S.runScan FL.head (1 `S.cons` undefined)
917-- The following example extracts the input stream up to a point where the
918-- running average of elements is no more than 10:
922-- $ S.map (fromJust . fst)
923-- $ S.takeWhile (\\(_,x) -> x <= 10)
924-- $ S.runPostscan ((,) \<$> FL.last \<*> avg) (S.enumerateFromTo 1.0 100.0)
927-- [1.0,2.0,3.0,4.0,5.0,6.0,7.0,8.0,9.0,10.0,11.0,12.0,13.0,14.0,15.0,16.0,17.0,18.0,19.0]
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list