3-- Copyright : (c) 2017 Composewell Technologies
6-- Maintainer : streamly@composewell.com
7-- Stability : experimental
10-- Streamly is a general purpose programming framework using cocnurrent data
11-- flow programming paradigm. It can be considered as a generalization of
12-- Haskell lists to monadic streaming with concurrent composition capability.
13-- The serial stream type in streamly @SerialT m a@ is like the list type @[a]@
14-- parameterized by the monad @m@. For example, @SerialT IO a@ is a moral
15-- equivalent of @[a]@ in the IO monad. Streams are constructed very much like
16-- lists, except that they use 'nil' and 'cons' instead of '[]' and ':'.
20-- > import "Streamly.Prelude" (cons, consM)
21-- > import qualified "Streamly.Prelude" as S
23-- > S.toList $ 1 \`cons` 2 \`cons` 3 \`cons` nil
27-- Unlike lists, streams can be constructed from monadic effects:
30-- > S.'toList' $ 'getLine' \`consM` 'getLine' \`consM` S.'nil'
36-- Streams are processed just like lists, with list like combinators, except
37-- that they are monadic and work in a streaming fashion. Here is a simple
38-- console echo program example:
41-- > S.drain $ S.repeatM getLine & S.mapM putStrLn
44-- @SerialT Identity a@ is a moral equivalent of pure lists. Streamly utilizes
45-- fusion for high performance, therefore, we can represent and process strings
46-- as streams of 'Char', encode and decode the streams to/from UTF8 and
47-- serialize them to @Array Word8@ obviating the need for special purpose
48-- libraries like @bytestring@ and @text@.
51-- please see the "Streamly.Tutorial" module and the examples directory in this
54{-# LANGUAGE NoMonomorphismRestriction #-}
55{-# OPTIONS_GHC -Wno-orphans #-}
59module Streamly {-# DEPRECATED "Please use \"Streamly.Prelude\" instead." #-}
61 -- -- * Concepts Overview
77 -- * Stream transformers
78 -- | A stream represents a sequence of pure or effectful actions. The
79 -- `cons` and `consM` operations and the corresponding operators '.:' and
80 -- '|:' can be used to join pure values or effectful actions in a sequence.
81 -- The effects in the stream can be executed in many different ways
82 -- depending on the type of stream. In other words, the behavior of 'consM'
83 -- depends on the type of the stream.
85 -- There are three high level categories of streams, /spatially ordered/
86 -- streams, /speculative/ streams and /time ordered/ streams. Spatially
87 -- ordered streams, 'SerialT' and 'WSerialT', execute the effects in serial
88 -- order i.e. one at a time and present the outputs of those effects to the
89 -- consumer in the same order. Speculative streams, 'AheadT', may execute
90 -- many effects concurrently but present the outputs to the consumer in the
91 -- specified spatial order. Time ordered streams, 'AsyncT', 'WAsyncT' and
92 -- 'ParallelT', may execute many effects concurrently and present the
93 -- outputs of those effects to the consumer in time order i.e. as soon as
94 -- the output is generated.
96 -- We described above how the effects in a sequence are executed for
97 -- different types of streams. The behvavior of the 'Semigroup' and 'Monad'
98 -- instances follow the behavior of 'consM'. Stream generation operations
99 -- like 'repeatM' also execute the effects differently for different
100 -- streams, providing a concurrent generation capability when used with
101 -- stream types that execute effects concurrently. Similarly, effectful
102 -- transformation operations like 'mapM' also execute the transforming
103 -- effects differently for different types of streams.
110 -- ** Speculative Streams
114 -- ** Asynchronous Streams
120 -- ** Zipping Streams
125 -- * Parallel Function Application
142 -- * Concurrency Control
155 -- * Stream Type Adapters
179 -- ** Folding Containers of Streams
180 -- | These are variants of standard 'Foldable' fold functions that use a
181 -- polymorphic stream sum operation (e.g. 'async' or 'wSerial') to fold a
182 -- finite container of streams. Note that these are just special cases of
183 -- the more general 'concatMapWith' operation.
196import Data.Semigroup (Semigroup(..))
197import Streamly.Internal.Control.Concurrent (MonadAsync)
198import Streamly.Internal.Data.SVar (Rate(..))
199import Streamly.Internal.Data.Stream.Ahead
200import Streamly.Internal.Data.Stream.Async
201import Streamly.Internal.Data.Stream.IsStream.Combinators
202import Streamly.Internal.Data.Stream.IsStream.Expand
203import Streamly.Internal.Data.Stream.IsStream.Type
205import qualified Streamly.Prelude as P
206import qualified Streamly.Internal.Data.Stream.IsStream as IP
207import qualified Streamly.Internal.Data.Stream.IsStream.Transform as Transform
209-- XXX provide good succinct examples of pipelining, merging, splitting etc.
214-- A program is expressed as a network of streams and folds. A stream is a
215-- source or generator of data elements and a fold is a consumer of data elements that
216-- reduces multiple input elements to a single value.
218-- In the following example, a 'Word8' stream is generated by using
219-- 'Streamly.FileSystem.Handle.read' on a file handle, then the
220-- 'Streamly.Prelude.splitBySuffix' transformation splits the stream on
221-- newlines (ascii value 10); it uses the 'Streamly.Data.Fold.drain' fold to reduce
222-- the resulting lines to unit values (@()@), 'Streamly.Prelude.length' fold
223-- then counts the unit elements in the resulting stream which gives us the
224-- number of lines in the file:
226-- > S.length $ S.splitOnSuffix FL.drain 10 $ FH.read fh
228-- The following example folds the lines to arrays of 'Word8' using the
229-- 'Streamly.Data.Array.Foreign.writeF' fold and then wraps the lines in square
230-- brackets before writing them to standard output using
231-- 'Streamly.FileSystem.Handle.write':
233-- > wrapLine ln = S.fromList "[" <> A.read ln <> S.fromList "]\n"
234-- > readLines = S.splitOnSuffix A.writeF 10
235-- > FH.write stdout $ S.concatMap wrapLine $ readLines fh1
237-- One stream can be appended after another:
239-- > FH.write stdout $ S.concatMap wrapLine $ readLines fh1 <> readLines fh2
241-- The following example reads two files concurrently, merges the lines from
242-- the two streams and writes the resulting stream to another file:
244-- > FH.write stdout $ S.concatMap wrapLine $ readLines fh1 `parallel` readLines fh2
246-- There are many ways to generate, merge, zip, transform and fold data
247-- streams. Many transformations can be chained in a stream pipeline. See
248-- "Streamly.Prelude" module for combinators to manipulate streams.
252-- The way stream types in this module like 'SerialT' represent data sources,
253-- the same way the 'Fold' type from "Streamly.Data.Fold" represents data sinks or
254-- reducers of streams. Reducers can be combined to consume a stream source in
255-- many ways. The simplest is to reduce a stream source using a fold e.g.:
257-- > S.fold FL.length $ S.enumerateTo 100
259-- Folds are consumers of streams and can be used to split a stream into
260-- multiple independent flows. Grouping transforms a stream by applying a fold
261-- on segments of a stream, distributing applies multiple folds in parallel on
262-- the same stream and combines them, partitioning sends different elements of
263-- a stream to different folds, unzipping divides the elements of a stream into
264-- parts and sends them through different folds. Parsers are nothing but a
265-- particular type of folds. Transformations can be applied contravariantly on
266-- the input of a fold.
270-- |---transform----Fold m a b--------|
271-- ---stream m a-->transform-->| |---f b c ...
272-- |---transform----Fold m a c--------|
280-- Streamly arrays (See "Streamly.Data.Array.Foreign") complement streams to provide an
281-- efficient computing paradigm. Streams are suitable for immutable
282-- transformations of /potentially infinite/ data using /sequential access/ and
283-- pipelined transformations whereas arrays are suitable for in-place
284-- transformations of /necessarily finite/ data using /random access/. Streams
285-- are synonymous with /sequential pipelined processing/ whereas arrays are
286-- synonymous with /efficient buffering and random access/.
288-- In general, a data processing pipeline reads data from some IO device, does
289-- some processing on it and finally writes the output to another IO device.
290-- Streams provide the overall framework of sequential processing pipeline in
291-- which arrays are used as buffering elements in the middle. In addition to
292-- buffering in the middle, arrays can also be used at the boundaries of the
293-- pipeline to efficiently interface with external storage systems like memory,
294-- files and network. If streams are the pipes in a water pipeline network
295-- then arrays are like the storage tanks in the middle. On the input side,
296-- think of arrays as buckets to fetch water to feed the pipeline and on the
297-- output side buckets to remove the processed water.
299-- 'ByteString' data type from the 'bytestring' package and the 'Text' data
300-- type from the 'text' package are special cases of arrays. 'ByteString' is
301-- like @Array Word8@ and 'Text' is like @utf16@ encoded @Array Word8@.
302-- Streamly arrays can be transformed as efficiently as @bytestring@ or @text@
303-- by using stream operations on them.
305-- Streams and arrays are equally important in computing. They are computing
306-- duals of each other.
309-- The basic stream type is 'Serial', it represents a sequence of IO actions,
310-- and is a 'Monad'. The type 'SerialT' is a monad transformer that can
311-- represent a sequence of actions in an arbitrary monad. The type 'Serial' is
312-- in fact a synonym for @SerialT IO@. There are a few more types similar to
313-- 'SerialT', all of them represent a stream and differ only in the
314-- 'Semigroup', 'Applicative' and 'Monad' compositions of the stream. 'Serial'
315-- and 'WSerial' types compose serially whereas 'Async' and 'WAsync'
316-- types compose concurrently. All these types can be freely inter-converted
317-- using type combinators without any cost. You can freely switch to any type
318-- of composition at any point in the program. When no type annotation or
319-- explicit stream type combinators are used, the default stream type is
320-- inferred as 'Serial'.
322-- This module exports stream types, instances and combinators for:
324-- * converting between different stream types
325-- * appending and concurrently merging streams
326-- * Concurrency control
327-- * Concurrent function application
328-- * Stream rate control
330-- This module is designed to be imported unqualified:
336-- See the "Streamly.Prelude" module for APIs for construction,
337-- generation, elimination and transformation of streams.
339------------------------------------------------------------------------------
340-- Eliminating a stream
341------------------------------------------------------------------------------
343-- | Same as "Streamly.Prelude.runStream".
345{-# DEPRECATED runStream "Please use Streamly.Prelude.drain instead." #-}
346runStream :: Monad m => SerialT m a -> m ()
350-- | Same as "Streamly.Prelude.foldWith".
352{-# DEPRECATED foldWith "Please use Streamly.Prelude.foldWith instead." #-}
353{-# INLINABLE foldWith #-}
354foldWith :: (IsStream t, Foldable f)
355 => (t m a -> t m a -> t m a) -> f (t m a) -> t m a
358-- | Same as "Streamly.Prelude.foldMapWith".
360{-# DEPRECATED foldMapWith "Please use Streamly.Prelude.foldMapWith instead." #-}
361{-# INLINABLE foldMapWith #-}
362foldMapWith :: (IsStream t, Foldable f)
363 => (t m b -> t m b -> t m b) -> (a -> t m b) -> f a -> t m b
364foldMapWith = P.foldMapWith
366-- | Same as "Streamly.Prelude.forEachWith".
368{-# DEPRECATED forEachWith "Please use Streamly.Prelude.forEachWith instead." #-}
369{-# INLINABLE forEachWith #-}
370forEachWith :: (IsStream t, Foldable f)
371 => (t m b -> t m b -> t m b) -> f a -> (a -> t m b) -> t m b
372forEachWith = P.forEachWith
375-- XXX Deprecate it in 0.8.0
377-- | Make a stream asynchronous, triggers the computation and returns a stream
378-- in the underlying monad representing the output generated by the original
379-- computation. The returned action is exhaustible and must be drained once. If
380-- not drained fully we may have a thread blocked forever and once exhausted it
381-- will always return 'empty'.
384{-# INLINABLE mkAsync #-}
385mkAsync :: (IsStream t, MonadAsync m) => t m a -> m (t m a)
386mkAsync = return . Transform.mkAsync
388------------------------------------------------------------------------------
390------------------------------------------------------------------------------
394-- When a stream consumer demands an element from a serial stream constructed
395-- as @a \`consM` b \`consM` ... nil@, the action @a@ at the head of the stream
396-- sequence is executed and the result is supplied to the consumer. When the
397-- next element is demanded, the action @b@ is executed and its result is
398-- supplied. Thus, the effects are performed and results are consumed strictly
399-- in a serial order. Serial streams can be considered as /spatially ordered/
400-- streams as the order of execution and consumption is the same as the spatial
401-- order in which the actions are composed by the programmer.
403-- Serial streams enforce the side effects as well as the results of the
404-- actions to be in the same order in which the actions are added to the
405-- stream. Therefore, the semigroup operation for serial streams is not
409-- a <> b is not the same as b <> a
412-- There are two serial stream types 'SerialT' and 'WSerialT'. The stream
413-- evaluation of both the variants works in the same way as described above,
414-- they differ only in the 'Semigroup' and 'Monad' implementaitons.
418-- When a stream consumer demands an element from a speculative stream
419-- constructed as @a \`consM` b \`consM` ... nil@, the action @a@ at the head
420-- of the stream is executed and the output of the action is supplied to the
421-- consumer. However, in addition to the action at the head multiple actions
422-- following it may also be executed concurrently and the results buffered.
423-- When the next element is demanded it may be served from the buffer and we
424-- may execute the next action in the sequence to keep the buffer adequately
425-- filled. Thus, the actions are executed concurrently but results consumed in
426-- serial order just like serial streams. `consM` can be used to fold an
427-- infinite lazy container of effects, as the number of concurrent executions
430-- Similar to 'consM', the monadic stream generation (e.g. replicateM) and
431-- transformation operations (e.g. mapM) on speculative streams can execute
432-- multiple effects concurrently in a speculative manner.
434-- How many effects can be executed concurrently and how many results can be
435-- buffered are controlled by 'maxThreads' and 'maxBuffer' combinators
436-- respectively. The actual number of concurrent threads is adjusted according
437-- to the rate at which the consumer is consuming the stream. It may even
438-- execute actions serially in a single thread if that is enough to match the
441-- Speculative streams enforce ordering of the results of actions in the stream
442-- but the side effects are only partially ordered. Therefore, the semigroup
443-- operation for speculative streams is not commutative from the pure outputs
444-- perspective but commutative from side effects perspective.
448-- /Scheduling and execution:/ In an asynchronous stream @a \`consM` b \`consM`
449-- c ...@, the actions @a@, @b@, and @c@ are executed concurrently with the
450-- consumer of the stream. The actions are /scheduled/ for execution in the
451-- same order as they are specified in the stream. Multiple scheduled actions
452-- may be /executed/ concurrently in parallel threads of execution. The
453-- actions may be executed out of order and they may complete at arbitrary
454-- times. Therefore, the /effects/ of the actions may be observed out of
457-- /Buffering:/ The /results/ from multiple threads of execution are queued in
458-- a buffer as soon as they become available. The consumer of the stream is
459-- served from this buffer. Therefore, the consumer may observe the results to
460-- be out of order. In other words, an asynchronous stream is an unordered
461-- stream i.e. order does not matter.
463-- /Concurrency control:/ Threads are suspended if the `maxBuffer` limit is
464-- reached, and resumed when the consumer makes space in the buffer. The
465-- maximum number of concurrent threads depends on `maxThreads`. Number of
466-- threads is increased or decreased based on the speed of the consumer.
468-- /Generation operations:/ Concurrent stream generation operations e.g.
469-- 'Streamly.Prelude.replicateM' when used in async style schedule and execute
470-- the stream generating actions in the manner described above. The generation
471-- actions run concurrently, effects and results of the actions as observed by
472-- the consumer of the stream may be out of order.
474-- /Transformation operations:/ Concurrent stream transformation operations
475-- e.g. 'Streamly.Prelude.mapM', when used in async style, schedule and
476-- execute transformation actions in the manner described above. Transformation
477-- actions run concurrently, effects and results of the actions may be
478-- observed by the consumer out of order.
480-- /Variants:/ There are two asynchronous stream types 'AsyncT' and 'WAsyncT'.
481-- They are identical with respect to single stream evaluation behavior. Their
482-- behaviors differ in how they combine multiple streams using 'Semigroup' or
483-- 'Monad' composition. Since the order of elements does not matter in
484-- asynchronous streams the 'Semigroup' operation is effectively commutative.
488-- 'ZipSerialM' and 'ZipAsyncM', provide 'Applicative' instances for zipping the
489-- corresponding elements of two streams together. Note that these types are
494-- Stream processing functions can be composed in a chain using function
495-- application with or without the '$' operator, or with reverse function
496-- application operator '&'. Streamly provides concurrent versions of these
497-- operators applying stream processing functions such that each stage of the
498-- stream can run in parallel. The operators start with a @|@; we can read '|$'
499-- as "@parallel dollar@" to remember that @|@ comes before '$'.
501-- Imports for the code snippets below:
505-- import qualified Streamly.Prelude as S
506-- import Control.Concurrent
510-- The 'Semigroup' operation '<>' of each stream type combines two streams in a
511-- type specific manner. This section provides polymorphic versions of '<>'
512-- which can be used to combine two streams in a predetermined way irrespective
515-- XXX An alternative design choice would be to let a control parameter affect
516-- the nearest SVar only and then it gets cleared. The benefit of the current
517-- choice is that it is simply just like global configuration, just like state
518-- behaves, so should be easy to comprehend. But it has the downside of leaking
519-- to undesired actions, that is we can forget to reset it.
523-- These combinators can be used at any point in a stream composition to set
524-- parameters to control the concurrency of the /argument stream/. A control
525-- parameter set at any point remains effective for any concurrent combinators
526-- used in the argument stream until it is reset by using the combinator again.
527-- These control parameters have no effect on non-concurrent combinators in the
528-- stream, or on non-concurrent streams.
530-- /Pitfall:/ Remember that 'maxBuffer' in the following example applies to
531-- 'mapM' and any other combinators that may follow it, and it does not apply
532-- to the combinators before it:
541-- If we use '&' instead of '$' the situation will reverse, in the following
542-- example, 'maxBuffer' does not apply to 'mapM', it applies to combinators
543-- that come before it, because those are the arguments to 'maxBuffer':
554-- You may want to use different stream composition styles at different points
555-- in your program. Stream types can be freely converted or adapted from one
556-- type to another. The 'IsStream' type class facilitates type conversion of
557-- one stream type to another. It is not used directly, instead the type
558-- combinators provided below are used for conversions.
560-- To adapt from one monomorphic type (e.g. 'AsyncT') to another monomorphic
561-- type (e.g. 'SerialT') use the 'adapt' combinator. To give a polymorphic code
562-- a specific interpretation or to adapt a specific type to a polymorphic type
563-- use the type specific combinators e.g. 'fromAsync' or 'fromWSerial'. You
564-- cannot adapt polymorphic code to polymorphic code, as the compiler would not know
565-- which specific type you are converting from or to. If you see a an
566-- @ambiguous type variable@ error then most likely you are using 'adapt'
567-- unnecessarily on polymorphic code.
569-- | Same as 'Streamly.Prelude.concatFoldableWith'
572{-# DEPRECATED foldWith "Please use 'Streamly.Prelude.concatFoldableWith' instead." #-}
573{-# INLINEABLE foldWith #-}
574foldWith :: (IsStream t, Foldable f) => (t m a -> t m a -> t m a) -> f (t m a) -> t m a
575foldWith = P.concatFoldableWith
577-- | Same as 'Streamly.Prelude.concatMapFoldableWith'
580{-# DEPRECATED foldMapWith "Please use 'Streamly.Prelude.concatMapFoldableWith' instead." #-}
581{-# INLINEABLE foldMapWith #-}
582foldMapWith :: (IsStream t, Foldable f) => (t m b -> t m b -> t m b) -> (a -> t m b) -> f a -> t m b
583foldMapWith = P.concatMapFoldableWith
585-- | Same as 'Streamly.Prelude.concatForFoldableWith'
588{-# DEPRECATED forEachWith "Please use 'Streamly.Prelude.concatForFoldableWith' instead." #-}
589{-# INLINEABLE forEachWith #-}
590forEachWith :: (IsStream t, Foldable f) => (t m b -> t m b -> t m b) -> f a -> (a -> t m b) -> t m b
591forEachWith = P.concatForFoldableWith
593{-# DEPRECATED serially "Please use 'Streamly.Prelude.fromSerial' instead." #-}
594serially :: IsStream t => SerialT m a -> t m a
597{-# DEPRECATED wSerially "Please use 'Streamly.Prelude.fromWSerial' instead." #-}
598wSerially :: IsStream t => WSerialT m a -> t m a
599wSerially = fromWSerial
601{-# DEPRECATED asyncly "Please use 'Streamly.Prelude.fromAsync' instead." #-}
602asyncly :: IsStream t => AsyncT m a -> t m a
605{-# DEPRECATED aheadly "Please use 'Streamly.Prelude.fromAhead' instead." #-}
606aheadly :: IsStream t => AheadT m a -> t m a
609{-# DEPRECATED wAsyncly "Please use 'Streamly.Prelude.fromWAsync' instead." #-}
610wAsyncly :: IsStream t => WAsyncT m a -> t m a
613{-# DEPRECATED parallely "Please use 'Streamly.Prelude.fromParallel' instead." #-}
614parallely :: IsStream t => ParallelT m a -> t m a
615parallely = fromParallel
617{-# DEPRECATED zipSerially "Please use 'Streamly.Prelude.fromZipSerial' instead." #-}
618zipSerially :: IsStream t => ZipSerialM m a -> t m a
619zipSerially = fromZipSerial
621{-# DEPRECATED zipAsyncly "Please use 'Streamly.Prelude.fromZipAsync' instead." #-}
622zipAsyncly :: IsStream t => ZipAsyncM m a -> t m a
623zipAsyncly = fromZipAsync
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list