@@ -8,7 +8,8 @@
88--
99-- All Stream related combinators including the streamly-core
1010-- "Streamly.Data.Stream" module, concurrency, time and lifted
11−-- exception operations.
11+-- exception operations. For more pre-release operations also see
12+-- "Streamly.Internal.Data.Stream.Prelude" module.
1213--
1314module Streamly.Data.Stream.Prelude
1415 (
@@ -17,22 +18,205 @@ module Streamly.Data.Stream.Prelude
1718 -- module. For more pre-release combinators also see
1819 -- "Streamly.Internal.Data.Stream" module.
1920 module Streamly.Data.Stream
21+
2022 -- * Concurrent Operations
21− -- | Also see "Streamly.Internal.Data.Stream.Concurrent" for more
22− -- pre-release functions.
23− , module Streamly.Data.Stream.Concurrent
23+ -- $concurrency
24+
25+ -- ** Types
26+ , MonadAsync
27+
28+ -- ** Configuration
29+ , Config
30+
31+ -- *** Limits
32+ , maxThreads
33+ , maxBuffer
34+
35+ -- *** Rate Control
36+ , Rate(..)
37+ , rate
38+ , avgRate
39+ , minRate
40+ , maxRate
41+ , constRate
42+
43+ -- *** Stop behavior
44+ , StopWhen (..)
45+ , stopWhen
46+
47+ -- *** Scheduling behavior
48+ , eager
49+ , ordered
50+ , interleaved
51+
52+ -- *** Diagnostics
53+ , inspect
54+
55+ -- ** Combinators
56+ -- | Stream combinators using a concurrent channel.
57+
58+ -- *** Evaluate
59+ -- | Evaluates a serial stream asynchronously using a concurency channel.
60+
61+ , parEval
62+
63+ -- *** Generate
64+ -- | Uses a concurrency channel to evaluate multiple actions concurrently.
65+ , parRepeatM
66+ , parReplicateM
67+ , fromCallback
68+
69+ -- *** Map
70+ -- | Uses a concurrency channel to evaluate multiple mapped actions
71+ -- concurrently.
72+
73+ , parMapM
74+ , parSequence
75+
76+ -- *** Combine two
77+ -- | Use a channel for each pair.
78+ , parZipWithM
79+ , parZipWith
80+ , parMergeByM
81+ , parMergeBy
82+
83+ -- *** List of streams
84+ -- | Shares a single channel across many streams.
85+
86+ , parList
87+ -- , zipWithM
88+ -- , zipWith
89+
90+ -- *** Stream of streams
91+ -- **** Apply
92+
93+ , parApply
94+
95+ -- **** Concat
96+ -- | Shares a single channel across many streams.
97+
98+ , parConcat
99+ , parConcatMap
100+
101+ -- **** ConcatIterate
102+ , parConcatIterate
103+
104+ -- *** Observation
105+ , parTapCount
106+
24107 -- * Time Related
25− -- | Also see "Streamly.Internal.Data.Stream.Time" for more pre-release
26− -- functions.
27− , module Streamly.Data.Stream.Time
108+
109+ -- ** Timers
110+ , interject
111+
112+ -- ** Trimming
113+ , takeInterval
114+ , dropInterval
115+
116+ -- ** Chunking
117+ , intervalsOf
118+
119+ -- ** Sampling
120+ , sampleIntervalEnd
121+ , sampleIntervalStart
122+ , sampleBurstEnd
123+ , sampleBurstStart
124+
28125 -- * Lifted Exceptions
29− -- | Also see "Streamly.Internal.Data.Stream.Exception.Lifted" for more
30− -- pre-release functions.
31− , module Streamly.Data.Stream.Exception
126+ , after
127+ , bracket
128+ -- , bracket3
129+ , finally
130+
131+ -- ** Deprecated
132+ , tapCount
32133 )
33134where
34135
35136import Streamly.Data.Stream
36−import Streamly.Data.Stream.Concurrent
37−import Streamly.Data.Stream.Exception
38−import Streamly.Data.Stream.Time
137+import Streamly.Internal.Data.Stream.Prelude
138+
139+-- $concurrency
140+--
141+-- == Channels
142+--
143+-- At a lower level, concurrency is implemented using channels that support
144+-- concurrent evaluation of streams. We create a channel, and add one or more
145+-- streams to it. The channel evaluates multiple streams concurrently and then
146+-- generates a single output stream from the results. How the streams are
147+-- combined depends on the configuration of the channel.
148+--
149+-- == Primitives
150+--
151+-- There are only a few fundamental abstractions for concurrency, 'parEval',
152+-- 'parConcatMap', and 'parConcatIterate', all concurrency combinators can be
153+-- expressed in terms of these.
154+--
155+-- 'parEval', evaluates a single stream asynchronously, a worker thread runs
156+-- the stream and buffers the results, and the consumer of the stream runs in
157+-- another thread consuming it from the buffer, thus decoupling the production
158+-- and consumption of the stream. This can be used to run different stages of a
159+-- pipeline concurrently.
160+--
161+-- 'parConcatMap' is used to evaluate multiple streams concurrently and combine
162+-- the results. A stream generator function is mapped to the input stream and
163+-- all the generated streams are then evaluated concurrently, and the results
164+-- are combined.
165+--
166+-- 'parConcatIterate' is like 'parConcatMap' but iterates a stream generator
167+-- function recursively over the stream. This can be used to traverse trees or
168+-- graphs.
169+--
170+-- == Configuration
171+--
172+-- Concurrent combinators take a 'Config' argument which controls the
173+-- concurrent behavior. For example, maximum number of threads to be used
174+-- ('maxThreads') or the maxmimum size of the buffer ('maxBuffer'), or how the
175+-- streams are scheduled with respect to each other ('interleaved'), or how the
176+-- results are consumed ('ordered').
177+--
178+-- Configuration is specified as @Config -> Config@ modifier functions that can
179+-- be composed together using function composition. For example, to specify the
180+-- maximum threads we can use @parConcatMap (maxThreads 10)@ if we also want to
181+-- specify the maximum buffer we can compose the two options @parConcatMap
182+-- (maxThreads 10 . maxBuffer 100)@. To use default configuration use 'id' as
183+-- the config modifier e.g. @parConcatMap id@.
184+--
185+-- See the @Configuration@ section and individual configuration options'
186+-- documentation for the default behavior and default values of configuration
187+-- parameters.
188+--
189+-- == Scheduling
190+--
191+-- The most important configuration option is to control whether the output of
192+-- the concurrent execution is consumed in the same order as the corresponding
193+-- actions in the input stream or as soon as they arrive. The default is the
194+-- latter, however, we can enforce the original order by using the 'ordered'
195+-- option.
196+--
197+-- Another important option controls whether the number of worker threads are
198+-- automatically increased and decreased based on the consumption rate or
199+-- threads are started as aggresively as possible until the 'maxThreads' or
200+-- 'maxBuffer' limits are hit. The default is the former. However, the 'eager'
201+-- option can be enabled to use the latter behavior. When 'eager' is on, even
202+-- if the stream consumer thread blocks it does not make any impact on the
203+-- scheduling of the available tasks.
204+--
205+-- == Combinators
206+--
207+-- Using the few fundamental concurrency primitives we can implement all the
208+-- usual streaming combinators with concurrent behavior. Combinators like
209+-- 'unfoldrM', 'iterateM' that are inherently serial can be evaluated
210+-- concurrently with respect to the consumer pipeline using 'parEval'.
211+-- Combinators like 'zipWithM', 'mergeByM' can also use 'parEval' on the input
212+-- streams to evaluate them concurrently before combining.
213+--
214+-- Combinators like 'repeatM', 'replicateM', 'fromListM', 'sequence', 'mapM' in
215+-- which all actions are independent of each other can be made concurrent using
216+-- the 'parConcatMap' operation.
217+--
218+-- A concurrent 'repeatM' repeats an action using multiple concurrent
219+-- executions of the action. Similarly, a concurrent 'mapM' performs the mapped
220+-- action in independent threads.
221+--
222+-- Some common concurrent combinators are provided in this module.