git.iko.soy/streamly
on master help sign in
update docs, readme, cabal description and tutorial
commitca0e9a935c3d2714f76eb3e005f9a16e92cd7937 authorHarendra Kumar <harendra.kumar@gmail.com> dateSat 5 May 2018, 00:23 (8 years ago) parentb214a85
6 files changed+406−284
compare with master
unifiedsplit
Changelog.md+3 README.md+120−74 src/Streamly.hs+70−96 src/Streamly/Prelude.hs+6 src/Streamly/Tutorial.hs+165−99 streamly.cabal+42−15
Changelog.md+3
@@ -64,6 +64,9 @@
6464
6565### Enhancements
6666* Add the following functions:
67+ * `consM` and `|:` operator to construct streams from monadic actions
68+ * `once` to create a singleton stream from a monadic action
69+ * `repeatM` to construct a stream by repeating a monadic action
6770 * `scanl'` strict left scan
6871 * `foldl'` strict left fold
6972 * `foldlM'` strict left fold with a monadic fold function
README.md+120−74
@@ -8,23 +8,118 @@
88
99## Stream`ing` `Concurrent`ly
1010
11−Streamly is a monad transformer unifying non-determinism
12−([list-t](https://hackage.haskell.org/package/list-t)/[logict](https://hackage.haskell.org/package/logict)),
13−concurrency ([async](https://hackage.haskell.org/package/async)),
14−streaming ([conduit](https://hackage.haskell.org/package/conduit)\/[pipes](https://hackage.haskell.org/package/pipes)),
15−and FRP ([Yampa](https://hackage.haskell.org/package/Yampa)\/[reflex](https://hackage.haskell.org/package/reflex))
16−functionality in a concise and intuitive API.
17−High level concurrency makes concurrent applications almost indistinguishable
18−from non-concurrent ones. By changing a single combinator you can control
19−whether the code runs serially or concurrently. It naturally integrates
20−concurrency with streaming rather than adding it as an afterthought.
21−Moreover, it interworks with the popular streaming libraries.
22−
23−See the haddock documentation for full reference. It is recommended that you
24−read `Streamly.Tutorial` first. Also see `Streamly.Examples` for some working
25−examples.
26−
27−## Non-determinism
11+Streamly, short for stream concurrently, is a simple yet powerful streaming
12+library with concurrent merging and concurrent nested looping support. A stream
13+is just like a list except that it is a list of monadic actions rather than
14+pure values. Streamly streams can be generated, consumed, combined, or
15+transformed serially or concurrently. We can loop over a stream serially or
16+concurrently. We can also have serial or concurrent nesting of loops. For
17+those familiar with the list transformer concept this is a concurrent list
18+transformer. Streamly uses standard composition abstractions. Concurrent
19+composition is just the same as serial composition except that we use a simple
20+combinator to request a concurrent composition instead of serial. The
21+programmer does not have to be aware of threads, locking or synchronization to
22+write scalable concurrent programs.
23+
24+Streamly provides functionality that is equivalent to streaming libraries
25+like [pipes](https://hackage.haskell.org/package/pipes) and
26+[conduit](https://hackage.haskell.org/package/conduit) but with a list like
27+API. The streaming API of streamly is close to the monadic streams API of the
28+[vector](https://hackage.haskell.org/package/vector) package and similar in
29+concept to the [streaming](https://hackage.haskell.org/package/streaming)
30+package. In addition to streaming, streamly subsumes the functionality of list
31+transformer libraries like `pipes` or
32+[list-t](https://hackage.haskell.org/package/list-t) and also the logic
33+programming library [logict](https://hackage.haskell.org/package/logict). On
34+the concurrency side, it subsumes the functionality of the
35+[async](https://hackage.haskell.org/package/async) package. Because it streams
36+and supports concurrency we can write FRP applications similar in concept to
37+[Yampa](https://hackage.haskell.org/package/Yampa) or
38+[reflex](https://hackage.haskell.org/package/reflex).
39+
40+Streamly has excellent performance, see
41+[streaming-benchmarks](https://github.com/composewell/streaming-benchmarks)
42+for a comparison of popular streaming libraries on micro-benchmarks. For
43+more information:
44+
45+ * [Streamly.Tutorial](https://hackage.haskell.org/package/streamly-0.1.2/docs/Streamly-Tutorial.html) module in the haddock documentation for a detailed introduction
46+ * [examples](https://github.com/composewell/streamly/tree/master/examples) directory in the package for some simple practical examples
47+
48+## Streaming Pipelines
49+
50+Unlike `pipes` or `conduit` and like `vector` and `streaming` `streamly`
51+composes stream data instead of stream processors (functions). A stream is
52+just like a list and is explicitly passed around to functions that process the
53+stream. Therefore, no special operator is needed to join stages in a streaming
54+pipeline, just a forward (`$`) or reverse (`&`) function application operator
55+is enough. Combinators are provided in `Streamly.Prelude` to transform or fold
56+streams.
57+
58+```haskell
59+import Streamly
60+import qualified Streamly.Prelude as S
61+import Data.Function ((&))
62+
63+main = runStream $
64+ S.repeatM getLine
65+ & fmap read
66+ & S.filter even
67+ & S.takeWhile (<= 9)
68+ & fmap (\x -> x * x)
69+ & S.mapM print
70+```
71+
72+## Serial and Concurrent Merging
73+
74+Semigroup and Monoid instances can be used to fold streams serially or
75+concurrently. In the following example we are composing ten actions in the
76+stream each with a delay of 1 to 10 seconds. Since all the actions are
77+concurrent we see one output printed every second:
78+
79+``` haskell
80+import Streamly
81+import qualified Streamly.Prelude as S
82+import Control.Concurrent (threadDelay)
83+
84+main = S.toList $ parallely $ foldMap delay [1..10]
85+ where delay n = S.once $ threadDelay (n * 1000000) >> print n
86+```
87+
88+Streams can be combined together in many ways (see the tutorial for more ways):
89+
90+``` haskell
91+import Streamly
92+import qualified Streamly.Prelude as S
93+import Control.Concurrent
94+
95+delay n = S.once $ do
96+ threadDelay (n * 1000000)
97+ tid <- myThreadId
98+ putStrLn (show tid ++ ": Delay " ++ show n)
99+```
100+### Serial
101+
102+```haskell
103+main = runStream $ delay 3 <> delay 2 <> delay 1
104+```
105+```
106+ThreadId 36: Delay 3
107+ThreadId 36: Delay 2
108+ThreadId 36: Delay 1
109+```
110+
111+### Parallel
112+
113+```haskell
114+main = runStream . parallely $ delay 3 <> delay 2 <> delay 1
115+```
116+```
117+ThreadId 42: Delay 1
118+ThreadId 41: Delay 2
119+ThreadId 40: Delay 3
120+```
121+
122+## Nested Loops (aka List Transformer)
28123
29124The monad instance composes like a list monad.
30125
@@ -46,7 +141,7 @@ main = runStream loops
46141(2,4)
47142```
48143
49−## Magical Concurrency
144+## Concurrent Nested Loops
50145
51146To run the above code with demand-driven concurrency i.e. each iteration in the
52147loops can run concurrently depending on the consumer rate:
@@ -67,6 +162,8 @@ To run it serially but interleaving the outer and inner loop iterations:
67162main = runStream $ costreamly $ loops
68163```
69164
165+## Magical Concurrency
166+
70167Streams can perform semigroup (<>) and monadic bind (>>=) operations
71168concurrently using combinators like `coparallelly`, `parallelly`. For example,
72169to concurrently generate squares of a stream of numbers and then concurrently
@@ -112,65 +209,14 @@ The concurrency facilities provided by streamly can be compared with
112209[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative
113210expression.
114211
115−## Streaming
116−
117−Streaming is effortless, simple and straightforward. Streamly data type behaves
118−just like a list and combinators are provided in `Streamly.Prelude` to
119−transform or fold streamly streams. Unlike other libraries and like `streaming`
120−library the combinators explicitly consume a stream and produce a stream,
121−therefore, no special operator is needed to join stream stages, just a forward
122−(`$`) or reverse (`&`) function application operator is enough.
123−
124−```haskell
125−import Streamly
126−import qualified Streamly.Prelude as S
127−import Data.Function ((&))
128−
129−main = S.fromFoldable [1..10]
130− & fmap (+ 1)
131− & S.drop 2
132− & S.filter even
133− & fmap (* 3)
134− & S.takeWhile (< 25)
135− & S.mapM (\x -> putStrLn ("saw " ++ show x) >> return x)
136− & S.toList
137− >>= print
138−```
139−
140−Fold style combinators can be used to fold purely or monadically. You can also
141−use the beautiful `foldl` library for folding.
142−
143−```haskell
144−main = S.fromFoldable [1..10]
145− & S.foldl (+) 0 id
146− >>= print
147−```
148−
149−Streams can be combined together in multiple ways:
150−
151−```haskell
152−main = do
153− let p s = toList s >>= print
154− -- serial, this is the default even if you omit the `streamly` combinator
155− p $ streamly $ S.fromFoldable [1..10] <> S.fromFoldable [11..20]
156− -- serial but interleaved
157− p $ costreamly $ S.fromFoldable [1..10] <> S.fromFoldable [11..20]
158− -- left-biased, demand driven parallel
159− p $ coparallely $ S.fromFoldable [1..10] <> S.fromFoldable [11..20]
160− -- round-robin parallel
161− p $ parallely $ S.fromFoldable [1..10] <> S.fromFoldable [11..20]
162−```
163−
164−As we have already seen streams can be combined using monadic composition in a
165−non-deterministic manner. This allows arbitrary manipulation and combining of
166−streams. See `Streamly.Examples.MergeSortedStreams` for a more complicated
167−example.
168−
169212## Reactive Programming (FRP)
170213
171214Streamly is a foundation for first class reactive programming as well by virtue
172−of integrating concurrency and streaming. See `Streamly.Examples.AcidRainGame`
173−and `Streamly.Examples.CirclingSquare` for an SDL based animation example.
215+of integrating concurrency and streaming. See
216+[AcidRain.hs](https://github.com/composewell/streamly/tree/master/examples/AcidRain.hs)
217+for a console based FRP game example and
218+[CirclingSquare.hs](https://github.com/composewell/streamly/tree/master/examples/CirclingSquare.hs)
219+for an SDL based animation example.
174220
175221## Performance
176222
src/Streamly.hs+70−96
@@ -6,28 +6,60 @@
66-- Maintainer : harendra.kumar@gmail.com
77-- Stability : experimental
88-- Portability : GHC
9+--
10+-- The way a list represents a sequence of pure values, a stream represents a
11+-- sequence of monadic actions. The monadic stream API offered by Streamly is
12+-- very close to the Haskell "Prelude" pure lists' API, it can be considered as
13+-- a natural extension of lists to monadic actions. Streamly streams provide
14+-- concurrent composition and merging of streams. It can be considered as a
15+-- concurrent list transformer. In contrast to the "Prelude" lists, merging or
16+-- appending streams of arbitrary length is scalable and inexpensive.
17+--
18+-- The basic stream type is 'Stream', it represents a sequence of IO actions,
19+-- and is a 'Monad'. The type 'StreamT' is a monad transformer that can
20+-- represent a sequence of actions in an arbitrary monad. The type 'Stream' is
21+-- in fact a synonym for @StreamT IO@. There are a few more types similar to
22+-- 'StreamT', all of them represent a stream and differ only in the
23+-- 'Semigroup', 'Applicative' and 'Monad' compositions of the stream. 'Stream'
24+-- and 'Costream' types compose serially whereas 'Coparallel' and 'Parallel'
25+-- types compose concurrently. All these types can be freely inter-converted
26+-- using type combinators without any cost. You can freely switch to any type
27+-- of composition at any point in the program. When no type annotation or
28+-- explicit stream type combinators are used, the default stream type is
29+-- inferred as 'Stream'.
30+--
31+-- Here is a simple console echo program example:
32+--
33+-- @
34+-- > runStream $ S.repeatM getLine & S.mapM putStrLn
35+-- @
36+--
37+-- For more details please see the "Streamly.Tutorial" module and the examples
38+-- directory in this package.
39+--
40+-- This module exports stream types, instances and some basic operations.
41+-- Functionality exported by this module include:
42+--
43+-- * Semigroup append ('<>') instances as well as explicit operations for merging streams
44+-- * Monad and Applicative instances for looping over streams
45+-- * Zip Applicatives for zipping streams
46+-- * Stream type combinators to convert between different composition styles
47+-- * Some basic utilities to run and fold streams
48+--
49+-- See the "Streamly.Prelude" module for comprehensive APIs for construction,
50+-- generation, elimination and transformation of streams.
51+--
52+-- This module is designed to be imported unqualified:
53+--
54+-- @
55+-- import Streamly
56+-- @
957
1058module Streamly
1159 (
12− -- * Background
13− -- $background
14−
15− -- * Overview
16− -- $overview
17−
1860 MonadParallel
1961
20− -- * IO Streams
21− , Stream
22− , Costream
23− , Coparallel
24− , Parallel
25− , ZipStream
26− , ZipParallel
27−
2862 -- * Stream transformers
29− , IsStream
30−
3163 -- ** Serial Streams
3264 -- $serial
3365 , StreamT
@@ -43,8 +75,17 @@ module Streamly
4375 , ZipStreamM
4476 , ZipParallelM
4577
78+ -- * Polymorphic Sum Operations
79+ -- $sum
80+ , splice
81+ , cosplice
82+ , coparallel
83+ , parallel
84+
4685 -- * Stream Type Adapters
4786 -- $adapters
87+ , IsStream
88+
4889 , streamly
4990 , costreamly
5091 , coparallely
@@ -53,19 +94,20 @@ module Streamly
5394 , zipParallely
5495 , adapt
5596
97+ -- * IO Streams
98+ , Stream
99+ , Costream
100+ , Coparallel
101+ , Parallel
102+ , ZipStream
103+ , ZipParallel
104+
56105 -- * Running Streams
57106 , runStream
58107
59108 -- * Transformation
60109 , async
61110
62− -- * Polymorphic Sum Operations
63− -- $sum
64− , splice
65− , cosplice
66− , coparallel
67− , parallel
68−
69111 -- * Polymorphic Fold Utilities
70112 -- $foldutils
71113 , foldWith
@@ -99,76 +141,6 @@ where
99141import Streamly.Streams
100142import Data.Semigroup (Semigroup(..))
101143
102−-- $background
103−--
104−-- Streamly provides a monad transformer that extends the product style
105−-- composition of monads to streams of many elements of the same type; it is a
106−-- functional programming equivalent of nested loops from imperative
107−-- programming. Composing each element in one stream with each element in the
108−-- other stream generalizes the monadic product of single elements. You can
109−-- think of the IO monad as a special case of the more general @StreamT IO@
110−-- monad; with single element streams. List transformers and logic programming
111−-- monads also provide a similar product style composition of streams, however
112−-- streamly generalizes it with the time dimension; allowing streams to be
113−-- composed in an asynchronous and concurrent fashion in many different ways.
114−-- It also provides multiple alternative ways of composing streams e.g.
115−-- serial, interleaved or concurrent.
116−--
117−-- The seemingly simple addition of asynchronicity and concurrency to product
118−-- style streaming composition unifies a number of disparate abstractions into
119−-- one powerful and elegant abstraction. A wide variety of programming
120−-- problems can be solved elegantly with this abstraction. In particular, it
121−-- unifies three major programming domains namely non-deterministic (logic)
122−-- programming, concurrent programming and functional reactive programming. In
123−-- other words, you can do everything with this one abstraction that you could
124−-- with list transformers (e.g.
125−-- <https://hackage.haskell.org/package/list-t list-t>), logic programming
126−-- monads (e.g. <https://hackage.haskell.org/package/logict logict>),
127−-- streaming libraries (a lot of what
128−-- <https://hackage.haskell.org/package/conduit conduit> or
129−-- <https://hackage.haskell.org/package/pipes pipes> can do), concurrency
130−-- libraries (e.g. <https://hackage.haskell.org/package/async async>) and FRP
131−-- libraries (e.g. <https://hackage.haskell.org/package/Yampa Yampa> or
132−-- <https://hackage.haskell.org/package/reflex reflex>).
133−
134−-- $overview
135−--
136−-- Streamly provides six distinct stream types i.e. 'StreamT', 'CostreamT',
137−-- 'CoparallelT' and 'ParallelT', 'ZipStreamM' and 'ZipParallelM', each representing a
138−-- stream of elements. All these types have the same underlying representation
139−-- and can be adapted from one to another using type adaptor combinators
140−-- described later. Each of these types belongs to the 'IsStream' type class
141−-- which helps converting the specific type to and from the underlying generic
142−-- stream type.
143−--
144−-- The types 'StreamT', 'CostreamT', 'CoparallelT' and 'ParallelT' are 'Monad'
145−-- transformers with the monadic bind operation combining streams in a product
146−-- style in much the same way as a list monad or a list transformer i.e. each
147−-- element from one stream is combined with every element of the other stream.
148−-- However, the applicative and monadic composition of these types differ in
149−-- terms of the ordering and time sequence in which the elements from two
150−-- streams are combined. 'StreamT' and 'CostreamT' compose streams streamly
151−-- whereas 'CoparallelT' and 'ParallelT' are their concurrent counterparts. See the
152−-- documentation of the respective types for more details.
153−--
154−-- The types 'ZipStreamM' and 'ZipParallelM' provide 'Applicative' instances to zip
155−-- two streams together i.e. each element in one stream is combined with the
156−-- corresponding element in the other stream. 'ZipStreamM' generates the streams
157−-- being zipped serially whereas 'ZipParallelM' produces both the elements being
158−-- zipped concurrently.
159−--
160−-- Two streams of the same type can be merged using a sum style composition to
161−-- generate a stream of the same type where the output stream would contain all
162−-- elements of both the streams. However, the sequence or the manner
163−-- (concurrent or serial) in which the elements in the resulting stream are
164−-- produced depends on the type of the stream.
165−--
166−-- For more details please see the "Streamly.Tutorial" and "Streamly.Examples"
167−-- (the latter is available only when built with the 'examples' build flag).
168−
169−-- A simple inline example here illustrating applicative, monad and alternative
170−-- compositions.
171−
172144-- $serial
173145--
174146-- Serial streams compose serially or non-concurrently. In a composed stream,
@@ -205,9 +177,11 @@ import Data.Semigroup (Semigroup(..))
205177
206178-- $adapters
207179--
208−-- You may want to use different stream types at different points in your
209−-- program. Stream types can be converted or adapted from one type to another
210−-- to make them interwork with each other.
180+-- You may want to use different stream composition styles at different points
181+-- in your program. Stream types can be freely converted or adapted from one
182+-- type to another. The 'IsStream' type class facilitates type conversion of
183+-- one stream type to another. It is not used directly, instead the type
184+-- combinators provided below are used for conversions.
211185--
212186-- To adapt from one monomorphic type (e.g. 'ParallelT') to another monomorphic
213187-- type (e.g. 'StreamT') use the 'adapt' combinator. To give a polymorphic code
src/Streamly/Prelude.hs+6
@@ -14,6 +14,12 @@
1414-- Stability : experimental
1515-- Portability : GHC
1616--
17+-- This module is designed to be imported qualified:
18+--
19+-- @
20+-- import qualified Streamly.Prelude as S
21+-- @
22+--
1723-- Functions with the suffix @M@ are general functions that work on monadic
1824-- arguments. The corresponding functions without the suffix @M@ work on pure
1925-- arguments and can in general be derived from their monadic versions but are
src/Streamly/Tutorial.hs+165−99
@@ -6,40 +6,41 @@
66-- License : BSD3
77-- Maintainer : harendra.kumar@gmail.com
88--
9−-- Streamly, short for stream concurrently, combines the essence of
10−-- non-determinism, streaming and concurrency in functional programming.
11−-- Concurrent and non-concurrent applications are almost indistinguishable,
12−-- concurrency capability does not at all impact the performance of
13−-- non-concurrent case.
14−-- Streaming enables writing modular, composable and scalable applications with
15−-- ease and concurrency allows you to make them scale and perform well.
16−-- Streamly enables writing concurrent applications without being aware of
17−-- threads or synchronization. No explicit thread control is needed, where
18−-- applicable the concurrency rate is automatically controlled based on the
19−-- demand by the consumer. However, combinators are provided to fine tune the
20−-- concurrency control.
9+-- Streamly is a general computing framework based on streaming IO. The IO
10+-- monad and pure lists are a special case of streamly. On one hand, streamly
11+-- extends the lists of pure values to lists of monadic actions, on the other
12+-- hand it extends the IO monad with concurrrent non-determinism. In simple
13+-- imperative terms we can say that streamly extends the IO monad with @for@
14+-- loops and nested @for@ loops with concurrency support. You can understand
15+-- this analogy better once you can go through this tutorial.
16+--
17+-- Streaming in general enables writing modular, composable and scalable
18+-- applications with ease, and concurrency allows you to make them scale and
19+-- perform well. Streamly enables writing scalable concurrent applications
20+-- without being aware of threads or synchronization. No explicit thread
21+-- control is needed, where applicable the concurrency rate is automatically
22+-- controlled based on the demand by the consumer. However, combinators can be
23+-- used to fine tune the concurrency control.
24+--
2125-- Streaming and concurrency together enable expressing reactive applications
22−-- conveniently. See "Streamly.Examples" for a simple SDL based FRP example.
23−--
24−-- Streamly streams are very much like the Haskell lists and most of the
25−-- functions that work on lists have a counterpart that works on streams.
26−-- However, streamly streams can be generated, consumed or combined
27−-- concurrently. In this tutorial we will go over the basic concepts and how to
28−-- use the library. The documentation of @Streamly@ module has more details on
29−-- core APIs. For more APIs for constructing, folding, filtering, mapping and
30−-- zipping etc. see the documentation of "Streamly.Prelude" module. For
31−-- examples and other ways to use the library see the module
32−-- "Streamly.Examples" as well.
26+-- conveniently. See the @CirclingSquare@ example in the examples directory for
27+-- a simple SDL based FRP example. To summarize, streamly provides a unified
28+-- computing framework for streaming, non-determinism and functional reactive
29+-- programming in an elegant and simple API that is a natural extension of pure
30+-- lists to monadic streams.
31+--
32+-- In this tutorial we will go over the basic concepts and how to
33+-- use the library. See the last section for further reading resources.
3334
3435module Streamly.Tutorial
3536 (
3637 -- * Streams
3738 -- $streams
3839
39− -- ** Generating Streams
40+ -- * Generating Streams
4041 -- $generating
4142
42− -- ** Eliminating Streams
43+ -- * Eliminating Streams
4344 -- $eliminating
4445
4546 -- * Transforming Streams
@@ -125,6 +126,9 @@ module Streamly.Tutorial
125126
126127 -- * Comparison with Existing Packages
127128 -- $comparison
129+
130+ -- * Where to go next?
131+ -- $furtherReading
128132 )
129133where
130134
@@ -138,12 +142,43 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
138142
139143-- $streams
140144--
141−-- Streamly provides many different stream types depending on the desired
142−-- composition style. The simplest type is 'Stream', @Stream a@ represents a
143−-- stream of elements of type @a@ in IO monad. Streams are very much like lists
144−-- the only difference is that they are lists of monadic actions instead of
145−-- pure values. We can call them monadic lists. You can apply all the
146−-- constructs related to lists to monadic streams as well.
145+-- The way a list represents a sequence of pure values, a stream represents a
146+-- sequence of monadic actions. The monadic stream API offered by Streamly is
147+-- very close to the Haskell "Prelude" pure lists' API, it can be considered as a
148+-- natural extension of lists to monadic actions. Streamly streams provide
149+-- concurrent composition and merging of streams. It can be considered as a
150+-- concurrent list transformer. In contrast to the "Prelude" lists, merging or
151+-- appending streams of arbitrary length is scalable and inexpensive.
152+--
153+-- The basic stream type is 'Stream', it represents a sequence of IO actions,
154+-- and is a 'Monad'. The type 'StreamT' is a monad transformer that can
155+-- represent a sequence of actions in an arbitrary monad. The type 'Stream' is
156+-- in fact a synonym for @StreamT IO@. There are a few more types similar to
157+-- 'StreamT' that all represent streams and differ only in the 'Semigroup',
158+-- 'Applicative' and 'Monad' compositions of the streams. 'Stream' and
159+-- 'Costream' types compose serially whereas 'Coparallel' and 'Parallel' types
160+-- compose concurrently. All these types can be freely inter-converted using
161+-- type combinators without any cost. You can freely switch to any type of
162+-- composition at any point in the program. When no type annotation or
163+-- explicit stream type combinators are used, the default stream type is
164+-- inferred as 'Stream'.
165+--
166+-- The 'Stream' monad is almost a drop in replacement for the 'IO' monad, IO
167+-- monad is a special case of the 'Stream' monad; IO monad represents a single
168+-- IO action whereas the 'Stream' monad represents a sequence of IO actions.
169+-- The only change you need to make to go from 'IO' to 'Stream' is to use
170+-- 'runStream' to run the monad and to prefix the IO actions with 'once'. You
171+-- can also use 'liftIO' instead of 'once'. In fact, if you use liftIO you can
172+-- switch from 'Stream' to IO monad by simply removing the 'runStream'
173+-- function; no other changes are needed unless you have used some stream
174+-- specific composition or combinators.
175+--
176+-- Similarly, the 'Stream' type is almost a drop in replacement for pure lists,
177+-- pure lists are a special case of monadic streams. If you use 'nil' in place
178+-- of '[]' and '|:' in place ':' you can replace a list with a 'Stream'. The
179+-- only difference is that the elements must be monadic type and to operate on
180+-- the streams we must use the corresponding functions from "Streamly.Prelude"
181+-- instead of using the base "Prelude".
147182
148183-- $monadtransformers
149184--
@@ -161,22 +196,31 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
161196-- singleton stream use 'lift' and to lift from an IO action use 'liftIO'.
162197--
163198-- @
164−-- > runStream $ liftIO $ putStrLn "Hello world!"
199+-- > 'runStream' $ liftIO $ putStrLn "Hello world!"
165200-- Hello world!
166−-- > runStream $ lift $ putStrLn "Hello world!"
201+-- > 'runStream' $ lift $ putStrLn "Hello world!"
167202-- Hello world!
168203-- @
169204--
170205
171206-- $generating
172207--
208+-- We will assume the following imports in this tutorial. Go ahead, fire up a
209+-- GHCi session and import these lines to start playing.
210+--
211+-- @
212+-- > import "Streamly"
213+-- > import "Streamly.Prelude" (|:)
214+-- > import qualified "Streamly.Prelude" as S
215+-- @
216+--
173217-- 'nil' represents an empty stream and 'consM' or its operator form '|:' adds
174218-- a monadic action at the head of the stream.
175219--
176220-- @
177−-- > toList nil
221+-- > S.'toList' S.'nil'
178222-- []
179−-- > toList $ getLine |: getLine |: nil
223+-- > S.'toList' $ 'getLine' |: 'getLine' |: S.'nil'
180224-- hello
181225-- world
182226-- ["hello","world"]
@@ -186,9 +230,9 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
186230-- singleton stream from a monadic action use 'once'.
187231--
188232-- @
189−-- > toList $ pure 1
233+-- > S.'toList' $ 'pure' 1
190234-- [1]
191−-- > toList $ once getLine
235+-- > S.'toList' $ S.'once' 'getLine'
192236-- hello
193237-- ["hello"]
194238-- @
@@ -197,9 +241,9 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
197241-- 'fromFoldable' which is equivalent to a fold using 'cons' and 'nil':
198242--
199243-- @
200−-- > toList $ fromFoldable [1..3]
244+-- > S.'toList' $ S.'fromFoldable' [1..3]
201245-- [1,2,3]
202−-- > toList $ Prelude.foldr cons nil [1..3]
246+-- > S.'toList' $ 'Prelude.foldr' S.'cons' S.'nil' [1..3]
203247-- [1,2,3]
204248-- @
205249--
@@ -207,7 +251,7 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
207251-- right fold using 'consM' and 'nil':
208252--
209253-- @
210−-- > runStream $ Prelude.foldr (|:) nil [putStr "Hello ", putStrLn "world!"]
254+-- > 'runStream' $ 'Prelude.foldr' ('|:') S.'nil' ['putStr' "Hello ", 'putStrLn' "world!"]
211255-- Hello world!
212256-- @
213257--
@@ -236,7 +280,7 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
236280-- forever:
237281--
238282-- @
239−-- > runStream $ S.repeatM getLine & S.mapM putStrLn
283+-- > 'runStream' $ S.'repeatM' getLine & S.'mapM' putStrLn
240284-- @
241285--
242286-- The following code snippet reads lines from standard input, filters blank
@@ -244,19 +288,19 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
244288-- numbers them and prints them:
245289--
246290-- @
247−-- import Streamly
248−-- import qualified Streamly.Prelude as S
291+-- import "Streamly"
292+-- import qualified "Streamly.Prelude" as S
249293-- import Data.Char (toUpper)
250294-- import Data.Function ((&))
251295--
252−-- main = runStream $
253−-- S.repeatM getLine
254−-- & S.filter (not . null)
255−-- & S.drop 1
256−-- & S.take 2
296+-- main = 'runStream' $
297+-- S.'repeatM' getLine
298+-- & S.'filter' (not . null)
299+-- & S.'drop' 1
300+-- & S.'take' 2
257301-- & fmap (map toUpper)
258−-- & S.zipWith (\\n s -> show n ++ " " ++ s) (S.fromFoldable [1..])
259−-- & S.mapM putStrLn
302+-- & S.'zipWith' (\\n s -> show n ++ " " ++ s) (S.'fromFoldable' [1..])
303+-- & S.'mapM' putStrLn
260304-- @
261305
262306-- $semigroup
@@ -279,10 +323,11 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
279323--
280324-- @
281325-- import "Streamly"
282−-- import "Streamly.Prelude"(once)
326+-- import "Streamly.Prelude" (nil)
327+-- import qualified "Streamly.Prelude" as S
283328-- import Control.Concurrent
284329--
285−-- delay n = once $ do
330+-- delay n = S.'once' $ do
286331-- threadDelay (n * 1000000)
287332-- tid \<- myThreadId
288333-- putStrLn (show tid ++ ": Delay " ++ show n)
@@ -477,7 +522,7 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
477522--
478523-- @
479524-- main = 'runStream' . 'coparallely' $ traced (sqrt 9) '<>' traced (sqrt 16) '<>' traced (sqrt 25)
480−-- where traced m = once (myThreadId >>= print) >> return m
525+-- where traced m = S.'once' (myThreadId >>= print) >> return m
481526-- @
482527-- @
483528-- ThreadId 40
@@ -538,7 +583,7 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
538583--
539584-- @
540585-- import "Streamly"
541−-- import Streamly.Prelude (once)
586+-- import qualified Streamly.Prelude as S
542587-- import Network.HTTP.Simple
543588--
544589-- main = 'runStream' . 'parallely' $ google \<> bing \<> duckduckgo
@@ -546,7 +591,7 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
546591-- google = get "https://www.google.com/search?q=haskell"
547592-- bing = get "https://www.bing.com/search?q=haskell"
548593-- duckduckgo = get "https://www.duckduckgo.com/?q=haskell"
549−-- get s = once (httpNoBody (parseRequest_ s) >> putStrLn (show s))
594+-- get s = S.'once' (httpNoBody (parseRequest_ s) >> putStrLn (show s))
550595-- @
551596--
552597-- In the following example we can see that the fairly parallel composition
@@ -611,15 +656,15 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
611656--
612657-- @
613658-- import "Streamly"
614−-- import "Streamly.Prelude"
659+-- import qualified "Streamly.Prelude" as S
615660-- import Control.Concurrent
616661--
617662-- main = do
618−-- runStream $ parallely $ foldMap delay [1..10]
619−-- runStream $ 'foldWith' parallel (map delay [1..10])
620−-- runStream $ 'foldMapWith' parallel delay [1..10]
621−-- runStream $ 'forEachWith' parallel [1..10] delay
622−-- where delay n = once $ threadDelay (n * 1000000) >> print n
663+-- 'runStream' $ 'parallely' $ foldMap delay [1..10]
664+-- 'runStream' $ 'foldWith' 'parallel' (map delay [1..10])
665+-- 'runStream' $ 'foldMapWith' 'parallel' delay [1..10]
666+-- 'runStream' $ 'forEachWith' 'parallel' [1..10] delay
667+-- where delay n = S.'once' $ threadDelay (n * 1000000) >> print n
623668-- @
624669
625670-- $nesting
@@ -665,10 +710,10 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
665710--
666711-- @
667712-- import "Streamly"
668−-- import "Streamly.Prelude"
713+-- import qualified "Streamly.Prelude" as S
669714--
670715-- main = 'runStream' $ do
671−-- x <- 'fromFoldable' [3,2,1]
716+-- x <- S.'fromFoldable' [3,2,1]
672717-- delay x
673718-- @
674719-- @
@@ -687,8 +732,9 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
687732--
688733-- @
689734-- import "Streamly"
735+-- import qualified "Streamly.Prelude" as S
690736--
691−-- main = 'runStream' $ forever $ once getLine >>= once . putStrLn
737+-- main = 'runStream' $ forever $ S.once getLine >>= S.once . putStrLn
692738-- @
693739--
694740-- When multiple streams are composed using this style they nest in a DFS
@@ -698,12 +744,12 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
698744--
699745-- @
700746-- import "Streamly"
701−-- import "Streamly.Prelude"
747+-- import qualified "Streamly.Prelude" as S
702748--
703749-- main = 'runStream' $ do
704−-- x <- 'fromFoldable' [1,2]
705−-- y <- 'fromFoldable' [3,4]
706−-- once $ putStrLn $ show (x, y)
750+-- x <- S.'fromFoldable' [1,2]
751+-- y <- S.'fromFoldable' [3,4]
752+-- S.'once' $ putStrLn $ show (x, y)
707753-- @
708754-- @
709755-- (1,3)
@@ -757,12 +803,12 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
757803--
758804-- @
759805-- import "Streamly"
760−-- import "Streamly.Prelude"
806+-- import qualified "Streamly.Prelude" as S
761807--
762808-- main = 'runStream' . 'coparallely' $ do
763−-- x <- 'fromFoldable' [1,2]
764−-- y <- 'fromFoldable' [3,4]
765−-- once $ putStrLn $ show (x, y)
809+-- x <- S.'fromFoldable' [1,2]
810+-- y <- S.'fromFoldable' [3,4]
811+-- S.'once' $ putStrLn $ show (x, y)
766812-- @
767813-- @
768814-- (1,3)
@@ -782,12 +828,12 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
782828--
783829-- @
784830-- import "Streamly"
785−-- import "Streamly.Prelude"
831+-- import qualified "Streamly.Prelude" as S
786832--
787833-- main = 'runStream' . 'costreamly' $ do
788−-- x <- 'fromFoldable' [1,2]
789−-- y <- 'fromFoldable' [3,4]
790−-- once $ putStrLn $ show (x, y)
834+-- x <- S.'fromFoldable' [1,2]
835+-- y <- S.'fromFoldable' [3,4]
836+-- S.once $ putStrLn $ show (x, y)
791837-- @
792838-- @
793839-- (1,3)
@@ -808,10 +854,10 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
808854--
809855-- @
810856-- import "Streamly"
811−-- import "Streamly.Prelude"
857+-- import qualified "Streamly.Prelude" as S
812858--
813859-- main = 'runStream' . 'parallely' $ do
814−-- x <- 'fromFoldable' [3,2,1]
860+-- x <- S.'fromFoldable' [3,2,1]
815861-- delay x
816862-- @
817863-- @
@@ -829,20 +875,20 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
829875--
830876-- @
831877-- import "Streamly"
832−-- import "Streamly.Prelude"
878+-- import qualified "Streamly.Prelude" as S
833879--
834880-- composed :: 'IsStream' t => t m a
835881-- composed = do
836882-- sz <- sizes
837883-- cl <- colors
838884-- sh <- shapes
839−-- once $ putStrLn $ show (sz, cl, sh)
885+-- S.'once' $ putStrLn $ show (sz, cl, sh)
840886--
841887-- where
842888--
843−-- sizes = 'fromFoldable' [1, 2, 3]
844−-- colors = 'fromFoldable' ["red", "green", "blue"]
845−-- shapes = 'fromFoldable' ["triangle", "square", "circle"]
889+-- sizes = S.'fromFoldable' [1, 2, 3]
890+-- colors = S.'fromFoldable' ["red", "green", "blue"]
891+-- shapes = S.'fromFoldable' ["triangle", "square", "circle"]
846892-- @
847893--
848894-- Now we can interpret this in whatever way we want:
@@ -865,8 +911,9 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
865911--
866912-- @
867913-- import "Streamly"
914+-- import qualified "Streamly.Prelude" as S
868915--
869−-- main = ('toList' $ fmap show $ 'fromFoldable' [1..10]) >>= print
916+-- main = (S.'toList' $ fmap show $ S.'fromFoldable' [1..10]) >>= print
870917-- @
871918--
872919-- Also see the 'mapM' and 'sequence' functions for mapping actions, in the
@@ -883,14 +930,14 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
883930--
884931-- @
885932-- import "Streamly"
886−-- import "Streamly.Prelude"
933+-- import qualified "Streamly.Prelude" as S
887934-- import Control.Concurrent
888935--
889936-- s1 = d 1 <> d 2
890937-- s2 = d 3 <> d 4
891938-- d n = delay n >> return n
892939--
893−-- main = ('toList' . 'streamly' $ (,) \<$> s1 \<*> s2) >>= print
940+-- main = (S.'toList' . 'streamly' $ (,) \<$> s1 \<*> s2) >>= print
894941-- @
895942-- @
896943-- ThreadId 36: Delay 1
@@ -906,7 +953,7 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
906953-- order but since it is serial it takes a total of 17 seconds:
907954--
908955-- @
909−-- main = ('toList' . 'costreamly' $ (,) \<$> s1 \<*> s2) >>= print
956+-- main = (S.'toList' . 'costreamly' $ (,) \<$> s1 \<*> s2) >>= print
910957-- @
911958-- @
912959-- ThreadId 36: Delay 1
@@ -922,7 +969,7 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
922969-- of 10 seconds (1 + 2 + 3 + 4):
923970--
924971-- @
925−-- main = ('toList' . 'coparallely' $ (,) \<$> s1 \<*> s2) >>= print
972+-- main = (S.'toList' . 'coparallely' $ (,) \<$> s1 \<*> s2) >>= print
926973-- @
927974-- @
928975-- ThreadId 34: Delay 1
@@ -938,7 +985,7 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
938985-- therefore takes a total of 10 seconds (1 + 2 + 3 + 4):
939986--
940987-- @
941−-- main = ('toList' . 'parallely' $ (,) \<$> s1 \<*> s2) >>= print
988+-- main = (S.'toList' . 'parallely' $ (,) \<$> s1 \<*> s2) >>= print
942989-- @
943990-- @
944991-- ThreadId 34: Delay 1
@@ -987,14 +1034,14 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
9871034--
9881035-- @
9891036-- import "Streamly"
990−-- import "Streamly.Prelude"
1037+-- import qualified "Streamly.Prelude" as S
9911038-- import Control.Concurrent
9921039--
9931040-- d n = delay n >> return n
9941041-- s1 = 'streamly' $ d 1 <> d 2
9951042-- s2 = 'streamly' $ d 3 <> d 4
9961043--
997−-- main = ('toList' . 'zipStreamly' $ (,) \<$> s1 \<*> s2) >>= print
1044+-- main = (S.'toList' . 'zipStreamly' $ (,) \<$> s1 \<*> s2) >>= print
9981045-- @
9991046--
10001047-- This takes total 10 seconds to zip, which is (1 + 2 + 3 + 4) since
@@ -1017,7 +1064,7 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
10171064--
10181065-- @
10191066-- import "Streamly"
1020−-- import "Streamly.Prelude"
1067+-- import qualified "Streamly.Prelude" as S
10211068-- import Control.Concurrent
10221069-- import System.IO (stdout, hSetBuffering, BufferMode(LineBuffering))
10231070--
@@ -1027,7 +1074,7 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
10271074--
10281075-- main = do
10291076-- hSetBuffering stdout LineBuffering
1030−-- ('toList' . 'zipParallely' $ (,) \<$> s1 \<*> s2) >>= print
1077+-- (S.'toList' . 'zipParallely' $ (,) \<$> s1 \<*> s2) >>= print
10311078-- @
10321079--
10331080-- This takes 7 seconds to zip, which is max (1,3) + max (2,4) because 1 and 3
@@ -1059,11 +1106,11 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
10591106--
10601107-- @
10611108-- import "Streamly"
1062−-- import "Streamly.Prelude" (toList)
1109+-- import qualified "Streamly.Prelude" as S
10631110-- import Data.List (sum)
10641111--
10651112-- main = do
1066−-- z \<- 'toList'
1113+-- z \<- S.'toList'
10671114-- $ 'streamly' -- Serial monadic processing (sqrt below)
10681115-- $ do
10691116-- x2 \<- 'forEachWith' 'coparallel' [1..100] $ -- Concurrent @"for"@ loop
@@ -1252,10 +1299,22 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
12521299
12531300-- $comparison
12541301--
1255−-- Streamly unifies non-determinism, streaming, concurrency and FRP
1256−-- functionality that is otherwise covered by several disparate packages, and
1257−-- it does that with a surprisingly concise API. Here is a list of popular and
1258−-- well-known packages in all these areas:
1302+-- List transformers and logic programming monads also provide a product style
1303+-- composition similar to streamly, however streamly generalizes it with the
1304+-- time dimension; allowing streams to be composed in an asynchronous and
1305+-- concurrent fashion in many different ways. It also provides multiple
1306+-- alternative ways of composing streams e.g. serial, interleaved or
1307+-- concurrent.
1308+--
1309+-- This seemingly simple addition of asynchronicity and concurrency to product
1310+-- style streaming composition unifies a number of disparate abstractions into
1311+-- one powerful, concise and elegant abstraction. A wide variety of
1312+-- programming problems can be solved elegantly with this abstraction. In
1313+-- particular, it unifies three major programming domains namely
1314+-- non-deterministic (logic) programming, concurrent programming and functional
1315+-- reactive programming. In other words, you can do everything with this one
1316+-- abstraction that you could do with the popular libraries listed under these
1317+-- categories in the list below.
12591318--
12601319-- @
12611320-- +-----------------+----------------+
@@ -1318,3 +1377,10 @@ import Control.Monad.Trans.Class (MonadTrans (lift))
13181377-- similar to @dunai@. dunai was designed from a FRP perspective and streamly
13191378-- was originally designed from a concurrency perspective. However, both have
13201379-- similarity at the core.
1380+
1381+-- $furtherReading
1382+--
1383+-- * Read the documentation of "Streamly" module
1384+-- * Read the documentation of "Streamly.Prelude" module
1385+-- * See the examples in the "examples" directory of the package
1386+-- * See the tests in the "test" directory of the package
streamly.cabal+42−15
@@ -2,22 +2,49 @@ name: streamly
22version: 0.1.2
33synopsis: Beautiful Streaming, Concurrent and Reactive Composition
44description:
5− Streamly is a monad transformer unifying non-determinism
6− (<https://hackage.haskell.org/package/list-t list-t>\/<https://hackage.haskell.org/package/logict logict>),
7− concurrency (<https://hackage.haskell.org/package/async async>),
8− streaming (<https://hackage.haskell.org/package/conduit conduit>\/<https://hackage.haskell.org/package/pipes pipes>),
9− and FRP (<https://hackage.haskell.org/package/Yampa Yampa>\/<https://hackage.haskell.org/package/reflex reflex>)
10− functionality in a concise and intuitive API.
11− High level concurrency makes concurrent applications almost indistinguishable
12− from non-concurrent ones. By changing a single combinator you can control
13− whether the code runs serially or concurrently. It naturally integrates
14− concurrency with streaming rather than adding it as an afterthought.
15− Moreover, it interworks with the popular streaming libraries.
5+ Streamly, short for streaming concurrently, is a simple yet powerful
6+ streaming library with concurrent merging and concurrent nested looping
7+ support. A stream is just like a list except that it is a list of monadic
8+ actions rather than pure values. Streamly streams can be generated,
9+ consumed, combined, or transformed serially or concurrently. We can loop over
10+ a stream serially or concurrently. We can also have serial or concurrent
11+ nesting of loops. For those familiar with list transformer concept streamly
12+ is a concurrent list transformer. Streamly uses standard composition
13+ abstractions. Concurrent composition is just the same as serial composition
14+ except that we use a simple combinator to request a concurrent composition
15+ instead of serial. The programmer does not have to be aware of threads,
16+ locking or synchronization to write scalable concurrent programs.
1617 .
17− See the README for an overview and the haddock documentation for full
18− reference. It is recommended to read the comprehensive tutorial module
19− "Streamly.Tutorial" first. Also see "Streamly.Examples" for some working
20− examples.
18+ Streamly provides functionality that is equivalent to streaming libraries
19+ like <https://hackage.haskell.org/package/pipes pipes> and
20+ <https://hackage.haskell.org/package/conduit conduit> but with a simple list
21+ like API. The streaming API of streamly is close to the monadic streams API
22+ of the <https://hackage.haskell.org/package/vector vector> package and
23+ similar in concept to the
24+ <https://hackage.haskell.org/package/streaming streaming> package. In
25+ addition to the streaming functionality, streamly subsumes the functionality
26+ of list transformer libraries like @pipes@ or
27+ <https://hackage.haskell.org/package/list-t list-t> and also the logic
28+ programming library <https://hackage.haskell.org/package/logict logict>. On
29+ the concurrency side, it subsumes the functionality of the
30+ <https://hackage.haskell.org/package/async async> package. Because it
31+ supports streaming with concurrency we can write FRP applications similar in
32+ concept to <https://hackage.haskell.org/package/Yampa Yampa> or
33+ <https://hackage.haskell.org/package/reflex reflex>.
34+ .
35+ Streamly has excellent performance, see
36+ <https://github.com/composewell/streaming-benchmarks streaming-benchmarks>
37+ for a comparison of popular streaming libraries on micro-benchmarks. For
38+ file IO, currently the library provides only one API to stream the lines in
39+ the file as Strings. Future versions will provide better streaming file IO
40+ options. Streamly interworks with the popular streaming libraries, see the
41+ interworking section in "Streamly.Tutorial".
42+ .
43+ Where to find more information:
44+ .
45+ * @README@ shipped with the package for a quick overview
46+ * "Streamly.Tutorial" module in the haddock documentation for a detailed introduction
47+ * @examples@ directory in the package for some simple practical examples
2148
2249homepage: https://github.com/composewell/streamly
2350bug-reports: https://github.com/composewell/streamly/issues