@@ -6,38 +6,79 @@
66[](https://coveralls.io/github/composewell/streamly?branch=master)
77
88## Stream`ing` `Concurrent`ly
9−Streamly unifies concurrency and streaming in a single monad transformer with a
10−concise and simple API. It provides two ways to combine streams, a monadic
11−product composition as well as the standard pipelined composition provided by
12−streaming libraries. A natural extension of regular monadic composition to
13−streaming and concurrency makes it intuitive and concise with almost universal
14−application. You can write concurrent or non-concurrent applications using
15−simple IO, logic programming, streaming IO or reactive programming (FRP) using
16−the same API. You can also think about it as representing concurrent and
17−composable state machines in imperative terms. It unifies the core
18−functionality provided by [async](https://hackage.haskell.org/package/async),
19−[logict](https://hackage.haskell.org/package/logict),
20−[list-t](https://hackage.haskell.org/package/list-t),
21−[conduit](https://hackage.haskell.org/package/conduit)\/[pipes](https://hackage.haskell.org/package/pipes),
22−[Yampa](https://hackage.haskell.org/package/Yampa)\/[reflex](https://hackage.haskell.org/package/reflex)
23−under one type and API. It interworks with the existing streaming libraries.
9+
10+Streamly is a monad transformer unifying non-determinism
11+([list-t](https://hackage.haskell.org/package/list-t)/[logict](https://hackage.haskell.org/package/logict)),
12+concurrency ([async](https://hackage.haskell.org/package/async)),
13+streaming ([conduit](https://hackage.haskell.org/package/conduit)\/[pipes](https://hackage.haskell.org/package/pipes)),
14+and FRP ([Yampa](https://hackage.haskell.org/package/Yampa)\/[reflex](https://hackage.haskell.org/package/reflex))
15+functionality in a concise and intuitive API.
16+High level concurrency makes concurrent applications almost indistinguishable
17+from non-concurrent ones. By changing a single combinator you can control
18+whether the code runs serially or concurrently. It naturally integrates
19+concurrency with streaming rather than adding it as an afterthought.
20+Moreover, it interworks with the popular streaming libraries.
21+
22+See the haddock documentation for full reference. It is recommended to read
23+the comprehensive tutorial module `Streamly.Tutorial` first. Also see
24+`Streamly.Examples` for some working examples.
25+
26+## Non-determinism
27+
28+The monad instance composes like a list monad.
29+
30+``` haskell
31+loops = $ do
32+ x <- each [1,2]
33+ y <- each [3,4]
34+ liftIO $ putStrLn $ show (x, y)
35+
36+main = runStreaming $ serially $ loops
37+```
38+```
39+(1,3)
40+(1,4)
41+(2,3)
42+(2,4)
43+```
2444
2545## Magical Concurrency
26−Streamly provides high level concurrency primitives (higher level than async)
27−and hides the low level concurrency details completely from the programmer.
28−Concurrency can be used with ease in applicative or monadic contexts. The
29−programmer just expresses whether a task can run in parallel with another.
30−Threads, synchronization and concurrency rate control are handled
31−automatically under the hood. The concurrency facilities provided by streamly
32−can be compared with [OpenMP](https://en.wikipedia.org/wiki/OpenMP) and
33−[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative
34−expression. Concurrency support does not compromise performance in
35−non-concurrent cases, the performance of the library is at par or better than
36−most of the existing streaming libraries.
3746
38−## Example
39−Here is a simple example to concurrently and recursively list the contents of a
40−directory tree:
47+To run the above code with demand-driven concurrency i.e. each iteration in the
48+loops can run concurrently depending on the consumer rate:
49+
50+``` haskell
51+main = runStreaming $ asyncly $ loops
52+```
53+
54+To run it with full parallelism irrespective of demand:
55+
56+``` haskell
57+main = runStreaming $ parallely $ loops
58+```
59+
60+To run it serially but interleaving the outer and inner loop iterations:
61+
62+``` haskell
63+main = runStreaming $ interleaving $ loops
64+```
65+
66+You can fold multiple streams or IO actions using parallel combinators like
67+`<|`, `<|>`. For example, to concurrently generate the squares and then
68+concurrently sum the square roots of all combinations:
69+
70+``` haskell
71+main = do
72+ print $ sum $ asyncly $ do
73+ -- Squaring is concurrent (<|)
74+ x2 <- forEachWith (<|) [1..100] $ \x -> return $ x * x
75+ y2 <- forEachWith (<|) [1..100] $ \y -> return $ y * y
76+ -- sqrt is concurrent (asyncly)
77+ return $ sqrt (x2 + y2)
78+```
79+
80+Of course, the actions running in parallel could be arbitrary IO actions. To
81+concurrently list the contents of a directory tree recursively:
4182
4283``` haskell
4384import Path.IO (listDir, getCurrentDir)
@@ -47,10 +88,85 @@ main = runStreaming $ serially $ getCurrentDir >>= readdir
4788 where readdir d = do
4889 (dirs, files) <- lift $ listDir d
4990 liftIO $ mapM_ putStrLn $ map show files
91+ -- read the subdirs concurrently
5092 foldMapWith (<|>) readdir dirs
5193```
5294
53−See "Streamly.Tutorial" and "Streamly.Examples" for more details.
95+In the above examples we do not think in terms of threads, locking or
96+synchronization, rather we think in terms of what can run in parallel, the rest
97+is taken care of automatically. With `asyncly` and `<|` the programmer does not
98+have to worry about how many threads are to be created they are automatically
99+adjusted based on the demand of the consumer.
100+
101+The concurrency facilities provided by streamly can be compared with
102+[OpenMP](https://en.wikipedia.org/wiki/OpenMP) and
103+[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative
104+expression. Concurrency support does not compromise performance in
105+non-concurrent cases, the performance of the library is at par or better than
106+most of the existing streaming libraries.
107+
108+## Streaming
109+
110+Streaming is effortless, simple and straightforward. Streamly data type behaves
111+just like a list and combinators are provided in `Streamly.Prelude` to
112+transform or fold streamly streams. Unlike other libraries and like `streaming`
113+library the combinators explicitly consume a stream and produce a stream,
114+therefore, no special operator is needed to join stream stages, just a forward
115+(`$`) or reverse (`&`) function application operator is enough.
116+
117+```haskell
118+import Streamly
119+import Streamly.Prelude as S
120+import Data.Function ((&))
121+
122+main = S.each [1..10]
123+ & fmap (+ 1)
124+ & S.drop 2
125+ & S.filter even
126+ & fmap (* 3)
127+ & S.takeWhile (< 25)
128+ & S.mapM (\x -> putStrLn ("saw " ++ show x) >> return x)
129+ & S.toList . serially
130+ >>= print
131+```
132+
133+Fold style combinators can be used to fold purely or monadically. You can also
134+use the beautiful `foldl` library for folding.
135+
136+```haskell
137+main = S.each [1..10]
138+ & serially
139+ & S.foldl (+) 0 id
140+ >>= print
141+```
142+
143+Streams can be combined together in multiple ways:
144+
145+```haskell
146+return 1 <> return 2 -- serial, combine atoms
147+S.each [1..10] <> S.each [11..20] -- serial
148+S.each [1..10] <| S.each [11..20] -- demand driven parallel
149+S.each [1..10] <=> S.each [11..20] -- serial but interleaved
150+S.each [1..10] <|> S.each [11..20] -- fully parallel
151+```
152+
153+As we have already seen streams can be combined using monadic composition in a
154+non-deterministic manner. This allows arbitrary manipulation and combining of
155+streams. See `Streamly.Examples.MergeSortedStreams` for a more complicated
156+example.
157+
158+## Reactive Programming (FRP)
159+
160+Streamly is a foundation for first class reactive programming as well by virtue
161+of integrating concurrency and streaming. See `Streamly.Examples.AcidRainGame`
162+and `Streamly.Examples.CirclingSquare` for an SDL based animation example.
163+
164+## Contributing
165+
166+The code is available under BSD-3 license [on
167+github](https://github.com/composewell/streamly). Join the [gitter
168+chat](https://gitter.im/composewell/streamly) channel for discussions. All
169+contributions are welcome!
54170
55−This library was originally inspired by the `transient`
56−package authored by Alberto G. Corona.
171+This library was originally inspired by the `transient` package authored by
172+Alberto G. Corona.