@@ -7,70 +7,113 @@
77[](https://circleci.com/gh/composewell/streamly/tree/master)
88[](https://coveralls.io/github/composewell/streamly?branch=master)
99
10−## Stream`ing` `Concurrent`ly
1110
12−Streamly, short for streaming concurrently, provides monadic streams, with a
13−simple API, almost identical to standard lists and vector, and an in-built
14−support for concurrency. By using stream-style combinators on stream
15−composition, streams can be generated, merged, chained, mapped, zipped, and
16−consumed concurrently – providing a generalized high level programming
17−framework unifying streaming and concurrency. Controlled concurrency allows
18−even infinite streams to be evaluated concurrently. Concurrency is auto scaled
19−based on feedback from the stream consumer. The programmer does not have to be
20−aware of threads, locking or synchronization to write scalable concurrent
21−programs.
11+## What is streamly?
12+
13+Haskell lists provide an elegant data flow programming model for pure
14+computations. With lists, we can express pure computations using composable
15+stream operations like `:`, `unfold`, `map`, `filter`, `zip` and `fold`.
16+Streamly extends this data flow programming model to concurrent monadic
17+computations. We use the same list primitives except that we deal with lists of
18+monadic computations (streams) instead of lists of pure computations.
19+
20+We express concurrency declaratively with the list primitives themselves
21+without having to know any low level notions of concurrency like threads,
22+locking or synchronization. Concurrency is automatically scaled up or down
23+based on the need of the application, so that we can say goodbye to managing
24+thread pools and associated sizing issues as well. That's what true fearless
25+concurrency is. Streamly can be thought of as concurrent monadic lists, if you
26+know Haskell lists then you already know how to use streamly.
27+
28+## Where to use streamly?
29+
30+Everywhere. The answer to this question would be similar to the answer to -
31+"Where do I use Haskell lists?". Streamly generalizes lists to monadic
32+streams, and IO to non-deterministic stream composition with concurrency. The
33+`IO` monad becomes a special case of streamly, if we use single element streams
34+the behavior of streamly is identical to the IO monad. It can be replaced with
35+streamly by just prefixing IO actions with `liftIO`, without any loss of
36+performance. Pure lists too become a special case of streamly; if we use the
37+`Identity` monad, streams turn into pure lists. Non-concurrent programs become
38+a special case of concurrent ones, by just adding a combinator, a non-concurrent
39+program becomes concurrent.
40+
41+We can say that streamly is a superset of lists and IO, with builtin
42+concurrency. If you want to write a program that involves IO, concurrent or
43+not, then you can just use streamly as the base monad, heck, you could even use
44+streamly for pure computations, as streamly performs at par with pure lists or
45+`vector`.
46+
47+If you need convincing for using streaming or data flow programming paradigm
48+itself then try to answer this question - why do we use lists? It boils down to
49+why we use functional programming in the first place, and Haskell is successful
50+in enforcing this for pure computations, but not for monadic computations. In
51+the absence of a standard, easy to use or enforced data flow programming
52+library for monadic computations, and the IO monad providing an escape hatch to
53+an imperative model, we just love to fall into the imperative trap.
54+
55+## Show me an example
56+
57+Here is an IO monad code to list a directory recursively:
58+
59+```haskell
60+import Control.Monad.IO.Class (liftIO)
61+import Path.IO (listDir, getCurrentDir) -- from path-io package
62+
63+listDirRecursive = getCurrentDir >>= readdir
64+ where
65+ readdir dir = do
66+ (dirs, files) <- listDir dir
67+ liftIO $ mapM_ putStrLn
68+ $ map show dirs ++ map show files
69+ foldMap readdir dirs
70+```
2271
23−The basic streaming functionality of streamly is equivalent to that provided by
24−streaming libraries like
25−[vector](https://hackage.haskell.org/package/vector),
26−[streaming](https://hackage.haskell.org/package/streaming),
27−[pipes](https://hackage.haskell.org/package/pipes), and
28−[conduit](https://hackage.haskell.org/package/conduit).
29−In addition to providing streaming functionality, streamly subsumes
30−the functionality of list transformer libraries like `pipes` or
31−[list-t](https://hackage.haskell.org/package/list-t), and also the logic
32−programming library [logict](https://hackage.haskell.org/package/logict). On
33−the concurrency side, it subsumes the functionality of the
34−[async](https://hackage.haskell.org/package/async) package, and provides even
35−higher level concurrent composition. Because it supports
36−streaming with concurrency we can write FRP applications similar in concept to
37−[Yampa](https://hackage.haskell.org/package/Yampa) or
38−[reflex](https://hackage.haskell.org/package/reflex).
72+This is your usual IO monad code, with no streamly specific code whatsoever.
73+This is how you can run this:
3974
40−Why use streamly?
75+``` haskell
76+main :: IO ()
77+main = listDirRecursive
78+```
4179
42− * _Simplicity_: Simple list like streaming API, if you know how to use lists
43− then you know how to use streamly. This library is built with simplicity
44− and ease of use as a design goal.
45− * _Concurrency_: Simple, powerful, and scalable concurrency. Concurrency is
46− built-in, and not intrusive, concurrent programs are written exactly the
47− same way as non-concurrent ones.
48− * _Generality_: Unifies functionality provided by several disparate packages
49− (streaming, concurrency, list transformer, logic programming, reactive
50− programming) in a concise API.
51− * _Performance_: Streamly is designed for high performance. It employs stream
52− fusion optimizations for best possible performance. Serial peformance is
53− equivalent to the venerable `vector` library in most cases and even better
54− in some cases. Concurrent performance is unbeatable. See
55− [streaming-benchmarks](https://github.com/composewell/streaming-benchmarks)
56− for a comparison of popular streaming libraries on micro-benchmarks.
80+And, this is how you can run exactly the same code using streamly with
81+lookahead style concurrency, the only difference is that this time multiple
82+directories are read concurrently:
83+
84+``` haskell
85+import Streamly (runStream, aheadly)
86+
87+main :: IO ()
88+main = runStream $ aheadly $ listDirRecursive
89+```
90+
91+Isn't that magical? What's going on here? Streamly does not introduce any new
92+abstractions, it just uses the standard abstractions like `Semigroup` or
93+`Monoid` to combine monadic streams concurrently, the same way lists combine a
94+sequence of pure values non-concurrently. Therefore, the `foldMap` in the code
95+above turns into a concurrent monoidal composition of a stream of `readdir`
96+computations.
97+
98+## How does it perform?
99+
100+Providing monadic streaming and high level declarative concurrency does not
101+mean that `streamly` compromises with performance in any way. The
102+non-concurrent performance of `streamly` competes with lists and the `vector`
103+library. The concurrent performance is as good as it gets, see [concurrency
104+benchmarks](https://github.com/composewell/concurrency-benchmarks) for detailed
105+performance results and a comparison with the `async` package.
57106
58107The following chart shows a summary of the cost of key streaming operations
59−processing a million elements. The timings for streamly and vector are in the
60−600-700 microseconds range and therefore can barely be seen in the graph.
108+processing a million elements. The timings for `streamly` and `vector` are in
109+the 600-700 microseconds range and therefore can barely be seen in the graph.
110+For more details, see [streaming
111+benchmarks](https://github.com/composewell/streaming-benchmarks).
61112
62113
63114
64115## Streaming Pipelines
65116
66−Unlike `pipes` or `conduit` and like `vector` and `streaming`, `streamly`
67−composes stream data instead of stream processors (functions). A stream is
68−just like a list and is explicitly passed around to functions that process the
69−stream. Therefore, no special operator is needed to join stages in a streaming
70−pipeline, just the standard function application (`$`) or reverse function
71−application (`&`) operator is enough. Combinators are provided in
72−`Streamly.Prelude` to transform or fold streams.
73−
74117The following snippet provides a simple stream composition example that reads
75118numbers from stdin, prints the squares of even numbers and exits if an even
76119number more than 9 is entered.
@@ -89,6 +132,14 @@ main = runStream $
89132 & S.mapM print
90133```
91134
135+Unlike `pipes` or `conduit` and like `vector` and `streaming`, `streamly`
136+composes stream data instead of stream processors (functions). A stream is
137+just like a list and is explicitly passed around to functions that process the
138+stream. Therefore, no special operator is needed to join stages in a streaming
139+pipeline, just the standard function application (`$`) or reverse function
140+application (`&`) operator is enough. Combinators are provided in
141+`Streamly.Prelude` to transform or fold streams.
142+
92143## Concurrent Stream Generation
93144
94145Monadic construction and generation functions e.g. `consM`, `unfoldrM`,
@@ -270,28 +321,6 @@ main = do
270321 print s
271322```
272323
273−Of course, the actions running in parallel could be arbitrary IO actions. For
274−example, to concurrently list the contents of a directory tree recursively:
275−
276−``` haskell
277−import Path.IO (listDir, getCurrentDir)
278−import Streamly
279−import qualified Streamly.Prelude as S
280−
281−main = runStream $ aheadly $ getCurrentDir >>= readdir
282− where readdir d = do
283− (dirs, files) <- S.yieldM $ listDir d
284− S.yieldM $ mapM_ putStrLn $ map show files
285− -- read the subdirs concurrently, (<>) is concurrent
286− foldMap readdir dirs
287−```
288−
289−In the above examples we do not think in terms of threads, locking or
290−synchronization, rather we think in terms of what can run in parallel, the rest
291−is taken care of automatically. When using `aheadly` the programmer does
292−not have to worry about how many threads are to be created, they are
293−automatically adjusted based on the demand of the consumer.
294−
295324The concurrency facilities provided by streamly can be compared with
296325[OpenMP](https://en.wikipedia.org/wiki/OpenMP) and
297326[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative
@@ -328,6 +357,56 @@ for a console based FRP game example and
328357[CirclingSquare.hs](https://github.com/composewell/streamly/tree/master/examples/CirclingSquare.hs)
329358for an SDL based animation example.
330359
360+## Conclusion
361+
362+Streamly, short for streaming concurrently, provides monadic streams, with a
363+simple API, almost identical to standard lists, and an in-built
364+support for concurrency. By using stream-style combinators on stream
365+composition, streams can be generated, merged, chained, mapped, zipped, and
366+consumed concurrently – providing a generalized high level programming
367+framework unifying streaming and concurrency. Controlled concurrency allows
368+even infinite streams to be evaluated concurrently. Concurrency is auto scaled
369+based on feedback from the stream consumer. The programmer does not have to be
370+aware of threads, locking or synchronization to write scalable concurrent
371+programs.
372+
373+Streamly is a programmer first library, designed to be useful and friendly to
374+programmers for solving practical problems in a simple and concise manner. Some
375+key points in favor of streamly are:
376+
377+ * _Simplicity_: Simple list like streaming API, if you know how to use lists
378+ then you know how to use streamly. This library is built with simplicity
379+ and ease of use as a design goal.
380+ * _Concurrency_: Simple, powerful, and scalable concurrency. Concurrency is
381+ built-in, and not intrusive, concurrent programs are written exactly the
382+ same way as non-concurrent ones.
383+ * _Generality_: Unifies functionality provided by several disparate packages
384+ (streaming, concurrency, list transformer, logic programming, reactive
385+ programming) in a concise API.
386+ * _Performance_: Streamly is designed for high performance. It employs stream
387+ fusion optimizations for best possible performance. Serial peformance is
388+ equivalent to the venerable `vector` library in most cases and even better
389+ in some cases. Concurrent performance is unbeatable. See
390+ [streaming-benchmarks](https://github.com/composewell/streaming-benchmarks)
391+ for a comparison of popular streaming libraries on micro-benchmarks.
392+
393+The basic streaming functionality of streamly is equivalent to that provided by
394+streaming libraries like
395+[vector](https://hackage.haskell.org/package/vector),
396+[streaming](https://hackage.haskell.org/package/streaming),
397+[pipes](https://hackage.haskell.org/package/pipes), and
398+[conduit](https://hackage.haskell.org/package/conduit).
399+In addition to providing streaming functionality, streamly subsumes
400+the functionality of list transformer libraries like `pipes` or
401+[list-t](https://hackage.haskell.org/package/list-t), and also the logic
402+programming library [logict](https://hackage.haskell.org/package/logict). On
403+the concurrency side, it subsumes the functionality of the
404+[async](https://hackage.haskell.org/package/async) package, and provides even
405+higher level concurrent composition. Because it supports
406+streaming with concurrency we can write FRP applications similar in concept to
407+[Yampa](https://hackage.haskell.org/package/Yampa) or
408+[reflex](https://hackage.haskell.org/package/reflex).
409+
331410## Further Reading
332411
333412For more information, see: