@@ -7,70 +7,113 @@
7[](https://circleci.com/gh/composewell/streamly/tree/master)
7[](https://circleci.com/gh/composewell/streamly/tree/master)
8[](https://coveralls.io/github/composewell/streamly?branch=master)
8[](https://coveralls.io/github/composewell/streamly?branch=master)
10−## Stream`ing` `Concurrent`ly
12−Streamly, short for streaming concurrently, provides monadic streams, with a
11+## What is streamly?
13−simple API, almost identical to standard lists and vector, and an in-built
12+
14−support for concurrency. By using stream-style combinators on stream
13+Haskell lists provide an elegant data flow programming model for pure
15−composition, streams can be generated, merged, chained, mapped, zipped, and
14+computations. With lists, we can express pure computations using composable
16−consumed concurrently – providing a generalized high level programming
15+stream operations like `:`, `unfold`, `map`, `filter`, `zip` and `fold`.
17−framework unifying streaming and concurrency. Controlled concurrency allows
16+Streamly extends this data flow programming model to concurrent monadic
18−even infinite streams to be evaluated concurrently. Concurrency is auto scaled
17+computations. We use the same list primitives except that we deal with lists of
19−based on feedback from the stream consumer. The programmer does not have to be
18+monadic computations (streams) instead of lists of pure computations.
20−aware of threads, locking or synchronization to write scalable concurrent
19+
21−programs.
20+We express concurrency declaratively with the list primitives themselves
21+without having to know any low level notions of concurrency like threads,
22+locking or synchronization. Concurrency is automatically scaled up or down
23+based on the need of the application, so that we can say goodbye to managing
24+thread pools and associated sizing issues as well. That's what true fearless
25+concurrency is. Streamly can be thought of as concurrent monadic lists, if you
26+know Haskell lists then you already know how to use streamly.
28+## Where to use streamly?
30+Everywhere. The answer to this question would be similar to the answer to -
31+"Where do I use Haskell lists?". Streamly generalizes lists to monadic
32+streams, and IO to non-deterministic stream composition with concurrency. The
33+`IO` monad becomes a special case of streamly, if we use single element streams
34+the behavior of streamly is identical to the IO monad. It can be replaced with
35+streamly by just prefixing IO actions with `liftIO`, without any loss of
36+performance. Pure lists too become a special case of streamly; if we use the
37+`Identity` monad, streams turn into pure lists. Non-concurrent programs become
38+a special case of concurrent ones, by just adding a combinator, a non-concurrent
39+program becomes concurrent.
41+We can say that streamly is a superset of lists and IO, with builtin
42+concurrency. If you want to write a program that involves IO, concurrent or
43+not, then you can just use streamly as the base monad, heck, you could even use
44+streamly for pure computations, as streamly performs at par with pure lists or
47+If you need convincing for using streaming or data flow programming paradigm
48+itself then try to answer this question - why do we use lists? It boils down to
49+why we use functional programming in the first place, and Haskell is successful
50+in enforcing this for pure computations, but not for monadic computations. In
51+the absence of a standard, easy to use or enforced data flow programming
52+library for monadic computations, and the IO monad providing an escape hatch to
53+an imperative model, we just love to fall into the imperative trap.
57+Here is an IO monad code to list a directory recursively:
60+import Control.Monad.IO.Class (liftIO)
61+import Path.IO (listDir, getCurrentDir) -- from path-io package
63+listDirRecursive = getCurrentDir >>= readdir
66+ (dirs, files) <- listDir dir
67+ liftIO $ mapM_ putStrLn
68+ $ map show dirs ++ map show files
23−The basic streaming functionality of streamly is equivalent to that provided by
72+This is your usual IO monad code, with no streamly specific code whatsoever.
24−streaming libraries like
73+This is how you can run this:
25−[vector](https://hackage.haskell.org/package/vector),
26−[streaming](https://hackage.haskell.org/package/streaming),
27−[pipes](https://hackage.haskell.org/package/pipes), and
28−[conduit](https://hackage.haskell.org/package/conduit).
29−In addition to providing streaming functionality, streamly subsumes
30−the functionality of list transformer libraries like `pipes` or
31−[list-t](https://hackage.haskell.org/package/list-t), and also the logic
32−programming library [logict](https://hackage.haskell.org/package/logict). On
33−the concurrency side, it subsumes the functionality of the
34−[async](https://hackage.haskell.org/package/async) package, and provides even
35−higher level concurrent composition. Because it supports
36−streaming with concurrency we can write FRP applications similar in concept to
37−[Yampa](https://hackage.haskell.org/package/Yampa) or
38−[reflex](https://hackage.haskell.org/package/reflex).
40−Why use streamly?
75+``` haskell
77+main = listDirRecursive
42− * _Simplicity_: Simple list like streaming API, if you know how to use lists
80+And, this is how you can run exactly the same code using streamly with
43− then you know how to use streamly. This library is built with simplicity
81+lookahead style concurrency, the only difference is that this time multiple
44− and ease of use as a design goal.
82+directories are read concurrently:
45− * _Concurrency_: Simple, powerful, and scalable concurrency. Concurrency is
83+
46− built-in, and not intrusive, concurrent programs are written exactly the
84+``` haskell
47− same way as non-concurrent ones.
85+import Streamly (runStream, aheadly)
48− * _Generality_: Unifies functionality provided by several disparate packages
86+
49− (streaming, concurrency, list transformer, logic programming, reactive
87+main :: IO ()
50− programming) in a concise API.
88+main = runStream $ aheadly $ listDirRecursive
51− * _Performance_: Streamly is designed for high performance. It employs stream
89+```
52− fusion optimizations for best possible performance. Serial peformance is
90+
53− equivalent to the venerable `vector` library in most cases and even better
91+Isn't that magical? What's going on here? Streamly does not introduce any new
54− in some cases. Concurrent performance is unbeatable. See
92+abstractions, it just uses the standard abstractions like `Semigroup` or
93+`Monoid` to combine monadic streams concurrently, the same way lists combine a
94+sequence of pure values non-concurrently. Therefore, the `foldMap` in the code
95+above turns into a concurrent monoidal composition of a stream of `readdir`
98+## How does it perform?
100+Providing monadic streaming and high level declarative concurrency does not
101+mean that `streamly` compromises with performance in any way. The
102+non-concurrent performance of `streamly` competes with lists and the `vector`
103+library. The concurrent performance is as good as it gets, see [concurrency
55− [streaming-benchmarks](https://github.com/composewell/streaming-benchmarks)
104+benchmarks](https://github.com/composewell/concurrency-benchmarks) for detailed
56− for a comparison of popular streaming libraries on micro-benchmarks.
105+performance results and a comparison with the `async` package.
58The following chart shows a summary of the cost of key streaming operations
107The following chart shows a summary of the cost of key streaming operations
59−processing a million elements. The timings for streamly and vector are in the
108+processing a million elements. The timings for `streamly` and `vector` are in
60−600-700 microseconds range and therefore can barely be seen in the graph.
109+the 600-700 microseconds range and therefore can barely be seen in the graph.
110+For more details, see [streaming
111+benchmarks](https://github.com/composewell/streaming-benchmarks).
62
113
64## Streaming Pipelines
115## Streaming Pipelines
66−Unlike `pipes` or `conduit` and like `vector` and `streaming`, `streamly`
67−composes stream data instead of stream processors (functions). A stream is
68−just like a list and is explicitly passed around to functions that process the
69−stream. Therefore, no special operator is needed to join stages in a streaming
70−pipeline, just the standard function application (`$`) or reverse function
71−application (`&`) operator is enough. Combinators are provided in
72−`Streamly.Prelude` to transform or fold streams.
74The following snippet provides a simple stream composition example that reads
117The following snippet provides a simple stream composition example that reads
75numbers from stdin, prints the squares of even numbers and exits if an even
118numbers from stdin, prints the squares of even numbers and exits if an even
76number more than 9 is entered.
119number more than 9 is entered.
@@ -89,6 +132,14 @@ main = runStream $
89 & S.mapM print
132 & S.mapM print
135+Unlike `pipes` or `conduit` and like `vector` and `streaming`, `streamly`
136+composes stream data instead of stream processors (functions). A stream is
137+just like a list and is explicitly passed around to functions that process the
138+stream. Therefore, no special operator is needed to join stages in a streaming
139+pipeline, just the standard function application (`$`) or reverse function
140+application (`&`) operator is enough. Combinators are provided in
141+`Streamly.Prelude` to transform or fold streams.
92## Concurrent Stream Generation
143## Concurrent Stream Generation
94Monadic construction and generation functions e.g. `consM`, `unfoldrM`,
145Monadic construction and generation functions e.g. `consM`, `unfoldrM`,
@@ -270,28 +321,6 @@ main = do
273−Of course, the actions running in parallel could be arbitrary IO actions. For
274−example, to concurrently list the contents of a directory tree recursively:
277−import Path.IO (listDir, getCurrentDir)
279−import qualified Streamly.Prelude as S
281−main = runStream $ aheadly $ getCurrentDir >>= readdir
282− where readdir d = do
283− (dirs, files) <- S.yieldM $ listDir d
284− S.yieldM $ mapM_ putStrLn $ map show files
285− -- read the subdirs concurrently, (<>) is concurrent
286− foldMap readdir dirs
289−In the above examples we do not think in terms of threads, locking or
290−synchronization, rather we think in terms of what can run in parallel, the rest
291−is taken care of automatically. When using `aheadly` the programmer does
292−not have to worry about how many threads are to be created, they are
293−automatically adjusted based on the demand of the consumer.
295The concurrency facilities provided by streamly can be compared with
324The concurrency facilities provided by streamly can be compared with
296[OpenMP](https://en.wikipedia.org/wiki/OpenMP) and
325[OpenMP](https://en.wikipedia.org/wiki/OpenMP) and
297[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative
326[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative
@@ -328,6 +357,56 @@ for a console based FRP game example and
328[CirclingSquare.hs](https://github.com/composewell/streamly/tree/master/examples/CirclingSquare.hs)
357[CirclingSquare.hs](https://github.com/composewell/streamly/tree/master/examples/CirclingSquare.hs)
329for an SDL based animation example.
358for an SDL based animation example.
362+Streamly, short for streaming concurrently, provides monadic streams, with a
363+simple API, almost identical to standard lists, and an in-built
364+support for concurrency. By using stream-style combinators on stream
365+composition, streams can be generated, merged, chained, mapped, zipped, and
366+consumed concurrently – providing a generalized high level programming
367+framework unifying streaming and concurrency. Controlled concurrency allows
368+even infinite streams to be evaluated concurrently. Concurrency is auto scaled
369+based on feedback from the stream consumer. The programmer does not have to be
370+aware of threads, locking or synchronization to write scalable concurrent
373+Streamly is a programmer first library, designed to be useful and friendly to
374+programmers for solving practical problems in a simple and concise manner. Some
375+key points in favor of streamly are:
377+ * _Simplicity_: Simple list like streaming API, if you know how to use lists
378+ then you know how to use streamly. This library is built with simplicity
379+ and ease of use as a design goal.
380+ * _Concurrency_: Simple, powerful, and scalable concurrency. Concurrency is
381+ built-in, and not intrusive, concurrent programs are written exactly the
382+ same way as non-concurrent ones.
383+ * _Generality_: Unifies functionality provided by several disparate packages
384+ (streaming, concurrency, list transformer, logic programming, reactive
385+ programming) in a concise API.
386+ * _Performance_: Streamly is designed for high performance. It employs stream
387+ fusion optimizations for best possible performance. Serial peformance is
388+ equivalent to the venerable `vector` library in most cases and even better
389+ in some cases. Concurrent performance is unbeatable. See
390+ [streaming-benchmarks](https://github.com/composewell/streaming-benchmarks)
391+ for a comparison of popular streaming libraries on micro-benchmarks.
393+The basic streaming functionality of streamly is equivalent to that provided by
394+streaming libraries like
395+[vector](https://hackage.haskell.org/package/vector),
396+[streaming](https://hackage.haskell.org/package/streaming),
397+[pipes](https://hackage.haskell.org/package/pipes), and
398+[conduit](https://hackage.haskell.org/package/conduit).
399+In addition to providing streaming functionality, streamly subsumes
400+the functionality of list transformer libraries like `pipes` or
401+[list-t](https://hackage.haskell.org/package/list-t), and also the logic
402+programming library [logict](https://hackage.haskell.org/package/logict). On
403+the concurrency side, it subsumes the functionality of the
404+[async](https://hackage.haskell.org/package/async) package, and provides even
405+higher level concurrent composition. Because it supports
406+streaming with concurrency we can write FRP applications similar in concept to
407+[Yampa](https://hackage.haskell.org/package/Yampa) or
408+[reflex](https://hackage.haskell.org/package/reflex).
331## Further Reading
410## Further Reading
333For more information, see:
412For more information, see: