git.iko.soy/streamly
on 5cbcd5b help sign in
streamly/docs/streamly-vs-async.md
202 lines 6.9 kB history raw
1# Streamly
3Streamly is a library to make concurrent programming a joy. The venerable
4`async` package is the go to package for concurrent programming for most
5Haskellers. Streamly is a higher level library than `async` and provides a lot
6more power and functionality, using a simpler and concise expression of
7concurrency. At a high level, you should be able to express everything with
8streamly that you can with `async`, if you can't please raise an issue. If you
9are familiar with `async`, in this document we highlight how streamly can be
10used where you would use `async`.
12## `async/wait` vs Concurrent Streams
14Unlike `async`, streamly does not use a spawn and `wait` model. Streamly uses
15a more high level approach to concurrency and has no explicit notion of
16threads. In streamly, we compose multiple actions as a stream and then express
17whether you want to run the actions in the stream `serially` or `parallely`.
18There are many different ways in which you can run streams concurrently, see
19the reference documentation for details.
21Since there is no explicit notion of threads in streamly, there are no
22equivalents of `async`, `wait`, `cancel`, `poll` or `link` combinators from the
23`async` package.
25Since streamly is a monad transformer it can work with all monads and not just
26IO, you won't need adaptations like `lifted-async` to use it for a generic
27monad.
29## Using Streamly for Concurrency
31You can write all of your program in a streamly monad and use the full power of
32the library. Streamly can be used as a direct replacement of the IO monad with
33no loss of performance, and no change in code except using `liftIO` or `yieldM`
34to run any IO actions. Streamly IO monads (e.g. `SerialT IO`) are just a
35generalization of the IO monad with non-deterministic composition of streams
36added on top.
38However, if you would like to just run only some concurrent portions of your
39program using streamly, you can do that too. Just use `runStream` if you want
40to run the stream without collecting the outputs of the concurrent actions or
41use `toList` if you want to convert the output stream into a list. Other
42stream folding operations can also be used, see the docs for more details.
44## Features as Compared with `async`
46Use the following imports to run the snippets shown below:
48```haskell
49import Streamly
50import Streamly.Prelude ((|:))
51import qualified Streamly.Prelude as S
52import qualified Data.Text as Text
53import Control.Concurrent (threadDelay)
54```
56Let us simulate a URL fetch with a delay of `n` seconds using the following
57functions:
59```haskell
60getURL :: Int -> IO String
61getURL n = threadDelay (n * 1000000) >> return (show n)
62getURLString = getURL
63getURLText n = getURL n >>= return . Text.pack
64```
66### concurrently
68You can run any number of actions concurrently. For example, to fetch two URLs
69concurrently:
71```haskell
72 urls <- S.toList $ parallely $ getURL 2 |: getURL 1 |: S.nil
73```
75This would return the results in their arrival order i.e. first 1 and then 2.
76If you want to preserve the order of the results, use the lookahead style
77stream `aheadly` instead. In the following example both URLs are fetched
78concurrently, and even though URL 1 arrives before URL 2 the results will
79return 2 first and then 1.
81```haskell
82 urls <- S.toList $ aheadly $ getURL 2 |: getURL 1 |: S.nil
83```
85### concurrently_
87Use `runStream` instead of `toList` to run the actions but ignore the results:
89```haskell
90 runStream $ parallely $ getURL 1 |: getURL 2 |: S.nil
91```
93### Concurrent Applicative
95If the actions that you are executing result in different output types you can
96use applicative zip to collect the results or to directly apply them to a
97function:
99```haskell
100 tuples <- S.toList $ zipAsyncly $
101 (,) <$> S.yieldM (getURLString 1) <*> S.yieldM (getURLText 2)
102```
104### race
106To run multiple actions and use the first result:
108```haskell
109 urls <- S.toList $ S.take 1 $ parallely $ getURL 1 |: getURL 2 |: S.nil
110```
112After the first result arrives, the rest of the actions are canceled
113automatically. You can take first `n` results as they arrive:
115```haskell
116 urls <- S.toList $ S.take 2 $ parallely $ getURL 1 |: getURL 2 |: S.nil
117```
119### mapConcurrently
121There are many ways to map concurrently on a container and collect the results:
123You can create a concurrent stream from a `Foldable` container of monadic
124actions:
126```haskell
127 urls <- S.toList $ aheadly $ S.fromFoldableM $ fmap getURL [1..3]
128```
130You can first convert a `Foldable` into a stream and then map an action on the
131stream concurrently:
133```haskell
134 urls <- S.toList $ aheadly $ S.mapM getURL $ foldMap return [1..3]
135```
137You can map a monadic action to a `Foldable` container to convert it into a
138stream and at the same time fold it:
140```haskell
141 urls <- S.toList $ aheadly $ foldMap (S.yieldM . getURL) [1..3]
142```
144### replicateConcurrently
146Streamly has not just the equivalent of `replicateConcurrently` which is
147`replicateM` but many more ways to generate concurrent streams, for example,
148`|:`, `unfoldrM`, `repeatM`, `iterateM`, `fromFoldableM` etc. See the
149[Streamly.Prelude](https://hackage.haskell.org/package/streamly/docs/Streamly-Prelude.html)
150module documentation for more details.
152```haskell
153 xs <- S.toList $ parallely $ S.replicateM 2 $ getURL 1
154```
156### Functor
158The stream resulting from concurrent actions can be mapped serially or
159concurrently.
161To map serially just use `fmap`:
163```haskell
164 xs <- S.toList $ parallely $ fmap (+1) $ return 1 |: return 2 |: S.nil
165```
167To map a monadic action concurrently on all elements of the stream use `mapM`:
169```haskell
170 xs <- S.toList $ parallely $ S.mapM (\x -> return (x + 1))
171 $ return 1 |: return 2 |: S.nil
172```
174### Semigroup
176The `Semigroup` instances of streamly merge multiple streams serially or
177concurrently.
179### Monad
181The `Monad` instances of streamly nest loops concurrently (concurrent
182non-determinism).
184### Performance
186Streamly has very little concurrency overhead (ranging from a few 100
187nanoseconds to a few microseconds on a 2.2 GHz Intel Core i7), you can even run
188very lightweight actions in parallel without worrying about the overhead of
189concurrency. See the performance benchmarks [comparing streamly with the `async`
190package in this repo](https://github.com/composewell/concurrency-benchmarks).
192## Further Reading
194There is much more that you can do with streamly. For example, you can use the
195`maxThreads` combinator to restrict the total number of concurrent threads or
196use the `maxBuffer` combinator to restrict the total number of bufferred
197results or you can use the `avgRate` combinator to control the rate at which
198the concurrent actions are executed.
200See the [haddock documentation on
201hackage](https://hackage.haskell.org/package/streamly) and [a comprehensive tutorial
202here](https://hackage.haskell.org/package/streamly/docs/Streamly-Tutorial.html).