git.iko.soy/streamly
on master help sign in
streamly/docs/User/HowTo/streamly-vs-async.md
237 lines 8.0 kB history raw
1# Streamly
3THIS TUTORIAL IS OBSOLETE.
5Streamly is a library to make concurrent programming a joy. The venerable
6`async` package is the go to package for concurrent programming for most
7Haskellers. Streamly is a higher level library than `async` and provides a lot
8more power and functionality, using a simpler and concise expression of
9concurrency. At a high level, you should be able to express everything with
10streamly that you can with `async`, if you can't please raise an issue. If you
11are familiar with `async`, in this document we highlight how streamly can be
12used where you would use `async`.
14## `async/wait` vs Concurrent Streams
16Unlike `async`, streamly does not use a spawn and `wait` model. Streamly uses
17a more high level approach to concurrency and has no explicit notion of
18threads. In streamly, we compose multiple actions as a stream and then express
19whether you want to run the actions in the stream `serially` or `parallely`.
20There are many different ways in which you can run streams concurrently, see
21the reference documentation for details.
23Since there is no explicit notion of threads in streamly, there are no
24equivalents of `async`, `wait`, `cancel`, `poll` or `link` combinators from the
25`async` package.
27Since streamly is a monad transformer it can work with all monads and not just
28IO, you won't need adaptations like `lifted-async` to use it for a generic
29monad.
31## Using Streamly for Concurrency
33You can write all of your program in a streamly monad and use the full power of
34the library. Streamly can be used as a direct replacement of the IO monad with
35no loss of performance, and no change in code except using `liftIO` or `fromEffect`
36to run any IO actions. Streamly IO monads (e.g. `SerialT IO`) are just a
37generalization of the IO monad with non-deterministic composition of streams
38added on top.
40However, if you would like to just run only some concurrent portions of your
41program using streamly, you can do that too. Just use `drain` if you want
42to run the stream without collecting the outputs of the concurrent actions or
43use `toList` if you want to convert the output stream into a list. Other
44stream folding operations can also be used, see the docs for more details.
46## Features as Compared with `async`
48Use the following imports to run the snippets shown below:
50```haskell
51import Streamly
52import Streamly.Prelude ((|:))
53import qualified Streamly.Prelude as S
54import qualified Data.Text as Text
55import Control.Concurrent (threadDelay)
56```
58Let us simulate a URL fetch with a delay of `n` seconds using the following
59functions:
61```haskell
62getURL :: Int -> IO String
63getURL n = threadDelay (n * 1000000) >> return (show n)
64getURLString = getURL
65getURLText n = getURL n >>= return . Text.pack
66```
68### concurrently
70You can run any number of actions concurrently. For example, to fetch two URLs
71concurrently:
73```haskell
74 urls <- S.toList $ fromParallel $ getURL 2 |: getURL 1 |: S.nil
75```
77This would return the results in their arrival order i.e. first 1 and then 2.
78If you want to preserve the order of the results, use the lookahead style
79using `fromAhead` instead. In the following example both URLs are fetched
80concurrently, and even though URL 1 arrives before URL 2 the results will
81return 2 first and then 1.
83```haskell
84 urls <- S.toList $ fromAhead $ getURL 2 |: getURL 1 |: S.nil
85```
87### concurrently_
89Use `drain` instead of `toList` to run the actions but ignore the results:
91```haskell
92 S.drain $ fromParallel $ getURL 1 |: getURL 2 |: S.nil
93```
95### Concurrent Applicative
97If the actions that you are executing result in different output types you can
98use applicative zip to collect the results or to directly apply them to a
99function:
101```haskell
102 tuples <- S.toList $ fromZipAsync $
103 (,) <$> S.fromEffect (getURLString 1) <*> S.fromEffect (getURLText 2)
104```
106### race
108There are two ways to achieve the race functionality, using `take` or using
109exceptions.
111#### `race` Using `take`
113We can run multiple actions concurrently and take the first result that
114arrives:
116```haskell
117 urls <- S.toList $ S.take 1 $ fromParallel $ getURL 1 |: getURL 2 |: S.nil
118```
120After the first result arrives, the rest of the actions are canceled
121automatically. In general, we can take first `n` results as they arrive:
123```haskell
124 urls <- S.toList $ S.take 2 $ fromParallel $ getURL 1 |: getURL 2 |: S.nil
125```
127#### `race` Using Exceptions
129When an exception occurs in a concurrent stream all the concurrently running
130actions are cacnceled on arrival of the exception. This can be used to
131implement the race functionality. Each action in the stream can use an
132exception to communicate the result. As soon as the first result arrives all
133other actions will be canceled, for example:
135```haskell
136 data Result = Result String deriving Show
137 instance Exception Result
139 main = do
140 url <- try $ S.drain $ fromParallel $
141 (getURL 2 >>= throwM . Result)
142 |: (getURL 1 >>= throwM . Result)
143 |: S.nil
144 case url of
145 Left (e :: SomeException) -> print e
146 Right _ -> undefined
147```
149### mapConcurrently
151There are many ways to map concurrently on a container and collect the results:
153You can create a concurrent stream from a `Foldable` container of monadic
154actions:
156```haskell
157 urls <- S.toList $ fromAhead $ S.fromFoldableM $ fmap getURL [1..3]
158```
160You can first convert a `Foldable` into a stream and then map an action on the
161stream concurrently:
163```haskell
164 urls <- S.toList $ fromAhead $ S.mapM getURL $ foldMap return [1..3]
165```
167You can map a monadic action to a `Foldable` container to convert it into a
168stream and at the same time fold it:
170```haskell
171 urls <- S.toList $ fromAhead $ foldMap (S.fromEffect . getURL) [1..3]
172```
174### replicateConcurrently
176Streamly has not just the equivalent of `replicateConcurrently` which is
177`replicateM` but many more ways to generate concurrent streams, for example,
178`|:`, `unfoldrM`, `repeatM`, `iterateM`, `fromFoldableM` etc. See the
179[Streamly.Prelude](https://hackage.haskell.org/package/streamly/docs/Streamly-Prelude.html)
180module documentation for more details.
182```haskell
183 xs <- S.toList $ fromParallel $ S.replicateM 2 $ getURL 1
184```
186### Functor
188The stream resulting from concurrent actions can be mapped serially or
189concurrently.
191To map serially just use `fmap`:
193```haskell
194 xs <- S.toList $ fromParallel $ fmap (+1) $ return 1 |: return 2 |: S.nil
195```
197To map a monadic action concurrently on all elements of the stream use `mapM`:
199```haskell
200 xs <- S.toList $ fromParallel $ S.mapM (\x -> return (x + 1))
201 $ return 1 |: return 2 |: S.nil
202```
204### Semigroup
206The `Semigroup` instances of streamly merge multiple streams serially or
207concurrently.
209### Monad
211The `Monad` instances of streamly nest loops concurrently (concurrent
212non-determinism).
214### Performance
216Streamly has very little concurrency overhead (ranging from a few 100
217nanoseconds to a few microseconds on a 2.2 GHz Intel Core i7), you can even run
218very lightweight actions in parallel without worrying about the overhead of
219concurrency. See the performance benchmarks [comparing streamly with the `async`
220package in this repo](https://github.com/composewell/concurrency-benchmarks).
222## Further Reading
224There is much more that you can do with streamly. For example, you can use the
225`maxThreads` combinator to restrict the total number of concurrent threads or
226use the `maxBuffer` combinator to restrict the total number of bufferred
227results or you can use the `avgRate` combinator to control the rate at which
228the concurrent actions are executed.
230See the [haddock documentation on
231hackage](https://hackage.haskell.org/package/streamly) and [a comprehensive tutorial
232here](https://streamly.composewell.com/streamly-0.8.2/Tutorial.html).
234## References
236* https://hackage.haskell.org/package/async
237* https://hackage.haskell.org/package/lifted-async