@@ -1,72 +1,106 @@
1−# [Streamly](https://streamly.composewell.com)
1+# [Streamly][]: Idiomatic Haskell with the Performance of C
3[](https://gitter.im/composewell/streamly)
3[](https://gitter.im/composewell/streamly)
4[](https://hackage.haskell.org/package/streamly)
4[](https://hackage.haskell.org/package/streamly)
6−## Idiomatic Haskell at the speed of C
6+ [Streamly]: https://streamly.composewell.com/
8−Streamly is a Haskell library/framework providing basic building blocks
8+Streamly is a Haskell library that provides the building blocks to build
9−or combinators to build safe, scalable, modular and high performance
9+safe, scalable, modular and high performance software. Streamly offers:
10−software systems. The key features it provides are:
12−* Speed of C
11+* The type safety of Haskell.
13−* Safety of Haskell
12+* The performance of C programs.
13+* Powerful abstractions for structuring your code.
14−* Idiomatic functional programming
14+* Idiomatic functional programming.
15−* Powerful abstractions
16−* Declarative concurrency
15+* Declarative concurrency for the seamless use of multiprocessing hardware.
18−This guide introduces you to streamly using some practical examples.
17+## About This Document
19−See [Streamly Homepage](https://streamly.composewell.com) and [Streamly
20−Examples](https://github.com/composewell/streamly-examples) to learn
23−## Installing and using
19+This guide introduces programming with [Streamly][] using a few practical
25−Please see [getting started with streamly](./INSTALL.md) for
22+* We will start with a simple program that [counts the number of words
26−instructions on how to use streamly with your Haskell build tool or
23+ in a text](#modular-word-counting). We will then transform this program
27−package manager.
24+ into a [concurrent](#concurrent-word-counting) program that can efficiently
25+ use multiprocessing hardware.
26+* Next, we will create a [concurrent network
27+ server](#a-concurrent-network-server). We then show
28+ how to write a network server that [merges multiple
29+ streams](#merging-incoming-streams) concurrently.
30+* Our third example shows how to list a directory tree concurrently,
31+ by reading [multiple directories in
32+ parallel](#listing-directories-recursivelyconcurrently).
33+* Finally, we will look at how to [rate limit](#rate-limiting) stream
29−To build your streamly application for best performance see the
36+The guide then looks at how Streamly achieves its
30−[build and compile options guide](docs/Build.md).
37+[performance](#performance). It [concludes](#notes) with a brief
38+discussion about Streamly's design philosophy, and with suggestions for
32−## Documentation
41+## Getting Started
34−* [Streamly Homepage](https://streamly.composewell.com)
43+### Installing Streamly
35−* [Streamly Examples](https://github.com/composewell/streamly-examples)
44+
36−* [Streaming Benchmarks](https://github.com/composewell/streaming-benchmarks) | [Concurrency Benchmarks](https://github.com/composewell/concurrency-benchmarks)
45+If you wish to follow along with this guide, you will need to have
37−* [Functional Conf 2019 Video](https://www.youtube.com/watch?v=uzsqgdMMgtk) | [Functional Conf 2019 Slides](https://www.slideshare.net/HarendraKumar10/streamly-concurrent-data-flow-programming)
46+[Streamly][] installed.
48+Please see the [Getting Started With The Streamly Package](./INSTALL.md)
49+guide for instructions on how to install [Streamly][].
51+If you wish to run benchmarks, please be sure to build your
52+application using the instructions in the [Build And Compile Options
53+Guide](docs/Build.md).
55+### An overview of the types used in these examples
40−## Types Overview
57+As an expository device, we have indicated the types at the intermediate
58+stages of stream computations as comments in the examples below.
59+The meaning of these types are:
42−Quick overview of basic types for this tutorial:
61+* A `SerialT IO a` is a serial stream of values of type `a` in the IO Monad.
62+* An `AsyncT IO a` is a concurrent (asynchronous) stream of values of type
64+* An `Unfold IO a b` is a representation of a function that converts a seed
65+ value of type `a` into a stream of values of type `b` in the IO Monad.
66+* A `Fold IO a b` is a representation of a function that converts a stream of
67+ type `a` to a final accumulator of type `b` in the IO Monad.
44−* `SerialT IO a` is a serial stream of values of type `a` in IO Monad.
69+### A Note on Module Naming
45−* `AsyncT IO a` is a concurrent (async) stream of values of type `a` in IO
47−* `Unfold IO a b` is a representation of a function that converts a seed
48− value of type `a` to a stream of values of type `b` in IO Monad.
49−* `Fold IO a b` is a representation of a function that converts a stream of
50− type `a` to a final accumulator of type `b` in IO Monad.
52−## Modular Word Counting
71+Some of the examples below use modules from the `Internal` Streamly package
72+hierarchy. These are not really internal to the library. We classify
73+`Streamly` modules into two categories:
54−In this example, we will use folds to count bytes, words and lines in
75+* _Released modules and APIs_: These modules and APIs are
55−a file. A `Fold` is a composable stream consumer. We will see how
76+ stable. Significant changes to these modules and APIs will cause
56−individual folds can be composed together to perform all the three
77+ Streamly's version number to change according to the package versioning
57−counts at once with the same performance.
78+ policy.
79+* _Pre-release modules and APIs_: These modules and APIs have not been
80+ formally released yet. They may change in the near future, and such
81+ changes will not necessarily be reflected in Streamly's package
82+ version number. As yet unreleased modules and APIs reside in the
83+ `Internal` namespace.
59−See [WordCountModular.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountModular.hs)
85+## The Examples
60−for full working code including imports that we may have omitted
61−here. Note, the `Internal` modules imported here are `pre-release`
62−modules that have been tested and are ready for use except for some
63−minor signature changes planned before we release them.
65−### Count bytes (wc -c)
87+### Modular Word Counting
67−Count bytes in a file.
89+A `Fold` in Streamly is a composable stream consumer. For our first
90+example, we will use `Fold`s to count the number of bytes, words and lines
91+present in a file. We will then compose individual `Fold`s together to
92+count words, bytes and lines at the same time.
95+[WordCountModular.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountModular.hs)
96+for the complete example program, including the imports that we have
99+#### Count Bytes (wc -c)
101+We start with a code fragment that counts the number of bytes in a file:
69−``` haskell
103+```haskell
70import qualified Streamly.Data.Fold as Fold
104import qualified Streamly.Data.Fold as Fold
71import qualified Streamly.Internal.FileSystem.File as File
105import qualified Streamly.Internal.FileSystem.File as File
72import qualified Streamly.Prelude as Stream
106import qualified Streamly.Prelude as Stream
@@ -77,15 +111,12 @@ wcb file =
77 & Stream.fold Fold.length -- IO Int
111 & Stream.fold Fold.length -- IO Int
80−If you are wondering about the `Internal` module import, see [A Note on
114+### Count Lines (wc -l)
81−Internal Modules](#a-note-on-internal-modules) section.
83−### Count lines (wc -l)
116+The next code fragment shows how to count the number of lines in a file:
85−Count lines in a file.
87−``` haskell
118+```haskell
88−-- ASCII character 10 is newline
119+-- ASCII character 10 is a newline.
89countl :: Int -> Word8 -> Int
120countl :: Int -> Word8 -> Int
90countl n ch = if ch == 10 then n + 1 else n
121countl n ch = if ch == 10 then n + 1 else n
@@ -99,18 +130,19 @@ wcl file =
99 & Stream.fold nlines -- IO Int
130 & Stream.fold nlines -- IO Int
102−### Count words (wc -w)
133+### Count Words (wc -w)
104−Count words in a file.
135+Our final code fragment counts the number of whitespace-separated words
106−``` haskell
138+```haskell
107countw :: (Int, Bool) -> Word8 -> (Int, Bool)
139countw :: (Int, Bool) -> Word8 -> (Int, Bool)
108countw (n, wasSpace) ch =
140countw (n, wasSpace) ch =
109 if isSpace $ chr $ fromIntegral ch
141 if isSpace $ chr $ fromIntegral ch
110 then (n, True)
142 then (n, True)
111 else (if wasSpace then n + 1 else n, False)
143 else (if wasSpace then n + 1 else n, False)
113−-- The fold accepts a stream of `Word8` and returns a word count (`Int`)
145+-- The fold accepts a stream of `Word8` and returns a word count (`Int`).
114nwords :: Monad m => Fold m Word8 Int
146nwords :: Monad m => Fold m Word8 Int
115nwords = fst <$> Fold.foldl' countw (0, True)
147nwords = fst <$> Fold.foldl' countw (0, True)
@@ -120,17 +152,19 @@ wcw file =
120 & Stream.fold nwords -- IO Int
152 & Stream.fold nwords -- IO Int
123−### Count bytes, words and lines
155+### Counting Bytes, Words and Lines Together
125−We can compose the three folds together into a single fold using `Tee`
157+By using the `Tee` combinator we can compose the three folds that count
158+bytes, lines and words individually into a single fold that counts all
126−to do all the three things at once. The applicative instance of `Tee`
159+three at once. The applicative instance of `Tee` distributes its input
160+to all the supplied folds (`Fold.length`, `nlines`, and `nwords`) and
127−distributes the input to all the folds and combines the outputs using the
161+then combines the outputs from the folds using the supplied combiner
128−supplied function.
162+function (`(,,)`).
130−``` haskell
164+```haskell
131import qualified Streamly.Internal.Data.Fold.Tee as Tee
165import qualified Streamly.Internal.Data.Fold.Tee as Tee
133−-- The fold accepts a stream of `Word8` and returns the three counts
167+-- The fold accepts a stream of `Word8` and returns the three counts.
134countAll :: Fold IO Word8 (Int, Int, Int)
168countAll :: Fold IO Word8 (Int, Int, Int)
135countAll = Tee.toFold $ (,,) <$> Tee Fold.length <*> Tee nlines <*> Tee nwords
169countAll = Tee.toFold $ (,,) <$> Tee Fold.length <*> Tee nlines <*> Tee nwords
@@ -140,16 +174,18 @@ wc file =
140 & Stream.fold countAll -- IO (Int, Int, Int)
174 & Stream.fold countAll -- IO (Int, Int, Int)
143−This example demonstrates the simple and concise API of streamly with
177+This example demonstrates the excellent modularity offered by
144−excellent modularity. Experienced Haskellers would notice that we have
178+[Streamly][]'s simple and concise API. Experienced Haskellers will
145−not used bytestrings, we simply use a stream of `Word8`, simplifying the
179+notice that we have not used bytestrings—we instead used a stream of
146−program.
180+`Word8` values, simplifying our program.
182+### The Performance of Word Counting
148−## Word Counting Performance
184+We compare two equivalent implementations: one using [Streamly][],
185+and the other using C.
150−We compare two equivalent implementations, one using Haskell Streamly and the
187+The performance of the [Streamly word counting
152−[Haskell Streamly word counting implementation](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.hs):
188+implementation](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.hs) is:
155$ time WordCount-hs gutenberg-500MB.txt
191$ time WordCount-hs gutenberg-500MB.txt
@@ -160,7 +196,7 @@ user 0m1.697s
160sys 0m0.128s
196sys 0m0.128s
163−[Equivalent BSD wc implementation in C](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.c):
199+The performance of an equivalent [wc implementation in C](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.c) is:
166$ time WordCount-c gutenberg-500MB.txt
202$ time WordCount-c gutenberg-500MB.txt
@@ -171,20 +207,24 @@ user 0m1.935s
171sys 0m0.165s
207sys 0m0.165s
174−## Concurrent Word Counting
210+### Concurrent Word Counting
176−To count words in parallel we divide the stream into chunks (arrays),
212+In our next example we show how the task of counting words, lines,
177−count properties in each chunk and then add all the counts. We use the
213+and bytes could be done in parallel on multiprocessor hardware.
178−same code as above except that we use an array input instead of using a
181−See
215+To count words in parallel we first divide the stream into chunks
182−[WordCountParallel.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountParallel.hs).
216+(arrays), do the counting within each chunk, and then add all the
183−for full working code including the imports that we may have omitted below.
217+counts across chunks. We use the same code as above except that we use
218+arrays for our input data.
185−Get the line, word, char counts in one chunk.
220+Please see the file
221+[WordCountParallel.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountParallel.hs)
222+for the complete working code for this example, including the imports
223+that we have omitted below.
225+The `countArray` function counts the line, word, char counts in one chunk:
187−``` haskell
227+```haskell
188import qualified Streamly.Data.Array.Foreign as Array
228import qualified Streamly.Data.Array.Foreign as Array
190countArray :: Array Word8 -> IO Counts
230countArray :: Array Word8 -> IO Counts
@@ -193,14 +233,18 @@ countArray arr =
193 & Stream.decodeLatin1 -- SerialT IO Char
233 & Stream.decodeLatin1 -- SerialT IO Char
194 & Stream.foldl' count (Counts 0 0 0 True) -- IO Counts
234 & Stream.foldl' count (Counts 0 0 0 True) -- IO Counts
237+Here the function `count` and the `Counts` data type are defined in the
238+`WordCount` helper module defined in [WordCount.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.hs).
196−When combining the counts in two contiguous chunks, we would also need
240+When combining the counts in two contiguous chunks, we need to check
197−to know whether the first element of the next chunk was a space char or
241+whether the first element of the next chunk is a whitespace character
198−non-space to know whether the same word is continuing to the next chunk
242+in order to determine if the same word continues in the next chunk or
243+whether the chunk starts with a new word. The `partialCounts` function
199−or if it is a new word. `partialCounts` adds a `Bool` flag to `Counts`
244+adds a `Bool` flag to `Counts` returned by `countArray` to indicate
200−returned by `countArray` to indicate whether the first character in the
245+whether the first character in the chunk is a space.
201−chunk is a space.
246+
203−``` haskell
247+```haskell
204partialCounts :: Array Word8 -> IO (Bool, Counts)
248partialCounts :: Array Word8 -> IO (Bool, Counts)
205partialCounts arr = do
249partialCounts arr = do
206 let r = Array.getIndex arr 0
250 let r = Array.getIndex arr 0
@@ -211,20 +255,23 @@ partialCounts arr = do
211 Nothing -> return (False, Counts 0 0 0 True)
255 Nothing -> return (False, Counts 0 0 0 True)
214−Combine the counts from two consecutive chunks.
258+`addCounts` then adds the counts from two consecutive chunks:
215−``` haskell
260+```haskell
216addCounts :: (Bool, Counts) -> (Bool, Counts) -> (Bool, Counts)
261addCounts :: (Bool, Counts) -> (Bool, Counts) -> (Bool, Counts)
217addCounts (sp1, Counts l1 w1 c1 ws1) (sp2, Counts l2 w2 c2 ws2) =
262addCounts (sp1, Counts l1 w1 c1 ws1) (sp2, Counts l2 w2 c2 ws2) =
218 let wcount =
263 let wcount =
219− if not ws1 && not sp2 -- no space between two chunks
264+ if not ws1 && not sp2 -- No space between two chunks.
220 then w1 + w2 - 1
265 then w1 + w2 - 1
221 else w1 + w2
266 else w1 + w2
222 in (sp1, Counts (l1 + l2) wcount (c1 + c2) ws2)
267 in (sp1, Counts (l1 + l2) wcount (c1 + c2) ws2)
225−Now put it all together, we only need to divide the stream into arrays,
270+To count in parallel we now only need to divide the stream into arrays,
226−apply our counting function to each array and then combine all the counts.
271+apply our counting function to each array, and then combine the counts
227−``` haskell
274+```haskell
228wc :: String -> IO (Bool, Counts)
275wc :: String -> IO (Bool, Counts)
229wc file = do
276wc file = do
230 Stream.unfold File.readChunks file -- AheadT IO (Array Word8)
277 Stream.unfold File.readChunks file -- AheadT IO (Array Word8)
@@ -234,12 +281,14 @@ wc file = do
234 & Stream.foldl' addCounts (False, Counts 0 0 0 True) -- IO (Bool, Counts)
281 & Stream.foldl' addCounts (False, Counts 0 0 0 True) -- IO (Bool, Counts)
237−Note that `Stream.fromAhead` is the only difference in a concurrent and
284+Please note that the only difference between a concurrent and a
285+non-concurrent program lies in the use of the `Stream.fromAhead`
286+combinator. If we remove the call to `Stream.fromAhead`, we would
238−non-concurrent program. If we remove that we still have a perfectly valid,
287+still have a perfectly valid and performant serial program. Notice
239−well performing serial program. Notice, how succinctly and idiomatically
240−we expressed the concurrent word counting problem.
288+how succinctly and idiomatically we have expressed the concurrent word
242−Benchmarked with 2 CPUs:
291+A benchmark with 2 CPUs:
245$ time WordCount-hs-parallel gutenberg-500MB.txt
294$ time WordCount-hs-parallel gutenberg-500MB.txt
@@ -250,32 +299,40 @@ user 0m1.952s
250sys 0m0.140s
299sys 0m0.140s
253−If you want to get serious about word counting, here is a
302+These example programs have assumed ASCII encoded
303+input data. For UTF-8 streams, we have a [concurrent wc
254−[concurrent wc implementation with UTF-8 decoding](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountUTF8.hs).
304+implementation](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountUTF8.hs)
305+with UTF-8 decoding. This concurrent implementation performs as well
255−It performs as well as the stock wc in serial benchmarks, and of course
306+as the standard `wc` program in serial benchmarks. In concurrent mode
256−in concurrent mode it can use multiple cores so can be much faster.
307+[Streamly][]'s implementation can utilise multiple processing cores if
257−
308+these are present, and can thereby run much faster than the standard
258−Streamly provides concurrency facilities similar to
311+Streamly provides concurrency facilities similar
259−[OpenMP](https://en.wikipedia.org/wiki/OpenMP) and
312+to [OpenMP](https://en.wikipedia.org/wiki/OpenMP) and
260[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative
313[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative
261−expression. You can write concurrent loops with ease, with different types of
314+style of expression. With Streamly you can write concurrent programs
262−concurrent scheduling.
315+with ease, with support for different types of concurrent scheduling.
264−## Concurrent Network Server
317+### A Concurrent Network Server
266−Slightly more complicated example. A dictionary lookup server, the server
319+We now move to a slightly more complicated example: we simulate a
320+dictionary lookup server which can serve word meanings to multiple
267−serves word meanings to multiple clients concurrently. It uses the concurrent
321+clients concurrently. This example demonstrates the use of the concurrent
268`mapM` combinator.
322`mapM` combinator.
270−See
324+Please see the file
271[WordServer.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordServer.hs)
325[WordServer.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordServer.hs)
272−for full working code including the imports that we may have omitted below.
326+for the complete code for this example, including the imports that we
274−``` haskell
329+```haskell
330+import qualified Streamly.Data.Fold as Fold
275import qualified Streamly.Network.Inet.TCP as TCP
331import qualified Streamly.Network.Inet.TCP as TCP
332+import qualified Streamly.Network.Socket as Socket
276import qualified Streamly.Unicode.Stream as Unicode
333import qualified Streamly.Unicode.Stream as Unicode
278−-- Simulate network/db query by adding a delay
335+-- Simulate network/db query by adding a delay.
279fetch :: String -> IO (String, String)
336fetch :: String -> IO (String, String)
280fetch w = threadDelay 1000000 >> return (w,w)
337fetch w = threadDelay 1000000 >> return (w,w)
@@ -311,23 +368,26 @@ main =
311 & Stream.drain -- IO ()
368 & Stream.drain -- IO ()
314−## Merging Incoming Streams
371+### Merging Incoming Streams
316−Assume you have logs coming from multiple nodes in your network and
373+In the next example, we show how to merge logs coming from multiple
317−you want to merge all the logs at line boundaries and send the merged
374+nodes in your network. These logs are merged at line boundaries and
318−stream to a file or to a network destination. It uses the amazing
375+the merged logs are written to a file or to a network destination.
319−`concatMapWith` combinator to merge multiple streams concurrently.
376+This example uses the `concatMapWith` combinator to merge multiple
377+streams concurrently.
321−See
379+Please see the file
322[MergeServer.hs](https://github.com/composewell/streamly-examples/blob/master/examples/MergeServer.hs)
380[MergeServer.hs](https://github.com/composewell/streamly-examples/blob/master/examples/MergeServer.hs)
323−for full working code including the imports that we may have omitted below.
381+for the complete working code, including the imports that we have
325−``` haskell
384+```haskell
326import qualified Streamly.Data.Unfold as Unfold
385import qualified Streamly.Data.Unfold as Unfold
327import qualified Streamly.Network.Socket as Socket
386import qualified Streamly.Network.Socket as Socket
329−-- | Read a line stream from a socket. Note, lines are buffered, we could add
388+-- | Read a line stream from a socket.
389+-- Note: lines are buffered, and we could add a limit to the
330−-- a limit to the buffering for safety.
390+-- buffering for safety.
331readLines :: Socket -> SerialT IO (Array Char)
391readLines :: Socket -> SerialT IO (Array Char)
332readLines sk =
392readLines sk =
333 Stream.unfold Socket.read sk -- SerialT IO Word8
393 Stream.unfold Socket.read sk -- SerialT IO Word8
@@ -353,30 +413,30 @@ main :: IO ()
353main = withFile "output.txt" AppendMode server
413main = withFile "output.txt" AppendMode server
356−## Listing Directories Recursively/Concurrently
416+### Listing Directories Recursively/Concurrently
358−The following example lists a directory tree recursively, reading
418+Our next example lists a directory tree recursively, reading
359multiple directories concurrently.
419multiple directories concurrently.
361−It uses the wonderful tree traversing combinator
421+This example uses the tree traversing combinator `iterateMapLeftsWith`.
362−`iterateMapLeftsWith`. It maps a stream generator on the `Left` values
422+This combinator maps a stream generator on the `Left` values in its
363−(directories in this case) of the input stream, feeds the resulting
423+input stream (directory names in this case), feeding the resulting `Left`
364−'Left' values back to the input, and lets the `Right` values (files in
424+values back to the input, while it lets the `Right` values (file names
365−this case) pass through to the output. The `Stream.ahead` stream joining
425+in this case) pass through to the output. The `Stream.ahead` stream
366−combinator makes it iterate on the directories concurrently.
426+joining combinator then makes it iterate on the directories concurrently.
368−See
428+Please see the file
369[ListDir.hs](https://github.com/composewell/streamly-examples/blob/master/examples/ListDir.hs)
429[ListDir.hs](https://github.com/composewell/streamly-examples/blob/master/examples/ListDir.hs)
370−for full working code including the imports that we may have omitted below.
430+for the complete working code, including the imports that we have
372```haskell
433```haskell
374import Streamly.Internal.Data.Stream.IsStream (iterateMapLeftsWith)
434import Streamly.Internal.Data.Stream.IsStream (iterateMapLeftsWith)
376import qualified Streamly.Prelude as Stream
436import qualified Streamly.Prelude as Stream
377import qualified Streamly.Internal.FileSystem.Dir as Dir (toEither)
437import qualified Streamly.Internal.FileSystem.Dir as Dir (toEither)
379−-- Lists a dir as a stream of (Either Dir File)
439+-- Lists a directory as a stream of (Either Dir File).
380listDir :: String -> SerialT IO (Either String String)
440listDir :: String -> SerialT IO (Either String String)
381listDir dir =
441listDir dir =
382 Dir.toEither dir -- SerialT IO (Either String String)
442 Dir.toEither dir -- SerialT IO (Either String String)
@@ -384,7 +444,7 @@ listDir dir =
385 where mkAbs x = dir ++ "/" ++ x
445 where mkAbs x = dir ++ "/" ++ x
387−-- | List the current directory recursively using concurrent processing
447+-- | List the current directory recursively using concurrent processing.
388main :: IO ()
448main :: IO ()
390 hSetBuffering stdout LineBuffering
450 hSetBuffering stdout LineBuffering
@@ -393,12 +453,13 @@ main = do
393 & Stream.mapM_ print
453 & Stream.mapM_ print
396−## Rate Limiting
456+### Rate Limiting
398−For bounded concurrent streams, stream yield rate can be specified. For
458+For bounded concurrent streams, a stream yield rate can be specified
399−example, to print "tick" once every second you can simply write this:
459+easily. For example, to print "tick" once every second you can simply
401−``` haskell
462+```haskell
402main :: IO ()
463main :: IO ()
404 Stream.repeatM (pure "tick") -- AsyncT IO String
465 Stream.repeatM (pure "tick") -- AsyncT IO String
@@ -408,133 +469,139 @@ main =
408 & Stream.mapM_ print -- IO ()
469 & Stream.mapM_ print -- IO ()
411−See
472+Please see the file
412[Rate.hs](https://github.com/composewell/streamly-examples/blob/master/examples/Rate.hs)
473[Rate.hs](https://github.com/composewell/streamly-examples/blob/master/examples/Rate.hs)
413−for full working code.
474+for the complete working code.
476+The concurrency of the stream is automatically controlled to match the
477+specified rate. [Streamly][]'s rate control works precisely even at
478+throughputs as high as millions of yields per second.
415−Concurrency of the stream is automatically controlled to match the specified
416−rate. Rate control works precisely even at throughputs as high as millions of
417−yields per second. For more sophisticated rate control see the haddock
480+For more sophisticated rate control needs please see the [Haddock
418−documentation.
481+documentation](https://hackage.haskell.org/package/streamly-0.7.3/docs/Streamly.html)
420−## Reactive Programming
484+### Reactive Programming
422−Streamly supports reactive and time domain programming inherently because of
486+Streamly supports reactive (time domain) programming because of its
423−declarative concurrency. See the `Streamly.Prelude` module for some time
487+support for declarative concurrency. Please see the `Streamly.Prelude`
424−specific combinators like `intervalsOf` and folds like `takeInterval` in
488+module for time-specific combinators like `intervalsOf`, and
489+folds like `takeInterval` in `Streamly.Internal.Data.Fold`.
425−`Streamly.Internal.Data.Fold`. Also see pre-release sampling combinators in
490+Please also see the pre-release sampling combinators in the
426−the `Streamly.Internal.Data.Stream.IsStream.Top` module including `throttle`
491+`Streamly.Internal.Data.Stream.IsStream.Top` module for `throttle` and
427−and `debounce` like operations.
492+`debounce` like operations.
430[AcidRain.hs](https://github.com/composewell/streamly-examples/tree/master/examples/AcidRain.hs)
495[AcidRain.hs](https://github.com/composewell/streamly-examples/tree/master/examples/AcidRain.hs)
432−[CirclingSquare.hs](https://github.com/composewell/streamly-examples/tree/master/examples/CirclingSquare.hs).
497+[CirclingSquare.hs](https://github.com/composewell/streamly-examples/tree/master/examples/CirclingSquare.hs)
436−Many more examples can be found in the [streamly-examples
437−repository](https://github.com/composewell/streamly-examples).
439−Streamly comes equipped with a very powerful set of abstractions to accomplish
440−any kind of programming tasks that you may want to throw at it. It provides,
441−streams, arrays, file-io, fsnotify, network-io, time domain programming
442−(reactive programming). See the [streamly
498+demonstrate reactive programming using [Streamly][].
443−documentation](https://streamly.composewell.com) to learn more.
499+
444−
500+### More Examples
446−
502+If you would like to view more examples, please visit the [Streamly
447−Streamly uses lock-free synchronization for low overhead
503+Examples](https://github.com/composewell/streamly-examples) web page.
448−concurrency. The number of tasks performed concurrently are
504+
449−determined automatically based on the rate at which a consumer is
505+### Further Reading
450−consuming the results. In other words, you do not need to manage
506+
451−thread pools and decide how many threads to use for a particular
507+* [Streaming Benchmarks](https://github.com/composewell/streaming-benchmarks) | [Concurrency Benchmarks](https://github.com/composewell/concurrency-benchmarks)
452−task. For CPU bound tasks it tries to keep the number of threads
508+* [Functional Conf 2019 Video](https://www.youtube.com/watch?v=uzsqgdMMgtk) | [Functional Conf 2019 Slides](https://www.slideshare.net/HarendraKumar10/streamly-concurrent-data-flow-programming)
453−close to the number of CPUs available whereas for IO bound tasks
509+* [Other Guides](docs/)
454−more threads can be utilized.
510+* [Streamly Homepage](https://streamly.composewell.com)
456−Parallelism can be utilized with little overhead even if
457−the task size is very small, because it can automatically
458−switch to serial mode or batch multiple tasks on the
459−same CPU if that is more efficient. See [concurrency
460−benchmarks](https://github.com/composewell/concurrency-benchmarks) for
461−detailed performance results and a comparison with the `async` package.
463## Performance
512## Performance
465−As you have seen above in the word count example, streamly enables
514+As you have seen in the word count example above, [Streamly][] offers
466−highly modular abstractions with the best possible performance (close to
515+highly modular abstractions for building programs while also offering
467−an equivalent C program).
516+the performance close to an equivalent (imperative) C program.
469−Streamly provides excellent performance even for byte level stream
518+Streamly offers excellent performance even for byte-at-a-time stream
470−operations, it is made possible by employing efficient abstractions like
519+operations using efficient abstractions like `Unfold`s and terminating
471−`Unfold`s and terminating `Fold`s. Byte level stream operations make
520+`Fold`s. Byte-at-a-time stream operations can simplify programming
472−programming simpler because you do not have to deal with chunking and
521+because the developer does not have to deal explicitly with chunking
473−re-combining.
522+and re-combining data.
475−If you can write a program significantly faster in some other way or
524+Streamly exploits GHC's stream fusion optimizations (`case-of-case` and
476−with some other language, please let us know and we will improve.
525+`spec-constr`) aggressively to achieve C-like speed, while also offering
526+highly modular abstractions to developers.
528+[Streamly][] will usually perform very well without any
529+compiler plugins. However, we have fixed some deficiencies
530+that we had noticed in GHC's optimizer using a [compiler
531+plugin](https://github.com/composewell/fusion-plugin). We hope to fold
532+these optimizations into GHC in the future; until then we recommend that
533+you use this plugin for applications that are performance sensitive.
478−## Benchmarks
535+### Benchmarks
480We measured several Haskell streaming implementations
537We measured several Haskell streaming implementations
481−on various micro-benchmarks. Please see [streaming
538+using various micro-benchmarks. Please see the [streaming
482benchmarks](https://github.com/composewell/streaming-benchmarks) page
539benchmarks](https://github.com/composewell/streaming-benchmarks) page
483−for detailed comparison of streamly with other streaming libraries.
540+for a detailed comparison of Streamly against other streaming libraries.
542+Our results show that [Streamly][] is the fastest effectful streaming
543+implementation on almost all the measured microbenchmarks. In many cases
544+it runs up to 100x faster, and in some cases even 1000x faster than
545+some of the tested alternatives. In some composite operation benchmarks
546+[Streamly][] turns out to be significantly faster than Haskell's list
549+*Note*: If you can write a program in some other way or with some other
550+language that runs significantly faster than what [Streamly][] offers,
551+please let us know and we will improve.
555+Streamly comes equipped with a very powerful set of abstractions to
556+accomplish many kinds of programming tasks: it provides support for
557+programming with streams and arrays, for reading and writing from the
558+file system and from the network, for time domain programming (reactive
559+programming), and for reacting to file system events using `fsnotify`.
561+Please view [Streamly's documentation](https://streamly.composewell.com)
562+for more information about Streamly's features.
566+Streamly uses lock-free synchronization for achieving concurrent
567+operation with low overheads. The number of tasks performed concurrently
568+are determined automatically based on the rate at which a consumer
569+is consuming the results. In other words, you do not need to manage
570+thread pools or decide how many threads to use for a particular task.
571+For CPU-bound tasks Streamly will try to keep the number of threads
572+close to the number of CPUs available; for IO-bound tasks it will utilize
575+The parallelism available during program execution can be utilized with
576+very little overhead even where the task size is very
577+small, because Streamly will automatically switch between
578+serial or batched execution of tasks on the same CPU depending
579+on whichever is more efficient. Please see our [concurrency
580+benchmarks](https://github.com/composewell/concurrency-benchmarks) for
581+more detailed performance measurements, and for a comparison with the
485−These results show that streamly is the fastest effectful streaming
584+### Design Goals
486−implementation on almost all the measured micro benchmarks. In many cases
487−it is up to 100x faster and in some cases even 1000x faster. In many
488−composite operation benchmarks streamly turns out to be significantly
489−faster than the Haskell lists implementation.
491−## Design Goals
586+Our goals for [Streamly][] from the very beginning have been:
493−The goals of streamly from the very beginning have been, (1) simplicity
494−by unifying abstractions, (2) high performance. These are hard to
588+1. To achieve simplicity by unifying abstractions.
495−achieve at the same time because they are usually inversely related. We
589+2. To offer high performance.
496−have spent many years trying to get the abstractions right without
497−compromising performance.
591+These goals are hard to achieve simultaneously because they are usually
592+inversely related. We have spent many years trying to get the abstractions
593+right without compromising performance.
499−`Unfold` is an example of an abstraction that we have created to
595+`Unfold` is an example of an abstraction that we have created to achieve
500−achieve high performance when mapping streams on streams. It allows
596+high performance when mapping streams on streams. `Unfold` allows stream
501−stream generation to be optimized well by the compiler, employing
597+generation to be optimized well by the compiler through stream fusion.
502−stream fusion. `Fold` with termination capability is another example
598+A `Fold` with termination capability is another example which modularizes
503−which modularizes stream elimination operations with stream fusion.
599+stream elimination operations through stream fusion. Terminating folds
504−Terminating folds can perform many simple parsing tasks that do not
600+can perform many simple parsing tasks that do not require backtracking.
505−require backtracking. `Parser`s in streamly are a natural extension
601+In Streamly, `Parser`s are a natural extension to terminating `Fold`s;
506−of terminating `Fold`s just adding backtracking capability to folds.
602+`Parser`s add the ability to backtrack to `Fold`s. Unification leads
507−Unification leads to simpler abstractions, lesser cognitive overhead
603+to simpler abstractions and lower cognitive overheads while also not
508−without compromising performance.
604+compromising performance.
510−Streamly exploits GHC stream fusion optimizations (`case-of-case`
511−and `spec-constr`) aggressively to bring C like speed with highly
512−modular abstractions. It performs very well without any compiler
513−plugins. However, we have fixed some deficiencies in GHC optimizer
514−via a [compiler plugin](https://github.com/composewell/fusion-plugin).
515−We hope to bring these optimizations to GHC in future but until
516−then we recommend that you use the plugin for performance sensitive
519−## A Note on Internal Modules
521−You may see some `Internal` modules imported in the examples. Those are
522−not really internal to the library. We can classify streamly modules into
525−* Released Modules: These are the modules that have a stable API, any
526− API changes conform to the package versioning policy.
527−* Pre-release Modules: These modules are not yet released due to some planned
528− changes in near future, they will be released soon. These modules reside in
529− the `Internal` namespace for now.
531−Pre-release APIs: Some of the APIs that are recently introduced
532−and require some soak time for stability are kept in the
533−internal modules corresponding to the released module (e.g.
534−Streamly.Internal.Data.Fold).
536−We usually try to change even the unstable APIs in `Internal` module
537−namespace in a major release version.
539## Credits
606## Credits
@@ -545,20 +612,26 @@ significant way:
545 * Gabriel Gonzalez ([foldl](https://hackage.haskell.org/package/foldl))
612 * Gabriel Gonzalez ([foldl](https://hackage.haskell.org/package/foldl))
546 * Alberto G. Corona ([transient](https://hackage.haskell.org/package/transient))
613 * Alberto G. Corona ([transient](https://hackage.haskell.org/package/transient))
615+Please see the [`credits`](./credits/README.md) directory for a full
548−See the `credits` directory for full list of contributors, credits and licenses.
616+list of contributors, credits and licenses.
620+Streamly is an [open source](https://github.com/composewell/streamly)
621+project available under a liberal [BSD-3-Clause
622+license](https://github.com/composewell/streamly/blob/master/LICENSE).
550−## Contributing
624+## Contributing to Streamly
626+As an open project we welcome contributions:
552* [Streamly Contributor's Guide](CONTRIBUTING.md)
628* [Streamly Contributor's Guide](CONTRIBUTING.md)
553* [Contact the streamly development team](mailto:streamly@composewell.com)
629* [Contact the streamly development team](mailto:streamly@composewell.com)
555−## License
631+## Getting Support
557−Streamly is an [open source](https://github.com/composewell/streamly)
558−project available under [BSD-3-Clause
559−license](https://github.com/composewell/streamly/blob/master/LICENSE).
561−## Support
633+Professional support is available for [Streamly][]: please contact
634+[support@composewell.com](mailto:support@composewell.com).
636+You can also join our [community chat
563−* Join streamly [online community chat](https://gitter.im/composewell/streamly)
637+channel](https://gitter.im/composewell/streamly) on Gitter.
564−* For professional support please contact [support@composewell.com](mailto:support@composewell.com)