@@ -1,72 +1,106 @@
1−# [Streamly](https://streamly.composewell.com)
1+# [Streamly][]: Idiomatic Haskell with the Performance of C
22
33[](https://gitter.im/composewell/streamly)
44[](https://hackage.haskell.org/package/streamly)
55
6−## Idiomatic Haskell at the speed of C
6+ [Streamly]: https://streamly.composewell.com/
77
8−Streamly is a Haskell library/framework providing basic building blocks
9−or combinators to build safe, scalable, modular and high performance
10−software systems. The key features it provides are:
8+Streamly is a Haskell library that provides the building blocks to build
9+safe, scalable, modular and high performance software. Streamly offers:
1110
12−* Speed of C
13−* Safety of Haskell
14−* Idiomatic functional programming
15−* Powerful abstractions
16−* Declarative concurrency
11+* The type safety of Haskell.
12+* The performance of C programs.
13+* Powerful abstractions for structuring your code.
14+* Idiomatic functional programming.
15+* Declarative concurrency for the seamless use of multiprocessing hardware.
1716
18−This guide introduces you to streamly using some practical examples.
19−See [Streamly Homepage](https://streamly.composewell.com) and [Streamly
20−Examples](https://github.com/composewell/streamly-examples) to learn
21−more.
17+## About This Document
2218
23−## Installing and using
19+This guide introduces programming with [Streamly][] using a few practical
20+examples:
2421
25−Please see [getting started with streamly](./INSTALL.md) for
26−instructions on how to use streamly with your Haskell build tool or
27−package manager.
22+* We will start with a simple program that [counts the number of words
23+ in a text](#modular-word-counting). We will then transform this program
24+ into a [concurrent](#concurrent-word-counting) program that can efficiently
25+ use multiprocessing hardware.
26+* Next, we will create a [concurrent network
27+ server](#a-concurrent-network-server). We then show
28+ how to write a network server that [merges multiple
29+ streams](#merging-incoming-streams) concurrently.
30+* Our third example shows how to list a directory tree concurrently,
31+ by reading [multiple directories in
32+ parallel](#listing-directories-recursivelyconcurrently).
33+* Finally, we will look at how to [rate limit](#rate-limiting) stream
34+ processing.
2835
29−To build your streamly application for best performance see the
30−[build and compile options guide](docs/Build.md).
36+The guide then looks at how Streamly achieves its
37+[performance](#performance). It [concludes](#notes) with a brief
38+discussion about Streamly's design philosophy, and with suggestions for
39+further reading.
3140
32−## Documentation
41+## Getting Started
3342
34−* [Streamly Homepage](https://streamly.composewell.com)
35−* [Streamly Examples](https://github.com/composewell/streamly-examples)
36−* [Streaming Benchmarks](https://github.com/composewell/streaming-benchmarks) | [Concurrency Benchmarks](https://github.com/composewell/concurrency-benchmarks)
37−* [Functional Conf 2019 Video](https://www.youtube.com/watch?v=uzsqgdMMgtk) | [Functional Conf 2019 Slides](https://www.slideshare.net/HarendraKumar10/streamly-concurrent-data-flow-programming)
38−* [Guides](docs)
43+### Installing Streamly
44+
45+If you wish to follow along with this guide, you will need to have
46+[Streamly][] installed.
47+
48+Please see the [Getting Started With The Streamly Package](./INSTALL.md)
49+guide for instructions on how to install [Streamly][].
50+
51+If you wish to run benchmarks, please be sure to build your
52+application using the instructions in the [Build And Compile Options
53+Guide](docs/Build.md).
54+
55+### An overview of the types used in these examples
3956
40−## Types Overview
57+As an expository device, we have indicated the types at the intermediate
58+stages of stream computations as comments in the examples below.
59+The meaning of these types are:
4160
42−Quick overview of basic types for this tutorial:
61+* A `SerialT IO a` is a serial stream of values of type `a` in the IO Monad.
62+* An `AsyncT IO a` is a concurrent (asynchronous) stream of values of type
63+ `a` in the IO Monad.
64+* An `Unfold IO a b` is a representation of a function that converts a seed
65+ value of type `a` into a stream of values of type `b` in the IO Monad.
66+* A `Fold IO a b` is a representation of a function that converts a stream of
67+ type `a` to a final accumulator of type `b` in the IO Monad.
4368
44−* `SerialT IO a` is a serial stream of values of type `a` in IO Monad.
45−* `AsyncT IO a` is a concurrent (async) stream of values of type `a` in IO
46− Monad.
47−* `Unfold IO a b` is a representation of a function that converts a seed
48− value of type `a` to a stream of values of type `b` in IO Monad.
49−* `Fold IO a b` is a representation of a function that converts a stream of
50− type `a` to a final accumulator of type `b` in IO Monad.
69+### A Note on Module Naming
5170
52−## Modular Word Counting
71+Some of the examples below use modules from the `Internal` Streamly package
72+hierarchy. These are not really internal to the library. We classify
73+`Streamly` modules into two categories:
5374
54−In this example, we will use folds to count bytes, words and lines in
55−a file. A `Fold` is a composable stream consumer. We will see how
56−individual folds can be composed together to perform all the three
57−counts at once with the same performance.
75+* _Released modules and APIs_: These modules and APIs are
76+ stable. Significant changes to these modules and APIs will cause
77+ Streamly's version number to change according to the package versioning
78+ policy.
79+* _Pre-release modules and APIs_: These modules and APIs have not been
80+ formally released yet. They may change in the near future, and such
81+ changes will not necessarily be reflected in Streamly's package
82+ version number. As yet unreleased modules and APIs reside in the
83+ `Internal` namespace.
5884
59−See [WordCountModular.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountModular.hs)
60−for full working code including imports that we may have omitted
61−here. Note, the `Internal` modules imported here are `pre-release`
62−modules that have been tested and are ready for use except for some
63−minor signature changes planned before we release them.
85+## The Examples
6486
65−### Count bytes (wc -c)
87+### Modular Word Counting
6688
67−Count bytes in a file.
89+A `Fold` in Streamly is a composable stream consumer. For our first
90+example, we will use `Fold`s to count the number of bytes, words and lines
91+present in a file. We will then compose individual `Fold`s together to
92+count words, bytes and lines at the same time.
6893
69−``` haskell
94+Please see the file
95+[WordCountModular.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountModular.hs)
96+for the complete example program, including the imports that we have
97+omitted here.
98+
99+#### Count Bytes (wc -c)
100+
101+We start with a code fragment that counts the number of bytes in a file:
102+
103+```haskell
70104import qualified Streamly.Data.Fold as Fold
71105import qualified Streamly.Internal.FileSystem.File as File
72106import qualified Streamly.Prelude as Stream
@@ -77,15 +111,12 @@ wcb file =
77111 & Stream.fold Fold.length -- IO Int
78112```
79113
80−If you are wondering about the `Internal` module import, see [A Note on
81−Internal Modules](#a-note-on-internal-modules) section.
114+### Count Lines (wc -l)
82115
83−### Count lines (wc -l)
116+The next code fragment shows how to count the number of lines in a file:
84117
85−Count lines in a file.
86−
87−``` haskell
88−-- ASCII character 10 is newline
118+```haskell
119+-- ASCII character 10 is a newline.
89120countl :: Int -> Word8 -> Int
90121countl n ch = if ch == 10 then n + 1 else n
91122
@@ -99,18 +130,19 @@ wcl file =
99130 & Stream.fold nlines -- IO Int
100131```
101132
102−### Count words (wc -w)
133+### Count Words (wc -w)
103134
104−Count words in a file.
135+Our final code fragment counts the number of whitespace-separated words
136+in a stream:
105137
106−``` haskell
138+```haskell
107139countw :: (Int, Bool) -> Word8 -> (Int, Bool)
108140countw (n, wasSpace) ch =
109141 if isSpace $ chr $ fromIntegral ch
110142 then (n, True)
111143 else (if wasSpace then n + 1 else n, False)
112144
113−-- The fold accepts a stream of `Word8` and returns a word count (`Int`)
145+-- The fold accepts a stream of `Word8` and returns a word count (`Int`).
114146nwords :: Monad m => Fold m Word8 Int
115147nwords = fst <$> Fold.foldl' countw (0, True)
116148
@@ -120,17 +152,19 @@ wcw file =
120152 & Stream.fold nwords -- IO Int
121153```
122154
123−### Count bytes, words and lines
155+### Counting Bytes, Words and Lines Together
124156
125−We can compose the three folds together into a single fold using `Tee`
126−to do all the three things at once. The applicative instance of `Tee`
127−distributes the input to all the folds and combines the outputs using the
128−supplied function.
157+By using the `Tee` combinator we can compose the three folds that count
158+bytes, lines and words individually into a single fold that counts all
159+three at once. The applicative instance of `Tee` distributes its input
160+to all the supplied folds (`Fold.length`, `nlines`, and `nwords`) and
161+then combines the outputs from the folds using the supplied combiner
162+function (`(,,)`).
129163
130−``` haskell
164+```haskell
131165import qualified Streamly.Internal.Data.Fold.Tee as Tee
132166
133−-- The fold accepts a stream of `Word8` and returns the three counts
167+-- The fold accepts a stream of `Word8` and returns the three counts.
134168countAll :: Fold IO Word8 (Int, Int, Int)
135169countAll = Tee.toFold $ (,,) <$> Tee Fold.length <*> Tee nlines <*> Tee nwords
136170
@@ -140,16 +174,18 @@ wc file =
140174 & Stream.fold countAll -- IO (Int, Int, Int)
141175```
142176
143−This example demonstrates the simple and concise API of streamly with
144−excellent modularity. Experienced Haskellers would notice that we have
145−not used bytestrings, we simply use a stream of `Word8`, simplifying the
146−program.
177+This example demonstrates the excellent modularity offered by
178+[Streamly][]'s simple and concise API. Experienced Haskellers will
179+notice that we have not used bytestrings—we instead used a stream of
180+`Word8` values, simplifying our program.
181+
182+### The Performance of Word Counting
147183
148−## Word Counting Performance
184+We compare two equivalent implementations: one using [Streamly][],
185+and the other using C.
149186
150−We compare two equivalent implementations, one using Haskell Streamly and the
151−other using C. The
152−[Haskell Streamly word counting implementation](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.hs):
187+The performance of the [Streamly word counting
188+implementation](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.hs) is:
153189
154190```
155191$ time WordCount-hs gutenberg-500MB.txt
@@ -160,7 +196,7 @@ user 0m1.697s
160196sys 0m0.128s
161197```
162198
163−[Equivalent BSD wc implementation in C](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.c):
199+The performance of an equivalent [wc implementation in C](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.c) is:
164200
165201```
166202$ time WordCount-c gutenberg-500MB.txt
@@ -171,20 +207,24 @@ user 0m1.935s
171207sys 0m0.165s
172208```
173209
174−## Concurrent Word Counting
210+### Concurrent Word Counting
175211
176−To count words in parallel we divide the stream into chunks (arrays),
177−count properties in each chunk and then add all the counts. We use the
178−same code as above except that we use an array input instead of using a
179−file input.
212+In our next example we show how the task of counting words, lines,
213+and bytes could be done in parallel on multiprocessor hardware.
180214
181−See
182−[WordCountParallel.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountParallel.hs).
183−for full working code including the imports that we may have omitted below.
215+To count words in parallel we first divide the stream into chunks
216+(arrays), do the counting within each chunk, and then add all the
217+counts across chunks. We use the same code as above except that we use
218+arrays for our input data.
184219
185−Get the line, word, char counts in one chunk.
220+Please see the file
221+[WordCountParallel.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountParallel.hs)
222+for the complete working code for this example, including the imports
223+that we have omitted below.
186224
187−``` haskell
225+The `countArray` function counts the line, word, char counts in one chunk:
226+
227+```haskell
188228import qualified Streamly.Data.Array.Foreign as Array
189229
190230countArray :: Array Word8 -> IO Counts
@@ -193,14 +233,18 @@ countArray arr =
193233 & Stream.decodeLatin1 -- SerialT IO Char
194234 & Stream.foldl' count (Counts 0 0 0 True) -- IO Counts
195235```
196−When combining the counts in two contiguous chunks, we would also need
197−to know whether the first element of the next chunk was a space char or
198−non-space to know whether the same word is continuing to the next chunk
199−or if it is a new word. `partialCounts` adds a `Bool` flag to `Counts`
200−returned by `countArray` to indicate whether the first character in the
201−chunk is a space.
202−
203−``` haskell
236+
237+Here the function `count` and the `Counts` data type are defined in the
238+`WordCount` helper module defined in [WordCount.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.hs).
239+
240+When combining the counts in two contiguous chunks, we need to check
241+whether the first element of the next chunk is a whitespace character
242+in order to determine if the same word continues in the next chunk or
243+whether the chunk starts with a new word. The `partialCounts` function
244+adds a `Bool` flag to `Counts` returned by `countArray` to indicate
245+whether the first character in the chunk is a space.
246+
247+```haskell
204248partialCounts :: Array Word8 -> IO (Bool, Counts)
205249partialCounts arr = do
206250 let r = Array.getIndex arr 0
@@ -211,20 +255,23 @@ partialCounts arr = do
211255 Nothing -> return (False, Counts 0 0 0 True)
212256```
213257
214−Combine the counts from two consecutive chunks.
215−``` haskell
258+`addCounts` then adds the counts from two consecutive chunks:
259+
260+```haskell
216261addCounts :: (Bool, Counts) -> (Bool, Counts) -> (Bool, Counts)
217262addCounts (sp1, Counts l1 w1 c1 ws1) (sp2, Counts l2 w2 c2 ws2) =
218263 let wcount =
219− if not ws1 && not sp2 -- no space between two chunks
264+ if not ws1 && not sp2 -- No space between two chunks.
220265 then w1 + w2 - 1
221266 else w1 + w2
222267 in (sp1, Counts (l1 + l2) wcount (c1 + c2) ws2)
223268```
224269
225−Now put it all together, we only need to divide the stream into arrays,
226−apply our counting function to each array and then combine all the counts.
227−``` haskell
270+To count in parallel we now only need to divide the stream into arrays,
271+apply our counting function to each array, and then combine the counts
272+from each chunk.
273+
274+```haskell
228275wc :: String -> IO (Bool, Counts)
229276wc file = do
230277 Stream.unfold File.readChunks file -- AheadT IO (Array Word8)
@@ -234,12 +281,14 @@ wc file = do
234281 & Stream.foldl' addCounts (False, Counts 0 0 0 True) -- IO (Bool, Counts)
235282```
236283
237−Note that `Stream.fromAhead` is the only difference in a concurrent and
238−non-concurrent program. If we remove that we still have a perfectly valid,
239−well performing serial program. Notice, how succinctly and idiomatically
240−we expressed the concurrent word counting problem.
284+Please note that the only difference between a concurrent and a
285+non-concurrent program lies in the use of the `Stream.fromAhead`
286+combinator. If we remove the call to `Stream.fromAhead`, we would
287+still have a perfectly valid and performant serial program. Notice
288+how succinctly and idiomatically we have expressed the concurrent word
289+counting problem.
241290
242−Benchmarked with 2 CPUs:
291+A benchmark with 2 CPUs:
243292
244293```
245294$ time WordCount-hs-parallel gutenberg-500MB.txt
@@ -250,32 +299,40 @@ user 0m1.952s
250299sys 0m0.140s
251300```
252301
253−If you want to get serious about word counting, here is a
254−[concurrent wc implementation with UTF-8 decoding](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountUTF8.hs).
255−It performs as well as the stock wc in serial benchmarks, and of course
256−in concurrent mode it can use multiple cores so can be much faster.
257−
258−Streamly provides concurrency facilities similar to
259−[OpenMP](https://en.wikipedia.org/wiki/OpenMP) and
302+These example programs have assumed ASCII encoded
303+input data. For UTF-8 streams, we have a [concurrent wc
304+implementation](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountUTF8.hs)
305+with UTF-8 decoding. This concurrent implementation performs as well
306+as the standard `wc` program in serial benchmarks. In concurrent mode
307+[Streamly][]'s implementation can utilise multiple processing cores if
308+these are present, and can thereby run much faster than the standard
309+binary.
310+
311+Streamly provides concurrency facilities similar
312+to [OpenMP](https://en.wikipedia.org/wiki/OpenMP) and
260313[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative
261−expression. You can write concurrent loops with ease, with different types of
262−concurrent scheduling.
314+style of expression. With Streamly you can write concurrent programs
315+with ease, with support for different types of concurrent scheduling.
263316
264−## Concurrent Network Server
317+### A Concurrent Network Server
265318
266−Slightly more complicated example. A dictionary lookup server, the server
267−serves word meanings to multiple clients concurrently. It uses the concurrent
319+We now move to a slightly more complicated example: we simulate a
320+dictionary lookup server which can serve word meanings to multiple
321+clients concurrently. This example demonstrates the use of the concurrent
268322`mapM` combinator.
269323
270−See
324+Please see the file
271325[WordServer.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordServer.hs)
272−for full working code including the imports that we may have omitted below.
326+for the complete code for this example, including the imports that we
327+have omitted below.
273328
274−``` haskell
329+```haskell
330+import qualified Streamly.Data.Fold as Fold
275331import qualified Streamly.Network.Inet.TCP as TCP
332+import qualified Streamly.Network.Socket as Socket
276333import qualified Streamly.Unicode.Stream as Unicode
277334
278−-- Simulate network/db query by adding a delay
335+-- Simulate network/db query by adding a delay.
279336fetch :: String -> IO (String, String)
280337fetch w = threadDelay 1000000 >> return (w,w)
281338
@@ -311,23 +368,26 @@ main =
311368 & Stream.drain -- IO ()
312369```
313370
314−## Merging Incoming Streams
371+### Merging Incoming Streams
315372
316−Assume you have logs coming from multiple nodes in your network and
317−you want to merge all the logs at line boundaries and send the merged
318−stream to a file or to a network destination. It uses the amazing
319−`concatMapWith` combinator to merge multiple streams concurrently.
373+In the next example, we show how to merge logs coming from multiple
374+nodes in your network. These logs are merged at line boundaries and
375+the merged logs are written to a file or to a network destination.
376+This example uses the `concatMapWith` combinator to merge multiple
377+streams concurrently.
320378
321−See
379+Please see the file
322380[MergeServer.hs](https://github.com/composewell/streamly-examples/blob/master/examples/MergeServer.hs)
323−for full working code including the imports that we may have omitted below.
381+for the complete working code, including the imports that we have
382+omitted below.
324383
325−``` haskell
384+```haskell
326385import qualified Streamly.Data.Unfold as Unfold
327386import qualified Streamly.Network.Socket as Socket
328387
329−-- | Read a line stream from a socket. Note, lines are buffered, we could add
330−-- a limit to the buffering for safety.
388+-- | Read a line stream from a socket.
389+-- Note: lines are buffered, and we could add a limit to the
390+-- buffering for safety.
331391readLines :: Socket -> SerialT IO (Array Char)
332392readLines sk =
333393 Stream.unfold Socket.read sk -- SerialT IO Word8
@@ -353,30 +413,30 @@ main :: IO ()
353413main = withFile "output.txt" AppendMode server
354414```
355415
356−## Listing Directories Recursively/Concurrently
416+### Listing Directories Recursively/Concurrently
357417
358−The following example lists a directory tree recursively, reading
418+Our next example lists a directory tree recursively, reading
359419multiple directories concurrently.
360420
361−It uses the wonderful tree traversing combinator
362−`iterateMapLeftsWith`. It maps a stream generator on the `Left` values
363−(directories in this case) of the input stream, feeds the resulting
364−'Left' values back to the input, and lets the `Right` values (files in
365−this case) pass through to the output. The `Stream.ahead` stream joining
366−combinator makes it iterate on the directories concurrently.
421+This example uses the tree traversing combinator `iterateMapLeftsWith`.
422+This combinator maps a stream generator on the `Left` values in its
423+input stream (directory names in this case), feeding the resulting `Left`
424+values back to the input, while it lets the `Right` values (file names
425+in this case) pass through to the output. The `Stream.ahead` stream
426+joining combinator then makes it iterate on the directories concurrently.
367427
368−See
428+Please see the file
369429[ListDir.hs](https://github.com/composewell/streamly-examples/blob/master/examples/ListDir.hs)
370−for full working code including the imports that we may have omitted below.
430+for the complete working code, including the imports that we have
431+omitted below.
371432
372433```haskell
373−...
374434import Streamly.Internal.Data.Stream.IsStream (iterateMapLeftsWith)
375435
376436import qualified Streamly.Prelude as Stream
377437import qualified Streamly.Internal.FileSystem.Dir as Dir (toEither)
378438
379−-- Lists a dir as a stream of (Either Dir File)
439+-- Lists a directory as a stream of (Either Dir File).
380440listDir :: String -> SerialT IO (Either String String)
381441listDir dir =
382442 Dir.toEither dir -- SerialT IO (Either String String)
@@ -384,7 +444,7 @@ listDir dir =
384444
385445 where mkAbs x = dir ++ "/" ++ x
386446
387−-- | List the current directory recursively using concurrent processing
447+-- | List the current directory recursively using concurrent processing.
388448main :: IO ()
389449main = do
390450 hSetBuffering stdout LineBuffering
@@ -393,12 +453,13 @@ main = do
393453 & Stream.mapM_ print
394454```
395455
396−## Rate Limiting
456+### Rate Limiting
397457
398−For bounded concurrent streams, stream yield rate can be specified. For
399−example, to print "tick" once every second you can simply write this:
458+For bounded concurrent streams, a stream yield rate can be specified
459+easily. For example, to print "tick" once every second you can simply
460+write:
400461
401−``` haskell
462+```haskell
402463main :: IO ()
403464main =
404465 Stream.repeatM (pure "tick") -- AsyncT IO String
@@ -408,133 +469,139 @@ main =
408469 & Stream.mapM_ print -- IO ()
409470```
410471
411−See
472+Please see the file
412473[Rate.hs](https://github.com/composewell/streamly-examples/blob/master/examples/Rate.hs)
413−for full working code.
474+for the complete working code.
475+
476+The concurrency of the stream is automatically controlled to match the
477+specified rate. [Streamly][]'s rate control works precisely even at
478+throughputs as high as millions of yields per second.
414479
415−Concurrency of the stream is automatically controlled to match the specified
416−rate. Rate control works precisely even at throughputs as high as millions of
417−yields per second. For more sophisticated rate control see the haddock
418−documentation.
480+For more sophisticated rate control needs please see the [Haddock
481+documentation](https://hackage.haskell.org/package/streamly-0.7.3/docs/Streamly.html)
482+for the Streamly.
419483
420−## Reactive Programming
484+### Reactive Programming
421485
422−Streamly supports reactive and time domain programming inherently because of
423−declarative concurrency. See the `Streamly.Prelude` module for some time
424−specific combinators like `intervalsOf` and folds like `takeInterval` in
425−`Streamly.Internal.Data.Fold`. Also see pre-release sampling combinators in
426−the `Streamly.Internal.Data.Stream.IsStream.Top` module including `throttle`
427−and `debounce` like operations.
486+Streamly supports reactive (time domain) programming because of its
487+support for declarative concurrency. Please see the `Streamly.Prelude`
488+module for time-specific combinators like `intervalsOf`, and
489+folds like `takeInterval` in `Streamly.Internal.Data.Fold`.
490+Please also see the pre-release sampling combinators in the
491+`Streamly.Internal.Data.Stream.IsStream.Top` module for `throttle` and
492+`debounce` like operations.
428493
429−See
494+The examples
430495[AcidRain.hs](https://github.com/composewell/streamly-examples/tree/master/examples/AcidRain.hs)
431496and
432−[CirclingSquare.hs](https://github.com/composewell/streamly-examples/tree/master/examples/CirclingSquare.hs).
433−
434−## More examples
435−
436−Many more examples can be found in the [streamly-examples
437−repository](https://github.com/composewell/streamly-examples).
438−
439−Streamly comes equipped with a very powerful set of abstractions to accomplish
440−any kind of programming tasks that you may want to throw at it. It provides,
441−streams, arrays, file-io, fsnotify, network-io, time domain programming
442−(reactive programming). See the [streamly
443−documentation](https://streamly.composewell.com) to learn more.
444−
445−## Concurrency
446−
447−Streamly uses lock-free synchronization for low overhead
448−concurrency. The number of tasks performed concurrently are
449−determined automatically based on the rate at which a consumer is
450−consuming the results. In other words, you do not need to manage
451−thread pools and decide how many threads to use for a particular
452−task. For CPU bound tasks it tries to keep the number of threads
453−close to the number of CPUs available whereas for IO bound tasks
454−more threads can be utilized.
455−
456−Parallelism can be utilized with little overhead even if
457−the task size is very small, because it can automatically
458−switch to serial mode or batch multiple tasks on the
459−same CPU if that is more efficient. See [concurrency
460−benchmarks](https://github.com/composewell/concurrency-benchmarks) for
461−detailed performance results and a comparison with the `async` package.
497+[CirclingSquare.hs](https://github.com/composewell/streamly-examples/tree/master/examples/CirclingSquare.hs)
498+demonstrate reactive programming using [Streamly][].
499+
500+### More Examples
501+
502+If you would like to view more examples, please visit the [Streamly
503+Examples](https://github.com/composewell/streamly-examples) web page.
504+
505+### Further Reading
506+
507+* [Streaming Benchmarks](https://github.com/composewell/streaming-benchmarks) | [Concurrency Benchmarks](https://github.com/composewell/concurrency-benchmarks)
508+* [Functional Conf 2019 Video](https://www.youtube.com/watch?v=uzsqgdMMgtk) | [Functional Conf 2019 Slides](https://www.slideshare.net/HarendraKumar10/streamly-concurrent-data-flow-programming)
509+* [Other Guides](docs/)
510+* [Streamly Homepage](https://streamly.composewell.com)
462511
463512## Performance
464513
465−As you have seen above in the word count example, streamly enables
466−highly modular abstractions with the best possible performance (close to
467−an equivalent C program).
514+As you have seen in the word count example above, [Streamly][] offers
515+highly modular abstractions for building programs while also offering
516+the performance close to an equivalent (imperative) C program.
468517
469−Streamly provides excellent performance even for byte level stream
470−operations, it is made possible by employing efficient abstractions like
471−`Unfold`s and terminating `Fold`s. Byte level stream operations make
472−programming simpler because you do not have to deal with chunking and
473−re-combining.
518+Streamly offers excellent performance even for byte-at-a-time stream
519+operations using efficient abstractions like `Unfold`s and terminating
520+`Fold`s. Byte-at-a-time stream operations can simplify programming
521+because the developer does not have to deal explicitly with chunking
522+and re-combining data.
474523
475−If you can write a program significantly faster in some other way or
476−with some other language, please let us know and we will improve.
524+Streamly exploits GHC's stream fusion optimizations (`case-of-case` and
525+`spec-constr`) aggressively to achieve C-like speed, while also offering
526+highly modular abstractions to developers.
477527
478−## Benchmarks
528+[Streamly][] will usually perform very well without any
529+compiler plugins. However, we have fixed some deficiencies
530+that we had noticed in GHC's optimizer using a [compiler
531+plugin](https://github.com/composewell/fusion-plugin). We hope to fold
532+these optimizations into GHC in the future; until then we recommend that
533+you use this plugin for applications that are performance sensitive.
534+
535+### Benchmarks
479536
480537We measured several Haskell streaming implementations
481−on various micro-benchmarks. Please see [streaming
538+using various micro-benchmarks. Please see the [streaming
482539benchmarks](https://github.com/composewell/streaming-benchmarks) page
483−for detailed comparison of streamly with other streaming libraries.
540+for a detailed comparison of Streamly against other streaming libraries.
541+
542+Our results show that [Streamly][] is the fastest effectful streaming
543+implementation on almost all the measured microbenchmarks. In many cases
544+it runs up to 100x faster, and in some cases even 1000x faster than
545+some of the tested alternatives. In some composite operation benchmarks
546+[Streamly][] turns out to be significantly faster than Haskell's list
547+implementation.
548+
549+*Note*: If you can write a program in some other way or with some other
550+language that runs significantly faster than what [Streamly][] offers,
551+please let us know and we will improve.
552+
553+## Notes
554+
555+Streamly comes equipped with a very powerful set of abstractions to
556+accomplish many kinds of programming tasks: it provides support for
557+programming with streams and arrays, for reading and writing from the
558+file system and from the network, for time domain programming (reactive
559+programming), and for reacting to file system events using `fsnotify`.
560+
561+Please view [Streamly's documentation](https://streamly.composewell.com)
562+for more information about Streamly's features.
563+
564+### Concurrency
565+
566+Streamly uses lock-free synchronization for achieving concurrent
567+operation with low overheads. The number of tasks performed concurrently
568+are determined automatically based on the rate at which a consumer
569+is consuming the results. In other words, you do not need to manage
570+thread pools or decide how many threads to use for a particular task.
571+For CPU-bound tasks Streamly will try to keep the number of threads
572+close to the number of CPUs available; for IO-bound tasks it will utilize
573+more threads.
574+
575+The parallelism available during program execution can be utilized with
576+very little overhead even where the task size is very
577+small, because Streamly will automatically switch between
578+serial or batched execution of tasks on the same CPU depending
579+on whichever is more efficient. Please see our [concurrency
580+benchmarks](https://github.com/composewell/concurrency-benchmarks) for
581+more detailed performance measurements, and for a comparison with the
582+`async` package.
484583
485−These results show that streamly is the fastest effectful streaming
486−implementation on almost all the measured micro benchmarks. In many cases
487−it is up to 100x faster and in some cases even 1000x faster. In many
488−composite operation benchmarks streamly turns out to be significantly
489−faster than the Haskell lists implementation.
584+### Design Goals
490585
491−## Design Goals
586+Our goals for [Streamly][] from the very beginning have been:
492587
493−The goals of streamly from the very beginning have been, (1) simplicity
494−by unifying abstractions, (2) high performance. These are hard to
495−achieve at the same time because they are usually inversely related. We
496−have spent many years trying to get the abstractions right without
497−compromising performance.
588+1. To achieve simplicity by unifying abstractions.
589+2. To offer high performance.
498590
499−`Unfold` is an example of an abstraction that we have created to
500−achieve high performance when mapping streams on streams. It allows
501−stream generation to be optimized well by the compiler, employing
502−stream fusion. `Fold` with termination capability is another example
503−which modularizes stream elimination operations with stream fusion.
504−Terminating folds can perform many simple parsing tasks that do not
505−require backtracking. `Parser`s in streamly are a natural extension
506−of terminating `Fold`s just adding backtracking capability to folds.
507−Unification leads to simpler abstractions, lesser cognitive overhead
508−without compromising performance.
509−
510−Streamly exploits GHC stream fusion optimizations (`case-of-case`
511−and `spec-constr`) aggressively to bring C like speed with highly
512−modular abstractions. It performs very well without any compiler
513−plugins. However, we have fixed some deficiencies in GHC optimizer
514−via a [compiler plugin](https://github.com/composewell/fusion-plugin).
515−We hope to bring these optimizations to GHC in future but until
516−then we recommend that you use the plugin for performance sensitive
517−applications.
518−
519−## A Note on Internal Modules
520−
521−You may see some `Internal` modules imported in the examples. Those are
522−not really internal to the library. We can classify streamly modules into
523−two categories:
524−
525−* Released Modules: These are the modules that have a stable API, any
526− API changes conform to the package versioning policy.
527−* Pre-release Modules: These modules are not yet released due to some planned
528− changes in near future, they will be released soon. These modules reside in
529− the `Internal` namespace for now.
530−
531−Pre-release APIs: Some of the APIs that are recently introduced
532−and require some soak time for stability are kept in the
533−internal modules corresponding to the released module (e.g.
534−Streamly.Internal.Data.Fold).
535−
536−We usually try to change even the unstable APIs in `Internal` module
537−namespace in a major release version.
591+These goals are hard to achieve simultaneously because they are usually
592+inversely related. We have spent many years trying to get the abstractions
593+right without compromising performance.
594+
595+`Unfold` is an example of an abstraction that we have created to achieve
596+high performance when mapping streams on streams. `Unfold` allows stream
597+generation to be optimized well by the compiler through stream fusion.
598+A `Fold` with termination capability is another example which modularizes
599+stream elimination operations through stream fusion. Terminating folds
600+can perform many simple parsing tasks that do not require backtracking.
601+In Streamly, `Parser`s are a natural extension to terminating `Fold`s;
602+`Parser`s add the ability to backtrack to `Fold`s. Unification leads
603+to simpler abstractions and lower cognitive overheads while also not
604+compromising performance.
538605
539606## Credits
540607
@@ -545,20 +612,26 @@ significant way:
545612 * Gabriel Gonzalez ([foldl](https://hackage.haskell.org/package/foldl))
546613 * Alberto G. Corona ([transient](https://hackage.haskell.org/package/transient))
547614
548−See the `credits` directory for full list of contributors, credits and licenses.
615+Please see the [`credits`](./credits/README.md) directory for a full
616+list of contributors, credits and licenses.
549617
550−## Contributing
618+## Licensing
619+
620+Streamly is an [open source](https://github.com/composewell/streamly)
621+project available under a liberal [BSD-3-Clause
622+license](https://github.com/composewell/streamly/blob/master/LICENSE).
623+
624+## Contributing to Streamly
625+
626+As an open project we welcome contributions:
551627
552628* [Streamly Contributor's Guide](CONTRIBUTING.md)
553629* [Contact the streamly development team](mailto:streamly@composewell.com)
554630
555−## License
556−
557−Streamly is an [open source](https://github.com/composewell/streamly)
558−project available under [BSD-3-Clause
559−license](https://github.com/composewell/streamly/blob/master/LICENSE).
631+## Getting Support
560632
561−## Support
633+Professional support is available for [Streamly][]: please contact
634+[support@composewell.com](mailto:support@composewell.com).
562635
563−* Join streamly [online community chat](https://gitter.im/composewell/streamly)
564−* For professional support please contact [support@composewell.com](mailto:support@composewell.com)
636+You can also join our [community chat
637+channel](https://gitter.im/composewell/streamly) on Gitter.