@@ -18,8 +18,8 @@
1818## Idiomatic Haskell at the speed of C
1919
2020Streamly is a Haskell library/framework providing basic building blocks
21−or combinators to perform fundamental as well as advanced programming
22−tasks with ease. The key features it provides are:
21+or combinators to build safe, scalable, modular and high performance
22+software systems. The key features it provides are:
2323
2424* Speed of C
2525* Safety of Haskell
@@ -43,10 +43,10 @@ repository](https://github.com/composewell/streamly-examples).
4343
4444## Modular Word Counting
4545
46−The `Fold` data type in streamly represents a consumer of stream. In
47−this example, we will use individual folds to count bytes, words and
48−lines in a file. We will see how these individual folds can be composed
49−together to do all the three at once with the same performance.
46+In this example, we will use folds to count bytes, words and lines in
47+a file. A `Fold` is a composable stream consumer. We will see how
48+individual folds can be composed together to perform all the three
49+counts at once with the same performance.
5050
5151See [WordCountModular.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountModular.hs)
5252for full working code including imports that we may have omitted
@@ -129,13 +129,10 @@ wc file =
129129 & Stream.fold countAll -- IO (Int, Int, Int)
130130```
131131
132−### Observations
133−
134−This example shows:
135−
136−* Excellent modularity
137−* Simple and concise API and types
138−* No bytestrings required, just streams of Word8
132+This example demonstrates the simple and concise API of streamly with
133+excellent modularity. Experienced Haskellers would notice that we have
134+not used bytestrings, we simply use a stream of `Word8`, simplifying the
135+program.
139136
140137## Performance
141138
@@ -165,7 +162,7 @@ sys 0m0.165s
165162
166163## Concurrent Word Counting
167164
168−To do word counting in parallel we divide the stream in chunks (arrays),
165+To count words in parallel we divide the stream into chunks (arrays),
169166count properties in each chunk and then add all the counts. We use the
170167same code as above except that we use an array input instead of using a
171168file input.
@@ -185,10 +182,12 @@ countArray arr =
185182 & Stream.decodeLatin1 -- SerialT IO Char
186183 & Stream.foldl' count (Counts 0 0 0 True) -- IO Counts
187184```
188−When combining the counts in two contiguous chunks, we would also need to
189−know whether the first element of the next chunk was a space char or
190−non-space to know whether the same word is continuing to the next chunk or
191−if it is a new word. So add that too, giving (firstCharWasSpace, Counts).
185+When combining the counts in two contiguous chunks, we would also need
186+to know whether the first element of the next chunk was a space char or
187+non-space to know whether the same word is continuing to the next chunk
188+or if it is a new word. `partialCounts` adds a `Bool` flag to `Counts`
189+returned by `countArray` to indicate whether the first character in the
190+chunk is a space.
192191
193192``` haskell
194193partialCounts :: Array Word8 -> IO (Bool, Counts)
@@ -224,7 +223,7 @@ wc file = do
224223 & Stream.foldl' addCounts (False, Counts 0 0 0 True) -- IO (Bool, Counts)
225224```
226225
227−Note that `Stream.aheadly` is the only difference in a concurrent and
226+Note that `Stream.fromAhead` is the only difference in a concurrent and
228227non-concurrent program. If we remove that we still have a perfectly valid,
229228well performing serial program. Notice, how succinctly and idiomatically
230229we expressed the concurrent word counting problem.
@@ -383,10 +382,6 @@ main = do
383382 & Stream.mapM_ print
384383```
385384
386−Notice, how succinctly and idiomatically we expressed the concurrent
387−directory tree traversal problem. Let us know if you can do it
388−significantly better with any other language or framework.
389−
390385## Rate Limiting
391386
392387For bounded concurrent streams, stream yield rate can be specified. For
@@ -438,13 +433,21 @@ documentation](https://streamly.composewell.com) to know more.
438433
439434## Concurrency
440435
441−Streamly uses lock-free synchronization for concurrent operations. It employs
442−auto-scaling of the degree of concurrency based on demand. For CPU bound tasks
443−it tries to keep the threads close to the number of CPUs available whereas for
444−IO bound tasks more threads can be utilized. Parallelism can be utilized with
445−little overhead even if the task size is very small. See [concurrency
446−benchmarks](https://github.com/composewell/concurrency-benchmarks) for detailed
447−performance results and a comparison with the `async` package.
436+Streamly uses lock-free synchronization for low overhead
437+concurrency. The number of tasks performed concurrently are
438+determined automatically based on the rate at which a consumer is
439+consuming the results. In other words, you do not need to manage
440+thread pools and decide how many threads to use for a particular
441+task. For CPU bound tasks it tries to keep the number of threads
442+close to the number of CPUs available whereas for IO bound tasks
443+more threads can be utilized.
444+
445+Parallelism can be utilized with little overhead even if
446+the task size is very small, because it can automatically
447+switch to serial mode or batch multiple tasks on the
448+same CPU if that is more efficient. See [concurrency
449+benchmarks](https://github.com/composewell/concurrency-benchmarks) for
450+detailed performance results and a comparison with the `async` package.
448451
449452## Performance
450453
@@ -452,41 +455,29 @@ As you have seen above in the word count example, streamly enables
452455highly modular abstractions with the best possible performance (close to
453456an equivalent C program).
454457
455−For example, none of the existing Haskell libraries can work with byte
456−level streams with acceptable performance. Streamly provides excellent
457−performance even for byte level stream operations, it is made possible
458−by employing efficient abstractions like `Unfold`s and terminating
459−`Fold`s. Byte level stream operations make programming simpler because
460−you do not have to deal with chunking and re-combining.
458+Streamly provides excellent performance even for byte level stream
459+operations, it is made possible by employing efficient abstractions like
460+`Unfold`s and terminating `Fold`s. Byte level stream operations make
461+programming simpler because you do not have to deal with chunking and
462+re-combining.
461463
462464If you can write a program significantly faster in some other way or
463−with some other language, please let us know and we will improve. We
464−still have some areas to improve but we will surely get there for most
465−cases.
465+with some other language, please let us know and we will improve.
466466
467467## Benchmarks
468468
469−When it comes to stream operations, Haskell lists is the fastest
470−existing implementation even though it supports only pure (not monadic)
471−streams. Other streaming libraries are many times slower compared to
472−lists, so lists is our gold standard for performance comparison.
473−
474−In the following chart, positive y-axis displays how many times worse
475−is a list operation compared to the corresponding streamly operation,
476−negative y-axis shows the same where streamly is worse compared to
477−lists.
478−
479−
480−
481−This chart show only those operations where streamly and list
482−performance differs by more than 10%. For all other operations the
483−performance of both is more or less comparable.
469+We measured several Haskell streaming implementations
470+on various micro-benchmarks. Please see [streaming
471+benchmarks](https://github.com/composewell/streaming-benchmarks) page
472+for detailed comparison of streamly with other streaming libraries.
484473
485−See [streaming
486−benchmarks](https://github.com/composewell/streaming-benchmarks) for
487−detailed comparison with other Haskell streaming libraries.
474+These results show that streamly is the fastest effectful streaming
475+implementation on almost all the measured micro benchmarks. In many cases
476+it is up to 100x faster and in some cases even 1000x faster. In many
477+composite operation benchmarks streamly turns out to be significantly
478+faster than the Haskell lists implementation.
488479
489−## Performance and Modularity
480+## Design Goals
490481
491482The goals of streamly from the very beginning have been, (1) simplicity
492483by unifying abstractions, (2) high performance. These are hard to