@@ -1,17 +1,4 @@
1−# [Streamly][]: Idiomatic Haskell with the Performance of C
2−
3−[](https://gitter.im/composewell/streamly)
4−[](https://hackage.haskell.org/package/streamly)
5−
6−Streamly is a Haskell library that provides the building blocks to build
7−safe, scalable, modular and high performance software. Streamly offers:
8−
9−* The type safety of Haskell.
10−* The performance of C programs.
11−* Powerful abstractions for structuring your code.
12−* Idiomatic functional programming.
13−* Declarative concurrency for the seamless use of multiprocessing hardware.
14−
1+# [Streamly][] Quick Tutorial
152## About This Document
163
174This guide introduces programming with [Streamly][] using a few practical
@@ -31,10 +18,7 @@ examples:
3118* Finally, we will look at how to [rate limit](#rate-limiting) stream
3219 processing.
3320
34−The guide then looks at how Streamly achieves its
35−[performance](#performance). It [concludes](#notes) with a brief
36−discussion about Streamly's design philosophy, and with suggestions for
37−further reading.
21+It [concludes](#further-reading) with suggestions for further reading.
3822
3923## Getting Started
4024
@@ -55,7 +39,8 @@ As an expository device, we have indicated the types at the intermediate
5539stages of stream computations as comments in the examples below.
5640The meaning of these types are:
5741
58−* A `Stream IO a` is a serial stream of values of type `a` in the IO Monad.
42+* A `Stream IO a` is a representation of a sequence of values of type
43+ `a` in the IO Monad.
5944* An `Unfold IO a b` is a representation of a function that converts a seed
6045 value of type `a` into a stream of values of type `b` in the IO Monad.
6146* A `Fold IO a b` is a representation of a function that converts a stream of
@@ -82,6 +67,10 @@ using the pre-release (internal) modules.
8267
8368## The Examples
8469
70+The code snippets below should work in GHCi if all of those are typed
71+in sequence. For brevity, imports that are already used in earlier
72+snippets are omitted from the latter ones.
73+
8574### Modular Word Counting
8675
8776A `Fold` in Streamly is a composable stream consumer. For our first
@@ -90,7 +79,7 @@ present in a file. We will then compose individual `Fold`s together to
9079count words, bytes and lines at the same time.
9180
9281Please see the file [WordCountModular.hs][] for the complete example
93−program, including the imports that we have omitted here.
82+program.
9483
9584#### Count Bytes (wc -c)
9685
@@ -121,7 +110,7 @@ import Streamly.Data.Fold (Fold)
121110countl :: Int -> Word8 -> Int
122111countl n ch = if ch == 10 then n + 1 else n
123112
124−-- The fold accepts a stream of `Word8` and returns a line count (`Int`).
113+-- The `nlines` fold accepts a stream of `Word8` and returns a line count (`Int`).
125114nlines :: Monad m => Fold m Word8 Int
126115nlines = Fold.foldl' countl 0
127116
@@ -145,7 +134,7 @@ countw (n, wasSpace) ch =
145134 then (n, True)
146135 else (if wasSpace then n + 1 else n, False)
147136
148−-- The fold accepts a stream of `Word8` and returns a word count (`Int`).
137+-- The `nwords` fold accepts a stream of `Word8` and returns a word count (`Int`).
149138nwords :: Monad m => Fold m Word8 Int
150139nwords = fst <$> Fold.foldl' countw (0, True)
151140
@@ -178,9 +167,7 @@ wc file =
178167```
179168
180169This example demonstrates the excellent modularity offered by
181−[Streamly][]'s simple and concise API. Experienced Haskellers will
182−notice that we have not used bytestrings—we instead used a stream of
183−`Word8` values, simplifying our program.
170+[Streamly][]'s simple and concise API.
184171
185172### The Performance of Word Counting
186173
@@ -188,15 +175,15 @@ We compare two equivalent implementations: one using [Streamly][],
188175and the other using C.
189176
190177The performance of the [Streamly word counting
191−implementation][WordCount.hs] is:
178+implementation][WordCount.hs] (using ghc-9.4.4 and fusion-plugin) is:
192179
193180```
194181$ time WordCount-hs gutenberg-500MB.txt
19518211242220 97050938 574714449 gutenberg-500MB.txt
196183
197−real 0m1.825s
198−user 0m1.697s
199−sys 0m0.128s
184+real 0m2.033s
185+user 0m1.821s
186+sys 0m0.209s
200187```
201188
202189The performance of an equivalent [wc implementation in C][WordCount.c] is:
@@ -205,9 +192,9 @@ The performance of an equivalent [wc implementation in C][WordCount.c] is:
205192$ time WordCount-c gutenberg-500MB.txt
20619311242220 97050938 574714449 gutenberg-500MB.txt
207194
208−real 0m2.100s
209−user 0m1.935s
210−sys 0m0.165s
195+real 0m2.113s
196+user 0m1.928s
197+sys 0m0.185s
211198```
212199
213200### Concurrent Word Counting
@@ -318,9 +305,9 @@ A benchmark with 2 CPUs:
318305$ time WordCount-hs-parallel gutenberg-500MB.txt
31930611242220 97050938 574714449 gutenberg-500MB.txt
320307
321−real 0m1.284s
322−user 0m1.952s
323−sys 0m0.140s
308+real 0m1.443s
309+user 0m2.095s
310+sys 0m0.202s
324311```
325312
326313These example programs have assumed ASCII encoded input data. For UTF-8
@@ -344,7 +331,7 @@ dictionary lookup server which can serve word meanings to multiple
344331clients concurrently.
345332
346333Please see the file [WordServer.hs][] for the complete code for this
347−example, including the imports that we have omitted below.
334+example.
348335
349336```haskell ghci
350337import Control.Concurrent (threadDelay)
@@ -517,131 +504,6 @@ Examples][streamly-examples] web page.
517504* [Other Guides](/)
518505* [Streamly Homepage][Streamly]
519506
520−## Performance
521−
522−As you have seen in the word count example above, [Streamly][] offers
523−highly modular abstractions for building programs while also offering
524−the performance close to an equivalent (imperative) C program.
525−
526−Streamly offers excellent performance even for byte-at-a-time stream
527−operations using efficient abstractions like `Unfold`s and terminating
528−`Fold`s. Byte-at-a-time stream operations can simplify programming
529−because the developer does not have to deal explicitly with chunking
530−and re-combining data.
531−
532−Streamly exploits GHC's stream fusion optimizations (`case-of-case` and
533−`spec-constr`) aggressively to achieve C-like speed, while also offering
534−highly modular abstractions to developers.
535−
536−[Streamly][] will usually perform very well without any
537−compiler plugins. However, we have fixed some deficiencies
538−that we had noticed in GHC's optimizer using a [compiler
539−plugin](https://github.com/composewell/fusion-plugin). We hope to fold
540−these optimizations into GHC in the future; until then we recommend that
541−you use this plugin for applications that are performance sensitive.
542−
543−### Benchmarks
544−
545−We measured several Haskell streaming implementations
546−using various micro-benchmarks. Please see the [streaming
547−benchmarks][streaming-benchmarks] page for a detailed comparison of
548−Streamly against other streaming libraries.
549−
550−Our results show that [Streamly][] is the fastest effectful streaming
551−implementation on almost all the measured microbenchmarks. In many cases
552−it runs up to 100x faster, and in some cases even 1000x faster than
553−some of the tested alternatives. In some composite operation benchmarks
554−[Streamly][] turns out to be significantly faster than Haskell's list
555−implementation.
556−
557−*Note*: If you can write a program in some other way or with some other
558−language that runs significantly faster than what [Streamly][] offers,
559−please let us know and we will improve.
560−
561−## Notes
562−
563−Streamly comes equipped with a very powerful set of abstractions to
564−accomplish many kinds of programming tasks: it provides support for
565−programming with streams and arrays, for reading and writing from the
566−file system and from the network, for time domain programming (reactive
567−programming), and for reacting to file system events using `fsnotify`.
568−
569−Please view [Streamly's documentation][Streamly] for more information
570−about Streamly's features.
571−
572−### Concurrency
573−
574−Streamly uses lock-free synchronization for achieving concurrent
575−operation with low overheads. The number of tasks performed concurrently
576−are determined automatically based on the rate at which a consumer
577−is consuming the results. In other words, you do not need to manage
578−thread pools or decide how many threads to use for a particular task.
579−For CPU-bound tasks Streamly will try to keep the number of threads
580−close to the number of CPUs available; for IO-bound tasks it will utilize
581−more threads.
582−
583−The parallelism available during program execution can be utilized with
584−very little overhead even where the task size is very
585−small, because Streamly will automatically switch between
586−serial or batched execution of tasks on the same CPU depending
587−on whichever is more efficient. Please see our [concurrency
588−benchmarks][concurrency-benchmarks] for more detailed performance
589−measurements, and for a comparison with the `async` package.
590−
591−### Design Goals
592−
593−Our goals for [Streamly][] from the very beginning have been:
594−
595−1. To achieve simplicity by unifying abstractions.
596−2. To offer high performance.
597−
598−These goals are hard to achieve simultaneously because they are usually
599−inversely related. We have spent many years trying to get the abstractions
600−right without compromising performance.
601−
602−`Unfold` is an example of an abstraction that we have created to achieve
603−high performance when mapping streams on streams. `Unfold` allows stream
604−generation to be optimized well by the compiler through stream fusion.
605−A `Fold` with termination capability is another example which modularizes
606−stream elimination operations through stream fusion. Terminating folds
607−can perform many simple parsing tasks that do not require backtracking.
608−In Streamly, `Parser`s are a natural extension to terminating `Fold`s;
609−`Parser`s add the ability to backtrack to `Fold`s. Unification leads
610−to simpler abstractions and lower cognitive overheads while also not
611−compromising performance.
612−
613−## Credits
614−
615−The following authors/libraries have influenced or inspired this library in a
616−significant way:
617−
618− * Roman Leshchinskiy ([vector](http://hackage.haskell.org/package/vector))
619− * Gabriella Gonzalez ([foldl](https://hackage.haskell.org/package/foldl))
620− * Alberto G. Corona ([transient](https://hackage.haskell.org/package/transient))
621−
622−Please see the [`credits`](/docs/User/ProjectRelated/Credits.md) directory for a full
623−list of contributors, credits and licenses.
624−
625−## Licensing
626−
627−Streamly is an [open source](https://github.com/composewell/streamly)
628−project available under a liberal [BSD-3-Clause license][LICENSE]
629−
630−## Contributing to Streamly
631−
632−As an open project we welcome contributions:
633−
634−* [Streamly Contributor's Guide][CONTRIBUTING.md]
635−* [Contact the streamly development team](mailto:streamly@composewell.com)
636−
637−## Getting Support
638−
639−Professional support is available for [Streamly][]: please contact
640−[support@composewell.com](mailto:support@composewell.com).
641−
642−You can also join our [community chat
643−channel](https://gitter.im/composewell/streamly) on Gitter.
644−
645507<!--
646508Link References.
647509-->