@@ -3,7 +3,37 @@
33[](https://gitter.im/composewell/streamly)
44[](https://hackage.haskell.org/package/streamly)
55
6− [Streamly]: https://streamly.composewell.com/
6+<!--
7+Link References.
8+-->
9+
10+[Streamly]: https://streamly.composewell.com/
11+[streamly-examples]: https://github.com/composewell/streamly-examples
12+[streaming-benchmarks]: https://github.com/composewell/streaming-benchmarks
13+[concurrency-benchmarks]: https://github.com/composewell/concurrency-benchmarks
14+
15+<!--
16+Keep all the unstable links here so that they can be updated to stable
17+links (for online docs) before we release.
18+-->
19+
20+<!-- examples -->
21+[WordCountModular.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/WordCountModular.hs
22+[WordCount.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.hs
23+[WordCount.c]: https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.c
24+[WordCountParallel.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/WordCountParallel.hs
25+[WordCountUTF8.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/WordCountUTF8.hs
26+[WordServer.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/WordServer.hs
27+[MergeServer.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/MergeServer.hs
28+[ListDir.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/ListDir.hs
29+[Rate.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/Rate.hs
30+[AcidRain.hs]: https://github.com/composewell/streamly-examples/tree/master/examples/AcidRain.hs
31+[CirclingSquare.hs]: https://github.com/composewell/streamly-examples/tree/master/examples/CirclingSquare.hs
32+
33+<!-- local files -->
34+[LICENSE]: https://github.com/composewell/streamly/blob/master/LICENSE
35+[CONTRIBUTING.md]: https://github.com/composewell/streamly/blob/master/CONTRIBUTING.md
36+[docs]: https://github.com/composewell/streamly/blob/master/docs/
737
838Streamly is a Haskell library that provides the building blocks to build
939safe, scalable, modular and high performance software. Streamly offers:
@@ -91,10 +121,8 @@ example, we will use `Fold`s to count the number of bytes, words and lines
91121present in a file. We will then compose individual `Fold`s together to
92122count words, bytes and lines at the same time.
93123
94−Please see the file
95−[WordCountModular.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountModular.hs)
96−for the complete example program, including the imports that we have
97−omitted here.
124+Please see the file [WordCountModular.hs][] for the complete example
125+program, including the imports that we have omitted here.
98126
99127#### Count Bytes (wc -c)
100128
@@ -185,7 +213,7 @@ We compare two equivalent implementations: one using [Streamly][],
185213and the other using C.
186214
187215The performance of the [Streamly word counting
188−implementation](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.hs) is:
216+implementation][WordCount.hs] is:
189217
190218```
191219$ time WordCount-hs gutenberg-500MB.txt
@@ -196,7 +224,7 @@ user 0m1.697s
196224sys 0m0.128s
197225```
198226
199−The performance of an equivalent [wc implementation in C](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.c) is:
227+The performance of an equivalent [wc implementation in C][WordCount.c] is:
200228
201229```
202230$ time WordCount-c gutenberg-500MB.txt
@@ -217,10 +245,8 @@ To count words in parallel we first divide the stream into chunks
217245counts across chunks. We use the same code as above except that we use
218246arrays for our input data.
219247
220−Please see the file
221−[WordCountParallel.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountParallel.hs)
222−for the complete working code for this example, including the imports
223−that we have omitted below.
248+Please see the file [WordCountParallel.hs][] for the complete working
249+code for this example, including the imports that we have omitted below.
224250
225251The `countArray` function counts the line, word, char counts in one chunk:
226252
@@ -235,7 +261,7 @@ countArray arr =
235261```
236262
237263Here the function `count` and the `Counts` data type are defined in the
238−`WordCount` helper module defined in [WordCount.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.hs).
264+`WordCount` helper module defined in [WordCount.hs][].
239265
240266When combining the counts in two contiguous chunks, we need to check
241267whether the first element of the next chunk is a whitespace character
@@ -299,9 +325,8 @@ user 0m1.952s
299325sys 0m0.140s
300326```
301327
302−These example programs have assumed ASCII encoded
303−input data. For UTF-8 streams, we have a [concurrent wc
304−implementation](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountUTF8.hs)
328+These example programs have assumed ASCII encoded input data. For UTF-8
329+streams, we have a [concurrent wc implementation][WordCountUTF8.hs]
305330with UTF-8 decoding. This concurrent implementation performs as well
306331as the standard `wc` program in serial benchmarks. In concurrent mode
307332[Streamly][]'s implementation can utilise multiple processing cores if
@@ -321,10 +346,8 @@ dictionary lookup server which can serve word meanings to multiple
321346clients concurrently. This example demonstrates the use of the concurrent
322347`mapM` combinator.
323348
324−Please see the file
325−[WordServer.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordServer.hs)
326−for the complete code for this example, including the imports that we
327−have omitted below.
349+Please see the file [WordServer.hs][] for the complete code for this
350+example, including the imports that we have omitted below.
328351
329352```haskell
330353import qualified Streamly.Data.Fold as Fold
@@ -376,10 +399,8 @@ the merged logs are written to a file or to a network destination.
376399This example uses the `concatMapWith` combinator to merge multiple
377400streams concurrently.
378401
379−Please see the file
380−[MergeServer.hs](https://github.com/composewell/streamly-examples/blob/master/examples/MergeServer.hs)
381−for the complete working code, including the imports that we have
382−omitted below.
402+Please see the file [MergeServer.hs][] for the complete working code,
403+including the imports that we have omitted below.
383404
384405```haskell
385406import qualified Streamly.Data.Unfold as Unfold
@@ -425,10 +446,8 @@ values back to the input, while it lets the `Right` values (file names
425446in this case) pass through to the output. The `Stream.ahead` stream
426447joining combinator then makes it iterate on the directories concurrently.
427448
428−Please see the file
429−[ListDir.hs](https://github.com/composewell/streamly-examples/blob/master/examples/ListDir.hs)
430−for the complete working code, including the imports that we have
431−omitted below.
449+Please see the file [ListDir.hs][] for the complete working code,
450+including the imports that we have omitted below.
432451
433452```haskell
434453import Streamly.Internal.Data.Stream.IsStream (iterateMapLeftsWith)
@@ -469,17 +488,14 @@ main =
469488 & Stream.mapM_ print -- IO ()
470489```
471490
472−Please see the file
473−[Rate.hs](https://github.com/composewell/streamly-examples/blob/master/examples/Rate.hs)
474−for the complete working code.
491+Please see the file [Rate.hs][] for the complete working code.
475492
476493The concurrency of the stream is automatically controlled to match the
477494specified rate. [Streamly][]'s rate control works precisely even at
478495throughputs as high as millions of yields per second.
479496
480−For more sophisticated rate control needs please see the [Haddock
481−documentation](https://hackage.haskell.org/package/streamly-0.7.3/docs/Streamly.html)
482−for the Streamly.
497+For more sophisticated rate control needs please see the Streamly [reference
498+documentation][Streamly].
483499
484500### Reactive Programming
485501
@@ -491,23 +507,21 @@ Please also see the pre-release sampling combinators in the
491507`Streamly.Internal.Data.Stream.IsStream.Top` module for `throttle` and
492508`debounce` like operations.
493509
494−The examples
495−[AcidRain.hs](https://github.com/composewell/streamly-examples/tree/master/examples/AcidRain.hs)
496−and
497−[CirclingSquare.hs](https://github.com/composewell/streamly-examples/tree/master/examples/CirclingSquare.hs)
498−demonstrate reactive programming using [Streamly][].
510+The examples [AcidRain.hs][] and [CirclingSquare.hs][] demonstrate
511+reactive programming using [Streamly][].
499512
500513### More Examples
501514
502515If you would like to view more examples, please visit the [Streamly
503−Examples](https://github.com/composewell/streamly-examples) web page.
516+Examples][streamly-examples] web page.
504517
505518### Further Reading
506519
507−* [Streaming Benchmarks](https://github.com/composewell/streaming-benchmarks) | [Concurrency Benchmarks](https://github.com/composewell/concurrency-benchmarks)
508−* [Functional Conf 2019 Video](https://www.youtube.com/watch?v=uzsqgdMMgtk) | [Functional Conf 2019 Slides](https://www.slideshare.net/HarendraKumar10/streamly-concurrent-data-flow-programming)
520+* [Streaming Benchmarks][streaming-benchmarks]
521+* [Concurrency Benchmarks][concurrency-benchmarks]
522+* Functional Conf 2019 [Video](https://www.youtube.com/watch?v=uzsqgdMMgtk) | [Slides](https://www.slideshare.net/HarendraKumar10/streamly-concurrent-data-flow-programming)
509523* [Other Guides](docs/)
510−* [Streamly Homepage](https://streamly.composewell.com)
524+* [Streamly Homepage][Streamly]
511525
512526## Performance
513527
@@ -536,8 +550,8 @@ you use this plugin for applications that are performance sensitive.
536550
537551We measured several Haskell streaming implementations
538552using various micro-benchmarks. Please see the [streaming
539−benchmarks](https://github.com/composewell/streaming-benchmarks) page
540−for a detailed comparison of Streamly against other streaming libraries.
553+benchmarks][streaming-benchmarks] page for a detailed comparison of
554+Streamly against other streaming libraries.
541555
542556Our results show that [Streamly][] is the fastest effectful streaming
543557implementation on almost all the measured microbenchmarks. In many cases
@@ -558,8 +572,8 @@ programming with streams and arrays, for reading and writing from the
558572file system and from the network, for time domain programming (reactive
559573programming), and for reacting to file system events using `fsnotify`.
560574
561−Please view [Streamly's documentation](https://streamly.composewell.com)
562−for more information about Streamly's features.
575+Please view [Streamly's documentation][Streamly] for more information
576+about Streamly's features.
563577
564578### Concurrency
565579
@@ -577,9 +591,8 @@ very little overhead even where the task size is very
577591small, because Streamly will automatically switch between
578592serial or batched execution of tasks on the same CPU depending
579593on whichever is more efficient. Please see our [concurrency
580−benchmarks](https://github.com/composewell/concurrency-benchmarks) for
581−more detailed performance measurements, and for a comparison with the
582−`async` package.
594+benchmarks][concurrency-benchmarks] for more detailed performance
595+measurements, and for a comparison with the `async` package.
583596
584597### Design Goals
585598
@@ -618,14 +631,13 @@ list of contributors, credits and licenses.
618631## Licensing
619632
620633Streamly is an [open source](https://github.com/composewell/streamly)
621−project available under a liberal [BSD-3-Clause
622−license](https://github.com/composewell/streamly/blob/master/LICENSE).
634+project available under a liberal [BSD-3-Clause license][LICENSE]
623635
624636## Contributing to Streamly
625637
626638As an open project we welcome contributions:
627639
628−* [Streamly Contributor's Guide](CONTRIBUTING.md)
640+* [Streamly Contributor's Guide][CONTRIBUTING.md]
629641* [Contact the streamly development team](mailto:streamly@composewell.com)
630642
631643## Getting Support