11e1693 · streamly · git.iko.soy

git.iko.soy/streamly

on master help sign in

Split README into README and tutorial, and update it

11e1693Harendra Kumar4 years ago

README.md+1−1

compare with master

unifiedsplit

@@ -1 +1 @@

1−docs/User/Tutorials/Introduction.md

1+docs/User/Tutorials/README.md

docs/User/Tutorials/Introduction.md+23−161

@@ -1,17 +1,4 @@

1−# [Streamly][]: Idiomatic Haskell with the Performance of C

1+# [Streamly][] Quick Tutorial

2−

3−[![Gitter chat](https://badges.gitter.im/composewell/gitter.svg)](https://gitter.im/composewell/streamly)

4−[![Hackage](https://img.shields.io/hackage/v/streamly.svg?style=flat)](https://hackage.haskell.org/package/streamly)

5−

6−Streamly is a Haskell library that provides the building blocks to build

7−safe, scalable, modular and high performance software. Streamly offers:

8−

9−* The type safety of Haskell.

10−* The performance of C programs.

11−* Powerful abstractions for structuring your code.

12−* Idiomatic functional programming.

13−* Declarative concurrency for the seamless use of multiprocessing hardware.

14−

15## About This Document

2## About This Document

17This guide introduces programming with [Streamly][] using a few practical

4This guide introduces programming with [Streamly][] using a few practical

@@ -31,10 +18,7 @@ examples:

31* Finally, we will look at how to [rate limit](#rate-limiting) stream

18* Finally, we will look at how to [rate limit](#rate-limiting) stream

32 processing.

19 processing.

34−The guide then looks at how Streamly achieves its

21+It [concludes](#further-reading) with suggestions for further reading.

35−[performance](#performance). It [concludes](#notes) with a brief

36−discussion about Streamly's design philosophy, and with suggestions for

37−further reading.

39## Getting Started

23## Getting Started

@@ -55,7 +39,8 @@ As an expository device, we have indicated the types at the intermediate

55stages of stream computations as comments in the examples below.

39stages of stream computations as comments in the examples below.

56The meaning of these types are:

40The meaning of these types are:

58−* A `Stream IO a` is a serial stream of values of type `a` in the IO Monad.

42+* A `Stream IO a` is a representation of a sequence of values of type

43+ `a` in the IO Monad.

59* An `Unfold IO a b` is a representation of a function that converts a seed

44* An `Unfold IO a b` is a representation of a function that converts a seed

60 value of type `a` into a stream of values of type `b` in the IO Monad.

45 value of type `a` into a stream of values of type `b` in the IO Monad.

61* A `Fold IO a b` is a representation of a function that converts a stream of

46* A `Fold IO a b` is a representation of a function that converts a stream of

@@ -82,6 +67,10 @@ using the pre-release (internal) modules.

83## The Examples

68## The Examples

70+The code snippets below should work in GHCi if all of those are typed

71+in sequence. For brevity, imports that are already used in earlier

72+snippets are omitted from the latter ones.

73+

85### Modular Word Counting

74### Modular Word Counting

87A `Fold` in Streamly is a composable stream consumer. For our first

76A `Fold` in Streamly is a composable stream consumer. For our first

@@ -90,7 +79,7 @@ present in a file. We will then compose individual `Fold`s together to

90count words, bytes and lines at the same time.

79count words, bytes and lines at the same time.

92Please see the file [WordCountModular.hs][] for the complete example

81Please see the file [WordCountModular.hs][] for the complete example

93−program, including the imports that we have omitted here.

82+program.

95#### Count Bytes (wc -c)

84#### Count Bytes (wc -c)

@@ -121,7 +110,7 @@ import Streamly.Data.Fold (Fold)

121countl :: Int -> Word8 -> Int

110countl :: Int -> Word8 -> Int

122countl n ch = if ch == 10 then n + 1 else n

111countl n ch = if ch == 10 then n + 1 else n

123

112

124−-- The fold accepts a stream of `Word8` and returns a line count (`Int`).

113+-- The `nlines` fold accepts a stream of `Word8` and returns a line count (`Int`).

125nlines :: Monad m => Fold m Word8 Int

114nlines :: Monad m => Fold m Word8 Int

126nlines = Fold.foldl' countl 0

115nlines = Fold.foldl' countl 0

127

116

@@ -145,7 +134,7 @@ countw (n, wasSpace) ch =

145 then (n, True)

134 then (n, True)

146 else (if wasSpace then n + 1 else n, False)

135 else (if wasSpace then n + 1 else n, False)

147

136

148−-- The fold accepts a stream of `Word8` and returns a word count (`Int`).

137+-- The `nwords` fold accepts a stream of `Word8` and returns a word count (`Int`).

149nwords :: Monad m => Fold m Word8 Int

138nwords :: Monad m => Fold m Word8 Int

150nwords = fst <$> Fold.foldl' countw (0, True)

139nwords = fst <$> Fold.foldl' countw (0, True)

151

140

@@ -178,9 +167,7 @@ wc file =

178```

167```

179

168

180This example demonstrates the excellent modularity offered by

169This example demonstrates the excellent modularity offered by

181−[Streamly][]'s simple and concise API. Experienced Haskellers will

170+[Streamly][]'s simple and concise API.

182−notice that we have not used bytestrings—we instead used a stream of

183−`Word8` values, simplifying our program.

184

171

185### The Performance of Word Counting

172### The Performance of Word Counting

186

173

@@ -188,15 +175,15 @@ We compare two equivalent implementations: one using [Streamly][],

188and the other using C.

175and the other using C.

189

176

190The performance of the [Streamly word counting

177The performance of the [Streamly word counting

191−implementation][WordCount.hs] is:

178+implementation][WordCount.hs] (using ghc-9.4.4 and fusion-plugin) is:

192

179

193```

180```

194$ time WordCount-hs gutenberg-500MB.txt

181$ time WordCount-hs gutenberg-500MB.txt

19511242220 97050938 574714449 gutenberg-500MB.txt

18211242220 97050938 574714449 gutenberg-500MB.txt

196

183

197−real 0m1.825s

184+real 0m2.033s

198−user 0m1.697s

185+user 0m1.821s

199−sys 0m0.128s

186+sys 0m0.209s

200```

187```

201

188

202The performance of an equivalent [wc implementation in C][WordCount.c] is:

189The performance of an equivalent [wc implementation in C][WordCount.c] is:

@@ -205,9 +192,9 @@ The performance of an equivalent [wc implementation in C][WordCount.c] is:

205$ time WordCount-c gutenberg-500MB.txt

192$ time WordCount-c gutenberg-500MB.txt

20611242220 97050938 574714449 gutenberg-500MB.txt

19311242220 97050938 574714449 gutenberg-500MB.txt

207

194

208−real 0m2.100s

195+real 0m2.113s

209−user 0m1.935s

196+user 0m1.928s

210−sys 0m0.165s

197+sys 0m0.185s

211```

198```

212

199

213### Concurrent Word Counting

200### Concurrent Word Counting

@@ -318,9 +305,9 @@ A benchmark with 2 CPUs:

318$ time WordCount-hs-parallel gutenberg-500MB.txt

305$ time WordCount-hs-parallel gutenberg-500MB.txt

31911242220 97050938 574714449 gutenberg-500MB.txt

30611242220 97050938 574714449 gutenberg-500MB.txt

320

307

321−real 0m1.284s

308+real 0m1.443s

322−user 0m1.952s

309+user 0m2.095s

323−sys 0m0.140s

310+sys 0m0.202s

324```

311```

325

312

326These example programs have assumed ASCII encoded input data. For UTF-8

313These example programs have assumed ASCII encoded input data. For UTF-8

@@ -344,7 +331,7 @@ dictionary lookup server which can serve word meanings to multiple

344clients concurrently.

331clients concurrently.

345

332

346Please see the file [WordServer.hs][] for the complete code for this

333Please see the file [WordServer.hs][] for the complete code for this

347−example, including the imports that we have omitted below.

334+example.

348

335

349```haskell ghci

336```haskell ghci

350import Control.Concurrent (threadDelay)

337import Control.Concurrent (threadDelay)

@@ -517,131 +504,6 @@ Examples][streamly-examples] web page.

517* [Other Guides](/)

504* [Other Guides](/)

518* [Streamly Homepage][Streamly]

505* [Streamly Homepage][Streamly]

519

506

520−## Performance

521−

522−As you have seen in the word count example above, [Streamly][] offers

523−highly modular abstractions for building programs while also offering

524−the performance close to an equivalent (imperative) C program.

525−

526−Streamly offers excellent performance even for byte-at-a-time stream

527−operations using efficient abstractions like `Unfold`s and terminating

528−`Fold`s. Byte-at-a-time stream operations can simplify programming

529−because the developer does not have to deal explicitly with chunking

530−and re-combining data.

531−

532−Streamly exploits GHC's stream fusion optimizations (`case-of-case` and

533−`spec-constr`) aggressively to achieve C-like speed, while also offering

534−highly modular abstractions to developers.

535−

536−[Streamly][] will usually perform very well without any

537−compiler plugins. However, we have fixed some deficiencies

538−that we had noticed in GHC's optimizer using a [compiler

539−plugin](https://github.com/composewell/fusion-plugin). We hope to fold

540−these optimizations into GHC in the future; until then we recommend that

541−you use this plugin for applications that are performance sensitive.

542−

543−### Benchmarks

544−

545−We measured several Haskell streaming implementations

546−using various micro-benchmarks. Please see the [streaming

547−benchmarks][streaming-benchmarks] page for a detailed comparison of

548−Streamly against other streaming libraries.

549−

550−Our results show that [Streamly][] is the fastest effectful streaming

551−implementation on almost all the measured microbenchmarks. In many cases

552−it runs up to 100x faster, and in some cases even 1000x faster than

553−some of the tested alternatives. In some composite operation benchmarks

554−[Streamly][] turns out to be significantly faster than Haskell's list

555−implementation.

556−

557−*Note*: If you can write a program in some other way or with some other

558−language that runs significantly faster than what [Streamly][] offers,

559−please let us know and we will improve.

560−

561−## Notes

562−

563−Streamly comes equipped with a very powerful set of abstractions to

564−accomplish many kinds of programming tasks: it provides support for

565−programming with streams and arrays, for reading and writing from the

566−file system and from the network, for time domain programming (reactive

567−programming), and for reacting to file system events using `fsnotify`.

568−

569−Please view [Streamly's documentation][Streamly] for more information

570−about Streamly's features.

571−

572−### Concurrency

573−

574−Streamly uses lock-free synchronization for achieving concurrent

575−operation with low overheads. The number of tasks performed concurrently

576−are determined automatically based on the rate at which a consumer

577−is consuming the results. In other words, you do not need to manage

578−thread pools or decide how many threads to use for a particular task.

579−For CPU-bound tasks Streamly will try to keep the number of threads

580−close to the number of CPUs available; for IO-bound tasks it will utilize

581−more threads.

582−

583−The parallelism available during program execution can be utilized with

584−very little overhead even where the task size is very

585−small, because Streamly will automatically switch between

586−serial or batched execution of tasks on the same CPU depending

587−on whichever is more efficient. Please see our [concurrency

588−benchmarks][concurrency-benchmarks] for more detailed performance

589−measurements, and for a comparison with the `async` package.

590−

591−### Design Goals

592−

593−Our goals for [Streamly][] from the very beginning have been:

594−

595−1. To achieve simplicity by unifying abstractions.

596−2. To offer high performance.

597−

598−These goals are hard to achieve simultaneously because they are usually

599−inversely related. We have spent many years trying to get the abstractions

600−right without compromising performance.

601−

602−`Unfold` is an example of an abstraction that we have created to achieve

603−high performance when mapping streams on streams. `Unfold` allows stream

604−generation to be optimized well by the compiler through stream fusion.

605−A `Fold` with termination capability is another example which modularizes

606−stream elimination operations through stream fusion. Terminating folds

607−can perform many simple parsing tasks that do not require backtracking.

608−In Streamly, `Parser`s are a natural extension to terminating `Fold`s;

609−`Parser`s add the ability to backtrack to `Fold`s. Unification leads

610−to simpler abstractions and lower cognitive overheads while also not

611−compromising performance.

612−

613−## Credits

614−

615−The following authors/libraries have influenced or inspired this library in a

616−significant way:

617−

618− * Roman Leshchinskiy ([vector](http://hackage.haskell.org/package/vector))

619− * Gabriella Gonzalez ([foldl](https://hackage.haskell.org/package/foldl))

620− * Alberto G. Corona ([transient](https://hackage.haskell.org/package/transient))

621−

622−Please see the [`credits`](/docs/User/ProjectRelated/Credits.md) directory for a full

623−list of contributors, credits and licenses.

624−

625−## Licensing

626−

627−Streamly is an [open source](https://github.com/composewell/streamly)

628−project available under a liberal [BSD-3-Clause license][LICENSE]

629−

630−## Contributing to Streamly

631−

632−As an open project we welcome contributions:

633−

634−* [Streamly Contributor's Guide][CONTRIBUTING.md]

635−* [Contact the streamly development team](mailto:streamly@composewell.com)

636−

637−## Getting Support

638−

639−Professional support is available for [Streamly][]: please contact

640−[support@composewell.com](mailto:support@composewell.com).

641−

642−You can also join our [community chat

643−channel](https://gitter.im/composewell/streamly) on Gitter.

644−

645<!--

507<!--

646Link References.

508Link References.

647-->

509-->

docs/User/Tutorials/README.mdnew file+165

@@ -0,0 +1,165 @@

1+# [Streamly][]: Idiomatic Haskell with the Performance of C

3+[![Gitter chat](https://badges.gitter.im/composewell/gitter.svg)](https://gitter.im/composewell/streamly)

4+[![Hackage](https://img.shields.io/hackage/v/streamly.svg?style=flat)](https://hackage.haskell.org/package/streamly)

6+Streamly is a Haskell library that provides building blocks to build

7+safe, scalable, modular and high performance software. Streamly offers:

9+* The type safety of Haskell.

10+* The performance of C programs.

11+* Powerful building blocks for modular code.

12+* Idiomatic functional programming.

13+* Declarative, fearless concurrency.

14+* Ecossytem libraries for quick development.

15+

16+Please read [Using Streamly](/docs/User/Tutorials/getting-started.md) and

17+[Streamly Quick Tutorial](/docs/User/Tutorials/Introduction.md) to get a taste

18+of the library. Streamly comes with comprehensive documentation, please

19+visit the [Haskell Streamly website][Streamly] for documentation.

20+

21+## Performance with Modularity

22+

23+Usually, you have to pick one of the two, performance or

24+modularity. Using [Haskell Streamly][Streamly] you can write highly

25+modular code and still achieve performance close to an equivalent

26+(imperative) C program. Streamly exploits GHC's stream fusion

27+optimizations (`case-of-case` and `spec-constr`) aggressively to achieve

28+both modularity and performance at the same time.

29+

30+Streamly offers excellent performance even for byte-at-a-time stream

31+operations using efficient abstractions like `Unfold`s and

32+`Fold`s. Byte-at-a-time stream operations can simplify programming

33+because the developer does not have to deal explicitly with chunking

34+and re-combining data.

35+

36+### GHC Plugin for Stream Fusion

37+

38+[Streamly][] usually performs very well without any compiler plugins.

39+However, we have fixed some deficiencies in GHC's optimizer using a

40+[compiler plugin](https://github.com/composewell/fusion-plugin). We

41+hope to fold these optimizations into GHC in the future; until then we

42+recommend that you use this plugin for applications that are performance

43+sensitive.

44+

45+### Performance Benchmarks

46+

47+We measured several Haskell streaming implementations

48+using various micro-benchmarks. Please see the [streaming

49+benchmarks][streaming-benchmarks] page for a detailed comparison of

50+Streamly against other streaming libraries.

51+

52+Our results show that [Streamly][] is the fastest effectful streaming

53+implementation on almost all the measured microbenchmarks. In many cases

54+it runs up to 100x faster, and in some cases even 1000x faster than

55+some of the tested alternatives. In some composite operation benchmarks

56+[Streamly][] turns out to be significantly faster than Haskell's list

57+implementation.

58+

59+*Note*: If you can write a program in some other way or with some other

60+language that runs significantly faster than what [Streamly][] offers,

61+please let us know and we will improve.

62+

63+## Applications

64+

65+Streamly comes equipped with a very powerful set of abstractions to

66+accomplish many kinds of programming tasks: it provides support for

67+programming with streams and arrays, for reading and writing from the

68+file system and from the network, for time domain programming (reactive

69+programming), and for reacting to file system events using `fsnotify`.

70+

71+Please view [Streamly's documentation][Streamly] for more information

72+about Streamly's features.

73+

74+## Design

75+

76+### Design Goals

77+

78+Our goals for [Streamly][] from the very beginning have been:

79+

80+1. To achieve simplicity by unifying abstractions.

81+2. To offer high performance.

82+

83+These goals are hard to achieve simultaneously because they are usually

84+inversely related. We have spent many years trying to get the abstractions

85+right without compromising performance.

86+

87+`Unfold` is an example of an abstraction that we have created to achieve

88+high performance when mapping streams on streams. `Unfold` allows stream

89+generation to be optimized well by the compiler through stream fusion.

90+A `Fold` with termination capability is another example which modularizes

91+stream elimination operations through stream fusion. Terminating folds

92+can perform many simple parsing tasks that do not require backtracking.

93+In Streamly, `Parser`s are a natural extension to terminating `Fold`s;

94+`Parser`s add the ability to backtrack to `Fold`s. Unification leads

95+to simpler abstractions and lower cognitive overheads while also not

96+compromising performance.

97+

98+### Concurrency Design

99+

100+Streamly uses lock-free synchronization for achieving concurrent

101+operation with low overheads. The number of tasks performed concurrently

102+are determined automatically based on the rate at which a consumer

103+is consuming the results. In other words, you do not need to manage

104+thread pools or decide how many threads to use for a particular task.

105+For CPU-bound tasks Streamly will try to keep the number of threads

106+close to the number of CPUs available; for IO-bound tasks it will utilize

107+more threads.

108+

109+The parallelism available during program execution can be utilized with

110+very little overhead even where the task size is very small, because

111+Streamly will automatically switch between serial or batched execution

112+of tasks on the same CPU depending on whichever is more efficient.

113+Please see our [concurrency benchmarks][concurrency-benchmarks] for more

114+detailed performance measurements, and for a comparison with the `async`

115+package.

116+

117+## Credits

118+

119+The following authors/libraries have influenced or inspired this library in a

120+significant way:

121+

122+ * Roman Leshchinskiy ([vector](http://hackage.haskell.org/package/vector))

123+ * Gabriella Gonzalez ([foldl](https://hackage.haskell.org/package/foldl))

124+ * Alberto G. Corona ([transient](https://hackage.haskell.org/package/transient))

125+

126+Please see the [`credits`](/docs/User/ProjectRelated/Credits.md) directory for a full

127+list of contributors, credits and licenses.

128+

129+## Licensing

130+

131+Streamly is an [open source](https://github.com/composewell/streamly)

132+project available under a liberal [BSD-3-Clause license][LICENSE]

133+

134+## Contributing to Streamly

135+

136+As an open project we welcome contributions:

137+

138+* [Streamly Contributor's Guide][CONTRIBUTING.md]

139+* [Contact the streamly development team](mailto:streamly@composewell.com)

140+

141+## Getting Support

142+

143+Professional support is available for [Streamly][]: please contact

144+[support@composewell.com](mailto:support@composewell.com).

145+

146+You can also join our [community chat

147+channel](https://gitter.im/composewell/streamly) on Gitter.

148+

149+<!--

150+Link References.

151+-->

152+

153+[Streamly]: https://streamly.composewell.com/

154+[streaming-benchmarks]: https://github.com/composewell/streaming-benchmarks

155+[concurrency-benchmarks]: https://github.com/composewell/concurrency-benchmarks

156+

157+<!--

158+Keep all the unstable links here so that they can be updated to stable

159+links (for online docs) before we release.

160+-->

161+

162+

163+[LICENSE]: /LICENSE

164+[CONTRIBUTING.md]: /CONTRIBUTING.md

165+[docs]: docs/