cf63b4e · streamly · git.iko.soy

git.iko.soy/streamly

on master help sign in

Fix README and Overview.md

cf63b4eHarendra Kumar5 years ago

README.md+49−58

compare with master

unifiedsplit

@@ -18,8 +18,8 @@

18## Idiomatic Haskell at the speed of C

20Streamly is a Haskell library/framework providing basic building blocks

21−or combinators to perform fundamental as well as advanced programming

21+or combinators to build safe, scalable, modular and high performance

22−tasks with ease. The key features it provides are:

22+software systems. The key features it provides are:

24* Speed of C

25* Safety of Haskell

@@ -43,10 +43,10 @@ repository](https://github.com/composewell/streamly-examples).

44## Modular Word Counting

46−The `Fold` data type in streamly represents a consumer of stream. In

47−this example, we will use individual folds to count bytes, words and

46+In this example, we will use folds to count bytes, words and lines in

47+a file. A `Fold` is a composable stream consumer. We will see how

48−lines in a file. We will see how these individual folds can be composed

48+individual folds can be composed together to perform all the three

49−together to do all the three at once with the same performance.

49+counts at once with the same performance.

51See [WordCountModular.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountModular.hs)

52for full working code including imports that we may have omitted

@@ -129,13 +129,10 @@ wc file =

129 & Stream.fold countAll -- IO (Int, Int, Int)

130```

131

132−### Observations

132+This example demonstrates the simple and concise API of streamly with

133−

133+excellent modularity. Experienced Haskellers would notice that we have

134−This example shows:

134+not used bytestrings, we simply use a stream of `Word8`, simplifying the

135−

135+program.

136−* Excellent modularity

137−* Simple and concise API and types

138−* No bytestrings required, just streams of Word8

139

136

140## Performance

137## Performance

141

138

@@ -165,7 +162,7 @@ sys 0m0.165s

165

162

166## Concurrent Word Counting

163## Concurrent Word Counting

167

164

168−To do word counting in parallel we divide the stream in chunks (arrays),

165+To count words in parallel we divide the stream into chunks (arrays),

169count properties in each chunk and then add all the counts. We use the

166count properties in each chunk and then add all the counts. We use the

170same code as above except that we use an array input instead of using a

167same code as above except that we use an array input instead of using a

171file input.

168file input.

@@ -185,10 +182,12 @@ countArray arr =

185 & Stream.decodeLatin1 -- SerialT IO Char

182 & Stream.decodeLatin1 -- SerialT IO Char

186 & Stream.foldl' count (Counts 0 0 0 True) -- IO Counts

183 & Stream.foldl' count (Counts 0 0 0 True) -- IO Counts

187```

184```

188−When combining the counts in two contiguous chunks, we would also need to

185+When combining the counts in two contiguous chunks, we would also need

189−know whether the first element of the next chunk was a space char or

186+to know whether the first element of the next chunk was a space char or

190−non-space to know whether the same word is continuing to the next chunk or

187+non-space to know whether the same word is continuing to the next chunk

191−if it is a new word. So add that too, giving (firstCharWasSpace, Counts).

188+or if it is a new word. `partialCounts` adds a `Bool` flag to `Counts`

189+returned by `countArray` to indicate whether the first character in the

190+chunk is a space.

192

191

193``` haskell

192``` haskell

194partialCounts :: Array Word8 -> IO (Bool, Counts)

193partialCounts :: Array Word8 -> IO (Bool, Counts)

@@ -224,7 +223,7 @@ wc file = do

224 & Stream.foldl' addCounts (False, Counts 0 0 0 True) -- IO (Bool, Counts)

223 & Stream.foldl' addCounts (False, Counts 0 0 0 True) -- IO (Bool, Counts)

225```

224```

226

225

227−Note that `Stream.aheadly` is the only difference in a concurrent and

226+Note that `Stream.fromAhead` is the only difference in a concurrent and

228non-concurrent program. If we remove that we still have a perfectly valid,

227non-concurrent program. If we remove that we still have a perfectly valid,

229well performing serial program. Notice, how succinctly and idiomatically

228well performing serial program. Notice, how succinctly and idiomatically

230we expressed the concurrent word counting problem.

229we expressed the concurrent word counting problem.

@@ -383,10 +382,6 @@ main = do

383 & Stream.mapM_ print

382 & Stream.mapM_ print

384```

383```

385

384

386−Notice, how succinctly and idiomatically we expressed the concurrent

387−directory tree traversal problem. Let us know if you can do it

388−significantly better with any other language or framework.

389−

390## Rate Limiting

385## Rate Limiting

391

386

392For bounded concurrent streams, stream yield rate can be specified. For

387For bounded concurrent streams, stream yield rate can be specified. For

@@ -438,13 +433,21 @@ documentation](https://streamly.composewell.com) to know more.

438

433

439## Concurrency

434## Concurrency

440

435

441−Streamly uses lock-free synchronization for concurrent operations. It employs

436+Streamly uses lock-free synchronization for low overhead

442−auto-scaling of the degree of concurrency based on demand. For CPU bound tasks

437+concurrency. The number of tasks performed concurrently are

438+determined automatically based on the rate at which a consumer is

439+consuming the results. In other words, you do not need to manage

440+thread pools and decide how many threads to use for a particular

441+task. For CPU bound tasks it tries to keep the number of threads

443−it tries to keep the threads close to the number of CPUs available whereas for

442+close to the number of CPUs available whereas for IO bound tasks

444−IO bound tasks more threads can be utilized. Parallelism can be utilized with

443+more threads can be utilized.

444+

445+Parallelism can be utilized with little overhead even if

445−little overhead even if the task size is very small. See [concurrency

446+the task size is very small, because it can automatically

447+switch to serial mode or batch multiple tasks on the

448+same CPU if that is more efficient. See [concurrency

446−benchmarks](https://github.com/composewell/concurrency-benchmarks) for detailed

449+benchmarks](https://github.com/composewell/concurrency-benchmarks) for

447−performance results and a comparison with the `async` package.

450+detailed performance results and a comparison with the `async` package.

448

451

449## Performance

452## Performance

450

453

@@ -452,41 +455,29 @@ As you have seen above in the word count example, streamly enables

452highly modular abstractions with the best possible performance (close to

455highly modular abstractions with the best possible performance (close to

453an equivalent C program).

456an equivalent C program).

454

457

455−For example, none of the existing Haskell libraries can work with byte

456−level streams with acceptable performance. Streamly provides excellent

457−performance even for byte level stream operations, it is made possible

458+Streamly provides excellent performance even for byte level stream

458−by employing efficient abstractions like `Unfold`s and terminating

459+operations, it is made possible by employing efficient abstractions like

459−`Fold`s. Byte level stream operations make programming simpler because

460+`Unfold`s and terminating `Fold`s. Byte level stream operations make

460−you do not have to deal with chunking and re-combining.

461+programming simpler because you do not have to deal with chunking and

462+re-combining.

461

463

462If you can write a program significantly faster in some other way or

464If you can write a program significantly faster in some other way or

463−with some other language, please let us know and we will improve. We

465+with some other language, please let us know and we will improve.

464−still have some areas to improve but we will surely get there for most

465−cases.

466

467## Benchmarks

468

469−When it comes to stream operations, Haskell lists is the fastest

469+We measured several Haskell streaming implementations

470−existing implementation even though it supports only pure (not monadic)

470+on various micro-benchmarks. Please see [streaming

471−streams. Other streaming libraries are many times slower compared to

471+benchmarks](https://github.com/composewell/streaming-benchmarks) page

472−lists, so lists is our gold standard for performance comparison.

472+for detailed comparison of streamly with other streaming libraries.

473−

474−In the following chart, positive y-axis displays how many times worse

475−is a list operation compared to the corresponding streamly operation,

476−negative y-axis shows the same where streamly is worse compared to

477−lists.

478−

479−![Streamly vs Lists (time) comparison](charts-0/streamly-vs-list-time.svg)

480−

481−This chart show only those operations where streamly and list

482−performance differs by more than 10%. For all other operations the

483−performance of both is more or less comparable.

484

473

485−See [streaming

474+These results show that streamly is the fastest effectful streaming

486−benchmarks](https://github.com/composewell/streaming-benchmarks) for

475+implementation on almost all the measured micro benchmarks. In many cases

487−detailed comparison with other Haskell streaming libraries.

476+it is up to 100x faster and in some cases even 1000x faster. In many

477+composite operation benchmarks streamly turns out to be significantly

478+faster than the Haskell lists implementation.

488

479

489−## Performance and Modularity

480+## Design Goals

490

481

491The goals of streamly from the very beginning have been, (1) simplicity

482The goals of streamly from the very beginning have been, (1) simplicity

492by unifying abstractions, (2) high performance. These are hard to

483by unifying abstractions, (2) high performance. These are hard to

docs/Overview.md+4−141

@@ -1,19 +1,8 @@

1−# Streamly

1+## Overview

3−[![Hackage](https://img.shields.io/hackage/v/streamly.svg?style=flat)](https://hackage.haskell.org/package/streamly)

3+These are some notes from an old version of README that may be

4−[![Gitter chat](https://badges.gitter.im/composewell/gitter.svg)](https://gitter.im/composewell/streamly)

4+useful. For a quick introduction please read the README.md at the repo

5−[![Travis](https://travis-ci.com/composewell/streamly.svg?branch=master)](https://travis-ci.com/composewell/streamly)

5+root first.

6−[![Appveyor](https://ci.appveyor.com/api/projects/status/ajxg0c79raou9ned?svg=true)](https://ci.appveyor.com/project/harendra-kumar/streamly)

7−[![CircleCI](https://circleci.com/gh/composewell/streamly/tree/master.svg?style=svg)](https://circleci.com/gh/composewell/streamly/tree/master)

8−[![Coverage Status](https://coveralls.io/repos/composewell/streamly/badge.svg?branch=master&service=github)](https://coveralls.io/github/composewell/streamly?branch=master)

9−

10−## Learning Materials

11−

12−* Documentation: [Quick](#streaming-concurrently) | [Tutorial](https://hackage.haskell.org/package/streamly/docs/Streamly-Tutorial.html) | [Reference (Hackage)](https://hackage.haskell.org/package/streamly) | [Reference (Latest)](https://composewell.github.io/streamly) | [Guides](docs)

13−* Installing: [Installing](./INSTALL.md) | [Building for optimal performance](docs/Build.md)

14−* Examples: [streamly](examples) | [streamly-examples](https://github.com/composewell/streamly-examples)

15−* Benchmarks: [Streaming](https://github.com/composewell/streaming-benchmarks) | [Concurrency](https://github.com/composewell/concurrency-benchmarks)

16−* Talks: [Functional Conf 2019 Video](https://www.youtube.com/watch?v=uzsqgdMMgtk) | [Functional Conf 2019 Slides](https://www.slideshare.net/HarendraKumar10/streamly-concurrent-data-flow-programming)

18## Streaming Concurrently

7## Streaming Concurrently

@@ -71,45 +60,6 @@ computations, and the IO monad providing an escape hatch to an imperative

71model, we just love to fall into the imperative trap, and start asking the same

60model, we just love to fall into the imperative trap, and start asking the same

72fundamental question again - why do we have to use the streaming data model?

61fundamental question again - why do we have to use the streaming data model?

74−## Comparative Performance

75−

76−High performance and simplicity are the two primary goals of streamly.

77−`Streamly` employs two different stream representations (CPS and direct style)

78−and interconverts between the two to get the best of both worlds on different

79−operations. It uses both foldr/build (for CPS style) and stream fusion (for

80−direct style) techniques to fuse operations. In terms of performance,

81−Streamly's goal is to compete with equivalent C programs. Streamly redefines

82−"blazing fast" for streaming libraries, it competes with lists and `vector`.

83−Other streaming libraries like "streaming", "pipes" and "conduit" are orders of

84−magnitude slower on most microbenchmarks. See [streaming

85−benchmarks](https://github.com/composewell/streaming-benchmarks) for detailed

86−comparison.

87−

88−The following chart shows a comparison of those streamly and list operations

89−where performance of the two differs by more than 10%. Positive y-axis displays

90−how many times worse is a list operation compared to the same streamly

91−operation, negative y-axis shows where streamly is worse compared to lists.

92−

93−![Streamly vs Lists (time) comparison](charts-0/streamly-vs-list-time.svg)

94−

95−Streamly uses lock-free synchronization for concurrent operations. It employs

96−auto-scaling of the degree of concurrency based on demand. For CPU bound tasks

97−it tries to keep the threads close to the number of CPUs available whereas for

98−IO bound tasks more threads can be utilized. Parallelism can be utilized with

99−little overhead even if the task size is very small. See [concurrency

100−benchmarks](https://github.com/composewell/concurrency-benchmarks) for detailed

101−performance results and a comparison with the `async` package.

102−

103−## Installing and using

104−

105−Please see [INSTALL.md](./INSTALL.md) for instructions on how to use streamly

106−with your Haskell build tool or package manager. You may want to go through it

107−before jumping to run the examples below.

108−

109−The module `Streamly.Prelude` provides the core stream types and combinators

110−for type casting, controlling concurrency, stream construction, transformation,

111−folding, merging, and zipping.

112−

113## Streaming Pipelines

63## Streaming Pipelines

114

115The following snippet provides a simple stream composition example that reads

65The following snippet provides a simple stream composition example that reads

@@ -170,14 +120,6 @@ The following finishes in 10 seconds (100 seconds when serial):

170S.drain $ S.fromAsync $ S.replicateM 10 $ p 10

120S.drain $ S.fromAsync $ S.replicateM 10 $ p 10

171```

121```

172

122

173−## Concurrency Auto Scaling

174−

175−Concurrency is auto-scaled i.e. more actions are executed concurrently if the

176−consumer is consuming the stream at a higher speed. How many tasks are executed

177−concurrently can be controlled by `maxThreads` and how many results are

178−buffered ahead of consumption can be controlled by `maxBuffer`. See the

179−documentation in the `Streamly.Prelude` module.

180−

181## Concurrent Streaming Pipelines

123## Concurrent Streaming Pipelines

182

124

183Use `|&` or `|$` to apply stream processing functions concurrently. The

125Use `|&` or `|$` to apply stream processing functions concurrently. The

@@ -306,39 +248,6 @@ main = do

306 print s

248 print s

307```

249```

308

250

309−The concurrency facilities provided by streamly can be compared with

310−[OpenMP](https://en.wikipedia.org/wiki/OpenMP) and

311−[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative

312−expression.

313−

314−## Example: Listing Directories Recursively/Concurrently

315−

316−The following code snippet lists a directory tree recursively, reading multiple

317−directories concurrently:

318−

319−```haskell

320−import Control.Monad.IO.Class (liftIO)

321−import Path.IO (listDir, getCurrentDir) -- from path-io package

322−import Streamly.Prelude (AsyncT, adapt)

323−import qualified Streamly.Prelude as S

324−

325−listDirRecursive :: AsyncT IO ()

326−listDirRecursive = getCurrentDir >>= readdir >>= liftIO . mapM_ putStrLn

327− where

328− readdir dir = do

329− (dirs, files) <- listDir dir

330− S.yield (map show dirs ++ map show files) <> foldMap readdir dirs

331−

332−main :: IO ()

333−main = S.drain $ adapt $ listDirRecursive

334−```

335−

336−`AsyncT` is a stream monad transformer. If you are familiar with a list

337−transformer, it is nothing but `ListT` with concurrency semantics. For example,

338−the semigroup operation `<>` is concurrent. This makes `foldMap` concurrent

339−too. You can replace `AsyncT` with `SerialT` and the above code will become

340−serial, exactly equivalent to a `ListT`.

341−

342## Rate Limiting

251## Rate Limiting

343

252

344For bounded concurrent streams, stream yield rate can be specified. For

253For bounded concurrent streams, stream yield rate can be specified. For

@@ -489,16 +398,6 @@ llhisto =

489main = withArg llhisto >>= print

398main = withArg llhisto >>= print

490```

399```

491

400

492−## Socket IO

493−

494−Its easy to build concurrent client and server programs using streamly.

495−`Streamly.Network.*` modules provide easy combinators to build network servers

496−and client programs using streamly. See

497−[FromFileClient.hs](https://github.com/composewell/streamly/tree/master/examples/FromFileClient.hs),

498−[EchoServer.hs](https://github.com/composewell/streamly/tree/master/examples/EchoServer.hs),

499−[FileSinkServer.hs](https://github.com/composewell/streamly/tree/master/examples/FileSinkServer.hs)

500−in the examples directory.

501−

502## Exceptions

401## Exceptions

503

402

504Exceptions can be thrown at any point using the `MonadThrow` instance. Standard

403Exceptions can be thrown at any point using the `MonadThrow` instance. Standard

@@ -516,15 +415,6 @@ asynchronous exceptions to deal with. You can just ignore the zillions of

516blogs, talks, caveats about async exceptions. Async exceptions just don't

415blogs, talks, caveats about async exceptions. Async exceptions just don't

517exist. Please don't use things like `myThreadId` and `throwTo` just for fun!

416exist. Please don't use things like `myThreadId` and `throwTo` just for fun!

518

417

519−## Reactive Programming (FRP)

520−

521−Streamly is a foundation for first class reactive programming as well by virtue

522−of integrating concurrency and streaming. See

523−[AcidRain.hs](https://github.com/composewell/streamly/tree/master/examples/AcidRain.hs)

524−for a console based FRP game example and

525−[CirclingSquare.hs](https://github.com/composewell/streamly/tree/master/examples/CirclingSquare.hs)

526−for an SDL based animation example.

527−

528## Conclusion

418## Conclusion

529

419

530Streamly, short for streaming concurrently, provides monadic streams, with a

420Streamly, short for streaming concurrently, provides monadic streams, with a

@@ -577,30 +467,3 @@ streaming with concurrency we can write FRP applications similar in concept to

577

467

578See the `Comparison with existing packages` section at the end of the

468See the `Comparison with existing packages` section at the end of the

579[tutorial](https://hackage.haskell.org/package/streamly/docs/Streamly-Tutorial.html).

469[tutorial](https://hackage.haskell.org/package/streamly/docs/Streamly-Tutorial.html).

580−

581−## Support

582−

583−Please feel free to ask questions on the

584−[streamly gitter channel](https://gitter.im/composewell/streamly).

585−If you require professional support, consulting, training or timely

586−enhancements to the library please contact

587−[support@composewell.com](mailto:support@composewell.com).

588−

589−## Credits

590−

591−The following authors/libraries have influenced or inspired this library in a

592−significant way:

593−

594− * Roman Leshchinskiy ([vector](http://hackage.haskell.org/package/vector))

595− * Gabriel Gonzalez ([foldl](https://hackage.haskell.org/package/foldl))

596− * Alberto G. Corona ([transient](https://hackage.haskell.org/package/transient))

597−

598−See the `credits` directory for full list of contributors, credits and licenses.

599−

600−## Contributing

601−

602−The code is available under BSD-3 license

603−[on github](https://github.com/composewell/streamly). Join the [gitter

604−chat](https://gitter.im/composewell/streamly) channel for discussions. Please

605−ask any questions on the gitter channel or [contact the maintainer

606−directly](mailto:streamly@composewell.com). All contributions are welcome!