53aae83 · streamly · git.iko.soy

git.iko.soy/streamly

on master help sign in

Fix links, use link refs kept at one place

53aae83Harendra Kumar5 years ago

INSTALL.md+11−10

compare with master

unifiedsplit

@@ -6,7 +6,7 @@ SPDX-License-Identifer: BSD-3-Clause

6# Getting started with the `streamly` package

8This guide will show you how to write a simple program that uses

9−`streamly`, using the [`cabal`](https://www.haskell.org/cabal/) build

9+[Streamly][], using the [`cabal`](https://www.haskell.org/cabal/) build

10manager for Haskell.

12<!-- TODO: Add instructions for `stack` and `nix`.

@@ -21,13 +21,11 @@ to run a command-line shell under `msys` and some of the commands

21below may also need to be changed in small ways to make them work.

23**Note**: If you are new to Haskell you may find the [Haskell Getting

24−Started

25−Guide](https://github.com/composewell/haskell-dev/blob/master/getting-started.rst)

24+Started Guide][haskell-getting-started] to be useful.

26−to be useful.

28If you know your way around Haskell, and have an up to date toolchain

26If you know your way around Haskell, and have an up to date toolchain

29already installed, then you can jump straight to the section titled

27already installed, then you can jump straight to the section titled

30−"[_Prepare Your Build Directory_](#user-content-prepare-your-build-directory)".

28+"[_Prepare Your Build Directory_](#prepare-your-build-directory)".

32## Getting started with `streamly` using `cabal`

30## Getting started with `streamly` using `cabal`

@@ -284,15 +282,18 @@ streamly][streamly-github] on Github.

284If you got this far successfully, congratulations!

282If you got this far successfully, congratulations!

285

283

286* For an overview of the `streamly` package, please read the

284* For an overview of the `streamly` package, please read the

287− [Streamly Tutorial][tutorial].

285+ [Streamly Quick Overview](README.md) and [Streamly Tutorial][tutorial].

288* Please browse the code examples in the

286* Please browse the code examples in the

289− [`examples`](./examples/README.md) directory.

287+ [`Streamly examples repository`][streamly-examples].

290* If you would like to start on a real world project, please take a

288* If you would like to start on a real world project, please take a

291− look at the [recommended compilation options](./docs/Build.md).

289+ look at the [Streamly Build and Optimization Guide](./docs/Build.md).

292−

290+* For more information please visit the [Streamly Homepage][Streamly].

293

291

294

292

295

293

294+ [Streamly]: https://streamly.composewell.com/

296− [tutorial]: https://hackage.haskell.org/package/streamly/docs/Streamly-Tutorial.html

295+ [tutorial]: https://hackage.haskell.org/package/streamly-0.7.3/docs/Streamly-Tutorial.html

297 [streamly-hackage]: https://hackage.haskell.org/package/streamly

296 [streamly-hackage]: https://hackage.haskell.org/package/streamly

298 [streamly-github]: https://github.com/composewell/streamly

297 [streamly-github]: https://github.com/composewell/streamly

298+ [streamly-examples]: https://github.com/composewell/streamly-examples

299+ [haskell-getting-started]: https://github.com/composewell/haskell-dev/blob/master/getting-started.rst

README.md+64−52

@@ -3,7 +3,37 @@

3[![Gitter chat](https://badges.gitter.im/composewell/gitter.svg)](https://gitter.im/composewell/streamly)

4[![Hackage](https://img.shields.io/hackage/v/streamly.svg?style=flat)](https://hackage.haskell.org/package/streamly)

6+<!--

7+Link References.

8+-->

6− [Streamly]: https://streamly.composewell.com/

10+[Streamly]: https://streamly.composewell.com/

11+[streamly-examples]: https://github.com/composewell/streamly-examples

12+[streaming-benchmarks]: https://github.com/composewell/streaming-benchmarks

13+[concurrency-benchmarks]: https://github.com/composewell/concurrency-benchmarks

14+

15+<!--

16+Keep all the unstable links here so that they can be updated to stable

17+links (for online docs) before we release.

18+-->

19+

20+

21+[WordCountModular.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/WordCountModular.hs

22+[WordCount.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.hs

23+[WordCount.c]: https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.c

24+[WordCountParallel.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/WordCountParallel.hs

25+[WordCountUTF8.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/WordCountUTF8.hs

26+[WordServer.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/WordServer.hs

27+[MergeServer.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/MergeServer.hs

28+[ListDir.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/ListDir.hs

29+[Rate.hs]: https://github.com/composewell/streamly-examples/blob/master/examples/Rate.hs

30+[AcidRain.hs]: https://github.com/composewell/streamly-examples/tree/master/examples/AcidRain.hs

31+[CirclingSquare.hs]: https://github.com/composewell/streamly-examples/tree/master/examples/CirclingSquare.hs

32+

33+

34+[LICENSE]: https://github.com/composewell/streamly/blob/master/LICENSE

35+[CONTRIBUTING.md]: https://github.com/composewell/streamly/blob/master/CONTRIBUTING.md

36+[docs]: https://github.com/composewell/streamly/blob/master/docs/

8Streamly is a Haskell library that provides the building blocks to build

38Streamly is a Haskell library that provides the building blocks to build

9safe, scalable, modular and high performance software. Streamly offers:

39safe, scalable, modular and high performance software. Streamly offers:

@@ -91,10 +121,8 @@ example, we will use `Fold`s to count the number of bytes, words and lines

91present in a file. We will then compose individual `Fold`s together to

121present in a file. We will then compose individual `Fold`s together to

92count words, bytes and lines at the same time.

122count words, bytes and lines at the same time.

123

94−Please see the file

124+Please see the file [WordCountModular.hs][] for the complete example

95−[WordCountModular.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountModular.hs)

96−for the complete example program, including the imports that we have

125+program, including the imports that we have omitted here.

97−omitted here.

126

99#### Count Bytes (wc -c)

127#### Count Bytes (wc -c)

100

128

@@ -185,7 +213,7 @@ We compare two equivalent implementations: one using [Streamly][],

185and the other using C.

213and the other using C.

186

214

187The performance of the [Streamly word counting

215The performance of the [Streamly word counting

188−implementation](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.hs) is:

216+implementation][WordCount.hs] is:

189

217

190```

218```

191$ time WordCount-hs gutenberg-500MB.txt

219$ time WordCount-hs gutenberg-500MB.txt

@@ -196,7 +224,7 @@ user 0m1.697s

196sys 0m0.128s

224sys 0m0.128s

197```

225```

198

226

199−The performance of an equivalent [wc implementation in C](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.c) is:

227+The performance of an equivalent [wc implementation in C][WordCount.c] is:

200

228

201```

229```

202$ time WordCount-c gutenberg-500MB.txt

230$ time WordCount-c gutenberg-500MB.txt

@@ -217,10 +245,8 @@ To count words in parallel we first divide the stream into chunks

217counts across chunks. We use the same code as above except that we use

245counts across chunks. We use the same code as above except that we use

218arrays for our input data.

246arrays for our input data.

219

247

220−Please see the file

248+Please see the file [WordCountParallel.hs][] for the complete working

221−[WordCountParallel.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountParallel.hs)

222−for the complete working code for this example, including the imports

249+code for this example, including the imports that we have omitted below.

223−that we have omitted below.

224

250

225The `countArray` function counts the line, word, char counts in one chunk:

251The `countArray` function counts the line, word, char counts in one chunk:

226

252

@@ -235,7 +261,7 @@ countArray arr =

235```

261```

236

262

237Here the function `count` and the `Counts` data type are defined in the

263Here the function `count` and the `Counts` data type are defined in the

238−`WordCount` helper module defined in [WordCount.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordCount.hs).

264+`WordCount` helper module defined in [WordCount.hs][].

239

265

240When combining the counts in two contiguous chunks, we need to check

266When combining the counts in two contiguous chunks, we need to check

241whether the first element of the next chunk is a whitespace character

267whether the first element of the next chunk is a whitespace character

@@ -299,9 +325,8 @@ user 0m1.952s

299sys 0m0.140s

325sys 0m0.140s

300```

326```

301

327

302−These example programs have assumed ASCII encoded

328+These example programs have assumed ASCII encoded input data. For UTF-8

303−input data. For UTF-8 streams, we have a [concurrent wc

329+streams, we have a [concurrent wc implementation][WordCountUTF8.hs]

304−implementation](https://github.com/composewell/streamly-examples/blob/master/examples/WordCountUTF8.hs)

305with UTF-8 decoding. This concurrent implementation performs as well

330with UTF-8 decoding. This concurrent implementation performs as well

306as the standard `wc` program in serial benchmarks. In concurrent mode

331as the standard `wc` program in serial benchmarks. In concurrent mode

307[Streamly][]'s implementation can utilise multiple processing cores if

332[Streamly][]'s implementation can utilise multiple processing cores if

@@ -321,10 +346,8 @@ dictionary lookup server which can serve word meanings to multiple

321clients concurrently. This example demonstrates the use of the concurrent

346clients concurrently. This example demonstrates the use of the concurrent

322`mapM` combinator.

347`mapM` combinator.

323

348

324−Please see the file

349+Please see the file [WordServer.hs][] for the complete code for this

325−[WordServer.hs](https://github.com/composewell/streamly-examples/blob/master/examples/WordServer.hs)

326−for the complete code for this example, including the imports that we

350+example, including the imports that we have omitted below.

327−have omitted below.

328

351

329```haskell

352```haskell

330import qualified Streamly.Data.Fold as Fold

353import qualified Streamly.Data.Fold as Fold

@@ -376,10 +399,8 @@ the merged logs are written to a file or to a network destination.

376This example uses the `concatMapWith` combinator to merge multiple

399This example uses the `concatMapWith` combinator to merge multiple

377streams concurrently.

400streams concurrently.

378

401

379−Please see the file

402+Please see the file [MergeServer.hs][] for the complete working code,

380−[MergeServer.hs](https://github.com/composewell/streamly-examples/blob/master/examples/MergeServer.hs)

381−for the complete working code, including the imports that we have

403+including the imports that we have omitted below.

382−omitted below.

383

404

384```haskell

405```haskell

385import qualified Streamly.Data.Unfold as Unfold

406import qualified Streamly.Data.Unfold as Unfold

@@ -425,10 +446,8 @@ values back to the input, while it lets the `Right` values (file names

425in this case) pass through to the output. The `Stream.ahead` stream

446in this case) pass through to the output. The `Stream.ahead` stream

426joining combinator then makes it iterate on the directories concurrently.

447joining combinator then makes it iterate on the directories concurrently.

427

448

428−Please see the file

449+Please see the file [ListDir.hs][] for the complete working code,

429−[ListDir.hs](https://github.com/composewell/streamly-examples/blob/master/examples/ListDir.hs)

430−for the complete working code, including the imports that we have

450+including the imports that we have omitted below.

431−omitted below.

432

451

433```haskell

452```haskell

434import Streamly.Internal.Data.Stream.IsStream (iterateMapLeftsWith)

453import Streamly.Internal.Data.Stream.IsStream (iterateMapLeftsWith)

@@ -469,17 +488,14 @@ main =

469 & Stream.mapM_ print -- IO ()

488 & Stream.mapM_ print -- IO ()

470```

489```

471

490

472−Please see the file

473−[Rate.hs](https://github.com/composewell/streamly-examples/blob/master/examples/Rate.hs)

474−for the complete working code.

491+Please see the file [Rate.hs][] for the complete working code.

475

492

476The concurrency of the stream is automatically controlled to match the

493The concurrency of the stream is automatically controlled to match the

477specified rate. [Streamly][]'s rate control works precisely even at

494specified rate. [Streamly][]'s rate control works precisely even at

478throughputs as high as millions of yields per second.

495throughputs as high as millions of yields per second.

479

496

480−For more sophisticated rate control needs please see the [Haddock

497+For more sophisticated rate control needs please see the Streamly [reference

481−documentation](https://hackage.haskell.org/package/streamly-0.7.3/docs/Streamly.html)

498+documentation][Streamly].

482−for the Streamly.

483

499

484### Reactive Programming

500### Reactive Programming

485

501

@@ -491,23 +507,21 @@ Please also see the pre-release sampling combinators in the

491`Streamly.Internal.Data.Stream.IsStream.Top` module for `throttle` and

507`Streamly.Internal.Data.Stream.IsStream.Top` module for `throttle` and

492`debounce` like operations.

508`debounce` like operations.

493

509

494−The examples

510+The examples [AcidRain.hs][] and [CirclingSquare.hs][] demonstrate

495−[AcidRain.hs](https://github.com/composewell/streamly-examples/tree/master/examples/AcidRain.hs)

496−and

497−[CirclingSquare.hs](https://github.com/composewell/streamly-examples/tree/master/examples/CirclingSquare.hs)

498−demonstrate reactive programming using [Streamly][].

511+reactive programming using [Streamly][].

499

512

500### More Examples

513### More Examples

501

514

502If you would like to view more examples, please visit the [Streamly

515If you would like to view more examples, please visit the [Streamly

503−Examples](https://github.com/composewell/streamly-examples) web page.

516+Examples][streamly-examples] web page.

504

517

505### Further Reading

518### Further Reading

506

519

520+* [Streaming Benchmarks][streaming-benchmarks]

507−* [Streaming Benchmarks](https://github.com/composewell/streaming-benchmarks) | [Concurrency Benchmarks](https://github.com/composewell/concurrency-benchmarks)

521+* [Concurrency Benchmarks][concurrency-benchmarks]

508−* [Functional Conf 2019 Video](https://www.youtube.com/watch?v=uzsqgdMMgtk) | [Functional Conf 2019 Slides](https://www.slideshare.net/HarendraKumar10/streamly-concurrent-data-flow-programming)

522+* Functional Conf 2019 [Video](https://www.youtube.com/watch?v=uzsqgdMMgtk) | [Slides](https://www.slideshare.net/HarendraKumar10/streamly-concurrent-data-flow-programming)

509* [Other Guides](docs/)

523* [Other Guides](docs/)

510−* [Streamly Homepage](https://streamly.composewell.com)

524+* [Streamly Homepage][Streamly]

511

525

512## Performance

526## Performance

513

527

@@ -536,8 +550,8 @@ you use this plugin for applications that are performance sensitive.

536

550

537We measured several Haskell streaming implementations

551We measured several Haskell streaming implementations

538using various micro-benchmarks. Please see the [streaming

552using various micro-benchmarks. Please see the [streaming

539−benchmarks](https://github.com/composewell/streaming-benchmarks) page

553+benchmarks][streaming-benchmarks] page for a detailed comparison of

540−for a detailed comparison of Streamly against other streaming libraries.

554+Streamly against other streaming libraries.

541

555

542Our results show that [Streamly][] is the fastest effectful streaming

556Our results show that [Streamly][] is the fastest effectful streaming

543implementation on almost all the measured microbenchmarks. In many cases

557implementation on almost all the measured microbenchmarks. In many cases

@@ -558,8 +572,8 @@ programming with streams and arrays, for reading and writing from the

558file system and from the network, for time domain programming (reactive

572file system and from the network, for time domain programming (reactive

559programming), and for reacting to file system events using `fsnotify`.

573programming), and for reacting to file system events using `fsnotify`.

560

574

561−Please view [Streamly's documentation](https://streamly.composewell.com)

575+Please view [Streamly's documentation][Streamly] for more information

562−for more information about Streamly's features.

576+about Streamly's features.

563

577

564### Concurrency

578### Concurrency

565

579

@@ -577,9 +591,8 @@ very little overhead even where the task size is very

577small, because Streamly will automatically switch between

591small, because Streamly will automatically switch between

578serial or batched execution of tasks on the same CPU depending

592serial or batched execution of tasks on the same CPU depending

579on whichever is more efficient. Please see our [concurrency

593on whichever is more efficient. Please see our [concurrency

580−benchmarks](https://github.com/composewell/concurrency-benchmarks) for

594+benchmarks][concurrency-benchmarks] for more detailed performance

581−more detailed performance measurements, and for a comparison with the

595+measurements, and for a comparison with the `async` package.

582−`async` package.

583

596

584### Design Goals

597### Design Goals

585

598

@@ -618,14 +631,13 @@ list of contributors, credits and licenses.

618## Licensing

631## Licensing

619

632

620Streamly is an [open source](https://github.com/composewell/streamly)

633Streamly is an [open source](https://github.com/composewell/streamly)

621−project available under a liberal [BSD-3-Clause

634+project available under a liberal [BSD-3-Clause license][LICENSE]

622−license](https://github.com/composewell/streamly/blob/master/LICENSE).

623

635

624## Contributing to Streamly

636## Contributing to Streamly

625

637

626As an open project we welcome contributions:

638As an open project we welcome contributions:

627

639

628−* [Streamly Contributor's Guide](CONTRIBUTING.md)

640+* [Streamly Contributor's Guide][CONTRIBUTING.md]

629* [Contact the streamly development team](mailto:streamly@composewell.com)

641* [Contact the streamly development team](mailto:streamly@composewell.com)

630

642

631## Getting Support

643## Getting Support

streamly.cabal+3−3

@@ -1,11 +1,11 @@

1cabal-version: 2.2

2name: streamly

3version: 0.7.2

4−synopsis: Data flow programming and declarative concurrency

4+synopsis: Dataflow programming and declarative concurrency

5description:

6 <https://streamly.composewell.com Streamly> is a framework to build

7 reliable and scalable software systems from modular building blocks

8− using data flow programming (streaming) and declarative concurrency.

8+ using dataflow programming (streaming) and declarative concurrency.

9 Powerful stream fusion framework in streamly allows high-performance,

10 modular combinatorial programming. Key features include:

11 .

@@ -28,7 +28,7 @@ description:

28 <https://streamly.composewell.com Streamly homepage>.

29 .

31−homepage: https://github.com/composewell/streamly

31+homepage: https://streamly.composewell.com

32bug-reports: https://github.com/composewell/streamly/issues

33license: BSD-3-Clause

34license-file: LICENSE