4synopsis: Beautiful Streaming, Concurrent and Reactive Composition
6 Streamly is a framework for writing programs in a high level, declarative
7 data flow programming paradigm. It provides a simple API, very close to
8 standard Haskell lists. A program is expressed as a composition of data
9 processing pipes, generally known as streams. Streams can be generated,
10 merged, chained, mapped, zipped, and consumed concurrently – enabling a high
11 level, declarative yet concurrent composition of programs. Programs can be
12 concurrent or non-concurrent without any significant change. Concurrency is
13 auto scaled based on consumption rate. Programmers do not have to be aware
14 of threads, locking or synchronization to write scalable concurrent programs.
15 Streamly provides C like performance, orders of magnitude better compared to
16 existing streaming libraries.
18 Streamly is designed to express the full spectrum of programs with highest
19 performance. Do not think that if you are writing a small and simple program
20 it may not be for you. It expresses a small "hello world" program with the
21 same efficiency, simplicity and elegance as a large scale concurrent
22 application. It unifies many different aspects of special purpose libraries
23 into a single yet simple framework.
25 Streamly covers the functionality provided by Haskell lists as well as the
26 functionality provided by streaming libraries like
27 <https://hackage.haskell.org/package/streaming streaming>,
28 <https://hackage.haskell.org/package/pipes pipes>, and
29 <https://hackage.haskell.org/package/conduit conduit> with a simpler API and
30 better performance. Streamly provides
31 advanced stream composition including various ways of appending, merging,
32 zipping, splitting, grouping, distributing, partitioning and unzipping of
33 streams with true streaming and with concurrency. Streamly subsumes the
34 functionality of list transformer libraries like @pipes@ or
35 <https://hackage.haskell.org/package/list-t list-t> and also the logic
36 programming library <https://hackage.haskell.org/package/logict logict>.
37 The grouping, splitting and windowing combinators in streamly can be compared
38 to the window operators in <https://flink.apache.org/ Apache Flink>.
39 However, compared to Flink streamly has a pure functional, succinct and
42 The concurrency capabilities of streamly are much more advanced and powerful
43 compared to the basic concurrency functionality provided by the
44 <https://hackage.haskell.org/package/async async> package. Streamly is a
45 first class reactive programming library. If you are familiar with
46 <http://reactivex.io/ Reactive Extensions> you will find that it is very
47 similar. For most RxJs combinators you can find or write corresponding ones
48 in streamly. Streamly can be used as an alternative to
49 <https://hackage.haskell.org/package/Yampa Yampa> or
50 <https://hackage.haskell.org/package/reflex reflex> as well.
52 Streamly focuses on practical engineering with high performance. From well
53 written streamly programs one can expect performance competitive to C. High
54 performance streaming eliminates the need for string and text libraries like
55 <https://hackage.haskell.org/package/bytestring bytestring>,
56 <https://hackage.haskell.org/package/text text> and their lazy and strict
57 flavors. The confusion and cognitive overhead arising from different
58 string types is eliminated. The two fundamental types in streamly are arrays
59 for storage and streams for processing. Strings and text are simply streams
60 or arrays of 'Char' as they should be. Arrays in streamly have performance
61 at par with the vector library.
63 Where to find more information:
65 * /Quick Overview/: <src/README.md README file> in the package
66 * /Building/: <src/docs/Build.md Build guide> for optimal performance
67 * /Detailed Tutorial/: "Streamly.Tutorial" module in the haddock documentation
68 * /Interoperation/: "Streamly.Tutorial" module for interop with other
70 * /Reference Documentation/: Haddock documentation for the respective modules
71 * /Examples/: <src/examples examples directory> in the package
72 * /Guides/: <src/docs docs directory> in the package, for documentation on
73 advanced topics, limitations, semantics of the library or on specific use
75 * <https://github.com/composewell/streaming-benchmarks Streaming Benchmarks>
76 * <https://github.com/composewell/concurrency-benchmarks Concurrency Benchmarks>
78 For additional unreleased/experimental APIs, build the haddock docs using:
80 > $ cabal haddock --haddock-option="--show-all"
81 > $ stack haddock --haddock-arguments "--show-all" --no-haddock-deps
84homepage: https://github.com/composewell/streamly
85bug-reports: https://github.com/composewell/streamly/issues
88tested-with: GHC==7.10.3
94maintainer: streamly@composewell.com
95copyright: 2017 Harendra Kumar
96category: Control, Concurrency, Streaming, Reactivity
97stability: Experimental
103 credits/base-4.12.0.0.txt
104 credits/bjoern-2008-2009.txt
105 credits/clock-0.7.2.txt
106 credits/foldl-1.4.5.txt
107 credits/pipes-concurrency-2.0.8.txt
108 credits/pipes-concurrency.txt
109 credits/transient-0.5.5.txt
110 credits/vector-0.12.0.2.txt
111 credits/Yampa-0.10.6.2.txt
113 docs/streamly-vs-async.md
114 docs/streamly-vs-lists.md
121 src/Streamly/Internal/Data/Stream/Instances.hs
122 src/Streamly/Internal/Data/Stream/inline.hs
125 src/Streamly/Internal/Data/Time/config.h.in
131 src/Streamly/Internal/Data/Time/config.h
133source-repository head
135 location: https://github.com/composewell/streamly
138 description: Benchmark build
143 description: Enable inspection testing
148 description: Debug build with asserts enabled
153 description: Development build
158 description: Use llvm backend for better performance
163 description: Disable chart generation
168 description: Disable rewrite rules
173 description: Use CPS style streams when possible
178 description: Build including examples
183 description: Build including SDL examples
187-------------------------------------------------------------------------------
189-------------------------------------------------------------------------------
191common compile-options
192 default-language: Haskell2010
194 cpp-options: -DUSE_STREAMK_ONLY
197 cpp-options: -DDISABLE_FUSION
200 cpp-options: -DDEVBUILD
203 cpp-options: -DINSPECTION
211 ghc-options: -Wmissed-specialisations
212 -Wall-missed-specialisations
214 if flag(dev) || flag(debug)
215 ghc-options: -fno-ignore-asserts
218 ghc-options: -Wcompat
219 -Wunrecognised-warning-flags
221 -Wincomplete-record-updates
222 -Wincomplete-uni-patterns
223 -Wredundant-constraints
224 -Wnoncanonical-monad-instances
226common optimization-options
228 -fspec-constr-recursive=16
231common threading-options
232 ghc-options: -threaded
235-- We need optimization options here to optimize internal (non-inlined)
236-- versions of functions. Also, we have some benchmarking inspection tests
237-- part of the library when built with --benchmarks flag. Thos tests fail
238-- if we do not use optimization options here. It was observed that due to
239-- -O2 here some concurrent/nested benchmarks improved and others regressed.
240-- We can investigate a bit more here why the regression occurred.
242 import: compile-options, optimization-options
244-- Compilation for coverage builds on CI machines takes too long without -O0
245-- XXX we should use coverage flag for that, -O0 may take too long to run tests
248 import: compile-options, threading-options
252-- Used by maxrate test, benchmarks and executables
254 import: compile-options, optimization-options, threading-options
256-- Some benchmarks are threaded some are not
258 import: compile-options, optimization-options
259 ghc-options: -with-rtsopts "-T"
261-------------------------------------------------------------------------------
263-------------------------------------------------------------------------------
267 js-sources: jsbits/clock.js
268 include-dirs: src/Streamly/Internal/Data/Time
269 , src/Streamly/Streams
271 c-sources: src/Streamly/Internal/Data/Time/Windows.c
273 c-sources: src/Streamly/Internal/Data/Time/Darwin.c
277 Streamly.Memory.Malloc
278 , Streamly.Memory.Ring
281 , Streamly.Internal.Data.Stream.StreamK.Type
282 , Streamly.Internal.Data.Stream.StreamK
283 , Streamly.Internal.Data.Stream.StreamDK.Type
284 , Streamly.Internal.Data.Stream.StreamDK
285 , Streamly.Internal.Data.Stream.StreamD
286 , Streamly.Internal.Data.Stream.Enumeration
287 , Streamly.Internal.Data.Stream.Prelude
289 -- Higher level streams
290 , Streamly.Internal.Data.Stream.SVar
291 , Streamly.Internal.Data.Stream.Serial
292 , Streamly.Internal.Data.Stream.Async
293 , Streamly.Internal.Data.Stream.Parallel
294 , Streamly.Internal.Data.Stream.Ahead
295 , Streamly.Internal.Data.Stream.Zip
296 , Streamly.Internal.Data.Stream.Combinators
298 , Streamly.FileSystem.IOVec
299 , Streamly.FileSystem.FDIO
300 , Streamly.FileSystem.FD
302 exposed-modules: Streamly.Prelude
304 , Streamly.Data.Array
306 , Streamly.Data.Unfold
307 , Streamly.Data.Unicode.Stream
310 , Streamly.Memory.Array
311 , Streamly.FileSystem.Handle
316 , Streamly.Internal.Control.Monad
317 , Streamly.Internal.Data.Strict
318 , Streamly.Internal.Data.Atomics
319 , Streamly.Internal.Data.Time
320 , Streamly.Internal.Data.Time.Units
321 , Streamly.Internal.Data.Time.Clock
322 , Streamly.Internal.Data.Stream.StreamD.Type
323 , Streamly.Internal.Data.SVar
324 , Streamly.Internal.Data.Array
325 , Streamly.Internal.Memory.Array.Types
326 , Streamly.Internal.Memory.Array
327 , Streamly.Internal.Memory.ArrayStream
328 , Streamly.Internal.Data.Fold.Types
329 , Streamly.Internal.Data.Fold
330 , Streamly.Internal.Data.Sink.Types
331 , Streamly.Internal.Data.Sink
332 , Streamly.Internal.Data.Unfold.Types
333 , Streamly.Internal.Data.Unfold
334 , Streamly.Internal.Data.Pipe.Types
335 , Streamly.Internal.Data.Pipe
336 , Streamly.Internal.Data.List
337 , Streamly.Internal.FileSystem.Handle
338 , Streamly.Internal.FileSystem.Dir
339 , Streamly.Internal.FileSystem.File
340 , Streamly.Internal.Data.Unicode.Stream
341 , Streamly.Internal.Data.Unicode.Char
342 , Streamly.Internal.Memory.Unicode.Array
343 , Streamly.Internal.Prelude
346 Streamly.Network.Socket
347 , Streamly.Network.Inet.TCP
349 , Streamly.Internal.Network.Socket
350 , Streamly.Internal.Network.Inet.TCP
354 Streamly.Internal.Benchmark.FileIO.Array
355 , Streamly.Internal.Benchmark.FileIO.Stream
356 , Streamly.Internal.Benchmark.Prelude
358 build-depends: base >= 4.8 && < 5
359 , ghc-prim >= 0.2 && < 0.6
360 , deepseq >= 1.4.1 && < 1.5
361 , containers >= 0.5.8.2 && < 0.7
362 , heaps >= 0.3 && < 0.4
363 , directory >= 1.3 && < 1.4
364 , primitive >= 0.6.4.0 && < 0.8
367 , atomic-primops >= 0.8 && < 0.9
368 , lockfree-queue >= 0.2.3 && < 0.3
371 , exceptions >= 0.8 && < 0.11
372 , monad-control >= 1.0 && < 2
374 , transformers >= 0.4 && < 0.6
375 , transformers-base >= 0.4 && < 0.5
378 build-depends: template-haskell >= 2.14 && < 2.16
379 , inspection-testing >= 0.4 && < 0.5
383 network >= 2.6 && < 4
386 semigroups >= 0.18 && < 0.19
388-------------------------------------------------------------------------------
390-------------------------------------------------------------------------------
394 type: exitcode-stdio-1.0
396 js-sources: jsbits/clock.js
401 , hspec >= 2.0 && < 3
402 , containers >= 0.5.8.2 && < 0.7
403 , transformers >= 0.4 && < 0.6
405 , exceptions >= 0.8 && < 0.11
406 default-language: Haskell2010
408test-suite pure-streams-base
410 type: exitcode-stdio-1.0
411 main-is: PureStreams.hs
416 , hspec >= 2.0 && < 3
417 default-language: Haskell2010
419test-suite pure-streams-streamly
421 type: exitcode-stdio-1.0
422 main-is: PureStreams.hs
424 cpp-options: -DUSE_STREAMLY_LIST
428 , hspec >= 2.0 && < 3
429 default-language: Haskell2010
433 type: exitcode-stdio-1.0
435 js-sources: jsbits/clock.js
440 , QuickCheck >= 2.10 && < 2.14
441 , hspec >= 2.0 && < 3
444 transformers >= 0.4 && < 0.6
445 default-language: Haskell2010
449 type: exitcode-stdio-1.0
451 js-sources: jsbits/clock.js
456 , QuickCheck >= 2.10 && < 2.14
457 , hspec >= 2.0 && < 3
460 transformers >= 0.4 && < 0.6
461 default-language: Haskell2010
463test-suite unpinned-array-test
465 type: exitcode-stdio-1.0
466 main-is: Streamly/Test/Data/Array.hs
467 js-sources: jsbits/clock.js
472 , QuickCheck >= 2.10 && < 2.14
473 , hspec >= 2.0 && < 3
476 transformers >= 0.4 && < 0.6
477 default-language: Haskell2010
479test-suite string-test
481 type: exitcode-stdio-1.0
483 js-sources: jsbits/clock.js
488 , QuickCheck >= 2.10 && < 2.14
489 , hspec >= 2.0 && < 3
492 transformers >= 0.4 && < 0.6
493 default-language: Haskell2010
497 type: exitcode-stdio-1.0
498 default-language: Haskell2010
500 js-sources: jsbits/clock.js
507 , clock >= 0.7.1 && < 0.9
508 , hspec >= 2.0 && < 3
509 , random >= 1.0.0 && < 2
515 type: exitcode-stdio-1.0
516 default-language: Haskell2010
523test-suite nested-loops
525 type: exitcode-stdio-1.0
526 default-language: Haskell2010
527 main-is: nested-loops.hs
532 , random >= 1.0.0 && < 2
534test-suite parallel-loops
536 type: exitcode-stdio-1.0
537 default-language: Haskell2010
538 main-is: parallel-loops.hs
543 , random >= 1.0.0 && < 2
545-------------------------------------------------------------------------------
547-------------------------------------------------------------------------------
550 import: bench-options
551 type: exitcode-stdio-1.0
552 hs-source-dirs: benchmark
559 , deepseq >= 1.4.1 && < 1.5
560 , random >= 1.0 && < 2.0
561 , gauge >= 0.2.4 && < 0.3
564 transformers >= 0.4 && < 0.6
568benchmark linear-async
569 import: bench-options
570 cpp-options: -DLINEAR_ASYNC
571 type: exitcode-stdio-1.0
572 hs-source-dirs: benchmark
573 main-is: LinearAsync.hs
579 , deepseq >= 1.4.1 && < 1.5
580 , random >= 1.0 && < 2.0
581 , gauge >= 0.2.4 && < 0.3
584 transformers >= 0.4 && < 0.6
589 import: bench-options
590 type: exitcode-stdio-1.0
591 hs-source-dirs: benchmark
592 main-is: LinearRate.hs
598 , deepseq >= 1.4.1 && < 1.5
599 , random >= 1.0 && < 2.0
600 , gauge >= 0.2.4 && < 0.3
603 transformers >= 0.4 && < 0.6
608 import: bench-options
609 type: exitcode-stdio-1.0
610 hs-source-dirs: benchmark
612 other-modules: NestedOps
616 , deepseq >= 1.4.1 && < 1.5
617 , random >= 1.0 && < 2.0
618 , gauge >= 0.2.4 && < 0.3
621 transformers >= 0.4 && < 0.6
623benchmark nestedUnfold
624 import: bench-options
625 type: exitcode-stdio-1.0
626 hs-source-dirs: benchmark
627 main-is: NestedUnfold.hs
628 other-modules: NestedUnfoldOps
632 , deepseq >= 1.4.1 && < 1.5
633 , random >= 1.0 && < 2.0
634 , gauge >= 0.2.4 && < 0.3
637 transformers >= 0.4 && < 0.6
639benchmark unpinned-array
640 import: bench-options
641 type: exitcode-stdio-1.0
642 hs-source-dirs: benchmark
643 main-is: Streamly/Benchmark/Data/Array.hs
644 other-modules: Streamly.Benchmark.Data.ArrayOps
648 , deepseq >= 1.4.1 && < 1.5
649 , random >= 1.0 && < 2.0
650 , gauge >= 0.2.4 && < 0.3
653 transformers >= 0.4 && < 0.6
656 import: bench-options
657 type: exitcode-stdio-1.0
658 hs-source-dirs: benchmark
660 other-modules: ArrayOps
664 , deepseq >= 1.4.1 && < 1.5
665 , random >= 1.0 && < 2.0
666 , gauge >= 0.2.4 && < 0.3
669 transformers >= 0.4 && < 0.6
672 import: bench-options
673 type: exitcode-stdio-1.0
674 -- A value of 400 works better for some benchmarks, however, it takes
675 -- extraordinary amount of time to compile with that.
676 -- ghc-options: -funfolding-use-threshold=150
677 hs-source-dirs: benchmark
684 , gauge >= 0.2.4 && < 0.3
685 , typed-process >= 0.2.3 && < 0.3
686 , deepseq >= 1.4.1 && < 1.5
691 import: bench-options
692 type: exitcode-stdio-1.0
693 hs-source-dirs: benchmark
694 main-is: Concurrent.hs
700 , gauge >= 0.2.4 && < 0.3
704-------------------------------------------------------------------------------
705-- Internal benchmarks for unexposed modules
706-------------------------------------------------------------------------------
708-- We have to copy the streamly library modules here because there is no
709-- way to use unexposed modules from the library.
712 import: bench-options
713 type: exitcode-stdio-1.0
714 include-dirs: src/Streamly/Internal/Data/Time
715 , src/Streamly/Streams
717 c-sources: src/Streamly/Internal/Data/Time/Windows.c
719 c-sources: src/Streamly/Internal/Data/Time/Darwin.c
720 hs-source-dirs: benchmark, src
721 main-is: BaseStreams.hs
722 other-modules: Streamly.Internal.Data.Atomics
723 , Streamly.Internal.Data.Stream.StreamD.Type
724 , Streamly.Internal.Data.SVar
725 , Streamly.Internal.Data.Time.Units
726 , Streamly.Internal.Data.Time.Clock
727 , Streamly.Internal.Data.Stream.StreamDK.Type
728 , Streamly.Internal.Data.Stream.StreamDK
729 , Streamly.Internal.Data.Stream.StreamK.Type
730 , Streamly.Internal.Data.Stream.StreamK
731 , Streamly.Internal.Data.Stream.StreamD
732 , Streamly.Internal.Data.Stream.Prelude
733 , Streamly.FileSystem.IOVec
743 , deepseq >= 1.4.1 && < 1.5
744 , random >= 1.0 && < 2.0
745 , gauge >= 0.2.4 && < 0.3
747 , ghc-prim >= 0.2 && < 0.6
748 , containers >= 0.5.8.2 && < 0.7
749 , heaps >= 0.3 && < 0.4
752 , atomic-primops >= 0.8 && < 0.9
753 , lockfree-queue >= 0.2.3 && < 0.3
755 , exceptions >= 0.8 && < 0.11
756 , monad-control >= 1.0 && < 2
758 , transformers >= 0.4 && < 0.6
759 , transformers-base >= 0.4 && < 0.5
763 semigroups >= 0.18 && < 0.19
768 import: bench-options
769 hs-source-dirs: benchmark, src
770 include-dirs: src/Streamly/Internal/Data/Time
771 , src/Streamly/Streams
773 c-sources: src/Streamly/Internal/Data/Time/Windows.c
775 c-sources: src/Streamly/Internal/Data/Time/Darwin.c
776 main-is: NanoBenchmarks.hs
777 other-modules: Streamly.Internal.Data.Atomics
778 , Streamly.Internal.Data.Stream.StreamD.Type
779 , Streamly.Internal.Data.SVar
780 , Streamly.Internal.Data.Time.Units
781 , Streamly.Internal.Data.Time.Clock
782 , Streamly.Internal.Data.Stream.StreamK.Type
783 , Streamly.Internal.Data.Stream.StreamK
784 , Streamly.FileSystem.IOVec
785 , Streamly.Internal.Data.Stream.StreamD
790 , gauge >= 0.2.4 && < 0.3
791 , ghc-prim >= 0.2 && < 0.6
792 , containers >= 0.5.8.2 && < 0.7
793 , deepseq >= 1.4.1 && < 1.5
794 , heaps >= 0.3 && < 0.4
795 , random >= 1.0 && < 2.0
798 , atomic-primops >= 0.8 && < 0.9
799 , lockfree-queue >= 0.2.3 && < 0.3
801 , exceptions >= 0.8 && < 0.11
802 , monad-control >= 1.0 && < 2
804 , transformers >= 0.4 && < 0.6
805 , transformers-base >= 0.4 && < 0.5
810 import: bench-options
811 hs-source-dirs: benchmark
813 default-language: Haskell2010
819 , gauge >= 0.2.4 && < 0.3
820 , random >= 1.0 && < 2.0
825 default-language: Haskell2010
827 hs-source-dirs: benchmark
829 if flag(dev) && !flag(no-charts) && !impl(ghcjs)
833 , bench-show >= 0.3 && < 0.4
835 , transformers >= 0.4 && < 0.6
839-------------------------------------------------------------------------------
841-------------------------------------------------------------------------------
843executable SearchQuery
845 main-is: SearchQuery.hs
846 hs-source-dirs: examples
847 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
852 , http-conduit >= 2.2.2 && < 2.4
859 hs-source-dirs: examples
860 if flag(examples) || flag(examples-sdl)
865 , directory >= 1.3 && < 1.4
868 transformers >= 0.4 && < 0.6
874 main-is: MergeSort.hs
875 hs-source-dirs: examples
876 if flag(examples) || flag(examples-sdl)
881 , random >= 1.0.0 && < 2
888 hs-source-dirs: examples
889 if flag(examples) || flag(examples-sdl)
897 semigroups >= 0.18 && < 0.19
898 , transformers >= 0.4 && < 0.6
902executable CirclingSquare
904 main-is: CirclingSquare.hs
905 hs-source-dirs: examples
906 if flag(examples-sdl)
911 , SDL >= 0.6.5 && < 0.7
915executable ControlFlow
917 main-is: ControlFlow.hs
918 hs-source-dirs: examples
919 if flag(examples) || flag(examples-sdl)
924 , exceptions >= 0.8 && < 0.11
925 , transformers >= 0.4 && < 0.6
926 , transformers-base >= 0.4 && < 0.5
929 semigroups >= 0.18 && < 0.19
936 hs-source-dirs: examples
937 if flag(examples) || flag(examples-sdl)
945executable FileIOExamples
947 main-is: FileIOExamples.hs
948 hs-source-dirs: examples
949 if flag(examples) || flag(examples-sdl)
959 main-is: EchoServer.hs
960 hs-source-dirs: examples
961 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
966 , network >= 2.6 && < 4
969 transformers >= 0.4 && < 0.6
973executable FileSinkServer
975 main-is: FileSinkServer.hs
976 hs-source-dirs: examples
977 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
982 , network >= 2.6 && < 4
985 transformers >= 0.4 && < 0.6
989executable FromFileClient
991 main-is: FromFileClient.hs
992 hs-source-dirs: examples
993 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
1001executable WordClassifier
1003 main-is: WordClassifier.hs
1004 hs-source-dirs: examples
1005 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
1009 , base >= 4.8 && < 5
1010 , hashable >= 1.2 && < 1.4
1011 , unordered-containers >= 0.2 && < 0.3
1017 main-is: WordCount.hs
1018 hs-source-dirs: examples
1019 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
1023 , base >= 4.8 && < 5
1024 , vector >= 0.12 && < 0.13
1030 main-is: CamelCase.hs
1031 hs-source-dirs: examples
1032 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
1036 , base >= 4.8 && < 5
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list