4synopsis: Beautiful Streaming, Concurrent and Reactive Composition
6 Streamly is a framework for writing programs in a high level, declarative
7 data flow programming paradigm. It provides a simple API, very close to
8 standard Haskell lists. A program is expressed as a composition of data
9 processing pipes, generally known as streams. Streams can be generated,
10 merged, chained, mapped, zipped, and consumed concurrently – enabling a high
11 level, declarative yet concurrent composition of programs. Programs can be
12 concurrent or non-concurrent without any significant change. Concurrency is
13 auto scaled based on consumption rate. Programmers do not have to be aware
14 of threads, locking or synchronization to write scalable concurrent programs.
15 Streamly provides C like performance, orders of magnitude better compared to
16 existing streaming libraries.
18 Streamly is designed to express the full spectrum of programs with highest
19 performance. Do not think that if you are writing a small and simple program
20 it may not be for you. It expresses a small "hello world" program with the
21 same efficiency, simplicity and elegance as a large scale concurrent
22 application. It unifies many different aspects of special purpose libraries
23 into a single yet simple framework.
25 Streamly covers the functionality provided by Haskell lists as well as the
26 functionality provided by streaming libraries like
27 <https://hackage.haskell.org/package/streaming streaming>,
28 <https://hackage.haskell.org/package/pipes pipes>, and
29 <https://hackage.haskell.org/package/conduit conduit> with a simpler API and
30 better performance. Streamly provides
31 advanced stream composition including various ways of appending, merging,
32 zipping, splitting, grouping, distributing, partitioning and unzipping of
33 streams with true streaming and with concurrency. Streamly subsumes the
34 functionality of list transformer libraries like @pipes@ or
35 <https://hackage.haskell.org/package/list-t list-t> and also the logic
36 programming library <https://hackage.haskell.org/package/logict logict>.
37 The grouping, splitting and windowing combinators in streamly can be compared
38 to the window operators in <https://flink.apache.org/ Apache Flink>.
39 However, compared to Flink streamly has a pure functional, succinct and
42 The concurrency capabilities of streamly are much more advanced and powerful
43 compared to the basic concurrency functionality provided by the
44 <https://hackage.haskell.org/package/async async> package. Streamly is a
45 first class reactive programming library. If you are familiar with
46 <http://reactivex.io/ Reactive Extensions> you will find that it is very
47 similar. For most RxJs combinators you can find or write corresponding ones
48 in streamly. Streamly can be used as an alternative to
49 <https://hackage.haskell.org/package/Yampa Yampa> or
50 <https://hackage.haskell.org/package/reflex reflex> as well.
52 Streamly focuses on practical engineering with high performance. From well
53 written streamly programs one can expect performance competitive to C. High
54 performance streaming eliminates the need for string and text libraries like
55 <https://hackage.haskell.org/package/bytestring bytestring>,
56 <https://hackage.haskell.org/package/text text> and their lazy and strict
57 flavors. The confusion and cognitive overhead arising from different
58 string types is eliminated. The two fundamental types in streamly are arrays
59 for storage and streams for processing. Strings and text are simply streams
60 or arrays of 'Char' as they should be. Arrays in streamly have performance
61 at par with the vector library.
63 Where to find more information:
65 * /Quick Overview/: <src/README.md README file> in the package
66 * /Building/: <src/docs/Build.md Build guide> for optimal performance
67 * /Detailed Tutorial/: "Streamly.Tutorial" module in the haddock documentation
68 * /Interoperation/: "Streamly.Tutorial" module for interop with other
70 * /Reference Documentation/: Haddock documentation for the respective modules
71 * /Examples/: <src/examples examples directory> in the package
72 * /Guides/: <src/docs docs directory> in the package, for documentation on
73 advanced topics, limitations, semantics of the library or on specific use
75 * <https://github.com/composewell/streaming-benchmarks Streaming Benchmarks>
76 * <https://github.com/composewell/concurrency-benchmarks Concurrency Benchmarks>
78homepage: https://github.com/composewell/streamly
79bug-reports: https://github.com/composewell/streamly/issues
82tested-with: GHC==7.10.3
88maintainer: streamly@composewell.com
89copyright: 2017 Harendra Kumar
90category: Control, Concurrency, Streaming, Reactivity
91stability: Experimental
97 credits/clock-0.7.2.txt
98 credits/foldl-1.4.5.txt
99 credits/pipes-concurrency-2.0.8.txt
100 credits/pipes-concurrency.txt
101 credits/transient-0.5.5.txt
102 credits/vector-0.12.0.2.txt
103 credits/Yampa-0.10.6.2.txt
105 docs/streamly-vs-async.md
109 src/Streamly/Streams/Instances.hs
110 src/Streamly/Streams/inline.hs
113 src/Streamly/Internal/Data/Time/config.h.in
119 src/Streamly/Internal/Data/Time/config.h
121source-repository head
123 location: https://github.com/composewell/streamly
126 description: Benchmark build
131 description: Enable inspection testing
136 description: Debug build with asserts enabled
141 description: Development build
146 description: Use llvm backend for better performance
151 description: Disable chart generation
156 description: Disable rewrite rules
161 description: Use CPS style streams when possible
166 description: Build including examples
171 description: Build including SDL examples
175-------------------------------------------------------------------------------
177-------------------------------------------------------------------------------
179common compile-options
180 default-language: Haskell2010
182 cpp-options: -DUSE_STREAMK_ONLY
185 cpp-options: -DDISABLE_FUSION
188 cpp-options: -DDEVBUILD
191 cpp-options: -DINSPECTION
199 ghc-options: -Wmissed-specialisations
200 -Wall-missed-specialisations
202 if flag(dev) || flag(debug)
203 ghc-options: -fno-ignore-asserts
206 ghc-options: -Wcompat
207 -Wunrecognised-warning-flags
209 -Wincomplete-record-updates
210 -Wincomplete-uni-patterns
211 -Wredundant-constraints
212 -Wnoncanonical-monad-instances
214common optimization-options
216 -fspec-constr-recursive=16
219common threading-options
220 ghc-options: -threaded
223-- We need optimization options here to optimize internal (non-inlined)
224-- versions of functions. Also, we have some benchmarking inspection tests
225-- part of the library when built with --benchmarks flag. Thos tests fail
226-- if we do not use optimization options here. It was observed that due to
227-- -O2 here some concurrent/nested benchmarks improved and others regressed.
228-- We can investigate a bit more here why the regression occurred.
230 import: compile-options, optimization-options
232-- Compilation for coverage builds on CI machines takes too long without -O0
233-- XXX we should use coverage flag for that, -O0 may take too long to run tests
236 import: compile-options, threading-options
240-- Used by maxrate test, benchmarks and executables
242 import: compile-options, optimization-options, threading-options
244-- Some benchmarks are threaded some are not
246 import: compile-options, optimization-options
247 ghc-options: -with-rtsopts "-T"
249-------------------------------------------------------------------------------
251-------------------------------------------------------------------------------
255 js-sources: jsbits/clock.js
256 include-dirs: src/Streamly/Internal/Data/Time
257 , src/Streamly/Streams
259 c-sources: src/Streamly/Internal/Data/Time/Windows.c
261 c-sources: src/Streamly/Internal/Data/Time/Darwin.c
265 Streamly.Memory.Malloc
266 , Streamly.Memory.Ring
269 , Streamly.Streams.StreamK.Type
270 , Streamly.Streams.StreamK
271 , Streamly.Streams.StreamDK.Type
272 , Streamly.Streams.StreamDK
273 , Streamly.Streams.StreamD
274 , Streamly.Streams.Enumeration
275 , Streamly.Streams.Prelude
277 -- Higher level streams
278 , Streamly.Streams.SVar
279 , Streamly.Streams.Serial
280 , Streamly.Streams.Async
281 , Streamly.Streams.Parallel
282 , Streamly.Streams.Ahead
283 , Streamly.Streams.Zip
284 , Streamly.Streams.Combinators
286 , Streamly.FileSystem.IOVec
287 , Streamly.FileSystem.FDIO
288 , Streamly.FileSystem.FD
290 exposed-modules: Streamly.Prelude
293 , Streamly.Data.Unfold
294 , Streamly.Data.Unicode.Stream
297 , Streamly.Memory.Array
298 , Streamly.FileSystem.Handle
303 , Streamly.Internal.Data.Strict
304 , Streamly.Internal.Data.Atomics
305 , Streamly.Internal.Data.Time
306 , Streamly.Internal.Data.Time.Units
307 , Streamly.Internal.Data.Time.Clock
308 , Streamly.Internal.Data.Stream.StreamD.Type
309 , Streamly.Internal.Data.SVar
310 , Streamly.Internal.Memory.Array.Types
311 , Streamly.Internal.Memory.Array
312 , Streamly.Internal.Memory.ArrayStream
313 , Streamly.Internal.Data.Fold.Types
314 , Streamly.Internal.Data.Fold
315 , Streamly.Internal.Data.Sink.Types
316 , Streamly.Internal.Data.Sink
317 , Streamly.Internal.Data.Unfold.Types
318 , Streamly.Internal.Data.Unfold
319 , Streamly.Internal.Data.Pipe.Types
320 , Streamly.Internal.Data.Pipe
321 , Streamly.Internal.Data.List
322 , Streamly.Internal.FileSystem.Handle
323 , Streamly.Internal.FileSystem.Dir
324 , Streamly.Internal.FileSystem.File
325 , Streamly.Internal.Data.Unicode.Stream
326 , Streamly.Internal.Data.Unicode.Char
327 , Streamly.Internal.Memory.Unicode.Array
328 , Streamly.Internal.Prelude
331 Streamly.Network.Socket
332 , Streamly.Network.Inet.TCP
334 , Streamly.Internal.Network.Socket
335 , Streamly.Internal.Network.Inet.TCP
339 Streamly.Benchmark.FileIO.Array
340 , Streamly.Benchmark.FileIO.Stream
341 , Streamly.Benchmark.Prelude
343 build-depends: base >= 4.8 && < 5
344 , ghc-prim >= 0.2 && < 0.6
345 , deepseq >= 1.4.1 && < 1.5
346 , containers >= 0.5.8.2 && < 0.7
347 , heaps >= 0.3 && < 0.4
348 , directory >= 1.3 && < 1.4
351 , atomic-primops >= 0.8 && < 0.9
352 , lockfree-queue >= 0.2.3 && < 0.3
355 , exceptions >= 0.8 && < 0.11
356 , monad-control >= 1.0 && < 2
358 , transformers >= 0.4 && < 0.6
359 , transformers-base >= 0.4 && < 0.5
362 build-depends: template-haskell >= 2.14 && < 2.16
363 , inspection-testing >= 0.4 && < 0.5
367 network >= 2.6 && < 4
370 semigroups >= 0.18 && < 0.19
372-------------------------------------------------------------------------------
374-------------------------------------------------------------------------------
378 type: exitcode-stdio-1.0
380 js-sources: jsbits/clock.js
385 , hspec >= 2.0 && < 3
386 , containers >= 0.5.8.2 && < 0.7
387 , transformers >= 0.4 && < 0.6
389 , exceptions >= 0.8 && < 0.11
390 default-language: Haskell2010
392test-suite pure-streams-base
394 type: exitcode-stdio-1.0
395 main-is: PureStreams.hs
400 , hspec >= 2.0 && < 3
401 default-language: Haskell2010
403test-suite pure-streams-streamly
405 type: exitcode-stdio-1.0
406 main-is: PureStreams.hs
408 cpp-options: -DUSE_STREAMLY_LIST
412 , hspec >= 2.0 && < 3
413 default-language: Haskell2010
417 type: exitcode-stdio-1.0
419 js-sources: jsbits/clock.js
424 , QuickCheck >= 2.10 && < 2.14
425 , hspec >= 2.0 && < 3
428 transformers >= 0.4 && < 0.6
429 default-language: Haskell2010
433 type: exitcode-stdio-1.0
435 js-sources: jsbits/clock.js
440 , QuickCheck >= 2.10 && < 2.14
441 , hspec >= 2.0 && < 3
444 transformers >= 0.4 && < 0.6
445 default-language: Haskell2010
447test-suite string-test
449 type: exitcode-stdio-1.0
451 js-sources: jsbits/clock.js
456 , QuickCheck >= 2.10 && < 2.14
457 , hspec >= 2.0 && < 3
460 transformers >= 0.4 && < 0.6
461 default-language: Haskell2010
465 type: exitcode-stdio-1.0
466 default-language: Haskell2010
468 js-sources: jsbits/clock.js
475 , clock >= 0.7.1 && < 0.9
476 , hspec >= 2.0 && < 3
477 , random >= 1.0.0 && < 2
483 type: exitcode-stdio-1.0
484 default-language: Haskell2010
491test-suite nested-loops
493 type: exitcode-stdio-1.0
494 default-language: Haskell2010
495 main-is: nested-loops.hs
500 , random >= 1.0.0 && < 2
502test-suite parallel-loops
504 type: exitcode-stdio-1.0
505 default-language: Haskell2010
506 main-is: parallel-loops.hs
511 , random >= 1.0.0 && < 2
513-------------------------------------------------------------------------------
515-------------------------------------------------------------------------------
518 import: bench-options
519 type: exitcode-stdio-1.0
520 hs-source-dirs: benchmark
527 , deepseq >= 1.4.1 && < 1.5
528 , random >= 1.0 && < 2.0
529 , gauge >= 0.2.4 && < 0.3
532 transformers >= 0.4 && < 0.6
536benchmark linear-async
537 import: bench-options
538 cpp-options: -DLINEAR_ASYNC
539 type: exitcode-stdio-1.0
540 hs-source-dirs: benchmark
541 main-is: LinearAsync.hs
547 , deepseq >= 1.4.1 && < 1.5
548 , random >= 1.0 && < 2.0
549 , gauge >= 0.2.4 && < 0.3
552 transformers >= 0.4 && < 0.6
557 import: bench-options
558 type: exitcode-stdio-1.0
559 hs-source-dirs: benchmark
560 main-is: LinearRate.hs
566 , deepseq >= 1.4.1 && < 1.5
567 , random >= 1.0 && < 2.0
568 , gauge >= 0.2.4 && < 0.3
571 transformers >= 0.4 && < 0.6
576 import: bench-options
577 type: exitcode-stdio-1.0
578 hs-source-dirs: benchmark
580 other-modules: NestedOps
584 , deepseq >= 1.4.1 && < 1.5
585 , random >= 1.0 && < 2.0
586 , gauge >= 0.2.4 && < 0.3
589 transformers >= 0.4 && < 0.6
591benchmark nestedUnfold
592 import: bench-options
593 type: exitcode-stdio-1.0
594 hs-source-dirs: benchmark
595 main-is: NestedUnfold.hs
596 other-modules: NestedUnfoldOps
600 , deepseq >= 1.4.1 && < 1.5
601 , random >= 1.0 && < 2.0
602 , gauge >= 0.2.4 && < 0.3
605 transformers >= 0.4 && < 0.6
608 import: bench-options
609 type: exitcode-stdio-1.0
610 hs-source-dirs: benchmark
612 other-modules: ArrayOps
616 , deepseq >= 1.4.1 && < 1.5
617 , random >= 1.0 && < 2.0
618 , gauge >= 0.2.4 && < 0.3
621 transformers >= 0.4 && < 0.6
624 import: bench-options
625 type: exitcode-stdio-1.0
626 -- A value of 400 works better for some benchmarks, however, it takes
627 -- extraordinary amount of time to compile with that.
628 -- ghc-options: -funfolding-use-threshold=150
629 hs-source-dirs: benchmark
636 , gauge >= 0.2.4 && < 0.3
637 , typed-process >= 0.2.3 && < 0.3
638 , deepseq >= 1.4.1 && < 1.5
643 import: bench-options
644 type: exitcode-stdio-1.0
645 hs-source-dirs: benchmark
646 main-is: Concurrent.hs
652 , gauge >= 0.2.4 && < 0.3
656-------------------------------------------------------------------------------
657-- Internal benchmarks for unexposed modules
658-------------------------------------------------------------------------------
660-- We have to copy the streamly library modules here because there is no
661-- way to use unexposed modules from the library.
664 import: bench-options
665 type: exitcode-stdio-1.0
666 include-dirs: src/Streamly/Internal/Data/Time
667 , src/Streamly/Streams
669 c-sources: src/Streamly/Internal/Data/Time/Windows.c
671 c-sources: src/Streamly/Internal/Data/Time/Darwin.c
672 hs-source-dirs: benchmark, src
673 main-is: BaseStreams.hs
674 other-modules: Streamly.Internal.Data.Atomics
675 , Streamly.Internal.Data.Stream.StreamD.Type
676 , Streamly.Internal.Data.SVar
677 , Streamly.Internal.Data.Time.Units
678 , Streamly.Internal.Data.Time.Clock
679 , Streamly.Streams.StreamDK.Type
680 , Streamly.Streams.StreamDK
681 , Streamly.Streams.StreamK.Type
682 , Streamly.Streams.StreamK
683 , Streamly.Streams.StreamD
684 , Streamly.Streams.Prelude
685 , Streamly.FileSystem.IOVec
695 , deepseq >= 1.4.1 && < 1.5
696 , random >= 1.0 && < 2.0
697 , gauge >= 0.2.4 && < 0.3
699 , ghc-prim >= 0.2 && < 0.6
700 , containers >= 0.5.8.2 && < 0.7
701 , heaps >= 0.3 && < 0.4
704 , atomic-primops >= 0.8 && < 0.9
705 , lockfree-queue >= 0.2.3 && < 0.3
707 , exceptions >= 0.8 && < 0.11
708 , monad-control >= 1.0 && < 2
710 , transformers >= 0.4 && < 0.6
711 , transformers-base >= 0.4 && < 0.5
715 semigroups >= 0.18 && < 0.19
720 import: bench-options
721 hs-source-dirs: benchmark, src
722 include-dirs: src/Streamly/Internal/Data/Time
723 , src/Streamly/Streams
725 c-sources: src/Streamly/Internal/Data/Time/Windows.c
727 c-sources: src/Streamly/Internal/Data/Time/Darwin.c
728 main-is: NanoBenchmarks.hs
729 other-modules: Streamly.Internal.Data.Atomics
730 , Streamly.Internal.Data.Stream.StreamD.Type
731 , Streamly.Internal.Data.SVar
732 , Streamly.Internal.Data.Time.Units
733 , Streamly.Internal.Data.Time.Clock
734 , Streamly.Streams.StreamK.Type
735 , Streamly.Streams.StreamK
736 , Streamly.FileSystem.IOVec
737 , Streamly.Streams.StreamD
742 , gauge >= 0.2.4 && < 0.3
743 , ghc-prim >= 0.2 && < 0.6
744 , containers >= 0.5.8.2 && < 0.7
745 , deepseq >= 1.4.1 && < 1.5
746 , heaps >= 0.3 && < 0.4
747 , random >= 1.0 && < 2.0
750 , atomic-primops >= 0.8 && < 0.9
751 , lockfree-queue >= 0.2.3 && < 0.3
753 , exceptions >= 0.8 && < 0.11
754 , monad-control >= 1.0 && < 2
756 , transformers >= 0.4 && < 0.6
757 , transformers-base >= 0.4 && < 0.5
762 import: bench-options
763 hs-source-dirs: benchmark
765 default-language: Haskell2010
771 , gauge >= 0.2.4 && < 0.3
772 , random >= 1.0 && < 2.0
777 default-language: Haskell2010
779 hs-source-dirs: benchmark
781 if flag(dev) && !flag(no-charts) && !impl(ghcjs)
785 , bench-show >= 0.3 && < 0.4
787 , transformers >= 0.4 && < 0.6
791-------------------------------------------------------------------------------
793-------------------------------------------------------------------------------
795executable SearchQuery
797 main-is: SearchQuery.hs
798 hs-source-dirs: examples
799 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
804 , http-conduit >= 2.2.2 && < 2.4
811 hs-source-dirs: examples
812 if flag(examples) || flag(examples-sdl)
817 , directory >= 1.3 && < 1.4
820 transformers >= 0.4 && < 0.6
826 main-is: MergeSort.hs
827 hs-source-dirs: examples
828 if flag(examples) || flag(examples-sdl)
833 , random >= 1.0.0 && < 2
840 hs-source-dirs: examples
841 if flag(examples) || flag(examples-sdl)
849 semigroups >= 0.18 && < 0.19
850 , transformers >= 0.4 && < 0.6
854executable CirclingSquare
856 main-is: CirclingSquare.hs
857 hs-source-dirs: examples
858 if flag(examples-sdl)
863 , SDL >= 0.6.5 && < 0.7
867executable ControlFlow
869 main-is: ControlFlow.hs
870 hs-source-dirs: examples
871 if flag(examples) || flag(examples-sdl)
876 , exceptions >= 0.8 && < 0.11
877 , transformers >= 0.4 && < 0.6
878 , transformers-base >= 0.4 && < 0.5
881 semigroups >= 0.18 && < 0.19
888 hs-source-dirs: examples
889 if flag(examples) || flag(examples-sdl)
897executable FileIOExamples
899 main-is: FileIOExamples.hs
900 hs-source-dirs: examples
901 if flag(examples) || flag(examples-sdl)
911 main-is: EchoServer.hs
912 hs-source-dirs: examples
913 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
918 , network >= 2.6 && < 4
921 transformers >= 0.4 && < 0.6
925executable FileSinkServer
927 main-is: FileSinkServer.hs
928 hs-source-dirs: examples
929 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
934 , network >= 2.6 && < 4
937 transformers >= 0.4 && < 0.6
941executable FromFileClient
943 main-is: FromFileClient.hs
944 hs-source-dirs: examples
945 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
953executable WordClassifier
955 main-is: WordClassifier.hs
956 hs-source-dirs: examples
957 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
962 , hashable >= 1.2 && < 1.4
963 , unordered-containers >= 0.2 && < 0.3
969 main-is: WordCount.hs
970 hs-source-dirs: examples
971 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
976 , vector >= 0.12 && < 0.13
982 main-is: CamelCase.hs
983 hs-source-dirs: examples
984 if (flag(examples) || flag(examples-sdl)) && !impl(ghcjs)
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list