@@ -0,0 +1,260 @@
1+# Unified Functional Abstractions in Streamly
2+
3+The goal of Streamly is to provide unified abstractions with high
4+performance. The basic abstractions in Streamly are streams and arrays
5+and yet it provides the functionality of many packages in the Haskell
6+ecosystem.
7+
8+This document discusses the related packages in the Haskell ecosystem
9+and how streamly unifies, overlaps, or compares with those. We provide
10+simple code snippets for illustrations, for a better overview of the
11+library please see the [Streamly Quick Overview](../README.md) document.
12+
13+## Existing Haskell Libraries
14+
15+In the Haskell ecosystem effectful streaming functionality is provided
16+by several streaming libraries e.g. `streaming`, nested looping by list
17+transformers e.g. `list-t`, interleaved scheduling by `logict`,
18+concurrency by `async`, time-domain programming by FRP libraries like
19+`Yampa` and `reflex`, and array functionality by `bytestring`, `text`,
20+`vector`, and `arrays` packages.
21+
22+For basic programming needs, one needs to discover and learn all of
23+these or roll something on their own. These libraries have evolved
24+independently for specialized needs. They are not designed with a big
25+picture in mind to eliminate duplicate functionality and to come up with
26+the most optimal abstractions on the whole. For example, there is no
27+reason why streaming, list-t, and logict cannot be provided by the same
28+implementation. Concurrency and reactive programming also naturally fit
29+with the streaming model and can be unified. `Bytestring`, `text`,
30+`vector`, and `arrays` all provide the same functionality which can
31+be unified under one umbrella of arrays. Moreover, there are many
32+of flavors some of these packages like lazy bytestring, Char8 bytestrings,
33+lazy text which can be eliminated by using streams instead.
34+
35+On the performance side, existing streaming libraries lack stream fusion
36+leading to a function call overhead in each iteration of the loop,
37+therefore, providing orders of magnitude lower performance in tight
38+loops compared to writing a monolithic loop by hand. Similarly, `logict`
39+shows quadratic performance characteristics in some basic use cases.
40+These problems are solved by Streamly.
41+
42+## How Streamly Unifies them?
43+
44+Streamly unifies streaming, nested looping, scheduling, concurrency,
45+time, and array functionality using two basic constructs, namely streams
46+and arrays. Streams provide efficient, immutable, composable serial
47+processing capabilities for in-flight data, whereas arrays provide
48+efficient storage, mutability, and random access for data at rest.
49+
50+Streamly builds all the functionality on top of these two building
51+blocks. It takes a big picture view of programming in general and
52+provides well-integrated APIs with the same look and feel. Moreover,
53+performance is a primary goal of streamly, all the functionality is
54+built for performance comparable to C.
55+
56+Streamly streams are like lists and provide non-determinism just like
57+lists. Streamly also adds asynchronicity and concurrency to
58+streaming composition. This seemingly simple change unifies several
59+disparate abstractions into one powerful, concise, and elegant
60+abstraction. A wide variety of programming problems can be solved
61+elegantly with this abstraction. In particular, it unifies three major
62+programming domains namely non-deterministic (logic) programming,
63+concurrent programming, and functional Reactive programming.
64+
65+## Streaming
66+
67+The basic, bare-bones functionality of Streamly is processing streams of
68+data. In simple terms, stream processing is the functional equivalent
69+of loops in imperative programming.
70+
71+Like Haskell lists, `vector`, and `streaming` packages, Streamly composes
72+streams of data rather than stream processor functions as in other
73+streaming libraries like `pipes` and `machines`. This makes the types
74+and API very simple and is very similar to Haskell lists which is
75+familiar to everyone. The fundamental difference is that Streamly adds
76+concurrency support but the good thing is that it does not change the
77+API.
78+
79+This simple console echo program shows the simplicity of Streamly API
80+and its similarity with the list API:
81+
82+```haskell
83+echo =
84+ Stream.repeatM getLine
85+ & Stream.mapM putStrLn
86+ & Stream.drain
87+```
88+
89+Streamly uses dual representation for streams. On top it uses Scott
90+encoded CPS streams which provide a way to incrementally construct and
91+append streams efficiently. Under the good, for tight loops, Streamly
92+uses a vector-like stream representation which is amenable to
93+`case-of-case` and `SPEC constructor` optimizations by GHC resulting in
94+low-level code having performance comparable to C.
95+
96+To further understand the similarity with list API, please see [Streamly vs.
97+lists](https://github.com/composewell/streamly/blob/master/docs/streamly-vs-lists.md).
98+For comparison of Streamly performance with other libraries see
99+[streaming benchmarks](https://github.com/composewell/streaming-benchmarks).
100+
101+## Non-determinism
102+
103+Roughly speaking, non-determinism is a fancy term that functional
104+programmers use for what you call nested loops in imperative
105+programming. List transformers are the basic implementations for
106+non-determinism.
107+
108+The stream monad in Streamly is a list-transformer with behavior similar
109+to the list monad. It provides the functionality provided by `list-t` or
110+`logict` packages for free. Here is an example of nested looping using
111+the serial stream monad:
112+
113+``` haskell
114+import qualified Streamly.Prelude as S
115+
116+loops = do
117+ x <- Stream.fromFoldable [1,2]
118+ y <- Stream.fromFoldable [3,4]
119+ Stream.fromEffect $ putStrLn $ show (x, y)
120+```
121+
122+Moreover, the list transformer in Streamly can be concurrent. The Scott
123+encoding of streams also avoids the quadratic performance issue of
124+`logict`.
125+
126+## Scheduling Behaviors
127+
128+When we execute a stream or combine two streams, the actions in the
129+stream need to be scheduled for execution. Existing streaming libraries
130+are limited to just one form of scheduling of actions in the stream
131+which is serial execution. Streamly provides several ways of scheduling
132+the actions in the stream. The good thing is the API does not change, we
133+just need to use a combinator or a different type to change the
134+scheduling behavior.
135+
136+In a single stream, the actions in the stream can be executed
137+concurrently with different concurrent scheduling behaviors. Two or
138+more streams can be combined in several ways. For example, the actions
139+from two streams being combined can be interleaved. Similarly, streams
140+can be interleaved with concurrent execution providing a fair,
141+concurrent round-robin scheduling of streams.
142+
143+The monad instance provides a convenient way to combine streams in
144+different ways. It is just non-determinism with different flavors of
145+scheduling behavior. The example from the previous section can be run
146+with interleaved scheduling behavior as follows, without changing the
147+code at all:
148+
149+```haskell
150+main = Stream.drain $ Stream.fromWserial loops
151+```
152+
153+Scheduling is fundamental to expressing many common programming problems
154+in an idiomatic functional manner. One example application of
155+interleaving is breadth-first search mechanism for logic programming.
156+Please see [mini kanren implementation using streamly](https://github.com/composewell/ds-kanren).
157+
158+## Declarative Concurrency
159+
160+The same combinators that are used for serial streams e.g. 'unfoldrM',
161+'replicateM', 'repeatM' work concurrently when used at the appropriate type.
162+It allows concurrent programs to be written declaratively and composed
163+idiomatically. They are not much different than serial programs. See
164+[Streamly vs async](https://github.com/composewell/streamly/blob/master/docs/streamly-vs-async.md)
165+for a comparison of Streamly with the `async` package.
166+
167+Streamly provides concurrent scheduling and looping similar to to
168+[OpenMP](https://en.wikipedia.org/wiki/OpenMP) and
169+[Cilk](https://en.wikipedia.org/wiki/Cilk) but with a more declarative
170+style. The list transformer example can be run with concurrent
171+execution of loop iterations as follows, without changing the code at
172+all:
173+
174+```haskell
175+main = Stream.drain $ Stream.fromAhead loops
176+```
177+
178+And interleaving with concurrent execution of the loop iterations can be
179+written like this:
180+
181+```haskell
182+main = Stream.drain $ Stream.fromWAsync loops
183+```
184+
185+All this comes with no change in the streaming APIs.
186+
187+## Reactive Programming
188+
189+The combination of non-determinism, concurrency, and streaming makes
190+Streamly a strong reactive programming library as well. Reactive
191+programming fundamentally deals with streams of events that can be
192+processed concurrently. The
193+[https://github.com/composewell/streamly-examples/tree/master/AcidRain.hs](Acid
194+Rain) and [Circling
195+Square](https://github.com/composewell/streamly-examples/tree/master/CirclingSquare.hs)
196+examples demonstrate the basic reactive capability of Streamly.
197+
198+In core concepts, Streamly is strikingly similar to `dunai`. `dunai` was
199+designed from a FRP perspective and Streamly was originally designed
200+from a concurrency perspective. However, both have similarities at the
201+core.
202+
203+## Arrays
204+
205+Streamly provides immutable, mutable, pinned, unpinned, boxed, and
206+unboxed arrays with streaming interfaces. The combination of efficient
207+streaming and polymorphic arrays lets it express the functionality of
208+`bytestring` and `text` packages as special cases of arrays with no loss
209+of performance. Since we can use explicit streaming, the lazy versions
210+of `bytestring` and `text` are not required.
211+
212+## Conclusion
213+
214+Streamly, provides effectful streams, with a simple API, almost
215+identical to standard lists, and in-built support for concurrency.
216+By using stream-style combinators on stream composition, streams can be
217+generated, merged, chained, mapped, zipped, and consumed concurrently –
218+providing a generalized high-level programming framework unifying
219+streaming and concurrency. Controlled concurrency allows even infinite
220+streams to be evaluated concurrently. Concurrency is auto-scaled based
221+on feedback from the stream consumer.
222+
223+Streamly is a programmer-first library, designed to be useful and
224+friendly to programmers for solving practical problems in a simple and
225+concise manner. Some key points that Streamly stresses are:
226+
227+* _Simplicity_: Simple list like streaming API, if you know how to use lists
228+ then you know how to use Streamly. This library is built with simplicity
229+ and ease of use as a design goal.
230+* _Concurrency_: Simple, powerful, and scalable concurrency. Concurrency is
231+ built-in, and not intrusive, concurrent programs are written exactly
232+ as non-concurrent ones.
233+* _Generality_: Unifies functionality provided by several disparate packages
234+ (streaming, concurrency, list transformer, logic programming, reactive
235+ programming) in a concise API.
236+* _Performance_: Streamly is designed for high performance. It employs stream
237+ fusion optimizations for the best possible performance. Serial performance is
238+ equivalent to the venerable `vector` library in most cases and even better
239+ in some cases. Concurrent performance is unbeatable. See
240+ [streaming-benchmarks](https://github.com/composewell/streaming-benchmarks)
241+ for a comparison of popular streaming libraries on micro-benchmarks.
242+
243+## References
244+
245+The basic streaming functionality of Streamly is equivalent to that
246+provided by streaming libraries like
247+[vector](https://hackage.haskell.org/package/vector),
248+[streaming](https://hackage.haskell.org/package/streaming),
249+[pipes](https://hackage.haskell.org/package/pipes), and
250+[conduit](https://hackage.haskell.org/package/conduit).
251+In addition to providing streaming functionality, Streamly subsumes
252+the functionality of list transformer libraries like `pipes` or
253+[list-t](https://hackage.haskell.org/package/list-t), and also the logic
254+programming library [logict](https://hackage.haskell.org/package/logict). On
255+the concurrency side, it subsumes the functionality of the
256+[async](https://hackage.haskell.org/package/async) package and provides an even
257+higher level concurrent composition. Because it supports
258+streaming with concurrency we can write FRP applications similar in concept to
259+[Yampa](https://hackage.haskell.org/package/Yampa) or
260+[reflex](https://hackage.haskell.org/package/reflex).