@@ -0,0 +1,117 @@
1+Streaming Paradigms
2+-------------------
3+
4+There are two dual paradigms for stream processing in Haskell. In the first
5+paradigm we represent a stream as a data type and use functions to work on it.
6+In the second paradigm we represent *stream processors* as data types and
7+provide them individual data elements to process, there is no explicit
8+representation of the stream as a data type. In the first paradigm we work with
9+data representation and in the second paradigm we work with function
10+representations. Both of these paradigms have equal expressive power. The
11+latter uses the monadic composition for data flow whereas the former does not
12+need monadic composition for straight line stream processing and therefore can
13+use it for higher level composition e.g. to compose streams in a product
14+style.
15+
16+To see an example of the first paradigm, let us use the ``vector`` package to
17+represent a monadic stream of integers as ``Stream IO Int``. This data
18+representation of stream is passed explicitly to the stream processing
19+functions like ``filter`` and ``drop`` to manipulate it::
20+
21+ import qualified Data.Vector.Fusion.Stream.Monadic as S
22+
23+ stream :: S.Stream IO Int
24+ stream = S.fromList [1..100]
25+
26+ main = do
27+ let str = (S.filter even . S.drop 10) stream
28+ toList str >>= putStrLn . show
29+
30+Pure lists and vectors are the most basic examples of streams in this paradigm.
31+The streaming IO libraries just extend the same paradigm to monadic streaming.
32+The API of these libraries is very much similar to lists with a monad parameter
33+added.
34+
35+The second paradigm is direct opposite of the first one, there is no stream
36+representation in this paradigm, instead we represent *stream processors* as
37+data types. A stream processor represents a particular process rather than
38+data, and we compose them together to create composite processors. We can call
39+them stream transducers or simply pipes. Using the ``machines`` package::
40+
41+ import qualified Data.Machine as S
42+
43+ producer :: S.SourceT IO Int
44+ producer = S.enumerateFromTo 1 100
45+
46+ main = do
47+ let processor = producer S.~> S.dropping 10 S.~> S.filtered even
48+ S.runT processor >>= putStrLn . show
49+
50+Both of these paradigms look almost the same, right? To see the difference
51+let's take a look at some types. In the first paradigm we have an explicit
52+stream type and the processing functions take the stream as input and produce
53+the transformed stream::
54+
55+ stream :: S.Stream IO Int
56+ filter :: Monad m => (a -> Bool) -> Stream m a -> Stream m a
57+
58+In the second paradigm, there is no stream data type, there are stream
59+processors, let's call them boxes that represent a process. We have a
60+*SourceT* box that represents a singled ended producer and a *Process* box or a
61+pipe that has two ends, an input end and an output end, a ``MachineT``
62+represents any kind of box. We put these boxes together using the ``~>``
63+operator and then run the resulting machine using ``runT``::
64+
65+ producer :: S.SourceT IO Int
66+ filtered :: (a -> Bool) -> Process a a
67+ dropping :: Int -> Process a a
68+ (~>) :: Monad m => MachineT m k b -> ProcessT m b c -> MachineT m k c
69+
70+Custom pipes can be created using a Monadic composition and primitives to
71+receive and send data usually called ``await`` and ``yield``.
72+
73+.. |str| replace:: `streamly <https://github.com/composewell/streamly>`__
74+
75++-----------------------------------------------------------------------------+
76+| Streaming libraries using the direct paradigm. |
77++------------------------+----------------------------------------------------+
78+| Library | Remarks |
79++========================+====================================================+
80+| vector | The simplest in this category, provides |
81+| | transformation and combining of monadic |
82+| | streams but no monadic composition of streams. |
83+| | Provides a very simple list like API. |
84++------------------------+----------------------------------------------------+
85+| streaming | * Encodes a return value to be supplied when the |
86+| | stream ends. The monad instance passes on the |
87+| | streams and combines the return values. |
88+| | * Functor general |
89+| | * The API is more complicated than vector because |
90+| | of the return value and the functor layer. |
91++------------------------+----------------------------------------------------+
92+| list-t | Provides straight line composition of streams |
93+| | as well as a list like monadic composition. |
94+| | The API is simple, just like ``vector``. |
95++------------------------+----------------------------------------------------+
96+| | Like list-t, in addition to straight line |
97+| | composition it provides a list like monadic |
98+| | composition of streams, supports combining streams |
99+| | concurrently supports concurrent applicative and |
100+| | monadic composition. |
101+| |str| | The basic API is very much like lists and |
102+| | almost identical to ``vector`` streams. |
103++------------------------+----------------------------------------------------+
104+
105++-----------------------------------------------------------------------------+
106+| Streaming libraries using the pipes paradigm. |
107++------------------------+----------------------------------------------------+
108+| Library | Remarks |
109++========================+====================================================+
110+| conduit | ``await`` and ``yield`` data to upstream or |
111+| | downstream pipes; supports pushing leftovers back. |
112++------------------------+----------------------------------------------------+
113+| pipes | ``await`` and ``yield`` data to upstream or |
114+| | downstream pipes |
115++------------------------+----------------------------------------------------+
116+| machines | Can await from two sources, left and right. |
117++------------------------+----------------------------------------------------+