@@ -0,0 +1,178 @@
1+Conduits are an approach to the streaming data problem. It is meant as an
2+alternative to enumerators/iterators, hoping to address the same issues with
3+different trade-offs based on real-world experience with enumerators.
4+
5+General Goal
6+===========================
7+
8+Let's start by defining the goal of enumerators, iterators, and conduits. We
9+want a standard interface to represent streaming data from one point to
10+another, possibly modifying the data along the way.
11+
12+This goal is also achieved by lazy I/O; the problem with lazy I/O, however, is
13+that of deterministic resource cleanup. That is to say, with lazy I/O, you
14+cannot be guaranteed that your file handles will be closed as soon as you have
15+finished reading data from them.
16+
17+We want to keep the same properties of constant memory usage from lazy I/O, yet
18+have guarantees that scarce resources will be freed as early as possible.
19+
20+Enumerator
21+===========================
22+
23+__Note__: This is biased towards John Millikin's enumerator package, as that is
24+the package with which I have the most familiarity.
25+
26+The concept of an enumerator is fairly simple. We have an `Iteratee` which
27+"consumes" data. It keeps its state while being fed data by an `Enumerator`.
28+The `Enumerator` will feed data a few chunks at a time to an `Iteratee`,
29+transforming the `Iteratee`'s state at each call. Additionally, there is an
30+`Enumeratee` that acts as both an `Enumerator` and `Iteratee`.
31+
32+As a result, there are a few changes to code structure that need to take place
33+in order to fully leverage enumerators:
34+
35+* The `Enumerator`s control code flow. This is an Inversion of Control (IoC)
36+ technique.
37+
38+ __Practical ramification__: `Iteratee` code can be more difficult to
39+ structure.
40+
41+* `Iteratee`s are not able to allocate scarce resources. Since they do not
42+ have any control of the flow of the program, they cannot guarantee that
43+ the resources will be released, especially in the presence of exceptions.
44+
45+ __Practical ramification__: There is no way to create an `iterFile`, which
46+ will stream data into a file. Instead, you must allocate a file handle
47+ before entering the `Iteratee` and pass that in. In some cases, such an
48+ approach would mean file handles are kept open too long.
49+
50+* None of this plays nicely with monad transformers, though this does not
51+ seem to be an inherent problem with enumerators, instead with the current
52+ library.
53+
54+ __Practical ramification__: You cannot enumerate a file when running in a
55+ `ReaderT IO`.
56+
57+* Instead of passing around a `Handle` to pull data from, your code should
58+ live inside an `Iteratee`. This makes it difficult and/or impossible to
59+ interleave two different sources.
60+
61+ __Practical ramification__: Even with libraries designed to interoperate
62+ (like http-enumerator and warp), it is not possible to create a proper
63+ streaming HTTP proxy.
64+
65+* While the concepts are simple, actually writing low-level Iteratee code is
66+ very complex. This in turn intimidates users from adopting the approach.
67+
68+Conduits
69+===========================
70+
71+Conduits attempt to provide a similar high-level API to enumerators, while
72+providing a drastically different low-level implementation. The first question
73+to visit is: why does the enumerator need to control flow of the program? The
74+main purpose is to ensure that resources are released properly. But this in
75+fact solved only *half* the problem; iteratees still cannot release resources.
76+
77+ResourceT
78+---------------------------
79+
80+So our first issue to address is to create a new way to deal with resource
81+allocation. We represent this as a monad transformer, `ResourceT`. It works as
82+follows:
83+
84+* You can register a cleanup action, which will return a `ReleaseKey`.
85+
86+* If you pass your `ReleaseKey` to the `release` function, your action will be
87+ called automatically, and your action will be unregistered.
88+
89+* When the monad is exited (via `runRelease`), all remaining registered actions
90+ will be called.
91+
92+* All of this is provided in an exception-safe manner.
93+
94+For example, you would be able to open a file handle, and then register an
95+action to close the file handle. In your code, you would call `release` on your
96+`ReleaseKey` as soon as you reach the end of the contents you are streaming. If
97+that code is never reached, the file handle will be released when the monad
98+terminates.
99+
100+Source
101+---------------------------
102+
103+Now that we have a way to deal with resources, we can take a radically
104+different approach to production of data streams. Instead of a push system,
105+where the enumerators sends data down the pipeline, we have a pull system,
106+where data is requested from the source. Additionally, a source allows
107+buffering of input data, so data can be "pushed back" onto the source to be
108+available for a later call.
109+
110+Sink
111+---------------------------
112+
113+A `Sink` is the corrollary to an `Iteratee`. It takes a stream of data, and can
114+return a result, consisting of leftover input and an output. Like an
115+`Iteratee`, a `Sink` provides a `Monad` instance, which allows easy chaining
116+together of `Sink`s.
117+
118+However, a big difference is that your code needn't live in the `Sink` monad.
119+You can easily pass around your sources and connect them to different `Sink`s.
120+As a practical example, when the Web Application Interface (WAI) is translated
121+to conduits, the application lives in the `ResourceT IO` monad, and the
122+`Request` value contains a `requestBody` record, which is a `Source IO
123+ByteString`.
124+
125+Conduit
126+---------------------------
127+
128+Conduits are simply functions that that a stream of input data and return
129+leftover input as well as a stream of output data. Conduits are far simpler to
130+implement than their corrollary, `Enumeratee`s.
131+
132+Connecting
133+---------------------------
134+
135+While you can directly pull data from a `Source`, or directly push to a `Sink`, the easiest approach is to use the built-in connect operators. These follow the naming convention from the enumerator package, e.g.:
136+
137+ sourceFile "myfile.txt" $$ sinkFile "mycopy.txt"
138+ sourceFile "myfile.txt" $= uppercase {- a conduit -} $$ sinkFile "mycopy.txt"
139+ fromList [1..10] $$ Data.Conduit.List.map (+ 1) =$ fold (+) 0
140+
141+Trade-offs
142+===========================
143+
144+Overall, the approach achieves the goals I had hoped for. The main downside in
145+its current form is its reliance on mutable data. Instead of having an
146+`Iteratee` return a new `Iteratee`, thereby provide an illusion of mutability,
147+in conduit the sources and sinks must maintain their state internally. As a
148+result, code must live in IO and usually use something like an IORef to keep
149+track of the current state.
150+
151+I believe this to be an acceptable trade-off, since:
152+
153+1. Virtually all conduit code will be performing I/O, so staying in the `IO`
154+ monad is reasonable.
155+2. By using `monad-control`, conduit can work with any monad *based* on `IO`,
156+ meaning all standard transformers (except `ContT`) can be used.
157+3. Enumerator experience has shown that the majority of the time, you construct
158+ `Iteratee`s by using built-in functions, such as fold and map. Therefore,
159+ the complication of tracking mutable state will usually be abstracted from
160+ users.
161+
162+Another minor point is that, in order to provide an efficient `Monad` instance,
163+the `Sink` type is complicated with tracking two cases: a `Sink` which expects
164+data and one which does not. As expressed in point (3) above, this should not
165+have a major impact for users.
166+
167+Finally, since most `Source`s and `Sink`s begin their life by allocating some
168+mutable variable, both types allow some arbitrary monadic action to be run
169+before actual processing begins. The monad (et al) instances and connect
170+functions are all built to run this action once and then continue operation.
171+
172+Status
173+===========================
174+
175+This is currently no more than a proof-of-concept, to see the differences
176+between enumerators and conduits for practical problems. This may serve as a
177+basis for WAI and Yesod in the future, but that will only be after careful
178+vetting of the idea. Your input is greatly appreciated!