@@ -1,178 +0,0 @@
1−Conduits are an approach to the streaming data problem. It is meant as an
2−alternative to enumerators/iterators, hoping to address the same issues with
3−different trade-offs based on real-world experience with enumerators.
4−
5−General Goal
6−===========================
7−
8−Let's start by defining the goal of enumerators, iterators, and conduits. We
9−want a standard interface to represent streaming data from one point to
10−another, possibly modifying the data along the way.
11−
12−This goal is also achieved by lazy I/O; the problem with lazy I/O, however, is
13−that of deterministic resource cleanup. That is to say, with lazy I/O, you
14−cannot be guaranteed that your file handles will be closed as soon as you have
15−finished reading data from them.
16−
17−We want to keep the same properties of constant memory usage from lazy I/O, yet
18−have guarantees that scarce resources will be freed as early as possible.
19−
20−Enumerator
21−===========================
22−
23−__Note__: This is biased towards John Millikin's enumerator package, as that is
24−the package with which I have the most familiarity.
25−
26−The concept of an enumerator is fairly simple. We have an `Iteratee` which
27−"consumes" data. It keeps its state while being fed data by an `Enumerator`.
28−The `Enumerator` will feed data a few chunks at a time to an `Iteratee`,
29−transforming the `Iteratee`'s state at each call. Additionally, there is an
30−`Enumeratee` that acts as both an `Enumerator` and `Iteratee`.
31−
32−As a result, there are a few changes to code structure that need to take place
33−in order to fully leverage enumerators:
34−
35−* The `Enumerator`s control code flow. This is an Inversion of Control (IoC)
36− technique.
37−
38− __Practical ramification__: `Iteratee` code can be more difficult to
39− structure.
40−
41−* `Iteratee`s are not able to allocate scarce resources. Since they do not
42− have any control of the flow of the program, they cannot guarantee that
43− the resources will be released, especially in the presence of exceptions.
44−
45− __Practical ramification__: There is no way to create an `iterFile`, which
46− will stream data into a file. Instead, you must allocate a file handle
47− before entering the `Iteratee` and pass that in. In some cases, such an
48− approach would mean file handles are kept open too long.
49−
50−* None of this plays nicely with monad transformers, though this does not
51− seem to be an inherent problem with enumerators, instead with the current
52− library.
53−
54− __Practical ramification__: You cannot enumerate a file when running in a
55− `ReaderT IO`.
56−
57−* Instead of passing around a `Handle` to pull data from, your code should
58− live inside an `Iteratee`. This makes it difficult and/or impossible to
59− interleave two different sources.
60−
61− __Practical ramification__: Even with libraries designed to interoperate
62− (like http-enumerator and warp), it is not possible to create a proper
63− streaming HTTP proxy.
64−
65−* While the concepts are simple, actually writing low-level Iteratee code is
66− very complex. This in turn intimidates users from adopting the approach.
67−
68−Conduits
69−===========================
70−
71−Conduits attempt to provide a similar high-level API to enumerators, while
72−providing a drastically different low-level implementation. The first question
73−to visit is: why does the enumerator need to control flow of the program? The
74−main purpose is to ensure that resources are released properly. But this in
75−fact solved only *half* the problem; iteratees still cannot release resources.
76−
77−ResourceT
78−---------------------------
79−
80−So our first issue to address is to create a new way to deal with resource
81−allocation. We represent this as a monad transformer, `ResourceT`. It works as
82−follows:
83−
84−* You can register a cleanup action, which will return a `ReleaseKey`.
85−
86−* If you pass your `ReleaseKey` to the `release` function, your action will be
87− called automatically, and your action will be unregistered.
88−
89−* When the monad is exited (via `runRelease`), all remaining registered actions
90− will be called.
91−
92−* All of this is provided in an exception-safe manner.
93−
94−For example, you would be able to open a file handle, and then register an
95−action to close the file handle. In your code, you would call `release` on your
96−`ReleaseKey` as soon as you reach the end of the contents you are streaming. If
97−that code is never reached, the file handle will be released when the monad
98−terminates.
99−
100−Source
101−---------------------------
102−
103−Now that we have a way to deal with resources, we can take a radically
104−different approach to production of data streams. Instead of a push system,
105−where the enumerators sends data down the pipeline, we have a pull system,
106−where data is requested from the source. Additionally, a source allows
107−buffering of input data, so data can be "pushed back" onto the source to be
108−available for a later call.
109−
110−Sink
111−---------------------------
112−
113−A `Sink` is the corrollary to an `Iteratee`. It takes a stream of data, and can
114−return a result, consisting of leftover input and an output. Like an
115−`Iteratee`, a `Sink` provides a `Monad` instance, which allows easy chaining
116−together of `Sink`s.
117−
118−However, a big difference is that your code needn't live in the `Sink` monad.
119−You can easily pass around your sources and connect them to different `Sink`s.
120−As a practical example, when the Web Application Interface (WAI) is translated
121−to conduits, the application lives in the `ResourceT IO` monad, and the
122−`Request` value contains a `requestBody` record, which is a `Source IO
123−ByteString`.
124−
125−Conduit
126−---------------------------
127−
128−Conduits are simply functions that that a stream of input data and return
129−leftover input as well as a stream of output data. Conduits are far simpler to
130−implement than their corrollary, `Enumeratee`s.
131−
132−Connecting
133−---------------------------
134−
135−While you can directly pull data from a `Source`, or directly push to a `Sink`, the easiest approach is to use the built-in connect operators. These follow the naming convention from the enumerator package, e.g.:
136−
137− sourceFile "myfile.txt" $$ sinkFile "mycopy.txt"
138− sourceFile "myfile.txt" $= uppercase {- a conduit -} $$ sinkFile "mycopy.txt"
139− fromList [1..10] $$ Data.Conduit.List.map (+ 1) =$ fold (+) 0
140−
141−Trade-offs
142−===========================
143−
144−Overall, the approach achieves the goals I had hoped for. The main downside in
145−its current form is its reliance on mutable data. Instead of having an
146−`Iteratee` return a new `Iteratee`, thereby provide an illusion of mutability,
147−in conduit the sources and sinks must maintain their state internally. As a
148−result, code must live in IO and usually use something like an IORef to keep
149−track of the current state.
150−
151−I believe this to be an acceptable trade-off, since:
152−
153−1. Virtually all conduit code will be performing I/O, so staying in the `IO`
154− monad is reasonable.
155−2. By using `monad-control`, conduit can work with any monad *based* on `IO`,
156− meaning all standard transformers (except `ContT`) can be used.
157−3. Enumerator experience has shown that the majority of the time, you construct
158− `Iteratee`s by using built-in functions, such as fold and map. Therefore,
159− the complication of tracking mutable state will usually be abstracted from
160− users.
161−
162−Another minor point is that, in order to provide an efficient `Monad` instance,
163−the `Sink` type is complicated with tracking two cases: a `Sink` which expects
164−data and one which does not. As expressed in point (3) above, this should not
165−have a major impact for users.
166−
167−Finally, since most `Source`s and `Sink`s begin their life by allocating some
168−mutable variable, both types allow some arbitrary monadic action to be run
169−before actual processing begins. The monad (et al) instances and connect
170−functions are all built to run this action once and then continue operation.
171−
172−Status
173−===========================
174−
175−This is currently no more than a proof-of-concept, to see the differences
176−between enumerators and conduits for practical problems. This may serve as a
177−basis for WAI and Yesod in the future, but that will only be after careful
178−vetting of the idea. Your input is greatly appreciated!