@@ -0,0 +1,471 @@
1++++
2+title = "Input and Output"
3+weight = 37
4+template = "doc.html"
5++++
6+
7+Let's talk about IO in Urbit. I won't say much about how events and effects are
8+processed by the runtime; it suffices to note that the runtime is a normal Unix
9+program which (1) listens for various events, (2) passes them to Arvo (Urbit
10+OS), and (3) processes the effects produced by Arvo. Instead, I’ll focus on how
11+IO works from the perspective of Hoon programs, like Arvo or apps running on
12+Arvo. But first, let's review IO paradigms in other languages.
13+
14+All examples are in a made-up language, since it's useful to understand this
15+information even before you're comfortable in Hoon. If you're learning Hoon and
16+want a challenge, try translating them!
17+
18+## The three styles of IO
19+
20+I will call these styles imperative IO, monadic IO, and state machines (note
21+that this classification refers only to the experience of using them and the
22+program structure they dictate). An imperative programming language can use any
23+of these and a functional programming language can use any of them.
24+
25+### Imperative IO
26+
27+Imperative languages (such as C and Python) perform IO by calling a function in
28+the middle of whatever they were doing. Anywhere, in any statement, you could
29+have IO.
30+
31+In this example and others to follow, the task will be to read a filename from
32+the command line, read a URL out of that file, and fetch the contents at that
33+URL.
34+
35+ output = fetchUrl(stripWhitespace(readFile(cliInput())))
36+
37+For readability, intermediate variables would be a good idea, but nothing stops
38+you from nesting IO this way.
39+
40+### Monadic IO
41+
42+Languages that use explicit monads for IO (such as Haskell) perform IO by
43+running IO behind a bind operation, which I'll notate with `<-`. Thus, while you
44+can't put IO *anywhere*, the general structure is a sequence of IO operations.
45+
46+ fileName <- cliInput()
47+ url <- readFile(fileName)
48+ output <- fetchUrl(stripWhitespace(url))
49+
50+[Tackling the Awkward
51+Squad](https://www.microsoft.com/en-us/research/wp-content/uploads/2016/07/mark.pdf)
52+is the canonical description of monadic IO, and does a great job of explaining
53+how the Haskell community decided to use this as their primary IO style.
54+
55+### State machines
56+
57+Languages that use explicit state machines for IO (such as Elm) perform IO by
58+accepting events and producing a list of effects.
59+
60+ event :: Initialize
61+ | CLIInput text
62+ | FileInput text
63+ | HTTPResponse body
64+ effect :: CLIInput
65+ | ReadFile fileName
66+ | FetchUrl url
67+ | Print text
68+ main(event) {
69+ switch event {
70+ Initialize -> [CLIInput ~]
71+ CLIinput fileName -> [ReadFile fileName]
72+ FileInput url -> [FetchUrl stripWhitespace(url)]
73+ HTTPResponse body -> [Print body]
74+ }
75+ }
76+
77+The effects produced are not functions but data.
78+
79+It's important to note that this example is chosen to highlight a particular
80+kind of IO — sequences of events. Further, this example has no persistent state
81+— each IO operation is a pure function of the output of the previous one.
82+
83+## A stateful, loopy example
84+
85+Let's look at another type of program: a simple HTTP server that you can give
86+text at the CLI, which will respond to all HTTP requests with that text.
87+
88+Imperative:
89+
90+ text = "no text entered yet"
91+ while (true)
92+ cli = readCli()
93+ if (cli != Null)
94+ text = cli
95+ request = readHttp()
96+ if (request != Null)
97+ respond(request,text)
98+
99+Monadic:
100+
101+ loop("no text entered yet")
102+
103+ where:
104+
105+ loop :: text -> IO Null
106+ loop =
107+ input <- readInput()
108+ switch input
109+ CLIInput newText -> loop(newText)
110+ HTTPRequest request ->
111+ _ <- respond(request,text)
112+ loop(text)
113+
114+State machine:
115+
116+ text = "no text entered yet"
117+ main(text,event) {
118+ switch event {
119+ CLIInput newText -> [newText, Null]
120+ HTTPRequest request -> [text, HTTPResponse text]
121+ }
122+ }
123+
124+Again, note the response in the state machine is data, not a function.
125+
126+I'm not trying to make a point about which type is better in general. There's a
127+pretty strong argument that in the first example, imperative was the best
128+option, followed by monadic, followed by state machine. There's an equally
129+strong argument that the second example is most naturally represented by the
130+state machine.
131+
132+Look at how state is handled in the monadic example. Formally, bind composes
133+functions, so that every time you see a `<-` there's actually a new function
134+(closure/delimited continuation) being created and stored for when we receive a
135+response. All variables in scope are stored in that closure, so we can
136+conveniently access them later without explicitly storing them. However, the
137+state we preserve over time varies a lot. In this example, it varies between
138+storing `loop`, `[loop input]`, `[loop input newText]`, and `[loop input
139+request]`.
140+
141+By comparison, the state machine does not store intermediate values, it only
142+stores its explicit state. While this is somewhat less flexible, it puts the
143+permanent state front-and-center.
144+
145+I'll reiterate that all of these are possible to implement in purely functional
146+languages, and further they can all be implemented in terms of each other. State
147+machines are trivially implementable -- it's just a function [state event] ->
148+[new-state (list effect)]. Monadic IO can be implemented directly (as in
149+Haskell) or on top of state machines by storing continuations in your state.
150+Imperative IO is somewhat more difficult but should be possible through
151+algebraic effects.
152+
153+When inventing a new system, you should consider which of these you intend to
154+support based on the sort of code you expect to see. For example, a scripting
155+language will almost always benefit from imperative (or at least monadic) style.
156+
157+## In Urbit
158+
159+So much for exploring the space of IO solutions. In terms of deployed code on
160+Earth, the split is about 90% imperative, 5% monadic, and 5% state machine. In
161+Urbit, the split is about 90% state machine and 10% monadic. This is what I
162+intend to explain and defend.
163+
164+Urbit's OS is called Arvo; it has several "kernel modules" called vanes, and it
165+has userspace applications. Colloquially these are called "apps," but when
166+speaking about them abstractly, it's helpful to use their more specific name:
167+["agents"](https://urbit.org/docs/tutorials/arvo/gall/#agents). Arvo, the vanes,
168+and agents are all structured as explicit state machines.
169+
170+In addition, there are
171+["threads"](https://urbit.org/docs/tutorials/arvo/gall/#threads), which are
172+structured as monadic IO. Note there's only a superficial resemblance to Unix
173+threads: they are not executed in parallel and they don't share memory, or at
174+least not any more than agents do. They're simply a computation that takes an
175+argument and produces a result, possibly after doing some IO, structured
176+monadically.
177+
178+As you can see, all the lowest levels of Arvo are explicit state machines, and
179+only the highest layer supports monadic IO. To see why this is the case, let's
180+list advantages and disadvantages to each type. Note that, while I'm speaking
181+specifically of agents and threads in the context of Urbit, these are
182+fundamental properties of monadic and state machine IO.
183+
184+Agents are:
185+
186+- Upgradable. Since the state is explicit, you can upgrade an agent in-place by
187+ supplying a new state machine and a function from the old state type to the
188+ new state type. This is very similar to a database migration.
189+
190+- Robust. A state machine must accept any input at any time. If it receives
191+ unexpected input, it must have had explicit error handling for that input,
192+ since it's one of the types in the switch statement. Thus, it's relatively
193+ easy to stay in a consistent state.
194+
195+- Permanent. Since it's upgradable and robust, there's no reason to stop its
196+ running.
197+
198+- Hard to write long sequences of IO. Long sequences of IO create many possible
199+ states, and you must handle any input in any state. This can result in
200+ complex, hard-to-follow code with code paths that are rarely if ever taken.
201+
202+Threads are:
203+
204+- Not upgradable in-place. Since the state is implicit, you can only cleanly
205+ upgrade by (1) killing the thread and restarting it or (2) letting it resolve
206+ to a defined "upgradable state" periodically, usually at the top of a "main
207+ loop". If you do (2), then you are in fact structuring it as a state machine,
208+ so the other advantages of threads are compromised.
209+
210+- Fragile. If we don't receive expected input, the default behavior is to be
211+ stuck waiting for input. You must remember to add any error-handling features
212+ you want, and it's easy to end up in an inconsistent state.
213+
214+- Impermanent. Since a thread can't be upgraded and may get stuck in case of
215+ unexpected input, a thread can't be expected to last forever. It must be
216+ "rebooted" from time to time.
217+
218+- Easy to write long sequences of IO. Just...write the IO one line at a time.
219+
220+Arvo proper and the vanes are permanent pieces of software and, for this reason,
221+state machines are more natural.
222+
223+I'll note that other kernels, like the Unix kernel, use imperative IO. The
224+difference is that Unix isn't permanent in itself — it only lasts until you
225+reboot. Since your permanent state is stored externally, you don't need to be
226+upgradable or permanent. With sufficient discipline you can write robust C code,
227+so it makes sense to use imperative style. Even so, a lot of C code is written
228+in a state machine style; even if it doesn't need to be upgradable, it's worth
229+it for the robustness and concurrency advantages.
230+
231+However, Arvo is a single-level store — all of its state is permanent. This is
232+very convenient and eliminates many long-tail bugs, but it also means we need to
233+code for permanence, and that's much easier when you structure your code with
234+explicit state.
235+
236+Further, Arvo and the vanes rarely perform long sequences of IO. Generally, they
237+do one thing and emit some effects, similar to the HTTP server example above.
238+Sometimes they "pass through" a request. For example, the Eyre vane passes HTTP
239+requests to userspace agents or threads. In this case, there is a sequence of
240+IO, which in monadic code would be:
241+
242+ loop()
243+
244+ where:
245+
246+ loop =
247+ request <- receiveRequest()
248+ agent = lookupAgent(request)
249+ response <- callAgent(agent, request)
250+ _ <- sendResponse(request, response)
251+ loop()
252+
253+This is clear code to read, but it leaves open the question of how to respond
254+during the callAgent() function. If another request comes in, can we handle it
255+in parallel? If we need to upgrade ourselves before the agent responds, can we
256+make sure to properly handle the response?
257+
258+The naive solution in monadic code is to say that you can only handle one
259+request at a time, and if an upgrade happens you just drop outstanding requests.
260+You can resolve both of these by introducing a sort of main loop, which is just
261+a way of saying "turn it into a state machine".
262+
263+In a state machine, the naive solution is:
264+
265+ state :: (map request agent)
266+ main(state, event) {
267+ switch event {
268+ HTTPRequest request ->
269+ agent = lookupAgent(request)
270+ [put(state, request, agent), CallAgent agent request]
271+ AgentResponse request response ->
272+ [del(state, request), HTTPRespond request response]
273+ }
274+ }
275+
276+Upgrading this is trivial, since the state (map of requests to outstanding calls
277+to agents) is explicit. It also trivially handles concurrent requests.
278+
279+I won't claim this is easier or harder to write than the monadic version, but it
280+has the properties we need at this level of the system.
281+
282+Userspace agents similarly are permanent entities that need to not lose data or
283+get stuck. For this reason they’re structured as state machines.
284+
285+A helpful comparison is that explicit state machines are basically Mealy
286+machines, the kind you might have learned about in an introductory digital
287+design class. This shouldn't be surprising; a digital circuit must always be
288+consistent because it can't be manually "rebooted", and their input and output
289+is highly formalized. By contrast, a Rube Goldberg machine doesn't have the same
290+constraints; which is why they're a lot more fun!
291+
292+It's notable also that explicit state machines are considerably more declarative
293+than imperative or monadic IO. Not everyone thinks that's a good thing, but
294+anyone who defends functional programming should see some advantages to that.
295+
296+However, userspace sometimes needs to perform long sequences of IO. Let's look
297+at an example where we want to fetch the front page of a site and the first
298+couple comments on each story. HTTP is unreliable, so on failure we retry five
299+times.
300+
301+Monadic:
302+
303+ topStories <- fetch(topStoriesUrl)
304+ loop(topStories)
305+
306+ where:
307+
308+ loop :: topStories -> IO Null
309+ if topStories is Null
310+ return Null
311+ comments <- retryLoop(5,head(topStories))
312+ otherComments <- loop(tail(topStories))
313+ append(comments,otherComments)
314+
315+ retryLoop :: [n story] -> IO Comments
316+ comments <- fetchComments(story)
317+ if comments is HTTPError:
318+ if n == 0:
319+ bail
320+ else:
321+ retryLoop(n-1, story)
322+
323+State machine:
324+
325+ state :: Initial
326+ | FetchingTop
327+ | FetchingStory comments stories retries
328+ | Done comments
329+
330+ main(state, event) {
331+ switch event {
332+ Initialize ->
333+ assert state == Initial
334+ [FetchingTop, Fetch topStoriesUrl]
335+ HttpResponse response ->
336+ switch state {
337+ FetchingTop ->
338+ [FetchingStory Null stories 5, FetchComments head(stories)]
339+ FetchingStory comments stories retries ->
340+ if response is HTTPError:
341+ if retries == 0:
342+ [Initial, Null]
343+ else:
344+ [FetchingStory comments stories retries-1,
345+ FetchComments head(stories)]
346+ else:
347+ comments = append(response,comments)
348+ if tail(stories) is Null:
349+ [Done comments, Null]
350+ else:
351+ [FetchingStory append(comments,response) tail(stories) 5,
352+ FetchComments head(tail(stories))]
353+ }
354+ }
355+ }
356+
357+With practice, the state machine version of this can be written correctly.
358+However, (1) it's verbose, (2) the control flow jumps all over the place, and
359+(3) in practice it's challenging to get exactly right. This is still a small
360+example, and it gets much worse as the length and complexity of the IO sequence
361+grows.
362+
363+The monadic version, on the other hand, flows cleanly down the page. We can
364+factor out generic functionality like "retry this request n times" to make the
365+essence of the computation clear.
366+
367+Suppose you have to upgrade this. In the state machine, you can see exactly
368+which stage of the computation you're in and write specific code to upgrade
369+cleanly.
370+
371+Threads, as we've discussed, can't be upgraded in-place. Let's be real though:
372+we're downloading comment sections, not regulating a nuclear reactor. If we get
373+a better version of this script, just kill the old one and start the new one
374+from scratch. What's a couple extra HTTP requests?
375+
376+This suggests a general division of workloads: code which needs to be permanent,
377+upgradable, and concurrent should generally be a state machine. Code with long
378+or complex sequences of IO and definite termination should generally be a
379+thread.
380+
381+These aren't completely mutually exclusive, but it's surprising how often they
382+are.
383+
384+### Jael: a case study
385+
386+When you need to do something permanent and upgradable, but you also need to do
387+long sequences of IO, the main trick is to factor your problem into both an
388+agent and a thread (or several).
389+
390+As a case study, consider Jael, one of our vanes. This maintains our PKI state,
391+which it downloads over HTTP from an Ethereum node. Downloading from an Ethereum
392+node is a complex sequence that starts with fetching its most recent block
393+number, then fetching all the blocks we haven't seen and scanning for
394+transactions we care about. We also maintain a map of block hashes to block
395+number, and if we see that a block's parent doesn't have the hash we expect,
396+that means a reorganization has occurred, so we must "rewind" one block at a
397+time until we find where the fork happened, then restart going forward.
398+
399+You don't have to understand this sequence of IO; it's sufficient to see that
400+it's complex and must be exactly right. We used to do this in a state machine
401+directly in Jael. This data is, of course, permanent and needs to be accessible
402+to the rest of the system, so Jael must be a state machine.
403+
404+However, we had many small mistakes in this sequence of IO, some of which only
405+appeared during unusual events, such as when an HTTP request failed at the same
406+time as a reorganization happened.
407+
408+We're in a bit of a bind, though: Jael has to be a state machine. The solution
409+is to factor out the act of getting updates into a thread while the main PKI
410+state is stored in the state machine in Jael.
411+
412+This thread has a definite purpose: given the most recent block number we
413+already know about, fetch all the PKI transactions since then. The function
414+signature is:
415+
416+ syncEthereum oldBlockNumber -> IO [newBlockNumber, newTxs]
417+
418+Jael runs this thread every five minutes and, if it succeeds, then we update our
419+block number and PKI state. If the thread fails, gets stuck, or we need to
420+upgrade it, we just kill the old thread and start a new one. Again, a few extra
421+HTTP requests don’t matter.
422+
423+From Jael's perspective, it looks something like:
424+
425+ state :: State block txs outstandingThread
426+ main(state,event) {
427+ switch event {
428+ Initialize -> [State 0 Null Null, StartTimer (now + five minutes)]
429+ TimerFired ->
430+ effects = If outstandingThread is Null
431+ then Null
432+ else [Kill outstandingThread]
433+ newId = newThreadId()
434+ [State txs newId,
435+ append(effects,
436+ StartThread newId syncEthereum(block),
437+ StartTimer (now + five minutes))]
438+ ThreadFinished Failure -> [State block txs Null, Null]
439+ ThreadFinished Success newBlockNumber newTxs ->
440+ [State newBlockNumber append(txs,newTxs) Null, Null]
441+ }
442+ }
443+
444+In other words, from Jael's perspective, it's just a single IO event that
445+encapsulates arbitrarily complex IO in the thread. A single IO event is easy
446+enough, and we can maintain all our permanency and robustness guarantees,
447+because we know the thread will have one of three results: success, failure, or
448+it hangs. If we set a timeout, hang becomes failure, so there's only two
449+possible results: success or failure. If we handle both of those correctly,
450+we've handled all the possible IO errors that could have occurred.
451+
452+What about upgrading? We've already said we can handle the thread dying for
453+arbitrary reasons, so if an upgrade comes in, we just kill it. We can upgrade
454+Jael's state machine easily since its state is very simple.
455+
456+## Recap
457+
458+Let's recap what we've covered:
459+
460+- There are three styles of IO: imperative, monadic, and explicit state
461+ machines.
462+- Urbit supports explicit state machines (agents) and monadic IO (threads).
463+- Explicit state machines are preferred for permanent, robust, concurrent code.
464+- Monadic IO is preferred for short-lived code with complex IO.
465+- If a state machine needs to perform complex IO, it should encapsulate it in a
466+ thread so that, from the perspective of the state machine, it's simple IO.
467+
468+Hopefully it's fairly clear why Urbit uses explicit state machines in most
469+situations. More than anything else, our goal is to build software that can last
470+forever. Explicit state machines are a crucial tool for that.
471+