@@ -0,0 +1,399 @@
1++++
2+title = "Ford Fusion"
3+date = 2020-07-10
4+description = "Ford Fusion was an overhaul of Urbit's over-the-air upgrade process and a rewrite of its build system. The new update system corrects a few long-standing bugs with the previous one, and the new build system is simpler, smaller (by around 5,000 lines), and easier to manage."
5+[extra]
6+author = "Ted Blackman"
7+ship = "~rovnys-ricfer"
8+image ="https://media.urbit.org/site/posts/essays/ford-fusion.png"
9++++
10+
11+<br>
12+
13+<img class="ba" src="https://media.urbit.org/site/posts/essays/ford-fusion.png">
14+
15+<br>
16+
17+## Overview and Rationale
18+
19+Ford Fusion was an overhaul of Urbit's over-the-air upgrade process and
20+a rewrite of its build system. The new update system corrects a few
21+long-standing bugs with the previous one, and the new build system is
22+simpler, smaller (by around 5,000 lines), and easier to manage.
23+
24+Urbit has always been able to update itself over-the-air (OTA), but this
25+process has often been rocky. Updating an operating system kernel
26+on-the-fly is a difficult problem in general, like performing heart
27+replacement surgery on yourself while running a marathon. Code that
28+allows Linux to update its kernel in this way became a startup called
29+Ksplice, won multiple awards, and sold to Oracle. Even that, as
30+impressive as it is, and as brilliant as its [programmers](https://nelhage.com) are,
31+can only perform certain limited kinds of patches to the kernel.
32+
33+Urbit isn't exactly a traditional operating system, so the comparison is
34+somewhat unfair, but the purpose of better architecture is to create
35+unfair comparisons. In this case, because the Nock layer is frozen,
36+upgrading everything above that layer is easier. Upgrades are also
37+facilitated by pure-functional semantics, transactional event
38+processing, a type system oriented toward concrete data, and orthogonal
39+persistence. These features make it feasible for Urbit to upgrade
40+itself in the general case, not just some special cases.
41+
42+While Urbit has always been able to upgrade itself, this process has
43+historically been fraught with issues. There have been many specific
44+bugs when upgrading, but they boil down to failure to achieve a few
45+properties that Ford Fusion does achieve:
46+
47+- Atomic: the update should complete or fail in one transaction. If it
48+ fails, the system shouldn't get stuck.
49+- Self-contained: there must be no implicit dependencies or hysteresis
50+ (dependence on previous system states) when building the new software
51+ from source.
52+- Ordered: updates must be monotonically sequenced from the system's
53+ lowest layer to highest.
54+
55+Like much of Urbit, these requirements are obvious in retrospect, but
56+took many years and rewrites to identify.
57+
58+In previous versions of Urbit, updates failed atomicity by deferring
59+parts of the update to later events, which are separate transactions
60+that can fail independently. Each deferral caused an exponential
61+increase in the number of failure states that needed to be handled.
62+
63+We've learned that asynchronicity is an entropic state. A system will
64+tend toward more asynchronicity over time unless effort is put into
65+keeping it synchronous. As Jonathan Blow has noted, the language server
66+protocol has turned every editor plugin into a distributed system, since
67+now it has to communicate asynchronously with the main editor process.
68+
69+[Steve Yegge's "platform
70+rant"](https://gist.github.com/chitchcock/1281611) describes a Bezosian
71+edict prohibiting synchronous communication among modules through direct
72+linking. This can be seen as an acknowledgment of the difficulty Amazon
73+was going to have when it needed to turn internal services external. If
74+your software needs to run in hell, build it that way from the start.
75+
76+Urbit is not an industrial computing system; each Urbit instance exists
77+for just one person. Web-scale performance is not the primary goal.
78+Rather, the goal is a system a single person can understand, rely on,
79+and build on.
80+
81+Before Ford Fusion, each commit to the Clay filesystem depended
82+implicitly on the previous commit to define this commit's filetypes
83+(called "marks"). This could cause bugs if the filetype definitions had
84+changed in a backward-incompatible manner, and it forced adding files of
85+new filetypes to be split into two commits: one to define the filetypes,
86+and a second to add files of that new type. More theoretically, it
87+caused history-dependence. It was possible for the validated contents
88+of files in a commit to vary based on the history of commits that led to
89+this one.
90+
91+Another violation of the idea that source code is self-contained was
92+that the source code had access to symmetry-breaking information, namely
93+ship, desk (Urbit's answer to a git branch), and [faked] date. It also
94+had access to Urbit's immutable global namespace, which the kernel makes
95+available as an implicit argument to userspace Hoon code. This
96+namespace is called the "scry" namespace.
97+
98+The scry namespace is immutable and referentially transparent, i.e. a
99+request must always yield the same result for all time. The kernel can
100+always decline to answer a namespace request, though, if you ask for a
101+resource, say, to which you don't have permission, or at a future date,
102+or that would need to be retrieved from another ship.
103+
104+If some code that runs during the build process makes a scry request and
105+is denied by the kernel, the build system has no choice but to treat it
106+as a nondeterministic error. Nondeterministic errors can never be fully
107+eradicated, if for no other reason than that the user always has the
108+option to defenestrate the machine—there's nothing deterministic
109+about that. But we should clearly strive to minimize them, and
110+especially to minimize uncertainty as to under what conditions they
111+might occur.
112+
113+So, as of this update, user code can no longer scry at build time. Once
114+built, though, a Gall agent's scry requests work just fine using a
115+handler constructed by Gall, as has been the case for years.
116+
117+The final kind of failure fixed by Ford Fusion was the lack of ordered
118+layering during a software update. The most common form of this failure
119+was that old Ford had a tendency to try to build userspace code using
120+the previous version's standard library. This didn't work too well,
121+unsurprisingly.
122+
123+Emerging from this underworld required making a number of changes to the
124+Arvo kernel, Clay, Gall, and the procedure for kernel updates. To avoid
125+turning into a pillar of salt, I'll skip the details of how the old
126+system worked and instead describe the new update procedure.
127+
128+## How Updates Work Now
129+
130+These are the layers of the stack that update according to this process,
131+listed from lowest layer to highest:
132+
133++ hoon: the hoon language definition and compiler, defined in `/sys/hoon/hoon`
134++ arvo: the arvo kernel proper and related type definitions, in `/sys/arvo/hoon`
135++ zuse: the standard library, in `/sys/zuse/hoon`
136++ vanes: arvo kernel modules, including clay itself, in folder `/sys/vane`
137++ userspace: apps, marks, ancillary source code like libraries, and user data
138+
139+Clay is responsible for enforcing the layering of updates. An update to
140+one layer necessitates a reload to all layers above it; e.g. a change to
141+Zuse should trigger updates to the vanes and userspace. Conversely, an
142+update to a higher layer should not cause a spurious reload of lower
143+layers, which should not be affected by the change; for example, an
144+update to just userspace should not cause any reloads of system code.
145+
146+An update to a module is triggered when an attempt is made to commit a
147+change to Clay that affects one or more files needed to build the
148+module. For example, if the 'foo' agent's source, defined in
149+`/app/foo/hoon`, imports the 'bar' library from `/lib/bar/hoon`, then a
150+modification to `/lib/bar/hoon` triggers an update to the 'foo' agent.
151+
152+The process of updating a layer varies by layer. The Hoon and Zuse
153+layers are stateless, so their newly rebuilt cores must be stored, but
154+they have no state that would need to be migrated. The Arvo kernel,
155+vanes, and userspace agents are all live, stateful programs, so in order
156+to update one of those, the system must extract the state from the old
157+program, pass that data into the newly built program, then discard the
158+old program and install the new one. Arvo and agent state injection
159+routines can emit effects, but vane updates cannot.
160+
161+As an aside, future iterations of the upgrade system might change that
162+to allow vanes to emit effects on update. There are some difficulties
163+in maintaining the integrity of the duct stack if going down that route,
164+but they might be surmountable.
165+
166+Clay determines which layers need to be updated based on which files
167+have changed in a proposed commit and which modules depend on those
168+files. If it needs to update kernelspace (there's a change to `/sys` on
169+the `%home` desk, which is the only "live" desk), it sends a sequence of
170+moves to Arvo to ask Arvo to perform any necessary updates to hoon, the
171+kernel, Zuse, and vanes. This sequence is terminated by an extra move
172+back to Clay itself, which will be received by the updated version of
173+Clay after migrating its state, so that the rebuilt Clay can use the
174+rebuilt version of Zuse to rebuild userspace and notify clients of the
175+update. One client is Gall, which reloads its agents when it hears that
176+Clay has updated Zuse.
177+
178+Clay triggers updates, but the Arvo kernel is responsible for performing
179+updates to all kernelspace layers, and Gall is responsible for updating
180+userspace agents. Agents are stored in Gall's state, but all other
181+layers are stored directly in the Arvo core's state, so the kernel
182+contains the routines that reload Hoon, itself, Zuse, and its vanes.
183+
184+The Arvo kernel reloads itself by receiving its new source code as a
185+`%lyra` move from Clay, compiling this source into a core with empty
186+state, then calling the gate produced by the new core's `+load` arm with
187+the relevant parts of the old state. This now includes not just the
188+vane cores and their states, but also the Arvo "duct" call stack, which
189+maintains a stack of queues of moves to be passed from one vane to
190+another, and a list of moves enqueued to be emitted to Unix as effects
191+at the end of the current Arvo event. This allows an Arvo update to
192+happen in the middle of a more complex event without disturbing other
193+sequences of processing steps happening concurrently in the vanes.
194+
195+## How Clay Validates a Desk
196+
197+A desk is Urbit's answer to a git branch. It's almost identical, except
198+every commit also has a semantically meaningful revision number, and all
199+files are typed and validated.
200+
201+If Clay has been asked to perform a commit, after possibly updating
202+kernelspace, it needs to validate all the files in this desk and notify
203+all subscribers to live queries of this desk's data. Gall, for example,
204+maintains live queries on builds of its live agents. Validation uses
205+the Ford build system, which as of this update, is no longer a
206+standalone vane but a core within Clay.
207+
208+A Clay commit, like a git commit, is specified as the current value of
209+all its changed files (and, separately, references its parent commits by
210+hash), not as the diff from a parent commit. Unlike git, Clay is typed,
211+and every file must be validated according to its "mark". A mark is
212+named like a file extension, e.g. `%txt`, `%png`, or `%noun`, and Clay
213+maintains a mapping from that name to behaviors of values of that type
214+under various operations. The last path segment of any Clay path
215+specifies the mark to use for operations on that file, including
216+validation.
217+
218+Mark operations include conversion to and from other marks (such as
219+converting `%json` to `%txt`), revision control operations (diff, patch,
220+join, and merge), and validating an untyped noun. Operations for mark
221+`%foo-bar` are defined by a core built using the source code at
222+`/mar/foo-bar/hoon`, or if that doesn't exist, at `/mar/foo/bar/hoon`.
223+
224+Consider a file at `/web/foo/json`. In order to validate this file,
225+Clay must load the mark definition core and use its validation routine
226+to ensure the untyped value of `/web/foo/json` is in fact valid JSON.
227+To obtain this core, Clay must build the file at `/mar/json/hoon` from
228+source and then process the resulting core using some mild
229+metaprogramming to get a standard interface core for dealing with marks,
230+called a `$dais`, whose type is defined in Zuse.
231+
232+Since building a source file only makes sense if the file has been
233+validated as a valid `%hoon` file, but mark definitions themselves must
234+be built from source, there's a logical dependency cycle. To break this
235+cycle, Clay hard-codes the validation of `%hoon` files. This allows any
236+file to depend on any other file of any mark, as long as there are no
237+cyclic dependencies. The Ford build system, as of this update, now
238+verifies that there are no cyclic dependencies as it runs.
239+
240+Since building a file is a pure function, Clay memoizes the results of
241+all builds, including builds of marks, mark conversions, and hoon source
242+files. These memoization results are stored along with the desk and are
243+used by later revisions of that desk. Future work should allow memoized
244+builds to be pulled from all parents in a merge commit, but for now only
245+the previous revision of the current desk is used. This is a major
246+simplification of previous Ford architectures, which maintained much
247+more complex caches with less clear eviction semantics. Now on every
248+commit, we just throw away any unused memoized builds from the previous
249+revision's Ford cache.
250+
251+Once Clay has validated every file in this new revision of a desk, it
252+constructs and sends updates to any subscriptions that other vanes or
253+agents have requested. More Ford builds may be run to fulfill these
254+requests, including builds for any running agents whose dependencies
255+changed in this commit.
256+
257+When Gall receives a newly rebuilt agent from Clay, it calls the gate
258+produced by the `+on-load` arm of the new agent with the state extracted
259+from the old agent. If there is a crash in any `+on-load` calls or in
260+the handling of any effects they emit (which can include more agent
261+activations), then the whole event crashes, canceling the commit. This
262+effectively gives any agent the ability to abort a commit by crashing.
263+
264+It is a bit counterintuitive that an app reload failure could prevent a
265+kernel update. The reason is that we don't want the system to update itself
266+into a broken state. An Urbit can be rendered practically unusable by
267+the presence of broken agents, even if the kernel hasn't lost integrity,
268+so it's kinder to the user not to break their agents by installing an
269+incompatible kernel update. This also puts virtuous pressure on kernel
270+developers not to "break userspace", the importance of which has been
271+insisted on for decades by Linus Torvalds, among others.
272+
273+## Ford Build Semantics
274+
275+### The Three Types of Ford Builds: Files, Marks, and Casts
276+
277+The Ford build semantics have been simplified. There are now three
278+kinds of builds that Ford can perform: files, marks, and casts, all of
279+which happen synchronously as function calls inside Clay and are
280+available (without memoization) as scry interfaces.
281+
282+File builds are the most basic form of build. A file build takes in a
283+filepath containing Ford runes and Hoon source, runs the Ford runes to
284+perform imports, and then compiles the source, producing a `$vase`, a
285+noun tagged with its Hoon type. Clay exposes file builds into the scry
286+namespace with `%ca`: as an example, `.^(vase %ca
287+/~zod/home/3/lib/sole/hoon)` will build the `sole` library.
288+
289+A mark build produces a `$dais` mark-interface core. It first performs
290+a file build on the Hoon file in `/mar` that defines the mark core, then
291+it does some metaprogramming to make the operations more convenient to
292+use, possibly loading another mark core that the raw mark core specified
293+as delegate for revision control operations. Clay exposes mark builds
294+into the scry namespace with `%cb`: as an example, `.^(dais:clay %cb
295+/~zod/home/3/mar/foo/hoon)` builds a `$dais` for the `%foo` mark.
296+
297+A cast build produces a `$tube`: a gate that takes a value of one mark
298+as input and converts it to a valid value of another mark or crashes. A
299+mark can convert a value to another mark by providing an arm in its
300+`+grow` core named after the destination mark, and it can convert a
301+value from another mark using an arm in its `+grab` core. A mark can
302+also chain a conversion through an intermediary using an arm in its
303+`+jump` core or by having a `+grab` arm produce a delegate mark instead
304+of directly defining a conversion gate. Clay exposes cast builds into
305+the scry namespace with `%cc`: as an example, `.^(tube:clay %cc
306+/~zod/home/3/foo/bar)` builds a `$tube` conversion gate from `%foo` to
307+`%bar`.
308+
309+### Ford Runes
310+
311+There are now only four Ford runes. A file can contain zero, one, or
312+many of each, but each Ford expression can only be one line, and they
313+must be in the standard order of `/-`s, `/+`s, `/=`s, and then `/*`s.
314+
315+```
316+/- foo, *bar, baz=qux
317+```
318+
319+The `/-` rune imports a structures file from `/sur`. You can import it
320+as just `foo`, in which case the build result of that file (usually a
321+core with mold definitions) will be pinned into the compilation subject
322+with the face `foo`. If you prefix it with a `*` as in `*bar`, the
323+result will be pinned into the subject with no face; if the structures
324+file compiled to a core, this exposes all the arms into the namespace of
325+the compilation subject. Finally, if you import it as `baz=qux`, the
326+`baz` face will be applied instead of `qux`. This is similar to "import
327+as" in other languages.
328+
329+```
330+/+ foo, *bar, baz=qux
331+```
332+
333+The `/+` rune imports a library file from `/lib`. Aside from the
334+different source folder, the syntax and semantics are the same as for
335+`/-`.
336+
337+```
338+/= clay-raw /sys/vane/clay
339+```
340+
341+The `/=` rune imports the result of building a hoon file from a
342+user-specified path (the second argument), wrapping it in a face
343+specified by the first argument. The final `/hoon` at the end of the
344+path must be omitted. This is mostly useful for importing a file for
345+testing. The file at the specified path will be built as a normal
346+userspace hoon file; i.e. its compilation subject will be Zuse augmented
347+with the results of any Ford runes it has at the top of the file.
348+
349+```
350+/* hello-gen %hoon /gen/hello/hoon
351+```
352+
353+The `/*` rune imports the contents of a file in the desk, specified as
354+the third argument with the full path including the trailing mark,
355+converted to the mark specified by the second argument, and pinned into
356+the compilation subject wrapped in the face specified by the first
357+argument. This can be used to import static data at build-time, such as
358+a data file, a media file, or, in the case of this example, a hoon file
359+as source text rather than already built.
360+
361+A valid userspace hoon file must contain a nonempty list of hoons (hoon
362+source expressions) below the Ford runes, separated by gap (more than
363+one space, or at least one newline). The system wraps this list of
364+hoons in a `=~` expression so that the result of the previous hoon is
365+used as the subject of the next hoon. The result of the Ford runes is
366+used as the compilation subject for this `=~` hoon; informally, the
367+shape of the compilation subject can be thought of as:
368+```
369+:* fastar-2 fastar-1
370+ fastis-2 fastis-1
371+ faslus-2 faslus-1
372+ fashep-2 fashep-1
373+ <zuse>
374+==
375+```
376+
377+## Conclusion
378+
379+Since deployment of Ford Fusion to the livenet a week ago, over-the-air
380+updates (OTAs) have been much smoother. Instead of taking several
381+hours, frequently using too much memory, and sometimes leaving ships in
382+inconsistent states, multiple OTAs have been pushed out since, including
383+kernelspace changes, and most users didn't even notice.
384+
385+Future work will entail making better use of desks other than `%home`
386+and improving the development process given the tighter coupling between
387+source code and kernel and tighter criteria for accepting an update.
388+
389+This work also hopefully provides a good foundation of a package
390+management and software distribution system for Urbit. As
391+`~wicdev-wisryt` has said, a user should be able to run `|install
392+~norsyr-torryn/canvas` to load and build remote source. No one should
393+experience dependency hell on Urbit, but we're not there yet. At least
394+now, building a desk has no dependencies, other than a Ford with a
395+compatible Hoon compiler. No decisions have been made on this yet, but
396+Ford might get moved to inside the desk, possibly by making Zuse
397+callable. This could allow a desk to expose a Nock interface in
398+addition to a typed Hoon interface, which could even let a desk be used
399+as a "pill" bootloader.