@@ -1,10 +1,10 @@
2title = "Ford Fusion"
2title = "Ford Fusion"
3−date = 2020-07-10
3+date = 2020-07-15
4description = "Ford Fusion was an overhaul of Urbit's over-the-air upgrade process and a rewrite of its build system. The new update system corrects a few long-standing bugs with the previous one, and the new build system is simpler, smaller (by around 5,000 lines), and easier to manage."
4description = "Ford Fusion was an overhaul of Urbit's over-the-air upgrade process and a rewrite of its build system. The new update system corrects a few long-standing bugs with the previous one, and the new build system is simpler, smaller (by around 5,000 lines), and easier to manage."
6−author = "Ted Blackman"
6+author = "Ted Blackman + Philip Monk"
7−ship = "~rovnys-ricfer"
7+ship = "~rovnys-ricfer + ~wicdev-wisryt"
8image ="https://media.urbit.org/site/posts/essays/ford-fusion.png"
8image ="https://media.urbit.org/site/posts/essays/ford-fusion.png"
@@ -16,34 +16,16 @@ image ="https://media.urbit.org/site/posts/essays/ford-fusion.png"
17## Overview and Rationale
17## Overview and Rationale
19−Ford Fusion was an overhaul of Urbit's over-the-air upgrade process and
19+Ford Fusion was an overhaul of Urbit's over-the-air upgrade process and a rewrite of its build system. The new update system corrects a few long-standing bugs with the previous one, and the new build system is simpler, smaller (by around 5,000 lines), and easier to manage.
20−a rewrite of its build system. The new update system corrects a few
20+
21−long-standing bugs with the previous one, and the new build system is
21+Since deployment of Ford Fusion to the livenet in late June, over-the-air updates (OTAs) have been much smoother. Before Ford Fusion, it was common for an OTA to take several hours, use too much memory, and leave ships in inconsistent states. After Ford Fusion, multiple OTAs have been pushed out, including kernelspace changes, and most users didn't even notice.
22−simpler, smaller (by around 5,000 lines), and easier to manage.
24−Urbit has always been able to update itself over-the-air (OTA), but this
25−process has often been rocky. Updating an operating system kernel
26−on-the-fly is a difficult problem in general, like performing heart
27−replacement surgery on yourself while running a marathon. Code that
28−allows Linux to update its kernel in this way became a startup called
29−Ksplice, won multiple awards, and sold to Oracle. Even that, as
30−impressive as it is, and as brilliant as its [programmers](https://nelhage.com) are,
31−can only perform certain limited kinds of patches to the kernel.
33−Urbit isn't exactly a traditional operating system, so the comparison is
34−somewhat unfair, but the purpose of better architecture is to create
35−unfair comparisons. In this case, because the Nock layer is frozen,
36−upgrading everything above that layer is easier. Upgrades are also
37−facilitated by pure-functional semantics, transactional event
38−processing, a type system oriented toward concrete data, and orthogonal
39−persistence. These features make it feasible for Urbit to upgrade
40−itself in the general case, not just some special cases.
42−While Urbit has always been able to upgrade itself, this process has
23+Urbit has always been able to update itself OTA, but this process has often been rocky. Updating an operating system kernel on-the-fly is a difficult problem in general, like performing heart replacement surgery on yourself while running a marathon. Code that allows Linux to update its kernel in this way became a startup called Ksplice, won multiple awards, and sold to Oracle. Even that, as impressive as it is, and as brilliant as its [programmers](https://nelhage.com) are, can only perform certain limited kinds of patches to the kernel.
43−historically been fraught with issues. There have been many specific
24+
44−bugs when upgrading, but they boil down to failure to achieve a few
25+Urbit isn't exactly a traditional operating system, so the comparison is somewhat unfair, but the purpose of better architecture is to create unfair comparisons. In this case, because the Nock layer is frozen, upgrading everything above that layer is easier. Upgrades are also facilitated by pure-functional semantics, transactional event processing, a type system oriented toward concrete data, and orthogonal persistence. These features make it feasible for Urbit to upgrade
45−properties that Ford Fusion does achieve:
26+itself in the general case, not just some special cases.
28+Ford Fusion has fixed the major upgrade issues of the past by guaranteeing three properties that in retrospect are obvious requirements, but, like much of Urbit, took many years and rewrites to identify as such:
47- Atomic: the update should complete or fail in one transaction. If it
29- Atomic: the update should complete or fail in one transaction. If it
48 fails, the system shouldn't get stuck.
30 fails, the system shouldn't get stuck.
49- Self-contained: there must be no implicit dependencies or hysteresis
31- Self-contained: there must be no implicit dependencies or hysteresis
@@ -52,19 +34,19 @@ properties that Ford Fusion does achieve:
52- Ordered: updates must be monotonically sequenced from the system's
34- Ordered: updates must be monotonically sequenced from the system's
53 lowest layer to highest.
35 lowest layer to highest.
55−Like much of Urbit, these requirements are obvious in retrospect, but
37+Let’s walk through them one by one.
56−took many years and rewrites to identify.
38+
41+In previous versions of Urbit, updates failed atomicity by deferring parts of the update to later events, which are separate transactions that can fail independently. Generally, each deferral causes an exponential increase in the number of failure states that needed to be handled.
43+We've learned that asynchronicity is an entropic state. A system will tend toward more asynchronicity over time unless effort is put into keeping it synchronous. [As Jonathan Blow has noted](https://youtu.be/pW-SOdj4Kkk?t=2547), the language server protocol has turned every editor plugin into a distributed system, since now it has to communicate asynchronously with the main editor process.
45+Consider an update system that took multiple Arvo events to complete. An ad-hoc higher-level transaction system would need to be built to roll back the effects of the first few events in case of failure. It’s important for various parts of the system to be able to emit effects on upgrade; since those effects would need to be rolled back if a later event in this upgrade fails, the system would need to maintain a queue of those events and only apply them once all the other upgrade events have completed.
58−In previous versions of Urbit, updates failed atomicity by deferring
47+Note that the asynchronicity has now spread. Some effects that would normally be guaranteed to be processed synchronously might now be asynchronous. Entropy has begun to take hold, chipping away at the set of invariants the system is capable of guaranteeing.
59−parts of the update to later events, which are separate transactions
60−that can fail independently. Each deferral caused an exponential
61−increase in the number of failure states that needed to be handled.
63−We've learned that asynchronicity is an entropic state. A system will
49+This observation is not purely theoretical. False modularity was the cause of internal asynchronicity in Clay where it had to wait for responses in a complex dance with Ford, which was another vane (Arvo kernel module); moving Ford into Clay allowed function calls that were synchronous from Clay’s perspective, which allowed further simplifications, culminating in about a twenty percent reduction of source code size of the Arvo kernelspace.
64−tend toward more asynchronicity over time unless effort is put into
65−keeping it synchronous. As Jonathan Blow has noted, the language server
66−protocol has turned every editor plugin into a distributed system, since
67−now it has to communicate asynchronously with the main editor process.
69[Steve Yegge's "platform
51[Steve Yegge's "platform
70rant"](https://gist.github.com/chitchcock/1281611) describes a Bezosian
52rant"](https://gist.github.com/chitchcock/1281611) describes a Bezosian
@@ -73,62 +55,29 @@ linking. This can be seen as an acknowledgment of the difficulty Amazon
73was going to have when it needed to turn internal services external. If
55was going to have when it needed to turn internal services external. If
74your software needs to run in hell, build it that way from the start.
56your software needs to run in hell, build it that way from the start.
76−Urbit is not an industrial computing system; each Urbit instance exists
58+An Urbit ship is not an enterprise SaaS product and does not need to run in this hell; it exists for just one person, with natural pressure pushing it in the opposite direction from Amazon’s web services. Instead of needing a ship’s state and functionality sharded into microservices strewn across multiple clusters, an Urbit instance is easiest to manage as a single server with all its state unified into one data structure and its event log stored as one totally ordered sequence of state updates -- the antithesis of a distributed system.
77−for just one person. Web-scale performance is not the primary goal.
59+
78−Rather, the goal is a system a single person can understand, rely on,
60+### Self-Contained
80−
62+Before Ford Fusion, each commit to the Clay filesystem validated its files using filetypes (called “marks”) defined by files in the previous commit. This could cause bugs if the filetype definitions had changed in a backward-incompatible manner. It also meant a commit could not add both a new filetype and new files of that type; instead, you needed two commits: one to define the filetypes, and a second to add files of that new type. More theoretically, it caused history-dependence. The validated contents of files in a commit could vary based on the history of commits that led to this one.
81−Before Ford Fusion, each commit to the Clay filesystem depended
63+
82−implicitly on the previous commit to define this commit's filetypes
64+Another way source code failed to be self-contained was that it had access to symmetry-breaking information at build-time, namely ship, desk (Urbit's answer to a git branch), and (faked) date. A build recipe should be able to shared, cached, and rerun without dependence on local conditions, so user code now no longer learns its ship, desk, or date until runtime.
83−(called "marks"). This could cause bugs if the filetype definitions had
65+
84−changed in a backward-incompatible manner, and it forced adding files of
66+Source code also had build-time access to Urbit's immutable global namespace, called the “scry namespace”, which the kernel makes available as an implicit argument to userspace Hoon code. The scry namespace is immutable and referentially transparent, i.e. a request must always yield the same result for all time, but if an agent asks the kernel for a resource that’s from the future, hosted on another ship, or to which that agent doesn’t have permission, the kernel will deny the request.
85−new filetypes to be split into two commits: one to define the filetypes,
67+
86−and a second to add files of that new type. More theoretically, it
68+If the kernel denies a scry request that user code made during the build process, the build system has no choice but to treat it as a nondeterministic error. Nondeterministic errors can never be fully eradicated, if for no other reason than that the user always has the option to defenestrate the machine—there's nothing deterministic about that. But we try to minimize them, and especially to minimize uncertainty as to under what conditions they might occur.
87−caused history-dependence. It was possible for the validated contents
69+
88−of files in a commit to vary based on the history of commits that led to
70+No build should be killed by the absence of files outside the desk, so as of this update, user code can no longer scry at build time. Once built, userspace programs can scry if run in a context with a scry handler; a Gall agent's runtime scry requests still work just fine.
91−Another violation of the idea that source code is self-contained was
73+
92−that the source code had access to symmetry-breaking information, namely
74+The final kind of failure fixed by Ford Fusion was the lack of ordered layering during a software update. The most common form of this failure was that old Ford had a tendency to try to build userspace code using the previous version's standard library. This didn't work too well, unsurprisingly.
93−ship, desk (Urbit's answer to a git branch), and [faked] date. It also
75+
94−had access to Urbit's immutable global namespace, which the kernel makes
95−available as an implicit argument to userspace Hoon code. This
96−namespace is called the "scry" namespace.
98−The scry namespace is immutable and referentially transparent, i.e. a
99−request must always yield the same result for all time. The kernel can
100−always decline to answer a namespace request, though, if you ask for a
101−resource, say, to which you don't have permission, or at a future date,
102−or that would need to be retrieved from another ship.
104−If some code that runs during the build process makes a scry request and
105−is denied by the kernel, the build system has no choice but to treat it
106−as a nondeterministic error. Nondeterministic errors can never be fully
107−eradicated, if for no other reason than that the user always has the
108−option to defenestrate the machine—there's nothing deterministic
109−about that. But we should clearly strive to minimize them, and
110−especially to minimize uncertainty as to under what conditions they
113−So, as of this update, user code can no longer scry at build time. Once
114−built, though, a Gall agent's scry requests work just fine using a
115−handler constructed by Gall, as has been the case for years.
117−The final kind of failure fixed by Ford Fusion was the lack of ordered
118−layering during a software update. The most common form of this failure
119−was that old Ford had a tendency to try to build userspace code using
120−the previous version's standard library. This didn't work too well,
123−Emerging from this underworld required making a number of changes to the
76+Emerging from this underworld required making a number of changes to the Arvo kernel, Clay, Gall, and the procedure for kernel updates. To avoid turning into a pillar of salt, I'll skip the details of how the old system worked and instead describe the new update procedure.
124−Arvo kernel, Clay, Gall, and the procedure for kernel updates. To avoid
125−turning into a pillar of salt, I'll skip the details of how the old
126−system worked and instead describe the new update procedure.
128## How Updates Work Now
78## How Updates Work Now
130−These are the layers of the stack that update according to this process,
80+These are the layers of the stack that update themselves on the fly, from lowest to highest:
131−listed from lowest layer to highest:
133+ hoon: the hoon language definition and compiler, defined in `/sys/hoon/hoon`
82+ hoon: the hoon language definition and compiler, defined in `/sys/hoon/hoon`
134+ arvo: the arvo kernel proper and related type definitions, in `/sys/arvo/hoon`
83+ arvo: the arvo kernel proper and related type definitions, in `/sys/arvo/hoon`
@@ -136,235 +85,115 @@ listed from lowest layer to highest:
136+ vanes: arvo kernel modules, including clay itself, in folder `/sys/vane`
85+ vanes: arvo kernel modules, including clay itself, in folder `/sys/vane`
137+ userspace: apps, marks, ancillary source code like libraries, and user data
86+ userspace: apps, marks, ancillary source code like libraries, and user data
139−Clay is responsible for enforcing the layering of updates. An update to
88+An update to one layer necessitates a reload of all layers above it; e.g. a change to Zuse should trigger updates to the vanes and userspace. Conversely, an update to a higher layer should not cause a spurious reload of lower layers, which should not be affected by the change; for example, an update to just userspace should not cause any reloads of system code.
140−one layer necessitates a reload to all layers above it; e.g. a change to
89+
141−Zuse should trigger updates to the vanes and userspace. Conversely, an
90+Clay is responsible for enforcing the layering of updates. An update to a module is triggered when an attempt is made to commit a change to Clay that affects one or more files needed to build the module. For example, if the 'foo' agent's source, defined in `/app/foo/hoon`, imports the 'bar' library from `/lib/bar/hoon`, then a modification to `/lib/bar/hoon` triggers an update to the 'foo' agent. All vanes and userspace files depend on Zuse, which depends on the Arvo and Hoon sources, so a change to the Hoon, Arvo, or Zuse sources will trigger updates to all vanes and userspace files.
142−update to a higher layer should not cause a spurious reload of lower
91+
143−layers, which should not be affected by the change; for example, an
92+When asked to perform a commit, Clay determines which layers need to be updated based on which files have changed and which modules depend on those files. For now, all running programs load their source from the `%home` desk, so only changes to `%home` trigger stateful updates. Files in other desks can be built, but not installed into the system. This might be relaxed in the future.
144−update to just userspace should not cause any reloads of system code.
93+
145−
94+The process of updating varies by layer. The Hoon and Zuse layers are stateless, so their newly rebuilt cores (Nock executables) must be stored (somewhere in the system’s Nock tree, in memory; remember, Urbit is a single-level store), but they have no state that would need to be migrated. The Arvo kernel, vanes, and userspace agents are all live, stateful programs, so in order to update one of those, the system must extract the state from the old program, pass that data into the newly built program, then discard the old program and store the new one. Arvo and agent state injection routines can emit effects, but vane updates cannot.
146−An update to a module is triggered when an attempt is made to commit a
95+
147−change to Clay that affects one or more files needed to build the
96+To work around this limitation, Gall has a two-phase update process. First it enters a dormant “pupal” phase that stores not running agent cores, but only the agent states that the old Gall had extracted from its agents. When Clay notifies Gall that its agents have been rebuilt, Gall “molts” back into normal functionality by loading the agent cores from Clay and then running their `+on-load` routines to inject the old state.
148−module. For example, if the 'foo' agent's source, defined in
97+
149−`/app/foo/hoon`, imports the 'bar' library from `/lib/bar/hoon`, then a
98+If there's a change to files in `/sys` on the `%home` desk, Clay asks Arvo to update kernelspace. Clay sends a sequence of moves (effects) to Arvo to ask Arvo to perform any necessary updates to hoon, the kernel, Zuse, and vanes. This sequence is terminated by an extra move back to Clay itself, which will be received by the updated version of Clay after migrating its state. The rebuilt Clay can then use the newly rebuilt version of Zuse to rebuild userspace and notify clients of the update. One client is Gall, which molts when Clay notifies it.
150−modification to `/lib/bar/hoon` triggers an update to the 'foo' agent.
99+
151−
100+Clay triggers updates, but the Arvo kernel is responsible for performing updates to all kernelspace layers, and Gall is responsible for updating userspace agents. Agents are stored in Gall's state, but all other layers are stored directly in the Arvo core's state, so the Arvo kernel contains the routines that reload Hoon, the kernel itself, Zuse, and the vanes.
152−The process of updating a layer varies by layer. The Hoon and Zuse
101+
153−layers are stateless, so their newly rebuilt cores must be stored, but
102+The Arvo kernel reloads itself by receiving its new source code as a `%lyra` move from Clay, compiling this source into a core with empty state, then calling the new core's `+load` routine with the relevant parts of the old state. The state passed to the new Arvo now includes not just the vane cores and their states, but also the Arvo "duct" call `stack`, which maintains a stack of queues of moves to be passed from one vane to another, and a list of effects to emit to Unix at the end of the current Arvo event. If needed, Arvo could migrate the outstanding moves themselves -- if, say, the duct datatype changes.
154−they have no state that would need to be migrated. The Arvo kernel,
103+
155−vanes, and userspace agents are all live, stateful programs, so in order
156−to update one of those, the system must extract the state from the old
157−program, pass that data into the newly built program, then discard the
158−old program and install the new one. Arvo and agent state injection
159−routines can emit effects, but vane updates cannot.
161−As an aside, future iterations of the upgrade system might change that
162−to allow vanes to emit effects on update. There are some difficulties
163−in maintaining the integrity of the duct stack if going down that route,
164−but they might be surmountable.
166−Clay determines which layers need to be updated based on which files
167−have changed in a proposed commit and which modules depend on those
168−files. If it needs to update kernelspace (there's a change to `/sys` on
169−the `%home` desk, which is the only "live" desk), it sends a sequence of
170−moves to Arvo to ask Arvo to perform any necessary updates to hoon, the
171−kernel, Zuse, and vanes. This sequence is terminated by an extra move
172−back to Clay itself, which will be received by the updated version of
173−Clay after migrating its state, so that the rebuilt Clay can use the
174−rebuilt version of Zuse to rebuild userspace and notify clients of the
175−update. One client is Gall, which reloads its agents when it hears that
176−Clay has updated Zuse.
178−Clay triggers updates, but the Arvo kernel is responsible for performing
179−updates to all kernelspace layers, and Gall is responsible for updating
180−userspace agents. Agents are stored in Gall's state, but all other
181−layers are stored directly in the Arvo core's state, so the kernel
182−contains the routines that reload Hoon, itself, Zuse, and its vanes.
184−The Arvo kernel reloads itself by receiving its new source code as a
185−`%lyra` move from Clay, compiling this source into a core with empty
186−state, then calling the gate produced by the new core's `+load` arm with
187−the relevant parts of the old state. This now includes not just the
188−vane cores and their states, but also the Arvo "duct" call stack, which
189−maintains a stack of queues of moves to be passed from one vane to
190−another, and a list of moves enqueued to be emitted to Unix as effects
191−at the end of the current Arvo event. This allows an Arvo update to
192−happen in the middle of a more complex event without disturbing other
104+Passing the Arvo call stack state to new Arvo allows a kernel update to happen in the middle of a more complex event without disturbing other sequences of processing steps happening concurrently in the vanes.
193−sequences of processing steps happening concurrently in the vanes.
105+
106+This entire update process happens in one Arvo event and doesn’t break event-dispatching semantics. This not only provides atomic rollback, but allows the update to be combined with other actions into a larger transaction -- for example, to stage complex changes, user code could trigger two kernel updates in a row, both in the same event.
108+Note that this is the opposite situation from the entropically leaking asynchronicity described earlier. Now the guarantees don’t deteriorate; they can be composed into stronger guarantees.
195## How Clay Validates a Desk
110## How Clay Validates a Desk
197−A desk is Urbit's answer to a git branch. It's almost identical, except
112+A desk is Urbit's answer to a git branch. It's almost identical, except all files are typed and validated, and whenever a commit becomes the equivalent of git’s ‘HEAD’, it’s assigned a semantically meaningful revision number, and all files are typed and validated.
198−every commit also has a semantically meaningful revision number, and all
113+
199−files are typed and validated.
114+If Clay has been asked to perform a commit, it needs to validate all the files in this desk and notify all subscribers to live queries of this desk's data. Gall, for example, maintains live queries on builds of its live agents. Validation uses the Ford build system, which as of this update is no longer a standalone vane but a core within Clay.
201−If Clay has been asked to perform a commit, after possibly updating
116+A Clay commit, like a git commit, is specified as the current value of all its changed files (and, separately, references its parent commits by hash), not as the diff from a parent commit. Unlike git, Clay is typed, and every file must be validated according to its "mark". A mark is named like a file extension, e.g. `%txt`, `%png`, or `%noun`, and Clay maintains a mapping from that name to behaviors of values of that type under various operations. The last segment of any Clay path specifies the mark to use for operations on that file, including validation.
202−kernelspace, it needs to validate all the files in this desk and notify
117+
203−all subscribers to live queries of this desk's data. Gall, for example,
118+Mark operations include conversion to and from other marks (such as converting `%json` to `%txt`), revision control operations (diff, patch, and merge), and validating an untyped noun. Operations for mark `%foo-bar` are defined by a core built using the source code at `/mar/foo-bar/hoon`, or if that doesn't exist, at `/mar/foo/bar/hoon`.
204−maintains live queries on builds of its live agents. Validation uses
119+
205−the Ford build system, which as of this update, is no longer a
120+Consider a file at `/web/foo/json`. In order to validate this file, Clay must load the mark definition core and use its validation routine to ensure the untyped value of `/web/foo/json` is in fact valid JSON. To obtain this core, Clay must build the file at `/mar/json/hoon` from source and then process the resulting raw mark core using some mild metaprogramming to get a standard interface core for dealing with marks, called a `$dais`, whose type is defined in Zuse.
206−standalone vane but a core within Clay.
121+
207−
122+Since building a source file only makes sense if the file has been validated as a `%hoon` file, but mark definitions themselves must be built from source, there's a logical dependency cycle -- who validates the validators? To break this cycle, Clay hard-codes the validation of `%hoon` files. This allows mark definitions to be built from source, and in fact any file can depend on any other file of any mark as long as there are no cycles. As of Ford Fusion, Ford performs a cycle check to ensure acyclicity.
208−A Clay commit, like a git commit, is specified as the current value of
123+
209−all its changed files (and, separately, references its parent commits by
124+Since building a file is a pure function, Clay memoizes the results of all builds, including builds of marks, mark conversions, and hoon source files. These memoization results are stored along with the desk and are used by later revisions of that desk. Future work should allow merge commits to pull memoized builds from all parents, but for now only the previous revision of the current desk is used. This is a major simplification of previous Ford architectures, which maintained much more complex caches with less clear eviction semantics. Now on every commit, we just throw away any unused memoized builds from the previous revision's Ford cache.
210−hash), not as the diff from a parent commit. Unlike git, Clay is typed,
125+
211−and every file must be validated according to its "mark". A mark is
126+Once Clay has validated every file in this new revision of a desk, it constructs and sends updates to any subscriptions that other vanes or agents have requested. More Ford builds may be run to fulfill these requests, including builds for any running agents whose dependencies changed in this commit.
212−named like a file extension, e.g. `%txt`, `%png`, or `%noun`, and Clay
127+
213−maintains a mapping from that name to behaviors of values of that type
128+When Gall receives a newly rebuilt agent from Clay, it calls the gate produced by the `+on-load` arm of the new agent with the state extracted from the old agent. If there is a crash in any `+on-load` calls or in the handling of any effects they emit (which can include more agent activations), then the whole event crashes, canceling the commit. This effectively gives any agent the ability to abort a commit by crashing.
214−under various operations. The last path segment of any Clay path
129+
215−specifies the mark to use for operations on that file, including
218−Mark operations include conversion to and from other marks (such as
219−converting `%json` to `%txt`), revision control operations (diff, patch,
220−join, and merge), and validating an untyped noun. Operations for mark
221−`%foo-bar` are defined by a core built using the source code at
222−`/mar/foo-bar/hoon`, or if that doesn't exist, at `/mar/foo/bar/hoon`.
224−Consider a file at `/web/foo/json`. In order to validate this file,
225−Clay must load the mark definition core and use its validation routine
226−to ensure the untyped value of `/web/foo/json` is in fact valid JSON.
227−To obtain this core, Clay must build the file at `/mar/json/hoon` from
228−source and then process the resulting core using some mild
229−metaprogramming to get a standard interface core for dealing with marks,
230−called a `$dais`, whose type is defined in Zuse.
232−Since building a source file only makes sense if the file has been
233−validated as a valid `%hoon` file, but mark definitions themselves must
234−be built from source, there's a logical dependency cycle. To break this
235−cycle, Clay hard-codes the validation of `%hoon` files. This allows any
236−file to depend on any other file of any mark, as long as there are no
237−cyclic dependencies. The Ford build system, as of this update, now
238−verifies that there are no cyclic dependencies as it runs.
240−Since building a file is a pure function, Clay memoizes the results of
241−all builds, including builds of marks, mark conversions, and hoon source
242−files. These memoization results are stored along with the desk and are
243−used by later revisions of that desk. Future work should allow memoized
244−builds to be pulled from all parents in a merge commit, but for now only
245−the previous revision of the current desk is used. This is a major
246−simplification of previous Ford architectures, which maintained much
247−more complex caches with less clear eviction semantics. Now on every
248−commit, we just throw away any unused memoized builds from the previous
249−revision's Ford cache.
251−Once Clay has validated every file in this new revision of a desk, it
252−constructs and sends updates to any subscriptions that other vanes or
253−agents have requested. More Ford builds may be run to fulfill these
254−requests, including builds for any running agents whose dependencies
255−changed in this commit.
257−When Gall receives a newly rebuilt agent from Clay, it calls the gate
258−produced by the `+on-load` arm of the new agent with the state extracted
259−from the old agent. If there is a crash in any `+on-load` calls or in
260−the handling of any effects they emit (which can include more agent
261−activations), then the whole event crashes, canceling the commit. This
262−effectively gives any agent the ability to abort a commit by crashing.
264−It is a bit counterintuitive that an app reload failure could prevent a
130+It is a bit counterintuitive that an app reload failure could prevent a kernel update. The reason is that we don't want the system to update itself into a broken state. An Urbit can be rendered practically unusable by the presence of broken agents, even if the kernel hasn't lost integrity,
265−kernel update. The reason is that we don't want the system to update itself
266−into a broken state. An Urbit can be rendered practically unusable by
267−the presence of broken agents, even if the kernel hasn't lost integrity,
268−so it's kinder to the user not to break their agents by installing an
269−incompatible kernel update. This also puts virtuous pressure on kernel
131+so it's kinder to the user not to break their agents by installing an incompatible kernel update. This also puts virtuous pressure on kernel developers not to "break userspace", the importance of which has been insisted on for decades by Linus Torvalds, among others.
270−developers not to "break userspace", the importance of which has been
132+
271−insisted on for decades by Linus Torvalds, among others.
133+If an agent does crash a commit event that included a kernel update, the attempted commit is now trivially rolled back, and the system can deliver an error message to the user. This does not leave the system in an inconsistent or stuck state, so the user could modify the failing agent and try the kernel update again later. Supporting better workflows for keeping third-party agents up-to-date will be an important aspect of Urbit’s upcoming software distribution work.
273## Ford Build Semantics
135## Ford Build Semantics
275### The Three Types of Ford Builds: Files, Marks, and Casts
137### The Three Types of Ford Builds: Files, Marks, and Casts
277−The Ford build semantics have been simplified. There are now three
139+The Ford build semantics have been simplified. There are now three kinds of builds that Ford can perform: files, marks, and casts, all of which happen synchronously as function calls inside Clay and are available (without memoization) as scry interfaces.
278−kinds of builds that Ford can perform: files, marks, and casts, all of
140+
279−which happen synchronously as function calls inside Clay and are
141+#### File Builds
280−available (without memoization) as scry interfaces.
142+
282−File builds are the most basic form of build. A file build takes in a
283−filepath containing Ford runes and Hoon source, runs the Ford runes to
143+A file build takes in a filepath containing Ford runes and Hoon source, runs the Ford runes to perform imports, and then compiles the source, producing a `$vase`, a noun tagged with its Hoon type.
284−perform imports, and then compiles the source, producing a `$vase`, a
144+
285−noun tagged with its Hoon type. Clay exposes file builds into the scry
286−namespace with `%ca`: as an example, `.^(vase %ca
287−/~zod/home/3/lib/sole/hoon)` will build the `sole` library.
145+Clay exposes file builds into the scry namespace with `%ca`: as an example, `.^(vase %ca /~zod/home/3/lib/sole/hoon)` will build the `sole` library.
289−A mark build produces a `$dais` mark-interface core. It first performs
147+#### Mark Builds
290−a file build on the Hoon file in `/mar` that defines the mark core, then
148+
291−it does some metaprogramming to make the operations more convenient to
149+A mark build produces a `$dais` mark-interface core. It first performs a file build on the Hoon file in `/mar` that defines the mark core, then it does some metaprogramming to make the operations more convenient to use. If the raw mark core delegated revision control operations to another mark core, the mark build will also load the delegate mark core and resolve the result into the `$dais`.
292−use, possibly loading another mark core that the raw mark core specified
150+
293−as delegate for revision control operations. Clay exposes mark builds
294−into the scry namespace with `%cb`: as an example, `.^(dais:clay %cb
151+Clay exposes mark builds into the scry namespace with `%cb`: as an example, `.^(dais:clay %cb /~zod/home/3/mar/foo/hoon)` builds a `$dais` for the `%foo` mark.
295−/~zod/home/3/mar/foo/hoon)` builds a `$dais` for the `%foo` mark.
152+
297−A cast build produces a `$tube`: a gate that takes a value of one mark
154+
298−as input and converts it to a valid value of another mark or crashes. A
155+A cast build produces a `$tube`: a gate that takes a value of one mark as input and converts it to a valid value of another mark or crashes. To convert from mark `%foo` to mark `%bar`, Clay tries the following operations, in order:
299−mark can convert a value to another mark by providing an arm in its
156+
300−`+grow` core named after the destination mark, and it can convert a
157++ direct grow from `%foo`
301−value from another mark using an arm in its `+grab` core. A mark can
158++ direct grab from `%bar`
302−also chain a conversion through an intermediary using an arm in its
159++ indirect jump from `%foo` through `%qux`
303−`+jump` core or by having a `+grab` arm produce a delegate mark instead
160++ indirect grab from `%bar` through `%qux`
304−of directly defining a conversion gate. Clay exposes cast builds into
161+
162+The `%foo` mark can “grow to” `%bar` by providing an arm in its `+grow` core named `+bar`. `%bar` can convert from `%foo` using a `+foo` arm in its `+grab` core. `%foo` can also chain a conversion through an intermediary using an arm in its `+jump` core, and `%bar` can specify an “indirect grab” by having a `+grab` arm produce a delegate mark instead of directly defining a conversion gate.
305−the scry namespace with `%cc`: as an example, `.^(tube:clay %cc
164+Clay exposes cast builds into the scry namespace with `%cc`: as an example, `.^(tube:clay %cc
306−/~zod/home/3/foo/bar)` builds a `$tube` conversion gate from `%foo` to
165+/~zod/home/3/foo/bar)` builds a `$tube` conversion gate from `%foo` to `%bar`.
309### Ford Runes
167### Ford Runes
311−There are now only four Ford runes. A file can contain zero, one, or
312−many of each, but each Ford expression can only be one line, and they
169+There are now only four Ford runes. A file can contain zero, one, or many of each, but each Ford expression can only be one line, and they must be in the standard order of `/-`s, `/+`s, `/=`s, and then `/*`s.
313−must be in the standard order of `/-`s, `/+`s, `/=`s, and then `/*`s.
316/- foo, *bar, baz=qux
172/- foo, *bar, baz=qux
319−The `/-` rune imports a structures file from `/sur`. You can import it
175+The `/-` rune imports a structures file from `/sur`. You can import it as just `foo`, in which case the build result of that file (usually a core with mold definitions) will be pinned into the compilation subject with the face `foo`. If you prefix it with a `*` as in `*bar`, the result will be pinned into the subject with no face; if the structures file compiled to a core, this exposes all the arms into the namespace of the compilation subject. Finally, if you import it as `baz=qux`, the `baz` face will be applied instead of `qux`. This is similar to "import as" in other languages.
320−as just `foo`, in which case the build result of that file (usually a
321−core with mold definitions) will be pinned into the compilation subject
322−with the face `foo`. If you prefix it with a `*` as in `*bar`, the
323−result will be pinned into the subject with no face; if the structures
324−file compiled to a core, this exposes all the arms into the namespace of
325−the compilation subject. Finally, if you import it as `baz=qux`, the
326−`baz` face will be applied instead of `qux`. This is similar to "import
327−as" in other languages.
330/+ foo, *bar, baz=qux
178/+ foo, *bar, baz=qux
333−The `/+` rune imports a library file from `/lib`. Aside from the
334−different source folder, the syntax and semantics are the same as for
181+The `/+` rune imports a library file from `/lib`. Aside from the different source folder, the syntax and semantics are the same as for `/-`.
338/= clay-raw /sys/vane/clay
184/= clay-raw /sys/vane/clay
341−The `/=` rune imports the result of building a hoon file from a
187+The `/=` rune imports the result of building a hoon file from a user-specified path (the second argument), wrapping it in a face specified by the first argument. The final `/hoon` at the end of the path must be omitted. This is mostly useful for importing a file for testing. The file at the specified path will be built as a normal userspace hoon file; i.e. its compilation subject will be Zuse augmented with the results of any Ford runes it has at the top of the file.
342−user-specified path (the second argument), wrapping it in a face
343−specified by the first argument. The final `/hoon` at the end of the
344−path must be omitted. This is mostly useful for importing a file for
345−testing. The file at the specified path will be built as a normal
346−userspace hoon file; i.e. its compilation subject will be Zuse augmented
347−with the results of any Ford runes it has at the top of the file.
350/* hello-gen %hoon /gen/hello/hoon
190/* hello-gen %hoon /gen/hello/hoon
353−The `/*` rune imports the contents of a file in the desk, specified as
193+The `/*` rune imports the contents of a file in the desk, specified as the third argument with the full path including the trailing mark, converted to the mark specified by the second argument, and pinned into the compilation subject wrapped in the face specified by the first argument. This can be used to import static data at build-time, such as a data file, a media file, or, in the case of this example, a hoon file as source text rather than already built.
354−the third argument with the full path including the trailing mark,
194+
355−converted to the mark specified by the second argument, and pinned into
195+A valid userspace hoon file must contain a nonempty list of hoons (hoon source expressions) below the Ford runes, separated by gap (more than one space, or at least one newline). The system wraps this list of hoons in a `=~` expression so that the result of the previous hoon is used as the subject of the next hoon. The result of the Ford runes is used as the compilation subject for this `=~` hoon; informally, the shape of the compilation subject can be thought of as:
356−the compilation subject wrapped in the face specified by the first
196+
357−argument. This can be used to import static data at build-time, such as
358−a data file, a media file, or, in the case of this example, a hoon file
359−as source text rather than already built.
361−A valid userspace hoon file must contain a nonempty list of hoons (hoon
362−source expressions) below the Ford runes, separated by gap (more than
363−one space, or at least one newline). The system wraps this list of
364−hoons in a `=~` expression so that the result of the previous hoon is
365−used as the subject of the next hoon. The result of the Ford runes is
366−used as the compilation subject for this `=~` hoon; informally, the
367−shape of the compilation subject can be thought of as:
369:* fastar-2 fastar-1
198:* fastar-2 fastar-1
370 fastis-2 fastis-1
199 fastis-2 fastis-1
@@ -374,26 +203,24 @@ shape of the compilation subject can be thought of as:
208+Urbit still needs to make better use of desks other than `%home` and the development process should be adjusted given the tighter coupling between source code and kernel and tighter criteria for accepting an update.
210+This work also hopefully provides a good foundation of a package management and software distribution system for Urbit. As `~wicdev-wisryt` has said, a user should be able to run `|install
211+~norsyr-torryn/canvas` to load and build remote source. No one should experience dependency hell on Urbit, but we're not there yet.
213+At least now, building a desk has no dependencies, other than a Ford with a compatible Hoon compiler. No decisions have been made on this yet, but
214+Ford might get moved to inside the desk, possibly by making Zuse callable. This could allow a desk to expose a Nock interface in addition to a typed Hoon interface, which could even let a desk be used as a "pill" bootloader.
377## Conclusion
216## Conclusion
379−Since deployment of Ford Fusion to the livenet a week ago, over-the-air
218+`~littel-ponnys` and I spent most of 2018 rewriting Ford with the intent of improving its performance. Compared to its predecessor, its result was better in some ways but worse in others. The caching system was labyrinthine and poorly factored, making the system difficult to debug or prove correct, even informally. Some things were faster, but the caching and dependency tracking were actually complex enough that a number of common operations, like mark conversion, were too slow.
380−updates (OTAs) have been much smoother. Instead of taking several
219+
381−hours, frequently using too much memory, and sometimes leaving ships in
220+In early 2020, `~master-morzod` suggested moving Ford into Clay to reduce asynchronicity. It seemed absurd at first, but at some point I realized I could combine that idea with a simpler build-caching scheme and self-contained desk builds, and `~wicdev-wisryt` realized he could use that to further simplify Clay’s commit and merge code, which he did as part of this project.
382−inconsistent states, multiple OTAs have been pushed out since, including
221+
383−kernelspace changes, and most users didn't even notice.
222+The first time I rewrote Ford, it took me six months, with help from `~littel-ponnys`, and it weighed in at 6,000 lines of code. The second time, in late 2018, took a few weeks. The third time, in January 2020, took a week. I wrote `+ford` in Ford Fusion in one long day, and it’s about 500 lines of synchronous, functional code.
385−Future work will entail making better use of desks other than `%home`
224+It has taken me two or three years to understand this problem as well as I do, and I expect there are parts of it I still don’t understand. The code itself isn’t the issue; it’s finding the right answer to ontological and teleological questions. What *is* Ford? What will it be in a hundred years? I’m confident Ford Fusion is more similar than its predecessor to the Ford of 2120, because it’s smaller, more functional, and easier to understand and administer.
386−and improving the development process given the tighter coupling between
225+
387−source code and kernel and tighter criteria for accepting an update.
226+As an engineering discipline and organizational practice, working on a system intended to be frozen yields surprising simplifications like this every so often. Urbit is now reaching the point where we’re starting to see more of the obsidian edges of the frozen future system emerge from the lava.
389−This work also hopefully provides a good foundation of a package
390−management and software distribution system for Urbit. As
391−`~wicdev-wisryt` has said, a user should be able to run `|install
392−~norsyr-torryn/canvas` to load and build remote source. No one should
393−experience dependency hell on Urbit, but we're not there yet. At least
394−now, building a desk has no dependencies, other than a Ford with a
395−compatible Hoon compiler. No decisions have been made on this yet, but
396−Ford might get moved to inside the desk, possibly by making Zuse
397−callable. This could allow a desk to expose a Nock interface in
398−addition to a typed Hoon interface, which could even let a desk be used
399−as a "pill" bootloader.