@@ -1,1006 +0,0 @@
1−+++
2−title = "Toward a New %clay"
3−date = 2016-07-14
4−description = "Urbit's revision-control system, %clay, is itself due for a (medium-sized) revision!"
5−aliases = ["/posts/essays/toward-a-new-clay", "/posts/toward-a-new-clay", "/blog/clay"]
6−[extra]
7−author = "Curtis Yarvin"
8−ship = "~sorreg-namtyv"
9−+++
10−
11−TLDR: (a) Urbit's revision-control system, `%clay`, is itself due
12−for a (medium-sized) revision! (b) this time, you yourself can
13−really help! (c) please don't give away the buried spoiler!
14−
15−This explanation does not (or at least should not) require you to
16−know anything about Urbit.
17−
18−### But %clay is already pretty cool
19−
20−Of course, nothing in Urbit is allowed to be _lame_. `%clay` is
21−actually quite cool. It does need some work. But first, let's
22−talk about what it is and why it's cool.
23−
24−`%clay` is the Arvo vane (Urbit kernel module) for revision
25−control. It's an authenticated, human-memorable global immutable
26−namespace which is also a typed revision-control system. It's
27−also very good at subscription and synchronization. Urbit
28−currently uses `%clay` to distribute live hotpatches; we
29−transparently upgrade the whole network with one checkin.
30−
31−### Compared to other revision-control systems
32−
33−`%clay` is clearly a vanilla DVCS in the modern `git` style. Of
34−deployed DVCS these days, `%clay` is probably closest in spirit
35−to `bzr` or maybe `fossil`.
36−
37−Obviously, as a part of a larger whole, you would expect any
38−Urbit RCS to be brutally spartan. `%clay` is. It's even more
39−spartan than `bzr`.
40−
41−(In fact, it's really too spartan to fully use at the moment. We
42−ourselves still use `%clay` mainly as a distribution system, and
43−do all internal development on GitHub. This is because the
44−present `%clay` lacks a few little conveniences, like commit
45−messages. See upgrades, below.)
46−
47−There are lots of normal DVCS things in `%clay`. But there are
48−also cool and unusual ones. Let's take a brief tour of the
49−high-level system, then describe what we think should change
50−about it.
51−
52−### Anatomy of a DVCS
53−
54−A DVCS is a historical state graph. A point in this graph is a
55−snapshot of some tree-structured system, like a filesystem.
56−
57−These snapshots, including recursive internal nodes, are stored in
58−a content-addressable store. Pointers in this store are used to
59−construct a revision graph, whose edges are edit and/or merge
60−actions. Some sort of namespace attaches meaningful names to
61−current revisions in this graph.
62−
63−There are six big questions in the design of a DVCS:
64−
65−- how are snapshots named?
66−- what is the data structure within a snapshot?
67−- how does it map to the Unix filesystem?
68−- how is the system queried?
69−- how is the revision graph defined?
70−- how does the content-addressable store work?
71−
72−## Anatomy of %clay
73−
74−A good way to summarize `%clay` (at present) is that its answers
75−to the first four questions are interesting, and its answers to
76−the last two are boring. We'll talk about all six, though.
77−
78−### Global namespace: [ship desk case]
79−
80−The permanent global name of a `%clay` snapshot is a `++beak`. A beak
81−is a triple: `[ship desk case]`.
82−
83−The `++ship` is an Urbit ship. A ship is a network address, a
84−human-memorable name and a cryptographic identity. The address
85−width is variable; shorter addresses are more desirable. Most
86−users have 32-bit addresses, which we map to four-syllable
87−synthetic names like `~sorreg-namtyv`.
88−
89−The `++desk` is a 4-letter word meaning "branch."
90−
91−The `++case` is a version, one of three kinds: a version sequence
92−number, a named label, or a date.
93−
94−In the Urbit path syntax, the beak is the first three segments of
95−the path: `/ship/desk/case/my/cat/picture`.
96−
97−In most cases, a path parser has a default ship, desk and case,
98−each of which can be replaced by `=`. So `/=home=/my/cat/picture`
99−is the data at `/my/cat/picture` on your own ship, in the `%home`
100−desk, right now. Immutability doesn't have to be inconvenient.
101−
102−`%clay` has no concept of a mutable name or a magic `HEAD`. Use
103−the current date. If you're referring to remote data and you're
104−worried about clock skew, subtract a second or three, or be
105−prepared to block. (Or keep your clock unskewed, which shouldn't
106−be _that_ hard.) It's not like mutable-name schemes work any
107−better than this anyway.
108−
109−(Immutability is central to `%clay`. `git` merely adopted the
110−reflog. `%clay` was born in it, molded by it...)
111−
112−`%clay` is objective; each Urbit node hosts only one ship, but
113−treats domestic and foreign data (almost) the same way. If you
114−ask your `%clay` repository for someone else's data, it will
115−block the request until it gets the data, which it then caches.
116−At present, we always fulfill requests from the origin ship,
117−though in future we could certainly use it as a tracker.
118−
119−If `%clay` discards cached data, it keeps a hash, so that your
120−urbit can't violate the local illusion of immutability. We
121−can't, of course, prevent anyone else's ship from sending
122−conflicting claims in different directions. But we can prevent
123−programs on our own ship from hearing about them, meaning that we
124−retain local referential transparency.
125−
126−### Local structure: dome, arch, leaf and mark
127−
128−A `%clay` snapshot is called a `++dome`. A dome is the `%clay`
129−equivalent of a tree-structured filesystem.
130−
131−Each node in this tree is one uniform structure, an `++arch`,
132−which is both file and directory. A `++leaf` or data value in
133−this arch is a typed document.
134−
135−A logical document type is called a `++mark`. A mark is a
136−symbolic pointer to a Hoon source file, in the same dome, for an
137−engine that can validate, compare, and/or convert
138−documents of that mark.
139−
140−Let's go over arches and marks a little more closely...
141−
142−#### An arch is a uniform filesystem node
143−
144−Most filesystems and RCS have modal nodes. For instance, a git
145−data node can be a blob or a tree. `%clay` (and the Urbit
146−namespace in general) has one kind of node, an `arch`:
147−
148−```
149− ++ arch (pair (unit leaf) (map knot arch))
150− ++ leaf (pair mark noun)
151−```
152−
153−An arch has an optional typed-data leaf, and optional named
154−children. In Unix terms, it's a file, a directory, or both. It
155−can't be neither: an empty arch, with no data and no children, is
156−auto-pruned. As in `git` (but not in Unix), the hierarchy is a
157−pure function of its leaves. Paths exist only because of the
158−data they point to.
159−
160−The `++knot` names in an arch path are also a little special.
161−They're typed, in a sense; any knot can be parsed invertibly into
162−a typed atom. The atom is an arbitrary unsigned integer. Its
163−type (an `++aura` if you know Hoon) defines the semantics and
164−presentation of the atom: for instance, a symbol, a decimal,
165−a date, etc.
166−
167−Ultimately a filesystem is a database, and a path is a primary
168−key. It's very useful to be able to store simple data in this
169−key, in a standard and unambiguous way.
170−
171−Why uniform nodes? One, they map nicely to information trees
172−where a subtree may need some custom collective presentation,
173−not an automatically generated Apache style directory listing.
174−Two, non-uniform nodes introduce a whole pointless category of
175−node type conflicts -- for which life is far too short.
176−
177−#### A mark is an executable document type
178−
179−A mark is "like an executable MIME type." It's best defined as a
180−symbolic mapping from a filesystem to a schema engine.
181−
182−If you have a MIME type `text/html`, you need to watch the
183−Internets to track what the W3C thinks HTML is. If you have an
184−Urbit mark `%html`, you can functionally construct (using our
185−insanely powerful functional build vane `%ford`) a core which
186−can validate or normalize HTML, apply patch and diff logic, and
187−translate to and from other marks. `%ford` can even reason about
188−translation paths and construct multistep conversions.
189−
190−(Urbit does not actually have this super-smart `%html` mark right
191−now. But it probably should, huh. And this page was composed in
192−Markdown and translated to HTML by `%ford`.)
193−
194−Where does the source code of a mark live? Within the same dome
195−as the data it describes. `%html` files are described by the
196−Hoon source at `/===/mar/html`, which perhaps gets mapped into
197−your Unix filesystem at `$ship/home/mar/html.hoon`.
198−
199−This Hoon file is not self-contained. It has dependencies. The
200−`%html` mark may depend on various structures and libraries
201−elsewhere in Urbit. But the source code for all them is within
202−this dome.
203−
204−One constraint this puts on domes is that _all domes are
205−descendants of the base system dome_. Domes are versions of one
206−reality, not fragments of that reality. You can't create a new
207−empty dome -- you'd have no mark source, and be unable to install
208−any validated files.
209−
210−Enforcing the rule that all domes are congruent (if only in that
211−they all contain the base system files) costs us nothing in the
212−implementation, does not inconvenience the user, and ensures that
213−weird collisions between non-parallel things don't happen.
214−
215−### Unix mapping and mounting
216−
217−Every DVCS is its own filesystem in a sense. But every DVCS
218−works by mapping and mounting itself, in some sense, to Unix.
219−
220−#### Mapping arches to Unix
221−
222−How do we map arches into Unix? The arch structure maps 1:1 to
223−the Unix directory tree. When the arch has a leaf, we derive a
224−dot extension from the file's mark.
225−
226−So, supposing we have a leaf `foo` which is an HTML document
227−(mark `%html`), with two children `bar` and `moo`, also HTML
228−documents, in Unix we'll see:
229−
230−```
231−foo.html
232−foo/bar.html
233−foo/moo.html
234−```
235−
236−This maps invertibly back into `%clay`, and it looks perfectly
237−unremarkable from the Unix side. File completion sometimes gets
238−a little annoying.
239−
240−It would be if nice every possible Unix tree, even every
241−plausible Unix tree, mapped cleanly and completely to a `%clay`
242−dome. There are always horrid exceptions, even in common use --
243−like extensionless Makefiles. The trivial solution is that if
244−`%clay` finds a file mysterious, it won't track it.
245−
246−One often unquestioned assumption in new system software is that
247−compatibility with old content is important. It's nice, but not
248−that important. Most content in a new system is new content.
249−The important compatibility direction is export, not import. So
250−the new system should subset, not superset, the ancient formats.
251−
252−### Mounting desks to Unix
253−
254−If a `%clay` desk is _mounted_, it has a two-way live sync to
255−the Unix directory of the same name within the pier.
256−
257−When files in a mount directory change, `%clay` notices it
258−through `libuv` filesystem watching (ultimately `inotify()` or
259−`FSEvents`), computes diffs, and automatically commits them to
260−the desk. And if `%clay` creates a new version for its own
261−urbity reasons, it will revise the Unix files.
262−
263−Keeping all mounted files in the same Unix subdirectory makes it
264−easy to move the whole pier in a tarfile. Edits applied while
265−the `urbit` process isn't running will be detected and applied
266−the next time it starts.
267−
268−In conventional DVCS terms, `%clay` has a 1:1 static mapping from
269−branches to working directories. Why?
270−
271−Why not? Do we need that "what branches are checked out in what
272−directories" piece of state? Why force the user to worry about
273−which branches are mounted to which directories? Why not just
274−mount them under their own names? Is there some crippling global
275−shortage of disk space for source files? Our planet is running
276−out of inodes? Wut?
277−
278−### Queries, subscriptions and synchronization
279−
280−There are two ways to get data out of `%clay`: dereference the Urbit
281−namespace, or send a request/subscription. Sync is in userspace.
282−
283−#### Dereferencing the namespace
284−
285−`%clay` is just part of the full Urbit namespace. The full namespace
286−has a two-character prefix which expresses what vane to query
287−(always `%c` for `%clay`). The suffix is a mode flag; `%x` for
288−data, `%y` for a list of children, `%z` for the whole arch
289−subtree, `%w` for the version number. (Each vane has its own
290−modes.)
291−
292−For example, the full path to my cat picture is
293−
294−```
295−/cx/~sorreg-namtyv/home/[now]/my/cat/picture
296−```
297−
298−(where `now` is the current date).
299−
300−If the data model is `++image` (hypothetical -- we don't really
301−do image processing in Urbit right now), the normal way to load
302−this data in a Hoon program is
303−```
304−.^(image /cx/~sorreg-namtyv/home/[now]/my/cat/picture)
305−```
306−
307−This produces a statically typed value (which works by
308−dynamically checking `image` against the actual type of the
309−value, but this comparison caches well).
310−
311−Internally, an Urbit namespace request produces a `++cage`, which
312−has a mark, a `++span` (language-level type), and an actual noun.
313−The product is doubly optional, `(unit (unit cage))`; this is `~`
314−if we don't know the value of this name at present but might
315−learn it in future, `[~ ~]` if we know we'll never know it, or
316−`[~ ~ value]` if we know it now.
317−
318−In high-level shell programming (`:dojo`), we can make apparently
319−"blocking" requests to `%clay`. What actually happens: when `%clay`
320−produces `~`, we stop trying to compute this expression and wait
321−on one or more blocking requests. When we are unblocked, we
322−recompute again from scratch; the intermediate computation is not
323−saved at all.
324−
325−Urbit doesn't like to block. It likes to save coherent data
326−structures, not incomplete computation contexts. When Urbit
327−pretends to block for you, this is a courtesy service. You
328−can return the courtesy of not expecting miracles from it.
329−When in doubt, send an actual request.
330−
331−### Requests and subscriptions
332−
333−A true request (local or remote) has access to the same
334−namespace, but can (a) read across case (revision) ranges,
335−and (b) filter by subtree.
336−
337−If the revision range in a request stretches into the future, the
338−request becomes a subscription. Urbit messaging always regards a
339−single response as a special case of a subscription.
340−
341−Generally, the response stream created by a request starts with
342−all the data the server has available right now; updates will
343−follow when they become available; the subscription will be
344−closed when the request range is complete.
345−
346−#### Userspace synchronization
347−
348−Synchronization per se is done in userspace. In `%clay`, sync just
349−means an application (typically the `%kiln` module in the system
350−app `:hood`) subscribes to a remote desk, and saves its change
351−stream locally. So sync is really an emergent feature of `%clay`.
352−
353−### Revision graph
354−
355−`%clay`'s revision graph is completely uninteresting at present.
356−Some changes are proposed below, however.
357−
358−As in any DVCS, a dome is the child of one parent dome (an edit)
359−or more parents (a merge). Often the parent of any revision is
360−the previous version in the sequence, but it can be anything.
361−
362−### Storage model
363−
364−`%clay`'s storage model is completely uninteresting. It's also a
365−little funky and I want to change it, so I'll be super vague.
366−
367−A `++lobe`, a 128-bit atom, is the Merkle hash of a `blob`, which
368−is essentially a leaf. All blobs are stored in a hash table
369−(`(map lobe blob)`), and reference each other through lobes.
370−
371−At present we don't GC this space at all, because we don't
372−actually delete any roots.
373−
374−A `blob` may be `%direct`, meaning it's defined as the data
375−itself, or `%delta`, defined as a change against another blob.
376−This is motivated by the desire not to store a full snapshot
377−on every file change. But it may be misguided (see changes #0
378−and #1 below).
379−
380−We cache the full arch tree (`++ankh`) of the latest state in
381−each desk. For current queries, this mitigates a lot of the
382−opportunities for slowness that this data model creates. But the
383−disparity between the speed of the current version and the
384−speed previous version is an undesirable glitch in the API.
385−
386−## Our motivation for change
387−
388−We actually have _three_ motivations for change...
389−
390−### We need to develop on %clay
391−
392−At present, we're still using `git` as a development platform.
393−`%clay` is certainly used, but just for its strength --
394−deployment. This is starving Urbit of essential experience and
395−basically keeping it lame. So, change is necessary.
396−
397−### Change must come from the community
398−
399−"Why?" you ask? "Why must change come from the community?" Well,
400−one, duh. Change always comes from the community.
401−
402−But two, a great force for change has departed. Philip Monk,
403−late maintainer of `%clay` (`~wictuc-folrex` in the old era,
404−`~wicdev-wisryt` in the new), is tired of types and typing. He
405−spent September roaming with the bison in Montana, and is now
406−literally on his way to Patagonia. On the one hand, Phil still
407−exists and is not at all unwilling to answer questions, even dumb
408−questions. On the other hand, _Patagonia_.
409−
410−So %clay needs a new owner, or owners. At this point, we think
411−the community might be indoctrinated enough to take a stab.
412−
413−Alas, `%clay` already has a pretty checkered ownership history.
414−Long ago I built a `%clay` that was really the only serious
415−design failure in Arvo. Then Jared Hance, a brilliant summer
416−intern, took my broken `%clay` and reformed it into a minimal `git`
417−in about a week. Then Philip modified Jared's `%clay` to be a
418−_typed_ revision-control system, which is a pretty cool thing.
419−
420−But... in none of these transitions did `%clay` actually get
421−_rewritten_ per se. While it's a reasonably good vane, it
422−does contain one or two historical decisions.
423−
424−We know a lot more about Arvo than we used to, and I think we can
425−revisit many of these decisions. The point of this post is to
426−start a `%clay` design discussion that will evolve into actual
427−collective action.
428−
429−### Pioneers may find it worth their time
430−
431−We're planning a rewards program for contributions to `%clay`.
432−It'll simply involve setting a time period and a pool of real
433−estate, and allocating the pool by judged contribution.
434−
435−This is an experimental program. We'll say more about it later.
436−It may not even happen. It may happen very differently. When in
437−doubt, don't talk about Fight Club.
438−
439−## Some openings for change
440−
441−Change, of course, must come from the community. No one can
442−dictate change from the top down. We _know_ this. But...
443−
444−This list of suggested changes for the next `%clay` is not in any
445−sort of order: priority, structure, difficulty, etc. Let's talk
446−about the changes first; make actual plans second. The numbers
447−here are just for convenience.
448−
449−*All changes in this program will happen on the next-generation
450−(`cc-release`) branch.*
451−
452−### Change #0: state reform
453−
454−As mentioned above, the storage model in `%clay`, as the code is
455−at present, feels a little funky to me. There's nothing terribly
456−awful about it, but it doesn't feel quite right and its scaling
457−properties are very questionable.
458−
459−Briefly, `%clay` today actually derives the hierarchy from its
460−leaves -- rather than _logically_ so deriving it. Only leaves
461−are in the object store. This is just weird, and you can feel
462−`%clay` trying to be something other than what it is. (I blame
463−Jared, though it's actually my fault.)
464−
465−To store a dome snapshot (`++yaki`), the present code keeps a
466−`(map path blob)` at the root of the dome. This is enough to
467−reconstruct the `++arch` structures the namespace exports.
468−However, all kinds of basic arch operations, like basically `ls`,
469−become `O(n)` in the number of leaves in the dome. This is
470−obviously not quite right.
471−
472−I believe the storage structure should be (removing everything
473−that isn't needed to serve the namespace):
474−
475−```
476− ++ clay (pair safe (map term desk))
477− ++ desk (pair numb (map numb root))
478− ++ dome (pair date root)
479− ++ leaf (pair mark noun)
480− ++ lobe @uvH :: 128-bit base32
481− ++ numb @ud :: decimal version number
482− ++ root (pair (unit leaf) (map knot lobe))
483− ++ safe (map lobe root)
484−```
485−
486−Here a `++root` is the universal node, with children addressed by
487−hash, but logically representing the abstract `++arch`.
488−
489−Users of hierarchical filesystems basically expect performance
490−qualities that match the filesystem structure. A huge directory
491−in one place shouldn't affect search speeds somewhere else. And
492−of course, `ls` can't be O(n) in the size of the filesystem.
493−
494−### Change #1: store edges, cache snapshots
495−
496−The current `%clay` is snapshot-oriented, like `git`. The actual
497−changes are discarded. I think this is a bad choice.
498−
499−`%clay` is not a research project, and has no idea of going as
500−far as `darcs` in the "set-of-patches" direction. However, at
501−present we actually throw away the patch (`++nori`).
502−
503−I think the right DVCS design for `%clay` is a snapshot-oriented
504−system which also keeps the patches. The revision graph has the
505−same general shape as in the old `git` style. But rather than
506−just deriving the diffs from the snapshots, we can also derive
507−the snapshots from the diffs. When in doubt, we save space by
508−dropping the snapshot, not the diff.
509−
510−For an RCS which is also a general-purpose typed filesystem, it's
511−clear that considering edges (deltas, diffs, changes, etc) as the
512−primary state is optimal. You can use diff functions to derive
513−deltas from snapshots, but in many data models this will always
514−risk losing information from the original action. When in
515−doubt, save what the user did.
516−
517−This is even more true for operations on the tree structure, not
518−on individual leaves. Consider the heuristic energy that `git` has
519−to go through to figure out when `mv` has been used -- even when
520−`git mv` has been used! (According to the `git` faq, this is
521−because the lowest common denominator remains Linus' style of
522−working with emailed patches.)
523−
524−Edges probably don't even need to be deduplicated. They also can
525−contain metadata such as commit messages (which `%clay` has no
526−place for at present).
527−
528−Once edges are recognized as unreclaimed primary state, snapshots
529−can always be derived from the edges. So we can reclaim memory
530−being used for snapshots.
531−
532−It's also straightforward to have a kind of edge which is a list
533−of edges, letting us collapse feature branches trivially.
534−
535−Finally, I think references in edges should wherever possible
536−indirect through the namespace, rather than directly to the
537−content address. The user has strictly more information if you
538−say you merged `/~sorreg-namtyv/base/987/`, than if all you have
539−is the hash that this reference refers to.
540−
541−### Change #2: unconfuse merge and rebase
542−
543−The dispute between merge and rebase has all the bad smells of
544−a historical wart. But here I'm just not sure what's right.
545−
546−The user has one _intent_ whether merging or rebasing, which is
547−to combine changes on one desk into another. The _algorithm_
548−used to construct this change, whether operational transformation
549−(rebasing) or a 3-way diff (merging) has one goal: use all
550−available information to model the authors' editing intent as
551−realistically as possible. If we save edges, we should use
552−edges.
553−
554−Whether to define the resulting change graph with the typical
555−structure of a merge, or the typical structure of a rebase,
556−strikes me as quite arbitrary, and well into the land of "pick
557−one and force everyone to stick with it." But this call demands
558−a real expert on RCS workflow, which I'm not.
559−
560−Since `%clay` is a humble part of a larger system and needs to
561−keep its weight down, "do it both ways because people have
562−different preferences" is not usually the right answer.
563−
564−### Change #3: comment jihad
565−
566−`%clay` needs to be brought up to the documentation standards of
567−`%jael`. This means (a) the formatting matches; (b) the comments
568−are at the same level of quality and ubiquity.
569−
570−The basic commenting standards are: (a) every core (class) needs
571−a block comment explanation; (b) every gate (function) needs a
572−line comment; (c) every face (variable) needs a line comment.
573−
574−This is obviously not just a clerical task. To write these
575−comments, you need to actually understand the code. This is a
576−nontrivial but extremely rewarding task, even if the code you're
577−commenting will get replaced.
578−
579−Also, the _symbols_ in `%clay` must be brought up to `%jael`
580−quality. There is still far too much four-letterism here.
581−
582−NB: `%clay` is about 3000 lines of pretty clean Hoon, which is
583−certainly not at all undocumented. We're not exactly looking at
584−a Matterhorn of reverse engineering here.
585−
586−### Change #4: knot to coin internal paths
587−
588−One feature of Urbit paths, briefly mentioned above, is that a
589−path segment or `++knot` (which is just a `@tas` text string, an
590−ASCII symbol as an atom) must be within the range of the
591−invertible noun printer, `++co`.
592−
593−`++co` maps a described data value, a `++coin`, to an ASCII
594−string which is a valid URL segment and Unix filename. Its
595−friend `++so` inverts the function, perfectly.
596−
597−Almost every knot is the simplest kind of `coin`, a `dime.` A
598−dime is a single atom with a rendering type (`aura` if you know
599−Hoon). Auras include `@tas` for symbolic text, `@ud` for
600−unsigned decimal, `@p` for Urbit ship, etc. `++co` can also
601−print arbitrary cells, though less prettily.
602−
603−This feature is cool because it lets us stuff data into names in
604−a precise and predictable way. Usually the names in a namespace
605−are human-meaningful and human-originated, and are therefore
606−symbolic, but there are exceptions. For example, blogposts may
607−want to be named by date.
608−
609−However, it is slightly retarded for the fundamental state of our
610−arches to store the _rendered_ string (`++knot`), not the source
611−data (`++coin`). It leads to all kinds of inefficiencies where
612−we're doing way more parsing than we want to. (Parsing also
613−should be jet-propelled, but is not.) It also makes it way too
614−easy to sneak in bad data and make it `%clay`'s problem.
615−
616−So, logically, we'd be shifting from
617−
618−```
619− ++ arch (pair (unit (pair mark noun)) (map knot arch))
620−```
621−
622−```
623− ++ arch (pair (unit (pair mark noun)) (map coin arch))
624−```
625−
626−Also, a vanilla `++map` might not be quite the right data
627−structure at this point -- we probably want a custom sorted tree,
628−which separates auras and imposes a custom order on each. This
629−would let us serve simple range queries quite efficiently, so
630−long as they match the standard aura order (eg, alphabetical
631−order for symbols).
632−
633−### Change #5: dome semantics by reference
634−
635−While the principle that a dome contains its own mark source is
636−elegant, it's a pain in the ass to implement properly. `%clay`
637−certainly does not do it justice at present.
638−
639−The complexity is in checking every change to see whether it
640−includes a change to the mark source (which is very unlikely),
641−or a dependency of the mark source (slightly less unlikely).
642−And if so, do we have to update all the leaves in the dome?
643−This is also very unlikely. And yet, it happens.
644−
645−Instead, we could simply designate, by beak (ship, desk, and
646−case) a dome which contains the canonical mark source. This
647−would rarely change -- you could upgrade a desk to a new mark
648−dome, but it would be a manual operation and probably a rare one.
649−
650−The common case would just be that everyone uses the current
651−version of the `%base` desk when a desk is created, and never
652−changes it. In practice, this means not every two-byte edit
653−needs to get involved in detecting and/or applying schema
654−evolution, a welcome simplification.
655−
656−### Change #6: mark containers
657−
658−Marks in present `%clay` are atomic and symbolic (like `%html`).
659−Since they're also used for network protocols, and the proper
660−design is one mark per message structure, the mark namespace gets
661−overloaded fast.
662−
663−So we added an app hierarchy within the mark namespace, using hep
664−`-` as a divider. The `:talk` app expects messages with the mark
665−`talk-message`, whose sources is in `/===/mar/talk/message`.
666−
667−This helps, but it doesn't solve another two problems: trivial
668−nouns and containers.
669−
670−It's retarded to have to add a source file to send a `@ud` or a
671−`*`. An aura (see above under `coin`) should be a trivial mark.
672−
673−Also, we need to be able to construct standard containers of
674−marks: lists, maps, sets, and tuples (or, as a database person
675−would say, "rows").
676−
677−### Change #7: meta-marks
678−
679−Another kind of complex mark that needs to exist: conflict,
680−difference, and invalid marks.
681−
682−Conflict data and difference data are two types of noun that are
683−generated in normal RCS operation. If we have the concept of an
684−`%html` noun, we also need the concept of an invertible change
685−from one `%html` file to another; a merge conflict between two
686−`%html` files; and even a conflict between two marks.
687−
688−There are two kinds of changes to a dome: manual and automatic.
689−Manual changes are initiated by the user and should crash, unless
690−manually overridden. Automatic changes should not crash, ever,
691−and should not lose data. If they create problems, which they
692−shouldn't, these problems will need to be cleaned up manually.
693−
694−This is a complex space to navigate, but it requires a conflicted
695−merge to be able to resolve to a valid dome containing conflict
696−leaves, not crash. The present `%clay` goes halfway there and
697−resolves to a scratch desk, but we can go farther: the whole `git`
698−dumpster-fire of "merge state" deserves destruction. One change
699−is a merge that produces conflicts, the next is an edit that
700−resolves them.
701−
702−Similarly, all sorts of errors can produce invalid leaves. One
703−particularly rare case which is nonetheless important is schema
704−evolution: what happens to files in a mark, when the mark source
705−code changes. They all need to be re-normalized. And this
706−operation can fail, producing an invalid value. (We don't do any
707−of this now, BTW, although `%ford` has the dependency mechanism to
708−make it possible.) Again, in some cases this should crash the
709−update to the mark source; in other cases, there is no local way
710−to handle the crash.
711−
712−So there must be a way to preserve invalid nouns that can be
713−repaired by repair tools that expect invalid content, while not
714−actually marking them as valid so that innocent code is deluded
715−into using the bad content.
716−
717−The bottom line: a mark needs to be split into two models:
718−`++mark`, the original symbolic mark with its source engine, and
719−something like `++logo`, the meta-mark or mark container. The
720−mark is the building block; the logo is the building.
721−
722−### Change #8: filesystem hierarchy standard
723−
724−While all filesystems remain forks of one ur-filesystem, the
725−standard tends to enforce itself. But not define itself.
726−
727−I've always hated the current `/mar` for marks, for instance. I
728−prefer `/mac`. But others hate this. And so on. Also, by
729−necessity, our nascent "FHS" is heavily biased toward system
730−data, away from user data. This is retarded.
731−
732−Please focus all bikeshedding energy on this problem! Better to
733−bikeshed about what color to paint the bikeshed, then what shape
734−the bike gears should be.
735−
736−### Change #9: security and access control
737−
738−Oh, yeah, that! They always say not to design security in at the
739−last minute. Unfortunately, this is exactly what we did. There
740−is no security at all in present `%clay`. (There is also no push
741−operation in `%clay`, so it's not as mad as you think -- anyone
742−can read anything, no one can write anything, which is fine in a
743−young network where there are no secrets.)
744−
745−Fortunately, we don't have anything super fancy planned in this
746−area, just basic access-control lists. These obviously need to
747−be designed along with the filesystem-hierarchy standard (see
748−elsewhere).
749−
750−Is it even possible to do ACLs properly in `%clay`? At this
751−point, yes. There is an event-oriented security model in Arvo,
752−though it isn't really hooked up to anything at the moment.
753−There is also a privilege badge system in `%jael`. The
754−combination of these two features is quite sufficient for a
755−decent ACL model.
756−
757−The idea of event-based security is that every event passed to
758−`%clay` has a security `++mask`. The mask tells us:
759−
760−- the set of ships that has _contaminated_ this event
761−- the set of ships that has _endorsed_ this event
762−- the set of ships that this event may be _released_ to.
763−
764−To simplify this system a little and maybe explain it, the
765−security question in any event is: (a) who has _caused_ this
766−event? (b) who may receive its _effects_? (These sets never
767−include your urbit itself, of course, only neighbors.)
768−
769−For example, if we're doing something because ship X sent us a
770−packet, and we're going to leak the results of this something
771−back in a response to ship X, once `%clay` knows what our ship
772−thinks of its neighbor X, we should have a good picture of what
773−effects this event should be allowed to cause. If `%jael` knows
774−that X is our friend, `%clay` can know that X can see our cat
775−picture.
776−
777−Causality is separated out into contamination (data from ship X
778−got into the event) and endorsement (ship X signed the whole
779−event). (This may be overdesign.) All these sets can be defined
780−as the empty set (eg, releasing to the empty set means no output)
781−or the universal set (input from the universal set could be from
782−anyone, ie, is unauthenticated).
783−
784−### Change #10: stop using mark-extended paths
785−
786−When you read a leaf from a `%clay` dome, `%clay` is happy to
787−perform any necessary mark conversions (the actual work is done
788−by the `%ford` build vane). You can also just ask for the leaf
789−in whatever mark `%clay` has it.
790−
791−It's confusing to think of this mark extension as part of the
792−path. You shouldn't say `+cat /===/app/talk/hoon`. You should say
793−`+cat /===/app/talk`, or maybe `+cat %hoon /===/app/talk`.
794−
795−A mark-extended namespace may be added to `%ford`, to compensate.
796−But type-converting reads are the rare case, not the common case.
797−
798−### Change #11: real queries and indexes
799−
800−A filesystem should postpone being a database until it can no
801−longer do so. So, please don't think about this problem yet.
802−I'm joking, of course!
803−
804−### Change #12: meta-query
805−
806−Make it possible to query the list of desks in a ship :-)
807−
808−### Change #13: completely redesign Unix mounting
809−
810−This is actually a series of changes, described below. But
811−broadly...
812−
813−One of Urbit's experiments is live two-way synchronization
814−between the Unix filesystem and the revision-control store.
815−Dropbox style, as it were. I think this experiment is basically
816−a failure. It may be right for cat pictures, but I don't think
817−it's right for Urbit.
818−
819−The fundamental problem is that there are all sorts of weird,
820−flaky ways applications use a mutable filesystem. It's not at
821−all unusual for an editor to change a file by deleting the old
822−file, then writing the new one in its place.
823−
824−You really don't want this to produce a delete change. A simple
825−cooldown timer for the sync helps a lot with this, but causes
826−other weirdnesses. In theory, a sync can even cause another
827−change to the same version of the same desk. And so on.
828−
829−The thing we fear is just too funky to work (there is some
830−strange "reflux bug" still affecting it) is any sort of _two-way_
831−synchronization between Unix and Urbit.
832−
833−And what I think puts this feature into the deadpool, or at least
834−cold storage, is that there's no useful workflow that actually
835−uses it -- at least, when editing Urbit code.
836−
837−It's actually really nice to have files on a disk that nothing
838−will automatically modify. If you're using `git` and `%clay` in
839−parallel, it's nice to mount your `git` working directory. The
840−original Urbit workflow was to work out of a mounted directory,
841−but the two-way updates were just too weird and scary, so now we
842−tend to edit elsewhere and copy in -- which is lame.
843−
844−### Change #14: figure out what to do with FUSE
845−
846−We actually have two forms of filesystem access to `%clay`: read-only
847−FUSE (in a somewhat experimental state), and filesystem watching
848−(using `inotify()` and `FSEvents` under a `libuv` wrapper).
849−
850−The FUSE interface is a good one, though it needs more work.
851−Moreover, it exports the whole Urbit namespace to Unix, not just
852−`%clay`. For reading Urbit data, FUSE is basically ideal.
853−
854−The obvious impulse is to make a read-write FUSE. But I don't
855−think FUSE should be used for recording changes to an RCS, even
856−though the FUSE API sends you changes as events.
857−
858−There are three reasons for this. One, the actual filesystem
859−writes that Unix apps make are typically of low quality and
860−confusing. Some crack-brained editor will find it completely
861−normal and unremarkable to modify a byte in a file, by deleting
862−the file and rewriting it from scratch. You can use a cooldown
863−timer to wait for this nonsense to converge, but leaving data in
864−heuristic limbo seems like a very bad idea in FUSE-land.
865−
866−The second is that people derive a lot of comfort from seeing
867−actual files on an actual disk. The idea that your datas are
868−stuck in Urbit is slightly terrifying as a user experience.
869−We claim to be reliable. It's actually hard to imagine datas that
870−couldn't be recovered from either snapshot or log. But how can
871−you possibly believe us on this?
872−
873−So I think it's correct to lean all the weight of reading data
874−from Urbit on FUSE, and use filesystem watching for writing.
875−Heuristic timeouts are fine in filesystem watching, because at
876−least the data got written when the disk says it got written.
877−And at least it's on the disk.
878−
879−Also, FUSE is a daemon by definition, which forces us to think a
880−little harder about how to run the Urbit process as a daemon.
881−But this is a separate discussion.
882−
883−### Change #15: one-way, nonced working desk
884−
885−A one-way mount is a much less crazy thing. It works the same
886−way our two-way mount works today, but with only one writer: the
887−user, via Unix.
888−
889−Since a mount is one-way, it needs to have an absolute write lock
890−on the desk it's writing to. We cannot have a `%home` desk which
891−we are also editing with filesystem watching.
892−
893−Instead, we have a temporary and anonymous state that's connected
894−itself with your `%home` desk. Desk names have to be valid
895−symbols, but we can use `hep`, `-`. A reasonable choice is to
896−pick a random 32-bit name for each mount instance -- so the desk
897−name is not `%home`, but `%home-bosdut-sigder`. (Randomizing
898−instead of always using a fixed suffix, say, `%home-work`,
899−just makes all kinds of screwups less likely.)
900−
901−You create a mount simply by mkdiring the desk name in your pier.
902−Just the base name, Urbit will come up with the `%bosdut-sigder`
903−part and put it in a `.desk` file at the desk root. To forget
904−the mount, just delete this file. This avoids the ugly confusion
905−between unmounting a mount, and deleting all the data in it.
906−
907−(Urbit will keep a set of the mount desks it's used and
908−forgotten, not reuse them, and block all requests into them. By
909−blocking, we can delete without losing referential transparency.
910−Yes, Virginia, Urbit can forget!)
911−
912−Note that the auto-reload behavior of applications is extremely
913−useful when testing. When testing, you can either switch the
914−active application desk from `%home` to `%home-bosdut-sigder`,
915−or activate autosync from the latter to the former.
916−
917−### Change #16: commands as metafiles
918−
919−A new idea I'd like to play with is that, since the basic model
920−of the mount process is to read Unix files and suck their
921−contents into Urbit, you should be able to control a mounted
922−`%clay` desk through Unix filesystem operations alone.
923−
924−No Urbit command-line access, not even any `urb/` HTTP control
925−channels, should be required for the user to generate `%clay` data
926−through filesystem mount, or to set up the mount.
927−
928−An RCS experience that works entirely through Unix tools, in
929−which the only connection between content creation tools and
930−revision-control tools is shared access to the filesystem itself,
931−is essential if your RCS has something so repulsive as a daemon.
932−Otherwise, you're basically, like, reinventing Perforce.
933−
934−More concretely, the Urbit experience today requires `:hood`
935−pokes (Urbit shell commands, basically) for all nontrivial RCS
936−operations, from mounting to merging. Again, there are lots of
937−situations in which you can write into a mounted filesystem (the
938−protocol for Unix communication could be easily adapted to
939−communicate with a browser-based editor, for instance), but you
940−don't want the user to have to bother with an Urbit shell.
941−
942−Even commands like mounting and merging are easily communicated,
943−through filesystem writes that any Unix tool (including many
944−macro editors).
945−
946−This is very easy for a filesystem watcher. One such protocol:
947−if you want to send the daemon a command relative to some
948−directory, write a `.ur-command` metafile in that directory. The
949−daemon accepts the command by deleting it.
950−
951−The user experience is that you configure your mounts by simply
952−making the directories and configuring dotfiles in them. Unlike
953−in the present system, Urbit will never modify your datas
954−silently in the background.
955−
956−Here's what this looks like:
957−
958−Suppose you want to edit some data in the `%home` desk. Your
959−data is in `/=home=/mar/talk`. You also want a read-only view
960−of `/=home=/arvo`.
961−
962−From Unix, you do something like this:
963−
964−```
965− $ cd $pier
966− $ mkdir -p home/mar/talk
967− $ touch home/mar/talk/.edit
968− $ mkdir home/arvo
969− $ touch home/arvo/.read
970−```
971−
972−Want to commit your changes? That means merging the working
973−desk, `%home-bosdut-sigder`, into the `%home`. Put a commit message
974−into `home/.commit` and it'll happen. If there's an error, it
975−will appear as `.error`.
976−
977−The mount has a .version file which simply contains the change
978−number (of the working desk) which any edits are relative to. If
979−the urbit daemon is running and watching, edits will be absorbed
980−quickly (probably after a heuristic timeout), and the version
981−incremented. But we always are confident that the edits are
982−against the latest version, or at least we can use this as a
983−sanity check. All kinds of insane things can happen to get a
984−filesystem out of consistency.
985−
986−This design obviously needs a little fleshing out, but I'm quite
987−confident it's a convenient and reliable way to drive `%clay`
988−from Unix.
989−
990−## In conclusion
991−
992−There's obviously a lot of fun here! There are two things you
993−can do right now.
994−
995−One, talk about `%clay` on [this forum
996−thread](http://urbit.org/~~/fora/posts/~2016.10.25..17.37.47..6937~/) or
997−on [:talk](http://urbit.org/stream). Both hosted on Urbit
998−itself!
999−
1000−Two, poke around the
1001−[code](https://github.com/urbit/arvo/blob/cc-release/arvo/clay.hoon).
1002−Make sure you're on the development branch (`cc-release`).
1003−
1004−Phasing in this work will be a fun and interesting challenge.
1005−(My suspicion is that the right first task is the Unix mount.)
1006−Let's talk...