@@ -0,0 +1,1006 @@
1++++
2+title = "Toward a New %clay"
3+date = 2016-07-14
4+description = "Urbit's revision-control system, %clay, is itself due for a (medium-sized) revision!"
5+aliases = ["/posts/essays/toward-a-new-clay", "/posts/toward-a-new-clay", "/blog/clay"]
6+[extra]
7+author = "Curtis Yarvin"
8+ship = "~sorreg-namtyv"
9++++
10+
11+TLDR: (a) Urbit's revision-control system, `%clay`, is itself due
12+for a (medium-sized) revision! (b) this time, you yourself can
13+really help! (c) please don't give away the buried spoiler!
14+
15+This explanation does not (or at least should not) require you to
16+know anything about Urbit.
17+
18+### But %clay is already pretty cool
19+
20+Of course, nothing in Urbit is allowed to be _lame_. `%clay` is
21+actually quite cool. It does need some work. But first, let's
22+talk about what it is and why it's cool.
23+
24+`%clay` is the Arvo vane (Urbit kernel module) for revision
25+control. It's an authenticated, human-memorable global immutable
26+namespace which is also a typed revision-control system. It's
27+also very good at subscription and synchronization. Urbit
28+currently uses `%clay` to distribute live hotpatches; we
29+transparently upgrade the whole network with one checkin.
30+
31+### Compared to other revision-control systems
32+
33+`%clay` is clearly a vanilla DVCS in the modern `git` style. Of
34+deployed DVCS these days, `%clay` is probably closest in spirit
35+to `bzr` or maybe `fossil`.
36+
37+Obviously, as a part of a larger whole, you would expect any
38+Urbit RCS to be brutally spartan. `%clay` is. It's even more
39+spartan than `bzr`.
40+
41+(In fact, it's really too spartan to fully use at the moment. We
42+ourselves still use `%clay` mainly as a distribution system, and
43+do all internal development on GitHub. This is because the
44+present `%clay` lacks a few little conveniences, like commit
45+messages. See upgrades, below.)
46+
47+There are lots of normal DVCS things in `%clay`. But there are
48+also cool and unusual ones. Let's take a brief tour of the
49+high-level system, then describe what we think should change
50+about it.
51+
52+### Anatomy of a DVCS
53+
54+A DVCS is a historical state graph. A point in this graph is a
55+snapshot of some tree-structured system, like a filesystem.
56+
57+These snapshots, including recursive internal nodes, are stored in
58+a content-addressable store. Pointers in this store are used to
59+construct a revision graph, whose edges are edit and/or merge
60+actions. Some sort of namespace attaches meaningful names to
61+current revisions in this graph.
62+
63+There are six big questions in the design of a DVCS:
64+
65+- how are snapshots named?
66+- what is the data structure within a snapshot?
67+- how does it map to the Unix filesystem?
68+- how is the system queried?
69+- how is the revision graph defined?
70+- how does the content-addressable store work?
71+
72+## Anatomy of %clay
73+
74+A good way to summarize `%clay` (at present) is that its answers
75+to the first four questions are interesting, and its answers to
76+the last two are boring. We'll talk about all six, though.
77+
78+### Global namespace: [ship desk case]
79+
80+The permanent global name of a `%clay` snapshot is a `++beak`. A beak
81+is a triple: `[ship desk case]`.
82+
83+The `++ship` is an Urbit ship. A ship is a network address, a
84+human-memorable name and a cryptographic identity. The address
85+width is variable; shorter addresses are more desirable. Most
86+users have 32-bit addresses, which we map to four-syllable
87+synthetic names like `~sorreg-namtyv`.
88+
89+The `++desk` is a 4-letter word meaning "branch."
90+
91+The `++case` is a version, one of three kinds: a version sequence
92+number, a named label, or a date.
93+
94+In the Urbit path syntax, the beak is the first three segments of
95+the path: `/ship/desk/case/my/cat/picture`.
96+
97+In most cases, a path parser has a default ship, desk and case,
98+each of which can be replaced by `=`. So `/=home=/my/cat/picture`
99+is the data at `/my/cat/picture` on your own ship, in the `%home`
100+desk, right now. Immutability doesn't have to be inconvenient.
101+
102+`%clay` has no concept of a mutable name or a magic `HEAD`. Use
103+the current date. If you're referring to remote data and you're
104+worried about clock skew, subtract a second or three, or be
105+prepared to block. (Or keep your clock unskewed, which shouldn't
106+be _that_ hard.) It's not like mutable-name schemes work any
107+better than this anyway.
108+
109+(Immutability is central to `%clay`. `git` merely adopted the
110+reflog. `%clay` was born in it, molded by it...)
111+
112+`%clay` is objective; each Urbit node hosts only one ship, but
113+treats domestic and foreign data (almost) the same way. If you
114+ask your `%clay` repository for someone else's data, it will
115+block the request until it gets the data, which it then caches.
116+At present, we always fulfill requests from the origin ship,
117+though in future we could certainly use it as a tracker.
118+
119+If `%clay` discards cached data, it keeps a hash, so that your
120+urbit can't violate the local illusion of immutability. We
121+can't, of course, prevent anyone else's ship from sending
122+conflicting claims in different directions. But we can prevent
123+programs on our own ship from hearing about them, meaning that we
124+retain local referential transparency.
125+
126+### Local structure: dome, arch, leaf and mark
127+
128+A `%clay` snapshot is called a `++dome`. A dome is the `%clay`
129+equivalent of a tree-structured filesystem.
130+
131+Each node in this tree is one uniform structure, an `++arch`,
132+which is both file and directory. A `++leaf` or data value in
133+this arch is a typed document.
134+
135+A logical document type is called a `++mark`. A mark is a
136+symbolic pointer to a Hoon source file, in the same dome, for an
137+engine that can validate, compare, and/or convert
138+documents of that mark.
139+
140+Let's go over arches and marks a little more closely...
141+
142+#### An arch is a uniform filesystem node
143+
144+Most filesystems and RCS have modal nodes. For instance, a git
145+data node can be a blob or a tree. `%clay` (and the Urbit
146+namespace in general) has one kind of node, an `arch`:
147+
148+```
149+ ++ arch (pair (unit leaf) (map knot arch))
150+ ++ leaf (pair mark noun)
151+```
152+
153+An arch has an optional typed-data leaf, and optional named
154+children. In Unix terms, it's a file, a directory, or both. It
155+can't be neither: an empty arch, with no data and no children, is
156+auto-pruned. As in `git` (but not in Unix), the hierarchy is a
157+pure function of its leaves. Paths exist only because of the
158+data they point to.
159+
160+The `++knot` names in an arch path are also a little special.
161+They're typed, in a sense; any knot can be parsed invertibly into
162+a typed atom. The atom is an arbitrary unsigned integer. Its
163+type (an `++aura` if you know Hoon) defines the semantics and
164+presentation of the atom: for instance, a symbol, a decimal,
165+a date, etc.
166+
167+Ultimately a filesystem is a database, and a path is a primary
168+key. It's very useful to be able to store simple data in this
169+key, in a standard and unambiguous way.
170+
171+Why uniform nodes? One, they map nicely to information trees
172+where a subtree may need some custom collective presentation,
173+not an automatically generated Apache style directory listing.
174+Two, non-uniform nodes introduce a whole pointless category of
175+node type conflicts -- for which life is far too short.
176+
177+#### A mark is an executable document type
178+
179+A mark is "like an executable MIME type." It's best defined as a
180+symbolic mapping from a filesystem to a schema engine.
181+
182+If you have a MIME type `text/html`, you need to watch the
183+Internets to track what the W3C thinks HTML is. If you have an
184+Urbit mark `%html`, you can functionally construct (using our
185+insanely powerful functional build vane `%ford`) a core which
186+can validate or normalize HTML, apply patch and diff logic, and
187+translate to and from other marks. `%ford` can even reason about
188+translation paths and construct multistep conversions.
189+
190+(Urbit does not actually have this super-smart `%html` mark right
191+now. But it probably should, huh. And this page was composed in
192+Markdown and translated to HTML by `%ford`.)
193+
194+Where does the source code of a mark live? Within the same dome
195+as the data it describes. `%html` files are described by the
196+Hoon source at `/===/mar/html`, which perhaps gets mapped into
197+your Unix filesystem at `$ship/home/mar/html.hoon`.
198+
199+This Hoon file is not self-contained. It has dependencies. The
200+`%html` mark may depend on various structures and libraries
201+elsewhere in Urbit. But the source code for all them is within
202+this dome.
203+
204+One constraint this puts on domes is that _all domes are
205+descendants of the base system dome_. Domes are versions of one
206+reality, not fragments of that reality. You can't create a new
207+empty dome -- you'd have no mark source, and be unable to install
208+any validated files.
209+
210+Enforcing the rule that all domes are congruent (if only in that
211+they all contain the base system files) costs us nothing in the
212+implementation, does not inconvenience the user, and ensures that
213+weird collisions between non-parallel things don't happen.
214+
215+### Unix mapping and mounting
216+
217+Every DVCS is its own filesystem in a sense. But every DVCS
218+works by mapping and mounting itself, in some sense, to Unix.
219+
220+#### Mapping arches to Unix
221+
222+How do we map arches into Unix? The arch structure maps 1:1 to
223+the Unix directory tree. When the arch has a leaf, we derive a
224+dot extension from the file's mark.
225+
226+So, supposing we have a leaf `foo` which is an HTML document
227+(mark `%html`), with two children `bar` and `moo`, also HTML
228+documents, in Unix we'll see:
229+
230+```
231+foo.html
232+foo/bar.html
233+foo/moo.html
234+```
235+
236+This maps invertibly back into `%clay`, and it looks perfectly
237+unremarkable from the Unix side. File completion sometimes gets
238+a little annoying.
239+
240+It would be if nice every possible Unix tree, even every
241+plausible Unix tree, mapped cleanly and completely to a `%clay`
242+dome. There are always horrid exceptions, even in common use --
243+like extensionless Makefiles. The trivial solution is that if
244+`%clay` finds a file mysterious, it won't track it.
245+
246+One often unquestioned assumption in new system software is that
247+compatibility with old content is important. It's nice, but not
248+that important. Most content in a new system is new content.
249+The important compatibility direction is export, not import. So
250+the new system should subset, not superset, the ancient formats.
251+
252+### Mounting desks to Unix
253+
254+If a `%clay` desk is _mounted_, it has a two-way live sync to
255+the Unix directory of the same name within the pier.
256+
257+When files in a mount directory change, `%clay` notices it
258+through `libuv` filesystem watching (ultimately `inotify()` or
259+`FSEvents`), computes diffs, and automatically commits them to
260+the desk. And if `%clay` creates a new version for its own
261+urbity reasons, it will revise the Unix files.
262+
263+Keeping all mounted files in the same Unix subdirectory makes it
264+easy to move the whole pier in a tarfile. Edits applied while
265+the `urbit` process isn't running will be detected and applied
266+the next time it starts.
267+
268+In conventional DVCS terms, `%clay` has a 1:1 static mapping from
269+branches to working directories. Why?
270+
271+Why not? Do we need that "what branches are checked out in what
272+directories" piece of state? Why force the user to worry about
273+which branches are mounted to which directories? Why not just
274+mount them under their own names? Is there some crippling global
275+shortage of disk space for source files? Our planet is running
276+out of inodes? Wut?
277+
278+### Queries, subscriptions and synchronization
279+
280+There are two ways to get data out of `%clay`: dereference the Urbit
281+namespace, or send a request/subscription. Sync is in userspace.
282+
283+#### Dereferencing the namespace
284+
285+`%clay` is just part of the full Urbit namespace. The full namespace
286+has a two-character prefix which expresses what vane to query
287+(always `%c` for `%clay`). The suffix is a mode flag; `%x` for
288+data, `%y` for a list of children, `%z` for the whole arch
289+subtree, `%w` for the version number. (Each vane has its own
290+modes.)
291+
292+For example, the full path to my cat picture is
293+
294+```
295+/cx/~sorreg-namtyv/home/[now]/my/cat/picture
296+```
297+
298+(where `now` is the current date).
299+
300+If the data model is `++image` (hypothetical -- we don't really
301+do image processing in Urbit right now), the normal way to load
302+this data in a Hoon program is
303+```
304+.^(image /cx/~sorreg-namtyv/home/[now]/my/cat/picture)
305+```
306+
307+This produces a statically typed value (which works by
308+dynamically checking `image` against the actual type of the
309+value, but this comparison caches well).
310+
311+Internally, an Urbit namespace request produces a `++cage`, which
312+has a mark, a `++span` (language-level type), and an actual noun.
313+The product is doubly optional, `(unit (unit cage))`; this is `~`
314+if we don't know the value of this name at present but might
315+learn it in future, `[~ ~]` if we know we'll never know it, or
316+`[~ ~ value]` if we know it now.
317+
318+In high-level shell programming (`:dojo`), we can make apparently
319+"blocking" requests to `%clay`. What actually happens: when `%clay`
320+produces `~`, we stop trying to compute this expression and wait
321+on one or more blocking requests. When we are unblocked, we
322+recompute again from scratch; the intermediate computation is not
323+saved at all.
324+
325+Urbit doesn't like to block. It likes to save coherent data
326+structures, not incomplete computation contexts. When Urbit
327+pretends to block for you, this is a courtesy service. You
328+can return the courtesy of not expecting miracles from it.
329+When in doubt, send an actual request.
330+
331+### Requests and subscriptions
332+
333+A true request (local or remote) has access to the same
334+namespace, but can (a) read across case (revision) ranges,
335+and (b) filter by subtree.
336+
337+If the revision range in a request stretches into the future, the
338+request becomes a subscription. Urbit messaging always regards a
339+single response as a special case of a subscription.
340+
341+Generally, the response stream created by a request starts with
342+all the data the server has available right now; updates will
343+follow when they become available; the subscription will be
344+closed when the request range is complete.
345+
346+#### Userspace synchronization
347+
348+Synchronization per se is done in userspace. In `%clay`, sync just
349+means an application (typically the `%kiln` module in the system
350+app `:hood`) subscribes to a remote desk, and saves its change
351+stream locally. So sync is really an emergent feature of `%clay`.
352+
353+### Revision graph
354+
355+`%clay`'s revision graph is completely uninteresting at present.
356+Some changes are proposed below, however.
357+
358+As in any DVCS, a dome is the child of one parent dome (an edit)
359+or more parents (a merge). Often the parent of any revision is
360+the previous version in the sequence, but it can be anything.
361+
362+### Storage model
363+
364+`%clay`'s storage model is completely uninteresting. It's also a
365+little funky and I want to change it, so I'll be super vague.
366+
367+A `++lobe`, a 128-bit atom, is the Merkle hash of a `blob`, which
368+is essentially a leaf. All blobs are stored in a hash table
369+(`(map lobe blob)`), and reference each other through lobes.
370+
371+At present we don't GC this space at all, because we don't
372+actually delete any roots.
373+
374+A `blob` may be `%direct`, meaning it's defined as the data
375+itself, or `%delta`, defined as a change against another blob.
376+This is motivated by the desire not to store a full snapshot
377+on every file change. But it may be misguided (see changes #0
378+and #1 below).
379+
380+We cache the full arch tree (`++ankh`) of the latest state in
381+each desk. For current queries, this mitigates a lot of the
382+opportunities for slowness that this data model creates. But the
383+disparity between the speed of the current version and the
384+speed previous version is an undesirable glitch in the API.
385+
386+## Our motivation for change
387+
388+We actually have _three_ motivations for change...
389+
390+### We need to develop on %clay
391+
392+At present, we're still using `git` as a development platform.
393+`%clay` is certainly used, but just for its strength --
394+deployment. This is starving Urbit of essential experience and
395+basically keeping it lame. So, change is necessary.
396+
397+### Change must come from the community
398+
399+"Why?" you ask? "Why must change come from the community?" Well,
400+one, duh. Change always comes from the community.
401+
402+But two, a great force for change has departed. Philip Monk,
403+late maintainer of `%clay` (`~wictuc-folrex` in the old era,
404+`~wicdev-wisryt` in the new), is tired of types and typing. He
405+spent September roaming with the bison in Montana, and is now
406+literally on his way to Patagonia. On the one hand, Phil still
407+exists and is not at all unwilling to answer questions, even dumb
408+questions. On the other hand, _Patagonia_.
409+
410+So %clay needs a new owner, or owners. At this point, we think
411+the community might be indoctrinated enough to take a stab.
412+
413+Alas, `%clay` already has a pretty checkered ownership history.
414+Long ago I built a `%clay` that was really the only serious
415+design failure in Arvo. Then Jared Hance, a brilliant summer
416+intern, took my broken `%clay` and reformed it into a minimal `git`
417+in about a week. Then Philip modified Jared's `%clay` to be a
418+_typed_ revision-control system, which is a pretty cool thing.
419+
420+But... in none of these transitions did `%clay` actually get
421+_rewritten_ per se. While it's a reasonably good vane, it
422+does contain one or two historical decisions.
423+
424+We know a lot more about Arvo than we used to, and I think we can
425+revisit many of these decisions. The point of this post is to
426+start a `%clay` design discussion that will evolve into actual
427+collective action.
428+
429+### Pioneers may find it worth their time
430+
431+We're planning a rewards program for contributions to `%clay`.
432+It'll simply involve setting a time period and a pool of real
433+estate, and allocating the pool by judged contribution.
434+
435+This is an experimental program. We'll say more about it later.
436+It may not even happen. It may happen very differently. When in
437+doubt, don't talk about Fight Club.
438+
439+## Some openings for change
440+
441+Change, of course, must come from the community. No one can
442+dictate change from the top down. We _know_ this. But...
443+
444+This list of suggested changes for the next `%clay` is not in any
445+sort of order: priority, structure, difficulty, etc. Let's talk
446+about the changes first; make actual plans second. The numbers
447+here are just for convenience.
448+
449+*All changes in this program will happen on the next-generation
450+(`cc-release`) branch.*
451+
452+### Change #0: state reform
453+
454+As mentioned above, the storage model in `%clay`, as the code is
455+at present, feels a little funky to me. There's nothing terribly
456+awful about it, but it doesn't feel quite right and its scaling
457+properties are very questionable.
458+
459+Briefly, `%clay` today actually derives the hierarchy from its
460+leaves -- rather than _logically_ so deriving it. Only leaves
461+are in the object store. This is just weird, and you can feel
462+`%clay` trying to be something other than what it is. (I blame
463+Jared, though it's actually my fault.)
464+
465+To store a dome snapshot (`++yaki`), the present code keeps a
466+`(map path blob)` at the root of the dome. This is enough to
467+reconstruct the `++arch` structures the namespace exports.
468+However, all kinds of basic arch operations, like basically `ls`,
469+become `O(n)` in the number of leaves in the dome. This is
470+obviously not quite right.
471+
472+I believe the storage structure should be (removing everything
473+that isn't needed to serve the namespace):
474+
475+```
476+ ++ clay (pair safe (map term desk))
477+ ++ desk (pair numb (map numb root))
478+ ++ dome (pair date root)
479+ ++ leaf (pair mark noun)
480+ ++ lobe @uvH :: 128-bit base32
481+ ++ numb @ud :: decimal version number
482+ ++ root (pair (unit leaf) (map knot lobe))
483+ ++ safe (map lobe root)
484+```
485+
486+Here a `++root` is the universal node, with children addressed by
487+hash, but logically representing the abstract `++arch`.
488+
489+Users of hierarchical filesystems basically expect performance
490+qualities that match the filesystem structure. A huge directory
491+in one place shouldn't affect search speeds somewhere else. And
492+of course, `ls` can't be O(n) in the size of the filesystem.
493+
494+### Change #1: store edges, cache snapshots
495+
496+The current `%clay` is snapshot-oriented, like `git`. The actual
497+changes are discarded. I think this is a bad choice.
498+
499+`%clay` is not a research project, and has no idea of going as
500+far as `darcs` in the "set-of-patches" direction. However, at
501+present we actually throw away the patch (`++nori`).
502+
503+I think the right DVCS design for `%clay` is a snapshot-oriented
504+system which also keeps the patches. The revision graph has the
505+same general shape as in the old `git` style. But rather than
506+just deriving the diffs from the snapshots, we can also derive
507+the snapshots from the diffs. When in doubt, we save space by
508+dropping the snapshot, not the diff.
509+
510+For an RCS which is also a general-purpose typed filesystem, it's
511+clear that considering edges (deltas, diffs, changes, etc) as the
512+primary state is optimal. You can use diff functions to derive
513+deltas from snapshots, but in many data models this will always
514+risk losing information from the original action. When in
515+doubt, save what the user did.
516+
517+This is even more true for operations on the tree structure, not
518+on individual leaves. Consider the heuristic energy that `git` has
519+to go through to figure out when `mv` has been used -- even when
520+`git mv` has been used! (According to the `git` faq, this is
521+because the lowest common denominator remains Linus' style of
522+working with emailed patches.)
523+
524+Edges probably don't even need to be deduplicated. They also can
525+contain metadata such as commit messages (which `%clay` has no
526+place for at present).
527+
528+Once edges are recognized as unreclaimed primary state, snapshots
529+can always be derived from the edges. So we can reclaim memory
530+being used for snapshots.
531+
532+It's also straightforward to have a kind of edge which is a list
533+of edges, letting us collapse feature branches trivially.
534+
535+Finally, I think references in edges should wherever possible
536+indirect through the namespace, rather than directly to the
537+content address. The user has strictly more information if you
538+say you merged `/~sorreg-namtyv/base/987/`, than if all you have
539+is the hash that this reference refers to.
540+
541+### Change #2: unconfuse merge and rebase
542+
543+The dispute between merge and rebase has all the bad smells of
544+a historical wart. But here I'm just not sure what's right.
545+
546+The user has one _intent_ whether merging or rebasing, which is
547+to combine changes on one desk into another. The _algorithm_
548+used to construct this change, whether operational transformation
549+(rebasing) or a 3-way diff (merging) has one goal: use all
550+available information to model the authors' editing intent as
551+realistically as possible. If we save edges, we should use
552+edges.
553+
554+Whether to define the resulting change graph with the typical
555+structure of a merge, or the typical structure of a rebase,
556+strikes me as quite arbitrary, and well into the land of "pick
557+one and force everyone to stick with it." But this call demands
558+a real expert on RCS workflow, which I'm not.
559+
560+Since `%clay` is a humble part of a larger system and needs to
561+keep its weight down, "do it both ways because people have
562+different preferences" is not usually the right answer.
563+
564+### Change #3: comment jihad
565+
566+`%clay` needs to be brought up to the documentation standards of
567+`%jael`. This means (a) the formatting matches; (b) the comments
568+are at the same level of quality and ubiquity.
569+
570+The basic commenting standards are: (a) every core (class) needs
571+a block comment explanation; (b) every gate (function) needs a
572+line comment; (c) every face (variable) needs a line comment.
573+
574+This is obviously not just a clerical task. To write these
575+comments, you need to actually understand the code. This is a
576+nontrivial but extremely rewarding task, even if the code you're
577+commenting will get replaced.
578+
579+Also, the _symbols_ in `%clay` must be brought up to `%jael`
580+quality. There is still far too much four-letterism here.
581+
582+NB: `%clay` is about 3000 lines of pretty clean Hoon, which is
583+certainly not at all undocumented. We're not exactly looking at
584+a Matterhorn of reverse engineering here.
585+
586+### Change #4: knot to coin internal paths
587+
588+One feature of Urbit paths, briefly mentioned above, is that a
589+path segment or `++knot` (which is just a `@tas` text string, an
590+ASCII symbol as an atom) must be within the range of the
591+invertible noun printer, `++co`.
592+
593+`++co` maps a described data value, a `++coin`, to an ASCII
594+string which is a valid URL segment and Unix filename. Its
595+friend `++so` inverts the function, perfectly.
596+
597+Almost every knot is the simplest kind of `coin`, a `dime.` A
598+dime is a single atom with a rendering type (`aura` if you know
599+Hoon). Auras include `@tas` for symbolic text, `@ud` for
600+unsigned decimal, `@p` for Urbit ship, etc. `++co` can also
601+print arbitrary cells, though less prettily.
602+
603+This feature is cool because it lets us stuff data into names in
604+a precise and predictable way. Usually the names in a namespace
605+are human-meaningful and human-originated, and are therefore
606+symbolic, but there are exceptions. For example, blogposts may
607+want to be named by date.
608+
609+However, it is slightly retarded for the fundamental state of our
610+arches to store the _rendered_ string (`++knot`), not the source
611+data (`++coin`). It leads to all kinds of inefficiencies where
612+we're doing way more parsing than we want to. (Parsing also
613+should be jet-propelled, but is not.) It also makes it way too
614+easy to sneak in bad data and make it `%clay`'s problem.
615+
616+So, logically, we'd be shifting from
617+
618+```
619+ ++ arch (pair (unit (pair mark noun)) (map knot arch))
620+```
621+
622+```
623+ ++ arch (pair (unit (pair mark noun)) (map coin arch))
624+```
625+
626+Also, a vanilla `++map` might not be quite the right data
627+structure at this point -- we probably want a custom sorted tree,
628+which separates auras and imposes a custom order on each. This
629+would let us serve simple range queries quite efficiently, so
630+long as they match the standard aura order (eg, alphabetical
631+order for symbols).
632+
633+### Change #5: dome semantics by reference
634+
635+While the principle that a dome contains its own mark source is
636+elegant, it's a pain in the ass to implement properly. `%clay`
637+certainly does not do it justice at present.
638+
639+The complexity is in checking every change to see whether it
640+includes a change to the mark source (which is very unlikely),
641+or a dependency of the mark source (slightly less unlikely).
642+And if so, do we have to update all the leaves in the dome?
643+This is also very unlikely. And yet, it happens.
644+
645+Instead, we could simply designate, by beak (ship, desk, and
646+case) a dome which contains the canonical mark source. This
647+would rarely change -- you could upgrade a desk to a new mark
648+dome, but it would be a manual operation and probably a rare one.
649+
650+The common case would just be that everyone uses the current
651+version of the `%base` desk when a desk is created, and never
652+changes it. In practice, this means not every two-byte edit
653+needs to get involved in detecting and/or applying schema
654+evolution, a welcome simplification.
655+
656+### Change #6: mark containers
657+
658+Marks in present `%clay` are atomic and symbolic (like `%html`).
659+Since they're also used for network protocols, and the proper
660+design is one mark per message structure, the mark namespace gets
661+overloaded fast.
662+
663+So we added an app hierarchy within the mark namespace, using hep
664+`-` as a divider. The `:talk` app expects messages with the mark
665+`talk-message`, whose sources is in `/===/mar/talk/message`.
666+
667+This helps, but it doesn't solve another two problems: trivial
668+nouns and containers.
669+
670+It's retarded to have to add a source file to send a `@ud` or a
671+`*`. An aura (see above under `coin`) should be a trivial mark.
672+
673+Also, we need to be able to construct standard containers of
674+marks: lists, maps, sets, and tuples (or, as a database person
675+would say, "rows").
676+
677+### Change #7: meta-marks
678+
679+Another kind of complex mark that needs to exist: conflict,
680+difference, and invalid marks.
681+
682+Conflict data and difference data are two types of noun that are
683+generated in normal RCS operation. If we have the concept of an
684+`%html` noun, we also need the concept of an invertible change
685+from one `%html` file to another; a merge conflict between two
686+`%html` files; and even a conflict between two marks.
687+
688+There are two kinds of changes to a dome: manual and automatic.
689+Manual changes are initiated by the user and should crash, unless
690+manually overridden. Automatic changes should not crash, ever,
691+and should not lose data. If they create problems, which they
692+shouldn't, these problems will need to be cleaned up manually.
693+
694+This is a complex space to navigate, but it requires a conflicted
695+merge to be able to resolve to a valid dome containing conflict
696+leaves, not crash. The present `%clay` goes halfway there and
697+resolves to a scratch desk, but we can go farther: the whole `git`
698+dumpster-fire of "merge state" deserves destruction. One change
699+is a merge that produces conflicts, the next is an edit that
700+resolves them.
701+
702+Similarly, all sorts of errors can produce invalid leaves. One
703+particularly rare case which is nonetheless important is schema
704+evolution: what happens to files in a mark, when the mark source
705+code changes. They all need to be re-normalized. And this
706+operation can fail, producing an invalid value. (We don't do any
707+of this now, BTW, although `%ford` has the dependency mechanism to
708+make it possible.) Again, in some cases this should crash the
709+update to the mark source; in other cases, there is no local way
710+to handle the crash.
711+
712+So there must be a way to preserve invalid nouns that can be
713+repaired by repair tools that expect invalid content, while not
714+actually marking them as valid so that innocent code is deluded
715+into using the bad content.
716+
717+The bottom line: a mark needs to be split into two models:
718+`++mark`, the original symbolic mark with its source engine, and
719+something like `++logo`, the meta-mark or mark container. The
720+mark is the building block; the logo is the building.
721+
722+### Change #8: filesystem hierarchy standard
723+
724+While all filesystems remain forks of one ur-filesystem, the
725+standard tends to enforce itself. But not define itself.
726+
727+I've always hated the current `/mar` for marks, for instance. I
728+prefer `/mac`. But others hate this. And so on. Also, by
729+necessity, our nascent "FHS" is heavily biased toward system
730+data, away from user data. This is retarded.
731+
732+Please focus all bikeshedding energy on this problem! Better to
733+bikeshed about what color to paint the bikeshed, then what shape
734+the bike gears should be.
735+
736+### Change #9: security and access control
737+
738+Oh, yeah, that! They always say not to design security in at the
739+last minute. Unfortunately, this is exactly what we did. There
740+is no security at all in present `%clay`. (There is also no push
741+operation in `%clay`, so it's not as mad as you think -- anyone
742+can read anything, no one can write anything, which is fine in a
743+young network where there are no secrets.)
744+
745+Fortunately, we don't have anything super fancy planned in this
746+area, just basic access-control lists. These obviously need to
747+be designed along with the filesystem-hierarchy standard (see
748+elsewhere).
749+
750+Is it even possible to do ACLs properly in `%clay`? At this
751+point, yes. There is an event-oriented security model in Arvo,
752+though it isn't really hooked up to anything at the moment.
753+There is also a privilege badge system in `%jael`. The
754+combination of these two features is quite sufficient for a
755+decent ACL model.
756+
757+The idea of event-based security is that every event passed to
758+`%clay` has a security `++mask`. The mask tells us:
759+
760+- the set of ships that has _contaminated_ this event
761+- the set of ships that has _endorsed_ this event
762+- the set of ships that this event may be _released_ to.
763+
764+To simplify this system a little and maybe explain it, the
765+security question in any event is: (a) who has _caused_ this
766+event? (b) who may receive its _effects_? (These sets never
767+include your urbit itself, of course, only neighbors.)
768+
769+For example, if we're doing something because ship X sent us a
770+packet, and we're going to leak the results of this something
771+back in a response to ship X, once `%clay` knows what our ship
772+thinks of its neighbor X, we should have a good picture of what
773+effects this event should be allowed to cause. If `%jael` knows
774+that X is our friend, `%clay` can know that X can see our cat
775+picture.
776+
777+Causality is separated out into contamination (data from ship X
778+got into the event) and endorsement (ship X signed the whole
779+event). (This may be overdesign.) All these sets can be defined
780+as the empty set (eg, releasing to the empty set means no output)
781+or the universal set (input from the universal set could be from
782+anyone, ie, is unauthenticated).
783+
784+### Change #10: stop using mark-extended paths
785+
786+When you read a leaf from a `%clay` dome, `%clay` is happy to
787+perform any necessary mark conversions (the actual work is done
788+by the `%ford` build vane). You can also just ask for the leaf
789+in whatever mark `%clay` has it.
790+
791+It's confusing to think of this mark extension as part of the
792+path. You shouldn't say `+cat /===/app/talk/hoon`. You should say
793+`+cat /===/app/talk`, or maybe `+cat %hoon /===/app/talk`.
794+
795+A mark-extended namespace may be added to `%ford`, to compensate.
796+But type-converting reads are the rare case, not the common case.
797+
798+### Change #11: real queries and indexes
799+
800+A filesystem should postpone being a database until it can no
801+longer do so. So, please don't think about this problem yet.
802+I'm joking, of course!
803+
804+### Change #12: meta-query
805+
806+Make it possible to query the list of desks in a ship :-)
807+
808+### Change #13: completely redesign Unix mounting
809+
810+This is actually a series of changes, described below. But
811+broadly...
812+
813+One of Urbit's experiments is live two-way synchronization
814+between the Unix filesystem and the revision-control store.
815+Dropbox style, as it were. I think this experiment is basically
816+a failure. It may be right for cat pictures, but I don't think
817+it's right for Urbit.
818+
819+The fundamental problem is that there are all sorts of weird,
820+flaky ways applications use a mutable filesystem. It's not at
821+all unusual for an editor to change a file by deleting the old
822+file, then writing the new one in its place.
823+
824+You really don't want this to produce a delete change. A simple
825+cooldown timer for the sync helps a lot with this, but causes
826+other weirdnesses. In theory, a sync can even cause another
827+change to the same version of the same desk. And so on.
828+
829+The thing we fear is just too funky to work (there is some
830+strange "reflux bug" still affecting it) is any sort of _two-way_
831+synchronization between Unix and Urbit.
832+
833+And what I think puts this feature into the deadpool, or at least
834+cold storage, is that there's no useful workflow that actually
835+uses it -- at least, when editing Urbit code.
836+
837+It's actually really nice to have files on a disk that nothing
838+will automatically modify. If you're using `git` and `%clay` in
839+parallel, it's nice to mount your `git` working directory. The
840+original Urbit workflow was to work out of a mounted directory,
841+but the two-way updates were just too weird and scary, so now we
842+tend to edit elsewhere and copy in -- which is lame.
843+
844+### Change #14: figure out what to do with FUSE
845+
846+We actually have two forms of filesystem access to `%clay`: read-only
847+FUSE (in a somewhat experimental state), and filesystem watching
848+(using `inotify()` and `FSEvents` under a `libuv` wrapper).
849+
850+The FUSE interface is a good one, though it needs more work.
851+Moreover, it exports the whole Urbit namespace to Unix, not just
852+`%clay`. For reading Urbit data, FUSE is basically ideal.
853+
854+The obvious impulse is to make a read-write FUSE. But I don't
855+think FUSE should be used for recording changes to an RCS, even
856+though the FUSE API sends you changes as events.
857+
858+There are three reasons for this. One, the actual filesystem
859+writes that Unix apps make are typically of low quality and
860+confusing. Some crack-brained editor will find it completely
861+normal and unremarkable to modify a byte in a file, by deleting
862+the file and rewriting it from scratch. You can use a cooldown
863+timer to wait for this nonsense to converge, but leaving data in
864+heuristic limbo seems like a very bad idea in FUSE-land.
865+
866+The second is that people derive a lot of comfort from seeing
867+actual files on an actual disk. The idea that your datas are
868+stuck in Urbit is slightly terrifying as a user experience.
869+We claim to be reliable. It's actually hard to imagine datas that
870+couldn't be recovered from either snapshot or log. But how can
871+you possibly believe us on this?
872+
873+So I think it's correct to lean all the weight of reading data
874+from Urbit on FUSE, and use filesystem watching for writing.
875+Heuristic timeouts are fine in filesystem watching, because at
876+least the data got written when the disk says it got written.
877+And at least it's on the disk.
878+
879+Also, FUSE is a daemon by definition, which forces us to think a
880+little harder about how to run the Urbit process as a daemon.
881+But this is a separate discussion.
882+
883+### Change #15: one-way, nonced working desk
884+
885+A one-way mount is a much less crazy thing. It works the same
886+way our two-way mount works today, but with only one writer: the
887+user, via Unix.
888+
889+Since a mount is one-way, it needs to have an absolute write lock
890+on the desk it's writing to. We cannot have a `%home` desk which
891+we are also editing with filesystem watching.
892+
893+Instead, we have a temporary and anonymous state that's connected
894+itself with your `%home` desk. Desk names have to be valid
895+symbols, but we can use `hep`, `-`. A reasonable choice is to
896+pick a random 32-bit name for each mount instance -- so the desk
897+name is not `%home`, but `%home-bosdut-sigder`. (Randomizing
898+instead of always using a fixed suffix, say, `%home-work`,
899+just makes all kinds of screwups less likely.)
900+
901+You create a mount simply by mkdiring the desk name in your pier.
902+Just the base name, Urbit will come up with the `%bosdut-sigder`
903+part and put it in a `.desk` file at the desk root. To forget
904+the mount, just delete this file. This avoids the ugly confusion
905+between unmounting a mount, and deleting all the data in it.
906+
907+(Urbit will keep a set of the mount desks it's used and
908+forgotten, not reuse them, and block all requests into them. By
909+blocking, we can delete without losing referential transparency.
910+Yes, Virginia, Urbit can forget!)
911+
912+Note that the auto-reload behavior of applications is extremely
913+useful when testing. When testing, you can either switch the
914+active application desk from `%home` to `%home-bosdut-sigder`,
915+or activate autosync from the latter to the former.
916+
917+### Change #16: commands as metafiles
918+
919+A new idea I'd like to play with is that, since the basic model
920+of the mount process is to read Unix files and suck their
921+contents into Urbit, you should be able to control a mounted
922+`%clay` desk through Unix filesystem operations alone.
923+
924+No Urbit command-line access, not even any `urb/` HTTP control
925+channels, should be required for the user to generate `%clay` data
926+through filesystem mount, or to set up the mount.
927+
928+An RCS experience that works entirely through Unix tools, in
929+which the only connection between content creation tools and
930+revision-control tools is shared access to the filesystem itself,
931+is essential if your RCS has something so repulsive as a daemon.
932+Otherwise, you're basically, like, reinventing Perforce.
933+
934+More concretely, the Urbit experience today requires `:hood`
935+pokes (Urbit shell commands, basically) for all nontrivial RCS
936+operations, from mounting to merging. Again, there are lots of
937+situations in which you can write into a mounted filesystem (the
938+protocol for Unix communication could be easily adapted to
939+communicate with a browser-based editor, for instance), but you
940+don't want the user to have to bother with an Urbit shell.
941+
942+Even commands like mounting and merging are easily communicated,
943+through filesystem writes that any Unix tool (including many
944+macro editors).
945+
946+This is very easy for a filesystem watcher. One such protocol:
947+if you want to send the daemon a command relative to some
948+directory, write a `.ur-command` metafile in that directory. The
949+daemon accepts the command by deleting it.
950+
951+The user experience is that you configure your mounts by simply
952+making the directories and configuring dotfiles in them. Unlike
953+in the present system, Urbit will never modify your datas
954+silently in the background.
955+
956+Here's what this looks like:
957+
958+Suppose you want to edit some data in the `%home` desk. Your
959+data is in `/=home=/mar/talk`. You also want a read-only view
960+of `/=home=/arvo`.
961+
962+From Unix, you do something like this:
963+
964+```
965+ $ cd $pier
966+ $ mkdir -p home/mar/talk
967+ $ touch home/mar/talk/.edit
968+ $ mkdir home/arvo
969+ $ touch home/arvo/.read
970+```
971+
972+Want to commit your changes? That means merging the working
973+desk, `%home-bosdut-sigder`, into the `%home`. Put a commit message
974+into `home/.commit` and it'll happen. If there's an error, it
975+will appear as `.error`.
976+
977+The mount has a .version file which simply contains the change
978+number (of the working desk) which any edits are relative to. If
979+the urbit daemon is running and watching, edits will be absorbed
980+quickly (probably after a heuristic timeout), and the version
981+incremented. But we always are confident that the edits are
982+against the latest version, or at least we can use this as a
983+sanity check. All kinds of insane things can happen to get a
984+filesystem out of consistency.
985+
986+This design obviously needs a little fleshing out, but I'm quite
987+confident it's a convenient and reliable way to drive `%clay`
988+from Unix.
989+
990+## In conclusion
991+
992+There's obviously a lot of fun here! There are two things you
993+can do right now.
994+
995+One, talk about `%clay` on [this forum
996+thread](http://urbit.org/~~/fora/posts/~2016.10.25..17.37.47..6937~/) or
997+on [:talk](http://urbit.org/stream). Both hosted on Urbit
998+itself!
999+
1000+Two, poke around the
1001+[code](https://github.com/urbit/arvo/blob/cc-release/arvo/clay.hoon).
1002+Make sure you're on the development branch (`cc-release`).
1003+
1004+Phasing in this work will be a fun and interesting challenge.
1005+(My suspicion is that the right first task is the Unix mount.)
1006+Let's talk...