@@ -0,0 +1,1012 @@
1++++
2+title = "Toward a New %clay"
3+date = "2016-07-13"
4+description = "Urbit's revision-control system, %clay, is itself due for a (medium-sized) revision!"
5+aliases = [
6+ "/posts/essays/toward-a-new-clay",
7+ "/posts/toward-a-new-clay",
8+ "/blog/clay"
9+]
10+
11+[extra]
12+author = "Curtis Yarvin"
13+ship = "~sorreg-namtyv"
14++++
15+
16+TLDR: (a) Urbit's revision-control system, `%clay`, is itself due
17+for a (medium-sized) revision! (b) this time, you yourself can
18+really help! (c) please don't give away the buried spoiler!
19+
20+This explanation does not (or at least should not) require you to
21+know anything about Urbit.
22+
23+### But %clay is already pretty cool
24+
25+Of course, nothing in Urbit is allowed to be _lame_. `%clay` is
26+actually quite cool. It does need some work. But first, let's
27+talk about what it is and why it's cool.
28+
29+`%clay` is the Arvo vane (Urbit kernel module) for revision
30+control. It's an authenticated, human-memorable global immutable
31+namespace which is also a typed revision-control system. It's
32+also very good at subscription and synchronization. Urbit
33+currently uses `%clay` to distribute live hotpatches; we
34+transparently upgrade the whole network with one checkin.
35+
36+### Compared to other revision-control systems
37+
38+`%clay` is clearly a vanilla DVCS in the modern `git` style. Of
39+deployed DVCS these days, `%clay` is probably closest in spirit
40+to `bzr` or maybe `fossil`.
41+
42+Obviously, as a part of a larger whole, you would expect any
43+Urbit RCS to be brutally spartan. `%clay` is. It's even more
44+spartan than `bzr`.
45+
46+(In fact, it's really too spartan to fully use at the moment. We
47+ourselves still use `%clay` mainly as a distribution system, and
48+do all internal development on GitHub. This is because the
49+present `%clay` lacks a few little conveniences, like commit
50+messages. See upgrades, below.)
51+
52+There are lots of normal DVCS things in `%clay`. But there are
53+also cool and unusual ones. Let's take a brief tour of the
54+high-level system, then describe what we think should change
55+about it.
56+
57+### Anatomy of a DVCS
58+
59+A DVCS is a historical state graph. A point in this graph is a
60+snapshot of some tree-structured system, like a filesystem.
61+
62+These snapshots, including recursive internal nodes, are stored in
63+a content-addressable store. Pointers in this store are used to
64+construct a revision graph, whose edges are edit and/or merge
65+actions. Some sort of namespace attaches meaningful names to
66+current revisions in this graph.
67+
68+There are six big questions in the design of a DVCS:
69+
70+- how are snapshots named?
71+- what is the data structure within a snapshot?
72+- how does it map to the Unix filesystem?
73+- how is the system queried?
74+- how is the revision graph defined?
75+- how does the content-addressable store work?
76+
77+## Anatomy of %clay
78+
79+A good way to summarize `%clay` (at present) is that its answers
80+to the first four questions are interesting, and its answers to
81+the last two are boring. We'll talk about all six, though.
82+
83+### Global namespace: [ship desk case]
84+
85+The permanent global name of a `%clay` snapshot is a `++beak`. A beak
86+is a triple: `[ship desk case]`.
87+
88+The `++ship` is an Urbit ship. A ship is a network address, a
89+human-memorable name and a cryptographic identity. The address
90+width is variable; shorter addresses are more desirable. Most
91+users have 32-bit addresses, which we map to four-syllable
92+synthetic names like `~sorreg-namtyv`.
93+
94+The `++desk` is a 4-letter word meaning "branch."
95+
96+The `++case` is a version, one of three kinds: a version sequence
97+number, a named label, or a date.
98+
99+In the Urbit path syntax, the beak is the first three segments of
100+the path: `/ship/desk/case/my/cat/picture`.
101+
102+In most cases, a path parser has a default ship, desk and case,
103+each of which can be replaced by `=`. So `/=home=/my/cat/picture`
104+is the data at `/my/cat/picture` on your own ship, in the `%home`
105+desk, right now. Immutability doesn't have to be inconvenient.
106+
107+`%clay` has no concept of a mutable name or a magic `HEAD`. Use
108+the current date. If you're referring to remote data and you're
109+worried about clock skew, subtract a second or three, or be
110+prepared to block. (Or keep your clock unskewed, which shouldn't
111+be _that_ hard.) It's not like mutable-name schemes work any
112+better than this anyway.
113+
114+(Immutability is central to `%clay`. `git` merely adopted the
115+reflog. `%clay` was born in it, molded by it...)
116+
117+`%clay` is objective; each Urbit node hosts only one ship, but
118+treats domestic and foreign data (almost) the same way. If you
119+ask your `%clay` repository for someone else's data, it will
120+block the request until it gets the data, which it then caches.
121+At present, we always fulfill requests from the origin ship,
122+though in future we could certainly use it as a tracker.
123+
124+If `%clay` discards cached data, it keeps a hash, so that your
125+urbit can't violate the local illusion of immutability. We
126+can't, of course, prevent anyone else's ship from sending
127+conflicting claims in different directions. But we can prevent
128+programs on our own ship from hearing about them, meaning that we
129+retain local referential transparency.
130+
131+### Local structure: dome, arch, leaf and mark
132+
133+A `%clay` snapshot is called a `++dome`. A dome is the `%clay`
134+equivalent of a tree-structured filesystem.
135+
136+Each node in this tree is one uniform structure, an `++arch`,
137+which is both file and directory. A `++leaf` or data value in
138+this arch is a typed document.
139+
140+A logical document type is called a `++mark`. A mark is a
141+symbolic pointer to a Hoon source file, in the same dome, for an
142+engine that can validate, compare, and/or convert
143+documents of that mark.
144+
145+Let's go over arches and marks a little more closely...
146+
147+#### An arch is a uniform filesystem node
148+
149+Most filesystems and RCS have modal nodes. For instance, a git
150+data node can be a blob or a tree. `%clay` (and the Urbit
151+namespace in general) has one kind of node, an `arch`:
152+
153+```
154+ ++ arch (pair (unit leaf) (map knot arch))
155+ ++ leaf (pair mark noun)
156+```
157+
158+An arch has an optional typed-data leaf, and optional named
159+children. In Unix terms, it's a file, a directory, or both. It
160+can't be neither: an empty arch, with no data and no children, is
161+auto-pruned. As in `git` (but not in Unix), the hierarchy is a
162+pure function of its leaves. Paths exist only because of the
163+data they point to.
164+
165+The `++knot` names in an arch path are also a little special.
166+They're typed, in a sense; any knot can be parsed invertibly into
167+a typed atom. The atom is an arbitrary unsigned integer. Its
168+type (an `++aura` if you know Hoon) defines the semantics and
169+presentation of the atom: for instance, a symbol, a decimal,
170+a date, etc.
171+
172+Ultimately a filesystem is a database, and a path is a primary
173+key. It's very useful to be able to store simple data in this
174+key, in a standard and unambiguous way.
175+
176+Why uniform nodes? One, they map nicely to information trees
177+where a subtree may need some custom collective presentation,
178+not an automatically generated Apache style directory listing.
179+Two, non-uniform nodes introduce a whole pointless category of
180+node type conflicts -- for which life is far too short.
181+
182+#### A mark is an executable document type
183+
184+A mark is "like an executable MIME type." It's best defined as a
185+symbolic mapping from a filesystem to a schema engine.
186+
187+If you have a MIME type `text/html`, you need to watch the
188+Internets to track what the W3C thinks HTML is. If you have an
189+Urbit mark `%html`, you can functionally construct (using our
190+insanely powerful functional build vane `%ford`) a core which
191+can validate or normalize HTML, apply patch and diff logic, and
192+translate to and from other marks. `%ford` can even reason about
193+translation paths and construct multistep conversions.
194+
195+(Urbit does not actually have this super-smart `%html` mark right
196+now. But it probably should, huh. And this page was composed in
197+Markdown and translated to HTML by `%ford`.)
198+
199+Where does the source code of a mark live? Within the same dome
200+as the data it describes. `%html` files are described by the
201+Hoon source at `/===/mar/html`, which perhaps gets mapped into
202+your Unix filesystem at `$ship/home/mar/html.hoon`.
203+
204+This Hoon file is not self-contained. It has dependencies. The
205+`%html` mark may depend on various structures and libraries
206+elsewhere in Urbit. But the source code for all them is within
207+this dome.
208+
209+One constraint this puts on domes is that _all domes are
210+descendants of the base system dome_. Domes are versions of one
211+reality, not fragments of that reality. You can't create a new
212+empty dome -- you'd have no mark source, and be unable to install
213+any validated files.
214+
215+Enforcing the rule that all domes are congruent (if only in that
216+they all contain the base system files) costs us nothing in the
217+implementation, does not inconvenience the user, and ensures that
218+weird collisions between non-parallel things don't happen.
219+
220+### Unix mapping and mounting
221+
222+Every DVCS is its own filesystem in a sense. But every DVCS
223+works by mapping and mounting itself, in some sense, to Unix.
224+
225+#### Mapping arches to Unix
226+
227+How do we map arches into Unix? The arch structure maps 1:1 to
228+the Unix directory tree. When the arch has a leaf, we derive a
229+dot extension from the file's mark.
230+
231+So, supposing we have a leaf `foo` which is an HTML document
232+(mark `%html`), with two children `bar` and `moo`, also HTML
233+documents, in Unix we'll see:
234+
235+```
236+foo.html
237+foo/bar.html
238+foo/moo.html
239+```
240+
241+This maps invertibly back into `%clay`, and it looks perfectly
242+unremarkable from the Unix side. File completion sometimes gets
243+a little annoying.
244+
245+It would be if nice every possible Unix tree, even every
246+plausible Unix tree, mapped cleanly and completely to a `%clay`
247+dome. There are always horrid exceptions, even in common use --
248+like extensionless Makefiles. The trivial solution is that if
249+`%clay` finds a file mysterious, it won't track it.
250+
251+One often unquestioned assumption in new system software is that
252+compatibility with old content is important. It's nice, but not
253+that important. Most content in a new system is new content.
254+The important compatibility direction is export, not import. So
255+the new system should subset, not superset, the ancient formats.
256+
257+### Mounting desks to Unix
258+
259+If a `%clay` desk is _mounted_, it has a two-way live sync to
260+the Unix directory of the same name within the pier.
261+
262+When files in a mount directory change, `%clay` notices it
263+through `libuv` filesystem watching (ultimately `inotify()` or
264+`FSEvents`), computes diffs, and automatically commits them to
265+the desk. And if `%clay` creates a new version for its own
266+urbity reasons, it will revise the Unix files.
267+
268+Keeping all mounted files in the same Unix subdirectory makes it
269+easy to move the whole pier in a tarfile. Edits applied while
270+the `urbit` process isn't running will be detected and applied
271+the next time it starts.
272+
273+In conventional DVCS terms, `%clay` has a 1:1 static mapping from
274+branches to working directories. Why?
275+
276+Why not? Do we need that "what branches are checked out in what
277+directories" piece of state? Why force the user to worry about
278+which branches are mounted to which directories? Why not just
279+mount them under their own names? Is there some crippling global
280+shortage of disk space for source files? Our planet is running
281+out of inodes? Wut?
282+
283+### Queries, subscriptions and synchronization
284+
285+There are two ways to get data out of `%clay`: dereference the Urbit
286+namespace, or send a request/subscription. Sync is in userspace.
287+
288+#### Dereferencing the namespace
289+
290+`%clay` is just part of the full Urbit namespace. The full namespace
291+has a two-character prefix which expresses what vane to query
292+(always `%c` for `%clay`). The suffix is a mode flag; `%x` for
293+data, `%y` for a list of children, `%z` for the whole arch
294+subtree, `%w` for the version number. (Each vane has its own
295+modes.)
296+
297+For example, the full path to my cat picture is
298+
299+```
300+/cx/~sorreg-namtyv/home/[now]/my/cat/picture
301+```
302+
303+(where `now` is the current date).
304+
305+If the data model is `++image` (hypothetical -- we don't really
306+do image processing in Urbit right now), the normal way to load
307+this data in a Hoon program is
308+
309+```
310+.^(image /cx/~sorreg-namtyv/home/[now]/my/cat/picture)
311+```
312+
313+This produces a statically typed value (which works by
314+dynamically checking `image` against the actual type of the
315+value, but this comparison caches well).
316+
317+Internally, an Urbit namespace request produces a `++cage`, which
318+has a mark, a `++span` (language-level type), and an actual noun.
319+The product is doubly optional, `(unit (unit cage))`; this is `~`
320+if we don't know the value of this name at present but might
321+learn it in future, `[~ ~]` if we know we'll never know it, or
322+`[~ ~ value]` if we know it now.
323+
324+In high-level shell programming (`:dojo`), we can make apparently
325+"blocking" requests to `%clay`. What actually happens: when `%clay`
326+produces `~`, we stop trying to compute this expression and wait
327+on one or more blocking requests. When we are unblocked, we
328+recompute again from scratch; the intermediate computation is not
329+saved at all.
330+
331+Urbit doesn't like to block. It likes to save coherent data
332+structures, not incomplete computation contexts. When Urbit
333+pretends to block for you, this is a courtesy service. You
334+can return the courtesy of not expecting miracles from it.
335+When in doubt, send an actual request.
336+
337+### Requests and subscriptions
338+
339+A true request (local or remote) has access to the same
340+namespace, but can (a) read across case (revision) ranges,
341+and (b) filter by subtree.
342+
343+If the revision range in a request stretches into the future, the
344+request becomes a subscription. Urbit messaging always regards a
345+single response as a special case of a subscription.
346+
347+Generally, the response stream created by a request starts with
348+all the data the server has available right now; updates will
349+follow when they become available; the subscription will be
350+closed when the request range is complete.
351+
352+#### Userspace synchronization
353+
354+Synchronization per se is done in userspace. In `%clay`, sync just
355+means an application (typically the `%kiln` module in the system
356+app `:hood`) subscribes to a remote desk, and saves its change
357+stream locally. So sync is really an emergent feature of `%clay`.
358+
359+### Revision graph
360+
361+`%clay`'s revision graph is completely uninteresting at present.
362+Some changes are proposed below, however.
363+
364+As in any DVCS, a dome is the child of one parent dome (an edit)
365+or more parents (a merge). Often the parent of any revision is
366+the previous version in the sequence, but it can be anything.
367+
368+### Storage model
369+
370+`%clay`'s storage model is completely uninteresting. It's also a
371+little funky and I want to change it, so I'll be super vague.
372+
373+A `++lobe`, a 128-bit atom, is the Merkle hash of a `blob`, which
374+is essentially a leaf. All blobs are stored in a hash table
375+(`(map lobe blob)`), and reference each other through lobes.
376+
377+At present we don't GC this space at all, because we don't
378+actually delete any roots.
379+
380+A `blob` may be `%direct`, meaning it's defined as the data
381+itself, or `%delta`, defined as a change against another blob.
382+This is motivated by the desire not to store a full snapshot
383+on every file change. But it may be misguided (see changes #0
384+and #1 below).
385+
386+We cache the full arch tree (`++ankh`) of the latest state in
387+each desk. For current queries, this mitigates a lot of the
388+opportunities for slowness that this data model creates. But the
389+disparity between the speed of the current version and the
390+speed previous version is an undesirable glitch in the API.
391+
392+## Our motivation for change
393+
394+We actually have _three_ motivations for change...
395+
396+### We need to develop on %clay
397+
398+At present, we're still using `git` as a development platform.
399+`%clay` is certainly used, but just for its strength --
400+deployment. This is starving Urbit of essential experience and
401+basically keeping it lame. So, change is necessary.
402+
403+### Change must come from the community
404+
405+"Why?" you ask? "Why must change come from the community?" Well,
406+one, duh. Change always comes from the community.
407+
408+But two, a great force for change has departed. Philip Monk,
409+late maintainer of `%clay` (`~wictuc-folrex` in the old era,
410+`~wicdev-wisryt` in the new), is tired of types and typing. He
411+spent September roaming with the bison in Montana, and is now
412+literally on his way to Patagonia. On the one hand, Phil still
413+exists and is not at all unwilling to answer questions, even dumb
414+questions. On the other hand, _Patagonia_.
415+
416+So %clay needs a new owner, or owners. At this point, we think
417+the community might be indoctrinated enough to take a stab.
418+
419+Alas, `%clay` already has a pretty checkered ownership history.
420+Long ago I built a `%clay` that was really the only serious
421+design failure in Arvo. Then Jared Hance, a brilliant summer
422+intern, took my broken `%clay` and reformed it into a minimal `git`
423+in about a week. Then Philip modified Jared's `%clay` to be a
424+_typed_ revision-control system, which is a pretty cool thing.
425+
426+But... in none of these transitions did `%clay` actually get
427+_rewritten_ per se. While it's a reasonably good vane, it
428+does contain one or two historical decisions.
429+
430+We know a lot more about Arvo than we used to, and I think we can
431+revisit many of these decisions. The point of this post is to
432+start a `%clay` design discussion that will evolve into actual
433+collective action.
434+
435+### Pioneers may find it worth their time
436+
437+We're planning a rewards program for contributions to `%clay`.
438+It'll simply involve setting a time period and a pool of real
439+estate, and allocating the pool by judged contribution.
440+
441+This is an experimental program. We'll say more about it later.
442+It may not even happen. It may happen very differently. When in
443+doubt, don't talk about Fight Club.
444+
445+## Some openings for change
446+
447+Change, of course, must come from the community. No one can
448+dictate change from the top down. We _know_ this. But...
449+
450+This list of suggested changes for the next `%clay` is not in any
451+sort of order: priority, structure, difficulty, etc. Let's talk
452+about the changes first; make actual plans second. The numbers
453+here are just for convenience.
454+
455+_All changes in this program will happen on the next-generation
456+(`cc-release`) branch._
457+
458+### Change #0: state reform
459+
460+As mentioned above, the storage model in `%clay`, as the code is
461+at present, feels a little funky to me. There's nothing terribly
462+awful about it, but it doesn't feel quite right and its scaling
463+properties are very questionable.
464+
465+Briefly, `%clay` today actually derives the hierarchy from its
466+leaves -- rather than _logically_ so deriving it. Only leaves
467+are in the object store. This is just weird, and you can feel
468+`%clay` trying to be something other than what it is. (I blame
469+Jared, though it's actually my fault.)
470+
471+To store a dome snapshot (`++yaki`), the present code keeps a
472+`(map path blob)` at the root of the dome. This is enough to
473+reconstruct the `++arch` structures the namespace exports.
474+However, all kinds of basic arch operations, like basically `ls`,
475+become `O(n)` in the number of leaves in the dome. This is
476+obviously not quite right.
477+
478+I believe the storage structure should be (removing everything
479+that isn't needed to serve the namespace):
480+
481+```
482+ ++ clay (pair safe (map term desk))
483+ ++ desk (pair numb (map numb root))
484+ ++ dome (pair date root)
485+ ++ leaf (pair mark noun)
486+ ++ lobe @uvH :: 128-bit base32
487+ ++ numb @ud :: decimal version number
488+ ++ root (pair (unit leaf) (map knot lobe))
489+ ++ safe (map lobe root)
490+```
491+
492+Here a `++root` is the universal node, with children addressed by
493+hash, but logically representing the abstract `++arch`.
494+
495+Users of hierarchical filesystems basically expect performance
496+qualities that match the filesystem structure. A huge directory
497+in one place shouldn't affect search speeds somewhere else. And
498+of course, `ls` can't be O(n) in the size of the filesystem.
499+
500+### Change #1: store edges, cache snapshots
501+
502+The current `%clay` is snapshot-oriented, like `git`. The actual
503+changes are discarded. I think this is a bad choice.
504+
505+`%clay` is not a research project, and has no idea of going as
506+far as `darcs` in the "set-of-patches" direction. However, at
507+present we actually throw away the patch (`++nori`).
508+
509+I think the right DVCS design for `%clay` is a snapshot-oriented
510+system which also keeps the patches. The revision graph has the
511+same general shape as in the old `git` style. But rather than
512+just deriving the diffs from the snapshots, we can also derive
513+the snapshots from the diffs. When in doubt, we save space by
514+dropping the snapshot, not the diff.
515+
516+For an RCS which is also a general-purpose typed filesystem, it's
517+clear that considering edges (deltas, diffs, changes, etc) as the
518+primary state is optimal. You can use diff functions to derive
519+deltas from snapshots, but in many data models this will always
520+risk losing information from the original action. When in
521+doubt, save what the user did.
522+
523+This is even more true for operations on the tree structure, not
524+on individual leaves. Consider the heuristic energy that `git` has
525+to go through to figure out when `mv` has been used -- even when
526+`git mv` has been used! (According to the `git` faq, this is
527+because the lowest common denominator remains Linus' style of
528+working with emailed patches.)
529+
530+Edges probably don't even need to be deduplicated. They also can
531+contain metadata such as commit messages (which `%clay` has no
532+place for at present).
533+
534+Once edges are recognized as unreclaimed primary state, snapshots
535+can always be derived from the edges. So we can reclaim memory
536+being used for snapshots.
537+
538+It's also straightforward to have a kind of edge which is a list
539+of edges, letting us collapse feature branches trivially.
540+
541+Finally, I think references in edges should wherever possible
542+indirect through the namespace, rather than directly to the
543+content address. The user has strictly more information if you
544+say you merged `/~sorreg-namtyv/base/987/`, than if all you have
545+is the hash that this reference refers to.
546+
547+### Change #2: unconfuse merge and rebase
548+
549+The dispute between merge and rebase has all the bad smells of
550+a historical wart. But here I'm just not sure what's right.
551+
552+The user has one _intent_ whether merging or rebasing, which is
553+to combine changes on one desk into another. The _algorithm_
554+used to construct this change, whether operational transformation
555+(rebasing) or a 3-way diff (merging) has one goal: use all
556+available information to model the authors' editing intent as
557+realistically as possible. If we save edges, we should use
558+edges.
559+
560+Whether to define the resulting change graph with the typical
561+structure of a merge, or the typical structure of a rebase,
562+strikes me as quite arbitrary, and well into the land of "pick
563+one and force everyone to stick with it." But this call demands
564+a real expert on RCS workflow, which I'm not.
565+
566+Since `%clay` is a humble part of a larger system and needs to
567+keep its weight down, "do it both ways because people have
568+different preferences" is not usually the right answer.
569+
570+### Change #3: comment jihad
571+
572+`%clay` needs to be brought up to the documentation standards of
573+`%jael`. This means (a) the formatting matches; (b) the comments
574+are at the same level of quality and ubiquity.
575+
576+The basic commenting standards are: (a) every core (class) needs
577+a block comment explanation; (b) every gate (function) needs a
578+line comment; (c) every face (variable) needs a line comment.
579+
580+This is obviously not just a clerical task. To write these
581+comments, you need to actually understand the code. This is a
582+nontrivial but extremely rewarding task, even if the code you're
583+commenting will get replaced.
584+
585+Also, the _symbols_ in `%clay` must be brought up to `%jael`
586+quality. There is still far too much four-letterism here.
587+
588+NB: `%clay` is about 3000 lines of pretty clean Hoon, which is
589+certainly not at all undocumented. We're not exactly looking at
590+a Matterhorn of reverse engineering here.
591+
592+### Change #4: knot to coin internal paths
593+
594+One feature of Urbit paths, briefly mentioned above, is that a
595+path segment or `++knot` (which is just a `@tas` text string, an
596+ASCII symbol as an atom) must be within the range of the
597+invertible noun printer, `++co`.
598+
599+`++co` maps a described data value, a `++coin`, to an ASCII
600+string which is a valid URL segment and Unix filename. Its
601+friend `++so` inverts the function, perfectly.
602+
603+Almost every knot is the simplest kind of `coin`, a `dime.` A
604+dime is a single atom with a rendering type (`aura` if you know
605+Hoon). Auras include `@tas` for symbolic text, `@ud` for
606+unsigned decimal, `@p` for Urbit ship, etc. `++co` can also
607+print arbitrary cells, though less prettily.
608+
609+This feature is cool because it lets us stuff data into names in
610+a precise and predictable way. Usually the names in a namespace
611+are human-meaningful and human-originated, and are therefore
612+symbolic, but there are exceptions. For example, blogposts may
613+want to be named by date.
614+
615+However, it is slightly retarded for the fundamental state of our
616+arches to store the _rendered_ string (`++knot`), not the source
617+data (`++coin`). It leads to all kinds of inefficiencies where
618+we're doing way more parsing than we want to. (Parsing also
619+should be jet-propelled, but is not.) It also makes it way too
620+easy to sneak in bad data and make it `%clay`'s problem.
621+
622+So, logically, we'd be shifting from
623+
624+```
625+ ++ arch (pair (unit (pair mark noun)) (map knot arch))
626+```
627+
628+```
629+ ++ arch (pair (unit (pair mark noun)) (map coin arch))
630+```
631+
632+Also, a vanilla `++map` might not be quite the right data
633+structure at this point -- we probably want a custom sorted tree,
634+which separates auras and imposes a custom order on each. This
635+would let us serve simple range queries quite efficiently, so
636+long as they match the standard aura order (eg, alphabetical
637+order for symbols).
638+
639+### Change #5: dome semantics by reference
640+
641+While the principle that a dome contains its own mark source is
642+elegant, it's a pain in the ass to implement properly. `%clay`
643+certainly does not do it justice at present.
644+
645+The complexity is in checking every change to see whether it
646+includes a change to the mark source (which is very unlikely),
647+or a dependency of the mark source (slightly less unlikely).
648+And if so, do we have to update all the leaves in the dome?
649+This is also very unlikely. And yet, it happens.
650+
651+Instead, we could simply designate, by beak (ship, desk, and
652+case) a dome which contains the canonical mark source. This
653+would rarely change -- you could upgrade a desk to a new mark
654+dome, but it would be a manual operation and probably a rare one.
655+
656+The common case would just be that everyone uses the current
657+version of the `%base` desk when a desk is created, and never
658+changes it. In practice, this means not every two-byte edit
659+needs to get involved in detecting and/or applying schema
660+evolution, a welcome simplification.
661+
662+### Change #6: mark containers
663+
664+Marks in present `%clay` are atomic and symbolic (like `%html`).
665+Since they're also used for network protocols, and the proper
666+design is one mark per message structure, the mark namespace gets
667+overloaded fast.
668+
669+So we added an app hierarchy within the mark namespace, using hep
670+`-` as a divider. The `:talk` app expects messages with the mark
671+`talk-message`, whose sources is in `/===/mar/talk/message`.
672+
673+This helps, but it doesn't solve another two problems: trivial
674+nouns and containers.
675+
676+It's retarded to have to add a source file to send a `@ud` or a
677+`*`. An aura (see above under `coin`) should be a trivial mark.
678+
679+Also, we need to be able to construct standard containers of
680+marks: lists, maps, sets, and tuples (or, as a database person
681+would say, "rows").
682+
683+### Change #7: meta-marks
684+
685+Another kind of complex mark that needs to exist: conflict,
686+difference, and invalid marks.
687+
688+Conflict data and difference data are two types of noun that are
689+generated in normal RCS operation. If we have the concept of an
690+`%html` noun, we also need the concept of an invertible change
691+from one `%html` file to another; a merge conflict between two
692+`%html` files; and even a conflict between two marks.
693+
694+There are two kinds of changes to a dome: manual and automatic.
695+Manual changes are initiated by the user and should crash, unless
696+manually overridden. Automatic changes should not crash, ever,
697+and should not lose data. If they create problems, which they
698+shouldn't, these problems will need to be cleaned up manually.
699+
700+This is a complex space to navigate, but it requires a conflicted
701+merge to be able to resolve to a valid dome containing conflict
702+leaves, not crash. The present `%clay` goes halfway there and
703+resolves to a scratch desk, but we can go farther: the whole `git`
704+dumpster-fire of "merge state" deserves destruction. One change
705+is a merge that produces conflicts, the next is an edit that
706+resolves them.
707+
708+Similarly, all sorts of errors can produce invalid leaves. One
709+particularly rare case which is nonetheless important is schema
710+evolution: what happens to files in a mark, when the mark source
711+code changes. They all need to be re-normalized. And this
712+operation can fail, producing an invalid value. (We don't do any
713+of this now, BTW, although `%ford` has the dependency mechanism to
714+make it possible.) Again, in some cases this should crash the
715+update to the mark source; in other cases, there is no local way
716+to handle the crash.
717+
718+So there must be a way to preserve invalid nouns that can be
719+repaired by repair tools that expect invalid content, while not
720+actually marking them as valid so that innocent code is deluded
721+into using the bad content.
722+
723+The bottom line: a mark needs to be split into two models:
724+`++mark`, the original symbolic mark with its source engine, and
725+something like `++logo`, the meta-mark or mark container. The
726+mark is the building block; the logo is the building.
727+
728+### Change #8: filesystem hierarchy standard
729+
730+While all filesystems remain forks of one ur-filesystem, the
731+standard tends to enforce itself. But not define itself.
732+
733+I've always hated the current `/mar` for marks, for instance. I
734+prefer `/mac`. But others hate this. And so on. Also, by
735+necessity, our nascent "FHS" is heavily biased toward system
736+data, away from user data. This is retarded.
737+
738+Please focus all bikeshedding energy on this problem! Better to
739+bikeshed about what color to paint the bikeshed, then what shape
740+the bike gears should be.
741+
742+### Change #9: security and access control
743+
744+Oh, yeah, that! They always say not to design security in at the
745+last minute. Unfortunately, this is exactly what we did. There
746+is no security at all in present `%clay`. (There is also no push
747+operation in `%clay`, so it's not as mad as you think -- anyone
748+can read anything, no one can write anything, which is fine in a
749+young network where there are no secrets.)
750+
751+Fortunately, we don't have anything super fancy planned in this
752+area, just basic access-control lists. These obviously need to
753+be designed along with the filesystem-hierarchy standard (see
754+elsewhere).
755+
756+Is it even possible to do ACLs properly in `%clay`? At this
757+point, yes. There is an event-oriented security model in Arvo,
758+though it isn't really hooked up to anything at the moment.
759+There is also a privilege badge system in `%jael`. The
760+combination of these two features is quite sufficient for a
761+decent ACL model.
762+
763+The idea of event-based security is that every event passed to
764+`%clay` has a security `++mask`. The mask tells us:
765+
766+- the set of ships that has _contaminated_ this event
767+- the set of ships that has _endorsed_ this event
768+- the set of ships that this event may be _released_ to.
769+
770+To simplify this system a little and maybe explain it, the
771+security question in any event is: (a) who has _caused_ this
772+event? (b) who may receive its _effects_? (These sets never
773+include your urbit itself, of course, only neighbors.)
774+
775+For example, if we're doing something because ship X sent us a
776+packet, and we're going to leak the results of this something
777+back in a response to ship X, once `%clay` knows what our ship
778+thinks of its neighbor X, we should have a good picture of what
779+effects this event should be allowed to cause. If `%jael` knows
780+that X is our friend, `%clay` can know that X can see our cat
781+picture.
782+
783+Causality is separated out into contamination (data from ship X
784+got into the event) and endorsement (ship X signed the whole
785+event). (This may be overdesign.) All these sets can be defined
786+as the empty set (eg, releasing to the empty set means no output)
787+or the universal set (input from the universal set could be from
788+anyone, ie, is unauthenticated).
789+
790+### Change #10: stop using mark-extended paths
791+
792+When you read a leaf from a `%clay` dome, `%clay` is happy to
793+perform any necessary mark conversions (the actual work is done
794+by the `%ford` build vane). You can also just ask for the leaf
795+in whatever mark `%clay` has it.
796+
797+It's confusing to think of this mark extension as part of the
798+path. You shouldn't say `+cat /===/app/talk/hoon`. You should say
799+`+cat /===/app/talk`, or maybe `+cat %hoon /===/app/talk`.
800+
801+A mark-extended namespace may be added to `%ford`, to compensate.
802+But type-converting reads are the rare case, not the common case.
803+
804+### Change #11: real queries and indexes
805+
806+A filesystem should postpone being a database until it can no
807+longer do so. So, please don't think about this problem yet.
808+I'm joking, of course!
809+
810+### Change #12: meta-query
811+
812+Make it possible to query the list of desks in a ship :-)
813+
814+### Change #13: completely redesign Unix mounting
815+
816+This is actually a series of changes, described below. But
817+broadly...
818+
819+One of Urbit's experiments is live two-way synchronization
820+between the Unix filesystem and the revision-control store.
821+Dropbox style, as it were. I think this experiment is basically
822+a failure. It may be right for cat pictures, but I don't think
823+it's right for Urbit.
824+
825+The fundamental problem is that there are all sorts of weird,
826+flaky ways applications use a mutable filesystem. It's not at
827+all unusual for an editor to change a file by deleting the old
828+file, then writing the new one in its place.
829+
830+You really don't want this to produce a delete change. A simple
831+cooldown timer for the sync helps a lot with this, but causes
832+other weirdnesses. In theory, a sync can even cause another
833+change to the same version of the same desk. And so on.
834+
835+The thing we fear is just too funky to work (there is some
836+strange "reflux bug" still affecting it) is any sort of _two-way_
837+synchronization between Unix and Urbit.
838+
839+And what I think puts this feature into the deadpool, or at least
840+cold storage, is that there's no useful workflow that actually
841+uses it -- at least, when editing Urbit code.
842+
843+It's actually really nice to have files on a disk that nothing
844+will automatically modify. If you're using `git` and `%clay` in
845+parallel, it's nice to mount your `git` working directory. The
846+original Urbit workflow was to work out of a mounted directory,
847+but the two-way updates were just too weird and scary, so now we
848+tend to edit elsewhere and copy in -- which is lame.
849+
850+### Change #14: figure out what to do with FUSE
851+
852+We actually have two forms of filesystem access to `%clay`: read-only
853+FUSE (in a somewhat experimental state), and filesystem watching
854+(using `inotify()` and `FSEvents` under a `libuv` wrapper).
855+
856+The FUSE interface is a good one, though it needs more work.
857+Moreover, it exports the whole Urbit namespace to Unix, not just
858+`%clay`. For reading Urbit data, FUSE is basically ideal.
859+
860+The obvious impulse is to make a read-write FUSE. But I don't
861+think FUSE should be used for recording changes to an RCS, even
862+though the FUSE API sends you changes as events.
863+
864+There are three reasons for this. One, the actual filesystem
865+writes that Unix apps make are typically of low quality and
866+confusing. Some crack-brained editor will find it completely
867+normal and unremarkable to modify a byte in a file, by deleting
868+the file and rewriting it from scratch. You can use a cooldown
869+timer to wait for this nonsense to converge, but leaving data in
870+heuristic limbo seems like a very bad idea in FUSE-land.
871+
872+The second is that people derive a lot of comfort from seeing
873+actual files on an actual disk. The idea that your datas are
874+stuck in Urbit is slightly terrifying as a user experience.
875+We claim to be reliable. It's actually hard to imagine datas that
876+couldn't be recovered from either snapshot or log. But how can
877+you possibly believe us on this?
878+
879+So I think it's correct to lean all the weight of reading data
880+from Urbit on FUSE, and use filesystem watching for writing.
881+Heuristic timeouts are fine in filesystem watching, because at
882+least the data got written when the disk says it got written.
883+And at least it's on the disk.
884+
885+Also, FUSE is a daemon by definition, which forces us to think a
886+little harder about how to run the Urbit process as a daemon.
887+But this is a separate discussion.
888+
889+### Change #15: one-way, nonced working desk
890+
891+A one-way mount is a much less crazy thing. It works the same
892+way our two-way mount works today, but with only one writer: the
893+user, via Unix.
894+
895+Since a mount is one-way, it needs to have an absolute write lock
896+on the desk it's writing to. We cannot have a `%home` desk which
897+we are also editing with filesystem watching.
898+
899+Instead, we have a temporary and anonymous state that's connected
900+itself with your `%home` desk. Desk names have to be valid
901+symbols, but we can use `hep`, `-`. A reasonable choice is to
902+pick a random 32-bit name for each mount instance -- so the desk
903+name is not `%home`, but `%home-bosdut-sigder`. (Randomizing
904+instead of always using a fixed suffix, say, `%home-work`,
905+just makes all kinds of screwups less likely.)
906+
907+You create a mount simply by mkdiring the desk name in your pier.
908+Just the base name, Urbit will come up with the `%bosdut-sigder`
909+part and put it in a `.desk` file at the desk root. To forget
910+the mount, just delete this file. This avoids the ugly confusion
911+between unmounting a mount, and deleting all the data in it.
912+
913+(Urbit will keep a set of the mount desks it's used and
914+forgotten, not reuse them, and block all requests into them. By
915+blocking, we can delete without losing referential transparency.
916+Yes, Virginia, Urbit can forget!)
917+
918+Note that the auto-reload behavior of applications is extremely
919+useful when testing. When testing, you can either switch the
920+active application desk from `%home` to `%home-bosdut-sigder`,
921+or activate autosync from the latter to the former.
922+
923+### Change #16: commands as metafiles
924+
925+A new idea I'd like to play with is that, since the basic model
926+of the mount process is to read Unix files and suck their
927+contents into Urbit, you should be able to control a mounted
928+`%clay` desk through Unix filesystem operations alone.
929+
930+No Urbit command-line access, not even any `urb/` HTTP control
931+channels, should be required for the user to generate `%clay` data
932+through filesystem mount, or to set up the mount.
933+
934+An RCS experience that works entirely through Unix tools, in
935+which the only connection between content creation tools and
936+revision-control tools is shared access to the filesystem itself,
937+is essential if your RCS has something so repulsive as a daemon.
938+Otherwise, you're basically, like, reinventing Perforce.
939+
940+More concretely, the Urbit experience today requires `:hood`
941+pokes (Urbit shell commands, basically) for all nontrivial RCS
942+operations, from mounting to merging. Again, there are lots of
943+situations in which you can write into a mounted filesystem (the
944+protocol for Unix communication could be easily adapted to
945+communicate with a browser-based editor, for instance), but you
946+don't want the user to have to bother with an Urbit shell.
947+
948+Even commands like mounting and merging are easily communicated,
949+through filesystem writes that any Unix tool (including many
950+macro editors).
951+
952+This is very easy for a filesystem watcher. One such protocol:
953+if you want to send the daemon a command relative to some
954+directory, write a `.ur-command` metafile in that directory. The
955+daemon accepts the command by deleting it.
956+
957+The user experience is that you configure your mounts by simply
958+making the directories and configuring dotfiles in them. Unlike
959+in the present system, Urbit will never modify your datas
960+silently in the background.
961+
962+Here's what this looks like:
963+
964+Suppose you want to edit some data in the `%home` desk. Your
965+data is in `/=home=/mar/talk`. You also want a read-only view
966+of `/=home=/arvo`.
967+
968+From Unix, you do something like this:
969+
970+```
971+ $ cd $pier
972+ $ mkdir -p home/mar/talk
973+ $ touch home/mar/talk/.edit
974+ $ mkdir home/arvo
975+ $ touch home/arvo/.read
976+```
977+
978+Want to commit your changes? That means merging the working
979+desk, `%home-bosdut-sigder`, into the `%home`. Put a commit message
980+into `home/.commit` and it'll happen. If there's an error, it
981+will appear as `.error`.
982+
983+The mount has a .version file which simply contains the change
984+number (of the working desk) which any edits are relative to. If
985+the urbit daemon is running and watching, edits will be absorbed
986+quickly (probably after a heuristic timeout), and the version
987+incremented. But we always are confident that the edits are
988+against the latest version, or at least we can use this as a
989+sanity check. All kinds of insane things can happen to get a
990+filesystem out of consistency.
991+
992+This design obviously needs a little fleshing out, but I'm quite
993+confident it's a convenient and reliable way to drive `%clay`
994+from Unix.
995+
996+## In conclusion
997+
998+There's obviously a lot of fun here! There are two things you
999+can do right now.
1000+
1001+One, talk about `%clay` on [this forum
1002+thread](http://urbit.org/~~/fora/posts/~2016.10.25..17.37.47..6937~/) or
1003+on [:talk](http://urbit.org/stream). Both hosted on Urbit
1004+itself!
1005+
1006+Two, poke around the
1007+[code](https://github.com/urbit/arvo/blob/cc-release/arvo/clay.hoon).
1008+Make sure you're on the development branch (`cc-release`).
1009+
1010+Phasing in this work will be a fun and interesting challenge.
1011+(My suspicion is that the right first task is the Unix mount.)
1012+Let's talk...