@@ -0,0 +1,1012 @@
2+title = "Toward a New %clay"
4+description = "Urbit's revision-control system, %clay, is itself due for a (medium-sized) revision!"
6+ "/posts/essays/toward-a-new-clay",
7+ "/posts/toward-a-new-clay",
12+author = "Curtis Yarvin"
13+ship = "~sorreg-namtyv"
16+TLDR: (a) Urbit's revision-control system, `%clay`, is itself due
17+for a (medium-sized) revision! (b) this time, you yourself can
18+really help! (c) please don't give away the buried spoiler!
20+This explanation does not (or at least should not) require you to
21+know anything about Urbit.
23+### But %clay is already pretty cool
25+Of course, nothing in Urbit is allowed to be _lame_. `%clay` is
26+actually quite cool. It does need some work. But first, let's
27+talk about what it is and why it's cool.
29+`%clay` is the Arvo vane (Urbit kernel module) for revision
30+control. It's an authenticated, human-memorable global immutable
31+namespace which is also a typed revision-control system. It's
32+also very good at subscription and synchronization. Urbit
33+currently uses `%clay` to distribute live hotpatches; we
34+transparently upgrade the whole network with one checkin.
36+### Compared to other revision-control systems
38+`%clay` is clearly a vanilla DVCS in the modern `git` style. Of
39+deployed DVCS these days, `%clay` is probably closest in spirit
40+to `bzr` or maybe `fossil`.
42+Obviously, as a part of a larger whole, you would expect any
43+Urbit RCS to be brutally spartan. `%clay` is. It's even more
46+(In fact, it's really too spartan to fully use at the moment. We
47+ourselves still use `%clay` mainly as a distribution system, and
48+do all internal development on GitHub. This is because the
49+present `%clay` lacks a few little conveniences, like commit
50+messages. See upgrades, below.)
52+There are lots of normal DVCS things in `%clay`. But there are
53+also cool and unusual ones. Let's take a brief tour of the
54+high-level system, then describe what we think should change
59+A DVCS is a historical state graph. A point in this graph is a
60+snapshot of some tree-structured system, like a filesystem.
62+These snapshots, including recursive internal nodes, are stored in
63+a content-addressable store. Pointers in this store are used to
64+construct a revision graph, whose edges are edit and/or merge
65+actions. Some sort of namespace attaches meaningful names to
66+current revisions in this graph.
68+There are six big questions in the design of a DVCS:
70+- how are snapshots named?
71+- what is the data structure within a snapshot?
72+- how does it map to the Unix filesystem?
73+- how is the system queried?
74+- how is the revision graph defined?
75+- how does the content-addressable store work?
79+A good way to summarize `%clay` (at present) is that its answers
80+to the first four questions are interesting, and its answers to
81+the last two are boring. We'll talk about all six, though.
83+### Global namespace: [ship desk case]
85+The permanent global name of a `%clay` snapshot is a `++beak`. A beak
86+is a triple: `[ship desk case]`.
88+The `++ship` is an Urbit ship. A ship is a network address, a
89+human-memorable name and a cryptographic identity. The address
90+width is variable; shorter addresses are more desirable. Most
91+users have 32-bit addresses, which we map to four-syllable
92+synthetic names like `~sorreg-namtyv`.
94+The `++desk` is a 4-letter word meaning "branch."
96+The `++case` is a version, one of three kinds: a version sequence
97+number, a named label, or a date.
99+In the Urbit path syntax, the beak is the first three segments of
100+the path: `/ship/desk/case/my/cat/picture`.
102+In most cases, a path parser has a default ship, desk and case,
103+each of which can be replaced by `=`. So `/=home=/my/cat/picture`
104+is the data at `/my/cat/picture` on your own ship, in the `%home`
105+desk, right now. Immutability doesn't have to be inconvenient.
107+`%clay` has no concept of a mutable name or a magic `HEAD`. Use
108+the current date. If you're referring to remote data and you're
109+worried about clock skew, subtract a second or three, or be
110+prepared to block. (Or keep your clock unskewed, which shouldn't
111+be _that_ hard.) It's not like mutable-name schemes work any
112+better than this anyway.
114+(Immutability is central to `%clay`. `git` merely adopted the
115+reflog. `%clay` was born in it, molded by it...)
117+`%clay` is objective; each Urbit node hosts only one ship, but
118+treats domestic and foreign data (almost) the same way. If you
119+ask your `%clay` repository for someone else's data, it will
120+block the request until it gets the data, which it then caches.
121+At present, we always fulfill requests from the origin ship,
122+though in future we could certainly use it as a tracker.
124+If `%clay` discards cached data, it keeps a hash, so that your
125+urbit can't violate the local illusion of immutability. We
126+can't, of course, prevent anyone else's ship from sending
127+conflicting claims in different directions. But we can prevent
128+programs on our own ship from hearing about them, meaning that we
129+retain local referential transparency.
131+### Local structure: dome, arch, leaf and mark
133+A `%clay` snapshot is called a `++dome`. A dome is the `%clay`
134+equivalent of a tree-structured filesystem.
136+Each node in this tree is one uniform structure, an `++arch`,
137+which is both file and directory. A `++leaf` or data value in
138+this arch is a typed document.
140+A logical document type is called a `++mark`. A mark is a
141+symbolic pointer to a Hoon source file, in the same dome, for an
142+engine that can validate, compare, and/or convert
143+documents of that mark.
145+Let's go over arches and marks a little more closely...
147+#### An arch is a uniform filesystem node
149+Most filesystems and RCS have modal nodes. For instance, a git
150+data node can be a blob or a tree. `%clay` (and the Urbit
151+namespace in general) has one kind of node, an `arch`:
154+ ++ arch (pair (unit leaf) (map knot arch))
155+ ++ leaf (pair mark noun)
158+An arch has an optional typed-data leaf, and optional named
159+children. In Unix terms, it's a file, a directory, or both. It
160+can't be neither: an empty arch, with no data and no children, is
161+auto-pruned. As in `git` (but not in Unix), the hierarchy is a
162+pure function of its leaves. Paths exist only because of the
165+The `++knot` names in an arch path are also a little special.
166+They're typed, in a sense; any knot can be parsed invertibly into
167+a typed atom. The atom is an arbitrary unsigned integer. Its
168+type (an `++aura` if you know Hoon) defines the semantics and
169+presentation of the atom: for instance, a symbol, a decimal,
172+Ultimately a filesystem is a database, and a path is a primary
173+key. It's very useful to be able to store simple data in this
174+key, in a standard and unambiguous way.
176+Why uniform nodes? One, they map nicely to information trees
177+where a subtree may need some custom collective presentation,
178+not an automatically generated Apache style directory listing.
179+Two, non-uniform nodes introduce a whole pointless category of
180+node type conflicts -- for which life is far too short.
182+#### A mark is an executable document type
184+A mark is "like an executable MIME type." It's best defined as a
185+symbolic mapping from a filesystem to a schema engine.
187+If you have a MIME type `text/html`, you need to watch the
188+Internets to track what the W3C thinks HTML is. If you have an
189+Urbit mark `%html`, you can functionally construct (using our
190+insanely powerful functional build vane `%ford`) a core which
191+can validate or normalize HTML, apply patch and diff logic, and
192+translate to and from other marks. `%ford` can even reason about
193+translation paths and construct multistep conversions.
195+(Urbit does not actually have this super-smart `%html` mark right
196+now. But it probably should, huh. And this page was composed in
197+Markdown and translated to HTML by `%ford`.)
199+Where does the source code of a mark live? Within the same dome
200+as the data it describes. `%html` files are described by the
201+Hoon source at `/===/mar/html`, which perhaps gets mapped into
202+your Unix filesystem at `$ship/home/mar/html.hoon`.
204+This Hoon file is not self-contained. It has dependencies. The
205+`%html` mark may depend on various structures and libraries
206+elsewhere in Urbit. But the source code for all them is within
209+One constraint this puts on domes is that _all domes are
210+descendants of the base system dome_. Domes are versions of one
211+reality, not fragments of that reality. You can't create a new
212+empty dome -- you'd have no mark source, and be unable to install
215+Enforcing the rule that all domes are congruent (if only in that
216+they all contain the base system files) costs us nothing in the
217+implementation, does not inconvenience the user, and ensures that
218+weird collisions between non-parallel things don't happen.
220+### Unix mapping and mounting
222+Every DVCS is its own filesystem in a sense. But every DVCS
223+works by mapping and mounting itself, in some sense, to Unix.
225+#### Mapping arches to Unix
227+How do we map arches into Unix? The arch structure maps 1:1 to
228+the Unix directory tree. When the arch has a leaf, we derive a
229+dot extension from the file's mark.
231+So, supposing we have a leaf `foo` which is an HTML document
232+(mark `%html`), with two children `bar` and `moo`, also HTML
233+documents, in Unix we'll see:
241+This maps invertibly back into `%clay`, and it looks perfectly
242+unremarkable from the Unix side. File completion sometimes gets
245+It would be if nice every possible Unix tree, even every
246+plausible Unix tree, mapped cleanly and completely to a `%clay`
247+dome. There are always horrid exceptions, even in common use --
248+like extensionless Makefiles. The trivial solution is that if
249+`%clay` finds a file mysterious, it won't track it.
251+One often unquestioned assumption in new system software is that
252+compatibility with old content is important. It's nice, but not
253+that important. Most content in a new system is new content.
254+The important compatibility direction is export, not import. So
255+the new system should subset, not superset, the ancient formats.
257+### Mounting desks to Unix
259+If a `%clay` desk is _mounted_, it has a two-way live sync to
260+the Unix directory of the same name within the pier.
262+When files in a mount directory change, `%clay` notices it
263+through `libuv` filesystem watching (ultimately `inotify()` or
264+`FSEvents`), computes diffs, and automatically commits them to
265+the desk. And if `%clay` creates a new version for its own
266+urbity reasons, it will revise the Unix files.
268+Keeping all mounted files in the same Unix subdirectory makes it
269+easy to move the whole pier in a tarfile. Edits applied while
270+the `urbit` process isn't running will be detected and applied
271+the next time it starts.
273+In conventional DVCS terms, `%clay` has a 1:1 static mapping from
274+branches to working directories. Why?
276+Why not? Do we need that "what branches are checked out in what
277+directories" piece of state? Why force the user to worry about
278+which branches are mounted to which directories? Why not just
279+mount them under their own names? Is there some crippling global
280+shortage of disk space for source files? Our planet is running
283+### Queries, subscriptions and synchronization
285+There are two ways to get data out of `%clay`: dereference the Urbit
286+namespace, or send a request/subscription. Sync is in userspace.
288+#### Dereferencing the namespace
290+`%clay` is just part of the full Urbit namespace. The full namespace
291+has a two-character prefix which expresses what vane to query
292+(always `%c` for `%clay`). The suffix is a mode flag; `%x` for
293+data, `%y` for a list of children, `%z` for the whole arch
294+subtree, `%w` for the version number. (Each vane has its own
297+For example, the full path to my cat picture is
300+/cx/~sorreg-namtyv/home/[now]/my/cat/picture
303+(where `now` is the current date).
305+If the data model is `++image` (hypothetical -- we don't really
306+do image processing in Urbit right now), the normal way to load
307+this data in a Hoon program is
310+.^(image /cx/~sorreg-namtyv/home/[now]/my/cat/picture)
313+This produces a statically typed value (which works by
314+dynamically checking `image` against the actual type of the
315+value, but this comparison caches well).
317+Internally, an Urbit namespace request produces a `++cage`, which
318+has a mark, a `++span` (language-level type), and an actual noun.
319+The product is doubly optional, `(unit (unit cage))`; this is `~`
320+if we don't know the value of this name at present but might
321+learn it in future, `[~ ~]` if we know we'll never know it, or
322+`[~ ~ value]` if we know it now.
324+In high-level shell programming (`:dojo`), we can make apparently
325+"blocking" requests to `%clay`. What actually happens: when `%clay`
326+produces `~`, we stop trying to compute this expression and wait
327+on one or more blocking requests. When we are unblocked, we
328+recompute again from scratch; the intermediate computation is not
331+Urbit doesn't like to block. It likes to save coherent data
332+structures, not incomplete computation contexts. When Urbit
333+pretends to block for you, this is a courtesy service. You
334+can return the courtesy of not expecting miracles from it.
335+When in doubt, send an actual request.
337+### Requests and subscriptions
339+A true request (local or remote) has access to the same
340+namespace, but can (a) read across case (revision) ranges,
341+and (b) filter by subtree.
343+If the revision range in a request stretches into the future, the
344+request becomes a subscription. Urbit messaging always regards a
345+single response as a special case of a subscription.
347+Generally, the response stream created by a request starts with
348+all the data the server has available right now; updates will
349+follow when they become available; the subscription will be
350+closed when the request range is complete.
352+#### Userspace synchronization
354+Synchronization per se is done in userspace. In `%clay`, sync just
355+means an application (typically the `%kiln` module in the system
356+app `:hood`) subscribes to a remote desk, and saves its change
357+stream locally. So sync is really an emergent feature of `%clay`.
361+`%clay`'s revision graph is completely uninteresting at present.
362+Some changes are proposed below, however.
364+As in any DVCS, a dome is the child of one parent dome (an edit)
365+or more parents (a merge). Often the parent of any revision is
366+the previous version in the sequence, but it can be anything.
370+`%clay`'s storage model is completely uninteresting. It's also a
371+little funky and I want to change it, so I'll be super vague.
373+A `++lobe`, a 128-bit atom, is the Merkle hash of a `blob`, which
374+is essentially a leaf. All blobs are stored in a hash table
375+(`(map lobe blob)`), and reference each other through lobes.
377+At present we don't GC this space at all, because we don't
378+actually delete any roots.
380+A `blob` may be `%direct`, meaning it's defined as the data
381+itself, or `%delta`, defined as a change against another blob.
382+This is motivated by the desire not to store a full snapshot
383+on every file change. But it may be misguided (see changes #0
386+We cache the full arch tree (`++ankh`) of the latest state in
387+each desk. For current queries, this mitigates a lot of the
388+opportunities for slowness that this data model creates. But the
389+disparity between the speed of the current version and the
390+speed previous version is an undesirable glitch in the API.
392+## Our motivation for change
394+We actually have _three_ motivations for change...
396+### We need to develop on %clay
398+At present, we're still using `git` as a development platform.
399+`%clay` is certainly used, but just for its strength --
400+deployment. This is starving Urbit of essential experience and
401+basically keeping it lame. So, change is necessary.
403+### Change must come from the community
405+"Why?" you ask? "Why must change come from the community?" Well,
406+one, duh. Change always comes from the community.
408+But two, a great force for change has departed. Philip Monk,
409+late maintainer of `%clay` (`~wictuc-folrex` in the old era,
410+`~wicdev-wisryt` in the new), is tired of types and typing. He
411+spent September roaming with the bison in Montana, and is now
412+literally on his way to Patagonia. On the one hand, Phil still
413+exists and is not at all unwilling to answer questions, even dumb
414+questions. On the other hand, _Patagonia_.
416+So %clay needs a new owner, or owners. At this point, we think
417+the community might be indoctrinated enough to take a stab.
419+Alas, `%clay` already has a pretty checkered ownership history.
420+Long ago I built a `%clay` that was really the only serious
421+design failure in Arvo. Then Jared Hance, a brilliant summer
422+intern, took my broken `%clay` and reformed it into a minimal `git`
423+in about a week. Then Philip modified Jared's `%clay` to be a
424+_typed_ revision-control system, which is a pretty cool thing.
426+But... in none of these transitions did `%clay` actually get
427+_rewritten_ per se. While it's a reasonably good vane, it
428+does contain one or two historical decisions.
430+We know a lot more about Arvo than we used to, and I think we can
431+revisit many of these decisions. The point of this post is to
432+start a `%clay` design discussion that will evolve into actual
435+### Pioneers may find it worth their time
437+We're planning a rewards program for contributions to `%clay`.
438+It'll simply involve setting a time period and a pool of real
439+estate, and allocating the pool by judged contribution.
441+This is an experimental program. We'll say more about it later.
442+It may not even happen. It may happen very differently. When in
443+doubt, don't talk about Fight Club.
445+## Some openings for change
447+Change, of course, must come from the community. No one can
448+dictate change from the top down. We _know_ this. But...
450+This list of suggested changes for the next `%clay` is not in any
451+sort of order: priority, structure, difficulty, etc. Let's talk
452+about the changes first; make actual plans second. The numbers
453+here are just for convenience.
455+_All changes in this program will happen on the next-generation
456+(`cc-release`) branch._
458+### Change #0: state reform
460+As mentioned above, the storage model in `%clay`, as the code is
461+at present, feels a little funky to me. There's nothing terribly
462+awful about it, but it doesn't feel quite right and its scaling
463+properties are very questionable.
465+Briefly, `%clay` today actually derives the hierarchy from its
466+leaves -- rather than _logically_ so deriving it. Only leaves
467+are in the object store. This is just weird, and you can feel
468+`%clay` trying to be something other than what it is. (I blame
469+Jared, though it's actually my fault.)
471+To store a dome snapshot (`++yaki`), the present code keeps a
472+`(map path blob)` at the root of the dome. This is enough to
473+reconstruct the `++arch` structures the namespace exports.
474+However, all kinds of basic arch operations, like basically `ls`,
475+become `O(n)` in the number of leaves in the dome. This is
476+obviously not quite right.
478+I believe the storage structure should be (removing everything
479+that isn't needed to serve the namespace):
482+ ++ clay (pair safe (map term desk))
483+ ++ desk (pair numb (map numb root))
484+ ++ dome (pair date root)
485+ ++ leaf (pair mark noun)
486+ ++ lobe @uvH :: 128-bit base32
487+ ++ numb @ud :: decimal version number
488+ ++ root (pair (unit leaf) (map knot lobe))
489+ ++ safe (map lobe root)
492+Here a `++root` is the universal node, with children addressed by
493+hash, but logically representing the abstract `++arch`.
495+Users of hierarchical filesystems basically expect performance
496+qualities that match the filesystem structure. A huge directory
497+in one place shouldn't affect search speeds somewhere else. And
498+of course, `ls` can't be O(n) in the size of the filesystem.
500+### Change #1: store edges, cache snapshots
502+The current `%clay` is snapshot-oriented, like `git`. The actual
503+changes are discarded. I think this is a bad choice.
505+`%clay` is not a research project, and has no idea of going as
506+far as `darcs` in the "set-of-patches" direction. However, at
507+present we actually throw away the patch (`++nori`).
509+I think the right DVCS design for `%clay` is a snapshot-oriented
510+system which also keeps the patches. The revision graph has the
511+same general shape as in the old `git` style. But rather than
512+just deriving the diffs from the snapshots, we can also derive
513+the snapshots from the diffs. When in doubt, we save space by
514+dropping the snapshot, not the diff.
516+For an RCS which is also a general-purpose typed filesystem, it's
517+clear that considering edges (deltas, diffs, changes, etc) as the
518+primary state is optimal. You can use diff functions to derive
519+deltas from snapshots, but in many data models this will always
520+risk losing information from the original action. When in
521+doubt, save what the user did.
523+This is even more true for operations on the tree structure, not
524+on individual leaves. Consider the heuristic energy that `git` has
525+to go through to figure out when `mv` has been used -- even when
526+`git mv` has been used! (According to the `git` faq, this is
527+because the lowest common denominator remains Linus' style of
528+working with emailed patches.)
530+Edges probably don't even need to be deduplicated. They also can
531+contain metadata such as commit messages (which `%clay` has no
532+place for at present).
534+Once edges are recognized as unreclaimed primary state, snapshots
535+can always be derived from the edges. So we can reclaim memory
536+being used for snapshots.
538+It's also straightforward to have a kind of edge which is a list
539+of edges, letting us collapse feature branches trivially.
541+Finally, I think references in edges should wherever possible
542+indirect through the namespace, rather than directly to the
543+content address. The user has strictly more information if you
544+say you merged `/~sorreg-namtyv/base/987/`, than if all you have
545+is the hash that this reference refers to.
547+### Change #2: unconfuse merge and rebase
549+The dispute between merge and rebase has all the bad smells of
550+a historical wart. But here I'm just not sure what's right.
552+The user has one _intent_ whether merging or rebasing, which is
553+to combine changes on one desk into another. The _algorithm_
554+used to construct this change, whether operational transformation
555+(rebasing) or a 3-way diff (merging) has one goal: use all
556+available information to model the authors' editing intent as
557+realistically as possible. If we save edges, we should use
560+Whether to define the resulting change graph with the typical
561+structure of a merge, or the typical structure of a rebase,
562+strikes me as quite arbitrary, and well into the land of "pick
563+one and force everyone to stick with it." But this call demands
564+a real expert on RCS workflow, which I'm not.
566+Since `%clay` is a humble part of a larger system and needs to
567+keep its weight down, "do it both ways because people have
568+different preferences" is not usually the right answer.
570+### Change #3: comment jihad
572+`%clay` needs to be brought up to the documentation standards of
573+`%jael`. This means (a) the formatting matches; (b) the comments
574+are at the same level of quality and ubiquity.
576+The basic commenting standards are: (a) every core (class) needs
577+a block comment explanation; (b) every gate (function) needs a
578+line comment; (c) every face (variable) needs a line comment.
580+This is obviously not just a clerical task. To write these
581+comments, you need to actually understand the code. This is a
582+nontrivial but extremely rewarding task, even if the code you're
583+commenting will get replaced.
585+Also, the _symbols_ in `%clay` must be brought up to `%jael`
586+quality. There is still far too much four-letterism here.
588+NB: `%clay` is about 3000 lines of pretty clean Hoon, which is
589+certainly not at all undocumented. We're not exactly looking at
590+a Matterhorn of reverse engineering here.
592+### Change #4: knot to coin internal paths
594+One feature of Urbit paths, briefly mentioned above, is that a
595+path segment or `++knot` (which is just a `@tas` text string, an
596+ASCII symbol as an atom) must be within the range of the
597+invertible noun printer, `++co`.
599+`++co` maps a described data value, a `++coin`, to an ASCII
600+string which is a valid URL segment and Unix filename. Its
601+friend `++so` inverts the function, perfectly.
603+Almost every knot is the simplest kind of `coin`, a `dime.` A
604+dime is a single atom with a rendering type (`aura` if you know
605+Hoon). Auras include `@tas` for symbolic text, `@ud` for
606+unsigned decimal, `@p` for Urbit ship, etc. `++co` can also
607+print arbitrary cells, though less prettily.
609+This feature is cool because it lets us stuff data into names in
610+a precise and predictable way. Usually the names in a namespace
611+are human-meaningful and human-originated, and are therefore
612+symbolic, but there are exceptions. For example, blogposts may
613+want to be named by date.
615+However, it is slightly retarded for the fundamental state of our
616+arches to store the _rendered_ string (`++knot`), not the source
617+data (`++coin`). It leads to all kinds of inefficiencies where
618+we're doing way more parsing than we want to. (Parsing also
619+should be jet-propelled, but is not.) It also makes it way too
620+easy to sneak in bad data and make it `%clay`'s problem.
622+So, logically, we'd be shifting from
625+ ++ arch (pair (unit (pair mark noun)) (map knot arch))
629+ ++ arch (pair (unit (pair mark noun)) (map coin arch))
632+Also, a vanilla `++map` might not be quite the right data
633+structure at this point -- we probably want a custom sorted tree,
634+which separates auras and imposes a custom order on each. This
635+would let us serve simple range queries quite efficiently, so
636+long as they match the standard aura order (eg, alphabetical
639+### Change #5: dome semantics by reference
641+While the principle that a dome contains its own mark source is
642+elegant, it's a pain in the ass to implement properly. `%clay`
643+certainly does not do it justice at present.
645+The complexity is in checking every change to see whether it
646+includes a change to the mark source (which is very unlikely),
647+or a dependency of the mark source (slightly less unlikely).
648+And if so, do we have to update all the leaves in the dome?
649+This is also very unlikely. And yet, it happens.
651+Instead, we could simply designate, by beak (ship, desk, and
652+case) a dome which contains the canonical mark source. This
653+would rarely change -- you could upgrade a desk to a new mark
654+dome, but it would be a manual operation and probably a rare one.
656+The common case would just be that everyone uses the current
657+version of the `%base` desk when a desk is created, and never
658+changes it. In practice, this means not every two-byte edit
659+needs to get involved in detecting and/or applying schema
660+evolution, a welcome simplification.
662+### Change #6: mark containers
664+Marks in present `%clay` are atomic and symbolic (like `%html`).
665+Since they're also used for network protocols, and the proper
666+design is one mark per message structure, the mark namespace gets
669+So we added an app hierarchy within the mark namespace, using hep
670+`-` as a divider. The `:talk` app expects messages with the mark
671+`talk-message`, whose sources is in `/===/mar/talk/message`.
673+This helps, but it doesn't solve another two problems: trivial
674+nouns and containers.
676+It's retarded to have to add a source file to send a `@ud` or a
677+`*`. An aura (see above under `coin`) should be a trivial mark.
679+Also, we need to be able to construct standard containers of
680+marks: lists, maps, sets, and tuples (or, as a database person
683+### Change #7: meta-marks
685+Another kind of complex mark that needs to exist: conflict,
686+difference, and invalid marks.
688+Conflict data and difference data are two types of noun that are
689+generated in normal RCS operation. If we have the concept of an
690+`%html` noun, we also need the concept of an invertible change
691+from one `%html` file to another; a merge conflict between two
692+`%html` files; and even a conflict between two marks.
694+There are two kinds of changes to a dome: manual and automatic.
695+Manual changes are initiated by the user and should crash, unless
696+manually overridden. Automatic changes should not crash, ever,
697+and should not lose data. If they create problems, which they
698+shouldn't, these problems will need to be cleaned up manually.
700+This is a complex space to navigate, but it requires a conflicted
701+merge to be able to resolve to a valid dome containing conflict
702+leaves, not crash. The present `%clay` goes halfway there and
703+resolves to a scratch desk, but we can go farther: the whole `git`
704+dumpster-fire of "merge state" deserves destruction. One change
705+is a merge that produces conflicts, the next is an edit that
708+Similarly, all sorts of errors can produce invalid leaves. One
709+particularly rare case which is nonetheless important is schema
710+evolution: what happens to files in a mark, when the mark source
711+code changes. They all need to be re-normalized. And this
712+operation can fail, producing an invalid value. (We don't do any
713+of this now, BTW, although `%ford` has the dependency mechanism to
714+make it possible.) Again, in some cases this should crash the
715+update to the mark source; in other cases, there is no local way
718+So there must be a way to preserve invalid nouns that can be
719+repaired by repair tools that expect invalid content, while not
720+actually marking them as valid so that innocent code is deluded
721+into using the bad content.
723+The bottom line: a mark needs to be split into two models:
724+`++mark`, the original symbolic mark with its source engine, and
725+something like `++logo`, the meta-mark or mark container. The
726+mark is the building block; the logo is the building.
728+### Change #8: filesystem hierarchy standard
730+While all filesystems remain forks of one ur-filesystem, the
731+standard tends to enforce itself. But not define itself.
733+I've always hated the current `/mar` for marks, for instance. I
734+prefer `/mac`. But others hate this. And so on. Also, by
735+necessity, our nascent "FHS" is heavily biased toward system
736+data, away from user data. This is retarded.
738+Please focus all bikeshedding energy on this problem! Better to
739+bikeshed about what color to paint the bikeshed, then what shape
740+the bike gears should be.
742+### Change #9: security and access control
744+Oh, yeah, that! They always say not to design security in at the
745+last minute. Unfortunately, this is exactly what we did. There
746+is no security at all in present `%clay`. (There is also no push
747+operation in `%clay`, so it's not as mad as you think -- anyone
748+can read anything, no one can write anything, which is fine in a
749+young network where there are no secrets.)
751+Fortunately, we don't have anything super fancy planned in this
752+area, just basic access-control lists. These obviously need to
753+be designed along with the filesystem-hierarchy standard (see
756+Is it even possible to do ACLs properly in `%clay`? At this
757+point, yes. There is an event-oriented security model in Arvo,
758+though it isn't really hooked up to anything at the moment.
759+There is also a privilege badge system in `%jael`. The
760+combination of these two features is quite sufficient for a
763+The idea of event-based security is that every event passed to
764+`%clay` has a security `++mask`. The mask tells us:
766+- the set of ships that has _contaminated_ this event
767+- the set of ships that has _endorsed_ this event
768+- the set of ships that this event may be _released_ to.
770+To simplify this system a little and maybe explain it, the
771+security question in any event is: (a) who has _caused_ this
772+event? (b) who may receive its _effects_? (These sets never
773+include your urbit itself, of course, only neighbors.)
775+For example, if we're doing something because ship X sent us a
776+packet, and we're going to leak the results of this something
777+back in a response to ship X, once `%clay` knows what our ship
778+thinks of its neighbor X, we should have a good picture of what
779+effects this event should be allowed to cause. If `%jael` knows
780+that X is our friend, `%clay` can know that X can see our cat
783+Causality is separated out into contamination (data from ship X
784+got into the event) and endorsement (ship X signed the whole
785+event). (This may be overdesign.) All these sets can be defined
786+as the empty set (eg, releasing to the empty set means no output)
787+or the universal set (input from the universal set could be from
788+anyone, ie, is unauthenticated).
790+### Change #10: stop using mark-extended paths
792+When you read a leaf from a `%clay` dome, `%clay` is happy to
793+perform any necessary mark conversions (the actual work is done
794+by the `%ford` build vane). You can also just ask for the leaf
795+in whatever mark `%clay` has it.
797+It's confusing to think of this mark extension as part of the
798+path. You shouldn't say `+cat /===/app/talk/hoon`. You should say
799+`+cat /===/app/talk`, or maybe `+cat %hoon /===/app/talk`.
801+A mark-extended namespace may be added to `%ford`, to compensate.
802+But type-converting reads are the rare case, not the common case.
804+### Change #11: real queries and indexes
806+A filesystem should postpone being a database until it can no
807+longer do so. So, please don't think about this problem yet.
808+I'm joking, of course!
810+### Change #12: meta-query
812+Make it possible to query the list of desks in a ship :-)
814+### Change #13: completely redesign Unix mounting
816+This is actually a series of changes, described below. But
819+One of Urbit's experiments is live two-way synchronization
820+between the Unix filesystem and the revision-control store.
821+Dropbox style, as it were. I think this experiment is basically
822+a failure. It may be right for cat pictures, but I don't think
823+it's right for Urbit.
825+The fundamental problem is that there are all sorts of weird,
826+flaky ways applications use a mutable filesystem. It's not at
827+all unusual for an editor to change a file by deleting the old
828+file, then writing the new one in its place.
830+You really don't want this to produce a delete change. A simple
831+cooldown timer for the sync helps a lot with this, but causes
832+other weirdnesses. In theory, a sync can even cause another
833+change to the same version of the same desk. And so on.
835+The thing we fear is just too funky to work (there is some
836+strange "reflux bug" still affecting it) is any sort of _two-way_
837+synchronization between Unix and Urbit.
839+And what I think puts this feature into the deadpool, or at least
840+cold storage, is that there's no useful workflow that actually
841+uses it -- at least, when editing Urbit code.
843+It's actually really nice to have files on a disk that nothing
844+will automatically modify. If you're using `git` and `%clay` in
845+parallel, it's nice to mount your `git` working directory. The
846+original Urbit workflow was to work out of a mounted directory,
847+but the two-way updates were just too weird and scary, so now we
848+tend to edit elsewhere and copy in -- which is lame.
850+### Change #14: figure out what to do with FUSE
852+We actually have two forms of filesystem access to `%clay`: read-only
853+FUSE (in a somewhat experimental state), and filesystem watching
854+(using `inotify()` and `FSEvents` under a `libuv` wrapper).
856+The FUSE interface is a good one, though it needs more work.
857+Moreover, it exports the whole Urbit namespace to Unix, not just
858+`%clay`. For reading Urbit data, FUSE is basically ideal.
860+The obvious impulse is to make a read-write FUSE. But I don't
861+think FUSE should be used for recording changes to an RCS, even
862+though the FUSE API sends you changes as events.
864+There are three reasons for this. One, the actual filesystem
865+writes that Unix apps make are typically of low quality and
866+confusing. Some crack-brained editor will find it completely
867+normal and unremarkable to modify a byte in a file, by deleting
868+the file and rewriting it from scratch. You can use a cooldown
869+timer to wait for this nonsense to converge, but leaving data in
870+heuristic limbo seems like a very bad idea in FUSE-land.
872+The second is that people derive a lot of comfort from seeing
873+actual files on an actual disk. The idea that your datas are
874+stuck in Urbit is slightly terrifying as a user experience.
875+We claim to be reliable. It's actually hard to imagine datas that
876+couldn't be recovered from either snapshot or log. But how can
877+you possibly believe us on this?
879+So I think it's correct to lean all the weight of reading data
880+from Urbit on FUSE, and use filesystem watching for writing.
881+Heuristic timeouts are fine in filesystem watching, because at
882+least the data got written when the disk says it got written.
883+And at least it's on the disk.
885+Also, FUSE is a daemon by definition, which forces us to think a
886+little harder about how to run the Urbit process as a daemon.
887+But this is a separate discussion.
889+### Change #15: one-way, nonced working desk
891+A one-way mount is a much less crazy thing. It works the same
892+way our two-way mount works today, but with only one writer: the
895+Since a mount is one-way, it needs to have an absolute write lock
896+on the desk it's writing to. We cannot have a `%home` desk which
897+we are also editing with filesystem watching.
899+Instead, we have a temporary and anonymous state that's connected
900+itself with your `%home` desk. Desk names have to be valid
901+symbols, but we can use `hep`, `-`. A reasonable choice is to
902+pick a random 32-bit name for each mount instance -- so the desk
903+name is not `%home`, but `%home-bosdut-sigder`. (Randomizing
904+instead of always using a fixed suffix, say, `%home-work`,
905+just makes all kinds of screwups less likely.)
907+You create a mount simply by mkdiring the desk name in your pier.
908+Just the base name, Urbit will come up with the `%bosdut-sigder`
909+part and put it in a `.desk` file at the desk root. To forget
910+the mount, just delete this file. This avoids the ugly confusion
911+between unmounting a mount, and deleting all the data in it.
913+(Urbit will keep a set of the mount desks it's used and
914+forgotten, not reuse them, and block all requests into them. By
915+blocking, we can delete without losing referential transparency.
916+Yes, Virginia, Urbit can forget!)
918+Note that the auto-reload behavior of applications is extremely
919+useful when testing. When testing, you can either switch the
920+active application desk from `%home` to `%home-bosdut-sigder`,
921+or activate autosync from the latter to the former.
923+### Change #16: commands as metafiles
925+A new idea I'd like to play with is that, since the basic model
926+of the mount process is to read Unix files and suck their
927+contents into Urbit, you should be able to control a mounted
928+`%clay` desk through Unix filesystem operations alone.
930+No Urbit command-line access, not even any `urb/` HTTP control
931+channels, should be required for the user to generate `%clay` data
932+through filesystem mount, or to set up the mount.
934+An RCS experience that works entirely through Unix tools, in
935+which the only connection between content creation tools and
936+revision-control tools is shared access to the filesystem itself,
937+is essential if your RCS has something so repulsive as a daemon.
938+Otherwise, you're basically, like, reinventing Perforce.
940+More concretely, the Urbit experience today requires `:hood`
941+pokes (Urbit shell commands, basically) for all nontrivial RCS
942+operations, from mounting to merging. Again, there are lots of
943+situations in which you can write into a mounted filesystem (the
944+protocol for Unix communication could be easily adapted to
945+communicate with a browser-based editor, for instance), but you
946+don't want the user to have to bother with an Urbit shell.
948+Even commands like mounting and merging are easily communicated,
949+through filesystem writes that any Unix tool (including many
952+This is very easy for a filesystem watcher. One such protocol:
953+if you want to send the daemon a command relative to some
954+directory, write a `.ur-command` metafile in that directory. The
955+daemon accepts the command by deleting it.
957+The user experience is that you configure your mounts by simply
958+making the directories and configuring dotfiles in them. Unlike
959+in the present system, Urbit will never modify your datas
960+silently in the background.
962+Here's what this looks like:
964+Suppose you want to edit some data in the `%home` desk. Your
965+data is in `/=home=/mar/talk`. You also want a read-only view
968+From Unix, you do something like this:
972+ $ mkdir -p home/mar/talk
973+ $ touch home/mar/talk/.edit
975+ $ touch home/arvo/.read
978+Want to commit your changes? That means merging the working
979+desk, `%home-bosdut-sigder`, into the `%home`. Put a commit message
980+into `home/.commit` and it'll happen. If there's an error, it
981+will appear as `.error`.
983+The mount has a .version file which simply contains the change
984+number (of the working desk) which any edits are relative to. If
985+the urbit daemon is running and watching, edits will be absorbed
986+quickly (probably after a heuristic timeout), and the version
987+incremented. But we always are confident that the edits are
988+against the latest version, or at least we can use this as a
989+sanity check. All kinds of insane things can happen to get a
990+filesystem out of consistency.
992+This design obviously needs a little fleshing out, but I'm quite
993+confident it's a convenient and reliable way to drive `%clay`
998+There's obviously a lot of fun here! There are two things you
1001+One, talk about `%clay` on [this forum
1002+thread](http://urbit.org/~~/fora/posts/~2016.10.25..17.37.47..6937~/) or
1003+on [:talk](http://urbit.org/stream). Both hosted on Urbit
1006+Two, poke around the
1007+[code](https://github.com/urbit/arvo/blob/cc-release/arvo/clay.hoon).
1008+Make sure you're on the development branch (`cc-release`).
1010+Phasing in this work will be a fun and interesting challenge.
1011+(My suspicion is that the right first task is the Unix mount.)