@@ -1,1012 +0,0 @@
1−+++
2−title = "Toward a New %clay"
3−date = "2016-07-13"
4−description = "Urbit's revision-control system, %clay, is itself due for a (medium-sized) revision!"
5−aliases = [
6− "/posts/essays/toward-a-new-clay",
7− "/posts/toward-a-new-clay",
8− "/blog/clay"
9−]
10−
11−[extra]
12−author = "Curtis Yarvin"
13−ship = "~sorreg-namtyv"
14−+++
15−
16−TLDR: (a) Urbit's revision-control system, `%clay`, is itself due
17−for a (medium-sized) revision! (b) this time, you yourself can
18−really help! (c) please don't give away the buried spoiler!
19−
20−This explanation does not (or at least should not) require you to
21−know anything about Urbit.
22−
23−### But %clay is already pretty cool
24−
25−Of course, nothing in Urbit is allowed to be _lame_. `%clay` is
26−actually quite cool. It does need some work. But first, let's
27−talk about what it is and why it's cool.
28−
29−`%clay` is the Arvo vane (Urbit kernel module) for revision
30−control. It's an authenticated, human-memorable global immutable
31−namespace which is also a typed revision-control system. It's
32−also very good at subscription and synchronization. Urbit
33−currently uses `%clay` to distribute live hotpatches; we
34−transparently upgrade the whole network with one checkin.
35−
36−### Compared to other revision-control systems
37−
38−`%clay` is clearly a vanilla DVCS in the modern `git` style. Of
39−deployed DVCS these days, `%clay` is probably closest in spirit
40−to `bzr` or maybe `fossil`.
41−
42−Obviously, as a part of a larger whole, you would expect any
43−Urbit RCS to be brutally spartan. `%clay` is. It's even more
44−spartan than `bzr`.
45−
46−(In fact, it's really too spartan to fully use at the moment. We
47−ourselves still use `%clay` mainly as a distribution system, and
48−do all internal development on GitHub. This is because the
49−present `%clay` lacks a few little conveniences, like commit
50−messages. See upgrades, below.)
51−
52−There are lots of normal DVCS things in `%clay`. But there are
53−also cool and unusual ones. Let's take a brief tour of the
54−high-level system, then describe what we think should change
55−about it.
56−
57−### Anatomy of a DVCS
58−
59−A DVCS is a historical state graph. A point in this graph is a
60−snapshot of some tree-structured system, like a filesystem.
61−
62−These snapshots, including recursive internal nodes, are stored in
63−a content-addressable store. Pointers in this store are used to
64−construct a revision graph, whose edges are edit and/or merge
65−actions. Some sort of namespace attaches meaningful names to
66−current revisions in this graph.
67−
68−There are six big questions in the design of a DVCS:
69−
70−- how are snapshots named?
71−- what is the data structure within a snapshot?
72−- how does it map to the Unix filesystem?
73−- how is the system queried?
74−- how is the revision graph defined?
75−- how does the content-addressable store work?
76−
77−## Anatomy of %clay
78−
79−A good way to summarize `%clay` (at present) is that its answers
80−to the first four questions are interesting, and its answers to
81−the last two are boring. We'll talk about all six, though.
82−
83−### Global namespace: [ship desk case]
84−
85−The permanent global name of a `%clay` snapshot is a `++beak`. A beak
86−is a triple: `[ship desk case]`.
87−
88−The `++ship` is an Urbit ship. A ship is a network address, a
89−human-memorable name and a cryptographic identity. The address
90−width is variable; shorter addresses are more desirable. Most
91−users have 32-bit addresses, which we map to four-syllable
92−synthetic names like `~sorreg-namtyv`.
93−
94−The `++desk` is a 4-letter word meaning "branch."
95−
96−The `++case` is a version, one of three kinds: a version sequence
97−number, a named label, or a date.
98−
99−In the Urbit path syntax, the beak is the first three segments of
100−the path: `/ship/desk/case/my/cat/picture`.
101−
102−In most cases, a path parser has a default ship, desk and case,
103−each of which can be replaced by `=`. So `/=home=/my/cat/picture`
104−is the data at `/my/cat/picture` on your own ship, in the `%home`
105−desk, right now. Immutability doesn't have to be inconvenient.
106−
107−`%clay` has no concept of a mutable name or a magic `HEAD`. Use
108−the current date. If you're referring to remote data and you're
109−worried about clock skew, subtract a second or three, or be
110−prepared to block. (Or keep your clock unskewed, which shouldn't
111−be _that_ hard.) It's not like mutable-name schemes work any
112−better than this anyway.
113−
114−(Immutability is central to `%clay`. `git` merely adopted the
115−reflog. `%clay` was born in it, molded by it...)
116−
117−`%clay` is objective; each Urbit node hosts only one ship, but
118−treats domestic and foreign data (almost) the same way. If you
119−ask your `%clay` repository for someone else's data, it will
120−block the request until it gets the data, which it then caches.
121−At present, we always fulfill requests from the origin ship,
122−though in future we could certainly use it as a tracker.
123−
124−If `%clay` discards cached data, it keeps a hash, so that your
125−urbit can't violate the local illusion of immutability. We
126−can't, of course, prevent anyone else's ship from sending
127−conflicting claims in different directions. But we can prevent
128−programs on our own ship from hearing about them, meaning that we
129−retain local referential transparency.
130−
131−### Local structure: dome, arch, leaf and mark
132−
133−A `%clay` snapshot is called a `++dome`. A dome is the `%clay`
134−equivalent of a tree-structured filesystem.
135−
136−Each node in this tree is one uniform structure, an `++arch`,
137−which is both file and directory. A `++leaf` or data value in
138−this arch is a typed document.
139−
140−A logical document type is called a `++mark`. A mark is a
141−symbolic pointer to a Hoon source file, in the same dome, for an
142−engine that can validate, compare, and/or convert
143−documents of that mark.
144−
145−Let's go over arches and marks a little more closely...
146−
147−#### An arch is a uniform filesystem node
148−
149−Most filesystems and RCS have modal nodes. For instance, a git
150−data node can be a blob or a tree. `%clay` (and the Urbit
151−namespace in general) has one kind of node, an `arch`:
152−
153−```
154− ++ arch (pair (unit leaf) (map knot arch))
155− ++ leaf (pair mark noun)
156−```
157−
158−An arch has an optional typed-data leaf, and optional named
159−children. In Unix terms, it's a file, a directory, or both. It
160−can't be neither: an empty arch, with no data and no children, is
161−auto-pruned. As in `git` (but not in Unix), the hierarchy is a
162−pure function of its leaves. Paths exist only because of the
163−data they point to.
164−
165−The `++knot` names in an arch path are also a little special.
166−They're typed, in a sense; any knot can be parsed invertibly into
167−a typed atom. The atom is an arbitrary unsigned integer. Its
168−type (an `++aura` if you know Hoon) defines the semantics and
169−presentation of the atom: for instance, a symbol, a decimal,
170−a date, etc.
171−
172−Ultimately a filesystem is a database, and a path is a primary
173−key. It's very useful to be able to store simple data in this
174−key, in a standard and unambiguous way.
175−
176−Why uniform nodes? One, they map nicely to information trees
177−where a subtree may need some custom collective presentation,
178−not an automatically generated Apache style directory listing.
179−Two, non-uniform nodes introduce a whole pointless category of
180−node type conflicts -- for which life is far too short.
181−
182−#### A mark is an executable document type
183−
184−A mark is "like an executable MIME type." It's best defined as a
185−symbolic mapping from a filesystem to a schema engine.
186−
187−If you have a MIME type `text/html`, you need to watch the
188−Internets to track what the W3C thinks HTML is. If you have an
189−Urbit mark `%html`, you can functionally construct (using our
190−insanely powerful functional build vane `%ford`) a core which
191−can validate or normalize HTML, apply patch and diff logic, and
192−translate to and from other marks. `%ford` can even reason about
193−translation paths and construct multistep conversions.
194−
195−(Urbit does not actually have this super-smart `%html` mark right
196−now. But it probably should, huh. And this page was composed in
197−Markdown and translated to HTML by `%ford`.)
198−
199−Where does the source code of a mark live? Within the same dome
200−as the data it describes. `%html` files are described by the
201−Hoon source at `/===/mar/html`, which perhaps gets mapped into
202−your Unix filesystem at `$ship/home/mar/html.hoon`.
203−
204−This Hoon file is not self-contained. It has dependencies. The
205−`%html` mark may depend on various structures and libraries
206−elsewhere in Urbit. But the source code for all them is within
207−this dome.
208−
209−One constraint this puts on domes is that _all domes are
210−descendants of the base system dome_. Domes are versions of one
211−reality, not fragments of that reality. You can't create a new
212−empty dome -- you'd have no mark source, and be unable to install
213−any validated files.
214−
215−Enforcing the rule that all domes are congruent (if only in that
216−they all contain the base system files) costs us nothing in the
217−implementation, does not inconvenience the user, and ensures that
218−weird collisions between non-parallel things don't happen.
219−
220−### Unix mapping and mounting
221−
222−Every DVCS is its own filesystem in a sense. But every DVCS
223−works by mapping and mounting itself, in some sense, to Unix.
224−
225−#### Mapping arches to Unix
226−
227−How do we map arches into Unix? The arch structure maps 1:1 to
228−the Unix directory tree. When the arch has a leaf, we derive a
229−dot extension from the file's mark.
230−
231−So, supposing we have a leaf `foo` which is an HTML document
232−(mark `%html`), with two children `bar` and `moo`, also HTML
233−documents, in Unix we'll see:
234−
235−```
236−foo.html
237−foo/bar.html
238−foo/moo.html
239−```
240−
241−This maps invertibly back into `%clay`, and it looks perfectly
242−unremarkable from the Unix side. File completion sometimes gets
243−a little annoying.
244−
245−It would be if nice every possible Unix tree, even every
246−plausible Unix tree, mapped cleanly and completely to a `%clay`
247−dome. There are always horrid exceptions, even in common use --
248−like extensionless Makefiles. The trivial solution is that if
249−`%clay` finds a file mysterious, it won't track it.
250−
251−One often unquestioned assumption in new system software is that
252−compatibility with old content is important. It's nice, but not
253−that important. Most content in a new system is new content.
254−The important compatibility direction is export, not import. So
255−the new system should subset, not superset, the ancient formats.
256−
257−### Mounting desks to Unix
258−
259−If a `%clay` desk is _mounted_, it has a two-way live sync to
260−the Unix directory of the same name within the pier.
261−
262−When files in a mount directory change, `%clay` notices it
263−through `libuv` filesystem watching (ultimately `inotify()` or
264−`FSEvents`), computes diffs, and automatically commits them to
265−the desk. And if `%clay` creates a new version for its own
266−urbity reasons, it will revise the Unix files.
267−
268−Keeping all mounted files in the same Unix subdirectory makes it
269−easy to move the whole pier in a tarfile. Edits applied while
270−the `urbit` process isn't running will be detected and applied
271−the next time it starts.
272−
273−In conventional DVCS terms, `%clay` has a 1:1 static mapping from
274−branches to working directories. Why?
275−
276−Why not? Do we need that "what branches are checked out in what
277−directories" piece of state? Why force the user to worry about
278−which branches are mounted to which directories? Why not just
279−mount them under their own names? Is there some crippling global
280−shortage of disk space for source files? Our planet is running
281−out of inodes? Wut?
282−
283−### Queries, subscriptions and synchronization
284−
285−There are two ways to get data out of `%clay`: dereference the Urbit
286−namespace, or send a request/subscription. Sync is in userspace.
287−
288−#### Dereferencing the namespace
289−
290−`%clay` is just part of the full Urbit namespace. The full namespace
291−has a two-character prefix which expresses what vane to query
292−(always `%c` for `%clay`). The suffix is a mode flag; `%x` for
293−data, `%y` for a list of children, `%z` for the whole arch
294−subtree, `%w` for the version number. (Each vane has its own
295−modes.)
296−
297−For example, the full path to my cat picture is
298−
299−```
300−/cx/~sorreg-namtyv/home/[now]/my/cat/picture
301−```
302−
303−(where `now` is the current date).
304−
305−If the data model is `++image` (hypothetical -- we don't really
306−do image processing in Urbit right now), the normal way to load
307−this data in a Hoon program is
308−
309−```
310−.^(image /cx/~sorreg-namtyv/home/[now]/my/cat/picture)
311−```
312−
313−This produces a statically typed value (which works by
314−dynamically checking `image` against the actual type of the
315−value, but this comparison caches well).
316−
317−Internally, an Urbit namespace request produces a `++cage`, which
318−has a mark, a `++span` (language-level type), and an actual noun.
319−The product is doubly optional, `(unit (unit cage))`; this is `~`
320−if we don't know the value of this name at present but might
321−learn it in future, `[~ ~]` if we know we'll never know it, or
322−`[~ ~ value]` if we know it now.
323−
324−In high-level shell programming (`:dojo`), we can make apparently
325−"blocking" requests to `%clay`. What actually happens: when `%clay`
326−produces `~`, we stop trying to compute this expression and wait
327−on one or more blocking requests. When we are unblocked, we
328−recompute again from scratch; the intermediate computation is not
329−saved at all.
330−
331−Urbit doesn't like to block. It likes to save coherent data
332−structures, not incomplete computation contexts. When Urbit
333−pretends to block for you, this is a courtesy service. You
334−can return the courtesy of not expecting miracles from it.
335−When in doubt, send an actual request.
336−
337−### Requests and subscriptions
338−
339−A true request (local or remote) has access to the same
340−namespace, but can (a) read across case (revision) ranges,
341−and (b) filter by subtree.
342−
343−If the revision range in a request stretches into the future, the
344−request becomes a subscription. Urbit messaging always regards a
345−single response as a special case of a subscription.
346−
347−Generally, the response stream created by a request starts with
348−all the data the server has available right now; updates will
349−follow when they become available; the subscription will be
350−closed when the request range is complete.
351−
352−#### Userspace synchronization
353−
354−Synchronization per se is done in userspace. In `%clay`, sync just
355−means an application (typically the `%kiln` module in the system
356−app `:hood`) subscribes to a remote desk, and saves its change
357−stream locally. So sync is really an emergent feature of `%clay`.
358−
359−### Revision graph
360−
361−`%clay`'s revision graph is completely uninteresting at present.
362−Some changes are proposed below, however.
363−
364−As in any DVCS, a dome is the child of one parent dome (an edit)
365−or more parents (a merge). Often the parent of any revision is
366−the previous version in the sequence, but it can be anything.
367−
368−### Storage model
369−
370−`%clay`'s storage model is completely uninteresting. It's also a
371−little funky and I want to change it, so I'll be super vague.
372−
373−A `++lobe`, a 128-bit atom, is the Merkle hash of a `blob`, which
374−is essentially a leaf. All blobs are stored in a hash table
375−(`(map lobe blob)`), and reference each other through lobes.
376−
377−At present we don't GC this space at all, because we don't
378−actually delete any roots.
379−
380−A `blob` may be `%direct`, meaning it's defined as the data
381−itself, or `%delta`, defined as a change against another blob.
382−This is motivated by the desire not to store a full snapshot
383−on every file change. But it may be misguided (see changes #0
384−and #1 below).
385−
386−We cache the full arch tree (`++ankh`) of the latest state in
387−each desk. For current queries, this mitigates a lot of the
388−opportunities for slowness that this data model creates. But the
389−disparity between the speed of the current version and the
390−speed previous version is an undesirable glitch in the API.
391−
392−## Our motivation for change
393−
394−We actually have _three_ motivations for change...
395−
396−### We need to develop on %clay
397−
398−At present, we're still using `git` as a development platform.
399−`%clay` is certainly used, but just for its strength --
400−deployment. This is starving Urbit of essential experience and
401−basically keeping it lame. So, change is necessary.
402−
403−### Change must come from the community
404−
405−"Why?" you ask? "Why must change come from the community?" Well,
406−one, duh. Change always comes from the community.
407−
408−But two, a great force for change has departed. Philip Monk,
409−late maintainer of `%clay` (`~wictuc-folrex` in the old era,
410−`~wicdev-wisryt` in the new), is tired of types and typing. He
411−spent September roaming with the bison in Montana, and is now
412−literally on his way to Patagonia. On the one hand, Phil still
413−exists and is not at all unwilling to answer questions, even dumb
414−questions. On the other hand, _Patagonia_.
415−
416−So %clay needs a new owner, or owners. At this point, we think
417−the community might be indoctrinated enough to take a stab.
418−
419−Alas, `%clay` already has a pretty checkered ownership history.
420−Long ago I built a `%clay` that was really the only serious
421−design failure in Arvo. Then Jared Hance, a brilliant summer
422−intern, took my broken `%clay` and reformed it into a minimal `git`
423−in about a week. Then Philip modified Jared's `%clay` to be a
424−_typed_ revision-control system, which is a pretty cool thing.
425−
426−But... in none of these transitions did `%clay` actually get
427−_rewritten_ per se. While it's a reasonably good vane, it
428−does contain one or two historical decisions.
429−
430−We know a lot more about Arvo than we used to, and I think we can
431−revisit many of these decisions. The point of this post is to
432−start a `%clay` design discussion that will evolve into actual
433−collective action.
434−
435−### Pioneers may find it worth their time
436−
437−We're planning a rewards program for contributions to `%clay`.
438−It'll simply involve setting a time period and a pool of real
439−estate, and allocating the pool by judged contribution.
440−
441−This is an experimental program. We'll say more about it later.
442−It may not even happen. It may happen very differently. When in
443−doubt, don't talk about Fight Club.
444−
445−## Some openings for change
446−
447−Change, of course, must come from the community. No one can
448−dictate change from the top down. We _know_ this. But...
449−
450−This list of suggested changes for the next `%clay` is not in any
451−sort of order: priority, structure, difficulty, etc. Let's talk
452−about the changes first; make actual plans second. The numbers
453−here are just for convenience.
454−
455−_All changes in this program will happen on the next-generation
456−(`cc-release`) branch._
457−
458−### Change #0: state reform
459−
460−As mentioned above, the storage model in `%clay`, as the code is
461−at present, feels a little funky to me. There's nothing terribly
462−awful about it, but it doesn't feel quite right and its scaling
463−properties are very questionable.
464−
465−Briefly, `%clay` today actually derives the hierarchy from its
466−leaves -- rather than _logically_ so deriving it. Only leaves
467−are in the object store. This is just weird, and you can feel
468−`%clay` trying to be something other than what it is. (I blame
469−Jared, though it's actually my fault.)
470−
471−To store a dome snapshot (`++yaki`), the present code keeps a
472−`(map path blob)` at the root of the dome. This is enough to
473−reconstruct the `++arch` structures the namespace exports.
474−However, all kinds of basic arch operations, like basically `ls`,
475−become `O(n)` in the number of leaves in the dome. This is
476−obviously not quite right.
477−
478−I believe the storage structure should be (removing everything
479−that isn't needed to serve the namespace):
480−
481−```
482− ++ clay (pair safe (map term desk))
483− ++ desk (pair numb (map numb root))
484− ++ dome (pair date root)
485− ++ leaf (pair mark noun)
486− ++ lobe @uvH :: 128-bit base32
487− ++ numb @ud :: decimal version number
488− ++ root (pair (unit leaf) (map knot lobe))
489− ++ safe (map lobe root)
490−```
491−
492−Here a `++root` is the universal node, with children addressed by
493−hash, but logically representing the abstract `++arch`.
494−
495−Users of hierarchical filesystems basically expect performance
496−qualities that match the filesystem structure. A huge directory
497−in one place shouldn't affect search speeds somewhere else. And
498−of course, `ls` can't be O(n) in the size of the filesystem.
499−
500−### Change #1: store edges, cache snapshots
501−
502−The current `%clay` is snapshot-oriented, like `git`. The actual
503−changes are discarded. I think this is a bad choice.
504−
505−`%clay` is not a research project, and has no idea of going as
506−far as `darcs` in the "set-of-patches" direction. However, at
507−present we actually throw away the patch (`++nori`).
508−
509−I think the right DVCS design for `%clay` is a snapshot-oriented
510−system which also keeps the patches. The revision graph has the
511−same general shape as in the old `git` style. But rather than
512−just deriving the diffs from the snapshots, we can also derive
513−the snapshots from the diffs. When in doubt, we save space by
514−dropping the snapshot, not the diff.
515−
516−For an RCS which is also a general-purpose typed filesystem, it's
517−clear that considering edges (deltas, diffs, changes, etc) as the
518−primary state is optimal. You can use diff functions to derive
519−deltas from snapshots, but in many data models this will always
520−risk losing information from the original action. When in
521−doubt, save what the user did.
522−
523−This is even more true for operations on the tree structure, not
524−on individual leaves. Consider the heuristic energy that `git` has
525−to go through to figure out when `mv` has been used -- even when
526−`git mv` has been used! (According to the `git` faq, this is
527−because the lowest common denominator remains Linus' style of
528−working with emailed patches.)
529−
530−Edges probably don't even need to be deduplicated. They also can
531−contain metadata such as commit messages (which `%clay` has no
532−place for at present).
533−
534−Once edges are recognized as unreclaimed primary state, snapshots
535−can always be derived from the edges. So we can reclaim memory
536−being used for snapshots.
537−
538−It's also straightforward to have a kind of edge which is a list
539−of edges, letting us collapse feature branches trivially.
540−
541−Finally, I think references in edges should wherever possible
542−indirect through the namespace, rather than directly to the
543−content address. The user has strictly more information if you
544−say you merged `/~sorreg-namtyv/base/987/`, than if all you have
545−is the hash that this reference refers to.
546−
547−### Change #2: unconfuse merge and rebase
548−
549−The dispute between merge and rebase has all the bad smells of
550−a historical wart. But here I'm just not sure what's right.
551−
552−The user has one _intent_ whether merging or rebasing, which is
553−to combine changes on one desk into another. The _algorithm_
554−used to construct this change, whether operational transformation
555−(rebasing) or a 3-way diff (merging) has one goal: use all
556−available information to model the authors' editing intent as
557−realistically as possible. If we save edges, we should use
558−edges.
559−
560−Whether to define the resulting change graph with the typical
561−structure of a merge, or the typical structure of a rebase,
562−strikes me as quite arbitrary, and well into the land of "pick
563−one and force everyone to stick with it." But this call demands
564−a real expert on RCS workflow, which I'm not.
565−
566−Since `%clay` is a humble part of a larger system and needs to
567−keep its weight down, "do it both ways because people have
568−different preferences" is not usually the right answer.
569−
570−### Change #3: comment jihad
571−
572−`%clay` needs to be brought up to the documentation standards of
573−`%jael`. This means (a) the formatting matches; (b) the comments
574−are at the same level of quality and ubiquity.
575−
576−The basic commenting standards are: (a) every core (class) needs
577−a block comment explanation; (b) every gate (function) needs a
578−line comment; (c) every face (variable) needs a line comment.
579−
580−This is obviously not just a clerical task. To write these
581−comments, you need to actually understand the code. This is a
582−nontrivial but extremely rewarding task, even if the code you're
583−commenting will get replaced.
584−
585−Also, the _symbols_ in `%clay` must be brought up to `%jael`
586−quality. There is still far too much four-letterism here.
587−
588−NB: `%clay` is about 3000 lines of pretty clean Hoon, which is
589−certainly not at all undocumented. We're not exactly looking at
590−a Matterhorn of reverse engineering here.
591−
592−### Change #4: knot to coin internal paths
593−
594−One feature of Urbit paths, briefly mentioned above, is that a
595−path segment or `++knot` (which is just a `@tas` text string, an
596−ASCII symbol as an atom) must be within the range of the
597−invertible noun printer, `++co`.
598−
599−`++co` maps a described data value, a `++coin`, to an ASCII
600−string which is a valid URL segment and Unix filename. Its
601−friend `++so` inverts the function, perfectly.
602−
603−Almost every knot is the simplest kind of `coin`, a `dime.` A
604−dime is a single atom with a rendering type (`aura` if you know
605−Hoon). Auras include `@tas` for symbolic text, `@ud` for
606−unsigned decimal, `@p` for Urbit ship, etc. `++co` can also
607−print arbitrary cells, though less prettily.
608−
609−This feature is cool because it lets us stuff data into names in
610−a precise and predictable way. Usually the names in a namespace
611−are human-meaningful and human-originated, and are therefore
612−symbolic, but there are exceptions. For example, blogposts may
613−want to be named by date.
614−
615−However, it is slightly retarded for the fundamental state of our
616−arches to store the _rendered_ string (`++knot`), not the source
617−data (`++coin`). It leads to all kinds of inefficiencies where
618−we're doing way more parsing than we want to. (Parsing also
619−should be jet-propelled, but is not.) It also makes it way too
620−easy to sneak in bad data and make it `%clay`'s problem.
621−
622−So, logically, we'd be shifting from
623−
624−```
625− ++ arch (pair (unit (pair mark noun)) (map knot arch))
626−```
627−
628−```
629− ++ arch (pair (unit (pair mark noun)) (map coin arch))
630−```
631−
632−Also, a vanilla `++map` might not be quite the right data
633−structure at this point -- we probably want a custom sorted tree,
634−which separates auras and imposes a custom order on each. This
635−would let us serve simple range queries quite efficiently, so
636−long as they match the standard aura order (eg, alphabetical
637−order for symbols).
638−
639−### Change #5: dome semantics by reference
640−
641−While the principle that a dome contains its own mark source is
642−elegant, it's a pain in the ass to implement properly. `%clay`
643−certainly does not do it justice at present.
644−
645−The complexity is in checking every change to see whether it
646−includes a change to the mark source (which is very unlikely),
647−or a dependency of the mark source (slightly less unlikely).
648−And if so, do we have to update all the leaves in the dome?
649−This is also very unlikely. And yet, it happens.
650−
651−Instead, we could simply designate, by beak (ship, desk, and
652−case) a dome which contains the canonical mark source. This
653−would rarely change -- you could upgrade a desk to a new mark
654−dome, but it would be a manual operation and probably a rare one.
655−
656−The common case would just be that everyone uses the current
657−version of the `%base` desk when a desk is created, and never
658−changes it. In practice, this means not every two-byte edit
659−needs to get involved in detecting and/or applying schema
660−evolution, a welcome simplification.
661−
662−### Change #6: mark containers
663−
664−Marks in present `%clay` are atomic and symbolic (like `%html`).
665−Since they're also used for network protocols, and the proper
666−design is one mark per message structure, the mark namespace gets
667−overloaded fast.
668−
669−So we added an app hierarchy within the mark namespace, using hep
670−`-` as a divider. The `:talk` app expects messages with the mark
671−`talk-message`, whose sources is in `/===/mar/talk/message`.
672−
673−This helps, but it doesn't solve another two problems: trivial
674−nouns and containers.
675−
676−It's retarded to have to add a source file to send a `@ud` or a
677−`*`. An aura (see above under `coin`) should be a trivial mark.
678−
679−Also, we need to be able to construct standard containers of
680−marks: lists, maps, sets, and tuples (or, as a database person
681−would say, "rows").
682−
683−### Change #7: meta-marks
684−
685−Another kind of complex mark that needs to exist: conflict,
686−difference, and invalid marks.
687−
688−Conflict data and difference data are two types of noun that are
689−generated in normal RCS operation. If we have the concept of an
690−`%html` noun, we also need the concept of an invertible change
691−from one `%html` file to another; a merge conflict between two
692−`%html` files; and even a conflict between two marks.
693−
694−There are two kinds of changes to a dome: manual and automatic.
695−Manual changes are initiated by the user and should crash, unless
696−manually overridden. Automatic changes should not crash, ever,
697−and should not lose data. If they create problems, which they
698−shouldn't, these problems will need to be cleaned up manually.
699−
700−This is a complex space to navigate, but it requires a conflicted
701−merge to be able to resolve to a valid dome containing conflict
702−leaves, not crash. The present `%clay` goes halfway there and
703−resolves to a scratch desk, but we can go farther: the whole `git`
704−dumpster-fire of "merge state" deserves destruction. One change
705−is a merge that produces conflicts, the next is an edit that
706−resolves them.
707−
708−Similarly, all sorts of errors can produce invalid leaves. One
709−particularly rare case which is nonetheless important is schema
710−evolution: what happens to files in a mark, when the mark source
711−code changes. They all need to be re-normalized. And this
712−operation can fail, producing an invalid value. (We don't do any
713−of this now, BTW, although `%ford` has the dependency mechanism to
714−make it possible.) Again, in some cases this should crash the
715−update to the mark source; in other cases, there is no local way
716−to handle the crash.
717−
718−So there must be a way to preserve invalid nouns that can be
719−repaired by repair tools that expect invalid content, while not
720−actually marking them as valid so that innocent code is deluded
721−into using the bad content.
722−
723−The bottom line: a mark needs to be split into two models:
724−`++mark`, the original symbolic mark with its source engine, and
725−something like `++logo`, the meta-mark or mark container. The
726−mark is the building block; the logo is the building.
727−
728−### Change #8: filesystem hierarchy standard
729−
730−While all filesystems remain forks of one ur-filesystem, the
731−standard tends to enforce itself. But not define itself.
732−
733−I've always hated the current `/mar` for marks, for instance. I
734−prefer `/mac`. But others hate this. And so on. Also, by
735−necessity, our nascent "FHS" is heavily biased toward system
736−data, away from user data. This is retarded.
737−
738−Please focus all bikeshedding energy on this problem! Better to
739−bikeshed about what color to paint the bikeshed, then what shape
740−the bike gears should be.
741−
742−### Change #9: security and access control
743−
744−Oh, yeah, that! They always say not to design security in at the
745−last minute. Unfortunately, this is exactly what we did. There
746−is no security at all in present `%clay`. (There is also no push
747−operation in `%clay`, so it's not as mad as you think -- anyone
748−can read anything, no one can write anything, which is fine in a
749−young network where there are no secrets.)
750−
751−Fortunately, we don't have anything super fancy planned in this
752−area, just basic access-control lists. These obviously need to
753−be designed along with the filesystem-hierarchy standard (see
754−elsewhere).
755−
756−Is it even possible to do ACLs properly in `%clay`? At this
757−point, yes. There is an event-oriented security model in Arvo,
758−though it isn't really hooked up to anything at the moment.
759−There is also a privilege badge system in `%jael`. The
760−combination of these two features is quite sufficient for a
761−decent ACL model.
762−
763−The idea of event-based security is that every event passed to
764−`%clay` has a security `++mask`. The mask tells us:
765−
766−- the set of ships that has _contaminated_ this event
767−- the set of ships that has _endorsed_ this event
768−- the set of ships that this event may be _released_ to.
769−
770−To simplify this system a little and maybe explain it, the
771−security question in any event is: (a) who has _caused_ this
772−event? (b) who may receive its _effects_? (These sets never
773−include your urbit itself, of course, only neighbors.)
774−
775−For example, if we're doing something because ship X sent us a
776−packet, and we're going to leak the results of this something
777−back in a response to ship X, once `%clay` knows what our ship
778−thinks of its neighbor X, we should have a good picture of what
779−effects this event should be allowed to cause. If `%jael` knows
780−that X is our friend, `%clay` can know that X can see our cat
781−picture.
782−
783−Causality is separated out into contamination (data from ship X
784−got into the event) and endorsement (ship X signed the whole
785−event). (This may be overdesign.) All these sets can be defined
786−as the empty set (eg, releasing to the empty set means no output)
787−or the universal set (input from the universal set could be from
788−anyone, ie, is unauthenticated).
789−
790−### Change #10: stop using mark-extended paths
791−
792−When you read a leaf from a `%clay` dome, `%clay` is happy to
793−perform any necessary mark conversions (the actual work is done
794−by the `%ford` build vane). You can also just ask for the leaf
795−in whatever mark `%clay` has it.
796−
797−It's confusing to think of this mark extension as part of the
798−path. You shouldn't say `+cat /===/app/talk/hoon`. You should say
799−`+cat /===/app/talk`, or maybe `+cat %hoon /===/app/talk`.
800−
801−A mark-extended namespace may be added to `%ford`, to compensate.
802−But type-converting reads are the rare case, not the common case.
803−
804−### Change #11: real queries and indexes
805−
806−A filesystem should postpone being a database until it can no
807−longer do so. So, please don't think about this problem yet.
808−I'm joking, of course!
809−
810−### Change #12: meta-query
811−
812−Make it possible to query the list of desks in a ship :-)
813−
814−### Change #13: completely redesign Unix mounting
815−
816−This is actually a series of changes, described below. But
817−broadly...
818−
819−One of Urbit's experiments is live two-way synchronization
820−between the Unix filesystem and the revision-control store.
821−Dropbox style, as it were. I think this experiment is basically
822−a failure. It may be right for cat pictures, but I don't think
823−it's right for Urbit.
824−
825−The fundamental problem is that there are all sorts of weird,
826−flaky ways applications use a mutable filesystem. It's not at
827−all unusual for an editor to change a file by deleting the old
828−file, then writing the new one in its place.
829−
830−You really don't want this to produce a delete change. A simple
831−cooldown timer for the sync helps a lot with this, but causes
832−other weirdnesses. In theory, a sync can even cause another
833−change to the same version of the same desk. And so on.
834−
835−The thing we fear is just too funky to work (there is some
836−strange "reflux bug" still affecting it) is any sort of _two-way_
837−synchronization between Unix and Urbit.
838−
839−And what I think puts this feature into the deadpool, or at least
840−cold storage, is that there's no useful workflow that actually
841−uses it -- at least, when editing Urbit code.
842−
843−It's actually really nice to have files on a disk that nothing
844−will automatically modify. If you're using `git` and `%clay` in
845−parallel, it's nice to mount your `git` working directory. The
846−original Urbit workflow was to work out of a mounted directory,
847−but the two-way updates were just too weird and scary, so now we
848−tend to edit elsewhere and copy in -- which is lame.
849−
850−### Change #14: figure out what to do with FUSE
851−
852−We actually have two forms of filesystem access to `%clay`: read-only
853−FUSE (in a somewhat experimental state), and filesystem watching
854−(using `inotify()` and `FSEvents` under a `libuv` wrapper).
855−
856−The FUSE interface is a good one, though it needs more work.
857−Moreover, it exports the whole Urbit namespace to Unix, not just
858−`%clay`. For reading Urbit data, FUSE is basically ideal.
859−
860−The obvious impulse is to make a read-write FUSE. But I don't
861−think FUSE should be used for recording changes to an RCS, even
862−though the FUSE API sends you changes as events.
863−
864−There are three reasons for this. One, the actual filesystem
865−writes that Unix apps make are typically of low quality and
866−confusing. Some crack-brained editor will find it completely
867−normal and unremarkable to modify a byte in a file, by deleting
868−the file and rewriting it from scratch. You can use a cooldown
869−timer to wait for this nonsense to converge, but leaving data in
870−heuristic limbo seems like a very bad idea in FUSE-land.
871−
872−The second is that people derive a lot of comfort from seeing
873−actual files on an actual disk. The idea that your datas are
874−stuck in Urbit is slightly terrifying as a user experience.
875−We claim to be reliable. It's actually hard to imagine datas that
876−couldn't be recovered from either snapshot or log. But how can
877−you possibly believe us on this?
878−
879−So I think it's correct to lean all the weight of reading data
880−from Urbit on FUSE, and use filesystem watching for writing.
881−Heuristic timeouts are fine in filesystem watching, because at
882−least the data got written when the disk says it got written.
883−And at least it's on the disk.
884−
885−Also, FUSE is a daemon by definition, which forces us to think a
886−little harder about how to run the Urbit process as a daemon.
887−But this is a separate discussion.
888−
889−### Change #15: one-way, nonced working desk
890−
891−A one-way mount is a much less crazy thing. It works the same
892−way our two-way mount works today, but with only one writer: the
893−user, via Unix.
894−
895−Since a mount is one-way, it needs to have an absolute write lock
896−on the desk it's writing to. We cannot have a `%home` desk which
897−we are also editing with filesystem watching.
898−
899−Instead, we have a temporary and anonymous state that's connected
900−itself with your `%home` desk. Desk names have to be valid
901−symbols, but we can use `hep`, `-`. A reasonable choice is to
902−pick a random 32-bit name for each mount instance -- so the desk
903−name is not `%home`, but `%home-bosdut-sigder`. (Randomizing
904−instead of always using a fixed suffix, say, `%home-work`,
905−just makes all kinds of screwups less likely.)
906−
907−You create a mount simply by mkdiring the desk name in your pier.
908−Just the base name, Urbit will come up with the `%bosdut-sigder`
909−part and put it in a `.desk` file at the desk root. To forget
910−the mount, just delete this file. This avoids the ugly confusion
911−between unmounting a mount, and deleting all the data in it.
912−
913−(Urbit will keep a set of the mount desks it's used and
914−forgotten, not reuse them, and block all requests into them. By
915−blocking, we can delete without losing referential transparency.
916−Yes, Virginia, Urbit can forget!)
917−
918−Note that the auto-reload behavior of applications is extremely
919−useful when testing. When testing, you can either switch the
920−active application desk from `%home` to `%home-bosdut-sigder`,
921−or activate autosync from the latter to the former.
922−
923−### Change #16: commands as metafiles
924−
925−A new idea I'd like to play with is that, since the basic model
926−of the mount process is to read Unix files and suck their
927−contents into Urbit, you should be able to control a mounted
928−`%clay` desk through Unix filesystem operations alone.
929−
930−No Urbit command-line access, not even any `urb/` HTTP control
931−channels, should be required for the user to generate `%clay` data
932−through filesystem mount, or to set up the mount.
933−
934−An RCS experience that works entirely through Unix tools, in
935−which the only connection between content creation tools and
936−revision-control tools is shared access to the filesystem itself,
937−is essential if your RCS has something so repulsive as a daemon.
938−Otherwise, you're basically, like, reinventing Perforce.
939−
940−More concretely, the Urbit experience today requires `:hood`
941−pokes (Urbit shell commands, basically) for all nontrivial RCS
942−operations, from mounting to merging. Again, there are lots of
943−situations in which you can write into a mounted filesystem (the
944−protocol for Unix communication could be easily adapted to
945−communicate with a browser-based editor, for instance), but you
946−don't want the user to have to bother with an Urbit shell.
947−
948−Even commands like mounting and merging are easily communicated,
949−through filesystem writes that any Unix tool (including many
950−macro editors).
951−
952−This is very easy for a filesystem watcher. One such protocol:
953−if you want to send the daemon a command relative to some
954−directory, write a `.ur-command` metafile in that directory. The
955−daemon accepts the command by deleting it.
956−
957−The user experience is that you configure your mounts by simply
958−making the directories and configuring dotfiles in them. Unlike
959−in the present system, Urbit will never modify your datas
960−silently in the background.
961−
962−Here's what this looks like:
963−
964−Suppose you want to edit some data in the `%home` desk. Your
965−data is in `/=home=/mar/talk`. You also want a read-only view
966−of `/=home=/arvo`.
967−
968−From Unix, you do something like this:
969−
970−```
971− $ cd $pier
972− $ mkdir -p home/mar/talk
973− $ touch home/mar/talk/.edit
974− $ mkdir home/arvo
975− $ touch home/arvo/.read
976−```
977−
978−Want to commit your changes? That means merging the working
979−desk, `%home-bosdut-sigder`, into the `%home`. Put a commit message
980−into `home/.commit` and it'll happen. If there's an error, it
981−will appear as `.error`.
982−
983−The mount has a .version file which simply contains the change
984−number (of the working desk) which any edits are relative to. If
985−the urbit daemon is running and watching, edits will be absorbed
986−quickly (probably after a heuristic timeout), and the version
987−incremented. But we always are confident that the edits are
988−against the latest version, or at least we can use this as a
989−sanity check. All kinds of insane things can happen to get a
990−filesystem out of consistency.
991−
992−This design obviously needs a little fleshing out, but I'm quite
993−confident it's a convenient and reliable way to drive `%clay`
994−from Unix.
995−
996−## In conclusion
997−
998−There's obviously a lot of fun here! There are two things you
999−can do right now.
1000−
1001−One, talk about `%clay` on [this forum
1002−thread](http://urbit.org/~~/fora/posts/~2016.10.25..17.37.47..6937~/) or
1003−on [:talk](http://urbit.org/stream). Both hosted on Urbit
1004−itself!
1005−
1006−Two, poke around the
1007−[code](https://github.com/urbit/arvo/blob/cc-release/arvo/clay.hoon).
1008−Make sure you're on the development branch (`cc-release`).
1009−
1010−Phasing in this work will be a fun and interesting challenge.
1011−(My suspicion is that the right first task is the Unix mount.)
1012−Let's talk...