@@ -1,63 +1,68 @@
2title = "Toward a New %clay"
2title = "Toward a New %clay"
3−date = 2016-07-14
3+date = "2016-07-13"
4description = "Urbit's revision-control system, %clay, is itself due for a (medium-sized) revision!"
4description = "Urbit's revision-control system, %clay, is itself due for a (medium-sized) revision!"
5−aliases = ["/posts/essays/toward-a-new-clay", "/posts/toward-a-new-clay", "/blog/clay"]
6+ "/posts/essays/toward-a-new-clay",
7+ "/posts/toward-a-new-clay",
7author = "Curtis Yarvin"
12author = "Curtis Yarvin"
8ship = "~sorreg-namtyv"
13ship = "~sorreg-namtyv"
11TLDR: (a) Urbit's revision-control system, `%clay`, is itself due
16TLDR: (a) Urbit's revision-control system, `%clay`, is itself due
12−for a (medium-sized) revision! (b) this time, you yourself can
17+for a (medium-sized) revision! (b) this time, you yourself can
13−really help! (c) please don't give away the buried spoiler!
18+really help! (c) please don't give away the buried spoiler!
15This explanation does not (or at least should not) require you to
20This explanation does not (or at least should not) require you to
16know anything about Urbit.
21know anything about Urbit.
18### But %clay is already pretty cool
23### But %clay is already pretty cool
20−Of course, nothing in Urbit is allowed to be _lame_. `%clay` is
25+Of course, nothing in Urbit is allowed to be _lame_. `%clay` is
21−actually quite cool. It does need some work. But first, let's
26+actually quite cool. It does need some work. But first, let's
22talk about what it is and why it's cool.
27talk about what it is and why it's cool.
24`%clay` is the Arvo vane (Urbit kernel module) for revision
29`%clay` is the Arvo vane (Urbit kernel module) for revision
25−control. It's an authenticated, human-memorable global immutable
30+control. It's an authenticated, human-memorable global immutable
26−namespace which is also a typed revision-control system. It's
31+namespace which is also a typed revision-control system. It's
27−also very good at subscription and synchronization. Urbit
32+also very good at subscription and synchronization. Urbit
28currently uses `%clay` to distribute live hotpatches; we
33currently uses `%clay` to distribute live hotpatches; we
29transparently upgrade the whole network with one checkin.
34transparently upgrade the whole network with one checkin.
31### Compared to other revision-control systems
36### Compared to other revision-control systems
33−`%clay` is clearly a vanilla DVCS in the modern `git` style. Of
38+`%clay` is clearly a vanilla DVCS in the modern `git` style. Of
34deployed DVCS these days, `%clay` is probably closest in spirit
39deployed DVCS these days, `%clay` is probably closest in spirit
35to `bzr` or maybe `fossil`.
40to `bzr` or maybe `fossil`.
37Obviously, as a part of a larger whole, you would expect any
42Obviously, as a part of a larger whole, you would expect any
38−Urbit RCS to be brutally spartan. `%clay` is. It's even more
43+Urbit RCS to be brutally spartan. `%clay` is. It's even more
39spartan than `bzr`.
44spartan than `bzr`.
41−(In fact, it's really too spartan to fully use at the moment. We
46+(In fact, it's really too spartan to fully use at the moment. We
42ourselves still use `%clay` mainly as a distribution system, and
47ourselves still use `%clay` mainly as a distribution system, and
43−do all internal development on GitHub. This is because the
48+do all internal development on GitHub. This is because the
44present `%clay` lacks a few little conveniences, like commit
49present `%clay` lacks a few little conveniences, like commit
45−messages. See upgrades, below.)
50+messages. See upgrades, below.)
47−There are lots of normal DVCS things in `%clay`. But there are
52+There are lots of normal DVCS things in `%clay`. But there are
48−also cool and unusual ones. Let's take a brief tour of the
53+also cool and unusual ones. Let's take a brief tour of the
49high-level system, then describe what we think should change
54high-level system, then describe what we think should change
52### Anatomy of a DVCS
57### Anatomy of a DVCS
54−A DVCS is a historical state graph. A point in this graph is a
59+A DVCS is a historical state graph. A point in this graph is a
55snapshot of some tree-structured system, like a filesystem.
60snapshot of some tree-structured system, like a filesystem.
57These snapshots, including recursive internal nodes, are stored in
62These snapshots, including recursive internal nodes, are stored in
58−a content-addressable store. Pointers in this store are used to
63+a content-addressable store. Pointers in this store are used to
59construct a revision graph, whose edges are edit and/or merge
64construct a revision graph, whose edges are edit and/or merge
60−actions. Some sort of namespace attaches meaningful names to
65+actions. Some sort of namespace attaches meaningful names to
61current revisions in this graph.
66current revisions in this graph.
63There are six big questions in the design of a DVCS:
68There are six big questions in the design of a DVCS:
@@ -73,16 +78,16 @@ There are six big questions in the design of a DVCS:
74A good way to summarize `%clay` (at present) is that its answers
79A good way to summarize `%clay` (at present) is that its answers
75to the first four questions are interesting, and its answers to
80to the first four questions are interesting, and its answers to
76−the last two are boring. We'll talk about all six, though.
81+the last two are boring. We'll talk about all six, though.
78### Global namespace: [ship desk case]
83### Global namespace: [ship desk case]
80−The permanent global name of a `%clay` snapshot is a `++beak`. A beak
85+The permanent global name of a `%clay` snapshot is a `++beak`. A beak
81is a triple: `[ship desk case]`.
86is a triple: `[ship desk case]`.
83−The `++ship` is an Urbit ship. A ship is a network address, a
88+The `++ship` is an Urbit ship. A ship is a network address, a
84−human-memorable name and a cryptographic identity. The address
89+human-memorable name and a cryptographic identity. The address
85−width is variable; shorter addresses are more desirable. Most
90+width is variable; shorter addresses are more desirable. Most
86users have 32-bit addresses, which we map to four-syllable
91users have 32-bit addresses, which we map to four-syllable
87synthetic names like `~sorreg-namtyv`.
92synthetic names like `~sorreg-namtyv`.
@@ -95,44 +100,44 @@ In the Urbit path syntax, the beak is the first three segments of
95the path: `/ship/desk/case/my/cat/picture`.
100the path: `/ship/desk/case/my/cat/picture`.
97In most cases, a path parser has a default ship, desk and case,
102In most cases, a path parser has a default ship, desk and case,
98−each of which can be replaced by `=`. So `/=home=/my/cat/picture`
103+each of which can be replaced by `=`. So `/=home=/my/cat/picture`
99is the data at `/my/cat/picture` on your own ship, in the `%home`
104is the data at `/my/cat/picture` on your own ship, in the `%home`
100−desk, right now. Immutability doesn't have to be inconvenient.
105+desk, right now. Immutability doesn't have to be inconvenient.
102−`%clay` has no concept of a mutable name or a magic `HEAD`. Use
107+`%clay` has no concept of a mutable name or a magic `HEAD`. Use
103−the current date. If you're referring to remote data and you're
108+the current date. If you're referring to remote data and you're
104worried about clock skew, subtract a second or three, or be
109worried about clock skew, subtract a second or three, or be
105−prepared to block. (Or keep your clock unskewed, which shouldn't
110+prepared to block. (Or keep your clock unskewed, which shouldn't
106−be _that_ hard.) It's not like mutable-name schemes work any
111+be _that_ hard.) It's not like mutable-name schemes work any
107better than this anyway.
112better than this anyway.
109−(Immutability is central to `%clay`. `git` merely adopted the
114+(Immutability is central to `%clay`. `git` merely adopted the
110−reflog. `%clay` was born in it, molded by it...)
115+reflog. `%clay` was born in it, molded by it...)
112`%clay` is objective; each Urbit node hosts only one ship, but
117`%clay` is objective; each Urbit node hosts only one ship, but
113−treats domestic and foreign data (almost) the same way. If you
118+treats domestic and foreign data (almost) the same way. If you
114ask your `%clay` repository for someone else's data, it will
119ask your `%clay` repository for someone else's data, it will
115block the request until it gets the data, which it then caches.
120block the request until it gets the data, which it then caches.
116At present, we always fulfill requests from the origin ship,
121At present, we always fulfill requests from the origin ship,
117though in future we could certainly use it as a tracker.
122though in future we could certainly use it as a tracker.
119If `%clay` discards cached data, it keeps a hash, so that your
124If `%clay` discards cached data, it keeps a hash, so that your
120−urbit can't violate the local illusion of immutability. We
125+urbit can't violate the local illusion of immutability. We
121can't, of course, prevent anyone else's ship from sending
126can't, of course, prevent anyone else's ship from sending
122−conflicting claims in different directions. But we can prevent
127+conflicting claims in different directions. But we can prevent
123programs on our own ship from hearing about them, meaning that we
128programs on our own ship from hearing about them, meaning that we
124retain local referential transparency.
129retain local referential transparency.
126### Local structure: dome, arch, leaf and mark
131### Local structure: dome, arch, leaf and mark
128−A `%clay` snapshot is called a `++dome`. A dome is the `%clay`
133+A `%clay` snapshot is called a `++dome`. A dome is the `%clay`
129equivalent of a tree-structured filesystem.
134equivalent of a tree-structured filesystem.
131Each node in this tree is one uniform structure, an `++arch`,
136Each node in this tree is one uniform structure, an `++arch`,
132−which is both file and directory. A `++leaf` or data value in
137+which is both file and directory. A `++leaf` or data value in
133this arch is a typed document.
138this arch is a typed document.
135−A logical document type is called a `++mark`. A mark is a
140+A logical document type is called a `++mark`. A mark is a
136symbolic pointer to a Hoon source file, in the same dome, for an
141symbolic pointer to a Hoon source file, in the same dome, for an
137engine that can validate, compare, and/or convert
142engine that can validate, compare, and/or convert
138documents of that mark.
143documents of that mark.
@@ -141,8 +146,8 @@ Let's go over arches and marks a little more closely...
142#### An arch is a uniform filesystem node
147#### An arch is a uniform filesystem node
144−Most filesystems and RCS have modal nodes. For instance, a git
149+Most filesystems and RCS have modal nodes. For instance, a git
145−data node can be a blob or a tree. `%clay` (and the Urbit
150+data node can be a blob or a tree. `%clay` (and the Urbit
146namespace in general) has one kind of node, an `arch`:
151namespace in general) has one kind of node, an `arch`:
@@ -151,24 +156,24 @@ namespace in general) has one kind of node, an `arch`:
153An arch has an optional typed-data leaf, and optional named
158An arch has an optional typed-data leaf, and optional named
154−children. In Unix terms, it's a file, a directory, or both. It
159+children. In Unix terms, it's a file, a directory, or both. It
155can't be neither: an empty arch, with no data and no children, is
160can't be neither: an empty arch, with no data and no children, is
156−auto-pruned. As in `git` (but not in Unix), the hierarchy is a
161+auto-pruned. As in `git` (but not in Unix), the hierarchy is a
157−pure function of its leaves. Paths exist only because of the
162+pure function of its leaves. Paths exist only because of the
158data they point to.
163data they point to.
160The `++knot` names in an arch path are also a little special.
165The `++knot` names in an arch path are also a little special.
161They're typed, in a sense; any knot can be parsed invertibly into
166They're typed, in a sense; any knot can be parsed invertibly into
162−a typed atom. The atom is an arbitrary unsigned integer. Its
167+a typed atom. The atom is an arbitrary unsigned integer. Its
163type (an `++aura` if you know Hoon) defines the semantics and
168type (an `++aura` if you know Hoon) defines the semantics and
164presentation of the atom: for instance, a symbol, a decimal,
169presentation of the atom: for instance, a symbol, a decimal,
165a date, etc.
170a date, etc.
167Ultimately a filesystem is a database, and a path is a primary
172Ultimately a filesystem is a database, and a path is a primary
168−key. It's very useful to be able to store simple data in this
173+key. It's very useful to be able to store simple data in this
169key, in a standard and unambiguous way.
174key, in a standard and unambiguous way.
171−Why uniform nodes? One, they map nicely to information trees
176+Why uniform nodes? One, they map nicely to information trees
172where a subtree may need some custom collective presentation,
177where a subtree may need some custom collective presentation,
173not an automatically generated Apache style directory listing.
178not an automatically generated Apache style directory listing.
174Two, non-uniform nodes introduce a whole pointless category of
179Two, non-uniform nodes introduce a whole pointless category of
@@ -176,34 +181,34 @@ node type conflicts -- for which life is far too short.
177#### A mark is an executable document type
182#### A mark is an executable document type
179−A mark is "like an executable MIME type." It's best defined as a
184+A mark is "like an executable MIME type." It's best defined as a
180symbolic mapping from a filesystem to a schema engine.
185symbolic mapping from a filesystem to a schema engine.
182If you have a MIME type `text/html`, you need to watch the
187If you have a MIME type `text/html`, you need to watch the
183−Internets to track what the W3C thinks HTML is. If you have an
188+Internets to track what the W3C thinks HTML is. If you have an
184Urbit mark `%html`, you can functionally construct (using our
189Urbit mark `%html`, you can functionally construct (using our
185insanely powerful functional build vane `%ford`) a core which
190insanely powerful functional build vane `%ford`) a core which
186can validate or normalize HTML, apply patch and diff logic, and
191can validate or normalize HTML, apply patch and diff logic, and
187−translate to and from other marks. `%ford` can even reason about
192+translate to and from other marks. `%ford` can even reason about
188translation paths and construct multistep conversions.
193translation paths and construct multistep conversions.
190(Urbit does not actually have this super-smart `%html` mark right
195(Urbit does not actually have this super-smart `%html` mark right
191−now. But it probably should, huh. And this page was composed in
196+now. But it probably should, huh. And this page was composed in
192Markdown and translated to HTML by `%ford`.)
197Markdown and translated to HTML by `%ford`.)
194−Where does the source code of a mark live? Within the same dome
199+Where does the source code of a mark live? Within the same dome
195−as the data it describes. `%html` files are described by the
200+as the data it describes. `%html` files are described by the
196Hoon source at `/===/mar/html`, which perhaps gets mapped into
201Hoon source at `/===/mar/html`, which perhaps gets mapped into
197your Unix filesystem at `$ship/home/mar/html.hoon`.
202your Unix filesystem at `$ship/home/mar/html.hoon`.
199−This Hoon file is not self-contained. It has dependencies. The
204+This Hoon file is not self-contained. It has dependencies. The
200`%html` mark may depend on various structures and libraries
205`%html` mark may depend on various structures and libraries
201−elsewhere in Urbit. But the source code for all them is within
206+elsewhere in Urbit. But the source code for all them is within
202this dome.
207this dome.
204One constraint this puts on domes is that _all domes are
209One constraint this puts on domes is that _all domes are
205−descendants of the base system dome_. Domes are versions of one
210+descendants of the base system dome_. Domes are versions of one
206−reality, not fragments of that reality. You can't create a new
211+reality, not fragments of that reality. You can't create a new
207empty dome -- you'd have no mark source, and be unable to install
212empty dome -- you'd have no mark source, and be unable to install
208any validated files.
213any validated files.
@@ -214,13 +219,13 @@ weird collisions between non-parallel things don't happen.
215### Unix mapping and mounting
220### Unix mapping and mounting
217−Every DVCS is its own filesystem in a sense. But every DVCS
222+Every DVCS is its own filesystem in a sense. But every DVCS
218works by mapping and mounting itself, in some sense, to Unix.
223works by mapping and mounting itself, in some sense, to Unix.
220#### Mapping arches to Unix
225#### Mapping arches to Unix
222−How do we map arches into Unix? The arch structure maps 1:1 to
227+How do we map arches into Unix? The arch structure maps 1:1 to
223−the Unix directory tree. When the arch has a leaf, we derive a
228+the Unix directory tree. When the arch has a leaf, we derive a
224dot extension from the file's mark.
229dot extension from the file's mark.
226So, supposing we have a leaf `foo` which is an HTML document
231So, supposing we have a leaf `foo` which is an HTML document
@@ -234,19 +239,19 @@ foo/moo.html
236This maps invertibly back into `%clay`, and it looks perfectly
241This maps invertibly back into `%clay`, and it looks perfectly
237−unremarkable from the Unix side. File completion sometimes gets
242+unremarkable from the Unix side. File completion sometimes gets
238a little annoying.
243a little annoying.
240It would be if nice every possible Unix tree, even every
245It would be if nice every possible Unix tree, even every
241plausible Unix tree, mapped cleanly and completely to a `%clay`
246plausible Unix tree, mapped cleanly and completely to a `%clay`
242−dome. There are always horrid exceptions, even in common use --
247+dome. There are always horrid exceptions, even in common use --
243−like extensionless Makefiles. The trivial solution is that if
248+like extensionless Makefiles. The trivial solution is that if
244`%clay` finds a file mysterious, it won't track it.
249`%clay` finds a file mysterious, it won't track it.
246One often unquestioned assumption in new system software is that
251One often unquestioned assumption in new system software is that
247−compatibility with old content is important. It's nice, but not
252+compatibility with old content is important. It's nice, but not
248−that important. Most content in a new system is new content.
253+that important. Most content in a new system is new content.
249−The important compatibility direction is export, not import. So
254+The important compatibility direction is export, not import. So
250the new system should subset, not superset, the ancient formats.
255the new system should subset, not superset, the ancient formats.
252### Mounting desks to Unix
257### Mounting desks to Unix
@@ -257,36 +262,36 @@ the Unix directory of the same name within the pier.
257When files in a mount directory change, `%clay` notices it
262When files in a mount directory change, `%clay` notices it
258through `libuv` filesystem watching (ultimately `inotify()` or
263through `libuv` filesystem watching (ultimately `inotify()` or
259`FSEvents`), computes diffs, and automatically commits them to
264`FSEvents`), computes diffs, and automatically commits them to
260−the desk. And if `%clay` creates a new version for its own
265+the desk. And if `%clay` creates a new version for its own
261urbity reasons, it will revise the Unix files.
266urbity reasons, it will revise the Unix files.
263Keeping all mounted files in the same Unix subdirectory makes it
268Keeping all mounted files in the same Unix subdirectory makes it
264−easy to move the whole pier in a tarfile. Edits applied while
269+easy to move the whole pier in a tarfile. Edits applied while
265the `urbit` process isn't running will be detected and applied
270the `urbit` process isn't running will be detected and applied
266the next time it starts.
271the next time it starts.
268In conventional DVCS terms, `%clay` has a 1:1 static mapping from
273In conventional DVCS terms, `%clay` has a 1:1 static mapping from
269−branches to working directories. Why?
274+branches to working directories. Why?
271−Why not? Do we need that "what branches are checked out in what
276+Why not? Do we need that "what branches are checked out in what
272−directories" piece of state? Why force the user to worry about
277+directories" piece of state? Why force the user to worry about
273−which branches are mounted to which directories? Why not just
278+which branches are mounted to which directories? Why not just
274−mount them under their own names? Is there some crippling global
279+mount them under their own names? Is there some crippling global
275shortage of disk space for source files? Our planet is running
280shortage of disk space for source files? Our planet is running
276−out of inodes? Wut?
281+out of inodes? Wut?
278### Queries, subscriptions and synchronization
283### Queries, subscriptions and synchronization
280There are two ways to get data out of `%clay`: dereference the Urbit
285There are two ways to get data out of `%clay`: dereference the Urbit
281−namespace, or send a request/subscription. Sync is in userspace.
286+namespace, or send a request/subscription. Sync is in userspace.
283#### Dereferencing the namespace
288#### Dereferencing the namespace
285−`%clay` is just part of the full Urbit namespace. The full namespace
290+`%clay` is just part of the full Urbit namespace. The full namespace
286has a two-character prefix which expresses what vane to query
291has a two-character prefix which expresses what vane to query
287−(always `%c` for `%clay`). The suffix is a mode flag; `%x` for
292+(always `%c` for `%clay`). The suffix is a mode flag; `%x` for
288data, `%y` for a list of children, `%z` for the whole arch
293data, `%y` for a list of children, `%z` for the whole arch
289−subtree, `%w` for the version number. (Each vane has its own
294+subtree, `%w` for the version number. (Each vane has its own
292For example, the full path to my cat picture is
297For example, the full path to my cat picture is
@@ -300,6 +305,7 @@ For example, the full path to my cat picture is
300If the data model is `++image` (hypothetical -- we don't really
305If the data model is `++image` (hypothetical -- we don't really
301do image processing in Urbit right now), the normal way to load
306do image processing in Urbit right now), the normal way to load
302this data in a Hoon program is
307this data in a Hoon program is
304.^(image /cx/~sorreg-namtyv/home/[now]/my/cat/picture)
310.^(image /cx/~sorreg-namtyv/home/[now]/my/cat/picture)
@@ -316,15 +322,15 @@ learn it in future, `[~ ~]` if we know we'll never know it, or
316`[~ ~ value]` if we know it now.
322`[~ ~ value]` if we know it now.
318In high-level shell programming (`:dojo`), we can make apparently
324In high-level shell programming (`:dojo`), we can make apparently
319−"blocking" requests to `%clay`. What actually happens: when `%clay`
325+"blocking" requests to `%clay`. What actually happens: when `%clay`
320produces `~`, we stop trying to compute this expression and wait
326produces `~`, we stop trying to compute this expression and wait
321−on one or more blocking requests. When we are unblocked, we
327+on one or more blocking requests. When we are unblocked, we
322recompute again from scratch; the intermediate computation is not
328recompute again from scratch; the intermediate computation is not
323saved at all.
329saved at all.
325−Urbit doesn't like to block. It likes to save coherent data
331+Urbit doesn't like to block. It likes to save coherent data
326−structures, not incomplete computation contexts. When Urbit
332+structures, not incomplete computation contexts. When Urbit
327−pretends to block for you, this is a courtesy service. You
333+pretends to block for you, this is a courtesy service. You
328can return the courtesy of not expecting miracles from it.
334can return the courtesy of not expecting miracles from it.
329When in doubt, send an actual request.
335When in doubt, send an actual request.
@@ -335,7 +341,7 @@ namespace, but can (a) read across case (revision) ranges,
335and (b) filter by subtree.
341and (b) filter by subtree.
337If the revision range in a request stretches into the future, the
343If the revision range in a request stretches into the future, the
338−request becomes a subscription. Urbit messaging always regards a
344+request becomes a subscription. Urbit messaging always regards a
339single response as a special case of a subscription.
345single response as a special case of a subscription.
341Generally, the response stream created by a request starts with
347Generally, the response stream created by a request starts with
@@ -345,10 +351,10 @@ closed when the request range is complete.
346#### Userspace synchronization
352#### Userspace synchronization
348−Synchronization per se is done in userspace. In `%clay`, sync just
354+Synchronization per se is done in userspace. In `%clay`, sync just
349means an application (typically the `%kiln` module in the system
355means an application (typically the `%kiln` module in the system
350app `:hood`) subscribes to a remote desk, and saves its change
356app `:hood`) subscribes to a remote desk, and saves its change
351−stream locally. So sync is really an emergent feature of `%clay`.
357+stream locally. So sync is really an emergent feature of `%clay`.
353### Revision graph
359### Revision graph
@@ -356,16 +362,16 @@ stream locally. So sync is really an emergent feature of `%clay`.
356Some changes are proposed below, however.
362Some changes are proposed below, however.
358As in any DVCS, a dome is the child of one parent dome (an edit)
364As in any DVCS, a dome is the child of one parent dome (an edit)
359−or more parents (a merge). Often the parent of any revision is
365+or more parents (a merge). Often the parent of any revision is
360the previous version in the sequence, but it can be anything.
366the previous version in the sequence, but it can be anything.
362### Storage model
368### Storage model
364−`%clay`'s storage model is completely uninteresting. It's also a
370+`%clay`'s storage model is completely uninteresting. It's also a
365little funky and I want to change it, so I'll be super vague.
371little funky and I want to change it, so I'll be super vague.
367A `++lobe`, a 128-bit atom, is the Merkle hash of a `blob`, which
373A `++lobe`, a 128-bit atom, is the Merkle hash of a `blob`, which
368−is essentially a leaf. All blobs are stored in a hash table
374+is essentially a leaf. All blobs are stored in a hash table
369(`(map lobe blob)`), and reference each other through lobes.
375(`(map lobe blob)`), and reference each other through lobes.
371At present we don't GC this space at all, because we don't
377At present we don't GC this space at all, because we don't
@@ -374,12 +380,12 @@ actually delete any roots.
374A `blob` may be `%direct`, meaning it's defined as the data
380A `blob` may be `%direct`, meaning it's defined as the data
375itself, or `%delta`, defined as a change against another blob.
381itself, or `%delta`, defined as a change against another blob.
376This is motivated by the desire not to store a full snapshot
382This is motivated by the desire not to store a full snapshot
377−on every file change. But it may be misguided (see changes #0
383+on every file change. But it may be misguided (see changes #0
378and #1 below).
384and #1 below).
380We cache the full arch tree (`++ankh`) of the latest state in
386We cache the full arch tree (`++ankh`) of the latest state in
381−each desk. For current queries, this mitigates a lot of the
387+each desk. For current queries, this mitigates a lot of the
382−opportunities for slowness that this data model creates. But the
388+opportunities for slowness that this data model creates. But the
383disparity between the speed of the current version and the
389disparity between the speed of the current version and the
384speed previous version is an undesirable glitch in the API.
390speed previous version is an undesirable glitch in the API.
@@ -391,38 +397,38 @@ We actually have _three_ motivations for change...
392At present, we're still using `git` as a development platform.
398At present, we're still using `git` as a development platform.
393`%clay` is certainly used, but just for its strength --
399`%clay` is certainly used, but just for its strength --
394−deployment. This is starving Urbit of essential experience and
400+deployment. This is starving Urbit of essential experience and
395−basically keeping it lame. So, change is necessary.
401+basically keeping it lame. So, change is necessary.
397### Change must come from the community
403### Change must come from the community
399−"Why?" you ask? "Why must change come from the community?" Well,
405+"Why?" you ask? "Why must change come from the community?" Well,
400−one, duh. Change always comes from the community.
406+one, duh. Change always comes from the community.
402−But two, a great force for change has departed. Philip Monk,
408+But two, a great force for change has departed. Philip Monk,
403late maintainer of `%clay` (`~wictuc-folrex` in the old era,
409late maintainer of `%clay` (`~wictuc-folrex` in the old era,
404−`~wicdev-wisryt` in the new), is tired of types and typing. He
410+`~wicdev-wisryt` in the new), is tired of types and typing. He
405spent September roaming with the bison in Montana, and is now
411spent September roaming with the bison in Montana, and is now
406−literally on his way to Patagonia. On the one hand, Phil still
412+literally on his way to Patagonia. On the one hand, Phil still
407exists and is not at all unwilling to answer questions, even dumb
413exists and is not at all unwilling to answer questions, even dumb
408−questions. On the other hand, _Patagonia_.
414+questions. On the other hand, _Patagonia_.
410−So %clay needs a new owner, or owners. At this point, we think
416+So %clay needs a new owner, or owners. At this point, we think
411the community might be indoctrinated enough to take a stab.
417the community might be indoctrinated enough to take a stab.
413Alas, `%clay` already has a pretty checkered ownership history.
419Alas, `%clay` already has a pretty checkered ownership history.
414Long ago I built a `%clay` that was really the only serious
420Long ago I built a `%clay` that was really the only serious
415−design failure in Arvo. Then Jared Hance, a brilliant summer
421+design failure in Arvo. Then Jared Hance, a brilliant summer
416intern, took my broken `%clay` and reformed it into a minimal `git`
422intern, took my broken `%clay` and reformed it into a minimal `git`
417−in about a week. Then Philip modified Jared's `%clay` to be a
423+in about a week. Then Philip modified Jared's `%clay` to be a
418_typed_ revision-control system, which is a pretty cool thing.
424_typed_ revision-control system, which is a pretty cool thing.
420But... in none of these transitions did `%clay` actually get
426But... in none of these transitions did `%clay` actually get
421−_rewritten_ per se. While it's a reasonably good vane, it
427+_rewritten_ per se. While it's a reasonably good vane, it
422does contain one or two historical decisions.
428does contain one or two historical decisions.
424We know a lot more about Arvo than we used to, and I think we can
430We know a lot more about Arvo than we used to, and I think we can
425−revisit many of these decisions. The point of this post is to
431+revisit many of these decisions. The point of this post is to
426start a `%clay` design discussion that will evolve into actual
432start a `%clay` design discussion that will evolve into actual
427collective action.
433collective action.
@@ -432,41 +438,41 @@ We're planning a rewards program for contributions to `%clay`.
432It'll simply involve setting a time period and a pool of real
438It'll simply involve setting a time period and a pool of real
433estate, and allocating the pool by judged contribution.
439estate, and allocating the pool by judged contribution.
435−This is an experimental program. We'll say more about it later.
441+This is an experimental program. We'll say more about it later.
436−It may not even happen. It may happen very differently. When in
442+It may not even happen. It may happen very differently. When in
437doubt, don't talk about Fight Club.
443doubt, don't talk about Fight Club.
439## Some openings for change
445## Some openings for change
441−Change, of course, must come from the community. No one can
447+Change, of course, must come from the community. No one can
442−dictate change from the top down. We _know_ this. But...
448+dictate change from the top down. We _know_ this. But...
444This list of suggested changes for the next `%clay` is not in any
450This list of suggested changes for the next `%clay` is not in any
445−sort of order: priority, structure, difficulty, etc. Let's talk
451+sort of order: priority, structure, difficulty, etc. Let's talk
446−about the changes first; make actual plans second. The numbers
452+about the changes first; make actual plans second. The numbers
447here are just for convenience.
453here are just for convenience.
449−*All changes in this program will happen on the next-generation
455+_All changes in this program will happen on the next-generation
450−(`cc-release`) branch.*
456+(`cc-release`) branch._
452### Change #0: state reform
458### Change #0: state reform
454As mentioned above, the storage model in `%clay`, as the code is
460As mentioned above, the storage model in `%clay`, as the code is
455−at present, feels a little funky to me. There's nothing terribly
461+at present, feels a little funky to me. There's nothing terribly
456awful about it, but it doesn't feel quite right and its scaling
462awful about it, but it doesn't feel quite right and its scaling
457properties are very questionable.
463properties are very questionable.
459Briefly, `%clay` today actually derives the hierarchy from its
465Briefly, `%clay` today actually derives the hierarchy from its
460−leaves -- rather than _logically_ so deriving it. Only leaves
466+leaves -- rather than _logically_ so deriving it. Only leaves
461−are in the object store. This is just weird, and you can feel
467+are in the object store. This is just weird, and you can feel
462−`%clay` trying to be something other than what it is. (I blame
468+`%clay` trying to be something other than what it is. (I blame
463Jared, though it's actually my fault.)
469Jared, though it's actually my fault.)
465To store a dome snapshot (`++yaki`), the present code keeps a
471To store a dome snapshot (`++yaki`), the present code keeps a
466−`(map path blob)` at the root of the dome. This is enough to
472+`(map path blob)` at the root of the dome. This is enough to
467reconstruct the `++arch` structures the namespace exports.
473reconstruct the `++arch` structures the namespace exports.
468However, all kinds of basic arch operations, like basically `ls`,
474However, all kinds of basic arch operations, like basically `ls`,
469−become `O(n)` in the number of leaves in the dome. This is
475+become `O(n)` in the number of leaves in the dome. This is
470obviously not quite right.
476obviously not quite right.
472I believe the storage structure should be (removing everything
478I believe the storage structure should be (removing everything
@@ -487,46 +493,46 @@ Here a `++root` is the universal node, with children addressed by
487hash, but logically representing the abstract `++arch`.
493hash, but logically representing the abstract `++arch`.
489Users of hierarchical filesystems basically expect performance
495Users of hierarchical filesystems basically expect performance
490−qualities that match the filesystem structure. A huge directory
496+qualities that match the filesystem structure. A huge directory
491−in one place shouldn't affect search speeds somewhere else. And
497+in one place shouldn't affect search speeds somewhere else. And
492of course, `ls` can't be O(n) in the size of the filesystem.
498of course, `ls` can't be O(n) in the size of the filesystem.
494### Change #1: store edges, cache snapshots
500### Change #1: store edges, cache snapshots
496−The current `%clay` is snapshot-oriented, like `git`. The actual
502+The current `%clay` is snapshot-oriented, like `git`. The actual
497−changes are discarded. I think this is a bad choice.
503+changes are discarded. I think this is a bad choice.
499`%clay` is not a research project, and has no idea of going as
505`%clay` is not a research project, and has no idea of going as
500−far as `darcs` in the "set-of-patches" direction. However, at
506+far as `darcs` in the "set-of-patches" direction. However, at
501present we actually throw away the patch (`++nori`).
507present we actually throw away the patch (`++nori`).
503I think the right DVCS design for `%clay` is a snapshot-oriented
509I think the right DVCS design for `%clay` is a snapshot-oriented
504−system which also keeps the patches. The revision graph has the
510+system which also keeps the patches. The revision graph has the
505−same general shape as in the old `git` style. But rather than
511+same general shape as in the old `git` style. But rather than
506just deriving the diffs from the snapshots, we can also derive
512just deriving the diffs from the snapshots, we can also derive
507−the snapshots from the diffs. When in doubt, we save space by
513+the snapshots from the diffs. When in doubt, we save space by
508dropping the snapshot, not the diff.
514dropping the snapshot, not the diff.
510For an RCS which is also a general-purpose typed filesystem, it's
516For an RCS which is also a general-purpose typed filesystem, it's
511clear that considering edges (deltas, diffs, changes, etc) as the
517clear that considering edges (deltas, diffs, changes, etc) as the
512−primary state is optimal. You can use diff functions to derive
518+primary state is optimal. You can use diff functions to derive
513deltas from snapshots, but in many data models this will always
519deltas from snapshots, but in many data models this will always
514−risk losing information from the original action. When in
520+risk losing information from the original action. When in
515doubt, save what the user did.
521doubt, save what the user did.
517This is even more true for operations on the tree structure, not
523This is even more true for operations on the tree structure, not
518−on individual leaves. Consider the heuristic energy that `git` has
524+on individual leaves. Consider the heuristic energy that `git` has
519to go through to figure out when `mv` has been used -- even when
525to go through to figure out when `mv` has been used -- even when
520−`git mv` has been used! (According to the `git` faq, this is
526+`git mv` has been used! (According to the `git` faq, this is
521because the lowest common denominator remains Linus' style of
527because the lowest common denominator remains Linus' style of
522working with emailed patches.)
528working with emailed patches.)
524−Edges probably don't even need to be deduplicated. They also can
530+Edges probably don't even need to be deduplicated. They also can
525contain metadata such as commit messages (which `%clay` has no
531contain metadata such as commit messages (which `%clay` has no
526place for at present).
532place for at present).
528Once edges are recognized as unreclaimed primary state, snapshots
534Once edges are recognized as unreclaimed primary state, snapshots
529−can always be derived from the edges. So we can reclaim memory
535+can always be derived from the edges. So we can reclaim memory
530being used for snapshots.
536being used for snapshots.
532It's also straightforward to have a kind of edge which is a list
538It's also straightforward to have a kind of edge which is a list
@@ -534,27 +540,27 @@ of edges, letting us collapse feature branches trivially.
535Finally, I think references in edges should wherever possible
541Finally, I think references in edges should wherever possible
536indirect through the namespace, rather than directly to the
542indirect through the namespace, rather than directly to the
537−content address. The user has strictly more information if you
543+content address. The user has strictly more information if you
538say you merged `/~sorreg-namtyv/base/987/`, than if all you have
544say you merged `/~sorreg-namtyv/base/987/`, than if all you have
539is the hash that this reference refers to.
545is the hash that this reference refers to.
541### Change #2: unconfuse merge and rebase
547### Change #2: unconfuse merge and rebase
543The dispute between merge and rebase has all the bad smells of
549The dispute between merge and rebase has all the bad smells of
544−a historical wart. But here I'm just not sure what's right.
550+a historical wart. But here I'm just not sure what's right.
546The user has one _intent_ whether merging or rebasing, which is
552The user has one _intent_ whether merging or rebasing, which is
547−to combine changes on one desk into another. The _algorithm_
553+to combine changes on one desk into another. The _algorithm_
548used to construct this change, whether operational transformation
554used to construct this change, whether operational transformation
549(rebasing) or a 3-way diff (merging) has one goal: use all
555(rebasing) or a 3-way diff (merging) has one goal: use all
550available information to model the authors' editing intent as
556available information to model the authors' editing intent as
551−realistically as possible. If we save edges, we should use
557+realistically as possible. If we save edges, we should use
554Whether to define the resulting change graph with the typical
560Whether to define the resulting change graph with the typical
555structure of a merge, or the typical structure of a rebase,
561structure of a merge, or the typical structure of a rebase,
556strikes me as quite arbitrary, and well into the land of "pick
562strikes me as quite arbitrary, and well into the land of "pick
557−one and force everyone to stick with it." But this call demands
563+one and force everyone to stick with it." But this call demands
558a real expert on RCS workflow, which I'm not.
564a real expert on RCS workflow, which I'm not.
560Since `%clay` is a humble part of a larger system and needs to
566Since `%clay` is a humble part of a larger system and needs to
@@ -564,23 +570,23 @@ different preferences" is not usually the right answer.
564### Change #3: comment jihad
570### Change #3: comment jihad
566`%clay` needs to be brought up to the documentation standards of
572`%clay` needs to be brought up to the documentation standards of
567−`%jael`. This means (a) the formatting matches; (b) the comments
573+`%jael`. This means (a) the formatting matches; (b) the comments
568are at the same level of quality and ubiquity.
574are at the same level of quality and ubiquity.
570The basic commenting standards are: (a) every core (class) needs
576The basic commenting standards are: (a) every core (class) needs
571a block comment explanation; (b) every gate (function) needs a
577a block comment explanation; (b) every gate (function) needs a
572line comment; (c) every face (variable) needs a line comment.
578line comment; (c) every face (variable) needs a line comment.
574−This is obviously not just a clerical task. To write these
580+This is obviously not just a clerical task. To write these
575−comments, you need to actually understand the code. This is a
581+comments, you need to actually understand the code. This is a
576nontrivial but extremely rewarding task, even if the code you're
582nontrivial but extremely rewarding task, even if the code you're
577commenting will get replaced.
583commenting will get replaced.
579Also, the _symbols_ in `%clay` must be brought up to `%jael`
585Also, the _symbols_ in `%clay` must be brought up to `%jael`
580−quality. There is still far too much four-letterism here.
586+quality. There is still far too much four-letterism here.
582NB: `%clay` is about 3000 lines of pretty clean Hoon, which is
588NB: `%clay` is about 3000 lines of pretty clean Hoon, which is
583−certainly not at all undocumented. We're not exactly looking at
589+certainly not at all undocumented. We're not exactly looking at
584a Matterhorn of reverse engineering here.
590a Matterhorn of reverse engineering here.
586### Change #4: knot to coin internal paths
592### Change #4: knot to coin internal paths
@@ -591,26 +597,26 @@ ASCII symbol as an atom) must be within the range of the
591invertible noun printer, `++co`.
597invertible noun printer, `++co`.
593`++co` maps a described data value, a `++coin`, to an ASCII
599`++co` maps a described data value, a `++coin`, to an ASCII
594−string which is a valid URL segment and Unix filename. Its
600+string which is a valid URL segment and Unix filename. Its
595friend `++so` inverts the function, perfectly.
601friend `++so` inverts the function, perfectly.
597−Almost every knot is the simplest kind of `coin`, a `dime.` A
603+Almost every knot is the simplest kind of `coin`, a `dime.` A
598dime is a single atom with a rendering type (`aura` if you know
604dime is a single atom with a rendering type (`aura` if you know
599−Hoon). Auras include `@tas` for symbolic text, `@ud` for
605+Hoon). Auras include `@tas` for symbolic text, `@ud` for
600−unsigned decimal, `@p` for Urbit ship, etc. `++co` can also
606+unsigned decimal, `@p` for Urbit ship, etc. `++co` can also
601print arbitrary cells, though less prettily.
607print arbitrary cells, though less prettily.
603This feature is cool because it lets us stuff data into names in
609This feature is cool because it lets us stuff data into names in
604−a precise and predictable way. Usually the names in a namespace
610+a precise and predictable way. Usually the names in a namespace
605are human-meaningful and human-originated, and are therefore
611are human-meaningful and human-originated, and are therefore
606−symbolic, but there are exceptions. For example, blogposts may
612+symbolic, but there are exceptions. For example, blogposts may
607want to be named by date.
613want to be named by date.
609However, it is slightly retarded for the fundamental state of our
615However, it is slightly retarded for the fundamental state of our
610arches to store the _rendered_ string (`++knot`), not the source
616arches to store the _rendered_ string (`++knot`), not the source
611−data (`++coin`). It leads to all kinds of inefficiencies where
617+data (`++coin`). It leads to all kinds of inefficiencies where
612−we're doing way more parsing than we want to. (Parsing also
618+we're doing way more parsing than we want to. (Parsing also
613−should be jet-propelled, but is not.) It also makes it way too
619+should be jet-propelled, but is not.) It also makes it way too
614easy to sneak in bad data and make it `%clay`'s problem.
620easy to sneak in bad data and make it `%clay`'s problem.
616So, logically, we'd be shifting from
622So, logically, we'd be shifting from
@@ -625,7 +631,7 @@ So, logically, we'd be shifting from
626Also, a vanilla `++map` might not be quite the right data
632Also, a vanilla `++map` might not be quite the right data
627structure at this point -- we probably want a custom sorted tree,
633structure at this point -- we probably want a custom sorted tree,
628−which separates auras and imposes a custom order on each. This
634+which separates auras and imposes a custom order on each. This
629would let us serve simple range queries quite efficiently, so
635would let us serve simple range queries quite efficiently, so
630long as they match the standard aura order (eg, alphabetical
636long as they match the standard aura order (eg, alphabetical
631order for symbols).
637order for symbols).
@@ -633,23 +639,23 @@ order for symbols).
633### Change #5: dome semantics by reference
639### Change #5: dome semantics by reference
635While the principle that a dome contains its own mark source is
641While the principle that a dome contains its own mark source is
636−elegant, it's a pain in the ass to implement properly. `%clay`
642+elegant, it's a pain in the ass to implement properly. `%clay`
637certainly does not do it justice at present.
643certainly does not do it justice at present.
639The complexity is in checking every change to see whether it
645The complexity is in checking every change to see whether it
640includes a change to the mark source (which is very unlikely),
646includes a change to the mark source (which is very unlikely),
641or a dependency of the mark source (slightly less unlikely).
647or a dependency of the mark source (slightly less unlikely).
642And if so, do we have to update all the leaves in the dome?
648And if so, do we have to update all the leaves in the dome?
643−This is also very unlikely. And yet, it happens.
649+This is also very unlikely. And yet, it happens.
645Instead, we could simply designate, by beak (ship, desk, and
651Instead, we could simply designate, by beak (ship, desk, and
646−case) a dome which contains the canonical mark source. This
652+case) a dome which contains the canonical mark source. This
647would rarely change -- you could upgrade a desk to a new mark
653would rarely change -- you could upgrade a desk to a new mark
648dome, but it would be a manual operation and probably a rare one.
654dome, but it would be a manual operation and probably a rare one.
650The common case would just be that everyone uses the current
656The common case would just be that everyone uses the current
651version of the `%base` desk when a desk is created, and never
657version of the `%base` desk when a desk is created, and never
652−changes it. In practice, this means not every two-byte edit
658+changes it. In practice, this means not every two-byte edit
653needs to get involved in detecting and/or applying schema
659needs to get involved in detecting and/or applying schema
654evolution, a welcome simplification.
660evolution, a welcome simplification.
@@ -661,14 +667,14 @@ design is one mark per message structure, the mark namespace gets
661overloaded fast.
667overloaded fast.
663So we added an app hierarchy within the mark namespace, using hep
669So we added an app hierarchy within the mark namespace, using hep
664−`-` as a divider. The `:talk` app expects messages with the mark
670+`-` as a divider. The `:talk` app expects messages with the mark
665`talk-message`, whose sources is in `/===/mar/talk/message`.
671`talk-message`, whose sources is in `/===/mar/talk/message`.
667This helps, but it doesn't solve another two problems: trivial
673This helps, but it doesn't solve another two problems: trivial
668nouns and containers.
674nouns and containers.
670It's retarded to have to add a source file to send a `@ud` or a
676It's retarded to have to add a source file to send a `@ud` or a
671−`*`. An aura (see above under `coin`) should be a trivial mark.
677+`*`. An aura (see above under `coin`) should be a trivial mark.
673Also, we need to be able to construct standard containers of
679Also, we need to be able to construct standard containers of
674marks: lists, maps, sets, and tuples (or, as a database person
680marks: lists, maps, sets, and tuples (or, as a database person
@@ -680,32 +686,32 @@ Another kind of complex mark that needs to exist: conflict,
680difference, and invalid marks.
686difference, and invalid marks.
682Conflict data and difference data are two types of noun that are
688Conflict data and difference data are two types of noun that are
683−generated in normal RCS operation. If we have the concept of an
689+generated in normal RCS operation. If we have the concept of an
684`%html` noun, we also need the concept of an invertible change
690`%html` noun, we also need the concept of an invertible change
685from one `%html` file to another; a merge conflict between two
691from one `%html` file to another; a merge conflict between two
686`%html` files; and even a conflict between two marks.
692`%html` files; and even a conflict between two marks.
688There are two kinds of changes to a dome: manual and automatic.
694There are two kinds of changes to a dome: manual and automatic.
689Manual changes are initiated by the user and should crash, unless
695Manual changes are initiated by the user and should crash, unless
690−manually overridden. Automatic changes should not crash, ever,
696+manually overridden. Automatic changes should not crash, ever,
691−and should not lose data. If they create problems, which they
697+and should not lose data. If they create problems, which they
692shouldn't, these problems will need to be cleaned up manually.
698shouldn't, these problems will need to be cleaned up manually.
694This is a complex space to navigate, but it requires a conflicted
700This is a complex space to navigate, but it requires a conflicted
695merge to be able to resolve to a valid dome containing conflict
701merge to be able to resolve to a valid dome containing conflict
696−leaves, not crash. The present `%clay` goes halfway there and
702+leaves, not crash. The present `%clay` goes halfway there and
697resolves to a scratch desk, but we can go farther: the whole `git`
703resolves to a scratch desk, but we can go farther: the whole `git`
698−dumpster-fire of "merge state" deserves destruction. One change
704+dumpster-fire of "merge state" deserves destruction. One change
699is a merge that produces conflicts, the next is an edit that
705is a merge that produces conflicts, the next is an edit that
700resolves them.
706resolves them.
702−Similarly, all sorts of errors can produce invalid leaves. One
708+Similarly, all sorts of errors can produce invalid leaves. One
703particularly rare case which is nonetheless important is schema
709particularly rare case which is nonetheless important is schema
704evolution: what happens to files in a mark, when the mark source
710evolution: what happens to files in a mark, when the mark source
705−code changes. They all need to be re-normalized. And this
711+code changes. They all need to be re-normalized. And this
706−operation can fail, producing an invalid value. (We don't do any
712+operation can fail, producing an invalid value. (We don't do any
707of this now, BTW, although `%ford` has the dependency mechanism to
713of this now, BTW, although `%ford` has the dependency mechanism to
708−make it possible.) Again, in some cases this should crash the
714+make it possible.) Again, in some cases this should crash the
709update to the mark source; in other cases, there is no local way
715update to the mark source; in other cases, there is no local way
710to handle the crash.
716to handle the crash.
@@ -716,46 +722,46 @@ into using the bad content.
717The bottom line: a mark needs to be split into two models:
723The bottom line: a mark needs to be split into two models:
718`++mark`, the original symbolic mark with its source engine, and
724`++mark`, the original symbolic mark with its source engine, and
719−something like `++logo`, the meta-mark or mark container. The
725+something like `++logo`, the meta-mark or mark container. The
720mark is the building block; the logo is the building.
726mark is the building block; the logo is the building.
722### Change #8: filesystem hierarchy standard
728### Change #8: filesystem hierarchy standard
724While all filesystems remain forks of one ur-filesystem, the
730While all filesystems remain forks of one ur-filesystem, the
725−standard tends to enforce itself. But not define itself.
731+standard tends to enforce itself. But not define itself.
727−I've always hated the current `/mar` for marks, for instance. I
733+I've always hated the current `/mar` for marks, for instance. I
728−prefer `/mac`. But others hate this. And so on. Also, by
734+prefer `/mac`. But others hate this. And so on. Also, by
729necessity, our nascent "FHS" is heavily biased toward system
735necessity, our nascent "FHS" is heavily biased toward system
730−data, away from user data. This is retarded.
736+data, away from user data. This is retarded.
732−Please focus all bikeshedding energy on this problem! Better to
738+Please focus all bikeshedding energy on this problem! Better to
733bikeshed about what color to paint the bikeshed, then what shape
739bikeshed about what color to paint the bikeshed, then what shape
734the bike gears should be.
740the bike gears should be.
736### Change #9: security and access control
742### Change #9: security and access control
738−Oh, yeah, that! They always say not to design security in at the
744+Oh, yeah, that! They always say not to design security in at the
739−last minute. Unfortunately, this is exactly what we did. There
745+last minute. Unfortunately, this is exactly what we did. There
740−is no security at all in present `%clay`. (There is also no push
746+is no security at all in present `%clay`. (There is also no push
741operation in `%clay`, so it's not as mad as you think -- anyone
747operation in `%clay`, so it's not as mad as you think -- anyone
742can read anything, no one can write anything, which is fine in a
748can read anything, no one can write anything, which is fine in a
743young network where there are no secrets.)
749young network where there are no secrets.)
745Fortunately, we don't have anything super fancy planned in this
751Fortunately, we don't have anything super fancy planned in this
746−area, just basic access-control lists. These obviously need to
752+area, just basic access-control lists. These obviously need to
747be designed along with the filesystem-hierarchy standard (see
753be designed along with the filesystem-hierarchy standard (see
748elsewhere).
754elsewhere).
750−Is it even possible to do ACLs properly in `%clay`? At this
756+Is it even possible to do ACLs properly in `%clay`? At this
751−point, yes. There is an event-oriented security model in Arvo,
757+point, yes. There is an event-oriented security model in Arvo,
752though it isn't really hooked up to anything at the moment.
758though it isn't really hooked up to anything at the moment.
753−There is also a privilege badge system in `%jael`. The
759+There is also a privilege badge system in `%jael`. The
754combination of these two features is quite sufficient for a
760combination of these two features is quite sufficient for a
755decent ACL model.
761decent ACL model.
757The idea of event-based security is that every event passed to
763The idea of event-based security is that every event passed to
758−`%clay` has a security `++mask`. The mask tells us:
764+`%clay` has a security `++mask`. The mask tells us:
760- the set of ships that has _contaminated_ this event
766- the set of ships that has _contaminated_ this event
761- the set of ships that has _endorsed_ this event
767- the set of ships that has _endorsed_ this event
@@ -763,20 +769,20 @@ The idea of event-based security is that every event passed to
764To simplify this system a little and maybe explain it, the
770To simplify this system a little and maybe explain it, the
765security question in any event is: (a) who has _caused_ this
771security question in any event is: (a) who has _caused_ this
766−event? (b) who may receive its _effects_? (These sets never
772+event? (b) who may receive its _effects_? (These sets never
767include your urbit itself, of course, only neighbors.)
773include your urbit itself, of course, only neighbors.)
769For example, if we're doing something because ship X sent us a
775For example, if we're doing something because ship X sent us a
770packet, and we're going to leak the results of this something
776packet, and we're going to leak the results of this something
771back in a response to ship X, once `%clay` knows what our ship
777back in a response to ship X, once `%clay` knows what our ship
772thinks of its neighbor X, we should have a good picture of what
778thinks of its neighbor X, we should have a good picture of what
773−effects this event should be allowed to cause. If `%jael` knows
779+effects this event should be allowed to cause. If `%jael` knows
774that X is our friend, `%clay` can know that X can see our cat
780that X is our friend, `%clay` can know that X can see our cat
777Causality is separated out into contamination (data from ship X
783Causality is separated out into contamination (data from ship X
778got into the event) and endorsement (ship X signed the whole
784got into the event) and endorsement (ship X signed the whole
779−event). (This may be overdesign.) All these sets can be defined
785+event). (This may be overdesign.) All these sets can be defined
780as the empty set (eg, releasing to the empty set means no output)
786as the empty set (eg, releasing to the empty set means no output)
781or the universal set (input from the universal set could be from
787or the universal set (input from the universal set could be from
782anyone, ie, is unauthenticated).
788anyone, ie, is unauthenticated).
@@ -785,11 +791,11 @@ anyone, ie, is unauthenticated).
786When you read a leaf from a `%clay` dome, `%clay` is happy to
792When you read a leaf from a `%clay` dome, `%clay` is happy to
787perform any necessary mark conversions (the actual work is done
793perform any necessary mark conversions (the actual work is done
788−by the `%ford` build vane). You can also just ask for the leaf
794+by the `%ford` build vane). You can also just ask for the leaf
789in whatever mark `%clay` has it.
795in whatever mark `%clay` has it.
791It's confusing to think of this mark extension as part of the
797It's confusing to think of this mark extension as part of the
792−path. You shouldn't say `+cat /===/app/talk/hoon`. You should say
798+path. You shouldn't say `+cat /===/app/talk/hoon`. You should say
793`+cat /===/app/talk`, or maybe `+cat %hoon /===/app/talk`.
799`+cat /===/app/talk`, or maybe `+cat %hoon /===/app/talk`.
795A mark-extended namespace may be added to `%ford`, to compensate.
801A mark-extended namespace may be added to `%ford`, to compensate.
@@ -798,7 +804,7 @@ But type-converting reads are the rare case, not the common case.
798### Change #11: real queries and indexes
804### Change #11: real queries and indexes
800A filesystem should postpone being a database until it can no
806A filesystem should postpone being a database until it can no
801−longer do so. So, please don't think about this problem yet.
807+longer do so. So, please don't think about this problem yet.
802I'm joking, of course!
808I'm joking, of course!
804### Change #12: meta-query
810### Change #12: meta-query
@@ -807,24 +813,24 @@ Make it possible to query the list of desks in a ship :-)
808### Change #13: completely redesign Unix mounting
814### Change #13: completely redesign Unix mounting
810−This is actually a series of changes, described below. But
816+This is actually a series of changes, described below. But
811broadly...
817broadly...
813One of Urbit's experiments is live two-way synchronization
819One of Urbit's experiments is live two-way synchronization
814between the Unix filesystem and the revision-control store.
820between the Unix filesystem and the revision-control store.
815−Dropbox style, as it were. I think this experiment is basically
821+Dropbox style, as it were. I think this experiment is basically
816−a failure. It may be right for cat pictures, but I don't think
822+a failure. It may be right for cat pictures, but I don't think
817it's right for Urbit.
823it's right for Urbit.
819The fundamental problem is that there are all sorts of weird,
825The fundamental problem is that there are all sorts of weird,
820−flaky ways applications use a mutable filesystem. It's not at
826+flaky ways applications use a mutable filesystem. It's not at
821all unusual for an editor to change a file by deleting the old
827all unusual for an editor to change a file by deleting the old
822file, then writing the new one in its place.
828file, then writing the new one in its place.
824−You really don't want this to produce a delete change. A simple
830+You really don't want this to produce a delete change. A simple
825cooldown timer for the sync helps a lot with this, but causes
831cooldown timer for the sync helps a lot with this, but causes
826−other weirdnesses. In theory, a sync can even cause another
832+other weirdnesses. In theory, a sync can even cause another
827−change to the same version of the same desk. And so on.
833+change to the same version of the same desk. And so on.
829The thing we fear is just too funky to work (there is some
835The thing we fear is just too funky to work (there is some
830strange "reflux bug" still affecting it) is any sort of _two-way_
836strange "reflux bug" still affecting it) is any sort of _two-way_
@@ -835,8 +841,8 @@ cold storage, is that there's no useful workflow that actually
835uses it -- at least, when editing Urbit code.
841uses it -- at least, when editing Urbit code.
837It's actually really nice to have files on a disk that nothing
843It's actually really nice to have files on a disk that nothing
838−will automatically modify. If you're using `git` and `%clay` in
844+will automatically modify. If you're using `git` and `%clay` in
839−parallel, it's nice to mount your `git` working directory. The
845+parallel, it's nice to mount your `git` working directory. The
840original Urbit workflow was to work out of a mounted directory,
846original Urbit workflow was to work out of a mounted directory,
841but the two-way updates were just too weird and scary, so now we
847but the two-way updates were just too weird and scary, so now we
842tend to edit elsewhere and copy in -- which is lame.
848tend to edit elsewhere and copy in -- which is lame.
@@ -849,25 +855,25 @@ FUSE (in a somewhat experimental state), and filesystem watching
850The FUSE interface is a good one, though it needs more work.
856The FUSE interface is a good one, though it needs more work.
851Moreover, it exports the whole Urbit namespace to Unix, not just
857Moreover, it exports the whole Urbit namespace to Unix, not just
852−`%clay`. For reading Urbit data, FUSE is basically ideal.
858+`%clay`. For reading Urbit data, FUSE is basically ideal.
854−The obvious impulse is to make a read-write FUSE. But I don't
860+The obvious impulse is to make a read-write FUSE. But I don't
855think FUSE should be used for recording changes to an RCS, even
861think FUSE should be used for recording changes to an RCS, even
856though the FUSE API sends you changes as events.
862though the FUSE API sends you changes as events.
858−There are three reasons for this. One, the actual filesystem
864+There are three reasons for this. One, the actual filesystem
859writes that Unix apps make are typically of low quality and
865writes that Unix apps make are typically of low quality and
860−confusing. Some crack-brained editor will find it completely
866+confusing. Some crack-brained editor will find it completely
861normal and unremarkable to modify a byte in a file, by deleting
867normal and unremarkable to modify a byte in a file, by deleting
862−the file and rewriting it from scratch. You can use a cooldown
868+the file and rewriting it from scratch. You can use a cooldown
863timer to wait for this nonsense to converge, but leaving data in
869timer to wait for this nonsense to converge, but leaving data in
864heuristic limbo seems like a very bad idea in FUSE-land.
870heuristic limbo seems like a very bad idea in FUSE-land.
866The second is that people derive a lot of comfort from seeing
872The second is that people derive a lot of comfort from seeing
867−actual files on an actual disk. The idea that your datas are
873+actual files on an actual disk. The idea that your datas are
868stuck in Urbit is slightly terrifying as a user experience.
874stuck in Urbit is slightly terrifying as a user experience.
869−We claim to be reliable. It's actually hard to imagine datas that
875+We claim to be reliable. It's actually hard to imagine datas that
870−couldn't be recovered from either snapshot or log. But how can
876+couldn't be recovered from either snapshot or log. But how can
871you possibly believe us on this?
877you possibly believe us on this?
873So I think it's correct to lean all the weight of reading data
879So I think it's correct to lean all the weight of reading data
@@ -882,35 +888,35 @@ But this is a separate discussion.
883### Change #15: one-way, nonced working desk
889### Change #15: one-way, nonced working desk
885−A one-way mount is a much less crazy thing. It works the same
891+A one-way mount is a much less crazy thing. It works the same
886way our two-way mount works today, but with only one writer: the
892way our two-way mount works today, but with only one writer: the
887user, via Unix.
893user, via Unix.
889Since a mount is one-way, it needs to have an absolute write lock
895Since a mount is one-way, it needs to have an absolute write lock
890−on the desk it's writing to. We cannot have a `%home` desk which
896+on the desk it's writing to. We cannot have a `%home` desk which
891we are also editing with filesystem watching.
897we are also editing with filesystem watching.
893Instead, we have a temporary and anonymous state that's connected
899Instead, we have a temporary and anonymous state that's connected
894−itself with your `%home` desk. Desk names have to be valid
900+itself with your `%home` desk. Desk names have to be valid
895−symbols, but we can use `hep`, `-`. A reasonable choice is to
901+symbols, but we can use `hep`, `-`. A reasonable choice is to
896pick a random 32-bit name for each mount instance -- so the desk
902pick a random 32-bit name for each mount instance -- so the desk
897−name is not `%home`, but `%home-bosdut-sigder`. (Randomizing
903+name is not `%home`, but `%home-bosdut-sigder`. (Randomizing
898instead of always using a fixed suffix, say, `%home-work`,
904instead of always using a fixed suffix, say, `%home-work`,
899just makes all kinds of screwups less likely.)
905just makes all kinds of screwups less likely.)
901You create a mount simply by mkdiring the desk name in your pier.
907You create a mount simply by mkdiring the desk name in your pier.
902Just the base name, Urbit will come up with the `%bosdut-sigder`
908Just the base name, Urbit will come up with the `%bosdut-sigder`
903−part and put it in a `.desk` file at the desk root. To forget
909+part and put it in a `.desk` file at the desk root. To forget
904−the mount, just delete this file. This avoids the ugly confusion
910+the mount, just delete this file. This avoids the ugly confusion
905between unmounting a mount, and deleting all the data in it.
911between unmounting a mount, and deleting all the data in it.
907(Urbit will keep a set of the mount desks it's used and
913(Urbit will keep a set of the mount desks it's used and
908−forgotten, not reuse them, and block all requests into them. By
914+forgotten, not reuse them, and block all requests into them. By
909blocking, we can delete without losing referential transparency.
915blocking, we can delete without losing referential transparency.
910Yes, Virginia, Urbit can forget!)
916Yes, Virginia, Urbit can forget!)
912Note that the auto-reload behavior of applications is extremely
918Note that the auto-reload behavior of applications is extremely
913−useful when testing. When testing, you can either switch the
919+useful when testing. When testing, you can either switch the
914active application desk from `%home` to `%home-bosdut-sigder`,
920active application desk from `%home` to `%home-bosdut-sigder`,
915or activate autosync from the latter to the former.
921or activate autosync from the latter to the former.
@@ -933,7 +939,7 @@ Otherwise, you're basically, like, reinventing Perforce.
934More concretely, the Urbit experience today requires `:hood`
940More concretely, the Urbit experience today requires `:hood`
935pokes (Urbit shell commands, basically) for all nontrivial RCS
941pokes (Urbit shell commands, basically) for all nontrivial RCS
936−operations, from mounting to merging. Again, there are lots of
942+operations, from mounting to merging. Again, there are lots of
937situations in which you can write into a mounted filesystem (the
943situations in which you can write into a mounted filesystem (the
938protocol for Unix communication could be easily adapted to
944protocol for Unix communication could be easily adapted to
939communicate with a browser-based editor, for instance), but you
945communicate with a browser-based editor, for instance), but you
@@ -943,20 +949,20 @@ Even commands like mounting and merging are easily communicated,
943through filesystem writes that any Unix tool (including many
949through filesystem writes that any Unix tool (including many
944macro editors).
950macro editors).
946−This is very easy for a filesystem watcher. One such protocol:
952+This is very easy for a filesystem watcher. One such protocol:
947if you want to send the daemon a command relative to some
953if you want to send the daemon a command relative to some
948−directory, write a `.ur-command` metafile in that directory. The
954+directory, write a `.ur-command` metafile in that directory. The
949daemon accepts the command by deleting it.
955daemon accepts the command by deleting it.
951The user experience is that you configure your mounts by simply
957The user experience is that you configure your mounts by simply
952−making the directories and configuring dotfiles in them. Unlike
958+making the directories and configuring dotfiles in them. Unlike
953in the present system, Urbit will never modify your datas
959in the present system, Urbit will never modify your datas
954silently in the background.
960silently in the background.
956Here's what this looks like:
962Here's what this looks like:
958−Suppose you want to edit some data in the `%home` desk. Your
964+Suppose you want to edit some data in the `%home` desk. Your
959−data is in `/=home=/mar/talk`. You also want a read-only view
965+data is in `/=home=/mar/talk`. You also want a read-only view
960of `/=home=/arvo`.
966of `/=home=/arvo`.
962From Unix, you do something like this:
968From Unix, you do something like this:
@@ -969,18 +975,18 @@ From Unix, you do something like this:
969 $ touch home/arvo/.read
975 $ touch home/arvo/.read
972−Want to commit your changes? That means merging the working
978+Want to commit your changes? That means merging the working
973−desk, `%home-bosdut-sigder`, into the `%home`. Put a commit message
979+desk, `%home-bosdut-sigder`, into the `%home`. Put a commit message
974−into `home/.commit` and it'll happen. If there's an error, it
980+into `home/.commit` and it'll happen. If there's an error, it
975will appear as `.error`.
981will appear as `.error`.
977The mount has a .version file which simply contains the change
983The mount has a .version file which simply contains the change
978−number (of the working desk) which any edits are relative to. If
984+number (of the working desk) which any edits are relative to. If
979the urbit daemon is running and watching, edits will be absorbed
985the urbit daemon is running and watching, edits will be absorbed
980quickly (probably after a heuristic timeout), and the version
986quickly (probably after a heuristic timeout), and the version
981−incremented. But we always are confident that the edits are
987+incremented. But we always are confident that the edits are
982against the latest version, or at least we can use this as a
988against the latest version, or at least we can use this as a
983−sanity check. All kinds of insane things can happen to get a
989+sanity check. All kinds of insane things can happen to get a
984filesystem out of consistency.
990filesystem out of consistency.
986This design obviously needs a little fleshing out, but I'm quite
992This design obviously needs a little fleshing out, but I'm quite
@@ -989,12 +995,12 @@ from Unix.
990## In conclusion
996## In conclusion
992−There's obviously a lot of fun here! There are two things you
998+There's obviously a lot of fun here! There are two things you
993can do right now.
999can do right now.
995One, talk about `%clay` on [this forum
1001One, talk about `%clay` on [this forum
996thread](http://urbit.org/~~/fora/posts/~2016.10.25..17.37.47..6937~/) or
1002thread](http://urbit.org/~~/fora/posts/~2016.10.25..17.37.47..6937~/) or
997−on [:talk](http://urbit.org/stream). Both hosted on Urbit
1003+on [:talk](http://urbit.org/stream). Both hosted on Urbit
1000Two, poke around the
1006Two, poke around the