@@ -1,68 +1,63 @@
2title = "Toward a New %clay"
2title = "Toward a New %clay"
3−date = "2016-07-13"
3+date = 2016-07-14
4description = "Urbit's revision-control system, %clay, is itself due for a (medium-sized) revision!"
4description = "Urbit's revision-control system, %clay, is itself due for a (medium-sized) revision!"
6− "/posts/essays/toward-a-new-clay",
5+aliases = ["/posts/essays/toward-a-new-clay", "/posts/toward-a-new-clay", "/blog/clay"]
7− "/posts/toward-a-new-clay",
12author = "Curtis Yarvin"
7author = "Curtis Yarvin"
13ship = "~sorreg-namtyv"
8ship = "~sorreg-namtyv"
16TLDR: (a) Urbit's revision-control system, `%clay`, is itself due
11TLDR: (a) Urbit's revision-control system, `%clay`, is itself due
17−for a (medium-sized) revision! (b) this time, you yourself can
12+for a (medium-sized) revision! (b) this time, you yourself can
18−really help! (c) please don't give away the buried spoiler!
13+really help! (c) please don't give away the buried spoiler!
20This explanation does not (or at least should not) require you to
15This explanation does not (or at least should not) require you to
21know anything about Urbit.
16know anything about Urbit.
23### But %clay is already pretty cool
18### But %clay is already pretty cool
25−Of course, nothing in Urbit is allowed to be _lame_. `%clay` is
20+Of course, nothing in Urbit is allowed to be _lame_. `%clay` is
26−actually quite cool. It does need some work. But first, let's
21+actually quite cool. It does need some work. But first, let's
27talk about what it is and why it's cool.
22talk about what it is and why it's cool.
29`%clay` is the Arvo vane (Urbit kernel module) for revision
24`%clay` is the Arvo vane (Urbit kernel module) for revision
30−control. It's an authenticated, human-memorable global immutable
25+control. It's an authenticated, human-memorable global immutable
31−namespace which is also a typed revision-control system. It's
26+namespace which is also a typed revision-control system. It's
32−also very good at subscription and synchronization. Urbit
27+also very good at subscription and synchronization. Urbit
33currently uses `%clay` to distribute live hotpatches; we
28currently uses `%clay` to distribute live hotpatches; we
34transparently upgrade the whole network with one checkin.
29transparently upgrade the whole network with one checkin.
36### Compared to other revision-control systems
31### Compared to other revision-control systems
38−`%clay` is clearly a vanilla DVCS in the modern `git` style. Of
33+`%clay` is clearly a vanilla DVCS in the modern `git` style. Of
39deployed DVCS these days, `%clay` is probably closest in spirit
34deployed DVCS these days, `%clay` is probably closest in spirit
40to `bzr` or maybe `fossil`.
35to `bzr` or maybe `fossil`.
42Obviously, as a part of a larger whole, you would expect any
37Obviously, as a part of a larger whole, you would expect any
43−Urbit RCS to be brutally spartan. `%clay` is. It's even more
38+Urbit RCS to be brutally spartan. `%clay` is. It's even more
44spartan than `bzr`.
39spartan than `bzr`.
46−(In fact, it's really too spartan to fully use at the moment. We
41+(In fact, it's really too spartan to fully use at the moment. We
47ourselves still use `%clay` mainly as a distribution system, and
42ourselves still use `%clay` mainly as a distribution system, and
48−do all internal development on GitHub. This is because the
43+do all internal development on GitHub. This is because the
49present `%clay` lacks a few little conveniences, like commit
44present `%clay` lacks a few little conveniences, like commit
50−messages. See upgrades, below.)
45+messages. See upgrades, below.)
52−There are lots of normal DVCS things in `%clay`. But there are
47+There are lots of normal DVCS things in `%clay`. But there are
53−also cool and unusual ones. Let's take a brief tour of the
48+also cool and unusual ones. Let's take a brief tour of the
54high-level system, then describe what we think should change
49high-level system, then describe what we think should change
57### Anatomy of a DVCS
52### Anatomy of a DVCS
59−A DVCS is a historical state graph. A point in this graph is a
54+A DVCS is a historical state graph. A point in this graph is a
60snapshot of some tree-structured system, like a filesystem.
55snapshot of some tree-structured system, like a filesystem.
62These snapshots, including recursive internal nodes, are stored in
57These snapshots, including recursive internal nodes, are stored in
63−a content-addressable store. Pointers in this store are used to
58+a content-addressable store. Pointers in this store are used to
64construct a revision graph, whose edges are edit and/or merge
59construct a revision graph, whose edges are edit and/or merge
65−actions. Some sort of namespace attaches meaningful names to
60+actions. Some sort of namespace attaches meaningful names to
66current revisions in this graph.
61current revisions in this graph.
68There are six big questions in the design of a DVCS:
63There are six big questions in the design of a DVCS:
@@ -78,16 +73,16 @@ There are six big questions in the design of a DVCS:
79A good way to summarize `%clay` (at present) is that its answers
74A good way to summarize `%clay` (at present) is that its answers
80to the first four questions are interesting, and its answers to
75to the first four questions are interesting, and its answers to
81−the last two are boring. We'll talk about all six, though.
76+the last two are boring. We'll talk about all six, though.
83### Global namespace: [ship desk case]
78### Global namespace: [ship desk case]
85−The permanent global name of a `%clay` snapshot is a `++beak`. A beak
80+The permanent global name of a `%clay` snapshot is a `++beak`. A beak
86is a triple: `[ship desk case]`.
81is a triple: `[ship desk case]`.
88−The `++ship` is an Urbit ship. A ship is a network address, a
83+The `++ship` is an Urbit ship. A ship is a network address, a
89−human-memorable name and a cryptographic identity. The address
84+human-memorable name and a cryptographic identity. The address
90−width is variable; shorter addresses are more desirable. Most
85+width is variable; shorter addresses are more desirable. Most
91users have 32-bit addresses, which we map to four-syllable
86users have 32-bit addresses, which we map to four-syllable
92synthetic names like `~sorreg-namtyv`.
87synthetic names like `~sorreg-namtyv`.
@@ -100,44 +95,44 @@ In the Urbit path syntax, the beak is the first three segments of
100the path: `/ship/desk/case/my/cat/picture`.
95the path: `/ship/desk/case/my/cat/picture`.
102In most cases, a path parser has a default ship, desk and case,
97In most cases, a path parser has a default ship, desk and case,
103−each of which can be replaced by `=`. So `/=home=/my/cat/picture`
98+each of which can be replaced by `=`. So `/=home=/my/cat/picture`
104is the data at `/my/cat/picture` on your own ship, in the `%home`
99is the data at `/my/cat/picture` on your own ship, in the `%home`
105−desk, right now. Immutability doesn't have to be inconvenient.
100+desk, right now. Immutability doesn't have to be inconvenient.
107−`%clay` has no concept of a mutable name or a magic `HEAD`. Use
102+`%clay` has no concept of a mutable name or a magic `HEAD`. Use
108−the current date. If you're referring to remote data and you're
103+the current date. If you're referring to remote data and you're
109worried about clock skew, subtract a second or three, or be
104worried about clock skew, subtract a second or three, or be
110−prepared to block. (Or keep your clock unskewed, which shouldn't
105+prepared to block. (Or keep your clock unskewed, which shouldn't
111−be _that_ hard.) It's not like mutable-name schemes work any
106+be _that_ hard.) It's not like mutable-name schemes work any
112better than this anyway.
107better than this anyway.
114−(Immutability is central to `%clay`. `git` merely adopted the
109+(Immutability is central to `%clay`. `git` merely adopted the
115−reflog. `%clay` was born in it, molded by it...)
110+reflog. `%clay` was born in it, molded by it...)
117`%clay` is objective; each Urbit node hosts only one ship, but
112`%clay` is objective; each Urbit node hosts only one ship, but
118−treats domestic and foreign data (almost) the same way. If you
113+treats domestic and foreign data (almost) the same way. If you
119ask your `%clay` repository for someone else's data, it will
114ask your `%clay` repository for someone else's data, it will
120block the request until it gets the data, which it then caches.
115block the request until it gets the data, which it then caches.
121At present, we always fulfill requests from the origin ship,
116At present, we always fulfill requests from the origin ship,
122though in future we could certainly use it as a tracker.
117though in future we could certainly use it as a tracker.
124If `%clay` discards cached data, it keeps a hash, so that your
119If `%clay` discards cached data, it keeps a hash, so that your
125−urbit can't violate the local illusion of immutability. We
120+urbit can't violate the local illusion of immutability. We
126can't, of course, prevent anyone else's ship from sending
121can't, of course, prevent anyone else's ship from sending
127−conflicting claims in different directions. But we can prevent
122+conflicting claims in different directions. But we can prevent
128programs on our own ship from hearing about them, meaning that we
123programs on our own ship from hearing about them, meaning that we
129retain local referential transparency.
124retain local referential transparency.
131### Local structure: dome, arch, leaf and mark
126### Local structure: dome, arch, leaf and mark
133−A `%clay` snapshot is called a `++dome`. A dome is the `%clay`
128+A `%clay` snapshot is called a `++dome`. A dome is the `%clay`
134equivalent of a tree-structured filesystem.
129equivalent of a tree-structured filesystem.
136Each node in this tree is one uniform structure, an `++arch`,
131Each node in this tree is one uniform structure, an `++arch`,
137−which is both file and directory. A `++leaf` or data value in
132+which is both file and directory. A `++leaf` or data value in
138this arch is a typed document.
133this arch is a typed document.
140−A logical document type is called a `++mark`. A mark is a
135+A logical document type is called a `++mark`. A mark is a
141symbolic pointer to a Hoon source file, in the same dome, for an
136symbolic pointer to a Hoon source file, in the same dome, for an
142engine that can validate, compare, and/or convert
137engine that can validate, compare, and/or convert
143documents of that mark.
138documents of that mark.
@@ -146,8 +141,8 @@ Let's go over arches and marks a little more closely...
147#### An arch is a uniform filesystem node
142#### An arch is a uniform filesystem node
149−Most filesystems and RCS have modal nodes. For instance, a git
144+Most filesystems and RCS have modal nodes. For instance, a git
150−data node can be a blob or a tree. `%clay` (and the Urbit
145+data node can be a blob or a tree. `%clay` (and the Urbit
151namespace in general) has one kind of node, an `arch`:
146namespace in general) has one kind of node, an `arch`:
@@ -156,24 +151,24 @@ namespace in general) has one kind of node, an `arch`:
158An arch has an optional typed-data leaf, and optional named
153An arch has an optional typed-data leaf, and optional named
159−children. In Unix terms, it's a file, a directory, or both. It
154+children. In Unix terms, it's a file, a directory, or both. It
160can't be neither: an empty arch, with no data and no children, is
155can't be neither: an empty arch, with no data and no children, is
161−auto-pruned. As in `git` (but not in Unix), the hierarchy is a
156+auto-pruned. As in `git` (but not in Unix), the hierarchy is a
162−pure function of its leaves. Paths exist only because of the
157+pure function of its leaves. Paths exist only because of the
163data they point to.
158data they point to.
165The `++knot` names in an arch path are also a little special.
160The `++knot` names in an arch path are also a little special.
166They're typed, in a sense; any knot can be parsed invertibly into
161They're typed, in a sense; any knot can be parsed invertibly into
167−a typed atom. The atom is an arbitrary unsigned integer. Its
162+a typed atom. The atom is an arbitrary unsigned integer. Its
168type (an `++aura` if you know Hoon) defines the semantics and
163type (an `++aura` if you know Hoon) defines the semantics and
169presentation of the atom: for instance, a symbol, a decimal,
164presentation of the atom: for instance, a symbol, a decimal,
170a date, etc.
165a date, etc.
172Ultimately a filesystem is a database, and a path is a primary
167Ultimately a filesystem is a database, and a path is a primary
173−key. It's very useful to be able to store simple data in this
168+key. It's very useful to be able to store simple data in this
174key, in a standard and unambiguous way.
169key, in a standard and unambiguous way.
176−Why uniform nodes? One, they map nicely to information trees
171+Why uniform nodes? One, they map nicely to information trees
177where a subtree may need some custom collective presentation,
172where a subtree may need some custom collective presentation,
178not an automatically generated Apache style directory listing.
173not an automatically generated Apache style directory listing.
179Two, non-uniform nodes introduce a whole pointless category of
174Two, non-uniform nodes introduce a whole pointless category of
@@ -181,34 +176,34 @@ node type conflicts -- for which life is far too short.
182#### A mark is an executable document type
177#### A mark is an executable document type
184−A mark is "like an executable MIME type." It's best defined as a
179+A mark is "like an executable MIME type." It's best defined as a
185symbolic mapping from a filesystem to a schema engine.
180symbolic mapping from a filesystem to a schema engine.
187If you have a MIME type `text/html`, you need to watch the
182If you have a MIME type `text/html`, you need to watch the
188−Internets to track what the W3C thinks HTML is. If you have an
183+Internets to track what the W3C thinks HTML is. If you have an
189Urbit mark `%html`, you can functionally construct (using our
184Urbit mark `%html`, you can functionally construct (using our
190insanely powerful functional build vane `%ford`) a core which
185insanely powerful functional build vane `%ford`) a core which
191can validate or normalize HTML, apply patch and diff logic, and
186can validate or normalize HTML, apply patch and diff logic, and
192−translate to and from other marks. `%ford` can even reason about
187+translate to and from other marks. `%ford` can even reason about
193translation paths and construct multistep conversions.
188translation paths and construct multistep conversions.
195(Urbit does not actually have this super-smart `%html` mark right
190(Urbit does not actually have this super-smart `%html` mark right
196−now. But it probably should, huh. And this page was composed in
191+now. But it probably should, huh. And this page was composed in
197Markdown and translated to HTML by `%ford`.)
192Markdown and translated to HTML by `%ford`.)
199−Where does the source code of a mark live? Within the same dome
194+Where does the source code of a mark live? Within the same dome
200−as the data it describes. `%html` files are described by the
195+as the data it describes. `%html` files are described by the
201Hoon source at `/===/mar/html`, which perhaps gets mapped into
196Hoon source at `/===/mar/html`, which perhaps gets mapped into
202your Unix filesystem at `$ship/home/mar/html.hoon`.
197your Unix filesystem at `$ship/home/mar/html.hoon`.
204−This Hoon file is not self-contained. It has dependencies. The
199+This Hoon file is not self-contained. It has dependencies. The
205`%html` mark may depend on various structures and libraries
200`%html` mark may depend on various structures and libraries
206−elsewhere in Urbit. But the source code for all them is within
201+elsewhere in Urbit. But the source code for all them is within
207this dome.
202this dome.
209One constraint this puts on domes is that _all domes are
204One constraint this puts on domes is that _all domes are
210−descendants of the base system dome_. Domes are versions of one
205+descendants of the base system dome_. Domes are versions of one
211−reality, not fragments of that reality. You can't create a new
206+reality, not fragments of that reality. You can't create a new
212empty dome -- you'd have no mark source, and be unable to install
207empty dome -- you'd have no mark source, and be unable to install
213any validated files.
208any validated files.
@@ -219,13 +214,13 @@ weird collisions between non-parallel things don't happen.
220### Unix mapping and mounting
215### Unix mapping and mounting
222−Every DVCS is its own filesystem in a sense. But every DVCS
217+Every DVCS is its own filesystem in a sense. But every DVCS
223works by mapping and mounting itself, in some sense, to Unix.
218works by mapping and mounting itself, in some sense, to Unix.
225#### Mapping arches to Unix
220#### Mapping arches to Unix
227−How do we map arches into Unix? The arch structure maps 1:1 to
222+How do we map arches into Unix? The arch structure maps 1:1 to
228−the Unix directory tree. When the arch has a leaf, we derive a
223+the Unix directory tree. When the arch has a leaf, we derive a
229dot extension from the file's mark.
224dot extension from the file's mark.
231So, supposing we have a leaf `foo` which is an HTML document
226So, supposing we have a leaf `foo` which is an HTML document
@@ -239,19 +234,19 @@ foo/moo.html
241This maps invertibly back into `%clay`, and it looks perfectly
236This maps invertibly back into `%clay`, and it looks perfectly
242−unremarkable from the Unix side. File completion sometimes gets
237+unremarkable from the Unix side. File completion sometimes gets
243a little annoying.
238a little annoying.
245It would be if nice every possible Unix tree, even every
240It would be if nice every possible Unix tree, even every
246plausible Unix tree, mapped cleanly and completely to a `%clay`
241plausible Unix tree, mapped cleanly and completely to a `%clay`
247−dome. There are always horrid exceptions, even in common use --
242+dome. There are always horrid exceptions, even in common use --
248−like extensionless Makefiles. The trivial solution is that if
243+like extensionless Makefiles. The trivial solution is that if
249`%clay` finds a file mysterious, it won't track it.
244`%clay` finds a file mysterious, it won't track it.
251One often unquestioned assumption in new system software is that
246One often unquestioned assumption in new system software is that
252−compatibility with old content is important. It's nice, but not
247+compatibility with old content is important. It's nice, but not
253−that important. Most content in a new system is new content.
248+that important. Most content in a new system is new content.
254−The important compatibility direction is export, not import. So
249+The important compatibility direction is export, not import. So
255the new system should subset, not superset, the ancient formats.
250the new system should subset, not superset, the ancient formats.
257### Mounting desks to Unix
252### Mounting desks to Unix
@@ -262,36 +257,36 @@ the Unix directory of the same name within the pier.
262When files in a mount directory change, `%clay` notices it
257When files in a mount directory change, `%clay` notices it
263through `libuv` filesystem watching (ultimately `inotify()` or
258through `libuv` filesystem watching (ultimately `inotify()` or
264`FSEvents`), computes diffs, and automatically commits them to
259`FSEvents`), computes diffs, and automatically commits them to
265−the desk. And if `%clay` creates a new version for its own
260+the desk. And if `%clay` creates a new version for its own
266urbity reasons, it will revise the Unix files.
261urbity reasons, it will revise the Unix files.
268Keeping all mounted files in the same Unix subdirectory makes it
263Keeping all mounted files in the same Unix subdirectory makes it
269−easy to move the whole pier in a tarfile. Edits applied while
264+easy to move the whole pier in a tarfile. Edits applied while
270the `urbit` process isn't running will be detected and applied
265the `urbit` process isn't running will be detected and applied
271the next time it starts.
266the next time it starts.
273In conventional DVCS terms, `%clay` has a 1:1 static mapping from
268In conventional DVCS terms, `%clay` has a 1:1 static mapping from
274−branches to working directories. Why?
269+branches to working directories. Why?
276−Why not? Do we need that "what branches are checked out in what
271+Why not? Do we need that "what branches are checked out in what
277−directories" piece of state? Why force the user to worry about
272+directories" piece of state? Why force the user to worry about
278−which branches are mounted to which directories? Why not just
273+which branches are mounted to which directories? Why not just
279−mount them under their own names? Is there some crippling global
274+mount them under their own names? Is there some crippling global
280shortage of disk space for source files? Our planet is running
275shortage of disk space for source files? Our planet is running
281−out of inodes? Wut?
276+out of inodes? Wut?
283### Queries, subscriptions and synchronization
278### Queries, subscriptions and synchronization
285There are two ways to get data out of `%clay`: dereference the Urbit
280There are two ways to get data out of `%clay`: dereference the Urbit
286−namespace, or send a request/subscription. Sync is in userspace.
281+namespace, or send a request/subscription. Sync is in userspace.
288#### Dereferencing the namespace
283#### Dereferencing the namespace
290−`%clay` is just part of the full Urbit namespace. The full namespace
285+`%clay` is just part of the full Urbit namespace. The full namespace
291has a two-character prefix which expresses what vane to query
286has a two-character prefix which expresses what vane to query
292−(always `%c` for `%clay`). The suffix is a mode flag; `%x` for
287+(always `%c` for `%clay`). The suffix is a mode flag; `%x` for
293data, `%y` for a list of children, `%z` for the whole arch
288data, `%y` for a list of children, `%z` for the whole arch
294−subtree, `%w` for the version number. (Each vane has its own
289+subtree, `%w` for the version number. (Each vane has its own
297For example, the full path to my cat picture is
292For example, the full path to my cat picture is
@@ -305,7 +300,6 @@ For example, the full path to my cat picture is
305If the data model is `++image` (hypothetical -- we don't really
300If the data model is `++image` (hypothetical -- we don't really
306do image processing in Urbit right now), the normal way to load
301do image processing in Urbit right now), the normal way to load
307this data in a Hoon program is
302this data in a Hoon program is
310.^(image /cx/~sorreg-namtyv/home/[now]/my/cat/picture)
304.^(image /cx/~sorreg-namtyv/home/[now]/my/cat/picture)
@@ -322,15 +316,15 @@ learn it in future, `[~ ~]` if we know we'll never know it, or
322`[~ ~ value]` if we know it now.
316`[~ ~ value]` if we know it now.
324In high-level shell programming (`:dojo`), we can make apparently
318In high-level shell programming (`:dojo`), we can make apparently
325−"blocking" requests to `%clay`. What actually happens: when `%clay`
319+"blocking" requests to `%clay`. What actually happens: when `%clay`
326produces `~`, we stop trying to compute this expression and wait
320produces `~`, we stop trying to compute this expression and wait
327−on one or more blocking requests. When we are unblocked, we
321+on one or more blocking requests. When we are unblocked, we
328recompute again from scratch; the intermediate computation is not
322recompute again from scratch; the intermediate computation is not
329saved at all.
323saved at all.
331−Urbit doesn't like to block. It likes to save coherent data
325+Urbit doesn't like to block. It likes to save coherent data
332−structures, not incomplete computation contexts. When Urbit
326+structures, not incomplete computation contexts. When Urbit
333−pretends to block for you, this is a courtesy service. You
327+pretends to block for you, this is a courtesy service. You
334can return the courtesy of not expecting miracles from it.
328can return the courtesy of not expecting miracles from it.
335When in doubt, send an actual request.
329When in doubt, send an actual request.
@@ -341,7 +335,7 @@ namespace, but can (a) read across case (revision) ranges,
341and (b) filter by subtree.
335and (b) filter by subtree.
343If the revision range in a request stretches into the future, the
337If the revision range in a request stretches into the future, the
344−request becomes a subscription. Urbit messaging always regards a
338+request becomes a subscription. Urbit messaging always regards a
345single response as a special case of a subscription.
339single response as a special case of a subscription.
347Generally, the response stream created by a request starts with
341Generally, the response stream created by a request starts with
@@ -351,10 +345,10 @@ closed when the request range is complete.
352#### Userspace synchronization
346#### Userspace synchronization
354−Synchronization per se is done in userspace. In `%clay`, sync just
348+Synchronization per se is done in userspace. In `%clay`, sync just
355means an application (typically the `%kiln` module in the system
349means an application (typically the `%kiln` module in the system
356app `:hood`) subscribes to a remote desk, and saves its change
350app `:hood`) subscribes to a remote desk, and saves its change
357−stream locally. So sync is really an emergent feature of `%clay`.
351+stream locally. So sync is really an emergent feature of `%clay`.
359### Revision graph
353### Revision graph
@@ -362,16 +356,16 @@ stream locally. So sync is really an emergent feature of `%clay`.
362Some changes are proposed below, however.
356Some changes are proposed below, however.
364As in any DVCS, a dome is the child of one parent dome (an edit)
358As in any DVCS, a dome is the child of one parent dome (an edit)
365−or more parents (a merge). Often the parent of any revision is
359+or more parents (a merge). Often the parent of any revision is
366the previous version in the sequence, but it can be anything.
360the previous version in the sequence, but it can be anything.
368### Storage model
362### Storage model
370−`%clay`'s storage model is completely uninteresting. It's also a
364+`%clay`'s storage model is completely uninteresting. It's also a
371little funky and I want to change it, so I'll be super vague.
365little funky and I want to change it, so I'll be super vague.
373A `++lobe`, a 128-bit atom, is the Merkle hash of a `blob`, which
367A `++lobe`, a 128-bit atom, is the Merkle hash of a `blob`, which
374−is essentially a leaf. All blobs are stored in a hash table
368+is essentially a leaf. All blobs are stored in a hash table
375(`(map lobe blob)`), and reference each other through lobes.
369(`(map lobe blob)`), and reference each other through lobes.
377At present we don't GC this space at all, because we don't
371At present we don't GC this space at all, because we don't
@@ -380,12 +374,12 @@ actually delete any roots.
380A `blob` may be `%direct`, meaning it's defined as the data
374A `blob` may be `%direct`, meaning it's defined as the data
381itself, or `%delta`, defined as a change against another blob.
375itself, or `%delta`, defined as a change against another blob.
382This is motivated by the desire not to store a full snapshot
376This is motivated by the desire not to store a full snapshot
383−on every file change. But it may be misguided (see changes #0
377+on every file change. But it may be misguided (see changes #0
384and #1 below).
378and #1 below).
386We cache the full arch tree (`++ankh`) of the latest state in
380We cache the full arch tree (`++ankh`) of the latest state in
387−each desk. For current queries, this mitigates a lot of the
381+each desk. For current queries, this mitigates a lot of the
388−opportunities for slowness that this data model creates. But the
382+opportunities for slowness that this data model creates. But the
389disparity between the speed of the current version and the
383disparity between the speed of the current version and the
390speed previous version is an undesirable glitch in the API.
384speed previous version is an undesirable glitch in the API.
@@ -397,38 +391,38 @@ We actually have _three_ motivations for change...
398At present, we're still using `git` as a development platform.
392At present, we're still using `git` as a development platform.
399`%clay` is certainly used, but just for its strength --
393`%clay` is certainly used, but just for its strength --
400−deployment. This is starving Urbit of essential experience and
394+deployment. This is starving Urbit of essential experience and
401−basically keeping it lame. So, change is necessary.
395+basically keeping it lame. So, change is necessary.
403### Change must come from the community
397### Change must come from the community
405−"Why?" you ask? "Why must change come from the community?" Well,
399+"Why?" you ask? "Why must change come from the community?" Well,
406−one, duh. Change always comes from the community.
400+one, duh. Change always comes from the community.
408−But two, a great force for change has departed. Philip Monk,
402+But two, a great force for change has departed. Philip Monk,
409late maintainer of `%clay` (`~wictuc-folrex` in the old era,
403late maintainer of `%clay` (`~wictuc-folrex` in the old era,
410−`~wicdev-wisryt` in the new), is tired of types and typing. He
404+`~wicdev-wisryt` in the new), is tired of types and typing. He
411spent September roaming with the bison in Montana, and is now
405spent September roaming with the bison in Montana, and is now
412−literally on his way to Patagonia. On the one hand, Phil still
406+literally on his way to Patagonia. On the one hand, Phil still
413exists and is not at all unwilling to answer questions, even dumb
407exists and is not at all unwilling to answer questions, even dumb
414−questions. On the other hand, _Patagonia_.
408+questions. On the other hand, _Patagonia_.
416−So %clay needs a new owner, or owners. At this point, we think
410+So %clay needs a new owner, or owners. At this point, we think
417the community might be indoctrinated enough to take a stab.
411the community might be indoctrinated enough to take a stab.
419Alas, `%clay` already has a pretty checkered ownership history.
413Alas, `%clay` already has a pretty checkered ownership history.
420Long ago I built a `%clay` that was really the only serious
414Long ago I built a `%clay` that was really the only serious
421−design failure in Arvo. Then Jared Hance, a brilliant summer
415+design failure in Arvo. Then Jared Hance, a brilliant summer
422intern, took my broken `%clay` and reformed it into a minimal `git`
416intern, took my broken `%clay` and reformed it into a minimal `git`
423−in about a week. Then Philip modified Jared's `%clay` to be a
417+in about a week. Then Philip modified Jared's `%clay` to be a
424_typed_ revision-control system, which is a pretty cool thing.
418_typed_ revision-control system, which is a pretty cool thing.
426But... in none of these transitions did `%clay` actually get
420But... in none of these transitions did `%clay` actually get
427−_rewritten_ per se. While it's a reasonably good vane, it
421+_rewritten_ per se. While it's a reasonably good vane, it
428does contain one or two historical decisions.
422does contain one or two historical decisions.
430We know a lot more about Arvo than we used to, and I think we can
424We know a lot more about Arvo than we used to, and I think we can
431−revisit many of these decisions. The point of this post is to
425+revisit many of these decisions. The point of this post is to
432start a `%clay` design discussion that will evolve into actual
426start a `%clay` design discussion that will evolve into actual
433collective action.
427collective action.
@@ -438,41 +432,41 @@ We're planning a rewards program for contributions to `%clay`.
438It'll simply involve setting a time period and a pool of real
432It'll simply involve setting a time period and a pool of real
439estate, and allocating the pool by judged contribution.
433estate, and allocating the pool by judged contribution.
441−This is an experimental program. We'll say more about it later.
435+This is an experimental program. We'll say more about it later.
442−It may not even happen. It may happen very differently. When in
436+It may not even happen. It may happen very differently. When in
443doubt, don't talk about Fight Club.
437doubt, don't talk about Fight Club.
445## Some openings for change
439## Some openings for change
447−Change, of course, must come from the community. No one can
441+Change, of course, must come from the community. No one can
448−dictate change from the top down. We _know_ this. But...
442+dictate change from the top down. We _know_ this. But...
450This list of suggested changes for the next `%clay` is not in any
444This list of suggested changes for the next `%clay` is not in any
451−sort of order: priority, structure, difficulty, etc. Let's talk
445+sort of order: priority, structure, difficulty, etc. Let's talk
452−about the changes first; make actual plans second. The numbers
446+about the changes first; make actual plans second. The numbers
453here are just for convenience.
447here are just for convenience.
455−_All changes in this program will happen on the next-generation
449+*All changes in this program will happen on the next-generation
456−(`cc-release`) branch._
450+(`cc-release`) branch.*
458### Change #0: state reform
452### Change #0: state reform
460As mentioned above, the storage model in `%clay`, as the code is
454As mentioned above, the storage model in `%clay`, as the code is
461−at present, feels a little funky to me. There's nothing terribly
455+at present, feels a little funky to me. There's nothing terribly
462awful about it, but it doesn't feel quite right and its scaling
456awful about it, but it doesn't feel quite right and its scaling
463properties are very questionable.
457properties are very questionable.
465Briefly, `%clay` today actually derives the hierarchy from its
459Briefly, `%clay` today actually derives the hierarchy from its
466−leaves -- rather than _logically_ so deriving it. Only leaves
460+leaves -- rather than _logically_ so deriving it. Only leaves
467−are in the object store. This is just weird, and you can feel
461+are in the object store. This is just weird, and you can feel
468−`%clay` trying to be something other than what it is. (I blame
462+`%clay` trying to be something other than what it is. (I blame
469Jared, though it's actually my fault.)
463Jared, though it's actually my fault.)
471To store a dome snapshot (`++yaki`), the present code keeps a
465To store a dome snapshot (`++yaki`), the present code keeps a
472−`(map path blob)` at the root of the dome. This is enough to
466+`(map path blob)` at the root of the dome. This is enough to
473reconstruct the `++arch` structures the namespace exports.
467reconstruct the `++arch` structures the namespace exports.
474However, all kinds of basic arch operations, like basically `ls`,
468However, all kinds of basic arch operations, like basically `ls`,
475−become `O(n)` in the number of leaves in the dome. This is
469+become `O(n)` in the number of leaves in the dome. This is
476obviously not quite right.
470obviously not quite right.
478I believe the storage structure should be (removing everything
472I believe the storage structure should be (removing everything
@@ -493,46 +487,46 @@ Here a `++root` is the universal node, with children addressed by
493hash, but logically representing the abstract `++arch`.
487hash, but logically representing the abstract `++arch`.
495Users of hierarchical filesystems basically expect performance
489Users of hierarchical filesystems basically expect performance
496−qualities that match the filesystem structure. A huge directory
490+qualities that match the filesystem structure. A huge directory
497−in one place shouldn't affect search speeds somewhere else. And
491+in one place shouldn't affect search speeds somewhere else. And
498of course, `ls` can't be O(n) in the size of the filesystem.
492of course, `ls` can't be O(n) in the size of the filesystem.
500### Change #1: store edges, cache snapshots
494### Change #1: store edges, cache snapshots
502−The current `%clay` is snapshot-oriented, like `git`. The actual
496+The current `%clay` is snapshot-oriented, like `git`. The actual
503−changes are discarded. I think this is a bad choice.
497+changes are discarded. I think this is a bad choice.
505`%clay` is not a research project, and has no idea of going as
499`%clay` is not a research project, and has no idea of going as
506−far as `darcs` in the "set-of-patches" direction. However, at
500+far as `darcs` in the "set-of-patches" direction. However, at
507present we actually throw away the patch (`++nori`).
501present we actually throw away the patch (`++nori`).
509I think the right DVCS design for `%clay` is a snapshot-oriented
503I think the right DVCS design for `%clay` is a snapshot-oriented
510−system which also keeps the patches. The revision graph has the
504+system which also keeps the patches. The revision graph has the
511−same general shape as in the old `git` style. But rather than
505+same general shape as in the old `git` style. But rather than
512just deriving the diffs from the snapshots, we can also derive
506just deriving the diffs from the snapshots, we can also derive
513−the snapshots from the diffs. When in doubt, we save space by
507+the snapshots from the diffs. When in doubt, we save space by
514dropping the snapshot, not the diff.
508dropping the snapshot, not the diff.
516For an RCS which is also a general-purpose typed filesystem, it's
510For an RCS which is also a general-purpose typed filesystem, it's
517clear that considering edges (deltas, diffs, changes, etc) as the
511clear that considering edges (deltas, diffs, changes, etc) as the
518−primary state is optimal. You can use diff functions to derive
512+primary state is optimal. You can use diff functions to derive
519deltas from snapshots, but in many data models this will always
513deltas from snapshots, but in many data models this will always
520−risk losing information from the original action. When in
514+risk losing information from the original action. When in
521doubt, save what the user did.
515doubt, save what the user did.
523This is even more true for operations on the tree structure, not
517This is even more true for operations on the tree structure, not
524−on individual leaves. Consider the heuristic energy that `git` has
518+on individual leaves. Consider the heuristic energy that `git` has
525to go through to figure out when `mv` has been used -- even when
519to go through to figure out when `mv` has been used -- even when
526−`git mv` has been used! (According to the `git` faq, this is
520+`git mv` has been used! (According to the `git` faq, this is
527because the lowest common denominator remains Linus' style of
521because the lowest common denominator remains Linus' style of
528working with emailed patches.)
522working with emailed patches.)
530−Edges probably don't even need to be deduplicated. They also can
524+Edges probably don't even need to be deduplicated. They also can
531contain metadata such as commit messages (which `%clay` has no
525contain metadata such as commit messages (which `%clay` has no
532place for at present).
526place for at present).
534Once edges are recognized as unreclaimed primary state, snapshots
528Once edges are recognized as unreclaimed primary state, snapshots
535−can always be derived from the edges. So we can reclaim memory
529+can always be derived from the edges. So we can reclaim memory
536being used for snapshots.
530being used for snapshots.
538It's also straightforward to have a kind of edge which is a list
532It's also straightforward to have a kind of edge which is a list
@@ -540,27 +534,27 @@ of edges, letting us collapse feature branches trivially.
541Finally, I think references in edges should wherever possible
535Finally, I think references in edges should wherever possible
542indirect through the namespace, rather than directly to the
536indirect through the namespace, rather than directly to the
543−content address. The user has strictly more information if you
537+content address. The user has strictly more information if you
544say you merged `/~sorreg-namtyv/base/987/`, than if all you have
538say you merged `/~sorreg-namtyv/base/987/`, than if all you have
545is the hash that this reference refers to.
539is the hash that this reference refers to.
547### Change #2: unconfuse merge and rebase
541### Change #2: unconfuse merge and rebase
549The dispute between merge and rebase has all the bad smells of
543The dispute between merge and rebase has all the bad smells of
550−a historical wart. But here I'm just not sure what's right.
544+a historical wart. But here I'm just not sure what's right.
552The user has one _intent_ whether merging or rebasing, which is
546The user has one _intent_ whether merging or rebasing, which is
553−to combine changes on one desk into another. The _algorithm_
547+to combine changes on one desk into another. The _algorithm_
554used to construct this change, whether operational transformation
548used to construct this change, whether operational transformation
555(rebasing) or a 3-way diff (merging) has one goal: use all
549(rebasing) or a 3-way diff (merging) has one goal: use all
556available information to model the authors' editing intent as
550available information to model the authors' editing intent as
557−realistically as possible. If we save edges, we should use
551+realistically as possible. If we save edges, we should use
560Whether to define the resulting change graph with the typical
554Whether to define the resulting change graph with the typical
561structure of a merge, or the typical structure of a rebase,
555structure of a merge, or the typical structure of a rebase,
562strikes me as quite arbitrary, and well into the land of "pick
556strikes me as quite arbitrary, and well into the land of "pick
563−one and force everyone to stick with it." But this call demands
557+one and force everyone to stick with it." But this call demands
564a real expert on RCS workflow, which I'm not.
558a real expert on RCS workflow, which I'm not.
566Since `%clay` is a humble part of a larger system and needs to
560Since `%clay` is a humble part of a larger system and needs to
@@ -570,23 +564,23 @@ different preferences" is not usually the right answer.
570### Change #3: comment jihad
564### Change #3: comment jihad
572`%clay` needs to be brought up to the documentation standards of
566`%clay` needs to be brought up to the documentation standards of
573−`%jael`. This means (a) the formatting matches; (b) the comments
567+`%jael`. This means (a) the formatting matches; (b) the comments
574are at the same level of quality and ubiquity.
568are at the same level of quality and ubiquity.
576The basic commenting standards are: (a) every core (class) needs
570The basic commenting standards are: (a) every core (class) needs
577a block comment explanation; (b) every gate (function) needs a
571a block comment explanation; (b) every gate (function) needs a
578line comment; (c) every face (variable) needs a line comment.
572line comment; (c) every face (variable) needs a line comment.
580−This is obviously not just a clerical task. To write these
574+This is obviously not just a clerical task. To write these
581−comments, you need to actually understand the code. This is a
575+comments, you need to actually understand the code. This is a
582nontrivial but extremely rewarding task, even if the code you're
576nontrivial but extremely rewarding task, even if the code you're
583commenting will get replaced.
577commenting will get replaced.
585Also, the _symbols_ in `%clay` must be brought up to `%jael`
579Also, the _symbols_ in `%clay` must be brought up to `%jael`
586−quality. There is still far too much four-letterism here.
580+quality. There is still far too much four-letterism here.
588NB: `%clay` is about 3000 lines of pretty clean Hoon, which is
582NB: `%clay` is about 3000 lines of pretty clean Hoon, which is
589−certainly not at all undocumented. We're not exactly looking at
583+certainly not at all undocumented. We're not exactly looking at
590a Matterhorn of reverse engineering here.
584a Matterhorn of reverse engineering here.
592### Change #4: knot to coin internal paths
586### Change #4: knot to coin internal paths
@@ -597,26 +591,26 @@ ASCII symbol as an atom) must be within the range of the
597invertible noun printer, `++co`.
591invertible noun printer, `++co`.
599`++co` maps a described data value, a `++coin`, to an ASCII
593`++co` maps a described data value, a `++coin`, to an ASCII
600−string which is a valid URL segment and Unix filename. Its
594+string which is a valid URL segment and Unix filename. Its
601friend `++so` inverts the function, perfectly.
595friend `++so` inverts the function, perfectly.
603−Almost every knot is the simplest kind of `coin`, a `dime.` A
597+Almost every knot is the simplest kind of `coin`, a `dime.` A
604dime is a single atom with a rendering type (`aura` if you know
598dime is a single atom with a rendering type (`aura` if you know
605−Hoon). Auras include `@tas` for symbolic text, `@ud` for
599+Hoon). Auras include `@tas` for symbolic text, `@ud` for
606−unsigned decimal, `@p` for Urbit ship, etc. `++co` can also
600+unsigned decimal, `@p` for Urbit ship, etc. `++co` can also
607print arbitrary cells, though less prettily.
601print arbitrary cells, though less prettily.
609This feature is cool because it lets us stuff data into names in
603This feature is cool because it lets us stuff data into names in
610−a precise and predictable way. Usually the names in a namespace
604+a precise and predictable way. Usually the names in a namespace
611are human-meaningful and human-originated, and are therefore
605are human-meaningful and human-originated, and are therefore
612−symbolic, but there are exceptions. For example, blogposts may
606+symbolic, but there are exceptions. For example, blogposts may
613want to be named by date.
607want to be named by date.
615However, it is slightly retarded for the fundamental state of our
609However, it is slightly retarded for the fundamental state of our
616arches to store the _rendered_ string (`++knot`), not the source
610arches to store the _rendered_ string (`++knot`), not the source
617−data (`++coin`). It leads to all kinds of inefficiencies where
611+data (`++coin`). It leads to all kinds of inefficiencies where
618−we're doing way more parsing than we want to. (Parsing also
612+we're doing way more parsing than we want to. (Parsing also
619−should be jet-propelled, but is not.) It also makes it way too
613+should be jet-propelled, but is not.) It also makes it way too
620easy to sneak in bad data and make it `%clay`'s problem.
614easy to sneak in bad data and make it `%clay`'s problem.
622So, logically, we'd be shifting from
616So, logically, we'd be shifting from
@@ -631,7 +625,7 @@ So, logically, we'd be shifting from
632Also, a vanilla `++map` might not be quite the right data
626Also, a vanilla `++map` might not be quite the right data
633structure at this point -- we probably want a custom sorted tree,
627structure at this point -- we probably want a custom sorted tree,
634−which separates auras and imposes a custom order on each. This
628+which separates auras and imposes a custom order on each. This
635would let us serve simple range queries quite efficiently, so
629would let us serve simple range queries quite efficiently, so
636long as they match the standard aura order (eg, alphabetical
630long as they match the standard aura order (eg, alphabetical
637order for symbols).
631order for symbols).
@@ -639,23 +633,23 @@ order for symbols).
639### Change #5: dome semantics by reference
633### Change #5: dome semantics by reference
641While the principle that a dome contains its own mark source is
635While the principle that a dome contains its own mark source is
642−elegant, it's a pain in the ass to implement properly. `%clay`
636+elegant, it's a pain in the ass to implement properly. `%clay`
643certainly does not do it justice at present.
637certainly does not do it justice at present.
645The complexity is in checking every change to see whether it
639The complexity is in checking every change to see whether it
646includes a change to the mark source (which is very unlikely),
640includes a change to the mark source (which is very unlikely),
647or a dependency of the mark source (slightly less unlikely).
641or a dependency of the mark source (slightly less unlikely).
648And if so, do we have to update all the leaves in the dome?
642And if so, do we have to update all the leaves in the dome?
649−This is also very unlikely. And yet, it happens.
643+This is also very unlikely. And yet, it happens.
651Instead, we could simply designate, by beak (ship, desk, and
645Instead, we could simply designate, by beak (ship, desk, and
652−case) a dome which contains the canonical mark source. This
646+case) a dome which contains the canonical mark source. This
653would rarely change -- you could upgrade a desk to a new mark
647would rarely change -- you could upgrade a desk to a new mark
654dome, but it would be a manual operation and probably a rare one.
648dome, but it would be a manual operation and probably a rare one.
656The common case would just be that everyone uses the current
650The common case would just be that everyone uses the current
657version of the `%base` desk when a desk is created, and never
651version of the `%base` desk when a desk is created, and never
658−changes it. In practice, this means not every two-byte edit
652+changes it. In practice, this means not every two-byte edit
659needs to get involved in detecting and/or applying schema
653needs to get involved in detecting and/or applying schema
660evolution, a welcome simplification.
654evolution, a welcome simplification.
@@ -667,14 +661,14 @@ design is one mark per message structure, the mark namespace gets
667overloaded fast.
661overloaded fast.
669So we added an app hierarchy within the mark namespace, using hep
663So we added an app hierarchy within the mark namespace, using hep
670−`-` as a divider. The `:talk` app expects messages with the mark
664+`-` as a divider. The `:talk` app expects messages with the mark
671`talk-message`, whose sources is in `/===/mar/talk/message`.
665`talk-message`, whose sources is in `/===/mar/talk/message`.
673This helps, but it doesn't solve another two problems: trivial
667This helps, but it doesn't solve another two problems: trivial
674nouns and containers.
668nouns and containers.
676It's retarded to have to add a source file to send a `@ud` or a
670It's retarded to have to add a source file to send a `@ud` or a
677−`*`. An aura (see above under `coin`) should be a trivial mark.
671+`*`. An aura (see above under `coin`) should be a trivial mark.
679Also, we need to be able to construct standard containers of
673Also, we need to be able to construct standard containers of
680marks: lists, maps, sets, and tuples (or, as a database person
674marks: lists, maps, sets, and tuples (or, as a database person
@@ -686,32 +680,32 @@ Another kind of complex mark that needs to exist: conflict,
686difference, and invalid marks.
680difference, and invalid marks.
688Conflict data and difference data are two types of noun that are
682Conflict data and difference data are two types of noun that are
689−generated in normal RCS operation. If we have the concept of an
683+generated in normal RCS operation. If we have the concept of an
690`%html` noun, we also need the concept of an invertible change
684`%html` noun, we also need the concept of an invertible change
691from one `%html` file to another; a merge conflict between two
685from one `%html` file to another; a merge conflict between two
692`%html` files; and even a conflict between two marks.
686`%html` files; and even a conflict between two marks.
694There are two kinds of changes to a dome: manual and automatic.
688There are two kinds of changes to a dome: manual and automatic.
695Manual changes are initiated by the user and should crash, unless
689Manual changes are initiated by the user and should crash, unless
696−manually overridden. Automatic changes should not crash, ever,
690+manually overridden. Automatic changes should not crash, ever,
697−and should not lose data. If they create problems, which they
691+and should not lose data. If they create problems, which they
698shouldn't, these problems will need to be cleaned up manually.
692shouldn't, these problems will need to be cleaned up manually.
700This is a complex space to navigate, but it requires a conflicted
694This is a complex space to navigate, but it requires a conflicted
701merge to be able to resolve to a valid dome containing conflict
695merge to be able to resolve to a valid dome containing conflict
702−leaves, not crash. The present `%clay` goes halfway there and
696+leaves, not crash. The present `%clay` goes halfway there and
703resolves to a scratch desk, but we can go farther: the whole `git`
697resolves to a scratch desk, but we can go farther: the whole `git`
704−dumpster-fire of "merge state" deserves destruction. One change
698+dumpster-fire of "merge state" deserves destruction. One change
705is a merge that produces conflicts, the next is an edit that
699is a merge that produces conflicts, the next is an edit that
706resolves them.
700resolves them.
708−Similarly, all sorts of errors can produce invalid leaves. One
702+Similarly, all sorts of errors can produce invalid leaves. One
709particularly rare case which is nonetheless important is schema
703particularly rare case which is nonetheless important is schema
710evolution: what happens to files in a mark, when the mark source
704evolution: what happens to files in a mark, when the mark source
711−code changes. They all need to be re-normalized. And this
705+code changes. They all need to be re-normalized. And this
712−operation can fail, producing an invalid value. (We don't do any
706+operation can fail, producing an invalid value. (We don't do any
713of this now, BTW, although `%ford` has the dependency mechanism to
707of this now, BTW, although `%ford` has the dependency mechanism to
714−make it possible.) Again, in some cases this should crash the
708+make it possible.) Again, in some cases this should crash the
715update to the mark source; in other cases, there is no local way
709update to the mark source; in other cases, there is no local way
716to handle the crash.
710to handle the crash.
@@ -722,46 +716,46 @@ into using the bad content.
723The bottom line: a mark needs to be split into two models:
717The bottom line: a mark needs to be split into two models:
724`++mark`, the original symbolic mark with its source engine, and
718`++mark`, the original symbolic mark with its source engine, and
725−something like `++logo`, the meta-mark or mark container. The
719+something like `++logo`, the meta-mark or mark container. The
726mark is the building block; the logo is the building.
720mark is the building block; the logo is the building.
728### Change #8: filesystem hierarchy standard
722### Change #8: filesystem hierarchy standard
730While all filesystems remain forks of one ur-filesystem, the
724While all filesystems remain forks of one ur-filesystem, the
731−standard tends to enforce itself. But not define itself.
725+standard tends to enforce itself. But not define itself.
733−I've always hated the current `/mar` for marks, for instance. I
727+I've always hated the current `/mar` for marks, for instance. I
734−prefer `/mac`. But others hate this. And so on. Also, by
728+prefer `/mac`. But others hate this. And so on. Also, by
735necessity, our nascent "FHS" is heavily biased toward system
729necessity, our nascent "FHS" is heavily biased toward system
736−data, away from user data. This is retarded.
730+data, away from user data. This is retarded.
738−Please focus all bikeshedding energy on this problem! Better to
732+Please focus all bikeshedding energy on this problem! Better to
739bikeshed about what color to paint the bikeshed, then what shape
733bikeshed about what color to paint the bikeshed, then what shape
740the bike gears should be.
734the bike gears should be.
742### Change #9: security and access control
736### Change #9: security and access control
744−Oh, yeah, that! They always say not to design security in at the
738+Oh, yeah, that! They always say not to design security in at the
745−last minute. Unfortunately, this is exactly what we did. There
739+last minute. Unfortunately, this is exactly what we did. There
746−is no security at all in present `%clay`. (There is also no push
740+is no security at all in present `%clay`. (There is also no push
747operation in `%clay`, so it's not as mad as you think -- anyone
741operation in `%clay`, so it's not as mad as you think -- anyone
748can read anything, no one can write anything, which is fine in a
742can read anything, no one can write anything, which is fine in a
749young network where there are no secrets.)
743young network where there are no secrets.)
751Fortunately, we don't have anything super fancy planned in this
745Fortunately, we don't have anything super fancy planned in this
752−area, just basic access-control lists. These obviously need to
746+area, just basic access-control lists. These obviously need to
753be designed along with the filesystem-hierarchy standard (see
747be designed along with the filesystem-hierarchy standard (see
754elsewhere).
748elsewhere).
756−Is it even possible to do ACLs properly in `%clay`? At this
750+Is it even possible to do ACLs properly in `%clay`? At this
757−point, yes. There is an event-oriented security model in Arvo,
751+point, yes. There is an event-oriented security model in Arvo,
758though it isn't really hooked up to anything at the moment.
752though it isn't really hooked up to anything at the moment.
759−There is also a privilege badge system in `%jael`. The
753+There is also a privilege badge system in `%jael`. The
760combination of these two features is quite sufficient for a
754combination of these two features is quite sufficient for a
761decent ACL model.
755decent ACL model.
763The idea of event-based security is that every event passed to
757The idea of event-based security is that every event passed to
764−`%clay` has a security `++mask`. The mask tells us:
758+`%clay` has a security `++mask`. The mask tells us:
766- the set of ships that has _contaminated_ this event
760- the set of ships that has _contaminated_ this event
767- the set of ships that has _endorsed_ this event
761- the set of ships that has _endorsed_ this event
@@ -769,20 +763,20 @@ The idea of event-based security is that every event passed to
770To simplify this system a little and maybe explain it, the
764To simplify this system a little and maybe explain it, the
771security question in any event is: (a) who has _caused_ this
765security question in any event is: (a) who has _caused_ this
772−event? (b) who may receive its _effects_? (These sets never
766+event? (b) who may receive its _effects_? (These sets never
773include your urbit itself, of course, only neighbors.)
767include your urbit itself, of course, only neighbors.)
775For example, if we're doing something because ship X sent us a
769For example, if we're doing something because ship X sent us a
776packet, and we're going to leak the results of this something
770packet, and we're going to leak the results of this something
777back in a response to ship X, once `%clay` knows what our ship
771back in a response to ship X, once `%clay` knows what our ship
778thinks of its neighbor X, we should have a good picture of what
772thinks of its neighbor X, we should have a good picture of what
779−effects this event should be allowed to cause. If `%jael` knows
773+effects this event should be allowed to cause. If `%jael` knows
780that X is our friend, `%clay` can know that X can see our cat
774that X is our friend, `%clay` can know that X can see our cat
783Causality is separated out into contamination (data from ship X
777Causality is separated out into contamination (data from ship X
784got into the event) and endorsement (ship X signed the whole
778got into the event) and endorsement (ship X signed the whole
785−event). (This may be overdesign.) All these sets can be defined
779+event). (This may be overdesign.) All these sets can be defined
786as the empty set (eg, releasing to the empty set means no output)
780as the empty set (eg, releasing to the empty set means no output)
787or the universal set (input from the universal set could be from
781or the universal set (input from the universal set could be from
788anyone, ie, is unauthenticated).
782anyone, ie, is unauthenticated).
@@ -791,11 +785,11 @@ anyone, ie, is unauthenticated).
792When you read a leaf from a `%clay` dome, `%clay` is happy to
786When you read a leaf from a `%clay` dome, `%clay` is happy to
793perform any necessary mark conversions (the actual work is done
787perform any necessary mark conversions (the actual work is done
794−by the `%ford` build vane). You can also just ask for the leaf
788+by the `%ford` build vane). You can also just ask for the leaf
795in whatever mark `%clay` has it.
789in whatever mark `%clay` has it.
797It's confusing to think of this mark extension as part of the
791It's confusing to think of this mark extension as part of the
798−path. You shouldn't say `+cat /===/app/talk/hoon`. You should say
792+path. You shouldn't say `+cat /===/app/talk/hoon`. You should say
799`+cat /===/app/talk`, or maybe `+cat %hoon /===/app/talk`.
793`+cat /===/app/talk`, or maybe `+cat %hoon /===/app/talk`.
801A mark-extended namespace may be added to `%ford`, to compensate.
795A mark-extended namespace may be added to `%ford`, to compensate.
@@ -804,7 +798,7 @@ But type-converting reads are the rare case, not the common case.
804### Change #11: real queries and indexes
798### Change #11: real queries and indexes
806A filesystem should postpone being a database until it can no
800A filesystem should postpone being a database until it can no
807−longer do so. So, please don't think about this problem yet.
801+longer do so. So, please don't think about this problem yet.
808I'm joking, of course!
802I'm joking, of course!
810### Change #12: meta-query
804### Change #12: meta-query
@@ -813,24 +807,24 @@ Make it possible to query the list of desks in a ship :-)
814### Change #13: completely redesign Unix mounting
808### Change #13: completely redesign Unix mounting
816−This is actually a series of changes, described below. But
810+This is actually a series of changes, described below. But
817broadly...
811broadly...
819One of Urbit's experiments is live two-way synchronization
813One of Urbit's experiments is live two-way synchronization
820between the Unix filesystem and the revision-control store.
814between the Unix filesystem and the revision-control store.
821−Dropbox style, as it were. I think this experiment is basically
815+Dropbox style, as it were. I think this experiment is basically
822−a failure. It may be right for cat pictures, but I don't think
816+a failure. It may be right for cat pictures, but I don't think
823it's right for Urbit.
817it's right for Urbit.
825The fundamental problem is that there are all sorts of weird,
819The fundamental problem is that there are all sorts of weird,
826−flaky ways applications use a mutable filesystem. It's not at
820+flaky ways applications use a mutable filesystem. It's not at
827all unusual for an editor to change a file by deleting the old
821all unusual for an editor to change a file by deleting the old
828file, then writing the new one in its place.
822file, then writing the new one in its place.
830−You really don't want this to produce a delete change. A simple
824+You really don't want this to produce a delete change. A simple
831cooldown timer for the sync helps a lot with this, but causes
825cooldown timer for the sync helps a lot with this, but causes
832−other weirdnesses. In theory, a sync can even cause another
826+other weirdnesses. In theory, a sync can even cause another
833−change to the same version of the same desk. And so on.
827+change to the same version of the same desk. And so on.
835The thing we fear is just too funky to work (there is some
829The thing we fear is just too funky to work (there is some
836strange "reflux bug" still affecting it) is any sort of _two-way_
830strange "reflux bug" still affecting it) is any sort of _two-way_
@@ -841,8 +835,8 @@ cold storage, is that there's no useful workflow that actually
841uses it -- at least, when editing Urbit code.
835uses it -- at least, when editing Urbit code.
843It's actually really nice to have files on a disk that nothing
837It's actually really nice to have files on a disk that nothing
844−will automatically modify. If you're using `git` and `%clay` in
838+will automatically modify. If you're using `git` and `%clay` in
845−parallel, it's nice to mount your `git` working directory. The
839+parallel, it's nice to mount your `git` working directory. The
846original Urbit workflow was to work out of a mounted directory,
840original Urbit workflow was to work out of a mounted directory,
847but the two-way updates were just too weird and scary, so now we
841but the two-way updates were just too weird and scary, so now we
848tend to edit elsewhere and copy in -- which is lame.
842tend to edit elsewhere and copy in -- which is lame.
@@ -855,25 +849,25 @@ FUSE (in a somewhat experimental state), and filesystem watching
856The FUSE interface is a good one, though it needs more work.
850The FUSE interface is a good one, though it needs more work.
857Moreover, it exports the whole Urbit namespace to Unix, not just
851Moreover, it exports the whole Urbit namespace to Unix, not just
858−`%clay`. For reading Urbit data, FUSE is basically ideal.
852+`%clay`. For reading Urbit data, FUSE is basically ideal.
860−The obvious impulse is to make a read-write FUSE. But I don't
854+The obvious impulse is to make a read-write FUSE. But I don't
861think FUSE should be used for recording changes to an RCS, even
855think FUSE should be used for recording changes to an RCS, even
862though the FUSE API sends you changes as events.
856though the FUSE API sends you changes as events.
864−There are three reasons for this. One, the actual filesystem
858+There are three reasons for this. One, the actual filesystem
865writes that Unix apps make are typically of low quality and
859writes that Unix apps make are typically of low quality and
866−confusing. Some crack-brained editor will find it completely
860+confusing. Some crack-brained editor will find it completely
867normal and unremarkable to modify a byte in a file, by deleting
861normal and unremarkable to modify a byte in a file, by deleting
868−the file and rewriting it from scratch. You can use a cooldown
862+the file and rewriting it from scratch. You can use a cooldown
869timer to wait for this nonsense to converge, but leaving data in
863timer to wait for this nonsense to converge, but leaving data in
870heuristic limbo seems like a very bad idea in FUSE-land.
864heuristic limbo seems like a very bad idea in FUSE-land.
872The second is that people derive a lot of comfort from seeing
866The second is that people derive a lot of comfort from seeing
873−actual files on an actual disk. The idea that your datas are
867+actual files on an actual disk. The idea that your datas are
874stuck in Urbit is slightly terrifying as a user experience.
868stuck in Urbit is slightly terrifying as a user experience.
875−We claim to be reliable. It's actually hard to imagine datas that
869+We claim to be reliable. It's actually hard to imagine datas that
876−couldn't be recovered from either snapshot or log. But how can
870+couldn't be recovered from either snapshot or log. But how can
877you possibly believe us on this?
871you possibly believe us on this?
879So I think it's correct to lean all the weight of reading data
873So I think it's correct to lean all the weight of reading data
@@ -888,35 +882,35 @@ But this is a separate discussion.
889### Change #15: one-way, nonced working desk
883### Change #15: one-way, nonced working desk
891−A one-way mount is a much less crazy thing. It works the same
885+A one-way mount is a much less crazy thing. It works the same
892way our two-way mount works today, but with only one writer: the
886way our two-way mount works today, but with only one writer: the
893user, via Unix.
887user, via Unix.
895Since a mount is one-way, it needs to have an absolute write lock
889Since a mount is one-way, it needs to have an absolute write lock
896−on the desk it's writing to. We cannot have a `%home` desk which
890+on the desk it's writing to. We cannot have a `%home` desk which
897we are also editing with filesystem watching.
891we are also editing with filesystem watching.
899Instead, we have a temporary and anonymous state that's connected
893Instead, we have a temporary and anonymous state that's connected
900−itself with your `%home` desk. Desk names have to be valid
894+itself with your `%home` desk. Desk names have to be valid
901−symbols, but we can use `hep`, `-`. A reasonable choice is to
895+symbols, but we can use `hep`, `-`. A reasonable choice is to
902pick a random 32-bit name for each mount instance -- so the desk
896pick a random 32-bit name for each mount instance -- so the desk
903−name is not `%home`, but `%home-bosdut-sigder`. (Randomizing
897+name is not `%home`, but `%home-bosdut-sigder`. (Randomizing
904instead of always using a fixed suffix, say, `%home-work`,
898instead of always using a fixed suffix, say, `%home-work`,
905just makes all kinds of screwups less likely.)
899just makes all kinds of screwups less likely.)
907You create a mount simply by mkdiring the desk name in your pier.
901You create a mount simply by mkdiring the desk name in your pier.
908Just the base name, Urbit will come up with the `%bosdut-sigder`
902Just the base name, Urbit will come up with the `%bosdut-sigder`
909−part and put it in a `.desk` file at the desk root. To forget
903+part and put it in a `.desk` file at the desk root. To forget
910−the mount, just delete this file. This avoids the ugly confusion
904+the mount, just delete this file. This avoids the ugly confusion
911between unmounting a mount, and deleting all the data in it.
905between unmounting a mount, and deleting all the data in it.
913(Urbit will keep a set of the mount desks it's used and
907(Urbit will keep a set of the mount desks it's used and
914−forgotten, not reuse them, and block all requests into them. By
908+forgotten, not reuse them, and block all requests into them. By
915blocking, we can delete without losing referential transparency.
909blocking, we can delete without losing referential transparency.
916Yes, Virginia, Urbit can forget!)
910Yes, Virginia, Urbit can forget!)
918Note that the auto-reload behavior of applications is extremely
912Note that the auto-reload behavior of applications is extremely
919−useful when testing. When testing, you can either switch the
913+useful when testing. When testing, you can either switch the
920active application desk from `%home` to `%home-bosdut-sigder`,
914active application desk from `%home` to `%home-bosdut-sigder`,
921or activate autosync from the latter to the former.
915or activate autosync from the latter to the former.
@@ -939,7 +933,7 @@ Otherwise, you're basically, like, reinventing Perforce.
940More concretely, the Urbit experience today requires `:hood`
934More concretely, the Urbit experience today requires `:hood`
941pokes (Urbit shell commands, basically) for all nontrivial RCS
935pokes (Urbit shell commands, basically) for all nontrivial RCS
942−operations, from mounting to merging. Again, there are lots of
936+operations, from mounting to merging. Again, there are lots of
943situations in which you can write into a mounted filesystem (the
937situations in which you can write into a mounted filesystem (the
944protocol for Unix communication could be easily adapted to
938protocol for Unix communication could be easily adapted to
945communicate with a browser-based editor, for instance), but you
939communicate with a browser-based editor, for instance), but you
@@ -949,20 +943,20 @@ Even commands like mounting and merging are easily communicated,
949through filesystem writes that any Unix tool (including many
943through filesystem writes that any Unix tool (including many
950macro editors).
944macro editors).
952−This is very easy for a filesystem watcher. One such protocol:
946+This is very easy for a filesystem watcher. One such protocol:
953if you want to send the daemon a command relative to some
947if you want to send the daemon a command relative to some
954−directory, write a `.ur-command` metafile in that directory. The
948+directory, write a `.ur-command` metafile in that directory. The
955daemon accepts the command by deleting it.
949daemon accepts the command by deleting it.
957The user experience is that you configure your mounts by simply
951The user experience is that you configure your mounts by simply
958−making the directories and configuring dotfiles in them. Unlike
952+making the directories and configuring dotfiles in them. Unlike
959in the present system, Urbit will never modify your datas
953in the present system, Urbit will never modify your datas
960silently in the background.
954silently in the background.
962Here's what this looks like:
956Here's what this looks like:
964−Suppose you want to edit some data in the `%home` desk. Your
958+Suppose you want to edit some data in the `%home` desk. Your
965−data is in `/=home=/mar/talk`. You also want a read-only view
959+data is in `/=home=/mar/talk`. You also want a read-only view
966of `/=home=/arvo`.
960of `/=home=/arvo`.
968From Unix, you do something like this:
962From Unix, you do something like this:
@@ -975,18 +969,18 @@ From Unix, you do something like this:
975 $ touch home/arvo/.read
969 $ touch home/arvo/.read
978−Want to commit your changes? That means merging the working
972+Want to commit your changes? That means merging the working
979−desk, `%home-bosdut-sigder`, into the `%home`. Put a commit message
973+desk, `%home-bosdut-sigder`, into the `%home`. Put a commit message
980−into `home/.commit` and it'll happen. If there's an error, it
974+into `home/.commit` and it'll happen. If there's an error, it
981will appear as `.error`.
975will appear as `.error`.
983The mount has a .version file which simply contains the change
977The mount has a .version file which simply contains the change
984−number (of the working desk) which any edits are relative to. If
978+number (of the working desk) which any edits are relative to. If
985the urbit daemon is running and watching, edits will be absorbed
979the urbit daemon is running and watching, edits will be absorbed
986quickly (probably after a heuristic timeout), and the version
980quickly (probably after a heuristic timeout), and the version
987−incremented. But we always are confident that the edits are
981+incremented. But we always are confident that the edits are
988against the latest version, or at least we can use this as a
982against the latest version, or at least we can use this as a
989−sanity check. All kinds of insane things can happen to get a
983+sanity check. All kinds of insane things can happen to get a
990filesystem out of consistency.
984filesystem out of consistency.
992This design obviously needs a little fleshing out, but I'm quite
986This design obviously needs a little fleshing out, but I'm quite
@@ -995,12 +989,12 @@ from Unix.
996## In conclusion
990## In conclusion
998−There's obviously a lot of fun here! There are two things you
992+There's obviously a lot of fun here! There are two things you
999can do right now.
993can do right now.
1001One, talk about `%clay` on [this forum
995One, talk about `%clay` on [this forum
1002thread](http://urbit.org/~~/fora/posts/~2016.10.25..17.37.47..6937~/) or
996thread](http://urbit.org/~~/fora/posts/~2016.10.25..17.37.47..6937~/) or
1003−on [:talk](http://urbit.org/stream). Both hosted on Urbit
997+on [:talk](http://urbit.org/stream). Both hosted on Urbit
1006Two, poke around the
1000Two, poke around the