@@ -1,61 +1,68 @@
2title = "Toward a New %clay"
2title = "Toward a New %clay"
3−date = 2016-07-14
3+date = "2016-07-13"
4−author = "Curtis Yarvin"
5description = "Urbit's revision-control system, %clay, is itself due for a (medium-sized) revision!"
4description = "Urbit's revision-control system, %clay, is itself due for a (medium-sized) revision!"
6−[taxonomies]
5+aliases = [
7−posts = ["Essays"]
6+ "/posts/essays/toward-a-new-clay",
7+ "/posts/toward-a-new-clay",
12+author = "Curtis Yarvin"
13+ship = "~sorreg-namtyv"
9TLDR: (a) Urbit's revision-control system, `%clay`, is itself due
16TLDR: (a) Urbit's revision-control system, `%clay`, is itself due
10−for a (medium-sized) revision! (b) this time, you yourself can
17+for a (medium-sized) revision! (b) this time, you yourself can
11−really help! (c) please don't give away the buried spoiler!
18+really help! (c) please don't give away the buried spoiler!
13This explanation does not (or at least should not) require you to
20This explanation does not (or at least should not) require you to
14know anything about Urbit.
21know anything about Urbit.
16### But %clay is already pretty cool
23### But %clay is already pretty cool
18−Of course, nothing in Urbit is allowed to be _lame_. `%clay` is
25+Of course, nothing in Urbit is allowed to be _lame_. `%clay` is
19−actually quite cool. It does need some work. But first, let's
26+actually quite cool. It does need some work. But first, let's
20talk about what it is and why it's cool.
27talk about what it is and why it's cool.
22`%clay` is the Arvo vane (Urbit kernel module) for revision
29`%clay` is the Arvo vane (Urbit kernel module) for revision
23−control. It's an authenticated, human-memorable global immutable
30+control. It's an authenticated, human-memorable global immutable
24−namespace which is also a typed revision-control system. It's
31+namespace which is also a typed revision-control system. It's
25−also very good at subscription and synchronization. Urbit
32+also very good at subscription and synchronization. Urbit
26currently uses `%clay` to distribute live hotpatches; we
33currently uses `%clay` to distribute live hotpatches; we
27transparently upgrade the whole network with one checkin.
34transparently upgrade the whole network with one checkin.
29### Compared to other revision-control systems
36### Compared to other revision-control systems
31−`%clay` is clearly a vanilla DVCS in the modern `git` style. Of
38+`%clay` is clearly a vanilla DVCS in the modern `git` style. Of
32deployed DVCS these days, `%clay` is probably closest in spirit
39deployed DVCS these days, `%clay` is probably closest in spirit
33to `bzr` or maybe `fossil`.
40to `bzr` or maybe `fossil`.
35Obviously, as a part of a larger whole, you would expect any
42Obviously, as a part of a larger whole, you would expect any
36−Urbit RCS to be brutally spartan. `%clay` is. It's even more
43+Urbit RCS to be brutally spartan. `%clay` is. It's even more
37spartan than `bzr`.
44spartan than `bzr`.
39−(In fact, it's really too spartan to fully use at the moment. We
46+(In fact, it's really too spartan to fully use at the moment. We
40ourselves still use `%clay` mainly as a distribution system, and
47ourselves still use `%clay` mainly as a distribution system, and
41−do all internal development on Github. This is because the
48+do all internal development on GitHub. This is because the
42present `%clay` lacks a few little conveniences, like commit
49present `%clay` lacks a few little conveniences, like commit
43−messages. See upgrades, below.)
50+messages. See upgrades, below.)
45−There are lots of normal DVCS things in `%clay`. But there are
52+There are lots of normal DVCS things in `%clay`. But there are
46−also cool and unusual ones. Let's take a brief tour of the
53+also cool and unusual ones. Let's take a brief tour of the
47high-level system, then describe what we think should change
54high-level system, then describe what we think should change
50### Anatomy of a DVCS
57### Anatomy of a DVCS
52−A DVCS is a historical state graph. A point in this graph is a
59+A DVCS is a historical state graph. A point in this graph is a
53snapshot of some tree-structured system, like a filesystem.
60snapshot of some tree-structured system, like a filesystem.
55These snapshots, including recursive internal nodes, are stored in
62These snapshots, including recursive internal nodes, are stored in
56−a content-addressable store. Pointers in this store are used to
63+a content-addressable store. Pointers in this store are used to
57construct a revision graph, whose edges are edit and/or merge
64construct a revision graph, whose edges are edit and/or merge
58−actions. Some sort of namespace attaches meaningful names to
65+actions. Some sort of namespace attaches meaningful names to
59current revisions in this graph.
66current revisions in this graph.
61There are six big questions in the design of a DVCS:
68There are six big questions in the design of a DVCS:
@@ -71,16 +78,16 @@ There are six big questions in the design of a DVCS:
72A good way to summarize `%clay` (at present) is that its answers
79A good way to summarize `%clay` (at present) is that its answers
73to the first four questions are interesting, and its answers to
80to the first four questions are interesting, and its answers to
74−the last two are boring. We'll talk about all six, though.
81+the last two are boring. We'll talk about all six, though.
76### Global namespace: [ship desk case]
83### Global namespace: [ship desk case]
78−The permanent global name of a `%clay` snapshot is a `++beak`. A beak
85+The permanent global name of a `%clay` snapshot is a `++beak`. A beak
79is a triple: `[ship desk case]`.
86is a triple: `[ship desk case]`.
81−The `++ship` is an Urbit ship. A ship is a network address, a
88+The `++ship` is an Urbit ship. A ship is a network address, a
82−human-memorable name and a cryptographic identity. The address
89+human-memorable name and a cryptographic identity. The address
83−width is variable; shorter addresses are more desirable. Most
90+width is variable; shorter addresses are more desirable. Most
84users have 32-bit addresses, which we map to four-syllable
91users have 32-bit addresses, which we map to four-syllable
85synthetic names like `~sorreg-namtyv`.
92synthetic names like `~sorreg-namtyv`.
@@ -93,44 +100,44 @@ In the Urbit path syntax, the beak is the first three segments of
93the path: `/ship/desk/case/my/cat/picture`.
100the path: `/ship/desk/case/my/cat/picture`.
95In most cases, a path parser has a default ship, desk and case,
102In most cases, a path parser has a default ship, desk and case,
96−each of which can be replaced by `=`. So `/=home=/my/cat/picture`
103+each of which can be replaced by `=`. So `/=home=/my/cat/picture`
97is the data at `/my/cat/picture` on your own ship, in the `%home`
104is the data at `/my/cat/picture` on your own ship, in the `%home`
98−desk, right now. Immutability doesn't have to be inconvenient.
105+desk, right now. Immutability doesn't have to be inconvenient.
100−`%clay` has no concept of a mutable name or a magic `HEAD`. Use
107+`%clay` has no concept of a mutable name or a magic `HEAD`. Use
101−the current date. If you're referring to remote data and you're
108+the current date. If you're referring to remote data and you're
102worried about clock skew, subtract a second or three, or be
109worried about clock skew, subtract a second or three, or be
103−prepared to block. (Or keep your clock unskewed, which shouldn't
110+prepared to block. (Or keep your clock unskewed, which shouldn't
104−be _that_ hard.) It's not like mutable-name schemes work any
111+be _that_ hard.) It's not like mutable-name schemes work any
105better than this anyway.
112better than this anyway.
107−(Immutability is central to `%clay`. `git` merely adopted the
114+(Immutability is central to `%clay`. `git` merely adopted the
108−reflog. `%clay` was born in it, molded by it...)
115+reflog. `%clay` was born in it, molded by it...)
110`%clay` is objective; each Urbit node hosts only one ship, but
117`%clay` is objective; each Urbit node hosts only one ship, but
111−treats domestic and foreign data (almost) the same way. If you
118+treats domestic and foreign data (almost) the same way. If you
112ask your `%clay` repository for someone else's data, it will
119ask your `%clay` repository for someone else's data, it will
113block the request until it gets the data, which it then caches.
120block the request until it gets the data, which it then caches.
114At present, we always fulfill requests from the origin ship,
121At present, we always fulfill requests from the origin ship,
115though in future we could certainly use it as a tracker.
122though in future we could certainly use it as a tracker.
117If `%clay` discards cached data, it keeps a hash, so that your
124If `%clay` discards cached data, it keeps a hash, so that your
118−urbit can't violate the local illusion of immutability. We
125+urbit can't violate the local illusion of immutability. We
119can't, of course, prevent anyone else's ship from sending
126can't, of course, prevent anyone else's ship from sending
120−conflicting claims in different directions. But we can prevent
127+conflicting claims in different directions. But we can prevent
121programs on our own ship from hearing about them, meaning that we
128programs on our own ship from hearing about them, meaning that we
122retain local referential transparency.
129retain local referential transparency.
124### Local structure: dome, arch, leaf and mark
131### Local structure: dome, arch, leaf and mark
126−A `%clay` snapshot is called a `++dome`. A dome is the `%clay`
133+A `%clay` snapshot is called a `++dome`. A dome is the `%clay`
127equivalent of a tree-structured filesystem.
134equivalent of a tree-structured filesystem.
129Each node in this tree is one uniform structure, an `++arch`,
136Each node in this tree is one uniform structure, an `++arch`,
130−which is both file and directory. A `++leaf` or data value in
137+which is both file and directory. A `++leaf` or data value in
131this arch is a typed document.
138this arch is a typed document.
133−A logical document type is called a `++mark`. A mark is a
140+A logical document type is called a `++mark`. A mark is a
134symbolic pointer to a Hoon source file, in the same dome, for an
141symbolic pointer to a Hoon source file, in the same dome, for an
135engine that can validate, compare, and/or convert
142engine that can validate, compare, and/or convert
136documents of that mark.
143documents of that mark.
@@ -139,8 +146,8 @@ Let's go over arches and marks a little more closely...
140#### An arch is a uniform filesystem node
147#### An arch is a uniform filesystem node
142−Most filesystems and RCS have modal nodes. For instance, a git
149+Most filesystems and RCS have modal nodes. For instance, a git
143−data node can be a blob or a tree. `%clay` (and the Urbit
150+data node can be a blob or a tree. `%clay` (and the Urbit
144namespace in general) has one kind of node, an `arch`:
151namespace in general) has one kind of node, an `arch`:
@@ -149,24 +156,24 @@ namespace in general) has one kind of node, an `arch`:
151An arch has an optional typed-data leaf, and optional named
158An arch has an optional typed-data leaf, and optional named
152−children. In Unix terms, it's a file, a directory, or both. It
159+children. In Unix terms, it's a file, a directory, or both. It
153can't be neither: an empty arch, with no data and no children, is
160can't be neither: an empty arch, with no data and no children, is
154−auto-pruned. As in `git` (but not in Unix), the hierarchy is a
161+auto-pruned. As in `git` (but not in Unix), the hierarchy is a
155−pure function of its leaves. Paths exist only because of the
162+pure function of its leaves. Paths exist only because of the
156data they point to.
163data they point to.
158The `++knot` names in an arch path are also a little special.
165The `++knot` names in an arch path are also a little special.
159They're typed, in a sense; any knot can be parsed invertibly into
166They're typed, in a sense; any knot can be parsed invertibly into
160−a typed atom. The atom is an arbitrary unsigned integer. Its
167+a typed atom. The atom is an arbitrary unsigned integer. Its
161type (an `++aura` if you know Hoon) defines the semantics and
168type (an `++aura` if you know Hoon) defines the semantics and
162presentation of the atom: for instance, a symbol, a decimal,
169presentation of the atom: for instance, a symbol, a decimal,
163a date, etc.
170a date, etc.
165Ultimately a filesystem is a database, and a path is a primary
172Ultimately a filesystem is a database, and a path is a primary
166−key. It's very useful to be able to store simple data in this
173+key. It's very useful to be able to store simple data in this
167key, in a standard and unambiguous way.
174key, in a standard and unambiguous way.
169−Why uniform nodes? One, they map nicely to information trees
176+Why uniform nodes? One, they map nicely to information trees
170where a subtree may need some custom collective presentation,
177where a subtree may need some custom collective presentation,
171not an automatically generated Apache style directory listing.
178not an automatically generated Apache style directory listing.
172Two, non-uniform nodes introduce a whole pointless category of
179Two, non-uniform nodes introduce a whole pointless category of
@@ -174,34 +181,34 @@ node type conflicts -- for which life is far too short.
175#### A mark is an executable document type
182#### A mark is an executable document type
177−A mark is "like an executable MIME type." It's best defined as a
184+A mark is "like an executable MIME type." It's best defined as a
178symbolic mapping from a filesystem to a schema engine.
185symbolic mapping from a filesystem to a schema engine.
180If you have a MIME type `text/html`, you need to watch the
187If you have a MIME type `text/html`, you need to watch the
181−Internets to track what the W3C thinks HTML is. If you have an
188+Internets to track what the W3C thinks HTML is. If you have an
182Urbit mark `%html`, you can functionally construct (using our
189Urbit mark `%html`, you can functionally construct (using our
183insanely powerful functional build vane `%ford`) a core which
190insanely powerful functional build vane `%ford`) a core which
184can validate or normalize HTML, apply patch and diff logic, and
191can validate or normalize HTML, apply patch and diff logic, and
185−translate to and from other marks. `%ford` can even reason about
192+translate to and from other marks. `%ford` can even reason about
186translation paths and construct multistep conversions.
193translation paths and construct multistep conversions.
188(Urbit does not actually have this super-smart `%html` mark right
195(Urbit does not actually have this super-smart `%html` mark right
189−now. But it probably should, huh. And this page was composed in
196+now. But it probably should, huh. And this page was composed in
190Markdown and translated to HTML by `%ford`.)
197Markdown and translated to HTML by `%ford`.)
192−Where does the source code of a mark live? Within the same dome
199+Where does the source code of a mark live? Within the same dome
193−as the data it describes. `%html` files are described by the
200+as the data it describes. `%html` files are described by the
194Hoon source at `/===/mar/html`, which perhaps gets mapped into
201Hoon source at `/===/mar/html`, which perhaps gets mapped into
195your Unix filesystem at `$ship/home/mar/html.hoon`.
202your Unix filesystem at `$ship/home/mar/html.hoon`.
197−This Hoon file is not self-contained. It has dependencies. The
204+This Hoon file is not self-contained. It has dependencies. The
198`%html` mark may depend on various structures and libraries
205`%html` mark may depend on various structures and libraries
199−elsewhere in Urbit. But the source code for all them is within
206+elsewhere in Urbit. But the source code for all them is within
200this dome.
207this dome.
202One constraint this puts on domes is that _all domes are
209One constraint this puts on domes is that _all domes are
203−descendants of the base system dome_. Domes are versions of one
210+descendants of the base system dome_. Domes are versions of one
204−reality, not fragments of that reality. You can't create a new
211+reality, not fragments of that reality. You can't create a new
205empty dome -- you'd have no mark source, and be unable to install
212empty dome -- you'd have no mark source, and be unable to install
206any validated files.
213any validated files.
@@ -212,13 +219,13 @@ weird collisions between non-parallel things don't happen.
213### Unix mapping and mounting
220### Unix mapping and mounting
215−Every DVCS is its own filesystem in a sense. But every DVCS
222+Every DVCS is its own filesystem in a sense. But every DVCS
216works by mapping and mounting itself, in some sense, to Unix.
223works by mapping and mounting itself, in some sense, to Unix.
218#### Mapping arches to Unix
225#### Mapping arches to Unix
220−How do we map arches into Unix? The arch structure maps 1:1 to
227+How do we map arches into Unix? The arch structure maps 1:1 to
221−the Unix directory tree. When the arch has a leaf, we derive a
228+the Unix directory tree. When the arch has a leaf, we derive a
222dot extension from the file's mark.
229dot extension from the file's mark.
224So, supposing we have a leaf `foo` which is an HTML document
231So, supposing we have a leaf `foo` which is an HTML document
@@ -232,19 +239,19 @@ foo/moo.html
234This maps invertibly back into `%clay`, and it looks perfectly
241This maps invertibly back into `%clay`, and it looks perfectly
235−unremarkable from the Unix side. File completion sometimes gets
242+unremarkable from the Unix side. File completion sometimes gets
236a little annoying.
243a little annoying.
238It would be if nice every possible Unix tree, even every
245It would be if nice every possible Unix tree, even every
239plausible Unix tree, mapped cleanly and completely to a `%clay`
246plausible Unix tree, mapped cleanly and completely to a `%clay`
240−dome. There are always horrid exceptions, even in common use --
247+dome. There are always horrid exceptions, even in common use --
241−like extensionless Makefiles. The trivial solution is that if
248+like extensionless Makefiles. The trivial solution is that if
242`%clay` finds a file mysterious, it won't track it.
249`%clay` finds a file mysterious, it won't track it.
244One often unquestioned assumption in new system software is that
251One often unquestioned assumption in new system software is that
245−compatibility with old content is important. It's nice, but not
252+compatibility with old content is important. It's nice, but not
246−that important. Most content in a new system is new content.
253+that important. Most content in a new system is new content.
247−The important compatibility direction is export, not import. So
254+The important compatibility direction is export, not import. So
248the new system should subset, not superset, the ancient formats.
255the new system should subset, not superset, the ancient formats.
250### Mounting desks to Unix
257### Mounting desks to Unix
@@ -255,36 +262,36 @@ the Unix directory of the same name within the pier.
255When files in a mount directory change, `%clay` notices it
262When files in a mount directory change, `%clay` notices it
256through `libuv` filesystem watching (ultimately `inotify()` or
263through `libuv` filesystem watching (ultimately `inotify()` or
257`FSEvents`), computes diffs, and automatically commits them to
264`FSEvents`), computes diffs, and automatically commits them to
258−the desk. And if `%clay` creates a new version for its own
265+the desk. And if `%clay` creates a new version for its own
259urbity reasons, it will revise the Unix files.
266urbity reasons, it will revise the Unix files.
261Keeping all mounted files in the same Unix subdirectory makes it
268Keeping all mounted files in the same Unix subdirectory makes it
262−easy to move the whole pier in a tarfile. Edits applied while
269+easy to move the whole pier in a tarfile. Edits applied while
263the `urbit` process isn't running will be detected and applied
270the `urbit` process isn't running will be detected and applied
264the next time it starts.
271the next time it starts.
266In conventional DVCS terms, `%clay` has a 1:1 static mapping from
273In conventional DVCS terms, `%clay` has a 1:1 static mapping from
267−branches to working directories. Why?
274+branches to working directories. Why?
269−Why not? Do we need that "what branches are checked out in what
276+Why not? Do we need that "what branches are checked out in what
270−directories" piece of state? Why force the user to worry about
277+directories" piece of state? Why force the user to worry about
271−which branches are mounted to which directories? Why not just
278+which branches are mounted to which directories? Why not just
272−mount them under their own names? Is there some crippling global
279+mount them under their own names? Is there some crippling global
273shortage of disk space for source files? Our planet is running
280shortage of disk space for source files? Our planet is running
274−out of inodes? Wut?
281+out of inodes? Wut?
276### Queries, subscriptions and synchronization
283### Queries, subscriptions and synchronization
278There are two ways to get data out of `%clay`: dereference the Urbit
285There are two ways to get data out of `%clay`: dereference the Urbit
279−namespace, or send a request/subscription. Sync is in userspace.
286+namespace, or send a request/subscription. Sync is in userspace.
281#### Dereferencing the namespace
288#### Dereferencing the namespace
283−`%clay` is just part of the full Urbit namespace. The full namespace
290+`%clay` is just part of the full Urbit namespace. The full namespace
284has a two-character prefix which expresses what vane to query
291has a two-character prefix which expresses what vane to query
285−(always `%c` for `%clay`). The suffix is a mode flag; `%x` for
292+(always `%c` for `%clay`). The suffix is a mode flag; `%x` for
286data, `%y` for a list of children, `%z` for the whole arch
293data, `%y` for a list of children, `%z` for the whole arch
287−subtree, `%w` for the version number. (Each vane has its own
294+subtree, `%w` for the version number. (Each vane has its own
290For example, the full path to my cat picture is
297For example, the full path to my cat picture is
@@ -298,6 +305,7 @@ For example, the full path to my cat picture is
298If the data model is `++image` (hypothetical -- we don't really
305If the data model is `++image` (hypothetical -- we don't really
299do image processing in Urbit right now), the normal way to load
306do image processing in Urbit right now), the normal way to load
300this data in a Hoon program is
307this data in a Hoon program is
302.^(image /cx/~sorreg-namtyv/home/[now]/my/cat/picture)
310.^(image /cx/~sorreg-namtyv/home/[now]/my/cat/picture)
@@ -314,15 +322,15 @@ learn it in future, `[~ ~]` if we know we'll never know it, or
314`[~ ~ value]` if we know it now.
322`[~ ~ value]` if we know it now.
316In high-level shell programming (`:dojo`), we can make apparently
324In high-level shell programming (`:dojo`), we can make apparently
317−"blocking" requests to `%clay`. What actually happens: when `%clay`
325+"blocking" requests to `%clay`. What actually happens: when `%clay`
318produces `~`, we stop trying to compute this expression and wait
326produces `~`, we stop trying to compute this expression and wait
319−on one or more blocking requests. When we are unblocked, we
327+on one or more blocking requests. When we are unblocked, we
320recompute again from scratch; the intermediate computation is not
328recompute again from scratch; the intermediate computation is not
321saved at all.
329saved at all.
323−Urbit doesn't like to block. It likes to save coherent data
331+Urbit doesn't like to block. It likes to save coherent data
324−structures, not incomplete computation contexts. When Urbit
332+structures, not incomplete computation contexts. When Urbit
325−pretends to block for you, this is a courtesy service. You
333+pretends to block for you, this is a courtesy service. You
326can return the courtesy of not expecting miracles from it.
334can return the courtesy of not expecting miracles from it.
327When in doubt, send an actual request.
335When in doubt, send an actual request.
@@ -333,7 +341,7 @@ namespace, but can (a) read across case (revision) ranges,
333and (b) filter by subtree.
341and (b) filter by subtree.
335If the revision range in a request stretches into the future, the
343If the revision range in a request stretches into the future, the
336−request becomes a subscription. Urbit messaging always regards a
344+request becomes a subscription. Urbit messaging always regards a
337single response as a special case of a subscription.
345single response as a special case of a subscription.
339Generally, the response stream created by a request starts with
347Generally, the response stream created by a request starts with
@@ -343,10 +351,10 @@ closed when the request range is complete.
344#### Userspace synchronization
352#### Userspace synchronization
346−Synchronization per se is done in userspace. In `%clay`, sync just
354+Synchronization per se is done in userspace. In `%clay`, sync just
347means an application (typically the `%kiln` module in the system
355means an application (typically the `%kiln` module in the system
348app `:hood`) subscribes to a remote desk, and saves its change
356app `:hood`) subscribes to a remote desk, and saves its change
349−stream locally. So sync is really an emergent feature of `%clay`.
357+stream locally. So sync is really an emergent feature of `%clay`.
351### Revision graph
359### Revision graph
@@ -354,16 +362,16 @@ stream locally. So sync is really an emergent feature of `%clay`.
354Some changes are proposed below, however.
362Some changes are proposed below, however.
356As in any DVCS, a dome is the child of one parent dome (an edit)
364As in any DVCS, a dome is the child of one parent dome (an edit)
357−or more parents (a merge). Often the parent of any revision is
365+or more parents (a merge). Often the parent of any revision is
358the previous version in the sequence, but it can be anything.
366the previous version in the sequence, but it can be anything.
360### Storage model
368### Storage model
362−`%clay`'s storage model is completely uninteresting. It's also a
370+`%clay`'s storage model is completely uninteresting. It's also a
363little funky and I want to change it, so I'll be super vague.
371little funky and I want to change it, so I'll be super vague.
365A `++lobe`, a 128-bit atom, is the Merkle hash of a `blob`, which
373A `++lobe`, a 128-bit atom, is the Merkle hash of a `blob`, which
366−is essentially a leaf. All blobs are stored in a hash table
374+is essentially a leaf. All blobs are stored in a hash table
367(`(map lobe blob)`), and reference each other through lobes.
375(`(map lobe blob)`), and reference each other through lobes.
369At present we don't GC this space at all, because we don't
377At present we don't GC this space at all, because we don't
@@ -372,12 +380,12 @@ actually delete any roots.
372A `blob` may be `%direct`, meaning it's defined as the data
380A `blob` may be `%direct`, meaning it's defined as the data
373itself, or `%delta`, defined as a change against another blob.
381itself, or `%delta`, defined as a change against another blob.
374This is motivated by the desire not to store a full snapshot
382This is motivated by the desire not to store a full snapshot
375−on every file change. But it may be misguided (see changes #0
383+on every file change. But it may be misguided (see changes #0
376and #1 below).
384and #1 below).
378We cache the full arch tree (`++ankh`) of the latest state in
386We cache the full arch tree (`++ankh`) of the latest state in
379−each desk. For current queries, this mitigates a lot of the
387+each desk. For current queries, this mitigates a lot of the
380−opportunities for slowness that this data model creates. But the
388+opportunities for slowness that this data model creates. But the
381disparity between the speed of the current version and the
389disparity between the speed of the current version and the
382speed previous version is an undesirable glitch in the API.
390speed previous version is an undesirable glitch in the API.
@@ -389,38 +397,38 @@ We actually have _three_ motivations for change...
390At present, we're still using `git` as a development platform.
398At present, we're still using `git` as a development platform.
391`%clay` is certainly used, but just for its strength --
399`%clay` is certainly used, but just for its strength --
392−deployment. This is starving Urbit of essential experience and
400+deployment. This is starving Urbit of essential experience and
393−basically keeping it lame. So, change is necessary.
401+basically keeping it lame. So, change is necessary.
395### Change must come from the community
403### Change must come from the community
397−"Why?" you ask? "Why must change come from the community?" Well,
405+"Why?" you ask? "Why must change come from the community?" Well,
398−one, duh. Change always comes from the community.
406+one, duh. Change always comes from the community.
400−But two, a great force for change has departed. Philip Monk,
408+But two, a great force for change has departed. Philip Monk,
401late maintainer of `%clay` (`~wictuc-folrex` in the old era,
409late maintainer of `%clay` (`~wictuc-folrex` in the old era,
402−`~wicdev-wisryt` in the new), is tired of types and typing. He
410+`~wicdev-wisryt` in the new), is tired of types and typing. He
403spent September roaming with the bison in Montana, and is now
411spent September roaming with the bison in Montana, and is now
404−literally on his way to Patagonia. On the one hand, Phil still
412+literally on his way to Patagonia. On the one hand, Phil still
405exists and is not at all unwilling to answer questions, even dumb
413exists and is not at all unwilling to answer questions, even dumb
406−questions. On the other hand, _Patagonia_.
414+questions. On the other hand, _Patagonia_.
408−So %clay needs a new owner, or owners. At this point, we think
416+So %clay needs a new owner, or owners. At this point, we think
409the community might be indoctrinated enough to take a stab.
417the community might be indoctrinated enough to take a stab.
411Alas, `%clay` already has a pretty checkered ownership history.
419Alas, `%clay` already has a pretty checkered ownership history.
412Long ago I built a `%clay` that was really the only serious
420Long ago I built a `%clay` that was really the only serious
413−design failure in Arvo. Then Jared Hance, a brilliant summer
421+design failure in Arvo. Then Jared Hance, a brilliant summer
414intern, took my broken `%clay` and reformed it into a minimal `git`
422intern, took my broken `%clay` and reformed it into a minimal `git`
415−in about a week. Then Philip modified Jared's `%clay` to be a
423+in about a week. Then Philip modified Jared's `%clay` to be a
416_typed_ revision-control system, which is a pretty cool thing.
424_typed_ revision-control system, which is a pretty cool thing.
418But... in none of these transitions did `%clay` actually get
426But... in none of these transitions did `%clay` actually get
419−_rewritten_ per se. While it's a reasonably good vane, it
427+_rewritten_ per se. While it's a reasonably good vane, it
420does contain one or two historical decisions.
428does contain one or two historical decisions.
422We know a lot more about Arvo than we used to, and I think we can
430We know a lot more about Arvo than we used to, and I think we can
423−revisit many of these decisions. The point of this post is to
431+revisit many of these decisions. The point of this post is to
424start a `%clay` design discussion that will evolve into actual
432start a `%clay` design discussion that will evolve into actual
425collective action.
433collective action.
@@ -430,41 +438,41 @@ We're planning a rewards program for contributions to `%clay`.
430It'll simply involve setting a time period and a pool of real
438It'll simply involve setting a time period and a pool of real
431estate, and allocating the pool by judged contribution.
439estate, and allocating the pool by judged contribution.
433−This is an experimental program. We'll say more about it later.
441+This is an experimental program. We'll say more about it later.
434−It may not even happen. It may happen very differently. When in
442+It may not even happen. It may happen very differently. When in
435doubt, don't talk about Fight Club.
443doubt, don't talk about Fight Club.
437## Some openings for change
445## Some openings for change
439−Change, of course, must come from the community. No one can
447+Change, of course, must come from the community. No one can
440−dictate change from the top down. We _know_ this. But...
448+dictate change from the top down. We _know_ this. But...
442This list of suggested changes for the next `%clay` is not in any
450This list of suggested changes for the next `%clay` is not in any
443−sort of order: priority, structure, difficulty, etc. Let's talk
451+sort of order: priority, structure, difficulty, etc. Let's talk
444−about the changes first; make actual plans second. The numbers
452+about the changes first; make actual plans second. The numbers
445here are just for convenience.
453here are just for convenience.
447−*All changes in this program will happen on the next-generation
455+_All changes in this program will happen on the next-generation
448−(`cc-release`) branch.*
456+(`cc-release`) branch._
450### Change #0: state reform
458### Change #0: state reform
452As mentioned above, the storage model in `%clay`, as the code is
460As mentioned above, the storage model in `%clay`, as the code is
453−at present, feels a little funky to me. There's nothing terribly
461+at present, feels a little funky to me. There's nothing terribly
454awful about it, but it doesn't feel quite right and its scaling
462awful about it, but it doesn't feel quite right and its scaling
455properties are very questionable.
463properties are very questionable.
457Briefly, `%clay` today actually derives the hierarchy from its
465Briefly, `%clay` today actually derives the hierarchy from its
458−leaves -- rather than _logically_ so deriving it. Only leaves
466+leaves -- rather than _logically_ so deriving it. Only leaves
459−are in the object store. This is just weird, and you can feel
467+are in the object store. This is just weird, and you can feel
460−`%clay` trying to be something other than what it is. (I blame
468+`%clay` trying to be something other than what it is. (I blame
461Jared, though it's actually my fault.)
469Jared, though it's actually my fault.)
463To store a dome snapshot (`++yaki`), the present code keeps a
471To store a dome snapshot (`++yaki`), the present code keeps a
464−`(map path blob)` at the root of the dome. This is enough to
472+`(map path blob)` at the root of the dome. This is enough to
465reconstruct the `++arch` structures the namespace exports.
473reconstruct the `++arch` structures the namespace exports.
466However, all kinds of basic arch operations, like basically `ls`,
474However, all kinds of basic arch operations, like basically `ls`,
467−become `O(n)` in the number of leaves in the dome. This is
475+become `O(n)` in the number of leaves in the dome. This is
468obviously not quite right.
476obviously not quite right.
470I believe the storage structure should be (removing everything
478I believe the storage structure should be (removing everything
@@ -485,46 +493,46 @@ Here a `++root` is the universal node, with children addressed by
485hash, but logically representing the abstract `++arch`.
493hash, but logically representing the abstract `++arch`.
487Users of hierarchical filesystems basically expect performance
495Users of hierarchical filesystems basically expect performance
488−qualities that match the filesystem structure. A huge directory
496+qualities that match the filesystem structure. A huge directory
489−in one place shouldn't affect search speeds somewhere else. And
497+in one place shouldn't affect search speeds somewhere else. And
490of course, `ls` can't be O(n) in the size of the filesystem.
498of course, `ls` can't be O(n) in the size of the filesystem.
492### Change #1: store edges, cache snapshots
500### Change #1: store edges, cache snapshots
494−The current `%clay` is snapshot-oriented, like `git`. The actual
502+The current `%clay` is snapshot-oriented, like `git`. The actual
495−changes are discarded. I think this is a bad choice.
503+changes are discarded. I think this is a bad choice.
497`%clay` is not a research project, and has no idea of going as
505`%clay` is not a research project, and has no idea of going as
498−far as `darcs` in the "set-of-patches" direction. However, at
506+far as `darcs` in the "set-of-patches" direction. However, at
499present we actually throw away the patch (`++nori`).
507present we actually throw away the patch (`++nori`).
501I think the right DVCS design for `%clay` is a snapshot-oriented
509I think the right DVCS design for `%clay` is a snapshot-oriented
502−system which also keeps the patches. The revision graph has the
510+system which also keeps the patches. The revision graph has the
503−same general shape as in the old `git` style. But rather than
511+same general shape as in the old `git` style. But rather than
504just deriving the diffs from the snapshots, we can also derive
512just deriving the diffs from the snapshots, we can also derive
505−the snapshots from the diffs. When in doubt, we save space by
513+the snapshots from the diffs. When in doubt, we save space by
506dropping the snapshot, not the diff.
514dropping the snapshot, not the diff.
508For an RCS which is also a general-purpose typed filesystem, it's
516For an RCS which is also a general-purpose typed filesystem, it's
509clear that considering edges (deltas, diffs, changes, etc) as the
517clear that considering edges (deltas, diffs, changes, etc) as the
510−primary state is optimal. You can use diff functions to derive
518+primary state is optimal. You can use diff functions to derive
511deltas from snapshots, but in many data models this will always
519deltas from snapshots, but in many data models this will always
512−risk losing information from the original action. When in
520+risk losing information from the original action. When in
513doubt, save what the user did.
521doubt, save what the user did.
515This is even more true for operations on the tree structure, not
523This is even more true for operations on the tree structure, not
516−on individual leaves. Consider the heuristic energy that `git` has
524+on individual leaves. Consider the heuristic energy that `git` has
517to go through to figure out when `mv` has been used -- even when
525to go through to figure out when `mv` has been used -- even when
518−`git mv` has been used! (According to the `git` faq, this is
526+`git mv` has been used! (According to the `git` faq, this is
519because the lowest common denominator remains Linus' style of
527because the lowest common denominator remains Linus' style of
520working with emailed patches.)
528working with emailed patches.)
522−Edges probably don't even need to be deduplicated. They also can
530+Edges probably don't even need to be deduplicated. They also can
523contain metadata such as commit messages (which `%clay` has no
531contain metadata such as commit messages (which `%clay` has no
524place for at present).
532place for at present).
526Once edges are recognized as unreclaimed primary state, snapshots
534Once edges are recognized as unreclaimed primary state, snapshots
527−can always be derived from the edges. So we can reclaim memory
535+can always be derived from the edges. So we can reclaim memory
528being used for snapshots.
536being used for snapshots.
530It's also straightforward to have a kind of edge which is a list
538It's also straightforward to have a kind of edge which is a list
@@ -532,27 +540,27 @@ of edges, letting us collapse feature branches trivially.
533Finally, I think references in edges should wherever possible
541Finally, I think references in edges should wherever possible
534indirect through the namespace, rather than directly to the
542indirect through the namespace, rather than directly to the
535−content address. The user has strictly more information if you
543+content address. The user has strictly more information if you
536say you merged `/~sorreg-namtyv/base/987/`, than if all you have
544say you merged `/~sorreg-namtyv/base/987/`, than if all you have
537is the hash that this reference refers to.
545is the hash that this reference refers to.
539### Change #2: unconfuse merge and rebase
547### Change #2: unconfuse merge and rebase
541The dispute between merge and rebase has all the bad smells of
549The dispute between merge and rebase has all the bad smells of
542−a historical wart. But here I'm just not sure what's right.
550+a historical wart. But here I'm just not sure what's right.
544The user has one _intent_ whether merging or rebasing, which is
552The user has one _intent_ whether merging or rebasing, which is
545−to combine changes on one desk into another. The _algorithm_
553+to combine changes on one desk into another. The _algorithm_
546used to construct this change, whether operational transformation
554used to construct this change, whether operational transformation
547(rebasing) or a 3-way diff (merging) has one goal: use all
555(rebasing) or a 3-way diff (merging) has one goal: use all
548available information to model the authors' editing intent as
556available information to model the authors' editing intent as
549−realistically as possible. If we save edges, we should use
557+realistically as possible. If we save edges, we should use
552Whether to define the resulting change graph with the typical
560Whether to define the resulting change graph with the typical
553structure of a merge, or the typical structure of a rebase,
561structure of a merge, or the typical structure of a rebase,
554strikes me as quite arbitrary, and well into the land of "pick
562strikes me as quite arbitrary, and well into the land of "pick
555−one and force everyone to stick with it." But this call demands
563+one and force everyone to stick with it." But this call demands
556a real expert on RCS workflow, which I'm not.
564a real expert on RCS workflow, which I'm not.
558Since `%clay` is a humble part of a larger system and needs to
566Since `%clay` is a humble part of a larger system and needs to
@@ -562,23 +570,23 @@ different preferences" is not usually the right answer.
562### Change #3: comment jihad
570### Change #3: comment jihad
564`%clay` needs to be brought up to the documentation standards of
572`%clay` needs to be brought up to the documentation standards of
565−`%jael`. This means (a) the formatting matches; (b) the comments
573+`%jael`. This means (a) the formatting matches; (b) the comments
566are at the same level of quality and ubiquity.
574are at the same level of quality and ubiquity.
568The basic commenting standards are: (a) every core (class) needs
576The basic commenting standards are: (a) every core (class) needs
569a block comment explanation; (b) every gate (function) needs a
577a block comment explanation; (b) every gate (function) needs a
570line comment; (c) every face (variable) needs a line comment.
578line comment; (c) every face (variable) needs a line comment.
572−This is obviously not just a clerical task. To write these
580+This is obviously not just a clerical task. To write these
573−comments, you need to actually understand the code. This is a
581+comments, you need to actually understand the code. This is a
574nontrivial but extremely rewarding task, even if the code you're
582nontrivial but extremely rewarding task, even if the code you're
575commenting will get replaced.
583commenting will get replaced.
577Also, the _symbols_ in `%clay` must be brought up to `%jael`
585Also, the _symbols_ in `%clay` must be brought up to `%jael`
578−quality. There is still far too much four-letterism here.
586+quality. There is still far too much four-letterism here.
580NB: `%clay` is about 3000 lines of pretty clean Hoon, which is
588NB: `%clay` is about 3000 lines of pretty clean Hoon, which is
581−certainly not at all undocumented. We're not exactly looking at
589+certainly not at all undocumented. We're not exactly looking at
582a Matterhorn of reverse engineering here.
590a Matterhorn of reverse engineering here.
584### Change #4: knot to coin internal paths
592### Change #4: knot to coin internal paths
@@ -589,26 +597,26 @@ ASCII symbol as an atom) must be within the range of the
589invertible noun printer, `++co`.
597invertible noun printer, `++co`.
591`++co` maps a described data value, a `++coin`, to an ASCII
599`++co` maps a described data value, a `++coin`, to an ASCII
592−string which is a valid URL segment and Unix filename. Its
600+string which is a valid URL segment and Unix filename. Its
593friend `++so` inverts the function, perfectly.
601friend `++so` inverts the function, perfectly.
595−Almost every knot is the simplest kind of `coin`, a `dime.` A
603+Almost every knot is the simplest kind of `coin`, a `dime.` A
596dime is a single atom with a rendering type (`aura` if you know
604dime is a single atom with a rendering type (`aura` if you know
597−Hoon). Auras include `@tas` for symbolic text, `@ud` for
605+Hoon). Auras include `@tas` for symbolic text, `@ud` for
598−unsigned decimal, `@p` for Urbit ship, etc. `++co` can also
606+unsigned decimal, `@p` for Urbit ship, etc. `++co` can also
599print arbitrary cells, though less prettily.
607print arbitrary cells, though less prettily.
601This feature is cool because it lets us stuff data into names in
609This feature is cool because it lets us stuff data into names in
602−a precise and predictable way. Usually the names in a namespace
610+a precise and predictable way. Usually the names in a namespace
603are human-meaningful and human-originated, and are therefore
611are human-meaningful and human-originated, and are therefore
604−symbolic, but there are exceptions. For example, blogposts may
612+symbolic, but there are exceptions. For example, blogposts may
605want to be named by date.
613want to be named by date.
607However, it is slightly retarded for the fundamental state of our
615However, it is slightly retarded for the fundamental state of our
608arches to store the _rendered_ string (`++knot`), not the source
616arches to store the _rendered_ string (`++knot`), not the source
609−data (`++coin`). It leads to all kinds of inefficiencies where
617+data (`++coin`). It leads to all kinds of inefficiencies where
610−we're doing way more parsing than we want to. (Parsing also
618+we're doing way more parsing than we want to. (Parsing also
611−should be jet-propelled, but is not.) It also makes it way too
619+should be jet-propelled, but is not.) It also makes it way too
612easy to sneak in bad data and make it `%clay`'s problem.
620easy to sneak in bad data and make it `%clay`'s problem.
614So, logically, we'd be shifting from
622So, logically, we'd be shifting from
@@ -623,7 +631,7 @@ So, logically, we'd be shifting from
624Also, a vanilla `++map` might not be quite the right data
632Also, a vanilla `++map` might not be quite the right data
625structure at this point -- we probably want a custom sorted tree,
633structure at this point -- we probably want a custom sorted tree,
626−which separates auras and imposes a custom order on each. This
634+which separates auras and imposes a custom order on each. This
627would let us serve simple range queries quite efficiently, so
635would let us serve simple range queries quite efficiently, so
628long as they match the standard aura order (eg, alphabetical
636long as they match the standard aura order (eg, alphabetical
629order for symbols).
637order for symbols).
@@ -631,23 +639,23 @@ order for symbols).
631### Change #5: dome semantics by reference
639### Change #5: dome semantics by reference
633While the principle that a dome contains its own mark source is
641While the principle that a dome contains its own mark source is
634−elegant, it's a pain in the ass to implement properly. `%clay`
642+elegant, it's a pain in the ass to implement properly. `%clay`
635certainly does not do it justice at present.
643certainly does not do it justice at present.
637The complexity is in checking every change to see whether it
645The complexity is in checking every change to see whether it
638includes a change to the mark source (which is very unlikely),
646includes a change to the mark source (which is very unlikely),
639or a dependency of the mark source (slightly less unlikely).
647or a dependency of the mark source (slightly less unlikely).
640And if so, do we have to update all the leaves in the dome?
648And if so, do we have to update all the leaves in the dome?
641−This is also very unlikely. And yet, it happens.
649+This is also very unlikely. And yet, it happens.
643Instead, we could simply designate, by beak (ship, desk, and
651Instead, we could simply designate, by beak (ship, desk, and
644−case) a dome which contains the canonical mark source. This
652+case) a dome which contains the canonical mark source. This
645would rarely change -- you could upgrade a desk to a new mark
653would rarely change -- you could upgrade a desk to a new mark
646dome, but it would be a manual operation and probably a rare one.
654dome, but it would be a manual operation and probably a rare one.
648The common case would just be that everyone uses the current
656The common case would just be that everyone uses the current
649version of the `%base` desk when a desk is created, and never
657version of the `%base` desk when a desk is created, and never
650−changes it. In practice, this means not every two-byte edit
658+changes it. In practice, this means not every two-byte edit
651needs to get involved in detecting and/or applying schema
659needs to get involved in detecting and/or applying schema
652evolution, a welcome simplification.
660evolution, a welcome simplification.
@@ -659,14 +667,14 @@ design is one mark per message structure, the mark namespace gets
659overloaded fast.
667overloaded fast.
661So we added an app hierarchy within the mark namespace, using hep
669So we added an app hierarchy within the mark namespace, using hep
662−`-` as a divider. The `:talk` app expects messages with the mark
670+`-` as a divider. The `:talk` app expects messages with the mark
663`talk-message`, whose sources is in `/===/mar/talk/message`.
671`talk-message`, whose sources is in `/===/mar/talk/message`.
665This helps, but it doesn't solve another two problems: trivial
673This helps, but it doesn't solve another two problems: trivial
666nouns and containers.
674nouns and containers.
668It's retarded to have to add a source file to send a `@ud` or a
676It's retarded to have to add a source file to send a `@ud` or a
669−`*`. An aura (see above under `coin`) should be a trivial mark.
677+`*`. An aura (see above under `coin`) should be a trivial mark.
671Also, we need to be able to construct standard containers of
679Also, we need to be able to construct standard containers of
672marks: lists, maps, sets, and tuples (or, as a database person
680marks: lists, maps, sets, and tuples (or, as a database person
@@ -678,32 +686,32 @@ Another kind of complex mark that needs to exist: conflict,
678difference, and invalid marks.
686difference, and invalid marks.
680Conflict data and difference data are two types of noun that are
688Conflict data and difference data are two types of noun that are
681−generated in normal RCS operation. If we have the concept of an
689+generated in normal RCS operation. If we have the concept of an
682`%html` noun, we also need the concept of an invertible change
690`%html` noun, we also need the concept of an invertible change
683from one `%html` file to another; a merge conflict between two
691from one `%html` file to another; a merge conflict between two
684`%html` files; and even a conflict between two marks.
692`%html` files; and even a conflict between two marks.
686There are two kinds of changes to a dome: manual and automatic.
694There are two kinds of changes to a dome: manual and automatic.
687Manual changes are initiated by the user and should crash, unless
695Manual changes are initiated by the user and should crash, unless
688−manually overridden. Automatic changes should not crash, ever,
696+manually overridden. Automatic changes should not crash, ever,
689−and should not lose data. If they create problems, which they
697+and should not lose data. If they create problems, which they
690shouldn't, these problems will need to be cleaned up manually.
698shouldn't, these problems will need to be cleaned up manually.
692This is a complex space to navigate, but it requires a conflicted
700This is a complex space to navigate, but it requires a conflicted
693merge to be able to resolve to a valid dome containing conflict
701merge to be able to resolve to a valid dome containing conflict
694−leaves, not crash. The present `%clay` goes halfway there and
702+leaves, not crash. The present `%clay` goes halfway there and
695resolves to a scratch desk, but we can go farther: the whole `git`
703resolves to a scratch desk, but we can go farther: the whole `git`
696−dumpster-fire of "merge state" deserves destruction. One change
704+dumpster-fire of "merge state" deserves destruction. One change
697is a merge that produces conflicts, the next is an edit that
705is a merge that produces conflicts, the next is an edit that
698resolves them.
706resolves them.
700−Similarly, all sorts of errors can produce invalid leaves. One
708+Similarly, all sorts of errors can produce invalid leaves. One
701particularly rare case which is nonetheless important is schema
709particularly rare case which is nonetheless important is schema
702evolution: what happens to files in a mark, when the mark source
710evolution: what happens to files in a mark, when the mark source
703−code changes. They all need to be re-normalized. And this
711+code changes. They all need to be re-normalized. And this
704−operation can fail, producing an invalid value. (We don't do any
712+operation can fail, producing an invalid value. (We don't do any
705of this now, BTW, although `%ford` has the dependency mechanism to
713of this now, BTW, although `%ford` has the dependency mechanism to
706−make it possible.) Again, in some cases this should crash the
714+make it possible.) Again, in some cases this should crash the
707update to the mark source; in other cases, there is no local way
715update to the mark source; in other cases, there is no local way
708to handle the crash.
716to handle the crash.
@@ -714,46 +722,46 @@ into using the bad content.
715The bottom line: a mark needs to be split into two models:
723The bottom line: a mark needs to be split into two models:
716`++mark`, the original symbolic mark with its source engine, and
724`++mark`, the original symbolic mark with its source engine, and
717−something like `++logo`, the meta-mark or mark container. The
725+something like `++logo`, the meta-mark or mark container. The
718mark is the building block; the logo is the building.
726mark is the building block; the logo is the building.
720### Change #8: filesystem hierarchy standard
728### Change #8: filesystem hierarchy standard
722While all filesystems remain forks of one ur-filesystem, the
730While all filesystems remain forks of one ur-filesystem, the
723−standard tends to enforce itself. But not define itself.
731+standard tends to enforce itself. But not define itself.
725−I've always hated the current `/mar` for marks, for instance. I
733+I've always hated the current `/mar` for marks, for instance. I
726−prefer `/mac`. But others hate this. And so on. Also, by
734+prefer `/mac`. But others hate this. And so on. Also, by
727necessity, our nascent "FHS" is heavily biased toward system
735necessity, our nascent "FHS" is heavily biased toward system
728−data, away from user data. This is retarded.
736+data, away from user data. This is retarded.
730−Please focus all bikeshedding energy on this problem! Better to
738+Please focus all bikeshedding energy on this problem! Better to
731bikeshed about what color to paint the bikeshed, then what shape
739bikeshed about what color to paint the bikeshed, then what shape
732the bike gears should be.
740the bike gears should be.
734### Change #9: security and access control
742### Change #9: security and access control
736−Oh, yeah, that! They always say not to design security in at the
744+Oh, yeah, that! They always say not to design security in at the
737−last minute. Unfortunately, this is exactly what we did. There
745+last minute. Unfortunately, this is exactly what we did. There
738−is no security at all in present `%clay`. (There is also no push
746+is no security at all in present `%clay`. (There is also no push
739operation in `%clay`, so it's not as mad as you think -- anyone
747operation in `%clay`, so it's not as mad as you think -- anyone
740can read anything, no one can write anything, which is fine in a
748can read anything, no one can write anything, which is fine in a
741young network where there are no secrets.)
749young network where there are no secrets.)
743Fortunately, we don't have anything super fancy planned in this
751Fortunately, we don't have anything super fancy planned in this
744−area, just basic access-control lists. These obviously need to
752+area, just basic access-control lists. These obviously need to
745be designed along with the filesystem-hierarchy standard (see
753be designed along with the filesystem-hierarchy standard (see
746elsewhere).
754elsewhere).
748−Is it even possible to do ACLs properly in `%clay`? At this
756+Is it even possible to do ACLs properly in `%clay`? At this
749−point, yes. There is an event-oriented security model in Arvo,
757+point, yes. There is an event-oriented security model in Arvo,
750though it isn't really hooked up to anything at the moment.
758though it isn't really hooked up to anything at the moment.
751−There is also a privilege badge system in `%jael`. The
759+There is also a privilege badge system in `%jael`. The
752combination of these two features is quite sufficient for a
760combination of these two features is quite sufficient for a
753decent ACL model.
761decent ACL model.
755The idea of event-based security is that every event passed to
763The idea of event-based security is that every event passed to
756−`%clay` has a security `++mask`. The mask tells us:
764+`%clay` has a security `++mask`. The mask tells us:
758- the set of ships that has _contaminated_ this event
766- the set of ships that has _contaminated_ this event
759- the set of ships that has _endorsed_ this event
767- the set of ships that has _endorsed_ this event
@@ -761,20 +769,20 @@ The idea of event-based security is that every event passed to
762To simplify this system a little and maybe explain it, the
770To simplify this system a little and maybe explain it, the
763security question in any event is: (a) who has _caused_ this
771security question in any event is: (a) who has _caused_ this
764−event? (b) who may receive its _effects_? (These sets never
772+event? (b) who may receive its _effects_? (These sets never
765include your urbit itself, of course, only neighbors.)
773include your urbit itself, of course, only neighbors.)
767For example, if we're doing something because ship X sent us a
775For example, if we're doing something because ship X sent us a
768packet, and we're going to leak the results of this something
776packet, and we're going to leak the results of this something
769back in a response to ship X, once `%clay` knows what our ship
777back in a response to ship X, once `%clay` knows what our ship
770thinks of its neighbor X, we should have a good picture of what
778thinks of its neighbor X, we should have a good picture of what
771−effects this event should be allowed to cause. If `%jael` knows
779+effects this event should be allowed to cause. If `%jael` knows
772that X is our friend, `%clay` can know that X can see our cat
780that X is our friend, `%clay` can know that X can see our cat
775Causality is separated out into contamination (data from ship X
783Causality is separated out into contamination (data from ship X
776got into the event) and endorsement (ship X signed the whole
784got into the event) and endorsement (ship X signed the whole
777−event). (This may be overdesign.) All these sets can be defined
785+event). (This may be overdesign.) All these sets can be defined
778as the empty set (eg, releasing to the empty set means no output)
786as the empty set (eg, releasing to the empty set means no output)
779or the universal set (input from the universal set could be from
787or the universal set (input from the universal set could be from
780anyone, ie, is unauthenticated).
788anyone, ie, is unauthenticated).
@@ -783,11 +791,11 @@ anyone, ie, is unauthenticated).
784When you read a leaf from a `%clay` dome, `%clay` is happy to
792When you read a leaf from a `%clay` dome, `%clay` is happy to
785perform any necessary mark conversions (the actual work is done
793perform any necessary mark conversions (the actual work is done
786−by the `%ford` build vane). You can also just ask for the leaf
794+by the `%ford` build vane). You can also just ask for the leaf
787in whatever mark `%clay` has it.
795in whatever mark `%clay` has it.
789It's confusing to think of this mark extension as part of the
797It's confusing to think of this mark extension as part of the
790−path. You shouldn't say `+cat /===/app/talk/hoon`. You should say
798+path. You shouldn't say `+cat /===/app/talk/hoon`. You should say
791`+cat /===/app/talk`, or maybe `+cat %hoon /===/app/talk`.
799`+cat /===/app/talk`, or maybe `+cat %hoon /===/app/talk`.
793A mark-extended namespace may be added to `%ford`, to compensate.
801A mark-extended namespace may be added to `%ford`, to compensate.
@@ -796,7 +804,7 @@ But type-converting reads are the rare case, not the common case.
796### Change #11: real queries and indexes
804### Change #11: real queries and indexes
798A filesystem should postpone being a database until it can no
806A filesystem should postpone being a database until it can no
799−longer do so. So, please don't think about this problem yet.
807+longer do so. So, please don't think about this problem yet.
800I'm joking, of course!
808I'm joking, of course!
802### Change #12: meta-query
810### Change #12: meta-query
@@ -805,24 +813,24 @@ Make it possible to query the list of desks in a ship :-)
806### Change #13: completely redesign Unix mounting
814### Change #13: completely redesign Unix mounting
808−This is actually a series of changes, described below. But
816+This is actually a series of changes, described below. But
809broadly...
817broadly...
811One of Urbit's experiments is live two-way synchronization
819One of Urbit's experiments is live two-way synchronization
812between the Unix filesystem and the revision-control store.
820between the Unix filesystem and the revision-control store.
813−Dropbox style, as it were. I think this experiment is basically
821+Dropbox style, as it were. I think this experiment is basically
814−a failure. It may be right for cat pictures, but I don't think
822+a failure. It may be right for cat pictures, but I don't think
815it's right for Urbit.
823it's right for Urbit.
817The fundamental problem is that there are all sorts of weird,
825The fundamental problem is that there are all sorts of weird,
818−flaky ways applications use a mutable filesystem. It's not at
826+flaky ways applications use a mutable filesystem. It's not at
819all unusual for an editor to change a file by deleting the old
827all unusual for an editor to change a file by deleting the old
820file, then writing the new one in its place.
828file, then writing the new one in its place.
822−You really don't want this to produce a delete change. A simple
830+You really don't want this to produce a delete change. A simple
823cooldown timer for the sync helps a lot with this, but causes
831cooldown timer for the sync helps a lot with this, but causes
824−other weirdnesses. In theory, a sync can even cause another
832+other weirdnesses. In theory, a sync can even cause another
825−change to the same version of the same desk. And so on.
833+change to the same version of the same desk. And so on.
827The thing we fear is just too funky to work (there is some
835The thing we fear is just too funky to work (there is some
828strange "reflux bug" still affecting it) is any sort of _two-way_
836strange "reflux bug" still affecting it) is any sort of _two-way_
@@ -833,8 +841,8 @@ cold storage, is that there's no useful workflow that actually
833uses it -- at least, when editing Urbit code.
841uses it -- at least, when editing Urbit code.
835It's actually really nice to have files on a disk that nothing
843It's actually really nice to have files on a disk that nothing
836−will automatically modify. If you're using `git` and `%clay` in
844+will automatically modify. If you're using `git` and `%clay` in
837−parallel, it's nice to mount your `git` working directory. The
845+parallel, it's nice to mount your `git` working directory. The
838original Urbit workflow was to work out of a mounted directory,
846original Urbit workflow was to work out of a mounted directory,
839but the two-way updates were just too weird and scary, so now we
847but the two-way updates were just too weird and scary, so now we
840tend to edit elsewhere and copy in -- which is lame.
848tend to edit elsewhere and copy in -- which is lame.
@@ -847,25 +855,25 @@ FUSE (in a somewhat experimental state), and filesystem watching
848The FUSE interface is a good one, though it needs more work.
856The FUSE interface is a good one, though it needs more work.
849Moreover, it exports the whole Urbit namespace to Unix, not just
857Moreover, it exports the whole Urbit namespace to Unix, not just
850−`%clay`. For reading Urbit data, FUSE is basically ideal.
858+`%clay`. For reading Urbit data, FUSE is basically ideal.
852−The obvious impulse is to make a read-write FUSE. But I don't
860+The obvious impulse is to make a read-write FUSE. But I don't
853think FUSE should be used for recording changes to an RCS, even
861think FUSE should be used for recording changes to an RCS, even
854though the FUSE API sends you changes as events.
862though the FUSE API sends you changes as events.
856−There are three reasons for this. One, the actual filesystem
864+There are three reasons for this. One, the actual filesystem
857writes that Unix apps make are typically of low quality and
865writes that Unix apps make are typically of low quality and
858−confusing. Some crack-brained editor will find it completely
866+confusing. Some crack-brained editor will find it completely
859normal and unremarkable to modify a byte in a file, by deleting
867normal and unremarkable to modify a byte in a file, by deleting
860−the file and rewriting it from scratch. You can use a cooldown
868+the file and rewriting it from scratch. You can use a cooldown
861timer to wait for this nonsense to converge, but leaving data in
869timer to wait for this nonsense to converge, but leaving data in
862heuristic limbo seems like a very bad idea in FUSE-land.
870heuristic limbo seems like a very bad idea in FUSE-land.
864The second is that people derive a lot of comfort from seeing
872The second is that people derive a lot of comfort from seeing
865−actual files on an actual disk. The idea that your datas are
873+actual files on an actual disk. The idea that your datas are
866stuck in Urbit is slightly terrifying as a user experience.
874stuck in Urbit is slightly terrifying as a user experience.
867−We claim to be reliable. It's actually hard to imagine datas that
875+We claim to be reliable. It's actually hard to imagine datas that
868−couldn't be recovered from either snapshot or log. But how can
876+couldn't be recovered from either snapshot or log. But how can
869you possibly believe us on this?
877you possibly believe us on this?
871So I think it's correct to lean all the weight of reading data
879So I think it's correct to lean all the weight of reading data
@@ -880,35 +888,35 @@ But this is a separate discussion.
881### Change #15: one-way, nonced working desk
889### Change #15: one-way, nonced working desk
883−A one-way mount is a much less crazy thing. It works the same
891+A one-way mount is a much less crazy thing. It works the same
884way our two-way mount works today, but with only one writer: the
892way our two-way mount works today, but with only one writer: the
885user, via Unix.
893user, via Unix.
887Since a mount is one-way, it needs to have an absolute write lock
895Since a mount is one-way, it needs to have an absolute write lock
888−on the desk it's writing to. We cannot have a `%home` desk which
896+on the desk it's writing to. We cannot have a `%home` desk which
889we are also editing with filesystem watching.
897we are also editing with filesystem watching.
891Instead, we have a temporary and anonymous state that's connected
899Instead, we have a temporary and anonymous state that's connected
892−itself with your `%home` desk. Desk names have to be valid
900+itself with your `%home` desk. Desk names have to be valid
893−symbols, but we can use `hep`, `-`. A reasonable choice is to
901+symbols, but we can use `hep`, `-`. A reasonable choice is to
894pick a random 32-bit name for each mount instance -- so the desk
902pick a random 32-bit name for each mount instance -- so the desk
895−name is not `%home`, but `%home-bosdut-sigder`. (Randomizing
903+name is not `%home`, but `%home-bosdut-sigder`. (Randomizing
896instead of always using a fixed suffix, say, `%home-work`,
904instead of always using a fixed suffix, say, `%home-work`,
897just makes all kinds of screwups less likely.)
905just makes all kinds of screwups less likely.)
899You create a mount simply by mkdiring the desk name in your pier.
907You create a mount simply by mkdiring the desk name in your pier.
900Just the base name, Urbit will come up with the `%bosdut-sigder`
908Just the base name, Urbit will come up with the `%bosdut-sigder`
901−part and put it in a `.desk` file at the desk root. To forget
909+part and put it in a `.desk` file at the desk root. To forget
902−the mount, just delete this file. This avoids the ugly confusion
910+the mount, just delete this file. This avoids the ugly confusion
903between unmounting a mount, and deleting all the data in it.
911between unmounting a mount, and deleting all the data in it.
905(Urbit will keep a set of the mount desks it's used and
913(Urbit will keep a set of the mount desks it's used and
906−forgotten, not reuse them, and block all requests into them. By
914+forgotten, not reuse them, and block all requests into them. By
907blocking, we can delete without losing referential transparency.
915blocking, we can delete without losing referential transparency.
908Yes, Virginia, Urbit can forget!)
916Yes, Virginia, Urbit can forget!)
910Note that the auto-reload behavior of applications is extremely
918Note that the auto-reload behavior of applications is extremely
911−useful when testing. When testing, you can either switch the
919+useful when testing. When testing, you can either switch the
912active application desk from `%home` to `%home-bosdut-sigder`,
920active application desk from `%home` to `%home-bosdut-sigder`,
913or activate autosync from the latter to the former.
921or activate autosync from the latter to the former.
@@ -931,7 +939,7 @@ Otherwise, you're basically, like, reinventing Perforce.
932More concretely, the Urbit experience today requires `:hood`
940More concretely, the Urbit experience today requires `:hood`
933pokes (Urbit shell commands, basically) for all nontrivial RCS
941pokes (Urbit shell commands, basically) for all nontrivial RCS
934−operations, from mounting to merging. Again, there are lots of
942+operations, from mounting to merging. Again, there are lots of
935situations in which you can write into a mounted filesystem (the
943situations in which you can write into a mounted filesystem (the
936protocol for Unix communication could be easily adapted to
944protocol for Unix communication could be easily adapted to
937communicate with a browser-based editor, for instance), but you
945communicate with a browser-based editor, for instance), but you
@@ -941,20 +949,20 @@ Even commands like mounting and merging are easily communicated,
941through filesystem writes that any Unix tool (including many
949through filesystem writes that any Unix tool (including many
942macro editors).
950macro editors).
944−This is very easy for a filesystem watcher. One such protocol:
952+This is very easy for a filesystem watcher. One such protocol:
945if you want to send the daemon a command relative to some
953if you want to send the daemon a command relative to some
946−directory, write a `.ur-command` metafile in that directory. The
954+directory, write a `.ur-command` metafile in that directory. The
947daemon accepts the command by deleting it.
955daemon accepts the command by deleting it.
949The user experience is that you configure your mounts by simply
957The user experience is that you configure your mounts by simply
950−making the directories and configuring dotfiles in them. Unlike
958+making the directories and configuring dotfiles in them. Unlike
951in the present system, Urbit will never modify your datas
959in the present system, Urbit will never modify your datas
952silently in the background.
960silently in the background.
954Here's what this looks like:
962Here's what this looks like:
956−Suppose you want to edit some data in the `%home` desk. Your
964+Suppose you want to edit some data in the `%home` desk. Your
957−data is in `/=home=/mar/talk`. You also want a read-only view
965+data is in `/=home=/mar/talk`. You also want a read-only view
958of `/=home=/arvo`.
966of `/=home=/arvo`.
960From Unix, you do something like this:
968From Unix, you do something like this:
@@ -967,18 +975,18 @@ From Unix, you do something like this:
967 $ touch home/arvo/.read
975 $ touch home/arvo/.read
970−Want to commit your changes? That means merging the working
978+Want to commit your changes? That means merging the working
971−desk, `%home-bosdut-sigder`, into the `%home`. Put a commit message
979+desk, `%home-bosdut-sigder`, into the `%home`. Put a commit message
972−into `home/.commit` and it'll happen. If there's an error, it
980+into `home/.commit` and it'll happen. If there's an error, it
973will appear as `.error`.
981will appear as `.error`.
975The mount has a .version file which simply contains the change
983The mount has a .version file which simply contains the change
976−number (of the working desk) which any edits are relative to. If
984+number (of the working desk) which any edits are relative to. If
977the urbit daemon is running and watching, edits will be absorbed
985the urbit daemon is running and watching, edits will be absorbed
978quickly (probably after a heuristic timeout), and the version
986quickly (probably after a heuristic timeout), and the version
979−incremented. But we always are confident that the edits are
987+incremented. But we always are confident that the edits are
980against the latest version, or at least we can use this as a
988against the latest version, or at least we can use this as a
981−sanity check. All kinds of insane things can happen to get a
989+sanity check. All kinds of insane things can happen to get a
982filesystem out of consistency.
990filesystem out of consistency.
984This design obviously needs a little fleshing out, but I'm quite
992This design obviously needs a little fleshing out, but I'm quite
@@ -987,12 +995,12 @@ from Unix.
988## In conclusion
996## In conclusion
990−There's obviously a lot of fun here! There are two things you
998+There's obviously a lot of fun here! There are two things you
991can do right now.
999can do right now.
993One, talk about `%clay` on [this forum
1001One, talk about `%clay` on [this forum
994thread](http://urbit.org/~~/fora/posts/~2016.10.25..17.37.47..6937~/) or
1002thread](http://urbit.org/~~/fora/posts/~2016.10.25..17.37.47..6937~/) or
995−on [:talk](http://urbit.org/stream). Both hosted on Urbit
1003+on [:talk](http://urbit.org/stream). Both hosted on Urbit
998Two, poke around the
1006Two, poke around the