@@ -1,79 +1,82 @@
2title = "Architecture"
2title = "Architecture"
4template = "doc.html"
4template = "doc.html"
7−Clay is the primary filesystem for the arvo operating system,
7+Clay is the primary filesystem for the [Arvo](@/docs/Arvo/overview.md) operating system,
8−which is the [core](/docs/glossary/core/) of an urbit. The architecture of clay is
8+which is the [core](/docs/glossary/core/) of an urbit. The architecture of Clay is
9−intrinsically connected with arvo, but we assume no knowledge of
9+intrinsically connected with Arvo, but for this section we assume no knowledge of
10−either arvo or urbit. We will point out only those features of
10+either Arvo or Urbit. We will point out only those features of
11−arvo that are necessary for an understanding of clay, and we will
11+Arvo that are necessary for an understanding of Clay, and we will
12do so only when they arise.
12do so only when they arise.
14−The first relevant feature of arvo is that it is a deterministic
14+The first relevant feature of Arvo is that it is a deterministic
15system where input and output are defined as a series of events
15system where input and output are defined as a series of events
16−and effects. The state of arvo is simply a function of its event
16+and effects. The state of Arvo is simply a pure function of its event
17log. None of the effects from an event are emitted until the
17log. None of the effects from an event are emitted until the
18event is entered in the log and persisted, either to disk or
18event is entered in the log and persisted, either to disk or
19another trusted source of persistence, such as a Kafka cluster.
19another trusted source of persistence, such as a Kafka cluster.
20−Consequently, arvo is a single-level store: everything in its
20+Consequently, Arvo is a single-level store: everything in its
21state is persistent.
21state is persistent.
23In a more traditional OS, everything in RAM can be erased at any
23In a more traditional OS, everything in RAM can be erased at any
24time by power failure, and is always erased on reboot. Thus, a
24time by power failure, and is always erased on reboot. Thus, a
25primary purpose of a filesystem is to ensure files persist across
25primary purpose of a filesystem is to ensure files persist across
26−power failures and reboots. In arvo, both power failures and
26+power failures and reboots. In Arvo, both power failures and
27reboots are special cases of suspending computation, which is
27reboots are special cases of suspending computation, which is
28done safely since our event log is already persistent. Therefore,
28done safely since our event log is already persistent. Therefore,
29−clay is not needed in arvo for persistence. Why, then, do we have a
29+Clay is not needed in Arvo for persistence. Why, then, do we have a
30filesystem? There are two answers to this question.
30filesystem? There are two answers to this question.
32−First, clay provides a filesystem tree, which is a convenient
32+First, Clay provides a filesystem tree, which is a convenient
33user interface for some applications. Unix has the useful concept
33user interface for some applications. Unix has the useful concept
34of virtual filesystems, which are used for everything from direct
34of virtual filesystems, which are used for everything from direct
35access to devices, to random number generators, to the /proc
35access to devices, to random number generators, to the /proc
36tree. It is easy and intuitive to read from and write to a
36tree. It is easy and intuitive to read from and write to a
37filesystem tree.
37filesystem tree.
39−Second, clay has a distributed revision-control system baked into
39+Second, Clay has a distributed revision-control system baked into
40it. Traditional filesystems are not revision controlled, so
40it. Traditional filesystems are not revision controlled, so
41userspace software -- such as git -- is written on top of them to
41userspace software -- such as git -- is written on top of them to
42−do so. clay natively provides the same functionality as modern
42+do so. Clay natively provides the same functionality as modern
43DVCSes, and more.
43DVCSes, and more.
45−clay has two other unique properties that we'll cover later on:
45+Clay has two other unique properties that we'll cover later on:
46it supports typed data and is referentially transparent.
46it supports typed data and is referentially transparent.
48### Revision Control
48### Revision Control
50−Every urbit has one or more "desks", which are independently
50+Every urbit has one or more `desk`s, which are independently
51−revision-controlled branches. Each desk contains its own mark
51+revision-controlled branches. Each `desk` contains its own `mark`
52−definitions, apps, doc, and so forth.
52+definitions, apps, and so forth.
54−Traditionally, an urbit has at least a base and a home desk. The
54+Traditionally, an urbit has at least a `%home` `desk` and `%kids` `desk`. The
55−base desk has all the system software from the distribution. the
55+`%home` `desk` has all the system software from the distribution, along with your
56−home desk is a fork of base with all the stuff specific to the
56+personal files. The `%home` `desk` is a fork of the `%kids` desk of whichever ship
57−user of the urbit.
57+you download [OTAs](@/docs/glossary/ota-updates.md) from - typically your sponsor,
58+but theoretically may be any ship. You will also find references to a `%base`
59+`desk` - which is not a real `desk`, rather it just a hash associated to the
60+`desk` from which your `%home` `desk` was forked.
59−A desk is a series of numbered commits, the most recent of which
62+A `desk` is a series of numbered commits, the most recent of which
60−represents the current state of the desk. A commit is composed of
63+represents the current state of the `desk`. A commit is composed of
61(1) an absolute time when it was created, (2) a list of zero or
64(1) an absolute time when it was created, (2) a list of zero or
62more parents, and (3) a map from paths to data.
65more parents, and (3) a map from paths to data.
64Most commits have exactly one parent, but the initial commit on a
67Most commits have exactly one parent, but the initial commit on a
65−desk may have zero parents, and merge commits have more than one
68+`desk` may have zero parents, and merge commits have more than one
68−The non-meta data is stored in the map of paths to data. It's
71+The non-metadata is stored as a `map` of `path`s to data. It's
69−worth noting that no constraints are put on this map, so, for
72+worth noting that no constraints are put on this `map`, so, for
70−example, both /a/b and /a/b/c could have data. This is impossible
73+example, both `/a/b` and `/a/b/c` could have data. This is impossible
71−in a traditional Unix filesystem since it means that /a/b is both
74+in a traditional Unix filesystem since it means that `/a/b` is both
72a file and a directory. Conventionally, the final element in the
75a file and a directory. Conventionally, the final element in the
73−path is its mark -- much like a filename extension in Unix. Thus,
76+path is its `mark` -- much like a filename extension in Unix. Thus,
74−/doc/readme.md in Unix is stored as /doc/readme/md in urbit.
77+`/doc/readme.md` in Unix is stored as `/doc/readme/md` in urbit.
76−The data is not stored directly in the map; rather, a hash of the
79+The data is not stored directly in the `map`; rather, a hash of the
77data is stored, and we maintain a master blob store. Thus, if the
80data is stored, and we maintain a master blob store. Thus, if the
78same data is referred to in multiple commits (as, for example,
81same data is referred to in multiple commits (as, for example,
79when a file doesn't change between commits), only the hash is
82when a file doesn't change between commits), only the hash is
@@ -85,8 +88,8 @@ only on the data within and not on whether or not it's stored
85directly, so we may on occasion rearrange the contents of the
88directly, so we may on occasion rearrange the contents of the
86blob store for performance reasons.
89blob store for performance reasons.
88−Recall that a desk is a series of numbered commits. Not every
91+Recall that a `desk` is a series of numbered commits. Not every
89−commit in a desk must be numbered. For example, if the base desk
92+commit in a `desk` must be numbered. For example, if the base `desk`
90has had 50 commits since home was forked from it, then a merge
93has had 50 commits since home was forked from it, then a merge
91from base to home will only add a single revision number to home,
94from base to home will only add a single revision number to home,
92although the full commit history will be accessible by traversing
95although the full commit history will be accessible by traversing
@@ -101,20 +104,20 @@ There are three ways to refer to particular commits in the
101revision history. Firstly, one can use the revision number.
104revision history. Firstly, one can use the revision number.
102Secondly, one can use any absolute time between the one numbered
105Secondly, one can use any absolute time between the one numbered
103commit and the next (inclusive of the first, exclusive of the
106commit and the next (inclusive of the first, exclusive of the
104−second). Thirdly, every desk has a map of labels to revision
107+second). Thirdly, every `desk` has a `map` of labels to revision
105numbers. These labels may be used to refer to specific commits.
108numbers. These labels may be used to refer to specific commits.
107−Additionally, clay is a global filesystem, so data on other urbit
110+Additionally, Clay is a global filesystem, so data on other urbits
108is easily accessible the same way as data on our local urbit. In
111is easily accessible the same way as data on our local urbit. In
109−general, the path to a particular revision of a desk is
112+general, the path to a particular revision of a `desk` is
110−/~urbit-name/desk-name/revision. Thus, to get /try/readme/md
113+`/~urbit-name/desk-name/revision`. Thus, to get `/try/readme/md`
111−from revision 5 of the home desk on ~sampel-sipnym, we refer to
114+from revision 5 of the home `desk` on `~sampel-sipnym`, we refer to
112−/~sampel-sipnym/home/5/try/readme/md. Clay's namespace is thus
115+`/~sampel-sipnym/home/5/try/readme/md`. Clay's namespace is thus
113global and referentially transparent.
116global and referentially transparent.
115### A Typed Filesystem
118### A Typed Filesystem
117−Since clay is a general filesystem for storing data of arbitrary
120+Since Clay is a general filesystem for storing data of arbitrary
118types, in order to revision control correctly it needs to be
121types, in order to revision control correctly it needs to be
119aware of types all the way through. Traditional revision control
122aware of types all the way through. Traditional revision control
120does an excellent job of handling source code, so for source code
123does an excellent job of handling source code, so for source code
@@ -164,33 +167,33 @@ a change. Two diffs consisting of a single added word to
164different sentences may be flagged as a conflict. In general,
167different sentences may be flagged as a conflict. In general,
165prose should be diffed by sentence, not by line.
168prose should be diffed by sentence, not by line.
167−As far as the author is aware, clay is the first generalized,
170+As far as we are aware, Clay is the first generalized,
168type-aware revision control system. We'll go into the workings
171type-aware revision control system. We'll go into the workings
169of this system in some detail.
172of this system in some detail.
173−Central to a typed filesystem is the idea of types. In clay, we
176+Central to a typed filesystem is the idea of types. In Clay, we
174−call these "marks". A mark is a file that defines a type,
177+call these `mark`s. A `mark` is a file that defines a type,
175−conversion routines to and from the mark, and diff, patch, and
178+conversion routines to and from the `mark`, and diff, patch, and
176merge routines.
179merge routines.
178For example, a `%txt` mark may be a list of lines of text, and it
181For example, a `%txt` mark may be a list of lines of text, and it
179may include conversions to `%mime` to allow it to be serialized
182may include conversions to `%mime` to allow it to be serialized
180−and sent to a browswer or to the unix filesystem. It will also
183+and sent to a browser or to the Unix filesystem. It will also
181include Hunt-McIlroy diff, patch, and merge algorithms.
184include Hunt-McIlroy diff, patch, and merge algorithms.
183−A `%json` mark would be defined as a json object in the code, and
186+A `%json` `mark` would be defined as a JSON object in the code, and
184it would have a parser to convert from `%txt` and a printer to
187it would have a parser to convert from `%txt` and a printer to
185convert back to `%txt`. The diff, patch, and merge algorithms are
188convert back to `%txt`. The diff, patch, and merge algorithms are
186−fairly straightforward for json, though they're very different
189+fairly straightforward for JSON, though they're very different
187from the text ones.
190from the text ones.
189−More formally, a mark is a core with three [arms](/docs/glossary/arm/), `++grab`,
192+More formally, a `mark` is a core with three [arms](/docs/glossary/arm/): `+grab`,
190−`++grow`, and `++grad`. In `++grab` is a series of functions to
193+`+grow`, and `+grad`. In `+grab` is a series of functions to
191−convert from other marks to the given mark. In `++grow` is a
194+convert from other `mark`s to the given `mark`. In `+grow` is a
192−series of functions to convert from the given mark to other
195+series of functions to convert from the given `mark` to other
193−marks. In `++grad` is `++diff`, `++pact`, `++join`, and `++mash`.
196+`mark`s. In `+grad` is `+diff`, `+pact`, `+join`, and `+mash`.
195The types are as follows, in an informal pseudocode:
198The types are as follows, in an informal pseudocode:
@@ -210,41 +213,41 @@ The types are as follows, in an informal pseudocode:
210 ++ mash: (<diff-type>, <diff-type>) -> <diff-type>
213 ++ mash: (<diff-type>, <diff-type>) -> <diff-type>
213−These types are basically what you would expect. Not every mark
216+These types are basically what you would expect. Not every `mark`
214has each of these functions defined -- all of them are optional
217has each of these functions defined -- all of them are optional
215in the general case.
218in the general case.
217−In general, for a particular mark, the `++grab` and `++grow` entries
220+In general, for a particular `mark`, the `+grab` and `+grow` entries
218(if they exist) should be inverses of each other.
221(if they exist) should be inverses of each other.
220−In `++grad`, `++diff` takes two instances of a mark and produces
223+In `+grad`, `+diff` takes two instances of a `mark` and produces
221−a diff of them. `++pact` takes an instance of a mark and patches
224+a diff of them. `+pact` takes an instance of a `mark` and patches
222−it with the given diff. `++join` takes two diffs and attempts to
225+it with the given diff. `+join` takes two diffs and attempts to
223merge them into a single diff. If there are conflicts, it
226merge them into a single diff. If there are conflicts, it
224−produces null. `++mash` takes two diffs and forces a merge,
227+produces null. `+mash` takes two diffs and forces a merge,
225annotating any conflicts.
228annotating any conflicts.
227−In general, if `++diff` called with A and B produces diff D, then
230+In general, if `+diff` called with A and B produces diff D, then
228−`++pact` called with A and D should produce B. Also, if `++join`
231+`+pact` called with A and D should produce B. Also, if `+join`
229−of two diffs does not produce null, then `++mash` of the same
232+of two diffs does not produce null, then `+mash` of the same
230diffs should produce the same result.
233diffs should produce the same result.
232−Alternately, instead of `++diff`, `++pact`, `++join`, and
235+Alternately, instead of `+diff`, `+pact`, `+join`, and
233−`++mash`, a mark can provide the same functionality by defining
236+`+mash`, a `mark` can provide the same functionality by defining
234−`++sted` to be the name of another mark to which we wish to
237+`+grad` to be the name of another `mark` to which we wish to
235delegate the revision control responsibilities. Then, before
238delegate the revision control responsibilities. Then, before
236−running any of those functions, clay will convert to the other
239+running any of those functions, Clay will convert to the other
237−mark, and convert back afterward. For example, the `%hoon` mark
240+`mark`, and convert back afterward. For example, the `%hoon` `mark`
238−is revision-controlled in the same way as `%txt`, so its `++grad`
241+is revision-controlled in the same way as `%txt`, so its `+grad`
239−is simply `++sted %txt`. Of course, `++txt` must be defined in
242+is simply `++ grad %txt`. Of course, `+txt` must be defined in
240−`++grow` and `++grab` as well.
243+`+grow` and `+grab` as well.
242−Every file in clay has a mark, and that mark must have a
245+Every file in Clay has a `mark`, and that `mark` must have a
243−fully-functioning `++grad`. Marks are used for more than just
246+fully-functioning `+grad`. `Mark`s are used for more than just
244−clay, and other marks don't need a `++grad`, but if a piece of
247+Clay, and other `mark`s don't need a `+grad`, but if a piece of
245−data is to be saved to clay, we must know how to revision-control
248+data is to be saved to Clay, we must know how to revision-control
248−Additionally, if a file is to be synced out to unix, then it must
251+Additionally, if a file is to be synced out to Unix, then it must
249−have conversion routines to and from the `%mime` mark.
252+have conversion routines to and from the `%mime` `mark`.