@@ -25,4 +25,2614 @@ zuse. Second is the write, query, and subscription logic. Finally, there
25is the logic for communicating requests to, and receiving requests from,
25is the logic for communicating requests to, and receiving requests from,
26foreign ships.
26foreign ships.
30+ ;div(id "architecture"): Architecture
34+Clay is the primary filesystem for the arvo operating system,
35+which is the core of an urbit. The architecture of clay is
36+intrinsically connected with arvo, but we assume no knowledge of
37+either arvo or urbit. We will point out only those features of
38+arvo that are necessary for an understanding of clay, and we will
39+do so only when they arise.
41+The first relevant feature of arvo is that it is a deterministic
42+system where input and output are defined as a series of events
43+and effects. The state of arvo is simply a function of its event
44+log. None of the effects from an event are emitted until the
45+event is entered in the log and persisted, either to disk or
46+another trusted source of persistence, such as a Kafka cluster.
47+Consequently, arvo is a single-level store: everything in its
50+In a more traditional OS, everything in RAM can be erased at any
51+time by power failure, and is always erased on reboot. Thus, a
52+primary purpose of a filesystem is to ensure files persist across
53+power failures and reboots. In arvo, both power failures and
54+reboots are special cases of suspending computation, which is
55+done safely since our event log is already persistent. Therefore,
56+clay is not needed in arvo for persistence. Why, then, do we have a
57+filesystem? There are two answers to this question.
59+First, clay provides a filesystem tree, which is a convenient
60+user interface for some applications. Unix has the useful concept
61+of virtual filesystems, which are used for everything from direct
62+access to devices, to random number generators, to the /proc
63+tree. It is easy and intuitive to read from and write to a
66+Second, clay has a distributed revision control system baked into
67+it. Traditional filesystems are not revision controlled, so
68+userspace software -- such as git -- is written on top of them to
69+do so. clay natively provides the same functionality as modern
72+clay has two other unique properties that we'll cover later on:
73+it supports typed data and is referentially transparent.
77+Every urbit has one or more "desks", which are independently
78+revision-controlled branches. Each desk contains its own mark
79+definitions, apps, doc, and so forth.
81+Traditionally, an urbit has at least a base and a home desk. The
82+base desk has all the system software from the distribution. the
83+home desk is a fork of base with all the stuff specific to the
86+A desk is a series of numbered commits, the most recent of which
87+represents the current state of the desk. A commit is composed of
88+(1) an absolute time when it was created, (2) a list of zero or
89+more parents, and (3) a map from paths to data.
91+Most commits have exactly one parent, but the initial commit on a
92+desk may have zero parents, and merge commits have more than one
95+The non-meta data is stored in the map of paths to data. It's
96+worth noting that no constraints are put on this map, so, for
97+example, both /a/b and /a/b/c could have data. This is impossible
98+in a traditional Unix filesystem since it means that /a/b is both
99+a file and a directory. Conventionally, the final element in the
100+path is its mark -- much like a filename extension in Unix. Thus,
101+/doc/readme.md in Unix is stored as /doc/readme/md in urbit.
103+The data is not stored directly in the map; rather, a hash of the
104+data is stored, and we maintain a master blob store. Thus, if the
105+same data is referred to in multiple commits (as, for example,
106+when a file doesn't change between commits), only the hash is
109+In the master blob store, we either store the data directly, or
110+else we store a diff against another blob. The hash is dependent
111+only on the data within and not on whether or not it's stored
112+directly, so we may on occasion rearrange the contents of the
113+blob store for performance reasons.
115+Recall that a desk is a series of numbered commits. Not every
116+commit in a desk must be numbered. For example, if the base desk
117+has had 50 commits since home was forked from it, then a merge
118+from base to home will only add a single revision number to home,
119+although the full commit history will be accessible by traversing
120+the parentage of the individual commits.
122+We do guarantee that the first commit is numbered 1, commits are
123+numbered consecutively after that (i.e. there are no "holes"),
124+the topmost commit is always numbered, and every numbered commit
125+is an ancestor of every later numbered commit.
127+There are three ways to refer to particular commits in the
128+revision history. Firstly, one can use the revision number.
129+Secondly, one can use any absolute time between the one numbered
130+commit and the next (inclusive of the first, exclusive of the
131+second). Thirdly, every desk has a map of labels to revision
132+numbers. These labels may be used to refer to specific commits.
134+Additionally, clay is a global filesystem, so data on other urbit
135+is easily accessible the same way as data on our local urbit. In
136+general, the path to a particular revision of a desk is
137+/~urbit-name/desk-name/revision. Thus, to get /try/readme/md
138+from revision 5 of the home desk on ~sampel-sipnym, we refer to
139+/~sampel-sipnym/home/5/try/readme/md. Clay's namespace is thus
140+global and referentially transparent.
144+### A Typed Filesystem
146+Since clay is a general filesystem for storing data of arbitrary
147+types, in order to revision control correctly it needs to be
148+aware of types all the way through. Traditional revision control
149+does an excellent job of handling source code, so for source code
150+we act very similar to traditional revision control. The
151+challenge is to handle other data similarly well.
153+For example, modern VCSs generally support "binary files", which
154+are files for which the standard textual diffing, patching, and
155+merging algorithms are not helpful. A "diff" of two binary files
156+is just a pair of the files, "patching" this diff is just
157+replacing the old file with the new one, and "merging"
158+non-identical diffs is always a conflict, which can't even be
159+helpfully annotated. Without knowing anything about the structure
160+of a blob of data, this is the best we can do.
162+Often, though, "binary" files have some internal structure, and
163+it is possible to create diff, patch, and merge algorithms that
164+take advantage of this structure. An image may be the result of a
165+base image with some set of operations applied. With algorithms
166+aware of this set of operations, not only can revision control
167+software save space by not having to save every revision of the
168+image individually, these transformations can be made on parallel
169+branches and merged at will.
171+Suppose Alice is tasked with touching up a picture, improving the
172+color balance, adjusting the contrast, and so forth, while Bob
173+has the job of cropping the picture to fit where it's needed and
174+adding textual overlay. Without type-aware revision control,
175+these changes must be made serially, requiring Alice and Bob to
176+explicitly coordinate their efforts. With type-aware revision
177+control, these operations may be performed in parallel, and then
178+the two changesets can be merged programmatically.
180+Of course, even some kinds of text files may be better served by
181+diff, patch, and merge algorithms aware of the structure of the
182+files. Consider a file containing a pretty-printed JSON object.
183+Small changes in the JSON object may result in rather significant
184+changes in how the object is pretty-printed (for example, by
185+addding an indentation level, splitting a single line into
188+A text file wrapped at 80 columns also reacts suboptimally with
189+unadorned Hunt-McIlroy diffs. A single word inserted in a
190+paragraph may push the final word or two of the line onto the
191+next line, and the entire rest of the paragraph may be flagged as
192+a change. Two diffs consisting of a single added word to
193+different sentences may be flagged as a conflict. In general,
194+prose should be diffed by sentence, not by line.
196+As far as the author is aware, clay is the first generalized,
197+type-aware revision control system. We'll go into the workings
198+of this system in some detail.
202+Central to a typed filesystem is the idea of types. In clay, we
203+call these "marks". A mark is a file that defines a type,
204+conversion routines to and from the mark, and diff, patch, and
207+For example, a `%txt` mark may be a list of lines of text, and it
208+may include conversions to `%mime` to allow it to be serialized
209+and sent to a browswer or to the unix filesystem. It will also
210+include Hunt-McIlroy diff, patch, and merge algorithms.
212+A `%json` mark would be defined as a json object in the code, and
213+it would have a parser to convert from `%txt` and a printer to
214+convert back to `%txt`. The diff, patch, and merge algorithms are
215+fairly straightforward for json, though they're very different
218+More formally, a mark is a core with three arms, `++grab`,
219+`++grow`, and `++grad`. In `++grab` is a series of functions to
220+convert from other marks to the given mark. In `++grow` is a
221+series of functions to convert from the given mark to other
222+marks. In `++grad` is `++diff`, `++pact`, `++join`, and `++mash`.
224+The types are as follows, in an informal pseudocode:
228+ ++ mime: <mime> -> <mark-type>
229+ ++ txt: <txt> -> <mark-type>
232+ ++ mime: <mark-type> -> <mime>
233+ ++ txt: <mark-type> -> <txt>
236+ ++ diff: (<mark-type>, <mark-type>) -> <diff-type>
237+ ++ pact: (<mark-type>, <diff-type>) -> <mark-type>
238+ ++ join: (<diff-type>, <diff-type>) -> <diff-type> or NULL
239+ ++ mash: (<diff-type>, <diff-type>) -> <diff-type>
242+These types are basically what you would expect. Not every mark
243+has each of these functions defined -- all of them are optional
246+In general, for a particular mark, the `++grab` and `++grow` entries
247+(if they exist) should be inverses of each other.
249+In `++grad`, `++diff` takes two instances of a mark and produces
250+a diff of them. `++pact` takes an instance of a mark and patches
251+it with the given diff. `++join` takes two diffs and attempts to
252+merge them into a single diff. If there are conflicts, it
253+produces null. `++mash` takes two diffs and forces a merge,
254+annotating any conflicts.
256+In general, if `++diff` called with A and B produces diff D, then
257+`++pact` called with A and D should produce B. Also, if `++join`
258+of two diffs does not produce null, then `++mash` of the same
259+diffs should produce the same result.
261+Alternately, instead of `++diff`, `++pact`, `++join`, and
262+`++mash`, a mark can provide the same functionality by defining
263+`++sted` to be the name of another mark to which we wish to
264+delegate the revision control responsibilities. Then, before
265+running any of those functions, clay will convert to the other
266+mark, and convert back afterward. For example, the `%hoon` mark
267+is revision-controlled in the same way as `%txt`, so its `++grad`
268+is simply `++sted %txt`. Of course, `++txt` must be defined in
269+`++grow` and `++grab` as well.
271+Every file in clay has a mark, and that mark must have a
272+fully-functioning `++grad`. Marks are used for more than just
273+clay, and other marks don't need a `++grad`, but if a piece of
274+data is to be saved to clay, we must know how to revision-control
277+Additionally, if a file is to be synced out to unix, then it must
278+have conversion routines to and from the `%mime` mark.
282+### Reading and Subscribing
284+When reading from Clay, there are three types of requests. A
285+`%sing` request asks for data at single revsion. A `%next`
286+request asks to be notified the next time there's a change to
287+given file. A `%many` request asks to be notified on every
288+change in a desk for a range of changes.
290+For `%sing` and `%next`, there are generally three things to be
291+queried. A `%u` request simply checks for the existence of a
292+file at a path. A `%x` request gets the data in the file at a
293+path. A `%y` request gets a hash of the data in the file at the
294+path combined with all its children and their data. Thus, `%y`
295+of a node changes if it or any of its children change.
297+A `%sing` request is fulfilled immediately if possible. If the
298+requested revision is in the future, or is on another ship for
299+which we don't have the result cached, we don't respond
300+immediately. If the requested revision is in the future, we wait
301+until the revision happens before we respond to the request. If
302+the request is for data on another ship, we pass on the request
303+to the other ship. In general, Clay subscriptions, like most
304+things in Urbit, aren't guaranteed to return immediately.
305+They'll return when they can, and they'll do so in a
306+referentially transparent manner.
308+A `%next` request checks query at the given revision, and it
309+produces the result of the query the next time it changes, along
310+with the revsion number when it changes. Thus, a `%next` of a
311+`%u` is triggered when a file is added or deleted, a `%next` of a
312+`%x` is triggered when a file is added, deleted, or changed, and
313+a `%next` of a `%y` is triggered when a file or any of its
314+children is added, deleted, or changed.
316+A `%many` request is triggered every time the given desk has a
317+new revision. Unlike a `%next`, a `%many` has both a start and
318+an end revsion, after which it stops returning. For `%next`, a
319+single change is reported, and if the caller wishes to hear of
320+the next change, it must resubscribe. For `%many`, every revsion
321+from the start to the end triggers a response. Since a `%many`
322+request doesn't ask for any particular data, there aren't `%u`,
323+`%x`, and `%y` versions for it.
327+One of the primary functions of clay is as a convenient user
328+interface. While tools exist to use clay from within urbit, it's
329+often useful to be able to treat clay like any other filesystem
330+from the Unix perspective -- to "mount" it, as it were.
332+From urbit, you can run `|mount /path/to/directory %mount-point`,
333+and this will mount the given clay directory to the mount-point
334+directory in Unix. Every file is converted to `%mime` before it's
335+written to Unix, and converted back when read from Unix. The
336+entire directory is watched (a la Dropbox), and every change is
337+auto-committed to clay.
341+Merging is a fundamental operation for a distributed revision
342+control system. At their root, clay's merges are similar to
343+git's, but with some additions to accomodate typed data. There
344+are seven different merge strategies.
346+Throughout our discussion, we'll say that the merge is from
347+Alice's desk to Bob's. Recall that a commit is a date (for all
348+new commits this will be the current date), a list of parents,
351+A `%init` merge should be used iff it's the first commit to a
352+desk. The head of Alice's desk is used as the number 1 commit to
353+Bob's desk. Obviously, the ancestry remains intact through
354+traversing the parentage of the commit even though previous
355+commits are not numbered for Bob's desk.
357+A `%this` merge means to keep what's in Bob's desk, but join the
358+ancestry. Thus, the new commit has the head of each desk as
359+parents, but the data is exactly what's in Bob's desk. For those
360+following along in git, this is the 'ours' merge strategy, not
361+the '--ours' option to the 'recursive' merge strategy. In other
362+words, even if Alice makes a change that does not conflict with
363+Bob, we throw it away. It's Bob's way or the highway.
365+A `%that` merge means to take what's in Alice's desk, but join
366+the ancestry. This is the reverse of `%this`.
368+A `%fine` merge is a "fast-forward" merge. This succeeds iff one
369+head is in the ancestry of the other. In this case, we use the
370+descendant as our new head.
372+For `%meet`, `%mate`, and `%meld` merges, we first find the most
373+recent common ancestor to use as our merge base. If we have no
374+common ancestors, then we fail. If we have more than one most
375+recent common ancestor, then we have a criss-cross situation,
376+which should be handled delicately. At present, we delicately
377+throw up our hands and give up, but something akin to git's
378+'recursive' strategy should be implemented in the future.
380+There's a functional inclusion ordering on `%fine`, `%meet`,
381+`%mate`, and `%meld` such that if an earlier strategy would have
382+succeeded, then every later strategy will produce the same
383+result. Put another way, every earlier strategy is the same as
384+every later strategy except with a restricted domain.
386+A `%meet` merge only succeeds if the changes from the merge base
387+to Alice's head (hereafter, "Alice's changes") are in different
388+files than Bob's changes. In this case, the parents are both
389+Alice's and Bob's heads, and the data is the merge base plus
390+Alice's changed files plus Bob's changed files.
392+A `%mate` merge attempts to merge changes to the same file when
393+both Alice and bob change it. If the merge is clean, we use it;
394+otherwise, we fail. A merge between different types of changes --
395+for example, deleting a file vs changing it -- is always a
396+conflict. If we succeed, the parents are both Alice's and Bob's
397+heads, and the data is the merge base plus Alice's changed files
398+plus Bob's changed files plus the merged files.
400+A `%meld` merge will succeed even if there are conflicts. If
401+there are conflicts in a file, then we use the merge base's
402+version of that file, and we produce a set of files with
403+conflicts. The parents are both Alice's and Bob's heads, and the
404+data is the merge base plus Alice's changed files plus Bob's
405+changed files plus the successfully merged files plus the merge
406+base's version of the conflicting files.
408+That's the extent of the merge options in clay proper. In
409+userspace there's a final option `%auto`, which is the most
410+common. `%auto` checks to see if Bob's desk exists, and if it
411+doesn't we use a `%init` merge. Otherwise, we progressively try
412+`%fine`, `%meet`, and `%mate` until one succeeds.
414+If none succeed, we merge Bob's desk into a scratch desk. Then,
415+we merge Alice's desk into the scratch desk with the `%meld`
416+option to force the merge. For each file in the produced set of
417+conflicting files, we call the `++mash` function for the
418+appropriate mark, which annotates the conflicts if we know how.
420+Finally, we display a message to the user informing them of the
421+scratch desk's existence, which files have annotated conflicts,
422+and which files have unannotated conflicts. When the user has
423+resolved the conflicts, they can merge the scratch desk back into
424+Bob's desk. This will be a `%fine` merge since Bob's head is in
425+the ancestry of the scratch desk.
429+Tracking and staying in sync with another desk is another
430+fundamental operation. We call this "autosync". This doesn't mean
431+simply mirroring a desk, since that wouldn't allow local changes.
432+We simply want to apply changes as they are made upstream, as
433+long as there are no conflicts with local changes.
435+This is implemented by watching the other desk, and, when it has
436+changes, merging these changes into our desk with the usual merge
439+Note that it's quite reasonable for two desks to be autosynced to
440+each other. This results in any change on one desk being mirrored
441+to the other and vice versa.
443+Additionally, it's fine to set up an autosync even if one desk,
444+the other desk, or both desks do not exist. The sync will be
445+activated when the upstream desk comes into existence and will
446+create the downstream desk if needed.
450+ ;div(id "commentary"): Commentary
454+The first part of this section will be reference documentation for the data
455+types used by our filesystem. In fact, as a general guide, we recommend
456+reading and attempting to understand the data structures used in any
457+Hoon code before you try to read the code itself. Although complete
458+understanding of the data structures is impossible without seeing them
459+used in the code, an 80% understanding greatly clarifies the code. As
460+another general guide, when reading Hoon, it rarely pays off to
461+understand every line of code when it appears. Try to get the gist of
462+it, and then move on. The next time you come back to it, it'll likely
463+make a lot more sense.
465+After a description of the data models, we'll give an overview of the
466+interface that vanes and applications can use to interact with the
469+Finally, we'll dive into the code and the algorithms themselves. You
474+As you're reading through this section, remember you can always come
475+back to this when you run into these types later on. You're not going to
476+remember everything the first time through, but it is worth reading, or
477+at least skimming, this so that you get a rough idea of how our state is
480+The types that are certainly worth reading are `++raft`, `++room`,
481+`++dome`, `++ankh`, `++rung`, `++rang`, `++blob`, `++yaki`, and `++nori`
482+(possibly in that order). All in all, though, this section isn't too
483+long, so many readers may wish to quickly read through all of it. If you
484+get bored, though, just skip to the next section. You can always come
485+back when you need to.
487+### `++raft`, formal state
490+ ++ raft :: filesystem
491+ $: fat=(map ship room) :: domestic
492+ hoy=(map ship rung) :: foreign
497+This is the state of our vane. Anything that must be remembered between
498+calls to clay is stored in this state.
500+`fat` is the set of domestic servers. This stores all the information
501+that is specfic to a particular ship on this pier. The keys to this map
502+are the ships on the current pier. all the information that is specific
503+to a particular foreign ship. The keys to this map are all the ships
504+whose filesystems we have attempted to access through clay.
506+`ran` is the store of all commits and deltas, keyed by hash. The is
507+where all the "real" data we know is stored; the rest is "just
510+#### `++room`, filesystem per domestic ship
513+ ++ room :: fs per ship
514+ $: hun=duct :: terminal duct
515+ hez=(unit duct) :: sync duch
516+ dos=(map desk dojo) :: native desk
520+This is the representation of the filesystem of a ship on our pier.
522+`hun` is the duct we use to send messages to dill to display
523+notifications of filesystem changes. Only `%note` gifts should be
524+produced along this duct. This is set by the `%init` kiss.
526+`hez`, if present, is the duct we use to send sync messages to unix so
527+that they end up in the pier unix directory. Only `%ergo` gifts should
528+be producd along this duct. This is set by `%into` and `%invo` kisses.
530+`dos` is a well-known operating system released in 1981. It is also the
531+set of desks on this ship, mapped to their data.
533+#### `++desk`, filesystem branch
536+ ++ desk ,@tas :: ship desk case spur
539+This is the name of a branch of the filesystem. The default desks are
540+"arvo", "main", and "try". More may be created by simply referencing
541+them. Desks have independent histories and states, and they may be
542+merged into each other.
544+### `++dojo`, domestic desk state
547+ ++ dojo ,[p=cult q=dome] :: domestic desk state
550+This is the all the data that is specific to a particular desk on a
551+domestic ship. `p` is the set of subscribers to this desk and `q` is the
554+#### `++cult`, subscriptions
557+ ++ cult (map duct rave) :: subscriptions
560+This is the set of subscriptions to a particular desk. The keys are the
561+ducts from where the subscriptions requests came. The results will be
562+produced along these ducts. The values are a description of the
563+requested information.
565+#### `++rave`, general subscription request
568+ ++ rave :: general request
569+ $% [& p=mood] :: single request
570+ [| p=moat] :: change range
574+This represents a subscription request for a desk. The request can be
575+for either a single item in the desk or else for a range of changes on
578+### `++rove`, stored general subscription request
581+ ++ rove (each mood moot) :: stored request
584+When we store a request, we store subscriptions with a little extra
585+information so that we can determine whether new versions actually
586+affect the path we're subscribed to.
588+#### `++mood`, single subscription request
591+ ++ mood ,[p=care q=case r=path] :: request in desk
594+This represents a request for the state of the desk at a particular
595+commit, specfied by `q`. `p` specifies what kind of information is
596+desired, and `r` specifies the path we are requesting.
598+#### `++moat`, range subscription request
601+ ++ moat ,[p=case q=case r=path] :: change range
604+This represents a request for all changes between `p` and `q` on path
605+`r`. You will be notified when a change is made to the node referenced
606+by the path or to any of its children.
608+### `++moot`, stored range subscription request
611+ ++ moot ,[p=case q=case r=path s=(map path lobe)] ::
614+This is just a `++moat` plus a map of paths to lobes. This map
615+represents the data at the node referenced by the path at case `p`, if
616+we've gotten to that case (else null). We only send a notification along
617+the subscription if the data at a new revision is different than it was.
619+### ` `++care`, clay submode
622+ ++ care ?(%u %v %w %x %y %z) :: clay submode
625+This specifies what type of information is requested in a subscription
628+`%u` requests the `++rang` at the current moment. Because this
629+information is not stored for any moment other than the present, we
630+crash if the `++case` is not a `%da` for now.
632+`%v` requests the `++dome` at the specified commit.
634+`%w` requests the revsion number of the desk.
636+`%x` requests the file at a specified path at the specified commit. If
637+there is no node at that path or if the node has no contents (that is,
638+if `q:ankh` is null), then this produces null.
640+`%y` requests a `++arch` of the specfied commit at the specified path.
642+`%z` requests the `++ankh` of the specified commit at the specfied path.
644+### `++arch`, shallow filesystem node
647+ ++ arch ,[p=@uvI q=(unit ,@uvI) r=(map ,@ta ,~)] :: fundamental node
650+This is analogous to `++ankh` except that the we have neither our
651+contents nor the ankhs of our children. The other fields are exactly the
652+same, so `p` is a hash of the associated ankh, `u.q`, if it exists, is a
653+hash of the contents of this node, and the keys of `r` are the names of
654+our children. `r` is a map to null rather than a set so that the
655+ordering of the map will be equivalent to that of `r:ankh`, allowing
656+efficient conversion.
658+#### `++case`, specifying a commit
661+ ++ case :: ship desk case spur
662+ $% [%da p=@da] :: date
663+ [%tas p=@tas] :: label
664+ [%ud p=@ud] :: number
668+A commit can be referred to in three ways: `%da` refers to the commit
669+that was at the head on date `p`, `%tas` refers to the commit labeled
670+`p`, and `%ud` refers to the commit numbered `p`. Note that since these
671+all can be reduced down to a `%ud`, only numbered commits may be
672+referenced with a `++case`.
674+#### `++dome`, desk data
677+ ++ dome :: project state
678+ $: ang=agon :: pedigree
681+ hit=(map ,@ud tako) :: changes by id
682+ lab=(map ,@tas ,@ud) :: labels
686+This is the data that is actually stored in a desk.
688+`ang` is unused and should be removed.
690+`ank` is the current state of the desk. Thus, it is the state of the
691+filesystem at revison `let`. The head of a desk is always a numbered
694+`let` is the number of the most recently numbered commit. This is also
695+the total number of numbered commits.
697+`hit` is a map of numerical ids to hashes of commits. These hashes are
698+mapped into their associated commits in `hut:rang`. In general, the keys
699+of this map are exactly the numbers from 1 to `let`, with no gaps. Of
700+course, when there are no numbered commits, `let` is 0, so `hit` is
701+null. Additionally, each of the commits is an ancestor of every commit
702+numbered greater than this one. Thus, each is a descendant of every
703+commit numbered less than this one. Since it is true that the date in
704+each commit (`t:yaki`) is no earlier than that of each of its parents,
705+the numbered commits are totally ordered in the same way by both
706+pedigree and date. Of course, not every commit is numbered. If that
707+sounds too complicated to you, don't worry about it. It basically
708+behaves exactly as you would expect.
710+`lab` is a map of textual labels to numbered commits. Note that labels
711+can only be applied to numbered commits. Labels must be unique across a
714+#### `++ankh`, filesystem node
717+ ++ ankh :: fs node (new)
718+ $: p=cash :: recursive hash
719+ q=(unit ,[p=cash q=*]) :: file
720+ r=(map ,@ta ankh) :: folders
724+This is a single node in the filesystem. This may be file or a directory
725+or both. In earth filesystems, a node is a file xor a directory. On
726+mars, we're inclusive, so a node is a file ior a directory.
728+`p` is a recursive hash that depends on the contents of the this file or
729+directory and on any children.
731+`q` is the contents of this file, if any. `p.q` is a hash of the
732+contents while `q.q` is the data itself.
734+`r` is the set of children of this node. In the case of a pure file,
735+this is empty. The keys are the names of the children and the values
736+are, recursively, the nodes themselves.
738+#### `++cash`, ankh hash
741+ ++ cash ,@uvH :: ankh hash
744+This is a 128-bit hash of an ankh. These are mostly stored within ankhs
745+themselves, and they are used to check for changes in possibly-deep
748+#### `++rung`, filesystem per neighbor ship
751+ ++ rung $: rus=(map desk rede) :: neighbor desks
755+This is the filesystem of a neighbor ship. The keys to this map are all
756+the desks we know about on their ship.
758+#### `++rede`, desk state
761+ ++ rede :: universal project
762+ $: lim=@da :: complete to
763+ qyx=cult :: subscribers
764+ ref=(unit rind) :: outgoing requests
765+ dom=dome :: revision state
769+This is our knowledge of the state of a desk, either foreign or
772+`lim` is the date of the last full update. We only respond to requests
773+for stuff before this time.
775+`qyx` is the list of subscribers to this desk. For domestic desks, this
776+is simply `p:dojo`, all subscribers to the desk, while in foreign desks
777+this is all the subscribers from our ship to the foreign desk.
779+`ref` is the request manager for the desk. For domestic desks, this is
780+null since we handle requests ourselves.
782+`dom` is the actual data in the desk.
784+#### `++rind`, request manager
787+ ++ rind :: request manager
788+ $: nix=@ud :: request index
789+ bom=(map ,@ud ,[p=duct q=rave]) :: outstanding
790+ fod=(map duct ,@ud) :: current requests
791+ haw=(map mood (unit)) :: simple cache
795+This is the request manager for a foreign desk.
797+`nix` is one more than the index of the most recent request. Thus, it is
798+the next available request number.
800+`bom` is the set of outstanding requests. The keys of this map are some
801+subset of the numbers between 0 and one less than `nix`. The members of
802+the map are exactly those requests that have not yet been fully
805+`fod` is the same set as `bom`, but from a different perspective. In
806+particular, the values of `fod` are the same as the values of `bom`, and
807+the `p` out of the values of `bom` are the same as the keys of `fod`.
808+Thus, we can map ducts to their associated request number and `++rave`,
809+and we can map numbers to their associated duct and `++rave`.
811+`haw` is a map from simple requests to their values. This acts as a
812+cache for requests that have already been made. Thus, the second request
813+for a particular `++mood` is nearly instantaneous.
815+#### `++rang`, data store
818+ ++ rang $: hut=(map tako yaki) ::
819+ lat=(map lobe blob) ::
823+This is a set of data keyed by hash. Thus, this is where the "real" data
824+is stored, but it is only meaningful if we know the hash of what we're
827+`hut` is a map from hashes to commits. We often get the hashes from
828+`hit:dome`, which keys them by logical id. Not every commit has an id.
830+`lat` is a map from hashes to the actual data. We often get the hashes
831+from a `++yaki`, a commit, which references this map to get the data.
832+There is no `++blob` in any `++yaki`. They are only accessible through
835+#### `++tako`, commit reference
838+ ++ tako ,@ :: yaki ref
841+This is a hash of a `++yaki`, a commit. These are most notably used as
842+the keys in `hut:rang`, where they are associated with the actual
843+`++yaki`, and as the values in `hit:dome`, where sequential ids are
844+associated with these.
846+#### `++yaki`, commit
849+ ++ yaki ,[p=(list tako) q=(map path lobe) r=tako t=@da] :: commit
852+This is a single commit.
854+`p` is a list of the hashes of the parents of this commit. In most
855+cases, this will be a single commit, but in a merge there may be more
856+parents. In theory, there may be an arbitrary number of parents, but in
857+practice merges have exactly two parents. This may change in the future.
858+For commit 1, there is no parent.
860+`q` is a map of the paths on a desk to the data at that location. If you
861+understand what a `++lobe` and a `++blob` is, then the type signature
862+here tells the whole story.
864+`r` is the hash associated with this commit.
866+`t` is the date at which this commit was made.
868+#### `++lobe`, data reference
871+ ++ lobe ,@ :: blob ref
874+This is a hash of a `++blob`. These are most notably used in `lat:rang`,
875+where they are associated with the actual `++blob`, and as the values in
876+`q:yaki`, where paths are associated with their data in a commit.
881+ ++ blob $% [%delta p=lobe q=lobe r=udon] :: delta on q
882+ [%direct p=lobe q=* r=umph] ::
883+ [%indirect p=lobe q=* r=udon s=lobe] ::
887+This is a node of data. In every case, `p` is the hash of the blob.
889+`%delta` is the case where we define the data by a delta on other data.
890+In practice, the other data is always the previous commit, but nothing
891+depends on this. `q` is the hash of the parent blob, and `r` is the
894+`%direct` is the case where we simply have the data directly. `q` is the
895+data itself, and `r` is any preprocessing instructions. These almost
896+always come from the creation of a file.
898+`%indirect` is both of the preceding cases at once. `q` is the direct
899+data, `r` is the delta, and `s` is the parent blob. It should always be
900+the case that applying `r` to `s` gives the same data as `q` directly
901+(with the prepreprocessor instructions in `p.r`). This exists purely for
902+performance reasons. This is unused, at the moment, but in general these
903+should be created when there are a long line of changes so that we do
904+not have to traverse the delta chain back to the creation of the file.
906+#### `++udon`, abstract delta
909+ ++ udon :: abstract delta
910+ $: p=umph :: preprocessor
912+ $% [%a p=* q=*] :: trivial replace
913+ [%b p=udal] :: atomic indel
914+ [%c p=(urge)] :: list indel
915+ [%d p=upas q=upas] :: tree edit
920+This is an abstract change to a file. This is a superset of what would
921+normally be called diffs. Diffs usually refer to changes in lines of
922+text while we have the ability to do more interesting deltas on
923+arbitrary data structures.
925+`p` is any preprocessor instructions.
927+`%a` refers to the trival delta of a complete replace of old data with
930+`%b` refers to changes in an opaque atom on the block level. This has
931+very limited usefulness, and is not used at the moment.
933+`%c` refers to changes in a list of data. This is often lines of text,
934+which is your classic diff. We, however, will work on any list of data.
936+`%d` refers to changes in a tree of data. This is general enough to
937+describe changes to any hoon noun, but often more special-purpose delta
938+should be created for different content types. This is not used at the
939+moment, and may in fact be unimplemented.
941+#### `++urge`, list change
944+ ++ urge |*(a=_,* (list (unce a))) :: list change
947+This is a parametrized type for list changes. For example, `(urge ,@t)`
948+is a list change for lines of text.
950+#### `++unce`, change part of a list.
953+ ++ unce |* a=_,* :: change part
954+ $% [%& p=@ud] :: skip[copy]
955+ [%| p=(list a) q=(list a)] :: p -> q[chunk]
959+This is a single change in a list of elements of type `a`. For example,
960+`(unce ,@t)` is a single change in a lines of text.
962+`%&` means the next `p` lines are unchanged.
964+`%|` means the lines `p` have changed to `q`.
966+#### `++umph`, preprocessing information
969+ ++ umph :: change filter
970+ $| $? %a :: no filter
974+ $% [%d p=@ud] :: blocklist
978+This space intentionally left undocumented. This stuff will change once
979+we get a well-typed clay.
981+#### `++upas`, tree change
984+ ++ upas :: tree change (%d)
985+ $& [p=upas q=upas] :: cell
986+ $% [%0 p=axis] :: copy old
987+ [%1 p=*] :: insert new
988+ [%2 p=axis q=udon] :: mutate!
992+This space intentionally left undocumented. This stuff is not known to
993+work, and will likely change when we get a well-typed clay. Also, this
994+is not a complicated type; it is not difficult to work out the meaning.
996+#### `++nori`, repository action
999+ ++ nori :: repository action
1000+ $% [& q=soba] :: delta
1001+ [| p=@tas] :: label
1005+This describes a change that we are asking clay to make to the desk.
1006+There are two kinds of changes that may be made: we can modify files or
1007+we can apply a label to a commit.
1009+In the `|` case, we will simply label the current commit with the given
1010+label. In the `&` case, we will apply the given changes.
1012+#### `++soba`, delta
1015+ ++ soba ,[p=cart q=(list ,[p=path q=miso])] :: delta
1018+This describes a set of changes to make to a desk. The `cart` is simply
1019+a pair of the old hash and the new hash of the desk. The list is a list
1020+of changes keyed by the file they're changing. Thus, the paths are paths
1021+to files to be changed while `miso` is a description of the change
1024+#### `++miso`, ankh delta
1027+ ++ miso :: ankh delta
1028+ $% [%del p=*] :: delete
1029+ [%ins p=*] :: insert
1030+ [%mut p=udon] :: mutate
1034+There are three kinds of changes that may be made to a node in a desk.
1035+We can insert a file, in which case `p` is the contents of the new file.
1036+We can delete a file, in which case `p` is the contents of the old file.
1037+Finally, we can mutate that file, in which case the `udon` describes the
1038+changes we are applying to the file.
1040+#### `++mizu`, merged state
1043+ ++ mizu ,[p=@u q=(map ,@ud tako) r=rang] :: new state
1046+This is the input to the `%merg` kiss, which allows us to perform a
1047+merge. The `p` is the number of the new head commit. The `q` is a map
1048+from numbers to commit hashes. This is all the new numbered commits that
1049+are to be inserted. The keys to this should always be the numbers from
1050+`let.dom` plus one to `p`, inclusive. The `r` is the maps of all the new
1051+commits and data. Since these are merged into the current state, no old
1052+commits or data need be here.
1054+#### `++riff`, request/desist
1057+ ++ riff ,[p=desk q=(unit rave)] :: request/desist
1060+This represents a request for data about a particular desk. If `q`
1061+contains a `rave`, then this opens a subscription to the desk for that
1062+data. If `q` is null, then this tells clay to cancel the subscription
1065+#### `++riot`, response
1068+ ++ riot (unit rant) :: response/complete
1071+A riot is a response to a subscription. If null, the subscription has
1072+been completed, and no more responses will be sent. Otherwise, the
1073+`rant` is the produced data.
1075+#### `++rant`, response data
1078+ ++ rant :: namespace binding
1079+ $: p=[p=care q=case r=@tas] :: clade release book
1085+This is the data at a particular node in the filesystem. `p.p` specifies
1086+the type of data that was requested (and is produced). `q.p` gives the
1087+specific version reported (since a range of versions may be requested in
1088+a subscription). `r.p` is the desk. `q` is the path to the filesystem
1089+node. `r` is the data itself (in the format specified by `p.p`).
1091+### `++nako`, subscription response data
1094+ ++ nako $: gar=(map ,@ud tako) :: new ids
1096+ lar=(set yaki) :: new commits
1097+ bar=(set blob) :: new content
1101+This is the data that is produced by a request for a range of revisions
1102+of a desk. This allows us to easily keep track of a remote repository --
1103+all the new information we need is contained in the `nako`.
1105+`gar` is a map of the revisions in the range to the hash of the commit
1106+at that revision. These hashes can be used with `hut:rang` to find the
1109+`let` is either the last revision number in the range or the most recent
1110+revision number, whichever is smaller.
1112+`lar` is the set of new commits, and `bar` is the set of new content.
1116+As with all vanes, there are exactly two ways to interact with clay.
1117+`%clay` exports a namespace accessible through `.^`, which is described
1118+above under `++care`. The primary way of interacting with clay, though,
1119+is by sending kisses and receiving gifts.
1122+ ++ gift :: out result <-$
1123+ $% [%ergo p=@p q=@tas r=@ud] :: version update
1124+ [%note p=@tD q=tank] :: debug message
1125+ [%writ p=riot] :: response
1127+ ++ kiss :: in request ->$
1128+ $% [%info p=@p q=@tas r=nori] :: internal edit
1129+ [%ingo p=@p q=@tas r=nori] :: internal noun edit
1130+ [%init p=@p] :: report install
1131+ [%into p=@p q=@tas r=nori] :: external edit
1132+ [%invo p=@p q=@tas r=nori] :: external noun edit
1133+ [%merg p=@p q=@tas r=mizu] :: internal change
1134+ [%wart p=sock q=@tas r=path s=*] :: network request
1135+ [%warp p=sock q=riff] :: file request
1139+There are only a small number of possible kisses, so it behooves us to
1140+describe each in detail.
1143+ $% [%info p=@p q=@tas r=nori] :: internal edit
1145+ [%into p=@p q=@tas r=nori] :: external edit
1148+These two kisses are nearly identical. At a high level, they apply
1149+changes to the filesystem. Whenever we add, remove, or edit a file, one
1150+of these cards is sent. The `p` is the ship whose filesystem we're
1151+trying to change, the `q` is the desk we're changing, and the `r` is the
1152+request change. For the format of the requested change, see the
1153+documentation for `++nori` above.
1155+When a file is changed in the unix filesystem, vere will send a `%into`
1156+kiss. This tells clay that the duct over which the kiss was sent is the
1157+duct that unix is listening on for changes. From within Arvo, though, we
1158+should never send a `%into` kiss. The `%info` kiss is exactly identical
1159+except it does not reset the duct.
1162+ [%ingo p=@p q=@tas r=nori] :: internal noun edit
1164+ [%invo p=@p q=@tas r=nori] :: external noun edit
1167+These kisses are currently identical to `%info` and `%into`, though this
1168+will not always be the case. The intent is for these kisses to allow
1169+typed changes to clay so that we may store typed data. This is currently
1173+ [%init p=@p] :: report install
1176+Init is called when a ship is started on our pier. This simply creates a
1177+default `room` to go into our `raft`. Essentially, this initializes the
1178+filesystem for a ship.
1181+ [%merg p=@p q=@tas r=mizu] :: internal change
1184+This is called to perform a merge. This is most visibly called by
1185+:update to update the filesystem of the current ship to that of its
1186+sein. The `p` and `q` are as in `%info`, and the `r` is the description
1187+of the merge. See `++mizu` above.
1190+`XX [%wake ~] :: timer activate XX`
1192+XX This card is sent by unix at the time specified by `++doze`. This
1193+time is XX usually the closest time specified in a subscription request.
1194+When `%wake` is XX called, we update our subscribers if there have been
1198+ [%wart p=sock q=@tas r=path s=*] :: network request
1201+This is a request that has come across the network for a particular
1202+file. When another ship asks for a file from us, that request comes to
1203+us in the form of a `%wart` kiss. This is handled by trivially turning
1207+ [%warp p=sock q=riff] :: file request
1210+This is a request for information about a particular desk. This is, in
1211+its most general form, a subscription, though in many cases it is the
1212+trivial case of a subscription -- a read. See `++riff` for the format of
1215+## Lifecycle of a Local Read
1217+There are two real types of interaction with a filesystem: you can read,
1218+and you can write. We'll describe each process, detailing both the flow
1219+of control followed by the kernel and the algorithms involved. The
1220+simpler case is that of the read, so we'll begin with that.
1222+When a vane or an application wishes to read a file from the filesystem,
1223+it sends a `%warp` kiss, as described above. Of course, you may request
1224+a file on another ship and, being a global filesystem, clay will happily
1225+produce it for you. That code pathway will be described in another
1226+section; here, we will restrict ourselves to examining the case of a
1227+read from a ship on our own pier.
1229+The kiss can request either a single version of a file node or a range
1230+of versions of a desk. Here, we'll deal only with a request for a single
1233+As in all vanes, a kiss enters clay via a call to `++call`. Scanning
1234+through the arm, we quickly see where `%warp` is handled.
1237+ ?: =(p.p.q.hic q.p.q.hic)
1238+ =+ une=(un p.p.q.hic now ruf)
1239+ =+ wex=(di:une p.q.q.hic)
1243+ (eave:wex hen u.q.q.q.hic)
1246+ [-.woo abet:(pish:une p.q.q.hic +.woo ran.wao)]
1249+We're following the familar patern of producing a list of moves and an
1250+updated state. In this case, the state is `++raft`.
1252+We first check to see if the sending and receiving ships are the same.
1253+If they're not, then this is a request for data on another ship. We
1254+describe that process later. Here, we discuss only the case of a local
1257+At a high level, the call to `++un` sets up the core for the domestic
1258+ship that contains the files we're looking for. The call to `++di` sets
1259+up the core for the particular desk we're referring to.
1261+After this, we perform the actual request. If there is no rave in the
1262+riff, then that means we are cancelling a request, so we call
1263+`++ease:de`. Otherwise, we start a subscription with `++eave:de`. We
1264+call `++abet:de` to resolve our various types of output into actual
1265+moves. We produce the moves we found above and the `++un` core resolved
1266+with `++pish:un` (putting the modified desk in the room) and `++abet:un`
1267+(putting the modified room in the raft).
1269+Much of this is fairly straightforward, so we'll only describe `++ease`,
1270+`++eave`, and `++abet:de`. Feel free to look up the code to the other
1271+steps -- it should be easy to follow.
1273+Although it's called last, it's usually worth examining `++abet` first,
1274+since it defines in what ways we can cause side effects. Let's do that,
1275+and also a few of the lines at the beginning of `++de`.
1278+ =| yel=(list ,[p=duct q=gift])
1279+ =| byn=(list ,[p=duct q=riot])
1280+ =| vag=(list ,[p=duct q=gift])
1281+ =| say=(list ,[p=duct q=path r=ship s=[p=@ud q=riff]])
1282+ =| tag=(list ,[p=duct q=path c=note])
1285+ ^- [(list move) rede]
1289+ |=([a=duct b=gift] [hun %give b])
1292+ |=([a=duct b=riot] [a %give [%writ b]])
1295+ |=([a=duct b=gift] [a %give b])
1298+ |= [a=duct b=path c=ship d=[p=@ud q=riff]]
1300+ [%pass b %a %want [who c] [%q %re p.q.d (scot %ud p.d) ~] q.d]
1303+ |=([a=duct b=path c=note] [a %pass b c])
1307+This is very simple code. We see there are exactly five different kinds
1308+of side effects we can generate.
1310+In `yel` we put gifts that we wish to be sent along the `hun:room` duct
1311+to dill. See the documentation for `++room` above. This is how we
1312+display messages to the terminal.
1314+In `byn` we put riots that we wish returned to subscribers. Recall that
1315+a riot is a response to a subscription. These are returned to our
1316+subscribers in the form of a `%writ` gift.
1318+In `vag` we put gifts along with the ducts on which to send them. This
1319+allows us to produce arbitrary gifts, but in practice this is only used
1320+to produce `%ergo` gifts.
1322+In `say` we put messages we wish to pass to ames. These messages are
1323+used to request information from clay on other piers. We must provide
1324+not only the duct and the request (the riff), but also the return path,
1325+the other ship to talk to, and the sequence number of the request.
1327+In `tag` we put arbitrary notes we wish to pass to other vanes. For now,
1328+the only notes we pass here are `%wait` and `%rest` to the timer vane.
1330+Now that we know what kinds of side effects we may have, we can jump
1331+into the handling of requests.
1334+ ++ ease :: release request
1338+ =+ rov=(~(got by qyx) hen)
1339+ =. qyx (~(del by qyx) hen)
1340+ (mabe rov (cury best hen))
1341+ =. qyx (~(del by qyx) hen)
1343+ =+ nux=(~(get by fod.u.ref) hen)
1346+ say [[hen [(scot %ud u.nux) ~] for [u.nux syd ~]] say]
1347+ fod.u.ref (~(del by fod.u.ref) hen)
1348+ bom.u.ref (~(del by bom.u.ref) u.nux)
1352+This is called when we're cancelling a subscription. For domestic desks,
1353+`ref` is null, so we're going to cancel any timer we might have created.
1354+We first delete the duct from our map of requests, and then we call
1355+`++mabe` with `++best` to send a `%rest` kiss to the timer vane if we
1356+have started a timer. We'll describe `++best` and `++mabe` momentarily.
1358+Although we said we're not going to talk about foreign requests yet,
1359+it's easy to see that for foreign desks, we cancel any outstanding
1360+requests for this duct and send a message over ames to the other ship
1361+telling them to cancel the subscription.
1365+ |= [hen=duct tym=@da]
1366+ %_(+> tag :_(tag [hen /tyme %t %rest tym]))
1369+This simply pushes a `%rest` note onto `tag`, from where it will be
1370+passed back to arvo to be handled. This cancels the timer at the given
1371+duct (with the given time).
1374+ ++ mabe :: maybe fire function
1375+ |* [rov=rove fun=$+(@da _+>.^$)]
1382+ ?. ?=(%da -.q.p.rov) ~
1387+ ?. ?=(%da -.p.mot) ~
1388+ ?.((lth now p.p.mot) ~ [~ p.p.mot])
1389+ ?. ?=(%da -.q.mot) ~
1390+ ?.((lth now p.q.mot) [~ now] [~ p.q.mot])
1394+This decides whether the given request can only be satsified in the
1395+future. In that case, we call the given function with the time in the
1396+future when we expect to have an update to give to this request. This is
1397+called with `++best` to cancel timers and with `++bait` to start them.
1399+For single requests, we have a time if the request is for a particular
1400+time (which is assumed to be in the future). For ranges of requests, we
1401+check both the start and end cases to see if they are time cases. If so,
1402+we choose the earlier time.
1404+If any of those give us a time, then we call the given funciton with the
1407+The more interesting case is, of course, when we're not cancelling a
1408+subscription but starting one.
1411+ ++ eave :: subscribe
1412+ |= [hen=duct rav=rave]
1416+ ?: &(=(p.p.rav %u) !=(p.q.p.rav now))
1417+ ~& [%clay-fail p.q.p.rav %now now]
1419+ =+ ver=(aver p.rav)
1424+ (blab hen p.rav u.u.ver)
1427+There are two types of subscriptions -- either we're requesting a single
1428+file or we're requesting a range of versions of a desk. We'll dicuss the
1431+First, we check that we're not requesting the `rang` from any time other
1432+than the present. Since we don't store that information for any other
1433+time, we can't produce it in a referentially transparent manner for any
1434+time other than the present.
1436+Then, we try to read the requested `mood` `p.rav`. If we can't access
1437+the request data right now, we call `++duce` to put the request in our
1438+queue to be satisfied when the information becomes available.
1440+This case occurs when we make a request for a case whose (1) date is
1441+after the current date, (2) number is after the current number, or (3)
1442+label is not yet used.
1445+ ++ duce :: produce request
1446+ |= [hen=duct rov=rove]
1448+ =. qyx (~(put by qyx) hen rov)
1450+ (mabe rov (cury bait hen))
1451+ |- ^+ +>+.$ :: XX why?
1454+ ?. ?=([%& %v *] rav) rav
1455+ [%| [%ud let.dom] `case`q.p.rav r.p.rav]
1458+ say [[hen [(scot %ud inx) ~] for [inx syd ~ vaw]] say]
1459+ nix.u.ref +(nix.u.ref)
1460+ bom.u.ref (~(put by bom.u.ref) inx [hen vaw])
1461+ fod.u.ref (~(put by fod.u.ref) hen inx)
1465+The code for `++duce` is nearly the exact inverse of `++ease`, which in
1466+the case of a domestic desk is very simple -- we simply put the duct and
1467+rave into `qyx` and possibly start a timer with `++mabe` and `++bait`.
1468+Recall that `ref` is null for domestic desks and that `++mabe` fires the
1469+given function with the time we need to be woken up at, if we need to be
1470+woken up at a particular time.
1474+ |= [hen=duct tym=@da]
1475+ %_(+> tag :_(tag [hen /tyme %t %wait tym]))
1478+This sets an alarm by sending a `%wait` card with the given time to the
1481+Back in `++eave`, if `++aver` returned `[~ ~]`, then we cancel the
1482+subscription. This occurs when we make (1) a `%x` request for a file
1483+that does not exist, (2) a `%w` request with a case that is not a
1484+number, or (3) a `%w` request with a nonempty path. The `++blub` is
1485+exactly what you would expect it to be.
1488+ ++ blub :: ship stop
1490+ %_(+> byn [[hen ~] byn])
1493+We notify the duct that we're cancelling their subscription since it
1496+Otherwise, we have received the desired information, so we send it on to
1497+the subscriber with `++blab`.
1500+ ++ blab :: ship result
1501+ |= [hen=duct mun=mood dat=*]
1503+ +>(byn [[hen ~ [p.mun q.mun syd] r.mun dat] byn])
1506+The most interesting arm called in `++eave` is, of course, `++aver`,
1507+where we actually try to read the data.
1512+ ^- (unit (unit ,*))
1513+ ?: &(=(p.mun %u) !=(p.q.mun now)) :: prevent bad things
1514+ ~& [%clay-fail p.q.mun %now now]
1516+ =+ ezy=?~(ref ~ (~(get by haw.u.ref) mun))
1518+ =+ nao=(~(case-to-aeon ze lim dom ran) q.mun)
1519+ ?~(nao ~ [~ (~(read-at-aeon ze lim dom ran) u.nao mun)])
1522+We check immediately that we're not requesting the `rang` for any time
1523+other than the present.
1525+If this is a foreign desk, then we check our cache for the specific
1526+request. If either this is a domestic desk or we don't have the request
1527+in our cache, then we have to actually go read the data from our dome.
1529+We need to do two things. First, we try to find the number of the commit
1530+specified by the given case, and then we try to get the data there.
1532+Here, we jump into `arvo/zuse.hoon`, which is where much of the
1533+algorithmic code is stored, as opposed to the clay interface, which is
1534+stored in `arvo/clay.hoon`. We examine `++case-to-aeon:ze`.
1537+ ++ case-to-aeon :: case-to-aeon:ze
1538+ |= lok=case :: act count through
1542+ ?: (gth p.lok lim) ~
1544+ ?: =(0 let) [~ 0] :: avoid underflow
1553+ %tas (~(get by lab) p.lok)
1554+ %ud ?:((gth p.lok let) ~ [~ p.lok])
1558+We handle each type of `case` differently. The latter two types are
1561+If we're requesting a revision by label, then we simply look up the
1562+requested label in `lab` from the given dome. If it exists, that is our
1563+aeon; else we produce null, indicating the requested revision does not
1566+If we're requesting a revision by number, we check if we've yet reached
1567+that number. If so, we produce the number; else we produce null.
1569+If we're requesting a revision by date, we check first if the date is in
1570+the future, returning null if so. Else we start from the most recent
1571+revision and scan backwards until we find the first revision committed
1572+before that date, and we produce that. If we requested a date before any
1573+revisions were committed, we produce `0`.
1575+The definitions of `++aeon-to-tako` and `++tako-to-yaki` are trivial.
1578+ ++ aeon-to-tako ~(got by hit)
1580+ ++ tako-to-yaki ~(got by hut) :: grab yaki
1583+We simply look up the aeon or tako in their respective maps (`hit` and
1586+Assuming we got a valid version number, `++aver` calls
1587+`++read-at-aeon:ze`, which reads the requested data at the given
1591+ ++ read-at-aeon :: read-at-aeon:ze
1592+ |= [oan=aeon mun=mood] :: seek and read
1594+ ?: &(?=(%w p.mun) !?=(%ud -.q.mun)) :: NB only for speed
1595+ ?^(r.mun ~ [~ oan])
1596+ (read:(rewind oan) mun)
1599+If we're requesting the revision number with a case other than by
1600+number, then we go ahead and just produce the number we were given.
1601+Otherwise, we call `++rewind` to rewind our state to the given revision,
1602+and then we call `++read` to get the requested information.
1605+ ++ rewind :: rewind:ze
1606+ |= oan=aeon :: rewind to aeon
1609+ ?: (gth oan let) !! :: don't have version
1610+ +>(ank (checkout-ankh q:(tako-to-yaki (aeon-to-tako oan))), let oan)
1613+If we're already at the requested version, we do nothing. If we're
1614+requesting a version later than our head, we are unable to comply.
1616+Otherwise, we get the hash of the commit at the number, and from that we
1617+get the commit itself (the yaki), which has the map of path to lobe that
1618+represents a version of the filesystem. We call `++checkout-ankh` to
1619+checkout the commit, and we replace `ank` in our context with the
1623+ ++ checkout-ankh :: checkout-ankh:ze
1624+ |= hat=(map path lobe) :: checkout commit
1627+ %+ roll (~(tap by hat) ~)
1628+ |= [[pat=path bar=lobe] ank=ankh]
1632+ =+ zar=(lobe-to-noun bar)
1633+ ank(q [~ (sham zar) zar])
1634+ =+ nak=(~(get by r.ank) i.pat)
1636+ r %+ ~(put by r.ank) i.pat
1637+ $(pat t.pat, ank (fall nak _ankh))
1641+Twice we call `++cosh`, which hashes a commit, updating `p` in an
1642+`ankh`. Let's jump into that algorithm before we describe
1646+ ++ cosh :: locally rehash
1647+ |= ank=ankh :: NB v/unix.c
1648+ ank(p rehash:(zu ank))
1651+We simply replace `p` in the hash with the `cash` we get from a call to
1655+ ++ zu !: :: filesystem
1656+ |= ank=ankh :: filesystem state
1657+ =| myz=(list ,[p=path q=miso]) :: changes in reverse
1658+ =| ram=path :: reverse path into
1660+ ++ rehash :: local rehash
1662+ %+ mix ?~(q.ank 0 p.u.q.ank)
1667+ (shaf %dash (mix axe (shaf %dush (mix p.n.r.ank p.q.n.r.ank))))
1668+ $(r.ank l.r.ank, axe (peg axe 2))
1669+ $(r.ank r.r.ank, axe (peg axe 3))
1673+`++zu` is a core we set up with a particular filesystem node to traverse
1674+a checkout of the filesystem and access the actual data inside it. One
1675+of the things we can do with it is to create a recursive hash of the
1678+In `++rehash`, if this node is a file, then we xor the remainder of the
1679+hash with the hash of the contents of the file. The remainder of the
1680+hash is `0` if we have no children, else we descend into our children.
1681+Basically, we do a half SHA-256 of the xor of the axis of this child and
1682+the half SHA-256 of the xor of the name of the child and the hash of the
1683+child. This is done for each child and all the results are xored
1686+Now we return to our discussion of `++checkout-ankh`.
1688+We fold over every path in this version of the filesystem and create a
1689+great ankh out of them. First, we call `++lobe-to-noun` to get the raw
1690+data referred to be each lobe.
1693+ ++ lobe-to-noun :: grab blob
1699+This converts a lobe into the raw data it refers to by first getting the
1700+blob with `++lobe-to-blob` and converting that into data with
1704+ ++ lobe-to-blob ~(got by lat) :: grab blob
1707+This just grabs the blob that the lobe refers to.
1710+ ++ blob-to-noun :: grab blob
1713+ %delta (lump r.p (lobe-to-noun q.p))
1719+If we have either a direct or an indirect blob, then the data is stored
1720+right in the blob. Otherwise, we have to reconstruct it from the diffs.
1721+We do this by calling `++lump` on the diff in the blob with the data
1722+obtained by recursively calling the parent of this blob.
1725+ ++ lump :: apply patch
1726+ |= [don=udon src=*]
1728+ ?+ p.don ~|(%unsupported !!)
1730+ ?+ -.q.don ~|(%unsupported !!)
1732+ %c (lurk ((hard (list)) src) p.q.don)
1733+ %d (lure src p.q.don)
1737+ =+ dst=(lore ((hard ,@) src))
1739+ ?+ -.q.don ~|(%unsupported !!)
1740+ %a ((hard (list ,@t)) q.q.don)
1741+ %c (lurk dst p.q.don)
1746+This is defined in `arvo/hoon.hoon` for historical reasons which are
1747+likely no longer applicable. Since the `++umph` structure will likely
1748+change we convert clay to be a typed filesystem, we'll only give a
1749+high-level description of this process. If we have a `%a` udon, then
1750+we're performing a trivial replace, so we produce simply `q.q.don`. If
1751+we have a `%c` udon, then we're performing a list merge (as in, for
1752+example, lines of text). The merge is performed by `++lurk`.
1755+ ++ lurk :: apply list patch
1756+ |* [hel=(list) rug=(urge)]
1765+ hel (slag p.i.rug hel)
1766+ war (weld (flop (scag p.i.rug hel)) war)
1772+ hel =+ gur=(flop p.i.rug)
1775+ ?>(&(?=(^ hel) =(i.gur i.hel)) $(hel t.hel, gur t.gur))
1776+ war (weld q.i.rug war)
1781+We accumulate our final result in `war`. If there's nothing more in our
1782+list of merge instructions (unces), we just reverse `war` and produce
1783+it. Otherwise, we process another unce. If the unce is of type `&`, then
1784+we have `p.i.rug` lines of no changes, so we just copy them over from
1785+`hel` to `war`. If the unice is of type `|`, then we verify that the
1786+source lines (in `hel`) are what we expect them to be (`p.i.rug`),
1787+crashing on failure. If they're good, then we append the new lines in
1788+`q.i.rug` onto `war`.
1790+And that's really it. List merges are pretty easy. Anyway, if you
1791+recall, we were discussing `++checkout-ankh`.
1794+ ++ checkout-ankh :: checkout-ankh:ze
1795+ |= hat=(map path lobe) :: checkout commit
1798+ %+ roll (~(tap by hat) ~)
1799+ |= [[pat=path bar=lobe] ank=ankh]
1803+ =+ zar=(lobe-to-noun bar)
1804+ ank(q [~ (sham zar) zar])
1805+ =+ nak=(~(get by r.ank) i.pat)
1807+ r %+ ~(put by r.ank) i.pat
1808+ $(pat t.pat, ank (fall nak _ankh))
1812+If the path is null, then we calculate `zar`, the raw data at the path
1813+`pat` in this version. We produce the given ankh with the correct data.
1815+Otherwise, we try to get the child we're looking at from our parent
1816+ankh. If it's already been created, this succeeds; otherwise, we simply
1817+create a default blank ankh. We place ourselves in our parent after
1818+recursively computing our children.
1820+This algorithm really isn't that complicated, but it may not be
1821+immediately obvious why it works. An example should clear everything up.
1823+Suppose `hat` is a map of the following information.
1826+ /greeting --> "customary upon meeting"
1827+ /greeting/english --> "hello"
1828+ /greeting/spanish --> "hola"
1829+ /greeting/russian/short --> "привет"
1830+ /greeting/russian/long --> "Здравствуйте"
1831+ /farewell/russian --> "до свидания"
1834+Furthermore, let's say that we process them in this order:
1838+ /greeting/russian/short
1839+ /greeting/russian/long
1845+Then, the first path we process is `/greeting/english` . Since our path
1846+is not null, we try to get `nak`, but because our ankh is blank at this
1847+point it doesn't find anything. Thus, update our blank top-level ankh
1848+with a child `%greeting`. and recurse with the blank `nak` to create the
1849+ankh of the new child.
1851+In the recursion, we our path is `/english` and our ankh is again blank.
1852+We try to get the `english` child of our ankh, but this of course fails.
1853+Thus, we update our blank `/greeting` ankh with a child `english`
1854+produced by recursing.
1856+Now our path is null, so we call `++lobe-to-noun` to get the actual
1857+data, and we place it in the brand-new ankh.
1859+Next, we process `/greeting/russian/short`. Since our path is not null,
1860+we try to get the child named `%greeting`, which does exist since we
1861+created it earlier. We put modify this child by recursing on it. Our
1862+path is now `/russian/short`, so we look for a `%russian` child in our
1863+`/greeting` ankh. This doesn't exist, so we add it by recursing. Our
1864+path is now `/short`, so we look for a `%short` child in our
1865+`/greeting/russian` ankh. This doesn't exist, so we add it by recursing.
1866+Now our path is null, so we set the contents of this node to `"привет"`,
1867+and we're done processing this path.
1869+Next, we process `/greeting/russian/long`. This proceeds similarly to
1870+the previous except that now the ankh for `/greeting/russian` already
1871+exists, so we simply reuse it rather than creating a new one. Of course,
1872+we still must create a new `/greeting/russian/long` ankh.
1874+Next, we process `/greeting`. This ankh already exists, so after we've
1875+recursed once, our path is null, and our ankh is not blank -- it already
1876+has two children (and two grandchildren). We don't touch those, though,
1877+since a node may be both a file and a directory. We just add the
1878+contents of the file -- "customary upon meeting" -- to the existing
1881+Next, we process `/greeting/spanish`. Of course, the `/greeting` ankh
1882+already exists, but it doesn't have a `%spanish` child, so we create
1883+that, taking care not to disturb the contents of the `/greeting` file.
1884+We put "hola" into the ankh and call it good.
1886+Finally, we process `/farewell/russian`. Here, the `/farewell` ankh
1887+doesn't exist, so we create it. Clearly the newly-created ankh doesn't
1888+have any children, so we have to add a `%russian` child, and in this
1889+child we put our last content -- "до свидания".
1891+We hope it's fairly obvious that the order we process the paths doesn't
1892+affect the final ankh tree. The tree will be constructed in a very
1893+different order depending on what order the paths come in, but the
1894+resulting tree is independent of order.
1896+At any rate, we were talking about something important, weren't we? If
1897+you recall, that concludes our discussion of `++rewind`, which was
1898+called from `++read-at-aeon`. In summary, `++rewind` returns a context
1899+in which our current state is (very nearly) as it was when the specified
1900+version of the desk was the head. This allows `++read-at-aeon` to call
1901+`++read` to read the requested information.
1905+ |= mun=mood :: read at point
1908+ [~ `dome`+<+<.read]
1909+ ?: &(?=(%w p.mun) !?=(%ud -.q.mun))
1910+ ?^(r.mun ~ [~ let])
1916+ ?^(r.mun ~ [~ [t.yak (forge-nori yak)]])
1917+ ::?> ?=(^ hit) ?^(r.mun ~ [~ i.hit]) :: what do?? need [@da nori]
1918+ (query(ank ank:(descend-path:(zu ank) r.mun)) p.mun)
1921+If we're requesting the dome, then we just return that immediately.
1923+If we're requesting the revision number of the desk and we're not
1924+requesting it by number, then we just return the current number of this
1925+desk. Note of course that this was really already handled in
1928+If we're requesting a `%w` with a specific revision number, then we do
1929+something or other with the commit there. It's kind of weird, and it
1930+doesn't seem to work, so we'll ignore this case.
1932+Otherwise, we descend into the ankh tree to the given path with
1933+`++descend-path:zu`, and then we handle specific request in `++query`.
1936+ ++ descend-path :: descend recursively
1939+ ?~(way +> $(way t.way, +> (descend i.way)))
1942+This is simple recursion down into the ankh tree. `++descend` descends
1943+one level, so this will eventually get us down to the path we want.
1946+ ++ descend :: descend
1949+ =+ you=(~(get by r.ank) lol)
1950+ +>.$(ram [lol ram], ank ?~(you [*cash ~ ~] u.you))
1953+`ram` is the path that we're at, so to descend one level we push the
1954+name of this level onto that path. We update the ankh with the correct
1955+one at that path if it exists; else we create a blank one.
1957+Once we've decscended to the correct level, we need to actually deal
1961+ ++ query :: query:ze
1962+ |= ren=?(%u %v %x %y %z) :: endpoint query
1965+ %u [~ `rang`+<+>.query]
1966+ %v [~ `dome`+<+<.query]
1967+ %x ?~(q.ank ~ [~ q.u.q.ank])
1973+Now that everything's set up, it's really easy. If they're requesting
1974+the rang, dome, or ankh, we give it to them. If the contents of a file,
1975+we give it to them if it is in fact a file. If the `arch`, then we
1976+calculate it with `++as-arch`.
1979+ ++ as-arch :: as-arch:ze
1980+ ^- arch :: arch report
1982+ ?~(q.ank ~ [~ p.u.q.ank])
1983+ |- ^- (map ,@ta ,~)
1985+ [[p.n.r.ank ~] $(r.ank l.r.ank) $(r.ank r.r.ank)]
1988+This very simply strips out all the "real" data and returns just our own
1989+hash, the hash of the file contents (if we're a file), and a map of the
1990+names of our immediate children.
1992+## Lifecycle of a Local Subscription
1994+A subscription to a range of revisions of a desk initially follows the
1995+same path that a single read does. In `++aver`, we checked the head of
1996+the given rave. If the head was `&`, then it was a single request, so we
1997+handled it above. If `|`, then we handle it with the following code.
2000+ =+ nab=(~(case-to-aeon ze lim dom ran) p.p.rav)
2002+ ?> =(~ (~(case-to-aeon ze lim dom ran) q.p.rav))
2003+ (duce hen (rive rav))
2004+ =+ huy=(~(case-to-aeon ze lim dom ran) q.p.rav)
2005+ ?: &(?=(^ huy) |((lth u.huy u.nab) &(=(0 u.huy) =(0 u.nab))))
2007+ =+ top=?~(huy let.dom u.huy)
2008+ =+ sar=(~(lobes-at-path ze lim dom ran) u.nab r.p.rav)
2009+ =+ ear=(~(lobes-at-path ze lim dom ran) top r.p.rav)
2012+ =+ fud=(~(make-nako ze lim dom ran) u.nab top)
2013+ (bleb hen u.nab fud)
2018+ (duce hen `rove`[%| ptr q.p.rav r.p.rav ear])
2022+Recall that `++case-to-aeon:ze` produces the revision number that a case
2023+corresponds to, if it corresponds to any. If it doesn't yet correspond
2024+to a revision, then it produces null.
2026+Thus, we first check to see if we've even gotten to the beginning of the
2027+range of revisions requested. If not, then we assert that we haven't yet
2028+gotten to the end of the range either, because that would be really
2029+strange. If not, then we immediately call `++duce`, which, if you
2030+recall, for a local request, simply puts this duct and rove into our
2031+cult `qyx`, so that we know who to respond to when the revision does
2034+If we've already gotten to the first revision, then we can produce some
2035+content immediately. If we've also gotten to the final revision, and
2036+that revision is earlier than the start revision, then it's a bad
2037+request and we call `++blub`, which tells the subscriber that his
2038+subscription will not be satisfied.
2040+Otherwise, we find the data at the given path at the beginning of the
2041+subscription and at the last available revision in the subscription. If
2042+they're the same, then we don't send a notification. Otherwise, we call
2043+`++gack`, which creates the `++nako` we need to produce. We call
2044+`++bleb` to actually produce the information.
2046+If we already have the last requested revision, then we also tell the
2047+subscriber with `++blub` that the subscription will receive no further
2050+If there will be more revisions in the subscription, then we call
2051+`++duce`, adding the duct to our subscribers. We modify the rove to
2052+start at the next revision since we've already handled all the revisions
2055+We glossed over the calls to `++lobes-at-path`, `++make-nako`, and
2056+`++bleb`, so we'll get back to those right now. `++bleb` is simple, so
2057+we'll start with that.
2060+ ++ bleb :: ship sequence
2061+ |= [hen=duct ins=@ud hip=nako]
2063+ (blab hen [%w [%ud ins] ~] hip)
2066+We're given a duct, the beginning revision number, and the nako that
2067+contains the updates since that revision. We use `++blab` to produce
2068+this result to the subscriber. The case is `%w` with a revision number
2069+of the beginning of the subscription, and the data is the nako itself.
2071+We call `++lobes-at-path:ze` to get the data at the particular path.
2074+ ++ lobes-at-path :: lobes-at-path:ze
2075+ |= [oan=aeon pax=path] :: data at path
2090+ $(p +.p, pax +.pax)
2094+At revision zero, the theoretical common revision between all
2095+repositories, there is no data, so we produce null.
2097+We get the list of paths (paired with their lobe) in the revision
2098+referred to by the given number and we keep only those paths which begin
2099+with `pax`. Converting to a map, we now have a map from the subpaths at
2100+the given path to the hash of their data. This is simple and efficient
2101+to calculate and compare to later revisions. This allows us to easily
2102+tell if a node or its children have changed.
2104+Finally, we will describe `++make-nako:ze`.
2107+ ++ make-nako :: gack a through b
2109+ ^- [(map aeon tako) aeon (set yaki) (set blob)]
2111+ =- [(takos-to-yakis -<) (lobes-to-blobs ->)]
2112+ %+ reachable-between-takos
2113+ (~(get by hit) a) :: if a not found, a=0
2116+ %- mo %+ skim (~(tap by hit) ~)
2118+ &((gth p a) (lte p b))
2121+We need to produce four things -- the numbers of the new commits, the
2122+number of the latest commit, the new commits themselves, and the new
2125+The first is fairly easy to produce. We simply go over our map of
2126+numbered commits and produce all those numbered greater than `a` and not
2129+The second is even easier to produce -- `b` is clearly our most recent
2132+The third and fourth are slightly more interesting, though not too
2133+terribly difficult. First, we call `++reachable-between-takos`.
2136+ ++ reachable-between-takos
2137+ |= [a=(unit tako) b=tako] :: pack a through b
2138+ ^- [(set tako) (set lobe)]
2141+ (reachable-takos r:(tako-to-yaki u.a))
2142+ =+ yak=`yaki`(tako-to-yaki b)
2143+ %+ new-lobes-takos (new-lobes ~ sar) :: get lobes
2144+ |- ^- (set tako) :: walk onto sar
2145+ ?: (~(has in sar) r.yak)
2147+ =+ ber=`(set tako)`(~(put in `(set tako)`~) `tako`r.yak)
2151+ |= [yek=tako bar=(set tako)]
2153+ ?: (~(has in bar) yek) :: save some time
2156+ ^$(yak (tako-to-yaki yek))
2159+We take a possible starting commit and a definite ending commit, and we
2160+produce the set of commits and the set of data between them.
2162+We let `sar` be the set of commits reachable from `a`. If `a` is null,
2163+then obviously no commits are reachable. Otherwise, we call
2164+`++reachable-takos` to calculate this.
2167+ ++ reachable-takos :: reachable
2168+ |= p=tako :: XX slow
2170+ =+ y=(tako-to-yaki p)
2171+ =+ t=(~(put in _(set tako)) p)
2174+ ?: (~(has in s) q) :: already done
2176+ (~(uni in s) ^$(p q)) :: otherwise traverse
2179+We very simply produce the set of the given tako plus its parents,
2182+Back in `++reachable-between-takos`, we let `yak` be the yaki of `b`,
2183+the ending commit. With this, we create a set that is the union of `sar`
2184+and all takos reachable from `b`.
2186+We pass `sar` into `++new-lobes` to get all the lobes referenced by any
2187+tako referenced by `a`. The result is passed into `++new-lobes-takos` to
2188+do the same, but not recomputing those in already calculated last
2189+sentence. This produces the sets of takos and lobes we need.
2192+ ++ new-lobes :: object hash set
2193+ |= [b=(set lobe) a=(set tako)] :: that aren't in b
2195+ %+ roll (~(tap in a) ~)
2196+ |= [tak=tako bar=(set lobe)]
2198+ =+ yak=(tako-to-yaki tak)
2199+ %+ roll (~(tap by q.yak) ~)
2200+ |= [[path lob=lobe] far=_bar]
2202+ ?~ (~(has in b) lob) :: don't need
2204+ =+ gar=(lobe-to-blob lob)
2206+ %direct (~(put in far) lob)
2207+ %delta (~(put in $(lob q.gar)) lob)
2208+ %indirect (~(put in $(lob s.gar)) lob)
2212+Here, we're creating a set of lobes referenced in a commit in `a`. We
2213+start out with `b` as the initial set of lobes, so we don't need to
2214+recompute any of the lobes referenced in there.
2216+The algorithm is pretty simple, so we won't bore you with the details.
2217+We simply traverse every commit in `a`, looking at every blob referenced
2218+there, and, if it's not already in `b`, we add it to `b`. In the case of
2219+a direct blob, we're done. For a delta or an indirect blob, we
2220+recursively add every blob referenced within the blob.
2223+ ++ new-lobes-takos :: garg & repack
2224+ |= [b=(set lobe) a=(set tako)]
2225+ ^- [(set tako) (set lobe)]
2226+ [a (new-lobes b a)]
2229+Here, we just update the set of lobes we're given with the commits we're
2230+given and produce both sets.
2232+This concludes our discussion of a local subscription.
2234+## Lifecycle of a Foreign Read or Subscription
2236+Foreign reads and subscriptions are handled in much the same way as
2237+local ones. The interface is the same -- a vane or app sends a `%warp`
2238+kiss with the request. The difference is simply that the `sock` refers
2239+to the foreign ship.
2241+Thus, we start in the same place -- in `++call`, handling `%warp`.
2242+However, since the two side of the `sock` are different, we follow a
2246+ =+ wex=(do now p.q.hic p.q.q.hic ruf)
2249+ abet:(ease:wex hen)
2250+ abet:(eave:wex hen u.q.q.q.hic)
2251+ [-.woo (posh q.p.q.hic p.q.q.hic +.woo ruf)]
2254+If we compare this to how the local case was handled, we see that it's
2255+not all that different. We use `++do` rather than `++un` and `++de` to
2256+set up the core for the foreign ship. This gives us a `++de` core, so we
2257+either cancel or begin the request by calling `++ease` or `++eave`,
2258+exactly as in the local case. In either case, we call `++abet:de` to
2259+resolve our various types of output into actual moves, as described in
2260+the local case. Finally, we call `++posh` to update our raft, putting
2261+the modified rung into the raft.
2263+We'll first trace through `++do`.
2267+ |= [now=@da [who=ship him=ship] syd=@tas ruf=raft]
2269+ =+ rug=(~(get by hoy.ruf) him)
2270+ ?^(rug u.rug *rung)
2272+ =+ yit=(~(get by rus.rug) syd)
2273+ ?^(yit u.yit `rede`[~2000.1.1 ~ [~ *rind] *dome])
2274+ ((de now ~ ~) [who him] syd red ran.ruf)
2277+If we already have a rung for this foreign ship, then we use that.
2278+Otherwise, we create a new blank one. If we already have a rede in this
2279+rung, then we use that, otherwise we create a blank one. An important
2280+point to note here is that we let `ref` in the rede be `[~ *rind]`.
2281+Recall, for domestic desks `ref` is null. We use this to distinguish
2282+between foreign and domestic desks in `++de`.
2284+With this information, we create a `++de` core as usual.
2286+Although we've already covered `++ease` and `++eave`, we'll go through
2287+them quickly again, highlighting the case of foreign request.
2290+ ++ ease :: release request
2294+ =+ rov=(~(got by qyx) hen)
2295+ =. qyx (~(del by qyx) hen)
2296+ (mabe rov (cury best hen))
2297+ =. qyx (~(del by qyx) hen)
2299+ =+ nux=(~(get by fod.u.ref) hen)
2302+ say [[hen [(scot %ud u.nux) ~] for [u.nux syd ~]] say]
2303+ fod.u.ref (~(del by fod.u.ref) hen)
2304+ bom.u.ref (~(del by bom.u.ref) u.nux)
2308+Here, we still remove the duct from our cult (we maintain a cult even
2309+for foreign desks), but we also need to tell the foreign desk to cancel
2310+our subscription. We do this by sending a request (by appending to
2311+`say`, which gets resolved in `++abet:de` to a `%want` kiss to ames) to
2312+the foreign ship to cancel the subscription. Since we don't anymore
2313+expect a response on this duct, we remove it from `fod` and `bom`, which
2314+are the maps between ducts, raves, and request sequence numbers.
2315+Basically, we remove every trace of the subscription from our request
2318+In the case of `++eave`, where we're creating a new request, everything
2319+is exactly identical to the case of the local request except `++duce`.
2320+We said that `++duce` simply puts the request into our cult. This is
2321+true for a domestic request, but distinctly untrue for foreign requests.
2324+ ++ duce :: produce request
2325+ |= [hen=duct rov=rove]
2327+ =. qyx (~(put by qyx) hen rov)
2329+ |- ^+ +>+.$ :: XX why?
2332+ ?. ?=([%& %v *] rav) rav
2333+ [%| [%ud let.dom] `case`q.p.rav r.p.rav]
2336+ say [[hen [(scot %ud inx) ~] for [inx syd ~ vaw]] say]
2337+ nix.u.ref +(nix.u.ref)
2338+ bom.u.ref (~(put by bom.u.ref) inx [hen vaw])
2339+ fod.u.ref (~(put by fod.u.ref) hen inx)
2343+If we have a request manager (i.e. this is a foreign desk), then we do
2344+the approximate inverse of `++ease`. We create a rave out of the given
2345+request and send it off to the foreign desk by putting it in `say`. Note
2346+that the rave is created to request the information starting at the next
2347+revision number. Since this is a new request, we put it into `fod` and
2348+`bom` to associate the request with its duct and its sequence number.
2349+Since we're using another sequence number, we must increment `nix`,
2350+which represents the next available sequence number.
2352+And that's really it for this side of the request. Requesting foreign
2353+information isn't that hard. Let's see what it looks like on the other
2354+side. When we get a request from another ship for information on our
2355+ship, that comes to us in the form of a `%wart` from ames.
2357+We handle a `%wart` in `++call`, right next to where we handle the
2363+ =+ ryf=((hard riff) s.q.hic)
2366+ :^ %pass [(scot %p p.p.q.hic) (scot %p q.p.q.hic) r.q.hic]
2368+ [%warp [p.p.q.hic p.p.q.hic] ryf]
2372+Every request we receive should be of type `riff`, so we coerce it into
2373+that type. We just convert this into a new `%warp` kiss that we pass to
2374+ourself. This gets handled like normal, as a local request. When the
2375+request produces a value, it does so like normal as a `%writ`, which is
2376+returned to `++take` along the path we just sent it on.
2381+ =+ our=(need (slaw %p i.tea))
2382+ =+ him=(need (slaw %p i.t.tea))
2385+ [%pass ~ %a [%want [our him] [%r %re %c t.t.tea] p.+.q.hin]]
2389+Since we encoded the ship we need to respond to in the path, we can just
2390+pass our `%want` back to ames, so that we tell the requesting ship about
2393+This comes back to the original ship as a `%waft` from ames, which comes
2394+into `++take`, right next to where we handled `%writ`.
2399+ =+ syd=(need (slaw %tas i.tea))
2400+ =+ inx=(need (slaw %ud i.t.tea))
2403+ (knit:(do now p.+.q.hin syd ruf) [inx ((hard riot) q.+.q.hin)])
2406+ [-.zot (posh q.p.+.q.hin syd +.zot ruf)]
2407+ [mos ..^$(ran.ruf ran.zat)] :: merge in new obj
2410+This gets the desk and sequence number from the path the request was
2411+sent over. This determines exactly which request is being responded to.
2412+We call `++knit:de` to apply the changes to our local desk, and we call
2413+`++wake` to update our subscribers. Then we call `++abet:de` and
2414+`++posh` as normal (like in `++eave`).
2416+We'll examine `++knit` and `++wake`, in that order.
2419+ ++ knit :: external change
2420+ |= [inx=@ud rot=riot]
2424+ =+ ruv=(~(get by bom.u.ref) inx)
2426+ => ?. |(?=(~ rot) ?=(& -.q.u.ruv)) .
2428+ bom.u.ref (~(del by bom.u.ref) inx)
2429+ fod.u.ref (~(del by fod.u.ref) p.u.ruv)
2432+ =+ rav=`rave`q.u.ruv
2435+ ?.(&(?=(| -.rav) ?=(%da -.q.p.rav)) lim `@da`p.q.p.rav)
2438+ ?. ?=(& -.rav) haw.u.ref
2439+ (~(put by haw.u.ref) p.rav ~)
2441+ ?< ?=(%v p.p.u.rot)
2443+ (~(put by haw.u.ref) [p.p.u.rot q.p.u.rot q.u.rot] ~ r.u.rot)
2444+ ?. ?=(%w p.p.u.rot) +>+.$
2446+ =+ nez=[%w [%ud let.dom] ~]
2447+ =+ nex=(~(get by haw.u.ref) nez)
2449+ ?~ u.nex +>+.^$ :: should never happen
2450+ =. +>+.^$ =+ roo=(edis ((hard nako) u.u.nex))
2451+ ?>(?=(^ ref.roo) roo)
2453+ haw.u.ref (~(del by haw.u.ref) nez)
2457+This is kind of a long gate, but don't worry, it's not bad at all.
2459+First, we assert that we're not a domestic desk. That wouldn't make any
2462+Since we have the sequence number of the request, we can get the duct
2463+and rave from `bom` in our request manager. If we didn't actually
2464+request this data (or the request was canceled before we got it), then
2467+Else, we remove the request from `bom` and `fod` unless this was a
2468+subscription request and we didn't receive a null riot (which would
2469+indicate the last message on the subscription).
2471+Now, if we received a null riot, then if this was a subscription request
2472+by date, then we update `lim` in our request manager (representing the
2473+latest time at which we have complete information for this desk) to the
2474+date that was requested. If this was a single read request, then we put
2475+the result in our simple cache `haw` to make future requests faster.
2478+If we received actual data, then we put it into our cache `haw`. Unless
2479+it's a `%w` request, we're done.
2481+If it is a `%w` request, then we try to get the `%w` at our current head
2482+from the cache. In general, that should be the thing we just put in a
2483+moment ago, but that is not guaranteed. The most common case where this
2484+is different is when we receive desk updates out of order. At any rate,
2485+since we now have new information, we need to apply it to our local copy
2486+of the desk. We do so in `++edis`, and then we remove the stuff we just
2487+applied from the cache, since it's not really a true "single read", like
2488+what should really be in the cache.
2491+ ++ edis :: apply subscription
2495+ hit.dom (~(uni by hit.dom) gar.nak)
2497+ lat.ran %+ roll (~(tap in bar.nak) ~)
2499+ |= [sar=blob yeb=(map lobe blob)]
2500+ =+ zax=(blob-to-lobe sar)
2501+ %+ ~(put by yeb) zax sar
2502+ hut.ran %+ roll (~(tap in lar.nak) ~)
2504+ |= [sar=yaki yeb=(map tako yaki)]
2505+ %+ ~(put by yeb) r.sar sar
2509+This shows, of course, exactly why nako is defined the way it is. To
2510+become completely up to date with the foreign desk, we need to merge
2511+`hit` with the foreign one so that we have all the revision numbers. We
2512+update `let` so that we know which revision is the head.
2514+We merge the new blobs in `lat`, keying them by their hash, which we get
2515+from a call to `++blob-to-lobe`. Recall that the hash is stored in the
2516+actual blob itself. We do the same thing to the new yakis, putting them
2517+in `hut`, keyed by their hash.
2519+Now, our local dome should be exactly the same as the foreign one.
2521+This concludes our discussion of `++knit`. Now the changes have been
2522+applied to our local copy of the desk, and we just need to update our
2523+subscribers. We do so in `++wake:de`.
2526+ ++ wake :: update subscribers
2528+ =+ xiq=(~(tap by qyx) ~)
2529+ =| xaq=(list ,[p=duct q=rove])
2532+ ..wake(qyx (~(gas by *cult) xaq))
2535+ =+ cas=?~(ref ~ (~(get by haw.u.ref) `mood`p.q.i.xiq))
2539+ ..wake ?~ u.cas (blub p.i.xiq)
2540+ (blab p.i.xiq p.q.i.xiq u.u.cas)
2542+ =+ nao=(~(case-to-aeon ze lim dom ran) q.p.q.i.xiq)
2543+ ?~ nao $(xiq t.xiq, xaq [i.xiq xaq])
2544+ $(xiq t.xiq, ..wake (balk p.i.xiq u.nao p.q.i.xiq))
2547+ =+ mot=`moot`p.q.i.xiq
2548+ =+ nab=(~(case-to-aeon ze lim dom ran) p.mot)
2550+ $(xiq t.xiq, xaq [i.xiq xaq])
2551+ =+ huy=(~(case-to-aeon ze lim dom ran) q.mot)
2553+ =+ ptr=[%ud +(let.dom)]
2556+ xaq [[p.i.xiq [%| ptr q.mot r.mot s.mot]] xaq]
2558+ (~(lobes-at-path ze lim dom ran) let.dom r.p.q.i.xiq)
2559+ ?: =(s.p.q.i.xiq ear) ..wake
2560+ =+ fud=(~(make-nako ze lim dom ran) u.nab let.dom)
2561+ (bleb p.i.xiq let.dom fud)
2565+ ..wake =- (blub:- p.i.xiq)
2567+ (~(lobes-at-path ze lim dom ran) u.huy r.p.q.i.xiq)
2568+ ?: =(s.p.q.i.xiq ear) (blub p.i.xiq)
2569+ =+ fud=(~(make-nako ze lim dom ran) u.nab u.huy)
2570+ (bleb p.i.xiq +(u.nab) fud)
2576+This is even longer than `++knit`, but it's pretty similar to `++eave`.
2577+We loop through each of our subscribers `xiq`, processing each in turn.
2578+When we're done, we just put the remaining subscribers back in our
2581+If the subscriber is a single read, then, if this is a foreign desk
2582+(note that `++wake` is called from other arms, and not only on foreign
2583+desks). Obviously, if we find an identical request there, then we can
2584+produce the result immediately. Referential transparency for the win. We
2585+produce the result with a call to `++blab`. If this is a foreign desk
2586+but the result is not in the cache, then we produce `++blub` (canceled
2587+subscription with no data) for reasons entirely opaque to me. Seriously,
2588+it seems like we should wait until we get an actual response to the
2589+request. If someone figures out why this is, let me know. At any rate,
2592+If this is a domestic desk, then we check to see if the case exists yet.
2593+If it doesn't, then we simply move on to the next subscriber, consing
2594+this one onto `xaq` so that we can check again the next time around. If
2595+it does exist, then we call `++balk` to fulfill the request and produce
2598+`++balk` is very simple, so we'll describe it here before we get to the
2602+ ++ balk :: read and send
2603+ |= [hen=duct oan=@ud mun=mood]
2605+ =+ vid=(~(read-at-aeon ze lim dom ran) oan mun)
2606+ ?~ vid (blub hen) (blab hen mun u.vid)
2609+We call `++read-at-aeon` on the given request and aeon. If you recall,
2610+this processes a `mood` at a particular aeon and produces the result, if
2611+there is one. If there is data at the requested location, then we
2612+produce it with `++blab`. Else, we call `++blub` to notify the
2613+subscriber that no data can ever come over this subscriptioin since it
2614+is now impossible for there to ever be data for the given request.
2615+Because referential transparency.
2617+At any rate, back to `++wake`. If the given rave is a subscription
2618+request, then we proceed similarly to how we do in `++eave`. We first
2619+try to get the aeon referred to by the starting case. If it doesn't
2620+exist yet, then we can't do anything interesting with this subscription,
2621+so we move on to the next one.
2623+Otherwise, we try to get the aeon referred to by the ending case. If it
2624+doesn't exist yet, then we produce all the information we can. We call
2625+`++lobes-at-path` at the given aeon and path to see if the requested
2626+path has actually changed. If it hasn't, then we don't produce anything;
2627+else, we produce the correct nako by calling `++bleb` on the result of
2628+`++make-nako`, as in `++eave`. At any rate, we move on to the next
2629+subscription, putting back into our cult the current subscription with a
2630+new start case of the next aeon after the present.
2632+If the aeon referred to by the ending case does exist, then we drop this
2633+subscriber from our cult and satisfy its request immediately. This is
2634+the same as before -- we check to see if the data at the path has
2635+actually changed, producing it if it has; else, we call `++blub` since
2636+no more data can be produced over this subscription.
2638+This concludes our discussion of foreign requests.