@@ -23,2727 +23,42 @@ zuse. Second is the write, query, and subscription logic. Finally, there
23is the logic for communicating requests to, and receiving requests from,
23is the logic for communicating requests to, and receiving requests from,
24foreign ships.
24foreign ships.
26−## Merge strategies
26+Clay documentation is extensive, but the reader much tread carefully as portions
27+of it are out of date as of March 2021. We are actively working on updating
28+them. If you'd like to help, please reach out to us in the Docs channel on Urbit
29+Community or submit a PR at
30+[https://github.com/urbit/docs](https://github.com/urbit/docs).
28−*This section is up to date as of November 2020*
32+### User documentation
30−Usage:
34+[Filesystem](@/using/os/filesystem.md)
32−|merge %destination-desk ~source-ship %source-desk
33−|merge %destination-desk ~source-ship %source-desk, =gem %strategy
34−|merge %destination-desk ~source-ship %source-desk, =cas ud+5
37−Clay supports various merge strategies. A "commit" is a snapshot of
36+How to interact with the Clay filesystem via Dojo. This includes basics such as
38−the files with a list of parents plus a date. Most commits have
37+mounting to Unix, changing directory, merging, and listing files.
39−one parent; a "merge" commit is a commit with two parents. The
40−%home desk starts with an initial commit with no parents; commits
41−with several parents ("octopus merges") are possible but we don't
42−generate them right now.
44−Unless otherwise specified, all of the following create a new commit
45−with the source and destination commits as parents.
47−Several strategies need a "merge-base". They find it by identifying
48−the most recent common ancestor of the two desks. If none, fail
49−with `%merge-no-merge-base`; if there are two or more, pick one.
51−`%init`: the only way to create a desk. Not a true merge, since it
52−simply assigns the source commit to the destination.
54−`%fine`: if source or destination are in the ancestry of each other,
55−use the newer one; else abort. If the destination is ahead of the
56−source, succeed but do nothing. If the source is ahead of the
57−destination, assign the next revision number to the source commit.
58−Some call this "fast-forward".
60−`%meet`: combine changes, failing if both sides changed the same file.
61−Specifically, take diff(merge-base,source) and
62−diff(merge-base,destination) and combine them as long as those diffs
63−touch different files.
65−`%mate`: combine changes, failing if both sides changed the same part
66−of a file. Identical to `%meet`, except that some marks, like `%hoon`,
67−allow intelligent merge of changes to different parts of a file.
69−`%meld`: combine changes; if both sides changed the same part of a
70−file, use the version of the file in the merge-base.
72−`%only-this`: create a merge commit with exactly the contents of the
75−`%only-that`: create a merge commit with exactly the contents of the
78−`%take-this`: create a merge commit with exactly the contents of the
79−destination desk except take any files from the source commit which
80−are not in the destination desk.
82−`%take-that`: create a merge commit with exactly the contents of the
83−source commit except preserve any files from the destination desk
84−which are not in the source commit.
86−`%meet-this`: merge as in `%meet`, except if both sides changed the same
87−file, use the version in the destination desk.
89−`%meet-that`: merge as in `%meet`, except if both sides changed the same
90−file, use the version in the source commit.
92−### Examples and notes:
94−The most common merge strategy is %mate, which is a normal 3-way
95−merge which aborts on conflict.
97−`%take-that` is useful to "force" an OTA. After running `%take-that`,
98−you're guaranteed to have exactly the files in the source commit plus
99−any files you separately added.
101−We speak of merging into a destination *desk* from a source *commit*
102−because while you can only merge on top of a desk, you can merge from
103−historical commits. For example,
105−|merge %old our %home, =cas ud+5, =gem %init
107−will create a new desk called `%old` with the 5th commit in `%home`.
108−You can revert the contents of a desk to what they were yesterday
111−|merge %home our %home, =cas da+(sub now ~d1), =gem %only-that
114−Note this is a normal `%only-that merge`, which means you're creating a
115−*new* commit with the old *contents*.
117−`%meld` is rarely used on its own, however if you specify `%auto` or
118−omit the merge strategy, `%kiln` will run a `%meld` merge into a scratch
119−desk and then annotate the conflicts there.
121−If you resolve merge conflicts manually, for example by mounting the
122−desks, copying the files in Unix and then running `|commit`, you
123−should usually run an `%only-this merge`. This will not change the
124−newly-fixed contents of your desk, but it will record that the merge
125−happened so that those conflicts don't reappear in later merges.
127−If you get a `%merge-no-merge-base` error, this means you're trying to
128−merge two desks which have no common ancestors. You need to give
129−them a common ancestor by choosing a merge strategy which doesn't
130−need a merge-base, like `%only-this`, `%only-that`, `%take-this`, or
133−`%take-this` could be useful to install 3rd party software, but you
134−couldn't get subsequent updates this way, since the files would
135−already exist in the destination desk. Something like "take only
136−the files which aren't in my OTA source or any other 3rd party app"
137−would be basically correct. This would require a parameter listing
138−the desks to not conflict with.
140−`%meet-this` and `%meet-that` imply the existence of `%mate-this` and
141−`%mate-that`, but those don't exist yet.
39+### Developer Documentation
144−*The remainder of this document is out of date as of November 2020, but certain parts are still
41+[Architecture](@/docs/arvo/clay/architecture.md)
145−accurate. Please independently confirm information here before acting on it.*
147−## Architecture
43+A conceptual overview of how Clay was designed. (Up to date as of March 2021)
149−Clay is the primary filesystem for the arvo operating system,
45+[Using Clay](@/docs/arvo/clay/using.md)
150−which is the [core](/docs/glossary/core/) of an urbit. The architecture of clay is
151−intrinsically connected with arvo, but we assume no knowledge of
152−either arvo or urbit. We will point out only those features of
153−arvo that are necessary for an understanding of clay, and we will
154−do so only when they arise.
156−The first relevant feature of arvo is that it is a deterministic
47+A quick overview of how the most common tasks involving Clay are performed:
157−system where input and output are defined as a series of events
48+reading and subscribing, syncing to Unix, and merging.
158−and effects. The state of arvo is simply a function of its event
159−log. None of the effects from an event are emitted until the
160−event is entered in the log and persisted, either to disk or
161−another trusted source of persistence, such as a Kafka cluster.
162−Consequently, arvo is a single-level store: everything in its
165−In a more traditional OS, everything in RAM can be erased at any
50+[Data Types](@/docs/arvo/clay/data-types.md)
166−time by power failure, and is always erased on reboot. Thus, a
167−primary purpose of a filesystem is to ensure files persist across
168−power failures and reboots. In arvo, both power failures and
169−reboots are special cases of suspending computation, which is
170−done safely since our event log is already persistent. Therefore,
171−clay is not needed in arvo for persistence. Why, then, do we have a
172−filesystem? There are two answers to this question.
174−First, clay provides a filesystem tree, which is a convenient
52+Explanations of the many data types found throughout Clay.
175−user interface for some applications. Unix has the useful concept
176−of virtual filesystems, which are used for everything from direct
177−access to devices, to random number generators, to the /proc
178−tree. It is easy and intuitive to read from and write to a
181−Second, clay has a distributed revision-control system baked into
54+[Local Reads and Writes](@/docs/arvo/clay/local-reads.md)
182−it. Traditional filesystems are not revision controlled, so
183−userspace software -- such as git -- is written on top of them to
184−do so. clay natively provides the same functionality as modern
187−clay has two other unique properties that we'll cover later on:
56+How local reads and writes work.
188−it supports typed data and is referentially transparent.
190−### Revision Control
58+[Local Subscriptions](@/docs/arvo/clay/local-sub.md)
192−Every urbit has one or more "desks", which are independently
60+How local subscriptions work.
193−revision-controlled branches. Each desk contains its own mark
194−definitions, apps, doc, and so forth.
196−Traditionally, an urbit has at least a base and a home desk. The
62+[Foreign Requests](@/docs/arvo/clay/foreign.md)
197−base desk has all the system software from the distribution. the
198−home desk is a fork of base with all the stuff specific to the
201−A desk is a series of numbered commits, the most recent of which
64+How foreign reads and subscriptions work.
202−represents the current state of the desk. A commit is composed of
203−(1) an absolute time when it was created, (2) a list of zero or
204−more parents, and (3) a map from paths to data.
206−Most commits have exactly one parent, but the initial commit on a
207−desk may have zero parents, and merge commits have more than one
210−The non-meta data is stored in the map of paths to data. It's
211−worth noting that no constraints are put on this map, so, for
212−example, both /a/b and /a/b/c could have data. This is impossible
213−in a traditional Unix filesystem since it means that /a/b is both
214−a file and a directory. Conventionally, the final element in the
215−path is its mark -- much like a filename extension in Unix. Thus,
216−/doc/readme.md in Unix is stored as /doc/readme/md in urbit.
218−The data is not stored directly in the map; rather, a hash of the
219−data is stored, and we maintain a master blob store. Thus, if the
220−same data is referred to in multiple commits (as, for example,
221−when a file doesn't change between commits), only the hash is
224−In the master blob store, we either store the data directly, or
225−else we store a diff against another blob. The hash is dependent
226−only on the data within and not on whether or not it's stored
227−directly, so we may on occasion rearrange the contents of the
228−blob store for performance reasons.
230−Recall that a desk is a series of numbered commits. Not every
231−commit in a desk must be numbered. For example, if the base desk
232−has had 50 commits since home was forked from it, then a merge
233−from base to home will only add a single revision number to home,
234−although the full commit history will be accessible by traversing
235−the parentage of the individual commits.
237−We do guarantee that the first commit is numbered 1, commits are
238−numbered consecutively after that (i.e. there are no "holes"),
239−the topmost commit is always numbered, and every numbered commit
240−is an ancestor of every later numbered commit.
242−There are three ways to refer to particular commits in the
243−revision history. Firstly, one can use the revision number.
244−Secondly, one can use any absolute time between the one numbered
245−commit and the next (inclusive of the first, exclusive of the
246−second). Thirdly, every desk has a map of labels to revision
247−numbers. These labels may be used to refer to specific commits.
249−Additionally, clay is a global filesystem, so data on other urbit
250−is easily accessible the same way as data on our local urbit. In
251−general, the path to a particular revision of a desk is
252−/~urbit-name/desk-name/revision. Thus, to get /try/readme/md
253−from revision 5 of the home desk on ~sampel-sipnym, we refer to
254−/~sampel-sipnym/home/5/try/readme/md. Clay's namespace is thus
255−global and referentially transparent.
259−### A Typed Filesystem
261−Since clay is a general filesystem for storing data of arbitrary
262−types, in order to revision control correctly it needs to be
263−aware of types all the way through. Traditional revision control
264−does an excellent job of handling source code, so for source code
265−we act very similar to traditional revision control. The
266−challenge is to handle other data similarly well.
268−For example, modern VCSs generally support "binary files", which
269−are files for which the standard textual diffing, patching, and
270−merging algorithms are not helpful. A "diff" of two binary files
271−is just a pair of the files, "patching" this diff is just
272−replacing the old file with the new one, and "merging"
273−non-identical diffs is always a conflict, which can't even be
274−helpfully annotated. Without knowing anything about the structure
275−of a blob of data, this is the best we can do.
277−Often, though, "binary" files have some internal structure, and
278−it is possible to create diff, patch, and merge algorithms that
279−take advantage of this structure. An image may be the result of a
280−base image with some set of operations applied. With algorithms
281−aware of this set of operations, not only can revision control
282−software save space by not having to save every revision of the
283−image individually, these transformations can be made on parallel
284−branches and merged at will.
286−Suppose Alice is tasked with touching up a picture, improving the
287−color balance, adjusting the contrast, and so forth, while Bob
288−has the job of cropping the picture to fit where it's needed and
289−adding textual overlay. Without type-aware revision control,
290−these changes must be made serially, requiring Alice and Bob to
291−explicitly coordinate their efforts. With type-aware revision
292−control, these operations may be performed in parallel, and then
293−the two changesets can be merged programmatically.
295−Of course, even some kinds of text files may be better served by
296−diff, patch, and merge algorithms aware of the structure of the
297−files. Consider a file containing a pretty-printed JSON object.
298−Small changes in the JSON object may result in rather significant
299−changes in how the object is pretty-printed (for example, by
300−addding an indentation level, splitting a single line into
303−A text file wrapped at 80 columns also reacts suboptimally with
304−unadorned Hunt-McIlroy diffs. A single word inserted in a
305−paragraph may push the final word or two of the line onto the
306−next line, and the entire rest of the paragraph may be flagged as
307−a change. Two diffs consisting of a single added word to
308−different sentences may be flagged as a conflict. In general,
309−prose should be diffed by sentence, not by line.
311−As far as the author is aware, clay is the first generalized,
312−type-aware revision control system. We'll go into the workings
313−of this system in some detail.
317−Central to a typed filesystem is the idea of types. In clay, we
318−call these "marks". A mark is a file that defines a type,
319−conversion routines to and from the mark, and diff, patch, and
322−For example, a `%txt` mark may be a list of lines of text, and it
323−may include conversions to `%mime` to allow it to be serialized
324−and sent to a browswer or to the unix filesystem. It will also
325−include Hunt-McIlroy diff, patch, and merge algorithms.
327−A `%json` mark would be defined as a json object in the code, and
328−it would have a parser to convert from `%txt` and a printer to
329−convert back to `%txt`. The diff, patch, and merge algorithms are
330−fairly straightforward for json, though they're very different
333−More formally, a mark is a core with three [arms](/docs/glossary/arm/), `++grab`,
334−`++grow`, and `++grad`. In `++grab` is a series of functions to
335−convert from other marks to the given mark. In `++grow` is a
336−series of functions to convert from the given mark to other
337−marks. In `++grad` is `++diff`, `++pact`, `++join`, and `++mash`.
339−The types are as follows, in an informal pseudocode:
343− ++ mime: <mime> -> <mark-type>
344− ++ txt: <txt> -> <mark-type>
347− ++ mime: <mark-type> -> <mime>
348− ++ txt: <mark-type> -> <txt>
351− ++ diff: (<mark-type>, <mark-type>) -> <diff-type>
352− ++ pact: (<mark-type>, <diff-type>) -> <mark-type>
353− ++ join: (<diff-type>, <diff-type>) -> <diff-type> or NULL
354− ++ mash: (<diff-type>, <diff-type>) -> <diff-type>
357−These types are basically what you would expect. Not every mark
358−has each of these functions defined -- all of them are optional
361−In general, for a particular mark, the `++grab` and `++grow` entries
362−(if they exist) should be inverses of each other.
364−In `++grad`, `++diff` takes two instances of a mark and produces
365−a diff of them. `++pact` takes an instance of a mark and patches
366−it with the given diff. `++join` takes two diffs and attempts to
367−merge them into a single diff. If there are conflicts, it
368−produces null. `++mash` takes two diffs and forces a merge,
369−annotating any conflicts.
371−In general, if `++diff` called with A and B produces diff D, then
372−`++pact` called with A and D should produce B. Also, if `++join`
373−of two diffs does not produce null, then `++mash` of the same
374−diffs should produce the same result.
376−Alternately, instead of `++diff`, `++pact`, `++join`, and
377−`++mash`, a mark can provide the same functionality by defining
378−`++sted` to be the name of another mark to which we wish to
379−delegate the revision control responsibilities. Then, before
380−running any of those functions, clay will convert to the other
381−mark, and convert back afterward. For example, the `%hoon` mark
382−is revision-controlled in the same way as `%txt`, so its `++grad`
383−is simply `++sted %txt`. Of course, `++txt` must be defined in
384−`++grow` and `++grab` as well.
386−Every file in clay has a mark, and that mark must have a
387−fully-functioning `++grad`. Marks are used for more than just
388−clay, and other marks don't need a `++grad`, but if a piece of
389−data is to be saved to clay, we must know how to revision-control
392−Additionally, if a file is to be synced out to unix, then it must
393−have conversion routines to and from the `%mime` mark.
397−### Reading and Subscribing
399−When reading from Clay, there are three types of requests. A
400−`%sing` request asks for data at single revsion. A `%next`
401−request asks to be notified the next time there's a change to
402−given file. A `%many` request asks to be notified on every
403−change in a desk for a range of changes.
405−For `%sing` and `%next`, there are generally three things to be
406−queried. A `%u` request simply checks for the existence of a
407−file at a path. A `%x` request gets the data in the file at a
408−path. A `%y` request gets a hash of the data in the file at the
409−path combined with all its children and their data. Thus, `%y`
410−of a node changes if it or any of its children change.
412−A `%sing` request is fulfilled immediately if possible. If the
413−requested revision is in the future, or is on another ship for
414−which we don't have the result cached, we don't respond
415−immediately. If the requested revision is in the future, we wait
416−until the revision happens before we respond to the request. If
417−the request is for data on another ship, we pass on the request
418−to the other ship. In general, Clay subscriptions, like most
419−things in Urbit, aren't guaranteed to return immediately.
420−They'll return when they can, and they'll do so in a
421−referentially transparent manner.
423−A `%next` request checks query at the given revision, and it
424−produces the result of the query the next time it changes, along
425−with the revsion number when it changes. Thus, a `%next` of a
426−`%u` is triggered when a file is added or deleted, a `%next` of a
427−`%x` is triggered when a file is added, deleted, or changed, and
428−a `%next` of a `%y` is triggered when a file or any of its
429−children is added, deleted, or changed.
431−A `%many` request is triggered every time the given desk has a
432−new revision. Unlike a `%next`, a `%many` has both a start and
433−an end revsion, after which it stops returning. For `%next`, a
434−single change is reported, and if the caller wishes to hear of
435−the next change, it must resubscribe. For `%many`, every revsion
436−from the start to the end triggers a response. Since a `%many`
437−request doesn't ask for any particular data, there aren't `%u`,
438−`%x`, and `%y` versions for it.
442−One of the primary functions of clay is as a convenient user
443−interface. While tools exist to use clay from within urbit, it's
444−often useful to be able to treat clay like any other filesystem
445−from the Unix perspective -- to "mount" it, as it were.
447−From urbit, you can run `|mount /path/to/directory %mount-point`,
448−and this will mount the given clay directory to the mount-point
449−directory in Unix. Every file is converted to `%mime` before it's
450−written to Unix, and converted back when read from Unix. The
451−entire directory is watched (a la Dropbox), and every change is
452−committed once you run `|commit %mount-point`.
456−Merging is a fundamental operation for a distributed revision
457−control system. At their root, clay's merges are similar to
458−git's, but with some additions to accomodate typed data. There
459−are seven different merge strategies.
461−Throughout our discussion, we'll say that the merge is from
462−Alice's desk to Bob's. Recall that a commit is a date (for all
463−new commits this will be the current date), a list of parents,
466−A `%init` merge should be used iff it's the first commit to a
467−desk. The head of Alice's desk is used as the number 1 commit to
468−Bob's desk. Obviously, the ancestry remains intact through
469−traversing the parentage of the commit even though previous
470−commits are not numbered for Bob's desk.
472−A `%this` merge means to keep what's in Bob's desk, but join the
473−ancestry. Thus, the new commit has the head of each desk as
474−parents, but the data is exactly what's in Bob's desk. For those
475−following along in git, this is the 'ours' merge strategy, not
476−the '--ours' option to the 'recursive' merge strategy. In other
477−words, even if Alice makes a change that does not conflict with
478−Bob, we throw it away. It's Bob's way or the highway.
480−A `%that` merge means to take what's in Alice's desk, but join
481−the ancestry. This is the reverse of `%this`.
483−A `%fine` merge is a "fast-forward" merge. This succeeds iff one
484−head is in the ancestry of the other. In this case, we use the
485−descendant as our new head.
487−For `%meet`, `%mate`, and `%meld` merges, we first find the most
488−recent common ancestor to use as our merge base. If we have no
489−common ancestors, then we fail. If we have more than one most
490−recent common ancestor, then we have a criss-cross situation,
491−which should be handled delicately. At present, we delicately
492−throw up our hands and give up, but something akin to git's
493−'recursive' strategy should be implemented in the future.
495−There's a functional inclusion ordering on `%fine`, `%meet`,
496−`%mate`, and `%meld` such that if an earlier strategy would have
497−succeeded, then every later strategy will produce the same
498−result. Put another way, every earlier strategy is the same as
499−every later strategy except with a restricted domain.
501−A `%meet` merge only succeeds if the changes from the merge base
502−to Alice's head (hereafter, "Alice's changes") are in different
503−files than Bob's changes. In this case, the parents are both
504−Alice's and Bob's heads, and the data is the merge base plus
505−Alice's changed files plus Bob's changed files.
507−A `%mate` merge attempts to merge changes to the same file when
508−both Alice and bob change it. If the merge is clean, we use it;
509−otherwise, we fail. A merge between different types of changes --
510−for example, deleting a file vs changing it -- is always a
511−conflict. If we succeed, the parents are both Alice's and Bob's
512−heads, and the data is the merge base plus Alice's changed files
513−plus Bob's changed files plus the merged files.
515−A `%meld` merge will succeed even if there are conflicts. If
516−there are conflicts in a file, then we use the merge base's
517−version of that file, and we produce a set of files with
518−conflicts. The parents are both Alice's and Bob's heads, and the
519−data is the merge base plus Alice's changed files plus Bob's
520−changed files plus the successfully merged files plus the merge
521−base's version of the conflicting files.
523−That's the extent of the merge options in clay proper. In
524−userspace there's a final option `%auto`, which is the most
525−common. `%auto` checks to see if Bob's desk exists, and if it
526−doesn't we use a `%init` merge. Otherwise, we progressively try
527−`%fine`, `%meet`, and `%mate` until one succeeds.
529−If none succeed, we merge Bob's desk into a scratch desk. Then,
530−we merge Alice's desk into the scratch desk with the `%meld`
531−option to force the merge. For each file in the produced set of
532−conflicting files, we call the `++mash` function for the
533−appropriate mark, which annotates the conflicts if we know how.
535−Finally, we display a message to the user informing them of the
536−scratch desk's existence, which files have annotated conflicts,
537−and which files have unannotated conflicts. When the user has
538−resolved the conflicts, they can merge the scratch desk back into
539−Bob's desk. This will be a `%fine` merge since Bob's head is in
540−the ancestry of the scratch desk.
544−Tracking and staying in sync with another desk is another
545−fundamental operation. We call this "autosync". This doesn't mean
546−simply mirroring a desk, since that wouldn't allow local changes.
547−We simply want to apply changes as they are made upstream, as
548−long as there are no conflicts with local changes.
550−This is implemented by watching the other desk, and, when it has
551−changes, merging these changes into our desk with the usual merge
554−Note that it's quite reasonable for two desks to be autosynced to
555−each other. This results in any change on one desk being mirrored
556−to the other and vice versa.
558−Additionally, it's fine to set up an autosync even if one desk,
559−the other desk, or both desks do not exist. The sync will be
560−activated when the upstream desk comes into existence and will
561−create the downstream desk if needed.
565−The first part of this section will be reference documentation for the data
566−types used by our filesystem. In fact, as a general guide, we recommend
567−reading and attempting to understand the data structures used in any
568−Hoon code before you try to read the code itself. Although complete
569−understanding of the data structures is impossible without seeing them
570−used in the code, an 80% understanding greatly clarifies the code. As
571−another general guide, when reading Hoon, it rarely pays off to
572−understand every line of code when it appears. Try to get the gist of
573−it, and then move on. The next time you come back to it, it'll likely
574−make a lot more sense.
576−After a description of the data models, we'll give an overview of the
577−interface that vanes and applications can use to interact with the
580−Finally, we'll dive into the code and the algorithms themselves. You
585−As you're reading through this section, remember you can always come
586−back to this when you run into these types later on. You're not going to
587−remember everything the first time through, but it is worth reading, or
588−at least skimming, this so that you get a rough idea of how our state is
591−The types that are certainly worth reading are `++raft`, `++room`,
592−`++dome:clay`, `++ankh:clay`, `++rung:clay`, `++rang:clay`, `++blob:clay`, `++yaki:clay`, and `++nori:clay`
593−(possibly in that order). All in all, though, this section isn't too
594−long, so many readers may wish to quickly read through all of it. If you
595−get bored, though, just skip to the next section. You can always come
596−back when you need to.
598−### `++raft`, formal state
601− ++ raft :: filesystem
602− $: fat=(map ship room) :: domestic
603− hoy=(map ship rung) :: foreign
608−This is the state of our vane. Anything that must be remembered between
609−calls to clay is stored in this state.
611−`fat` is the set of domestic servers. This stores all the information
612−that is specfic to a particular ship on this pier. The keys to this map
613−are the ships on the current pier. all the information that is specific
614−to a particular foreign ship. The keys to this map are all the ships
615−whose filesystems we have attempted to access through clay.
617−`ran` is the store of all commits and deltas, keyed by hash. The is
618−where all the "real" data we know is stored; the rest is "just
621−#### `++room`, filesystem per domestic ship
624− ++ room :: fs per ship
625− $: hun=duct :: terminal duct
626− hez=(unit duct) :: sync duch
627− dos=(map desk dojo) :: native desk
631−This is the representation of the filesystem of a ship on our pier.
633−`hun` is the duct we use to send messages to dill to display
634−notifications of filesystem changes. Only `%note` gifts should be
635−produced along this duct. This is set by the `%init` kiss.
637−`hez`, if present, is the duct we use to send sync messages to unix so
638−that they end up in the pier unix directory. Only `%ergo` gifts should
639−be producd along this duct. This is set by `%into` and `%invo` kisses.
641−`dos` is a well-known operating system released in 1981. It is also the
642−set of desks on this ship, mapped to their data.
644−#### `++desk`, filesystem branch
647− ++ desk ,@tas :: ship desk case spur
650−This is the name of a branch of the filesystem. The default desks are
651−"arvo", "main", and "try". More may be created by simply referencing
652−them. Desks have independent histories and states, and they may be
653−merged into each other.
655−### `++dojo`, domestic desk state
658− ++ dojo ,[p=cult q=dome] :: domestic desk state
661−This is the all the data that is specific to a particular desk on a
662−domestic ship. `p` is the set of subscribers to this desk and `q` is the
665−#### `++cult`, subscriptions
668− ++ cult (map duct rave) :: subscriptions
671−This is the set of subscriptions to a particular desk. The keys are the
672−ducts from where the subscriptions requests came. The results will be
673−produced along these ducts. The values are a description of the
674−requested information.
676−#### `++rave:clay`, general subscription request
679− ++ rave :: general request
680− $% [& p=mood] :: single request
681− [| p=moat] :: change range
685−This represents a subscription request for a desk. The request can be
686−for either a single item in the desk or else for a range of changes on
689−### `++rove`, stored general subscription request
692− ++ rove (each mood moot) :: stored request
695−When we store a request, we store subscriptions with a little extra
696−information so that we can determine whether new versions actually
697−affect the path we're subscribed to.
699−#### `++mood:clay`, single subscription request
702− ++ mood ,[p=care q=case r=path] :: request in desk
705−This represents a request for the state of the desk at a particular
706−commit, specfied by `q`. `p` specifies what kind of information is
707−desired, and `r` specifies the path we are requesting.
709−#### `++moat:clay`, range subscription request
712− ++ moat ,[p=case q=case r=path] :: change range
715−This represents a request for all changes between `p` and `q` on path
716−`r`. You will be notified when a change is made to the node referenced
717−by the path or to any of its children.
719−### `++moot`, stored range subscription request
722− ++ moot ,[p=case q=case r=path s=(map path lobe)] ::
725−This is just a `++moat:clay` plus a map of paths to lobes. This map
726−represents the data at the node referenced by the path at case `p`, if
727−we've gotten to that case (else null). We only send a notification along
728−the subscription if the data at a new revision is different than it was.
730−### `++care:clay`, clay submode
733− ++ care ?(%u %v %w %x %y %z) :: clay submode
736−This specifies what type of information is requested in a subscription
739−`%u` requests the `++rang:clay` at the current moment. Because this
740−information is not stored for any moment other than the present, we
741−crash if the `++case:clay` is not a `%da` for now.
743−`%v` requests the `++dome:clay` at the specified commit.
745−`%w` requests the revsion number of the desk.
747−`%x` requests the file at a specified path at the specified commit. If
748−there is no node at that path or if the node has no contents (that is,
749−if `q:ankh` is null), then this produces null.
751−`%y` requests a `++arch` of the specfied commit at the specified path.
753−`%z` requests the `++ankh` of the specified commit at the specfied path.
755−### `++arch`, shallow filesystem node
758− ++ arch ,[p=@uvI q=(unit ,@uvI) r=(map ,@ta ,~)] :: fundamental node
761−This is analogous to `++ankh:clay` except that the we have neither our
762−contents nor the ankhs of our children. The other fields are exactly the
763−same, so `p` is a hash of the associated ankh, `u.q`, if it exists, is a
764−hash of the contents of this node, and the keys of `r` are the names of
765−our children. `r` is a map to null rather than a set so that the
766−ordering of the map will be equivalent to that of `r:ankh`, allowing
767−efficient conversion.
769−#### `++case:clay`, specifying a commit
772− ++ case :: ship desk case spur
773− $% [%da p=@da] :: date
774− [%tas p=@tas] :: label
775− [%ud p=@ud] :: number
779−A commit can be referred to in three ways: `%da` refers to the commit
780−that was at the head on date `p`, `%tas` refers to the commit labeled
781−`p`, and `%ud` refers to the commit numbered `p`. Note that since these
782−all can be reduced down to a `%ud`, only numbered commits may be
783−referenced with a `++case:clay`.
785−#### `++dome:clay`, desk data
788− ++ dome :: project state
789− $: ang=agon :: pedigree
792− hit=(map ,@ud tako) :: changes by id
793− lab=(map ,@tas ,@ud) :: labels
797−This is the data that is actually stored in a desk.
799−`ang` is unused and should be removed.
801−`ank` is the current state of the desk. Thus, it is the state of the
802−filesystem at revison `let`. The head of a desk is always a numbered
805−`let` is the number of the most recently numbered commit. This is also
806−the total number of numbered commits.
808−`hit` is a map of numerical ids to hashes of commits. These hashes are
809−mapped into their associated commits in `hut:rang:clay`. In general, the keys
810−of this map are exactly the numbers from 1 to `let`, with no gaps. Of
811−course, when there are no numbered commits, `let` is 0, so `hit` is
812−null. Additionally, each of the commits is an ancestor of every commit
813−numbered greater than this one. Thus, each is a descendant of every
814−commit numbered less than this one. Since it is true that the date in
815−each commit (`t:yaki`) is no earlier than that of each of its parents,
816−the numbered commits are totally ordered in the same way by both
817−pedigree and date. Of course, not every commit is numbered. If that
818−sounds too complicated to you, don't worry about it. It basically
819−behaves exactly as you would expect.
821−`lab` is a map of textual labels to numbered commits. Note that labels
822−can only be applied to numbered commits. Labels must be unique across a
825−#### `++ankh`, filesystem node
828− ++ ankh :: fs node (new)
829− $: p=cash :: recursive hash
830− q=(unit ,[p=cash q=*]) :: file
831− r=(map ,@ta ankh) :: folders
835−This is a single node in the filesystem. This may be file or a directory
836−or both. In earth filesystems, a node is a file xor a directory. On
837−mars, we're inclusive, so a node is a file ior a directory.
839−`p` is a recursive hash that depends on the contents of the this file or
840−directory and on any children.
842−`q` is the contents of this file, if any. `p.q` is a hash of the
843−contents while `q.q` is the data itself.
845−`r` is the set of children of this node. In the case of a pure file,
846−this is empty. The keys are the names of the children and the values
847−are, recursively, the nodes themselves.
849−#### `++cash`, ankh hash
852− ++ cash ,@uvH :: ankh hash
855−This is a 128-bit hash of an ankh. These are mostly stored within ankhs
856−themselves, and they are used to check for changes in possibly-deep
859−#### `++rung`, filesystem per neighbor ship
862− ++ rung $: rus=(map desk rede) :: neighbor desks
866−This is the filesystem of a neighbor ship. The keys to this map are all
867−the desks we know about on their ship.
869−#### `++rede`, desk state
872− ++ rede :: universal project
873− $: lim=@da :: complete to
874− qyx=cult :: subscribers
875− ref=(unit rind) :: outgoing requests
876− dom=dome :: revision state
880−This is our knowledge of the state of a desk, either foreign or
883−`lim` is the date of the last full update. We only respond to requests
884−for stuff before this time.
886−`qyx` is the list of subscribers to this desk. For domestic desks, this
887−is simply `p:dojo`, all subscribers to the desk, while in foreign desks
888−this is all the subscribers from our ship to the foreign desk.
890−`ref` is the request manager for the desk. For domestic desks, this is
891−null since we handle requests ourselves.
893−`dom` is the actual data in the desk.
895−#### `++rind`, request manager
898− ++ rind :: request manager
899− $: nix=@ud :: request index
900− bom=(map ,@ud ,[p=duct q=rave]) :: outstanding
901− fod=(map duct ,@ud) :: current requests
902− haw=(map mood (unit)) :: simple cache
906−This is the request manager for a foreign desk.
908−`nix` is one more than the index of the most recent request. Thus, it is
909−the next available request number.
911−`bom` is the set of outstanding requests. The keys of this map are some
912−subset of the numbers between 0 and one less than `nix`. The members of
913−the map are exactly those requests that have not yet been fully
916−`fod` is the same set as `bom`, but from a different perspective. In
917−particular, the values of `fod` are the same as the values of `bom`, and
918−the `p` out of the values of `bom` are the same as the keys of `fod`.
919−Thus, we can map ducts to their associated request number and `++rave:clay`,
920−and we can map numbers to their associated duct and `++rave:clay`.
922−`haw` is a map from simple requests to their values. This acts as a
923−cache for requests that have already been made. Thus, the second request
924−for a particular `++mood:clay` is nearly instantaneous.
926−#### `++rang:clay`, data store
929− ++ rang $: hut=(map tako yaki) ::
930− lat=(map lobe blob) ::
934−This is a set of data keyed by hash. Thus, this is where the "real" data
935−is stored, but it is only meaningful if we know the hash of what we're
938−`hut` is a map from hashes to commits. We often get the hashes from
939−`hit:dome:clay`, which keys them by logical id. Not every commit has an id.
941−`lat` is a map from hashes to the actual data. We often get the hashes
942−from a `++yaki`, a commit, which references this map to get the data.
943−There is no `++blob:clay` in any `++yaki:clay`. They are only accessible through
946−#### `++tako:clay`, commit reference
949− ++ tako ,@ :: yaki ref
952−This is a hash of a `++yaki:clay`, a commit. These are most notably used as
953−the keys in `hut:rang:clay`, where they are associated with the actual
954−`++yaki:clay`, and as the values in `hit:dome:clay`, where sequential ids are
955−associated with these.
957−#### `++yaki:clay`, commit
960− ++ yaki ,[p=(list tako) q=(map path lobe) r=tako t=@da] :: commit
963−This is a single commit.
965−`p` is a list of the hashes of the parents of this commit. In most
966−cases, this will be a single commit, but in a merge there may be more
967−parents. In theory, there may be an arbitrary number of parents, but in
968−practice merges have exactly two parents. This may change in the future.
969−For commit 1, there is no parent.
971−`q` is a map of the paths on a desk to the data at that location. If you
972−understand what a `++lobe:clay` and a `++blob:clay` is, then the type signature
973−here tells the whole story.
975−`r` is the hash associated with this commit.
977−`t` is the date at which this commit was made.
979−#### `++lobe:clay`, data reference
982− ++ lobe ,@ :: blob ref
985−This is a hash of a `++blob:clay`. These are most notably used in `lat:rang:clay`,
986−where they are associated with the actual `++blob:clay`, and as the values in
987−`q:yaki:clay`, where paths are associated with their data in a commit.
989−#### `++blob:clay`, data
992− ++ blob $% [%delta p=lobe q=lobe r=udon] :: delta on q
993− [%direct p=lobe q=* r=umph] ::
994− [%indirect p=lobe q=* r=udon s=lobe] ::
998−This is a node of data. In every case, `p` is the hash of the blob.
1000−`%delta` is the case where we define the data by a delta on other data.
1001−In practice, the other data is always the previous commit, but nothing
1002−depends on this. `q` is the hash of the parent blob, and `r` is the
1005−`%direct` is the case where we simply have the data directly. `q` is the
1006−data itself, and `r` is any preprocessing instructions. These almost
1007−always come from the creation of a file.
1009−`%indirect` is both of the preceding cases at once. `q` is the direct
1010−data, `r` is the delta, and `s` is the parent blob. It should always be
1011−the case that applying `r` to `s` gives the same data as `q` directly
1012−(with the prepreprocessor instructions in `p.r`). This exists purely for
1013−performance reasons. This is unused, at the moment, but in general these
1014−should be created when there are a long line of changes so that we do
1015−not have to traverse the delta chain back to the creation of the file.
1017−#### `++udon`, abstract delta
1020− ++ udon :: abstract delta
1021− $: p=umph :: preprocessor
1023− $% [%a p=* q=*] :: trivial replace
1024− [%b p=udal] :: atomic indel
1025− [%c p=(urge)] :: list indel
1026− [%d p=upas q=upas] :: tree edit
1031−This is an abstract change to a file. This is a superset of what would
1032−normally be called diffs. Diffs usually refer to changes in lines of
1033−text while we have the ability to do more interesting deltas on
1034−arbitrary data structures.
1036−`p` is any preprocessor instructions.
1038−`%a` refers to the trival delta of a complete replace of old data with
1041−`%b` refers to changes in an opaque atom on the block level. This has
1042−very limited usefulness, and is not used at the moment.
1044−`%c` refers to changes in a list of data. This is often lines of text,
1045−which is your classic diff. We, however, will work on any list of data.
1047−`%d` refers to changes in a tree of data. This is general enough to
1048−describe changes to any hoon [noun](/docs/glossary/noun/), but often more special-purpose delta
1049−should be created for different content types. This is not used at the
1050−moment, and may in fact be unimplemented.
1052−#### `++urge`, list change
1055− ++ urge |*(a=_,* (list (unce a))) :: list change
1058−This is a parametrized type for list changes. For example, `(urge ,@t)`
1059−is a list change for lines of text.
1061−#### `++unce`, change part of a list.
1064− ++ unce |* a=_,* :: change part
1065− $% [%& p=@ud] :: skip[copy]
1066− [%| p=(list a) q=(list a)] :: p -> q[chunk]
1070−This is a single change in a list of elements of type `a`. For example,
1071−`(unce ,@t)` is a single change in a lines of text.
1073−`%&` means the next `p` lines are unchanged.
1075−`%|` means the lines `p` have changed to `q`.
1077−#### `++umph`, preprocessing information
1080− ++ umph :: change filter
1081− $| $? %a :: no filter
1085− $% [%d p=@ud] :: blocklist
1089−This space intentionally left undocumented. This stuff will change once
1090−we get a well-typed clay.
1092−#### `++upas`, tree change
1095− ++ upas :: tree change (%d)
1096− $& [p=upas q=upas] :: cell
1097− $% [%0 p=axis] :: copy old
1098− [%1 p=*] :: insert new
1099− [%2 p=axis q=udon] :: mutate!
1103−This space intentionally left undocumented. This stuff is not known to
1104−work, and will likely change when we get a well-typed clay. Also, this
1105−is not a complicated type; it is not difficult to work out the meaning.
1107−#### `++nori:clay`, repository action
1110− ++ nori :: repository action
1111− $% [& q=soba] :: delta
1112− [| p=@tas] :: label
1116−This describes a change that we are asking clay to make to the desk.
1117−There are two kinds of changes that may be made: we can modify files or
1118−we can apply a label to a commit.
1120−In the `|` case, we will simply label the current commit with the given
1121−label. In the `&` case, we will apply the given changes.
1123−#### `++soba:clay`, delta
1126− ++ soba ,[p=cart q=(list ,[p=path q=miso])] :: delta
1129−This describes a set of changes to make to a desk. The `cart` is simply
1130−a pair of the old hash and the new hash of the desk. The list is a list
1131−of changes keyed by the file they're changing. Thus, the paths are paths
1132−to files to be changed while `miso` is a description of the change
1135−#### `++miso:clay`, ankh delta
1138− ++ miso :: ankh delta
1139− $% [%del p=*] :: delete
1140− [%ins p=*] :: insert
1141− [%mut p=udon] :: mutate
1145−There are three kinds of changes that may be made to a node in a desk.
1146−We can insert a file, in which case `p` is the contents of the new file.
1147−We can delete a file, in which case `p` is the contents of the old file.
1148−Finally, we can mutate that file, in which case the `udon` describes the
1149−changes we are applying to the file.
1151−#### `++mizu:clay`, merged state
1154− ++ mizu ,[p=@u q=(map ,@ud tako) r=rang] :: new state
1157−This is the input to the `%merg` kiss, which allows us to perform a
1158−merge. The `p` is the number of the new head commit. The `q` is a map
1159−from numbers to commit hashes. This is all the new numbered commits that
1160−are to be inserted. The keys to this should always be the numbers from
1161−`let.dom` plus one to `p`, inclusive. The `r` is the maps of all the new
1162−commits and data. Since these are merged into the current state, no old
1163−commits or data need be here.
1165−#### `++riff:clay`, request/desist
1168− ++ riff ,[p=desk q=(unit rave)] :: request/desist
1171−This represents a request for data about a particular desk. If `q`
1172−contains a `rave`, then this opens a subscription to the desk for that
1173−data. If `q` is null, then this tells clay to cancel the subscription
1176−#### `++riot:clay`, response
1179− ++ riot (unit rant) :: response/complete
1182−A riot is a response to a subscription. If null, the subscription has
1183−been completed, and no more responses will be sent. Otherwise, the
1184−`rant` is the produced data.
1186−#### `++rant:clay`, response data
1189− ++ rant :: namespace binding
1190− $: p=[p=care q=case r=@tas] :: clade release book
1196−This is the data at a particular node in the filesystem. `p.p` specifies
1197−the type of data that was requested (and is produced). `q.p` gives the
1198−specific version reported (since a range of versions may be requested in
1199−a subscription). `r.p` is the desk. `q` is the path to the filesystem
1200−node. `r` is the data itself (in the format specified by `p.p`).
1202−### `++nako`, subscription response data
1205− ++ nako $: gar=(map ,@ud tako) :: new ids
1207− lar=(set yaki) :: new commits
1208− bar=(set blob) :: new content
1212−This is the data that is produced by a request for a range of revisions
1213−of a desk. This allows us to easily keep track of a remote repository --
1214−all the new information we need is contained in the `nako`.
1216−`gar` is a map of the revisions in the range to the hash of the commit
1217−at that revision. These hashes can be used with `hut:rang:clay` to find the
1220−`let` is either the last revision number in the range or the most recent
1221−revision number, whichever is smaller.
1223−`lar` is the set of new commits, and `bar` is the set of new content.
1227−As with all vanes, there are exactly two ways to interact with clay.
1228−`%clay` exports a namespace accessible through `.^`, which is described
1229−above under `++care:clay`. The primary way of interacting with clay, though,
1230−is by sending kisses and receiving gifts.
1233− ++ gift :: out result <-$
1234− $% [%ergo p=@p q=@tas r=@ud] :: version update
1235− [%note p=@tD q=tank] :: debug message
1236− [%writ p=riot] :: response
1238− ++ kiss :: in request ->$
1239− $% [%info p=@p q=@tas r=nori] :: internal edit
1240− [%ingo p=@p q=@tas r=nori] :: internal noun edit
1241− [%init p=@p] :: report install
1242− [%into p=@p q=@tas r=nori] :: external edit
1243− [%invo p=@p q=@tas r=nori] :: external noun edit
1244− [%merg p=@p q=@tas r=mizu] :: internal change
1245− [%wart p=sock q=@tas r=path s=*] :: network request
1246− [%warp p=sock q=riff] :: file request
1250−There are only a small number of possible kisses, so it behooves us to
1251−describe each in detail.
1254− $% [%info p=@p q=@tas r=nori] :: internal edit
1256− [%into p=@p q=@tas r=nori] :: external edit
1259−These two kisses are nearly identical. At a high level, they apply
1260−changes to the filesystem. Whenever we add, remove, or edit a file, one
1261−of these cards is sent. The `p` is the ship whose filesystem we're
1262−trying to change, the `q` is the desk we're changing, and the `r` is the
1263−request change. For the format of the requested change, see the
1264−documentation for `++nori:clay` above.
1266−When a file is changed in the unix filesystem, vere will send a `%into`
1267−kiss. This tells clay that the duct over which the kiss was sent is the
1268−duct that unix is listening on for changes. From within Arvo, though, we
1269−should never send a `%into` kiss. The `%info` kiss is exactly identical
1270−except it does not reset the duct.
1273− [%ingo p=@p q=@tas r=nori] :: internal noun edit
1275− [%invo p=@p q=@tas r=nori] :: external noun edit
1278−These kisses are currently identical to `%info` and `%into`, though this
1279−will not always be the case. The intent is for these kisses to allow
1280−typed changes to clay so that we may store typed data. This is currently
1284− [%init p=@p] :: report install
1287−Init is called when a ship is started on our pier. This simply creates a
1288−default `room` to go into our `raft`. Essentially, this initializes the
1289−filesystem for a ship.
1292− [%merg p=@p q=@tas r=mizu] :: internal change
1295−This is called to perform a merge. This is most visibly called by
1296−:update to update the filesystem of the current ship to that of its
1297−sein. The `p` and `q` are as in `%info`, and the `r` is the description
1298−of the merge. See `++mizu:clay` above.
1301−`XX [%wake ~] :: timer activate XX`
1303−XX This card is sent by unix at the time specified by `++doze`. This
1304−time is XX usually the closest time specified in a subscription request.
1305−When `%wake` is XX called, we update our subscribers if there have been
1309− [%wart p=sock q=@tas r=path s=*] :: network request
1312−This is a request that has come across the network for a particular
1313−file. When another ship asks for a file from us, that request comes to
1314−us in the form of a `%wart` kiss. This is handled by trivially turning
1318− [%warp p=sock q=riff] :: file request
1321−This is a request for information about a particular desk. This is, in
1322−its most general form, a subscription, though in many cases it is the
1323−trivial case of a subscription -- a read. See `++riff:clay` for the format of
1326−## Lifecycle of a Local Read
1328−There are two real types of interaction with a filesystem: you can read,
1329−and you can write. We'll describe each process, detailing both the flow
1330−of control followed by the kernel and the algorithms involved. The
1331−simpler case is that of the read, so we'll begin with that.
1333−When a vane or an application wishes to read a file from the filesystem,
1334−it sends a `%warp` kiss, as described above. Of course, you may request
1335−a file on another ship and, being a global filesystem, clay will happily
1336−produce it for you. That code pathway will be described in another
1337−section; here, we will restrict ourselves to examining the case of a
1338−read from a ship on our own pier.
1340−The kiss can request either a single version of a file node or a range
1341−of versions of a desk. Here, we'll deal only with a request for a single
1344−As in all vanes, a kiss enters clay via a call to `++call`. Scanning
1345−through the arm, we quickly see where `%warp` is handled.
1348− ?: =(p.p.q.hic q.p.q.hic)
1349− =+ une=(un p.p.q.hic now ruf)
1350− =+ wex=(di:une p.q.q.hic)
1354− (eave:wex hen u.q.q.q.hic)
1357− [-.woo abet:(pish:une p.q.q.hic +.woo ran.wao)]
1360−We're following the familar patern of producing a list of moves and an
1361−updated state. In this case, the state is `++raft`.
1363−We first check to see if the sending and receiving ships are the same.
1364−If they're not, then this is a request for data on another ship. We
1365−describe that process later. Here, we discuss only the case of a local
1368−At a high level, the call to `++un` sets up the core for the domestic
1369−ship that contains the files we're looking for. The call to `++di` sets
1370−up the core for the particular desk we're referring to.
1372−After this, we perform the actual request. If there is no rave in the
1373−riff, then that means we are cancelling a request, so we call
1374−`++ease:de`. Otherwise, we start a subscription with `++eave:de`. We
1375−call `++abet:de` to resolve our various types of output into actual
1376−moves. We produce the moves we found above and the `++un` core resolved
1377−with `++pish:un` (putting the modified desk in the room) and `++abet:un`
1378−(putting the modified room in the raft).
1380−Much of this is fairly straightforward, so we'll only describe `++ease`,
1381−`++eave`, and `++abet:de`. Feel free to look up the code to the other
1382−steps -- it should be easy to follow.
1384−Although it's called last, it's usually worth examining `++abet` first,
1385−since it defines in what ways we can cause side effects. Let's do that,
1386−and also a few of the lines at the beginning of `++de`.
1389− =| yel=(list ,[p=duct q=gift])
1390− =| byn=(list ,[p=duct q=riot])
1391− =| vag=(list ,[p=duct q=gift])
1392− =| say=(list ,[p=duct q=path r=ship s=[p=@ud q=riff]])
1393− =| tag=(list ,[p=duct q=path c=note])
1396− ^- [(list move) rede]
1400− |=([a=duct b=gift] [hun %give b])
1403− |=([a=duct b=riot] [a %give [%writ b]])
1406− |=([a=duct b=gift] [a %give b])
1409− |= [a=duct b=path c=ship d=[p=@ud q=riff]]
1411− [%pass b %a %want [who c] [%q %re p.q.d (scot %ud p.d) ~] q.d]
1414− |=([a=duct b=path c=note] [a %pass b c])
1418−This is very simple code. We see there are exactly five different kinds
1419−of side effects we can generate.
1421−In `yel` we put gifts that we wish to be sent along the `hun:room` duct
1422−to dill. See the documentation for `++room` above. This is how we
1423−display messages to the terminal.
1425−In `byn` we put riots that we wish returned to subscribers. Recall that
1426−a riot is a response to a subscription. These are returned to our
1427−subscribers in the form of a `%writ` gift.
1429−In `vag` we put gifts along with the ducts on which to send them. This
1430−allows us to produce arbitrary gifts, but in practice this is only used
1431−to produce `%ergo` gifts.
1433−In `say` we put messages we wish to pass to ames. These messages are
1434−used to request information from clay on other piers. We must provide
1435−not only the duct and the request (the riff), but also the return path,
1436−the other ship to talk to, and the sequence number of the request.
1438−In `tag` we put arbitrary notes we wish to pass to other vanes. For now,
1439−the only notes we pass here are `%wait` and `%rest` to the timer vane.
1441−Now that we know what kinds of side effects we may have, we can jump
1442−into the handling of requests.
1445− ++ ease :: release request
1449− =+ rov=(~(got by qyx) hen)
1450− =. qyx (~(del by qyx) hen)
1451− (mabe rov (cury best hen))
1452− =. qyx (~(del by qyx) hen)
1454− =+ nux=(~(get by fod.u.ref) hen)
1457− say [[hen [(scot %ud u.nux) ~] for [u.nux syd ~]] say]
1458− fod.u.ref (~(del by fod.u.ref) hen)
1459− bom.u.ref (~(del by bom.u.ref) u.nux)
1463−This is called when we're cancelling a subscription. For domestic desks,
1464−`ref` is null, so we're going to cancel any timer we might have created.
1465−We first delete the duct from our map of requests, and then we call
1466−`++mabe` with `++best` to send a `%rest` kiss to the timer vane if we
1467−have started a timer. We'll describe `++best` and `++mabe` momentarily.
1469−Although we said we're not going to talk about foreign requests yet,
1470−it's easy to see that for foreign desks, we cancel any outstanding
1471−requests for this duct and send a message over ames to the other ship
1472−telling them to cancel the subscription.
1476− |= [hen=duct tym=@da]
1477− %_(+> tag :_(tag [hen /tyme %t %rest tym]))
1480−This simply pushes a `%rest` note onto `tag`, from where it will be
1481−passed back to arvo to be handled. This cancels the timer at the given
1482−duct (with the given time).
1485− ++ mabe :: maybe fire function
1486− |* [rov=rove fun=$+(@da _+>.^$)]
1493− ?. ?=(%da -.q.p.rov) ~
1498− ?. ?=(%da -.p.mot) ~
1499− ?.((lth now p.p.mot) ~ [~ p.p.mot])
1500− ?. ?=(%da -.q.mot) ~
1501− ?.((lth now p.q.mot) [~ now] [~ p.q.mot])
1505−This decides whether the given request can only be satsified in the
1506−future. In that case, we call the given function with the time in the
1507−future when we expect to have an update to give to this request. This is
1508−called with `++best` to cancel timers and with `++bait` to start them.
1510−For single requests, we have a time if the request is for a particular
1511−time (which is assumed to be in the future). For ranges of requests, we
1512−check both the start and end cases to see if they are time cases. If so,
1513−we choose the earlier time.
1515−If any of those give us a time, then we call the given funciton with the
1518−The more interesting case is, of course, when we're not cancelling a
1519−subscription but starting one.
1522− ++ eave :: subscribe
1523− |= [hen=duct rav=rave]
1527− ?: &(=(p.p.rav %u) !=(p.q.p.rav now))
1528− ~& [%clay-fail p.q.p.rav %now now]
1530− =+ ver=(aver p.rav)
1535− (blab hen p.rav u.u.ver)
1538−There are two types of subscriptions -- either we're requesting a single
1539−file or we're requesting a range of versions of a desk. We'll dicuss the
1542−First, we check that we're not requesting the `rang` from any time other
1543−than the present. Since we don't store that information for any other
1544−time, we can't produce it in a referentially transparent manner for any
1545−time other than the present.
1547−Then, we try to read the requested `mood:clay` `p.rav`. If we can't access
1548−the request data right now, we call `++duce` to put the request in our
1549−queue to be satisfied when the information becomes available.
1551−This case occurs when we make a request for a case whose (1) date is
1552−after the current date, (2) number is after the current number, or (3)
1553−label is not yet used.
1556− ++ duce :: produce request
1557− |= [hen=duct rov=rove]
1559− =. qyx (~(put by qyx) hen rov)
1561− (mabe rov (cury bait hen))
1562− |- ^+ +>+.$ :: XX why?
1565− ?. ?=([%& %v *] rav) rav
1566− [%| [%ud let.dom] `case`q.p.rav r.p.rav]
1569− say [[hen [(scot %ud inx) ~] for [inx syd ~ vaw]] say]
1570− nix.u.ref +(nix.u.ref)
1571− bom.u.ref (~(put by bom.u.ref) inx [hen vaw])
1572− fod.u.ref (~(put by fod.u.ref) hen inx)
1576−The code for `++duce` is nearly the exact inverse of `++ease`, which in
1577−the case of a domestic desk is very simple -- we simply put the duct and
1578−rave into `qyx` and possibly start a timer with `++mabe` and `++bait`.
1579−Recall that `ref` is null for domestic desks and that `++mabe` fires the
1580−given function with the time we need to be woken up at, if we need to be
1581−woken up at a particular time.
1585− |= [hen=duct tym=@da]
1586− %_(+> tag :_(tag [hen /tyme %t %wait tym]))
1589−This sets an alarm by sending a `%wait` card with the given time to the
1592−Back in `++eave`, if `++aver` returned `[~ ~]`, then we cancel the
1593−subscription. This occurs when we make (1) a `%x` request for a file
1594−that does not exist, (2) a `%w` request with a case that is not a
1595−number, or (3) a `%w` request with a nonempty path. The `++blub` is
1596−exactly what you would expect it to be.
1599− ++ blub :: ship stop
1601− %_(+> byn [[hen ~] byn])
1604−We notify the duct that we're cancelling their subscription since it
1607−Otherwise, we have received the desired information, so we send it on to
1608−the subscriber with `++blab`.
1611− ++ blab :: ship result
1612− |= [hen=duct mun=mood dat=*]
1614− +>(byn [[hen ~ [p.mun q.mun syd] r.mun dat] byn])
1617−The most interesting arm called in `++eave` is, of course, `++aver`,
1618−where we actually try to read the data.
1623− ^- (unit (unit ,*))
1624− ?: &(=(p.mun %u) !=(p.q.mun now)) :: prevent bad things
1625− ~& [%clay-fail p.q.mun %now now]
1627− =+ ezy=?~(ref ~ (~(get by haw.u.ref) mun))
1629− =+ nao=(~(case-to-aeon ze lim dom ran) q.mun)
1630− ?~(nao ~ [~ (~(read-at-aeon ze lim dom ran) u.nao mun)])
1633−We check immediately that we're not requesting the `rang` for any time
1634−other than the present.
1636−If this is a foreign desk, then we check our cache for the specific
1637−request. If either this is a domestic desk or we don't have the request
1638−in our cache, then we have to actually go read the data from our dome.
1640−We need to do two things. First, we try to find the number of the commit
1641−specified by the given case, and then we try to get the data there.
1643−Here, we jump into `arvo/zuse.hoon`, which is where much of the
1644−algorithmic code is stored, as opposed to the clay interface, which is
1645−stored in `arvo/clay.hoon`. We examine `++case-to-aeon:ze`.
1648− ++ case-to-aeon :: case-to-aeon:ze
1649− |= lok=case :: act count through
1653− ?: (gth p.lok lim) ~
1655− ?: =(0 let) [~ 0] :: avoid underflow
1664− %tas (~(get by lab) p.lok)
1665− %ud ?:((gth p.lok let) ~ [~ p.lok])
1669−We handle each type of `case:clay` differently. The latter two types are
1672−If we're requesting a revision by label, then we simply look up the
1673−requested label in `lab` from the given dome. If it exists, that is our
1674−aeon; else we produce null, indicating the requested revision does not
1677−If we're requesting a revision by number, we check if we've yet reached
1678−that number. If so, we produce the number; else we produce null.
1680−If we're requesting a revision by date, we check first if the date is in
1681−the future, returning null if so. Else we start from the most recent
1682−revision and scan backwards until we find the first revision committed
1683−before that date, and we produce that. If we requested a date before any
1684−revisions were committed, we produce `0`.
1686−The definitions of `++aeon-to-tako` and `++tako-to-yaki` are trivial.
1689− ++ aeon-to-tako ~(got by hit)
1691− ++ tako-to-yaki ~(got by hut) :: grab yaki
1694−We simply look up the aeon or tako in their respective maps (`hit` and
1697−Assuming we got a valid version number, `++aver` calls
1698−`++read-at-aeon:ze`, which reads the requested data at the given
1702− ++ read-at-aeon :: read-at-aeon:ze
1703− |= [oan=aeon mun=mood] :: seek and read
1705− ?: &(?=(%w p.mun) !?=(%ud -.q.mun)) :: NB only for speed
1706− ?^(r.mun ~ [~ oan])
1707− (read:(rewind oan) mun)
1710−If we're requesting the revision number with a case other than by
1711−number, then we go ahead and just produce the number we were given.
1712−Otherwise, we call `++rewind` to rewind our state to the given revision,
1713−and then we call `++read` to get the requested information.
1716− ++ rewind :: rewind:ze
1717− |= oan=aeon :: rewind to aeon
1720− ?: (gth oan let) !! :: don't have version
1721− +>(ank (checkout-ankh q:(tako-to-yaki (aeon-to-tako oan))), let oan)
1724−If we're already at the requested version, we do nothing. If we're
1725−requesting a version later than our head, we are unable to comply.
1727−Otherwise, we get the hash of the commit at the number, and from that we
1728−get the commit itself (the yaki), which has the map of path to lobe that
1729−represents a version of the filesystem. We call `++checkout-ankh` to
1730−checkout the commit, and we replace `ank` in our context with the
1734− ++ checkout-ankh :: checkout-ankh:ze
1735− |= hat=(map path lobe) :: checkout commit
1738− %+ roll (~(tap by hat) ~)
1739− |= [[pat=path bar=lobe] ank=ankh]
1743− =+ zar=(lobe-to-noun bar)
1744− ank(q [~ (sham zar) zar])
1745− =+ nak=(~(get by r.ank) i.pat)
1747− r %+ ~(put by r.ank) i.pat
1748− $(pat t.pat, ank (fall nak _ankh))
1752−Twice we call `++cosh`, which hashes a commit, updating `p` in an
1753−`ankh`. Let's jump into that algorithm before we describe
1757− ++ cosh :: locally rehash
1758− |= ank=ankh :: NB v/unix.c
1759− ank(p rehash:(zu ank))
1762−We simply replace `p` in the hash with the `cash` we get from a call to
1766− ++ zu !: :: filesystem
1767− |= ank=ankh :: filesystem state
1768− =| myz=(list ,[p=path q=miso]) :: changes in reverse
1769− =| ram=path :: reverse path into
1771− ++ rehash :: local rehash
1773− %+ mix ?~(q.ank 0 p.u.q.ank)
1778− (shaf %dash (mix axe (shaf %dush (mix p.n.r.ank p.q.n.r.ank))))
1779− $(r.ank l.r.ank, axe (peg axe 2))
1780− $(r.ank r.r.ank, axe (peg axe 3))
1784−`++zu` is a core we set up with a particular filesystem node to traverse
1785−a checkout of the filesystem and access the actual data inside it. One
1786−of the things we can do with it is to create a recursive hash of the
1789−In `++rehash`, if this node is a file, then we xor the remainder of the
1790−hash with the hash of the contents of the file. The remainder of the
1791−hash is `0` if we have no children, else we descend into our children.
1792−Basically, we do a half SHA-256 of the xor of the axis of this child and
1793−the half SHA-256 of the xor of the name of the child and the hash of the
1794−child. This is done for each child and all the results are xored
1797−Now we return to our discussion of `++checkout-ankh`.
1799−We fold over every path in this version of the filesystem and create a
1800−great ankh out of them. First, we call `++lobe-to-noun` to get the raw
1801−data referred to be each lobe.
1804− ++ lobe-to-noun :: grab blob
1810−This converts a lobe into the raw data it refers to by first getting the
1811−blob with `++lobe-to-blob` and converting that into data with
1815− ++ lobe-to-blob ~(got by lat) :: grab blob
1818−This just grabs the blob that the lobe refers to.
1821− ++ blob-to-noun :: grab blob
1824− %delta (lump r.p (lobe-to-noun q.p))
1830−If we have either a direct or an indirect blob, then the data is stored
1831−right in the blob. Otherwise, we have to reconstruct it from the diffs.
1832−We do this by calling `++lump` on the diff in the blob with the data
1833−obtained by recursively calling the parent of this blob.
1836− ++ lump :: apply patch
1837− |= [don=udon src=*]
1839− ?+ p.don ~|(%unsupported !!)
1841− ?+ -.q.don ~|(%unsupported !!)
1843− %c (lurk ((hard (list)) src) p.q.don)
1844− %d (lure src p.q.don)
1848− =+ dst=(lore ((hard ,@) src))
1850− ?+ -.q.don ~|(%unsupported !!)
1851− %a ((hard (list ,@t)) q.q.don)
1852− %c (lurk dst p.q.don)
1857−This is defined in `arvo/hoon.hoon` for historical reasons which are
1858−likely no longer applicable. Since the `++umph` structure will likely
1859−change we convert clay to be a typed filesystem, we'll only give a
1860−high-level description of this process. If we have a `%a` udon, then
1861−we're performing a trivial replace, so we produce simply `q.q.don`. If
1862−we have a `%c` udon, then we're performing a list merge (as in, for
1863−example, lines of text). The merge is performed by `++lurk`.
1866− ++ lurk :: apply list patch
1867− |* [hel=(list) rug=(urge)]
1876− hel (slag p.i.rug hel)
1877− war (weld (flop (scag p.i.rug hel)) war)
1883− hel =+ gur=(flop p.i.rug)
1886− ?>(&(?=(^ hel) =(i.gur i.hel)) $(hel t.hel, gur t.gur))
1887− war (weld q.i.rug war)
1892−We accumulate our final result in `war`. If there's nothing more in our
1893−list of merge instructions (unces), we just reverse `war` and produce
1894−it. Otherwise, we process another unce. If the unce is of type `&`, then
1895−we have `p.i.rug` lines of no changes, so we just copy them over from
1896−`hel` to `war`. If the unice is of type `|`, then we verify that the
1897−source lines (in `hel`) are what we expect them to be (`p.i.rug`),
1898−crashing on failure. If they're good, then we append the new lines in
1899−`q.i.rug` onto `war`.
1901−And that's really it. List merges are pretty easy. Anyway, if you
1902−recall, we were discussing `++checkout-ankh`.
1905− ++ checkout-ankh :: checkout-ankh:ze
1906− |= hat=(map path lobe) :: checkout commit
1909− %+ roll (~(tap by hat) ~)
1910− |= [[pat=path bar=lobe] ank=ankh]
1914− =+ zar=(lobe-to-noun bar)
1915− ank(q [~ (sham zar) zar])
1916− =+ nak=(~(get by r.ank) i.pat)
1918− r %+ ~(put by r.ank) i.pat
1919− $(pat t.pat, ank (fall nak _ankh))
1923−If the path is null, then we calculate `zar`, the raw data at the path
1924−`pat` in this version. We produce the given ankh with the correct data.
1926−Otherwise, we try to get the child we're looking at from our parent
1927−ankh. If it's already been created, this succeeds; otherwise, we simply
1928−create a default blank ankh. We place ourselves in our parent after
1929−recursively computing our children.
1931−This algorithm really isn't that complicated, but it may not be
1932−immediately obvious why it works. An example should clear everything up.
1934−Suppose `hat` is a map of the following information.
1937− /greeting --> "customary upon meeting"
1938− /greeting/english --> "hello"
1939− /greeting/spanish --> "hola"
1940− /greeting/russian/short --> "привет"
1941− /greeting/russian/long --> "Здравствуйте"
1942− /farewell/russian --> "до свидания"
1945−Furthermore, let's say that we process them in this order:
1949− /greeting/russian/short
1950− /greeting/russian/long
1956−Then, the first path we process is `/greeting/english` . Since our path
1957−is not null, we try to get `nak`, but because our ankh is blank at this
1958−point it doesn't find anything. Thus, update our blank top-level ankh
1959−with a child `%greeting`. and recurse with the blank `nak` to create the
1960−ankh of the new child.
1962−In the recursion, we our path is `/english` and our ankh is again blank.
1963−We try to get the `english` child of our ankh, but this of course fails.
1964−Thus, we update our blank `/greeting` ankh with a child `english`
1965−produced by recursing.
1967−Now our path is null, so we call `++lobe-to-noun` to get the actual
1968−data, and we place it in the brand-new ankh.
1970−Next, we process `/greeting/russian/short`. Since our path is not null,
1971−we try to get the child named `%greeting`, which does exist since we
1972−created it earlier. We put modify this child by recursing on it. Our
1973−path is now `/russian/short`, so we look for a `%russian` child in our
1974−`/greeting` ankh. This doesn't exist, so we add it by recursing. Our
1975−path is now `/short`, so we look for a `%short` child in our
1976−`/greeting/russian` ankh. This doesn't exist, so we add it by recursing.
1977−Now our path is null, so we set the contents of this node to `"привет"`,
1978−and we're done processing this path.
1980−Next, we process `/greeting/russian/long`. This proceeds similarly to
1981−the previous except that now the ankh for `/greeting/russian` already
1982−exists, so we simply reuse it rather than creating a new one. Of course,
1983−we still must create a new `/greeting/russian/long` ankh.
1985−Next, we process `/greeting`. This ankh already exists, so after we've
1986−recursed once, our path is null, and our ankh is not blank -- it already
1987−has two children (and two grandchildren). We don't touch those, though,
1988−since a node may be both a file and a directory. We just add the
1989−contents of the file -- "customary upon meeting" -- to the existing
1992−Next, we process `/greeting/spanish`. Of course, the `/greeting` ankh
1993−already exists, but it doesn't have a `%spanish` child, so we create
1994−that, taking care not to disturb the contents of the `/greeting` file.
1995−We put "hola" into the ankh and call it good.
1997−Finally, we process `/farewell/russian`. Here, the `/farewell` ankh
1998−doesn't exist, so we create it. Clearly the newly-created ankh doesn't
1999−have any children, so we have to add a `%russian` child, and in this
2000−child we put our last content -- "до свидания".
2002−We hope it's fairly obvious that the order we process the paths doesn't
2003−affect the final ankh tree. The tree will be constructed in a very
2004−different order depending on what order the paths come in, but the
2005−resulting tree is independent of order.
2007−At any rate, we were talking about something important, weren't we? If
2008−you recall, that concludes our discussion of `++rewind`, which was
2009−called from `++read-at-aeon`. In summary, `++rewind` returns a context
2010−in which our current state is (very nearly) as it was when the specified
2011−version of the desk was the head. This allows `++read-at-aeon` to call
2012−`++read` to read the requested information.
2016− |= mun=mood :: read at point
2019− [~ `dome`+<+<.read]
2020− ?: &(?=(%w p.mun) !?=(%ud -.q.mun))
2021− ?^(r.mun ~ [~ let])
2027− ?^(r.mun ~ [~ [t.yak (forge-nori yak)]])
2028− ::?> ?=(^ hit) ?^(r.mun ~ [~ i.hit]) :: what do?? need [@da nori]
2029− (query(ank ank:(descend-path:(zu ank) r.mun)) p.mun)
2032−If we're requesting the dome, then we just return that immediately.
2034−If we're requesting the revision number of the desk and we're not
2035−requesting it by number, then we just return the current number of this
2036−desk. Note of course that this was really already handled in
2039−If we're requesting a `%w` with a specific revision number, then we do
2040−something or other with the commit there. It's kind of weird, and it
2041−doesn't seem to work, so we'll ignore this case.
2043−Otherwise, we descend into the ankh tree to the given path with
2044−`++descend-path:zu`, and then we handle specific request in `++query`.
2047− ++ descend-path :: descend recursively
2050− ?~(way +> $(way t.way, +> (descend i.way)))
2053−This is simple recursion down into the ankh tree. `++descend` descends
2054−one level, so this will eventually get us down to the path we want.
2057− ++ descend :: descend
2060− =+ you=(~(get by r.ank) lol)
2061− +>.$(ram [lol ram], ank ?~(you [*cash ~ ~] u.you))
2064−`ram` is the path that we're at, so to descend one level we push the
2065−name of this level onto that path. We update the ankh with the correct
2066−one at that path if it exists; else we create a blank one.
2068−Once we've decscended to the correct level, we need to actually deal
2072− ++ query :: query:ze
2073− |= ren=?(%u %v %x %y %z) :: endpoint query
2076− %u [~ `rang`+<+>.query]
2077− %v [~ `dome`+<+<.query]
2078− %x ?~(q.ank ~ [~ q.u.q.ank])
2084−Now that everything's set up, it's really easy. If they're requesting
2085−the rang, dome, or ankh, we give it to them. If the contents of a file,
2086−we give it to them if it is in fact a file. If the `arch`, then we
2087−calculate it with `++as-arch`.
2090− ++ as-arch :: as-arch:ze
2091− ^- arch :: arch report
2093− ?~(q.ank ~ [~ p.u.q.ank])
2094− |- ^- (map ,@ta ,~)
2096− [[p.n.r.ank ~] $(r.ank l.r.ank) $(r.ank r.r.ank)]
2099−This very simply strips out all the "real" data and returns just our own
2100−hash, the hash of the file contents (if we're a file), and a map of the
2101−names of our immediate children.
2103−## Lifecycle of a Local Subscription
2105−A subscription to a range of revisions of a desk initially follows the
2106−same path that a single read does. In `++aver`, we checked the head of
2107−the given rave. If the head was `&`, then it was a single request, so we
2108−handled it above. If `|`, then we handle it with the following code.
2111− =+ nab=(~(case-to-aeon ze lim dom ran) p.p.rav)
2113− ?> =(~ (~(case-to-aeon ze lim dom ran) q.p.rav))
2114− (duce hen (rive rav))
2115− =+ huy=(~(case-to-aeon ze lim dom ran) q.p.rav)
2116− ?: &(?=(^ huy) |((lth u.huy u.nab) &(=(0 u.huy) =(0 u.nab))))
2118− =+ top=?~(huy let.dom u.huy)
2119− =+ sar=(~(lobes-at-path ze lim dom ran) u.nab r.p.rav)
2120− =+ ear=(~(lobes-at-path ze lim dom ran) top r.p.rav)
2123− =+ fud=(~(make-nako ze lim dom ran) u.nab top)
2124− (bleb hen u.nab fud)
2129− (duce hen `rove`[%| ptr q.p.rav r.p.rav ear])
2133−Recall that `++case-to-aeon:ze` produces the revision number that a case
2134−corresponds to, if it corresponds to any. If it doesn't yet correspond
2135−to a revision, then it produces null.
2137−Thus, we first check to see if we've even gotten to the beginning of the
2138−range of revisions requested. If not, then we assert that we haven't yet
2139−gotten to the end of the range either, because that would be really
2140−strange. If not, then we immediately call `++duce`, which, if you
2141−recall, for a local request, simply puts this duct and rove into our
2142−cult `qyx`, so that we know who to respond to when the revision does
2145−If we've already gotten to the first revision, then we can produce some
2146−content immediately. If we've also gotten to the final revision, and
2147−that revision is earlier than the start revision, then it's a bad
2148−request and we call `++blub`, which tells the subscriber that his
2149−subscription will not be satisfied.
2151−Otherwise, we find the data at the given path at the beginning of the
2152−subscription and at the last available revision in the subscription. If
2153−they're the same, then we don't send a notification. Otherwise, we call
2154−`++gack`, which creates the `++nako` we need to produce. We call
2155−`++bleb` to actually produce the information.
2157−If we already have the last requested revision, then we also tell the
2158−subscriber with `++blub` that the subscription will receive no further
2161−If there will be more revisions in the subscription, then we call
2162−`++duce`, adding the duct to our subscribers. We modify the rove to
2163−start at the next revision since we've already handled all the revisions
2166−We glossed over the calls to `++lobes-at-path`, `++make-nako`, and
2167−`++bleb`, so we'll get back to those right now. `++bleb` is simple, so
2168−we'll start with that.
2171− ++ bleb :: ship sequence
2172− |= [hen=duct ins=@ud hip=nako]
2174− (blab hen [%w [%ud ins] ~] hip)
2177−We're given a duct, the beginning revision number, and the nako that
2178−contains the updates since that revision. We use `++blab` to produce
2179−this result to the subscriber. The case is `%w` with a revision number
2180−of the beginning of the subscription, and the data is the nako itself.
2182−We call `++lobes-at-path:ze` to get the data at the particular path.
2185− ++ lobes-at-path :: lobes-at-path:ze
2186− |= [oan=aeon pax=path] :: data at path
2201− $(p +.p, pax +.pax)
2205−At revision zero, the theoretical common revision between all
2206−repositories, there is no data, so we produce null.
2208−We get the list of paths (paired with their lobe) in the revision
2209−referred to by the given number and we keep only those paths which begin
2210−with `pax`. Converting to a map, we now have a map from the subpaths at
2211−the given path to the hash of their data. This is simple and efficient
2212−to calculate and compare to later revisions. This allows us to easily
2213−tell if a node or its children have changed.
2215−Finally, we will describe `++make-nako:ze`.
2218− ++ make-nako :: gack a through b
2220− ^- [(map aeon tako) aeon (set yaki) (set blob)]
2222− =- [(takos-to-yakis -<) (lobes-to-blobs ->)]
2223− %+ reachable-between-takos
2224− (~(get by hit) a) :: if a not found, a=0
2227− %- mo %+ skim (~(tap by hit) ~)
2229− &((gth p a) (lte p b))
2232−We need to produce four things -- the numbers of the new commits, the
2233−number of the latest commit, the new commits themselves, and the new
2236−The first is fairly easy to produce. We simply go over our map of
2237−numbered commits and produce all those numbered greater than `a` and not
2240−The second is even easier to produce -- `b` is clearly our most recent
2243−The third and fourth are slightly more interesting, though not too
2244−terribly difficult. First, we call `++reachable-between-takos`.
2247− ++ reachable-between-takos
2248− |= [a=(unit tako) b=tako] :: pack a through b
2249− ^- [(set tako) (set lobe)]
2252− (reachable-takos r:(tako-to-yaki u.a))
2253− =+ yak=`yaki`(tako-to-yaki b)
2254− %+ new-lobes-takos (new-lobes ~ sar) :: get lobes
2255− |- ^- (set tako) :: walk onto sar
2256− ?: (~(has in sar) r.yak)
2258− =+ ber=`(set tako)`(~(put in `(set tako)`~) `tako`r.yak)
2262− |= [yek=tako bar=(set tako)]
2264− ?: (~(has in bar) yek) :: save some time
2267− ^$(yak (tako-to-yaki yek))
2270−We take a possible starting commit and a definite ending commit, and we
2271−produce the set of commits and the set of data between them.
2273−We let `sar` be the set of commits reachable from `a`. If `a` is null,
2274−then obviously no commits are reachable. Otherwise, we call
2275−`++reachable-takos` to calculate this.
2278− ++ reachable-takos :: reachable
2279− |= p=tako :: XX slow
2281− =+ y=(tako-to-yaki p)
2282− =+ t=(~(put in _(set tako)) p)
2285− ?: (~(has in s) q) :: already done
2287− (~(uni in s) ^$(p q)) :: otherwise traverse
2290−We very simply produce the set of the given tako plus its parents,
2293−Back in `++reachable-between-takos`, we let `yak` be the yaki of `b`,
2294−the ending commit. With this, we create a set that is the union of `sar`
2295−and all takos reachable from `b`.
2297−We pass `sar` into `++new-lobes` to get all the lobes referenced by any
2298−tako referenced by `a`. The result is passed into `++new-lobes-takos` to
2299−do the same, but not recomputing those in already calculated last
2300−sentence. This produces the sets of takos and lobes we need.
2303− ++ new-lobes :: object hash set
2304− |= [b=(set lobe) a=(set tako)] :: that aren't in b
2306− %+ roll (~(tap in a) ~)
2307− |= [tak=tako bar=(set lobe)]
2309− =+ yak=(tako-to-yaki tak)
2310− %+ roll (~(tap by q.yak) ~)
2311− |= [[path lob=lobe] far=_bar]
2313− ?~ (~(has in b) lob) :: don't need
2315− =+ gar=(lobe-to-blob lob)
2317− %direct (~(put in far) lob)
2318− %delta (~(put in $(lob q.gar)) lob)
2319− %indirect (~(put in $(lob s.gar)) lob)
2323−Here, we're creating a set of lobes referenced in a commit in `a`. We
2324−start out with `b` as the initial set of lobes, so we don't need to
2325−recompute any of the lobes referenced in there.
2327−The algorithm is pretty simple, so we won't bore you with the details.
2328−We simply traverse every commit in `a`, looking at every blob referenced
2329−there, and, if it's not already in `b`, we add it to `b`. In the case of
2330−a direct blob, we're done. For a delta or an indirect blob, we
2331−recursively add every blob referenced within the blob.
2334− ++ new-lobes-takos :: garg & repack
2335− |= [b=(set lobe) a=(set tako)]
2336− ^- [(set tako) (set lobe)]
2337− [a (new-lobes b a)]
2340−Here, we just update the set of lobes we're given with the commits we're
2341−given and produce both sets.
2343−This concludes our discussion of a local subscription.
2345−## Lifecycle of a Foreign Read or Subscription
2347−Foreign reads and subscriptions are handled in much the same way as
2348−local ones. The interface is the same -- a vane or app sends a `%warp`
2349−kiss with the request. The difference is simply that the `sock` refers
2350−to the foreign ship.
2352−Thus, we start in the same place -- in `++call`, handling `%warp`.
2353−However, since the two side of the `sock` are different, we follow a
2357− =+ wex=(do now p.q.hic p.q.q.hic ruf)
2360− abet:(ease:wex hen)
2361− abet:(eave:wex hen u.q.q.q.hic)
2362− [-.woo (posh q.p.q.hic p.q.q.hic +.woo ruf)]
2365−If we compare this to how the local case was handled, we see that it's
2366−not all that different. We use `++do` rather than `++un` and `++de` to
2367−set up the core for the foreign ship. This gives us a `++de` core, so we
2368−either cancel or begin the request by calling `++ease` or `++eave`,
2369−exactly as in the local case. In either case, we call `++abet:de` to
2370−resolve our various types of output into actual moves, as described in
2371−the local case. Finally, we call `++posh` to update our raft, putting
2372−the modified rung into the raft.
2374−We'll first trace through `++do`.
2378− |= [now=@da [who=ship him=ship] syd=@tas ruf=raft]
2380− =+ rug=(~(get by hoy.ruf) him)
2381− ?^(rug u.rug *rung)
2383− =+ yit=(~(get by rus.rug) syd)
2384− ?^(yit u.yit `rede`[~2000.1.1 ~ [~ *rind] *dome])
2385− ((de now ~ ~) [who him] syd red ran.ruf)
2388−If we already have a rung for this foreign ship, then we use that.
2389−Otherwise, we create a new blank one. If we already have a rede in this
2390−rung, then we use that, otherwise we create a blank one. An important
2391−point to note here is that we let `ref` in the rede be `[~ *rind]`.
2392−Recall, for domestic desks `ref` is null. We use this to distinguish
2393−between foreign and domestic desks in `++de`.
2395−With this information, we create a `++de` core as usual.
2397−Although we've already covered `++ease` and `++eave`, we'll go through
2398−them quickly again, highlighting the case of foreign request.
2401− ++ ease :: release request
2405− =+ rov=(~(got by qyx) hen)
2406− =. qyx (~(del by qyx) hen)
2407− (mabe rov (cury best hen))
2408− =. qyx (~(del by qyx) hen)
2410− =+ nux=(~(get by fod.u.ref) hen)
2413− say [[hen [(scot %ud u.nux) ~] for [u.nux syd ~]] say]
2414− fod.u.ref (~(del by fod.u.ref) hen)
2415− bom.u.ref (~(del by bom.u.ref) u.nux)
2419−Here, we still remove the duct from our cult (we maintain a cult even
2420−for foreign desks), but we also need to tell the foreign desk to cancel
2421−our subscription. We do this by sending a request (by appending to
2422−`say`, which gets resolved in `++abet:de` to a `%want` kiss to ames) to
2423−the foreign ship to cancel the subscription. Since we don't anymore
2424−expect a response on this duct, we remove it from `fod` and `bom`, which
2425−are the maps between ducts, raves, and request sequence numbers.
2426−Basically, we remove every trace of the subscription from our request
2429−In the case of `++eave`, where we're creating a new request, everything
2430−is exactly identical to the case of the local request except `++duce`.
2431−We said that `++duce` simply puts the request into our cult. This is
2432−true for a domestic request, but distinctly untrue for foreign requests.
2435− ++ duce :: produce request
2436− |= [hen=duct rov=rove]
2438− =. qyx (~(put by qyx) hen rov)
2440− |- ^+ +>+.$ :: XX why?
2443− ?. ?=([%& %v *] rav) rav
2444− [%| [%ud let.dom] `case`q.p.rav r.p.rav]
2447− say [[hen [(scot %ud inx) ~] for [inx syd ~ vaw]] say]
2448− nix.u.ref +(nix.u.ref)
2449− bom.u.ref (~(put by bom.u.ref) inx [hen vaw])
2450− fod.u.ref (~(put by fod.u.ref) hen inx)
2454−If we have a request manager (i.e. this is a foreign desk), then we do
2455−the approximate inverse of `++ease`. We create a rave out of the given
2456−request and send it off to the foreign desk by putting it in `say`. Note
2457−that the rave is created to request the information starting at the next
2458−revision number. Since this is a new request, we put it into `fod` and
2459−`bom` to associate the request with its duct and its sequence number.
2460−Since we're using another sequence number, we must increment `nix`,
2461−which represents the next available sequence number.
2463−And that's really it for this side of the request. Requesting foreign
2464−information isn't that hard. Let's see what it looks like on the other
2465−side. When we get a request from another ship for information on our
2466−ship, that comes to us in the form of a `%wart` from ames.
2468−We handle a `%wart` in `++call`, right next to where we handle the
2474− =+ ryf=((hard riff) s.q.hic)
2477− :^ %pass [(scot %p p.p.q.hic) (scot %p q.p.q.hic) r.q.hic]
2479− [%warp [p.p.q.hic p.p.q.hic] ryf]
2483−Every request we receive should be of type `riff`, so we coerce it into
2484−that type. We just convert this into a new `%warp` kiss that we pass to
2485−ourself. This gets handled like normal, as a local request. When the
2486−request produces a value, it does so like normal as a `%writ`, which is
2487−returned to `++take` along the path we just sent it on.
2492− =+ our=(need (slaw %p i.tea))
2493− =+ him=(need (slaw %p i.t.tea))
2496− [%pass ~ %a [%want [our him] [%r %re %c t.t.tea] p.+.q.hin]]
2500−Since we encoded the ship we need to respond to in the path, we can just
2501−pass our `%want` back to ames, so that we tell the requesting ship about
2504−This comes back to the original ship as a `%waft` from ames, which comes
2505−into `++take`, right next to where we handled `%writ`.
2510− =+ syd=(need (slaw %tas i.tea))
2511− =+ inx=(need (slaw %ud i.t.tea))
2514− (knit:(do now p.+.q.hin syd ruf) [inx ((hard riot) q.+.q.hin)])
2517− [-.zot (posh q.p.+.q.hin syd +.zot ruf)]
2518− [mos ..^$(ran.ruf ran.zat)] :: merge in new obj
2521−This gets the desk and sequence number from the path the request was
2522−sent over. This determines exactly which request is being responded to.
2523−We call `++knit:de` to apply the changes to our local desk, and we call
2524−`++wake` to update our subscribers. Then we call `++abet:de` and
2525−`++posh` as normal (like in `++eave`).
2527−We'll examine `++knit` and `++wake`, in that order.
2530− ++ knit :: external change
2531− |= [inx=@ud rot=riot]
2535− =+ ruv=(~(get by bom.u.ref) inx)
2537− => ?. |(?=(~ rot) ?=(& -.q.u.ruv)) .
2539− bom.u.ref (~(del by bom.u.ref) inx)
2540− fod.u.ref (~(del by fod.u.ref) p.u.ruv)
2543− =+ rav=`rave`q.u.ruv
2546− ?.(&(?=(| -.rav) ?=(%da -.q.p.rav)) lim `@da`p.q.p.rav)
2549− ?. ?=(& -.rav) haw.u.ref
2550− (~(put by haw.u.ref) p.rav ~)
2552− ?< ?=(%v p.p.u.rot)
2554− (~(put by haw.u.ref) [p.p.u.rot q.p.u.rot q.u.rot] ~ r.u.rot)
2555− ?. ?=(%w p.p.u.rot) +>+.$
2557− =+ nez=[%w [%ud let.dom] ~]
2558− =+ nex=(~(get by haw.u.ref) nez)
2560− ?~ u.nex +>+.^$ :: should never happen
2561− =. +>+.^$ =+ roo=(edis ((hard nako) u.u.nex))
2562− ?>(?=(^ ref.roo) roo)
2564− haw.u.ref (~(del by haw.u.ref) nez)
2568−This is kind of a long [gate](/docs/glossary/gate/), but don't worry, it's not bad at all.
2570−First, we assert that we're not a domestic desk. That wouldn't make any
2573−Since we have the sequence number of the request, we can get the duct
2574−and rave from `bom` in our request manager. If we didn't actually
2575−request this data (or the request was canceled before we got it), then
2578−Else, we remove the request from `bom` and `fod` unless this was a
2579−subscription request and we didn't receive a null riot (which would
2580−indicate the last message on the subscription).
2582−Now, if we received a null riot, then if this was a subscription request
2583−by date, then we update `lim` in our request manager (representing the
2584−latest time at which we have complete information for this desk) to the
2585−date that was requested. If this was a single read request, then we put
2586−the result in our simple cache `haw` to make future requests faster.
2589−If we received actual data, then we put it into our cache `haw`. Unless
2590−it's a `%w` request, we're done.
2592−If it is a `%w` request, then we try to get the `%w` at our current head
2593−from the cache. In general, that should be the thing we just put in a
2594−moment ago, but that is not guaranteed. The most common case where this
2595−is different is when we receive desk updates out of order. At any rate,
2596−since we now have new information, we need to apply it to our local copy
2597−of the desk. We do so in `++edis`, and then we remove the stuff we just
2598−applied from the cache, since it's not really a true "single read", like
2599−what should really be in the cache.
2602− ++ edis :: apply subscription
2606− hit.dom (~(uni by hit.dom) gar.nak)
2608− lat.ran %+ roll (~(tap in bar.nak) ~)
2610− |= [sar=blob yeb=(map lobe blob)]
2611− =+ zax=(blob-to-lobe sar)
2612− %+ ~(put by yeb) zax sar
2613− hut.ran %+ roll (~(tap in lar.nak) ~)
2615− |= [sar=yaki yeb=(map tako yaki)]
2616− %+ ~(put by yeb) r.sar sar
2620−This shows, of course, exactly why nako is defined the way it is. To
2621−become completely up to date with the foreign desk, we need to merge
2622−`hit` with the foreign one so that we have all the revision numbers. We
2623−update `let` so that we know which revision is the head.
2625−We merge the new blobs in `lat`, keying them by their hash, which we get
2626−from a call to `++blob-to-lobe`. Recall that the hash is stored in the
2627−actual blob itself. We do the same thing to the new yakis, putting them
2628−in `hut`, keyed by their hash.
2630−Now, our local dome should be exactly the same as the foreign one.
2632−This concludes our discussion of `++knit`. Now the changes have been
2633−applied to our local copy of the desk, and we just need to update our
2634−subscribers. We do so in `++wake:de`.
2637− ++ wake :: update subscribers
2639− =+ xiq=(~(tap by qyx) ~)
2640− =| xaq=(list ,[p=duct q=rove])
2643− ..wake(qyx (~(gas by *cult) xaq))
2646− =+ cas=?~(ref ~ (~(get by haw.u.ref) `mood`p.q.i.xiq))
2650− ..wake ?~ u.cas (blub p.i.xiq)
2651− (blab p.i.xiq p.q.i.xiq u.u.cas)
2653− =+ nao=(~(case-to-aeon ze lim dom ran) q.p.q.i.xiq)
2654− ?~ nao $(xiq t.xiq, xaq [i.xiq xaq])
2655− $(xiq t.xiq, ..wake (balk p.i.xiq u.nao p.q.i.xiq))
2658− =+ mot=`moot`p.q.i.xiq
2659− =+ nab=(~(case-to-aeon ze lim dom ran) p.mot)
2661− $(xiq t.xiq, xaq [i.xiq xaq])
2662− =+ huy=(~(case-to-aeon ze lim dom ran) q.mot)
2664− =+ ptr=[%ud +(let.dom)]
2667− xaq [[p.i.xiq [%| ptr q.mot r.mot s.mot]] xaq]
2669− (~(lobes-at-path ze lim dom ran) let.dom r.p.q.i.xiq)
2670− ?: =(s.p.q.i.xiq ear) ..wake
2671− =+ fud=(~(make-nako ze lim dom ran) u.nab let.dom)
2672− (bleb p.i.xiq let.dom fud)
2676− ..wake =- (blub:- p.i.xiq)
2678− (~(lobes-at-path ze lim dom ran) u.huy r.p.q.i.xiq)
2679− ?: =(s.p.q.i.xiq ear) (blub p.i.xiq)
2680− =+ fud=(~(make-nako ze lim dom ran) u.nab u.huy)
2681− (bleb p.i.xiq +(u.nab) fud)
2687−This is even longer than `++knit`, but it's pretty similar to `++eave`.
2688−We loop through each of our subscribers `xiq`, processing each in turn.
2689−When we're done, we just put the remaining subscribers back in our
2692−If the subscriber is a single read, then, if this is a foreign desk
2693−(note that `++wake` is called from other arms, and not only on foreign
2694−desks). Obviously, if we find an identical request there, then we can
2695−produce the result immediately. Referential transparency for the win. We
2696−produce the result with a call to `++blab`. If this is a foreign desk
2697−but the result is not in the cache, then we produce `++blub` (canceled
2698−subscription with no data) for reasons entirely opaque to me. Seriously,
2699−it seems like we should wait until we get an actual response to the
2700−request. If someone figures out why this is, let me know. At any rate,
2703−If this is a domestic desk, then we check to see if the case exists yet.
2704−If it doesn't, then we simply move on to the next subscriber, consing
2705−this one onto `xaq` so that we can check again the next time around. If
2706−it does exist, then we call `++balk` to fulfill the request and produce
2709−`++balk` is very simple, so we'll describe it here before we get to the
2713− ++ balk :: read and send
2714− |= [hen=duct oan=@ud mun=mood]
2716− =+ vid=(~(read-at-aeon ze lim dom ran) oan mun)
2717− ?~ vid (blub hen) (blab hen mun u.vid)
2720−We call `++read-at-aeon` on the given request and aeon. If you recall,
2721−this processes a `mood` at a particular aeon and produces the result, if
2722−there is one. If there is data at the requested location, then we
2723−produce it with `++blab`. Else, we call `++blub` to notify the
2724−subscriber that no data can ever come over this subscriptioin since it
2725−is now impossible for there to ever be data for the given request.
2726−Because referential transparency.
2728−At any rate, back to `++wake`. If the given rave is a subscription
2729−request, then we proceed similarly to how we do in `++eave`. We first
2730−try to get the aeon referred to by the starting case. If it doesn't
2731−exist yet, then we can't do anything interesting with this subscription,
2732−so we move on to the next one.
2734−Otherwise, we try to get the aeon referred to by the ending case. If it
2735−doesn't exist yet, then we produce all the information we can. We call
2736−`++lobes-at-path` at the given aeon and path to see if the requested
2737−path has actually changed. If it hasn't, then we don't produce anything;
2738−else, we produce the correct nako by calling `++bleb` on the result of
2739−`++make-nako`, as in `++eave`. At any rate, we move on to the next
2740−subscription, putting back into our cult the current subscription with a
2741−new start case of the next aeon after the present.
2743−If the aeon referred to by the ending case does exist, then we drop this
2744−subscriber from our cult and satisfy its request immediately. This is
2745−the same as before -- we check to see if the data at the path has
2746−actually changed, producing it if it has; else, we call `++blub` since
2747−no more data can be produced over this subscription.
2749−This concludes our discussion of foreign requests.