@@ -1,783 +0,0 @@
2−title = "Local Reads and Writes"
7−Parts of this document may be out of date as of March 2021.
9−There are two real types of interaction with a filesystem: you can read,
10−and you can write. We'll describe each process, detailing both the flow
11−of control followed by the kernel and the algorithms involved. The
12−simpler case is that of the read, so we'll begin with that.
14−When a vane or an application wishes to read a file from the filesystem,
15−it sends a `%warp` kiss, as described above. Of course, you may request
16−a file on another ship and, being a global filesystem, clay will happily
17−produce it for you. That code pathway will be described in another
18−section; here, we will restrict ourselves to examining the case of a
19−read from a ship on our own pier.
21−The kiss can request either a single version of a file node or a range
22−of versions of a desk. Here, we'll deal only with a request for a single
25−As in all vanes, a kiss enters clay via a call to `++call`. Scanning
26−through the arm, we quickly see where `%warp` is handled.
29− ?: =(p.p.q.hic q.p.q.hic)
30− =+ une=(un p.p.q.hic now ruf)
31− =+ wex=(di:une p.q.q.hic)
35− (eave:wex hen u.q.q.q.hic)
38− [-.woo abet:(pish:une p.q.q.hic +.woo ran.wao)]
41−We're following the familar patern of producing a list of moves and an
42−updated state. In this case, the state is `++raft`.
44−We first check to see if the sending and receiving ships are the same.
45−If they're not, then this is a request for data on another ship. We
46−describe that process later. Here, we discuss only the case of a local
49−At a high level, the call to `++un` sets up the core for the domestic
50−ship that contains the files we're looking for. The call to `++di` sets
51−up the core for the particular desk we're referring to.
53−After this, we perform the actual request. If there is no rave in the
54−riff, then that means we are cancelling a request, so we call
55−`++ease:de`. Otherwise, we start a subscription with `++eave:de`. We
56−call `++abet:de` to resolve our various types of output into actual
57−moves. We produce the moves we found above and the `++un` core resolved
58−with `++pish:un` (putting the modified desk in the room) and `++abet:un`
59−(putting the modified room in the raft).
61−Much of this is fairly straightforward, so we'll only describe `++ease`,
62−`++eave`, and `++abet:de`. Feel free to look up the code to the other
63−steps -- it should be easy to follow.
65−Although it's called last, it's usually worth examining `++abet` first,
66−since it defines in what ways we can cause side effects. Let's do that,
67−and also a few of the lines at the beginning of `++de`.
70− =| yel=(list ,[p=duct q=gift])
71− =| byn=(list ,[p=duct q=riot])
72− =| vag=(list ,[p=duct q=gift])
73− =| say=(list ,[p=duct q=path r=ship s=[p=@ud q=riff]])
74− =| tag=(list ,[p=duct q=path c=note])
77− ^- [(list move) rede]
81− |=([a=duct b=gift] [hun %give b])
84− |=([a=duct b=riot] [a %give [%writ b]])
87− |=([a=duct b=gift] [a %give b])
90− |= [a=duct b=path c=ship d=[p=@ud q=riff]]
92− [%pass b %a %want [who c] [%q %re p.q.d (scot %ud p.d) ~] q.d]
95− |=([a=duct b=path c=note] [a %pass b c])
99−This is very simple code. We see there are exactly five different kinds
100−of side effects we can generate.
102−In `yel` we put gifts that we wish to be sent along the `hun:room` duct
103−to dill. See the documentation for `++room` above. This is how we
104−display messages to the terminal.
106−In `byn` we put riots that we wish returned to subscribers. Recall that
107−a riot is a response to a subscription. These are returned to our
108−subscribers in the form of a `%writ` gift.
110−In `vag` we put gifts along with the ducts on which to send them. This
111−allows us to produce arbitrary gifts, but in practice this is only used
112−to produce `%ergo` gifts.
114−In `say` we put messages we wish to pass to ames. These messages are
115−used to request information from clay on other piers. We must provide
116−not only the duct and the request (the riff), but also the return path,
117−the other ship to talk to, and the sequence number of the request.
119−In `tag` we put arbitrary notes we wish to pass to other vanes. For now,
120−the only notes we pass here are `%wait` and `%rest` to the timer vane.
122−Now that we know what kinds of side effects we may have, we can jump
123−into the handling of requests.
126− ++ ease :: release request
130− =+ rov=(~(got by qyx) hen)
131− =. qyx (~(del by qyx) hen)
132− (mabe rov (cury best hen))
133− =. qyx (~(del by qyx) hen)
135− =+ nux=(~(get by fod.u.ref) hen)
138− say [[hen [(scot %ud u.nux) ~] for [u.nux syd ~]] say]
139− fod.u.ref (~(del by fod.u.ref) hen)
140− bom.u.ref (~(del by bom.u.ref) u.nux)
144−This is called when we're cancelling a subscription. For domestic desks,
145−`ref` is null, so we're going to cancel any timer we might have created.
146−We first delete the duct from our map of requests, and then we call
147−`++mabe` with `++best` to send a `%rest` kiss to the timer vane if we
148−have started a timer. We'll describe `++best` and `++mabe` momentarily.
150−Although we said we're not going to talk about foreign requests yet,
151−it's easy to see that for foreign desks, we cancel any outstanding
152−requests for this duct and send a message over ames to the other ship
153−telling them to cancel the subscription.
157− |= [hen=duct tym=@da]
158− %_(+> tag :_(tag [hen /tyme %t %rest tym]))
161−This simply pushes a `%rest` note onto `tag`, from where it will be
162−passed back to arvo to be handled. This cancels the timer at the given
163−duct (with the given time).
166− ++ mabe :: maybe fire function
167− |* [rov=rove fun=$+(@da _+>.^$)]
174− ?. ?=(%da -.q.p.rov) ~
179− ?. ?=(%da -.p.mot) ~
180− ?.((lth now p.p.mot) ~ [~ p.p.mot])
181− ?. ?=(%da -.q.mot) ~
182− ?.((lth now p.q.mot) [~ now] [~ p.q.mot])
186−This decides whether the given request can only be satsified in the
187−future. In that case, we call the given function with the time in the
188−future when we expect to have an update to give to this request. This is
189−called with `++best` to cancel timers and with `++bait` to start them.
191−For single requests, we have a time if the request is for a particular
192−time (which is assumed to be in the future). For ranges of requests, we
193−check both the start and end cases to see if they are time cases. If so,
194−we choose the earlier time.
196−If any of those give us a time, then we call the given funciton with the
199−The more interesting case is, of course, when we're not cancelling a
200−subscription but starting one.
203− ++ eave :: subscribe
204− |= [hen=duct rav=rave]
208− ?: &(=(p.p.rav %u) !=(p.q.p.rav now))
209− ~& [%clay-fail p.q.p.rav %now now]
216− (blab hen p.rav u.u.ver)
219−There are two types of subscriptions -- either we're requesting a single
220−file or we're requesting a range of versions of a desk. We'll dicuss the
223−First, we check that we're not requesting the `rang` from any time other
224−than the present. Since we don't store that information for any other
225−time, we can't produce it in a referentially transparent manner for any
226−time other than the present.
228−Then, we try to read the requested `mood:clay` `p.rav`. If we can't access
229−the request data right now, we call `++duce` to put the request in our
230−queue to be satisfied when the information becomes available.
232−This case occurs when we make a request for a case whose (1) date is
233−after the current date, (2) number is after the current number, or (3)
234−label is not yet used.
237− ++ duce :: produce request
238− |= [hen=duct rov=rove]
240− =. qyx (~(put by qyx) hen rov)
242− (mabe rov (cury bait hen))
243− |- ^+ +>+.$ :: XX why?
246− ?. ?=([%& %v *] rav) rav
247− [%| [%ud let.dom] `case`q.p.rav r.p.rav]
250− say [[hen [(scot %ud inx) ~] for [inx syd ~ vaw]] say]
251− nix.u.ref +(nix.u.ref)
252− bom.u.ref (~(put by bom.u.ref) inx [hen vaw])
253− fod.u.ref (~(put by fod.u.ref) hen inx)
257−The code for `++duce` is nearly the exact inverse of `++ease`, which in
258−the case of a domestic desk is very simple -- we simply put the duct and
259−rave into `qyx` and possibly start a timer with `++mabe` and `++bait`.
260−Recall that `ref` is null for domestic desks and that `++mabe` fires the
261−given function with the time we need to be woken up at, if we need to be
262−woken up at a particular time.
266− |= [hen=duct tym=@da]
267− %_(+> tag :_(tag [hen /tyme %t %wait tym]))
270−This sets an alarm by sending a `%wait` card with the given time to the
273−Back in `++eave`, if `++aver` returned `[~ ~]`, then we cancel the
274−subscription. This occurs when we make (1) a `%x` request for a file
275−that does not exist, (2) a `%w` request with a case that is not a
276−number, or (3) a `%w` request with a nonempty path. The `++blub` is
277−exactly what you would expect it to be.
280− ++ blub :: ship stop
282− %_(+> byn [[hen ~] byn])
285−We notify the duct that we're cancelling their subscription since it
288−Otherwise, we have received the desired information, so we send it on to
289−the subscriber with `++blab`.
292− ++ blab :: ship result
293− |= [hen=duct mun=mood dat=*]
295− +>(byn [[hen ~ [p.mun q.mun syd] r.mun dat] byn])
298−The most interesting arm called in `++eave` is, of course, `++aver`,
299−where we actually try to read the data.
305− ?: &(=(p.mun %u) !=(p.q.mun now)) :: prevent bad things
306− ~& [%clay-fail p.q.mun %now now]
308− =+ ezy=?~(ref ~ (~(get by haw.u.ref) mun))
310− =+ nao=(~(case-to-aeon ze lim dom ran) q.mun)
311− ?~(nao ~ [~ (~(read-at-aeon ze lim dom ran) u.nao mun)])
314−We check immediately that we're not requesting the `rang` for any time
315−other than the present.
317−If this is a foreign desk, then we check our cache for the specific
318−request. If either this is a domestic desk or we don't have the request
319−in our cache, then we have to actually go read the data from our dome.
321−We need to do two things. First, we try to find the number of the commit
322−specified by the given case, and then we try to get the data there.
324−Here, we jump into `arvo/zuse.hoon`, which is where much of the
325−algorithmic code is stored, as opposed to the clay interface, which is
326−stored in `arvo/clay.hoon`. We examine `++case-to-aeon:ze`.
329− ++ case-to-aeon :: case-to-aeon:ze
330− |= lok=case :: act count through
334− ?: (gth p.lok lim) ~
336− ?: =(0 let) [~ 0] :: avoid underflow
345− %tas (~(get by lab) p.lok)
346− %ud ?:((gth p.lok let) ~ [~ p.lok])
350−We handle each type of `case:clay` differently. The latter two types are
353−If we're requesting a revision by label, then we simply look up the
354−requested label in `lab` from the given dome. If it exists, that is our
355−aeon; else we produce null, indicating the requested revision does not
358−If we're requesting a revision by number, we check if we've yet reached
359−that number. If so, we produce the number; else we produce null.
361−If we're requesting a revision by date, we check first if the date is in
362−the future, returning null if so. Else we start from the most recent
363−revision and scan backwards until we find the first revision committed
364−before that date, and we produce that. If we requested a date before any
365−revisions were committed, we produce `0`.
367−The definitions of `++aeon-to-tako` and `++tako-to-yaki` are trivial.
370− ++ aeon-to-tako ~(got by hit)
372− ++ tako-to-yaki ~(got by hut) :: grab yaki
375−We simply look up the aeon or tako in their respective maps (`hit` and
378−Assuming we got a valid version number, `++aver` calls
379−`++read-at-aeon:ze`, which reads the requested data at the given
383− ++ read-at-aeon :: read-at-aeon:ze
384− |= [oan=aeon mun=mood] :: seek and read
386− ?: &(?=(%w p.mun) !?=(%ud -.q.mun)) :: NB only for speed
388− (read:(rewind oan) mun)
391−If we're requesting the revision number with a case other than by
392−number, then we go ahead and just produce the number we were given.
393−Otherwise, we call `++rewind` to rewind our state to the given revision,
394−and then we call `++read` to get the requested information.
397− ++ rewind :: rewind:ze
398− |= oan=aeon :: rewind to aeon
401− ?: (gth oan let) !! :: don't have version
402− +>(ank (checkout-ankh q:(tako-to-yaki (aeon-to-tako oan))), let oan)
405−If we're already at the requested version, we do nothing. If we're
406−requesting a version later than our head, we are unable to comply.
408−Otherwise, we get the hash of the commit at the number, and from that we
409−get the commit itself (the yaki), which has the map of path to lobe that
410−represents a version of the filesystem. We call `++checkout-ankh` to
411−checkout the commit, and we replace `ank` in our context with the
415− ++ checkout-ankh :: checkout-ankh:ze
416− |= hat=(map path lobe) :: checkout commit
419− %+ roll (~(tap by hat) ~)
420− |= [[pat=path bar=lobe] ank=ankh]
424− =+ zar=(lobe-to-noun bar)
425− ank(q [~ (sham zar) zar])
426− =+ nak=(~(get by r.ank) i.pat)
428− r %+ ~(put by r.ank) i.pat
429− $(pat t.pat, ank (fall nak _ankh))
433−Twice we call `++cosh`, which hashes a commit, updating `p` in an
434−`ankh`. Let's jump into that algorithm before we describe
438− ++ cosh :: locally rehash
439− |= ank=ankh :: NB v/unix.c
440− ank(p rehash:(zu ank))
443−We simply replace `p` in the hash with the `cash` we get from a call to
447− ++ zu !: :: filesystem
448− |= ank=ankh :: filesystem state
449− =| myz=(list ,[p=path q=miso]) :: changes in reverse
450− =| ram=path :: reverse path into
452− ++ rehash :: local rehash
454− %+ mix ?~(q.ank 0 p.u.q.ank)
459− (shaf %dash (mix axe (shaf %dush (mix p.n.r.ank p.q.n.r.ank))))
460− $(r.ank l.r.ank, axe (peg axe 2))
461− $(r.ank r.r.ank, axe (peg axe 3))
465−`++zu` is a core we set up with a particular filesystem node to traverse
466−a checkout of the filesystem and access the actual data inside it. One
467−of the things we can do with it is to create a recursive hash of the
470−In `++rehash`, if this node is a file, then we xor the remainder of the
471−hash with the hash of the contents of the file. The remainder of the
472−hash is `0` if we have no children, else we descend into our children.
473−Basically, we do a half SHA-256 of the xor of the axis of this child and
474−the half SHA-256 of the xor of the name of the child and the hash of the
475−child. This is done for each child and all the results are xored
478−Now we return to our discussion of `++checkout-ankh`.
480−We fold over every path in this version of the filesystem and create a
481−great ankh out of them. First, we call `++lobe-to-noun` to get the raw
482−data referred to be each lobe.
485− ++ lobe-to-noun :: grab blob
491−This converts a lobe into the raw data it refers to by first getting the
492−blob with `++lobe-to-blob` and converting that into data with
496− ++ lobe-to-blob ~(got by lat) :: grab blob
499−This just grabs the blob that the lobe refers to.
502− ++ blob-to-noun :: grab blob
505− %delta (lump r.p (lobe-to-noun q.p))
511−If we have either a direct or an indirect blob, then the data is stored
512−right in the blob. Otherwise, we have to reconstruct it from the diffs.
513−We do this by calling `++lump` on the diff in the blob with the data
514−obtained by recursively calling the parent of this blob.
517− ++ lump :: apply patch
520− ?+ p.don ~|(%unsupported !!)
522− ?+ -.q.don ~|(%unsupported !!)
524− %c (lurk ((hard (list)) src) p.q.don)
525− %d (lure src p.q.don)
529− =+ dst=(lore ((hard ,@) src))
531− ?+ -.q.don ~|(%unsupported !!)
532− %a ((hard (list ,@t)) q.q.don)
533− %c (lurk dst p.q.don)
538−This is defined in `arvo/hoon.hoon` for historical reasons which are
539−likely no longer applicable. Since the `++umph` structure will likely
540−change we convert clay to be a typed filesystem, we'll only give a
541−high-level description of this process. If we have a `%a` udon, then
542−we're performing a trivial replace, so we produce simply `q.q.don`. If
543−we have a `%c` udon, then we're performing a list merge (as in, for
544−example, lines of text). The merge is performed by `++lurk`.
547− ++ lurk :: apply list patch
548− |* [hel=(list) rug=(urge)]
557− hel (slag p.i.rug hel)
558− war (weld (flop (scag p.i.rug hel)) war)
564− hel =+ gur=(flop p.i.rug)
567− ?>(&(?=(^ hel) =(i.gur i.hel)) $(hel t.hel, gur t.gur))
568− war (weld q.i.rug war)
573−We accumulate our final result in `war`. If there's nothing more in our
574−list of merge instructions (unces), we just reverse `war` and produce
575−it. Otherwise, we process another unce. If the unce is of type `&`, then
576−we have `p.i.rug` lines of no changes, so we just copy them over from
577−`hel` to `war`. If the unice is of type `|`, then we verify that the
578−source lines (in `hel`) are what we expect them to be (`p.i.rug`),
579−crashing on failure. If they're good, then we append the new lines in
580−`q.i.rug` onto `war`.
582−And that's really it. List merges are pretty easy. Anyway, if you
583−recall, we were discussing `++checkout-ankh`.
586− ++ checkout-ankh :: checkout-ankh:ze
587− |= hat=(map path lobe) :: checkout commit
590− %+ roll (~(tap by hat) ~)
591− |= [[pat=path bar=lobe] ank=ankh]
595− =+ zar=(lobe-to-noun bar)
596− ank(q [~ (sham zar) zar])
597− =+ nak=(~(get by r.ank) i.pat)
599− r %+ ~(put by r.ank) i.pat
600− $(pat t.pat, ank (fall nak _ankh))
604−If the path is null, then we calculate `zar`, the raw data at the path
605−`pat` in this version. We produce the given ankh with the correct data.
607−Otherwise, we try to get the child we're looking at from our parent
608−ankh. If it's already been created, this succeeds; otherwise, we simply
609−create a default blank ankh. We place ourselves in our parent after
610−recursively computing our children.
612−This algorithm really isn't that complicated, but it may not be
613−immediately obvious why it works. An example should clear everything up.
615−Suppose `hat` is a map of the following information.
618− /greeting --> "customary upon meeting"
619− /greeting/english --> "hello"
620− /greeting/spanish --> "hola"
621− /greeting/russian/short --> "привет"
622− /greeting/russian/long --> "Здравствуйте"
623− /farewell/russian --> "до свидания"
626−Furthermore, let's say that we process them in this order:
630− /greeting/russian/short
631− /greeting/russian/long
637−Then, the first path we process is `/greeting/english` . Since our path
638−is not null, we try to get `nak`, but because our ankh is blank at this
639−point it doesn't find anything. Thus, update our blank top-level ankh
640−with a child `%greeting`. and recurse with the blank `nak` to create the
641−ankh of the new child.
643−In the recursion, we our path is `/english` and our ankh is again blank.
644−We try to get the `english` child of our ankh, but this of course fails.
645−Thus, we update our blank `/greeting` ankh with a child `english`
646−produced by recursing.
648−Now our path is null, so we call `++lobe-to-noun` to get the actual
649−data, and we place it in the brand-new ankh.
651−Next, we process `/greeting/russian/short`. Since our path is not null,
652−we try to get the child named `%greeting`, which does exist since we
653−created it earlier. We put modify this child by recursing on it. Our
654−path is now `/russian/short`, so we look for a `%russian` child in our
655−`/greeting` ankh. This doesn't exist, so we add it by recursing. Our
656−path is now `/short`, so we look for a `%short` child in our
657−`/greeting/russian` ankh. This doesn't exist, so we add it by recursing.
658−Now our path is null, so we set the contents of this node to `"привет"`,
659−and we're done processing this path.
661−Next, we process `/greeting/russian/long`. This proceeds similarly to
662−the previous except that now the ankh for `/greeting/russian` already
663−exists, so we simply reuse it rather than creating a new one. Of course,
664−we still must create a new `/greeting/russian/long` ankh.
666−Next, we process `/greeting`. This ankh already exists, so after we've
667−recursed once, our path is null, and our ankh is not blank -- it already
668−has two children (and two grandchildren). We don't touch those, though,
669−since a node may be both a file and a directory. We just add the
670−contents of the file -- "customary upon meeting" -- to the existing
673−Next, we process `/greeting/spanish`. Of course, the `/greeting` ankh
674−already exists, but it doesn't have a `%spanish` child, so we create
675−that, taking care not to disturb the contents of the `/greeting` file.
676−We put "hola" into the ankh and call it good.
678−Finally, we process `/farewell/russian`. Here, the `/farewell` ankh
679−doesn't exist, so we create it. Clearly the newly-created ankh doesn't
680−have any children, so we have to add a `%russian` child, and in this
681−child we put our last content -- "до свидания".
683−We hope it's fairly obvious that the order we process the paths doesn't
684−affect the final ankh tree. The tree will be constructed in a very
685−different order depending on what order the paths come in, but the
686−resulting tree is independent of order.
688−At any rate, we were talking about something important, weren't we? If
689−you recall, that concludes our discussion of `++rewind`, which was
690−called from `++read-at-aeon`. In summary, `++rewind` returns a context
691−in which our current state is (very nearly) as it was when the specified
692−version of the desk was the head. This allows `++read-at-aeon` to call
693−`++read` to read the requested information.
697− |= mun=mood :: read at point
701− ?: &(?=(%w p.mun) !?=(%ud -.q.mun))
708− ?^(r.mun ~ [~ [t.yak (forge-nori yak)]])
709− ::?> ?=(^ hit) ?^(r.mun ~ [~ i.hit]) :: what do?? need [@da nori]
710− (query(ank ank:(descend-path:(zu ank) r.mun)) p.mun)
713−If we're requesting the dome, then we just return that immediately.
715−If we're requesting the revision number of the desk and we're not
716−requesting it by number, then we just return the current number of this
717−desk. Note of course that this was really already handled in
720−If we're requesting a `%w` with a specific revision number, then we do
721−something or other with the commit there. It's kind of weird, and it
722−doesn't seem to work, so we'll ignore this case.
724−Otherwise, we descend into the ankh tree to the given path with
725−`++descend-path:zu`, and then we handle specific request in `++query`.
728− ++ descend-path :: descend recursively
731− ?~(way +> $(way t.way, +> (descend i.way)))
734−This is simple recursion down into the ankh tree. `++descend` descends
735−one level, so this will eventually get us down to the path we want.
738− ++ descend :: descend
741− =+ you=(~(get by r.ank) lol)
742− +>.$(ram [lol ram], ank ?~(you [*cash ~ ~] u.you))
745−`ram` is the path that we're at, so to descend one level we push the
746−name of this level onto that path. We update the ankh with the correct
747−one at that path if it exists; else we create a blank one.
749−Once we've decscended to the correct level, we need to actually deal
753− ++ query :: query:ze
754− |= ren=?(%u %v %x %y %z) :: endpoint query
757− %u [~ `rang`+<+>.query]
758− %v [~ `dome`+<+<.query]
759− %x ?~(q.ank ~ [~ q.u.q.ank])
765−Now that everything's set up, it's really easy. If they're requesting
766−the rang, dome, or ankh, we give it to them. If the contents of a file,
767−we give it to them if it is in fact a file. If the `arch`, then we
768−calculate it with `++as-arch`.
771− ++ as-arch :: as-arch:ze
772− ^- arch :: arch report
774− ?~(q.ank ~ [~ p.u.q.ank])
777− [[p.n.r.ank ~] $(r.ank l.r.ank) $(r.ank r.r.ank)]
780−This very simply strips out all the "real" data and returns just our own
781−hash, the hash of the file contents (if we're a file), and a map of the
782−names of our immediate children.