@@ -1,1938 +1,113 @@
2title = "Ames"
2title = "Ames"
4template = "doc.html"
4template = "doc.html"
5aliases = ["/docs/learn/arvo/ames/"]
5aliases = ["/docs/learn/arvo/ames/"]
7Our networking protocol.
7Our networking protocol.
9−`%ames` is the name of both our network and the vane that communicates
9+Ames is the name of both our network and the vane that communicates
10over it. When Unix receives a packet over the correct UDP port, it pipes
10over it. When Unix receives a packet over the correct UDP port, it pipes
11−it straight into `%ames` for handling. Also, all packets sent over the
11+it straight into Ames for handling. Also, all packets sent over the
12−`%ames` network are sent by the `%ames` vane. Apps and vanes may use
12+Ames network are sent by the Ames vane. Apps and vanes may use
13−`%ames` to directly send messages to other ships. In general, apps use
13+Ames to directly send messages to other ships. In general, apps use
14+[Gall](@/docs/tutorials/arvo/gall.md) and [Clay](@/docs/tutorials/arvo/clay.md)
14−gall and clay to communicate with other ships rather than using `%ames`
15+to communicate with other ships rather than using Ames
15−directly, but this isn't a requirement. Of course, gall and clay use
16+directly, but this isn't a requirement. Of course, Gall and Clay use
16−`%ames` behind the scenes to communicate across the network. These are
17+Ames behind the scenes to communicate across the network. Jael is the only other
17−the only two vanes that use `%ames`.
18+vane to utilize Ames.
19−`%ames` includes several significant components. Although the actual
20+Ames includes several significant components. Although the actual
20−crypto algorithms are defined in zuse, they're used extensively in
21+crypto algorithms are defined in `zuse`, they're used extensively in
21−`%ames` for encrypting and decrypting packets. Congestion control and
22+Ames for encrypting and decrypting packets. Congestion control and
22−routing is handled entirely in `%ames`. Finally, the actual `%ames`
23+routing is handled entirely in Ames. Finally, the actual Ames
23protocol itself, including how to route incoming packets to the correct
24protocol itself, including how to route incoming packets to the correct
24−vane or app, is defined in `%ames`.
25+vane or app, is defined in Ames.
26−## Commentary
27+## Technical Overview
27−
28+Ames extends [Arvo's](@/docs/tutorials/arvo/arvo.md) `%pass`/`%give` `move` semantics across the network.
28−`%ames` is our networking protocol.
29+
29−
30+Ames receives packets as Arvo events and emits packets as Arvo
30−First we give commentary on the code, the algorithms involved, and the
31+effects. The runtime is responsible for transferring the bytes in
31−protocol. We trace through the code touched when a packet is sent,
32+an Ames packet across a physical network to another ship.
32−received, acknowledged, and that acknowledgment applied. This is fairly
33+
33−comprehensive, and contains many implementation details, but if you
34+The runtime tells Ames which physical address a packet came from,
34−understand this, then you understand `%ames`.
35+represented as an opaque atom. Ames can emit a packet effect to
35−
36+one of those opaque atoms or to the Urbit address of a galaxy
36−If you've scrolled down this page, you may be intimidated by the amount
37+(root node), which the runtime is responsible for translating to a
37−of Hoon code, especially if you are new to the language. Don't be afraid
38+physical address. One runtime implementation sends UDP packets
38−of it, you don't have to read any of it if you don't want to -- every
39+using IPv4 addresses for ships and DNS lookups for galaxies, but
39−interesting action the code takes is explained in plain English. In
40+other implementations may overlay over other kinds of networks.
40−fact, if you are new to the language, this may be a good learning
41+
41−opportunity. Even if you don't understand every line of Hoon code,
42+A local vane can pass Ames a `%plea` request message. Ames
42−you'll hopefully be able to follow most lines. By the time you've worked
43+transmits the message over the wire to the peer ship's Ames, which
43−through this, you'll have seen many common patterns and best practices.
44+passes the message to the destination vane.
44−Hoon, much more than other languages, is best learned by reading and
45+
45−understanding large quantities of existing code. In this way, it is
46+Once the peer has processed the `%plea` message, it sends a
46−similar to learning a natural language. All of this code is in
47+message-acknowledgment packet over the wire back to the local
47−`arvo/ames.hoon`.
48+Ames. This "ack" can either be positive to indicate the request was
48−
49+processed, or negative to indicate the request failed, in which
49−After the commentary, we have reference documentation for all the data
50+case it's called a "nack". (Don't confuse Ames nacks with TCP
50−structures that are specific to `%ames`. If you see a data structure or
51+nacks, which are a different concept).
51−a variable used that you don't recognize, search for it in the code, and
52+
52−it's very likely defined in one of these data structures. We recommend
53+When the local Ames receives either a positive message-ack or a
53−that another tab is kept open for easy access to the data structure
54+combination of a nack and "naxplanation" (explained in more detail
54−reference documentation. The code for these is split between
55+below), it gives an `%done` `move` to the local vane that had
55−`arvo/ames.hoon` and `arvo/zuse.hoon`.
56+requested the original `%plea` message be sent.
57−The Lifecycle of a Packet (or, How a Packet Becomes Law)
58+A local vane can give Ames zero or more `%boon` response messages in
58−
59+response to a `%plea`, on the same duct that Ames used to pass the
59−---
60+`%plea` to the vane. Ames transmits a `%boon` over the wire to the
60−
61+peer's Ames, which gives it to the destination vane on the same
61−Here, we will trace a packet as it makes its way through ames. There are
62+duct the vane had used to pass the original `%plea` to Ames.
62−actually two pathways through ames: the legacy path through `%want`, and
63+
63−the modern way, entered through `%wont`, with full end-to-end
64+`%boon` messages are acked automatically by the receiver Ames. They
64−acknowledgments. Here we will only trace the modern way, though much of
65+cannot be nacked, and Ames only uses the ack internally, without
65−the path is the same for both.
66+notifying the client vane that gave Ames the `%boon`.
67−When an app (or a vane) wishes to send a packet to another ship, it must
68+If the Arvo event that completed receipt of a `%boon` message
68−send a `%wont` card:
69+crashes, Ames instead sends the client vane a `%lost` message
69−
70+indicating the `%boon` was missed.
71− [%wont p=sock q=path r=*] :: e2e send message
72+`%plea` messages can be nacked, in which case the peer will send
72−```
73+both a message-nack packet and a naxplanation message, which is
73−
74+sent in a way that does not interfere with normal operation. The
74−This card takes three arguments. The `p` is a `sock`, that is, a pair of
75+naxplanation is sent as a full Ames message, instead of just a
75−two ships, the first of which is the sender and the second is the
76+packet, because the contained error information can be arbitrarily
76−receiver. But wait, you ask, why do I get to decide who is the sender?
77+large. A naxplanation can only give rise to a positive ack --
77−Can I fake like I'm someone else? The reason is that there are
78+never ack an ack, and never nack a naxplanation.
78−potentially multiple ships on the same pier, and the kernel can send a
79+
79−message from any of them. If you attempt to send a message from a ship
80+Ames guarantees a total ordering of messages within a "flow",
80−not on your pier, then ames will refuse to send it. If you hack around
81+identified in other vanes by a duct and over the wire by a "bone":
81−in your own copy of ames to go ahead and send it anyway, then the other
82+an opaque number. Each flow has a FIFO (first-in-first-out) queue of `%plea` requests
82−ship will reject it because your key is bad. Only send messages from
83+from the requesting ship to the responding ship and a FIFO queue
83−yourself.
84+of `%boon`'s in the other direction.
85−The `q` is a path, representing the place on the other side that you
86+Message order across flows is not specified and may vary based on
86−want to receive your message. It is approximately equivalent to a port
87+network conditions.
87−number. Messages on the same path are guaranteed to arrive in the same
88+
88−order as they were sent. No such guarantees are made across paths.
89+Ames guarantees that a message will only be delivered once to the
90−The `r` is the actual data that you are sending. As the type implies,
91+
91−this can be an arbitrary noun, and it will be transferred to the
92+Ames encrypts every message using symmetric-key encryption by
92−receiver exactly as-is, in a well-typed way. Of course, this is data
93+performing an elliptic curve Diffie-Hellman using our private key
93−that is sent over the wire, so be careful not to send anything too
94+and the public key of the peer. For ships in the Jael PKI
94−massive unless you're willing to wait.
95+(public-key infrastructure), Ames looks up the peer's public key
95−
96+from Jael. Comets (128-bit ephemeral addresses) are not
96−But enough about the interface. Grepping in ames.hoon for `%wont`, we
97+cryptographic assets and must self-attest over Ames by sending a
97−find that it appears in `++knob`. We see that we go
98+single self-signed packet containing their public key.
98−directly into `++wise:am`.
99+
99−
100+When a peer suffers a continuity breach, Ames removes all
100−```hoon
101+messaging state related to it. Ames does not guarantee that all
101− ++ wise :: wise:am
102+messages will be fully delivered to the now-stale peer. From
102− |= [soq=sock hen=duct cha=path val=* ete=?] :: send a statement
103+Ames's perspective, the newly restarted peer is a new ship.
103− ^- [p=(list boon) q=fort]
104+Ames's guarantees are not maintained across a breach.
104− zork:zank:(wool:(ho:(um p.soq) q.soq) hen cha val ete)
105+
105−```
106+A vane can pass Ames a `%heed` `$task` to request Ames track a peer's
106−
107+responsiveness. If our `%boon`'s to it start backing up locally,
107−The inputs to this gate are exactly the sort of thing you'd expect. In
108+Ames will give a `%clog` back to the requesting vane containing the
108−particular, everything in the `%wont` gate is here plus the calling duct
109+unresponsive peer's urbit address. This interaction does not use
109−so that we know where to send the acknowledgment and `ete` to determine
110+ducts as unique keys. Stop tracking a peer by sending Ames a
110−if we're going to do the modern end-to-end acknowledgments.
111+`%jilt` `$task`.
112−The actual line of code looks intimidating, but it's really not all that
113+Debug output can be adjusted using `%sift` and `%spew` `$task`'s.
113−bad. Working from the inside out, the call to `++um` sets up our
114−domestic server, and the call to `++ho` sets up our knowledge about the
115−neighbor we're sending to. From the outside, `++zork` and `++zank` just
116−apply the changes made to our `++um` and `++am` cores, respectively. If
117−you're familiar with the common idiom of `++abet`, that's all this is.
118−The code predates the widespread usage of that name.
120−The interesting part, then, is in `++wool:ho:um:am`. Let's look at the
124− ++ wool :: wool:ho:um:am
125− |= [hen=duct cha=path val=* ete=?] :: send a statement
128− =+ rol=(~(get by ryl.bah) cha)
131− :: ~& [%tx [our her] cha sex]
133− %+ ~(put by ryl.bah) cha
134− rol(sed +(sed.rol), san (~(put by san.rol) sex hen))
135− =+ cov=[p=p:sen:gus q=clon:diz]
138− [%bund q.cov cha sex val]
139− [%bond q.cov cha sex val]
142−This is slightly more complicated, but it's still not all that bad. Our
143−inputs, at least, are fairly obvious.
145−If you glance at the code for a second, you'll see that
146−`++wind:ho:um:am` seems to be able to send a message, or `++meal:ames`, given
147−a `++soup`. This gate, then, just sets up the things we need to for
148−`++wind` to do its job.
150−We first get `rol`, which is a `++rill:ames`, that is, a particular outbound
151−stream. This stream is specific to the path on which we're sending. If
152−the path hasn't been used before, then we create it. We let `sex` be the
153−number of messages we've already sent on this path.
155−Then, we update the outbound stream by incrementing the number of
156−messages sent and placing an entry in `san.rol` that associates the
157−message number with the `duct` that sent the message. This allows us to
158−give the acknowledgment to the one who sent the message.
160−We let `cov` be the current life of our crypto and our neighbor's
161−crypto. At the moment, we only need our neighbor's life, which we put
164−Finally, we call `++wind:ho:um:am` with the `++soup` of the path and
165−message number and the `++meal:ames` of the payload itself. For end-to-end
166−acknowledged messages, we use `%bund`.
169− [%bund p=life q=path r=@ud s=*] :: e2e message
172−Looking at how we create the `%bund`, we can easily see what each field
175−Following the trail a little further, we go to `++wind:ho:um:am`.
178− ++ wind :: wind:ho:um:am
179− |= [gom=soup ham=meal]
180− :: ~& [%wind her gom]
182− =^ wyv diz (zuul:diz now ham)
183− =^ feh puz (whap:puz now gom wyv)
187−`++wind` does three things: it (1) encodes the message into a list of
188−possibly-encrypted packets, (2) puts the message into the packet pump,
189−and (3) sends any packets that are ready to be sent. Yes, our nice
190−little linear run of each gate calling exactly one other interesting
191−gate is over. We'll go in order here.
193−`++zuul:lax:as:go` is the what converts a `++meal:ames` into a list of
197− ++ zuul :: zuul:lax:as:go
198− |= [now=@da ham=meal] :: encode message
199− ^- [p=(list rock) q=_+>]
201− ++ wasp :: null security
202− ++ weft :: fragment message
203− ++ wisp :: generate message
206−For organizational purposes, `++zuul` constructs an internal core with
207−three arms. `++wasp` encodes the meal into an atom with no encryption.
208−`++wisp` encodes a meal with possible encryption (else it simply calls
209−`++wasp`). `++weft` takes the result of `++wisp` and splits it into
213− ++ wasp :: null security
214− ^-([p=skin q=@] [%none (jam ham)])
217−This simply jams the meal, wrapping it with the `skin` of `%none`,
218−meaning no encryption.
220−Since `++wisp` is a little long, we'll go through it line-by-line.
223− ++ wisp :: generate message
224− ^- [[p=skin q=@] q=_..wisp]
227−`++wisp` produces a pair of a `skin` and an atom, which is the meal
228−encoded as a single atom and possibly encrypted.
235−If the meal that we're encoding is a `%carp`, then we don't encrypt it.
236−A `%carp` meal is a partial meal, used when a message is more than 1KB.
237−Since the entire message is already encrypted, we don't need to encrypt
238−each packet individually again.
241− ?: !=(~ yed.caq.dur)
242− ?> ?=(^ yed.caq.dur)
247− (en:r:cluy q.u.yed.caq.dur (jam ham))
250−If we have a symmetric key set up with this neighbor, then we simply use
251−it. The skin `%fast` is used to indicate a symmetric key.
254− ?: &(=(~ lew.wod.dur) |(=(%back -.ham) =(%buck -.ham)))
258−If we do not yet have our neighbor's will, then there is no way that we
259−can seal the message so that only they may read it. If what we're
260−sending is an acknowledgment, then we go ahead and just send it in the
265− ?:(=(~ lew.wod.dur) [*code +>.$] (griz now))
268−If we don't have our neighbor's will, then we "encrypt" with a key of 0.
269−If we do have their will, then we generate a new symmetric key that we
275− =+ bil=law.saf :: XX send whole will
279−`yig` will be the life and engine for our current crypto. `bil` is our
280−will. `hom` is the meal encoded as a single atom.
288− (sign:as:q.yig tuy hom)
291−If we do not have our neighbor's will, then we send our current life
292−along with our will and the message. The message itself is "signed" with
299− [`life`p.cay `life`p.yig]
301− (seal:as:q.yig pub:ex:r.cay tuy hom)
302− -- :: --zuul:lax:as:go
305−If we do have our neighbor's will, then we send our perception of their
306−current life, our current life, our will, and the message. The message
307−is sealed with their public key so that only they can read our message.
309−Once we have the message encoded as an atom, `++weft` goes to work.
312− ++ weft :: fragment message
313− ^- [p=(list rock) q=_+>.$]
319−We're going to produce a list of the packets to send. First, we use the
320−aforementioned `++wisp` to get the message as an atom.
323− =+ wit=(met 13 q.gim)
327−`wit` is the number of 1KB (2\^13 bit) blocks in the message. We assert
328−that there is at least one block.
332− =+ yup=(spit [our her] p.gim q.gim)
336−If there is exactly one block, then we just call `++spit` to turn the
337−message into a packet. We'll explain what `++spit` does momentarily.
340− =+ ruv=(rip 13 q.gim)
341− =+ gom=(shaf %thug q.gim)
345−If there is more than one block, then we rip it into blocks in `ruv`.
346−`gom` is a hash of the message, used as an id. `inx` is the number of
347−packets we've already made.
355− wasp(ham [%carp (ksin p.gim) inx wit gom i.ruv])
357− $(ruv t.ruv, inx +(inx))
360−Here we package each block into a packet with `++spit` and produce the
364− ++ spit :: cake to packet
366− =+ wim=(met 3 p.p.kec)
367− =+ dum=(met 3 q.p.kec)
368− =+ yax=?:((lte wim 2) 0 ?:((lte wim 4) 1 ?:((lte wim 8) 2 3)))
369− =+ qax=?:((lte dum 2) 0 ?:((lte dum 4) 1 ?:((lte dum 8) 2 3)))
372− =+ bod=:(mix p.p.kec (lsh 3 wix q.p.kec) (lsh 3 (add wix vix) r.kec))
385−This is how we turn a message into a real packet. This has the
386−definition of the packet format.
388−`wim` is the length of the sending ship, and `dum` is the length of the
389−receiving ship. There are only five possibilities for each of those,
390−corresponding to carriers, cruisers, destroyers, yachts, and submarines.
391−These are encoded in `yax` and `qax` as 0, 0, 1, 2, and 3, respectively.
392−Thus, `wix` and `vix` are the number of bytes that must be reserved for
393−the ship names in a packet.
395−Next, we construct `bod` by simply concatenating the sending ship, the
396−receiving ship, and the body of the message. Then, we get the encryption
397−mechanism from `++skin:ames`, which may be a 0, 1, 2, or 3, and put it in
400−Next, we concatenate together, bit by bit, some final metadata. We use
401−three bits for our protocol number, which is incremented modulo eight
402−when there is a continuity breach or the protocol changes. We use the
403−final twenty bits of a hash of the body (which, we suppose, makes it a
404−twenty bit hash) for error-checking. We use two bits to tell how much
405−room is used in the body for the sending ship, and another two bits for
406−the receiving ship. Finally, we use five bits to store the encryption
407−type. Note that since there are only two bits worth of encryption types,
408−there are three unused bits here. This adds up to 32 bits of header
409−data. Finally, we concatenate this onto the front of the packet. Thus,
410−we can summarize the packet header format as follows.
414− 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
415− +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
416− |Proto| Hash of Body |yax|qax| Crypto |
417− +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
420−After this, there are `yax` bits of the sender name, `qax` bits of the
421−receiver name, and up to 8192 bits of data. Thus, the maximum size of a
422−packet is achieved in a message between two submarines with 8192 bits of
423−data. This will require 32+128+128+8192 = 8480 bits, or 1060 bytes.
425−This concludes our discussion of `++zuul:lax:as:go`. If you recall from
426−`++wind:ho:um:am`, the list of packets from `++zuul` is passed into
427−`++whap:pu` to update the packet pump and get any packets that can be
432− |= [now=@da gom=soup wyv=(list rock)] :: send a message
433− ^- [(list rock) _+>]
434− =. pyz (~(put by pyz) gom (lent wyv))
441− diq (~(put by diq) (shaf %flap i.wyv) nus)
442− puq (~(put to puq) [nus `soul`[gom 0 | ~2000.1.1 i.wyv]])
447−First, we put into `pyz` the id for this message and the number of its
448−packets that have not yet been acknowledged, which is of course the
449−total number of packets since we haven't even sent the packets.
451−For every packet, we change three things in the state (`++shed:ames`) of our
452−packet pump: (1) we increment `nus`, the number of packets sent; (2) we
453−put the packet number into `diq` keyed by a hash of the packet; and (3)
454−we put the packet into the packet queue, with the basic metadata of its
455−id `gom`, 0 transmissions, not live yet, last sent in the year 2000, and
458−Finally, we harvest the packet pump.
462− |= now=@da :: harvest queue
463− ^- [(list rock) _+>]
464− ?: =(~ puq) [~ +>(rtn ~)]
465− ?. (gth caw nif) [~ +>]
466− =+ wid=(sub caw nif)
472−`++harv` contains a core for most of its work. The meat is in `++apse`.
473−First, though, it sets itself up. If there aren't any packets in the
474−queue, then we simply do nothing except set `rtn`, our next timeout, to
475−nil because we don't have any packets that may need to be retransmitted.
476−If we have more live (that is, sent and unacknowledged) packets than our
477−window size, then we don't do anything.
479−Otherwise, we let `wid` be the width of our remaining packet window, and
480−we initialize `rub` to nil. `rub` will be the list of packets that are
481−ready to be sent. We then call `++apse` and pass the result to `++abet`.
482−`++apse` decides which packets are ready to be sent.
492− ?. =(| liv.q.n.puq) .
493− :: ~& [%harv nux.q.n.puq p.n.puq]
496− rub [pac.q.n.puq rub]
499− nux.q.n.puq +(nux.q.n.puq)
504−If there are no remaining packets to send, or if we've filled the packet
505−window, do nothing. We call `++rigt` and `++left` to process the left
506−and right branches of the packet queue.
508−Now we assert that the queue is not empty, and we again check that we
509−haven't filled the packet window. We will operate on the head of the
510−queue. If the packet is live, then do nothing. Otherwise, we go ahead
513−To send, we (1) decrement `wid`, our packet window width; (2) cons the
514−packet onto the `rub`, which will be returned as the list of packets to
515−send; (3) increment `nif`, the number of live packets; (4) set the
516−packet to be live; (5) increment the number of transmissions of the
517−packet; and (6) set the last sent time of the packet to now.
522− ^+(. =+(lef=apse(puq l.puq) lef(puq [n.puq puq.lef r.puq])))
525− ^+(. =+(rig=apse(puq r.puq) rig(puq [n.puq l.puq puq.rig])))
528−These do exactly what you would expect: they traverse the packet queue
529−so that `++apse` gets called recursively through it.
531−Finally, `++abet` gets called, which resolves the changes.
536− [(flop rub) +>.$(rtn [~ (add rto now)])]
539−This returns the packets that we wish to send, and it updates the
540−timeout so that we know when to try resending unacknowledged packets.
542−This concludes our discussion of `++whap:pu`. To finish
543−`++wind:ho:um:am`, we just need to delve into `++busk:ho:um:am`. But
544−wait, in the call to `++busk`, the first argument is `xong:diz`. What is
545−this? This, my dear reader, is one more detour, this time into
549− ++ xong :: xong:lax:as:go
550− ^- (list ship) :: route unto
551− =+ [fro=xen too=xeno]
552− =+ ^= oot ^- (list ship)
556− ?: (lien fro |=(a=ship =(a i.too))) ~
557− [i.too $(too t.too)]
558− :: ~& [%xong-to [our her] (weld oot ?>(?=(^ fro) t.fro))]
559− (weld oot ?>(?=(^ fro) t.fro))
562−This gets the list of intermediate ships needed to get a packet from us
563−to our neighbor. First, we get `fro` and `too`, the "canons" of ourself
564−and our neighbor, respectively.
566−What is this "canon", you ask? A canon is simply a ship plus its
567−"ancestors", as defined by `++sein:title`. For example, the canon of
571− ~hoclur-bicrel/try=> (saxo:title ~hoclur-bicrel)
572− ~[~hoclur-bicrel ~tasruc ~tug]
575−If we follow the algorithm in `++xong`, we see that we are simply
576−creating a list of ships that form a path from our neighbor to ourself.
577−Essentially, we look through the canon of our neighbor until we find
578−something in our own cannon -- a common ancestor. Or, if we are from
579−different carriers, then there is no common ancestor. We then weld this
580−onto the tail of our own canon. In the end, this is simply a list of
581−possible ships to try to route via to get to our neighbor, ordered by
582−preferability (that is, closeness to our neighbor). We will end up
583−trying, in order, to find a lane to these.
585−Now, we can finally get to `++busk:ho:um:am`.
588− ++ busk :: busk:ho:um:am
589− |= [waz=(list ship) pax=(list rock)] :: send packets
594− $(pax t.pax, bin (weld (flop (wist:diz now waz ~ i.pax)) bin))
598−Thankfully, `++busk` is fairly simple. We go through the list of packets
599−and convert them to `++boon:ames`s with `++wist:lax:as:go`. These boons are
600−placed into `bin`, and they end up getting processed by `++clop` (this
601−happens in `++knob`).
604− ++ wist :: wist:lax:as:go
605− |= $: now=@da :: route via
611− ?: =(our her) [[%ouzo *lane pac] ~]
613− =+ dyr=?:(=(her i.waz) dur (gur i.waz))
618− :_ ?: ?=(%ix -.u.lun.wod.dyr)
621− :+ %ouzo u.lun.wod.dyr
622− ?: &(=(i.waz her) =(~ ryn)) pac
623− =+ mal=(jam `meal`[%fore her ryn pac])
627− ?~ yed.caq.dyr [%none mal]
631− (en:crub:crypto q.u.yed.caq.dyr mal)
635−This takes a sample of the current time, the list of ships that we just
636−generated, a lane if we already know it, and the packet itself.
638−First, if we are sending a message to ourself, then we simply create a
639−`%ouzo` boon with a bunted lane. Otherwise, if there are no routing
640−candidates, there is nothing we can do, so we return nil.
642−Next, we get the `++dore:ames` of the first routing candidate. If we're looking
643−at the neighbor to whom we're trying to send the message, then we simply
644−use the `++dore:ames` that we already have. Otherwise, we get a default `++dore:ames`.
646−If we're the first routing candidate, or if we have don't have a lane to
647−this candidate, then we skip this candidate and move on to the next one.
649−If we have only a provisional ip address, then we try to send on it, but
650−we also try to send on later routing candidates as well. Otherwise, we
651−only send on this one candidate.
653−Finally, we create the actual `%ouzo` boon. The lane is the one from our
654−`++dore:ames`. If we're sending it directly to our intended recipient, and we
655−haven't been told to use a specific lane, then we just send the packet
656−directly. Otherwise, we wrap it in a little `%fore` meal, telling the
657−intermediary to whom we wish it to be sent. If we have already set up a
658−symmetric key with the intermediary, then we encrypt it with that.
659−Otherwise, we send it in the clear.
661−Now, if you recall, we have traced all the way through from the
662−beginning when, in `++knob`, the `%wont` card was handled by a call to
663−`++wise`. There is only one more step before the packet is finally sent.
664−Looking in `++knob`, we see that the resultant list of boons is passed
665−into `++clop`, which will execute the correct actions and return a list
666−of moves. In `++clop`, we see the handling of each specific boon. The
667−one we are interested in is `%ouzo`, since that is the only one we have
672− :: ~& [%send now p.bon `@p`(mug (shaf %flap q.bon))]
674− [[gad.fox [%give %send p.bon q.bon]] ~]
677−Very simply, we give a `%send` gift along the special duct that goes
678−straight into the bowels of unix. This is the last stop before we drop
679−into vere, and later libuv. And then... the world.
681−The packet, after its creation, embarks on a journey across physical
682−time and space into the great unknown. Hurtling through fiber-optic
683−cables at hundreds of thousands of kilometers per second, it finally
684−arrives at our neighbor's network adapter. The adapter tells unix, unix
685−tells libuv, libuv tells vere, and vere sends a `%hear` task to ames.
686−And now we reenter the kernel.
688−The `%hear` task goes straight to `++knob`, just as did the `%wont` task
693− (~(gnaw am [now fox]) %good p.kyz q.kyz)
696−Here, though, we call `++gnaw:am` to process the packet. The arguments
697−to `++gnaw` are the same as those to the `%hear` task: the lane on which
698−the packet was received and the packet itself. The other argument is
699−just `%good`, which is a `++cape:ames` saying that we expect the packet to
700−succeed. If a formal error occurs, then since we have a transactional
701−event system, the `%hear` event will never be considered to have
702−actually happened, and unix will send a `%hole` task so that we may send
703−a negative acknowledgment.
707− |= [kay=cape ryn=lane pac=rock] :: process packet
708− ^- [p=(list boon) q=fort]
709− ?. =(2 (end 0 3 pac)) [~ fox]
711− ?: (goop p.p.kec) [~ fox]
712− ?. (~(has by urb.ton.fox) q.p.kec)
716− %- ~(chew la:(ho:(um q.p.kec) p.p.kec) kay ryn %none (shaf %flap pac))
720−First, we check the protocol number. If it is not correct, then we
721−simply ignore the packet entirely. Otherwise, we parse the packet with
722−`++bite`, which converts a packet atom into a `++cake:ames`, that is, a triple
723−of the `sock` (pair of sender and receiver), the `skin` (encryption
727− ++ bite :: packet to cake
729− =+ [mag=(end 5 1 pac) bod=(rsh 5 1 pac)]
730− =+ :* vez=(end 0 3 mag) :: protocol version
731− chk=(cut 0 [3 20] mag) :: checksum
732− wix=(bex +((cut 0 [23 2] mag))) :: width of receiver
733− vix=(bex +((cut 0 [25 2] mag))) :: width of sender
734− tay=(cut 0 [27 5] mag) :: message type
737− ?> =(chk (end 0 20 (mug bod)))
738− :+ [(end 3 wix bod) (cut 3 [wix vix] bod)]
740− (rsh 3 (add wix vix) bod)
743−This is exactly the inverse of `++spit`. Note that here we check both
744−the protocol number and the hash, crashing on error. Remember that a
745−crash will result in a negative acknowledgment being sent.
747−Continuing in `++gnaw`, we see that if the intended recipient is not on
748−our pier, then we drop the packet.
750−If we've gotten this far, then we wish to process the packet. Recall
751−that `++ho` and `++um` set up the domestic server and foreign client
752−cores, respectively, and that `++zork` and `++zank` resolve any changes
755−The new stuff here, then, is the `++la` core and the `++chew` arm. The
756−`++la` sets up a core for this particular packet, containing the current
757−success/failure `++cape:ames`, the lane it was sent on, the encryption type,
758−and a hash of the packet, used as an id.
760−`++chew` is called with the encryption type and the message itself. It
761−contains a little helper core inside of it, which starts immediately
768− =+ neg==(~ yed.caq.dur.diz)
770− =+ eng==(~ yed.caq.dur.diz)
774− :_(bin [%wine [our her] " is ok"])
777− :_(bin [%wine [our her] " is your neighbor"])
781−First, we let `oub` be true if our neighbor hasn't been responding to us
782−for more than sixteen seconds. Let `neg` be true if we haven't yet
783−proposed a symmetric key, meaning that we haven't yet corresponded with
784−this ship, so they are not our neighbor. Next, we run `++east`, which
785−we'll go into in just a minute.
787−We now do the same two checks and store the results in `eng` and `bou`.
788−If our neighbor has, like the prodigal son, returned after an extended
789−absence, then we send a `%wine` boon as the proverbial fatted calf,
790−which is simply printed out to the console. Likewise, if we are meeting
791−one with whom we have never had the pleasure of acquainting ourselves,
792−we send a message to the console to that effect.
794−We skipped over `++east`, which contains the meat of the processing. It
795−first decrypts the message, then calls `++chow:la:ho:um:am` with the
796−resultant meal. We'll go through each of the four cases in turn, but
797−first since each one calls `++bilk:pu`, we'll take a brief detour.
801− |= now=@da :: inbound packet
807− rtn ?~(puq ~ [~ (add now trt)])
811−This updates the timing information in our packet pump. `rue`, the last
812−time we have heard from this neighbor, is set to now. `rto`, the
813−retransmit timeout is set to twice the current ping time, and if there
814−is anything in the packet queue, then we reset the next timeout, since
815−we've just heard a message.
821− =. puz (bilk:puz now)
822− (chow ((hard meal) (cue msg)))
825−The simplest case is when the encryption type is `%none`. We first call
826−`++bilk` to update the packet pump, then we cue (unjam) the message into
827−a meal. We hard cast it into a meal -- if the cast fails, then we do
828−want to crash since someone is sending us malformed data. Finally, we
829−send the result to `++chow` for interpretation and handling.
833− =+ [mag=`hand`(end 7 1 msg) bod=(rsh 7 1 msg)]
834− =+ dey=(kuch:diz mag)
836− ~& [%bad-key her mag]
837− +>.$ :: ignore unknown key
838− =. puz (bilk:puz now)
840− (chow(aut sin) ((hard meal) (cue (dy:q:sen:gus key bod))))
843−For symmetric encryption, we first get the `++hand`, which is the hash of
844−the symmetric key. We pass it to `++kuch:lax:as:go`, which returns the
845−key if we either have used it before or we have proposed it. If we have
846−proposed it, then we change its status from proposed to real. If
847−`++kuch` fails, then we drop the packet and print out a `%bad-key`
850−Otherwise, we call `++bilk` as before to update the packet pump and pass
851−into `++chow` the decrypted data.
855− =+ mex=((hard ,[p=[p=life q=life] q=will r=@]) (cue msg))
856− =. diz (deng:diz q.mex)
859− =+ gey=(sev:gus p.p.mex)
860− =+ mes=(need (tear:as:q.gey pub:ex:r.wug r.mex))
861− =. diz (wasc:diz p.mes)
862− =. puz (bilk:puz now)
866−For sealed asymmetric encryption, we first take off the the layer of
867−data that gives us the life and will of our neighbor, and we apply try
868−to extend their former will with the new data. `++deng` will fail if
871−Next, we get our most current understanding of our neighbor's crypto,
872−and we verify that it's the same life as what they're sending. Then, we
873−get our own crypto from `++sev` and decrypt the message with the public
874−key from our neighbor's crypto. We register the proposed symmetric key,
875−update the packet pump, and call `++west`, which simply casts the
876−message to a meal and calls `++chow`, reporting any error.
880− =+ mex=((hard ,[p=[~ q=life] q=will r=@]) (cue msg))
881− =. diz (deng:diz q.mex)
884− =+ mes=(need (sure:as:r.wug *code r.mex))
885− =. puz (bilk:puz now)
889−Finally, for signed asymmetric encryption, we, as before, take off the
890−layer of data that gives us the life and will of our neighbor. This
891−time, of course, we do not get our own crypto -- only that of our
894−The rest you have seen. We call `++deng` to extend the will, we verify
895−that their crypto life is what we think it ought to be, we "decrypt" the
896−data, we update the packet pump, and we call `++west` to call `++chow`.
899− ++ chow :: chow:la:ho:um:am
900− |= fud=meal :: interpret meal
902− =. diz ?:(=(%none aut) diz (wast:diz ryn))
906−Here, if the message was encrypted at all, then we call
907−`++wast:lax:as:go`, which simply updates the lane (route) to our
908−neighbor (unless we're given a provisional route). This ensures that we
909−always have the most direct possible path to them.
911−We've been handling this meal for so long, we've almost forgotten what
912−we want to do with it. The telos is of any meal to be dined on. We will
913−choose out the cases here that are important to our current
920− ?. ?=(%if -.u.q.fud) u.q.fud
924− (emit %mead lyn r.fud)
925− =+ zid=(myx:gus p.fud)
926− (emir (wist:zid now xong:zid [~ lyn] r.fud))
929−Forwarding is the simplest case, since we've seen all the arms before,
930−except perhaps `++emit` and `++emir`, which simply take a boon or list
931−of boons respectively and queue them up to be handled when the core
932−resolves. If we're told to forward a packet to ourselves, then we emit a
933−`%mead` boon which simply sends another `%hear` task to ourselves with
934−the data. Otherwise, we try to find a route to the recipient, as before.
938− =+ zol=(~(get by olz.weg) s.fud)
939− ?^ zol cock(kay u.zol)
941− =+ neb=(~(get by nys.weg) s.fud)
942− ?^ neb [u.neb nys.weg]
943− =+ neb=`bait`[(kins p.fud) 0 r.fud ~]
944− [neb (~(put by nys.weg) s.fud neb)]
945− ?> (lth q.fud p.r.neb)
946− ?> =((kins p.fud) p.neb)
948− =+ doy=`(unit ,@)`(~(get by q.r.neb) q.fud)
951− q.r.neb (~(put by q.r.neb) q.fud t.fud)
954− :: ~& [%carp q.fud s.fud q.neb p.r.neb]
956− =: nys.weg (~(del by nys.weg) s.fud)
957− olz.weg (~(put by olz.weg) s.fud kay)
961− +>.$(nys.weg (~(put by nys.weg) s.fud neb))
964−Here, we have received a partial message, and we're just assembling the
965−individual packets into a message. Most of this code is fairly
966−algorithmic, so we'll just hit the high points. In the beginning, we
967−check if we've already received this message, and if so, we resend the
968−acknowledgment. Remember, "always ack a dupe, never ack an ack".
970−In `nys.weg` we keep track of an incoming set of partial packets,
971−indexed by the `flap` hash that comes with every packet. We check to see
972−if we have already received this partial message, and if so we
973−acknowledge it. Otherwise, we put it in `nys.weg` unless this is the
974−last message, in which case we ack the last partial message, move the
975−complete message into `olz.weg`, and call `++golf`, which assembles the
976−message and calls `++chew`, to start the dance again with the complete
981− :: ~& [%bund q.fud r.fud]
982− ?> =(p:sen:gus p.fud)
983− (deer q.fud r.fud ?-(kay %dead ~, %good [~ s.fud]))
986−What if we're just receiving a regular old, garden variety message? We
987−call `++deer` with the data from the message. If we already know that
988−the message processing will fail (that is, if we got a `%hole` card from
989−unix rather than a `%hear` card), then we don't even send the data at
990−all. Remember, if a packet fails to process, it's as if it never even
991−arrived, except that we send a negative acknowledgment.
994− ++ deer :: deer:la:ho:um:am
995− |= [cha=path num=@ud dut=(unit)] :: interpret message
997− =+ rum=(fall (~(get by raz.bah) cha) *race)
1000− ?. (gte num did.rum) :: always ack a dup
1001− (cook (~(get by bum.rum) num) cha ~ ryn dam)
1003− (coat cha rum(mis (~(put by mis.rum) num [kay ryn dam dut])))
1006− %+ ~(put by raz.bah) cha
1007− rum(mis (~(put by mis.rum) num [kay ryn dam dut]))
1012−First, we get the race for this particular triple of sender, receiver,
1013−and path, creating it if it doesn't exist. If we've already acked the
1014−message, then we resend the ack. Note that `did.rum` is the number of
1015−packets we acknowledged, positively or negatively while `bum.rum` is a
1016−map of message numbers to negative acknowledgments. Thus, if a message
1017−number is less than `did.rum`, then if it's in `bum.rum` then it was
1018−negatively acknowledged, otherwise it's positively acknowledged. Thus, we
1019−are constant in space with the number of successful messages and linear
1020−in the number of failed messages. We'll document `++cook` later on, but
1021−suffice it to say that it sends an acknowledgment. It is to end-to-end
1022−acknowledgments what `++cock` is to packet-level acknowledgments.
1024−If we are still processing a message (that is, `dod.rum` is false), then
1025−we simply put this message in the map of misordered packets to be
1026−processed when their time comes. "Processing a message" in this case
1027−means that we've received the message and notified the correct
1028−application, but we're still waiting for the application-level
1031−Otherwise, we're ready for a packet, so we process it.
1034− ++ coat :: coat:ho:um:am
1035− |= [cha=path rum=race] :: update input race
1037− =+ cun=(~(get by mis.rum) did.rum)
1039− +>.$(raz.bah (~(put by raz.bah) cha rum))
1040− ?. =(%good p.u.cun) +>.$
1043− raz.bah (~(put by raz.bah) cha rum(dod |))
1048− `soap:ames`[[p:sen:gus clon:diz] cha did.rum]
1053−First, we grab the message we want to process and store it in `cun`. If
1054−it's a good packet, then we change `dod.rum` to false, meaning that
1055−we're in the middle of processing a packet and should not start
1056−processing another one. We also put a `%mulk` boon into the queue so
1057−that, when it all resolves, we send a mesage to the intended recipient
1058−application. The boon contains the sender, the receiver, the identity of
1059−the message, and the message itself.
1061−This bubbles up all the way back to `++knob`, where we were handling the
1062−`%hear` card. Following the logic in `++knob`, we can see that the boons
1063−get sent into `++clop` to be turned into actual arvo-level moves. We've
1064−been here before, if you recall, when we handled the `%cake` boon to
1065−send a message. Now, we're handling the `%mulk` boon, which is
1066−unfortunately slightly more complicated.
1070− :: ~& [%mulk p.bon q.bon]
1071− ?> ?=([@ @ *] q.q.bon)
1072− ?> ?=(%q i.q.q.bon)
1076− :~ :- (claw p.p.bon)
1077− [%sick %wart p.bon i.t.q.q.bon t.t.q.q.bon r.bon]
1079− %ge :: %gall request
1080− ?> ?=([@ ~] t.t.q.q.bon)
1081− =+ app=`term`(need ((sand %tas) i.t.t.q.q.bon))
1083− :+ (scot %p p.p.bon)
1086− :_ fox [hen %pass pax %g %rote p.bon app r.bon]~
1087− %gh :: %gall response
1088− ?> ?=([@ ~] t.t.q.q.bon)
1089− =+ app=`term`(need ((sand %tas) i.t.t.q.q.bon))
1091− :+ (scot %p p.p.bon)
1094− :_ fox [hen %pass pax %g %roth p.bon app r.bon]~
1098−We're dispatching messages based on the prefix of their path. Since only
1099−`%gall` apps use end-to-end acknowledgments at the moment, every path
1100−must have at least two elements, and the first one must be `%q`. Beyond
1101−that, we handle the `/q/ge` and `/q/gh` cases for gall requests and
1102−responses, respectively.
1104−In both cases, we require the next term in the path to be the name of
1105−the intended recipient `%gall` app. Thus, a message to `/q/ge/talk` for
1106−example, will send a message to the talk app.
1108−We then send a message to the app itself. The message is either a
1109−`%rote` or a `%roth` for a request and a response, respectively. The
1110−content is the `rook` or `roon` that was sent (stored in `r.bon`), but
1111−we don't actually handle that at all here. That's completely a
1112−`%gall`-level thing. We're just the messenger.
1114−Notice the path we send this over. We encode the sender, the receiver,
1115−and the path over which it was sent. This fully specifies the `race` so
1116−that when the app gives us the acknowledgment we know where to send it.
1118−We now have another interlude. We have entrusted our precious data, so
1119−carefully guarded and guided from the app on that far-away ship, to our
1120−local app. It has the ability to do whatever it pleases with it. It may
1121−take a significant amount of time to process. When the message has been
1122−handled by this app, though, it must produce an acknowledgment. Our
1123−final task is to deliver this acknowledgment to the sending app.
1125−We should describe here what exactly these oft-mentioned acknowledgments
1126−actually consist of. There are two kinds of acknowledgments: positive
1127−and negative. A positive acknowledgment contains no data other than its
1128−existence. A negative acknowledgment may optionally include a reason for
1129−said negativity. Formally, a negative acknowledgment is an `ares`, which
1130−is a unit pair of a term and a list of tanks. If this is null, this is
1131−simply a failure with no associated information. If the pair exists, the
1132−term is a short error code that is usually both human and computer
1133−readable. For example, if you try to send a message to a valid `%gall`
1134−app that doesn't have any `++poke` to handle it, then `%gall` will give
1135−a negative acknowledgment with error term `%poke-find-fail`. The list of
1136−tanks is a human-readable description of the error. This often contains
1137−a stack trace. At any rate, all this information is returned to the
1138−sending app on the other end of the wire.
1140−After this brief interlude, our story resumes in `++knap`, where we
1141−receive responses. In particular, a `%mean` indicates a negative
1142−acknowledgment while a `%nice` indicates a positive acknowledgment.
1146− ?> ?=([@ @ @ *] tea)
1147− =+ soq=[(slav %p i.tea) (slav %p i.t.tea)]
1151− %^ ~(rack am [now fox]) soq pax
1152− ?-(+<.sih %mean `p.+.sih, %nice ~)
1155− |- ^- [p=(list move) q=_+>.^$]
1158− =^ toe fox (clop now hen i.p.fuy)
1159− $(p.fuy t.p.fuy, out (weld (flop toe) out))
1162−Recall the format of the path we sent the message on, and you'll
1163−understand why `soq` and `pax` are the sender/receiver pair and path on
1164−which the message was sent. The rest of this is structured much like
1165−`++knob`, so we call `++rack:am` and send the resulting boons to
1166−`++clop`. Business as usual.
1170− |= [soq=sock cha=path cop=coop] :: e2e ack
1171− =+ oh=(ho:(um p.soq) q.soq)
1172− =. oh (cook:oh cop cha ~)
1176−First, we set up `++um` and `++ho`, as we've done twice before, for our
1177−domestic and foreign servers, respectively. The other two things are
1178−new, though. Well, `++cook` is not actually new, but we delayed the
1179−explanation saying only that it sends an acknowledgment. The time has
1183− ++ cook :: cook:ho:um:am
1184− |= [cop=coop cha=path ram=(unit ,[ryn=lane dam=flap])]
1185− ^+ +> :: acknowledgment
1186− =+ rum=(need (~(get by raz.bah) cha))
1187− =+ lat=(~(get by mis.rum) did.rum)
1188− ?: &(?=(~ lat) ?=(~ ram)) ~&(%ack-late-or-redundant +>.$)
1189− =+ ^- [ryn=lane dam=flap]
1190− ?^ ram [ryn.u.ram dam.u.ram]
1195− %+ ~(put by raz.bah) cha
1196− rum(dod &, bum ?~(cop bum.rum (~(put by bum.rum) did.rum u.cop)))
1197− =^ roc diz (zuul:diz now [%buck cop dam ~s0])
1198− (busk(diz (wast:diz ryn)) xong:diz roc)
1201−If we are acknowledging a message that we have already acked, the `ram`
1202−will contain the new lane and flap to send the duplicate ack to. This
1203−happens if we call `++cook` in `++deer`, but it doesn't happen from
1204−`++rack`. If there is no message waiting to be acknowledged and we're
1205−not given an explicit lane and flap (that is, we're not sending a
1206−duplicate ack), then the app must have sent us multiple acknowledgments.
1207−We do the only sensible thing we can do and drop all acknowledgments
1208−after the first, printing a message. This is, in fact, an error, so it
1209−could be argued that we ought to crash. Whatever you do, don't depend on
1212−First, we grab the race specified by the given path, and we get the most
1213−recent in-order message, which must be the one which is being
1216−Then, we decide which lane/flap to respond on/to. Basically, in the
1217−usual case we respond on the lane through which the initial message was
1218−sent, which is stored along with the other packet information in
1219−`mis.rum`, since it has to be remembered across calls to ames. However,
1220−if we receive a duplicate message, then we must respond to the new
1221−message. It's quite possible the reason the other acknowledgment didn't
1222−get returned was that the lane between the ships was broken.
1224−At any rate, we update the race by saying that we've finished processing
1225−this packet (unless we're sending a duplicate ack) and, if we're sending
1226−a negative acknowledgment, putting the negative ack into `bum.rum` so
1227−that we can resend it if necessary.
1229−We encode our new message, updating the packet pump, with `++zuul`, as
1230−before, and we send it off with `++busk`, routed via `++wast` to one of
1231−the ships in `++xong`. Of course, in practice, we don't even look at the
1232−ships in `++xong` because we already have a lane directly to our
1233−neighbor (the one over which they sent their message to us).
1235−We glossed over the actual message we're sending back. We're sending a
1236−`%buck` meal, which is an acknowledgment. The `cop` specifies whether
1237−this is a positive or a negative ack, `dam` specifies the message we're
1238−acknowledging, and the `~s0` is a placeholder for the processing time
1239−required. This time is neither calculated (though it is hopefully
1240−obvious how to do so) nor used at present, but this information may be
1241−used in the future for improved congestion control. Since the round-trip
1242−time for an end-to-end acknowledged packet includes the processing time
1243−on the other end, most common congestion control algorithms will stumble
1244−when some messages take much longer to process than others. As noted,
1245−though, this is simply an opportunity for improvement -- our congestion
1246−control algorithms are relatively naive at the moment.
1248−Recall that `++busk` calls `++wist` to put the actual `%ouzo` boon in
1249−the queue, which gets handled by `++clop` to actually send the message.
1250−This is the same pipeline as sending any other message, so we'll refer
1251−you to the explanation above if you've forgotten it.
1253−The last thing we need to do on this ship is move on to the next packet
1254−in the queue if there is one. If you recall, in `++rack` after the call
1255−to `++cook` there was a call to `++cans:ho:um:am`.
1258− ++ cans :: cans:ho:um:am
1260− =+ rum=(need (~(get by raz.bah) cha))
1264− mis (~(del by mis.rum) did.rum)
1269−This is very simple. We increment the number of packets that we've
1270−acknowledged on this race and we delete the packet that we just
1271−acknowledged from the set of misordered packets.
1273−Then, we call `++coat` again to process the next packet if we've already
1274−received it. And that's it for this.
1276−The acknowledgment now travels the same path that its forebearer, the
1277−original message, once tread, but this time not into the great unknown.
1278−The weary traveler is seeking out its familial roots, finding the app
1279−from whom sprung forth the original message way back in paragraph three.
1280−When it arrives at the network adapter of its ancestors, the adapter
1281−tells unix, unix tells libuv, libuv tells vere, and vere sends a `%hear`
1282−task to ames. Once more into the kernel.
1284−The `%hear` task is handled in `++knob` as before, leading to `++gnaw`,
1285−going over to `++chew`, `++apse`, `++chow`, and eventualy to `++dine`.
1286−We've seen most of the cases in `++dine`, but we haven't yet looked at
1287−the handling of this `%buck` meal.
1291− =. +> ?.(=(%full aut) +> cock) :: finish key exch
1292− +>(..la (tock p.fud q.fud r.fud))
1295−We send a packet level acknowledgment if we're finishing a key exchange,
1296−else we call `++tock` to process the acknowledgment.
1298−This will get a little involved, so if you don't much care about how
1299−exactly an acknowledgment happens, just know that the result gets gifted
1300−as a `%woot` card back to the app who sent it. For those brave souls who
1301−wish to see this thing through to the end, it's once more into the
1305− ++ tock :: tock:ho:um:am
1306− |= [cop=coop fap=flap cot=@dr] :: e2e ack by hash
1308− =^ yoh puz (bick:puz now fap)
1312− (done p.u.p.yoh q.u.p.yoh)
1317− `boon`[%cake [our her] [[p:sen:gus clon:diz] u.p.yoh] cop u.hud]
1319− (busk xong:diz q.yoh)
1322−We're going to work through this one a little backwards since it's
1323−mostly fairly simple except the call to `++bick:pu`. In fact, we'll just
1324−skip `++bick` for the moment and finish the rest.
1326−If `++bick` succesfully acks the message, then we call `++done`.
1329− ++ done :: done:ho:um:am
1330− |= [cha=path num=@ud] :: complete outgoing
1331− ^- [(unit duct) _+>]
1332− =+ rol=(need (~(get by ryl.bah) cha))
1333− =+ rix=(~(get by san.rol) num)
1338− (~(put by ryl.bah) cha rol(san (~(del by san.rol) num)))
1342−This very simply gets the rill (the outgoing counterpart to a race, if
1343−you recall), pulls out of the map of outstanding messages the duct over
1344−which the original message was sent, and produces this duct while
1345−deleting that entry from the map of outstanding messages.
1347−Going back to `++tock`, we now have the duct we need to return the
1348−result over. We do the very sensible thing and put a `%cake` boon in the
1349−queue to be processed later by `++clop`.
1351−In `q.yoh` we have a list of messages that may need to be sent, which we
1352−pass to `++busk` to send, as usual. When an acknowledgment arrives, that
1353−may trigger other messages immediately. This often happens when sending
1354−more messages than the width of the logical window since for congestion
1355−control reasons another message cannot be sent until some of the earlier
1356−ones have been acknowledged.
1358−We'll look at the processing of the `%cake` boon in `++clop` before we
1359−get back to talking about `++bick`.
1364− :~ [s.bon %give %woot q.p.bon r.bon]
1368−We very simply give, along the duct we found above, a `%woot` card with
1369−the ship who sent us the ack and the ack itself. This allows the
1370−application to decide what to do about the result. In case of a failure,
1371−we usually either resend the message or display it to the user.
1372−Sometimes, we recognize the error term and handle it internally. In any
1373−case, the decision of how to handle the acknowledgment is entirely up to
1374−the application. Our job is done.
1376−Well, except that we skipped `++bick:pu`. Let's go back to that.
1380− |= [now=@da fap=flap] :: ack by hash
1381− ^- [[p=(unit soup) q=(list rock)] _+>]
1382− =+ sun=(~(get by diq) fap)
1385− =. diq (~(del by diq) fap)
1386− =^ gub +>.$ (bock now u.sun)
1387− =^ yop +>.$ (harv now)
1391−If you recall, in `++whap:pu` we created the packet pump's
1392−representation of the message, which included putting the message into
1393−`diq`, which maps from packet hashes to packet sequence numbers. Thus,
1394−`u.sun` is the sequence number of this particular message.
1396−We delete this message from `diq` since we have now received an ack for
1397−it. We call `++bock` to perform the ack by sequence number. We call
1398−`++harv` to harvest the packet queue, sending any messages that are now
1401−In `++bock`, there are three arms we haven't seen before: `++bine`,
1402−`+wept`, and `++beet`. We'll describe each of these before we get to
1403−`++bock`. `++bine` looks scariest.
1407− |= [now=@da num=@ud] :: apply ack
1408− ^- [(unit soup) _+>]
1411− ?: (gth num p.n.puq)
1412− =+ lef=$(puq l.puq)
1413− [-.lef +.lef(puq [n.puq puq.lef r.puq])]
1414− =+ rig=$(puq r.puq)
1415− [-.rig +.rig(puq [n.puq l.puq puq.rig])]
1416− =: rtt ?. &(liv.q.n.puq =(1 nux.q.n.puq)) rtt
1417− =+ gap=(sub now lys.q.n.puq)
1418− :: ~& [%bock-trip num (div gap (div ~s1 1.000))]
1419− (div (add (mul 2 rtt) gap) 3)
1420− nif (sub nif !liv.q.n.puq)
1422− =+ lez=(dec (need (~(get by pyz) gom.q.n.puq)))
1425− [[~ gom.q.n.puq] (~(del by pyz) gom.q.n.puq)]
1426− [~ (~(put by pyz) gom.q.n.puq lez)]
1428− +>.$(puq ~(nap to puq))
1431−The first few lines are simply looking through the packet queue until we
1432−find the correct packet to ack. This is basic queue manipulation that
1433−operates directly on the treap structure of the queue. If you understand
1434−treap queues, the logic is easy to follow. Otherwise, just trust us that
1435−by the time we get to the `=:`, the packet with sequence number `num` is
1436−on the top of the packet queue (that is, at `n.puq`).
1438−We first update the round-trip time. If the packet is either not alive
1439−or had to be transmitted more than once, then we don't have any reliable
1440−way of calculating the round-trip time since we're unsure of exactly
1441−which transmission was acknowledged. Otherwise, the round-trip time is
1442−the difference between now and when the packet was last sent. We set
1443−`rtt` by a little weighted average where the previous smoothed RTT is
1444−weighted twice as much as the RTT of the current packet. Thus,
1445−`(2*rtt+gap)/3`. This gives us a nice smooth RTT that is somewhat
1446−resilient to outlier data while still being responsive to our
1447−ever-changing world.
1449−If the packet wasn't already dead, then we decrement the number of live
1450−packets, which may allow more packets to be sent.
1452−We decrement the number of unacknowledged packets in our `pyz` for this
1453−particular message. If you recall, this was set in `++whap` to the
1454−number of packets required to send a message.
1456−If that was the last packet in the messge that needed to be acked, then
1457−we delete the messgae reference from `pyz` and produce the id of the
1458−message. Otherwise, we simply update `pyz` with the new number of
1459−unacked messages. In either case, we remove the packet from the packet
1464− |= [fip=@ud lap=@ud] :: fip thru lap-1
1471− ?: (lth p.n.puq fip) ?~(l.puq . left)
1472− ?: (gte p.n.puq lap) ?~(r.puq . rigt)
1475− ?.(liv.q.n.puq . .(nif (dec nif), liv.q.n.puq |))
1479− ^+(. =+(lef=apse(puq l.puq) lef(puq [n.puq puq.lef r.puq])))
1482− ^+(. =+(rig=apse(puq r.puq) rig(puq [n.puq l.puq puq.rig])))
1486−The algorithm is a simple case of traversing the packet queue.
1487−Essentialy, we mark as dead all packets in the queue between `fip` and
1488−`(dec lap)`. We also update `nif`, the number of live packets. Lest you
1489−mourn too much the passing of these packets, know that they shall soon
1490−rise again. Recall that in `++bick` after the call to `++bock` we call
1491−`++harv`. This will resend the packets that have just been labeled dead.
1495− ^+ . :: advance unacked
1496− =- +(nep ?~(foh nus u.foh))
1500− ?: (lte p.n.puq nep) $(puq l.puq)
1501− =+ rig=$(puq r.puq)
1502− ?^(rig rig [~ p.n.puq])
1505−Here we search for the next expected packet number. Basically, we search
1506−the queue for the leftmost packet whose number is greater than the
1507−current `nep`. If we don't find any such packet, we just use the total
1508−number of packets sent.
1510−We can now dive into `++bock`, our last arm.
1514− |= [now=@da num=@ud] :: ack by sequence
1515− ^- [(unit soup) _+>]
1516− =^ gym +> (bine now num)
1519− =+ cam=(max 2 (div caw 2))
1520− :: ~& [%bock-hole num nep cam]
1521− beet:(wept(nep num, cag cam, caw cam) nep num)
1522− =. caw ?: (lth caw cag) +(caw)
1523− (add caw !=(0 (mod (mug now) caw)))
1525− :: ~& [%bock-fine num nif caw cag]
1527− :: ~& [%bock-fill num nif caw cag]
1531−First, we call `++bine` to apply the ack to the packet pump information.
1532−We produce `gym`, which, if it exists, is the id of the packet that was
1533−acked. If we received an ack for a packet later than the one we
1534−expected, then we halve the logical packet window and kill all the
1535−earlier packets so that they may be resent.
1537−Otherwise, we possibly increase the congestion window. If the window is
1538−less than the congestion threshold, then we increment the size of the
1539−window. Otherwise, we only increment one out of every `caw` times.
1541−If we received an ack for the packet we expected, then we simply advance
1542−`nep` with `++beet`. If we received an ack for a packet earlier than we
1543−expected, we do nothing.
1545−It may be hard to believe, but we are, in fact, done. The message has
1546−been sent, received, acknowledged, and the acknowledgment has been
1547−returned to the original sender. We hope it's clear that, while the
1548−process has been somewhat involved, the algorithms are not all that
1549−complicated. If you've read this far, you know `%ames`. The only other
1550−code involves initialization, timeouts, and the like.
1552−Below, we give detailed reference documentation for the data models
1557−### `++sufi:ames`, domestic host
1560− ++ sufi :: domestic host
1561− $: hoy=(list ship) :: hierarchy
1562− val=wund :: private keys
1563− law=will :: server will
1564− seh=(map hand ,[p=ship q=@da]) :: key cache
1565− hoc=(map ship dore) :: neighborhood
1569−This is the security state of a domestic server.
1571−`hoy` is a list of the ships directly above us in the hierarchy of
1572−ships. For example, for `~hoclur-bicrel`, this would be `~tasruc` and
1573−`~tug`. See `++sein:title`.
1575−`val` is a list of our private keys.
1577−`law` is our certificate, which is a list of the XXX
1581−`hoc` is a map of ships to `++dore:ames`. The stores all the security
1582−information about foreign ships. The keys to this map are the neighbors
1583−(ships we have been in contact with) of this domestic server.
1585−### `++wund:ames`, private keys
1588− ++ wund (list ,[p=life q=ring r=acru]) :: mace in action
1591−This is a list of our own private keys, indexed by life. The key itself
1592−is the `++ring`, and the `++acru:ames` is the encryption engine. We generate
1593−the `++acru:ames` from the private key by calling `++weur`. Thus, we can at
1594−any time regenerate our `++wund:ames` from a `++mace:ames`. The current crypto is
1595−at the head of the list and can be accessed with `++sen:as:go`.
1597−### `++ring`, private key
1600− ++ ring ,@ :: private key
1603−This is a private key. The first byte is reserved to identify the type
1604−of cryptography. Lower-case means public key, upper-case means public
1605−key, and the letter identifies which `++acru:ames` to use.
1607−### `++pass`, public key
1610− ++ pass ,@ :: public key
1613−This is a public key. The first byte is reserved to identify the type of
1614−cryptography. Lower-case means public key, upper-case means public key,
1615−and the letter identifies which `++acru:ames` to use.
1617−### `++mace:ames`, private secrets
1620− ++ mace (list ,[p=life q=ring]) :: private secrets
1623−This is a list of the our private keys, indexed by life. From this we
1624−can generate a `++wund:ames` for actual use.
1626−### `++skin:ames`, encoding stem
1629− ++ skin ?(%none %open %fast %full) :: encoding stem
1632−This defines the type of encryption used for each message. `%none`
1633−refers to messages sent in the clear, `%open` refers to signed messages,
1634−`%full` refers to sealed messages, and `%fast` refers to symmetrically
1635−encrypted messages. See `++acru:ames` for details.
1637−### `++acru:ames`, asymmetric cryptosuite
1640− ++ acru :: asym cryptosuite
1641− $_ ^? |% :: opaque object
1642− ++ as ^? :: asym ops
1643− |% ++ seal |=([a=pass b=@ c=@] _@) :: encrypt to a
1644− ++ sign |=([a=@ b=@] _@) :: certify as us
1645− ++ sure |=([a=@ b=@] *(unit ,@)) :: authenticate from us
1646− ++ tear |= [a=pass b=@] :: accept from a
1647− *(unit ,[p=@ q=@]) ::
1649− ++ de |+([a=@ b=@] *(unit ,@)) :: symmetric de, soft
1650− ++ dy |+([a=@ b=@] _@) :: symmetric de, hard
1651− ++ en |+([a=@ b=@] _@) :: symmetric en
1653− |% ++ fig _@uvH :: fingerprint
1654− ++ pac _@uvG :: default passcode
1655− ++ pub *pass :: public key
1656− ++ sec *ring :: private key
1658− ++ nu ^? :: reconstructors
1659− |% ++ pit |=([a=@ b=@] ^?(..nu)) :: from [width seed]
1660− ++ nol |=(a=@ ^?(..nu)) :: from naked ring
1661− ++ com |=(a=@ ^?(..nu)) :: from naked pass
1666−This is an opaque interface for a general asymmetric cryptosuite. Any
1667−form of asymmetric cryptography can be dropped in to be used instead of
1668−the default. Right now, there is one cryptosuite: `++crub:crypto:crypto`,
1669−which is elliptic-curve cryptography.
1671−#### `++as:acru:ames`, asymmetric operations
1674− ++ as ^? :: asym ops
1675− |% ++ seal |=([a=pass b=@ c=@] _@) :: encrypt to a
1676− ++ sign |=([a=@ b=@] _@) :: certify as us
1677− ++ sure |=([a=@ b=@] *(unit ,@)) :: authenticate from us
1678− ++ tear |= [a=pass b=@] :: accept from a
1679− *(unit ,[p=@ q=@]) ::
1683−This is the core that defines the standard asymmetric cryptography
1686−`++seal:as:acru:ames` allows us to send a message encrypted with someone's
1687−public key so that only they may read it. If Alice seals a message with
1688−Bob's public key, then she can be sure that Bob is the only one who can
1689−read it. This is associated with the `++skin:ames` `%full`.
1691−`++sign:as:acru:ames` allows us to sign a message with our private key so
1692−that others can verify that we sent the message. If Alice signs a
1693−message with her private key, then Bob can verify with her public key
1694−that it was indeed Alice who sent it. This is associated with the
1695−`++skin:ames` `%open`.
1697−`++sure:as:acru:ames` is the dual to `++sign:as:acru:ames`. It allows us to verify
1698−that a message we have received is indeed from the claimed sender. If
1699−Alice sends a message with her private key, then Bob can use this arm to
1700−verify that it was indeed Alice who sent it. This is associated with the
1701−`++skin:ames` `%open`.
1703−`++tear:as:acru:ames` is the dual to `++seal:as:acru:ames`. It allows us to read a
1704−message that we can be sure is only read by us. If Alice seals a message
1705−with Bob's public key, then Bob can use this arm to read it. This is
1706−associated with the `++skin:ames` `%full`.
1708−#### `++de:acru:ames`, `++dy:acru:ames`, and `++en:acru:ames`, symmetric encryption/decryption
1711− ++ de |+([a=@ b=@] *(unit ,@)) :: symmetric de, soft
1712− ++ dy |+([a=@ b=@] _@) :: symmetric de, hard
1713− ++ en |+([a=@ b=@] _@) :: symmetric en
1716−Symmetric encryption is associated with the `++skin:ames` `%fast`.
1718−`++de:acru:ames` decrypts a message with a symmetric key, returning `~` on
1719−failure and `[~ u=data]` on success.
1721−`++dy:acru:ames` decrypts a message with a symmetric key, crashing on
1722−failure. This should almost always be defined as, and should always be
1723−semantically equivalent to, `(need (de a b))`.
1725−`++en:acru:ames` encrypts a message with a symmetric key.
1727−#### `++ex:acru:ames`, exporting data
1731− |% ++ fig _@uvH :: fingerprint
1732− ++ pac _@uvG :: default passcode
1733− ++ pub *pass :: public key
1734− ++ sec *ring :: private key
1738−`++fig:ex:acru:ames` is our fingerprint, usually a hash of our public key.
1739−This is used, for example, in `++zeno`, where every carrier owner's
1740−fingerprint is stored so that we can ensure that carriers are indeed
1741−owned by their owners
1743−`++pac:ex:acru:ames` is our default passcode, which is unused at present.
1745−`++pub:ex:acru:ames` is the `++pass` form of our public key.
1747−`++sec:ex:acru:ames` is the `++ring` form of our private key.
1749−#### `++nu:acru:ames`, reconstructors
1752− ++ nu ^? :: reconstructors
1753− |% ++ pit |=([a=@ b=@] ^?(..nu)) :: from [width seed]
1754− ++ nol |=(a=@ ^?(..nu)) :: from naked ring
1755− ++ com |=(a=@ ^?(..nu)) :: from naked pass
1759−These arms allow us to reconstruct a `++acru:ames` from basic data.
1761−`++pit:nu:acru:ames` constructs a `++acru:ames` from the width of our intended key
1762−and seed entropy. This is usually used in the initial construction of
1765−`++nol:nu:acru:ames` constructs a `++acru:ames` from a "naked ring", meaning a
1766−`++ring` without the initial byte identifying the type of crypto.
1768−`++com:nu:acru:ames` constructs a `++acru:ames` from a "naked pass", meaning a
1769−`++ring` without the initial byte identifying the type of crypto.
1771−### `++will`, certificate
1774− ++ will (list deed) :: certificate
1777−This is a list of deeds associated with the current ship. There should
1778−be an item in this list for every ship from this point up in the
1779−hierarchy times the number of lives that each ship has had. For example,
1780−\~hoclur-bicrel may have a will with three items: one for itself, one
1781−for \~tasruc (who issued \~hoclur-bicrel's deed) and one for \~tug (who
1782−issued \~tasruc's deed).
1784−### `++deed`, identity
1787− ++ deed ,[p=@ q=step r=?] :: sig, stage, fake?
1790−`p` is the signature of a particular deed, which is a signed copy of
1793−`q` is the stage in the identity.
1795−`r` is true if we're working on a fake network, where we don't check
1796−that the carrier fingerprints are correct. This allows us to create fake
1797−networks for development without interfering with the real network.
1799−### `++step:ames`, identity stage
1802− ++ step ,[p=bray q=gens r=pass] :: identity stage
1805−This is a single stage in our identity. Thus, this is specific to a
1806−single life in a single ship. Everything in here may change between
1813−`r` is the public key for this stage in the identity.
1818− ++ bray ,[p=life q=(unit life) r=ship s=@da] :: our parent us now
1823−### `++gens:ames`, general identity
1826− ++ gens ,[p=lang q=gcos] :: general identity
1829−`p` is the IETF language code for the preferred language of this
1830−identity. This is unused at the moment, but in the future text should be
1831−localized based on this.
1833−`q` is the description of the ship.
1835−### `++gcos:ames`, identity description
1838− ++ gcos :: id description
1839− $% [%czar ~] :: 8-bit ship
1840− [%duke p=what] :: 32-bit ship
1841− [%earl p=@t] :: 64-bit ship
1842− [%king p=@t] :: 16-bit ship
1843− [%pawn p=(unit ,@t)] :: 128-bit ship
1847−This is the description of the identity of a ship. Most types of
1848−identity have a `@t` field, which is their human-readable name. The
1849−identity of a `%duke` is more involved.
1851−A `%czar`, a carrier, is a ship with an 8-bit address. Thus, there are
1852−only 256 carriers. These are at the top of the namespace hierarchy, and
1853−the fingerprint of each carrier is stored in `++zeno`. These are the
1854−"senators" of Urbit.
1856−A `%king`, a cruiser, is a ship with a 16-bit address. Thus, there are
1857−65,536 cruisers. Each carrier may issue 256 cruisers. These are the
1858−infrastructure of Urbit.
1860−A `%duke`, a destroyer, is a ship with a 32-bit address. Thus, there are
1861−4,294,967,296 destroyers. Each cruiser may issue 65,536 cruisers. These
1862−are the individuals of Urbit.
1864−A `%earl`, a yacht, is a ship with a 64-bit address. Thus, there are
1865−18,446,744,073,709,551,616 yachts. Each destroyer may issue
1866−4,294,967,296 yachts. These are the devices of Urbit.
1868−A `%pawn`, a submarine, is a ship with a 128-bit address. Thus, there
1869−are a lot of submarines. The chance of random name collision is
1870−negligible, so submarines are not issued by any ship. They must simply
1871−assert their presence, and they are all considered children of \~zod.
1872−This is the underworld of Urbit, where anonymity reigns supreme.
1874−### `++what:ames`, logical destroyer identity
1877− ++ what :: logical identity
1878− $% [%anon ~] :: anonymous
1879− [%lady p=whom] :: female person ()
1880− [%lord p=whom] :: male person []
1881− [%punk p=sect q=@t] :: opaque handle ""
1885−This is the logical identity of a destroyer.
1887−A `%anon` is a completely anonymous destroyer. The difference between
1888−this and a submarine is that a submarine is ephemeral while a `%anon`
1889−destroyer is not. Thus, we may not know who \~hoclur-bicrel is, but we
1890−do know that it's always the same person.
1892−A `%lady` is a female person. The name used here should be a real name.
1894−A `%lord` is a male person. The name used here should be a real name.
1896−A `%punk` is a person who is identified only by a handle.
1898−### `++whom:ames`, real person
1901− ++ whom ,[p=@ud q=govt r=sect s=name] :: year/govt/id
1904−Ths is the information associated with a real person. It is mostly
1905−information that could be observed with the briefest of interactions.
1907−`p` is the birth year.
1909−`q` is the location of a user, usually of the form "country/zip".
1911−`r` is the sect of the user.
1913−`s` is the real name of the person.
1918− ++ govt path :: country/postcode
1921−This is the location of the user, usually of the form "country/zip".
1926− ++ sect ?(%black %blue %red %orange %white) :: banner
1934− ++ name ,[p=@t q=(unit ,@t) r=(unit ,@t) s=@t] :: first mid/nick last
1937−This is the given name, possible middle name/initial, possible nickname,
1938−and surname of a user.