@@ -25,6 +25,9 @@ protocol itself, including how to route incoming packets to the correct
2525vane or app, is defined in Ames.
2626
2727## Technical Overview
28+This section summarizes the design of Ames. Beyond this section are deeper
29+elaborations on the concepts presented here.
30+
2831Ames extends [Arvo's](@/docs/tutorials/arvo/arvo.md) `%pass`/`%give` `move` semantics across the network.
2932
3033Ames receives packets as Arvo events and emits packets as Arvo
@@ -78,7 +81,7 @@ large. A naxplanation can only give rise to a positive ack --
7881never ack an ack, and never nack a naxplanation.
7982
8083Ames guarantees a total ordering of messages within a "flow",
81−identified in other vanes by a duct and over the wire by a "bone":
84+identified in other vanes by a duct and over the wire by a `bone`:
8285an opaque number. Each flow has a FIFO (first-in-first-out) queue of `%plea` requests
8386from the requesting ship to the responding ship and a FIFO queue
8487of `%boon`'s in the other direction.
@@ -112,21 +115,22 @@ ducts as unique keys. Stop tracking a peer by sending Ames a
112115
113116Debug output can be adjusted using `%sift` and `%spew` `$task`'s.
114117
115−## Packet structure
118+## Packets
116119
117120Ames datagram packets are handled as nouns internally by Arvo but as serial data
118−by Unix. Here we document their serial structure.
119−For further information, see `+encode-packet` and `+decode-packet` in `ames.hoon`.
121+by Unix. In this section we describe how packets are formed, serialized, and relayed.
122+
123+### Packet format
120124
121125There is a 32-bit header followed by a variable width body.
122126
123−### Header
127+#### Header
124128
125129The 32-bit header is given by the following data, presented in order:
126130
127131 - 3 bits: Ames `protocol-version`,
128132 - 20 bits: a checksum as a truncated insecure hash of the body, done with
129− `+mug`,
133+ [`+mug`](@/docs/reference/library/2e.md#mug),
130134 - 2 bits: the bit width of the sender address encoded as a 2-bit enum,
131135 - 2 bits: the bit width of the receiver address encoded as a 2-bit enum,
132136 - 1 bit: whether the packet is encrypted or not,
@@ -134,18 +138,286 @@ The 32-bit header is given by the following data, presented in order:
134138
135139 Every packet sent between ships is encrypted except for self-signed attestation packets from 128-bit comets.
136140
137−### Body
141+#### Body
138142
139143The body is of variable length and consists of three parts in this order:
140144
141145 - The `@p` of the sending ship,
142146 - The `@p` of the receiving ship,
143− - The payload, which is the `+jam` (i.e. serialization) of the noun `[origin content]`.
147+ - The payload, which is the [`+jam`](@/docs/reference/libary/2p.md#jam) (i.e. serialization) of the noun `[origin content]`.
144148
145149 `origin` is the IP and port of the original sender if the packet was proxied
146150 through a relay and null otherwise. `content` is a noun that is either an encrypted ack or an
147151 encrypted message fragment, unless it is a comet attestation packet in which
148− case it is unencrypted.
152+ case it is unencrypted. `content` is always 1kB in size or less.
149153
150154 The sender and receiver live outside of the jammed data section to simplify
151155 packet filtering for the interpreter.
156+
157+### Packeting
158+
159+When Ames has a message to be sent it must first determine how many packets are
160+required to send the message. To do this, it first `+jam`s the
161+message, producing an atom. Ames checks how large the atom is, and if it is
162+bigger than a kilobyte it will split it into packets whose payloads are 1 kB or
163+less. It then numbers each one - this is message 17, packet 12, this is message
164+17, packet 13, etc., so that when the receiver receives these packets it knows
165+which number they are. Finally it encrypts each individual packet and enqueues
166+them to be sent along their stated flow.
167+
168+Network packets aren't always received in order, so this numbering is important
169+for reconstruction, and also packets may get lost. So Ames does transmission
170+control (the TC in TCP) to solve this problem. It makes sure that all packets
171+eventually get through, and when the other side gets them it can put them in the
172+correct order. If Ames doesn't get an ack on a packet then it will resend it
173+until it does. The logic for determining how many packets to send or re-send at what time is performed by an Ames-specific variant of TCP's "NewReno" congestion control algorithm.
174+
175+As each packet in a message is received, Ames decrypts it and stores the message fragment. Once it's received every packet for a message, Ames concatenates the fragments back into a single large atom and uses `+cue` to deserialize that back into the original message noun.
176+
177+
178+
179+### Acks and Nacks
180+
181+In this section we discuss acks and nacks. In Ames, an "ack", short for "acknowledgment", is a small packet attesting that a piece of information (either a packet or a whole message) was received. Ames makes use of
182+acks to maintain synchronization between two communicating parties. Nacks are 'negative
183+acknowledgments' and are used when something goes wrong.
184+
185+#### Acks
186+
187+Every message (i.e. a `%plea` or `%boon`) is split up by Ames into some number
188+of _fragments_ that are 1kB in size or less. The fragments are then encrypted
189+and encapsulated into
190+packets and sent along a flow. The message will be considered successfully
191+received once the sender has received the appropriate set of acks in response,
192+defined as follows.
193+
194+There are two types of acks: fragment acks and message acks. Acks are
195+not considered messages, and thus are not `%plea`s or `%boon`s. Given a message split into N
196+fragments, the sender of the message will expect N-1 fragment acks followed
197+by exactly one message ack (ignoring any duplicate packets, which are idempotent
198+from the perspective of the application). This is because the receiver
199+will send a fragment ack for the first N-1 packets it received, and what
200+would have been the final fragment ack will instead be a message ack.
201+
202+Acks are considered to be part of the flow in which that `%plea` or
203+`%boon` lives, as the packets containing their fragments and packets acking the
204+receipt of those packets are considered to be what makes up a given message.
205+Thus a message-level ack must be received before the next message on the flow
206+can begin. The full story is more complicated than this; see the section on
207+[flows](#flows).
208+
209+
210+#### Nacks
211+
212+A nack indicates a negative acknowledgement to a `%plea`, meaning that the
213+requested action was not performed.
214+
215+`%boon`s and naxplanations are never nacked.
216+Individual packets are also never nacked, only complete `%plea` messages are.
217+Whether a malformed packet causes a `%plea` to be nacked depends on its content.
218+In the case of an incorrect checksum or a failure to decrypt then Ames drops the
219+packet and it is as though it never happened. However, if the packet does
220+decrypt and has invalid ciphertext (i.e. something other than a jammed
221+`$shutpacket` data structure), then Ames will nack the whole `%plea` since that
222+indicates that the peer is misbehaving. Eventually Ames should also present a
223+warning to the user that the peer is untrustworthy.
224+
225+A nack will be accompanied by a naxplanation, which is a special type of`%boon`
226+that uses its own `bone` (see [flows](#flows)). Ames won't give the vane that requested the
227+initial `%plea` a nack until it also receives the naxplanation, which it will send to
228+the vane as a `%done` gift. Naxplanations may only be acked, never nacked.
229+Furthermore, naxplanations can only ever be sent as a rejection of a `%plea` -
230+the receiver will never both perform a `%plea` and return a naxplanation.
231+
232+#### (N)ack packets
233+
234+A fragment ack's contents (before encryption) are a pure function of the
235+following noun:
236+```hoon
237+[our-life her-life bone message-num fragment-num]
238+```
239+This means all re-sends of an ack packet will be bitwise identical to each other, unless one of the peers changes its encryption keys.
240+
241+Each datum in this noun is an atom with the aura `@ud` or an aura that nests
242+under `@ud`.
243+
244+Here, `our-life` refers to the [`life`](@/docs/glossary/breach), or revision
245+number, of the acking ship's networking keys, and `her-life` is the `life` of
246+the ack-receiving ship's networking keys. `bone` is an opaque number identifying
247+the flow. `message-num` denotes the number of the
248+message in the flow identified by `bone`. `fragment-num` denotes the number of
249+the fragment of the message identified by `message-num` that is being acked.
250+
251+A message (n)ack is a different kind of ack that is obtained by encrypting the
252+`+jam` of the following noun:
253+
254+```hoon
255+[our-life her-life bone message-num ok=? lag=@dr]
256+```
257+`ok` is a flag that is set to `%.y` for a message ack and `%.n` for a message
258+nack. In the future, `lag` will be used to
259+describe the time it took to process a message, but for now it is always zero.
260+
261+
262+### Flows
263+
264+Flows are asymmetric communication channels along which two ships send and
265+receive packets, and all Ames packets are part of some flow.
266+
267+To create a new flow, a ship sends a
268+`%plea` to another ship. Only the creator of the flow may send `%plea`s, and the
269+other party may only send `%boon`s. In this sense flows are asymmetric.
270+
271+The sequence of `%plea`s in a flow is totally ordered, though
272+the packets which make them up need not be. `%boon`s are totally ordered in the
273+same fashion, but there is not a coordinated ordering between `%plea`s and
274+`%boon`s in a given flow beyond the implicit one arising from the fact that a
275+`%boon` is always a response to a `%plea`. A `%plea` can give rise to zero, one,
276+or many `%boon`s.
277+
278+Inside of Ames, each flow has four sequential opaque `@ud`s
279+called bones that are unique to that flow. Thus the flow itself is often
280+referred to by its first bone. Each bone is a one-way street
281+for packets to travel along, so e.g. acks to packets making up a `%plea` are
282+sent along a different bone than the `%plea`.
283+
284+We give an example of such a partition. Let flow 12 be a flow between `~bacbel-tagfeg` and `~worwel-sipnum` with
285+`~bacbel-tagfeg` as the `%plea` sender. Then bones 12-15 are associated with the
286+flow, and the types of packets for each bone are:
287+
288+ - bone 12, `%plea`s and acks to `~worwel-sipnum`,
289+ - bone 13, `%boon`s and (n)acks to `~bacbel-tagfeg`,
290+ - bone 14, acks of naxplanations to `~worwel-sipnum`,
291+ - bone 15, naxplanations to `~bacbel-tagfeg`.
292+
293+Each bone is handled separately by congestion control, and this is one reason
294+for their segregation. For example, say a two-packet `%plea` is sent on bone 12,
295+with `~bacbel-tagfeg` requesting to join a group on `~worwel-sipnum`.
296+Then `~worwel-sipnum` can ack the first packet on bone 13, and then send a nack
297+on bone 13 as well. A nack by itself contains no information as to why the nack
298+happened. Then `~worwel-sipnum` can also send a naxplanation on bone 15 saying
299+to `~bacbel-tagfeg` that they cannot join the group, to which an ack is received
300+on bone 14.
301+
302+Another reason to separate bones is to avoid race conditions. If
303+`~worwel-sipnum` and `~bacbel-tagfeg` attempt to start a flow with each other at
304+the same time we do not wish there to be a conflict when they receive each
305+others' packets. Flipping the last bit of the bone based on whether a packet is
306+an ack or a message fragment allows for `~worwel-sipnum` to create flow 8 with
307+`~bacbel-tagfeg` without coming into conflict with the flow 8 `~bacbel-tagfeg`
308+created for `~worwel-sipnum`.
309+
310+`%plea`s and `%boon`s are handled on separate bones so that e.g. sending a large
311+`%boon` doesn't stop an additional `%plea` from being received. It is also
312+simply cleaner to make each bone one-way. Acks and nacks are very small packets
313+as well as integral to controlling the total orderings on `%plea`s and `%boon`s
314+and so they are included on these bones as well.
315+
316+Naxplanations can potentially be very large, as they often contain things like stack
317+traces or other crash reports. Thus it is undesirable to have them share a bone
318+with `%boon`s and create congestion there. So naxplanations have their own bone,
319+and so do acks to packets that make up a naxplanation, as well as the
320+message-level naxplanation ack.
321+
322+### Packet relaying
323+
324+Here we describes how the Ames network relays packets. We ignore all details
325+about UDP, which occurs at a lower layer than Ames.
326+
327+When a ship first contacts another ship, it may only know its `@p`, but an IP
328+address and port are necessary for peer-to-peer communication to occur.
329+Thus the initial correspondence between two ships, or additional correspondence
330+following a change in IP or port of one of the parties, will need to be relayed
331+by an intermediary that knows the IP and port of the receiving ship.
332+
333+For now packet relaying is handled entirely by galaxies. Every galaxy is
334+responsible for knowing the IP and port of every planet underneath it. In the
335+future, galaxies will only need to know the star sponsoring a planet, and stars
336+will be responsible for the final step of the relay.
337+
338+The following diagram summarizes the packet creation and forwarding process.
339+
340+<div style="text-align:center">
341+<img src="https://media.urbit.org/docs/arvo/datagram-long.png">
342+</div>
343+
344+We elaborate on the above diagram.
345+
346+A typical relay will look something like this: `~bacbel-tagfeb` wishes to
347+contact `~worwel-sipnum` but does not know the IP address and port at which
348+`~worwel-sipnum` resides. Both planets live under `~zod`, which knows both of
349+their IP and port numbers by virtue of being the galaxy that both live under.
350+
351+To prepare, `~bacbel-tagfeb` forms a Diffie-Hellman symmetric key with their own
352+private key and the public key of `~worwel-sipnum`, obtained via Jael. Then
353+`~bacbel-tagfeb` sends a packet addressed to `~worwel-sipnum` to `~zod`.
354+
355+The packet has the following format:
356+ - The standard Ames header described in [header](#header), where the checksum
357+ is the `+mug` of the body, and the sender and receiver ship types are `01`,
358+ denoting that the sender and receiver are planets. The remainder of the
359+ packet is the body.
360+ - The body of this packet will be the `@p` of the
361+sender `~bacbel-tagfeb`, followed by the `@p` of the receiver `~worwel-sipnum`,
362+followed by the payload.
363+ - The payload of this packet will be the `+jam` of `[origin content]`. `origin`
364+for this initial packet will be `~`, which implies that the packet is
365+originating from the sender. `content` is the `+jam` of the message, encrypted
366+using the Diffie-Hellman symmetric key that was previously computed.
367+
368+`~zod` receives the packet and reads the body. It sees that it is not the
369+intended recipient of the packet, and so gets ready to forward it to
370+`~worwel-sipnum`. First, it [`+cue`s](@/docs/reference/library/2p.md#cue)
371+(deserializes) the payload and changes the `origin` to the IP and port of
372+`~bacbel-tagfeb`. Then it `+jam`s `[origin content]` to form a new payload, and
373+`+mug`s that payload to get a new checksum. It replaces the old payload and
374+checksum with the new, and then sends the packet along to `~worwel-sipnum`.
375+
376+Once `~worwel-sipnum` processes the packet, it will know the IP and port of
377+`~bacbel-tagfeb` since `~zod` included it when it forwarded the packet. Thus
378+`~worwel-sipnum` will send an ack packet directly to `~bacbel-tagfeb`, unless a
379+NAT and/or firewall prevents a direct peer-to-peer connection, in which case
380+`~zod` will continue to relay packets. Absent these factors preventing a
381+peer-to-peer connection, communication between the two ships will now be direct
382+until one of them changes their IP address, port, or networking keys.
383+
384+
385+## The Serf and the King
386+
387+Urbit's functionality is split between the two binaries `urbit-worker` (sometimes
388+called the Serf) and `urbit-king` (sometimes called the King). This division of
389+labor is currently not well-documented outside of the [Vere documents](@/docs/tutorials/vere/_index.md), but we summarize it here.
390+
391+In short, the Serf is the Nock runtime and so keeps track of the current state
392+of Arvo as a Nock noun and updates the state by `%poke`ing it with nouns, and
393+then informs the King of the new state. The King manages snapshots of the Arvo
394+state and handles I/O with Unix, among other things. The Serf only ever talks to
395+the King, while the King talks with both the Serf and Unix.
396+
397+### Ames I/O submodule
398+
399+The King has several submodules, one of them being an Ames I/O submodule. This
400+submodule is responsible for wrapping outgoing Ames packets as UDP packets, and
401+unwrapping incoming UDP packets into Ames packets and forwarding them to the
402+worker. It also maintains an incoming packet queue.
403+
404+This division is summarized in the following diagram, describing how
405+`~bacbel-tagfeb` requests a subscription to the `recipes` notebook of
406+`~worwel-sipnum` in the Publish app.
407+
408+<div style="text-align:center">
409+<img src="https://media.urbit.org/docs/arvo/packet.png">
410+</div>
411+
412+Ames, as a part of Arvo, handles `+jam`ming, packetizing, encryption, and
413+forming Ames packets. Once it is ready to send an Ames packet, it `%give`s to
414+Unix a `%send` `gift` containing that packet. This will be a Nock noun
415+containing the `@tas` `%send` as well as the serialized packet.
416+
417+"Unix", in this case, is actually the King. The King receives the `%send`
418+instruction, wraps the packet contained within as a UDP packet, and immediately
419+hands it off the the Unix network interface to be sent.
420+
421+Now the receiving King is handed a UDP packet by Unix. The King removes the UDP
422+wrapper, `+jam`s the `lane` on which it heard the packet, and delivers the packet
423+to the Ames vane as an atom by copying the bytes it heard on the UDP port.