@@ -25,6 +25,9 @@ protocol itself, including how to route incoming packets to the correct
25vane or app, is defined in Ames.
25vane or app, is defined in Ames.
27## Technical Overview
27## Technical Overview
28+This section summarizes the design of Ames. Beyond this section are deeper
29+elaborations on the concepts presented here.
28Ames extends [Arvo's](@/docs/tutorials/arvo/arvo.md) `%pass`/`%give` `move` semantics across the network.
31Ames extends [Arvo's](@/docs/tutorials/arvo/arvo.md) `%pass`/`%give` `move` semantics across the network.
30Ames receives packets as Arvo events and emits packets as Arvo
33Ames receives packets as Arvo events and emits packets as Arvo
@@ -78,7 +81,7 @@ large. A naxplanation can only give rise to a positive ack --
78never ack an ack, and never nack a naxplanation.
81never ack an ack, and never nack a naxplanation.
80Ames guarantees a total ordering of messages within a "flow",
83Ames guarantees a total ordering of messages within a "flow",
81−identified in other vanes by a duct and over the wire by a "bone":
84+identified in other vanes by a duct and over the wire by a `bone`:
82an opaque number. Each flow has a FIFO (first-in-first-out) queue of `%plea` requests
85an opaque number. Each flow has a FIFO (first-in-first-out) queue of `%plea` requests
83from the requesting ship to the responding ship and a FIFO queue
86from the requesting ship to the responding ship and a FIFO queue
84of `%boon`'s in the other direction.
87of `%boon`'s in the other direction.
@@ -112,21 +115,22 @@ ducts as unique keys. Stop tracking a peer by sending Ames a
113Debug output can be adjusted using `%sift` and `%spew` `$task`'s.
116Debug output can be adjusted using `%sift` and `%spew` `$task`'s.
115−## Packet structure
118+## Packets
117Ames datagram packets are handled as nouns internally by Arvo but as serial data
120Ames datagram packets are handled as nouns internally by Arvo but as serial data
118−by Unix. Here we document their serial structure.
121+by Unix. In this section we describe how packets are formed, serialized, and relayed.
119−For further information, see `+encode-packet` and `+decode-packet` in `ames.hoon`.
122+
121There is a 32-bit header followed by a variable width body.
125There is a 32-bit header followed by a variable width body.
123−### Header
127+#### Header
125The 32-bit header is given by the following data, presented in order:
129The 32-bit header is given by the following data, presented in order:
127 - 3 bits: Ames `protocol-version`,
131 - 3 bits: Ames `protocol-version`,
128 - 20 bits: a checksum as a truncated insecure hash of the body, done with
132 - 20 bits: a checksum as a truncated insecure hash of the body, done with
129− `+mug`,
133+ [`+mug`](@/docs/reference/library/2e.md#mug),
130 - 2 bits: the bit width of the sender address encoded as a 2-bit enum,
134 - 2 bits: the bit width of the sender address encoded as a 2-bit enum,
131 - 2 bits: the bit width of the receiver address encoded as a 2-bit enum,
135 - 2 bits: the bit width of the receiver address encoded as a 2-bit enum,
132 - 1 bit: whether the packet is encrypted or not,
136 - 1 bit: whether the packet is encrypted or not,
@@ -134,18 +138,286 @@ The 32-bit header is given by the following data, presented in order:
135 Every packet sent between ships is encrypted except for self-signed attestation packets from 128-bit comets.
139 Every packet sent between ships is encrypted except for self-signed attestation packets from 128-bit comets.
137−### Body
141+#### Body
139The body is of variable length and consists of three parts in this order:
143The body is of variable length and consists of three parts in this order:
141 - The `@p` of the sending ship,
145 - The `@p` of the sending ship,
142 - The `@p` of the receiving ship,
146 - The `@p` of the receiving ship,
143− - The payload, which is the `+jam` (i.e. serialization) of the noun `[origin content]`.
147+ - The payload, which is the [`+jam`](@/docs/reference/libary/2p.md#jam) (i.e. serialization) of the noun `[origin content]`.
145 `origin` is the IP and port of the original sender if the packet was proxied
149 `origin` is the IP and port of the original sender if the packet was proxied
146 through a relay and null otherwise. `content` is a noun that is either an encrypted ack or an
150 through a relay and null otherwise. `content` is a noun that is either an encrypted ack or an
147 encrypted message fragment, unless it is a comet attestation packet in which
151 encrypted message fragment, unless it is a comet attestation packet in which
148− case it is unencrypted.
152+ case it is unencrypted. `content` is always 1kB in size or less.
150 The sender and receiver live outside of the jammed data section to simplify
154 The sender and receiver live outside of the jammed data section to simplify
151 packet filtering for the interpreter.
155 packet filtering for the interpreter.
159+When Ames has a message to be sent it must first determine how many packets are
160+required to send the message. To do this, it first `+jam`s the
161+message, producing an atom. Ames checks how large the atom is, and if it is
162+bigger than a kilobyte it will split it into packets whose payloads are 1 kB or
163+less. It then numbers each one - this is message 17, packet 12, this is message
164+17, packet 13, etc., so that when the receiver receives these packets it knows
165+which number they are. Finally it encrypts each individual packet and enqueues
166+them to be sent along their stated flow.
168+Network packets aren't always received in order, so this numbering is important
169+for reconstruction, and also packets may get lost. So Ames does transmission
170+control (the TC in TCP) to solve this problem. It makes sure that all packets
171+eventually get through, and when the other side gets them it can put them in the
172+correct order. If Ames doesn't get an ack on a packet then it will resend it
173+until it does. The logic for determining how many packets to send or re-send at what time is performed by an Ames-specific variant of TCP's "NewReno" congestion control algorithm.
175+As each packet in a message is received, Ames decrypts it and stores the message fragment. Once it's received every packet for a message, Ames concatenates the fragments back into a single large atom and uses `+cue` to deserialize that back into the original message noun.
181+In this section we discuss acks and nacks. In Ames, an "ack", short for "acknowledgment", is a small packet attesting that a piece of information (either a packet or a whole message) was received. Ames makes use of
182+acks to maintain synchronization between two communicating parties. Nacks are 'negative
183+acknowledgments' and are used when something goes wrong.
187+Every message (i.e. a `%plea` or `%boon`) is split up by Ames into some number
188+of _fragments_ that are 1kB in size or less. The fragments are then encrypted
189+and encapsulated into
190+packets and sent along a flow. The message will be considered successfully
191+received once the sender has received the appropriate set of acks in response,
194+There are two types of acks: fragment acks and message acks. Acks are
195+not considered messages, and thus are not `%plea`s or `%boon`s. Given a message split into N
196+fragments, the sender of the message will expect N-1 fragment acks followed
197+by exactly one message ack (ignoring any duplicate packets, which are idempotent
198+from the perspective of the application). This is because the receiver
199+will send a fragment ack for the first N-1 packets it received, and what
200+would have been the final fragment ack will instead be a message ack.
202+Acks are considered to be part of the flow in which that `%plea` or
203+`%boon` lives, as the packets containing their fragments and packets acking the
204+receipt of those packets are considered to be what makes up a given message.
205+Thus a message-level ack must be received before the next message on the flow
206+can begin. The full story is more complicated than this; see the section on
212+A nack indicates a negative acknowledgement to a `%plea`, meaning that the
213+requested action was not performed.
215+`%boon`s and naxplanations are never nacked.
216+Individual packets are also never nacked, only complete `%plea` messages are.
217+Whether a malformed packet causes a `%plea` to be nacked depends on its content.
218+In the case of an incorrect checksum or a failure to decrypt then Ames drops the
219+packet and it is as though it never happened. However, if the packet does
220+decrypt and has invalid ciphertext (i.e. something other than a jammed
221+`$shutpacket` data structure), then Ames will nack the whole `%plea` since that
222+indicates that the peer is misbehaving. Eventually Ames should also present a
223+warning to the user that the peer is untrustworthy.
225+A nack will be accompanied by a naxplanation, which is a special type of`%boon`
226+that uses its own `bone` (see [flows](#flows)). Ames won't give the vane that requested the
227+initial `%plea` a nack until it also receives the naxplanation, which it will send to
228+the vane as a `%done` gift. Naxplanations may only be acked, never nacked.
229+Furthermore, naxplanations can only ever be sent as a rejection of a `%plea` -
230+the receiver will never both perform a `%plea` and return a naxplanation.
234+A fragment ack's contents (before encryption) are a pure function of the
237+[our-life her-life bone message-num fragment-num]
239+This means all re-sends of an ack packet will be bitwise identical to each other, unless one of the peers changes its encryption keys.
241+Each datum in this noun is an atom with the aura `@ud` or an aura that nests
244+Here, `our-life` refers to the [`life`](@/docs/glossary/breach), or revision
245+number, of the acking ship's networking keys, and `her-life` is the `life` of
246+the ack-receiving ship's networking keys. `bone` is an opaque number identifying
247+the flow. `message-num` denotes the number of the
248+message in the flow identified by `bone`. `fragment-num` denotes the number of
249+the fragment of the message identified by `message-num` that is being acked.
251+A message (n)ack is a different kind of ack that is obtained by encrypting the
252+`+jam` of the following noun:
255+[our-life her-life bone message-num ok=? lag=@dr]
257+`ok` is a flag that is set to `%.y` for a message ack and `%.n` for a message
258+nack. In the future, `lag` will be used to
259+describe the time it took to process a message, but for now it is always zero.
264+Flows are asymmetric communication channels along which two ships send and
265+receive packets, and all Ames packets are part of some flow.
267+To create a new flow, a ship sends a
268+`%plea` to another ship. Only the creator of the flow may send `%plea`s, and the
269+other party may only send `%boon`s. In this sense flows are asymmetric.
271+The sequence of `%plea`s in a flow is totally ordered, though
272+the packets which make them up need not be. `%boon`s are totally ordered in the
273+same fashion, but there is not a coordinated ordering between `%plea`s and
274+`%boon`s in a given flow beyond the implicit one arising from the fact that a
275+`%boon` is always a response to a `%plea`. A `%plea` can give rise to zero, one,
278+Inside of Ames, each flow has four sequential opaque `@ud`s
279+called bones that are unique to that flow. Thus the flow itself is often
280+referred to by its first bone. Each bone is a one-way street
281+for packets to travel along, so e.g. acks to packets making up a `%plea` are
282+sent along a different bone than the `%plea`.
284+We give an example of such a partition. Let flow 12 be a flow between `~bacbel-tagfeg` and `~worwel-sipnum` with
285+`~bacbel-tagfeg` as the `%plea` sender. Then bones 12-15 are associated with the
286+flow, and the types of packets for each bone are:
288+ - bone 12, `%plea`s and acks to `~worwel-sipnum`,
289+ - bone 13, `%boon`s and (n)acks to `~bacbel-tagfeg`,
290+ - bone 14, acks of naxplanations to `~worwel-sipnum`,
291+ - bone 15, naxplanations to `~bacbel-tagfeg`.
293+Each bone is handled separately by congestion control, and this is one reason
294+for their segregation. For example, say a two-packet `%plea` is sent on bone 12,
295+with `~bacbel-tagfeg` requesting to join a group on `~worwel-sipnum`.
296+Then `~worwel-sipnum` can ack the first packet on bone 13, and then send a nack
297+on bone 13 as well. A nack by itself contains no information as to why the nack
298+happened. Then `~worwel-sipnum` can also send a naxplanation on bone 15 saying
299+to `~bacbel-tagfeg` that they cannot join the group, to which an ack is received
302+Another reason to separate bones is to avoid race conditions. If
303+`~worwel-sipnum` and `~bacbel-tagfeg` attempt to start a flow with each other at
304+the same time we do not wish there to be a conflict when they receive each
305+others' packets. Flipping the last bit of the bone based on whether a packet is
306+an ack or a message fragment allows for `~worwel-sipnum` to create flow 8 with
307+`~bacbel-tagfeg` without coming into conflict with the flow 8 `~bacbel-tagfeg`
308+created for `~worwel-sipnum`.
310+`%plea`s and `%boon`s are handled on separate bones so that e.g. sending a large
311+`%boon` doesn't stop an additional `%plea` from being received. It is also
312+simply cleaner to make each bone one-way. Acks and nacks are very small packets
313+as well as integral to controlling the total orderings on `%plea`s and `%boon`s
314+and so they are included on these bones as well.
316+Naxplanations can potentially be very large, as they often contain things like stack
317+traces or other crash reports. Thus it is undesirable to have them share a bone
318+with `%boon`s and create congestion there. So naxplanations have their own bone,
319+and so do acks to packets that make up a naxplanation, as well as the
320+message-level naxplanation ack.
324+Here we describes how the Ames network relays packets. We ignore all details
325+about UDP, which occurs at a lower layer than Ames.
327+When a ship first contacts another ship, it may only know its `@p`, but an IP
328+address and port are necessary for peer-to-peer communication to occur.
329+Thus the initial correspondence between two ships, or additional correspondence
330+following a change in IP or port of one of the parties, will need to be relayed
331+by an intermediary that knows the IP and port of the receiving ship.
333+For now packet relaying is handled entirely by galaxies. Every galaxy is
334+responsible for knowing the IP and port of every planet underneath it. In the
335+future, galaxies will only need to know the star sponsoring a planet, and stars
336+will be responsible for the final step of the relay.
338+The following diagram summarizes the packet creation and forwarding process.
340+<div style="text-align:center">
341+<img src="https://media.urbit.org/docs/arvo/datagram-long.png">
344+We elaborate on the above diagram.
346+A typical relay will look something like this: `~bacbel-tagfeb` wishes to
347+contact `~worwel-sipnum` but does not know the IP address and port at which
348+`~worwel-sipnum` resides. Both planets live under `~zod`, which knows both of
349+their IP and port numbers by virtue of being the galaxy that both live under.
351+To prepare, `~bacbel-tagfeb` forms a Diffie-Hellman symmetric key with their own
352+private key and the public key of `~worwel-sipnum`, obtained via Jael. Then
353+`~bacbel-tagfeb` sends a packet addressed to `~worwel-sipnum` to `~zod`.
355+The packet has the following format:
356+ - The standard Ames header described in [header](#header), where the checksum
357+ is the `+mug` of the body, and the sender and receiver ship types are `01`,
358+ denoting that the sender and receiver are planets. The remainder of the
360+ - The body of this packet will be the `@p` of the
361+sender `~bacbel-tagfeb`, followed by the `@p` of the receiver `~worwel-sipnum`,
362+followed by the payload.
363+ - The payload of this packet will be the `+jam` of `[origin content]`. `origin`
364+for this initial packet will be `~`, which implies that the packet is
365+originating from the sender. `content` is the `+jam` of the message, encrypted
366+using the Diffie-Hellman symmetric key that was previously computed.
368+`~zod` receives the packet and reads the body. It sees that it is not the
369+intended recipient of the packet, and so gets ready to forward it to
370+`~worwel-sipnum`. First, it [`+cue`s](@/docs/reference/library/2p.md#cue)
371+(deserializes) the payload and changes the `origin` to the IP and port of
372+`~bacbel-tagfeb`. Then it `+jam`s `[origin content]` to form a new payload, and
373+`+mug`s that payload to get a new checksum. It replaces the old payload and
374+checksum with the new, and then sends the packet along to `~worwel-sipnum`.
376+Once `~worwel-sipnum` processes the packet, it will know the IP and port of
377+`~bacbel-tagfeb` since `~zod` included it when it forwarded the packet. Thus
378+`~worwel-sipnum` will send an ack packet directly to `~bacbel-tagfeb`, unless a
379+NAT and/or firewall prevents a direct peer-to-peer connection, in which case
380+`~zod` will continue to relay packets. Absent these factors preventing a
381+peer-to-peer connection, communication between the two ships will now be direct
382+until one of them changes their IP address, port, or networking keys.
385+## The Serf and the King
387+Urbit's functionality is split between the two binaries `urbit-worker` (sometimes
388+called the Serf) and `urbit-king` (sometimes called the King). This division of
389+labor is currently not well-documented outside of the [Vere documents](@/docs/tutorials/vere/_index.md), but we summarize it here.
391+In short, the Serf is the Nock runtime and so keeps track of the current state
392+of Arvo as a Nock noun and updates the state by `%poke`ing it with nouns, and
393+then informs the King of the new state. The King manages snapshots of the Arvo
394+state and handles I/O with Unix, among other things. The Serf only ever talks to
395+the King, while the King talks with both the Serf and Unix.
397+### Ames I/O submodule
399+The King has several submodules, one of them being an Ames I/O submodule. This
400+submodule is responsible for wrapping outgoing Ames packets as UDP packets, and
401+unwrapping incoming UDP packets into Ames packets and forwarding them to the
402+worker. It also maintains an incoming packet queue.
404+This division is summarized in the following diagram, describing how
405+`~bacbel-tagfeb` requests a subscription to the `recipes` notebook of
406+`~worwel-sipnum` in the Publish app.
408+<div style="text-align:center">
409+<img src="https://media.urbit.org/docs/arvo/packet.png">
412+Ames, as a part of Arvo, handles `+jam`ming, packetizing, encryption, and
413+forming Ames packets. Once it is ready to send an Ames packet, it `%give`s to
414+Unix a `%send` `gift` containing that packet. This will be a Nock noun
415+containing the `@tas` `%send` as well as the serialized packet.
417+"Unix", in this case, is actually the King. The King receives the `%send`
418+instruction, wraps the packet contained within as a UDP packet, and immediately
419+hands it off the the Unix network interface to be sent.
421+Now the receiving King is handed a UDP packet by Unix. The King removes the UDP
422+wrapper, `+jam`s the `lane` on which it heard the packet, and delivers the packet
423+to the Ames vane as an atom by copying the bytes it heard on the UDP port.