ames: update comment docs

pkg/arvo/sys/vane/ames.hoon

@@ -1,55 +1,89 @@
 ::    Ames extends Arvo's %pass/%give move semantics across the network.
 ::
-::    A "forward flow" message, which is like a request, is passed to
+::    Ames receives packets as Arvo events and emits packets as Arvo
-::    Ames from a local vane.  Ames transmits the message to the peer's
+::    effects.  The runtime is responsible for transferring the bytes in
-::    Ames, which passes the message to the destination vane.
+::    an Ames packet across a physical network to another ship.
 ::
-::    Once the peer has processed the "forward flow" message, it sends a
+::    The runtime tells Ames which physical address a packet came from,
-::    message acknowledgment over the wire back to the local Ames.  This
+::    represented as an opaque atom.  Ames can emit a packet effect to
-::    ack can either be positive or negative, in which case we call it a
+::    one of those opaque atoms or to the Urbit address of a galaxy
-::    "nack".  (Don't confuse Ames nacks with TCP nacks, which are a
+::    (root node), which the runtime is responsible for translating to a
-::    different concept).
+::    physical address.  One runtime implementation sends UDP packets
+::    using IPv4 addresses for ships and DNS lookups for galaxies, but
+::    other implementations may overlay over other kinds of networks.
 ::
-::    When the local Ames receives either a positive message ack or a
+::    A local vane can pass Ames a %plea request message.  Ames
-::    combination of a nack and nack-trace (explained in more detail
+::    transmits the message over the wire to the peer ship's Ames, which
+::    passes the message to the destination vane.
+::
+::    Once the peer has processed the %plea message, it sends a
+::    message-acknowledgment packet over the wire back to the local
+::    Ames.  This ack can either be positive to indicate the request was
+::    processed, or negative to indicate the request failed, in which
+::    case it's called a "nack".  (Don't confuse Ames nacks with TCP
+::    nacks, which are a different concept).
+::
+::    When the local Ames receives either a positive message-ack or a
+::    combination of a nack and naxplanation (explained in more detail
 ::    below), it gives an %done move to the local vane that had
-::    requested the original "forward flow" message be sent.
+::    requested the original %plea message be sent.
 ::
-::    A "backward flow" message, which is similar to a response or a
+::    A local vane can give Ames zero or more %boon response messages in
-::    subscription update, is given to Ames from a local vane.  Ames
+::    response to a %plea, on the same duct that Ames used to pass the
-::    transmits the message to the peer's Ames, which gives the message
+::    %plea to the vane.  Ames transmits a %boon over the wire to the
-::    to the destination vane.
+::    peer's Ames, which gives it to the destination vane on the same
+::    duct the vane had used to pass the original %plea to Ames.
 ::
-::    Ames will give a %memo to a vane upon hearing the message from a
+::    %boon messages are acked automatically by the receiver Ames.  They
-::    remote. This message is a "backward flow" message, forming one of
+::    cannot be nacked, and Ames only uses the ack internally, without
-::    potentially many responses to a "forward flow" message that a
+::    notifying the client vane that gave Ames the %boon.
-::    local vane had passed to our local Ames, and which local Ames had
-::    relayed to the remote.  Ames gives the %memo on the same duct the
-::    local vane had originally used to pass Ames the "forward flow"
-::    message.
 ::
-::    Backward flow messages are acked automatically by the receiver.
+::    If the Arvo event that completed receipt of a %boon message
-::    They cannot be nacked, and Ames only uses the ack internally,
+::    crashes, Ames instead sends the client vane a %lost message
-::    without notifying the client vane.
+::    indicating the %boon was missed.
 ::
-::    Forward flow messages can be nacked, in which case the peer will
+::    %plea messages can be nacked, in which case the peer will send
-::    send both a message-nack packet and a nack-trace message, which is
+::    both a message-nack packet and a naxplanation message, which is
-::    sent on a special diagnostic flow so as not to interfere with
+::    sent in a way that does not interfere with normal operation.  The
-::    normal operation.  The nack-trace is sent as a full Ames message,
+::    naxplanation is sent as a full Ames message, instead of just a
-::    instead of just a packet, because the contained error information
+::    packet, because the contained error information can be arbitrarily
-::    can be arbitrarily large.
+::    large.  A naxplanation can only give rise to a positive ack --
+::    never ack an ack, and never nack a naxplanation.
 ::
-::    Once the local Ames has received the nack-trace, it knows the peer
+::    Ames guarantees a total ordering of messages within a "flow",
-::    has received the full message and failed to process it.  This
+::    identified in other vanes by a duct and over the wire by a "bone":
-::    means if we later hear an ack packet on the failed message, we can
+::    an opaque number.  Each flow has a FIFO queue of %plea requests
-::    ignore it.
+::    from the requesting ship to the responding ship and a FIFO queue
+::    of %boon's in the other direction.
 ::
-::    Also, due to Ames's exactly-once delivery semantics, we know that
+::    Message order across flows is not specified and may vary based on
-::    when we receive a nack-trace for message n, we know the peer has
+::    network conditions.
-::    positively acked all messages m+1 through n-1, where m is the last
+::
-::    message for which we heard a nack-trace.  If we haven't heard acks
+::    Ames guarantees that a message will only be delivered once to the
-::    on all those messages, we apply positive acks when we hear the
+::    destination vane.
-::    nack-trace.
+::
+::    Ames encrypts every message using symmetric-key encryption by
+::    performing an elliptic curve Diffie-Hellman using our private key
+::    and the public key of the peer.  For ships in the Jael PKI
+::    (public-key infrastructure), Ames looks up the peer's public key
+::    from Jael.  Comets (128-bit ephemeral addresses) are not
+::    cryptographic assets and must self-attest over Ames by sending a
+::    single self-signed packet containing their public key.
+::
+::    When a peer suffers a continuity breach, Ames removes all
+::    messaging state related to it.  Ames does not guarantee that all
+::    messages will be fully delivered to the now-stale peer.  From
+::    Ames's perspective, the newly restarted peer is a new ship.
+::    Ames's guarantees are not maintained across a breach.
+::
+::    A vane can pass Ames a %heed $task to request Ames track a peer's
+::    responsiveness.  If our %boon's to it start backing up locally,
+::    Ames will give a %clog back to the requesting vane containing the
+::    unresponsive peer's urbit address.  This interaction does not use
+::    ducts as unique keys.  Stop tracking a peer by sending Ames a
+::    %jilt $task.
+::
+::    Debug output can be adjusted using %sift and %spew $task's.
 ::
 ::  protocol-version: current version of the ames wire protocol
 ::
@@ -530,7 +564,15 @@
 ::    The first bone is 0. They increment by 4, since each flow includes
 ::    a bit for each message determining forward vs. backward and a
 ::    second bit for whether the message is on the normal flow or the
-::    associated diagnostic flow (for nack-traces).
+::    associated diagnostic flow (for naxplanations).
+::
+::    The least significant bit of a $bone is:
+::    1 if "forward", i.e. we send %plea's on this flow, or
+::    0 if "backward", i.e. we receive %plea's on this flow.
+::
+::    The second-least significant bit is 1 if the bone is a
+::    naxplanation bone, and 0 otherwise.  Only naxplanation
+::    messages can be sent on a naxplanation bone, as %boon's.
 ::
 +$  ossuary
   $:  =next=bone
@@ -542,43 +584,40 @@
 ::    Messages queue up in |message-pump's .unsent-messages until they
 ::    can be packetized and fed into |packet-pump for sending.  When we
 ::    pop a message off .unsent-messages, we push as many fragments as
-::    we can into |packet-pump, then place the remaining in
+::    we can into |packet-pump, which sends every packet it eats.
-::    .unsent-fragments.
+::    Packets rejected by |packet-pump are placed in .unsent-fragments.
 ::
-::    When we hear a packet ack, we send it to |packet-pump.  If we
+::    When we hear a packet ack, we send it to |packet-pump to be
-::    haven't seen it before, |packet-pump reports the fresh ack.
+::    removed from its queue of unacked packets.
 ::
 ::    When we hear a message ack (positive or negative), we treat that
 ::    as though all fragments have been acked.  If this message is not
-::    .current, then it's a future message and .current has not yet been
+::    .current, then this ack is for a future message and .current has
-::    acked, so we place the ack in .queued-message-acks.
+::    not yet been acked, so we place the ack in .queued-message-acks.
 ::
-::    If we hear a message ack before we've sent all the
+::    If we hear a message ack before we've sent all the fragments for
-::    fragments for that message, clear .unsent-fragments. If the
+::    that message, clear .unsent-fragments and have |packet-pump delete
-::    message ack was positive, print it out because it indicates the
+::    all sent fragments from the message. If this early message ack was
-::    peer is not behaving properly.
+::    positive, print it out because it indicates the peer is not
+::    behaving properly.
 ::
-::    If the ack is for the current message, emit the message ack,
+::    If the ack is for the current message, have |packet-pump delete
-::    increment .current, and check if this next message is in
+::    all packets from the message, give the message ack back
-::    .queued-message-acks.  If it is, emit the message (n)ack,
+::    to the client vane, increment .current, and check if this next
-::    increment .current, and check the next message.  Repeat until
+::    message is in .queued-message-acks.  If it is, emit the message
-::    .current is not fully acked.
+::    (n)ack, increment .current, and check the next message.  Repeat
-::
+::    until .current is not fully acked.
-::    When we hear a message nack, we send it to |packet-pump, which
-::    deletes all packets from that message.  If .current gets nacked,
-::    clear .unsent-fragments and go into the same flow as when we hear
-::    the last packet ack on a message.
 ::
 ::    The following equation is always true:
 ::    .next - .current == number of messages in flight
 ::
 ::    At the end of a task, |message-pump sends a %halt task to
 ::    |packet-pump, which can trigger a timer to be set or cleared based
-::    on congestion control calculations. When it fires, the timer will
+::    on congestion control calculations. When the timer fires, it will
-::    generally cause one or more packets to be resent.
+::    generally cause a packet to be re-sent.
 ::
-::    Message sequence numbers start at 1 so the first message will be
+::    Message sequence numbers start at 1 so that the first message will
-::    greater than .last-acked.message-sink-state on the receiver.
+::    be greater than .last-acked.message-sink-state on the receiver.
 ::
 ::    current: sequence number of earliest message sent or being sent
 ::    next: sequence number of next message to send
@@ -618,7 +657,14 @@
 ::    algorithm.  The information signals and their responses are
 ::    identical to those of the "NewReno" variant of Reno; the
 ::    implementation differs because Ames acknowledgments differ from
-::    TCP's and because we're using functional data structures.
+::    TCP's, because this code uses functional data structures, and
+::    because TCP's sequence numbers reset when a peer becomes
+::    unresponsive, whereas Ames sequence numbers only change when a
+::    ship breaches.
+::
+::    A deviation from Reno is +fast-resend-after-ack, which re-sends
+::    timed-out packets when a peer starts responding again after a
+::    period of unresponsiveness.
 ::
 ::    If .skips reaches 3, we perform a fast retransmit and fast
 ::    recovery.  This corresponds to Reno's handling of "three duplicate
@@ -697,8 +743,6 @@
   ==
 ::  $note: request to other vane
 ::
-::    TODO: specialize gall interface for subscription management
-::
 ::    Ames passes a %plea note to another vane when it receives a
 ::    message on a "forward flow" from a peer, originally passed from
 ::    one of the peer's vanes to the peer's Ames.
@@ -729,13 +773,6 @@
   ==  ==  ==
 ::  $sign: response from other vane
 ::
-::    A vane gives a %boon sign to Ames on a duct on which it had
-::    previously received a message on a "forward flow".  Ames will
-::    transmit the message to the peer that had originally sent the
-::    message on the forward flow.  The peer's Ames will then give the
-::    message to the remote vane from which the forward flow message
-::    originated.
-::
 +$  sign
   $~  [%b %wake ~]
   $%  $:  %b
@@ -1512,6 +1549,7 @@
     ::
     ::    Abandon all pretense of continuity and delete all messaging state
     ::    associated with .ship, including sent and unsent messages.
+    ::    Also cancel all timers related to .ship.
     ::
     ++  on-publ-breach
       |=  =ship
@@ -2053,7 +2091,11 @@
     ++  send-shut-packet
       |=  =shut-packet
       ^+  peer-core
-      ::  swizzle bone just before sending; TODO document
+      ::  swizzle last bone bit before sending
+      ::
+      ::    The peer has the opposite perspective from ours about what
+      ::    kind of flow this is (forward/backward), so flip the bit
+      ::    here.
       ::
       =.  bone.shut-packet  (mix 1 bone.shut-packet)
       ::
@@ -2201,7 +2243,7 @@
       ::  +on-sink-boon: handle response message received by |message-sink
       ::
       ::    .bone must be mapped in .ossuary.peer-state, or we crash.
-      ::    This means a malformed message will kill a channel.  We
+      ::    This means a malformed message will kill a flow.  We
       ::    could change this to a no-op if we had some sort of security
       ::    reporting.
       ::
@@ -2654,7 +2696,7 @@
   ::  +on-hear: handle ack on a live packet
   ::
   ::    If the packet was in our queue, delete it and update our
-  ::    metrics.  Otherwise, no-op.
+  ::    metrics, possibly re-sending skipped packets.  Otherwise, no-op.
   ::
   ++  on-hear
     |=  [=message-num =fragment-num]