on master help sign in
PROTOCOL-REVIEW: add #9 federation retry loop starvation

backgroundRetryForwards runs retryPendingForwards inline (not forked) to prevent double-counting recordFailure. With many slow/unreachable peers, a sweep can block for minutes and starve pruneExpired's atomic writes to *messages and *forwardFailures. Design tradeoff โ€” document as accepted until fetch supports a deadline. Also filed Knot compiler hang as ilyakooo0/knot#71 (partially-applied recursive helper in tail position hangs indefinitely).

commit4acbb4064cf86049755546d9c43c91ec2154e6ec authoriko <mail@iko.soy> dateSun 12 Jul 2026, 01:41 (3 months ago) parent254f6a0
1 file changed+32
PROTOCOL-REVIEW.md+32
PROTOCOL-REVIEW.md+32
@@ -181,6 +181,36 @@ protection without the proxy caveat. Recommend documenting it.
181181
182182---
183183
184+## ๐ŸŸก 9. Federation retry loop can starve the prune sweep
185+
186+**Severity: low โ€” design tradeoff, not a code bug.**
187+
188+`backgroundRetryForwards` runs `retryPendingForwards` inline (not via `fork`),
189+so a sweep that hits many slow or timing-out peers blocks until every
190+`fetch` in the batch resolves. Each `fetch` has no explicit timeout โ€” the
191+runtime's HTTP client holds the connection โ€” so a sweep over N unreachable
192+peers can take N ร— (TCP connect timeout) before returning. During that time
193+`backgroundPrune` (on its own `fork`) is free to run, but its writes to
194+`*messages` and `*forwardFailures` conflict with `retryPendingForwards`'s
195+`atomic` blocks, so `pruneExpired`'s atomic retries lose to the retry loop
196+and are effectively starved: expired messages, sessions, and challenges
197+pile up until the retry sweep finishes.
198+
199+In practice this only matters under federation with many dead peers and a
200+low `retryBackoffMax` โ€” the `shouldRetryServer` gate skips peers in backoff,
201+so after one failed sweep most peers are gated out of the next one. The
202+first sweep after a network partition is the worst case.
203+
204+**No fix applied.** The deliberate design (inline, not forked) prevents two
205+concurrent `retryPendingForwards` from seeing the same `*forwards` snapshot
206+and double-counting `recordFailure`. Forking it would reintroduce that race.
207+A real fix would either (a) give `fetch` a deadline and bound the sweep
208+time, or (b) run `retryPendingForwards` in a transaction that yields
209+`*messages`/`*forwardFailures` to `pruneExpired` between per-peer batches.
210+Both need runtime support the current `fetch` does not expose.
211+
212+---
213+
184214## Suggested follow-ups
185215
1862161. **Spec edits (safe, no behavior change):** fold #2, #4, #6, #8 caveats into
@@ -190,3 +220,5 @@ protection without the proxy caveat. Recommend documenting it.
190220 guard) are **done**.
1912213. #2 and #4 are inherent to open, unauthenticated federation โ€” decide whether to
192222 document-as-accepted or add peer authentication.
223+4. **#9** is a design tradeoff; document as accepted until `fetch` supports a
224+ deadline.