@@ -1,10 +1,14 @@
1−:: gossip: data sharing with pals (and pals of pals, etc)
2−::
3−:: v1.1.1: sneaky whisper
1+:: gossip: data sharing with pals
42::
53:: automates using /app/pals for peer & content discovery,
64:: letting the underlying agent focus on handling the data.
75::
6+:: at the time of writing, this library has not seen much real-world
7+:: use yet. it may be subject to behavioral or interface changes.
8+:: it tries its best to not generate obscene amounts of network load,
9+:: but has not yet proven itself in the real world.
10+:: for the time being, please tread carefully.
11+::
812:: usage
913::
1014:: to use this library, call its +agent arm with an initial
@@ -40,10 +44,6 @@
4044:: setting hops to 1 distributes the data to direct pals only.
4145:: setting hops to 0 prevents emission of any gossip data at all,
4246:: even during initial subscription setup.
43−:: setting pass to true causes 50% of locally-originating data to be
44−:: proxied through a random "tell" peer. note that the proxy will have
45−:: a 50% chance of proxying in turn. for low amounts of hops, data could
46−:: escape the local social graph entirely.
4747::
4848:: (we introduce the /~/etc path prefix convention to indicate paths
4949:: that are for library-specific use only.
@@ -75,12 +75,6 @@
7575:: we may want to use pokes exclusively, instead of watches/facts,
7676:: making it easier to exclude the src.bowl, include metadata, etc.
7777::
78−:: we currently only emit every rumor as a fact once at most. but we might
79−:: receive it again later, with a higher hop count. if we want to try for
80−:: maximum reach within the given hops, we should re-send if we receive a
81−:: known rumor with higher hop counter. but this may not be worth the added
82−:: complexity...
83−::
8478:: what if this was a userspace-infrastructure app instead of a wrapper?
8579:: how would ensuring installation of this app-dependency work?
8680:: it gains us... a higher chance of peers having this installed.
@@ -99,7 +93,6 @@
9993:: - if pals ever watch-nacks (it shouldn't), we try rewatching after ~m1.
10094::
10195:: - for facts produced on /~/gossip/source, we
102−:: - 50% chance to redirect into the "pass flow" (see below)
10396:: - wrap them as %gossip-rumor to send them out on /~/gossip/gossip
10497:: - for new pals matching the hear mode, we watch their /~/gossip/gossip
10598:: - for gone pals, we leave that watch
@@ -109,16 +102,6 @@
109102:: - re-emit them as facts on /~/gossip/gossip if there are hops left
110103:: - for nacks on those watches, we retry after ~m30
111104::
112−:: - other gossip instances may poke us with rumors directly
113−:: - the inner agent learns the rumor normally, as if it came from a fact
114−:: - based on randomness (50/50), we do one of two things:
115−:: - publish the rumor as a fact to our subscribers, like the above
116−:: - poke a randomly selected peer (in the "tell" set) with the rumor
117−:: - when poking, store the rumor until we receive a poke-ack
118−:: - upon receiving a positive poke-ack, delete the local datum
119−:: - upon receiving a negative poke-ack, do the 50/50 again
120−:: - if we do not receive a poke-ack within some time, do the 50/50 again
121−::
122105/- pals
123106/+ lp=pals, dbug, verb
124107::
@@ -129,8 +112,6 @@
129112 ==
130113+$ meta-0 hops=_0
131114::
132−+$ hash @uv
133−::
134115+$ whos
135116 $? %anybody :: any ship discoverable through pals
136117 %targets :: any ship we've added as a pal
@@ -142,11 +123,7 @@
142123 :: (1 hop is pals only, 0 stops exposing data at all)
143124 hear=whos :: who to subscribe to
144125 tell=whos :: who to allow subscriptions from
145− pass=? :: whether to (50/50) emit data through proxy
146126 ==
147−::
148−++ pass-timeout ~s30
149−::
150127++ invent
151128 |= =cage
152129 ^- card:agent:gall
@@ -169,27 +146,23 @@
169146 ^- $-(agent:gall agent:gall)
170147 |^ agent
171148 ::
172− +$ state-3
173− $: %3
174− manner=config :: latest config
175− memory=(set hash) :: datums seen (by inner agent)
176− shared=(set hash) :: datums shared (as fact)
177− passed=(map hash [rumor @da]) :: pending relays & timeouts
178− misses=(set ship) :: ships that won't relay
179− future=(list rumor) :: rumors of unknown kinds
149+ +$ state-0
150+ $: %0
151+ manner=config :: latest config
152+ memory=(set @uv) :: datums seen
153+ future=(list rumor) :: rumors of unknown kinds
180154 ==
181155 ::
182156 +$ card card:agent:gall
183157 ::
184158 ++ helper
185− |_ [=bowl:gall state-3]
159+ |_ [=bowl:gall state-0]
186160 +* state +<+
187161 pals ~(. lp bowl)
188162 ++ en-cage
189163 |= =(cask *)
190164 ^- cage
191165 ?^ to=(~(get by grab) p.cask)
192− ::TODO +soft or otherwise virtualize? don't want to risk crashes, right?
193166 [p.cask (u.to q.cask)]
194167 ~& [gossip+dap.bowl %no-mark p.cask]
195168 [%gossip-unknown !>(cask)]
@@ -204,10 +177,6 @@
204177 ~| [%en-rumor initial-hops=hops.manner]
205178 [%0 `meta-0`(dec hops.manner)]
206179 ::
207− ++ en-hash
208− |= rumor
209− (sham data)
210− ::
211180 ++ play-card :: en-rumor relevant facts, handle config changes
212181 |= =card
213182 ^- (quip ^card _state)
@@ -242,68 +211,11 @@
242211 ?> =(/~/gossip/source path)
243212 :: if hops is configured at 0, we don't broadcast at all.
244213 ::
214+ =. memory (~(put in memory) (sham (de-cage cage.p.card)))
215+ ?: =(0 hops.manner) [caz state]
216+ =- [[- caz] state]
245217 =/ =rumor (en-rumor cage.p.card)
246− =. memory (~(put in memory) (en-hash rumor))
247− ?: =(0 hops.manner)
248− [caz state]
249− =^ cas state (emit-rumor rumor)
250− [(weld cas caz) state]
251− ::
252− ++ emit-rumor :: gossip a rumor as-is
253− |= =rumor
254− ^- (quip card _state)
255− =/ =hash (en-hash rumor)
256− =* fact
257− :- [%give %fact [/~/gossip/gossip]~ %gossip-rumor !>(rumor)]~
258− %_ state
259− passed (~(del by passed) hash)
260− shared (~(put in shared) hash)
261− ==
262− :: if we don't want to proxy, always send as fact
263− ::
264− ?. pass.manner
265− fact
266− :: if we want to proxy, do so 50% of the time
267− ::
268− ?. =(0 (~(rad og eny.bowl) 2))
269− fact
270− :: if we're proxying, but there's no reasonable targets, send as fact
271− ::
272− =/ aides=(set ship)
273− :: reasonable targets do not include ourselves, whoever
274− :: caused us to want to (re)send this rumor, or ships that failed
275− :: to proxy for us before.
276− :: that last one is important because we need to avoid the mistake of
277− :: occasionally sending them pokes that just linger in our outbound
278− :: ames flows and keep retrying. (ideally we'd do those all pokes for
279− :: a target on the same flow, but we really like storing information
280− :: in the wire and don't want to do per-relay queues in state. we also
281− :: don't particularly care about ordering guarantees for gossip.)
282− :: we don't just get the set of ships with active incoming or outgoing
283− :: subscriptions, because that constrains the set unnecessarily. for
284− :: some configs, ships in .tell (that aren't in .hear) might be
285− :: running the agent without actively listening to us.
286− ::
287− =- (~(dif in (~(del in (~(del in -) our.bowl)) src.bowl)) misses)
288− ?- tell.manner
289− %anybody (~(uni in (targets:pals ~.)) leeches:pals)
290− %targets (targets:pals ~.)
291− %mutuals (mutuals:pals ~.)
292− ==
293− =/ count=@ud ~(wyt in aides)
294− ?: =(0 count)
295− fact
296− :: poke a randomly chosen proxy with the rumor
297− ::
298− =/ proxy=ship (snag (~(rad og +(eny.bowl)) count) ~(tap in aides))
299− =/ =time (add now.bowl pass-timeout)
300− =. passed (~(put by passed) hash [rumor time])
301− :_ state
302− =/ =wire /~/gossip/passed/(scot %p proxy)/(scot %uv hash)
303− =/ =cage gossip-rumor+!>(rumor)
304− :~ [%pass wire %agent [proxy dap.bowl] %poke cage]
305− [%pass wire %arvo %b %wait time]
306− ==
218+ card(paths.p [/~/gossip/gossip]~, cage.p [%gossip-rumor !>(rumor)])
307219 ::
308220 ++ play-cards
309221 |= cards=(list card)
@@ -330,18 +242,15 @@
330242 [%gossip-rumor !>((en-rumor cage.p.i.cards))]
331243 $(out (snoc out i.cards), cards t.cards)
332244 ::
333− ++ jump-rumor :: relay a rumor if we haven't yet
245+ ++ resend-rumor
334246 |= =rumor
335− ^- (quip card _state)
336− =/ =hash (en-hash rumor)
337− ?: (~(has in shared) hash) [~ state]
247+ ^- (unit card)
338248 ?> =(%0 kind.rumor) ::NOTE should have been checked for already
339− ?~ meta=((soft ,hops=@ud) meta.rumor) [~ state]
249+ ?~ meta=((soft ,hops=@ud) meta.rumor) ~
340250 =* hops hops.u.meta
341− ?: =(0 hops) [~ state]
251+ ?: =(0 hops) ~
342252 =. meta.rumor (dec hops)
343− :- [%give %fact [/~/gossip/gossip]~ %gossip-rumor !>(rumor)]~
344− state(shared (~(put in shared) (en-hash rumor)))
253+ `[%give %fact [/~/gossip/gossip]~ %gossip-rumor !>(rumor)]
345254 ::
346255 ++ may-watch
347256 |= who=ship
@@ -455,7 +364,7 @@
455364 ::
456365 ++ agent
457366 |= inner=agent:gall
458− =| state-3
367+ =| state-0
459368 =* state -
460369 %+ verb |
461370 %- agent:dbug
@@ -473,70 +382,22 @@
473382 =^ cards state (play-cards:up cards)
474383 [(weld watch-pals:up cards) this]
475384 ::
476− ++ on-save (slop !>([%gossip state]) on-save:og)
385+ ++ on-save !>([[%gossip state] on-save:og])
477386 ++ on-load
478387 |= ole=vase
479388 ^- (quip card _this)
480389 ?. ?=([[%gossip *] *] q.ole)
481390 =. manner init
482− =^ cards inner (on-load:og ole)
391+ =^ cards inner (on-load:og ole)
483392 =^ cards state (play-cards:up cards)
484393 [(weld watch-pals:up cards) this]
485394 ::
486− |^ =+ !<([%gossip old=state-any] (slot 2 ole))
487− =/ ile=vase (slot 3 ole)
488− =? old ?=(%0 -.old) (state-0-to-1 old)
489− =? old ?=(%1 -.old) (state-1-to-2 old)
490− :: before version 3, we stored the inner +on-save vase inside
491− :: our own +on-save vase. undoing this double-vasing coincided
492− :: with the h135 type-of-type upgrade.
493− ::
494− =? ile ?=(%2 -.old) (next-vase:h136 !<(vase:h136 ile))
495− =? old ?=(%2 -.old) (state-2-to-3 old)
496− ?> ?=(%3 -.old)
497− =. state old
498− =^ cards inner (on-load:og ile)
499− =^ cards state (play-cards:up cards)
500− ::TODO for later versions, add :future retry logic as needed
501− [cards this]
502− ::
503− +$ state-any $%(state-0 state-1 state-2 state-3)
504− ::
505− +$ state-2
506− $: %2
507− manner=config
508− memory=(set hash)
509− shared=(set hash)
510− passed=(map hash [rumor @da])
511− misses=(set ship)
512− future=(list rumor)
513− ==
514− ++ state-2-to-3
515− |=(s=state-2 s(- %3))
516− ::
517− +$ state-1
518− $: %1
519− manner=config
520− memory=(set hash)
521− shared=(set hash)
522− passed=(map hash [rumor @da])
523− future=(list rumor)
524− ==
525− ++ state-1-to-2
526− |= state-1
527− [%2 manner memory shared passed ~ future]
528− ::
529− +$ state-0 [%0 manner=config-0 memory=(set hash) future=(list rumor)]
530− +$ config-0 [hops=_1 hear=whos tell=whos]
531− ++ state-0-to-1
532− |= state-0
533− ^- state-1
534− [%1 (config-0-to-1 manner) memory memory ~ future]
535− ++ config-0-to-1
536− |= config-0
537− ^- config
538− [hops hear tell pass.init]
539− --
395+ =+ !<([[%gossip old=state-0] ile=vase] ole)
396+ =. state old
397+ =^ cards inner (on-load:og ile)
398+ =^ cards state (play-cards:up cards)
399+ ::TODO for later versions, add :future retry logic as needed
400+ [cards this]
540401 ::
541402 ++ on-watch
542403 |= =path
@@ -553,43 +414,13 @@
553414 =^ cards state (play-first-cards:up cards)
554415 [cards this]
555416 ::
556− ++ on-poke
557− |= [=mark =vase]
558− ^- (quip card _this)
559− ::TODO gossip config pokes
560− ?. =(%gossip-rumor mark)
561− =^ cards inner (on-poke:og +<)
562− =^ cards state (play-cards:up cards)
563− [cards this]
564− ?. (want-target:up src.bowl)
565− ~|(%gossip-rejected !!)
566− :: we're getting pokes from them, so we should try relaying through
567− :: them again if we stopped doing so
568− ::NOTE this only ever cleans out .misses for ships producing content
569− ::
570− =. misses (~(del in misses) src.bowl)
571− ::TODO dedupe with +on-agent %fact
572− =+ !<(=rumor vase)
573− =/ =hash (en-hash:up rumor)
574− ?: (~(has in memory) hash)
575− [~ this]
576− ?. =(%0 kind.rumor)
577− ~& [gossip+dap.bowl %delaying-unknown-rumor-kind kind.rumor]
578− [~ this(future [rumor future])]
579− =. memory (~(put in memory) hash)
580− =/ mage=cage (en-cage:up data.rumor)
581− =^ cards inner (on-agent:og /~/gossip/gossip %fact mage)
582− =^ caz1 state (play-cards:up cards)
583− =^ caz2 state (emit-rumor:up rumor)
584− [(weld caz1 caz2) this]
585− ::
586417 ++ on-agent
587418 |= [=wire =sign:agent:gall]
588419 ^- (quip card _this)
589420 ?. ?=([%~.~ %gossip *] wire)
590421 =^ cards inner (on-agent:og wire sign)
591422 =^ cards state (play-cards:up cards)
592− [cards this]
423+ [cards this]
593424 ::
594425 ?+ t.t.wire ~|([%gossip %unexpected-wire wire] !!)
595426 [%gossip @ ~]
@@ -602,26 +433,18 @@
602433 ?. =(%gossip-rumor mark)
603434 ~& [gossip+dap.bowl %ignoring-unexpected-fact mark=mark]
604435 [~ this]
605− :: we're getting facts from them, so we should try relaying through
606− :: them again if we stopped doing so
607− ::NOTE this only ever cleans out .misses for ships in .hear
608− ::
609− =. misses (~(del in misses) src.bowl)
610− ::TODO de-dupe with +on-poke
611436 =+ !<(=rumor vase)
612− =/ =hash (en-hash:up rumor)
437+ =/ hash (sham data.rumor)
613438 ?: (~(has in memory) hash)
614− =^ cards state (jump-rumor:up rumor)
615− [cards this]
439+ [~ this]
616440 ?. =(%0 kind.rumor)
617441 ~& [gossip+dap.bowl %delaying-unknown-rumor-kind kind.rumor]
618442 [~ this(future [rumor future])]
619− =. memory (~(put in memory) hash)
620443 =/ mage=cage (en-cage:up data.rumor)
621444 =^ cards inner (on-agent:og /~/gossip/gossip sign(cage mage))
622− =^ caz1 state (play-cards:up cards)
623− =^ caz2 state (jump-rumor:up rumor)
624− [(weld caz1 caz2) this]
445+ =^ cards state (play-cards:up cards)
446+ :_ this(memory (~(put in memory) hash))
447+ (weld (drop (resend-rumor:up rumor)) cards)
625448 ::
626449 %watch-ack
627450 :_ this
@@ -648,32 +471,6 @@
648471 ~& [gossip+dap.bowl %unexpected-poke-ack wire]
649472 [~ this]
650473 ==
651− ::
652− [%passed ?([@ ~] [@ @ ~])]
653− ?. ?=(%poke-ack -.sign)
654− ~& [gossip+dap.bowl %unexpected-sign wire -.sign]
655− [~ this]
656− ~| t.t.wire
657− :: regardless of anything else: if they acked the poke, they are
658− :: accepting of relay requests, if they nacked they're unaccepting
659− ::
660− =. misses
661− ?: ?=(~ p.sign)
662− (~(del in misses) src.bowl)
663− (~(put in misses) src.bowl)
664− =/ =hash
665− ?: ?=([@ ~] t.t.t.wire)
666− (slav %uv i.t.t.t.wire)
667− (slav %uv i.t.t.t.t.wire)
668− ?~ rum=(~(get by passed) hash)
669− [~ this]
670− ::NOTE emitting rest is cute, but doesn't actually work reliably,
671− :: due to userspace duct shenanigans. %wake logic will have to
672− :: be defensive...
673− =/ rest=card [%pass wire %arvo %b %rest +.u.rum]
674− ?~ p.sign [[rest]~ this(passed (~(del by passed) hash))]
675− =^ caz state (emit-rumor:up -.u.rum)
676− [[rest caz] this]
677474 ::
678475 [%pals @ ~]
679476 ?- -.sign
@@ -745,14 +542,6 @@
745542 ++ on-peek
746543 |= =path
747544 ^- (unit (unit cage))
748− ?: =(/x/whey path)
749− :+ ~ ~
750− :- %mass
751− !> ^- (list mass)
752− :- %gossip^&+state
753− =/ dat (on-peek:og path)
754− ?: ?=(?(~ [~ ~]) dat) ~
755− (fall ((soft (list mass)) q.q.u.u.dat) ~)
756545 ?: =(/x/dbug/state path)
757546 ``noun+(slop on-save:og !>(gossip=state))
758547 ?. ?=([@ %~.~ %gossip *] path)
@@ -771,6 +560,14 @@
771560 =^ cards state (play-cards:up cards)
772561 [cards this]
773562 ::
563+ ++ on-poke
564+ |= cage
565+ ^- (quip card _this)
566+ ::TODO gossip config pokes?
567+ =^ cards inner (on-poke:og +<)
568+ =^ cards state (play-cards:up cards)
569+ [cards this]
570+ ::
774571 ++ on-arvo
775572 |= [=wire sign=sign-arvo:agent:gall]
776573 ^- (quip card _this)
@@ -778,41 +575,19 @@
778575 =^ cards inner (on-arvo:og wire sign)
779576 =^ cards state (play-cards:up cards)
780577 [cards this]
781− ?+ t.t.wire ~|(wire !!)
782− [%passed ?([@ ~] [@ @ ~])]
783− =/ =hash
784− ?: ?=([@ ~] t.t.t.wire)
785− (slav %uv i.t.t.t.wire)
786− (slav %uv i.t.t.t.t.wire)
787− ?~ rum=(~(get by passed) hash) [~ this]
788− :: since timer cancellation isn't fully reliable, this timer fire
789− :: might be from a relay that nacked our poke. if the timer fired
790− :: earlier than expected, assume that to be the case and no-op.
791− ::
792− ?: (gth +.u.rum now.bowl) [~ this]
793− :: they failed to respond to our relay request in time, mark them as
794− :: unaccepting (if we know who they are from a the new-style wire)
795− :: and try sending the rumor again
796− ::
797− =? misses ?=([@ @ ~] t.t.t.wire)
798− (~(put in misses) (slav %p i.t.t.t.wire))
799− =^ cards state (emit-rumor:up -.u.rum)
800− [cards this]
578+ ?> ?=([%retry *] t.t.wire)
579+ ?> ?=(%wake +<.sign)
580+ ?+ t.t.t.wire ~|(wire !!)
581+ [%pals *]
582+ ::NOTE this might result in subscription wire re-use,
583+ :: but if we hit this path we should be loud anyway.
584+ [watch-pals:up this]
801585 ::
802− [%retry *]
803− ?> ?=(%wake +<.sign)
804− ?+ t.t.t.wire ~|(wire !!)
805− [%pals *]
806− ::NOTE this might result in subscription wire re-use,
807− :: but if we hit this path we should be loud anyway.
808− [watch-pals:up this]
809− ::
810− [%watch @ ~]
811− :_ this
812− =/ target=ship (slav %p i.t.t.t.t.wire)
813− ?. (want-target:up target) ~
814− (watch-target:up target)
815− ==
586+ [%watch @ ~]
587+ :_ this
588+ =/ target=ship (slav %p i.t.t.t.t.wire)
589+ ?. (want-target:up target) ~
590+ (watch-target:up target)
816591 ==
817592 ::
818593 ++ on-fail
@@ -823,4 +598,4 @@
823598 [cards this]
824599 --
825600 --
826−--
601+--