@@ -1,10 +1,14 @@
1−:: gossip: data sharing with pals (and pals of pals, etc)
1+:: gossip: data sharing with pals
3−:: v1.1.1: sneaky whisper
5:: automates using /app/pals for peer & content discovery,
3:: automates using /app/pals for peer & content discovery,
6:: letting the underlying agent focus on handling the data.
4:: letting the underlying agent focus on handling the data.
6+:: at the time of writing, this library has not seen much real-world
7+:: use yet. it may be subject to behavioral or interface changes.
8+:: it tries its best to not generate obscene amounts of network load,
9+:: but has not yet proven itself in the real world.
10+:: for the time being, please tread carefully.
10:: to use this library, call its +agent arm with an initial
14:: to use this library, call its +agent arm with an initial
@@ -40,10 +44,6 @@
40:: setting hops to 1 distributes the data to direct pals only.
44:: setting hops to 1 distributes the data to direct pals only.
41:: setting hops to 0 prevents emission of any gossip data at all,
45:: setting hops to 0 prevents emission of any gossip data at all,
42:: even during initial subscription setup.
46:: even during initial subscription setup.
43−:: setting pass to true causes 50% of locally-originating data to be
44−:: proxied through a random "tell" peer. note that the proxy will have
45−:: a 50% chance of proxying in turn. for low amounts of hops, data could
46−:: escape the local social graph entirely.
48:: (we introduce the /~/etc path prefix convention to indicate paths
48:: (we introduce the /~/etc path prefix convention to indicate paths
49:: that are for library-specific use only.
49:: that are for library-specific use only.
@@ -75,12 +75,6 @@
75:: we may want to use pokes exclusively, instead of watches/facts,
75:: we may want to use pokes exclusively, instead of watches/facts,
76:: making it easier to exclude the src.bowl, include metadata, etc.
76:: making it easier to exclude the src.bowl, include metadata, etc.
78−:: we currently only emit every rumor as a fact once at most. but we might
79−:: receive it again later, with a higher hop count. if we want to try for
80−:: maximum reach within the given hops, we should re-send if we receive a
81−:: known rumor with higher hop counter. but this may not be worth the added
84:: what if this was a userspace-infrastructure app instead of a wrapper?
78:: what if this was a userspace-infrastructure app instead of a wrapper?
85:: how would ensuring installation of this app-dependency work?
79:: how would ensuring installation of this app-dependency work?
86:: it gains us... a higher chance of peers having this installed.
80:: it gains us... a higher chance of peers having this installed.
@@ -99,7 +93,6 @@
99:: - if pals ever watch-nacks (it shouldn't), we try rewatching after ~m1.
93:: - if pals ever watch-nacks (it shouldn't), we try rewatching after ~m1.
101:: - for facts produced on /~/gossip/source, we
95:: - for facts produced on /~/gossip/source, we
102−:: - 50% chance to redirect into the "pass flow" (see below)
103:: - wrap them as %gossip-rumor to send them out on /~/gossip/gossip
96:: - wrap them as %gossip-rumor to send them out on /~/gossip/gossip
104:: - for new pals matching the hear mode, we watch their /~/gossip/gossip
97:: - for new pals matching the hear mode, we watch their /~/gossip/gossip
105:: - for gone pals, we leave that watch
98:: - for gone pals, we leave that watch
@@ -109,16 +102,6 @@
109:: - re-emit them as facts on /~/gossip/gossip if there are hops left
102:: - re-emit them as facts on /~/gossip/gossip if there are hops left
110:: - for nacks on those watches, we retry after ~m30
103:: - for nacks on those watches, we retry after ~m30
112−:: - other gossip instances may poke us with rumors directly
113−:: - the inner agent learns the rumor normally, as if it came from a fact
114−:: - based on randomness (50/50), we do one of two things:
115−:: - publish the rumor as a fact to our subscribers, like the above
116−:: - poke a randomly selected peer (in the "tell" set) with the rumor
117−:: - when poking, store the rumor until we receive a poke-ack
118−:: - upon receiving a positive poke-ack, delete the local datum
119−:: - upon receiving a negative poke-ack, do the 50/50 again
120−:: - if we do not receive a poke-ack within some time, do the 50/50 again
123/+ lp=pals, dbug, verb
106/+ lp=pals, dbug, verb
@@ -129,8 +112,6 @@
130+$ meta-0 hops=_0
113+$ meta-0 hops=_0
135 $? %anybody :: any ship discoverable through pals
116 $? %anybody :: any ship discoverable through pals
136 %targets :: any ship we've added as a pal
117 %targets :: any ship we've added as a pal
@@ -142,11 +123,7 @@
142 :: (1 hop is pals only, 0 stops exposing data at all)
123 :: (1 hop is pals only, 0 stops exposing data at all)
143 hear=whos :: who to subscribe to
124 hear=whos :: who to subscribe to
144 tell=whos :: who to allow subscriptions from
125 tell=whos :: who to allow subscriptions from
145− pass=? :: whether to (50/50) emit data through proxy
152 ^- card:agent:gall
129 ^- card:agent:gall
@@ -169,27 +146,23 @@
169 ^- $-(agent:gall agent:gall)
146 ^- $-(agent:gall agent:gall)
172− +$ state-3
149+ +$ state-0
174− manner=config :: latest config
151+ manner=config :: latest config
175− memory=(set hash) :: datums seen (by inner agent)
152+ memory=(set @uv) :: datums seen
176− shared=(set hash) :: datums shared (as fact)
177− passed=(map hash [rumor @da]) :: pending relays & timeouts
178− misses=(set ship) :: ships that won't relay
179− future=(list rumor) :: rumors of unknown kinds
153+ future=(list rumor) :: rumors of unknown kinds
182 +$ card card:agent:gall
156 +$ card card:agent:gall
184 ++ helper
158 ++ helper
185− |_ [=bowl:gall state-3]
159+ |_ [=bowl:gall state-0]
186 +* state +<+
160 +* state +<+
187 pals ~(. lp bowl)
161 pals ~(. lp bowl)
188 ++ en-cage
162 ++ en-cage
189 |= =(cask *)
163 |= =(cask *)
191 ?^ to=(~(get by grab) p.cask)
165 ?^ to=(~(get by grab) p.cask)
192− ::TODO +soft or otherwise virtualize? don't want to risk crashes, right?
193 [p.cask (u.to q.cask)]
166 [p.cask (u.to q.cask)]
194 ~& [gossip+dap.bowl %no-mark p.cask]
167 ~& [gossip+dap.bowl %no-mark p.cask]
195 [%gossip-unknown !>(cask)]
168 [%gossip-unknown !>(cask)]
@@ -204,10 +177,6 @@
204 ~| [%en-rumor initial-hops=hops.manner]
177 ~| [%en-rumor initial-hops=hops.manner]
205 [%0 `meta-0`(dec hops.manner)]
178 [%0 `meta-0`(dec hops.manner)]
211 ++ play-card :: en-rumor relevant facts, handle config changes
180 ++ play-card :: en-rumor relevant facts, handle config changes
213 ^- (quip ^card _state)
182 ^- (quip ^card _state)
@@ -242,68 +211,11 @@
242 ?> =(/~/gossip/source path)
211 ?> =(/~/gossip/source path)
243 :: if hops is configured at 0, we don't broadcast at all.
212 :: if hops is configured at 0, we don't broadcast at all.
214+ =. memory (~(put in memory) (sham (de-cage cage.p.card)))
215+ ?: =(0 hops.manner) [caz state]
245 =/ =rumor (en-rumor cage.p.card)
217 =/ =rumor (en-rumor cage.p.card)
246− =. memory (~(put in memory) (en-hash rumor))
249− =^ cas state (emit-rumor rumor)
250− [(weld cas caz) state]
252− ++ emit-rumor :: gossip a rumor as-is
254− ^- (quip card _state)
255− =/ =hash (en-hash rumor)
257− :- [%give %fact [/~/gossip/gossip]~ %gossip-rumor !>(rumor)]~
218+ card(paths.p [/~/gossip/gossip]~, cage.p [%gossip-rumor !>(rumor)])
259− passed (~(del by passed) hash)
260− shared (~(put in shared) hash)
262− :: if we don't want to proxy, always send as fact
266− :: if we want to proxy, do so 50% of the time
268− ?. =(0 (~(rad og eny.bowl) 2))
270− :: if we're proxying, but there's no reasonable targets, send as fact
273− :: reasonable targets do not include ourselves, whoever
274− :: caused us to want to (re)send this rumor, or ships that failed
275− :: to proxy for us before.
276− :: that last one is important because we need to avoid the mistake of
277− :: occasionally sending them pokes that just linger in our outbound
278− :: ames flows and keep retrying. (ideally we'd do those all pokes for
279− :: a target on the same flow, but we really like storing information
280− :: in the wire and don't want to do per-relay queues in state. we also
281− :: don't particularly care about ordering guarantees for gossip.)
282− :: we don't just get the set of ships with active incoming or outgoing
283− :: subscriptions, because that constrains the set unnecessarily. for
284− :: some configs, ships in .tell (that aren't in .hear) might be
285− :: running the agent without actively listening to us.
287− =- (~(dif in (~(del in (~(del in -) our.bowl)) src.bowl)) misses)
289− %anybody (~(uni in (targets:pals ~.)) leeches:pals)
290− %targets (targets:pals ~.)
291− %mutuals (mutuals:pals ~.)
293− =/ count=@ud ~(wyt in aides)
296− :: poke a randomly chosen proxy with the rumor
298− =/ proxy=ship (snag (~(rad og +(eny.bowl)) count) ~(tap in aides))
299− =/ =time (add now.bowl pass-timeout)
300− =. passed (~(put by passed) hash [rumor time])
302− =/ =wire /~/gossip/passed/(scot %p proxy)/(scot %uv hash)
303− =/ =cage gossip-rumor+!>(rumor)
304− :~ [%pass wire %agent [proxy dap.bowl] %poke cage]
305− [%pass wire %arvo %b %wait time]
308 ++ play-cards
220 ++ play-cards
309 |= cards=(list card)
221 |= cards=(list card)
@@ -330,18 +242,15 @@
330 [%gossip-rumor !>((en-rumor cage.p.i.cards))]
242 [%gossip-rumor !>((en-rumor cage.p.i.cards))]
331 $(out (snoc out i.cards), cards t.cards)
243 $(out (snoc out i.cards), cards t.cards)
333− ++ jump-rumor :: relay a rumor if we haven't yet
245+ ++ resend-rumor
334 |= =rumor
246 |= =rumor
335− ^- (quip card _state)
247+ ^- (unit card)
336− =/ =hash (en-hash rumor)
337− ?: (~(has in shared) hash) [~ state]
338 ?> =(%0 kind.rumor) ::NOTE should have been checked for already
248 ?> =(%0 kind.rumor) ::NOTE should have been checked for already
339− ?~ meta=((soft ,hops=@ud) meta.rumor) [~ state]
249+ ?~ meta=((soft ,hops=@ud) meta.rumor) ~
340 =* hops hops.u.meta
250 =* hops hops.u.meta
341− ?: =(0 hops) [~ state]
251+ ?: =(0 hops) ~
342 =. meta.rumor (dec hops)
252 =. meta.rumor (dec hops)
343− :- [%give %fact [/~/gossip/gossip]~ %gossip-rumor !>(rumor)]~
253+ `[%give %fact [/~/gossip/gossip]~ %gossip-rumor !>(rumor)]
344− state(shared (~(put in shared) (en-hash rumor)))
346 ++ may-watch
255 ++ may-watch
347 |= who=ship
256 |= who=ship
@@ -455,7 +364,7 @@
457 |= inner=agent:gall
366 |= inner=agent:gall
458− =| state-3
367+ =| state-0
459 =* state -
368 =* state -
460 %+ verb |
369 %+ verb |
461 %- agent:dbug
370 %- agent:dbug
@@ -473,70 +382,22 @@
473 =^ cards state (play-cards:up cards)
382 =^ cards state (play-cards:up cards)
474 [(weld watch-pals:up cards) this]
383 [(weld watch-pals:up cards) this]
476− ++ on-save (slop !>([%gossip state]) on-save:og)
385+ ++ on-save !>([[%gossip state] on-save:og])
477 ++ on-load
386 ++ on-load
478 |= ole=vase
387 |= ole=vase
479 ^- (quip card _this)
388 ^- (quip card _this)
480 ?. ?=([[%gossip *] *] q.ole)
389 ?. ?=([[%gossip *] *] q.ole)
481 =. manner init
390 =. manner init
482− =^ cards inner (on-load:og ole)
391+ =^ cards inner (on-load:og ole)
483 =^ cards state (play-cards:up cards)
392 =^ cards state (play-cards:up cards)
484 [(weld watch-pals:up cards) this]
393 [(weld watch-pals:up cards) this]
486− |^ =+ !<([%gossip old=state-any] (slot 2 ole))
395+ =+ !<([[%gossip old=state-0] ile=vase] ole)
487− =/ ile=vase (slot 3 ole)
488− =? old ?=(%0 -.old) (state-0-to-1 old)
489− =? old ?=(%1 -.old) (state-1-to-2 old)
490− :: before version 3, we stored the inner +on-save vase inside
491− :: our own +on-save vase. undoing this double-vasing coincided
492− :: with the h135 type-of-type upgrade.
494− =? ile ?=(%2 -.old) (next-vase:h136 !<(vase:h136 ile))
495− =? old ?=(%2 -.old) (state-2-to-3 old)
497− =. state old
396+ =. state old
498− =^ cards inner (on-load:og ile)
397+ =^ cards inner (on-load:og ile)
499− =^ cards state (play-cards:up cards)
398+ =^ cards state (play-cards:up cards)
500− ::TODO for later versions, add :future retry logic as needed
399+ ::TODO for later versions, add :future retry logic as needed
501− [cards this]
400+ [cards this]
503− +$ state-any $%(state-0 state-1 state-2 state-3)
510− passed=(map hash [rumor @da])
515− |=(s=state-2 s(- %3))
522− passed=(map hash [rumor @da])
527− [%2 manner memory shared passed ~ future]
529− +$ state-0 [%0 manner=config-0 memory=(set hash) future=(list rumor)]
530− +$ config-0 [hops=_1 hear=whos tell=whos]
534− [%1 (config-0-to-1 manner) memory memory ~ future]
538− [hops hear tell pass.init]
541 ++ on-watch
402 ++ on-watch
@@ -553,43 +414,13 @@
553 =^ cards state (play-first-cards:up cards)
414 =^ cards state (play-first-cards:up cards)
554 [cards this]
415 [cards this]
558− ^- (quip card _this)
559− ::TODO gossip config pokes
560− ?. =(%gossip-rumor mark)
561− =^ cards inner (on-poke:og +<)
562− =^ cards state (play-cards:up cards)
564− ?. (want-target:up src.bowl)
565− ~|(%gossip-rejected !!)
566− :: we're getting pokes from them, so we should try relaying through
567− :: them again if we stopped doing so
568− ::NOTE this only ever cleans out .misses for ships producing content
570− =. misses (~(del in misses) src.bowl)
571− ::TODO dedupe with +on-agent %fact
573− =/ =hash (en-hash:up rumor)
574− ?: (~(has in memory) hash)
577− ~& [gossip+dap.bowl %delaying-unknown-rumor-kind kind.rumor]
578− [~ this(future [rumor future])]
579− =. memory (~(put in memory) hash)
580− =/ mage=cage (en-cage:up data.rumor)
581− =^ cards inner (on-agent:og /~/gossip/gossip %fact mage)
582− =^ caz1 state (play-cards:up cards)
583− =^ caz2 state (emit-rumor:up rumor)
584− [(weld caz1 caz2) this]
586 ++ on-agent
417 ++ on-agent
587 |= [=wire =sign:agent:gall]
418 |= [=wire =sign:agent:gall]
588 ^- (quip card _this)
419 ^- (quip card _this)
589 ?. ?=([%~.~ %gossip *] wire)
420 ?. ?=([%~.~ %gossip *] wire)
590 =^ cards inner (on-agent:og wire sign)
421 =^ cards inner (on-agent:og wire sign)
591 =^ cards state (play-cards:up cards)
422 =^ cards state (play-cards:up cards)
592− [cards this]
423+ [cards this]
594 ?+ t.t.wire ~|([%gossip %unexpected-wire wire] !!)
425 ?+ t.t.wire ~|([%gossip %unexpected-wire wire] !!)
595 [%gossip @ ~]
426 [%gossip @ ~]
@@ -602,26 +433,18 @@
602 ?. =(%gossip-rumor mark)
433 ?. =(%gossip-rumor mark)
603 ~& [gossip+dap.bowl %ignoring-unexpected-fact mark=mark]
434 ~& [gossip+dap.bowl %ignoring-unexpected-fact mark=mark]
605− :: we're getting facts from them, so we should try relaying through
606− :: them again if we stopped doing so
607− ::NOTE this only ever cleans out .misses for ships in .hear
609− =. misses (~(del in misses) src.bowl)
610− ::TODO de-dupe with +on-poke
611 =+ !<(=rumor vase)
436 =+ !<(=rumor vase)
612− =/ =hash (en-hash:up rumor)
437+ =/ hash (sham data.rumor)
613 ?: (~(has in memory) hash)
438 ?: (~(has in memory) hash)
614− =^ cards state (jump-rumor:up rumor)
615− [cards this]
439+ [~ this]
616 ?. =(%0 kind.rumor)
440 ?. =(%0 kind.rumor)
617 ~& [gossip+dap.bowl %delaying-unknown-rumor-kind kind.rumor]
441 ~& [gossip+dap.bowl %delaying-unknown-rumor-kind kind.rumor]
618 [~ this(future [rumor future])]
442 [~ this(future [rumor future])]
619− =. memory (~(put in memory) hash)
620 =/ mage=cage (en-cage:up data.rumor)
443 =/ mage=cage (en-cage:up data.rumor)
621 =^ cards inner (on-agent:og /~/gossip/gossip sign(cage mage))
444 =^ cards inner (on-agent:og /~/gossip/gossip sign(cage mage))
622− =^ caz1 state (play-cards:up cards)
445+ =^ cards state (play-cards:up cards)
623− =^ caz2 state (jump-rumor:up rumor)
446+ :_ this(memory (~(put in memory) hash))
624− [(weld caz1 caz2) this]
447+ (weld (drop (resend-rumor:up rumor)) cards)
626 %watch-ack
449 %watch-ack
@@ -648,32 +471,6 @@
648 ~& [gossip+dap.bowl %unexpected-poke-ack wire]
471 ~& [gossip+dap.bowl %unexpected-poke-ack wire]
652− [%passed ?([@ ~] [@ @ ~])]
653− ?. ?=(%poke-ack -.sign)
654− ~& [gossip+dap.bowl %unexpected-sign wire -.sign]
657− :: regardless of anything else: if they acked the poke, they are
658− :: accepting of relay requests, if they nacked they're unaccepting
662− (~(del in misses) src.bowl)
663− (~(put in misses) src.bowl)
665− ?: ?=([@ ~] t.t.t.wire)
666− (slav %uv i.t.t.t.wire)
667− (slav %uv i.t.t.t.t.wire)
668− ?~ rum=(~(get by passed) hash)
670− ::NOTE emitting rest is cute, but doesn't actually work reliably,
671− :: due to userspace duct shenanigans. %wake logic will have to
673− =/ rest=card [%pass wire %arvo %b %rest +.u.rum]
674− ?~ p.sign [[rest]~ this(passed (~(del by passed) hash))]
675− =^ caz state (emit-rumor:up -.u.rum)
678 [%pals @ ~]
475 [%pals @ ~]
679 ?- -.sign
476 ?- -.sign
@@ -745,14 +542,6 @@
745 ++ on-peek
542 ++ on-peek
747 ^- (unit (unit cage))
544 ^- (unit (unit cage))
753− =/ dat (on-peek:og path)
754− ?: ?=(?(~ [~ ~]) dat) ~
755− (fall ((soft (list mass)) q.q.u.u.dat) ~)
756 ?: =(/x/dbug/state path)
545 ?: =(/x/dbug/state path)
757 ``noun+(slop on-save:og !>(gossip=state))
546 ``noun+(slop on-save:og !>(gossip=state))
758 ?. ?=([@ %~.~ %gossip *] path)
547 ?. ?=([@ %~.~ %gossip *] path)
@@ -771,6 +560,14 @@
771 =^ cards state (play-cards:up cards)
560 =^ cards state (play-cards:up cards)
772 [cards this]
561 [cards this]
565+ ^- (quip card _this)
566+ ::TODO gossip config pokes?
567+ =^ cards inner (on-poke:og +<)
568+ =^ cards state (play-cards:up cards)
774 ++ on-arvo
571 ++ on-arvo
775 |= [=wire sign=sign-arvo:agent:gall]
572 |= [=wire sign=sign-arvo:agent:gall]
776 ^- (quip card _this)
573 ^- (quip card _this)
@@ -778,41 +575,19 @@
778 =^ cards inner (on-arvo:og wire sign)
575 =^ cards inner (on-arvo:og wire sign)
779 =^ cards state (play-cards:up cards)
576 =^ cards state (play-cards:up cards)
780 [cards this]
577 [cards this]
578+ ?> ?=([%retry *] t.t.wire)
579+ ?> ?=(%wake +<.sign)
781− ?+ t.t.wire ~|(wire !!)
580+ ?+ t.t.t.wire ~|(wire !!)
782− [%passed ?([@ ~] [@ @ ~])]
581+ [%pals *]
783− =/ =hash
582+ ::NOTE this might result in subscription wire re-use,
784− ?: ?=([@ ~] t.t.t.wire)
583+ :: but if we hit this path we should be loud anyway.
785− (slav %uv i.t.t.t.wire)
584+ [watch-pals:up this]
786− (slav %uv i.t.t.t.t.wire)
787− ?~ rum=(~(get by passed) hash) [~ this]
788− :: since timer cancellation isn't fully reliable, this timer fire
789− :: might be from a relay that nacked our poke. if the timer fired
790− :: earlier than expected, assume that to be the case and no-op.
792− ?: (gth +.u.rum now.bowl) [~ this]
793− :: they failed to respond to our relay request in time, mark them as
794− :: unaccepting (if we know who they are from a the new-style wire)
795− :: and try sending the rumor again
797− =? misses ?=([@ @ ~] t.t.t.wire)
798− (~(put in misses) (slav %p i.t.t.t.wire))
799− =^ cards state (emit-rumor:up -.u.rum)
803− ?> ?=(%wake +<.sign)
804− ?+ t.t.t.wire ~|(wire !!)
806− ::NOTE this might result in subscription wire re-use,
807− :: but if we hit this path we should be loud anyway.
808− [watch-pals:up this]
810− [%watch @ ~]
586+ [%watch @ ~]
812− =/ target=ship (slav %p i.t.t.t.t.wire)
588+ =/ target=ship (slav %p i.t.t.t.t.wire)
813− ?. (want-target:up target) ~
589+ ?. (want-target:up target) ~
814− (watch-target:up target)
590+ (watch-target:up target)
818 ++ on-fail
593 ++ on-fail
@@ -823,4 +598,4 @@
823 [cards this]
598 [cards this]