@@ -0,0 +1,911 @@
2+title = "8. Subscriptions"
7+In this lesson we're going to look at subscriptions. Subscriptions are probably
8+the most complicated part of writing agents, so there's a fair bit to cover.
9+Before we get into the nitty-gritty details, we'll give a brief overview of
10+Gall's subscription mechanics.
12+The basic unit of subscriptions is the _path_. An agent will typically define a
13+number of subscription paths in its `on-watch` arm, and other agents (local or
14+remote) can subscribe to those paths. The agent will then send out updates
15+called `%fact`s on one or more of its paths, and _all_ subscribers of those
16+paths will receive them. An agent cannot send out updates to specific
17+subscribers, it can only target its paths. An agent can kick subscribers from
18+its paths, and subscribers can unsubscribe from any paths.
20+The subscription paths an agent defines can be simple and fixed like
21+`/foo/bar/baz`. They can also be dynamic, containing data of a particular atom
22+aura encoded in certain elements of the path. These paths can therefore be as
23+simple or complex as you need for your particular application.
25+Note it's not strictly necessary to define subscription paths explicitly. As
26+long as the arm doesn't crash, the subscription will succeed. In practice,
27+however, it's nearly always appropriate to define them explicitly and crash on
30+For a deeper explanation of subscription mechanics in Arvo, you can refer to
31+Arvo's [Subscriptions](/docs/arvo/concepts/subscriptions) section.
33+## Incoming subscriptions
35+Subscription requests from other entities arrive in your agent's `on-watch` arm.
36+The `on-watch` arm takes the `path` to which they're subscribing, and produces a
46+Your agent's subscription paths would be defined in this arm, typically in a
47+wutlus (`?+`) expression or similar:
50+?+ path (on-watch:def path)
58+ =/ when=@da (slav %da i.t.path)
62+ ?+ t.t.path (on-watch:def path)
70+Subscription paths can be simple and fixed like the first two examples above:
71+`/updates` and `/blah/blah`. They can also contain "wildcard" elements, with an
72+atom of a particular aura encoded in an element of the `path`, as in the `[%foo @ ~]` example. The type pattern matcher is quite limited, so we just specify
73+such variable elements as `@`, and then decode them with something like `(slav %da i.t.path)` (for a `@da`), as in the example. The incoming `path` in this
74+example would look like `/foo/~2021.11.14..13.30.39..6b17`. For more information
75+on decoding atoms in strings, see the [Strings
76+Guide](/docs/hoon/guides/strings#decoding-from-text).
78+In the last case of `[%bar baz *]`, we're allowing a variable number of elements
79+in the path. First we check it's `/foo/bar/...something...`, and then we check
80+what the "something" is in another wutlus expression and handle it
81+appropriately. In this case, it could be `/foo/bar`, `/foo/bar/abc/def`, or
82+`/foo/bar/blah`. You could of course also have "wildcard" elements here too, so
83+there's not really a limit to the complexity of your subscription paths, or the
84+data that might be encoded therein.
86+Permissions can be checked as described in the previous lesson, comparing the
87+source `@p` of the request in `src.bowl` to `our.bowl` or any other logic you
90+If a permission check fails, the path is not valid, or any other reason you want
91+to reject the subscription request, your agent can simply crash. The behavior
92+here is the same as with `on-poke` - Gall will send a `%watch-ack` card in
93+response, which is either an ack (positive acknowledgement) or a nack (negative
94+acknowledgement). The `(unit tang)` in the `%watch-ack` will be null if
95+processing succeeded, and non-null if it crashed, with a stack trace in the
96+`tang`. Like with `poke-ack`s, you don't need to explicitly send a
97+`%watch-ack` - Gall will do it automatically.
99+As well as sending a `%watch-ack`, Gall will also record the subscription in the
100+`sup` field of the `bowl`, if it succeeded. Then, when you send updates out to
101+subscribers of the `path` in question, the new subscriber will begin receiving
104+Updates to subscribers would usually be sent from other arms, but there's one
105+special case for `on-watch` which is very useful. Normally updates can only be
106+sent to all subscribers of a particular path - you can't target a specific
107+subscriber. There's one exception to this: In `on-watch`, when there's a new
108+subscription, you can send a `%fact` back with an empty `(list path)`, and it'll
109+only go to the new subscriber. This is most useful when you want to give the
110+subscriber some initial state, which you otherwise couldn't do without sending
111+it to everyone. It might look something like this:
115+:~ [%give %fact ~ %todo-update !>(`update:todo`initial+tasks)]
119+## Sending updates to subscribers
121+Once your agent has subscribers, it's easy to send them out updates. All you
122+need to do is produce `card`s with `%fact`s in them:
126+:~ [%give %fact ~[/some/path /another/path] %some-mark !>('some data')]
127+ [%give %fact ~[/some/path] %some-mark !>('more data')]
132+The `(list path)` in the `%fact` specifies which subscription `path`s the
133+`%fact` should be sent on. All subscribers of all `path`s specified will receive
134+the `%fact`. Any agent arm which produces a `(quip card _this)` can send
135+`%fact`s to subscribers. Most often they will be produced in the `on-poke` arm,
136+since new data will often be added in `poke`s.
138+## Kicking subscribers
140+To kick a subscriber, you just send a `%kick` `card`:
143+[%give %kick ~[/some/path] `~sampel-palnet]
146+The `(list path)` specifies which subscription `path`s the ship should be kicked
147+from, and the `(unit ship)` specifies which ship to kick. The `(unit ship)` can also be null, like so:
150+[%give %kick ~[/some/path] ~]
153+In this case, all subscribers to the specified `path`s will be kicked.
155+Note that `%kick`s are not exclusively sent by the agent itself - Gall itself
156+can also kick subscribers under certain network conditions. Because of this,
157+`%kick`s are not assumed to be intentional, and the usual behavior is for a
158+kicked agent to try and resubscribe. Therefore, if you want to disallow a
159+particular subscriber, your agent's `on-watch` arm should reject further
160+subscription requests from them - your agent should not just `%kick` them and
163+## Outgoing subscriptions
165+Now that we've covered incoming subscriptions, we'll look at the other side of
166+it: Subscribing to other agents. This is done by `%pass`ing the target agent a
167+`%watch` task in a `card`:
170+[%pass /some/wire %agent [~some-ship %some-agent] %watch /some/path]
173+If your agent's subscription request is successful, updates will come in to your
174+agent's `on-agent` arm on the `wire` specified (`/some/wire` in this example).
175+The `wire` can be anything you like - its purpose is for your agent to figure
176+out which subscription the updates came from. The `[ship term]` pair specifies the
177+ship and agent you're trying to subscribe to, and the final `path` (`/some/path`
178+in this example) is the path you want to subscribe to - a `path` the target
179+agent has defined in its `on-watch` arm.
181+Gall will deliver the `card` to the target agent and call that agent's
182+`on-watch` arm, which will process the request [as described
183+above](#incoming-subscription-requests), accept or reject it, and send back
184+either a positive or negative `%watch-ack`. The `%watch-ack` will come back in
185+to your agent's `on-agent` arm in a `sign`, along with the `wire` you specified
186+(`/some/wire` in this example). Recall in the lesson on pokes, the `on-agent`
191+ |= [=wire =sign:agent:gall]
192+ ^- (quip card _this)
196+The `sign` will be of the following format:
199+[%watch-ack p=(unit tang)]
202+How you want to handle the `%watch-ack` really depends on the particular agent.
203+In the simplest case, you can just pass it to the `on-agent` arm of
204+`default-agent`, which will just accept it and do nothing apart from printing
205+the error in the `%watch-ack` `tang` if it's a nack. You shouldn't have your
206+agent crash on a `%watch-ack` - even if it's a nack your agent should process it
207+successfully. If you wanted to apply some additional logic on receipt of the
208+`%watch-ack`, you'd typically first test the `wire`, then test whether it's a
209+`%watch-ack`, then test whether it's an ack or a nack and do whatever's
214+ |= [=wire =sign:agent:gall]
215+ ^- (quip card _this)
216+ ?+ wire (on-watch:def wire sign)
218+ ?+ -.sign (on-watch:def wire sign)
221+ ...(do something if ack)...
222+ ...(do something if nack)...
226+The `on-agent` arm produces a `(quip card _this)`, so you can produce new
227+`card`s and update your agent's state, as appropriate.
229+One further thing to note with subscriptions is that you can subscribe multiple
230+times to the same `path` on the same ship and agent, as long as the `wire` is
231+unique. If the ship, agent, `path` and `wire` are all the same as an existing
232+subscription, Gall will not allow the request to be sent, and instead fail with
233+an error message fed into the `on-fail` arm of your agent.
237+Assuming the `%watch` succeeded, your agent will now begin receiving any
238+`%fact`s the other agent publishes on the `path` to which you've subscribed. These
239+`%fact`s will also come in to your agent's `on-agent` arm in a `sign`, just like
240+the initial `%watch-ack`. The `%fact` `sign` will have the following format:
246+You would typically handle such `%fact`s in the following manner: Test the
247+`wire`, test whether the `sign` is a `%fact`, test the `mark` in the `cage`,
248+extract the data from the `vase` in the `cage`, and apply your logic. Again, routing on `wire` before `sign` is one of the [Precepts](/docs/development/precepts#specifics). For example:
252+ |= [=wire =sign:agent:gall]
253+ ^- (quip card _this)
254+ ?+ wire (on-watch:def wire sign)
256+ ?+ -.sign (on-watch:def wire sign)
258+ ?+ p.cage.sign (on-watch:def wire sign)
260+ =/ foo !<(expected-type q.cage.sign)
265+Note that Gall will not allow `sign`s to come into `on-agent` unsolicited, so
266+you don't necessarily need to include permission logic in this arm.
268+The `on-agent` arm produces a `(quip card _this)`, so you can produce new
269+`card`s and update your agent's state, as appropriate.
273+For whatever reason, the agent you're `%watch`ing might want to kick your agent
274+from a `path` to which it's suscribed, ending your subscription and ceasing to
275+send your agent `%fact`s. To do this, it will send your agent a `%kick` card [as
276+described above](#kicking-subscribers). The `%kick` will come in to your agent's
277+`on-agent` arm in a `sign`, like `%watch-ack`s and `%fact`s do. The `%kick`
278+`sign` will have the following format:
284+Since the `%kick` itself contains no information, you'll need to consider the
285+`wire` it comes in on to know what it pertains to. As explained previously,
286+`%kick`s aren't always intentional - sometimes Gall will kick subscribers due to
287+network issues. Your `on-agent` arm therefore has no way to know whether the
288+other agent actually intended to kick it. This means _your agent should almost
289+always try to resubscribe if it gets kicked_. Then, if the resubscribe `%watch`
290+request is rejected with a negative `%watch-ack`, you can conclude that it was
291+intentional and give up. The logic would look something like this:
295+ |= [=wire =sign:agent:gall]
296+ ^- (quip card _this)
297+ ?+ wire (on-agent:def wire sign)
299+ ?+ -.sign (on-agent:def wire sign)
302+ :~ [%pass /some/wire %agent [src.bowl dap.bowl] %watch /some/path]
307+## Leaving a subscription
309+Eventually you may wish to unsubscribe from a `path` in another agent and stop
310+receiving updates. This is done by `%pass`ing a `%leave` task to the agent in
314+[%pass /some/wire %agent [~some-ship %some-agent] %leave ~]
317+The subcription to be ended is determined by the combination of the `wire`, ship
318+and agent, so the `%leave` task itself always just has `~` at the end.
322+Here we're going to give a pretty well fleshed out example. It will demonstrate
323+both inbound and outbound subscriptions, most of the concepts we've discussed
324+here, as well as some from the previous lesson - `/sur` files, `mark` files, and
327+In previous lessons we've only dealt with things on a local ship - this example
328+will demonstrate messages being sent over the network.
330+The example will be composed of two separate agents - a publisher called
331+`/app/todo.hoon` and a subscriber called `/app/todo-watcher.hoon`, which will
332+live on separate ships. It will be a very rudimentary To-Do app - to-do tasks
333+will be poked into the publisher and sent out to the subscriber as `%fact`s,
334+which will just print them to the dojo. It will have its types defined in
335+`/sur/todo.hoon`, and it will have a couple of `mark` files for pokes and
336+updates: `/mar/todo/action.hoon` and `/mar/todo/update.hoon`.
338+Before we get into trying it out, we'll first walk through the `/sur` file, mark
339+files, and each agent.
343+#### `/sur/todo.hoon`
349++$ task [=name done=?]
350++$ tasks (map id task)
371+This file defines most of the types for the agents. The list of to-do tasks will
372+be stored in the state of the publisher agent as the `tasks` type, a `(map id task)`, where a `task` is a `[=name done=?]`. The set of ships allowed to
373+subscribe will be stored in `friends`, a `(set @p)`, also in the publisher's
374+state. After that, there are the head-tagged unions of accepted poke `action`s
375+and `update`s for subscribers.
377+#### `/mar/todo/action.hoon`
394+This is a very simple mark file for the `action` type.
396+#### `/mar/todo/update.hoon`
413+This is a very simple mark file for the `update` type.
417+#### `/app/todo.hoon`
421+/+ default-agent, dbug
426++$ state-0 [%0 =friends:todo =tasks:todo]
427++$ card card:agent:gall
435+ def ~(. (default-agent this %.n) bowl)
438+ ^- (quip card _this)
447+ ^- (quip card _this)
448+ =/ old !<(versioned-state old-state)
455+ ^- (quip card _this)
457+ ?> =(src.bowl our.bowl)
458+ ?+ mark (on-poke:def mark vase)
461+ (handle-poke !<(action:todo vase))
466+ ^- (quip card _state)
469+ ?: (~(has by tasks) now.bowl)
470+ $(now.bowl (add now.bowl ~s0..0001))
471+ :_ state(tasks (~(put by tasks) now.bowl [name.action %.n]))
472+ :~ :* %give %fact ~[/updates] %todo-update
473+ !>(`update:todo`[%add now.bowl name.action])
478+ :_ state(tasks (~(del by tasks) id.action))
479+ :~ :* %give %fact ~[/updates] %todo-update
480+ !>(`update:todo`action)
485+ :- :~ :* %give %fact ~[/updates] %todo-update
486+ !>(`update:todo`action)
489+ tasks %+ ~(jab by tasks)
491+ |=(=task:todo task(done !done.task))
495+ :- :~ :* %give %fact ~[/updates] %todo-update
496+ !>(`update:todo`action)
499+ tasks %+ ~(jab by tasks)
501+ |=(=task:todo task(name name.action))
504+ `state(friends (~(put in friends) who.action))
507+ :_ state(friends (~(del in friends) who.action))
508+ :~ [%give %kick ~[/updates] `who.action]
515+ ^- (quip card _this)
516+ ?+ path (on-watch:def path)
518+ ?> (~(has in friends) src.bowl)
520+ :~ [%give %fact ~ %todo-update !>(`update:todo`initial+tasks)]
524+++ on-leave on-leave:def
525+++ on-peek on-peek:def
526+++ on-agent on-agent:def
527+++ on-arvo on-arvo:def
528+++ on-fail on-fail:def
532+This is the publisher agent, `todo.hoon`. The bulk of its logic is in its
533+`on-poke` arm, where it handles the various possible actions like `%add`ing a
534+task, `%toggle`ing its "done" state, `%rename`ing a task, and so on. It also has
535+a couple of `action`s for `%allow`ing and `%kick`ing subscribers.
537+Most of these cases both update the state of the agent, as well as producing
538+`%fact` cards to send out to subscribers with the new data.
540+You'll notice it only allows these pokes from the local ship, and enforces this
544+?> =(src.bowl our.bowl)
547+Additionally, you might notice the `%add` case in `handle-poke` begins with the
551+?: (~(has by tasks) now.bowl)
552+ $(now.bowl (add now.bowl ~s0..0001))
555+Back in lesson two, we mentioned that the bowl is only repopulated when there's
556+a new Arvo event, so simultaneous messages from a local agent or web client
557+would be processed with the same bowl. Since we're using `now.bowl` for the task
558+ID, this means multiple `%add` actions could collide. To handle this case, we
559+check if there's already an entry in the `tasks` map with the current date-time,
560+and if there is, we increase the time by a fraction of a second and try again.
562+Let's now look at `on-watch`:
567+ ^- (quip card _this)
568+ ?+ path (on-watch:def path)
570+ ?> (~(has in friends) src.bowl)
572+ :~ [%give %fact ~ %todo-update !>(`update:todo`initial+tasks)]
577+When `on-watch` gets a subscription request, it checks whether the requesting
578+ship is in the `friends` set, and crashes if it is not. If they're in `friends`,
579+it produces a `%fact` card with a null `(list path)`, which means it goes only
580+to the new subscriber. This `%fact` contains the entire `tasks` map as it
581+currently exists, getting the new subscriber up to date.
585+#### `/app/todo-watcher.hoon`
589+/+ default-agent, dbug
595++$ card card:agent:gall
603+ def ~(. (default-agent this %.n) bowl)
606+ ^- (quip card _this)
615+ ^- (quip card _this)
616+ =/ old !<(versioned-state old-state)
623+ ^- (quip card _this)
624+ ?> =(src.bowl our.bowl)
625+ ?+ mark (on-poke:def mark vase)
627+ =/ action !<(?([%sub @p] [%unsub @p]) vase)
631+ :~ [%pass /todos %agent [+.action %todo] %watch /updates]
635+ :~ [%pass /todos %agent [+.action %todo] %leave ~]
640+++ on-watch on-watch:def
641+++ on-leave on-leave:def
642+++ on-peek on-peek:def
645+ |= [=wire =sign:agent:gall]
646+ ^- (quip card _this)
647+ ?+ wire (on-agent:def wire sign)
649+ ?+ -.sign (on-agent:def wire sign)
652+ ((slog '%todo-watcher: Subscribe succeeded!' ~) `this)
653+ ((slog '%todo-watcher: Subscribe failed!' ~) `this)
656+ %- (slog '%todo-watcher: Got kick, resubscribing...' ~)
658+ :~ [%pass /todos %agent [src.bowl %todo] %watch /updates]
662+ ?+ p.cage.sign (on-agent:def wire sign)
664+ ~& !<(update:todo q.cage.sign)
670+++ on-arvo on-arvo:def
671+++ on-fail on-fail:def
675+This is the subscriber agent. Since it's just for demonstrative purposes, it has
676+no state and just prints the updates it receives. In practice it would keep the
677+`tasks` map it receives in its own state, and then update it as it receives new
680+The `on-poke` arm is fairly simple - it accepts two pokes, to either `[%sub ~some-ship]` or `[%unsub ~some-ship]`.
682+The `on-agent` arm will print whether a subscription request succeeded or
683+failed, as well as printing a message when it gets kicked. When it receives a
684+`%fact` from the publisher agent, it will just print it to the terminal with a
689+We're going to try this between two different ships. The first ship will be the
690+usual fakezod. We'll add both `mark` files, the `/sur` file, and the `todo.hoon`
691+agent to the `%base` desk of our fakezod, putting them in the following
706+In `~zod`'s dojo, we can `|commit %base`, and then start the `%todo` agent:
709+|rein %base [& %todo]
712+Now we need to spin up another fake ship. We'll use `~nut` in this example:
718+Once it's booted, we can `|mount %base` and then add just the `update.hoon` mark
719+file, the `/sur` file, and the `todo-watcher.hoon` agent like so:
724+│ └── todo-watcher.hoon
732+On `~nut` we can then `|commit %base`, and start the `%todo-watcher` agent:
735+|rein %base [& %todo-watcher]
738+Now, on `~nut`, let's try subscribing:
741+> :todo-watcher [%sub ~zod]
743+%todo-watcher: Subscribe failed!
746+Our `%todo-watcher` agent tried, but received a negative `%watch-ack` from
747+`%todo`, because we haven't yet added `~nut` to the `friends` set of allowed
748+ships. Let's now remedy that on `~zod`:
751+> :todo &todo-action [%allow ~nut]
755+Let's also add a couple of to-do tasks, on `~zod`:
758+> :todo &todo-action [%add 'foo']
760+> :todo &todo-action [%add 'bar']
764+If we now check its state with `+dbug`, we'll see they're in the `tasks` map,
765+and `~nut` will also now be in the `friends` set:
771+ { [ p=170.141.184.505.349.079.206.522.766.950.035.095.552
772+ q=[name='foo' done=%.n]
774+ [ p=170.141.184.505.349.079.278.538.984.166.386.565.120
775+ q=[name='bar' done=%.n]
783+Let's now try subscribing again on `~nut`:
786+> :todo-watcher [%sub ~zod]
788+%todo-watcher: Subscribe succeeded!
791+ { [ p=170.141.184.505.349.079.206.522.766.950.035.095.552
792+ q=[name='foo' done=%.n]
794+ [ p=170.141.184.505.349.079.278.538.984.166.386.565.120
795+ q=[name='bar' done=%.n]
801+As you can see, this time it's worked, and we've immediately received the
804+Now, let's try adding another task on `~zod`:
807+> :todo &todo-action [%add 'baz']
811+On `~nut`, we'll see it has received the `%fact` with the new task in it:
815+ id=170.141.184.505.349.082.779.030.192.959.445.270.528
820+Let's try toggle its done state on `~zod`:
823+> :todo &todo-action [%toggle 170.141.184.505.349.082.779.030.192.959.445.270.528]
827+`~nut` will again get the `%fact`:
831+ id=170.141.184.505.349.082.779.030.192.959.445.270.528
835+Recall that incoming subscriptions are stored in `sup.bowl`, and outgoing
836+subscriptions are stored in `wex.bowl`. Let's have a look at the incoming
837+subscription on `~zod`:
842+ duct=~[/gall/sys/req/~nut/todo /ames/bone/~nut/1 //ames]
844+> :todo +dbug [%incoming %ship ~nut]
848+On `~nut`, let's look at the outgoing subscription:
851+> [wire=/todos agnt=[~zod %todo] path=/updates ackd=%.y]
852+> :todo-watcher +dbug [%outgoing %ship ~zod]
856+Now on `~zod`, let's try kicking `~nut` and removing it from our `friends` set:
859+> :todo &todo-action [%kick ~nut]
863+On `~nut`, we'll see it got the `%kick`, tried resubscribing automatically, but
864+was rejected because `~nut` is no longer in `friends`:
867+%todo-watcher: Got kick, resubscribing...
868+%todo-watcher: Subscribe failed!
873+- Incoming subscription requests arrive in an agent's `on-watch` arm.
874+- An agent will define various subscription `path`s in its `on-watch` arm, which
875+ others can subscribe to.
876+- Gall will automatically produce a negative `%watch-ack` if `on-watch` crashed,
877+ and a positive one if it was successful.
878+- Incoming subscribers are recorded in the `sup` field of the `bowl`.
879+- `on-watch` can produce a `%fact` with a null `(list path)` which will go only
880+ to the new subscriber.
881+- Updates are sent to subscribers in `%fact` cards, and contain a `cage` with a
882+ `mark` and some data in a `vase`.
883+- `%fact`s are sent to all subscribers of the paths specified in the `(list path)`.
884+- A subscriber can be kicked from subscription paths with a `%kick` card
885+ specifying the ship in the `(unit ship)`. All subscribers of the specified
886+ paths will be kicked if the `(unit ship)` is null.
887+- An outgoing subscription can be initiated with a `%watch` card.
888+- The `%watch-ack` will come back in to the subscriber's `on-agent` arm as a
889+ `sign`, and may be positive or negative, depending on whether the `(unit tang)` is null.
890+- `%kick`s will also arrive in the subscriber's `on-agent` arm as a `sign`.
891+ Since kicks may not be intentional, the subscriber should attempt to
892+ resubscribe and only give up if the subsequent `%watch-ack` is negative.
893+- `%fact`s will also arrive in the subscriber's `on-agent` arm.
894+- All such `sign`s that arrive in `on-agent` will also have a `wire`.
895+- The `wire` for subscription updates to arrive on is specified in the initial
897+- A subscriber can unsubscribe by passing a `%leave` card on the original
902+- Have a look at the [Strings Guide](/docs/hoon/guides/strings) if you're not
903+ already familiar with decoding/encoding atoms in strings.
904+- Try running through the [example](#example) yourself, if you've not done so
906+- Try modifying `%todo-watcher` to recording the data it receives in its state,
907+ rather than simply printing it to the terminal.
908+- If you'd like, try going back to [lesson
909+ 6](/docs/userspace/gall-guide/6-pokes) (on pokes) and modifying the agents
910+ with an appropriate permission system, and also try running them on separate