@@ -1,47 +1,56 @@
2−# Talk architecture
2+# Hall architecture
4−This document is complemented by talk's source code, but doesn't require it. Definitions of data structures, code snippets and diagrams will be provided where useful.
4+This document is complemented by Hall's source code, but doesn't require it. Definitions of data structures, code snippets and diagrams will be provided where useful.
6+Hall's implementation is structured according to the new Gall model. Familiarity with its concepts is assumed. See [the new Gall model](gall.md).
6## Table of contents
8## Table of contents
8−* [Overview](#overview)
10+* [Introduction](#introduction)
9* [Structures & functionality](#structures--functionality)
11* [Structures & functionality](#structures--functionality)
10 * [Partners](#partners)
12 * [Partners](#partners)
11 * [Messages](#messages)
13 * [Messages](#messages)
12− * [Metadata](#metadata)
14+ * [Participant metadata](#participant-metadata)
13− * [Access control](#access-control)
15+ * [Configurations](#configurations)
14− * [Federation](#federation)
15 * [Stories](#stories)
16 * [Stories](#stories)
16 * [General use](#general-use)
17 * [General use](#general-use)
18+ * [Federation](#federation)
17* [Interfaces for applications](#interfaces-for-applications)
19* [Interfaces for applications](#interfaces-for-applications)
18 * [Interactions](#interactions)
20 * [Interactions](#interactions)
19− * [Subscriptions](#subscriptions)
21+ * [Queries, prizes and rumors](#queries-prizes-and-rumors)
20−* [Communication between brokers](#communication-between-brokers)
22+* [Communication between Halls](#communication-between-halls)
21 * [Interactions](#interactions-1)
23 * [Interactions](#interactions-1)
22− * [Subscriptions](#subscriptions-1)
24+ * [Querying](#querying)
25+ * [Federation](#federation)
23−* [Broker implementation](#broker-implementation)
26+* [Hall implementation](#hall-implementation)
24− * [Subscriptions](#subscriptions-2)
27+ * [Subscriptions](#subscriptions)
25− * [Reports](#reports)
28+ * [Rumors](#rumors)
26 * [Messaging](#messaging)
29 * [Messaging](#messaging)
27* [The future](#the-future)
30* [The future](#the-future)
28− * [Architectural issues](#architectural-issues)
29 * [Features and functionality](#features-and-functionality)
31 * [Features and functionality](#features-and-functionality)
30* [Further reading](#further-reading)
32* [Further reading](#further-reading)
34−Talk was Urbit's first big user-facing application, and continues to enjoy a prominent role in the Urbit landscape as the primary messaging bus.
37+Talk was Urbit's first big user-facing application. It continues to enjoy a prominent role in the Urbit landscape, but does so as two separate applications this time.
39+The messaging parts of talk have been separated from its user interface parts. What we ended up with is a shiny new generic messaging bus, and the chat interface we all know and love. The interface retains the `talk` name. The messaging bus, which will prove useful to many more applications, will be named `Hall`.
41+@TODO mention clients are sometimes referred to as "readers".
43+@TODO probably rewrite intro entirely.
36To be clear, we make a distinction between "talk the platform" and "talk the chat application". The latter makes use of the former for most of its functionality, and serves as an example of things that can be built on top of the platform. (Also see the small Twitter clone, [feed](http://urbit.org/fora/posts/~2017.4.12..21.14.00..fe17~/).)
45To be clear, we make a distinction between "talk the platform" and "talk the chat application". The latter makes use of the former for most of its functionality, and serves as an example of things that can be built on top of the platform. (Also see the small Twitter clone, [feed](http://urbit.org/fora/posts/~2017.4.12..21.14.00..fe17~/).)
37−This document, however, focusses on the platform. In talk's case, we refer to this "server" (or `guardian`, in Urbit terminology) as the *broker*. It manages an identity's messages, subscriptions, and more.
46+This document, however, focusses on the platform. In talk's case, we refer to this "server" (or `guardian`, in Urbit terminology) as the *Hall*. It manages an identity's messages, subscriptions, and more.
40## Structures & functionality
49## Structures & functionality
42### Partners
51### Partners
44−One of the broker's strengths is that it can not only interface with other brokers, but also external services. We define a single structure that can be used to specify both kinds of messaging targets.
53+One of Hall's strengths is that it can not only interface with other Hall instances (Halls, if you will), but also external services. We define a single structure that can be used to specify both kinds of messaging targets.
47++ partner (each circle passport) :< message target
56++ partner (each circle passport) :< message target
@@ -51,7 +60,7 @@ One of the broker's strengths is that it can not only interface with other broke
54−A `partner` is either a `circle` or a `passport`. A `circle` is a native partner, essentially a named collection of messages created by and hosted on a ship's broker, usually represented as `~ship-name/circle-name`.
63+A `partner` is either a `circle` or a `passport`. A `circle` is a native partner, essentially a named collection of messages created by and hosted on a ship's Hall, usually represented as `~ship-name/circle-name`.
55A `passport` is a non-native partner, such as a Twitter feed. The possibilities for passports are endless... but none of them are properly implemented yet. For the rest of this document we'll assume a `partner` is always a `circle`.
64A `passport` is a non-native partner, such as a Twitter feed. The possibilities for passports are endless... but none of them are properly implemented yet. For the rest of this document we'll assume a `partner` is always a `circle`.
57### Messages
66### Messages
@@ -59,7 +68,7 @@ A `passport` is a non-native partner, such as a Twitter feed. The possibilities
59When we subscribe to a circle, the primary thing we're interested in is its messages. Message data itself isn't that complicated, but a fair amount of metadata comes into play when actually sending a message. Let's work our way up, starting at the contents.
68When we subscribe to a circle, the primary thing we're interested in is its messages. Message data itself isn't that complicated, but a fair amount of metadata comes into play when actually sending a message. Let's work our way up, starting at the contents.
62−++ speech :> narrative action
71+++ speech :> content body
63 $% {$lin pat/? msg/cord} :< no/@ text line
72 $% {$lin pat/? msg/cord} :< no/@ text line
64 {$ire tos/serial sep/speech} :< in-reply-to
73 {$ire tos/serial sep/speech} :< in-reply-to
65 {$url url/purf} :< parsed url
74 {$url url/purf} :< parsed url
@@ -69,15 +78,15 @@ When we subscribe to a circle, the primary thing we're interested in is its mess
72−At the heart of every message lies a `speech` that describes the message contents. There's a large number of different speech types, from simple text messages to parsed URLs, Hoon expressions and more. Thought the system strives to accommodate all use cases, the `%ext` speech type provides some extensibility for applications that feel their needs aren't covered.
81+At the heart of every message lies a `speech` that describes the message body. There's a large number of different speech types, from simple text messages to parsed URLs, Hoon expressions and more. Thought the system strives to eventually accommodate all use cases, the `%ext` speech type provides extensibility for applications that feel their needs aren't covered.
75−++ statement {wen/@da boq/bouquet sep/speech} :< when this
84+++ statement {wen/@da boq/bouquet sep/speech} :< when what body
76++ bouquet (set flavor) :< complete aroma
85++ bouquet (set flavor) :< complete aroma
77++ flavor path :< content flavor
86++ flavor path :< content flavor
80−Message contents are wrapped in a `statement` which specifies its timestamp and the `flavor` of its content, ¿¿¿ which can be used to filter out content they prefer not to see. ???
89+Message bodies are wrapped in a `statement` which specifies the timestamp and `flavor` of its content, the latter of which can in the future be used to filter out the kinds of content you personally do not wish to see.
83++ audience (map partner (pair envelope delivery)) :< destination + state
92++ audience (map partner (pair envelope delivery)) :< destination + state
@@ -95,14 +104,14 @@ When specifying a message's destination, an `audience` is used. It contains all
97++ telegram {aut/ship tot/thought} :< who thought
106++ telegram {aut/ship tot/thought} :< who thought
98−++ thought {uid/serial aud/audience sam/statement} :< which whom what
107+++ thought {uid/serial aud/audience sam/statement} :< which whom this
99++ serial @uvH :< unique identifier
108++ serial @uvH :< unique identifier
102The message and its destination get put into a `thought`, which also carries a universally unique identifier. This way, recipients can easily check whether they're receiving a new message or a potentially modified version of an existing one.
111The message and its destination get put into a `thought`, which also carries a universally unique identifier. This way, recipients can easily check whether they're receiving a new message or a potentially modified version of an existing one.
103Finally, all this gets put into a `telegram` which tags it with the message's author.
112Finally, all this gets put into a `telegram` which tags it with the message's author.
105−### Metadata
114+### Participant metadata
107Messages aren't the only thing a subscription gets us. We're also kept up to date with relevant metadata.
116Messages aren't the only thing a subscription gets us. We're also kept up to date with relevant metadata.
@@ -111,7 +120,7 @@ Messages aren't the only thing a subscription gets us. We're also kept up to dat
111++ group (map ship status) :< presence map
120++ group (map ship status) :< presence map
112++ status {pec/presence man/human} :< participant
121++ status {pec/presence man/human} :< participant
113++ presence :> status type
122++ presence :> status type
114− $? $gone :< left
123+ $? $gone :< absent
115 $idle :< idle
124 $idle :< idle
116 $hear :< present
125 $hear :< present
117 $talk :< typing
126 $talk :< typing
@@ -125,27 +134,30 @@ Messages aren't the only thing a subscription gets us. We're also kept up to dat
125`status` is user-set metadata that describes, well, the status of users in a circle. This encompasses their `presence`, which shows their activity, and their `human` identity, which includes their display handle.
134`status` is user-set metadata that describes, well, the status of users in a circle. This encompasses their `presence`, which shows their activity, and their `human` identity, which includes their display handle.
126For reasons we'll discover shortly, circles keep track of both their own `group` and those of the partners they're subscribed to. `crowd` encapsulates this.
135For reasons we'll discover shortly, circles keep track of both their own `group` and those of the partners they're subscribed to. `crowd` encapsulates this.
129++ lobby {loc/config rem/(map circle config)} :< our & srcs configs
140++ lobby {loc/config rem/(map circle config)} :< our & srcs configs
130++ config :> circle config
141++ config :> circle config
131− $: src/(set partner) :< pulls from
142+ $: src/(set partner) :< active sources
132 cap/cord :< description
143 cap/cord :< description
144+ fit/filter :< message rules
133 con/control :< restrictions
145 con/control :< restrictions
134− fed/federal :< federators
146+ == ::
147+++ filter :> content filters
148+ $: cas/? :< dis/allow capitals
149+ utf/? :< dis/allow non-ascii
136++ control {sec/security ses/(set ship)} :< access control
151++ control {sec/security ses/(set ship)} :< access control
137−++ federal {may/(set ship) fes/(set ship)} :< federation control
140Another part of metadata we get from circle subscriptions is configurations. Again, circles want to remember their own configuration, as well as those of their subscriptions. But why, precisely?
154Another part of metadata we get from circle subscriptions is configurations. Again, circles want to remember their own configuration, as well as those of their subscriptions. But why, precisely?
142−The `config` structure contains `src`, a set of partners. These indicate the different sources a circle may pull content from. This allows circles to aggregate messages from multiple places. In doing so, it also receives metadata from those places, hence why we have structures for storing "remote" presences and configurations alongside local ones.
156+The `config` structure contains `src`, a set of partners. These indicate the different sources a circle is currently pulling content from. This allows circles to aggregate messages from multiple places. In doing so, it also receives metadata from those places, hence why we have structures for storing "remote" presences and configurations alongside local ones.
144−Aside from a description, a configuration also contains structures to define access and federation control. They're important parts of functionality in their own right, so they get their own sections below.
158+Aside from that, `config` contains a description for the circle, which is exactly what it sounds like. It also specifies a `filter`, which are content formatting rules all messages passing through this circle are made to adhere to.
148−A `control` structure contains both a security mode and a list of ships, which is either a white- or blacklist depending on the aforementioned mode. There are four such modes available.
160+Lastly, there's a `control` structure that contains both a security mode and a list of ships, which is either a white- or blacklist depending on the aforementioned mode. There are four such modes available.
151++ security :> security mode
163++ security :> security mode
@@ -161,20 +173,12 @@ A `village` is privately readable and writable, with a whitelist for inviting.
161A `journal` is publicly readable and privately writable, with a whitelist for authors.
173A `journal` is publicly readable and privately writable, with a whitelist for authors.
162A `mailbox` is readable by its owner and publicly writable, with a blacklist for blocking.
174A `mailbox` is readable by its owner and publicly writable, with a blacklist for blocking.
166−One of the primary aspects of Urbit is its decentralized nature. To make full use of that, the broker can federate circles. That is, circles can be hosted on multiple ships simultaneously. They share all messages, presence and configuration, regardless of where that data or change originated.
167−This can help spread heavy load across multiple ships, makes circles easier to access, and allows for fallback in case one of the hosts goes offline.
169−The `fed` configuration attribute is used to determine which ships are allowed to federate a circle, and keep track of the ones that are currently doing so.
172### Stories
176### Stories
174−To see how that all ties together, we're going to take a look at the broker's state.
178+To see how that all ties together, we're going to take a look at Hall's state.
177−++ state :> broker state
181+++ state :> hall state
178 $: stories/(map knot story) :< conversations
182 $: stories/(map knot story) :< conversations
179 :: ... ::
183 :: ... ::
@@ -188,59 +192,64 @@ To see how that all ties together, we're going to take a look at the broker's st
191−Stories are the primary driver behind the broker. They are the structures that are used to power circles, which we can now say are named stories hosted on ships.
195+Stories are the primary driver behind Hall. They are the structures that are used to power circles, which we can now say are named stories hosted on ships.
193With the configuration described above in mind, we can try and imagine the things we can do with stories. Knowing that we can subscribe them to any number of sources, they can function as central hubs for our messaging, aggregate specific kinds of data feeds, or simply accept whatever messages get sent to it like a regular old chatroom.
197With the configuration described above in mind, we can try and imagine the things we can do with stories. Knowing that we can subscribe them to any number of sources, they can function as central hubs for our messaging, aggregate specific kinds of data feeds, or simply accept whatever messages get sent to it like a regular old chatroom.
195### General use
199### General use
197−Upon initial startup, the broker creates a default story, a mailbox named `porch`. (This name is different for stars and galaxies.) This mailbox is the primary target for anyone and anything that wants to reach its owner. Applications can use it to send notifications and other information to the user (the broker itself does this as well), and users can use it to direct-message others.
201+Upon initial startup, Hall creates a default story, a mailbox named `porch`. (This name is different for stars and galaxies: `floor` and `court` respectively.) This mailbox is the primary target for anyone and anything that wants to reach its owner. Applications can use it to send notifications and other information to the user (Hall itself does this as well), and users can use it to direct-message others.
203+Applications that use Hall are encouraged to use the default mailbox as the primary messaging hub. This way, users can easily switch between different applications without "losing" their subscriptions, message backlog, etc.
205+As an example, Talk operates like this, serving as an interface for reading from and managing the user's mailbox. Local stories are subscribed to through the `porch`, which ends up containing all messages the user receives.
199−Talk reader applications are encouraged to use the default mailbox as the primary messaging hub. This way, users can easily switch between different applications without "losing" their subscriptions, message backlog, etc.
207+### Federation
209+Some places, like `urbit-meta`, will want to be easily accessible to everyone on the network, and actually support the load that comes with that. Both these desired can be met through federation.
201−As an example, the "official" talk reader operates like this, serving as an interface for reading from and managing the user's mailbox. Local stories are subscribed to through the `porch`, which ends up containing all messages the user receives.
211+In federation, stories and any changes to them trickle down from ~zod, to other galaxies, to their stars. Only changes to messages and presence can also move upstream, from stars, to galaxies, to ~zod (and then back down again).
212+This way, all planets and comets will be able to subscribe to `/urbit-meta` and get all updates on the story, regardless of where on the network they originated.
204## Interfaces for applications
215## Interfaces for applications
206−Applications can interact with the broker in two complementary ways: they can tell it what actions to perform, and they can subscribe to its state changes.
217+Applications can interact with Hall in two complementary ways: they can tell it what actions to perform, and they can subscribe to its state changes.
208### Interactions
219### Interactions
210−The broker can be commanded by poking it with `action`s.
221+Hall can be commanded by poking it with `action`s.
213++ action :> user action
224++ action :> user action
214 $% :: circle configuration ::
225 $% :: circle configuration ::
215 {$create nom/knot des/cord sec/security} :< create circle
226 {$create nom/knot des/cord sec/security} :< create circle
216− {$delete nom/knot ano/(unit cord)} :< delete + announce
227+ {$delete nom/knot why/(unit cord)} :< delete + announce
217 {$depict nom/knot des/cord} :< change description
228 {$depict nom/knot des/cord} :< change description
229+ {$filter nom/knot fit/filter} :< change message rules
218 {$permit nom/knot inv/? sis/(set ship)} :< invite/banish
230 {$permit nom/knot inv/? sis/(set ship)} :< invite/banish
219 {$source nom/knot sub/? src/(set partner)} :< un/sub to/from src
231 {$source nom/knot sub/? src/(set partner)} :< un/sub to/from src
220− {$enlist nom/knot fed/? sis/(set ship)} :< dis/allow federation
221− {$burden circle} :< help federate
222 :: messaging ::
232 :: messaging ::
223 {$convey tos/(list thought)} :< post exact
233 {$convey tos/(list thought)} :< post exact
224 {$phrase aud/(set partner) ses/(list speech)} :< post easy
234 {$phrase aud/(set partner) ses/(list speech)} :< post easy
225 :: personal metadata ::
235 :: personal metadata ::
226− {$status cis/(set circle) sat/status} :< our status update
236+ {$notify nos/(set knot) pes/presence} :< our presence update
237+ {$naming nos/(set knot) man/human} :< our name update
227 :: changing shared ui ::
238 :: changing shared ui ::
228 {$human sip/ship man/human} :< new identity
239 {$human sip/ship man/human} :< new identity
229 {$glyph gyf/char pas/(set partner) bin/?} :< un/bind a glyph
240 {$glyph gyf/char pas/(set partner) bin/?} :< un/bind a glyph
233−The largest part of these actions concern themselves with managing local circles. `nom` is always the name used to identify the relevant story. To disambiguate between "add" and "delete" type actions (for permissions, subscriptions and federation), a loob `?` is used.
244+The largest part of these actions concern themselves with managing local circles. `nom` is always the name used to identify the relevant story. To disambiguate between "add" and "delete" type actions (for permissions and subscriptions), a loob `?` is used.
235To send messages, two interfaces are available. `%convey` lets you specify all details of the messages, including its timestamp, serial, and fully assembled audience. `%phrase`, on the other hand, takes care of that for you, allowing you to specify just the target partners and message contents.
246To send messages, two interfaces are available. `%convey` lets you specify all details of the messages, including its timestamp, serial, and fully assembled audience. `%phrase`, on the other hand, takes care of that for you, allowing you to specify just the target partners and message contents.
237−`%status` is useful for setting your own presence and nickname in a circle, so others can see if you're active.
248+`%notify` and `%naming` are useful for setting your own presence and nickname in a circle respectively, so others can see if you're active and what to call you by.
239−> **Current implementation** implements a `%status` command that only works for local circles.
241"Shared UI" encompasses UI data that should be consistent across applications. For example, if the user sets a local nickname for an identity, they expect to see that nickname regardless of the application they're currently using. The same goes for glyph bindings, for easy audience targeting.
250"Shared UI" encompasses UI data that should be consistent across applications. For example, if the user sets a local nickname for an identity, they expect to see that nickname regardless of the application they're currently using. The same goes for glyph bindings, for easy audience targeting.
243−Sometimes, in response to a user action, the broker will respond with a `reaction`.
252+Sometimes, in response to a user action, Hall will respond with a `reaction`.
246++ reaction :> user information
255++ reaction :> user information
@@ -252,218 +261,262 @@ Sometimes, in response to a user action, the broker will respond with a `reactio
253These are used to inform the application (and thus the user) of any unexpected side effects or errors that occurred as a result of the action they issued.
262These are used to inform the application (and thus the user) of any unexpected side effects or errors that occurred as a result of the action they issued.
255−### Subscriptions
264+### Queries, prizes and rumors
257−To receive data from a broker, applications will have to subscribe to it. A `%peer` move for doing so looks something like this:
266+To receive data from Hall, applications will have to query it. There are two paths that are useful for readers to query. Let's look at them and their results.
260−:* ost.bol :< bone
269+++ query :> query paths
261− %peer :< move type
270+ $% {$reader $~} :< shared ui state
262− /story/[some-story] :< diff path
271+ {$circle nom/knot ran/range} :< story query
263− [our %broker] :< peer target
272+ :: ... ::
264− /reader/[some-story] :< peer path
273+ == ::
274+++ range (unit {hed/place tal/(unit place)}) :< msg range, @ud/@da
275+++ place $%({$da @da} {$ud @ud}) :< point for range
276+++ prize :> query result
277+ $% $: $reader :< /reader
278+ gys/(jug char (set partner)) :< glyph bindings
279+ nis/(map ship cord) :< nicknames
281+ {$circle burden} :< /circle
284+++ burden :< full story state
285+ $: gaz/(list telegram) :< all messages
286+ cos/lobby :< loc & rem configs
287+ pes/crowd :< loc & rem presences
268−The diff path is for the application to use to identify what subscription a diff came from. That is, what story the diff relates to. Not all applications will care for this.
291+To be clear, the paths for those queries are `/reader` and `/circle/[name]/[start]/[end]`, where `start` and `end` are optional and can be either a date or a message number.
269−The peer path informs the broker of the intent of the subscription. It always starts with `/reader` to indicate we're a client application, not another broker. If the path ends there, it subscribes to shared UI changes. If it goes on to specify a story name, it subscribes to changes to that story.
292+`/reader` queries produce shared UI state. That is, glyph bindings and nicknames.
293+`/circle` queries produce the entire public state of the requested story, including remotes.
271−These changes get sent as `lowdown`s.
295+Any changes to the results of these queries are communicated via the following `rumor`s. It's a fairly long list of potential changes,
274−++ lowdown :> new/changed state
298+++ rumor :< query result change
275− $% :: story state ::
299+ $% $: $reader :< /reader
276− $: $confs :< configs
300+ $= dif ::
277− loc/(unit config) :< local config
301+ $% {$glyph diff-glyph} ::
278− rem/(map circle (unit config)) :< remote configs
302+ {$nick diff-nick} ::
280− {$precs reg/crowd} :< presences
305+ {$circle dif/diff-story} :< /circle
308+++ diff-glyph {bin/? gyf/char pas/(set partner)} :< un/bound glyph
309+++ diff-nick {who/ship nic/cord} :< changed nickname
311+ $% {$new cof/config} :< new story
281− {$grams num/@ud gaz/(list telegram)} :< messages
312+ {$grams gaz/(list telegram)} :< new/changed msgs
282− :: ui state ::
313+ {$config cir/circle dif/diff-config} :< new/changed config
283− {$glyph (jug char (set partner))} :< glyph bindings
314+ {$status pan/partner who/ship dif/diff-status} :< new/changed status
284− {$names (map ship (unit human))} :< nicknames
315+ {$follow sub/? pas/(set partner)} :< un/subscribe
316+ {$remove $~} :< removed story
319+++ diff-config :> config change
320+ $% {$full cof/config} :< set w/o side-effects
321+ {$source add/? pas/(set partner)} :< add/rem sources
322+ {$caption cap/cord} :< changed description
323+ {$filter fit/filter} :< changed filter
324+ {$secure sec/security} :< changed security
325+ {$permit add/? sis/(set ship)} :< add/rem to b/w-list
326+ {$remove $~} :< removed config
328+++ diff-status :> status change
329+ $% {$full sat/status} :< fully changed status
330+ {$presence pec/presence} :< changed presence
331+ {$human dif/diff-human} :< changed name
332+ {$remove $~} :< removed config
334+++ diff-human :> name change
335+ $% {$full man/human} :< fully changed name
336+ {$true tru/(unit (trel cord (unit cord) cord))} :< changed true name
337+ {$handle han/(unit cord)} :< changed handle
288−Aside from the initial lowdown that is sent when a subscription starts, these contains exclusively the changes as they occur. Configurations are wrapped in `unit`s so that deletion can be indicated. This also means that simple applications don't necessarily need to keep state. If all they want to do is show events as they happen, then the broker subscription will provide all they need.
341+These are, as enforced by Gall, the changes as they happen. This makes it possible for very minimal Hall readers to be implemented that only show a stream of changes, rather than keeping state themselves.
291−## Communication between brokers
344+## Communication between Halls
293−Brokers can communicate with other brokers to send commands and subscription updates.
346+Halls can communicate with other Halls to send commands and subscription updates.
295### Interactions
348### Interactions
297−To request changes to a story, brokers can send `command`s.
350+To request changes to a story, Halls can send `command`s.
300++ command :> effect on story
353++ command :> effect on story
301− $% {$review tos/(list thought)} :< deliver
354+ $% {$publish tos/(list thought)} :< deliver
302− $: $burden :< starting fed state
355+ {$present nos/(set knot) dif/diff-status} :< status update
303− nom/knot ::
356+ :: ... ::
306− gaz/(list telegram) ::
308− {$relief nom/knot who/(set ship)} :< federation ended
312−A `%review` command contains messages. These are sent to foreign brokers to be published to one of their stories.
360+A `%publish` command contains messages. These are sent to foreign Halls to be published to their stories.
314−`%burden` and `%relief` commands are used for federation. The former signals the sender wants to start federating story `nom`, and contains any pre-existing state that needs to be merged into it. The latter signals that the hosts in `who` will no longer be federating the story.
362+The `%present` command is used for sending status updates about our ship.
316−### Subscriptions
364+### Querying
318−Brokers subscribe to other brokers whenever a source gets added to a story's configuration. (Likewise, they unsubscribe when a source is removed.) Here's the peer move we send when we start a subscription:
366+Halls can query other halls. Here's the peer move we send when we open a query on a foreign circle:
321:* ost.bol :< bone
369:* ost.bol :< bone
322 %peer :< move type
370 %peer :< move type
323− /friend/show/[our-circle]/[host]/[their-circle] :< diff path
371+ /circle/[our-circle]/[host]/[their-circle] :< rumor path
324− [host %broker] :< peer target
372+ [host %hall] :< query target
325− /[their-circle]/[start]/[end] :< peer path
373+ /circle/[their-circle]/[start]/[end] :< query path
377+Again, the rumor path is what our Hall uses to identify what query a received message originates from. Our circle is specified so that we know what story is interested in the changes we get.
329−Again, the diff path is just what our broker uses to identify what subscription a received message originates from. The peer path is more interesting. Of course it specifies the name of their story we want to subscribe our story to, but also a "start" and "end". These can be used to specify how long we want our subscription to last. Maybe we want messages starting from last week, up to the end of the month. We may also specify message numbers instead of dates, if we're so inclined.
378+The query path is slightly more interesting. Of course it specifies the name of their story we want to subscribe our story to, but also a "start" and "end". These can (optionally) be used to specify the range of messages we want to get from our query. Once we're out of that range, we stop receiving updates.
331−For as long as their subscriptions last, subscribers get sent `report`s whenever changes occur.
380+### Federation
382+Along the way so far, we've skipped some parts of structures. Most of these relate to federation. Let's see what we missed.
385+++ query :> query paths
386+ $% {$burden who/ship} :< duties to share
334−++ report :> update
387+ {$report $~} :< duty reports
335− $% {$lobby cab/lobby} :< config neighborhood
388+ :: ... ::
336− {$crowd reg/crowd} :< presence
389+ == ::
337− {$grams num/@ud gaz/(list telegram)} :< thoughts
390+++ prize :> query result
391+ $% {$burden sos/(map knot burden)} :< /burden
394+++ rumor :< query result change
395+ $% {$burden nom/knot dif/diff-story} :< /burden
399+ $% {$bear bur/burden} :< new inherited story
400+ {$burden bur/?} :< burden flag
341−There are three different kinds of reports, for configurations, presences, and messages.
405+We briefly touched upon how federation works on the higher level. On the lower, this is how things go down.
343−Attentive eyes may have seen this in lowdowns as well, they contains both local and remote data. This is required for federation to share full state, and some regular subscribers might take interest in that data as well.
407+1. When Hall boots on a star or galaxy, it starts querying its parent's (~zod for galaxies) `/burden` path.
408+2. Upon receiving that query, the parent sends a `%burden` prize containing all state of its channels (fully public circles) to the child. It also subscribes to `/report` on that child.
409+3. When a new story gets created on the parent, its children get a `%burden` rumor with a `%bear` story diff, and they create a local copy of that channel.
410+4. When something about a parent's story changes, a `%burden` rumor is sent to all its children (because they're querying `/burden`). The children apply this change to their local version of the story.
411+5. When a child's story has a `%grams` or `%status` change, a `%burden` rumor is sent to the parent (because they're querying `/report`). The parent applies this change to their local version of the story, and #4 happens.
346−## Broker implementation
414+## Hall implementation
348Now that we know what gets sent over the wire and why, let's see what happens whenever such events happen. We won't be covering everything, but enough to illustrate the general flow.
416Now that we know what gets sent over the wire and why, let's see what happens whenever such events happen. We won't be covering everything, but enough to illustrate the general flow.
350−First though, it's useful to understand how the broker's code is structured. It consists of two primary cores that are used as engines for applying changes to state and producing moves for output.
418+First though, it's useful to understand how Hall's code is structured. It consists of primary cores, designed to work optimally with Gall.
420+* `++ta` is the transaction core. For every event that happens, be it a poke, peer or diff, the transaction core gets put to work to process it. Once it's done, it produces a list of deltas that describe how our state needs to be modified.
421+* `++da` is the delta application core. It is used to apply `++ta`'s deltas to our application state, and produce a list of side-effects.
352−* `++ta` is the transaction core. For every event that happens, be it a poke, peer of diff, the transaction core gets put to work to process it. Once it's done, it produces a list of moves and the new state.
423+Both those cores also have a core within themselves dedicated solely to dealing with stories, `++so` and `++sa` respectively.
353−* `++so` is the story core, used by `++ta` for applying changes to stories. In doing so, it may add moves to the transaction core for it to produce.
355−Once all work in an engine core has finished, its `-done` arm is called to produce moves and apply the changes it made back into the application state.
425+Once all work in an engine core has finished, its `-done` arm is called to produce the result of all the core's computation.
357In the diagrams that follow, regular lines indicate flows that always happen. Dashed lines indicate flows that are traversed if the described condition is met.
427In the diagrams that follow, regular lines indicate flows that always happen. Dashed lines indicate flows that are traversed if the described condition is met.
358Below the diagrams you will find brief explanations of the involved arms. In a particularly bright future, these might be available on-hover instead.
428Below the diagrams you will find brief explanations of the involved arms. In a particularly bright future, these might be available on-hover instead.
359For brevity, utility arms that aren't a direct and important part of the flow have been omitted.
429For brevity, utility arms that aren't a direct and important part of the flow have been omitted.
431+(You will find that the origins still use old Gall arms, with some glue code in between. At the time of writing, new Gall has not yet been implemented, so this will have to do.)
433+We will generally only display the delta generation parts of the flow. Delta application is usually implemented simply enough to deduce what it does from looking at the label.
361### Subscriptions
435### Subscriptions
363−Broker-to-broker subscriptions happen when a source gets added to a story. This is done with a `%source` action, and results in a `%peer` move being sent. Upon receiving a peer, the broker validates the subscription and sends the current state.
437+Hall-to-Hall subscriptions happen when a source gets added to a story. This is done with a `%source` action, and results in a `%peer` move being sent (prompted by a `%story %follow` delta).
438+Upon receiving a valid peer on a `/circle` path, the subscribing ship is added to that circle's presence map.
365
440
368−++ poke-talk-action :< we got poked with an action.
443+++ poke-talk-action :< we got poked with an action.
369− ++ ta-action :< applies an action.
444+ ++ ta-action :< applies an action.
370− ++ ta-config :< (re)configures a story.
445+ ++ ta-config :< (re)configures a story.
371− ++ so-reform :< changes the configuration of the story.
372− ++ so-acquire :< subscribes the story to new sources.
374−++ peer :< we got peered.
447+++ peer :< we got peered.
375− ++ ta-subscribe :< sends initial data: current state.
448+ ++ g-query :< resolves query to prize.
376− ++ so-report-lobby :< sends a report with configurations.
449+ ++ ta-subscribe :< act upon subscription.
377− ++ so-notify :< changes a ship's presence in the story.
450+ ++ so-attend :< adds a ship's status to the story.
378− ++ so-report-crowd :< sends a report with presences.
379− ++ so-start :< determines the backlog of messages to send.
380− ++ so-first-grams :< sends a range of messages and adds follower.
383−### Reports
453+### Rumors
385−Once a subscription has been established, the broker will receive reports. `%lobby` and `%group` reports are applied through the following flow. Note how, when a `%lobby` from a federating ship is received, this authoritatively changes the story's configuration.
455+Once a query has opened, Hall will receive updates on it, rumors. The changes describes in these rumors are applied via the following flow.
387−
457+
390−++ diff-talk-report :< we got a subscription report.
460+++ diff-talk-rumor :< we got a query update.
391− ++ ta-diff-report :< applies a report to the story it's intended for.
461+ ++ ta-hear :< applies a rumor to the story it's intended for.
392− ++ so-diff-report :< applies a report.
462+ ++ so-hear :< applies a %circle rumor to the story.
393− ++ so-lobby :< stores updated configuration.
463+ ++ so-config-full :< splits a %full diff-config into separate changes.
394− ++ so-reform :< changes the configuration of the story.
464+ ++ so-bear :< accept burden, assimilate into the story's state.
395− ++ so-report-lobby :< sends a report with configuration.
396− ++ so-remind :< stores updated presences.
397− ++ so-report-crowd :< sends a report with presences.
465+ ++ so-lesson :< sends a report with presences.
400### Messaging
468### Messaging
402−To send messages, the user sends a `%convey` or `%phrase` action, resulting in a `%review` command being sent to the involved partners. Receiving a `%review` command causes its messages to be added to the story, which sends a `%grams` report to subscribers and a `%grams` lowdown to all reader applications.
470+To send messages, the user sends a `%convey` or `%phrase` action, resulting in a `%publish` command being sent to the involved partners. Receiving a `%publish` command causes its messages to be added to the story through the creation of a `%story %grams` delta.
404
472
407−++ poke-talk-action :< we got poked with an action.
475+++ poke-talk-action :< we got poked with an action.
408− ++ ta-action :< applies an action.
476+ ++ ta-action :< applies an action.
410−++ poke-talk-command :< we get poked with a command.
478+++ poke-talk-command :< we get poked with a command.
411− ++ ta-apply :< applies a command.
479+ ++ ta-apply :< applies a command.
412− ++ ta-think :< consumes each message in the given list.
480+ ++ ta-think :< consumes each message in the given list.
413− ++ ta-consume :< conducts a message to each partner in audience.
481+ ++ ta-consume :< conducts a message to each partner in audience.
414− ++ ta-conduct :< records or sends a message.
482+ ++ ta-conduct :< records or sends a message.
415− ++ ta-transmit :< sends a message to a partner.
483+ ++ ta-transmit :< sends a message to a partner.
416− ++ ta-record :< stores a message in a story.
484+ ++ ta-record :< stores a message in a story.
418−++ diff-talk-report :< we got a subscription report.
419− ++ ta-diff-report :< applies a report to the story it's intended for.
420− ++ so-diff-report :< applies a report.
421− ++ so-lesson :< learns a list of messages.
422− ++ so-learn :< either adds or modifies a message.
485+ ++ so-learn :< either adds or modifies a message.
423− ++ so-append :< adds a new message to the story.
424− ++ so-revise :< modifies an existing message in the story.
425− ++ so-refresh :< sends messages to all interested subscribers.
429## The future
489## The future
431−Talk is neither complete nor perfect. There are still problems that need to be solved and features that need to be implemented.
491+Hall is neither complete nor perfect. There are features that need to be implemented, varying from small quality of life changes to broad-reaching functionality.
433−### Architectural issues
435−To distribute commands and reports to all that are affected by it, the broker takes a "propagate changes" approach. If, in processing a report, any state changes, new state gets sent to all interested brokers... even those it got the report from in the first place. This results in a lot of unnecessary data transfer, where everyone always gets notified of each other's state changes, regardless of whether those changes are old news or not.
437−While not a practical problem (yet, things work fine at the current scale of use), it does underline the need for a more structured approach to the broker's subscription model. In an ideal world, every interested node gets notified of a set of changes exactly once.
439−That still becomes unwieldy for circles that have thousands of subscribers, since a single node would still be sending thousands of network events. An obvious potential solution is to apply Urbit's hierarchical structure to this, and send messages to stars for them to send to their children.
441−While on the topic of subscriptions, the current implementation treats them as semantic data, explicitly storing them in application state. It would be desirable to push this to the background again.
443### Features and functionality
493### Features and functionality
445To turn talk into a [sustainable social platform](https://urbit.org/fora/posts/~2017.4.26..18.00.25..b93c~/), it needs a number of things.
495To turn talk into a [sustainable social platform](https://urbit.org/fora/posts/~2017.4.26..18.00.25..b93c~/), it needs a number of things.
447−Primarily, discoverability of circles. A possible solution for realizing this is making it possible for users to add friends, which would have their broker subscribe to a list of circles the added friends are in. Of course, one will be able to set joined circles as "private" to keep them from being visible to others.
497+Primarily, discoverability of circles. A possible solution for realizing this is making it possible for users to add friends, which would have their Hall subscribe to a list of circles the added friends are in. Of course, one will be able to set joined circles as "private" to keep them from being visible to others.
449Eventually, content self-moderation might need to be implemented. Users would flag their content if it is potentially offensive. Others would easily be able to filter out content that might offend them.
499Eventually, content self-moderation might need to be implemented. Users would flag their content if it is potentially offensive. Others would easily be able to filter out content that might offend them.
450This functionality brings its own large sets of challenges that need to be tackled.
500This functionality brings its own large sets of challenges that need to be tackled.
452−Additionally, federation might be put to broader use. Knowing a circle is federated could help fall back to alternative hosts in case the one the broker originally subscribed to is unavailable.
502+Additionally, federation might be put to broader use. Knowing a circle is federated could help fall back to alternative hosts in case the one Hall originally subscribed to is unavailable.
453Implementation-wise, care would need to be taken for this to not get ugly. Not to mention, when/how would availability of a host be verified?
503Implementation-wise, care would need to be taken for this to not get ugly. Not to mention, when/how would availability of a host be verified?
505+Other changes that might have a fair impact on the functioning of Hall are un/read states for messages, and being able to use moons to allow your friends to chat with you in a local channel.
455Smaller functionality that still needs to be implemented includes:
507Smaller functionality that still needs to be implemented includes:
456* Extended permissions management. Being able to black-/whitelist entire ship classes.
508* Extended permissions management. Being able to black-/whitelist entire ship classes.
457* Improved presence functionality. Actually using the presence system, and sending "typing", "idle", etc. statuses.
509* Improved presence functionality. Actually using the presence system, and sending "typing", "idle", etc. statuses.
458−* Per-circle message rules. Dis/allow unicode, enforce line length, etc.
459* Flexible on-subscribe scrollback. Some cases may want more than just a day's worth.
510* Flexible on-subscribe scrollback. Some cases may want more than just a day's worth.
460−* Talkpolls.
511+* Polls.
513+And of course there's a lot Talk, the client, can improve on as well.
463## Further reading
516## Further reading
465−To gain a more thorough understanding of the broker's inner workings, take a look at its source code. It comes with inline documentation.
518+To gain a more thorough understanding of Hall's inner workings, take a look at its source code. It comes with inline documentation.
468−To see an expansive example of an application that uses the broker, take a look at the code of the talk reader. This, too, comes with inline documentation.
521+To see an expansive example of an application that uses Hall, take a look at the code of Talk. It, too, comes with inline documentation.