@@ -6,11 +6,13 @@ This document is complemented by talk's source code, but doesn't require it. Def
66## Table of contents
77
88* [Overview](#overview)
9−* [Core concepts & functionality](#core-concepts--functionality)
10− * [Stories](#stories)
9+* [Structures & functionality](#structures--functionality)
1110 * [Partners](#partners)
11+ * [Messages](#messages)
12+ * [Metadata](#metadata)
1213 * [Access control](#access-control)
1314 * [Federation](#federation)
15+ * [Stories](#stories)
1416 * [General use](#general-use)
1517* [Interfaces for applications](#interfaces-for-applications)
1618 * [Interactions](#interactions)
@@ -35,39 +37,96 @@ To be clear, we make a distinction between "talk the platform" and "talk the cha
3537This document, however, focusses on the platform. In talk's case, we refer to this "server" (or `guardian`, in Urbit terminology) as the *broker*. It manages an identity's messages, subscriptions, and more.
3638
3739
38−## Core concepts & functionality
40+## Structures & functionality
41+
42+### Partners
3943
40−To begin to gain an understanding of how the broker functions, let's look at its state model.
44+One of the broker's strengths is that it can not only interface with other brokers, but also external services. We define a single structure that can be used to specify both kinds of messaging targets.
4145
4246```
43−++ state :> broker state
44− $: stories/(map knot story) :< conversations
45− readers/(map bone (set knot)) :< our message readers
46− :: ... ::
47+++ partner (each circle passport) :< message target
48+++ circle {hos/ship nom/knot} :< native target
49+++ passport :> foreign target
50+ $% {$twitter p/cord} :< twitter handle
4751 == ::
4852```
4953
50−Aside from keeping track of `readers`, the different applications that use the broker, it also stores a map of named stories.
54+A `partner` is either a `circle` or a `passport`. A `circle` is a native partner, essentially a named collection of messages created by and hosted on a ship's broker, usually represented as `~ship-name/circle-name`.
55+A `passport` is a non-native partner, such as a Twitter feed. The possibilities for passports are endless... but none of them are properly implemented yet. For the rest of this document we'll assume a `partner` is always a `circle`.
5156
52−### Stories
57+### Messages
58+
59+When we subscribe to a circle, the primary thing we're interested in is its messages. Message data itself isn't that complicated, but a fair amount of metadata comes into play when actually sending a message. Let's work our way up, starting at the contents.
5360
5461```
55−++ story :> wire content
56− $: grams/(list telegram) :< all messages
57− locals/group :< local presence
58− remotes/(map partner group) :< remote presence
59− shape/config :< configuration
60− mirrors/(map circle config) :< remote config
61− followers/(map bone river) :< subscribers
62+++ speech :> narrative action
63+ $% {$lin pat/? msg/cord} :< no/@ text line
64+ {$ire tos/serial sep/speech} :< in-reply-to
65+ {$url url/purf} :< parsed url
66+ {$app app/term msg/cord} :< app message
67+ {$ext nom/term dat/*} :< extended action
6268 :: ... ::
6369 == ::
6470```
6571
66−Stories are the primary driver behind the broker. Whether you're hosting a group chat, or just want a place to aggregate messages into, a `story` will be created for it. They contain all data they need to be complete, from messages to metadata. Let's see what that metadata is made up of.
72+At the heart of every message lies a `speech` that describes the message contents. There's a large number of different speech types, from simple text messages to parsed URLs, Hoon expressions and more. Thought the system strives to accommodate all use cases, the `%ext` speech type provides some extensibility for applications that feel their needs aren't covered.
73+
74+```
75+++ statement {wen/@da boq/bouquet sep/speech} :< when this
76+++ bouquet (set flavor) :< complete aroma
77+++ flavor path :< content flavor
78+```
79+
80+Message contents are wrapped in a `statement` which specifies its timestamp and the `flavor` of its content, ¿¿¿ which can be used to filter out content they prefer not to see. ???
6781
6882```
83+++ audience (map partner (pair envelope delivery)) :< destination + state
84+++ envelope {vis/? sen/(unit partner)} :< visible sender
85+++ delivery :> delivery state
86+ $? $pending :< undelivered
87+ $received :< delivered
88+ $rejected :< undeliverable
89+ $released :< sent one-way
90+ $accepted :< fully processed
91+ == ::
92+```
93+
94+When specifying a message's destination, an `audience` is used. It contains all the recipients of the message, as well as the delivery state for each of those recipients.
95+
96+```
97+++ telegram {aut/ship tot/thought} :< who thought
98+++ thought {uid/serial aud/audience sam/statement} :< which whom what
99+++ serial @uvH :< unique identifier
100+```
101+
102+The message and its destination get put into a `thought`, which also carries a universally unique identifier. This way, recipients can easily check whether they're receiving a new message or a potentially modified version of an existing one.
103+Finally, all this gets put into a `telegram` which tags it with the message's author.
104+
105+### Metadata
106+
107+Messages aren't the only thing a subscription gets us. We're also kept up to date with relevant metadata.
108+
109+```
110+++ crowd {loc/group rem/(map partner group)} :< our & srcs presences
69111++ group (map ship status) :< presence map
70112++ status {pec/presence man/human} :< participant
113+++ presence :> status type
114+ $? $gone :< left
115+ $idle :< idle
116+ $hear :< present
117+ $talk :< typing
118+ == ::
119+++ human :> human identifier
120+ $: tru/(unit (trel cord (unit cord) cord)) :< true name
121+ han/(unit cord) :< handle
122+ == ::
123+```
124+
125+`status` is user-set metadata that describes, well, the status of users in a circle. This encompasses their `presence`, which shows their activity, and their `human` identity, which includes their display handle.
126+For reasons we'll discover shortly, circles keep track of both their own `group` and those of the partners they're subscribed to. `crowd` encapsulates this.
127+
128+```
129+++ lobby {loc/config rem/(map circle config)} :< our & srcs configs
71130++ config :> circle config
72131 $: src/(set partner) :< pulls from
73132 cap/cord :< description
@@ -78,32 +137,15 @@ Stories are the primary driver behind the broker. Whether you're hosting a group
78137++ federal {may/(set ship) fes/(set ship)} :< federation control
79138```
80139
81−A `group` consists of ships and their `status` within a story. A status encompasses both `presence` (whether they're online, idle, typing, etc.) and `human` (their display name).
82−
83−A `config` contains all configuration for a story, from its description to its access control. (More on [`con`](#access-control) and [`fed`](#federation) below.) The `src` attribute in configurations is worth elaborating on.
84−
85−Every story has a set of sources from which it receives data. If there's nothing in this set, the story will only receive messages people explicitly target towards it. If there *are* sources in that set, then their message will get forwarded to the story automatically. Presences and configurations for those sources are kept track of as well, which is why we see not only "local" but also "remote" data in the story structure.
86−
87−In short, stories store messages. They get these either directly, or through any sources they have subscribed to.
140+Another part of metadata we get from circle subscriptions is configurations. Again, circles want to remember their own configuration, as well as those of their subscriptions. But why, precisely?
88141
89−### Partners
90−
91−Though these sources are `partner` structures, we can see remote configurations are only stored for so-called `circle`s.
92−
93−```
94−++ partner (each circle passport) :< message target
95−++ circle {hos/ship nom/knot} :< native target
96−++ passport :> foreign target
97− $% {$twitter p/cord} :< twitter handle
98− == ::
99−```
142+The `config` structure contains `src`, a set of partners. These indicate the different sources a circle may pull content from. This allows circles to aggregate messages from multiple places. In doing so, it also receives metadata from those places, hence why we have structures for storing "remote" presences and configurations alongside local ones.
100143
101−A `partner` is either a `circle` or a `passport`. A `circle` is a native partner, one created by the broker: a named story hosted on a ship, usually represented as `~ship/story`.
102−A `passport` is a non-native partner, such as a Twitter feed. The possibilities for passports are endless... but none of them are properly implemented yet, so for the rest of this document we'll assume a `partner` is always a `circle`.
144+Aside from a description, a configuration also contains structures to define access and federation control. They're important parts of functionality in their own right, so they get their own sections below.
103145
104146### Access control
105147
106−Though not important to the broker's architecture, the way access control for stories works is useful to understand. There are currently four security modes:
148+A `control` structure contains both a security mode and a list of ships, which is either a white- or blacklist depending on the aforementioned mode. There are four such modes available.
107149
108150```
109151++ security :> security mode
@@ -114,7 +156,7 @@ Though not important to the broker's architecture, the way access control for st
114156 == ::
115157```
116158
117−A `channel` is publicly readable and writable, with a blacklist for banishing.
159+A `channel` is publicly readable and writable, with a blacklist for blocking.
118160A `village` is privately readable and writable, with a whitelist for inviting.
119161A `journal` is publicly readable and privately writable, with a whitelist for authors.
120162A `mailbox` is readable by its owner and publicly writable, with a blacklist for blocking.
@@ -126,6 +168,30 @@ This can help spread heavy load across multiple ships, makes circles easier to a
126168
127169The `fed` configuration attribute is used to determine which ships are allowed to federate a circle, and keep track of the ones that are currently doing so.
128170
171+
172+### Stories
173+
174+To see how that all ties together, we're going to take a look at the broker's state.
175+
176+```
177+++ state :> broker state
178+ $: stories/(map knot story) :< conversations
179+ :: ... ::
180+ == ::
181+++ story :> wire content
182+ $: grams/(list telegram) :< all messages
183+ locals/group :< local presence
184+ remotes/(map partner group) :< remote presence
185+ shape/config :< configuration
186+ mirrors/(map circle config) :< remote config
187+ :: ... ::
188+ == ::
189+```
190+
191+Stories are the primary driver behind the broker. They are the structures that are used to power circles, which we can now say are named stories hosted on ships.
192+
193+With the configuration described above in mind, we can try and imagine the things we can do with stories. Knowing that we can subscribe them to any number of sources, they can function as central hubs for our messaging, aggregate specific kinds of data feeds, or simply accept whatever messages get sent to it like a regular old chatroom.
194+
129195### General use
130196
131197Upon initial startup, the broker creates a default story, a mailbox named `porch`. (This name is different for stars and galaxies.) This mailbox is the primary target for anyone and anything that wants to reach its owner. Applications can use it to send notifications and other information to the user (the broker itself does this as well), and users can use it to direct-message others.
@@ -191,7 +257,7 @@ These are used to inform the application (and thus the user) of any unexpected s
191257To receive data from a broker, applications will have to subscribe to it. A `%peer` move for doing so looks something like this:
192258
193259```
194−:* ost.hid :< bone
260+:* ost.bol :< bone
195261 %peer :< move type
196262 /story/[some-story] :< diff path
197263 [our %broker] :< peer target
@@ -217,7 +283,6 @@ These changes get sent as `lowdown`s.
217283 {$glyph (jug char (set partner))} :< glyph bindings
218284 {$names (map ship (unit human))} :< nicknames
219285 == ::
220−++ crowd {loc/group rem/(map partner group)} :< our & srcs presences
221286```
222287
223288Aside from the initial lowdown that is sent when a subscription starts, these contains exclusively the changes as they occur. Configurations are wrapped in `unit`s so that deletion can be indicated. This also means that simple applications don't necessarily need to keep state. If all they want to do is show events as they happen, then the broker subscription will provide all they need.
@@ -242,8 +307,6 @@ To request changes to a story, brokers can send `command`s.
242307 == ::
243308 {$relief nom/knot who/(set ship)} :< federation ended
244309 == ::
245−++ lobby {loc/config rem/(map circle config)} :< our & srcs configs
246−++ crowd {loc/group rem/(map partner group)} :< our & srcs presences
247310```
248311
249312A `%review` command contains messages. These are sent to foreign brokers to be published to one of their stories.
@@ -252,7 +315,20 @@ A `%review` command contains messages. These are sent to foreign brokers to be p
252315
253316### Subscriptions
254317
255−Brokers subscribe to other brokers whenever a source gets added to a story's configuration. (Likewise, they unsubscribe when a source is removed.) Subscribers, also knows as story "followers", get sent `report`s whenever changes occur.
318+Brokers subscribe to other brokers whenever a source gets added to a story's configuration. (Likewise, they unsubscribe when a source is removed.) Here's the peer move we send when we start a subscription:
319+
320+```
321+:* ost.bol :< bone
322+ %peer :< move type
323+ /friend/show/[our-circle]/[host]/[their-circle] :< diff path
324+ [host %broker] :< peer target
325+ /[their-circle]/[start]/[end] :< peer path
326+== ::
327+```
328+
329+Again, the diff path is just what our broker uses to identify what subscription a received message originates from. The peer path is more interesting. Of course it specifies the name of their story we want to subscribe our story to, but also a "start" and "end". These can be used to specify how long we want our subscription to last. Maybe we want messages starting from last week, up to the end of the month. We may also specify message numbers instead of dates, if we're so inclined.
330+
331+For as long as their subscriptions last, subscribers get sent `report`s whenever changes occur.
256332
257333```
258334++ report :> update
@@ -260,8 +336,6 @@ Brokers subscribe to other brokers whenever a source gets added to a story's con
260336 {$crowd reg/crowd} :< presence
261337 {$grams num/@ud gaz/(list telegram)} :< thoughts
262338 == ::
263−++ lobby {loc/config rem/(map circle config)} :< our & srcs configs
264−++ crowd {loc/group rem/(map partner group)} :< our & srcs presences
265339```
266340
267341There are three different kinds of reports, for configurations, presences, and messages.
@@ -325,7 +399,7 @@ Once a subscription has been established, the broker will receive reports. `%lob
325399
326400### Messaging
327401
328−To send messages, the user sends a `%convey` or `%phrase` action, resulting in a `%review` command being sent to the involved partners. Receiving a `%review` command causes its messages to be added to the story, which sends a `%grams` report to followers and a `%grams` lowdown to all reader applications.
402+To send messages, the user sends a `%convey` or `%phrase` action, resulting in a `%review` command being sent to the involved partners. Receiving a `%review` command causes its messages to be added to the story, which sends a `%grams` report to subscribers and a `%grams` lowdown to all reader applications.
329403
330404
331405
@@ -362,6 +436,10 @@ To distribute commands and reports to all that are affected by it, the broker ta
362436
363437While not a practical problem (yet, things work fine at the current scale of use), it does underline the need for a more structured approach to the broker's subscription model. In an ideal world, every interested node gets notified of a set of changes exactly once.
364438
439+That still becomes unwieldy for circles that have thousands of subscribers, since a single node would still be sending thousands of network events. An obvious potential solution is to apply Urbit's hierarchical structure to this, and send messages to stars for them to send to their children.
440+
441+While on the topic of subscriptions, the current implementation treats them as semantic data, explicitly storing them in application state. It would be desirable to push this to the background again.
442+
365443### Features and functionality
366444
367445To turn talk into a [sustainable social platform](https://urbit.org/fora/posts/~2017.4.26..18.00.25..b93c~/), it needs a number of things.