@@ -6,11 +6,13 @@ This document is complemented by talk's source code, but doesn't require it. Def
6## Table of contents
6## Table of contents
8* [Overview](#overview)
8* [Overview](#overview)
9−* [Core concepts & functionality](#core-concepts--functionality)
9+* [Structures & functionality](#structures--functionality)
10− * [Stories](#stories)
11 * [Partners](#partners)
10 * [Partners](#partners)
11+ * [Messages](#messages)
12+ * [Metadata](#metadata)
12 * [Access control](#access-control)
13 * [Access control](#access-control)
13 * [Federation](#federation)
14 * [Federation](#federation)
15+ * [Stories](#stories)
14 * [General use](#general-use)
16 * [General use](#general-use)
15* [Interfaces for applications](#interfaces-for-applications)
17* [Interfaces for applications](#interfaces-for-applications)
16 * [Interactions](#interactions)
18 * [Interactions](#interactions)
@@ -35,39 +37,96 @@ To be clear, we make a distinction between "talk the platform" and "talk the cha
35This document, however, focusses on the platform. In talk's case, we refer to this "server" (or `guardian`, in Urbit terminology) as the *broker*. It manages an identity's messages, subscriptions, and more.
37This document, however, focusses on the platform. In talk's case, we refer to this "server" (or `guardian`, in Urbit terminology) as the *broker*. It manages an identity's messages, subscriptions, and more.
38−## Core concepts & functionality
40+## Structures & functionality
40−To begin to gain an understanding of how the broker functions, let's look at its state model.
44+One of the broker's strengths is that it can not only interface with other brokers, but also external services. We define a single structure that can be used to specify both kinds of messaging targets.
43−++ state :> broker state
47+++ partner (each circle passport) :< message target
44− $: stories/(map knot story) :< conversations
48+++ circle {hos/ship nom/knot} :< native target
45− readers/(map bone (set knot)) :< our message readers
49+++ passport :> foreign target
46− :: ... ::
50+ $% {$twitter p/cord} :< twitter handle
50−Aside from keeping track of `readers`, the different applications that use the broker, it also stores a map of named stories.
54+A `partner` is either a `circle` or a `passport`. A `circle` is a native partner, essentially a named collection of messages created by and hosted on a ship's broker, usually represented as `~ship-name/circle-name`.
55+A `passport` is a non-native partner, such as a Twitter feed. The possibilities for passports are endless... but none of them are properly implemented yet. For the rest of this document we'll assume a `partner` is always a `circle`.
52−### Stories
57+### Messages
59+When we subscribe to a circle, the primary thing we're interested in is its messages. Message data itself isn't that complicated, but a fair amount of metadata comes into play when actually sending a message. Let's work our way up, starting at the contents.
55−++ story :> wire content
62+++ speech :> narrative action
56− $: grams/(list telegram) :< all messages
63+ $% {$lin pat/? msg/cord} :< no/@ text line
57− locals/group :< local presence
64+ {$ire tos/serial sep/speech} :< in-reply-to
58− remotes/(map partner group) :< remote presence
65+ {$url url/purf} :< parsed url
59− shape/config :< configuration
66+ {$app app/term msg/cord} :< app message
60− mirrors/(map circle config) :< remote config
67+ {$ext nom/term dat/*} :< extended action
61− followers/(map bone river) :< subscribers
66−Stories are the primary driver behind the broker. Whether you're hosting a group chat, or just want a place to aggregate messages into, a `story` will be created for it. They contain all data they need to be complete, from messages to metadata. Let's see what that metadata is made up of.
72+At the heart of every message lies a `speech` that describes the message contents. There's a large number of different speech types, from simple text messages to parsed URLs, Hoon expressions and more. Thought the system strives to accommodate all use cases, the `%ext` speech type provides some extensibility for applications that feel their needs aren't covered.
75+++ statement {wen/@da boq/bouquet sep/speech} :< when this
76+++ bouquet (set flavor) :< complete aroma
77+++ flavor path :< content flavor
80+Message contents are wrapped in a `statement` which specifies its timestamp and the `flavor` of its content, ¿¿¿ which can be used to filter out content they prefer not to see. ???
83+++ audience (map partner (pair envelope delivery)) :< destination + state
84+++ envelope {vis/? sen/(unit partner)} :< visible sender
85+++ delivery :> delivery state
86+ $? $pending :< undelivered
87+ $received :< delivered
88+ $rejected :< undeliverable
89+ $released :< sent one-way
90+ $accepted :< fully processed
94+When specifying a message's destination, an `audience` is used. It contains all the recipients of the message, as well as the delivery state for each of those recipients.
97+++ telegram {aut/ship tot/thought} :< who thought
98+++ thought {uid/serial aud/audience sam/statement} :< which whom what
99+++ serial @uvH :< unique identifier
102+The message and its destination get put into a `thought`, which also carries a universally unique identifier. This way, recipients can easily check whether they're receiving a new message or a potentially modified version of an existing one.
103+Finally, all this gets put into a `telegram` which tags it with the message's author.
107+Messages aren't the only thing a subscription gets us. We're also kept up to date with relevant metadata.
110+++ crowd {loc/group rem/(map partner group)} :< our & srcs presences
69++ group (map ship status) :< presence map
111++ group (map ship status) :< presence map
70++ status {pec/presence man/human} :< participant
112++ status {pec/presence man/human} :< participant
113+++ presence :> status type
119+++ human :> human identifier
120+ $: tru/(unit (trel cord (unit cord) cord)) :< true name
121+ han/(unit cord) :< handle
125+`status` is user-set metadata that describes, well, the status of users in a circle. This encompasses their `presence`, which shows their activity, and their `human` identity, which includes their display handle.
126+For reasons we'll discover shortly, circles keep track of both their own `group` and those of the partners they're subscribed to. `crowd` encapsulates this.
129+++ lobby {loc/config rem/(map circle config)} :< our & srcs configs
71++ config :> circle config
130++ config :> circle config
72 $: src/(set partner) :< pulls from
131 $: src/(set partner) :< pulls from
73 cap/cord :< description
132 cap/cord :< description
@@ -78,32 +137,15 @@ Stories are the primary driver behind the broker. Whether you're hosting a group
78++ federal {may/(set ship) fes/(set ship)} :< federation control
137++ federal {may/(set ship) fes/(set ship)} :< federation control
81−A `group` consists of ships and their `status` within a story. A status encompasses both `presence` (whether they're online, idle, typing, etc.) and `human` (their display name).
140+Another part of metadata we get from circle subscriptions is configurations. Again, circles want to remember their own configuration, as well as those of their subscriptions. But why, precisely?
83−A `config` contains all configuration for a story, from its description to its access control. (More on [`con`](#access-control) and [`fed`](#federation) below.) The `src` attribute in configurations is worth elaborating on.
85−Every story has a set of sources from which it receives data. If there's nothing in this set, the story will only receive messages people explicitly target towards it. If there *are* sources in that set, then their message will get forwarded to the story automatically. Presences and configurations for those sources are kept track of as well, which is why we see not only "local" but also "remote" data in the story structure.
87−In short, stories store messages. They get these either directly, or through any sources they have subscribed to.
89−### Partners
142+The `config` structure contains `src`, a set of partners. These indicate the different sources a circle may pull content from. This allows circles to aggregate messages from multiple places. In doing so, it also receives metadata from those places, hence why we have structures for storing "remote" presences and configurations alongside local ones.
91−Though these sources are `partner` structures, we can see remote configurations are only stored for so-called `circle`s.
94−++ partner (each circle passport) :< message target
95−++ circle {hos/ship nom/knot} :< native target
96−++ passport :> foreign target
97− $% {$twitter p/cord} :< twitter handle
101−A `partner` is either a `circle` or a `passport`. A `circle` is a native partner, one created by the broker: a named story hosted on a ship, usually represented as `~ship/story`.
144+Aside from a description, a configuration also contains structures to define access and federation control. They're important parts of functionality in their own right, so they get their own sections below.
102−A `passport` is a non-native partner, such as a Twitter feed. The possibilities for passports are endless... but none of them are properly implemented yet, so for the rest of this document we'll assume a `partner` is always a `circle`.
104### Access control
146### Access control
106−Though not important to the broker's architecture, the way access control for stories works is useful to understand. There are currently four security modes:
148+A `control` structure contains both a security mode and a list of ships, which is either a white- or blacklist depending on the aforementioned mode. There are four such modes available.
109++ security :> security mode
151++ security :> security mode
@@ -114,7 +156,7 @@ Though not important to the broker's architecture, the way access control for st
117−A `channel` is publicly readable and writable, with a blacklist for banishing.
159+A `channel` is publicly readable and writable, with a blacklist for blocking.
118A `village` is privately readable and writable, with a whitelist for inviting.
160A `village` is privately readable and writable, with a whitelist for inviting.
119A `journal` is publicly readable and privately writable, with a whitelist for authors.
161A `journal` is publicly readable and privately writable, with a whitelist for authors.
120A `mailbox` is readable by its owner and publicly writable, with a blacklist for blocking.
162A `mailbox` is readable by its owner and publicly writable, with a blacklist for blocking.
@@ -126,6 +168,30 @@ This can help spread heavy load across multiple ships, makes circles easier to a
127The `fed` configuration attribute is used to determine which ships are allowed to federate a circle, and keep track of the ones that are currently doing so.
169The `fed` configuration attribute is used to determine which ships are allowed to federate a circle, and keep track of the ones that are currently doing so.
174+To see how that all ties together, we're going to take a look at the broker's state.
177+++ state :> broker state
178+ $: stories/(map knot story) :< conversations
181+++ story :> wire content
182+ $: grams/(list telegram) :< all messages
183+ locals/group :< local presence
184+ remotes/(map partner group) :< remote presence
185+ shape/config :< configuration
186+ mirrors/(map circle config) :< remote config
191+Stories are the primary driver behind the broker. They are the structures that are used to power circles, which we can now say are named stories hosted on ships.
193+With the configuration described above in mind, we can try and imagine the things we can do with stories. Knowing that we can subscribe them to any number of sources, they can function as central hubs for our messaging, aggregate specific kinds of data feeds, or simply accept whatever messages get sent to it like a regular old chatroom.
129### General use
195### General use
131Upon initial startup, the broker creates a default story, a mailbox named `porch`. (This name is different for stars and galaxies.) This mailbox is the primary target for anyone and anything that wants to reach its owner. Applications can use it to send notifications and other information to the user (the broker itself does this as well), and users can use it to direct-message others.
197Upon initial startup, the broker creates a default story, a mailbox named `porch`. (This name is different for stars and galaxies.) This mailbox is the primary target for anyone and anything that wants to reach its owner. Applications can use it to send notifications and other information to the user (the broker itself does this as well), and users can use it to direct-message others.
@@ -191,7 +257,7 @@ These are used to inform the application (and thus the user) of any unexpected s
191To receive data from a broker, applications will have to subscribe to it. A `%peer` move for doing so looks something like this:
257To receive data from a broker, applications will have to subscribe to it. A `%peer` move for doing so looks something like this:
194−:* ost.hid :< bone
260+:* ost.bol :< bone
195 %peer :< move type
261 %peer :< move type
196 /story/[some-story] :< diff path
262 /story/[some-story] :< diff path
197 [our %broker] :< peer target
263 [our %broker] :< peer target
@@ -217,7 +283,6 @@ These changes get sent as `lowdown`s.
217 {$glyph (jug char (set partner))} :< glyph bindings
283 {$glyph (jug char (set partner))} :< glyph bindings
218 {$names (map ship (unit human))} :< nicknames
284 {$names (map ship (unit human))} :< nicknames
220−++ crowd {loc/group rem/(map partner group)} :< our & srcs presences
223Aside from the initial lowdown that is sent when a subscription starts, these contains exclusively the changes as they occur. Configurations are wrapped in `unit`s so that deletion can be indicated. This also means that simple applications don't necessarily need to keep state. If all they want to do is show events as they happen, then the broker subscription will provide all they need.
288Aside from the initial lowdown that is sent when a subscription starts, these contains exclusively the changes as they occur. Configurations are wrapped in `unit`s so that deletion can be indicated. This also means that simple applications don't necessarily need to keep state. If all they want to do is show events as they happen, then the broker subscription will provide all they need.
@@ -242,8 +307,6 @@ To request changes to a story, brokers can send `command`s.
243 {$relief nom/knot who/(set ship)} :< federation ended
308 {$relief nom/knot who/(set ship)} :< federation ended
245−++ lobby {loc/config rem/(map circle config)} :< our & srcs configs
246−++ crowd {loc/group rem/(map partner group)} :< our & srcs presences
249A `%review` command contains messages. These are sent to foreign brokers to be published to one of their stories.
312A `%review` command contains messages. These are sent to foreign brokers to be published to one of their stories.
@@ -252,7 +315,20 @@ A `%review` command contains messages. These are sent to foreign brokers to be p
253### Subscriptions
316### Subscriptions
255−Brokers subscribe to other brokers whenever a source gets added to a story's configuration. (Likewise, they unsubscribe when a source is removed.) Subscribers, also knows as story "followers", get sent `report`s whenever changes occur.
318+Brokers subscribe to other brokers whenever a source gets added to a story's configuration. (Likewise, they unsubscribe when a source is removed.) Here's the peer move we send when we start a subscription:
323+ /friend/show/[our-circle]/[host]/[their-circle] :< diff path
324+ [host %broker] :< peer target
325+ /[their-circle]/[start]/[end] :< peer path
329+Again, the diff path is just what our broker uses to identify what subscription a received message originates from. The peer path is more interesting. Of course it specifies the name of their story we want to subscribe our story to, but also a "start" and "end". These can be used to specify how long we want our subscription to last. Maybe we want messages starting from last week, up to the end of the month. We may also specify message numbers instead of dates, if we're so inclined.
331+For as long as their subscriptions last, subscribers get sent `report`s whenever changes occur.
258++ report :> update
334++ report :> update
@@ -260,8 +336,6 @@ Brokers subscribe to other brokers whenever a source gets added to a story's con
260 {$crowd reg/crowd} :< presence
336 {$crowd reg/crowd} :< presence
261 {$grams num/@ud gaz/(list telegram)} :< thoughts
337 {$grams num/@ud gaz/(list telegram)} :< thoughts
263−++ lobby {loc/config rem/(map circle config)} :< our & srcs configs
264−++ crowd {loc/group rem/(map partner group)} :< our & srcs presences
267There are three different kinds of reports, for configurations, presences, and messages.
341There are three different kinds of reports, for configurations, presences, and messages.
@@ -325,7 +399,7 @@ Once a subscription has been established, the broker will receive reports. `%lob
326### Messaging
400### Messaging
328−To send messages, the user sends a `%convey` or `%phrase` action, resulting in a `%review` command being sent to the involved partners. Receiving a `%review` command causes its messages to be added to the story, which sends a `%grams` report to followers and a `%grams` lowdown to all reader applications.
402+To send messages, the user sends a `%convey` or `%phrase` action, resulting in a `%review` command being sent to the involved partners. Receiving a `%review` command causes its messages to be added to the story, which sends a `%grams` report to subscribers and a `%grams` lowdown to all reader applications.
330
404
@@ -362,6 +436,10 @@ To distribute commands and reports to all that are affected by it, the broker ta
363While not a practical problem (yet, things work fine at the current scale of use), it does underline the need for a more structured approach to the broker's subscription model. In an ideal world, every interested node gets notified of a set of changes exactly once.
437While not a practical problem (yet, things work fine at the current scale of use), it does underline the need for a more structured approach to the broker's subscription model. In an ideal world, every interested node gets notified of a set of changes exactly once.
439+That still becomes unwieldy for circles that have thousands of subscribers, since a single node would still be sending thousands of network events. An obvious potential solution is to apply Urbit's hierarchical structure to this, and send messages to stars for them to send to their children.
441+While on the topic of subscriptions, the current implementation treats them as semantic data, explicitly storing them in application state. It would be desirable to push this to the background again.
365### Features and functionality
443### Features and functionality
367To turn talk into a [sustainable social platform](https://urbit.org/fora/posts/~2017.4.26..18.00.25..b93c~/), it needs a number of things.
445To turn talk into a [sustainable social platform](https://urbit.org/fora/posts/~2017.4.26..18.00.25..b93c~/), it needs a number of things.