on master help sign in
First draft of revised talk architecture doc.
e9b2823Fang
docs/talk/architecture.md+125−262
@@ -1,346 +1,209 @@
1# Talk architecture
1# Talk architecture
2
2
3−This document is complemented by talk's source code, but doesn't require it. Definitions of data structures will be provided where useful. Descriptions of processes will be accompanied by diagrams describing this flow in code arms. For an overview of the different cores and their most important arms, see the end of this document.
3+This document is complemented by talk's source code, but doesn't require it. Definitions of data structures will be provided where useful.
4
4
5>**Current implementation** remarks mention where the [current implementation](https://github.com/Fang-/arvo/tree/talk-split) differs from what is described, and why changing that to match this document would be desirable.
5>**Current implementation** remarks mention where the [current implementation](https://github.com/Fang-/arvo/tree/talk-split) differs from what is described, and why changing that to match this document would be desirable.
6
6
7
7
8## Overview
8## Overview
9
9
10−At the time of writing, talk is Urbit's biggest user-facing application, and fresh out of a big restructuring. Previously, talk was a single application. Now, it has been split into two, to improve maintainability and extensibility.
10+At the time of writing, talk is Urbit's biggest user-facing application, and fresh out of a big restructuring. Previously, talk the platform and talk the application were one and the same. Now, they have been split to help clarify their distinction, provide an example of how the platform can be used, and to aid in maintaining and extending it.
11
11
12−Official Urbit terminology calls these the `guardian` (like a unix daemon) and `agent`. You may also view them as your personal server and client. To be more semantically appropriate for talk, we're calling them `broker` and `reader` respectively.
12+Official Urbit terminology calls the platform a `guardian` (like a unix daemon) and the application an `agent`. You may also view them as your personal server and client. To be more semantically appropriate for talk, we're calling them `broker` and `reader` respectively.
13
13
14−Everyone (per identity) has a single talk broker that does all the heavy lifting. Connected to it can be any number of readers, whose task it is to present the user with an interface for interacting with talk. Brokers and readers will only interact with each other if they are in the same team. That is, if they belong to the same identity, with moons counting as their planets.
14+>It may be desirable to separate the platform from the "talk" name entirely, to decouple it from the client implementation we know and love. Also, "reader" is starting to feel more and more incorrect.
15
15
16−### Partners
16+Everyone (per identity) has a single broker that does all the heavy lifting. This broker can be used by any number of readers to help them realize their messaging functionality. Examples include the talk application we're all familiar with and the brand new Twitter clone [feed](http://urbit.org/fora/posts/~2017.4.12..21.14.00..fe17~/).
17
17
18−```
18+Since reader implementations will vary widely, we'll be looking exclusively at the functionality the broker provides and how that can be used. In order to do so, we first look at the core concepts that govern talk's architecture, then delve continuously deeper by examining the interfaces the broker offers, seeing how communication between brokers happens, and finally looking at how it's all implemented.
19−++ partner (each station passport) :: interlocutor
20−```
21−
22−A `partner` is either a `station` (a "chat room" hosted on a ship, represented as `~ship-name/station-name`) or a `passport`, which represents a non-station target. An example of a passport is a Twitter handle. Since passports aren't yet fully implemented for any kind of usage, we'll ignore them for now.
23−
24−
25−## Broker
26−
27−The broker is responsible for all primary talk functionality, everything that happens "under the hood". It can send, receive and store messages, keeps track of presence lists and configurations, and manages the stations you've created.
28
19
29−> **Note**: You may see `lowdown`s as output in some of the arm flows. These are sent to readers. Their purpose is described later in this document.
20+## Core concepts
30
21
31−### General messaging flow (broker to broker)
22+Before we dive into interfacing with the broker, we need to understand how it sees the world. Let's pull up part of the structure for the broker's state, and some structures that relate to it.
32
23
33```
24```
34−++ command :: effect on party
25+++ house :: broker state
35− $% {$publish (list thought)} :: originate
26+ $: stories/(map knot story) :: conversations
36− {$review (list thought)} :: deliver
27+ readers/(map bone (set knot)) :: our message readers
37− :: ... ::
38− == ::
39−++ thought (trel serial audience statement) :: which whom what
40−++ report :: talk update
41− $% {$grams (pair @ud (list telegram))} :: beginning thoughts
42 :: ... ::
28 :: ... ::
43 == ::
29 == ::
44−```
45−
46−Posting a message is done by poking your own broker with a `%publish` `command`, which prompts the broker to verifies that command originated from someone in its `team` (its own ship, or one of its moons), and to actually publish it.
47−
48−Having received and verified the command to publish a message, your broker sends a `%review` `command` to the brokers of all members of the `audience`. That is, all ships of the stations in the audience. (Again, messaging to other kinds of partners isn't implemented yet.)
49−
50−Upon getting poked with a command to review a message, a broker seeks out the appropriate story and (after checking if the sender has write permissions) adds the message to the story. Doing this causes a `%grams` `report` to be sent to everyone that has subscribed to that story.
51−
52−When a broker gets a diff ("subscription poke") with a `%grams` report, it adds all messages in it to the appropriate story.
53−
54−![messaging implementation flow](./diagrams/talk_flow-messaging.png "messaging implementation flow")
55−
56−```
57−++ diff-talk-report ::
58− ++ ra-diff-talk-report :: sends report to its appropriate story.
59− ++ pa-diff-talk-report :: processes a report for a story.
60− ++ pa-lesson :: learns a list of messages.
61−++ poke-talk-command ::
62− ++ ra-apply :: applies a talk command.
63− ++ ra-think :: consumes a list of messages.
64− ++ ra-consume :: conducts a message to each partner in audience.
65− ++ ra-conduct :: records a message or send it.
66− ++ ra-transmit :: sends a message to a partner.
67− ++ ra-record :: stores a message in a story.
68− ++ pa-learn :: either adds or modifies a message.
69− ++ pa-append :: adds a message to the story.
70− ++ pa-revise :: modifies a message in the story.
71− ++ pa-refresh :: sends messages to all interested subscribers.
72−```
73−
74−### Stories (broker to broker)
75−
76−```
77++ story :: wire content
30++ story :: wire content
78− $: count/@ud :: (lent grams)
79− grams/(list telegram) :: all history
31+ $: grams/(list telegram) :: all messages
80 locals/atlas :: local presence
32 locals/atlas :: local presence
81− sequence/(map partner @ud) :: partners heard
33+ remotes/(map partner atlas) :: remote presence
82 shape/config :: configuration
34 shape/config :: configuration
83− known/(map serial @ud) :: messages heard
35+ mirrors/(map station config) :: remote config
84 followers/(map bone river) :: subscribers
36 followers/(map bone river) :: subscribers
85− == ::
86−```
87−
88−A `story` is the structure in which a station's full state is stored. Its the canonical source for its messages, configuration, presence and subscribers.
89−
90−#### Telling stories
91−
92−```
93−++ house :: broker state
94− $: stories/(map knot story) :: conversations
95 :: ... ::
37 :: ... ::
96 == ::
38 == ::
97−++ command :: effect on party
39+++ config :: party configuration
98− $% {$design (pair knot (unit config))} :: configure+destroy
40+ $: sources/(set partner) :: pulls from
99 :: ... ::
41 :: ... ::
100 == ::
42 == ::
101```
43```
102
44
103−To create a station, you send a `%design` command to your broker, containing the name of the station and its initial configuration. After verifying you are in the broker's team, it sets the configuration for a story with the specified name, creating that story if it doesn't yet exist.
45+### Stories
46+
47+A `story` is the structure in which a local `station`'s full state is stored. All its messages and configuration, as well as those of other stations.
104
48
105−When people subscribe to a station it sends a `%peer` to the station's ship's broker. In the subscription path, the name of the station is specified, as well as the range of the subscription. For example, a subscriber may only be interested in messages up to a month from now.
49+Why those of other stations? Observe that a station's configuration contains `sources`. These are the station's subscriptions, from which is receives messages and metadata like presences and configurations. When you subscribe to a foreign station, you're not subscribing your identity to it. Rather, you're telling whatever story your reader is using to subscribe to it.
106
50
107−>**Current implementation** always gets you messages from up to a day old, and all messages after that. Giving the user the option to specify the range of the subscription should prove useful to future applications of the talk system. The same can be said for making a range enforceable on a per-station basis.
51+Because subscriptions are essentially story-to-story instead of broker-to-story, it is easy for different reader implementations to isolate their functionality from each other. For example, cli-talk uses the `%mailbox` story for subscriptions and message management, and feed uses a `%feed` story for doing the same. Neither have to deal with each other's data, but they can still choose to subscribe to it if they want to (and know the name of the story the other application uses).
108
52
109−When a broker gets peered, it checks the source for read permissions on the specified story before adding it to the story's list of subscribers. When this happens, the broker sends a bunch of different reports to the new subscriber to bring them up to speed on configuration, presences and messages of the station.
53+>**Current implementation** of "cli-talk" (and web-talk too) instantiates with default mailbox and journal stations. This is likely not desirable. Applications should specify their own stories. Using defaults will just end up with them becoming bloated streams of unfiltered noise.
110
54
111−![subscribe implementation flow](./diagrams/talk_flow-subscribe.png "subscribe implementation flow")
55+### `partner` or `station`?
112
56
113```
57```
114−++ poke-talk-command ::
58+++ partner (each station passport) :: interlocutor
115− ++ ra-apply :: applies a talk command.
59+++ station (pair ship knot) :: domestic flow
116− ++ ra-config :: (re)configures a story.
117− ++ pa-reform :: changes the configuration of the story.
118−++ peer ::
119− ++ ra-subscribe :: adds a subscriber to a story, sends them info.
120− ++ pa-report-cabal :: sends subscribers a report with configuration.
121− ++ pa-notify :: changes a ship's presence in the story.
122− ++ pa-report-group :: sends subscribers a report with presences.
123− ++ pa-first-grams :: determines the backlog of messages to send.
124− ++ pa-start :: sends a range of messages.
125```
60```
126
61
127−#### Hearing stories
62+A `partner` is either a `station` (a "chat room" hosted on a ship's broker, represented as `~ship-name/station-name`) or a `passport`, which represents a non-station target. An example of a passport is a Twitter handle. Since passports aren't yet fully implemented for any kind of usage, we'll ignore them.
63+
64+## Interfaces for readers
65+
66+Now that we know how a broker keeps track of things, we should have an easier time understanding the different interfaces it offers for user/reader actions.
67+
68+>The interfaces presented here are suggestions, backed by functional implementations. Adding a new interface for existing functionality is trivial. These may be changed or added to as the use cases for talk (the platform) evolve.
128
69
129```
70```
130−++ house :: broker state
71+++ action :: user action
131− $: remotes/(map partner atlas) :: remote presence
72+ $% :: station configuration ::
132− mirrors/(map station config) :: remote config
73+ {$create (trel knot cord posture)} :: create station
133− :: ... ::
74+ {$depict (pair knot cord)} :: change description
75+ {$adjust (pair knot posture)} :: change posture
76+ {$delete (pair knot (unit @t))} :: delete + announce
77+ {$permit (trel knot ? (set ship))} :: invite/banish
78+ {$source (trel knot ? (set partner))} :: un/sub p to/from r
79+ :: messaging ::
80+ {$convey (list thought)} :: post exact
81+ {$phrase (pair (set partner) (list speech))} :: post easy
82+ :: personal metadata ::
83+ {$status (pair (set partner) status)} :: our status update
84+ :: changing shared ui ::
85+ {$human (pair ship human)} :: new identity
86+ {$glyph (pair char (set partner))} :: bind a glyph
134 == ::
87 == ::
135−++ report :: talk update
88+++ reaction :: user information
136− $% {$cabal config} :: config neighborhood
89+ $: kind/?($info $fail) :: result
137− {$group atlas} :: presence
90+ what/@t :: explain
138− :: ... ::
91+ why/(unit action) :: cause
139 == ::
92 == ::
140```
93```
141
94
142−When subscribing, no stories are created. A story is the one true canonical source of a station, there can only be one. A subscriber keeps a station's data and other
95+>**Current implementation** doesn't include *all* of the above actions yet, but adding them in is trivial since the functionality is already there.
143
96
144−Every instance of a talk broker comes with a default mailbox. For planets, this is their `%porch`. It can be written to by everyone, but only read by its owner. The mailbox is used to store *all* incoming messages from *all* subscriptions. This means that if you are subscribed to a station you host, messages will be stored in both the station's story *and* your mailbox. (This is not a problem, and actually a good thing. It provides clear separation between subscriptions and stations themselves and makes it simple to not be subscribed to your own stations.)
97+>**Current implementation** uses a `(set knot)` for `%status` commands, which limits us to updating our status on local stations only.
145
98
146−>**Current implementation** still draws this line a bit unclearly, because of how the mailbox itself is also a regular story, and the storage of relevant data being as described below. The effort for changing this is not worth the small gains in clarity, however, and may even end up increasing clutter. (You'd almost think a broker-host and broker-guest would be appropriate, but likely not.)
99+Most of these `action`s should (hopefully) be fairly self-explanatory. Loobs `?` are used to differentiate between "add" and "delete" actions, `knot`s are used to specify local stations, `partner`s are targets of actions and...
100+"shared ui"? Yes, some UI state gets stored in the broker. Currently these are nicknames and glyph bindings. Those get kept up to date across readers to ensure the user can easily identify other users and stations in a familiar manner, regardless of what reader they're using. (Of course, readers can choose to ignore this if they want to.)
147
101
148−Presence lists and configurations of stations you've joined are stored in maps in the broker's general state. These are updated whenever a `%cabal` or `%group` `report` is received. Such updates are sent to all subscribers of a story whenever changes occur.
102+To give the broker a way to communicate warnings and errors to the readers, it can send them a `reaction`. This way, even though the broker itself isn't user-facing, it can still inform the user of potential failure of their action, or other warnings.
149
103
150−>**Current implementation** doesn't store presence and config of local stations in the maps, instead keeping this strictly in the stories themselves. Also adding local story presences and configs to the global maps would help simplify communication with readers and readers themselves. (See also "Reader" below.)
104+By using these interfaces exclusively, a reader as functional as our current talk can be implemented. Most of the reader code will go towards UI-related tasks, since the broker generically implements all messaging functionality.
151
105
152−>**Current implementation** includes remote presences and configs in `%cabal` and `%group` reports. Leaving those out should not hinder functionality. "But federation?" No party in a federation scenario should care where presence/config originated.
106+### Reader subscriptions
153
107
154−![subscription reports implementation flow](./diagrams/talk_flow-reports.png "subscription reports implementation flow")
108+Those interfaces alone can't provide the reader with any data, however. It still needs to subscribe itself to the stories it's interested in. Let's take a look at a `%peer` move that does just that!
155
109
156```
110```
157−++ diff-talk-report ::
111+:* ost.hid :: bone
158− ++ ra-diff-talk-report :: applies a report to the story it's intended for.
112+ %peer :: move type
159− ++ pa-diff-talk-report :: processes a report.
113+ /story/some-story :: diff path
160− ++ pa-cabal :: stores remote configuration.
114+ [our %broker] :: peer target
161− ++ pa-report-cabal :: sends subscribers a report with configuration.
115+ /reader/some-story :: peer path
162− ++ pa-remind :: stores remote presence.
116+==
163− ++ pa-report-group :: sends subscribers a report with presences.
164```
117```
165
118
119+Fairly standard. We should mention the paths being used here, however.
120+The diff path is simply something for the reader to use to identify what subscription a diff came from. That is, what story the diff relates to. Not all readers will need this (cli-talk doesn't!), but that's entirely up to the implementation.
121+The peer path informs the broker as to what kind of subscription this is. We always start with `/reader` to identify this as a reader subscription. If we leave the path at that, we subscribe to changes to the shared UI state. If we append a story name to it, we subscribe to changes to that story.
166
122
167−## Reader
123+Over a subscription we can either get sent a `reaction` as described above, or a `lowdown`. Let's see what that looks like.
168−
169−A reader is an application used for interfacing with a talk broker. As such, it communicates solely with the broker of the identity it is associated with. No other brokers. No other readers. That is all left up to the broker itself.
170−
171−>**Current implementation** still stores different tales (subsets of stories). Ideally, and taking the changes described below into account, readers should be able to get away with a single list of messages and maps for presences and configurations. Practically, you could say it's like subscribing to the broker's mailbox and the changes made to broker's presence and config lists. This means that the `man/knot` that's still passed around to cores in the reader has become mostly meaningless, and should be removed.
172−
173−### Staying informed (broker to reader)
174
124
175```
125```
176−++ lowdown :: reader update
126+++ lowdown :: changed shared state
127+ $% :: story state ::
177− $% {$confs (map station (unit config))} :: changed config
128+ {$confs (unit config) (map station (unit config))}:: changed configs
178− {$precs (map partner atlas)} :: changed presence
129+ {$precs (pair atlas (map partner atlas))} :: changed presences
179− {$grams (pair knot (pair @ud (list telegram)))} :: new grams
130+ {$grams (pair @ud (list telegram))} :: new grams
180− :: ... ::
131+ :: ui state ::
132+ {$glyph (jug char (set partner))} :: new bindings
133+ {$names (map ship (unit human))} :: new identities
181 == ::
134 == ::
182```
135```
183
136
184−As silently illustrated in the flow diagrams shown earlier in this document, `lowdown`s may get sent whenever something about a story changes: new or changed messages (`%grams`), presence (`%precs`) or configuration (`%confs`). Those lowdowns contain exclusively the changes compared to what the reader knows. When it first boots up, this is all state of all the broker's subscriptions. Afterwards, this is just whatever change the broker recently recorded.
137+`lowdown`s inform a reader of changes to story or UI state. You'll see these cover most of the story state we described above.
185−
186−>**Current implementation** has many more lowdowns, which include different lowdowns for local and remote station changes. Taking the changes described above into account (ditching tales for simpler storage) would make it possible to simplify reader state and the lowdowns that update it considerably.
187
138
188−>**Current implementation** sends much more than just the change for most lowdowns, often sending the entire new state. This places the burden of calculating the differences with the reader, even though the broker had that information readily available. Sending the change alone would free the reader from the required logic, and potentially allow the reader to stop mirroring broker state altogether. Reports might also be updated to behave this way. If needed, ++scry arms could be added to the broker for (hopefully infrequent) state access.
139+It's important to note that lowdowns contain just that which has changed, just the "diff". This is why configurations are wrapped in `unit`s, so we can signify deletion. This means simple readers that don't even keep state can still inform you when someones presence has changed.
189
140
190−![subscription lowdowns implementation flow](./diagrams/talk_flow-lowdowns-subs.png "subscription lowdowns implementation flow")
141+>Sending "just the diff" is a little bit trickier for `config`s, since they're a more complex structure than "just a map of maps". For now, the best we do is send only the configs that have changed, but not the individual changes within those configs.
191−
192−```
193−++ diff-talk-lowdown
194− ++ ra-lowdown :: processes a lowdown.
195− ++ ra-low-precs :: applies changes from a presence lowdown.
196− ++ sh-low-precs :: displays changes from a presence lowdown.
197− ++ ra-low-confs :: applies changes from a configuration lowdown.
198− ++ sh-low-confs :: displays changes from a configuration lowdown.
199− ++ ra-low-grams :: applies changes from a messages lowdown.
200− ++ sh-low-grams :: displays changes from a messages lowdown.
201− ++ pa-lesson :: learns a list of messages.
202− ++ pa-learn :: either adds or modifies a message.
203− ++ pa-append :: adds a message to the story.
204− ++ pa-revise :: modifies a message in the story.
205−```
206
142
207−### Changing shared UI (reader to broker to reader)
143+## Communication between brokers
208
144
209```
145```
210−++ update :: change shared state
146+++ command :: effect on party
211− $% {$status (pair (set partner) status)} :: our status update
147+ $% {$review (list thought)} :: deliver
212− {$human (pair ship human)} :: new identity
213− {$bind (pair char (set partner))} :: bind a glyph
214− == ::
215−++ lowdown :: reader update
216− $% {$glyfs (jug char (set partner))} :: new bindings
217− {$names (map ship (unit human))} :: new identities
218− :: ... ::
219 == ::
148 == ::
220```
149```
221
150
222−Generally, readers should be free to carry their own configurations, independent from other readers. Some of the currently available UI configuration, however, is worth syncing across readers to present information to the user in a consistent way. These are glyph bindings and nicknames, and are shared across readers through the broker.
151+>**Current implementation** hasn't deprecated the `%design` and `%publish` commands yet. They should be, because they have `action` equivalents and are exclusively used by your own identity.
223−On top of that, readers are in a good position for determining a user's status, like "idle", "active" or "typing", or allow the user to set such a status themselves. This, too, needs to be done through the broker.
224−
225−When a reader wants something done, it sends an `update` to its broker, containing the change that needs to happen. The reader applies the change to its own state before sending a `lowdown` describing it to all readers. Even the reader that requested the change gets informed, so it knows for sure that its change went through.
226−In the case of a `%status` update, we follow a similar flow similar to when we receive a `%group` `report`.
227−
228−>**Current implementation** doesn't deal with status updates very well. You want to be able to set your status per partner, but it's currently set by knot, which means it only works for local stations. What's more, there's no `command` for telling a foreign station you want to change your status, so unless you own the station your change won't be propagated.
229
152
230−>**Current implementation** also considers a `status` to be both `presence` and `human`. This same `human` structure is also used in storing local nicknames, however. All this needs some work for clarity and simplicity. (What's a "true name"? Demonology?) Allowing users to set per-station handles is probably fine, but this needs to actually be implemented.
153+When you tell your broker to post a message (through the `%convey` or `%phrase` commands), it looks at the audience you want the message to be sent to. For each station, it sends a `%review` command to the host ship, asking for the message to be reviewed and accepted.
231
154
232−![UI lowdowns implementation flow](./diagrams/talk_flow-lowdowns-ui.png "UI lowdowns implementation flow")
155+When a message gets added to a story, or a presence or configuration in it gets changed, all who are subscribed to that story get notified. This is done via `report`s.
233
156
234```
157```
235−++ poke-talk-update ::
158+++ report :: talk update
236− ++ ra-update :: processes an update.
159+ $% {$cabal config} :: local config
237− ++ pa-notify :: changes a ship's presence in the story.
160+ {$group register} :: presences
238− ++ pa-report-group :: sends subscribers a report with presences.
161+ {$grams (pair @ud (list telegram))} :: thoughts
239− ++ ra-inform :: sends a lowdown with changes to all readers.
162+ == ::
240−++ diff-talk-lowdown ::
163+++ register (pair atlas (map partner atlas)) :: ping me, ping srcs
241− ++ ra-lowdown :: processes a lowdown.
242− ++ ra-low-names :: applies new or changed local identities.
243− ++ ra-low-glyfs :: applies new or changed glyph bindings.
244```
164```
245
165
246−## Implementation overview
166+>Whereas lowdowns send just the information that has changed, reports always send the entire thing. It might technically be possible to send just the diff, but this needs to be looked into further. Diffs are in a bit of a vague spot currently anyway, see below.
247−
248−We're not going to be explaining talk's usage of cores here. ~~Please see the "core shenanigans" document for that.~~
249−
250−The different cores used by the talk applications are described below. Renaming (of both cores and their arms) is going to happen.
251−
252−### `++ra` transactions (broker and reader)
253−
254−Whenever an event happens (poke, peer, diff, etc.) `++ra` core arms are usually the first to be called.
255−
256−Important arms in the broker's `++ra` include:
257−* `++ra-apply` for processing `command` pokes.
258−* `++ra-update` for processing `update` pokes.
259−* `++ra-diff-talk-report` for processing `report` diffs, delegates to `++pa-diff-talk-report`.
260−* `++ra-subscribe` for peers.
261−* `++ra-cancel` for pulls.
262−
263−Important arms in the reader's `++ra` include:
264−* `++ra-low` for processing `lowdown` diffs.
265−* `++ra-sole` for processing `sole-action` pokes.
266−
267−When it needs to apply changes to complex structures (stories, shells), it directs flow into the appropriate core.
268−
269−It modifies state and produces `command`s, `report`s and `lowdown`s.
270−
271−### `++pa` stories (broker)
272−
273−Used for modifying stories.
274−
275−>**Current implementation** still has a `++pa` core in the reader, for dealing with tales. All it does is add messages to the specified tale. When ditching tales, this functionality should just get moved into the reader's `++ra` core.
276
167
277−Important arms include:
168+Non-message reports include both local and remote presences. This is useful in federation, where subscribers are also interested in the presences on related stations.
278−* `++pa-diff-talk-report` for processing `report`s.
279−* `++pa-acquire` for subscribing a partner to the story.
280−* `++pa-abjure` for unsubscribing a partner from the story.
281
169
282−It modifies a story's state and produces `report`s and `lowdown`s.
170+>Federation is, at the time of writing, completely untested. Messages should work fine, but proper configuration and presence propagation has yet to be seen. We'll probably need a `/federate` subscription path to integrate this cleanly into the existing system. It's an entirely new can of worms though.
283
171
284−### `++sh` shell (reader)
172+>**Current implementation** also sends remote configurations in the `%cabal` report. This doesn't seem to be useful for anything. In case of federation, you would want all stations to copy each other's configuration.
285
173
286−Used for doing sole (console, cli) work. Does work on input and prints changes to the screen.
287
174
288−Important arms include:
175+## Broker implementation
289−* `++sh-sole` for applying `sole-action`s.
290−* `++sh-scad` for parsing cli input and turning it into a work item.
291−* `++sh-work` for doing a work item.
292
176
293−For constructing printable strings of structures like `partner`s and `telegram`s, it uses their rendering cores.
177+For every event (poke, peer, etc.) the broker receives, it calls an arm from the `++ra` core. The event gets processed, and if needed, the `++pa` core is invoked to make changes to stories. If anything interesting happens, an event is sent to relevant brokers and/or readers.
294
178
295−It modifies the shell's state and produces `command`s, `update`s and `sole-effect`s.
179+Below you'll find diagrams outlining the arm flows for various common events. If you want to see what they do in more detail, looking at the code might be a good idea.
296−
297−### `++sn` station rendering (reader)
298−
299−Used in both station and ship rendering. Creates tapes representing the station or ship.
300−
301−### `++ta` partner rendering (reader)
302−
303−Used in partner rendering. Creates tapes representing the partner. Makes use of `++sn`.
304−
305−### `++te` audience rendering (reader)
306−
307−Used in audience rendering. Creates tapes representing the set of partners. Makes use of `++ta`.
308−
309−### `++tr` telegram rendering (reader)
310−
311−Used in telegram rendering. Can turn telegrams into their full representations, or a single 64-character line. Makes use of `++sn` and `++te`.
312−
313−
314−@TODO maybe insert full flow diagram?
315−
316−
317−## Notes on the talk API
318−
319−The broker does "the heavy lifting", but still leaves a lot of light work up to the readers. For example, readers can instruct a broker to send messages, but they have to send an entire thought. The client should not be responsible for, say, generating a message serial when that's something the broker can handle perfectly fine.
320−
321−Similarly, I'm currently working on implementations for `;invite` and `;banish` for modifying blacklists (channels, mailboxes) and whitelists (journals, villages). I find myself doing checks on the station type and constructing an updated config within the reader, and even building an `%inv` message to send! These are all things the broker should take care of, because they will be (should be) implemented that way for all readers.
322−
323−If we want to make talk an easy platform to use for messaging applications, it needs a better API. The current command structure works, but isn't as simple as it could be. The broker also has no way of sending an informative message back to the reader if anything fails, so doing checks on the reader-side is currently mandatory if you want to be able tot ell your user what is up.
324−
325−```
326−++ action :: user action
327− $% {$create (pair knot (pair cord posture))} :: configure + destroy
328− {$permit (trel knot ? (set ship))} :: invite/banish
329− {$say (list (trel audience statement))} :: originate
330− :: probably include ++update in this. ::
331− == ::
332−++ reaction :: user information
333− $: kind/?($info $fail) :: result
334− what/@t :: explain
335− why/(unit action) :: cause
336− == ::
337−```
338
180
339−You want the broker to do as much lifting as possible, so that readers don't have to reimplement the same things over and over again. The structure above helps the reader instruct the broker in a simpler, more low-effort way, and enables the broker to pass on a potential failure cause.
181+@TODO update diagrams where necessary, put them here, walk the reader through them briefly.
340
182
341−(Readers can of course choose to not display all reactions they get. At this point I don't think it's worth implementing something like error codes for readers to display their own custom messages, but very much possibly to add that in the future if it becomes desirable.)
342
183
184+## Roadmap
343
185
344−## Miscellaneous notes
186+Things currently on the to-do list, roughly in order of priority:
187+(Cleanup, rewriting for clarity and improving inline documentation will be done once we have a shippable talk.)
345
188
346−* Federation might be implemented as a new subscription type, where multiple brokers subscribe to a station, mirror its state, and (after processing) relay commands/reports to the other federation nodes.
189+* **Architecture** / high-impact changes
190+ * Implement federation.
191+ * Should config be determined by the original station, or should "federators" be allowed to make changes as well?
192+ * Is anyone allowed to federate any station they can subscribe to? (Definitely not if the above is "allowed to make changes"!)
193+ * Move fora things out of the broker.
194+ * *Maybe* make it its own reader app?
195+ * Improve presence capabilities.
196+ * Separate presence from handles.
197+ * Command for setting our presence in a foreign station.
198+ * "Typing", "idle", etc.
199+ * Extended permissions management?
200+ * It might be nice to be able to white/blacklist entire classes
201+* **Functionality**
202+ * Flexible subscribe ranges.
203+ * As opposed to the current always-default "a day old, and everything newer forever". of ships (ie, disallow all comets.)
204+ * List (accessible) stations on ships.
205+ * Talkpolls!
206+ * Per-channel message sanitization configuration?
207+ * Allow/disallow capitals, unicode, etc.
208+ * Better glyph management.
209+ * Better line splitting.