@@ -1,29 +1,19 @@
1−# Talk
2−## Architecture
1+# Talk architecture
32
4−##### notes to self
5−
6−Justify everything!
3+This document is complemented by talk's source code. Definitions of data structures will be provided where useful. Descriptions of processes will be accompanied by diagrams describing this flow in code arms. [@TODO? For more detailed explanations, see the implementation overview at the end of this document.]
74
8−Include proposed changes:
9−* mirror config and state for local stories into remotes, allowing for:
10− * less different lowdowns,
11− * no more config/locals in tales in reader,
12− * potentially no config/locals in stories, simplifying and clarifying.
13−* something to do with the mailbox story, also see above.
14−* something to allow federation?
5+***Current implementation*** *remarks mention where the [current implementation](https://github.com/Fang-/arvo/tree/talk-split) differs from what is described, and why changing that to match this document would be desirable.*
156
167
17−This document is complemented by talk's source code. Definitions of data structures will be provided where useful. Descriptions of processes will be accompanied by diagrams describing this flow in code arms. [@TODO? For more detailed explanations, see the implementation overview at the end of this document.]
8+## Overview
189
19−***Current implementation*** *remarks mention where the [current implementation](https://github.com/Fang-/arvo/tree/talk-split) differs from what is described, and why changing that to match this document would be desirable.*
10+At the time of writing, talk is Urbit's biggest user-facing application, and fresh out of a big restructuring. Previously, talk was a single application. Now, it has been split into two, to improve maintainability and extensibility.
2011
21−### Overview
12+Official Urbit terminology calls these the `guardian` (like a unix daemon) and `agent`. You may also view them as your personal server and client. To be more semantically appropriate for talk, we're calling them `broker` and `reader` respectively.
2213
23−Talk has been split into two, because x y z.
24−Official Urbit terminology calls these the `guardian` (like a unix daemon) and `agent`. To be more semantically appropriate for talk, we're calling them `broker` and `reader` respectively.
14+Everyone (per identity) has a single talk broker that does all the heavy lifting. Connected to it can be any number of readers, whose task it is to present the user with an interface for interacting with talk. Brokers and readers will only interact with each other if they are in the same team. That is, if they belong to the same identity, with moons counting as they planets.
2515
26−#### Partners
16+### Partners
2717
2818```
2919++ partner (each station passport) :: interlocutor
@@ -32,13 +22,13 @@ Official Urbit terminology calls these the `guardian` (like a unix daemon) and `
3222A `partner` is either a `station` (a "chat room" hosted on a ship, represented as `~ship-name/station-name`) or a `passport`, which represents a non-station target. An example of a passport is a Twitter handle. Since passports aren't yet fully implemented for any kind of usage, we'll ignore them for now.
3323
3424
35−### Broker
25+## Broker
3626
3727The broker is responsible for all primary talk functionality, everything that happens "under the hood". It can send, receive and store messages, keeps track of presence lists and configurations, and manages the stations you've created.
3828
3929Note: *You may see `lowdown`s as output in some of the arm flows. These are sent to readers. Their purpose is described later in this document.*
4030
41−#### General messaging flow (broker to broker)
31+### General messaging flow (broker to broker)
4232
4333```
4434++ command :: effect on party
@@ -63,7 +53,7 @@ When a broker gets a diff ("subscription poke") with a `%grams` report, it adds
6353
6454
6555
66−#### Stories (broker to broker)
56+### Stories (broker to broker)
6757
6858```
6959++ story :: wire content
@@ -79,7 +69,7 @@ When a broker gets a diff ("subscription poke") with a `%grams` report, it adds
7969
8070A `story` is the structure in which a station's full state is stored. Its the canonical source for its messages, configuration, presence and subscribers.
8171
82−##### Telling stories
72+#### Telling stories
8373
8474```
8575++ house :: broker state
@@ -94,14 +84,15 @@ A `story` is the structure in which a station's full state is stored. Its the ca
9484
9585To create a station, you send a `%design` command to your broker, containing the name of the station and its initial configuration. After verifying you are in the broker's team, it sets the configuration for a story with the specified name, creating that story if it doesn't yet exist.
9686
97−When people subscribe to a station it sends a `%peer` to the station's ship's broker. In the subscription path, the name of the station is specified, as well as the range of the subscription. For example, a subscriber may only be interested in messages up to a month from now.
87+When people subscribe to a station it sends a `%peer` to the station's ship's broker. In the subscription path, the name of the station is specified, as well as the range of the subscription. For example, a subscriber may only be interested in messages up to a month from now.
88+
9889***Current implementation*** *always gets you messages from up to a day old, and all messages after that. Giving the user the option to specify the range of the subscription should prove useful to future applications of the talk system. The same can be said for making a range enforceable on a per-station basis.*
9990
10091When a broker gets peered, it checks the source for read permissions on the specified story before adding it to the story's list of subscribers. When this happens, the broker sends a bunch of different reports to the new subscriber to bring them up to speed on configuration, presences and messages of the station.
10192
10293
10394
104−##### Hearing stories
95+#### Hearing stories
10596
10697```
10798++ house :: broker state
@@ -116,23 +107,27 @@ When a broker gets peered, it checks the source for read permissions on the spec
116107 == ::
117108```
118109
119−When subscribing, no stories are created. A story is the one true canonical source of a station, there can only be one. So where does a subscriber keep a station's messages and other data?
110+When subscribing, no stories are created. A story is the one true canonical source of a station, there can only be one. A subscriber keeps a station's data and other
120111
121−Every instance of a talk broker comes with a default mailbox. For planets, this is their `%porch`. It can be written to by everyone, but only read by its owner. The mailbox is used to store *all* incoming messages from *all* subscriptions. This means that if you are subscribed to a station you host, messages will be stored in both the station's story *and* your mailbox.
112+Every instance of a talk broker comes with a default mailbox. For planets, this is their `%porch`. It can be written to by everyone, but only read by its owner. The mailbox is used to store *all* incoming messages from *all* subscriptions. This means that if you are subscribed to a station you host, messages will be stored in both the station's story *and* your mailbox. (This is not a problem, and actually a good thing. It provides clear separation between subscriptions and stations themselves and makes it simple to not be subscribed to your own stations.)
113+
114+***Current implementation*** *still draws this line a bit unclearly, because of how the mailbox itself is also a regular story, and the storage of relevant data being as described below. The effort for changing this is not worth the small gains in clarity, however, and may even end up increasing clutter. (You'd almost think a broker-host and broker-guest would be appropriate, but likely not.)*
115+
116+Presence lists and configurations of stations you've joined are stored in maps in the broker's general state. These are updated whenever a `%cabal` or `%group` `report` is received. Such updates are sent to all subscribers of a story whenever changes occur.
122117
123−Presence lists and configurations of stations you've joined are stored in maps in the broker's general state. These are updated whenever a `%cabal` or `%group` `report` is received. Such updates are sent to all subscribers of a story whenever changes occur.
124118***Current implementation*** *doesn't store presence and config of local stations in the maps, instead keeping this strictly in the stories themselves. Also adding local story presences and configs to the global maps would help simplify communication with readers and readers themselves. (See also "Reader" below.)*
125119***Current implementation*** *includes remote presences and configs in `%cabal` and `%group` reports. Leaving those out should not hinder functionality. "But federation?" No party in a federation scenario should care where presence/config originated.*
126120
127121
128122
129−### Reader
123+
124+## Reader
130125
131126A reader is an application used for interfacing with a talk broker. As such, it communicates solely with the broker of the identity it is associated with. No other brokers. No other readers. That is all left up to the broker itself.
132127
133−***Current implementation*** *still stores different tales (subsets of stories). Ideally, and taking the changes described below into account, readers should be able to get away with a single list of messages and maps for presences and configurations. Practically, you could say it's like subscribing to the broker's mailbox and the changes made to broker's presence and config lists.*
128+***Current implementation*** *still stores different tales (subsets of stories). Ideally, and taking the changes described below into account, readers should be able to get away with a single list of messages and maps for presences and configurations. Practically, you could say it's like subscribing to the broker's mailbox and the changes made to broker's presence and config lists. This means that the `man/knot` that's still passed around to cores in the reader has become mostly meaningless, and should be removed.*
134129
135−#### Staying informed (broker to reader)
130+### Staying informed (broker to reader)
136131
137132```
138133++ lowdown :: reader update
@@ -143,17 +138,18 @@ A reader is an application used for interfacing with a talk broker. As such, it
143138 == ::
144139```
145140
146−As silently illustrated in the flow diagrams shown earlier in this document, `lowdown`s may get sent whenever something about a story changes: new or changed messages (`%grams`), presence (`%precs`) or configuration (`%confs`). Those lowdowns contain exclusively the changes compared to what the reader knows. When it first boots up, this is all state of all the broker's subscriptions. Afterwards, this is just whatever change the broker recently recorded.
141+As silently illustrated in the flow diagrams shown earlier in this document, `lowdown`s may get sent whenever something about a story changes: new or changed messages (`%grams`), presence (`%precs`) or configuration (`%confs`). Those lowdowns contain exclusively the changes compared to what the reader knows. When it first boots up, this is all state of all the broker's subscriptions. Afterwards, this is just whatever change the broker recently recorded.
142+
147143***Current implementation*** *has many more lowdowns, which include different lowdowns for local and remote station changes. Taking the changes described above into account (ditching tales for simpler storage) would make it possible to simplify reader state and the lowdowns that update it considerably.*
148144***Current implementation*** *sends much more than just the change for most lowdowns, often sending the entire new state. This places the burden of calculating the differences with the reader, even though the broker had that information readily available. Sending the change alone would free the reader from the required logic, and potentially allow the reader to stop mirroring broker state altogether. Reports might also be updated to behave this way. If needed, ++scry arms could be added to the broker for (hopefully infrequent) state access.*
149145
150146
151147
152−#### Changing shared UI (reader to broker to reader)
148+### Changing shared UI (reader to broker to reader)
153149
154150```
155151++ update :: change shared state
156− $% {$status (pair (set knot) status)} :: our status update
152+ $% {$status (pair (set partner) status)} :: our status update
157153 {$human (pair ship human)} :: new identity
158154 {$bind (pair char (set partner))} :: bind a glyph
159155 == ::
@@ -163,27 +159,93 @@ As silently illustrated in the flow diagrams shown earlier in this document, `lo
163159 :: ... ::
164160 == ::
165161```
166−@TODO maybe update the above molds a bit to better fit the "send only diffs" design?
167−@TODO status per partner, not per story, right?
168162
169−@TODO
170−Sometimes when a user changes something about their UI, we want that to synchronize across all readers. This includes nicknames and glyph bindings. Also, status changes.
171−Broker receives these updates, processes them, then propagates to all readers.
163+Generally, readers should be free to carry their own configurations, independent from other readers. Some of the currently available UI configuration, however, is worth syncing across readers to present information to the user in a consistent way. These are glyph bindings and nicknames, and are shared across readers through the broker.
164+On top of that, readers are in a good position for determining a user's status, like "idle", "active" or "typing", or allow the user to set such a status themselves. This, too, needs to be done through the broker.
165+
166+When a reader wants something done, it sends an `update` to its broker, containing the change that needs to happen. The reader applies the change to its own state before sending a `lowdown` describing it to all readers. Even the reader that requested the change gets informed, so it knows for sure that its change went through.
167+In the case of a `%status` update, we follow a similar flow similar to when we receive a `%group` `report`.
168+
169+***Current implementation*** *doesn't deal with status updates very well. You want to be able to be able to set your status per partner, but it's currently set by knot, which means it only works for local stations. What's more, there's no `command` for telling a foreign station you want to change your status, so unless you own the station your change won't be propagated.*
170+***Current implementation*** *also considers a `status` to be both `presence` and `human`. This same `human` structure is also used in storing local nicknames, however. All this needs some work for clarity and simplicity. (What's a "true name"? Demonology?) Allowing users to set per-station handles is probably fine, but this needs to actually be implemented.*
172171
173172
174173
175−### Implementation overview
174+
175+## Implementation overview
176176
177177@TODO per core: task/subject, important arms, general flow description
178178
179179We're not going to be explaining talk's usage of cores here. ~~Please see the "core shenanigans" document for that.~~
180180
181−#### `++ra` transactions
182−#### `++pa` stories/tales
183−#### `++sh` shell
184−#### `++sn` station rendering
185−#### `++ta` partner rendering
186−#### `++te` audience rendering
187−#### `++tr` telegram rendering
181+The different cores are described below. Renaming (of both cores and their arms) is going to happen.
182+
183+### `++ra` transactions (broker and reader)
184+
185+Whenever an event happens (poke, peer, diff, etc.) `++ra` core arms are usually the first to be called.
186+
187+Important arms in the broker's `++ra` include:
188+* `++ra-apply` for processing `command` pokes.
189+* `++ra-update` for processing `update` pokes.
190+* `++ra-diff-talk-report` for processing `report` diffs, delegates to `++pa-diff-talk-report`.
191+* `++ra-subscribe` for peers.
192+* `++ra-cancel` for pulls.
193+
194+Important arms in the reader's `++ra` include:
195+* `++ra-low` for processing `lowdown` diffs.
196+* `++ra-sole` for processing `sole-action` pokes.
197+
198+When it needs to apply changes to complex structures (stories, shells), it directs flow into the appropriate core.
199+
200+It modifies state and produces `command`s, `report`s and `lowdown`s.
201+
202+### `++pa` stories (broker)
203+
204+Used for modifying stories.
205+
206+***Current implementation*** *still has a `++pa` core in the reader, for dealing with tales. All it does is add messages to the specified tale. When ditching tales, this functionality should just get moved into the reader's `++ra` core.*
207+
208+Important arms include:
209+* `++pa-diff-talk-report` for processing `report`s.
210+* `++pa-acquire` for subscribing a partner to the story.
211+* `++pa-abjure` for unsubscribing a partner from the story.
212+
213+It modifies a story's state and produces `report`s and `lowdown`s.
214+
215+### `++sh` shell (reader)
216+
217+Used for doing sole (console, cli) work. Does work on input and prints changes to the screen.
218+
219+Important arms include:
220+* `++sh-sole` for applying `sole-action`s.
221+* `++sh-scad` for parsing cli input and turning it into a work item.
222+* `++sh-work` for doing a work item.
223+
224+For constructing printable strings of structures like `partner`s and `telegram`s, it uses their rendering cores.
225+
226+It modifies the shell's state and produces `command`s, `update`s and `sole-effect`s.
227+
228+### `++sn` station rendering (reader)
229+
230+Used in both station and ship rendering. Creates tapes representing the station or ship.
231+
232+### `++ta` partner rendering (reader)
233+
234+Used in partner rendering. Creates tapes representing the partner. Makes use of `++sn`.
235+
236+### `++te` audience rendering (reader)
237+
238+Used in audience rendering. Creates tapes representing the set of partners. Makes use of `++ta`.
239+
240+### `++tr` telegram rendering (reader)
241+
242+Used in telegram rendering. Can turn telegrams into their full representations, or a single 64-character line. Makes use of `++sn` and `++te`.
243+
244+
245+@TODO maybe insert full flow diagram?
246+
247+
248+## Notes
188249
189−@TODO insert/link to arm flow diagrams
250+* Federation might be implemented as a new subscription type, where multiple brokers subscribe to a station, mirror its state, and (after processing) relay commands/reports to the other federation nodes.
251+* ...