@@ -1,29 +1,19 @@
1−# Talk
1+# Talk architecture
4−##### notes to self
3+This document is complemented by talk's source code. Definitions of data structures will be provided where useful. Descriptions of processes will be accompanied by diagrams describing this flow in code arms. [@TODO? For more detailed explanations, see the implementation overview at the end of this document.]
8−Include proposed changes:
5+***Current implementation*** *remarks mention where the [current implementation](https://github.com/Fang-/arvo/tree/talk-split) differs from what is described, and why changing that to match this document would be desirable.*
9−* mirror config and state for local stories into remotes, allowing for:
10− * less different lowdowns,
11− * no more config/locals in tales in reader,
12− * potentially no config/locals in stories, simplifying and clarifying.
13−* something to do with the mailbox story, also see above.
14−* something to allow federation?
17−This document is complemented by talk's source code. Definitions of data structures will be provided where useful. Descriptions of processes will be accompanied by diagrams describing this flow in code arms. [@TODO? For more detailed explanations, see the implementation overview at the end of this document.]
8+## Overview
19−***Current implementation*** *remarks mention where the [current implementation](https://github.com/Fang-/arvo/tree/talk-split) differs from what is described, and why changing that to match this document would be desirable.*
10+At the time of writing, talk is Urbit's biggest user-facing application, and fresh out of a big restructuring. Previously, talk was a single application. Now, it has been split into two, to improve maintainability and extensibility.
21−### Overview
12+Official Urbit terminology calls these the `guardian` (like a unix daemon) and `agent`. You may also view them as your personal server and client. To be more semantically appropriate for talk, we're calling them `broker` and `reader` respectively.
23−Talk has been split into two, because x y z.
14+Everyone (per identity) has a single talk broker that does all the heavy lifting. Connected to it can be any number of readers, whose task it is to present the user with an interface for interacting with talk. Brokers and readers will only interact with each other if they are in the same team. That is, if they belong to the same identity, with moons counting as they planets.
24−Official Urbit terminology calls these the `guardian` (like a unix daemon) and `agent`. To be more semantically appropriate for talk, we're calling them `broker` and `reader` respectively.
26−#### Partners
16+### Partners
29++ partner (each station passport) :: interlocutor
19++ partner (each station passport) :: interlocutor
@@ -32,13 +22,13 @@ Official Urbit terminology calls these the `guardian` (like a unix daemon) and `
32A `partner` is either a `station` (a "chat room" hosted on a ship, represented as `~ship-name/station-name`) or a `passport`, which represents a non-station target. An example of a passport is a Twitter handle. Since passports aren't yet fully implemented for any kind of usage, we'll ignore them for now.
22A `partner` is either a `station` (a "chat room" hosted on a ship, represented as `~ship-name/station-name`) or a `passport`, which represents a non-station target. An example of a passport is a Twitter handle. Since passports aren't yet fully implemented for any kind of usage, we'll ignore them for now.
35−### Broker
25+## Broker
37The broker is responsible for all primary talk functionality, everything that happens "under the hood". It can send, receive and store messages, keeps track of presence lists and configurations, and manages the stations you've created.
27The broker is responsible for all primary talk functionality, everything that happens "under the hood". It can send, receive and store messages, keeps track of presence lists and configurations, and manages the stations you've created.
39Note: *You may see `lowdown`s as output in some of the arm flows. These are sent to readers. Their purpose is described later in this document.*
29Note: *You may see `lowdown`s as output in some of the arm flows. These are sent to readers. Their purpose is described later in this document.*
41−#### General messaging flow (broker to broker)
31+### General messaging flow (broker to broker)
44++ command :: effect on party
34++ command :: effect on party
@@ -63,7 +53,7 @@ When a broker gets a diff ("subscription poke") with a `%grams` report, it adds
64
54
66−#### Stories (broker to broker)
56+### Stories (broker to broker)
69++ story :: wire content
59++ story :: wire content
@@ -79,7 +69,7 @@ When a broker gets a diff ("subscription poke") with a `%grams` report, it adds
80A `story` is the structure in which a station's full state is stored. Its the canonical source for its messages, configuration, presence and subscribers.
70A `story` is the structure in which a station's full state is stored. Its the canonical source for its messages, configuration, presence and subscribers.
82−##### Telling stories
72+#### Telling stories
85++ house :: broker state
75++ house :: broker state
@@ -94,14 +84,15 @@ A `story` is the structure in which a station's full state is stored. Its the ca
95To create a station, you send a `%design` command to your broker, containing the name of the station and its initial configuration. After verifying you are in the broker's team, it sets the configuration for a story with the specified name, creating that story if it doesn't yet exist.
85To create a station, you send a `%design` command to your broker, containing the name of the station and its initial configuration. After verifying you are in the broker's team, it sets the configuration for a story with the specified name, creating that story if it doesn't yet exist.
97−When people subscribe to a station it sends a `%peer` to the station's ship's broker. In the subscription path, the name of the station is specified, as well as the range of the subscription. For example, a subscriber may only be interested in messages up to a month from now.
87+When people subscribe to a station it sends a `%peer` to the station's ship's broker. In the subscription path, the name of the station is specified, as well as the range of the subscription. For example, a subscriber may only be interested in messages up to a month from now.
98***Current implementation*** *always gets you messages from up to a day old, and all messages after that. Giving the user the option to specify the range of the subscription should prove useful to future applications of the talk system. The same can be said for making a range enforceable on a per-station basis.*
89***Current implementation*** *always gets you messages from up to a day old, and all messages after that. Giving the user the option to specify the range of the subscription should prove useful to future applications of the talk system. The same can be said for making a range enforceable on a per-station basis.*
100When a broker gets peered, it checks the source for read permissions on the specified story before adding it to the story's list of subscribers. When this happens, the broker sends a bunch of different reports to the new subscriber to bring them up to speed on configuration, presences and messages of the station.
91When a broker gets peered, it checks the source for read permissions on the specified story before adding it to the story's list of subscribers. When this happens, the broker sends a bunch of different reports to the new subscriber to bring them up to speed on configuration, presences and messages of the station.
102
93
104−##### Hearing stories
95+#### Hearing stories
107++ house :: broker state
98++ house :: broker state
@@ -116,23 +107,27 @@ When a broker gets peered, it checks the source for read permissions on the spec
119−When subscribing, no stories are created. A story is the one true canonical source of a station, there can only be one. So where does a subscriber keep a station's messages and other data?
110+When subscribing, no stories are created. A story is the one true canonical source of a station, there can only be one. A subscriber keeps a station's data and other
121−Every instance of a talk broker comes with a default mailbox. For planets, this is their `%porch`. It can be written to by everyone, but only read by its owner. The mailbox is used to store *all* incoming messages from *all* subscriptions. This means that if you are subscribed to a station you host, messages will be stored in both the station's story *and* your mailbox.
112+Every instance of a talk broker comes with a default mailbox. For planets, this is their `%porch`. It can be written to by everyone, but only read by its owner. The mailbox is used to store *all* incoming messages from *all* subscriptions. This means that if you are subscribed to a station you host, messages will be stored in both the station's story *and* your mailbox. (This is not a problem, and actually a good thing. It provides clear separation between subscriptions and stations themselves and makes it simple to not be subscribed to your own stations.)
114+***Current implementation*** *still draws this line a bit unclearly, because of how the mailbox itself is also a regular story, and the storage of relevant data being as described below. The effort for changing this is not worth the small gains in clarity, however, and may even end up increasing clutter. (You'd almost think a broker-host and broker-guest would be appropriate, but likely not.)*
116+Presence lists and configurations of stations you've joined are stored in maps in the broker's general state. These are updated whenever a `%cabal` or `%group` `report` is received. Such updates are sent to all subscribers of a story whenever changes occur.
123−Presence lists and configurations of stations you've joined are stored in maps in the broker's general state. These are updated whenever a `%cabal` or `%group` `report` is received. Such updates are sent to all subscribers of a story whenever changes occur.
124***Current implementation*** *doesn't store presence and config of local stations in the maps, instead keeping this strictly in the stories themselves. Also adding local story presences and configs to the global maps would help simplify communication with readers and readers themselves. (See also "Reader" below.)*
118***Current implementation*** *doesn't store presence and config of local stations in the maps, instead keeping this strictly in the stories themselves. Also adding local story presences and configs to the global maps would help simplify communication with readers and readers themselves. (See also "Reader" below.)*
125***Current implementation*** *includes remote presences and configs in `%cabal` and `%group` reports. Leaving those out should not hinder functionality. "But federation?" No party in a federation scenario should care where presence/config originated.*
119***Current implementation*** *includes remote presences and configs in `%cabal` and `%group` reports. Leaving those out should not hinder functionality. "But federation?" No party in a federation scenario should care where presence/config originated.*
127
121
129−### Reader
124+## Reader
131A reader is an application used for interfacing with a talk broker. As such, it communicates solely with the broker of the identity it is associated with. No other brokers. No other readers. That is all left up to the broker itself.
126A reader is an application used for interfacing with a talk broker. As such, it communicates solely with the broker of the identity it is associated with. No other brokers. No other readers. That is all left up to the broker itself.
133−***Current implementation*** *still stores different tales (subsets of stories). Ideally, and taking the changes described below into account, readers should be able to get away with a single list of messages and maps for presences and configurations. Practically, you could say it's like subscribing to the broker's mailbox and the changes made to broker's presence and config lists.*
128+***Current implementation*** *still stores different tales (subsets of stories). Ideally, and taking the changes described below into account, readers should be able to get away with a single list of messages and maps for presences and configurations. Practically, you could say it's like subscribing to the broker's mailbox and the changes made to broker's presence and config lists. This means that the `man/knot` that's still passed around to cores in the reader has become mostly meaningless, and should be removed.*
135−#### Staying informed (broker to reader)
130+### Staying informed (broker to reader)
138++ lowdown :: reader update
133++ lowdown :: reader update
@@ -143,17 +138,18 @@ A reader is an application used for interfacing with a talk broker. As such, it
146−As silently illustrated in the flow diagrams shown earlier in this document, `lowdown`s may get sent whenever something about a story changes: new or changed messages (`%grams`), presence (`%precs`) or configuration (`%confs`). Those lowdowns contain exclusively the changes compared to what the reader knows. When it first boots up, this is all state of all the broker's subscriptions. Afterwards, this is just whatever change the broker recently recorded.
141+As silently illustrated in the flow diagrams shown earlier in this document, `lowdown`s may get sent whenever something about a story changes: new or changed messages (`%grams`), presence (`%precs`) or configuration (`%confs`). Those lowdowns contain exclusively the changes compared to what the reader knows. When it first boots up, this is all state of all the broker's subscriptions. Afterwards, this is just whatever change the broker recently recorded.
147***Current implementation*** *has many more lowdowns, which include different lowdowns for local and remote station changes. Taking the changes described above into account (ditching tales for simpler storage) would make it possible to simplify reader state and the lowdowns that update it considerably.*
143***Current implementation*** *has many more lowdowns, which include different lowdowns for local and remote station changes. Taking the changes described above into account (ditching tales for simpler storage) would make it possible to simplify reader state and the lowdowns that update it considerably.*
148***Current implementation*** *sends much more than just the change for most lowdowns, often sending the entire new state. This places the burden of calculating the differences with the reader, even though the broker had that information readily available. Sending the change alone would free the reader from the required logic, and potentially allow the reader to stop mirroring broker state altogether. Reports might also be updated to behave this way. If needed, ++scry arms could be added to the broker for (hopefully infrequent) state access.*
144***Current implementation*** *sends much more than just the change for most lowdowns, often sending the entire new state. This places the burden of calculating the differences with the reader, even though the broker had that information readily available. Sending the change alone would free the reader from the required logic, and potentially allow the reader to stop mirroring broker state altogether. Reports might also be updated to behave this way. If needed, ++scry arms could be added to the broker for (hopefully infrequent) state access.*
150
146
152−#### Changing shared UI (reader to broker to reader)
148+### Changing shared UI (reader to broker to reader)
155++ update :: change shared state
151++ update :: change shared state
156− $% {$status (pair (set knot) status)} :: our status update
152+ $% {$status (pair (set partner) status)} :: our status update
157 {$human (pair ship human)} :: new identity
153 {$human (pair ship human)} :: new identity
158 {$bind (pair char (set partner))} :: bind a glyph
154 {$bind (pair char (set partner))} :: bind a glyph
@@ -163,27 +159,93 @@ As silently illustrated in the flow diagrams shown earlier in this document, `lo
163 :: ... ::
159 :: ... ::
166−@TODO maybe update the above molds a bit to better fit the "send only diffs" design?
167−@TODO status per partner, not per story, right?
169−@TODO
163+Generally, readers should be free to carry their own configurations, independent from other readers. Some of the currently available UI configuration, however, is worth syncing across readers to present information to the user in a consistent way. These are glyph bindings and nicknames, and are shared across readers through the broker.
170−Sometimes when a user changes something about their UI, we want that to synchronize across all readers. This includes nicknames and glyph bindings. Also, status changes.
164+On top of that, readers are in a good position for determining a user's status, like "idle", "active" or "typing", or allow the user to set such a status themselves. This, too, needs to be done through the broker.
171−Broker receives these updates, processes them, then propagates to all readers.
165+
166+When a reader wants something done, it sends an `update` to its broker, containing the change that needs to happen. The reader applies the change to its own state before sending a `lowdown` describing it to all readers. Even the reader that requested the change gets informed, so it knows for sure that its change went through.
167+In the case of a `%status` update, we follow a similar flow similar to when we receive a `%group` `report`.
169+***Current implementation*** *doesn't deal with status updates very well. You want to be able to be able to set your status per partner, but it's currently set by knot, which means it only works for local stations. What's more, there's no `command` for telling a foreign station you want to change your status, so unless you own the station your change won't be propagated.*
170+***Current implementation*** *also considers a `status` to be both `presence` and `human`. This same `human` structure is also used in storing local nicknames, however. All this needs some work for clarity and simplicity. (What's a "true name"? Demonology?) Allowing users to set per-station handles is probably fine, but this needs to actually be implemented.*
173
172
175−### Implementation overview
175+## Implementation overview
177@TODO per core: task/subject, important arms, general flow description
177@TODO per core: task/subject, important arms, general flow description
179We're not going to be explaining talk's usage of cores here. ~~Please see the "core shenanigans" document for that.~~
179We're not going to be explaining talk's usage of cores here. ~~Please see the "core shenanigans" document for that.~~
181+The different cores are described below. Renaming (of both cores and their arms) is going to happen.
181−#### `++ra` transactions
183+### `++ra` transactions (broker and reader)
182−#### `++pa` stories/tales
184+
183−#### `++sh` shell
185+Whenever an event happens (poke, peer, diff, etc.) `++ra` core arms are usually the first to be called.
187+Important arms in the broker's `++ra` include:
188+* `++ra-apply` for processing `command` pokes.
189+* `++ra-update` for processing `update` pokes.
190+* `++ra-diff-talk-report` for processing `report` diffs, delegates to `++pa-diff-talk-report`.
191+* `++ra-subscribe` for peers.
192+* `++ra-cancel` for pulls.
194+Important arms in the reader's `++ra` include:
195+* `++ra-low` for processing `lowdown` diffs.
196+* `++ra-sole` for processing `sole-action` pokes.
198+When it needs to apply changes to complex structures (stories, shells), it directs flow into the appropriate core.
200+It modifies state and produces `command`s, `report`s and `lowdown`s.
202+### `++pa` stories (broker)
204+Used for modifying stories.
206+***Current implementation*** *still has a `++pa` core in the reader, for dealing with tales. All it does is add messages to the specified tale. When ditching tales, this functionality should just get moved into the reader's `++ra` core.*
208+Important arms include:
209+* `++pa-diff-talk-report` for processing `report`s.
210+* `++pa-acquire` for subscribing a partner to the story.
211+* `++pa-abjure` for unsubscribing a partner from the story.
213+It modifies a story's state and produces `report`s and `lowdown`s.
215+### `++sh` shell (reader)
217+Used for doing sole (console, cli) work. Does work on input and prints changes to the screen.
219+Important arms include:
220+* `++sh-sole` for applying `sole-action`s.
221+* `++sh-scad` for parsing cli input and turning it into a work item.
222+* `++sh-work` for doing a work item.
224+For constructing printable strings of structures like `partner`s and `telegram`s, it uses their rendering cores.
226+It modifies the shell's state and produces `command`s, `update`s and `sole-effect`s.
184−#### `++sn` station rendering
228+### `++sn` station rendering (reader)
230+Used in both station and ship rendering. Creates tapes representing the station or ship.
185−#### `++ta` partner rendering
232+### `++ta` partner rendering (reader)
234+Used in partner rendering. Creates tapes representing the partner. Makes use of `++sn`.
186−#### `++te` audience rendering
236+### `++te` audience rendering (reader)
238+Used in audience rendering. Creates tapes representing the set of partners. Makes use of `++ta`.
187−#### `++tr` telegram rendering
240+### `++tr` telegram rendering (reader)
242+Used in telegram rendering. Can turn telegrams into their full representations, or a single 64-character line. Makes use of `++sn` and `++te`.
245+@TODO maybe insert full flow diagram?
189−@TODO insert/link to arm flow diagrams
250+* Federation might be implemented as a new subscription type, where multiple brokers subscribe to a station, mirror its state, and (after processing) relay commands/reports to the other federation nodes.