d2cfce1 · urbit.org · git.iko.soy

11

2−# Talk architecture

2+# Hall architecture

33

4−This document is complemented by talk's source code, but doesn't require it. Definitions of data structures, code snippets and diagrams will be provided where useful.

4+This document is complemented by Hall's source code, but doesn't require it. Definitions of data structures, code snippets and diagrams will be provided where useful.

5+

6+Hall's implementation is structured according to the new Gall model. Familiarity with its concepts is assumed. See [the new Gall model](gall.md).

57

68## Table of contents

79

8−* [Overview](#overview)

10+* [Introduction](#introduction)

911* [Structures & functionality](#structures--functionality)

1012 * [Partners](#partners)

1113 * [Messages](#messages)

12− * [Metadata](#metadata)

13− * [Access control](#access-control)

14− * [Federation](#federation)

14+ * [Participant metadata](#participant-metadata)

15+ * [Configurations](#configurations)

1516 * [Stories](#stories)

1617 * [General use](#general-use)

18+ * [Federation](#federation)

1719* [Interfaces for applications](#interfaces-for-applications)

1820 * [Interactions](#interactions)

19− * [Subscriptions](#subscriptions)

20−* [Communication between brokers](#communication-between-brokers)

21+ * [Queries, prizes and rumors](#queries-prizes-and-rumors)

22+* [Communication between Halls](#communication-between-halls)

2123 * [Interactions](#interactions-1)

22− * [Subscriptions](#subscriptions-1)

23−* [Broker implementation](#broker-implementation)

24− * [Subscriptions](#subscriptions-2)

25− * [Reports](#reports)

24+ * [Querying](#querying)

25+ * [Federation](#federation)

26+* [Hall implementation](#hall-implementation)

27+ * [Subscriptions](#subscriptions)

28+ * [Rumors](#rumors)

2629 * [Messaging](#messaging)

2730* [The future](#the-future)

28− * [Architectural issues](#architectural-issues)

2931 * [Features and functionality](#features-and-functionality)

3032* [Further reading](#further-reading)

3133

32−## Overview

3334

34−Talk was Urbit's first big user-facing application, and continues to enjoy a prominent role in the Urbit landscape as the primary messaging bus.

35+## Introduction

36+

37+Talk was Urbit's first big user-facing application. It continues to enjoy a prominent role in the Urbit landscape, but does so as two separate applications this time.

38+

39+The messaging parts of talk have been separated from its user interface parts. What we ended up with is a shiny new generic messaging bus, and the chat interface we all know and love. The interface retains the `talk` name. The messaging bus, which will prove useful to many more applications, will be named `Hall`.

40+

41+@TODO mention clients are sometimes referred to as "readers".

42+

43+@TODO probably rewrite intro entirely.

3544

3645To be clear, we make a distinction between "talk the platform" and "talk the chat application". The latter makes use of the former for most of its functionality, and serves as an example of things that can be built on top of the platform. (Also see the small Twitter clone, [feed](http://urbit.org/fora/posts/~2017.4.12..21.14.00..fe17~/).)

37−This document, however, focusses on the platform. In talk's case, we refer to this "server" (or `guardian`, in Urbit terminology) as the *broker*. It manages an identity's messages, subscriptions, and more.

46+This document, however, focusses on the platform. In talk's case, we refer to this "server" (or `guardian`, in Urbit terminology) as the *Hall*. It manages an identity's messages, subscriptions, and more.

3847

3948

4049## Structures & functionality

4150

4251### Partners

4352

44−One of the broker's strengths is that it can not only interface with other brokers, but also external services. We define a single structure that can be used to specify both kinds of messaging targets.

53+One of Hall's strengths is that it can not only interface with other Hall instances (Halls, if you will), but also external services. We define a single structure that can be used to specify both kinds of messaging targets.

4554

4655```

4756++ partner (each circle passport) :< message target

5160 == ::

5261```

5362

54−A `partner` is either a `circle` or a `passport`. A `circle` is a native partner, essentially a named collection of messages created by and hosted on a ship's broker, usually represented as `~ship-name/circle-name`.

63+A `partner` is either a `circle` or a `passport`. A `circle` is a native partner, essentially a named collection of messages created by and hosted on a ship's Hall, usually represented as `~ship-name/circle-name`.

5564A `passport` is a non-native partner, such as a Twitter feed. The possibilities for passports are endless... but none of them are properly implemented yet. For the rest of this document we'll assume a `partner` is always a `circle`.

5665

5766### Messages

5968When we subscribe to a circle, the primary thing we're interested in is its messages. Message data itself isn't that complicated, but a fair amount of metadata comes into play when actually sending a message. Let's work our way up, starting at the contents.

6069

6170```

62−++ speech :> narrative action

71+++ speech :> content body

6372 $% {$lin pat/? msg/cord} :< no/@ text line

6473 {$ire tos/serial sep/speech} :< in-reply-to

6574 {$url url/purf} :< parsed url

6978 == ::

7079```

7180

72−At the heart of every message lies a `speech` that describes the message contents. There's a large number of different speech types, from simple text messages to parsed URLs, Hoon expressions and more. Thought the system strives to accommodate all use cases, the `%ext` speech type provides some extensibility for applications that feel their needs aren't covered.

81+At the heart of every message lies a `speech` that describes the message body. There's a large number of different speech types, from simple text messages to parsed URLs, Hoon expressions and more. Thought the system strives to eventually accommodate all use cases, the `%ext` speech type provides extensibility for applications that feel their needs aren't covered.

7382

7483```

75−++ statement {wen/@da boq/bouquet sep/speech} :< when this

84+++ statement {wen/@da boq/bouquet sep/speech} :< when what body

7685++ bouquet (set flavor) :< complete aroma

7786++ flavor path :< content flavor

7887```

7988

80−Message contents are wrapped in a `statement` which specifies its timestamp and the `flavor` of its content, ¿¿¿ which can be used to filter out content they prefer not to see. ???

89+Message bodies are wrapped in a `statement` which specifies the timestamp and `flavor` of its content, the latter of which can in the future be used to filter out the kinds of content you personally do not wish to see.

8190

8291```

8392++ audience (map partner (pair envelope delivery)) :< destination + state

95104

96105```

97106++ telegram {aut/ship tot/thought} :< who thought

98−++ thought {uid/serial aud/audience sam/statement} :< which whom what

107+++ thought {uid/serial aud/audience sam/statement} :< which whom this

99108++ serial @uvH :< unique identifier

100109```

101110

102111The message and its destination get put into a `thought`, which also carries a universally unique identifier. This way, recipients can easily check whether they're receiving a new message or a potentially modified version of an existing one.

103112Finally, all this gets put into a `telegram` which tags it with the message's author.

104113

105−### Metadata

114+### Participant metadata

106115

107116Messages aren't the only thing a subscription gets us. We're also kept up to date with relevant metadata.

108117

111120++ group (map ship status) :< presence map

112121++ status {pec/presence man/human} :< participant

113122++ presence :> status type

114− $? $gone :< left

123+ $? $gone :< absent

115124 $idle :< idle

116125 $hear :< present

117126 $talk :< typing

125134`status` is user-set metadata that describes, well, the status of users in a circle. This encompasses their `presence`, which shows their activity, and their `human` identity, which includes their display handle.

126135For reasons we'll discover shortly, circles keep track of both their own `group` and those of the partners they're subscribed to. `crowd` encapsulates this.

127136

137+### Configurations

138+

128139```

129140++ lobby {loc/config rem/(map circle config)} :< our & srcs configs

130141++ config :> circle config

131− $: src/(set partner) :< pulls from

142+ $: src/(set partner) :< active sources

132143 cap/cord :< description

144+ fit/filter :< message rules

133145 con/control :< restrictions

134− fed/federal :< federators

146+ == ::

147+++ filter :> content filters

148+ $: cas/? :< dis/allow capitals

149+ utf/? :< dis/allow non-ascii

135150 == ::

136151++ control {sec/security ses/(set ship)} :< access control

137−++ federal {may/(set ship) fes/(set ship)} :< federation control

138152```

139153

140154Another part of metadata we get from circle subscriptions is configurations. Again, circles want to remember their own configuration, as well as those of their subscriptions. But why, precisely?

141155

142−The `config` structure contains `src`, a set of partners. These indicate the different sources a circle may pull content from. This allows circles to aggregate messages from multiple places. In doing so, it also receives metadata from those places, hence why we have structures for storing "remote" presences and configurations alongside local ones.

156+The `config` structure contains `src`, a set of partners. These indicate the different sources a circle is currently pulling content from. This allows circles to aggregate messages from multiple places. In doing so, it also receives metadata from those places, hence why we have structures for storing "remote" presences and configurations alongside local ones.

143157

144−Aside from a description, a configuration also contains structures to define access and federation control. They're important parts of functionality in their own right, so they get their own sections below.

158+Aside from that, `config` contains a description for the circle, which is exactly what it sounds like. It also specifies a `filter`, which are content formatting rules all messages passing through this circle are made to adhere to.

145159

146−### Access control

147−

148−A `control` structure contains both a security mode and a list of ships, which is either a white- or blacklist depending on the aforementioned mode. There are four such modes available.

160+Lastly, there's a `control` structure that contains both a security mode and a list of ships, which is either a white- or blacklist depending on the aforementioned mode. There are four such modes available.

149161

150162```

151163++ security :> security mode

161173A `journal` is publicly readable and privately writable, with a whitelist for authors.

162174A `mailbox` is readable by its owner and publicly writable, with a blacklist for blocking.

163175

164−### Federation

165−

166−One of the primary aspects of Urbit is its decentralized nature. To make full use of that, the broker can federate circles. That is, circles can be hosted on multiple ships simultaneously. They share all messages, presence and configuration, regardless of where that data or change originated.

167−This can help spread heavy load across multiple ships, makes circles easier to access, and allows for fallback in case one of the hosts goes offline.

168−

169−The `fed` configuration attribute is used to determine which ships are allowed to federate a circle, and keep track of the ones that are currently doing so.

170−

171−

172176### Stories

173177

174−To see how that all ties together, we're going to take a look at the broker's state.

178+To see how that all ties together, we're going to take a look at Hall's state.

175179

176180```

177−++ state :> broker state

181+++ state :> hall state

178182 $: stories/(map knot story) :< conversations

179183 :: ... ::

180184 == ::

188192 == ::

189193```

190194

191−Stories are the primary driver behind the broker. They are the structures that are used to power circles, which we can now say are named stories hosted on ships.

195+Stories are the primary driver behind Hall. They are the structures that are used to power circles, which we can now say are named stories hosted on ships.

192196

193197With the configuration described above in mind, we can try and imagine the things we can do with stories. Knowing that we can subscribe them to any number of sources, they can function as central hubs for our messaging, aggregate specific kinds of data feeds, or simply accept whatever messages get sent to it like a regular old chatroom.

194198

195199### General use

196200

197−Upon initial startup, the broker creates a default story, a mailbox named `porch`. (This name is different for stars and galaxies.) This mailbox is the primary target for anyone and anything that wants to reach its owner. Applications can use it to send notifications and other information to the user (the broker itself does this as well), and users can use it to direct-message others.

201+Upon initial startup, Hall creates a default story, a mailbox named `porch`. (This name is different for stars and galaxies: `floor` and `court` respectively.) This mailbox is the primary target for anyone and anything that wants to reach its owner. Applications can use it to send notifications and other information to the user (Hall itself does this as well), and users can use it to direct-message others.

202+

203+Applications that use Hall are encouraged to use the default mailbox as the primary messaging hub. This way, users can easily switch between different applications without "losing" their subscriptions, message backlog, etc.

204+

205+As an example, Talk operates like this, serving as an interface for reading from and managing the user's mailbox. Local stories are subscribed to through the `porch`, which ends up containing all messages the user receives.

198206

199−Talk reader applications are encouraged to use the default mailbox as the primary messaging hub. This way, users can easily switch between different applications without "losing" their subscriptions, message backlog, etc.

207+### Federation

208+

209+Some places, like `urbit-meta`, will want to be easily accessible to everyone on the network, and actually support the load that comes with that. Both these desired can be met through federation.

200210

201−As an example, the "official" talk reader operates like this, serving as an interface for reading from and managing the user's mailbox. Local stories are subscribed to through the `porch`, which ends up containing all messages the user receives.

211+In federation, stories and any changes to them trickle down from ~zod, to other galaxies, to their stars. Only changes to messages and presence can also move upstream, from stars, to galaxies, to ~zod (and then back down again).

212+This way, all planets and comets will be able to subscribe to `/urbit-meta` and get all updates on the story, regardless of where on the network they originated.

202213

203214

204215## Interfaces for applications

205216

206−Applications can interact with the broker in two complementary ways: they can tell it what actions to perform, and they can subscribe to its state changes.

217+Applications can interact with Hall in two complementary ways: they can tell it what actions to perform, and they can subscribe to its state changes.

207218

208219### Interactions

209220

210−The broker can be commanded by poking it with `action`s.

221+Hall can be commanded by poking it with `action`s.

211222

212223```

213224++ action :> user action

214225 $% :: circle configuration ::

215226 {$create nom/knot des/cord sec/security} :< create circle

216− {$delete nom/knot ano/(unit cord)} :< delete + announce

227+ {$delete nom/knot why/(unit cord)} :< delete + announce

217228 {$depict nom/knot des/cord} :< change description

229+ {$filter nom/knot fit/filter} :< change message rules

218230 {$permit nom/knot inv/? sis/(set ship)} :< invite/banish

219231 {$source nom/knot sub/? src/(set partner)} :< un/sub to/from src

220− {$enlist nom/knot fed/? sis/(set ship)} :< dis/allow federation

221− {$burden circle} :< help federate

222232 :: messaging ::

223233 {$convey tos/(list thought)} :< post exact

224234 {$phrase aud/(set partner) ses/(list speech)} :< post easy

225235 :: personal metadata ::

226− {$status cis/(set circle) sat/status} :< our status update

236+ {$notify nos/(set knot) pes/presence} :< our presence update

237+ {$naming nos/(set knot) man/human} :< our name update

227238 :: changing shared ui ::

228239 {$human sip/ship man/human} :< new identity

229240 {$glyph gyf/char pas/(set partner) bin/?} :< un/bind a glyph

230241 == ::

231242```

232243

233−The largest part of these actions concern themselves with managing local circles. `nom` is always the name used to identify the relevant story. To disambiguate between "add" and "delete" type actions (for permissions, subscriptions and federation), a loob `?` is used.

244+The largest part of these actions concern themselves with managing local circles. `nom` is always the name used to identify the relevant story. To disambiguate between "add" and "delete" type actions (for permissions and subscriptions), a loob `?` is used.

234245

235246To send messages, two interfaces are available. `%convey` lets you specify all details of the messages, including its timestamp, serial, and fully assembled audience. `%phrase`, on the other hand, takes care of that for you, allowing you to specify just the target partners and message contents.

236247

237−`%status` is useful for setting your own presence and nickname in a circle, so others can see if you're active.

238−

239−> **Current implementation** implements a `%status` command that only works for local circles.

248+`%notify` and `%naming` are useful for setting your own presence and nickname in a circle respectively, so others can see if you're active and what to call you by.

240249

241250"Shared UI" encompasses UI data that should be consistent across applications. For example, if the user sets a local nickname for an identity, they expect to see that nickname regardless of the application they're currently using. The same goes for glyph bindings, for easy audience targeting.

242251

243−Sometimes, in response to a user action, the broker will respond with a `reaction`.

252+Sometimes, in response to a user action, Hall will respond with a `reaction`.

244253

245254```

246255++ reaction :> user information

252261

253262These are used to inform the application (and thus the user) of any unexpected side effects or errors that occurred as a result of the action they issued.

254263

255−### Subscriptions

264+### Queries, prizes and rumors

256265

257−To receive data from a broker, applications will have to subscribe to it. A `%peer` move for doing so looks something like this:

266+To receive data from Hall, applications will have to query it. There are two paths that are useful for readers to query. Let's look at them and their results.

258267

259268```

260−:* ost.bol :< bone

261− %peer :< move type

262− /story/[some-story] :< diff path

263− [our %broker] :< peer target

264− /reader/[some-story] :< peer path

265−== ::

269+++ query :> query paths

270+ $% {$reader $~} :< shared ui state

271+ {$circle nom/knot ran/range} :< story query

272+ :: ... ::

273+ == ::

274+++ range (unit {hed/place tal/(unit place)}) :< msg range, @ud/@da

275+++ place $%({$da @da} {$ud @ud}) :< point for range

276+++ prize :> query result

277+ $% $: $reader :< /reader

278+ gys/(jug char (set partner)) :< glyph bindings

279+ nis/(map ship cord) :< nicknames

280+ == ::

281+ {$circle burden} :< /circle

282+ :: ... ::

283+ == ::

284+++ burden :< full story state

285+ $: gaz/(list telegram) :< all messages

286+ cos/lobby :< loc & rem configs

287+ pes/crowd :< loc & rem presences

288+ == ::

266289```

267290

268−The diff path is for the application to use to identify what subscription a diff came from. That is, what story the diff relates to. Not all applications will care for this.

269−The peer path informs the broker of the intent of the subscription. It always starts with `/reader` to indicate we're a client application, not another broker. If the path ends there, it subscribes to shared UI changes. If it goes on to specify a story name, it subscribes to changes to that story.

291+To be clear, the paths for those queries are `/reader` and `/circle/[name]/[start]/[end]`, where `start` and `end` are optional and can be either a date or a message number.

292+`/reader` queries produce shared UI state. That is, glyph bindings and nicknames.

293+`/circle` queries produce the entire public state of the requested story, including remotes.

270294

271−These changes get sent as `lowdown`s.

295+Any changes to the results of these queries are communicated via the following `rumor`s. It's a fairly long list of potential changes,

272296

273297```

274−++ lowdown :> new/changed state

275− $% :: story state ::

276− $: $confs :< configs

277− loc/(unit config) :< local config

278− rem/(map circle (unit config)) :< remote configs

298+++ rumor :< query result change

299+ $% $: $reader :< /reader

300+ $= dif ::

301+ $% {$glyph diff-glyph} ::

302+ {$nick diff-nick} ::

303+ == ::

279304 == ::

280− {$precs reg/crowd} :< presences

281− {$grams num/@ud gaz/(list telegram)} :< messages

282− :: ui state ::

283− {$glyph (jug char (set partner))} :< glyph bindings

284− {$names (map ship (unit human))} :< nicknames

305+ {$circle dif/diff-story} :< /circle

306+ :: ... ::

307+ == ::

308+++ diff-glyph {bin/? gyf/char pas/(set partner)} :< un/bound glyph

309+++ diff-nick {who/ship nic/cord} :< changed nickname

310+++ diff-story ::

311+ $% {$new cof/config} :< new story

312+ {$grams gaz/(list telegram)} :< new/changed msgs

313+ {$config cir/circle dif/diff-config} :< new/changed config

314+ {$status pan/partner who/ship dif/diff-status} :< new/changed status

315+ {$follow sub/? pas/(set partner)} :< un/subscribe

316+ {$remove $~} :< removed story

317+ :: ... ::

318+ == ::

319+++ diff-config :> config change

320+ $% {$full cof/config} :< set w/o side-effects

321+ {$source add/? pas/(set partner)} :< add/rem sources

322+ {$caption cap/cord} :< changed description

323+ {$filter fit/filter} :< changed filter

324+ {$secure sec/security} :< changed security

325+ {$permit add/? sis/(set ship)} :< add/rem to b/w-list

326+ {$remove $~} :< removed config

327+ == ::

328+++ diff-status :> status change

329+ $% {$full sat/status} :< fully changed status

330+ {$presence pec/presence} :< changed presence

331+ {$human dif/diff-human} :< changed name

332+ {$remove $~} :< removed config

333+ == ::

334+++ diff-human :> name change

335+ $% {$full man/human} :< fully changed name

336+ {$true tru/(unit (trel cord (unit cord) cord))} :< changed true name

337+ {$handle han/(unit cord)} :< changed handle

285338 == ::

286339```

287340

288−Aside from the initial lowdown that is sent when a subscription starts, these contains exclusively the changes as they occur. Configurations are wrapped in `unit`s so that deletion can be indicated. This also means that simple applications don't necessarily need to keep state. If all they want to do is show events as they happen, then the broker subscription will provide all they need.

341+These are, as enforced by Gall, the changes as they happen. This makes it possible for very minimal Hall readers to be implemented that only show a stream of changes, rather than keeping state themselves.

289342

290343

291−## Communication between brokers

344+## Communication between Halls

292345

293−Brokers can communicate with other brokers to send commands and subscription updates.

346+Halls can communicate with other Halls to send commands and subscription updates.

294347

295348### Interactions

296349

297−To request changes to a story, brokers can send `command`s.

350+To request changes to a story, Halls can send `command`s.

298351

299352```

300353++ command :> effect on story

301− $% {$review tos/(list thought)} :< deliver

302− $: $burden :< starting fed state

303− nom/knot ::

304− cof/lobby ::

305− pes/crowd ::

306− gaz/(list telegram) ::

307− == ::

308− {$relief nom/knot who/(set ship)} :< federation ended

354+ $% {$publish tos/(list thought)} :< deliver

355+ {$present nos/(set knot) dif/diff-status} :< status update

356+ :: ... ::

309357 == ::

310358```

311359

312−A `%review` command contains messages. These are sent to foreign brokers to be published to one of their stories.

360+A `%publish` command contains messages. These are sent to foreign Halls to be published to their stories.

313361

314−`%burden` and `%relief` commands are used for federation. The former signals the sender wants to start federating story `nom`, and contains any pre-existing state that needs to be merged into it. The latter signals that the hosts in `who` will no longer be federating the story.

362+The `%present` command is used for sending status updates about our ship.

315363

316−### Subscriptions

364+### Querying

317365

318−Brokers subscribe to other brokers whenever a source gets added to a story's configuration. (Likewise, they unsubscribe when a source is removed.) Here's the peer move we send when we start a subscription:

366+Halls can query other halls. Here's the peer move we send when we open a query on a foreign circle:

319367

320368```

321369:* ost.bol :< bone

322370 %peer :< move type

323− /friend/show/[our-circle]/[host]/[their-circle] :< diff path

324− [host %broker] :< peer target

325− /[their-circle]/[start]/[end] :< peer path

371+ /circle/[our-circle]/[host]/[their-circle] :< rumor path

372+ [host %hall] :< query target

373+ /circle/[their-circle]/[start]/[end] :< query path

326374== ::

327375```

328376

329−Again, the diff path is just what our broker uses to identify what subscription a received message originates from. The peer path is more interesting. Of course it specifies the name of their story we want to subscribe our story to, but also a "start" and "end". These can be used to specify how long we want our subscription to last. Maybe we want messages starting from last week, up to the end of the month. We may also specify message numbers instead of dates, if we're so inclined.

377+Again, the rumor path is what our Hall uses to identify what query a received message originates from. Our circle is specified so that we know what story is interested in the changes we get.

378+The query path is slightly more interesting. Of course it specifies the name of their story we want to subscribe our story to, but also a "start" and "end". These can (optionally) be used to specify the range of messages we want to get from our query. Once we're out of that range, we stop receiving updates.

330379

331−For as long as their subscriptions last, subscribers get sent `report`s whenever changes occur.

380+### Federation

381+

382+Along the way so far, we've skipped some parts of structures. Most of these relate to federation. Let's see what we missed.

332383

333384```

334−++ report :> update

335− $% {$lobby cab/lobby} :< config neighborhood

336− {$crowd reg/crowd} :< presence

337− {$grams num/@ud gaz/(list telegram)} :< thoughts

385+++ query :> query paths

386+ $% {$burden who/ship} :< duties to share

387+ {$report $~} :< duty reports

388+ :: ... ::

389+ == ::

390+++ prize :> query result

391+ $% {$burden sos/(map knot burden)} :< /burden

392+ :: ... ::

393+ == ::

394+++ rumor :< query result change

395+ $% {$burden nom/knot dif/diff-story} :< /burden

396+ :: ... ::

397+ == ::

398+++ diff-story ::

399+ $% {$bear bur/burden} :< new inherited story

400+ {$burden bur/?} :< burden flag

401+ :: ... ::

338402 == ::

339403```

340404

341−There are three different kinds of reports, for configurations, presences, and messages.

405+We briefly touched upon how federation works on the higher level. On the lower, this is how things go down.

342406

343−Attentive eyes may have seen this in lowdowns as well, they contains both local and remote data. This is required for federation to share full state, and some regular subscribers might take interest in that data as well.

407+1. When Hall boots on a star or galaxy, it starts querying its parent's (~zod for galaxies) `/burden` path.

408+2. Upon receiving that query, the parent sends a `%burden` prize containing all state of its channels (fully public circles) to the child. It also subscribes to `/report` on that child.

409+3. When a new story gets created on the parent, its children get a `%burden` rumor with a `%bear` story diff, and they create a local copy of that channel.

410+4. When something about a parent's story changes, a `%burden` rumor is sent to all its children (because they're querying `/burden`). The children apply this change to their local version of the story.

411+5. When a child's story has a `%grams` or `%status` change, a `%burden` rumor is sent to the parent (because they're querying `/report`). The parent applies this change to their local version of the story, and #4 happens.

344412

345413

346−## Broker implementation

414+## Hall implementation

347415

348416Now that we know what gets sent over the wire and why, let's see what happens whenever such events happen. We won't be covering everything, but enough to illustrate the general flow.

349417

350−First though, it's useful to understand how the broker's code is structured. It consists of two primary cores that are used as engines for applying changes to state and producing moves for output.

418+First though, it's useful to understand how Hall's code is structured. It consists of primary cores, designed to work optimally with Gall.

419+

420+* `++ta` is the transaction core. For every event that happens, be it a poke, peer or diff, the transaction core gets put to work to process it. Once it's done, it produces a list of deltas that describe how our state needs to be modified.

421+* `++da` is the delta application core. It is used to apply `++ta`'s deltas to our application state, and produce a list of side-effects.

351422

352−* `++ta` is the transaction core. For every event that happens, be it a poke, peer of diff, the transaction core gets put to work to process it. Once it's done, it produces a list of moves and the new state.

353−* `++so` is the story core, used by `++ta` for applying changes to stories. In doing so, it may add moves to the transaction core for it to produce.

423+Both those cores also have a core within themselves dedicated solely to dealing with stories, `++so` and `++sa` respectively.

354424

355−Once all work in an engine core has finished, its `-done` arm is called to produce moves and apply the changes it made back into the application state.

425+Once all work in an engine core has finished, its `-done` arm is called to produce the result of all the core's computation.

356426

357427In the diagrams that follow, regular lines indicate flows that always happen. Dashed lines indicate flows that are traversed if the described condition is met.

358428Below the diagrams you will find brief explanations of the involved arms. In a particularly bright future, these might be available on-hover instead.

359429For brevity, utility arms that aren't a direct and important part of the flow have been omitted.

360430

431+(You will find that the origins still use old Gall arms, with some glue code in between. At the time of writing, new Gall has not yet been implemented, so this will have to do.)

432+

433+We will generally only display the delta generation parts of the flow. Delta application is usually implemented simply enough to deduce what it does from looking at the label.

434+

361435### Subscriptions

362436

363−Broker-to-broker subscriptions happen when a source gets added to a story. This is done with a `%source` action, and results in a `%peer` move being sent. Upon receiving a peer, the broker validates the subscription and sends the current state.

437+Hall-to-Hall subscriptions happen when a source gets added to a story. This is done with a `%source` action, and results in a `%peer` move being sent (prompted by a `%story %follow` delta).

438+Upon receiving a valid peer on a `/circle` path, the subscribing ship is added to that circle's presence map.

364439

365440![subscriptions implementation flow](./diagrams/flow-subscriptions.png "subscriptions implementation flow")

366441

367442```

368−++ poke-talk-action :< we got poked with an action.

369− ++ ta-action :< applies an action.

370− ++ ta-config :< (re)configures a story.

371− ++ so-reform :< changes the configuration of the story.

372− ++ so-acquire :< subscribes the story to new sources.

373−:: ::

374−++ peer :< we got peered.

375− ++ ta-subscribe :< sends initial data: current state.

376− ++ so-report-lobby :< sends a report with configurations.

377− ++ so-notify :< changes a ship's presence in the story.

378− ++ so-report-crowd :< sends a report with presences.

379− ++ so-start :< determines the backlog of messages to send.

380− ++ so-first-grams :< sends a range of messages and adds follower.

443+++ poke-talk-action :< we got poked with an action.

444+ ++ ta-action :< applies an action.

445+ ++ ta-config :< (re)configures a story.

446+:: ::

447+++ peer :< we got peered.

448+ ++ g-query :< resolves query to prize.

449+ ++ ta-subscribe :< act upon subscription.

450+ ++ so-attend :< adds a ship's status to the story.

381451```

382452

383−### Reports

453+### Rumors

384454

385−Once a subscription has been established, the broker will receive reports. `%lobby` and `%group` reports are applied through the following flow. Note how, when a `%lobby` from a federating ship is received, this authoritatively changes the story's configuration.

455+Once a query has opened, Hall will receive updates on it, rumors. The changes describes in these rumors are applied via the following flow.

386456

387−![report diff implementation flow](./diagrams/flow-reports.png "report diff implementation flow")

457+![rumor implementation flow](./diagrams/flow-rumors.png "report diff implementation flow")

388458

389459```

390−++ diff-talk-report :< we got a subscription report.

391− ++ ta-diff-report :< applies a report to the story it's intended for.

392− ++ so-diff-report :< applies a report.

393− ++ so-lobby :< stores updated configuration.

394− ++ so-reform :< changes the configuration of the story.

395− ++ so-report-lobby :< sends a report with configuration.

396− ++ so-remind :< stores updated presences.

397− ++ so-report-crowd :< sends a report with presences.

460+++ diff-talk-rumor :< we got a query update.

461+ ++ ta-hear :< applies a rumor to the story it's intended for.

462+ ++ so-hear :< applies a %circle rumor to the story.

463+ ++ so-config-full :< splits a %full diff-config into separate changes.

464+ ++ so-bear :< accept burden, assimilate into the story's state.

465+ ++ so-lesson :< sends a report with presences.

398466```

399467

400468### Messaging

401469

402−To send messages, the user sends a `%convey` or `%phrase` action, resulting in a `%review` command being sent to the involved partners. Receiving a `%review` command causes its messages to be added to the story, which sends a `%grams` report to subscribers and a `%grams` lowdown to all reader applications.

470+To send messages, the user sends a `%convey` or `%phrase` action, resulting in a `%publish` command being sent to the involved partners. Receiving a `%publish` command causes its messages to be added to the story through the creation of a `%story %grams` delta.

403471

404472![messaging implementation flow](./diagrams/flow-messaging.png "messaging implementation flow")

405473

406474```

407−++ poke-talk-action :< we got poked with an action.

408− ++ ta-action :< applies an action.

409−:: ::

410−++ poke-talk-command :< we get poked with a command.

411− ++ ta-apply :< applies a command.

412− ++ ta-think :< consumes each message in the given list.

413− ++ ta-consume :< conducts a message to each partner in audience.

414− ++ ta-conduct :< records or sends a message.

415− ++ ta-transmit :< sends a message to a partner.

416− ++ ta-record :< stores a message in a story.

417−:: ::

418−++ diff-talk-report :< we got a subscription report.

419− ++ ta-diff-report :< applies a report to the story it's intended for.

420− ++ so-diff-report :< applies a report.

421− ++ so-lesson :< learns a list of messages.

422− ++ so-learn :< either adds or modifies a message.

423− ++ so-append :< adds a new message to the story.

424− ++ so-revise :< modifies an existing message in the story.

425− ++ so-refresh :< sends messages to all interested subscribers.

475+++ poke-talk-action :< we got poked with an action.

476+ ++ ta-action :< applies an action.

477+:: ::

478+++ poke-talk-command :< we get poked with a command.

479+ ++ ta-apply :< applies a command.

480+ ++ ta-think :< consumes each message in the given list.

481+ ++ ta-consume :< conducts a message to each partner in audience.

482+ ++ ta-conduct :< records or sends a message.

483+ ++ ta-transmit :< sends a message to a partner.

484+ ++ ta-record :< stores a message in a story.

485+ ++ so-learn :< either adds or modifies a message.

426486```

427487

428488

429489## The future

430490

431−Talk is neither complete nor perfect. There are still problems that need to be solved and features that need to be implemented.

432−

433−### Architectural issues

434−

435−To distribute commands and reports to all that are affected by it, the broker takes a "propagate changes" approach. If, in processing a report, any state changes, new state gets sent to all interested brokers... even those it got the report from in the first place. This results in a lot of unnecessary data transfer, where everyone always gets notified of each other's state changes, regardless of whether those changes are old news or not.

436−

437−While not a practical problem (yet, things work fine at the current scale of use), it does underline the need for a more structured approach to the broker's subscription model. In an ideal world, every interested node gets notified of a set of changes exactly once.

438−

439−That still becomes unwieldy for circles that have thousands of subscribers, since a single node would still be sending thousands of network events. An obvious potential solution is to apply Urbit's hierarchical structure to this, and send messages to stars for them to send to their children.

440−

441−While on the topic of subscriptions, the current implementation treats them as semantic data, explicitly storing them in application state. It would be desirable to push this to the background again.

491+Hall is neither complete nor perfect. There are features that need to be implemented, varying from small quality of life changes to broad-reaching functionality.

442492

443493### Features and functionality

444494

445495To turn talk into a [sustainable social platform](https://urbit.org/fora/posts/~2017.4.26..18.00.25..b93c~/), it needs a number of things.

446496

447−Primarily, discoverability of circles. A possible solution for realizing this is making it possible for users to add friends, which would have their broker subscribe to a list of circles the added friends are in. Of course, one will be able to set joined circles as "private" to keep them from being visible to others.

497+Primarily, discoverability of circles. A possible solution for realizing this is making it possible for users to add friends, which would have their Hall subscribe to a list of circles the added friends are in. Of course, one will be able to set joined circles as "private" to keep them from being visible to others.

448498

449499Eventually, content self-moderation might need to be implemented. Users would flag their content if it is potentially offensive. Others would easily be able to filter out content that might offend them.

450500This functionality brings its own large sets of challenges that need to be tackled.

451501

452−Additionally, federation might be put to broader use. Knowing a circle is federated could help fall back to alternative hosts in case the one the broker originally subscribed to is unavailable.

502+Additionally, federation might be put to broader use. Knowing a circle is federated could help fall back to alternative hosts in case the one Hall originally subscribed to is unavailable.

453503Implementation-wise, care would need to be taken for this to not get ugly. Not to mention, when/how would availability of a host be verified?

454504

505+Other changes that might have a fair impact on the functioning of Hall are un/read states for messages, and being able to use moons to allow your friends to chat with you in a local channel.

506+

455507Smaller functionality that still needs to be implemented includes:

456508* Extended permissions management. Being able to black-/whitelist entire ship classes.

457509* Improved presence functionality. Actually using the presence system, and sending "typing", "idle", etc. statuses.

458−* Per-circle message rules. Dis/allow unicode, enforce line length, etc.

459510* Flexible on-subscribe scrollback. Some cases may want more than just a day's worth.

460−* Talkpolls.

511+* Polls.

512+

513+And of course there's a lot Talk, the client, can improve on as well.

461514

462515

463516## Further reading

464517

465−To gain a more thorough understanding of the broker's inner workings, take a look at its source code. It comes with inline documentation.

518+To gain a more thorough understanding of Hall's inner workings, take a look at its source code. It comes with inline documentation.

466519[link]

467520

468−To see an expansive example of an application that uses the broker, take a look at the code of the talk reader. This, too, comes with inline documentation.

521+To see an expansive example of an application that uses Hall, take a look at the code of Talk. It, too, comes with inline documentation.

469522[link]