6title: Hall architecture
11This document is complemented by Hall's source code, but doesn't require it. Definitions of data structures, code snippets and diagrams will be provided where useful.
13Hall's implementation is structured according to the new Gall model. Familiarity with its concepts is assumed. See [the new Gall model](../new-gall).
17* [Introduction](#introduction)
18* [Structures & functionality](#structures--functionality)
20 * [Messages](#messages)
21 * [Participant metadata](#participant-metadata)
22 * [Configurations](#configurations)
24 * [General use](#general-use)
25 * [Federation](#federation)
26 * [Public membership](#public-membership)
27* [Interfaces for applications](#interfaces-for-applications)
28 * [Interactions](#interactions)
29 * [Queries, prizes and rumors](#queries-prizes-and-rumors)
30* [Communication between Halls](#communication-between-halls)
31 * [Interactions](#interactions-1)
32 * [Querying](#querying)
33 * [Federation](#federation)
34* [Hall implementation](#hall-implementation)
35 * [Subscriptions](#subscriptions)
37 * [Messaging](#messaging)
38* [The future](#the-future)
39 * [Features and functionality](#features-and-functionality)
40* [Further reading](#further-reading)
45Talk was Urbit's first big user-facing application. It continues to enjoy a prominent role in the Urbit landscape, but does so as two separate applications this time.
47The messaging parts of Talk have been separated from its user interface parts. What we ended up with is a shiny new generic messaging bus, and the chat interface we all know and love. The messaging bus, which should now prove useful to many different applications, will be named `Hall`. Applications that use it are referred to as *clients*. One such application, as you might have guessed, is `Talk`.
50## Structures & functionality
55++ circle {hos/ship nom/term} :< native target
58A `circle` is essentially a named collection of messages created by and hosted on a ship's Hall, usually represented as `~ship-name/circle-name`. Most of Hall revolves around doing things with circles.
62When we subscribe to a circle, the primary thing we're interested in is its messages. Message data itself isn't that complicated, but a fair amount of metadata comes into play when actually sending a message. Let's work our way up, starting at the contents.
65++ speech :> content body
66 $% {$lin pat/? msg/cord} :< no/@ text line
67 {$url url/purf:eyre} :< parsed url
68 {$exp exp/cord res/(list tank)} :< hoon line
69 {$ire top/serial sep/speech} :< in reply to
70 {$fat tac/attache sep/speech} :< attachment
71 {$app app/term sep/speech} :< app message
72 {$inv inv/? cir/circle} :< inv/ban for circle
76At the heart of every message lies a `speech` that describes the message body. There's a large number of different speech types, from simple text messages to parsed URLs, Hoon expressions and more.
79++ audience (set circle) :< destinations
80++ serial @uvH :< unique identifier
81++ thought :> inner message
82 $: uid/serial :< unique identifier
83 aud/audience :< destinations
89A `speech` is always accompanied by various bits of metadata. We include a unique identifier, a message timestamp and an `audience`, the set of all intended recipients of the message.
92++ telegram {aut/ship thought} :< whose message
93++ envelope {num/@ud gam/telegram} :< outward message
96Finally, before sending the message over the wire, we add on the message's original author. In most cases, we also include a sequence number, the index of the message in the originating circle's list of messages. This can be a useful point of reference for clients.
98### Participant metadata
100Messages aren't the only thing a subscription gets us. We're also kept up to date with relevant metadata.
103++ crowd {loc/group rem/(map partner group)} :< our & srcs presences
104++ group (map ship status) :< presence map
105++ status {pec/presence man/human} :< participant
106++ presence :> status type
112++ human :> human identifier
113 $: han/(unit cord) :< handle
114 tru/(unit truename) :< true name
116++ truename {fir/cord mid/(unit cord) las/cord} :< real-life name
119`status` is user-set metadata that describes, well, the status of users in a circle. This encompasses their `presence`, which shows their activity, and their `human` identity, which includes their display handle.
120For reasons we'll discover shortly, circles keep track of both their own `group` and those of the circles they're subscribed to. `crowd` encapsulates this.
125++ lobby {loc/config rem/(map circle config)} :< our & srcs configs
126++ config :> circle config
127 $: src/(set circle) :< active sources
128 cap/cord :< description
129 fit/filter :< message rules
130 con/control :< restrictions
132++ filter :> content filters
133 $: cas/? :< dis/allow capitals
134 utf/? :< dis/allow non-ascii
136++ control {sec/security ses/(set ship)} :< access control
139Another part of metadata we get from circle subscriptions is configurations. Again, circles want to remember their own configuration, as well as those of their subscriptions. But why, precisely?
141The `config` structure contains `src`, a set of partners. These indicate the different sources a circle is currently pulling content from. This allows circles to aggregate messages from multiple places. In doing so, it also receives metadata from those places, hence why we have structures for storing "remote" presences and configurations alongside local ones.
143Aside from that, `config` contains a description for the circle, which is exactly what it sounds like. It also specifies a `filter`, which are content formatting rules all messages passing through this circle are made to adhere to.
145Lastly, there's a `control` structure that contains both a security mode and a list of ships, which is either a white- or blacklist depending on the aforementioned mode. There are four such modes available.
148++ security :> security mode
149 $? $channel :< blacklist
150 $village :< whitelist
151 $journal :< pub r, whitelist w
152 $mailbox :< our r, blacklist w
156A `channel` is publicly readable and writable, with a blacklist for blocking.
157A `village` is privately readable and writable, with a whitelist for inviting.
158A `journal` is publicly readable and privately writable, with a whitelist for authors.
159A `mailbox` is readable by its owner and publicly writable, with a blacklist for blocking.
163To see how that all ties together, we're going to take a look at Hall's state.
166++ state :> hall state
167 $: stories/(map term story) :< conversations
170++ story :> wire content
171 $: grams/(list telegram) :< all messages
172 locals/group :< local presence
173 remotes/(map circle group) :< remote presence
174 shape/config :< configuration
175 mirrors/(map circle config) :< remote config
180Stories are the primary driver behind Hall. They are the structures that are used to power circles, which we can now say are named stories hosted on ships.
182With the configuration described above in mind, we can try and imagine the things we can do with stories. Knowing that we can subscribe them to any number of sources, they can function as central hubs for our messaging, aggregate specific kinds of data feeds, or simply accept whatever messages get sent to it like a regular old chatroom.
186Upon initial startup, Hall creates a default story, a mailbox named `inbox`. This mailbox is the primary target for anyone and anything that wants to reach its owner. Applications can use it to send notifications and other information to the user (Hall itself does this as well), and users can use it to send direct messages to each other.
188Applications that use Hall, especially clients, are encouraged to use the default mailbox as the primary messaging hub. This way, users can easily switch between different applications without "losing" their subscriptions, message backlog, etc.
190As an example, Talk operates like this, serving as an interface for reading from and managing the user's mailbox. Local stories are subscribed to through the `inbox`, which ends up containing all messages the user receives.
194Some places, like `urbit-meta`, will want to be easily accessible to everyone on the network, and actually support the load that comes with that. Both these desired can be met through federation.
196In federation, stories and any changes to them trickle down from ~zod, to other galaxies, to their stars. Only changes to messages and presence can also move upstream, from stars, to galaxies, to ~zod (and then back down again).
197This way, all planets and comets will be able to subscribe to `/urbit-meta` (that is, the `urbit-meta` circle on their parent ship) and get all updates on the story, regardless of where on the network they originated.
201To aid in discoverability of circles, it is possible for users to make their participation in any given circle public. This data can then be used by things like profile widgets, or read out directly by other users.
204## Interfaces for applications
206Applications can interact with Hall in two complementary ways: they can tell it what actions to perform, and they can subscribe to its state and the changes made to it.
210Hall can be commanded by poking it with `action`s.
213++ action :> user action
214 $% :: circle configuration ::
215 {$create nom/term des/cord sec/security} :< create circle
216 {$delete nom/term why/(unit cord)} :< delete + announce
217 {$depict nom/term des/cord} :< change description
218 {$filter nom/term fit/filter} :< change message rules
219 {$permit nom/term inv/? sis/(set ship)} :< invite/banish
220 {$source nom/term sub/? srs/(set source)} :< un/sub to/from src
222 {$convey tos/(list thought)} :< post exact
223 {$phrase aud/audience ses/(list speech)} :< post easy
224 :: personal metadata ::
225 {$notify aud/audience pes/(unit presence)} :< our presence update
226 {$naming aud/audience man/human} :< our name update
227 :: changing shared ui ::
228 {$glyph gyf/char aud/audience bin/?} :< un/bind a glyph
229 {$nick who/ship nic/cord} :< new identity
231 {$public add/? cir/circle} :< show/hide membership
235The largest part of these actions concern themselves with managing local circles. `nom` is always the name used to identify the relevant story. To disambiguate between "add" and "delete" type actions (for permissions and subscriptions), a loob `?` is used.
237To send messages, two interfaces are available. `%convey` lets you specify all details of the messages, including its timestamp, serial, and fully assembled audience. `%phrase`, on the other hand, takes care of that for you, allowing you to specify just the target partners and message contents.
239`%notify` and `%naming` are useful for setting your own presence and nickname in a circle respectively, so others can see if you're active and what to call you by.
241"Shared UI" encompasses UI data that should be consistent across applications. For example, if the user sets a local nickname for an identity, they expect to see that nickname regardless of the application they're currently using. The same goes for glyph bindings, for easy audience targeting.
243### Queries, prizes and rumors
245To receive data from Hall, applications will have to query it. There are two paths that are useful for clients to query. Let's look at them and their results.
248++ query :> query paths
249 $% {$client $~} :< shared ui state
250 $: $circle :> story query
251 nom/naem :< circle name
252 wat/(set circle-data) :< data to get
253 ran/range :< query duration
257++ circle-data :> kinds of circle data
258 $? $grams :< messages
259 $group-l :< local presence
260 $group-r :< remote presences
261 $config-l :< local config
262 $config-r :< remote configs
264++ range :> inclusive msg range
265 %- unit :< ~ means everything
266 $: hed/place :< start of range
267 tal/(unit place) :< opt end of range
269++ place :> range indicators
271 {$ud @ud} :< message number
273++ prize :> query result
274 $% $: $client :< /client
275 gys/(jug char (set partner)) :< glyph bindings
276 nis/(map ship cord) :< nicknames
278 {$circle burden} :< /circle
281++ burden :< full story state
282 $: gaz/(list telegram) :< all messages
283 cos/lobby :< loc & rem configs
284 pes/crowd :< loc & rem presences
288To be clear, the paths for those queries are `/client` and `/circle/[name]/[data]/[start]/[end]`, where data is at least one `circle-data`, and `start` and `end` are optional and can be either a date or a message number.
289`/client` queries produce shared UI state. That is, glyph bindings and nicknames.
290`/circle` queries produce the entire public state of the requested story, including remotes.
292Any changes to the results of these queries are communicated via the following `rumor`s. It's a fairly long list of potential changes,
295++ rumor :< query result change
296 $% {$client rum/rumor-client} :< /client
297 {$circle rum/rumor-story} :< /circle
300++ rumor-client :< changed ui state
301 $% {$glyph diff-glyph} :< un/bound glyph
302 {$nick diff-nick} :< changed nickname
304++ diff-glyph {bin/? gyf/char aud/audience} :< un/bound glyph
305++ diff-nick {who/ship nic/cord} :< changed nickname
306++ diff-story :> story change
307 $% {$new cof/config} :< new story
308 {$config cir/circle dif/diff-config} :< new/changed config
309 {$status cir/circle who/ship dif/diff-status} :< new/changed status
310 {$remove $~} :< removed story
313++ rumor-story ::> story rumor
314 $? diff-story ::< both in & outward
315 $% {$gram nev/envelope} ::< new/changed msgs
317++ diff-config :> config change
318 $% {$full cof/config} :< set w/o side-effects
319 {$source add/? src/source} :< add/rem sources
320 {$caption cap/cord} :< changed description
321 {$filter fit/filter} :< changed filter
322 {$secure sec/security} :< changed security
323 {$permit add/? sis/(set ship)} :< add/rem to b/w-list
324 {$remove $~} :< removed config
326++ diff-status :> status change
327 $% {$full sat/status} :< fully changed status
328 {$presence pec/presence} :< changed presence
329 {$human dif/diff-human} :< changed name
330 {$remove $~} :< removed status
332++ diff-human :> name change
333 $% {$full man/human} :< fully changed name
334 {$handle han/(unit cord)} :< changed handle
335 {$true tru/(unit truename)} :< changed true name
339These are, as enforced by Gall, the changes as they happen. This makes it possible for very minimal Hall clients to be implemented that only display a stream of changes, rather than keeping state themselves.
342## Communication between Halls
344Halls can communicate with other Halls to send commands and subscription updates.
348To request changes to a story, Halls can send `command`s.
351++ command :> effect on story
352 $% {$publish tos/(list thought)} :< deliver
353 {$present nos/(set term) dif/diff-status} :< status update
358A `%publish` command contains messages. These are sent to foreign Halls to be published to their stories.
360The `%present` command is used for sending status updates about our ship.
364Halls can query other halls. Here's the peer move we send when we open a query on a foreign circle:
369 /[our-circle]/[host]/[query-path] :< rumor path
370 [host %hall] :< query target
371 /circle/[their-circle]/[data]/[start]/[end] :< query path
375Again, the rumor path is what our Hall uses to identify what query a received message originates from. Our circle is specified so that we know what story is interested in the changes we get.
376The query path is slightly more interesting. Of course it specifies the name of their story we want to subscribe our story to, but also a "start" and "end". These can (optionally) be used to specify the range of messages we want to get from our query. Once that range has passed, we stop receiving updates.
380Along the way so far, we've skipped some parts of structures. Most of these relate to federation. Let's see what we missed.
383++ query :> query paths
384 $% {$burden who/ship} :< duties to share
385 {$report $~} :< duty reports
388++ prize :> query result
389 $% {$burden sos/(map term burden)} :< /burden
392++ rumor :< query result change
393 $% {$burden nom/term rum/rumor-story} :< /burden
397 $% {$bear bur/burden} :< new inherited story
400++ burden :> full story state
401 $: gaz/(list telegram) :< all messages
402 cos/lobby :< loc & rem configs
403 pes/crowd :< loc & rem presences
407We briefly touched upon how federation works on the higher level. On the lower, this is how things go down.
4091. When Hall boots on a star or galaxy, it starts querying its parent's `/burden` path. (Galaxies query ~zod.)
4102. Upon receiving that query, the parent sends a `%burden` prize containing all state of its channels (fully public circles) to the child. It also subscribes to `/report` on that child.
4113. When a new story gets created on the parent, its children get a `%burden` rumor with a `%bear` story diff, and they create a local copy of that channel.
4124. When something about a parent's story changes, a `%burden` rumor is sent to all its children (because they're querying `/burden`). The children apply this change to their local version of the story.
4135. When a child's story has a `%grams` or `%status` change, a `%burden` rumor is sent to the parent (because they're querying `/report`). The parent applies this change to their local version of the story, and #4 happens.
416## Hall implementation
418Now that we know what gets sent over the wire and why, let's see what happens whenever such events happen. We won't be covering everything, but enough to illustrate the general flow.
420First though, it's useful to understand how Hall's code is structured. It consists of primary cores, designed to work optimally with Gall.
422* `++ta` is the transaction core. For every event that happens, be it a poke, peer or diff, the transaction core gets put to work to process it. Once it's done, it produces a list of deltas that describe how our state needs to be modified.
423* `++da` is the delta application core. It is used to apply `++ta`'s deltas to our application state, and produce a list of side-effects.
425Both those cores also have a core within themselves dedicated solely to dealing with stories, `++so` and `++sa` respectively.
427Once all work in an engine core has finished, its `-done` arm is called to produce the result of all the core's computation.
429In the diagrams that follow, regular lines indicate flows that always happen. Dashed lines indicate flows that are traversed if the described condition is met.
430Below the diagrams you will find brief explanations of the involved arms. In a particularly bright future, these might be available on-hover instead.
431For brevity, utility arms that aren't a direct and important part of the flow have been omitted.
433(You will find that the origins still use old Gall arms, with some glue code in between. At the time of writing, new Gall has not yet been implemented, so this will have to do.)
435We will only display the delta generation parts of the flow. Delta application is usually implemented simply enough to deduce what it does from looking at the label.
439Hall-to-Hall subscriptions happen when a source gets added to a story. This is done with a `%source` action, and results in a `%peer` move being sent (prompted by a `%story %follow` delta).
440Upon receiving a valid peer on a `/circle` path, the subscribing ship is added to that circle's presence map.
442
445++ poke-talk-action :< we got poked with an action.
446 ++ ta-action :< applies an action.
447 ++ ta-config :< (re)configures a story.
449++ peer :< we got peered.
450 ++ g-query :< resolves query to prize.
451 ++ ta-subscribe :< act upon subscription.
452 ++ so-attend :< adds a ship's status to the story.
457Once a query has opened, Hall will receive updates on it, rumors. The changes describes in these rumors are applied via the following flow.
459
462++ diff-talk-rumor :< we got a query update.
463 ++ ta-hear :< applies a rumor to the story it's intended for.
464 ++ so-hear :< applies a %circle rumor to the story.
465 ++ so-config-full :< splits a %full diff-config into separate changes.
466 ++ so-bear :< accept burden, assimilate into the story's state.
467 ++ so-lesson :< sends a report with presences.
472To send messages, the user sends a `%convey` or `%phrase` action, resulting in a `%publish` command being sent to the involved partners. Receiving a `%publish` command causes its messages to be added to the story through the creation of a `%story %grams` delta.
474
477++ poke-talk-action :< we got poked with an action.
478 ++ ta-action :< applies an action.
480++ poke-talk-command :< we get poked with a command.
481 ++ ta-apply :< applies a command.
482 ++ ta-think :< consumes each message in the given list.
483 ++ ta-consume :< conducts a message to each partner in audience.
484 ++ ta-conduct :< records or sends a message.
485 ++ ta-transmit :< sends a message to a partner.
486 ++ ta-record :< stores a message in a story.
487 ++ so-learn :< either adds or modifies a message.
493Hall is neither complete nor perfect. There are features that need to be implemented, varying from small quality of life changes to broad-reaching functionality.
495### Features and functionality
497To turn Talk into a [sustainable social platform](https://urbit.org/fora/posts/~2017.4.26..18.00.25..b93c~/), it needs a number of things.
499Primarily, discoverability of circles. A possible solution for realizing this is making it possible for users to add friends, which would have their Hall subscribe to a list of circles the added friends are in. The existing public membership functionality can be leveraged for this.
501Eventually, content self-moderation might need to be implemented. Users would flag their content if it is potentially offensive. Others would easily be able to filter out content that might offend them.
502This functionality brings its own large sets of challenges that need to be tackled.
504Additionally, federation might be put to broader use. Knowing a circle is federated could help fall back to alternative hosts in case the one Hall originally subscribed to is unavailable.
505Implementation-wise, care would need to be taken for this to not get ugly. Not to mention, when/how would availability of a host be verified?
507Other changes that might have a fair impact on the functioning of Hall are un/read states for messages, and being able to use moons to allow your friends to chat with you in a local channel.
509Smaller functionality that still needs to be implemented includes:
510* Extended permissions management. Being able to black-/whitelist entire ship classes.
511* Improved presence functionality. Actually using the presence system, and sending "typing", "idle", etc. statuses.
514And of course there's a lot Talk, the client, can improve on as well.
519To gain a more thorough understanding of Hall's inner workings, take a look at its source code. It comes with inline documentation.
520[On Github.](https://github.com/urbit/arvo/blob/master/app/hall.hoon)
522To see an expansive example of a Hall client, take a look at the code of Talk. It, too, comes with inline documentation.
523[On Github.](https://github.com/urbit/arvo/blob/master/app/talk.hoon)
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list