@@ -0,0 +1,516 @@
4+This document is complemented by Hall's source code, but doesn't require it. Definitions of data structures, code snippets and diagrams will be provided where useful.
6+Hall's implementation is structured according to the new Gall model. Familiarity with its concepts is assumed. See [the new Gall model](gall.md).
10+* [Introduction](#introduction)
11+* [Structures & functionality](#structures--functionality)
12+ * [Circles](#circles)
13+ * [Messages](#messages)
14+ * [Participant metadata](#participant-metadata)
15+ * [Configurations](#configurations)
16+ * [Stories](#stories)
17+ * [General use](#general-use)
18+ * [Federation](#federation)
19+ * [Public membership](#public-membership)
20+* [Interfaces for applications](#interfaces-for-applications)
21+ * [Interactions](#interactions)
22+ * [Queries, prizes and rumors](#queries-prizes-and-rumors)
23+* [Communication between Halls](#communication-between-halls)
24+ * [Interactions](#interactions-1)
25+ * [Querying](#querying)
26+ * [Federation](#federation)
27+* [Hall implementation](#hall-implementation)
28+ * [Subscriptions](#subscriptions)
30+ * [Messaging](#messaging)
31+* [The future](#the-future)
32+ * [Features and functionality](#features-and-functionality)
33+* [Further reading](#further-reading)
38+Talk was Urbit's first big user-facing application. It continues to enjoy a prominent role in the Urbit landscape, but does so as two separate applications this time.
40+The messaging parts of Talk have been separated from its user interface parts. What we ended up with is a shiny new generic messaging bus, and the chat interface we all know and love. The messaging bus, which should now prove useful to many different applications, will be named `Hall`. Applications that use it are referred to as *clients*. One such application, as you might have guessed, is `Talk`.
43+## Structures & functionality
48+++ circle {hos/ship nom/term} :< native target
51+A `circle` is essentially a named collection of messages created by and hosted on a ship's Hall, usually represented as `~ship-name/circle-name`. Most of Hall revolves around doing things with circles.
55+When we subscribe to a circle, the primary thing we're interested in is its messages. Message data itself isn't that complicated, but a fair amount of metadata comes into play when actually sending a message. Let's work our way up, starting at the contents.
58+++ speech :> content body
59+ $% {$lin pat/? msg/cord} :< no/@ text line
60+ {$url url/purf:eyre} :< parsed url
61+ {$exp exp/cord res/(list tank)} :< hoon line
62+ {$ire top/serial sep/speech} :< in reply to
63+ {$fat tac/attache sep/speech} :< attachment
64+ {$app app/term sep/speech} :< app message
65+ {$inv inv/? cir/circle} :< inv/ban for circle
69+At the heart of every message lies a `speech` that describes the message body. There's a large number of different speech types, from simple text messages to parsed URLs, Hoon expressions and more.
72+++ audience (set circle) :< destinations
73+++ serial @uvH :< unique identifier
74+++ thought :> inner message
75+ $: uid/serial :< unique identifier
76+ aud/audience :< destinations
78+ sep/speech :< content
82+A `speech` is always accompanied by various bits of metadata. We include a unique identifier, a message timestamp and an `audience`, the set of all intended recipients of the message.
85+++ telegram {aut/ship thought} :< whose message
86+++ envelope {num/@ud gam/telegram} :< outward message
89+Finally, before sending the message over the wire, we add on the message's original author. In most cases, we also include a sequence number, the index of the message in the originating circle's list of messages. This can be a useful point of reference for clients.
91+### Participant metadata
93+Messages aren't the only thing a subscription gets us. We're also kept up to date with relevant metadata.
96+++ crowd {loc/group rem/(map partner group)} :< our & srcs presences
97+++ group (map ship status) :< presence map
98+++ status {pec/presence man/human} :< participant
99+++ presence :> status type
105+++ human :> human identifier
106+ $: han/(unit cord) :< handle
107+ tru/(unit truename) :< true name
109+++ truename {fir/cord mid/(unit cord) las/cord} :< real-life name
112+`status` is user-set metadata that describes, well, the status of users in a circle. This encompasses their `presence`, which shows their activity, and their `human` identity, which includes their display handle.
113+For reasons we'll discover shortly, circles keep track of both their own `group` and those of the circles they're subscribed to. `crowd` encapsulates this.
118+++ lobby {loc/config rem/(map circle config)} :< our & srcs configs
119+++ config :> circle config
120+ $: src/(set circle) :< active sources
121+ cap/cord :< description
122+ fit/filter :< message rules
123+ con/control :< restrictions
125+++ filter :> content filters
126+ $: cas/? :< dis/allow capitals
127+ utf/? :< dis/allow non-ascii
129+++ control {sec/security ses/(set ship)} :< access control
132+Another part of metadata we get from circle subscriptions is configurations. Again, circles want to remember their own configuration, as well as those of their subscriptions. But why, precisely?
134+The `config` structure contains `src`, a set of partners. These indicate the different sources a circle is currently pulling content from. This allows circles to aggregate messages from multiple places. In doing so, it also receives metadata from those places, hence why we have structures for storing "remote" presences and configurations alongside local ones.
136+Aside from that, `config` contains a description for the circle, which is exactly what it sounds like. It also specifies a `filter`, which are content formatting rules all messages passing through this circle are made to adhere to.
138+Lastly, there's a `control` structure that contains both a security mode and a list of ships, which is either a white- or blacklist depending on the aforementioned mode. There are four such modes available.
141+++ security :> security mode
142+ $? $channel :< blacklist
143+ $village :< whitelist
144+ $journal :< pub r, whitelist w
145+ $mailbox :< our r, blacklist w
149+A `channel` is publicly readable and writable, with a blacklist for blocking.
150+A `village` is privately readable and writable, with a whitelist for inviting.
151+A `journal` is publicly readable and privately writable, with a whitelist for authors.
152+A `mailbox` is readable by its owner and publicly writable, with a blacklist for blocking.
156+To see how that all ties together, we're going to take a look at Hall's state.
159+++ state :> hall state
160+ $: stories/(map term story) :< conversations
163+++ story :> wire content
164+ $: grams/(list telegram) :< all messages
165+ locals/group :< local presence
166+ remotes/(map circle group) :< remote presence
167+ shape/config :< configuration
168+ mirrors/(map circle config) :< remote config
173+Stories are the primary driver behind Hall. They are the structures that are used to power circles, which we can now say are named stories hosted on ships.
175+With the configuration described above in mind, we can try and imagine the things we can do with stories. Knowing that we can subscribe them to any number of sources, they can function as central hubs for our messaging, aggregate specific kinds of data feeds, or simply accept whatever messages get sent to it like a regular old chatroom.
179+Upon initial startup, Hall creates a default story, a mailbox named `inbox`. This mailbox is the primary target for anyone and anything that wants to reach its owner. Applications can use it to send notifications and other information to the user (Hall itself does this as well), and users can use it to send direct messages to each other.
181+Applications that use Hall, especially clients, are encouraged to use the default mailbox as the primary messaging hub. This way, users can easily switch between different applications without "losing" their subscriptions, message backlog, etc.
183+As an example, Talk operates like this, serving as an interface for reading from and managing the user's mailbox. Local stories are subscribed to through the `inbox`, which ends up containing all messages the user receives.
187+Some places, like `urbit-meta`, will want to be easily accessible to everyone on the network, and actually support the load that comes with that. Both these desired can be met through federation.
189+In federation, stories and any changes to them trickle down from ~zod, to other galaxies, to their stars. Only changes to messages and presence can also move upstream, from stars, to galaxies, to ~zod (and then back down again).
190+This way, all planets and comets will be able to subscribe to `/urbit-meta` (that is, the `urbit-meta` circle on their parent ship) and get all updates on the story, regardless of where on the network they originated.
192+### Public membership
194+To aid in discoverability of circles, it is possible for users to make their participation in any given circle public. This data can then be used by things like profile widgets, or read out directly by other users.
197+## Interfaces for applications
199+Applications can interact with Hall in two complementary ways: they can tell it what actions to perform, and they can subscribe to its state and the changes made to it.
203+Hall can be commanded by poking it with `action`s.
206+++ action :> user action
207+ $% :: circle configuration ::
208+ {$create nom/term des/cord sec/security} :< create circle
209+ {$delete nom/term why/(unit cord)} :< delete + announce
210+ {$depict nom/term des/cord} :< change description
211+ {$filter nom/term fit/filter} :< change message rules
212+ {$permit nom/term inv/? sis/(set ship)} :< invite/banish
213+ {$source nom/term sub/? srs/(set source)} :< un/sub to/from src
215+ {$convey tos/(list thought)} :< post exact
216+ {$phrase aud/audience ses/(list speech)} :< post easy
217+ :: personal metadata ::
218+ {$notify aud/audience pes/(unit presence)} :< our presence update
219+ {$naming aud/audience man/human} :< our name update
220+ :: changing shared ui ::
221+ {$glyph gyf/char aud/audience bin/?} :< un/bind a glyph
222+ {$nick who/ship nic/cord} :< new identity
224+ {$public add/? cir/circle} :< show/hide membership
228+The largest part of these actions concern themselves with managing local circles. `nom` is always the name used to identify the relevant story. To disambiguate between "add" and "delete" type actions (for permissions and subscriptions), a loob `?` is used.
230+To send messages, two interfaces are available. `%convey` lets you specify all details of the messages, including its timestamp, serial, and fully assembled audience. `%phrase`, on the other hand, takes care of that for you, allowing you to specify just the target partners and message contents.
232+`%notify` and `%naming` are useful for setting your own presence and nickname in a circle respectively, so others can see if you're active and what to call you by.
234+"Shared UI" encompasses UI data that should be consistent across applications. For example, if the user sets a local nickname for an identity, they expect to see that nickname regardless of the application they're currently using. The same goes for glyph bindings, for easy audience targeting.
236+### Queries, prizes and rumors
238+To receive data from Hall, applications will have to query it. There are two paths that are useful for clients to query. Let's look at them and their results.
241+++ query :> query paths
242+ $% {$client $~} :< shared ui state
243+ $: $circle :> story query
244+ nom/naem :< circle name
245+ wat/(set circle-data) :< data to get
246+ ran/range :< query duration
250+++ circle-data :> kinds of circle data
251+ $? $grams :< messages
252+ $group-l :< local presence
253+ $group-r :< remote presences
254+ $config-l :< local config
255+ $config-r :< remote configs
257+++ range :> inclusive msg range
258+ %- unit :< ~ means everything
259+ $: hed/place :< start of range
260+ tal/(unit place) :< opt end of range
262+++ place :> range indicators
263+ $% {$da @da} :< date
264+ {$ud @ud} :< message number
266+++ prize :> query result
267+ $% $: $client :< /client
268+ gys/(jug char (set partner)) :< glyph bindings
269+ nis/(map ship cord) :< nicknames
271+ {$circle burden} :< /circle
274+++ burden :< full story state
275+ $: gaz/(list telegram) :< all messages
276+ cos/lobby :< loc & rem configs
277+ pes/crowd :< loc & rem presences
281+To be clear, the paths for those queries are `/client` and `/circle/[name]/[data]/[start]/[end]`, where data is at least one `circle-data`, and `start` and `end` are optional and can be either a date or a message number.
282+`/client` queries produce shared UI state. That is, glyph bindings and nicknames.
283+`/circle` queries produce the entire public state of the requested story, including remotes.
285+Any changes to the results of these queries are communicated via the following `rumor`s. It's a fairly long list of potential changes,
288+++ rumor :< query result change
289+ $% {$client rum/rumor-client} :< /client
290+ {$circle rum/rumor-story} :< /circle
293+++ rumor-client :< changed ui state
294+ $% {$glyph diff-glyph} :< un/bound glyph
295+ {$nick diff-nick} :< changed nickname
297+++ diff-glyph {bin/? gyf/char aud/audience} :< un/bound glyph
298+++ diff-nick {who/ship nic/cord} :< changed nickname
299+++ diff-story :> story change
300+ $% {$new cof/config} :< new story
301+ {$config cir/circle dif/diff-config} :< new/changed config
302+ {$status cir/circle who/ship dif/diff-status} :< new/changed status
303+ {$remove $~} :< removed story
306+++ rumor-story ::> story rumor
307+ $? diff-story ::< both in & outward
308+ $% {$gram nev/envelope} ::< new/changed msgs
310+++ diff-config :> config change
311+ $% {$full cof/config} :< set w/o side-effects
312+ {$source add/? src/source} :< add/rem sources
313+ {$caption cap/cord} :< changed description
314+ {$filter fit/filter} :< changed filter
315+ {$secure sec/security} :< changed security
316+ {$permit add/? sis/(set ship)} :< add/rem to b/w-list
317+ {$remove $~} :< removed config
319+++ diff-status :> status change
320+ $% {$full sat/status} :< fully changed status
321+ {$presence pec/presence} :< changed presence
322+ {$human dif/diff-human} :< changed name
323+ {$remove $~} :< removed status
325+++ diff-human :> name change
326+ $% {$full man/human} :< fully changed name
327+ {$handle han/(unit cord)} :< changed handle
328+ {$true tru/(unit truename)} :< changed true name
332+These are, as enforced by Gall, the changes as they happen. This makes it possible for very minimal Hall clients to be implemented that only display a stream of changes, rather than keeping state themselves.
335+## Communication between Halls
337+Halls can communicate with other Halls to send commands and subscription updates.
341+To request changes to a story, Halls can send `command`s.
344+++ command :> effect on story
345+ $% {$publish tos/(list thought)} :< deliver
346+ {$present nos/(set term) dif/diff-status} :< status update
351+A `%publish` command contains messages. These are sent to foreign Halls to be published to their stories.
353+The `%present` command is used for sending status updates about our ship.
357+Halls can query other halls. Here's the peer move we send when we open a query on a foreign circle:
362+ /[our-circle]/[host]/[query-path] :< rumor path
363+ [host %hall] :< query target
364+ /circle/[their-circle]/[data]/[start]/[end] :< query path
368+Again, the rumor path is what our Hall uses to identify what query a received message originates from. Our circle is specified so that we know what story is interested in the changes we get.
369+The query path is slightly more interesting. Of course it specifies the name of their story we want to subscribe our story to, but also a "start" and "end". These can (optionally) be used to specify the range of messages we want to get from our query. Once that range has passed, we stop receiving updates.
373+Along the way so far, we've skipped some parts of structures. Most of these relate to federation. Let's see what we missed.
376+++ query :> query paths
377+ $% {$burden who/ship} :< duties to share
378+ {$report $~} :< duty reports
381+++ prize :> query result
382+ $% {$burden sos/(map term burden)} :< /burden
385+++ rumor :< query result change
386+ $% {$burden nom/term rum/rumor-story} :< /burden
390+ $% {$bear bur/burden} :< new inherited story
393+++ burden :> full story state
394+ $: gaz/(list telegram) :< all messages
395+ cos/lobby :< loc & rem configs
396+ pes/crowd :< loc & rem presences
400+We briefly touched upon how federation works on the higher level. On the lower, this is how things go down.
402+1. When Hall boots on a star or galaxy, it starts querying its parent's `/burden` path. (Galaxies query ~zod.)
403+2. Upon receiving that query, the parent sends a `%burden` prize containing all state of its channels (fully public circles) to the child. It also subscribes to `/report` on that child.
404+3. When a new story gets created on the parent, its children get a `%burden` rumor with a `%bear` story diff, and they create a local copy of that channel.
405+4. When something about a parent's story changes, a `%burden` rumor is sent to all its children (because they're querying `/burden`). The children apply this change to their local version of the story.
406+5. When a child's story has a `%grams` or `%status` change, a `%burden` rumor is sent to the parent (because they're querying `/report`). The parent applies this change to their local version of the story, and #4 happens.
409+## Hall implementation
411+Now that we know what gets sent over the wire and why, let's see what happens whenever such events happen. We won't be covering everything, but enough to illustrate the general flow.
413+First though, it's useful to understand how Hall's code is structured. It consists of primary cores, designed to work optimally with Gall.
415+* `++ta` is the transaction core. For every event that happens, be it a poke, peer or diff, the transaction core gets put to work to process it. Once it's done, it produces a list of deltas that describe how our state needs to be modified.
416+* `++da` is the delta application core. It is used to apply `++ta`'s deltas to our application state, and produce a list of side-effects.
418+Both those cores also have a core within themselves dedicated solely to dealing with stories, `++so` and `++sa` respectively.
420+Once all work in an engine core has finished, its `-done` arm is called to produce the result of all the core's computation.
422+In the diagrams that follow, regular lines indicate flows that always happen. Dashed lines indicate flows that are traversed if the described condition is met.
423+Below the diagrams you will find brief explanations of the involved arms. In a particularly bright future, these might be available on-hover instead.
424+For brevity, utility arms that aren't a direct and important part of the flow have been omitted.
426+(You will find that the origins still use old Gall arms, with some glue code in between. At the time of writing, new Gall has not yet been implemented, so this will have to do.)
428+We will only display the delta generation parts of the flow. Delta application is usually implemented simply enough to deduce what it does from looking at the label.
432+Hall-to-Hall subscriptions happen when a source gets added to a story. This is done with a `%source` action, and results in a `%peer` move being sent (prompted by a `%story %follow` delta).
433+Upon receiving a valid peer on a `/circle` path, the subscribing ship is added to that circle's presence map.
435+
438+++ poke-talk-action :< we got poked with an action.
439+ ++ ta-action :< applies an action.
440+ ++ ta-config :< (re)configures a story.
442+++ peer :< we got peered.
443+ ++ g-query :< resolves query to prize.
444+ ++ ta-subscribe :< act upon subscription.
445+ ++ so-attend :< adds a ship's status to the story.
450+Once a query has opened, Hall will receive updates on it, rumors. The changes describes in these rumors are applied via the following flow.
452+
455+++ diff-talk-rumor :< we got a query update.
456+ ++ ta-hear :< applies a rumor to the story it's intended for.
457+ ++ so-hear :< applies a %circle rumor to the story.
458+ ++ so-config-full :< splits a %full diff-config into separate changes.
459+ ++ so-bear :< accept burden, assimilate into the story's state.
460+ ++ so-lesson :< sends a report with presences.
465+To send messages, the user sends a `%convey` or `%phrase` action, resulting in a `%publish` command being sent to the involved partners. Receiving a `%publish` command causes its messages to be added to the story through the creation of a `%story %grams` delta.
467+
470+++ poke-talk-action :< we got poked with an action.
471+ ++ ta-action :< applies an action.
473+++ poke-talk-command :< we get poked with a command.
474+ ++ ta-apply :< applies a command.
475+ ++ ta-think :< consumes each message in the given list.
476+ ++ ta-consume :< conducts a message to each partner in audience.
477+ ++ ta-conduct :< records or sends a message.
478+ ++ ta-transmit :< sends a message to a partner.
479+ ++ ta-record :< stores a message in a story.
480+ ++ so-learn :< either adds or modifies a message.
486+Hall is neither complete nor perfect. There are features that need to be implemented, varying from small quality of life changes to broad-reaching functionality.
488+### Features and functionality
490+To turn Talk into a [sustainable social platform](https://urbit.org/fora/posts/~2017.4.26..18.00.25..b93c~/), it needs a number of things.
492+Primarily, discoverability of circles. A possible solution for realizing this is making it possible for users to add friends, which would have their Hall subscribe to a list of circles the added friends are in. The existing public membership functionality can be leveraged for this.
494+Eventually, content self-moderation might need to be implemented. Users would flag their content if it is potentially offensive. Others would easily be able to filter out content that might offend them.
495+This functionality brings its own large sets of challenges that need to be tackled.
497+Additionally, federation might be put to broader use. Knowing a circle is federated could help fall back to alternative hosts in case the one Hall originally subscribed to is unavailable.
498+Implementation-wise, care would need to be taken for this to not get ugly. Not to mention, when/how would availability of a host be verified?
500+Other changes that might have a fair impact on the functioning of Hall are un/read states for messages, and being able to use moons to allow your friends to chat with you in a local channel.
502+Smaller functionality that still needs to be implemented includes:
503+* Extended permissions management. Being able to black-/whitelist entire ship classes.
504+* Improved presence functionality. Actually using the presence system, and sending "typing", "idle", etc. statuses.
507+And of course there's a lot Talk, the client, can improve on as well.
512+To gain a more thorough understanding of Hall's inner workings, take a look at its source code. It comes with inline documentation.
513+[On Github.](https://github.com/urbit/arvo/blob/master/app/hall.hoon)
515+To see an expansive example of a Hall client, take a look at the code of Talk. It, too, comes with inline documentation.
516+[On Github.](https://github.com/urbit/arvo/blob/master/app/talk.hoon)