2title = "Sample Application: Library"
6insert_anchor_links = "right"
9Library is a sample social media application in which you can create a collection of books, called a library, which can contain any number of books.
10You can share individual collections per-ship. The creator of the collection has de-facto admin powers; he is the only one who can add or remove books to/from the library,
11remove the library itself, and add comments or remove anyone's comments. Guest ships may request access to specific libraries, which if granted,
12allows them to request any book from the library.
14Using this application, you can:
16- Create libraries, where a library is a collection of books
17- Add and remove books from a library, if you are the owner
18- Allow others to view your library based on various permissioning schemes (policies)
19- Add and remove comments from a library, if you are the owner or were granted access
23This overview is a companion document to the _Library_ application, and assumes you are familiar with `%graph-store`. Before continuing, **you must familiarize yourself with the sample application**, which can be found [here](https://github.com/ynx0/library). See the [detailed usage guide](https://github.com/ynx0/library/blob/master/README.md) for download and installation instructions.
27Here is the directory structure of our app.
31│ └── library-proxy.hoon
47- The main code of the application lives in [`app/library-proxy.hoon`](https://github.com/ynx0/library/blob/master/app/library-proxy.hoon). This contains the gall agent which proxies the `%graph-store` updates between ships.
49- [`lib/library.hoon`](https://github.com/ynx0/library/blob/master/lib/library.hoon) contains miscellaneous helper arms which the proxy uses extensively. It mainly contains arms that construct different `%graph-store` updates various actions that a user performed.
51- The `mar/library` folder contains the definitions of the pokes that `%library-proxy` uses.
53- Contained in [`mar/graph/validator/library.hoon`](https://github.com/ynx0/library/blob/master/mar/graph/validator/library.hoon) is the definition of the validation logic that `%graph-store` uses to enforce the schema of the library applications graph data.
55- [`sur/library.hoon`](https://github.com/ynx0/library/blob/master/sur/library.hoon) contains all the various type definitions used by `%library-store`.
57- [`install.sh`](https://github.com/ynx0/library/blob/master/install.sh) is a script that automates copying the source files into a ship's pier / home desk.
59## Implementation Summary
61The project contains a custom `%graph-store` validator tht specifies the application's schema and a custom gall agent, called `%library-proxy`, which sends `%graph-store` updates between the host ship and subscribers.
62The gall agent also generates `%graph-store` updates based on commands and actions that a user issues. Commands can only be issued to a proxy by the ship owner,
63while actions can be issued by any ship, but may fail based on permissions. A host ship's `%library-proxy` talks to the subscriber ship's `%library-proxy` through the normal gall app channels (pokes/peeks)
64to send them any updates that have taken place on the library graph. It is the host's responsibility to forward all relevant updates to subscribers
65(kept track of in a data structure defined [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/sur/library.hoon#L14-L15)),
66while subscribers must only trust graph updates that it receives from the owner of that resource.
68Here are the proxy's responsibilities:
70- **Application-specific API** - presents an interface for a user to interact with both his/her own library proxy and others' by directly defining and implementing a user-facing API as pokes and scries,
71 as opposed to forcing the user to deal with the graph-store API directly. The `command`/`action`/`response` poke types are defined [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/sur/library.hoon#L17-L42),
72 while the scries are defined [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L132).
73- **Graph store update creation** - creates the appropriate graph update for a given user-facing action
74- **Graph store update proxying** - handles the networking and subscription logic required to send graph store updates between host and subscriber
75- **Access control** - allows or denies ships access to libraries, checks for proper permissions before processing a `command` or `action`
77## Interaction with `%graph-store`
79There are a couple of ways that our application uses and interacts with `%graph-store`.
80Firstly, the app subscribes to `%graph-store` in `+on-init` on path `/updates`, meaning it will be notified of every single `graph-update` that occurs on the local ship. (_Recall the type of `graph-update`, defined [here](https://github.com/urbit/urbit/blob/ceed4b78d068d7cb70350b3cd04e7525df1c7e2d/pkg/arvo/sur/graph-store.hoon#L29), the reference for which can be found [here](https://urbit.org/docs/userspace/graph-store/data-structure-overview/#update-part-1)_). We also define a validator for our application that `%graph-store` uses for every _Library_ graph. Since `%graph-store` registers all validators under the path `%/graph/validator/<name of app>/hoon`, we don't have to do anything special to register our validator. We just need to correctly name our file, and provide an `+graph-indexed-post` arm which should perform all schema validation.
82### Synchronizing Graphs Between Ships
84An owner is responsible for forwarding any updates to clients. Whenever the `%library-proxy` gall agent receives an update from it's local `%graph-store`, it checks to see whether it's for a graph it owns or not. If it's not, we skip sending out updates since we don't own the resource\*. If it is, then we generate cards to poke each subscriber with that same `graph-update`. This logic occurs within the agent's `+on-watch` arm, with the logic residing in the `handle-outgoing-graph-update` arm, found [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L345).
86On the receiving end, since we know that we are not the source of the `graph-update`, we handle the update in the `handle-incoming-graph-store` helper arm, which makes sure to only process and forward graph store updates to local graph store that are sent by the owner, and no one else. Once it passes this permissions check, it is poked into the local graph like any other graph-update.
88\*_One could imagine a gossip style protocol where the data is like a fact so it doesn't matter where it came from, but in this app the library host is defined as the source of truth._
92There are two main application-side data structures that we define.
93The first type is [`book`](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/sur/library.hoon#L4-L9) which contains a title and an isbn.
94The second type is a [`comment`](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/sur/library.hoon#L3), which is just a simple string of text,
95meant to represent a comment on any given book.
97### Representing Data using `%graph-store`
99The data types we defined for our application do not fit within `%graph-store` out of the box. `%graph-store` doesn't allow arbitrarily typed data in a node's content field, so we create an ad-hoc representation that we can cleanly convert to and from our own data types and `%graph-store` types.
101Here is the conversion code: [(link)](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/lib/library.hoon#L12-L19)
107 ~[[%text title.book] [%text isbn.book]]
108++ make-comment-contents
109 |= [comment-text=comment:library]
111 ~[[%text comment-text]]
114Each arm takes in either a `book` or a `comment`, our application specific data type, and spits out a `(list content)`, which is the type that a `node` uses to store its content.
116### Organizing Data within `%graph-store`
118We define 3 core structures:
120- **Library** - The fundamental data structure. A library is a graph that contains books
121- **Book** - A data structure that contains an entry of a given book's metadata (i.e. title and isbn) and any comments associated with it. It is represented by a top-level node within a library graph
122- **Comment** - A node that represents a user's comment on a given book. Represented as a child node of a book's comment container node.
124We then define the layout of the data as follows:
128 - Metadata Revision Container
129 - Specific Metadata Revision
133Every graph created with the `%graph-validator-library` is a _Library_ graph, and thus it has the following characteristics:
135- Every Library graph represents a library, or a collection of books
136- Every top level node in a library represents a book and all related contents
137- Within a book there are always two structural nodes: a container node for metadata revisions, and a container node that holds all comments, modeling a comments section
139To make things more concrete, let's look at an example of an actual graph.
141(_For the sake of clarity, we have replaced each index with its representation in its original aura in following output._)
144[ p=[entity=~zod name=%library1]
147 { [ key=~2021.6.27..14.18.57..b3dc
153 index=~[~2021.6.27..14.18.57..b3dc]
154 time-sent=~2021.6.27..14.18.57..b3dc
169 index=~[~2021.6.27..14.18.57..b3dc %comments]
170 time-sent=~2021.6.27..14.18.57..b3dc
179 { [ key=~2021.6.27..14.20.00.b5fc
185 index=~[~2021.6.27..14.18.57..b3dc %meta ~2021.6.27..14.20.00.b5fc]
186 time-sent=~2021.6.27..14.20.00.b5fc
187 contents=~[[%text text='dune is pretty good']]
205 index=~[~2021.6.27..14.18.57..b3dc %meta]
206 time-sent=~2021.6.27..14.18.57..b3dc
221 index=~[~2021.6.27..14.18.57..b3dc %meta 1]
222 time-sent=~2021.6.27..14.18.58..a4ed
223 contents=~[[%text text='Dune'] [%text text='0441172717']]
237 index=~[~2021.6.27..14.18.57..b3dc %meta 1]
238 time-sent=~2021.6.27..14.18.57..b3dc
239 contents=~[[%text text='Dune......'] [%text text='0444444444']]
256 q=[~ %graph-validator-library]
261
263- The above graph represents a library that has a single book, whose metadata is stored under the `%meta` revision container. The metadata associated with this specific book entry is currently with title: `"Dune"` and isbn `"0441172717"`.
264 The reason we have a revision container for the book metadata is so that in case someone makes an error, they may correct it. A potential frontend could then simply show the most recent version.
265- Each comment is simply a node under a book's `%comment` node with a single `%text` content.
266 We skipped revision containers for comments to keep schema simple, but you could imagine duplicating the same logic for comments as well.
267- Every book node has an index the datetime of when it was created.
268- Each structural node in a book has a constant index fragment, either `%meta` for the metadata revision container node,
269 or `%comments` for the comments container node.
270- Every comment has an index of the datetime of when it was posted.
271- The metadata revisions are a single incrementing number starting from 1, so the initial metadata has a revision count of `1`, and a subsequent edit has a revision count of `2`, and so on.
273### Schema Enforcement
275Recall that when using `%graph-store`, all data that is to be added to a graph passes through our [custom validator](https://github.com/ynx0/library/blob/master/mar/graph/validator/library.hoon).
276Take a moment to read through it and cross-check your understanding with the following summary.
278**Validator logic summary**
280- None of the structural nodes require data, so we simply assert that their contents are empty
281- For a _Specific Comment Node_, a post with an index matching the structure `[@ %comments @ ~]`, we assert that `contents` only contains a single `%text` content.
282 This choice is arbitrary but meant to keep the application simple and so that a potential frontend does not have to do complex rendering.
283- For a _Specific Metadata Revision_, a post with index matching the structure `[@ %meta @ ~]`, we first ensure that it's contents only contain two `%text` content instances.
284 We then use two helper arms `+is-title-valid` and `+is-isbn-valid` to validate the two values. For us, any title value is accepted, while only strings with length 10 or 13 are valid ISBNs.
285 (We don't do any true ISBN validation for simplicity's sake)
291There are explicit access control rules called `policy`s, (defined [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/sur/library.hoon#L45-L49))
292which are set by the user per-library at the time of creation (stored [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L179)).
293These specify who can or cannot gain access to a library.
295Here's what `policy` looks like:
299 $% [%open ~] :: any ship is allowed
300 [%children ~] :: any children (i.e. moons) are allowed
301 [%whitelist ships=(set ship)] :: any ships in the provided set are allowed
305There are also implicit rules, defined as follows:
307For any given library that one owns,
310 - add a top level book node to any library
311 - add a metadata revision to any book
312 - add or remove any comment on any book
313- A reader, that is, a another ship granted access to the library, can:
314 - add a comment to any book that they have requested
315 - remove any comment that they are the author of
317Implicitly, all readers are given permission to get any book when granted access to the library.
321- `policies` is a `(map @tas policy)`. That is, a map between the names of libraries that we own and what policy should be enforced on each one and is a part of the agent state (shown [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L13)).
323- The `+is-allowed` arm, (found [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/lib/library.hoon#L21))
324 implements each `policy`'s behavior, and is reproduced below.
328 |= [requester=ship host=ship =policy:library]
330 ?: =(requester host) :: host is always allowed
334 %children (team:title host requester)
335 %whitelist (~(has in ships.policy) requester)
339 Given the host ship of a resource and the policy to be enforced for a resource, `+is-allowed` returns a boolean for whether or not a requesting ship should be allowed access to that resource.
341- `+is-allowed` is used in `+on-watch` [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L113), where it only allows a ship to subscribe to a library if it passes the permissions check.
342- It is also used [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L299) and [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L310), so that when a ship wants to know about what libraries and books exist, only data they are allowed to see gets revealed to them.
343- Some of the more ad-hoc/implicit permission rules are implemented at the following locations: [[1](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L318) [2](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L48) [3](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L256) [4](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L299)]
345## Alternative Methods
347There are a number of architectural decisions made in our application that may not be suitable for other projects.
348Many decisions made such as the definition of the schema, the user-facing and inter-proxy API, and the permissioning model could be done differently.
349For instance, many decisions in the schema and thus the validator were made to keep the schema simple. One could imagine allowing any content type and amount,
350while right now it is restricted to a single text content, and stricter validation of or a custom for the ISBN.
351The subscription model is also just one way to handle subscriptions and by no means the only one.
352The permissioning scheme was also implemented relatively simple; it has some limitations, like the fact that you cannot change the permissions after creating the a library, or the fact that there are only 3 policy types.
354### Comparison between `%library-proxy` and `%graph-push-hook`
356`%library-proxy` performs the functions of both `%graph-push-hook` and `%graph-pull-hook`.
357`%graph-pull-hook` is responsible for asking other `%graph-push-hook`s for graph data
358by trying to initiate a subscription on the host ship's `%graph-push-hook`, which the host only allows if they have permissions.
359When it succeeds, `%graph-push-hook` then sends out all `graph-updates` to those subscribers who need to hear about it.
360`%graph-pull-hook` then merges in graph updates it hears into the local `%graph-store`.
361So far, all the functionality is in line with `%library-proxy`.
363The two main places where the hooks differ is in their choice of permissioning model and subscription model.
364To know whether a person is allowed to access a resource, pull hook uses checks to see whether they are a member of a group, while `%library-proxy` uses its own policy type.
365For subscriptions, the hooks do it per-resource, (i.e. one subscription path per resource, many ships subscribe to that),
366while library proxy incorporates both the resource and the ship in its path, making the subscription model per-resource per-ship.
368In general, if your application's use case does not fall neatly with the permissioning and subscription model used by the hooks, you'll need to make your own proxy agent.
369Otherwise, you will likely want to use the existing hooks and save yourself the trouble of reimplementing all of the functionality that they offer.
keysesc to close
- j k
- move down, up
- enter
- open
- esc
- back, or up a level
- 1 2 3 4
- readme, files, commits, refs
- /
- filter
- y
- copy the clone command
- c
- compare the chosen commit with another (or shift-click both)
- ?
- show or hide this list