@@ -0,0 +1,369 @@
1++++
2+title = "Sample Application: Library"
3+weight = 4
4+sort_by = "weight"
5+template = "doc.html"
6+insert_anchor_links = "right"
7++++
8+
9+
10+
11+Library is a sample social media application in which you can create a collection of books, called a library, which can contain any number of books.
12+You can share individual collections per-ship. The creator of the collection has de-facto admin powers; he is the only one who can add or remove books to/from the library,
13+remove the library itself, and add comments or remove anyone's comments. Guest ships may request access to specific libraries, which if granted,
14+allows them to request any book from the library.
15+
16+Using this application, you can:
17+
18+- Create libraries, where a library is a collection of books
19+- Add and remove books from a library, if you are the owner
20+- Allow others to view your library based on various permissioning schemes (policies)
21+- Add and remove comments from a library, if you are the owner or were granted access
22+
23+## Prerequisites
24+
25+This overview is a companion document to the *Library* application, and assumes you are familiar with `%graph-store`. Before continuing, **you must familiarize yourself with the sample application**, which can be found [here](https://github.com/ynx0/library). See the [detailed usage guide](https://github.com/ynx0/library/blob/master/README.md) for download and installation instructions.
26+
27+
28+## Project Structure
29+
30+Here is the directory structure of our app.
31+```
32+├── app
33+│ └── library-proxy.hoon
34+├── lib
35+│ └── library.hoon
36+├── mar
37+│ ├── graph
38+│ │ └── validator
39+│ │ └── library.hoon
40+│ └── library
41+│ ├── action.hoon
42+│ ├── command.hoon
43+│ └── response.hoon
44+├── sur
45+│ └── library.hoon
46+└── install.sh
47+```
48+
49+- The main code of the application lives in [`app/library-proxy.hoon`](https://github.com/ynx0/library/blob/master/app/library-proxy.hoon). This contains the gall agent which proxies the `%graph-store` updates between ships.
50+
51+- [`lib/library.hoon`](https://github.com/ynx0/library/blob/master/lib/library.hoon) contains miscellaneous helper arms which the proxy uses extensively. It mainly contains arms that construct different `%graph-store` updates various actions that a user performed.
52+
53+- The `mar/library` folder contains the definitions of the pokes that `%library-proxy` uses.
54+
55+- Contained in [`mar/graph/validator/library.hoon`](https://github.com/ynx0/library/blob/master/mar/graph/validator/library.hoon) is the definition of the validation logic that `%graph-store` uses to enforce the schema of the library applications graph data.
56+
57+- [`sur/library.hoon`](https://github.com/ynx0/library/blob/master/sur/library.hoon) contains all the various type definitions used by `%library-store`.
58+
59+- [`install.sh`](https://github.com/ynx0/library/blob/master/install.sh) is a script that automates copying the source files into a ship's pier / home desk.
60+
61+
62+## Implementation Summary
63+
64+The project contains a custom `%graph-store` validator tht specifies the application's schema and a custom gall agent, called `%library-proxy`, which sends `%graph-store` updates between the host ship and subscribers.
65+The gall agent also generates `%graph-store` updates based on commands and actions that a user issues. Commands can only be issued to a proxy by the ship owner,
66+while actions can be issued by any ship, but may fail based on permissions. A host ship's `%library-proxy` talks to the subscriber ship's `%library-proxy` through the normal gall app channels (pokes/peeks)
67+to send them any updates that have taken place on the library graph. It is the host's responsibility to forward all relevant updates to subscribers
68+(kept track of in a data structure defined [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/sur/library.hoon#L14-L15)),
69+while subscribers must only trust graph updates that it receives from the owner of that resource.
70+
71+Here are the proxy's responsibilities:
72+
73+- **Application-specific API** - presents an interface for a user to interact with both his/her own library proxy and others' by directly defining and implementing a user-facing API as pokes and scries,
74+ as opposed to forcing the user to deal with the graph-store API directly. The `command`/`action`/`response` poke types are defined [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/sur/library.hoon#L17-L42),
75+ while the scries are defined [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L132).
76+- **Graph store update creation** - creates the appropriate graph update for a given user-facing action
77+- **Graph store update proxying** - handles the networking and subscription logic required to send graph store updates between host and subscriber
78+- **Access control** - allows or denies ships access to libraries, checks for proper permissions before processing a `command` or `action`
79+
80+## Interaction with `%graph-store`
81+
82+There are a couple of ways that our application uses and interacts with `%graph-store`.
83+Firstly, the app subscribes to `%graph-store` in `+on-init` on path `/updates`, meaning it will be notified of every single `graph-update` that occurs on the local ship. (*Recall the type of `graph-update`, defined [here](https://github.com/urbit/urbit/blob/ceed4b78d068d7cb70350b3cd04e7525df1c7e2d/pkg/arvo/sur/graph-store.hoon#L29), the reference for which can be found [here](https://urbit.org/docs/userspace/graph-store/data-structure-overview/#update-part-1)*). We also define a validator for our application that `%graph-store` uses for every *Library* graph. Since `%graph-store` registers all validators under the path `%/graph/validator/<name of app>/hoon`, we don't have to do anything special to register our validator. We just need to correctly name our file, and provide an `+graph-indexed-post` arm which should perform all schema validation.
84+
85+### Synchronizing Graphs Between Ships
86+
87+An owner is responsible for forwarding any updates to clients. Whenever the `%library-proxy` gall agent receives an update from it's local `%graph-store`, it checks to see whether it's for a graph it owns or not. If it's not, we skip sending out updates since we don't own the resource\*. If it is, then we generate cards to poke each subscriber with that same `graph-update`. This logic occurs within the agent's `+on-watch` arm, with the logic residing in the `handle-outgoing-graph-update` arm, found [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L345).
88+
89+On the receiving end, since we know that we are not the source of the `graph-update`, we handle the update in the `handle-incoming-graph-store` helper arm, which makes sure to only process and forward graph store updates to local graph store that are sent by the owner, and no one else. Once it passes this permissions check, it is poked into the local graph like any other graph-update.
90+
91+\**One could imagine a gossip style protocol where the data is like a fact so it doesn't matter where it came from, but in this app the library host is defined as the source of truth.*
92+
93+## Data Model
94+
95+There are two main application-side data structures that we define.
96+The first type is [`book`](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/sur/library.hoon#L4-L9) which contains a title and an isbn.
97+The second type is a [`comment`](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/sur/library.hoon#L3), which is just a simple string of text,
98+meant to represent a comment on any given book.
99+
100+### Representing Data using `%graph-store`
101+
102+The data types we defined for our application do not fit within `%graph-store` out of the box. `%graph-store` doesn't allow arbitrarily typed data in a node's content field, so we create an ad-hoc representation that we can cleanly convert to and from our own data types and `%graph-store` types.
103+
104+Here is the conversion code: [(link)](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/lib/library.hoon#L12-L19)
105+
106+```
107+++ make-meta-contents
108+ |= [=book:library]
109+ ^- (list content)
110+ ~[[%text title.book] [%text isbn.book]]
111+++ make-comment-contents
112+ |= [comment-text=comment:library]
113+ ^- (list content)
114+ ~[[%text comment-text]]
115+```
116+
117+Each arm takes in either a `book` or a `comment`, our application specific data type, and spits out a `(list content)`, which is the type that a `node` uses to store its content.
118+
119+### Organizing Data within `%graph-store`
120+
121+We define 3 core structures:
122+
123+- **Library** - The fundamental data structure. A library is a graph that contains books
124+- **Book** - A data structure that contains an entry of a given book's metadata (i.e. title and isbn) and any comments associated with it. It is represented by a top-level node within a library graph
125+- **Comment** - A node that represents a user's comment on a given book. Represented as a child node of a book's comment container node.
126+
127+We then define the layout of the data as follows:
128+
129+* Library
130+ - Book
131+ + Metadata Revision Container
132+ * Specific Metadata Revision
133+ + Comments Container
134+ * Specific Comment
135+
136+Every graph created with the `%graph-validator-library` is a *Library* graph, and thus it has the following characteristics:
137+
138+- Every Library graph represents a library, or a collection of books
139+- Every top level node in a library represents a book and all related contents
140+- Within a book there are always two structural nodes: a container node for metadata revisions, and a container node that holds all comments, modeling a comments section
141+
142+To make things more concrete, let's look at an example of an actual graph.
143+
144+(*For the sake of clarity, we have replaced each index with its representation in its original aura in following output.*)
145+
146+```
147+[ p=[entity=~zod name=%library1]
148+ q
149+ [ p
150+ { [ key=~2021.6.27..14.18.57..b3dc
151+ val
152+ [ post
153+ [ %.y
154+ p
155+ [ author=~zod
156+ index=~[~2021.6.27..14.18.57..b3dc]
157+ time-sent=~2021.6.27..14.18.57..b3dc
158+ contents=~
159+ hash=~
160+ signatures={}
161+ ]
162+ ]
163+ children
164+ [ %graph
165+ p
166+ { [ key=%comments
167+ val
168+ [ post
169+ [ %.y
170+ p
171+ [ author=~zod
172+ index=~[~2021.6.27..14.18.57..b3dc %comments]
173+ time-sent=~2021.6.27..14.18.57..b3dc
174+ contents=~
175+ hash=~
176+ signatures={}
177+ ]
178+ ]
179+ children
180+ [ %graph
181+ p
182+ { [ key=~2021.6.27..14.20.00.b5fc
183+ val
184+ [ post
185+ [ %.y
186+ p
187+ [ author=~pub
188+ index=~[~2021.6.27..14.18.57..b3dc %meta ~2021.6.27..14.20.00.b5fc]
189+ time-sent=~2021.6.27..14.20.00.b5fc
190+ contents=~[[%text text='dune is pretty good']]
191+ hash=~
192+ signatures={}
193+ ]
194+ ]
195+ children=[%empty ~]
196+ ]
197+ ]
198+ }
199+ ]
200+ ]
201+ ]
202+ [ key=%meta
203+ val
204+ [ post
205+ [ %.y
206+ p
207+ [ author=~zod
208+ index=~[~2021.6.27..14.18.57..b3dc %meta]
209+ time-sent=~2021.6.27..14.18.57..b3dc
210+ contents=~
211+ hash=~
212+ signatures={}
213+ ]
214+ ]
215+ children
216+ [ %graph
217+ p
218+ { [ key=2
219+ val
220+ [ post
221+ [ %.y
222+ p
223+ [ author=~zod
224+ index=~[~2021.6.27..14.18.57..b3dc %meta 1]
225+ time-sent=~2021.6.27..14.18.58..a4ed
226+ contents=~[[%text text='Dune'] [%text text='0441172717']]
227+ hash=~
228+ signatures={}
229+ ]
230+ ]
231+ children=[%empty ~]
232+ ]
233+ ]
234+ [ key=1
235+ val
236+ [ post
237+ [ %.y
238+ p
239+ [ author=~zod
240+ index=~[~2021.6.27..14.18.57..b3dc %meta 1]
241+ time-sent=~2021.6.27..14.18.57..b3dc
242+ contents=~[[%text text='Dune......'] [%text text='0444444444']]
243+ hash=~
244+ signatures={}
245+ ]
246+ ]
247+ children=[%empty ~]
248+ ]
249+ ]
250+ }
251+ ]
252+ ]
253+ ]
254+ }
255+ ]
256+ ]
257+ ]
258+ }
259+ q=[~ %graph-validator-library]
260+ ]
261+]
262+```
263+
264+
265+
266+* The above graph represents a library that has a single book, whose metadata is stored under the `%meta` revision container. The metadata associated with this specific book entry is currently with title: `"Dune"` and isbn `"0441172717"`.
267+ The reason we have a revision container for the book metadata is so that in case someone makes an error, they may correct it. A potential frontend could then simply show the most recent version.
268+* Each comment is simply a node under a book's `%comment` node with a single `%text` content.
269+ We skipped revision containers for comments to keep schema simple, but you could imagine duplicating the same logic for comments as well.
270+* Every book node has an index the datetime of when it was created.
271+* Each structural node in a book has a constant index fragment, either `%meta` for the metadata revision container node,
272+ or `%comments` for the comments container node.
273+* Every comment has an index of the datetime of when it was posted.
274+* The metadata revisions are a single incrementing number starting from 1, so the initial metadata has a revision count of `1`, and a subsequent edit has a revision count of `2`, and so on.
275+
276+### Schema Enforcement
277+
278+Recall that when using `%graph-store`, all data that is to be added to a graph passes through our [custom validator](https://github.com/ynx0/library/blob/master/mar/graph/validator/library.hoon).
279+Take a moment to read through it and cross-check your understanding with the following summary.
280+
281+**Validator logic summary**
282+
283+- None of the structural nodes require data, so we simply assert that their contents are empty
284+- For a *Specific Comment Node*, a post with an index matching the structure `[@ %comments @ ~]`, we assert that `contents` only contains a single `%text` content.
285+ This choice is arbitrary but meant to keep the application simple and so that a potential frontend does not have to do complex rendering.
286+- For a *Specific Metadata Revision*, a post with index matching the structure `[@ %meta @ ~]`, we first ensure that it's contents only contain two `%text` content instances.
287+ We then use two helper arms `+is-title-valid` and `+is-isbn-valid` to validate the two values. For us, any title value is accepted, while only strings with length 10 or 13 are valid ISBNs.
288+ (We don't do any true ISBN validation for simplicity's sake)
289+
290+## Access Control
291+
292+### Rules
293+
294+There are explicit access control rules called `policy`s, (defined [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/sur/library.hoon#L45-L49))
295+which are set by the user per-library at the time of creation (stored [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L179)).
296+These specify who can or cannot gain access to a library.
297+
298+Here's what `policy` looks like:
299+```
300++$ policy
301+ $% [%open ~] :: any ship is allowed
302+ [%children ~] :: any children (i.e. moons) are allowed
303+ [%whitelist ships=(set ship)] :: any ships in the provided set are allowed
304+ ==
305+```
306+
307+There are also implicit rules, defined as follows:
308+
309+For any given library that one owns,
310+- An owner can:
311+ + add a top level book node to any library
312+ + add a metadata revision to any book
313+ + add or remove any comment on any book
314+- A reader, that is, a another ship granted access to the library, can:
315+ + add a comment to any book that they have requested
316+ + remove any comment that they are the author of
317+
318+Implicitly, all readers are given permission to get any book when granted access to the library.
319+
320+### Implementation
321+
322+- `policies` is a `(map @tas policy)`. That is, a map between the names of libraries that we own and what policy should be enforced on each one and is a part of the agent state (shown [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L13)).
323+
324+- The `+is-allowed` arm, (found [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/lib/library.hoon#L21))
325+ implements each `policy`'s behavior, and is reproduced below.
326+ ```
327+ ++ is-allowed
328+ |= [requester=ship host=ship =policy:library]
329+ ^- ?
330+ ?: =(requester host) :: host is always allowed
331+ %.y
332+ ?- -.policy
333+ %open %.y
334+ %children (team:title host requester)
335+ %whitelist (~(has in ships.policy) requester)
336+ ==
337+ ```
338+ Given the host ship of a resource and the policy to be enforced for a resource, `+is-allowed` returns a boolean for whether or not a requesting ship should be allowed access to that resource.
339+
340+
341+- `+is-allowed` is used in `+on-watch` [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L113), where it only allows a ship to subscribe to a library if it passes the permissions check.
342+- It is also used [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L299) and [here](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L310), so that when a ship wants to know about what libraries and books exist, only data they are allowed to see gets revealed to them.
343+- Some of the more ad-hoc/implicit permission rules are implemented at the following locations: [[1](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L318) [2](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L48) [3](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L256) [4](https://github.com/ynx0/library/blob/9104145bb24e8ec949a5e9685139665b07161dd6/app/library-proxy.hoon#L299)]
344+
345+## Alternative Methods
346+
347+There are a number of architectural decisions made in our application that may not be suitable for other projects.
348+Many decisions made such as the definition of the schema, the user-facing and inter-proxy API, and the permissioning model could be done differently.
349+For instance, many decisions in the schema and thus the validator were made to keep the schema simple. One could imagine allowing any content type and amount,
350+while right now it is restricted to a single text content, and stricter validation of or a custom for the ISBN.
351+The subscription model is also just one way to handle subscriptions and by no means the only one.
352+The permissioning scheme was also implemented relatively simple; it has some limitations, like the fact that you cannot change the permissions after creating the a library, or the fact that there are only 3 policy types.
353+
354+### Comparison between `%library-proxy` and `%graph-push-hook`
355+
356+`%library-proxy` performs the functions of both `%graph-push-hook` and `%graph-pull-hook`.
357+`%graph-pull-hook` is responsible for asking other `%graph-push-hook`s for graph data
358+by trying to initiate a subscription on the host ship's `%graph-push-hook`, which the host only allows if they have permissions.
359+When it succeeds, `%graph-push-hook` then sends out all `graph-updates` to those subscribers who need to hear about it.
360+`%graph-pull-hook` then merges in graph updates it hears into the local `%graph-store`.
361+So far, all the functionality is in line with `%library-proxy`.
362+
363+The two main places where the hooks differ is in their choice of permissioning model and subscription model.
364+To know whether a person is allowed to access a resource, pull hook uses checks to see whether they are a member of a group, while `%library-proxy` uses its own policy type.
365+For subscriptions, the hooks do it per-resource, (i.e. one subscription path per resource, many ships subscribe to that),
366+while library proxy incorporates both the resource and the ship in its path, making the subscription model per-resource per-ship.
367+
368+In general, if your application's use case does not fall neatly with the permissioning and subscription model used by the hooks, you'll need to make your own proxy agent.
369+Otherwise, you will likely want to use the existing hooks and save yourself the trouble of reimplementing all of the functionality that they offer.