git.iko.soy/urbit.org
on master help sign in
Add graph store documenation
6614443Yaseen
userspace/graph-store/_index.mdnew file+23
compare with master
unifiedsplit
@@ -0,0 +1,23 @@
1++++
2+title = "Graph Store"
3+weight = 50
4+sort_by = "weight"
5+template = "sections/docs/chapters.html"
6+insert_anchor_links = "right"
7++++
8+
9+## [Overview](@/docs/userspace/graph-store/overview.md)
10+
11+Basic concepts and architectural overview of Graph Store
12+
13+## [`sur` File Walkthrough](@/docs/userspace/graph-store/sur-file-walkthrough.md)
14+
15+A walkthrough of the various data structures and types used in Graph Store.
16+
17+## [Validator Walkthrough](@/docs/userspace/graph-store/validator-walkthrough.md)
18+
19+A walkthrough of existing applications built using Graph Store
20+
21+## [Advanced Information](@/docs/userspace/graph-store/advanced-info.md)
22+
23+Advanced information and selected implementation details about Graph Store.
userspace/graph-store/advanced-info.mdnew file+36
@@ -0,0 +1,36 @@
1++++
2+title = "Advanced Info"
3+weight = 50
4+sort_by = "weight"
5+template = "doc.html"
6+insert_anchor_links = "right"
7++++
8+
9+This section is not required, but does shed light on some implicit assumptions that Graph Store makes.
10+
11+
12+### Miscellaneous
13+- Only nodes that successfully typecheck under the validator will be added to the graph
14+- Graphs are validated recursively (see [`app/graph-store.hoon#L598-L616`](https://github.com/urbit/urbit/blob/5cb6af0433a65fb28b4bc957be10cb436781392d/pkg/arvo/app/graph-store.hoon#L598-L616))
15+ - First, validate all top level nodes of a graph.
16+ - Then, recursively validate all children nodes of those nodes.
17+ - Base case: empty children is always valid
18+- Nodes are also validated against the graph's mark when inserted individually (see [`app/graph-store.hoon#L380`](https://github.com/urbit/urbit/blob/e2ad6e3e9219c8bfad62f27f05c7cac94c9effa8/pkg/arvo/app/graph-store.hoon#L380))
19+- Only the root level graphs get validated with marks. There is only one mark / validator per graph. All child graphs get validated with the same mark as the root/top-level one.
20+- The way in which Graph Store works is by mirroring all data from a given social media channel. Thus, anything you see is your copy of it, and anything you do is sent as a request to the hosting ship.
21+
22+
23+### What happens when you add or remove a node?
24+
25+When adding nodes, Graph Store takes in a flat list of nodes that have nodes whose `index`es are arbitrarily deep. It does not allow for a node which has non-existent ancestors to be added (i.e. it doesn't silently create the node). Thus, it assumes every node up until the 2nd to last index is created (meaning it needs every ancestor to exist execpt it's parent), either already existing or included in the `graph-update`. To ensure consistent behavior, all nodes are added shallowest-index first, which ensures that no child is added before it's parent (if it exists). This is why any non-leaf node must have its parent exist, either already within the graph or in the update. However, children do not have to be nested within the parent's data structure in the update. They simply have to exist somewhere in the update, even at the top level is fine. Graph Store merges the flat list of nodes from an `%add-nodes` update into the fully connected graph structure.
26+
27+Another constraint to be aware of is that in a map of indexed nodes: for each node, the index in map entry has to be the same index as the in that `node`'s post index.
28+
29+To see how Graph Store handles add-nodes, take a look at [`app/graph-store.hoon#L370-L417`](https://github.com/urbit/urbit/blob/e2ad6e3e9219c8bfad62f27f05c7cac94c9effa8/pkg/arvo/app/graph-store.hoon#L370-L417).
30+
31+
32+### Permissions internals
33+Internally, the permissions part of a mark (the grow arm) are built and then watches for changes by Graph Store. See [`app/graph-push-hook.hoon#L154`](https://github.com/urbit/urbit/blob/ac096d85ae847fcfe8786b51039c92c69abc006e/pkg/arvo/app/graph-push-hook.hoon#L154) for more info.
34+
35+
36+It is important to note that the push hook and permissions system is bespoke, and doesn't necessarily fit all permissions schemes out there. In addition, the permissions structure is likely to change, so it can make sense to implement your own permissioning structure (if you are not using `graph-push-hook`), especially in the case where you are making your own resource control system. The `graph-push-hook` permissions system is not the definite way to implement permissions, and is simply one pre-existing way to do it. The validator mark doesn't need any permissions to compile; it just needs a grow arm with at least `noun`. The way to implement your own permissioning structure is in the form of your own grow arm definitions in the validator. There's nothing special about the `graph-permission-add` arms; they are just constants, arms which are known to push-hook. As stated before, Graph Store proper (`app/graph-store.hoon`) doesn't know anything about the permissions.
userspace/graph-store/overview.mdnew file+110
@@ -0,0 +1,110 @@
1++++
2+title = "Graph Store Overview"
3+weight = 1
4+sort_by = "weight"
5+template = "doc.html"
6+insert_anchor_links = "right"
7++++
8+
9+
10+## Intro
11+Graph store is a non-relational database suitable for use in building social media applications. It is primarily designed for storing text-based content and data that is generally threaded and nested. It is not suitable as a general purpose database for highly structured or pure binary data.
12+
13+<p align="center">
14+ <img src="images/image1.png"/>
15+</p>
16+
17+Graph Store is mainly in charge of two things: data storage and retrieval, and schema validation. There are two related tools: Graph Push Hook, which provides permissioning support to Graph Store and acts as a proxy layer to Graph Store for outside ships to access, and Graph Pull Hook, which can be used to request Graph Store data from other ships. Importantly, Graph Store itself does not have a built-in permissions system (or rather, is permissions-agnostic), and acts in a purely trusted manner, and assumes that all inputs are trusted This is why Graph Push Hook exists: to mediate untrusted requests from outside ships to your instance of Graph Store, rejecting invalid ones where a ship should not be able to modify the data.
18+
19+## Data Structures
20+### Posts
21+
22+<p align="center">
23+ <img src="images/image2.png"/>
24+</p>
25+
26+Above, we can see a representation of a post on the left, along with its table form on the right.
27+A post is the most basic building block of a graph.
28+Every post is made up of an:
29+- **Author** - who created the post
30+- **Index** - the unique path/identifier of the post on the graph (more on this later)
31+- **Time Sent** - when the post was created
32+- **Contents** - data that the user intended to post
33+
34+Currently, Graph Store supports 5 data types for content that is to be stored within a graph:
35+- **Text** - plain text data
36+- **Url** - specific data type for urls
37+- **Mention** - mentioning another ship
38+- **Code** - a pair of a piece of code that was executed and its result (this is static data, and is not used to execute code on other's ships)
39+- **Reference** - a reference to another post
40+
41+
42+### Graphs and Nodes
43+
44+<p align="center">
45+ <img src="images/image3.png"/>
46+</p>
47+
48+<p align="center">
49+ <img src="images/image4.png"/>
50+</p>
51+
52+A graph is a flat, ordered map of nodes, where each node can have a child graph, which is itself a flat ordered map of nodes. Nodes contain a post and a child graph. In the above diagram, we can see an example of a basic graph on the top, along with the underlying structure of the data in table form underneath.
53+
54+A few vocab terms:
55+- **Root graph** refers to the outermost graph, G in this case
56+- **Top level** is a loose term usually used to refer to a node that exists in the root graph. **A** and **D** would be considered nodes at the top level, but **B** and all its descendants would not qualify.
57+- **Sibling nodes** are two nodes that reside next to each other, meaning that they reside in the same `graph`. **A** and **D** are an example of sibling nodes.
58+- We say a node is a **child** of another node if it directly resides within the child graph of that node
59+- A node is a **parent** to another node if it directly contains the node within its children. An example is the relationship between **B** and **C**: **C** is a child of **B**, while **B** is the parent of **C**
60+- Another way of saying a node is a child is by saying that node **B** is nested within node **A** Strictly speaking, **C** is not a child of **A**, but we do say that **A** is an **ancestor** of **C**, while **C** is a **descendant** of **A**.
61+- **Leaf nodes** refer to nodes that do not have children. **C** and **D** are both examples of leaf nodes
62+
63+### Index
64+
65+<p align="center">
66+ <img src="images/image5.png"/>
67+</p>
68+
69+Indexes are a way of uniquely identifying a node within a graph. You can think of `index`es as similar to file paths, although they aren't exactly the same. Roughly, a file path is a unique reference to a file or folder located in the filesystem. Similarly, an index is a unique reference to a node nested within a graph. The written syntax for a full index is very similar to file paths. It consists of every index fragment in order separated by a slash. A node's **level of nesting** refers to how deeply it is nested within the context of the root graph. The level of nesting directly corresponds to the number of items in the index. An index fragment is the atom by which a node is uniquely identified within it's graph, and roughly corresponds to a specific name of a directory along a path. In the diagrams that follow, we'll use the index fragment instead of the index to avoid repeating redundant information, but please note that internally graph-store still store the full index in every node's post.
70+
71+In the above diagram, we would say that nodes **A** and **D** are nested 1 level deep, while **B** would be at the 2nd level of nesting, and **C** would be nested 3 levels deep.
72+
73+Indexes are usually numbers and are stored as such in the database. Most commonly, they can represent:
74+- Date or time of posting
75+- A sequence of numbers starting from 1 increasing
76+- Structural/constant value: values which are associated with a specific meaning in the context of the schema of an application.
77+
78+However, there is no strict requirement for them to be numbers; they can be cords as well as other data types. As we'll see in the later sections, it is up to the app developer to decide this when creating their application.
79+
80+
81+### Structural Nodes vs. Content-Centric Nodes
82+
83+<p align="center">
84+ <img src="images/image6.png"/>
85+</p>
86+
87+When using Graph Store, there is a notion of **structural nodes** vs. **content-centric nodes**. In the example diagram, we've color coded the different nodes based on what type of node they are. **Content-centric nodes** represent data created or consumed directly by the user. **Structural nodes**, on the other hand do not directly represent user data, and instead represent a higher level relationship between different user data. Structural nodes are used to implement the structure of the schema that is being implemented. In other words, they exist away from the eyes of the user, solely for organizational purposes.
88+
89+Note that this differentiation between content-centric vs. structural nodes is purely developer-facing, and not encoded within the actual system. Although these patterns aren't hard or fast rules, we'll see how they can be used in practice in the validator walkthrough section.
90+
91+## Validator Overview - Schema and Permissions
92+
93+Every social application has a minimum amount of information it needs to function along with the structure that the information must follow. We'll call this the application's schema. This is enforced by a **validator**. Given a schema, a validator's primary function is to encode its constraints and validate arbitrary data against it. Graph Store uses clay marks to represent validators. Validators are a special case of a mark, so the terms will be used interchangeably.
94+
95+In addition, validators can also encode structural permissions. **Structural permissions** govern who is allowed to add or remove a given node (and by extension modify its children) based on the node's characteristics.
96+
97+There are 3 different classes of users:
98+- **Admin** - An owner of a resource or someone who's been delegated similar privileges
99+- **Writer** - Someone who can create and modify their own content but cannot modify others'
100+- **Reader** - Someone who is only given permission to access but not create or modify, except in special cases
101+
102+There are two different types of privileges:
103+- **Add privileges** - permission to create a given node and create any child nodes
104+- **Remove privileges** - permission to remove a given node and remove any of its children
105+
106+There are also three different levels of access a given permission level can have:
107+- **%no** - user does not have add or remove privileges for this node
108+- **%yes** - user has add or remove privileges for this node, whether or not they authored the parent node
109+- **%self** - user has add or remove privileges for child nodes only if they authored the parent node (determined by `author` of post, i.e., they are the author of the post)
110+
userspace/graph-store/sur-file-walkthrough.mdnew file+336
@@ -0,0 +1,336 @@
1++++
2+title = "`sur` File Walkthrough"
3+weight = 2
4+sort_by = "weight"
5+template = "doc.html"
6+insert_anchor_links = "right"
7++++
8+
9+
10+Let's go through the type definitions of some of the most used types when working with Graph Store.
11+
12+## Post
13+Here's `sur/post.hoon`
14+```
15++$ index (list atom)
16++$ uid [=resource =index]
17+::
18+:: +sham (half sha-256) hash of +validated-portion
19++$ hash @ux
20+::
21++$ signature [p=@ux q=ship r=life]
22++$ signatures (set signature)
23++$ post
24+ $: author=ship
25+ =index
26+ time-sent=time
27+ contents=(list content)
28+ hash=(unit hash)
29+ =signatures
30+ ==
31+::
32++$ indexed-post [a=atom p=post]
33+::
34++$ validated-portion
35+ $: parent-hash=(unit hash)
36+ author=ship
37+ time-sent=time
38+ contents=(list content)
39+ ==
40+::
41++$ content
42+ $% [%text text=cord]
43+ [%mention =ship]
44+ [%url url=cord]
45+ [%code expression=cord output=(list tank)]
46+ [%reference =uid]
47+ :: TODO: maybe use a cask?
48+ ::[%cage =cage]
49+ ==
50+```
51+
52+
53+### Index
54+```
55++$ index (list atom)
56++$ uid [=resource =index]
57+```
58+
59+`index` is a list of atoms (big integers). It represents a path to a specific node on a graph that is nested arbitrarily deep.
60+
61+`uid` is simply a pair of a `resource` and an `index`. These two pieces of information combine to form an unambiguous way to identify and reference a node. It is used in the `content` type to model a reference.
62+
63+An index fragment is not an explicitly defined type, but since an index is `(list atom)`, it follows that the type of an index fragment is `atom`. This is what gives the developer the flexibility to use more than just numbers in an index: any type that is representable as an atom can be used in the index.
64+
65+### Hashing (Part 1)
66+```
67+:: +sham (half sha-256) hash of +validated-portion
68++$ hash @ux
69+::
70++$ signature [p=@ux q=ship r=life]
71++$ signatures (set signature)
72+```
73+
74+These types are used to cryptographically sign a given post, so that the host of some content cannot act as an imposter, and post content impersonating as someone else. `signature` represents a triple of signed message of hash, author, and author's life at time of posting. These can be used to cryptographically attest to a message. The implementation is a form of asymmetric/public-key encryption, where `q` and `r` are data necessary to look up a ship's public key on azimuth, which can be used to verify the validity of the message.
75+
76+### Content Types
77+```
78++$ content
79+ $% [%text text=cord]
80+ [%mention =ship]
81+ [%url url=cord]
82+ [%code expression=cord output=(list tank)]
83+ [%reference =uid]
84+ :: TODO: maybe use a cask?
85+ ::[%cage =cage]
86+ ==
87+--
88+```
89+
90+`content` enumerates all the possible content types that a post can have. The possible content types can be:
91+
92+- **Text** - representing plain text content
93+- **Url** - specific data type for urls
94+- **Mention** - mentioning another ship
95+- **Code** - a pair of a piece of code that was executed and its result (static data, no execution takes place inside of Graph Store)
96+- **Reference** - a reference to another post. Uses the `uid` type under the hood
97+
98+Currently, these are the only content types supported by Graph Store, although there is potential for dynamic content support in the form of a `cage`.
99+
100+### Post
101+```
102++$ post
103+ $: author=ship
104+ =index
105+ time-sent=time
106+ contents=(list content)
107+ hash=(unit hash)
108+ =signatures
109+ ==
110+::
111++$ indexed-post [a=atom p=post]
112+```
113+
114+Post is a fundamental type that represents what we normally think of as a post on social media. Most of the rest of the types are self-explanatory. `hash` is the optional hash of the post, and `signatures` is the (potentially empty) set of `signature`s if the post is cryptographically signed.
115+
116+An `indexed-post` is a post with an associated index fragment that can be used to validate a post's index with an index fragment that is expected at the end of the index list.
117+
118+### Hashing (Part 2)
119+```
120++$ validated-portion
121+ $: parent-hash=(unit hash)
122+ author=ship
123+ time-sent=time
124+ contents=(list content)
125+ ==
126+::
127+```
128+
129+The parts of a `post` that are actually hashed to obtain a value of type the earlier type `hash`.
130+
131+
132+## Graph Store
133+
134+Here's `sur/graph-store.hoon`
135+
136+```
137++$ graph ((mop atom node) gth)
138++$ marked-graph [p=graph q=(unit mark)]
139+::
140++$ node [=post children=internal-graph]
141++$ graphs (map resource marked-graph)
142+::
143++$ internal-graph
144+ $~ [%empty ~]
145+ $% [%graph p=graph]
146+ [%empty ~]
147+ ==
148+::
149++$ tag-queries (jug term resource)
150+::
151+::
152++$ network
153+ $: =graphs
154+ =tag-queries
155+ =update-logs
156+ archive=graphs
157+ validators=(set mark)
158+ ==
159+::
160++$ update
161+ $% [%0 p=time q=update-0]
162+ ==
163+::
164++$ update-log ((mop time logged-update) gth)
165++$ update-logs (map resource update-log)
166+::
167++$ logged-update
168+ $% [%0 p=time q=logged-update-0]
169+ ==
170+::
171++$ logged-update-0
172+ $% [%add-graph =resource =graph mark=(unit mark) overwrite=?]
173+ [%add-nodes =resource nodes=(map index node)]
174+ [%remove-nodes =resource indices=(set index)]
175+ [%add-signatures =uid =signatures]
176+ [%remove-signatures =uid =signatures]
177+ ==
178+::
179+
180+
181++$ update-0
182+ $% logged-update-0
183+ [%remove-graph =resource]
184+ ::
185+ [%add-tag =term =resource]
186+ [%remove-tag =term =resource]
187+ ::
188+ [%archive-graph =resource]
189+ [%unarchive-graph =resource]
190+ [%run-updates =resource =update-log]
191+ ::
192+ :: NOTE: cannot be sent as pokes
193+ ::
194+ [%keys =resources]
195+ [%tags tags=(set term)]
196+ [%tag-queries =tag-queries]
197+ ==
198+--
199++$ permissions
200+ [admin=permission-level writer=permission-level reader=permission-level]
201+::
202+:: $permission-level: levels of permissions in increasing order
203+::
204+:: %no: May not add/remove node
205+:: %self: May only nodes beneath nodes that were added by
206+:: the same pilot, may remove nodes that the pilot 'owns'
207+:: %yes: May add a node or remove node
208++$ permission-level
209+ ?(%no %self %yes)
210+```
211+
212+### Graph, Node, and Related Objects
213+```
214++$ graph ((mop atom node) gth)
215++$ marked-graph [p=graph q=(unit mark)]
216+::
217++$ node [=post children=internal-graph]
218++$ graphs (map resource marked-graph)
219+::
220++$ internal-graph
221+ $~ [%empty ~]
222+ $% [%graph p=graph]
223+ [%empty ~]
224+ ==
225+::
226+
227++$ network
228+ $: =graphs
229+ =tag-queries
230+ =update-logs
231+ archive=graphs
232+ validators=(set mark)
233+ ==
234+::
235+```
236+A graph is a loosely interconnected set of data which can reference each other and be arbitrarily nested and interconnected. This concept is based off of the mathematical concept of a graph: a collection of nodes, and connections between nodes called edges.
237+
238+`graph` is a `mop`, a map which maintains ordering on its keys based on a sorting function also known as an ordered map. A graph's keys are `atom`s representing a node's index fragment and whose values are `node`s, where entries are sorted by largest valued keys first, using `gth` as the sorting function. This is the fundamental data structure used in Graph Store.
239+
240+Here are some helpful Wikipedia pages for more info on what this data type represents:
241+- [Graph (abstract data type)](https://en.wikipedia.org/wiki/Graph_(abstract_data_type))
242+- [Graph database](https://en.wikipedia.org/wiki/Graph_database#Background)
243+- [Graph (discrete mathematics)](https://en.wikipedia.org/wiki/Graph_(discrete_mathematics))
244+
245+`node` represents a pair of a `post` and all of its children, which is an `internal-graph`.
246+
247+`internal-graph` is a tagged union representing the state that children can be in. Either a `node` has children in the form of a `graph`, or does not have any and is labeled as `%empty`.
248+
249+`marked-graph` is the pair of a `graph` and an optionally present `mark`, which is used by Graph Store to validate a graph against the provided validator.
250+
251+`graphs` is a mapping between `resource`s and`marked-graph`s
252+
253+`network` is the highest level data structure used by Graph Store to represent all the information that the agent is aware of.
254+
255+
256+
257+### Tag Queries
258+
259+```
260++$ tag-queries (jug term resource)
261+```
262+
263+`tag-queries` is a mapping where the keys are `term`s and the values are a `set` of `resource`s (this pattern is called a `jug`). It is a simple tagging system that allows for various ad-hoc collections, similar to filesystem tags being used to sort different files/folders. While the type's name is `tag-queries`, there is no complex querying system as of now. Currently, you can add term/resources pairs into the tag queries, get a list of all terms in tag-queries, and get the whole `jug` out of Graph Store.
264+
265+### Update (Part 1)
266+```
267++$ update
268+ $% [%0 p=time q=update-0]
269+ ==
270+::
271++$ logged-update-0
272+ $% [%add-graph =resource =graph mark=(unit mark) overwrite=?]
273+ [%add-nodes =resource nodes=(map index node)]
274+ [%remove-nodes =resource indices=(set index)]
275+ [%add-signatures =uid =signatures]
276+ [%remove-signatures =uid =signatures]
277+ ==
278+::
279++$ update-0
280+ $% logged-update-0
281+ [%remove-graph =resource]
282+ ::
283+ [%add-tag =term =resource]
284+ [%remove-tag =term =resource]
285+ ::
286+ [%archive-graph =resource]
287+ [%unarchive-graph =resource]
288+ [%run-updates =resource =update-log]
289+ ::
290+ :: NOTE: cannot be sent as pokes
291+ ::
292+ [%keys =resources]
293+ [%tags tags=(set term)]
294+ [%tag-queries =tag-queries]
295+ ==
296+--
297+```
298+
299+The `update` type is what is used to interact with Graph Store. It is used both to update subscribers with data (outgoing data) and to write to Graph Store itself (incoming data). The first six actions are sent as pokes to graph-store in the form of a `graph-update`, which is an alias for `update` above. All actions defined here allow you to create/read/update/delete various objects in a running Graph Store gall agent. An `update-0` encapsulates all `logged-update-0` (i.e. any `logged-update-0` is an `update-0` but not necessarily the other way around). The last three actions are scries which allow you to ask a Graph Store agent for its current state regarding the three entries.
300+
301+If you want to check out a relevant code listing to see how graph store handles these pokes, see [`app/graph-store.hoon#L221-L227`](https://github.com/urbit/urbit/blob/e2ad6e3e9219c8bfad62f27f05c7cac94c9effa8/pkg/arvo/app/graph-store.hoon#L221-L227)
302+
303+### Update (Part 2)
304+```
305++$ update-log ((mop time logged-update) gth)
306++$ update-logs (map resource update-log)
307+::
308++$ logged-update
309+ $% [%0 p=time q=logged-update-0]
310+ ==
311+::
312+```
313+
314+`update-log` is an ordered map where the keys are a timestamp (time is an alias for `@da`, an absolute datetime) and the values are `logged-update`s, where entries are sorted with the most recent timestamp first. It represents a history of updates applied to a graph. `update-logs` is a mapping where keys are resources and values are `update-log`s. This is the data structure used by Graph Store to store the history of actions associated with all graphs that it knows about, where each graph has a unique resource that identifies it. A `logged-update` is a data structure that holds any `logged-update-0` along with a time identifying when the update happened. It follows a versioning pattern similar to the versioned state of a gall agent.
315+
316+It is important to note that the source of truth is actually the `update-logs`, and not the `graphs`. Similar to the Urbit event log, Graph Store also stores all updates that it receives, so that it can rebuild its current state on demand. The current state of the database is more of a product of the event log, like a checkpoint, or a materialized db view, and is not the source of truth. As a result, the Graph Store database is immutable in nature, where all data is preserved and deleted data is only inaccessible in the current view or checkpoint, and is still present and recoverable by replaying the log.
317+
318+The reason for having the main CRUD actions being `logged-update`s is so that Graph Store knows which order to process the log entries in when it is rebuilding its current state. The time associated with the `logged-update` is a way of specifying the canonical order to process operations. All other actions that aren't part of logged-update stand on their own and don't need a timestamp in order to properly apply them.
319+
320+### Permissions
321+```
322++$ permissions
323+ [admin=permission-level writer=permission-level reader=permission-level]
324+::
325+:: $permission-level: levels of permissions in increasing order
326+::
327+:: %no: May not add/remove node
328+:: %self: May only nodes beneath nodes that were added by
329+:: the same pilot, may remove nodes that the pilot 'owns'
330+:: %yes: May add a node or remove node
331++$ permission-level
332+ ?(%no %self %yes)
333+```
334+
335+These are the types from the permissioning system explained earlier. `permissions` is just a length-3 cell of `permissions-level`s for Admins, Writers, and Readers respectively.
336+
userspace/graph-store/validator-walkthrough.mdnew file+486
@@ -0,0 +1,486 @@
1++++
2+title = "Validator Walkthrough"
3+weight = 3
4+sort_by = "weight"
5+template = "doc.html"
6+insert_anchor_links = "right"
7++++
8+
9+
10+## Schema and Permissioning Implementation
11+In this section, we will analyze various applications and see how they implement their schema and permissions.
12+
13+
14+### Current State of Permissioning
15+Please note that as it stands today, the current permissioning system is likely subject to change. It is by no means the only manner in which to set up permissionings. For many applications, writing a small and bespoke permissioning library may be suitable as an alternative to the `graph-push-hook` permissioning scheme which is what will be covered in the following examples.
16+
17+Also, please be aware that the current set of applications use a special type known as `vip-metadata`, which stands for "variation in permission".
18+It is extra metadata attached to a post that is available to the permissioning arms that is mainly used to specify whether reader comments are enabled or disabled.
19+Here's the source if you want to explore: [`sur/metadata-store.hoon#L20-L30`](https://github.com/urbit/urbit/blob/ac096d85ae847fcfe8786b51039c92c69abc006e/pkg/arvo/sur/metadata-store.hoon#L20-L30)
20+
21+
22+### Chat
23+#### Schema
24+Here's what the schema of chat looks like:
25+
26+<p align="center">
27+ <img src="images/image7.png"/>
28+</p>
29+
30+A chat is a flat graph, where all chat messages are nodes appended to the root of the graph.
31+The graph represents a chat channel and contains all chat messages in order. A chat message is a child node of the root graph.
32+
33+
34+Here's the definition of the schema in the chat validator mark:
35+File: `mar/graph/validator/chat.hoon`
36+```
37+++ grab
38+ |%
39+ ++ noun
40+ |= p=* :: 1
41+ =/ ip ;;(indexed-post p) :: 2
42+ ?> ?=([@ ~] index.p.ip) :: 3
43+ ip :: 4
44+ --
45+::
46+```
47+
48+Here are the steps:
49+1. Given a noun (we expect an indexed-post)
50+1. Try to coerce p to an indexed-post, crash if it doesn't cast
51+1. Assert that the index of the post of the indexed post is only a single atom, i.e., that it is only nested one level deep
52+1. Return the indexed post
53+
54+Notably, under this set of rules, there is no nesting allowed. Put another way, no node is allowed to have any children. Nodes can only be added to the root graph. Step 3 is what enforces the flat hierarchy. If someone were to manually try to submit a node with children, Graph Store would reject it, preventing them from sending an invalid chat message.
55+
56+Since the schema of the chat application is simple enough, it has no need for structural nodes at all.
57+
58+#### Permissioning
59+
60+<p align="center">
61+ <img src="images/image8.png"/>
62+</p>
63+
64+Let's take a look at the permissions table in the diagram.
65+
66+Chat Message `[@ ~]`
67+- Add Privileges
68+ - Admins and Writers have `%yes` add permissions for all nodes at the top level, meaning that they have the ability to post chat messages, even if they did not create the chat channel
69+ - Readers have `%no` add privileges for any nodes at the root level, so they do not have the ability to post chat messages
70+- Remove Privileges
71+ - Admins and Writers have `%self` remove privileges, meaning that they may only delete chat messages that they posted, not anyone else's
72+ - Readers have `%no` remove privileges for any nodes, meaning they cannot delete any chat messages
73+
74+This follows our general intuition of how permissions for chat messages should be structured.
75+For example, it wouldn't make sense to give readers `%self`, because they do not have the ability to create nodes in the first place, so they will never be in a position to delete any nodes.
76+
77+Let's see how this permissioning system is implemented in the validator code.
78+
79+Here is the `grow` arm of `mar/validator/chat.hoon`
80+```
81+|_ i=indexed-post :: A
82+++ grow
83+ |%
84+ ++ graph-permissions-add
85+ |= vip=vip-metadata:met :: 1
86+ ?+ index.p.i !! :: 2
87+ [@ ~] [%yes %yes %no] :: 3
88+ ==
89+ ::
90+ ++ graph-permissions-remove
91+ |= vip=vip-metadata:met :: 4
92+ ?+ index.p.i !! :: 5
93+ [@ ~] [%self %self %no] :: 6
94+ ==
95+ ::
96+ --
97+::
98+```
99+
100+In line A, we accept an `indexed-post` that is used in the rest of the `grow` arm.
101+
102+`graph-permissions-add`
103+1. Accept a noun `vip` of type `vip-metadata`
104+2. Switch on the index of the post found in `i`, crashing if no successful matches occur
105+3. If the index is nested one level deep
106+4. Return a `permissions` noun defined as: [admin: %yes, writer: %yes, reader: %no]
107+
108+
109+ `graph-permissions-remove`
110+
111+5. Accept a noun `vip` of type vip-metadata
112+6. Switch on the index, crashing if no successful matches occur
113+7. If the index is nested one level deep
114+8. Return a `permissions` noun defined as: [admin: %self, writer: %self, reader: %no]
115+
116+In this example, a switch statement is used to determine the `permissions` values based on the index of the post.
117+
118+
119+### Links
120+#### Schema
121+
122+<p align="center">
123+ <img src="images/image9.png"/>
124+</p>
125+
126+The root graph represents the whole Links collection. Every Links entry is a child node of this graph. Every Links entry is made up of:
127+- The link and its description
128+- A comments section
129+
130+
131+The comments section holds all individual comment nodes, but comments are not simple leaf nodes. An individual comment is actually a structural node that acts as a revision container, storing the comment's full edit history by storing each edit as a child node. The front-end is responsible for properly displaying the latest revision of the comment.
132+
133+
134+Here's the validator, located at `mar/graph/validator/link.hoon`:
135+```
136+++ grab
137+ |%
138+ ++ noun
139+ |= p=* :: 1
140+ =/ ip ;;(indexed-post p) :: 2
141+ ?+ index.p.ip ~|(index+index.p.ip !!) :: 3
142+ :: top-level link post; title and url
143+ ::
144+ [@ ~] :: 4
145+ ?> ?=([[%text @] [%url @] ~] contents.p.ip) :: 4a
146+ ip
147+ ::
148+ :: comment on link post; container structure
149+ ::
150+ [@ @ ~] :: 5
151+ ?> ?=(~ contents.p.ip) :: 5a
152+ ip
153+ ::
154+ :: comment on link post; comment text
155+ ::
156+ [@ @ @ ~] :: 6
157+ ?> ?=(^ contents.p.ip) :: 6a
158+ ip
159+ ==
160+ --
161+```
162+
163+
164+<ol>
165+ <li>Get the post as a noun</li>
166+ <li>Force cast to indexed post</li>
167+ <li>
168+ Switch on index of post, crash if no match occurs
169+ <ol type="a">
170+ <li>If the node is nested one level deep, (if the index is made up of a single atom)</li>
171+ <li>Ensure that it is a cell that has two pieces of data, whose content types are text and url</li>
172+ <li>If the node is nested two levels deep</li>
173+ <li>Ensure that it is empty, this is the structural node for holding comment revisions; it should not contain any content</li>
174+ </ol>
175+ </li>
176+ <li>
177+ If the node is nested three levels deep
178+ <ol type="a">
179+ <li>Ensure that it is a cell, this is a specific revision of a comment under a revision container</li>
180+ </ol>
181+ </li>
182+</ol>
183+
184+
185+It is important to note that you cannot directly edit the url or link afterwards, only the whole link entry itself. This is because you can only add or remove nodes to a graph, not modify them. Comments, however, can be “edited”. This is possible by using a technique called a revision container. Instead of having a single leaf node holding the comment text in its contents, a node with empty contents is created. All revisions of the comment are added as children to this note, and the frontend simply shows the most recent one only.
186+
187+#### Permissioning
188+
189+<p align="center">
190+ <img src="images/image10.png"/>
191+</p>
192+
193+<p align="center">
194+ <img src="images/image11.png"/>
195+</p>
196+
197+Let's analyze the permissions structure.
198+
199+**TODO** *stubbed out awaiting potential changes in source*
200+
201+Here's how it is implemented:
202+
203+```
204+|_ i=indexed-post
205+++ grow
206+ |%
207+ ++ noun i
208+ ::
209+ ++ graph-permissions-add
210+ |= vip=vip-metadata:met :: 1
211+ =/ reader :: 2
212+ ?=(%reader-comments vip)
213+ ?+ index.p.i !! :: 3
214+ [@ ~] [%yes %yes %no] :: 3a
215+ [@ @ ~] [%yes %yes ?:(reader %yes %no)] :: 3b
216+ [@ @ @ ~] [%self %self %self] :: 3c
217+ ==
218+ ::
219+ ++ graph-permissions-remove
220+ |= vip=vip-metadata:met :: 4
221+ =/ reader :: 5
222+ ?=(%reader-comments vip)
223+ ?+ index.p.i !! :: 6
224+ [@ ~] [%yes %self %self] :: 6a
225+ [@ @ ~] [%yes %self %self] ::
226+ [@ @ @ ~] [%yes %self %self] ::
227+ ==
228+ ::
229+```
230+
231+`graph-permissions-add`
232+<ol>
233+ <li>Accept a noun <code>vip</code> of type vip-metadata
234+ <ul>
235+ <li>Declare a variable <code>reader</code>, a flag which is true if reader comments are enabled, false otherwise</li>
236+ </ul>
237+ </li>
238+ <li>Switch on the index of the post found in <code>i</code>, crashing if no successful matches occur</li>
239+ <li>If the index is nested one level deep, return a <code>permissions</code> noun defined as:
240+ <ul>
241+ <li>Admin - <code>%yes</code></li>
242+ <li>Writer - <code>%yes</code></li>
243+ <li>Reader: <code>%no</code></li>
244+ </ul>
245+ </li>
246+ <li>If the index is nested two levels deep, return a <code>permissions</code> noun defined as:
247+ <ul>
248+ <li>Admin - <code>%yes</code></li>
249+ <li>Writer - <code>%yes</code></li>
250+ <li>Reader: <code>%yes</code> if reader comments are enabled, else <code>%no</code></li>
251+ </ul>
252+ </li>
253+ <li>If the index is nested three levels deep, return a <code>permissions</code> noun defined as:
254+ <ul>
255+ <li>Admin - <code>%yes</code></li>
256+ <li>Writer - <code>%yes</code></li>
257+ <li>Reader: <code>%yes</code> if reader comments are enabled, else <code>%no</code></li>
258+ </ul>
259+ </li>
260+</ol>
261+
262+
263+`graph-permissions-remove`
264+<ol>
265+ <li>Accept a noun <code>vip</code> of type vip-metadata</li>
266+ <li>Declare a variable <code>reader</code>, a flag which is true if reader comments are enabled, false otherwise</li>
267+ <li>Switch on the index of the post found in <code>i</code>, crashing if no successful matches occur</li>
268+ <li>If the index is nested one level deep, two levels deep, or three levels deep, return a <code>permissions</code> noun defined as:
269+ <ul>
270+ <li>Admin - <code>%yes</code></li>
271+ <li>Writer - <code>%self</code></li>
272+ <li>Reader: <code>%self</code></li>
273+ </ul>
274+ </li>
275+</ol>
276+
277+
278+
279+### Publish
280+
281+#### Schema
282+
283+<p align="center">
284+ <img src="images/image12.png"/>
285+</p>
286+
287+Here, a notebook, which is a collection of blog posts (called notes), is represented by the root graph. All data associated with the blog post is represented by the top level node, which is the note itself along with the associated comments. One level deeper, we see two container structures. The first one is the post revision container; it holds the edit history of your blog post. Every child node of this corresponds to the actual title and text of your blog post. The second one is the comments container. This represents the comment section of your blog post. Every child node of this is not a comment, but a comment revision container, which, as before, contains the edit history of your comment.
288+
289+Here's its validator
290+```
291+ ++ noun
292+ |= p=* :: 1
293+ =/ ip ;;(indexed-post p) :: 2
294+ ?+ index.p.ip !! :: 3
295+ :: top level post must have no content
296+ [@ ~] :: 4
297+ ?> ?=(~ contents.p.ip) :: 4a
298+ ip
299+ :: container for revisions
300+ ::
301+ [@ %1 ~] :: 5
302+ ?> ?=(~ contents.p.ip) :: 5a
303+ ip
304+ :: specific revision
305+ :: first content is the title
306+ :: revisions are numbered by the revision count
307+ :: starting at one
308+ [@ %1 @ ~] :: 6
309+ ?> ?=([* * *] contents.p.ip) :: 6a
310+ ?> ?=(%text -.i.contents.p.ip) :: 6b
311+ ip
312+ :: container for comments
313+ ::
314+ [@ %2 ~] :: 7
315+ ?> ?=(~ contents.p.ip) :: 7a
316+ ip
317+ :: container for comment revisions
318+ ::
319+ [@ %2 @ ~] :: 8
320+ ?> ?=(~ contents.p.ip) :: 8a
321+ ip
322+ :: specific comment revision
323+ ::
324+ [@ %2 @ @ ~] :: 9
325+ ?> ?=(^ contents.p.ip) :: 9a
326+ ip
327+ ==
328+ --
329+```
330+
331+Walkthrough
332+
333+
334+<ol>
335+ <!-- WHYYYYY GITHUB WHY -->
336+ <li>Get the post as a noun</li>
337+ <li>Force cast to indexed post</li>
338+ <li>Switch on index of post, crashing (reject) if no matches found</li>
339+ <li>If the node is nested one level deep
340+ <ol type="a"><li>Ensure that its contents are empty. The top level node is a structural node containing all of the post and associated data.</li></ol>
341+ </li>
342+ <li>If it is nested 2 levels deep and contains a 1 as it's last index fragment
343+ <ol type="a"><li>Ensure that its contents are empty. This is a structural node for holding revisions to the blog post.</li></ol>
344+ </li>
345+ <li>If the node is nested three levels deep, and has a 1 as it's second index fragment
346+ <ol type="a">
347+ <li>Ensure that its contents is a list of at least 2 elements</li>
348+ <li>Ensure that the first element of contents has a content type of text</li>
349+ </ol>
350+ </li>
351+ <li>If the node is nested two levels deep, and has a 2 as it's last index fragment
352+ <ol type="a"><li>Ensure that its contents are empty. This is a structural node for holding comments.</li></ol>
353+ </li>
354+ <li>If the node is nested three levels deep, and has a 2 as it's second index fragment
355+ <ol type="a"><li>Ensure that its contents are empty. This is a structural node for holding revisions of a specific comment.</li></ol>
356+ </li>
357+ <li>If the node is nested four levels deep, and has a 2 as it's second index fragment
358+ <ol type="a"><li>Ensure that contents has type `cell`. This is a specific revision of a comment</li></ol>
359+ </li>
360+</ol>
361+
362+Items 1-3 are setting up the validator
363+Items 4-6 are enforcing the schema for the post in general
364+Items 7-9 are enforcing the schema for comments specifically
365+
366+Notably, the revision container for the blog post itself allows the post to be edited, unlike the link entry in the previous example. In addition, in step six, the reason that the validator is made this way is because the first element of the contents is interpreted as the title of the post, and the rest of the elements are interpreted as the body of the post. Otherwise, the structure is unchanged from the Links example.
367+
368+#### Permissioning
369+
370+<p align="center">
371+ <img src="images/image13.png"/>
372+</p>
373+
374+<p align="center">
375+ <img src="images/image14.png"/>
376+</p>
377+
378+Let's take a look at the permissioning structure for Publish.
379+
380+**TODO** *stubbed out awaiting potential changes in source*
381+
382+```
383+|_ i=indexed-post
384+++ grow
385+ |%
386+ ++ noun i
387+ ++ graph-permissions-add
388+ |= vip=vip-metadata:met :: 1
389+ ?+ index.p.i !! :: 2
390+ [@ ~] [%yes %yes %no] :: new note :: 2a
391+ [@ %1 @ ~] [%self %self %no] :: 2b
392+ [@ %2 @ ~] [%yes %yes ?:(?=(%reader-comments vip) %yes %no)] :: 2c
393+ [@ %2 @ @ ~] [%self %self %self] :: 2d
394+ ==
395+ ::
396+ ++ graph-permissions-remove
397+ |= vip=vip-metadata:met :: 3
398+ ?+ index.p.i !! :: 4
399+ [@ ~] [%yes %self %self] :: 4a
400+ [@ %1 @ @ ~] [%yes %self %self] ::
401+ [@ %2 @ ~] [%yes %self %self] ::
402+ [@ %2 @ @ ~] [%yes %self %self] ::
403+ ==
404+::
405+```
406+
407+`graph-permissions-add`
408+
409+ <ol>
410+ <li>Accept a noun <code>vip</code> of type vip-metadata</li>
411+ <li>Switch on the index of the post found in <code>i</code>, crashing if no successful matches occur</li>
412+ <li>If the index is nested one level deep, return a <code>permissions</code> noun defined as:
413+ <ul>
414+ <li>Admin - <code>%yes</code></li>
415+ <li>Writer - <code>%yes</code></li>
416+ <li>Reader: <code>%no</code></li>
417+ </ul>
418+ </li>
419+ <li>If the index is nested three levels deep and has a 1 as its 2nd index fragment, return a <code>permissions</code> noun defined as:
420+ <ul>
421+ <li>Admin - <code>%self</code></li>
422+ <li>Writer - <code>%self</code></li>
423+ <li>Reader: <code>%no</code></li>
424+ </ul>
425+ </li>
426+ <li>If the index is nested three levels deep and has a 2 as its 2nd index fragment, return a <code>permissions</code> noun defined as:
427+ <ul>
428+ <li>Admin - <code>%yes</code></li>
429+ <li>Writer - <code>%yes</code></li>
430+ <li>Reader: <code>%yes</code> if reader comments are enabled, else <code>%no</code></li>
431+ </ul>
432+ </li>
433+ <li>If the index is nested four levels deep and has a 2 as its 2nd index fragment, return a <code>permissions</code> noun defined as:
434+ <ul>
435+ <li>Admin - <code>%self</code></li>
436+ <li>Writer - <code>%self</code></li>
437+ <li>Reader: <code>%self</code></li>
438+ </ul>
439+ </li>
440+</ol>
441+
442+
443+`graph-permissions-remove`
444+<ol>
445+ <li>Accept a noun <code>vip</code> of type <code>vip-metadata</code></li>
446+ <li>Declare a variable <code>reader</code>, a flag which is true if reader comments are enabled, false otherwise</li>
447+ <li>Switch on the index of the post found in <code>i</code>, crashing if no successful matches occur</li>
448+ <li>If the index is nested one level deep, two levels deep, or three levels deep, return a <code>permissions</code> noun defined as:
449+ <ul>
450+ <li>Admin - <code>%yes</code></li>
451+ <li>Writer - <code>%self</code></li>
452+ <li>Reader: <code>%self</code></li>
453+ </ul>
454+ </li>
455+</ol>
456+
457+## General Patterns and Best Practices
458+### General Patterns
459+#### Schemas
460+The general pattern for enforcing a schema in a validator is to:
461+1. Cast the incoming data as an `indexed-post`
462+1. Switch on the index, matching by its depth and structure
463+1. Validate the structure of the contents of the post based on what we expect to see at that level of nesting. This logic can definitely be more involved than simply presence or absence of data.
464+
465+When designing your schema, it may help to decide first what data the user will be directly consuming or producing, then try to think of what extra information and structure is needed to support that. For instance, if you may want a user to be able to enter a comment, then later realize that you need a structural node that contains the whole comments section. One pattern that we can see is that all structural nodes tend to have empty contents. Another pattern we can see is that all leaf nodes have their contents set.
466+
467+In general, validators can be as robust and expressive as desired, because the mark system already sets proper limitations of what you can and cannot do, with the main restriction being no side effects can be produced by a mark.
468+
469+
470+#### Permissions
471+In general, you want to first have your schema finalized, then at every node ask the question: who should be able to modify (add/remove) this node, and its children if present in your schema. You may find that you don't need to set permission for every single type of node afterwards, although being thorough can help to find bugs in permissioning early on. Writing out the permissions in plain words and bullet points, sketching them out in the form of a table, then splitting it up into the code can also make the design process easier.
472+
473+### Earth/Mars Interface Details
474+There are a few general ways to talk to a Graph Store gall agent. You can either interact with the agent in the form of pokes/scries/spiders either directly from the dojo, or through HTTP requests via Eyre.
475+
476+- All graph store applications are going to have a mark, which are applied to data going in or coming out
477+- Eyre lets you choose the mark you scry for by specifying the desired format in the url while making the HTTP request
478+- In the case of data going in, you pass in a `marked-graph` which the Graph Store agent uses to validate the whole `graph` with. In the case of adding a node to a graph, the node will simply be checked for validity using the existing mark since the mark is already set on a per-graph basis
479+- In the case of data coming out, Eyre forcibly tries to convert the hoon noun into json, and silently fails if no json conversion exists. This process is handled by Graph Store under the hood because all graphs have a well-defined en-json/de-json format, so you never have to worry about making your own de/serialization arms.
480+
481+
482+Code References
483+- [`interface/src/logic/api/base.ts#L62`](https://github.com/urbit/urbit/blob/master/pkg/interface/src/logic/api/base.ts#L62)
484+- [`mar/graph/update.hoon`](https://github.com/urbit/urbit/blob/e2ad6e3e9219c8bfad62f27f05c7cac94c9effa8/pkg/arvo/mar/graph/update.hoon)
485+- [`sys/vane/eyre.hoon#L1617-L1625`](https://github.com/urbit/urbit/blob/ac096d85ae847fcfe8786b51039c92c69abc006e/pkg/arvo/sys/vane/eyre.hoon#L1617-L1625) shows how Eyre applies marks
486+