on master help sign in
Merge pull request #1131 from tinnus-napbus/json-guide

add json guide

commitf7923d85b0058ee1e10c75771e62ab95522ad424 authortinnus-napbus <77721746+tinnus-napbus@users.noreply.github.com> dateSat 23 Oct 2021, 14:29 (5 years ago) parent80bd56224f22e9
3 files changed+561−21
content/docs/hoon/guides/_index.md+2 content/docs/hoon/guides/json-guide.md+511 content/docs/hoon/guides/parsing.md+48−21
content/docs/hoon/guides/_index.md+2
@@ -13,3 +13,5 @@ insert_anchor_links = "right"
1313## [Parsing](/docs/hoon/guides/parsing)
1414
1515## [Strings](/docs/hoon/guides/strings)
16+
17+## [JSON](/docs/hoon/guides/json-guide)
content/docs/hoon/guides/json-guide.mdnew file+511
@@ -0,0 +1,511 @@
1++++
2+title = "JSON"
3+weight = 4
4+template = "doc.html"
5++++
6+
7+If you are working on a Gall agent with any kind of web interface, it's likely you will encounter the problem of converting Hoon data structures to JSON and vice versa. This is what we'll look at in this document.
8+
9+Urbit represents JSON data with the `$json` structure (defined in `lull.hoon`). You can refer to the [json type](#the-json-type) section below for details.
10+
11+JSON data on the web is encoded in text, so Urbit has two functions in `zuse.hoon` for dealing with this:
12+
13+- [`+en-json:html`](/docs/hoon/reference/zuse/2e_2-3#en-jsonhtml) - For printing `$json` to a text-encoded form.
14+- [`+de-json:html`](/docs/hoon/reference/zuse/2e_2-3#de-jsonhtml) - For parsing text-encoded JSON to a `$json` structure.
15+
16+You typically want `$json` data converted to some other `noun` structure or vice versa, so Urbit has three collections of functions for this purpose, also in `zuse.hoon`:
17+
18+- [`+enjs:format`](/docs/hoon/reference/zuse/2d_1-5#enjsformat) - Functions for converting various atoms and structures to `$json`.
19+- [`+dejs:format`](/docs/hoon/reference/zuse/2d_6#dejsformat) - Many "reparsers" for converting `$json` data to atoms and other structures.
20+- [`+dejs-soft:format`](/docs/hoon/reference/zuse/2d_7#dejs-softformat) - Largely the same as `+dejs:format` except its reparsers produce `unit`s which are null upon failure rather than simply crashing.
21+
22+The relationship between these types and functions look like this:
23+
24+![json diagram](https://media.urbit.org/docs/json-diagram.svg)
25+
26+Note this diagram is a simplification - the `+dejs:format` and `+enjs:format` collections in particular are tools to be used in writing conversion functions rather than simply being used by themselves, but it demonstrates the basic relationships. Additionally, it is less common to do printing/parsing manually - this would typically be handled automatically by Eyre, though it may be necessary if you're retrieving JSON data via the web client vane Iris.
27+
28+### In practice
29+
30+A typical Gall agent will have a number of structures defined in a file in the `/sur` directory. These will define the type of data it expects to be `%poke`ed with, the type of data it will `%give` to subscribers, and the type of data its scry endpoints produce.
31+
32+If the agent only interacts with other agents inside Urbit, it may just use a `%noun` `mark`. If, however, it needs to talk to a web interface of some kind, it usually must handle `$json` data with a `%json` mark.
33+
34+Sometimes an agent's interactions with a web interface are totally distinct from its interactions with other agents. If so, the agent could just have separate scry endpoints, poke handlers, etc, that directly deal with `$json` data with a `%json` mark. In such a case, you can include `$json` encoding/decoding functions directly in the agent or associated libraries, using the general techniques demonstrated in the [$json encoding and decoding example](#json-encoding-and-decoding-example) section below.
35+
36+If, on the other hand, you want a unified interface (whether interacting with a web client or within Urbit), a different approach is necessary. Rather than taking or producing either `%noun` or `%json` marked data, custom `mark` files can be created which specify conversion methods for both `%noun` and `%json` marked data.
37+
38+With this approach, an agent would take and/or produce data with some `mark` like `%my-custom-mark`. Then, when the agent must interact with a web client, the webserver vane Eyre can automatically convert `%my-custom-mark` to `%json` or vice versa. This way the agent only ever has to handle the `%my-custom-mark` data. This approach is used by `%graph-store` with its `%graph-update-2` mark, for example, and a number of other agents.
39+
40+For details of creating a `mark` file for this purpose, the [mark file example](#mark-file-example) section below walks through a practical example.
41+
42+## The `$json` type
43+
44+Urbit represents JSON data with the `$json` structure (defined in `/sys/lull.hoon`):
45+
46+```hoon
47++$ json :: normal json value
48+ $@ ~ :: null
49+ $% [%a p=(list json)] :: array
50+ [%b p=?] :: boolean
51+ [%o p=(map @t json)] :: object
52+ [%n p=@ta] :: number
53+ [%s p=@t] :: string
54+ == ::
55+```
56+
57+The correspondence of `$json` to JSON types is fairly self-evident, but here's a table comparing the two for additional clarity:
58+
59+| JSON Type | `$json` Type | JSON Example | `$json` Example |
60+| :-------- | :--------------------- | ------------------------- | :----------------------------------------------------------- |
61+| Null | `~` | `null` | `~` |
62+| Boolean | `[%b p=?]` | `true` | `[%b p=%.y]` |
63+| Number | `[%n p=@ta]` | `123` | `[%n p=~.123]` |
64+| String | `[%s p=@t]` | `"foo"` | `[%s p='foo']` |
65+| Array | `[%a p=(list json)]` | `["foo",123]` | `[%a p=~[[%s p='foo'] [%n p=~.123]]]` |
66+| Object | `[%o p=(map @t json)]` | `{"foo":"xyz","bar":123}` | `[%o p={[p='bar' q=[%n p=~.123]] [p='foo' q=[%s p='xyz']]}]` |
67+
68+Since the `$json` `%o` object and `%a` array types may themselves contain any `$json`, you can see how JSON structures of arbitrary complexity can be represented. Note the `%n` number type is a `@ta` rather than something like a `@ud` that you might expect. This is because JSON's number type may be either an integer or floating point, so it's left as a `knot` which can then be parsed to a `@ud` or `@rd` with the appropriate [`+dejs:format`](/docs/hoon/reference/zuse/2d_6) function.
69+
70+## `$json` encoding and decoding example
71+
72+Let's have a look at a practical example. Here's a core with three arms. It has the structure arm `$user`, and then two more: `+to-js` converts a `$user` structure to `$json`, and `+from-js` does the opposite. Usually we'd define the structure in a separate `/sur` file, but for simplicity it's all in the one core.
73+
74+#### `json-test.hoon`
75+
76+```hoon
77+|%
78++$ user
79+ $: username=@t
80+ name=[first=@t mid=@t last=@t]
81+ joined=@da
82+ email=@t
83+ ==
84+++ to-js
85+ |= usr=user
86+ |^ ^- json
87+ %- pairs:enjs:format
88+ :~
89+ ['username' s+username.usr]
90+ ['name' name]
91+ ['joined' (sect:enjs:format joined.usr)]
92+ ['email' s+email.usr]
93+ ==
94+ ++ name
95+ :- %a
96+ :~
97+ [%s first.name.usr]
98+ [%s mid.name.usr]
99+ [%s last.name.usr]
100+ ==
101+ --
102+++ from-js
103+ =, dejs:format
104+ ^- $-(json user)
105+ %- ot
106+ :~
107+ [%username so]
108+ [%name (at ~[so so so])]
109+ [%joined du]
110+ [%email so]
111+ ==
112+--
113+```
114+
115+### Try it out
116+
117+First we'll try using our `$json` encoding/decoding library, and afterwards we'll take a closer look at its construction. To begin, save the code above in `/lib/json-test.hoon` of the `%base` desk on a fake ship and `|commit` it:
118+
119+```
120+> |commit %base
121+>=
122++ /~zod/base/5/lib/json-test/hoon
123+```
124+
125+Then we need to build it so we can use it. We'll give it a face of `user-lib`:
126+
127+```
128+> =user-lib -build-file %/lib/json-test/hoon
129+```
130+
131+Let's now create an example of a `$user` structure:
132+
133+```
134+> =usr `user:user-lib`['john456' ['John' 'William' 'Smith'] now 'john.smith@example.com']
135+> usr
136+[ username='john456'
137+ name=[first='John' mid='William' last='Smith']
138+ joined=~2021.9.12..09.47.58..1b65
139+ email='john.smith@example.com'
140+]
141+```
142+
143+Now we can try calling the `+to-js` function with our data to convert it to `$json`:
144+
145+```
146+> =usr-json (to-js:user-lib usr)
147+> usr-json
148+[ %o
149+ p
150+ { [p='email' q=[%s p='john.smith@example.com']]
151+ [p='name' q=[%a p=~[[%s p='John'] [%s p='William'] [%s p='Smith']]]]
152+ [p='username' q=[%s p='john456']]
153+ [p='joined' q=[%n p=~.1631440078]]
154+ }
155+]
156+```
157+
158+Let's also see how that `$json` would look as real JSON encoded in text. We can do that with `+en-json:html`:
159+
160+```
161+> (crip (en-json:html (to-js:user-lib usr)))
162+'{"joined":1631440078,"username":"john456","name":["John","William","Smith"],"email":"john.smith@example.com"}'
163+```
164+
165+Finally, let's try converting the `$json` back to a `$user` again with our `+from-js` arm:
166+
167+```
168+> (from-js:user-lib usr-json)
169+[ username='john456'
170+ name=[first='John' mid='William' last='Smith']
171+ joined=~2021.9.12..09.47.58
172+ email='john.smith@example.com'
173+]
174+```
175+
176+### Analysis
177+
178+#### Converting to `$json`
179+
180+Here's our arm that converts a `$user` structure to `$json`:
181+
182+```hoon
183+++ to-js
184+ |= usr=user
185+ |^ ^- json
186+ %- pairs:enjs:format
187+ :~
188+ ['username' s+username.usr]
189+ ['name' name]
190+ ['joined' (sect:enjs:format joined.usr)]
191+ ['email' s+email.usr]
192+ ==
193+ ++ name
194+ :- %a
195+ :~
196+ [%s first.name.usr]
197+ [%s mid.name.usr]
198+ [%s last.name.usr]
199+ ==
200+ --
201+```
202+
203+There are different ways we could represent our `$user` structure as JSON, but in this case we've opted to encapsulate it in an object and have the `name` as an array (since JSON arrays preserve order).
204+
205+[`+enjs:format`](/docs/hoon/reference/zuse/2d_1-5#enjsformat)includes the convenient [`+pairs`](/docs/hoon/reference/zuse/2d_1-5#pairsenjsformat) function, which converts a list of `[@t json]` to an object containing those key-value pairs. We've used this to assemble the final object. Note that if you happen to have only a single key-value pair rather than a list, you can use [`+frond`](/docs/hoon/reference/zuse/2d_1-5#frondenjsformat) instead of `+pairs`.
206+
207+For the `joined` field, we've used the [`+sect`](/docs/hoon/reference/zuse/2d_1-5#sectenjsformat) function from `+enjs` to convert the `@da` to a Unix seconds timestamp in a `$json` number. The `+sect` function, like others in `+enjs`, takes in a noun (in this case a `@da`) and produces `$json` (in this case a `[%n @ta]` number). `+enjs` contains a handful of useful functions like this, but for the rest we've just hand-made the `$json` structure. This is fairly typical when encoding `$json`, it's usually [decoding](#converting-from-json) that makes more extensive use of the `$json` utility functions in `+format`.
208+
209+For the `name` field we've just formed a cell of `%a` and a list of `$json` strings, since a `$json` array is `[%a p=(list json)]`. Note we've separated this part into its own arm and wrapped the whole thing in a `|^` - a core with a `$` arm that's computed immediately. This is simply for readability - our structure here is quite simple but when dealing with deeply-nested `$json` structures or complex logic, having a single giant function can quickly become unwieldy.
210+
211+#### Converting from `$json`
212+
213+Here's our arm that converts `$json` to our `$user` structure:
214+
215+```hoon
216+++ from-js
217+ =, dejs:format
218+ ^- $-(json user)
219+ %- ot
220+ :~
221+ [%username so]
222+ [%name (at ~[so so so])]
223+ [%joined du]
224+ [%email so]
225+ ==
226+```
227+
228+This is the inverse of the [encoding](#converting-to-json) function described in the previous section.
229+
230+We make extensive use of [`+dejs:format`](/docs/hoon/reference/zuse/2d_6) functions here, so we've used `=,` to expose the namespace and allow succinct `+dejs` function calls.
231+
232+We use the [`+ot`](/docs/hoon/reference/zuse/2d_6#otdejsformat) function from `+dejs:format` to decode the `$json` object to a n-tuple. It's a wet gate that takes a list of pairs of keys and other `+dejs` functions and produces a new gate that takes the `$json` to be decoded (which we've given it in `jon`).
233+
234+The [`+so`](/docs/hoon/reference/zuse/2d_6#sodejsformat) functions just decode `$json` strings to `cord`s. The [`+at`](/docs/hoon/reference/zuse/2d_6#atdejsformat) function converts a `$json` array to a tuple, decoding each element with the respective function given in its argument list. Like `+ot`, `+at` is also a wet gate that produces a gate that takes `$json`. In our case we've used `+so` for each element, since they're all strings.
235+
236+For `joined`, we've used the [`+du`](/docs/hoon/reference/zuse/2d_6#dudejsformat) function, which converts a Unix seconds timestamp in a `$json` number to a `@da` (it's basically the inverse of the `+sect:enjs:format` we used earlier).
237+
238+Notice how `+ot` takes in other `+dejs` functions in its argument. One of its arguments includes the `+at` function which itself takes in other `+dejs` functions. There are several `+dejs` functions like this that allow complex nested JSON structures to be decoded. For other examples of common `+dejs` functions like this, see the [More `+dejs`](#more-dejs) section below.
239+
240+There are dozens of different functions in [`+dejs:format`](/docs/hoon/reference/zuse/2d_6) that will cover a great many use cases. If there isn't a `+dejs` function for a particular case, you can also just write a custom function - it just has to take `$json`. Note there's also the [`+dejs-soft:format`](/docs/hoon/reference/zuse/2d_7) functions - these are similar to `+dejs` functions except they produce `unit`s rather than simply crashing if decoding fails.
241+
242+## More `+dejs`
243+
244+We looked at the commonly used `+ot` function in the [first example](#converting-from-json), now let's look at a couple more common `+dejs` functions.
245+
246+### `+of`
247+
248+The [`+of`](/docs/hoon/reference/zuse/2d_6#ofdejsformat) function takes an object containing a single key-value pair, decodes the value with the corresponding `+dejs` function in a key-function list, and produces a key-value tuple. This is useful when there are multiple possible objects you might receive, and tagged unions are a common data structure in hoon.
249+
250+Let's look at an example. Here's a gate that takes in some `$json`, decodes it with an `+of` function that can handle three possible objects, casts the result to a tagged union, switches against its head with `?-`, performs some transformation and finally returns the result. You can save it as `gen/of-test.hoon` in the `%base` desk of a fake ship and `|commit %base`.
251+
252+#### `of-test.hoon`
253+
254+```hoon
255+|= jon=json
256+|^ ^- @t
257+=/ =fbb
258+ (to-fbb jon)
259+?- -.fbb
260+ %foo (cat 3 +.fbb '!!!')
261+ %bar ?:(+.fbb 'Yes' 'No')
262+ %baz :((cury cat 3) p.fbb q.fbb r.fbb)
263+==
264++$ fbb
265+ $% [%foo @t]
266+ [%bar ?]
267+ [%baz p=@t q=@t r=@t]
268+ ==
269+++ to-fbb
270+=, dejs:format
271+%- of
272+:~ foo+so
273+ bar+bo
274+ baz+(at ~[so so so])
275+==
276+--
277+```
278+
279+Let's try it:
280+
281+```
282+> +of-test (need (de-json:html '{"foo":"Hello"}'))
283+'Hello!!!'
284+
285+> +of-test (need (de-json:html '{"bar":true}'))
286+'Yes'
287+
288+> +of-test (need (de-json:html '{"baz":["a","b","c"]}'))
289+'abc'
290+```
291+
292+### `+ou`
293+
294+The [`+ou`](/docs/hoon/reference/zuse/2d_6#oudejsformat) function decodes a `$json` object to an n-tuple using the matching functions in a key-function list. Additionally, it lets you set some key-value pairs in an object as optional and others as mandatory. The mandatory ones crash if they're missing and the optional ones are replaced with a given noun.
295+
296+`+ou` is different to other `+dejs` functions - the functions it takes are `$-((unit json) grub)` rather than the usual `$-(json grub)` of most `+dejs` functions. There are only two `+dejs` functions that fit this - [`+un`](/docs/hoon/reference/zuse/2d_6#undejsformat) and [`+uf`](/docs/hoon/reference/zuse/2d_6#ufdejsformat). These are intended to be used with `+ou` - you would wrap each function in the key-function list of `+ou` with either `+un` or `+uf`.
297+
298+`+un` crashes if its argument is `~`. `+ou` gives functions a `~` if the matching key-value pair is missing in the `$json` object, so `+un` crashes if the key-value pair is missing. Therefore, `+un` lets you set key-value pairs as mandatory.
299+
300+`+uf` takes two arguments - a noun and a `+dejs` function. If the `(unit json)` it's given by `+ou` is `~`, it produces the noun it was given rather than the product of the `+dejs` function. This lets you specify key-value pairs as optional, replacing missing ones with whatever you want.
301+
302+Let's look at a practical example. Here's a generator you can save in the `%base` desk of a fake ship in `gen/ou-test.hoon` and `|commit %base`. It takes in a `$json` object and produces a triple. The `+ou` in `+decode` has three key-function pairs - the first two are mandatory and the last is optional, producing the bunt of a set if the `%baz` key is missing.
303+
304+#### `ou-test.hoon`
305+
306+```hoon
307+|= jon=json
308+|^ ^- [@t ? (set @ud)]
309+(decode jon)
310+++ decode
311+=, dejs:format
312+%- ou
313+:~ foo+(un so)
314+ bar+(un bo)
315+ baz+(uf *(set @ud) (as ni))
316+==
317+--
318+```
319+
320+Let's try it:
321+
322+```
323+> +ou-test (need (de-json:html '{"foo":"hello","bar":true,"baz":[1,2,3,4]}'))
324+['hello' %.y {1 2 3 4}]
325+
326+> +ou-test (need (de-json:html '{"foo":"hello","bar":true}'))
327+['hello' %.y {}]
328+
329+> +ou-test (need (de-json:html '{"foo":"hello"}'))
330+[%key 'bar']
331+dojo: hoon expression failed
332+```
333+
334+### `+su`
335+
336+The [`+su`](/docs/hoon/reference/zuse/2d_6#sudejsformat) function parses a string with the given parsing `rule`. Hoon's functional parsing library is very powerful and lets you create arbitrarily complex parsers. JSON will often have data types encoded in strings, so this function can be very useful. The writing of parsers is outside the scope of this guide, but you can see the [Parsing Guide](/docs/hoon/guides/parsing) and sections 4e to 4j of the standard library documentation for details.
337+
338+Here are some simple examples of using `+su` to parse strings:
339+
340+```
341+> `@ux`((su:dejs:format hex) s+'deadbeef1337f00D')
342+0xdead.beef.1337.f00d
343+
344+> `(list @)`((su:dejs:format (most lus dem)) s+'1+2+3+4')
345+~[1 2 3 4]
346+
347+> `@ub`((su:dejs:format ven) s+'+>-<->+<+')
348+0b11.1000.1101
349+```
350+
351+Here's a more complex parser that will parse a GUID like `824e7749-4eac-9c00-db16-4cb816cd6f19` to a `@ux`:
352+
353+#### `su-test.hoon`
354+
355+```hoon
356+|= jon=json
357+^- @ux
358+%. jon
359+%- su:dejs:format
360+%+ cook
361+|= parts=(list [step @])
362+^- @ux
363+(can 3 (flop parts))
364+;~ plug
365+ (stag 4 ;~(sfix (bass 16 (stun 8^8 six:ab)) hep))
366+ (stag 2 ;~(sfix qix:ab hep))
367+ (stag 2 ;~(sfix qix:ab hep))
368+ (stag 2 ;~(sfix qix:ab hep))
369+ (stag 6 (bass 16 (stun 12^12 six:ab)))
370+ (easy ~)
371+==
372+```
373+
374+Save it in the `/gen` directory of the `%base` desk and `|commit` it. We can then try it with:
375+
376+```
377+> +su-test s+'5323a61d-0c26-d8fa-2b73-18cdca805fd8'
378+0x5323.a61d.0c26.d8fa.2b73.18cd.ca80.5fd8
379+```
380+
381+If we delete the last character it'll no longer be a valid GUID and the parsing will fail:
382+
383+```
384+> +su-test s+'5323a61d-0c26-d8fa-2b73-18cdca805fd'
385+/gen/su-test/hoon:<[2 1].[16 3]>
386+/gen/su-test/hoon:<[3 1].[16 3]>
387+{1 36}
388+syntax error
389+dojo: naked generator failure
390+```
391+
392+## `mark` file example
393+
394+Here's a simple `mark` file for the `$user` structure we created in the [first example](#json-encoding-and-decoding-example). It imports the [json-test.hoon](#json-testhoon) library we created and saved in our `%base` desk's `/lib` directory.
395+
396+#### `user.hoon`
397+
398+```hoon
399+/+ *json-test
400+|_ usr=user
401+++ grab
402+ |%
403+ ++ noun user
404+ ++ json from-js
405+ --
406+++ grow
407+ |%
408+ ++ noun usr
409+ ++ json (to-js usr)
410+ --
411+++ grad %noun
412+--
413+```
414+
415+The [Marks section](/docs/arvo/clay/marks/marks) of the Clay documentation covers `mark` files comprehensively and is worth reading through if you want to write a mark file.
416+
417+In brief, a mark file contains a `door` with three arms. The door's sample type is the type of the data in question - in our case the `$user` structure. The `+grab` arm contains methods for converting _to_ our mark, and the `+grow` arm contains methods for converting _from_ our mark. The `+noun` arms are mandatory, and then we've added `+json` arms which respectively call the `+from-js` and `+to-js` functions from our `json-test.hoon` library. The final `+grad` arm defines various revision control functions, in our case we've delegated these to the `%noun` mark.
418+
419+From this mark file, Clay can build mark conversion gates between the `%json` mark and our `%user` mark, allowing the conversion of `$json` data to a `$user` structure and vice versa.
420+
421+### Try it out
422+
423+First, we'll save the code above as `user.hoon` in the `/mar` directory our of `%base` desk:
424+
425+```
426+> |commit %base
427+>=
428++ /~zod/base/9/mar/user/hoon
429+```
430+
431+Let's quickly create a `$json` object to work with:
432+
433+```
434+> =jon (need (de-json:html '{"joined":1631440078,"username":"john456","name":["John","William","Smith"],"email":"john.smith@example.com"}'))
435+> jon
436+[ %o
437+ p
438+ { [p='email' q=[%s p='john.smith@example.com']]
439+ [p='name' q=[%a p=~[[%s p='John'] [%s p='William'] [%s p='Smith']]]]
440+ [p='username' q=[%s p='john456']]
441+ [p='joined' q=[%n p=~.1631440078]]
442+ }
443+]
444+```
445+
446+We'll also build our library so we can use its types from the dojo:
447+
448+```
449+> =user-lib -build-file %/lib/json-test/hoon
450+```
451+
452+Now we can ask Clay to build a mark conversion gate from a `%json` mark to our `%user` mark. We'll use a scry with a `%f` `care` which produces a static mark conversion gate:
453+
454+```
455+> =json-to-user .^($-(json user:user-lib) %cf /===/json/user)
456+```
457+
458+Let's try converting our `$json` to a `$user` structure with our new mark conversion gate:
459+
460+```
461+> =usr (json-to-user jon)
462+> usr
463+[ username='john456'
464+ name=[first='John' mid='William' last='Smith']
465+ joined=~2021.9.12..09.47.58
466+ email='john.smith@example.com'
467+]
468+```
469+
470+Now let's try the other direction. We'll again scry Clay to build a static mark conversion gate, this time _from_ `%user` _to_ `%json` rather than the reverse:
471+
472+```
473+> =user-to-json .^($-(user:user-lib json) %cf /===/user/json)
474+```
475+
476+Let's test it out by giving it our `$user` data:
477+
478+```
479+> (user-to-json usr)
480+[ %o
481+ p
482+ { [p='email' q=[%s p='john.smith@example.com']]
483+ [p='name' q=[%a p=~[[%s p='John'] [%s p='William'] [%s p='Smith']]]]
484+ [p='username' q=[%s p='john456']]
485+ [p='joined' q=[%n p=~.1631440078]]
486+ }
487+]
488+```
489+
490+Finally, let's see how that looks as JSON encoded in text:
491+
492+```
493+> (crip (en-json:html (user-to-json usr)))
494+'{"joined":1631440078,"username":"john456","name":["John","William","Smith"],"email":"john.smith@example.com"}'
495+```
496+
497+Usually (though not in all cases) these mark conversions will be performed implicitly by Gall or Eyre and you'd not deal with the mark conversion gates directly, but it's still informative to see them work explicitly.
498+
499+## Further reading
500+
501+[The Zuse library reference](/docs/hoon/reference/zuse/table-of-contents) - This includes documentation of the JSON parsing, printing, encoding and decoding functions.
502+
503+[The Marks section of the Clay documentation](/docs/arvo/clay/marks/marks) - Comprehensive documentation of `mark`s.
504+
505+[The External API Reference section of the Eyre documentation](/docs/arvo/eyre/external-api-ref) - Details of the webserver vane Eyre's external API.
506+
507+[The Iris documentation](/docs/arvo/iris/iris) - Details of the web client vane Iris, which may be used to fetch external JSON data among other things.
508+
509+[Strings Guide](/docs/hoon/guides/strings) - Atom printing functions like `+scot` will often be useful for JSON encoding - see the [Encoding in Text](/docs/hoon/guides/strings#encoding-in-text) section for usage.
510+
511+[Parsing Guide](/docs/hoon/guides/parsing) - Learn how to write functional parsers in hoon which can be used with `+su`.
content/docs/hoon/guides/parsing.md+48−21
@@ -9,6 +9,8 @@ knowledge of parsing is required, and we will explain the basic structure of how
99parsing works in a purely functional language such as Hoon before moving on to
1010how it is implemented in Hoon.
1111
12+**Note:** For JSON printing/parsing and encoding/decoding, see the [JSON Guide](/docs/hoon/guides/json-guide).
13+
1214## What is parsing? {#what-is-parsing}
1315
1416A program which takes a raw sequence of characters as an input and produces a data
@@ -42,20 +44,24 @@ the parsing routine on the string `"1"` and looks for an integer, and so it
4244returns the integer `1`. However, taking into account what was said above about
4345parsers returning the unparsed portion of the string as well, we should
4446represent this return as a tuple. So we should expect something like this:
47+
4548```
4649> (parse "1" integer)
4750[1 ""]
4851> (parse "123" integer)
4952[1 "23"]
5053```
54+
5155What if we wish to parse the rest of the string? We would need to apply the
5256`parse` function again:
57+
5358```
5459> (parse (parse "123" integer) integer)
5560[12 "3"]
5661> (parse (parse (parse "123" integer) integer) integer)
5762[123 ""]
5863```
64+
5965So we see that we can parse strings larger than one character by stringing
6066together parsing functions for single characters. Thus in addition to parsing
6167functions for single input characters, we want _parser combinators_ that
@@ -75,15 +81,14 @@ are quite a few), but ought to be
7581sufficient to get started with parsing in Hoon and be equipped to discover the
7682remainder yourself.
7783
78−
7984## Basic types
8085
8186In this section we discuss the types most commonly used for Hoon parsers. In short:
8287
83− - A `hair` is the position in the text the parser is at,
84− - A `nail` is parser input,
85− - An `edge` is parser output,
86− - A `rule` is a parser.
88+- A `hair` is the position in the text the parser is at,
89+- A `nail` is parser input,
90+- An `edge` is parser output,
91+- A `rule` is a parser.
8792
8893### `hair`
8994
@@ -102,6 +107,7 @@ is in case it hits something unexpected during parsing.
102107```hoon
103108++ nail [p=hair q=tape]
104109```
110+
105111We recall from our [discussion above](#what-is-parsing) that parsing functions must keep
106112track of both what has been parsed and what has yet to be parsed. Thus a `nail`
107113consists of both a `hair`, giving the line and column up to which the input
@@ -122,6 +128,7 @@ what has theoretically already been parsed up to that point.
122128```hoon
123129++ edge [p=hair q=(unit [p=* q=nail])]
124130```
131+
125132An `edge` is the output of a parser. If parsing succeeded, `p` is the location
126133of the original input `tape `up to which the text has been parsed. If parsing
127134failed, `p` will be the first `hair` at which parsing failed.
@@ -136,6 +143,7 @@ of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.
136143```hoon
137144++ rule _|:($:nail $:edge)
138145```
146+
139147A `rule` is a gate which takes in a `nail` and returns an `edge` - in other
140148words, a parser.
141149
@@ -160,6 +168,7 @@ of the input `nail`.
160168> edg
161169[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]
162170```
171+
163172We note that `p.edg` is `[p=1 q=2]`, indicating that the next character to be
164173parsed is in line 1, column 2. `q.edg` is not null, indicating that parsing
165174succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is the same as `p.edg`, which is always the case for
@@ -167,11 +176,13 @@ succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is
167176`q.q.edg` is `"bc"`, which is the part of the input `tape` that has yet to be parsed.
168177
169178Now let's see what happens when parsing fails.
179+
170180```
171181> =edg ((just 'b') [[1 1] "abc"])
172182> edg
173183[p=[p=1 q=1] q=~]
174184```
185+
175186Now we have that `p.edg` is the same as the input `hair`, `[1 1]`, meaning the
176187parser has not advanced since parsing failed. `q.edg` is null, indicating that
177188parsing has failed.
@@ -186,11 +197,13 @@ produces a `rule` that attempts to match that `cord` against the beginning of
186197the input.
187198
188199Let's see what happens when we successfully parse the entire input `tape`.
200+
189201```
190202> =edg ((jest 'abc') [[1 1] "abc"])
191203> edg
192204[p=[p=1 q=4] q=[~ [p='abc' q=[p=[p=1 q=4] q=""]]]]
193205```
206+
194207`p.edg` is `[p=1 q=4]`, indicating that the next character to be parsed is at
195208line 1, column 4. Of course, this does not exist since the input `tape` was only
1962093 characters long, so this actually indicates that the entire `tape` has been
@@ -199,21 +212,25 @@ successfully parsed (since the `hair` does not advance in the case of failure).
199212remains to be parsed.
200213
201214What happens if we only match some of the input `tape`?
215+
202216```
203217> =edg ((jest 'ab') [[1 1] "abc"])
204218> edg
205219[p=[p=1 q=3] q=[~ [p='ab' q=[p=[p=1 q=3] q="c"]]]]
206220```
221+
207222Now we have that the result, `p.q.edg`, is `'ab'`, while the remainder `q.q.q.edg`
208223is `"c"`. So `+jest` has successfully parsed the first two characters, while the
209224last character remains. Furthermore, we still have the information that the
210225remaining character was in line 1 column 3 from `p.edg` and `p.q.q.edg`.
211226
212227What happens when `+jest` fails?
228+
213229```
214230> ((jest 'bc') [[1 1] "abc"])
215231[p=[p=1 q=1] q=~]
216232```
233+
217234Despite the fact that `'bc'` appears in `"abc"`, because it was not at the
218235beginning the parse failed. We will see in [parser
219236combinators](#parser-combinators) how to modify this `rule` so that it
@@ -237,6 +254,7 @@ replaces the parsing result with `cus`.
237254
238255Here we see that `p.q` of the `edge` returned by the `rule` created with `+cold`
239256is `%foo`.
257+
240258```
241259> ((cold %foo (just 'a')) [[1 1] "abc"])
242260[p=[p=1 q=2] q=[~ u=[p=%foo q=[p=[p=1 q=2] q="bc"]]]]
@@ -246,7 +264,6 @@ One common scenario where `+cold` sees play is when writing [command line
246264interface (CLI) apps](/docs/hoon/guides/cli-tutorial). We usher the
247265reader there to find an example where `+cold` is used.
248266
249−
250267### `+knee`
251268
252269Another important function in the parser builder library is `+knee`, used for building
@@ -254,7 +271,6 @@ recursive parsers. We delay discussion of `+knee` to the
254271[section below](#recursive-parsers) as more context is needed to explain it
255272properly.
256273
257−
258274## Outside callers
259275
260276Since `hair`s, `nail`s, etc. are only utilized within the context of writing
@@ -295,7 +311,6 @@ For additional information including additional examples see [4g: Parsing (Outsi
295311For the remainder of this tutorial we will make use of `+scan` so that we do not
296312need to deal directly with `nail`s except where it is ilustrative to do so.
297313
298−
299314### Parsing atoms
300315
301316[Recall](/docs/hoon/hoon-school/lists) that `cord`s are atoms with the aura
@@ -312,7 +327,6 @@ The standard library provides a number of gates that take a `rule` and
312327produce a new modified `rule` according to some process. We call these _parser modifiers_. These are
313328documented among the [parser builders](/docs/hoon/reference/stdlib/4f).
314329
315−
316330### `+ifix`
317331
318332`+ifix` modifies a `rule` so that it matches that `rule` only when it is
@@ -322,11 +336,11 @@ surrounded on both sides by text that matches a pair of `rule`s, which is discar
322336> (scan "(42)" (ifix [pal par] (jest '42')))
323337'42'
324338```
339+
325340`+pal` and `+par` are shorthand for `(just '(')` and `(just ')')`, respectively. All
326341ASCII glyphs have counterparts of this sort, documented
327342[here](/docs/hoon/reference/stdlib/4h).
328343
329−
330344### `+star` {#star}
331345
332346`+star` is used to apply a `rule` repeatedly. Recall that `+just` only parses
@@ -343,7 +357,9 @@ We can use `+star` to get the rest of the `tape`:
343357> ((star (just 'a')) [[1 1] "aaa"])
344358[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q=""]]]]
345359```
360+
346361and we note that the parsing ceases when it fails.
362+
347363```
348364> ((star (just 'a')) [[1 1] "aaab"])
349365[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q="b"]]]]
@@ -358,9 +374,11 @@ combinators](/docs/hoon/reference/stdlib/4e). First we introduce a few
358374combinators, then we examine more closely how `;~` is used to chain them together.
359375
360376The syntax to combine `rule`s is
377+
361378```hoon
362379;~(combinator rule1 rule2 ... ruleN)
363380```
381+
364382The `rule`s are composed together using the combinator as an
365383intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns
366384it into a sample (a `nail`) for the next `rule` to handle. We elaborate on this
@@ -409,42 +427,46 @@ take this opportunity to think about it carefully.
409427The `rule` created by `;~(combinator [list rule])` may be understood
410428inductively. To do this, let's consider the base case where our `[list rule]` has only a
411429single entry.
430+
412431```
413432> (scan "star" ;~(plug (jest 'star')))
414433'star'
415434```
435+
416436Our output is identical to that given by `scan "star" (jest 'star')`. This is to
417437be expected. The combinator `+plug` is specifically used for chaining together
418438`rule`s in the `[list rule]`, but if there is only one `rule`, there is nothing
419439to chain. Thus, swapping out `+plug` for another combinator makes no difference here:
440+
420441```
421442> (scan "star" ;~(pose (jest 'star')))
422443'star'
423444> (scan "star" ;~((glue com) (jest 'star')))
424445'star'
425446```
447+
426448`;~` and the combinator only begin to play a role once the `[list rule]` has at
427449least two elements. So let's look at an example done with `+plug`, the simplest
428450combinator.
451+
429452```
430453> (scan "star" ;~(plug (jest 'st') (jest 'ar')))
431454['st' 'ar']
432455```
456+
433457Our return suggests that we first parsed `"star"` with the `rule` `(jest 'st')` and passed
434−the resulting `edge` to `(jest 'ar')` - in other words, we called `+plug` on `(jest
435−'st')` and the `edge` returned once it had been used to parse `"star"`. Thus
458+the resulting `edge` to `(jest 'ar')` - in other words, we called `+plug` on `(jest 'st')` and the `edge` returned once it had been used to parse `"star"`. Thus
436459`+plug` was the glue that allowed us to join the two `rule`s, and `;~` performed
437460the gluing operation. And so, swapping `+plug` for `+pose` results in a crash,
438461which clues us into the fact that the combinator now has an effect since there
439462is more than one `rule`.
463+
440464```
441465> (scan "star" ;~(pose (jest 'st') (jest 'ar')))
442466{1 3}
443467syntax error
444468```
445469
446−
447−
448470## Parsing numbers
449471
450472Functions for parsing numbers are documented in [4j: Parsing (Bases and Base
@@ -461,15 +483,19 @@ for parsing decimal numbers.
461483## Recursive parsers
462484
463485Naively attempting to write a recursive `rule`, i.e. like
486+
464487```
465488> |-(;~(pose den ;~(pose $ (easy ~))))
466489```
490+
467491results in an error.
492+
468493```
469494-find.,.+6
470495-find.,.+6
471496rest-loop
472497```
498+
473499Thus some special sauce is required, the `+knee` function.
474500
475501`+knee` takes in a noun that is the default value of the parser, typically given
@@ -494,7 +520,6 @@ the parser to improve performance. In [parsing arithmetic
494520expressions](#parsing-arithmetic-expressions) we will be writing a recursive
495521parser making use of `+knee` and `~+` throughout.
496522
497−
498523# Parsing arithmetic expressions
499524
500525In this section we will be applying what we have learned to write a parser for
@@ -514,6 +539,7 @@ for arithmetic expressions informally.
514539First lets look at the code we're going to use, and then dive into explaining
515540it. If you'd like to follow along, save the following as `expr-parse.hoon` in
516541your `gen/` folder.
542+
517543```hoon
518544:: expr-parse: parse arithmetic expressions
519545::
@@ -544,9 +570,10 @@ your `gen/` folder.
544570```
545571
546572Informally, the grammar here is:
547− - A factor is either an integer or an expression surrounded by parentheses.
548− - A term is either a factor or a factor times a term.
549− - An expression is either a term or a term plus an expression.
573+
574+- A factor is either an integer or an expression surrounded by parentheses.
575+- A term is either a factor or a factor times a term.
576+- An expression is either a term or a term plus an expression.
550577
551578### Factors, terms, and expressions
552579
@@ -568,8 +595,7 @@ for expressions.
568595A _factor_ is either a decimal number or an expression surrounded by parentheses. Put
569596into Hoon terms, a decimal number is parsed by the `rule` `+dem` and an
570597expression is parsed by removing the surrounding parentheses and then passing
571−the result to the expression parser arm `+expr`, given by the `rule` `(ifix [pal
572−par] expr)`. Since we want to parse our expression with one or the other, we
598+the result to the expression parser arm `+expr`, given by the `rule` `(ifix [pal par] expr)`. Since we want to parse our expression with one or the other, we
573599chain these two `rule`s together using the monadic applicator rune `;~` along
574600with `+pose`, which says to try each rule in succession until one of them works.
575601
@@ -578,6 +604,7 @@ recursive rule. Thus we need to make use of `+knee`. The first argument for
578604`+knee` is `*@ud`, since our final answer should be a `@ud`.
579605
580606Then follows the definition of the gate utilized by `+knee`:
607+
581608```hoon
582609 |. ~+
583610 ;~ pose
@@ -585,6 +612,7 @@ Then follows the definition of the gate utilized by `+knee`:
585612 (ifix [pal par] expr)
586613 ==
587614```
615+
588616`~+` is used to cache the parser, so that it does not need to be computed over
589617and over again. Then it follows the `rule` we described above.
590618
@@ -615,7 +643,6 @@ accomplish this:
615643The rest of the `+expr` arm is structured just like how `+factor` is, and for
616644the same reasons.
617645
618−
619646#### Parsing terms
620647
621648A _term_ is either a factor times a term or a factor. This is handled similarly