@@ -0,0 +1,354 @@
1++++
2+title = "JSON"
3+weight = 4
4+template = "doc.html"
5++++
6+
7+If one is working on a Gall agent with any kind of web interface, it's likely one will encounter the problem of converting Hoon data structures to JSON and vice versa. This is what we'll look at in this document.
8+
9+Urbit represents JSON data with the `$json` structure defined in `lull.hoon`. You can refer to the [json type](#the-json-type) section below for details. Since JSON data on the web is typically encoded in text, Urbit has two functions in `zuse.hoon` - [+en-json:html](/docs/hoon/reference/zuse/2e_2-3#en-jsonhtml) for printing `$json` to a text-encoded form, and [+de-json:html](/docs/hoon/reference/zuse/2e_2-3#de-jsonhtml) for parsing text-encoded JSON to a `$json` structure.
10+
11+Since one typically wants `$json` data converted to some other `noun` structure or vice versa, Urbit has three collections of functions for this purpose, also in `zuse.hoon`. The first is [+enjs:format](/docs/hoon/reference/zuse/2d_1-5#enjsformat), which contains functions for converting various atoms and structures to `$json`. The second is [+dejs:format](/docs/hoon/reference/zuse/2d_6#dejsformat), which contains many JSON "reparsers" for converting `$json` data to atoms and other structures. The third is [+dejs-soft:format](/docs/hoon/reference/zuse/2d_7#dejs-softformat), which is largely the same as `+dejs:format` except its reparsers produce `unit`s which are null upon failure rather than simply crashing.
12+
13+The relationship between these types and functions look like this:
14+
15+
16+
17+Note this diagram is a simplification - the `+dejs:format` and `+enjs:format` collections in particular are tools to be used in writing conversion functions rather than simply being used by themselves, but it demonstrates the basic relationships. Additionally, it would be less common to perform the printing/parsing directly - this would typically be handled implicitly by Eyre, though it may be necessary if one is requesting external JSON data via the web client vane Iris.
18+
19+### In practice
20+
21+A typical Gall agent will have a number of structures defined in a file in the `/sur` directory. These will define the type of data it expects to be `%poke`ed with, the type of data it will `%give` to subscribers, and the type of data its scry endpoints produce.
22+
23+If the agent is expected to only interact with local agents or agents on remote ships over Ames, it may just take and produce its structures with a `%noun` `mark`. If, however, it needs to talk to a web interface of some kind, it usually must take and produce `$json` data with a `%json` mark.
24+
25+It may be the case that its dealings with a web interface are totally distinct from its dealings with other agents. If so, the agent could just have separate scry endpoints, poke handling code, etc, that just directly take and produce `$json` data with a `%json` mark. In such a case one can include `$json` encoding/decoding functions directly in the agent or associated libraries, using the general techniques demonstrated in the [$json encoding and decoding example](#json-encoding-and-decoding-example) section below.
26+
27+If, on the other hand, one wants to be able to use the same basic structures in requests and responses, and the same code to deal with them, whether from the web or within Urbit, a different approach is necessary. Rather than taking or producing either `%noun` or `%json` marked data, custom `mark` files can be created for the relevant structures, which specify conversion methods for both `%noun` and `%json` marked data.
28+
29+With this approach, an agent would take and/or produce data with some `mark` like `%my-custom-mark`. Then, when the agent is queried from a web client or must produce data for a subscribed web client, the webserver vane Eyre can automatically convert `%my-custom-mark` to `%json` or vice versa. Otherwise, when interacting within Urbit, it can deal with its ordinary data types. This approach is used by `%graph-store` with its `%graph-update-2` mark, for example, and a number of other agents.
30+
31+For details of creating a `mark` file for this purpose, the [mark file example](#mark-file-example) section below walks through a practical example.
32+
33+## The `$json` type
34+
35+Urbit represents JSON data with the `$json` structure (defined in `/sys/lull.hoon`):
36+
37+```hoon
38++$ json :: normal json value
39+ $@ ~ :: null
40+ $% [%a p=(list json)] :: array
41+ [%b p=?] :: boolean
42+ [%o p=(map @t json)] :: object
43+ [%n p=@ta] :: number
44+ [%s p=@t] :: string
45+ == ::
46+```
47+
48+The correspondence of `$json` to JSON types is fairly self-evident, but here's a table comparing the two for additional clarity:
49+
50+| JSON Type | `$json` Type | JSON Example | `$json` Example |
51+| :-------- | :--------------------- | ------------------------- | :----------------------------------------------------------- |
52+| Null | `~` | `null` | `~` |
53+| Boolean | `[%b p=?]` | `true` | `[%b p=%.y]` |
54+| Number | `[%n p=@ta]` | `123` | `[%n p=~.123]` |
55+| String | `[%s p=@t]` | `"foo"` | `[%s p='foo']` |
56+| Array | `[%a p=(list json)]` | `["foo",123]` | `[%a p=~[[%s p='foo'] [%n p=~.123]]]` |
57+| Object | `[%o p=(map @t json)]` | `{"foo":"xyz","bar":123}` | `[%o p={[p='bar' q=[%n p=~.123]] [p='foo' q=[%s p='xyz']]}]` |
58+
59+Since the `$json` `%o` object and `%a` array types may themselves contain any `$json`, you can see how JSON structures of arbitrary complexity can be represented. Note the `%n` number type is a `@ta` rather than something like a `@ud` that one might expect. This is because JSON's number type may be either an integer or floating point, so it's left as a `knot` which can then be parsed to a `@ud` or `@rd` with the appropriate [+dejs:format](/docs/hoon/reference/zuse/2d_6) function.
60+
61+## `$json` encoding and decoding example
62+
63+Let's have a look at a practical example. Here's a core with three arms. It has the structure arm `$user`, and then two more: `+to-js` converts a `$user` structure to `$json`, and `+from-js` does the opposite. Usually we'd define the structure in a separate `/sur` file, but for simplicity it's all in the one core.
64+
65+#### `json-test.hoon`
66+
67+```hoon
68+|%
69++$ user
70+ $: username=@t
71+ name=[first=@t mid=@t last=@t]
72+ joined=@da
73+ email=@t
74+ ==
75+++ to-js
76+ |= usr=user
77+ |^ ^- json
78+ :- %o
79+ %- malt
80+ %- limo
81+ :~
82+ ['username' s+username.usr]
83+ ['name' name]
84+ ['joined' (sect:enjs:format joined.usr)]
85+ ['email' s+email.usr]
86+ ==
87+ ++ name
88+ :- %a
89+ :~
90+ [%s first.name.usr]
91+ [%s mid.name.usr]
92+ [%s last.name.usr]
93+ ==
94+ --
95+++ from-js
96+ =, dejs:format
97+ |= jon=json
98+ ^- user
99+ %. jon
100+ %- ot
101+ :~
102+ [%username so]
103+ [%name (at ~[so so so])]
104+ [%joined du]
105+ [%email so]
106+ ==
107+--
108+```
109+
110+### Try it out
111+
112+First we'll try using our `$json` encoding/decoding library, and afterwards we'll take a closer look at its construction. To begin, save the code above in `/lib/json-test.hoon` and `|commit` it:
113+
114+```
115+> |commit %home
116+>=
117++ /~zod/home/5/lib/json-test/hoon
118+```
119+
120+Then we need to build it so we can use it. We'll give it a face of `user-lib`:
121+
122+```
123+> =user-lib -build-file %/lib/json-test/hoon
124+```
125+
126+Let's now create an example of a `$user` structure:
127+
128+```
129+> =usr `user:user-lib`['john456' ['John' 'William' 'Smith'] now 'john.smith@example.com']
130+> usr
131+[ username='john456'
132+ name=[first='John' mid='William' last='Smith']
133+ joined=~2021.9.12..09.47.58..1b65
134+ email='john.smith@example.com'
135+]
136+```
137+
138+Now we can try calling the `+to-js` function with our data to convert it to `$json`:
139+
140+```
141+> =usr-json (to-js:user-lib usr)
142+> usr-json
143+[ %o
144+ p
145+ { [p='email' q=[%s p='john.smith@example.com']]
146+ [p='name' q=[%a p=~[[%s p='John'] [%s p='William'] [%s p='Smith']]]]
147+ [p='username' q=[%s p='john456']]
148+ [p='joined' q=[%n p=~.1631440078]]
149+ }
150+]
151+```
152+
153+Let's also see how that `$json` would look as real JSON encoded in text. We can do that with `+en-json:html`:
154+
155+```
156+> (crip (en-json:html (to-js:user-lib usr)))
157+'{"joined":1631440078,"username":"john456","name":["John","William","Smith"],"email":"john.smith@example.com"}'
158+```
159+
160+Finally, let's try converting the `$json` back to a `$user` again with our `+from-js` arm:
161+
162+```
163+> (from-js:user-lib usr-json)
164+[ username='john456'
165+ name=[first='John' mid='William' last='Smith']
166+ joined=~2021.9.12..09.47.58
167+ email='john.smith@example.com'
168+]
169+```
170+
171+### Analysis
172+
173+#### Converting to `$json`
174+
175+Here's our arm that converts a `$user` structure to `$json`:
176+
177+```hoon
178+++ to-js
179+ |= usr=user
180+ |^ ^- json
181+ :- %o
182+ %- malt
183+ %- limo
184+ :~
185+ ['username' s+username.usr]
186+ ['name' name]
187+ ['joined' (sect:enjs:format joined.usr)]
188+ ['email' s+email.usr]
189+ ==
190+ ++ name
191+ :- %a
192+ :~
193+ [%s first.name.usr]
194+ [%s mid.name.usr]
195+ [%s last.name.usr]
196+ ==
197+ --
198+```
199+
200+There are different ways we could represent our `$user` structure as JSON, but in this case we've opted to encapsulate it in an object and have the `name` as an array (since JSON arrays preserve order).
201+
202+Since an object is represented as a `[%o p=(map @t json)]`, we can use `malt` to convert a `list` of key-value pairs to a `(map @t json)`.
203+
204+For the `joined` field, we've used the [+sect](/docs/hoon/reference/zuse/2d_1-5#sectenjsformat) function from [+enjs:format](/docs/hoon/reference/zuse/2d_1-5#enjsformat) to convert the `@da` to a Unix seconds timestamp in a `$json` number. The `+sect` function, like others in `+enjs`, takes in a noun (in this case a `@da`) and produces `$json` (in this case a `[%n @ta]` number). `+enjs` contains a handful of useful functions like this, but for the rest we've just hand-made the `$json` structure. This is fairly typical when encoding `$json`, it's usually [decoding](#converting-from-json) that makes more extensive use of the `$json` utility functions in `+format`.
205+
206+For the `name` field we've just formed a cell of `%a` and a list of `$json` strings, since a `$json` array is `[%a p=(list json)]`. Note we've separated this part into its own arm and wrapped the whole thing in a `|^` - a core with a `$` arm that's computed immediately. This is simply for readability - our structure here is quite simple but when dealing with deeply-nested `$json` structures or complex logic, having a single giant function can quickly become unwieldy.
207+
208+#### Converting from `$json`
209+
210+Here's our arm that converts `$json` to our `$user` structure:
211+
212+```hoon
213+++ from-js
214+ =, dejs:format
215+ |= jon=json
216+ ^- user
217+ %. jon
218+ %- ot
219+ :~
220+ [%username so]
221+ [%name (at ~[so so so])]
222+ [%joined du]
223+ [%email so]
224+ ==
225+```
226+
227+This is the inverse of the [encoding](#converting-to-json) function described in the previous section.
228+
229+We make extensive use of [+dejs:format](/docs/hoon/reference/zuse/2d_6) functions here, so we've used `=,` to expose the namespace and allow succinct `+dejs` function calls.
230+
231+We use the [+ot](/docs/hoon/reference/zuse/2d_6#otdejsformat) function from `+dejs:format` to decode the `$json` object to a n-tuple. It's a wet gate that takes a list of pairs of keys and other `+dejs` functions and produces a new gate that takes the `$json` to be decoded (which we've given it in `jon`).
232+
233+The [+so](/docs/hoon/reference/zuse/2d_6#sodejsformat) functions just decode `$json` strings to `cord`s. The [+at](/docs/hoon/reference/zuse/2d_6#atdejsformat) function converts a `$json` array to a tuple, decoding each element with the respective function given in its argument list. Like `+ot`, `+at` is also a wet gate that produces a gate that takes `$json`. In our case we've used `+so` for each element, since they're all strings.
234+
235+For `joined`, we've used the [+du](/docs/hoon/reference/zuse/2d_6#dudejsformat) function, which converts a Unix seconds timestamp in a `$json` number to a `@da` (it's basically the inverse of the `+sect:enjs:format` we used earlier).
236+
237+There are dozens of different functions in [+dejs:format](/docs/hoon/reference/zuse/2d_6) that will cover a great many use cases. If there isn't a `+dejs` function for a particular case, one can also just write a custom function - it just has to take `$json`. Note there's also the [+dejs-soft:format](/docs/hoon/reference/zuse/2d_7) functions - these are similar to `+dejs` functions except they produce `unit`s rather than simply crashing if decoding fails.
238+
239+## `mark` file example
240+
241+Here's a simple `mark` file for the `$user` structure we created in the [previous section](#json-encoding-and-decoding-example). It imports the [json-test.hoon](#json-testhoon) library we created and saved in our `%home` desk's `/lib` directory.
242+
243+#### `user.hoon`
244+
245+```hoon
246+/+ *json-test
247+|_ usr=user
248+++ grab
249+ |%
250+ ++ noun user
251+ ++ json from-js
252+ --
253+++ grow
254+ |%
255+ ++ noun usr
256+ ++ json (to-js usr)
257+ --
258+++ grad %noun
259+--
260+```
261+
262+The [Marks section](/docs/arvo/clay/marks/marks) of the Clay documentation covers `mark` files comprehensively and is worth reading through if one wants to write a mark file.
263+
264+In brief, a mark file contains a `door` with three arms. The door's sample type is the type of the data in question - in our case the `$user` structure. The `+grab` arm contains methods for converting _to_ our mark, and the `+grow` arm contains methods for converting _from_ our mark. The `+noun` arms are mandatory, and then we've added `+json` arms which respectively call the `+from-js` and `+to-js` functions from our `json-test.hoon` library. The final `+grad` arm defines various revision control functions, in our case we've delegated these to the `%noun` mark.
265+
266+From this mark file, Clay can build mark conversion `gate`s between the `%json` mark and our `%user` mark, allowing the conversion of `$json` data to a `$user` structure and vice versa.
267+
268+### Try it out
269+
270+First, we'll save the code above as `user.hoon` in the `/mar` directory our of `%home` desk:
271+
272+```
273+> |commit %home
274+>=
275++ /~zod/home/9/mar/user/hoon
276+```
277+
278+Let's quickly create a `$json` object to work with:
279+
280+```
281+> =jon (need (de-json:html '{"joined":1631440078,"username":"john456","name":["John","William","Smith"],"email":"john.smith@example.com"}'))
282+> jon
283+[ %o
284+ p
285+ { [p='email' q=[%s p='john.smith@example.com']]
286+ [p='name' q=[%a p=~[[%s p='John'] [%s p='William'] [%s p='Smith']]]]
287+ [p='username' q=[%s p='john456']]
288+ [p='joined' q=[%n p=~.1631440078]]
289+ }
290+]
291+```
292+
293+We'll also build our library so we can use its types from the Dojo:
294+
295+```
296+> =user-lib -build-file %/lib/json-test/hoon
297+```
298+
299+Now we can ask Clay to build a mark conversion gate from a `%json` mark to our `%user` mark. We'll use a scry with a `%f` `care` which produces a static mark conversion gate:
300+
301+```
302+> =json-to-user .^($-(json user:user-lib) %cf /===/json/user)
303+```
304+
305+Let's try convert our `$json` to a `$user` structure with our new mark conversion gate:
306+
307+```
308+> =usr (json-to-user jon)
309+> usr
310+[ username='john456'
311+ name=[first='John' mid='William' last='Smith']
312+ joined=~2021.9.12..09.47.58
313+ email='john.smith@example.com'
314+]
315+```
316+
317+Now let's try the other direction. We'll again scry Clay to build a static mark conversion gate, this time _from_ `%user` _to_ `%json` rather than the reverse:
318+
319+```
320+> =user-to-json .^($-(user:user-lib json) %cf /===/user/json)
321+```
322+
323+Let's test it out by giving it our `$user` data:
324+
325+```
326+> (user-to-json usr)
327+[ %o
328+ p
329+ { [p='email' q=[%s p='john.smith@example.com']]
330+ [p='name' q=[%a p=~[[%s p='John'] [%s p='William'] [%s p='Smith']]]]
331+ [p='username' q=[%s p='john456']]
332+ [p='joined' q=[%n p=~.1631440078]]
333+ }
334+]
335+```
336+
337+Finally, let's see how that looks as JSON encoded in text:
338+
339+```
340+> (crip (en-json:html (user-to-json usr)))
341+'{"joined":1631440078,"username":"john456","name":["John","William","Smith"],"email":"john.smith@example.com"}'
342+```
343+
344+Usually (though not in all cases) these mark conversions will be performed implicitly by Gall or Eyre and you'd not deal with the mark conversion gates directly, but it's still informative to see them work explicitly.
345+
346+## Further reading
347+
348+[The Zuse library reference](/docs/hoon/reference/zuse/table-of-contents) - This includes documentation of the JSON parsing, printing, encoding and decoding functions.
349+
350+[The Marks section of the Clay documentation](/docs/arvo/clay/marks/marks) - Comprehensive documentation of `mark`s.
351+
352+[The External API Reference section of the Eyre documentation](/docs/arvo/eyre/external-api-ref) - Details of the webserver vane Eyre's external API.
353+
354+[The Iris documentation](/docs/arvo/iris/iris) - Details of the web client vane Iris, which may be used to fetch external JSON data among other things.