f7923d8 · urbit.org · git.iko.soy

git.iko.soy/urbit.org

on master help sign in

Merge pull request #1131 from tinnus-napbus/json-guide

f7923d8tinnus-napbus5 years ago

content/docs/hoon/guides/_index.md+2

compare with master

unifiedsplit

@@ -13,3 +13,5 @@ insert_anchor_links = "right"

13## [Parsing](/docs/hoon/guides/parsing)

15## [Strings](/docs/hoon/guides/strings)

16+

17+## [JSON](/docs/hoon/guides/json-guide)

content/docs/hoon/guides/json-guide.mdnew file+511

@@ -0,0 +1,511 @@

1++++

2+title = "JSON"

3+weight = 4

4+template = "doc.html"

5++++

7+If you are working on a Gall agent with any kind of web interface, it's likely you will encounter the problem of converting Hoon data structures to JSON and vice versa. This is what we'll look at in this document.

9+Urbit represents JSON data with the `$json` structure (defined in `lull.hoon`). You can refer to the [json type](#the-json-type) section below for details.

10+

11+JSON data on the web is encoded in text, so Urbit has two functions in `zuse.hoon` for dealing with this:

12+

13+- [`+en-json:html`](/docs/hoon/reference/zuse/2e_2-3#en-jsonhtml) - For printing `$json` to a text-encoded form.

14+- [`+de-json:html`](/docs/hoon/reference/zuse/2e_2-3#de-jsonhtml) - For parsing text-encoded JSON to a `$json` structure.

15+

16+You typically want `$json` data converted to some other `noun` structure or vice versa, so Urbit has three collections of functions for this purpose, also in `zuse.hoon`:

17+

18+- [`+enjs:format`](/docs/hoon/reference/zuse/2d_1-5#enjsformat) - Functions for converting various atoms and structures to `$json`.

19+- [`+dejs:format`](/docs/hoon/reference/zuse/2d_6#dejsformat) - Many "reparsers" for converting `$json` data to atoms and other structures.

20+- [`+dejs-soft:format`](/docs/hoon/reference/zuse/2d_7#dejs-softformat) - Largely the same as `+dejs:format` except its reparsers produce `unit`s which are null upon failure rather than simply crashing.

21+

22+The relationship between these types and functions look like this:

23+

24+![json diagram](https://media.urbit.org/docs/json-diagram.svg)

25+

26+Note this diagram is a simplification - the `+dejs:format` and `+enjs:format` collections in particular are tools to be used in writing conversion functions rather than simply being used by themselves, but it demonstrates the basic relationships. Additionally, it is less common to do printing/parsing manually - this would typically be handled automatically by Eyre, though it may be necessary if you're retrieving JSON data via the web client vane Iris.

27+

28+### In practice

29+

30+A typical Gall agent will have a number of structures defined in a file in the `/sur` directory. These will define the type of data it expects to be `%poke`ed with, the type of data it will `%give` to subscribers, and the type of data its scry endpoints produce.

31+

32+If the agent only interacts with other agents inside Urbit, it may just use a `%noun` `mark`. If, however, it needs to talk to a web interface of some kind, it usually must handle `$json` data with a `%json` mark.

33+

34+Sometimes an agent's interactions with a web interface are totally distinct from its interactions with other agents. If so, the agent could just have separate scry endpoints, poke handlers, etc, that directly deal with `$json` data with a `%json` mark. In such a case, you can include `$json` encoding/decoding functions directly in the agent or associated libraries, using the general techniques demonstrated in the [$json encoding and decoding example](#json-encoding-and-decoding-example) section below.

35+

36+If, on the other hand, you want a unified interface (whether interacting with a web client or within Urbit), a different approach is necessary. Rather than taking or producing either `%noun` or `%json` marked data, custom `mark` files can be created which specify conversion methods for both `%noun` and `%json` marked data.

37+

38+With this approach, an agent would take and/or produce data with some `mark` like `%my-custom-mark`. Then, when the agent must interact with a web client, the webserver vane Eyre can automatically convert `%my-custom-mark` to `%json` or vice versa. This way the agent only ever has to handle the `%my-custom-mark` data. This approach is used by `%graph-store` with its `%graph-update-2` mark, for example, and a number of other agents.

39+

40+For details of creating a `mark` file for this purpose, the [mark file example](#mark-file-example) section below walks through a practical example.

41+

42+## The `$json` type

43+

44+Urbit represents JSON data with the `$json` structure (defined in `/sys/lull.hoon`):

45+

46+```hoon

47++$ json :: normal json value

48+ $@ ~ :: null

49+ $% [%a p=(list json)] :: array

50+ [%b p=?] :: boolean

51+ [%o p=(map @t json)] :: object

52+ [%n p=@ta] :: number

53+ [%s p=@t] :: string

54+ == ::

55+```

56+

57+The correspondence of `$json` to JSON types is fairly self-evident, but here's a table comparing the two for additional clarity:

58+

60+| :-------- | :--------------------- | ------------------------- | :----------------------------------------------------------- |

61+| Null | `~` | `null` | `~` |

62+| Boolean | `[%b p=?]` | `true` | `[%b p=%.y]` |

63+| Number | `[%n p=@ta]` | `123` | `[%n p=~.123]` |

64+| String | `[%s p=@t]` | `"foo"` | `[%s p='foo']` |

66+| Object | `[%o p=(map @t json)]` | `{"foo":"xyz","bar":123}` | `[%o p={[p='bar' q=[%n p=~.123]] [p='foo' q=[%s p='xyz']]}]` |

67+

68+Since the `$json` `%o` object and `%a` array types may themselves contain any `$json`, you can see how JSON structures of arbitrary complexity can be represented. Note the `%n` number type is a `@ta` rather than something like a `@ud` that you might expect. This is because JSON's number type may be either an integer or floating point, so it's left as a `knot` which can then be parsed to a `@ud` or `@rd` with the appropriate [`+dejs:format`](/docs/hoon/reference/zuse/2d_6) function.

69+

70+## `$json` encoding and decoding example

71+

72+Let's have a look at a practical example. Here's a core with three arms. It has the structure arm `$user`, and then two more: `+to-js` converts a `$user` structure to `$json`, and `+from-js` does the opposite. Usually we'd define the structure in a separate `/sur` file, but for simplicity it's all in the one core.

73+

74+#### `json-test.hoon`

75+

76+```hoon

77+|%

78++$ user

79+ $: username=@t

80+ name=[first=@t mid=@t last=@t]

81+ joined=@da

82+ email=@t

83+ ==

84+++ to-js

85+ |= usr=user

86+ |^ ^- json

87+ %- pairs:enjs:format

88+ :~

89+ ['username' s+username.usr]

90+ ['name' name]

91+ ['joined' (sect:enjs:format joined.usr)]

92+ ['email' s+email.usr]

93+ ==

94+ ++ name

95+ :- %a

96+ :~

97+ [%s first.name.usr]

98+ [%s mid.name.usr]

99+ [%s last.name.usr]

100+ ==

101+ --

102+++ from-js

103+ =, dejs:format

104+ ^- $-(json user)

105+ %- ot

106+ :~

107+ [%username so]

108+ [%name (at ~[so so so])]

109+ [%joined du]

110+ [%email so]

111+ ==

112+--

113+```

114+

115+### Try it out

116+

117+First we'll try using our `$json` encoding/decoding library, and afterwards we'll take a closer look at its construction. To begin, save the code above in `/lib/json-test.hoon` of the `%base` desk on a fake ship and `|commit` it:

118+

119+```

120+> |commit %base

121+>=

122++ /~zod/base/5/lib/json-test/hoon

123+```

124+

125+Then we need to build it so we can use it. We'll give it a face of `user-lib`:

126+

127+```

128+> =user-lib -build-file %/lib/json-test/hoon

129+```

130+

131+Let's now create an example of a `$user` structure:

132+

133+```

134+> =usr `user:user-lib`['john456' ['John' 'William' 'Smith'] now 'john.smith@example.com']

135+> usr

136+[ username='john456'

137+ name=[first='John' mid='William' last='Smith']

138+ joined=~2021.9.12..09.47.58..1b65

139+ email='john.smith@example.com'

140+]

141+```

142+

143+Now we can try calling the `+to-js` function with our data to convert it to `$json`:

144+

145+```

146+> =usr-json (to-js:user-lib usr)

147+> usr-json

148+[ %o

149+ p

150+ { [p='email' q=[%s p='john.smith@example.com']]

151+ [p='name' q=[%a p=~[[%s p='John'] [%s p='William'] [%s p='Smith']]]]

152+ [p='username' q=[%s p='john456']]

153+ [p='joined' q=[%n p=~.1631440078]]

154+ }

155+]

156+```

157+

158+Let's also see how that `$json` would look as real JSON encoded in text. We can do that with `+en-json:html`:

159+

160+```

161+> (crip (en-json:html (to-js:user-lib usr)))

162+'{"joined":1631440078,"username":"john456","name":["John","William","Smith"],"email":"john.smith@example.com"}'

163+```

164+

165+Finally, let's try converting the `$json` back to a `$user` again with our `+from-js` arm:

166+

167+```

168+> (from-js:user-lib usr-json)

169+[ username='john456'

170+ name=[first='John' mid='William' last='Smith']

171+ joined=~2021.9.12..09.47.58

172+ email='john.smith@example.com'

173+]

174+```

175+

176+### Analysis

177+

178+#### Converting to `$json`

179+

180+Here's our arm that converts a `$user` structure to `$json`:

181+

182+```hoon

183+++ to-js

184+ |= usr=user

185+ |^ ^- json

186+ %- pairs:enjs:format

187+ :~

188+ ['username' s+username.usr]

189+ ['name' name]

190+ ['joined' (sect:enjs:format joined.usr)]

191+ ['email' s+email.usr]

192+ ==

193+ ++ name

194+ :- %a

195+ :~

196+ [%s first.name.usr]

197+ [%s mid.name.usr]

198+ [%s last.name.usr]

199+ ==

200+ --

201+```

202+

203+There are different ways we could represent our `$user` structure as JSON, but in this case we've opted to encapsulate it in an object and have the `name` as an array (since JSON arrays preserve order).

204+

205+[`+enjs:format`](/docs/hoon/reference/zuse/2d_1-5#enjsformat)includes the convenient [`+pairs`](/docs/hoon/reference/zuse/2d_1-5#pairsenjsformat) function, which converts a list of `[@t json]` to an object containing those key-value pairs. We've used this to assemble the final object. Note that if you happen to have only a single key-value pair rather than a list, you can use [`+frond`](/docs/hoon/reference/zuse/2d_1-5#frondenjsformat) instead of `+pairs`.

206+

207+For the `joined` field, we've used the [`+sect`](/docs/hoon/reference/zuse/2d_1-5#sectenjsformat) function from `+enjs` to convert the `@da` to a Unix seconds timestamp in a `$json` number. The `+sect` function, like others in `+enjs`, takes in a noun (in this case a `@da`) and produces `$json` (in this case a `[%n @ta]` number). `+enjs` contains a handful of useful functions like this, but for the rest we've just hand-made the `$json` structure. This is fairly typical when encoding `$json`, it's usually [decoding](#converting-from-json) that makes more extensive use of the `$json` utility functions in `+format`.

208+

209+For the `name` field we've just formed a cell of `%a` and a list of `$json` strings, since a `$json` array is `[%a p=(list json)]`. Note we've separated this part into its own arm and wrapped the whole thing in a `|^` - a core with a `$` arm that's computed immediately. This is simply for readability - our structure here is quite simple but when dealing with deeply-nested `$json` structures or complex logic, having a single giant function can quickly become unwieldy.

210+

211+#### Converting from `$json`

212+

213+Here's our arm that converts `$json` to our `$user` structure:

214+

215+```hoon

216+++ from-js

217+ =, dejs:format

218+ ^- $-(json user)

219+ %- ot

220+ :~

221+ [%username so]

222+ [%name (at ~[so so so])]

223+ [%joined du]

224+ [%email so]

225+ ==

226+```

227+

228+This is the inverse of the [encoding](#converting-to-json) function described in the previous section.

229+

230+We make extensive use of [`+dejs:format`](/docs/hoon/reference/zuse/2d_6) functions here, so we've used `=,` to expose the namespace and allow succinct `+dejs` function calls.

231+

232+We use the [`+ot`](/docs/hoon/reference/zuse/2d_6#otdejsformat) function from `+dejs:format` to decode the `$json` object to a n-tuple. It's a wet gate that takes a list of pairs of keys and other `+dejs` functions and produces a new gate that takes the `$json` to be decoded (which we've given it in `jon`).

233+

234+The [`+so`](/docs/hoon/reference/zuse/2d_6#sodejsformat) functions just decode `$json` strings to `cord`s. The [`+at`](/docs/hoon/reference/zuse/2d_6#atdejsformat) function converts a `$json` array to a tuple, decoding each element with the respective function given in its argument list. Like `+ot`, `+at` is also a wet gate that produces a gate that takes `$json`. In our case we've used `+so` for each element, since they're all strings.

235+

236+For `joined`, we've used the [`+du`](/docs/hoon/reference/zuse/2d_6#dudejsformat) function, which converts a Unix seconds timestamp in a `$json` number to a `@da` (it's basically the inverse of the `+sect:enjs:format` we used earlier).

237+

238+Notice how `+ot` takes in other `+dejs` functions in its argument. One of its arguments includes the `+at` function which itself takes in other `+dejs` functions. There are several `+dejs` functions like this that allow complex nested JSON structures to be decoded. For other examples of common `+dejs` functions like this, see the [More `+dejs`](#more-dejs) section below.

239+

240+There are dozens of different functions in [`+dejs:format`](/docs/hoon/reference/zuse/2d_6) that will cover a great many use cases. If there isn't a `+dejs` function for a particular case, you can also just write a custom function - it just has to take `$json`. Note there's also the [`+dejs-soft:format`](/docs/hoon/reference/zuse/2d_7) functions - these are similar to `+dejs` functions except they produce `unit`s rather than simply crashing if decoding fails.

241+

242+## More `+dejs`

243+

244+We looked at the commonly used `+ot` function in the [first example](#converting-from-json), now let's look at a couple more common `+dejs` functions.

245+

246+### `+of`

247+

248+The [`+of`](/docs/hoon/reference/zuse/2d_6#ofdejsformat) function takes an object containing a single key-value pair, decodes the value with the corresponding `+dejs` function in a key-function list, and produces a key-value tuple. This is useful when there are multiple possible objects you might receive, and tagged unions are a common data structure in hoon.

249+

250+Let's look at an example. Here's a gate that takes in some `$json`, decodes it with an `+of` function that can handle three possible objects, casts the result to a tagged union, switches against its head with `?-`, performs some transformation and finally returns the result. You can save it as `gen/of-test.hoon` in the `%base` desk of a fake ship and `|commit %base`.

251+

252+#### `of-test.hoon`

253+

254+```hoon

255+|= jon=json

256+|^ ^- @t

257+=/ =fbb

258+ (to-fbb jon)

259+?- -.fbb

260+ %foo (cat 3 +.fbb '!!!')

261+ %bar ?:(+.fbb 'Yes' 'No')

262+ %baz :((cury cat 3) p.fbb q.fbb r.fbb)

263+==

264++$ fbb

265+ $% [%foo @t]

266+ [%bar ?]

267+ [%baz p=@t q=@t r=@t]

268+ ==

269+++ to-fbb

270+=, dejs:format

271+%- of

272+:~ foo+so

273+ bar+bo

274+ baz+(at ~[so so so])

275+==

276+--

277+```

278+

279+Let's try it:

280+

281+```

282+> +of-test (need (de-json:html '{"foo":"Hello"}'))

283+'Hello!!!'

284+

285+> +of-test (need (de-json:html '{"bar":true}'))

286+'Yes'

287+

288+> +of-test (need (de-json:html '{"baz":["a","b","c"]}'))

289+'abc'

290+```

291+

292+### `+ou`

293+

294+The [`+ou`](/docs/hoon/reference/zuse/2d_6#oudejsformat) function decodes a `$json` object to an n-tuple using the matching functions in a key-function list. Additionally, it lets you set some key-value pairs in an object as optional and others as mandatory. The mandatory ones crash if they're missing and the optional ones are replaced with a given noun.

295+

296+`+ou` is different to other `+dejs` functions - the functions it takes are `$-((unit json) grub)` rather than the usual `$-(json grub)` of most `+dejs` functions. There are only two `+dejs` functions that fit this - [`+un`](/docs/hoon/reference/zuse/2d_6#undejsformat) and [`+uf`](/docs/hoon/reference/zuse/2d_6#ufdejsformat). These are intended to be used with `+ou` - you would wrap each function in the key-function list of `+ou` with either `+un` or `+uf`.

297+

298+`+un` crashes if its argument is `~`. `+ou` gives functions a `~` if the matching key-value pair is missing in the `$json` object, so `+un` crashes if the key-value pair is missing. Therefore, `+un` lets you set key-value pairs as mandatory.

299+

300+`+uf` takes two arguments - a noun and a `+dejs` function. If the `(unit json)` it's given by `+ou` is `~`, it produces the noun it was given rather than the product of the `+dejs` function. This lets you specify key-value pairs as optional, replacing missing ones with whatever you want.

301+

302+Let's look at a practical example. Here's a generator you can save in the `%base` desk of a fake ship in `gen/ou-test.hoon` and `|commit %base`. It takes in a `$json` object and produces a triple. The `+ou` in `+decode` has three key-function pairs - the first two are mandatory and the last is optional, producing the bunt of a set if the `%baz` key is missing.

303+

304+#### `ou-test.hoon`

305+

306+```hoon

307+|= jon=json

308+|^ ^- [@t ? (set @ud)]

309+(decode jon)

310+++ decode

311+=, dejs:format

312+%- ou

313+:~ foo+(un so)

314+ bar+(un bo)

315+ baz+(uf *(set @ud) (as ni))

316+==

317+--

318+```

319+

320+Let's try it:

321+

322+```

323+> +ou-test (need (de-json:html '{"foo":"hello","bar":true,"baz":[1,2,3,4]}'))

324+['hello' %.y {1 2 3 4}]

325+

326+> +ou-test (need (de-json:html '{"foo":"hello","bar":true}'))

327+['hello' %.y {}]

328+

329+> +ou-test (need (de-json:html '{"foo":"hello"}'))

330+[%key 'bar']

331+dojo: hoon expression failed

332+```

333+

334+### `+su`

335+

336+The [`+su`](/docs/hoon/reference/zuse/2d_6#sudejsformat) function parses a string with the given parsing `rule`. Hoon's functional parsing library is very powerful and lets you create arbitrarily complex parsers. JSON will often have data types encoded in strings, so this function can be very useful. The writing of parsers is outside the scope of this guide, but you can see the [Parsing Guide](/docs/hoon/guides/parsing) and sections 4e to 4j of the standard library documentation for details.

337+

338+Here are some simple examples of using `+su` to parse strings:

339+

340+```

341+> `@ux`((su:dejs:format hex) s+'deadbeef1337f00D')

342+0xdead.beef.1337.f00d

343+

344+> `(list @)`((su:dejs:format (most lus dem)) s+'1+2+3+4')

345+~[1 2 3 4]

346+

347+> `@ub`((su:dejs:format ven) s+'+>-<->+<+')

348+0b11.1000.1101

349+```

350+

351+Here's a more complex parser that will parse a GUID like `824e7749-4eac-9c00-db16-4cb816cd6f19` to a `@ux`:

352+

353+#### `su-test.hoon`

354+

355+```hoon

356+|= jon=json

357+^- @ux

358+%. jon

359+%- su:dejs:format

360+%+ cook

361+|= parts=(list [step @])

362+^- @ux

363+(can 3 (flop parts))

364+;~ plug

365+ (stag 4 ;~(sfix (bass 16 (stun 8^8 six:ab)) hep))

366+ (stag 2 ;~(sfix qix:ab hep))

367+ (stag 2 ;~(sfix qix:ab hep))

368+ (stag 2 ;~(sfix qix:ab hep))

369+ (stag 6 (bass 16 (stun 12^12 six:ab)))

370+ (easy ~)

371+==

372+```

373+

374+Save it in the `/gen` directory of the `%base` desk and `|commit` it. We can then try it with:

375+

376+```

377+> +su-test s+'5323a61d-0c26-d8fa-2b73-18cdca805fd8'

378+0x5323.a61d.0c26.d8fa.2b73.18cd.ca80.5fd8

379+```

380+

381+If we delete the last character it'll no longer be a valid GUID and the parsing will fail:

382+

383+```

384+> +su-test s+'5323a61d-0c26-d8fa-2b73-18cdca805fd'

385+/gen/su-test/hoon:<[2 1].[16 3]>

386+/gen/su-test/hoon:<[3 1].[16 3]>

387+{1 36}

388+syntax error

389+dojo: naked generator failure

390+```

391+

392+## `mark` file example

393+

394+Here's a simple `mark` file for the `$user` structure we created in the [first example](#json-encoding-and-decoding-example). It imports the [json-test.hoon](#json-testhoon) library we created and saved in our `%base` desk's `/lib` directory.

395+

396+#### `user.hoon`

397+

398+```hoon

399+/+ *json-test

400+|_ usr=user

401+++ grab

402+ |%

403+ ++ noun user

404+ ++ json from-js

405+ --

406+++ grow

407+ |%

408+ ++ noun usr

409+ ++ json (to-js usr)

410+ --

411+++ grad %noun

412+--

413+```

414+

415+The [Marks section](/docs/arvo/clay/marks/marks) of the Clay documentation covers `mark` files comprehensively and is worth reading through if you want to write a mark file.

416+

417+In brief, a mark file contains a `door` with three arms. The door's sample type is the type of the data in question - in our case the `$user` structure. The `+grab` arm contains methods for converting _to_ our mark, and the `+grow` arm contains methods for converting _from_ our mark. The `+noun` arms are mandatory, and then we've added `+json` arms which respectively call the `+from-js` and `+to-js` functions from our `json-test.hoon` library. The final `+grad` arm defines various revision control functions, in our case we've delegated these to the `%noun` mark.

418+

419+From this mark file, Clay can build mark conversion gates between the `%json` mark and our `%user` mark, allowing the conversion of `$json` data to a `$user` structure and vice versa.

420+

421+### Try it out

422+

423+First, we'll save the code above as `user.hoon` in the `/mar` directory our of `%base` desk:

424+

425+```

426+> |commit %base

427+>=

428++ /~zod/base/9/mar/user/hoon

429+```

430+

431+Let's quickly create a `$json` object to work with:

432+

433+```

434+> =jon (need (de-json:html '{"joined":1631440078,"username":"john456","name":["John","William","Smith"],"email":"john.smith@example.com"}'))

435+> jon

436+[ %o

437+ p

438+ { [p='email' q=[%s p='john.smith@example.com']]

439+ [p='name' q=[%a p=~[[%s p='John'] [%s p='William'] [%s p='Smith']]]]

440+ [p='username' q=[%s p='john456']]

441+ [p='joined' q=[%n p=~.1631440078]]

442+ }

443+]

444+```

445+

446+We'll also build our library so we can use its types from the dojo:

447+

448+```

449+> =user-lib -build-file %/lib/json-test/hoon

450+```

451+

452+Now we can ask Clay to build a mark conversion gate from a `%json` mark to our `%user` mark. We'll use a scry with a `%f` `care` which produces a static mark conversion gate:

453+

454+```

455+> =json-to-user .^($-(json user:user-lib) %cf /===/json/user)

456+```

457+

458+Let's try converting our `$json` to a `$user` structure with our new mark conversion gate:

459+

460+```

461+> =usr (json-to-user jon)

462+> usr

463+[ username='john456'

464+ name=[first='John' mid='William' last='Smith']

465+ joined=~2021.9.12..09.47.58

466+ email='john.smith@example.com'

467+]

468+```

469+

470+Now let's try the other direction. We'll again scry Clay to build a static mark conversion gate, this time _from_ `%user` _to_ `%json` rather than the reverse:

471+

472+```

473+> =user-to-json .^($-(user:user-lib json) %cf /===/user/json)

474+```

475+

476+Let's test it out by giving it our `$user` data:

477+

478+```

479+> (user-to-json usr)

480+[ %o

481+ p

482+ { [p='email' q=[%s p='john.smith@example.com']]

483+ [p='name' q=[%a p=~[[%s p='John'] [%s p='William'] [%s p='Smith']]]]

484+ [p='username' q=[%s p='john456']]

485+ [p='joined' q=[%n p=~.1631440078]]

486+ }

487+]

488+```

489+

490+Finally, let's see how that looks as JSON encoded in text:

491+

492+```

493+> (crip (en-json:html (user-to-json usr)))

494+'{"joined":1631440078,"username":"john456","name":["John","William","Smith"],"email":"john.smith@example.com"}'

495+```

496+

497+Usually (though not in all cases) these mark conversions will be performed implicitly by Gall or Eyre and you'd not deal with the mark conversion gates directly, but it's still informative to see them work explicitly.

498+

499+## Further reading

500+

501+[The Zuse library reference](/docs/hoon/reference/zuse/table-of-contents) - This includes documentation of the JSON parsing, printing, encoding and decoding functions.

502+

503+[The Marks section of the Clay documentation](/docs/arvo/clay/marks/marks) - Comprehensive documentation of `mark`s.

504+

505+[The External API Reference section of the Eyre documentation](/docs/arvo/eyre/external-api-ref) - Details of the webserver vane Eyre's external API.

506+

507+[The Iris documentation](/docs/arvo/iris/iris) - Details of the web client vane Iris, which may be used to fetch external JSON data among other things.

508+

509+[Strings Guide](/docs/hoon/guides/strings) - Atom printing functions like `+scot` will often be useful for JSON encoding - see the [Encoding in Text](/docs/hoon/guides/strings#encoding-in-text) section for usage.

510+

511+[Parsing Guide](/docs/hoon/guides/parsing) - Learn how to write functional parsers in hoon which can be used with `+su`.

content/docs/hoon/guides/parsing.md+48−21

@@ -9,6 +9,8 @@ knowledge of parsing is required, and we will explain the basic structure of how

9parsing works in a purely functional language such as Hoon before moving on to

10how it is implemented in Hoon.

12+**Note:** For JSON printing/parsing and encoding/decoding, see the [JSON Guide](/docs/hoon/guides/json-guide).

13+

12## What is parsing? {#what-is-parsing}

14## What is parsing? {#what-is-parsing}

14A program which takes a raw sequence of characters as an input and produces a data

16A program which takes a raw sequence of characters as an input and produces a data

@@ -42,20 +44,24 @@ the parsing routine on the string `"1"` and looks for an integer, and so it

42returns the integer `1`. However, taking into account what was said above about

44returns the integer `1`. However, taking into account what was said above about

43parsers returning the unparsed portion of the string as well, we should

45parsers returning the unparsed portion of the string as well, we should

44represent this return as a tuple. So we should expect something like this:

46represent this return as a tuple. So we should expect something like this:

47+

45```

48```

46> (parse "1" integer)

49> (parse "1" integer)

47[1 ""]

50[1 ""]

48> (parse "123" integer)

51> (parse "123" integer)

49[1 "23"]

52[1 "23"]

50```

53```

54+

51What if we wish to parse the rest of the string? We would need to apply the

55What if we wish to parse the rest of the string? We would need to apply the

52`parse` function again:

56`parse` function again:

57+

53```

58```

54> (parse (parse "123" integer) integer)

59> (parse (parse "123" integer) integer)

55[12 "3"]

60[12 "3"]

56> (parse (parse (parse "123" integer) integer) integer)

61> (parse (parse (parse "123" integer) integer) integer)

57[123 ""]

62[123 ""]

58```

63```

64+

59So we see that we can parse strings larger than one character by stringing

65So we see that we can parse strings larger than one character by stringing

60together parsing functions for single characters. Thus in addition to parsing

66together parsing functions for single characters. Thus in addition to parsing

61functions for single input characters, we want _parser combinators_ that

67functions for single input characters, we want _parser combinators_ that

@@ -75,15 +81,14 @@ are quite a few), but ought to be

75sufficient to get started with parsing in Hoon and be equipped to discover the

81sufficient to get started with parsing in Hoon and be equipped to discover the

76remainder yourself.

82remainder yourself.

78−

79## Basic types

84## Basic types

81In this section we discuss the types most commonly used for Hoon parsers. In short:

86In this section we discuss the types most commonly used for Hoon parsers. In short:

83− - A `hair` is the position in the text the parser is at,

88+- A `hair` is the position in the text the parser is at,

84− - A `nail` is parser input,

89+- A `nail` is parser input,

85− - An `edge` is parser output,

90+- An `edge` is parser output,

86− - A `rule` is a parser.

91+- A `rule` is a parser.

88### `hair`

93### `hair`

@@ -102,6 +107,7 @@ is in case it hits something unexpected during parsing.

102```hoon

107```hoon

103++ nail [p=hair q=tape]

108++ nail [p=hair q=tape]

104```

109```

110+

105We recall from our [discussion above](#what-is-parsing) that parsing functions must keep

111We recall from our [discussion above](#what-is-parsing) that parsing functions must keep

106track of both what has been parsed and what has yet to be parsed. Thus a `nail`

112track of both what has been parsed and what has yet to be parsed. Thus a `nail`

107consists of both a `hair`, giving the line and column up to which the input

113consists of both a `hair`, giving the line and column up to which the input

@@ -122,6 +128,7 @@ what has theoretically already been parsed up to that point.

122```hoon

128```hoon

123++ edge [p=hair q=(unit [p=* q=nail])]

129++ edge [p=hair q=(unit [p=* q=nail])]

124```

130```

131+

125An `edge` is the output of a parser. If parsing succeeded, `p` is the location

132An `edge` is the output of a parser. If parsing succeeded, `p` is the location

126of the original input `tape `up to which the text has been parsed. If parsing

133of the original input `tape `up to which the text has been parsed. If parsing

127failed, `p` will be the first `hair` at which parsing failed.

134failed, `p` will be the first `hair` at which parsing failed.

@@ -136,6 +143,7 @@ of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.

136```hoon

143```hoon

137++ rule _|:($:nail $:edge)

144++ rule _|:($:nail $:edge)

138```

145```

146+

139A `rule` is a gate which takes in a `nail` and returns an `edge` - in other

147A `rule` is a gate which takes in a `nail` and returns an `edge` - in other

140words, a parser.

148words, a parser.

141

149

@@ -160,6 +168,7 @@ of the input `nail`.

160> edg

168> edg

161[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]

169[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]

162```

170```

171+

163We note that `p.edg` is `[p=1 q=2]`, indicating that the next character to be

172We note that `p.edg` is `[p=1 q=2]`, indicating that the next character to be

164parsed is in line 1, column 2. `q.edg` is not null, indicating that parsing

173parsed is in line 1, column 2. `q.edg` is not null, indicating that parsing

165succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is the same as `p.edg`, which is always the case for

174succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is the same as `p.edg`, which is always the case for

@@ -167,11 +176,13 @@ succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is

167`q.q.edg` is `"bc"`, which is the part of the input `tape` that has yet to be parsed.

176`q.q.edg` is `"bc"`, which is the part of the input `tape` that has yet to be parsed.

168

177

169Now let's see what happens when parsing fails.

178Now let's see what happens when parsing fails.

179+

170```

180```

171> =edg ((just 'b') [[1 1] "abc"])

181> =edg ((just 'b') [[1 1] "abc"])

172> edg

182> edg

173[p=[p=1 q=1] q=~]

183[p=[p=1 q=1] q=~]

174```

184```

185+

175Now we have that `p.edg` is the same as the input `hair`, `[1 1]`, meaning the

186Now we have that `p.edg` is the same as the input `hair`, `[1 1]`, meaning the

176parser has not advanced since parsing failed. `q.edg` is null, indicating that

187parser has not advanced since parsing failed. `q.edg` is null, indicating that

177parsing has failed.

188parsing has failed.

@@ -186,11 +197,13 @@ produces a `rule` that attempts to match that `cord` against the beginning of

186the input.

197the input.

187

198

188Let's see what happens when we successfully parse the entire input `tape`.

199Let's see what happens when we successfully parse the entire input `tape`.

200+

189```

201```

190> =edg ((jest 'abc') [[1 1] "abc"])

202> =edg ((jest 'abc') [[1 1] "abc"])

191> edg

203> edg

192[p=[p=1 q=4] q=[~ [p='abc' q=[p=[p=1 q=4] q=""]]]]

204[p=[p=1 q=4] q=[~ [p='abc' q=[p=[p=1 q=4] q=""]]]]

193```

205```

206+

194`p.edg` is `[p=1 q=4]`, indicating that the next character to be parsed is at

207`p.edg` is `[p=1 q=4]`, indicating that the next character to be parsed is at

195line 1, column 4. Of course, this does not exist since the input `tape` was only

208line 1, column 4. Of course, this does not exist since the input `tape` was only

1963 characters long, so this actually indicates that the entire `tape` has been

2093 characters long, so this actually indicates that the entire `tape` has been

@@ -199,21 +212,25 @@ successfully parsed (since the `hair` does not advance in the case of failure).

199remains to be parsed.

212remains to be parsed.

200

213

201What happens if we only match some of the input `tape`?

214What happens if we only match some of the input `tape`?

215+

202```

216```

203> =edg ((jest 'ab') [[1 1] "abc"])

217> =edg ((jest 'ab') [[1 1] "abc"])

204> edg

218> edg

205[p=[p=1 q=3] q=[~ [p='ab' q=[p=[p=1 q=3] q="c"]]]]

219[p=[p=1 q=3] q=[~ [p='ab' q=[p=[p=1 q=3] q="c"]]]]

206```

220```

221+

207Now we have that the result, `p.q.edg`, is `'ab'`, while the remainder `q.q.q.edg`

222Now we have that the result, `p.q.edg`, is `'ab'`, while the remainder `q.q.q.edg`

208is `"c"`. So `+jest` has successfully parsed the first two characters, while the

223is `"c"`. So `+jest` has successfully parsed the first two characters, while the

209last character remains. Furthermore, we still have the information that the

224last character remains. Furthermore, we still have the information that the

210remaining character was in line 1 column 3 from `p.edg` and `p.q.q.edg`.

225remaining character was in line 1 column 3 from `p.edg` and `p.q.q.edg`.

211

226

212What happens when `+jest` fails?

227What happens when `+jest` fails?

228+

213```

229```

214> ((jest 'bc') [[1 1] "abc"])

230> ((jest 'bc') [[1 1] "abc"])

215[p=[p=1 q=1] q=~]

231[p=[p=1 q=1] q=~]

216```

232```

233+

217Despite the fact that `'bc'` appears in `"abc"`, because it was not at the

234Despite the fact that `'bc'` appears in `"abc"`, because it was not at the

218beginning the parse failed. We will see in [parser

235beginning the parse failed. We will see in [parser

219combinators](#parser-combinators) how to modify this `rule` so that it

236combinators](#parser-combinators) how to modify this `rule` so that it

@@ -237,6 +254,7 @@ replaces the parsing result with `cus`.

237

254

238Here we see that `p.q` of the `edge` returned by the `rule` created with `+cold`

255Here we see that `p.q` of the `edge` returned by the `rule` created with `+cold`

239is `%foo`.

256is `%foo`.

257+

240```

258```

241> ((cold %foo (just 'a')) [[1 1] "abc"])

259> ((cold %foo (just 'a')) [[1 1] "abc"])

242[p=[p=1 q=2] q=[~ u=[p=%foo q=[p=[p=1 q=2] q="bc"]]]]

260[p=[p=1 q=2] q=[~ u=[p=%foo q=[p=[p=1 q=2] q="bc"]]]]

@@ -246,7 +264,6 @@ One common scenario where `+cold` sees play is when writing [command line

246interface (CLI) apps](/docs/hoon/guides/cli-tutorial). We usher the

264interface (CLI) apps](/docs/hoon/guides/cli-tutorial). We usher the

247reader there to find an example where `+cold` is used.

265reader there to find an example where `+cold` is used.

248

266

249−

250### `+knee`

267### `+knee`

251

268

252Another important function in the parser builder library is `+knee`, used for building

269Another important function in the parser builder library is `+knee`, used for building

@@ -254,7 +271,6 @@ recursive parsers. We delay discussion of `+knee` to the

254[section below](#recursive-parsers) as more context is needed to explain it

271[section below](#recursive-parsers) as more context is needed to explain it

255properly.

272properly.

256

273

257−

258## Outside callers

274## Outside callers

259

275

260Since `hair`s, `nail`s, etc. are only utilized within the context of writing

276Since `hair`s, `nail`s, etc. are only utilized within the context of writing

@@ -295,7 +311,6 @@ For additional information including additional examples see [4g: Parsing (Outsi

295For the remainder of this tutorial we will make use of `+scan` so that we do not

311For the remainder of this tutorial we will make use of `+scan` so that we do not

296need to deal directly with `nail`s except where it is ilustrative to do so.

312need to deal directly with `nail`s except where it is ilustrative to do so.

297

313

298−

299### Parsing atoms

314### Parsing atoms

300

315

301[Recall](/docs/hoon/hoon-school/lists) that `cord`s are atoms with the aura

316[Recall](/docs/hoon/hoon-school/lists) that `cord`s are atoms with the aura

@@ -312,7 +327,6 @@ The standard library provides a number of gates that take a `rule` and

312produce a new modified `rule` according to some process. We call these _parser modifiers_. These are

327produce a new modified `rule` according to some process. We call these _parser modifiers_. These are

313documented among the [parser builders](/docs/hoon/reference/stdlib/4f).

328documented among the [parser builders](/docs/hoon/reference/stdlib/4f).

314

329

315−

316### `+ifix`

330### `+ifix`

317

331

318`+ifix` modifies a `rule` so that it matches that `rule` only when it is

332`+ifix` modifies a `rule` so that it matches that `rule` only when it is

@@ -322,11 +336,11 @@ surrounded on both sides by text that matches a pair of `rule`s, which is discar

322> (scan "(42)" (ifix [pal par] (jest '42')))

336> (scan "(42)" (ifix [pal par] (jest '42')))

323'42'

337'42'

324```

338```

339+

325`+pal` and `+par` are shorthand for `(just '(')` and `(just ')')`, respectively. All

340`+pal` and `+par` are shorthand for `(just '(')` and `(just ')')`, respectively. All

326ASCII glyphs have counterparts of this sort, documented

341ASCII glyphs have counterparts of this sort, documented

327[here](/docs/hoon/reference/stdlib/4h).

342[here](/docs/hoon/reference/stdlib/4h).

328

343

329−

330### `+star` {#star}

344### `+star` {#star}

331

345

332`+star` is used to apply a `rule` repeatedly. Recall that `+just` only parses

346`+star` is used to apply a `rule` repeatedly. Recall that `+just` only parses

@@ -343,7 +357,9 @@ We can use `+star` to get the rest of the `tape`:

343> ((star (just 'a')) [[1 1] "aaa"])

357> ((star (just 'a')) [[1 1] "aaa"])

344[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q=""]]]]

358[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q=""]]]]

345```

359```

360+

346and we note that the parsing ceases when it fails.

361and we note that the parsing ceases when it fails.

362+

347```

363```

348> ((star (just 'a')) [[1 1] "aaab"])

364> ((star (just 'a')) [[1 1] "aaab"])

349[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q="b"]]]]

365[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q="b"]]]]

@@ -358,9 +374,11 @@ combinators](/docs/hoon/reference/stdlib/4e). First we introduce a few

358combinators, then we examine more closely how `;~` is used to chain them together.

374combinators, then we examine more closely how `;~` is used to chain them together.

359

375

360The syntax to combine `rule`s is

376The syntax to combine `rule`s is

377+

361```hoon

378```hoon

362;~(combinator rule1 rule2 ... ruleN)

379;~(combinator rule1 rule2 ... ruleN)

363```

380```

381+

364The `rule`s are composed together using the combinator as an

382The `rule`s are composed together using the combinator as an

365intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns

383intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns

366it into a sample (a `nail`) for the next `rule` to handle. We elaborate on this

384it into a sample (a `nail`) for the next `rule` to handle. We elaborate on this

@@ -409,42 +427,46 @@ take this opportunity to think about it carefully.

409The `rule` created by `;~(combinator [list rule])` may be understood

427The `rule` created by `;~(combinator [list rule])` may be understood

410inductively. To do this, let's consider the base case where our `[list rule]` has only a

428inductively. To do this, let's consider the base case where our `[list rule]` has only a

411single entry.

429single entry.

430+

412```

431```

413> (scan "star" ;~(plug (jest 'star')))

432> (scan "star" ;~(plug (jest 'star')))

414'star'

433'star'

415```

434```

435+

416Our output is identical to that given by `scan "star" (jest 'star')`. This is to

436Our output is identical to that given by `scan "star" (jest 'star')`. This is to

417be expected. The combinator `+plug` is specifically used for chaining together

437be expected. The combinator `+plug` is specifically used for chaining together

418`rule`s in the `[list rule]`, but if there is only one `rule`, there is nothing

438`rule`s in the `[list rule]`, but if there is only one `rule`, there is nothing

419to chain. Thus, swapping out `+plug` for another combinator makes no difference here:

439to chain. Thus, swapping out `+plug` for another combinator makes no difference here:

440+

420```

441```

421> (scan "star" ;~(pose (jest 'star')))

442> (scan "star" ;~(pose (jest 'star')))

422'star'

443'star'

423> (scan "star" ;~((glue com) (jest 'star')))

444> (scan "star" ;~((glue com) (jest 'star')))

424'star'

445'star'

425```

446```

447+

426`;~` and the combinator only begin to play a role once the `[list rule]` has at

448`;~` and the combinator only begin to play a role once the `[list rule]` has at

427least two elements. So let's look at an example done with `+plug`, the simplest

449least two elements. So let's look at an example done with `+plug`, the simplest

428combinator.

450combinator.

451+

429```

452```

430> (scan "star" ;~(plug (jest 'st') (jest 'ar')))

453> (scan "star" ;~(plug (jest 'st') (jest 'ar')))

431['st' 'ar']

454['st' 'ar']

432```

455```

456+

433Our return suggests that we first parsed `"star"` with the `rule` `(jest 'st')` and passed

457Our return suggests that we first parsed `"star"` with the `rule` `(jest 'st')` and passed

434−the resulting `edge` to `(jest 'ar')` - in other words, we called `+plug` on `(jest

458+the resulting `edge` to `(jest 'ar')` - in other words, we called `+plug` on `(jest 'st')` and the `edge` returned once it had been used to parse `"star"`. Thus

435−'st')` and the `edge` returned once it had been used to parse `"star"`. Thus

436`+plug` was the glue that allowed us to join the two `rule`s, and `;~` performed

459`+plug` was the glue that allowed us to join the two `rule`s, and `;~` performed

437the gluing operation. And so, swapping `+plug` for `+pose` results in a crash,

460the gluing operation. And so, swapping `+plug` for `+pose` results in a crash,

438which clues us into the fact that the combinator now has an effect since there

461which clues us into the fact that the combinator now has an effect since there

439is more than one `rule`.

462is more than one `rule`.

463+

440```

464```

441> (scan "star" ;~(pose (jest 'st') (jest 'ar')))

465> (scan "star" ;~(pose (jest 'st') (jest 'ar')))

442{1 3}

466{1 3}

443syntax error

467syntax error

444```

468```

445

469

446−

447−

448## Parsing numbers

470## Parsing numbers

449

471

450Functions for parsing numbers are documented in [4j: Parsing (Bases and Base

472Functions for parsing numbers are documented in [4j: Parsing (Bases and Base

@@ -461,15 +483,19 @@ for parsing decimal numbers.

461## Recursive parsers

483## Recursive parsers

462

484

463Naively attempting to write a recursive `rule`, i.e. like

485Naively attempting to write a recursive `rule`, i.e. like

486+

464```

487```

465> |-(;~(pose den ;~(pose $ (easy ~))))

488> |-(;~(pose den ;~(pose $ (easy ~))))

466```

489```

490+

467results in an error.

491results in an error.

492+

468```

493```

469-find.,.+6

494-find.,.+6

470-find.,.+6

495-find.,.+6

471rest-loop

496rest-loop

472```

497```

498+

473Thus some special sauce is required, the `+knee` function.

499Thus some special sauce is required, the `+knee` function.

474

500

475`+knee` takes in a noun that is the default value of the parser, typically given

501`+knee` takes in a noun that is the default value of the parser, typically given

@@ -494,7 +520,6 @@ the parser to improve performance. In [parsing arithmetic

494expressions](#parsing-arithmetic-expressions) we will be writing a recursive

520expressions](#parsing-arithmetic-expressions) we will be writing a recursive

495parser making use of `+knee` and `~+` throughout.

521parser making use of `+knee` and `~+` throughout.

496

522

497−

498# Parsing arithmetic expressions

523# Parsing arithmetic expressions

499

524

500In this section we will be applying what we have learned to write a parser for

525In this section we will be applying what we have learned to write a parser for

@@ -514,6 +539,7 @@ for arithmetic expressions informally.

514First lets look at the code we're going to use, and then dive into explaining

539First lets look at the code we're going to use, and then dive into explaining

515it. If you'd like to follow along, save the following as `expr-parse.hoon` in

540it. If you'd like to follow along, save the following as `expr-parse.hoon` in

516your `gen/` folder.

541your `gen/` folder.

542+

517```hoon

543```hoon

518:: expr-parse: parse arithmetic expressions

544:: expr-parse: parse arithmetic expressions

519::

545::

@@ -544,9 +570,10 @@ your `gen/` folder.

544```

570```

545

571

546Informally, the grammar here is:

572Informally, the grammar here is:

573+

547− - A factor is either an integer or an expression surrounded by parentheses.

574+- A factor is either an integer or an expression surrounded by parentheses.

548− - A term is either a factor or a factor times a term.

575+- A term is either a factor or a factor times a term.

549− - An expression is either a term or a term plus an expression.

576+- An expression is either a term or a term plus an expression.

550

577

551### Factors, terms, and expressions

578### Factors, terms, and expressions

552

579

@@ -568,8 +595,7 @@ for expressions.

568A _factor_ is either a decimal number or an expression surrounded by parentheses. Put

595A _factor_ is either a decimal number or an expression surrounded by parentheses. Put

569into Hoon terms, a decimal number is parsed by the `rule` `+dem` and an

596into Hoon terms, a decimal number is parsed by the `rule` `+dem` and an

570expression is parsed by removing the surrounding parentheses and then passing

597expression is parsed by removing the surrounding parentheses and then passing

571−the result to the expression parser arm `+expr`, given by the `rule` `(ifix [pal

598+the result to the expression parser arm `+expr`, given by the `rule` `(ifix [pal par] expr)`. Since we want to parse our expression with one or the other, we

572−par] expr)`. Since we want to parse our expression with one or the other, we

573chain these two `rule`s together using the monadic applicator rune `;~` along

599chain these two `rule`s together using the monadic applicator rune `;~` along

574with `+pose`, which says to try each rule in succession until one of them works.

600with `+pose`, which says to try each rule in succession until one of them works.

575

601

@@ -578,6 +604,7 @@ recursive rule. Thus we need to make use of `+knee`. The first argument for

578`+knee` is `*@ud`, since our final answer should be a `@ud`.

604`+knee` is `*@ud`, since our final answer should be a `@ud`.

579

605

580Then follows the definition of the gate utilized by `+knee`:

606Then follows the definition of the gate utilized by `+knee`:

607+

581```hoon

608```hoon

582 |. ~+

609 |. ~+

583 ;~ pose

610 ;~ pose

@@ -585,6 +612,7 @@ Then follows the definition of the gate utilized by `+knee`:

585 (ifix [pal par] expr)

612 (ifix [pal par] expr)

586 ==

613 ==

587```

614```

615+

588`~+` is used to cache the parser, so that it does not need to be computed over

616`~+` is used to cache the parser, so that it does not need to be computed over

589and over again. Then it follows the `rule` we described above.

617and over again. Then it follows the `rule` we described above.

590

618

@@ -615,7 +643,6 @@ accomplish this:

615The rest of the `+expr` arm is structured just like how `+factor` is, and for

643The rest of the `+expr` arm is structured just like how `+factor` is, and for

616the same reasons.

644the same reasons.

617

645

618−

619#### Parsing terms

646#### Parsing terms

620

647

621A _term_ is either a factor times a term or a factor. This is handled similarly

648A _term_ is either a factor times a term or a factor. This is handled similarly