@@ -9,6 +9,8 @@ knowledge of parsing is required, and we will explain the basic structure of how
99parsing works in a purely functional language such as Hoon before moving on to
1010how it is implemented in Hoon.
1111
12+**Note:** For JSON printing/parsing and encoding/decoding, see the [JSON Guide](/docs/hoon/guides/json-guide).
13+
1214## What is parsing? {#what-is-parsing}
1315
1416A program which takes a raw sequence of characters as an input and produces a data
@@ -42,20 +44,24 @@ the parsing routine on the string `"1"` and looks for an integer, and so it
4244returns the integer `1`. However, taking into account what was said above about
4345parsers returning the unparsed portion of the string as well, we should
4446represent this return as a tuple. So we should expect something like this:
47+
4548```
4649> (parse "1" integer)
4750[1 ""]
4851> (parse "123" integer)
4952[1 "23"]
5053```
54+
5155What if we wish to parse the rest of the string? We would need to apply the
5256`parse` function again:
57+
5358```
5459> (parse (parse "123" integer) integer)
5560[12 "3"]
5661> (parse (parse (parse "123" integer) integer) integer)
5762[123 ""]
5863```
64+
5965So we see that we can parse strings larger than one character by stringing
6066together parsing functions for single characters. Thus in addition to parsing
6167functions for single input characters, we want _parser combinators_ that
@@ -75,15 +81,14 @@ are quite a few), but ought to be
7581sufficient to get started with parsing in Hoon and be equipped to discover the
7682remainder yourself.
7783
78−
7984## Basic types
8085
8186In this section we discuss the types most commonly used for Hoon parsers. In short:
8287
83− - A `hair` is the position in the text the parser is at,
84− - A `nail` is parser input,
85− - An `edge` is parser output,
86− - A `rule` is a parser.
88+- A `hair` is the position in the text the parser is at,
89+- A `nail` is parser input,
90+- An `edge` is parser output,
91+- A `rule` is a parser.
8792
8893### `hair`
8994
@@ -102,6 +107,7 @@ is in case it hits something unexpected during parsing.
102107```hoon
103108++ nail [p=hair q=tape]
104109```
110+
105111We recall from our [discussion above](#what-is-parsing) that parsing functions must keep
106112track of both what has been parsed and what has yet to be parsed. Thus a `nail`
107113consists of both a `hair`, giving the line and column up to which the input
@@ -122,6 +128,7 @@ what has theoretically already been parsed up to that point.
122128```hoon
123129++ edge [p=hair q=(unit [p=* q=nail])]
124130```
131+
125132An `edge` is the output of a parser. If parsing succeeded, `p` is the location
126133of the original input `tape `up to which the text has been parsed. If parsing
127134failed, `p` will be the first `hair` at which parsing failed.
@@ -136,6 +143,7 @@ of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.
136143```hoon
137144++ rule _|:($:nail $:edge)
138145```
146+
139147A `rule` is a gate which takes in a `nail` and returns an `edge` - in other
140148words, a parser.
141149
@@ -160,6 +168,7 @@ of the input `nail`.
160168> edg
161169[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]
162170```
171+
163172We note that `p.edg` is `[p=1 q=2]`, indicating that the next character to be
164173parsed is in line 1, column 2. `q.edg` is not null, indicating that parsing
165174succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is the same as `p.edg`, which is always the case for
@@ -167,11 +176,13 @@ succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is
167176`q.q.edg` is `"bc"`, which is the part of the input `tape` that has yet to be parsed.
168177
169178Now let's see what happens when parsing fails.
179+
170180```
171181> =edg ((just 'b') [[1 1] "abc"])
172182> edg
173183[p=[p=1 q=1] q=~]
174184```
185+
175186Now we have that `p.edg` is the same as the input `hair`, `[1 1]`, meaning the
176187parser has not advanced since parsing failed. `q.edg` is null, indicating that
177188parsing has failed.
@@ -186,11 +197,13 @@ produces a `rule` that attempts to match that `cord` against the beginning of
186197the input.
187198
188199Let's see what happens when we successfully parse the entire input `tape`.
200+
189201```
190202> =edg ((jest 'abc') [[1 1] "abc"])
191203> edg
192204[p=[p=1 q=4] q=[~ [p='abc' q=[p=[p=1 q=4] q=""]]]]
193205```
206+
194207`p.edg` is `[p=1 q=4]`, indicating that the next character to be parsed is at
195208line 1, column 4. Of course, this does not exist since the input `tape` was only
1962093 characters long, so this actually indicates that the entire `tape` has been
@@ -199,21 +212,25 @@ successfully parsed (since the `hair` does not advance in the case of failure).
199212remains to be parsed.
200213
201214What happens if we only match some of the input `tape`?
215+
202216```
203217> =edg ((jest 'ab') [[1 1] "abc"])
204218> edg
205219[p=[p=1 q=3] q=[~ [p='ab' q=[p=[p=1 q=3] q="c"]]]]
206220```
221+
207222Now we have that the result, `p.q.edg`, is `'ab'`, while the remainder `q.q.q.edg`
208223is `"c"`. So `+jest` has successfully parsed the first two characters, while the
209224last character remains. Furthermore, we still have the information that the
210225remaining character was in line 1 column 3 from `p.edg` and `p.q.q.edg`.
211226
212227What happens when `+jest` fails?
228+
213229```
214230> ((jest 'bc') [[1 1] "abc"])
215231[p=[p=1 q=1] q=~]
216232```
233+
217234Despite the fact that `'bc'` appears in `"abc"`, because it was not at the
218235beginning the parse failed. We will see in [parser
219236combinators](#parser-combinators) how to modify this `rule` so that it
@@ -237,6 +254,7 @@ replaces the parsing result with `cus`.
237254
238255Here we see that `p.q` of the `edge` returned by the `rule` created with `+cold`
239256is `%foo`.
257+
240258```
241259> ((cold %foo (just 'a')) [[1 1] "abc"])
242260[p=[p=1 q=2] q=[~ u=[p=%foo q=[p=[p=1 q=2] q="bc"]]]]
@@ -246,7 +264,6 @@ One common scenario where `+cold` sees play is when writing [command line
246264interface (CLI) apps](/docs/hoon/guides/cli-tutorial). We usher the
247265reader there to find an example where `+cold` is used.
248266
249−
250267### `+knee`
251268
252269Another important function in the parser builder library is `+knee`, used for building
@@ -254,7 +271,6 @@ recursive parsers. We delay discussion of `+knee` to the
254271[section below](#recursive-parsers) as more context is needed to explain it
255272properly.
256273
257−
258274## Outside callers
259275
260276Since `hair`s, `nail`s, etc. are only utilized within the context of writing
@@ -295,7 +311,6 @@ For additional information including additional examples see [4g: Parsing (Outsi
295311For the remainder of this tutorial we will make use of `+scan` so that we do not
296312need to deal directly with `nail`s except where it is ilustrative to do so.
297313
298−
299314### Parsing atoms
300315
301316[Recall](/docs/hoon/hoon-school/lists) that `cord`s are atoms with the aura
@@ -312,7 +327,6 @@ The standard library provides a number of gates that take a `rule` and
312327produce a new modified `rule` according to some process. We call these _parser modifiers_. These are
313328documented among the [parser builders](/docs/hoon/reference/stdlib/4f).
314329
315−
316330### `+ifix`
317331
318332`+ifix` modifies a `rule` so that it matches that `rule` only when it is
@@ -322,11 +336,11 @@ surrounded on both sides by text that matches a pair of `rule`s, which is discar
322336> (scan "(42)" (ifix [pal par] (jest '42')))
323337'42'
324338```
339+
325340`+pal` and `+par` are shorthand for `(just '(')` and `(just ')')`, respectively. All
326341ASCII glyphs have counterparts of this sort, documented
327342[here](/docs/hoon/reference/stdlib/4h).
328343
329−
330344### `+star` {#star}
331345
332346`+star` is used to apply a `rule` repeatedly. Recall that `+just` only parses
@@ -343,7 +357,9 @@ We can use `+star` to get the rest of the `tape`:
343357> ((star (just 'a')) [[1 1] "aaa"])
344358[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q=""]]]]
345359```
360+
346361and we note that the parsing ceases when it fails.
362+
347363```
348364> ((star (just 'a')) [[1 1] "aaab"])
349365[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q="b"]]]]
@@ -358,9 +374,11 @@ combinators](/docs/hoon/reference/stdlib/4e). First we introduce a few
358374combinators, then we examine more closely how `;~` is used to chain them together.
359375
360376The syntax to combine `rule`s is
377+
361378```hoon
362379;~(combinator rule1 rule2 ... ruleN)
363380```
381+
364382The `rule`s are composed together using the combinator as an
365383intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns
366384it into a sample (a `nail`) for the next `rule` to handle. We elaborate on this
@@ -409,42 +427,46 @@ take this opportunity to think about it carefully.
409427The `rule` created by `;~(combinator [list rule])` may be understood
410428inductively. To do this, let's consider the base case where our `[list rule]` has only a
411429single entry.
430+
412431```
413432> (scan "star" ;~(plug (jest 'star')))
414433'star'
415434```
435+
416436Our output is identical to that given by `scan "star" (jest 'star')`. This is to
417437be expected. The combinator `+plug` is specifically used for chaining together
418438`rule`s in the `[list rule]`, but if there is only one `rule`, there is nothing
419439to chain. Thus, swapping out `+plug` for another combinator makes no difference here:
440+
420441```
421442> (scan "star" ;~(pose (jest 'star')))
422443'star'
423444> (scan "star" ;~((glue com) (jest 'star')))
424445'star'
425446```
447+
426448`;~` and the combinator only begin to play a role once the `[list rule]` has at
427449least two elements. So let's look at an example done with `+plug`, the simplest
428450combinator.
451+
429452```
430453> (scan "star" ;~(plug (jest 'st') (jest 'ar')))
431454['st' 'ar']
432455```
456+
433457Our return suggests that we first parsed `"star"` with the `rule` `(jest 'st')` and passed
434−the resulting `edge` to `(jest 'ar')` - in other words, we called `+plug` on `(jest
435−'st')` and the `edge` returned once it had been used to parse `"star"`. Thus
458+the resulting `edge` to `(jest 'ar')` - in other words, we called `+plug` on `(jest 'st')` and the `edge` returned once it had been used to parse `"star"`. Thus
436459`+plug` was the glue that allowed us to join the two `rule`s, and `;~` performed
437460the gluing operation. And so, swapping `+plug` for `+pose` results in a crash,
438461which clues us into the fact that the combinator now has an effect since there
439462is more than one `rule`.
463+
440464```
441465> (scan "star" ;~(pose (jest 'st') (jest 'ar')))
442466{1 3}
443467syntax error
444468```
445469
446−
447−
448470## Parsing numbers
449471
450472Functions for parsing numbers are documented in [4j: Parsing (Bases and Base
@@ -461,15 +483,19 @@ for parsing decimal numbers.
461483## Recursive parsers
462484
463485Naively attempting to write a recursive `rule`, i.e. like
486+
464487```
465488> |-(;~(pose den ;~(pose $ (easy ~))))
466489```
490+
467491results in an error.
492+
468493```
469494-find.,.+6
470495-find.,.+6
471496rest-loop
472497```
498+
473499Thus some special sauce is required, the `+knee` function.
474500
475501`+knee` takes in a noun that is the default value of the parser, typically given
@@ -494,7 +520,6 @@ the parser to improve performance. In [parsing arithmetic
494520expressions](#parsing-arithmetic-expressions) we will be writing a recursive
495521parser making use of `+knee` and `~+` throughout.
496522
497−
498523# Parsing arithmetic expressions
499524
500525In this section we will be applying what we have learned to write a parser for
@@ -514,6 +539,7 @@ for arithmetic expressions informally.
514539First lets look at the code we're going to use, and then dive into explaining
515540it. If you'd like to follow along, save the following as `expr-parse.hoon` in
516541your `gen/` folder.
542+
517543```hoon
518544:: expr-parse: parse arithmetic expressions
519545::
@@ -544,9 +570,10 @@ your `gen/` folder.
544570```
545571
546572Informally, the grammar here is:
547− - A factor is either an integer or an expression surrounded by parentheses.
548− - A term is either a factor or a factor times a term.
549− - An expression is either a term or a term plus an expression.
573+
574+- A factor is either an integer or an expression surrounded by parentheses.
575+- A term is either a factor or a factor times a term.
576+- An expression is either a term or a term plus an expression.
550577
551578### Factors, terms, and expressions
552579
@@ -568,8 +595,7 @@ for expressions.
568595A _factor_ is either a decimal number or an expression surrounded by parentheses. Put
569596into Hoon terms, a decimal number is parsed by the `rule` `+dem` and an
570597expression is parsed by removing the surrounding parentheses and then passing
571−the result to the expression parser arm `+expr`, given by the `rule` `(ifix [pal
572−par] expr)`. Since we want to parse our expression with one or the other, we
598+the result to the expression parser arm `+expr`, given by the `rule` `(ifix [pal par] expr)`. Since we want to parse our expression with one or the other, we
573599chain these two `rule`s together using the monadic applicator rune `;~` along
574600with `+pose`, which says to try each rule in succession until one of them works.
575601
@@ -578,6 +604,7 @@ recursive rule. Thus we need to make use of `+knee`. The first argument for
578604`+knee` is `*@ud`, since our final answer should be a `@ud`.
579605
580606Then follows the definition of the gate utilized by `+knee`:
607+
581608```hoon
582609 |. ~+
583610 ;~ pose
@@ -585,6 +612,7 @@ Then follows the definition of the gate utilized by `+knee`:
585612 (ifix [pal par] expr)
586613 ==
587614```
615+
588616`~+` is used to cache the parser, so that it does not need to be computed over
589617and over again. Then it follows the `rule` we described above.
590618
@@ -615,7 +643,6 @@ accomplish this:
615643The rest of the `+expr` arm is structured just like how `+factor` is, and for
616644the same reasons.
617645
618−
619646#### Parsing terms
620647
621648A _term_ is either a factor times a term or a factor. This is handled similarly