@@ -0,0 +1,638 @@
2+title = "Parsing tutorial"
9+This document serves as an introduction to parsing text with Hoon. No prior
10+knowledge of parsing is required, and we will explain the basic structure of how
11+parsing works in a purely functional language such as Hoon before moving on to
12+how it is implemented in Hoon.
14+## What is parsing? {#what-is-parsing}
16+A program which takes a raw sequence of characters as an input and produces a data
17+structure as an output is known as a _parser_. The data structure produced
18+depends on the use case, but often it may be represented as a tree and the
19+output is thought of as a structural representation of the input. Parsers are ubiquitous in
20+computing, commonly used for to perform tasks such as reading files, compiling
21+source code, or understanding commands input in a command line interface.
23+Parsing a string is rarely done all at once. Instead, it is usually done
24+character-by-character, and the return contains the data structure representing
25+what has been parsed thus far as well as the remainder of the string to be
26+parsed. They also need to be able to fail in case the input is improperly
27+formed. We will see each of these standard practices implemented in Hoon below.
31+How parsers are built varies substantially depending on what sort of programming
32+language it is written in. As Hoon is a functional programming language, we will
33+be focused on understanding _functional parsers_, also known as _combinator
36+Complex functional parsers are built piece by piece from simpler parsers
37+that are plugged into one another in various ways to perform the desired task.
39+The basic building blocks, or primitives, are parsers that read only a
40+single character. There are frequently a few types of possible input characters,
41+such as letters, numbers, and symbols. For example, `parse(integer, "1")` calls
42+the parsing routine on the string `"1"` and looks for an integer, and so it
43+returns the integer `1`. However, taking into account what was said above about
44+parsers returning the unparsed portion of the string as well, we should
45+represent this return as a tuple. So we should expect something like this:
49+> parse(integer, "123")
52+What if we wish to parse the rest of the string? We would need to apply the
53+`parse(integer, -)` function again:
55+> parse(integer, parse(integer, "123"))
57+> parse(integer, parse(integer, parse(integer, "123")))
60+So we see that we can parse strings larger than one character by stringing
61+together parsing functions for single characters. Thus in addition to parsing
62+functions for single input characters, we want _parser combinators_ that
63+allow you to combine two or more parsers to form a more complex one.
64+Combinators come in a few shapes and sizes, and typical operations they may
65+perform would be to repeat the same parsing operation until the string is
66+consumed, try a few different parsing operations until one of them works,
67+or perform a sequence of parsing operations. We will see how all of this is done
68+with Hoon in the [Parsing in Hoon](#parsing-in-hoon) section.
72+In this section we will cover the basic types, parser functions, and parser
73+combinators that are utilized for parsing in Hoon. This is not a complete guide
74+to every parsing-related functionality in the standard library (of which there
75+are quite a few), but ought to be
76+sufficient to get started with parsing in Hoon and be equipped to discover the
79+We encourage you to follow along with this section by inputting the relevant
80+commands in dojo as they are presented.
84+In this section we discuss the types most commonly used for Hoon parsers. In short:
86+ - A `hair` is the position in the text the parser is at,
87+ - A `nail` is parser input,
88+ - An `edge` is parser output,
89+ - A `rule` is a parser.
97+A `hair` is a pair of `@ud` used to keep track of what has already been parsed.
98+`p` represents the column and `q` represents the line.
103+++ nail [p=hair q=tape]
105+We recall from our [discussion above](#what-is-parsing) that parsing functions must keep
106+track of both what has been parsed and what has yet to be parsed. Thus a `nail`
107+consists of both a `hair`, giving the line and column up to which the input
108+sequence has already been parsed, and a `tape`, consisting of what remains of
109+the original input string (i.e. everything after the location indicated by the
110+`nail`, including the character at that `nail`).
112+For example, if you wish to feed the entire `tape` `"abc"` into a parser, you
113+would pass it as the `nail` `[[1 1] "abc"]`. If the parser successfully parses the first
114+character, the `nail` it returns will be `[[2 1] "bc"]` (though we note that
115+parser outputs are actually `edge`s which contain a `nail`, see the following).
120+++ edge [p=hair q=(unit [p=* q=nail])]
122+An `edge` is the output of a parser. If the parse succeeded, `p` is the location
123+of the original input `tape `up to which the text has been parsed. If the parse
124+failed, `p` will be the first `hair` at which the parse failed.
126+`q` may be `~`, indicating that the parse has failed .
127+If the parse did not fail, `p.q` is the data structure that is the result of the
128+parse up to this point, while `q.q` is the `nail` which contains the remainder
129+of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.
134+++ rule _|:($:nail $:edge)
136+A `rule` is a gate which takes in a `nail` and returns an `edge` - in other
137+words, a parser. Not totally sure why its written like this, though.
139+Maybe mention this from `/app/shoe.hoon` as well?
141+|~(nail *(like [? command]))
147+These functions are used to build `rule`s (i.e. parsers), and thus are often
148+called rule-builders. For a complete list of parser builders, see [4f: Parsing
149+(Rule-Builders)](@/docs/reference/library/4f.md), but also the more specific
150+functions in [4h: Parsing (ASCII
151+Glyphs)](@/docs/reference/library/4h.md), [4i: Parsing (Useful
152+Idioms)](@/docs/reference/library/4i.md), [4j: Parsing (Bases and Base
153+Digits)](@/docs/reference/library/4j.md), [4l: Atom Parsing](@/docs/reference/library/4l.md).
157+The most basic rule builder, `+just` takes in a single `char` and produces a
158+`rule` that attempts to match that `char` to the first character in the `tape`
162+> =edg ((just 'a') [[1 1] "abc"])
164+[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]
166+We note that `p.edg` is `[p=1 q=2]`, indicating that the next character to be
167+parsed is in line 1, column 2. `q.edg` is not null, indicating that the parse
168+succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is the same as `p.edg`, which is always the case for
169+`rule`s built using standard library functions when the parse succeeds. Lastly,
170+`q.q.edg` is `"bc"`, which is the part of the input `tape` that has yet to be parsed.
172+Now let's see what happens when the parse fails.
174+> =edg ((just 'b') [[1 1] "abc"])
178+Now we have that `p.edg` is the same as the input `hair`, `[1 1]`, meaning the
179+parser has not advanced since the parse failed. `q.edg` is null, indicating that
180+the parse has failed.
182+Later we will use [+star](#star) to string together a sequence of `+just`s in
183+order to parse multiple characters at once.
187+`+jest` is a `rule` builder used to match a `cord`. It takes an input `cord` and
188+produces a `rule` that attempts to match that `cord` against the beginning of
189+the `tape` in an input `nail`.
191+Let's see what happens when we successfully parse the entire input `tape`.
193+> =edg ((jest 'abc') [[1 1] "abc"])
195+[p=[p=1 q=4] q=[~ [p='abc' q=[p=[p=1 q=4] q=""]]]]
197+`p.edg` is `[p=1 q=4]`, indicating that the next character to be parsed is at
198+line 1, column 4. Of course, this does not exist since the input `tape` was only
199+3 characters long, so this actually indicates that the entire `tape` has been
200+successfully parsed (since the `hair` does not advance in the case of failure).
201+`p.q.edg` is `'abc'`, as expected. `q.q.edg` is `""`, indicating that nothing
202+remains to be parsed.
204+What happens if we only match some of the input `tape`?
206+> =edg ((jest 'ab') [[1 1] "abc"])
208+[p=[p=1 q=3] q=[~ [p='ab' q=[p=[p=1 q=3] q="c"]]]]
210+Now we have that the result, `p.q.edg`, is `'ab'`, while the remainder `q.q.q.edg`
211+is `"c"`. So `+jest` has successfully parsed the first two characters, while the
212+last character remains. Furthermore, we still have the information that the
213+remaining character was in line 1 column 3 from `p.edg` and `p.q.q.edg`.
215+What happens when `+jest` fails?
217+> ((jest 'bc') [[1 1] "abc"])
220+Despite the fact that `'bc'` appears in `"abc"`, because it was not at the
221+beginning the parse failed. We will see in [parser
222+combinators](#parser-combinators) how to modify this `rule` so that it
223+finds `bc` successfully.
227+`+shim` is used to parse characters within a given range. It takes in two atoms
228+and returns a `rule`.
231+> ((shim 'a' 'z') [[1 1] "abc"])
232+[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]
237+`+cold` is a `rule` builder that takes in a constant noun we'll call `cus` and a
238+`rule` we'll call `sef` and returns a `rule` identical to the `sef` except it
239+returns `cus` when it parses successfully.
241+Here we see that `p.q` of the `edge` returned by the `rule` created with `+cold`
244+> ((cold %foo (just 'a')) [[1 1] "abc"])
245+[p=[p=1 q=2] q=[~ u=[p=%foo q=[p=[p=1 q=2] q="bc"]]]]
248+One common scenario where `+cold` sees play is when writing [command line
249+interface (CLI) apps](@/docs/tutorials/arvo/cli-tutorial.md). We usher the
250+reader there to find an example where `+cold` is used.
255+Another important function in the parser builder library is `+knee`, used for building
256+recursive parsers. We delay discussion of `+knee` to the
257+[section below](#recursive-parsers) as more context is needed to explain it
263+Since `hair`s, `nail`s, etc. are only utilized within the context of writing
264+parsers, we'd like to hide them from the rest of the code of a program that
265+utilizes parsers. That is to say, you'd like the programmer to only worry about
266+passing `tape`s to the parser, and not have to dress up the `tape` as a `nail`
267+themselves. Thus we have several functions for exactly this purpose.
269+These functions take in either a `tape` or an `@` (typically representing a
270+`cord`) with a `rule` and attempts to parse the input with the `rule`. If the
271+parse succeeds, it returns the result, otherwise it crashes with a stack trace or returns null.
273+Here we cover the basics, for complete information including additional examples see [4g: Parsing (Outside Caller)](@/docs/reference/library/4g.md).
277+`+scan` takes in a `tape` and a `rule` and attempts to parse the `tape` with the `rule`.
280+> (scan "hello" (jest 'hello'))
282+> (scan "hello zod" (jest 'hello'))
287+`+rust` works the same way as `+scan`, except it returns a `unit` of the result,
288+which is null if the parse failed.
291+> (rust "a" (just 'a'))
293+> (rust "a" (just 'b'))
297+For the remainder of this tutorial we will make use of `+scan` so that we do not
298+need to deal directly with `nail`s except where it is ilustrative to do so.
302+The standard library provides a number of gates that take a `rule` and
303+produce a new modified `rule` according to some process. We call these _parser modifiers_. These are
304+documented among the [parser builders](@/docs/reference/library/4f.md).
308+`+star` is used to apply a `rule` repeatedly. Recall that `+just` only parses
309+the first character in the input `tape.`
312+> ((just 'a') [[1 1] "aaaa"])
313+[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="aaa"]]]]
316+We can use `+star` to get the rest of the `tape`:
319+> ((star (just 'a')) [[1 1] "aaa"])
320+[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q=""]]]]
322+and we note that the parsing ceases when it fails.
324+> ((star (just 'a')) [[1 1] "aaab"])
325+[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q="b"]]]]
330+`+ifix` modifies a `rule` so that it matches that `rule` only when it is
331+surrounded on both sides by text that matches a pair of `rule`s, which is discarded.
334+> (scan "(42)" (ifix [lit rit] (jest '42')))
337+`+lit` and `+rit` are shorthand for `(just '(')` and `(just ')')`, respectively. All
338+ASCII glyphs have counterparts of this sort, documented
339+[here](@/docs/reference/library/4h.md).
342+## Parser combinators
344+Building complex parsers from simpler parsers is accomplished in Hoon with the
345+use of two tools: the monadic applicator rune
346+[`;~`](@/docs/reference/hoon-expressions/rune/mic.md#micsig) and [parsing
347+combinators](@/docs/reference/library/4e.md). First we introduce a few
348+combinators, then we examine more closely how `;~` is used to chain them together.
350+The syntax to combine `rule`s is
352+;~(combinator [list rule])
354+The `rule`s are composed together using the combinator as an
355+intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns
356+it into a sample (a `nail`) for the next `rule` to handle. We elaborate on this
357+behavior [below](#micsig).
361+`+plug` simply takes the `nail` in the `edge` produced by one rule and passes it
362+to the next `rule`, forming a cell of the results as it proceeds.
365+> (scan "starship" ;~(plug (jest 'star') (jest 'ship')))
371+`+pose` tries each `rule` you hand it successively until it finds one that
375+> (scan "a" ;~(pose (just 'a') (just 'b')))
377+> (scan "b" ;~(pose (just 'a') (just 'b')))
383+`+glue` parses a delimiter in between each `rule` and forms a cell of the
384+results of each `rule`.
387+> (scan "a,b" ;~((glue com) (just 'a') (just 'b')))
389+> (scan "a,b,a" ;~((glue com) (just 'a') (just 'b')))
396+Understanding the rune `;~` is essential to building parsers with Hoon. Let's
397+take this opportunity to think about it carefully.
399+The `rule` created by `;~(combinator [list rule])` may be understood
400+inductively. To this, let's consider the base case where our `[list rule]` has only a
403+> (scan "star" ;~(plug (jest 'star')))
406+Our output is identical to that given by `scan "star" (jest 'star')`. This is to
407+be expected. The combinator `+plug` is specifically used for chaining together
408+`rule`s in the `[list rule]`, but if there is only one `rule`, there is nothing
409+to chain. Thus, swapping out `+plug` for another combinator makes no difference here:
411+> (scan "star" ;~(pose (jest 'star')))
413+> (scan "star" ;~((glue com) (jest 'star')))
416+`;~` and the combinator only begin to play a role once the `[list rule]` has at
417+least two elements. So let's look at an example done with `+plug`, the simplest
420+> (scan "star" ;~(plug (jest 'st') (jest 'ar')))
423+Our return suggests that we first parsed `"star"` with the `rule` `(jest 'st')` and passed
424+the resulting `edge` to `(jest 'ar')` - in other words, we called `+plug` on `(jest
425+'st')` and the `edge` returned once it had been used to parse `"star"`. Thus
426+`+plug` was the glue that allowed us to join the two `rule`s, and `;~` performed
427+the gluing operation. And so, swapping `+plug` for `+pose` results in a crash,
428+which clues us into the fact that the combinator now has an effect since there
429+is more than one `rule`.
431+`> (scan "star" ;~(pose (jest 'st') (jest 'ar')))
438+[Recall](@/docs/tutorials/hoon/lists.md) that `cord`s are atoms with the aura
439+`@t` and are typically used to represent strings internally as data, as atoms
440+are faster for the computer to work with than `tape`s, which are `list`s of
441+`@tD` atoms. `+rash` and `+rush` are for parsing atoms, with `+rash` being
442+analogous to `+scan` and `+rush` being analogous to `rust`. Under the hood, `+rash`
443+calls `+scan` after converting the input atom to a `tape`, and `+rush` does
444+similary for `+rust`.
448+Functions for parsing numbers are documented in [4j: Parsing (Bases and Base
449+Digits)](@/docs/reference/library/4j.md). In particualr, `dem` is a `rule`
450+for parsing decimal numbers.
455+> (add 1 (scan "42" dem))
461+Naively attempting to write a recursive `rule`, i.e. like
463+> |-(;~(pose den ;~(pose $ (easy ~))))
465+results in an error, as the compiler will attempt to build an infinite function.
471+Thus some special sauce is required, the `+knee` function.
473+`+knee` takes in a noun that is the default value of the parser, typically given
474+as the bunt value of the type that the `rule` produces, as well as a gate that
475+accepts a `rule`. `+knee` produces a `rule` that implements any recursive calls
476+in the `rule` in a manner acceptable to the compiler. Thus the preferred manner
477+to write the above `rule` is as follows:
479+|-(;~(plug prn ;~(pose (knee *tape |.(^$)) (easy ~))))
482+You may want to utilize the `~+` rune when writing recursive parsers to cache
483+the results as you go to improve performance. In [parsing arithmetic
484+expressions](#parsing-arithmetic-expressions) we will be writing a recursive
485+parser making use of `+knee` and `~+` throughout.
488+# Parsing arithmetic expressions
490+In this section we will be applying what we have learned to write a parser for
491+arithmetic expressions in Hoon. That is, we will make a `rule` that takes in
492+`tape`s of the form `"(2+3)*4"` and returns `20` as a `@ud`.
494+We call a `tape` consisting of some consistent arithmetic string of numbers,
495+`+,*`, `(`, and `)` an _expression_. We wish to build a `rule` that takes in an
496+expression and returns the result of the arithmetic computation described by the
497+expression as a `@ud`.
499+To build a parser it is a helpful exercise to first describe its
500+[grammar](https://en.wikipedia.org/wiki/Parsing_expression_grammar). This has a
501+formal mathematical definition, but we will manage to get by here describing the grammar
502+for arithmetic expressions informally.
504+First lets look at the code we're going to use, and then dive into explaining
505+it. If you'd like to follow along, save the following as `expr-parse.hoon` in
508+:: This generator parses arithmetic expressions
522+ (ifix [lit rit] expr)
527+ ((slug mul) tar ;~(pose factor turm))
533+ ((slug add) lus ;~(pose turm expr))
539+### Factors, terms, and expressions
541+Our grammar consists of three `rule`s: one for factors, one for terms, and one
551+ (ifix [lit rit] expr)
555+A _factor_ is either a decimal number or an expression surrounded by parentheses. Put
556+into Hoon terms, a decimal number is parsed by the `rule` `+dem` and an
557+expression is parsed by removing the surrounding parentheses and then passing
558+the result to the expression parser arm `+expr`, given by the `rule` `(ifix [lit
559+rit] expr)`. Since we want to parse our expression with one or the other, we
560+chain these two `rule`s together using the monadic applicator rune `;~` along
561+with `+pose`, which says to try each rule in succession until one of them works.
563+Since expressions ultimately reduce to factors, we are actually building a
564+recursive rule. Thus we need to make use of `+knee`. The first argument for
565+`+knee` is `*int`, since our final answer should be a `@ud`, to which we have
566+assigned the macro `int`.
568+Then follows the definition of the gate utilized by `+knee`:
572+ (ifix [lit rit] expr)
575+`|.` is used instead of `|-` since... `~+` is used to cache results as we go -
576+there is no point in computing `2+2` more than once. Then follows the `rule` we
579+#### Parsing expressions
581+An _expression_ is either a term plus an expression or a term.
584+term plus an expression, we actually must compute what that equals. Thus we will
585+make use of [`+slug`](@/docs/reference/library/4f.md#slug), which parses a
586+delimited list into `tape`s separated by a given delimiter and then composes
587+them by folding with a binary gate. In this case, our delimiter is `+` and our
588+binary gate is `+add`. That is to say, we will split the input string into terms
589+and expressions separated by luses, parse each term and expression until they
590+reduce to a `@ud`, and then add them together. This is accomplished with the
591+rule `((slug add) lus ;~(pose turm expr))`.
593+If the above `rule` does not parse the expression, it must be a `turm`, so we
594+pass the `tape` to `+turm` to be evaluated. Again we use `;~` and `pose` to
599+ ((slug add) lus ;~(pose turm expr))
604+The rest of the `+expr` arm is structured just like how `+factor` is, and for
610+A _term_ is either a factor times a term or a factor. This is handled similarly
611+for expressions, we just need to swap `lus` for `tar`, `add` for `mul`, and
612+`;~(pose factor turm)` instead of `;(pose turm expr)`.
618+ ((slug add) lus ;~(pose turm expr))
625+Let's feed some expressions to `+expr-parse` and see how it does.
632+> +expr-parse "3+3+(2*3)+(4+2)*(4+1)"
634+> +expr-parse "3+3+2*3"
638+As an exercise, add exponentiation (e.g. `2^3 = 8`) to `+expr-parse`.