@@ -11,7 +11,7 @@ knowledge of parsing is required, and we will explain the basic structure of how
11parsing works in a purely functional language such as Hoon before moving on to
11parsing works in a purely functional language such as Hoon before moving on to
12how it is implemented in Hoon.
12how it is implemented in Hoon.
14−## What is parsing?
14+## What is parsing? {#what-is-parsing}
16A program which takes a raw sequence of characters as an input and produces a data
16A program which takes a raw sequence of characters as an input and produces a data
17structure as an output is known as a _parser_. The data structure produced
17structure as an output is known as a _parser_. The data structure produced
@@ -60,7 +60,7 @@ What if we wish to parse the rest of the string? We would need to apply the
60So we see that we can parse strings larger than one character by stringing
60So we see that we can parse strings larger than one character by stringing
61together parsing functions for single characters. Thus in addition to parsing
61together parsing functions for single characters. Thus in addition to parsing
62functions for single input characters, we want _parser combinators_ that
62functions for single input characters, we want _parser combinators_ that
63−allow you to combine two or more simple parsers to form more complex ones.
63+allow you to combine two or more parsers to form a more complex one.
64Combinators come in a few shapes and sizes, and typical operations they may
64Combinators come in a few shapes and sizes, and typical operations they may
65perform would be to repeat the same parsing operation until the string is
65perform would be to repeat the same parsing operation until the string is
66consumed, try a few different parsing operations until one of them works,
66consumed, try a few different parsing operations until one of them works,
@@ -69,11 +69,224 @@ with Hoon in the [Parsing in Hoon](#parsing-in-hoon) section.
70# Parsing in Hoon
70# Parsing in Hoon
72−In this section we will cover the basic types and functions that are utilized for parsing in
72+In this section we will cover the basic types, parser functions, and parser
73−Hoon, and then build some simple parsers.
73+combinators that are utilized for parsing in Hoon. This is not a complete guide
74+to every parsing-related functionality in the standard library, but ought to be
75+sufficient to get started with parsing in Hoon and be equipped to discover the
78+We encourage you to follow along with this section by inputting the commands in dojo.
75## Basic types
80## Basic types
77−hair, edge, etc
82+In this section we discuss the types most commonly used for Hoon parsers. In short:
84+ - A `hair` is the position in the text the parser is at,
85+ - A `nail` is parser input,
86+ - An `edge` is parser output,
87+ - A `rule` is a parser.
95+A `hair` is a pair of `@ud` used to keep track of what has already been parsed.
96+`p` represents the column and `q` represents the line.
101+++ nail [p=hair q=tape]
103+We recall from our [discussion above](#what-is-parsing) that parsing functions must keep
104+track of both what has been parsed and what has yet to be parsed. Thus a `nail`
105+consists of both a `hair`, giving the line and column up to which the input
106+sequence has already been parsed, and a `tape`, consisting of what remains of
107+the original input string (i.e. everything after the location indicated by the
108+`nail`, including the character at that `nail`).
110+For example, if you wish to feed the entire `tape` `"abc"` into a parser, you
111+would pass it as the `nail` `[[1 1] "abc"]`. If the parser successfully parses the first
112+character, the `nail` it returns will be `[[2 1] "bc"]` (though we note that
113+parser outputs are actually `edge`s which contain a `nail`, see the following).
118+++ edge [p=hair q=(unit [p=* q=nail])]
120+An `edge` is the output of a parser. If the parse succeeded, `p` is the location
121+of the original input `tape `up to which the text has been parsed. If the parse
122+failed, `p` will be the first `hair` at which the parse failed.
124+`q` may be `~`, indicating that the parse has failed .
125+If the parse did not fail, `p.q` is the data structure that is the result of the
126+parse up to this point, while `q.q` is the `nail` which contains the remainder
127+of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.
132+++ rule _|:($:nail $:edge)
134+A `rule` is a gate which takes in a `nail` and returns an `edge` - in other
135+words, a parser. Not totally sure why its written like this, though.
137+Maybe mention this from `/app/shoe.hoon` as well?
139+|~(nail *(like [? command]))
143+++ edge {p/hair q/(unit {p/* q/nail})} :: parsing output
144+++ hair {p/@ud q/@ud} :: parsing trace
145+++ like |* a/$-(* *) :: generic edge
146+ |: b=`*`[(hair) ~] ::
151+ u=[p=(a +>-.b) q=[p=(hair -.b) q=(tape +.b)]] ::
152+++ nail {p/hair q/tape} :: parsing input
153+++ pint {p/{p/@ q/@} q/{p/@ q/@}} :: line+column range
154+++ rule _|:($:nail $:edge) :: parsing rule
155+++ spot {p/path q/pint} :: range in file
160+These functions are used to build `rule`s (i.e. parsers), and thus are often
161+called rule-builders. For a complete list of parser builders, see [4f: Parsing
162+(Rule-Builders)](@/docs/reference/library/4f.md), but also the more specific
163+functions in [4h: Parsing (ASCII
164+Glyphs)](@/docs/reference/library/4h.md), [4i: Parsing (Useful
165+Idioms)](@/docs/reference/library/4i.md), [4j: Parsing (Bases and Base
166+Digits)](@/docs/reference/library/4j.md), [4l: Atom Parsing](@/docs/reference/library/4l.md).
170+The most basic rule builder, `just` takes in a single `char` and produces a
171+`rule` that attempts to match that `char` to the first character in the `tape`
175+> =edg ((just 'a') [[1 1] "abc"])
177+[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]
179+We note that `p.edg` is `[p=1 q=2]`, indicating that the next character to be
180+parsed is in line 1, column 2. `q.edg` is not null, indicating that the parse
181+succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is the same as `p.edg`, which is always the case for
182+`rule`s built using standard library functions when the parse succeeds. Lastly,
183+`q.q.edg` is `"bc"`, which is the part of the input `tape` that has yet to be parsed.
185+Now let's see what happens when the parse fails.
187+> =edg ((just 'b') [[1 1] "abc"])
191+Now we have that `p.edg` is the same as the input `hair`, `[1 1]`, meaning the
192+parser has not advanced since the parse failed. `q.edg` is null, indicating that
193+the parse has failed.
195+In [parsing arithmetic expressions](#parsing-arithmetic-expressions) we will see
196+how to string together a sequence of `just`s in order to parse an entire string.
197+(maybe in the next section instead? seems like a good example for `;~`)
201+`jest` is a `rule` builder used to match a `cord`. It takes an input `cord` and
202+produces a `rule` that attempts to match that `cord` against the beginning of
203+the `tape` in an input `nail`.
205+Let's see what happens when we successfully parse the entire input `tape`.
207+> =edg ((jest 'abc') [[1 1] "abc"])
209+[p=[p=1 q=4] q=[~ [p='abc' q=[p=[p=1 q=4] q=""]]]]
211+`p.edg` is `[p=1 q=4]`, indicating that the next character to be parsed is at
212+line 1, column 4. Of course, this does not exist since the input `tape` was only
213+3 characters long, so this actually indicates that the entire `tape` has been
214+successfully parsed (since the `hair` does not advance in the case of failure).
215+`p.q.edg` is `'abc'`, as expected. `q.q.edg` is `""`, indicating that nothing
216+remains to be parsed.
218+What happens if we only match some of the input `tape`?
220+> =edg ((jest 'ab') [[1 1] "abc"])
222+[p=[p=1 q=3] q=[~ [p='ab' q=[p=[p=1 q=3] q="c"]]]]
224+Now we have that the result, `p.q.edg`, is `'ab'`, while the remainder `q.q.q.edg`
225+is `"c"`. So `jest` has successfully parsed the first two characters, while the
226+last character remains. Furthermore, we still have the information that the
227+remaining character was in line 1 column 3 from `p.edg` and `p.q.q.edg`.
229+What happens when `jest` fails?
231+> ((jest 'bc') [[1 1] "abc"])
234+Despite the fact that `'bc'` appears in `"abc"`, because it was not at the
235+beginning the parse failed. We will see in [basic parser
236+combinators](#basic-parser-combinators) how to modify this `rule` so that it
237+finds `bc` successfully.
239+## Basic parser combinators
245+Since `hair`s, `nail`s, etc. are only utilized within the context of writing
246+parsers, we'd like to hide them from the rest of the code of a program that
247+utilizes parsers. That is to say, you'd like the programmer to only worry about
248+passing `tape`s to the parser, and not have to dress up the `tape` as a `nail`
249+themselves. Thus we have several functions for exactly this purpose.
251+These functions take in either a `tape` or an `@` (typically representing a
252+`cord`) with a `rule` and attempts to parse the input with the `rule`. If the
253+parse succeeds, it returns the result, otherwise it crashes with a stack trace or returns null.
255+Here we cover the basics, for complete information including additional examples see [4g: Parsing (Outside Caller)](@/docs/reference/library/4g.md).
259+`scan` takes in a `tape` and a `rule` and attempts to parse the `tape` with the rule.
262+> (scan "hello" (jest 'hello'))
264+> (scan "hello zod" (jest 'hello'))
269+`rust` works the same way as `scan`, except it returns a `unit` of the result,
270+which is null if the parse failed.
273+> (rust "a" (just 'a'))
275+> (rust "a" (just 'b'))
281+[Recall](@/docs/tutorials/hoon/lists.md) that `cord`s are atoms with the aura
282+`@t` and are typically used to represent strings internally as data, as atoms
283+are faster for the computer to work with than `tape`s, which are `list`s of
284+`@tD` atoms. `rash` and `rush` are for parsing atoms, with `rash` being
285+analogous to `scan` and `rush` being analogous to `rust`. Under the hood, `rash`
286+calls `scan` after converting the input atom to a `tape`, and `rush` does
79## Parsing arithmetic expressions
290## Parsing arithmetic expressions