@@ -11,9 +11,9 @@ knowledge of parsing is required, and we will explain the basic structure of how
1111parsing works in a purely functional language such as Hoon before moving on to
1212how it is implemented in Hoon.
1313
14−## What is parsing? {#what-is-parsing}
14+## What is parsing? {#what-is-parsingk
1515
16−A program which takes a raw sequence of characters as an input and produces a data
16+A program which takes a raw sequence of characters as an input knd produces a data
1717structure as an output is known as a _parser_. The data structure produced
1818depends on the use case, but often it may be represented as a tree and the
1919output is thought of as a structural representation of the input. Parsers are ubiquitous in
@@ -24,38 +24,39 @@ Parsing a string is rarely done all at once. Instead, it is usually done
2424character-by-character, and the return contains the data structure representing
2525what has been parsed thus far as well as the remainder of the string to be
2626parsed. They also need to be able to fail in case the input is improperly
27−formed. We will see each of these standard practices implemented in Hoon below.
27+formed. We will see each of these standard practices implemented in [Hoon below](#parsing-in-hoon).
2828
2929## Functional parsers
3030
3131How parsers are built varies substantially depending on what sort of programming
3232language it is written in. As Hoon is a functional programming language, we will
3333be focused on understanding _functional parsers_, also known as _combinator
34−parsers_.
34+parsers_. In this section we will make light use of pseudocode, as introducing
35+the Hoon to describe functional parsers here creates a chicken-and-egg problem.
3536
3637Complex functional parsers are built piece by piece from simpler parsers
3738that are plugged into one another in various ways to perform the desired task.
3839
3940The basic building blocks, or primitives, are parsers that read only a
4041single character. There are frequently a few types of possible input characters,
41−such as letters, numbers, and symbols. For example, `parse(integer, "1")` calls
42+such as letters, numbers, and symbols. For example, `(parse "1" integer)` calls
4243the parsing routine on the string `"1"` and looks for an integer, and so it
4344returns the integer `1`. However, taking into account what was said above about
4445parsers returning the unparsed portion of the string as well, we should
4546represent this return as a tuple. So we should expect something like this:
4647```
47−> parse(integer, "1")
48−(1, "")
49−> parse(integer, "123")
50−(1, "23")
48+> (parse "1" integer)
49+[1 ""]
50+> (parse "123" integer)
51+[1 "23"]
5152```
5253What if we wish to parse the rest of the string? We would need to apply the
53−`parse(integer, -)` function again:
54+`parse` function again:
5455```
55−> parse(integer, parse(integer, "123"))
56−(12, "3")
57−> parse(integer, parse(integer, parse(integer, "123")))
58−(123, "")
56+> (parse (parse "123" integer) integer)
57+[12 "3"]
58+> (parse (parse (parse "123" integer) integer) integer)
59+[123 ""]
5960```
6061So we see that we can parse strings larger than one character by stringing
6162together parsing functions for single characters. Thus in addition to parsing
@@ -76,8 +77,6 @@ are quite a few), but ought to be
7677sufficient to get started with parsing in Hoon and be equipped to discover the
7778remainder yourself.
7879
79−We encourage you to follow along with this section by inputting the relevant
80−commands in dojo as they are presented.
8180
8281## Basic types
8382
@@ -94,7 +93,10 @@ In this section we discuss the types most commonly used for Hoon parsers. In sho
9493++ hair [p=@ud q=@ud]
9594```
9695
97−A `hair` is a pair of `@ud` used to keep track of what has already been parsed.
96+A `hair` is a pair of `@ud` used to keep track of what has already been parsed
97+for stack tracing purposes. This allows the parser to reveal where the problem
98+is in case it hits something unexpected during parsing.
99+
98100`p` represents the column and `q` represents the line.
99101
100102### `nail`
@@ -111,20 +113,23 @@ the original input string (i.e. everything after the location indicated by the
111113
112114For example, if you wish to feed the entire `tape` `"abc"` into a parser, you
113115would pass it as the `nail` `[[1 1] "abc"]`. If the parser successfully parses the first
114−character, the `nail` it returns will be `[[2 1] "bc"]` (though we note that
116+character, the `nail` it returns will be `[[1 2] "bc"]` (though we note that
115117parser outputs are actually `edge`s which contain a `nail`, see the following).
118+The `nail` only matters for book-keeping reasons - it could be any value here
119+since it doesn't refer to a specific portion of the string being input, but only
120+what has theoretically already been parsed up to that point.
116121
117122### `edge`
118123
119124```hoon
120125++ edge [p=hair q=(unit [p=* q=nail])]
121126```
122−An `edge` is the output of a parser. If the parse succeeded, `p` is the location
123−of the original input `tape `up to which the text has been parsed. If the parse
124−failed, `p` will be the first `hair` at which the parse failed.
127+An `edge` is the output of a parser. If parsing succeeded, `p` is the location
128+of the original input `tape `up to which the text has been parsed. If parsing
129+failed, `p` will be the first `hair` at which parsing failed.
125130
126−`q` may be `~`, indicating that the parse has failed .
127−If the parse did not fail, `p.q` is the data structure that is the result of the
131+`q` may be `~`, indicating that parsing has failed .
132+If parsing did not fail, `p.q` is the data structure that is the result of the
128133parse up to this point, while `q.q` is the `nail` which contains the remainder
129134of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.
130135
@@ -134,13 +139,7 @@ of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.
134139++ rule _|:($:nail $:edge)
135140```
136141A `rule` is a gate which takes in a `nail` and returns an `edge` - in other
137−words, a parser. Not totally sure why its written like this, though.
138−
139−Maybe mention this from `/app/shoe.hoon` as well?
140−```hoon
141−|~(nail *(like [? command]))
142−```
143−
142+words, a parser.
144143
145144## Parser builders
146145
@@ -164,20 +163,20 @@ of the input `nail`.
164163[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]
165164```
166165We note that `p.edg` is `[p=1 q=2]`, indicating that the next character to be
167−parsed is in line 1, column 2. `q.edg` is not null, indicating that the parse
166+parsed is in line 1, column 2. `q.edg` is not null, indicating that parsing
168167succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is the same as `p.edg`, which is always the case for
169−`rule`s built using standard library functions when the parse succeeds. Lastly,
168+`rule`s built using standard library functions when parsing succeeds. Lastly,
170169`q.q.edg` is `"bc"`, which is the part of the input `tape` that has yet to be parsed.
171170
172−Now let's see what happens when the parse fails.
171+Now let's see what happens when parsing fails.
173172```
174173> =edg ((just 'b') [[1 1] "abc"])
175174> edg
176175[p=[p=1 q=1] q=~]
177176```
178177Now we have that `p.edg` is the same as the input `hair`, `[1 1]`, meaning the
179−parser has not advanced since the parse failed. `q.edg` is null, indicating that
180−the parse has failed.
178+parser has not advanced since parsing failed. `q.edg` is null, indicating that
179+parsing has failed.
181180
182181Later we will use [+star](#star) to string together a sequence of `+just`s in
183182order to parse multiple characters at once.
@@ -186,7 +185,7 @@ order to parse multiple characters at once.
186185
187186`+jest` is a `rule` builder used to match a `cord`. It takes an input `cord` and
188187produces a `rule` that attempts to match that `cord` against the beginning of
189−the `tape` in an input `nail`.
188+the input.
190189
191190Let's see what happens when we successfully parse the entire input `tape`.
192191```
@@ -235,8 +234,8 @@ and returns a `rule`.
235234### `+cold`
236235
237236`+cold` is a `rule` builder that takes in a constant noun we'll call `cus` and a
238−`rule` we'll call `sef` and returns a `rule` identical to the `sef` except it
239−returns `cus` when it parses successfully.
237+`rule` we'll call `sef`. It returns a `rule` identical to the `sef` except it
238+replaces the parsing result with `cus`.
240239
241240Here we see that `p.q` of the `edge` returned by the `rule` created with `+cold`
242241is `%foo`.
@@ -266,15 +265,17 @@ utilizes parsers. That is to say, you'd like the programmer to only worry about
266265passing `tape`s to the parser, and not have to dress up the `tape` as a `nail`
267266themselves. Thus we have several functions for exactly this purpose.
268267
269−These functions take in either a `tape` or an `@` (typically representing a
270−`cord`) with a `rule` and attempts to parse the input with the `rule`. If the
271−parse succeeds, it returns the result, otherwise it crashes with a stack trace or returns null.
268+These functions take in either a `tape` or a `cord`,
269+alongside a `rule`, and attempt to parse the input with the `rule`. If the
270+parse succeeds, it returns the result. There are crashing and unitized versions
271+of each caller, corresponding to what happens when a parse fails.
272272
273−Here we cover the basics, for complete information including additional examples see [4g: Parsing (Outside Caller)](@/docs/reference/library/4g.md).
273+For additional information including additional examples see [4g: Parsing (Outside Caller)](@/docs/reference/library/4g.md).
274274
275275### Parsing `tape`s
276276
277−`+scan` takes in a `tape` and a `rule` and attempts to parse the `tape` with the `rule`.
277+`+scan` takes in a `tape` and a `rule` and attempts to parse the `tape` with the
278+`rule`.
278279
279280```
280281> (scan "hello" (jest 'hello'))
@@ -284,8 +285,7 @@ Here we cover the basics, for complete information including additional examples
284285'syntax-error'
285286```
286287
287−`+rust` works the same way as `+scan`, except it returns a `unit` of the result,
288−which is null if the parse failed.
288+`+rust` is the unitized version of `+scan`.
289289
290290```
291291> (rust "a" (just 'a'))
@@ -297,12 +297,38 @@ which is null if the parse failed.
297297For the remainder of this tutorial we will make use of `+scan` so that we do not
298298need to deal directly with `nail`s except where it is ilustrative to do so.
299299
300+
301+### Parsing atoms
302+
303+[Recall](@/docs/tutorials/hoon/lists.md) that `cord`s are atoms with the aura
304+`@t` and are typically used to represent strings internally as data, as atoms
305+are faster for the computer to work with than `tape`s, which are `list`s of
306+`@tD` atoms. `+rash` and `+rush` are for parsing atoms, with `+rash` being
307+analogous to `+scan` and `+rush` being analogous to `rust`. Under the hood, `+rash`
308+calls `+scan` after converting the input atom to a `tape`, and `+rush` does
309+similary for `+rust`.
310+
300311## Parser modifiers
301312
302313The standard library provides a number of gates that take a `rule` and
303314produce a new modified `rule` according to some process. We call these _parser modifiers_. These are
304315documented among the [parser builders](@/docs/reference/library/4f.md).
305316
317+
318+### `+ifix`
319+
320+`+ifix` modifies a `rule` so that it matches that `rule` only when it is
321+surrounded on both sides by text that matches a pair of `rule`s, which is discarded.
322+
323+```
324+> (scan "(42)" (ifix [lit rit] (jest '42')))
325+'42'
326+```
327+`+lit` and `+rit` are shorthand for `(just '(')` and `(just ')')`, respectively. All
328+ASCII glyphs have counterparts of this sort, documented
329+[here](@/docs/reference/library/4h.md).
330+
331+
306332### `+star` {#star}
307333
308334`+star` is used to apply a `rule` repeatedly. Recall that `+just` only parses
@@ -325,20 +351,6 @@ and we note that the parsing ceases when it fails.
325351[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q="b"]]]]
326352```
327353
328−### `+ifix`
329−
330−`+ifix` modifies a `rule` so that it matches that `rule` only when it is
331−surrounded on both sides by text that matches a pair of `rule`s, which is discarded.
332−
333−```
334−> (scan "(42)" (ifix [lit rit] (jest '42')))
335−'42'
336−```
337−`+lit` and `+rit` are shorthand for `(just '(')` and `(just ')')`, respectively. All
338−ASCII glyphs have counterparts of this sort, documented
339−[here](@/docs/reference/library/4h.md).
340−
341−
342354## Parser combinators
343355
344356Building complex parsers from simpler parsers is accomplished in Hoon with the
@@ -349,7 +361,7 @@ combinators, then we examine more closely how `;~` is used to chain them togethe
349361
350362The syntax to combine `rule`s is
351363```hoon
352−;~(combinator [list rule])
364+;~(combinator rule1 rule2 ... ruleN)
353365```
354366The `rule`s are composed together using the combinator as an
355367intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns
@@ -433,15 +445,7 @@ is more than one `rule`.
433445syntax error
434446```
435447
436−## Parsing atoms
437448
438−[Recall](@/docs/tutorials/hoon/hoon-school/lists.md) that `cord`s are atoms with the aura
439−`@t` and are typically used to represent strings internally as data, as atoms
440−are faster for the computer to work with than `tape`s, which are `list`s of
441−`@tD` atoms. `+rash` and `+rush` are for parsing atoms, with `+rash` being
442−analogous to `+scan` and `+rush` being analogous to `rust`. Under the hood, `+rash`
443−calls `+scan` after converting the input atom to a `tape`, and `+rush` does
444−similary for `+rust`.
445449
446450## Parsing numbers
447451
@@ -462,7 +466,7 @@ Naively attempting to write a recursive `rule`, i.e. like
462466```
463467> |-(;~(pose den ;~(pose $ (easy ~))))
464468```
465−results in an error, as the compiler will attempt to build an infinite function.
469+results in an error.
466470```
467471-find.,.+6
468472-find.,.+6
@@ -475,12 +479,20 @@ as the bunt value of the type that the `rule` produces, as well as a gate that
475479accepts a `rule`. `+knee` produces a `rule` that implements any recursive calls
476480in the `rule` in a manner acceptable to the compiler. Thus the preferred manner
477481to write the above `rule` is as follows:
482+
478483```hoon
479−|-(;~(plug prn ;~(pose (knee *tape |.(^$)) (easy ~))))
484+++ pars
485+ |-
486+ ;~ plug prn
487+ ;~ pose
488+ (knee *tape |.(^$))
489+ (easy ~)
490+ ==
491+ ==
480492```
481493
482494You may want to utilize the `~+` rune when writing recursive parsers to cache
483−the results as you go to improve performance. In [parsing arithmetic
495+the parser to improve performance. In [parsing arithmetic
484496expressions](#parsing-arithmetic-expressions) we will be writing a recursive
485497parser making use of `+knee` and `~+` throughout.
486498
@@ -492,7 +504,7 @@ arithmetic expressions in Hoon. That is, we will make a `rule` that takes in
492504`tape`s of the form `"(2+3)*4"` and returns `20` as a `@ud`.
493505
494506We call a `tape` consisting of some consistent arithmetic string of numbers,
495−`+,*`, `(`, and `)` an _expression_. We wish to build a `rule` that takes in an
507+`+`, `*`, `(`, and `)` an _expression_. We wish to build a `rule` that takes in an
496508expression and returns the result of the arithmetic computation described by the
497509expression as a `@ud`.
498510
@@ -505,31 +517,28 @@ First lets look at the code we're going to use, and then dive into explaining
505517it. If you'd like to follow along, save the following as `expr-parse.hoon` in
506518your `gen/` folder.
507519```hoon
508−:: This generator parses arithmetic expressions
520+:: expr-parse: parse arithmetic expressions
509521::
510−|= [xprs=tape]
511−=<
512−(print xprs)
513−|%
514−++ int @ud
515−++ print
516− |= [message=tape]
517− (scan message expr)
522+|= xprs=tape
523+|^ (scan xprs tape)
518524++ factor
519− %+ knee *int
520− |. ~+ ;~ pose
525+ %+ knee *@ud
526+ |. ~+
527+ ;~ pose
521528 dem
522529 (ifix [lit rit] expr)
523530 ==
524531++ turm
525− %+ knee *int
526− |. ~+ ;~ pose
532+ %+ knee *@ud
533+ |. ~+
534+ ;~ pose
527535 ((slug mul) tar ;~(pose factor turm))
528536 factor
529537 ==
530538++ expr
531− %+ knee *int
532− |. ~+ ;~ pose
539+ %+ knee *@ud
540+ |. ~+
541+ ;~ pose
533542 ((slug add) lus ;~(pose turm expr))
534543 turm
535544 ==
@@ -550,8 +559,9 @@ for expressions.
550559
551560```hoon
552561++ factor
553− %+ knee *int
554− |. ~+ ;~ pose
562+ %+ knee *@ud
563+ |. ~+
564+ ;~ pose
555565 dem
556566 (ifix [lit rit] expr)
557567 ==
@@ -567,26 +577,24 @@ with `+pose`, which says to try each rule in succession until one of them works.
567577
568578Since expressions ultimately reduce to factors, we are actually building a
569579recursive rule. Thus we need to make use of `+knee`. The first argument for
570−`+knee` is `*int`, since our final answer should be a `@ud`, to which we have
571−assigned the macro `int`.
580+`+knee` is `*@ud`, since our final answer should be a `@ud`.
572581
573582Then follows the definition of the gate utilized by `+knee`:
574583```hoon
575− |. ~+ ;~ pose
584+ |. ~+
585+ ;~ pose
576586 dem
577587 (ifix [lit rit] expr)
578588 ==
579589```
580−`|.` is used instead of `|-` since... `~+` is used to cache results as we go -
581−there is no point in computing `2+2` more than once. Then follows the `rule` we
582−described above.
590+`~+` is used to cache the parser, so that it does not need to be computed over
591+and over again. Then it follows the `rule` we described above.
583592
584593#### Parsing expressions
585594
586595An _expression_ is either a term plus an expression or a term.
587596
588−In the case of a
589−term plus an expression, we actually must compute what that equals. Thus we will
597+In the case of a term plus an expression, we actually must compute what that equals. Thus we will
590598make use of [`+slug`](@/docs/reference/library/4f.md#slug), which parses a
591599delimited list into `tape`s separated by a given delimiter and then composes
592600them by folding with a binary gate. In this case, our delimiter is `+` and our
@@ -595,15 +603,15 @@ and expressions separated by luses, parse each term and expression until they
595603reduce to a `@ud`, and then add them together. This is accomplished with the
596604rule `((slug add) lus ;~(pose turm expr))`.
597605
598−If the above `rule` does not parse the expression, it must be a `turm`, so we
599−pass the `tape` to `+turm` to be evaluated. Again we use `;~` and `pose` to
606+If the above `rule` does not parse the expression, it must be a `turm`, so the
607+`tape` is automatically passed to `+turm` to be evaluated. Again we use `;~` and `pose` to
600608accomplish this:
601609
602610```hoon
603611;~ pose
604− ((slug add) lus ;~(pose turm expr))
605− turm
606− ==
612+ ((slug add) lus ;~(pose turm expr))
613+ turm
614+==
607615```
608616
609617The rest of the `+expr` arm is structured just like how `+factor` is, and for
@@ -614,12 +622,13 @@ the same reasons.
614622
615623A _term_ is either a factor times a term or a factor. This is handled similarly
616624for expressions, we just need to swap `lus` for `tar`, `add` for `mul`, and
617−`;~(pose factor turm)` instead of `;(pose turm expr)`.
625+`;~(pose factor turm)` instead of `;~(pose turm expr)`.
618626
619627```hoon
620628++ expr
621− %+ knee *int
622− |. ~+ ;~ pose
629+ %+ knee *@ud
630+ |. ~+
631+ ;~ pose
623632 ((slug add) lus ;~(pose turm expr))
624633 turm
625634 ==