@@ -11,9 +11,9 @@ knowledge of parsing is required, and we will explain the basic structure of how
11parsing works in a purely functional language such as Hoon before moving on to
11parsing works in a purely functional language such as Hoon before moving on to
12how it is implemented in Hoon.
12how it is implemented in Hoon.
14−## What is parsing? {#what-is-parsing}
14+## What is parsing? {#what-is-parsingk
16−A program which takes a raw sequence of characters as an input and produces a data
16+A program which takes a raw sequence of characters as an input knd produces a data
17structure as an output is known as a _parser_. The data structure produced
17structure as an output is known as a _parser_. The data structure produced
18depends on the use case, but often it may be represented as a tree and the
18depends on the use case, but often it may be represented as a tree and the
19output is thought of as a structural representation of the input. Parsers are ubiquitous in
19output is thought of as a structural representation of the input. Parsers are ubiquitous in
@@ -24,38 +24,39 @@ Parsing a string is rarely done all at once. Instead, it is usually done
24character-by-character, and the return contains the data structure representing
24character-by-character, and the return contains the data structure representing
25what has been parsed thus far as well as the remainder of the string to be
25what has been parsed thus far as well as the remainder of the string to be
26parsed. They also need to be able to fail in case the input is improperly
26parsed. They also need to be able to fail in case the input is improperly
27−formed. We will see each of these standard practices implemented in Hoon below.
27+formed. We will see each of these standard practices implemented in [Hoon below](#parsing-in-hoon).
29## Functional parsers
29## Functional parsers
31How parsers are built varies substantially depending on what sort of programming
31How parsers are built varies substantially depending on what sort of programming
32language it is written in. As Hoon is a functional programming language, we will
32language it is written in. As Hoon is a functional programming language, we will
33be focused on understanding _functional parsers_, also known as _combinator
33be focused on understanding _functional parsers_, also known as _combinator
34−parsers_.
34+parsers_. In this section we will make light use of pseudocode, as introducing
35+the Hoon to describe functional parsers here creates a chicken-and-egg problem.
36Complex functional parsers are built piece by piece from simpler parsers
37Complex functional parsers are built piece by piece from simpler parsers
37that are plugged into one another in various ways to perform the desired task.
38that are plugged into one another in various ways to perform the desired task.
39The basic building blocks, or primitives, are parsers that read only a
40The basic building blocks, or primitives, are parsers that read only a
40single character. There are frequently a few types of possible input characters,
41single character. There are frequently a few types of possible input characters,
41−such as letters, numbers, and symbols. For example, `parse(integer, "1")` calls
42+such as letters, numbers, and symbols. For example, `(parse "1" integer)` calls
42the parsing routine on the string `"1"` and looks for an integer, and so it
43the parsing routine on the string `"1"` and looks for an integer, and so it
43returns the integer `1`. However, taking into account what was said above about
44returns the integer `1`. However, taking into account what was said above about
44parsers returning the unparsed portion of the string as well, we should
45parsers returning the unparsed portion of the string as well, we should
45represent this return as a tuple. So we should expect something like this:
46represent this return as a tuple. So we should expect something like this:
47−> parse(integer, "1")
48+> (parse "1" integer)
49−> parse(integer, "123")
50+> (parse "123" integer)
52What if we wish to parse the rest of the string? We would need to apply the
53What if we wish to parse the rest of the string? We would need to apply the
53−`parse(integer, -)` function again:
54+`parse` function again:
55−> parse(integer, parse(integer, "123"))
56+> (parse (parse "123" integer) integer)
57−> parse(integer, parse(integer, parse(integer, "123")))
58+> (parse (parse (parse "123" integer) integer) integer)
60So we see that we can parse strings larger than one character by stringing
61So we see that we can parse strings larger than one character by stringing
61together parsing functions for single characters. Thus in addition to parsing
62together parsing functions for single characters. Thus in addition to parsing
@@ -76,8 +77,6 @@ are quite a few), but ought to be
76sufficient to get started with parsing in Hoon and be equipped to discover the
77sufficient to get started with parsing in Hoon and be equipped to discover the
77remainder yourself.
78remainder yourself.
79−We encourage you to follow along with this section by inputting the relevant
80−commands in dojo as they are presented.
82## Basic types
81## Basic types
@@ -94,7 +93,10 @@ In this section we discuss the types most commonly used for Hoon parsers. In sho
94++ hair [p=@ud q=@ud]
93++ hair [p=@ud q=@ud]
97−A `hair` is a pair of `@ud` used to keep track of what has already been parsed.
96+A `hair` is a pair of `@ud` used to keep track of what has already been parsed
97+for stack tracing purposes. This allows the parser to reveal where the problem
98+is in case it hits something unexpected during parsing.
98`p` represents the column and `q` represents the line.
100`p` represents the column and `q` represents the line.
100### `nail`
102### `nail`
@@ -111,20 +113,23 @@ the original input string (i.e. everything after the location indicated by the
112For example, if you wish to feed the entire `tape` `"abc"` into a parser, you
114For example, if you wish to feed the entire `tape` `"abc"` into a parser, you
113would pass it as the `nail` `[[1 1] "abc"]`. If the parser successfully parses the first
115would pass it as the `nail` `[[1 1] "abc"]`. If the parser successfully parses the first
114−character, the `nail` it returns will be `[[2 1] "bc"]` (though we note that
116+character, the `nail` it returns will be `[[1 2] "bc"]` (though we note that
115parser outputs are actually `edge`s which contain a `nail`, see the following).
117parser outputs are actually `edge`s which contain a `nail`, see the following).
118+The `nail` only matters for book-keeping reasons - it could be any value here
119+since it doesn't refer to a specific portion of the string being input, but only
120+what has theoretically already been parsed up to that point.
117### `edge`
122### `edge`
120++ edge [p=hair q=(unit [p=* q=nail])]
125++ edge [p=hair q=(unit [p=* q=nail])]
122−An `edge` is the output of a parser. If the parse succeeded, `p` is the location
127+An `edge` is the output of a parser. If parsing succeeded, `p` is the location
123−of the original input `tape `up to which the text has been parsed. If the parse
128+of the original input `tape `up to which the text has been parsed. If parsing
124−failed, `p` will be the first `hair` at which the parse failed.
129+failed, `p` will be the first `hair` at which parsing failed.
126−`q` may be `~`, indicating that the parse has failed .
131+`q` may be `~`, indicating that parsing has failed .
127−If the parse did not fail, `p.q` is the data structure that is the result of the
132+If parsing did not fail, `p.q` is the data structure that is the result of the
128parse up to this point, while `q.q` is the `nail` which contains the remainder
133parse up to this point, while `q.q` is the `nail` which contains the remainder
129of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.
134of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.
@@ -134,13 +139,7 @@ of what is to be parsed. If `q` is not null, `p` and `p.q.q` are identical.
134++ rule _|:($:nail $:edge)
139++ rule _|:($:nail $:edge)
136A `rule` is a gate which takes in a `nail` and returns an `edge` - in other
141A `rule` is a gate which takes in a `nail` and returns an `edge` - in other
137−words, a parser. Not totally sure why its written like this, though.
142+words, a parser.
139−Maybe mention this from `/app/shoe.hoon` as well?
141−|~(nail *(like [? command]))
145## Parser builders
144## Parser builders
@@ -164,20 +163,20 @@ of the input `nail`.
164[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]
163[p=[p=1 q=2] q=[~ [p='a' q=[p=[p=1 q=2] q="bc"]]]]
166We note that `p.edg` is `[p=1 q=2]`, indicating that the next character to be
165We note that `p.edg` is `[p=1 q=2]`, indicating that the next character to be
167−parsed is in line 1, column 2. `q.edg` is not null, indicating that the parse
166+parsed is in line 1, column 2. `q.edg` is not null, indicating that parsing
168succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is the same as `p.edg`, which is always the case for
167succeeded. `p.q.edg` is `'a'`, which is the result of the parse. `p.q.q.edg` is the same as `p.edg`, which is always the case for
169−`rule`s built using standard library functions when the parse succeeds. Lastly,
168+`rule`s built using standard library functions when parsing succeeds. Lastly,
170`q.q.edg` is `"bc"`, which is the part of the input `tape` that has yet to be parsed.
169`q.q.edg` is `"bc"`, which is the part of the input `tape` that has yet to be parsed.
172−Now let's see what happens when the parse fails.
171+Now let's see what happens when parsing fails.
174> =edg ((just 'b') [[1 1] "abc"])
173> =edg ((just 'b') [[1 1] "abc"])
176[p=[p=1 q=1] q=~]
175[p=[p=1 q=1] q=~]
178Now we have that `p.edg` is the same as the input `hair`, `[1 1]`, meaning the
177Now we have that `p.edg` is the same as the input `hair`, `[1 1]`, meaning the
179−parser has not advanced since the parse failed. `q.edg` is null, indicating that
178+parser has not advanced since parsing failed. `q.edg` is null, indicating that
180−the parse has failed.
179+parsing has failed.
182Later we will use [+star](#star) to string together a sequence of `+just`s in
181Later we will use [+star](#star) to string together a sequence of `+just`s in
183order to parse multiple characters at once.
182order to parse multiple characters at once.
@@ -186,7 +185,7 @@ order to parse multiple characters at once.
187`+jest` is a `rule` builder used to match a `cord`. It takes an input `cord` and
186`+jest` is a `rule` builder used to match a `cord`. It takes an input `cord` and
188produces a `rule` that attempts to match that `cord` against the beginning of
187produces a `rule` that attempts to match that `cord` against the beginning of
189−the `tape` in an input `nail`.
188+the input.
191Let's see what happens when we successfully parse the entire input `tape`.
190Let's see what happens when we successfully parse the entire input `tape`.
@@ -235,8 +234,8 @@ and returns a `rule`.
235### `+cold`
234### `+cold`
237`+cold` is a `rule` builder that takes in a constant noun we'll call `cus` and a
236`+cold` is a `rule` builder that takes in a constant noun we'll call `cus` and a
238−`rule` we'll call `sef` and returns a `rule` identical to the `sef` except it
237+`rule` we'll call `sef`. It returns a `rule` identical to the `sef` except it
239−returns `cus` when it parses successfully.
238+replaces the parsing result with `cus`.
241Here we see that `p.q` of the `edge` returned by the `rule` created with `+cold`
240Here we see that `p.q` of the `edge` returned by the `rule` created with `+cold`
242is `%foo`.
241is `%foo`.
@@ -266,15 +265,17 @@ utilizes parsers. That is to say, you'd like the programmer to only worry about
266passing `tape`s to the parser, and not have to dress up the `tape` as a `nail`
265passing `tape`s to the parser, and not have to dress up the `tape` as a `nail`
267themselves. Thus we have several functions for exactly this purpose.
266themselves. Thus we have several functions for exactly this purpose.
269−These functions take in either a `tape` or an `@` (typically representing a
268+These functions take in either a `tape` or a `cord`,
270−`cord`) with a `rule` and attempts to parse the input with the `rule`. If the
269+alongside a `rule`, and attempt to parse the input with the `rule`. If the
271−parse succeeds, it returns the result, otherwise it crashes with a stack trace or returns null.
270+parse succeeds, it returns the result. There are crashing and unitized versions
271+of each caller, corresponding to what happens when a parse fails.
273−Here we cover the basics, for complete information including additional examples see [4g: Parsing (Outside Caller)](@/docs/reference/library/4g.md).
273+For additional information including additional examples see [4g: Parsing (Outside Caller)](@/docs/reference/library/4g.md).
275### Parsing `tape`s
275### Parsing `tape`s
277−`+scan` takes in a `tape` and a `rule` and attempts to parse the `tape` with the `rule`.
277+`+scan` takes in a `tape` and a `rule` and attempts to parse the `tape` with the
280> (scan "hello" (jest 'hello'))
281> (scan "hello" (jest 'hello'))
@@ -284,8 +285,7 @@ Here we cover the basics, for complete information including additional examples
284'syntax-error'
285'syntax-error'
287−`+rust` works the same way as `+scan`, except it returns a `unit` of the result,
288+`+rust` is the unitized version of `+scan`.
288−which is null if the parse failed.
291> (rust "a" (just 'a'))
291> (rust "a" (just 'a'))
@@ -297,12 +297,38 @@ which is null if the parse failed.
297For the remainder of this tutorial we will make use of `+scan` so that we do not
297For the remainder of this tutorial we will make use of `+scan` so that we do not
298need to deal directly with `nail`s except where it is ilustrative to do so.
298need to deal directly with `nail`s except where it is ilustrative to do so.
303+[Recall](@/docs/tutorials/hoon/lists.md) that `cord`s are atoms with the aura
304+`@t` and are typically used to represent strings internally as data, as atoms
305+are faster for the computer to work with than `tape`s, which are `list`s of
306+`@tD` atoms. `+rash` and `+rush` are for parsing atoms, with `+rash` being
307+analogous to `+scan` and `+rush` being analogous to `rust`. Under the hood, `+rash`
308+calls `+scan` after converting the input atom to a `tape`, and `+rush` does
309+similary for `+rust`.
300## Parser modifiers
311## Parser modifiers
302The standard library provides a number of gates that take a `rule` and
313The standard library provides a number of gates that take a `rule` and
303produce a new modified `rule` according to some process. We call these _parser modifiers_. These are
314produce a new modified `rule` according to some process. We call these _parser modifiers_. These are
304documented among the [parser builders](@/docs/reference/library/4f.md).
315documented among the [parser builders](@/docs/reference/library/4f.md).
320+`+ifix` modifies a `rule` so that it matches that `rule` only when it is
321+surrounded on both sides by text that matches a pair of `rule`s, which is discarded.
324+> (scan "(42)" (ifix [lit rit] (jest '42')))
327+`+lit` and `+rit` are shorthand for `(just '(')` and `(just ')')`, respectively. All
328+ASCII glyphs have counterparts of this sort, documented
329+[here](@/docs/reference/library/4h.md).
306### `+star` {#star}
332### `+star` {#star}
308`+star` is used to apply a `rule` repeatedly. Recall that `+just` only parses
334`+star` is used to apply a `rule` repeatedly. Recall that `+just` only parses
@@ -325,20 +351,6 @@ and we note that the parsing ceases when it fails.
325[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q="b"]]]]
351[p=[p=1 q=4] q=[~ [p=[i='a' t=<|a a|>] q=[p=[p=1 q=4] q="b"]]]]
330−`+ifix` modifies a `rule` so that it matches that `rule` only when it is
331−surrounded on both sides by text that matches a pair of `rule`s, which is discarded.
334−> (scan "(42)" (ifix [lit rit] (jest '42')))
337−`+lit` and `+rit` are shorthand for `(just '(')` and `(just ')')`, respectively. All
338−ASCII glyphs have counterparts of this sort, documented
339−[here](@/docs/reference/library/4h.md).
342## Parser combinators
354## Parser combinators
344Building complex parsers from simpler parsers is accomplished in Hoon with the
356Building complex parsers from simpler parsers is accomplished in Hoon with the
@@ -349,7 +361,7 @@ combinators, then we examine more closely how `;~` is used to chain them togethe
350The syntax to combine `rule`s is
362The syntax to combine `rule`s is
352−;~(combinator [list rule])
364+;~(combinator rule1 rule2 ... ruleN)
354The `rule`s are composed together using the combinator as an
366The `rule`s are composed together using the combinator as an
355intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns
367intermediate function, which takes product of a `rule` (an `edge`) and a `rule` and turns
@@ -433,15 +445,7 @@ is more than one `rule`.
433syntax error
445syntax error
438−[Recall](@/docs/tutorials/hoon/hoon-school/lists.md) that `cord`s are atoms with the aura
439−`@t` and are typically used to represent strings internally as data, as atoms
440−are faster for the computer to work with than `tape`s, which are `list`s of
441−`@tD` atoms. `+rash` and `+rush` are for parsing atoms, with `+rash` being
442−analogous to `+scan` and `+rush` being analogous to `rust`. Under the hood, `+rash`
443−calls `+scan` after converting the input atom to a `tape`, and `+rush` does
444−similary for `+rust`.
446## Parsing numbers
450## Parsing numbers
@@ -462,7 +466,7 @@ Naively attempting to write a recursive `rule`, i.e. like
463> |-(;~(pose den ;~(pose $ (easy ~))))
467> |-(;~(pose den ;~(pose $ (easy ~))))
465−results in an error, as the compiler will attempt to build an infinite function.
469+results in an error.
467-find.,.+6
471-find.,.+6
468-find.,.+6
472-find.,.+6
@@ -475,12 +479,20 @@ as the bunt value of the type that the `rule` produces, as well as a gate that
475accepts a `rule`. `+knee` produces a `rule` that implements any recursive calls
479accepts a `rule`. `+knee` produces a `rule` that implements any recursive calls
476in the `rule` in a manner acceptable to the compiler. Thus the preferred manner
480in the `rule` in a manner acceptable to the compiler. Thus the preferred manner
477to write the above `rule` is as follows:
481to write the above `rule` is as follows:
479−|-(;~(plug prn ;~(pose (knee *tape |.(^$)) (easy ~))))
488+ (knee *tape |.(^$))
482You may want to utilize the `~+` rune when writing recursive parsers to cache
494You may want to utilize the `~+` rune when writing recursive parsers to cache
483−the results as you go to improve performance. In [parsing arithmetic
495+the parser to improve performance. In [parsing arithmetic
484expressions](#parsing-arithmetic-expressions) we will be writing a recursive
496expressions](#parsing-arithmetic-expressions) we will be writing a recursive
485parser making use of `+knee` and `~+` throughout.
497parser making use of `+knee` and `~+` throughout.
@@ -492,7 +504,7 @@ arithmetic expressions in Hoon. That is, we will make a `rule` that takes in
492`tape`s of the form `"(2+3)*4"` and returns `20` as a `@ud`.
504`tape`s of the form `"(2+3)*4"` and returns `20` as a `@ud`.
494We call a `tape` consisting of some consistent arithmetic string of numbers,
506We call a `tape` consisting of some consistent arithmetic string of numbers,
495−`+,*`, `(`, and `)` an _expression_. We wish to build a `rule` that takes in an
507+`+`, `*`, `(`, and `)` an _expression_. We wish to build a `rule` that takes in an
496expression and returns the result of the arithmetic computation described by the
508expression and returns the result of the arithmetic computation described by the
497expression as a `@ud`.
509expression as a `@ud`.
@@ -505,31 +517,28 @@ First lets look at the code we're going to use, and then dive into explaining
505it. If you'd like to follow along, save the following as `expr-parse.hoon` in
517it. If you'd like to follow along, save the following as `expr-parse.hoon` in
506your `gen/` folder.
518your `gen/` folder.
508−:: This generator parses arithmetic expressions
520+:: expr-parse: parse arithmetic expressions
510−|= [xprs=tape]
522+|= xprs=tape
511−=<
523+|^ (scan xprs tape)
519− %+ knee *int
525+ %+ knee *@ud
520− |. ~+ ;~ pose
527+ ;~ pose
522 (ifix [lit rit] expr)
529 (ifix [lit rit] expr)
525− %+ knee *int
532+ %+ knee *@ud
526− |. ~+ ;~ pose
534+ ;~ pose
527 ((slug mul) tar ;~(pose factor turm))
535 ((slug mul) tar ;~(pose factor turm))
531− %+ knee *int
539+ %+ knee *@ud
532− |. ~+ ;~ pose
541+ ;~ pose
533 ((slug add) lus ;~(pose turm expr))
542 ((slug add) lus ;~(pose turm expr))
@@ -550,8 +559,9 @@ for expressions.
553− %+ knee *int
562+ %+ knee *@ud
554− |. ~+ ;~ pose
564+ ;~ pose
556 (ifix [lit rit] expr)
566 (ifix [lit rit] expr)
@@ -567,26 +577,24 @@ with `+pose`, which says to try each rule in succession until one of them works.
568Since expressions ultimately reduce to factors, we are actually building a
578Since expressions ultimately reduce to factors, we are actually building a
569recursive rule. Thus we need to make use of `+knee`. The first argument for
579recursive rule. Thus we need to make use of `+knee`. The first argument for
570−`+knee` is `*int`, since our final answer should be a `@ud`, to which we have
580+`+knee` is `*@ud`, since our final answer should be a `@ud`.
571−assigned the macro `int`.
573Then follows the definition of the gate utilized by `+knee`:
582Then follows the definition of the gate utilized by `+knee`:
575− |. ~+ ;~ pose
585+ ;~ pose
577 (ifix [lit rit] expr)
587 (ifix [lit rit] expr)
580−`|.` is used instead of `|-` since... `~+` is used to cache results as we go -
590+`~+` is used to cache the parser, so that it does not need to be computed over
581−there is no point in computing `2+2` more than once. Then follows the `rule` we
582−described above.
591+and over again. Then it follows the `rule` we described above.
584#### Parsing expressions
593#### Parsing expressions
586An _expression_ is either a term plus an expression or a term.
595An _expression_ is either a term plus an expression or a term.
589−term plus an expression, we actually must compute what that equals. Thus we will
597+In the case of a term plus an expression, we actually must compute what that equals. Thus we will
590make use of [`+slug`](@/docs/reference/library/4f.md#slug), which parses a
598make use of [`+slug`](@/docs/reference/library/4f.md#slug), which parses a
591delimited list into `tape`s separated by a given delimiter and then composes
599delimited list into `tape`s separated by a given delimiter and then composes
592them by folding with a binary gate. In this case, our delimiter is `+` and our
600them by folding with a binary gate. In this case, our delimiter is `+` and our
@@ -595,15 +603,15 @@ and expressions separated by luses, parse each term and expression until they
595reduce to a `@ud`, and then add them together. This is accomplished with the
603reduce to a `@ud`, and then add them together. This is accomplished with the
596rule `((slug add) lus ;~(pose turm expr))`.
604rule `((slug add) lus ;~(pose turm expr))`.
598−If the above `rule` does not parse the expression, it must be a `turm`, so we
606+If the above `rule` does not parse the expression, it must be a `turm`, so the
599−pass the `tape` to `+turm` to be evaluated. Again we use `;~` and `pose` to
607+`tape` is automatically passed to `+turm` to be evaluated. Again we use `;~` and `pose` to
600accomplish this:
608accomplish this:
604− ((slug add) lus ;~(pose turm expr))
612+ ((slug add) lus ;~(pose turm expr))
609The rest of the `+expr` arm is structured just like how `+factor` is, and for
617The rest of the `+expr` arm is structured just like how `+factor` is, and for
@@ -614,12 +622,13 @@ the same reasons.
615A _term_ is either a factor times a term or a factor. This is handled similarly
623A _term_ is either a factor times a term or a factor. This is handled similarly
616for expressions, we just need to swap `lus` for `tar`, `add` for `mul`, and
624for expressions, we just need to swap `lus` for `tar`, `add` for `mul`, and
617−`;~(pose factor turm)` instead of `;(pose turm expr)`.
625+`;~(pose factor turm)` instead of `;~(pose turm expr)`.
621− %+ knee *int
629+ %+ knee *@ud
622− |. ~+ ;~ pose
631+ ;~ pose
623 ((slug add) lus ;~(pose turm expr))
632 ((slug add) lus ;~(pose turm expr))